.NET Inside Out Part 18 — Handling StackOverflowException with custom CLR host

This is the eighteenth part of the .NET Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 – Virtual and non-virtual calls in C#

We saw in Part 8 — Handling Stack Overflow Exception in C# with VEH how to handle SOE in C#. Obviously, that method is very unsafe and shouldn’t be used in production. But what can we do if we really need to handle SOE?

The answer is — nothing. .NET will always kill the CLR, we cannot stop that, we cannot do anything about that. We can try handling the situation to log errors but it is very dangerous and there is no reliable way to do so. Let’s try, at least.

Let’s begin with the following application in C#:

Nothing big, we just loop and print the argument. The thing here is we cannot run this application just like that. To handle StackOverflowException we need to use custom CLR loader. So let’s use this C++ code:

Let’s go through this line by line. Initially we just import some libraries to have an access to CLR methods. Next, in lines 27-35 we just load .NET 4.5 (it’s still based on version 4). You can load different .NET here (for instance version 2).

Next, in line 43 we tell .NET to unload application domain in case of SOE. The name says eRudeUnloadAppDomain which should suggest you that this is not nice.

Next, in line 46 we start the runtime and in line 49 we add VEH handler to report the SOE.

Handler is in lines 13-22. If you run this application you see the following output:

There are some more exceptions afterwards related to closing the application but they are not interesting. You can see multiple exceptions being reported. These are:

What is “Soft SOE”? There are actually two types of StackOverflowException. One is reported by MMU and has code 0xC00000FD. The other one is thrown by .NET when CLR realizes there is not enough stack, and the exception code is 0xE053534F.

So we can see that we cannot handle SOE but we can at least run some code when it happens and not kill the application. If you comment out line 43 then the application will get terminated as soon as SOE is thrown.

However, this is very tricky. Notice that we do printf and puts in VEH handler. Is it safe? Well, not really. Let’s modify the code:

And now our application crashes and VEH “is not called”. It is called, in fact, but it allocates some memory on the stack and crashes the process. Let’s change it to

And now we get this output:

If you attach the debugger you can actually see that this time we get

To sum up:

  • You cannot stop .NET application from terminating in case of SOE (unless you do the magic from Part 8)
  • You can stop the process from terminating (and continue running in native space)
  • You can execute code on SOE (soft and hard) but it is unreliable (and very dangerous)

So what should you do in reality? If you really need to handle SOE then it seems like you could set some flag atomically in VEH and pause the thread. Then, some other thread could observe the flag, dump the memory when needed, and clean up. You probably cannot do anything on the thread with insufficient stack as it is easy to crash the process.