.NET Inside Out Part 22 – Your application is always multithreaded and it’s not easy to exit properly

This is the twentieth second part of the .NET Inside Out series. For your convenience you can find other parts in the table of contents in Part 1 – Virtual and non-virtual calls in C#

We know that a .NET application is always multithreaded. GC creates threads for running finalizers and (depending on the settings) to run collection. This means that each app has multiple threads running and we may get a race condition when cleaning resources.

Let’s take the following code:

using System;
using System.Diagnostics;
using System.IO;
using System.Threading;

namespace SerializationTester
{
	class Program
	{
		static void Main(string[] args)
		{
			var content = @"abc";
			
			if(args.Length > 0)
			{
				TestFile(content);
			}else {
				while (true)
				{
					Process.Start("SerializationTester.exe", "abc").WaitForExit();
					if (new FileInfo("0").Length - 3 > 0)
					{
						Debugger.Launch();
						Debugger.Break();
						Thread.Sleep(100000000);
					}
					Thread.Sleep(1);
				}
			}
		}

		static bool TestFile(string content)
		{
			var thread1 = new Thread(() =>
			{
				Write(content);
				Thread.Yield();
			});
			var thread2 = new Thread(() => Environment.Exit(1));
			thread1.Start();
			thread2.Start();

			return true;
		}

		static void Write(string content)
		{
			using (StreamWriter writer = new StreamWriter(new FileStream("0", FileMode.Create, FileAccess.Write, FileShare.None)))
			{
				writer.Write(content);
			}
		}
	}
}

using System;

using System.Diagnostics;

using System.IO;

using System.Threading;

namespace SerializationTester

{

class Program

{

static void Main(string[] args)

{

var content = @"abc";

if(args.Length > 0)

{

TestFile(content);

}else {

while (true)

{

Process.Start("SerializationTester.exe", "abc").WaitForExit();

if (new FileInfo("0").Length - 3 > 0)

{

Debugger.Launch();

Debugger.Break();

Thread.Sleep(100000000);

}

Thread.Sleep(1);

}

static bool TestFile(string content)

{

var thread1 = new Thread(() =>

{

Write(content);

Thread.Yield();

});

var thread2 = new Thread(() => Environment.Exit(1));

thread1.Start();

thread2.Start();

return true;

}

static void Write(string content)

{

using (StreamWriter writer = new StreamWriter(new FileStream("0", FileMode.Create, FileAccess.Write, FileShare.None)))

{

writer.Write(content);

}

We run application recursively in two modes. In first mode (bootstraper) we enter the loop in line 18 and then constantly run the application. In other mode we start writing to file in line 16. We create two threads: first of them is writing the content, second is exiting the application with Environment.Exit.

Then, bootstraper checks the file length in line 21. If you run this for long enough (few seconds on my machine) you should see that the file is longer than 3 characters. Actually, instead of being abc it is abcabc. How is that possible?

The reason is in line 39 where we terminate the application. What’s happening? Let’s grab dnspy and decompile the code writing to the file. If we go through the using construct we can see that it calls FileStream.WriteFileNative method under the hood through the Dispose method. So let’s see, what is exactly going on (for .NET Framework):

using System;
using System.Diagnostics;
using System.IO;
using System.Threading;
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using Microsoft.Win32.SafeHandles;

namespace SerializationTester
{
    class Program
	{
        static string stacktrace = null;

		static void Main(string[] args)
		{
			var content = @"abc";

            if (args.Length > 0)
            {
                HijackMethod(
                    typeof(FileStream).GetMethod("WriteFileNative", BindingFlags.NonPublic | BindingFlags.Instance),
                    typeof(Program).GetMethod(nameof(Hijacked), BindingFlags.Instance | BindingFlags.Public)
                );
                TestFile(content);
            }
            else
            {

                while (true)
                {
                    var process = Process.Start(new ProcessStartInfo
                    {
                        Arguments = "abc",
                        RedirectStandardOutput = true,
                        FileName = "SerializationTester.exe",
                        UseShellExecute = false
                    });
                    process.WaitForExit();
                    var output = process.StandardOutput.ReadToEnd();

                    if (new FileInfo("0").Length - 3 > 0 || !string.IsNullOrEmpty(output))
                    {
                        Console.WriteLine(output);
                        Debugger.Launch();
                        Debugger.Break();
                        Thread.Sleep(100000000);
                    }
                    Thread.Sleep(1);
                }
            }
		}

		static bool TestFile(string content)
		{
			var thread1 = new Thread(() =>
			{
				Write(content);
				Thread.Yield();
			});
			var thread2 = new Thread(() => Environment.Exit(1));
			thread1.Start();
			thread2.Start();

			return true;
		}

		static void Write(string content)
		{
			using (StreamWriter writer = new StreamWriter(new FileStream("0", FileMode.Create, FileAccess.Write, FileShare.None)))
			{
				writer.Write(content);
			}
		}

        public unsafe int Hijacked(SafeFileHandle handle, byte[] bytes, int offset, int count, NativeOverlapped* overlapped, out int hr)
        {
            if(stacktrace != null)
            {
                Console.WriteLine("Current stacktrace");
                Console.WriteLine(Environment.StackTrace);
                Console.WriteLine("Other stacktrace");
                Console.WriteLine(stacktrace);
            }
            
            stacktrace = Environment.StackTrace;
            Thread.Yield();

            hr = 0;
            return 3;
        }

        public static void HijackMethod(MethodBase source, MethodBase target)
        {
            RuntimeHelpers.PrepareMethod(source.MethodHandle);
            RuntimeHelpers.PrepareMethod(target.MethodHandle);

            var sourceAddress = (IntPtr)(Marshal.ReadInt32(source.MethodHandle.Value, 8) + (int)source.MethodHandle.Value + 8);
            var targetAddress = (long)target.MethodHandle.GetFunctionPointer();

            UnlockPage((int)sourceAddress);
            UnlockPage((int)targetAddress);


            int offset = (int)(targetAddress - (long)sourceAddress - 4 - 1); // four bytes for relative address and one byte for opcode

            byte[] instruction = {
                0xE9, // Long jump relative instruction
				(byte)(offset & 0xFF),
                (byte)((offset >> 8) & 0xFF),
                (byte)((offset >> 16) & 0xFF),
                (byte)((offset >> 24) & 0xFF)
            };

            Marshal.Copy(instruction, 0, sourceAddress, instruction.Length);
        }

        public enum Protection
        {
            PAGE_NOACCESS = 0x01,
            PAGE_READONLY = 0x02,
            PAGE_READWRITE = 0x04,
            PAGE_WRITECOPY = 0x08,
            PAGE_EXECUTE = 0x10,
            PAGE_EXECUTE_READ = 0x20,
            PAGE_EXECUTE_READWRITE = 0x40,
            PAGE_EXECUTE_WRITECOPY = 0x80,
            PAGE_GUARD = 0x100,
            PAGE_NOCACHE = 0x200,
            PAGE_WRITECOMBINE = 0x400
        }

        [DllImport("kernel32.dll", SetLastError = true)]
        static extern bool VirtualProtect(IntPtr lpAddress, uint dwSize, uint flNewProtect, out uint lpflOldProtect);

        private static void UnlockPage(int address)
        {
            uint old;
            VirtualProtect((IntPtr)address, 4096, (uint)Protection.PAGE_EXECUTE_READWRITE, out old);
        }
    }
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

using System;

using System.Diagnostics;

using System.IO;

using System.Threading;

using System.Reflection;

using System.Runtime.CompilerServices;

using System.Runtime.InteropServices;

using Microsoft.Win32.SafeHandles;

namespace SerializationTester

{

class Program

{

static string stacktrace = null;

static void Main(string[] args)

{

var content = @"abc";

if (args.Length > 0)

{

HijackMethod(

typeof(FileStream).GetMethod("WriteFileNative", BindingFlags.NonPublic | BindingFlags.Instance),

typeof(Program).GetMethod(nameof(Hijacked), BindingFlags.Instance | BindingFlags.Public)

);

TestFile(content);

}

else

{

while (true)

{

var process = Process.Start(new ProcessStartInfo

{

Arguments = "abc",

RedirectStandardOutput = true,

FileName = "SerializationTester.exe",

UseShellExecute = false

});

process.WaitForExit();

var output = process.StandardOutput.ReadToEnd();

if (new FileInfo("0").Length - 3 > 0 || !string.IsNullOrEmpty(output))

{

Console.WriteLine(output);

Debugger.Launch();

Debugger.Break();

Thread.Sleep(100000000);

}

Thread.Sleep(1);

}

static bool TestFile(string content)

{

var thread1 = new Thread(() =>

{

Write(content);

Thread.Yield();

});

var thread2 = new Thread(() => Environment.Exit(1));

thread1.Start();

thread2.Start();

return true;

}

static void Write(string content)

{

using (StreamWriter writer = new StreamWriter(new FileStream("0", FileMode.Create, FileAccess.Write, FileShare.None)))

{

writer.Write(content);

}

public unsafe int Hijacked(SafeFileHandle handle, byte[] bytes, int offset, int count, NativeOverlapped* overlapped, out int hr)

{

if(stacktrace != null)

{

Console.WriteLine("Current stacktrace");

Console.WriteLine(Environment.StackTrace);

Console.WriteLine("Other stacktrace");

Console.WriteLine(stacktrace);

}

stacktrace = Environment.StackTrace;

Thread.Yield();

hr = 0;

return 3;

}

public static void HijackMethod(MethodBase source, MethodBase target)

{

RuntimeHelpers.PrepareMethod(source.MethodHandle);

RuntimeHelpers.PrepareMethod(target.MethodHandle);

var sourceAddress = (IntPtr)(Marshal.ReadInt32(source.MethodHandle.Value, 8) + (int)source.MethodHandle.Value + 8);

var targetAddress = (long)target.MethodHandle.GetFunctionPointer();

UnlockPage((int)sourceAddress);

UnlockPage((int)targetAddress);

int offset = (int)(targetAddress - (long)sourceAddress - 4 - 1); // four bytes for relative address and one byte for opcode

byte[] instruction = {

0xE9, // Long jump relative instruction

(byte)(offset & 0xFF),

(byte)((offset >> 8) & 0xFF),

(byte)((offset >> 16) & 0xFF),

(byte)((offset >> 24) & 0xFF)

};

Marshal.Copy(instruction, 0, sourceAddress, instruction.Length);

}

public enum Protection

{

PAGE_NOACCESS = 0x01,

PAGE_READONLY = 0x02,

PAGE_READWRITE = 0x04,

PAGE_WRITECOPY = 0x08,

PAGE_EXECUTE = 0x10,

PAGE_EXECUTE_READ = 0x20,

PAGE_EXECUTE_READWRITE = 0x40,

PAGE_EXECUTE_WRITECOPY = 0x80,

PAGE_GUARD = 0x100,

PAGE_NOCACHE = 0x200,

PAGE_WRITECOMBINE = 0x400

}

[DllImport("kernel32.dll", SetLastError = true)]

static extern bool VirtualProtect(IntPtr lpAddress, uint dwSize, uint flNewProtect, out uint lpflOldProtect);

private static void UnlockPage(int address)

{

uint old;

VirtualProtect((IntPtr)address, 4096, (uint)Protection.PAGE_EXECUTE_READWRITE, out old);

}

We want to hijack .NET method and log the stacktraces when we call it multiple times (as we suspect that’s the reason why the file has incorrect content). Based on this great explanation we figure out how to hijack method. So we start that in line 22, in line 99 we get the offset and calculate target address, then do the normal jump instruction to end in line 79.

There, we want to store information that this method was already called. We don’t care about memory consistency here so we don’t use any volatiles, in worst case we’ll just need to wait a little longer. We store the stacktrace of first call in line 87 and then we explicitly yield. However, we need to maintain the logic of the .NET method, so we set the HRESULT in line 90 and return 3 because that’s the length of the content to write.

How do we do this for .NET Core 3.1.101? First, WriteFileNative is different:

public unsafe int Hijacked(SafeFileHandle handle, ReadOnlySpan<byte> buffer, NativeOverlapped* overlapped, out int errorCode)

1	public unsafe int Hijacked(SafeFileHandle handle, ReadOnlySpan<byte> buffer, NativeOverlapped* overlapped, out int errorCode)

Second, we need to disable ReadyToRun before running the child

Environment.SetEnvironmentVariable("COMPlus_ReadyToRun", "0");

1	Environment.SetEnvironmentVariable("COMPlus_ReadyToRun", "0");

Finally, machine code address is 12 bytes from the beginning, not 8.

If we run this for long enough (~20 seconds on my box), we should get the race condition. In that case lines 81-84 print out the stracktraces which are handled in line 45. And that’s the output:

Current stacktrace
   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at SerializationTester.Program.Hijacked(SafeFileHandle handle, Byte[] bytes,Int32 offset, Int32 count, NativeOverlapped* overlapped, Int32& hr)
   at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.FileStream.FlushWrite(Boolean calledFromFinalizer)
   at System.IO.FileStream.Dispose(Boolean disposing)
   at System.IO.FileStream.Finalize()
Other stacktrace
   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at SerializationTester.Program.Hijacked(SafeFileHandle handle, Byte[] bytes,Int32 offset, Int32 count, NativeOverlapped* overlapped, Int32& hr) in C:\Users\afish\Desktop\SerializationTester\Program.cs:line 87
   at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.FileStream.FlushInternalBuffer()
   at System.IO.FileStream.Flush(Boolean flushToDisk)
   at System.IO.FileStream.Flush()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at SerializationTester.Program.Write(String content) in C:\Users\afish\Desktop\SerializationTester\Program.cs:line 74
   at SerializationTester.Program.<>c__DisplayClass2_0.<TestFile>b__0() in C:\Users\afish\Desktop\SerializationTester\Program.cs:line 59
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Current stacktrace

at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)

at System.Environment.get_StackTrace()

at SerializationTester.Program.Hijacked(SafeFileHandle handle, Byte[] bytes,Int32 offset, Int32 count, NativeOverlapped* overlapped, Int32& hr)

at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count)

at System.IO.FileStream.FlushWrite(Boolean calledFromFinalizer)

at System.IO.FileStream.Dispose(Boolean disposing)

at System.IO.FileStream.Finalize()

Other stacktrace

at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)

at System.Environment.get_StackTrace()

at SerializationTester.Program.Hijacked(SafeFileHandle handle, Byte[] bytes,Int32 offset, Int32 count, NativeOverlapped* overlapped, Int32& hr) in C:\Users\afish\Desktop\SerializationTester\Program.cs:line 87

at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count)

at System.IO.FileStream.FlushInternalBuffer()

at System.IO.FileStream.Flush(Boolean flushToDisk)

at System.IO.FileStream.Flush()

at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)

at System.IO.StreamWriter.Dispose(Boolean disposing)

at System.IO.TextWriter.Dispose()

at SerializationTester.Program.Write(String content) in C:\Users\afish\Desktop\SerializationTester\Program.cs:line 74

at SerializationTester.Program.<>c__DisplayClass2_0.<TestFile>b__0() in C:\Users\afish\Desktop\SerializationTester\Program.cs:line 59

at System.Threading.ThreadHelper.ThreadStart_Context(Object state)

at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

at System.Threading.ThreadHelper.ThreadStart()

We can see that first stacktrace comes from finalizer and the second one from normal using construct.

So what’s going on exactly? First thread writes to the file and then cleans it up with Dispose method which causes the buffer to be flushed to the disk. But in the meantime we terminate the application which causes finalizer to run which in turn goes through the same path and writes to the file again.

Is this something we should rely on? No. Finalizers are tricky, as shown by Eric Lippert here and here.

Okay, how do we fix it? We need to change the logic to not call the finalizer:

static void Write(string content)
{
	using(var stream = new FileStream("0", FileMode.Create, FileAccess.Write, FileShare.None))
	using (StreamWriter writer = new StreamWriter(stream))
	{
		GC.SuppressFinalize(stream);
		GC.SuppressFinalize(writer);
		writer.Write(content);
	}
}

static void Write(string content)

{

using(var stream = new FileStream("0", FileMode.Create, FileAccess.Write, FileShare.None))

using (StreamWriter writer = new StreamWriter(stream))

{

GC.SuppressFinalize(stream);

GC.SuppressFinalize(writer);

writer.Write(content);

}

Thanks to this we don’t call the finalizer but we still flush with Dispose.

After this change I couldn’t reproduce the issue for few minutes so I presume it’s fixed.

Takeaway? Finalizers can cause race conditions and exiting the application is not simple.