Coding – Random IT Utensils

Async Wandering Part 15 — How async in C# tricks you and how to order async continuations

afish — Sat, 09 Nov 2024 21:10:33 +0000

This is the fifteenth part of the Async Wandering series. For your convenience you can find other parts in the table of contents in Part 1 – Why creating Form from WinForms in unit tests breaks async?

You probably heard that async is all about not blocking the operating system level thread. This is the fundamental principle of asynchronous programming. If you block the operating system level thread, you lose all the benefits of the asynchronous code.

You also need to keep in mind how to write your C# code. You probably heard that you should keep async all the way up. This is rather easy to keep because the compiler takes care of that. What’s slightly harder to remember is to keep ConfigureAwait(false) all the way down. If you don’t do it this way, the compiler won’t help you and you may run into some nasty deadlocking issues, especially if you use some weird SynchronizationContext.

Last but not least, you probably know that the asynchronous code is only useful if your code is IO-bound. You probably heard that many times. However, what might be very surprising is that C# actually does a lot to make your application work even if your code is CPU-bound and you still use async. This is very misleading and may let you believe that you know async, whereas you only know async in C#. Let’s see why.

There is plenty of no threads!

One of the best articles about async is C# is titled There Is No Thread. Stephen Cleary shows that it’s all about continuations and juggling the lambdas to run your code when some IO-bound operation finishes. I even used this title in my Internals of Async talk in which I explain all the internals of synchronization contexts, continuations, and the machinery behind the scenes.

However, it’s only a figure of speech. At the very end of the day, we need to have some thread to run the code. Depending on your synchronization context, there may be some dedicated thread to run the continuations (like in desktop or Blazor applications), or we can use threads from the thread pool. If you think carefully about the asynchronous code, you should notice that this is the place where C# either bites you hard (and causes many deadlocks) or saves your application even if you are doing something very wrong. How? Because C# uses many threads.

By default, C# uses the thread pool to run continuations. The thread pool runs some not-so-trivial algorithm to spawn new threads when there is plenty of work to be done. This is not part of the asynchronous programming paradigm per se. This is just the implementation detail of C#’s asynchronous code which heavily impacts how your applications scale. Other languages don’t do it in the same way and what works well in C# may fail badly somewhere else. For instance, Python’s asyncio uses just one thread even though Python supports multithreading. While this is just an implementation detail, it have tremendous performance implications. Let’s see why.

One thread can kill you

Let’s take a typical message processing flow. We take a message from the service bus, refresh the lease periodically, and process the message in the background. Let’s say that our flow is IO-bound and we use async to benefit from non-blocking thread instead of spawning multiple threads. Let’s simulate the system. You can find the whole code in this gist.
We start with a message that will store when we received it, when we refreshed the lease for the last time, the identifier of the message, and the final status (if we lost the message or finished successfully):

public class Message
{
	public DateTime LastRefreshTime { get; set; }
	public DateTime ReceiveTime { get; set; }
	public bool WasLost { get; set; }
	public bool WasFinished { get; set; }
	public int Id { get; set; }
}

Now, we want to configure timings in our application. We can specify how long it takes to receive the message from the bus, how many operations we need to perform on each message, and how long they all take:

public class Settings
{
	public int MessagesCount { get; set; }
	public TimeSpan ReceivingDuration { get; set; }
	public TimeSpan ProcessingCPUDuration { get; set; }
	public TimeSpan ProcessingIODuration { get; set; }
	public int ProcessingLoops { get; set; }
	public TimeSpan RefreshDelay { get; set; }
	public TimeSpan RefreshDuration { get; set; }
	public TimeSpan MessageLeaseTimeout { get; set; }
}

Finally, we have some statistics showing how we did:

public class Stats
{
	public int GeneratedMessages { get; set; }
	public int ProcessedSuccessfully { get; set; }
	public int Lost { get; set; }
}

Let’s now see the scaffodling code:

public static async Task Simulate()
{
	Initialize();
	StartMonitoringThread();
	await RunLoop();
}

private static void Initialize()
{
	CurrentStats = new Stats();
	CurrentSettings = new Settings
	{
		ReceivingDuration = TimeSpan.FromMilliseconds(250),
		RefreshDelay = TimeSpan.FromSeconds(2),
		RefreshDuration = TimeSpan.FromMilliseconds(250),
		MessageLeaseTimeout = TimeSpan.FromSeconds(5),
		ProcessingCPUDuration = TimeSpan.FromMilliseconds(100),
		ProcessingIODuration = TimeSpan.FromMilliseconds(1000),
		ProcessingLoops = 20,
		MessagesCount = 100
	};
}

private static void StartMonitoringThread()
{
	Thread monitoringThread = new Thread(() =>
	{
		while (true)
		{
			Thread.Sleep(TimeSpan.FromSeconds(3));
			Log($"Received messages {CurrentStats.GeneratedMessages}, " +
				$"success {CurrentStats.ProcessedSuccessfully}, " +
				$"failed {CurrentStats.Lost}, " +
				$"still running {CurrentStats.GeneratedMessages - CurrentStats.ProcessedSuccessfully - CurrentStats.Lost}");
		}
	});
	monitoringThread.IsBackground = true;
	monitoringThread.Start();
}

We have the Simulate method that runs the magic. It starts by initializing the timings and setting up some monitoring thread to print statistics every three seconds.

When it comes to the timings: we will run 20 loops for each message. In each loop’s iteration, we will do some CPU-bound operation (taking 100 milliseconds), and then some IO-bound operation (taking 1000 milliseconds). We can see that the CPU operation is 10 times shorter than the IO-bound one.

Finally, we have the heart of our system:

private static async Task RunLoop()
{
	while (true)
	{
		var message = await ReceiveMessage();
		if (message == null) continue;

		KeepLease(message);
		ProcessMessage(message);
	}
}

We receive the message, keep refreshing the lease, and process the message. Some error-handling code is omitted for brevity.

Receiving the message is rather straightforward – we check if we have more messages in the queue, then take one, otherwise we simply return:

public static async Task ReceiveMessage()
{
	await Task.Yield();
	await Task.Delay(CurrentSettings.ReceivingDuration);

	if (CurrentSettings.MessagesCount-- > 0)
	{
		CurrentStats.GeneratedMessages++;
		Message message = new Message
		{
			LastRefreshTime = DateTime.Now,
			WasLost = false,
			Id = CurrentStats.GeneratedMessages,
			ReceiveTime = DateTime.Now
		};
		Log($"New message received with id {message.Id}");
		return message;
	}
	else
	{
		return null;
	}
}

Keeping a lease is also clear – we wait for some time, then refresh the lease and check if we made it on time:

public static async Task KeepLease(Message message)
{
	await Task.Yield();

	while (message.WasFinished == false) // This is unsafe according to memory model
	{
		await Task.Delay(CurrentSettings.RefreshDelay);

		if (DateTime.Now > message.LastRefreshTime + CurrentSettings.MessageLeaseTimeout)
		{
			message.WasLost = true;
			CurrentStats.Lost++;
			Log($"Lost lease for message {message.Id}");
			return;
		}
		else
		{
			await Task.Delay(CurrentSettings.RefreshDuration);
			Log($"Refreshed lease for message {message.Id}");
			message.LastRefreshTime = DateTime.Now;
		}
	}

	CurrentStats.ProcessedSuccessfully++;
}

Finally, the heart of our message processing. We simply run a loop and do the work:

public static async Task ProcessMessage(Message message)
{
	await Task.Yield();

	for (int part = 0; part < CurrentSettings.ProcessingLoops && message.WasLost == false; ++part)
	{
		Thread.Sleep(CurrentSettings.ProcessingCPUDuration); // CPU-bound part

		await Task.Delay(CurrentSettings.ProcessingIODuration); // IO-bound part
	}

	message.WasFinished = true;
	if (!message.WasLost)
	{
		Log($"Finished message with id {message.Id} in {DateTime.Now - message.ReceiveTime}");
	}
}

Notice that we block the thread for the CPU-bound operation and use await for the IO-bound one.

We also have this logging method that prints the timestamp, thread ID, and the message:

public static void Log(string message)
{
	Console.WriteLine($"{DateTime.Now}\t{Thread.CurrentThread.ManagedThreadId}\t{message}");
}

Let’s run the code, let it go for a while, and then see what happens:

11/9/2024 12:05:06 PM   4       New message received with id 1
11/9/2024 12:05:06 PM   6       New message received with id 2
11/9/2024 12:05:06 PM   6       New message received with id 3
11/9/2024 12:05:07 PM   8       New message received with id 4
11/9/2024 12:05:07 PM   8       New message received with id 5
11/9/2024 12:05:07 PM   4       New message received with id 6
11/9/2024 12:05:08 PM   4       New message received with id 7
11/9/2024 12:05:08 PM   4       New message received with id 8
11/9/2024 12:05:08 PM   11      New message received with id 9
11/9/2024 12:05:08 PM   8       Refreshed lease for message 1
11/9/2024 12:05:08 PM   12      New message received with id 10
11/9/2024 12:05:08 PM   11      Refreshed lease for message 2
11/9/2024 12:05:09 PM   12      New message received with id 11
11/9/2024 12:05:09 PM   7       Received messages 11, success 0, failed 0, still running 11
11/9/2024 12:05:09 PM   11      Refreshed lease for message 3
11/9/2024 12:05:09 PM   11      New message received with id 12
11/9/2024 12:05:09 PM   12      Refreshed lease for message 4
11/9/2024 12:05:09 PM   6       New message received with id 13
11/9/2024 12:05:09 PM   8       Refreshed lease for message 5
11/9/2024 12:05:09 PM   12      New message received with id 14
11/9/2024 12:05:10 PM   12      Refreshed lease for message 6
11/9/2024 12:05:10 PM   12      New message received with id 15
11/9/2024 12:05:10 PM   11      Refreshed lease for message 7
11/9/2024 12:05:10 PM   11      New message received with id 16
11/9/2024 12:05:10 PM   6       Refreshed lease for message 8
11/9/2024 12:05:10 PM   12      New message received with id 17
11/9/2024 12:05:10 PM   11      Refreshed lease for message 9
11/9/2024 12:05:10 PM   8       New message received with id 18
11/9/2024 12:05:10 PM   11      Refreshed lease for message 1
11/9/2024 12:05:11 PM   4       Refreshed lease for message 10
11/9/2024 12:05:11 PM   13      New message received with id 19
11/9/2024 12:05:11 PM   12      Refreshed lease for message 2
11/9/2024 12:05:11 PM   13      Refreshed lease for message 11
11/9/2024 12:05:11 PM   11      New message received with id 20
11/9/2024 12:05:11 PM   6       Refreshed lease for message 3
11/9/2024 12:05:11 PM   4       Refreshed lease for message 12
11/9/2024 12:05:11 PM   13      New message received with id 21
11/9/2024 12:05:11 PM   13      Refreshed lease for message 4
11/9/2024 12:05:11 PM   4       Refreshed lease for message 13
11/9/2024 12:05:11 PM   4       New message received with id 22
11/9/2024 12:05:12 PM   4       Refreshed lease for message 5
11/9/2024 12:05:12 PM   6       Refreshed lease for message 14
11/9/2024 12:05:12 PM   13      New message received with id 23
11/9/2024 12:05:12 PM   7       Received messages 23, success 0, failed 0, still running 23
11/9/2024 12:05:12 PM   4       Refreshed lease for message 6
11/9/2024 12:05:12 PM   4       Refreshed lease for message 15
...
11/9/2024 12:05:50 PM   15      Finished message with id 84 in 00:00:22.3550821
11/9/2024 12:05:50 PM   15      Refreshed lease for message 83
11/9/2024 12:05:50 PM   4       Refreshed lease for message 100
11/9/2024 12:05:50 PM   8       Finished message with id 85 in 00:00:22.3554494
11/9/2024 12:05:50 PM   15      Refreshed lease for message 84
11/9/2024 12:05:50 PM   20      Refreshed lease for message 92
11/9/2024 12:05:50 PM   20      Finished message with id 86 in 00:00:22.3882717
11/9/2024 12:05:50 PM   8       Refreshed lease for message 93
11/9/2024 12:05:51 PM   4       Refreshed lease for message 85
11/9/2024 12:05:51 PM   20      Finished message with id 87 in 00:00:22.3452990
11/9/2024 12:05:51 PM   4       Refreshed lease for message 94
11/9/2024 12:05:51 PM   20      Refreshed lease for message 86
11/9/2024 12:05:51 PM   7       Received messages 100, success 86, failed 0, still running 14
11/9/2024 12:05:51 PM   14      Finished message with id 88 in 00:00:22.3968974
11/9/2024 12:05:51 PM   13      Refreshed lease for message 87
11/9/2024 12:05:51 PM   4       Refreshed lease for message 95
11/9/2024 12:05:51 PM   4       Refreshed lease for message 88
11/9/2024 12:05:51 PM   14      Finished message with id 89 in 00:00:22.3782384
11/9/2024 12:05:51 PM   4       Refreshed lease for message 96
11/9/2024 12:05:51 PM   15      Finished message with id 90 in 00:00:22.3557212
11/9/2024 12:05:51 PM   15      Refreshed lease for message 89
11/9/2024 12:05:52 PM   13      Refreshed lease for message 97
11/9/2024 12:05:52 PM   20      Refreshed lease for message 90
11/9/2024 12:05:52 PM   15      Finished message with id 91 in 00:00:22.4805351
11/9/2024 12:05:52 PM   15      Refreshed lease for message 98
11/9/2024 12:05:52 PM   20      Refreshed lease for message 91
11/9/2024 12:05:52 PM   15      Refreshed lease for message 99
11/9/2024 12:05:52 PM   15      Finished message with id 92 in 00:00:22.3979587
11/9/2024 12:05:52 PM   4       Refreshed lease for message 100
11/9/2024 12:05:52 PM   20      Finished message with id 93 in 00:00:22.3374987
11/9/2024 12:05:53 PM   4       Refreshed lease for message 92
11/9/2024 12:05:53 PM   4       Finished message with id 94 in 00:00:22.3451488
11/9/2024 12:05:53 PM   4       Refreshed lease for message 93
11/9/2024 12:05:53 PM   13      Refreshed lease for message 94
11/9/2024 12:05:53 PM   13      Finished message with id 95 in 00:00:22.3784563
11/9/2024 12:05:53 PM   13      Finished message with id 96 in 00:00:22.3800325
11/9/2024 12:05:53 PM   13      Refreshed lease for message 95
11/9/2024 12:05:54 PM   4       Finished message with id 97 in 00:00:22.3312738
11/9/2024 12:05:54 PM   20      Refreshed lease for message 96
11/9/2024 12:05:54 PM   7       Received messages 100, success 96, failed 0, still running 4
11/9/2024 12:05:54 PM   13      Finished message with id 98 in 00:00:22.3502617
11/9/2024 12:05:54 PM   4       Refreshed lease for message 97
11/9/2024 12:05:54 PM   13      Finished message with id 99 in 00:00:22.3527442
11/9/2024 12:05:54 PM   4       Refreshed lease for message 98
11/9/2024 12:05:54 PM   4       Finished message with id 100 in 00:00:22.3675039
11/9/2024 12:05:54 PM   13      Refreshed lease for message 99
11/9/2024 12:05:55 PM   13      Refreshed lease for message 100
11/9/2024 12:05:57 PM   7       Received messages 100, success 100, failed 0, still running 0

We can see that all messaged were processed successfully in around 50 seconds. Processing a message was taking around 22 seconds which makes perfect sense since we had 20 iterations taking around 1100 milliseconds each. No failures, all was good.

Let’s now increase the CPU-bound operation time to 1 second (to match the IO-bound part). This is what happens:

11/9/2024 12:06:57 PM   8       New message received with id 1
11/9/2024 12:06:57 PM   11      New message received with id 2
11/9/2024 12:06:58 PM   11      New message received with id 3
11/9/2024 12:06:58 PM   4       New message received with id 4
11/9/2024 12:06:58 PM   13      New message received with id 5
11/9/2024 12:06:58 PM   13      New message received with id 6
11/9/2024 12:06:59 PM   14      New message received with id 7
11/9/2024 12:06:59 PM   11      New message received with id 8
11/9/2024 12:06:59 PM   4       New message received with id 9
11/9/2024 12:06:59 PM   15      Refreshed lease for message 1
11/9/2024 12:06:59 PM   4       New message received with id 10
11/9/2024 12:07:00 PM   12      Refreshed lease for message 2
11/9/2024 12:07:00 PM   12      New message received with id 11
11/9/2024 12:07:00 PM   7       Received messages 11, success 0, failed 0, still running 11
11/9/2024 12:07:00 PM   6       Refreshed lease for message 3
11/9/2024 12:07:00 PM   16      New message received with id 12
11/9/2024 12:07:00 PM   14      Refreshed lease for message 4
11/9/2024 12:07:00 PM   14      New message received with id 13
11/9/2024 12:07:00 PM   15      Refreshed lease for message 5
11/9/2024 12:07:00 PM   4       New message received with id 14
11/9/2024 12:07:01 PM   13      Refreshed lease for message 6
11/9/2024 12:07:01 PM   12      New message received with id 15
11/9/2024 12:07:01 PM   6       Refreshed lease for message 7
11/9/2024 12:07:01 PM   16      New message received with id 16
11/9/2024 12:07:01 PM   14      Refreshed lease for message 8
11/9/2024 12:07:01 PM   4       Refreshed lease for message 9
11/9/2024 12:07:02 PM   13      Refreshed lease for message 1
11/9/2024 12:07:02 PM   12      Refreshed lease for message 10
11/9/2024 12:07:02 PM   12      New message received with id 17
11/9/2024 12:07:02 PM   6       Refreshed lease for message 2
11/9/2024 12:07:02 PM   16      Refreshed lease for message 11
11/9/2024 12:07:02 PM   14      Refreshed lease for message 3
11/9/2024 12:07:02 PM   14      Refreshed lease for message 12
11/9/2024 12:07:02 PM   4       Refreshed lease for message 4
11/9/2024 12:07:03 PM   13      Refreshed lease for message 13
11/9/2024 12:07:03 PM   13      Refreshed lease for message 5
11/9/2024 12:07:03 PM   12      Refreshed lease for message 14
11/9/2024 12:07:03 PM   12      New message received with id 18
11/9/2024 12:07:03 PM   7       Received messages 18, success 0, failed 0, still running 18
11/9/2024 12:07:03 PM   16      Refreshed lease for message 6
11/9/2024 12:07:03 PM   16      Refreshed lease for message 15
...
11/9/2024 12:08:38 PM   21      Finished message with id 88 in 00:00:42.4181405
11/9/2024 12:08:38 PM   42      Finished message with id 90 in 00:00:41.9138898
11/9/2024 12:08:38 PM   8       Refreshed lease for message 91
11/9/2024 12:08:38 PM   28      Refreshed lease for message 85
11/9/2024 12:08:38 PM   28      Refreshed lease for message 88
11/9/2024 12:08:39 PM   24      Finished message with id 85 in 00:00:43.9296927
11/9/2024 12:08:39 PM   16      Finished message with id 93 in 00:00:40.6588087
11/9/2024 12:08:39 PM   7       Received messages 100, success 88, failed 0, still running 12
11/9/2024 12:08:39 PM   21      Refreshed lease for message 93
11/9/2024 12:08:39 PM   23      Refreshed lease for message 94
11/9/2024 12:08:39 PM   16      Refreshed lease for message 92
11/9/2024 12:08:40 PM   23      Refreshed lease for message 97
11/9/2024 12:08:40 PM   16      Refreshed lease for message 99
11/9/2024 12:08:40 PM   21      Refreshed lease for message 96
11/9/2024 12:08:40 PM   42      Refreshed lease for message 98
11/9/2024 12:08:40 PM   16      Refreshed lease for message 90
11/9/2024 12:08:40 PM   42      Refreshed lease for message 95
11/9/2024 12:08:40 PM   42      Refreshed lease for message 100
11/9/2024 12:08:40 PM   24      Finished message with id 94 in 00:00:41.3801588
11/9/2024 12:08:40 PM   8       Finished message with id 92 in 00:00:41.9121719
11/9/2024 12:08:40 PM   8       Finished message with id 95 in 00:00:41.1802881
11/9/2024 12:08:40 PM   24      Finished message with id 96 in 00:00:40.9180345
11/9/2024 12:08:40 PM   24      Refreshed lease for message 91
11/9/2024 12:08:40 PM   52      Refreshed lease for message 85
11/9/2024 12:08:41 PM   24      Finished message with id 98 in 00:00:41.3325727
11/9/2024 12:08:41 PM   21      Finished message with id 91 in 00:00:43.1595798
11/9/2024 12:08:42 PM   24      Refreshed lease for message 94
11/9/2024 12:08:42 PM   42      Refreshed lease for message 92
11/9/2024 12:08:42 PM   24      Refreshed lease for message 99
11/9/2024 12:08:42 PM   8       Refreshed lease for message 97
11/9/2024 12:08:42 PM   42      Refreshed lease for message 96
11/9/2024 12:08:42 PM   24      Finished message with id 97 in 00:00:42.6127819
11/9/2024 12:08:42 PM   42      Refreshed lease for message 98
11/9/2024 12:08:42 PM   42      Finished message with id 99 in 00:00:40.4341505
11/9/2024 12:08:42 PM   7       Received messages 100, success 95, failed 0, still running 5
11/9/2024 12:08:42 PM   21      Refreshed lease for message 95
11/9/2024 12:08:42 PM   42      Refreshed lease for message 100
11/9/2024 12:08:43 PM   42      Refreshed lease for message 91
11/9/2024 12:08:43 PM   8       Finished message with id 100 in 00:00:41.1363357
11/9/2024 12:08:44 PM   21      Refreshed lease for message 97
11/9/2024 12:08:44 PM   52      Refreshed lease for message 99
11/9/2024 12:08:44 PM   52      Refreshed lease for message 100
11/9/2024 12:08:45 PM   7       Received messages 100, success 100, failed 0, still running 0

This time it took nearly 2 minutes to process all the messages. Each message is now taking around 40 seconds. Still, all worked.

Let’s now talk about threads. You can see that the examples use multiple threads to handle the messages. In the second execution, there were around 60 active messages at one time, so this created many threads (we can see that at least 50 threads were created based on the log above). Our application scales well and we can’t complain. Seems like async is doing a really good job!

However, what would happen if we moved this code to some other asynchronous platform? For instance, to Python’s asyncio that uses only single thread? We can emulate that in C# by running the code above in a WinForms context that forces continuations to go through one thread. Let’s change the CPU-bound operation duration to 100 milliseconds (to the original value) and let’s run this from the WinForms app now:

11/9/2024 12:19:20 PM   1       New message received with id 1
11/9/2024 12:19:20 PM   1       New message received with id 2
11/9/2024 12:19:20 PM   1       New message received with id 3
11/9/2024 12:19:21 PM   1       New message received with id 4
11/9/2024 12:19:21 PM   1       New message received with id 5
11/9/2024 12:19:22 PM   1       New message received with id 6
11/9/2024 12:19:22 PM   1       New message received with id 7
11/9/2024 12:19:22 PM   1       Refreshed lease for message 1
11/9/2024 12:19:22 PM   1       New message received with id 8
11/9/2024 12:19:22 PM   11      Received messages 8, success 0, failed 0, still running 8
11/9/2024 12:19:23 PM   1       Refreshed lease for message 2
11/9/2024 12:19:23 PM   1       New message received with id 9
11/9/2024 12:19:23 PM   1       Refreshed lease for message 3
11/9/2024 12:19:23 PM   1       New message received with id 10
11/9/2024 12:19:23 PM   1       Refreshed lease for message 4
11/9/2024 12:19:24 PM   1       Refreshed lease for message 5
11/9/2024 12:19:24 PM   1       New message received with id 11
11/9/2024 12:19:24 PM   1       Refreshed lease for message 6
11/9/2024 12:19:24 PM   1       New message received with id 12
11/9/2024 12:19:25 PM   1       Refreshed lease for message 7
11/9/2024 12:19:25 PM   1       Refreshed lease for message 1
11/9/2024 12:19:25 PM   1       Refreshed lease for message 8
11/9/2024 12:19:25 PM   1       New message received with id 13
11/9/2024 12:19:25 PM   11      Received messages 13, success 0, failed 0, still running 13
11/9/2024 12:19:26 PM   1       Refreshed lease for message 2
11/9/2024 12:19:26 PM   1       Refreshed lease for message 9
...
11/9/2024 12:21:15 PM   1       Refreshed lease for message 58
11/9/2024 12:21:15 PM   1       Refreshed lease for message 62
11/9/2024 12:21:15 PM   1       Finished message with id 46 in 00:00:42.5373955
11/9/2024 12:21:16 PM   1       Refreshed lease for message 49
11/9/2024 12:21:16 PM   1       Refreshed lease for message 51
11/9/2024 12:21:16 PM   1       Refreshed lease for message 53
11/9/2024 12:21:17 PM   1       New message received with id 65
11/9/2024 12:21:17 PM   1       Refreshed lease for message 55
11/9/2024 12:21:17 PM   1       Refreshed lease for message 46
11/9/2024 12:21:17 PM   1       Refreshed lease for message 57
11/9/2024 12:21:17 PM   1       Refreshed lease for message 59
11/9/2024 12:21:17 PM   11      Received messages 65, success 43, failed 3, still running 19
11/9/2024 12:21:17 PM   1       Refreshed lease for message 61
11/9/2024 12:21:17 PM   1       Refreshed lease for message 48
...
11/9/2024 12:22:53 PM   11      Received messages 100, success 90, failed 3, still running 7
11/9/2024 12:22:53 PM   1       Refreshed lease for message 97
11/9/2024 12:22:53 PM   1       Refreshed lease for message 99
11/9/2024 12:22:54 PM   1       Refreshed lease for message 94
11/9/2024 12:22:54 PM   1       Refreshed lease for message 96
11/9/2024 12:22:54 PM   1       Finished message with id 95 in 00:00:32.1189187
11/9/2024 12:22:54 PM   1       Refreshed lease for message 98
11/9/2024 12:22:54 PM   1       Refreshed lease for message 100
11/9/2024 12:22:55 PM   1       Refreshed lease for message 95
11/9/2024 12:22:56 PM   1       Finished message with id 96 in 00:00:31.0536654
11/9/2024 12:22:56 PM   1       Refreshed lease for message 99
11/9/2024 12:22:56 PM   1       Refreshed lease for message 97
11/9/2024 12:22:56 PM   11      Received messages 100, success 92, failed 3, still running 5
11/9/2024 12:22:56 PM   1       Refreshed lease for message 96
11/9/2024 12:22:57 PM   1       Refreshed lease for message 98
11/9/2024 12:22:57 PM   1       Refreshed lease for message 100
11/9/2024 12:22:57 PM   1       Finished message with id 97 in 00:00:30.1211740
11/9/2024 12:22:58 PM   1       Refreshed lease for message 99
11/9/2024 12:22:58 PM   1       Refreshed lease for message 97
11/9/2024 12:22:58 PM   1       Finished message with id 98 in 00:00:29.1238261
11/9/2024 12:22:59 PM   1       Refreshed lease for message 98
11/9/2024 12:22:59 PM   1       Refreshed lease for message 100
11/9/2024 12:22:59 PM   11      Received messages 100, success 95, failed 3, still running 2
11/9/2024 12:22:59 PM   1       Finished message with id 99 in 00:00:28.1643467
11/9/2024 12:23:00 PM   1       Refreshed lease for message 99
11/9/2024 12:23:00 PM   1       Finished message with id 100 in 00:00:27.2119750
11/9/2024 12:23:01 PM   1       Refreshed lease for message 100
11/9/2024 12:23:02 PM   11      Received messages 100, success 97, failed 3, still running 0

It wasn’t that bad and we can see that we indeed ran on a single thread. First, notice that now it took nearly 4 minutes to complete. That’s understandable as we now run things on a single thread. Also, notice that each message was taking around 30-40 seconds to complete. That is much longer than before. This is because messages compete for the CPU time and we don’t have any parallelism. It’s also worth noting that we lost 3 messages. That’s not that bad. The system overscaled just a bit and couldn’t deal with the load but the stabilized and finished processing.

Let’s now increase the CPU-bound duration to 1 second and try again:

11/9/2024 12:24:26 PM   1       New message received with id 1
11/9/2024 12:24:27 PM   1       New message received with id 2
11/9/2024 12:24:29 PM   12      Received messages 2, success 0, failed 0, still running 2
11/9/2024 12:24:29 PM   1       New message received with id 3
11/9/2024 12:24:29 PM   1       Refreshed lease for message 1
11/9/2024 12:24:32 PM   12      Received messages 3, success 0, failed 0, still running 3
11/9/2024 12:24:32 PM   1       Refreshed lease for message 2
11/9/2024 12:24:33 PM   1       New message received with id 4
11/9/2024 12:24:34 PM   1       Lost lease for message 3
11/9/2024 12:24:34 PM   1       Refreshed lease for message 1
11/9/2024 12:24:35 PM   12      Received messages 4, success 0, failed 1, still running 3
11/9/2024 12:24:38 PM   12      Received messages 4, success 0, failed 1, still running 3
11/9/2024 12:24:39 PM   1       Refreshed lease for message 2
11/9/2024 12:24:39 PM   1       New message received with id 5
11/9/2024 12:24:39 PM   1       Lost lease for message 4
11/9/2024 12:24:39 PM   1       Refreshed lease for message 1
11/9/2024 12:24:41 PM   12      Received messages 5, success 0, failed 2, still running 3
11/9/2024 12:24:43 PM   1       New message received with id 6
11/9/2024 12:24:44 PM   12      Received messages 6, success 0, failed 2, still running 4
11/9/2024 12:24:46 PM   1       Refreshed lease for message 5
...
11/9/2024 12:31:48 PM   1       Lost lease for message 98
11/9/2024 12:31:51 PM   12      Received messages 99, success 7, failed 88, still running 4
11/9/2024 12:31:52 PM   1       Refreshed lease for message 97
11/9/2024 12:31:52 PM   1       Refreshed lease for message 92
11/9/2024 12:31:52 PM   1       Refreshed lease for message 86
11/9/2024 12:31:54 PM   12      Received messages 99, success 7, failed 88, still running 4
11/9/2024 12:31:54 PM   1       New message received with id 100
11/9/2024 12:31:54 PM   1       Lost lease for message 99
11/9/2024 12:31:54 PM   1       Finished message with id 86 in 00:01:04.5795457
11/9/2024 12:31:57 PM   12      Received messages 100, success 7, failed 89, still running 4
11/9/2024 12:31:57 PM   1       Refreshed lease for message 86
11/9/2024 12:31:57 PM   1       Refreshed lease for message 97
11/9/2024 12:31:58 PM   1       Refreshed lease for message 92
11/9/2024 12:31:59 PM   1       Lost lease for message 100
11/9/2024 12:32:00 PM   12      Received messages 100, success 8, failed 90, still running 2
11/9/2024 12:32:01 PM   1       Refreshed lease for message 97
11/9/2024 12:32:02 PM   1       Refreshed lease for message 92
11/9/2024 12:32:03 PM   12      Received messages 100, success 8, failed 90, still running 2
11/9/2024 12:32:04 PM   1       Refreshed lease for message 97
11/9/2024 12:32:05 PM   1       Refreshed lease for message 92
11/9/2024 12:32:06 PM   12      Received messages 100, success 8, failed 90, still running 2
11/9/2024 12:32:08 PM   1       Refreshed lease for message 97
11/9/2024 12:32:09 PM   12      Received messages 100, success 8, failed 90, still running 2
11/9/2024 12:32:09 PM   1       Refreshed lease for message 92
11/9/2024 12:32:11 PM   1       Refreshed lease for message 97
11/9/2024 12:32:12 PM   12      Received messages 100, success 8, failed 90, still running 2
11/9/2024 12:32:12 PM   1       Finished message with id 92 in 00:00:55.4921671
11/9/2024 12:32:13 PM   1       Refreshed lease for message 92
11/9/2024 12:32:14 PM   1       Refreshed lease for message 97
11/9/2024 12:32:15 PM   12      Received messages 100, success 9, failed 90, still running 1
11/9/2024 12:32:17 PM   1       Refreshed lease for message 97
11/9/2024 12:32:18 PM   12      Received messages 100, success 9, failed 90, still running 1
11/9/2024 12:32:19 PM   1       Refreshed lease for message 97
11/9/2024 12:32:21 PM   12      Received messages 100, success 9, failed 90, still running 1
11/9/2024 12:32:21 PM   1       Refreshed lease for message 97
11/9/2024 12:32:24 PM   1       Refreshed lease for message 97
11/9/2024 12:32:24 PM   12      Received messages 100, success 9, failed 90, still running 1
11/9/2024 12:32:27 PM   1       Refreshed lease for message 97
11/9/2024 12:32:27 PM   12      Received messages 100, success 9, failed 90, still running 1
11/9/2024 12:32:28 PM   1       Finished message with id 97 in 00:00:50.4062632
11/9/2024 12:32:29 PM   1       Refreshed lease for message 97
11/9/2024 12:32:30 PM   12      Received messages 100, success 10, failed 90, still running 0

And here things start to collapse. It took us 8 minutes to process all messages, each of them was taking around 1 minute, and we failed to process 90 out of one hundred. We lost 90% of all of the messages. Our system became unreliable just because we increased the CPU-bound part of the message processing. But why did it break the application exactly? What happened?

You don’t control the priority of continuations

Our application runs three distinct operations in total:

Take the message from the queue
Refresh the lease of the message
Do some processing of the message

Every single time we await the task, we release the thread and let it do something else. Once the IO-bound operation finishes, we schedule it to run on the same thread. However, the order of continuations doesn’t reflect the importance of what we should do.

In order to keep the system stable, we need to refresh the leases. Therefore, if there is any continuation that wants to refresh the lease (the continuation in KeepLease method), it should run before everything else.

Once we don’t have any continuations for refreshing the leases, we should run continuations for message processing. Obviously, if some KeepLease continuation gets scheduled, it should preempt other continuations.

Finally, when we have no continuations for refreshing the leases or processing the messages, we should run the continuation for getting new message from the queue. In other words, we receive a new message only when we have some idle CPU time that we can use to process something more.

Unfortunately, the async in C# doesn’t let you easily prioritize the continuations. However, this is not a problem most of the times because C# uses multiple threads! Once a continuation is free to run, the thread pool will grow to run the continuation earlier if possible. This is not part of the async programming paradigm and you can’t take it for granted. However, when we run things on a single thread, then continuations have no priorities and message processing continuations may stop lease refreshing continuations from running. Even worse, we may run continuation that receives new message from the bus even though we are already overloaded.

Depending on the nature of your platform (be it C# with different synchronization context, Python with single-threaded asyncio, or JavaScript with one and only one thread), you may get different results. Your application may scale well or may fail badly.

Let’s fix it

We can fix this issue in many ways. Conceptually, we need three different queues: the first one represents the lease refreshments, the second is for message processing, and the third is for getting new message from the bus. We would then have one processor that would check each of the queues in order and execute the operations accordingly. Unfortunately, rewriting the application from async paradigm to a consumer with multiple queues is not straightforward.

Instead, we can reorder the continuations. The trick is to introduce a priority for each continuation. We do the following:

We store a desired priority of continuations running on a thread
When a continuation wants to run, it checks if the desired priority is equal to the priority of the continuation
If it’s equal, then the continuation resets the desired priority to some invalid value and continues
Otherwise, the continuation bumps the priority if possible and lets other continuations run

The protocol is based on the following idea: some continuation sets the desired priority to be at least the priority of the continuation and then lets other continuations to run. If there are other continuations of lower priority, they will simply release the CPU and reschedule themselves. If there are continuations of some higher priority, they will bump the desired priority. And if there are no continuations, then th original continuation will finally get the CPU, run the code, and reset the priority, so other continuations can run the same dance over and over. Here is the code:

public static async Task Prioritize(int priority)
{
	var cookie = Guid.NewGuid();

	while (true)
	{
		if (CurrentSettings.DesiredPriority == priority && CurrentSettings.Cookie == cookie)
		{
			CurrentSettings.DesiredPriority = -1;
			CurrentSettings.Cookie = Guid.Empty;
			return;
		}
		else
		{
			if (CurrentSettings.DesiredPriority < priority)
			{
				CurrentSettings.DesiredPriority = priority;
				CurrentSettings.Cookie = cookie;
				await Task.Yield();
				continue;
			}
			else
			{
				await Task.Yield();
				continue;
			}
		}
	}
}

We need to run this method every single time when we run await. For instance, KeepLease becomes this:

public static async Task KeepLease(Message message)
{
	await Task.Yield();
	await Prioritize(3);

	while (message.WasFinished == false) // This is unsafe according to memory model
	{
		await Task.Delay(CurrentSettings.RefreshDelay);
		await Prioritize(3);

		if (DateTime.Now > message.LastRefreshTime + CurrentSettings.MessageLeaseTimeout)
		{
			message.WasLost = true;
			CurrentStats.Lost++;
			Log($"Lost lease for message {message.Id}");
			return;
		}
		else
		{
			await Task.Delay(CurrentSettings.RefreshDuration);
			await Prioritize(3);
			Log($"Refreshed lease for message {message.Id}");
			message.LastRefreshTime = DateTime.Now;
		}
	}

	CurrentStats.ProcessedSuccessfully++;
}

You can find the full snippet here. Let’s see it in action:

11/9/2024 12:35:42 PM   1       New message received with id 1
11/9/2024 12:35:43 PM   1       New message received with id 2
11/9/2024 12:35:45 PM   12      Received messages 2, success 0, failed 0, still running 2
11/9/2024 12:35:45 PM   1       Refreshed lease for message 1
11/9/2024 12:35:46 PM   1       Refreshed lease for message 2
11/9/2024 12:35:48 PM   12      Received messages 2, success 0, failed 0, still running 2
11/9/2024 12:35:48 PM   1       Refreshed lease for message 1
11/9/2024 12:35:49 PM   1       Refreshed lease for message 2
11/9/2024 12:35:51 PM   12      Received messages 2, success 0, failed 0, still running 2
11/9/2024 12:35:51 PM   1       Refreshed lease for message 1
11/9/2024 12:35:52 PM   1       Refreshed lease for message 2
11/9/2024 12:35:53 PM   1       New message received with id 3
11/9/2024 12:35:54 PM   12      Received messages 3, success 0, failed 0, still running 3
11/9/2024 12:35:55 PM   1       Refreshed lease for message 1
11/9/2024 12:35:56 PM   1       Refreshed lease for message 3
11/9/2024 12:35:57 PM   12      Received messages 3, success 0, failed 0, still running 3
11/9/2024 12:35:58 PM   1       Refreshed lease for message 1
11/9/2024 12:35:58 PM   1       Refreshed lease for message 2
11/9/2024 12:35:59 PM   1       Refreshed lease for message 3
11/9/2024 12:36:00 PM   12      Received messages 3, success 0, failed 0, still running 3
11/9/2024 12:36:01 PM   1       Refreshed lease for message 1
11/9/2024 12:36:02 PM   1       Refreshed lease for message 3
11/9/2024 12:36:03 PM   12      Received messages 3, success 0, failed 0, still running 3
...
11/9/2024 1:09:00 PM    1       Refreshed lease for message 100
11/9/2024 1:09:00 PM    1       Refreshed lease for message 98
11/9/2024 1:09:02 PM    12      Received messages 100, success 97, failed 0, still running 3
11/9/2024 1:09:03 PM    1       Refreshed lease for message 99
11/9/2024 1:09:03 PM    1       Refreshed lease for message 100
11/9/2024 1:09:04 PM    1       Refreshed lease for message 98
11/9/2024 1:09:05 PM    1       Finished message with id 98 in 00:00:46.5274062
11/9/2024 1:09:05 PM    12      Received messages 100, success 97, failed 0, still running 3
11/9/2024 1:09:06 PM    1       Refreshed lease for message 100
11/9/2024 1:09:06 PM    1       Refreshed lease for message 99
11/9/2024 1:09:07 PM    1       Refreshed lease for message 98
11/9/2024 1:09:08 PM    12      Received messages 100, success 98, failed 0, still running 2
11/9/2024 1:09:09 PM    1       Refreshed lease for message 100
11/9/2024 1:09:09 PM    1       Refreshed lease for message 99
11/9/2024 1:09:11 PM    12      Received messages 100, success 98, failed 0, still running 2
11/9/2024 1:09:12 PM    1       Refreshed lease for message 99
11/9/2024 1:09:12 PM    1       Refreshed lease for message 100
11/9/2024 1:09:14 PM    12      Received messages 100, success 98, failed 0, still running 2
11/9/2024 1:09:15 PM    1       Refreshed lease for message 99
11/9/2024 1:09:15 PM    1       Refreshed lease for message 100
11/9/2024 1:09:17 PM    1       Finished message with id 99 in 00:00:51.5793525
11/9/2024 1:09:17 PM    1       Refreshed lease for message 99
11/9/2024 1:09:17 PM    1       Refreshed lease for message 100
11/9/2024 1:09:17 PM    12      Received messages 100, success 99, failed 0, still running 1
11/9/2024 1:09:19 PM    1       Refreshed lease for message 100
11/9/2024 1:09:20 PM    12      Received messages 100, success 99, failed 0, still running 1
11/9/2024 1:09:22 PM    1       Refreshed lease for message 100
11/9/2024 1:09:23 PM    12      Received messages 100, success 99, failed 0, still running 1
11/9/2024 1:09:25 PM    1       Refreshed lease for message 100
11/9/2024 1:09:26 PM    12      Received messages 100, success 99, failed 0, still running 1
11/9/2024 1:09:27 PM    1       Refreshed lease for message 100
11/9/2024 1:09:29 PM    12      Received messages 100, success 99, failed 0, still running 1
11/9/2024 1:09:29 PM    1       Refreshed lease for message 100
11/9/2024 1:09:30 PM    1       Finished message with id 100 in 00:00:54.5227973
11/9/2024 1:09:32 PM    1       Refreshed lease for message 100
11/9/2024 1:09:32 PM    12      Received messages 100, success 100, failed 0, still running 0

We can see that the code runs much slower and it takes 35 minutes to complete. However, all messages are processed successfully and the code scales automatically. We don’t need to manually control the thread pool size, but the application simply processes fewer or more messages depending on the actual CPU-bound processing time.

Summary

async programming is very hard. We were told many times that it’s as simple as putting async here and await there. C# did a lot to get rid of deadlocks as nearly every platform now uses no synchronization context (as compared with old ASP.NET which had its own context or all the desktop apps that were running with a single thread). Also, C# uses a thread pool and can fix many programmer’s mistakes that can limit the scalability.

However, asynchronous programming can be implemented in many other ways. You can’t assume that it will use many threads or that the coroutine will be triggered immediately. Many quirks can decrease the performance. Python’s asyncio is a great example of how asynchronous programming can work much differently, especially if you take the Python’s performance into consideration. What’s IO-bound in C#, can easily become CPU-bound in Python because Python is way slower.

RPA Part 1 — Sharing Word/Excel between objects in Blue Prism

afish — Mon, 09 Sep 2024 16:22:45 +0000

It’s quite common for Blue Prism developers to use MS Word VBO or similar to interact with Word instances in workflows. However, a problem arises when we want to extend the object with some custom functionality that utilizes Word COM object. The issue is that MS Word VBO doesn’t expose the Word instance and we can only use the handle. Due to that, Blue Prism developers often extend this object and add actions directly in it. This breaks many principles and is very nasty. Let’s see why and how to fix that.

Why extending MS Word VBO is not the best idea

Nothing stops us from extending the object. However, this is not ideal because:

IT administrators become scared of updating the object since it is hard to migrate the newly added actions
We stop understanding what’s in the object. Apart from new actions, someone may have actually modified existing ones and broken the object. We can’t trust it anymore
This breaks Open Closed principle – objects should be open for extension but closed for modification
MS Word VBO is written in VB.NET. If we want to use C#, we can’t do that as we can’t mix languages inside an object
It’s hard to share functionality between projects. Ideally, we’d like to be able to share our actions between projects and separate them from actions that we don’t want to share. This is much harder if we put these actions directly on the object
Actions are much further from the business context. If we need a specific action for one of our processes, then it’s probably better to keep the action with other actions in the process-specific object

There are probably many more reasons to avoid that. Generally, apart from technical limitations (like being unable to use C#), this is just a matter of best practices and good design.

Why it’s hard to put actions outside of the object

First and foremost, Word instance is a COM object. Without going into details, it’s an object that is globally visible in the operating system and exposes many APIs. You can call this object from Blue Prism, Visual Basic for Applications, PowerShell, Visual Basic Script, C#, Java, and much more.

However, actions in Blue Prism objects cannot return any type they want. They can deal with numbers, texts, collections, and some other things. However, they can’t return an arbitrary object type. This means that the action cannot return a Word instance. To work that around, MS Word Object returns a handle which is an integer (a number). So when we call MS Word.Create Instance, then the method returns a number that identifies the word instance held behind the scenes. If we then call a method like MS Word.Open, the MS Word Object must find the actual Word instance based on the number.

Technically, it works in the following way. When creating an instance, this is the code that is executed:

Dim word as Object = CreateObject("Word.Application")

' Create a GUID with which we can kill the instance later
' if we have to play hardball to get rid of it.
word.Caption = System.Guid.NewGuid().ToString().ToUpper()

handle = GetHandle(word)

We create an instance of the Word object, and then we get the handle for it. This is GetHandle:

' Gets the handle for a given instance
'
' If the instance is not yet held, then it is added to the 
' 	map and a handle is assigned to it. It is also set as the
' 	'current' instance, accessed with a handle of zero in the
' 	below methods.
'
' Either way, the handle which identifies the instance is returned
'
' @param Instance The instance for which a handle is required
'
' @return The handle of the instance
Protected Function GetHandle(Instance As Object) As Integer

	If Instance Is Nothing Then
		Throw New ArgumentNullException("Tried to add an empty instance")
	End If

	' Check if we already have this instance - if so, return it.
	If InstanceMap.ContainsKey(Instance) Then
		CurrentInstance = Instance
		Return InstanceMap(Instance)
	End If

	Dim key as Integer
	For key = 1 to Integer.MaxValue
		If Not HandleMap.ContainsKey(key)
			HandleMap.Add(key, Instance)
			InstanceMap.Add(Instance, key)
			Me.CurrentInstance = Instance
			Return key
		End If
	Next key

	Return 0

End Function

We first check if the Instance is empty. It is not, as we just created it. We then check if the instance is already in the InstanceMap collection that holds all the Word instances we already created. At this point, it is not there. In that case, we simply iterate over all numbers from 1 going up, and then we find the first unused number. We add the instance to the HandleMap with key equal to 1, and then we return 1 as the handle that the user will use later on.

Let’s now say that you call Open. This is what it does behind the scenes:

' Just ensure that the handle references a valid instance
HandleMissing = (GetInstance(handle) is Nothing)

This calls GetInstance which looks like this:

' Gets the instance corresponding to the given handle, setting
' 	the instance as the 'current' instance for future calls
'
' A value of 0 will provide the 'current' instance, which
' 	is set each time an instance is added or accessed.
'
' This will return Nothing if the given handle does not
' correspond to a registered instance, or if the current
' instance was closed and the reference has not been updated.
'
' @param Handle The handle representing the instance required,
' 		or zero to get the 'current' instance.
Protected Function GetInstance(Handle As Integer) As Object

	Dim Instance As Object = Nothing
	
	If Handle = 0 Then
		If CurrentInstance Is Nothing Then
			' Special case - getting the current instance when the
			' instance is not set, try and get a current open instance.
			' If none there, create a new one and assign a handle as if
			' CreateInstance() had been called
		'	Try
		'		Instance = GetObject(,"Word.Application")
		'	Catch ex as Exception ' Not running
		'		Instance = Nothing
		'	End Try
		'	If Instance Is Nothing Then
				Create_Instance(Handle)
				' Instance = CreateObject("Word.Application")
				' Force the instance into the maps.
				' GetHandle(Instance)
				' CurrentInstance should now be set.
				' If it's not, we have far bigger problems
		'	End If
		End If
		Return CurrentInstance
	End If
	
	If Not HandleMap.ContainsKey(Handle)
            Throw New ArgumentException("A valid attachment to the application cannot been detected. Please check that Blue Prism is attached to an instance of the application.")
	End If

	Instance = HandleMap(Handle)
	If Not Instance Is Nothing Then
		CurrentInstance = Instance
	End If
	Return Instance

End Function

We just check if the handle is in the HandleMap and then store it in the CurrentInstance. So this is how we get the Word instance back from the integer passed by the user.

To use the Word instance in some other object, we would need to access the Word instance. However, we can’t just return it from the action as Blue Prism doesn’t support returning any type. We could also try to get access to HandleMap and then extract the instance. This is doable but far from straightforward. However, we can use some clever tricks to get access.

One of possible workarounds – truly global variable in Blue Prism

The idea is to create a global dictionary for sharing any data between any objects in Blue Prism process. The solution will work like this:

We create a code block that creates a globally-accessible dictionary storing string-object pairs
We add one action to MS Word VBO to put the Word COM instance in the dictionary based on the handle
In some other object (like our custom MS Word Object), we extract the com instance from the globally-accessible dictionary and use it accordingly

Let’s see that in action.

First, let’s create an object CustomWordObject.

We need these two external references added:

System.Windows.Forms.dll

and

C:\Program Files (x86)\Microsoft Visual Studio\Shared\Visual Studio Tools for Office\PIA\Office15\Microsoft.Office.Interop.Word.dll

You may need to adjust the path to the Word dll based on your Word version.

Now, let’s create an action named CreateGlobalDictionary with the following content:

16965f9d-eb5e-44bc-bb24-fb27327560631e2e0509-5b3b-4b0b-8d20-dcc5b5e9174316965f9d-eb5e-44bc-bb24-fb273275606316965f9d-eb5e-44bc-bb24-fb2732756063ced18beb-abce-4eac-92d4-e627fa99a391), System.Reflection.FieldAttributes.Public | System.Reflection.FieldAttributes.Static);
	Type t = typeBuilder.CreateType();
	System.Reflection.FieldInfo fieldInfo = t.GetField("GlobalDictionary");
	fieldInfo.SetValue(null, new System.Collections.Generic.Dictionary());
}catch(Exception e){
	System.Windows.Forms.MessageBox.Show(e.ToString());
}]]>

Specifically, there is a code block that does the following:

try{
	var name = new System.Reflection.AssemblyName("GlobalAssemblyForSharingData");
	var assemblyBuilder = System.Reflection.Emit.AssemblyBuilder.DefineDynamicAssembly(name, System.Reflection.Emit.AssemblyBuilderAccess.Run);
	var moduleBuilder = assemblyBuilder.DefineDynamicModule(name.Name ?? "GlobalAssemblyForSharingDataModule");
	var typeBuilder = moduleBuilder.DefineType("GlobalType", System.Reflection.TypeAttributes.Public);
	var fieldBuilder = typeBuilder.DefineField("GlobalDictionary", typeof(System.Collections.Generic.Dictionary), System.Reflection.FieldAttributes.Public | System.Reflection.FieldAttributes.Static);
	Type t = typeBuilder.CreateType();
	System.Reflection.FieldInfo fieldInfo = t.GetField("GlobalDictionary");
	fieldInfo.SetValue(null, new System.Collections.Generic.Dictionary());
}catch(Exception e){
	System.Windows.Forms.MessageBox.Show(e.ToString());
}

We use some .NET magic to dynamically create an assembly named GlobalAssemblyForSharingData, add a type named GlobalType to it, add a global field named GlobalDictionary, and initialize it accordingly.

Next, let’s add the action to MS Word VBO named ExportWord with the following content:

5600ecd3-0257-4793-8c9b-58fba32fc417064c4864-b486-4ff5-9e44-0d9b0e11c0035600ecd3-0257-4793-8c9b-58fba32fc4175600ecd3-0257-4793-8c9b-58fba32fc4172ef37518-b895-403e-93ee-76b9743c2cf05600ecd3-0257-4793-8c9b-58fba32fc417number5600ecd3-0257-4793-8c9b-58fba32fc417text

Let’s see the code in detail:

Dim identifier = Guid.NewGuid().ToString()
Dim globalDictionary As System.Collections.Generic.Dictionary(Of String, Object) = Nothing


For Each assembly In System.AppDomain.CurrentDomain.GetAssemblies()
	Try
		For Each type In assembly.GetTypes()
			If type.Name = "GlobalType" Then
				globalDictionary = CType(type.GetField("GlobalDictionary").GetValue(Nothing), System.Collections.Generic.Dictionary(Of String, Object))
			End If
		Next

	Catch e As Exception
	End Try
		
Next

globalDictionary(identifier) = GetInstance(handle)

word = identifier

We list all the assemblies, then we find all the types, then we find the type with the global dictionary. Next, we get the dictionary and put the COM object in it. Finally, we return the identifier.

With the regular actions on the MS Word Object, we need to identify the Word instance by using an integer handle. For our custom actions, we will use a string identifier that serves the same purpose. Just like the regular actions accept handle and other parameters, we’ll accept the same with only a different identifier type.

Lastly, we create a new action in our custom object. Let’s say that we would like to show the instance of the Word application. This is the action:

c76f8dc4-c63e-4820-b619-475abe9c3adc659b998f-463b-401a-b72f-e7f809dc5151c76f8dc4-c63e-4820-b619-475abe9c3adcc76f8dc4-c63e-4820-b619-475abe9c3adctextc76f8dc4-c63e-4820-b619-475abe9c3adc5a5fb9ef-562a-457d-945f-432e8ac1610c globalDictionary = null;
		foreach(var assembly in System.AppDomain.CurrentDomain.GetAssemblies()){
			try{
				foreach(var type in assembly.GetTypes()){
					if(type.Name == "GlobalType"){
						globalDictionary = (System.Collections.Generic.Dictionary)type.GetField("GlobalDictionary").GetValue(null);
					}
				}
			}catch(Exception e){
			}
		}
((Microsoft.Office.Interop.Word.ApplicationClass)globalDictionary[identifier]).Visible = true;]]>

And here is the code specifically:

System.Collections.Generic.Dictionary globalDictionary = null;
		foreach(var assembly in System.AppDomain.CurrentDomain.GetAssemblies()){
			try{
				foreach(var type in assembly.GetTypes()){
					if(type.Name == "GlobalType"){
						globalDictionary = (System.Collections.Generic.Dictionary)type.GetField("GlobalDictionary").GetValue(null);
					ac	}
				}
			}catch(Exception e){
			}
		}
((Microsoft.Office.Interop.Word.ApplicationClass)globalDictionary[identifier]).Visible = true;

You can see that it’s the same code for obtaining the dictionary as before. We list all the types, we then find the dictionary and extract the COM instance based on the identifier obtained when exporting the instance. We then cast the object to the known interface and then simply change the property to show the Word application.

This is how we would use it in the workflow:

24a17572-5a8c-4ad1-8119-36c634d3cc7554e185fe-aa0e-4796-b4d9-ab85cf14aee2number8cfb4920-496f-4db5-8098-ee503aec2b896c70b7da-9c20-461e-97e3-99b3617d210916f8e48f-b508-4bad-b79c-7e38fa5f7cf8text480cfc9d-c4c1-4556-958f-17c471c330ef

You can see that we first open the Word instance and open the file using the regular Word object. We then call CreateGlobalDictionary and ExportWord to export the instance. Finally, we call ShowWord from our custom object. This way, you can add any action in your business object and not touch the Blue Prism’s stock object anymore.

Further improvements

It’s worth noting that this solution provides a truly global dictionary for sharing any data between any objects. Nothing stops us from sharing other things. Just don’t abuse the mechanism.

It’s also worth noticing that there is a big code duplication. We could put it in a NuGet package to reuse the code easily.

Bit Twiddling Part 6 — Stop RDP from detaching GUI when the client disconnects

afish — Wed, 05 Jun 2024 11:06:31 +0000

This is the sixth part of the Bit Twiddling series. For your convenience you can find other parts in the table of contents in Par 1 — Modifying Android application on a binary level

Today we’re going to solve the problem when the remote server disconnects GUI when we lock the workstation or disconnect.

Let’s start by explaining the problem a little bit more. Imagine that you connect like this:

Local ---> Remote

If you now lock the Local workstation (with WIN+L) or disconnect, then the GUI on Remote will break. The user on Remote will not be logged out, the applications will continue to work, but the UI will not be there. This effectively breaks things like VNC or apps that take screenshots or click on the screen.

Similarly, the same happens when you connection with a jump host:

Local ---> Windows Server 2019 ---> Remote

If you now disconnect Local or lock Local workstation, then the GUI on Remote will break. However, it’s different if you use Windows Server 2016:

Local ---> Windows Server 2016 ---> Remote

If you now disconnect Local or lock the workstation, then the GUI on Remote will still be there. This suggests that something has changed in Windows Server 2019. This is not surprising as this is the time when Microsoft implemented RemoteFX heavily into their RDP to support cameras, USB redirection, improve security and much more.

At this point we have one solution – just use the Windows Server 2016 jump host and it fixes the issue. However, this way you won’t use the camera (it doesn’t work in 2016 typically) and you need to keep another machine along the way (this could be a virtual machine on your Remote, though). Let’s see if we can do better.

Fixing the workstation lock

The first thing to notice is that if you suspend the mstsc.exe on Local (with Task Manager or debugger) and then lock the workstation, then Remote doesn’t lose the UI. This suggests that disconnecting the GUI is an explicit action of the RDP implementation. However, we can easily fix that.

mstsc.exe registers for session notifications with WTSRegisterSessionNotification. We can see that with ApiMonitor:

We can simply hook this method to not register for any notifications. We can use that with the following WinDBG script:

.sympath srv*C:\tmp*http://msdl.microsoft.com/download/symbols
.reload
bu	WTSAPI32!WTSRegisterSessionNotification	""e @rip 0x48 0xc7 0xc0 0x01 0x00 0x00 0x00 0xC3; qd""
g

The shell code we put here is simply:

move rax, 1
ret

You can run mstsc.exe, attach the debugger, add the breakpoint, resume the application, and then connect to the server. The debugger will detach shortly after you open the connection and you can check that now you can lock the workstation and the GUI will still be there.

Fixing the disconnection

Fixing the case on disconnect is slightly harder. Once again, when we pause the mstsc.exe, then the remote session still works even if we keep mstsc.exe paused for a long time. This suggests that the session is up as long as the connection is alive. Even though we paused the mstsc.exe application, the operating system keeps the connection up on the TCP level. We can exhibit that.

The idea is to run a proxy on the Remote. We’ll then connect like this:

Local ---> Proxy ---> Remote

The proxy must just route the connection to the RDP server (on the port 3389),but do not lose the connection to Remote when the connection from Local disappears. This way we can simply kill the mstsc.exe on Local, and then the connection will still be alive and the GUI will persist. However, if you close mstsc.exe gracefully, then the client will terminate the GUI on the remote server. Therefore, just kill the client. You can also put your machine to sleep and the UI should still work.

Bit Twiddling Part 5 — Fixing audio latency in mstsc.exe (RDP)

afish — Thu, 30 May 2024 10:34:31 +0000

This is the fifth part of the Bit Twiddling series. For your convenience you can find other parts in the table of contents in Par 1 — Modifying Android application on a binary level

Today we’re going to fix the audio latency in mstsc.exe. Something that people really ask about on the Internet and there is no definite solution. I’ll show how to hack mstsc.exe to fix the latency. First, I’ll explain why existing solutions do not work and what needs to be done, and at the end of this post I provide an automated PowerShell script that does the magic.

What is the issue

First, a disclaimer. I’ll describe what I suspect is happening. I’m not an expert of the RDP and I didn’t see the source code of mstsc.exe. I’m just guessing what happens based on what I see.

RDP supports multiple channels. There is a separate channel for video and another one for audio. Unfortunately, these two channels are not synchronized which means that they are synchronized on the client on a best effort basis (or rather sheer luck). To understand why it’s hard to synchronize two independent streams, we need to understand how much different they are.

Video is something that doesn’t need to be presented “for some time”. We can simply take the last video frame and show it to the user. If frames are buffered, we just process them as fast as possible to present the last state. You can see that mstsc.exe does exactly that by connecting to a remote host, playing some video, and then suspending the mstsc.exe process with ProcessExplorer. When you resume it after few seconds, you’ll see the video moving much faster until the client catches up with the latest state.

When it comes to audio, things are much different. You can’t just jump to the latest audio because you’d loose the content as it would be unintelligible. Each audio piece has its desired length. When you get delayed, you could just skip the packets (and lose some audio), play it faster (which would make it less intelligible for some time), or just play it as it goes. However, to decide what to do, you would need to understand whether the piece you want to play is delayed or not. You can’t tell that without timestamps or time markers, and I believe RDP doesn’t send those.

As long as you’re getting audio packets “on time”, there is no issue. You just play them. The first part is if you get them “in time”. This depends on your network quality and on the server that sends you the sound. From my experience, Windows Server is better when it comes to speed of sending updates. I can see my mouse moving faster and audio delayed less often when I connect to the Windows Server than Windows 10 (or other client edition). Therefore, use Windows Server where possible. Just keep in mind that you’ll need CAL licenses to send microphone and camera to the server which is a bummer (client edition lets you do that for free). Making the packets to be sent as fast as possible is the problem number one.

However, there is another part to that. If your client gets delayed for whatever reason (CPU spike, overload, or preemption), your sound will effectively slow down. You will just play it “later” even though you received it “on time”. Unfortunately, RDP cannot detect whether this happened because there are no timestamps in the stream. As far as I can tell, this is the true reason why your sound is often delayed. You can get “no latency audio” over the Internet and I had it many times. However, the longer you run the client, the higher the chance that you’ll go out of sync. This is the problem number two.

Therefore, we need to “resync” the client. Let’s see how to do it.

Why existing solutions don’t work and what fix we need

First, let me explain why existing solutions won’t work. Many articles on the Internet tell you to change the audio quality to High in Group Policy and enforce the quality in your rdp.file by setting audioqualitymode:i:2. This fixes the problem number one (although I don’t see much difference to be honest), but it doesn’t address the problem number two.

Some other articles suggest other fixes on the remote side. All these fixes have one thing in common – they don’t fix the client. If the mstsc.exe client cannot catch up when it gets delayed, then the only thing you can do is to reset the audio stream. Actually, this is how you can fix the delay easily – just restart the audio service:

net stop audiosrv & timeout 3 & net start audiosrv

I add the timeout to give some time to clear the buffer on the client. Once the buffer is empty, we restart the service and then the audio should be in sync. Try this to verify if it’s physically possible to deliver the audio “on time” in your networking conditions.

Unfortunately, restarting the audio service have many issues. First, it resets the devices on the remote end, so your audio streams may break and you’ll need to restart them. Second, this simply takes time. You probably don’t want to lose a couple of seconds of audio and microphone when you’re presenting (and unfortunately, you’ll get delayed exactly during that time).

What we need is to fix the client to catch up when there is a delay. However, how can we do that when we don’t have any timestamps? Well, the solution is simple – just drop some audio packages (like 10%) periodically to sync over time. This will decrease the audio quality to some extent and won’t fix the delay immediately, but after few seconds we’ll get back on track. Obviously, you could implement some better heuristics and solutions. There is one problem, though – we need to do that in the mstsc.exe itself. And here comes the low level magic. Let’s implement the solution that drops the audio frames to effectively “resync” the audio.

How to identify

We need to figure out how the sound is played. Let’s take ApiMonitor to trace the application. Luckily enough, it seems that the waveOutPrepareHeader is used, as we can see in the screenshot:

Let’s break there win WinDBG:

bu 0x00007ffb897d3019

kb

 # RetAddr               : Args to Child                                                           : Call Site														
00 00007ffb`897d1064     : 00000252`6d68beb8 00000252`6d68beb8 00000000`000014ac 00000252`6d68be00 : mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+0x6d														
01 00007ffb`897e7ee8     : 00000000`00000000 00000073`bfc7fcb9 00000252`6d68be00 00000252`7e69df80 : mstscax!CRdpWinAudioWaveoutPlayback::vcwaveWritePCM+0xec														
02 00007ffb`898e73bf     : 00000000`00000001 00000000`00000003 00000073`bfc7d055 00000073`00001000 : mstscax!CRdpWinAudioWaveoutPlayback::RenderThreadProc+0x2c8														
03 00007ffc`02e57344     : 00000000`000000ac 00000252`6d68be00 00000000`00000000 00000000`00000000 : mstscax!CRdpWinAudioWaveoutPlayback::STATIC_ThreadProc+0xdf														
04 00007ffc`046826b1     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x14														
05 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

We can see a method named mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite. Let’s see it:

u mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+0x6d		
									
mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite:											
00007ffb`897d2fac 48895c2410      mov     qword ptr [rsp+10h],rbx											
00007ffb`897d2fb1 55              push    rbp											
00007ffb`897d2fb2 56              push    rsi											
00007ffb`897d2fb3 57              push    rdi											
00007ffb`897d2fb4 4883ec40        sub     rsp,40h											
00007ffb`897d2fb8 488bf2          mov     rsi,rdx											
00007ffb`897d2fbb 488bd9          mov     rbx,rcx											
00007ffb`897d2fbe 488b0543287500  mov     rax,qword ptr [mstscax!WPP_GLOBAL_Control (00007ffb`89f25808)]											
00007ffb`897d2fc5 488d2d3c287500  lea     rbp,[mstscax!WPP_GLOBAL_Control (00007ffb`89f25808)]											
00007ffb`897d2fcc 483bc5          cmp     rax,rbp											
00007ffb`897d2fcf 740a            je      mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+0x2f (00007ffb`897d2fdb)											
00007ffb`897d2fd1 f6401c01        test    byte ptr [rax+1Ch],1											
00007ffb`897d2fd5 0f85e8000000    jne     mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+0x117 (00007ffb`897d30c3)											
00007ffb`897d2fdb 83bbc000000000  cmp     dword ptr [rbx+0C0h],0											
00007ffb`897d2fe2 7418            je      mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+0x50 (00007ffb`897d2ffc)											
00007ffb`897d2fe4 488b8bb8000000  mov     rcx,qword ptr [rbx+0B8h]											
00007ffb`897d2feb 4885c9          test    rcx,rcx											
00007ffb`897d2fee 740c            je      mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+0x50 (00007ffb`897d2ffc)											
00007ffb`897d2ff0 48ff15a9295c00  call    qword ptr [mstscax!_imp_EnterCriticalSection (00007ffb`89d959a0)]											
00007ffb`897d2ff7 0f1f440000      nop     dword ptr [rax+rax]											
00007ffb`897d2ffc 488b4b70        mov     rcx,qword ptr [rbx+70h]											
00007ffb`897d3000 4885c9          test    rcx,rcx											
00007ffb`897d3003 0f84f2000000    je      mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+0x14f (00007ffb`897d30fb)											
00007ffb`897d3009 41b830000000    mov     r8d,30h											
00007ffb`897d300f 488bd6          mov     rdx,rsi											
00007ffb`897d3012 48ff15e7a87900  call    qword ptr [mstscax!_imp_waveOutPrepareHeader (00007ffb`89f6d900)]											
00007ffb`897d3019 0f1f440000      nop     dword ptr [rax+rax]

Okay, we can see that this method passes the audio packet to the API. When we look at WAVEHDR structure, we can see that it has the following fields:

typedef struct wavehdr_tag {
  LPSTR              lpData;
  DWORD              dwBufferLength;
  DWORD              dwBytesRecorded;
  DWORD_PTR          dwUser;
  DWORD              dwFlags;
  DWORD              dwLoops;
  struct wavehdr_tag  *lpNext;
  DWORD_PTR          reserved;
} WAVEHDR, *LPWAVEHDR;

This is exactly what we see in ApiMonitor. Seems like the dwBufferLength is what we might want to change. When we shorten this buffer, we’ll effectively make the audio last shorter. We can do that for some of the packets to not break the quality much, and then all should be good.

We can verify that this works with this breakpoint:

bp 00007ffb`897d3012 "r @$t0 = poi(rdx + 8); r @$t1 = @$t0 / 2; ed rdx+8 @$t1; g"

Unfortunately, this makes the client terribly slow. We need to patch the code in place. Effectively, we need to inject a shellcode.

First, we need to allocate some meory with VirtualAllocEx via .dvalloc.

.dvalloc 1024

The debugger allocates the memory. In my case the address is 25fb8960000.

The address of the WinAPI function is in the memory, so we need to remember to extract the pointer from the address:

00007ffb`897d3012 48ff15e7a87900  call    qword ptr [mstscax!_imp_waveOutPrepareHeader (00007ffb`89f6d900)]

Now we need to do two things: first, we need to patch the call site to call our shellcode instead of [mstscax!_imp_waveOutPrepareHeader (00007ffb89f6d900)]. Second, we need to construct a shell code that fixes the audio packet for some of the packets, and then calls [mstscax!_imp_waveOutPrepareHeader (00007ffb89f6d900)] correctly.

To do the first thing, we can do the absolute jump trick. We put the address in the rax register, push it on the stack, and then return. This is the code:

mov rax, 0x25fb8960000	;Move the address to the register
push rax		;Push the address on the stack
ret			;Return. This takes the address from the stuck and jumps
nop			;Nops are just to not break the following instructions when you disassemble with u
nop
nop
nop

We can compile the code with Online assembler and we should get a shell code. We can then put it in place with this line:

e	mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+5d		0x48 0xB8 0x00 0x00 0x96 0xb8 0x5f 0x02 0x00 0x00 0x50 0xC3 0x90 0x90 0x90 0x90

Unfortunately, this patch is long. We need to break few lines and then restore them in the shellcode. So our shell code starts with the lines that we broke:

mov r8d, 0x30	;Preserved code
mov rdx,rsi	;Preserved code

Next, we need to preserve our working registers:

push rbx
push rdx

Okay, now we can do the logic. We want to modify the buffer length. However, we don’t want to do it for all of the packets. We need some source of randomness, like time, random values, or something else. Fortunately, the WAVEHDR structure has the field dwUser which may be “random enough” for our needs. Let’s take that value modulo some constant, and then change the packet length only for some cases.

First, let’s preserve the buffer length for the sake of what we do later:

mov rax, [rdx + 8]	;Load buffer length (that's the second field of the structure)
push rax		;Store buffer length on the stack

Now, let’s load dwUser and divide it by some constant like 23:

mov rax, [rdx + 16]	;Load dwUser which is the fourth field
mov rbx, 23		;Move the constant to the register
xor rdx, rdx		;Clear upper divisor part
div rbx			;Divide

Now, we can restore the buffer length to rax:

pop rax	;Restore buffer length

At this point we have rax with the buffer length, and rdx with the remainder. We can now compare the reminder and skip the code modifying the pucket length if needed:

cmp rdx, 20	;Compare with 20	
jbe 0x17	;Skip the branch

We can see that we avoid the buffer length modification if the remainder is at most 20. Effectively, we have 20/22 = 0.909% chance that we won’t modify the package. This means that we modify something like 9% of the packages, assuming the dwUser has a good distribution. The code is written in this way so you can tune the odds of changing the packet.

Now, let’s modify the package. We want to divide the buffer length by 2, however, we want to keep it generic to be able to experiment with other values:

mov rbx, 1	;Move 1 to rbx to multiply by 1/2
xor rdx, rdx	;Clear remainder
mul rbx		;Multiply
mov rbx, 2	;Store 2 to rbx to multiply by 1/2
xor rdx, rdx	;Clear remainder
div rbx		;Divide

You can play with other values, obviously. From my experiments, halving the value works the best.

Now it’s rather easy. rax has the new buffer length or the original one if we decided not to modify it. Let’s restore other registers:

pop rdx
pop rbx

Let’s update the buffer length:

mov [rdx + 8], rax

Now, we need to prepare the jump addresses. First, we want to put the original return address of the method mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite which is mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+0x6d:

mov rax, 0x00007ffb897d3019
push rax

Now, we can jump to the WinAPI method:

push rbx
mov rbx, 0x00007ffbe6c6a860
mov rax, [rbx]
pop rbx
push rax
ret

That’s it. The final shellcode looks like this:

mov r8d, 0x30
mov rdx,rsi
push rbx
push rdx
mov rax, [rdx + 8]
push rax
mov rax, [rdx + 16]
mov rbx, 23
xor rdx, rdx
div rbx
pop rax
cmp rdx, 20
jbe 0x17
mov rbx, 1
xor rdx, rdx
mul rbx
mov rbx, 2
xor rdx, rdx
div rbx
pop rdx
pop rbx
mov [rdx + 8], rax
mov rax, 0x00007ffb897d3019
push rax
push rbx
mov rbx, 0x00007ffbe6c6a860
mov rax, [rbx]
pop rbx
push rax
ret

We can implant it with this:

e	25f`b8960000		0x41 0xB8 0x30 0x00 0x00 0x00 0x48 0x89 0xF2 0x53 0x52 0x48 0x8B 0x42 0x08 0x50 0x48 0x8B 0x42 0x10 0x48 0xC7 0xc3 0x17 0x00 0x00 0x00 0x48 0x31 0xD2 0x48 0xF7 0xF3 0x58 0x48 0x83 0xFA 0x14 0x76 0x1A 0x48 0xC7 0xC3 0x01 0x00 0x00 0x00 0x48 0x31 0xD2 0x48 0xF7 0xE3 0x48 0xC7 0xC3 0x02 0x00 0x00 0x00 0x48 0x31 0xD2 0x48 0xF7 0xF3 0x5A 0x5B 0x48 0x89 0x42 0x08 0x48 0xB8 0x19 0x30 0x7D 0x89 0xFB 0x7F 0x00 0x00 0x50 0x48 0xBB 0x60 0xA8 0xC6 0xE6 0xFB 0x7F 0x00 0x00 0x50 0xC3

Automated fix

We can now fix the code automatically. We need to do the following:

Start mstsc.exe
Find it’s process ID
Attach the debugger and find all the addresses: free memory, mstsc.exe method, WinAPI method
Construct the shellcode
Attach the debugger and patch the code
Detach the debugger

We can do all of that with PowerShell. Here is the code:

Function Run-Mstsc($rdpPath, $cdbPath, $numerator, $denominator){
	$id = get-random
	$code = @"
using System;
using System.Collections;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading;
	
namespace MstscPatcher
{
	public class Env$id {
		public static void Start() {
			Process[] processes = Process.GetProcessesByName("mstsc");
			RunProcess("mstsc.exe", "$rdpPath");
			Thread.Sleep(3000);
			Process[] processes2 = Process.GetProcessesByName("mstsc");
			var idToPatch = processes2.Select(p => p.Id).OrderBy(i => i).Except(processes.Select(p => p.Id).OrderBy(i => i)).First();
			Patch(idToPatch);
		}
		
		public static void Patch(int id){
			Console.WriteLine(id);
			var addresses = RunCbd(id, @"
.sympath srv*C:\tmp*http://msdl.microsoft.com/download/symbols
.reload
!address
u mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+100
.dvalloc 1024
qd
			");
			
			string freeMemoryAddress = addresses.Where(o => o.Contains("Allocated 2000 bytes starting at")).First().Split(' ').Last().Trim();
			Console.WriteLine("Free memory: " + freeMemoryAddress);
			
			var patchAddress = addresses.SkipWhile(o => !o.StartsWith("mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite:"))
				.SkipWhile(o => !o.Contains("r8d,30h"))
				.First().Split(' ').First().Trim();
			Console.WriteLine("Patch address: " + patchAddress);
			var returnAddress = (Convert.ToUInt64(patchAddress.Replace(((char)96).ToString(),""), 16) + 0x10).ToString("X").Replace("0x", "");
			Console.WriteLine("Return address: " + returnAddress);
			
			var winApiAddress = addresses.SkipWhile(o => !o.Contains("[mstscax!_imp_waveOutPrepareHeader"))
				.First().Split('(')[1].Split(')')[0].Trim();
			Console.WriteLine("WinAPI address: " + winApiAddress);
			
			Func> splitInPairs = address => address.Where((c, i) => i % 2 == 0).Zip(address.Where((c, i) => i % 2 == 1), (first, second) => first.ToString() + second.ToString());			
			Func translateToBytes = address => string.Join(" ", splitInPairs(address.Replace(((char)96).ToString(), "").PadLeft(16, '0')).Reverse().Select(p => "0x" + p));
						
			var finalScript = @"
.sympath srv*C:\tmp*http://msdl.microsoft.com/download/symbols
.reload
e	" + patchAddress + @"	0x48 0xB8 " + translateToBytes(freeMemoryAddress) + @" 0x50 0xC3 0x90 0x90 0x90 0x90	
e	" + freeMemoryAddress + @"	0x41 0xB8 0x30 0x00 0x00 0x00 0x48 0x89 0xF2 0x53 0x52 0x48 0x8B 0x42 0x08 0x50 0x48 0x8B 0x42 0x10 0x48 0xC7 0xc3 $denominator 0x00 0x00 0x00 0x48 0x31 0xD2 0x48 0xF7 0xF3 0x58 0x48 0x83 0xFA $numerator 0x76 0x1A 0x48 0xC7 0xC3 0x01 0x00 0x00 0x00 0x48 0x31 0xD2 0x48 0xF7 0xE3 0x48 0xC7 0xC3 0x02 0x00 0x00 0x00 0x48 0x31 0xD2 0x48 0xF7 0xF3 0x5A 0x5B 0x48 0x89 0x42 0x08 0x48 0xB8 " + translateToBytes(returnAddress) + @" 0x50 0x53 0x48 0xBB " + translateToBytes(winApiAddress) + @" 0x48 0x8B 0x03 0x5B 0x50 0xC3	
qd
			";
			Console.WriteLine(finalScript);
			RunCbd(id, finalScript);
		}
		
		public static string[] RunCbd(int id, string script) {
			Console.WriteLine(script);
			File.WriteAllText("mstsc.txt", script);
			string output = "";
			Process process = RunProcess("$cdbPath", "-p " + id + " -cf mstsc.txt", text => output += text + "\n");
			process.WaitForExit();
			File.Delete("mstsc.txt");
			
			return output.Split('\n');
		}
		
		public static Process RunProcess(string fileName, string arguments, Action outputReader = null){
			ProcessStartInfo startInfo = new ProcessStartInfo
			{
				FileName = fileName,
				Arguments = arguments,
				UseShellExecute = outputReader == null,
				RedirectStandardOutput = outputReader != null,
				RedirectStandardError = outputReader != null
			};
			
			if(outputReader != null){
				var process = new Process{
					StartInfo = startInfo
				};
				process.OutputDataReceived += (sender, args) => outputReader(args.Data);
				process.ErrorDataReceived += (sender, args) => outputReader(args.Data);

				process.Start();
				process.BeginOutputReadLine();
				process.BeginErrorReadLine();
				return process;
			}else {
				return Process.Start(startInfo);
			}
		}
	}
}
"@.Replace('$id', $id)
	$assemblies = ("System.Core","System.Xml.Linq","System.Data","System.Xml", "System.Data.DataSetExtensions", "Microsoft.CSharp")
	Add-Type -referencedAssemblies $assemblies -TypeDefinition $code -Language CSharp
	iex "[MstscPatcher.Env$id]::Start()"
}


Run-Mstsc "my_rdp_settings.rdp".Replace("\", "\\") "cdb.exe".Replace("\", "\\") "0x14" "0x17"

We compile some C# code on the fly. First, we find existing mstsc.exe instances (line 16), then run the new instance (line 17), wait a bit for the mstsc.exe to spawn a child process, and then find the id (lines 19-20). We can then patch the existing id.

First, we look for addresses. We do all the manual steps we did above to find the memory address, and two function addresses. The script is in lines 27-32. Notice that I load symbols as we need them and CDB may not have them configured on the box.

We can now parse the output. We first extract the allocated memory in lines 35-36.

Next, we look for the call site. We dump the whole method, and then find the first occurrence of mov 8d,30h. That’s our call site. This is in lines 38-41.

Next, we calculate the return address which is 16 bytes further. This is in lines 42-43.

Finally, I calculate the WinAPI method address. I extract the location of the pointer for the method (lines 45-47).

Next, we need to construct the shell code. This is exactly what we did above. We just need to format addresses properly (this is in helper methods in lines 49-50), and then build the script (lines 52-558). We can run it and that’s it. The last thing is customization of the values. You can see in line 108 that I made two parameters to change numerator and denominator for the odds of modifying the package. This way you can easily change how many packets are broken. The more you break, the faster you resynchronize, however, the worse the sound is.

That’s it. I verified that on Windows 10 22H2 x64, Windows 11 22H2 x64, Windows Server 2016 1607 x64, and Windows Server 2019 1809 x64, and it worked well. Your mileage may vary, however, the approach should work anywhere. Generally, to make this script work somewhere else, you just need to adjust how we find the call site, the return address, and the address of the WinAPI function. Assuming that the WinAPI is still called via the pointer stored in memory, then you won’t need to touch the machine code payload.

Below is the script for x86 bit (worked on Windows 10 10240 x86). Main differences are in how we access the data structure as the pointer is on the stack (and not in the register).

Function Run-Mstsc($rdpPath, $cdbPath, $numerator, $denominator){
	$id = get-random
	$code = @"
using System;
using System.Collections;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading;
	
namespace MstscPatcher
{
	public class Env$id {
		public static void Start() {
			Process[] processes = Process.GetProcessesByName("mstsc");
			RunProcess("mstsc.exe", "$rdpPath");
			Thread.Sleep(3000);
			Process[] processes2 = Process.GetProcessesByName("mstsc");
			var idToPatch = processes2.Select(p => p.Id).OrderBy(i => i).Except(processes.Select(p => p.Id).OrderBy(i => i)).First();
			Patch(idToPatch);
		}
		
		public static void Patch(int id){
			Console.WriteLine(id);
			var addresses = RunCbd(id, @"
.sympath srv*C:\tmp*http://msdl.microsoft.com/download/symbols
.reload
!address
u mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite+100
.dvalloc 1024
qd
			");
			
			string freeMemoryAddress = addresses.Where(o => o.Contains("Allocated 2000 bytes starting at")).First().Split(' ').Last().Trim();
			Console.WriteLine("Free memory: " + freeMemoryAddress);
			
			var patchAddress = addresses.SkipWhile(o => !o.StartsWith("mstscax!CRdpWinAudioWaveoutPlayback::vcwaveOutWrite:"))
				.SkipWhile(o => !(o.Contains("dword ptr [ebx+3Ch]") && o.Contains("push")))
				.First().Split(' ').First().Trim();
			Console.WriteLine("Patch address: " + patchAddress);
			var returnAddress = (Convert.ToUInt64(patchAddress.Replace(((char)96).ToString(),""), 16) + 0x9).ToString("X").Replace("0x", "");
			Console.WriteLine("Return address: " + returnAddress);
			
			var winApiAddress = addresses.SkipWhile(o => !o.Contains("[mstscax!_imp__waveOutPrepareHeader"))
				.First().Split('(')[1].Split(')')[0].Trim();
			Console.WriteLine("WinAPI address: " + winApiAddress);
			
			Func> splitInPairs = address => address.Where((c, i) => i % 2 == 0).Zip(address.Where((c, i) => i % 2 == 1), (first, second) => first.ToString() + second.ToString());			
			Func translateToBytes = address => string.Join(" ", splitInPairs(address.Replace(((char)96).ToString(), "").PadLeft(8, '0')).Reverse().Select(p => "0x" + p));
						
			var finalScript = @"
.sympath srv*C:\tmp*http://msdl.microsoft.com/download/symbols
.reload
e	" + patchAddress + @"	0xB8 " + translateToBytes(freeMemoryAddress) + @" 0x50 0xC3 0x90 0x90	
e	" + freeMemoryAddress + @"	0xFF 0x73 0x3C 0x53 0x52 0x8B 0x53 0x3C 0x52 0x8B 0x42 0x04 0x50 0x8B 0x42 0x0C 0xBB $denominator 0x00 0x00 0x00 0x31 0xD2 0xF7 0xF3 0x58 0x83 0xFA $numerator 0x76 0x12 0xBB 0x01 0x00 0x00 0x00 0x31 0xD2 0xF7 0xE3 0xBB 0x02 0x00 0x00 0x00 0x31 0xD2 0xF7 0xF3 0x5A 0x89 0x42 0x04 0x5A 0x5B 0xB8 " + translateToBytes(returnAddress) + @" 0x50 0x53 0xBB " + translateToBytes(winApiAddress) + @" 0x8B 0x03 0x5B 0x50 0xC3	
qd
			";
			Console.WriteLine(finalScript);
			RunCbd(id, finalScript);
		}
		
		public static string[] RunCbd(int id, string script) {
			Console.WriteLine(script);
			File.WriteAllText("mstsc.txt", script);
			string output = "";
			Process process = RunProcess("$cdbPath", "-p " + id + " -cf mstsc.txt", text => output += text + "\n");
			process.WaitForExit();
			File.Delete("mstsc.txt");
			
			return output.Split('\n');
		}
		
		public static Process RunProcess(string fileName, string arguments, Action outputReader = null){
			ProcessStartInfo startInfo = new ProcessStartInfo
			{
				FileName = fileName,
				Arguments = arguments,
				UseShellExecute = outputReader == null,
				RedirectStandardOutput = outputReader != null,
				RedirectStandardError = outputReader != null
			};
			
			if(outputReader != null){
				var process = new Process{
					StartInfo = startInfo
				};
				process.OutputDataReceived += (sender, args) => outputReader(args.Data);
				process.ErrorDataReceived += (sender, args) => outputReader(args.Data);

				process.Start();
				process.BeginOutputReadLine();
				process.BeginErrorReadLine();
				return process;
			}else {
				return Process.Start(startInfo);
			}
		}
	}
}
"@.Replace('$id', $id)
	$assemblies = ("System.Core","System.Xml.Linq","System.Data","System.Xml", "System.Data.DataSetExtensions", "Microsoft.CSharp")
	Add-Type -referencedAssemblies $assemblies -TypeDefinition $code -Language CSharp
	iex "[MstscPatcher.Env$id]::Start()"
}

Run-Mstsc "my_rdp_settings.rdp".Replace("\", "\\") "cdb.exe".Replace("\", "\\") "0x14" "0x17"

Some keyword to make this easier to find on the Internet

Her some keywords to make this article easier to find on the Internet.

audio latency in mstsc.exe
audio latency in rdp
audio delay in mstsc.exe
audio delay in rdp
laggy sound in rdp
sound desynchronized in rdp
sound latency in rdp
slow audio
how to fix audio in rdp

Enjoy!

Serializing collections with Jackson in Scala and renaming the nested element

afish — Mon, 20 May 2024 07:55:06 +0000

This is a workaround for XML collection duplicated element names that I was fixing recently.

We use case classes in Scala and the Jackson library for XML serialization. We have a class that has a list of some nested class. When serializing it, we would like to name the nested element properly.

We start with these classes:

case class Person(val Name: String)
 
@JsonPropertyOrder(Array("People"))
case class Base
(
	val People: List[Person]
)

When we try to serialize it, we’ll get something like this:

< Base>
    < People>
        < People>
            < Name>Some name here

You can see that we have Base -> People -> People instead of Base -> People -> Person. We can try to fix it with regular annotations:

case class Base
(
	@JacksonXmlElementWrapper(localName = "People")
	@JacksonXmlProperty(localName = "Person")
	val People: List[Person]
)

It now serializes correctly. However, when you try to deserialize it, you get the following exception:

Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Could not find creator property with name 'Person'

This fails because we use JacksonXmlProperty to rename the property and this gets handled incorrectly.

The issue is with how Scala implements case classes. Fields are always stored as private fields with getters (and setters if you use var), and the annotations are propagated to the constructor parameters.

My way of fixing it was the following:

Do not propagate annotations to the constructor
Create another constructor that has a different signature that the default one
Mark the constructor as @JsonCreator

This way, Jackson creates the object with default values using the new constructor, and then sets the fields with reflection. So, this is how the code should look like:

case class Base
(
	@(JacksonXmlElementWrapper @field @getter)(localName = "People")
	@(JacksonXmlProperty @field @getter)(localName = "Person")
	val People: List[Person]
) {
   @JsonCreator
   def this(
	v1: List[Person]
	ignored: String
   ) = {
    this(v1)
   }
}

You could go with some different constructor if needed (like a parameterless one and pass nulls explicitly).

This works with Jackson 2.13.0.

Bit Twiddling Part 4 — Disabling CTRL+ALT+HOME in mstsc.exe (Windows RDP client)

afish — Mon, 20 May 2024 07:40:05 +0000

This is the fourth part of the Bit Twiddling series. For your convenience you can find other parts in the table of contents in Par 1 — Modifying Android application on a binary level

Today we’re going to disable CTRL+ALT+HOME shortcut in mstsc. We want to disable the shortcut so it doesn’t “unfocus” the RDP session but still works inside the connection. I needed that for easier work in nested RDPs and I couldn’t find any decent RDP client for Windows that wouldn’t have this shortcut. The only one I found was Thincast but it lacked some other features as well.

Let’s begin.

Finding the entry point

The hardest part is as always finding “the place”. I took API Monitor which is like strace for Windows. I started the mstsc.exe, connected to some machine, pressed CTRL+ALT+HOME and observed what happened. After some digging here and there I found that when I press CTRL+ALT+HOME, the WM_USER+19 message is sent inside the application:

This is clearly a hint what’s going on. We can see that the key combination is captured by mstscax.dll which is the ActiveX part of the RDP. We also found the call stack and the address 0x000007ffec2d82fe2.

Analyzing the code

Now we can use WinDBG or whatever other debugger to figure out what’s going on. I attached to the running mstsc.exe, added a breakpoint with bu 0x00007ffec2d82fe2 and then pressed CTLR+ALT+HOME. As expected, the breakpoint was hit and I could observe the call stack:

0:003> kb
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ffe`c2d27655     : 00000000`00000000 00000000`00000000 00000000`00000024 00000000`00008000 : mstscax!CTSCoreEventSource::InternalFireAsyncNotification+0xca
01 00007ffe`c2d262a5     : 00000000`00000100 00000000`00000000 00000000`00000024 00000135`bc82ea28 : mstscax!CTSInput::IHPostMessageToMainWindow+0x1c5
02 00007ffe`c2d261c8     : 00000000`00000001 00000000`01470001 00000000`003f0b28 00000000`00000113 : mstscax!CTSInput::IHInputCaptureWndProc+0x85
03 00007fff`5a04ef75     : 00000000`00000001 00000033`4faff2d0 00000000`003f0b28 00000000`80000022 : mstscax!CTSInput::IHStaticInputCaptureWndProc+0x58
04 00007fff`5a04e69d     : 00000000`00000000 00007ffe`c2d26170 00000033`4f5d3800 00007fff`3f646aae : USER32!UserCallWinProcCheckWow+0x515
05 00007fff`3f64ab32     : 00007ffe`c2d26170 00000000`00000000 00000000`ffffffff 00007fff`3f646a35 : USER32!DispatchMessageWorker+0x49d
06 00007fff`3f643997     : 00000000`00000000 00000000`00000000 00000135`bc830b00 00007fff`00000000 : apimonitor_drv_x64+0xab32
07 00000135`bc8fb89f     : 00000135`bc25da6a 00000000`00000001 00000000`00000001 00000000`00000001 : apimonitor_drv_x64+0x3997
08 00000135`bc25da6a     : 00000000`00000001 00000000`00000001 00000000`00000001 00000135`bd33e0e0 : 0x00000135`bc8fb89f
09 00000000`00000001     : 00000000`00000001 00000000`00000001 00000135`bd33e0e0 00007ffe`c2cd761c : 0x00000135`bc25da6a
0a 00000000`00000001     : 00000000`00000001 00000135`bd33e0e0 00007ffe`c2cd761c 00000033`4faff480 : 0x1
0b 00000000`00000001     : 00000135`bd33e0e0 00007ffe`c2cd761c 00000033`4faff480 00000135`bba40000 : 0x1
0c 00000135`bd33e0e0     : 00007ffe`c2cd761c 00000033`4faff480 00000135`bba40000 00000000`00000083 : 0x1
0d 00007ffe`c2cd761c     : 00000033`4faff480 00000135`bba40000 00000000`00000083 00000000`7ffef000 : 0x00000135`bd33e0e0
0e 00007ffe`c2cd7444     : 00000000`00000000 4e478f48`d4f63727 00000000`000051c0 0000aa52`0fad086d : mstscax!PAL_System_CondWait+0x1cc
0f 00007ffe`c2cf0f75     : 00000000`00000400 00000000`00000000 00000033`4faff8c0 00000135`bd35a720 : mstscax!CTSThreadInternal::ThreadSignalWait+0x34
10 00007ffe`c2cf1f8d     : 00000000`00000000 00000000`00000000 00000135`bd35a720 00000000`00000400 : mstscax!CTSThread::internalMsgPump+0x6d
11 00007ffe`c2d8693c     : 00000000`00000000 00007ffe`c2cee22d 00000135`bd344640 00007ffe`c2fdf3f0 : mstscax!CTSThread::internalThreadMsgLoop+0x14d
12 00007ffe`c3128550     : 00007ffe`c3475808 00000033`4faff8c0 00000000`00000000 00000135`ba29b8c0 : mstscax!CTSThread::ThreadMsgLoop+0x1c
13 00007ffe`c2fdeca8     : 00000135`bd35a720 00000135`ba29b8c0 00000135`bd35a720 00000135`ba29b748 : mstscax!CSND::SND_Main+0x148
14 00007ffe`c2fe73c2     : 00000135`bd35d100 00000135`bd35a720 00000033`4f67e3f0 00000000`00000000 : mstscax!CTSThread::TSStaticThreadEntry+0x258
15 00007fff`5a1f7344     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : mstscax!PAL_System_Win32_ThreadProcWrapper+0x32
16 00007fff`5b7a26b1     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x14
17 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

Perfect! We can see the method names and they are really useful. We can immediately see that some IHStaticInputCaptureWndProc method captures the regular message from the OS, then it calls the window procedure IHInputCaptureWndProc, and then the method posts the message to the main mstsc.exe window in IHPostMessageToMainWindow.

This shows us the way. We can also ask the WinDBG to get the call stack with addresses to find out that mstscax!CTSInput::IHInputCaptureWndProc+0x85 is mstscax+0x562a5. We can now dump the code:

0:004> u mstscax+0x562a5-0x85 mstscax+0x562a5
mstscax+0x56220:
00007ffe`c2d26220 4053            push    rbx
00007ffe`c2d26222 55              push    rbp
00007ffe`c2d26223 56              push    rsi
00007ffe`c2d26224 57              push    rdi
00007ffe`c2d26225 4154            push    r12
00007ffe`c2d26227 4155            push    r13
00007ffe`c2d26229 4156            push    r14
00007ffe`c2d2622b 4157            push    r15
00007ffe`c2d2622d 4881ece8000000  sub     rsp,0E8h
00007ffe`c2d26234 488b05cdc37500  mov     rax,qword ptr [mstscax!DllUnregisterServer+0x6f1308 (00007ffe`c3482608)]
00007ffe`c2d2623b 4833c4          xor     rax,rsp
00007ffe`c2d2623e 48898424d0000000 mov     qword ptr [rsp+0D0h],rax
00007ffe`c2d26246 33ed            xor     ebp,ebp
00007ffe`c2d26248 0f57c0          xorps   xmm0,xmm0
00007ffe`c2d2624b 448bf5          mov     r14d,ebp
00007ffe`c2d2624e 896c245c        mov     dword ptr [rsp+5Ch],ebp
00007ffe`c2d26252 f30f7f442470    movdqu  xmmword ptr [rsp+70h],xmm0
00007ffe`c2d26258 498bf1          mov     rsi,r9
00007ffe`c2d2625b 418bd8          mov     ebx,r8d
00007ffe`c2d2625e 4c8be2          mov     r12,rdx
00007ffe`c2d26261 488bf9          mov     rdi,rcx
00007ffe`c2d26264 488b059df57400  mov     rax,qword ptr [mstscax!DllUnregisterServer+0x6e4508 (00007ffe`c3475808)]
00007ffe`c2d2626b 4c8d2d96f57400  lea     r13,[mstscax!DllUnregisterServer+0x6e4508 (00007ffe`c3475808)]
00007ffe`c2d26272 4c8bbc2450010000 mov     r15,qword ptr [rsp+150h]
00007ffe`c2d2627a 493bc5          cmp     rax,r13
00007ffe`c2d2627d 740a            je      mstscax+0x56289 (00007ffe`c2d26289)
00007ffe`c2d2627f f6401c01        test    byte ptr [rax+1Ch],1
00007ffe`c2d26283 0f85c6010000    jne     mstscax+0x5644f (00007ffe`c2d2644f)
00007ffe`c2d26289 39af88030000    cmp     dword ptr [rdi+388h],ebp
00007ffe`c2d2628f 0f85f1020000    jne     mstscax+0x56586 (00007ffe`c2d26586)
00007ffe`c2d26295 4d8bcf          mov     r9,r15
00007ffe`c2d26298 4c8bc6          mov     r8,rsi
00007ffe`c2d2629b 8bd3            mov     edx,ebx
00007ffe`c2d2629d 488bcf          mov     rcx,rdi
00007ffe`c2d262a0 e8eb110000      call    mstscax+0x57490 (00007ffe`c2d27490)
00007ffe`c2d262a5 85c0            test    eax,eax

Great. We can see that the line 00007ffe`c2d262a0 calls some internal method and then tests if the output of the call is equal to 0 (this is the meaning of test eax,eax). We can now comment out this line and see what happens (mind the empty line):

a 00007ffe`c2d262a0
xor eax,eax
nop
nop
nop

g

We come back to mstsc.exe and we can see that CTRL+ALT+HOME doesn’t unfocus the window anymore! We can also check that the combination is handled properly inside the connection so it still unfocuses nested RDPs.

Automation

Automating this is pretty straightforward with cdb. We could modify the dll in place, but that would affect all the RDP connections we ever make. If we want to disable CTRL+ALT+HOME for some of them only, then this is what we can do:

First, create the file mstsc.txt with the following code:

a mstscax+0x562a0
xor eax, eax
nop
nop
nop

qd

Next, create batch file no_home.bat

cdb.exe -p %1 -cf mstsc.txt

Finally, run it like this:

no_home.bat PROCESS_ID_OF_MSTSC_YOU_WANT_TO_HACK

Changing shortcut to something else

How about we change shortcut instead of getting rid of it entirely? Let’s disassemble the method a bit more:

0:001> u mstscax+0x57655-0x1c5 mstscax+0x57655
mstscax!CTSInput::IHPostMessageToMainWindow:
00007ffe`c2d27490 48895c2408      mov     qword ptr [rsp+8],rbx
00007ffe`c2d27495 48896c2410      mov     qword ptr [rsp+10h],rbp
00007ffe`c2d2749a 4889742418      mov     qword ptr [rsp+18h],rsi
00007ffe`c2d2749f 57              push    rdi
00007ffe`c2d274a0 4156            push    r14
00007ffe`c2d274a2 4157            push    r15
00007ffe`c2d274a4 4883ec30        sub     rsp,30h
00007ffe`c2d274a8 33db            xor     ebx,ebx
00007ffe`c2d274aa 4d8bf1          mov     r14,r9
00007ffe`c2d274ad 498be8          mov     rbp,r8
00007ffe`c2d274b0 8bf2            mov     esi,edx
00007ffe`c2d274b2 488bf9          mov     rdi,rcx
00007ffe`c2d274b5 399958030000    cmp     dword ptr [rcx+358h],ebx
00007ffe`c2d274bb 0f8582000000    jne     mstscax!CTSInput::IHPostMessageToMainWindow+0xb3 (00007ffe`c2d27543)
00007ffe`c2d274c1 81fa00010000    cmp     edx,100h
00007ffe`c2d274c7 0f8489000000    je      mstscax!CTSInput::IHPostMessageToMainWindow+0xc6 (00007ffe`c2d27556)
00007ffe`c2d274cd b901000000      mov     ecx,1
00007ffe`c2d274d2 8d82fcfeffff    lea     eax,[rdx-104h]
00007ffe`c2d274d8 3bc1            cmp     eax,ecx
00007ffe`c2d274da 0f47cb          cmova   ecx,ebx
00007ffe`c2d274dd 399f24030000    cmp     dword ptr [rdi+324h],ebx
00007ffe`c2d274e3 0f84c3040000    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x51c (00007ffe`c2d279ac)
00007ffe`c2d274e9 448bc3          mov     r8d,ebx
00007ffe`c2d274ec 81fe04010000    cmp     esi,104h
00007ffe`c2d274f2 0f84c7040000    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x52f (00007ffe`c2d279bf)
00007ffe`c2d274f8 8bd3            mov     edx,ebx
00007ffe`c2d274fa 85c9            test    ecx,ecx
00007ffe`c2d274fc 0f85d1040000    jne     mstscax!CTSInput::IHPostMessageToMainWindow+0x543 (00007ffe`c2d279d3)
00007ffe`c2d27502 8bc3            mov     eax,ebx
00007ffe`c2d27504 85d2            test    edx,edx
00007ffe`c2d27506 0f8558010000    jne     mstscax!CTSInput::IHPostMessageToMainWindow+0x1d4 (00007ffe`c2d27664)
00007ffe`c2d2750c 85c0            test    eax,eax
00007ffe`c2d2750e 0f8550010000    jne     mstscax!CTSInput::IHPostMessageToMainWindow+0x1d4 (00007ffe`c2d27664)
00007ffe`c2d27514 8bc3            mov     eax,ebx
00007ffe`c2d27516 4585c0          test    r8d,r8d
00007ffe`c2d27519 0f85c8040000    jne     mstscax!CTSInput::IHPostMessageToMainWindow+0x557 (00007ffe`c2d279e7)
00007ffe`c2d2751f 85c0            test    eax,eax
00007ffe`c2d27521 0f85c0040000    jne     mstscax!CTSInput::IHPostMessageToMainWindow+0x557 (00007ffe`c2d279e7)
00007ffe`c2d27527 488b6c2458      mov     rbp,qword ptr [rsp+58h]
00007ffe`c2d2752c 8bc3            mov     eax,ebx
00007ffe`c2d2752e 488b5c2450      mov     rbx,qword ptr [rsp+50h]
00007ffe`c2d27533 488b742460      mov     rsi,qword ptr [rsp+60h]
00007ffe`c2d27538 4883c430        add     rsp,30h
00007ffe`c2d2753c 415f            pop     r15
00007ffe`c2d2753e 415e            pop     r14
00007ffe`c2d27540 5f              pop     rdi
00007ffe`c2d27541 c3              ret
00007ffe`c2d27542 cc              int     3
00007ffe`c2d27543 81fe13010000    cmp     esi,113h
00007ffe`c2d27549 0f851f010000    jne     mstscax!CTSInput::IHPostMessageToMainWindow+0x1de (00007ffe`c2d2766e)
00007ffe`c2d2754f bb01000000      mov     ebx,1
00007ffe`c2d27554 ebd1            jmp     mstscax!CTSInput::IHPostMessageToMainWindow+0x97 (00007ffe`c2d27527)
00007ffe`c2d27556 8b8128030000    mov     eax,dword ptr [rcx+328h]
00007ffe`c2d2755c be00800000      mov     esi,8000h
00007ffe`c2d27561 483be8          cmp     rbp,rax
00007ffe`c2d27564 0f8470010000    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x24a (00007ffe`c2d276da)
00007ffe`c2d2756a 8b8744030000    mov     eax,dword ptr [rdi+344h]
00007ffe`c2d27570 483be8          cmp     rbp,rax
00007ffe`c2d27573 0f84dc010000    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x2c5 (00007ffe`c2d27755)
00007ffe`c2d27579 8b8748030000    mov     eax,dword ptr [rdi+348h]
00007ffe`c2d2757f 483be8          cmp     rbp,rax
00007ffe`c2d27582 0f8440020000    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x338 (00007ffe`c2d277c8)
00007ffe`c2d27588 8b874c030000    mov     eax,dword ptr [rdi+34Ch]
00007ffe`c2d2758e 483be8          cmp     rbp,rax
00007ffe`c2d27591 0f84a4020000    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x3ab (00007ffe`c2d2783b)
00007ffe`c2d27597 8b8750030000    mov     eax,dword ptr [rdi+350h]
00007ffe`c2d2759d 483be8          cmp     rbp,rax
00007ffe`c2d275a0 0f8408030000    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x41e (00007ffe`c2d278ae)
00007ffe`c2d275a6 4883fd24        cmp     rbp,24h
00007ffe`c2d275aa 0f8471030000    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x491 (00007ffe`c2d27921)
00007ffe`c2d275b0 4883fd2d        cmp     rbp,2Dh
00007ffe`c2d275b4 0f856dffffff    jne     mstscax!CTSInput::IHPostMessageToMainWindow+0x97 (00007ffe`c2d27527)
00007ffe`c2d275ba 8d4de5          lea     ecx,[rbp-1Bh]
00007ffe`c2d275bd 48ff15acec5b00  call    qword ptr [mstscax!_imp_GetKeyState (00007ffe`c32e6270)]
00007ffe`c2d275c4 0f1f440000      nop     dword ptr [rax+rax]
00007ffe`c2d275c9 6685c6          test    si,ax
00007ffe`c2d275cc 0f8455ffffff    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x97 (00007ffe`c2d27527)
00007ffe`c2d275d2 8d4de4          lea     ecx,[rbp-1Ch]
00007ffe`c2d275d5 48ff1594ec5b00  call    qword ptr [mstscax!_imp_GetKeyState (00007ffe`c32e6270)]
00007ffe`c2d275dc 0f1f440000      nop     dword ptr [rax+rax]
00007ffe`c2d275e1 6685c6          test    si,ax
00007ffe`c2d275e4 0f843dffffff    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x97 (00007ffe`c2d27527)
00007ffe`c2d275ea e8c581fcff      call    mstscax!CClientUtilsWin32::UT_IsRunningInAppContainer (00007ffe`c2cef7b4)
00007ffe`c2d275ef 85c0            test    eax,eax
00007ffe`c2d275f1 0f8430ffffff    je      mstscax!CTSInput::IHPostMessageToMainWindow+0x97 (00007ffe`c2d27527)
00007ffe`c2d275f7 488b050ae27400  mov     rax,qword ptr [mstscax!WPP_GLOBAL_Control (00007ffe`c3475808)]
00007ffe`c2d275fe 4c8d3d03e27400  lea     r15,[mstscax!WPP_GLOBAL_Control (00007ffe`c3475808)]
00007ffe`c2d27605 493bc7          cmp     rax,r15
00007ffe`c2d27608 7430            je      mstscax!CTSInput::IHPostMessageToMainWindow+0x1aa (00007ffe`c2d2763a)
00007ffe`c2d2760a f6401c01        test    byte ptr [rax+1Ch],1
00007ffe`c2d2760e 742a            je      mstscax!CTSInput::IHPostMessageToMainWindow+0x1aa (00007ffe`c2d2763a)
00007ffe`c2d27610 80781904        cmp     byte ptr [rax+19h],4
00007ffe`c2d27614 7224            jb      mstscax!CTSInput::IHPostMessageToMainWindow+0x1aa (00007ffe`c2d2763a)
00007ffe`c2d27616 e82de60000      call    mstscax!RdpWppGetCurrentThreadActivityIdPrefix (00007ffe`c2d35c48)
00007ffe`c2d2761b 488b0de6e17400  mov     rcx,qword ptr [mstscax!WPP_GLOBAL_Control (00007ffe`c3475808)]
00007ffe`c2d27622 4c8d0557ca5c00  lea     r8,[mstscax!WPP_f5f71bb7bac236b27f26969128cc1e12_Traceguids (00007ffe`c32f4080)]
00007ffe`c2d27629 448bc8          mov     r9d,eax
00007ffe`c2d2762c bafd000000      mov     edx,0FDh
00007ffe`c2d27631 488b4910        mov     rcx,qword ptr [rcx+10h]
00007ffe`c2d27635 e85ecdfcff      call    mstscax!WPP_SF_D (00007ffe`c2cf4398)
00007ffe`c2d2763a 488b8f60050000  mov     rcx,qword ptr [rdi+560h]
00007ffe`c2d27641 4533c0          xor     r8d,r8d
00007ffe`c2d27644 488b01          mov     rax,qword ptr [rcx]
00007ffe`c2d27647 418d5010        lea     edx,[r8+10h]
00007ffe`c2d2764b 488b4048        mov     rax,qword ptr [rax+48h]
00007ffe`c2d2764f ff1523f75b00    call    qword ptr [mstscax!_guard_dispatch_icall_fptr (00007ffe`c32e6d78)]
00007ffe`c2d27655 c7871404000001000000 mov dword ptr [rdi+414h],1

The most interesting line is:

00007ffe`c2d275a6 4883fd24        cmp     rbp,24h

0x24 is the code key of HOME. We can replace it with something else, like insert which is 2D:

e mstscax+0x575A9 0x2D

And there you go. You can now press CTRL+ALT+HOME to unfocus nested RDP and CTRL+ALT+INSERT to unfocus the outer one. This gives you two levels of unfocusing.

Availability Anywhere Part 25 — Supercharge your VR experience

afish — Fri, 22 Mar 2024 20:15:30 +0000

This is the twentieth fifth part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows

Today we’re going to make our VR experience even better. This is continuation of the previous part.

As mentioned before, there are issues with ALT+TAB and WINDOWS key. I remapped CAPS LOCK to WINDOWS with PowerToys, but today we’re doing something better.

First, I was looking for a good full-size keyboard with touchpad that I could use with VR. The best thing I found is Perixx PERIBOARD-313. Touchpad is in some weird position but I got used to that. It has this irritating Fn key, lacks Right WINDOWS, and the space is quite short, but generally the keyboard is good enough.

However, it’s a wired keyboard. You can plug that into your VR headset and it works well. The keyboard has additional USB slots, so you can plug more devices to your VR this way. Still, we’d like to make it wireless. To do that, I’m using Bluetooth Adapter for Keyboard & Mouse (BT-500). This adapter has a couple of interesting features. First, it turns your wired keyboard or mouse into a wireless one. Second, it can remap keys on the fly, add macros, or have custom settings that you can switch between easily.

I’m using the adapter to remap Left WINDOWS to CAPS LOCK, and TAB to SCROLL LOCK. This is my setup:

map add l_com caps_lock
map add tab scroll_lock
save
reboot

Next, I remap CAPS LOCK to Left WINDOWS back using PowerToys. I do the same for SCROLL LOCK and TAB.

Finally, I use another adapter to turn any wired mouse into wireless one. You could go with Bluetooth mouse instead. With these things in place, I now have a “regular” wireless keyboard that I can use with VR. Even though it’s Android behind the scenes, all is just like a regular Windows laptop.

As a side note, I tried remapping keys to F13 and similar special keys. Unfortunately, Quest 3 doesn’t handle these properly and my remote machine I connect to over vSpatial doesn’t receive the keys.

Availability Anywhere Part 24 — Make RDP retain position of windows and stop moving them around

afish — Sat, 09 Mar 2024 08:53:42 +0000

This is the twentieth fourth part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows

Today we’re going to solve a problem of RDP session moving your windows around when you connect from multiple machines. Let’s see how it works.

Problem statement

First, a couple of assumptions what I’m trying to solve:

I’m using mstsc.exe to connect to the RDP server. mstsc.exe is the regular Windows RDP client (so called the “blue” client). I’m using the regular application, not the UWP one
I have two machines that I use to connect to the RDP server. These machines are Client1 and Client2. Both Clients have three screens attached. While I’m using physical screens, you’re free to use IddSampleDriver or vSpatial virtual screens. I think the same should apply to other solutions like spacedesk and similar. The same should apply to screens created by BetterDisplay (when you RDP from Mac to your Client1).
Screen1 is “at front”, Screen2 is “on the right”, Screen3 is “on the left”. The actual ordering doesn’t matter but I’ll refer to this later in this post
I use Client1 to connect to the RDP server and I open maximized Browser on Screen1 and maximized Notepad on Screen2. I then disconnect and connect to the RDP server from Client2. At this point, I want Browser to still be on Screen1 (“at front”) and Notepad on Screen2 (“on the right”).
Screen1 is my “main display” and it has the taskbar. I want the same in the RDP server (so taskbar should be on Screen1)

I’m going to use screenshots to explain my setup. They are a little trickier to let’s walk one by one.

My setup

First, my Client1 has five physical screens in total. However, I use Screen4 to duplicate my Screen1, and I use Screen5 to keep things outside of RDP. Therefore, I only want to use Screen1, Screen2, and Screen3 in the RDP session (and obviously Screen4 which is implicitly showing Screen1). I also have 5 additional virtual screens created with IddSampleDriver and 2 more virtual screens created with vSpatial. I don’t use these virtual screens (they are turned off).

This is what “Display” shows me which from now on I’ll call “Display Setup”:

So this is the mapping:

Screen1 (“at front”) is Display Setup Screen 1
Screen2 (“on the right”) is Display Setup Screen 12
Screen3 (“on the left”) is Display Setup Screen 11
Screen4 (duplicating Screen1) is Display Setup Screen 9
Screen5 (kept outside of RDP) is Display Setup Screen 10
Display Setup Screen 2-6 are IddSampleDriver virtual screens
Display Setup Screen 7-8 are vSpatial virtual screens

Now I want to use screens Screen1 + Screen2 + Screen3 in RDP session only. Scott Hanselman explains how to do it. We first need to list the screens with mstsc /l and this is what I get (which from now I’ll call “MSTSC Setup”):

So here comes the mapping:

Screen1 (“at front”) is MSTSC Setup Screen 44
Screen2 (“on the right”) is MSTSC Setup Screen 46
Screen3 (“on the left”) is MSTSC Setup Screen 47
Screen4 (duplicating Screen1) is MSTSC Setup Screen 44 (the same as Screen1)
Screen5 (kept outside of RDP) is MSTSC Setup Screen 45

It’s important to notice that numbers do not match between Display Setup and MSTSC Setup. This is very important and makes the whole trouble. Let’s now solve the problems.

Using only a subset of displays in the RDP session

This is actually easy. When you see Scott Hanselman’s blog, it mentions the parameter called selectedmonitors. You use it to specify the comma-separated-list of monitors that you want to use for the RDP session.

In my case, I need to use MSTSC Setup Screen 44, MSTSC Setup Screen 46, and MSTSC Setup Screen 47. So the parameter in my case is:

selectedmonitors:s:44,46,47

After connecting to RDP, this is what “Display” shows (which I’ll refer to as Display RDP Setup):

In textual form each line is screenId: width x height; (xBegin, yBegin, xEnd, yEnd):

44: 1920 x 1080; (0, 0, 1919, 1079)
45: 1080 x 1920; (363, 1080, 1442, 2999)
46: 2560 x 1440; (1920, 0, 4479, 1439)
47: 2560 x 1440; (-2560, 0, -1, 1439)

So, here is the mapping:

Screen1 (“at front”) is Display RDP Setup Screen 1
Screen2 (“on the right”) is Display RDP Setup Screen 2
Screen3 (“on the left”) is Display RDP Setup Screen 3

You can see that there are only 3 screens in the RDP session. Exactly as I wanted.

Controlling the main display in RDP

selectedmonitors parameter controls this. The main display is the first monitor from this list. So, if you have selectedmonitors:s:x,y,z, then the screen x is going to be the main display in RDP. This doesn’t need to match your main display on the host. In my case, my parameter is set to selectedmonitors:s:44,46,47, so the main display in RDP is going to be MSTSC Setup Screen 44.

Making RDP not move windows

This is the trickiest part. Remember that I open maximized Browser on Screen1 and maximized Notepad on Screen2. Based on our mappings above, the Browser is on Screen1 = Display Setup Screen 1 = Display RDP Setup Screen 1 = MSTSC Setup Screen 44. Similarly, Notepad is on Screen2 = Display Setup Screen 12 = Display RDP Setup Screen 2 = MSTSC Setup Screen 46.

Now, these are the things that I believe are correct. I found them empirically and they work for me across 4 different laptops, but your mileage my vary.

The order of screens in selectedmonitors parameter doesn’t matter (apart from choosing the “main display” explained in the previous section)
The numbers from Display Setup do not matter! The numbers from MSTSC Setup do not matter! The only thing that matters is the order of screens from MSTSC Setup
The resolutions of the screens do not matter! Windows remembers maximized windows positions not based on the geometry (like “the window’s top left corner is in pixel 0,0) but rather as “the window is maximized on Display RDP Setup Screen 1“
You can’t control numbers from MSTSC Setup programmatically. You can only control them by physically moving your screens to different ports/connectors/docking stations/USB adapters, etc.
To get the windows not move, you need to have screen “passed to RDP” in the same order between Client1 and Client2 (between different clients)

That’s it. It may be a little bit cryptic, so let’s delve into how I think it works.

mstsc.exe uses some API to enumerate displays and then applies logic to merge windows or reorder them. Notice, that in my case Display Setup shows 12 screens while MSTSC Setup shows only 4. That’s because 7 virtual screens are turned off, and Screen4 duplicates Screen1.

Now, it seems that mstsc.exe enumerates the screens, and then for each screen checks if it was selected in selectedmonitors. If yes, then it registers the screen in the RDP session. So the code looks conceptually like:

foreach(screen in enumerateScreens()):
    if(screen.id in selectedmonitors) then
        registerScreenInRdpSession(screen)
    end
end

This explains why the order of screens in selectedmonitors doesn’t matter. Assuming that, this is what we need to get:

We want the Browser to still be presented on the Screen1 (“at front”). This screen was registered first in the loop above when connecting from Client1. Therefore, when Client2 enumerates the screens in mstsc.exe, Screen1 must be the first one that matches the if condition. In other words, Screen1 can be in whatever position according to MSTSC Setup and can have any number, but there must be no other selectedmonitors screen earlier in the list of mstsc.exe /l
We don’t care about resolutions. Screen1 on Client1 can have different resolution than Screen1 on Client2. The Browser will still be “maximized on the Display RDP Setup Screen 1
We want the Notepad to still be presented on the Screen2 (“on the right”). Same logic applies: mstsc.exe continues enumerating the screens and the second screen that gets picked must be the screen on the right

Now, how do we control the order of screens enumerated by mstsc.exe /l? The short answer is: we can’t do that programmatically. We need to physically change cables! What’s worse, this order breaks when you change settings like Duplicate screen or Extend desktop to this screen.

How to change cables physically? There are solutions:

Just plug cable to some other slot in your laptop/PC
Use docking station and change the order of cables (or move some monitors to your laptop and some through your docking station)
Use another docking station! Yes, you can have many of them (I actually have 2, that’s a long story)
Use USB adapters like i-tec USB Type C to DisplayPort Adapter, 4K Video, Supports 2 External Displays. I have two of them and they are great. Your mileage may vary, obviously
Try hacking mstsc.exe with things like API Monitor (I haven’t tried that but that seems plausible)

I tested this setup across four different laptops – one with physical screens, two with virtual screens by IddSampleDriver, one with virtual screens by vSpatial. In my case, windows are not moved anymore. I can literally switch from one computer to another with different screen resolutions but all the windows stay where they were. And yes, I had to switch my cables like crazy. In my case it was even worse, because Screen1 was attached directly (and so was enumerated first by Display Setup) but Screen4 was attached by an adapter (and was taking precedence in mstsc.exe /l). It works, though.

Availability Anywhere Part 23 — RDP over VR goggles with no PCVR

afish — Wed, 21 Feb 2024 21:10:54 +0000

This is the twentieth third part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows

In this part we’re going to see how we can work remotely from VR goggles with no PCVR around. Effectively, we can work remotely with goggles and a bluetooth keyboard, nothing else.

My workstation

My home workstation is quite powerful. I have 5 physical screens which I use to display 4 different screens. I also use VirtuaWin to have many desktops. I also work remotely from a dedicated machine to which I connect using MS RDP (mstsc.exe).

My typical setup works like this:

My main physical screen is 24” with FullHD resolution. I duplicate this physical screen to another physical screen which is 10” and has touch capabilities. I use the touch to draw diagrams.
My two additional side screens are 32” with 4K resolution
My fifth screen is 10” big and I use it to display all the instant messengers I need (slacks, discords, meets etc). That’s basically a browser window that I configure to be visible on all of the desktops.
I have 11 desktops configured in VirtuaWin.
To connect remotely to my machine, I move to some other VirtuaWin desktop and use four physical screens (FullHD + 2x4K + touch-enabled screen) to open mstsc.exe.
When I need to connect to some other machine, I basically do CTRL+ALT+HOME to unfocus the current RDP, then I switch to another VirtuaWin desktop, and there I connect to the server. This way I can quickly switch between multiple machines I have

This setup works great for me. I can work remotely from any place around the planet. When I’m not at home, I can use my road runner laptop with two physical screens or my mobile phone to connect over RDP to the remote server. I do various things over RDP – I code, I write/read documents, I postprocess videos, I record videos (yes, I record them over RDP!), I browse the Internet. Typical office work.

I wanted to replicate this setup as much as possible with VR goggles. I managed to do that with Meta Quest 3 and vSpatial. Let’s see how I did that.

What we need

First, you need the goggles. I used Meta Quest 3 and they were quite good. I can’t complain on the image quality and I definitely can do all the stuff I need. The battery lasts for around 90-120 minutes which is okayish.

Next, you need to have a machine with many physical displays. That’s the most tricky part if you want to configure it entirely over RDP and use free version of vSpatial. I use two dedicated machines: one is Windows Server and the other is Mac. I first connect to Windows Server from my mobile phone over the regular RDP and log in as User1. At this point, I have a session in Windows Server that has just one display. Next, I use NoMachine to connect from Windows Server to my Mac. In my Mac, I create multiple physical screens using Better Display. Next, I connect from my Mac to my Windows Server and log in as User2 using the regular RDP application. At this point, I have a session in Windows Server that has multiple physical screens. All of that done entirely from my mobile phone, so I can do that when I’m travelling. While it works and allows for any setup of monitors, it’s probably good enough to just use 4 screens from paid vSpatial. Up to you.

Next, we need the vSpatial server. I install it in Windows Server for User2. I then log in to Windows Server using vSpatial application in Meta Quest 3 goggles. The vSpatial client correctly recognizes all 4 displays and now I have access to a regular Windows machine. I can install VirtuaWin and configure desktops the regular way. It all works just like on the screenshot below:

You can see one screen at the top and three screens in the middle. This is nearly the same setup as I have at home.

Finally, I just pair my bluetooth keyboard with touchpad and that’s it. CTRL+ALT+HOME works, so I can work the same way as at home.

Now, sky is the limit. Since I managed to RDP into a regular Windows machine, I can RDP anywhere I need, I can install any application I need, and I can do all of that using my goggles and the bluetooth keyboard only. I want to stress that there is no PCVR here. The Windows Machine with vSpatial server is completely virtualized and has no dedicated graphics card. Also, I don’t use powerful Internet connection. I just connect over regular WiFi. This works from mobile hotspot as well as it’s just RDP.

When it comes to online meetings, I can take them from my mobile phone. If I want to share the screen, I can just join the same meeting from my mobile phone and from my remote machine, and I can share the screen from Windows Server. However, you can also start browser on VR and join meeting like in Google Meet. The meeting will work even after you switch to vSpatial.

Quirks

There are quirks, obviously.

First, Meta Quest 3 seems to be based on Android. Bluetooth keyboards don’t work 100% properly with Android when you RDP to Windows. One issue is that the WIN key doesn’t work. Another, some shortcuts (like ALT+TAB) are captured by the host (goggles in this case).

To solve the lack of WIN key, I use PowerToys. I just remap CAPS LOCK to LEFT WIN on my Windows Server. You can also use AutoHotkey with the following script:

*CapsLock::
    Send {Blind}{LWin Down}
Return

*CapsLock up::
    Send {Blind}{LWin Up}
Return

The script rebinds CAPS LOCK to LEFT WIN. No matter which method you use, you can just use CAPS LOCK in all the shortcuts (like maximizing the window with LEFT WIN+ARROW UP which becomes CAPS LOCK+ARROW UP). While it’s a bit painful, it works. You need to install AutoHotkey in all the remote machines you connect to.

I didn’t solve the ALT+TAB issue with AutoHotkey. I solved it with PowerToys by remapping SCROLL LOCK to TAB. I think you could use AutoHotkey to remap it as well.

Second, vSpatial has a concept of active (focused) desktop. If you worked on one screen and want to click on another one, the mouse click changes the focus. You need to click again to do the actual click you want. This is not a big deal and I got used to it very fast.

Third, I think vSpatial is slightly slower than the regular RDP. I don’t find this as a problem, though, as it’s fast enough for me.

Fourth, the goggles don’t last for long (up to two hours). On the other hand, maybe it’s better for your eyes. You can also buy an additional strap with built-it powerbank.

Summary

It took me quite a while to figure out how to configure this setup. However, now I can truly work using goggles and no laptop around. That sounds like a great approach when traveling – imagine sitting at the airport and having multiple screens, just like at home. Now, the only issue is around the goggles. I hope one day they will be smaller, lighter, and will have stronger batteries. You can always plug them into powerbank if needed, though.

Availability Anywhere Part 22 — Customer Experience Improvement Program restarts

afish — Sun, 11 Feb 2024 10:10:10 +0000

This is the twentieth second part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows

I was recently investigating a case of a computer restarting every Sunday at 3AM UTC. I couldn’t figure out what was going on and I suspected Customer Experience Improvement Program as I found the following event in event viewer:

User Logoff Notification for Customer Experience Improvement Program

- 
- 
		 
		1074 
		0 
		4 
		0 
		0 
		0x8080000000000000 
		 
		404828 
		 
		 
		System 
		computerName 
		 
	
- 
		C:\Windows\system32\shutdown.EXE (computerName) 
		computerName 
		No title for this reason could be found 
		0x800000ff 
		restart 
		System is scheduled to reboot in 10 minutes. Please save your work. 
		NT AUTHORITY\SYSTEM

I checked many solutions on the Internet and none of them helped. However, later I realized it was not due to Customer Experience Improvement Program. After carefully checking the event viewer, I found the following:

The process C:\Windows\system32\shutdown.EXE (computerName) has initiated the restart of computer computerName on behalf of user NT AUTHORITY\SYSTEM for the following reason: No title for this reason could be found
Reason Code: 0x800000ff
Shutdown Type: restart
Comment: System is scheduled to reboot in 10 minutes. Please save your work.

- 
- 
		 
		7002 
		0 
		4 
		1102 
		0 
		0x2000200000000000 
		 
		404868 
		 
		 
		System 
		computerName 
		 
	
- 
		2 
		S-1-5-21-1801674531-515967899-839522115-19733726

So it seems that something triggered the restart. I wasn’t sure what that was and I just solved it by cancelling the restart. Just run this batch script to cancel any shutdowns every 30 seconds:

:start
shutdown /a
timeout 30
goto start

It worked good enough.