Availability Anywhere Part 27 — Tools for remote work from laptops and XR/VR/AR

~ afish

This is the twentieth seventh part of the Availability Anywhere series. For your convenience you can find other parts in the table of contents in Part 1 – Connecting to SSH tunnel automatically in Windows

Last time we discussed how to work remotely. Let’s now look at that from a different perspective and see various tools that we can use and how they work.

Sessions, input, and output

Before seeing the tools let’s understand some basics of how computers work. This will be important to understand what tools we need and how they should work. I’ll be talking mostly from the perspective of Windows OS, but the same applies to other systems.

Whenever you try to use a computer, you need to have a session. Session is an object that controls your human interfaces, most importantly input and output. Depending on the version of Windows, you may be able to create many sessions.

The main session (the one you have when you “sit at a computer”) is called CONSOLE. You can see it with qwinsta command:

CONSOLE is what you create when you use your computer “normally”. This session handles your keyboard, mouse, and (most importantly) your monitors.

You can create more sessions. Whenever you use mstsc.exe, you connect to a computer and create a session:

Obviously, when you create a new session, it must have some way of getting input (mouse and keyboard) and producing output (monitors).

It’s important to understand that one physical monitor can be in one session only (at least I have no idea how to use it in many sessions and I don’t think that would make sense). After all, you just have “one picture” on the monitor and there is no point in showing many sessions in it. Sure, you could fake it to some extent maybe to do picture-in-picture, but I really don’t see a decent use case for that.

While all of that sounds super clear, good understanding is crucial for picking the right tools.

What do we really need

Let’s now think what we really need to do the remote work.

Regarding sessions:

  • Do we want to connect to an existing session? In other words, do we want to connect to the state that the “person sitting next to the computer” sees?
  • Do we want to create a new session? This will start things from scratch just like we “restarted” the computer

However, the more important question we need to consider is how do we want to interact with the session?

  • Do we want to control it at the destination (“remote”)?
  • Do we want to control it from both places (“remote” and “local”)? This is “shared” control.
  • Or maybe only “local” should control it? This is “takeover” control.
  • Do you want the keyboard shortcuts pressed on “local” to be sent to “remote”?

If you read this carefully, you may realize some interesting facts that may change the way you look for the remote work software:

  • If you want to create a session and control it from “local” then you need RDP-like (mstsc.exe-like) application
  • If you want to connect to the existing session and control it from “local” then you need VNC-like software
  • If you want to connect to the existing session and control it from “remote” then you basically need a screen share

While all these things may seem rather trivial, they have crucial impact on what we’re actually trying to achieve. Let’s see the typical scenarios:

  • You want to connect to a remote server (like VDI, Azure VM, etc.) – in that case you want to create a new session and you want to control it from “local” which is RDP-like software
  • You want to help someone else with their computer – in that case you connect to the existing session and you want to let “remote” control it which is a screen share
  • You want to play a game on your VR goggles from your PVCR – in that case you connect to the existing session and you want to control it from “local” or from both places which is a VNC-like software
  • You want to use VR goggles to work from your laptop – in that case you connect to the existing session and you want to let “remote” control it which is a screen share

Notice that a regular “monitor” can be thought of a remote connection to the existing session that is controlled from “remote” so it’s basically a screen share. However, you could have a TV that you use to connect to the existing session and control it from “local” with your remote control (no pun intended) which is a VNC-like software.

Multiple monitors

Now we can ask the question of the monitors. This is crucial as here are many ways we can configure our machines.
Let’s assume that our “local” has 3 screens and our “remote” has 2 physical screens. So the geometry in “remote” may look like (1920, 1080, 0, 0) and (1920, 1080, 1920, 0) (which is (width, height, left, top)). Similarly, geometry in “local” may be (1920, 1080, 0, 0) + (1920, 1080, -1920, 0) + (1920, 1080, 1920, 1080).

Let’s consider two scenarios.

Creating new session in remote

If we create a new session in remote then we need to provide monitors for it. Effectively, it doesn’t matter what monitors are there in “remote” because we won’t use them. The only important thing that matters here is that we have 3 monitors in “local”. To understand what’s going on, we now need to decide on how many virtual monitors we want to create in “remote” and how we want to map them in “local”. There are typically the following options:

  • 1:1 – You create one monitor in “remote” and you map it to one monitor in “local” – this is how most RDP clients work
  • 1:n – You create one monitor in “remote” and you show it on multiple monitors in “local” – this is how xfreerdp workd for me. Sometimes this is called “span”
  • n:n – You create as many monitors in “remote” as you have in “local” – this is how mstsc.exe works when you select “use all displays”. Sometimes this is called “multimon”
  • m:m – You create fewer monitors in “remote” then you have in “local” but each monitor is mapped to one monitor. This is how mstsc.exe works when you modify the .rdp file directly
  • m:n – You create some monitors in “remote” and you map them to some configuration in “local”. I don’t know any software that would do this. Notice that this doesn’t mean that you use entire “local” monitor. You could come up with any geometry that you need.

Whenever I talk about “multiple monitors” I mean the n:n or m:m scenario from the above.

Connecting to he existing session in remote

In this scenario the “remote” has 2 monitors. You now want to map them somehow in your machine. You have the following options:

  • 1:1 – You show only one “remote” monitor in one “local”
  • m:1 – You show all “remote” monitors in one “local” (typically one window) – this is how most VNC apps work
  • m:m – You show “remote” monitors separately in “local”. This rarely works on Windows
  • m:n – You configure custom geometry. I’m not aware of any software that does that

Let’s now see the software for various scenarios.

Software

Applications that create new session (RDP-like) (and therefore “local” must control the session):

I’m not aware of an RDP-like application that would work in VR goggles.

VNC-like applications:

  • TightVNC, TigerVNC, UltraVNC and similar – they typically support 1:1 and “shared” control. You can often change the geometry on the server (I didn’t see any client that would let me change the geometry, though).
  • vSpatial – this supports n:n, and “shared” or “takeover”
  • noVNC – this is a web VNC client. It supports 1:1 and “shared”. It’s just an HTML file with some JS, so you can easily adjust it to your needs
  • Immersed – this supports n:n and “shared”
  • Virtual Desktop, Multirooms – they support 1:1 and “shared”

Screen share applications:

  • vdo.ninja – this shares one screen and works in browser, so you can share as many screens as you need
  • Google Meets, Teams, Amazon Chime, Zoom, Webex and many others – they share one screen and work in browser or in dedicated apps. You can open up many bridges to share many screens

And now the most important part (kind of TL;DR):

  • You want to have the “true” RDP experience with multiple monitors – mstsc.exe or Thincast Remote Desktop Client
  • You want to interact with some remote session from your computer – any VNC client that works for your case (and your mileage may vary significantly) – TightVNC, UltraVNC, TigerVNC, noVNC, many more. Also, vSpatial
  • You just want to see the remote session only – any screen share or VNC. Specifically, you don’t need Immersed or other “built-in” VR apps that stream the screen. You can just open up a browser and use vdo.ninja!

Being that said. If you just want to use your goggles as a “monitor” for your laptop, then you don’t need any CPU/GPU-heavy applications for that. Just share the screen in browser and it works.

And if you want to offload that even more, then use RDP + Screen share:

  • Take your road runner and connect over RDP to a remote machine. This way you can create a session there with monitors (and you can use IndirectDisplayDriver or BetterDisplay to create virtual monitors)
  • Use your VR goggles to connect to see the screen shared from the remote machine (with screen share solutions like browser, VNC, vSpatial or whatever else)

This way your road runner does nearly nothing. It just connects to the remote machine. Similarly, your VR goggles don’t work hard. They just run a browser or something light.