How I ran Cy̶b̶e̶r̶p̶u̶n̶k Minecraft in the cloud: part 3

Don't bother yourself with things that are not relevant to the present. You still have to be able to get into the future. Victor Pelevin – Chapaev and Emptiness.

The first part is dedicated to the server side of cloud gaming, the second part is about the client side, and the third is RDP 🙂

This article consists of three sections: the background of the RDP problem, approaches to solving the problem, and an unexpected solution.

Background of the RDP problem

Since the fall of 2022, Moscow State University has been conducting a course on Industrial Research in Artificial Intelligence, IIvII. This year, as the organizer of the course, I invited students to complete exercise on the topic LLM-based Multi-Agents Systems, in which you need to use LLM to control a character in the Minecraft gaming environment on MSU map. Since I'm a proponent of thin clients, I had a need to run Minecraft in the cloud.

To run Minecraft, you need an OpenGL implementation. Of course you can use software implementationbut it works slowly. Therefore, it is better to have a video card.

I chose RDP as my thin client because it has its own display, as you can see in the comparison table at the end of the second article. And RDP support is best implemented in Windows.

Thus, we got a combination of GPU+Windows+RDP, then we had to select a cloud.

It is important to note that this post is not a promotion for the Selectel cloud, but I chose this particular cloud for two reasons. Firstly, in this cloud you can rent a virtual machine with Windows Server 2019 and GTX 1080 for only 30 rubles. per hour Secondly, in this cloud the freezing function is available when the virtual machine is turned off, but the GPU is not charged.

After installation on a virtual machine running Windows Server 2019 simple Soviet drivers for GPU 565.90-desktop-win10-win11-64bit-international-nsd-dch-whql.exe and remote connection via RDP Minecraft started.

But there was a problem with the camera, or rather with the mouse…

A search showed that this problem is best known as Erratic mouse movement in 3D games over RDP with RemoteFX (this link is key and then I refer to it several times). It turns out that RDP has two modes for transmitting cursor position: absolute mouse movement and relative mouse movement. This is even written about in official documentation from Microsoft. They say that in games, supposedly for the sake of performance, you need to transmit the relative displacement of the cursor. But personally, it seems to me that this was done so that games would not be played via RDP…

Approaches to solving the problem

It turned out that when connecting via RDP, using a mouse without relative mode is extremely non-trivial. By the way, Parsec has an automatic conversion from relative to absolute. But with RDP everything is much more complicated…

Basic approach via RemoteFX USB redirection

The main idea of ​​the solution key link is that the mouse can be transferred via RDP as a RemoteFX device. This procedure is suitable for many USB devices and is called RemoteFX USB redirection. In this case, the cursor displacement is transmitted in absolute mouse movement mode.

Attempts to enable this mode following the manual key link were unsuccessful because the mouse did not appear in the list of RemoteFX devices in the Local Resources panel in RDP. Separately, I note that the manual for key link refers to the official documentation from Microsoft, which describes how to add a device to the list of RemoteFX devices via registering the GUID in the registry.

Also on the Microsoft portal in response to the question How to fix redirected USB mouse with RemoteFX not working over RDP? asked if the mouse was wireless. Oddly enough, what follows will really reveal the difference in using a mouse via RDP depending on the presence of a wire.

To summarize the basic approach, I will add that I have verified that this approach does not work on one machine running Windows 10 Home and another machine running Windows 10 Pro. I will leave without comment the option when, in order for Remote FX USB redirection to work, it is recommended upgrade Windows version to Enterprise.

USB forwarding via third party solutions

The next option we considered was when the mouse is transmitted not via RDP, but through a third-party solution. An analogy with audio transmission in Parsec via VB-Audio Virtual Cable is appropriate here. I tested the following three solutions for USB forwarding:

1. USB Network Gate

   Interestingly, at the end of the page the link says that: You can't allow RemoteFX to utilize USB devices while connecting to an RDP server if your Windows 10 HOME edition lacks gpedit.msc. So maybe upgrading Windows to Enterprise makes sense…

2. USB over Network

3. VirtualHere

   In my opinion, this is the most interesting solution. Need to run on local machine VirtualHere USB Server. On a virtual machine in the cloud, you need to disable Windows Defender Firewall. You also need to run in the cloud VirtualHere USB Client. And then all that remains is to connect to the local machine via Reverse Clients.

All three solutions didn't work for my mouse, it just wouldn't forward. However, remembering the comment above about whether the mouse is wireless, I decided to check these solutions for Bluetooth mice, since before this the tests were for wired mice…

Surprisingly, it worked and with the help of VirtualHere I was able to forward the wireless mouse when connecting via RDP. I knew the forwarding was working because I happened to see the cursor moving in the console of the virtual machine in the cloud, but not moving on the local machine monitor and on the RDP virtual display.

I noticed that when connecting via RDP, a virtual mouse device is created that runs in the RDP virtual display. Thus, to complete the analysis of the current approach it was necessary:
– Learn to disable RDP Desktop Mouse Device
– Learn to disable the virtual machine console in the cloud

It is possible that doing both of these actions would result in the mouse cursor moving on the RDP virtual display after the wireless mouse was transferred, but I decided not to test this hypothesis. However, I tested the hypothesis of connecting the PS/2 mouse via RDP, since both the wired and wireless mice were previously USB devices. Unfortunately, when connecting a PS/2 mouse via RDP, the problem persisted.

Alternative approaches

I also found the following alternative approaches to solving the problem:

1. Use Chrome Remote Desktop as a thin client

2. Create a virtual display using another thin client, For exampleremote.it and then connect to it using Parsec

3. Use a gamepad or keyboard arrows to control the camera

Option with an asterisk: to create a display, you can also use a physical dongle, for example, the familiar HDMI-VGA adapter, but for some reason cloud providers do not provide such a service…

I was ready to start looking at alternative approaches when I suddenly found a solution!

Unexpected decision

As the classic said: reality is full of disappointments. The solution was found where I didn't expect to find it: on the Microsoft portal…

One architect in Covid times also faced the same problem and really asked Microsoft engineers to solve it.

And in the comments he was advised to pay attention to the following link: https://github.com/TKMAX777/RemoteRelativeInput

And the same author had another solution: https://github.com/TKMAX777/RDPRelativeInput

In a sense, the author of the solution implemented a plugin for RDP, which allows you to use a mouse without relative mode when connected, which is what was needed for Minecraft!

Message to myself

Thus, the problem with the availability of the cursor in the game when connecting via RDP from the second article has been resolved. And now RDP Also suitable for cloud gaming. All that remains is to deal with Horizon…