accessible virtual reality based on smartphones

Hi all! My name is Evgeniy, I am the developer of the EnJoyTheVR application – an application that allows you to immerse yourself in virtual reality with 6DOF tracking of head and hand movements directly through your smartphone. In this article I would like to talk about the project, its idea, stages of development, and also share positive and negative aspects.

Now almost everyone knows what VR is, but not everyone has the opportunity to purchase expensive equipment for complete immersion in virtual reality. There's been a thing called Google Cardboard for a long time, but most apps built around it are limited in functionality, offering only basic head tracking and often basic interaction. In contrast, EnJoyTheVR provides full 6DOF tracking of both the head and hands, making the experience much more realistic and rich.

It was this idea that formed the basis of my school project 🙂 Let's see how successful it was in implementing it.

For development, I chose Unity as the main platform. It provides simple tools for creating applications with good graphics and optimization (which became the second most important task after implementing hand tracking). In addition, Unity allows you to develop two versions of the application at once – for Android and iOS – almost without unnecessary problems. For example, the ARCore and ARKit components are used in the same way.

The most difficult task was to implement hand tracking. I won’t go into technical details, but I will say that after a long search for a solution, I settled on a convolutional neural network (CNN). This approach resulted in fairly accurate tracking, although not perfect.

Next, I modernized the example scene from the official asset for developing VR applications XR Interaction Toolkit. A virtual controller was “barbarously glued” to the hand, and the controls were remade for Joy-cons.

After these improvements, I took part in my first school scientific conference, where I took first place among the 10th and 11th grades. This became a source of motivation for me to continue working on the project.

After a complete overhaul of the stick-on virtual controller method and several experiments with different mechanics, it became clear that one-handed control was not enough. I've modified the code to support second hand. At this point, the application looked much more functional and complete compared to its earlier versions.

Soon I decided to try my hand at the city scientific conference “Engineers of the Future”, which was held at MPEI. My project won in the “3D modeling, 3D printing and VR/AR technologies” category.

To show that this project can be used not only as a separate toy, but also as a simple tool for porting VR games from other platforms, in a couple of summer days (with the help of the magic of decompilation) I ported two games: MineSouls III (from Oculus Quest 2) and Rick and Morty: Virtual Rick-ality (with SteamVR).

Then the main menu, settings, support for third-party levels and much more were made.

The result of long work was a VR application with 6DOF tracking of the head and hands, the ability to fully interact with the gaming space, support for high-Hz screens, the ability to record video (both from the player’s point of view and from a virtual camera that can be mounted anywhere), and also a built-in application store where enthusiasts can download their games and applications.