Virtual reality at your fingertips

Imagine that you are sitting at your computer and typing, playing the piano, or simply banging your fingers on the table out of boredom. Each movement of the finger, each tapping generates a vibration unique of its kind, passing from the phalanx to the base of the hand.

Researchers from the Swiss Higher Technical School of Zurich (ETH) have used this physical feature of human hands in their invention – a two-sensor bracelet that allows you to interact directly with virtual objects.

Virtual reality is already widely used in various fields, but it is mainly needed to absorb content, not to create it. But Swiss scientists are on their way to addressing this injustice.

One of the ETH professors, Christian Holtz, sees VR technology as a potential replacement for desktop PCs. In his opinion, “decoupling from the screen” will make it possible to fully utilize the virtual space: to place “working tools” around oneself and interact with it in the most natural way – by hand.

Each finger vibrates differently

Why are there still no virtual computers with an unlimited number of monitors and peripherals? Holtz says it’s all about the absence of normal interaction mechanisms between humans and VR objects. Now VR systems are equipped not only with helmets / goggles, but also with special controllers and cameras to record the user’s movements. What kind of productivity and focus at work can we talk about when you have to interact with such a sophisticated system?

“If you have to constantly keep your hands on the weight (so that the camera can detect them), then you will quickly get tired. And you often have to work for several hours without a break, ”adds Holz.

The second problem is that when typing on the keyboard, the hands move differently than when playing tennis, so it is difficult for the camera to capture and correctly interpret the movement of the limbs. In addition, in this case, there is no physical return.

Holz’s team resigned themselves to the fact that additional passive interfaces were indispensable, but decided to rethink them, turning fingers and their unique vibrations that occur when touching any hard surface into such an interface. To implement the idea, TapID technology was created, which will be presented at the IEEE VR conference at the end of March 2021. Holz and his team will bring TapID prototypes there in the form of multisensor rubber bracelets.

Sensors in the bracelet register the moment a finger touches any surface and tell the system which finger was used by the person. To do this, the sensor analyzes the vibration at the base of the hand and, based on the characteristics of this vibration, determines which finger is being used at the current time. The bracelet works in conjunction with a camera installed in VR glasses and an advanced neural network developed specifically for TapID. Together, they allowed the researchers to achieve extremely high precision in recording “keystrokes” on virtual keys. The developers of TapID have already demonstrated their brainchild in work on the example of two applications: a virtual keyboard and a virtual piano.