What touch virtual walls?

Over the past few years, devices that allow the user to plunge into virtual reality (VR) have become much more complex and better. VR is used in education, and in art, and in entertainment, and even in medicine (no, I’m not talking about Surgeon Simulator). Thanks to VR, a person can see something that he would never have seen in real reality, such a pun. However, for a complete immersion in the world of dreams, one of the most important sensations, which provides a person’s contact with the outside world, is lacking – touch. We constantly feel something to the touch: keyboard keys, doorknob, tea cup, etc. In VR, all objects, no matter how realistic they seem to the eyes, are devoid of any physical representation in reality. A group of scientists from Carnegie Mellon University (USA) decided to fix this using a fairly simple device in concept. What the invention consists of and how it works, we learn from the report of scientists. Go.

Study basis

Modern VR systems use controllers (and even ordinary gamepads from game consoles) to track the position of the user’s hands, control the actions of their virtual avatar and for vibration-tactile response.

Those of us who have a game console know that gamepads can vibrate at certain points in the game (battle with the boss, collision with an opponent in races, etc.). VR also uses this method of communication with the user, however, it does not give that fullness of sensations that would give a touch.

If we talk about the walls in VR, then they successfully implement the concept of the Schrödinger cat – they seem to be there, but at the same time they are not. The user can go through them (in whole or in part) intentionally or not. And this greatly affects the feeling of the world created in VR.

To get rid of the above disadvantages is quite real, although difficult in terms of design and implementation. There are several options for creating tactile contact with VR objects, from the exoskeleton that organizes the user’s movements, to the physical restriction of the VR system in real space (i.e. real walls). However, both of these methods limit user freedom of movement, and therefore cannot provide a complete and comfortable immersion in the virtual world.

Image # 1: The concept of the Wireality system.

In the study that we are considering today, scientists decided to create their own version of the tactile contact system by means of threads attached to sensors on the user’s hands, which looks like Pinocchio. Such a system is autonomous and very mobile, not to mention its low cost – up to $ 50 apiece for mass production. The inventors named their creations Wireality, combining wire (wire, thread) and reality (reality) in one word.

Wireality system implementation

So, the main physical properties of the system that its creators wanted to get were ease of use (the system will be attached to the user), energy efficiency (an autonomous power supply is needed) and relatively low cost.

The conceptual and mechanical basis of the system are threads that will in a certain way stop certain sections of the user’s brush at the right time. As a material for the threads, it was decided to use fairly strong and thin steel cables with a nylon coating.

Image No. 2: prototype of the Wireality Thread Tagging Module.

When the threads are the limiters of the position of the hand, they should not be allowed to sag or to excess tension, because this will disrupt the system.

To prevent this from happening, it was necessary to use something that would effectively tighten the threads. The first options for the tightening module, as the scientists themselves admit, turned out to be cumbersome due to the engines, and the problem of heat removal appeared (the scheme above).

Image # 3: Final version of Wireality Thread Tagging Module.

Instead, flat springs were used, as in measuring tape. This option is much easier, cheaper, stronger and more compact. Even though the flat springs provide an exceptionally fixed traction, it was found that a retractor module with a traction force of 80 g perfectly eliminates the problem of sagging threads.

The threads are selected, the pulling module, too, it remains to determine the module for fixing the position of the brush. One of the most important factors in obtaining the most realistic sensation is the delay in the response of the fixation module to user movements. Wireality systems utilized cam and ratchet locking mechanisms.

The final design of the locking module consists of an acrylic ratchet mechanism with ratchet (gear with sharp teeth, roughly speaking) with a resolution of 8 ° per tooth and a tightening module in which the spring, spool and thread are installed.

Since the thread is wound on a spool with a smaller diameter than the ratchet, each tooth is equal to 0.84 mm “string travel”. The cam, in turn, attaches to the push pull DC solenoid (12 V).

Therefore, when the user’s hand is in contact with the virtual object, the system program starts the solenoid, pushing the latch into the ratchet, which blocks the coil from further rotation, i.e. prevents further release of threads, which stops the position of the hand. And due to the effort exerted on the thread by the user himself, the ratchet holds the latch, so the solenoid can be turned off quickly enough, which reduces energy consumption.

Image # 4: thread lock module.

One blocking module is responsible for one section of the user’s hand (for example, for the index finger), because they need several. Compact size (4.1x7x1.4 cm) and lightness make it possible to connect them into clusters (photo below).

Image # 5: cluster of lock modules.

Each module is housed inside a case of nylon and carbon fiber printed on a 3D printer.

The electronic component of the interlocking modules consists of the HBridge L298N dual motor driver controlled by the Teensy 3.2 microcontroller, which receives commands from the VR system via regular USB.

These electronic components are compact and practically do not consume energy when the solenoids are not involved. The dimensions and cost of the module can be reduced by using a transistor, which will be considered in further studies.

When Teensy instructs to block one of the threads, it activates the corresponding solenoid for 40 ms, which is the shortest interval for reliable blocking.

Having gathered all the modules together, scientists thought about where all this joy would be attached to the human body. The prototype, of course, is not yet ideal, and its size can be reduced, but even at this stage it is quite convenient to wear it. It was decided to place the Wireality system on the user’s shoulder by means of a special vest that distributes the load on the body (image No. 1).

As for the hands, it was necessary to understand the optimal number of contact “sensors”, which directly determines the number of blocking modules and, therefore, the number of threads used. Scientists have found that with a large number of contact points (for example, the tip of the finger, metacarpophalangeal and interphalangeal joints), more complex geometric shapes can be modeled. However, this inevitably leads to an increase in the dimensions of the device and its weight (a total of 15 modules are needed: three for the index, middle, ring and little fingers, 2 on the thumb and 1 on the wrist).

During the experiments it was found that such a large number of sensors is not necessary to obtain realistic tactile contact with a virtual object. Therefore, in the end, the number of strings and modules was reduced to seven: 5 for each finger, 1 on the palm and 1 on the wrist.

Sensors with threads are attached to the user’s hand through thimbles printed on a 3D printer and Velcro straps, which helps to distribute concentrated pressure.

To hold the user’s fingers at points in three-dimensional space to simulate complex virtual objects, precise tracking of multiple joints of the arm is required. In early prototypes (one of the options in image No. 2), a potentiometer was used, which made it possible to accurately track the distance to a point from the module, but not the azimuth or altitude.

Ultimately, Leap Motion, attached to the front of the VR headset, was used to track hands. Leap Motion provides data on the position of all joints of the arm in three-dimensional space. It is curious that the thimbles that are worn on the user’s fingers were made of a material that does not interfere with the operation of Leap Motion, since in the infrared they look the same as leather. Another important detail was the threads themselves, which were supposed to remain invisible to the camera, because their thickness was also not chosen by chance.

Oculus Rift was used as a VR system, and all virtual experimental sites were created using Unity. To detect contact with virtual objects, an ObjectCollider was assigned to each object or obstacle on the site (wall). When a contact occurs, an OnTriggerEnter event is dispatched. The event handler sends a lock command to the driver board via USB for the corresponding joint, i.e. the corresponding lock module.

Contact detection is performed in parallel for all joints of the fingers and hand, and the actuation of the solenoids does not interfere with each other, providing completely independent control, which is necessary for tactile visualization of complex geometries.