Telepresence, 3D video communication and NON-holographic booths

There are three ways to show a 3D image to a person:

1. Give him special glasses ↑ to create the desired picture for each eye.

2. To create the illusion of a three-dimensional object in ordinary human reality using special displays ↓, effects, booths – this is called autostereoscopywhen nothing needs to be placed directly in front of the user's eyes to perceive volume. Modern 3D displays can already track the position of your eyes in three dimensions and rebuild the image on the fly, but this does not stop the illusion from being an illusion.

3. Making a real hologram is the most advanced autostereoscopy.

One-way 3D video communication in real time

Stereo cinema that's good, they learned to show it quite a long time ago. But we are interested in technologies that can or are going to transmit a three-dimensional image of a real person in real time, so that it can be called telepresence.

Real holograms have not yet been learned to transmit over a distance. There is also digital holography – Computer-Generated Hologram or CGHThe program creates a model that does not require a real object. [но это не наш метод, нам надо телепортировать живого человека]or live shooting is calculated to create a model. The computer hologram is displayed in front of the viewer in 3D glasses or on a special 3D display.

Why is the universal desire for “holographic” communication not taking off? Content is first inconvenient to shoot and difficult to process, then difficult to transmit and even more difficult to reproduce. And if the first two problems are solved by increasing computing power and machine learning, the problem of reproduction is still relevant.

In 2008, the University of Arizona created a display measuring approximately 10×10 centimeters that could refresh monochrome 3D images every few minutes.

“Imagine watching the World Cup final or the long-awaited sequel to Sex and the City and seeing all the action unfold in 3D on your coffee table.” CNN scares us

The prospects were assessed very differently. The authors hoped to find sponsors in the consumer electronics market, wanted to enter the holographic television and consumer holographic video communication systems in 5-10 years. But they believed more in education, the military and medicine. Colleagues in the industry were skeptical: “It's one thing to demonstrate something in a laboratory, quite another to be able to produce it cheaply and effectively enough to distribute it to the mass market.”

In 2014, OTOY, a company invested in by Yuri Milner and Autodesk, was advised by Eric Schmidt, showed the first portable light field capture system and cloud pipeline for image rendering that bridges the gap between offline and real-time rendering.

“The same level of incredibly precise rendering is used to create special effects in major motion pictures. It would typically take hours, if not days, to render a single, highly polished and realistic frame for use in a film. OTOY has demonstrated the same enormous computational complexity, implemented in hundredths of a second, on the screen of one of the least powerful devices we use every day – a smartphone. The result is video with lifelike image quality that can be navigated and explored from any angle, otherwise known as holographic video,” Press release

The content was planned to be viewed using Oculus Rift glasses or in a separate volumetric holographic display box, which was located on development stage. Commercial use was planned to begin in 2015. All this together was called Holodeck, inspired by Star Trek.

One investor: “While the holographic displays we envisioned don’t exist yet, VR headsets give us an incredible opportunity to bring the Holodeck to life in a slightly different way.”

That is, again, everything rests on reproduction. Our brain can see a three-dimensional interlocutor either through a special headset or through an installation/display that reproduces the illusion of volume.

In 2016, Microsoft HoloLens project manager Alex Kipman showed a full-size hologram of a NASA professor on the TED stage, startling the host. The professor was in a room across the street equipped with 3D cameras.

The function was called Holoportation. The professor could only be seen through a headset. Kipman inserted a special lens into his glasses, the image from which was transmitted to the audience on a large screen. TED video with timecode

In 2017, between New Jersey and Seoul via 5G networks called each other 3D employees of Verizon and Korean Telecom. The interlocutors were displayed as “holograms” on each other's monitors; the technology did not allow for the reproduction of objects in the air. The service was called Live Call.

“If the 5G network is commercialized, video calls with 3D hologram will be available as one of the typical 5G-based services,” KT officials predicted.

In 2021, mobile calls with “hologram” participants were offered Slovak company Matsuko. You can still see your 3D interlocutors only with special glasses, but the shooting is done with the iPhone's front camera using an app. The regular video in the cloud is converted into 3D and from there hangs in the air opposite the one wearing the glasses. Visual video Here And Here.

The developers promised to learn how to conduct “holographic” video calls with a large number of participants simultaneously. The largest operators Deutsche Telekom, Orange, Telefonica, Vodafone – they are responsible for 5G – have agreed to cooperate with the project.

At the end of 2021, Cisco, one of the main vendors of classic video communications, showed a solution Webex Hologram. They used light field technology, which is based on the principle of changing color perception depending on the viewing angle. 12 cameras capture color reflections from different angles, then the resulting images are synchronized with information about where the viewer's eyes are and how they move. A person wearing HoloLens or MagicLeap glasses can see a three-dimensional photorealistic figure of the interlocutor. At the end of 2023, the updated installation will already sent clients for testing.

So, for now we can only take high-quality pictures of the speaker. And show it as a volumetric one-way view to a person or several people who have individual glasses or an individual display. We can also make it a volumetric illusion for the auditorium, where again only the speaker is three-dimensional.

Mass performances and education

From here, the areas where 3D speakers have begun to spread and gain momentum logically follow – this is the entertainment industry, education, politics. And also a convenient, simple kiosk-like format that helps to visually recreate the illusion of presence, while maintaining “live” contact with the audience via video communication.

In 2019, a notable event was the American startup Portl, later Proto Hologram. They created a light booth with a screen, speakers at head level, and a video camera that films the audience. The installation was well described ZDNET journalists described it as a “full-size, vending-machine-sized machine.” Then came a mini version.

To record a person's image, all you need is a camera and a white background; in 2022, it was already possible to shoot on smartphone with the app, albeit with a loss of quality. Additionally, you can overlay any image on the hologram, try on clothes, etc.

Technologically, a pseudo-hologram is reproduced in the box – Pepper's Ghost illusionThe system is interactive, the audience is also filmed, so the main character can see the interlocutors and conduct a dialogue.

The equipment for visually reproducing a life-size human being cost from
$60 thousand. The developers “resurrected” dead celebrities, held stand-ups. In 2023, it was already used to test a court hearing, where 3D witnesses spoke ↓

In 2025, 3D lectures based on Proto Epic will officially plans introduce into the curriculum of the British University of Loughborough. In addition to inviting remote lecturers, the developers have already thought about creating realistic characters whose personalities can be recreated using artificial intelligence, but are still working out legal issues.

2021 ARHT Media for $25 thousand. organizes session spiritualism telepresence of your speaker in a remote location. They are filmed against a green screen and then projected onto an almost invisible grid. The speaker sees the audience in front of them on large displays and can respond to questions from the audience.

ARHT Media HoloPresence technology was almost scaled in WeWork coworking spaces. entered into an agreement with the developers and were going to equip 100 locations. Knowledgeable specialists they saidthat the solution works best in darkened rooms – for example, in large auditoriums. For the office, the above-described Portl was recommended.

An example of a political performance is the simultaneous performance in 2022 by French presidential candidate Jean-Luc Mélenchon in 12 cities.

There is a holiday on our street too. Russian TrueConf also showed in 2024 “holographic” booth ↓

The speaker's image is captured by a 4K camera connected to a PC with the TrueConf app, a smartphone camera will do as well. The video stream is sent via a video communication server to the TrueBox booth. No additional equipment is required, just good lighting and a white background.

The device uses a transparent 4K display with a depth effect that creates the illusion of a three-dimensional image and a live presence.

The system is interactive, the speaker sees and hears the viewer. The viewer can control the booth from a phone by scanning a QR code, or can press directly on the touch front screen. Video

Mutual 3D from video conferencing vendors

Two-way communication, where both participants are three-dimensional and without glasses, has become available relatively recently. It logically arose in the depths of large vendors engaged in classic video communication.

Google and Logitech may have developed glasses-free two-way volumetric communication, but instead of glasses they offer impressive booths/installations for each participant. It is not yet possible to throw off the shackles. The cheating way is to hang out as avatars in metaverses like Microsoft Mesh.

Google Project Starline

↑ Google in 2021 showed massive experimental setup called Project Starline. Like Cisco Webex Hologram, it used light field technology, but instead of 3D glasses, it had a 3D display.

NoJitter Analyst Dave Michels compared Project Starline with NASA's invention of a space pen made of famous tale. In his opinion, this is more of a large-scale scientific project than a real product. The technology is not available outside the laboratory and is prohibitively expensive.

In May 2023, Google reconsidered its approach and proposed a more compact version of Project Starline ↓

The bulky cabin was replaced with a display with cameras that would fit into a regular conference room. The project involved new artificial intelligence algorithms that rely on several standard cameras instead of a complex hardware system with infrared emitters to create a 3D model of a person. Very interesting demonstration.

In May 2024, the technology was released from the lab to the market. Google has teamed up with HP and plans to start making money on this topic in 2025.

Logitech Project Ghost

Almost simultaneously with the mini-version of the Google project, in 2023, Logitech proposed a simpler, but no less dimensional design.

Technology – “Pepper's Ghost”the effect of the interlocutor's presence is achieved by reflecting the image from the display on a special glass placed in front of it. Inside the furniture set from Steelcase is an affordable commercial system Logitech Rally Pluswhich gives the project a better chance of survival and distribution.

This year at ISE 2024, Logitech showed a two-seater booth, a ceiling and the ability to close the door with a seal to create a space isolated from outside noise. Before that, users complained about the lack of privacy.

Promises quick commercialization. The exhibition sample was sold with Microsoft Teams Rooms integration, options with Google Meet and Zoom are provided.

In short, the format is still not suitable for mass use. Individual organizations with many branches will be able to please their employees with exclusive communications, but you can’t give everyone a room-sized lab booth. We can only hope for a total transition to inexpensive 3D displays or other magical display devices for everyday use that visionaries dream of.

Why is all this necessary?

There is already a revolt against classic video communication approximately everything. Some due to loss of time, others – health. Mainly mental. Zoom fatigue raged across the planet, with varying success forcing people to lose interest in work, to become nervous, do plastic surgery and many other strange things.

3D developers see their volumetric worlds as more natural for people to live and communicate in. Google will even conducted a studyto find out how much more effective Project Starline is than a regular video meeting.

Users of Google's Project Starline 3D video communication system:

• They talk more intensively than in regular video conferences. The number of so-called “turns” of conversations, that is, the number of lines passed to each other, was 2-3 more in a two-minute segment.

• They use 43% more gestures, 26% more head nods and 49% more eyebrow movements.

• Maintain eye contact with the person you are talking to 14% longer. This is roughly equivalent to eye contact during a face-to-face conversation.

• 31% less fatigue as measured by the Zoom Exhaustion and Fatigue Scale (ZEF).

• 12% faster response to cognitive tasks after a session.

• They describe the conversation in more detail, with an average of 28% more characters in the description than users of traditional video conferences.

• They rate their conversations on Starline as significantly more natural (quotes: smooth, easy, not awkward).

hope dies last

↑ The holographic display market is expected to almost triple in size by 2029. “However, the prohibitively high cost of producing holographic displays limits market expansion.”

In a separate line, analysts mention that at the end of 2023, the company Realfiction held the first public presentations of the multi-user 3D display ECHO. It is based on the patented technology of directional pixels – Directional Pixel Technology (DPT) – and displays independent 2D and 3D images to multiple people at the same time without loss of quality. So independent that passengers in the car can watch action movieand the driver looks at the navigator. And all this on one screen.

One of the areas of development is telepresence and group calls with all the usual attributes: digital boards, 3D content demonstration and other collaboration, as well as family gatherings. There is a demo with simultaneous showing of the presentation to several people, each in their own language.

The mass distribution of displays is still a long way off, but there are already attempts to make holograms tangible using laser tingling And air supply different temperatures.

Expert opinion

In 2019, the Russian company VRCAST developed a telepresence system for top management. Managers can remotely connect via a browser from any device or in VR glasses to a special 360° camera and “personally be present” at a production facility or in the office. Two-way video communication works in real time and in film quality.

VRCAST founder, international expert in immersive technologies Anatoly Goronesko:

“Companies with solid investments have proven unable to sell holography, augmented and virtual reality to the mass user. As an example: the closure of the volumetric system Intel TrueView and Intel Studios, a massive domed arena with a green screen and over 100 8K cameras for shooting from any angle.

I think that it is currently impossible to make a working holographic video communication in real time for the mass segment. This is more of a marketing and advertising application for the wow effect. The hardware and software part, algorithms, processing are not sufficiently developed – all this is still individual, cumbersome and very expensive. The problems with communication channels, computing power, transmission delay have not been solved. New discoveries in optics, improvement of laser projection technologies, lidars are needed. My forecast is not earlier than 2030.

“Microsoft has an interesting and very old technology called Kinect. It's a camera with a depth sensor that can do essentially what we've been waiting for – create a point cloud and subtract the background. Right now it's not very high resolution, it's not very wide-ranging, but the evolution of that technology seems to me to be the most realistic path forward for 3D communication.”