how MTS and Daiger created software for industrial AR glasses
Who developed what and why?
In short, the combined team of MTS and Daiger created software for Rokid’s industrial AR glasses so that the “glasses + software” combination could be used at domestic enterprises.
The developed software is called “Digital Control”. It is intended for such complex industrial facilities as oil and gas fields, mining enterprises, airports, logistics terminals, etc.
The final system was supposed to solve several major production problems:
complexity and/or high cost of delivering an expert to a remote site
the complexity or impossibility of objective control of routine operations – these are walk-throughs, inventory, data collection, technological operations, etc.
the impossibility of simultaneously performing a number of operations, as well as the difficulty of using a phone/tablet in parallel to obtain reference information and/or photo and video control of key steps
various operations that require prompt or even instantaneous transfer of information between several employees
We decided to use AR glasses as the hardware basis of the project. Looking ahead, let's say we chose a supplier – the Rokid company. It has been on the market since approximately 2017, and its products are quite well known. Her glasses were chosen because they have excellent screen brightness. And this is important because critical information is often projected onto displays. Another point: information can be displayed for each eye separately, which is not the case with all competitors. All this significantly affects the convenience of work. Here are the partner device models we worked with:
The first model, Rokid X Craft, Designed for use in industrial environments. These are flip-up goggles with a high degree of protection and two cameras.
Here are their characteristics:
display: 1280 × 720, FOV 40°, contrast ratio 400:1, 1600 nits maximum brightness
photo resolution: 3840 × 2448 pixels
video recording: 8 MP
battery: 7,200 mAh, 8 hours, ability to connect external power supplies
built-in memory capacity: 64/128 GB + 4 GB
CPU: Amlogic A311D
controls: touchpad, power button, return button, SOS button, multiple interactive modes
connection: support for 5G module, ultra-high speed Wi-Fi 6, BT 5.0
interfaces: Type-C, audio 3.5 mm
weight: 600 g
certification and degrees of protection: ATEX TPS 22 ATEX 113979 0001 XCertificate Number: TPS 22 ATEX 113979 0001 X Rev. 01ATEX Zone 1/2 and Zone 21/22IECEx: TPS 22.0007XEX Marking: II 2G Ex ib IIC T4 Gb II 2D Ex ib IIIB T130°C Db.Tamb: -40° to +50 °C
Second, Rokid Glass 2, is universal and can be used in various industries.
Its characteristics:
viewing angle: 40°
sensors: accelerometer, gyroscope, IR sensor
Battery life: 8 ocloc'k
controls: touchpad, voice
audio: 2 microphones, stereo sound, active noise reduction
interfaces: USB (Micro/Type-C)
OS compatibility: PC, Mac
wireless connection: WiFi
weight: 96 g
Third, Rokid Air Pro, — the next stage in the development of Rokid Glass 2. These are AR glasses for everyday use.
Here are their characteristics:
weight: 85 g
equipment: Rokid Air Pro
audio: HDx2 directional speaker, 2 high-performance noise-canceling microphones
display resolution: 1,920 × 1,080 × 2
camera type: 8 MP
viewing angle: 43°
streaming support: streaming video 1 080p
update frequency: 60 Hz
controls: voice commands, gestures, screen on/off button
other: designed for myopia from 0.00 to -5.00 D
display: Micro-O LED
sensors: advanced 9-axis (IMU, magnetometer), sensor fusion circuit, proximity sensor
Fourth, Rokid Air, is positioned as an AR device for civil servants. It has a flip-up blackout cover and a great sporty design. There is a possibility of voice control.
Here are the characteristics of this device:
display: 1,920 × 1,080, FOV 43°, 100,000:1 contrast ratio, 1,800 nits maximum brightness
photo resolution: 3264×2448
video recording: Full HD
battery: 7,620 mAh, 8 hours, ability to connect external power supplies
built-in memory capacity: 128 GB + 4 GB
CPU: Qualcomm Snapdragon XR2
controls: touchpad, power button, voice control, multiple interactive modes
connection: support for 5G module, ultra-high speed Wi-Fi 6, BT 5.0
interfaces: Type-C port
weight: 83 g
additional features: 9-axis IMU, GPS, proximity sensor, microphone, directional speakers
The fifth, lightweight model is called INMO Air 2. Control is carried out using a joystick ring.
Here are the characteristics of these AR glasses:
display: FOV 26°, Micro OLED, sRGB 100%
photo resolution: 640×400
video recording: HD
battery: 500 mAh, video playback duration – up to 2.5 hours
built-in memory capacity: 32 GB + 2 GB
CPU: ZiGuang ZhanRui AI/4 cores
controls: touchpad, power button, voice control, multiple interactive modes
connection: support for 5G module, ultra-high speed Wi-Fi 6, BT 5.0
interfaces: Type-C
weight: 99 g
additional features: GPS, proximity sensor, microphone, directional speakers
How it all started
The idea of creating a software platform arose in June-July 2022. Several options have been proposed:
VR development
video analytics
AR, augmented reality
We analyzed the market for about six months. And not in vain, because in the end they realized that the most open niche is AR. At that time, basically everyone used Realware products, which only had a monocular, but high-quality devices were already appearing, for example from Rokid.
We had a hypothesis that the solution would be in demand in all areas of business where control of technological processes is needed, including by qualified foreign specialists. At the same time, control objects are often located in hard-to-reach or remote locations – and this feature had to be taken into account.
From the beginning of 2023, we began active development: we tried various models of gadgets, for which we traveled and assessed the capabilities of the devices in person. As a result, we decided to write software specifically for Rokid. Almost immediately we encountered a problem: for all these years, Rokid has not provided a good SDK and documentation, and the community is very limited. But we were able to figure out how to use the poorly documented API.
It is worth noting that the project was implemented by 8 Daiger people under the leadership of the company’s technical director Sergei Golovin: a product, a designer, a DevOps, two front-end developers, three back-end specialists and an Android developer. On the MTS side, the project was supervised by Product Director Viktor Kasparov.
For the backend, we used JS, Nginx, Node.js, Express.js, Socket.io, Prisma, PostgreSQL, COTURN and the Pm2 process manager.
As for the web application, we used Typescript, React, Redux, Znt Design, Socket.io.
The software for AR glasses is written using Kotlin and the library for real-time data exchange Socket.io.
The product, in addition to the problems already mentioned above, helps solve others:
monitor key information about the surveillance object in real time
assess situations at the enterprise
control the precise actions of the operator
transmit telemetry, instructions and video to the operator’s augmented reality glasses
view records from glasses based on the results of operators walking around the object
Thanks to the active participation of colleagues from Daiger, we confirmed the hypothesis in practice and moved on to launching the product on the market. The pilot project received positive feedback. Users noted the stable connection and user-friendly interface, including a clear picture on the operator’s AR glasses: you can work even outdoors in bright daylight.
The launch of Digital Control into commercial operation is planned for the first quarter of 2024.
How does it all work
Interaction with software and hardware systems based on AR glasses involves three categories of system participants: administrator, expert and operator.
Employees of the first two categories work via the Internet in a web application. They use a PC, laptop, tablet or phone. Here are the technical computer requirements for running our software:
operating system: MS Windows 10 (64 bit)
volume and type of RAM: from 8 GB DDR4
Hard drive capacity: free space from 10 GB
clock frequency and processor type: AMD/INTEL 2.1 GHz 4 cores or more, support for SSE/SSE2.1, AVX system bus frequency – from 8 GT/s
Graphics accelerator: Intel Iris Plus series above 655 model or Nvidia GTX series above 970 model
availability of peripheral devices: high-speed USB 3.0 port, 1,000 Mb/s network adapter
Here is an example of what the software platform interface looks like for an administrator, expert and operator:
The administrator creates users, groups, manages tasks, assigns executors, and also fills the knowledge base.
The expert supervises the execution of tasks, advises operators via video communication, schedules conferences and fills in the technical database of facilities.
Well, the operator already performs tasks according to checklists, calls experts for consultation and conducts inspections of objects using a knowledge base and telemetry. The Digital Control application for AR glasses provides operator access to the system to interact with the expert and the announced functionality.
And now – about cases
Let's talk about three different situations with three industrial enterprises. In fact, there may be much more cases.
Oil service company. It is necessary to conduct an audit and flaw detection of equipment at oil production sites, speed up the preparation of an industrial safety report, and reduce the company’s travel expenses. And also optimize the man-hours of expensive specialists.
The solution is the implementation of the “Digital Control” system with the “Remote Expert” module, the output of telemetric information and technological maps, as well as the use of dashboards.
The result is that a flaw detectorist at the field, under the guidance of an industrial safety inspector from the central office, inspects the necessary components using AR glasses. The inspector quickly generates a report, and without personal presence at the site. Accordingly, the time required to conduct inspections is significantly reduced, and the costs of travel and related expenses are reduced.
Coal mining company. Here the task is different – to automate typical business processes:
acceptance of wagons with coal
accounting of the condition of control and measuring instruments
inspection and repair of coal feed belt
monitoring the serviceability of railway tracks, cars and car dumpers
In addition, to improve the quality of work of teams while simultaneously reducing the stay of people in an unfavorable environment and recording the walk-through process until subsequent analysis.
The solution is still the same – the introduction of “Digital Control” with the “Remote Expert” module, the output of telemetry and maps, as well as the use of the “Industrial Safety” dashboard based on Safe Zone.
The result is the acceleration of most processes, including onboarding new employees, expert consultations, recording defects and filling out reports. In addition, it also controls the movement of personnel around the industrial site.
Industrial enterprise. In this case, it is necessary to carry out commissioning of the production line without the arrival of specialists from another region.
The solution is still the implementation of the “Digital Control” system with the “Remote Expert” module, output of telemetry and maps, the use of dashboards with output of data from MRO (maintenance and repair of equipment).
The result is a reduction in the enterprise's expenses for expensive business trips, as well as independent commissioning and adjustment of equipment under the guidance of experts from the production line vendor.
We analyzed the results of the cases and found that digital control reduces the time for checking and setting up equipment. It also removes the burden from contractors, improves the quality of service and speeds up the following processes:
search and acceptance of materials and equipment
filling out acts and protocols
equipment inspection and problem correction
carrying out maintenance
entering measuring instruments into the accounting system
In addition, there are other results. For example:
the employee gets acquainted with the instructions approximately 10 times faster
Equipment defects are described 5 times faster
reports after a crawl are completed approximately 10 times faster
data on objects in accounting systems is updated up to 120 times faster than usual
Everything is fine, but we are planning to upgrade the system in the near future. In particular, we are going to expand the line of supported devices: Rokid is releasing new models, and new manufacturers are appearing. We are also connecting additional services; now on the agenda is the integration of the MTS AI solution in the field of video analytics and voice recognition.
That's all. If you have questions or just want to discuss the topic of the article, write in the comments.
