A Look at Augmented Reality in Surgery

Hello, reader!

I recently had the chance to talk to innovators from the world of augmented reality (AR). This is a research and practical team that conducts research and training on the use of AR technologies in surgical operations. You didn’t think that augmented reality is just a simulation in games, did you? Anyway, under the cut — to an interview with the developers and the history of combining advanced visual technologies with surgical instruments.

At a meeting on the pet project, colleagues mentioned that doctors they knew had augmented reality glasses and used them during surgeries. The technologies themselves, individually — surgery, tracking objects in space, augmented reality applications — have existed for quite a long time. However, combining them into one product requires a lot of work. I contacted the system developers to get acquainted with the Medgital project, try interacting with augmented reality, and share information.

To demonstrate the operation of the devices and programs, colleagues allowed me to take pictures in the office with previously used models and mock-ups. I am posting the photos and videos with their approval and consent.

First 100 operations

The developer of the augmented reality navigation system, Ildar Rafaelevich Mamaev, answers the questions.

Tell us how the project began?

The history of the project begins at the Polytechnic University of St. Petersburg in the Department of Information Technology. On its basis there was a laboratory of computer graphics visualization. Initially, it focused on research projects, and since 2015 it has switched to medical ones.

The laboratory specialists developed a simulator for open surgery at the request of one of the clinics. Then, in 2015, their cooperation began. The results of the first project were demonstrated at exhibitions, we received several awards. The events received a great response from those interested in developments using innovative methods of computer graphics and visualization.

The research team discovered the great potential of visual technologies. They established new contacts with clinics and constantly tried to apply various technologies in medicine. They worked out various devices and consulted with doctors to find optimal niches and potentially interesting projects. During the research, they developed technologies for reconstructing and building a three-dimensional model of the heart.

How did you get from visualization to mixed reality?

In 2018, we purchased Microsoft Hololens augmented reality glasses. The idea immediately arose to visualize the anatomy in these glasses as a 3D model and apply the method of matching it with the patient. Within a year, we created the first prototype and published several articles on this topic. However, the project was somewhat slowed down due to the technical limitations of the glasses and the difficulties associated with tracking and positioning.

When a new generation of devices appeared, we realized that optimized positioning technology could be used in surgeries. In late 2020 and early 2021, we were able to interest the First Medical University and the Almazov Clinic in performing surgeries using the new technology. Initially, a crude version of the positioning system was used, so it had to be adjusted for each patient individually.

What was the result after the first operations with the prototype system?

The success of these operations prompted the laboratory to new research and development. And for this, money was needed. In the summer of 2021, two investors became interested and financed the creation of the MVP. Over the next year, the system was finalized, making it adaptable for different patients without the need for individual production of elements. Already in 2022, about 20 operations were performed in the field of dentistry and maxillofacial surgery. That same year, they received a grant from the Bortnik Foundation and began registering a medical device. Later, they became residents of Skolkovo.

In 2023, the number of operations performed using our system was approaching 100. As of July 2024, we work with twelve different clinics in St. Petersburg, Moscow and other regions.

In addition to entertainment, these technologies are also used in Marketing, for educational purposes, and, more recently, in medicine.

Training in working with the system

What tasks do you perform as a developer in a project?

First of all, this is the development of new features that scale our project in other directions. I also support the software that is used in augmented reality glasses. My responsibilities include preparing training materials, training surgeons and other clients who purchase our software and equipment. I periodically go on business trips, participate in exhibitions. In the future, I plan to speak at conferences.

Is special training required for development?

I studied to be a physicist, then retrained to be a programmer. I am not a doctor, although the necessary minimum of knowledge, for example in the field of anatomy, is inevitably acquired in the process of work.

Doctor of Medical Sciences, Professor Mikhail Sergeevich Korzhuk answers questions.

What experience do specialists need to work with glasses?

Augmented reality is a rather complex tool that requires certain skills from the surgeon. The advantages of its use: choosing the best options for separating and connecting tissues and organs, reducing the risk of complications, reducing the time of the operation. But there are also disadvantages: a large amount of preparatory work and the unfamiliarity of operating in AR glasses. Not all, but some of the disadvantages can be overcome using remote online work.

The staff using the glasses are mostly experienced surgeons who need to get advanced surgical information, prevent and eliminate potential complications. They highly value the technology. Young specialists see the system as nothing more than a spectacle.

How is the system used in education?

We have accumulated a database of 3D models of patients. With their permission, we use it in teaching medical students to illustrate both the normal anatomical picture and the corresponding lesions. We demonstrate the models through AR glasses or on a monitor. You can think of it as a virtual anatomical atlas. We also print out some elements of the 3D models from plastic for educational purposes.

Technology in action

Colleagues provided an opportunity to test adding a model and combining it with equipment. It felt like Tony Stark in Iron Man throwing blueprints around in space. Unfortunately, the recording differs from reality and has delays and minor inconsistencies.

What is the application of augmented reality technologies in medicine?

The essence of AR is to simultaneously provide the user with information from two sources: from his senses (perception of the real world) and from the computer. If information from the virtual world is used to influence a real object, the technology is called “mixed reality”. It uses DICOM files of CT or MRI as initial data. The 3D-Slicer program visualizes, edits and saves information about the patient and the disease in the format of a 3D model. In some cases, we also save data about the stages of the operation.

How do you prepare for surgery?

The first stage — preparation for surgery — includes performing CT or MRI. Then a team of specialists of different profiles (surgeon, oncologist, traumatologist, radiologist) forms a 3D model. This part of the work is called segmentation. Particular attention is paid to the key features of the patient and the affected organs. Then the council evaluates the detail and quality of the model, sometimes recommending improvements.

The model can be used as a personal anatomical atlas of the patient, a surgical navigation map with marking of zones and stages of increased surgical risk, as well as for the production of medical equipment. In the latter case, three-dimensional models of medical products are prepared and transferred to 3D printing. For example, a plastic dock splint created for a specific patient provides the stable position of the limb necessary for manipulations. Depending on how the 3D model is planned to be used, a device for its display is selected. This can be a monitor, a projector, AR glasses.

As an example, here is a clinical case of one of the patients with a foreign body in the foot.

The navigation system, which includes augmented reality glasses, allows you to link the 3D model to the patient. To do this, the model is modified using Medgital Studio software, developed using the Unity graphics engine. The program windows mark the axes of comparison, the axes for inserting the instrument, and the reference points of the model. At least three static points are marked to fix the model marker. The marker is included in the navigation system kit and has a QR code for binding.

Next, the original DICOM files and 3D model files are loaded into MegitalVision. The application configures the color display and transparency functions of image elements. 3D models of axes, reference points, foreign bodies and additional layers, such as the skin surface, are loaded. The models are aligned with the scan. Each added model has its own color. The final step is to add QR code attachment points. The files are then automatically transferred to the glasses memory.

The final stage is the operation. During preparation and after anesthesia, the patient is put in the position in which the CT scan was performed. The limb can be fixed using equipment. A plate with a QR code is attached to previously marked places. The operating doctor, with the help of an assistant, puts on glasses and selects the patient model file in the virtual menu. The QR code is used to automatically align the model with the patient's body. Then comes the manipulation itself. Axes, reference points, and organ images are used as reference points for the stages of the operation.

You mentioned that 3D printing is used. How common is it in surgery? I know that the technology is most widely used in dentistry.

Many authors of scientific papers use printed models of the patient's organs for better orientation in the anatomical picture. We do not particularly need this, since it is possible to study the model in augmented reality glasses. But for the production of personal medical products, 3D printing is a good option. In the example above, the splint contained guide channels for rods indicating the location of the foreign body.

Plans for the promotion of the system and its use

Ildar Mamaev:

We have demonstrated our system at international exhibitions, including a presentation in Bahrain, the Ministry of Health in Dubai and Turkey. We see great interest from potential customers. At the moment, we have a registration certificate for a medical device. The solution created is unique on the Russian market and our system is actively used in the country's leading clinics.

Mikhail Korzhuk:

Augmented reality is used in a number of medical organizations in St. Petersburg and Moscow. The number of patients treated with its help is in the hundreds. Most of them have been successfully operated. The main vector of our work at present is to simplify the technology for the end user-doctor, to form a professional community for the use of augmented reality technology in surgery. We have created a course for advanced training of doctors in this area. Thus, we are creating the foundation for the widespread use of this complex, but extremely useful for the health of patients section of IT.