how the automation of microscopy is developing in Russia and in the world

New microscopy techniques such as live cell imaging, slide scanning, high contrast screening, and 3D electron microscopy are generating huge amounts of data.

To process them, often in real-time mode, specific systems are needed that support automation at all levels of work. In the sequel, we will talk about how microscopes became smart and saved a person from routine tasks – and not only in medicine.

How it all began

At the beginning of the 17th century, Galileo Galilei, manipulating convex and concave lenses, noticed that not only distant, but also close objects of small size can be viewed in a telescope in an enlarged view. Already in 1620, Cornelius Drebbel invented the first compound microscope. Half a century later, in the 1670s, Anthony van Leeuwenhoek began experimenting with very high magnification single-lens microscopes, and he designed them himself. The main element in his microscopes were specially polished lenses.

More than three centuries later, microscopy has become a vast field of application in many fields, from industry to medicine. Nowadays, microscopes are becoming more and more “smart”, it is no longer just an optical, but a computer-optical system. The rise of automation, the paradigm shift towards Industry 4.0 has played an important role in transforming the microscope design: equipment manufacturers are forced to quickly innovate using increasingly intelligent systems in order to maintain a competitive advantage.

Why microscopes are important in industry and how to make them smart

Digital microscopes, developed back in the mid-1980s, are still popular today for medical research. They are also used for general control and quality assurance of products on industrial lines. Digital microscopy has already turned optical microscopes into digital systems that support a wide range of functions, from sharing images to analyzing and measuring objects.

The capabilities of different digital optical systems depend on the industry where they are planned to be used. So the manufacturer of ultra-high definition digital control systems INSPECTIS introduced 4K video recording into their microscopes. The ability to track and record the entire surveillance process, including in order to ensure safety, is in demand in the pharmaceutical industry and in the development of medical technology.

Another typical application of digital microscopes, but already in the electronic business, is the automated optical quality control of the printed circuit board – AOI. If the AOI detects a malfunction, the system also identifies the cause of the incident. But despite this, the operator’s opinion is still required: only a person is still able to understand whether the malfunction in the board is associated with an incorrect temperature regime or a poor-quality soldering process. AI plays the role of an assistant here.

Microscopes to reconstruct surfaces and spot imperfections

Three-dimensional optical microscopes that appeared in the 1980s, including profilometers for measuring microroughness on precision surfaces, continue to develop today. Bruker, a manufacturer of scientific instruments, is one of the industry leaders in this area: in 2018, the company acquired Alicona, a provider of optical metrology solutions.

Precisely Alicona has developed new technology for three-dimensional optical microscopes. This is a focus variation that calculates a sharpened image and measures the depth of unevenness using optics with a very limited depth of field. For example, the Contour LS-K 3D optical profiler enables high-resolution imaging, providing the researcher with quantifiable data.

This is important for OEMs who require higher frame rates and higher bandwidth measurements for improved accuracy and quality control. This is where automation and self-adapting systems come into play, which have self-adapting algorithms built into them. The system takes measurements at the surface and then, based on the criteria it has for analyzing frequencies and amplitudes, decides which algorithm is best to use to recreate the surface topography.

Engineers are forcing a change in the approach to microscopy

Smart data management has become part of microscopy, and companies such as ZEISS are developing in this direction. The manufacturer enhances the intelligence of industrial microscope systems to obtain the best results regardless of human factors, i.e. operator. This is essential for modern QC assurance where data performance and reliability are key. ZEISS is also confident that smart algorithms will not take the place of the operator yet. Instead, people will begin to use automated systems more flexibly.

Digital microscopist: what smart systems do in medicine

The machine learning that microscope manufacturers use today for image segmentation is not limited to industrial applications such as failure analysis and quality control. These technologies are also used in medicine, where they have already become an important part of the automation of the processing of laboratory analyzes, the creation of data sets and the release of medical personnel from routine processes.

The tasks of a modern microscopist include not only counting certain cells on a sample taken from a patient, but also a whole range of issues that require attentiveness and perseverance. First of all, this is the correct determination of cell types, a preliminary interpretation of the results and the transfer of data to a medical specialist, whose competence already includes the diagnosis and further treatment of the patient.

Smart technology from Celly.AI, which is based on computer vision and machine learning, decides these tasks. The doctor has only control and solution of extraordinary problems associated with anomalies. The fact is that it is still difficult to train AI systems to detect all anomalies. Nevertheless, this can still be done – the algorithm will simply add an unusual case to its dataset for training and will take this case into account in the future. The primary data is marked up by a human physician.

By analyzing images using convolutional neural networks, the system automatically determines the types of tissue cells, their number, and actually performs all of his daily tasks for the microscopist. To simplify the introduction of innovations in such a conservative industry as medicine, the company proposed a rather elegant solution – an iPhone is connected to the microscope eyepiece using an adapter lens. iOS application using neural networks analyzes the picture of the smear in real time. The research results are automatically uploaded to the cloud service, which allows you to instantly share data with colleagues, request their advice and ensure the availability of medical services for remote geographic locations.

There are other useful developments in this area. So, researchers from Japan have developed an automated computer program that can accurately and reproducibly count the number of micronuclei of tissue cells in stained images. Micronuclei are small nuclear structures that are markers for pathologies such as cancer. The model, called CAMDi (Calculating Automatic Micronuclei Distinction), is capable of counting microkernels despite their relatively small size.

Legacy automated systems have traditionally used images from only one tissue level. To understand why this is important, imagine that a ball, anchored in space, is cut in cross section. If you cut it closer to the top or bottom, the cross-sectional size will be much smaller than if you chose the slice closer to the center, so when the cross-section is made close to the periphery of the ball, the nucleus can easily be mistaken for a microkernel. To solve this problem, researchers from the University of Tsukuba made photographs at different levels and created a program capable of analyzing the received three-dimensional information.

A collaborative team of researchers from Oxford and the University of Warwick developeda a method to better understand and evaluate the pleomorphism of viruses. It was being developed amid a pandemic to aid in coronavirus research.

Unfortunately, electron microscopy is still too expensive and slow for large-scale use in such studies, so scientists have created a technique for high-performance imaging of filamentous virions, combining direct stochastic optical reconstruction microscopy (dSTORM). This is a method with a resolution of less than 20 nm. In addition, the researchers have also developed software for rapid automated analysis that allows the identification and analysis of thousands of virions.

The main advantages of neural network algorithms are that they can work endlessly, analyze thousands of images and at the same time self-learn. This excludes the human factor, which can be associated with both incompetence and the usual fatigue or inattention of the microscopist. But tasks that are not part of the routine cannot be completely devoted to developing AI. In this case, the functionality of the digital microscopy ecosystem makes working with them as convenient and efficient as possible: remote access to information and its processing and presentation of data in an easy-to-understand visual form make smart systems an indispensable assistant for a microscopist.