Deep learning for deformable medical image registration

Image registration in medical diagnostics. Large Deformation Diffeomorphic Metric Mapping.

In general, the problem of image registration is posed as searching for a mapping for a pair of images that will transform the initial image into the final one. Registration can be divided into two classes: rigid and deformable. Rigid registration methods consider image transformations as a whole, that is, they only include rotations and shifts of images. Unlike rigid registration, deformable registration makes it possible to describe nonlinear local nonlinear changes in the image.

Let's consider everything described above from a mathematical point of view: let us be given the initial and final images having ${n\times m\times p}$ voxels. Deformable image registration can be described as searching for a mapping that minimizes the following loss functional:

$\hat\phi =\underset{\phi\ \in\ Diff\ R^{n\times m \times p}}{\operatorname{argmin}} (I_m \circ \phi, I_f),$

Whereis the initial image, and– finite (in foreign literature moving and fixed, respectively).

In other words, we want to obtain a mapping from the class of diffeomorphisms in which the difference between the transformed initial and final images is minimal. The diffeomorphicity of the transformation is necessary for the resulting mapping to be invertible. However, how can one check that a mapping is diffeomorphic on a computer that can only process discrete variables?

This problem is solved using the Large Deformation Diffeomorphic Metric Mapping (LDDMM) method. To describe the idea briefly, without going into complex mathematical details, we must introduce a grid onto the images and calculate the Jacobian of the mapping at each point in the image. If the Jacobian at each point of the grid is positive, then the resulting mapping will be considered diffeomorphic. A more detailed description of the LDDMM technique is presented Here.

Generally speaking, image registration can be carried out not only for a pair of images, but also for a time series of images. In this case, the resulting display will be interpolated in intermediate images.

So, the theory is over, it's time to move on to practice!

Application of deformable image registration in medicine.

Deformable image registration has a wide range of applications. Image sources for deformable registration can be images from computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, fluorography, etc. Here are some examples of the use of deformable image registration in medicine:

Oncology: Deformable registration is used to track changes in the size and shape of tumors over time, which is critical for treatment planning and evaluation of treatment effectiveness. Example article: Deformable image registration in radiation therapy.
Cardiology: Strain registration is used to analyze the movement and deformation of the heart muscle, which helps in the diagnosis and treatment of cardiovascular diseases. Example article: Deformable Image Registration Using Vision Transformers for Cardiac Motion Estimation from Cine Cardiac MRI Images.
Orthopedics: In orthopedics, deformation registration can be used to compare images of joints before and after surgery, which helps in evaluating the results of surgery. Example article: “Model-based registration for assessment of spinal deformities in idiopathic scoliosis“.
Brain Research: Deformable recording helps in brain research by allowing brain scans of different people to be compared to study anatomical differences and activity patterns Article example: “A reproducible evaluation of ANTs similarity metric performance in brain image registration“.

Deformable registration for reconstruction of vascular behavior.

Classical approaches to deformable image registration.

Classical methods of deformable image registration optimize the loss functional using gradient descent methods and also check the diffeomorphicity of the resulting mapping. The most popular deformable registration algorithms are presented in software packages ANTs And greedy. Follow the links to find a detailed description of these methods with examples of their use.

The advantage of these methods is that they are quite simple to run, and they also provide the opportunity to improve registration accuracy by taking into account the marking of anatomical regions using class masks. The code for obtaining a mapping for deformable registration is as follows (the solution to the deformable registration problem using greedy is shown as an example):

greedy -d 3 \ 
-i path-to-fixed-image.nii.gz path-to-moving-image.nii.gz \
-gm path-to-fixed-segmentation.nii.gz \
-mm path-to-moving-segmentation.nii.gz \
-o path-to-warp.nii.gz \ 
-sv -n 100x50x10 -m NCC 4x4x4

Now apply the resulting mapping to the initial image:

greedy -d 3 \
-rf path-to-fixed-image.nii.gz \
-rm path-to-moving-image.nii.gz path-to-warped-image.nii.gz \
-ri LABEL 0.2vox \
-rm path-to-moving-segmentation.nii.gz path-to-warped-segmentation.nii.gz \
-r path-to-warp.nii.gz

However, all classical approaches have disadvantages. First of all, they are not universal. It is necessary to select parameters for launch each time. The second drawback is even more significant: we are looking for a mapping in the class of diffeomorphisms, which is quite extensive. This means that the resulting displays may incorrectly reflect real physical processes occurring in the observation area over a long period of time.

Neural network approaches to image registration. VoxelMorph and its modifications.

The solution to the previously stated shortcomings of classical approaches is the use of neural networks. Recently, a neural network has become widespread for solving problems of deformable registration of medical images. VoxelMorph.

VoxelMorph is based on a convolutional neural network similar to U-Net. It receives two images and class masks (optional) as input, and the output is a mapping that transforms the initial image into the final one.

VoxelMorph architecture using deformable brain recording as an example.

In addition to the generality advantage that neural networks provide, there is the advantage that we can modify the loss function by adding regularizers to it to obtain a physically based mapping.

First, let’s add a condition for the smoothness of the mapping and take into account the accuracy of the transformation of class masks:

$\hat\phi =\underset{\phi\ \in\ Diff\ R^{n\times m \times p}}{\operatorname{argmin}} (I_m \circ \phi, I_f) + \lambda\|\ nabla \phi\|_2+ \gamma(S_m \circ \phi, S_f)$

HyperMorph

Now it is important to ask the question: how to select hyperparameters $\lambda$ And $\gamma$ with regularizers? On the one hand, it is possible to train several models with different hyperparameter values, but this process takes a lot of time and resources. To solve this problem, a modification of VoxelMorph was developed – HyperMorph.

HyperMorph is a fully connected neural network add-on $h_\theta$ over the main VoxelMorph network. In addition to images, HyperMorph takes as input randomly generated values of hyperparameters in the form of a set and calculates the values of all weights UNet core network $g_\theta$ , which makes up the original VoxelMorph architecture. Thus, all trainable weights are only online not in $g_\theta$ .

At each iteration of HyperMorph training, input data is generated, which, in the case of training on labeled images, contains , ,, as well as the values of the hyperparameters $\Lambda = \{{\lambda', \gamma'}\}$ obtained as random numbers from a uniform distribution . The model prediction error function from Eq. is modified as follows:

$L_{hyp} (I_m, I_f , \phi, Λ) = (1−\lambda')(1−\gamma' )L_{sim}(I_m \circ \phi, I_f)+\lambda' L_{smooth} (\phi)+\\+ \gamma' L_{seg}(S_m\circ\phi, S_f )$

Factor $(1−\lambda')(1−\gamma' )$ before the first term allows you to cover all possible weighting relationships between each error term using parameter values ${\lambda',\gamma'}$ from 0 to 1.

As noted above, in addition to the smoothness of the display and the similarity between the transformed initial and final images, it is important to obtain displays that will reflect the physical processes occurring in the observation area over time. To obtain physically based transformations, you can use regularizers, which are based on a physical model that describes the mechanics of the medium. In particular, in the article Learning Physics-Inspired Regularization for Medical Image
Registration with Hypernetworks A model of an elastic medium was proposed for the problem of deformable registration of lung images. At the same time, the hyperparameters obtained during the HyperMorph training process do not have an abstract, but a physical meaning! This means that in this way it is possible to estimate the parameters of the environment within the limits of applicability of the physical model used.

VoxelMorph++

Another modification of VoxelMorph is the architecture VoxelMorph++, which allows you to improve the quality of deformable registration by taking into account key points in regions of interest. This architecture replaces the head of the VoxelMorph network to focus on the displacement of corresponding key points between images, which improves the accuracy of capturing large deformations between images.

This approach allowed the authors of the article to improve the quality of deformable image registration for lung images. In addition, it is assumed that VoxelMorph++ can be used to obtain more accurate images when dealing with images of soft tissues of the body.

Conclusion.

Deformable image registration plays an important role in modern medical diagnostics, allowing accurate tracking of changes over time and providing high accuracy in treatment planning and evaluation of its effectiveness.

Progress in registration methods: With the development of deep learning methods such as VoxelMorph and its modifications, new opportunities are opening up to improve the quality and accuracy of deformable registration. These methods offer more universal and adaptive solutions compared to classical approaches.

Physical basis of registration: The introduction of physically based regularizers into the training process of neural networks allows us to obtain displays that better reflect the real physical processes occurring in tissues, which is a significant progress in the field of medical imaging.

Development prospects: Continuous improvement of algorithms and neural network architectures, such as HyperMorph and VoxelMorph++, opens new horizons for research and clinical application of deformable registration, making it more accurate and efficient.

Overall, deformable image registration using deep learning techniques is a powerful tool that will continue to evolve and have a significant impact on medical diagnosis and treatment.

The article was written with the support of the Master's program “Applied Machine Learning and Big Data” of NSU.