Deformable medical image registration based on CNN

Abstract

Objective:

This study aims to develop and test a new image registration method in which full-scale skip connections in the encoding process is added into the registration network, which can predict the deformation field more accurately by retaining more features and information in the decoding process.

Methods:

Two improved registration networks are connected in series in the registration framework. Each registration network uses the unsupervised learning registration method to predict a small deformation field, and the last two small deformation fields are superimposed to obtain the final deformation field. The model is evaluated by the oasis datasets (brain T1-weighted MRI images), one image is selected as the fixed image, while 383 images are used as training images and 30 images are used as test images. Wavelet decomposition and reconstruction are also used to enhance the image.

Results:

Compared with the affine method, the voxelmorph-1 method and the voxelmorph-2 method, applying the new registration network was proposed by this study improves the registration accuracy by 28.6%, 1.2% and 0.2%, respectively.

Conclusion:

The experimental results demonstrate that the method proposed in this study can improve the accuracy of image registration effectively.

Keywords

Image registration wavelet decomposition and reconstruction LDIRnet CNN

1 Introduction

Medical imaging technology can obtain information about patients’ condition without trauma, which is the most powerful auxiliary means of treatment. Medical image registration is an important part of medical image processing, and there are many types of medical image registration. For example, according to the dimension of the image, medical image registration can be divided into three categories, namely 2D and 2D image registration, 3D and 3D image registration and 2D and 3D image registration, which also can be classified by the subjects, the way that the image is obtained, and the types of spatial transformation. Medical image registration be used for lesion detection, clinical diagnosis, surgical planning, surgical navigation, efficacy evaluation, etc. It has important implications for the development of clinical medicine.

The traditional image’s registration method seeks the optimal spatial transformation between two images by iterating and optimizing each pair of data. Traditional image registration includes feature-based registration and gray-based registration and so on. For example, the feature-based image registration is to extract the key feature points between the fixed image and the moving image, and it seeks the geometric transformation relationship between the two images by the relationship between the feature points of the two images. The disadvantages are that extracting features is difficult and image registration accuracy is not high. Common feature-based image registration methods include SIFT [1], SURF [2], Harris [3] and so on. The grayscale-based method is based on the grayscale values of two images, uses a search method to find the maximum or minimum similarity of the grayscale values of the two images, and determines the transformation parameters between the fixed image and the moving image. This method is simple to implement and does not require complex preprocessing for fixed and moving images. The disadvantage is that the amount of computation is large and cannot be directly used to correct nonlinear transformations of images. It mainly includes cross-correlation method [4], maximum mutual information method [5], maximum likelihood matching method [6] and Demons [7], etc.

The registration method of supervised learning requires the real deformation field as the gold standard, which is marked manually. The Affine Image Registration Network (AIRNet) was proposed by Chee et al. [8] This network can estimate the spatial transformation parameters of two 3D brain MRI images by the supervised learning method. The method uses the MSE function to train the network. The results show that the performance of this registration method is better than the iterative algorithm. The CNN network model was proposed by Sloan et al. [9] The performance of this registration method is better than the FCN method in the registration of unimodal and multimodal T1- and T2-weighted brain MRI images. The real deformation field not only is difficult to obtain but also cannot be guaranteed in quality.

The registration method of unsupervised learning overcomes this shortcoming. This method directly uses the warped image to calculate the loss by adding the spatial transformer network (STN network) [10] into the registration network, thereby an end-to-end registration method is realized. A first end-to-end unsupervised Deformable Image Registration Network (DIRNet) was proposed by Devos et al. [11] The network used the MNIST and SCD datasets for model evaluation. The experimental results show that the registration effect is better than the supervised learning method. An unsupervised registration method based on CNN was proposed by Balakrishnan et al. [12]. The structure of network is similar to U-Net and it was named VoxelMorph, which is a commonly used deep learning image registration method. A deep recursive cascaded network structure was proposed by Zhao et al. [13]. Multiple networks are connected in series in model. The model gradually forms a deformation field and improves the registration accuracy. A large deformable image registration network (LDIRnet) model based on multiple deep convolutional neural networks are connected in series, which was proposed by Wei et al [14]. The method is two or three registration networks in series. The small deformation field is obtained by each registration network, and the final large deformation field is obtained by voxel summation of the small deformation field. In this study, the registration network of LDIRnet is improved by adding full-scale skip connections, in order to retain more image information. The paper also enhances the image to improve the accuracy of image registration effectively. A cycle-consistent deformable image registration was proposed by Kim et al. [15]. The registration model contains two registration networks (G_X, G_Y). The deformation field of X to Y is obtained by the registration network G_X, and the deformation field of Y to X is obtained by G_Y. The registration accuracy is improved by matching the moving image and the fixed image with each other. An image registration method that introduces amortized hyperparameter learning was proposed by Andrew et al. [16] This method adds a hypernetwork to the registration model, which takes in an input hyperparameter and modulates a registration network to produce the optimal deformation field for that hyperparameter value.

Few medical image samples and scarcity of annotations have always been two major problems in medical image processing. For supervised learning, the accuracy of registration is determined by the quality of annotations. Unsupervised learning networks require a large amount of data for training, but there are few samples of medical images, so how to use less data for network training is also an issue that needs to be further studied.

The LDIRnet model will lose a lot of information. In order to retain more image information, the registration network of the LDIRnet model is improved in this study. Full-scale skip connections are added to the registration network, so that the deformation field is more accurately predicted by more information. The detailed information of the training images in this study is enhanced by wavelet decomposition and reconstruction, thereby improving the registration accuracy.

2 Materials and methods

2.1 A registration network with full-scale jump connections is added

Full-scale skip connections in the encoding process is added into the registration network in the paper, which likes UNet, and fine-grained semantics and coarse-grained semantics are added in the decoding process to predict the deformation field more accurately. The registration network structure is shown in Fig. 1. The down-sampling operation in the encoding process and the convolution operation in the decoding process are implemented by a convolutional layer with a convolution kernel of 3×3and a stride of 2, a LeakReLu layer with a parameter of 0.2. The up-sampling operation in the process is implemented by a deconvolution layer with a convolution kernel of 3×3 and a stride of 2, a LeakReLu layer with a parameter of 0.2. In the skip connection, the maximum pooling is used to reduce the size of the feature map to solve the problem of different sizes of the feature images, thus, to realize the aggregation of features. The image that realized the aggregation of features is then subjected to convolution operations to gradually form a deformation field.

Fig. 1

Structural diagram of the improved registration network.

2.2 Registration framework

This study adopts the LDIRnet registration framework, two improved registration networks are connected in series. Each registration network learns a small deformation field, and the final large deformation field is obtained by summation of the two small deformation fields. The registration network framework is shown in Fig. 2.

Fig. 2

Model structure diagram.

First, the deformation field φ₁ is obtained by the first registration network, and then the fixed image and the image that are obtained by the first registration network are the input of the second registration network. The deformation field φ₂ is obtained by the second registration network, and the final large deformation field φ is obtained by summation of the two deformation fields.

2.3 Loss function

The loss function of registration method of the unsupervised learning method is divided into two parts, the first part is the similarity measure between the registered image and the fixed image, and the other part is the regularization loss added to avoid the discontinuity of the deformation field. The loss function is computed using Equation (1). $L (F, M, φ) = L_{sim} (F, M (φ)) + λ L_{smooth} (φ)$ (1) where F is the fixed image, M is the moving image, φ is the deformation field, M (φ) represents M is warped by φ, L_sim (F, M (φ)) measures the image similarity between F and M (φ), L_smooth (φ) is the regularization loss, and λ is the regularization trade-off parameter.

In this study, the similarity measure of the fixed image and the warped image selects a negative local cross-correlation, the calculation window is 9 × 9, and the local cross-correlation is computed using Equation (2) as below. $\begin{matrix} L_{sim} (F, M (φ)) = - CC (F, M (φ)) = \\ - \sum_{p \in Ω} [\frac{{(\sum_{q \in N (p)} (F_{q} - {\bar{F}}_{q}) (M_{φ (q)} - {\bar{M}}_{φ (q)}))}^{2}}{({\sum_{q \in N (p)} (F_{q} - {\bar{F}}_{q})}^{2}) (\sum_{q \in N (p)} {(M_{φ (q)} - {\bar{M}}_{φ (q)})}^{2}) + ɛ}] \end{matrix}$ (2) where Ω represents the entire area, N (p) is the selected local area, F_q and M_φ(q) are the gray value of F and M (φ) at the q point in the area N (p), ${\bar{F}}_{q}$ and ${\bar{M}}_{φ (q)}$ are the mean values in area N (p), and ɛ is a very small number, in order to avoid the preblem that the denominator is 0.

The purpose of training the model is to train a registered image that is most similar to the fixed image, so that the L_sim (F, M (φ)) function is as small as possible, but the deformation field that may be obtained is discontinuous, so a regularization loss is added to smooth the deformation field. In this study, the l₂ norm is selected as the regularization loss. Equation (3) is the computation formula. $L_{smooth} (φ) = \sum_{p \in Ω} {∥ \nabla φ (p) ∥}^{2}$ (3) where, ∇φ (p) is the gradient of φ.

2.4 Datasets

2.4.1 Datasets preprocessing

The experimental data in this study comes from the oasis [17] datasets in the experimental data of HyperMorph. The oasis datasets includes 3D brain T1-weighted MRI images of 414 subjects. The data that was used in this article is the slice of preprocessing 3D brain T1 weighted MRI image. The preprocessing operation is as follows: first, FreeSurfer [18] was used to obtain the original data, and the original data was resampled into a grid of 256 × 256 × 256 with a voxel spacing of 1mm × 1mm × 1mm. Then, the obtained origina data were removed from the skull, the data was normalized, and the data was scaled to the [0, 1] interval. Finally, the data used affine transform and cropped the image size to 160 × 192 × 224.

2.4.2 Image enhancement

In this study, the image is enhanced by wavelet decomposition and reconstruction. The process is shown in Fig. 3. Wavelet decomposition of the image uses the wavelet basis to decompose the image into four images, which are horizontal low frequency (LL), vertical low frequency respectively (LH), horizontal high frequency (HL) and vertical high frequency (HH). The four images contain different information, and the three images that are LH, HL and HH contain the detailed information of the original image. Thus, in this study these three images multiplied β (greater than 1) parameter, and then the training data was reconstructed by LL, β LH, β HL and β HH, in order to enhance details. In this study, the Biorthogonal wavelet function was used as wavelet base function in wavelet decomposition and reconstruction. This study used one image for the fixed data, while 383 images for model training and 30 images for validation.

Fig. 3

Image enhancement.

2.5 Implement

The experiments in this study used Windows10 system with GPU, the programming language was Python, and the model was constructed by the pyTorch. The torch version is 1.8.0 + cu111. During training, batch_size was 1 that a set of data was used for training each times, the model was trained for 15,000 epochs and using Adam optimization with learning rate 1 × 10^-4. This study sets parameter as λ = 1.0, β = 1.1.

2.6 Evaluation method

In this study, the registration performance is evaluated by segmentation labels of brain anatomy. The dice similarity coefficient (DSC) is chosen as the evaluation method. Equation (4) is used to compute the DSC coefficient. $DSC (A^{i}, B^{i}) = 2 * \frac{| A^{i} \cap B^{i} |}{| A^{i} | + | B^{i} |}$ (4) where Aⁱ and Bⁱ represent the set of i structures in the fixed image F and the warped image MI (φ). Respectively, the DSC coefficient reflects the degree of overlap between the two images. The closer the DSC is to 1, the better the registration effect.

3 Results

In this study, 23 anatomical structures that were segmented by FreeSurfer are selected to calculate the DSC coefficients, and the 23 anatomical structures are shown in Table 1.

Table 1
Anatomy table

Index Name

1 Left-Cerebral-White-Matter

2 Left-Cerebral-Cortex

3 Left-Lateral-Ventricle

4 Left-Inf-Lat-Ventricle

5 Left-Thalamus

6 Left-Caudate

7 Left-Putamen

8 Left-Pallidum

9 3rd-Ventricle

10 Brain-Stem

11 Left-Hippocampus

12 Left-Ventral-DC

13 Left-Choroid-Plexus

14 Right-Cerebral-White-Matter

15 Right-Cerebral-Cortex

16 Right-Lateral-Ventricle

17 Right-Inf-Lat-Ventricle

18 Right-Thalamus

19 Right-Caudate

20 Right-Putamen

21 Right-Hippocampus

22 Right-Ventral-DC

23 Right-Choroid-Plexus

Index	Name
1	Left-Cerebral-White-Matter
2	Left-Cerebral-Cortex
3	Left-Lateral-Ventricle
4	Left-Inf-Lat-Ventricle
5	Left-Thalamus
6	Left-Caudate
7	Left-Putamen
8	Left-Pallidum
9	3rd-Ventricle
10	Brain-Stem
11	Left-Hippocampus
12	Left-Ventral-DC
13	Left-Choroid-Plexus
14	Right-Cerebral-White-Matter
15	Right-Cerebral-Cortex
16	Right-Lateral-Ventricle
17	Right-Inf-Lat-Ventricle
18	Right-Thalamus
19	Right-Caudate
20	Right-Putamen
21	Right-Hippocampus
22	Right-Ventral-DC
23	Right-Choroid-Plexus

In this study, the registration network framework was trained, and the experimental results were compared with affine transformation and voxelmorph models. The registration effect was quantified by the average DSC coefficients of 23 brain anatomical structures, and the experimental results are shown in Table 2. It can be seen from the table that the registration accuracy of the improved registration model is better than the Affine transform method and the voxelmorph-1 method, and which is comparable to the voxelmorph-2 method.

Table 2

Comparison table of experimental results

Method	Average DSC	Average standard deviation	p-value
Affine	0.457	0.153	–
voxelmorph-1	0.731	0.151	0.00000966
voxelmorph-2	0.741	0.151	0.292
This article	0.743	0.150	–

The following figure shows the average DSC coefficient of each brain anatomy. As can be seen from Fig. 4, for each anatomy, the registration effect of method in this study is better than the Affine transform. Compared with the voxelmorph-1 method, except for Left-Choroid-Plexus the registration effect of the other brain structures is better than or comparable to the voxelmorph-1 method. Compared with the voxelmorph-2 method, except for the three brain structures that are Left-Inf-Lat-Ventricle, Left-Putamen and Left-Choroid-Plexus, the registration effect of other brain structures in this study is better than the voxelmorph-2 method or comparable to the voxelmorph-2 method. In this study, the experimental results were hypothesis tested by paired t-test, and p = 9.66 × 10^-6 was obtained between the paper model and the voxelmorph-1 model, and the p = 0.292 was obtained between the voxelmorph-2 model.

Fig. 4

Comparison of anatomical structures.

Figure 5 shows the visualization of the partial registration results of the above registration method, including the fixed image, the moving image, warped images which were obtain by the voxelmorph-1 method, vocelmorph-2 method and the method in this study and the deformation fields that were obtained by the method, respectively.

Fig. 5

The registration results.

4 Discussion and conclusion

In this study, the LDIRnet model is improved by adding the full-scale jump connection of the encoding process into the registration network, and more image information is retained by adding fine-grained semantics and coarse-grained semantics in the decoding process. The deformation field is more accurately predicted by retaining more image information. The overall framework is a series of two registration networks, and the final transformation field is the sum of the transformation fields that are predicted by the two registration networks. In this study, image enhancement is carried out before the model training, and the detailed information of the image is enhanced by wavelet decomposition and reconstruction. During training, more features are learned. Experimental results show that compared with the affine method, the voxelmorph-1 method and the voxelmorph-2 method, average DSC coefficient is improved by 28.6%, 1.2% and 0.2%. It shows that the registration effect of the registration model in this study is better. Medical image registration is the basis of disease diagnosis, medical image segmentation, etc. So, accuracy such as disease diagnosis or image segmentation is improved when the model in this study is applied to these aspects. The method proposed in this study can improve the accuracy of image registration effectively. The Unsupervised image registration method does not require the real deformation field as the gold standard, which saves the cost of labor as well as ensures the quality of the registration. Thus, it has potential to be a new development direction of the future registration field.

Footnotes

Acknowledgements

This work was supported in part by National Natural Science Foundation of China (No. 42274173).

References

Lowe

D.G.

, Distinctive image features from scale-invariant key-points, International Journal of Computer Vision 60(2) (2004), 91–110.

Bay

, Ess

, Tuytelaars

, et al. Speeded-up robust features (SURF), Computer Vision and Image Understanding 110(3) (2008), 346–359.

Harris

and Stephens

, A combined corner and edge detector, Proceedings of 4th Alvey Vision Conference (1988), 147–151.

Martinez

, Garcia-Consuegra

and Abad

, A correlation-symbolic approach to automatic remotely sensed image rectification, IEEE 1999 Geoscience and Remote Sensing Symposium 1 (1999), 336–338.

Johnson

, Cole-Rhodes

, Zavorin

, et al., Mutual information as a similarity measure for remote sensing image registration, International Society for Optics and Photonics 4383 (2001), 51–61.

and Leung

, A maximum likelihood approach for image registration using control point and intensity, IEEE Transactions on Image Processing 13(8) (2004), 1115–1127.

Thirion

J.P.

, Image matching as a diffusion process: an analogy with Maxwell’s demons, Medical Image Analysis 2(3) (1998), 243–260.

Chee

and Wu

Z.Z.

, AIRNet: self-supervised affine registration for 3D medical images using neural networks, arXiv (2018), 1810.02583.

Sloan

J.M.

, Goatman

K.A.

and Siebert

J.P.

, Learning rigid image registration utilizing convolutional neural networks for medical image registration, 11th International Joint Conference on Biomedical Engineering Systems and Technologies, Funchal, Madeira, Portugal, (2018), 89–99.

10.

Jaderberg

M.M.

, Simonyan

, Zisserman

, et al., Spatial transformer networks, Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, Quebec, Canada, (2015), 2017–2025.

11.

Devos

B.D.

, Berendsen

F.F.

, Viergever

M.A.

, et al., End-to-end unsupervised deformable image registration with a convolutional neural network, A deep metric for multimodal registration, International Workshop on Multimodal Learning for Clinical Decision Support, Springer, (2017), 204–212.

12.

Balakrishnan

, Zhao

, Sabuncu

M.R.

, et al, An unsupervised learning model for deformable medical image registration, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), 9252–9260.

13.

Zhao

S.Y.

, Dong

, Chang

E.I.

, et al., Recursive cascaded networks for unsupervised medical image registration, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019), 10600–10610.

14.

Wei

Z.J.

, Liu

G.C.

and Gu

D.D.

, Registering large deformed images based on multi-stage concatenated deep convolutional neural networks, Chinese Medical Image in Technology 38(04) (2022), 588–593.

15.

Kim

, Kim

, Lee

J.G.

, et al., Unsupervised deformable image registration using cycle-consistent, Medical Image Computing and Computer-Assisted Intervention, Springer (2019), 74–166.

16.

Andrew

, Malte

, Bruce

, et al., HyperMorph: Amortized hyperparameter learning for image registration, International Conference on Information Processing in Medical Imaging, Springer (2021), 3–17.

17.

Marcus

D.S.

, Wang

T.H.

, Parker

, et al., Open access series of imaging studies (oasis): cross-sectional mridata in young, middle aged, nondemented, and demented older adults, Journal of Cognitive Neuroscience 19(9) (2007), 1498–1507.

18.

Fischl

, Freesurfer, Neuroimage 62(2) (2012), 774–781.

19.

Hang

, Lin

, Tong

, et al, Unet 3+: A full-scale connected Unet for medical image segmentation, ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, (2020), 4–8.

20.

Kaelbling

L.P.

, Littman

M.L.

and Moore

A.W.

, Reinforcement learning: A survey, Journal of Artificial Intelligence Research 4 (2020), 237–285.

21.

Antonini

, Barlaud

, Mathieu

, et al., Image coding using wavelet Transform, IEEE Transactions on Image Processing 1(2) (1992), 205–220.

22.

Simonovsky

, Guti’errez-Becker

, Mateus

, et al., A deep metric for multimodal registration, Springer, In International Conference on Medical Image Computing and Computer-Assisted Intervention (2016), 10–18.

23.

Ronneberger

, Fischer

and Brox

, U-net: Convolutional networks for biomedical image segmentation, International Conference on Medical Image Computing and Computer-Assisted Intervention (2015), 234–241.

24.

Yang

, Kwitt

and Niethammer

, Fast predictive image registration, In Deep Learning and Data Labeling for Medical Applications, Springer (2016), 48–57.

25.

Y.B.

, Lei

, Wang

T.H.

, et al., Deep learning in medical image registration:a review, Physics in Medicine & Biology 65(20) (2020), 1–50.