Multiscale unsupervised network for deformable image registration

Abstract

BACKGROUND:

Deformable image registration (DIR) plays an important part in many clinical tasks, and deep learning has made significant progress in DIR over the past few years.

OBJECTIVE:

To propose a fast multiscale unsupervised deformable image registration (referred to as FMIRNet) method for monomodal image registration.

METHODS:

We designed a multiscale fusion module to estimate the large displacement field by combining and refining the deformation fields of three scales. The spatial attention mechanism was employed in our fusion module to weight the displacement field pixel by pixel. Except mean square error (MSE), we additionally added structural similarity (ssim) measure during the training phase to enhance the structural consistency between the deformed images and the fixed images.

RESULTS:

Our registration method was evaluated on EchoNet, CHAOS and SLIVER, and had indeed performance improvement in terms of SSIM, NCC and NMI scores. Furthermore, we integrated the FMIRNet into the segmentation network (FCN, UNet) to boost the segmentation task on a dataset with few manual annotations in our joint leaning frameworks. The experimental results indicated that the joint segmentation methods had performance improvement in terms of Dice, HD and ASSD scores.

CONCLUSIONS:

Our proposed FMIRNet is effective for large deformation estimation, and its registration capability is generalizable and robust in joint registration and segmentation frameworks to generate reliable labels for training segmentation tasks.

Keywords

Deformable image registration multiscale fusion spatial attention unsupervised learning image segmentation

1 Introduction

As a fundamental task in medical image analysis, deformable image registration (DIR) has been used in many medical applications such as image segmentation [3 , 8]. DIR is the process of establishing dense nonlinear spatial correspondences Φ between moving images m and fixed images f . The typical DIR can be formulated as: $Φ^{*} = \underset{Φ}{\arg \min} L_{sim} (f, m_{Φ}) + λ L_{reg} (Φ),$ (1) where λ is the regularization weight, Φ^* is the optimal displacement field, $L_{sim} (f, m_{Φ})$ means the dissimilarity of the fixed image f and the deformed image m _Φ = m ∘ Φ, $L_{reg} (Φ)$ represents the regularization constraint on displacement field Φ.

Traditional DIR approaches, such as Demons [21], LDDMM [6] and SyN [1] align images by solving computationally expensive iterative optimization. Therefore, classcial DIR algorithms are time-consuming and not practical in actual clinical applications.

Recently, deep learning was introduced to image registration for its high efficiency. DIR methods based on deep leaning regard image registration as mappings from input to output, which seek the best deformation vector fields by updating and optimizing the similarity objective function in the training dataset. The popular evaluation metrics currently include mean square error (MSE), cross correlation (CC) and normalized cross correlation (NCC), mutual information (MI) and normalized mutual information (NMI). Given the situation where reliable ground truth deformation fields are difficult to obtain, unsupervised deformable image registration (UDIR) schemes are becoming popular.

DIR algorithms based on deep learning have achieved significant success. However, large deformation estimation still is a key and difficult problem for deformable image registration. There are two kinds of methods for large deformation decomposition, which means learning a large deformation field by combining several subfields. One is cascade-based methods, which decompose the warp process into multistage operation. The entire network formed by stacking multiple identical subnetworks together to estimate the deformation field in a recursive manner. The other is coarse-to-fine based methods, which employ the multiscale features to estimate a deformation field. Each scale deformation component of multiscale-based methods is unequal [10], the low scale deformation field enjoys large receptive fields to deal with large deformation components, while the high scale deformation enjoys small receptive fields to deal with small deformation components.

In this paper, we proposed a fast multiscale network (FMIRNet) for unsupervised deformable image registration of monomodal medical images. Our multiscale module was designed for large deformation estimation. We first obtained three displacement fields at different scales, then inputed them to our proposed multiscale fusion module, where spatial attention was employed to refine the deformation field. Additionally, we integrated FMIRNet into Joint Registration and Segmentation (JRS) frameworks to improve segmentation tasks with few annotations. In the JRS methods, our FMIRNet can generate reliable annotations for images without ground truth labels.

On the whole, the detailed work in this paper can be summarized as follows:

(1) We propose a multiscale unsupervised deformable image registration method for large displacement estimation;

(2) We design a multiscale fusion module for combining and refine multiscale displacement fields;

(3) We integrate the FMIRNet into segmentation networks to form joint frameworks, where the FMIRNet is used to generate reliable labels from the few available annotations for supervised training to boost the segmentation task.

(4) We conduct extensive experiments and ablation studies on the registration and joint learning networks to validate the effectiveness of our work.

2 Related work

Deep learning technologies have recently been applied to medical image registration. In supervised methods for deformable image registration, ground truth deformation fields for supervised training are either obtained by traditional approaches, or synthesized by manual deformation transformations. FlowNet [4] used synthetic data for training and estimating pixel-level loss. Sokooti et al. [20] augmented their dataset using random displacement vector fields (DVFs). Fan et al. [5] used the deformation field obtained by LCC-Demons [15] and SyN [1] to train a CNN. Realtime and robust registration has been made possible by supervised transformation estimation. However, ground truth informations are difficult to collect. Additionally, it is crucial to ensure that simulated data is sufficiently similar to clinical data. These challenges make supervised registration impractical.

Recently, unsupervised registration methods have gradually become a research hotspot, which use the image-wise similarities instead of the synthesized unreal deformation fields during the training process. Balakrishnan et al. [2] customized the registration as a parameter function for the first time and modeled it through a convolutional neural network, in which a spatial transformer network (STN) [11] was used to reconstruct one image from another. Sheikhjafari et al. [19] exploited the learned latent representations that were used as input to a network composed of eight fully connected layers to obtain the transformation for deformable registration in 2D cardiac cine MR volumes. However, unsupervised registration methods which are based on similarity metrics are sensitive to local optimas during optimization [17], so that these methods may fail to estimate large displacements in complex deformation fields [12].

Recent efforts have been devoted to handle estimation of large displacements by decomposing a large deformation field into combination of several subfields in either multistage or multiscale manner. Zhao et al. proposed the Volume Tweening Network (VTN) [27] to decompose the large deformation into a series of deformation subfields by several cascade CNNs. In subsequent research, Zhao et al. [26] enlarged the number of cascades and demonstrated that, as the number of cascades increases, the performance improves accordingly. UDRSNet proposed by wang et al. [22] adopted the same way as VTN and used a structural similarity measure to align images. Mok et al. [16] proposed LapIRN, which takes a Laplacian image pyramid as inputs to estimate and refine deformation fields. Kang et al. [12] developed Dual-PRNet for a coarse-to-fine deformation field estimation.

3 Method

3.1 Multiscale unsupervised registration

Figure 1 illustrates our proposed multiscale unsupervised network for deformable image registration (Taking 2D images as an example). We denote the moving image as m and the fixed image as f . In our multiscale framework, we first generated multiscale images m _i ∈ { m ₁, m ₂, m ₃} (and f _i ∈ { f ₁, f ₂, f ₃}) by downsampling the original moving and fixed image with bilinear interpolation, where m _i means the downsampled version of m with a scale factor 0 . 5^i-1. We concatenated the moving and fixed image [ m , f ] as input. After the encoding and decoding process, two feature maps Φ_x and Φ_y are obtained, and Φ = [Φ_x, Φ_y] is the dense flow (i.e., displacement field). Each element in a flow indicates the displacement of a particular pixel from its original location to a new location.

Fig. 1

Flowchart of the proposed FMIRNet.

As shown in Fig. 1(a), we employed a ResUnet [22, 25] to model three dense nonlinear transformations (Φ₁, Φ₂ and Φ₃) from multiscale image pairs ([ m ₁, f ₁], [ m ₂, f ₂] and [ m ₃, f ₃]) respectively. The “Shared Parameters” means using the same network to estimate the displacement fields of three different scales of inputs. In this way, we can estimate three displacement fields in one iteration. Therefore, our model is more efficient compared to learning deformation fields separately at each scale.

We proposed a spatial attention module to combine and refine the information contained by the multiscale displacement fields. As illustrated in Fig. 1(b), our attention mechanism was introduced after the displacement estimation module. We first upsampled Φ₂ and Φ₃ by factor 2 (denoted as Φ^'₂) and factor 4 (denoted as Φ^'₃), the preliminary fused features can be obtained as follows: $Φ_{p} = {Φ'}_{1} + {Φ'}_{2} + {Φ'}_{3},$ (2)

Next, we applied the spatial attention module on Φ_p. Our attention module was composed of a Conv layer followed by a Sigmoid activate layer. The Sigmoid layer was configured to generate spatial weights W for each element in Φ_p. The learned attention W was applied to Φ_p by W ⊗ Φ_p. Finally, we obtained the multiscale fused deformation field as follows: $Φ = Φ_{p} + W \otimes Φ_{p},$ (3) where ⊗ denotes the element-wise multiplication operator. Then generate the deformed image through Grid Sampling operator, the grid is computed as: $G = G_{I} + Φ,$ (4) where G_I is the unit grid, taking 2D images as example, the deformed image is generated by bilinear interpolation sampling as: $m_{Φ} (\to p) = \sum_{\to q} m (\to q) \prod_{d} | 1 - [Φ (\to q)]_{d} |,$ (5) where →p and →q are pixel locations, d ∈ {x, y} denotes a horizontal direction or vertical direction, and [Φ (→p)] _d means the d-component of Φ (→p).

3.2 Joint registration and segmentation

Recent segmentation methods using deep learning technologies often require massive manually annotated labels, however that is labor intensive and expensive. Our proposed multiscale unsupervised deformable registration method can transfer a moving image to a fixed image, so we employed such an ability to map the available annotated labels from a moving domain to a fixed domain where the annotations are not provided.

As shown in Fig. 2, we integrated the registration network into a segmentation network to form a unified framework similar to [12, 24]. We learned two tasks simultaneously using the proposed multiscale architecture as a registration network and using FCN [14], UNet [18] as segmentation networks, respectively.

Fig. 2

Joint registration and segmentation framework, where the proposed FMIRNet is used as registration subnetwork (RegNet). FMIRNet is applied to warp moving images and their labels, and then moving images and their labels, as well as, warped images and their labels are used as inputs for segmentation subnetwork for supervised training.

Figure 2 provides a detailed explanation of the proposed joint learning framework. Given a moving m with annotated label l _m and fixed image f , our multiscale registration network was used to estimate the deformation field Φ, which was then applied to warp the moving labels. The segmentation network took m , l _m, m _Φ, l _m ∘ Φ as input and has two segmentation maps denoted as S _m, S _d. In the inference stage, segmentation subnetwork can be used separately.

Our joint learning method is designed for boosting segmentation with few annotations samples. For each 3D volume data selected from CHAOS, firstly, we divided volume into n groups 2D image slices, then choose the transverse-section slice as a moving image and the remaining slices as fixed images. In our training phase, only the moving image has annotated labels, the deformed labels were generated by warping l _m by deformation field Φ. Few annotations indicate that only few images have ground truth labels in training dataset, the ratio of annotated images to the total image can be computed as: $r = \frac{num (l_{m})}{num (m) + num (f)},$ (6) where num means the number of images.

3.3 Loss function

In our model, the deformed image is represented as m _Φ, similarity estimation $l_{s i m} = \frac{1}{N} ‖ f - m_{Φ} ‖_{2}^{2}$ between f and m _Φ was defined as the mean square error (MSE) loss, $l_{d e f} = ‖ Φ ‖_{2}^{2}$ was employed to guarantee the smooth of deformation field Φ. Structural similarity (SSIM) estimation measures images by defining structural information as independent of brightness and contrast, reflecting the properties of object structures in the scene,and can simultaneously constrain the consistency of two images in terms of brightness, contrast, and structure. We formulated the ssim loss as follows:

$\begin{matrix} l_{ssim} = & 1 - ssim (m_{Φ}, f) \\ = & 1 - \frac{(2 μ_{m_{Φ}} μ_{f} + C_{1}) (2 σ_{m_{Φ} f} + C_{2})}{(μ_{m_{Φ}}^{2} + μ_{f}^{2} + C_{1}) (σ_{m_{Φ}}^{2} + σ_{f}^{2} + C_{2})}, \end{matrix}$ (7) where μ is the mean value of pixel intensity values, σ represents covariance and variance, C₁ = (K₁R) ² and C₂ = (K₂R) ² are set to maintain computational stability. R = 2^{#bits per pixel} - 1 is the range of pixel intensity values [23]. We used single channel grayscale images (bits per pixel = 8) for the experiments, setting R = 255 as their maximum pixel intensity value is 255, K₁ = 0.01, K₂ = 0.03 are the default values.

Formally, the registration loss was defined as: $\begin{matrix} l_{reg} & = l_{sim} + α l_{def} + β l_{ssim} \\ = \frac{1}{N} ∥ f - m_{Φ} ∥_{2}^{2} + α ∥ Φ ∥_{2}^{2} + β (1 - ssim (m_{Φ}, f)), \end{matrix}$ (8) where α and β are the weights to balance the contributions of deformation field loss and structural similarity loss, and N is the number of pixels in the reference image.

The segmentation network takes m , l _m, m ∘ Φ and l _m ∘ Φ as input and has two segmentation maps denoted S _m S _d. We use ground truth labels and generated labels to train the segmentation network in a regular supervised manner, therefore the segmentation loss can be computed as: $l_{seg} = l_{cro} (l_{m}, S_{m}) + l_{cro} (l_{m} \circ Φ, S_{d})$ (9) where l_cro (· , ·) is the Cross-Entropy loss function.

4 Experiment

4.1 Data and pre-processing

We first evaluated our proposed multiscale registration network on 2D ultrasound images selected from EchoNet 1 . EchoNet consists of videos of parasternal long-axis echocardiography. For each video, we selected two frames as moving and fixed images to obtain 1362 image pairs, and divided the training, validation, and testing sets in a 7:2:1 ratio. Then registration experiment of 2D CT images and our joint registration and segmentation method is performed on CT images seclected from the CHAOS competition 2 . The CT dataset contains 20 volumes of data. For each CT volume data, divide the volume data into n sets of 2D slices in order, then select the transverse-section slice as the moving image and the remaining slices as fixed images (including adjacent slices) to obtain a set of image pairs. After that, all images were resampled at 128 * 128, and we randomly divided the data set into 738, 210, and 104 image pairs for training, validation, and test sets, respectively. As for 3D registration, we performed experiments on 400 pairs scans selected from the CT dataset SLIVER [9], and divided the training, validation, and testing sets in a 7:2:1 ratio. The resolution for scans of this dataset is 128 × 128 × 128.

4.2 Implementation details

The network training process was implemented using the PyTorch toolkit and optimized through the Adam optimizer with a learning rate of 1e - 4. The regularization parameters for CT images α = 1e - 7, β = 0.1 and ultrasound images α = 1e - 5, β = 1 were set to ensure that the magnitude of each loss element was the same. The models were trained for 40 epochs, with a batch size of 4. The experimental setup included NVIDIA GeForce RTX3090 and Intel Core i9-10920X@3.50GHz.

4.3 Evaluation metrics

In order to evaluate the registration performance of our proposed multiscale unsupervised network, normalized cross-correlation (NCC), normalized mutual information (NMI) and ssim were adopted. NCC is a metric of degree of correlation, the higher the ncc value, the better the registration effect. NMI values are within the range of [0, 1], the larger the better.

Dice, Hausdorff Distance (HD) and Average Symmetric Surface Distance (ASSD) are considered to measure the performance of the segmentation subnetwork in our joint learning framework. A higher Dice score between two regions denotes a higher degree of their overlap, meaning higher segmentation performance. A smaller value of HD denotes high proximity between the segmentation prediction and ground truth. ASSD measures the averaged mismatch between the surfaces of two volumes.

4.4 Results

4.4.1 Results of registration

We choose the unsupervised registration network VoxelMorph [2] and UDRSNet [22] to compare the registration performance of our proposed model. Meanwhile, we also compared our algorithms with state-of-art traditional registration algorithms including Elastix [13] and SyN [1]. The quantitative results are presented in Table 1. We can see that our FMIRNet performs better in terms of SSIM, NCC, and NMI scores; for example, FMIRNet improved SSIM (0.05), NCC (0.06) than UDRSNet. We attribute these improvements to our multiscale fusion module for refining deformable field.

Table 1
Results of different methods on EchoNet. Standard deviations across instances are in parentheses

Method SSIM NCC NMI

Elastix 0.7257 (0.0449) 0.8790 (0.0366) 0.4665 (0.0091)

SyN 0.8278 (0.0455) 0.7090 (0.1653) 0.6849 (0.0281)

UDRSNet 0.8691 (0.0311) 0.9000 (0.0360) 0.9367 (0.0252)

VoxelMorph 0.8781 (0.0293) 0.9442 (0.0204) 0.9655(0.0171)

FMIRNet 0.9279 (0.0206) 0.9643 (0.0146) 0.9630 (0.0169)

Method	SSIM	NCC	NMI
Elastix	0.7257 (0.0449)	0.8790 (0.0366)	0.4665 (0.0091)
SyN	0.8278 (0.0455)	0.7090 (0.1653)	0.6849 (0.0281)
UDRSNet	0.8691 (0.0311)	0.9000 (0.0360)	0.9367 (0.0252)
VoxelMorph	0.8781 (0.0293)	0.9442 (0.0204)	0.9655(0.0171)
FMIRNet	0.9279 (0.0206)	0.9643 (0.0146)	0.9630 (0.0169)

We took three pairs of EchoNet images as examples and showed the visualization results of different methods in Fig. 3. From Fig. 3, we can see that the proposed FMIRNet method achieves an impressive improvement in the registration results, compared to the competing methods.

Fig. 3

Registration results of different methods for deforming source images (left) to target images (right) on EchoNet.

In addition to ultrasound images, we also conducted registration experiments on CT images. Table 2 gives a summary of the quantitative results on CHAOS. The visualization results can be find in Fig. 4.

Table 2

Results of different methods on CHAOS. Standard deviations across instances are in parentheses

Method	SSIM	NCC	NMI
Elastix	0.4642 (0.0478)	0.5098 (0.0202)	0.6359 (0.0187)
SyN	0.5796 (0.0839)	0.5745 (0.1465)	0.6459 (0.0222)
UDRSNet	0.7082 (0.1107)	0.8794 (0.0923)	0.9720 (0.0246)
VoxelMorph	0.6444 (0.0860)	0.9526 (0.0646)	0.9813 (0.0213)
FMIRNet	0.7800 (0.0894)	0.9603 (0.0544)	0.9828 (0.0179)

Fig. 4

Samples of registration results of our joint learning framework on CHAOS.

4.4.2 Results of joint registration and segmentation

We further evaluated the performance of our joint registration and segmentation framework using the proposed FMIRNet, which can be integrated into a 2D segmentation network. FCN and UNet were adopted as segmentation networks in our joint framework to verify the universality of our registration method. Figure 4 shows some registration visualization samples, we can see that our FMIRNet not only aligns moving image to fixed image well, but also generates reliable annotation label to deformed image. These generated images and deformed labels help us to train a better segmentation network. In the inference stage, the segmentation subnetworks in our joint framework are used independently.

The segmentation results are shown in Table 3 and Fig. 5. FCN denotes training segmentation network in supervised manner on images with annotations, RFCN is a joint registration and segmentation network, where registration task employs our proposed FMIRNet, RUNet is also like this. Corresponding, VUNet and VFCN mean using voxmorph as registration subnetwork in joint framework. From Table 3 and Fig. 5, we can observe that both RFCN and RUNet achieve performance improvement compared to FCN and RUNet. The reason of this is that our joint registration and segmentation method has more images with reliable annotations, generated by registration subnetwork.

Table 3
Results of different methods of segmentation task on CHAOS. Standard deviations across instances are in parentheses

Method Dice HD ASSD

FCN 0.9213 (0.0117) 1.0099 (0.1415) 0.2519 (0.0561)

RFCN 0.9500 (0.0086) 0.2320 (0.4221) 0.1141 (0.0254)

VFCN 0.9509 (0.0087) 0.3520 (0.4776) 0.1107 (0.0212)

UNet 0.9446 (0.0171) 0.4667 (0.4989) 0.1263 (0.0445)

RUNet 0.9518 (0.0096) 0.3120 (0.4633) 0.1008 (0.0426)

VUNet 0.9513 (0.0096) 0.3333 (0.4714) 0.1189 (0.0238)

Method	Dice	HD	ASSD
FCN	0.9213 (0.0117)	1.0099 (0.1415)	0.2519 (0.0561)
RFCN	0.9500 (0.0086)	0.2320 (0.4221)	0.1141 (0.0254)
VFCN	0.9509 (0.0087)	0.3520 (0.4776)	0.1107 (0.0212)
UNet	0.9446 (0.0171)	0.4667 (0.4989)	0.1263 (0.0445)
RUNet	0.9518 (0.0096)	0.3120 (0.4633)	0.1008 (0.0426)
VUNet	0.9513 (0.0096)	0.3333 (0.4714)	0.1189 (0.0238)

Fig. 5

Samples of segmentation results of our joint learning framework on CHAOS.

4.5 Ablation study

Multiscale Module. We used the ResUnet to estimate the displacement field, which has three encoding blocks, including three downsampling operations. We resized the image to a size of 128 × 128 × 128 (128 × 128) and set the convolution kernel size to 3 × 3 × 3 (3 × 3). Therefore, to ensure that the size of the smallest feature map in the multiscale scheme is greater than the size of the convolution kernel, we used three scale inputs to estimate the displacement field.

In Table 4, we compare FMIRNet and its variant FMIRNet-1 and FMIRNet-2. Only Φ₁ is estimated as deformation field in FMIRNet-1, while FMIRNet-2 employs Φ₁, Φ₂ to estimate deformation field. We can get conclusion that our proposed multiscale module learns a more accurate and reliable spatial transformation.

Table 4
Ablation study of multiscale module. Standard deviations across instances are in parentheses

Method SSIM NCC NMI

EchoNet FMIRNet-1 0.9193 (0.0200) 0.9602 (0.0153) 0.9618 (0.0169)

FMIRNet-2 0.9262 (0.0183) 0.9640 (0.0134) 0.9632 (0.0161)

FMIRNet 0.9279 (0.0206) 0.9643 (0.0146) 0.9630 (0.0169)

CHAOS FMIRNet-1 0.7782 (0.0821) 0.9596 (0.0519) 0.9822 (0.0179)

FMIRNet-2 0.7800 (0.0894) 0.9526 (0.0646) 0.9813 (0.0213)

FMIRNet 0.7869 (0.0802) 0.9603 (0.0544) 0.9828 (0.0189)

	Method	SSIM	NCC	NMI
EchoNet	FMIRNet-1	0.9193 (0.0200)	0.9602 (0.0153)	0.9618 (0.0169)
	FMIRNet-2	0.9262 (0.0183)	0.9640 (0.0134)	0.9632 (0.0161)
	FMIRNet	0.9279 (0.0206)	0.9643 (0.0146)	0.9630 (0.0169)
CHAOS	FMIRNet-1	0.7782 (0.0821)	0.9596 (0.0519)	0.9822 (0.0179)
	FMIRNet-2	0.7800 (0.0894)	0.9526 (0.0646)	0.9813 (0.0213)
	FMIRNet	0.7869 (0.0802)	0.9603 (0.0544)	0.9828 (0.0189)

Regularization parameter α. We conducted comparative experiments to determine a better α value, Table 5 records the registration experiment results for different α values. The β value was set based on the principle that the magnitude of the ssim loss is the same as that of the mse loss.

Table 5

Ablation studies of the regularization weight α values. Standard deviations across instances are in parentheses

EchoNet
	α = 1	α = 1e - 2	α = 1e - 3	α = 1e - 5	α = 1e - 7
SSIM	0.7548 (0.0486)	0.7549 (0.0486)	0.8188 (0.0473)	0.9279 (0.0206)	0.9154 (0.0203)
NCC	0.8180 (0.0652)	0.8181 (0.0652)	0.8817 (0.0494)	0.9643 (0.0146)	0.9595 (0.0154)
NMI	0.9635 (0.0214)	0.9635 (0.0214)	0.9636 (0.0211)	0.9630 (0.0169)	0.9592 (0.0154)
CHAOS
	α = 1	α = 1e - 3	α = 1e - 5	α = 1e - 7	α = 1e - 8
SSIM	0.4784 (0.1315)	0.4786 (0.1315)	0.7817 (0.0833)	0.7800 (0.0894)	0.7782 (0.0799)
NCC	0.8228 (0.1201)	0.8230 (0.1201)	0.9574 (0.0552)	0.9603 (0.0544)	0.9603 (0.0479)
NMI	0.9730 (0.0287)	0.9730 (0.0287)	0.9805 (0.0209)	0.9828 (0.0189)	0.9823 (0.0177)

Results of 3D Registration. In order to evaluate the proposed model with 3D medical image registration task, we extended the proposed FMIRNet for registration of 3D volumes by replacing the corresponding convolution operations and spatial transformation layers. We utilized the subject-to-subject registration strategy for our experiments. Subject-to-subject registration means that the moving and fixed volumes collected from different subjects, therefore we used dice score between the deformed liver label and the fixed label as measure as well as SSIM and NCC. Table 6 presents the results of different method on SLIVER, and Fig. 6 visualizes 3D registration results of three samples. The 3D registration task is more difficult, as shown in Fig. 6, there are other organs such as the intestines around the liver, which have a negative impact on the deformation of the liver area.

Table 6

Results of different methods on SLIVER. Standard deviations across instances are in parentheses

Method	SSIM	NCC	Dice
Elastix	0.4642 (0.0478)	0.5583 (0.1628)	0.6097 (0.0132)
SyN	0.5796 (0.0839)	0.5500 (0.2978)	0.6923 (0.0237)
UDRSNet	0.6469 (0.0216)	0.6949 (0.2160)	0.8241 (0.0378)
VoxelMorph	0.5583 (0.0219)	0.6392 (0.2262)	0.8024 (0.0390)
FMIRNet	0.6756 (0.0228)	0.6878 (0.2391)	0.8250 (0.0395)

Fig. 6

Samples of 3D registration results on SLIVER.

Number of r. Table 7 presents results with respect to different r. The test data have the ground truth labels annotated by experts.Our joint frameworks are designed to improve the performance of segmentation tasks when only few annotated labels is available for supervised training. The smaller the r, the fewer annotated images there are. As shown in Table 7, joint learning methods (RFCN, RUNet, VFCN and VUNet) all have performance improvement than FCN and UNet. The 2D slices of volume data vary greatly in the position and size of the liver, which is alleviated when there are more annotated moving images, namely r becomes larger. Therefore, compared to the case of r = 0.01, the HD errors of RUNet reduced 0.176 when r = 0.1.

In the joint framework, registration subnetwork is adopted to generate new labels from manually annotated labels by experts. Therefore, the segmentation network in joint network was trained using few annotated labels and deformed labels, while FCN and UNet were trained using only few annotated labels. Therefore, better segmentation performances on test data of the joint methods indicating that the registration network generates reliable and accurate annotations for unlabeled slices, that is, the learned spatial transformations are accurate and reliable.

We can get two conclusions from the data in Table 7. On the one hand, the proposed FMIRNet is reliable to generate accurate labels for unlabeled fix images, as RFCN (RUNet) is trained on moving labels and deformed labels and has better segmentation performance than FCN (UNet). On the one hand, the proposed FMIRNet has a good generalization ability in joint registration and segmentation framework, as the segmentation performances of RUNet and RFCN all have been improved.

Table 7

Ablation studies of different r on CHAOS. Standard deviations across instances are in parentheses

	Method	Dice	HD	ASSD
r=0.01	FCN	0.9213 (0.0117)	1.0099 (0.1415)	0.2519 (0.0561)
	RFCN	0.9506 (0.0086)	0.2320 (0.4221)	0.1008 (0.0254)
	VFCN	0.9509 (0.0087)	0.3520 (0.4776)	0.1107 (0.0212)
	UNet	0.9446 (0.0171)	0.4667 (0.4989)	0.1263 (0.0445)
	RUNet	0.9518 (0.0096)	0.3120 (0.4633)	0.1141 (0.0426)
	VUNet	0.9513 (0.0096)	0.3333 (0.4714)	0.1189 (0.0238)
r=0.05	FCN	0.9298 (0.0171)	0.9444 (0.2291)	0.1834 (0.0382)
	RFCN	0.9538 (0.0078)	0.2640 (0.4408)	0.1034 (0.0210)
	VFCN	0.9484 (0.0076)	0.2944 (0.4899)	0.1146 (0.0132)
	UNet	0.9451 (0.0145)	0.4943 (0.5376)	0.1274 (0.0275)
	RUNet	0.9538 (0.0074)	0.2240 (0.4169)	0.1002 (0.0264)
	VUNet	0.9530 (0.0077)	0.3333 (0.4714)	0.1134 (0.0201)
r=0.1	FCN	0.9391 (0.0170)	0.8943 (0.3655)	0.1386 (0.0248)
	RFCN	0.9551 (0.0074)	0.1920 (0.3939)	0.1002 (0.0264)
	VFCN	0.9547 (0.0057)	0.1333 (0.3499)	0.1016 (0.0152)
	UNet	0.9451 (0.0145)	0.4934 (0.5376)	0.1274 (0.0275)
	RUNet	0.9563 (0.0065)	0.1360 (0.3428)	0.1004 (0.0158)
	VUNet	0.9532 (0.0084)	0.3333 (0.4714)	0.1182 (0.0314)

5 Discussion

In our multiscale deformation fields estimation module, Φ_p is computed as Equation 2, which is based a assumption that Φ₁, and have the same influence to generate the final deformation field Φ. However, this simple assumption may not necessarily hold true in practical applications, as the low scale deformation field enjoy large receptive fields, while the high scale enjoy small receptive fields. In our further research, we plan to incorporate channel attention to adaptively learn the weights of displacement fields at various scales.

The dataset we used in our proposed joint registration and segmentation frameworks is composed of 2D slices of 3D volume data. Between different 2D slices, the liver exhibits positional movement and varying volume sizes, and these factors all have an impact on the performance of registration tasks. According to Equation 6 and Table 7, the larger r, the greater the correlation between moving and fixed images, and the accuracy of the registration tasks will also be improved, providing more reliable annotation labels for segmentation tasks.

We design the joint registration and segmentation frameworks by integrating FMIRNet into FCN (denoted as RFCN) and UNet (denoted as RUnet). Experimental results illustrate that both RFCN and RUNet have performance improvement in terms of Dice, HD and ASSD scores; these confirm the generalization ability and robustness of our FMIRNet for boosting segmentation task with few annotation labels. Therefore, we can theoretically conclude that FMIRNet can be integrated into any segmentation network as a plug-and-play block.

Currently, the proposed FMIRNet is designed for 2D registration tasks. 3D registration tasks are more difficult and complicated than 2D registration tasks, the proposed FMIRNet can be extended for 3D registration task by directly replacing the convolution layers and spatial transformation layers. However, due to the influence of other organs around the liver, the registration performance deteriorates obviously. In our future research, for 3D registration tasks, we plan to design some region perception modules to restrict the deformation regions.

6 Conclusion

In this paper, we propose a fast multiscale unsupervised network (FMIRNet) for monomodal deformable image registration in an end-to-end manner. Compared to single scale registration scheme, the FMIRNet has indeed performance improvements in terms of SSIM, NCC and NMI scores. The good registration ability of FMIRNet inspires us to integrate it into the joint registration and segmentation framework to boost segmentation task with few annotated labels. Using FCN and UNet as segmentation networks, respectively, the joint methods RFCN and RUNet have performance gains in terms of Dice, HD, and ASSD scores, compared to using FCN and UNet alone. Still, the FMIRNet has several limitations in its current implementation. First, the FMIRNet is designed for monomodal image registration tasks. Therefore, how to efficiently register multimodal images is one of our future work. Second, the FMIRNet is designed for 2D registration tasks and has slightly inferior performance for 3D registration tasks. Therefore, improving FMIRNet also deserves exploration in the future.

Footnotes

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant Nos. 12090020 and 12090025).

References

Avants

Brian B.

, Epstein

Charles L.

, Grossman

Murray

and Gee

James C.

, Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain, Medical Image Analysis 12(1) (2008), 26–41.

Balakrishnan

Guha

, Zhao

Amy

, Sabuncu

Mert R.

, Guttag

John

and Dalca

Adrian V.

, An unsupervised learning model for deformable medical image registration, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9252–9260, 2018.

Brock

Kristy K.

, Mutic

Sasa

, McNutt

Todd R.

, Li

Hua

and Kessler

Marc L.

, Use of image registration and fusion algorithms and techniques in radiotherapy: Report of the aapm radiation therapy committee task group no. 132, Medical Physics 44(7) (2017), e43–e76.

Dosovitskiy

Alexey

, Fischer

Philipp

, Ilg

Eddy

, Hausser

Philip

, Hazirbas

Caner

, Golkov

Vladimir

, Van Der Smagt

Patrick

, Cremers

Daniel

and Brox

Thomas

, Flownet: Learning optical flow with convolutional networks, In Proceedings of the IEEE international conference on computer vision, pages 2758–2766, 2015.

Fan

Jingfan

, Cao

Xiaohuan

, Yap

Pew-Thian

and Shen

Dinggang

, Birnet: Brain image registration using dual-supervised fully convolutional networks, Medical Image Analysis 54 (2019), 193–206.

Glaunes

Joan

, Qiu

Anqi

, Miller

Michael I.

and Younes

Laurent

, Large deformation diffeomorphic metric curve mapping, International Journal of Computer Vision 80 (2008), 317–336.

Yuting

, Ge

Rongjun

, Qi

Xiaoming

, Chen

Yang

, Wu

Jiasong

, Coatrieux

Jean-Louis

, Yang

Guanyu

and Li

Shuo

, Learning better registration to learn better few-shot medical image segmentation: Authenticity, diversity, and robustness, IEEE Transactions on Neural Networks and Learning Systems, 2022.

Yuting

, Li

Tiantian

, Ge

Rongjun

, Yang

Jian

, Kong

Youyong

, Zhu

Jian

, Shu

Huazhong

, Yang

Guanyu

and Li

Shuo

, Few-shot learning for deformable medical image registration with perceptioncorrespondence decoupling and reverse teaching, IEEE Journal of Biomedical and Health Informatics 26(3) (2021), 1177–1187.

Heimann

Tobias

, Ginneken

Bram Van

, Styner

Martin A.

, Arzhaeva

Yulia

, Aurich

Volker

, Bauer

Christian

, Beck

Andreas

, Becker

Christoph

, Beichel

Reinhard

, Bekes

György

, et al., Comparison and evaluation of methods for liver segmentation from ct datasets, IEEE Transactions on Medical Imaging 28(8) (2009), 1251–1265.

10.

, Zhou

Shenglong

, Xiong

Zhiwei

and Wu

Feng

, Recursive decomposition network for deformable image registration, IEEE Journal of Biomedical and Health Informatics 26(10) (2022), 5130–5141.

11.

Jaderberg

Max

, Simonyan

Karen

, Zisserman

Andrew

, et al., Spatial transformer networks, Advances in Neural Information Processing Systems 28 (2015).

12.

Kang

Miao

, Hu

Xiaojun

, Huang

Weilin

, Scott

Matthew R

and Reyes

Mauricio

, Dual-stream pyramid registration network, Medical Image Analysis 78 (2022), 102379.

13.

Klein

Stefan

, Staring

Marius

, Murphy

Keelin

, Viergever

Max A.

and Pluim

Josien PW

, Elastix: a toolbox for intensity-based medical image registration, IEEE Transactions on Medical Imaging 29(1) (2009), 196–205.

14.

Long

Jonathan

, Shelhamer

Evan

and Darrell

Trevor

, Fully convolutional networks for semantic segmentation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.

15.

Lorenzi

Marco

, Ayache

Nicholas

, Frisoni

Giovanni B.

and Pennec

Xavier

, Alzheimer’s Disease Neuroimaging Initiative (ADNI, et al., Lcc-demons: a robust and accurate symmetric diffeomorphic registration algorithm, NeuroImage 81 (2013), 470–483.

16.

Mok

Tony C.W.

and Chung

Albert C.S.

, Large deformation diffeomorphic image registration with laplacian pyramid networks, In Medical Image Computing and Computer Assisted Intervention– MICCAI 2020:23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23, pages 211–221. Springer, 2020.

17.

Rohlfing

Torsten

, Image similarity and tissue overlaps as surrogates for image registration accuracy: widely used but unreliable, IEEE Transactions on Medical Imaging 31(2) (2011), 153–163.

18.

Ronneberger

Olaf

, Fischer

Philipp

and Brox

Thomas

, U-net: Convolutional networks for biomedical image segmentation, In Medical image computing and computer-assisted intervention–MICCAI 2015:18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 pages 234–241. Springer, 2015.

19.

Sheikhjafari

Ameneh

, Noga

Michelle

, Punithakumar

Kumaradevan

and Ray

Nilanjan

, Unsupervised deformable image registration with fully connected generative neural network, In Medical imaging with deep learning, 2022.

20.

Sokooti

Hessam

, Vos

Bob De

, Berendsen

Floris

, Lelieveldt

Boudewijn P.F.

, Išgum

Ivana

and Staring

Marius

, Nonrigid image registration using multi-scale 3d convolutional neural networks, In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017:20th International Conference, Quebec City, QC, Canada, September 11–13, 2017, Proceedings, Part I 20, pages 232–239. Springer, 2017.

21.

Vercauteren

Tom

, Pennec

Xavier

, Perchant

Aymeric

and Ayache

Nicholas

, Diffeomorphic demons: Efficient non-parametric image registration, NeuroImage 45(1) (2009), S61–S72.

22.

Wang

Yun

, Huang

Chongfei

, Chang

Wanru

, Lu

Wenliang

, Hui

Qinglei

, Jiang

Siyuan

, Ouyang

Xiaoping

and Kong

Dexing

, Udrsnet: An unsupervised deformable registration module based on image structure similarity, Medical Physics 2024.

23.

Wang

Zhou

, Bovik

Alan C.

, Sheikh

Hamid R.

and Simoncelli

Eero P.

, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing 13(4) (2004), 600–612.

24.

Zhenlin

and Niethammer

Marc

, Deepatlas: Joint semi-supervised learning of image registration and segmentation, In Medical Image Computing and Computer Assisted Intervention–MICCAI 2019:22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part II 22, pages 420–429. Springer, 2019.

25.

Zhang

Zhengxin

, Liu

Qingjie

and Wang

Yunhong

, Road extraction by deep residual u-net, IEEE Geoscience and Remote Sensing Letters 15(5) (2018), 749–753.

26.

Zhao

Shengyu

, Dong

Yue

, Chang

Eric I.

, Xu

Yan

, et al., Recursive cascaded networks for unsupervised medical image registration, In Proceedings of the IEEE/CVF international conference on computer vision, pages 10600–10610, 2019.

27.

Zhao

Shengyu

, Lau

Tingfung

, Luo

, Eric

, Chang

Chao

and Xu

Yan

, Unsupervised 3d endto- end medical image registration with volume tweening network, IEEE Journal of Biomedical and Health Informatics 24(5) (2019), 1394–1404.