New improved model for joint segmentation and registration of multi-modality images: application to medical images

Abstract

Joint segmentation and registration of images is a focused area of research nowadays. Jointly segmenting and registering noisy images and images having weak boundaries/intensity inhomogeneity is a challenging task. In medical image processing, joint segmentation and registration are essential methods that aid in distinguishing structures and aligning images for precise diagnosis and therapy. However, these methods encounter challenges, such as computational complexity and sensitivity to variations in image quality, which may reduce their effectiveness in real-world applications. Another major issue is still attaining effective joint segmentation and registration in the presence of artifacts or anatomical deformations. In this paper, a new nonparametric joint model is proposed for the segmentation and registration of multi-modality images having weak boundaries/noise. For segmentation purposes, the model will be utilizing local binary fitting data term and for registration, it is utilizing conditional mutual information. For regularization of the model, we are using linear curvature. The new proposed model is more efficient to segmenting and registering multi-modality images having intensity inhomogeneity, noise and/or weak boundaries. The proposed model is also tested on the images obtained from the freely available CHOAS dataset and compare the results of the proposed model with the other existing models using statistical measures such as the Jaccard similarity index, relative reduction, Dice similarity coefficient and Hausdorff distance. It can be seen that the proposed model outperforms the other existing models in terms of quantitatively and qualitatively.

Keywords

Image segmentation,,,,image registration linear curvature (LC)conditional mutual information (CMI)Jaccard similarity index (JSI)

1 Introduction

Image segmentation and registration are considered the most demanding issues in image processing faced in many areas like in medical imaging analysis [1, 2], pattern recognition [4], geophysics [5], data comparison common reference frame [3], shape tracking, etc. In the above-mentioned cases, both image segmentation and registration depend on each other and can be treated at the same time within the joint framework. The aim of image registration is to find an optimal geometric transformation which aligns different images data [6], whereas image segmentation aims to partition an image based on some certain special features w.r.t. color, texture, intensity etc.

In last two decades, few joint segmentation and registration models have been proposed for mono-modality images. In joint models, it is important that better registration of moving image should lead to better segmentation of target image and vice verse. This requires a match of the intensity distribution and geometric features of the moving and target images. On the other hand, the multi-modalities images are produced by different image modalities and different intensity ranges leading to extreme intensity differences. Therefore, the aim of registration process for multi-modalities images is to correlate the given images from two or more different modalities without changing the modality of the moving image. Multi-modality image registration enables the combination of complementary information from images acquired using various modalities [15, 17]. The multi-modality image registration task is shown to be more difficult than mono-modal image registration because finding an optimal similarity measurement for it is very difficult. The image registration for such cases requires outputs that show physiological and anatomical information. Such common examples are found in the fields of medicine in multiple diagnostics where MRI and CT images introduce various image perspectives and information.

In literature, there exist several simultaneous unsupervised and supervised models for mono-modality images. Well known simultaneous models for mono modality are classified as rigid registration based models [7, 8], non-rigid registration based models [9, 10] and atlas-based registration models [11] etc. For the first time segmentation and registration was jointly presented as a model by Yezzi et al. in [8]. In [13], Le-Guyader et al. presented a joint model for non-rigid mono-modality images. The authors employ a non-linear elastic term as a similarity measure. The model registers the segmented moving image with the target image and allows large deformations. This model is able to construct topology retaining segmentation which warps the level set of the moving image to the level set of the target image without blending or breaking. In comparison to this model, Ibrahim et al. [14] presented a joint model using linear curvature as smoother. This model can deal with the mono-modal images containing more than one object and severe deformations but still can not deal with multi-modal images.

Joint segmentation and registration models for multi-modality images are just a few that have been worked upon and among those, the first one is introduced by Wang et al. [12], with the segmentation of the target image without changing the non-rigidity. Although this model is doing well in accommodating the images having different intensities but can not cope with large deformations. A variational model based on a local energy density for multi-modality image registration is presented in [18], which utilizes a generalized notion of image morphology. In [16] a non-information theocratical measure for image registration is considered using multi-modality images where the error of segmentation is considered as registration cost function. This model is designed only for rigid registration leading to the limitation of this model. In [25] the author presented a joint model, which combines both nonlocal shape descriptors and total variations. The limitation of this model is that this model depends on many parameters. In [19] Ademaj et al. presented a joint regmentation model using linear curvature smoother and the mutual information [21] similarity measure for image registration through an active contour [23, 37] representation which easily can cope with the merging and splitting for a proper segmentation. Even though the model show relatively good results for some hard cases where rigid or non-rigid image registration is required, the model may stuck in local minima and struggles with severe non-rigid deformations. Due to the use of mutual information based similarity measure, this issue is demonstrated in experimental section in detail.

Moreover, it is observed that joint segmentation and registration model may produce better results than applying segmentation and registration separately. In Fig. 1, segmentation and registration models are applied separately and in Fig. 1a the joint segmentation and registration model is used. From Fig. 1a we can see that jointly the model produce better results as compared to separately applying segmentation and registration models in Fig. 1.

Fig. 1

Applying segmentation and registration separately. Results of segmentation with LBF model and then using linear curvature model based on CMI for registration of synthetic noisy image. (a) Segmented moving image, M with $\hat{ψ} (\dot{x})$ . (b) Target image I₀ with Gaussian noise. (c) The deformation field. (d) The target image I₀ with $\hat{ψ} (\dot{x} + u (\dot{x}))$ . (e) The transformed moving image $M (\dot{x} + u (\dot{x}))$ .

Fig. 2

Applying segmentation and registration jointly by using LBF model for segmentation and linear curvature model based on CMI for registration of synthetic noisy image. (a) Segmented moving image, M with $\hat{ψ} (\dot{x})$ . (b) Target image I₀ with Gaussian noise. (c) The deformation field. (d) The target image I₀ with $\hat{ψ} (\dot{x} + u (\dot{x}))$ . (e) The transformed moving image $M (\dot{x} + u (\dot{x}))$ .

Working with multi-modality medical imaging data presents challenges, which is typically the driving force for joint models. The capacity to effectively combine data from several imaging modalities can offer an enhanced understanding of the patient’s status. The recognition of shortcomings in current approaches could serve as the driving force. The goal of creating a joint model could be motivated by the need to fill any gaps in the methods used today for image segmentation and registration in order to get more reliable and accurate outcomes.

To address these issues, in this paper, we propose a joint model for segmentation and registration of multi-modal images. The proposed model is based on conditional mutual information (CMI) similarity measure to adjust the spatial positions. The CMI similarity measure is flexible to non-rigid deformation [24]. For the segmentation part, the model’s energy is based on local information of the image through the kernel windows for better segmentation results. For smooth deformation field, we use linear curvature as a regularization term. The proposed model has the following advantages over the existing models. (i) This model is using Conditional Mutual Information (CMI), which is far better from MI when applied in images having noise and intensity inhomogeneity. (ii) This joint model is using local binary fittings for the selection of region of interest (ROI), which is proven to give better results in images with intensity inhomogeneity. (iii) For regularization we have used linear curvature which give smooth deformation with linear computational complexity. For detecting region of interest, we are not using length term as a regularization and does not require any re-initilization of the level set function.

The rest of the article is structured in the following way: In section 2, some classical models are reviewed and we indicate towards their constraints. Section 3 describes the variational formulation of our proposed model. In section 4 we evaluate our model experimentally and in section 5 the conclusion of this paper with a final discussion and future work is given.

2 Related work

A brief discussion of related models is given in this section, that are utilized in the experimental section for comparison.

In [14], Ibrahim et al. proposed a joint model for segmentation and registration of mono-modal images, which is the improvement of model given in [13]. Their proposed model is based on distance measure (SSD), Chen-Vese data term and linear curvature regularizer. They proposed the following model:

$\begin{matrix} min_{c_{in}, c_{out}, u (\dot{x})} J \\ = & ν_{1} \int_{Ω} | I_{0} (\dot{x}) - c_{in} |^{2} H_{*} (\hat{ψ} (\dot{x} + u (\dot{x}))) d \dot{x} \\ + & ν_{2} \int_{Ω} | I_{0} (\dot{x}) - c_{out} |^{2} (1 - H_{*} (\hat{ψ} (\dot{x} + u (\dot{x})))) d \dot{x} \\ + & γ S^{(LC)} (u) + D^{SSDH} (M, I_{0}, \hat{ψ} (\dot{x}), u (\dot{x})), \end{matrix}$ (1) where M is the moving image, I₀ is the target image and $\hat{ψ} (\dot{x})$ is the transformation vector such that $M (\hat{ψ} (\dot{x})) = I_{0} (\dot{x})$ . ν₁, ν₂ are positive constants and γ is a trade-off parameter. The similarity measure $D^{SSDH}$ weighted by the parameter ν₃ is defined as:

$\begin{matrix} D^{SSDH} (M, I_{0}, \hat{ψ} (\dot{x}), u (\dot{x})) = ν_{3} \int_{Ω} \\ (M (\dot{x} + u (\dot{x})) - I_{0} (\dot{x}))^{2} H_{*} (\hat{ψ} (\dot{x} + u (\dot{x}))) d \dot{x}, \end{matrix}$ (2) and

$S^{(LC)} (u) = \sum_{m = 1}^{2} \int_{Ω} [(Δ u_{m})^{2}] d \dot{x},$ (3) is the regularization term. Here, H_* is an approximation of the Heaviside function H, whereas the δ_* is the approximation of the Delta function, given as follows: $\begin{matrix} H_{*} (\hat{ψ} (\dot{x} + u (\dot{x}))) \\ = \frac{1}{2} (1 + \frac{2}{π} arctan \frac{\hat{ψ} (\dot{x} + u (\dot{x}))}{*}), \end{matrix}$ (4) $\begin{matrix} δ_{*} (\hat{ψ} (\dot{x} + u (\dot{x}))) \\ = \frac{*}{π (*^{2} + {(\hat{ψ} (\dot{x} + u (\dot{x})))}^{2})} . \end{matrix}$ (5) c_out and c_in in (1) are respectively the average intensity values outside and inside of the boundary Ω of the target image. Minimization of (1) with respect to c_in and c_out can be estimated as: $c_{in} = \frac{\int_{Ω} I_{0} (\dot{x}) H_{*} (\hat{ψ} (\dot{x} + u (\dot{x}))) d \dot{x}}{\int_{Ω} H_{*} (\hat{ψ} (\dot{x} + u (\dot{x}))) d \dot{x}},$ (6) and $c_{out} = \frac{\int_{Ω} I_{0} (\dot{x}) (1 - H_{*} (\hat{ψ} (\dot{x} + u (\dot{x})))) d \dot{x}}{\int_{Ω} 1 - H_{*} (\hat{ψ} (\dot{x} + u (\dot{x}))) d \dot{x}} .$ (7) For optimization and solution see [14]. The limitation of this model is that it works only in mono-modality images.

In multi-modal images, intensities in moving and target images are completely different as they are acquired by using multiple modalities. MI is the best option for a measure of similarity in multi-modal image registration. Some of well known models based on MI are [20 –22]. We give here a brief discussion about Modersitzki model [35]. He proposed the following energy functional: $J (\dot{x} + u ((\dot{x}))) = D^{MI} [M, I_{0}] + γ S^{LC},$ (8) The fitting term $D^{MI} [M, I_{0}]$ is based on mutual information which measures the similarity between the moving image M and target image I₀ and is defined by:

$\begin{matrix} D^{MI} & = & \int_{Ω} ρ_{[M]} log ρ_{[M]} ds + \int_{Ω} ρ_{[I_{0}]} log ρ_{[I_{0}]} dt \\ - & \int_{Ω^{2}} ρ_{[M, I_{0}]} log ρ_{[M, I_{0}]} d (s, t), \end{matrix}$ (9) where $ρ_{[M]} (s) = \int_{Ω} ρ_{[M, I_{0}]} (s, t) ds,$ (10) and

$ρ_{[I_{0}]} (t) = \int_{Ω} ρ_{[M, I_{0}]} (s, t) dt,$ (11) are marginal probability densities of image M and I₀ and its discretized forms are $ρ_{[M]} (s) = \frac{1}{n} \sum_{i = 1}^{n} G (s - M ({\dot{x}}_{i}), σ),$ (12) $ρ_{[I_{0}]} (t) = \frac{1}{n} \sum_{i = 1}^{n} G (t - M ({\dot{x}}_{i}), σ) .$ (13) Whereas ρ_[M,I₀] (s, t) is the joint probability density of image M and I₀ and its discretized form is given as:

$\begin{matrix} ρ_{[M, I_{0}]} (s, t) \\ = \frac{1}{n} \sum_{i = 1}^{n} G (s - M ({\dot{x}}_{i}), σ) G (t - I_{0} ({\dot{x}}_{i}), σ), \end{matrix}$ (14) where σ is the Parzen window density parameter, G is the kernel function, and n is the image size. The positions of the kernels in the moving and target images are denoted by s and t, respectively.

In [19], Ademaj et al. proposed a joint model for segmentation and registration of multi-modal images, known as regmentation model, which is the modification of model given in [14]. Their proposed model is based on linear curvature regulariser, an active contour model without edges and MI similarity measure. They proposed the following model:

$\begin{matrix} min_{c_{in}, c_{out}, u (\dot{x})} J \\ = ν_{1} \int_{Ω} | I_{0} (\dot{x}) - c_{in} |^{2} H_{*} (\hat{ψ} (\dot{x} + u (\dot{x}))) d \dot{x} \\ + & ν_{2} \int_{Ω} | I_{0} (\dot{x}) - c_{out} |^{2} (1 - H_{*} (\hat{ψ} (\dot{x} + u (\dot{x})))) d \dot{x} \\ + & γ S^{(LC)} (u (\dot{x})) + ν_{3} D^{MIH} (M, I_{0}, \hat{ψ} (\dot{x}), u (\dot{x})), \end{matrix}$ (15) where ν₁, ν₂ and ν₃ are the positive constants. γ is a trade-off parameter. In (2) I₀ be the given target image consists of two regions of constant average intensities having different values i.e. c_in and c_out. Minimization of (2) w.r.t. c_in and c_out can be estimated as in (6) and (7). Here, $S^{LC} (u (\dot{x}))$ represents the regularization term as defined in (3)and $D^{MIH}$ is the mutual information similarity measure between the foregrounds of the moving and the target images i.e. MH and I₀H, respectively. The term $D^{MIH}$ can be expressed in the from as in (9). The energy functional given in (15) is solved by using discretization and then optimization approach. For optimization and solution see [19].

3 The proposed model

Registration of multi-modality images is very important and is a challenging problem, which is usually done in variational framework through MI. MI does not compare intensities and compare their spatial positions globally. Due to this reason, it may not work very well in images having intensity inhomogeneity or noise. Conditional MI also compares spatial positions but locally, which perform better in images having intensity inhomogeneity. This motivate us to use CMI as data fitting term for registration. Doing registration of images separately may not give good results in complex medical images with noise/intensity inhomogeneity. So we propose a joint segmentation and registration model for the better registration of complex medical images. This process will decrease the computational cost of the algorithm. For the segmentation part of the model we have introduced local binary fitting energy, which perform better in images with noise/inhomogeniety. The process of combined segmentation and registration can be done in the following two ways: 1) Registration and then segmentation separately or vice versa. 2) Joint model for segmentation and registration. In the first case, segmentation of moving image is done by using a suitable segmentation model then registration model is applied on the image to get a transformed image and the final contour of the moving image is deformed to the target image. This process is explained in Fig. 1, where (a) is the segmented moving image with $\hat{ψ} (\dot{x})$ , (b) is the target image I₀ with Gaussian noise, (c) is the deformation field, (d) is the target image I₀ with $\hat{ψ} (\dot{x} + u (\dot{x}))$ and (e) is the transformed moving image $M (\dot{x} + u (\dot{x}))$ . This process may lead to poor registration results which can be observed in Fig. 1. For the second case, some models [16, 19] are proposed for joint segmentation and registration. As discussed earlier, these models have some drawbacks due to use of MI and Chan-Vese energies.

Here, a joint model for segmenting and registering of multi-modality images is proposed, which overcomes the drawbacks of the existing models by utilizing different types of energies for both segmentation and registration. The proposed model is utilizing local binary fitting data term and for registration, it is utilizing conditional mutual information. For regularization of the model, we are using linear curvature. The new proposed model has an ability to segment and register multi-modality images with intensity inhomogeneity, noise and/or weak boundaries.

Our proposed model comprises of two terms namely segmentation and registration. The term responsible for segmentation of moving images uses local information of the moving image. Binary functions are calculated in each local window for fitting the image data. This type of data fitting can segment images with weak object’s boundaries, images with intensity inhomogeneity and images have strong noise. The similarity measure is responsible for registration of images, which is based on conditional mutual information (CMI) [30]. Conditional mutual information (CMI) is introduced to estimate feature redundancy and interaction, respectively in moving and target images of multi modalities and leads towards an optimal transformation between moving and target images. In active contour methods, the level curves becomes too flat and need re-initialization to make it unit distance function. To decrease the computational cost, we regularize the level set function by using Gaussian kernel instead of re-initialization.

As a result, a joint segmentation and registration model is proposed having the following energy functional:

$\begin{matrix} min_{\hat{ψ}, {\hat{f}}_{1}, {\hat{f}}_{2}, u (\dot{x})} J (\hat{ψ}, {\hat{f}}_{1}, {\hat{f}}_{2}, u (\dot{x})) \\ = ν_{1} \int \int G_{σ} (\dot{y} - \dot{x}) {| I_{0} (\dot{x}) - {\hat{f}}_{1} (y) |}^{2} \\ H_{*} (\hat{ψ} (\dot{x} + u (\dot{x}))) d \dot{y} d \dot{x} \\ + ν_{2} \int \int G_{σ} (\dot{y} - \dot{x}) {| I_{0} (\dot{x}) - {\hat{f}}_{2} (\dot{y}) |}^{2} \\ (1 - H_{*} (\hat{ψ} (\dot{x} + u (\dot{x}))) d \dot{y} d \dot{x} \\ + ν_{3} D^{CMIH} (M, I_{0}, \hat{ψ} (\dot{x}), u (\dot{x})) \\ + γ S^{(LC)} (u (\dot{x})) . \end{matrix}$ (16) The first two terms in the functional are responsible for segmentation and the last two terms are used for registration of images. ν₁, ν₂ and ν₃ be the positive constants. γ be the trade off parameter between the CMI and linear curvature regularizer. To provide well aligned images, we have to carefully select γ because it is unknown.

$G_{σ} (\dot{x}) = \frac{1}{(2 π)^{n / 2} σ^{n}} e^{- | \dot{x} |^{2} / 2 σ^{2}},$ (17) is the Gaussian kernel. Note that $G_{σ} (\dot{x} - \dot{y})$ approaches to zero if $\dot{x}$ is far away from $\dot{y}$ , i.e. $∣ \dot{x} - \dot{y} ∣$ is large [26]. We use the regularized Heaviside function H_* which is given in (4). The numbers ${\hat{f}}_{1} (\dot{y})$ and ${\hat{f}}_{2} (\dot{y})$ in (16) are the local average intensities of the moving image near the center $\dot{y}$ . Furthermore, $S^{LC} (u (\dot{x}))$ represents the linear curvature regularizer and is defined as in (3). The similarity measure based on CMI is denoted by $D^{CMIH}$ and is defined as:

$\begin{matrix} D^{CMIH} & = & \int_{Ω^{2}} ρ_{[{MH}_{*}, I_{0} H *]} log ρ_{[{MH}_{*}, I_{0} H_{*}]} d (s, t) \\ - & \int_{Ω} ρ_{[{MH}_{*}]} log ρ_{[{MH}_{*}]} ds, \end{matrix}$ (18) where ρ_{[MH_*]} (s) and ρ_[MH,I₀H] (s, t) are marginal densities and joint densities of intensities s and t respectively, as given in (10), (11) and (14).

To discretize the energy functional in (16) we use the following:

$\begin{matrix} Ω^{\hat{h}} = {{\dot{x}}_{m, n} = ({\dot{x}}_{1, m}, {\dot{x}}_{2, n}) \\ = ((m - 0.5) \hat{h}, (n - 0.5) \hat{h})}, \end{matrix}$ where

$1 \leq m, n \leq N .$ U be the solution vector and is given by:

$U = {[\begin{matrix} u_{1} \\ u_{2} \end{matrix}]}_{2 N^{2} \times 1}, \dot{x} = {[\begin{matrix} {\dot{x}}_{1} \\ {\dot{x}}_{2} \end{matrix}]}_{2 N^{2} \times 1},$ (19) where ${\dot{x}}_{1}, {\dot{x}}_{2}$ be the position vectors and u₁, u₂ be the displacement vectors. The discrete form of the energy functional in (16) is given by:

\min_{\hat{ψ}, {\hat{f}}_{1}, {\hat{f}}_{2}, U} J^{h} = ν_{1} \sum_{m, n = 1}^{N} G_{σ} (y - x_{m, n}) | I_{0} ({\dot{x}}_{m, n}) - {\hat{f}}_{1} |^{2} H_{ϵ} (\hat{ψ} ({\dot{x}}_{m, n} + u ({\dot{x}}_{m, n}))) + ν_{2} \sum_{m, n = 1}^{N} G_{σ} (y - x_{m, n}) | I_{0} ({\dot{x}}_{m, n}) - {\hat{f}}_{2} |^{2} (1 - H_{ϵ} (\hat{ψ} ({\dot{x}}_{m, n} + u ({\dot{x}}_{m, n})))) + ν_{3} (η_{s} η_{t} \sum_{m = 1}^{n_{t}} \sum_{n = 1}^{n_{s}} ρ_{m, n} \log (ρ_{m, n} + ϵ) - η_{s} \sum_{m = 1}^{n_{t}} ρ_{m} \log (ρ_{m} + ϵ)) + γ \sum_{l = 1}^{2} \sum_{m = 1, n = 1}^{N} (- 4 u_{l} ({\dot{x}}_{m, n}) + u_{l} ({\dot{x}}_{m + 1, n}) + u_{l} ({\dot{x}}_{m - 1, n}) + u_{l} ({\dot{x}}_{m, n + 1}) + u_{l} ({\dot{x}}_{m, n - 1}))^{2},

(20)

multline where η_s = (s_n - s₀)/n_s and η_t = (t_n - t₀)/n_t, n_s and n_t shows the number of mesh points. The very small positive constant term * is used to avoid indeterminate forms. Note that the Neumann boundary conditions are used, i.e.

$\begin{matrix} u_{l} ({\dot{x}}_{m, 1}) = u_{l} ({\dot{x}}_{m, 2}), u_{l} ({\dot{x}}_{1, n}) = u_{l} ({\dot{x}}_{2, n}), \\ u_{l} ({\dot{x}}_{m, N - 1}) = u_{l} ({\dot{x}}_{m, N}), u_{l} ({\dot{x}}_{N - 1, n}) = u_{l} ({\dot{x}}_{N, n}), \\ l = 1, 2 . \end{matrix}$ Minimizing (31) with respect to ${\hat{f}}_{1}, {\hat{f}}_{2}$ by keeping other dependent variables fixed, we have:

${\hat{f}}_{1} (\dot{y}) = \frac{G_{σ} ({\dot{x}}_{m, n}) * [H_{*} (\hat{ψ} ({\dot{x}}_{m, n} + u ({\dot{x}}_{m, n}))) I_{0} ({\dot{x}}_{m, n})]}{G_{σ} ({\dot{x}}_{m, n}) * [H_{*} (\hat{ψ} ({\dot{x}}_{m, n} + u ({\dot{x}}_{m, n})))]},$ (21)

${\hat{f}}_{2} (\dot{y}) = \frac{G_{σ} ({\dot{x}}_{m, n}) * [(1 - H_{*} (\hat{ψ} ({\dot{x}}_{m, n} + u ({\dot{x}}_{m, n})))) I_{0} ({\dot{x}}_{m, n})]}{G_{σ} ({\dot{x}}_{m, n}) * [1 - H_{*} (\hat{ψ} ({\dot{x}}_{m, n} + u ({\dot{x}}_{m, n})))]},$ (22)

here * represents discrete convolution.

To get the optimal value of $\hat{ψ}$ , we take first variation of (31) with respect to $\hat{ψ}$ we have:

$Δ_{\hat{ψ}} J ({\hat{f}}_{1}, {\hat{f}}_{2}, U, \hat{ψ}) = 0 .$ (23) Solving this equation for level set function $\hat{ψ}$ and then it is regularized by using Gaussian kernel instead of re-initialization. After the minimization of (31) with respect to (u₁, u₂), the following nonlinear system of equations is obtained:

$Δ_{U} J ({\hat{f}}_{1}, {\hat{f}}_{2}, U, \hat{ψ}) = 0,$ (24) where

$\begin{matrix} Δ J & = & ν_{1} \sum_{m, n = 1}^{N} G_{σ} (y - x_{m, n}) | I_{0} ({\dot{x}}_{m, n}) \\ - {\hat{f}}_{1} |^{2} H_{*} (\hat{ψ} ({\dot{x}}_{m, n} + u ({\dot{x}}_{m, n}))) \\ + & ν_{2} \sum_{m, n = 1}^{N} G_{σ} (y - x_{m, n}) | I_{0} ({\dot{x}}_{m, n}) \\ - {\hat{f}}_{2} |^{2} (1 - H_{*} (\hat{ψ} ({\dot{x}}_{m, n} + u ({\dot{x}}_{m, n})))) \\ + & ν_{3} D^{CMIH} + γ u^{T} A u . \end{matrix}$ (25) The coefficient matrix A is given as:

$A = [\begin{matrix} P^{T} P 0 \\ 0 P^{T} P \end{matrix}],$ (26) where P be the block tri-diagonal matrix derived from the discretization of $S^{LC} (u (\dot{x}))$ . We have set U = 0 initially and the quasi-Newton method is used to solve the following system of nonlinear equations:

$H δ U = - G,$ (27) where

$G = \nabla_{u} MH (\dot{x} + u (\dot{x})) = \frac{\partial D^{CMIH}}{\partial u (\dot{x})} \frac{\partial MH (\dot{x} + u (\dot{x}))}{\partial (u (\dot{x}))},$ (28) for δU and updated U ↩ U + ζδU where ζ is the line search parameter [32] which minimizes the energy functional $J$ . Here, $H$ and $G$ be respectively the Hessian and Gradient matrices.

In general, the proposed approach can improve diagnosis accuracy by providing more accurate information about the location and boundaries of structures or abnormalities by joint image segmentation and registration. For medical professionals, this can save time and facilitate easier decision-making. The capacity of the model to handle several imaging modalities can result in a better understanding of the patient’s situation. For the purpose of diagnosis and treatment planning, integrating data from several modalities can offer a more comprehensive perspective.

Fig. 3

Block diagram of the proposed joint model.

Algorithm 1 An algorithm for joint segmentation and registration
model of multi-modal images:
1.	$(D^{CMIH}, S^{(LC)}, \hat{ψ} (\dot{x}), u (\dot{x})) \leftarrow JSRCMI (M, I_{0}, ν_{1}, ν_{2}, ν_{3}, {\hat{ψ}}^{0})$
2.	Step1. In the first step initialize $\hat{ψ} (\dot{x})$ as a binary function,
	${\hat{ψ}}^{0} = \hat{ψ} (\dot{x}, t = 0) = {\begin{matrix} - ω & if \dot{x} \in Ω_{0} - \partial Ω_{0} \\ 0 & if \dot{x} \in \partial Ω_{0} \\ ω & if \dot{x} \in Ω - Ω_{0} \end{matrix}$ (29)
	where ω > 0 is a constant, Ω₀ ⊂ Ω. ∂Ω₀ is taken as the boundary of Ω₀.
3.	U is considered as the zero initial guess for the deformation field.
4.	Step2. Multilevel method is used for minlevel = 4 and maxlevel = 7 to find the deformation field:
5.	for level= coarseLevel: finestLevel do.
6.	Compute the deformation field on each level.
	U^level← Registration $(M^{level}, {I_{0}}^{level}, U^{level}, {\hat{ψ}}^{0})$ .
7.	Quasi Newton method is utilized to obtain the following:
8.	Update ${\hat{f}}_{1} (y)$ , ${\hat{f}}_{2} (y)$ using (21) and (22).
9.	Update U^level and solve (27).
10.	Continue until the stopping criterion is satisfied.
11.	To each next finer level after registration U^level is interpolated.
12.	endfor.

4 Implementation and results

In this section, we aim to investigate the performance of the proposed model and those of other three existing models [14, 35] on medical images of different modalities. The experimental results of the proposed model are compared with three existing state of the art models qualitatively and quantitatively. The proposed model has been validated on images having intensity inhomogeneity, large deformations and high level of noise. The metric relative reduction of similarity measure is defined as follows and is used for quantitative comparison to judge the registration quality for multi-modality images.

${RR}^{CMI} = \frac{D^{C M I} (M, I_{0}, u (\dot{x}))}{D^{C M I} (M, I_{0})},$ (30) where $D^{C M I} (M, I_{0}, u (\dot{x}))$ is the conditional mutual information distance measure term after registration of the transformed moving image and target image, and $D^{C M I} (M, I_{0})$ shows the conditional mutual information distance measure before registration. To prove the efficiency of the proposed model with those of other existing models, Jaccard similarity index is used for the quantitative comparison. The Jacobian matrix ( J ) is given by:

$J = [\begin{matrix} 1 + \frac{\partial u_{1}}{\partial {\dot{x}}_{1}} & \frac{\partial u_{1}}{\partial {\dot{x}}_{2}} \\ \frac{\partial u_{2}}{\partial {\dot{x}}_{1}} & 1 + \frac{\partial u_{2}}{\partial {\dot{x}}_{2}} \end{matrix}], P = \min (\det (J)),$ (31) In all of our experiments, the regularization parameters are tuned in such away that P > 0, which shows bending and breaking free deformation field. Here, $\frac{\partial u_{1}}{\partial {\dot{x}}_{1}}$ and $\frac{\partial u_{1}}{\partial {\dot{x}}_{2}}$ denotes the gradient of u₁ whereas $\frac{\partial u_{2}}{\partial {\dot{x}}_{1}}$ and $\frac{\partial u_{2}}{\partial {\dot{x}}_{2}}$ denotes the gradient of u₂. Experiments were done by using Matlab 2018b with Intel Core i7, 7th Gen with 8 GB of RAM.

In Fig. 1a, the proposed model has produced satisfactory results and successfully jointly segment and register the noisy images whose output results are given in Fig. 1a (c)-(e). Parameters used have the values ν₁ = 2, ν₂ = 2, ν₃ = 0.9 and γ = 0.4. Stopping Criterion: One of the most commonly used stopping criterion to judge whether the moving image has been correctly deformed to the target image is the cross correlation coefficient. However, such a similarity metric does not work well in this CT-CBCT images. We therefore use a convergence criterion based on the difference between successive deformation fields. We define a relative norm

${Err}^{(k)} = Σ | u^{k + 1} | / Σ | u^{k} |$ (32) we use Err^(k-10) - Err^(k) ≤ * where * = 1.0 × 10^-4 as our stopping criterion [28].

4.1 Experimental results of the proposed model

Here, we have tested our proposed model on four different multi-modal medical images (Thorax, Brain and Liver [19]) of size 128 × 128. In Fig. 4 we have shown a successful performance of our proposed model in terms of segmentation and registration for images having large deformations, severe intensity inhomogeneity and weak boundaries. In Fig. 4 the first column shows moving images M with the contour Γ, the second column shows the target images I₀, the third column shows the deformation fields with the values of P > 0, means no mesh folding and cracking occurs. The fourth column shows the target images I₀ with the final contour of the transformed moving images $\hat{ψ} (\dot{x} + u (\dot{x})) = 0$ and in the fifth column we have given the transformed moving images $M (\dot{x} + u (\dot{x}))$ . For optimal results, we have set the parameters according to the images.

Fig. 4

Results of the proposed model for different multi-modality medical images. The first and second columns shows the given moving images with initial contours and target images, respectively. The third and fourth columns shows the deformation fields and the target images with the final contours, respectively. The fifth column shows the transformed moving images.

4.2 Quantitative comparison

In this section, we discuss quantitative comparison of the proposed model with the above mentioned existing models say Modersitzki model [35] and regmentation model [19] through Jaccard Similarity Index (JSI) and relative reduction of similarity measure. For testing, we have used different type of synthetic and medical images.

It must be noted that the value of JSI always lies between 0 and 1, where 0 means no overlapping and 1 means complete alignment. The JSI value closer to 1 indicates that the transformed moving image is quite similar to the target image, whereas, JSI value closer to 0 shows a low resemblance. Table 1, shows the JSI values of the proposed model and two other existing models given in [19, 35]. From Table 1, it can be observed that the proposed model has outperformed the existing models quantitatively as the JSI values of the proposed model are higher than the existing models.

Table 1
JSI values of the proposed model in comparison with the Regmentation model [19] and the Modersitzki model [35]

Image data Proposed model Regmentation model Modersitzki model

Thorax 0.7407 0.7400 0.6696

Brain(1) 0.8038 0.7041 0.7248

Liver 0.9043 0.9000 0.8822

Brain(2) 0.9689 0.9534 0.9391

Image data	Proposed model	Regmentation model	Modersitzki model
Thorax	0.7407	0.7400	0.6696
Brain(1)	0.8038	0.7041	0.7248
Liver	0.9043	0.9000	0.8822
Brain(2)	0.9689	0.9534	0.9391

We have also compared the results of proposed model with other existing models by using comparison metric relative reduction which is defined in (30). In Table 2, we have given the relative reduction value of the similarity measure of the proposed model and other existing models. Table 2, shows that the relative reduction values for the proposed model is smaller than the other existing models which shows the better performance of the proposed model.

Table 2

Relative Reduction (RR) values of the proposed model in comparison with the Regmentation model [19] and the Modersitzki model [35]

Images	RR values of the proposed model	RR values of the Regmentation model	RR values of the Modersitzki model
Thorax	0.907374	2.15547	1.07000
Brain(1)	0.763422	0.953374	0.99918
Liver	0.98055	1.21200	1.20000
Brain(2)	0.99270	1.02000	1.19000

Computational Complexity: It can be seen from (16) that it contains three different terms. One term is based on CMI which is given in (18), and is based on two entropies terms each one has linear time computational complexity i.e $O (n)$ [33]. The other term is the segmentation term which is based on local binary fitting term which has also linear time computational complexity i.e $O (n)$ . The third term is the regularization term, which is based on linear curvature which has also linear time computational complexity [34]. So the total time computational complexity of the model is $O (n)$ . Same behaviour of the model can be observed from the Table 3.

Table 3

Table of the proposed model by using images of different resolutions

Images [36]	Resolution/Time(sec)
	32×32	64×64	128×128	256×256	512×512
1	41	85	333	900	2200
2	45	86	360	1000	2400
3	43	82	376	850	2300
4	42	77	486	800	1980
5	41	85	334	900	2200
6	34	83	358	865	2150
7	40	81	350	822	2000
8	50	90	374	1100	2500
9	49	81	383	970	2000
10	46	88	367	1070	2290

4.3 Qualitative comparison

Here, we have compared the proposed model with the existing models on a variety of synthetic and real-medical images. We have included a few of them below.

On multi-modal medical images with intensity inhomogeneity and weak boundaries, Fig. 5 illustrates the comparative results of the proposed model with the Modersitzki model [35]. In Fig. 5 the first and second columns shows the moving and target images respectively. The third column shows the moving images M with final contours. The fourth column shows the deformation fields, the fifth column shows target images I₀ with final contours and the sixth column shows the transformed moving images of the proposed model. The seventh and eighth columns are the results of Modersitzki model [35]. In all of these experimental results, it can be observed visually that the proposed model has produced better results than the Modersitzki model [35] on multi-modal medical images.

Fig. 5

Comparative results of the proposed model with the Modersitzki model [35]. (a) Moving images M. (b) Target images I₀. (c) Moving images M, with zero level set. (d) The deformation fields. (e) Images I₀ with final contour. (f) The transformed moving images $M (\dot{x} + u (\dot{x}))$ , shows the results of the proposed model. (g) The deformation fields and (h) shows the transformed moving images $M (\dot{x} + u (\dot{x}))$ , of the Modersitzki model [35].

Fig. 6

Comparative results of the proposed model with the regmentation model [19]. First row shows the results of the proposed model. Second row shows the results of the regmentation model.

Fig. 7

Comparative results of the proposed model with the regmentation model [19]. First row shows the results of the proposed model. Second row shows the results of the regmentation model.

Fig. 8

Comparative results of the proposed model with the regmentation model [19]. First row shows the results of the proposed model. Second row shows the results of the regmentation model.

Fig. 9

Comparative results of the proposed model with the regmentation model [19]. First row shows the results of the proposed model. Second row shows the results of the regmentation model.

In Figs. 6–9, the first and second rows shows the qualitative comparative results of proposed model and the regmentation model [19], respectively on four different medical images having intensity inhomogeneity. In Figs. 6–9, (a) shows the moving images M, (b) shows the target images I₀, (c) show the moving images M with zero level set $\hat{ψ} (\dot{x})$ , (d) shows the transformations $(\dot{x} + u (\dot{x}))$ with P = 0.13764 > 0, P = 0.15666 > 0, P = 0.570 > 0 and P = 0.50497 > 0, respectively, (e) shows I₀ with final contour $\hat{ψ} (\dot{x} + u (\dot{x}))$ and (f) shows the transformed moving images $M (\dot{x} + u (\dot{x}))$ , RR^CMI = 0.907374, RR^CMI = 0.763422, RR^CMI = 0.98055 and 0.99270, respectively.

In Figs. 8–9, (a) shows the moving images M, (b) shows the target images I₀, (c) show the moving images M with zero level set $\hat{ψ} (\dot{x})$ ,(d)shows the transformations $(\dot{x} + u (\dot{x}))$ with P = 0.2891 > 0, P = 0.15666 > 0, P = 0.948 > 0 and P = 0.92398 > 0, respectively, (e) shows I₀ with final contour $\hat{ψ} (\dot{x} + u (\dot{x}))$ and (f) shows the transformed moving images $M (\dot{x} + u (\dot{x}))$ with RR^MI = 2.15547, RR^MI = 0.9553374, RR^MI = 1.212 and RR^MI = 1.0200, respectively. From visual view point, it can be observed that the results of proposed model are efficient than the results of the regmentation model [19].

4.4 CHAOS dataset

Additionally, we evaluated our proposed model using the CHAOS dataset (Combined (CT-MR) Healthy Abdominal Organ Segmentation) [36]. The collection contains the ground truth segmentation of the abdominal organs, i.e. (liver, kidney, and spleen etc). For the joint registration and segmentation model, we employed the full T1-DUAL and T2-SPIR (Spectral Pre-Saturation Inversion Recovery) abdominal MRI sequences. A 1.5T MRI scanner was used to collect the dataset. The target image is T2-SPIR and the moving image is T1-DUAL in mode. MRI sequences have a resolution of 256 × 256. To achieve a resolution of 128 × 128, the images are further down-scaled.

In this dataset, we have carried out our experiments on 10 cases. We find the relative reduction applied directly on the gray-scale images of the target and the transformed moving image. Moreover, the Dice Similarity Coefficient (DSC) and Hausdorff distance ( $H d$ ) is found for the evaluation of the internal anatomical structure registration through segmentation. For this purpose, liver is considered and the ground truth labeled data is used. The transformations obtained as a result of registration process is applied to the ground truth image of moving liver. DSC and $H d$ is measured between the transformed ground truth liver moving image and the ground truth liver target image. The outliers in the image have an impact on the $H d$ . After removing the outliers from the segmented images using morphological operations (erosion and dilation of the same structural element), we calculated the $H d$ .

The proposed approach’s evaluation is compared with the other models. The average computational cost of the proposed model is 372 seconds whereas the average computational cost of the regmentation and Modersitzki models are 1433 seconds and 1670 seconds respectively. Which shows that the computational cost of the proposed model is less than the existing state of the art models as shown in Table 4.

Table 4
Comparison of the proposed model with the regmentation model and Modersitzki model in terms of CPU time

Images [36]		CPU time/sec
	Proposed Model	Regmentation Model	Modersitzki Model
1	1l333	1l1479	1500
2	1l360	1l1300	1650
3	1l376	1l1350	1523
4	1l486	1l1450	1724
5	1l334	1l1500	1543
6	1l358	1l1400	1645
7	1l350	1l1360	1789
8	1l374	1l1500	1723
9	1l383	1l1500	1799
10	1l367	1l1487	1800

Also, the quantitative comparison of the proposed model with the regmentation and Modersitzki model using this dataset is presented in Table 5 and Figs. 10–12. The higher value of DSC and lower values of $H d$ and RR shows better performance. DSC and $H d$ are defined as:

$DSC = \frac{2 (M (\dot{x} + u (\dot{x}))_{s} ⋂ I_{0} s)}{∣ M (\dot{x} + u (\dot{x}))_{s} ∣ + ∣ I_{0 s} ∣},$ (33) with its values in [0, 1] range. The higher values near to one shows perfect segmentation. Here, I₀s and $M (\dot{x} + u (\dot{x}))_{s}$ are the segmented target and transformed moving images, respectively. $H d (X, Y) = max_{\dot{x} \in X} {min_{\dot{y} \in Y} {d (\dot{x}, \dot{y})}},$ (34) where $\dot{x} \in X$ and $\dot{y} \in Y$ , respectively, and $d (\dot{x}, \dot{y})$ is the Euclidian distance between $\dot{x}$ and $\dot{y}$ . The smaller $H d$ value gives the best match of the transformed moving and target images. We have calculated MI over CMI for the comparison purpose.

We also find the statistical measures of CHAOS dataset to show the statistical significance of our proposed model. We calculated the means and standard deviations of our proposed model, the regmentation model and the Modersitzki model. The DSC is computed on CHAOS dataset images given in Table 5 and the following results are obtained in the form (mean±s.d), (0.9085 ± 0.0241), (0.8586 ± 0.0242), (0.8621 ± 0.0281), respectively, for the proposed model, regmentation model and Modersitzki model. This clearly shows that the mean of the DSC values of our proposed model is high as compared to the other existing models. Furthermore, the standard deviation of DSC of our proposed model is low. The high mean DSC and low standard deviation DSC of our proposed model is statically significant as compared to the regmentation and Modersitzki models. This clearly indicates the better performance of our proposed model on various MR images from the CHAOS dataset.

Table 5

Comparison table of the proposed model with the existing state of the art models using DSC-Liver, Hd-Liver and RR

Images		DSC_Liver			Hd_Liver			RR
	Proposed Model	Regmentation Model	Modersitzki Model	Proposed Model	Regmentation Model	Modersitzki Model	Proposed Model	Regmentation Model	Modersitzki Model
1	1l0.9192	1l0.8669	0.8687	1l4.1231	1l7	7	1l0.9723	1l0.9725	0.997
2	1l0.9048	1l0.8556	0.8594	1l5	1l7	7	1l0.9792	1l0.9955	0.9961
3	1l0.9501	1l0.8779	0.8883	1l3	1l7	7	1l0.9805	1l0.9887	0.9963
4	1l0.9342	1l0.8852	0.8967	1l4.2426	1l7	6.3246	1l0.979	1l0.978	0.9966
5	1l0.9292	1l0.8765	0.8789	1l5.8828	1l6	6.0828	1l0.9791	1l0.9882	0.9969
6	1l0.8969	1l0.8797	0.8822	1l8	1l8	8	1l0.9764	1l0.9825	0.9962
7	1l0.8633	1l0.8637	0.8669	1l9.3619	1l9.4868	9.4868	1l0.9639	1l0.9641	0.9966
8	1l0.8886	1l0.8125	0.8068	1l5	1l7.6158	7.2801	1l0.9538	1l0.9862	0.9961
9	1l0.9006	1l0.832	0.8309	1l6.3246	1l8.6023	7.6158	1l0.9478	1l0.9707	0.997
10	1l0.903	1l0.8365	0.8419	1l7.2801	1l8.4853	8	1l0.954	1l0.9703	0.9962

Fig. 10

Comparison of the proposed model with Regmentation model [19] and Modersitzki model [35] using Dice Similarity.

Fig. 11

Comparison of the proposed model with Regmentation model [19] and Modersitzki model [35] using Hausdorff distance values.

Fig. 12

Comparison of the proposed model with Regmentation model [19] and Modersitzki model [35] by using Relative Reduction of Similarity measure.

4.5 Parameter sensitivity analysis

In this section, we discuss about the sensitivity of the proposed joint model on a parameter γ. γ is one of the important parameters of the proposed model that can effect the registration results of the medical images. Figure 13 shows the parameter sensitivity of our proposed model. The regularization parameter here is γ and is applied on the multimodal images taken from CHAOS dataset [36]. The joint registration and segmentation is performed with varying γ values to analyze the parameter sensitivity. The dice similarity coefficient is recorded versus γ. From the Fig. 13 it is clear that it have a high peak of DSC value at 0.9501 at γ = 1.05. The range of γ is taken as [1 2]. Also we have minimum peak of DSC 0.9436 at γ = 1.75. All the values of DSC in the range of γ values [1 2] lies in between 0.9436 and 0.9501. The resultant DSC values show the results are highly correlated even in case of varying γ parameter. However, we choose the best performing parameter γ for joint registration and segmentation.

Fig. 13

Sensitivity of the proposed model on parameter γ vs Dice Similarity Coefficient for image 3 taken from CHAOS dataset [13].

5 Conclusion

In this paper, a new joint model for the segmentation and registration of multi-modality images having severe intensity inhomogeneity/noise/weak boundaries is proposed. For segmentation purposes, the model will be utilizing local binary fitting data term and for registration, it is utilizing conditional mutual information. For regularization of the model, we are using linear curvature. The new proposed model is capable of segmenting and registering multi-modal images having intensity inhomogeneity, noise and/or weak boundaries. The proposed model is compared quantitatively and qualitatively with the existing state of the art models through statistical metrics like Jaccard similarity index, relative reduction of similarity measure, Dice Similarity measure and Hausdorff distance. The relative reduction of similarity measure (RR^CMI) is computed to evaluate the registration quality of multi-modality images. We have used MI over CMI for the comparison purpose. We have also tested our proposed model on CHAOS Dataset. For both comparison types, it can be observed that the proposed model has produced better results than the other existing models.

6 Future work

Future work may include optimization of the parameters of the regularisation terms as well as a further modification which will include the use of the Gaussian curvature term as regularizer. Developing fast numerical methods for the solution of joint segmentation and registration models to deal with large size medical images.

References

Mahapatra

, Li

, Vos

and Buhmann

, Joint segentation and registration of cardiac DCE MRI using sparse data representation, IEEE 12th International Symposium on Biomed. Imaging (ISBI), Brooklyn, NY, USA (2015), 1312–1315.

Schnabel

J.A.

, Heinrich

M.P.

, Papiez

B.W.

and Brady

S.J.M.

, Advances and challehges in deformable image registration: From image fusion to complex motion modelling, Med. Image Anal. Elsevier 33 (2016), 145–148.

Pohl

K.M.

, Fisher

, Grimson

W.E.L.

, Kikinis

and wells

W.M.

, A bayesian model for joint image segmentation and registration, Neuro Image, Elsevier 31 (2006), 228–239.

Chaoui

, Hamitouche

, Stindel

and Roux

, Recognition-based segmentation and registration method for image guided shoulder surgery, 33rd Annual International Conference of the IEEE EBMS. Boston, Massachusetts USA, in Med. Biology Society (2011), 6212–6215.

Ravasi

and Birnie

, A joint inversion-segmentation approach to assisted seismic interpretation, arXive preprint arXive:, (2102), 03860.

begum

, Badshah

, Ibrahim

, Ashfaq

, Minallah

and Atta

, On two algorithms for multi-modality image registration based on Gaussian curvature and application to medical images, IEEE Access 9 (2021), 10586–10603.

Dydenko

, Friboulet

, Magnin

I.E.

A variational framework for affine registration and segmentation with shape prior: application in echocardiographic imaging, IEEE Workshop on V. G. L. S. M. in Comp. Vision, Citeseer, 2003.

Yezzi

, Zollei

and Kapur

, A variational framework for joint segmentation and registration, Proceedings IEEE Workshop on Mathematical Methods in Biomed. Image Anal. (MMBIA 2001) (2001), 44–51.

Alvarez

, Weickert

and Sanchez

, Reliable estimation of dense optical flow fields with large displacements, International Journal of Computer Vision, Springer 39(1) (2000), 41–56.

10.

Unal

and Slabaugh

, Coupled PDEs for non-rigid registration and segmentation, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) 1 (2005), 168–175.

11.

Gooya

, Pohl

K.M.

, Bilello

, Biros

and Davatzikos

, Joint segmentation and deformable registration of brain scans guided by a tumor growth model, Toronto, Canada, Springer, 14th International Conference on Med. Image Computing and Computer-Assisted Intervention (2011), 532–540.

12.

Wang

and Vemuri

B.C.

, Simultaneous registration and segmentation of anatomical structures from brain MRI, Proceedings of the 8th International Conference on Med. Image Computing and Computer-Assisted Intervention, Springer-VerlagBerlin, Heidelberg (2005), 17–25.

13.

Guyader

C.L.

and Vese

L.A.

, A combined segmentation and registration framework with a nonlinear elasticity smoother, Computer Vision and Image Understanding, Elsevier 115(12) (2011), 1689–1709.

14.

Ibrahim

, Chen

and Rada

, An improved model for joint segmentation and registration based on linear curvature smoother, J. of Algorithms and Computational Technology, SAGE Publications Sage UK: London, England 10(4) (2016), 314–324.

15.

, Yang

, Liu

, Coatrieux

J.L.

and Shu

, Multimodal medical image registration based on an informationtheory measure with histogram estimation of continuous image representation, Mathematical Problems in Engineering, Hindawi 2018, 2018.

16.

Aganj

and Fischl

, Multimodal image registration through simultaneous segmentation, IEEE Signal Processing Letters 24(11) (2017), 1661–1665.

17.

Azam

M.A.

, Khan

K.B.

, Ahmad

and Mazzara

, Multimodal medical image registration and fusion for quality enhancement, CMC-Computers Materials and Aterials and Continua, TECH. Science Press 871 Coronado Center DR, SUTE 200, HENDERSON, NV 89052 USA 68(1) (2021), 821–840.

18.

Droske

and Rumpf

, Multiscale joint segmentation and registration of image morphology, IEEE Transactions on Pattern Analysis and Machine Intelligence 29(12) (2007), 2181–2194.

19.

Ademaj

, Rada

, Ibrahim

and Chen

, A variational joint segmentation and registration framework for multimodal images, 23rd Annual Conference on Med. Image Understanding and Analysis proceedings, Liverpool, UK, Springer (2019), 305–316.

20.

Wells

W.M.

III , Viola

, Atsumi

, Nakajima

and Kikinis

, Multi-modal volume registration by maximization of mutual information, Med. Image Anal. 1(1) (1996), 35–51.

21.

Viola

and Wells

W.M.

, III, Alignment by maximization of mutual information, IJCV 24(2) (1997), 137–154.

22.

Maes

, Collignon

, Vandermeulen

, Marchal

and Suetens

, Multi-modality image registration by maximization of mutual information, IEEE Trans. on Med. Imaging 16(2) (1997), 187–198.

23.

Chan

and Vese

, An active contour model without edges, (1999), 141–151.

24.

Loeckx

, Slagmolen

, Maes

, Vandermeulen

and Suetens

, Nonrigid image registration using conditional mutual information, IEEE Trans. on Med. Imaging 29(1) (2010), 19–29.

25.

Debroux

and Guyader

C.L.

, A joint segmentation /registration model based on a nonlocal characterization of weighted total variation and nonlocal shape descriptors, SIAM Journal on Imaging Sciences 11(2) (2018), 957–990.

26.

, Kao

, Gore

J.C.

and Ding

, Implicit active contour driven by local binary fitting energy, Minnesota, USA, IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis (2007), 1–7.

27.

Modersitzki

and Fischer

, Curvature based image resgtration, J. Math. Imaging Vis. 18(1) (2003), 81–85.

28.

Zhen

, Gu

, Yan

, Zhou

, Jia

and Jiang

S.B.

, CT to cone-beam CT deformable registration with simultaneous intensity correction, Physics in Medicine and Biology 57(21) (2012), 6807.

29.

Dirk

, Pieter

, Frederik

, Dirk

and Paul

, Nonrigid image registration using conditional mutual information, IEEE Trans. on Med. Imaging 29(1) (2009), 19–29.

30.

Cover

T.M.

, Thomas

J.A.

Elements of information theory, 2nd ed. New York: Wiley, 2006.

31.

D.H.

and Fukushima

, On the global convergence of the BFGS method for nonconvex unconstrained optimization problems, SIAM Journal on Optimization 11(4) (2011), 1054–1064.

32.

Jorge

, Wright

Numerical optimization, Springer Science and Business Media, 2006.

33.

Liu

, Jiang

and Xu

, A super fast algorithm for estimating sample entropy, Entropy, MDPI 24(4) (2022), 524.

34.

Gong

and Sbalzarini

I.F.

, Curvature filters efficiently reduce certain variational energies, IEEE Trans. on Image Processing 26(4) (2017), 1786–1798.

35.

Modersitzki

FAIR: flexible algorithms for image registration, SIAM, 2009.

36.

Kavur

A.E.

, Gezer

N.S.

, Baris

, Aslan

, Conze

, Groza

, Pham

D.D.

, Chatterjee

, Ernst

, Ozkan

, BayBaydar

, Lachinov

, Han

, Pauli

, Isensee

, Perkonigg

, Sathish

, Rajan

, Sheet

, Dovletov

, Speck

, Nurnberger

, Maier-Hein

K.H.

, Akar

G.B.

, Unal

, Dicle

and Selver

M.A.

, Challenge - combined (CT-MR) healthy abdominal organ segmentation (CHAOS), Med. Imag. Analysis, Elsevier BV 69 (2021), 101950.

37.

Mahajan

, Mittal

and Pandit

A.K.

, Image segmentation approach based on adaptiveflower pollination algorithm and type II fuzzy entropy,–, Multimedia Tools and Applications 82 (2023), 8537–8559.

New improved model for joint segmentation and registration of multi-modality images: application to medical images

Abstract

Keywords

1 Introduction

Table 1 JSI values of the proposed model in comparison with the Regmentation model [19] and the Modersitzki model [35] Image data Proposed model Regmentation model Modersitzki model Thorax 0.7407 0.7400 0.6696 Brain(1) 0.8038 0.7041 0.7248 Liver 0.9043 0.9000 0.8822 Brain(2) 0.9689 0.9534 0.9391

Table 4 Comparison of the proposed model with the regmentation model and Modersitzki model in terms of CPU time

6 Future work

References

Table 4
Comparison of the proposed model with the regmentation model and Modersitzki model in terms of CPU time