Deep learning supported disease detection with multi-modality image fusion

Abstract

Multi-modal image fusion techniques aid the medical experts in better disease diagnosis by providing adequate complementary information from multi-modal medical images. These techniques enhance the effectiveness of medical disorder analysis and classification of results. This study aims at proposing a novel technique using deep learning for the fusion of multi-modal medical images. The modified 2D Adaptive Bilateral Filters (M-2D-ABF) algorithm is used in the image pre-processing for filtering various types of noises. The contrast and brightness are improved by applying the proposed Energy-based CLAHE algorithm in order to preserve the high energy regions of the multimodal images. Images from two different modalities are first registered using mutual information and then registered images are fused to form a single image. In the proposed fusion scheme, images are fused using Siamese Neural Network and Entropy (SNNE)-based image fusion algorithm. Particularly, the medical images are fused by using Siamese convolutional neural network structure and the entropy of the images. Fusion is done on the basis of score of the SoftMax layer and the entropy of the image. The fused image is segmented using Fast Fuzzy C Means Clustering Algorithm (FFCMC) and Otsu Thresholding. Finally, various features are extracted from the segmented regions. Using the extracted features, classification is done using Logistic Regression classifier. Evaluation is performed using publicly available benchmark dataset. Experimental results using various pairs of multi-modal medical images reveal that the proposed multi-modal image fusion and classification techniques compete the existing state-of-the-art techniques reported in the literature.

Keywords

Image fusion ABF DCNN CT MRI multi-modal

1 Introduction

With the recent advancements in imaging techniques, the internal structure of human brain can be easily visualized. Each image modality provides unique information about the human brain. Computed Tomography (CT) images provide information about the structure of the bones and the Magnetic Resonance Imaging (MRI) images give information about the soft-tissues [25]. Fusion of these two images provides excellent information that can be used for reliable tumor diagnosis. The key steps involved in the brain disease detection are registration [7 , 20], fusion [10 , 35] and classification [15, 21].

Image registration was done using deep learning and affine transformation in [7]. Here, multiple ConvNet units were stacked to achieve registration. Both affine and deformable image registration was achieved in this paper. A survey on different registration techniques was presented in [20]. Analysis was done based on 9 different registration criteria like dimensionality, domain, registration basis, transformation, interaction, object, modalities, subject and optimization. A scheme for optimization of registration was given in [19]. The proposed optimization technique was based on the modified particle swarm optimization technique. A technique for medical image fusion was proposed in [18]. Here, conjugate gradient technique was used to optimize the coefficients. CT and MRI image fusion using fuzzy logic was presented in [35]. Image was decomposed to low and high bands by using discrete wavelet transform. Fuzzy logic was used for fusion of low bands and average fusion was used for the high frequency bands. Both traditional and hybrid fusion system was presented in [20]. All the techniques were analyzed using fusion metrics like entropy, gradient, etc. Image fusion based on CNN was presented in [30]. This system used both NSST and spiking cortical model for the fusion of CT and MRI images. Segmentation of abnormal intensities from brain structures is accomplished to extract tumour properties like size, shape and location [14 , 38]. Tumor classification using MRI images was proposed in [21]. Here, feature extraction was done using discrete wavelet transform. The extracted features were classified using SVM [24]. A hybrid approach for medical image classification was proposed in [15]. Here features were extracted using DWT, feature reduction was done using genetic algorithm and classification was done using SVM.

The main contributions of this paper are

A new medical image classification methodology based on image fusion is presented.

A novel image enhancement technique using Energy based CLAHE (ECLAHE) is proposed.

A novel weighted medical image fusion algorithm based on Siamese Neural Network and Entropy (SNNE) is proposed.

Classification of fused medical images using logistic regression modelling is presented.

Quantitative evaluation using various metrics is presented using publicly available dataset.

In earlier work only single modality image considered, in this work fused images are used to improve disease detections.

2 Literature survey

A framework for the fusion of MRI [28, 36], and CT images was proposed by Hou et al. [10]. The authors employed convolutional neural network architecture for the fusion. In addition, a dual-channel spiking model was also employed in this paper. The entire image was decomposed using Non-Sub sampled Shearlet Transform (NSST). At first, source image was decomposed at three levels. Then the low-frequency components were fused using convolutional neural network architecture. Later, using the convolutional neural network, a weight map was generated. The weights in the weight map were designed based on the activity level measurements. The high frequency band coefficients were fused using dual-channel spiking model. The dual-channel was employed to raise the performance of detail extraction in the dark image regions. Finally, the fused image was obtained by using the inverse NSST transform.

Huang et al. [11] proposed frog leaping algorithm for image fusion using. In this paper, two schemes were combined to form a fusion technique. The first scheme comprised of shuffled frog leaping scheme and the second was the pulse coupled neural network structure. The SPECT input image was decomposed into hue, saturation and intensity. The intensity image was then decomposed using NSST transform. However, NSST transform was directly applied to the CT image. Finally, inverse NSST was applied to get the intensity layer. Then, inverse HIS operation was used to get the final fused image. In the frog leaping algorithm, the fitness of each frog was computed and arranged in the descending order. This system produced good fusion results Xia et al. [13] proposed a scheme for the detection of brain tumor using medical image fusion. In this paper, two types of features were used. The first type was the multi-scale transform features and the second type was the deep convolutional neural features. The input image was decomposed into sub-images using two filters namely, the Gauss Laplace filter and the Gaussian filter. The initialization of the kernel for convolution was performed using Hek technique. Using the initialization, the basic unit was generated. The basic unit was then trained using the back-propagation technique. Many such units were stacked together to form a deep stacked structure. Finally, the fused coefficients were used to generate the fusion result. Following is a summary of the previously related studies in this research topic reported in the literature.

Author	Technique used	Findings	Focus/Comments
Hou et al. [10]	Convolutional neural network	A framework for the fusion of MRI and CT images	•The entire image was decomposed using Non-Sub sampled Shearlet Transform (NSST). At first, source image was decomposed at three levels.
			•The dual-channel was employed to raise the performance of detail extraction in the dark image regions. Finally, the fused image was obtained by using the inverse NSST transform
Huang et al. [11]	back-propagation	Proposedfrog leaping algorithm	•The first scheme comprised of shuffled frog leaping scheme and the second was the pulse coupled neural network structure.
			•The SPECT input image was decomposed into hue, saturation and intensity.The intensity image was then decomposed using NSST transform.
Xia et al. [13]	Hek technique	proposed a scheme for the detection of brain tumor using medical image fusion.	•The first type was the multi-scale transform features and the second type was the deep convolutional neural features.
			•The input image was decomposed into sub-images using two filters namely, the Gauss Laplace filter and the Gaussian filter.
Rajalingam et al. [3]	fuzzy and the neural network	proposed a framework for medical image fusion using neuro-fuzzy hybrid scheme	•Fuzzy logic rule was applied to the coefficients in which the coefficients were mapped to a fuzzy membership value that ranged from 0 to 255.
			•The output of this level was fed to the neural network structure. The obtained output was given to the neural network. Then, the final output was obtained using the neuro-fuzzy logic system.
Ouerghi et al. [23, 40]	pulse couple neural network (PCNN) structure	proposed a scheme for the fusion of MRI and PET images	•This system was designed to retain the functional characteristics of the PET image and also to retain the spatial details of the MRI image.
			•The input PET image was first converted to YIQ model. Adaptive-linking scheme was used for the fusion of high frequency bands.
Rajinikanth et al. [31]	Watershed segmentation	proposed image fusion using DWT and PCA.	•The MRI images were evaluated based on initial fusion, thresholding and tumor segmentation. The Shannon thresholding technique was done based on the social group optimization.
			•It involved two stages namely, the improving stage and the acquiring stage. In the improving stage, the locations were synchronized and in the acquiring stage the solutions were obtained.
Rajalingam et al. [29]	•Discrete cosine harmonic wavelet transforms •PCNN	proposed a scheme for identification of disease from the multimodal image data using hybrid intelligence system	•Initially, DCHWT was applied to the input images to decompose them into low and high frequency components. •The inverse DCHWT was then applied to two multi-modal images. The output of IDCHWT was then fed to the PCNN model.
Hermessi et al. [9]	Non Sub-sampled Contourlet (NSCT) transform	presented multimodal image fusion using convolutional neural networks	•Positive pair and negative pair patches of coefficients based on shearlet transform were used for training the neural network structure.
			•The feature mapping was then done based on the cross-correlation. Using the cross-correlation, a similarity metric was generated.
Zhu et al. [47]	Presented multi-modal fusion using sparse theory	K-SVD algorithm	•a novel dictionary learning technique was proposed for creating dictionaries
			•The dictionary had best representative features. These patches were then clustered into several groups based on local density peaks.s
Asokan et al. [2]	Proposed a scheme for the medical image fusion using stationary wavelet transform (SWT)	Principal Component Analysis(PCA)	•Two images belonging to different modalities are first decomposed using SWT. Different fusion rules were applied for the approximate and detail coefficients.
			•These coefficients were fused using principal component analysis and then reconstructed using inverse stationary wavelet transform.
Ramdlon et al. [32]	k-nearest neighbor (k-NN)	proposed a technique for the classification of images	•The images were first pre-processed using resize operation, cropping operation and filtration using median filter.
			•The filtered image was then segmented using morphological operator followed by binarization operation.
Zaw et al. [44]	Naïve Bayes (NB) classifier	presented a framework for detection of brain tumor	•The MRI images were first pre-processed using morphological operation with disk operator.
			•To separate the skull region from the entire image, the pixel subtraction operation was used. Segmentation was then performed using maximum entropy metric.
Anitha et al. [1]	random forest (RF) classification algorithm	proposed a scheme for the brain tumor recognition	•The MRI images were segmented and classified using computer aided methodology. The input noisy images were filtered using adaptive median filter.
			•Using the filtered image, the texture features and grey level co-occurrence matrix features were extracted.
Chinnu [6]	support vector machine (SVM) classifier	proposed a framework for tumor classification	•The tumor region was then segmented using histogram-based segmentation technique.
			•Then from the segmented regions, features were extracted whose dimension was reduced using PCA.
Feng et al. [8]	dual residual convolution neural network (DRCNN)	compared RED-CNN and DP-ResNet	•Feature extraction is performed simultaneously on both the sinogram-domain sub-net and the image-domain sub-net, which utilize the residual shortcut networks and play a complementary role in suppressing the projection noise and reducing image error.
Khan et al. [14]	directional weighted optimized Fuzzy C-Means (dwsFCM)	perform precise segmentation	•Directional weighted optimized Fuzzy C-Means (dwsFCM) method has been proposed for segmentation of brain MR images.
			•This method works by incorporating the spatial information of the pixels of the images and assigning the directional weights to the neighborhood.

3 Proposed methodology

The proposed methodology comprises of steps like image pre-processing (filtering and enhancement), registration, fusion, segmentation, extraction of relevant features and final classification. This is depicted in Fig. 1. The noisy MRI and noisy CT images are first obtained. These data are pre-processed using filtering and enhancement. Filtering is done using modified 2D Adaptive Bilateral Filters (M-2D-ABF). We aim to reduce the image noises such as Gaussian random noise by using the modified bilateral filters in such way that the image are smoothened however, while preserving the edges. And Enhancement is done using Energy-based CLAHE (ECLAHE) algorithm. These noiseless images are then registered using mutual information. The registered MRI and CT images are combined using the proposed Siamese Neural Network and Entropy (SNNE)-based image fusion algorithm. Then from the fused image, segmentation is done using clustering and thresholding. Then, from the segmented regions, the features are extracted and finally fused image classified into two cases, benign and malignant.

Fig.1

Flow chart of proposed methodology.

The main advantage of this paper is the proposal of a novel image enhancement technique based on Energy-based CLAHE (ECLAHE) algorithm. The main drawback of CLAHE algorithm that is the degradation of edge component due to uniform histogram equalization. This drawback is avoided in this paper. Here, the histogram equalization was applied only to the non-edge regions, so that the edge component is preserved. Another main advantage of the proposed scheme is the incorporation of DCNN and entropy features. The DCNN fusion weights for the fusion of MRI and CT images were determined based on the Siamese network and pixel entropy was computed by finding entropy value of 5×5 block surrounding a particular pixel. Based on these weights, fusion [39] weight was computed using which fusion was performed.

3.1 Image Pre-processing

Pre-processing was performed using two steps namely, filtering and enhancement. Image filtering was performed using modified 2D Adaptive Bilateral Filter (M-2D-ABF) and enhancement was performed using a novel Energy-based Contrast Limited Adaptive Histogram Equalization (ECLAHE) algorithm.

3.1.1 Image filtering using modified 2D Adaptive Bilateral Filters (M-2D-ABF)

The input image X ∈ R^U×V was filtered with modified 2D Adaptive Bilateral Filter (M-2D-ABF). The output obtained is F ∈ R^U×V. The operation of 2D Adaptive Bilateral Filter (2D-ABF) is given by $f [u, v] = \sum_{k} \sum_{l} R^{- 1} (k, l) H_{d} (u, v, k, l) H, (x [u, v], x [k, l]) x [k, l]$ (1)

where x [u, v] represents the input image and f [u, v] refers to the filtered output. Also, H_d and H_r represents domain and range filters respectively. These domain and range filters are given by $H_{d} (u, v; u_{0}, v_{0}) = e^{- (\frac{{(u - u_{0})}^{2} + {(v - v_{0})}^{2}}{2 σ_{d}^{2}})}$ (2) $H_{r} (x [u, v], x [u_{0}, v_{0}]) = e^{- (\frac{{(x [u, v] - x [u_{0}, v_{0}])}^{2}}{2 σ_{r}^{2}})}$ (3)

where [u₀, v₀] represents the center pixel.

The above two equations were modified by [45] by including an offset component. Thus, the modified 2D Adaptive Bilateral filters (M-2D-ABF) are defined as $H_{d} (u, v; u_{0}, v_{0}) = e^{- (\frac{{(u - u_{0})}^{2} + {(v - v_{0})}^{2}}{2 σ_{d}^{2}})}$ (4) $H_{r} (x [u, v], x [u_{0}, v_{0}]) = e^{- (\frac{{(x [u, v] - x [u_{0}, v_{0}] - ς [u_{0}, v_{0}])}^{2}}{2 σ_{r}^{2}})}$ (5)

The sharpness of the image is controlled by the value of ς. If the value of ς moves close to mean, the image gets more blurred and if the value of ς is moved away from the mean, the image gets sharper.

3.1.2 Image Enhancement using Energy-based CLAHE (ECLAHE) algorithm

The filtered images were then enhanced using a novel Energy-based CLAHE (ECLAHE) algorithm. The steps involved in the proposed algorithm are given in Algorithm 1. In traditional CLAHE algorithm, the histogram equalization is performed in a uniform manner, such that the edge components of the image get affected. The idea behind the proposed ECLAHE algorithm is to preserve the edge regions by retaining the regions containing more energy values. That is the image is divided into four groups namely the border blocks (BB), corner blocks (CB), high energy blocks (HEB) and low energy blocks (LEB). Here, the four corner blocks and the border blocks are retained without any histogram normalization similar to CLAHE algorithm. The blocks containing high energy more than a pre-defined threshold are also retained. However, the remaining blocks with low energy are alone processed for histogram equalization.

Algorithm 1: Proposed Energy-based CLAHE (ECLAHE) technique

Input: Filtered image F ∈ R^U×V.

Output: Enhanced image E ∈ R^U×V.

Steps:

1 Divide the entire image F ∈ R^U×V into blocks B of size b × b.

2 Compute energy of all blocks and normalize the values using l₂- norm.

3 Identify the corner blocks (CB) and border blocks (BB).

4 Select all the blocks whose energy value obtained in Step 2 is more than threshold alpha and mark them as high energy blocks (HEB).

5 Identify the remaining blocks whose energy value is less than threshold α and mark them as low energy blocks (LEB).

6 for every LEB

• Compute the histogram.

• Obtain uniform density function for each region by computing the CDF [42] of the histogram using,

C_{i, j} (n) = \frac{(N - 1)}{M} \sum_{k = 0}^{n} h_{i, j} (k); n = 1, 2, . . ., N - 1

• Transform each pixel in LEB based on the linear combination of results obtained from four neighbourhood blocks to obtain the Enhanced image E ∈ R^U×V.

3.2 Registration of MRI and CT images

The pre-processed images cannot be fused directly as they may not have the same alignment and orientation. Registration is a crucial step especially in the case of medical image fusion because the MRI and CT images are captured when the patients are made to lie in a still position [26]. Due to practical conditions both the images may not be perfectly aligned with each other. This misalignment may cause errors in image fusion if proper image registration is not performed. Registration is the method involved in finding a geometric transformation that can align the two input images. Here one image is called moving image and the other is called stationary image. The moving image is aligned with respect to the stationary image [4]. In this work, MRI image is taken as moving image and CT image is taken as stationary image. The transformation of moving image is performed using $(x, y) = T (p, q)$ (6)

where (p, q) are the original coordinates of the moving image and (x, y) are the transformed coordinates. In this work, we have employed mutual information [5] as a metric for finding the transformation matrix for the transformation of the coordinates of the moving image. Mutual information aims at aligning the voxels whose values processes similar probabilities of occurring in the two images.

3.3 Proposed image fusion methodology

In this work, we propose a novel image fusion methodology based on the Deep Convolutional Neural Networks (DCNN) and image entropy features. The proposed fusion algorithm is called Siamese Neural Network and Entropy (SNNE) based image fusion algorithm. Here, the DCNN fusion weights for the fusion of MRI and CT images were determined based on the Siamese network and pixel entropy. Siamese network is used for generating fusion scores. The Siamese neural network (SNN) has four convolutional layers, one concatenation layer, two fully connected layers and a final SoftMax layer. The output of the SoftMax layer produces a score value. Let the score value for the Siamese network for MRI and CT images be S^M (q, r) and S^C (q, r) for pixels at location (q, r). The entropy-based weights at (q, r) are computed by finding entropy value of 5×5 block surrounding pixel at location (q, r). These weights are denoted as E^M (q, r) and E^C (q, r). Using these two sets of weights, fusion is performed as in Algorithm 2.

For instance, assume that the score value of SNN for MRI image be S^M (q, r) =8 and that of CT image be S^C (q, r) =2. Also, assume the entropy value at location (q, r) for MRI image be E^M (q, r) =18 and for CT image be E^C (q, r) =6. Then, combined weight for MRI and CT images are T^M (q, r) =26 and T^C (q, r) =8 respectively. Then, the fusion weights are computed as $w^{M} (q, r) = \frac{26}{34} = 0.76$ (7) $w^{C} (q, r) = \frac{8}{34} = 0.24$ (8)

Thus, the fusion is performed using, $F (q, r) = 0.76 * p^{M} (q, r) + 0.24 * p^{C} (q, r)$ (9)

In this way, every pixel in the MRI and CT images are fused based on SNN score and entropy values.

Algorithm 2: Proposed medical image fusion using Siamese Neural Network and Entropy (SNNE)

Input: Enhanced MRI image E^M ∈ R^U×V and enhanced CT image E^C ∈ R^U×V.

Output: Fused image F ∈ R^U×V.

Steps:

1 Using MRI and CT images, construct Siamese neural network.

2 For every pixel location (q, r) where q = 1, 2, … U and r = 1, 2, …, V, compute the score values of the SoftMax layer as S^M (q, r) and S^C (q, r).

3 For every pixel location (q, r), compute the entropy of surrounding 5×5 block for the MRI and CT images to get the entropy scores as E^M (q, r) and E^C (q, r).

4 Form the combined weights using

T^M (q, r) = S^M (q, r) + E^M (q, r)

T^C (q, r) = S^C (q, r) + E^C (q, r)

5 The weights for the fusion of MRI and CT images are computed as

w^{M} (q, r) = \frac{T^{M} (q, r)}{T^{M} (q, r) + T^{C} (q, r)}

and

w^{C} (q, r) = \frac{T^{C} (q, r)}{T^{M} (q, r) + T^{C} (q, r)}

6 The fusion of MRI and CT images is performed using

F (q, r) = w^M (q, r) p^M (q, r) + w^C (q, r) p^C (q, r)

Where p^M (q, r) and p^C (q, r) represents the pixel values at location (q, r) of MRI and CT images respectively.

The Siamese neural network (SNN) has four convolutional layers, one concatenation layer, two fully connected layers and a final SoftMax layer. The four convolutional layers have a Kernel size of 3×3. The two fully connected layers have a dimension of 256 and 2 respectively. The fully connected layers have a Kernel size of 8×8. The SoftMax layer employed was a 2-way SoftMax function.

The system was trained using 273 input training images. These images were divided into 4×4 overlapping patches and used for training. The training dataset comprised of around 250000 positive training patches and 250000 negative training patches. The training batch size was set as 128 in this work.

Minimization of the loss gradient was performed using Stochastic Gradient Descent function. Loss function optimization was performed using all the 500000 training patches. The variation of loss function for number of iterations is given in Fig. 2. From Fig. 2 we observe that optimization is achieved after 70000 iterations.

Fig.2

Variation of loss function with iteration number.

3.4 Image segmentation

The fused images contain more information than the individual images. The next step in the proposed methodology is segmentation. In this study, segmentation is performed using clustering and thresholding.

3.4.1 Clustering using Fast Fuzzy C Means Clustering Algorithm (FFCMC)

Fuzzy C Means Clustering (FCMC) is a popularly used technique for segmentation of images since the accuracy of segmentation is better than other segmentation techniques. However, this technique is very slow. In order to overcome this drawback, Fast FCMC [43] was proposed. In this technique, image histogram was used instead of raw image pixels. The objective function is given by, $J = \sum_{i = 0}^{255} \sum_{q = 1}^{C} H_{i} F_{iq} D (i, θ_{q})$ (10)

where, H_i refers to the histogram, F_iq refers to the fuzzy membership between pixel x_i and histogram of cluster with center θ_q, D (i, θ_q) refers to the distance between pixel x_i and histogram of cluster with center θ_q.

3.4.2 Thresholding using Otsu Thresholding (OT)

Otsu threshold [22] is a popularly used image thresholding technique. This technique identifies a single global threshold that can be used to split the image pixels into two groups namely, the foreground and the background. All image pixels whose values are greater than this threshold are rendered as foreground and the other pixels are rendered as the background pixels.

3.5 Feature extraction

In this work, we have extracted intensity histogram based features [34] and Gray-Level Co-Occurrence Matrix features [46]. The intensity histogram-based features included mean, variance, skewness, kurtosis and entropy. GLCM feature exploits the special relationship between two pixels that are spaced by a particular distance in an image. The GLCM feature has a rapidly changing value in fine texture regions and a slowly changing value in the coarse texture regions. It is computed as $G (m, n) = \frac{# {\in S | f (m_{1}, n_{1}) = g_{1} & f (m_{2}, n_{2}) = g_{2}}}{# S}$ (11)

From the GLCM matrix, several statistical features like contrast, energy, homogeneity, correlation were extracted.

3.6 Classification

Using the segmentation results, classification of the segmented brain images is performed using Logistic Regression [33] model. We have employed ten-fold cross validation technique in which the dataset is divided into ten equal parts and evaluation is done. During the training phase, a logistic regression model is generated using the training dataset. The classification is then performed using the testing dataset and the trained classification model. In logistic regression, a feature vector v = (v₁, v₂, … v_q) is classified to a class label l ∈ {0, 1} using $P (l = 0 / v) = \frac{1}{1 + e^{z β + β_{0}}}$ (12) $P (l = 1 / v) = \frac{e^{z β + β_{0}}}{1 + e^{z β + β_{0}}}$ (13)

where β = (β₁, β₂, …, β_q) represents the weight vector and β₀ represents the intercept value. A test data sample is classified to a particular class by first evaluating the two probabilities namely the, P (l = 0/v) and P (l = 1/v) and then assigning to the class that has a higher probability.

4 Results and Discussion

4.1 Parameter settings

The proposed system was simulated using MATLAB software running on windows intel i3 core processor with 8GB RAM. Four different MRI and CT image pairs were considered for analysis in this study. The value of threshold α in Algorithm 1 was selected as 0.8. The value of block size b in the Algorithm 1 was set as 5. To evaluate the classification performance, metrics like overall accuracy, recall, precision, specificity and F-score were employed. For evaluation of the classification algorithm, we employed the Anatomical Tracings of Lesions After Stroke (ATLAS) dataset [17] that consists of 304 images. The dataset can be accessed at http://atlas.cern/resources/opendata. We performed 10-fold cross validation in which 273 images were used for training and 31 images were used for testing. The entire process was repeated 10 times until all the images were tested.

4.2 Simulation results

Figure 3 and Fig. 4 shows the sample input MRI and CT images respectively. From these Figs, it can be visually observed that the images are noisy. Hence, they are pre-processed using filtering and enhancement techniques.

Fig.3

(a)Noisy MRI image 1, (b) Noisy MRI image 2, (c) Noisy MRI image 3, (d) Noisy MRI image 4.

Fig.4

(a) Noisy CT image 1, (b) Noisy CT image 2, (c) Noisy CT image 3, (d) Noisy CT image 4.

Figure 5 and Fig. 6 shows MRI and CT images that are filtered using M-2D-ABF filter. From these Figs we infer that the filtered images have less noise compared to the original images.

Fig.5

(a) M-2D-ABF filtered MRI image 1, (b) M-2D-ABF filtered MRI image 2, (c) M-2D-ABF filtered MRI image 3, (d) M-2D-ABF filtered MRI image 4.

Fig.6

(a) M-2D-ABF filtered CT image 1, (b) M-2D-ABF filtered CT image 2, (c) M-2D-ABF filtered CT image 3, (d) M-2D-ABF filtered CT image 4.

Figure 7 and Fig. 8 shows the MRI and CT images enhanced using ECLAHE algorithm. From Fig. 7 and Fig. 8, it is obvious that the enhanced images have better visual clarity compared to the original images.

Fig.7

(a) ECLAHE enhanced MRI image 1, (b) ECLAHE enhanced MRI image 2, (c) ECLAHE enhanced MRI image 3, (d) ECLAHE enhanced MRI image 4.

Fig.8

(a) ECLAHE enhanced CT image 1, (b) ECLAHE enhanced CT image 2, (c) ECLAHE enhanced CT image 3, (d) ECLAHE enhanced CT image 4.

Figure 9 shows an example pair of misaligned MRI and CT images. From Fig. 9, it is evident that, these two images cannot be fused directly as the orientation of the two images are different. Figure 9 (c) shows the registered MRI image.

Fig.9

(a) Misaligned MRI image, (b) Misaligned CT image (c) Registered MRI image.

Figure 10 shows the images that depict the proposed SNNE algorithm results. The first two images are the input to the fusion algorithm. The final image is the fusion result. From the third image, we clearly see that the proposed algorithm produces excellent fusion result that contains the complimentary information from the two input images belonging to different modalities.

Fig.10

(a) MRI image, (b) CT image, (c)Fusion result using proposed SNNE algorithm.

Figure 11 shows the segmentation results using FFCMC clustering and Otsu thresholding. The segmentation results clearly show the regions containing the tumor cells. From these segmented regions, classification is performed using logistic regression classifier.

Fig.11

(a) Segmentation result of Image pair 1, (b) Segmentation result of Image pair 2, (c) Segmentation result of Image pair 3, (d) Segmentation result of Image pair 4.

4.2.2 Results of Proposed Algorithms on fusing BRATS MRI FLAIR and T2 Modality

BRATs 2015 is a well-known dataset for brain abnormalities investigation. The dataset contains axial brain MR slices with T2, T1c and FLAIR modalities. The dataset comprises the following properties listed in Table 1 (Image dimension: 216×160×3, Number of images: 1000, Modality: FLAIR|T2|T1C, Bit Depth: 8). The advantage of using BRATs MR series dataset is that, it holds expert ground truth (GT) information. The ground truth evidence helps in assessing the segmentation process of computer assisted procedure by analyzing the image similarity between the two images.

Table 1
BRATs 2015 Dataset

Image Fused Modality Image dimension Total images Training images Testing and validation images

Benign Flair + T2 216×160×3 1000 800 200

Malignant Flair + T2 216×160×3 1000 800 200

Image	Fused Modality	Image dimension	Total images	Training images	Testing and validation images
Benign	Flair + T2	216×160×3	1000	800	200
Malignant	Flair + T2	216×160×3	1000	800	200

Similar procedure is directed towards MR series of BRATS 2015 dataset. The test images of Flair and T2 modalities to be evaluated and its corresponding ground truth is shown in Fig. 12. The MRI images are initially subjected to M-2D-ABF filter then enhanced using ECLAHE algorithm. The generated images from the above process contain less noise and offers better visualization.

Fig.12

BRATS MR Series.

Figure 13 shows the fusion and segmentation results of the proposed SNNE algorithm. The FLAIR and T2 modalities of BRATs MR series is subjected to the fusion algorithm. The resultant image encloses the fused image of T2 and FLAIR modality. The segmentation results clearly depict that the proposed algorithm produces outstanding fusion results and it is reflected in the segmentation process in extracting the tumor core intensities.

Fig.13

Results of BRATS MR series. (a) Fusion result using proposed SNNE algorithm. (b) Segmentation results using FFCMC clustering and Otsu thresholding.

4.3 Quantitative analysis

4.3.1 Evaluation of Proposed Pre-processing (M-2D-ABF-ECLAHE) Algorithms

Table 2 shows the performance evaluation using mean square error. Evaluation was performed for the four pairs of datasets shown in the Fig. 2 and Fig. 3. They are denoted as Image Pair 1 (IP1), Image Pair 2 (IP2), Image Pair 3 (IP3) and Image Pair 4 (IP4). From Table 1 we see that, the average value of MSE for a combination of2D median filter [32] with CLAHE [37] algorithm is 5.86. Similarly, the average value of MSE for a combination of 2D Adaptive Median Filter [1] with CLAHE algorithm is 3.07. However, our M-2D-ABF Filter with proposed ECLAHE algorithm produces a very low MSE of 0.91. Thus, our proposed pre-processing technique achieves best performance in terms of MSE.

Table 2
Performance evaluation using Mean Square Error

Image Pair MSE

2D Median 2D Adaptive Median Proposed

Filter + CLAHE Filter + CLAHE M-2D-ABF + ECLAHE

IP1 5.42 3.12 0.85

IP2 6.23 3.42 0.99

IP3 6.12 2.23 0.89

IP4 5.68 3.52 0.93

Image Pair		MSE
IP1	5.42	3.12	0.85
IP2	6.23	3.42	0.99
IP3	6.12	2.23	0.89
IP4	5.68	3.52	0.93

Table 3 shows the performance evaluation using mean PSNR. From Table 3 we see that, the average value of PSNR for a combination of 2D Median Filter with CLAHE algorithm is 20.21. Similarly, the average value of PSNR for a combination of 2D Adaptive Median Filter with CLAHE algorithm is 24.40. However, our M-2D-ABF Filter with proposed ECLAHE algorithm produces a very high PSRN of 34.39. Thus, our proposed pre-processing technique achieves best performance in terms of PSNR.

Table 3

Performance evaluation using PSNR

Image Pair		PSNR (in dB)
	2D Median	2D Adaptive Median	Proposed
	Filter + CLAHE	Filter + CLAHE	M-2D-ABF + ECLAHE
IP1	19.29	23.42	34.62
IP2	18.93	23.21	33.65
IP3	21.83	26.73	35.78
IP4	20.81	24.25	33.51

4.3.2 Convergence graph of SSN

The proposed fusion methodology employs Siamese Neural Network (SNN) architecture. The convergence graph of the proposed SNN architecture is shown in Fig. 14. This graph shows the variation of loss and accuracy per epoch. From these plots, we observe that the SNN model has a comparable performance for the training and testing datasets in terms of loss and accuracy. These graphs also depict that the proposed SSN model has a good fit.

Fig.14

Convergence graph of SNN architecture.

4.3.3 Evaluation of proposed fusion methodology

The proposed fusion methodology is evaluated using the following metrics:

Entropy:

Entropy [16] is the information content of the fused image. It is computed as $E_{f} = - \sum_{p = 0}^{255} F_{p} {log}_{2} F_{p}$ (14)

where F_p is the probability of intensity p in the fused image F.

Standard Deviation:

Standard deviation [12] measures the overall contrast factor. It is computed as $SD = \sqrt{\frac{1}{MN} \sum_{m = 1}^{M} \sum_{n = 1}^{N} {(F (m, n) - μ)}^{2}}$ (15)

where μ is the mean. Higher values of standard deviation indicate better contrast factor in the fusion output.

MSE:

MSE gives the average difference between the original image O and the fused image F after applying the proposed fusion methodology. It is given by $(1 / N^{2}) \sum_{i = 1}^{N} \sum_{j = 1}^{N} {[O (i, j) - F (i, j)]}^{2}$ (16)

PSNR:

PSNR gives the fraction of actual signal value to the error value. It is calculated as $PSNR = 10 {log}_{10} \frac{255^{2}}{(1 / N^{2}) \sum_{i = 1}^{N} \sum_{j = 1}^{N} {[O (i, j) - F (i, j)]}^{2}}$ (17)

Structural Similarity Index (SSIM)

The metric SSIM [41]measures the luminance, contrast and structural similarities between two images and is given by, $SSIM = (\frac{2 μ_{o} μ_{f} + C_{1}}{{μ_{o}}^{2} + {μ_{f}}^{2} + C_{1}}) (\frac{2 σ_{of} + C_{2}}{{μ_{o}}^{2} + {μ_{f}}^{2} + C_{2}})$ (18)

The default value for C₁ and C₂ are 0.01 and 0.03. The maximum SSIM value is 1.

Homogeneity:

Homogeneity indicates the structural similarity of an image. It is computes as $H_{x, y} (m, n) = \frac{e^{- k {(F (x, y) - F (m, n))}^{2}}}{\sum_{m = 1}^{M} \sum_{n = 1}^{N} e^{- k {(F (x, y) - F (m, n))}^{2}}}$ (19)

Table 4 to Table 7 clearly shows the results obtained for various fusion methodologies proposed in the literature using Image Pair 1 to Image Pair 4 respectively. From these tables, it is clear that the proposed Siamese Neural Network and Entropy (SNNE) based fusion technique produces outstanding results in terms of various metrics. The efficacy of the proposed fusion technique is clearly seen from Table 4 to Table 7. For all the four image pairs, our system produces excellent results in terms of all the fusion performance evaluation metrics.

Table 4

Quantitative Metrics for Image Pair 1

Fusion Methodologies	Entropy	Standard Deviation	PSNR in dB	MSE	SSIM	Homogeneity
2D Discrete Wavelet Transform [14]	5.23	42.42	26.52	1.94	45.23	32.23
2D Stationary Wavelet Transform [4]	6.32	45.23	24.32	1.57	34.42	33.42
Non-Subsampled Contourlet Transform [34]	4.52	49.93	26.48	1.84	41.2	37.42
SNNE (Proposed System)	8.42	53.91	42.45	0.92	69.23	53.12

Table 5

Quantitative Metrics for Image Pair 2

Fusion Methodologies	Entropy	Standard Deviation	PSNR in dB	MSE	SSIM	Homogeneity
2D Discrete Wavelet Transform	3.91	34.53	26.34	1.43	33.65	31.43
2D Stationary Wavelet Transform	5.23	44.32	22.49	1.35	33.74	32.30
Non-Subsampled Contourlet Transform	5.83	37.32	31.52	1.53	45.34	34.23
SNNE (Proposed System)	8.33	54.32	45.46	1.03	65.34	54.32

Table 6

Quantitative Metrics for Image Pair 3

Fusion Methodologies	Entropy	Standard Deviation	PSNR in dB	MSE	SSIM	Homogeneity
2D Discrete Wavelet Transform	3.94	36.24	24.23	1.42	35.22	33.52
2D Stationary Wavelet Transform	4.63	45.23	22.16	1.11	38.73	36.22
Non-Subsampled Contourlet Transform	6.34	36.23	32.73	1.31	44.27	36.89
SNNE (Proposed System)	9.43	55.64	43.23	0.99	72.53	62.23

Table 7

Quantitative Metrics for Image Pair 4

Fusion Methodologies	Entropy	Standard Deviation	PSNR in dB	MSE	SSIM	Homogeneity
2D Discrete Wavelet Transform	3.11	42.23	24.22	1.35	42.52	33.52
2D Stationary Wavelet Transform	4.53	44.17	26.48	1.35	49.53	36.42
Non-Subsampled Contourlet Transform	5.23	36.53	36.78	1.53	53.32	35.26
SNNE (Proposed System)	7.24	49.43	48.42	1.01	74.23	54.32

4.3.4 Evaluation of Classification Algorithms

The classification is also performed using traditional algorithms like k-nearest neighbour (k-NN), Naïve Bayes (NB), Random Forest (RF), support vector machine (SVM) for comparison.

The confusion matrix obtained for the logistic regression classification scheme is shown in Table 7. From Table 8 we infer that the overall accuracy of the logistic regression classifier is 97.37%.

Table 8
Confusion matrix of logistic regression classifier

Predicted Class

Benign Malignant

Target class

Benign True positive (145) False negative (5) Sensitivity (96.6667)

Malignant False positive (3) True negative (151) Specificity (98.0519)

F1-Score Precision Accuracy

(97.3154) (97.9730) (97.3684)

	Predicted Class
Target class
Benign	True positive (145)	False negative (5)	Sensitivity (96.6667)
Malignant	False positive (3)	True negative (151)	Specificity (98.0519)
	F1-Score	Precision	Accuracy
	(97.3154)	(97.9730)	(97.3684)

Table 9 shows the results obtained in terms of overall accuracy. From Table 9, we see that Logistic Regression (LR) algorithm produces the best results compared to all other traditional classification algorithms.

Table 9

Comparison of Overall Accuracy

Classification algorithms	Overall accuracy (%)
k-NN [19]	81.39
Naïve Bayes [2]	84.29
Random Forest [21]	90.32
SVM [20]	94.82
Logistic Regression	97.37

Figure 15 represents the comparison of specificity for the ATLAS dataset. Form Fig. 15 we see that the value of specificity obtained byk-NN, NB, RF, SVM and LR are 88.12, 89.31, 91.39, 94.52 and 96.52 respectively. Thus, it is clearly seen that LR outperforms other algorithms.

Fig.15

Comparison of specificity.

Figure 16 represents the comparison of precision. Form Fig. 16 we see that the average value of precision obtained by k-NN, NB, RF, SVM and LR are 86.13, 88.22, 90.36, 93.69 and 96.65 respectively. Thus, it is clearly seen that LR is the best compared to other algorithms.

Fig.16

Comparison of precision.

Figure 17 represents the comparison of recall. Form Fig. 17, we see that the average value of recall obtained by k-NN, NB, RF, SVM and LR are 89.23, 90.12, 92.96, 93.42 and 96.93 respectively. Thus, it is clearly seen that LR performs better that other traditional classification algorithms.

Fig.17

Comparison of recall.

Figure 18 represents the comparison of F-score. Form Fig. 18, we see that the average value of F-score obtained by k-NN, NB, RF, SVM and LR are 90.42, 91.67, 93.33, 94.62 and 97.11 respectively. Thus, the F-score of LR is better than other algorithms.

Fig.18

Comparison of F-score.

The classification is also performed using traditional algorithms like k-nearest neighbour (k-NN), Naïve Bayes (NB), Random Forest (RF), support vector machine (SVM) for comparison.

4.3.5 Classification accuracy of BRATs Dataset

The performance of the proposed procedure is evaluated against BRATs dataset for estimating the classification accuracy. The classification is performed using the similar traditional algorithms like k-nearest neighbour (k-NN), Naïve Bayes (NB), Random Forest (RF), support vector machine (SVM) for evaluation. Table 10 lists the confusion matrix parameters for classifying the two distinct class of tumours. Figure 19 shows the predictive performance of the classifiers assessed with the following parameters, accuracy, sensitivity, specificity, precision and accuracy. From the plotted values of Fig. 19, LR outperforms in overall assessment towards classification accuracy.

Table 10
Comparison of performance indices of applying 5 models to BRATs Dataset

Classifier TP FN TN FP ACC PRE SEN SPE F1s

k-NN 182 18 186 14 0.92 92.85714 91 93 91.91919

Naïve Bayes 185 15 187 13 0.93 93.43434 92.5 93.5 92.96482

Random Forest 186 14 188 12 0.935 93.93939 93 94 93.46734

SVM 191 9 189 11 0.95 94.55446 95.5 94.5 95.02488

Logistic Regression 192 8 191 9 0.9575 95.52239 96 95.5 95.7606

Classifier	TP	FN	TN	FP	ACC	PRE	SEN	SPE	F1s
k-NN	182	18	186	14	0.92	92.85714	91	93	91.91919
Naïve Bayes	185	15	187	13	0.93	93.43434	92.5	93.5	92.96482
Random Forest	186	14	188	12	0.935	93.93939	93	94	93.46734
SVM	191	9	189	11	0.95	94.55446	95.5	94.5	95.02488
Logistic Regression	192	8	191	9	0.9575	95.52239	96	95.5	95.7606

Fig.19

Classifier accuracy of BRATs Dataset.

In future the proposed work can be extended by considering other pre-trained deep learning method existing in literature and used to text clinical MRI images.

5 Conclusion

A novel framework for fusion and classification of medical images was proposed in this research. In this work, the modified 2D Adaptive Bilateral Filtering algorithm was used for filtering. Then image enhancement was done using ECLAHE algorithm. The computed tomography and magnetic resonance imaging were then registered using mutual information and then fused to form a single fused image. Here, the images were fused using the proposed Siamese Neural Network and Entropy (SNNE)-based image fusion algorithm. Using the score of the SoftMax layer and the entropy of the image, fusion was done. Using the fused image, the tumor regions were segmented using FFCMC clustering and Otsu Thresholding. Finally, intensity histogram and GLCM features were extracted. Using the extracted features, classification was done using Logistic Regression classifier. The proposed fusion algorithm attained excellent results in terms of fusion metrics. It was observed that the logistic regression classifier achieved an accuracy of about 97.37% for ATLAS Dataset and an accuracy of about 95.76% for BRATs 2015 dataset.

References

Anitha

and Siva Sundhara Raja

, Development of computer-aided approach for brain tumor detection using random forest classifier, Int J Imaging Syst Technol 28 (2018), 48–53.

Asokan

, Kalaiselvi

T.C.

and Tamilarasi

, Medical image fusion using stationary wavelet transform with different wavelet families, Pakistan J Biotechnol 13 (2016), 10–14.

Rajalingam

and Priya

, Hybrid multimodality medical image fusion technique for feature enhancement inmedical diagnosis, Int J Eng Sci Invent (2018), 52–60.

Brock

K.K.

, Mutic

, McNutt

T.R.

, et al., Use of image registration and fusion algorithms and techniques in radiotherapy: Report of the AAPM Radiation Therapy Committee Task Group No. 132: Report Med Phys 44 (2017), e43–e76.

Chen

H.M.

and Varshney

P.K.

, Mutual Information-Based CT-MR Brain Image Registration Using Generalized Partial Volume Joint Histogram Estimation, IEEE Trans Med Imaging 22 (2003), 1111–1119.

Chinnu

, MRI Brain Tumor Classification Using SVM and Histogram Based Image Segmentation, Int J Sci Res 4 (2015), 1647–1650.

de Vos

B.D.

, Berendsen

F.F.

, Viergever

M.A.

, et al., A deep learning framework for unsupervised affine and deformable image registration, Med Image Anal 52 (2019), 128–143.

Feng

, Cai

, Wang

, et al., Dual residual convolutional neural network (DRCNN) for low-dose CT imaging, J Xray Sci Technol 29 (2021), 91–109.

Hermessi

, Mourali

and Zagrouba

, Convolutional neural network-based multimodal image fusion via similarity learning in the shearlet domain, Neural Comput Appl 30 (2018), 2029–2045.

10.

Hou

, Zhou

, Nie

, et al., Brain CT and MRI medical image fusion using convolutional neural networks and a dual-channel spiking cortical model, Med Biol Eng Comput 57 (2019), 887–900.

11.

Huang

, et al., A New Pulse Coupled Neural Network (PCNN) for Brain Medical Image Fusion Empowered by Shuffled Frog Leaping Algorithm, Front Neurosci 13 (2019), 1–10.

12.

Huang

and Jing

, Evaluation of focus measures in multi-focus image fusion, Pattern Recognit Lett 28 (2007), 493–500.

13.

Jian

, Xia

, Yin

and Wang

J.Q.

, A novel improved deep convolutional neural network model for medical image fusion, Cluster Comput 22 (2019), 1515–1527.

14.

Khan

S.U.

, Ullah

, Ahmed

, et al., Directional weighted spatial fuzzy C-means for segmentation of brain MRI images, J Xray Sci Technol 27 (2020), 1087–1099.

15.

Kumar

, Dabas

and Godara

, Classification of Brain MRI Tumor Images: A Hybrid Approach, Procedia Comput Sci 122 (2017), 510–517.

16.

Kv Ålseth

T.O.

, Entropy and Correlation: Some Comments, IEEE Trans Syst Man Cybern 17 (1987), 517–519.

17.

Liew

S.L.

, et al., A large, open source dataset of stroke anatomical brain images and manual lesion segmentations, Sci Data 5 (2018).

18.

Liu

, Mei

and Du

, Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion, Neurocomputing 235 (2017), 131–139.

19.

Maddaiah

P.N.

, Pournami

P.N.

and Govindan

V.K.

, Optimization of Image Registration for Medical Image Analysis, Int J Comput Sci Inf Technol 5 (2014), 3394–3398.

20.

Maintz

J.B.A.

and Viergever

M.A.

, A survey of medical image registration, Med Image Anal 2 (1998), 1–36.

21.

Mathew

A.R.

and Anto

P.B.

, Tumor detection and classification of MRI brain image using wavelet transform and SVM, Proc IEEE Int Conf Signal Process Commun ICSPC 2017 (2018), 75–78.

22.

Otsu

, Threshold Selection Method From Gray-Level Histograms, IEEE Trans Syst Man Cybern 1 (1979), 62–66.

23.

Ouerghi

, Mourali

and Zagrouba

, Non-subsampled shearlet transform based MRI and PET brain image fusion using simplified pulse coupled neural network and weight local features in YIQ colour space, IET Image Process 12 (2018), 1873–1880.

24.

Prabhu Chakkaravarthy

and Chandrasekar

, A Linear filtering on automatic decomposition and reconstruction of dermoscopy images using global thresholding, International Journal of Innovative Technology and Exploring Engineering 8 (2019), 4257–4263.

25.

Prabhu Chakkaravarthy

and Chandrasekar

, An automatic threshold segmentation and mining optimum credential features by using HSV model, 3D Research 10 (2019), 1–17.

26.

Prabhu Chakkaravarthy

and Chandrasekar

, Anatomical region segmentation method from dermoscopic images of pigmented skin lesions, International Journal of Imaging System and Technology 30 (2020), 636–652.

27.

Pugalenthi

, Oliver

A.S.

and Anuradha

, Impulse noise reduction using hybrid neuro-fuzzy filter with improved firefly algorithm from X-ray bio-images, International Journal of Imaging Systems and Technology 30 (2020), 1119–1131.

28.

Pugalenthi

, Rajakumar

M.P.

, Ramya

and Rajinikanth

, Evaluation and Classification of the Brain Tumor MRI using Machine Learning Technique, Journal of Control Engineering & Applied Informatics 21 (2019), 12–21.

29.

Rajalingam

P. B.

, Combining multi-modality medical image fusion based on hybrid intelligence for disease identification, Int J Adv Res Trends Eng Technol 5 (2018), 862–887.

30.

Rajalingam

and Priya

, Multimodality medical image fusion based on hybrid fusion techniques, 7 (2017), 22–29.

31.

Rajinikanth

, Satapathy

S.C.

, Dey

and Vijayarajan

, DWT-PCA image fusion technique to improve segmentation accuracy in brain tumor analysis, Lect Notes Electr Eng 471 (2018), 453–462.

32.

Ramdlon

R.H.

, Martiana Kusumaningtyas

and Karlita

, Brain tumor classification using MRI images with K-nearest neighbor method, IES 2019 - Int Electron Symp Role Techno-Intelligence Creat an Open Energy Syst Towar Energy Democr Proc (2019), 660–667.

33.

Rao

, Lee

, Gass

and Monsch

, Classification of Alzheimer’s disease from structural MRI using sparse logistic regression with optional spatial regularization, Conf Proc IEEE Eng Med Biol Soc 20 (2011), 4499–4502.

34.

Rouhi

, Jafari

, Kasaei

and Keshavarzian

, Benign and malignant breast tumors classification based on region growing and CNN segmentation, Expert Syst Appl 42 (2015), 990–1002.

35.

Sanjay

A.R.

, Soundrapandiyan

, Karuppiah

and Ganapathy

, CT and MRI image fusion based on discrete wavelet transform and Type-2 fuzzy logic, Int J Intell Eng Syst 10 (2017), 355–362.

36.

Sheryl Oliver

, Ganesan

, Yuvaraj

S.A.

, et al., Accurate prediction of heart disease based on bio system using regressive learning based neural network classifer, Journal of Ambient Intelligence and Humanized Computing (2021), 56.

37.

Sunny

and Gandhi

M.P.I.

, An efficient citrus canker detection method based on contrast limited adaptive histogram equalization enhancement, Int J Appl Eng Res 13 (2018), 809–815.

38.

Suresh

and Sakthi

, A soft-computing based hybrid tool to extract the tumour section from brain MRI, Multimedia Tools and Applications 79 (2020), 4133–4147.

39.

Tirupal

, Chandra Mohan

and Srinivas Kumar

, Multimodal medical image fusion techniques –A review, Current Signal Transduction Therapy 15 (2020), 1–22.

40.

Kamarthi

, Satyanarayana

and Giri Prasad

M.N.

, Multi-modal medical image fusion algorithm based on spatial frequency motivated PA-PCNN in NSST domain, Curr Med Imaging 16 (2020).

41.

Wang

, Bovik

A.C.

, Sheikh

H.R.

and Simoncelli

E.P.

, Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Process 13 (2004), 600–612.

42.

Yadav

, Maheshwari

and Agarwal

, Contrast limited adaptive histogramequalization based enhancement for real time video system, in Proceedings of the 2014 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2014 (2014), 2392–2397.

43.

Yong

, Chongxun

and Pan

, A novel fuzzy c-means clustering algorithm for image thresholding, Meas Sci Rev 4 (2004), 11–19.

44.

Zaw

H.T.

, Maneerat

and Win

K.Y.

, Brain tumor detection based on Naíve Bayes classification, Proceeding - 5th Int Conf Eng Appl Sci Technol ICEAST 2019 (2019), 1–4.

45.

Zhang

and Allebach

J. P.

, Adaptive bilateral filter for sharpness enhancement and noise removal, IEEE Trans Image Process 17 (2008), 664–678.

46.

Zhang

, Cui

, Wang

and Lin

, A study for texture feature extraction of high-resolution satellite images based on a direction measure and gray level co-occurrence matrix fusion algorithm, Sensors 17 (2017).

47.

Zhu

, Chai

, Yin

, Li

and Liu

, A novel dictionary learning approach for multi-modality medical image fusion, Neurocomputing 214 (2016), 471–482.

Deep learning supported disease detection with multi-modality image fusion

Abstract

Keywords

1 Introduction

2 Literature survey

3 Proposed methodology

3.1.1 Image filtering using modified 2D Adaptive Bilateral Filters (M-2D-ABF)

3.2 Registration of MRI and CT images

3.4.1 Clustering using Fast Fuzzy C Means Clustering Algorithm (FFCMC)

3.5 Feature extraction

4.1 Parameter settings

4.2 Simulation results

Table 1 BRATs 2015 Dataset Image Fused Modality Image dimension Total images Training images Testing and validation images Benign Flair + T2 216×160×3 1000 800 200 Malignant Flair + T2 216×160×3 1000 800 200

4.3.1 Evaluation of Proposed Pre-processing (M-2D-ABF-ECLAHE) Algorithms

Table 2 Performance evaluation using Mean Square Error Image Pair MSE 2D Median 2D Adaptive Median Proposed Filter + CLAHE Filter + CLAHE M-2D-ABF + ECLAHE IP1 5.42 3.12 0.85 IP2 6.23 3.42 0.99 IP3 6.12 2.23 0.89 IP4 5.68 3.52 0.93

References

Table 1
BRATs 2015 Dataset

Image Fused Modality Image dimension Total images Training images Testing and validation images

Benign Flair + T2 216×160×3 1000 800 200

Malignant Flair + T2 216×160×3 1000 800 200

Table 2
Performance evaluation using Mean Square Error

Image Pair MSE

2D Median 2D Adaptive Median Proposed

Filter + CLAHE Filter + CLAHE M-2D-ABF + ECLAHE

IP1 5.42 3.12 0.85

IP2 6.23 3.42 0.99

IP3 6.12 2.23 0.89

IP4 5.68 3.52 0.93