PCA based SVD fusion for MRI and CT medical images

Abstract

This paper presents a new approach to the multi-modal medical image fusion based on Principal Component Analysis (PCA) and Singular value decomposition (SVD).The main objective of the proposed approach is to facilitate its implementation on a hardware unit, so it works effectively at run time. To evaluate the presented approach, it was tested in fusing four different cases of a registered CT and MRI images. Eleven quality metrics (including Mutual Information and Universal Image Quality Index) were used in evaluating the fused image obtained by the proposed approach, and compare it with the images obtained by the other fusion approaches. In experiments, the quality metrics shows that the fused image obtained by the presented approach has better quality result and it proved effective in medical image fusion especially in MRI and CT images. It also indicates that the paper approach had reduced the processing time and the memory required during the fusion process, and leads to very cheap and fast hardware implementation of the presented approach.

Keywords

Image fusion PCA SVD medical images fusion

1 Introduction

Nowadays, medical images are a major method of diseases diagnosis such as tumor tissues among children and adults alike [1]. There are several modalities of imaging systems such as Computed Tomography (CT), and The Magnetic resonance (MR). For hard tissues CT imaging model would be the best choice, on the other side for soft tissues MR imaging would be the best. Sometimes brain tumorous contains soft and hard tissues together, because doctors need information from CT and MR images together, so CT model or MR model alone is not enough. That is why the fusion between different imaging modalities became important. Fusion process had been implemented using different methods. These methods are simple methods such as pixel averaging, moderate complicated methods such as PCA-based fusion, and advanced methods such as those in the different transform domains such as Wavelet-based fusion [2].

The image merging process is the merging of two or more images in a section to produce high-quality, high-resolution images without misrepresentation, so that the merged image can be better interpreted in several applications by giving more precise information [3 –5]. In this sense, a number of methods have been presented in the literature, such as the high-pass filtering method [6], analysis of the major components of transformation such as PCA, Hue and Saturation (HIS), and Brovey transformation and intensity [7].

Although these spatial methods [8 –10] provide better visual excellence images, several studies show that colors are visually distorted in merged images.

The high-pass filter technique conserves spatial detail, but there are noises within the merged image. The IHS-transform changes intensity components of high resolution panchromatic images, while spatial-resolution enhances the supernatural artifacts in a resulted appearance due to the replacement of the density element [11].

The PCA technique substitutes the first major element of the multispectral image with PAN images. It increases the longitudinal feature and the supernatural artifacts due to the deficiency of information in the reconstruction of the original image.

Lately, transformation domain methods using multi-resolution analysis have been developed, such as wavelet transformation. Wavelet transformations can store and preserve the details of time and frequency. The entered input images are disintegrating addicted to various frequency concentrations then, the fusion procedures are applied by substituting the feature measurements with element parts of added original image. Although image fusion based on wavelet gives high supernatural excellence, longitudinal/spatial resolution may reduce because of critical down-sampling [12]. Spatial information is needed for highly medical applications due to the convoluted nature of land use characteristics. Even though, several image fusion techniques are accessible in the works for integrating remotely-recognized data, not all methods meet the necessity for mapping the properties of the earth’s surface [10].

In recent years, researchers have used other multi-resolution analyzes, including wavelet packet transform (WPT), multi wavelet transform (MWT), discrete wavelet transform (DWT), curvelet transform (CUT), atrous wavelet transform (ATWT), Contourlet transform (CT), Framelet transform (FRT), and nonsupsampled Contourlet transform (NSCT). Great advances have been produced in image fusion knowledge but little investigation has been done on other purposes such as categorization and feature extraction [4 , 12].

Simple and moderate fusion methods give a fused image in a small processing time but without high quality in several metrics [13]. Advanced methods such as Discrete Wavelet Transform (DWT) and Curvelet fusions introduce a high quality image, but those take a long time compared to the simple methods especially the Curvelet fusion. According to the required fused image characteristics, and the time requirements, one of those methods would be selected [1].

This paper presents a proposed image fusion algorithm to integrate all the relevant information of both CT and MRI imaging modalities. The proposed algorithm is based on a combination of PCA and SVD, as a pre-processing stage to fuse the CT and MRI images. By using the proposed PCA based SVD fusion algorithm, the processing time and the memory requirement of the fusion process would be reduced, with the same quality of other algorithms images. This will be a good step to implement the proposed algorithm with a cheap and fast hardware unit, so it can be attached to any scanner of the CT or MRI scanners.

The results of the proposed algorithm in several cases and different image dimensions indicate that the obtained fused images by the presented algorithm is better than those of the other compared algorithms in most of the quality metrics values. Moreover, the elapsed time of the fusion process using the proposed algorithm had been reduced at least 50% and 60% compared to PCA, and Dual-tree Continuous Wavelet Transform, and more than 98% comparing to wavelet and the Curvelet based fusion algorithms. When the size of the input images increased the saved time would be increased also by using the proposed algorithm in comparing of other traditional algorithms.

This paper is organized as follows: Section 2 discusses and explores related works. Section 3 presents the image fusion methods. Section 4 includes the proposed approach based on PCA and SVD fusion methods. Section 5 shows how to use the fusion quality evaluations metrics to evaluate the presented approach compared to the other traditional approaches. The experimental results are presented in section 6. Finally, section 7 concludes the paper.

2 Related works

In [14], the authors introduced a multi-scale analysis coupling approximate sparse representation image fusion scheme. The image high and low frequencies information are obtained. After that, the image high and low frequencies information are estimated using the coefficients of approximate sparse, and the approximate sparseness is transformed using absolute maximum selection scheme to provide the low frequency subband approximate coefficient and the high frequency subband. Then, the decision mapping is formulated, and the matching degree of coefficients exist within the subband are investigated to determine the decision estimate, and the image can be matched and merged. In [15], the authors proposed a medical fusion scheme using Laplacian pyramid and convolutional neural network (CNN). The multimodal medical images are generated using CNN. After that, Laplacian pyramid can be employed on decomposition and fusion processes. The optimal decomposition layers number can be estimated experimentally. The fused image is produced using the inverse of Laplacian transform. The output fusion results have better quality performance. In [16], the authors proposed an efficient medical image fusion scheme. The input image is firstly decomposed using a hybrid three-layer decomposition model into structure layer, texture layer, and local mean brightness layer. After that, the patches nuclear norm achieved by a sliding window, are estimated to provide the structure and texture layers weight maps. Both the subjective/objective tests ensured the advantage of the suggested scheme with respect to the related state-of-art schemes.

In [17], the authors introduced a medical image fusion scheme using the segmented graph filter (SGF) and the sparse representation (SR). The images are decomposed using the SGF into the base and the detailed images. After that, the base images are fused using the normalized Shannon entropy while the detailed images are fused based on SR fusion. The final fused-image can be produced through fusing both the fused-base and fused-detailed images. The test results show that the proposed method the fusion performance is comparable to the related state-of-art schemes. In [18], the authors introduced a comprehensive survey of medical image fusion schemes including different fusion methods strategies like pixel-level, feature-level and decision-level. Also, the fusion performance experimental tests using various fusion categories are given. In [19], the authors introduced a three-layer medical image fusion scheme which has three stages. With the first stage, various features using structure tensor are utilized to decompose the anatomical medical MRI image into its three-layer image representation. With the second stage, spatial frequency metric is suggested for merging the decomposed intensity and detail layers. Finally, the fused image is produced through adding the fused intensity, detail, and base layers, the fused layer, and the fused base layer. The subjective and objective evaluations demonstrated the effectiveness of the proposed scheme.

In [20], the authors introduced a multimodal medical image fusion algorithm that depends on a lifting-based biorthogonal wavelet transformation. The multi-scale fusion is applied in wavelet domain for the multimodal medical images using multiple scales. The experimental outcomes indicated that the proposed scheme can provide good outcomes over wavelet transform-based fusion methods. In [21], the authors introduced a fusion scheme by combining the complementary information of different imaging modalities like SPECT, PET, and MRI through employing the empirical wavelet transform (EWT) representation and the local energy maxima (LEM) fusion rule. The outcomes of the proposed scheme show improved visual quality performance as well as fusion performance keys. In [22], the authors introduced a multimodal medical image fusion scheme using Non-Subsampled Shearlet Transformation (NSST). The fusion is employed based on the Whale Optimization Algorithm (WOA). The experimental tests of the proposed scheme demonstrated that the resultant fused image is robust. In [23], the authors introduced a multimodal medical image fusion scheme that utilizes an optimized weighted average fusion (OWAF). The experimental tests of the proposed scheme demonstrated superior performance compared to the existing traditional fusion methods.

In [24], the authors introduced a multimodal medical image fusion based on guided filter random walks (GFRWs) and spatial frequency. The framelet transform (FT) is employed to decompose the images to be fused. After that, the GFRWs and spatial frequency (SF) are utilized to obtain the fused residual image and the fused approximate image. The experiment tests demonstrated that the suggested scheme outperforms better with respect to both subjective and objective metrics. In [25], the authors introduced a multi-modal image fusion scheme using a segmentation map of the ant colony approach. The proposed approach performs a maximum selection rule in ensemble empirical mode decomposition (EEMD) transform. After that, the proposed approach employs the pseudo-color image information to determine pixels spatial regions of the same object. The suggested approach utilizes the majority voting procedure for merging both of the fusion map and segmentation map results. Experiment outcomes indicated that that the suggested approach produces enhanced fusion outcomes. In [26], the authors introduced a hybrid optimized medical fusion scheme using shark smell optimization and world cup optimization algorithms to produce a high quality fusion of medical images. The presented scheme utilizes the wavelet transform and the homomorphic filter to improve the model efficiency. Simulations tests result demonstrated that the proposed model gives better efficiency compared to the other the studied schemes.

In [27], the authors introduced a hybrid medical fusion scheme based on combining the NSCT with DTCWT. The hybrid medical fusion scheme is examined using a set of simulation experiments on distinct image multimodalities. The experimental outcomes are investigated with a set of well-known fusion evaluation keys, and the outcomes demonstrated that the suggested scheme outperforms well with respect to subjective and objective metrics. In [28], the authors introduced a multimodal MRI-PET medical image fusion scheme based on the 2D Hartley transform (HT) in HSV color space. The medical PET color image is transformed into HSV channels. Then, the MRI and PET image V component are split into 8*8 segments and after that employ a 2D Hartley transformation for on every block of two images to be fused. After that, estimate the variance of every block for the two images and choose the best blocks. Then, employ the inverse of 2D HT and all blocks are structured into one image. Finally, the V component, H, S are multiplexed to obtain the HSV image and transform HSV to RGB to obtain the resulted final fused image. The test result demonstrated the superiority of the suggested model. In [29], the authors introduced a medical image fusion scheme that adopts a rolling guidance filter to split the source image into detail and structural components. Laplacian Pyramid is utilized to fuse the structural component and a sum-modified-laplacian (SML) fusion rule is utilized to fuse the detail component. The resulted fused-image can be produced by merging the fused structural component and fused detail component. The experimental outcomes indicated that the proposed model is superior compared to traditional medical image fusion approaches. In [30], the authors introduced a deep learning-based multimodal medical image fusion scheme. The proposed deep learning-based multimodal medical image fusion model has the advantage of enhancing the fusion result, image clarity and efficiency. The examined experimental tests ensure the efficiency of the suggested model.

3 Image fusion methods

The main fusion process objective between two images of the same object is to collect all the needed information in a new fused image. This acquired fused image should have an improved quality than both of them. The following subsections some traditional fusion methods, which were used to evaluate the proposed approach will be discussed, and then the proposed PCA based SVD fusion algorithm will be discussed in detail.

3.1 Discrete wavelet transform (DWT)

Image processing in the transform domains has several advantages, such as a higher Signal to Noise Ratio (SNR), reduced features ... setc. [31]. One of the common transform domains used in image fusion is the wavelet transform [32, 33]. This method includes the following steps:

Both CT and MRI images are analyzed by using 2D-DWT and the four components LL, LH, HL, and HH are obtained.

Each component of the two images is fused by using appropriate fusion rule (ex: LL1 and LL2).

In most cases LL’s components are fused using the maximum fusion rule while the other components are fused by using the average rule [34].

Inverse DWT is applied on the obtained fused components to produce fused image.

3.2 Dual-tree complex wavelet transform fusion (DT-CWT)

DT-CWT is used as a fusion approach instead of DWT to increase the poor directionality using the complex extension of it [8]. This method includes analysis and synthesis stages. In analysis stage, four levels of analysis filter bank of the DT-CWT are applied on any signal X(n), and produces twelve directional wavelets; six of them are real trees and 6 are imaginary trees which improve the directionality [35]. The synthesis stage is the opposite of the analysis stage. In this stage both real and imaginary wavelets are composed to produce the original signal again.

The fusion process between two images using DT-CWT can be summarized as following [9 , 36]:

Both images are analyzed using 2D DT-CWT, this produces a real low pass image that represents the detail coefficients and other six complex high pass sub-images that represent the approximation coefficients) for each image.

Approximation coefficients from both images are fused using the uniform average fusion rule.

Absolute maximum fusion rule can be used for the detail coefficients.

Inverse 2D DT-CWT is employed on the fused coefficients to produce the fused image.

3.3 Curvelet fusion

In this method, Curvelet fusion is used in fusing the images that contain a lot of curved shapes in their details [37, 38]. The general steps of the fusion process based on Curvelet transform on the image are:

Apply the Ridgelet Transform to get the image sub-bands.

Use the maximum fusion rule to fuse the sub-bands.

Image Pis split into four components Δ1, Δ2, Δ3, and P3 via the additive wavelets transform as shown in Eq. (1) [38]: $P = \sum_{i = 1}^{n - 1} Δ i + ρ_{n}$ (1)

Apply the inverse Curvelet transform on P3 component of the input image, and on the fused Δ1, Δ2, Δ3 to get the fused image.

The tiling process is applied on Δ1, Δ2, and Δ3 sub-bands.

Discrete Ridgelet Transform is applied on each obtained tile.

3.4 Principal component analysis (PCA) based image fusion

The PCA transforms a series of correlated data into an uncorrelated data by removing the redundant data [39, 40]. The general steps of the image fusion algorithm using PCA can be summarized as follows:

Obtaining the main components (PC’s) of the two input images (to be fused)

Rebuild both images once more by using only their PC’s of the largest Eigen-values.

Fuse the new acquired images using appropriate fusion rule.

3.5 SVD background

SVD is a matrix factorization technique used to generate a low-range approximation of any matrix [41]. For a matrix A having a size of m×n, SVD is applied using the following equation [31]: $SVD (A) = U \times S \times V^{T}$ (2) where the matrices U, S, and V have the dimensions (m×m), (m×n), and (n×n), respectively. The matrices U and V are orthogonal matrices and their first r columns are the orthogonal Eigen-vectors corresponding to the r Eigen-values of the AAT and ATA matrixes. These matrices also designate the left and right singular vectors.

The matrix S is a diagonal matrix called a singular matrix where its first r-diagonal elements (s₁, s₂... s s_r) are the Eigen-values of the matrix A and have the property s_i >0 and s₁ ≥ s₂ ≥ s₃... s.≥s_r. All other entries in this matrix are 0’s.

4 The proposed PCA based SVD image fusion method

This section presents our fusion technique to the CT and MRI images. The proposed method depicted in Fig. 1 uses PCA and Singular value decomposition SVD to reduce both of the processing time and the needed memory during the fusion process. On the other hand, we get a fused image with better characteristics than others obtained by other traditional algorithms. This would allow us to build a hardware unit based on the implementation of this algorithm to apply the fusion process in real time. In this section, we introduce the CT and MRI images fusion by applying the fusion process on the new $S_{\underset{new}{CT}}$ and $S_{\underset{new}{MRI}}$ Matrixes instead of the original image matrix. The important feature of these new matrixes is its small size, which is a one dimensional matrix of size 1×M for an image of size N×M, and that is why the proposed algorithm has a very short processing time, and needs a small memory size compared to the other traditional algorithms. Assume that the images which are going to be fused are registered; the basic steps of the proposed scheme are listed below as:

Fig. 1

The proposed PCA based SVD Fusion Method Block Diagram.

The two images are normalized to reduce the differences in the grayscale values, mostly to normalize an image to have a specified mean M₀, and a specified variance V₀, can be performed using this equation [42]: $N (i, j) = {\begin{matrix} M_{0} + \sqrt{\frac{V_{0} (I (i, j) - M)^{2}}{V}} ifI (i, j) > M \\ M_{0} - \sqrt{\frac{V_{0} (I (i, j) - M)^{2}}{V}} otherwise \end{matrix}$ (3) where M is the mean value of the image, V is its variance, and I (i, j) is the grey value of the pixel (i, j), and N (i, j) is the normalized value of the pixel (i, j) [8, 12].

SVD is used in both obtained normalized images, to have the diagonal S_matrix for CT and MRI images.

For each of CT and MRI images, we use its S_matrix of M x M dimension to find a new matrix called S_new matrix, which will be one dimensional matrix (1 x M). S_new matrix is generated by taking only the diagonal values of S_matrix as in the following equation: $S_{new} [1, i] = S [i, i]$ (4)

where i=0, 1, 2, 3... s..m.

Generate the $S_{\underset{new}{CT}}$ for the CT image from the S_CT matrix by taking only the diagonal values of S_CT as in Eq. 4. $S_{\underset{new}{CT}} [1, i] = S_{CT} [i, i]$ (5) where i=0, 1, 2, 3... s..m. S_CT represents the diagonal matrix of the CT image and called a singular matrix of M x M dimensions, and $S_{\underset{new}{CT}}$ is one dimensional matrix of size of (1 x M),

Generate the $S_{\underset{new}{MRI}}$ for the MRI image from the S_MRI matrix by taking only the diagonal values of S_MRI as in Eq. 5. $S_{\underset{new}{MRI}} [1, i] = S_{MRI} [i, i]$ (6) where i=0, 1, 2, 3... s..m. S_MRI represents the diagonal matrix of the MRI image and called a singular matrix of M x M dimensions, and $S_{\underset{new}{MRI}}$ is one dimensional matrix of size of (1 x M),

Calculating the Eigen-values and the Eigen-vectors for S_new matrixes; $S_{\underset{new}{CT}}$ and $S_{\underset{new}{MRI}}$ together as a two datasets, and then calling the Eigen-vector of the largest Eigen-value Ev1, and the Eigen-vector of the second largest Eigen-value Ev2 [43].

Finding the fused image by applying the proposed adaptive fusion rule in the following equation: $Fusedimage = Ev 1 \times {CT}_{image} + Ev 2 \times {MRI}_{image}$ (7)

5 Fusion quality evaluations metrics

In our algorithm, obtaining the S_new matrixes; $S_{\underset{new}{CT}}$ and $S_{\underset{new}{MRI}}$ step is one of the most important steps. It reduces the time and the memory complexity of the algorithm by reducing the processing time and the memory required for obtaining the fused image. Mainly that is because of applying the processing on two 1 x M matrixes instead of two M x M matrixes of the original images, and this will lead to a simple hardware implementation of the proposed algorithm with a small resources and a low cost. To evaluate the proposed algorithm as compared to the other traditional algorithms, the visual quality and other eleven image quality metrics were applied on the fused image achieved by the fusion algorithms. These eleven metrics include the most common metrics such as Entropy, PSNR, Average Gradient, Standard Deviation (STD), Local Contrast, and Edge Intensity metrics [38 , 45] and advanced fusion metrics such as Mutual Information (MI) [46 , 48], Universal Image Quality Index (UIQI), Xydeas & Petrovic (Q^ab/f), Structural Similarity (SSIM), and Feature Similarity Index (FSIM) metrics). All these metrics are explained at [44 , 48].

6 Experimental results

The presented results were obtained through using PCA, DWT, DT-CWT, Curvelet, and the proposed PCA based SVD fusion algorithms on a set of CT and MRI real images. In all instances, the results of applying the five fusion algorithms on different four cases; each is 256×256 pixel grey scale image are shown. The last mentioned eleven fusion quality metrics are used in measuring the quality of the obtained fused images. The main purpose of the proposed algorithm is to be implemented as a hardware unit, which apply the fusion process on two registered images with the same quality of the known fusion algorithms. The proposed algorithm gives the best results for each of the metrics; Average gradient, PSNR, STD, Edge intensity, UIQI, MI, Q^ab/f, Elapsed time and the amount of saved memory, and a lower but reasonable values for SSIM, FSIM, Entropy, and local contrast metrics. Figure 2 shows the original CT and MRI images for Case 1 and also the results of the fusion process using five different techniques; PCA, DWT, DT-CWT, Curvelet, and the proposed PCA based SVD fusion approaches. Quality measurements for Case 1 were shown in Table 1. Figure 3 shows the CT and MRI images for Case 2 and also the fusion process results using the previous five different fusion techniques. Quality measurements for Case 2 were shown in Table 2. Figure 4 shows the original CT and MRI images for Case 3 and also the results of the fusion process using the previous five different fusion techniques. The quality measurements for Case 3 were shown in Table 3. Figure 5 shows the original CT and MRI images for Case 4 and also the results of the fusion process using the previous mentioned five different fusion techniques. The quality measurements for Case 4 are shown in Table 4. The observed results shows that the proposed algorithm depends on both the input images more than the other four algorithms such as PCA, DWT, DT-CWT, Curvelet and that appears from the values of MI metrics in Tables 1 –4. Also, there are more correlation and clearance offered by the proposed algorithm for the fused image, which is showed in the UIQI and STD results. The amount of transferred information from the input images to the fused image using the proposed algorithm is more than those by the other algorithms as shown in the results of Q^ab/f, and Edge Intensity metrics. Also, the PSNR values show a good results for the proposed algorithm compared with the other fusion methods. On the other hand, the amount of information in the fused image calculated by Entropy results is lower than the other four algorithms, and the same for the Local contrast of the image. Also, the results of the similarity, either regions similarity expressed in terms of SSIM, or edges similarity expressed in terms of FSIM are lower than the obtained by other fusion methods. As showed in Tables 1 –4, the proposed PCA based SVD fusion approach has a lower execution time compared to the other fusion algorithms, and the amount of reduced time is increased when the input images size increased. We can calculate the fusion time versus image sizes for the previous algorithms such as PCA, DWT, DT-DWT, Curvelet, and the proposed scheme. One of the previous cases was chosen randomly, and had been resized to 32×32 pixel size, 64×64 pixel size, 128×128 pixel size, and 256×256 pixel size. The fusion process time of each fusion algorithm is recorded as shown in Table 5. The conclusion of the results shows that the proposed PCA based SVD fusion approach saves more time compared to the other fusion algorithms, and the saved time is increased when the image size is increased. This is mainly because of using S_new matrix, which has 1×M dimension for an image of size N×M instead of using the whole image in the fusion process during the algorithm execution. Finally, to discuss the memory requirements for the proposed algorithm, it was found that for a grey scale image of size 16×16 pixel, each pixel of the image will be stored in 1-byte of the memory, and so the needed memory to store the whole image is 256 byte. To deal with different fusion approaches such as PCA, DWT, DT-CWT, Curvelet, the processing is applied on the both CT and MRI images at the same time. So, for fusion of two grey scale images of size 16 x 16, the fusion process needs 512 byte of the memory. For the proposed PCA based SVD fusion approach, the processing is applied on the S_new matrix which is only 1×M dimension matrix for M×N image instead of the original image. So, in this case it needs 1 x 16 x 1 byte=16 byte for each image and we will need only 32 bytes of the total memory, which saves 480 byte compared to the other four algorithms. The last calculations are applied on different image sizes and showed in Table 6. As shown in Table 6, the proposed algorithm uses a small amount of memory compared to the other four algorithms. The amount of saved memory by the proposed algorithm is increased when the size of the images we are going to fuse increased, and that is shown clearly in the last column of Table 6. This large amount of saved memory enables us to implement the proposed algorithm as a real time hardware unit.

Fig. 2

Fusion results for the original images (CT and MRI) of size 256×256 pixel with 5 different fusion methods for Case 1.

Table 1

Quantitative results for the original fused CT and MRI images of size 256×256 pixel with 5 different fusion methods for Case 1

Measurement method	PCA	DWT	DT-CWT	Curvelet	PCA-based SVD
Average gradient	0.01194	0.01783	0.02310	0.01611	2.95560
Local Contrast	0.35576	0.38750	0.64207	0.38343	0.33508
PSNR_CT	13.3571	23.69545	17.74137	23.72606	25.9693
PSNR_MRI	15.8939	14.21527	19.33152	14.96618	23.3166
STD	0.15837	0.19129	0.19198	0.17868	40.67747
Edge intensity	0.12926	0.18622	0.24974	0.17452	31.93803
Shannon entropy	6.08729	6.19920	7.00239	6.17380	6.07940
Elapsed time	0.17160	0.90481	0.10920	1111.2	0.04680
UIQI Ct	0.00008	0.53732	0.23277	0.35540	0.88544
UIQI MRI	0.00118	0.18956	0.55279	0.23660	0.64610
mSSIM Ct	0.00679	0.99952	0.99786	0.99959	0.76374
mSSIM MRI	0.16828	0.99478	0.99846	0.99561	0.55013
MI	0.21486	0.51659	0.30929	0.29254	0.75992
Q^ab/f	0.00019	0.73429	0.30478	0.21460	0.34374
FSIMCt	0.60440	0.94579	0.92197	0.94885	0.81444
FSIM MRI	0.50528	0.94592	0.96475	0.93971	0.79458

Fig. 3

Fusion results for the original images (CT and MRI) of size 256×256 pixel with 5 different fusion methods for Case 2.

Table 2

Quantitative results for the original fused CT and MRI images of size 256×256 pixel with 5 different fusion methods for Case 2

Measurement Method	PCA	DWT	DT-CWT	Curvelet	PCA-based SVD
Average gradient	0.02502	0.03864	0.04309	0.03406	6.23039
Local Contrast	0.53611	0.58691	0.87051	0.60572	0.50336
PSNR_CT	11.2534	12.5198	15.10513	12.90086	20.6716
PSNR_MRI	11.9828	12.5031	13.30015	12.96963	19.9294
STD	0.25603	0.29156	0.28868	0.28272	57.14593
Edge intensity	0.26334	0.39869	0.45589	0.36965	65.92772
Shannon entropy	7.60477	7.78112	7.55499	7.78239	7.57077
Elapsed time	0.65520	0.79561	0.10920	1145.31	0.03120
UIQI Ct	0.00007	0.41429	0.41609	0.31383	0.74686
UIQI MRI	0.00002	0.58212	0.51546	0.43828	0.84678
mSSIM Ct	0.04993	0.99238	0.99603	0.99319	0.56616
mSSIM MRI	0.00549	0.99259	0.99400	0.99356	0.62451
MI	0.83940	0.86000	0.49922	0.56137	0.94586
Q^ab/f	0.00016	0.45791	0.13962	0.14375	0.33148
FSIMCt	0.39207	0.95077	0.96075	0.95383	0.78909
FSIM MRI	0.31668	0.94324	0.91723	0.94402	0.75494

Fig. 4

Fusion results for the original images (CT and MRI) of size 256×256 pixel with 5 different fusion methods for Case 3.

Table 3

Quantitative results for the original fused CT and MRI images of size 256×256 pixel with 5 different fusion methods for Case 3

Measurement Method	PCA	DWT	DT-CWT	Curvelet	PCA-based SVD
Average gradient	0.01870	0.03093	0.03624	0.03014	4.79774
Local Contrast	0.40809	0.49923	0.75350	0.53639	0.40135
PSNR_CT	10.3423	14.60326	15.85625	15.26212	21.8422
PSNR_MRI	11.9223	12.17082	13.50069	12.73808	20.2322
STD	0.28689	0.33293	0.32272	0.31705	70.00427
Edge intensity	0.20312	0.32615	0.39283	0.32811	51.9878
Shannon entropy	7.05327	7.31251	7.26395	7.37808	7.06265
Elapsed time	0.26520	0.92040	0.09360	1205.46	0.01560
UIQI Ct	0.00013	0.43326	0.33207	0.33491	0.69215
UIQI MRI	0.00002	0.49792	0.43817	0.38740	0.53889
mSSIM Ct	0.17127	0.99564	0.99707	0.99644	0.58592
mSSIM MRI	0.02641	0.99195	0.99414	0.99327	0.61553
MI	0.99997	0.90351	0.54759	0.70382	0.88276
Q^ab/f	0.00014	0.31354	0.15674	0.14809	0.32469
FSIMCt	0.56266	0.94560	0.92047	0.94175	0.77548
FSIM MRI	0.44131	0.92979	0.93803	0.94245	0.75578

Fig. 5

Fusion results for the original images (CT and MRI) of size 256×256 pixel with 5 different fusion methods for Case 4.

Table 4

Quantitative results for the original fused CT and MRI images of size 256×256 pixel with 5 different fusion methods for Case 1

Measurement Method	PCA	DWT	DT-CWT	Curvelet	PCA-based SVD
Average gradient	0.01047	0.01429	0.015069	0.01333	2.18534
Local Contrast	0.68985	0.74702	1.31620	0.80712	0.70331
PSNR_CT	21.6092	16.0755	19.4910	16.3351	18.9072
PSNR_MRI	15.6883	24.8813	21.1737	25.16178	25.39167
STD	0.11115	0.12921	0.11333	0.12014	21.98655
Edge intensity	0.11382	0.15371	0.16401	0.14616	23.70420
Shannon entropy	5.62188	5.85859	5.29160	5.87930	5.33131
Elapsed time	0.06240	0.84240	0.09360	1161.17	0.06240
UIQI Ct	0.00028	0.10325	0.10446	0.07507	0.04697
UIQI MRI	0.00005	0.87053	0.42479	0.60191	0.84188
mSSIM Ct	0.82194	0.99622	0.99835	0.99651	0.17910
mSSIM MRI	0.05913	0.99961	0.99895	0.99967	0.86435
MI	0.06323	0.14533	0.00882	0.06019	0.66641
Q^ab/f	0.00017	0.72843	0.05235	0.13284	0.55331
FSIMCt	0.82421	0.96919	0.98867	0.96548	0.69821
FSIM MRI	0.67046	0.96066	0.94183	0.97049	0.92466

Table 5

Execution time of the fusion process for PCA, DWT, DT-CWT, Curvelet, and Proposed techniques applied on different image sizes

Image size	Elapsed time for each fusion approach in Sec
	PCA	DWT	DT-CWT	Curvelet	PCA-based SVD
32×32 pixel	0.09360	1.06081	0.05600	31.3562	0.0156
64×64 pixel	0.04680	1.01401	0.07800	119.6216	0.0234
128×128 pixel	0.0624	1.0452	0.0780	466.1466	0.0312
256×256 pixel	0.65520	0.79561	0.10920	1145.31	0.0390

Table 6

Memory requirements needed for each fusion approach in bytes

Image size	Memory needs for each fusion approach					Saved
	PCA	DWT	DT-CWT	Curvelet	PCA-based SVD	Memory
8×8 pixel	128	128	128	128	16	112
16×16 pixel	512	512	512	512	32	480
32×32 pixel	2048	2048	2048	2048	64	1984
64×64 pixel	8192	8192	8192	8192	128	8064
128×128 pixel	32864	32864	32864	32864	256	32608
256×256 pixel	131072	131072	131072	131072	512	130560
512×512 pixel	524288	524288	524288	524288	1024	523264

7 Conclusion

In this study, we presented an efficient CT/MRI images fusion approach based on the PCA and SVD techniques. Eleven quality metrics were used in estimating the fused image resulted by the proposed approach, and compare it with the images acquired by the other fusion approaches. The experimental results of the proposed technique are superior compared with other traditional techniques such as PCA, DWT, DT-CWT, and Curvelet fusion methods. The algorithm has been extensively tested and applied to 4 sets of different data in the form of CT and MRI images. The experimental results show that the proposed PCA based SVD fusion approach has better results for: (1) Detailed contrast and texture variation, which appeared in the results of average gradient measure. (2) SNR in the obtained fused image that appeared in the results of PSNR_CT and PSNR_MRI measures. (3) Clearness of the obtained fused image, which appeared in the results of standard deviation measure. (4) Edges clearness of the obtained fused image, which appeared in the edge intensity measure results. (5) Correlation value between the obtained fused image and input images, which appeared in the results of UIQI. (6) The results of Q^ab/f, and Edge Intensity metrics reveals that, the information transferred from CT and MRI images to the obtained fused one is increased. (7) The elapsed time of the fusion process is reduced compared to the other algorithms, and the amount of reduction is directly proportional to the size of the images to be fused, which makes the proposed PCA based SVD fusion algorithm very fast. (8) Finally, the memory requirements needed by the proposed algorithm had been reduced compared with the other fusion algorithms.

Footnotes

Acknowledgment

This study was funded by the Deanship of Scientific Research, Taif University Researchers Supporting Project number (TURSP-2020/08), Taif University, Taif, Saudi Arabia.

References

Guihong

, Dali

and Pingfan

, Medical image fusion by wavelet transform modulus maxima, Opt Express 9(4) (2001), 184–190.

El-Hoseny

H.M.

, El Kareh

Z.Z.

, Mohamed

W.A.

, El-Banby

G.M.

, Mahmoud

K.R.

, Faragallah

O.S.

, El-Rabaie

, El-Madbouly

and Abd El-Samie

F.E.

, An optimal wavelet-based multi-modality medical image fusion approach based on modified central force optimization and histogram matching, Multimedia Tools and Applications 78(18) (2019), 26373–26397.

Sulochana

, Vidhya

, Mohanraj

and Vijayasekaran

, Effect of wavelet based image fusion techniques with principal component analysis (PCA) and singular value decomposition (SVD) in supervised classification, (2017), 338–348.

El-Hoseny

H.M.

, El-Rahman

W.A.

, El-Shafai

, El-Rabaie

, Mahmoud

K.R.

, Abd El-Samie

F.E.

and Faragallah

O.S.

, Optimal multi-scale geometric fusion based on non-subsampled contourlet transform and modified central force optimization, International Journal of Imaging Systems and Technology 29(1) (2019), 4–18.

El-Hoseny

H.M.

, Abd El-Rahman

W.A.

, El-Rabaie

, Abd El-Samie

F.E.

and Faragallah

O.S.

, An efficient DT-CWT medical image fusion system based on modified central force optimization and histogram matching, Infrared Physics & Technology 94 (2018), 223–231.

Río

, Sánchez

, Conde

, Tsitiridis

, Gómez

J.R.

, de Diego

I.M.

and Cabello

, Face-based recognition systems in the ABC e-gates, In Systems Conference (SysCon), 2015 9th Annual IEEE International, pp. 340–346. IEEE, 2015.

Liang

, He

, Liu

and Zeng

, Image fusion using higher order singular value decomposition, IEEE transactions on Image Processing 21(5) (2012), 2898–2909.

Thirunavukkarasu

, Kumar

J.S.

, Chae

G.S.

and Kishorkumar

, Non-intrusive Forensic Detection Method Using DSWT with Reduced Feature Set for Copy-Move Image Tampering, Wireless Personal Communications 98(4) (2018), 3039–3057.

Farid

M.S.

, Mahmood

and Al-Maadeed

S.A.

, Multi-focus Image Fusion Using Content Adaptive Blurring, Information Fusion, (2018), 96–12.

10.

Vyas

, Kanumuri

and Sheoran

, Cross spectral iris recognition for surveillance based applications, Multimedia Tools and Applications, (2018), 1–19.

11.

Gupta

, Singh

and Kumar

, Fusion of texture and wavelet features of PALSAR image using LDA and PCA for land cover classification, International Journal of Image and Data Fusion 8(4) (2017), 354–374.

12.

Aktar

M.N.

, Lambert

A.J.

and Pickering

, An automatic fusion algorithm for multi-modal medical images, Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, (2017), 1–15.

13.

Zhou

, Liang

, Yang

, Xu

and Yang

, Recognition of Wheat Spike from Field Based Phenotype Platform Using Multi-Sensor Fusion and Improved Maximum Entropy Segmentation Algorithms, Remote Sensing 10(2) (2018), P246.

14.

Guan

J.-S.

, Kang

S.-B.

and Sun

, Medical image fusion algorithm based on multi-resolution analysis coupling approximate spare representation, Future Generation Computer Systems 98 (2019), 201–207.

15.

, Li

, Du

and Xiao

, Multimodal medical image fusion via laplacian pyramid and convolutional neural network reconstruction with local gradient energy strategy, Computers in Biology and Medicine 126 (2020), 104048.

16.

Wang

, Li*

and Huang

, Medical image fusion based on hybrid three-layer decomposition model and nuclear norm, Computers in Biology and Medicine 129 (2021), 104179.

17.

, Wang

, Chen

and Zhao

, Medical image fusion using segment graph filter and sparse representation, Computers in Biology and Medicine 131 (2021), 104239.

18.

Hermessia<

, Mouralia

and Zagroubaa

, Multimodal Medical Image Fusion Review: Theoretical Background and Recent Advances, Signal Processing, https://doi.org/10.1016/j.sigpro.2021.108036, 2021.

19.

, Li

and Tan

, Three-layer medical image fusion with tensor-based features, Information Sciences 525 (2020), 93–108.

20.

Prakash

, Park

C.M.

, Khare

, Jeon

and Gwak

, Multiscale fusion of multimodal medical images using lifting scheme based biorthogonal wavelet transform, Optik – International Journal for Light and Electron Optics 182 (2019), 995–1014.

21.

Polinati

and Dhuli

, Multimodal medical image fusion using empirical wavelet decomposition and local energy maxima, Optik – International Journal for Light and Electron Optics 205 (2020), 163947.

22.

Nair

R.R.

and Singh

, An Optimal Registration on Shearlet domain with Novel Weighted Energy fusion for Multi-Modal Medical Images, Optik – International Journal for Light and Electron Optics 225 (2021), 165742.

23.

Shehanaz

, Daniel

, Guntur

S.R.

and Satrasupalli

, Optimum weighted multimodal medical image fusion using particle swarm optimization, Optik – International Journal for Light and Electron Optics 231 (2021), 166413.

24.

Kong

, Chen

and Lei

, Medical image fusion using guided filter random walks and spatial frequency in framelet domain, Signal Processing 181 (2021), 107921.

25.

Shahdoosti*

H.R.

and Tabatabaei

, MRI and PET/SPECT image fusion at feature level using ant colony based segmentation, Biomedical Signal Processing and Control 47 (2019), 63–74.

26.

Xua

, Sia

, Jiangb

, Sunb

and Ebrahimian

, Medical image fusion using a modified shark smell optimizationalgorithm and hybrid wavelet-homomorphic filter, Biomedical Signal Processing and Control 59 (2020), 101885.

27.

Rajalingama

, Priya

and Bhavani

, Hybrid Multimodal Medical Image Fusion Using Combination of Transform Techniques for Disease Analysis, Procedia Computer Science 152 (2019), 150–157.

28.

Haribabu

and Guruvaiah

, Statistical Measurements of Multi Modal MRI–PET Medical Image Fusion using 2D–HT in HSV color Space, Procedia Computer Science 165 (2019), 209–215.

29.

Chen

, Zhang

, Lu

, Li

, Hu

and Yang

, A novel medical image fusion method based on Rolling Guidance Filtering, Internet of Things, https://doi.org/10.1016/j.iot.2020.100172, 2021.

30.

, Zhao

, Lv

and Li

, Medical image fusion method by deep learning, International Journal of Cognitive Computing in Engineering 2 (2021), 21–29.

31.

Krishnamoorthy

and Soman

K.P.

, Implementation and Comparative Study of Image Fusion Algorithms, International Journal of Computer Applications 9(11) (2010).

32.

Goshtasby

A.A.

and Nikolov

, Image fusion: Advances in the state of the art, Information Fusion 8(2) (2007), 114–118.

33.

Wang

, Sun

and Guan

, The Application of Wavelet Transform to Multi-modality Medical Image Fusion, Sensing and Control IEEE International Conference on, (2006), 70–274.

34.

Pohl

and van Genderen

J.L.

, Multisensor image fusion in remote sensing: concepts, methods and application, Int J of Remote Sensing 19 (1998), 823–854.

35.

Zheng

, Hou

, Bian

and Qin

, Effective Image Fusion Rules of Multi-scale Image Decomposition, International Symposium on Image and Signal Processing and Analysis, (27-29)(9) (2007), 362–366.

36.

, Yang

and Hu

, Performance comparison of different multi-resolution transforms for image fusion, Information Fusion 12(2) (2011), 74–84.

37.

Kannan

and Perumal

S.A.

, Optimal Decomposition Level of Discrete Wavelet Transform for Pixel Based Fusion of Multi-Focused Images, International Conference on Computational Intelligence and Multimedia Applications (12) (2007), 314–318.

38.

Nencini

, Garzelli

, Baronti

and Alparone

, Remote sensing image fusion using the curvelet transform, Information Fusion 8(2) (2007), 143–156.

39.

Soliman

R.F.

, Amin

and Abd El-Samie

F.E.

, C22, Enhanced fusion for infrared and visible images, 29th National Radio Science Conference (NRSC), (2012).

40.

Qiguang

and Baoshu

, A novel image fusion algorithm using FRIT AND PCA, 10th Information Fusion International Conference, (9-12)(7) (2007), 1–5.

41.

Zhou

, Principal Component Analysis Based Image Fusion Routine with Application to Stamping Split Detection, Dissertation presented to the Graduate School of Clemson University In Partial Fulfilment of the Requirements for the Degree Doctor of Philosophy Automotive Engineering, (8) (2010).

42.

Sarwar

, Karypis

, Konstan

and Riedl

, Incremental Singular Value Decom-position Algorithms for Highly Scalable Recommender Systems, Fifth International Conference on Computer and Information Science, (2002) 27–28.

43.

Vaclav

, Guenther

, Daniel

and Kadir

, Image normalization strategies for organ specific models of functional uptake in FDG-PET/CT, Nuclear Science Symposium Conference Record, (19-25)(10) (2008), 5225–5228.

44.

Faragallah

O.S.

, El-Hoseny

H.M.

, El-Shafai

, El-Rahman

W.A.

, El-Sayed

H.S.

, El-Rabaie

, Abd El-Samie

F.E.

and Geweid

G.G.N.

, A Comprehensive Survey Analysis for Present Solutions of Medical Image Fusion and Future Directions, IEEE Access 9 (2021), 11358–11371.

45.

Raut

G.N.

, Paikrao

P.L.

and Chaudhari

D.S.

, A Study of Quality Assessment Techniques For Fused Images, IJITEE 2(4) (2013).

46.

Yang

, Zhang

, Wang

and Liu

, A novel similarity based quality metric for image fusion, Information Fusion 9(2) (2008), 156–160.

47.

Zhang

and Mou

D.X.

, FSIM: A Feature Similarity Index for Image Quality Assessment, Image Processing, IEEE Transactions on 20(8) (2011), 2378–2386.

48.

El-Hoseny

H.M.

, Abd El-Rahman

, El-Shafai

, El-Banby

G.M.

, El-Rabaie

, Abd El-Samie

F.E.

, Faragallah

O.S.

and Mahmoud

K.R.

, Efficient multi-scale non-sub-sampled shearlet fusion system based on modified central force optimization and contrast enhancement, Infrared Physics & Technology 102 (2019), 102975.