Abstract
This paper presents a new approach to the multi-modal medical image fusion based on Principal Component Analysis (PCA) and Singular value decomposition (SVD).The main objective of the proposed approach is to facilitate its implementation on a hardware unit, so it works effectively at run time. To evaluate the presented approach, it was tested in fusing four different cases of a registered CT and MRI images. Eleven quality metrics (including Mutual Information and Universal Image Quality Index) were used in evaluating the fused image obtained by the proposed approach, and compare it with the images obtained by the other fusion approaches. In experiments, the quality metrics shows that the fused image obtained by the presented approach has better quality result and it proved effective in medical image fusion especially in MRI and CT images. It also indicates that the paper approach had reduced the processing time and the memory required during the fusion process, and leads to very cheap and fast hardware implementation of the presented approach.
Introduction
Nowadays, medical images are a major method of diseases diagnosis such as tumor tissues among children and adults alike [1]. There are several modalities of imaging systems such as Computed Tomography (CT), and The Magnetic resonance (MR). For hard tissues CT imaging model would be the best choice, on the other side for soft tissues MR imaging would be the best. Sometimes brain tumorous contains soft and hard tissues together, because doctors need information from CT and MR images together, so CT model or MR model alone is not enough. That is why the fusion between different imaging modalities became important. Fusion process had been implemented using different methods. These methods are simple methods such as pixel averaging, moderate complicated methods such as PCA-based fusion, and advanced methods such as those in the different transform domains such as Wavelet-based fusion [2].
The image merging process is the merging of two or more images in a section to produce high-quality, high-resolution images without misrepresentation, so that the merged image can be better interpreted in several applications by giving more precise information [3–5]. In this sense, a number of methods have been presented in the literature, such as the high-pass filtering method [6], analysis of the major components of transformation such as PCA, Hue and Saturation (HIS), and Brovey transformation and intensity [7].
Although these spatial methods [8–10] provide better visual excellence images, several studies show that colors are visually distorted in merged images.
The high-pass filter technique conserves spatial detail, but there are noises within the merged image. The IHS-transform changes intensity components of high resolution panchromatic images, while spatial-resolution enhances the supernatural artifacts in a resulted appearance due to the replacement of the density element [11].
The PCA technique substitutes the first major element of the multispectral image with PAN images. It increases the longitudinal feature and the supernatural artifacts due to the deficiency of information in the reconstruction of the original image.
Lately, transformation domain methods using multi-resolution analysis have been developed, such as wavelet transformation. Wavelet transformations can store and preserve the details of time and frequency. The entered input images are disintegrating addicted to various frequency concentrations then, the fusion procedures are applied by substituting the feature measurements with element parts of added original image. Although image fusion based on wavelet gives high supernatural excellence, longitudinal/spatial resolution may reduce because of critical down-sampling [12]. Spatial information is needed for highly medical applications due to the convoluted nature of land use characteristics. Even though, several image fusion techniques are accessible in the works for integrating remotely-recognized data, not all methods meet the necessity for mapping the properties of the earth’s surface [10].
In recent years, researchers have used other multi-resolution analyzes, including wavelet packet transform (WPT), multi wavelet transform (MWT), discrete wavelet transform (DWT), curvelet transform (CUT), atrous wavelet transform (ATWT), Contourlet transform (CT), Framelet transform (FRT), and nonsupsampled Contourlet transform (NSCT). Great advances have been produced in image fusion knowledge but little investigation has been done on other purposes such as categorization and feature extraction [4, 12].
Simple and moderate fusion methods give a fused image in a small processing time but without high quality in several metrics [13]. Advanced methods such as Discrete Wavelet Transform (DWT) and Curvelet fusions introduce a high quality image, but those take a long time compared to the simple methods especially the Curvelet fusion. According to the required fused image characteristics, and the time requirements, one of those methods would be selected [1].
This paper presents a proposed image fusion algorithm to integrate all the relevant information of both CT and MRI imaging modalities. The proposed algorithm is based on a combination of PCA and SVD, as a pre-processing stage to fuse the CT and MRI images. By using the proposed PCA based SVD fusion algorithm, the processing time and the memory requirement of the fusion process would be reduced, with the same quality of other algorithms images. This will be a good step to implement the proposed algorithm with a cheap and fast hardware unit, so it can be attached to any scanner of the CT or MRI scanners.
The results of the proposed algorithm in several cases and different image dimensions indicate that the obtained fused images by the presented algorithm is better than those of the other compared algorithms in most of the quality metrics values. Moreover, the elapsed time of the fusion process using the proposed algorithm had been reduced at least 50% and 60% compared to PCA, and Dual-tree Continuous Wavelet Transform, and more than 98% comparing to wavelet and the Curvelet based fusion algorithms. When the size of the input images increased the saved time would be increased also by using the proposed algorithm in comparing of other traditional algorithms.
This paper is organized as follows: Section 2 discusses and explores related works. Section 3 presents the image fusion methods. Section 4 includes the proposed approach based on PCA and SVD fusion methods. Section 5 shows how to use the fusion quality evaluations metrics to evaluate the presented approach compared to the other traditional approaches. The experimental results are presented in section 6. Finally, section 7 concludes the paper.
Related works
In [14], the authors introduced a multi-scale analysis coupling approximate sparse representation image fusion scheme. The image high and low frequencies information are obtained. After that, the image high and low frequencies information are estimated using the coefficients of approximate sparse, and the approximate sparseness is transformed using absolute maximum selection scheme to provide the low frequency subband approximate coefficient and the high frequency subband. Then, the decision mapping is formulated, and the matching degree of coefficients exist within the subband are investigated to determine the decision estimate, and the image can be matched and merged. In [15], the authors proposed a medical fusion scheme using Laplacian pyramid and convolutional neural network (CNN). The multimodal medical images are generated using CNN. After that, Laplacian pyramid can be employed on decomposition and fusion processes. The optimal decomposition layers number can be estimated experimentally. The fused image is produced using the inverse of Laplacian transform. The output fusion results have better quality performance. In [16], the authors proposed an efficient medical image fusion scheme. The input image is firstly decomposed using a hybrid three-layer decomposition model into structure layer, texture layer, and local mean brightness layer. After that, the patches nuclear norm achieved by a sliding window, are estimated to provide the structure and texture layers weight maps. Both the subjective/objective tests ensured the advantage of the suggested scheme with respect to the related state-of-art schemes.
In [17], the authors introduced a medical image fusion scheme using the segmented graph filter (SGF) and the sparse representation (SR). The images are decomposed using the SGF into the base and the detailed images. After that, the base images are fused using the normalized Shannon entropy while the detailed images are fused based on SR fusion. The final fused-image can be produced through fusing both the fused-base and fused-detailed images. The test results show that the proposed method the fusion performance is comparable to the related state-of-art schemes. In [18], the authors introduced a comprehensive survey of medical image fusion schemes including different fusion methods strategies like pixel-level, feature-level and decision-level. Also, the fusion performance experimental tests using various fusion categories are given. In [19], the authors introduced a three-layer medical image fusion scheme which has three stages. With the first stage, various features using structure tensor are utilized to decompose the anatomical medical MRI image into its three-layer image representation. With the second stage, spatial frequency metric is suggested for merging the decomposed intensity and detail layers. Finally, the fused image is produced through adding the fused intensity, detail, and base layers, the fused layer, and the fused base layer. The subjective and objective evaluations demonstrated the effectiveness of the proposed scheme.
In [20], the authors introduced a multimodal medical image fusion algorithm that depends on a lifting-based biorthogonal wavelet transformation. The multi-scale fusion is applied in wavelet domain for the multimodal medical images using multiple scales. The experimental outcomes indicated that the proposed scheme can provide good outcomes over wavelet transform-based fusion methods. In [21], the authors introduced a fusion scheme by combining the complementary information of different imaging modalities like SPECT, PET, and MRI through employing the empirical wavelet transform (EWT) representation and the local energy maxima (LEM) fusion rule. The outcomes of the proposed scheme show improved visual quality performance as well as fusion performance keys. In [22], the authors introduced a multimodal medical image fusion scheme using Non-Subsampled Shearlet Transformation (NSST). The fusion is employed based on the Whale Optimization Algorithm (WOA). The experimental tests of the proposed scheme demonstrated that the resultant fused image is robust. In [23], the authors introduced a multimodal medical image fusion scheme that utilizes an optimized weighted average fusion (OWAF). The experimental tests of the proposed scheme demonstrated superior performance compared to the existing traditional fusion methods.
In [24], the authors introduced a multimodal medical image fusion based on guided filter random walks (GFRWs) and spatial frequency. The framelet transform (FT) is employed to decompose the images to be fused. After that, the GFRWs and spatial frequency (SF) are utilized to obtain the fused residual image and the fused approximate image. The experiment tests demonstrated that the suggested scheme outperforms better with respect to both subjective and objective metrics. In [25], the authors introduced a multi-modal image fusion scheme using a segmentation map of the ant colony approach. The proposed approach performs a maximum selection rule in ensemble empirical mode decomposition (EEMD) transform. After that, the proposed approach employs the pseudo-color image information to determine pixels spatial regions of the same object. The suggested approach utilizes the majority voting procedure for merging both of the fusion map and segmentation map results. Experiment outcomes indicated that that the suggested approach produces enhanced fusion outcomes. In [26], the authors introduced a hybrid optimized medical fusion scheme using shark smell optimization and world cup optimization algorithms to produce a high quality fusion of medical images. The presented scheme utilizes the wavelet transform and the homomorphic filter to improve the model efficiency. Simulations tests result demonstrated that the proposed model gives better efficiency compared to the other the studied schemes.
In [27], the authors introduced a hybrid medical fusion scheme based on combining the NSCT with DTCWT. The hybrid medical fusion scheme is examined using a set of simulation experiments on distinct image multimodalities. The experimental outcomes are investigated with a set of well-known fusion evaluation keys, and the outcomes demonstrated that the suggested scheme outperforms well with respect to subjective and objective metrics. In [28], the authors introduced a multimodal MRI-PET medical image fusion scheme based on the 2D Hartley transform (HT) in HSV color space. The medical PET color image is transformed into HSV channels. Then, the MRI and PET image V component are split into 8*8 segments and after that employ a 2D Hartley transformation for on every block of two images to be fused. After that, estimate the variance of every block for the two images and choose the best blocks. Then, employ the inverse of 2D HT and all blocks are structured into one image. Finally, the V component, H, S are multiplexed to obtain the HSV image and transform HSV to RGB to obtain the resulted final fused image. The test result demonstrated the superiority of the suggested model. In [29], the authors introduced a medical image fusion scheme that adopts a rolling guidance filter to split the source image into detail and structural components. Laplacian Pyramid is utilized to fuse the structural component and a sum-modified-laplacian (SML) fusion rule is utilized to fuse the detail component. The resulted fused-image can be produced by merging the fused structural component and fused detail component. The experimental outcomes indicated that the proposed model is superior compared to traditional medical image fusion approaches. In [30], the authors introduced a deep learning-based multimodal medical image fusion scheme. The proposed deep learning-based multimodal medical image fusion model has the advantage of enhancing the fusion result, image clarity and efficiency. The examined experimental tests ensure the efficiency of the suggested model.
Image fusion methods
The main fusion process objective between two images of the same object is to collect all the needed information in a new fused image. This acquired fused image should have an improved quality than both of them. The following subsections some traditional fusion methods, which were used to evaluate the proposed approach will be discussed, and then the proposed PCA based SVD fusion algorithm will be discussed in detail.
Discrete wavelet transform (DWT)
Image processing in the transform domains has several advantages, such as a higher Signal to Noise Ratio (SNR), reduced features ... setc. [31]. One of the common transform domains used in image fusion is the wavelet transform [32, 33]. This method includes the following steps: Both CT and MRI images are analyzed by using 2D-DWT and the four components LL, LH, HL, and HH are obtained. Each component of the two images is fused by using appropriate fusion rule (ex: LL1 and LL2). In most cases LL’s components are fused using the maximum fusion rule while the other components are fused by using the average rule [34]. Inverse DWT is applied on the obtained fused components to produce fused image.
Dual-tree complex wavelet transform fusion (DT-CWT)
DT-CWT is used as a fusion approach instead of DWT to increase the poor directionality using the complex extension of it [8]. This method includes analysis and synthesis stages. In analysis stage, four levels of analysis filter bank of the DT-CWT are applied on any signal X(n), and produces twelve directional wavelets; six of them are real trees and 6 are imaginary trees which improve the directionality [35]. The synthesis stage is the opposite of the analysis stage. In this stage both real and imaginary wavelets are composed to produce the original signal again.
The fusion process between two images using DT-CWT can be summarized as following [9, 36]: Both images are analyzed using 2D DT-CWT, this produces a real low pass image that represents the detail coefficients and other six complex high pass sub-images that represent the approximation coefficients) for each image. Approximation coefficients from both images are fused using the uniform average fusion rule. Absolute maximum fusion rule can be used for the detail coefficients. Inverse 2D DT-CWT is employed on the fused coefficients to produce the fused image.
Curvelet fusion
In this method, Curvelet fusion is used in fusing the images that contain a lot of curved shapes in their details [37, 38]. The general steps of the fusion process based on Curvelet transform on the image are: Apply the Ridgelet Transform to get the image sub-bands. Use the maximum fusion rule to fuse the sub-bands. Image Pis split into four components Δ1, Δ2, Δ3, and P3 via the additive wavelets transform as shown in Eq. (1) [38]:
Apply the inverse Curvelet transform on P3 component of the input image, and on the fused Δ1, Δ2, Δ3 to get the fused image. The tiling process is applied on Δ1, Δ2, and Δ3 sub-bands. Discrete Ridgelet Transform is applied on each obtained tile.
The PCA transforms a series of correlated data into an uncorrelated data by removing the redundant data [39, 40]. The general steps of the image fusion algorithm using PCA can be summarized as follows: Obtaining the main components (PC’s) of the two input images (to be fused) Rebuild both images once more by using only their PC’s of the largest Eigen-values. Fuse the new acquired images using appropriate fusion rule.
SVD background
SVD is a matrix factorization technique used to generate a low-range approximation of any matrix [41]. For a matrix A having a size of m×n, SVD is applied using the following equation [31]:
The matrix S is a diagonal matrix called a singular matrix where its first r-diagonal elements (s1, s2... s s r ) are the Eigen-values of the matrix A and have the property s i >0 and s1 ≥ s2 ≥ s3... s.≥s r . All other entries in this matrix are 0’s.
This section presents our fusion technique to the CT and MRI images. The proposed method depicted in Fig. 1 uses PCA and Singular value decomposition SVD to reduce both of the processing time and the needed memory during the fusion process. On the other hand, we get a fused image with better characteristics than others obtained by other traditional algorithms. This would allow us to build a hardware unit based on the implementation of this algorithm to apply the fusion process in real time. In this section, we introduce the CT and MRI images fusion by applying the fusion process on the new

The proposed PCA based SVD Fusion Method Block Diagram.
The two images are normalized to reduce the differences in the grayscale values, mostly to normalize an image to have a specified mean M0, and a specified variance V0, can be performed using this equation [42]:
SVD is used in both obtained normalized images, to have the diagonal S
matrix
for CT and MRI images. For each of CT and MRI images, we use its S
matrix
of M x M dimension to find a new matrix called S
new
matrix, which will be one dimensional matrix (1 x M). S
new
matrix is generated by taking only the diagonal values of S
matrix
as in the following equation:
where i=0, 1, 2, 3... s..m. Generate the Generate the Calculating the Eigen-values and the Eigen-vectors for S
new
matrixes; Finding the fused image by applying the proposed adaptive fusion rule in the following equation:
In our algorithm, obtaining the S
new
matrixes;
Experimental results
The presented results were obtained through using PCA, DWT, DT-CWT, Curvelet, and the proposed PCA based SVD fusion algorithms on a set of CT and MRI real images. In all instances, the results of applying the five fusion algorithms on different four cases; each is 256×256 pixel grey scale image are shown. The last mentioned eleven fusion quality metrics are used in measuring the quality of the obtained fused images. The main purpose of the proposed algorithm is to be implemented as a hardware unit, which apply the fusion process on two registered images with the same quality of the known fusion algorithms. The proposed algorithm gives the best results for each of the metrics; Average gradient, PSNR, STD, Edge intensity, UIQI, MI, Qab/f, Elapsed time and the amount of saved memory, and a lower but reasonable values for SSIM, FSIM, Entropy, and local contrast metrics. Figure 2 shows the original CT and MRI images for Case 1 and also the results of the fusion process using five different techniques; PCA, DWT, DT-CWT, Curvelet, and the proposed PCA based SVD fusion approaches. Quality measurements for Case 1 were shown in Table 1. Figure 3 shows the CT and MRI images for Case 2 and also the fusion process results using the previous five different fusion techniques. Quality measurements for Case 2 were shown in Table 2. Figure 4 shows the original CT and MRI images for Case 3 and also the results of the fusion process using the previous five different fusion techniques. The quality measurements for Case 3 were shown in Table 3. Figure 5 shows the original CT and MRI images for Case 4 and also the results of the fusion process using the previous mentioned five different fusion techniques. The quality measurements for Case 4 are shown in Table 4. The observed results shows that the proposed algorithm depends on both the input images more than the other four algorithms such as PCA, DWT, DT-CWT, Curvelet and that appears from the values of MI metrics in Tables 1–4. Also, there are more correlation and clearance offered by the proposed algorithm for the fused image, which is showed in the UIQI and STD results. The amount of transferred information from the input images to the fused image using the proposed algorithm is more than those by the other algorithms as shown in the results of Qab/f, and Edge Intensity metrics. Also, the PSNR values show a good results for the proposed algorithm compared with the other fusion methods. On the other hand, the amount of information in the fused image calculated by Entropy results is lower than the other four algorithms, and the same for the Local contrast of the image. Also, the results of the similarity, either regions similarity expressed in terms of SSIM, or edges similarity expressed in terms of FSIM are lower than the obtained by other fusion methods. As showed in Tables 1–4, the proposed PCA based SVD fusion approach has a lower execution time compared to the other fusion algorithms, and the amount of reduced time is increased when the input images size increased. We can calculate the fusion time versus image sizes for the previous algorithms such as PCA, DWT, DT-DWT, Curvelet, and the proposed scheme. One of the previous cases was chosen randomly, and had been resized to 32×32 pixel size, 64×64 pixel size, 128×128 pixel size, and 256×256 pixel size. The fusion process time of each fusion algorithm is recorded as shown in Table 5. The conclusion of the results shows that the proposed PCA based SVD fusion approach saves more time compared to the other fusion algorithms, and the saved time is increased when the image size is increased. This is mainly because of using S new matrix, which has 1×M dimension for an image of size N×M instead of using the whole image in the fusion process during the algorithm execution. Finally, to discuss the memory requirements for the proposed algorithm, it was found that for a grey scale image of size 16×16 pixel, each pixel of the image will be stored in 1-byte of the memory, and so the needed memory to store the whole image is 256 byte. To deal with different fusion approaches such as PCA, DWT, DT-CWT, Curvelet, the processing is applied on the both CT and MRI images at the same time. So, for fusion of two grey scale images of size 16 x 16, the fusion process needs 512 byte of the memory. For the proposed PCA based SVD fusion approach, the processing is applied on the S new matrix which is only 1×M dimension matrix for M×N image instead of the original image. So, in this case it needs 1 x 16 x 1 byte=16 byte for each image and we will need only 32 bytes of the total memory, which saves 480 byte compared to the other four algorithms. The last calculations are applied on different image sizes and showed in Table 6. As shown in Table 6, the proposed algorithm uses a small amount of memory compared to the other four algorithms. The amount of saved memory by the proposed algorithm is increased when the size of the images we are going to fuse increased, and that is shown clearly in the last column of Table 6. This large amount of saved memory enables us to implement the proposed algorithm as a real time hardware unit.

Fusion results for the original images (CT and MRI) of size 256×256 pixel with 5 different fusion methods for Case 1.
Quantitative results for the original fused CT and MRI images of size 256×256 pixel with 5 different fusion methods for Case 1

Fusion results for the original images (CT and MRI) of size 256×256 pixel with 5 different fusion methods for Case 2.
Quantitative results for the original fused CT and MRI images of size 256×256 pixel with 5 different fusion methods for Case 2

Fusion results for the original images (CT and MRI) of size 256×256 pixel with 5 different fusion methods for Case 3.
Quantitative results for the original fused CT and MRI images of size 256×256 pixel with 5 different fusion methods for Case 3

Fusion results for the original images (CT and MRI) of size 256×256 pixel with 5 different fusion methods for Case 4.
Quantitative results for the original fused CT and MRI images of size 256×256 pixel with 5 different fusion methods for Case 1
Execution time of the fusion process for PCA, DWT, DT-CWT, Curvelet, and Proposed techniques applied on different image sizes
Memory requirements needed for each fusion approach in bytes
In this study, we presented an efficient CT/MRI images fusion approach based on the PCA and SVD techniques. Eleven quality metrics were used in estimating the fused image resulted by the proposed approach, and compare it with the images acquired by the other fusion approaches. The experimental results of the proposed technique are superior compared with other traditional techniques such as PCA, DWT, DT-CWT, and Curvelet fusion methods. The algorithm has been extensively tested and applied to 4 sets of different data in the form of CT and MRI images. The experimental results show that the proposed PCA based SVD fusion approach has better results for: (1) Detailed contrast and texture variation, which appeared in the results of average gradient measure. (2) SNR in the obtained fused image that appeared in the results of PSNR_CT and PSNR_MRI measures. (3) Clearness of the obtained fused image, which appeared in the results of standard deviation measure. (4) Edges clearness of the obtained fused image, which appeared in the edge intensity measure results. (5) Correlation value between the obtained fused image and input images, which appeared in the results of UIQI. (6) The results of Qab/f, and Edge Intensity metrics reveals that, the information transferred from CT and MRI images to the obtained fused one is increased. (7) The elapsed time of the fusion process is reduced compared to the other algorithms, and the amount of reduction is directly proportional to the size of the images to be fused, which makes the proposed PCA based SVD fusion algorithm very fast. (8) Finally, the memory requirements needed by the proposed algorithm had been reduced compared with the other fusion algorithms.
Footnotes
Acknowledgment
This study was funded by the Deanship of Scientific Research, Taif University Researchers Supporting Project number (TURSP-2020/08), Taif University, Taif, Saudi Arabia.
