Infrared and visible image fusion using dual-tree complex wavelet transform and convolutional sparse representation

Abstract

Infrared and visible image fusion refers to the technology that merges the visual details of visible images and thermal feature information of infrared images; it has been extensively adopted in numerous image processing fields. In this study, a dual-tree complex wavelet transform (DTCWT) and convolutional sparse representation (CSR)-based image fusion method was proposed. In the proposed method, the infrared images and visible images were first decomposed by dual-tree complex wavelet transform to characterize their high-frequency bands and low-frequency band. Subsequently, the high-frequency bands were enhanced by guided filtering (GF), while the low-frequency band was merged through convolutional sparse representation and choose-max strategy. Lastly, the fused images were reconstructed by inverse DTCWT. In the experiment, the objective and subjective comparisons with other typical methods proved the advantage of the proposed method. To be specific, the results achieved using the proposed method were more consistent with the human vision system and contained more texture detail information.

Keywords

image fusion dual-tree complex wavelet transform convolutional sparse representation guided filter

1 Introduction

Nowadays, infrared and visible image fusion acts as one of the critical factors in image processing [1]. It has been extensively adopted in numerous applications (e.g., surveillance, modern military and object detection) [2]. The source images applied for image fusion originate from different sensors. The infrared sensors can capture thermal radiation. They are less susceptible to light and exhibit high performance at night, whereas it will lose textural details. The visible sensors are capable of capturing reflected light and normally obtaining sufficient texture information and high-resolution images [3, 4]. However, in certain scenarios (e.g., low-brightness, heavy fog, and smog), the visible imaging sensor rarely captures effective information. Infrared and visible image fusion aims to combine the complementary information from different sensors. Accordingly, the fused images combine the vital feature information and texture details simultaneously.

As fueled by the advancement of image processing and analysis theory technology, considerable image processing methods have been developed over the past decades. These methods are commonly classified into spatial domain methods and transform domain methods [5, 6]. The spatial domain methods directly operate pixels of source images by following different strategies, which are simple and time-efficient, whereas these methods may loss some information for considering less about the differences of infrared and visible images [7]. The transform domain methods are the most commonly used methods to address the image fusion issues and achieve effective fused results, which is to transform source images of spatial domain into other domain via several specific transformations. The transform domain methods are split into several categories [8], namely, multi-scale transform (MST) [9 –11],sparse representation (SR) [12 –19], etc. MST are usually employed in image fusion, which cover pyramid contourlet based methods (e.g., ratio of low-frequency pyramid (RP) [20] and Laplacian pyramid (LP) [21, 22]), curvelet transform (CVT) [23], contourlet transform based methods (e.g., nonsubsampled contourlet transform (NSCT) [24, 25]), wavelet transform based methods (e.g., discrete wavelet transform (DWT) [26], and dual-tree complex wavelet transform (DTCWT) [27], etc. There exist some variants of sparse representation-based methods, covering convolutional form (CSR) [28] and adaptive form (ASR) [15], etc. However, they usually only apply “averaging rule” and “variance rule” to fuse base layers or detail layers, so the information of source images cannot be transformed into the fused images, thereby decreasing the contrast of fused images.

Over recent years, many image fusion methods have been proposed based on above technologies, nevertheless such methods may lose details information or consume too much time. To remedy these defects, this study proposed a hybrid method based on DTCWT and CSR. Two things are noteworthy about the principle of fusion [12]: (1) representing the image/signal effectively and appropriately; (2) combining the complementary information originating from source images captured by multi-sensors into a joint fused image. In our fusion framework proposed here, the DTCWT was first adopted to decompose the source images into high-frequency component and low-frequency components. Subsequently, the convolutional sparse representation (CSR) was employed to represent the low-frequency component for better detail-preserving, and the low-frequency component was fused following choose-max rule. Next, the high-frequency bands were optimized by guided filtering (GF), consequently the detailed information of high-frequency component was preserved better than other methods. Finally, the fused image was reconstructed via the inverse DTCWT. The whole process was split into three steps, namely, decomposition, fusion, and reconstruction. The main contributions of this work are presented below:

1) A hybrid method was proposed for infrared and visible image fusion, capable of capturing high-quality fused images. The experimental results demonstrated that this method outperforms other typical approaches in visual perception and objective measurement.

2) The proposed method integrates DTCWT, CSR and GF and remedies some defects of other methods. It also exhibits good effect in detail information preservation and efficient in computation, enhancing the performance of the method.

3) Effective strategies were presented for fusing low-frequency band and high-frequency bands. For the characteristic of CSR and GF, the generated images can tackle down two problems, namely, image representation and complementary information preservation.

The rest of this paper is structured as follows. In section 2, we introduce the related background and techniques briefly. In section 3, the scheme of image fusion is proposed and explained in detail. In section 4, we conduct comparative experiments, and the results show the superiority of our approach. Finally, section 5 summarizes this whole paper.

2 Related works

In this section, we introduce the background of our proposed method and the utilized technologies. Besides, we present the recent methods and their characteristics. To overcome drawbacks of these methods, three important technologies are applied.

2.1 Background and literature review

In recent years, the transform domain based algorithms are pretty popular in image fusion processing. For example, MST and SR based methods have been extensively adopted. Nencini et al. utilized curvelet transform (CVT) [29] to fuse remote sensing images; such method can generate the fused images exhibiting abundant details and edge information. However, CVT conducts down-sampling and up-sampling, leading to distortion and spectral aliasing in the final fused image. Ma et al. proposed an infrared and visible image fusion method based on gradient transfer and total variation minimization (GTF) [3, 30], capable of enhancing the applicability of high-precision registration and generating the fused images exhibiting abundant texture details. However, this method only significantly impacts images exhibiting high brightness. Recently, the fusion algorithms based on SR have also had wide applications in image processing; they exploit sliding window technology to reduce block artifacts. However, they are limited to save texture detail information. To solve this drawback, CSR is derived from SR. The result of CSR method is single-valued and optimized over the entire image rather than a local image patch; it can then optimize the entire image and preserve more details than SR. However, CSR is relatively time-consuming during the whole image processing.

For overcoming the above disadvantages, our method applies DTCWT, GF, and CSR theory. To acquire more texture detail information, CSR was adopted to represent the low-frequency bands of images. Meantime, since the result of CSR is single-value, the low-frequency component could be optimized by it. To enhance computational efficiency, the appropriate MST method was employed. The dual-tree wavelet transform complied with individual filter bank [8], thereby making the image processing more efficient. Moreover, it exhibits directional selectivity and preserves more texture details. Combined with DTCWT and CSR, the proposed method here can considerably reduce the impact of low computational efficiency caused by CSR in low-frequency band. The specific principles of these three technologies are described in detail as follows.

2.2 Dual-tree complex wavelet transform

Compared with DWT, DTCWT has excellent directional selectivity and high-efficiency [9, 31]. At the same time, the DTCWT is an over-complete wavelet transform that is consistent with the human visual system [27]. The DTCWT is an enhancement of the traditional wavelet transform. Thus, it has been widely used in the field of image processing.

DTCWT adopts two separated real DWTs with a 90° phase difference. The real part is provided by one DWT, and the imaginary part of the transform is presented by another DWT. Each tree acts on rows and columns to require dual-tree structure. No matter what the decomposition level of 2-D DTCWT is, the overall redundancy is 4:1. For each level of decomposition, it produces two low-pass sub-bands and six high-pass sub-bands. Fig. 1 shows the impulse responses of the 2-D DTCWT filters for six different directional high-pass sub-bands (-75°, -45°, -15°, 15°, 45°, 75°). Consequently, DTCWT has three more directions than DWT, which can improve the image quality, the accuracy of decomposition and reconstruction, and the ability of retaining image details. More details refer to the comprehensive study published by I.W. Selesnick [32].

Fig. 1

The impulse responses for 2-D DTCWT transform. (a)the six dictionary sub-bands of the real part: -75°, -45°, -15°, 15°, 45°, 75°. (b)The six dictionary sub-bands of the real part: -75°, -45°, -15°, 15°, 45°, 75°.

2.3 Guided image filtering

As an enhancement technology, guided image filtering (GF) has been proposed to implement edge-preserving [33]. The guided filter is defined as a local linear model [34] as shown in (1-4). $O_{i} = a_{k} I_{i} + b_{k} \forall i \in w_{k},$ (1) where I denotes the guidance image, O denotes the filtering output, w_k denotes the local window centered at pixel k, and |w| is the number of pixels in w_k. The linear coefficients a_k and b_k are calculated by the linear ridge regression model (3-4) [33]. $E (a_{k}, b_{k}) = \sum_{i \in w_{k}} ({(a_{k} I_{i} + b_{k} - P_{i})}^{2} + ε a_{k}^{2}),$ (2)

The coefficients a_k and b_k are indicated as follows: $a_{k} = \frac{\frac{1}{| w |} \sum_{i \in w_{k}} I_{i} P_{i} - μ_{k} {\bar{P}}_{k}}{δ_{k} + ε},$ (3) $b_{k} = {\bar{P}}_{k} - a_{k} μ_{k} .$ (4) where P_i indicates the input image, and ${\bar{P}}_{k}$ is the mean of P_i in the square window w_k. μ _k and δ _k indicate the mean and variance of guidance image in w_k, ε means a regularization parameter. In this paper, r and ε mean the filter size and blur degree during the guided filtering operation. As a fast linear model, the guided filter has the advantage of the structural transfer and edge-preserving smoothing. Therefore, it is widely applicable in image processing applications, including infrared and visible image fusion.

2.4 Convolutional sparse representation

SR has been applied in image processing tasks, and it addresses the sparsity of the training signals. The principle of it is based on constituting a sparse vector with only a few nonzero elements to represent the signals by the dictionary, and the dictionary can be learned by the K-SVD algorithm [35]. For a set of training signals $Y = {y_{1}, \dots, y_{M}} \in ℝ^{n \times M}$ , each signal can be represented as the sparse linear combination of the special dictionary $D \in ℝ^{n \times k}$ and coefficient vector $α_{i} \in ℝ^{N_{i}}$ [36, 37]: $min_{{α_{i}}} {∥ y_{i} - D α_{i} ∥}_{2}^{2} s . t . {∥ α_{i} ∥}_{0} \leq t_{,} 1 \leq i \leq M$ (5) where $y_{i} \in ℝ^{n}$ is the signal, $D \in ℝ^{n \times k}$ is the over-complete dictionary with k atoms, $α_{i} \in ℝ^{M}$ is the sparse linear coefficient, and t means the number of nonzero elements.

Sparse representation (SR) performs on a local image patch. And it can not optimize an entire image. In addition, SR is limited in local patch manner and detail preservation.

Consequently, convolutional sparse representation (CSR) is considered as an alternative representation structure. The image S is modeled as the sum over a series of convolutions of coefficient maps X_m and relevant dictionary filters d_m. And the sparse coefficient maps X_m can be obtained by following CSR model [28, 38]: $\begin{matrix} \underset{{X_{m}}}{arg min} \frac{1}{2} {∥ \sum_{m} d_{m} * X_{m} - S ∥}_{2}^{2} + λ \sum_{m} {∥ X_{m} ∥}_{1} \end{matrix}$ (6) where * denotes the convolutional operator, S denotes the input signal, λ is the regularization parameter of this model, d_m is a set of M dictionary filters, and the dictionary filter has been learned from the dictionary learning algorithms [28].

In this paper, CSR is regarded as an efficient form that can solve the existing problem of SR and optimize the whole image due to the result of single-value. Meanwhile, the principle of convolutional form is based on the entire image so that CSR can preserve more texture details during image fusion.

3 Proposed fusion scheme

In this section, we present the structure and rule of the proposed image fusion method in detail. For the low-frequency bands and high-frequency bands, we apply CSR and GF strategy respectively. The schematic diagram of our method is summarized in Fig. 2. The essential steps and fusion framework are introduced in the following subsections.

Fig. 2

The schematic diagram of our model.

3.1 Multi-scale image decomposition

The DTCWT decomposition works on each input image respectively to acquire their wavelet subimages [31]. In this paper, input images S₁ and S₂ are decomposed into low-frequency bands S_{1 _ low} and S_{2 _ low}, high-frequency bands S_{1 _ high} and S_{2 _ high}. The variation range of decomposition level is from 1 to 4, and the principle of decomposition is presented in section 2.

3.2 Fusion of high-frequency bands

The high-frequency bands S_{1 _ high} and S_{2 _ high} are processed by guided image filtering to obtain S_{1g _ high} and S_{2g _ high}, respectively. The fused high-frequency bands S_{1 _ high} is taken into equation (1) as the input image P_i to obtain S_{1g _ high}, and S_{2g _ high} also can be obtained in the same way. The detailed calculation process is shown in Section 2.2. This enhancement technology implements the edge-preserving of the fused image. In this paper, numerous experiments demonstrate that when the values of ε (blur degree of the guided filtering) and r (filter size) respectively are 10^-7 and 3, the best performance can be achieved. Then we adopt the “choose-max” strategy to get the fused high-frequency bands S_high.

3.3 Fusion of low-frequency bands

The low-frequency bands S_{1 _ low} and S_{2 _ low} are represented by CSR separately. The sparse coefficient maps C_k,m, which can be obtained by calculating the following CSR model. $\underset{{C_{k, m}}}{arg min} \frac{1}{2} {∥ \sum_{m}^{M} d_{m} * C_{k, m} - I_{k}^{d} ∥}_{2}^{2} + λ \sum_{m = 1}^{M} {∥ C_{k, m} ∥}_{1}$ (7) where the sparse coefficient maps C_k,m are N-dimensional vectors [28], N is the number of image pixels, and $I_{k}^{d}$ is the low-frequency band of the source image. The l₁-norm of C_k,m is considered to be the activity level measure of source images according to the SR method [14], hence, the activity level map is indicated as follows: $A_{k} (x, y) = {∥ C_{k, 1 : M} (x, y) ∥}_{1},$ (8)

Besides, the window-based averaging strategy for A_k (x, y) makes this method save more structure information so that we can obtain the optimized activity level map: ${\bar{A}}_{k} (x, y) = \frac{\sum_{p = - r^{'}}^{r^{'}} \sum_{q = - r^{'}}^{r^{'}} A_{k} (x + p, y + q)}{(2 r^{'} + 1)^{2}},$ (9) where r’ denotes the size of the window, and variables {p, q} are in (-r, r).

When r’ is too small, it may be incapable of representing characteristics of the image. However, when r’ is too large, it will lead to a great increase in complexity. Therefore, it is significant to employ a suitable r’ for image fusion. In this paper, through the verification of numerous experiments, this method works best when r’ equals 3. Then, the fused coefficient maps are required by implementing the choose-max strategy as below: $\begin{matrix} C_{f, 1 : M} (x, y) = C_{k^{*}, 1 : M} (x, y), \\ k^{*} = arg max_{k} ({\bar{A}}_{k} (x, y)) \end{matrix}$ (10) where C_f,1:M (x, y) means the fused sparse coefficient map.

The final fusion result of low-frequency band S_low is reconstructed by learning dictionary and convolutional sparse vectors: $S_{low} = \sum_{m = 1}^{M} d_{m} * C_{f, m} .$ (11)

3.4 Image reconstruction

In this paper, we comply with the corresponding inverse DTCWT to absolutely reconstruct the fused image. And the inverse DTCWT process can be easily implemented because it is the opposite of DTCWT decomposition. The experimental results and metrics of fused images are shown in the next section.

4 Experiments

In order to verify the superiority of our method, we conduct internal comparison experiments and external comparison experiments. We choose the optimal parameters via the comparison of fused results under different parameters. And the external experiments verify that our results are better than the other eight methods.

4.1 Experimental settings

We pick on multiple pairs of infrared and visible images to certificate the efficacy and benefit of the proposed method as shown in Fig. 3. All source images are from the public datasets: https://figshare.com/articles/TN_Image_Fusion_Dataset/1008029 and http://www.imagefusion.org/. Four metrics are adopted to evaluate the performance of image fusion. These experiments are conducted in MATLAB on a computer with a 2.5 GHz CPU and 8 GB RAM. The dictionary used in CSR is learned from natural image patches by the K-SVD method [39]. At the beginning of the experiment, we set N (the decomposition level) =4, ε (the blur degree of the guided filtering) =10^-6, r (the guided filter size) =7 as the initial values. However, they are not the optimal ones, we conduct the internal comparative experiments to select the most appropriate values later. We study the effect of decomposition level in terms of image fusion efficiency and four metrics.

Fig. 3

Four pairs of source images: The first row shows the infrared image, and the second row shows the visible image.(“Tank” a-infrared, b-visible; “Leaf” c-infrared, d-visible; “People” e-infrared, f-visible; “Street” g-infrared, h-visible)

In the external experiments, our method is compared with eight representative image fusion methods, namely, ratio of low-pass pyramid (RP) [20], dual-tree complex wavelet transform (DTCWT) [27], curvelet transform (CVT) [23], guided filtering fusion(GF) [33], adaptive sparse representation (ASR) [15], gradient transfer fusion (GTF) [3], convolutional sparse representation (CSR) [28], and DTCWT-CSR.

For all comparison methods, all the parameters are set to the default values by reference in the original paper. Besides, for DTCWT-SR, the size of the dictionary used in SR is united with CSR in our paper. RP is the most classic and early method. ASR and CSR are based on sparse methods. DTCWT is a pretty popular MST method. GTF and GF are recent and excellent methods. DTCWT-SR method can outperform plenty of typical fusion methods in recent surveys. Thus, it is necessary to compare it with the proposed method on the source images to verify the superiority of our approach. We carry out the subjective and objective comparisons between the eight comparative methods and our method.

4.2 Objective image fusion metrics

To evaluate the fusion property of the different methods objectively, many assessment methods are proposed in recent years. None of these metrics can appraise the quality of fused images accurately, so it is essential to employ several metrics simultaneously [40]. In this paper, four metrics are adopted, namely, entropy (EN) [19], mutual information (MI) [41], standard deviation (SD) [19], and visual information fidelity (VIF) [42]. The value of each metric is larger, the image fusion effect is better.

1. Entropy (EN). The EN is the average amount of information contained in the generative image. The larger the value is, the greater the amount of information it contains. It is calculated as follows: $EN = - \sum_{i = 0}^{L - 1} p_{i} {log}_{2} p_{i},$ (12)

where i denotes the gray level of an image (in this paper, i=256), and p_i means the distribution of gray scale.

2. Mutual information (MI). The quality metric MI is employed to measure the similarity between the fused image and the input image in the distribution of the gray scale. Thereby, it measures the degree of preservation that the information originates from input images in the generated image. It is defined as follows: $I_{XY} (x, y) = \sum_{x, y} p_{XY} (x, y) {log}_{2} \frac{p_{XY} (x, y)}{p_{X} (x) p_{Y} (y)},$ (13) $MI = 2 [\frac{I_{FX} (f, x)}{H (F) + H (X)} + \frac{I_{FY} (f, y)}{H (F) + H (y)}]$ (14) where H(A)=I(A, A), and more details are accessible in a paper written by M. Hossny [43].

3. Standard deviation (SD).The SD reflects the degree of dispersion of image grayscale relative to average grayscale. $SD = \sqrt{\frac{\sum_{i = 0}^{M - 1} \sum_{j = 0}^{N - 1} [F (i, j) - μ]^{2}}{M \times N}},$ (15) where M*N denotes the size of source images, F(i,j) means the grayscale value of a pixel in the image, and μ _k means the average value of the fused image.

4. Visual information fidelity (VIF). The VIF shows the performance for image quality prediction, which is in accordance with the principle of the human vision system. It is calculated as follows: ${VIFF}_{k} (A, B, F) = \frac{\sum_{b} FVI D_{k, b} (A, B, F)}{\sum_{b} FVN D_{k, b} (A, B, F)},$ (16)

$VIFF (A, B, F) = \sum_{k = 1}^{N} p_{k} \cdot {VIFF}_{k} (A, B, F),$ (17) where the algorithm of fusion visual information with distortion (FVID) and fusion visual information without distortion (FVIND) are introduced in the paper written by Yu Han [42].

4.3 Investigation of optimal parameters

In this subsection, we study the parameters (level N, radius r, and regularization ε) in our method. To investigate the effect of the decomposition level for image fusion, we conduct a comparative experiment. Fig. 3 shows four sets of source infrared images and visible images. The level of DTCWT decomposition is set from 1 to 4, respectively. And the fused images of different levels are shown in Fig. 4. There is little difference with different levels in subjective visual perception. Table 1 lists the objective assessments with different decomposition levels for image fusion processing. Under different decomposition levels, the maximum value of each objective metric (SD, EN, MI, VIF) of the four pictures is indicated in bold. As we can see that N is 1 or 2, the metrics tend to be larger. Table 2 shows the computational efficiency of different levels. When we increase the value of N, the running time is significantly reduced. From the result, we can know that larger decomposition layer leads to higher computational efficiency. However, N has a little impact on these four metrics of the fused image. Considering both the computational cost and quality metrics, 4 is the most appropriate value of N for DTCWT decomposition in this work.

Fig. 4

The fused images under different decomposition levels (a-“Tank”, b-“Leaf”, c-“People”, d-“Street”). From top to bottom, namely N=1, N=2, N=3, N=4.

Table 1

Quantitative assessment under different levels

Image	Metrics	Tank	Leaf	People	Street
N=1	EN	7.9212	6.8664	6.8379	6.7817
	MI	4.1283	5.2313	4.6788	3.1740
	SD	70.0886	43.7204	34.6548	36.5133
	VIF	0.1080	0.8926	0.6677	0.5635
N=2	EN	7.9318	6.8908	6.8297	6.8006
	MI	3.8227	4.9917	4.3609	3.0239
	SD	72.4936	43.8453	34.8436	36.9578
	VIF	0.1390	0.9055	0.6684	0.5819
N=3	EN	7.9117	6.8777	6.8195	6.8165
	MI	3.7275	4.7802	3.6856	2.8428
	SD	74.4452	43.4978	35.3480	36.8896
	VIF	0.1565	0.8903	0.7021	0.5656
N=4	EN	7.8827	6.8840	6.7824	6.7893
	MI	3.5134	4.6421	2.8114	2.8019
	SD	75.6742	43.3169	36.0041	36.1319
	VIF	0.1732	0.8838	0.7203	0.5435

Table 2

The running time (s) under different levels

Level	Tank	Leaf	People	Street
N=1	351.61	66.23	369.64	119.30
N=2	114.57	23.29	114.22	39.44
N=3	25.06	3.93	33.12	5.78
N=4	4.08	1.52	4.98	1.82

The parameter r and ε are the key value that influence the image fusion result. To investigate the effect of r and ε, we conduct another set of comparative experiments. When r changes from 1 to 7, the values of metrics are shown in Fig. 5. As we can see, the values of EN and MI are lager when r is 2, whereas the values of SD and VIF are larger when r is 3. However, when r is 2, the value of SD is much lower. Therefore, the most suitable value of r is set to 3. Fig. 6 shows the change of objective metrics with different ε, and it is set from 10^-1 to 10^-10. As we can see, with the increase of ε, the curve is on the rise and eventually tends to stabilize. Thus, The value ε is set 10^-7 in our work.

Fig. 5

The change of objective metrics with different r values.

Fig. 6

The change of objective metrics with different ε values.

4.4 Comparisons with other representative fusion methods

According to the above internal experiments, the parameters in our method are empirically set to N (the decomposition level) =4, ε (the blur degree of the guided filtering) =10^-7, and r (the filter size) =3. Fig. 3 shows the infrared images and visible images in four scenes, namely, “Tank”, “Leaf”, “People”, “Street”. We choose two representative scenes for comparison in detail. In Fig. 7-8, results of fusion are obtained to assess the subjective evaluation of fusion performance by eight compared methods (RP, DTCWT, CVT, GF, ASR, GTF, CSR, DTCWT-SR) and the proposed method. And all the parameters used in other comparative methods are set to the optimal values. For apparent comparisons, the green box region is selected to enlarge twice and put in the white box. Table 3-4 show the objective metrics of different methods. Four metrics are employed to quantitatively evaluate the fused images, namely, EN, MI, SD, and VIF, and the optimal values of objective metrics are indicated in bold. It shows that the proposed method almost has larger values of the objective evaluation compared with the other eight methods. The value of EN means that the fused images contain the amount of information, thereupon, the generated images based on our method get a greater amount of information. Due to larger value of MI, our method has a greater degree of original information preservation from source images. Besides, larger value of VIF indicates that the fused images merged by our method provide better visual effect and are more consistent with the human vision system. At the same time, the computational efficiency of different methods is compared. Table 5 shows the computational efficiency of fusing four sets of images.

Fig. 7

Source images and fused images of “People” by different methods.

Fig. 8

Source images and fused images of “Street” by different methods.

Table 3

Quantitative assessment of “People” with different image fusion methods

Metrics	RP	DTCWT	CVT	GF	ASR	GTF	CSR	DTCWT-SR	OUR
EN	6.1172	6.0379	6.0384	6.5342	6.0387	6.0495	6.0540	6.5342	6.7825
MI	1.1559	1.5181	1.4808	2.6693	1.9135	1.4228	2.0252	2.6693	2.8112
SD	28.3444	22.4200	22.1970	28.2595	21.6215	22.5638	21.1530	28.2595	36.0043
VIF	0.2527	0.4587	0.4485	0.4827	0.4188	0.1725	0.4082	0.4827	0.7203

Table 4

Quantitative assessment of “Street” with different image fusion methods

Metrics	RP	DTCWT	CVT	GF	ASR	GTF	CSR	DTCWT-SR	OUR
EN	6.4195	6.4252	6.4344	5.9772	6.1922	6.3221	6.1370	5.9772	6.7892
MI	1.3685	1.5694	1.4571	1.4385	1.9807	1.7821	2.0911	1.4385	2.8017
SD	32.5046	29.6677	29.3833	30.4521	26.3146	24.3411	25.2080	30.4521	36.1318
VIF	0.2425	0.4158	0.4158	0.2470	0.3978	0.2720	0.3985	0.2470	0.5436

Table 5

The running time (s) of different methods

Method	Tank	Leaf	People	Street
RP	0.04	0.10	0.04	0.04
DTCWT	0.11	0.31	0.14	0.11
CVT	0.52	1.40	0.74	0.60
GF	0.31	0.43	0.58	0.68
ASR	98.97	327.38	139.90	119.26
GTF	0.92	9.24	1.33	1.04
CSR	39.81	191.79	46.22	59.90
DTCWT-SR	0.33	0.13	0.48	0.17
OUR	4.08	1.52	4.98	1.82

Fig. 7(a)-(b) shows the infrared and visible image and Fig. 7(c)-(k) shows fused images obtained by different algorithms. In the area of junction marked by the green box, the edges of the rectangle are very clear and the details are well preserved in our fusion image. For the area of fist marked by the green box in Fig. 7(c)-(k), the RP based image fusion method is very blurry in the entire area and produces some noise. The DTCWT, CVT, GF, ASR, GTF, CSR based fusion methods produce some serious edge artifacts. Our method is more clear in the area of the finger and the edge of “fist’’, and it is easier to distinguish the fist from the background. Overall, the image obtained by our fusion method contains complementary information of visible images and infrared images, and the details of the characters are well preserved. Table 3 shows the objective metrics of “People” with different image fusion approaches. All metrics of our method are larger, especially the value of SD is much larger than the other methods. Consequently, the fused image contains richer information.

The fused images of “Street” with RP, DTCWT, CVT, GF, ASR, GTF, CSR, DTCWT-SR, and our method are shown in Fig. 8(c)-(k). The effort of RP is not ideal that some noise is introduced. The fused images based on the GTF image fusion method almost lose brightness and details. It can be seen that the fused images based on our method are closer to visible images in some parts and include more information on visible images, which is coordinate with the human visual system. Also, the generated images by our method can preserve edge information better. For example, the edge details in the two traffic lights of our method are better preserved in “Street”. In the area of a pair of pedestrians marked in the green box, it can be seen that the method we proposed preserves the edges and details of pedestrians better. Besides, the lines on the road are also clearer. Table 4 shows the objective metrics of “Street” with different image fusion approaches. All evaluation indexes with our method are much larger than other methods. These objective metrics fully demonstrate that our proposed method is better than other methods.

Experimental results show that our method is superior to the other eight methods in subjective and objective evaluation of scenes “Tank” and “leaf”. The details and edge information of fused images are well preserved. For the sake of space, the fusion results are not displayed one by one.

Table 5 shows that ASR has the lowest efficiency, and the RP has the highest computational efficiency. Even if the proposed method adopts CSR to represent the signal, the running time of the proposed method costs relatively less time than CSR due to the high computational efficiency of DTCWT. At the expense of reducing a little efficiency, the objective assessment and subjective assessment of our method are superior to others.

GF can enhance the edge information of images to make images clearer. Table 3-4 show the value of objective evaluation, and the maximum values are indicated in bold. The higher values of EN and SD mean the richer information contained by fused images. The value of MI evaluates the amount of information which transfers from the source image to the fused image. According to the experimental result, we can see that our proposed method performs well on details-preserving. Furthermore, the VIF plays a vital role in the human perception of images. However, other methods contain many ringing artifacts around the saliency features, and the details information is not distinct enough in some fused images. In contrast, the proposed method generates all four merge images perfectly, as more details are preserved in fused images, and the generated images are consistent with the human visual system. Therefore, better objective and subjective fusion performance can be achieved.

5 Conclusion

In this paper, an effective hybrid method for infrared and visible image fusion is presented and evaluated. It is vital for image fusion to choose a suitable tool for decomposition and an appropriate strategy of fusion. The basic kernel steps are as follows. Firstly, The source images are decomposed into their low-frequency band and high-frequency bands by DTCWT. The former contains structure information, while the latter contains more texture details. Secondly, for low-frequency bands, CSR is adopted to represent the structure of source images, and then the low-frequency bands of infrared and visible images are fused via the “choose-max” rule. For high-frequency bands, we employ guided filtering to enhance high-frequency bands information, which can preserve more texture details. And we fuse the high-frequency bands of source images via the median filter. Finally, inverse DTCWT was used to reconstruct the fused image.

The proposed method focuses on combining infrared images and visible images into fused images to obtain more efficient details. We evaluate the proposed method qualitatively and quantitatively, and the property of our proposed method is better than other compared methods. Above all, we introduce the hybrid structure containing DTCWT and CSR, and utilize GF to enhance images simultaneously. The experimental results show that the proposed method can require almost the best-performed results.

Footnotes

Acknowledgments

The authors would sincerely like to thank the editor and anonymous reviewers for their detailed review. The authors also sincerely thank Daguang Shao and professor Xiaomin Yang for their meaningful suggestions. The authors also thank Yu Liu and his research team for providing some codes on the public platform.

References

, Wu

and Kittler

, Densefuse: A fusion approach to infrared and visible images, IEEE Transactions on Image Processing 28(5) (2018), 2614–2623.

, Kang

, Fang

, et al., Pixel-level image fusion: A survey of the state of the art, Information Fusion 33 (2017), 100–112.

, Chen

, Li

, et al., Infrared and visible image fusion via gradient transfer and total variation minimization, Information Fusion 31 (2016), 100–109.

Zhu

, Yin

, et al., A novel multi-modality image fusion method based on image decomposition and sparse representation, Information Sciences 432 (2018), 516–529.

Stathaki

, Image Fusion: Algorithms and Applications, Elsevier Science (2011), 119–138.

Jian

, Yang

, Zhou

, et al., Multi-scale image fusion through rolling guidance filter, Information Fusion 83 (2018), 310–325.

Anantrasirichai

, Zheng

, et al., Image fusion via sparse regularization with non-convex penalties, Pattern Recognition Letters 131 (2020), 355–360.

, Ma

and Li

, Infrared and visible image fusion methods and applications: A survey, Information Fusion 45 (2019), 153–178.

, Yang

and Hu

, Performance comparison of different multi-resolution transforms for image fusion, Information Fusion 12(2) (2011), 74–84.

10.

Pajares

and Cruz

J.L.

, A wavelet-based image fusion tutorial, Pattern Recognition 37(9) (2004), 1855–1872.

11.

Bavirisetti

D.P.

and Dhuli

, Two-scale image fusion of visible and infrared images using saliency detection, Infrared Physics & Technology 76 (2016), 52–64.

12.

Liu

, Qi

and Ding

, Infrared and visible image fusion method based on saliency detection in sparse domain, Infrared Physics & Technology 83 (2017), 94–102.

13.

, Yin

and Fang

, Group-sparse representation with dictionary learning for medical image denoising and fusion, IEEE Transactions on Biomedical Engineering 59 (2012), 3450–3459.

14.

Nejati

, Samavi

and Shirani

, Multi-focus image fusion using dictionary-based sparse representation, Information Fusion 25 (2015), 72–84.

15.

Liu

and Wang

, Simultaneous image fusion and denoising with adaptive sparse representation, IET Image Processing 9 (2015), 347–357.

16.

Zong

and Qiu

, Medical image fusion based on sparse representation of classified image patches, Biomedical Signal Processing and Control 34 (2017), 195–205.

17.

Zhang

and Levine

M.D.

, Robust multi-focus image fusion using multi-task sparse representation and spatial context, IEEE Transactions on Image Processing 25 (2016), 2045–2058.

18.

Gao

, Vorobyov

S.A.

and Zhao

, Multi-focus image fusion via coupled dictionary training, in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2016), 1666–1670.

19.

Liu

, Liu

and Wang

, A general framework for image fusion based on multi-scale transform and sparse representation, Information Fusion 24 (2015), 147–164.

20.

Toet

, Image fusion by a ratio of low-pass pyramid, Pattern Recognition Letters 9 (1989), 245–253.

21.

Burt

P.J.

and Adelson

E.H.

, The laplacian pyramid as a compact image code, IEEE Transactions on Communications 31 (1983), 532–540.

22.

Liu

, Chen

, et al., Medical image fusion method by using Laplacian pyramid and convolutional sparse representation, Concurrency and Computation: Practice and Experience (2019), 5632–5644.

23.

Nencini

, Garzelli

, Baronti

, et al., Remote sensing image fusion using the curvelet transform, Information Fusion 8(2) (2007), 143–156. Special Issue on Image Fusion: Advances in the State of the Art.

24.

Zhang

and Long Guo

, Multifocus image fusion using the nonsubsampled contourlet transform, Signal Processing 89(7) (2009), 1334–1346.

25.

Zhang

and Maldague

, An adaptive fusion approach for infrared and visible images based on nsct and compressed sensing, Infrared Physics & Technology 74 (2016), 11–20.

26.

, Manjunath

and Mitra

, Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing 57(3) (1995), 235–245.

27.

Lewis

J.J.

, O’Callaghan

R.J.

, Nikolov

S.G.

, et al., Pixel- and region-based image fusionwith complexwavelets, Information Fusion 8(2) (2007), 119–130. Special Issue on Image Fusion: Advances in the State of the Art.

28.

Wohlberg

, Efficient algorithms for convolutional sparse representations, IEEE Transactions on Image Processing 25 (2016), 301–315.

29.

Yin

, Duan

, Liu

, et al., A novel infrared and visible image fusion algorithm based on shift-invariant dual-tree complex shearlet transform and sparse representation, Neurocomputing 226 (2017), 182–191.

30.

Chen

, Yang

, et al., An image fusion algorithm of infrared and visible imaging sensors for cyber physical systems, Journal of Intelligent & Fuzzy Systems 36 (2019), 1–16.

31.

Ray

L.A.

and Adhami

R.R.

, Dual tree discretewavelet transform with application to image fusion, in 2006 Proceeding of the Thirty-Eighth Southeastern Symposium on System Theory (2006), 430–433.

32.

Selesnick

I.W.

, Baraniuk

R.G.

and Kingsbury

N.G.

, The dualtree complex wavelet transform, IEEE Signal Processing Magazine 22 (2005), 123–151.

33.

, Kang

and Hu

, Image fusion with guided filtering, IEEE Transactions on Image Processing 22 (2013), 2864–2875.

34.

, Sun

and Tang

, Guided image filtering, IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (2013), 1397–1409.

35.

, Shen

, et al., Recovering Quantitative Remote Sensing Products Contaminated by Thick Clouds and Shadows Using Multitemporal Dictionary Learning, IEEE Transactions on Geoscience and Remote Sensing 52(11) (2014), 7086–7098.

36.

Wright

, Ma

, et al., Sparse Representation for Computer Vision and Pattern Recognition, Proceedings of the IEEE 98(6) (2010), 1031–1044.

37.

Mairal

, Elad

and Sapiro

, Sparse Representation for Color Image Restoration, IEEE Transactions on Image Processing 17(1) (2008), 53–69.

38.

Liu

, Chen

and Ward

R.K.

, Image fusion with convolutional sparse representation, IEEE Signal Processing Letters 23 (2016), 1882–1886.

39.

Aharon

, Elad

and Bruckstein

, K-svd: An algorithm for designing overcomplete dictionaries for sparse representation, IEEE Transactions on Signal Processing 54 (2006), 4311–4322.

40.

, Wu

and Kittler

, MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion, IEEE Transactions on Image Processing 29 (2020), 4733–4746.

41.

, Zhang

and Yan

, Information measure for performance of image fusion, Electronics Letters 38 (2002), 313–315.

42.

Han

, Cai

, Cao

, et al., A new image fusion performance metric based on visual information fidelity, Information Fusion 14(2) (2013), 127–135.

43.

Hossny

, Nahavandi

and Creighton

, Comments on “information measure for performance of image fusion”, Electronics Letters 44 (2008), 1066–1067.