The enhanced deep Plug-and-play super-resolution algorithm with residual channel attention networks

Abstract

In the field of super-resolution image reconstruction, as a learning-based method, deep plug-and-play super-resolution (DPSR) algorithm can be used to find the blur kernel by using the existing blind deblurring methods. However, DPSR is not flexible enough in processing images with high- and low-frequency information. Considering a channel attention mechanism can distinguish low-frequency information and features in low-resolution images, in this paper, we firstly introduce this mechanism and design a new residual channel attention networks (RCAN); then the RCAN is adopted to replace deep feature extraction part in DPSR to achieve the adaptive adjustment of channel characteristics. Through four test experiments based on Set5, Set14, Urban100 and BSD100 datasets, we find that, under different blur kernels and different scale factors, the average peak signal to noise ratio (PSNR) and structural similarity (SSIM) values of our proposed method increase by 0.31dB and 0.55%, respectively; under different noise levels, the average PSNR and SSIM values increase by 0.26dB and 0.51%, respectively.

Keywords

image reconstruction channel attention mechanism residual channel attention networks blur kernel

1 Introduction

The research on a single-image super-resolution (SISR) reconstruction problem, which is how to reconstruct low-resolution (LR) images into high-resolution (HR) ones perfectly and efficiently, has high academic and practical values [1]. In recent years, with the rapid development of deep learning technologies, super-resolution improvement methods based on deep learning have been widely adopted due to their ability to directly learn the mapping between LR and HR images by training an end-to-end network model [2]. At present, most of the SISR improvement methods based on deep learning focus on enhancing network architecture, learning strategies, model frameworks, up-sampling methods, and other aspects [3, 4]. Although all of these aspects can bring some performance improvement, most of the above SISR improvement methods are mainly based on the widely-used bicubic degenerate model. Therefore, it remains a considerable challenge to super-resolved LR images with arbitrary blur kernels for these improvement methods [1].

As one of the deep-learning-based SISR improvement methods, the deep plug-and-play super-resolution (DPSR) algorithm provides a new solution to the above challenge [5]. Specifically, the work of DPSR is reflected in the following aspects. Firstly, a new degradation model is designed in the DPSR, and it is different from the bicubic degradation model and the general degenerate model. The balance is made between the computation of degradation model and the characterization accuracy of degradation process. Secondly, to solve the SISR problem with the new degradation model, a DPSR framework is proposed, which is applicable beyond bicubic degradation and can handle LR images with arbitrary blur kernels. Based on the above measures, it is clear that the plug-and-play prior of SISR is not limited to Gaussian denoising, and the iterative scheme for solving the new degeneration-induced energy function has sound principles.

Although DPSR provides a new idea to solve the reconstruction problem of LR image with the arbitrary blurring kernel, it has a clear limitation. That is, LR images contain valuable low-frequency information, which can be directly forwarded to the final HR output, but DPSR treats a large amount of low-frequency information equally when processing the input LR images across different feature channels, and cannot learn the discrimination across feature channels. Hence, the flexibility in processing different types of information is lost, thus severely hindering the representation capability of neural networks. Given the above shortcoming of DPSR, this paper proposed the enhanced DPSR algorithm with channel attention mechanism (EDPSR), which can distinguish different image features to realize that the convolutional neural network (CNN) treats each channel feature differently, and improve the flexibility of processing different information.

2 Related work

2.1 Residual network

In 2015, He et al. proposed a 152-layer residual network (ResNet) [6]. Due to the effectiveness of ResNet in improving the training speed and prediction accuracy of a deep network, Ledig et al. proposed SRResNet, which is applied ResNet to solve the SISR problem with remarkable results [7]. SRResNet+ was proposed by Zhang et al. based on SRResNet [5], and the network structure is shown in Fig. 1.

Fig. 1

The Network Architecture of DPSR.

As can be seen from Fig. 1, the DPSR network structure consists of the following four parts. Firstly, a convolutional layer (Conv) is used to extract the shallow features of input LR images. Secondly, a long hop connection and a deep feature extractor composed of 16 basic blocks are used to extract the deep features of LR images (each of the 16 basic blocks is composed of a residual block). Thirdly, the features extracted by the deep feature extractor and the shallow feature extractor are fused into an upper sampler for scale adjustment. Finally, a Conv is used as an image reproducer to carry out high-resolution image reconstruction. Compared with SRResnet, SRResNet+ is improved in the following three aspects: (1) the batch normalization layer, which is detrimental to feature extraction and consumes a lot of memory, is removed by SRResNet+; (2) the number of SRResNet feature maps is increased by SRResNet+ from 64 to 96; (3) based on the original SRResNet input volume, SRResNet+ adds an additional noise level map as input. Based on the above improvements, the SRResNet+ can stack more network layers and extract more features in each layer using the same computing resources. However, SRResNet+ cannot distinguish image features and loses the flexibility to handle different types of information. In this paper, we give the following solution to the above problems, which is summarized as follows: based on SRResNet+, we introduce a channel attention mechanism to learn the image space and channel feature correlation, which can adaptively adjust the features through the interdependence between different channels and make the network focus more on the potential high-frequency information recovery of LR images, and finally improve the network feature expression capability.

2.2 Residual channel attention network

The attention mechanism mimics the signal processing of human brain, and the probability distribution of attention is used to emphasize the effectiveness of critical input on output [8]. Since 2014, it has performed well in natural language processing, speech recognition, and image recognition to improve the performance of neural networks. It has been favored by many scholars in the field of deep learning [9, 10]. In the field of image recognition, Zhang et al. proposed the residual channel attention networks (RCAN) in 2018 [11]. As shown in Fig. 2, the main work of RCAN can be divided into two aspects. On the one hand, the attention mechanism is introduced to treat different channels to improve the representative ability of the CNN. On the other hand, a new residual in residual (RIR) structure is introduced, allowing a large amount of low-frequency information to be transmitted through multiple skip connections, so that the CNN can focus on learning high-frequency information. Compared with algorithms without channel attention mechanism, RCAN realizes the ability to distinguish learning across feature channels. Hence, we introduce the RCAN module to transfer various types of high-frequency and low-frequency information from LR images to HR images, and then propose the EDPSR, which is described in detail in the following section.

Fig. 2

The Network Architecture of RCAN.

3 Design for deep Plug-and-Play super-resolution algorithm with residual channel attention networks

In Section 2, we analyze the limitations of the DPSR algorithm in network structure and explain the improvement of the attention mechanism introduced by RCAN. This section focuses on the design of the super-resolution model based on the channel attention mechanism.

3.1 Selection of blur kernels

There are many types of blur kernels, such as linear kernels, polynomial kernels, Gaussian kernels, exponential kernels, ANOVA kernels, etc. [12]. To show the improved algorithm’s effectiveness in this paper, as shown in Fig. 3, we have selected three types of blur kernels (disk blur kernels, Gaussian blur kernels, and motion blur kernels) with the same size in the DPSR algorithm [5].

Fig. 3

Examples of blur kernels.

3.2 The EDPSR algorithm

In this section, the EDPSR algorithm is proposed based on the introduction of residual group [11]. In Section 1 and 2.2, the DPSR algorithm achieves good results in both deblurring and denoising. However, considering it cannot effectively distinguish the feature information with different degrees of importance in LR images, much effective feature information may be lost in the reconstruction process. If this critical feature information is lost in the real image reconstruction, it will lead to low image quality after reconstruction. To solve the above problems, RCAN is introduced to enhance the perception ability of DPSR algorithm and the transmission ability during the reconstruction process. Specifically, it is divided into the following two parts:

The residual channel attention block (RCAB) is used to rescale channel-wise features adaptively by considering interdependencies among different channels;

the RIR structure is adopted to directly transmit the effective low-frequency characteristic information through multiple hop connection, under the cooperation of multiple short skip connection (SSC) and one long skip connection (LSC), so that the main network can focus on learning high-frequency information.

In this way, the network can intelligently understand the importance of different channel characteristics. Therefore, the introduction of RCAN in this article can effectively solve the information loss problem. As shown in Fig. 2, the RCAN network comprises four parts: shallow feature extraction part, deep feature extraction part, up-sampling module, and image reconstruction part. The working process is as follows: firstly, a 3×3 Conv is used to extract shallow features from the LR input, and a feature map is obtained; secondly, deep feature extraction is carried out through the RIR structure, which contains ten residual groups (RG), a Conv of 3×3 and an LSC, and each RG is formed by the superposition of 20 RCAB and one SSC; thirdly, the up-sampling is performed through the pixelshuffle module; Finally, image reconstruction is realized through a Conv of 3×3. Based on the above analysis, this paper proposes the enhanced deep plug-and-play super-resolution algorithm with residual channel attention networks, and its network structure is shown in Fig. 4. The improvement of EDPSR network is mainly reflected in the deep feature extraction part.

Fig. 4

The Network Architecture of EDPSR.

In Fig. 4, after the first Conv extracts the shallow feature F_SF of the input, F_SF is fed into the RIR structure to obtain depth feature F_DF. Before F_SF is introduced into the first basic block (RG) of the RIR, F_SF is identically mapped to the tail of the RIR structure through LSC. When F_SF enters the first RG of the RIR structure (what has been discussed above, we already know that RG is composed of N RCAB, an SSC, and a Conv), F_SF propagates backward through SSC identity mapping. At the same time, when F_SF is passed to the first RCAB in this RG, a Conv-Relu-Conv operation is performed, and the residual f_1,1 with dimension H × W × C is obtained. Then f_1,1 is rescaled by a CA module to obtain the adjusted feature f_c1,1. In the end, to obtain the output feature $F_{1, 1}^{'}$ of the first RCAB, the “element-wise sum” operation on F_SF and f_c1,1 is performed, and $F_{1, 1}^{'}$ is taken as the input of the second RCAB. After repeating the above processes, the output $F_{1, N}^{'}$ of the N-th RCAB is obtained. After convolving $F_{1, N}^{'}$ with a Conv, the “element-wise sum” operation on the convolved result and the F_SF of the previous SSC identity mapping is performed, and the output $F_{1}^{'}$ of the first RG is obtained. Repeat the above processes to get the output $F_{G}^{'}$ of the G-th RG module, $F_{G}^{'}$ is convolved with the last Conv at the end of the RIR structure. In order to obtain F_DF, the “element summation” operation is performed on the convolution result and F_SFof the previous LSC identity mapping. Ultimately, the HR image is obtained by inputting F_DF into the upsampling module and reconstruction module. It should be noted that the size of all Conv mentioned above is 3×3, and the number of feature maps is 96.

3.3 Implementation details

It should be emphasized here that, after the introduction of RCAN, by increasing the number of RGs in RCAN and increasing the number of stacked RCABs in each RG, the representable performance of the original neural network will be improved to some extent due to the deepening of network depth. However, given that the purpose of this paper is to explore the before-and-after differences caused by attentional mechanisms on DPSR, to avoid or reduce the effectiveness of other disturbance quantities (the network depth or width, etc.), we deliberately ignore the advantages of RCAN in enhancing the depth of the network. Clearly, during the design of deep feature extraction part in EDPSR, the number of RGs G is set to 16, and the number of RCABs N contained in each RG is set to 1, to ensure that the network depth of EDPSR is close to the depth of DPSR. In addition, the rest of EDPSR network is consistent with that of DPSR.

4 Experiments

We use the DIV2K dataset [13] as the training set and Set5 [14], Set14 [15], BSD100 [16], Urban100 [17] as the test sets. The minibatch size is set to 16. The patch size of LR input is set to 48 × 48. We use the Adam optimizer to train our model. The initial learning rate is set to 10^-4, decreasing half after every 5 × 10⁵ backpropagation iterations, and the training ends when the learning rate falls below 10^-7. The experimental environment is shown in Table 1.

Table 1
Experimental environment

Item Details Item Details

OS Windows 10 Language Python3.6

CPU AMD Ryzen
5 2600 GPU NIVIDA
GTX 1080Ti

Memory 32G Framework Pytorch1.1.0

Item	Details	Item	Details
OS	Windows 10	Language	Python3.6
CPU	AMD Ryzen 5 2600	GPU	NIVIDA GTX 1080Ti
Memory	32G	Framework	Pytorch1.1.0

4.1 Training of EDPSR

1) The evaluation index. Here, the peak signal to noise ratio (PSNR) and the structural similarity (SSIM) are selected to obtain the universal evaluation results of DPSR and EDPSR [18]. The PSNR solution is shown as follows: $PSNR = 10 \cdot {log}_{10} (\frac{MA X_{L}^{2}}{MSE})$ (1) where MSE represents the mean square error of reconstructed image L and reference image H. MAX_L is the maximum value representing the color of image points, which is usually 255. The unit of PSNR is dB; the value range of PSNR is [0, + ∞). The SSIM solution formula is obtained as follows: $\begin{matrix} SSIM (L, H) \\ = \frac{(2 u_{L} u_{H} + C_{1}) (2 σ_{L} σ_{H} + C_{2}) (σ_{LH} + C_{3})}{(u_{L}^{2} + u_{H}^{2} + C_{1}) (σ_{L}^{2} + σ_{H}^{2} + C_{2}) (σ_{L} σ_{H} + C_{3})} \end{matrix}$ (2) where u_L and u_H are the mean values of image L and H; σ_L and σ_H represent the variances of images L and H, respectively; σ_LH is the covariance of two images; C₁, C₂, and C₃ are constants. The value range of SSIM is [0,1], which can reflect the image reconstruction ability of the algorithm.

2) Different aspects of performance training. To visualize the EDPSR training process, we sample the first 5000 training iterations and compare the results of EDPSR with DPSR. As described above, we use the first 800 images from DIV2K as the training set and the Set5 dataset as the test set. Without doing anything special to Set5 (adding noise or blur kernels), and just setting the upsampling scale to 4, the test results of DPSR and EDPSR are shown in Fig. 5.

Fig. 5

Comparison between PSNR, SSIM and LOSS.

Fig. 5 shows the resultant curves of the average LOSS, PSNR, and SSIM values after 5000 training epochs for Set5. In Fig. 5a, when the LOSS value is equal to 0.05, compared with DPSR, the training iteration required by EDPSR decreased by 3.59%. Furtherly, it can be seen that when EDPSR drops, the fluctuation of its curve is greater than that of DPSR. The reason is that adding the attention mechanism can effectively improve the feature learning ability of the original model. In Fig. 5b and 5c, the curves of EDPSR are higher than that of DPSR in both speed and amplitude of rising. Compared with DPSR, when the PSNR value reaches 25dB, the training iteration required for EDPSR has been decreased by 27.87%. When the SSIM value is equal to 0.7, the least iteration needed by EDPSR has been dropped by 31.48%. The reason is that, DPSR does not effectively distinguish and treat input information with different levels of importance.

4.2 Comparison of deblurring results

In this section, we focus on the performance differences between EDPSR and DPSR in deblurring. Firstly, after convolution with three types of blur kernels and down-sampling, all HR images in the four public benchmarks are processed into LR images with blur kernels. Secondly, EDPSR is used to deblur and reconstruct LR images. Finally, the reconstructed images are compared with their corresponding HR images.

1) Quantitative results. Deblurring results of DPSR and EDPSR with different degradation settings on public benchmarks are shown in Table 2. Compared with DPSR, we can find the following four conclusions. Firstly, under the arbitrary scale factor, EDPSR consistently achieves higher values of PSNR or SSIM on pubic benchmarks than DPSR, regardless of the type of blur kernel selected. Secondly, as the test dataset changes from Set5, Set14, BSD100 to Urban100, the average PSNR values increase by 0.31dB, 0.32dB, 0.30dB, and 0.29dB, respectively; the average SSIM values increase by 0.58%, 0.68%, 0.28%, and 0.65%, respectively. Thirdly, when facing different scale factors, as the kernel types change from Gaussian, motion to disk, the average PSNR values increase by 0.31dB, 0.30dB, and 0.30dB, respectively; the average SSIM values increase by 2.65%, 0.66%, and 0.38%, respectively. Finally, when facing different kernel types, as the scale factor increase from 2 to 4, the average PSNR values increase by 0.33dB, 0.30dB, and 0.28dB, respectively; the average SSIM values increase by 0.71%, 0.44%, and 0.70%, respectively. The reasons for these four points are that our proposed method can achieve a better image deblurring effect based on DPSR. The introduction of CA mechanism provides DPSR with a more efficient modeling method to capture global context information. Under the same scaling factor, the CA mechanism significantly improves thenetwork’s sensitivity to useful information. It improves the stability and continuity of the original low-frequency information in the transmission process.

Table 2
Deblurring test results (PSNR(dB) /SSIM) on public benchmarks

Degradation Setting Set5 Set14 BSD100 Urban100

Scale Factor Kernel Type DPSR EDPSR DPSR EDPSR DPSR EDPSR DPSR EDPSR

×2 Guassian 30.84/0.867 31.15/0.872 27.31/0.760 27.68/0.767 27.07/0.745 27.44/0.752 25.64/0.793 25.93/0.800

Motion 34.01/0.918 34.48/0.923 29.44/0.826 29.81/0.833 29.77/0.855 30.05/0.859 29.36/0.892 29.68/0.898

Disk 32.22/0.888 32.60/0.892 27.55/0.828 27.79/0.835 28.33/0.798 28.62/0.804 27.51/0.843 27.84/0.849

×3 Guassian 28.76/0.819 28.99/0.823 26.13/0.715 26.42/0.718 25.97/0.686 26.28/0.689 24.24/0.726 24.53/0.731

Motion 32.23/0.893 32.53/0.900 28.77/0.808 29.12/0.812 28.23/0.795 28.49/0.780 27.47/0.845 27.78/0.852

Disk 30.44/0.873 30.79/0.878 26.53/0.727 26.79/0.734 27.01/0.737 27.33/0.742 25.74/0.787 26.08/0.793

×4 Guassian 26.63/0.802 26.84/0.807 23.43/0.611 23.76/0.614 23.95/0.589 24.30/0.594 21.81/0.623 22.13/0.628

Motion 29.54/0.841 29.79/0.846 24.91/0.686 25.17/0.691 25.51/0.677 25.78/0.681 24.03/0.737 24.29/0.742

Disk 28.03/0.805 28.32/0.809 24.11/0.614 24.46/0.620 24.69/0.631 24.94/0.637 22.67/0.681 22.89/0.686

Degradation Setting	Set5	Set14	BSD100	Urban100
×2	Guassian	30.84/0.867	31.15/0.872	27.31/0.760	27.68/0.767	27.07/0.745	27.44/0.752	25.64/0.793	25.93/0.800
	Motion	34.01/0.918	34.48/0.923	29.44/0.826	29.81/0.833	29.77/0.855	30.05/0.859	29.36/0.892	29.68/0.898
	Disk	32.22/0.888	32.60/0.892	27.55/0.828	27.79/0.835	28.33/0.798	28.62/0.804	27.51/0.843	27.84/0.849
×3	Guassian	28.76/0.819	28.99/0.823	26.13/0.715	26.42/0.718	25.97/0.686	26.28/0.689	24.24/0.726	24.53/0.731
	Motion	32.23/0.893	32.53/0.900	28.77/0.808	29.12/0.812	28.23/0.795	28.49/0.780	27.47/0.845	27.78/0.852
	Disk	30.44/0.873	30.79/0.878	26.53/0.727	26.79/0.734	27.01/0.737	27.33/0.742	25.74/0.787	26.08/0.793
×4	Guassian	26.63/0.802	26.84/0.807	23.43/0.611	23.76/0.614	23.95/0.589	24.30/0.594	21.81/0.623	22.13/0.628
	Motion	29.54/0.841	29.79/0.846	24.91/0.686	25.17/0.691	25.51/0.677	25.78/0.681	24.03/0.737	24.29/0.742
	Disk	28.03/0.805	28.32/0.809	24.11/0.614	24.46/0.620	24.69/0.631	24.94/0.637	22.67/0.681	22.89/0.686

2) Visual results. According to the above experimental steps, taking the butterfly image from the Set14, the glass pyramid from BSD100, and the station image of Urban100 as an example, when the scaling factor is set to 4, the image deblurring reconstruction results of DPSR and EDPSR are shown in Fig. 6. The blur kernel is shown in the upper-left corner of the LR image. It can be seen that under the three types of blur kernels (Disk, Gaussian, and Motion), DPSR has basically been able to achieve clearer image reconstruction, but compared with EDPSR, it is still slightly inferior in some detail textures, color saturation, and clarity. EDPSR could obtain more apparent reconstruction results and restore more high-frequency details (such as high contrast and sharp edges). Due to the limited input information of available LR images, although it is challenging to recover high-frequency information, EDPSR can still use a little LR information by attention mechanism to learn spatially and channel feature correlations and has a more robust representation ability and obtain in more accurate reconstruction results.

Fig. 6

Visual comparison of deblurring performance between DPSR and EDPSR.

4.3 Comparison of denoising results

In this section, we focus on the performance differences between EDPSR and DPSR in denoising. The procedure is roughly the same as in Section 4.2. Firstly, after adding different noise and down-sampling levels, all HR images in the four public benchmarks are processed into LR images with the noise level. Secondly, EDPSR is used to denoise and reconstruct LR images. Finally, the reconstructed images are compared with their corresponding HR images. However, the only difference here is that, we set the scale factor for 4 for a clean and straightforward comparison of denoising performance.

1) Quantitative results. The denoising results of DPSR and EDPSR with different degradation settings on public benchmarks are shown in Table 3, from which we have several observations. Firstly, under the arbitrary scale factor, EDPSR always achieves higher values of PSNR or SSIM on pubic benchmarks than DPSR, regardless of the difference of noise level selected. Secondly, as the test dataset changes from Set5, Set14, BSD100 to Urban100, the average PSNR value of EDPSR increase by 0.22 dB, 0.22dB, 0.31dB, and 0.28dB; the average SSIM value increase by 0.72%, 0.92%, 1.30%, and 0.78%. Thirdly, when facing different kernel types, as the noise level changes from 1%, 2% to 3 %, the average PSNR value increase by 1.31dB, 1.02dB and 0.24dB; the average SSIM value is increased by 0.71%, 0.98%, and 1.01%. Fourthly, when facing different noise levels, as the kernel type changes from Disk, Gaussian, and Motion, the average PSNR value of EDPSR increases by 1.77dB, 2.17dB, and 0.24dB; the average SSIM values increase by 0.80%, 0.87%, and 1%. Similarly, it can be seen from Table 3 that EDPSR has a better denoising effect than DPSR in different blur kernels and various noise levels under the same objective evaluation index. The reason is that, the CA mechanism can assign different weights to features with varying value scales in the input image. By focusing on reconstructing the most relevant aspects of the image, the noise and redundant information in the input image are largely ignored, resulting in a reconstructed image with a high level of useful information.

Table 3
Denoising test results (PSNR(dB) /SSIM) on public benchmarks

Degradation Setting Set5 Set14 BSD100 Urban100

Kernel Type Noise Level DPSR EDPSR DPSR EDPSR DPSR EDPSR DPSR EDPSR

Guassian 2.55(1%) 24.58/0.673 24.85/0.678 22.55/0.558 22.77/0.561 20.60/0.556 20.94/0.559 22.88/0.536 23.16/0.540

5.1(2%) 23.76/0.632 23.92/0.636 22.07/0.544 22.28/0.548 20.11/0.483 20.38/0.489 22.43/0.514 22.70/0.517

7.65(3%) 23.22/0.618 23.45/0.623 21.74/0.524 22.03/0.530 19.80/0.512 20.14/0.519 22.13/0.501 22.38/0.504

Disk 2.55(1%) 23.93/0.676 24.27/0.682 22.13/0.547 22.47/0.549 20.49/0.552 20.74/0.558 22.44/0.518 22.75/0.522

5.1(2%) 22.54/0.597 22.83/0.601 21.27/0.508 21.46/0.513 19.50/0.498 19.78/0.503 21.68/0.488 21.94/0.490

7.65(3%) 21.75/0.612 21.82/0.616 20.76/0.492 20.81/0.499 18.94/0.470 19.32/0.476 21.21/0.471 21.59/0.476

Motion 2.55(1%) 26.20/0.742 26.48/0.745 23.32/0.609 23.51/0.614 22.08/0.644 22.29/0.650 23.68/0.579 23.87/0.583

5.1(2%) 24.54/0.694 24.73/0.700 22.41/0.565 22.69/0.570 21.07/0.588 21.46/0.604 22.74/0.532 23.04/0.537

7.65(3%) 23.47/0.654 23.62/0.659 21.79/0.538 21.92/0.544 20.44/0.543 20.79/0.548 22.14/0.505 22.41/0.511

Degradation Setting	Set5	Set14	BSD100	Urban100
Guassian	2.55(1%)	24.58/0.673	24.85/0.678	22.55/0.558	22.77/0.561	20.60/0.556	20.94/0.559	22.88/0.536	23.16/0.540
	5.1(2%)	23.76/0.632	23.92/0.636	22.07/0.544	22.28/0.548	20.11/0.483	20.38/0.489	22.43/0.514	22.70/0.517
	7.65(3%)	23.22/0.618	23.45/0.623	21.74/0.524	22.03/0.530	19.80/0.512	20.14/0.519	22.13/0.501	22.38/0.504
Disk	2.55(1%)	23.93/0.676	24.27/0.682	22.13/0.547	22.47/0.549	20.49/0.552	20.74/0.558	22.44/0.518	22.75/0.522
	5.1(2%)	22.54/0.597	22.83/0.601	21.27/0.508	21.46/0.513	19.50/0.498	19.78/0.503	21.68/0.488	21.94/0.490
	7.65(3%)	21.75/0.612	21.82/0.616	20.76/0.492	20.81/0.499	18.94/0.470	19.32/0.476	21.21/0.471	21.59/0.476
Motion	2.55(1%)	26.20/0.742	26.48/0.745	23.32/0.609	23.51/0.614	22.08/0.644	22.29/0.650	23.68/0.579	23.87/0.583
	5.1(2%)	24.54/0.694	24.73/0.700	22.41/0.565	22.69/0.570	21.07/0.588	21.46/0.604	22.74/0.532	23.04/0.537
	7.65(3%)	23.47/0.654	23.62/0.659	21.79/0.538	21.92/0.544	20.44/0.543	20.79/0.548	22.14/0.505	22.41/0.511

2) Visual results. According to the above experimental steps, taking one doccinella septempunctata image and two frog images as an example, to visually compare the denoising performance of the DPSR and EDPSR, we just add different levels of noise to the picture above, when the noise level is changed from 1%, 2% to 3%, the image denoising reconstruction results of DPSR and EDPSR are shown in Fig. 7. It can be seen that under the three sizes of noise levels, DPSR can reduce noise to a certain extent, but compared with EDPSR, it is still slightly inferior in suppressing some noise points and detail textures. EDPSR could obtain better denoising results. The main reason is that, when different levels of noise are used as interference in the image reconstruction process, compared with DPSR, which treats the information in the input image equally, our improved algorithm utilizes the CA mechanism to place maximum attention on the recovery of some of the original more important features in the input image and gives them a larger weight. The weight of the interfering information from noise is reduced, which indirectly realizes the hindrance of noise interference and achieves a higher denoising performance.

Fig.7

Visual comparison of denoising performance between DPSR and EDPSR.

5 Conclusions

In this paper, the EDPSR is proposed to improve the performance of DPSR. During image reconstruction, low-resolution images contain a wealth of low-frequency information treated equally in DPSR and cannot be effectively distinguished, thus hindering the ability of CNN representation. We introduce an attentional mechanism into DPSR that could adaptively redistribute channel features by taking into account interdependencies between channels; then, with the cooperation of SSC and LSC, a large amount of low-frequency information is bypassed by multiple skip connections, allowing the main network to focus on learning high-frequency information. Compared with the DPSR, the experiment result proves that EDPSR has higher training and test accuracy and better image reconstruction effects in image deblurring or denoising. The proposed measures can demonstrate the effectiveness of the neural network attention mechanism in solving the super-resolution reconstruction problem of LR images with arbitrary blur kernels, and accelerate the progress of super-resolution image reconstruction technology in the fields of urban management, medical imaging, satellite, and aerial imaging.

Footnotes

Acknowledgments

This work is supported by National Science Foundation of China (51804249, 61603295), and Shaanxi Postdoctoral Science Foundation (2018BSHEDZZ124).

References

Zhang

, Zuo

, Zhang

, Learning a single convolutional super-resolution network for multipledegradations, in Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3262–3271, 2018.

Yang

, Zhang

, Tian

, Wang

, Xue

, Liao

, Deep learning for single image super-resolution: A briefreview, IEEE Transactions on Multimedia 21(12) (2019), 3106–3121.

Wang

, Chen

, Hoi

S.C.

, Deep learning for image super-resolution: A survey, IEEE Transactions onPattern Analysis and Machine Intelligence PP(99) (2020), 1–22.

Pan

, Mi

, Lei

, Deng

, A closed-loop brain-machine interface framework design for motorrehabilitation, Biomedical Signal Processing and Control, 2020, 10.1016/j.bspc.2020.101877

Zhang

, Zuo

, Zhang

, Deep plug-and-play super-resolution for arbitrary blur kernels, in Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1671–1681, 2019.

, Zhang

, Ren

, Sun

, Deep residual learning for image recognition, in Proceedings of the 2016IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016.

Ledig

, Theis

, Huszár

, Caballero

, Cunningham

, Acosta

, Aitken

, Tejani

, Totz

, Wang

, et al., Photo-realistic single image super-resolution using a generative adversarial network, in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4681–4690, 2017.

Treisman

A.M.

and Gelade

, A feature-integration theory of attention, Cognitive Psychology 12(1) (1980), 97–136.

Stark

, Oskoui

, High-resolution image recovery from image-plane arrays, using convex projections, Journal of The Optical Society of America A-optics Image Science and Vision 6(11) (1989), 1715–1726.

10.

Pan

, Mi

, Wen

, Zhong

, An adaptive decoder design based on the receding horizon optimization in bmisystem, Cognitive Neurodynamics 14 (2020), 281–290.

11.

Zhang

, Li

, Wang

, Zhong

, Fu

, Image super-resolution using very deep residual channelattention networks, in Proceedings of the 2018 European Conference on Computer Vision (ECCV), pp.286–301, 2018.

12.

Cho

, Paris

, Horn

S. B. K.

and Freeman

W. T.

, Blur kernel estimation using the radon transform, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 241–248, 2011.

13.

Agustsson

, Timofte

, Ntire 2017 challenge on single image super-resolution: Dataset and study, in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 126–135, 2017.

14.

Bevilacqua

, Roumy

, Guillemot

, L.

, Alberimorel, Low-complexity single-image super-resolution basedon nonnegative neighbor embedding, in Proceedings of the 2012 British Machine Vision Conference 135(2012), 126–135.

15.

Zeyde

, Elad

, Protter

, On single image scale-up using sparse-representations, in Proceedings ofthe 2010 International Conference on Curves and Surfaces, pp. 711–730, Springer, 2010.

16.

Martin

, Fowlkes

, Tal

, Malik

, A database of human segmented natural images and its application toevaluating segmentation algorithms and measuring ecological statistics, in Proceedings of the 8th IEEEInternational Conference on Computer Vision (CVPR), 2 (2001), 416–423, IEEE.

17.

Huang

, Singh

, Ahuja

, Single image super-resolution from transformed self-exemplars, in Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206, 2015.

18.

Agrawal

, ℓ_1 sabmis: ℓ_1-minimization and sparse approximation based blind multi-imagesteganography scheme, arXiv preprint arXiv:2007.05025, 2020.

The enhanced deep Plug-and-play super-resolution algorithm with residual channel attention networks

Abstract

Keywords

1 Introduction

2 Related work

2.1 Residual network

3.1 Selection of blur kernels

4 Experiments

Table 1 Experimental environment Item Details Item Details OS Windows 10 Language Python3.6 CPU AMD Ryzen5 2600 GPU NIVIDAGTX 1080Ti Memory 32G Framework Pytorch1.1.0

Footnotes

Acknowledgments

References

Table 1
Experimental environment

Item Details Item Details

OS Windows 10 Language Python3.6

CPU AMD Ryzen
5 2600 GPU NIVIDA
GTX 1080Ti

Memory 32G Framework Pytorch1.1.0