SuperVidConform: Conformation detail-preserving network (CDPN) for video super-resolution

2.1 Temporal correlation

Temporal correlations built into the VSR architecture [22, 25, 26, 33–35] are generally famous for end-to-end learning frameworks. Two strategies are widely used to model temporal information: Temporal Concatenation and Temporal Aggregation. Former is a widely used VSR method [17, 19, 20, 36] that concatenates several frames for preserving temporal information. Using multiple input images is an expansion of ISR using this method. However, the concatenation method can lead to difficult training as it fails to represent various motion regimes within a single input sequence correctly. Later uses temporal aggregation in which multiple SR inferences operate in various motion regimes as proposed by [8, 11, 22, 28] to overcome the dynamic motion issue in VSR. The final layer constructs an SR frame by combining the results of all branches

2.2 Exploitation of motion information

Several DL methods have recently been proposed to tackle the VSR problem explicitly and implicitly, such as [30, 37]. Most VSR methods use explicit MC, which takes a direct channel as MEMC with information fusion and upsampling. These methods include a joint motion compensation module proposed by VESPCN [26] and an SPMC module introduced by Tao et al. [20], task-oriented flow modules proposed by Xue et al. [9], and recurrent frame modules proposed by Sajjadi et al. [8]. They estimate motion frame-to-frame and align the results to a reference frame. However, a significant disadvantage of these techniques is high computational cost caused by MEMC. Conversely, implicit MC techniques reduce video SR computational load by implicitly using motion information rather than explicitly. This can be done by incorporating motion information into the network in a way that does not require additional computation. Methods with implicit MC [21–23, 25] develop an improved module that fully utilizes complementary information from different frames. Jo et al. [23] predicted a dynamic upsampling filter to recover the image, while Yi Huang et al. [25] fused spatial-temporal information. Huang et al. [38] used a bidirectional recurrent convolutional network; in [21] Fuoli used recurrent architecture in feature space. The proposed method in this paper utilizes an implicit MC technique to reduce the heavy computational burden of explicit MEMC. This method also takes advantage of the relationship between the current frame and the hidden state to actively use historical data within the hidden state, improving performance and lowering the risk of error accumulation.

2.3 Recurrent neural network

Recently, there has been a lot of use of recurrent networks in tasks related to video processing, such as VSR. These networks [11, 22, 28, 38–40] can handle input and output that involve time by analyzing sequential data by combining data from every frame while keeping track of their individual hidden states. However, a big challenge with this approach is that when working with long sequences of frames, the training process can become complex due to gradient vanishing [41–43]. To address this problem, the proposed method utilizes a Recurrent Residual Network (RRN), which uses a residual mapping between layers and includes identity skip connections, which helps to avoid the gradient vanishing risk in the training process. Therefore, motivated by these VSR aspects of temporal correlation, exploitation of motion information, and handling the level of recurrent network for temporal input and output, this work proposed a general framework for long-range encoding video.

3.1 Network framework

The proposed CDPN comprises the following components: a decomposition of images to preserve essential details, a module for extracting features, a module for increasing resolution, and a module for combining these elements to produce the final high-resolution result as shown in Fig. 1. The annotation that follows, It, I _t-1, I _{t C} , I _{t-1 C} , I_HRC, I_HR, and I_GT, denote the current and previous input, the conformation details of current and previous input, the recovered details of the image, the final high-resolution output, and the original high- resolution image that serves as the reference point, respectively. Equation (1) decomposes the input from the current and previous frames to get corresponding conformation details from the image.

OPEN IN VIEWER

Fig. 1

Schematic illustration of the proposed method.

F oi = H decomp ( I t , I t - 1 )(1) F o 1 = H f ( I t C ) , F o 2 = H f ( I t - 1 C ) where H_Decomp (·) denotes the decomposition operation. Then, the extracted details F_oi (i = 1, 2) put forward to Equation (2) F Dfi = H RRN ( F oi ) ( i = 1 , 2 )(2)

H_RRN (·) denotes the residual RNN feature extraction module, consisting of residual learning to extract deep features (DF). Furthermore, the extracted detail feature F_DFi is then upscaled via the upsampling module to eliminate the pixelation effect and estimate extra image details.F UPi = H UP ( F DFi ) ( i = 1 , 2 )(3)

H_UP (·) and F_UPi refer to the module for increasing resolution and the features that have been increased in resolution, respectively. This approach employs a post-resampling method that relies on the high-level detail obtained from the low-level space and yields improved outcomes for VSR than a predefined upsampling method. Then, the recovered conformation details input IHRC was estimated from F_UPi, where i = 1,2.I HRC = H REC ( F UPi )(4)

Where H_REC (·) stands for the recovered upsampled input. Finally, these recovered conformation details of IHRC are inputted into a composition module to produce the final HR image IHR, where H_comp (·) represents the composition module.I HR = H comp ( I HRC )(5)

Recovering the high-resolution image from its low-resolution counterpart can be difficult due to its ill-posed problem. To overcome this challenge, the authors in this paper utilized the conformation detailing approach, as shown in Fig. 2. This method learns the high-frequency information, such as texture, separately, thereby enhancing the final image quality.

OPEN IN VIEWER

Fig. 2

Illustration of the decomposition process.

The high-frequency information in an image, such as textures, can be extracted by taking the difference between the original image and a blurred version [27, 44]. This method has been used in low-level image processing tasks, such as identifying boundaries and assessing image quality [45].

Motivated by the above findings to recover high-frequency information, the input image is separated into its high-frequency components, which are obtained by subtracting the original image from a version that has been blurred using a Gaussian filter. These high-frequency details are then used in a composition module to generate the final high-resolution image, as shown in Equation 5.

This work comprehensively studies different temporal modeling frameworks, which include 2D CNN and RNN. Figure 3(a) and (b) illustrate these networks, respectively, and Fig. 3(c) shows the proposed hidden state architecture. The input frames are joined together in a 2D CNN. RNN, on the other hand, uses fewer frames as input and processes a video sequence repeatedly. Typically, the hidden state at a particular time step is composed of three components: the previous output from time step o_t-1, the previously hidden state representation h_t-1, and the two neighboring frames I_t–1,t. Intuitively, pixels in consecutive frames of a video sequence are often similar. The result is improved by using information from the previous layer to refine the t-th time step that contains the high-frequency details. However, like other video processing tasks, RNN in VSR [21] can be affected by the issue of gradient vanishing [41–43]. The recurrent residual network (RRN) is proposed to overcome this problem, which uses residual learning and includes identity skip connections [37].

OPEN IN VIEWER

Fig. 3

Frameworks for modelling temporal data that are frequently used: A) 2D CNN, B) RNN, C) Proposed RRN.

The recurrent residual network (RRN) design ensures that information flows smoothly, allowing it to retain texture information over a long period, making it easier for RNNs to handle longer videos and using a set of equations within the RRN framework, it may reduce the chance of gradient vanishing during training by using each time step is ‘t’ to produce two outputs, h_t, and o_t, which are then used to guide the next time step, ‘t + 1’:x o = σ ( W conv 2 D { [ I t – 1 , I t , o t – 1 , h t – 1 ] } )(6) x k = g ( x k – 1 ) + F ( x k – 1 ) ,(7)

where,

k ∈ [1, K]

h _t  = σ (W_conv2D {x_K}) ot = W_conv2D {x_K} Equation 6 uses the ReLU function represented by σ (·). In the k-th residual block, the term g (x_k–1) denotes a simple mapping, indicating that (x_k–1) = x_k–1 in Equation 7. In contrast, the newly obtained residual mapping is denoted by the notation F (x_k–1).

The CDPN is optimized using a training loss function, the sum of all outputs.L ( Θ ) = ω 1 ι 1 ( I HR , I GT )(8)

The symbol Θ represents the set of parameters for the proposed network, as shown in Fig. 2. The term ι ¹ (I_HR, I_GT) represents the overall loss for the network, where I_HR is the final HR output of the CDPN, and I_GT is the ground-truth HR image. The term ω_k represents the weight of each loss.

In the field of VSR, various loss functions have been proposed to guide the optimization of networks, such as pixel loss [8, 22, 46, 47] (e.g., ι1 loss), content loss [25, 31], etc. The ι ¹ loss function is a simple yet effective method to obtain a high peak signal-to-noise ratio (PSNR) with less complexity. The pixel loss promotes the recovered HR image (I_HR) to be visually similar to its ground truth (I_GT). Thus, to achieve a high PSNR and high-quality regional recovery, and accurate edges, the proposed method uses loss as a ι ¹ loss function:ι 1 ( I HR , I GT ) = ι pi x 1 ( I HR , I GT )(9)

Specifically, the L1 loss function is defined as follows: Given a set of N training data pairs {I_iHR, I_iGTN_i=1 where I_iHR is a recovered image and I_iGT is the corresponding ground truth image, the ι ¹ loss function measures the difference between the recovered and ground truth images using the ι1 metric.ι pi x 1 1 N ∑ i = 1 N ∥ I HR - I GT ∥ 1(10)

1) Difference to the CNN Model: CNN models [46, 48] detect image features adaptively. The differences between the CNN-based and proposed CDPN model are as follows: (i) The CNN-based model is PSNR-oriented and focuses on pursuing high accuracy only. In contrast, the CDPN aims to achieve high visual quality and PSNR. Unlike a CNN-based model that only uses gradient maps to recover images, CDPN uses gradient features from the original image to improve reconstruction. Additionally, while the CNN-based model only considers spatial correlation to extract features of LR images, CDPN uses a more comprehensive approach. In contrast, our CDPN uses spatial and temporal correlations in the network to increase performance. (iv) The CNN-based model uses optical flow algorithms for alignment, which increases the complexity, while CDPN does not require any particular alignment algorithm, resulting in reduced complexity. The experimental results are presented in section 4.

2) Difference to the RNN Model: RNN is a recurrent-based VSR model [8, 49, 50], and the key differences between RNN and CDPN are: (i) RNN only has hidden layers stacked, while CDPN has several residual groups with skip connections included, allowing for more low-frequency information to be preserved and inter-dependencies to be accounted for. (ii) RNN only uses contextual information within a limited area and cannot consider local information. In contrast, CDPN can alleviate this issue by introducing the decomposition of the original image, which captures spatial contextual information and conformation details. Section 4 presents proposed experimental results that aim to balance high PSNR and good visual quality.

3) Requirement of residual connection in the hidden state of RNN: Identity mapping is utilized in the hidden state, resulting in an advanced model [45, 48, 50]. The baseline model performs best in PSNR, using the hidden state. However, when the number of blocks increases, it suffers the vanishing gradient issues. Residual connections in the hidden state are added to stabilize progress as the number of blocks increases. The results highlighted that identity mapping helps to improve VSR performance while sustaining training. Adding more blocks can help the RRN function perform even better.

4.1 Datasets

Authors in this experimental analysis utilized the publicly available Vimeo-90k [9] dataset for model training. This dataset consists of 90k video scenes with high visual quality. Gaussian blur with σ = 1.6 was applied on a dataset, with patch size 64×64, and further down sampling was performed with a 4× scale factor. The performance evaluation for the proposed method was done using VID4 [14], SPMC [20], and UDM10 [25] standard benchmark datasets. Four scenes make up VID4, each with its motion. The most recent validation sets, SPMC and UDM10, feature a variety of senses with frames that are far higher in resolution than VID4. For a fair comparison, the quantitative results are evaluated using PSNR and structure similarity index metrics (SSIM).

4.2 Implementation details

This work considers two models for a given temporal method: CDPN-S and CDPN-L, where S represents five and L represents ten 2D residual blocks, respectively. Each residual block comprises two convolutional layers with ReLU activation in between. The convolutional layer has 128 channels and a 3×3 filter size. This method uses sub-pixel convolution to increase the resolution of the LR features to HR [35]. At the starting time of step t0, the previous estimation is set to zero. The learning rate starts at 1×10–4 and decreases by 0.1 every 60 epochs until 70 epochs to train the CDPN models. The models are trained using a pixel-wise loss function and an Adam [19] optimizer with parameters β1 = 0.9, β2 = 0.999, and a weight decay of 5×10–4. All experiments were conducted using Python 3.6.4 and Pytorch 1.1.

This work comprehensively studies and compares two temporal modeling methods, including 2D CNN and RNN. The proposed model specification comparison is shown in Table 1, with their respective residual blocks. The authors utilized 5 and 10 residual blocks as the hidden states S and L, respectively.

The results are measured using the luminance (Y) channel, with the L1 loss applied to all pixels between the ground truth frames and the network output. Further, PSNR (dB) values of VID4, SPMC and UDM10 benchmark datasets of these temporal modeling methods are given in Table 2.

This section validates CDPN with other state-of-the-art VSR methods to demonstrate its effectiveness, including Bicubic, FRVSR [8], DUF [23], RBPN [22], and RLSP [21]. For intense comparison, this work uses both explicit and implicit MC methods.

1) Quantitative comparison: Table 3 compares the proposed method quantitatively with state-of-the-art video super-resolution methods with and without alignment methods, such as CNN and RNN. For a fair comparison of reported methods, this work utilized three benchmark datasets with a scale factor of ×4. Specifically, comparing the recovered results at ×4 scale on the VID4, SPMC, and UDM10 datasets wrt to RLSP method, the proposed CDPN show an improvement of 0.43 dB, 0.78 dB, and 0.84 dB, respectively, in terms of PSNR. RLSP also passes historical information like the CDPN method without ME in feature space. The main reason for achieving this optimal performance is improved feature extraction from conformation detail information such as repetitive patterns, edges, and textures. In many VSR methods, recovering this information is a difficult task. Moreover, the conformation detail-learning enables the network to focus on information detail learning of the original image and obtains satisfying results. The RRN with residual learning focuses on learning abundant local features in images and achieves a high PSNR.

2) Qualitative Comparison: Most existing VSR methods, such as FRVSR, DUF, RBPN, and RLSP, focused on PSNR-based results and not perceptual details [8, 21–23]. The proposed temporal method reconstructs consistent frames with fewer flickering artefacts than other VSR methods. Figure 4 shows the superior visual quality of our CDPN in qualitative results.

OPEN IN VIEWER

Fig. 4

Qualitative comparison on the VID4, SPMC and UDM10 datasets for 4× VSR.

All dataset consists of rich information in scenes like sharp edges and fine texture with fewer artefacts. The proposed method produces sharper edges in the calendar scene, the fine texture of the RMVTG_011 scene, and a clear pattern in the auditorium scene. All dataset consists of rich information in scenes like sharp edges and fine texture with fewer artefacts. The proposed method produces sharper edges in the calendar scene, the fine texture of the RMVTG_011 scene, and a clear pattern in the auditorium scene. In contrast, it is difficult for the remaining methods to recover the high-definition outcomes as they suffer from unpleasant blurring artefacts and unclear structure. The proposed CDPN method recovered and obtained sharper results by detailed conformation. The results indicate that the proposed method generates more realistic and natural outputs than other methods.

3) Model Parameter Size Comparison: Table 4 compares the proposed and state-of-the-art method parameters. The proposed method produces better results with fewer parameters, resulting in the highest efficiency in terms of parameters compared to other methods, demonstrating exceptional performance with less computational overload.

This ablation study works in two models of 5 L and 10 L residual blocks. Table 5 displays the results of the ablation study performed on the surveillance dataset for a scale factor of 4 to evaluate the impact of conformation details.

For investing the impact of the kernel on the given dataset, this work used kernel_1 and kernel_3 on Model1, Model2, Model3, and Model4, respectively. Model 3 and Model 4 constructed a better image and high performance with more residual blocks compared to Model 1 and Model 2. Figure 5 shows the further analysis of the qualitative comparison of surveillance data with different residual blocks as 5 L and 10 L.

OPEN IN VIEWER

Fig. 5

Qualitative comparison of surveillance dataset for 4× VSR.

Moreover, this work plotted the PSNR with time to examine the information flow between various temporal modeling methods. Figure 6 shows the video series of the calendar sequence in VID4, RMVTG_011 sequences in SPMC, and auditorium sequences in UDM10. The method without conformation details falls overdue after a few frames. The proposed method outperforms the other method by incorporating conformation details, allowing for information accumulation over time.

OPEN IN VIEWER

Fig. 6

Information flow for VID4, SPMC, UDM10, and Surveillance data on sequences over time.

More interestingly, the CDPN-based method keeps improving while, without conformation detailing, the method suffers from performance degradation. Information from a previously hidden state is complementary to restoring missing details. In addition, this work also plotted surveillance data information flow with and without conformation details, demonstrating high performance. Through the above experimental results and analyses, the findings of this work are that both the network architecture and the proposed loss contribute to the visual improvements and that the network architecture plays a crucial role in achieving high performance.