Edge-preserving filter with adaptive L 0 gradient optimization

Abstract

Wireless multimedia sensor networks have recently emerged as one of the most important technologies to actively perceive physical world and empower a wide spectrum of potential applications in various areas. Due to the advantages of rapid deployment, flexible networking, and multimedia information perceiving, wireless multimedia sensor networks are suitable for transmitting mass multimedia data such as audio, video, and images. Two-dimensional images are among the nuclear ways to convey certain information, and there exists a large number of image data to be processed and transmitted; however, the complexity of environment and the instability of sensing component both can give rise to the insignificant information of the resulted images. Hence, image processing attracts a lot of research concerns in last several decades. Our concern in this article is filtering technology on image signal. Filtering is shown to be a key technique to ensure the validity and reliability of the wireless multimedia sensor networks images, which aims to preserve salient edges and remove low-amplitude structures. The well-known L₀ gradient minimization employs L₀ norm as gradient sparsity prior, and it is capable of preserving sharp edges. Similar to the total variation model, L₀ gradient minimization may easily suffer from the staircase effect and even lose part of the structure. Therefore, in this article, we propose an edge-preserving filter with adaptive L₀ gradient optimization. Different from original L₀ gradient minimization, we introduce an adaptive L₀ regularization. The newly proposed adaptive function is feature-driven and makes the utmost of the image gradient, enabling the filter to remove low-amplitude structures and preserve key edges. Furthermore, the proposed filter can effectively avoid staircase effect and is robust to noise. We develop an efficient optimization algorithm to solve the proposed model based on alternating minimization. Through extensive experiments, our method shows many attractive properties like preserving meaningful edges, avoiding staircase effect, robustness to noise, and so on. Applications including noise reduction, clip-art compression artifact removal, detail enhancement and edge extraction, image abstraction and pencil sketching, and high dynamic range tone mapping further demonstrate the effectiveness of the proposed method.

Keywords

Wireless multimedia sensor networks image filtering adaptive edge preserving

Introduction

Wireless multimedia sensor networks (WMSNs) are composed of a large number of wirelessly interconnected sensor nodes equipped with multimedia devices, which can support the multimedia data collection, processing, and transmission.¹ The introduction of inexpensive CCD cameras and CMOS sensors has made it possible to transmit multimedia data and to foster the development of WMSNs.^2,3 They are evolved from the traditional wireless sensor networks (WSNs) gradually and have extensive utilization fields, such as multimedia surveillance sensor networks, environmental monitoring, advanced health care delivery, and traffic congestion control.⁴ WMSNs have been successfully applied in monitoring system, but still face challenges in the video image processing. Due to the limitations of the monitoring environment and the technological conditions of WMSNs, the monitoring system may generate unsatisfied video images in practice. The instability of the image quality leads to some difficulties in identification, forensics, and incident analysis. This means such video images are unreliable. Therefore, researches on image filtering and processing, aiming to obtain clear images with sharp edge and good visual impression, are of great significance. Generally, the output video images of WMSNs are diverse and need various techniques for different practice purposes.

The relevant tasks mainly belong to computer vision community, including but not limited to image denoising, detail enhancement, edge extraction, and high dynamic range (HDR) mapping. Most of these tasks involve the essential image filtering technology. Images usually contain rich well-structured information, such as edges. How to effectively diminish salient edges from meaningless details is a key problem in image processing. Edge-preserving filters have been proposed in favor of their validity to image structures, thus becoming the fundamental tools in a wide range of applications. Actually, the edge-preserving filters are extremely useful for characterizing and enhancing image edges. An image can also be decomposed into structural and detailed elements by edge-preserving filter. Recently, numerous edge-preserving filters have been proposed via different strategies and these filters have the same purpose of removing weak edges while preserving strong ones.

Most early proposed image filters are widely employed to reduce noise and/or extract useful image structures. Bilateral filter (BF)⁵ is well applicable for removing noise-like structures and extracting details at a fine-spatial scale. While this filter is effective in many situations, it may generate gradient reversal artifacts^6–8 near the edges. The efficiency is also a problem especially when the kernel radius is large. To overcome the limitation of gradient reversal, guided filter (GF) has been proposed; this filter performs better near edges without generating gradient reversal artifacts. However, halo artifacts may be introduced in the filter output. The weighted least squares (WLS) filter⁸ is a robust method with flexible optimization framework. While this method often generates high-quality results with halo-free, it comes with the price of long computational time. Total variation (TV)⁹ is another type of edge-preserving regularization. Recently, various TV-based models^10–14 have been developed and achieve better results in a great variety of computer vision; these TV models have the common limitation of staircase effect. Moreover, as the improvement of the classical TV method,⁹ $L_{0}$ gradient minimization¹⁵ performs more effectively than most TV-based methods and also suffers from staircase effect and over-sharpened artifacts.

Motivated by the success of the $L_{0}$ gradient minimization,¹⁵ we propose an adaptive $L_{0}$ gradient optimization image filter and introduce an additional pre-smoothing step to smooth the gradient of the input image. We solve the proposed method by alternating minimization,¹⁶ which has fast convergence property. Experimental results demonstrate that our method presents better performance than the state-of-the-art methods. The contributions of our work can be summarized as follows:

We propose a novel energy function using $L_{1}$ fidelity with adaptive $L_{0}$ penalty. The adaptive weight function designed in our paper is driven by the feature of the input image, thus improving the capability of removing details and preserving significant structures. In addition, $L_{1}$ fidelity is employed to enhance the robustness to noise.

We introduce an additional pre-smoothing step to smooth the gradient of the input image, which helps effectively avoid intermediate errors introduced during the alternate iteration. This pre-smoothing step is critical to avoid staircase effect.

Our method shows many attractive properties, for example, robustness to noise, avoiding staircase effect, preserving meaningful edges, avoiding visual artifacts. And our method helps efficiently empower applications like noise reduction, clip-art compression artifact removal, detail enhancement and edge extraction, image abstraction and pencil sketching, HDR tone mapping, and so on.

Related work

There are lots of researches on image processing technology of the WMSNs images and filtering is the foundation of image processing. In this section, we discuss some commonly used image filters.

Two main branches of edge-preserving image filters are average-based filters and optimization-based approaches. Average-based filtering methods process images through weighted average, which define different types of affinity between neighboring pixel pairs by considering intensity difference. The popular BF⁵ together and its extensions^6,17–20 belong to the category of average-based filtering. The key point of BF is to average the neighbor pixels by computing weights based on information of both spatial and range domain. BF can effectively preserve sharp edges and flatten small details. BF has benefited a wide range of applications, for example, image denoising,²¹ image abstraction,²² detail decomposition,²³ and image defogging.²⁴ Inevitably, BF may generate the gradient reversal artifacts near the edges. Another commonly used average-based filter is anisotropic diffusion.²⁵ It employs an edge-stopping technique to avoid smoothing strong edges. Based on the assumption that the filtered output is a local linear transformation of the guidance image, GF²⁶ is proposed. While GF has similar edge-preserving capability with BF, it can further generate artifact-free results. These average-based filters have the same limitation that edges or structures may be blurred or destroyed.

Optimization-based approaches often optimize energy functions to constrain the filtered output to be as smooth as possible except where large gradients exist in the input image, such as TV.⁹ The classical TV method⁹ penalizes large gradient magnitudes by utilizing $L_{1}$ norm regularization term. Subsequently, a series of optimization-based filters have been proposed, for example, WLS⁸ and local extrema (LE).²⁷ WLS method is able to avoid visual artifacts and is more adaptive than most of average-based filtering methods. LE method allows fine scale details to be extracted regardless of contrast. They are also successfully applied to multi-scale decomposition. According to the diverse structural properties, various TV-based methods^10–14 are proposed and they extremely extend the application range, such as TV-L₁,¹⁰ TV-G,²⁸ TV-Gabor.²⁹ Aujol et al.¹¹ analyzed four different decomposition models and demonstrated that TV-L₂⁹ and TV-G²⁸ model fit well with no a-priori knowledge of the texture. As the improved TV-based methods transform piecewise smooth function into piecewise constant function, TV-based methods always suffer from staircase effect near edges. Chan et al.³⁰ proposed a high order energy model to overcome staircase effect caused by edge enhancing. Xu et al.¹⁵ employed $L_{0}$ norm instead of $L_{1}$ norm to directly constrain the gradient sparsity and present a powerful smoothing method, that is, $L_{0}$ gradient minimization.¹⁵ Compared with Rudin et al.,⁹ the method in Xu et al.¹⁵ can generate more sparse solution. $L_{0}$ gradient minimization¹⁵ also performs more effectively than most TV-based methods. However, it may suffer from staircase effect and over-sharpened artifacts. Recently, a number of improved approaches based on $L_{0}$ smoothing were proposed. Instead of $L_{2}$ fidelity term, Shen et al.³¹ proposed to adopt the $L_{1}$ fidelity term to cope with inaccurate measurement of Xu et al.¹⁵ In addition, the advantages of using $L_{1}$ norm fidelity term can be found in Shen et al.³¹ and Liu et al.³² Unfortunately, for the above limitations encountered in the $L_{0}$ gradient minimization,¹⁵ these methods do not work. To solve the problem of losing structure, Cheng et al.³³ proposed an approximation algorithm for $L_{0}$ gradient minimization. However, due to the high consistency of the local pixels, more staircase effect may be generated in the filtered output.

Adaptive and sparse L₀ gradient optimization

We will first briefly review the $L_{0}$ gradient minimization¹⁵ in section “ $L_{0}$ gradient minimization.” Then we present the formulations of the proposed approach based on $L_{0}$ gradient optimization framework in section “Adaptive $L_{0}$ gradient minimization model.” Finally we put forward the solution procedures of the proposed energy function in section “Optimization.”

L ₀ gradient minimization

The $L_{0}$ gradient minimization model proposed by Xu et al.¹⁵ is defined as

S = \arg min_{S} {\sum_{i} {(U (i) - S (i))}^{2} + λ \cdot C (S)}

(1)

where U is the input image, S is the desired output, i is the pixel index, $λ$ is a positive smoothing parameter, $\nabla S (i) = (\partial_{x} S (i), \partial_{y} S (i))^{T}$ , and gradient measure is defined as

C (S) = # {i | | \partial_{x} S (i) | + | \partial_{y} S (i) | \neq 0}

(2)

where $# {\cdot}$ is a counting operator and it denotes the number of pixels whose gradient is non-zero. $∥ \nabla S ∥_{0}$ measures the sparsity of image gradient. In equation (1), the first term constrains that the output image should have similar structures with the input image, and the second term denotes the number of non-zero gradient value, which measures the sparsity.

As the $L_{0}$ norm term contained in equation (2) involves a non-convex discrete counting metric, equation (1) is difficult to optimize directly. Xu et al.¹⁵ propose to solve equation (1) by alternating minimization.¹⁶ To avoid staircase effect and over-sharpening limitations of Xu et al.,¹⁵ we propose an adaptive $L_{0}$ gradient optimization which is introduced in section “Adaptive $L_{0}$ gradient minimization model.”

Adaptive L₀ gradient minimization model

In this article, we propose an adaptive optimization-based image filter and employ an adaptive local function $θ$ . Specifically, given an image U, the output image S is solved by

min_{S} {\sum_{i} (| U (i) - S (i) | + λ θ (i) C (S))}

(3)

where $λ$ is a non-negative parameter and $θ$ is the adaptive local function, which is defined as

θ (i) = \exp (- ∥ r (i) ∥^{0.8})

(4)

where

r (i) = \frac{‖ \sum_{j \in N_{ε} (i)} \nabla U (j) ‖_{2}^{2}}{\sum_{j \in N_{ε} (i)} ‖ \nabla U (j) ‖_{2}^{2} + 0.5}

$N_{ε} (i)$ is a $ε \times ε$ window centered at pixel i , and j is the pixel index in $N_{ε} (i)$ .

We point out that the map r is proposed by Xu and Jia³⁴ to preserve salient edges and remove some narrow strips. As analyzed in Xu and Jia,³⁴ a large value of r indicates that there exist strong image structures in the local window, whereas a small value of r indicates that the area is flat.

Optimization

It is difficult to solve the energy function in equation (3) as it involves a hybrid norm. We use alternating minimization proposed by Wang et al.¹⁶ to solve in equation (3). By introducing new auxiliary variables $η$ , h, and v, we rewrite the energy function equation (3) as

\begin{matrix} min_{S, h, v, η} {\sum_{i} α (U (i) - S (i) - η (i))^{2} + | η (i) | + λ θ (i) C (h, v) \\ + β ((h (i) - \partial_{x} S (i {))}^{2} + (v (i) - \partial_{y} S (i))^{2})} \end{matrix}

(5)

where $α$ and $β$ are positive parameters, and solvers of equation (5) converge to equation (3) when $α$ , $β \to \infty$ . The optimization problem (equation (5)) can be divided into four tractable subproblems as follows

\min_{S} {\sum_{i} α {(U (i) - S (i) - η (i))}^{2} + β ((h (i) - \partial_{x} S (i {))}^{2} + {(v (i) - \partial_{y} S (i))}^{2})}

(6)

min_{η} {\sum_{i} α {(U (i) - S (i) - η (i))}^{2} + | η (i) |}

(7)

\begin{matrix} min_{h} {\sum_{i} λ θ (i) ∥ h ∥_{0} + β {(h (i) - \partial_{x} S (i))}^{2}} \end{matrix}

(8)

\begin{matrix} min_{v} {\sum_{i} λ θ (i) ∥ v ∥_{0} + β {(v (i) - \partial_{y} S (i))}^{2}} \end{matrix}

(9)

Solving S

As equation (6) is a least squares problem, we can obtain its closed-form solution based on the convolution theorem of Fourier transformations. By utilizing the fast Fourier transformation (FFT) to speed up the whole process, the closed-form solution of equation (6) is

S = F^{- 1} (\frac{α F (U) + α F (η) + β (\bar{F (\partial_{x})} F (h) + \bar{F (\partial_{y})} F (v))}{α F (1) + β (\bar{F (\partial_{x})} F (\partial_{x}) + \bar{F (\partial_{y})} F (\partial_{y}))})

(10)

where $F (\cdot)$ and $F^{- 1} (\cdot)$ denote the FFT and the inverse FFT, and $\bar{F (\cdot)}$ denotes the complex conjugate of $F (\cdot)$ .

Solving $η$

Given S, we can solve equation (7) by matrix calculus. Similar to Wang et al.,¹⁶ the solver of equation (7) can be given by

η (i) = ζ \cdot sgn (U (i) - S (i))

(11)

where $ζ (i) = \max {| U (i) - S (i) | - (1 / 2 α), 0}$ , $sgn (\cdot)$ is a sign function, and $α$ is a parameter.

Solving h

Problem (8) can be solved by $L_{0}$ minimization directly. However, the result is sensitive to noise-like structures. Thus, we introduce an additional pre-smoothing step. To remove small-scale details in $\partial_{x} S (i)$ , a smooth version of $\partial_{x} S (i)$ is pre-computed by

\tilde{S_{x} (i)} = \arg min_{\tilde{S_{x} (i)}} {\sum_{i} (\tilde{S_{x} (i}) - \partial_{x} S (i {))}^{2} + χ_{1} ∥ \nabla \tilde{S_{x}} ∥_{0}}

(12)

where $χ_{1}$ is a positive smoothing parameter. equation (12) is a $L_{0}$ gradient minimization¹⁵ and easy to solve. It is unnecessary to go into details here. Given $\tilde{S_{x}} (i)$ , we rewrite equation (8) and solve h by

\begin{matrix} min_{h} {\sum_{i} β (h (i) - \tilde{S_{x} (i} {))}^{2} + λ θ (i) ∥ h ∥_{0}} \end{matrix}

(13)

The shrinkage formula is utilized to obtain the closed-form solution h, which is given by

\begin{matrix} h (i) = {\begin{matrix} \tilde{S_{x} (i)}, & {(\tilde{S_{x} (i)})}^{2} \geq \frac{χ_{1} θ (i)}{β} \\ 0, & otherwise \end{matrix} \end{matrix}

(14)

Solving v

Similarly, a smooth version of $\partial_{y} S (i)$ can be computed by

\tilde{S_{y} (i)} = \arg min_{\tilde{S_{y} (i)}} {\sum_{i} {(\tilde{S_{y} (i)} - \partial_{y} S (i))}^{2} + χ_{2} ∥ \nabla \tilde{S_{y}} ∥_{0}}

(15)

where $χ_{2}$ is a positive smoothing parameter. equation (15) is also a $L_{0}$ gradient minimization¹⁵ and easy to solve. Given $\tilde{S_{y}} (i)$ , we rewrite equation (9) and solve v by

\begin{matrix} min_{v} {\sum_{i} λ θ (i) ∥ v ∥_{0} + β (v (i) - \tilde{S_{y} (i} {))}^{2}} \end{matrix}

(16)

The shrinkage formula is utilized to obtain the closed-form solution v, which is given by

\begin{matrix} v (i) = {\begin{matrix} \tilde{S_{y} (i)}, & (\tilde{S_{y} (i} {))}^{2} \geq \frac{χ_{2} θ (i)}{β} \\ 0, & otherwise \end{matrix} \end{matrix}

(17)

Algorithm 1 summarizes the proposed alternating minimization algorithm.

Algorithm 1. Adaptive $L_{0}$ gradient optimization.
Input: image U, parameters $λ$ , $α$ , $κ$ and $β_{\max}$ .
Initialize $S = U$ , $β_{0} = λ$ , $χ_{1} = χ_{2} = β_{0}^{2}$ , $t = 1$ .
while $β < β_{\max}$ do
solve $η^{t}$ by equation (11).
solve ${\tilde{S_{x}}}^{t}, {\tilde{S_{y}}}^{t}$ by minimizing equations (12) and (15).
solve $h^{t}$ and $v^{t}$ by equations (14) and (17).
solve $S^{t + 1}$ by equation (10).
update parameter $β \to κ β$ .
end while
Output: the filtered image S.

Analysis and discussion

In this section, we provide more insights and analysis on how the proposed algorithm performs on image filtering and demonstrate the function of each component of the algorithm separately.

Local weighted function $θ$

The regularization term usually plays a critical role in the optimization-based filtering methods. In this article, we propose an adaptive regularization, which involves an adaptive function $θ$ . As mentioned in section “Adaptive $L_{0}$ gradient minimization model,” $θ$ relies on another local gradient measurement r, which is used to measure the usefulness of image edges in motion deblurring.³⁴ By performing thresholding on r, narrow strips and tiny edges can be successfully removed. In the proposed model, we employ $\exp (- ∥ r ∥^{0.8})$ to form an adaptive function $θ$ . Indeed, $θ$ works as a structure descriptor and r is a pixel-wise adaptive map. Figure 1 visualizes r and $θ$ of a synthesized image. The noisy input image shown in Figure 1(a) is roughly piecewise constant and contains several step edges of different magnitudes. Figure 1(b) is the visualization of r, which shows that step edges imply large r. Correspondingly, the adaptive function $θ$ is small on step edges and large in roughly flat regions, as can be seen in Figure 1(c).

Figure 1.

The visualization of (b) r map and (c) adaptive weight function $θ$ of the input image (a).

In the proposed method, the adaptive function $θ$ is capable of removing littery details. To better illustrate the effectiveness of $θ$ , we show an example in Figure 2. Figure 2(a) is the input image. Figure 2(b) is generated by the proposed method without computing $θ$ , that is, $θ (i) = 1$ for each pixel. Figure 2(c) is the filtered output of our method. Figure 2(b) contains some small-scale details such as the floor’s texture. It can be seen that our method performs better.

Figure 2.

The function of $θ$ in removing tiny edges. (a) Input image. (b) Result without computing $θ$ . (c) Ours with $θ$ . Here, λ = 0.012, $κ = 2$ for (b) and (c).

Moreover, we find that $θ$ also helps to preserve middle-scale structures, as shown in Figure 3. Figure 3(a) is a 1D input signal with both large-scale structures and middle-scale structures. Figure 3(b) is generated by the proposed method without computing $θ$ . Figure 3(c) is the filtered output of our method. The method without computing $θ$ loses middle-scale structures. In contrast, our method preserves middle-scale structures adaptively, as shown in Figure 3(c).

Figure 3.

Comparisons of various smoothing methods on a 1D signal. Black curves represent the noisy input; red curves represent the smoothing results. (a) Input noisy 1D signal. (b) Without computing $θ$ . (c) Ours. For fair comparisons, we set λ = 0.0012, $κ = 2$ .

The effectiveness of pre-smoothing

In the proposed method, we use $(\tilde{S_{x}}, \tilde{S_{y}})$ instead of $(\partial_{x} S, \partial_{y} S)$ to solve the auxiliary variables $(h, v)$ . We find that this pre-smoothing step is very critical to avoid staircase effect. The efficient iterative algorithm enhances the smoothing ability and reduces the staircase effect. Figure 4 shows that our method can avoid staircase effect. Figure 4(a) is a natural image with analogous texture and inhomogeneous color. Figure 4(b) is generated by the proposed method without pre-smoothing step and Figure 4(c) is the filtered output of our method. Compared with our method, the method without pre-smoothing step generates staircase effect. Our method avoids staircase effect effectively.

Figure 4.

The effectiveness of pre-smoothing. (a) Input image. (b) Result of without pre-smoothing step. (c) Our method. (d)–(f) Close-ups of corresponding images in (a)–(c). Here, $λ = 0.0015, κ = 2$ for (b) and (c).

The effectiveness of $L_{1}$ fidelity term

As mentioned in Shen et al.,³¹ the $L_{2}$ fidelity term is less effective than $L_{1}$ fidelity term when the erroneous measurements exist. To verify the effectiveness of the $L_{1}$ fidelity, we show an example in Figure 5. Figure 5(a) shows a noisy image. Figure 5(b) is generated by $L_{2}$ fidelity term. Figure 5(c) is generated by the method of Shen et al.³¹ Figure 5(d) shows the filtered output of our method. As can be seen, our method is effective to removing noise-like structures and generates a much smoother result.

Figure 5.

Our method is effective to remove noise-like structures. (a) Input noisy image. (b) The method using $L_{2}$ fidelity term.(c) The method of Shen et al.³¹ using $L_{1}$ fidelity term. (d) Our method with $L_{1}$ fidelity term. We set $λ = 0.03$ , $κ = 2$ for (b)–(d). There are many residual noises in (b) and (c).

Experiments and applications

In this section, we apply the proposed filter to several applications in image editing, including noise removal, clip-art compression artifact removal, detail enhancement and edge extraction, image abstraction and pencil sketching, and HDR tone mapping.

Parameter analysis and settings

We first provide some analysis on the role of parameters. The key parameter $λ$ is the weight on the adaptive regularized term, and it is critical to effectively control the level of structure coarseness. Larger $λ$ usually results in a more smoothing output. On the contrary, if it is too small, the output image may be very similar to the input image. Figure 6 shows a visual example with results from varying values of $λ$ . $α$ and $β$ are introduced using alternating minimization technique. As $α$ controls the weight of data term when computing $η$ using equation (7), it is not suggested to be too large to avoid over fitting. By contrast, $β$ varies during each iteration. It is increasing at the rate of $κ$ . $ε$ is the window size when computing the adaptive function $θ$ . The choice of $ε$ is followed by Xu and Jia.³⁴ $χ_{1}$ and $χ_{2}$ are both introduced for solving h and v. They are regularization parameters in equations (12) and (15) for pre-smoothing on $\partial_{x} S$ and $\partial_{y} S$ . Thus, it should be set differently for different images.

Figure 6.

Effects of varying $λ$ settings on smoothing results of the proposed filter. Larger $λ$ leads to smoother output. For fair comparisons, we set $κ = 2$ for (b)–(e). (a) Input image, (b) λ = 0.001, (c) λ = 0.008, (d) λ = 0.015, and (e) λ = 0.03.

In all experiments, we empirically set $β_{\max} = 100, 000$ , $α = 1.1$ , $κ = 2$ , $ε = 7$ , $β_{0} = λ$ , $χ_{1} = χ_{2} = β_{0}^{2}$ . The range of $λ$ is $[0.001, 0.05]$ . Most computation is spent on FFT in equation (10) and pixel-wise algebraic operations in equations (12), (13), (15), and (16). It takes about 5 s to process a single-channel $600 \times 450$ image on an Intel Xeon CPU @ 2.53G with our MATLAB implementation.

Image denoising

As a smoothing image filter, the proposed method can be applied to image denoising. Figure 7 shows an example compared with commonly used image filters including relative total variation (RTV),³⁵ WLS,⁸ rolling guidance filter (RGF),³⁶ and $L_{0}$ gradient minimization ( $L_{0}$ ).¹⁵ Figure 7(a) is a noisy image. It can be seen that results of RTV, WLS, and RGF shown in Figure 7(b)–(d) still contain heavy noises and lose some salient structures. As shown in Figure 7(e), $L_{0}$ performs better than RTV, RGF, and WLS. However, it also fails to remove all the noises. Compared with these methods, our method generates result with clean structures and without noise residual, as shown in Figure 7(f).

Figure 7.

Comparisons of various smoothing methods on a noisy image. Results shown in (b)–(e) are generated by RTV,³⁵ WLS,⁸ RGF,³⁶ and $L_{0}$ .¹⁵ The parameters of relative total variation are $λ = 0.01$ , $σ = 3$ , the parameters of WLS are $λ = 1$ , $α = 1.2$ , the parameters of RGF are $σ_{s} = 3$ , $σ_{r} = 0.1$ , the parameters of $L_{0}$ and the proposed method are $λ = 0.006$ , $κ = 2$ . (a) Input, (b) RTV,³⁵ (c) WLS,⁸ (d) RGF,³⁶ (e) L₀,¹⁵ and (f) ours.

Clip-art compression artifact removal

Cartoon-like images are heavily compressed and contain severe compression artifacts. Xu et al.¹⁵ analyze that it is difficult to generate satisfied output as the compression artifact is highly dependent on edges. The edge-preserving filters can properly deal with this matter and do not require any learning process. As our method is capable of preserving salient edges, the proposed smoothing method is also suitable for clip-art compression artifact removal. We make a comparison with a set of edge-preserving methods, including BF,⁵ GF,²⁶ WLS,⁸ RTV,³⁵ RGF,³⁶ and $L_{0}$ .¹⁵ As shown in Figure 8(a), the input image contains some acute edges, which are difficult to deal with. Results of BF, WLS, and $L_{0}$ shown in Figures 8(d) and (f) and 7(i) fail to remove traces of compression. Results of GF, RTV, and RGF³⁶ shown in Figure 8(e), (g), and (h) blur the edges and lose some salient structures. Our method shown in Figure 8(j) removes the compression artifacts more successfully as compared to other methods.

Figure 8.

Comparisons on compression artifact removal for a cartoon image. (a) Input image. (b) Result of our method. (c)–(j) Close-ups of the input and results of BF,⁵ GF,²⁶ WLS,⁸ RTV,³⁵ RGF,³⁶ $L_{0}$ ,¹⁵ and our method.

Detail enhancement and edge extraction

Detail enhancement is the process of representing details in a magnified way. We briefly introduce the detail enhancement algorithm,²⁶ which formulates the enhanced image as

E = B + τ D

(18)

where B denotes the image base layer, D denotes the image detail layer, $τ$ is a parameter that controls the magnification, and E denotes the enhanced image. Generally, the base layer B can be generated by edge-preserving filter. A good base layer is very critical to the final enhanced result. If the base layer is not consistent with the input image, the gradient reversal may appear around edges, which is a problem encountered by BF.⁵ WLS⁸ is able to avoid gradient reversal safely. However, it is usually time-consuming to compute a five-point spatially inhomogeneous laplacian matrix for WLS.

The proposed method can be applied to detail enhancement based on the decomposition in equation (18). We show an example of detail enhancement in Figure 9. Figure 9(a) is a natural image. Result of BF⁵ shown in Figure 9(b) displays the gradient reversal artifacts. Figure 9(c) shows the result of $L_{0}$ gradient minimization.¹⁵ Compared with $L_{0}$ gradient minimization, our result contains more rich details and seems vividly as shown in Figure 9(d).

Figure 9.

Detail enhancement. The parameters of BF are $σ_{s} = 10$ , $σ_{r} = 0.2$ . We set $λ = 0.001, κ = 2$ for (b) and (c). For (b)–(d), we set $τ = 5$ . (a) Input, (b) BF,⁵ (c) L₀,¹⁵ and (d) ours.

Edge extraction is a basic pre-processor of natural image editing and high-level structure inference. It is also a challenging problem because most edge detectors are sensitive to complex structures and unavoidable noises. As analyzed in section “Local weighted function $θ$ ,” the proposed method is capable of suppressing low-amplitude details and preserving the significative structures. An example is shown in Figure 10. Figure 10(a) is a challenging input image and its gradient map is shown in Figure 10(d). Figure 10(g) is the edge extraction result of Figure 10(d) by canny edge detector. Figure 10(b) and (c) shows the corresponding results of $L_{0}$ gradient minimization¹⁵ and the proposed method. Figure 10(e) and (f) shows the corresponding gradient maps of Figure 10(b) and (c) with linear enhancement for visualization. Figure 10(h) and (i) shows the corresponding edge maps of Figure 10(b) and (c). Compared with $L_{0}$ gradient minimization,¹⁵ the proposed method preserves most significant structures (such as the woman’s shirt and the boundary of clouds), and the filtered output is more reliable.

Figure 10.

Edge enhancement and extraction. The parameters are $λ = 0.045, κ = 2$ for (b) and (c). Our method suppresses low-amplitude details and preserves high contrast edges. (a) Input, (b) L₀,¹⁵ (c) ours, (d) gradient map of (a), (e) gradient map of (b), (f) gradient map of (c), (g) edge map of (a), (h) edge map of (b), and (i) edge map of (c).

Image abstraction and pencil sketching

Usually, the visual perception of an image can be enriched with textures and details. In some cases, we prefer simplified stylistic pictures which are non-photorealistic abstraction with suppressing details and emphasizing edges from color images or videos. Conventional image abstraction methods contain two main steps—image smoothing step and edge component detecting step. To increase the visual distinctiveness of different regions, the extracted lines are added back to the smoothed image. Based on the video abstraction framework of Winnemöller et al.,²² we apply the proposed method to image abstraction task. Figure 11 shows an example. The result of $L_{0}$ gradient minimization¹⁵ is shown in Figure 11(b). Our method performs better than Xu et al.¹⁵ We also show pencil sketching results of Figure 11(a)–(c) in Figure 11(d)–(f). As can be seen, our pencil sketching result shown in Figure 11(f) contains more details than that of Xu et al.¹⁵

Figure 11.

Image abstraction and pencil sketching results. For fair comparisons, we set the parameters λ = 0.003, $κ = 2$ for (b) and (c). Our method removes the least important structures and generates a more hierarchical result. (a) Input, (b) L₀,¹⁵ (c) ours, (d) pencil stylization of (a), (e) pencil stylization of (b), and (f) pencil stylization of (c).

HDR tone mapping

HDR mapping is another popular image editing task which aims to generate low dynamic range images by compressing the base layer to some level. The base layer should be smooth enough to generate reasonable contrast maintenance in range compression. Within the tone mapping framework of Durand and Dorsey,⁶ the proposed filter can be used to generate the base layer. We show a tone mapping example in Figure 12 and compare it with some other commonly used methods. We first convert the input HDR radiance to a logarithmic scale (log10). Then we map the result to $[0, 1]$ and display it in Figure 12(a). The result of WLS⁸ shown in Figure 11(b) contains some dark regions. A blocky reflection also appears on the floor. The result of $L_{0}$ ¹⁵ is shown in Figure 12(c). As can be seen, our method shown in Figure 12(d) contains more details than Figure 12(b) and (c).

Figure 12.

Comparisons of HDR tone mapping using different methods. Top row: results by several different filtering methods. Bottom row are close-ups of corresponding images in the top row. The parameters of WLS are $λ = 20$ , $α = 1.2$ . We set the parameters λ = 0.003, $κ = 2$ for both (c) and (d) for fair comparison. (a) Input, (b) WLS,⁸ (c) L₀,¹⁵ and (d) ours.

Conclusion

In this article, to enrich the image processing technology of WMSNs images, we present an adaptive image filter based on $L_{0}$ gradient minimization framework. We utilize an adaptive function to facilitate detail removal and preservation of salient edges. We also develop an efficient iterative algorithm to solve the proposed model. The proposed filter behaves robustly to noise and avoids staircase effect effectively. Experimental results on several computer vision tasks show that the proposed method performs favorably against the state-of-the-art filtering methods. It is also appreciated to extend our method to more applications.

Our method is adaptive to preserve middle-scale details and remove small-scale details without employing prior texture information. If the structures and textures of the image are visually similar in scales, it is hard to distinguish between structures and textures; thus, part of the structures can be easily mistaken as textures. As can be seen from Figure 13, Figure 13(a) is a input image with scale and appearance similar to the underlying textures. Since the input image contains textures with strong contrast and the underlying textures are exceedingly too close to the scale and shape of these edges, the visually filtering output of our method is shown in Figure 13(b). As can be seen, our method is failed to preserve all of the structures.

Figure 13.

Difficult examples. Our method is failed to extract structures whose scale and appearance are similar to the textures. We set the parameters λ = 0.0015, $κ = 2$ . (a) Input and (b) ours.

Footnotes

Handling Editor: Eleonora Borgia

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work has been partially supported by National Natural Science Foundation of China (no. 61572099), Fundamental Research Funds for the Central Universities (no. 3132018192), and High-Tech Ship Research Program Support Project.

ORCID iD

Wanshu Fan

References

Akyildiz

Melodia

Chowdhury

KR.

A survey on wireless multimedia sensor networks. Comput Netw 2007; 51(4): 921–960.

Han

Sun

Xiao

et al . A camera nodes correlation model based on 3D sensing in wireless multimedia sensor networks. Int J Distrib Sens Netw 2012; 8(11): 154602.

Gao

Wen

Zhao

et al . Secure data aggregation in wireless multimedia sensor networks based on similarity matching. Int J Distrib Sens Netw 2014; 10: 494853.

Wang

A flexible and reliable traffic control protocol for wireless multimedia sensor networks. Int J Distrib Sens Netw 2014; 10: 102742.

Tomasi

Manduchi

. Bilateral filtering for gray and color images. In: Proceedings of the sixth international conference on computer vision—ICCV, Bombay, India, 7 January 1998, pp.839–846. New York: IEEE.

Durand

Dorsey

Fast bilateral filtering for the display of high-dynamic-range images. ACM T Graphic 2002; 21(3): 257–266.

Bae

Paris

Durand

Two-scale tone management for photographic look. ACM T Graphic 2006; 25(3): 637–645.

Farbman

Fattal

Lischinski

et al . Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM T Graphic 2008; 27(3): 1–67.

Rudin

Osher

Fatemi

Nonlinear total variation based noise removal algorithms. Physica D: Nonlin Phen 1992; 60(1): 259–268.

10.

Chan

Esedoglu

Aspects of total variation regularized c function approximation. SIAM J Appl Math 2005; 65(5): 1817–1837.

11.

Aujol

Gilboa

Chan

et al . Structure-texture image decomposition¡ªmodeling, algorithms, and parameter selection. Int J Comput Vis 2006; 67(1): 111–136.

12.

Coupé

Hellier

Kervrann

et al . Nonlocal means-based speckle filtering for ultrasound images. IEEE T Image Proces 2009; 18(10): 2221–2229.

13.

Shen

Liu

et al . A fast implementation algorithm of TV inpainting model based on operator splitting method. Comput Electr Eng 2011; 37(5): 782–788.

14.

Zhang

Wang

Edge adaptive directional total variation. J Eng 2013; 1(1): 1–2.

15.

et al . Image smoothing via

L_{0}

gradient minimization. ACM T Graphic 2011; 30(6): 1741.

16.

Wang

Yang

Yin

et al . A new alternating minimization algorithm for total variation image reconstruction. SIAM J Imag Sci 2008; 1(3): 248–272.

17.

Paris

Durand

. A fast approximation of the bilateral filter using a signal processing approach. In: Proceedings of the 9th European conference on computer vision—ECCV, Graz, 7–13 May 2006, pp.568–580. New York: ACM.

18.

Chen

Paris

Durand

Real-time edge-aware image processing with the bilateral grid. ACM T Graphic 2007; 26(3): 103

19.

Yang

Tan

Ahuja

Real-time O(1) bilateral filtering. In: Proceedings of the IEEE conference on computer vision and pattern recognition—CVPR, Miami, FL, 20–25 June 2009, pp.557–564. New York: IEEE.

20.

Yang

. Recursive bilateral filtering. In: Proceedings of the 12th European conference on computer vision—ECCV, Florence, 7–13 October 2012, pp.399–413. New York: IEEE.

21.

Liu

Freeman

Szeliski

et al . Noise estimation from a single image. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR’06), New York, NY, 17–22 June 2006, pp.901–908. New York: IEEE.

22.

Winnemöller

Olsen

Gooch

Real-time video abstraction. ACM T Graphic 2006; 25(3): 1221–1226.

23.

Fattal

Agrawala

Rusinkiewicz

Multiscale shape and detail enhancement from multi-light image collections. ACM T Graphic 2007; 26(3): 51.

24.

Tripathi

Mukhopadhyay

Single image fog removal using bilateral filter. In: Proceedings of the IEEE international conference on signal processing, computing and control, Waknaghat, India, 15–17 March 2012, pp.1–6. New York: IEEE.

25.

Perona

Malik

Scale-space and edge detection using anisotropic diffusion. IEEE T Pattern Anal Mach Intel 1990; 12(7): 629–639.

26.

Sun

Tang

Guided image filtering. IEEE T Pattern Anal Mach Intel 2013; 35(6): 1397–1409.

27.

Subr

Soler

Durand

Edge-preserving multiscale image decomposition based on local extrema. ACM T Graphic 2009; 28(5): 1471–1479.

28.

Meyer

Oscillating patterns in image processing and nonlinear evolution equations the fifteenth Dean Jacqueline B. Lewis memorial lectures. Providence, RI: American Mathematical Society, 2001.

29.

Aujol

Chambolle

Dual norms and image decomposition models. Int J Comput Vis 2005; 63(1): 85–104.

30.

Chan

Esedoglu

Park

. A fourth order dual method for staircase reduction in texture extraction and image restoration problems. In: Proceedings of the IEEE international conference on image processing—ICIP, Hong Kong, China, 26–29 September 2010, pp.4137–4140. New York: IEEE.

31.

Shen

Chang

Hung

et al . Edge-preserving image decomposition using

L_{1}

fidelity with

L_{0}

gradient. In: Proceedings of the Asia 2012 technical briefs SIGGRAPH, Singapore, 28 November–12 December 2012, pp.1–4. New York: ACM.

32.

Liu

Xiong

Yang

et al . A generalized relative total variation method for image smoothing. Multimed Tool Appl 2016; 75(13): 7909–7930.

33.

Cheng

Zeng

Liu

Feature-preserving filtering with

L_{0}

gradient minimization. Comput Graph 2014; 38: 150–157

34.

Jia

. Two-phase kernel estimation for robust motion deblurring. In: Proceedings of the European conference on computer vision—ECCV, Crete, Greece, 5–11 September 2010, pp.157–170. New York: Springer.

35.

Yan

Xia

et al . Structure extraction from texture via relative total variation. ACM T Graphic 2012; 31(6): 139.

36.

Zhang

Shen

et al . Rolling guidance filter. In: Proceedings of the European conference on computer vision—ECCV, Zurich, 6–12 September 2014, pp.815–830. New York: Springer.

Edge-preserving filter with adaptive L 0 gradient optimization

Abstract

Keywords

Introduction

Related work

Adaptive and sparse L0 gradient optimization

L 0 gradient minimization

Adaptive L0 gradient minimization model

Optimization

Solving S

Solving η

Solving h

Solving v

Analysis and discussion

Local weighted function θ

The effectiveness of pre-smoothing

The effectiveness of L 1 fidelity term

Experiments and applications

Parameter analysis and settings

Image denoising

Clip-art compression artifact removal

Detail enhancement and edge extraction

Image abstraction and pencil sketching

HDR tone mapping

Conclusion

Footnotes

Declaration of conflicting interests

Funding

ORCID iD

References

Adaptive and sparse L₀ gradient optimization

L ₀ gradient minimization

Adaptive L₀ gradient minimization model

Solving $η$

Local weighted function $θ$

The effectiveness of $L_{1}$ fidelity term