Depth estimation for image dehazing of surveillance on education

Abstract

Foggy weather brings lots of inconvenience for outdoor safety surveillance in the densely populated school education area. Research on image and video dehazing is able to solve this problem. Most existing methods recover the haze-free scenes relying on the atmospheric scattering model in image dehazing, which often suffer from halo artifacts because of the indistinct edges in the scene depth map. L₀ gradient minimization is introduced to better preserve and locate important edges globally to optimize the scene depth map, making use of this physical model in this paper. Firstly, a rough scene depth map based on the inherent boundary constraint prior on the scene is estimated. Secondly, the rough scene depth map in bright regions is compensated with an adaptive term. Then this compensated scene depth map is put into an optimizing framework to get a refined depth map to make it closer to the ideal scene depth. Finally, with the refined depth map and global atmospheric light, we can recover the haze-free scenes using the atmospheric scatting model. Experimental results show the proposed is better to obtain haze-free scenes with sharp edges, abundant details and vivid color while dealing well with bright areas.

Keywords

Image dehazing scene depth map haze-free scenes

1 Introduction

Outdoor images are often contaminated by suspended atmospheric aerosols such as haze, fog, smoke and others. Therefore, the reflected light from objects, before it reaches the camera, is attenuated and absorbed by these particles in the atmosphere. As a result, contrast, color and visibility of images are drastically degraded, which makes it unable to meet the requirements of most image perception systems [10 –12] and algorithms for surveillance, intelligent vehicles, object recognition, and target location. Foggy weather brings our daily life lots of inconvenience, especially for outdoor safety surveillance. In the densely populated school education area with weaker security management, outdoor supervision security risk exists brought by foggy weather. Thus, image and video dehazing becomes particularly important [18, 19].

The goal of image dehazing is to remove the weather effect caused by suspended atmospheric particles, alleviate loss of contrast and color distortion, and eventually make images more visible. Lots of techniques have been proposed to tackle this problem. In general, they can be categorized into two classes: one [1, 2] is the enhancement-based methods, which can significantly enhance contrast, enrich details, and improve visual quality of hazy images, but may also lead to a loss of salient details and unnatural color as they are not physical valid and cannot adaptively make improvements according to image characters. The other [3] is dedicated to recover the scene relying on the atmospheric scatting model. This restoration is based on image degradation mechanism and special for image haze removal, whose result images seem more natural and vivid. Meanwhile, it is better to process complex scenes and the details and structural information are preserved more completely.

Single image dehazing methods based on this physical model have been appealing enough in recent years thanks to its convenience of input image, adaptation of algorithm and its compelling results. However, this model-based restoration is severely ill-posed thus needs additional assumptions over the scene to tackle the problem. Considerable statistical priors have been explored for single image dehazing in previous work to recover haze-free scenes. In [22], a high-contrast prior on haze-free image, which assumes haze-free images have higher contrast than hazy ones, was proposed to maximize local contrast. Fattal [4] estimated the albedo of the scene and the medium transmission map on the assumption that transmission map and surface shading are statistically uncorrelated. In [9], Fattal used the color-line prior in natural images, where pixels of small image patches typically exhibit a one-dimensional distribution in RGB color space under assumptions, to restore the scene. He et al. [5] proposed the dark channel prior to roughly estimate the image depth map. This prior comes from a statistical observation that most local patches in haze-free images often contain some low intensity pixels.

Depth map and atmospheric light are the essential tasks of single image dehazing based on the physical model. Several optimized techniques have been investigated in recent work to estimate depth map making use of priors or constraints to remove haze. Nishino et al. [14] modeled the image with a factorial Markov random field to jointly estimate the scene albedo and depth map by levering their latent statistical structures. But, it needs more constraints to improve the accuracy. Carr [13] modeled geometry constraint and made the optimization work with the help of α-expansion method. However, edge information cannot be presented well in depth map because of its over-smoothing and local patch-based operation. Meng et al. [17] explored the inherent boundary constraint of the scene to obtain a rough estimation of scene depth map, which was then put into an optimization work making use of a weighted L₁-norm regularization method. Reference [26] estimated the scene depth map using its proposed weighted guided image filter and the modified dark channel prior. However, its performance in preserving edged is not well because of its local operator. In general, as approaches cited above are more or less based on local patch-based operation, edges of scene depth can not estimated accurately, which inevitably leads to halo artifacts and ambiguous detail and edges in result haze-free scenes.

To overcome the above problems, we propose a single image dehazing method based on L₀ gradient minimization to refine the scene depth map. The L₀ gradient minimization [16], which preserves salient edges and structures in a global measure, can dramatically alleviate halo artifacts caused by local patches. Moreover, small-resolution objects and thin edges can be faithfully maintained. Because of its nature, we put the L₀ gradient minimization into refining the depth map obtained from priors, which can efficiently preserve sharp edges and alleviate halo artifacts in result images that traditional methods suffered. Moreover, to address problems like halo artifacts, noise amplification and over-darkness in bright regions because of the undervalued scene depth, we introduce an adaptive term to compensate the scene depth obtained from the prior before putting it into the optimization framework using L₀ gradient minimization. The global atmospheric light is obtained by optimizing method in [5]. Finally, we restore the image with the refined depth map and atmospheric light using this physical model. Extensive experiments demonstrate that the proposed is better able to recover scenes with sharp edges, abundant details and vivid color while dealing well with bright areas.

This paper is organized as follows. Section 1 discusses several image dehazing works and problems traditional methods suffered. Section 2 gives an instruction of the atmospheric scatting model and details the main work of the algorithm. In Section 3,we show some results on real images and make a comparison with several methods. A conclusion is provided in Section 4.

2 Single image dehazing based on L₀ gradient minimization

Most of the methods are devoted to recover haze-free scenes relying on the atmospheric scattering model through estimating its scene depth map and atmospheric light. The proposed is also following this idea. Given a hazy image, a rough depth map is obtained based on the inherent boundary constraint on the haze scene. Then an adaptive compensation strategy is proposed to compensate the scene depth map in bright region. Next this compensated scene depth is modeled into an optimization function using the L₀ gradient minimization to get the final refined depth map. Finally, with the refined scene depth and global atmospheric light, we recover haze-free scenes using the physical model. Figure 1 shows the framework of this algorithm.

2.1 Scene depth map based on the boundary constraint of the scene

The atmospheric scatting model [6 –8] proposed by McCartney [21] and derived by Narasimhan [20], is described as: $I (x) = J (x) t (x) + A (1 - t (x))$ (1)

Where x indexes the pixel of the observed hazy image I (x), J (x) is the haze-free image we desired, A is the atmospheric light, and the scene depth t (x) describes the portion of the light that is not scattered and reaches the camera. The goal of haze removal is to obtain A, and t (x), and eventually recover J (x) from Equation (1). The pixel of I (x) contaminated by fog will approach the global atmospheric light A. The scene radiance J (x) meets an inherent boundary constraint of the scene [17], C₀ ≤ J (x) ≤ C₁, ∀x ∈ Ω, where Ω is the local neighborhood of pixel x in radiance cube [17]. Then the scene depth can be described as the ratio of two line segments as illustrated in (2), according to the atmospheric scatting model, which leads to the following boundary constraint on t (x) :0 ≤ t_b (x) ≤ t (x) ≤1. The pixels-wise t_b (x) strands for the lower bounder of t (x), given by (3). Under this condition, a patch-wise scene depth t_s (x) can be calculated by (4) to eliminate effects of little brighter pixels. $\frac{1}{t (x)} = \frac{∥ J (x) - A ∥}{∥ I (x) - A ∥}$ (2) $\begin{matrix} t_{b} (x) & = & min {max_{c \in {r, g, b}} \\ (\frac{A^{c} - I^{c} (x)}{A^{c} - C_{0}^{c}}, \frac{A^{c} - I^{c} (x)}{A^{c} - C_{1}^{c}}), 1} \end{matrix}$ (3) $t_{s} (x) = min_{y \in w_{x}} max_{z \in w_{y}} t_{b} (z)$ (4)

In (3), I^c, A^c, $C_{0}^{c}$ and $C_{1}^{c}$ are the color channels of I, A, C₀, C₁ respectively. Where C₀ = (20, 20, 20) ^T, C₁ = (300, 300, 300) ^T. The scene depth map $\tilde{t} (x)$ obtained from [17] is more reliable compared with the scene depth derived by dark channel prior.

2.2 Adaptive compensation strategy on the scene depth map in bright region

The captured images more or less suffer from noise in bright region, especially in sky region. Denoting the captured image noise as n, the noise magnitude is n/t. Since the obtained depth map in bright region is close to zero, differences of neighboring pixels or noise are amplified excessively, which severely degrade the result images. To deal with this issue, an adaptive compensation term Δt is introduced to slightly adjust the initial value of scene depth t_s in bright region, given as Equations (5) and (6). After that, the brightness of the depth map in bright regions is enhanced independently. $Δ t = (1 - t_{s} ({id}_{x})) \times 0.1$ (5) $\tilde{t} = {\begin{matrix} t_{s} ({id}_{x}) + Δ t, & bright region \\ t_{s}, & otherwise \end{matrix}$ (6)

In (5), $\tilde{t}$ stands for the adjusted depth map. id_x is the coordinate of a pixel in bright region. It is detected by selecting pixel satisfying the condition |I (x) - A| < K. Where A presents the atmospheric light, K is a threshold determined by histogram of the haze image [15].

2.3 Refining scene depth map with L₀ gradient minimization

For the depth map $\tilde{t} (x)$ obtained from the scene boundary constraint and adjusted in bright regions, it contains too much detail and cannot stick out salient structures, which cannot reflect real depth information of the scene. As the effectiveness of L₀ gradient minimization in sharpening major edges and its operation of globally locating edges, which can dramatically alleviates halo artifacts caused by local patches in most image dehazing algorithm, we put the L₀ gradient minimization into an objective function to refine the adjusted scene depth obtained above, expressed as (7). Through minimizing (7), the refined scene depth map t (x) is obtained which can capture the sharp edge discontinuities and outline the profile of objects. $min_{t} {\sum_{x} {(t (x) - \tilde{t} (x))}^{2} + λ \cdot C (t)}$ (7) $C (t) = # {p | | \partial_{x^{'}} t (x) | + | \partial_{y^{'}} t (x) | \neq 0}$ (8)

Where $\tilde{t} (x)$ is the adjusted depth map under prior of boundary constraint. t (x) is the final refined depth map and C (t) denotes the algorithm of L₀ norm, defined as counting the number of non-zero elements of the gradient field, where # is the counting operator, expressed as (8). λ is a parameter controlling the level of smoothness.

As C (t) in (7) contains the pixel-wise difference and global discontinuity, we adopt the half-quadratic method by introducing auxiliary variables (h, v) to approximate their corresponding gradients ∂_x′t_x, ∂_y′t_x respectively to solve this problem and rewrite the objective function (7) as (9):

$\begin{matrix} min_{t, h, v} {\sum_{x} {(t_{x} - {\tilde{t}}_{x})}^{2} + λ \cdot C (h, v) \\ + β ((\partial_{x^{'}} t_{x} - h_{x})^{2} + (\partial_{y^{'}} t_{x} - v_{x})^{2})} \end{matrix}$ (9)

Problem (9) is solved through alternatively minimizing (h, v) and t. The similarity between auxiliary variables (h, v) and ∂_x′t_x, ∂_y′t_x is controlled by parameter β, which is set as a small value 2λ at the beginning, multiplied by a constant κ each time, and finally ended when β is larger than 10⁵. κ is set as 2 and λ as 0.03.

Computing t when h, v are fixed, the sub-problem is equal to (10) and the final result of t is (11): $\begin{matrix} min_{t} {\sum_{x} (t_{x} - I_{x})^{2} + β ((\partial_{x^{'}} t_{x} - h_{x})^{2} + (\partial_{y^{'}} t_{x} - v_{x})^{2})} \end{matrix}$ (10) $\begin{matrix} t = F^{- 1} (\frac{F (I) + β (F (\partial_{x^{'}}) * F (h) + F (\partial_{y^{'}}) * F (v))}{F (1) + β (F (\partial_{x^{'}}) * F (\partial_{x^{'}}) + F (\partial_{y^{'}}) * F (\partial_{y^{'}}))}) \end{matrix}$ (11)

Computing (h, v) when t is fixed, we can rewrite the sub-problem as (12) and the corresponding result is (13): $\begin{matrix} min_{h, v} {\sum_{x} ((\partial_{x^{'}} t_{x} - h_{x})^{2} + (\partial_{y^{'}} t_{x} - v_{x})^{2}) \\ + \frac{λ}{β} C (h, v)} \end{matrix}$ (12) $(h_{p}, v_{p}) = {\begin{matrix} (0, 0) & (\partial_{x} t_{p})^{2} + (\partial_{y} t_{p})^{2} \leq λ / β \\ (\partial_{x} t_{p}, \partial_{y} t_{p}) & otherwise \end{matrix}$ (13)

2.4 Estimating the atmospheric light

As we known, inaccurate estimation of the atmospheric light may result in inaccurate luminance and distorted color in result images [27]. In [5], the atmospheric light is estimated using dark channel prior with a fixed window size. But in some cases with large bright objects or multiple light sources, the global atmospheric light may be estimated troublesomely and inaccurately with a fixed window size, shown as the left image [5] in Fig. 2. We filter the dark channel image with a moving window w_x which is set as 3% of the largest dimension of the gray-level image in most cases. Then pick the top 0.1% brightest pixels in the dark channel. Among these pixels, the pixel with highest intensity in the input hazy mage I is selected as the atmospheric light, given as (14), (15) and (16). Where x and y index pixels in hazy image I, c presents R, G, B color channel. $I_{dark} = min_{y \in w_{x}} {min_{c \in {r, g, b}} I (y)}$ (14) $A = max_{x \in Φ} (I (x))$ (15) $\begin{matrix} Φ & = & {x | I (x) is the 0.1 % of brightest \\ {pixels in dark channel image I}_{dark}} \end{matrix}$ (16)

2.5 Recovering the scene radiance

With the atmospheric light and the refined scene depth map, we can recover the scene radiance according to (17): $J (x) = \frac{I (x) - A}{[max (t (x), ɛ)]^{δ}} + A$ (17)

Where ɛ is a small constant to avoiding division by zero, the exponent δ is used for fine-tuning the dehazing effects and set as 0.85.

3 Comparison experiments

To demonstrate the effectiveness of the proposed method, we select different types of outdoor images in our experiments.

3.1 Qualitative comparison

Figure 3 shows a comparison between results obtained by [17] and our algorithm. The first column are the hazy images, next are the results obtained by [17] and the proposed, followed by a zoom of the result images. The last column displays the corresponding depth map of these partial enlarged results.

As can be seen from the partial enlarged images of first and third line in Fig. 3, our algorithm is better able to keep sharp edges and remove haze than Meng’s method, which remains residual haze in and around these regions while our results do not. In the second line, considerable improved contrast and details have been got by [17] and our method. However, the proposed is superior to [17] in processing details of branches. This is mainly due to the fact that Meng’ method is based on a local patch measure which is one cause of halo artifacts. Our method is bases on the global smoothing measure of L₀ gradient minimization, thus alleviate halo problems generated by local patch or average operation.

3.2 Quantitative comparison

Figures 4 and 5 further present comparisons against the classical and recent dehazing techniques. The blind measure of Hautière et al. [23], which consists in three indicators e, $\bar{r}$ , Σ, is used to quantitative assess and rate these algorithms. Among these indicators, indicator e represents edges newly visible after restoration, the mean $\bar{r}$ computes the ratio of gradient norms at visible edges, while Σ shows the percentage of pixels which are saturated (black or white) after restoration. In most cases, good performance is described by high values of e, $\bar{r}$ and low values of Σ. In addition, in order to evaluate the degree of haze removal, we introduce an indicator D [24], which evaluates the perceptual fog density of images. The smaller the value of D is, the lower the degree of haze is, the better the performance of haze removal is. The quantitative comparison results are given in Tables 1 and 2.

In Fig. 4, we make comparisons between our method, and recent works such as Chio et al. [24] and He et al. [5]. From these images, we can see that Chio’s result often suffers from oversaturation, which can be seen from the value of Σ. Analyzing results of Table 1 in general, while all the methods yield nearly the same and small values of Σ, our method produces higher values of e, which shows that our algorithm outperforms than others in terms of contrast and visibility enhancement. Meanwhile, our value of D is always lower than others, which show the proposed method is more effective for removing haze.

Figure 5 shows comparisons of our method with several traditional dehazing algorithms: Fattal [4], He et al. [5] and Tarel et al. [25]. As can be seen, the result images produced by Tarel often seem oversaturated, over-dark and contain halo artifacts. Fattal’s results remain too much haze in general. Seen from Table 2, in most cases, our algorithm gives the relative big value of e and small value of Σ, which indicates that our algorithm enhances the contrast more efficiently and recovers much more details while does not contain too much saturated pixels. Moreover, for all the images, the value of D in our result images is the smallest, indicating that the degree of haze removal by the proposed is thorough and again demonstrating the efficiency and effectiveness of our algorithm in haze removal.

4 Conclusion

Single image dehazing is a challenging problem because of the unknown depth map. In this paper, we analyze the problems most existing methods suffered and propose a single image dehazing method based on L₀ gradient minimization to refine the scene depth map. For halo artifacts caused by local measure and indistinct edges of scene depth, the L₀ gradient minimization is applied to optimize the depth obtained from the prior because of its nature in sharping major edges and processing on a global measure. Moreover, small-resolution objects and thin edges can be faithfully maintained. In addition, thanks to the adaptive compensation term for the rough scene depth map obtained from the prior, the proposed can also deal well with bright regions. Our algorithm bring the school education surveillance great convenience and make a big difference in the field of image dehazing.

Footnotes

Acknowledgments

This research was supported partially by the National Natural Science Foundation of China (Nos. 61372130, 61432014, 61501349), the Fundamental Research Funds for the Central Universities (Nos. BDY081426, JB140214, XJS14042), the Program for New Scientific and Technological Star of Shaanxi Province (No. 2014KJXX-47).

References

Zimmerman

J.B.

, Pizer

S.M.

, Staab

E.V.

et al., An evaluation of the effectiveness of adaptive histogram equalization for contrast enhancement, IEEE Transactions on Medical Imaging7(4) (1988), 304–312.

Hines

, Rahman

, Jobson

, et al., Single-scale retinex using digital signal processors, In: Global Signal Processing Conference, 27(30) 2004.

Narasimhan

S.G.

and Nayar

S.K.

, Chromatic framework for vision in bad weather, Proceeings of 2000 IEEE Conference on Computer Vision and Pattern Recognition, 2000, pp. 598–605.

Fattal

, Single image dehazing, ACM Transactions on Graphics27(3) (2008), 72.

, Sun

and Tang

, Single image haze removal using dark channel prior, IEEE Transactions on Pattern Analysis and Machine Intelligence33(12) (2009), 2341–2353.

Nayar

S.K.

and Narashiman

S.G.

, Vision in bad weather, Proceedings of the 7th IEEE International Conference on Computer Vision, 2, 1999, pp. 820–827.

Narasimhan

S.G.

and Nayar

S.K.

, Vision and the atmosphere, International Journal of Computer Vision48(3) (2002), 233–254.

Narasimhan

S.G.

and Nayar

S.K.

, Contrast restoration of weather degraded images, IEEE Transactions on Pattern Analysis and Machine Intelligence25(6) (2003), 713–724.

Fattal

, Dehazing using color-lines, ACM Transactions on Graphics34(1) (2014), 13.

10.

Lin

, Yang

, Lv

, Wei

and Song

, A self-assessment stereo capture model applicable to the internet of things, Sensors15(8) (2015), 20925–20944.

11.

Yang

, Chen

, Zhou

and Lv

, A low-power and portable biomedical device for respiratory monitoring with a stable power source, Sensors15(8) (2015), 19618–19632.

12.

Yang

, He

, Lin

and Lv

, Multimedia cloud transmission and storage system based on internet of things, Multimedia Tools and Applications (2015), 1–16.

13.

Carr

and Hartley

, Improved single image dehazing using geometry, In Digital Image Computing: Techniques and Applications, 2009, pp. 103–110.

14.

Nishino

, Kratz

and Lombardi

, Bayesian defogging, International Journal of Computer Vision98(3) (2012), 263–278.

15.

Dai

S.K.

and Tarel

J.P.

, Adaptive sky detection and preservation in dehazing algorithm, IEEE International Symposium on Intelligent Signal Processing and Communication Systems (2015), 6.

16.

, Lu

, Xu

et al., Image smoothing via L₀ gradient minimization, ACM Transactions on Graphics30(6) (2011), 174.

17.

Meng

, Wang

, Duan

et al., Efficient image dehazing with boundary constraint and contextual regularization, Proceedings of the 2013 IEEE International Conference on Computer Vision, 2013, pp. 617–624.

18.

Yang

, Zhou

, Lv

, Wei

and Song

, A real-time monitoring system of industry carbon monoxide based on wireless sensor networks, Sensors15(11) (2015), 29535–29546.

19.

Yang

, Lin

, Gao

, Lv

, Wei

and Song

, Quality index for stereoscopic images by separately evaluating adding and subtracting, PLoS One10(12) (2015), e0145800.

20.

Narasimhan

S.G.

and Nayar

S.K.

, Removing weather effects from monochrome images, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, pp. 186–193.

21.

McCartney

, Optics of the atmosphere: Scattering by molecules and particles. John Wiley and Sons, 1976.

22.

Tan

R.T.

, Visibility in bad weather from a single image, In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp. 1–8.

23.

Nicolas

and Tarel

J.P.

, Blind contrast enhancement assessment by gradient ratioing at visible edges, Image Analysis & Stereology Journal27(2) (2008), 87–95.

24.

Choi

L.K.

, You

and Bovik

A.C.

, Referenceless prediction of perceptual fog density and perceptual image defogging, IEEE Transactions on Image Processing24(11) (2015), 3888–3901.

25.

Tarel

J.P.

and Nicolas

, Fast visibility restoration from a single color or gray level image, IEEE International Conference on Computer Vision, 2009, pp. 2201–2208.

26.

and Zheng

, Edge-preserving decomposition-based single image haze removal, IEEE Transactions on Image Processing24(12) (2015), 5432–5441.

27.

Sulami

, Glatzer

, Fattal

and Werman

, Automatic recovery of the atmospheric light in hazy images, IEEE International Conference on Computational Photography, 2014, pp. 1–11.

Depth estimation for image dehazing of surveillance on education

Abstract

Keywords

1 Introduction

2 Single image dehazing based on L0 gradient minimization

2.1 Scene depth map based on the boundary constraint of the scene

3.1 Qualitative comparison

3.2 Quantitative comparison

4 Conclusion

Footnotes

Acknowledgments

References

2 Single image dehazing based on L₀ gradient minimization