Abstract
Video smoke detection benefits life safety and environment protection. A method of video smoke detection using shape, color and dynamic texture features is presented in this paper. Firstly, an algorithm identifying cone geometry feature is used to distinguish conical region from dynamic regions. Secondly, conical regions are filtered by a color filtering algorithm to further test the candidate smoke region. Finally, a texture filtering algorithm is used to differentiate true smoke from candidate smoke regions. Experiments show that the proposed method is effective and it results in earlier and more reliable detection than the other two methods reported in the literature. The processing rate of the smoke detection method achieves 25 frames per second with an image size of 320×240 pixels.
Keywords
Introduction
Smoke is a lethal fire hazard causing casualties when fire occurs. Early detection of smoke is crucial for saving lives. Smoke detection has been attached more and more importance. However, traditional sensors for smoke detection can only be applied in closed space or indoor scenarios. With development of video surveillance system, a new technology of real-time video smoke detection has attracted more and more attention in recent years. This technology uses various image process algorithms to identify smoke from video sequences.
Many methods of smoke detection are centered around smoke motion detection then supported by other filtering algorithms. Xiong et al. [1] used background subtraction to detect moving objects, followed by flickering extraction, contour initialization and contour classification to distinguish smoke from other moving objects. Toreyin et al. [2, 3] also employed background subtraction method to extract moving objects. Then, flickering analysis, measure of turbulence and energy variation were provided to detect smoke. A different technique, namely integral image technique was used in Yuan’s [4] work to detect moving objects which is further processed with an accumulative motion model for video smoke detection. Tung et al. [5] employed a median method combined with a fuzzy c-means method to segment moving regions and cluster candidate smoke regions from moving regions. Then, parameter extraction method and support vector machine classifier were employed to discriminate smoke from candidate smokes. Wang et al. [6] used choquet fuzzy integral to separate moving object from video frames to detect smoke. Then, a swaying identification algorithm based on centroid calculation was used to distinguish candidate smoke region from other dynamic regions. Thirdly, smoke texture feature was used to differentiate smoke from other candidate smoke regions by Gray Level Co-occurrence Matrix (GLCM). Jia et al. [7] proposed a method of segmenting smoke region based on saliency detection.
Texture judgment is another approach to detect smoke. Ferrari et al. [8] used Hidden Markov Tree (HMT) to characterize steam texture pattern, and a Support Vector Machine (SVM) classifier was used to detect steam. Yuan [9] used histogram sequence of Local Binary Pattern (LBP) and Local Binary Pattern Variance (LBPV) pyramids to detect smoke. Cui et al. [10] combined tree-structured wavelet transform and GLCM to analyze texture feature of fire smoke. Ye [11] proposed a dynamic texture descriptor with surfacelet transform and HMT model for smoke detection.
Another group of methods are centered around color filtering. Simone et al. [12] presented a smoke detection method using image energy and color information. Yu et al. [13] used motion and color features to detect smoke object. After background estimation, moving pixels were checked using a color decision rule. Then, optical flow algorithm based on brightness was proposed to calculate optical flow of candidate regions. Finally, a back-propagation neural network was used to classify smoke features from non-fire smoke features.
There are some other methods to detect smoke. In the method presented by Gubbi et al. [14], statistical features, such as arithmetic mean, geometric mean, standard deviation, skewness, kurtosis and entropy, were computed on sub-bands of 3-level wavelet transformed images. Then, SVM was used for detection of smoke. Tian et al. [15] assumed an image to be a linear blending of a smoke component and a background image. The algorithm converted smoke detection into an optimization problem. Luo et al. [16] proposed a smoke detection method based on condensed image. After smoke images were condensed, the paper used special characteristics such as right-leaning line, smooth streamline, low-frequency, fixed source and vertical–horizontal ratio to detect smoke. He et al. [17] presented a smoke detection method based on a semi-supervised clustering model. Yuan et al. [18] proposed a video smoke detection based on semitransparent properties. The paper used the haze image optical model, region growing method and fuzzy clustering analysis to detect smoke. Li et al. [19] used back-propagation neural network based on Moderate Resolution Imaging Spectro-radiometer (MODIS) data to detect forest fire smoke. Chen et al. [20] proposed a method based on multiple features fusion to detect fire smoke in train carriages.
The literatures [1–20] give many detection methods based on smoke features such as motion, color, texture, etc. Smoke motion and texture is very unique. Few literatures further study smoke motion shape after extracted as moving object. There is also no literature to study smoke texture along smoke orientation. In order to improve the accuracy of smoke detection, these smoke features still need to be further studied.
When fire smoke occurs, smoke plume generally rises straight or tilts upwards under influence of buoyancy and wind. Smoke plume diameter gradually becomes greater and greater forming an inverted conical region [21]; smoke often displays grayish colors; the moving orientation of smoke varies continuously which leads to its texture varying continuously. An algorithm was developed to detect smoke based on the identification of the aforementioned combination of smoke features in this paper. Experiments show that the proposed method is effective and it results in earlier and more reliable detection than the other two methods reported in the literature.
Method
The proposed method consists of three major steps: Extracting conical region from video frames. Smoke color filtering in conical regions and getting candidate smoke region. Dynamic texture filtering and extracting true smoke from candidate smoke regions.
Details of the three steps are illustrated in the following subsections.
Conical region identification
The identification algorithm consists of two steps. Firstly, dynamic region is extracted from video frames. Secondly, an algorithm identifying cone geometry feature is used to distinguish conical region from dynamic regions.
Extracting dynamic region
In this paper, choquet fuzzy integral is used to extract dynamic regions from video frames. Fuzzy integral is based on the notion of a fuzzy measure, which can be viewed as the weight of importance of a set [6]. It can be used to realize data classification based on information fusion. Let the value of similarity measure μ : F → [0, 1], {f (x1) , f (x2) , …, f (x
n
)} follow a non-decreasing order as:
Choquet fuzzy integral can be calculated as:
Figure 1 shows the detection results using choquet fuzzy integral algorithm for three videos. Figure 1(a) presents a scene with an out-door smoke and a walking man. Figure 1(c) demonstrates a swaying plastic bag. Figure 1(e) presents a fire in a laboratory room. Figure 1(b, d, f) are binary images which show the detected dynamic regions of Fig. 1(a, c, e), respectively. As image morphology processing is used to eliminate noises, it also sets a few pixels in dynamic region as background pixels. However, this does not influence smoke detection result.

Dynamic regions detection.
When smoke occurs, smoke plume generally rises straight or tilts upwards under influence of buoyancy. Smoke often forms a slender volume. Along smoke diffusion orientation, smoke diameter graduallybecomes greater and greater. Generally, it presents an inverted conical region, which correspondingly presents a triangle geometry region in image. As can be seen from Fig. 2, smoke region surrounded by red lines in Fig. 2(a) and detected smoke region in Fig. 2(b) surrounded by red lines show triangle geometry shape.

Smoke cone regions.
An axis of elongation can be approximately used to define smoke orientation. The line is the axis which the integral of the square of the distance to points in the region is a minimum [24]. The integral is defined as:
Figure 3 is a binary image in which the red arrow line represents smoke orientation. The other red line represents smoke verticality orientation which is perpendicular to smoke orientation. Smoke verticality orientation passes through smoke region centroid and divides smoke region into top region and bottom region. The green signals ‘*’,’+’ and ‘o’ represent smoke centroid, bottom region centroid and top region centroid, respectively.

Smoke orientation.
For nth frame, C (X (n) , Y (n)) represents centroid coordinate of a dynamic region (DR (n)) in this paper. It can be computed as follows:
Rate1 and
As can be seen from Table 1, smoke region in Fig. 1(b) and the plastic bag in Fig. 1(d) are in agreement with filtering conditions of conical region. The man in Fig. 1(b) does not satisfy the conditions. Figure 4 shows the detection results after conical region filtering. In the following section, smoke color filtering is further used to differentiate candidate smoke region from conical regions.

Binary image of detected conical regions.
When smoke occurs in video frames, most area usually displays grayish colors except at the interface region where gray color appears to be blended with background color. For RGB color space, R component, G component and B component of smoke pixel are approximately equal [25]. Many literatures used RGB color space to judge smoke color. However, judging smoke color with more variable parameters easily leads to a bigger cumulative deviation. This may cause more false alarms for detecting smoke. In this paper, YCbCr color space is used to judge candidate smoke region from conical regions. By RGB space, Cb and Cr components can be calculated as Equations (10 and 11), respectively.
As smoke color approximates grayish color, its three components of RGB space are nearly equal. According to Equations (10 and 11), both Cb value and Cr value of smoke pixel are close to 128 (±10 %). Equations (12 and 13) are adopted for judging smoke pixel.
For a conical region, the ratio of pixel number which satisfies Equations (12 and 13) to the pixel number of the conical region is adopted for judging candidate smoke region. Rate2 and
Rate2 and
As can be seen from Table 2, conical region in Fig. 4(a) is in agreement with color filtering conditions. The corresponding region in Fig. 1(a) is regarded as candidate smoke region. The conical region in Fig. 4(b) is not satisfied with color filtering conditions. In the following section, dynamic texture filtering is further used to testify candidate smoke region.
When fire smoke occurs, smoke plume generally moves under influence of buoyancy, which forms a variable flow orientation. According to testing result for many smoke videos, the variation of adjacent pixel color along smoke orientation is less than that of smoke verticality orientation. This forms a unique dynamic texture. It is difficult to accurately describe smoke texture feature using a fixed orientation.
According to this smoke texture feature, a VO-GLCM (Variable Orientation Gray Level Co-occurrence Matrix) operator is presented in this paper. For frame n and smoke orientation θ1, a search needs to be carried out to compute the frequency of gray levels i and j (i, j ∈ 0, 1, 2, …, l - 1) occurring at a specified distance d and the specified angle θ1 to obtain a l × l VO-GLCM; For frame n + 1, smoke orientation changes to θ2 and a new l × l VO-GLCM is constructed at distance d and angle θ2. Figure 5 shows two VO-GLCMs for two consecutive frames. In the first frame, smoke orientation is 45 deg. In the second frame, it changes to 90 deg. Smoke orientation generally ranges from 0 deg to 180 deg in actual environment. For creating VO-GLCM, degrees in (0, 22.5), [22.5, 67.5), 67.5, 112.5), 112.5, 157.5) and 157.5, 180) are set as 0 deg, 45 deg, 90 deg, 135 deg and 180 deg respectively in this paper. Experiments show that the approximate direction does not influence smoke detection result.

VO-GLCM operator. (a) θ1 = 45° and d = 1 in the first frame. (b) The first small frame of size 5×5. (c) VO-GLCM for the first frame. (d) θ2 = 90° and d = 1 in the second frame. (e) The second small frame of size 5×5. (f) VO-GLCM for the second frame.
For candidate smoke region in a frame, two VO-GLCMs are created with two texture orientations to describe texture of candidate smoke region. As variation of adjacent pixel color along smoke orientation is less than that of smoke along smoke verticality orientation, the sum of elements values on main diagonal along smoke orientation is greater than the sum along smoke verticality orientation. Rate3 and
Rate3 and
In summary, the flowchart of the method presented in this paper can be depicted in Fig. 6.

Flowchart of smoke detection method.
The proposed method is implemented using Matlab. It is applied to 17 video clips to detect smoke in this paper. The videos are normalized to 25Hz and 320×240 pixels. Some videos were downloaded from the Internet (http://imagelab.ing.unimore.it and www.gettyimages.ae).

Eight different scenes of smoke.

Six scenes of smoke and non-smoke videos.
Figure 7 shows different video smokes occurring in different scenes. Figure 7(a) presents a smoke on a hill which is covered with mist. Figure 7(b) shows a grey smoke beside parking lot. Figure 7(c) presents a black smoke blended with fire in a testing house. Figure 7(d) is a white smoke in outdoor scenarios. Figure 7(e) shows a large amount of smoke beside fire laboratory. Figure 7(f) presents a chimney smoke in a factory. Figure 7(g) shows a chimney smoke from farmhouse. Figure 7(h) presents a slim smoke in laboratory. They are all correctly detected which show the method is effective.
The performance of the proposed method is compared with that of Wang et al’s [6] method and Yu et al’s [13] method. The images of the chosen video sequences including smoke scenes and non-smoke scenes are shown in Fig. 8.
The detection at frame n in Table 4 means that the smoke is detected at the nth frame after smoke is started at 0th frame. As shown in Table 4, the proposed method in the current study outperforms that of Wang et al and Yu et al. As optical flow algorithm in Yu et al’s method is sensitive to motion region where the gray levels change significantly, the slow change in gray levels in Movie (a), Movie (b) and Movie (c) resulted in a later smoke alarm than the other two methods. Too many constraints for texture filtering in Wang et al’s method led to a later smoke alarm in Movie (c). Therefore, in comparison to the other two methods, the new method of the current study is more conducive to early detection.
Smoke detection performance comparison
Some non-smoke movies in Fig. 8 were used to test false alarm rate of the three methods. Number of false alarms in Table 5 means the number of false alarms after testing all frames of a movie. As shown in Table 5, the proposed method has a lower false alarm rate than the other two methods. This shows the proposed method using shape, color and dynamic texture features to detect smoke is more reliable than the other two methods.
False alarm performance comparison
In this study, a method of early video smoke detection was developed based on shape, color and dynamic texture features. The conic shape of smoke plume helps to distinguish the disturbances from other motion modes, such as swaying fire, walking people. After smoke color filtering, smoke texture feature is analyzed to further separate smoke from other possible disturbances. The new method was subjected to a series of videos testing which involved many no-smoke scenes and smoke scenes with smoke color of wide grayish range from white to black. Experiments show that the proposed method is effective. Comparing to the two other existing methods, the proposed method has achieved earlier detection and lower false alarm rate.
The developed algorithm has been tested with a limited number of video clips. More experiments with a wider range of environment conditions are required to test the robustness of the algorithm. The development of a method for both smoke and fire detections will be conducted in our future work.
Footnotes
Acknowledgments
Supported by the Open Project Program of The State Key Laboratory of Fire Science (No. HZ2015-KF02), University of Science and Technology of China; Supported by Research fund for the Doctoral Program (No. 20140408), Anhui Jianzhu University. Supported by the Open Project Program of the National Laboratory of Pattern Recognition (NLPR) (NO. 201700014), Anhui Provincial Natural Science Foundation (No. 1708085MF167).
