Real-time unstructured road detection based on CNN and Gibbs energy function

Abstract

Road detection algorithms with high robustness as well as timeliness are the basis for developing intelligent assisted driving systems. To improve the robustness as well as the timeliness of unstructured road detection, a new algorithm is proposed in this paper. First, for the first frame in the video, the homography matrix H is estimated based on the improved random sample consensus (RANSAC) algorithm for different regions in the image, and the features of H are automatically extracted using convolutional neural network (CNN), which in turn enables road detection. Secondly, in order to improve the rate of subsequent similar frame detection, the color as well as texture features of the road are extracted from the detection results of the first frame, and the corresponding Gaussian mixture models (GMMs) are constructed based on Orchard-Bouman, and then the Gibbs energy function is used to achieve road detection in subsequent frames. Finally, the above algorithm is verified in a real unstructured road scene, and the experimental results show that the algorithm is 98.4% accurate and can process 58 frames per second with 1024×960 pixels.

Keywords

Unstructured road detection Gibbs energy function CNN

1 Introduction

Studies show that there are about 9.23 million traffic accidents caused by driver fatigue and inattentiveness worldwide each year, with more than 1.4 million fatalities [1]. In order to reduce the incidence of traffic accidents and enhance driving safety, many artificial intelligence-related research institutions and automobile manufacturers have invested a lot of manpower and resources in developing advanced driver assistance systems (ADAS) in recent years [2]. ADAS is an active safety technology that uses a variety of sensors installed in the vehicle to collect data from the environment inside and outside the vehicle and perform technical processing such as static and dynamic object recognition, detection and tracking, thus enabling the driver to detect possible hazards at the earliest possible time in order to draw attention and improve safety [3]. Real-time road detection algorithms based on vision sensors are the basis for developing driver assistance systems, which can help drivers determine whether the area ahead is a road area where vehicles can drive, whether there are obstacles, vehicles and non-road areas such as pedestrians, and also provide guidance for subsequent path planning [4, 5].

Roads can generally be divided into structured roads and unstructured roads. Structured roads have clear road markings, clear road boundaries and special color information, including: highways and urban roads [6]. Structured road problem detection can be reduced to the problem of road marking, road boundary and road color detection [7]. Unstructured roads do not have obvious road markings, road boundaries and fixed colors, and the road environment is complex and variable (e.g., there are shadows, water spots and complex obstacles, etc.), so the detection method for unstructured roads is still in the research stage [8].

The current methods on structured roads detection are as follows: Yao et al. [9] reduced the free-space estimation task to an inference problem on a one-dimensional graph by defining the potential function of the one-dimensional graph based on the image features of edges, colors, and vector machine, which eventually partitions the road space from the non-road space using the labels of the nodes. Wang et al. [10] transformed the image into the inverse perspective space, then constructed a variable road model using Bezie spline curve and detected the road based on the RANSAC algorithm to solve the model parameters. Cheng et al. [11] proposed a road surface segmentation method that fuses road geometric and appearance features, which achieves self-adaptation to different road scenes and significantly improves the segmentation accuracy compared to the method that uses geometric cues alone. Wang et al. [12] used a combination of Hough transform and parabolic model for lane detection, the algorithm consists of two parts: initial detection of road edges using Hough transform, then mid-to-side strategy is used to detect road boundary points and parabolic model is used to fit the road boundary for tracking purpose. Kong et al. [13] used a confidence-weighted Gabor filter to calculate the principal texture direction of each pixel and estimated the vanishing points of the road by an adaptive soft voting scheme based on variable voting regions, and then segmented the corresponding road regions according to the detected vanishing points. Fern $\overset{´}{a}$ ndez et al. [14] used the watershed transform to segment the original image into super pixels, and then extracted the colors and trained a decision tree-based classifier using the texture features of the super pixels to accomplish road detection. The above methods usually use the color features, boundary features and texture features of the road to achieve the detection and tracking of structured roads, and the computational complexity is not high, which can meet the real-time requirements of ADAS. However, in the unstructured road scenario, the robustness of unstructured road detection based on the above algorithm is low because the road itself does not have fixed surface features and structural features, and the road features and road environment are complex and variable.

The Current research on unstructured roads detection are as follows: Cai et al. [15] proposed a detection method that fuses road images with their corresponding local road maps to improve the robustness of drivable road region detection in unstructured road environments, which can accomplish the detection of drivable road regions in regions without clear lane markings or boundaries. Geng et al. [16] firstly segmented unstructured road images into uniformly sized super pixels using the Simple Linear Clustering algorithm (SLIC), based on this, the road areas as well as non-road areas are classified based on the full convolutional neural network (CNN), finally, the classification results of the CNN are optimized using the relationship between superpixel neighborhoods and using Markov Random Fields (MRF). Mendes et al. [17] proposed a detection method combining a context window with a multilayer convolutional neural network (CNN) that is capable of detecting drivable areas in a clear road image. Brostow et al. [18] extracted action as well as structural features of objects in 3D space, then projected these features into 2D space and implemented road segmentation using Random Decision Forest (RDF). Yuan et al. [19] proposed an online framework for feature vector-based structural cue mining, which first discriminates between road boundary and non-boundary instances by means of a structural support vector machine (SSVN), the discriminated boundaries are then used to fit a complete road boundary. On this basis, the road area is extrapolated accordingly, and the obtained results are taken as ground truth values.

Although the above unstructured road detection algorithm has strong robustness, it is difficult to be widely used in actual road scenarios because the detection model is more complex and has higher computational complexity, and does not have better timeliness. In order to overcome the problems of poor robustness due to uncertainty of road surface features and structural features and poor timeliness due to complex models and high computational complexity, a new real-time unstructured road detection and tracking algorithm is proposed in this paper.

Therefore, the real-time detection technology of drivable area of unstructured road needs to overcome the following two problems: (1) the problem of poor robustness caused by the uncertainty of road surface features and structural features, (2) The problem of poor timeliness caused by complex detection model and high computational complexity. In order to overcome the above two problems, a real-time driving area detection method of unstructured road based on CNN and Gibbs energy function is proposed in this paper.

In order to solve the first problem, the homography matrix features of different regions in the image are used as the basis to distinguish road regions and non-road regions. Because flatness is the only invariable characteristic of unstructured road, whether there is a large flat area in front of the field of view in the image can be used as the criterion for the detection of driveable area. The algorithm using flatness as the criterion has the following two advantages: 1) the flatness will not change with the change of road type, color, illumination and driving environment, so that the algorithm has high robustness; 2) A single feature can simplify the algorithm model, reduce the computational complexity of the whole algorithm, and make the algorithm have high timeliness. It can be seen from reference [46] that points on the same plane in the image have similar homography matrix. Therefore, the homography matrix features of different regions in the image can be used to judge whether the front region is flat.

In order to solve the second problem, this paper first estimates the homography matrix H of different regions in the first frame image based on the improved RANSAC algorithm, and uses CNN to automatically extract the features of H and obtain the large flat region in the image, that is, the road region, to realize road detection. Compared with the traditional CNN algorithm based on road pixel features, the CNN algorithm based on regional homography matrix features has lower computational complexity. However, homography matrix estimation and road detection based on CNN are still time-consuming. Because there is little difference in road surface features between the first frame and adjacent subsequent frames, the road detection algorithm based on road surface features can be used to replace the above algorithm for adjacent subsequent frames, so as to improve the detection rate of subsequent similar frames. The road detection algorithm based on road surface features can be divided into the following two processes: (1) the color and texture features of the road are extracted from the detection results of the first frame image, and the corresponding GMMs model is constructed based on Orchard Bouman, (2) Minimize Gibbs energy function and divide road area and non-road area.

The remainder of this paper is organized as follows: in section 2 and 3, we discuss the main aspect of the proposed method. The details of the planar homography and CNN method are introduced in Section 2. The color and texture features method are given in Section 3. Different experiments conducted using the database are described in Section 4. Finally, some concluding remarks and direction for future work are provided in Section 5.

2 Road detection based on planar homography and CNN

The overall process of this paper is shown in Fig. 1. The road detection algorithm based on road surface features can be divided into the following two processes: (1) The road color and texture features are extracted from the detection results of the first frame image, and the corresponding GMMs model is constructed based on Orchard Bouman; (2) Minimize Gibbs energy function and divide road area and non-road area.

Fig. 1

System description.

Since we use the flatness of the road as the criterion for detecting whether the image contains a road area, the first frame in a new road scene needs to determine whether it contains a large drivable plane area. There are two main methods on plane estimation as follows: The first method uses 3D reconstruction to calculate the world coordinates of pixel points based on dense parallax maps and camera parameters [21, 22]. This algorithm can accurately calculate the homography matrix between world coordinate points and pixel points, but the algorithm is computationally intensive and requires constant camera calibration. The second method uses the feature points matched by the road plane in the left camera image and the right camera image to estimate the homography matrix between pixels [23, 24]. This method is simpler and more effective than the first method based on stereo matching. To save computation time, the second method is chosen to estimate the single response matrix in this paper.

2.1 Plane homography of stereoscopic vision

The homography matrix describes the correspondence between three-dimensional points in the same plane in space in two images, while the stereo vision homography matrix describes the correspondence between the same point in space in two images. When the feature points in the scene fall on the same plane in the world coordinate system, the affine relationship between the feature points in the left image and the right image can be expressed by the homography matrix H . The descriptive equation is shown in Eq. (1). The imaging point of M in the plane π in the left camera is m_a, which can be corresponded to the imaging point m_b in the right camera by the single response matrix H induced by the plane π, as shown in Fig. 2. With the left camera image I_left as the reference, the homography matrix H ₀ between the feature points in the road area is calculated according to Eq. (1), and the right camera image I_right is transformed according to H ₀, and compare the transformed right image $I_{right}^{'}$ with I_left. If the points in $I_{right}^{'}$ overlap with the points in I_left, the area consisting of the overlapping points is the road area, otherwise it is the non-road area. $I_{right}^{'}$ vs. I_left is shown in Fig. 3. $m = m^{'} H$ (1)

Fig. 2

Homography of plane.

Fig. 3

Comparison between $I_{right}^{'}$ and I_left.

The matrix H is a nonsingular matrix (3×3).

Define H ′, which is homography matrix between the world plane and the image plane. If the homogeneous world coordinate point is M = (X, Y, 1) ^T and the corresponding homogeneous image coordinate point is M = (x, y, 1) ^T, the relationship between m and M is as follows: $m = S A [R_{3 \times 3} T_{3 \times 1}] M$ (2)

Where S is the scaling factor, A is the camera internal parameter matrix, which contains 4 parameters, camera focal length d and center point coordinates of camera (x_c, y_c, z_c); [ R _3×3 T _3×1] is the external parameter, which is the relative orientation between the world coordinate system and the camera coordinate system, R _3×3 is the rotation and T _3×1 is the translation.

The internal matrix of camera A and camera B are A_a and A_b, the rotation matrix and translation matrix corresponding to the conversion from 3D point in world coordinate system to camera A and camera B are [ R _a T _a], [ R _b T _b]. The corresponding pixels of feature point m in camera A and camera B are M _a = (x_a, y_a, 1), M _b = (x_b, y_b, 1). Therefore, the model matrices of the two cameras are as follows: $(\begin{matrix} x_{a} \\ y_{a} \\ 1 \end{matrix}) = S_{a} A_{a} [R_{a} T_{a}] (\begin{matrix} X \\ Y \\ 1 \end{matrix})$ (3) $(\begin{matrix} x_{b} \\ y_{b} \\ 1 \end{matrix}) = S_{b} A_{b} [R_{b} T_{b}] (\begin{matrix} X \\ Y \\ 1 \end{matrix})$ (4)

Eq. (3) can be transformed into: $(\begin{matrix} X \\ Y \\ 1 \end{matrix}) = S_{a}^{- 1} A_{a}^{- 1} {[R_{a} T_{a}]}^{- 1} (\begin{matrix} x_{a} \\ y_{a} \\ 1 \end{matrix})$ (5)

By substituting Eq. (5) into Eq. (4), we can get: $(\begin{matrix} x_{b} \\ y_{b} \\ 1 \end{matrix}) = S_{b} S_{a}^{- 1} A_{b} A_{a}^{- 1} [R_{b} T_{b}] {[R_{a} T T_{a}]}^{- 1} (\begin{matrix} x_{a} \\ y_{a} \\ 1 \end{matrix})$ (6)

$S_{b} S_{a}^{- 1} A_{b} A_{a}^{- 1} [R_{b} T_{b}] {[R_{a} T_{a}]}^{- 1}$ is a 3×3 matrix, so it can be simplified to H :

$H = [\begin{matrix} h_{1} & h_{2} & h_{3} \\ h_{4} & h_{5} & h_{6} \\ h_{7} & h_{8} & h_{9} \end{matrix}]$ (7)

Eq. (6) can be simplified to: $(\begin{matrix} x_{b} \\ y_{b} \\ 1 \end{matrix}) = [\begin{matrix} h_{1} & h_{2} & h_{3} \\ h_{4} & h_{5} & h_{6} \\ h_{7} & h_{8} & h_{9} \end{matrix}] (\begin{matrix} x_{a} \\ y_{a} \\ 1 \end{matrix})$ (8)

Expanding the Eq. (8) to get three equations, and substituting the third equation into the first two equations: $x_{b} = \frac{h_{1} x_{a} + h_{2} y_{a} + h_{3}}{h_{7} x_{a} + h_{8} y_{a} + h_{9}}$ (9) $y_{b} = \frac{h_{4} x_{a} + h_{5} y_{a} + h_{6}}{h_{7} x_{a} + h_{8} y_{a} + h_{9}}$ (10)

Expanding Eq. (9) and (10) by multiplying the denominators: $h_{1} x_{a} + h_{2} y_{a} + h_{3} - h_{7} x_{a} x_{b} - h_{8} y_{a} x_{b} - h_{9} x_{b} = 0$ (11) $h_{4} x_{a} + h_{5} y_{a} + h_{6} - h_{7} x_{a} y_{b} - h_{8} y_{a} y_{b} - h_{9} y_{b} = 0$ (12)

Assuming that N pairs of feature points can be obtained from the images of camera A and camera B, the following system of linear equations can be obtained:

The Eq. (22) is the Nth feature point pair in the two images $(\begin{matrix} x_{a}^{N} & y_{a}^{N} & 1 \end{matrix})$ , $(\begin{matrix} x_{a}^{N} & y_{a}^{N} & 1 \end{matrix})$ , and the right side of the equation is a 2N × 1 matrix. Since the homography matrix H contains ∥ H ∥ = 1 constraints, H with 8 degrees of freedom requires at least 4 corresponding points to calculate the homography matrix.

$[\begin{matrix} x_{a}^{1} & y_{a}^{1} & 1 & 0 & 0 & 0 & - x_{a}^{1} x_{b}^{1} & - y_{a}^{1} x_{b}^{1} & - x_{b}^{1} \\ 0 & 0 & 0 & x_{a}^{1} & y_{a}^{1} & 1 & - x_{a}^{1} y_{b}^{1} & - y_{a}^{1} y_{b}^{1} & - y_{b}^{1} \\ x_{a}^{2} & y_{a}^{2} & 1 & 0 & 0 & 0 & - x_{a}^{2} x_{b}^{2} & - y_{a}^{2} x_{b}^{2} & - x_{b}^{2} \\ 0 & 0 & 0 & x_{a}^{2} & y_{a}^{2} & 1 & - x_{a}^{2} y_{b}^{2} & - y_{a}^{2} y_{b}^{2} & - y_{b}^{2} \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ x_{a}^{N} & y_{a}^{N} & 1 & 0 & 0 & 0 & - x_{a}^{N} x_{b}^{N} & - y_{a}^{N} x_{b}^{N} & - x_{b}^{N} \\ 0 & 0 & 0 & x_{a}^{N} & y_{a}^{N} & 1 & - x_{a}^{N} y_{b}^{N} & - y_{a}^{N} y_{b}^{N} & - y_{b}^{N} \end{matrix}] [\begin{matrix} h_{1} \\ h_{2} \\ h_{3} \\ h_{4} \\ h_{5} \\ h_{6} \\ h_{7} \\ h_{8} \\ h_{9} \end{matrix}] = [\begin{matrix} 0 \\ 0 \\ ⋮ \\ ⋮ \\ ⋮ \\ ⋮ \\ ⋮ \\ 0 \end{matrix}]$ (13)

2.2 Homography matrix estimation based on improved RANSAC algorism

The traditional RANSAC algorithm can estimate the homography matrix from the data set containing a large number of external points by iteratively randomly selecting internal points, which eliminates a certain number of serious error effects, and can eliminate more mismatched points, thus improving the accuracy of the homography matrix [25]. However, RANSAC algorithm has the following two shortcomings: (1) When there are many mismatches of feature points in the image, the number of iterations of the algorithm increases greatly, which reduces the execution efficiency of the algorithm and affects the accuracy of homography matrix [26]; (2) Since RANSAC only has a certain probability to get a credible model, the probability of getting a credible homography matrix is directly proportional to the number of iterations, so there is no upper limit on the number of iterations. If the upper bound is set, the final homography matrix may not be the optimal result. Subsequent improvements of RANSAC, such as MLESAC [47] and PROSAC [48], are based on resampling. These methods aim to estimate the predefined transformation model between the matching point sets through repeated sampling in the initial matching, so as to find the maximum interior point set satisfying its estimation model as a matching pair. These methods rely heavily on the accuracy of sampling. When there are a large number of outliers in the initial matching, the required sampling times are significantly increased, which greatly reduces the efficiency of the method. In view of the above two shortcomings, this paper improves the traditional RANSAC algorithm as follows: (1) The improved fast algorithm is used to extract stable and directional feature points from the difference of Gaussian pyramid (DOG), then, SIFT matching method is used to optimize the matching results, and the mismatching points are eliminated to reduce the iteration times of homography matrix estimation; (2) LM (Levenberg Marquardt) nonlinear optimization method is introduced to improve the accuracy of homography matrix by minimizing the cost function iteration. The specific process of homography estimation based on improved RANSAC algorithm is as follows:

(1) Construction of DOG

The difference image contains stable features in different blur degrees and scales; therefore, the DOG can provide great convenience for the subsequent feature point extraction. Firstly, the low-pass filter is used to smooth the image, and the smooth image is down sampled to obtain the Gaussian pyramid. The specific construction process is shown in Fig. 4.

Fig. 4

Construction process of Gaussian pyramid.

A total of O groups of L layer images with different scales are generated in the Gaussian pyramid, the scale space (o, l) of Gaussian pyramid is composed of O and L, then, the layer l + 1 of group O of Gaussian pyramid is differentiated from the layer L of group O, get the O Group L layer image of the DOG, the scale space of the Gaussian pyramid and the construction process of the dog are shown in Fig. 5.

Fig. 5

Scale space of Gaussian pyramid and construction process of DOG.

The original image of size M × N is defined as: I (x, y), the group O image is I^o (x, y). In Figs. 5 and 6, the Gaussian smoothing function of layer 1 in group O is as follows: $G (x, y, 2^{o - 1} k^{l} δ) = \frac{1}{2 π {(2^{o - 1} k^{l} δ)}^{2}} e^{\frac{- (x^{2} + y^{2})}{2 {(2^{o - 1} k^{l} δ)}^{2}}}$ (14)

Where δ is the standard deviation of normal distribution, and δ = 1.6. The scale space expression of the l layer of the group o is as follows: $L (x, y, 2^{o - 1} k^{l} δ k^{l} δ) = G (x, y, 2^{o - 1} k^{l} δ k^{l} δ) \otimes I^{o} (x, y)$ (15)

Therefore, the scale space expression of the l layer of group O is as follows:

$D (x, y, 2^{o - 1} k^{l} δ) = (G (x, y, 2^{o - 1} k^{l} δ) - G (x, y, 2^{o - 1} k^{l - 1} δ)) \otimes I^{o} (x, y)$ (16)

It can be transformed into:

$D (x, y, 2^{o - 1} k^{l} δ) = L (x, y, 2^{o - 1} k^{l} δ) - L (x, y, 2^{o - 1} k^{l - 1} δ)$ (17)

(1) The improved FAST algorithm is used to extract and match the feature points;

Feature point detection algorithms mainly include Harris [27], SIFT [28] [29], SUSAN [30], SURF [31], etc., and corresponding improved algorithms: PCA-SIFT, ICA-SIFT, P-ASURF, R-ASURF, Radon SIFT, etc. [32]. Harris and SUSAN algorithms are difficult to repeatedly and accurately locate the points on the uniform area or object contour in different images. At the same time, these two algorithms can only solve the problem of rotation invariance, but cannot solve the problem of scale change [33]. However, road images are generally composed of uniform regions and simple contours, and there are scale changes between left and right images, therefore, Harris and SUSAN algorithms are not suitable for road image feature point detection. The feature points detected based on SIFT and SURF algorithm are not only invariant to rotation, scaling and brightness changes, but also stable to a certain extent to the change of viewing angle and noise [34], But its huge feature computation makes the process of feature point detection very time-consuming, so it is difficult to apply it to real-time road detection scene.

Therefore, in order to ensure that the feature extraction algorithm has scale invariance and high computational efficiency, this paper firstly uses the improved fast feature point detection algorithm to quickly extract the feature points in the left and right images, then the feature point descriptor is obtained by ORB algorithm, Finally, the initial matching results are obtained based on the FLANN matching algorithm, and then the SIFT matching method is used to optimize the matching results.

Fast algorithm is a fast method to extract feature points, but it has the following two disadvantages: (1) the detected feature points do not meet the scale change; (2) the detected feature points have no direction. To solve the above two problems, this paper proposes an improved fast feature point detection algorithm. In view of the first shortcoming, this paper uses the method in the above process (1) to construct the DOG of image, and detects the feature points in the dog, so as to make it meet the scale change effect. Aiming at the second shortcoming, this paper obtains the histogram of the gradient direction of the neighborhood points by counting the gradient directions of all the points in the neighborhood of the feature points, according to this, the direction of the feature points is determined, so that the feature points are invariant to the angle and rotation of the image. The specific process is shown in Fig. 6.

Fig. 6

Direction determination process of feature points.

The gradient direction of the neighboring points of p in C is calculated as follows: $θ (x, y) = {tan}^{- 1} [\frac{L (x, y + 1) - L (x, y - 1)}{L (x + 1, y) - L (x - 1, y)}]$ (18)

Where: θ (x, y) is the gradient direction of the neighboring points, L (x, y) is the brightness of pixel (x, y).

Gradient normalization method: the gradient direction is defined in the range [0°, 360°] and divided into 36 equal parts [49]. Define Δθ = θ (x, y) - n, if |Δθ| ⩽ 5, then θ (x, y) = n. The gradient direction after statistical normalization is obtained as the gradient histogram of the neighborhood points. The direction represented by the term with the largest vertical coordinate in the histogram is used as the principal direction of the feature point p. If there is a peak in the histogram greater than 80% of the peak in the main direction, the direction corresponding to the peak is the auxiliary direction of the feature point p, otherwise p has no auxiliary direction.

Randomly capture 560 images from the road video (280 images from each of the left and right cameras, and in pairs), after obtaining the FAST feature points with orientation from the above images, the descriptors of FAST feature points are extracted using SURF, ORB and SIFT algorithms in OpenCV, and using the FlANN matching algorithm to obtain the initial matching results of each of the above three algorithms, as shown in Fig. 6. On the basis of this previous, the matching results are optimized using the SIFT matching method. We set the ratio of the nearest distance to the next nearest distance to be less than the threshold value of 0.6, as shown in Fig. 7. The extraction results and time consumption of the above three algorithms are compared, and the comparison results are shown in Table 1.

Fig. 7

Initial matching results.

Table 1

Parameters of different network structures

Network structures	C1	Step-size	S1	Step-size	C2	Step-size	S2	Step-size	C3	Step-size	F1	F2
2C-0S-16-32	16*5²	3	–	–	32*3²	–	–	–	–	–	256	2
2C-2S-16-32	16*5²	3	2*2	2	32*3²	1	2*2	2	–	–	256	2
3C-2S-4-8-16	4*3²	3	2*2	2	8*3²	1	2*2	2	8*3²	1	256	2
3C-2S-16-32-64	16*3²	3	2*2	2	32*3²	1	2*2	2	64*3²	1	256	2
3C-2S-48-64-128	48*3²	3	2*2	2	64*3²	1	2*2	2	128*3²	1	256	2

It can be seen from Fig. 8 that the number of false matches in the initial matching result is high. It can be seen from Fig. 8 that SIFT matching method can remove unqualified matching points from the matching results, and the accuracy of the three algorithms has been greatly improved, moreover, orb algorithm takes the shortest time and the fastest matching speed, which is about 90 times of SIFT algorithm and 8 times of SURF algorithm.

Fig. 8

Optimized matching results.

(3) Homography matrix estimation based on improved RANSAC algorithm

Firstly, the left reference image is divided into N regions with an area of w × h, the set of feature points corresponding to the nth region is I_n, which contains S_n pairs of feature points, n = 0, 1, 2, . . . , N; then, the homography matrix $H_{n}^{best}$ of each region is estimated based on RANSAC algorithm, and get the corresponding interior point set $I_{inner}^{n}$ , finally, LM is used to optimize $H_{n}^{best}$ to get $H_{n}^{Best}$ , The total projection error corresponding to the set of interior points is minimized. The specific algorithm steps are as follows:

Judge whether the number of feature point pairs S_n in the region n is greater than 4. If S_n ⩾ 4, randomly select 4 feature point pairs from the region as the inner set point $I_{inner}^{n}$ , and calculate the homography matrix $H_{n}$ of the region, which is recorded as model M_n; otherwise, merge the adjacent regions until the number of feature point pairs in the merged region is greater than 4, and continue to go down to step 2), If all regions are merged and the number of feature points is still less than 4, the calculation ends;

The projection error of all feature point pairs in $I_{n} - I_{inner}^{n}$ and model M_n is calculated. If there is a point pair whose projection error is less than $T_{n}^{1}$ , $I_{inner}^{n}$ is added to it;

If the number of interior points in $I_{inner}^{n}$ is greater than $T_{n}^{2}$ , then $H_{n}$ is re estimated by using all the feature point pairs in $I_{inner}^{n}$ ; otherwise, go to step 1) reselect the feature point pairs to estimate the initial homography matrix;

Repeat 1) ∼ 3) k times, and evaluate the result of each iteration through the projection error to get the optimal homography matrix $H_{n}$ and the interior set point $I_{inner}^{n}$ , count the number of points P of $I_{inner}^{n}$ , and re estimate $H_{n}$ with $I_{inner}^{n}$ to get $H_{n}^{best}$ ;

Calculate the projection error $e_{p}^{n} (H_{n}^{best})$ and the error index function $E_{n} (H_{n}^{best})$ of each pair of feature points in $I_{inner}^{n}$ ;

If $E_{n} (H_{n}^{best})$ is less than the threshold of error index, the calculation is stopped and the optimal homography matrix $H_{n}^{Best}$ in the region n is obtained; otherwise, it goes to 7);

Calculate the partial reciprocal $J (H_{n}^{best})$ of each component of $H_{n}^{best}$ relative to $e_{p}^{n} (H_{n}^{best})$ , and then obtain the increment $Δ H_{n}^{best}$ , the new homography matrix is $H_{n}^{newbest} = H_{n}^{best} + Δ H_{n}^{best}$ . If $E_{n} (H_{n}^{newbest}) < E_{n} (H_{n}^{best})$ , reduce the value of ξ and let $H_{n}^{best} = H_{n}^{newbest}$ and return to 5). Otherwise, increase the value of ξ and retain the original homography matrix and return to 5).

The calculation formula of the minimum number of iterations k in the above algorithm is as follows: $k = \frac{lg (1 - p)}{lg (1 - ɛ^{min})}$ (19)

Where: ɛ is the ratio of the number of interior points to S_n, min is the minimum number of samples needed to calculate the model, p is the confidence, generally set at 0.95∼0.99. In this paper, we set min = 4, p = 0.95. Since the proportion of the interior point in the dataset is unknown, in general, ɛ can be set to the ratio of 4 to S_n, and then updated to the ratio of the current maximum interior point to S_n in the iterative process.

The calculation formulas of projection error $e_{p}^{n} (H_{n}^{best})$ , error index function $E_{n} (H_{n}^{best})$ and increment $Δ H_{n}^{best}$ of homography matrix based on LM are shown in Eqs. (20∼22) respectively. $e_{p}^{n} (H_{n}^{best}) = | H m_{p}^{b} - m_{p}^{a} |$ (20)

Where $m_{p}^{b}$ and $m_{p}^{a}$ are the feature point pairs matched in the region n respectively. $E (H_{n}^{best}) = \sum_{p}^{P} {(e_{p}^{n} (H_{n}^{best}))}^{2}$ (21)

$\begin{matrix} Δ H_{n}^{best} = - {[J^{T} (H_{n}^{best}) J (H_{n}^{best}) + ξ I]}^{- 1} \\ J^{T} (H_{n}^{best}) e (H_{n}^{best}) \end{matrix}$ (22)

Where $J (H_{n}^{best})$ is Jacobian matrix; I is identity matrix; when ξ decreases, LM algorithm approaches Gauss Newton method; when ξ increases, LM algorithm approaches gradient descent method.

2.3 Road detection based on CNN

According to the improved RANSAC algorithm, the homography matrices of N regions in the left reference image can be estimated. Whether the homography matrices have similar features can be used as a criterion to judge whether different regions belong to the same plane. So, feature extraction and classification of homography matrix become the key of road plane detection. Traditional classifiers need to design multiple features artificially, and it takes a lot of time to verify whether these features can distinguish road plane. CNN has intelligent learning mechanism, which transforms the feature expression of samples in the original space into a new feature space through layer-by-layer feature transformation, so as to obtain more abundant high-level features and achieve more accurate classification and recognition. At the same time, CNN has been widely used in target detection [35], target recognition [36], segmentation [37] and other fields.

2.3.1 Structure of CNN

Common CNN is mainly composed of input layer (I layer), convolution layer (C layer), pooling layer (S layer), full connection layer (F layer) and output layer (O layer). Its structure is shown in Fig. 9.

Fig. 9

Structure of CNN(1) Convolution layer.

Convolution layer mainly extracts features by convolution operation, which not only enhances the features of the original signal, but also reduces the noise. In order to extract more abundant features, a convolution layer often contains multiple filters, which means F_depth > 1. Therefore, the output of the convolution layer has F_depth component matrix, that is, the output depth of the convolution layer is D = F_depth. After convolution of the input data and the filter, the element values in the i - th row and j - th column of the output matrix at a certain output depth d are as follows: $z_{d, i, j} = \sum_{r = 0}^{R - 1} \sum_{c = 0}^{C - 1} w_{d, r, c} x_{i + r, j + c} + b_{d}$ (23)

Where R and C are the height and width dimensions of the filter respectively, and b_d is a constant.

The output of the current convolution layer activated in depth is obtained through the activation function: $a_{d, i, j} = Re LU (a_{d, i, j})$ (24)

The matrix at multiple output depths together constitutes the output a of the current volume layer, and serves as the input of the next layer.

(2) Pooling layer

After convolution processing, each region and adjacent regions in the image have relevant feature information, which will lead to information redundancy. In order to improve the performance and robustness of the algorithm, the output of the convolution layer is down sampled. The lower sampling layer uses the local correlation principle of image to sample the image, uses a fixed size filter to move on the matrix a in turn, and extracts the maximum or average value of the corresponding data block to form the elements of the output matrix as the sampling result. Two common pooling operations are maximum pooling and average pooling.

(3) Fully connected layer

After the convolution layer and pooling layer processing, the higher-level features are extracted, and then the classification is completed through a number of fully connected layers. Finally, the Softmax function transforms the classification results into probability distribution to obtain the road plane area.

2.3.2 CNN solution

CNN is essentially an input-output mapping, it can learn a lot of input-output mapping relationship. In this paper, the homography matrix of each region in the image is used as the input of CNN, and the road plane region is used as the final output. In the process of CNN training, the current network input and network weight are used to calculate the network output, that is, the road plane area detected by this round of training; then the error between the training result and the sample label is calculated; finally, the back-propagation algorithm is used to calculate the network weight error, and the weight update method is used to update the weight. After many times of training, the network parameters are constantly optimized, which makes the network structure tend to be stable.

The specific training process of CNN is as follows:

The homography matrix of each region in the left reference image is extracted by using the improved RANSAC algorithm;

The outer rectangle of each region is reserved with the region center as the center;

Then, each rectangle is marked (Road area is marked as 1, non-Road area is marked as 0), and all rectangles and their corresponding labels are used as training data;

CNN uses the homography moments of each region as the input of network training, and initializes the network parameters;

Forward propagation stage: calculate the corresponding actual output;

Backward propagation stage: calculate the difference between the actual output and the label, and adjust the weight matrix according to the method of minimizing the error.

The process of road detection based on CNN is as follows:

The homography matrix of each region in the left reference image is extracted by using the improved RANSAC algorithm;

The outer rectangle of each region is reserved with the region center as the center;

The trained CNN is used to determine the road area in the image.

3 Road detection based on color and texture features

The above algorithm can accurately detect the road area in the image, but the above algorithm is time-consuming, and the surface features of the road between adjacent frames are not different. Therefore, the above algorithm can be used to obtain the road area in the first frame of the image, on this basis, the road surface feature model is constructed, and then the model is used to detect the adjacent subsequent frames, so as to reduce the time consumption of the whole algorithm. Because the road region generally has obvious color and texture features, this paper can realize the road detection of subsequent frames based on the above two features and Gibbs energy function.

3.1 Building color GMMs model

It can be seen from reference [38] that in the construction of GMMs, Orchard-Bouman binary splitting algorithm is twice as fast as K-mean algorithm. Therefore, this paper uses the Orchard-Bouman binary splitting algorithm to build the GMMs model of color and texture of road pixels and non-road pixels, and its principle is shown in Fig. 10. The construction process of color GMMs model is as follows:

Fig. 10

Principle of Orchard-Bouman binary splitting algorithm.

In the first frame, the road region is defined as the road pixel, and the non-road region is defined as the non-road pixel;

Two GMMs models are defined. GMM₁ and GMM₀ are used to represent road and non-road color distribution respectively. GMM₁and GMM₀ contain K₁ and K₀ components respectively;

The components of GMM₁ and GMM₀ are obtained based on Orchard-Bouman binary splitting algorithm. The specific algorithm flow is shown in Fig. 11;

GMMs models are obtained based on the classification results, and each Gaussian model can be represented by K triples.

Fig. 11

Orchard–Bouman binary splitting algorithm.

$G M M = {(μ_{1}, \sum_{1}, ω_{1}), (μ_{2}, \sum_{2}, ω_{2}) \dots (μ_{K}, \sum_{K}, ω_{K})}$ (25)

In the above triples, μ _k is the mean value of pixel color or gray value, ∑_k is the 3×3 covariance matrix, and ω_k is the weight of the k - th component.

3.2 Building texture GMMs model

The construction process of texture GMMs model is the same as that of color GMMs model, but before the construction of texture GMMs model, it is necessary to convert the color value or gray value of each pixel into the feature value that can describe the texture. LBP (local binary patterns) is a non-parametric local texture feature descriptor with high computational efficiency. Because of its high feature discrimination and low computational complexity, LBP is selected to obtain the texture feature values of road and non-road regions.

3.3 Road detection based on Gibbs energy function

We simplify the road detection problem into a binary label problem. An image can be represented by a set V ={ Value (x, y) , (x, y) ∈ R ² }, where (x, y) represents the position of a pixel in the image. For the color image, Value (x, y) represents the vector composed of RGB values, and for the gray image, Value (x, y) represents the gray value. Let L ={ Lable (x, y) , (x, y) ∈ R ² } represents the label set of the pixel, where Lable (x, y) represents the label value of the pixel located at (x, y). Lable (x, y) = 0 is non-road pixel, Lable (x, y) = 1 is road pixel. The road detection problem can be transformed into the problem of finding the optimal label to minimize the Gibbs energy function. $E (U) = E {(U)}_{c} + η E {(U)}_{t}$ (26)

Where: E ( U ) _c is the difference between the pixel value and the color model, and the calculation method is shown in Eq. (27); E ( U ) _t is the difference between the pixel LBP value and the texture model, and the calculation method is shown in Eq. (28); η is the trade-off coefficient, when η = 2.5, the experiment in this paper can achieve better results. $E {(U)}_{c} = \sum_{(x, y) \in R^{2}} - log p (Value (x, y) Lable (x, y))$ (27)

Where: p (Value (x, y) Lable (x, y)) is the probability that the Value (x, y) value of the pixel matches the road model and non-road model. $E {(U)}_{c} = \sum_{(x, y) \in R^{2}} - log p ({LBP}_{Value (x, y)} Lable (x, y))$ (28)

Where: p (LBP_Value(x,y)Lable (x, y)) is the probability that the LBP_Value(x,y) value of the pixel matches the road model and non-road model.

As shown in Fig. 12: Fig. 12(a) is the first frame image in the video. Plane detection is performed on the first frame image based on the algorithm in Section 1, the detection result is shown in Fig. 12(b). In Fig. 11(b), the blue area is the road area, and the gray area is the non-road area. Figure 12(c), (d) and (e) are the minimum values of E ( U ) _c, E ( U ) _f and E ( U ) in the subsequent frame images, respectively. Figure 12(f) is processed by morphology, including erosion, expansion, denoising and filling, to get the maximum connection area, which is the final road area in subsequent frames.

Fig. 12

The road detection process based on road color, texture and Gibbs energy function.

3.4 Update strategy of GMMs model

In the process of detecting subsequent frames, if the E (U) value of the image is in range of [0, P], the subsequent frames can be detected according to the original color and texture GMMs model, otherwise, the frame is defined as the first frame image, and the algorithm in the first section is used to re detect the plane of the image, determine the driving plane area of the road, re calculate the color and texture GMMs model of the road, and then realize the continuous detection of the unstructured road in the real scene measurement.

4 Experiment

4.1 Experimental data set

Experimental software and hardware: vehicle camera (DF-8066), compilation environment (VS2012), operating system (64-bit windows 10 professional), memory (3GB), processor (2.27 GHz Intel (R) Core (TM) I3-350 M core). Experimental image: The image collected by the camera includes unstructured roads, that is, the image contains different road types, different backgrounds, different lighting conditions, and is affected by shadows and vehicles and pedestrians under different traffic conditions, as shown in Fig. 13. The original image size is 1024×960, a total of 31706 images, of which 23924 are used for training and 7782 for testing. Our experiment needs about 20 hours of training time, and the average test time of each image is 18 ms.

Fig. 13

Road images.

4.2 Parameter setting and network structure design

In order to get a higher accuracy of road plane area detection, we test various network structures in Section 1.3, but we finally choose detection convolutional neural network (RD-CNN). RD-CNN includes three C layers, two S layers and two F layers. The three C layers use 16, 32 and 64 convolution kernels to extract image features. The size of convolution kernels in each C layer is 3×3, and the step size of the first C layer is 3, and step size of the second and third C layers is 1. Both S layers use the maximum value for sampling. The number of neurons in the first F layer is 256, and the number of neurons in the second F layer is 2. The parameters of different network structures are shown in Table 1. The network structure 2C-0S-16-32 has no S layer, so the recognition effect is not ideal. The network structure 2C-2S-16-32 adds two S layers, based on 2C-0S-16-32, the recognition rate is significantly improved. The network structure 3C-2S-4-8-16 transforms the 5×5 convolution kernel into two 3×3 convolution kernels, which improves the depth of the network. The recognition rate is improved, but not significantly. On this basis, RD-CNN (3C-2S-16-32-64) adds convolution kernel, and the recognition rate is further improved to 98.6%.

In order to visually compare the influence of different network structure parameters on the recognition rate, Fig. 14 shows the recognition rate of 300000 iterations per network. It can be seen from Fig. 13 that when the iteration times are small, the network fluctuates greatly. With the increase of iteration times, the parameters of adjustable network are optimized gradually, and the recognition rate is gradually increased and stabilized. By increasing the number of S layers, we can reduce the dimension of output characteristics of layer C, thus reducing information redundancy and improving network efficiency. Increasing the number of C layers or convolution cores can improve the accuracy of road detection in a certain range, but also increase the computing time of the network. Therefore, considering the accuracy and calculation efficiency of the network, this paper selects RD-CNN network to complete the road detection of the first frame in the video.

Fig. 14

Recognition rate of different network structures.

In order to verify the robustness of the algorithm, this paper compares the algorithm in [39 –45] with the algorithm proposed in this paper, and uses the following four evaluation indexes: recall (R), percision (P), F-measure (F) and accuracy (A). The specific comparison results are shown in Table 2. The detection result of road area is shown in Fig. 15. The first line in Fig. 15 shows the input image, the second line shows the image of the actual road area that can be driven, and the third line shows the road result using vanishing point (VP) [40]. The fourth row of Fig. 15 shows the possibility of using the full convolution neural network (FCN) [41], the fifth row shows the output based on the random decision forest (RDF) [42], and the sixth row shows the road results using the conditional random field (CRF) [43]. The seventh line of Fig. 15 shows the detection results of the method combining the full convolution neural network and Markov random field (CNN-MRF) [45], and the last line shows the output of the algorithm.

Fig. 15

Unstructured road detection results based on different algorithms.

Table 2

Comparison of different algorithms

Methods	P	R	F	A
VP [39]	71.2	87.6	78.4	77.9
FCN [40]	89.8	87.6	88.2	89.9
SFMPC [41]	89.3	82.0	85.6	86.2
RFCF [42]	91.1	91.9	91.7	91.9
CPF. &CNN-7. [43]	89.1	86.1	88.1	87.1
RCC – Net [44]	83.4	84.2	83.8	82.5
CNN - MRF [45]	92.2	88.9	90.9	91.9
Our approach	98.4	96.3	97.6	98.5

The specific analysis is as follows: because the road type, road background, lighting conditions and traffic conditions in unstructured road image are uncertain, the method based on road intrinsic model [11, 39], road feature [12 –14] and road classifier [40 –45] cannot detect road well. However, this algorithm first extracts the feature value H which can represent the smoothness of the road, and then uses CNN to extract the features of H in different regions of the image, which can finally achieve high accuracy of road detection.

In order to verify the timeliness of the algorithm, 558 test images are arranged according to the sequence in the video stream, and the running time of each step of the algorithm in the test process is counted, as shown in Table 3. Firstly, the homography matrix of all test images is estimated and detected based on CNN. It can be seen from Table 3 that the road detection algorithm based on homography matrix estimation CNN takes a long time, and the detection rate is 53.04 ms/frame, while the output rate of video screen is 30 frames / second. Therefore, the algorithm cannot be well applied to the actual scene. Then, the road area and non-road area are manually marked in the test image, and the road GMMs model is constructed, and the road area is re-detected in the test image based on GMMs and Gibbs energy function. It can be seen from Table 3 that the road detection algorithm based on GMMs Gibbs energy function takes a short time, and the detection rate is 13.26 ms/frame, but the GMMs model needs to be constructed based on the marked image, which is not applicable in the actual driving scene. Finally, the road detection algorithm based on CNN Gibbs detects the unlabeled test image, and the time consumption is 17.24 ms/frame, which is one fourth of the time consumption of the road plane detection algorithm based on homography matrix estimation CNN, and less than the image acquisition rate. Therefore, the road detection algorithm based on CNN Gibbs can meet the needs of the actual scene.

Table 3

Algorithm time consumption per step

Step of road detection	Total time/ms	Average time/ms
Homography matrix estimation	3799.98	6.81
Road plane detection based on CNN	25796.34	46.23
Road plane detection based on homography matrix estimation CNN (unlabeled)	29596.32	53.04
Construction of GMMs model	2767.68	4.96
Road detection based on Gibbs energy function	4631.4	8.30
Road detection based on GMMs Gibbs energy function (labeled)	7399.08	13.26
Road detection based on CNN Gibbs (unlabeled)	9619.92	17.24

4.3 Algorithm verification and discussion

In order to further verify the effectiveness of this algorithm, this paper makes supplementary experiments to compare open source dataset. The open source dataset comes from literature [51], which includes four weather conditions: Dusk, rain, night and sunny, with 1399, 819, 2167, 1995 pictures respectively. In this paper, four groups of weather conditions are tested and compared respectively, and the final results are shown in Table 4.

Table 4
Results on UAS

Method Dusk set Night set Rain set Sun set

FCN 94.6% 89.9% 93.1% 95.2%

SFMPC 91.8% 90.9% 92.2% 93.2%

CPF.&CNN-7. 93.5% 91.9% 92.6% 94.9%

CNN-MRF 93.3% 96.2% 96.6% 94.1%

Our approach 98.3% 96.2% 98.1% 99.0%

Method	Dusk set	Night set	Rain set	Sun set
FCN	94.6%	89.9%	93.1%	95.2%
SFMPC	91.8%	90.9%	92.2%	93.2%
CPF.&CNN-7.	93.5%	91.9%	92.6%	94.9%
CNN-MRF	93.3%	96.2%	96.6%	94.1%
Our approach	98.3%	96.2%	98.1%	99.0%

It can be seen from Table 4 that the accuracy of this paper is the highest on Dusk set, Rain set and Sun set, and it also has a high accuracy on Night set.

The advantages of real-time unstructured road drivable area detection algorithm based on CNN Gibbs energy function can not only overcome the problem of poor robustness caused by the uncertainty of road surface and structural features, but also solve the problem of poor timeliness caused by complex detection model and high computational complexity. The deficiency of this paper is that the detection effect is not good for inclined roads. When the road is inclined, it is easy to detect the road area as a non-road area.

5 Conclusions

(1) In this paper, the smoothness of the road is used as the criterion to judge the driving area, Therefore, the algorithm is not affected by the inherent appearance characteristics of the road, illumination and driving environment. In the actual scene of unstructured road detection, this algorithm not only has high robustness (accuracy: 98.3%) and it takes less time (58frames/s), it can meet the needs of the actual scene and realize online detection.

(2) In order to improve the accuracy and efficiency of homography matrix estimation, this paper first uses the improved fast feature point extraction algorithm to quickly extract stable and directional feature points from DOG; Then, the SIFT matching method is used to optimize the matching results and eliminate the mismatched points; Finally, the LM nonlinear optimization method is introduced to iteratively optimize H by minimizing the cost function.

(3) In order to improve the detection accuracy of the first road image, this paper automatically extracts the high-dimensional features of homography matrix in different regions based on CNN, classifies them and outputs the road regions. In the process of network training, RD-CNN has high accuracy, and the network structure is simple, which makes the detection time shorter.

(4) In order to speed up the detection of subsequent similar frames, this paper uses the detection results of the first image and constructs the road color and texture GMMs model based on orchard-bouman, then, Gibbs energy function is used to assign each pixel to road label or non-road label, finally, the first image is redefined according to the update strategy of GMMs model to realize the continuous detection of unstructured road in real scene.

References

Research Institute of Highway Ministry of Transport, The Blue Book of Road Safety in China 2014 [M], Beijing: China Communications Press; 2015, p. 13–14.

, Li

, Mu

, et al., Motor Anomaly Detection for Unmanned Aerial Vehicles Using Reinforcement Learning[J], IEEE Internet of Things Journal PP(99) (2017), 1–1.

Broggi

, Zelinsky

, Parent, et al., Intelligent vehicles[M], Handbook of Robotics, 2016.

Ziomek

J.F.

, Intelligent Vehicle Technology and Trends – [Book review] [J], Instrumentation & Measurement Magazine IEEE 9(1) (2006), 54–54.

Martinez

F.J.

, Toh

C.K.

, Cano

J.C.

, et al., Emergency Services in Future Intelligent Transportation Systems Based on Vehicular Communication Networks[J], IEEE Intelligent Transportation Systems Magazine 2(2) (2010), 6–20.

Francisco Javier Ariza-López and José Luis García Balboa, Generalization-oriented road line segmentation by means of an artificial neural network applied over a moving window[J], Pattern Recognition 41(5) (2008), 1593–1609.

Ardiyanto

and Adji

T.B.

, Deep residual coalesced convolutional network for efficient semantic road segmentation[J], IPSJ Transactions on Computer Vision & Applications 9(1) (2017).

Kim

T.H.

, Lim

K.Y.

, Byun

H.R.

, et al., Road Sign Detection with Weather/Illumination Classifications and Adaptive Color Models in Various Road Images[J], Anesthesiology 4(11) (2015), 521–528.

Yao

, Ramalingam

, Taguchi

, et al., Estimating drivable collision-free space from monocular video[C], Applications of Computer Vision, 2015.

10.

Wang

, Huang

and Zhong

, Algorithm for urban road detection based on uncertain Bezier deformable template[J], Journal of Mechanical Engineering 49(8) (2013), 143–150.

11.

Cheng

, Qian

and Elder

J.H.

, Fusing Geometry and Appearance for Road Segmentation[C], IEEE International Conference on Computer Vision Workshop, IEEE, 2018.

12.

Wang

, Gu

, Zhang

, Lane boundary detection based on parabola model, In: IEEE International Conference on Information and Automation, Harbin: IEEE; 2010, p. 1729–1734.

13.

Kong

, Audibert

J.Y.

and Ponce

, Vanishing point detection for road detection, In: Computer Vision and Pattern Recognition (CVPR), 2009 IEEE Conference on. IEEE; 2009, p. 96–103.

14.

Fernández

, Izquierdo

, Llorca

D.F.

and Sotelo

M.A.

, A comparative analysis of decision trees-based classifiers for road detection in urban environments, Proceedings of the IEEE International Conference on Intelligent Transportation Systems (2015), 719–24 ;.

15.

Cai

, Li

, Zhou

, et al., Robust Drivable Road Region Detection for Fixed-Route Autonomous Vehicles Using Map-Fusion Images[J], Sensors 18(12) (2018).

16.

Geng

, Sun

, Xiao

, et al., Combining CNN and MRF for road detection[J], Computers & Electrical Engineering 70 (2017), 103–113.

17.

Mendes

C.C.T.

, Frémont

and Wolf

D.F.

, Exploiting fully convolutional neural networks for fast road detection, IEEE International Conference on Robotics and Automation, IEEE; 2016.

18.

Brostow

G.J.

, Shotton

, Fauqueur

, et al., Segmentation and Recognition Using Structure from Motion Point Clouds[C], European Conference on Computer Vision, Springer, Berlin, Heidelberg, 20081(44–57).

19.

Yuan

, Jiang

and Wang

, Video-based road detection via online structural learning[J], Neurocomputing 168 (2015), 336–347.

20.

Guo

, Mita

and Mcallester

, Robust Road Detection and Tracking in Challenging Scenarios Based on Markov Random Fields with Unsupervised Learning[J], Intelligent Transportation Systems, IEEE Transactions on 13(3) (2012), 1338–1354.

21.

Dipanda

and Woo

, Towards a real-time 3D shape reconstruction using a structured light system[J], Pattern Recognition 38(10) (2005), 1632–1650.

22.

Jia

, Chen

and Zhang

, Drivable Road Reconstruction for Intelligent Vehicles Based on Two-View Geometry[J], IEEE Transactions on Industrial Electronics 64(5) (2017), 3696–3706.

23.

Kubota

, Nakano

and Okamoto

, A Global Optimization Algorithm for Real-Time On-Board Stereo Obstacle Detection Systems[C], IEEE Intelligent Vehicles Symposium, IEEE (2007), 7–12.

24.

Nguyen

V.D.

, Nguyen

H.V.

, Tran

D.T.

, et al., Learning Framework for Robust Obstacle Detection, Recognition, and Tracking[J], IEEE Transactions on Intelligent Transportation Systems 18(6) (2016), 1633–1646.

25.

Fan

and Yuan

, On the quadratic convergence of the Levenberg-Marquardt method without nonsingularity assumption[J], Computing 74(1) (2005), 23–39.

26.

Bui

D.T.

, Pradhan

, Lofman

, et al., Landslide susceptibility assessment in the Hoa Binh province of Vietnam: A comparison of the Levenberg-Marquardt and Bayesian regularized neural networks[J], Geomorphology 169-170 (2012), 12–29.

27.

Wang

, Li

, Shao

, et al., Adaptive Harris corner detection algorithm based on iterative threshold[J], Modern Physics Letters B 31(15) (2017), 1750181.

28.

Delponte

, Isgrò

, Odone

, et al., SVD-matching using SIFT features[J], Graphical Models 68(5-6) (2006), 415–431.

29.

Schwind

, Suri

, Reinartz

, et al., Applicability of the SIFT operator to geometric SAR image registration[J], International Journal of Remote Sensing 31(2010-04) (2010), 1959–1980.

30.

Smith

S.M.

and Brady

J.M.

, SUSAN—A New Approach to Low Level Image Processing[J], International Journal of Computer Vision 23(1) (1997), 45–78.

31.

Wang

Y.T.

, Chi

C.T.

and Feng

Y.C.

, Robot mapping using local invariant feature detectors[J], Engineering Computations 31(2) (2014), 297–316.

32.

Possa

P.R.

, Mahmoudi

S.A.

, Harb

, et al., A Multi-Resolution FPGA-Based Architecture for Real-Time Edge and Corner Detection[J], Computers IEEE Transactions on 63(10) (2014), 2376–2388.

33.

Benjelloun

and Mahmoudi

, Spine Localization in X-ray Images Using Interest Point Detection[J], Journal of Digital Imaging 22(3) (2009), 309–318.

34.

Zhang

L.M.

, Fast and robust corner detector based on double-circle mask[J], Optical Engineering 49(12) (2010), 1–8.

35.

Kawakita

, Iwasawa

, Lopez-Gulliver

, et al., Glasses-free large-screen three-dimensional display and super multiview camera for highly realistic communication[J], Optical Engineering 57(6) (2018), 1.

36.

Yingzhong

, Huijuan

, Haoyang

, et al., Three-dimensional surface microtopography recovery from a multifocus image sequence using an omnidirectional modified Laplacian operator with adaptive window size[J], Applied Optics 56(22) (2017), 6300.

37.

Cherri

A.K.

and Karim

M.A.

, Optical symbolic substitution: Edge detection using Prewitt, Sobel, and Robert’s operators[J], Applied Optics 28(21) (1989), 4644–4648.

38.

Özdemir

, Akarun

and Akarun

, A fuzzy algorithm for color quantization of images[J], Pattern Recognition 35(8) (2002), 1785–1791.

39.

Kong

, Audibert

J.Y.

and Ponce

, Vanishing point detection for road detection, In: Computer Vision and Pattern Recognition (CVPR), 2009 IEEE Conference on, IEEE; 2009, p. 96–103.

40.

Mendes

C.C.T.

, Frémont

and Wolf

D.F.

, Exploiting fully convolutional neural networks for fast road detection, IEEE International Conference on Robotics and Automation, IEEE; 2016.

41.

Brostow

G.J.

, Shotton

, Fauqueur

, Cipolla

, Segmentation and recognition using structure from motion point clouds, In: ECCV, vol. 1; 2008, p. 44–57.

42.

Yuan

, Jiang

and Wang

, Video-based road detection via online structural learning, Neurocomputing 168(C) (2015), 336–347.

43.

Sturgess

, Alahari

, Ladicky

and Torr

P.H.S.

, Combining appearance and structure from motion features for road scene understanding, BMVC’09; 2009.

44.

Ardiyanto

and Adji

T.B.

, Deep residual coalesced convolutional network for efficient semantic road segmentation, IPSJ Trans Comput Vis Appl 9(1) (2017), 6.

45.

Geng

, Sun

, Xiao

, et al., Combining CNN and MRF for road detection[J], Computers & Electrical Engineering (2017), 70.

46.

D’Orazio

, et al., A visual approach for driver inattention detection, Pattern Recognition 40(8) (2007), 2341–2355.

47.

Torr

P.H.

and Zisserman

, Mlesac: A new robust estimator with application to estimating image geometry, Computer Vision and Image Understanding 78(1) (2000), 138–156.

48.

Chum

, Matas

and Kittler

, Locally optimized ransac, In Proceedings of the joint pattern recognition symposium, Springer, (2003), pp, 236–243.

49.

Lowe

D.G.

, Distinctive Image Features from Scale-Invariant Keypoints[J], International Journal of Computer Vision 60(2) (2004), 91–110.

50.

Bochkovskiy

, Wang

C.Y.

and Liao

, YOLOv4: Optimal Speed and Accuracy of Object Detection, (2020).

51.

Zhang

, Chen

, He

, et al., Road segmentation for all-day outdoor robot navigation, Neurocomputing 314 (2018), 316–325.

Real-time unstructured road detection based on CNN and Gibbs energy function

Abstract

Keywords

1 Introduction

2 Road detection based on planar homography and CNN

2.3.1 Structure of CNN

3 Road detection based on color and texture features

3.1 Building color GMMs model

3.3 Road detection based on Gibbs energy function

4 Experiment

4.1 Experimental data set

Table 4 Results on UAS Method Dusk set Night set Rain set Sun set FCN 94.6% 89.9% 93.1% 95.2% SFMPC 91.8% 90.9% 92.2% 93.2% CPF.&CNN-7. 93.5% 91.9% 92.6% 94.9% CNN-MRF 93.3% 96.2% 96.6% 94.1% Our approach 98.3% 96.2% 98.1% 99.0%

References

Table 4
Results on UAS

Method Dusk set Night set Rain set Sun set

FCN 94.6% 89.9% 93.1% 95.2%

SFMPC 91.8% 90.9% 92.2% 93.2%

CPF.&CNN-7. 93.5% 91.9% 92.6% 94.9%

CNN-MRF 93.3% 96.2% 96.6% 94.1%

Our approach 98.3% 96.2% 98.1% 99.0%