Marked and unmarked speed bump detection for autonomous vehicles using stereo vision

Abstract

A current challenge for autonomous vehicles is the detection of irregularities on road surfaces in order to prevent accidents; in particular, speed bump detection is an important task for safe and comfortable autonomous navigation. There are some techniques that have achieved acceptable speed bump detection under optimal road surface conditions, especially when signs are well-marked. However, in developing countries it is very common to find unmarked speed bumps and existing techniques fail. In this paper a methodology to detect both marked and unmarked speed bumps is proposed, for clearly painted speed bumps we apply local binary patterns technique to extract features from an image dataset. For unmarked speed bump detection, we apply stereo vision where point clouds obtained by the 3D reconstruction are converted to triangular meshes by applying Delaunay triangulation. A selection and extraction of the most relevant features is made to speed bump elevation on surfaces meshes. Results obtained have an important contribution and improve some of the existing techniques since the reconstruction of three-dimensional meshes provides relevant information for the detection of speed bumps by elevations on surfaces even though they are not marked.

Keywords

Speed bump detection road segmentation stereo vision triangular surface meshes machine learning

1 Introduction

Autonomous vehicles have been able to circulate in countries with optimal road infrastructure conditions, such as smart city roads. Some algorithms have been developed for these vehicles to perform perception, control and decision-making tasks to move from an origin to a destination safely, avoiding different obstacles on roads [8]. Autonomous cars are equipped with some sensors such as: video cameras, stereo cameras, LIDAR (LIght Detection And Ranging) devices, ultrasonic sensors, among others, to perceive different elements surrounding the road traffic [6].

One of the great challenges for safe and comfortable autonomous navigation is the detection of irregularities by monitoring road surface conditions [2 , 31]. Irregularities that have been frequently studied (because they affect the stability in vehicle navigation) are speed bumps (SB) and potholes. In developing countries, it is very common to find road irregularities with inadequate traffic signs that make detection difficult which could cause vehicle damages.

Speed bump detection is used in some of self-driving car applications, some of them are the speed reduction when detecting SB and the steering angle prediction to avoid accidents or damages that affect the safety of the passengers, in addition, it can provide an adequate performance in the vehicle in scenarios with high uncertainty [7 , 29].

In this work, results for marked speed bump detection are presented and some experimental results about the pre-processing phase for the detection of unmarked speed bump are presented. Therefore, a novel methodology for both marked and unmarked speed bump detection is presented to help analyze irregularities on road surfaces of developing countries. Our proposal is based on image processing, stereo vision and supervised learning.

2 Related work

Speed bump detection on roads has been a topic of interest for several years and recently a review of the most relevant techniques is reported in [28]. These techniques try to improve the precision in the SB detection under different conditions such as unmarked signs, poor visibility and weather adversities. Approaches that have been used in recent years for the detection of speed bumps are: image-based techniques, stereo vision, LIDAR sensors, smartphone sensors, and hybrid techniques that combine more than one approach.

Some works have applied image processing techniques and machine learning to detect speed bumps [16]. Basic image processing techniques, such as the Otsu thresholding method and morphological operators, have been useful to analyze SB signaling patterns when they are clearly painted and even when the pattern is noisy [5]. In addition, techniques such as texture and color analysis, contour extraction and segmentation of areas of interest to a set of images have been used, where the segmented region is compared with patterns of SB formed by diagonal lines [10, 13].

Likewise, features have been extracted via Local Binary Patterns and a classification model has been trained with Support Vector Machines (SVM) for SB detection [22]. Advanced techniques using deep learning, in particular, a deep convolutional neural network architecture for semantic pixel-wise segmentation has been applied to segment SB into image datasets [4].

The main advantage of segmentation algorithms is that they use inexpensive cameras for image acquisition. However, the precision of speed bump detection is low due to the fact that it is restricted to the visualization of the road marking, otherwise it is hard to detect speed bumps with these techniques. In addition, when signs of SB are very diverse it is difficult to apply the same vision technique to all types of patterns.

Despite the fact that these works presents results with high precision, they occurs when there is a good illumination in the environment, thus these techniques require optimal weather conditions, and mainly they work adequately for the detection of clearly painted SB but for unmarked speed bumps this technique fails.

Another approach for speed bump detection is stereo vision that is combined with deep learning, where three-dimensional scene reconstruction has been done from the estimation of stereo images obtained by a ZED camera [32]. Despite the fact that results of this work reveal an approximate precision of 93.88%, ZED camera does not work properly outdoors. The main advantage of these algorithms is that they use low-cost sensors, however, their great disadvantage is that the range is too short because the separation between the camera lenses is very small, so they do not work in real scale environments.

The use of LIDAR sensors has been very popular in autonomous car navigation. Through these sensors it is possible to obtain data of scenes by generating point clouds. A method that detects bumps according to appropriate threshold values to distinguish between a speed bump and other objects has been developed [25]. Despite the high precision of these sensors, they have some limitations: their cost is very high, they do not follow internationally compatible standards, they require high computing power to process large amounts of data, in addition, the signals from several sensors can overlap and generate errors [20, 26].

Another approach for speed bump detection is the use of smartphone sensors for capturing vibrations on road surfaces by analyzing acceleration signals [24]. Accelerometer and gyroscope sensors have been used to gather vibration data when vehicles pass through some SB and a GPS receiver to capture the geographical location coordinates [1, 3].

Mobile applications have been developed to send timely alerts to passengers by detecting SB in real time through the use of smartphone sensors and crowdsourcing where users share and get data [34]. Signal processing techniques and more advanced techniques such as genetic algorithms have been applied to detect SB from the collected data [9, 14]. The advantage of these applications is that their design is simple, inexpensive and within the reach of any user. Smartphone sensors have high precision errors, long delays when used in real time, are sensitive to environmental conditions and overload the network to which they are connected. Alert applications are useful when SB have a fixed location, however, for unplanned events, when new SB are placed without updating their location, they cannot be detected. Despite the various disadvantages, these technologies could be complemented with other techniques to improve SB detection accuracy.

There are works that combine some mentioned techniques to detect speed bumps in order to improve the individual performance of each one. The use of a LIDAR sensor has been combined with a camera to find speed bump through image pattern recognition with zebra crossing [37]. The LIDAR sensor has been used to estimate the distance to the SB. This method is restricted to the detection of clearly marked SB, otherwise it fails. A real-time system has been built which combines LIDAR sensors and digital image processing to detect SB with zebra cross patterns [18]. Results reveal that when the car travels at a standard speed, the detection of SB is performed with a 94% accuracy. However, when there are many vehicles with LIDAR sensors, their signals could overlap and generate errors.

Table 1 shows a summary of some works related to speed bump detection, results with high accuracy have been reported in the literature, but some limitations are observed. Techniques based on image processing are focused on signaling pattern recognition and they fail when the SB is not marked. The current results using stereo vision are good but some systems such as ZED cameras not work for outdoors conditions or real scales environments due to their short range optic. The use of smartphone provides acceptable results, and these sensors are widely used for the detection of SB as irregularities on the surfaces, but network overload, high delays and false alarms can occur caused by the error of these sensors.

Table 1
Summary of some related works to speed bump detection

Author Approach Sensors Employed Techniques Results Limitations

Irhebhude et al. [22] Image processing based

Cameras Hough transform, LBP and SVM 100% accuracy It does not work for unmarked SB

Babu et al. [5] Otsu’s algorithm, morphological operations 90% detection rate for marked SB

Devapriya et al. [16] Morphological operators 85% accuracy for marked SB

Varma et al. [32] Stereo vision ZED stereo camera CNN and stereo vision 97.44% accuracy for marked SB 93.88% for unmarked SB ZED cameras have short range and high error outdoors

Lee et al. [25] LIDAR sensors Accelerometer, LIDAR and

IMU sensor Hydraulic circuit for suspension control 78.6% accuracy at 5m

17.9% error rate Low accuracy and there are no international standars

Celaya-Padilla et al. [9] Gyroscope, accelerometer, and a GPS Genetic algorithm 97.1% accuracy, 1.8% FPR and 97.8% AUC

Al-Shargabi et al. [1] Smartphone sensors Accelerometer and GPS Noise filtering Android application High sensor error, network overload, high delays and false alarm

Vimalkumar et al. [34] Accelerometer and GPS Crowdsourcing Android alert application

Fernández et al. [18] Hybrid techniques LIDAR sensors and cameras LIDAR sensor, and computer vision 94% detection rate Multiple LIDAR can overlap signals

Author	Approach	Sensors	Employed Techniques	Results	Limitations
Irhebhude et al. [22]	Image processing based
		Cameras	Hough transform, LBP and SVM	100% accuracy	It does not work for unmarked SB
Babu et al. [5]			Otsu’s algorithm, morphological operations	90% detection rate for marked SB
Devapriya et al. [16]			Morphological operators	85% accuracy for marked SB
Varma et al. [32]	Stereo vision	ZED stereo camera	CNN and stereo vision	97.44% accuracy for marked SB 93.88% for unmarked SB	ZED cameras have short range and high error outdoors
Lee et al. [25]	LIDAR sensors	Accelerometer, LIDAR and
		IMU sensor	Hydraulic circuit for suspension control	78.6% accuracy at 5m
				17.9% error rate	Low accuracy and there are no international standars
Celaya-Padilla et al. [9]		Gyroscope, accelerometer, and a GPS	Genetic algorithm	97.1% accuracy, 1.8% FPR and 97.8% AUC
Al-Shargabi et al. [1]	Smartphone sensors	Accelerometer and GPS	Noise filtering	Android application	High sensor error, network overload, high delays and false alarm
Vimalkumar et al. [34]		Accelerometer and GPS	Crowdsourcing	Android alert application
Fernández et al. [18]	Hybrid techniques	LIDAR sensors and cameras	LIDAR sensor, and computer vision	94% detection rate	Multiple LIDAR can overlap signals

In this work is presented a novel approach based on the calculus of a 3D surface mesh, which is obtained from a full-scale stereo camera system. With this approach is possible to detect SB including scenarios when they are not well-marked. Section 3 shows the methodology proposed for speed bump detection.

3 Methodology

In this section, a novel methodology for detecting both painted and unpainted speed bumps on road surfaces are presented. Fig. 1 shows the proposed methodology and each stage is described below.

Fig. 1

Proposed methodology for marked and unmarked speed bump detection.

3.1 Stereo image capture

Using stereo vision, three-dimensional environments can be reconstructed from pairs of two dimensional images [21]. Therefore, a stereo vision system is employed which includes two cameras with the same characteristics. Simultaneous images are captured and processed to recover depth information from scenes. In this way, a dataset of road stereo images with and without speed bumps is built.

3.2 Road segmentation

Semantic segmentation is a deep learning approach that assigns a color to each pixel in an image according to the category to which it belongs. Deep learning is one of the most accurate in classifying images, which uses an amount of hidden layers in a neural network. In this stage, a pre-trained model is employed, applying the Fully Convolutional Network (FCN) architecture of the Fig. 2, for road semantic segmentation. The FCN is a particular case of a CNN (Convolutional Neural Network) where the input is each image and the output is the class (road or non-road) it belongs.

Fig. 2

Network architecture for road segmentation applying a FCN model [27].

Fig. 3 shows the result of the road semantic segmentation where the predicted region is colored. Applying a mask over the original image removes the area of no interest and the region of interest (ROI) is cropped.

3.3 Image pre-processing

From segmented road images, for marked speed bump detection, the following operations are applied as pre-processing:

Gray scale image conversion: using the luminosity equation L (P_R, P_G, P_B) =0.21P_R + 0.72P_G + 0.07P_B, where P_R, P_G, P_B are the R, G, B components of pixel P in the image I (u, v) ∈ [0, 255].

Contrast limited adaptive histogram equalization (CLAHE): a histogram h for a grayscale image I (u, v) is calculated by Equation 1: $h (i) = card {(u, v) | I (u, v) = i} .$ (1) where h (i) = the number of pixels in I with the intensity value i. Histogram equalization (HE) is obtained from the cumulative histogram H as follows: $H (i) = \sum_{j = 0}^{i} h (j), 0 \leq i < 255$ . Applying CLAHE improves the contrast of the images dividing each one into blocks 4x4 called tiles and the histogram of each one is equalized.

Gaussian filtering: employed for softening images. A 5x5 kernel is used and the Gaussian function $G (u, v) = \frac{1}{2 π σ^{2}} e^{- (\frac{u^{2} + v^{2}}{2 σ^{2}})}$ is applied where u, v are the positions of the image pixels in each RGB channel, and σ is the parameter of Gaussian standard deviation.

Otsu thresholding: the threshold value is calculated in such a way that the dispersion within each segment is as small as possible, but at the same time the dispersion is as high as possible between different segments. According to the threshold value, the images are binarized, separating the speed bump signs from the road which corresponds to the background. Fig. 4 a) shows the result of segmentation by thresholding.

Morphological operators: these operators are used to enhance the shape and contour of the SB signs. An opening operator is applied which performs the erosion operation followed by dilation as follow: A ∘ B = (A ⊖ B) ⊕ B, where A is an input Binary image, and B is the structure element. After, a closing operator is applied which performs the dilation followed by erosion operation as follows: A • B = (A ⊕ B) ⊖ B. A 3x3 kernel is used for the application of both opening and closing operators.

Gabor filters: it is a linear filter whose impulse response is a sinusoidal function that is multiplied by a Gaussian function. They are used to highlight the gradients corresponding to the outline of the marking pattern that facilitates speed bump detection. Fig. 4 b) shows results of applying Gabor filters to the segmented images.

Fig. 3

Road semantic segmentation using a pre-trained FCN model. Dataset obtained from [33].

3.4 Marked SB classification

After the images have been pre-processed and segmented in the previous stages, a supervised binary classification model is trained to predict images with or without speed bump by following these steps:

Feature extraction: The local binary patterns (LBP) histogram is the feature vector used to train a predictive model where, according to its value, the class to which each image belongs is labeled. LBP is a descriptor that calculates the local representation of textures that is constructed by comparing each pixel around its neighbors, generating an encoding. The total points considered for LBP feature extraction is 24, the neighborhood radius used is 8 pixels and the uniform method is applied for coding regions.

Model training: The set of images is divided into 80% for training and 20% for tests. To train the model, the support vector machines (SVM) algorithm is applied, which is a robust method and, unlike other classification algorithms, does not require high computational cost. A binary linear classifier is trained where the two classes are separated by an hyperplane, defined as the vector between the two points, and the good separation between classes generates a correct classification. Also Fig. 5 shows two examples of successful cases of the SB classification, a) true positive case (TP) and b) true negative case TN. Fig. 5 shows the result of the binary classification of images with and without SB.

Model evaluation: to measure the model performance, confusion matrix, evaluation metrics (precision, recall, and accuracy) and ROC curves (receiver operating characteristic) are calculated from the test images set. In section 4 results of the classification model evaluation are presented.

Fig. 4

Application of image pre-processing techniques.

Fig. 5

Classification results obtained from the predictive model.

3.5 Disparity map generation

The following phases of stereo vision are applied to reconstruct the three-dimensional scenes: camera calibration, image rectification and generation of disparity maps [13]. A disparity map estimates the depth of objects from a stereo image pair. Differences between the pixels of each stereo pair are calculated on the segmented road images. The pairs of corresponding points that are projections of the same point in space are found. Fig. 6 shows the disparity maps obtained for a road with and without SB.

Fig. 6

Generated disparity maps.

3.6 Point cloud reconstruction

3D reconstruction of the segmented road is generated using a disparity map and applying a triangulation process of stereo vision to estimate the different points (x, y, z) of the scene [12]. Fig. 7 shows the 3D road reconstruction with and without SB using point clouds.

Fig. 7

3D reconstruction using point clouds.

3.7 Surface mesh construction

Unstructured point clouds are used to generate three-dimensional surface meshes applying the Delaunay triangulation method [36]. The Delaunay condition of a triangle indicates that the circumscribed circle must not contain any other vertex of the triangulation in its interior. Due to the difficult handling and high computational cost to process the number of mesh vertices, a simplification of the 3D mesh triangles is performed. The objective is to reduce the size of the meshes while preserving their shape and topology and two methods are applied: decimate and pro-decimate.

The mesh decimation algorithm classifies each vertex using a distance measure between points; if a certain decimation achieve the vertex is removed and the resulting hole is triangulated. The pro-decimation algorithm generates progressive meshes to improve the basic decimation algorithm where a priority queue is used to place the ranked vertices. Furthermore, the mesh is divided into submeshes and each one is processed recursively. In Fig. 8 results of reducing the triangular meshes are displayed using the two mentioned methods, both with a reduction percentage of 80%; notice that, although the size of the meshes is reduced, the original shape of the road surface is preserved.

Fig. 8

Triangle mesh for 3D road surfaces.

3.8 3D mesh processing

The three-dimensional meshes are processed in order to make them more uniform and to remove outliers. The first mesh processing technique applied to generated surfaces is Laplacian smoothing. This algorithm is used to smooth polygonal meshes where a new position is chosen for each vertex according to the local information of its neighbors. Eq. 2 calculates the Laplacian to smooth a 3D mesh.

$\bar{x_{i}} = \frac{1}{N} \sum_{j = 1}^{N} x_{j}$ (2) where N = number of vertices adjacent to node i, x_j is the position of the j-th adjacent vertex and $\bar{x_{i}}$ is the new position of node i. Fig. 10 shows the result of 3D mesh smoothing where noisy vertices are removed from the surface.

Fig. 9

Decimate triangle meshes preserving their topology.

Fig. 10

Laplacian smoothing of triangle meshes.

3.9 Unmarked SB classification

From processed road surface meshes, a predictive model to detect unmarked speed bumps due to elevations on surfaces is generated according to the following steps:

Feature extraction: A set of features are extracted from the processed 3D surface meshes. Some features based on eigenvalues are: linearity, planarity, sphericity, curvatures and anisotropy.

Model training: From the generated feature vector, supervised classification algorithms are applied. The model makes a binary classification of 3D meshes with and without unmarked speed bump by elevations on the road surfaces.

Model evaluation: to evaluate the performance of the classification model, some metrics are applied and the corresponding ROC curves are calculated. In addition, an analysis is made of the algorithm that presents classification results with greater precision.

3.10 MUSBD Algorithm

The general pseudocode of the MUSBD (Marked and Unmarked Speed Bump Detection) algorithm, proposed in this work is depicted in Algorithm 1 and it is summarized below.

The input to the algorithm is a stereo image sequence <I₁, . . . , I_n>, where the i - th image is represented as I_i = (L_i, R_i) and the output returns a class C where each class indicates whether or not SB is detected. The first task is the camera calibration in order to obtain intrinsic and extrinsic parameters; the algorithm loops through each stereo pair (L_i, R_i) performing the following functions:

Algorithm 1 MUSBD algorithm

1: procedure Main<I₁, . . . , I_n>

2: S_p ← stereoCalibration ()

3: for all < I_i> do

4: $L_{i}^{S} \leftarrow ROAD_SEGMENT (L_{i})$

5: $R_{i}^{S} \leftarrow ROAD_SEGMENT (R_{i})$

6: $L_{i}^{P} \leftarrow IMAGE_PROCESS (L_{i}^{S})$

7: $d_{i} \leftarrow MARK_SB_DET (L_{i}^{P})$

8: ifd_i = TRUEthen

9: C ← " SB "

10: else

11: $m_{i} \leftarrow MESH_GEN (L_{i}^{S}, R_{i}^{S}, S_{p})$

12: d_i ← UNMARK _ SB _ DET (m_i)

13: ifd_i = TRUEthen

14: C ← " SB "

15: else

16: C ← " NO _ SB "

17: end if

18: end if

19: end for

20: returnC

21: end procedure

22: procedure ROAD_SEGMENTI_i

23: P_i ← bilateralFilter (I_i)

24: S_i ← segmentationFCN (P_i)

25: ROI_i ← cropROI (S_i)

26: returnROI_i

27: end procedure

28: procedure MARK_SB_DETL_i

29: FV_i ← extractFeatures (L_i)

30: d_i ← testTrainedModel (FV_i)

31: returnd_i

32: end procedure

33: procedure MESH_GEN $L_{i}^{S}, R_{i}^{S}, S_{p}$

34: $(L_{i}^{R}, R_{i}^{R}) \leftarrow stereoRect (L_{i}^{S}, R_{i}^{S}, S_{p})$

35: ${DM}_{i} \leftarrow generateDM_SGBM (L_{i}^{R}, R_{i}^{R})$

36: ${DM}_{i}^{P} \leftarrow WLSFilter ({DM}_{i})$

37: ${PC}_{i} \leftarrow generatePointClouds ({DM}_{i}^{P})$

38: mesh_i ← convertPCtoMesh (PC_i)

39: ${mesh}_{i}^{S} \leftarrow meshSimplification ({mesh}_{i})$

40: ${mesh}_{i}^{P} \leftarrow processMesh ({mesh}_{i}^{S})$

41: return ${mesh}_{i}^{P}$

42: end procedure

43: procedure UNMARK_SB_DETmesh_i

44: FV_i ← extractFeatures (mesh_i)

45: d_i ← testTrainedModel (FV_i)

46: returnd_i

47: end procedure

ROAD _ SEGMENT (Ii): this function receives a stereo image pair I_i and returns a pair of segmented road images ROI_i. A bilateral filter is applied to smooth the images while maintaining the edges (bilateralFilter (I_i)). A pre-trained FCN model is used for the semantic road segmentation (segmentationFCN (P_i)). The road region of interest is cropped (cropROI_i (S_i)) removing irrelevant information.

MARK _ SB _ DET (L_i): features are extracted from LBP histograms (extractFeatures (L_i)) and a model is trained using SVM for classification. The model is tested from input images using the function (testTrainedModel (FV_i)) and the model performance is validated by calculating metrics: precision, recall, and accuracy.

$MESH_GEN (L_{i}^{S}, R_{i}^{S}, S_{p})$ : this function generates 3D meshes from each pair of stereo images segmented with the ROI_i. For that, the left ( $L_{i}^{S}$ ) and right ( $R_{i}^{S}$ ) stereo images are rectified from the stereo calibration parameters ( $stereoRect (L_{i}^{S},$

$R_{i}^{S}, S_{p})$ ). A disparity map is generated by applying the semi-global block matching (SGBM) algorithm to find the correspondence between pixels [21].

A weighted least squares (WLS) filter is applied to smooth and make the disparity map more uniform (WLSFilter (DM_i)). 3D point clouds are generated from the processed disparity map ( $generatePointClouds ({DM}_{i}^{P})$ ). Point clouds are converted to triangular meshes by applying the Delaunay triangulation condition (convertPCtoMesh (PC_i)). The total number of points is simplified by applying the decimate and pro-decimate methods (meshSimplificati–on (mesh_i)). Finally, three-dimensional meshes are processed to make them more uniform.

UNMARK _ SB _ DET (mesh_i): Geometric features and eigenvalues are extracted from meshes (extractFeatures (mesh_i)) and some supervised learning algorithm can be trained to classify road surface meshes with and without speed bumps by elevations. Models are tested (testTrai-nedModel (FV_i)) and the one with the best performance is selected by calculating evaluation metrics.

Fig. 11

Road images captured with the stereo vision system.

4 Experimental results

In this section, some results obtained are shown considering a set of stereo images captured under environment conditions specified previously. For the capture, a stereo vision system of two Cannon Vixia HR R72 camcorders separated by a distance of 20cm were used. A set of 500 stereo road images with and without speed bumps has been captured, where experiments have been made under real conditions based on an illumination between 32,000 lux and 100,000 lux (see some examples in Fig 11). Algorithm 1 is implemented in the Python language and is applied to the set of stereo images.

4.1 Road segmentation

Fig. 12 presents a comparison between true and predicted semantic segmentation from two road images. To evaluate the performance of the model, two metrics are applied to a set of images of each stereo pair. A successful prediction is one where the overlap between estimated and true classes is maximized. Two widely used metrics for their high efficiency are the Jaccard index and the Dice coefficient. The Jaccard index, also known as IoU or intersection on union, is the area of overlap between the predicted segmentation and the ground truth or true divided by the area of the union, the predicted segmentation and the ground truth (see Equation 3). On the other hand, the Dice coefficient is twice the overlap divided by the total number of pixels in both images (see Equation 4). Both metrics vary between 0 and 1 where 0 indicates that there is no overlap and 1 corresponds to a perfectly overlapped segmentation. $IoU = \frac{Intersection}{Union} = \frac{TP}{TP + FP + FN}$ (3) $Dice = \frac{2 * Inter .}{Uni . + Inter .} = \frac{2 TP}{2 TP + FP + FN}$ (4)

Fig. 12

Comparison between ground truth and predicted road segmentation.

In Equation 3 and 4 TP are the true positives cases, FP the false positives and FN the false negatives. From a set of 500 stereo images, both evaluation metrics are calculated and averaged to obtain the final results. The IoU mean obtained is 91.6% while the Dice mean obtained is 93.8%. This shows that results obtained with the FCN model of semantic road segmentation shows high reliability.

4.2 Marked speed bump detection

The marked speed bump detection model is tested and evaluated to measure its performance by calculating the confusion matrices and evaluation metrics (precision, recall, and accuracy). Table 2 shows the numerical results obtained by the model, where the precision obtained is 84%, the recall value is 98% and the accuracy obtained is 89%. The ROC curve associated with the predictive model calculates the proportion of true positives versus the ratio of false positives, the area under the curve (AUC) is 91% (see Fig. 13). Therefore, it is shown that the model has acceptable results in the detection of speed bumps when the signs are well-marked.

Table 2
Metrics to evaluate the model performance

Model Total instances Metrics Value

SVM 500 images with Precision 84%

and without SB Recall 98%

Accuracy 89%

Model	Total instances	Metrics	Value
SVM	500 images with	Precision	84%
	and without SB	Recall	98%
		Accuracy	89%

Fig. 13

ROC curves for marked speed bump classification.

4.3 Unmarked speed bump

Applying camera calibration and stereo rectification generates disparity maps where color intensities indicate the depth of objects in the scene. Fig. 6 a) and b) shows two examples of the disparity map generation of roads with and without speed bumps respectively.

Fig. 7 shows results of the 3D road reconstruction through point clouds applying the stereo vision process. Point clouds are converted to triangular meshes representing road surfaces (Fig. 8). A simplification has been made to surface meshes for reduce their size while preserving their shape (see Fig. 9). The obtained meshes have been smoothed to eliminate outliers generated erroneously by the 3D reconstruction (Fig. 10).

The selected local 3D shape features based on eigenvalues are listed in Table 3 which is the feature vector. Fig. 14 shows the extracted histograms of selected features where a significant difference can be observed between the values of each mesh feature with and without speed bump.

Table 3
Feature selection based on eigenvalues for 3D meshes

Feature Expression

Lineality $L_{λ} = \frac{λ_{1} - λ_{2}}{λ_{1}}$ λ₁, λ₂: eigen

Planarity $P_{λ} = \frac{λ_{2} - λ_{3}}{λ_{1}}$ values of 3D points,

Sphericity $S_{λ} = \frac{λ_{3}}{λ_{1}}$ λ₁ ≥ λ₂ ≥ λ₃ ≥ 0

Curvatures $C_{λ} = \frac{λ_{3}}{\sum_{λ}}$ λ₁ + λ₂ + λ₃ = 1

Anisotropy $A_{λ} = \frac{λ_{1} - λ_{3}}{λ_{1}}$

Feature	Expression
Lineality	$L_{λ} = \frac{λ_{1} - λ_{2}}{λ_{1}}$	λ₁, λ₂: eigen
Planarity	$P_{λ} = \frac{λ_{2} - λ_{3}}{λ_{1}}$	values of 3D points,
Sphericity	$S_{λ} = \frac{λ_{3}}{λ_{1}}$	λ₁ ≥ λ₂ ≥ λ₃ ≥ 0
Curvatures	$C_{λ} = \frac{λ_{3}}{\sum_{λ}}$	λ₁ + λ₂ + λ₃ = 1
Anisotropy	$A_{λ} = \frac{λ_{1} - λ_{3}}{λ_{1}}$

Fig. 14

3D mesh feature extraction.

According to the obtained results, the proposed methodology is able to detect speed bumps by recognition of well-marked signs and also by shape analysis of road surfaces when the signs is not marked using road image datasets of developing countries. To reduce the computational cost, a road segmentation corresponding to the ROI is applied.

It has been shown that the road semantic segmentation, using a pre-trained model of an FCN, presents results with high precision, with a mean IoU of 91.6% and a mean Dice of 93.8%, under controlled environmental conditions. It should be noted that the model has been tested with images without the restriction that the road limits are clearly marked.

There are some works that have addressed the detection of marked speed bumps with characteristics and conditions similar to those considered in this work, where approaches based on image processing (IBP), machine learning (ML) and stereo vision (SV) are applied. The work by Irhebhude et al. [22] combines the IBP and ML approaches and despite obtaining an accuracy of 100% they only consider asphalt roads; Babu et al. [5] and Devapriya et al. [16] apply an IBP approach with an accuracy of 90% and 85% respectively, however, since no machine learning algorithms are applied, the detection becomes inaccurate due to color variations on roads and lighting; Varma et al. [32] combines a stereo vision approach with deep learning where an accuracy of 97.44% is obtained. In the present work, the IBP and ML approaches are combined, obtaining an accuracy for marked SB of 89%, which is acceptable with respect to related work; unlike the related approaches, in this work various colors and road material types are considered such as: asphalt, pavement and cobblestone. Moreover, in the work of Varma et al. [32] the detection of unmarked SB is addressed where an accuracy of 93.88% SB is obtained, however, a ZED stereo camera is used which is short range and has high error outdoors. In contrast to this approach, our work uses long-range video cameras that work well in real-scale scenarios; elevation detection on surfaces is independent of the road material and is more accurate than detection by pattern recognition.

For the detection of speed bumps without marked signaling, the stereo vision process has been applied to obtain depth information. To do this, a 3D road reconstruction that have been converted into triangular meshes has been made, which are optimized to reduce the processing time by decimate to simplify the vertices of the meshes while preserving their shape. 3D meshes are smoothed to remove outliers while preserving elevations. These results can be viewed in sections 3.6, 3.7, and 3.8. A selection of the most relevant features of triangular meshes, based on eigenvalues, has been done, so that the feature vector obtained is: lineality, planarity, sphericity, curvatures, and anisotropy. Surface meshes with and without SB have been analyzed and the histogram of each selected feature has been calculated. These are the ones that have the greatest difference between both classes and therefore help to detect speed bump by elevations on surfaces.

The experiments were executed on a computer with the following characteristics: AMD A10-8700P Radeon R6 1.8 GHz processor, 8 GB RAM and Windows operating system. The average execution times for each task of Algorithm 1 are: road segmentation 0.445 sec, Image preprocessing 0.009 sec, marked SB classification 0.044 sec, disparity map generation 9.845 sec, point cloud generation 39.186 sec, surface mesh 2.986 sec, 3D mesh processing 4.327 sec where the total average time is 56.842 seconds.

5 Conclusions

This work presents a new methodology to detect both marked and unmarked speed bumps applying MUSBD algorithm for decision making of autonomous vehicles. For this purpose, stereo images with a vision system were captured. For road segmentation, a pre-trained convolutional FCN model was tested, obtaining results with high precision with an IoU mean of 91.6% and a dice mean of 93.08% were obtained. Processing techniques were applied to segment the image ROI with well-marked markings, histograms of local binary patterns were extracted, and a classification model was trained using support vector machines. The algorithm detects successfully marked speed bumps with a precision of 84%, a recall of 98%, an accuracy of 89% and an AUC of 91%.

For unmarked speed bumps, stereo vision was applied to generate disparity maps between each pair of stereo images and to make the 3D reconstruction of roads using point clouds. These points were converted to triangular surface meshes applying Delaunay triangulation and a mesh decimation was done to simplify the vertices while preserving the topology. A Laplacian smoothing was applied to triangular meshes in order to eliminate outliers. Selection and extraction of the features, that significantly help to detect speed bumps on road surface meshes, has been made and these are based on eigenvalues: linearity, planarity, sphericity, curvatures, and anisotropy.

Results obtained have an important contribution to the detection of irregularities on road surfaces that affect safety and comfort in autonomous vehicle navigation. In addition to improving existing techniques when considering real-scale environments, various types of road material and acceptable detection results. The total average time for the execution of the proposed algorithm is 56.842 seconds; these times are subject to the hardware used and can be reduced using computers with GPUs.

The future scope of the work is to consider different scenarios, mainly with poor lighting caused by bad weather or by images captured at night, applying alternative techniques when conditions are not ideal. Additionally, a predictive model will be trained for the detection of speed bumps according to the extracted features from 3D surface meshes using some classification algorithms. For that, a more exact geometric feature and based on eigenvalues of the 3D meshes selection will be considered.

References

Al-Shargabi

, Hassan

and Al-Rousan

, A Novel Approach for the Detection of Road Speed Bumps using Accelerometer Sensor, Technology, Education, Management, Informatics Journal9(2) (2020), 469–476.

Alam

M.Y.

, Nandi

, Kumar

, Saha

, Nandi

and Chakraborty

, Crowdsourcing from the True crowd: Device, vehicle, road-surface and driving independent road profiling from smartphone sensors, Pervasive and Mobile Computing61 (2020), 101103.

Aljaafreh

, Alawasa

, Alja’afreh

and Abadleh

, Fuzzy inference system for speed bumps detection using smart phone accelerometer sensor, Journal of Telecommunication, Electronic and Computer Engineering (JTEC)9(2–7) (2017), 133–136.

Arunpriyan

, Variyar

V.S.

, Soman

K.P.

and Adarsh

, Real-Time Speed Bump Detection Using Image Segmentation for Autonomous Vehicles, International Conference on Intelligent Computing, Information and Control Systems (2019), 308–315.

Babu

C.N.K.

, Priya

W.D.

, Srihari

and Nandakumar

, Speed-bump Detection using Otsu’s Algorithm and Morphological Operation, International Journal on Emerging Technologies11(3) (2020), 989–994.

Badue

, Guidolini

, Carneiro

F.V.

, Azevedo

, Cardoso

V.B.

, Forechi

, Jesus

, Berriel

, Paixao

T.M.

, Mutz

, Veronesea

, Oliveira-Santos

and Ferreira De Souza

, Selfdriving cars: A survey, Expert Systems with Applications165(1) (2021), 113816.

Ballinas-Hernández

A.L.

, Olmos-Pineda

and Olvera-López

J.A.

, Speed Bump Detection on Roads using Artificial Vision, Research in Computing Science148(9) (2019), 71–82.

Buehler

, Iagnemma

and Singh

, The DARPA urban challenge: autonomous vehicles in city traffic, Springer56 (2009).

Celaya-Padilla

J.M.

, Galván-Tejada

C.E.

, López-Monteagudo

F.E.

, Alonso-González

, Moreno-Báez

, Martínez-Torteya

, Galván-Tejada

J.I.

, Arceo-Olague

J.G.

, Luna-García

and Gamboa-Rosales

, Speed bump detection using accelerometric features: A genetic algorithm approach, Sensors18(2) (2018), 443.

10.

Choi

, Lee

, Kim

, Soprani

, Cerri

, Broggi

and Yi

, Environment-detection-and-mapping algorithm for autonomous driving in rural or off-road environment, IEEE Transactions on Intelligent Transportation Systems13(2) (2012), 974–982.

11.

Corsini

, Cignoni

and Scopigno

, Efficient and flexible sampling with blue noise properties of triangular meshes, IEEE transactions on visualization and computer graphics18(6) (2012), 914–924.

12.

Cyganek

and Siebert

J.P.

, An introduction to 3Dcomputer vision techniques and algorithms, John Wiley & Sons (2011).

13.

Danti

, Kulkarni

and Hiremath

, A technique for bump detection in Indian road images using color segmentation and knowledge base object detection, International Journal of Scientific & Engineering Research4(8) (2013), 2229–5518.

14.

Darwiche

and El-Hajj-Chehade

, Speed bump detection for autonomous vehicles using signal-processing techniques, BAU Journal-Science and Technology1(1) (2019), 5.

15.

De la Escalera

, Visión por computador: fundamentos y mátodos, Pearson (2001).

16.

Devapriya

, Babu

C.N.K.

and Srihari

, Advance driver assistance system (ADAS)-speed bump detection, 2015 IEEE international conference on computational intelligence and computing research (ICCIC) (2015), 1–6.

17.

, Qiu

, Gao

, Hu

and Liu

, Abnormal road surface recognition based on smartphone acceleration sensor, Sensors20(2) (2020), 451.

18.

Fernández

, Gavilán

, Llorca

D.F.

, Parra

, Quintero

, Lorente

A.G.

, Vlacic

and Sotelo

, Free space and speed humps detection using lidar and vision for urban autonomous navigation, 2012 IEEE Intelligent Vehicles Symposium (2012), 698–703.

19.

Gidado

U.M.

, Chiroma

, Aljojo

, Abubakar

, Popoola

S.I.

and Al-Garadi

M.A.

, A Survey on Deep Learning for Steering Angle Prediction in Autonomous Vehicles, IEEE Access8 (2020), 163797–163817.

20.

Guerrero-Rascado

J.L.

, Costa

M.J.

, Bortoli

, Silva

A.M.

, Lyamani

and Alados-Arboledas

, Infrared lidar overlap function: an experimental determination, Optics Express18(19) (2010), 20350–20369.

21.

Hirschmuller

, Stereo processing by semiglobal matching and mutual information, IEEE Transactions on pattern analysis and machine intelligence30(2) (2007), 328–341.

22.

Irhebhude

M.E.

, Adeyemi

O.A.

and Kolawole

, Speed Breakers, Road Marking Detection and Recognition Using Image Processing Techniques, European Journal of Applied Science7(5) (2019), 30–42.

23.

Kuutti

, Bowden

, Jin

, Barber

and Fallah

, A survey of deep learning applications to autonomous vehicle control, IEEE Trans. Intell. Transp. Syst.22(2) (2020), 712–733.

24.

Lanjewar

, Khedkar

, Sagar

, Pawar Rasika

and Gosavi

, Survey of road bump and intensity detection algorithms using smartphone sensors, International Journal of Computer Science and Information Technologies6(6) (2015), 5133–5136.

25.

Lee

J.H.

, Kim

H.J.

, Cho

B.J.

, Choi

J.H.

and Kim

Y.J.

, Road Bump Detection Using LiDAR sensor for Semi-Active Control of Front Axle Suspension in an Agricultural Tractor, IFAC-Papers On Line51(17) (2018), 124–129.

26.

Lohani

, Ghosh

and Dashora

, A Review of Standards for Airborne LiDARData Acquisition, Processing, QA/QC, and Delivery. Geospatial Infrastructure, Applications and Technologies: India Case Studies (2018), 305–312.

27.

Long

, Shelhamer

and Darrell

, Fully convolutional networks for semantic segmentation, Proceedings of the IEEE conference on computer vision and pattern recognition (2015), 3431–3440.

28.

Marques

, Alves

, Oliveira

H.C.

, MendonÇa

and Souza

, An evaluation of machine learning methods for speed-bump detection on a GoPro dataset, Anais da Academia Brasileira de Ciências93 (2021).

29.

Oussama

and Mohamed

, A literature reviewof steering angle prediction algorithms for self-driving cars, International Conference on Advanced Intelligent Systems for Sustainable Development (2019), 30–38.

30.

Salau

H.B.

, Onumany

A.J.

, Aibinu

, Onwuka

E.N.

, Dukiya

J.J.

and Ohize

, A survey of accelerometer-based techniques for road anomalies detection and characterization, International Journal of Engineering Science and Application3(1) (2019), 8–20.

31.

Sattar

, Li

and Chapman

, Road surface monitoring using smartphone sensors: A review, Sensors18(11) (2018), 3845.

32.

Varma

, Adarsh

, Ramachandran

and Nair

B.B.

, Real time detection of speed hump/bump and distance estimation with deep learning using GPU and ZED stereo camera, Procedia computer science143 (2018), 988–997.

33.

Varma

, Speed hump/bump dataset, Mendeley Data, v1 (2018).

34.

Vimalkumar

, Vinodhini

R.E.

, Archanaa

, and others, An early detection-warning system to identify speed breakers and bumpy roads using sensors in smartphones, International Journal of Electrical and Computer Engineering7(3) (2017), 1377.

35.

Wang

, Gu

, Yu

, Tan

and Zhou

, A framework for 3d model reconstruction in reverse engineering, Computers & Industrial Engineering63(4) (2012), 1189–1200.

36.

Wöhler

, 3Dcomputer vision: efficient methods and applications, Springer Science & Business Media (2012).

37.

Yun

H.S.

, Kim

T.H.

and Park

T.H.

, Speed-Bump Detection for Autonomous Vehicles by Lidar and Camera, Journal of Electrical Engineering & Technology14(5) (2019), 2155–2162.

Marked and unmarked speed bump detection for autonomous vehicles using stereo vision

Abstract

Keywords

1 Introduction

2 Related work

3.2 Road segmentation

3.10 MUSBD Algorithm

4.1 Road segmentation

Table 2 Metrics to evaluate the model performance Model Total instances Metrics Value SVM 500 images with Precision 84% and without SB Recall 98% Accuracy 89%

References

Table 2
Metrics to evaluate the model performance

Model Total instances Metrics Value

SVM 500 images with Precision 84%

and without SB Recall 98%

Accuracy 89%