Automatic road crack detection and classification using multi-tasking faster RCNN

Abstract

Automatic road crack detection is a prominent challenging task, in view of that, a novel approach is proposed using multi-tasking Faster-RCNN to detect and classify road cracks. In this present study, we have collected the road images (a dataset of 19300 images) from the Outer Ring Road of Chennai, Tamil Nadu, India. The collected road images were pre-processed using various conventional image processing techniques to identify the ground-truth label of the bounding boxes for the cracks. We present a novel multi-tasking Faster-RCNN based approach using the Global Average Pooling(GAP) and Region of Interest (RoI) Align techniques to detect the road cracks. The RoI Align is used to avoid quantizing the stride. So that the information loss can be minimized and the bi-linear interpolation can be used to map the proposal to the input image. The resulting features from RoI Align are given as input to the GAP layer which drastically reduces the multi-dimension features into a single feature map. The output of the GAP layer is given to the fully connected layer for classification (softmax) and also to a regression model for predicting the crack location using a bounding box. F1-measure, precision, and recall were used to evaluate the results of classification and detection. The proposed model achieves the accuracy-97.97%, precision-99.12%, and recall-97.25% for classification using the MIT-CHN-ORR dataset. The experimental results show, that the proposed approach outperforms the other state-of-the-art methods.

Keywords

Multi-tasking faster-RCNN RoI align road crack detection road crack classification

1 Introduction

Road pavement cracks and damages are the most common issues that are found on road surfaces. As the majority of the goods are shipped on road, it is a general belief that well-paved road structures contribute more to the economic growth of a country. The measures taken for maintaining the quality of road has always been a challenge to the public authorities like the road engineers. The maintenance, inspection, and reporting of road conditions are their most important duties. However, the detection, identification, and classification of damaged or cracked roads specifically rely on human experts [22].

The members of the road maintenance department, provide the real status of the road like the severity of the road damage and the locations of the damaged road. It was claimed that the manual assessment made by the road surveyors was inaccurate most of the time. Whenever a road engineer inspects the road for maintenance, the road has to be blocked for a particular period which will affect the normal usage and bring inconveniences to the public [34]. In this work, we propose a novel approach for automatic crack detection using multi-tasking Faster-RCNN. So far, the statistical data were maintained manually by the users. We claim that the shortcomings of manual records can be overcome by the proposed approach.

In the present study, the different types of cracks and damages were categorized based on their shapes. Various image processing techniques and machine learning concepts were proposed for predicting the cracks and damages on the road surfaces. The proposed approach for crack and damage –detection and classification can roughly be divided into two major parts: the first part uses traditional image processing techniques with human intervention to derive ground truth labels for classification and detection. The second part includes the generation of automatic crack and damage –detection and classification for road surface using Convolution Neural Network.

In this work, we have collected road crack images from the outer ring road Chennai. During the collection of road images with cracks, we have also captured the longitude, latitude, date, and time of the specific location of the cracks. The statistical data is also stored and updated automatically for future reference.

There are many approaches proposed for road crack detection which can be classified into two types: 1. The traditional image processing approaches and 2. The CNN-based deep learning approaches. In traditional image processing, various approaches are used for feature extraction. These features are then used as input to a machine learning classifier. The CNN-based approach develops an end-to-end crack detection model. There are many CNN-based approaches proposed for crack detection or segmentation.

The main contributions of this paper is as follows,

We propose a novel multi-tasking Faster-RCNN for detection and classification of road crack images.

The proposed approach uses an RoI align in Faster-RCNN which does not quantize the stride. This reduces the error due to the quantization of strides in the traditional RoI mapping. Thus the proposed model will accurately map the region of the proposal to the original image.

The multi-tasking Faster-RCNN is designed using the GAP layer, which maps the output of the convolution layer to the final fully connected layer for classification and road crack detection.

The rest part of the paper is designed as follows. In Section 2, the various related works are discussed, and Section 3 explains the pre-processing techniques for processing the collected images and identification of ground-truth label of the cracks. Section 4 introduces the proposed approach for road surface crack detection, and classification. In Section 5, the dataset collection, experiments, and analysis of the results for the proposed method are discussed. In Section 6, we discuss the conclusion of the paper.

2 Related works

Road crack detection is an important application of computer vision. There have been many approaches proposed for road crack detection, segmentation, and classification. The road crack detection algorithms can be broadly classified into two types: 1. Image processing approaches 2. Deep learning approaches. The image processing techniques are used in the early period of developing road crack detection algorithms. One such approach proposed in [33] uses traditional image processing algorithm for accurate segmentation and classification of road cracks. From the input images, edges are detected using the Ostu algorithm and threshold division. But the basic method of Otsu’s thresholding detects only the edges in the images. Thus, it is not suitable for noisy images and it is unable to detect low-intensity cracks in the images.

In another image processing based algorithm [24] for road crack detection, the input images are first pre-processed then given as input to the crack detection algorithm. It uses a variance discriminant analysis technique that extracts the crack regions based on pixel variance. The threshold method applied here is similar to Otsu’s Binarization method, but the crack regions are extracted using the variance in the pixels. Based on variance calculation, the threshold value is set to classify a pixel as a crack pixel or non-crack pixel.

There are other approaches that use image processing techniques for pre-processing the images and then path selection algorithm was used to detect cracks [1 , 41]. The approach in [16] uses image processing technique for pavement crack detection. The input images are pre–processed using contrast stretching, morphological operations, and noise reduction. Then, it was processed by applying threshold conversion to detect the cracks. The detected broken cracks are connected and classified using a crack edge searching algorithm. CrackTree [41] method was developed to detect the crack automatically by removing the shadows of pavement images. To identify the crack connectivity, a crack probability map is constructed. These maps are connected using a minimum spanning tree to segment the complete crack regions.

A minimum path selection based approach was proposed in [1] for detecting cracks. The image is divided into smaller regions and one pixel is selected from each region called endpoints. From these endpoints, Dijkstra’s algorithm is used to select paths from one endpoint to another using intensity of the pixels. Cracks were detected in [13] using adaptive line detector in association with Hidden Marko Random Field and Expectation-Maximization algorithms (HMRF –EM). The input images are pre–processed using traditional image processing techniques like the bandpass filter which was used to reduce the high-frequency noise. The HMRF–EM algorithm calculates the spatial correlation and spatial constraints between the pixels. Then, the conditional connection algorithm was used to connect the crack segment.

The initial deep learning approaches performs classification of the images as crack or non-crack [5] and [39]. From the classified crack images, the cracks are detected using image processing methods. In [5], a deep learning approach is proposed for classification of crack images. After classification, segmentation is carried out using the image processing technique. The image is smoothed using a bilateral filter, the adaptive thresholding is used for the segmentation of cracks.

In [17], a spatially tuned robust multi-feature classifier is proposed for crack classification. It first extracts the curve regions from given images and features are extracted from these curve regions. These features are then classified using classifiers like Random Forest, SVM, etc. From the classified images, the cracks regions are identified by constructing a density map from the mosaic obtained from the images.

To identify the crack, in [30] they have introduced the multi-scale dilated attention based convolution module. The ResNet model is used to extract features from the images, then an attention mechanism was used for extracting the high-level features using upsampling. These features are then used for the classification of crack images.

The recent deep learning approaches uses end to end crack detection algorithm, which calculates the bounding boxes of the cracks or generates a binary segmented images [3 , 40]. The SDDNet [3] model was used to detect the damaged region of the concrete surface using standard convolutions and densely separable (DenSep) module along with modified spatial pooling for segmentation of cracks. The model is trained using modified intersection over union loss, thus the model performs binary segmentation of the crack regions. The approach [4] performs pixel-level crack detection using U-net architecture. The U-Net is trained along with features extracted from the pre-trained ResNet-34 and ResNet-50 model. The approach in [7] also uses U-Net for pixel-level crack identification, here auxiliary interaction loss is proposed to alleviate the problem of fractured crack regions.

DeepCrack [15] approach was developed to find the pixel-wise segmentation of crack regions. It has a deep convolutional network and Deep-Supervised Nets which learns the multi-scale and multi-level features from the convolutional layers. After detection of crack regions, to refine the final prediction of crack regions, guided filtering, as well as Conditional Random Fields, were used. In [19], the automated road surface distress analysis system was designed by using the YOLO v2 framework. It predicts the bounding box locations for the given input images. Another approach in [12] uses YOLO models for road crack detection.

Though many image processing techniques were proposed for crack identification and detection, they lack in performance when compared to the convolution neural network based approaches. In this work, a novel end-to-end road crack detection and classification method were developed using the multi-tasking Faster-RCNN approach by incorporating global average pooling layer and Region of Interest Align.

3 Ground truth label identification

The ground truth labeling was carried out after pre-processing the road crack images using various traditional image processing techniques like image enhancement, image transformation, image filtering, contrast stretching, and edge detection. After identifying the cracks on the images, the bounding box is created. Figure 1 shows the techniques that are used for the identification of ground truth labels for the crack images. Then we manually annotate the labels of cracks from the pre-processed input images. The various image processing techniques used in the present work to pre-process the images are explained in detail in the following sub-sections.

Fig. 1

Flow of deriving ground truth image from the obtained results of various traditional image processing techniques.

3.1 Image enhancement

It is the basic operation performed on an image to improve the intensity level or to suppress unwanted distortions. Also, it corrects the degraded portions of an image. To pre-process the images, a few image enhancement techniques [25] were employed and analyzed.

There are various image enhancement techniques available in the literature like sharpening, log transformation, histogram equalization, adaptive histogram equalization, dynamic fuzzy histogram equalization, etc. The sharpening technique improves the contrast between bright and dark regions to highlight the features. In [31], sharpening is used for multispectral images in remote sensing. It is found that sharpening has improved the spatial resolution of these multi-spectral images. Another image enhancement technique is log transformation performed by equation 1. $s = c \log (r + 1)$ (1) where, s and r represent the pixel values of the input and output image and c is a constant. Log transformation and the standard least-squares formulation are used to improve low contrast and smooth adjustment for the pixel intensity value of an image [21]. Histogram equalization helps to enhance the contrast for a given image which spreads out the high level intensity pixel values effectively [8], Adaptive histogram equalization [23] is specially designed based on the contextual region size. The term refers to the mapping of resultant pixel by taking the union of the surrounding four sample points associated with contextual regions appeared as Equivalent Contextual Region (ECR). Dynamic fuzzy histogram equalization is an enhancement of adaptive histogram equalization. This technique was used to improve the brightness and contrast to reduce its computational complexities.

Performance measuring metrics like Mean Square Error (MSE), Peak Signal to Noise Ratio (PSNR), Signal to Noise Ratio (SNR), and Structured Similarity Index Metric (SSIM) were checked for various image enhancement techniques. Table 1 compares the results of various image enhancement techniques for the MIT-CHN-ORR dataset. It was inferred that the sharpening achieves better results compared to the other image enhancement techniques.

Table 1

Various image enhancement techniques - results comparison based on MSE, PSNR, SNR, and SSIM

		Sharpening	Log transformation	Dynamic Fuzzy Histogram Equalization	Adaptive Histogram Equalization	Histogram Equalization
MSE	Mean	123.44	27519.3	1711.57	3756.80	5990.55
	Standard Deviation	19.51	709.06	293.50	203.74	290.61
PSNR	Mean	27.26	3.73	15.85	12.38	10.36
	Standard Deviation	0.68	0.10	0.73	0.22	0.19
SNR	Mean	23.65	-44.40	12.64	6.93	3.62
	Standard Deviation	0.69	0.10	0.66	0.21	0.19
SSIM	Mean	0.87	0.005	0.47	0.40	0.38
	Standard Deviation	0.008	0.0005	0.03	0.02	0.01

3.2 Noise removal (Wiener Filter)

A noise removal filter is used to remove unwanted and irrelevant information from an input image. Here, a wiener filter was used for the noise removal technique. It also helps in removing the noise with compression operation. Apart from them, various noise removal filters like Salt and Pepper Noise, Guided Filter, and wiener filter were exploited. Table 2 shows the comparison results for various noise removal techniques. Nevertheless, the wiener filter performed better compared with the other noise removal methods in the present study.

Table 2
Comparison results of various noise removal filters

Method Average MSE

Salt and Pepper Noise 27.24

Guided Filter 221.46

Gaussian Filter 107.36

Wiener Filter 0.27

Method	Average MSE
Salt and Pepper Noise	27.24
Guided Filter	221.46
Gaussian Filter	107.36
Wiener Filter	0.27

3.3 Contrast stretching

The main objective of contrast stretching is to revamp the intensity values of the images. It is used to make use of all possible intensity values in an image. Also, it had a feature to improve the dynamic range of an image. It uses two types of contrast stretching: global contrast stretching and linear contrast stretching. It is also known as the normalization of an image. The approaches in [14, 35], and [6] uses contrast stretching in various applications like remote sensing, Bio-medical image processing, and image enhancement. In this work, we use contrast stretching to pre-process the input images to develop ground truth labels and also to pre-process images given as input to the Faster-RCNN model.

3.4 Edge detection

It is a process to identify the boundary or edges in an image. It played a vital role in various applications like recognition of objects, analysis of moving objects, and recognition of patterns. In [20], detecting the edges was performed in two stages; stage one was used to identify the edge points, location, and orientation of the cubic facet model and stage two was used to decreased the noise and increased the edge structure.

Sobel operator was used in [26] to identify the edges of an input image by applying the derivative approximation method. A new double threshold technique was developed using hysteresis threshold to find out edges of an image [28]. Based on those threshold values, the improved canny edge detector was obtained.

In [2], canny edge detector was employed in scale multiplication method, which has two scales of response detection filters. Edge maps and detection criteria were used to identify the edge based on the localization criterion. MM -sobel operator and region growing were used in [29 , 37], and [9] to isolate the identified edges from the given image.

Edge detection techniques will work based on the threshold value calculated for the neighboring pixel with its respective deviation in [18]. Also, they measure the quality of an edge detection as mentioned in [38]. The following edge detection techniques like Sobel operator, Prewitt operator, Roberts operator, Laplacian of Gaussian, Zero cross edge detector, K-means, Canny edge were implemented and the resulting images are displayed in Fig. 2. From the results, it was concluded that the canny edge detection provides better results compared to other algorithms. Hence it was chosen in the present study to identify the edges in the images to pre-process the images. From the edge detected images, we have manually annotated the ground truth labels for the road crack images in the MIT-CHN-ORR dataset.

Fig. 2

Edge detection comparison results obtained for our proposed dataset.

4 Proposed multi-tasking faster RCNN for crack detection and classification

In this work, we propose a multi-tasking faster R–CNN for detection and classification of the crack images. The input image is passed through the various convolutional layers followed by pooling layers. The output of the pooling layers is passed through the ReLU activation function. We have also used some batch normalization layers. The input images are passed through the convolution and pooling layer of the CNN model to extract the feature map. From these feature maps, 12 anchor points are derived. The anchor points are again passed through various convolutional and pooling layers followed finally by the Global Average Pooling (GAP) layer. The extracted features are passed through softmax for classifying the cracks into various types. The same features were passed through the regressor for predicting the coordinates of such information, called as the object proposal.

The feature maps were compared with the fixed features maps and assigned with RoI Align. Consider an input image of 34 x 34 size, the stride can be calculated as 34/7=4.85, it will not be rounded off the value of stride as 5. Rather, it will process as the stride value obtained as such for generating the 7 x 7 output images. Figure 3 shows the proposed architecture of multi-tasking faster region-based convolution neural network using global average pooling and RoI align.

Fig. 3

Proposed architecture using Multi-tasking Faster R-CNN in association with Global Average Pooling and Region of Interest Align.

4.1 Object proposal region

In this work, the term ’object’ refers to the crack or damaged part of the road image. The object proposal network works based on a feature map. The feature map contains an anchor region, that points to a location in the image which is calculated by Intersection Over Union(IOU). The IOU_a threshold value for the mapping anchor region is finalized as 0.75 after fine-tuning all the parameters. The IOU_a was applied on an image using equation (2). ${IOU}_{a} = \frac{Anchorvalue ⋂ groundtruth}{Anchorvalue ⋃ groundtruth}$ (2) Then, it was passed into the convolution layer with an input size of 800 x 500. The output of the convolution layer is of size 50 x 32 (then 800/16 = 50 and 500/16 =31.25, approximately considered as 32). Later, it is given as input to the ResNet 152 model. The layer dimension or depth of 4 of convolution is 16. The size of the kernel window was fixed as 3 x 3, for all possible kernel window locations per anchor was taken.

A total of 6 anchors were obtained for the single dimension for all 16 filters based on the fourth layer of convolution. If it is 16, then, it will have a combination ratio of 1:1-16:16, 1:2-16:32, and 2:1-32:16, and similar anchors were obtained for a combination ratio of 8, 1:1-8:8, 1:2-8:16, and 2:1-16:8. While ignoring the occurrences of cross-boundary anchors, the value was approximately set as 7500. Then, by applying the Non –Minimal Suppression (NMS), the anchor is reduced to an approximate value of 1200. The output is then given as input to the global average pooling layer which calculates the average of every single feature map. It returns the single average value of all the returned features which is then passed to the fully connected layer. The softmax function is used to determine the type of the crack and the regressor is used for identifying the coordinates of axis, height, and width for generating a single bounding box.

4.2 Object proposal loss function

The object proposal consists of two loss functions: The loss function is used at the classifier to identify the object class. Another loss function is used at the regressor to obtain the coordinates x and y, width, and height for constructing the bounding box.

The loss function L (C_i) of the classifier is given by equation (3). $L (C_{i}) = \frac{1}{N_{cl}} \sum_{i} L_{cl} (C_{i}, C_{i}^{*})$ (3) where,

C_i - represents the value of predicted probability.

$C_{i}^{*}$ - the value will be 1 for positive anchor otherwise the value will be 0.

N_cl - number of anchor available in minibatch.

The loss function for regressor L (R_i) were calculated in equations (4). $L (R_{i}) = C \frac{1}{N_{reg}} \sum_{i \in (x, y, w, h)} C_{i}^{*} L_{reg}^{*} (R_{i}, R_{i}^{*})$ (4) where,

C –constant value.

N_reg –total number of anchors (approximately 1200).

$L_{Reg}^{*} (R_{i}, R_{i}^{*})$ is loss of the i^th region given by equation (5). $L_{reg}^{*} (R_{i}, R_{i}^{*}) = {smooth}_{L 1} (R_{i} - R_{i}^{*})$ (5) where,

the smooth function is given by equation (6). ${smooth}_{L 1} (j) = {\begin{matrix} 0.5 j^{2}, if | j | < 1 \\ | j | - 0.5, otherwise \end{matrix}$ (6) the region values for x, y, w and h are given by equation (7). $\begin{matrix} R_{x} & = (x - x_{a}) / w_{a}, R_{y} = (y - y_{a}) / h_{a} \\ R_{x}^{*} & = (x^{*} - x_{a}) / w_{a}, R_{y}^{*} = (y^{*} - y) / h_{a} \\ R_{w} & = \log (w / w_{a}), R_{h} = \log (h / h_{a}) \\ R_{w}^{*} & = \log (w^{*} / w_{a}), R_{h} = \log (h^{*} / h_{a}) \end{matrix}$ (7) where, x and y represent the predicted bounding box coordinates, x^* and y^* represent the ground truth bounding box coordinates. Here w and h represent the width and height of the predicted bounding box. w^* and h^* represent the width and height of the ground truth bounding box. R_x and R_y indicate the regression value of x and y of the predicted bounding box coordinates. $R_{x}^{*}$ and $R_{y}^{*}$ indicates the regression value of x and y of the ground truth bounding box coordinates. R_w and R_h indicates the regression value for the predicted bounding box’s width and height. $R_{w}^{*}$ and $R_{h}^{*}$ indicate the regression value for the ground truth bounding box’s width and height.

The overall loss function was calculated using the equation (8). $Lossfunction = \frac{1}{2} {L (C_{i}) + L (R_{i})}$ (8)

For predicting the class of an object, the softmax function was used for classification. Figure 4 shows a sample calculation carried out for normalizing the value. It also shows the calculation of the probability of whether the pixel is crack or non-crack from the output value.

Fig. 4

Sample calculation for normalization.

4.3 RoI align

The Region of Interest is used to select the feature map from the given input image. The input image had the size 102 x 102 and then, it was reduced to 8 x 8 size as the RoI. The stride was calculated by 12.75/3= 4.25. The value of stride will not be rounded off and the obtained value will be considered as such. Then, all the features were mapped as 3 x 3 for the region of interest and based on stride, it was divided into 4.25 for each bin. This creates a region of interest on the bottom right, top right, bottom left and top left. And each of those subcells was pooled using bilinear interpolation which continued with 4 values per cell. Finally, the cell values were computed as on average or on maximum over the 4 sub-values. Figure 5 shows the working principle of RoI Align. The RoI align preserved the spatial-oriented features with no data loss.

Fig. 5

An illustration of the Region of Interest align.

5 Experiments and results

In this section, we discuss the various results obtained from the experiments carried out in this work. The results obtained for pre-processing of images were evaluated using various metrics like PSNR, MSE, and SSIM. The results obtained for the crack detection and the classification were evaluated using various metrics like precision, recall, accuracy, and F1-measure.

5.1 Dataset collection and generation

In the present study, the data set was collected from Outer Ring Road, Chennai location, denoted as MIT-CHN–ORR data set. While capturing the images, the information was saved automatically as shown in Table 3. It saves information like the file name, latitude, longitude, elevation, altitude, accuracy, date, and time. The data set consists of 19,300 images with a combination of crack and non-crack images. The dataset consists of 9,875 crack and 9,425 non-crack images. The crack images are further classified into 5 types: Linear cracks - 3,847 images, non-linear cracks-2,894 images, Alligator cracks-2,108 images, and Damaged rods with pitfalls-1,026 images. The resolution of the images was 256x256 pixels. To compare the performance of our model, we have also used RDD dataset [17], which consists of 9,053 road damage images with 15,435 instances of road surface images.

Table 3
Dataset collection and information maintenance for statistical data

File -name Latitude Longitude Elevation Altitude Accuracy Time

20190825-102154.jpg 12.915764 80.101353 29.17 m –62.85 m 9.6 m 25-08-2019 10:21

20190825-102346.jpg 12.915694 80.101386 30.64 m –61.38 m 3.2 m 25-08-2019 10:23

20190825-102401.jpg 12.915701 80.101392 29.9 m –62.12 m 3.2 m 25-08-2019 10:24

20190825-102412.jpg 12.915705 80.101395 30.52 m –61.5 m 3.2 m 25-08-2019 10:24

20190825-102425.jpg 12.915714 80.101395 30.37 m –61.65 m 3.2 m 25-08-2019 10:24

20190825-102442.jpg 12.915722 80.101398 30.47 m –61.55 m 9.6 m 25-08-2019 10:24

20190825-102455.jpg 12.915723 80.101398 30.48 m –61.54 m 9.6 m 25-08-2019 10:24

20190825-102536.jpg 12.915738 80.101399 30.16 m –61.86 m 9.6 m 25-08-2019 10:25

20190825-102554.jpg 12.915761 80.101402 27.76 m –64.26 m 3.2 m 25-08-2019 10:25

20190825-102619.jpg 12.915755 80.10137 26.86 m –65.16 m 3.2 m 25-08-2019 10:26

File -name	Latitude	Longitude	Elevation	Altitude	Accuracy	Time
20190825-102154.jpg	12.915764	80.101353	29.17 m	–62.85 m	9.6 m	25-08-2019 10:21
20190825-102346.jpg	12.915694	80.101386	30.64 m	–61.38 m	3.2 m	25-08-2019 10:23
20190825-102401.jpg	12.915701	80.101392	29.9 m	–62.12 m	3.2 m	25-08-2019 10:24
20190825-102412.jpg	12.915705	80.101395	30.52 m	–61.5 m	3.2 m	25-08-2019 10:24
20190825-102425.jpg	12.915714	80.101395	30.37 m	–61.65 m	3.2 m	25-08-2019 10:24
20190825-102442.jpg	12.915722	80.101398	30.47 m	–61.55 m	9.6 m	25-08-2019 10:24
20190825-102455.jpg	12.915723	80.101398	30.48 m	–61.54 m	9.6 m	25-08-2019 10:24
20190825-102536.jpg	12.915738	80.101399	30.16 m	–61.86 m	9.6 m	25-08-2019 10:25
20190825-102554.jpg	12.915761	80.101402	27.76 m	–64.26 m	3.2 m	25-08-2019 10:25
20190825-102619.jpg	12.915755	80.10137	26.86 m	–65.16 m	3.2 m	25-08-2019 10:26

5.2 Evaluation metrics

5.2.1 Metrics used for ground label identification

PSNR

PSNR is calculated by comparing the original image with the reconstructed image. It is the ratio of maximum possible values of the signal to the power of distortion noise. If the obtained value is found to be high, then it will be considered as good quality.

MSE

MSE is used to calculate the error value by taking the difference between the original and the reconstructed image. If the obtained value is low, then the image quality will be considered as good. It is also referred to as Mean Square Deviation (MSD).

SSIM

SSIM is calculated by comparing the original and the reconstructed image. It is measured based on luminance, contrast, and structure. If the image had more similarities, then, it will be indicated as 1 otherwise it will be indicated as 0.

5.2.2 Metrics used for crack detection and classification

The proposed work calculates the F1 measure of the results based on the confusion matrix. The matrix consisted of four fields namely,

True Positive (TP) –crack images predicted accurately.

True Negative (TN) –the number of non-crack images predicted accurately.

False Positive (FP) –the number of crack images predicted as crack but they don’t have a crack. False Negative (FN) –the number of crack images predicted as non-crack but they have a crack.

Precision, Recall and F1-Measure are calculated as follows:

Precision

It is the ratio of the number of correct positive predictions to the total number of positive predictions, calculated by the equation (9). $precision = \frac{TruePositive}{TruePositive + FalsePositive}$ (9)

Recall

It is the ratio of the number of correct positive predictions to the total number of positive samples in the dataset calculated using equation (10). $Recall = \frac{TruePositive}{TruePositive + FalseNegative}$ (10)

F1–Measure

It is the weighted average of the precision and recall. It is computed using based on the equation (11). $F 1 Measure = 2 . \frac{Precision . Recall}{Precision + Recall}$ (11)

5.2.3 Implementation details

The multi-tasking Faster RCNN model for crack detection was implemented in Keras environment with TensorFlow backend. We have used a dataset split of 60% for training, 10% for validation, and 30% for testing. From the MIT-CHN-ORR dataset 11,580 images were used for training, 1930 images are used for validation, and 5790 images are used for testing. The input images consist of both crack and non-crack images. The stride for the CNN is set as 7. Batch Normalization (BN) is used for regularizing the Faster-RCNN model. We use ReLU as activation functions for the model. The model was trained for 2500 epochs. The Batch size is set to 64, and the learning rate is set to 0.002. We have used an ADAM optimizer with beta1=0.7 and beta2=0.99 for training the model.

5.3 Analysis of results

The proposed multi-tasking faster R-CNN is evaluated using various evaluation metrics. Table 4 shows the comparison results obtained for detection using the MIT-CHN-ORR dataset. Table 5 shows the comparison of results obtained for detection and classification using the existing dataset RDDC with the proposed approach.

Table 4
Comparison of results obtained for road crack detection using our MIT-CHN-ORR dataset

Method Precision Recall F-1 measure

Otsu’s thresholding [33] 0.54 0.51 0.53

YOLO v2 framework [19] 0.72 0.80 0.75

Supervised Deep CNN [39] 0.79 0.81 0.80

Faster R-CNN [17] 0.85 0.89 0.87

CNN + GAP 0.94 0.91 0.92

Mask-RCNN + RoI Pooling 0.97 0.96 0.97

Faster-RCNN + GAP 0.97 0.98 0.97

Multi-tasking Faster-RCNN using GAP and ROI Align} 0.99 0.97 0.98

Method	Precision	Recall	F-1 measure
Otsu’s thresholding [33]	0.54	0.51	0.53
YOLO v2 framework [19]	0.72	0.80	0.75
Supervised Deep CNN [39]	0.79	0.81	0.80
Faster R-CNN [17]	0.85	0.89	0.87
CNN + GAP	0.94	0.91	0.92
Mask-RCNN + RoI Pooling	0.97	0.96	0.97
Faster-RCNN + GAP	0.97	0.98	0.97
Multi-tasking Faster-RCNN using GAP and ROI Align}	0.99	0.97	0.98

Table 5

Comparison of results obtained for road crack detection using the existing RDDC dataset ([17])

Method	Precision	Recall	F-1 measure
Otsu’s thresholding [33]	0.45	0.56	0.50
YOLO v2 framework [19]	0.67	0.70	0.68
Supervised Deep CNN [39]	0.72	0.74	0.73
CNN + GAP	0.77	0.76	0.76
Mask-RCNN + RoI Pooling	0.75	0.77	0.76
Faster-RCNN + GAP	0.78	0.76	0.77
Multi-tasking Faster-RCNN using GAP and ROI Align	0.87	0.80	0.83

The following observations are made from the results shown in Tables 4 and 5.

The proposed approach provides better Precision, Recall, and F1-Measure score than the traditional image processing based algorithm like Otsu’s thresholding [33]. This is due to the end-to-end training of the model to detect the cracks.

The proposed approach performs better than the other state-of-the-art approaches like YOLO v2 framework [19], Supervised Deep CNN [39], and Faster R-CNN [17]. This is due to the pre-processing of the images using traditional image processing techniques. We have also used an RoI Align and the GAP layer in the proposed multi-tasking Faster-RCNN model for road crack detection.

The deep learning based approaches like [17 , 39], provides better performance than the traditional image processing approaches like Otsu’s thresholding [33].

For both the datasets, the proposed Multi-tasking Faster RCNN provides better performance compared to the basic CNN designed using only the GAP layer for segmentation. Thus the use of RoI Align and the multi-task training of the model helps the model in improving the performance.

The use of GAP and RoI Align increases the performance compared to the model which does not use the GAP layer and RoI Align. This is due to round-off error while calculating the strides in a normal RoI layer, whereas in RoI align does not quantize the stride, hence reduces the error.

The multi-tasking faster RCNN provides better performance compared to the mask RCNN with RoI Pooling. This is due to the use of the GAP layer and RoI Align which results in a lower error in RoI mapping.

Table 6 shows the result comparison of the various datasets available in the literature. The following observations are made from the results in Table 6.

It shows that the proposed approach of multi-tasking faster RCNN on the MIT-CHN-ORR dataset provides better results compared to all the other approaches. This is also due to the large number of images collected in our MIT-CHN-ORR dataset.

It shows that the traditional Random Structured Forests [27] provide better results for smaller dataset size like 118 and 38. But the Deep neural network based approaches [13 , 39] uses larger number of images for training the model. Thus the larger number of images in the dataset collected by us provides better performance.

Though the Faster R-CNN approach in [13] uses a larger dataset of 9,053 images it provides a lower performance score. The proposed approach uses 11,580 images for training the model but provides higher performance, this is due to the use of RoI align and GAP layer in the proposed approach. We have also proposed a multi-task Faster-RCNN which increases the performance of both classification and segmentation.

Table 6

Comparison of results obtained for road crack detection for the existing methods with the proposed method

Method	Dataset Name	Precision	Recall	F-1 measure
Random Structured Forests [27]	CFD Dataset –118 images	0.82	0.89	0.85
Random Structured Forests [27]	AIGLERN Dataset –38 images	0.90	0.86	0.88
Steerable matched filtering and an active contour model [11]	tested using 65 pavement images	0.92	0.85	0.88
Voting using tensor [10]	200 asphalt pavement images	0.88	0.93	0.90
HMRF-EM Algorithm [13]	CFD Dataset –118 images	0.90	0.94	0.93
Supervised Deep CNN [39]	Dataset collected from Temple University campus (more than 500 images)	0.86	0.92	0.89
DeepCrack-Aug(DeepCrack-augmented data networks) [15]	537 images	0.86	0.86	0.86
Faster R-CNN [13]	New dataset of 9,053 images	0.65	0.60	0.62
Multi-tasking Faster - RCNN	proposed dataset (MIT-CHN-ORR)	0.99	0.97	0.98

Table 7 shows the comparison results of classification on MIT-CHN-ORR dataset. Here ResNet+SVM is an SVM model trained using ResNet-152 features extracted from the road crack images. It shows that the multi-tasking faster RCNN provides higher when compared to basic ResNet+SVM and CNN models. Figure 6 shows the confusion matrix obtained for the classification performed using multi-tasking faster RCNN. We can see that most of the images are correctly classified. We can see that the wrong classification is more for non-crack images classified into other types of cracks. This is due to the availability of a large number of non-crack images compared to the other type of crack images.

Table 7

Comparison of classification results on MIT-CHN-ORR dataset

Method	Precision	Recall	Accuracy
ResNet+SVM	0.79	0.88	0.78
CNN	0.92	0.93	0.92
Multi-tasking Faster - RCNN	0.99	0.97	0.97

Fig. 6

Confusion matrix obtained using Multi-tasking Faster RCNN for classification.

Figure 7 shows the Receiving Operating Characteristics(ROC) curve for classification of the road crack images. It can be seen that the proposed multi-tasking Faster RCNN approach provides better performance and larger ROC-AUC value compared to the basic ResNet+SVM and CNN model. Figure 8 shows the sample road images with cracks, without cracks, ground truth image with bounding boxes on the cracks, and the classified images for our proposed MIT-CHN-ORR dataset. The figure shows that the proposed multi-tasking faster RCNN precisely calculates the bounding box of the cracks from the crack labels, we can see that the proposed model correctly classifies the types of crack.

Fig. 7

ROC-AUC curve for classification of road crack images.

Fig. 8

The first row shows the few sample non-crack images from the MIT-CHN-ORR dataset, the second row shows the few sample crack images from the MIT-CHN-ORR dataset, the third row shows a few sample ground truth for crack images with bounding boxes from the MIT-CHN-ORR dataset, fourth and fifth row shows few of the sample output images obtained using Multi-tasking Faster RCNN (linear crack, non-linear crack, alligator crack, road damage, and non-crack) from MIT-CHN-ORR dataset.

6 Conclusion

Maintaining roadways is a very tedious task as there are larger stretches of roads are available in a country. There are many approaches developed for road crack identification and detection. The present study puts forth a multi-tasking Faster-RCNN for crack detection and classification. We have also proposed global average pooling and region of interest align in the multi-tasking Faster RCNN for multi-tasking the classification and detection. The RoI Align reduces the error due to the quantization of the strides from the region proposal to the original image. The proposed models are evaluated using the MIT-CHN-ORR dataset and RDDC dataset. The performance results of the proposed methods were compared with the existing methods using various evaluation metrics. The evaluation results show that the proposed multi-tasking Faster RCNN outperforms the other state-of-the-art approaches for road crack detection and classification.

References

Amhaz

, Chambon

, Idier

and Baltazart

, Automatic crackdetection on 2d pavement images: An algorithm based on minimal pathselection, IEEE Transactions on Intelligent TransportationSystems 17 (2015), 2718–2729.

Bao

, Zhang

and Wu

, Canny edge detection enhancement byscale multiplication, IEEE Trans Pattern Anal Mach Intell 27 (2005), 1485–1490.

Choi

and Cha

, Sddnet: Real-time crack segmentation, IEEETransactions on Industrial Electronics 67 (2020), 8016–8025.

Deru

, Zhongdong

, Xiaoyang

and Dongchang

, Pixel-LevelRecognition of Pavement Distresses Based on U-Net, Advances inMaterials Science and Engineering 2021 (2021), 1–11.

Fan

, Bocus

, Zhu

, Jiao

, Wang

, Ma

, Cheng

and Liu

, Road crack detection using deep convolutional neural network and adaptive thresholding, (2019).

Kaur

and Choudhary

, Comparison of contrast stretching methodsof image enhancement techniques for acute leukemia imagess, International Journal of Engineering Research and Technologys 2 (2012), 1715–1719.

Fang

, Qu

and Yuan

, Distribution equalization learningmechanism for road crack detection, Neurocomputing 424(2021), 193–204.

Yang

, Zhong

and Miao

, On the image enhancement histogram processing, in: 2016 3rd International Conference on Informative and Cybernetics for Computational Social Systems (ICCSS) (2016), 252–255.

Koplowitz

and Greco

, On the edge location error for localmaximum and zero-crossing edge detectors, IEEE Trans PatternAnal Mach Intell 16 (1994), 1207–1212.

10.

, Wang

, Zhang

, Fei

and Sollazzo

, Automaticsegmentation and enhancement of pavement cracks based on 3d pavementimages, Journal of Advanced Transportation 2019 (2019), 1–9.

11.

, Cao

and Cai

, Automatic pavement-crack detection andsegmentation based on steerable matched filtering and an activecontour model, Journal of Computing in Civil Engineering 31 (2017).

12.

, Gu

, Xu

, Zhang

, Liu

and Dong

, Detectionof concealed cracks from ground penetrating radar images based ondeep learning algorithm, Construction and Building Materials 273 (2021).

13.

Lin

, Zhou

, Yan

and Xu

, Automatic pavement crack detection using hmrf-em algorithm, in: 2019 International Conference on Computer, Information and Telecommunication Systems (CITS), (2019), 1–5.

14.

Liu

, Zhou

, Chen

and Kang

, An efficient contrastenhancement method for remote sensing images, IEEE Geoscienceand Remote Sensing Letters 14 (2017), 1715–1719.

15.

Liu

, Yao

, Lu

, Xie

and Li

, Deepcrack: A deephierarchical feature learning architecture for crack segmentation, Neurocomputing 338 (2019), 139–153.

16.

Lyasheva

, Tregubov

and Shleymovich

, Detection and recognition of pavement cracks based on computer vision technology, in: 2019 International Conference on Industrial Engineering, Applications and Manufacturing (ICIEAM), 1–5, (2019).

17.

Maeda

, Sekimoto

, Seto

, Kashiyama

and Omata

, Road damage detection using deep neural networks with images captured through a smartphone, Arxiv, (2018).

18.

Mallick

, Roy

, Chaudhuri

S.S.

and Roy

, Optimization of laplace of gaussian (log) filter for enhanced edge detection: A new approach, in: Proceedings of The 2014 International Conference on Control, Instrumentation, Energy and Communication (CIEC) (2014), 658–661.

19.

Mandal

, Uong

and Adu-Gyamfi

, Automated road crack detection using deep convolutional neural networks, (2018), 5212–5215.

20.

Matalas

, Benjamin

and Kitney

, An edge detection techniqueusing the facet model and parameterized relaxation labeling, IEEE Transactions on Pattern Analysis and MachineIntelligence 19 (1997), 328–341.

21.

Meaney

, Grzegorczyk

, Jeon

and Paulsen

, Log transformation with gauss-newton microwave image reconstruction reduces incidence of local minima convergence, (2009), 1–4.

22.

Oliveira

and Correia

P.L.

, Automatic road crack detection andcharacterization, IEEE Transactions on IntelligentTransportation Systems 14 (2013), 155–168.

23.

Pizer

S.M.

, Amburn

E.P.

, Austin

J.D.

, Cromartie

, Geselowitz

, Greer

, Romeny

B.T.H.

and Zimmerman

J.B.

, Adaptive histogramequalization and its variations, Comput Vision Graph ImageProcess 39 (1987), 355–368.

24.

Premachandra

, Waruna

, Premachandra

and Parape

C.D.

, Image based automatic road surface crack detection for achieving smooth driving on deformed roads, in: 2013 IEEE International Conference on Systems, Man, and Cybernetics, (2013), 4018–4023.

25.

Radke

, Andra

, Al-Kofahi

and Roysam

, Image changedetection algorithms: A systematic survey, IEEE Transactions onImage Processing: A Publication of the IEEE Signal ProcessingSociety 14 (2005), 294–307.

26.

Gupta1

and Mazumdar

S.G.

, Sobel edge detection algorithm, International Journal of Computer Science and ManagementResearch 2 (2013), 1578–1583.

27.

Shi

, Cui

, Qi

, Meng

and Chen

, Automatic road crackdetection using random structured forests, IEEE Transactions onIntelligent Transportation Systems 17 (2016), 3434–3445.

28.

Sidhu

R.K.

, Improved canny edge detector in various color spaces, in: Proceedings of 3rd International Conference on Reliability, Infocom Technologies and Optimization, (2014), 1–6.

29.

Singh

, Kaur

and Gupta

, Robust edge detector using back propagation neural network with multi-thresholding, in: 2014 IEEE International Conference on Computational Intelligence and Computing Research, (2014), 1–6.

30.

Song

, Jia

and Zhu

, Automatic pavement crackdetection and classification using multiscale feature attentionnetwork, IEEE Access 7 (2019), 171001–171012.

31.

Sun

, Lv

, Tan

and Liu

, An image sharpening strategy based on multiframe super resolution for multispectral data, in: 2016 8th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), (2016), 1–5.

32.

Wang

, Wu

, Yang

and Wang

, Road damage detection and classification with faster r-cnn, (2018), 5220–5223.

33.

Wei

, Liu

and Bai

, Automatic road crack image preprocessing for detection and identification, in: Proceedings of the 2009 Second International Conference on Intelligent Networks and Intelligent Systems, IEEE Computer Society, (2009), 319–322.

34.

, Fang

, Zheng

and Li

, Sample and structure-guidednetwork for road crack detection, IEEE Access 7 (2019), 130032–130043.

35.

, Zhuang

, Tang

and Zhang

, Object-based multilevelcontrast stretching method for image enhancement, IEEETransactions on Consumer Electronics 56 (2010), 1746–1754.

36.

Yao

, Image segmentation based on sobel edge detection, 5th International Conference on Advanced Materials and Computer Science, (2016), 141–144.

37.

Zhang

, Zhu

and Guan

X.F.

, Probe into image segmentation based on sobel operator and maximum entropy algorithm, in: Proceedings of the 2012 International Conference on Computer Science and Service System, IEEE Computer Society, USA, (2012), 238–241.

38.

Zhang

, Chang

and Wu

, Edge detection based on general grey correlation and log operator, in: 2010 International Conference on Artificial Intelligence and Computational Intelligence, (2010), 480–483.

39.

Zhang

, Yang

, Zhang

Y.D.

and Zhu

Y.J.

, Road crack detection using deep convolutional neural network, 2016 IEEE International Conference on Image Processing (ICIP), (2016), 3708–3712.

40.

Zheng

, Lei

and Zhang

, Intelligent detection of buildingcracks based on deep learning, Image and Vision Computing 103 (2020), 103987.

41.

Zou

, Cao

, Li

, Mao

and Wang

, Crack tree: Automaticcrack detection from pavement images, Pattern RecognitionLetters 33 (2012), 227–238.

Automatic road crack detection and classification using multi-tasking faster RCNN

Abstract

Keywords

1 Introduction

2 Related works

3 Ground truth label identification

Table 2 Comparison results of various noise removal filters Method Average MSE Salt and Pepper Noise 27.24 Guided Filter 221.46 Gaussian Filter 107.36 Wiener Filter 0.27

3.4 Edge detection

5.1 Dataset collection and generation

5.2.1 Metrics used for ground label identification

5.2.2 Metrics used for crack detection and classification

5.3 Analysis of results

References

Table 2
Comparison results of various noise removal filters

Method Average MSE

Salt and Pepper Noise 27.24

Guided Filter 221.46

Gaussian Filter 107.36

Wiener Filter 0.27