CDCL-VRE: An ensemble deep learning-based model for distracted driver behavior detection

3.1 Body segmentation algorithm

Since the CNN classification model discriminates all pixels of the input image, the driver images contain the driver’s body parts (including the head, body parts, arms, and hands) and the driving background (including the cabin, seat, vehicle’s various components, and external environment). The driving backgrounds are varied and cluttered, which cannot provide useful information about driver distraction to the CNN model. The images processed by the body part detection algorithm were input to the CNN model so that the model’s attention was focused on relevant regions, such as the driver’s body parts. It helps to provide driver distraction pixels of information and improve detection accuracy.

Body part segmentation is a challenging problem in the field of image segmentation. In this paper, the CDCL algorithm is used to segment the body parts of drivers. Detection of body parts aims to detect a person in an image as multiple semantically consistent regions (head, arms, legs) and highlights interested regions.

3.2 CNN

For a given image, two pixels that are close to each other are similar. In a convolutional neural network, each pixel is connected to a neuron. It rapidly increases the computational load. CNNs make image processing controllable at the computational level by filtering the connections between neurons based on similarity. For a given convolutional layer, instead of connecting each input to each neuron, the CNN restricts the connections so that any neurons can only receive the input (3*3 or 5*5) of a small fraction from the previous layer. Thus, each neuron is responsible for processing one region of the image. This approach significantly reduces the computational effort of the network. Therefore, CNNs are selected for application in the regions of image recognition in this study.

VGG-16 is proposed by K. Simonyan and A. Zisserman of Oxford University in the paper “Very deep convolutional networks for large-scale image recognition” [26]. In the 2014 ImageNet Large Scale Visual Recognition Chall(ILSVRC) [27] competition, the model achieved 92.7% accuracy in the first 5 tests. The dataset contains more than 14 million images belonging to 1000 classes. It is one of the notable models submitted for ILSVRC-2014.

Vanilla network is the a simplest RNN, with a simple model structure, a small number of parameters, a single input, and a single output.

The Resnet network has more layers than the VGG network. Resnet is proposed by Kaiming in the paper “Deep Residual Learning for Image Recognition”. The model won first place in the ILSVRC-2015 classification task in 2015 and won first place in the ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation tasks. The main contribution of the residual neural network is the discovery of the “Degradation” phenomenon and the invention of the “Shortcut connection” for the degradation phenomenon, which greatly eliminates the difficulty of training neural networks with too much depth.

3.3 Activation function

The activation function introduces a nonlinear element to the neurons, which allows the neural network to approximate any nonlinear function at will, allowing the neural network to be applied to a multitude of nonlinear models.

In deep learning, the commonly use saturated activation functions are sigmoid, tanh, ReLU, and softplus. sigmoid and tanh activation functions, whose derivatives tend to 0 when the value tends to infinity, and both have the problem of gradient disappearance. But the ReLU (Rectified Linear Units) is a non-saturated activation, which has great strength in preventing gradient disappearance. The Softplus function can be regarded as a smoothing of the ReLU function.

The purpose of this experiment is to identify distracted driver behaviors into ten categories, which is a multi-classification problem. The Softmax function is used in this study, which can “compress” a K-dimensional vector with any genuine number into another K-dimensional real vector. The output of the neuron is the ratio of the input of each neuron to the sum of the input of all neurons in the current layer. The output of each neuron is in the range of (0,1), and the sum of all outputs is 1. The larger the output probability of the neuron, the higher the probability that the neuron’s corresponding category is the genuine category. The equation of Softmax function in the text as Equation (1). σ ( z j ) = e z j ∑ k = 1 K e z k(1) where is the output value of the j-th node and K is the number of output nodes, i.e., the number of classified categories.

In multi-classification problems, the Softmax activation function is used in collocation with the categorical_cros cross-entropy loss function in general. The loss function evaluates the difference between the probability distribution obtained from the current training and the genuine distribution. It describes the distance between the actual output (probability) and the expected output (probability), i.e., the smaller the value of cross-entropy, the closer the two probability distributions are.

The label of each category is a 10-dimensional vector. The vector is set to 1 at the index position corresponding to the present value, and the rest is set to 0. The equation of Loss function in the text as Equation (2). Loss = - ∑ k = 1 outputsize y i * log y ˜ i(2) where y _i denotes the true label and y ˜ i denotes the predicted label.

According to the formula find that since there are only two values of 0 and 1 to take. And when y _i  = 0, Loss = 0, when y _i  = 1, the Loss will presence a value. That means categorical_crossentropy focuses on only one result.

When training a neural network model, if the model has too many parameters and the number of training samples is too small, the trained model is likely to be overfitted. The overfitting problem is often encountered when training neural networks. The dropout [28] technique is proposed to resolve the overfitting problem. The key to the Dropout technique is to ignore the randomly selected neurons in some specific periods during the training process, which can alleviate the occurrence of overfitting more effectively and achieve the effect of regularization to some extent. Dropout technology lead to the network learning more robust features that help improve generalization errors.

Real and synthetic humans both have a skeleton (pose) representation, the skeletons can effectively bridge the synthetic and real domains during the training. The CDCL [15] algorithm takes advantage of the rich and realistic variations of the real data and the easily obtained labels of the synthetic data. The algorithmic framework is trained using a ResNet-101 network and contains two main components. The first component is the synthetic input trained to learn body parts and human poses in the synthetic domain. The second component for real input training shares the network parameters of the backbone, keypoint mapping head, and part affinity field head with the first component. CDCL algorithm solves the problem of data labeling at the pixel-level with the low cost.

The CDCL algorithm is used to detect each part of the human body (torso, arms, legs, head, etc.) more accurately despite the absence of any human markers or the poor quality of the pictures (low brightness, complex environment, the high number of people, etc.). The CDCL algorithm outperforms several other state-of-the-art methods that require human markers, including the method proposed in [29–35].

The input images are processed with the CDCL algorithm (the source code can be got in GitHub [36]). The comparisons of the images before and after detection are shown in Fig. 2.

OPEN IN VIEWER

Fig. 2

CDCL body part detection (the original images are listed in the first row and the images processed by the CDCL algorithm are listed in the second row).

VGG-16 pre-trained CNN architecture containing 13 convolutional layers (convolutional kernel size of 3×3) and 5 pooling layers (pooling kernel size of 2×2) to extract features. The last 3 fully connected layers are responsible for the classification task. Since the convolution kernel focuses on expanding the number of channels and the pooling kernel focuses on reducing the width and height, it makes the model architecturally deeper and broader while the computational effort increases slowly. The purpose of this model is to classify distracted driver behaviors into ten categories, so a fully-connected layer is added at the end of the classifier and the Softmax function classifies the final categories into ten classes and outputs the probabilities of the classes to which they belong.

The VGG19 network is based on the VGG16 network with three additional 3*3 convolutional layers. The pre-trained model is used and retrains only the fully connected layers, so the parameters to be retrained are the same for the VGG16 and VGG19 networks.

4.3 Vanilla network

The Vanilla network has a simple structure and a small number of parameters, consisting of 6 convolutional layers, 3 pooling layers, and 3 fully connected layers. Although the model of Vanilla Network is simple. However, it has low time complexity and high accuracy for driving image classification. The convolutional layer uses the convolutional kernel of 3*3 size and the ReLU activation function, and the pooling layer uses the pooling kernel of 2*2 size. Convolutional and pooling layers are responsible for extracting features. The fully connected layers are responsible for the classification task. Finally, the probability of belonging to each category is output by a Softmax function. The Vanilla network is summarized in Fig. 3.

OPEN IN VIEWER

Fig. 3

Vanilla network model architecture.

The depth of the network has a role in the feature representation and generalization ability, but simply stacking convolutional layers does not improve the generalization ability of the model well. resnet50 has significantly fewer training parameters compared to the VGG network, even less than the AlexNet network, despite the deepening of the number of layers. This is mainly due to removing the fully connected layers and instead using global average pooling.

Resnet50 network contains five stages. In the first stage, the input images successively go through the convolution layer, batch normalization, activation function, maximum pooling layer, and output; stages 2-5 alternately use two residual learning blocks, and two primary residual learning blocks are shown in Fig. 4. It is able to solve the problem of gradient disappearance during the training process.

OPEN IN VIEWER

Fig. 4

Two types of Bottleneck structures.

The input shape of Bottleneck1 is (C, W, W), which refers to the number of channels, width, and height of the input. Here the width and height are equal, so there are two variable parameters C and W. Assuming that the input shape (C, W, W) is x, F(x) can be got by the three convolution blocks, which is added with x to get (F(x) + x), and then a RELU activation function is used to get the output, which is still in the shape (C, W, W).

Bottleneck2 has four variable parameters C, W, C1, and S. The right side of Bottleneck2 has one more convolutional layer than Bottleneck1, assuming that the G(x) can be got by the right branch, the F(x) can be got by the left branch, and the two branches add up to get (F(x) + G(x)), and then go through a RELU activation function to get the output. The parameter S indicates whether the two 1×1 convolutional layers on the left and right of Bottleneck2 are downsampled; the parameters C and C1 indicate whether the first 1×1 convolutional layer on the left side of Bottleneck2 reduces the number of channels.

Pre-trained CNN architectures are trained by large datasets (the ImagetNet dataset [35]), and the pre-trained CNN architectures already can extract shallow base features and deep abstract features. The models can be initializers or extractors features for similar tasks. Since general experiments may not have big training datasets, the network is generally not trained from scratch. Using a pre-trained CNN architecture without fine-tuning, the network may run the risk of non-convergence, under-optimized parameters, low accuracy, weak model generalization ability, and over-fitting.

4.6 Ensemble learning

Distracted driver behaviors may lead to critical consequences and bring heavy loss to people’s families and society, so it is especially important to improve the accuracy of recognition for distracted behavior. With the increasing number of vehicles worldwide, it is possible to save countless lives and families as long as the accuracy rate is improved.

CNNs are highly capable of learning and very sensitive to the details of the training data. Different networks learn different features and different focus on the training datasets, and the output of prediction results will have a certain tendency. A better and more successful method is proposed to solve this problem, which trains multiple models instead of a single model, and obtains the final prediction by combining the results of each training model, called the ensemble method [38]. There are several ensemble methods available, such as weighted voting, simple averaging, and GASEN [39].

The weighted average ensemble algorithm is used in this paper. The contribution of each ensemble member is weighted by a factor indicating the trust or expected performance of the model. The weight value is a value between 0 and 1, so the sum of the weights of all ensemble members is 1.

5.1 Dataset

In this section, the performance of the proposed model was validated by experiments, the module detected ten types of driving behaviors by multi-category classification. Ten behaviors include normal driving, typing with the right hand, answering the phone with the right hand, typing with the left hand, Answering the phone with the left hand, operating the radio, drinking, reaching behind, fixing hair or makeup, and chatting with passengers.

In the experiments, two publicly available datasets are used to evaluate the recognition performance of the models. The first dataset is the State Farm distracted driver dataset [40] to train the model and obtain classification accuracy. The second dataset is the American University in Cairo (AUC) distracted driver dataset [41] to validate the generalization ability of the model.

5.2 The comparison experiment of the activation function

The pre-trained CNN architecture and fine-tune method were used in this study. The parameters of the fully connected layers were retrained. The Adam optimizer was used in this study. The learning rate was set to 1 * e-3 when training the network pre-trained model; since the Vanilla network has few layers, a simple structure, and a small number of parameters. The Vanilla network was trained from scratch and the learning rate was set to 0.001.

Four activation functions were put into the model for training, and their accuracies are summarized in Table 2.

Table 2 shows that the ReLU activation function performs the best among the other activation functions. Because the ReLU function is non-saturated, which means there is no issue with gradient saturation. Although the ReLU function is not centered at 0, it has a gradient value of 0 in the negative direction, and weights are not updated during back-propagation. However, in the positive direction, the gradient value is 1. Additionally, the ReLU function is linear and can effectively solve the problem of gradient disappearance, leading to improved operational speed. As a result, The ReLU activation function was used for our model. The equation of ReLU activation function in the text as Equation (3). f ( x ) = { 0 , x ≤ 0 x , x > 0(3)

Four pre-trained models were evaluated on the State Farm distracted driver dataset, compared to before and after using the body segmentation algorithm, and the accuracy and number of parameters for the four pre-trained models. The training results are summarized in Fig. 5.

OPEN IN VIEWER

Fig. 5

Pre-trained network comparison chart.

As can be seen in Fig. 5, the classification accuracy of the four pre-trained models using the body segmentation algorithm is significantly improved compared to without the body segmentation algorithm, which proves the effectiveness of the body segmentation algorithm; at the same time, the number of parameters of the Resnet50 and Vanilla networks is significantly reduced compared to that of the VGG network and the accuracy is high, so the Resnet50 and Vanilla networks are selected to ensemble in this paper to reduce the sensitivity of individual networks to the data.

The State Farm distracted driver dataset was put into our model for experiments. The images of body parts were obtained by the CDCL algorithm and then input to the Vanilla network and Resnet50 network for training and testing, respectively. The variation of the accuracy and loss of the network with an increasing number of epochs is shown in Fig. 6.

OPEN IN VIEWER

Fig. 6

The variation of the accuracy and loss of the network with an increasing number of epochs.

The loss function decreases and the accuracy rate increases with the increase of training epochs from Fig. 8. The Vanilla model reaches the accuracy rate of 99.28%. The Resnet50 model reaches the accuracy rate of 99.46%.

The Vanilla network and ResNet-50 were ensembled in this paper for several training sessions, and the results of five randomly selected experiments are summarized in Table 3. The average accuracy is about 99.58%.

It can be observed from Table 3 that using the vanilla network for driver abnormal behavior classification alone, an average accuracy of 99.27% can be obtained on the State Farm driver behavior dataset. Using the ResNet-50 model for driver diastracted behavior classification alone, an average accuracy of 99.26% on the State Farm driver distracted dataset can be obtained. It is difficult to improve the accuracy any further by training two networks individually, and the ensemble of the two models achieves an average accuracy of 99.58%. The proposed model can reduce the sensitivity of a single network to the input data, and improve the detection accuracy.

The confusion matrix helps us to visualize the number of correct and incorrect predictions. It gives us an idea of how well the classifier model works on the test set. The confusion matrix forms the basis of most classification metrics, such as accuracy, precision, and recall, as these metrics are derived from the confusion matrix [43]. Figure 7 shows the confusion matrix of the two network models for ten classes of behavior recognition.

OPEN IN VIEWER

Fig. 7

Confusion matrix.

In the confusion matrix output by the Vanilla network, The network misclassifies chatting with passengers as safe driving three times because drivers may keep their hands on the steering wheel and only turn their heads while chatting with passengers, which is very similar to the state of safe driving and prone to cause misclassification of the system; The Vanilla network misjudges the behavior of typing with the right hand as chatting with passengers behavior two times; The Vanilla network misjudges the behavior of chatting with passengers as fixing hair or makeup three times, which is because when chatting with passengers, drivers may have more behaviors about hand, sometimes with both hands on the steering wheel, which is also the reason why the network misjudges the driver’s behavior of chatting with passengers as normal driving behavior. The content of the driver’s chat with passengers and the driver’s state of driving the vehicle may cause the driver to behave in various ways. In the confusion matrix output by the Resnet50 network. Resnet50 network misjudges the behavior of chatting with passengers as safe driving two times, typing with the right hand as chatting with passengers behavior two times, and the behavior of fixing hair or makeup as chatting with passengers two times, which is similar to the reasons for misclassification by the Vanilla network.

The confusion matrix after using the ensemble algorithm is shown in Fig. 8.

OPEN IN VIEWER

Fig. 8

Confusion matrix of using ensemble algorithm.

By comparing the confusion matrices in Figs. 7 and 8, it can be seen that the ensemble learning algorithm reduces the misclassification rate. The Vanilla network misclassified chatting with passengers as normal driving behavior twice, which was reduced to once using the ensemble algorithm; The Vanilla network misclassified chatting with passengers as fixing hair or makeup behavior three times, which was reduced to once using the ensemble algorithm; The ResNet50 network misclassified typing with the right-hand as drinking behavior three times, which was reduced to zero using the ensemble algorithm; The ResNet50 network misclassified typing with the right-hand as normal driving behavior once, which was reduced to zero using the ensemble algorithm; The ResNet50 network misclassified adjust the radio as chatting with passengers behavior once, which was reduced to zero using the ensemble algorithm. The effectiveness of the ensemble algorithm was demonstrated, which reduced the sensitivity of individual networks to data and improved classification accuracy.

The evaluation metrics such as Precision, Recall, and F1-score can be calculated from the data in the confusion matrix with the equations in the text as Equations (4–6). Precision = TP TP + FP(4) Recall = TP TP + FN(5) F 1 - score = 2 PR P + R(6) Where P stands for Percision and R stands for Recall.

The Precision, Recall, F1-score, Support, micro avg, macro avg, weighted avg, and sample avg of the Vanilla network in each category of the test set are summarized in Table 4. The values of each category are above 0.98. The Precision, Recall, F1-score, Support, micro avg, macro avg, weighted avg, and sample avg of the Resnet50 network in each category of the test set are shown in Table 5.

The results of comparing the model proposed in this paper with other state-of-the-art methods using the same dataset are shown in Table 6. The method in this study outperformed other state-of-the-art methods in identifying driver distraction behavior.

The model proposed by Imen JEGHAM [15] used the SURF keypoint technique to detect driver body regions and used the K-NN method for classifying driver behaviors, obtaining an average accuracy of 70% on the State Farm dataset; The model proposed by Furkan Omerustaoglu [18] introduced a time series. They built a CNN model on the driver image data and an LSTM model on the time series of control data of the vehicle. Finally, the two models built were provided to the ensemble algorithm to obtain the final LSTM model and obtained 85.00% accuracy on the State Farm dataset; Ardhendu Behera [20] also introduced time series and used the driver image as well as the temporal information of the driver image to build an LSTM model on the State Farm dataset to obtain an accuracy of 91.25%; C. Huang [22] proposed the HCF framework using class activation mapping technique to highlight the region of interest of the pre-trained models, The features were extracted by three pre-trained CNN architectures: ResNet50, Inception V3 and Xception in parallel and were fused into a one-dimensional vector that was sent to the Softmax function for classification. An average of 96.74% was achieved on the State Farm distracted driver dataset. But the three pre-trained CNN architectures have a large number of parameters and high computational complexity; The ResneXt101-S2F model proposed by Lei Zhao [23] has a lower accuracy than the ResneXt101-S3F model, but is faster than the ResneXt101-S3F model, and it is difficult to balance the accuracy with the computational effort. The model proposed in this paper used a CDCL algorithm to highlight the region of interest of the CNN. The Resnet50 network and the Vanilla network were provided to the ensemble algorithm that determined the driver behaviors, and the model obtained an average accuracy of 99.58% on the State Farm dataset

The robustness and generalization of the model proposed in this paper were further evaluated using the AUC dataset. 80% of the dataset was used for training and the rest for testing for dividing the dataset. The authors of the AUC dataset used a temporal annotation mechanism to label the recorded videos. Each video frame was labeled as a category. However, by looking at the image markers, it can be noticed that in some cases the authors failed to correctly select the beginning and end of the video sequence, which led to some images being annotated with errors [21]. Therefore, we manually filtered all the classes. Removed and changed the incorrect video frames.

The manually processed dataset was applied to our proposed method. Five training results after several training sessions were randomly selected. The results are summarized in Table 7. The model achieved an average accuracy of 98.2%. Error annotations can interfere with the recognition of the model. The high accuracy of the AUC dataset in our model demonstrated the strong performance and generalization ability of our proposed model.