Recognition of driver’s fatigue expression using Local Multiresolution Derivative Pattern

Abstract

To develop the human-centric driver fatigue monitoring system for automatic understanding and charactering of driver’s conditions, a novel, efficient feature extraction approach, named Local Multiresolution Derivative Pattern (LMDP), is proposed to describe the driver’s fatigue expression images, and the Intersection Kernel Support Vector Machines classifier is then exploited to recognize three pre-defined classes of fatigue expressions, i.e., awake expressions, moderate fatigue expressions and severe fatigue expressions. With features extracted from a fatigue expressions dataset created at Southeast University, the holdout and cross-validation experiments on fatigue expressions classification are conducted by the Intersection Kernel Support Vector Machines classifier, compared with three commonly used classification methods including the k-nearest neighbor classifier, the multilayer perception classifier and the dissimilarity-based classifier. The experimental results of holdout and cross-validation showed that LMDP offers the better performance than Local Derivative Pattern, and the second order LMDP exceeds other order LMDP. With the second order LMDP and the Intersection Kernel Support Vector Machines classifier, the classification accuracies of the severe fatigue are over 90% in the holdout and cross-validation experiments, thus demonstrating the effectiveness of the proposed feature extraction method in automatically understanding the driver’s conditions towards the human-centric driver fatigue monitoring system.

Keywords

Fatigue expression feature extraction Local Multiresolution Derivative Pattern (LMDP)Local Derivative Pattern (LDP)Human-centric Driver Fatigue Monitoring System (DFMS)

1 Introduction

European Transport Safety Council (ETSC) defines four levels of sleepiness based on behavioral terms, i.e., completely awake, moderate sleepiness, severe sleepiness, and sleep [1]. Studies show that 25% –30% of driving accidents are fatigue related, and most sleepy drivers tried to fight against sleep with moderate sleepiness, severe sleepiness [2]. When a driver is fatigued, certain physical and physiological phenomena can be observed, including changes in brain waves or EEG, eye activity, facial expressions, body sagging posture, gripping force on the steering wheel, and other changes in body activities. Recently, Human-centric Driver Fatigue Monitoring System (DFMS) is developed for monitoring the attention status of the driver, and different countermeasures, depending on the types and levels of fatigue, should be taken to maintain driving safety.

The Physical measures of driver’s fatigue include blink frequency, eye closure duration, nodding frequency, fixed gaze, and frontal face pose. In [3], the different linguistic terms and their corresponding fuzzy sets were distributed in each of the inputs using induced knowledge based on the hierarchical fuzzy partitioning method, and three variables (fixed gaze, PERCLOS, and ECD) were determined to be crucial cues for detecting a driver’s fatigue. Suzuki et al. derived the following three factors from the blinking waveform [4], and these factors were then weighted using a multiple regression analysis for each individual to calculate the drowsiness level. Eskandarian et al. utilized artificial neural network to analyze vehicle parameter data and eye-closure data to infer driver fatigue, and identify the potential variables that were correlated with drowsiness [5]. Orazio et al. used a mixture Gaussian model to model the “normal behavior” statistics from the ECD and frequency of eye closure for each person to identify anomalous behaviors [6]. Barr et al. reviewed and evaluated the noninvasive drivers’ monitoring technologies of vehicle-based operator alertness/drowsiness/vigilance [7], such as DD850 Driver Fatigue Monitor (DFM) designed by Attention Technology Inc., Driver State Monitor (DSM) developed by Delphi Inc., Seeing Machines faceLAB^TM, Smart Eye AB, InSight^TM developed by SensoMotoric Instruments GmbH Inc., Video-based Eye Tracking Systems ETS-PC II developed by Applied Science Laboratories, Eyegaze Analysis System developed by LC Technologies Inc., Drowsy Driver Detection System designed by Johns Hopkins University Applied Physics Laboratory, RPI computer vision system for monitoring driver vigilance by Rensselaer Polytechnic University, Drowsiness Detecting System based on Artificial Neural Network developed by George Washington University. Friedrichs and Yang explored 18 features of eye movement for drowsiness detection [8], and chose the sequential floating forward selection algorithm to select the most promising features to construct a classifier. Fan et al. utilized a Gabor features representation of the face for fatigue detection [9], and then AdaBoost algorithm was used to extract the most critical features from the dynamic feature set and construct a strong classifier for fatigue detection. In [10], it also reported that sleep deprived drivers have a lower frequency of steering reversals, a deterioration of steering performance, a decrease in the steering-wheel reversing rate, more frequent steering maneuvers during wakeful periods, no steering correction for a prolonged period of time followed by a jerky motion during drowsy periods, low-velocity steering, large amplitude steering-wheel movements, and large standard deviations in the steering-wheel angle.

The biological measures of driver’s fatigue include Electroencephalography (EEG), Electrocardiogram (ECG), Electro-oculography (EOG), and Surface Electromyogram (SEMG). These signals can be collected through electrodes in contact with the skin of the human body. Recent research has proposed various methods of extracting features from a segment of raw EEG data for fatigue detection. Lin et al. established a linear regression model to estimate the drowsiness level from the independent component analysis of 33-channel EEG signals and could estimate the drowsiness level with 87% accuracy [11, 12], and they then implemented a real-time embedded EEG-based driver drowsiness estimate system, which adopted only four channels of EEG data. Damousis et al. selected eight eye activity features, extracted from EOG, to develop a fuzzy expert system for the detection of hypovigilance [13]. Yeo et al. trained SVM to classify EEG signals into four principal frequency bands and then to predict the transition from alertness to drowsiness [14]. Jap et al. accessed four EEG activities for 52 subjects during a monotonous driving session, and the results showed an increase in the ratio of slow wave to fast wave EEG activities over time [15]. Hu and Zheng employed a support vector machines to perform drowsiness prediction with 11 eyelid-related features extracted from EOG [16]. Yang et al. employed a dynamic BN with EEG and ECG to estimate fatigue, and a first-order HMM was employed to compute the dynamics of a BN at two different times slices [17]. In [18], kernel principal component analysis algorithm was employed to extract nonlinear features from the complexity parameters of EEG and improve the generalization performance of an HMM. Sibsambhu et al. presents a method based on a class of entropy measures on the recorded EEG signals of human subjects for relative quantification of fatigue during driving [19]. Rami et al. developed an efficient fuzzy mutual-information-based wavelet packet transform feature-extraction method for classifying the driver drowsiness state into one of predefined drowsiness levels [20]. Recently, a powerful operator, called Local Derivative Pattern (LDP) was proposed by Zhang for face recognition [21]. LDP encodes directional pattern features based on local derivative variations, and captures more detailed discriminative information of face. But LDP can’t obtain the relationship information of local high pattern for the intrinsic geometrical structures (locations and smoothed contours of mouth, eyes and eyebrows. This disappointing behaviour of LDP in fatigue expression indicates that more powerful representations are needed in higher dimensions.

With the aim to explore the intrinsic geometrical structure of fatigue expression images, we propose a novel, efficient features extraction approach, named Local Multiresolution Derivative Pattern (LMDP) for fatigue expressions descriptions of vehicle drivers. The paper is organized as follows. In Section 2, the background of SEU (Southeast University) fatigue facial expression dataset is outlined. In Section 3, Local Multiresolution Derivative Pattern (LMDP) for image feature description of fatigue expressions is introduced. Intersection Kernel Support Vector Machines (ISKVMs), compared with k-Nearst Neighbor (kNN), Multilayer Perceptron (MLP) and Dissimilarity classifier are presented in Section 4. Section 5 details the experiments and reports on the classification results for the fatigue expressions of vehicle drivers. Section 6 presents our conclusions.

2 Fatigue expression data acquisition

Traffic accidents are often related to moderate sleepiness and severe sleepiness. The sign of moderate sleepiness is that vehicle driver’s repeat yawning, and the sign of severe sleepiness is that vehicle drivers have difficulties in keeping eyes open and nodding off at the wheel [1]. We created a fatigue expressions dataset, named Southeast University (SEU) fatigue expressions dataset, which consists of three kinds of fatigue expressions images, i.e., awake expressions, moderate fatigue expressions and severe fatigue expressions. The SEU fatigue expression dataset was created using a side-mounted Logitech C905 CCD camera and includes 120 fatigue expressions images. There are 20 male drivers and 20 female drivers in SEU fatigue expression dataset, and the lighting conditions varied under the natural conditions, as the car was in an outdoor parking lot. In this paper, Viola-Jones face detection algorithm was used to detect the vehicle driver’s faces [22]. The major contributions of Viola-Jones face detection algorithm include the exploitation of Haar-like feature expression and AdaBoost learning. In our data acquisition and normalization, Viola-Jones face detection algorithm offers excellent performance in detecting the driver’s face even when a driver turns his/her face to look at rear-mirrors, or lower his/her head to operate the shift lever. At finally, SEU fatigue expression dataset consists of 40 subjects with 3 different sessions per subject and in total 120 sessions. The Fig. 1(a) shows example images of a driver inside the vehicle, and the corresponding face detection result is shown in Fig. 1(b).

3 Local Multiresolution Derivative Pattern

In this section, a brief review of Local Derivative Pattern (LDP) is presented, and then Local Multiresolution Derivative Pattern (LMDP) is introduced in details.

3.1 Local Derivative Pattern

Local Derivative Pattern (LDP) encodes the higher-order derivative information of an image [21], which is derived from a general definition of texture in a local neighbourhood. Given an image I (Z), the first-order derivative is denoted as I′ (Z). Let Z₀ be a point in I (Z), and Z_i, i = 1, …, 8 be the neighboring point around Z₀. The first-order derivatives in at Z = Z₀ can be written as $I^{'} (Z_{i}) = I (Z_{0}) - I (Z_{i})$ (1)

The second-order local derivative pattern, LDP² at Z = Z₀ is defined as $\begin{matrix} {LDP}^{2} (Z_{0}) & = {f (I^{'} (Z_{0}), I^{'} (Z_{1})), f (I^{'} (Z_{0}), \\ I^{'} (Z_{2})), \dots, f (I^{'} (Z_{0}), I^{'} (Z_{8}))} \end{matrix}$ (2)where f (· , ·) is a binary coding function determining the types of local pattern transitions, and encodes the co-occurrence of two directions at different neighboring pixels as $f (I^{'} (Z_{0}), I^{'} (Z_{i})) = {\begin{matrix} \begin{matrix} 0, & if I^{'} (Z_{0}) \cdot I^{'} (Z_{i}) > 0 \end{matrix} \\ \begin{matrix} 1, & if I^{'} (Z_{0}) \cdot I^{'} (Z_{i}) \leq 0 \end{matrix} \end{matrix} \begin{matrix} i = 1, 2, \dots, 8 \end{matrix}$ (3)

From Equation 3, it can be seen that the second-order LDP encodes the change of the neighbourhood derivative directions, which represents the second-order pattern information in the local region. To calculate the nth-order Local Derivative Pattern, the nth-order LDP is a binary string describing gradient trend changes in a local region of directional (n–1)th-order derivative images I^n-1 (Z) as

$\begin{matrix} {LDP}^{n} (Z_{0}) & = {f (I^{n - 1} (Z_{0}), I^{n - 1} (Z_{1})), \\ f (I^{n - 1} (Z_{0}), I^{n - 1} (Z_{2})), \dots, \\ f (I^{n - 1} (Z_{0}), I^{n - 1} (Z_{8}))} \end{matrix}$ (4)where I^n-1 (Z₀) is the (n - 1) th-order derivative in at Z = Z₀, and f (I^n-1 (Z₀) , I^n-1 (Z_i)) is defined as $f (I^{n - 1} (Z_{0}), I^{n - 1} (Z_{i})) = {\begin{matrix} \begin{matrix} 0, & if I^{n - 1} (Z_{0}) \cdot I^{n - 1} (Z_{i}) > 0 \end{matrix} \\ \begin{matrix} 1, & if I^{n - 1} (Z_{0}) \cdot I^{n - 1} (Z_{i}) \leq 0 \end{matrix} \end{matrix} \begin{matrix} i = 1, 2, \dots, 8 \end{matrix}$ (5)

The high-order local patterns provide a stronger discriminative capability in describing detailed texture information than the first-order local pattern. The higher the order is, the more details the local pattern operator can extract from the image, but over-detailed patterns tend to be noise instead of identity information.

3.2 Local Multiresolution Derivative Pattern

For a human visual system to capture the essential information of a natural scene, it is well-known that a computational image representation based on a local, directional and multiresolution expansion will be efficient. With this insight, the sparse expansion for fatigue expression image can be obtained by applying a Laplacian Pyramid (LP) with orthogonal filters [23], followed by a Directional Filter Bank (DFB) [24]. The LP with orthogonal filters is used to capture the point discontinuities, and the DFB is used to link point discontinuities into linear structures. In the frequency domain, the structure of LP with orthogonal filters, followed by a two-dimensional DFB, provides a multiscale and directional decomposition to obtain sparse expansions for images having smooth contours. And then a high-order Local Multiresolution Derivative Pattern (LMDP) can be used for capturing the detailed discriminative information of contours of mouth, eyes and eyebrows in sparse representation images of fatigue expressions. The structure of Local Multiresolution Derivative Pattern (LMDP) with 2 levels is presented in Fig. 2. Bandpass images from LP are fed into a DFB so that directional information can be captured. The scheme can be iterated on the coarse images, which can be decomposed into directional subbands at multiple scales. With orthogonal filters, the BFB is an orthogonal transform.

The Laplacian Pyramid (LP) decomposition with orthogonal filters at each level generates a downsampled lowpass version of the original and the difference between the original and the prediction, resulting in a bandpass image. In this paper, the “9-7” biorthogonal filters is adopted in the LP stage. Figure 3 depicts this decomposition process, where H, G and M are called (lowpass) analysis filter, symthesis filter and sampling matrix respectively. This process can be iterated on the coarse (downsampled lowpass) image, and the outputs are a coarse approximation a[n] and a difference b[n] between the original image and the prediction.

The orthogonal lowpass filter G in a multilevel LP is an orthogonal scaling function φ (t) ∈ L² (R) that generates a multiresolution analysis (MRA) represented by a sequence of nested subspaces {V_j } _j∈Z, …V₂⊂ V₁ ⊂ V₀ ⊂ V_-1 ⊂ V_-2 … with Closure (⋃ _j∈ZV_j) = L² (R) and ⋂_j∈ZV_j ={ 0 }. The scaling function φ is specified from the filter G via the two-scale equation: $φ (t) = {| M |}^{1 / 2} \sum_{n \in Z} g [n] φ (M t - n)$ (6)where φ_j,n (t) = |M|^-j/2φ (M^jt - n), j ∈ Z, n ∈ Z. Then the family {φ_j,n } _n∈Z is an orthonormal basis of V_j for all j ∈ Z. Define W_j to be the orthogonal complement of V_j in V_j-1: $V_{j - 1} = V_{j} \oplus W_{j}$ (7)

Denote F_i (0 ≤ i ≤ |M| - 1) the highpass filters, and associate to each of these filters a continuous function ψ⁽ⁱ⁾ where $ψ^{(i)} (t) = {| M |}^{1 / 2} \sum_{n \in Z} f_{i} [n] φ (M t - n)$ (8)

These functions also generate families of scaled and translated functions as $ψ_{j, n}^{(i)} = {| M |}^{- j / 2} ψ^{(i)} (M^{- j} t - n) j \in Z, n \in Z$ (9)

Using the two-scale equations for φ (t) and ψ (t), the coarser image c^(j) [n] and the detail image d^(j) [n] in j-th scale of the LP decomposed image using the “9-7” symmetric biorthogonal filters is the inner product of the input sequence c^(j-1) [n] at the j - 1 scale can be written as $c^{(j)} [n] = \sum_{k \in Z} c^{(j - 1)} [k] \cdot g [k - M n]$ (10) $d^{(j)} [n] = \sum_{k \in Z} c^{(j - 1)} [k] \cdot f_{i} [k - M n]$ (11)

The directionality, which is a crucial feature for an efficient image representation, is supported by recent studies to identify the sparse components of natural images. A simplified construction for the two Dimensional Filter Bank (DFB) was proposed [24], which is efficiently implemented via an l-level binary tree decomposition that leads to 2^l subbands with wedge-shaped frequency partitioning. The simplified DFB is intuitively constructed from two building blocks, which are a two-channel quincunx filter bank with fan filters and a shearing operator. Using multirate identities, it is instructive to view an l-level tree-structured DFB equivalently as a 2^l parallel channel filter bank with overall sampling matrices, as shown in Fig. 4.

Consider a Multi-channel Directional Filter Bank (MDFB), which results from 2^l channels with equivalent filters and diagonal sampling matrices $S_{k}^{(j)}$ . Obviously, all the channels in the MDFB have the same sampling density, which is equal to the number of channels, as det(S_k) = 2^l, for k = 0, …, 2^l - 1. Each bandpass image d^(j) [n] is further decomposed by the 2^j level DFB into the bandpass directional images $y_{k}^{(j)} [m]$ , k = 0, 1, …, 2^j - 1. The “23–45” biorthogonal quincunx filters is adopted in the DFB stage. $y_{k}^{(j)} [m] = 〈 d^{(j)} [n], E_{k}^{(j)} [S_{k}^{(j)} m - n] 〉$ (12)where k and m are the direction and location indices, respectively. And the corresponding diagonal sampling matrices is shown to have the following diagonal forms $S_{k}^{(j)} = {\begin{matrix} \begin{matrix} diag (2^{k}, 2), & for 0 \leq k < 2^{l} - 1 \end{matrix} \\ \begin{matrix} diag (2, 2^{k}), & {for 2}^{l - 1} \leq k < 2^{l} \end{matrix} \end{matrix}$ (13)

The family ${E_{k} [S_{k}^{(j)} - \cdot]}_{0 \leq k \leq 2^{l}, m \in Z}$ obtained from the time-reversed version of the analysis filters E_k, k = 0, …, 2^l - 1. The directional filter bank is a powerful mechanism for decomposing images into local and directional expansions. The filter bank is implemented efficiently with a tree structure.

Let $c_{0}^{(j)} [n]$ is a center point in low frequency content of LP image, and $c_{i}^{(j)} [n]$ , i = 1, …, 8, is the 8 neighboring points around $c_{0}^{(j)} [n]$ , among which the point $c_{8}^{(j)} [n]$ is the right side point of $c_{0}^{(j)} [n]$ . Let $d_{k, 0}^{(j)} [n]$ is a center point in high frequency content of DFB image in k direction, and $d_{k, i}^{(j)} [n]$ , i = 1, …, 8, is the 8 neighboring points around $d_{k, 0}^{(j)} [n]$ , among which the point $d_{k, 8}^{(j)} [n]$ is the right side point of $d_{k, 0}^{(j)} [n]$ . The first order Derivative Patterns (DP) of 8 neighboring points along the horizontal direction at point $c_{0}^{(j)} [n]$ and point $d_{k, 0}^{(j)} [n]$ can be written as $\begin{matrix} {DP}_{i}^{'} & = \sum_{k \in Z} c_{i}^{j - 1} [k] \cdot g [k - M n] \\ - \sum_{k \in Z} c_{8}^{j - 1} [k] \cdot g [k - M n], i = 0, \dots, 7 \end{matrix}$ (14) $\begin{matrix} {DP}_{k, i}^{'} & = 〈 \sum_{k \in Z} d_{i}^{(j - 1)} [k] \cdot f [k - M n], E_{k}^{(j)} [S_{k}^{(j)} m - n] 〉 \\ - 〈 \sum_{k \in Z} d_{8}^{(j - 1)} [k] \cdot f [k - M n], E_{k}^{(j)} [S_{k}^{(j)} m - n] 〉, \\ i = 0, \dots, 7 \end{matrix}$ (15)

The first-order Low Frequency Derivative Pattern (LFDP) at point $c_{0}^{(j)} [n]$ and the first-order High Frequency Derivative Pattern (HFDP) at point at point $d_{k, i}^{(j)} [n]$ are defined as the concatenation of the first-order derivatives at 8 neighboring points ${LFDP}^{'} (c_{0}^{(j)}) = {{DP}_{0}^{'}, {DP}_{1}^{'}, \dots, {DP}_{7}^{'}}$ (16) ${HFDP}^{'} (d_{k, 0}^{(j)}) = {{DP}_{k, 0}^{'}, {DP}_{k, 1}^{'}, \dots, {DP}_{k, 7}^{'}}$ (17)

The second-order LFDP and the second-order HFDP are defined as $\begin{matrix} {LFDP}^{2} (c_{0}^{(j)}) & = {f ({DP}_{0}^{'}, {DP}_{8}^{'}), \\ f ({DP}_{1}^{'}, {DP}_{8}^{'}), \dots, f ({DP}_{7}^{'}, {DP}_{8}^{'})} \end{matrix}$ (18) $\begin{matrix} {HFDP}^{2} (d_{k, 0}^{(j)}) & = {f ({DP}_{k, 0}^{'}, {DP}_{k, 8}^{'}), \\ f ({DP}_{k, 1}^{'}, {DP}_{k, 8}^{'}), \dots, \\ f ({DP}_{k, 7}^{'}, {DP}_{k, 8}^{'})} \end{matrix}$ (19)where f (· , ·) is a binary coding function determining the types of local pattern transitions. It encodes the co-occurrence of two derivative directions at different neighboring pixels as $f (A, B) = {\begin{matrix} \begin{matrix} 0, & if A \cdot B > 0 \end{matrix} \\ \begin{matrix} 1, & if A \cdot B \leq 0 \end{matrix} \end{matrix} \begin{matrix} i = 0, \dots, 7 \end{matrix}$ (20)

In a general formulation, the nth-order LFDPⁿ and the nth-order HFDPⁿ are a binary string describing gradient trend changes in a local region of directional (n - 1) th-order derivative images LFDPⁿ and HFDPⁿ - 1

$\begin{matrix} {LFDP}^{n} (c_{0}^{(j)}) & = {f ({DP}_{0}^{n - 1}, {DP}_{8}^{n - 1}), \\ f ({DP}_{1}^{n - 1}, {DP}_{8}^{n - 1}), \dots, \\ f ({DP}_{7}^{n - 1}, {DP}_{8}^{n - 1})} \end{matrix}$ (21)

$\begin{matrix} {HFDP}^{n} (d_{k, 0}^{(j)}) & = {f ({DP}_{k, 0}^{n - 1}, {DP}_{k, 8}^{n - 1}), \\ f ({DP}_{k, 1}^{n - 1}, {DP}_{k, 8}^{n - 1}), \dots, \\ f ({DP}_{k, 7}^{n - 1}, {DP}_{k, 8}^{n - 1})} \end{matrix}$ (22)where LFDPⁿ - 1 is the (n - 1) th-order derivative along the horizontal direction at point $c_{0}^{(j)} [n]$ in scale j, and HFDPⁿ - 1 is the (n - 1) th-order derivative along the scale j and k direction at point $d_{k, 0}^{(j)} [n]$ .

To extract the discriminative LMDP features of a fatigue expression image, the spatial histograms can be used to model the distribution of the high-order LMDP, because it is more robust against variations in an illumination than the holistic methods. In this paper, taking the spatial histograms of the subregions and concatenating them into an enhanced feature vector as the fatigue expression image descriptor. The special histogram of a fatigue expression image is represented as $H_{LMDP} = H_{{LFDP}^{n}} (c^{(j)}) + \sum_{k = 0}^{2 l - 1} H_{{HFDP}^{n}} (d_{k}^{j})$ (23)

To balance the identification accuracy and feature length, we selected the parameters of the 16×16 sub-regions with 32 histogram bins for representing the LMDP images. And the joint feature of histogram is 1×8192(16×16×32) dimension. LMDP offers a much richer set of directions and shapes, and thus they are more effective in capturing smooth contours and geometric structures in images. The visualized feature results of the 2th-order LDP and the 2-th-order LMDP for a fatigue expression image Fig. 1(b) are shown in Fig. 5.

4 Classification methods compared

Support vector machines (SVMs) are supervised learning models with associate learning algorithms that analyse data and recognize patterns, used for classification [25]. The k-nearest neighbour algorithm (k-NN) is a method for classifying objects based on closest training examples in feature space [29]. Multilayer perception (MLP) classifier is a modification of the standard linear perceptron [30], which can distinguish data that is not linearly separable. In Dissimilarity-based classifier [31], the dissimilarities computed between an object and its prototypes are used for object’s classification. The above four classifiers are most commonly used in pattern classification, and adopted to classify fatigue expressions in our study.

4.1 Intersection Kernel Support Vector Machines

Support Vector Machines (SVMs) were originally designed for binary-class classification problems, and the basic principle of binary-class SVMs is to find an optimal separating hyperplane (OSH) and separate two classes of patterns based on the training set and the decision boundary [25]. In order to solve the multi-class problem, a variety of schemes have been proposed in the literature [26, 27], such as One Against All (OAA), One Against One (OAO), Directed Acyclic Graph (DAA), Error Correcting Output Coding (ECOC), and Multiclass objective function by adding bias to the objective function. Recently, Maji et al. proposed a fast IKSVM with an approximation scheme whose time and space complexity is O (n), independent of the number of support vectors, and have been shown to be successful for the objects’ detection and recognition [28]. In this paper, we adopt the fast IKSVMs for the classification of fatigue expressions of vehicle drivers. The key idea of fast IKSVMs proposed by Maji et al. is that for a class of kernels including the intersection kernel, the classifier can be decomposed as a sum of functions, one for each histogram bin, each of which can be efficiently computed. For feature vectors x, $z \in R_{+}^{n}$ , the intersection kernel is $k (x, z) = \sum_{i = 1}^{n} min (x (i), z (i))$ (24)

and classification is based on evaluating $h (x) = \sum_{l = 1}^{m} α_{l} y_{l} (\sum_{i = 1}^{n} min (x (i), x_{l} (i))) + b$ (25)

Thus the runtime complexity of computing h (x) is O (n log m).

4.2 k-Nearest Neighbor (kNN) classifier

k-nearest neighbor (kNN) classifier [29] is a method for classifying objects based on closest training examples in the feature space. KNN is a type of instance-based learning, or lazy learning where the function is only approximated locally and all computation is deferred until classification. The k-nearest neighbor algorithm is amongst the simplest of all machine learning algorithms: an object is classified by a majority vote of its neighbors, with the object being assigned to the class most common amongst its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of its nearest neighbor. The kNN rule is optimal in the asymptotic case, i.e., the error tends to the Bayes error when the size of the training set tends to infinity. The major drawback of the kNN algorithm is the computational complexity, caused by the large number of distance computations.

4.3 Multilayer perception (MLP) classifier

A Multilayer perception (MLP) classifier is a feedforward artificial neural network model that maps sets of input data onto a set of appropriate output, and consists of a set of source nodes forming the inputlayer, one or more hidden layers of computation nodes, and an output layer of nodes [30]. The MLP utilizes a supervised learning technique called backpropagation for training the network and constructs input-output mappings that are a nested composition of nonlinearities with the form $y = f (\sum g (\sum (\cdot)))$ (26)where the number of function compositions is given by the number of network layers. An MLP can be trained by gradient descent using the back-propagation algorithm to optimize any derivable criterion, such as the mean squared error, Here, an MLP is trained to classify an input to be one of the given class labels. The input of the MLP is a vector corresponding to the features extracted from gait patterns. The output of the MLP is either 1 (if the input corresponds to a control group) or –1 (if the input corresponds to children with CP). Though MLPs have been proved to be able to virtually approximate any function with any desired accuracy, there is a common criticism for MLP that is very difficult to interpret the trained discriminant function.

4.4 Dissimilarity-based classifier

In the dissimilarity-based classification, the dissimilarity measure D (x_i, R) ={ d (x_i, p₁) , …, d (x_i, p_r) } between an object x_i ∈ T, 1 ≤ i ≤ n, and the prototypes R ={ p₁, …, p_r }, is a vector with r distance that associates x_i with all objects in the representation set R [31]. Therefore, the proximity D (T, R) is a dissimilarity matrix of size n × r, which refers objects in the training set to all objects in the representation set. Given a test set S, its representation D (S, R) is obtained by calculating the distances between its objects and prototypes in R. The dissimilarity measure is small when the objects x_i and p_h are similar, but it should be larger when the objects are more different. The distance d (x_i, p_h) = 0 when x_i and p_h are identical. In this paper, the Euclidean distances are used for the dissimilarity representation between the objects and prototypes.

5 Experiments

Two standard experimental procedures, named the holdout approach and the cross-validation approach, are used to evaluate the performance of LMDP versus LDP using IKSVMs classifier, compared with other three commonly used classifiers, namely, kNN classifier, MLP classifier and Dissimilarity-based classifier. SEU fatigue facial expression dataset, which is shown in Fig. 6, is used for the performance evaluation of the above two feature extraction approaches and four classifiers in the holdout and cross-validation experiments. There are 3 classes of fatigue expressions, i.e., awake, moderate fatigue and severe fatigue in SEU fatigue facial expression dataset. Each class of fatigue expressions consists of 40 images, which are captured in different time under the natural lighting conditions, as the car was in an outdoor parking lot.

5.1 Holdup experiments

Holdout experiments are based on randomly dividing feature vectors of fatigue expression images, extracted from images in SEU fatigue facial expression dataset, into a training dataset (80% feature vectors of fatigue expression extracted from images in SEU fatigue expression dataset) and a test dataset (the rest 20% feature vectors of fatigue expression extracted from the images in SEU fatigue expression dataset). Using the holdout experiment approach, only the test dataset is used to estimate the generalization error. We repeat the holdout experiment 100 times by randomly splitting the fatigue expression dataset, and recorded the classification results.

The comparative experiments between LMDP and LDP are first conducted in the first holdout experiments. The results of classification rate for fatigue expressions using LMDP versus LDP by IKSVMs classifier are displayed in the bar plots of Fig. 7(a) and box plots of Fig. 7(b). From Fig. 7, it is obvious that the average recognition accuracy of fatigue expressions is significantly improved using LMDP than using LDP. The experimental results in Fig. 7 also demonstrated that the high-order LMDP offers the better performance than the first-order LMDP, but the performance drops when the order of LMDP reach to the third-order and four-order. The experimental results reveal that the high-order local patterns, such as the second-order LMDP, can extract more detailed information than the first order, but it is incapable of dealing with further detailed information contained in the higher-order LMDP, such as the third-order and four-order LMDP.

In the second holdout experiment, the same set of training and testing are applied to IKSVMs, compared with kNN classifier, MLP classifier and Dissimilarity-based classifier, and their classification performances are simultaneously compared. The results of classification rate in average for fatigue expressions by IKSVMs with other three classifiers are displayed in the bar plots of Fig. 8(a) and box plots of Fig. 8(b). The average classification accuracies of IKSVMs classifier, kNN classifier, MLP classifier and Dissimilarity-based classifier, are 92.92% , 81.83% , 92.79% and 89.08% , respectively. From Fig. 8, it is obvious that IKSVMs classifier offers the best performance among the four classifiers in the second holdoutexperiments.

The confusion matrix, which represents the proportion of examples from one class classified into another class, is often used for further measuring the classification performance regarding the information about actual and predicted classifications acquired. In the holdout experiment, the confusion matrix that summarizes the detailed performance using the second-order LMDP and IKSVMs classifier is shown in Table 1. In the confusion matrix, the rows and columns indicate true and predicted class, respectively. The diagonal entries represent correct classification, while the off-diagonal entries represent incorrect ones. The rows and columns of confusion matrices express the fatigue expression classes of awake expressions, moderate fatigue expressions, and severe fatigue expressions. The corresponding classification accuracies are 88.24% , 98.77% and 90.46% , respectively. From confusion matrix of the holdout experiments, it is clear that the class of awake expression has the most recognition accuracy of three classes in the holdout experiments.

5.2 Cross-validation experiments

The k-fold cross validation approach is another commonly used technique that takes a set of m examples and randomly partitions them into k folds of size m/k. For each fold, the classifier is tested on one fold (consists of m/k examples) and trained on the other k–1 folds (consisting of m(1–1/k) examples) [32]. The cross-validation experiments are then repeated k times, with all of the k sub-samples used exactly once as the validation dataset, and the k experiment results from the folds are then averaged to produce a single classification rate. In this paper, 5-fold cross validation was used when comparing LMDP versus LDP using IKSVMs classifier and other three classifiers, i.e., kNN classifier, MLP classifier and Dissimilarity-based classifier. One set of feature vector corresponds to one fatigue expression image, and there are 120 sets of feature vectors extracted from the images of SEU fatigue facial expression dataset. The 120 sets of feature vectors of fatigue expression images are randomly divided into 5 disjoint subsets of equal size, and each subset consists of 24 sets of feature vectors. The cross-validation experiments were repeated 100 times by randomly splitting the fatigue expression dataset, and recorded the classification results in average.

In the first cross-validation experiment, the classification rates for fatigue expressions using LMDP versus LDP by IKSVMs classifier are displayed in the bar plots of Fig. 9(a) and box plots of Fig. 9(b), which shows again that the recognition accuracy of fatigue expressions in average is significantly improved using the proposed LMDP than using LDP. The experimental results in Fig. 9 also demonstrated that the performance of LMDP drops when the order reaches the third-order and four-order, and it reveals that the high-order LMDP can extract more detailed information than the first order. But the third-order and four-order LMDPs aren’t incapable of dealing with the further detailed information contained in a fatigue expression image.

In the second cross-validation experiment, the same set of training and testing are applied to the second-order LMDP and IKSVMs, compared with kNN classifier, MLP classifier and Dissimilarity-based classifier, and their classification performances are simultaneously recorded. The average classification accuracies of the 100 cross-validation experiments using the proposed LMDP and IKSVMS classifier, compared with kNN classifier, MLP classifier and Dissimilarity-based classifier, are displayed in the bar plots of Fig. 10(a) and box plots of Fig. 10(b). From Fig. 10, it is obvious IKSVMs classifier outperforms the other three classifiers, because it achieves the highest classification rates in the second cross-validation experiments.

In the cross-validation experiment, the confusion matrix that summarizes the detailed performance of the proposed second-order LMDP and IKSVMs classifier is shown in Table 2. The accuracies of three classes, (i.e., awake facial expression, etc.) are 89.80% , 99.57% , and 90.02% , and once again, it is clear that the class of awake expression has the most recognition accuracy of three classes in the cross-validation experiments.

5.3 Discussions

Most of the current techniques require that sensor devices must be attached to driver’s clothing or body, which is not a natural way to monitor a driver’s activities. It is unlikely that drivers would accept any tethered sensing solution, i.e. using wired sensors or wireless sensors attached to their bodies. The noninvasive technological approaches for detecting driver’s fatigue are only based on the local features of driver images, such as eye closure, eyelid movement, blink measurement, head pose and gaze direction. DD850 Driver Fatigue Monitor (DFM), designed by Attention Technology Inc., is a video-based drowsiness detection system for measuring slow eyelid closure, and the field of view is large enough to accommodate normal head movement. The disadvantages of DFM are that every driver has the individual eyes and it is hard to determine a universal drowsiness threshold. Driver State Monitor (DSM), developed by Delphi Inc., consists of the ForeWarn Drowsy Driver Alert system and the ForeWarn Driver Distraction Alert systems, and DSM analyses eye closures and head pose to infer fatigue or distraction level. InSight^TM, developed by SensoMotoric Instruments GmbH (SMI), is a noninvasive computer-vision based operator monitoring system that measures head position and orientation, gaze direction, eyelid opening, and pupil position and diameter. InSight^TM calculates PERCLOS to determine a driver’s state of alertness. Video-based Eye Tracking Systems ETS-PC II, developed by Applied Science Laboratories, and Eye gaze Analysis System, developed by LC Technologies Inc., are both the eye trackers-based systems utilizing the pupil reflection technique for measuring eye movements. Drowsy Driver Detection System, designed by Johns Hopkins University Applied Physics Laboratory, can monitor and quantitatively measure the speed, frequency, and duration of eyelid closure, rate of heartbeat and respiration, and pulse rate by analyzing the Doppler components in the reflected signal. RPI computer vision system for monitoring driver vigilance by Rensselaer Polytechnic University can simultaneously and unobtrusively monitor in real time several behaviors that typically characterize a driver’s level of alertness, and these visual cues include eyelid, gaze movements, pupil movement, head movement and facial expression. Drowsiness Detecting System based on Artificial Neural Network (ANN), developed by George Washington University (GWU), and the ANN observes the steering angle patterns and classifiers them into drowsy and non-drowsy driving intervals. GWU researchers trained and tested the ANN by conducting a driving simulator experiment. Different drivers have individual differences on eye blinking. Moreover, it is also affected by environment factors such as outside road lighting and oncoming headlights. Therefore, it is hard to determine the preset drowsiness threshold on eye blinking.

Image-based fatigue detection would be more applicable in a consumer car without the requirement of special markers or user intervention. A decisive step in developing image-based driver fatigue degree recognition system is to extract suitable features from the driver’s images and characterize differences between different driver fatigue expressions. When being fatigue, the drivers will repeat yawning, and have difficulties in keeping eyes open. The key features of fatigue expression images are locations and contours of mouth, eyes and eyebrows, which are localized in both location and direction. The images of fatigue expressions contain intrinsic geometrical structure (locations and contours of mouth, eyes and eyebrows), which are key features in visual information for fatigue expression recognition. In the paper, we presented a more reliable fatigue monitoring system based on comprehensive characterization of driver’s facial expressions, including eyes closure, position of eyelid, motion of cheek muscle and mouth movement. An efficient feature extraction approach, named as Local Multiresolution Derivative Pattern (LMDP), was proposed to describe the fatigue expressions of vehicle drivers. The experimental results of the holdout and cross-validation show that the recognition rates of driver’s fatigue expressions in average are above 92% .

The severe fatigue is the most dangerous situation since the driver may lose control of the vehicle and the chances of accidents could dramatically increase. Although our fatigue monitoring system obtains higher average recognition rates of driver’s fatigue expressions, the severe fatigue classification accuracy of 90% is lower than the class of moderate fatigue expressions. It is observed that the main difference between awake expressions of SEU image database is the openness of the eyes. So that the two types of driver’s expressions are likely to be misclassified to each other. In the future work, we will continue our investigations on feature extraction approaches of drivers’ fatigue expressions to enhance the eyes-around or eye-related features to further improve the performance of detecting the severe fatigue. The efficiency of our proposed system is also very important in the real situation, and how to improve the system efficiency of drivers’ fatigue recognition is worth our study in the future research.

6 Conclusions

Fatigue expressions recognition of vehicle drivers are investigated, and three contributions are presented in this paper. Firstly, in order to investigate the approach of recognizing the driver’s fatigue degree using comprehensive characterization of driver’s facial expressions, a fatigue expression dataset, named Southeast University (SEU) fatigue expressions dataset, is presented, and the SEU fatigue expression dataset consists of 40 subjects with 3 different sessions per subject and in total 120 sessions. Secondly, we proposed a novel local descriptor, named Local Multiresolution Derivative Pattern (LMDP), to describe fatigue expression images, and the experimental results show the more feasibility and effectiveness of high-order LMDP than high-order LDP for fatigue expression recognition. Thirdly, the confusion matrixes of the holdout and cross-validation experiments show that the class of moderate fatigue expressions has the most recognition accuracy of three, and the severe fatigue classification accuracy of 90% is lower than the class of moderate fatigue expressions.

Footnotes

Acknowledgments

This project is funded by National Natural Science Foundation of China (Project No. 51078087).

References

Croo

H.D.

, Bandmann

, Mackay

G.M.

, et al., The role of driver fatigue in commercial road transport crashes, Eur Transp Safety Council, Brussels, Belgium, Tech Rep, 2001.

The Royal Society for the Prevention of Accidents. Driver fatigue and road accidents: A literature review and position. Birmingham, UK, 2001

Bergasa

, Nuevo

, Sotelo

, et al., Real-time system for monitoring driver vigilance, IEEE Trans Intell Transp Syst 7(1) (2006), 63–77.

Suzuki

, Yamamoto

, et al., Measurement of driver’s consciousness by image processing–A method for presuming driver’s drowsiness by eye-blinks coping with individual differences, Proc IEEE Int Conf Syst, Man, Cybern 4 (2006), 2891–2896.

Eskandarian

, Sayed

, Delaigue

, et al., Advanced driver fatigue research, U.S. Dept Transp, Fed Motor Carrier Safety Admin, Washington, DC, Tech Rep, Rep 2007.

Orazio

T.D.

, Leo

and Guaragnella

, A visual approach for driver inattention detection, Pattern Recognit 40(8) (2007), 2341–2355.

Barr

, Howarath

, Popkin

, et al., A review and evaluation of emerging driver fatigue detection, measures and technologies, A Report of U.S. Department of Transportation, Washington, DC, 2009.

Friedrichs

and Yang

, Camera-based drowsiness reference for driver state classification under real driving conditions, Proc IEEE Intell Veh Symp (2010), 101–106.

Fan

, Sun

and Yin

, Gabor-based dynamic representation for human fatigue monitoring in facial image sequences, Pattern Recognit Lett 31(3) (2010), 234–243.

10.

Dong

Y.C.

, Hu

Z.C.

, Uchimura

, et al., Driver inattention monitoring system for intelligent vehicles: A review, IEEE Transactions on Intellignet Transportation Systems 12(2) (2011), 596–614.

11.

Lin

C.T.

, Wu

R.C.

, Liang

S.F.

, et al., EEG-based drowsiness estimation for safety driving using independent component analysis, IEEE Trans Circuits Syst I, Reg Papers 52(12) (2005), 2726–2738.

12.

Lin

C.T.

, Chen

Y.C.

, Huang

T.Y.

, et al., Development of wireless brain computer interface with embedded multitask scheduling and its application on real-time driver’s drowsiness detection and warning, IEEE Trans Biomed Eng 55(5) (2008), 1582–1591.

13.

Damousis

I.G.

and Tzovaras

, Fuzzy fusion of eyelid activity indicators for hypovigilance-related accident prediction, IEEE Trans Intell Transp Syst 9(3) (2008), 491–500.

14.

Yeo

M.V.M.

, Li

X.P.

and Shen

, Can SVM be used for automatic EEG detection of drowsiness during car driving?, Safety Sci 47(1) (2009), 115–124.

15.

Jap

B.T.

, Lal

, Fischer

, et al., Using EEG spectral components to assess algorithms for detecting fatigue, Expert Syst Appl 36(2) (2009), 2352–2359.

16.

and Zheng

, Driver drowsiness detection with eyelid-related parameters by support vector machine, Expert Syst Appl 36(4) (2009), 7651–7658.

17.

Yang

, Lin

and Bhattacharya

, A driver fatigue recognition model based on information fusion and dynamic Bayesian network, Inf Sci 180(10) (2010), 1942–1954.

18.

Liu

, Zhang

and Zheng

, EEG-based estimation of mental fatigue by using KPCA–HMM and complexity parameters, Biomed Signal Process Control 5(2) (2010), 124–130.

19.

Sibsambhu

, Mayank

and Aurobinda

, EEG signal analysis for the assessment and quantification of driver’s fatigue, Transportation Research Part F Traffic Psychology and Behaviour 13(5) (2010), 297–306.

20.

Khushaba

R.N.

, Kodagoda

, Sara

, et al., Driver drowsiness classification using fuzzy wavelet-packet-based feature extraction algorithm, IEEE Transactions on Biomedical Engineering 58(1) (2011), 121–131.

21.

Zhang

, Gao

, Zhao

, et al., Local derivative pattern versus local binary pattern: Face recognition with high-order local pattern descriptor, IEEE Transactions on Image Processing 19(2) (2010), 533–544.

22.

Paul

and Michael

J.J.

, Robust real-time face detection, International Journal of Computer Vision 57(2) (2004), 137–154.

23.

Minh

N.D.

and Martin

, Framing pyramids, IEEE Transactions on Signal Processing 51(9) (2003), 2329–2342.

24.

Park

S.I.

, smith

M.J.T.

and Mersereau

R.M.

, Improved structures of maximally decimated directional filter banks for spatial image analysis, IEEE Transactions on Signal Processing 13(9) (2004), 1424–1434.

25.

Cortes

and Vapnik

, Support-vetor networks, Machine Learning 20(3) (1995), 273–297.

26.

Platt

, Fast training of SVMs using sequential minimal optimization, Advances in Kernel Methods Support Vector Machine, MIT Press,Cambridge, 1999. pp. 185–208.

27.

Milgram

, Cheriet

and Sabourin

, One Against One” or “One Against All: Which One is Better for Handwriting Recognition with SVMs?, International workshop on Frontiers in handwriting Recognition, Montreal, Canada, 2006.

28.

Maji

, Berg

A.C.

, Malik

, Classification using Intersection Kernel Support Vector Machines is efficient, IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, 2008, pp. 1–8.

29.

Bremner

, Demaine

, Erickson

, et al., Output-sensitive algorithms for computing nearest-neighbor decision boundaries, Discrete and Computational Geometry 33(4) (2005), 593–604.

30.

Haykin

, Neural Networks: A comprehensive foundation (2ed.). Prentice Hall, 1998.

31.

Pekalska

, Paclik

and Duin

R.P.W.

, A generalized kernel approach to dissimilarity-based classification, Journal of Machine Learning Research 2 (2001), 175–211.

32.

Zhang

B.L.

and Zhang

Y.C.

, Classification of cerebral palsy gait by kernel fisher discriminant analysis, International Journal of Hybrid Intelligent Systems 5(4) (2008), 209–218.