Principle for the Validation of a Driving Support using a Computer Vision-Based Driver Modelization on a Simulator

Abstract

This paper presents a new structure for a driving support designed to compensate for the problems caused by the behaviour of the driver without causing a feeling of unease. This assistance is based on a shared control between the human and an automatic support that computes and applies an assisting torque on the steering wheel. This torque is computed from a representation of the hazards encountered on the road by virtual potentials. However, the equilibrium between the relative influences of the human and the support on the steering wheel are difficult to find and depend upon the situation. This is why this driving support includes a modelization of the driver based on an analysis of several face features using a computer vision algorithm. The goal is to determine whether the driver is drowsy or whether he is paying attention to some specific points in order to adapt the strength of the support. The accuracy of the measurements made on the face features is estimated, and the interest of the proposal as well as the concepts raised by such assistance are studied through simulations.

Keywords

Driving Assistance Virtual Force Field Facial Features Analysis Driving Simulator Steer-by-Wire

1. Introduction

After the first appearance of what can be described as an early prototype of a car in 1672 [1], the number of registered vehicles worldwide in 2010 was around 1,500,000,000 [2]. In 2012, production reached 84,000,000 vehicles [3].

Despite the crucial role of cars in everyday life, traffic accidents are among the main causes of avoidable deaths among 15–29-year-olds worldwide, and the number of deaths per country can be very high. If the number of road fatalities per year seems to decrease in some parts of the world where strict laws are enforced (in the EU, 54,900 people died on the road in 2001 and 26,000 in 2013 [4]), the global number of road traffic deaths is increasing. 1.18 million people died in 2002 and 1.24 million in 2010 [5]. The principal reason for these casualties on the road seems to be the behaviour of drivers. As such, it is of interest to consider a proper support for drivers as it could contribute to the reduction of traffic incidents.

The large number of parameters involved in driving tasks and the difficulty for drivers in staying focused over long periods has led to the progressive introduction of automated tools aiming to support users without disturbing them. This is why systems such as ABS, ESP, power steering, cruise control and even sleepiness sensors have been introduced in commercialized vehicles.

The purpose of realizing a driving support is to create an assistant able to correctly estimate the danger of a situation and react efficiently. However, due to the impossibility of creating a system that would never fail to estimate dangers or react correctly, it is preferable to retain the influence of the driver through the shared control of the vehicle, as humans seem to have better reactions in unpredictable situations.

The architecture of a driving assistant consists of combining the efficiency of an automated control and the ability to react to complex situations of humans. However, it is a complicated matter to determine accurately the expected level of assistance to be offered. If the effect of the assistant is too small, then the system loses its efficiency. Qn the other hand, if the effect of the assistant is too strong, the freedom of the driver is restricted and the situation corresponds to an autonomous control. To find an equilibrium between those influences, the system described in this paper proposes to adapt the strength of the supporting function on the state of the driver. Indeed, the condition of the user is valuable in determining the danger of a situation. If the driver is not focused on his task, or if he is drowsy, the hazard of a situation is increased. According to the National Sleep Foundation, one out of every six deadly traffic accidents, and one out of every eight crashes requiring hospitalization, are due to drowsy driving [6]. This is why a device able to detect this kind of state is particularly interesting in terms of efficiency and safety improvement. However, when the driver is paying attention to what he is doing, a too intrusive action on the part of the assistant could lead to bad reactions by the user or to a feeling of discomfort.

The method proposed here also aims to conform the level of assistance to the expectations of the user, based on his estimated condition, helping him when it is required and leaving him free when the situation is safe. It is thus possible to achieve better cooperation between the support and the driver, as well as to increase the acceptance of the system by the user.

In this paper, the state of the driver is estimated with a driver model using the analysis of face features. It is then possible to realize a temporal switch between freedom and efficiency in the functioning of the estimated state of the driver. This assistant must be able to prevent hazardous situations by determining a safe trajectory as well as to wake the driver if he begins to fall asleep. It is also crucial to reduce the influence of the support when it is not needed. Moreover, the use of a camera to determine the state of the driver is the less invasive possible measurement method. If bio-sensors can provide more accurate information, their application to driving situations remains problematic because of the discomfort for the user.

As an initial approach, a driving assistant can be considered as an evolution of obstacle avoidance applied to robots and manipulators. A particular approach to collision avoidance consists of representing hazards by different potential fields. This technique was applied to the control of manipulators and mobile robots in works similar to that presented by R. Volpe and P. Kholsa [7] and O. Kathib [8].

This technique was then applied on the particular field of driving support, and especially to quite simple cases, such as highway driving which is considered in this paper. As the death ratio of car incidents on highways is superior to the death ratio on other types of road, this limitation remains interesting. This principle was applied to lanekeeping assistance by Rossetter [9]. An extension of this work was studied by Wolf and Bur dick, who added potential fields representing the other cars on the road [10]. The efficiency of this approach was tested in relation to active assistance on the steering wheel by Carlier and Murakami [11].

Other works were conducted on active driving assistance, such as those presented by A. Kullack, I. Ehrenpfordt, K. Lemmer and F. Eggert, who proposed a lane-departure prevention system based on behavioural control [12].

The use of computer vision to model the human's state is an active field. This method can be used for almost every one, including persons with reduced mobility, and may be applied to many types of systems. The use of this kind of technology to realize human computer interfacing has been demonstrated by A. Krolak and P. Strumillo [13]. In the automobile field, this technique is used to detect the driver's inattention, as in the work of P. Jimenez, L. M. Bergasa, J. Nuevo, N. Hernandez and I. G. Daza [14].

The system described in this paper is a combination of several technologies and looks to create a new kind of adaptive and active driving support. Compared to other systems, the aim is to provide a support able to actively prevent incidents and to represent every type of danger. The intent is also to maximize acceptance and efficiency by adapting its effect, and by reacting in the function of the estimated state of the driver. The purpose is to provide to the user the exact amount of support required depending upon his situation. The main innovative aspect of the proposed method is its adaptive behaviour. This allows us to solve the equilibrium problem between efficiency and freedom raised by the nature of the active assistant. With this solution, there is no need to choose between the impact and efficiency of the driving support and the freedom of the driver. Indeed, a temporal switch between these two cases can be done depending upon the state of the driver. This assistant combines a potential field approach to represent the danger with a face-features analysis of the user. This analysis comprises the combination of several methods using one another's results in order to improve global efficiency.

The paper is organized as follows. First, the models used for the electric vehicle and driving simulator are described. Next, the driver modelling based on a computer vision algorithm is studied and the accuracy of several face-features analyses are estimated. Finally, the proposed support is evaluated using a driving simulator.

2. Electric Vehicle Modelling and Driving Simulator

This section briefly describes the model of the vehicle, the potential fields used to represent the hazards, and the driving simulator used to test the efficiency of the proposed driving assistant.

2.1 Vehicle Model

The model used to represent the car in this paper is described in Figure 1 and in Figure 2. This model corresponds to a highway driving situation [9], namely the scenario used in the simulation. As such, considering the braking, steering and throttle as electronically controllable, it is possible to propose the following equations:

Figure 1.

Dynamics model of the vehicle

Figure 2.

Kinematics of the wheels

m ({\dot{U}}_{x} - γ U_{y}) = F_{x r} + F_{x f} \cos δ - F_{y f} \sin δ

(1)

m ({\dot{U}}_{y} - γ U_{x}) = F_{y r} + F_{x f} \sin δ + F_{y f} \cos δ

(2)

I_{z} \dot{γ} = l_{1} F_{x f} \sin δ + l_{1} F_{y f} \cos δ - l_{2} F_{y r} + \frac{L}{2} (Δ F_{x r} + Δ F_{x f} \cos δ)

(3)

where:

F_{x f} = F_{x f r} + F_{x f l}

(4)

F_{x r} = F_{x r r} + F_{x r l}

(5)

Δ F_{x f} = F_{x f r} - F_{x f l}

(6)

Δ F_{x r} = F_{x r r} - F_{x r l}

(7)

According to Figure 2, the slip angles a are defined as:

α_{l r} = \arctan (\frac{U_{y} - γ l_{2}}{U_{x} - γ \frac{L}{2}})

(8)

α_{r r} = \arctan (\frac{U_{y} - γ l_{2}}{U_{x} + γ \frac{L}{2}})

(9)

If γ is small and U_x is large (which corresponds to a highway situation), it is possible to consider that $U_{x} + γ \frac{L}{2} \approx U_{x} - γ \frac{L}{2} \approx U_{x}$ and that $α_{l r} \approx α_{r r} \approx α_{r}$ . Then, with C_r the sum of the cornering stiffness of the rear wheels:

F_{y r} \approx - C_{r} α_{r} = - C_{r} \arctan (\frac{U_{y} - γ l_{2}}{U_{x}})

(10)

Similarly, with $C_{f}$ , the sum of the cornering stiffness of the front wheels:

F_{y f} \approx - C_{f} α_{f} = - C_{f} (\arctan (\frac{U_{y} + γ l_{1}}{U_{x}}) - δ)

(11)

F_{y f} \approx - C_{f} \arctan (\frac{U_{y} + γ l_{1}}{U_{x}}) + C_{f} δ = {\tilde{F}}_{y f} + F_{δ}

(12)

Finally, it is possible to describe the equation of motion as follows:

M \ddot{q} = d (\dot{q}) + g (u)

(13)

with the mass matrix,

M = [\begin{matrix} m & 0 & 0 \\ 0 & m & 0 \\ 0 & 0 & I_{z} \end{matrix}]

(14)

the velocity vector,

\dot{q} = {[U_{x}, U_{y}, γ]}^{T}

(15)

the uncontrolled or drift terms,

d (\dot{q}) = [\begin{matrix} - {\tilde{F}}_{y f} \sin δ + m γ U_{y} \\ F_{y r} + {\tilde{F}}_{y f} \cos δ - m γ U_{x} \\ - l_{2} F_{y r} + l_{1} {\tilde{F}}_{y f} \cos δ \end{matrix}]

(16)

the controlled components,

g (u) = [\begin{matrix} F_{x r} + F_{x f} \cos δ - F_{δ} \sin δ \\ F_{x f} \sin δ + f_{δ} \cos δ \\ l_{1} F_{x f} \sin δ + l_{1} F_{δ} \cos δ + \frac{L}{2} (Δ F_{x r} + Δ F_{x f} \cos δ) \end{matrix}]

(17)

and the control vector,

u = {[δ, F_{x f r}, F_{x f l}, F_{x r r}, F_{x r l}]}^{T}

(18)

Another important factor to take into account when simulating the behaviour of a vehicle is the tire model. The chosen model is the Dugoff model. This empirical model uses the slip angle, the tire slip, the normal force and two cornering constants representing all of the tire parameters, to define the lateral and longitudinal tire forces as stated in the following equations and in Figure 3.

Figure 3.

Distribution of velocities and tire forces

Two main constants are used in this model, C_x and C^y, namely the longitudinal and lateral stiffness of the tires. The slip angle α and the tire slip s_w are shown in Figure 3.

α = \arctan (\frac{V_{y}}{V_{x}})

(19)

During acceleration, the tire slip can be defined as:

s_{w} = \frac{R ω - V_{x}}{R ω}

(20)

During braking phases, the tire slip can be defined as:

s_{w} = \frac{R ω - V_{x}}{V_{x}}

(21)

where R is the tire radius, ω is the angular velocity of the tire, and V_x and V^y are the planar velocities of the tire centre.

As such, the tire forces can be described using the normal force applied, F_n, and the friction coefficient, μ, according to:

F_{x} = - \frac{f (λ)}{1 - s_{w}} C_{x} s_{w}

(22)

F_{y} = - \frac{f (λ)}{1 - s_{w}} C_{y} \tan (α)

(23)

where the boundary value λ is defined by:

λ = \frac{μ F_{n} (1 - s_{w})}{2 \sqrt{(F_{x}^{2} + F_{y}^{2})}}

(24)

and:

f (λ) = {\begin{matrix} (2 - λ) λ & when λ < 1 \\ 1 & when λ \geq 1 \end{matrix}

(25)

Finally, supposing that the tire slip and the slip angle are small and – in the case of acceleration – inferior to 0.5 g, the previous equations can be reduced as:

F_{x} \approx - C_{x} s_{w}

(26)

F_{y} \approx - C_{y} α

(27)

Another specificity of the used vehicle is the steer-by-wire system. This system suppresses the mechanical connection between the steering wheel and the steering shaft. The connection is made electronically by a control system that includes the driving assistant and two motors, one on the steering wheel and one on the steering shaft. This architecture has the advantage of simplifying the integration of the driving assistant. Indeed, the torque created by the support in order to reach a safer trajectory can be directly applied on the steering wheel. The control of the vehicle is then shared between the driver and the assistant as if two drivers were handling the steering wheel at the same time. The second advantage of this technology is the possibility of disconnecting the steering wheel and the steering shaft, suppressing the control of the driver over the vehicle. This can be useful in specific and limited-in-time situations. For instance, if the driver is detected as being sleepy, the driving support sends a haptic signal in the form of an oscillation to the user through the steering wheel. In this case, it is important to avoid short variations in the direction of the car by suppressing the connection between the upper and the lower part of the steer-by-wire.

2.2 Potential Fields

In this paper, the assisting torque to apply on the steering wheel is obtained by short-range path planning. The used method is based on virtual potential fields representing the different hazards that can be encountered on the road. This approach was created to simplify the trajectory planning task and was used in several fields, such as the control of a manipulator [7] [8], mobile robots [8] [15] and even driving assistance [9] [10].

The principle of virtual force fields is to assign to each point in the space in which the controlled object is evolving, with a measurement representing the level of hazard of some areas compared to their surroundings. As the potential fields are additive, it is possible to create a hazard map expressing all of the detected dangers. A local derivation on the location of the car on the hazard map allows for a quick computation of the short-range path planning. The main problem of this approach is caused by the possible appearance of local minima. However, as the proposed system is a driving support, this problem can be tackled by the driver, who can choose to leave a local minima to reach a better location.

In this work, two main potentials were used. The first one is in charge of representing the road. It is composed of Gaussian-like functions modelling the borders of the road and preventing leaving the road, and of Gaussian-like functions representing the borders of the lanes and allowing lane-changing in normal situations but preventing undesired lane-changing. The following equations describe this potential. Y_t is the coordinate of a point representing the border of the road, in the direction orthogonal to the road. This point is located a little outside of the road to avoid a derivation equal to 0 on the road borders. Y^j is the positions of the lines on the road, in the direction orthogonal to the road.

E_{t o t a l} = E_{l a n e s} + E_{r o a d}

(28)

E_{r o a d} = A_{r o a d} \sum_{i = 1}^{2} \exp (- \frac{{(Y - Y_{i})}^{2}}{2 σ_{r o a d}})

(29)

E_{l a n e} = A_{l a n e} \sum_{j = 1}^{2} \exp (- \frac{{(Y - Y_{j})}^{2}}{2 σ_{l a n e}})

(30)

where A_road and A_lane can evolve during the running time of the driving assistant as a function of the results of the driver modelling.

The second potential is used to represent the vehicles. Its goal is to avoid collisions but also, depending upon the speed difference between the controlled car and the obstacle car, to maintain a safe distance or to encourage a lane change. The shape of this potential involves a rectangle, representing the vehicle, and a triangle on the rear part to help the lane-changing. The potential used is the superquadric avoidance potential function proposed by Yukawa [7] [10]. It is defined by:

E_{c a r, n} (K) = A_{c a r} \frac{e^{- α K}}{K}

(31)

where K represents the pseudo-distance between the controlled vehicle and the car number n to avoid, A_car is a scaling factor and α is the slope of the potential field. However, K has to be dependent upon the considered direction. Qn the sides and the front part, it corresponds to the distance to the nearest point of the obstacle vehicle, and on the rear part this distance is modified as a function of the speed difference between the two vehicles.

2.3 Driving simulator

The principle behind and the efficiency of the proposed method are tested through several simulations performed on a driving simulator composed of a steer-by-wire system and a pedal system connected to a computer. This estimates the dynamic response of the vehicle based on the model of the car previously described, and displays the results using OpenGL rendering. Figure 5 describes the block diagram of the simulator and shows the dual interaction on the steering wheel of the driver and driving assistant. Table 1 gives the definitions of the different variables. The disturbance observer (DOB) and reaction torque observer (RTOB) were described by K. Ohnishi [16] and T. Murakami, F. Yu and K. Ohnishi[17].

Figure 4.

Example of a road section showing the resulting potential

Figure 5.

Block diagram of the driving simulator

Table 1.

Simulator variables

τ_d	Driver torque	θ_d	Driver input angle
$\hat{τ_{d}}$	Estimated driver torque	θ_da	Driving assistant angle
τ_da	Driving assistant torque	K_da	Driving assistant gain
τ_uref	Nominal upper motor torque	V_car	Car velocity
τ_lref	Nominal lower motor torque	V_driver	Command velocity
τ_tref	Nominal traction motor torque	V_da	Driving assistant velocity
$\hat{τ_{s a}}$	Estimated self-aligning torque	γ	Car yaw angle
θ_u	Upper motor angle	r	Front rack gear ratio
θ _l	Lower motor angle	c_f	Tire cornering stiffness

3. Driver Modelling

3.1 Architecture of the Computer Vision Algorithm

This section describes the methods that were used in the driver modelling. A representation of this is given in Figure 6. This program is composed of two main parts. First, an initialization step is conducted – this corresponds to the column on the left. The data used in the next steps of the process are acquired at this moment. A support vector regression (SVR) is initialized (its working will be described later), and some templates of the closed eyes of the driver are saved to be used later in a comparison. The two other columns describe the main loop of the program. It comprises a face detection algorithm and a face tracking algorithm. For face detection, it tries to find a face on the image returned by the camera. If a face is detected, then it is stored and sent to the face tracking algorithm. Here, several features of the face are studied. If the face is considered as lost – when no face feature is detected – then another face detection loop is launched. To reduce the influence of lighting variations, every frame goes trough a light filter in which the luminance of the frame is equilibrated.

Figure 6.

Architecture of the computer vision algorithm

3.2 Step-by-step Description

Some face features are detected very quickly and efficiently using the Viola-Jones method [18] based on cascaded Haar classifiers. These classifiers are composed of specifically-chosen objects called 'Haar features'. Those features are weak classifiers applied to specific areas of the frame. Each weak classifier returns a value that is compared to a threshold. If an area is able to pass successfully through all of the thresholds of a cascaded Haar classifier, then the feature corresponding to this classifier is considered as detected. This method is fast and efficient, but it is dependent upon the orientation of the object. This is thus a good way to detect a face in order to use it as an initialization of the tracking phase. This method is also used to detect the mouth, the nose and the eyes. In order to minimize the computation time, the sizes of the search areas are reduced at their minima and their locations are dependent upon the features to be detected. For instance, in the case of the mouth, the search area is located in the lower part of the face. These face features are detected to confirm that the tracked object is a face, but they are also to be used in the next step of the computer vision algorithm.

In this work, the face is tracked using the Camshift algorithm. Conversely to the detection step of the face, which can find a face in an image but is not able to say whether two consecutive detections are related to the same object, the purpose of the Camshift method is to find the same object in every frame. This algorithm tracks an object by constructing a colour histogram and then by computing the probability map of the location of the object to follow in the next frames. This algorithm is able to detect the location, the size and the roll angle of an object. Its main backlash is that false detections can appear if the colour of another object is really close to the colour of the object to be tracked. One of the uses of the detection of face features made previously is to then confirm whether the tracked object is a face or not. If no face is detected in several consecutive frames, the face is considered to be lost and another face detection step is launched. The results of the Camshift algorithm are used to compute a cropped version of the frame that focuses only on the face and which suppresses the roll angle of the face to simplify further detection of face features.

The blink detection of this computer vision algorithm is based on the utilization of several templates, namely small images of the right and left eyes obtained with the Viola-Jones Method. The closed eyes are acquired once, during the initialization step. Based on the location of the eyes in the previous frame, two areas are delimited. Their sizes must remain small in order to shorten the computation delay. In those areas, two template-matchings are operated with one opened-eye template and one closed-eye template. The results of the template-matchings are used to determine the new positions of the eyes. The evolution of the matching results indicates the evolution of the states of the eyes. By this means, an eye closure can be detected as a decrease in the correlations of the opened-eye templates, followed by an increase in the correlations of the closed-eye templates. To reduce the possibility of mistakes, the states of the two eyes are considered simultaneously. Moreover, to avoid erroneous results, the opened-eyed templates are suppressed every time any suspicion of mistake is detected. Finally, if a blink is detected, then its starting time and duration are stored in a list of blinks.

The yawning detection, by studying the geometry of the mouth, can decide whether the mouth is opened or closed. To detect the mouth, colourimetric filtering is used [19]. Again, the work area is reduced as much as possible by focusing upon the last known position of the mouth and its surroundings. This method can help to reduce some interference, since a colourimetric filter can detect other parts of the face as a function of the colour. First, a colour base translation of the research area is operated – the goal is to obtain a description of the image in a YCrCb base. Next, the colour filtering is applied. Several filters are then applied to this image to obtain a blob representing the lips. The dimensions of the mouth are extracted from contour detection operating on that blob. Finally, by comparing its width with its height, it is possible to determine whether the mouth is closed or opened. A pseudo-yawn comprises a mouth opening followed by a mouth closing. Each detected yawn is stored in a list in the form of a start time and a duration.

To estimate the direction of the gaze of the driver, the orientation of his head is studied, as in a driving situation, and the extra accuracy offered when studying the pupil orientation is not required. The roll angle estimation is done during the Camshift process. The pitch angle detection is based on the use of a SVR. SVR is a version of the support vector machine (SVM), a supervised learning model used for the classification into several categories of events characterized by different values. The difference is that the SVR is used to realize the non-linear regression of an estimated scalar function. From the previous step of the algorithm, the ratio of the vertical location of the mouth to the height of the face image, and the ratio of the vertical location of the eyes onto the same height, are sent to the SVR, which returns the pitch angle of the head. During the initialization step, the user is asked to look at several points on the screen corresponding to predetermined pitch angles. Based on an elliptical model of the head [20] using the locations of the borders of the head and the location of the centre of the face, it is possible to estimate the yaw angle of the head. These borders are detected through filters enhancing horizontal variations. The horizontal position of the centre of the face is given by the location of the mouth, obtained during the yawning detection step.

Finally, after the estimation of the orientation of the head, the projection of the gaze on the plan of the driving simulator screen is computed. In this calculation, the position of the head is taken into account. If the distance from the screen plan does not vary while driving – because the user remains still in his seat – the orthogonal projection of the centre of the head on the screen plan has an effect on the gaze projection to be estimated.

The presented method requires appropriate lighting conditions to be maintained. Indeed, the camera that is used and the colourimetric method for detecting yawning are not efficient in dark conditions. Of course, this is a restriction of the field of applications of the described driving assistant. However, loss of focus and drowsiness do not just appear at night – they are often encountered during the daytime on well-known routes, especially while commuting. In daylight condition, it is possible to adjust the parameters of the different methods used in the algorithm to increase efficiency and to compensate for the variations in lighting. In order to be able to adapt the described computer vision algorithm to night conditions, some modifications could be implemented. First, the detection has to be improved and a combination of an infrared emitter directed towards the head of the driver and an infrared camera could be efficient. The result would be a one-channel image on which most of the previously described methods could be applied (except for the colourimetric detection of the mouth). Another application of Haar-cascaded classifiers that could be combined with a learning method should be able to replace this colourimetric detection.

3.3 F ace features and Driver Modelling

This section focuses on the algorithm responsible for the driver's state estimation based on the previously studied face features. By studying the frequency of the mouth opening and its mean duration, it is possible to describe three situations. The algorithm maintains information about every instance of the mouth opening for a duration of five minutes. Yawns are mainly detected as long mouth-opening periods. As such, if the computed mean duration on the analysis period is superior to two seconds, the driver is considered as being sleepy. Moreover, if the frequency of the mouth-opening events is more than one opening every two seconds, then it is considered that the driver is speaking. Even if this situation is not considered as being dangerous, the attention of the driver is not focused. In other cases, the driver's status is normal.

Concerning blinking, the important factors are the blinking frequency and the blinking mean duration. These data are computed over a period of two minutes. In a normal state, the blinking frequency of an adult is between 15 and 30 blinks per minute [21]. If the frequency is lower, it might indicate that the driver is focused on something. Conversely, if the frequency is higher, this indicates stress or distraction. However, the main indicator of a driver's drowsiness is the duration of the blinks – if this duration is greater than 0.5 seconds, then the driver is considered as being sleepy.

The projection of the gaze is used to determine the direction in which the driver is looking. It also provides information about the state of the driver and his behaviour. Indeed, if the gaze of the driver remains steady for a long period, this can indicate that the driver is falling asleep, or at least that the driver is not focused on driving, since in a driving situation the driver must constantly check, visually, a lot of information distributed across the environment. On the other hand, if the gaze projection remains for more than 500 ms on a rear mirror, then it can be considered that the driver has checked it. In order to be able to determine the evolution of the gaze projection, a list of projection coordinates is stored.

Figure 7 shows the architecture of the driver modelling program and the transitions between the states. As the most hazardous status of the driver is a drowsy state, the detection of this state and the effects of that detection have priority over the other detected states. In fact, there are two mains states – drowsy and awake. If the driver is considered as being sleepy, no other condition is tested.

Figure 7.

Architecture of the driver modelling algorithm

4. Estimation Accuracy of Face-feature Detection

A crucial aspect of the efficiency of the proposed method is the accuracy of the computer vision algorithm. Indeed, without decent precision it is impossible to get a relevant state estimation of the driver. To measure its accuracy, several tests were conducted. In order to validate the final efficiency of the presented algorithm, the accuracy of the data used to determine the state of the driver is crucial. This is why the methods that are evaluated are those returning these data. This means that the three Euler angles representing the orientation of the head, the blink detection and the yawn detection are tested.

To estimate the efficiency of the roll angle measurement, the values returned by the Camshift algorithm were compared to the roll angles manually measured on the images on which the Camshift algorithm was applied. 100 such tests were conducted to estimate the mean error. The results given by the estimation indicate that the mean error is equal to 1.38 degrees. This precision is satisfying in the case of gaze projection. As this angle is only used in the final step of the gaze projection – as a rotation of this projection – an error of this order will not provoke significant mistakes. A more detailed repartition of the measured estimation errors is given in Figure 8. This Figure demonstrates that the great majority of the errors are very small, and that only a few measures indicate important differences. The median error is 0.47 degrees. These facts combined to the estimated mean error, show that the results of the roll angle estimation are good. The results returned by the Camshift must be processed in order to obtain this accuracy. An ellipse surrounding the tracked object is returned by this algorithm. Next, depending upon the evolution of the geometry of the followed object, the longest axis can become the shortest, leading to a sudden variation of 90 degrees in the roll angle. These variations need to be suppressed. Low-pass filtering is also applied in order to compensate for the short-duration instabilities. The major origins of the errors are the deformation of the face – especially around the neck – when the roll angle increases, and the fact that the lighting might not be homogeneous. To improve the results, these problems must be tackled by restraining the detection to the face and by excluding the other detected elements that share the same colours as the face, as is the case for the neck.

Figure 8.

Repartition of the roll angle measurement errors

Since the pitch estimation is done using SVR, the manner of estimating the accuracy is quite different from that which is done for the other estimations. The main factor for the accuracy is the initialization step. This is realized by asking the user to look at several points on the screen. The corresponding pitch angles are computed and used to initialize the SVR. The initialization points are randomly chosen and are defined in order to be placed across the entire screen. Next, in order to avoid later problems and to validate the initialization, an accuracy estimation is realized. For this purpose, the user is asked to look at some random points, and the values returned by the SVR are compared to the computed values of the pitch angle. The estimation mean error is then measured. The mean error returned in the test step is compared to a threshold. If the mean error is small enough, the initialization is validated. Otherwise, another initialization is required. The usual values of the mean errors are around two degrees. This validation is required because the influence of the pitch angle on the accuracy of the gaze projection is important. It is possible to modify the accuracy of the process by changing the number of points used in the initialization, but if the initialization duration is too long it can become annoying for the user. The assistant has to refrain from constraining the driver as much as possible in order to be completely accepted by the user, and by then be more efficient. Moreover, as this regression is dependant upon the geometry of the face of the user, if the experimental conditions remain the same between two simulations (for instance, if the location of the camera compared to the user remains constant), it is possible to use the data previously acquired in reducing the constraints on the driver.

To perform the evaluation of the efficiency of the yaw angle estimation, the contributions of several testers were monitored in order to study the effect on the detection of different facial geometries. Each tester had to look in several directions corresponding to predetermined yaw angles. The error between the nominal value and the value returned by the algorithm was measured. Nine series of tests were conducted. Figure 9 shows the results for all the test series and Figure 10 shows the mean estimation errors for the different angles. The results show that the estimation error is small when the yaw angle is small, but that it quickly increases when the yaw angle becomes bigger. Table 2 gives the different mean errors obtained in this experiment. If the accuracy is sufficient when the yaw angles are small, the efficiency will not be good for the largest angles. This problem is due to the simplicity of the hardware used. Indeed, since only one camera is used and because the face cannot be represented as a circle, when the yaw angle is too large, the borders of the face can be hidden. The algorithm is then unable to detect the correct borders of the face, and it is impossible to correctly estimate the yaw angle. This problem can be tackled by using two cameras. This system would allow the detection of borders with larger angles by selecting the information given by the camera such that the orientation will be closest to the orientation of the head of the driver. Moreover, as seen in Figure 9, the results are less accurate for some testers than for others. This can be explained by the elliptical model used for the angle estimation. This model uses parameters to represent the geometry of the analysed face. This is why the elliptical model has to be adapted as a function of the driver. This was not the case in this test.

Figure 9.

Estimation errors for the different users

Figure 10.

Mean estimation error as a function of the nominal angle

Table 2.

Summary of the results

Face feature	Evaluation of efficiency
Head's roll angle.	Mean estimation error: 1.38 degrees.
Head's yaw angle.	Mean estimation error: 3.83 degrees.
	For small angles: 1.18 degrees.
	For angles >30 degrees: 7.82 degrees.
Head's pitch angle.	Mean est. error: around 2 degrees.
Blinks detection.	Missed detection rate: 0.8 percent.
	False detection rate: 0.1 percent.
	Minimum duration detected: 61 ms.
Mouth opening.	Missed detection rate: 2 percent.
detection	False detection rate: 0.6 percent.

The efficiency of the blinking detection is measured by submitting to the program several series of 10 blinks. The number and the lengths of the blinks are saved. The algorithm is tested on several drivers to reach a total of 100 series of 10 blinks. The results of this estimation are given in Table 2. The rates of missed detections and false detections are very low, demonstrating the efficiency of the proposed method. Moreover, the minimum duration of the detected blinks is short enough considering that the drowsiness of the driver – the most dangerous situation – is characterized by long durations of blinks. This minimum duration is dependant upon the hardware used. Indeed, the camera is a simple USB camera with a sampling rate of 30 Hz. To detect a blink, the eyes must be closed on at least one frame. As such, in an ideal case, two frames are required to detect a blink if both eyes are closed on the same frame and opened on the next frame. In this theoretical case, the minimum duration that can de detected is 33 ms. Moreover, if the computation of a frame takes too long, the actual sampling time of the camera is lowered. Even by reducing the computation time by as much as possible, this factor is still important. To be able to detect shorter blinks, it would be necessary to use a high-frequency camera and a computer able to provide better performances.

The test procedure for the mouth opening is very similar to that for blink detection. One-hundred series of 10 mouth-openings are submitted to the computer vision algorithm. Several testers' contributions were used for this test. The number of mouth-openings and their durations are saved. The efficiency of this detection is lower than that for blink detection. However, the results shown in Table 2 remain good. The biggest problem for this method is that it is based on a colourimetric filter. As such, it is dependant upon the lighting.

5. Algorithm Evaluation by a Driving Simulator

5.1 Driving Simulator

Figure 11 shows the architecture of the simulation system used to represent in 3D the behaviour of the car in order to test the relevance and efficiency of the proposed support. The complete system comprises two computers. Computer 1 handles the computer vision and the state estimation of the driver. The parameters to be applied on the assistant as a function of the condition of the user are then sent to Computer 2 using a user datagram protocol. Finally, Computer 2 realizes the dynamical simulation and displays the results using OpenGL. It also computes the driving assistant's torque to be applied on the steering wheel and estimates the torque applied by the driver on the steering wheel using RTOB [17]. The reading of the sensor values as well as the transmission of the torques to the steering wheel and the pedal system are realized through a connection board.

Figure 11.

Experimental setup

5.2 Simulation 1

In the first simulation, the drivers were asked to simulate a sleepy state by closing their eyes at the beginning of the test. Detecting a sleepy state, the driving assistant sent a haptic signal through the steering wheel in order to wake the testers up. This haptic signal is an oscillation of the steering wheel. The idea is to provoke a quick return to an awake state, as the sense of touch enables fast reactions. The steer-by-wire system is essential in this situation to prevent the application of oscillations on the steering shaft. To avoid such incidents, when the haptic signal is sent to wake the driver, the inputs applied to the steering wheel are not taken into account for a given period. At that time, the direction of the car is only chosen by the driving assistant. In normal situations, a torque corresponding to the yaw angle computed by the driving assistant is applied to the steering wheel. As the reaction of a driver who is awoken by the assistant is difficult to foresee and potentially dangerous, if the driver involuntarily turns the steering wheel, the disconnection time between the two parts of the steer-by-wire can be prolonged. In this simulation, the effect of the variation of this delay was studied. After they felt the haptic signal, the testers were asked to remain in their current lane. Each driver had to proceed with the simulation for three different initial yaw angles. For each initial yaw angle, four different delays were tested. Several testers conducted a total number of 10 groups of simulations.

The results of the analysis can be analysed as follows.

The position variation decreases as the inactivity delay increases when the initial yaw angle of the car is equal to zero.

In the other cases, the position variation first decreases and then increases as the inactivity delay increases.

When the driver is ignored, the driving support is able to prevent lane-changing.

Overcompensation of the driving support can be observed when the inactivity delay is too long.

Some drivers exhibit better acceptance of intrusive assistance.

The deviation is delayed when the inactivity delay is increased.

Figure 12a) shows the effects of an increase in the delay when the initial yaw angle is null. The shorter the delay, the sooner a deviation is observed and the bigger that deviation is, producing oscillations. Indeed, the driver may be surprised and sometimes he will apply an undesired reaction on the steering wheel. A delay allows for the suppressing of this kind of reaction. This result is confirmed by Figure 12c). According to the mean position variation, increasing the delay tends to reduce the position variations. Conversely, Figure 12b) shows that this fact is modified if the initial yaw angle is not null. As the driver has to change his trajectory to return to the middle of his lane, the sole action of the driving support can be seen to be not quick enough by the driver, and over-compensation can occur if the inactivity delay of the user is too long. Even if this situation does not arise for every driver, Figure 12d) andFigure 12e) indicate that a long delay will have a bad impact on the deviation if the initial angle is not null. The inactivity thus has to be long enough to suppress the reflexes of the awakening driver but short enough to avoid the long-term restriction of this one. This simulation demonstrates the efficiency of the driving assistant-indeed, even with a very long delay the lanekeeping is achieved by the driving assistant. However, the haptic signal – even if efficient in waking the driver – has to be chosen carefully. A short delay can lead to oscillations, but a long one can create a feeling of uneasiness for the driver, who will then feel restrained by the driving assistant, leading to hazardous reactions.

Figure 12.

Results of the first simulation

5.3 Simulation 2

The second simulation aims to test the influences of diverted estimated states of the driver in a specific driving situation – lane-changing. The goal is to demonstrate the value of an adaptive support depending upon the situation of the driver. The testers were asked to change lanes with different results from the driver model. As a reference, no potentials were applied on the lines; then, potentials corresponding to a detected lane-changing intention were considered, potentials corresponding to a normal situation for the driver were later emulated and, finally a situation describing a sleepy driver was simulated. Each driver went through five simulations per situation in order to have sufficient statistical reliability. Ten series of simulations were conducted for the different drivers.

Several observations can be made from the results of this experiment.

The trajectories followed by the drivers remain very similar, even when the strength of the potentials increases.

A learning process can be observed as the drivers progressively manage to more efficiently use the driving support.

The torque used to stabilize the trajectory in the new lane is not dependant upon the state of the driver.

The drivers increase both their crossing times and the applied torques when the strength of the potential increases (although in different proportions, depending upon the driver).

The results are similar when no potential is applied and when a lane-changing intention was detected.

In the other case, the torques applied are very strong and the crossing time is long.

Figure 13a) and Figure 13b) offer a representation of the trajectories followed by two drivers with different strengths in the potential field. The trajectories during line-crossing are similar and the strengths of the potential fields do not have any influence, demonstrating that even if the driver tries to overcome the driving assistant, no dangerous behaviour results from that attempt. The Figures also show a learning process on the part of the driver as well as the fact that the potential fields can help drivers to find more easily the centre of the new lane. Indeed, they demonstrate that the deviation on the right is reduced when the potentials increase. In fact, the drivers become used to the driving assistance and learn how to use the haptic sensation returned by the automated support in order to acquire more information about their locations on the road. By feeling the torque created by the driving assistant, it is possible to determine precisely when the car has crossed the line, even with a low potential. This aspect of the driving assistant is very important because this kind of haptic information can be obtained even with small potentials. It demonstrates the interest in keeping a low assistance-level, even if the driver is in good driving conditions. As we can see in Figure 13c), Figure 13e) and Figure 13g), the stabilization torque used to keep the new lane – the positive torque – remains constant without consideration of the strength of the potential field. This means that the road potentials do not affect this torque as the driver is in a safe trajectory once the line is crossed. When the strengths of the potentials are increased several behaviours can be observed from the drivers. It can lead to an increase in the crossing time while the torque applied on the steering wheel does not vary importantly. This is the case for Driver 5, as shown in Figure 13e) and Figure 13f). A second possibility is an increasing of the applied torque without noticeable variations of the crossing time. This behaviour is represented by Driver 3 in Figure 13s) and Figure 13d). Finally a combination of those two behaviours can be observed like for Driver 6, as shown in Figure 13g) and Figure 13h). These results are confirmed by Figure 13i) and Figure 14. The important values of the standard deviations can be explained by the diversity of the reactions of the drivers, as the standard deviations for individual drivers are small. If a lane-changing intention is detected, the torque applied by the driver is very similar to that when no potential is considered. The same fact can be observed on the crossing time. This indicates that the chosen potential for the line-crossing combines the interest of the driving assistant, haptic information about the car's location, and the freedom of the driver observed when the driving support is not active. Moreover, the measures obtained when no driving intention is detected indicate that it is difficult to operate lane-changing in this situation. It is even more difficult when the driver is classified as drowsy, therefore, in this situation, the strength of the assistant makes impossible to leave the current lane unwillingly. The testers declared that in the cases where no intention of lane changing was detected, the strength to apply on the steering wheel was strong enough to dissuade possible dangerous actions.

Figure 13.

Results of Simulation 2

Figure 14.

Mean crossing torques considering all the drivers

6. Conclusion

The innovative aspect of the driving support presented in this paper is the combination of active driving assistance based on the representation of the hazards encountered on the road by virtual potential fields with a driver modelling, using the analysis of face features to adapt the assistance offered to the driver according to his situation in order to create a more efficient and widely accepted assistant. The evolution of the role of the automobile and the current state of safety on the road contribute to underline the interest of such an adaptive driving support. Even if this concept is not new, the proposed architecture can manage the equilibrium between the driving assistant and the driver in real-time in order to give the most appropriate help as a function of the state of the driver. The simulations demonstrated the value of this architecture compared to a non-adaptive driving support. A low level of assistance allows for the obtaining of haptic information and a high level can prevent incidents. The choice between these two situations is made according to the state of the driver in order to increase the acceptance and efficiency of this solution.

Footnotes

7.

This work was supported by the Fujiwara Scholarship.

References

Eckermann

, SAE International. World history of the Automobile. 2001.14p. ISBN:9780768008005

Number of registered vehicles, Data by country [Internet] World Health Organization website. Available from: http://apps.who.int/gho/data/node.main.A995. Accessed on 22 Sept 2014.

Production statistics [Internet] Organisation internationale des constructeurs automobiles (OICA) website. Available from: http://www.oica.net/category/production-statistics/2012-statistics/. Accessed on 10 Jan 2014.

Road fatalities in the EU since 2001 [Internet] European Commission website. Available from: http://ec.europa.eu/transport/road_safety/specialist/statistics/index_en.htm. Accessed on 03 Feb 2015.

Global Health Observatory (GHO) data, Road safety [Internet] World Health Organization website. Available from: http://www.who.int/gho/road_safety/en/. Accessed on 03 Feb 2015.

National Sleep Foundation White Paper [Internet] 2010. National Sleep Foundation website. Available from: “National Sleep Foundation White Paper”, http://www.sleepfoundation.org/article/white-papers/national-sleep-foundation-white-paper. Accessed on 29 Jan 2014.

Volpe

and Kholsa

: Manipulator control with superquadric artificial potential functions. IEEE TSMC. 1990, Vol. 20:1423–1436.

Kathib

: Real-time obstacle avoidance for manipulators and mobile robots. The IJRR. 1986, Vol. 5: 500–505.

Rossetter

E.J.

A potential field framework for active vehicle lanekeeping assistance [Ph.D thesis]. Stanford University; 2003.

10.

Wolf

M.T

and Burdick

J.W.

Artificial potential functions for highway driving with collision avoidance. In: IEEE International Conference on Robotics and Automation, Padena, CA, USA. May 19–23 2008. pp. 3731–3736.

11.

Carlier

and Murakami

: Virtual Force Field Based of Force-Feedback of Road Condition for Driving Assistant Design in Electric Vehicle. International Journal of Automation Technology. 2011, Vol. 5, No. 6: 908–915.

12.

Kullack

Ehrenpfordt

Lemmer

and Eggert

: ReflektAS: lane departure prevention system based on behavioural control. Intelligent Transport Systems, IET. 2008, Vol. 2: 285–293.

13.

Krolak

and Strumillo

Vision-Based Eye Blink Monitoring System for Human-Computer Interfacing. In: HSI, Krakow, Poland, May 25–27 2008, pp. 994–998.

14.

Jimenez

Bergasa

L.M

Nuevo

Hernandez

and Daza

I.G

: Gaze Fixation System for the Evaluation of Driver Distractions Induced by IVIS. IEEE Transactions on Intelligent Transportation Systems. 2012, Vol. 13, No. 3, September, pp. 1167–1178.

15.

Nakhaei

N.D

Tang

H.S

Karasfi

Motlagh

O.R.E

and Kit

A.C

:Virtual force field algorithm for a behaviour-based autonomous robot in unknown environments. Proceedings of the Institution of Mechanical Engineers, 2011, Part I: Journal of Systems and Control Engineering, pp. 51–62.

16.

Ohnishi

: Robust Motion Control by Disturbance Observer. Journal of the Robotic Society of Japan. 1993, Vol. 11, No. 4, pp. 486–493.

17.

Murakami

and Ohnishi

: Torque sensorless control in multidegree-of-freedom manipulator. IEEE Transactions on Industrial Electronics. 1993, Vol. 40, No. 2, pp. 259–265.

18.

Viola

Jones

Rapid object detection using a boosted cascade of simple features. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition 2001 pp. 511–518.

19.

Hariri

Abtahi

Shirmohammadi

and Martel

L.A Yawning

. Measurement Method to Detect Driver Drowsiness. Technical Papers. 2012. Available from: http://www.cognivue.com. Accessed on 15 Jun 2013.

20.

Lee

S.J

Jung

H.G

Park

K.R

and Kim

: Real-Time Gaze Estimator Based on Driver's Head Orientation for Forward Collision Warning System. IEEE Transactions on Intelligent Transportation Systems. 2011, Vol. 12, No. 1, pp. 254–267.

21.

Karson

C.N

Berman

K.F

Donnelly

E.F

Mendelson

W.B

Kleinman

J.E

and Wyatt

R.J

: Speaking, thinking and blinking. Psychiatry Res. 1981, Vol. 5, No. 3: 243–246.