Towards a cognitive warning system for safer hybrid traffic

1. Introduction

Since 2009, when Google started testing Google Chauffeur driven cars, they accomplished driving over 1.5 million miles with only 22 documented minor accidents [1]. Interestingly, human error was found underlying all but one of these [2]. This warns to the fact that in spite of self-driving cars being a safer mode of transportation [3], a hybrid traffic of human-driven and self-driving cars is still prone to human faults. Human drivers are object to biological limitations (e.g. drowsiness) and tend to do multitasking in the car, thus providing suboptimal response in emergency situations [4]. Several in-car warning system designs have been implemented in order to reduce the risk of fatal outcomes [5]. In the present paper, we propose that these warning systems should not only raise the driver’s attention, but could be also used to inform other participants of the traffic, namely self-driving cars.

Widespread availably of passenger cars in the middle of the 20th century raised attention to traffic safety [9]. Since then, several different kinds of accident risk evaluation systems have been proposed. Amongst these we can distinguish three main types based on the source of data they use for estimation. These are (1) traffic data based, (2) car position based, and (3) driver behaviour based approaches. Traffic data-based approaches are typically based on traffic surveillance data and use that to evaluate the risk of accident depending on timeslot, traffic frequency and area (highway, intersection) [10, 11, 12, 13, 14]. Not entirely different from these systems [15, 16, 17] are those that work on the single car basis and use sensors of the master vehicle to predict risks of the peers. Current self-driving concept cars rely mostly on this technology [18]. The third type of risk evaluation systems is the set of systems that collect information from the driver. Driver behaviour-based models use gaze [19, 20], facial coding [19, 21], EEG [22, 23], and motion trajectories [24, 25, 26] recorded with various sensors. These solutions give very good real-time estimates that can be used to warn the driver for a potential risk of falling asleep [24, 27], driving through a red light [26], or for optimal lane-changing trajectory [28]. Here, we propose that these warnings could help the hybrid traffic of human-driven and self-driving cars in the future. This way they work more as a communication channel between two agents and not as a one-way sensor, hence the term cognitive in the title.

While a human driver may not be able to evaluate a warning message from a lead car in a couple milliseconds, this is not a problem for the processor of a self-driving car. Automated vehicles constantly monitor their surroundings with several sensors to provide the safest transportation possible [29]. Nonetheless, information collected inside the car’s cockpit may forego the externally detectable risk with tens or, sometimes, hundreds of milliseconds. This is true even if we take the steering wheel, where there is a few millisecond delay between the steering action and the chassis response [30]. Thus, these warnings may be extremely helpful for self-driving cars.

The proposed solution could be a good example of how biological and artificial cognitive agents could co-evolve [31, 32], emerging in a safer traffic infrastructure. The current proposal is not the first that promote consideration of cognitive factors in traffic safety [9, 33, 34], or increased communication between traffic participants [35, 36]. However, it is unique in its emphasis on human-to-machine information flow. Ongoing research [17, 29, 36, 37] is focusing on the design of optimal wireless communication between vehicles (vehicle to vehicle, V2V) and between vehicles and road-side units (vehicle to infrastructure, V2I). These communication links support efficiently the drivers’ situational awareness. Although situational awareness often refers to human situational awareness [38], it bears relevance also in human-machine (or possibly even in machine-machine) situations as a general concept of information availability and use in an interaction [39]. To demonstrate whether we can potentially facilitate situational awareness of a machine, we validate our idea by predicting abrupt steering wheel turn actions of a human driver in a virtual reality simulator paradigm. Here, from time to time the driver had to make emergency steering movements to roadside stimuli [40]. In the present analysis we used the car trajectory and the steering wheel angle data to investigate how early we can detect the initiation of an emergency steering behaviour only based on data from either external sensor.

In the current proof-of-concept implementation we used a one-class support vector machine (OC-SVM). SVM [41, 42, 43] is a set of machine learning models that uses support vectors (i.e. hyperplanes) in high dimensional space for classification and regression problems. Our choice of model was motivated by three main reasons. First, SVM solutions are fast and are often used in real-time applications [44]. Second, such a model can be extended, for example, a recent study presented a hybrid model of an OC-SVM and a deep belief network that outperformed a deep autoencoder in terms of speed on an anomaly detection task in high dimensional data [45]. Third, SVM can be trained even on computers with modest processing power. This latter argument is important since the current ideas may later give birth to an actual product. Presumably, people who cannot afford buying new self-driving cars would adhere to using human-driven cars, and thus would be the target audience of such an instrument. This facilitates the design of an efficient, yet inexpensive device.

We hypothesized that abrupt steering movements can be readily detected using both steering and car trajectory data. Moreover, we predicted that emergency events are detected earlier based on steering than on trajectory data. We aimed to propose a general anomaly detection system that could potentially use multidimensional data (e.g. EEG, eye-tracking etc.). These sensors could provide even earlier detection of an emergency [46]. Therefore we did not include any prior expectation of the dangerous events, only data of normal driving and hence the use of OC-SVM.

OPEN IN VIEWER

As a first step, we divided the whole no event data to training and validation sets by randomly assigning half of the time points to one and the other half to the other. Because our aim was to build a model that uses both general and personalized information, we did not split the data to two pools of participants. The model gave very small amount of false alarms on the validation set: 4.86% for the steering angle data and 4.06% for the trajectory data. After this, we used the support vectors of this model to detect the earliest anomaly point in the event data. We expected significantly high detection rate of the emergency events, and earlier detection of anomalies in the steering wheel data than in the trajectory data.

Emergencies were detected 645.15 ( ± 219.67) msec after the onset of the event. In total 2735 emergency events were detected and 8 remained undetected. As can be seen in Fig. 2 this is in the beginning of the trajectory curvature in the emergency trials meaning that we detected emergency very early in time. On the trajectory data anomalies were detected 734.54 ( ± 269.44) msec after the onset of the event, significantly later than in the steering angle data ( t (1530) = - 17.24 0.001). The detection rate was not different: 2736 emergency events were detected and 7 were undetected. The reason why steering angle made earlier detection possible is the non-linear relationship between steering angle and vehicle position (see Fig. 3).

OPEN IN VIEWER

We visualized the anomaly detection thresholds based on the validation set and emergency event data points using the t-Distributed Stochastic Neighbour Embedding (t-SNE) method [55]. This method efficiently visualizes high-dimensional data by using joint probabilities of a low-dimensional embedding. The transformation was run using the Barnes-Hut approximation in order to perform calculation in quasi-linear time. The results of the t-SNE show that the no event and emergency event data points are easily differentiable (see Fig. 4).

Summarizing the results, we found that emergency events were readily detected both in wheel angle and in trajectory data using a OC-SVM. Steering data made possible earlier detection of emergency events than trajectory data.

In the current work we proposed an in-car risk detection and warning system that could inform automatic vehicles on the road about the cautious actions of the human driver (e.g. abrupt steering movement, falling asleep). We illustrated the benefits of the risk detection component by predicting dangerous steering movements earlier from wheel angle data than from vehicle trajectory data, because of the non-linear relationship between steering angle and vehicle lateral position [56, 57].

We used one class support vector machine for learning and prediction. These type of models are common in outlier detection scenarios for various problems [45, 58, 59]. Note, that by controlling the sparsity parameter of the SVM we can limit the number of support vectors used for prediction [54], there are even solutions to find the optimal number of support vectors for a given problem [60]. Moreover, while training an SVM (and potentially multitude of SVMs for each car on the road) would be infeasible inside a master vehicle, our proposal leads to computational efficiency since training and prediction could run on the individual peer vehicles. This fact opens the door to highly individualized models.

We found earlier detection of risk in wheel angle data than in trajectory data. Although this is in line with the expectations (i.e. because of steering backlash, vehicle inertia, tire stiffness), a limitation of the current study is that it was done in virtual reality. While reactions in virtual reality are comparable to those in the real-world [61], the physics of the virtual environment are simpler than reality. Not speaking of the large variance of normal driver behaviour in real world scenarios. While in our case there were only two tasks, outside of the simulator the driver faces all the challenges of traffic. This necessitates further exploration under more naturalistic circumstances. Nonetheless, our choice of virtual reality was motivated by the fact that only this way we were able to generate large amount of clean and labelled data for training and test without real risk of accident. Further studies should evaluate the effectiveness of such a system with more degrees of freedom. Here participants were only able to control the steering wheel angle but not the speed of the car, in reality steering wheel angle changes depends on the speed of the car too, also manufacturers apply speed steering solutions in today’s cars [56].

Worthy to note, that the change of the steering wheel angle is indicative of rather distant elements of the perception-action cycle. Hence, presumably more benefit we earn from such a model when more proximal cognitive variables are tracked. Eye and face tracking in the cockpit could help detecting drowsiness very early in time [21], but also – in situations like the current experiment – could also help identifying saccades to certain stimuli inside and outside the car [8]. Wearable sensors can monitor heart rate, and therefore can be used to inform traffic peers of medical emergency. Moreover, given the increasing availability of consumer EEG headsets, it is promising that research shows electrophysiological patterns can be extremely helpful as well [22, 23].

Another interesting field of exploration is the study of information transmission and potentially further propagation of data in a vehicle network [17, 62, 63]. This way the risk information is not only locally useful but can change the state of the global network. For example, the network could start organizing detours even when an inevitable accident has not happened yet. On the one hand, creating such a one-directional inter-cognitive link between an artificial and a biological cognitive system is an important step forward from the perspective of the applied field of cognitive infocommunications [31]. On the other hand though, it raises important concerns regarding privacy and security. These systems would monitor the driver’s reactions and while communication is only intended in case of risk, it is still a potential data breach. Moreover, malicious attack is also possible against the automated car by sending large amount of risk notifications. The communication link therefore must be secured. Indeed, current research on intelligent automated traffic, smart cities and situation awareness of self-driving cars is aware of these challenges [17, 35, 64, 65].

Researchers working on self-driving cars say that fully automated cars are still years or even decades ahead [29, 66]. Meanwhile, semi-automatic solutions are increasingly available (automatic parking, highway autopilot) [67, 68]. Thus, roads are becoming more and more a niche of biological and artificial drivers. In this situation we may want artificial cognitive agents to co-evolve with our biological cognitive systems. In the present work we detailed one aspect of this endeavour, namely inter-cognitive warning systems. The core of arguments was the importance of communication of the human drivers’ cognitive and behavioural states to self-driving cars to increase road safety in the future.