Abstract
Skraaning and Jamieson’s (2023) article defines, provides examples, and offers a taxonomy for automation failures. They also invite others to apply their concepts to other domains. Driving automation systems along with their myriad of failures provide the perfect test case. This article defines, characterizes, and discusses prevention mechanisms for driving automation system failures. By combining Skraaning and Jamieson’s (2023) original taxonomy with characterizations and prevention mechanisms of driving automation system failures, their work is extended and substantiated.
Skraaning and Jamieson’s (2023) article defines and introduces a taxonomy of automation failure mechanisms, with a focus on negative outcomes of automation for industrial applications. The taxonomy separates automation failures into three categories (elementary automation failures, systemic automation failures, and human–automation interaction breakdowns) based on which mechanisms lead to human performance challenges when dealing with automation. They invite others to apply this automation failure definitions and taxonomy to other domains and applications. Given the frequency with which humans interact with driving automation systems in their daily lives along with the opportunity for automation failure, driving automation systems appear to be a suitable target application. In this article, first, failures in driving automation systems are defined and characterized. Next, prevention and mitigation mechanisms for driving automation system failures are highlighted. Finally, characterization and mitigation mechanisms are combined with Skraaning and Jamieson’s (2023) original taxonomy to define a new taxonomy for driving automation systems.
Failures in Driving Automation Systems
Driving automation system failures have been well documented, especially among users of production systems (Dikmen & Burns, 2016; Endsley, 2017; Larsson, 2012). Failures can be caused by different mechanisms, like a limitation of the system or a malfunction within the system (DeGuzman et al., 2020). Failures in driving automation systems can be salient and transparent or silent and hidden (Bianchi Piccinini et al., 2020; Louw et al., 2019). Failures can be a minor nuisance or they can be safety critical (Mole et al., 2020; Strand et al., 2014). Failures can be due to rare, infrequent, uncommon situations or caused by everyday events (Pai et al., 2023). Perhaps most evidently, failures differ according to the level of automation.
The Society of Automotive Engineers have prescribed six levels of automation for vehicles (Society of Automotive Engineers (SAE) International, 2018). Within each automation level, the dynamic driving task is operating the vehicle in traffic, including maintaining lateral and longitudinal control as well as event detection and mitigation. At level 0, the driver is responsible for all aspects of the dynamic driving task. In level 1 and 2, the system is responsible for lateral and/or longitudinal control of the vehicle in limited conditions, but the driver must regain control of the vehicle when necessary or when requested by the system. In level 3 systems, the system is responsible for the dynamic driving task, but the driver must regain control of the vehicle when requested by the system. Level 4 systems are characterized by technology that handles the entire dynamic driving task in limited situations and the driver does not need to regain control of the vehicle in case of failure. Level 5 is an extension of level 4 except it functions in all situations.
Defining Driving Automation System Failure
With these five levels of automation in mind, a failure in driving automation systems can be defined as an event in which the system no longer performs the dynamic driving task at full capacity for a sustained amount of time. Note that this definition of a driving automation system failure is more encompassing than the SAE definition (SAE International, 2021) as the SAE definition ignores system or design limitations and only considers system malfunctions as automation failures. Failures vary depending on the automation level. For example, a failure in level 1 for an adaptive cruise control system would occur when the system no longer performs the task of longitudinal control due to a non-functioning sensor. A failure in a level 4 system would be an incapacitated vehicle that pulls itself over to the side of the road. Excluding level 0 because the automation is not engaged, across the five higher levels of automation, failures can be characterized by type, transparency, and responsibility.
Failure Type
Within levels 1, 2, and 3, there are automation failures due to known system limitations. Such limitation based failures generate takeover request that require drivers to take over control of the vehicle from the system (DeGuzman et al., 2020). For example, in a level 1 system, if it encounters poor weather conditions, thereby yielding its sensors incapable of reading the lane lines, it will request that the driver resume lateral control of the vehicle. Failures due to a system limitation are often obvious to the driver (because the system issues a warning), thereby allowing for a successful recovery from the automation failure. This is in comparison to a failure caused by a system malfunction, which is unforeseen by both the driver and the designer of the system (DeGuzman et al., 2020; Mishler & Chen, 2023). For example, poorly marked lane lines (i.e., infrastructure issues) can cause the vehicle to steer off course (Louw et al., 2019) or the brakes may completely malfunction in an automatic emergency braking/adaptive cruise control system (Strand et al., 2014). Failures due to a system malfunction can happen at any level of automation and tend to be ambiguous or not obvious, thereby yielding a successful recovery unlikely.
Failure Transparency
Automation failures can be salient or silent. Takeover request in levels 1, 2, and 3 that correspond to system limitations tend to be salient and transparent. The driving automation system often gives structured warnings to the driver to indicate it has reached a limitation and that the driver must soon resume control (Mole et al., 2020). On the other hand, system malfunctions in driving automation systems tend to be subtle and hidden from the driver as by definition, they do not include an explicit alert to the driver (Mole et al., 2020). Some researchers consider these silent failures to be unstructured, in the sense that the transition of control from automation to human is sudden and not planned (Blommer et al., 2017) as well as unpredictable (Mole et al., 2020). Research on silent failures is much less common (Louw et al., 2019), but often, leads to distrust in the system (Mishler & Chen, 2023)and the most safety critical situations (Strand et al., 2014) as takeover time in these situations can be upwards of 40 seconds (McDonald et al., 2019).
Failure Responsibility
Closely related to the taxonomy proposed by Skraaning and Jamieson (2023), failures in driving automation systems can be caused by the system or the human–system interaction. Failures caused by the system are well documented, whether they be a system limitation or a system malfunction. When it comes to failures caused by the human-system interaction, the literature highlights real world examples of human–automation breakdowns. For example, mode confusion, wherein the driver is confused about which level of automation is engaged, is a common topic (Eom & Lee, 2022; Wilson et al., 2020) that demonstrates a breakdown in the interaction between human and system. As another example, misuse and disuse of operational driving automation systems has been well documented (Kim et al., 2020; Nordhoff et al., 2023) and indicate the unintended ways in which driving automation systems are used. Note that the responsibility when it comes to automation failure is distinct from responsibility when it comes to a crash caused by an automation failure, which is discussed elsewhere (Bennett et al., 2020; Wotton et al., 2022).
Preventing and Mitigating Driving Automation System Failures
No matter the type of failure, it is equally important to focus on preventing and mitigating the negative effects of driving automation system failures. Within the space of driving automation systems, it has been well documented that there is very little formal education for drivers to learn about the systems before, during, or after their use (Abraham et al., 2017; Casner & Hutchins, 2019) and consequently, drivers have poor mental models (Endsley, 2017; Krampell et al., 2020; Merriman et al., 2023a, 2023b; Pai et al., 2023). Though many drivers often learn about driving automation systems through trial and error, it has been proven that trial and error while using ACC (and simply experiencing failures) does not lead to appropriate mental models (Beggiato & Krems, 2013; Endsley, 2017). Here, mental models are defined as “the rich and elaborate structure which reflects the user’s understanding about the system’s contents, its functionality and the concept and logic behind the functionality” (Carroll & Olson, 1987, p. 12). Skraaning and Jamieson (2023) highlight the importance of accurate mental models in recovering from automation failures. Relatedly, past research has indicated that there are many methods to improve driver’s understanding of driving automation systems and their mental models, mainly appropriate human–machine interface design and training (Pradhan et al., 2019).
Human–Machine Interfaces
Human–machine interfaces are essential for delivering information about the automation’s state to the driver, for giving feedback to the driver, and for warning the driver when the automation is or already has failed (Bazilinskyy & DeWinter, 2015; Naujoks et al., 2019; van den Beukel & van der Voort, 2017). The design of such interfaces and warnings has been thoroughly investigated, reviewed, and summarized (Mehrotra et al., 2022). While appropriate human–machine interface design can improve driver’s responses to automation failures (Roberts et al., 2022), it has shown mixed results in terms of improving driver’s mental models (Monsaingeon et al., 2021; Perrier et al., 2023). Additionally, such interfaces can only prevent or mitigate human–automation interaction breakdowns as well as elementary automation failures that are limitation-based, salient, and caused by the system. In other words, if the failure is unpredictable and silent, as in a systemic automation failure, an interface cannot warn the drive of automation failure.
Education Through Training
In the case of systemic automation failures that are malfunction-based, silent, and caused by the system or the human, a different approach is needed. A wider-reaching method to improve driver’s mental models of driving automation systems and their response to automation failures is education through the form of training. Not only does training improve trust (Ebnali et al., 2019), performance (Chen et al., 2023; Noble et al., 2019), and driver’s mental models of driving automation systems (Merriman et al., 2023a, 2023b; Pai et al., 2023), it has also been shown to improve overall driving skills such as hazard perception, visual search, and situation awareness (Krampell et al., 2020). Though many training programs and systems have been evaluated when it comes to improving drivers’ mental models, recent research has specified the minimum amount of information that should be given during training (Casner & Hutchins, 2019) along with a focus on training that highlights the driver’s responsibility when using automation (DeGuzman & Donmez, 2022).
Taxonomy for Driving Automation System Failures
Taking it one step further, instead of categorizing driving automation system failures by type, transparency, or responsibility and differentiating mechanisms to prevent and mitigate automation errors, it may be more useful to combine Skraaning and Jamieson’s (2023) taxonomy with these categorizations, as shown in Figure 1. Elementary automation failures, which are caused by isolated failures of individual automation components (Skraaning & Jamieson, 2023), are associated with driving automation system failures that are limitation-based, salient to the driver, and caused by the system. Such failures can be prevented or mitigated with good human–machine interface design as well as proper training. Systemic automation failures are caused by issues in the interaction of components, functions, or logic of automation (Skraaning & Jamieson, 2023). In driving automation systems, such failures are malfunction-based, not transparent to the driver, and caused by the system. Though these failures can be prevented or mitigated by training, good principles of human–machine interfaces do not help. Last are human–automation interaction breakdowns, which represent a mismatch between how the automation was designed and how humans function (Skraaning & Jamieson, 2023). Human–automation interaction breakdowns in driving automation systems represent themselves as failures that are (potentially) limitation based, either salient or silent, and caused by both the system and the human. Such failures can be prevented or mitigated with good human-machine interface design as well as proper training. Taxonomy of driving automation system failures in relation to the original taxonomy by Skraaning and Jamieson (2023).
There are many benefits of applying and expanding upon Skraaning and Jamieson’s (2023) initial taxonomy to failures in driving automation systems. First, driving automation systems are similar to, yet distinct from the aviation domain, which was the focus of the original article. While both are within the broad field of transportation, driving automation systems are operated by the general public on real world roads with unpredictable conditions, not specialists or experts in confined and controlled contexts. Additionally, driving automation systems failures are much more frequent and are (sometimes) scrutinized to a greater degree when crashes occur. As such, taking the initial taxonomy and adding components that are specific to driving automation systems allows one to see how the human automation interaction and cognitive engineering literature, which was the basis of the initial taxonomy, applies to the driving domain, similar to past research (Endsley, 2017).
Second, while taxonomies have been developed that describe driving automation system takeovers or handovers (McCall et al., 2016, 2019), a taxonomy focused on failures has yet to be realized, until now. Relatedly, while there are nearly hundreds of articles on failures in driving automation, mainly focused on takeovers (Morales-Alvarez et al., 2020; Zhang et al., 2019), there has yet to be a concerted push towards characterizing these failures in such a way as to focus on their prevention or mitigation. Taken together, a taxonomy delineating driving automation system failures will benefit the driving automation system community in identifying, characterizing, investigating, preventing, and mitigating failures. Such a taxonomy seems to be needed in a time when high profile driving automation system failures degrade user’s trust in these systems (Tapiro et al., 2022; Xu et al., 2021) and ultimately, hinder their development and widescale deployment.
Conclusion
Examining, defining, characterizing, and identifying prevention and mitigation mechanisms for driving automation system failures provide a nice complement to the original article (Skraaning & Jamieson, 2023) as it is a non-industrial application with publicly available information about (high profile) automation failures (e.g., in the public media and via National Transportation Safety Board reports). Additionally, failures in driving automation systems are observable, measurable, and allow for ample opportunities for recovery. The taxonomy of driving automation failures outlined here can be applied to other domains, including both industrial and non-industrial applications, where the user may use the automation infrequently with no formal education, yet has safety critical implications.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
