Abstract
Despite multiple taxonomies and descriptions of automation there is inconsistency in describing automation capabilities, making it difficult to interpret and replicate research. We conducted a systematic literature review to investigate how studies document automation. The Scopus® database was searched on January 13th, 2023, for vehicle automation studies published in the Proceedings of the Human Factors and Ergonomics Society Conference between 2012 and 2022. Twenty-one studies were identified. Sixteen of these studies described the vehicle automation in the protocol and apparatus and five were missing at least one element. We show inconsistency and insufficiency of these explanations, suggesting researchers and participants might not understand the levels of automation. We offer a guide for improving how researchers describe automation capabilities to improve the interpretability and replicability of studies.
Keywords
Introduction
There is no unified taxonomy for describing all variations of motor vehicle automation despite multiple attempts (Gasser & Westhoff, 2012; National Highway Traffic Safety Administration, 2013; SAE International, 2021). The most cited source for describing vehicle automation is the Society of Automotive Engineers (SAE) Levels of Driving Automation (SAE International, 2021). The SAE taxonomy partitions vehicle automation functions into six levels, each with driver responsibilities and automation capabilities (Table 1).
The role of the human and automation system according to SAE Levels of Automation (adapted from J3016).
The SAE framework can help communicate automation capabilities. However, drivers and researchers struggle to understand the user’s role and the automation capabilities in terms of the SAE levels (Novakazi et al., 2021). This might undermine the interpretation and replication of research on driver interaction with increasingly automated vehicles and varied capabilities within levels. Although general guidelines help researchers describe their experiments to ensure replication, the complexity of the SAE levels presents challenges. Specifically, they were designed as a reference for engineers and are often difficult for drivers to understand. This difference in understanding can be seen in the results of studies in which driver mental models do not align with the SAE levels (Novakazi et al., 2021). Likewise, researchers miscite the SAE levels when they state nonexistent levels of automation capabilities like “Level 2.5” (Ataya et al., 2021). The SAE levels are framed around three descriptive dimensions, the Dynamic Driving Task (DDT), Object Event Detection and Response (OEDR), and the Operational Design Domain (ODD). DDT is the operational (e.g., steering and acceleration) and tactical (e.g., honking horn, maneuvering, and signaling) functions required to operate a vehicle. The OEDR defines the sensing and control abilities to monitor and respond to the environment, which may include planning maneuvers. The ODD defines automation where and when the automation is meant to operate (e.g., time of day, roadway type, speed, etc.).
The SAE levels specify increasingly capable automation systems, each prescribing different driver roles. Level 0 only provides warnings and momentary assistance using vehicle safety subsystems such as antilock braking and stability control.
Level 1 requires driver-supervised automated steering or braking and performs the remainder of the DDT. Level 2 provides simultaneous longitudinal and lateral control, and the driver performs the remainder of the DDT. Level 3 performs the entire DDT in an ODD but assumes that the driver is receptive to a request to intervene. Level 4 can operate without driver engagement in a defined ODD. Level 5 can perform the entire DDT in any driver-manageable on-road conditions. Importantly, a manufacturer may generally represent the driving automation of their vehicle according to these SAE levels. However, the actual implementation of automation is qualified by the specifics of the Tactical Maneuver Competency of the DDT, the ODD, and the OEDR, within each SAE level. Therefore, driver roles can vary considerably between offerings within an SAE level according to the degree of competency described by each dimension. Specifically, each cell in a threedimensional framework of DDT, OEDR, and ODD define specific variance within each SAE level. On one hand, a framework with fewer discrete levels for each dimension may describe fewer potential systems. On the other hand, a finer scale of each dimension would describe a very large number of potential design variants, each with slightly different driver roles. This implies that the three-dimensional design space, whatever the specificity, will contain regions of variants associated with each SAE level. A key challenge is how to scale the framework for designing driver roles and describing research.
Although this taxonomy seems to unambiguously describe vehicle automation, in practice both researchers and drivers often struggle to appreciate the nuances within and between levels (Novakazi et al., 2021). For example, within Level 2, a vehicle may include features like intelligent speed control and automatic lane changes. Similarly, design affordances within Level 2 may vary across systems, for example, one design may afford a great deal of steering input, while another may afford very little. This paper addresses two research questions: 1) Do the method sections of HFES conference papers precisely describe the automation implemented for the study? and 2) Do these method sections precisely describe the automation to study participants? By addressing these questions, we identify how to precisely document research and experimental conditions.
Methods
Seventy-five articles from the Proceedings of the Human Factors and Ergonomics Society Conference were found using the keywords “driving” OR “vehicle” AND “automation” in the Scopus database. We included studies about vehicle automation in either a driving simulator or automated vehicle, and excluded all other study designs, such as survey studies or literature reviews, reducing this list to 21 articles.
The lead author then screened each article and extracted the explanation of the automation capabilities from the apparatus and protocol. The apparatus describes the automation’s capabilities. Automation capability refers to the control that automation exerts, under what conditions, and if the human must take over when prompted. The protocol instructions are the written description of how the participant was introduced, taught, or trained on how to use the automation.
This includes the participants’ responsibility and the automation capabilities once engaged.
The completeness of the information in the apparatus and protocol sections of the experimental methods were coded. We recorded whether the SAE levels were mentioned, and if the level of automation used in the experiment was described. We also rated how well the description covered information that may support appropriate understanding and trust in the automation: purpose, process, and performance (Lee & See, 2004). These dimensions were defined as: Purpose (P1)—What is the drivers’ role and why the automation was developed? Process (P2)—How it works (e.g., sensors, limits, etc.)? Performance (P3)—What should the user expect from warnings and how are they expected to interact? We rated how completely these dimensions are addressed on a 0 to 3 scale: 0—corresponding to the component’s absence, 1—present but scarce (e.g., states “capable of automated driving” but no details on what it can do), 2—partial details (e.g., States “Level 3” without stating the ODD), 3—enough detail to replicate the conditions.
Results
Table 2 summarizes how 21 papers published in the HFES conference proceedings from 2012–2022 applied the levels of automation and described the vehicle automation. Of these, 14% (3) failed to describe the apparatus, 9.5% (2) did not provide their protocol instructions, 42% cited the SAE levels of automation, and 21.8% (5) explicitly stated the level of automation used.
Explanations of automation capabilities to both the reader (apparatus) and the research participant (protocol). ◊ = Mentions SAE levels standard, ◑ = No participant explanation, ◐ = No automation explanation, ▲ = States level of automation. * = includes an explanation from a prior paper, (Clark &Feng, 2016).
The participant protocol instructions were rarely discussed. The most defined instruction was “familiarization” or practice drives where the participant used the automation before the experiment began. Thirteen studies described that the participant was allowed to explore and use the automation in a familiarization drive before the study began. Four of these study procedures specified that they instructed the participant on the automation’s capabilities and limitations. However, only two described instructing the driver on how to use the automation, and three stated that they gave them “information” about the automation system such as how to turn it on and off.
Discussion
Contents of the Method Sections
The method section is integral to understanding the study and replicating the research and should be thoughtfully written (Coverdale et al., 2006; Kallet, 2004; Kotz & Cals, 2013). It ensures that the experiment is described in enough detail to be replicated by others, allowing readers to judge the validity and generalizability and decide whether to incorporate the findings into practice (Kallet, 2004). When this information is missing fellow researchers cannot reproduce experiments, introduce new variations, or build upon theory. This section also helps others interpret findings and assess validity, such as biasing factors, and allows readers to decide whether to incorporate the findings into practice (Coverdale et al., 2006; Kallet, 2004).
We recommend, when appropriate, researchers write a detailed account of the participants’ introduction to the automation system. When there is limited space, such as with the HFES conference papers, researchers must be concise in describing the automation capabilities. Our guide in Table 3, shows what needs to be discussed for the purpose, process, and performance of the automation to be sufficiently described.
A guide describing automation capabilities and how they are introduced to participants in research papers.
A more general issue, affecting human subject research is the “psychologist’s fallacy”, a cognitive bias that describes how experimenters mistakenly assume their perception of a situation matches the participants (James, 1890). This is particularly critical in vehicle automation studies because drivers’ mental models are unlikely to be based on SAE levels and so may diverge from those of the SAE (Novakazi et al., 2021; Lee et al., In Press). Mental model quizzes can assess participants’ understanding of the automation and mitigate the psychologist’s fallacy.
The protocol describing the automation should specify the degree of interaction between the participant and the automation. The introduction is important as it indicates how “in the loop” and educated the participant is. A driver’s understanding and trust in automation can change how it is used (Körber et al., 2018; Novakazi et al., 2021). Naturally, researchers want to ensure that the participant understands what is expected of them and will often incorporate a practice drive, where the participant has an opportunity to become familiar with the automation capabilities. However, in studies involving control and experimental groups, participants may or may not be introduced to the automation, given prior hands-on exposure, or told certain details. This further raises the importance of describing any interface and associated design affordances as well as following strict protocols when communicating with subjects. The explanation to participants is critical because it can affect user trust, which influences automation acceptance, monitoring, and reliance (Choi & Ji, 2015; Körber et al., 2018; Lee & See, 2004; Sanders et al., 2019).
This brief review only included articles from the HFES conference proceedings where the five-page limit constrains what researchers can include. Future reviews should broaden the scope and consider journals that provide more space for discussing experimental design. Furthermore, the Scopus® database does not include all relevant published articles from 2012 to 2022. Future studies should use the SAGE Journals® database as it contains all HFES papers.
Automation Descriptions in Research
Although SAE levels provided a structure to classify and describe automation, many studies have found that a humancentric description of automation may be more valuable (Jamieson & Skraaning, 2018; Kaber, 2018; Novakazi et al., 2021; J. H. Yang et al., 2017). The current classification of automation levels does not fully account for human behavior or accommodate the complexity of automation with its simple hierarchy (Jamieson & Skraaning, 2018). Furthermore, our analysis reveals inconsistencies in automation research, suggesting potential shortcomings in its effectiveness in communicating automation capabilities.
Drivers’ understanding of automation often diverges from what is offered by SAE; instead of single levels, people have conceptual models containing elements of multiple levels that imply different assumptions about context, vehicle, and driver (Novakazi et al., 2021). Moreover, each level may contain different embodiments of DDT, OEDR, and ODD as offered by manufacturers, offering different features within a defined level. The methods section should have a comprehensive description of the human role in system control. (Kaber, 2018; J. H. Yang et al., 2017). This is particularly relevant for experiments focusing on specific scenarios (e.g., highway driving) with diverse user roles and automation competencies that span multiple SAE levels.
Footnotes
Acknowledgements
Thank you to members of the Cognitive Sciences Lab. This project was supported by the National Science Foundation under grant number 1839484. Any opinions and conclusions expressed within are those of the authors and do not necessarily reflect the views of the National Science Foundation.
