Abstract
Objective:
To present a structured, narrative review highlighting research into human perceptual-motor coordination that can be applied to automated vehicle (AV)–human transitions.
Background:
Manual control of vehicles is made possible by the coordination of perceptual-motor behaviors (gaze and steering actions), where active feedback loops enable drivers to respond rapidly to ever-changing environments. AVs will change the nature of driving to periods of monitoring followed by the human driver taking over manual control. The impact of this change is currently poorly understood.
Method:
We outline an explanatory framework for understanding control transitions based on models of human steering control. This framework can be summarized as a perceptual-motor loop that requires (a) calibration and (b) gaze and steering coordination. A review of the current experimental literature on transitions is presented in the light of this framework.
Results:
The success of transitions are often measured using reaction times, however, the perceptual-motor mechanisms underpinning steering quality remain relatively unexplored.
Conclusion:
Modeling the coordination of gaze and steering and the calibration of perceptual-motor control will be crucial to ensure safe and successful transitions out of automated driving.
Application:
This conclusion poses a challenge for future research on AV-human transitions. Future studies need to provide an understanding of human behavior that will be sufficient to capture the essential characteristics of drivers reengaging control of their vehicle. The proposed framework can provide a guide for investigating specific components of human control of steering and potential routes to improving manual control recovery.
Keywords
Introduction
The term automated vehicle encompasses a wide variety of systems that provide some form of driver assistance (SAE, 2016). Many Level 2 (SAE, 2016) semiautomated systems where lateral and longitudinal control can be temporarily relinquished by the driver to the vehicle, but the driver’s hands typically remain on the wheel, are already commercially available (e.g., traffic jam assist, automated parking; Chan, 2017; Sousa, Almeida, Coutinho-Rodrigues, & Natividade-Jesus, 2017). There is a long list of car companies trying to rapidly develop vehicles with higher levels of automation, who are promising widespread deployment of AVs by the early 2020s (Chan, 2017; Bagloee, Tavana, Asadi, & Oliver, 2016) with a number of such systems already being piloted on public roads. Most of these AVs will not be driverless (Level 5; SAE, 2016); rather, they will be Level 3 or 4 systems that are largely automated but still require a supervising driver who receives handover of control (during a period which we will refer to as a control transition) to manage situations where the AV is unable to safely maneuver.
Transitions might occur in systematic and planned ways (e.g., the AV always hands over control when leaving the motorway), but also for a variety of unplanned reasons where the AV system fails. Failures will encompass a multitude of situations where the AV no longer operates safely, so they could occur at a variety of timescales with differing degrees of warning for the driver depending on whether the AV system is able to identify that a failure state has occurred. Likely examples of AV failure states include situations where information about the environment has become uncertain (e.g., if a road has degraded lane markings or unusual signage, or if the vehicle sensor signals are disrupted due to weather conditions or a weak GPS signal; Sousa et al., 2017). AV reliability will continue to improve, but preventing failure conditions entirely is a massive challenge; therefore, it is likely that transitions will be a feature of AVs for the foreseeable future. The AV systems that are able to successfully transition control to the human driver will be the ones that can be deployed most readily: The AV can be given control during easier-to-automate situations where there is less uncertainty about the vehicle position relative to the external environment (such as driving along motorways) while also being able to relinquish control to the human driver when the environment is too complex or uncertain (e.g., driving through a busy city center or negotiating country lanes).
The core assumption of Level 2 to 4 AV systems is that humans can and will safely and rapidly take over control of a moving vehicle. Depending on the situation, however, the human driver may be faced with a set of environmental and vehicle characteristics that are drastically different from when they were last in control. The assumption that drivers can jump back in to control and produce actions that are appropriate for the conditions is not well aligned with our current understanding of how humans perform highly dynamic active control tasks such as steering (see Lappi & Mole, 2018, Mulder et al., 2017 for recent reviews). Nor is it supported by ergonomics research into human performance of monotonous “monitoring” tasks over prolonged periods where active input is only rarely required (Molloy & Parasuraman, 1996).
To illustrate the problem, consider the case of a driver relinquishing control to an AV prior to joining a motorway. When the driver was in control of steering, the vehicle was traveling at fairly slow speeds on a dry road. After the AV was given control, the driver does not feel a pressing need to keep gaze directed to the road ahead, he or she may look around at the scenery or even direct gaze to other tasks such as reading e-mail. The AV then detects road construction ahead and alerts the driver to take over control in response to this event. The driver’s task is to coordinate steering and/or braking actions to smoothly generate a safe path during the transition event, but in the intervening period (since the driver was last in control of the vehicle) it has rained, causing reduced road friction, and the car is also traveling at a higher speed than previously. If the driver is explicitly aware of the changing conditions, he or she could take some form of tactical precautionary measure (i.e., an arbitrary reduction in speed), but the driver’s sensorimotor system will also need to adapt rapidly to the altered control dynamics (due to lower adherence and increased vehicle speed). Because the driver has not been in control of the vehicle, his or her sensorimotor system may not be well calibrated to the new environmental conditions and/or vehicle dynamics, and the driver may not have access to useful perceptual information that would be needed to plan and execute the possible driving actions (either due to not looking at or attending to the road ahead). In this example, it seems likely that the ability of the human driver to successfully steer will be diminished (compared with a situation where control had been manual throughout), resulting in less safe lane-keeping or collision-avoidance maneuvers by the human driver.
Highly automated vehicles may populate our streets in the not-so-distant future (Chan, 2017), yet currently we do not have sufficient understanding of the factors affecting driving performance in transition scenarios to inform the design of these systems. This review examines control transitions from the perspective offered by the extensive literature on human perceptual-motor control. In order to apply these findings, it is first important to identify the theoretical framework in which we will situate this research.
Identifying a Framework for Examining Transitions
Driving is complex and can be broken down into numerous subcomponents, the nature of which will depend on the environment being driven through, the familiarity of the driver with the environment, and also the driver’s level of skill. Capturing the complexity of driving is usually approached using a framework that employs a hierarchy of distinct control loops (Donges, 1978, 1999; Hollnagel, Nåbo, & Lau, 2003; Lappi & Mole, 2018; McRuer, Allen, Weir, & Klein, 1977; Michon, 1985; Salvucci, 2006). Although the description of each control level varies across frameworks, many of the underpinning principles are shared. At the highest level, the driver sets navigation goals. The middle levels are responsible for composing the route from a sequence of actions, e.g., changing lanes within the constraints of the current traffic environment. The lowest level is responsible for controlling the underlying perceptual-motor behaviors (lateral and longitudinal control actions) that propel the vehicle along the desired trajectory (Figure 1). While lateral (steering) control and longitudinal (speed) control are clearly related (e.g., speed choice can preclude certain steering responses, and steering response gain will depend on speed) it is also the case that different perceptual variables provide useful information about lateral and longitudinal control. For this reason, research into human perceptual-motor control often considers these behaviors independently. The primary behavior considered in this review is the effect of automation on lateral (steering) control: adjusting the direction of travel to meet the current and upcoming requirements specified by the road.
The principles underlying multilevel driver models. Each level has its own control loop, with specific environment inputs and action outputs. The terms used in this figure are taken from Michon (1985), but the principles are similar for all multilevel driving models. The higher level, strategic control, is concerned with general plans that require infrequent updating, for example large scale route setting through the environment. The middle level(s), tactical control, is concerned with the organization and sequencing of actions that determine the course through the local environment. The lowest level, operational control, involves rapid perceptual-motor control loops that execute the necessary steering commands to keep the vehicle on the selected trajectory. The lowest operational control level is the primary focus of this manuscript.
Within the framework presented in Figure 1, the lower the control level, the higher the feedback frequency (Donges, 1999; Hollnagel et al., 2003; McRuer et al., 1977; Michon, 1985; Salvucci, 2006). Successful steering control (the lowest level, operational control) is supported by rapid perceptual-motor loops. System lags mean that for smooth control, drivers require perceptual inputs not only of the current vehicle lane position and heading, but also preview information obtained from 1–2 s ahead, in the direction of where the driver wishes to go (Chattington, Wilson, Ashford, & Marple-Horvat, 2007; Land & Lee, 1994; Lappi, Lehtonen, Pekkanen, & Itkonen, 2013; Lehtonen, Lappi, Koirikivi, & Summala, 2014; Lehtonen, Lappi, Kotkanen, & Summala, 2013; Salvucci & Gray, 2004; Wilkie, Wann, & Allison, 2008; Wilkie, Kountouriotis, Merat, & Wann, 2010) or even further along the road (if road regions are visible beyond this typical preview distance, drivers may make anticipatory look-ahead fixations; Lehtonen et al., 2013, 2014; Lehtonen, Lappi, & Summala, 2012; Mars & Navarro, 2012). Based on these perceptual inputs, the driver needs to determine quickly which path to take and how to coordinate steering actions to get there. The resulting motor commands set the conditions for new perceptual inputs, and the loop repeats. A consequence of this operational (perceptual-motor) control loop is that vehicle control and gaze tend to be tightly coupled during manual control of driving.
This article focuses on the consequences of automated driving, specifically the impact of automation upon the operational control loop. The operational control loop is the first loop to be disengaged, and is critical for all takeover scenarios: higher level loops can only affect ongoing activity through the pathway provided by perceptual-motor control. This focus is intended to complement existing human factors frameworks that adopt more holistic high-level approaches (e.g., see Merat et al., 2018 for a common definition for being “out of the loop”). Although considerations of driver control that introduce concepts such as “out of the loop” (Merat et al., 2018) or “situation awareness” (Endsley, 2017) provide useful high-level descriptions for capturing the nature of driving, this manuscript aims to bring greater clarity to the underlying role of perceptual-motor control. This focus necessitates the omission of a number of topics (such as driver distraction) that are important research challenges for control transitions but are outside the scope of the current manuscript (and for which reviews already exist, e.g., Engstrom et al., 2017). By concentrating on perceptual-motor control, this article will highlight concepts that may be less familiar to the human factors readership but in our view are no less critical.
Overview and Method
The section entitled The Perceptual-Motor Loop (Outside of Transitions) highlights the literature that underpins our current understanding of the human perceptual-motor loop for the operational control of steering. The section entitled Automated Driving Will Break the Perceptual-Motor Control Loop then considers the likely impact of automation on the perceptual-motor control loop and the possible implications of transitions of control. The section entitled Current Evidence: Human Responses During Transitions then relates these predictions to current evidence examining human performance when taking over control from AVs. Finally, in Conclusions and Future Directions, we look to the future and assess how current technological advances may address some of the issues raised in the previous sections.
This article is not intended to be a systematic review, rather our purpose is to take knowledge from one theoretical domain (the area of human perceptual-motor control) and apply it within the context of a newly emerging, distinct but related field (transitions out of automated driving). We met this aim through the pursuit and reporting of two distinct literature searches. The first is a structured, narrative review of papers in the domain of perceptual-motor control, selected based on the accumulated expertise of the authors in order to highlight literature that can be best related to transitions of control. Although this review is extensive, there were no strict inclusion criteria. The section entitled Current Evidence: Human Responses During Transitions presents a semistructured review of the way that perceptual-motor control is examined within the existing literature on transitions. We used Google Scholar to conduct specific searches on perceptual-motor calibration and gaze and steering coordination (the mechanisms that are this article’s focus) during automated driving (see Table A1 for search terms). We found 10 relevant papers using these searches. Further candidate articles were identified using existing reviews reported in Lu, Happee, Cabrall, Kyriakidis, and de Winter, 2016, and Eriksson and Stanton, 2017a, that were then complemented with additional articles found through ad-hoc searches and citation networks. Although there were no strict inclusion criteria, we gave preference to papers that could be accessed that were in English, and that: (a) described transitions out of a period of automated driving to human control of driving, (b) considered automation of lateral control, (c) reported empirical objective metrics on driver perceptual-motor control (recorded actions of the driver or the vehicle, rather than subjective report), and (d) were published from the year 2010 onwards to ensure relevance to automation of lateral control. In total, Current Evidence: Human Responses During Transitions is supported by 53 papers on transitions out of automated driving, which was deemed sufficient for assessing the current transition literature (see Table A2 for the full list of references).
The Perceptual-Motor Loop (Outside of Transitions)
To understand how automation could affect the perceptual-motor level in driving transitions, it is necessary to establish how the human perceptual-motor systems are normally involved during successful steering control. The following primer is aimed at readers unfamiliar with the existing perceptual-motor control literature, to provide sufficient background to appreciate how the issues raised apply to control transitions out of automated driving (for more in-depth reviews, see Land & Tatler, 2009; Lappi, 2014; Lappi & Mole, 2018; Regan & Gray, 2000). In particular, we introduce two key concepts in the following subsections: (1) perceptual-motor calibration and (2) gaze and steering coordination. Perceptual-motor calibration (fully described in the next section) refers to how individuals maintain appropriately scaled movements in conditions where there are changing task dynamics; although this is a common conceptual framework in the motor control literature (see Brand & de Oliveira, 2017; van Andel, Cole, & Pepping, 2017 for recent reviews), the issue is often not explicitly considered in the steering control literature, despite having important implications for control transitions. Gaze and steering coordination (fully described later) refers to the way that drivers use head and eye movements to anticipate upcoming steering requirements (and how the steering requirements themselves alter gaze patterns), and we (and others) consider this coordination to be central to understanding how humans drive (Chattington et al., 2007; Lappi & Mole, 2018; Land, 1992, 1998; Land & Lee, 1994; Land & Tatler, 2001; Lappi, 2014; Lehtonen et al., 2014; Mars, 2008a; Wilkie et al., 2008, 2010). In the section entitled Automated Driving Will Break the Perceptual-Motor Control Loop, we will apply these concepts to consider the case of control transitions.
Perceptual-Motor Calibration
Steering is a specific example of a broader set of actions that rely on visual information to guide movement (e.g., steering has been modeled as a reaching task; Kolekar, Mugge, & Abbink, 2018; and there is also evidence linking steering ability with manually tracing paths; Raw, Kountouriotis, Mon-Williams, & Wilkie, 2012). In the fields of experimental psychology and vision science, such visually guided actions have historically been modeled using mathematically specifiable control laws that translate perceptual cues more or less directly into movement commands, for example braking (Fajen, 2007, 2008; Lee, 1976) or steering (Fajen & Warren, 2003; Land & Lee, 1994; Salvucci & Gray, 2004; Wilkie & Wann, 2002). A number of perceptual cues are made optically available to humans by their environment (such as optic flow, Gibson, 1958; or optic expansion, Lee, 1976). A driver can learn relationships between available perceptual variables and the control states that produce desired task performance (Fajen, 2005). The learned relationship can be referred to as a perceptual-motor mapping. However, the exact mechanisms of the sensorimotor learning underpinning skilled actions is often unclear. Motor learning is routinely described as a set of internal models that support predictive feedforward control (Wolpert, Diedrichsen, & Flanagan, 2011), yet the presence of internal models is also contested in some online control accounts of visually guided action (Zhao & Warren, 2015). Irrespective of the precise mechanisms underlying perceptual-motor mapping and learning, adequate perceptual-motor mappings need to be established, maintained, and updated over time and across different conditions (see Lappi & Mole, 2018, for a more detailed discussion of the role of internal models in steering control).
The mappings from perceptual cues to motor actions will vary depending on environmental conditions, vehicle dynamics, and driver experience. For example, if someone tries to drive a new vehicle, the steering characteristics (e.g., wheel sensitivity) is likely to differ from their prior experience: Given identical perceptual stimuli, a different motor response will be needed, and so a new mapping needs to be acquired. If the new vehicle is more responsive (e.g., it has power-assisted steering, whereas the previous vehicle did not), the driver risks excessive initial steering inputs and/or overcorrecting for errors because responses will reflect an incorrect mapping. Analogous changes will occur even during a continuous drive of the same vehicle. Steering dynamics will alter across time due to changes in the vehicle (e.g., increased speed, reduced fuel load, wear in tire tread), the environment (change in surface texture, e.g., gravel vs. Tarmac, different weather conditions), and the driver (e.g., muscle fatigue). During such periods, a driver needs to adapt to frequently changing action capabilities and remain well attuned to the environment (Fajen, 2005; Shadmehr, Smith, & Krakauer, 2010).
We can define, at the broadest level, perceptual-motor calibration as maintaining appropriately scaled movements when conditions change (Fajen, 2005). We will follow recent reviews of how humans scale movements under changing conditions (Brand & de Oliveira, 2017; Redding, Rossetti, & Wallace, 2005; van Andel et al., 2017) and refer to this as perceptual-motor calibration, but note that calibration is a type of sensorimotor adaptation (Krakauer & Mazzoni, 2011; Wolpert et al., 2011), so often either term is used to describe changes very similar in nature (e.g., Bourgeois & Coello, 2012; Benson, Anguera, & Seidler, 2011). For readers unfamiliar with the perceptual-motor control literature, it is worth being explicit about the relationship between calibration and adaptation (Tresilian, 2012). Adaptation is broader in scope and refers to adjusting existing skills in new circumstances to maintain levels of performance. Calibration refers to a specific case of adaptation involving adjusting existing or learning new perceptual-motor mappings. All (re)calibration is adaptation, but not all adaptation is calibration, for example one can stiffen muscles to resist uncertain forces without updating perceptual-motor mappings. Both adaptation and calibration are distinct from acquisition, which involves learning a new skill (Tresilian, 2012). All are forms of learning.
The process of maintaining perceptual-motor mappings attuned to the environment can be examined at different levels of the sensorimotor system, from neuromuscular changes (Franklin, Wolpert, & Franklin, 2017), to scaling movements such as swinging a baseball bat (Scott & Gray, 2010), braking (Fajen, 2007) or reaching-to-grasp (Coats, Bingham, & Mon-Williams, 2007), to end-point accuracy of complex tasks with multiple coordinated submovements such as during driving simulator adaptation experiments (McGehee, Lee, Rizzo, Dawson, & Bateman, 2004; Ronen & Yair, 2013; Sahami & Sayed, 2010). Given the role of calibration in supporting successful action, it is important to consider how AVs might affect the acquisition and maintenance of well calibrated steering responses. We examine the likely impact of automation on calibration in Automated Driving Will Break the Perceptual-Motor Control Loop after we have completed outlining the fundamentals of the operational control loop underpinning steering control behaviors.
Gaze and Steering Coordination
There appears to be a growing consensus within the recent perceptual-motor steering control literature that successful steering naturally relies on close coordination with gaze behaviors (Chattington et al., 2007; Hollands, Patla, & Vickers, 2002; Jahn, Kalla, Karg, Strupp, & Brandt, 2006; Land, 1992, 1998; Land & Hayhoe, 2001; Land & Lee, 1994; Land & Tatler, 2001; Lappi, 2014; Lappi & Mole, 2018; Lehtonen et al., 2014; Mars, 2008a; Matthis, Yates, & Hayhoe, 2018; Vansteenkiste et al., 2014; Wilkie et al., 2008, 2010). Gaze behaviors that are tightly coupled with vehicle control maneuvers lead to fairly stereotypical behaviors (albeit with some interindividual variation) during routine driving (for illustrative examples, see Lappi, Rinkkala, & Pekkanen, 2017). It seems that to steer smoothly, drivers usually employ guiding fixations (GF): fixations directed about 1–2 s ahead (Land, 1992; Land & Lee, 1994; Lappi et al., 2013; Lehtonen et al., 2014). However, GFs are sometimes interleaved with rarer fixations even further ahead—referred to as lookahead fixations (LAFs; Lehtonen et al., 2013). This gaze polling behavior of alternating GF/LAF (Wilkie et al., 2008) has been observed in both laboratory and real-world tasks (Lappi et al., 2017; Lehtonen et al., 2014; Wilkie et al., 2008).
It has been hypothesized that these two classes of fixation have different functional roles. GFs seem to be useful for path modification when responding to changes determined by the upcoming road curvature, whereas LAFs are more useful for route planning decisions further ahead in both time and space (Lehtonen et al., 2013, 2014; Mars & Navarro, 2012; Mennie, Hayhoe, & Sullivan, 2007; Pelz & Canosa, 2001; Wilkie et al., 2008; see Lappi & Mole, 2018, for a review of the relevant evidence).
During routine steering, gaze patterns are very active, with a move-dwell-move pattern occurring 2 to 3 times per second (Lappi et al., 2017; Wilkie et al., 2010). It appears that gaze behaviors by themselves are important, but the interplay between steering and gaze control is more nuanced than simply needing to look ahead to observe the scene features relevant for determining the upcoming steering requirements. Gaze patterns change when the need for active vehicular control is removed: even when the viewed scene is identical, drivers look further ahead and make more LAFs than when they are no longer required to steer (Mackenzie & Harris, 2015; Mars & Navarro, 2012).
Complementing the visual input, the (forward) orientation of the head and eyes itself provides proprioceptive gaze direction information (from the muscles controlling the head and eyes) that is a useful input for steering control over and above the visual pattern on the retina (Authié, Hilt, Berthoz, & Bennequin, 2015; Wilkie & Wann, 2003a; Wilson, Stephenson, Chattington, & Marple-Horvat, 2007). If drivers are looking where they wish to go, the direction and magnitude of gaze relative to the current direction of locomotion in part signals the steering required to pass through the point of fixation. Steering can be biased by preventing normal eye movements (Robertshaw & Wilkie, 2008) or requiring drivers to fixate a point to the side of the path (Jahn et al., 2006; Kountouriotis, Floyd, Gardner, Merat, & Wilkie, 2012; Mars, 2008a; Readinger, Chatziastros, Cunningham, Bülthoff, & Cutting, 2002). It seems, therefore, that active gaze control is also critically important for maintaining locomotor perceptual-motor mappings. These eye and head movements actively shape the samples from the sensory array (including but not limited to the retina) that the brain receives as input (Ahissar & Assa, 2016). Therefore, a notable aspect of the coupling between steering and gaze is that it appears to be bidirectional: gaze influences steering, and steering influences gaze.
The evidence highlighted so far in this subsection demonstrates that gaze and steering coordination can be well captured within the framework of a perception-action loop. Neither behavior in isolation (gaze or steering) wholly determines the other: rather, where a driver will look (and consequently what is sampled retinally and extraretinally) depends on the current steering intentions, but the current steering is in turn influenced by where the driver is looking. It is worth mentioning that because these perception-action loops are modeled on successful human steering behaviors in laboratory steering tasks, they tend to describe well-calibrated behaviors (see e.g., Boer, 2016; Mars & Chevral, 2017; Salvucci & Gray, 2004; Wilkie et al., 2008), however the way that the human has become calibrated, and how calibration can adapt in more dynamically complex and labile environments, tend not to be explicitly addressed.
One cannot have synergy between steering and gaze without appropriately attuned bidirectional perceptual-motor mappings. The section entitled How Will AVs Affect Gaze and Steering Coordination? examines the potential impact on control transitions.
An Operational Control Loop That Reflects the Perceptual-Motor Demands of Steering
The operational control level outlined in Figure 1 can be expanded to highlight the components of perceptual-motor steering control that are most relevant to control transitions (Figure 2). Successful steering is here depicted as relying on a frequently updated perceptual-motor loop, where the perceptual inputs used to inform steering are supplied by gaze and steering control acting synergistically. The role of calibration is included as a critical property of this loop, attuning gaze and steering control outputs to the current vehicle and environmental conditions.
A schematic of the perceptual-motor control loop supporting efficient and safe driving. Perceptual variables (examples given: retinal flow refers to the continuous transformation of elements on the retina; gaze angle refers to proprioceptive information about direction of gaze relative to the locomotor heading; and force feedback refers to contact forces on the wheel) are mapped via calibration onto gaze and steering coordination: steering control (determining how rapidly, and by how much, to turn the steering wheel to produce the desired wheel angle, Benderius & Markkula, 2014) and gaze control (e.g., saccading to, then tracking, a waypoint on the future path then tracking this waypoint, Wann & Land, 2000; Wilkie & Wann, 2002). Mappings are frequently updated so the driver remains well calibrated to the environment. Note the bidirectional information flow between gaze control and steering control. Steering and gaze outputs, mediated through vehicle dynamics, produce a change in the world. The large black arrow represents the general direction of information flow in the action and perception loop, though not all perceptual information has to be mediated by the vehicle and the environment: perceptual variables can be obtained directly from driver outputs (e.g., predictions of steering commands, Markkula et al., 2018), from the vehicle (e.g., haptics), or from environmental changes (e.g., retinal flow). Boxes with dashed outlines represent distinct entities, and boxes with solid outlines represent key subcomponents for this manuscript. Bubbles give illustrative examples of the nature of processing within each subcomponent. Gray boxes are mechanisms that are the focus of this article.
When attempting to capture the nature of complex human behaviors (such as driving) it is essential to be explicit about the level of description being used. Figure 2 is a schematic representation, which does not attempt to describe an implemented steering model, nor does it make concrete proposals about the physiological or perceptual-cognitive nature of the key mechanisms underlying calibration and gaze and steering coordination.
This is because these phenomena are partly determined by the character of the subcomponents of the perceptual-motor loop: perceptual variables, steering control, and gaze control. Ongoing research continues to improve our understanding of these components, and there are a number of steering models proposing different candidate perceptual variables and alternative mechanisms for how candidate variables translate to vehicle control (e.g., Salvucci & Gray, 2004; Wilkie et al., 2008; also see Lappi, 2014, for a review). However, the level of description used in Figure 2 is sufficient for assessing the impact of these components upon smooth and safe control transitions.
One characteristic of the loop shown in Figure 2 is that for steering actions to remain well calibrated with respect to the environment, the driver will need to operate at a sufficiently high frequency. In the real world, drivers are able to look away from the road (or have the scene occluded) for around 1 to 2 s intermittently without a major impact upon performance (Horrey & Wickens, 2007; Pekkanen et al., 2018; Senders, Kristofferson, Levison, Dietrich, & Ward, 1967). Even when a driver is on task, there will be intermittency in the gaze input due to gaze polling behaviors (Lappi et al., 2017). Also, recent modeling advances suggest that intermittent control could be a fundamental property of steering (Markkula et al., 2018). Nevertheless, while the perceptual-motor loop can handle intermittent and irregular inputs, the manageable time scales of such interruptions appear to be in the order of seconds rather than minutes (Johns & Cole, 2015; Pekkanen, Lappi, Itkonen, & Summala, 2017). During periods of automated driving, the perceptual-motor loop is likely to be interrupted for considerably longer timescales, in the order of many tens of minutes, up to a number of hours. The next section will explore how the perceptual-motor control loop components identified so far may be disrupted during automated driving for prolonged periods.
Automated Driving Will Break the Perceptual-Motor Control Loop
During automated driving, the perceptual-motor loop depicted in Figure 2 will be disrupted. Specifically, the requirement for the human driver to produce steering control commands is removed because the AV takes control over steering (Figure 3). This change effectively breaks the perceptual-motor loop, which may have an impact on perceptual-motor calibration and also have consequences for the other behaviors normally exhibited during driving (e.g., eye-movement patterns).
A schematic of how automated driving may disrupt the perceptual-motor loops underpinning efficient and safe driving. AV control replaces driver actions. The operation control loop is broken: information only flows from perception to action, not from action to perception. Specifically, information flows to and from steering control are removed. The removal of steering control is likely to change gaze behavior, putting the utility of gaze-mediated perceptual variables in doubt. Without a perception-action loop, it is also unlikely that the driver will remain well calibrated to the environment.
How Will AVs Impact Upon Perceptual-Motor Calibration?
During manual control of steering, a driver remains calibrated to the vehicle dynamics and environmental conditions despite frequent changes to the mapping between motor action and resultant vehicle motion (see Perceptual-Motor Calibration). Most (if not all) current steering models implicitly assume that the driver is well calibrated (e.g., Boer, 2016; Mars & Chevrel, 2017; Salvucci & Gray, 2004; Wilkie et al., 2008). However, if a driver relinquishes control to an automated vehicle and then later takes back control (after some unspecified duration), some degree of miscalibration should be expected. This situation is analogous to classic recalibration paradigms (see Brand & de Oliveira, 2017, and Redding et al., 2005, for detailed reviews) where an individual is initially calibrated to baseline conditions (in our case, during a period of manual driving) with a subsequent disturbance whereby perceptual-motor mappings are altered (the automated driving period), followed by a rearrangement period (in the terminology used by Brand & de Oliveira, 2017) where perceptual-motor mappings are adjusted and reacquired (control transitions from automation back into manual control).
In this context the relevant question to ask about perceptual-motor calibration is how long the system remains well calibrated once the driver is no longer in active control of the vehicle. There is evidence that when perceptual feedback is removed the sensorimotor system quickly becomes inaccurate: for example proprioception accuracy has been shown to deteriorate within 1 min without visual feedback (Wann & Ibrahim, 1992), and when visual and kinesthetic motion feedback is removed driver’s actions can become inaccurate within a few seconds (Wallis, Chatziastros, & Bülthoff, 2002; Wallis, Chatziastros, Tresilian, & Tomasevic, 2007). These studies clearly demonstrate that perceptual-motor calibration can deteriorate rapidly without feedback, but a complete absence of feedback is the limit case, and it is unlikely to occur during automated driving. The driver will continue to receive positional feedback (e.g., from having the hands on the wheel) as well as visual feedback (when looking at the road ahead). However, even though such feedback may be available, the signals could be considered diminished compared with what is available when driving manually: drivers seem to be less likely to look at the road ahead during automated driving (see How Will AVs Affect Gaze and Steering Coordination?), and in L3/L4 automation they will not necessarily have their hands on the wheel (thus eliminating haptic/proprioceptive signals from their hands/arms).
Even if the perceptual-motor system is able to use some of the available information to prevent calibration drift (the decay/misalignment of perceptual-motor mappings), it is likely that during longer automation periods the environmental conditions will have altered (e.g., changes to the grip of road surface due to rain). In such cases, the driver is likely to need an acclimatization period to ensure recalibration. Research investigating recalibration tends to examine discrete movements over short trials, for example swinging a baseball bat that has increased mass (Scott & Gray, 2010) or decelerating a vehicle using a stronger brake (Fajen, 2007). Such studies show that participants are able to recalibrate fairly quickly (in the order of around 10 trials), even when feedback is restricted to 1s per trial (Fajen, 2007). However, these are repetitions of specific movements in controlled environments with consistent feedback—conditions that are likely to be favorable to rapid recalibration (Castro, Hadjiosif, Hemphill, & Smith, 2014; Huang, Haith, Mazzoni, & Krakauer, 2011). In the real world, drivers do not have the luxury of repeating a movement until it is optimal, rather they execute steering corrections of different magnitude in response to an ever-changing environment, and so rearrangement of steering may take much longer. Deborne, Gilles, and Kemeny (2012) showed that drivers on a simulated circular track could recalibrate (stabilize steering wheel angle) in response to a sudden increase in steering wheel self-aligning torque within a few seconds. However, the drivers in Deborne et al. (2012) were in active control the entire time, and there is evidence that drivers adapt to changes in steering torque particularly quickly (Russell et al., 2016; Toffin, Reymond, Kemeny, & Droulez, 2007). Therefore, although this evidence supports the idea that drivers can rapidly recalibrate to some changing conditions during active control, it may not be directly applicable to control transitions where drivers are taking over vehicle control after a period of disengagement when there may have been multiple changes to both the vehicle dynamics and environmental conditions.
An indirect way to examine the issue of a change in vehicle and environment is by measuring the behavior of drivers when they first use a driving simulator. Simulators differ from real-world driving in terms of vehicle dynamics and the available perceptual information (e.g., a fixed-base simulator will not provide vestibular feedback), so drivers usually need to have a period of time over which they adjust their perceptual-motor mappings. Simulator studies report that driver performance can take between 4 to 15 min to stabilize (i.e., stop noticeably improving). McGehee et al. (2004) report a stabilization time of 4 min (for wheel reversals), whereas Ronen and Yair (2013) report longer timescales that vary from 6.5 to 15 min depending on road type. Sahami and Sayed () report considerable interindividual variation in rate of stabilization, averaging ~7.5 min. It is reasonable to expect that the recalibration period during real-world control transitions would be shorter than these estimates because changes within the same vehicle are likely to be relatively small compared with the difference between a real vehicle and first-time use of a simulator (presuming the conditions during takeover are close to the preexisting perceptual-motor mappings). To the best of our knowledge, there is only one study to date that has looked in detail at motor recalibration in real cars during transitions out of automation. Russell et al. (2016) used a paradigm similar to classic motor control studies, where participants first experienced multiple handovers of lateral control with one set of vehicle dynamics, then vehicle dynamics were altered (the steering was made more sensitive or the steering wheel self-aligning torque increased). An increase in steering sensitivity initially led to jerkier steering as drivers overshot the required wheel angle. Within 10 trials (more than 1 min) on a controlled lane change task (each trial contained approximately 15 s of manual lateral control), drivers were able to bring steering back to levels of smoothness comparable to the baseline trials. The results of Russell et al. (2016) establish the existence, at least for some aspects of vehicle control (there was little effect of changing steering torque), that there is a critical period after takeover where drivers may be miscalibrated if conditions have changed from when they were last in manual control. The time needed to recalibrate in Russell et al. (2016) is considerably quicker than the 4- to 15-min timescales reported in the simulator studies discussed previously (though it is possible that the repeated controlled conditions favor rapid recalibration). Yet a timescale of around 1 min is still considerably slower than the ~10 s exposure time extrapolated from the perceptual-motor literature. It seems, then, that although drivers may be able to take up control within a few seconds (as indicated by reaction times; see the section entitled Reaction Times When Responding During Transitions for further evaluation), they may be prone to making miscalibrated steering responses during early phases of the transition (such as rapid evasive maneuvers; Navarro, François, & Mars, 2016; Russell et al., 2016).
It is critical to improve our understanding of which behaviors could be supported during automated driving to minimize the decay of perceptual-motor mappings and reduce the time required to recalibrate. Unfortunately, the task demands placed on the human operator during automated driving is really at odds with the indications of the perceptual-motor control literature, which suggests that a crucial requirement of successful calibration is active motor control (often termed action exploration, Brand & de Oliveira, 2017). For example, Pelah and Barlow (1996) report that treadmill runners recalibrate their relationship between locomotor speed and optic flow, so that after treadmill running there is a period of illusory accelerated self-motion that causes participants to walk more slowly than usual, however, the effect of recalibration was eliminated if participants were pushed on a wheelchair (so they had no active control, Pelah & Barlow, 1996). Furthermore, research suggests that the state of calibration is often unavailable for self-report, with participants able to recalibrate to a change in brake strength despite being unable to accurately detect the brake strength change (Fajen, 2007). Or, in contrast, participants are made aware of the manipulation (and were instructed to ignore it) but nevertheless were unable to resist recalibration (Mon-Williams & Bingham, 2007, Exp. 4; see also Benson et al., 2011). This evidence suggests that simply informing drivers of the need to recalibrate may not be sufficient (although it could help drivers reduce error if there are explicit strategies available, Benson et al., 2011; Taylor, Krakauer, & Ivry, 2014). This point would seem to be supported by the evidence of Russell et al. (2016), who informed participants about an upcoming change in steering but drivers nevertheless required more than 1 min to recalibrate. The extent to which individuals can recalibrate without “online” motor control (i.e., solely from “passive” perceptual signals) remains unclear, and there may well be mismatch situations where a driver is able to subjectively report changes in conditions during the automated drive yet their perceptual-motor calibration does not shift appropriately.
On the basis of the research reviewed in this section we identify three key open research questions pertinent to the design of safe AV systems: (RQ1) How long does a well-calibrated human driver’s mapping persist without active steering control? (RQ2) What factors determine how quickly a driver can recalibrate to new conditions after a control transition? (RQ3) How can we help drivers remain well-calibrated during automated driving?
How Will AVs Affect Gaze and Steering Coordination?
In the section entitled Gaze and Steering Coordination, we introduced the extensive literature suggesting that the coordination of gaze and steering is fundamental to effective steering behavior. This point is reinforced by real-world studies of the manual control of driving, demonstrating that even coarse indicators of gaze behavior (on- or off-road glances) can be reliable indicators of collision risk (Victor et al., 2015; Seppelt et al., 2017). It seems plausible, therefore, that steering actions during control transitions will be influenced by where drivers look, both in the seconds before takeover (which will affect the nature of the available perceptual information) and during the initial period of the control transition.
It is clear from existing empirical data that gaze behaviors during automated driving should be expected to be markedly different than during manual driving. Gaze patterns are characteristically less concentrated during automation than during manual driving: eye metric positions are more variable (Damböck, Weissgerber, Kienle, & Bengler, 2013; Mackenzie & Harris, 2015; Shen & Neyans, 2017), gaze dispersion metrics are higher (Louw & Merat, 2017), and less cumulative time is spent looking toward the road ahead (Carsten, Lai, Barnard, Jamson, & Merat, 2012; Feldhütter, Gold, Schneider, & Bengler, 2016; Jamson, Merat, Carsten, & Lai, 2013; Louw, Kountouriotis, Carsten, & Merat, 2015; Louw, Madigan, Carsten, & Merat, 2016).
Louw et al. (2016) linked changes in gaze behavior during automation to detrimental road safety outcomes, showing that drivers who looked the least often to the road ahead were most likely to crash (i.e., did not execute an evasive maneuver quickly enough; see also Zeeb, Buchner, & Schrauf, 2015). It may seem obvious that drivers who do not look at what is in front of them are unable to respond to events that they did not see. But the erratic patterns of sampling often observed during automation (Louw & Merat, 2017) may affect steering control by disrupting the coordination of gaze and steering (independent of the opportunity to sample useful visual information). Looking away from the direction of travel for long periods may lead to subsequent steering control being biased by where the driver was previously looking (e.g., steering response are coupled to gaze direction with a 1 to 2 s lag; Land & Lee, 1994; Wilkie & Wann, 2003b; see Gaze and Steering Coordination). Alternatively, if drivers have decoupled gaze from steering during automation (so that gaze direction is no longer informing steering) then recoupling will need to take place after takeover: this could lead to a period whereby gaze direction does not appropriately inform steering. Evidence during manual steering suggests that drivers can find it difficult to decouple gaze and steering and may unintentionally steer where they look (Kountouriotis et al., 2012; Robertshaw & Wilkie, 2008). It is currently unknown whether gaze direction information picked up during automation will carry over to manual steering after a handover, but if carryover is possible, then there hypothetically exists a critical period immediately after handover where a driver could be influenced by where they had previously been looking (during automation), not where they are currently looking (during manual control). Whether carryover exists—and the timescales of any effect—will have important implications for where a driver should be looking before a transition; these factors need to be empirically tested.
Ensuring the driver is on task (i.e., monitoring how the automated vehicle is steering) and looking to the road ahead should aid steering during transitions, but it may also be the case that the gaze patterns produced during automation differ compared with manual driving (Mars & Navarro, 2012; Navarro et al., 2016). The bidirectionality of gaze and steering coordination suggests that executing steering commands provides a driver with valuable information (e.g., efference copy, Franklin & Wolpert, 2011) of the likely consequences of current steering actions (Blaauw, Godthelp, & Milgram, 1984; Mars & Navarro, 2012; Nash, Cole, & Bigler, 2016; Markkula et al., 2018), informing drivers of where they need to look in order to obtain perceptual inputs for the next motor command controlling steering. In contrast, during periods of automation, drivers have to decide where to look based on the control outputs from the AV rather than their own sensorimotor system. Mars and Navarro (2012) showed that during automated driving, gaze patterns changed compared with when the driver was in full manual control (guiding fixations were executed less often, and look-ahead fixations more often). Navarro et al. (2016) demonstrated that these differences are amplified in critical scenarios: during obstacle avoidance, drivers spent less time sampling the region where GFs normally occur (compared with manual driving). Navarro et al. (2016) interpreted this result as a reorganization of the requirements for perceptual inputs: Since drivers are not actively controlling their trajectory (semiautomated control was used where steering was automated but drivers remained in control of speed), they made fewer GFs, and prioritized perceptual inputs from farther ahead in the scene. Interestingly, in Navarro et al. (2016) the manual drivers evaded obstacles less aggressively (had lower steering amplitudes and accelerations) than recently transitioned drivers, despite both groups looking to the obstacle at similar times (so had a similar preview time in which to prepare the evasive maneuver). Unstable steering posttransition may be directly linked to gaze reorganization that takes place during automation: Drivers make fewer GFs, so do not have available the necessary perceptual signals for smooth steering, and it takes time to reestablish successful coordination. However, it could be the case that even with optimal gaze sampling patterns, unstable steering occurs due to a decay in perceptual-motor mappings (resulting in poor calibration; see How Will AVs Impact Upon Perceptual-Motor Calibration?). When a driver is poorly calibrated, erratic steering is expected because there will be a mismatch between intended and executed actions (causing positional error that needs corrected for). Of course, it is also plausible that the unstable steering in Navarro et al. (2016) could be due to a combination of gaze and steering coordination and calibration mechanisms: The reorganization of gaze may remove perceptual inputs that would be available in manual driving, leading to a decay of perceptual-motor mappings that in turn cause unstable steering. At present, the relative impact of gaze and steering coordination and calibration on transitions are unknown, but this understanding will be invaluable for the design of safe AV systems.
In this section, we have identified three further research questions pertinent to the design of safe AV systems: (RQ4) How does gaze behavior change during automation? (RQ5) Do changes in gaze during automation affect steering control upon takeover? And if so, by what mechanism? (RQ6) How can we help drivers to maintain gaze patterns during automated driving that facilitate timely and well calibrated reengagement?
Current Evidence: Human Responses During Transitions
In this section, we assess the extent to which the AV literature maps onto the perceptual-motor control loop as outlined in the previous sections and the operational level of Figure 1. In particular, we review empirical studies that examine transitions out of automated steering (for complementary reviews, see de Winter, Happee, Martens, & Stanton, 2014; Lu et al., 2016), and focus on whether the research addresses concepts related to gaze and steering coordination and perceptual-motor calibration (see Table A2 for the references supporting this section, along with how they relate to the perceptual-motor behaviors discussed in this article). Among the papers examining transitions, the perceptual-motor loop has been examined using a variety of measures, ranging from coarse yet critical real-world outcomes such as crashes (Dixit et al., 2016; Louw et al., 2016; Strand, Nilsson, Karlsson, & Nilsson, 2014; van den Beukel & van der Voort, 2013, 2017; Wan & Wu, 2018) through to detailed measures of changes in gaze and/or steering behaviors (DinparastDjadid et al., 2017; Naujoks et al., 2017; Navarro et al., 2016; Petermeijer, Cieler, & de Winter, 2017; Vogelpohl et al., 2018). By far the most commonly discussed measure of transition performance, however, attempts to balance sensitivity with applicability to real-world scenarios. The reaction time (RT; how quickly individuals take control of their vehicles) has been used for assigning AV safety boundaries, which has led to RTs becoming the predominant way of evaluating drivers’ actions (Liu & Green, 2017; Eriksson & Stanton, 2017a; see Zhang, de Winter, Varotto, Happee, & Martens, 2019 for a meta-analysis). The use of RTs in the literature will consequently be the primary focus of this section, followed by an evaluation of the additional measures used in this literature.
Reaction Times When Responding During Transitions
RTs are typically recorded from the start of a takeover request (Blommer et al., 2017; Dogan et al., 2017; Eriksson, Banks, & Stanton, 2017; Eriksson & Stanton, 2017a; Feldhütter et al., 2016; Gold, Damböck, Lorenz, & Bengler, 2013; Körber, Gold, Lechner, & Bengler, 2016; Körber, Weißgerber, Kalb, Blaschke, & Farid, 2015; Lorenz, Kerschbaum, & Schumann, 2014; Melcher, Rauh, Diederichs, Widlroither, & Bauer, 2015; Naujoks et al., 2017; Naujoks, Mai, & Neukum, 2014; Naujoks, Purucker, Neukum, Wolter, & Steiger, 2015; Payre, Cestac, Dang, Vienne, & Delhomme, 2017; Petermeijer, Bazilinskyy, Bengler, & de Winter, 2017; Politis, Brewster, & Pollick, 2017; Radlmayr, Gold, Lorenz, Farid, & Bengler, 2014; Telpaz, Rhindress, Zelman, & Tsimhoni, 2015; van der Meulen, Janssen, & Kun, 2016; Vogelpohl et al., 2018; Wan & Wu, 2018; Zeeb, Buchner, & Schrauf, 2015) up to the point when the AV system becomes deactivated by the user (most commonly by the execution of a driving action: movement of the wheel or pedals, or a button press). Sometimes a takeover request is not present, so some studies identify RTs as starting from the moment that the event which precipitates the handover is initiated (e.g., a parked car becomes visible or a crosswind begins; Johns, Mok, Talamonti, Sibi, & Ju, 2017; Larsson, Kircher, Hultgren, & Andersson, 2014; Louw, Markkula, Boer, Madigan, Carsten, & Merat, 2017; Shen & Neyens, 2017; Strand et al., 2014). The assumption underlying the use of RTs is that an early response (shorter RT) will result in safer steering control, and therefore RTs act as a useful proxy measure of steering coordination. This logic aligns with sequentially stepping through the perceptual-motor control loop (the driver as depicted in the red box of Figure 3): if a driver is quicker to sample and process perceptual inputs, then they should be faster at selecting an appropriate action, resulting in a motor output that is safe because there is sufficient time to execute it smoothly and accurately (Benderius & Markkula, 2014). At the limits of the action time window, this assumption seems uncontroversial (slow RTs could cause braking that occurs so late that a collision is unavoidable), and there are suggestions that later steering responses will be more aggressive and jerky (Hoc et al., 2006; Navarro et al., 2016). Indeed, transition scenarios that elicit earlier responses are associated with less aggressive steering movements, causing smoother and less variable steering trajectories compared with scenarios that elicit delayed responses (Kircher, Larsson, & Hultgren, 2014; Louw et al., 2017; Louw, Merat, et al., 2015; Madigan, Louw, & Merat, 2018; Mok et al., 2015; Petermeijer, Cieler, et al., 2017; Politis et al., 2017; Shen & Neyens, 2017; Zeeb et al., 2016). However, as we shall discuss later, there are limits to applying these assumptions to all transition scenarios, particularly when the RT window is less critical.
As well as using RTs as a measure of steering response, they can also be used to measure gaze response. Rather than using the first steering response as the RT endpoint, the first glance can be used (either looking away from the secondary task or toward a predefined region of interest such as the car windscreen; Feldhütter et al., 2016; Gold et al., 2013; Kerschbaum, Lorenz, & Bengler, 2014; Lorenz et al., 2014; Vogelpohl et al., 2018; Zeeb et al., 2015, 2016; Zeeb, Härtel, Buchner, & Schrauf, 2017). RTs for eye movements are in some respects a purer measure of perceptual-motor performance than steering RTs: eye movements have a very low latency (Leigh & Zee, 2015) and will be near the lower bound of physiological responses to visual inputs relevant to steering, whereas steering responses may only be executed after the driver has decided which type of action to take (and therefore these measures may sometimes incorporate fairly high level decision making). However, the relevance of gaze RTs for driving safety depends on whether steering behavior can be inferred from early or late responses. In the literature (covered in Gaze and Steering Coordination), eye movements lead steering responses by ~1s in highly predictable conditions, such as curve negotiation (Land & Lee, 1994; Land & Tatler, 2001; Lappi et al., 2013; Lehtonen et al., 2014). In transition experiments, the lag between gaze and steering RTs are of a similar order of magnitude, ranging from approximately 1 to 2.5 s (Feldhütter et al., 2016; Gold et al., 2013; Kerschbaum et al., 2014; Lorenz et al., 2014; Vogelpohl et al., 2018; Zeeb et al., 2015, 2016, 2017). This similarity may suggest that gaze-steering RT lag times in transitions capture at least some aspects of the nature of gaze and steering coordination observed in manual steering. However, to measure the reliably of RTs, it is usually necessary to repeat the same task multiple times in order to derive an estimate of central tendency. This limitation in experimental design can make takeover events much more predictable than would be the case in real driving, possibly leading to artificially low RTs. It has also been well documented that subtly different gaze locations can lead to very different mechanistic explanations when considering gaze and steering coordination (Lappi, 2014). The risk of wrongfully inferring psychological mechanisms increases as the area of interest increases (Orquin & Holmqvist, 2017), so the wide catchment areas used for measuring gaze RTs (e.g., the first fixation to any area of the windscreen; Eriksson et al., 2019; Gold et al., 2013; Kerschbaum et al., 2014; Lorenz et al., 2014; Zeeb et al., 2015, 2016, 2017) risk inferential errors, for example assuming that drivers are sampling road information when they may be looking at an object close to the road. Louw et al. (2016) used an area of interest approach (by dividing the windscreen into a central 6° circle and four surrounded segments) to examine the fixation placement at 200 ms after takeover and found that only 35–55% (depending on condition) of fixations were directed to the road center, suggesting that gaze and steering coordination is only partially captured by typical RT measures.
It seems then that although RTs are useful for identifying safety boundaries, these sorts of temporal indicators (of gaze or steering) are restricted to time-stamping processes within the perceptual motor control loop (Figure 2). Gaze-on-road RT measures provide an estimate of when the process of sampling task-relevant perceptual inputs may have started, and turn-initiation RTs give an estimate of when the decision-making process is sufficiently advanced to trigger an initial motor output. However, it seems that there are some conditions (e.g., visual distractions during nonurgent control transitions) that have little influence over the time it takes to return hands to the wheel or eyes to the road, yet they will affect the nature of the steering response (Zeeb et al., 2016). Consequently, some authors have taken pains to draw a distinction between the time taken to return hands to the wheel or the eyes to the road ahead and the subsequent steering “quality” (smoothness and variability) of steering control (Louw et al., 2017; Vogelpohl et al., 2018; Zeeb et al., 2016, 2017). The next section briefly reviews the current approaches in the transition literature to reporting steering quality.
Beyond Reaction Times: Steering Quality During Transitions
Transition researchers have adopted a variety of approaches when reporting steering quality (see Table A3). The most frequently used metrics capture behavioral extremes (prevalent examples are maximum lateral error or maximum lateral acceleration), but these necessarily capture only a snapshot of the driver’s trajectory, which may or may not have ramifications for driver safety. For example, high lateral acceleration may occur during short and sharp obstacle avoidance maneuvers that are effective and safe. Some researchers have attempted to assess the quality of steering across whole trajectories by taking (most commonly) the standard deviation of lane position (SDLP). Variability of vehicle position is often used to infer how smoothly a driver is controlling the vehicle: Greater variability is associated with producing many steering corrections. Supporting the use of this metric, some papers have reported a spike in SDLP in the first few seconds after takeover (presumably when drivers are making the most corrections) that takes a further few seconds to dissipate (Dogan et al., 2017; Naujoks et al., 2017).
If these sorts of variability measures of vehicle behavior are considered in isolation, it can sometimes be difficult to classify good performance, or indeed infer what sort of control strategy underpins the trajectories taken. There is an implicit assumption that smoother trajectories are synonymous with better performance, however, fewer large steering corrections would usually be classified as smoother steering but could lead to large errors in lane positioning. It is often more informative, therefore, to couple vehicle position metrics with other metrics relating to the driver’s actions on the wheel. For example, Eriksson and Stanton (2017b) found little difference in lane position between manual and automation modes but found large differences in steering wheel angle variability.
With metrics from both steering actions and vehicle position, one can build a better picture of the driver’s control strategy (it is reassuring that recent studies have provided detailed plots of both steering wheel angle and vehicle position; see Table A3). For example, Louw et al. (2017) demonstrated that the steering wheel amplitude of an avoidance maneuver was related to the criticality of the near-collision situation. Considering both steering actions and vehicle position allows the evaluation of whether unstable steering is related to poor calibration (as in the case of Russell et al., 2016) or whether it is due to impaired coordination of gaze and steering (which may be the case in many of the papers cited in this review, especially for drivers looking away from the future path during takeover, for example when looking toward a displayed secondary task). Similarly, jerky steering during the few seconds after takeover can be examined to see whether this results from successful compensatory steering (jerky steering that keeps vehicle position within acceptable limits, Donges, 1978), a behavior that might be encouraged (e.g., by training the driver to look in the near region ahead of the vehicle). However, if jerky steering does not lead to good road positioning, perhaps different gaze behavior patterns that enable smoother trajectories should be encouraged. Addressing these issues around perceptual motor control requires a common framework among researchers, so the next section works to situate the existing evidence in the perceptual-motor control framework (Figure 3).
Comparing Existing Transition Evidence to the Perceptual-Motor Control Loop
This final section of the review of the transition literature highlights the usefulness of adopting the conceptual framework presented in Figure 3, both with helping to interpret existing transition phenomena and for the future of autonomous vehicle design.
One situation where the processes underpinning perceptual-motor calibration could offer insight is when RTs apparently fail to capture steering quality (e.g., where RTs are similar across conditions but steering quality differs; see Reaction Times When Responding During Transitions). Miscalibration does not necessarily lead to longer RTs (Benson et al., 2011). Furthermore, longer RTs might result from explicit corrective strategies that can improve performance by compensating for inaccurate calibration (Benson et al., 2011; Taylor et al., 2014). Therefore, while similar RTs for different transition conditions indicate that steering movements were initiated equally quickly, it would be wrong to assume that the driver is equally well calibrated in these cases. Transitions when the driver is poorly calibrated may be identified by examining steering quality in the first few seconds of takeover (e.g., Russell et al., 2016).
Perceptual-motor calibration may explain differences in steering quality across conditions where changes in perceptual stimuli and/or time spent without active control may have altered/decayed perceptual-motor mappings. However, steering quality can vary even when the perceptual stimuli and time spent without active control have been kept constant (so the mappings have not changed) and only the amount of time between the takeover request and manual control recovery (takeover lead time) has varied. Shorter takeover lead times are more likely to result in variable and unsafe steering behaviors than longer takeover request lead times (e.g., Gold et al., 2013; Mok et al., 2015; Wan & Wu, 2018). The difference could be partly due to changes in gaze and steering coordination: Longer lead times allow the driver to establish useful gaze behaviors (e.g., tracking a point where the driver wishes to go) that provide the necessary perceptual inputs to feed into safe and smooth steering behaviors. Examining how gaze relates to the driver’s future trajectory across takeover request lead times may shed light on the requisite perceptual-motor behaviors for smooth steering after takeover.
Despite the potential issues with interpreting RTs, there are some fairly concrete findings emerging from comparing RTs across different scenarios and pooling the results (Zhang et al., 2019). Transition RTs seem to increase with traffic density (Feldhütter et al., 2016; Gold, Körber, Lechner, & Bengler, 2016; Happee, Gold, Radlmayr, Hergeth, & Bengler, 2017; Körber et al., 2016; Radlmayr et al., 2014), or the addition of a secondary task (Dogan et al., 2017; Payre et al., 2017; Merat et al., 2014), whereas RTs are quicker when the takeover is cued auditorily rather than visually (Naujoks et al., 2014; Politis et al., 2017; Walch, Lange, Baumann, & Weber, 2015) or when drivers have prior experience taking control from automated systems (Happee et al., 2017; Hergeth, Lorenz, & Krems, 2017; Larsson et al., 2014; Payre, Cestac, & Delhomme, 2016; Zeeb et al., 2016). When combined, these findings begin to illustrate key situations where takeover may be unsafe (e.g., a novice user of automation who is distracted in heavy traffic). This being the case, one might wonder why it is necessary to look beyond RTs and consider the detailed issues highlighted earlier. We would contend that a theoretical understanding of the mechanisms underlying driver behavior during transitions will be invaluable. It is tempting to believe that enough experiments recording RTs from a multitude of different scenarios would be sufficient to capture human responses during transitions with a degree of precision sufficient to inform safe AV design. However the danger is that ignoring the issues identified in this article will only ever lead to a partial understanding and the measured disruption will be qualitatively different from the actual changes in steering control. New types of AV-transition scenarios are bound to occur, and for systems to be safe, we need to be confident in predicting likely steering behaviors in conditions beyond those currently studied. It is simply impossible to test all possible scenarios; however a deeper understanding should lead to viable computational models (e.g., Markkula et al., 2018) that allow virtual testing across a much wider range of scenarios, extrapolating our understanding of these cases to find those where there is most risk. Ultimately, we need to identify effective solutions that aid drivers reengaging with their vehicles, with the aim being to identify what information the driver needs in order to reengage the perceptual-motor loop most effectively. The concepts of calibration and gaze and steering coordination will help to provide an operational (and theoretically grounded) method of examining the extent to which someone is in control and safe to drive. As shall be discussed in the next section, there will be benefits to having this fundamental understanding of the underpinning processes in order to push forward research applied to the field of transitions.
Conclusions and Future Directions
This article has outlined two key perceptual-motor processes: perceptual-motor calibration and gaze and steering coordination. These processes may be disrupted during automation with potentially detrimental effects on steering capability upon takeover, but they have hitherto received only a small amount of attention in current transition research (see Russell et al., 2016 for some promising advances in this area). For each process, three concrete research questions have been highlighted that are important for the design of safe AVs. These research questions map onto three categories: (a) breaking, (b) reengaging, or (c) maintaining the operational control loop (see Table 1).
Six Key Research Questions Emerging From Applying Perceptual-Motor Control Processes to Transitions out of Automation Driving
The answers to the research questions arranged in Table 1 will ultimately determine the practical significance of the perceptual-motor control issues raised in this article. The next step will be to understand how the uncovered perceptual-motor issues connect with existing theoretical frameworks as currently used by those investigating automated driving. Recently, Merat et al. (2018) have usefully defined driver engagement using three distinct stages: in-the-loop, where a driver is in physical control of the vehicle and monitoring their environment; on-the-loop, where a driver is not in physical control but is monitoring the environment; and out-of-the-loop, where the driver is not monitoring the environment (regardless of whether he or she is in physical control). These definitions are complementary yet distinct from the perceptual-motor control issues raised in this article. Merat et al.’s proposals aim to provide a shared framework to shape research questions. However, judicious applications of the definitions proposed by Merat et al. require a precise description of what processes physical control and monitoring actually consist of. We contend that such a description needs to start at the perceptual-motor level (Figure 2 and 3): perceptual-motor control is central to any transition scenario, and the framework presented here (Figures 2 and 3) provides researchers with a common starting point for interpreting steering behavior after takeover. Experiments addressing RQ1 through RQ6 (Table 1) could lead to operational definitions for concepts such as physical control and monitoring, allowing researchers to specify how far a driver is from being safely in-the-loop (Figure 2).
One of the benefits of improving our understanding of the perceptual-motor processes supporting steering control is the facilitation of technological solutions to support automated driving. It is currently difficult to propose concrete practical solutions to the issues raised in this article. However, we see clear opportunities for the use of technological advances to address RQ3 and RQ6 once answers to RQs 1, 2, 4, and 5 (Table 1) have been resolved. Two main obstacles for maintaining good calibration during transitions of control are the potentially rapid decay of perceptual-motor mappings (RQ1), and the need for active control in order to recalibrate (RQ2). These obstacles arise from transitions between 100% AV control to 100% human control, but there are intermediate AV systems often referred to as haptic shared control systems (for a comprehensive review, see Abbink et al., 2018). With these systems, the AV and human jointly control the vehicle, with the steering commands of the human being mediated by the AV system (Abbink, Cleij, Mulder, & van Paassen, 2012; Mulder, Abbink, & Boer, 2012). The benefit of such systems is that they are able to provide haptic feedback to the driver, potentially allowing sufficient active exploration to prevent decay of perceptual-motor mappings during automated driving (Mars, Deroo, & Hoc, 2014; Mars, Deroo, & Charron, 2014) and/or reduce the time needed for recalibration during progressive transitions of control (Guo et al., in press).
In the section entitled How Will AVs Affect Gaze and Steering Coordination? we examined the potential issues with altered gaze behavior during automation (RQ4) and considered the effects of disrupting coordination on manual control (RQ5). Head-up displays (HUD) offer the possibility of superimposing visual information over the visual scene (using augmented reality) that could encourage useful and appropriate gaze behaviors to assist coordination before and during a transition. In the literature, HUDs have typically been employed to increase the salience of symbolic information (such as roadside hazards, signs, or other cars; Eyraud et al., 2015; Halmaoui, Joulan, Hautière, Cord, & Brémond, 2014; Langlois & Soualmi, 2016; Rusch et al., 2013) in order to aid decision making, but these systems could be adapted to aid the reestablishment of coupled gaze and steering. For instance, Mars (2008a, 2008b) demonstrated that guiding gaze by means of a virtual target moving ahead (as a function of the changes in road curvature) improved steering stability during manual driving. If gaze patterns during AV control are found to have a major impact on resumption of control, then the use of a HUD that informs drivers where they need to look based on their current direction of travel, and other environmental conditions, could be a promising avenue for investigation. The benefits of these types of technology (shared-control systems or HUDs) can only be realized after we have accurate and detailed models of human perceptual-motor control behaviors during (transitions out of) automated driving. Shared control systems will need not only models of human sensorimotor coordination (Abbink et al., 2012; Abbink, Mulder, Van der Helm, Mulder, & Boer, 2011; Mars, Saleh, Chevrel, Claveau, & Lafay, 2011; Mulder et al., 2011; Saleh, Chevrel, Mars, Lafay, & Claveau, 2011), but also an appreciation of the calibration mechanisms to ensure that appropriate feedback is provided. Similarly, HUDs require a sophisticated mechanistic model of how steering and gaze are coordinated in order to appropriately direct gaze.
Our hope is that by addressing the RQs raised in Table 1—and considering the perceptual-motor control issues raised in this article alongside existing practices—the field of transition research may come closer to realizing the benefits of automated driving technologies and ensure the automotive future is as bright as has been promised.
Key Points
During successful steering control, driving is supported by a rapid perception-action loop that regularly updates perceptual-motor mappings to remain well calibrated to changing conditions; this perception-action loop tightly couples gaze and steering behaviors.
The perceptual-motor loop is likely to be disrupted during automated driving if perceptual-motor mappings are allowed to decay and gaze control is no longer coordinated with steering control.
Miscalibration and uncoordinated gaze and steering behaviors are expected to lead to unstable steering control during the initial period of steering after transitions out of automated driving.
Incorporating an understanding of perceptual-motor mechanisms into transition research will lead to an improved ability to address the issues that arise during transitions out of automated driving.
Footnotes
Appendix
Nature of Steering Quality Reported in the Beyond Reaction Times Section
Acknowledgements
CDM, OTG, GM, and RMW were funded by EPSRC UK grant EP/P017517/1. CDM was supported by travel grants from the Experimental Psychology Society and the University of Leeds Research Mobility Programme to visit FM and OL. FM was funded by Agence Nationale de la Recherche (AUTOCONDUCT project, grant ANR-16-CE22-0007-05). OL was supported by a postdoctoral grant from the Finnish Cultural Foundation (00150514), and a researcher mobility period in Leeds for OL was supported by the Academy of Finland (279905).
Callum D. Mole is currently a postdoctoral researcher examining transitions out of automated driving at the School of Psychology, University of Leeds. He received his PhD in experimental psychology from University of Leeds in 2016.
Otto Lappi is a docent in cognitive science and leads the TRUlab, at the University of Helsinki. He obtained his PhD from University of Helsinki in 2014.
Oscar Giles is currently a postdoctoral research fellow at the School of Psychology and Institute of Transport Studies, University of Leeds. He obtained his PhD from University of Leeds in 2017.
Gustav Markkula is an associate professor in the Human Factors and Safety research group at the Institute for Transport Studies, University of Leeds. He obtained his PhD from Chalmers University of Technology in 2015.
Franck Mars is a CNRS researcher and leads the PACCE (human factors) group at the Laboratoire des Sciences du Numérique de Nantes (LS2N), Nantes. He received his PhD in Neuroscience from Université de la Méditerranée in 2001.
Richard M. Wilkie is professor of cognitive psychology at the University of Leeds, leading the Perception-Action-Cognition research group. He obtained a PhD from the University of Reading in 2003.