Analyzing Driver Strategy for Scheduled Takeovers Under the Effect of Multi-stage Takeover Request Design

Abstract

For scheduled takeover events in conditional driving automation, a longer lead time of takeover request (ToR) up to 60 seconds was recommended for drivers’ better situation awareness and subjective evaluations. However, the long ToR lead time could impair drivers’ estimation of the urgency of situation, which potentially results in risky takeover behavior. Four multi-stage ToR designs varying in warning modality that communicate the urgency within a long lead time were thus proposed to mitigate the negative effects. The present study involved a driving simulator experiment with 32 participants performing 12 takeover tasks for exiting a freeway with the aim to analyze driver strategy for scheduled takeover events and evaluate the multi-stage ToR designs. Eight driver strategies for attention management and takeover were identified. Using multi-stage ToR designs advanced drivers’ takeover actions by decreasing the likelihood of alternating attention between non-driving activities and road and reducing the duration of takeover preparation.

Keywords

conditional driving automation scheduled takeovers takeover request takeover strategy attention management

Introduction

Conditionally automated vehicles (AVs), known as the third level of driving automation, are designed to operate safely under certain conditions allowing drivers to turn their attention away from the road for an extended period of time. A common approach to defining the operational needs and capability boundaries of an automated driving system (ADS) is referred to as Operational Design Domain (ODD), which is a description of specific driving conditions (e.g., roadway features, speed range, time of day, and weather) that the ADS is designed to operate adequately (NHTSA, 2017). When a conditionally AV can predict it is approaching the boundary of its ODD, the driver will need to intervene and resume control upon a takeover request (ToR) initiated by system. The Highway Pilot feature, for instance, enables a car to drive itself in designated areas, but it will require the driver to take over control before the car exits a freeway and drive on roads that the Highway Pilot is not compatible with (Hungar et al., 2017).

Unlike unexpected, non-scheduled takeovers (e.g., emergency brake of a lead vehicle in congestion) in which drivers must respond within a short time window, the takeover events due to ODD exit such as exiting a freeway can be scheduled ahead of time rendering drivers ample time to perform a takeover task (McCall et al., 2019). Previous empirical studies adopted a ToR lead time ranging from 6 to 60 seconds (s) for freeway exiting takeovers (Holländer & Pfleging, 2018; Nobari et al., 2020; Petermeijer et al., 2017; Tan & Zhang, 2022; Wörle et al., 2020; Yun & Yang, 2020). These studies were mostly focused on takeover performance (such as takeover response time and vehicle control) and subjective experience. Yet, few research has been conducted to study driver behavior during the process of scheduled takeovers.

The general takeover process of conditionally AVs involves a sequence of actions: 1) redirect gaze from a nondriving related task (NDRT) to the road; 2) process and evaluate the situation; 3) establish cognitive readiness for takeover and determine actions to be taken after regaining control; 4) reposition the body to get the motor readiness for takeover; and 5) execute the takeover action and driving maneuver. When there is no time pressure in scheduled takeovers, drivers may not be motivated to implement any of these actions, and therefore take longer to regain control following a ToR (Eriksson & Stanton, 2017; Tan & Zhang, 2022). Drivers’ improper attention management and takeover strategies can result in life-threatening consequences (Hopp et al., 2005). For instance, late takeovers could increase the possibility of missing an exit or exit ramp crashes, while hasty takeovers with drivers not acquiring adequate situation awareness may lead to inappropriate driving maneuver after takeover. Thus, understanding drivers’ strategies for scheduled takeovers is important for conditionally AV driving safety.

The present study aims to identify the strategies that drivers take for regaining control of conditionally AVs to exit a freeway with a long ToR lead time. Borowsky et al. (2022) examined four strategies to manage a ToR with a time budget of 6 or 8 or 12 s to avoid potential hazards in conditionally AVs: 1) take over control and perform the NDRT and the driving task simultaneously; 2) take over control and focus on the driving task; 3) complete the NDRT first and then take over control; and 4) perform the NDRT and did not take over control. However, driver behavior may vary depending on the time budget. This work extends Borowsky et al.’s (2022) framework to analyze driver strategy for manage attention and takeover for exiting a freeway with a ToR being possibly scheduled up to 60 s beforehand.

Furthermore, this study also aims to evaluate the multistage ToR design for scheduled takeovers in conditionally AVs. A web-based behavioral study by Tan and Zhang (2022) has found the positive effects of a longer ToR lead time (up to 60 s) on situation awareness and system evaluations, however with a significant drop in drivers’ estimation of the remaining travel distance to exit a freeway compared to the 30 s lead time. Thus, multi-stage ToR design is proposed to communicate the urgency of takeover situation within a long ToR lead time with the purpose of reducing risky takeover behavior.

The present study is focused on the modality design of multistage ToR and assesses its effects on drivers’ strategy for scheduled freeway exiting takeovers in conditionally AVs.

Method

Participants

Thirty-two people (15 males, 17 females) aged 18 to 55 (Mean = 24.5, SD = 7.1) participated in the experiment. All the participants were licensed drivers for at least two years, with an average driving experience of 7.5 years (SD = 7.2) and an average annual mileage of 8,140 miles per year (SD = 5,756). After the experiment, each participant received $20.

Apparatus

The experiment was conducted using a fixed-based driving simulator (STISIM Drive^®), which includes three driving displays that enable a 135^° field of view, a high-fidelity, fullsize steering wheel with active force feedback and 900^° rotation, and two advanced foot pedals. The STISIM Drive^® Software is programmable using Open Module, which allows for automated driving programming and control transitions from driving simulator to driver.

A wearable eye tracking tool (Tobii Pro Glasses 2) was used to capture gaze behavior. The Tobii glasses are equipped with 4 eye cameras (each runs at 100 Hz), a HD (1980×1080) scene camera with 90^° field of view, and thin side pieces for completely unobstructed side view. The dual image sensor technology enables accurate compensation for head movements and is also used for pupil size measurement compensations.

Experiment Design and Driving Scenario

The experiment adopted a within-subjects design. Each participant was exposed to four multi-stage ToR designs (hereinafter referred to as four designs) as shown in Fig. 1. The ToR lead time for exiting a freeway was 60 s (one mile away from the exit with the car driving at 60 mph) in all the four designs. Following the initial ToR, two warnings were delivered at 45 and 30 s remaining to exit, generating three stages within a 60 s lead time – information stage (15 s), warning stage (15 s), and command stage (30 s). The three-stage design seeks to alert drivers to the situation with a gradually increasing sense of urgency over an extended period.

Figure 1.

Four ToR designs for freeway exiting takeovers.

The four designs differ in the modality of warnings issued at three stages. The multi-stage auditory (MA) design contained a white symbol of two hands holding a steering wheel on the dashboard and sent out a speech message at each stage, which was composed of a beep (1 kHz warning tone lasted for 0.15 s) followed by a signal word and the remaining distance to exit. The signal words for the three stages were Notice, Caution, and Danger, respectively. The speech was generated using a digitized human female voice with a speech rate of ~150 words/min at the loudness level of 70 dB. The multi-stage visual (MV) design used a beep only at the initial ToR and displayed a colored steering wheel symbol on the dashboard. The symbol adopted an icon with different colors to indicate the increasing urgency at three stages: a green symbol at the information stage, a yellow symbol containing an alarm lamp at the warning stage, and a red symbol containing an exclamation mark at the command stage. The colored symbol remained on the dashboard for the entire corresponding stage. The multi-stage auditory-visual (MAV) design was a combination of MA and MV designs. A baseline (B) design contained a beep issued at the initial ToR and a white steering wheel symbol remained for the entire ToR lead time before exiting a freeway. For all the four designs, a mockup of navigation map was displayed on the dashboard throughout the experiment. When the initial ToR was issued, the navigation map started displaying the information about the freeway exit, including which side of the road the desired exit was on, the remaining distance away from the exit, and the speed limit at the exit ramp.

The four designs were tested in the experiment following a Latin square design. Each participant went through four test sessions, with each session for testing one design. Each session contained three trials of performing the takeover task in random order, making it a total of 12 trials for each participant. To increase the variance of scenarios and reduce the carryover effect, the driving scenario of three takeover tasks in a session differed in the side of the road that the exit was on and the speed limit (30 or 45 or 60 mph) of exit ramp.

The driving context of all the takeover tasks was freeway, which involved a one-way four-lane roadway with no traffic lights, intersections, or pedestrian paths. The traffic density was 13-25 vehicles per mile per lane to follow the 4-second rule of following distance on freeway. Each trial was about 14,300 feet long (about 2.7 minutes’ driving). A freeway exit sign was placed one mile and right at the exit gore area.

At the beginning of each trial, the subject vehicle was in automated mode. Both the subjective vehicle and all the surrounding vehicles drove at the speed in line with the speed limit of 60 mph on freeway. When the initial ToR was issued at one mile away from the desired exit, a car (Car 1) was driving 250 feet ahead of the subject vehicle in the same lane (exit lane) and another car (Car 2) was driving 150 feet ahead of the subject vehicle in the adjacent lane. Car 2 had its turning lights on to signal its intention to change to the subject vehicle’s lane. Until arriving at the location with the remaining distance of 0.25 mile (15 s left), Car 2 smoothly cut in front of the subject vehicle while maintaining a safe distance of 150 feet between them. No collisions would occur even without participants’ intervention. After entering the exit ramp, Car 1 and Car 2 were programmed to drive in accordance with the speed limit.

Dependent Variables

The general takeover process to exit a freeway was illustrated as Fig. 2. Drivers’ takeover strategy was analyzed based on four key points in the takeover process: the initial ToR, the first gaze redirection to road after ToR, the last gaze redirection to road before takeover, and takeover action. The four points generated four measures, which were captured by eye tracking glasses. The gaze redirection time (time frame a in Fig. 2) referred to the duration from the delivery of the initial ToR to the first gaze redirection from NDRT to the road, which indicated drivers’ response to the auditory warning. The delay before last gaze redirection (time frame b in Fig. 2) represented the duration from the initial ToR to the last gaze redirection to road. The duration of takeover preparation (time frame c in Fig. 2) was defined as the duration between the last gaze redirection to road and takeover, which was used for restoring situation awareness and getting prepared for takeover. The takeover response time (time frame b+c in Fig. 2) counted from ToR to takeover action.

Figure 2.

An illustration of the general takeover process for exiting a freeway and dependent measures of takeover strategy. (a: Gaze redirection time; b: Delay before last gaze redirection; c: Duration of takeover preparation; b+c: takeover response time).

Procedure

In the recruitment stage, people who signed up to participate in the experiment first completed a pre-screening survey. The potential participants were asked to select one from three location ranges (0-0.5 mile, 0.5-0.75 mile, 0.75-1 mile away from the exit) that they were most likely to take over control of a conditionally AV to exit a freeway when a ToR was issued one mile away from the target exit. The purpose was to recruit people that would take over at various stages to evaluate multi-stage ToR designs.

Upon arrival, potential participants first wore eye tracking glasses and calibrated their gaze fixations. Only people who passed the calibration test were allowed to continue. After confirming the eligibility, participants signed a consent form and completed a survey on demographics. Then, they were asked to read an instruction document regarding the takeover task, the information displayed in the driving scene and on the car dashboard, and the meaning of four designs.

After participants confirmed that everything was clear, they were led to sit in the driving simulator and were shown the location of the takeover button, pedals, and steering wheel. They were told to play a mobile game named Snake VS Block on a Huawei P9 smartphone with no need to supervise automated driving. The mobile game was an endless running style game in which participants swiped a finger to guide a snake of balls and break the bricks, which required visual perception, motor control, and simple numeric comparison capabilities. Participants were instructed to press the takeover button after receiving ToR when they felt necessary and safe to regain control and exit a freeway. Before the formal test, participants had the opportunity to get familiar with the takeover task and four designs in a 10-minute practice session, which contained four trials in the same order (B, MV, MA, MAV). After the practice, the formal test containing four test sessions with 12 trials was conducted.

Results

The data analysis was conducted based on 382 trials after removing one trial with eye tracker recording buffering and one trial with the takeover being performed before ToR. In 65 trials, participants already looked up and checked the traffic situation at the delivery of ToR. For these cases, when computed takeover strategy measures for analysis, the gaze redirection time was counted as zero; the delay before the last gaze redirection was also counted as zero if the participant did not return to NDRT; the duration of takeover preparation was the amount of time between ToR and takeover. Table 1 summarized the sample size, sample mean, and standard deviation of dependent measures for four designs.

Table 1.

Descriptive means (standard deviations) and sample size (N).

	N	Gaze redirection time (s)	Delay before last gaze redirection (s)	Duration of takeover preparation (s)	Takeover response time (s)
B	95	5.40 (7.75)	21.21 (17.50)	10.64 (10.56)	31.85 (17.60)
MV	96	7.60 (11.20)	18.25 (16.70)	7.31 (9.05)	25.55 (16.89)
MA	95	9.84 (11.84)	19.70 (13.38)	8.08 (8.85)	27.79 (13.36)
MAV	96	8.28 (9.56)	18.06 (12.45)	6.95 (7.31)	25.01 (11.76)

Eight driver strategies (1a, 1b, 2a, 2b, 3a, 3b, 4a, 4b in Table 2) for attention management and takeover were identified based on combinations of data-related conditions regarding gaze redirection time, remaining time after takeover, and time difference between the first and last gaze redirection to the road. Specifically, a two-second criterion was used to determine whether drivers responded immediately to ToR. An eight-second criterion was used to differentiate between drivers who delayed takeover action and those who did not (Tan & Zhang, 2022). The time difference between the first and last gaze redirection was examined to determine whether drivers alternated their attention between the road and NDRT.

Table 2.

Data-Driven categorization criteria of eight takeover strategies (1a, 1b, 2a, 2b, 3a, 3b, 4a, 4b).

	Strategy		Nr	First gaze redirection time	Remaining time after takeover	First (F₁) and last (F_n) gaze redirection
1	Accept ToR and takeover	1a. Monitor AV driving	65			F¹ = F_n
		1b. Alternate attention	101	≤ 2 s	> 8 s	F₁ < F_n
2	Accept ToR and delay takeover	2a. Monitor AV driving	1			F₁ = F_n
		2b. Alternate attention	8	≤ 2 s	≤ 8 s	F₁ < F_n
3	Delay ToR response and accept takeover	3a. Monitor AV driving	105			F₁ = F_n
		3b. Alternate attention	92	> 2 s	> 8 s	F₁ < F_n
4	Delay ToR response and takeover	4a. Monitor AV driving	4			F₁ = F_n
		4b. Alternate attention	6	> 2 s	≤ 8 s	F₁ < F_n

Data Analysis

As a within-subjects design was adopted, the multilevel modeling test was used to quantify within-person and between-person variability. The analysis followed a “build-up” strategy for multilevel model testing (Heck et al., 2013). First, a random intercept model was tested for each dependent measure to examine whether there was significant variance across participants. The Wald Z tests (at a significance level of .025 for one-tailed test) and ICCs (at a threshold of .05) indicated substantial clustering of data in the gaze redirection time (Z = 3.13, p = .002; ICC = 0.23), the delay before the last gaze redirection (Z = 3.69, p < .001; ICC = 0.50), the duration of takeover preparation (Z = 3.47, p = .001; ICC = 0.36), and takeover response time (Z = 3.80, p < .001; ICC = 0.61), which supported the multilevel modeling test. Then, models were tested after adding task-level predictors (i.e., ToR design and sequence of trial within participants) to the random intercept model. On this basis, driver-level predictors (i.e., age, gender, and annual mileage) were added. By conducting likelihood ratio test, a significant increase in fit was observed by incorporating driver-level predictors for the delay before the last gaze redirection ( (3) = 9.02, p = .03) and takeover response time ( (3) = 8.09, p = .04), but not for the gaze redirection time or the duration of takeover preparation.

Takeover Strategy

Gaze redirection time

The ToR design had a significant effect on gaze redirection time, F (3, 350.10) = 4.49, p = .004. Post-hoc testing using pairwise comparisons of the estimated marginal means with Bonferroni adjusted alpha levels revealed a significantly longer gaze redirection time under the MA design compared to the Baseline design (see Fig. 3). The gaze redirection time was also significantly affected by the sequence of trial, b = .52, t(350.10) = 4.03, p < .001, 95% CI [.26, .77].

Figure 3.

Effects of ToR design on gaze redirection time.

Delay before last gaze redirection

The ToR design did not affect the delay before last gaze redirection. Nevertheless, the sequence of trial was a significant factor, b = .51, t(350.26) = 3.32, p = .001, 95% CI [.21, .82]. Male drivers (M = 23.72) had longer delay before the last gaze redirection than their female counterparts (M = 15.41), F (1, 32.27) = 6.62, p = .02.

Duration of takeover preparation

Results found significant effects of ToR design on the amount of time spent on preparing takeover, F (3, 350.07) = 5.24, p = .002. Compared to the MV and MAV designs, the Baseline design led to significantly longer duration of takeover preparation (see Fig. 4).

Figure 4.

Effects of ToR design on duration of takeover preparation.

Takeover response time

The takeover response time significantly depended on ToR design, F (3, 349.94) = 11.07, p < .001. The Baseline design led to longer takeover response time than the three multi-stage ToR designs (see Fig. 5). In addition, the takeover response time was also significantly affected by the sequence of trial (b = .45, t(350.25) = 3.42, p = .001, 95% CI [.19, .71]), annual mileage (b = -.001, t(39.72) = 2.05, p = .047, 95% CI [-.001, -1E-5]), and gender (F (1, 32.32) = 6.30, p = .02). Male drivers (M = 32.02) spent longer time to regain control than female drivers (M = 23.59).

Figure 5.

Effects of ToR design on takeover response time.

Takeover strategy

The multinomial regression showed that the takeover strategy was significantly affected by ToR design ((21) = 39.32, p = .01), sequence of trial ( (7) = 26.34, p < .001), age ((7) = 16.68, p = .02), gender ( (7) = 41.27, p < .001), and annual mileage ((7) = 23.40, p = .001). Using strategy 1a as a reference category, drivers were more likely to adopt strategy 3a when using MA and MAV designs and strategy 4a when using MA design, compared to the Baseline design (see Fig. 6)

Figure 6.

Frequency of eight takeover strategies under four ToR designs.

Discussion and Conclusion

The present study identified eight strategies that drivers used for scheduled freeway exiting takeovers in conditionally AVs. Some strategies were in line with Borowsky et al.’s (2022) interruption management in possible hazardous scenarios. For example, drivers may disengage NDRT and take over immediately or prioritize NDRT and postpone takeover action. However, some strategies were not observed, such as taking over immediately and performing the driving task and NDRT simultaneously. Furthermore, this study observed new takeover strategies in scheduled takeover events. Drivers alternated attention between NDRT and the road in about 54.2% of takeover events (strategies 1b, 2b, 3b, and 4b), which is a feature of scheduled takeovers with sufficient time budget. Nonemergency takeover events also featured a delay in the gaze redirection observed in 54.2% of takeover events (strategies 3a, 3b, 4a, and 4b). Repeated exposure to takeover tasks with the same lead time for freeway exiting may generate practice effects and adaptive takeover behavior.

The multi-stage ToR design had an impact on driver strategy for attention management and takeover. When the urgency of takeover situations was communicated using the three multi-stage ToR designs, drivers’ takeover action was advanced compared to using the Baseline design. The decrease in takeover response time under MV and MAV designs resulted from the shorter duration of takeover preparation. Furthermore, drivers were more likely to take over control shortly after a ToR rather than alternating attention between NDRT and the road when using MV design compared to Baseline design. Nevertheless, the MA design delayed drivers’ gaze redirection to the road after the ToR. The verbal warning involving a signal word and remaining distance may generate a lower perceived urgency of situation and higher driver trust (Bliss & Kilpatrick, 2000), whereas the abstract, nonverbal warning was more effective in urging drivers to disengage NDRT and check the traffic situation.

In conclusion, scheduled takeovers allow drivers to perform a takeover task at their own pace, resulting in a variety of strategies for gaze redirection and takeover. Informing drivers about the urgency of the takeover situation at multiple stages within a long lead time has changed driver strategy. The effectiveness of multi-stage ToR designs is examined in advancing drivers’ takeover action by decreasing the likelihood of alternating attention and reducing the time spent on preparing for takeover.

Footnotes

Acknowledgements

We gratefully acknowledge the support by PSU/Technion Marcus Funds.

ORCID iD

Xiaomei Tan

References

Bliss

J. P.

Kilpatrick

(2000). The Effect of Vocal Alarms on Operator Mistrust. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 44(22), 694–697.

Borowsky

Zangi

Oron-Gilad

(2022). Interruption Management in the Context of Take-Over-Requests in Conditional Driving Automation. IEEE Transactions on Human-Machine Systems, 52(5), 1015–1024.

Eriksson

Stanton

N. A.

(2017). Takeover Time in Highly Automated Vehicles: Noncritical Transitions to and From Manual Control. Human Factors, 59(4), 689–705.

Heck

R. H.

Thomas

S. L.

Tabata

L. N.

(2013). Multilevel and longitudinal modeling with IBM SPSS. Routledge.

Holländer

Pfleging

(2018). Preparing drivers for planned control transitions in automated cars. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia, 83-92.

Hopp

P. J.

Smith

C. A. P.

Clegg

B. A.

Heggestad

E. D.

(2005). Interruption management: The use of attention-directing tactile cues. Human factors, 47(1), 1-11.

Hungar

(n.d.). Test Specifications for Highly Automated Driving Functions: Highway Pilot.

NHTSA. (2017). Automated Driving Systems 2.0: a vision for safety (DOT HS 812 442). https://www.nhtsa.gov/sites/nhtsa.gov/files/documents/13069aads2.0_090617_v9a_tag.pdf

McCall

McGee

Mirnig

Meschtscherjakov

Louveton

Engel

Tscheligi

(2019). A taxonomy of autonomous vehicle handover situations. Transportation Research Part A: Policy and Practice, 124, 507–522.

10.

Nobari

K. D.

Albers

Bartsch

Bertram

(2020). Online Feedback Control for Driver-Vehicle Interaction in Automated Driving. International Conference on Applied Human Factors and Ergonomics, 159-165.

11.

Petermeijer

Doubek

de Winter

(2017). Driver response times to auditory, visual, and tactile take-over requests: A simulator study with 101 participants. 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 1505-1510.

12.

Tan

Zhang

(2022). The effects of takeover request lead time on drivers’ situation awareness for manually exiting from freeways: A webbased study on level 3 automated vehicles. Accident Analysis & Prevention, 168, 106593.

13.

Wörle

Metz

Othersen

Baumann

(2020). Sleep in highly automated driving: takeover performance after waking up. Accident Analysis & Prevention, 144, 105617.

14.

Yun

Yang

J. H.

(2020). Multimodal warning design for take-over request in conditionally automated driving. European transport research review, 12, 1-11.