Performance of an Additional Task During Level 2 Automated Driving: An On-Road Study Comparing Drivers With and Without Experience With Partial Automation

Abstract

Objective:

To investigate the influence of prior experience with Level 2 automation on additional task performance during manual and Level 2 partially automated driving.

Background:

Level 2 automation is now on the market, but its effects on driver behavior remain unclear. Based on previous studies, we could expect an increase in drivers’ engagement in secondary tasks during Level 2 automated driving, but it is yet unknown how drivers will integrate all the ongoing demands in such situations.

Method:

Twenty-one drivers (12 without, 9 with Level 2 automation experience) drove on a highway manually and with Level 2 automation (exemplified by Volvo Pilot Assist generation 2; PA2) while performing an additional task. In half of the conditions, the task could be interrupted (self-paced), and in the other half, it could not (system-paced). Drivers’ visual attention, additional task performance, and other compensatory strategies were analyzed.

Results:

Driving with PA2 led to decreased scores in the additional task and more visual attention to the dashboard. In the self-paced condition, all drivers looked more to the task and perceived a lower mental demand. The drivers experienced with PA2 used the system and the task more than the novice group and performed more overtakings.

Conclusions:

The additional task interfered more with Level 2 automation than with manual driving. The drivers, particularly the automation novice drivers, used some compensatory strategies.

Applications:

Automation designers need to consider these potential effects in the development of future automated systems.

Keywords

automated driving additional task performance visual attention compensatory strategies automation experience

Introduction

Different automated systems, such as active cruise control (ACC) and lane keeping assist (LKA), have recently been integrated to provide feet- and hands-off driving for short periods. Examples include Pilot Assist version 2 (PA2) by Volvo and Autopilot by Tesla. While such systems perform the longitudinal and lateral control of the vehicle, the driver still needs to be alert and react when the system boundaries are exceeded or any maneuver needs to be performed. Several car manufacturers decided to guarantee this alertness by not letting drivers remove their hands from the steering wheel for more than a few seconds (e.g., 10–15 seconds in the case of Volvo and 60 seconds in the case of Tesla). According to SAE’s classification (SAE j3016; SAE, 2016), this level of automation falls under Level 2 (partial automation), as compared to Level 3, where the car monitors the environment as well, thereby granting the driver longer periods of inattention to traffic.

Despite the expected benefits of automation on driver workload, well-being, and safety (Stanton & Marsden, 1996), unanticipated effects have been reported, such as overreliance or reduced situation awareness (Saffarian, de Winter, & Happee, 2012). Another effect is a higher proneness to engage in non-driving-related tasks (Banks, Eriksson, O’Donoghue, & Stanton, 2018; Carsten, Lai, Barnard, Jamson, & Merat, 2012; Llaneras, Salinger, & Green, 2013), which likely is a natural consequence of relieving the driver from the control of the vehicle position. In support of this, a meta-analysis by de Winter, Happee, Martens, and Stanton (2014) estimated that compared to manual driving, in Levels 1 and 3, drivers are able to complete 112% and 261% more tasks, respectively.

In comparison to Levels 1 and 3, the literature offers less information about Level 2 automation. Even though it might be expected that drivers’ workload should be lower in Level 2 than in Level 1, although still higher than in Level 3, the little evidence available is ambiguous. For example, Banks et al. (2018) observed that drivers tended to engage more in other tasks during on-road partially automated driving. While this could reflect a complacency effect whereby drivers temporarily forgo their supervisory responsibilities, this might also show a greater capacity to integrate other tasks into the ongoing monitoring demands. In another on-road study by Naujoks, Purucker, and Neukum (2016), the drivers were found to engage more often in an additional task during Level 1 (only ACC) and Level 2 (ACC and LKA) driving compared to manual driving. However, no differences were observed between Level 2 and Level 1. The study participants did not have any Level 2 experience, which could explain why they did not benefit from it.

The results of Naujoks et al. (2016) also indicate that mental workload might not decrease linearly as more tasks are automated, as may be expected. An explanation for this can be found within the framework of the driver in control (DiC) model (Hollnagel, Nåbo, & Lau, 2003). According to DiC, driving consists of simultaneous subgoals at different interconnected levels of control, from more compensatory, like tracking or regulating, to more anticipatory tasks, like monitoring and targeting. The model also predicts that changes in one level may propagate upward to more anticipatory levels or downward to more compensatory levels, thus influencing driver performance. In the context of Level 2 automation, new demands may be placed on the monitoring control level because of automating tasks on the tracking or regulatory levels. In such level, other cognitive abilities may be required that are likely to affect performance on tasks tapping into the same control level. Particularly, monitoring without active task engagement requires drivers to keep vigilant and anticipate when his or her intervention is necessary. Such tasks may be quite demanding even in understimulating conditions (Warm, Parasuraman, & Matthews, 2008) and could therefore affect the performance of other tasks.

In this line, Level 2 automated vehicles might require some effort and/or compensatory strategies from the driver to integrate additional tasks into the ongoing supervisory demands. Therefore, the requirements of the additional task play an important role as they may interfere with other ongoing tasks at the same or different control levels in the DiC model. Also, as predicted by the multiple resource model (Wickens, 2008), the interference level between the tasks may depend on the extent to which they require similar resources within the following dimensions: stage of processing, code of processing, and modality. Additionally, other features such as the type of presentation or the interruptability of the task (Kircher, Fors, & Ahlstrom, 2014; Platten, Schwalm, Hülsmann, & Krems, 2014) need to be considered. Regarding the latter, Platten et al. (2014) observed that when the additional task could be interrupted with no perceived loss of performance, drivers tended to do so in demanding driving situations, thus freeing up resources on the monitoring level. This shows that drivers actively integrated this feature into their strategy to manage the overall demands. Although their study was conducted in manual driving, the interruption strategy may remain useful in Level 2, and therefore, it will be investigated here. We expect the drivers to interrupt the additional task to compensate for peaks of workload occurring when driver intervention is necessary, leading to improved quality of performance on the additional task.

The DiC framework predicts that anticipatory behavior plays an important role in integrating all subtasks. As anticipation is gained by experience, previous studies on automated driving using participants with little or no previous experience of automation may be misleading. Nowadays, as Level 2 vehicles have been on the market for some time, it is possible to study drivers who have acquired experience naturally over a longer period of time. This provides a unique opportunity to learn how Level 2 automation is used by drivers with a more precise mental model of the system capabilities and limitations. To date, most studies on Level 1 (only ACC) indicate that prior experience with automated systems influences how drivers are affected and interact with the system. For example, Kopf and Nirschl (1997) observed lower levels of mental workload in drivers with prior experience with ACC compared to drivers without experience. Experienced drivers with ACC have also shown a more developed mental model of the system limitations (Larsson, Kircher, & Andersson Hultgren, 2014) and seem to effectively integrate this model in their tactical decisions (Kircher, Larsson, & Hultgren, 2014), indicating a higher share of anticipatory behavior. Interestingly, prior experience with ACC may also influence how drivers interact with Level 2 systems. As shown by Naujoks et al. (2016), previous familiarity with ACC increases the level of engagement in an additional task both while driving with only ACC (Level 1) and with a combination of ACC and LKA (Level 2).

As previously mentioned, one of the main effects of prior experience with an automated system is a more developed mental model of the system. Based on the DiC model, we hypothesize that drivers with naturally acquired experience with Level 2 automation will display a different monitoring strategy than drivers without such experience. Particularly, we expect experienced drivers to integrate the system functionality in their behavior, thus diminishing their traffic and system monitoring when driving under conditions in which they learned the system is reliable (e.g., straight highway with good visibility). The more detailed mental model of the system may also be used by the experienced drivers when making tactical decisions such as to overtake or integrate the performance of additional tasks. In this study, we investigate how prior experience with Level 2 influences drivers’ cognitive and behavioral strategies while driving with automation and performing an additional task.

Aims and Hypotheses

The aim of this on-road study was to analyze drivers’ visual attention and compensatory strategies while driving and performing an additional visuomotor task. Particularly, we analyzed the effect of the level of automation (manual vs. Level 2), prior experience with the system (automation novice and automation experienced drivers), and the additional task pacing (system-paced vs. self-paced). The Level 2 system that was used in the study was PA2 by Volvo. Our hypotheses were:

Hypothesis 1: Automation of distance and lane keeping shifts the driving task to higher levels within the DiC model, which frees up resources on the level of regulation and operation. This should result in longer glances toward the additional task, more engagement in the additional task, and fewer overtakings due to decreased engagement in the driving task.

Hypothesis 2: Experienced drivers are expected to be able to make a better use of the system functionality due to their increased anticipatory capabilities. They are expected to perform better on the additional task, time better the switching on/off of the task/system, experience lower workload, and glance more “efficiently,” for example, longer glances off forward.

Hypothesis 3: Being able to switch off the task leads to better task performance and lower workload as drivers do not feel the pressure to work on the task anyway even though it is “too much” at the moment.

Methods

Participants

Twenty-three drivers (2 women) were recruited into two groups, one of which consisted of people with no prior experience with automated vehicles (henceforth called automation novices). The other group was recruited from a list of owners of one of the Volvo models launched in the spring of 2016 and equipped with PA2 system (automation experienced). The participants were required to have used the system during at least three drives a week for more than a month. Two participants (one from each group) could not complete the experiment due to bad weather conditions. Each participant received 500 SEK (≈$60 USD) for participation. Information on demographics, driving experience, and automation experience for both groups is presented in Table 1.

Table 1:

Demographic, Driving, and Automation Experience Information for Both Groups of Participants

	Automation Novices (n = 12)	Automation Experienced (n = 9)
Women (n)	2	0
Age (mean ± SD)	40 ± 6.33	47 ± 10.74
Years with driving license (mean ± SD)	20 ± 6.9	29 ± 10.74
Hours driven per week (n)
0–10	2	1
11–20	6	4
≥21	4	4
Experience with other automated systems (n)
Cruise control	8	9
Active cruise control	2	9
Time owning a car with PA2 system (months)	—	4.5 ± 1.6
PA2 system weekly use	—	≥3 times a week

Note. PA2 = Volvo Pilot Assist generation 2.

This research complied with the American Psychological Association Code of Ethics and was approved by the Regional Ethics Review Board in Linköping, Sweden (DNR: 2016/411-31). All participants gave their informed consent prior to the experiment.

Equipment and Data Acquisition

Automated system

A 2017 Volvo S90 model equipped with the second version of the Pilot Assist system was used for the experiment. The PA2 consists of the combined operation of ACC and LKA. The driver may remove the hands from the steering wheel for no longer than approximately 15 seconds. To initiate the system, the driver must select the “Pilot Assist” option by using the keypad on the steering wheel (see Figure 1). The status of the system is indicated by a symbol representing a steering wheel, which is integrated in the speedometer (Figure 1). When the system is active, the symbol is represented in green. However, when the detection of the lane markings by the sensors is temporarily interrupted (e.g., degraded or snow-covered lane markings), the system enters a stand-by mode, and the symbol turns grey. Then, only the ACC is active until the detection of the lane markings is resumed. The change from active to stand-by mode is only informed by the change in the symbol color, and no other auditory or haptic warning is presented. When the system is turned off completely, the symbol is not shown. To take back full control from the car, the driver must touch the brake or select the manual driving mode via the keypad on the steering wheel (Figure 1).

Figure 1.

The vehicle cockpit with the Volvo Pilot Assist generation 2 (PA2) system and the additional task engaged.

Data acquisition

The front view of the vehicle and the PA2 system status were filmed with one camera each and recorded in a data acquisition unit (Video VBox Pro, RaceLogic, Buckingham, UK). UTC time, vehicle speed, and vehicle position were registered via the internal GPS (resolution 0.1 km/h) data logger in the data acquisition unit (sampling rate 10 Hz).

A head-mounted eye tracker (SMI Eye Tracking Glasses 2.0, SensoMotoric Instruments, Teltow, Germany) was used to register glance behavior as well as the drivers’ scene view (sampling rate 60 Hz).

Test Route

The experiment took place on a two-lane highway near the city of Linköping. The participants started driving on the highway for approximately 7.5 km, then exited and drove on a trunk road for 3 km, until they entered the same highway but in the opposite direction (Figure 2). The quality of the lane markings was within the PA2 working limits throughout the whole route. Overall, the traffic density was medium-low for all participants. Only the data collected on the highway were analyzed as traffic conditions here were more comparable between participants. The speed limit in the highway was 110 km/h.

Figure 2.

Overview of the training segment, test route, and trunk road used in the experiment (Google Maps, 2017).

Additional Task

The additional task was a visuomotor task presented on a tablet placed to the right of the steering wheel (Figure 1) that was based on the surrogate IVIS task used in Östlund et al. (2004). The participants had to determine whether an arrow pointing upward was present in a 4 × 4 matrix of arrows. All other arrows but the target always pointed downward. The target arrow was present in half of the trials in a random position in the matrix. To respond to the task, the participants touched either the yes or no button displayed on the screen below the arrows matrix. A new matrix was presented either when the driver gave a response or after 5 seconds if no response was recorded. Thus, the task was system-paced but allowed the drivers to self-regulate their performance to some extent.

In the system-paced task condition, the additional task could not be turned off by the driver. In the self-paced condition, an on/off button was available above the arrows matrix. The drivers were informed that switching off the task would not affect their performance but that they should turn it back on whenever they thought it was possible. To strike a balance between motivating the drivers to perform the task and keeping them from possibly risky over-engagement, a playful incentive without real material value was offered. The participants were told that they could win up to six chocolate bars depending on the score achieved and the time the task was on. All measurements obtained from this task are presented in Table 2.

Table 2:

Dependent Variables Obtained From the Driving and Additional Tasks, Subjective Mental Workload, and Glance Behavior

Dependent Variables		Objective
Driving
Speed (km/h)		To investigate adaptations directly related to the driving task
Overtakings per km
Percentage time with PA2 on^a
Percentage time on the left lane
Additional task
Number of trials completed per minute		To investigate indirect adaptations such as increased additional task engagement and its consequences for additional task performance
Percentage correct
Percentage misses
Percentage errors
Percentage time task on^b
Mean response time
Number of off-switches^b
Mental workload (NASA-TLX)
Mental demand		To investigate whether potential changes in driving and/or additional task performance were modulated by changes in experienced workload or effort
Physical demand
Temporal demand
Performance
Effort
Frustration
Global score
Glance behavior
Measurements	Areas of interest	Used as a proxy for attention to investigate whether drivers shift attentional focus between driving, monitoring, and additional task engagement
Percentage of time looking	Front
Percentage of time looking	Additional task
Number of glances (per minute)	Dashboard
Number of glances (per minute)	Left mirror
Mean and 90th glance duration to the additional task(s)	Inside mirror
Mean and 90th glance duration to the additional task(s)	Right mirror

Note. PA2 = Volvo Pilot Assist generation 2.

Only for the automated driving conditions.

Only for the self-paced conditions.

Experimental Design

A mixed 2 × 2 × 2 design with two within- and one between-subjects variables was used. The within-subjects variables were automation level (manual and Level 2 driving) and additional task pacing (system-paced and self-paced). The between-subjects variable was the level of experience with the PA2 system (automation experienced and automation novice). Each participant drove one lap per condition combination, resulting in four laps per participant. The order of the conditions was balanced to prevent learning effects, variations in traffic intensity, or fatigue symptoms.

Procedure

Upon arriving at VTI, the participant received written instructions about the study. Automation novice drivers were then briefed on the PA2 system by using the online Volvo owner’s manual (Volvo Cars, n.d.) and given practical training on how to turn on/off the system.

The participant practiced the additional task until he or she was able to perform 15 correct trials in a row and reported feeling confident with it. He or she was instructed to perform this task as well as possible while driving safely and also reminded that safety was paramount and that it was fully acceptable not to perform the additional task. The driver was asked to use PA2 as much as possible in the automated condition but to take over control when this was deemed necessary.

The eye tracker was mounted and calibrated, followed by a short training on nearby roads for familiarization with the PA2 system and the additional task (Figure 2). Only when the drivers reported feeling confident were they asked to drive to the test route. The duration of the training sessions ranged between 20 and 30 minutes. A trained researcher sat in the backseat to monitor driver and system behavior and the traffic conditions. After each condition, the NASA-TLX was administered verbally (Hart & Staveland, 1988). Once all conditions were completed, the participant was given the reward corresponding to their additional task performance.

Data Reduction and Analysis

The eye tracking data were superimposed onto the scene camera videos using BeGaze Version 3.7 (SensoMotoric Instruments, Teltow, Germany). The resulting gaze-overlay videos, visualizing the foveated glance target from the drivers’ point of view, were then manually coded using the Observer XT Version 13 software (Noldus Information Technology, Wageningen, The Netherlands) as glances to predefined areas of interest (AOI; see Table 2). All manual coding was carried out by one person, thus eliminating intercoder reliability issues. The percentage of time looking at the additional task and the number of glances to it were calculated based on the total time the task was on (i.e., 100% of the time in the system-paced task conditions).

For the statistical analyses, 2 × 2 × 2 repeated measures analyses of variance (ANOVA) were conducted with automation experience as between-subjects factor and automation level and additional task pacing as within-subjects factors. Main and interaction effects were analyzed for the dependent variables (see Table 2). Specific analyses (2 × 2 ANOVAs) were conducted to investigate differences in the time spent with the task on (in the self-paced task conditions) and the time spent using PA2 (in the automated conditions).

Greenhouse-Geisser corrections were applied when Mauchly’s test detected violations of sphericity. Sidak’s pairwise comparisons were used to correct for Type I error. The significance level was set to .05. Partial eta-squares were calculated as a measure of relative effect size.

Results

The results are presented per dependent variable. Table 3 summarizes all effects and effect sizes.

Table 3:

Summary Table of the Effects Found in the Analyses

Dependent Variable		EL	AL	TP	EL × AL	EL × TP	Direction of the Main Effects	Direction of Interactions
Percentage of time looking	Front	F = 5.41η_p² = 0.22*	F = 8.84η_p² = 0.32**	F = 0.26η_p² = 0.01	F = 0.02η_p² = 0.01	F = 5.12η_p² = 0.21*	N > E;MD > AD	When the task was Self-P, N > E
	Additional task	F = 4.77η_p² = 0.2*	F = 3.24η_p² = 0.14	F = 11.59η_p² = 0.38**	F = 0.54η_p² = 0.03	F = 0.29η_p² = 0.01	E > N;Self-P > Syst-P
	Dashboard	F = 0.95η_p² = 0.05	F = 19.42η_p² = 0.51**	F = 3.89η_p² = 0.06	F = 2.82η_p² = 0.13	F = 0.99η_p² = 0.01	AD > MD
	Left mirror	F = 0.05η_p² = 0.01	F = 4.56η_p² = 0.19*	F = 5,14η_p² = 0.21*	F = 0.06η_p² = 0.01	F = 0.07η_p² = 0.01	MD > AD;Syst-P > Self-P
	Inside mirror	F = 11.37η_p² = 0.37**	F = 0.06η_p² = 0.01	F = 0.05η_p² = 0.01	F = 0.17η_p² = 0.01	F = 0.26η_p² = 0.01	N > E
	Right mirror	F = 1.29η_p² = 0.06	F = 2.78η_p² = 0.13	F = 0.04η_p² = 0.02	F = 0.01η_p² = 0.01	F = 0.01η_p² = 0.01
Number of glances	Front	F = 0.04η_p² = 0.01	F = 0.06η_p² = 0.01	F = 0.04η_p² = 0.01	F = 1.39η_p² = 0.07	F = 0.05η_p² = 0.03
	Additional task	F = 2.45η_p² = 0.11	F = 1.72η_p² = 0.08	F = 6.4η_p² = 0.25*	F = 5.11η_p² = 0.21*	F = 1.67η_p² = 0.08	Self-P > Syst-P	Within E group, AD > MD
	Dashboard	F = 0.07η_p² = 0.01	F = 11.65η_p² = 0.38**	F = 2.34η_p² = 0.14	F = 2.32η_p² = 0.01	F = 0.25η_p² = 0.01	AD > MD
	Left mirror	F = 0.04η_p² = 0.01	F = 6.41η_p² = 0.25*	F = 3.31η_p² = 0.15	F = 0.02η_p² = 0.01	F = 0.15η_p² = 0.01	MD > AD
	Inside mirror	F = 14.22η_p² = 0.41*	F = 0.13η_p² = 0.01	F = 0.02η_p² = 0.01	F = 0.10η_p² = 0.01	F = 0.03η_p² = 0.01	N > E
	Right mirror	F = 0.05η_p² = 0.01	F = 1.08η_p² = 0.05	F = 0.01η_p² = 0.01	F = 0.06η_p² = 0.01	F = 0.01η_p² = 0.01
Glance durations to the additional task	Mean	F = 2.14η_p² = 0.10	F = 18.25η_p² = 0.5**	F = 0.36η_p² = 0.02	F = 5.95η_p² = 0.24*	F = 2.05η_p² = 0.10	AD > MD	Within E group, AD > MD
Glance durations to the additional task	90th percentile	F = 1.9η_p² = 0.09	F = 21.3η_p² = 0.53**	F = 2.23η_p² = 0.10	F = 3.55η_p² = 0.15	F = 1.1η_p² = 0.05	AD > MD
Driving	Speed	F = 3.57η_p² = 0.16	F = 0.75η_p² = 0.04	F = 0.84η_p² = 0.04	F = 0.14η_p² = 0.01	F = 0.07η_p² = 0.01
	Number of overtakings per km	F = 6.13η_p² = 0.24*	F = 1.51η_p² = 0.07	F = 0.01η_p² = 0.01	F = 1.51η_p² = 0.07	F = 0.02η_p² = 0.01	E > N
	Percentage time with PA2 on	F = 5.1η_p² = 0.18*	—	F = 0.01η_p² = 0.01	—	F = 2.79η_p² = 0.12	E > N
	Percentage time on the left lane	F = 5.4η_p² = 0.22*	F = 0.14η_p² = 0.01	F = 1.47η_p² = 0.07	F = 0.64η_p² = 0.03	F = 0.12η_p² = 0.01	E > N
Additional task	Number of trials completed per minute	F = 0.97η_p² = 0.05	F = 1.15η_p² = 0.06	F = 0.83η_p² = 0.04	F = 0.91η_p² = 0.05	F = 0.23η_p² = 0.01
	Percentage correct	F = 2.7η_p² = 0.13	F = 6.97η_p² = 0.27*	F = 0.43η_p² = 0.02	F = 0.02η_p² = 0.01	F = 0.61η_p² = 0.04	MD > AD
	Percentage misses	F = 3.35η_p² = 0.15	F = 9.07η_p² = 0.32**	F = 2.04η_p² = 0.09	F = 0.05η_p² = 0.01	F = 0.44η_p² = 0.02	AD > MD
	Percentage errors	F = 0.86η_p² = 0.04	F = 0.08η_p² = 0.01	F = 1.68η_p² = 0.08	F = 0.04η_p² = 0.01	F = 0.41η_p² = 0.02
	Percentage time task on	F = 9.23η_p² = 0.33**	F = 0.58η_p² = 0.03	—	F = 0.18η_p² = 0.01	—	E > N
	Mean response time	F = 0.54η_p² = 0.03	F = 0.01η_p² = 0.01	F = 1.6η_p² = 0.07	F = 1.49η_p² = 0.07	F = 0.19η_p² = 0.01
	Number of off-switches	F = 9.58η_p² = 0.34**	F = 0.54η_p² = 0.03	—	F = 0.18η_p² = 0.01	—	N > E
NASA-TLX	Mental demand	F = 1.01 η_p² = 0.05	F = 2.75η_p² = 0.13	F = 4.1η_p² = 0.17*	F = 0.41η_p² = 0.02	F = 0.15η_p² = 0.01	Syst-P > Self-P
	Physical demand	F = 0.01η_p² = 0.01	F = 2.12η_p² = 0.10	F = 0.07η_p² = 0.01	F = 0.35η_p² = 0.02	F = 0.03η_p² = 0.01
	Temporal demand	F = 2.21η_p² = 0.10	F = 0.19η_p² = 0.01	F = 2.01η_p² = 0.10	F = 1.15η_p² = 0.06	F = 0.03η_p² = 0.01
	Performance	F = 1.04η_p² = 0.06	F = 0.02η_p² = 0.01	F = 0.49η_p² = 0.03	F = 0.03η_p² = 0.02	F = 0.71η_p² = 0.04
	Effort	F = 0.13η_p² = 0.01	F = 11.94η_p² = 0.4**	F = 0.91η_p² = 0.05	F = 0.51η_p² = 0.03	F = 0.24η_p² = 0.01	MD > AD
	Frustration	F = 5.07η_p² = 0.22*	F = 0.09η_p² = 0.01	F = 3.07η_p² = 0.15	F = 1.03η_p² = 0.05	F = 0.03η_p² = 0.01	N > E
	Global score	F = 0.63η_p² = 0.03	F = 2.14η_p² = 0.11	F = 1.2η_p² = 0.06	F = 0.97η_p² = 0.05	F = 0.06η_p² = 0.01

Note. Interactions with p values > .05 were excluded for space reasons. The same degrees of freedom are used. EL = experience level; AL = automation level; TP = task pacing; N = novice group; E = experienced group; Self-P = self-paced task; MD = manual driving; AD = automated driving; Syst-P = system-paced task; PA2 = Volvo Pilot Assist generation 2.

p < .05. **p < .01.

Subjective Ratings on NASA-TLX

A main effect of the type of task on the mental demand subscale was found. When the task was system-paced, the drivers reported a higher mental demand (M = 62.39, SE = 3.89) than when it was self-paced (M = 57.9, SE = 3.75). Also, scores on the effort subscale were significantly higher in the manual (M = 78.59, SE = 3.14) than the automated driving conditions (M = 66.85, SE = 5.47). Finally, the novice group reported higher levels of frustration (M = 56.36, SE = 4.5) than the experienced group (M = 41.11, SE = 5.02).

Additional Task

Similar mean response times (approximately 2.1 seconds) and number of trials completed were found in all conditions (approximately 20 trials/min). The automation experienced drivers kept the task on for significantly more time than the novice drivers when it was self-paced (Figure 3 and Table 3). In addition, the novice group switched the task off more times (M = 2.78, SE = 0.46) than the automation experienced group (M = 0.5, SE = 0.54). These effects were observed in the manual and automated conditions.

Figure 3.

Boxplots showing the significant effects found on percentage correct and missed responses (main effect of automation level) and percentage time with the additional task on (main effect of automation experience).

Regarding the quality of the additional task performance, average scores were high in general (>90% correct responses in all conditions). Despite this, a main effect of automation level was found for the percentage correct responses and percentage misses (see Table 3). When driving manually, the participants obtained higher percentage correct responses and lower percentage misses than when driving with automation (see Figure 3 and Table 3).

Driving Variables

Compared to the automation novice drivers, the experienced drivers performed more overtakings per kilometer (novice M = 0.12, SE = .02; experienced M = 0.21, SE = 0.2), used the PA2 for a greater percentage of the time (novice M = 72%, SE = 4.01; experienced M = 84,4%, SE = 4.64), and spent more time in the left lane (novice M = 9.87%, SE = 2.65; experienced M = 19.32%, SE = 3.06). Average speeds were similar in all conditions (about 100 km/h), and no significant differences were found between them.

Glance Behavior

Glance duration distributions to the front and tablet per condition and group are shown in Figure 4. The participants were more likely to direct longer glances toward the tablet than to the front in all conditions. Moreover, automation experienced drivers tended to glance at the tablet for longer periods than the novice group. This effect becomes more pronounced in the automated conditions, as shown by the more right-skewed histograms of the experienced drivers group. In contrast, the novice drivers directed longer glances to the front in all conditions.

Figure 4.

Histograms of the probability of all glance durations to the front and tablet in each condition and for both groups of drivers.

The statistical analyses indicated that the drivers spent more time looking to the front and left mirror in the manual than the automated driving condition (Figure 5 and Table 3). Also, the glance frequency to the left mirror was significantly higher in the manual conditions (automated M = 2.21, SE = 0.47; manual M = 2.68, SE = 0.51). In contrast, more time was spent looking (Figure 5) and more glances were directed to the dashboard (which includes the PA2 keypad on the steering wheel) in the automated than in the manual drives (number of glances per minute in manual driving M = 1.9, SE = 0.42; automated driving M = 3.61, SE = 0.48). Additionally, two interaction effects were observed between the automation level and automation experience. The automation experienced drivers directed significantly more and longer glances to the tablet in the automated conditions (number of glances per minute M = 24.89, SE = 1.14; glance duration M = 1.34 seconds, SE = 0.07) than the manual conditions (number of glances per minute M = 23.659, SE = 1.03; glance duration M = 1.18 seconds, SE = 0.07), while no differences were observed in the novice group.

Figure 5.

Distribution of visual attention (percentages time spent looking at each area of interest) by the novice (N) and experienced (E) drivers during the manual and automated conditions. The figures represent the location of the different areas of interest from the driver’s perspective.

Two main effects of automation experience were observed. As shown in Figure 5, the drivers with prior automation experience spent more time looking at the additional task than the novice group. In contrast, the novice group spent more time (Figure 5) and looked more frequently to the inside mirror of the car (novices M = 3.76, SE = 0.49; experienced M = 0.9, SE = 0.56). In the self-paced task condition, the drivers spent more time looking (Figure 5) and directed more glances to the additional task than when it was system-paced (self-paced M = 23.98, SE = 0.76; system-paced M = 22.35, SE = 0.77). Additionally, when the task was system-paced, more time was spent looking to the left mirror. This factor also interacted with the automation experience variable. When the task was self-paced, the automation experienced drivers spent less time looking to the front than the novice (Figure 5 and Table 3).

Discussion

In contrast to popular belief (Kyriakidis, Happee, & De Winter, 2015), our findings suggest that Level 2 automation, exemplified by the PA2 system, did not support the drivers in the performance of an additional visuomotor task. When using the system, the drivers did not devote more visual attention to the additional task, and their performance on the additional task was lower than in the manual driving conditions. Regardless of the automation level, the task was tactically interrupted whenever that option was available, resulting in lower mental demands. These results contradict both the idea that mental load is reduced with increasing automation levels and that it will be easier to engage in other tasks when driving Level 2 automated (de Winter et al., 2014).

Effects of Automation Level (Hypothesis 1)

Contrary to our assumption and the evidence on Levels 1 and 3 (de Winter et al., 2014), the automation feature of study led to decrements in performance on the additional task. The fact that those drivers with automation experience directed longer glances at the additional task when driving with automation, as hypothesized, is particularly interesting as it indicates that the drivers could not use the larger share of visual resources to improve their performance. This could be explained by a general reduction in their attentional capacity (see malleable attentional resource theory, Young & Stanton, 2002) or the effort invested (Hancock & Warm, 1989). Also, the attention required to monitor the system may have affected the performance of the additional task, and the effort rating might only reflect the driver’s perception of the driving task. Potential increments in monitoring requirements may be explained by the fact that changes in the PA2 status were only visually informed through the system symbol, thus requiring the drivers to frequently look at it.

Driving with PA2 instead of manually did not overtly influence driving behavior, but the changes in glance behavior reflect a shift of focus away from the driving task itself to a monitoring behavior, indicating a transfer of control from a tracking and regulatory level to a monitoring level within the DiC model (Hollnagel et al., 2003). While tracking tasks can be performed with little effort (Hollnagel et al., 2003), monitoring tasks may rely more on cognitive resources to better anticipate changes in the traffic and system state. According to the cognitive control hypothesis (Engström, Markkula, Victor, & Merat, 2017), “cognitive load selectively impairs driving subtasks that rely on cognitive control but leaves automatic performance unaffected” (p. 3). In our study, we can assume that the additional task itself and the task switching requirements placed some cognitive demands on the drivers. As a result, the monitoring task in the automated conditions was probably more disruptive to the additional task performance than the tracking task in the manual conditions.

Effects of Prior Level 2 Automation Experience (Hypothesis 2)

Based on prior observations in experienced drivers with Level 1 automation (e.g., Kircher et al., 2014; Kopf & Nirschl, 1997; Naujoks et al., 2016), we assumed that experience with a Level 2 automation system would lead to improved performance on the additional task. This should have shown in interaction effects between the automation and experience factors with similar behavior and performance for the two driver groups in the manual conditions but different in the automated conditions. The only interaction effects for those factors were found for glance intensity parameters toward the tablet with the additional task, where the drivers with automation experience glanced at the tablet more often and for longer time periods under automation than automation novices did.

The only task-related difference between automation novices and experienced drivers was that the latter activated the additional task for a larger percentage of time; however, this effect was also found in manual driving. Possibly, the experienced group was more familiar with the car itself and felt more content with keeping the additional task on than the novice group. This could perhaps explain why the novice group scored higher in the frustration subscale. Another explanation is that drivers are willing to engage more in the additional task with automation but that this willingness shifts to manual driving over time. Should this effect be confirmed in other studies, it must be considered carefully as drivers who are used to automation systems might involuntarily disengage more from the driving task also when driving manually again. Another finding in this study indicates, however, that drivers with automation experience do not necessarily become passive as they overtake more than the automation novices and spent more time in the left lane while also using the automation system more. They might have internalized the system functionality more and are therefore able to integrate overtaking maneuvers into the behavior of the automated system, whereas novices prefer to keep in the same lane, thereby keeping the level of situation demand lower. All in all, this may indicate that automation does change behavior but also that a person’s driving style is changed more persistently, carrying over to manual driving.

Effects of Additional Task Pacing (Hypothesis 3)

In line with Platten et al.’s (2014) observations, we hypothesized that the option to turn off the additional task would lead to lower workload and better performance on the additional task. In the framework of the DiC model, such strategy would show that drivers strategically manage the combination of tasks that, across the different control levels, can be performed safely in a given situation. In our study, while the perceived mental load was indeed higher in the system-paced condition, the performance on the task did not change, even though one could have expected improved performance in the self-paced condition because of the possibility to turn the task off when it was difficult to pay attention to it. The higher mental load might therefore reflect that in the system-paced condition, the drivers pushed themselves, trying to solve the additional task, even though the driving situation was complex. Such a strategy would lead to mental fatigue eventually. In the self-paced condition, the drivers seem to have made use of the strategy to turn off the task during more complex and/or visually demanding traffic situations as they spent more time looking at the tablet during the phases when the tablet was on in the self-paced condition than during the system-paced condition. Drivers experienced with automation did not turn off the additional task as much as the automation novices did, which may indicate that they are better able to use the PA2 system to enable them to execute additional tasks. It must be noted, though, that they also kept the task on more than the automation novices in the manual condition, which supports the assumption that driving with Level 2 automation may eventually lead to a greater disengagement from driving, even when driving manually again.

Methodological Considerations

The study was conducted in real traffic, with a Level 2 equipped vehicle, and with drivers who had already gained experience with the same system used in the study. Thus, external validity was very high. Despite this, our findings cannot be generalized to other Level 2 systems available as they may offer different technical solutions than the PA2 (e.g., different interfaces), which may influence driver behavior in a different way. Furthermore, one drawback of this study is that the additional task was not intrinsically motivating, making it conceptually different from a real-life distraction task. As the task was the same throughout all conditions, this should affect all conditions similarly. Also, sample size was somewhat low and male biased because it was difficult to recruit drivers with Level 2 experience as the cars with that functionality had only been released half a year before the study was conducted. Furthermore, sporadic disengagements of the LKA system occurred in some of the experimental sessions. Consequently, drivers might have increased the visual monitoring of the system status, leading to a worse additional task performance. This relationship was not analyzed here.

Time and budget limitations led to recruiting drivers of different types of middle-class vehicles for the control group. While it is theoretically possible that not being familiar with a Volvo S90 had a confounding effect, it does not appear to be very likely as the driving task on the freeway was very simple, not even requiring gear changes. For the same reasons, there were differences in driving experience between the novice and experienced group.

Conclusions

Our findings indicate that it might not be easier to perform other visuomotor tasks in a Level 2 automated vehicle than when driving manually, but rather the opposite may occur. Moreover, prior experience with the Level 2 automation system used in the study does not seem to mitigate this effect. These results may be somewhat surprising given prior research on Levels 1 and 3. Probably, the additional task used in this study interfered more with the monitoring demands placed by the system than with the demands stemming from the manual control of the vehicle position. If so, this highlights the importance for automation designers to analyze the nature of the tasks that are automated and anticipate the effects on driver performance via theoretical models and empirical testing. We encourage future studies to continue investigating the effects of performing other tasks while driving with Level 2 automation and its implications for safety.

Key Points

The drivers unexpectedly performed worse on a visuomotor additional task in Level 2 automated driving, exemplified by the Volvo Pilot Assist generation 2 (PA2) system, compared to manual driving.

Prior experience with Level 2 automation did not mitigate this effect.

Level 2 automated driving demanded an increased visual attention to the dashboard and system controls, associated with the supervisory demands.

The drivers, particularly the automation novices, actively compensated the increased demands by switching off the automated system or the additional task.

Footnotes

Acknowledgements

The primary funding body for this project was FFI as part of the HATric project, an associated project at SAFER–Vehicle and Traffic Safety Centre at Chalmers. Additional funding was provided by the European Marie Curie ITN project “HFAuto” (Human Factors of Automated Driving; PITN-GA-2013-605817).

ORCID iD

Ignacio Solís-Marcos

Ignacio Solís Marcos earned an MSc in human neuropsychology from the University of Seville and is currently a Marie Curie Research Fellow in the Human Factors in Highly Automated Vehicles project (HF-Auto). He belongs to the Department of Human Factors in the Transport System at the Swedish National Road and Transport Research Institute (VTI).

Christer Ahlström is a researcher at the Department of Human Factors in the Transport System at the Swedish National Road and Transport Research Institute (VTI). He received his PhD in biomedical engineering from Linköping University in 2008.

Katja Kircher is research leader and belongs to the Department of Human Factors in the Transport System at the Swedish National Road and Transport Research Institute (VTI). She received her PhD in industrial ergonomics from Linköping University in 2002, where she also has been associate professor in the department of Behavioral Sciences and Learning since 2015.

References

Banks

V. A.

Eriksson

O’Donoghue

Stanton

N. A.

(2018). Is partially automated driving a bad idea? Observations from an on-road study. Applied Ergonomics, 68, 138–145. doi:10.1016/j.apergo.2017.11.010

Carsten

Lai

F. C. H.

Barnard

Jamson

A. H.

Merat

(2012). Control task substitution in semiautomated driving. Human Factors, 54, 747–761. doi:10.1177/0018720812460246

de Winter

J. C. F.

Happee

Martens

M. H.

Stanton

N. A

. (2014). Effects of adaptive cruise control and highly automated driving on workload and situation awareness: A review of the empirical evidence. Transportation Research Part F: Traffic Psychology and Behaviour, 27, 196–217. doi:10.1016/j.trf.2014.06.016

Engström

Markkula

Victor

Merat

(2017). Effects of cognitive load on driving performance. Human Factors, 59, 734–764. doi:10.1177/0018720817690639

Google. (2017). [Test route on the E.4. highway near Linköping, Sweden]. Retrieved from https://www.google.se/maps/@58.4358227,15.556334,14.93z.

Hancock

P. A.

Warm

J. S.

(1989). A dynamic model of stress and sustained attention. Human Factors, 31, 519–537. doi:10.1177/001872088903100503

Hart

S. G.

Staveland

L. E.

(1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52(C), 139–183. doi:10.1016/S0166-4115(08)62386-9

Hollnagel

Nåbo

Lau

I. V.

(2003). A systemic model for driver-in-control. In PROCEEDINGS of the Second International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design (pp. 86–91). Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.121.2294&rep=rep1&type=pdf

Kircher

Fors

Ahlstrom

(2014). Continuous versus intermittent presentation of visual eco-driving advice. Transportation Research Part F: Traffic Psychology and Behaviour, 24, 27–38. doi:10.1016/j.trf.2014.02.007

10.

Kircher

Larsson

Hultgren

J. A.

(2014). Tactical driving behavior with different levels of automation. IEEE Transactions on Intelligent Transportation Systems, 15(1), 158–167. doi:10.1109/TITS.2013.2277725

11.

Kopf

Nirschl

(1997). Driver-vehicle interaction while driving with ACC in borderline situations. In Proceedings of the 4th World Congress on Intelligent Transportation Systems, (2128, Paper No. 205). Retrieved from http://trid.trb.org/view.aspx?id=635723

12.

Kyriakidis

Happee

De Winter

J. C. F.

(2015). Public opinion on automated driving: Results of an international questionnaire among 5000 respondents. Transportation Research Part F: Traffic Psychology and Behaviour, 32, 127–140. doi:10.1016/j.trf.2015.04.014

13.

Larsson

A. F. L.

Kircher

Andersson Hultgren

(2014). Learning from experience: Familiarity with ACC and responding to a cut-in situation in automated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 27, 229–237. doi:10.1016/j.trf.2014.05.008

14.

Llaneras

R. E.

Salinger

Green

C. A.

(2013). Human factors issues associated with limited ability autonomous driving systems: Drivers’ allocation of visual attention to the forward roadway. In PROCEEDINGS of the Seventh International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design (pp. 92–98). Retrieved from http://drivingassessment.uiowa.edu/sites/default/files/DA2013/Papers/015_Llaneras_0.pdf

15.

Naujoks

Purucker

Neukum

(2016). Secondary task engagement and vehicle automation—Comparing the effects of different automation levels in an on-road experiment. Transportation Research Part F: Traffic Psychology and Behaviour, 38, 67–82. doi:10.1016/j.trf.2016.01.011

16.

Östlund

Nilsson

Carsten

Merat

Jamson

. . . Brouwer

. (2004). Haste. Deliverable 2. HMI and Safety-Related Driver Performance. Retrieved from http://www.its.leeds.ac.uk/projects/haste/Haste%20D2%20v1-3%20small.pdf

17.

Platten

Schwalm

Hülsmann

Krems

(2014). Analysis of compensative behavior in demanding driving situations. Transportation Research Part F: Traffic Psychology and Behaviour, 26, 38–48. doi:10.1016/j.trf.2014.06.006

18.

SAE International. (2016). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles, J3016_201609. Warrendale, PA: Author.

19.

Saffarian

de Winter

J. C. F.

Happee

(2012). Automated driving: Human-factors issues and design solutions. Proceedings of the Human Factors and Ergonomics Society 56th Annual Meeting (pp. 2296–2300). Santa Monica, CA: Human Factors and Ergonomics Society.

20.

Stanton

N. A.

Marsden

(1996). From fly-by-wire to drive-by-wire: Safety implications of automation in vehicles. Safety Science, 24(1), 35–49. doi:10.1016/S0925-7535(96)00067-7.

21.

Volvo Cars. (n.d.). Owner’s manuals. Retrieved from https://volvornt.harte-hanks.com/manuals/2017/S90_OwnersManual_MY17_en-US_TP22301.pdf

22.

Warm

J. S.

Parasuraman

Matthews

(2008). Vigilance requires hard mental work and is stressful. Human Factors, 50, 433–441. doi:10.1518/001872008X312152.

23.

Wickens

C. D.

(2008). Multiple resources and mental workload. Human Factors, 50, 449–455. doi:0.1518/001872008X288394

24.

Young

M. S.

Stanton

N. A.

(2002). Malleable attentional resources theory: A new explanation for the effects of mental underload on performance. Human Factors, 44, 365–375. doi:10.1518/0018720024497709