The Impact of Higher Levels of Automation on Performance and Situation Awareness

Levels of Automation (LOAs)

Researchers have introduced generic frameworks that classify the ways in which automation can assist humans (e.g., Endsley, 1987; Kaber & Endsley, 1997; Ntuen & Park, 1988; Parasuraman, Sheridan, & Wickens, 2000; Sheridan & Verplank, 1978). For example, Parasuraman et al. (2000) suggested that “automation can be applied to four broad classes of functions” (p. 288): (a) information acquisition, which represents gathering, registering, and preprocessing information from the environment; (b) information analysis, which represents the conscious perception of information and the conduct of cognitive operations, such as the integration of and reasoning about information; (c) decision selection, which represents choosing an action; and (d) action implementation, which represents executing the action. In each of these functions, automation systems can assist their users to varying levels. Parasuraman et al. (2000) also suggested how to classify these LOAs for each function. For example, a low level of information-analysis automation would extrapolate data over time. A moderate level would integrate data streams into a single value, and a high level would provide context-dependent summaries of the situation. With regard to decision-selection automation, Parasuraman et al. (2000) suggested systems be classified on the basis of the 10 LOAs of Sheridan and Verplank (1978). When one simplifies these LOAs by considering only decision-selection functions, a low level of decision-selection automation may recommend a set of actions. A moderate level may suggest one action, and a high level of automation may decide what to do, independent of the user.

This is why this paper defines LOA according to the individual levels of the four automation functions:

LOA = ( Level of information - acquisition automation , Level of information - analysis automation , Level of decision - selection automation , Level of action - implementation automation ) .

For example, McFarland (1997) introduced the User Request Evaluation Tool, which provided automatic conflict detection in en route air traffic control. The tool automated information acquisition to a high level, because it integrated various information sources (e.g., radar tracks, flight plan information) to predict a flight path. The level of the information-analysis automation would be low if it predicted only the development of single flight paths without considering the interrelation of the flight paths of all aircraft. The level of the decision-selection automation would be low if the tool did not propose actions. Finally, the level of the action implementation would be low if the user still had to implement actions manually. The LOA of the tool would thus be LOA = (high, low, low, low).

Inconsistent Effects of Levels of Automation on Performance and Situation Awareness

Performance and situation awareness were often used as criteria to evaluate the effects of automation. Situation awareness is an important and multidimensional construct in human factors research. Three special issues have been devoted to this topic in the past decades (see Gilson, 1995; Pritchett, 2015; Stanton, 2010). According to Endsley (1995b), situation awareness comprises the perception of relevant information from the environment, the comprehension of the current situation, and the projection of the situation into the near future. Parasuraman, Sheridan, and Wickens (2008) considered situation awareness as useful for understanding and predicting human reactions to complex systems. For example, researchers used situation awareness successfully to analyze incidents and accidents (e.g., Durso, Truitt, Hackworth, Crutchfield, & Manning, 1998; Gaba, Howard, & Small, 1995; Jones & Endsley, 1996; cf. Flach, 1995) and to predict operators’ ability to recover from automation failures (e.g., Endsley & Kiris, 1995). Situation awareness can be a predictor of how well operators reacted to automation failures (Onnasch, Wickens, Li, & Manzey, 2014).

Meta-analyses have been conducted to investigate the effects of higher LOAs on situation awareness and performance—measured in situations without automation failures (Onnasch et al., 2014; Wickens, Li, Santamaria, Sebok, & Sarter, 2010). Results indicated that the provision of higher LOAs can result in inconsistent patterns of changes in performance and situation awareness. For example, Wickens et al. (2010) found that most studies had only zero or even negative performance changes associated with systems with increases in level of information-analysis and decision-selection automation. In contrast, Onnasch et al. (2014) reported significant increases in performance and decreases in situation awareness when the systems had increased LOAs in information analysis and decision selection. The effect sizes that Onnasch et al. reported varied greatly, which points to moderator effects. Onnasch et al. considered moderator effects of different classification methods of LOAs, the training of the participants, the task setting, and the nature of the display used by the automation. However, only the classification method seemed to play an important role for performance; levels of significance were not reported. It is not yet fully clear what might have caused the varying effects of higher information-analysis and decision-selection automation.

Individual differences in cognitive abilities may moderate information-analysis and decision-selection automation effects on performance and situation awareness. Consideration of these effects may explain different outcomes between studies in the literature, which did not account for sample/population differences (Calhoun et al., 2009; Cummings & Mitchell, 2007; Endsley & Kaber, 1999; Ferris, Sarter, & Wickens, 2010; Kaber et al., 2000; Kaber & Endsley, 2004; Lin, Yenn, & Yang, 2010; Manzey et al., 2011; Parasuraman & Riley, 1997; Sarter & Schroeder, 2001). For example, Manzey et al. (2011) asked 14 advanced medical students to execute a simulated mastoidectomy. The students required significantly less time to complete the surgery task when they received assistance from the navigated control system, LOA = (moderate, moderate, moderate, moderate), than from the image-guided navigation system, LOA = (moderate, moderate, low, low). In contrast, six novice Air Force personnel completed their mission slower when they controlled unmanned aerial vehicles with a system of which the LOA changed from LOA = (high, high, low, low) to LOA = (high, high, moderate, low) in the study of Calhoun et al. (2009). The authors classified their system LOA on the basis of the scheme of Sheridan and Verplank (1978), which we reclassified here to compare the results with the other studies.

With regard to situation awareness, Kaber and Endsley (2004) asked 30 university students to conduct an abstract radar-monitoring task. Comprehension scores of situation awareness were higher for blended decision making, LOA = (moderate, moderate, moderate, high), when compared to batch processing, LOA = (moderate, low, low, high). We also reclassified the LOAs here because Kaber and Endsley (2004) applied the classification method of Kaber and Endsley (1997). In contrast to the study of Kaber and Endsley (2004), the advanced medical students of Manzey et al. (2011) were better able to answer questions about the specifics of anatomy when they executed a mastoidectomy with an image-guided navigation system, LOA = (moderate, moderate, low, low), than with a navigated control system, LOA = (moderate, moderate, moderate, moderate).

The cognitive-ability profiles of the advanced medical students (Manzey et al., 2011), the university students (Kaber & Endsley, 2004), and the Air Force personnel (Calhoun et al., 2009) likely differed but were not measured in these studies. Moreover, no available study investigated the impact of cognitive abilities on utilizing different levels of information-analysis and decision-selection automation to attain performance and situation awareness.

Relevance of Cognitive-Ability Profiles

Cognitive-ability profiles may play a moderating role because higher levels of information-analysis and decision-selection automation may increase the complexity of the mental model underlying task completion. Kaber et al. (2005) expected the complexity to increase because systems with higher levels of information-analysis and decision-selection automation perform more tasks that the operators must integrate into their mental model. This integration is crucial for successful joint task completion (Christoffersen & Woods, 2002; Rauterberg, 1992). For the integration, the operators need to collect the information the system uses as inputs, mirror the automated reasoning processes, collect information on the behavior of the automation, build the model for the automation, and modify it if necessary (Ramsbottom & Baker, 2003). Therefore, the operators’ information-processing ability and working-memory capacity may be more engaged. Appendix A provides definitions of the functions of information-processing ability and working-memory capacity under consideration here.

Operators deal with the increased cognitive demands differentially, as proposed by the elaboration likelihood model (Petty & Cacioppo, 1986). On the one hand, operators can avoid the cognitive effort and apply heuristics that reduce the intrinsic cognitive load (see Atwood & Polson, 1976). On the other hand, operators can feel stimulated to process this additional information and establish a more complex mental model. As Petty and Cacioppo (1986) pointed out, cognitive abilities can determine whether humans prefer heuristics or deep processing of information. For operators with higher abilities, the deep processing of the additional information can be easier. These operators likely deeply process information, attain better performance, and better maintain situation awareness. For operators with lower abilities, however, deep processing can be more difficult. These individuals likely use heuristics to solve the task. Consequently, their mental models would be insufficiently developed, which would lead to worse performance and lower situation awareness. In summary, operators with higher information-processing ability and working-memory capacity may profit from higher levels of information-analysis and decision-selection automation, and operators with lower abilities may suffer under such automation.

Hypotheses

The main goal of this study was to investigate whether information-processing ability and working-memory capacity moderate the effect of increasing levels of information-analysis and decision-selection automation on performance and situation awareness. Two propositions reflected in the following hypotheses underlay this goal:

The following hypotheses reflected the main research goal:

Participants

Sixty (29 male, 31 female) university students with normal or corrected-to-normal vision participated in this study and received about US$80 for their participation. The average age was 23.6 years (SD = 3.48; range = 19–30 years).

Experimental Design

This study followed a within-subjects design with two independent variables: automation configuration (three levels) and trial number (four levels). Automation configuration represented the manipulation of the level of information-analysis and decision-selection automation: The automation assistance varied between LOA1 = (moderate, low, low, low), LOA2 = (moderate, moderate, low, low), and LOA3 = (moderate, moderate, moderate, low). The automation was perfectly reliable, because failures would have depended on the currently active automation configuration and biased the manipulation of the variable. Trial number reflected the number of times we confronted the participants with the task. We used four trials to yield sufficient data for statistical analysis.

Simulation Environment

Task

We asked the participants to perform a simplified air traffic control task using a modified version of the Micro Air Ground Integration Environment (MAGIE). MAGIE allows investigating solutions for challenges in air traffic control (e.g., Oberheid, Weber, & Rudolph, 2009) and effects of automation (e.g., Oberheid, Hasselberg, & Söffker, 2011). MAGIE simulated inbound air traffic in the terminal area of an airport (Figure 1). Two types of aircraft entered the simulated airspace along standard approach routes: “Equipped” aircraft descended automatically from their approach routes to the runway. They were equipped with a high-end flight management system that enabled the aircraft to fly a continuous descent approach. This procedure reduces noise and emissions (see Alam et al., 2010; Warren & Tong, 2002). The aircraft without this equipment turned from the end of their approach routes automatically to the east and flew parallel to the runway away from the runway along the downwind leg (Lecchini, Glover, Lygeros, & Maciejowski, 2005).

OPEN IN VIEWER

Figure 1.

Screenshot from Micro Air Ground Integration Environment visually representing the display used for interaction, key concepts, and the routes that unequipped and equipped aircraft use to arrive at the runway.

The participants needed to give heading, speed, and altitude instructions to the “unequipped” aircraft in the control zone to meet three objectives:

The participants had to ensure that all aircraft remained at a horizontal distance of at least 3 NM (5.56 km) apart from each other, according to the standard radar separation requirement (see Thompson, 1997). Participants were neither instructed nor required to ensure that a certain vertical distance remained between the aircraft to avoid floor effects due to the high level of difficulty (based on the results of Oberheid et al., 2009). We asked the participants to ignore the vertical separation because we expected it to be more difficult to ignore the horizontal separation, due to the way the interface was designed.

The participants had to ensure that aircraft not equipped with the high-end flight management system adhered to minimum and maximum speed and altitude limits. The limits depended on the aircraft position and ensured that the aircraft systematically reduced speed and altitude so they could be landed on the runway.

The participants had to ensure that as many aircraft as possible arrived at the runway. Therefore, the participants had to instruct the unequipped aircraft to leave the downwind leg at a certain time, so that the planes turned in the direction of the centerline and followed it to the runway. Aircraft disappeared from the simulation as soon as they reached the end of the downwind leg. We implemented this simplification of real-world air traffic control because the aircraft left the control zone at the end of the downwind leg and the participants would no longer be able to correct their mistakes.

Several tasks had to be executed to achieve the objectives. The tasks can be sorted into the function classes propagated by the four-stage model of information processing (Parasuraman et al., 2000):

1. Information acquisition applied to registering information provided by simulated radar sensors and obtained from previous instructions. This information needed to be preprocessed to identify the current position and status of each aircraft. Preprocessing was also necessary to organize the aircraft in a timeline, enabling prioritization of aircraft, and to highlight current deviations from the separation, speed, and altitude regulations.

We considered the prioritizing and highlighting information as information-acquisition tasks for two reasons: First, Parasuraman et al. (2000) used the “organization of incoming information according to some criteria, e.g., a priority list” (p. 288) and “highlighting of some part of the information” (p. 288) as examples for acquisition. Second, this approach allowed us to display violations in all automation configurations, which counteracted the criticism of Süß (1999) that dynamic tasks provided unclear objectives, such that the reliability of the performance measures would be reduced.

2. Information analysis applied to extrapolating the current status of each aircraft into the future, integrating the current paths of the equipped and unequipped aircraft, and identifying potential violations of separation, speed, and altitude regulations. On this basis, ideal approach paths needed to be identified for the unequipped aircraft. These paths then had to be related to the current paths to identify deviations and a related need for actions.

3. Decision selection applied to defining instructions that would make the unequipped aircraft follow ideal approach paths, evaluating these instructions, and if necessary, modifying them. The instructions then needed to be prioritized, such that it was clear when certain instructions had to be issued.

4. Action implementation referred to instructing aircraft to change speed, altitude, or heading. Therefore, the participants had to click on the aircraft that should modify its speed, altitude, or heading. Then, MAGIE opened an interaction display (Figure 1) that provided the current speed, altitude, and heading instructions for the selected aircraft. In this display, the participants had to select the new target value and send it to the aircraft. Then, the automation masked the interaction display, and the aircraft started changing its flight profile such that it approached the target value.

Automation configurations

We developed three automation configurations by manipulating whether the participants or the automation executed the tasks (see Appendix B). Configuration 1 (Figure 2a) implemented LOA1 = (moderate, low, low, low). Configuration 2 (Figure 2b) implemented LOA2 = (moderate, moderate, low, low), and Configuration 3 (Figure 2c) implemented LOA3 = (moderate, moderate, moderate, low).

OPEN IN VIEWER

Figure 2.

Screenshots from the three automation configurations (for details, see Appendix B) of the simulation environment visually representing (a) the timeline and the highlighting of information (Configuration 1), (b) the additional targeting and ghosting (Configuration 2), and (c) the additional advisories (Configuration 3).

The moderate information-acquisition automation continuously displayed (a) the current positions of all aircraft together with the current and cleared states (speed and altitude), (b) the timeline, and (c) deviations to rules and limits on the screen (Figure 2a). The information was updated once per second. We considered the level of this information-acquisition automation as moderate because the automation organized and highlighted information without removing raw data (see Parasuraman et al., 2000).

The moderate information-analysis automation provided two types of visual cues to augment human cognition (Figure 2b): First, it continuously projected a ghost aircraft for each equipped aircraft in the simulation onto the approach path of the unequipped aircraft. Specifically, each ghost aircraft followed the approach path of the unequipped aircraft—in contrast to the real counterpart. At the intersection of the paths, each ghost aircraft merged with its real counterpart and was no longer visible. If the participants placed an unequipped aircraft close to a ghost aircraft, the unequipped aircraft would violate the separation rule at the intersection between its path and the path of the equipped aircraft. An alarm was provided only as soon as the separation rule was violated.

Second, the automation continuously projected a target aircraft for each unequipped aircraft in the simulation onto the centerline. These target aircraft showed the ideal position of the unequipped aircraft. If an unequipped aircraft was on top of its target on the centerline, the aircraft had the ideal position. If an unequipped aircraft was on the downwind leg and its target was on the centerline (as in Figure 2b), the aircraft needed to be turned in the direction of the centerline.

We considered this information-analysis automation as moderate because both visual cues eliminated the need to mentally integrate the approach paths of both aircraft types—a typical characteristic of moderate information-analysis automation according to Parasuraman et al. (2000). When the level of information-analysis automation was low, the target aircraft or ghost aircraft was not displayed.

The moderate decision-selection automation communicated necessary instructions to the participants as an advisory (Figure 2c). An example was to instruct the aircraft U18 to reduce its altitude to 3,000 feet in 5 s. The participants could ignore the advisory. The automation displayed these advisories underneath the aircraft symbol only when the participants placed the mouse cursor over a counter. The automation also displayed the counter underneath the aircraft symbol. The counter informed the participants exactly when they should give the subsequent instructions to the aircraft. The counter became visible 30 s prior to the time the participants should give the instruction. The automation masked the counter if the participants gave the instruction to the aircraft or if the participant gave another instruction that changed the situation. Alternatives to these instructions were not provided. Thus we considered this decision-selection automation as moderate, that is, Level 4 in the classification scheme set forth by Sheridan and Verplank (1978) and suggested by Parasuraman et al. (2000). When the level of decision-selection automation was low, the participants did not receive advisories.

Besides automation replacing pilot actions, action-implementation automation was not available. The level of the action-implementation automation was thus always low.

Measures

Performance

The performance score was computed as follows:

Performance = ( Compliance with the goal to enable as many aircraft as possible to be landed ) − ( Compliance with the separation rule ) − ( Compliance with the speed and altitude limits ) .

Details of how we quantified the indicators are provided in Appendix C. Larger positive numbers indicated better performance across a theoretical range from −1 to +1.

Procedure

Data collection for each participant occurred on two separate days and lasted about 4 hr per day. On the 1st day, the participants always controlled air traffic with MAGIE. The participants completed all training and test scenarios with one automation configuration before they were confronted with another one. Each automation configuration procedure consisted of the following steps: (1) The participants received a standard introduction in MAGIE and the automation configuration. (2) The participants completed the two training sessions (the first one without the situation awareness questionnaire, the second one with the questionnaire) and four test scenarios. Each scenario was completed in about 10 min. (3) The participants had a short break. Afterward, we repeated this procedure with another configuration until the participants had worked with all configurations.

On the 2nd day, we assessed the participants’ information-processing ability and working-memory capacity. Finally, the participants completed a demographic questionnaire.

We distributed the presentation order of the scenarios, the automation configurations, and the tests of the individual-difference measures randomly between participants to counterbalance order effects.

We instructed the participants that performance was more important than situation awareness to avoid participants’ actively memorizing the answers to the situation awareness probes during the assessment of performance (i.e., maximizing their scores on situation awareness at the expense of performance). We also told the participants to carefully monitor the automation because automation failures could occur. We did not tell the participants an exact indicator for the assumed automation reliability. However, the automation did not fail.

We did not ensure that the participants’ level of proficiency with an automation configuration met a specific criterion of expertise after training because it would have been difficult to ensure that the participants would be masters of the air traffic control task after training, as it takes years of training (see Hoffman, 1998). To avoid biased results, we used the trial number to statistically control differences in potential skill acquisition, in effects of individually differing reactions to skill acquisition, and in effects of the different automation configurations on skill acquisition.

Preanalyses

The data analyses were based on 57 participants because data were lost from 3 participants due to technical issues. We first calculated descriptive statistics of the measured variables (Table 1). These statistics showed that the means and standard deviations of the information-processing abilities were comparable to the characteristics of the test-norming group (Jäger et al., 1997). The test norming was population representative.

OPEN IN VIEWER

Table 1:

Descriptive Statistics and Intercorrelations Between the Dependent Measures and the Individual-Difference Measures

Measure	1	2	3	4	5	6	7	8	9	10	M	SD
Dependent measures
1. Performance a	—										0.51	0.71
2. Perception score of situation awareness a	.20	—								4	0.28	0.10
3. Comprehension score of situation awareness a	.30*	.60*	—								0.42	0.23
4. Prediction score of situation awareness a	.33*	.52**	.74**								0.26	0.24
Information-processing ability
5. Information-processing complexity	.26*	.35*	.28*	.16	—						102.11	9.06
6. Information-processing speed	.05	.19	.07	.15	.04	—					97.70	9.75
7. Verbal ability	.13	.03	.10	.26	.06	.06	—				100.89	9.90
8. Numerical ability	.26*	.22	.16	.07	.06	.05	.03	—			96.68	9.59
Working-memory capacity
9. Supervision	.11	.18	.19	.23	.06	.00	.15	.01	—		0.00	1.00
10. Coordination	.09	.25	.24	.17	.35**	.16	.04	.25	.13	—	0.77	0.10
11. Simultaneous storage and processing	.37**	.26	.26	.25	.57**	.29*	.24	.33*	.10	.13	0.00	1.00

Note. All levels of significance base on N = 57.

a

For calculating the bivariate correlations, we drew on the mean value (across trials and automation configurations) for each dependent variable. The Ms and SDs base on N = 684 (57 participants, who worked with three automation configurations and four trials).

*

p < .05 (two tailed). **p < .01 (two tailed).

Second, we ensured that the assumptions underlying the ANOVAs and the multiple regressions, which we used for evaluating the hypotheses, were not violated. The internal consistency (α) of performance was .77, the consistencies of the situation awareness scores were αs = .91. We calculated these consistencies by treating the scores’ indicators listed in Appendix C as individual items. Potential issues, including skewness in the distributions of the variables, multicollinearity, and outliers, were not present with one exception: The capacity to simultaneously store and process information was highly correlated with the ability to process complex information (Table 1). Thus we separately investigated (a) whether the working-memory capacities influenced the dependent measures, (b) whether the information-processing abilities influenced the dependent measures, (c) whether the working-memory capacities moderated effects of the automation configuration on the dependent measures, and (d) whether the information-processing abilities moderated effects of the automation configuration on the dependent measures.

Effects of Automation Configuration on Performance and Situation Awareness

We drew on repeated-measures ANOVAs to test Hypothesis 1a and Hypothesis 1b. We used the performance and the situation awareness scores as dependent variables and the automation configuration and trial number as independent variables—without considering the individual-difference measures. The presentation order had no significant main or interaction effect on any dependent variable and was thus excluded.

The results supported Hypothesis 1a and Hypothesis 1b, all ps < .001 (Table 2). Figure 3 shows that the performance of the participants improved with the LOA. Performance was lowest with LOA1 = (moderate, low, low, low), a bit better with LOA2 = (moderate, moderate, low, low), and best with LOA3 = (moderate, moderate, moderate, low). In contrast, the situation awareness scores decreased with an increasing LOA. The participants achieved their best scores with LOA1 = (moderate, low, low, low). These scores decreased with LOA2 = (moderate, moderate, low, low) and were worst with LOA3 = (moderate, moderate, moderate, low).

OPEN IN VIEWER

Table 2:

Summary of the Repeated-Measures ANOVAs Evaluating the Effects of Automation Configuration and Trial Number on Performance and Situation Awareness

Variable	F	dfs, dfe	p		η²p	1 – β
Performance
Automation configuration	24.16	2,112	<.001	***	0.30	.99
Trial number	34.46	3,168	<.001	***	0.38	.99
Automation Configuration × Trial Number	1.47	6,336	.189		0.03	.09
Situation awareness
Perception score
Automation configuration	10.86	2,112	<.001	***	0.17	.84
Trial number	11.13	3,168	<.001	***	0.17	.79
Automation Configuration ×  Trial Number	1.10	6,336	.360		0.02	.11
Comprehension score
Automation configuration	10.39	2,112	<.001	***	0.16	.80
Trial number	2.35	3,168	.074		0.04	.20
Automation Configuration ×  Trial Number	1.12	6,336	.353		0.02	.09
Prediction score
Automation configuration	10.24	2,112	<.001	***	0.16	.80
Trial number	2.51	3,168	.060		0.04	.19
Automation Configuration ×  Trial Number	1.49	6,336	.180		0.03	.09

Note. Power was calculated with G*Power (Version 3.1.9.2; see Faul, Erdfelder, Buchner, & Lang, 2009).

***

p < .001 (two tailed).

OPEN IN VIEWER

Figure 3.

Bar graphs with standard error bars visualizing the main effects of the automation configuration on (a) the performance score and (b) the situation awareness scores (mean across trials per automation configuration).

The repeated-measures ANOVAs also showed significant skill acquisition effects (Figure 4). The trial number, which we included to control such effects, affected performance and the perception score of situation awareness, ps < .001 (Table 2). These effects did not interact with the automation configuration, ps > .05. Due to these effects, we kept the trial number as a repeated-measures variable in the ANCOVAs when investigating the interaction effects.

OPEN IN VIEWER

Figure 4.

Line graphs with standard error bars visualizing the change of (a) performance and (b) perceptual situation awareness across the four trials (aggregated per automation configuration).

Ability Predictors of Performance and Situation Awareness

We evaluated Hypothesis 2 with multiple regressions. The criteria were the mean values of performance and situation awareness scores across automation configurations and trials. The individual-difference measures were the predictors. One the one hand, the results supported Hypothesis 2 (Table 3). Performance improved with the participants’ ability to process complex information and the capacity to simultaneously store and process information, ps < .05. Similarly, the perception score of situation awareness was higher for the individuals with higher abilities in processing complex information and better supervision capacity, ps < .05. Last, the prediction score increased when the participants could better process complex information, p < .05. On the other hand, the results (Table 3) did not show significant links between the abilities and the comprehension score, all ps > .05, and did not confirm Hypothesis 2.

OPEN IN VIEWER

Table 3:

Summary of the Multiple Regression Analyses

						Situation Awareness
	Performance a					Perception Score b					Comprehension Score c					Prediction Score d
Variable	B	SE (B)	β	t(49)	p	B	SE (B)	β	t(49)	p	B	SE (B)	β	t(49)	p	B	SE (B)	β	t(49)	p
Information-processing (IP) ability
IP complexity	.02	0.01	.28	1.83	.04*	.30	0.02	.31	1.66	.05*	.02	0.02	0.28	1.52	.06	.03	0.01	.29	1.71	.04*
IP speed	.02	0.01	.20	1.20	.12	.01	0.02	.16	0.86	.20	.02	0.01	0.22	1.25	.11	.02	0.02	.17	0.96	.17
Verbal ability	.02	0.01	.25	1.55	.06	.02	0.02	.23	1.35	.09	.01	0.01	0.12	0.73	.23	.02	0.02	.18	0.99	.16
Numerical ability	.02	0.02	.16	0.92	.18	.00	0.01	.03	0.16	.44	.01	0.01	0.12	0.78	.22	.01	0.01	.08	0.49	.31
Working-memory capacity
Coordination	.39	1.23	.04	0.32	.38	.77	1.35	.08	0.57	.29	.94	1.20	1.11	0.78	.22	.77	1.35	.08	0.57	.28
St. and processing	.31	0.16	.31	2.04	.02*	.06	0.18	.06	0.35	.36	.16	0.12	0.16	1.23	.11	.21	0.18	.21	1.36	.09
Supervision	.05	0.12	.05	0.22	.33	.32	0.15	.32	2.07	.02*	.77	0.12	0.16	0.12	.23	.17	0.13	.17	1.27	.10

Note. For these multiple regressions, we calculated the mean values of the dependent measures across trials and automation configurations for each participant. We used the resulting variables as the criteria in the multiple regressions. St. = simultaneous storage.

a

The predictors explained 17.8% (adjusted R²) of the variance of the performance score, F(7, 48) = 2.71, p < .05 (two sided).

b

The predictors explained 13.1% (adjusted R²) of the variance of the perception score, F(7, 48) = 2.24, p < .05 (two sided).

c

The predictors explained 10.3% (adjusted R²) of the variance of the comprehension score, F(7, 48) = 1.04, p > .05 (two sided).

d

The predictors explained 12.8% (adjusted R²) of the variance of the prediction score, F(7, 48) = 2.15, p < .05 (two sided).

*

p < .05 (one tailed).

Interaction Effects of Automation Configuration and Abilities on Performance and Situation Awareness

We evaluated Hypothesis 3 with repeated-measures ANCOVAs. We categorized the participants according to their individual-difference measures in three groups such that each group consisted of n = 19 participants with one exception: The group with a medium level of information-processing speed consisted of n = 20 participants; the group with a high level of information-processing speed had n = 18 participants. We implemented this exception because two participants achieved the same score here. For categorization, we did not consider the manipulation of the presentation order. This covariate was excluded because it had no significant main or interaction effect. For the ANCOVAs, we used the categorized individual-difference measures as the covariates and the automation configuration and the trial number as within-subject independent variables to predict performance and situation awareness.

The results supported Hypothesis 3 (Table 4). With regard to performance (Hypothesis 3a), the automation configuration interacted with the ability to process complex information and with the capacity for simultaneous storage and processing, both ps < .05. The directions of the effects are illustrated in the bar graphs presented in Figure 5a: The participants with low ability to process complex information performed best with LOA1 = (moderate, low, low, low) and worst with LOA3 = (moderate, moderate, moderate, low). In contrast, the participants with high ability to process complex information performed best with LOA3 = (moderate, moderate, moderate, low) and worst with LOA1 = (moderate, low, low, low). A similar fan effect was present with regard to the capacity to simultaneously store and process information (Figure 5b). We could thus confirm Hypothesis 3a.

OPEN IN VIEWER

Table 4:

Summary of the Repeated-Measures’ ANCOVAs Evaluating the Effects of Automation Configuration, Trial Number, Abilities, and Their Interactions on the Dependent Measures

						Situation Awareness
	Performance					Perception Score					Comprehension Score					Prediction Score
Variable	F	dfs, dfe	p	η2p	1 – β	F	dfs, dfe	p	η2p	1 – β	F	dfs, dfe	p	η2p	1 – β	F	dfs, dfe	p	η2p	1 – β
Information-processing (IP) ability
AC	11.67	2, 42	<.001***	0.24	.96	6.99	2, 42	.002***	0.23	.95	6.77	2, 42	.003**	0.18	.87	8.17	2, 42	<.001***	0.28	.99
AC * IP complexity	3.03	4, 42	.028*	0.14	.62	<1	4, 42	.682	0.05	.30	1.52	4, 42	.214	0.09	.40	3.71	4, 42	.011*	0.26	.94
AC × IP Speed	1.80	4, 42	.140	0.09	.40	3.33	4, 42	.019*	0.22	.94	2.64	4, 42	.047*	0.14	.63	1.57	4, 42	.200	0.13	.58
AC × Verbal Ability	2.76	4, 42	.040*	0.13	.58	<1	4, 42	.822	0.03	.20	<1	4, 42	.606	0.04	.18	<1	4, 42	.537	0.06	.26
AC × Numerical Ability	<1	4, 42	.719	0.03	.14	<1	4, 42	.467	0.07	.41	<1	4, 42	.719	0.03	.14	2.32	4, 42	.072	0.18	.77
Trial number	6.96	3, 63	<.001***	0.16	.70	4.58	3, 63	.005**	0.16	.76	2.46	3, 63	.071	0.07	.35	1.07	3, 63	.368	0.05	.22
Working-memory capacity
AC	15.42	2, 72	<.001***	0.27	.98	6.95	2, 72	.002***	0.16	.82	1.11	2, 72	.335	0.03	.19	1.07	2, 72	.348	0.03	.19
AC × Coordination	1.64	4, 72	.173	0.07	.31	1.11	4, 72	.360	0.06	.26	1.06	4, 72	.381	0.05	.22	1.45	4, 72	.240	0.07	.31
AC × St. and Processing	3.19	4, 72	.018*	0.14	.63	2.52	4, 72	.049*	0.12	.54	1.14	4, 72	.345	0.05	.22	<1	4, 72	.761	0.02	.11
AC × Supervision	<1	4, 72	.698	0.03	.14	2.15	4, 72	.083	0.11	.49	2.66	4, 72	.039*	0.12	.54	<1	4, 72	.425	0.05	.22
Trial number	11.34	3, 108	<.001***	0.22	.91	3.03	3, 108	.032*	0.08	.40	<1	3, 108	.407	0.02	.12	<1	3, 108	.680	0.01	.08

Note. We did not report the statistics (a) for the two-way interaction effects between the trial number and the individual-difference measures and (b) for the three-way interaction effects between the trial number, the automation configuration, and the individual-difference measures because none of the effects reached an appropriate level of significance, all ps > .05, and because such effects were not in the focus of this study. Power was calculated with G*Power (Version 3.1.9.2; see Faul, Erdfelder, Buchner, & Lang, 2009). AC = automation configuration; St. = simultaneous storage.

*

p < .05 (two tailed). **p < .01 (two tailed). ***p < .001 (two tailed).

OPEN IN VIEWER

Figure 5.

Bar graphs with standard error bars visualizing the interaction effects between (a) the categorized ability to process complex information and automation configuration and (b) the categorized capacity to simultaneously store and process information and automation configuration on performance (mean across trials per automation configuration).

The results of the ANCOVAs also supported Hypothesis 3b. First, the interaction effects between the automation configuration and the ability to quickly process cognitively easy information and between the automation configuration and the capacity to simultaneously store and process information significantly influenced the perception score of situation awareness, ps < . 05 (Table 4). The effects are illustrated in Figure 6: The participants with low information-processing speed and low capacity for simultaneous storage and processing achieved their best scores with LOA1 = (moderate, low, low, low) and their worst scores with LOA3 = (moderate, moderate, moderate, low). In contrast, the participants with medium and high information-processing speed and capacity for simultaneous storage and processing achieved better perception scores when LOA1 = (moderate, low, low, low) was not active.

OPEN IN VIEWER

Figure 6.

Bar graphs with standard error bars visualizing the interaction effects (a) between the categorized information-processing speed and automation configuration and (b) between the categorized simultaneous storage and processing and automation configuration on the perception score of situation awareness (mean across trials per automation configuration).

Second, the interaction effects between automation configuration and information-processing speed and between automation configuration and supervision influenced the comprehension score, ps < .05 (Table 4). The participants with low information-processing speed and supervision capacities achieved their best scores with LOA1 = (moderate, low, low, low) (Figure 7). With higher levels of information-analysis and decision-selection automation, their scores were lower. In contrast, the participants with high information-processing speed and supervision capacity benefited from higher levels of information-analysis and decision-selection automation.

OPEN IN VIEWER

Figure 7.

Bar graphs with standard error bars visualizing the interaction effects (a) between the categorized information-processing speed and automation configuration and (b) between the categorized supervision capacity and automation configuration on the comprehension score of situation awareness (mean across trials per automation configuration).

Finally, the interaction effect between the automation configuration and information-processing complexity significantly influenced the prediction score of situation awareness (Table 4). The direction of this effect is obvious when one inspects the bar graph presented in Figure 8. The participants with low information-processing complexity achieved their lowest prediction scores with LOA3 = (moderate, moderate, moderate, low). In contrast, participants with high information-processing complexity achieved their lowest scores with LOA1 = (moderate, low, low, low).

OPEN IN VIEWER

Figure 8.

Bar graph with standard error bars visualizing the interaction effect between information-processing complexity and automation configuration on the prediction score of situation awareness (mean across trials per automation configuration).

Abilities Can Determine Effects of LOAs on Performance and Situation Awareness

The ability profiles of the operators influenced the effects of higher levels of information-analysis and decision-selection automation on performance and situation awareness according to our results. In our study, the operators with higher information-processing ability and working-memory capacity benefited from higher levels of information-analysis and decision-selection automation. In contrast, the operators with lower information-processing ability and working-memory capacity seemed to be relatively disadvantaged when the level of information-analysis and decision-selection automation increased. No previous study has investigated the role of information-processing ability and working-memory capacity on effects of information-analysis and decision-selection automation. Previous attempts to identify the reasons for inconsistent effects of automation have mainly focused on task and automation characteristics (e.g., Manzey, Reichenbach, & Onnasch, 2012; Onnasch et al., 2014; Sarter & Schroeder, 2001; Wickens, Hollands, Banbury, & Parasuraman, 2012).

According to our results, information-processing ability and working-memory capacity appear to influence performance and situation awareness more when the levels of information-analysis and decision-selection automation increase from low to moderate. One reason may be an increasing complexity of the mental model, which the operators need for completing the task along with the automation (see also Kaber et al., 2005; Rauterberg, 1992). The complexity might have increased because the operators needed to incorporate the automated tasks into their own mental model. Specifically, the operators had to understand the logic of the higher level of information-analysis automation, that is, the ghosts and targets that pointed to a need for action and the higher level of decision-selection automation, that is, the advisories. Therefore, the operators had to attend to relevant information from the automation (e.g., perceive the ghosts), simultaneously store and process this information (e.g., relate the positioning of the ghosts with the real aircraft), and reason about the information (e.g., identify a need for action). The operators with higher ability to process complex and cognitively easy information and with higher capacity to simultaneously store and process information seemed to be better able to integrate this information into their mental model of the task. This integration, in turn, appeared to allow these operators to perform better and to establish better situation awareness. They made more effective use of the information from the higher levels of information-analysis and decision-selection automation. In contrast, the operators with lower ability to process complex and cognitively easy information and with lower capacity to simultaneously store and process information appeared not to be able to use this information for deep processing (see Petty & Cacioppo, 1986). They seemed more likely to avoid cognitive effort, reduced their intrinsic load (see Atwood & Polson, 1976), and thus neither improved their performance nor established better situation awareness. Consequently, the operators’ ability level determined whether higher levels of information-analysis and decision-selection automation fostered higher performance and situation awareness.

Abilities Can Influence Situation Awareness

Our results replicated previous findings regarding the cognitive processes underlying the establishment of situation awareness. First, we replicated the results of Harbluk, Noy, Trbovich, and Eizenman’s (2007) by relating supervision, which resembles the central executive (Oberauer et al., 2003), to perceptual situation awareness. The capability to actively guide attention contributed to accurately perceiving relevant information from the environment. However, we could not replicate the results of Sangeun (2008). Sangeun (2008) related measures of operational span, which can be interpreted as contributing to the simultaneous storage-and-processing function of working memory (Oberauer et al., 2000), with her score of perceptual situation awareness. One reason for this divergent result may be the task environment, as Sangeun drew on a driving task (see also Durso & Drews, 2010; Lau, Jamieson, & Skraaning, 2013). Second, we could not identify a significant association between information-processing ability or working-memory capacity and the comprehension score of situation awareness—consistent with Sulistyawati, Wickens, and Chui (2011). Finally, the ability to process complex information contributed to the establishment of predictive situation awareness, a connection that Sulistyawati et al. also reported. These corroborating results confirm the findings of the previous studies (for the need for replication, see Jones, Derby, & Schmidlin, 2010).

Conclusion and Future Work

Future work should investigate whether information-processing ability and working-memory capacity moderate the effects of higher levels of information-analysis and decision-selection automation on performance and situation awareness when automation failures occur. Our expectation would be that the experience of failures should increase the complexity of the mental model required for completing the task in question with the automation assistance (see Hoffman et al., 2014). Thus cognitive abilities may be more highly related to outcome measures.

Moreover, authors of future work should investigate how the relationship between cognitive abilities and automation configuration changes, if the levels of information-analysis and decision-selection automation increase even more, and if the levels of information-analysis and action-implementation automation change. First, we would expect that the effects may increase when the level of the information-analysis and decision-selection automation increases even more, because then the establishment of a valid mental model may be more complex. Second, we would not expect that the effects would change if the levels of information-analysis and decision-selection automation change, because Kaber et al. (2005) considered both as low-order information-processing functions. A change in the LOA may not make the mental model more or less complex.

Future work should also focus on replicating our results with participants with higher levels of proficiency. Several researchers proposed levels of categories of proficiency (e.g., Dreyfus & Dreyfus, 1986; Hoffman, 1998). For instance, Hoffman (1998) distinguished naives, novices, initiates, apprentices, journeymen, experts, and masters and provided criteria for when learners can be considered as having which level of proficiency. On the basis of these levels, our participants may be considered initiates at maximum—novices who began training in a specific field. This procedure was similar to that of many other studies in this context (e.g., Endsley & Kaber, 1999; Manzey et al., 2011; Parasuraman, Cosenzo, & De Visser, 2009). Still, it is crucial to know whether our results can be expended to experts and masters. Then, training and personnel selection strategies might be optimized by focusing on the alignment between operator ability levels and automation demands.

Furthermore, intraindividual differences in the epistemic knowledge that operators pick up from instructions (see Kluwe, 1997) and in motivation (see Szalma, 2009) may also be relevant. Petty and Cacioppo (1986) pointed out that motivation can also determine whether humans prefer heuristics or deep processing of information. Future work should thus also focus on investigating whether an experimental manipulation of motivation might also have influences similar to cognitive abilities. We would expect that the influences are similar, such that more motivated operators perform better with higher LOAs in general than do less motivated operators. One way to manipulate (extrinsic) motivation would be to give operators bonuses for better performance and situation awareness.

Finally, research is needed to explore the boundary conditions across (a) different domains (e.g., medicine), (b) different tasks in the same domain (e.g., tower control), (c) different automation systems in that domain (e.g., midterm conflict detection), and (d) different ways of implementing the same automation (e.g., displaying the advisories in a separate list in MAGIE). As Flach (2015) pointed out, “context matters” (p. 59), and the detailed relationships could vary between domains and tasks (e.g., Durso & Drews, 2010; Lau et al., 2013). If the results can be replicated in different settings, a definite answer of whether information-processing ability and working-memory capacity determine effects of increasing levels of information-analysis and decision-selection automation can be resolved.

Nevertheless, information-processing ability and working-memory capacity moderated the effects of higher levels of information-analysis and decision-selection automation on performance and situation awareness at least in our study. If these results can be replicated and generalized, it is important to enable the automation to balance these interindividual differences or to provide systems that assist humans to deal with the increased complexity of the mental model. Balancing interindividual differences in relevant abilities could be achieved by enabling automation to categorize the level of information-processing ability and working-memory capacity of its user and to adapt its LOA such that outcome measures are optimized—a procedure typical for adaptive automation systems (e.g., Kaber & Endsley, 2004; Scerbo, 1996). According to our results, the adaptation should ensure that the operators with lower abilities to process complex information receive assistance from LOA = (moderate, low, low, low) or LOA = (moderate, moderate, low, low). Operators with high ability to process complex information should receive assistance from LOA = (moderate, moderate, moderate, low).

Systems assisting humans to deal with the increased complexity of the mental model could make use of automation that mirrors human perception-cognition-action processes. Therefore, it may be an option to enable automation to observe the operator when executing the tasks manually and when thinking aloud about the tasks and to copy these processes. Another option may be to enable automation to clearly and comprehensively explain its own reasoning processes, decision, and actions to the operator. This last option may not reduce complexity but may assist operators in understanding the complexity more quickly. Both options may thus enable operators to build a shared problem representation more easily (see Christoffersen & Woods, 2002) and increase the transparency of the automation (see Kaber et al., 2005). These mechanisms may enable operators with lower abilities to also profit from higher levels of information-analysis and decision-selection automation.