The Impact of Higher Levels of Automation on Performance and Situation Awareness

Abstract

Systems that supported operators with higher levels of information-analysis and decision-selection automation had varying effects on human performance and situation awareness. We investigated whether information-processing and working-memory abilities moderate the effects of automation on human performance and situation awareness. To investigate such effects, we measured the information-processing ability and working-memory capacity of 60 participants. We also assessed their performance and situation awareness when they repeatedly controlled simulated air traffic with the support of different levels of information-analysis and decision-selection automation. Results indicated that performance increased, but situation awareness declined, when the levels of automation increased. The participants with better information-processing ability and working-memory capacity scored better in performance and situation awareness. The participants with higher information-processing ability and working-memory capacity profited from the higher levels of automation. In contrast, the participants with lower information-processing ability and working-memory capacity suffered under higher levels of automation. Authors of future research should thus consider individual differences when investigating the effects of automation and focus on identifying mechanisms that ensure that automation supports all operators.

Keywords

level of automation information-analysis automation decision-selection automation air traffic control individual differences

Introduction

Technological progress has enabled the development of systems that automate cognitively complex functions, such as information analysis and decision selection (Sheridan & Parasuraman, 2006). However, evaluations of such systems have revealed inconsistent effects: In some studies, the performance and situation awareness of human operators decreased (e.g., Kaber, Onal, & Endsley, 2000; Manzey et al., 2011); in other studies, performance and situation awareness increased (e.g., Calhoun, Draper, & Ruff, 2009). One reason for these inconsistent effects may be different cognitive profiles of the participants with whom the evaluations were conducted. For instance, Manzey et al. (2011) sampled advanced medical students, but Calhoun et al. (2009) sampled expert pilots. Different cognitive profiles may play a role, because the cognitive complexity of the mental model underlying task completion can increase with the additional automation (e.g., Kaber, Wright, Prinzel, & Clamann, 2005). That is, inconsistent effects of automation may reflect individual differences in information-processing ability and working-memory capacity. If individual differences in these abilities affect the development of mental models, then the effects of automation on performance and situation awareness may be different. As previous research had not focused on such moderator effects, we investigated whether information-processing ability and working-memory capacity moderated the effects of higher levels of information-analysis and decision-selection automation on performance and situation awareness.

Levels of Automation (LOAs)

Researchers have introduced generic frameworks that classify the ways in which automation can assist humans (e.g., Endsley, 1987; Kaber & Endsley, 1997; Ntuen & Park, 1988; Parasuraman, Sheridan, & Wickens, 2000; Sheridan & Verplank, 1978). For example, Parasuraman et al. (2000) suggested that “automation can be applied to four broad classes of functions” (p. 288): (a) information acquisition, which represents gathering, registering, and preprocessing information from the environment; (b) information analysis, which represents the conscious perception of information and the conduct of cognitive operations, such as the integration of and reasoning about information; (c) decision selection, which represents choosing an action; and (d) action implementation, which represents executing the action. In each of these functions, automation systems can assist their users to varying levels. Parasuraman et al. (2000) also suggested how to classify these LOAs for each function. For example, a low level of information-analysis automation would extrapolate data over time. A moderate level would integrate data streams into a single value, and a high level would provide context-dependent summaries of the situation. With regard to decision-selection automation, Parasuraman et al. (2000) suggested systems be classified on the basis of the 10 LOAs of Sheridan and Verplank (1978). When one simplifies these LOAs by considering only decision-selection functions, a low level of decision-selection automation may recommend a set of actions. A moderate level may suggest one action, and a high level of automation may decide what to do, independent of the user.

This is why this paper defines LOA according to the individual levels of the four automation functions:

LOA = (Level of information - acquisition automation, Level of information - analysis automation, Level of decision - selection automation, Level of action - implementation automation) .

For example, McFarland (1997) introduced the User Request Evaluation Tool, which provided automatic conflict detection in en route air traffic control. The tool automated information acquisition to a high level, because it integrated various information sources (e.g., radar tracks, flight plan information) to predict a flight path. The level of the information-analysis automation would be low if it predicted only the development of single flight paths without considering the interrelation of the flight paths of all aircraft. The level of the decision-selection automation would be low if the tool did not propose actions. Finally, the level of the action implementation would be low if the user still had to implement actions manually. The LOA of the tool would thus be LOA = (high, low, low, low).

Inconsistent Effects of Levels of Automation on Performance and Situation Awareness

Performance and situation awareness were often used as criteria to evaluate the effects of automation. Situation awareness is an important and multidimensional construct in human factors research. Three special issues have been devoted to this topic in the past decades (see Gilson, 1995; Pritchett, 2015; Stanton, 2010). According to Endsley (1995b), situation awareness comprises the perception of relevant information from the environment, the comprehension of the current situation, and the projection of the situation into the near future. Parasuraman, Sheridan, and Wickens (2008) considered situation awareness as useful for understanding and predicting human reactions to complex systems. For example, researchers used situation awareness successfully to analyze incidents and accidents (e.g., Durso, Truitt, Hackworth, Crutchfield, & Manning, 1998; Gaba, Howard, & Small, 1995; Jones & Endsley, 1996; cf. Flach, 1995) and to predict operators’ ability to recover from automation failures (e.g., Endsley & Kiris, 1995). Situation awareness can be a predictor of how well operators reacted to automation failures (Onnasch, Wickens, Li, & Manzey, 2014).

Meta-analyses have been conducted to investigate the effects of higher LOAs on situation awareness and performance—measured in situations without automation failures (Onnasch et al., 2014; Wickens, Li, Santamaria, Sebok, & Sarter, 2010). Results indicated that the provision of higher LOAs can result in inconsistent patterns of changes in performance and situation awareness. For example, Wickens et al. (2010) found that most studies had only zero or even negative performance changes associated with systems with increases in level of information-analysis and decision-selection automation. In contrast, Onnasch et al. (2014) reported significant increases in performance and decreases in situation awareness when the systems had increased LOAs in information analysis and decision selection. The effect sizes that Onnasch et al. reported varied greatly, which points to moderator effects. Onnasch et al. considered moderator effects of different classification methods of LOAs, the training of the participants, the task setting, and the nature of the display used by the automation. However, only the classification method seemed to play an important role for performance; levels of significance were not reported. It is not yet fully clear what might have caused the varying effects of higher information-analysis and decision-selection automation.

Individual differences in cognitive abilities may moderate information-analysis and decision-selection automation effects on performance and situation awareness. Consideration of these effects may explain different outcomes between studies in the literature, which did not account for sample/population differences (Calhoun et al., 2009; Cummings & Mitchell, 2007; Endsley & Kaber, 1999; Ferris, Sarter, & Wickens, 2010; Kaber et al., 2000; Kaber & Endsley, 2004; Lin, Yenn, & Yang, 2010; Manzey et al., 2011; Parasuraman & Riley, 1997; Sarter & Schroeder, 2001). For example, Manzey et al. (2011) asked 14 advanced medical students to execute a simulated mastoidectomy. The students required significantly less time to complete the surgery task when they received assistance from the navigated control system, LOA = (moderate, moderate, moderate, moderate), than from the image-guided navigation system, LOA = (moderate, moderate, low, low). In contrast, six novice Air Force personnel completed their mission slower when they controlled unmanned aerial vehicles with a system of which the LOA changed from LOA = (high, high, low, low) to LOA = (high, high, moderate, low) in the study of Calhoun et al. (2009). The authors classified their system LOA on the basis of the scheme of Sheridan and Verplank (1978), which we reclassified here to compare the results with the other studies.

With regard to situation awareness, Kaber and Endsley (2004) asked 30 university students to conduct an abstract radar-monitoring task. Comprehension scores of situation awareness were higher for blended decision making, LOA = (moderate, moderate, moderate, high), when compared to batch processing, LOA = (moderate, low, low, high). We also reclassified the LOAs here because Kaber and Endsley (2004) applied the classification method of Kaber and Endsley (1997). In contrast to the study of Kaber and Endsley (2004), the advanced medical students of Manzey et al. (2011) were better able to answer questions about the specifics of anatomy when they executed a mastoidectomy with an image-guided navigation system, LOA = (moderate, moderate, low, low), than with a navigated control system, LOA = (moderate, moderate, moderate, moderate).

The cognitive-ability profiles of the advanced medical students (Manzey et al., 2011), the university students (Kaber & Endsley, 2004), and the Air Force personnel (Calhoun et al., 2009) likely differed but were not measured in these studies. Moreover, no available study investigated the impact of cognitive abilities on utilizing different levels of information-analysis and decision-selection automation to attain performance and situation awareness.

Relevance of Cognitive-Ability Profiles

Cognitive-ability profiles may play a moderating role because higher levels of information-analysis and decision-selection automation may increase the complexity of the mental model underlying task completion. Kaber et al. (2005) expected the complexity to increase because systems with higher levels of information-analysis and decision-selection automation perform more tasks that the operators must integrate into their mental model. This integration is crucial for successful joint task completion (Christoffersen & Woods, 2002; Rauterberg, 1992). For the integration, the operators need to collect the information the system uses as inputs, mirror the automated reasoning processes, collect information on the behavior of the automation, build the model for the automation, and modify it if necessary (Ramsbottom & Baker, 2003). Therefore, the operators’ information-processing ability and working-memory capacity may be more engaged. Appendix A provides definitions of the functions of information-processing ability and working-memory capacity under consideration here.

Operators deal with the increased cognitive demands differentially, as proposed by the elaboration likelihood model (Petty & Cacioppo, 1986). On the one hand, operators can avoid the cognitive effort and apply heuristics that reduce the intrinsic cognitive load (see Atwood & Polson, 1976). On the other hand, operators can feel stimulated to process this additional information and establish a more complex mental model. As Petty and Cacioppo (1986) pointed out, cognitive abilities can determine whether humans prefer heuristics or deep processing of information. For operators with higher abilities, the deep processing of the additional information can be easier. These operators likely deeply process information, attain better performance, and better maintain situation awareness. For operators with lower abilities, however, deep processing can be more difficult. These individuals likely use heuristics to solve the task. Consequently, their mental models would be insufficiently developed, which would lead to worse performance and lower situation awareness. In summary, operators with higher information-processing ability and working-memory capacity may profit from higher levels of information-analysis and decision-selection automation, and operators with lower abilities may suffer under such automation.

Hypotheses

The main goal of this study was to investigate whether information-processing ability and working-memory capacity moderate the effect of increasing levels of information-analysis and decision-selection automation on performance and situation awareness. Two propositions reflected in the following hypotheses underlay this goal:

Hypothesis 1a: Performance will change when operators receive assistance from systems in which the level of information-analysis and decision-selection automation changes.

Hypothesis 1b: Situation awareness will change when operators receive assistance from systems in which the level of information-analysis and decision-selection automation changes.

Hypothesis 2: Information-processing ability and working-memory capacity will be positively related to performance and situation awareness.

The following hypotheses reflected the main research goal:

Hypothesis 3: There will be a significant interaction between the cognitive abilities (information-processing ability and working-memory capacity) and the level of information-analysis and decision-selection automation.

Hypothesis 3a: Operators with higher ability levels will perform better when the levels of information-analysis and decision-selection automation increase. Operators with lower ability levels will perform worse when the levels of information-analysis and decision-selection automation increase.

Hypothesis 3b: Operators with higher ability levels will establish better situation awareness when the levels of information-analysis and decision-selection automation increase. Operators with lower ability levels will establish worse situation awareness when the levels of information-analysis and decision-selection automation increase.

Method

Participants

Sixty (29 male, 31 female) university students with normal or corrected-to-normal vision participated in this study and received about US$80 for their participation. The average age was 23.6 years (SD = 3.48; range = 19–30 years).

Experimental Design

This study followed a within-subjects design with two independent variables: automation configuration (three levels) and trial number (four levels). Automation configuration represented the manipulation of the level of information-analysis and decision-selection automation: The automation assistance varied between LOA1 = (moderate, low, low, low), LOA2 = (moderate, moderate, low, low), and LOA3 = (moderate, moderate, moderate, low). The automation was perfectly reliable, because failures would have depended on the currently active automation configuration and biased the manipulation of the variable. Trial number reflected the number of times we confronted the participants with the task. We used four trials to yield sufficient data for statistical analysis.

Simulation Environment

Task

We asked the participants to perform a simplified air traffic control task using a modified version of the Micro Air Ground Integration Environment (MAGIE). MAGIE allows investigating solutions for challenges in air traffic control (e.g., Oberheid, Weber, & Rudolph, 2009) and effects of automation (e.g., Oberheid, Hasselberg, & Söffker, 2011). MAGIE simulated inbound air traffic in the terminal area of an airport (Figure 1). Two types of aircraft entered the simulated airspace along standard approach routes: “Equipped” aircraft descended automatically from their approach routes to the runway. They were equipped with a high-end flight management system that enabled the aircraft to fly a continuous descent approach. This procedure reduces noise and emissions (see Alam et al., 2010; Warren & Tong, 2002). The aircraft without this equipment turned from the end of their approach routes automatically to the east and flew parallel to the runway away from the runway along the downwind leg (Lecchini, Glover, Lygeros, & Maciejowski, 2005).

Figure 1.

Screenshot from Micro Air Ground Integration Environment visually representing the display used for interaction, key concepts, and the routes that unequipped and equipped aircraft use to arrive at the runway.

The participants needed to give heading, speed, and altitude instructions to the “unequipped” aircraft in the control zone to meet three objectives:

The participants had to ensure that all aircraft remained at a horizontal distance of at least 3 NM (5.56 km) apart from each other, according to the standard radar separation requirement (see Thompson, 1997). Participants were neither instructed nor required to ensure that a certain vertical distance remained between the aircraft to avoid floor effects due to the high level of difficulty (based on the results of Oberheid et al., 2009). We asked the participants to ignore the vertical separation because we expected it to be more difficult to ignore the horizontal separation, due to the way the interface was designed.

The participants had to ensure that aircraft not equipped with the high-end flight management system adhered to minimum and maximum speed and altitude limits. The limits depended on the aircraft position and ensured that the aircraft systematically reduced speed and altitude so they could be landed on the runway.

The participants had to ensure that as many aircraft as possible arrived at the runway. Therefore, the participants had to instruct the unequipped aircraft to leave the downwind leg at a certain time, so that the planes turned in the direction of the centerline and followed it to the runway. Aircraft disappeared from the simulation as soon as they reached the end of the downwind leg. We implemented this simplification of real-world air traffic control because the aircraft left the control zone at the end of the downwind leg and the participants would no longer be able to correct their mistakes.

Several tasks had to be executed to achieve the objectives. The tasks can be sorted into the function classes propagated by the four-stage model of information processing (Parasuraman et al., 2000):

1. Information acquisition applied to registering information provided by simulated radar sensors and obtained from previous instructions. This information needed to be preprocessed to identify the current position and status of each aircraft. Preprocessing was also necessary to organize the aircraft in a timeline, enabling prioritization of aircraft, and to highlight current deviations from the separation, speed, and altitude regulations.

We considered the prioritizing and highlighting information as information-acquisition tasks for two reasons: First, Parasuraman et al. (2000) used the “organization of incoming information according to some criteria, e.g., a priority list” (p. 288) and “highlighting of some part of the information” (p. 288) as examples for acquisition. Second, this approach allowed us to display violations in all automation configurations, which counteracted the criticism of Süß (1999) that dynamic tasks provided unclear objectives, such that the reliability of the performance measures would be reduced.

2. Information analysis applied to extrapolating the current status of each aircraft into the future, integrating the current paths of the equipped and unequipped aircraft, and identifying potential violations of separation, speed, and altitude regulations. On this basis, ideal approach paths needed to be identified for the unequipped aircraft. These paths then had to be related to the current paths to identify deviations and a related need for actions.

3. Decision selection applied to defining instructions that would make the unequipped aircraft follow ideal approach paths, evaluating these instructions, and if necessary, modifying them. The instructions then needed to be prioritized, such that it was clear when certain instructions had to be issued.

4. Action implementation referred to instructing aircraft to change speed, altitude, or heading. Therefore, the participants had to click on the aircraft that should modify its speed, altitude, or heading. Then, MAGIE opened an interaction display (Figure 1) that provided the current speed, altitude, and heading instructions for the selected aircraft. In this display, the participants had to select the new target value and send it to the aircraft. Then, the automation masked the interaction display, and the aircraft started changing its flight profile such that it approached the target value.

Automation configurations

We developed three automation configurations by manipulating whether the participants or the automation executed the tasks (see Appendix B). Configuration 1 (Figure 2a) implemented LOA1 = (moderate, low, low, low). Configuration 2 (Figure 2b) implemented LOA2 = (moderate, moderate, low, low), and Configuration 3 (Figure 2c) implemented LOA3 = (moderate, moderate, moderate, low).

Figure 2.

Screenshots from the three automation configurations (for details, see Appendix B) of the simulation environment visually representing (a) the timeline and the highlighting of information (Configuration 1), (b) the additional targeting and ghosting (Configuration 2), and (c) the additional advisories (Configuration 3).

The moderate information-acquisition automation continuously displayed (a) the current positions of all aircraft together with the current and cleared states (speed and altitude), (b) the timeline, and (c) deviations to rules and limits on the screen (Figure 2a). The information was updated once per second. We considered the level of this information-acquisition automation as moderate because the automation organized and highlighted information without removing raw data (see Parasuraman et al., 2000).

The moderate information-analysis automation provided two types of visual cues to augment human cognition (Figure 2b): First, it continuously projected a ghost aircraft for each equipped aircraft in the simulation onto the approach path of the unequipped aircraft. Specifically, each ghost aircraft followed the approach path of the unequipped aircraft—in contrast to the real counterpart. At the intersection of the paths, each ghost aircraft merged with its real counterpart and was no longer visible. If the participants placed an unequipped aircraft close to a ghost aircraft, the unequipped aircraft would violate the separation rule at the intersection between its path and the path of the equipped aircraft. An alarm was provided only as soon as the separation rule was violated.

Second, the automation continuously projected a target aircraft for each unequipped aircraft in the simulation onto the centerline. These target aircraft showed the ideal position of the unequipped aircraft. If an unequipped aircraft was on top of its target on the centerline, the aircraft had the ideal position. If an unequipped aircraft was on the downwind leg and its target was on the centerline (as in Figure 2b), the aircraft needed to be turned in the direction of the centerline.

We considered this information-analysis automation as moderate because both visual cues eliminated the need to mentally integrate the approach paths of both aircraft types—a typical characteristic of moderate information-analysis automation according to Parasuraman et al. (2000). When the level of information-analysis automation was low, the target aircraft or ghost aircraft was not displayed.

The moderate decision-selection automation communicated necessary instructions to the participants as an advisory (Figure 2c). An example was to instruct the aircraft U18 to reduce its altitude to 3,000 feet in 5 s. The participants could ignore the advisory. The automation displayed these advisories underneath the aircraft symbol only when the participants placed the mouse cursor over a counter. The automation also displayed the counter underneath the aircraft symbol. The counter informed the participants exactly when they should give the subsequent instructions to the aircraft. The counter became visible 30 s prior to the time the participants should give the instruction. The automation masked the counter if the participants gave the instruction to the aircraft or if the participant gave another instruction that changed the situation. Alternatives to these instructions were not provided. Thus we considered this decision-selection automation as moderate, that is, Level 4 in the classification scheme set forth by Sheridan and Verplank (1978) and suggested by Parasuraman et al. (2000). When the level of decision-selection automation was low, the participants did not receive advisories.

Besides automation replacing pilot actions, action-implementation automation was not available. The level of the action-implementation automation was thus always low.

Scenarios

We designed two scenarios for training and four for data collection. The scenarios had equal lengths of 7 min and comparable degrees of difficulty. Each scenario could be run with each automation configuration.

Within each scenario, 13 aircraft successively entered the airspace. The aircraft had a separation of at least 7 s before the critical separation was breached. Seven of the 13 aircraft were unequipped, and six were equipped. This ratio equaled typical ones used for evaluating the effects of mixed procedures in air traffic control (e.g., Dinges, 2007).

The scenario length allowed guiding three unequipped aircraft to the airport without violating the separation, speed, and altitude regulations. When the participants obeyed the regulations, four unequipped aircraft were left in the simulation at the end of each scenario. We left four unequipped aircraft in the simulation to avoid a decreasing task load.

Measures

Performance

The performance score was computed as follows:

Performance = (Compliance with the goal to enable as many aircraft as possible to be landed) - (Compliance with the separation rule) - (Compliance with the speed and altitude limits) .

Details of how we quantified the indicators are provided in Appendix C. Larger positive numbers indicated better performance across a theoretical range from −1 to +1.

Situation awareness

We measured situation awareness with the Situation Awareness Global Assessment Technique (Endsley, 1995a). Specifically, we interrupted the simulation once per trial to capture situation awareness. During the interruption, the system automatically administered a questionnaire, which was based on a goal-directed task analysis in accordance with Endsley, Bolte, and Jones (2003). The questionnaire included an empty screenshot of MAGIE and assessed the indicators for situation awareness summarized in Appendix C. After the participants finished answering the questions, which took on average between 3 and 4 min, MAGIE continued.

We favored this technique over self-ratings, as it provided a more valid measure of situation awareness, according to Endsley (1995a), and over real-time probes, such as the Situation Present Assessment Method (Durso et al., 1998), because it avoided having participants neglect their air traffic control task when answering real-time probes. Interrupting the simulation for probing situation awareness could also bias the performance score. To minimize these biases, we interrupted the simulation once during the last 40 s of each trial. The exact timing varied randomly across the trials. The system interrupted the participants late to give them time to establish situation awareness. The assessment of performance stopped as soon as the last 40 s started to avoid performance scores that would be based on scenarios of unequal durations.

We calculated three situation awareness scores on the basis of participant answers and in accordance with Endsley (1995b): (a) The perception score reflected the extent to which the participants had accurately perceived the information from the environment, (b) the comprehension score reflected the extent to which the participants had accurately understood the current situation, and (c) the prediction score reflected the extent to which the participants accurately projected the current situation into the future. Each score represented the mean value of its indicators (Appendix C). All scores had theoretical ranges from 0 to 1, with 1 representing maximal situation awareness.

Information-processing ability

The assessment of information-processing ability focused on information-processing complexity and speed (Appendix A). As control variables, we also measured the participants’ ability to process verbal and numerical material. We obtained all measures with the Berlin Intelligence Structure Test (Jäger, Süß, & Beauducel, 1997) for two reasons: First, it was the only German test based on the Berlin intelligence structure model (Jäger, 1982, 1984), and the first language of all participants was German. Using a test that is based on the Berlin intelligence structure model had the advantage that it assessed abilities with tasks that sampled across figural, numerical, and verbal content. Second, the test has satisfactory reliability and validity estimates (Jäger et al., 1997).

Working-memory capacity

We assessed one indicator for each of the capacities of working memory listed in Appendix A. Therefore, we selected three computer-based tasks from the task battery of Oberauer, Süß, Schulze, Wilhelm, and Wittmann (2000) and Oberauer, Süß, Wilhelm, and Wittmann (2003) according to two criteria: First, the tasks should provide reliable measures. Second, the measures should assess only one function each, to avoid confounds. Appendix D provides an overview over the selected tasks and how we calculated the scores for the capacities of the working-memory functions.

Procedure

Data collection for each participant occurred on two separate days and lasted about 4 hr per day. On the 1st day, the participants always controlled air traffic with MAGIE. The participants completed all training and test scenarios with one automation configuration before they were confronted with another one. Each automation configuration procedure consisted of the following steps: (1) The participants received a standard introduction in MAGIE and the automation configuration. (2) The participants completed the two training sessions (the first one without the situation awareness questionnaire, the second one with the questionnaire) and four test scenarios. Each scenario was completed in about 10 min. (3) The participants had a short break. Afterward, we repeated this procedure with another configuration until the participants had worked with all configurations.

On the 2nd day, we assessed the participants’ information-processing ability and working-memory capacity. Finally, the participants completed a demographic questionnaire.

We distributed the presentation order of the scenarios, the automation configurations, and the tests of the individual-difference measures randomly between participants to counterbalance order effects.

We instructed the participants that performance was more important than situation awareness to avoid participants’ actively memorizing the answers to the situation awareness probes during the assessment of performance (i.e., maximizing their scores on situation awareness at the expense of performance). We also told the participants to carefully monitor the automation because automation failures could occur. We did not tell the participants an exact indicator for the assumed automation reliability. However, the automation did not fail.

We did not ensure that the participants’ level of proficiency with an automation configuration met a specific criterion of expertise after training because it would have been difficult to ensure that the participants would be masters of the air traffic control task after training, as it takes years of training (see Hoffman, 1998). To avoid biased results, we used the trial number to statistically control differences in potential skill acquisition, in effects of individually differing reactions to skill acquisition, and in effects of the different automation configurations on skill acquisition.

Results

Preanalyses

The data analyses were based on 57 participants because data were lost from 3 participants due to technical issues. We first calculated descriptive statistics of the measured variables (Table 1). These statistics showed that the means and standard deviations of the information-processing abilities were comparable to the characteristics of the test-norming group (Jäger et al., 1997). The test norming was population representative.

Table 1:

Descriptive Statistics and Intercorrelations Between the Dependent Measures and the Individual-Difference Measures

Measure	1	2	3	4	5	6	7	8	9	10	M	SD
Dependent measures
1. Performance^a	—										0.51	0.71
2. Perception score of situation awareness^a	.20	—								⁴	0.28	0.10
3. Comprehension score of situation awareness^a	.30*	.60*	—								0.42	0.23
4. Prediction score of situation awareness^a	.33*	.52**	.74**								0.26	0.24
Information-processing ability
5. Information-processing complexity	.26*	.35*	.28*	.16	—						102.11	9.06
6. Information-processing speed	.05	.19	.07	.15	.04	—					97.70	9.75
7. Verbal ability	.13	.03	.10	.26	.06	.06	—				100.89	9.90
8. Numerical ability	.26*	.22	.16	.07	.06	.05	.03	—			96.68	9.59
Working-memory capacity
9. Supervision	.11	.18	.19	.23	.06	.00	.15	.01	—		0.00	1.00
10. Coordination	.09	.25	.24	.17	.35**	.16	.04	.25	.13	—	0.77	0.10
11. Simultaneous storage and processing	.37**	.26	.26	.25	.57**	.29*	.24	.33*	.10	.13	0.00	1.00

Note. All levels of significance base on N = 57.

For calculating the bivariate correlations, we drew on the mean value (across trials and automation configurations) for each dependent variable. The Ms and SDs base on N = 684 (57 participants, who worked with three automation configurations and four trials).

p < .05 (two tailed). **p < .01 (two tailed).

Second, we ensured that the assumptions underlying the ANOVAs and the multiple regressions, which we used for evaluating the hypotheses, were not violated. The internal consistency (α) of performance was .77, the consistencies of the situation awareness scores were αs = .91. We calculated these consistencies by treating the scores’ indicators listed in Appendix C as individual items. Potential issues, including skewness in the distributions of the variables, multicollinearity, and outliers, were not present with one exception: The capacity to simultaneously store and process information was highly correlated with the ability to process complex information (Table 1). Thus we separately investigated (a) whether the working-memory capacities influenced the dependent measures, (b) whether the information-processing abilities influenced the dependent measures, (c) whether the working-memory capacities moderated effects of the automation configuration on the dependent measures, and (d) whether the information-processing abilities moderated effects of the automation configuration on the dependent measures.

Effects of Automation Configuration on Performance and Situation Awareness

We drew on repeated-measures ANOVAs to test Hypothesis 1a and Hypothesis 1b. We used the performance and the situation awareness scores as dependent variables and the automation configuration and trial number as independent variables—without considering the individual-difference measures. The presentation order had no significant main or interaction effect on any dependent variable and was thus excluded.

The results supported Hypothesis 1a and Hypothesis 1b, all ps < .001 (Table 2). Figure 3 shows that the performance of the participants improved with the LOA. Performance was lowest with LOA1 = (moderate, low, low, low), a bit better with LOA2 = (moderate, moderate, low, low), and best with LOA3 = (moderate, moderate, moderate, low). In contrast, the situation awareness scores decreased with an increasing LOA. The participants achieved their best scores with LOA1 = (moderate, low, low, low). These scores decreased with LOA2 = (moderate, moderate, low, low) and were worst with LOA3 = (moderate, moderate, moderate, low).

Table 2:

Summary of the Repeated-Measures ANOVAs Evaluating the Effects of Automation Configuration and Trial Number on Performance and Situation Awareness

Variable	F	df_s, df_e	p		η²_p	1 – β
Performance
Automation configuration	24.16	2,112	<.001	^***	0.30	.99
Trial number	34.46	3,168	<.001	^***	0.38	.99
Automation Configuration × Trial Number	1.47	6,336	.189		0.03	.09
Situation awareness
Perception score
Automation configuration	10.86	2,112	<.001	^***	0.17	.84
Trial number	11.13	3,168	<.001	^***	0.17	.79
Automation Configuration × Trial Number	1.10	6,336	.360		0.02	.11
Comprehension score
Automation configuration	10.39	2,112	<.001	^***	0.16	.80
Trial number	2.35	3,168	.074		0.04	.20
Automation Configuration × Trial Number	1.12	6,336	.353		0.02	.09
Prediction score
Automation configuration	10.24	2,112	<.001	^***	0.16	.80
Trial number	2.51	3,168	.060		0.04	.19
Automation Configuration × Trial Number	1.49	6,336	.180		0.03	.09

Note. Power was calculated with G*Power (Version 3.1.9.2; see Faul, Erdfelder, Buchner, & Lang, 2009).

***

p < .001 (two tailed).

Figure 3.

Bar graphs with standard error bars visualizing the main effects of the automation configuration on (a) the performance score and (b) the situation awareness scores (mean across trials per automation configuration).

The repeated-measures ANOVAs also showed significant skill acquisition effects (Figure 4). The trial number, which we included to control such effects, affected performance and the perception score of situation awareness, ps < .001 (Table 2). These effects did not interact with the automation configuration, ps > .05. Due to these effects, we kept the trial number as a repeated-measures variable in the ANCOVAs when investigating the interaction effects.

Figure 4.

Line graphs with standard error bars visualizing the change of (a) performance and (b) perceptual situation awareness across the four trials (aggregated per automation configuration).

Ability Predictors of Performance and Situation Awareness

We evaluated Hypothesis 2 with multiple regressions. The criteria were the mean values of performance and situation awareness scores across automation configurations and trials. The individual-difference measures were the predictors. One the one hand, the results supported Hypothesis 2 (Table 3). Performance improved with the participants’ ability to process complex information and the capacity to simultaneously store and process information, ps < .05. Similarly, the perception score of situation awareness was higher for the individuals with higher abilities in processing complex information and better supervision capacity, ps < .05. Last, the prediction score increased when the participants could better process complex information, p < .05. On the other hand, the results (Table 3) did not show significant links between the abilities and the comprehension score, all ps > .05, and did not confirm Hypothesis 2.

Table 3:

Summary of the Multiple Regression Analyses

						Situation Awareness
	Performance^a					Perception Score^b					Comprehension Score^c					Prediction Score^d
Variable	B	SE (B)	β	t(49)	p	B	SE (B)	β	t(49)	p	B	SE (B)	β	t(49)	p	B	SE (B)	β	t(49)	p
Information-processing (IP) ability
IP complexity	.02	0.01	.28	1.83	.04*	.30	0.02	.31	1.66	.05*	.02	0.02	0.28	1.52	.06	.03	0.01	.29	1.71	.04*
IP speed	.02	0.01	.20	1.20	.12	.01	0.02	.16	0.86	.20	.02	0.01	0.22	1.25	.11	.02	0.02	.17	0.96	.17
Verbal ability	.02	0.01	.25	1.55	.06	.02	0.02	.23	1.35	.09	.01	0.01	0.12	0.73	.23	.02	0.02	.18	0.99	.16
Numerical ability	.02	0.02	.16	0.92	.18	.00	0.01	.03	0.16	.44	.01	0.01	0.12	0.78	.22	.01	0.01	.08	0.49	.31
Working-memory capacity
Coordination	.39	1.23	.04	0.32	.38	.77	1.35	.08	0.57	.29	.94	1.20	1.11	0.78	.22	.77	1.35	.08	0.57	.28
St. and processing	.31	0.16	.31	2.04	.02*	.06	0.18	.06	0.35	.36	.16	0.12	0.16	1.23	.11	.21	0.18	.21	1.36	.09
Supervision	.05	0.12	.05	0.22	.33	.32	0.15	.32	2.07	.02*	.77	0.12	0.16	0.12	.23	.17	0.13	.17	1.27	.10

Note. For these multiple regressions, we calculated the mean values of the dependent measures across trials and automation configurations for each participant. We used the resulting variables as the criteria in the multiple regressions. St. = simultaneous storage.

The predictors explained 17.8% (adjusted R²) of the variance of the performance score, F(7, 48) = 2.71, p < .05 (two sided).

The predictors explained 13.1% (adjusted R²) of the variance of the perception score, F(7, 48) = 2.24, p < .05 (two sided).

The predictors explained 10.3% (adjusted R²) of the variance of the comprehension score, F(7, 48) = 1.04, p > .05 (two sided).

The predictors explained 12.8% (adjusted R²) of the variance of the prediction score, F(7, 48) = 2.15, p < .05 (two sided).

p < .05 (one tailed).

Interaction Effects of Automation Configuration and Abilities on Performance and Situation Awareness

We evaluated Hypothesis 3 with repeated-measures ANCOVAs. We categorized the participants according to their individual-difference measures in three groups such that each group consisted of n = 19 participants with one exception: The group with a medium level of information-processing speed consisted of n = 20 participants; the group with a high level of information-processing speed had n = 18 participants. We implemented this exception because two participants achieved the same score here. For categorization, we did not consider the manipulation of the presentation order. This covariate was excluded because it had no significant main or interaction effect. For the ANCOVAs, we used the categorized individual-difference measures as the covariates and the automation configuration and the trial number as within-subject independent variables to predict performance and situation awareness.

The results supported Hypothesis 3 (Table 4). With regard to performance (Hypothesis 3a), the automation configuration interacted with the ability to process complex information and with the capacity for simultaneous storage and processing, both ps < .05. The directions of the effects are illustrated in the bar graphs presented in Figure 5a: The participants with low ability to process complex information performed best with LOA1 = (moderate, low, low, low) and worst with LOA3 = (moderate, moderate, moderate, low). In contrast, the participants with high ability to process complex information performed best with LOA3 = (moderate, moderate, moderate, low) and worst with LOA1 = (moderate, low, low, low). A similar fan effect was present with regard to the capacity to simultaneously store and process information (Figure 5b). We could thus confirm Hypothesis 3a.

Table 4:

Summary of the Repeated-Measures’ ANCOVAs Evaluating the Effects of Automation Configuration, Trial Number, Abilities, and Their Interactions on the Dependent Measures

						Situation Awareness
	Performance					Perception Score					Comprehension Score					Prediction Score
Variable	F	df_s, df_e	p	η²_p	1 – β	F	df_s, df_e	p	η²_p	1 – β	F	df_s, df_e	p	η²_p	1 – β	F	df_s, df_e	p	η²_p	1 – β
Information-processing (IP) ability
AC	11.67	2, 42	<.001***	0.24	.96	6.99	2, 42	.002***	0.23	.95	6.77	2, 42	.003**	0.18	.87	8.17	2, 42	<.001***	0.28	.99
AC * IP complexity	3.03	4, 42	.028*	0.14	.62	<1	4, 42	.682	0.05	.30	1.52	4, 42	.214	0.09	.40	3.71	4, 42	.011*	0.26	.94
AC × IP Speed	1.80	4, 42	.140	0.09	.40	3.33	4, 42	.019*	0.22	.94	2.64	4, 42	.047*	0.14	.63	1.57	4, 42	.200	0.13	.58
AC × Verbal Ability	2.76	4, 42	.040*	0.13	.58	<1	4, 42	.822	0.03	.20	<1	4, 42	.606	0.04	.18	<1	4, 42	.537	0.06	.26
AC × Numerical Ability	<1	4, 42	.719	0.03	.14	<1	4, 42	.467	0.07	.41	<1	4, 42	.719	0.03	.14	2.32	4, 42	.072	0.18	.77
Trial number	6.96	3, 63	<.001***	0.16	.70	4.58	3, 63	.005**	0.16	.76	2.46	3, 63	.071	0.07	.35	1.07	3, 63	.368	0.05	.22
Working-memory capacity
AC	15.42	2, 72	<.001***	0.27	.98	6.95	2, 72	.002***	0.16	.82	1.11	2, 72	.335	0.03	.19	1.07	2, 72	.348	0.03	.19
AC × Coordination	1.64	4, 72	.173	0.07	.31	1.11	4, 72	.360	0.06	.26	1.06	4, 72	.381	0.05	.22	1.45	4, 72	.240	0.07	.31
AC × St. and Processing	3.19	4, 72	.018*	0.14	.63	2.52	4, 72	.049*	0.12	.54	1.14	4, 72	.345	0.05	.22	<1	4, 72	.761	0.02	.11
AC × Supervision	<1	4, 72	.698	0.03	.14	2.15	4, 72	.083	0.11	.49	2.66	4, 72	.039*	0.12	.54	<1	4, 72	.425	0.05	.22
Trial number	11.34	3, 108	<.001***	0.22	.91	3.03	3, 108	.032*	0.08	.40	<1	3, 108	.407	0.02	.12	<1	3, 108	.680	0.01	.08

Note. We did not report the statistics (a) for the two-way interaction effects between the trial number and the individual-difference measures and (b) for the three-way interaction effects between the trial number, the automation configuration, and the individual-difference measures because none of the effects reached an appropriate level of significance, all ps > .05, and because such effects were not in the focus of this study. Power was calculated with G*Power (Version 3.1.9.2; see Faul, Erdfelder, Buchner, & Lang, 2009). AC = automation configuration; St. = simultaneous storage.

p < .05 (two tailed). **p < .01 (two tailed). ***p < .001 (two tailed).

Figure 5.

Bar graphs with standard error bars visualizing the interaction effects between (a) the categorized ability to process complex information and automation configuration and (b) the categorized capacity to simultaneously store and process information and automation configuration on performance (mean across trials per automation configuration).

The results of the ANCOVAs also supported Hypothesis 3b. First, the interaction effects between the automation configuration and the ability to quickly process cognitively easy information and between the automation configuration and the capacity to simultaneously store and process information significantly influenced the perception score of situation awareness, ps < . 05 (Table 4). The effects are illustrated in Figure 6: The participants with low information-processing speed and low capacity for simultaneous storage and processing achieved their best scores with LOA1 = (moderate, low, low, low) and their worst scores with LOA3 = (moderate, moderate, moderate, low). In contrast, the participants with medium and high information-processing speed and capacity for simultaneous storage and processing achieved better perception scores when LOA1 = (moderate, low, low, low) was not active.

Figure 6.

Bar graphs with standard error bars visualizing the interaction effects (a) between the categorized information-processing speed and automation configuration and (b) between the categorized simultaneous storage and processing and automation configuration on the perception score of situation awareness (mean across trials per automation configuration).

Second, the interaction effects between automation configuration and information-processing speed and between automation configuration and supervision influenced the comprehension score, ps < .05 (Table 4). The participants with low information-processing speed and supervision capacities achieved their best scores with LOA1 = (moderate, low, low, low) (Figure 7). With higher levels of information-analysis and decision-selection automation, their scores were lower. In contrast, the participants with high information-processing speed and supervision capacity benefited from higher levels of information-analysis and decision-selection automation.

Figure 7.

Bar graphs with standard error bars visualizing the interaction effects (a) between the categorized information-processing speed and automation configuration and (b) between the categorized supervision capacity and automation configuration on the comprehension score of situation awareness (mean across trials per automation configuration).

Finally, the interaction effect between the automation configuration and information-processing complexity significantly influenced the prediction score of situation awareness (Table 4). The direction of this effect is obvious when one inspects the bar graph presented in Figure 8. The participants with low information-processing complexity achieved their lowest prediction scores with LOA3 = (moderate, moderate, moderate, low). In contrast, participants with high information-processing complexity achieved their lowest scores with LOA1 = (moderate, low, low, low).

Figure 8.

Bar graph with standard error bars visualizing the interaction effect between information-processing complexity and automation configuration on the prediction score of situation awareness (mean across trials per automation configuration).

Discussion

Abilities Can Determine Effects of LOAs on Performance and Situation Awareness

The ability profiles of the operators influenced the effects of higher levels of information-analysis and decision-selection automation on performance and situation awareness according to our results. In our study, the operators with higher information-processing ability and working-memory capacity benefited from higher levels of information-analysis and decision-selection automation. In contrast, the operators with lower information-processing ability and working-memory capacity seemed to be relatively disadvantaged when the level of information-analysis and decision-selection automation increased. No previous study has investigated the role of information-processing ability and working-memory capacity on effects of information-analysis and decision-selection automation. Previous attempts to identify the reasons for inconsistent effects of automation have mainly focused on task and automation characteristics (e.g., Manzey, Reichenbach, & Onnasch, 2012; Onnasch et al., 2014; Sarter & Schroeder, 2001; Wickens, Hollands, Banbury, & Parasuraman, 2012).

According to our results, information-processing ability and working-memory capacity appear to influence performance and situation awareness more when the levels of information-analysis and decision-selection automation increase from low to moderate. One reason may be an increasing complexity of the mental model, which the operators need for completing the task along with the automation (see also Kaber et al., 2005; Rauterberg, 1992). The complexity might have increased because the operators needed to incorporate the automated tasks into their own mental model. Specifically, the operators had to understand the logic of the higher level of information-analysis automation, that is, the ghosts and targets that pointed to a need for action and the higher level of decision-selection automation, that is, the advisories. Therefore, the operators had to attend to relevant information from the automation (e.g., perceive the ghosts), simultaneously store and process this information (e.g., relate the positioning of the ghosts with the real aircraft), and reason about the information (e.g., identify a need for action). The operators with higher ability to process complex and cognitively easy information and with higher capacity to simultaneously store and process information seemed to be better able to integrate this information into their mental model of the task. This integration, in turn, appeared to allow these operators to perform better and to establish better situation awareness. They made more effective use of the information from the higher levels of information-analysis and decision-selection automation. In contrast, the operators with lower ability to process complex and cognitively easy information and with lower capacity to simultaneously store and process information appeared not to be able to use this information for deep processing (see Petty & Cacioppo, 1986). They seemed more likely to avoid cognitive effort, reduced their intrinsic load (see Atwood & Polson, 1976), and thus neither improved their performance nor established better situation awareness. Consequently, the operators’ ability level determined whether higher levels of information-analysis and decision-selection automation fostered higher performance and situation awareness.

Abilities Can Influence Situation Awareness

Our results replicated previous findings regarding the cognitive processes underlying the establishment of situation awareness. First, we replicated the results of Harbluk, Noy, Trbovich, and Eizenman’s (2007) by relating supervision, which resembles the central executive (Oberauer et al., 2003), to perceptual situation awareness. The capability to actively guide attention contributed to accurately perceiving relevant information from the environment. However, we could not replicate the results of Sangeun (2008). Sangeun (2008) related measures of operational span, which can be interpreted as contributing to the simultaneous storage-and-processing function of working memory (Oberauer et al., 2000), with her score of perceptual situation awareness. One reason for this divergent result may be the task environment, as Sangeun drew on a driving task (see also Durso & Drews, 2010; Lau, Jamieson, & Skraaning, 2013). Second, we could not identify a significant association between information-processing ability or working-memory capacity and the comprehension score of situation awareness—consistent with Sulistyawati, Wickens, and Chui (2011). Finally, the ability to process complex information contributed to the establishment of predictive situation awareness, a connection that Sulistyawati et al. also reported. These corroborating results confirm the findings of the previous studies (for the need for replication, see Jones, Derby, & Schmidlin, 2010).

Conclusion and Future Work

Future work should investigate whether information-processing ability and working-memory capacity moderate the effects of higher levels of information-analysis and decision-selection automation on performance and situation awareness when automation failures occur. Our expectation would be that the experience of failures should increase the complexity of the mental model required for completing the task in question with the automation assistance (see Hoffman et al., 2014). Thus cognitive abilities may be more highly related to outcome measures.

Moreover, authors of future work should investigate how the relationship between cognitive abilities and automation configuration changes, if the levels of information-analysis and decision-selection automation increase even more, and if the levels of information-analysis and action-implementation automation change. First, we would expect that the effects may increase when the level of the information-analysis and decision-selection automation increases even more, because then the establishment of a valid mental model may be more complex. Second, we would not expect that the effects would change if the levels of information-analysis and decision-selection automation change, because Kaber et al. (2005) considered both as low-order information-processing functions. A change in the LOA may not make the mental model more or less complex.

Future work should also focus on replicating our results with participants with higher levels of proficiency. Several researchers proposed levels of categories of proficiency (e.g., Dreyfus & Dreyfus, 1986; Hoffman, 1998). For instance, Hoffman (1998) distinguished naives, novices, initiates, apprentices, journeymen, experts, and masters and provided criteria for when learners can be considered as having which level of proficiency. On the basis of these levels, our participants may be considered initiates at maximum—novices who began training in a specific field. This procedure was similar to that of many other studies in this context (e.g., Endsley & Kaber, 1999; Manzey et al., 2011; Parasuraman, Cosenzo, & De Visser, 2009). Still, it is crucial to know whether our results can be expended to experts and masters. Then, training and personnel selection strategies might be optimized by focusing on the alignment between operator ability levels and automation demands.

Furthermore, intraindividual differences in the epistemic knowledge that operators pick up from instructions (see Kluwe, 1997) and in motivation (see Szalma, 2009) may also be relevant. Petty and Cacioppo (1986) pointed out that motivation can also determine whether humans prefer heuristics or deep processing of information. Future work should thus also focus on investigating whether an experimental manipulation of motivation might also have influences similar to cognitive abilities. We would expect that the influences are similar, such that more motivated operators perform better with higher LOAs in general than do less motivated operators. One way to manipulate (extrinsic) motivation would be to give operators bonuses for better performance and situation awareness.

Finally, research is needed to explore the boundary conditions across (a) different domains (e.g., medicine), (b) different tasks in the same domain (e.g., tower control), (c) different automation systems in that domain (e.g., midterm conflict detection), and (d) different ways of implementing the same automation (e.g., displaying the advisories in a separate list in MAGIE). As Flach (2015) pointed out, “context matters” (p. 59), and the detailed relationships could vary between domains and tasks (e.g., Durso & Drews, 2010; Lau et al., 2013). If the results can be replicated in different settings, a definite answer of whether information-processing ability and working-memory capacity determine effects of increasing levels of information-analysis and decision-selection automation can be resolved.

Nevertheless, information-processing ability and working-memory capacity moderated the effects of higher levels of information-analysis and decision-selection automation on performance and situation awareness at least in our study. If these results can be replicated and generalized, it is important to enable the automation to balance these interindividual differences or to provide systems that assist humans to deal with the increased complexity of the mental model. Balancing interindividual differences in relevant abilities could be achieved by enabling automation to categorize the level of information-processing ability and working-memory capacity of its user and to adapt its LOA such that outcome measures are optimized—a procedure typical for adaptive automation systems (e.g., Kaber & Endsley, 2004; Scerbo, 1996). According to our results, the adaptation should ensure that the operators with lower abilities to process complex information receive assistance from LOA = (moderate, low, low, low) or LOA = (moderate, moderate, low, low). Operators with high ability to process complex information should receive assistance from LOA = (moderate, moderate, moderate, low).

Systems assisting humans to deal with the increased complexity of the mental model could make use of automation that mirrors human perception-cognition-action processes. Therefore, it may be an option to enable automation to observe the operator when executing the tasks manually and when thinking aloud about the tasks and to copy these processes. Another option may be to enable automation to clearly and comprehensively explain its own reasoning processes, decision, and actions to the operator. This last option may not reduce complexity but may assist operators in understanding the complexity more quickly. Both options may thus enable operators to build a shared problem representation more easily (see Christoffersen & Woods, 2002) and increase the transparency of the automation (see Kaber et al., 2005). These mechanisms may enable operators with lower abilities to also profit from higher levels of information-analysis and decision-selection automation.

Footnotes

Appendix

Appendix D

Description of the Tasks and Measures Used to Capture the Capacities of the Working-Memory Functions

Working-Memory Capacity	Computer-Based Tasks Used to Capture the Working-Memory Capacity^a	Measure
Coordination	The dot-span task presented dots shortly and required participants to indicate whether the pattern formed by the dots was symmetrical.	The measure for coordination capacity was the number of dots with which the participants could still establish the structure correctly (Oberauer, Süß, Schulze, Wilhelm, & Wittmann, 2000). Larger numbers indicated better coordination capacity.
Simultaneous storage and processing	We also used the dot-span task to capture simultaneous storage and processing by evaluating whether the participants could also reproduce the positioning of the dots in the order of their presentation. Additionally, we captured simultaneous storage and processing with the memory-updating numerical task that required participants to memorize digits and their positions in a matrix, to execute simple mathematical operations with selected digits, and to recall the results together with their positions.	We measured the number of digits/dots the participants could still master successfully. The measures of both tasks were internally consistent (α > .80) and captured only simultaneous storage and processing (Oberauer et al., 2000; Oberauer, Süß, Wilhelm, & Wittmann, 2003). Thus we z-standardized the measures and calculated their mean value to yield one score for simultaneous storage and processing. Higher scores indicate participants have higher levels of simultaneous storage and processing ability.
Supervision	The task of figural task switching required participants to react to switching instructions with figural-spatial material.	We measured the mean accuracy and reaction time, z-standardized both measures, and calculated their mean value to yield one measure for supervision. Higher scores indicate higher levels of supervision ability.

For details about the tasks, their internal consistencies, and their validity, see Oberauer et al. (2000, 2003).

Acknowledgements

The authors would like to thank Nils Carstengerdes, Andreas Hasselberg, Peer Manske, and Jan Kraemer for their assistance with the data collection as well as Michael Eisele, Raja Parasuraman, and Uwe Teegen for their valuable feedback on an earlier version of this paper.

Meike Jipp is head of the Human Factors division at the Institute of Transportation Systems of the German Aerospace Centre (DLR) in Braunschweig, Germany. Her research focuses on determinants of human performance in interaction with automation systems. She received her PhD in psychology from the University of Mannheim, Germany.

Phillip L. Ackerman is a professor of psychology at the Georgia Institute of Technology. His research focuses on cognitive, affective, and conative determinants of individual differences in learning, skill acquisition, and performance. He received his PhD in psychology from the University of Illinois.

References

Ahlstrom

(2005). Work domain analysis for air traffic controller weather displays. Journal of Safety Research, 36, 159–169. doi:10.1016/j.jsr.2005.03.001

Alam

Nguyen

M. H.

Abbass

H. A.

Lokan

Ellejmi

Kirby

(2010). A dynamic continuous descent approach methodology for low noise and emission. Proceedings of the IEEE/AIAA Digital Avionics Systems Conference, 29, 1.E.5-1–1.E.5-18. doi:10.1109/DASC.2010.5655502

Atwood

M. E.

Polson

P. G.

(1976). A process model for water jug problems. Cognitive Psychology, 8, 191–216. doi:10.1016/0010-0285(76)90023-2

Baddeley

A. D.

(1986). Working memory. Oxford, UK: Clarendon.

Bainbridge

(1983). Ironies of automation. Automatica, 19, 775–779.

Blom

H. A.

Stroeve

S. H.

Everdij

M. H.

van der Park

M. N.

(2003). Human cognition performance model to evaluate safe spacing in air traffic. Human Factors and Aerospace Safety, 3, 59–82.

Bucik

Neubauer

A. C.

(1996). Bimodality in the Berlin model of intelligence structure (BIS): A replication study. Personality and Individual Differences, 21, 987–1005. doi:10.1016/S0191-8869(96)00129-8

Buckley

E. P.

DeBaryshe

B. D

Hitchner

Kohn

(1983). Methods and measurements in real time air traffic control system simulation (Report No. DOT/FAA/CT-TN83/26). Atlantic City, NJ: DOT/FAA Technical Center.

Calhoun

G. L.

Draper

M. H.

Ruff

H. A.

(2009). Effect of level of automation on unmanned aerial vehicle routing task. In Proceedings of the Human Factors and Ergonomics Society 53rd Annual Meeting (pp. 197–201). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1177/154193120905300408

10.

Christoffersen

Woods

D. D.

(2002). How to make automated systems team players. In Salas

(Ed.), Advances in human performance and cognitive engineering research (pp. 1–12). St. Louis, MO: Elsevier.

11.

Cummings

M. L.

Mitchell

P. J.

(2007). Operator scheduling strategies in supervisory control of multiple UAVs. Aerospace Science and Technology, 11, 339–348. doi:10.1016/j.ast.2006.10.007

12.

D’Arcy

Della Rocco

(2001). Air traffic control specialist decision making and strategic planning: A field survey (DOT/FAA/CT-TN01/05). Washington, DC: Federal Aviation Administration.

13.

Dinges

(2007, July). Determining the environmental benefits of implementing continuous descent approach procedures. Paper presented at the 7th USA/Europe Air Traffic Management R&D Seminar, Barcelona, Spain.

14.

Dreyfus

H. L.

Dreyfus

S. E

(1986). Mind over machine. New York, NY: Free Press.

15.

Durso

F. T.

Dattell

A. R.

(2006). Expertise and transportation. In Ericsson

K. A.

Charness

Feltovich

P. J.

Hoffman

R. R.

(Eds.), The Cambridge handbook of expertise and expert performance (pp. 1084–1134). New York, NY: Cambridge University Press.

16.

Durso

F. T.

Drews

F. A.

(2010). Health care, aviation, and ecosystems: A socio-natural systems perspective. Current Directions in Psychological Science, 19, 71–75. doi:10.1177/0963721410364728

17.

Durso

F. T.

Manning

C. A.

(2008). Air traffic control. Reviews of Human Factors and Ergonomics, 4, 195–244. doi:10.1518/155723408X342853

18.

Durso

F. T.

Truitt

T. R.

Hackworth

Crutchfield

Manning

C. A.

(1998). En route operational errors and situational awareness. International Journal of Aviation Psychology, 82, 177–194.

19.

Endsley

M. R.

(1987). The application of human factors to the development of expert systems for advanced cockpits. In Proceedings of the Human Factors Society and Ergonomics Society 31st Annual Meeting (pp. 1388–1392). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1177/154193128703101219

20.

Endsley

M. R.

(1995a). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65–84. doi:10.1518/001872095779049499

21.

Endsley

M. R.

(1995b). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64. doi:10.1518/001872095779049543

22.

Endsley

M. R.

Bolte

Jones

D. G.

(2003). Designing for situation awareness: An approach to human-centered design. London, UK: Taylor & Francis.

23.

Endsley

M. R.

Kaber

D. B.

(1999). Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics, 42, 462–492. doi:10.1080/001401399185595

24.

Endsley

M. R.

Kiris

E. O.

(1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37, 381–394. doi:10.1518/001872095779064555

25.

Faul

Erdfelder

Buchner

Lang

A.-G.

(2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149–1160. doi:10.3758/BRM.41.4.1149

26.

Ferris

Sarter

Wickens

C. D.

(2010). Cockpit automation: Still struggling to catch up. In Salas

Allard

Maurino

(Eds.), Human factors in aviation (2nd ed., pp. 479–503). Amsterdam, Netherlands: Elsevier.

27.

Flach

J. M.

(1995). Situation awareness: Proceed with caution. Human Factors, 37, 149–157. doi:10.1518/001872095779049480

28.

Flach

J. M.

(2015). Situation awareness: Context matters! A commentary on Endsley. Journal of Cognitive Engineering and Decision Making, 9, 59–72. doi:10.1177/1555343414561087

29.

Gaba

D. M.

Howard

S. K.

Small

S. D.

(1995). Situation awareness in anesthesiology. Human Factors, 37, 20–31. doi:10.1518/001872095779049435

30.

Galushka

Frederick

Mogford

Krois

(1995). Plan view display baseline research report (Report No. DOT/FAA/CT-TN95/45). Atlantic City, NJ: DOT/FAA Technical Center.

31.

Gilson

R. D.

(1995). Special issue preface. Human Factors, 37, 3–4. doi:10.1518/001872095779049426

32.

Guilford

J. P.

(1967). The nature of human intelligence. New York, NY: McGraw-Hill.

33.

Harbluk

J. L.

Noy

Y. I.

Trbovich

P. L.

Eizenman

(2007). An on-road assessment of cognitive distraction: Impacts on drivers’ visual behavior and braking performance. Accident Analysis & Prevention, 39, 372–379.

34.

Hoffman

R. R.

(1998). How can expertise be defined? Implications of research from cognitive psychology. In Williams

Faulkner

Fleck

(Eds.), Exploring expertise (pp. 81–100). New York, NY: Macmillan.

35.

Hoffman

R. R.

Ward

Feltovich

P. J.

DiBello

Fiore

S. M.

Andrews

D. H.

(2014). Accelerated expertise: Training for high proficiency in a complex world. New York, NY: Psychology Press.

36.

Jäger

A. O.

(1982). Mehrmodale Klassifikation von Intelligenzleistungen: Experimentell kontrollierte Weiterentwicklung eines deskriptiven Intellingenzstrukturmodells [Multimodal classification of intelligence achievements: Experimentally controlled development of a descriptive intelligence structure model]. Diagnostica, 28, 195–225.

37.

Jäger

A. O.

(1984). Intelligenzstrukturforschung, konkurrierende Modelle, neue Entwicklungen, Perspektiven [Intelligence structure research, competing models, new developments, perspectives]. Psychologische Rundschau, 35, 21–35.

38.

Jäger

A. O.

Süß

H.-M.

Beauducel

(1997). Berliner Intelligenzstruktur-Test. Form 4 [Berlin Intelligence Structure Test. Form 4]. Göttingen, Germany: Hogrefe.

39.

Jäger

A. O.

Tesch-Römer

(1988). Replikation des Berliner Intelligenzstrukturmodells (BIS) in den “Kit of Reference Tests for cognitive factors” nach French, Ekstrom & Price (1963): Eine Reanalyse der Daten von Scholl (1976) [Replication of the Berlin Intelligence Structure Model (BIS) in the “Kit of Reference Tests for Cognitive Factors” according to French, Ekstrom, & Price (1963): A reanalysis of data from Scholl (1976)]. Zeitschrift für Differentielle und Diagnostische Psychologie, 9, 77–96.

40.

Jones

D. G.

Endsley

M. R.

(1996). Sources of situation awareness errors in aviation. Aviation, Space and Environmental Medicine, 67, 507–512.

41.

Jones

K. S.

Derby

P. L.

Schmidlin

E. A.

(2010). An investigation of the prevalence of replication research in Human Factors. Human Factors, 52, 586–595. doi:10.1177/0018720810384394

42.

Kaber

D. B.

Endsley

M. R.

(1997). The combined effect of level of automation and adaptive automation on human performance with complex, dynamic control systems. In Proceedings of the Human Factors and Ergonomics Society 41st Annual Meeting (pp. 205–209). Santa Monica, CA: Human Factors and Ergonomics Society.

43.

Kaber

D. B.

Endsley

M. R.

(2004). The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theoretical Issues in Ergonomics Science, 5, 113–153. doi:10.1080/1463922021000054335

44.

Kaber

D. B.

Onal

Endsley

M. R.

(2000). Design of automation for telerobots and the effect on performance, operator situation awareness, and subjective workload. Human Factors and Ergonomics in Manufacturing & Service Industries, 10, 409–430. doi:10.1002/1520-6564(200023)

45.

Kaber

D. B.

Wright

M. C.

Prinzel

L J.

Clamann

M. P.

(2005). Adaptive automation of human–machine system information-processing functions. Human Factors, 47, 730–741. doi:10.1518/001872005775570989

46.

Kluwe

R. H.

(1997). Acquisition of knowledge in the control of a simulated technical system. Le Travail Humain, 60, 61–85.

47.

Lau

Jamieson

G. A.

Skraaning

(2013). Distinguishing three accounts of situation awareness based on their domains of origin. In Proceedings of the Human Factors and Ergonomics Society 57th Annual Meeting (pp. 220–224). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1177/1541931213571049

48.

Lecchini

Glover

Lygeros

Maciejowski

(2005). Air traffic control with an expected value criterion. Proceedings of the IFAC World Congress, 16, 13–18. doi:10.3182/20050703-6-CZ-1902.02025

49.

Lin

C. J.

Yenn

Yang

(2010). Automation design in advanced control rooms of the nuclear power plants. Safety Science, 48, 63–71.

50.

Manzey

Luz

Mueller

Dietz

Meixensberger

Strauss

(2011). Automation in surgery: The impact of navigated-control assistance on performance, workload, situation awareness, and acquisition of surgical skills. Human Factors, 53, 584–599. doi:10.1177/0018720811426141

51.

Manzey

Reichenbach

Onnasch

(2012). Human performance consequences of automated decision aids: The impact of degree of automation and system experience. Journal of Cognitive Engineering and Decision Making, 6, 57–87. doi:10.1177/1555343411433844.

52.

McFarland

A. L.

(1997). A conflict probe to provide early benefits for airspace users and controllers. McLean, VA: Mitre Corporation.

53.

Ntuen

C. A.

Park

E. H.

(1988). Human factors issues in teleoperated systems. In Karwowski

Parsaei

H. R.

Wilhelm

M. R.

(Eds.), Proceedings of the First International Conference on Ergonomics of Hybrid Automated Systems (pp. 203–210). Amsterdam, Netherlands: Elsevier.

54.

Oberauer

Süß

H.-M.

Schulze

Wilhelm

Wittmann

W. W.

(2000). Working memory capacity: Facets of a cognitive ability construct. Personality and Individual Differences, 29, 1017–1045. doi:10.1016/S0191-8869(99)00251-2

55.

Oberauer

Süß

H.-M.

Wilhelm

Wittmann

W. W.

(2003). The multiple faces of working memory: Storage, processing, supervision, and coordination. Intelligence, 31, 167–193. doi:10.1016/S0160-2896(02)00115-0

56.

Oberheid

Hasselberg

Söffker

(2011). Know your options: Analyzing human decision making in dynamic task environments. In de Waard

Gérard

Onnasch

Wiczorek

Manzey

(Eds.), Human-centred automation (pp. 285–300). Maastricht, Netherlands: Shaker.

57.

Oberheid

Weber

Rudolph

(2009). Visual controller aids to support late merging operations for fuel efficient and noise reduced approach procedures. In Proceedings of the Human Factors and Ergonomics Society 53rd Annual Meeting (pp. 11–15). Santa Monica, CA: Human Factors and Ergonomics Society. doi:10.1177/154193120905300104

58.

Onnasch

Wickens

C. D.

Manzey

(2014). Human performance consequences of stages and levels of automation: An integrated meta-analysis. Human Factors, 56, 476–488. doi:10.1177/0018720813501549

59.

Parasuraman

Cosenzo

K. A.

De Visser

(2009). Adaptive automation for human supervision of multiple uninhabited vehicles: Effects on change detection, situation awareness, and mental workload. Military Psychology, 21, 270–297. doi:10.1080/08995600902768800

60.

Parasuraman

Riley

V. A.

(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39, 230–253. doi:10.1518/001872097778543886

61.

Parasuraman

Sheridan

T. B.

Wickens

C. D.

(2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics, 30, 286–296. doi:10.1109/3468.844354

62.

Parasuraman

Sheridan

T. B.

Wickens

C. D.

(2008). Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. Journal of Cognitive Engineering and Decision Making, 2, 140–160. doi:10.1518/155534308X284417

63.

Petty

R. E.

Cacioppo

J. T.

(1986). The elaboration likelihood model of persuasion. In Berkowitz

(Ed.), Advances in experimental social psychology (pp. 123–205). New York, NY: Academic Press.

64.

Pritchett

A. R.

(2015). Preface to the JCEDM special issue on situation awareness. Journal of Cognitive Engineering and Decision Making, 9, 3. doi:10.1177/1555343415572807

65.

Ramsbottom

Baker

(2003). Kalman filtering for monitoring and control driveline in applications. SAE technical paper. doi:10.4271/2003-01-1280

66.

Rauterberg

(1992). A method of quantitative measurement of cognitive complexity. In van der Veer

Tauber

Bagnara

Antalovits

(Eds.), Human–computer interaction: Tasks and organisation (pp. 295–307). Rome, Italy: CUD.

67.

Rosas

(1991). Replikation des Berliner Intelligenzstrukturmodells (BIS) und Vorhersagbarkeit des Studienerfolgs bei chilenischen Studenten [Replication of the Berlin Intelligence Structure Model (BIS) and predictability of study success for Chilenian students]. Berlin, Germany: Free University of Berlin.

68.

Sangeun

(2008). The effect of driver cognitive abilities and distraction on situation awareness and performance under hazard conditions (Unpublished MSc thesis). Raleigh: North Carolina State University.

69.

Sarter

N. B.

Schroeder

(2001). Supporting decision making and action selection under time pressure and uncertainty: The case of in-flight icing. Human Factors, 43, 573–583. doi:10.1518/001872001775870403

70.

Scerbo

M. W.

(1996). Theoretical perspectives on adaptive automation. In Parasuraman

Mouloua

(Eds.), Automation and human performance: Theory and application (pp. 37–63). Mahwah, NJ: Lawrence Erlbaum.

71.

Sheridan

T. B.

Parasuraman

(2006). Human–automation interaction. Reviews of Human Factors and Ergonomics, 1, 89–129. doi:10.1518/155723405783703082

72.

Sheridan

T. B.

Verplank

W. L.

(1978). Human and computer control of undersea teleoperators (Report No. N00014-77-C-0256). Cambridge: Massachusetts Institute of Technology.

73.

Siegler

R. S.

Chen

(2002). Development of rules and strategies: Balancing the old and the new. Journal of Experimental Child Psychology, 81, 446–457. doi:10.1006/jecp.2002.2666

74.

Spearman

(1927). The abilities of man. New York, NY: Macmillan.

75.

Stanton

N. A.

(2010). Situation awareness: Where have we been, where are we now and where are we going? Theoretical Issues in Ergonomics Science, 11, 1–6. doi:10.1080/14639220903009870

76.

Sulistyawati

Wickens

C. D.

Chui

Y. P.

(2011). Prediction in situation awareness: Confidence bias and underlying cognitive abilities. International Journal of Aviation Psychology, 21, 153–174. doi:10.1080/10508414.2011.556492

77.

Süß

H.-M.

(1999). Intelligenz und komplexes Problemlösen: Perspektiven für eine Kooperation zwischen differentiell-psychometrischer und kognitionspsychologischer Forschung [Intelligence and complex problem solving: Perspectives for a cooperation between differential-psychometrical and cognitive psychological research]. Psychologische Rundschau, 50, 220–228.

78.

Szalma

J. L.

(2009). Individual differences in human-technology interaction: Incorporating variation in human characteristics into human factors research and design. Theoretical Issues in Ergonomics Science, 10, 381–397. doi:10.1080/14639220902893613

79.

Thompson

S. D.

(1997). Terminal area separation standards: Historical development, current standards, and processes for change (Report ATC-258). Lexington: Massachusetts Institute of Technology.

80.

Thurstone

L. L.

(1938). Primary mental abilities. Chicago, Illinois: University of Chicago Press.

81.

Warren

Tong

(2002). Development of continuous descent approach concepts for noise abatement. Proceedings of the AIAA/IEEE Digital Avionics Systems Conference, 21, 1E3-1–1E3-4. doi:10.1109/DASC.2002.1067906

82.

Wickens

C. D.

Hollands

J. G.

Banbury

Parasuraman

(2012). Engineering psychology and human performance. New York, NY: Pearson.

83.

Wickens

C. D.

Santamaria

Sebok

Sarter

N. B.

(2010). Stages and levels of automation: An integrated meta-analysis. In Proceedings of the Human Factors and Ergonomics Society 54th Annual Meeting (pp. 389–393). doi:10.1177/154193121005400425

84.

Wickens

C. D.

Mavor

Parasuraman

McGee

(1998). The future of air traffic control: Human operators and automation. Washington, DC: National Academy Press.