Understanding Automation Failure

Small Sample Size

The study involved only eight teams of operators each performing in four automation conditions, providing only eight measures of performance per condition. This is an extremely small number to detect differences between conditions. While Skraaning and Jamieson argue that somehow the power of the test is not important, this fails to address a key limitation of their study: perhaps eight measures of an event in each condition is simply insufficient for showing statistical significance between conditions? Another study cited by Skraaning and Jamieson as contradicting the lumberjack effect is Calhoun et al. (2009). Yet that study only had six participants, an issue that its authors credited for the performance measure failing to reach statistical significance even though the trends followed the expected direction of higher LOA yielding slower performance when the automation is unreliable (p = .16). Similarly, Cummings and Mitchell (2007) had only three participants in each LOA condition, yet still found that high automation (management by exception) was worse for performance (p = .07), particularly when there were a high number of replanning events. Skraaning and Jamison do not provide any trend data.

Limits of Testing

The Jamieson and Skraaning study involved eight teams (of three people each) performing in four scenarios. All scenarios involved a fault in which one valve remained open, varying only in terms of which valve and why it failed to open. After the first fault, it is highly unlikely the operators were surprised or not expecting a fault in the remaining scenarios. The authors do not report on any testing of order effects which could easily have contributed to the lack of statistically significant difference between conditions. With four LOA conditions and only eight teams, attempts at counter-balancing for order effects will have been only partial at best.

Poor SA Measure

Skraaning and Jamieson relied on IPAQ as a measure of SA. Yet this is an unvalidated measure that has not been used in any published literature. Their only citations for it are two unpublished internal documents from an organization that no longer exists and are not publicly available. This measure, administered at the end of the scenario, asked for operators to rate the importance of eight process parameters. As a key limitation, by assessing this information only at the end of the scenario, it is likely to only capture operators’ understanding at that point, well after participants may have figured out what was going on in the scenario (Endsley, 2021). Thus, it does not necessarily capture people’s SA at the time of the anomalous event or any delays in ascertaining what was really happening that could lead to negative outcomes in many domains. The measure therefore may suffer from significant hindsight bias. Further, a high degree of overlap is shown in the 95% confidence intervals on this measure between the four LOA conditions casting some doubt on the strength of their findings of SA differences between conditions.

Other Relevant Factors

There are several other factors that also could have contributed the lack of an LOA performance effect in the Jamieson and Skraaning study. The negative effects of high LOAs have been shown to be worse for continuous control tasks and those involving advanced queueing of tasks (Endsley & Kaber, 1999; Kaber et al., 2000), and when other tasks are present that compete for people’s attention (Kaber & Endsley, 2004). The Jamieson and Skraaning study, however, involved a checklist that operators were working through with different levels of automation. This type of task, particularly with no competing tasks, is less likely to fall prey to OOTL problems.

Secondly, it has been shown that display transparency can reduce or eliminate OOTL performance decrements (Bagheri & Jamieson, 2004; Bass et al., 2013; Bean et al., 2011; Dzindolet et al., 2002; Mercado et al., 2016; Selkowitz et al., 2017; Seppelt & Lee, 2007; Stowers et al., 2017) as well as SA problems (Boyce et al., 2015; Chen et al., 2014; Schmitt et al., 2018; Selkowitz, et al., 2017). It is entirely possible that Jamieson and Skraaning did a good job in providing automation transparency with their automated checklist display design. While it cannot be stated categorically that either of these factors contributed to the lack of significant differences in failure detection performance between LOAs in their study, neither can they be ruled out with enough certainty to call the lumberjack effect into question.

More than OOTL

Much discussion about the lumberjack effect is built around studies of OOTL. However, it is worth pointing out that automation-related performance problems result from not just OOTL, but also automation confusion and automation interaction errors. Wicken’s description of the lumberjack effect is not limited to OOTL; rather, catastrophic events can happen due to automation problems of varying kinds.

Probabilistic versus Deterministic Models

Much of Skraaning and Jamieson’s argument lies in the presumption that the “burden of proof for a prediction of human performance effects rests on those making the prediction,” and that even a few studies that have findings that contradict that model are enough to cast doubt on it. This viewpoint would only hold true if the lumberjack effect were deterministic rather than probabilistic. In fact, it is plainly obvious that automation in many cases works well much of the time and that people are often able to detect and correct for its shortcomings, or it would never be implemented at all. Rather, the lumberjack effect merely states that the probability of these OOTL events increases with automation. The more detailed research literature provides information on the many factors that affect that probability (e.g., LOA, competing tasks, time on task, task implementation, display transparency, and training) (Endsley, 2017b).

As for the burden of proof, not only does the wide body of research literature on this matter show a preponderance of evidence for the effect of LOA on OOTL from laboratory studies (Onnasch, et al., 2014) but also a large body of real-world examples (e.g. Gawron, 2019) shows that these problems are not just the result of the artificialities of the laboratory, but are a significant problem in complex settings with experienced operators.

SA and LOA

Some confusion seems to exist on the effects of higher LOAs on SA. The research base includes studies that show increases in SA at higher LOAs (Endsley & Kaber, 1999; Ma & Kaber, 2005); however, most studies show decreases in SA at higher LOAs (Endsley & Kiris, 1995; Franz et al., 2015; Jipp & Ackerman, 2016; Kaber, et al., 2000; Manzey et al., 2012; Sethumadhavan, 2009). Understanding this apparent discrepancy relies on looking more closely at the data.

High LOAs can have the advantage of freeing up cognitive resources so that people have more time to take-in information, at least initially. But when secondary tasks are introduced, this potential benefit disappears (Kaber & Endsley, 2004; Ma & Kaber, 2005; Weaver & DeLucia, 2022). Engagement decreases and attention to other tasks increases over time, lowering SA on automation-related information (Carsten et al., 2012) and increasing complacency (Wickens et al., 2015). This is more likely with automation that is more reliable which acts to increase trust (Wickens & Dixon, 2007). SA essentially can become more variable under increased automation, increasing with effort, but decreasing during periods of distraction and higher workload (Endsley, 2017a). Importantly, studies have shown a significant correlation between SA and manual take-over time following an automation problem (Clark et al., 2017; Sethumadhavan, 2009), demonstrating the importance of SA at the time that an event occurs for people’s ability to recover from the automation failure.

Thus, if SA actually did increase in the higher LOA conditions in Skraaning and Jamieson’s studies (which is questionable given their measure of SA), it would not be particularly damning with respect to the lumberjack effect. Rather it is somewhat consistent for a study in which participants were not subject to dual tasking.

Task Complexity

Jamieson and Skraaning’s central rationale for not finding an OOTL effect centers on the fact that their study employed experienced operators in a high-fidelity simulation environment, claiming the lumberjack effect does not extend to complex environments. However, there is not much evidence to back up this claim. Two of the other studies cited as additional evidence by Jamieson and Skraaning also had a very small number of inexperienced participants, with their trends generally supporting the lumberjack effect (Calhoun, et al., 2009; Cummings & Mitchell, 2007). The third study employed 20 experienced air traffic controllers using a conflict detection aid (Metzger & Parasuraman, 2005). It also supports the lumberjack effect showing improved performance with the aid in normal situations (p = .01) and a trend toward better manual performance compared to with the automated aid when it was unreliable (p = .14). The authors felt this finding had practical significance in safety critical environments, particularly since order effects were present and the less reliable condition was always presented last. SA was not measured in that study.

While many experimental research studies that have been done in this area employed either low- or medium-fidelity simulations, the large number of accidents and incidents from highly complex settings in the real world with experienced operators discussed at the beginning of this paper shows that the OOTL event occurs outside of laboratory settings as well.