A Regulatory Perspective: Have We Done Enough on Grasping Automation Failure?

Abstract

This paper responds to Skraaning and Jamieson’s target paper “The Failure to Grasp Automation Failure.” We acknowledge that the target paper made important contributions to automation research in the human factors community. It analyzed automation failure events in complex operational systems in contrast to the vast majority of laboratory research on human-automation interaction. The paper presented a taxonomy of automation failure. The analysis and taxonomy demonstrate the integration of approaches to grasping automation failures from system instrumentation and controls, human factors engineering, and human reliability analysis. We reviewed the regulatory framework related to use of automation in nuclear power plants and examined whether the framework elements adequately address “Failure to Grasp Automation Failure” using the taxonomy in the target paper. Overall, we believe that the target paper could enhance the consideration for potential automation failures in the design and regulatory review process of automation technologies.

Keywords

automation failure human factors engineering human reliability

Introduction

This paper responds to Skraaning and Jamieson’s target paper from the perspectives of regulatory applications. The authors of this paper have worked at the U.S. Nuclear Regulatory Commission (NRC), Office of Nuclear Regulatory Research for nearly two decades, and the content of this paper represents the authors’, not the NRC’s technical opinions. Both have research background in cognitive science and human factors engineering. Both lead the development of regulatory guidance and methods for reviewing human factors engineering (HFE) and human reliability analysis (HRA) in the design of new technologies in nuclear power plant (NPP). The regulatory guidance and methods require technical basis of state-of-art research. Studies of human-automation interaction on complex process control can address the challenges in regulatory activities. The target paper presents the results that can enhance the technical basis for reviewing automation technologies in NPPs. Before discussing the target paper, we provide a landscape of the NRC’s regulatory activities related to the emergent use of automation.

Advanced nuclear reactor technologies present new challenges. High level automation is expected to be prevalent in advanced NPPs and modernization of existing NPP control rooms. The NRC has approved the design of technologies proposing higher levels of automation including the Westinghouse AP1000 [NRC, 2011] and NuScale [NRC, 2020a]. Two AP1000 units are authorized for operation by the NRC [Power, 2023; U.S. Nuclear Regulatory Commission News, 2023, U.S. Nuclear Regulatory Commission News. (NRC, 2022a]; each unit is operated from a nearly fully digital control room. Meanwhile, modernization activities also engender implementations of control room automation. For example, modifications to traditional plants in the U.S. seek to employ Digital Instrumentation and Controls (DI&C) on safety systems. Knowledge about the implications of automation implementations on operator performance will enhance the technical basis for NRC’s regulatory activities.

Modernization efforts involve using DI&C, as well as automating operator manual actions. Novel elements of the design will likely include more advanced automation and, thus, will be targeted for HFE review to determine whether the applicant has reasonably assured the effective integration of automation and operators, and that the design supports safe operations. The NRC has the guidance DI&C-ISG-06 [NRC, 2020b] to review license amendment requests associated with safety-related DI&C equipment modifications. For modifications that may involve HFE considerations, an HFE safety evaluation should be performed in accordance with NUREG-0711 [NRC, 2012], “Human Factors Engineering Program Review Model”; and NUREG-1764 [NRC, 2007] and “Guidance for the Review of Changes to Human Actions.” More recently, the NRC staff developed the draft Guidance “Development of Scalable Human Factors Engineering Review Plans (DRO-ISG-2023-03) [NRC, 2022b], under the “Risk-Informed, Technology-Inclusive Regulatory Framework” [U.S. Nuclear Regulatory Commission. NRC, 2023b] for HFE review of advanced reactors. The guidance provides a risk-informed and performance-based process to identify the most safety-significant systems, systems with likely human factors challenges, and novel elements of the design.

Research has been conducted on human-automation systems for NPP control rooms. For example, the Halden Human Technology Organization (Halden HTO) at the Institute for Energy Technology (IFE) consolidated its two decades long research results studying human-automation interaction in NPP simulators [Skraaning et al., 2020]. Several organizations have reported human performance studies of computerized procedures with automation functionality embedded [Claire, Hildebrandt, McDonald, R., & Hughes, 2017]. Operational experience review has documented many DI&C and automation events in NPPs. Most DI&C events involve issues in HFE considerations. There is also research regarding automated vehicles related to the mitigation of attribution error in the design of automated and autonomous nuclear power plants [Hancock et al., 2023]. The target paper by Skraaning and Jamieson, “The Failure to Grasp Automation Failure [Skraaning & Jamieson, 2023],” reviewed recent aviation accidents involving automation failures and proposed an initial taxonomy of automation failure and automation-related human performance challenges. Xing and Green [Xing & Green, 2023] reported that deficiencies of human-automation integration led to automation failure events in several ways: (i) The automation system worked as expected but the deficiencies in human-automation interaction led to human failures; (ii) The automation failures led to, or aggravated, human-automation integration deficiencies and then led to human failing to identify or recover from automation failures; (iii) The DI&C element had unusual behaviors or deviated from operators’ understanding, thus leading to or aggravating human-automation integration deficiencies and then leading to human failures. Taken together, these different sources of information can provide an understanding of automation reliability and how that reliability impacts the operator’s use of automation.

In 2022, an interdisciplinary team of NRC staff working in DI&C, HFE, and risk analysis systematically evaluated the findings from investigative reports of BOEING 737 crashes [NRC, 2022c]. The team examined the recommendations in these investigative reports for their potential implementation in the NRC’s DI&C regulatory process. The team recommended four areas of focus to enhance DI&C licensing and regulatory oversight, two of which address HFE in DI&C reviews:

- The NRC staff should continue to improve integration and communication among DI&C technical reviews, HFE reviews, and subsequent inspection oversight for new or significantly different applications from conception to installation.

- The NRC staff should develop guidance for assessing systems engineering approaches for the DI&C design and human factors life-cycle evaluation, which are important for ensuring that approved DI&C designs are appropriately integrated to maintain safety functionality.

These underscore the importance of effective communication and integration among HFE and DI&C for the review of advanced reactor technologies.

General Comments on the Target Paper from Regulatory Application Perspectives

We assert that the target paper made several important contributions to human-automation research:

(i) The authors analysed automation failure events in complex operational systems in contrast to most laboratory research on human-automation interaction. Previous studies and our own experience of developing human factors engineering guidance demonstrated that results from human-automation-interaction laboratory experiments in simple multi-task contexts generally do not predict human performance with automation in complex control process systems [NRC, 2022c].

(ii) The authors analysed automation failure accidents with a human performance framework beyond the traditional human information processing models. The human performance framework considers the broad context of complex cognitive process tasks, while human information processing models typically represent a “slice” of the overall cognitive processes involved in complex operational tasks.

(iii) The analysis focuses on the causes and mechanisms of automation failure; the taxonomy of automation failure expands the themes of human-automation research that have been focused on making automation work better. It is prevalent that laboratory experiments study the effects of human-automation interaction characteristics such as level or type of automation on measures such as workload, situational awareness, and level of trust. However, there has been little evidence that those characteristics and measures can predict human performance, in particular, the reliability of human’s ability to “grasp automation failure.”

(iv) The taxonomy incorporates findings on automation systems, design logic and components, DI&C, human-automation integration, and human and organizational factors. It calls to attention that grasping automation failure needs integrated approaches from all these aspects. This echoes the recommendations made by the NRC team on integration of DI&C, HFE, and risk analysis.

(v) The taxonomy could inspire experimental research scenarios that are more industry relevant. It could serve as a roadmap for the design of future domain-specific experimental research aimed at how different automation features influence performance in complex operational contexts. System designers could use the framework to anticipate and identify systemic automation failures. Likewise, after proper evaluation, regulators may consider using a similar taxonomy as a checklist to evaluate if a particular design is vulnerable to systemic automation failures.

Potential to Enhance Regulatory Guidance for Reviewing New NPP Technologies

A Framework for Integrating DI&C and Human Factor Engineering

Regardless of the vast landscape of advanced technologies, NPP operations share the critical functional aspects in common:

• Operators work within complex control systems;

• Operators are in control of the systems, although the systems can run at high or full automation modes;

• Operators’ tasks involve monitoring, situational assessment, decision-making/planning, manipulation/control, and teamwork; Use of higher levels of automation may result in more monitoring tasks for the human operator versus manipulation and control types of tasks.

• Operator responses are procedure-based.

Xing and Green (2023) generalize the NRC’s regulatory and licensing activities in DI&C, HFE, and HRA into a framework depicted in Figure 1. This framework represents how human-automation integration works: (i) The DI&C elements achieve functions of automation systems; behaviors of DI&C elements impact elements of human-automation integration; (ii) Human-automation integration should ensure that the automation functions do not cause human errors and failures of DI&C elements do not propagate to human failures, (iii) The human cognition system should ensure that human operators are capable of identifying and recovering from automation failures. In the diagram, the box on the left represents DI&C elements that constitute an automation system. The box in the middle of the diagram represents HFE elements that support human-automation interaction. The box on the right represents the elements of human cognitive task performance.

Figure 1.

A framework for integrating DI&C systems and HFE.

The taxonomy of the target paper is organized by automation-induced human performance challenges in four columns (1) Elementary Automation Failures, (2) Systematic Automation Failures, (3) Human Automation Interaction Breakdowns, and (4) Negative Human Performance Outcomes that may occur in the presence of automation and, may be the result of a lack of adherence to human factors design principles. Next, we discuss how the target paper taxonomy may enhance the elements in the framework presented in Figure 1.

DI&C Systems

The NRC staff conducts DI&C design reviews to ensure DI&C system safety for NPP operations. The DI&C design review identifies potential design hazards and analyzes system or component failure modes. The first column of taxonomy contains Elementary Automation Failures, such as “Automatic functions are missing or lost,” and “Loss of power supply to automation,” corresponding to design hazards of automation systems. The listed hazards can enhance existing hazard identification methods that are not specifically developed for automation systems.

The second column of the taxonomy contains Systematic Automation Failures. Examples are “Automation works as intended but operates outside the design basis,” and “Automatic systems works in parallel but compromise each other.” This list can enrich our understanding of potential failure modes in DI&C elements of an automation system and help identifying potential automation failures that induce human performance challenges.

Human-Automation Integration

The NRC staff review human-automation integration using guidance contained in Human Factors Engineering Review Model documented in NUREG-0711 [NRC, 2012]. The HFE model includes twelve elements, some of which are shown in the middle box of Figure 1. The element Functional Requirements Analysis and Function Allocation is most relevant to automation design. NUREG-0711 states “The purpose of this element is to verify that the applicant defined those functions that must be carried out to satisfy the plant’s safety goals and that the assignment of responsibilities for those functions (function allocation) to personnel and automation in a way that takes advantage of human strengths and avoids human limitations.” The currently proposed 10 CFR Part 53 rule § 53.440 (n) (4) states, “A functional requirements analysis and function allocation must be used to ensure that plant design features address how safety functions and functional safety criteria are satisfied, and how the safety functions will be assigned to appropriate combinations of human action, automation, active safety features, passive safety features, or inherent safety characteristics.” [U.S. Nuclear Regulatory Commission. NRC, 2023b]

The third column of the taxonomy presents a list of Human-Automation Interaction Breakdowns. Examples are “Automation provides misleading support to operators,” and “Critical operator actions are unsuitably blocked by automation.” Essentially, all the listed breakdowns are deficiencies in functional requirement analysis and functional allocation. The HFE review can benefit by incorporating the “breakdowns” as a check list for asking questions.

Similarly, the taxonomy can also help HFE review the element on Human Factors Verification and Validation, which states that “Verification and validation (V&V) evaluations comprehensively determine that the final HFE design conforms to accepted design principles and enables personnel to successfully and safely perform their tasks to achieve operational goals.” When present in a design, the listed breakdowns represent design deficiencies that should be captured in V&V. The current HFE process described in NUREG-0711 focuses on ensuring that state-of-the-art human factors principles are incorporated into the design and implementation which is verified during V&V. In contrast, grasping automation failures requires risk-focused considerations of failure modes and failure causes.

Human Reliability Analysis

The fourth column of the taxonomy lists examples of Human and Organizational Slips/Misconceptions, such as “Faulty operator programming of automation” and “Inadvertent activation/deactivation of automation” which are not specifically addressed in our HFE review process. They are, however, addressed in HRA. The NRC’s new HRA methodology, the Integrated Human Event Analysis System (IDHEAS) [Xing et al., 2019, 2022], uses 20 performance influencing factors (PIFs), as shown in Table 1, to model human performance. Each PIF is represented by a set of attributes, which describe the ways the PIF challenges human performance and increases the likelihood of human errors. We ascertained that all the listed slips/misconceptions are included in IDHEAS PIF attributes. Since the target paper indicates that the list is an initial subset of a large pool, we recommend that future work considers using IDHEAS PIF structure to organize human and organizations slips/misconceptions.

Table 1.

Performance Influencing Factors in IDHEAS Method.

Environment and situation	System	Personnel	Task
• Work location accessibility and habitability	• System and I&C transparency to personnel	• Staffing	• Information availability and reliability
• Workplace visibility	• Human-system interfaces	• Procedures, guidelines, and instructions	• Scenario familiarity
• Noise in workplace and communication pathways	• Equipment and tools	• Training	• Multi-tasking, interruption and distraction
• Cold/heat/humidity		• Teamwork and organizational factors	• Task complexity
• Resistance to physical movement		• Work processes	• Mental fatigue
			• Time pressure and stress
			• Physical demands

Finally, we see that one piece missing from the taxonomy is the “so what,” a description of human failures due to the challenges or misconceptions. We recommend the target paper authors consider incorporating IDHEAS cognitive failure modes to represent human failures in using automation. The cognitive failure modes represent failures of macrocognitive functions, which are the basic cognitive elements to achieve complex operational tasks. The cognitive failure modes are human-centered, thus, they can be used to model human failures in any automation systems.

IDHEAS cognitive failure modes consist of the failures of the following five macrocognitive functions:

• Detection (D) is noticing cues or gathering information in the work environment.

• Understanding (U) is the integration of pieces of information with a person’s mental model to make sense of the scenario or situation.

• Decision-making (DM) includes selecting strategies, planning, adapting plans, evaluating options, and making judgments on qualitative information or quantitative parameters.

• Action execution (E) is the implementation of the decision or plan to change some physical component or system.

• Interteam coordination (T) focuses on how various teams interact and collaborate on an action.

Each macrocognitive function is achieved through a set of basic cognitive processors. IDHEAS uses the failure of the processors as detailed cognitive failure modes, as shown in Table 2. These detailed cognitive failure modes can specifically model types of human failures in human-automation integration.

Table 2.

IDHEAS Detailed Cognitive Failure Modes.

Failure of macrocognitive function	Detailed cognitive failure modes
Failure of detection (D)	D1. Fail to establish the correct mental model or to initiate detection
	D2. Fail to select, identify, or attend to sources of information
	D3. Incorrectly perceive or classify information
	D4. Fail to verify perceived information
	D5. Fail to retain, record, or communicate the acquired information
Failure of understanding (U)	U1. Fail to assess/select data
	U2. Fail to select/adapt/develop the mental model
	U3. Fail to integrate data with the mental model
	U4. Fail to verify and revise the outcome of understanding
	U5. Fail to export the outcome
Failure of decision-making (DM)	DM1. Fail to adapt the infrastructure of decision-making
	DM2. Fail to manage the goals and decision criteria
	DM3. Fail to acquire and select data for decision-making
	DM4. Fail to make decision (judgment, strategies, plans)
	DM5. Fail to simulate or evaluate the decision or plan
	DM6. Fail to communicate and authorize the decision
Failure of action execution (E)	E1. Fail to assess action plan and criteria
	E2. Fail to develop or modify action scripts
	E3. Fail to prepare or adapt infrastructure for action implementation
	E4. Fail to implement action scripts
	E5. Fail to verify and adjust execution outcomes
Failure of interteam coordination (T)	T1. Fail to establish or adapt teamwork infrastructure
	T2. Fail to manage information
	T3. Fail to maintain shared situational awareness
	T4. Fail to manage resources
	T5. Fail to plan interteam collaborative activities
	T6. Fail to implement decisions and commands
	T7. Fail to verify, modify, and control the implementation

Concluding Remarks

The target paper, “The Failure to Grasp Automation Failure” makes several important contributions towards moving the field of automation research forward. The present commentary is from a regulatory perspective and particularly focused on the impact for the nuclear domain. The target paper addressed a much richer set of operational challenges through the analysis of automation failure events than can be accomplished in most laboratory experiments concerning human-automation interaction. The framework used builds upon traditional human information processing models, resulting in a more integrated approach. By focusing on the causes and mechanisms of automation failure, the initial taxonomy proposed not only extends the human-automation research for improved automation, but the taxonomy itself represents a potentially successful integration of the interdisciplinary approaches for identifying automation failures including DI&C, HFE, and HRA. The introduction of this taxonomy is a clear step towards addressing the challenge of integrating these interdisciplinary intersections by providing a common language. We suggest several areas where the target paper authors may consider moving forward in the development of the taxonomy, such as incorporation of cognitive failure modes. We also propose several use cases for the taxonomy. The taxonomy may be used to guide experimental research aimed at how different automation features affect performance in complex operational contexts. System designers may use the framework to anticipate and identify systemic automation failures. Likewise, after proper evaluation, regulators may use a similar taxonomy as a checklist to evaluate if a particular design is vulnerable to systemic automation failures. Overall, the analysis and proposed taxonomy could benefit the human-automation research, particularly through the focused scope on complex operational contexts like the nuclear domain.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Niav Hughes Green

Jing Xing is a Sr Human Performance Engineer at the Office of Nuclear Regulatory Research, U.S. Nuclear Regulatory Commission (NRC). She joined the NRC in 2008 and has been the technical lead for developing Human Reliability Analysis methods. She holds a Bachelor degree in Electronic Engineering, a MS degree in Biophysics and computer vision, and a Ph.D in Cognitive Neuroscience. Prior to joining the NRC, she worked on human factors engineering at NASA and U.S. Federal Aviation Administration (FAA).

Niav Hughes Green is a Human Factors Psychologist at the Office of Nuclear Regulatory Research, U.S. Nuclear Regulatory Commission (NRC). She joined the NRC in 2006 and serves as technical lead for experimental research activities supporting guidance development to facilitate NRC licensing reviews of nuclear power plants. She holds a Bachelor of Science in Psychology from the University of Maryland and earned her Masters in Psychology and Ph.D. in Applied and Experimental Psychology from the Catholic University of America. Her graduate research focused on using a variety of human performance measurement techniques, including eye tracking, to assess the effects of cognitive mechanisms on the performance of visual search and navigation activities as well as basic nuclear tasks performed in the nuclear domain.

References

Claire

Hildebrandt

McDonald

Hughes

(2017). Operator response to failures of a computerised procedure system: Results from a training simulator study, halden working report HWR-1198. OECD Halden Reactor Project.

Hancock

P. A.

Lee

J. D.

Senders

J. W.

(2023). Attribution errors by people and intelligent machines. Human Factors, 65(7), 1293–1305. https://doi.org/10.1177/00187208211036323

Power

Georgia

(2023). Vogtle Unit 3 Goes into Operation Delivering on Georgia Power’s Commitment to Safe, Clean, Reliable & Affordable Energy. (2023, July). georgiapower.com. Retrieved January 29, 2024, from https://www.georgiapower.com/content/dam/georgia-power/pdfs/company-pdfs/vogtleonepager_073023.pdf

Skraaning

Jamieson

G. A.

(2023). The failure to grasp automation failure. Journal of Cognitive Engineering and Decision Making. https://doi.org/10.1177/15553434231189375

Skraaning

Jamieson

G. A.

Armando

Guanoluisa

(2020). Future challenges in human-automation interaction: Technology trends and operational experiences from other industries. Halden Working Report HWR-1308. OECD Halden Reactor Project.

U.S. Nuclear Regulatory Commission . (NRC, 2022b). “Predecisional – documents to support Part 53 - development of scalable human factors engineering review plans: Draft interim staff guidance”. DRO-ISG-2023-03. 10/18-19/2022 ACRS Public Meeting, ML22272A051. U.S. Nuclear Regulatory Commission.

U.S. Nuclear Regulatory Commission News . (NRC, 2023a). NRC authorizes vogtle unit 4 fuel loading and operation”. U.S. Nuclear Regulatory Commission.

U.S. Nuclear Regulatory Commission . (NRC, 2007). “Guidance for the review of changes to human actions”, NUREG-1764, revision 1. US Nuclear Regulatory Commission.

U.S. Nuclear Regulatory Commission . (NRC, 2011). “Final safety evaluation report related to certification of the AP1000 standard design” NUREG-1793, supplement 2. U.S. Nuclear Regulatory Commission.

10.

U.S. Nuclear Regulatory Commission . (NRC, 2012), “Human factors engineering Program review model,” NUREG-0711, rev. 3. US Nuclear Regulatory Commission.

11.

U.S. Nuclear Regulatory Commission , (NRC, 2020a). “Standard design approval for the nuscale power plant based on the nuscale standard plant design certification application”, ML20247J564. U.S. Nuclear Regulatory Commission.

12.

U.S. Nuclear Regulatory Commission . (NRC 2020b). “Digital instrumentation and controls licensing process interim staff guidance”, DI&C-ISG-06, revision 2. U.S. Nuclear Regulatory Commission.

13.

U.S. Nuclear Regulatory Commission . (NRC, 2022c). “Boeing 737 crashes: Lessons learned for NRC digital instrumentation and controls evaluation process”. U.S. Nuclear Regulatory Commission.

14.

U.S. Nuclear Regulatory Commission News. (NRC , 2022a). “NRC authorizes vogtle unit 3 fuel loading and operation”. ML22215A210. U.S. Nuclear Regulatory Commission.

15.

U.S. Nuclear Regulatory Commission. (NRC , 2023b). “Proposed rule: Risk-informed, technology-inclusive regulatory framework for advanced reactors (RIN 3150-AK31)”. SECY-23-0021: ML21162A093. U.S. Nuclear Regulatory Commission.

16.

Xing

Chang

Y. J.

DeJesus

(2019). The general methodology of an integrated human event analysis system (IDHEAS-G). U.S. Nuclear Regulatory Commission, NUREG-2198. (ADAMS Accession No. ML19235A161).

17.

Xing

Chang

Y. J.

DeJesus

(2022). Integrated human event analysis system for event and condition assessment (IDHEAS-ECA). U.S. Nuclear Regulatory Commission, NUREG-2256. (ADAMS Accession Number: ML22165A282).

18.

Xing

Green

N. H.

(2023). Failure modes in human-automation integration. Proceedings of the enlarged halden Program review group (EHPRG) meeting. OECD Halden Human-Technology-Organization (MTO) Project.