Abstract
To support new and novel concepts of operations for the anticipated wave of advanced nuclear reactor deployments, human factors and human reliability analysis researchers need to develop advanced simulationbased approaches. The lack of standardized HRA models and human performance data is not new and has outlasted numerous potential solutions. This paper presents a Rancor microworld simulation with computerbased procedures that serve as a framework to classify human behaviors without manual or subjective experimenter coding during scenarios. This method supports a detailed level of analysis at the task level. It is feasible for the collecting large sample sizes required to develop quantitative modelling elements that have historically challenged traditional full-scope simulator study approaches.
Keywords
Introduction
There is growing and widespread interest in advanced nuclear reactors as part of a larger portfolio of electric generators needed to support growing U.S. electricity demands. There is sufficient activity that one might conclude the foreshadowed nuclear renaissance (Boring et al., 2008) is emerging. This is further evidenced by significant and sufficient capital investments to realize advanced reactors that can compete with existing natural gas and renewable assets. Several key characteristics of advanced reactors afford competitive electric generation solutions. Smaller scale 2-50 MW microreactor and less than 300 MW small modular reactor (SMR) designs are promising, since they are flexible to support a diverse suite of use cases. Standard designs that can be manufactured and shipped as a modular unit further reduce the initial capital investment and eliminate many existing one-off construction costs and deployment risks facing traditional large-scale reactors. Lastly, designs that leverage passive safety mechanisms in conjunction with the incorporation of high levels of automation reduce operational costs. These designs are becoming increasingly completive on their own merit but have become even more competitive due to the Inflation Reduction Act of 2022, which provides additional incentives for green technologies, including advanced reactors, to support the U.S. effort to reach net zero carbon emissions by the year 2050 and accelerate to a more aggressive target date of 2030.
Much of the existing research and development focus is on the system design. The designs are reaching sufficient maturity that human factors and human reliability analysis research must begin supporting the development of advanced reactor concept of operations (CONOPS). Due to the diverse designs and niche use cases, many different types of CONOPSs will be pursued, but at the heart of each lies the root question of how these systems can be operated safely and efficiently under oversight from a human operator. Remote operations may entail a central operations center with a single operator overseeing a fleet of geographically distributed reactors. Onsite designs may entail more traditional configurations with a small crew of operators overseeing a plant comprised of several microreactors or SMRs.
Human Performance-Based Conops
There are many prescient questions and issues related to human performance in the new CONOPS, as many of the technologies have no operational experience history in the nuclear domain. Within the human factor’s domain, the technologies and strategies to support supervisory control must be identified, developed, integrated into an overall control room solution, evaluated and refined, and then validated to support specific implementations. Some of these technologies dramatically alter operator tasks away from existing traditional operations. For example, digital displays nest the information under a hierarchy of views in contrast to the analog, always visible, representation found in existing control rooms. Operators will need to access the information to become aware of it, and therefore other new technologies will be included to add operators in viewing necessary information at the appropriate time. Computer-based procedures (CBPs) are one means of achieving this organization, and significant research is needed to understand the most effective method to curate and present CBPs to operators.
Within the related, but distinct, human reliability analysis (HRA) domain, new methods, and techniques for both qualitatively and quantitatively modeling human actions must be developed to profile risks associated with different sociotechnical system solutions to allow vendors and utilities to make informed decisions about which to select for applications. There is also a fascinating and emerging opportunity to begin using computation based dynamic HRA techniques in novel ways outside of risk evaluation for established systems. For example, the dynamic HRA tool HUNTER (Boring et al., 2022) can use a virtual operator model using draft procedures to simulate the interactions with a proposed system design. The simulation uses a virtual human-machine interface (HMI), representing the operators access to system state information and a model of the reactor. The virtual operator and the model can respond dynamically as each simulation run is executed following a Monte Carlo approach to calculate task completion time with uncertainty estimates and human error estimates. The results of this analysis can directly inform the human factors processes as they iteratively develop the HMI and procedures.
The Data Problem
Idaho National Laboratory (INL) conducts extensive research on both the human factors and HRA domains to support the existing U.S. light water reactor fleet of commercial reactors. Recently this research has begun to identify methods to extend into advanced reactors as well, since new technologies developed for existing reactors can be applicable to advanced reactors and vice versus. The work presented here specifically aims to support the ability to collect much needed human performance data. The nuclear industry at large has experienced challenges collecting sufficient data due to difficulties in acquiring simulator access and licensed operators as participants in studies (Ulrich, Boring, & Lew, 2019). Simulator access has been expanded with generic plant models that are within the ability of well-funded university laboratories, but still require substantial expertise to install and maintain. Lastly, the type of researchers needed to conduct studies on human performance typically do not necessarily have nuclear engineering knowledge or operational expertise required to develop scenarios with appropriate operational aspects to maintain a realistic nuclear operations environment and conditions to elicit useful human performance data. Typical full-scope studies require an interdisciplinary team to achieve the overall level of expertise required to perform a scenariobased study. Even with an experienced team, operators are expensive and have demanding schedules occupied with their scheduled shifts to operate the plant and the remaining filled by demanding and extensive training. Ironically, commercial plants record a lot of human performance data; however, these data are proprietary. Consequently, the U.S. nuclear industry has suffered from a dearth of data on human performance, leaving the HRA community struggling to identify effective methods to develop accurate models of human performance.
The simple solution to obtaining human performance data within a nuclear context is a platform that is accessible to human performance researchers and can generate large volumes of data. The Rancor Microworld Simulator, henceforth simply referred to as Rancor, is a simplified nuclear process control simulator developed specifically to serve as a platform to evaluate human performance in a nuclear process control room setting (Ulrich et al., 2017). There are several key characteristics that distinguish Rancor from the traditional simulators used for human factors and HRA research within a nuclear context. The simplicity in the design eliminates much of overly challenging complexity of an actual nuclear control room while maintaining the core tasks and functions that operators perform. This allows the tool to be used by human factors or psychology researchers without extensive nuclear operations expertise to develop meaningful scenarios. Additionally, the simplicity allows for an inexperienced test subject pool outside of the typical licensed or formerly licensed nuclear power plant operators used in traditional full scope simulators. This last capability is perhaps one of the most powerful for this platform, since one of the greatest challenges for human factors researchers in this field is obtaining and funding operators to act as test subjects. This barrier has limited the research to large institutions that can afford operator participants and the compliment of simulator developers and operations experts needed to perform experiments. Since this approach is using students as a surrogate for operators, the key critic is the generalizability of inexperienced participants to that of seasoned nuclear operators.
The initial development and continued refinement of Rancor focused on ensuring the simulation functionally represents the actual nuclear process control tasks in a simplified manner (Ulrich, Boring, & Lew, 2019). Continuing validation research compares students and operators using fullscope and Rancor simulators to establish equivalency and identify discrepancies. The goal is to map the data collected from Rancor to actual operations to understand which constructs directly translate, which simply do not, and others that must be adjusted to generalize appropriately. This validation work has thus far demonstrated promise, with evidence to validate the generalizability has been found in several experiments (Ulrich et al., 2021, Park et al. 2022). The remainder of this paper describes the use of Rancor with an integrated computer-based procedure (CBP) system to support data human performance data collection to illustrate how the unique capabilities of Rancor provide the ability to use CBP as an experimental framework to collect large amounts of task level human performance data to fill the HRA data modelling gaps.
Computer-Based Procedures
Computer-Based Procedure Classifications
Type 1, 2, and 3 CBPs are defined in IEEE-1786 (2022) in which the types differ based on the level of digitization used to support particular types of functionality as can be seen in Table 1. Type 1 procedures recreate the basic paper procedure in digital form. Type 2 procedures also include live process parameter values (i.e., indicators) embedded within the step and highlighting to denote the current state of the parameter upholds the logic of the step or does not as green and red text highlighting respectively. Type 3 procedures extend the functionality of Type 2 with the addition of embedded soft controls such that the participant can perform the prescribed actions without having to use the typical control interface.
Definitions for CBP Types Based on IEEE-1786.
As noted in Boring, Ulrich, and Lew (In press), CBPs can represent automation for both information and control actions, which leads to complexities that different characterization schemes account for better than others. To provide some additional characterization, the IEEE-1786 based conceptualization can be mapped to the NUREG-6433 scheme (O’Hara et al., 2000). NUREG-6433 characterizes CBPs according to levels of automation. The scheme defines several types of procedure functions that fall within four main categories of monitoring and detection, situation assessment, response planning, and response implementation (see Table 2). For the present purposes, more important than levels of automation are two considerations:
CBPs provide a way to classify human activities during control room operations, which provides a framework for collecting human performance data.
CBPs provide a way to log human actions automatically, thereby eliminating manual or subjective notetaking during scenario runs.
Level of Automation of Procedure Functions from NUREG-6634 According to IEEE-1786 CBP Type.
Rancor CBP System
A version of Rancor was developed with selectable CBP system module with the three levels of CBPs defined in IEEE-1786. The CBP system module is integrated into the overall Rancor simulation, which affords data connections to support all the functionality required to represent Type 3 CBPs with embedded indications and soft controls. All procedure configurations allow users to select the appropriate procedure to expand its contents to reveal the first step. Users are required to manually execute all control actions and manually mark each procedure step complete (Figure 1).
Rancor CBP System Examples Illustrating the Three Types of CBPs from IEEE-1786.
CBPs as an Experimental Data Collection Framework
To the best of the authors’ knowledge, prior experiments focus solely on the human factors related constructs when investigating CBPs. Here a novel and central aim is to illustrate the use of a CBP system as a research tool itself to unlock new possibilities for human performance evaluation. CBPs can provide fine-grained analysis of human performance that is otherwise not possible without tedious manual event coding. To illustrate the analysis power afforded by the CBP system in Rancor, it is first necessary to explain how human performance is typically analyzed in full-scope simulator studies, then with Rancor without CBP, and finally with CBP enabled in Rancor.
Most full-scope simulators have limited data recording capabilities as they were originally intended to train operators rated by simulator trainers through observation. In an experimental setting a subject matter expert, often the simulator trainer acts as an expert observer. To eliminate subjectivity the experimenter may provide a scoring rubric and a list of key tasks to rate the operator(s) as they complete the scenario. Within the training realm, the rubric may simply be a binary success or fail metric, while in some experimental methods, such as SCORE, a Likert-scale is used to capture more nuance (Braarud, Eitrheim, & Fernandes, 2015). The observer may also record the time each key activity occurred, both when it was initiated and when it was completed. The time to complete tasks is a powerful and useful metric, though it is typically used only in a gross expert estimation approach in which the length of time is deemed to be appropriate or insufficient. This is typically the extent of the human performance evaluation and, as should be evident, it offers very limited data because the entire scenario is collapsed into a small number of data points associated with a human error. In small minority of experiments, based on this authors’ involvement from prior full-scope studies performing the analysis, some process parameters are used to serve as an additional type of metric for performance. A limited set of process parameters are recorded throughout the scenario so that a running mean-square error (MSE) can be calculated. By referencing the timesteps recorded by the expert observer and, as needed, viewing video recordings to verify recorded timestamps, portions of the MSE for relevant process parameters can be used to correlate with rated performance. This becomes a tedious process as it requires significant manual effort. As such, much of the data recorded during full-scope simulator studies goes unused since the hypothesis can typically be supported by the lower resolution methods. Indeed, the debrief following each scenario in which operators report their experience and area probed by observers to understand any challenges they witnessed is typically sufficient to address much of the basic human factors issues that are often targeted during these experiments.
Rancor was developed to augment full-scope studies and focuses explicitly on methods to efficiently output human performance data and plant process data to support higher resolution analysis than is typical in full-scope studies. Due to its simplistic model, the entire model can be output at each timestep such that an exact history of all process parameters can be used for analysis. Full-scope simulators with 10k to 100k parameters do not have this luxury. Additionally, Rancor records every action an operator takes to the sub-second accuracy. A set of Python-based analysis tools are freely available and provide aggregated data outputs that integrate the process parameters with operator actions to aid analysis. This essentially automates the tedious process required to integrate, time-synchronize, and code the disparate data sources as must be done in a full-scope experiment. It certainly is a step in the right direction, since it is much easier to develop analysis scripts that rely on the action logs to focus on key time windows that may be informative to human performance, but it is still necessary for the experimenter to manually define these. The following section will describe how CBP as implemented in Rancor can be used to support human performance evaluation at a detailed task resolution.
A recent study sampled twenty-seven individuals (19 males, 8 females) across three types of CBP configurations to evaluated performance differences. A normal operations startup and abnormal operations loss-of-feedwater scenario were examined across the three CBP types. Performance between these procedure types can inform human factors efforts to support CBP implementations, but the focus of this paper lies elsewhere and interested readers are encouraged to read details on the experiment itself in an accompanying paper by Hall et al. (in press). The initial data analysis used to prepare the data for the specific analyses documented in Hall et al. required developing a post-analysis Python application to integrate the data files recording plant state parameters at 1 second intervals, participant action logs, and procedure logs. This analysis became the impetus for this paper, as it revealed the potential to systematically collect large amounts of task level human performance data with Rancor. Second, the analysis revealed the data outputs were largely adequate, but with a few simple additional data outputs, much of the post study analysis required to extract the task level details from the various data sources could be automated to further reduce the barriers to evaluating task level data.
The potential power of CBPs as an experimental tool is due to the hierarchical structure of the procedures. This structure provides the framework that supports detailed task level resolution and overall scenario level resolution analysis. The suite of procedures represents a collection of possible goals operators can achieve with the system. Each procedure represents a single high-level goal and is comprised of steps that provide methods to achieve the overall goal. Based on the plant state, the primary procedure path that provides the optimal method to achieve the procedure goal may not be feasible. Therefore, the procedure contains contingencies that guide operators to identify the system state and then choose an appropriate path to maneuver the plant to the desired goal state. As such, a well-designed procedures system is deemed closed loop if it prescribes methods to accommodate all known plant states, though it is rare if impossible to truly achieve a closed loop procedure system. However, through continued use and revision of the procedures, the gamut of plant states gradually become sufficiently covered such that the procedures effectively cover all possible states, as is the case in nuclear process control for the existing U.S. fleet of commercial reactors.
Armed with this hierarchy, the CBP system serves as an automatic trial management system that is able to mark time intervals with explicit goals. In traditional full-scope studies, an experimenter must manually code the time intervals, which is often simply not feasible to support experiments aiming to achieve thousands of samples of individual step time intervals. The CBP system divides the scenario into discrete time intervals surrounding explicit goals. Since the CBP system is integrated into Rancor, it can query the model throughout each procedure step’s time interval. Several key elements are needed to evaluate human performance at this basic task level including initial plant state, actions, and post plant state. First, the initial state of the plant as the participant entered the step must be determined. Then the logic of the procedure step must be evaluated against the initial plant state to determine the appropriate actions needed to satisfy the goal of the step. The participant executes the actions they deem necessary based on their understanding of the step instructions as it relates to their mental model of the plant state. When the step is marked by the participant, the CBP evaluates a postcondition for the step to determine if the desired state has been achieved. A miniature trial of sorts has now been defined for a task level human action for to qualitative and quantitative characterization suitable for HRA methods.
There are many useful metrics that can be used to evaluate human performance within the task level, which will be dictated by the goals of the experiment. Since Rancor was developed to augment existing data collection methods for HRA purposes, metrics with direct implications for HRA model construction will be highlighted. Dynamic HRA methods, such as HUNTER need well-defined timing distributions for the basic tasks. HUNTER uses a GOMS-HRA framework, which consists of a dictionary of basic human actions, i.e,. task level primitives that can be combined to represent more complicated actions that are found within a procedure step (Boring & Rasmussen, 2016). The elapsed time to complete the action from entering the procedure step is the key element needed to populate the various GOMS primitives. Thus far, only manual actions have been considered since the actions are logged and available to analyze against the procedure logs. Information tasks such as reading indicators or making a decision are more challenging, since they do not have a specific element recording a timestamp and value. However, the CBP system does record the time that a step is completed, which serves as more crudely defined timestamp since the act of determining the step was complete and marking it are captured in the elapsed time. Eye tracking provides another method for acquiring accurate timestamps for identifying required plant parameter information. With this information, the CPB framework to identify the relevant time interval can then function in an identical manner to the actions but instead using the eye tracking events to evaluate against.
To evaluate performance in terms of human error probabilities, actions completed within a procedure step time interval can be evaluated. Each action can be evaluated to determine if the correct plant state was achieved, which represents sub-task level analyses. The sequence of tasks can be evaluated to determine if they were executed in the order prescribed by the procedure. Missing steps provide a metric for errors of omission, while extra unprescribed steps provide a metric for errors of commission. Quantitatively, actions that entail a manipulation occurring across a continuum can be evaluated in terms of error from target optimal value similarly to how MSE is calculated for process parameters at the scenario level in full-scope scenarios. Lastly, constructs such as dependency can be evaluated by examining subsequent errors to identify proximity in time and potentially system, i.e. the same system or an independent system.
The work represents merely the initial development of a CBP system and evaluation of a small set of data to serve as a proof of concept and demonstration of using a CBP system as an experimental tool to target detailed task level analysis. Future work is planned to add additional validation logic to automatically generate the data in a synthesized format for timing and human error performance analysis.
Footnotes
Acknowledgements
This work of authorship was prepared as an account of work sponsored by Idaho National Laboratory (under Contract DE-AC07-05ID14517), an agency of the U.S. Government. Neither the U.S. Government, nor any agency thereof, nor any of their employees makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights.
