Abstract
Drilling incidents have emphasized that offshore drillers require a high level of cognitive skills, including situation awareness and decision making, to maintain safe and efficient well control. Although a number of tools for supporting operators’ cognition are available in other high-risk industries, there is no specific tool for drilling. We developed a prototype monitoring task simulating drilling scenarios, Drillers’ Situation Awareness Task, with drilling experts and piloted with 14 drilling personnel. Preliminary results suggest that it is viable as a tool for examining drillers’ cognition and has the potential for training and formatively assessing cognitive skills in drilling.
Keywords
A prototype monitoring task simulation specifically for drillers is designed to measure situational awareness and decision making–and improve safety.
Four minutes later, gas that had escaped from the well ignited, causing the first of two explosions on the rig (see Figure 1). This incident resulted in the death of 11 members of the crew, including both the driller and assistant driller; the rig’s destruction; and the worst oil spill in U.S. history.
Deepwater Horizon drilling rig engulfed in flames as fireboats pour water onto the rig. Courtesy of U.S. Coast Guard.
In the wake of the disaster, investigation reports identified the drill crew’s situation awareness (SA) as contributing to the blowout (National Commission on the Deepwater Horizon Oil Spill and Offshore Drilling, 2011a, 2011b). SA is the state of knowing what is going on in the situation and using that understanding to anticipate how it will develop (Endsley, 1995a, 1995b). For example, Roberts, Flin, and Cleland (2015b) identified problems such as failure to monitor the well, misunderstanding of the well state, and strong erroneous expectations that the well was stable. The disaster clearly illustrated that offshore oilfield drillers require high-level SA, particularly during complex tasks such as well control. (Well control refers to using a hydrostatic column of drilling fluid to maintain control of the highly pressurized hydrocarbons and other fluids within the well bore; Roberts, Flin, & Cleland, 2016.)
The skills associated with SA are increasingly important given advancements in drilling technology that have resulted in more reliance on cognition. Recent research has identified the key cognitive skills that expert drillers use to develop and maintain SA of the well state and surrounding environment (Roberts, Flin, & Cleland, 2015a).
Further research into SA in drilling has been identified as vital for offshore safety (e.g., Ocean Energy Safety Institute, 2016). Measurement and training techniques for operator cognition associated with SA and decision making (e.g., in aviation; Endsley, 1990; Hauss & Eyferth, 2003) have had limited application in the oil and gas industry despite their potential value.
Our focus for this study is to adapt an existing monitoring task to the new context of offshore drillers’ cognition, with the aim of supporting and training these vital skills. It is hoped that this effort will have ramifications for safety and performance in not only drilling but similar high-risk, high-reliability domains that involve monitoring.
What Do Drillers Do?
Offshore oilfield drillers are responsible for the hazardous task of drilling into the sea bed and constructing a well bore to gain access to and extract hydrocarbons (e.g., oil and gas). As hydrocarbons are highly pressurized with high temperatures, they need to be controlled using a hydrostatic column of drilling fluids. In conjunction with the drill crew − including the assistant driller, roughnecks, and tool pusher (supervisor) − the driller controls the majority of the equipment and, subsequently, the well from the drill cabin.
On newer-generation drilling rigs and cyber-rigs, the driller is required to monitor up to eight LCD screens displaying information from equipment hundreds of feet below the drill deck and multiple closed-circuit television video feeds, navigating between different control panels. Operators also must keep an eye out the window onto the drill floor for the safety of the crew working with powerful and heavy equipment, as shown in Figure 2. Thus, the driller needs to interact with increasingly complex technology, which requires high-level cognitive skills, principally associated with SA, to monitor and interpret the significance of the information coming from the well and surrounding environment.
Driller monitors the drilling screens with the drill floor shown in the background. Courtesy of Maersk Drilling.
Although the drilling industry has recognized the complexity of the drillers’ task and the value of using low-fidelity simulations to aid training (Letbetter, 1975), only recently have higher-fidelity simulators been introduced. Material on cognitive skills is being incorporated into simulation training via teaching nontechnical skills (International Association of Oil and Gas Producers [IOGP], 2014a, 2014b) and team training methods (e.g., tactical decision games; Crichton, Henderson, & Thorogood, 2004). However, to our knowledge, there are no simulation training or measurement tools specifically designed for drillers’ cognitive skills associated with SA.
Why do drillers need SA?
Maintaining SA is critical for safe and effective performance in the drilling industry, yet research is limited with regard to understanding the underlying cognitive skills. Problems with drillers’ SA have been identified, such as difficulties with concentration and interpreting information (e.g., Sneddon, Mearns, & Flin, 2006).
Roberts et al. (2015b) identified the key cognitive components required by drillers to develop and maintain SA, including
attending to the drilling screens and recognizing a pattern from available cues;
comprehending the significance of cues to the situation using mental models, expectations, and experience; and
projecting how the situation may develop.
We used interview and observation data to produce the drillers’ SA (DSA) model (see Figure 3), based on Endsley’s (1995a, 1995b) model of SA, in which situation awareness is described as a cognitive product of three hierarchical levels: cue recognition and perception (Level 1), comprehension (Level 2), and prediction (Level 3).
The driller’s situation awareness (SA) model illustrating the key cognitive components and associated skills required for developing and maintaining SA for safe well control.
The model has been subsequently used to examine drillers’ SA in reports of the Deepwater Horizon blowout (Roberts et al., 2015a) and in a cognitive task analysis of kick detection. (Kick detection refers to monitoring changes in readings from the well that may indicate that the pressure within the well may exceed the downward hydrostatic pressure, potentially resulting in a well control situation; Roberts et al., 2016.) We aimed to use the data and DSA model to inform the design of the simulation-based measurement task.
How to measure SA?
A number of methods have been developed to examine expert operator SA (e.g., Loft, Morrell, & Huff, 2013), including real-time probes, as they offer a direct and relatively objective measurement in which the participant responds to questions during a simulation task (e.g., situation present assessment method; Durso, Hackworth, Truitt, Crutchfield, & Manning, 1998). An alternative but similar computer-based method, Expert Intensive Skills Evaluation (EXPERTise; Loveday, Wiggins, Searle, Festa, & Schell, 2013), was developed to examine aspects of SA expertise, including cue utilization and pattern recognition (e.g., power control operators; Loveday, Wiggins, Harris, O’Hare, & Smith, 2013).
Similar to real-time probes, with EXPERTise, participants monitor a domain-specific display (e.g., intensive care unit screen) and respond when they recognize key cues during different tasks. In particular, two tasks appeared to be suitable to adapt to examine drillers’ SA and decision making. One task measured the ability to extract diagnostic cues from the functional work environment (by clicking on an abnormal indicator) and another task measured the ability (accuracy) to discriminate the usefulness of available information via decision making.
Aim
Our aim was to develop a simulation-based monitoring task that examines drillers’ cognitive skills, including cue recognition, comprehension and anticipation, and decision making. First, we developed the prototype monitoring task with subject matter experts (Study 1) and then piloted it with a sample of drilling personnel to test its viability for examining drillers’ cognitive skills (Study 2).
Study 1: Task Development
The authors of EXPERTise generously gave us access to an early version (1.0) of their program to determine if it could be adapted. However, this program proved impractical, so we developed a new program using the programming platform Delphi 6 (Borland Software Corporation, 2009) with an experienced programmer (JU). The interface was based on generic drilling-parameter screen images supplied by the sponsoring company.
Scenarios
We developed four scenarios plus a practice trial in conjunction with two drilling experts, both of whom had more than 20 years’ experience in drilling and were now drilling instructors. In addition, we employed the sponsor’s simulation training well-control scenarios and well-control incident reports, technical well-control manuals, and a cognitive task analysis (Roberts et al., 2016).
SA requirements (Endsley, 2016; what the participants would need to know) for key points of each scenario were identified with the drilling instructors, including benchmarking data against which to examine the participants’ performance (e.g., minimum cues that needed to be recognized to take the correct decision). The four scenarios were drilling into a hard formation, drilling into a transition zone, drilling into a porous formation, and drilling into a hard formation while encountering equipment problems. Additional details on the scenarios, as well as example SA requirements, are in the Appendix.
Task
Participants monitored the simulated drilling parameter screen (see Figure 4). In drilling, cues are predominantly changes in the drilling parameters (e.g., increase in flow rate). Each line represents a drilling parameter or variable, with the dips and peaks representing changes in that parameter. The reader will notice that these changes often occur in patterns across the parameters (i.e., one parameter affects another).
Screenshot of the completed Scenario 4 running on the Drillers’ Situation Awareness Task program as displayed to particpants.
To indicate that they had recognized a cue, the participants clicked on the cue’s location on the screen. This action was a measure of cue recognition in the form of accuracy and latency (time taken to recognize cue since onset). Indicating recognition of a cue prompts a probe question with a multiple-choice response. A generic question, based on what the supervisor would typically ask the driller, was used: “What is the current situation?” Four response options were presented, typically consisting of incorrect, partly correct, and correct levels of understanding and the fourth indicating a higher level of understanding in the form of anticipation, depending on the scenario. Response scores consisted of completely wrong = 0, recognizing a cue = 1 to 3 (i.e., minimal awareness), comprehension of the situation = 4 to 6, and anticipating how the well state may progress = 7 to 9. The score within each category (e.g., 7–9) depended on the predetermined scoring of the particular option included in the multiple-choice question (MCQ), varying with the level of complexity and subtlety of the changes in the scenario (i.e., more complex scenarios required options to have a greater level of subtlety).
Similar to cue recognition, for each scenario there was a comprehension and anticipation minimum benchmark required needed to take the correct decision.
Two decision actions (based on the cognitive task analysis; Roberts et al., 2016) were included that could be selected at any time: either to flow check or shut in the well, both of which would terminate the scenario. The accuracy and latency of the choice of these two options was the performance measure of decision making.
Prepilot
After development, these scenarios were piloted on five novices (four postgraduate psychology students and an individual with experience in the oil industry but not in drilling). We found that novice participants responded to obvious, sudden cues rather than to gradual changes; they generally selected basic comprehension responses, and none took the correct decisions at the correct time.
Two drilling instructors also completed the task of identifying the cues, more frequently selecting the higher-anticipation responses and making the correct decisions quickly (i.e., small-latency responses). This prepilot illustrated that the task was functional in that both novices and experts understood what was required in terms of responses, but it still required a level of expertise to complete correctly (i.e., the task provided a basic differentiation between novice and expert).
Study 2: Pilot Study
The aim of Study 2 was to pilot the prototype Drillers’ Situation Awareness Task (DSAT) to test its preliminary viability for examining drillers’ SA and decision making during four drilling scenarios.
Procedure
The DSAT was piloted over a 5-week period at two of the sponsor’s training simulation facilities during training courses. Access was negotiated to a sample of drillers from the same company, attending Levels 3 and 4, mandatory well-control training courses (through personnel who had previously been involved in the project, i.e., “snowballing” recruitment; Marshall, 1996). Ethical approval was granted by the university’s Psychology Ethics Committee.
Before the task began, demographic information was gathered: information on age, current job position, years in that position, and time since last in the driller’s chair. Then the task instructions were given. At the beginning of each scenario, handover information was given (which could later be manipulated for priming), for example, “You are drilling ahead at 4,450 ft. You are not expecting any problems with the formation or equipment.”
Sample
Drillers typically completed the DSAT in classes of three or four individuals. The sample (N = 14) consisted of three drillers, an assistant driller, two tool pushers, five drilling instructors, and three offshore installation managers from drilling rigs. The age of the participants ranged between 25 and 55 years (25–35, n = 4; 36–45, n = 4; 46–55, n = 4; 56–65, n = 2). More than half of them had spent time in the driller’s chair in the past year (57%; past month, n = 5; past 6 months, n = 1; past year, n = 2; past 18 months, n = 1; past 2 years, n = 1; past 5+ years, n = 4). The participants had a mean of 15 years’ experience in the drilling industry (range 5–30 years, SD = 8). The majority had more than 10 years’ experience (79%).
Data analysis
We analyzed the responses using SPSS 21 (IBM, 2012). The analysis consisted of cue accuracy and latency, comprehension accuracy, and the decision selected and the time taken.
Results
Participants completed the DSAT in an average of 24 min (range = 18–34 min).
Cue recognition
The results (see Table 1) showed that on average, the participants responded to three cues (M = 2.6, SD = 1.1) out of a possible 5.5 cues (where two cues [M = 2.1, SD = 0.5] were the minimum benchmark) within a mean of 20.9 s of the cue’s appearance (SD = 30.5). This finding suggests that all participants were able to recognize and respond to sufficient cues to understand the developing situation.
Results From the DSAT Pilot Study Including Participants’ Cue Recognition, Comprehension and Anticipation, and Decision Making
Note. DSAT = Drillers’ Situation Awareness Task.
For Scenario 1, average decision time is given for where actions were incorrectly taken.
Comprehension and anticipation
On average, the participants scored 17.9 (SD = 6.9) out of a possible 43.8 for the comprehension and anticipation MCQs, where 9 was the minimum benchmark; a similar score was reflected across the scenarios (see Table 1). This finding suggests that the participants formed a sufficient understanding and/or anticipated how the scenario may develop to make a decision.
There is a discrepancy between the Level 2 and Level 3 SA responses (see Table 1). Although participants could be recognizing the cues and going directly to anticipation, it is more likely that they had already understood the situation before selecting the anticipatory MCQ response.
Decision making
The results suggest that despite variations in SA, the majority of the participants made the correct decision for each scenario (Table 1). In general, they took a decision within a minute of the correct decision point (i.e., benchmark time; see Task Outline), suggesting that they were responding relatively quickly.
Discussion
Considering the importance of drillers’ SA and decision making for maintaining well control − and consequently the safety of not only the drill crew but also the drilling rig − it is crucial to have tools that support their cognition. We developed the prototype DSAT as a tool for examining drillers’ key cognitive skills associated with SA.
Preliminary evaluation suggests that DSAT is a viable tool for measuring drillers’ cognition using performance measures, in that participants were able to identify cues (cue recognition accuracy and latency) to develop a sufficient understanding of the well-control situation (comprehension and anticipation accuracy) so as to take the correct decision (decision-making accuracy and latency). These measures related to Endsley’s (1995b, 2016) three key SA cognitive processes, and thus those in the DSA model, of perception and cue recognition (Level 1), comprehension and understanding (Level 2), and anticipation (Level 3) as well as subsequent decision making.
Informal feedback from the participants supported ecological validity, including comments that task and scenarios seemed realistic and that the tool would be valuable for training both technical and cognitive skills, particularly for less experienced drillers or assistant drillers. With further development and evaluation, the DSAT has the potential to be used as part of training and formatively assessing cognitive skills in drilling, supporting safe performance.
The DSAT adds to the limited existing methods for supporting and training cognition in drilling, which are mainly classroom-based exercises for teams (e.g., nontechnical skills training; IOGP, 2014a; tactical decision games; Crichton et al., 2004). The DSAT is derived from several established techniques (EXPERTise; Loveday, Wiggins, Searle, et al., 2013) and real-time probes (e.g., Durso et al., 1998). It has the potential to be a relatively objective measure of SA compared with self-rating techniques (e.g., Taylor, 1990) or observer rating tools (e.g., Matthews & Beal, 2002). In addition, the DSAT tool is portable and does not require participants to travel to a large, costly training facility.
The DSAT is a prototype tool and, as such, has a number of limitations. Possible solutions through future research are outlined. The MCQ options could be assisting or biasing the participants’ SA by priming their awareness or redirecting their attention (Salmon et al., 2009). To give a more accurate measure of awareness, the MCQ options could include both comprehension and anticipation, requiring participants to select as many as they think are correct.
Once the DSAT is further refined, a study could be conducted to evaluate its reliability, sensitivity, and validity, such as was done for the SAGAT (e.g., Endsley & Garland, 2000), and to develop benchmarking data for assessments and individualized feedback. This tool could be further refined to train specific drilling skills as outlined in the DSA model (e.g., significance of patterns of cues and possible anticipated outcomes) and to examine influencing factors (e.g., distractions or expectations) and system changes (e.g., shift patterns, interface design, or procedural change).
The DSAT also could be applied to other monitoring positions within drilling (e.g., mud logger) and oil and gas (e.g., crane operator). The computer-based method has the potential to be customized to measure domain-specific cognitive skills, such as in nuclear power control (e.g., control room operators) and health care (e.g., anesthetists), particularly for training low-frequency, high-risk situations.
There are also potential applications for the DSAT to be used in conjunction with crew resource management training, to evaluate the effectiveness of training of transferring desired behaviors during routine and abnormal operations, or as an assessment alternative to large simulations, for example, using the computer simulation in combination with behavioral markers within drilling (Roberts & Flin, 2016). Employing novel solutions, such as our portable computerized simulation task, is essential for maintaining safe and effective operations in the current unpredictable, cost-cutting climate.
Footnotes
Appendix
Details on Each Scenario Including Example SA Requirements
| Scenario | Length (minutes) | Key Features | Decision (time from onset) | Example SA Requirements |
|---|---|---|---|---|
| Practice | 2 | Hard formation | N/A | Increase in rate of penetration |
| 1 | 8 | Drilling into a transition zone followed by a number of hard formations before drilling into expected new formation | Flow check (150 s) | Fluctuating rate of penetration and torque, indicating a set of hard formations
Anticipating that after the hard formation has been drilled through, there will be the new formation |
| 2 | 3 | Drilling into a soft, porous formation, resulting in partial losses and escalating into total losses | Flow check (70 s) | Gradual decrease in active gain/loss with slight increase in rate of penetration
Indicates potential loss in the well; anticipate drilling into soft formation could result in loss of well control |
| 3 | 4 | Either drilling into a hard formation or problems with the bottom hole assembly | No action; wait | Fluctuating rate of penetration
Dip in hook load Could be problem with equipment or drilling through a hard formation |
| 4 | 7 | Mud transfer on the rig while drilling and a gradual influx from the well | Flow check (120 s) | Expected rapid increase in active gain/loss
Gradual increase in active gain/loss Slight decrease in stand pipe pressure could be from an influx of lighter hydrocarbons |
Note. SA = situation awareness; ROP = rate of penetration.
Acknowledgements
This article is based on a doctoral research project of the first author, which was sponsored by an international drilling rig operator. The views presented are those of the authors and should not be taken to represent the position or policy of the sponsor. The authors wish to thank the industrial supervisor, drilling consultants, drillers, assistant drillers, and all those who added to the project for their contribution. The authors also thank Mark Wiggins for giving access to the program EXPERTise.