Eye tracking during a simulated start of shift safety check: An observational analysis of gaze behavior of critical care nurses

Introduction

An effective handover is considered one of the “nine patient safety solutions” according to the World Health Organization (WHO). ¹ It is a core skill for intensive care unit (ICU) nurses and doctors, and is prone to many errors such as loss of important clinical information by either failure to convey or failure to recognize existing problems. ² However, no uniform guidelines or training recommendations exist. Handover within the ICU involves transfer of responsibility of care of complex and critically ill patients when patients are being admitted to or discharged from the ICU or during shift handover. It is a frequent but challenging task. Deficiency of communication or miscommunication in this process both can lead to professional dissatisfaction, increase health care costs, and compromise patient safety. ³

Efforts to standardize that process have increased in recent years, ranging from numerous mnemonic tools to system-based approaches or checklists. ⁴ Many of these reported standardized handover protocols have been shown to improve professional satisfaction, reduce technical errors, and improve retention of information,^5,6 which has led to increased support for their use.^{7 –9} However, the level of implementation, adherence and training in this area remain low and uncertain. ⁴ As such, appropriate training for handover has also been proposed in order to ensure its quality. ¹⁰ Handover is often plagued by errors in or inadequate communication. Its completeness is often affected by time pressure, distractions, and inappropriate surroundings, which in turn causes information loss and even critical errors.^11,12 It has been shown that up to 50% of clinical information is either not conveyed or not retained after handover amongst staff intensivists, which further emphasized the importance of communication strategies such as checklists, as well as brief clinical examinations. ¹³ The burden of an inadequate handover may be associated with key patient information being lost and treatment plans being missed. Undoubtedly, the consequence of those in the complex and vulnerable ICU patient population may become catastrophic. Another study has found that handover without using checklists is associated with more than 60% loss in information compared with using them. ¹⁴

In general, most existing studies focus merely on conveying information and communication during handover. However, there is no objective measure of how much clinical information is retained or whether potential critical errors are recognized by clinical examination during handover and subsequent shift start safety checks. The novel technology of eye tracking might be a suitable tool to analyse gaze behavior during handover as it gives insight into areas of interest associated with complex tasks. This can be helpful since the handover and shift start safety checks are very heterogeneous inter-individually as no clear guidelines exist. ³ Eye tracking involves the analysis of ocular and pupilar movements by using infra-red light reflected by the cornea and detected by recording cameras. ¹⁵ Distinct gaze metrics provide indices to assess visual behavior of participants. The technology minimizes effects related to expected behavior, while revealing objective information relating to gaze behavior that conventional observations normally miss. ¹⁵ As a substantial part of the cognitive information processed by health professionals at the bedside is visual information, the role of eye movements during the execution of complex tasks becomes particularly important. ¹⁶ In previous research, eye tracking has been proven to be a helpful tool to assess gaze behavior in different ICU settings, contributing to a more profound understanding of complex situations during the treatment of patients and of possible sources of human errors.^{17 –19} To our knowledge, eye tracking has however not been used for research in the setting of handovers and shift start safety checks, which may also be prone to the occurrence of potentially harmful errors.

In this study, we thus used eye tracking to analyze gaze behaviors of ICU nurses during a handover with a shift start safety check. We hypothesized that specific gaze behavior correlates with error recognition during the safety check.

Methods

Simulation room and errors

The simulation room was equipped with a mannequin (Lifecast Body Simulation London UK Adult mannequin) set up as an ICU patient including vital parameter monitoring and diverse diagnostic and therapeutic installations (vital parameter monitoring, compression stockings, peripheral and central venous lines, extra ventricular drainage). Further it included a handout specifying the medical conditions of the patient and a prescription sheet. The fictive 50-year-old female patient was hospitalized for a spontaneous subarachnoid hemorrhage after endovascular aneurysm treatment (coil embolization of the arteria cerebri posterior). An extraventricular drainage (EVD) was placed to prevent intracerebral hypertension. The previously healthy patient was allergic to metamizole and was lactose intolerant. An overview of the simulation setting is shown in Figure 1.

OPEN IN VIEWER

Figure 1.

Overview of the simulation setting. The five critical and the five minor errors are shown (red circles). Detailed descriptions of the errors can be found in Figure 2.

We incorporated five critical and five minor errors based on real safety hazards that had happened at the ICU and had been reported in the Critical Incident Reporting System (CIRS). We classified errors as “critical” when they were considered as potentially lethal in the shortest term (e.g. blood group incompatibility) and as “minor” when the potential harm to the patient was not immediate (e.g. lactose containing yogurt despite lactose intolerance). A detailed overview of the different errors is shown in Figure 2.

OPEN IN VIEWER

Figure 2.

Detailed description of the five critical and the five minor errors.

Results

Participants

In total, 49 ICU nurses were included in the study, of which six dropped out due to inaccurate recordings such as insufficient calibration or incomplete/erroneous fixation recording. Of the 43 remaining participants, 33 (77%) were female. Role and professional qualification of the included participants are presented in Table 1. Median general professional experience was 194 months [IQR 123–279] and median ICU experience was 72 months [IQR 33–192], respectively (Table 1).

OPEN IN VIEWER

Table 1.

Baseline characteristics of included ICU nurses (n = 43).

Female sex	33 (77%)
Role
ICU nurse with managerial function	5 (9%)
Certified ICU nurse	22 (51%)
ICU nurse student	11 (26%)
Intermediate care (IMC) nurse	5 (12%)
General professional experience (months)	194 [123–279]
ICU experience (months)	120 [33–192]

Categorical numbers are reported as absolute numbers and percentages (with percentages in brackets). Continuous data are reported as median and interquartile range (with interquartile ranges in square brackets).

Critical and minor errors

In Table 2, occurrence of errors (critical and minor errors) during the safety check is characterized. While all participants committed at least a minor error (n = 43, 100%), at least one critical error was committed by 29 (67%) of the 43 included ICU nurses (Table 2). A detailed description of the number of participants overlooking critical (numbers 1–5) and minor (numbers 6–10) errors is provided in Supplemental File 2.

OPEN IN VIEWER

Table 2.

Occurrence of critical and minor errors.

Participants missing a critical error	29 (67%)
Participants missing a minor error	43 (100%)
Number of all missed errors (critical and minor) a	4 [3–6]
Number of missed critical errors a	1 [0–2]
Number of missed minor errors a	3 [3–4]

Categorical numbers (i.e. number of participants committing errors) are reported as absolute numbers and percentages (with percentages in brackets). Continuous data (i.e. number of errors) are reported as median and interquartile range (with interquartile ranges in square brackets).

a

Refers to the median number of missed errors per participant. A missed error was defined as a safety hazard that was not recognized by the participant.

Mixed model analysis

In the mixed model analysis taking all errors (critical and minor) into account, it became apparent that the odds ratios of missing errors differed according to specific eye tracking gaze metrics (Figure 3). While a longer time to First Fixation or more Revisits were associated with increased odds ratios (OR) for missing errors (Time to First Fixation: OR 1.099 (95% CI 1.023–1.191, p = 0.0002), Revisits: OR 1.080 (95% CI 1.025–1.143, p = 0.0055)), the opposite effect was detectable for First Visual Intake Duration (OR 0.771 (95% CI 0.660–0.871, p = 0.0002)), Average Visual Intake Duration (OR 0.765 (95% CI 0.651–0.867, p = 0.0002)), and Dwell Time (OR 0.774 (95% CI 0.672–0.862, p < 0.0001)).

OPEN IN VIEWER

Figure 3.

Mixed model analysis taking all errors (critical and minor) into account. The odds ratio (OR) of error and 95% confidence interval (CI) are shown for time to first fixation, first visual intake duration, average visual intake duration, dwell time, and revisits.

In the mixed model analysis restricted to only critical errors (Figure 4), similar results as in the analysis considering all missed errors could be obtained. The OR of missing critical errors increased with higher Time to Fixation (OR 1.095 (95% CI 1.018–1.196, p = 0.0252)). In contrast, the OR of missing critical errors decreased with increases in First Visual Intake Duration (OR 0.834 (95% CI 0.687–0.967, p = 0.0282)), Average Visual Intake Duration (OR 0.839 (95% CI 0.690–0.972, p = 0.0338)), Dwell time (OR 0.814 (95% CI 0.659–0.936, p = 0.0145)), and Revisits (OR 0.916 (95% CI 0.837–0.994, p = 0.0441)).

OPEN IN VIEWER

Figure 4.

Mixed model analysis taking only critical errors into account. The odds ratio (OR) of error and 95% confidence interval (CI) are shown for time to first fixation, first visual intake duration, average visual intake duration, dwell time, and revisits.

Discussion

In this study, we observed the gaze behavior of critical care nurses at handover with a focus on shift start safety check in a simulated setting. The main results strengthen our hypothesis that specific gaze behavior correlates with error recognition. The risk of error occurrence increased with time to first fixation in both analyses (overall errors and critical errors only). Thus, the correct detection of errors was associated with shorter duration of time to fixation. Longer dwell times were associated with less missed errors. There was no difference in gaze behavior between more experienced and less experienced staff in the detection of errors.

We used the eye-tracking tool to better understand the relationship between identification of critical errors during shift start safety check and the associated gaze behavior of ICU nurses, where comparable studies focusing on eye movements do not exist to our knowledge. While conventional observations frequently are associated with biases such as observation biases, ²³ the technology of eye tracking in contrast can offer objective gaze metrics in various settings including critical care.^{24 –27} For example, eye tracking has previously been shown to effectively quantify the visual attention amongst critical care professionals performing daily bedside practice. ²⁸ The method can also be successfully used to minimize the occurrence of the Hawthorne effect, which overestimates compliance of professionals in settings where they are directly observed. ²⁹ Our present study was able to present an in-depth characterization of visual patterns during a shift start safety check, which would not be possible with other conventional methodologies of observation. As one of our main results, we found that the risk of overlooking an error during shift safety check increased with a higher time to first fixation. This finding is highly plausible since it took ICU nurses longer to identify the relevant AOI and thus to detect the corresponding errors. Moreover, longer dwell times, first intake durations and average visual intake durations were associated with less unrecognized errors in the overall analysis. In the context of our study design, we assume that those gaze metrics (i.e. Dwell time, first intake duration, and average visual intake duration) are an expression of accurate cognitive detection of errors. The longer participants gazed at distinct AOIs, the more likely they were to detect and identify the corresponding errors. It is plausible that the opposite (i.e. shorter overall dwell times and average visual intake times) would have led to missing more errors, as the errors simply might have been overlooked and visually disregarded. Interestingly, more revisits were associated with not recognizing more errors in the overall analysis. We explain this finding as an expression of the fact that participants struggled to detect the corresponding errors at first hand and thus frequently revisited AOIs, without drawing any consequences or recognizing the associated errors. In contrast, if an error once has been detected, no more revisits to the corresponding AOIs would have probably been necessary, which in turn leads to decreased revisit numbers.

In the analysis including only critical errors, we found comparable results to the overall analysis. Longer fixations (i.e. dwell time, first visual intake duration, and average visual intake duration) also have been associated with missing less critical errors. Especially for the detection of critical errors, longer attention seems necessary to visually grasp those errors and cognitively process them. In contrast to the overall analysis, more revisits were however associated with missing less critical errors. In previous eye tracking research, revisits have been characterized as a surrogate of re-checks, monitoring, and control glances.^30,31 This finding thus could be explained by the fact that participants frequently re-checked AOIs with associated critical errors, even if the error already had been detected before. Such re-checks and verification blinks might have direct clinical consequences during the treatment of the patient, especially with regard to critical errors that potentially compromise patient safety (such as e.g. pump rates at perfusors or dosages of medications with effects on the circulation). The technology of eye tracking thus can provide highly precise assessments and crystallize relevant visual patterns during processes that are associated with increased risks for ICU patients. This would not be possible with pure conventional observations. Eye tracking should thus be used to investigate handovers and safety checks on the ICU more in detail. It could be used to develop checklists and standardized processes during handover procedures. Previous studies have made a similar observation in high-risk events (e.g. ICU patient transfers) and have shown that standardized handover protocols and preparation time reduced safety errors and information retainment. This led to increased professional satisfaction whilst handover time did not change significantly. However, further improvement and the development of such structured approaches is needed to systematically reduce safety hazards in the future.^5,6,14,32,33

In a sub analysis comparing more and less experienced staff, we found no differences according to experience level, which is an interesting finding that we did not expect. More expertise and longer professional experience in the ICU apparently did not lead to lower error rates, even after long years of working in the field. Several possible reasons could explain this finding. On the one hand, more experienced staff simply might rely on their own subjective approaches at shift start safety checks and perform according to their own, long-established routine, which might prone them to incur errors and overlook relevant details. New trainees might be trained to approach their patients in a structured way, possibly leading to standardized procedures during handover and safety checks. The combination of those might have led to our observation and the lacking differences. On the other hand, we can’t rule out residual confounders owing to limited participant numbers, heterogenous levels of experience, and professional functions within our cohort, or statistical underpowering. Nevertheless, our findings highlight that also more, experienced and older staff need frequent training and teaching sessions, be it in real-life or in simulation rooms.

Our study has several strengths, including its novelty and innovative design. We provided comparable conditions in every observation by exposing each participant to the same setup. Moreover, we created conditions close to reality by using a clinical case based on previous CIRS cases that happened in real-life. From a methodological point of view, the participants were only instructed to perform a handover and shift start safety check, and they did not know if and how many errors were hidden, which might have enhanced reality and minimized biases. We could also exclude stress (a well-known factor affecting performance) as a biasing influence using a standardized questionnaire.

The study also has to account for some limitations. The number of participants was relatively limited and the study was conducted in a single center design. Larger study populations might thus overcome potential associated biases and shed more light on the direct impact of gaze behavior on error detection. Moreover, the study was conducted in a simulated scenario. The utilization of our findings in a real clinical setting might thus be difficult. However, other research has demonstrated elevated error rates within simulations, underscoring the potential impact of simulation findings for clinicians in real-life.^34,35 All participants were working in the same hospital, probably limiting external validity. It would be interesting to observe critical care nurses from different centers to evaluate if they perform differently.

This study assessed two aspects of the handover and shift start safety check. The first aspect was the case presentation and the second part the subsequent safety check procedure. Different performances of the first part might thus have had impacts on the subsequent checking procedure, which our study did not account for. Furthermore, different institutions might have differing protocols for handovers and safety checks. Thus, our assumption that checks for errors are made during handover (as is routinely performed at our institution) may not be generalizable to other institutions. In addition, our selection of errors – despite being based on real errors reported in our CIRS – might be biased to some extent.

In conclusion, our results show that error detection during a shift start safety check was associated with distinct gaze behavior of ICU nurses. The gaze characteristics of the participants who recognized more errors fit to a more structured approach. Our study implicates that transition to real-life eye tracking studies are an important next step to refine and further validate our findings. In this way, checklists could eventually be developed or guidelines generated for organized and safe handovers in ICUs. Moreover, future studies in real environments should investigate whether structured handovers can directly minimize the occurrence of human errors. This would be an important next step to improve patient safety in critical care.

Supplemental Material