Leveraging the Critical Decision Method to Develop Simulation-Based Training for Early Recognition of Sepsis

Abstract

Training hour reductions for resident physicians have resulted in fewer opportunities for novices to manage critically ill patients. Our goals were (a) to understand differences in how novices and experts notice and interpret clinical cues using sepsis as an exemplar and (b) to develop simulations that replicate clinical cues to facilitate acquisition of expertise. Researchers conducted 14 critical decision method (CDM) interviews with four novices (interns), four senior trainees (senior residents), and six faculty (expert) physicians. We interviewed across a spectrum of experience to better assess for experience-based differences in sepsis recognition. Investigators analyzed transcribed interviews using a card sort technique. Experts described more hypothesis testing and violated expectations than novices. Expert–novice differences in sepsis recognition informed the design and future piloting of training scenarios that require novices to seek, interpret, and act on relevant cues.

Keywords

expert–novice differences cues resident supervision duty hours critical decision method simulation-based training cognitive task analysis resident training

Introduction

Sepsis, a systemic response to infection and its corresponding organ dysfunction, remains a leading cause of death worldwide, exceeding acute myocardial infarction, stroke, or cancer (Angus et al., 2001). Subtle, nonspecific symptoms make diagnosis of early sepsis a challenge for novice healthcare providers, often delaying timely treatment. A delay in recognition and management of sepsis is associated with increased morbidity and mortality (Han et al., 2003; Lundberg et al., 1998; Odetola et al., 2008). The pediatric patient population is often nonverbal, early in development, and reliant on care providers to “interpret” symptoms. Regulatory factors may be widening expert–novice differences in recognition of pediatric sepsis. Changes in both the residency training requirements and the pediatric patient population have reduced opportunities for residents to treat patients at risk for life-threatening deterioration. Thus, it is critical to develop methods that promote early recognition of sepsis in order to improve patient survival. This article describes a project to identify the factors that differentiate between the novice’s and expert’s approach to the early recognition of sepsis at the bedside, a step towards developing improved training for novices in the recognition of sepsis.

Reduced Duty Hours

Medical school emphasizes learning basic medical knowledge, whereas clinical expertise (decision-making skills) is developed during intensive postgraduate training (e.g., residency and fellowship training). Following medical school, residents work as apprentice-type physicians and are assumed to be able to practice independently upon completion. Residency is considered the most difficult, physically taxing, and fatiguing portion of training and has been identified as a likely setting for medical error. In 2003, to improve patient safety by mitigating fatigue, the Accreditation Council for Graduate Medical Education (ACGME) imposed stricter regulations about the number of hours a resident can work, limiting their clinical care time to 80 hr a week. Shifts are limited to16 hr for an intern (novice), with a minimum of 8 hr off between shifts and at least 1 day off per week averaged over 4 weeks. This is a marked decrease in clinical hours worked per week, and residents must now frequently “sign out” or hand over patient care at a set time, irrespective of their patient’s clinical condition. An artificial construct is imposed; residents may not be able to follow cases from beginning to outcome, identified as closed-loop learning (Argyris, 1991; Argyris & Schön, 1978). Experiential learning is changed, and clinical experience is segmented as time between observations increases. These gaps in observation come at a cost to the novice learner as subtle cues of illness progression, and the path between them, can be lost. Thus, restricting work hours limits resident opportunities for both exposure and real-time feedback (Kerlin & Halpern, 2012), especially the rate of exposure to critical illness and the opportunity to gain crucial clinical experience (ACGME, 2007; Wheeler, Clapp, & Poss, 2003; Wheeler, Vaux, Starr, & Poss, 2000). Specific to sepsis, symptoms may be subtle and can develop over hours to days; thus, work hour restrictions reduce the opportunities to experience these patients while their sepsis is evolving, a critical closed-loop process required for cementing contextual knowledge (Chang, Hamilton, & Carter, 1998; Hutter, Kellogg, Ferguson, Abbott, & Warshaw, 2006; Wheeler et al., 2003). An unintended consequence of the work hour restrictions is the diminished exposure to the clinical situations that provide novices with the opportunity to develop the skills associated with recognition-primed decision making (Klein, 1998; Merton, 2008). Bowen (2006) has written:

deliberative analytic reasoning is the primary strategy when a case is complex or ill-defined, the clinical findings are unusual, or the physician has had little clinical experience with a particular disease entity. Non-analytic reasoning as exemplified by pattern recognition is essential to diagnostic expertise and this skill is developed through clinical experience. (p. 2220)

Resident Supervision

Another potential challenge limiting resident opportunity to gain clinical experience was the Centers for Medicare and Medicaid Services (CMS) regulation 1780 concerning documentation of teaching (attending) physician presence when residents deliver patient care. This regulation was a response to widely publicized cases levying large fines against teaching institutions by the Department of Health and Human Services, alleging fraudulent documentation of physician supervision of trainees. Correspondingly, the ACGME issued specific and stringent requirements of direct and indirect resident supervision. These stricter rules concern entrustable professional activities and what a resident can do autonomously—tasks done in the presence of a supervising physician (direct supervision) versus tasks a resident does independently of immediate oversight. A supervising physician determines entrustment, specifically whether the supervising physician will provide direct, indirect, or oversight supervision (Kerlin & Halpern, 2012). The additive effect of the CMS regulation and the ACGME resident supervision requirements has likely decreased residents’ opportunities for independent decision making. Contemporary residents see a patient and present the case to a supervising physician before any significant intervention; a previous generation of residents performed rapid assessments and made urgent critical decisions without supervision—that is, endotracheal intubation (insertion of a breathing tube). Attending physicians are now generally physically present in critical and high-risk areas such as an intensive care unit, and the attending physician, not the resident, is ultimately responsible for care. Thus, there are fewer situations where the resident acts autonomously. Although effective mentoring and coaching from a supervisor certainly has the potential to aid residents in learning to make good decisions (Hoffman et al., 2014), some research suggests the increased supervision regulations for physician residents may have detrimental effects. Specifically, residents may think the supervising physician is responsible for all serious decision making, which can shift the mindset from autonomous problem solving and action to a “just tell me what to do” or a reporting kind of mindset, resulting in diminished accountability, both perceived and actual (Fazio, Wheeler, & Poss, 2000; Kline-Krammes, Wheeler, Schwartz, Forbes, & Bigham, 2012; Wheeler, Sperring, Vaux, & Poss, 1999). Instead of assessment and sense-making skills, residents develop information gathering and reporting skills. Thus, current clinical contexts may be displacing and extending the timeframe for autonomous action while more carefully measuring the residents’ performance during training.

Changing Patient Population

Epidemiologic statistics have shown that sepsis accounts for significant morbidity and mortality in children. If deaths resulting from infectious diseases like pneumonia, gastroenteritis, and meningitis are factored in these statistics, sepsis is clearly the leading cause of non-trauma-related death in the pediatric age group (Riley, Basu, Kissoon, & Wheeler, 2012). The lack of a reliable case definition and improper attribution has clouded the true extent of the problem (Angus & Wax, 2001). Available studies in critically ill children and adults suggest that the overall incidence of sepsis (both children and adults) has increased to 750,000 new cases per year in the United States alone, with a rate of hospitalization that doubled from 11.6 per 10,000 population in 2000 to 22.1 per 10,000 population in 2008 (Angus & Wax, 2001; Centers for Disease Control and Prevention/National Center for Health Statistics, 2000-2008; Kumar et al., 2011; Wood & Angus, 2004). In a recent study comparing U.S. data from 2005 to previous years, there was an 81% increase in pediatric severe sepsis cases compared to 1995 and a 45% increase compared to 2000. This represents an increase in prevalence from 0.56 to 0.89 cases per 1,000 pediatric population. In 2005, there were 75,255 pediatric hospitalizations involving severe sepsis, at a healthcare cost of $4.8 billion (Hartman, Linde-Zwirble, Angus, & Watson, 2013). Despite advances in immunizations, recent studies reveal pediatric sepsis is becoming more common and even more deadly, with reported mortality rates approaching 25% (Weiss et al., 2015). The epidemiology of sepsis in children has changed over time, with higher numbers of opportunistic infections in immunocompromised patients becoming more common (Riley et al., 2012). Public policy and prevention efforts to improve immunization rates, implement perinatal antibiotic prophylaxis, and reduce hospital-acquired infections (e.g., central line infections) have contributed to the changing sepsis demographics (Bigham et al., 2009; Brilli et al., 2008; Greenhow, Hung, & Herz, 2012; Laupland, Gregson, Vanderkooi, Ross, & Kellner, 2009; Lin et al., 2011; Nowak et al., 2010; Riley et al., 2012; Riley & Wheeler, 2012; Yildirim, Stevenson, Hsu, & Pelton, 2012). Additionally, the pediatric population at risk has shifted to one centering on chronic illness (i.e., patients’ status posttransplant, undergoing chemotherapy, or with shortened intestines) and increased life expectancies of technology-dependent children where early sepsis is difficult to identify, particularly for novice physicians.

Advances in Training

One potential strategy in novice physician education to fill in the described gaps created by regulated work hours, increased supervision, and changing patient population is simulation-based training. Hoffman et al. (2014) describes simulation-based training incorporating expert knowledge models and challenging realistic scenarios as an important strategy for accelerating expertise. The medical community continues to leverage simulation to provide additional opportunities for targeted training experiences (Gaba, 2004; Issenberg, McGaghie, Petrusa, Gordon, & Scalese, 2005; Owen, Mugford, Follows, & Plummer, 2006). Simulation technology is common in medical education, and research indicates that students learn better in a safe and supportive environment of facilitated experiential learning (McGaghie, Issenberg, Petrusa, & Scalese, 2010). A key argument for simulation in healthcare education is that “participation in simulation-based education appears to help prevent participants from making mistakes in the future, by providing a set of clinical circumstances in which it is permissible to make mistakes and learn from them” (Lewis, Strachan, & Smith, 2012, p. 88). This is especially relevant and applicable with critical illnesses (i.e., sepsis) and injuries that are seen infrequently by individual providers. Simulation allows novice physicians the opportunity to increase their exposure to these types of cases while allowing autonomous decision-making in a safe learning environment.

Simulation fosters effective learning through active learner engagement, repetitive practice, the ability to vary difficulty and clinical complexity, as well as diagnostic performance measurement and intra-experience feedback (Issenberg et al., 2005; Okuda et al., 2009). Thus, a simulation-based curriculum for pediatric sepsis requires the creation of valid and reliable scenarios. “The critical factor is that the simulation scenario induces transfer-appropriate processing; that is, those cognitive processes required for performing a task under normal operating conditions” (Weaver et al., 2010, p. 370). Simulation training focused on sepsis recognition must recreate the nuances of the signs and symptoms that septic patients present with, including the correct timeline and the clinical cues upon which experts rely, ones that are less well-documented (Crandall & Getchell-Reiter, 1993). Current literature suggests that using simulation to train providers on sepsis management is highly rated by participants; however, there is yet to be a study that shows improvements in technical proficiencies or clinical care after such training, even when randomized against no intervention or a situation awareness course (Hänsel et al., 2012; Nishisaki et al., 2009).

Regarding sepsis specifically, one noteworthy study explored expert–novice cue utilization within a cohort of cardiac critical care nurses using a written simulation on sepsis. Investigators found that expert providers demonstrated a higher ratio of highly relevant cues to total cues than novices, supporting the idea that nurses develop domain-specific knowledge with experience and that diagnostic expertise is associated with highly relevant cue recognition (Reischman & Yarandi, 2002). The subtle and nonspecific ways that sepsis presents in a complex pediatric population demand inclusion of realistic cues in the context of challenging scenarios that adequately replicate real-world presentations. If these scenarios can be developed, they should provide increased contextual learning and translate to improved assessment and management outcomes. Because this aspect of the research program was highly exploratory, we did not state formal hypotheses. Our intent was to explore differences in cue recognition and utilization among physicians with different levels of expertise. We leveraged the critical decision method (CDM) to engage both pediatric novices and experts with the following objectives: (a) collect and understand critical incidents about sepsis recognition, (b) explore expert–novice differences in sepsis recognition, and (c) identify critical cues to inform the design of training scenarios. The global goal is to develop learning strategies to hasten recognition and treatment of a potentially devastating illness.

Method

Setting

This study was conducted at an urban, academic quaternary care pediatric institution with a large pediatric residency training program (currently 59 residents per year; over 200 residents total). Resident physicians have completed medical school (graduate level training) and then devote 3 or more years to clinical specialization, such as pediatrics, internal medicine, surgery, etc. Within this residency training program, pediatric residents care for patients in a variety of areas, including the Emergency Department (ED), Pediatric Intensive Care Unit (PICU), and Neonatal Intensive Care Unit (NICU), as well as outpatient clinics and general inpatient units. Throughout the hospital, the resident and the patient’s bedside nurse are usually the first providers to assess a patient. This initial assessment is then reported to a more senior physician. The study was approved by the Institutional Review Board of Cincinnati Children’s Hospital Medical Center.

Participants

Eligible participants were (a) novice physicians defined as first-year residents in pediatric training, (b) senior trainee physicians defined as third-year residents in pediatric training, and (c) expert physicians defined as faculty physicians in emergency medicine or critical care who had completed fellowship training and who had been functioning in the role of supervising (teaching) physician for at least 5 years prior to the interview. The only physicians excluded from enrollment were expert physicians who were investigators on this study (M.D.P., D.S.W., and G.L.G.). Additionally, any physician who was interviewed as part of the interviewer training (see description below) was then excluded from participation, and results of those practice interviews were not used in the analysis. Fourteen participants enrolled: four novices, four senior trainees, and six experts. No physician was interviewed more than once; however, some physicians provided multiple incidents within their interview session.

Because experience and expertise are not perfectly correlated (Ericsson, 2004; Ericsson, Whyte, & Ward, 2007; Hoffman et al., 2014), identifying experts is not generally straightforward. With regard to sepsis, there is reason to believe that acquisition of expertise may not correlate well with experience because exposure to relevant cases is highly variable. Examples of the types of encounters that define expert performance tend to occur with challenging cases as opposed to typical or routine cases (Ericsson, 2004; Ericsson et al., 2007; Norman, Young, & Brooks, 2007). Yet even in high-volume and well-regarded children’s hospitals, the overall volume of exposure to critical pediatric patients is minimal (Wheeler et al., 2000; Wheeler et al., 2003). Nevertheless, we expected faculty with at least 5 years of experience in a supervisory position to have more cases to draw from, and to be more skilled at recognizing sepsis-recognition than the novices and senior trainees. With regard to novices and senior trainees, we did not know whether we would see differences between these two groups, but we wanted to include participants across a spectrum of experience in this exploratory study.

Procedures

Interviewer training

Five study investigators were trained in the CDM interviewing approach. The training included review of CDM methodology articles, an online presentation, and two 4-hr in-person training sessions. These sessions were led by two internationally recognized content experts in CDM (L.G.M. and G.K.). The initial session included a live demonstration during which G.K. interviewed a local expert physician. The second session included a practice interview during which one of the investigators interviewed a senior trainee physician, allowing team members to observe and reflect on the interview approach. The investigators conducted multiple practice interviews, with a CDM expert in attendance to provide additional coaching. The practice interviewees included physicians from critical care and surgery as well as nurses from critical care units and the ED. During these training sessions and practice interviews, the investigators developed, and refined by consensus, an interview guide based on the CDM approach for recognition and early management of sepsis (see Online Appendix for the complete interview guide). This template followed the structure outlined by Crandall, Klein, and Hoffman (2006), which includes four phases or sweeps designed to elicit more information about the incident during the interview.

Study interviews

Participants were recruited to participate in a study interview centered on critically ill patients. Participants were not informed that the interview would focus on sepsis in order to prevent them from identifying incidents prior to the interview. Each interview was scheduled as a 2-hr session and was conducted by a panel of two trained study investigators, one physician and one qualitative researcher. One interviewer, usually the physician, functioned as the lead and created a timeline on a white board that visually depicted the incident(s). The second interviewer took notes on paper or electronically. To reduce demand characteristics, the lead interviewer explained the goals of the study, emphasized the importance of understanding the interviewee’s lived experience, and assured interviewees that information shared in the interview was confidential and was focused on eliciting the interviewee’s thought process concerning the specific case. Furthermore, the second interviewer, usually the qualitative researcher, was primed to interject and ask for clarification if specific cues, impressions, and assessments were not fully articulated. The entire interview was audio-recorded to allow transcription. Postinterview the timeline was photographed to ensure that no critical data were lost. An example timeline is demonstrated in Figure 1.

Figure 1.

Incident timeline constructed during critical decision method interview.

During the interview, the participant recalled a patient he or she had cared for whom sepsis was suspected or later discovered, including patients for whom sepsis was a “missed diagnosis” or for whom sepsis was “ruled out.” Each interview included four phases or sweeps designed to elicit progressively more detailed information about the incident. The four sweeps in the CDM include incident identification, timeline verification, deepening, and “what if” queries. During Sweep 1, the interview team inquired about a sepsis-related incident and listened to make sure the identified incident fit the study goals and that the participant played a key role. During Sweep 2, the interview team inquired about an overview of the sepsis incident, read back the overview, identified and recorded decision points/major events, looked for shifts in situation awareness, and asked clarifying questions to develop a starting timeline. During Sweep 3, the interview team utilized a question-and-answer technique to better understand the incident (including situation assessment and cues), better developed management goals during the incident, and repeated back confusing points to the interviewee for clarification. During this sweep, the interview team listened and recorded critical decisions, cues and their implications, ambiguous clues, strategies, and violated expectancies. In Sweep 4, the interview team used “what if” queries to tease out expert–novice differences, such as “what a new physician might have done” or “What do you think someone with more experience might have done?” Also included in this sweep were queries related to consideration of other alternatives or actions taken in relation to the incident. At the end of each phase (or sweep), the second interviewer would ask clarifying or deepening questions. Participants also reflected on the timeline generated, which often produced more detail. A second patient incident was identified and explored if time permitted.

Transcription process

Each interview was transcribed by a medical transcriptionist outside of the study team. To ensure content accuracy, each transcript was then reviewed by two physicians (D.S.W. and G.L.G.) on the study team who corrected transcription errors. These two investigators highlighted the content-rich areas of the interview transcripts. They included (highlighted) content that was pertinent to the incident; they excluded demographic information and extraneous conversation material.

Analysis

Analysis occurred in two stages. In the first stage, the data were coded for cue recognition. Cue recognition is defined as cues in the incident accounts that were identified by the interviewee in discussions of situation assessment. In the second stage, incidents were reviewed a second time to examine differences in cue utilization. Cue utilization includes statements describing how cues were interpreted, how they informed the overall assessment of the patient’s status, and how they drove specific actions. For example, if an interviewee stated that he or she had noted the patient’s tachycardia, fever, and that the blood pressure was “a little soft,” these would be examples of cue recognition in the first round of analysis. These data would be included in the second round of analysis only if the interviewee described how this information was used in making an assessment of uncompensated septic shock and acting on that assessment.

Cue recognition coding process development

Using information collected from the practice interviews, four of the investigators (M.D.P., A.B., L.G.M., and R.G.T.) independently identified cues. The investigators met to discuss the themes identified and came to a consensus on the themes that would be used in coding the actual study interviews. A card sort technique was used to organize cues into broad themes with subthemes, as shown in Table 1. Clear definitions for each theme, subtheme, and cue were developed, resulting in the creation of a codebook. An excerpt from the codebook showing the definitions developed for the theme “classic indicators of sepsis” is presented in Table 2 (see Online Appendix for the complete codebook).

Table 1:

Theme, Subtheme, and Cue Categories

Themes	Subthemes	Cues
Classic indicators of sepsis	Consensus criteria	Fever or hypothermia
		Tachycardia or bradycardia
		Tachypnea or bradypnea
		WBC abnormalities (high, low)
	Experienced-based criteria	Distal perfusion
		Mental status changes
		Ill appearance
		Hypotension
		Other lab abnormalities
		Rhythm disturbances
		Patient’s interaction with parent
Processing of information		Unclear story/presentation
		Can’t be explained by other things
		Placing presentation into more common
Violated expectancy		Failure to respond to treatment
		Unexpected response
		Steady decline
		Instability/volatility
Provider-related factors		Physiologic reaction
Things that can mislead you		Absence of fever
		Dark skin
		Fatigue of similar information
Risk factors for sepsis	Medical history	Obvious source of infection
		Chronic medical illness
		Age less than 2 months
		Indwelling plastic
		Lack of immunizations
		Fever of unknown origin
	Environment	Arrival during shift change/off hours
	Environment	Environmental focus on sepsis
Source of information		Family or caregiver concerns/anxiety
		Nonphysician healthcare provider concerns
		Strength of data
		Personal trust/respect
		Disparity between trainee and faculty impression
		Cue clusters

Note. WBC = white blood cell count.

Table 2:

Excerpt From Codebook Developed During Cue Recognition Coding Process

Classis Indicators of Sepsis	The Signs or Symptoms of Sepsis That Are Discussed in Textbooks, Articles, or More Traditional Methods of Learning
Consensus criteria	Published criteria on sepsis – SIRS criteria
Fever or hypothermia	Temperature above or below normal
Tachycardia or bradycardia	Heart rate above or below normal
Tachypnea or bradypnea	Respiratory rate above or below normal—that is, the patient was really working hard to breathe
WBC abnormalities (high or low)	Labs indicate a white blood cell count outside the normal range
Experience-based criteria	Recognized or accepted indicators of sepsis that may make it into some publications but are mainly gained through experience
Distal perfusion	Abnormal pulses, prolonged capillary refill, decreased extremity (i.e., toe) temperature, mottling and abnormal color, “clamped down”
Mental status changes	Disorientation to time, person, or place; difficult to arouse; lethargic; abnormal reaction to physician; different than baseline or usual per family/caregiver; change in activity level
Ill appearance	Walking into the room, the physician should be able to give a 5-sec impression/gestalt of well, ill, toxic, or unable to explain/nondescript but “not normal”
Hypotension	Blood pressure lower than the normal range for a patient that age
Other lab abnormalities	Anemia, elevated platelets, low platelets, neutropenia
Rhythm disturbances	Any arrhythmia, including low heart rate, extra beats (PAC, PVC), atrial rhythms like fibrillation or SVT, ventricular rhythms like V-tachycardia
Patient’s interaction with parent	The provider notes something significant in the way that the parent and the child are interacting—could be either positive or negative—including the patient’s bondedness with the parent. For example, there are kids with fever that are just different, the parent moves them, they don’t wake up and cry, they just remain limp

Note. SIRS = systemic inflammatory response syndrome, WBC = white blood cell count, PAC = premature atrial contractions, PVC = premature ventricular contractions; SVT = supraventricular tachycardia.

Each transcript was independently coded by the four team members. Three coders were qualitative researchers (A.B., L.G.M., and R.G.T.) and one was an ED pediatrician (M.D.P.). As some participants discussed multiple incidents of patients with sepsis during one interview, each incident was coded separately. A cue was coded each time it was mentioned as well as whether it increased suspicion of sepsis, decreased suspicion of sepsis, or was confusing/misleading in the generation of a diagnosis or differential for the physician. After the initial two incidents were coded, the coders met in person or by phone to discuss each item coded and came to a consensus on every code. During these discussions, codes were “redistributed,” meaning they might not remain in the context in which they originally presented, but were organized and recombined with other cues and clinical contexts as indicated. This process resulted in a set of master codes for each incident. Discussions and consensus then occurred after every two interviews to prevent coding drift and allow further revision of the coding sheet, if needed. This iterative process continued until each transcribed incident was coded and consensus had been reached. The frequency of each cue type was tabulated across all interviews and summarized by participant class (novice, senior trainee, and expert). Frequency was calculated based on a single mention of cue within an incident rather than the number of times a specific cue was mentioned during the retelling of the incident.

Cue utilization analysis

After cue frequencies were calculated, two researchers reviewed the incidents a second time to explore differences in cue utilization. Researchers reviewed each incident, extracted examples of cue utilization, and then met to discuss and reach consensus. No formal coding scheme was used. Rather, we used a grounded theory approach, searching for themes and documenting supporting examples.

Results

Fourteen physicians were enrolled in the study interviews. Four novice physicians provided four incidents, four senior-level trainees provided eight incidents, and six expert physicians supplied 11 incidents. In total, 23 incidents were analyzed.

Expert–Novice Differences in Cue Recognition

Although our sample size was too small for statistical comparison, we examined the cue frequencies for indicators of differences in cue recognition across the three experience levels. No clear trends in cue frequency emerged.

Cue Recognition

The first set of findings focused on critical cues—specifically cues interviewees mentioned that were noteworthy to the interviewee and led to or heightened awareness of the probability of sepsis. Table 3 shows the frequency of cues identified overall and by physician experience.

Table 3:

Cue Recognition by Level of Expertise

Cue Categories	All Incidents (n = 23)		Novice Incidents (n = 4)		Senior-Level Trainee Incidents (n = 8)		Expert Incidents (n = 11)
Cue Categories	Freq.	%	Freq.	%	Freq.	%	Freq.	%
Classic indicators of sepsis
Consensus criteria
Fever or hypothermia	18	78	3	75	5	62	10	91
Tachycardia or bradycardia	20	87	3	75	7	87	10	91
Tachypnea or bradypnea	10	43	2	50	3	37	5	45
WBC abnormalities (high, low)	6	26	1	25	3	37	2	18
Experienced-based criteria
Distal perfusion	15	65	2	50	5	62	8	73
Mental status changes	19	83	3	75	8	100	8	73
Ill appearance	13	57	3	75	4	50	6	55
Hypotension	13	57	1	25	7	87	5	45
Other lab abnormalities	14	61	2	50	5	62	7	64
Rhythm disturbances	2	9	0	0	1	12	1	9
Patient’s interaction with parent Processing of information	4	17	1	25	0	0	3	27
Unclear story/presentation	3	13	0	0	0	0	3	27
Can’t be explained by other things	5	22	0	0	1	12	4	36
Placing presentation into more common Violated expectancy	11	48	3	75	3	37	5	45
Failure to respond to treatment	10	43	1	25	4	50	5	45
Unexpected response	1	4	1	25	0	0	0	0
Steady decline	5	22	2	50	2	25	1	9
Instability/volatility Provider related factors	3	13	1	25	1	12	1	9
Physiologic reaction Things that can mislead you	1	4	0	0	0	0	1	9
Absence of fever	4	17	2	50	2	25	0	0
Dark skin	4	17	1	25	1	12	2	18
Fatigue of similar information Risk factors for sepsis	1	4	1	25	0	0	0	0
Medical history
Obvious source of infection	12	52	2	50	4	50	6	55
Chronic medical illness	15	65	3	75	5	62	7	64
Age less than 2 months	1	4	0	0	1	12	0	0
Indwelling plastic	7	30	2	50	2	25	3	27
Lack of immunizations	0	0	0	0	0	0	0	0
Fever of unknown origin	0	0	0	0	0	0	0	0
Environment
Arrival during shift change/off hours	4	17	1	25	2	25	1	9
Environmental focus on sepsis Source of information	3	13	1	25	1	12	1	9
Family or caregiver concerns/anxiety	11	47	3	75	4	50	4	36
Nonphysician healthcare provider concerns	6	26	0	0	3	37	3	27
Strength of data	3	13	1	25	1	12	1	9
Personal trust/respect	4	17	1	25	1	12	2	18
Disparity between trainee and faculty impression	5	22	1	25	2	25	2	18
Cue clusters	16	70	1	25	6	75	9	82

Note. Freq = number of incidents in which a cue was identified; % = percentage of incidents in which cue was identified.

Classic indicators/consensus criteria

As expected, the classic indicators of sepsis appeared frequently in the incidents related by interviewees. These indicators are traditionally associated with sepsis and can be considered the “textbook signs” of sepsis; they are part of the expert consensus definition taught to every resident in pediatrics (Fleisher & Ludwig, 2010; Kliegman, Stanton, St. Geme, Schor, & Behrman, 2011). Signs of fever and tachycardia (heart rate faster than normal) or bradycardia (heart rate slower than normal) were the most frequent classic indicators, with other common signs of sepsis such as tachypnea (breathing faster than normal) or bradypnea (breathing slower than normal) and white blood cell count abnormalities appearing somewhat less frequently. Another way of thinking about classic indicators of sepsis is that often a numeric value can be assigned to the indicators in this category.

Classic indicators/experience-based criteria

Experience-based criteria require more judgment and skill to interpret clinical saliency. It has been noted that experts in “domains with high problem-solving and reasoning demands, such as physics and medicine, are more likely to identify the essential principles underlying particular cases than domain novices, whose representations tend to gravitate toward surface features” (Charness & Tuffiash, 2008, p. 429). Experience-based criteria typically require a clinical judgment rather than a numeric value. For example, a heart rate of 190 is clearly abnormal, clearly tachycardic. Abnormal mental status is more subtle and potentially depends on developmental status, time of day, medication hanging, or altered mental status. Abnormal mental status is a well-known cue for patients in shock, a condition where the body is attempting to compensate either for an illness or injury that has dropped blood supply below demand. For example, in traumatic hemorrhagic shock, blood loss beyond the body’s ability to compensate results in decreasing mental status as one symptom of shock. The prevalence of mental status changes in the reported incidents is an important finding because mental status changes can be an early sign of septic shock. Pediatric physicians in an academic setting are often interacting with patients they do not know well, so it is hard to know a child’s baseline mental status. Even if these physicians have a good sense of children at different stages of development, many children coming to the academic centers are not typical. Medications, congenital and chronic illnesses, and/or developmental delay can influence mental status. The assessment may take place in the middle of the night when one would expect a child to be sleepy. In fact, one novice physician described a congenital heart patient as “just a little intimidating.” He did not have a sense of what would be typical mental status for a child with this condition so he was not sure how to tell if something was terribly wrong. In his words, he did not know how to judge whether “she is really sick or . . . totally fine.” This statement reflects the difficulty of using a particular experience-based criterion for the novice physician.

Another experience-based cue that appeared frequently in the critical incidents was distal extremity perfusion (15 out of 23 incidents). Distal extremity perfusion is an important cue and one that every resident should know as Pediatric Advanced Life Support, a course required for all pediatric residents that references poor distal perfusion and delayed capillary refill as important indicators of shock (Chameides, Samson, Schexnayder, & Hazinski, 2012). Poor distal extremity perfusion appeared in 7 of 12 resident incidents (2 of 4 novice and 5 of 8 senior-level trainee) and in 8 of 11 faculty incidents. Across incidents, descriptions of poor perfusion varied considerably, including slow capillary refill, a thready pulse, or a child’s hands or feet feeling cold to the touch. In other cases, the interviewee mentioned a change in skin color, varyingly described as pale, yellow, mottled, flushed, or a reticulated pattern. This disparity in strategies for recognizing poor distal perfusion suggests this as an important perceptual and judgment skill and one that may require substantial experience in order to recognize subtle differences in patients of different age, skin tone, and physical condition. It is important that residents realize and acknowledge distal extremity perfusion as a key indicator of sepsis and develop the ability to recognize poor distal extremity perfusion across a range of patients. There are multiple descriptions and no single number that defines poor distal perfusion. Thus, it may require exposure to multiple patients with these findings before the physician may reliably state that a patient has poor distal perfusion. Table 4 provides a list of descriptors used to describe important changes in distal perfusion.

Table 4:

Descriptors of Distal Perfusion

Distal Perfusion Descriptors
Skin color PalePaleish grayPastyPallor (African American patient)YellowishNo nice flush on cheeksMottledFlushedPurpleReticulated pattern	Extremities Mottling, especially lower extremitiesHands were mottledExtremities were coldPale extremitiesNose was yellowish	Temperature ColdMottled and warmSweatingPerfusion was warm	Other Delayed cap refillDecreased peripheral perfusionPoor peripheral pulsesVasoconstrictedPulses were thready

A third experienced-based cue frequently mentioned was ill appearance. Although this is not a classic indicator of sepsis, pediatricians commonly refer to the ability to distinguish a critically ill or “sick” child from those with a less serious condition. Ill appearance was mentioned in 13 of the 23 incidents (3 of 4 novices, 4 of 8 senior-level trainees, and 6 of 11 experts). When probed for additional description, some interviewees reported an important cue was the patient looked sicker than expected given the current diagnosis, saying the patient “didn’t match what you expect a kid with strep [meaning Group A streptococcal pharyngitis or strep throat] to look like.” Another reported the patient “didn’t look like a normal kid with a fever.” Others remember visual cues such as glossy-eyed, limp, pale, or dry lips. Often these visual cues were in combination with indicators of mental status change such as lethargic, hypoactive, sleepy or lying on mom, head back, or eyes closed. The advantage for the physician who is able to use experience-based criteria is that presence or absence of these cues provides additional information and evidence concerning the diagnosis.

Risk factors for sepsis/medical history

In 15 of the 23 incidents related, the patient had a chronic medical illness. For example, in one case, the patient was a 4-month-old with congenital heart disease. Tachypnea or rapid breathing is one of the classic, easily recognized indicators of sepsis. However, as the infant often experienced periods of rapid breathing due to heart disease, it was difficult to recognize that his breathing was worsening or different than baseline. In other cases, the patient had developmental delays that made it difficult to assess changes in mental status. In incidents involving children on chemotherapy, otherwise straightforward cues such as fever or abnormal blood cell counts were easily overlooked because they could be explained away as a result of the underlying cancer or cancer treatment. In many of these cases, a worsening clinical condition in spite of intervention that should have led to improvement was an indication that something more than the chronic condition was occurring.

Source of information/family or caregiver concern

In nearly half of the incidents, interviewees mentioned family or caregiver concerns as an important cue. In three of four novice incidents and four of eight senior-level trainee incidents, the resident remembered concern expressed by parents as an important cue that a child was experiencing a change in mental status and that the child may be seriously ill. Parental or caregiver concern appeared in 4 of 11 incidents reported by experienced physicians. Recognizing this concern was described by some experts as a deliberate strategy; that is, when a parent uses phrases such as “he just isn’t himself,” it should be considered an important indicator.

Cue Utilization

The second analysis included an examination of the incidents for differences in cue utilization. For this analysis, we were interested in understanding how interviewees used the cues they mentioned to determine criticality, make sense of the situation, and interpret the clinical picture. One qualitative researcher reviewed transcripts and extracted excerpts from each describing cue utilization. Two qualitative researchers reviewed the excerpts independently and then met to discuss emergent themes. Novices (interns) and senior trainees (senior residents) were collapsed into one less-experienced group and compared to the experts (faculty). Two experience-level differences emerged: interpreting cues, and hypothesis generation and testing.

Interpreting cues

One striking difference in the incident analysis was the difference in the role of other team members in interpreting cues. For novices and senior-level trainees, the judgments of other team members were an important cue regarding the seriousness of a patient’s condition. Sometimes, concern voiced by other clinicians was the initial cue that led the novice or senior-level trainee to suspect sepsis. For example, as one interviewee indicated, “it [sepsis] was obvious to them and less obvious to me.” Another reported, “the attending [expert] . . . started pimping me about my assessment of, you know, sepsis in this kid . . . I remember feeling very confused and not really following his explanation.” A third interviewee said, “I think the rest of the team realized before I did that this was going to turn into a very bad night.” The narratives told by novices had detailed accounts of the patient but were generally expressed through the voice of an observer. Actions were recounted using the pronoun “we.”

Conversely, for experts, their own personal interpretation of the available cues drove the assessment of the patient’s condition. In incidents related by experts, the interviewees often described their own impression of criticality. For example, as one expert recounted, “[the resident] said the child was febrile and didn’t look very good and then I sort of popped my head in there and looked at the kid and thought, wow, this kid looks more than not good. This kid looks horrible.” Another remembered an incident in which the urgent care team had sent the child to the ED but did not seem to understand the criticality of the situation:

it was actually I who came out and said this kid doesn’t look right, like this kid is actually pretty sick. I think I came out and said I am actually worried about this person. . . . There wasn’t a push to get him out fast because I think that they didn’t see this as, “Oh, this is a really, really sick kid.”

In the incidents reported by experts, both assessment and actions tend to be preceded by the pronoun “I.” The assessment and actions of other team members were questioned, often because other team members did not seem to realize the seriousness or urgency of the situation. One expert interviewee recalled an incident in which a patient “definitely exhibited signs of sepsis and septic shock,” but these signs were “underappreciated because people hadn’t been very aggressive with his management and kept on attributing it to fever.” The difference in narrative style between the experts and novices was dramatic, as novices tended to rely on the reactions of others on the healthcare team to interpret the cues available. Experts, in contrast, described their own impressions, at times recognizing sepsis before others on the healthcare team realized the seriousness of the situation. This difference in pronoun use between novice and experts, in our opinion, does not necessarily reflect a devaluing of the team but more likely reflects the experience, confidence, and expertise of the experts. These accounts are consistent with expertise literature; this type of recognition has been described as a “gut feeling” (Klein, 2007; Van den Bruel, Thompson, Buntinx, & Mant, 2012).

Hypothesis generation and testing

In a number of the expert interviews, the interviewees described hypotheses that drove their information seeking. Although the hypothesis was in many cases imprecise, it was specific enough to guide problem framing and next steps. For example, in one incident a patient was experiencing seizures and the team was treating the child for seizures. The interviewee, however, suspected that there was more going on: “[I was] questioning, you know, did I want to put all my eggs in one basket of seizures when . . . there seems like there could be other stuff.” (Note: The interviewee clarified that “stuff” referred to bacterial or viral pneumonia, or possibly sepsis.) To test this hypothesis, the interviewee ordered neurological tests to better understand the cause of the seizures and also labs to provide information about the presence of infection, acidosis, and other potential indicators of sepsis. Often, the interviewee described actions taken to gather more information that would aid in ruling in, ruling out, or narrowing the differential diagnosis; that is, “Let’s give him some fluid and see what happens with his tachycardia.” Experts described violated expectancies: “Despite the antipyretics, despite the fluid administration . . . he didn’t have the degree of improvement I would expect.” In one case, the expert described a situation in which she guided a less experienced physician to consider an infection and order tests to confirm or disconfirm: “The fellow said [the patient was] febrile and tachycardic as a reaction to stress, being in the ICU. I said these kids are at risk for infection. Let’s at least send a blood culture.” Notably, these descriptions of hypothesis formation and testing were absent from the novice interviews. Novices clearly remembered individual cues, but not always what tests were ordered (i.e. “I don’t think we did a urine check” or “we might have had X-ray come over and shoot X-rays”), illustrating the change from judgment-based thinking to compliance-oriented responses.

In sum, experts described specific actions used to test their hypothesis that the patient (in this case) does indeed have sepsis. More than a general belief of something being wrong, the experts used specific tests and management strategies (for example, the patient’s response to fluid boluses) to evaluate, confirm, or refute their theory that the patient has sepsis. Conversely, the novices often did not recall thinking of sepsis as a possibility (except when other team members raise the possibility). In addition, novices often did not recall specific tests or strategies used to rule in or rule out sepsis. These differences suggest that although the novices were able to recognize the cues relevant to sepsis when recounting an incident, their descriptions of incidents did not provide evidence that they were actively interpreting those cues and forming and testing hypotheses in the same way the experts were.

Training Design

The cognitive task analysis findings influenced the design and development of simulation-based training. First, collecting a compilation of 23 real-world incidents involving sepsis provided the foundation to develop a set of scenarios that would be complementary yet present differing combinations of sepsis-related cues in a variety of contexts. The rich set of incidents and detailed discussion of cues in real-world contexts identified the “building block” cues relevant to sepsis that contemporary residents would experience rarely, if at all, during residency. In total, incidents described in the CDM interviews and themes, subthemes, and cues identified in the analysis drove the design of five scenarios. One “garden path” scenario was developed to present a seemingly straightforward case of hypovolemia (severe dehydration due to vomiting and diarrhea), but as the scenario unfolds, additional information is presented that is not consistent with the initial working diagnosis. This method of scenario design contrasts common practice in simulation where scenarios are often based on a single near miss or adverse event as articulated during a root cause analysis.

Second, the findings supported the notion that although realistic and challenging training scenarios can play a critical role in skill acquisition (Hoffman et al., 2014), this may not be sufficient. There may be additional benefits to placing the resident in a situation in which he or she is directly experiencing the cues and must rely on his or her own ability to interpret the cues, generate hypotheses, and determine how to confirm or disconfirm the diagnosis. This is consistent with the data/frame theory of sense-making, which purports that early hypothesis generation is advantageous in that it facilitates efficient information gathering and generation of expectancies that can be violated as new information becomes available (Klein, Moon, & Hoffman, 2006b). In more traditional scenarios, it has been common to provide the participant with findings from radiographs, ECGs, or distal perfusion by simply reporting the results. Based on the CDM interviews, we realized it was more valuable to present “raw” data as a method to support the development of perceptual skills and pattern recognition in the context of a challenging incident. For example, instead of radiographic findings provided telephonically, we showed the image to encourage, and force, the learner to glean a first-hand interpretation. Presenting these perceptual cues was not always straightforward. The classic indicators are relatively easy to incorporate into scenarios because they can easily be represented either by a single value or a trend over time, and the monitors used to measure and display these cues are generally a part of the simulation scenario. However, many of the cues identified in this project required innovative strategies such as placing the feet of the mannequin in ice prior to training to simulate cold extremities. Table 5 shows the methodological changes that resulted from these interviews and analysis, in contrast to more traditional methods.

Table 5:

Novel Methods to Present “Raw” Data to Participants in Simulation-Based Training

Historical, Physical Exam, Lab, or Radiographic Information	Traditional Method of Presentation in Simulation	Novel Approach Driven by Results of CDM Interviews
Changes in skin appearance, consistent with poor distal perfusion	Facilitator tells the participants, if they ask, that the skin appears mottled, pale, or blue	Image is shown to participants, forcing them to identify abnormality and identify significance
Delayed capillary refill, consistent with poor distal perfusion	Facilitator tells the participants, if they ask, that the capillary refills is “4 seconds”	Video of capillary refill is shown to participants, forcing them to identify abnormality and identify significance
Cool extremities, consistent with poor distal perfusion	Facilitator tells the participants, if they ask, that the feet are cool to touch	Prior to simulation the hands and feet of simulator are wrapped in bags of ice for 5-10 min. If participants do not feel them, they would not know they are cool/cold
Decreased mental status	Some simulators have ability to verbalize preprogrammed statements but are limited to one voice and those select comments	Incorporation of voice modulator allowing facilitator to speak as 2-year-old girl, 6-year-old boy, etc., providing improved fidelity and variety of responses
Blood work abnormalities, such as acidosis or hypoglycemia (low blood sugar)	Facilitator tells the participants, if they ask, the results of blood work drawn	Blood work shown as screen shot, forcing team to interpret results
Chest radiograph (X-ray)	Results, such as pneumonia or normal, told to participants	Image was shown to participants, forcing them to identify abnormality and identify significance. If formal reading was needed, appropriate time delay was introduced before radiologist would “read” the film.
Consultation from other healthcare providers or family	Facilitator and other simulation center staff often play these roles to provide information that the participants request	Scripted comments from support staff and family members were included, incorporating verbatim statements from the critical incidents. Scripted telephone responses from consulting physicians and radiology techs were embedded to increase realism while limiting the number of actors needed to conduct the simulation.

Third, the difference in use of the “I” pronoun by faculty as compared to the “we” pronoun more commonly used by residents inspired scenarios that would provide an opportunity for junior residents to rapidly respond to a potentially critically ill patient, rather than rely on the team leader. Our intent was to shift the perspective from a focus on information gathering and reporting skills to problem solving and action. Thus, we placed the resident in a leadership role within the safe environment of the simulation lab, but without senior support. Additionally, the incidents collected highlighted the importance of time in sepsis recognition. Several of the CDM incidents included situations in which a patient’s condition slowly deteriorated, making it easy to miss subtle sepsis cues. Oftentimes, simulation scenarios are focused on emergent crisis situations in which the patient has impending or actual cardiorespiratory failure, leading to a medical code within a few minutes. In order to include experiences that would be more faithful to real-world timing and urgency, we designed scenarios that are more representative of the timeline in which a situation is likely to unfold on the unit by designing time-based responses for lab, radiologic, and consultation information. For example, in actual patient care, a critical care consult or a blood gas result is not immediately available, and patients may deteriorate while waiting for those consults or results; thus in scenario design, we would force the resident to act while awaiting that information.

To illustrate, one scenario involves a 6-year-old boy with developmental delays who is recovering from surgery for a broken arm. Initially, the child has a normal temperature but develops labored breathing and an increased heart rate (tachycardia). His eyes are closed and he cries intermittently. He is slightly blue around his lips and chin. As the resident examines the child, he becomes agitated and inconsolable (represented by the facilitator using voice modulation software to mimic a crying 6-year-old). Distal perfusion worsens, as demonstrated by a video showing delayed capillary refill (filling of the skin after pressure is applied) and cool hands and feet (which were iced before the scenario was begun). As the scenario unfolds, the patient’s condition changes depending on the intervention performed by the pediatrician. However, even if the pediatrician intervenes quickly and/or appropriately, signs of sepsis become more evident (i.e., heart rate increases, blood pressure drops, respiratory effort becomes less organized and slows, and level of consciousness deteriorates). In this scenario, we incorporated a number of cues mentioned in critical incidents including a chronic medical illness (developmental delay), poor distal perfusion (visible as delayed capillary refill on video), and changes in activity level and mental status in addition to standard consensus criteria (tachycardia and bradypnea).

Discussion and Conclusions

Multiple examples of cognitive task analysis, and specifically the CDM, to identify novice and expert differences in a variety of domains exist. In healthcare, this method has been successfully used with acute care nurses, neonatal nurses, and residents rotating in an intensive care unit (Crandall & Getchell-Reiter, 1993; Fackler et al., 2009; Gazarian, Henneman, & Chandler, 2010; Militello & Lim, 1995). Despite the recognition of the differences in the novice’s and expert’s approach to complex and difficult clinical problems, there has been little research directed at understanding cues and cue utilization strategies for experts and novices in healthcare (for an exception, see Crandall & Getchell-Reiter, 1993). Nor has there been any organized attempt to exploit these differences in an effort to establish the development of clinical expertise (Crandall & Wears, 2008).

The first step in developing strategies to aid in the acquisition of expertise is defining the gaps that need to be addressed. In this project, CDM interviews identified cues that physicians use to assess and make decisions in the context of real-world incidents involving sepsis. Many of these cues are consistent with the consensus criteria for sepsis but are infused with experience-based criteria that require judgment and skill to discern, including mental status changes, ill appearance, and poor distal perfusion. We identified that residents appear to be aware of the same cues as experts, but important differences in the ways expert and novice physicians use these cues were noted, including their ability to understand the implications of those cues for hypothesis generation and testing. First, although novices reported many of the same cues, they tended to rely on other members of the healthcare team to interpret them. This may be appropriate given that residents are working under supervision and thus defer to the supervisor’s judgment. However, it also represents evidence for the concern that increased supervision may reduce the resident’s opportunities to form his or her own assessments, to build mental models, and practice sense making. Second, we found that hypothesis testing was an important component of expert narratives as they recounted incidents in which they deliberately took an action (i.e., administered fluids, order tests, etc.) with the intent of ruling out, ruling in, or deepening a hypothesis about the patient’s condition based on the outcome of the action. Descriptions of hypothesis testing were absent from the resident incident accounts. These examples of active hypothesis generation, followed by information seeking to confirm or disconfirm an understanding of the situation, are consistent with accounts of expert sense making in other domains (Klein et al., 2006a, 2006b). The identification of key factors that characterize and distinguish the expert from the novice in the recognition of sepsis at the bedside is expected to be significant because it is the first step in the development of an approach to facilitate expertise acquisition in the recognition of sepsis, a condition where early resuscitation and reversal of shock is imperative to survival.

The resulting critical cue inventory, detailed examples, and contextual elements provide the foundation for creating realistic scenarios to be used in simulation-based training. The expert novice differences highlight important training goals for resident physicians. One goal is to provide simulation-based training that creates an opportunity for novice physicians to go beyond noting and reporting important cues in a supervised setting to actually forming assessments, taking actions, and observing outcomes for simulated patients. A second goal is to provide opportunities for novice physicians to rapidly develop hypotheses and seek confirming and disconfirming information even as they treat the patient. These emergent training goals are a deviation from traditional medical training. Conventionally, education of medical students and residents uses a clinical reasoning model and focuses on a differential diagnosis based on a complete history, physical examination, and supporting data. This is a rational choice model or analytic process method (Ark, Brooks, & Eva, 2006; Klein, 1998). Analytic processes are those that entail controlled, systematic consideration of all features and their relation to potential diagnoses. This method of clinical reasoning has traditionally been sanctioned by healthcare educators responsible for teaching medical students and residents. An unintended consequence of this emphasis is the diminished exposure to the situations that would provide novices with the opportunity to develop the skills associated with recognition primed decision making. There is also evidence suggesting that medical experts that use these recognitional (rather than rational choice) methods are more successful and more efficient in elucidating correct diagnoses than novices (Coderre, Mandin, Harasym, & Fick, 2003; Nendaz et al., 2006; Schmidt & Rikers, 2007).

Furthermore, these findings may provide some support for the notion that resident work hour restrictions have contributed to unanticipated patient safety threats, creating an urgent need to develop training that accelerates the acquisition of expertise in multiple clinical domains. These skills can only be obtained via first-hand experience. Acquisition of these skills is hindered by the current resident training environment in which residents have diminished exposure to critical incidents, reduced opportunity to follow a case from beginning to end, and are generally in a supervised, supporting role rather than serving as decision maker. The traditional method of clinical learning, moving from history to physical exam to assessment and then presentation to a senior physician, is relatively linear; thus, creating multiple intense and time-based sepsis scenarios would allow residents to practice a different method of critical judgment, aid development of autonomous decision-making skills, and help reinforce the actual clinical context in which they may occur.

Our approach to addressing unintended consequences of resident work hour restrictions and improving outcomes from sepsis was to design and implement simulation-based training that is grounded in the ways expert physicians notice and use clinical cues. This training needed to include multiple scenarios (bundled examples versus a single example) with widely varying cues and clinical contexts, as demonstrated in the CDM interviews. The training was aimed at increasing the novice clinician’s exposure to patients with sepsis and also focusing on the critical strategies practiced by experts when evaluating and managing patients with possible sepsis. The potential positive impact of this training is considerable. Earlier recognition of sepsis by novice clinicians, those who evaluate the patient first, should lead to earlier initiation of critical therapies and a decrease in sepsis-related morbidity and mortality. Our next step is to pilot the developed simulations within a cohort of pediatric intern (novice), senior-level trainee, and faculty (expert) physicians to assess recognition of sepsis and early goal-directed management across groups. This will allow us to build on findings from the CDM interviews and improve the validity and reliability of the educational intervention, as well as the proposed outcomes measures. Following this pilot, and any needed revisions, we will implement this as a formal course for the pediatric intern class (approximately 60 novice physicians) in an effort to accelerate their acquisition of expertise in sepsis recognition.

This project represents one of the first efforts in health care to explicitly characterize the differences in novice and expert clinicians in cue recognition and utilization related to sepsis. To our knowledge, this is also the first time that these methods were used in health care with the specific aim of developing simulation-based training to increase exposure and aid in developing expertise in a particular clinical domain. The use of cognitive task analysis methods and the development of simulations to delineate the novice’s divergence from expert behavior enabled the creation of training designed to develop expertise in this clinical domain. The strategy of defining novice expert differences promises to serve as the basis for acquiring expertise in multiple clinical domains.

Although this approach is initially focused on the recognition of sepsis, it is likely to be transferable to other critical, time-dependent medical conditions such as myocardial infarction and pulmonary embolism. The limitations in resident work hours and other regulatory changes require that we identify effective methods to facilitate the development of clinical expertise if we are to ensure competency in graduating residents. Clinical teachers should abandon the mythical ideal of the clinician as an objective impassive observer and instead should encourage learners at all levels to use their experience:

We have tended to discount the experiential component of clinical expertise, dismissing it as mere pattern recognition and disparaging experts who are guided by experience. . . . Our current understanding of medical expertise suggests that this bias is misguided; a critical element of becoming an expert is accruing the vast experience that enables experts to recognize patterns effortlessly most of the time—and to recognize, as well, when the signs and symptoms do not fit a pattern at all. (Norman, 2006, p. 2252)

It is often much more difficult to definitively assess a change in mental status, yet clinicians use this frequently as an early indication of inadequate brain perfusion. Change in a patient’s mental status was the most frequently mentioned experience-based cue by all levels of clinicians. Across all incidents collected, mental status changes were described by interviewees in 19 of the 23 cases discussed during the study interviews: 3 of 4 novice incidents, 8 of 8 senior-level trainee incidents, and 8 of 11 expert incidents.

Researchers independently coded transcripts then met to reach consensus on disparities. Cue frequencies were tabulated across interviews and by participant class. Classic sepsis indicators were identified frequently by participants, as well as experience-based criteria—that is, mental status changes. Findings suggest novices relied on others’ judgments to assist them in interpreting cues; experts relied on their own interpretations.

Footnotes

Acknowledgements

We would like to thank Christen Sushereba for her support in conducting the qualitative analysis. This research was funded by the Agency for Healthcare Research and Quality (Grant 1R18HS020455-01).

Mary D. Patterson, MD, MEd, is a pediatric emergency physician at Cincinnati Children’s Hospital. Previously she was the medical director of simulation at Akron Children’s Hospital and Cincinnati Children’s Center for Simulation and Research. She is a past president of the Society for Simulation in Healthcare. Her research interests are related to the use of healthcare simulation to study systems and teamwork and improve patient safety.

Laura G. Militello, MS, is cofounder and senior scientist at Applied Decision Science, LLC. She has been studying decision making in complex settings for over 20 years. Her research interests include the impact of electronic health records on clinical decision making, strategies for supporting expertise via technology design and training, and solving real-world problems.

Amy Bunger, PhD, is the assistant designated official at the University of Cincinnati Medical Center and faculty member in medical education in the College of Medicine. She serves as principal investigator for a hospital-wide baseline assessment of residents and supervises over 650 residents. Her research interests include patient safety, quality improvement, and transitions of care.

Regina G. Taylor, MA, CCRP, is a senior clinical research coordinator at Cincinnati Children’s Hospital Medical Center. She serves as a project manager and oversees all research endeavors for the Center for Simulation and Research. Her areas of expertise include research in emergency settings, the consent process, research involving vulnerable populations, and educational research. She has been active in clinical research since 2004 and is a certified clinical research professional and certified project manager.

Derek S. Wheeler, MD, FAAP, FCCP, FCCM, is an associate professor of clinical pediatrics at the University of Cincinnati College of Medicine and associate chief of staff at Cincinnati Children’s Hospital Medical Center. He is an internationally recognized expert in pediatric critical care medicine, quality improvement, and patient safety. His clinical/research interests include the early recognition of clinical deterioration and shock, rapid response systems, and high reliability organization theory.

Gary Klein, PhD, is a senior scientist at MacroCognition LLC. He received his PhD in experimental psychology from the University of Pittsburgh in 1969. He was an assistant professor of psychology at Oakland University (1970-1974) and a research psychologist for the U.S. Air Force (1974-1978) and founded Klein Associates (in 1978). His books include Sources of Power: How People Make Decisions (1998) and Seeing What Others Don’t: The Remarkable Ways We Gain Insights (2013).

Gary L. Geis, MD, is an associate professor of pediatrics at Cincinnati Children’s in the Division of Emergency Medicine as well as the medical director for the Center for Simulation and Research. He has been active in simulation-based medical education and training since 2005, with special interests in teamwork’s impact on resuscitative care, procedural training, and utilization of simulation to implement new teams, care environments, and care processes.

References

Accrediation Council for Graduate Medical Education. (2007). Common program requirements: Duty hour requirements. Retrieved from http://www.acgme.org/acWebsite/dutyHours/dh_ComProgrRequirmentsDutyHours0707.pdf

Angus

D. C.

Linde-Zwirble

W. T.

Lidicker

Clermont

Carcillo

Pinsky

M. R.

(2001). Epidemiology of severe sepsis in the United States: Analysis of incidence, outcome, and associated costs of care. Critical Care Medicine, 29(7), 1303-1310.

Angus

D. C.

Wax

R. S.

(2001). Epidemiology of sepsis: An update. Critical Care Medicine, 29(7), S109-S116.

Argyris

(1991). Teaching smart people how to learn. Reflections, 4(2), 4-15.

Argyris

Schön

(1978). Organizational learning: A theory of action perspective. Reading, MA: Addison Wesley.

Ark

T. K.

Brooks

L. R.

Eva

K. W.

(2006). Giving learners the best of both worlds: Do clinical teachers need to guard against teaching pattern recognition to novices? Academic Medicine, 81(4), 405-409.

Bigham

M. T.

Amato

Bondurrant

Fridriksson

Krawczeski

C. D.

Raake

. . . Brilli

R. J.

(2009). Ventilator-associated pneumonia in the pediatric intensive care unit: Characterizing the problem and implementing a sustainable solution. The Journal of Pediatrics, 154(4), 582-587.

Bowen

J. L.

(2006). Educational strategies to promote clinical diagnostic reasoning. New England Journal of Medicine, 355(21), 2217-2225.

Brilli

R. J.

Sparling

K. W.

Lake

M. R.

Butcher

Myers

S. S.

Clark

M. D.

Helpling

Stutler

M. E.

(2008). The business case for preventing ventilator-associated pneumonia in pediatric intensive care unit patients. Joint Commission Journal on Quality and Patient Safety, 34(11), 629-638.

10.

Centers for Disease Control and Prevention/National Center for Health Statistics. (2000-2008). National hospital discharge survey. Hyattsville, MD: Author.

11.

Chameides

Samson

R. A.

Schexnayder

S. M.

Hazinski

M. F.

(2012). Pediatric advanced life support: Provider manual.

12.

Chang

R. S.

Hamilton

R. J.

Carter

W. A.

(1998). Declining rate of cricothyrotomy in trauma patients with an emergency medicine residency: Implications for skills training. Academic Emergency Medicine, 5(3), 247-251.

13.

Charness

Tuffiash

(2008). The role of expertise research and human factors in capturing, explaining, and producing superior performance. Human Factors, 50(3), 427-432.

14.

Coderre

Mandin

Harasym

P. H.

Fick

G. H.

(2003). Diagnostic reasoning strategies and diagnostic success. Medical Education, 37(8), 695-703.

15.

Crandall

Getchell-Reiter

(1993). Critical decision method: A technique for eliciting concrete assessment indicators from the intuition of NICU nurses. Advances in Nursing Science, 16(1), 42-51.

16.

Crandall

Klein

G. A.

Hoffman

R. R.

(2006). Working minds: A practitioner’s guide to cognitive task analysis. Cambridge, MA: MIT Press.

17.

Crandall

Wears

R. L.

(2008). Expanding perspectives on misdiagnosis. American Journal of Medicine, 121(5 Suppl), S30-S33.

18.

Ericsson

K. A.

(2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79(10 Suppl), S70-S81.

19.

Ericsson

K. A.

Whyte

Ward

(2007). Expert performance in nursing: Reviewing research on expertise in nursing within the framework of the expert-performance approach. ANS Advances in Nursing Science, 30(1), E58-E71.

20.

Fackler

J. C.

Watts

Grome

Miller

Crandall

Pronovost

(2009). Critical care physician cognitive task analysis: An exploratory study. Critical Care, 13(2), R33.

21.

Fazio

R. F.

Wheeler

D. S.

Poss

W. B.

(2000). Resident training in pediatric critical care transport medicine: A survey of pediatric residency programs. Pediatric Emergency Care, 16(3), 166-169.

22.

Fleisher

G. R.

Ludwig

(2010). Textbook of pediatric emergency medicine (6th ed.). Philadelphia: Lippincott, Williams & Wilkins.

23.

Gaba

D. M.

(2004). The future vision of simulation in health care. Quality and Safety in Health Care, 13(suppl 1), i2-i10.

24.

Gazarian

P. K.

Henneman

E. A.

Chandler

G. E.

(2010). Nurse decision making in the prearrest period. Clinical Nursing Research, 19(1), 21-37.

25.

Greenhow

T. L.

Hung

Y. Y.

Herz

A. M.

(2012). Changing epidemiology of bacteremia in infants aged 1 week to 3 months. Pediatrics, 129(3), e590-e596.

26.

Han

Y. Y.

Carcillo

J. A.

Dragotta

M. A.

Bills

D. M.

Watson

R. S.

Westerman

M. E.

Orr

R. A.

(2003). Early reversal of pediatric-neonatal septic shock by community physicians is associated with improved outcome. Pediatrics, 112(4), 793-799.

27.

Hänsel

Winkelmann

A. M.

Hardt

Gijselaers

Hacker

Stiehl

Koch

Müller

M. P.

(2012). Impact of simulator training and crew resource management training on final-year medical students’ performance in sepsis resuscitation: a randomized trial. Minerva Anesthesiology, 78(8), 901-909.

28.

Hartman

M. E.

Linde-Zwirble

W. T.

Angus

D. C.

Watson

R. S.

(2013). Trends in the epidemiology of pediatric severe sepsis. Pediatric Critical Care Medicine, 14(7), 686-693.

29.

Hoffman

R. R.

Ward

Feltovich

P. J.

DiBello

Fiore

S. M.

Andrews

D. H.

(2014). Accelerated expertise: Training for high proficiency in a complex world. New York: Psychology Press, Taylor & Francis Group.

30.

Hutter

M. M.

Kellogg

K. C.

Ferguson

C. M.

Abbott

W. M.

Warshaw

A. L.

(2006). The impact of the 80-hour resident workweek on surgical residents and attending surgeons. Annals of Surgery, 243(6), 864.

31.

Issenberg

S. B.

McGaghie

W. C.

Petrusa

E. R.

Gordon

D. L.

Scalese

R. J.

(2005). Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review. Medical Teacher, 27(1), 10-28.

32.

Kerlin

M. P.

Halpern

S. D.

(2012). Twenty-four-hour intensivist staffing in teaching hospitals: Tensions between safety today and safety tomorrow. CHEST Journal, 141(5), 1315-1320.

33.

Klein

(1998). Sourcs of power: How people make decisions. Cambridge, MA: MIT Press.

34.

Klein

(2007). The power of intuition: How to use your gut feelings to make better decisions at work. New York: Random House.

35.

Klein

Moon

B. M.

Hoffman

R. R.

(2006a). Making sense of sensemaking 1: Alternative perspectives. IEEE Intelligent Systems, 21(4), 70-73.

36.

Klein

Moon

Hoffman

R. R.

(2006b). Making sense of sensemaking 2: A macrocognitive model. Intelligent Systems IEEE, 21(5), 88-92.

37.

Kliegman

R. M.

Stanton

St. Geme

J. W.

Schor

N. F.

Behrman

R. E.

(2011). Nelson textbook of pediatrics (19th ed.). Philadelphia: Elsevier Saunders.

38.

Kline-Krammes

Wheeler

D. S.

Schwartz

H. P.

Forbes

Bigham

M. T.

(2012). Missed opportunities during pediatric residency training: Report of a 10-year follow-up survey in critical care transport medicine. Pediatric Emergency Care, 28(1), 1-5.

39.

Kumar

Taneja

Kaleekal

Tarima

McGinley

. . . Nachal

(2011). Nationwide trends of severe sepsis in the 21st century (2000-2007). Chest Journal, 140(5), 1223-1231.

40.

Laupland

K. B.

Gregson

D. B.

Vanderkooi

O. G.

Ross

Kellner

J. D.

(2009). The changing burden of pediatric bloodstream infections in Calgary, Canada, 2000–2006. The Pediatric Infectious Disease Journal, 28(2), 114-117.

41.

Lewis

Strachan

Smith

M. M.

(2012). Is high fidelity simulation the most effective method for the development of non-technical skills in nursing? A review of the current evidence. The Open Nursing Journal, 6, 82.

42.

Lin

C. Y.

Hsu

C. H.

Huang

F. Y.

Chang

J. H.

Hung

H. Y.

Kao

H. A.

. . . Chen

C. P.

(2011). The changing face of early-onset neonatal sepsis after the implementation of a maternal group B Streptococcus screening and intrapartum prophylaxis policy—A study in one medical center. Pediatrics & Neonatology, 52(2), 78-84.

43.

Lundberg

J. S.

Perl

T. M.

Wiblin

Costigan

M. D.

Dawson

Nettleman

M. D.

Wenzel

R. P.

(1998). Septic shock: An analysis of outcomes for patients with onset on hospital wards versus intensive care units. Critical Care Medicine, 26(6), 1020-1024.

44.

McGaghie

W. C.

Issenberg

S. B.

Petrusa

E. R.

Scalese

R. J.

(2010). A critical review of simulation-based medical education research: 2003–2009. Medical Education, 44(1), 50-63.

45.

Merton

R. K.

(2008). The unanticipated consequences of purposive social action. American Sociological Review, 1(6), 895.

46.

Militello

Lim

(1995). Patient assessment skills: Assessing early cues of necrotizing enterocolitis. The Journal of Perinatal & Neonatal Nursing, 9(2), 42-52.

47.

Nendaz

M. R.

Gut

A. M.

Perrier

Louis-Simonet

Blondon-Choa

Herrmann

F. R.

Junod

A. F.

N. V.

(2006). Brief report: Beyond clinical experience: Features of data collection and interpretation that contribute to diagnostic accuracy. Journal of General Internal Medicine, 21(12), 1302-1305.

48.

Nishisaki

Hales

Biagas

Cheifetz

Corriveau

Garber

. . . Nadkarni

(2009). A multi-institutional high-fidelity simulation “boot camp” orientation and training program for first year pediatric critical care fellows. Pediatric Critical Care Medicine, 10(2), 157-162.

49.

Norman

(2006). Building on experience-the development of clinical reasoning. New England Journal of Medicine, 355(21), 2251-2252.

50.

Norman

Young

Brooks

(2007). Non-analytical models of clinical reasoning: The role of experience. Medical Education, 41(12), 1140-1145.

51.

Nowak

J. E.

Brilli

R. J.

Lake

M. R.

Sparling

K. W.

Butcher

Schulte

Wheeler

D. S.

(2010). Reducing catheter-associated bloodstream infections in the pediatric intensive care unit: Business case for quality improvement. Pediatric Critical Care Medicine, 11(5), 579-587.

52.

Odetola

F. O.

Rosenberg

A. L.

Davis

M. M.

Clark

S. J.

Dechert

R. E.

Shanley

T. P.

(2008). Do outcomes vary according to the source of admission to the pediatric intensive care unit?*. Pediatric Critical Care Medicine, 9(1), 20-25.

53.

Okuda

Bryson

E. O.

DeMaria

Jacobson

Quinones

Shen

Levine

A. I.

(2009). The utility of simulation in medical education: What is the evidence? Mount Sinai Journal of Medicine: A Journal of Translational and Personalized Medicine, 76(4), 330-343.

54.

Owen

Mugford

Follows

Plummer

J. L.

(2006). Comparison of three simulation-based training methods for management of medical emergencies. Resuscitation, 71(2), 204-211.

55.

Reischman

R. R.

Yarandi

H. N.

(2002). Critical care cardiovascular nurse expert and novice diagnostic cue utilization. Journal of Advanced Nursing, 39(1), 24-34.

56.

Riley

Basu

R. K.

Kissoon

Wheeler

D. S.

(2012). Pediatric sepsis: Preparing for the future against a global scourge. Current Infectious Disease Reports, 14(5), 503-511.

57.

Riley

Wheeler

D. S.

(2012). Prevention of sepsis in children: A new paradigm for public policy. Critical Care Research and Practice, Volume 2012, 1-8. doi:10.1155/2012/437139.

58.

Schmidt

H. G.

Rikers

R. M.

(2007). How expertise develops in medicine: Knowledge encapsulation and illness script formation. Medical Education, 41(12), 1133-1139.

59.

Van den Bruel

Thompson

Buntinx

Mant

(2012). Clinicians’ gut feeling about serious infections in children: Observational study. British Medical Journal, 345, e6144.

60.

Weaver

S. J.

Salas

Lyons

Lazzara

E. H.

Rosen

M. A.

DiazGranados

. . . King

(2010). Simulation-based team training at the sharp end: A qualitative study of simulation-based team training design, implementation, and evaluation in healthcare. Journal of Emergencies, Trauma and Shock, 3(4), 369-377.

61.

Weiss

S. L.

Fitzgerald

J. C.

Pappachan

Wheeler

Jaramillo-Bustamante

J. C.

Salloo

. . . Thomas

N. J.

(2015). Global epidemiology of pediatric severe sepsis: The sepsis prevalence, outcomes, and therapies study. American Journal of Respiratory Critical Care Medicine, 191(10), 1147-1157.

62.

Wheeler

D. S.

Clapp

C. R.

Poss

W. B.

(2003). Training in pediatric critical care medicine: A survey of pediatric residency training programs. Pediatric Emergency Care, 19(1), 1-5.

63.

Wheeler

D. S.

Sperring

J. L.

Vaux

K. K.

Poss

W. B.

(1999). Development of a pediatric critical care transport team: Experience at a military medical center. Military Medicine, 164(3), 188-193.

64.

Wheeler

D. S.

Vaux

K. K.

Starr

S. R.

Poss

W. B.

(2000). The pediatric critical care experience at Naval Hospital Guam: Suggestions for critical care training during residency. Military Medicine, 165(6), 441-444.

65.

Wood

K. A.

Angus

D. C.

(2004). Genetic epidemiology of sepsis and septic shock. Seminars in Respiratory and Critical Care Medicine, 25(6), 611-618.

66.

Yildirim

Stevenson

Hsu

K. K.

Pelton

S. I.

(2012). Evolving picture of invasive pneumococcal disease in Massachusetts children: A comparison of disease in 2007–2009 with earlier periods. The Pediatric Infectious Disease Journal, 31(10), 1016-1021.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.11 MB

0.10 MB