Using “Mystery Patient” Drills to Assess Hospital Ebola Preparedness in New York City,2014-2015

Abstract

In response to the Ebola virus disease (EVD) outbreak in West Africa, rapid measures were taken to ensure readiness at frontline New York City (NYC) healthcare facilities, including mandating monthly EVD mystery patient drills to test screening protocols. This study analyzed after-action reports to describe the use of mystery patient drills to test rapid identification and isolation of potential EVD cases in NYC emergency departments. NYC hospitals were required to develop protocols for EVD screening and isolation, and to conduct drills with an actor presenting to the emergency department with symptoms suggestive of EVD. Fifty-five hospitals that participate in NYC's hospital preparedness program were invited to submit after-action reports summarizing at least 1 drill conducted between October 2014 and April 2015. Summary statistics were generated from reported quantitative measures. Report narratives were reviewed, coded, extracted, and analyzed to identify strengths and challenges experienced. Forty-five hospitals submitted after-action reports (82%). The median time from patient entry to isolation was 9 minutes and from isolation to evaluation was 14 minutes. Recurrent strengths included consistent travel history screening and compliance with infection control protocols. Themes for improvement included ensuring timely screening, staff competency with personal protective equipment (PPE), and clarifying notification procedures and staff roles. Mystery patient drills gave hospitals the means to test screening and isolation protocols and identify key gaps, such as competency-based training in PPE, to improve their capacity to respond to highly communicable diseases. Findings from this study will inform the development of a standardized mystery patient drill program.

This study analyzed after-action reports describing the use of mystery patient drills to test rapid identification and isolation of potential EVD cases in NYC emergency departments. The drills gave hospitals the means to test screening and isolation protocols and identify key gaps, such as competency-based training in PPE, to improve their capacity to respond to highly communicable diseases.

Since March 2014, more than 28,000 probable and suspected cases of Ebola virus disease (EVD) have been reported in West Africa, with more than 11,000 deaths.¹ Concern for potential imported Ebola cases was realized in September 2014 when EVD was diagnosed in a man who had recently traveled from Liberia to Dallas, Texas. This case highlighted the risk for more imported cases, particularly in New York City (NYC), which is a major transit point for travelers coming from the Ebola-affected countries. During a 1-month period from October 11 to November 10, 2014, it was estimated that 47% of travelers arriving from West Africa entered the United States through JFK Airport in NYC.²

In order to prepare the healthcare system for EVD, the New York State health commissioner issued a special order on October 16, 2014, mandating that all general hospitals develop written “patient registration protocols for the immediate identification, isolation, and medical evaluation of patients at risk for EVD.” In addition, all hospitals were required to implement monthly screening and isolation drills to test their protocols and maintain staff readiness for EVD starting in late October.³

Although there is no consensus in the literature on the best methods by which to assess hospital readiness, drills or exercises along with checklists to evaluate hospital performance have been shown to validate preparedness.^4-7 Similar drills have been performed for other infectious agents, including avian influenza,^8-11 inhalational anthrax,¹² smallpox,¹³ and even a hypothetical novel infectious disease.⁵ A study by Adini and colleagues demonstrated that such drills were useful in evaluating the effectiveness of hospital protocols for avian influenza.⁸

In this study, we describe the use of “mystery patient drills” as a tool to evaluate hospital screening and isolation protocols and identify common challenges encountered by NYC emergency departments (EDs) preparing to respond to EVD and other communicable diseases of public health concern.

Methods

Developing Protocols and Conducting Drills

Starting in late October 2014, all hospitals in New York State were required to conduct monthly drills to test patient registration protocols and to evaluate staff compliance with screening and isolation procedures.³ Hospitals were expected to rapidly implement and train emergency department staff on their EVD patient registration and isolation protocols within 5 days of the commissioner's order, released on October 16, 2014. New York State required hospitals to “conduct drills with personnel on all shifts on the patient registration protocol initially within 5 days of receipt of the Order and then every month, following the initial drill. A written description of the drill, including the items reviewed, number of staff included in the drill, gaps identified, conclusions, and next steps, must be maintained and made available to the New York State Department of Health upon request.”³

The NYC Department of Health and Mental Hygiene (NYC DOHMH) provided hospitals with “Guidance for the Management of Patients Presenting to Emergency Departments with Communicable Diseases of Public Health Concern.”¹⁴ This document was initially developed in 2004 and updated and released to NYC hospitals on October 6, 2014. The document aimed to assist hospitals in developing and/or updating existing screening and isolation protocols with a focus on initial patient identification, infection control measures, notification, and patient evaluation, along with the identification and management of people potentially exposed in emergency departments. In order to identify patients with potential communicable disease of public health concern, the general guidance recommended that hospitals screen all patients presenting to the ED for fevers within the past 2 weeks. If a patient reported fevers, he or she should be screened for rash, travel within the past 2 weeks to an area experiencing or at risk for an outbreak of public health concern, and respiratory symptoms with epidemiologic risk factors. Epidemiologic risk factors include healthcare-related exposures or exposure to a cluster of 2 or more persons with similar symptoms.

To support hospitals in their efforts to prepare to identify, isolate, and safely treat patients with possible Ebola virus disease, NYC DOHMH encouraged hospitals to use the previously released guidance document to develop their Ebola screening and registration protocols. Because of the longer incubation period for Ebola virus disease, hospitals were instructed to screen for fever or travel within the past 21 days. NYC DOHMH then used existing Hospital Preparedness Program (HPP) funding from the Assistant Secretary for Preparedness and Response (ASPR) to support NYC hospitals for training triage staff and other personnel, to verify the competence of core clinical staff, and to conduct at least 1 unannounced drill using a “mystery patient.” Of the 55 hospitals that participate in NYC's HPP program, 45 chose to use HPP funding to complete this activity (see Figure 1). Required documentation for the mystery patient drill included:

Figure 1.

Mystery Patient Drill Process Flow

• date of drill

• time from entry to triage

• time from triage to isolation

• time from isolation to initial evaluation by clinician (nurse or doctor)

• lessons learned

• recommendations for improvement

Participating hospitals submitted after-action reports from at least 1 drill conducted between August 1, 2014, and May 29, 2015. Some hospitals chose to submit after-action reports from more than 1 drill. Some hospitals submitted after-action reports that did not meet all of the above-listed requirements and were only partially reimbursed for the activity. Hospitals were solely responsible for all elements of drill planning, conduct, and evaluation. Drill scenarios were developed based on the needs of the facility but were required to include a patient arriving at the emergency department with a history suggestive of EVD.

Analysis

NYC DOHMH analyzed quantitative and qualitative data from all after-action reports submitted for this project. Data gathered from the respective time stamps were extracted from the reports and used to determine median time from initial patient entry to isolation and median time from isolation to evaluation.

We then performed a qualitative analysis of all submitted after-action reports to identify strengths and challenges around a set of predetermined thematic elements including:

• patient screening and triage protocols

• isolation and infection control

• personal protective equipment (PPE) availability or use

• notification between first receivers in the ED and the hospital incident command staff

• communication concerns (between hospital staff or with the patient)

• setting-specific issues

Based on these elements, a codebook was developed, and any documentation reflective of 1 of these elements was classified as a strength or weakness.

Results

Of the 55 hospitals participating in NYC's Hospital Preparedness Program, 45 (82%) submitted documentation for at least 1 drill conducted from November 2014 to April 2015. Some submitted multiple reports, so a total of 82 mystery patient drill after-action reports were reviewed. Fifty-five included narrative comments and were included in the qualitative analysis. Forty-one of the participating hospitals (91%) submitted at least 1 after-action report that documented all the required time measures (time from entry to triage; time from triage to isolation; time from isolation to initial clinical evaluation), and 79 after-action reports included at least 1 of these times and were included in the quantitative analysis (Table 1). Of those, the median time from patient entry to triage was 2 minutes (IQR 1-5 min), and the median time from triage to isolation was 5 minutes (IQR 2-8.2 min), making the median time from patient entry to the patient's being placed in isolation 8.5 minutes (IQR 3.5-12.6 min).

Table 1.

Key Time Measures Reported in After-Action Reports

Time Measure	Frequency Reported	Median Time (minutes)	Interquartile Range (minutes)
Time from entry to triage	79	2	1-5
Time from triage to isolation	78	5	2-8.3
Time from entry to isolation	78	8.5	3.5-12.6
Time from isolation to clinical evaluation	47	15	7-25

Qualitative analysis identified key strengths and challenges across the 6 thematic elements. These findings are summarized below, and additional illustrative quotes from after-action reports are listed in Table 2.

Table 2.

Illustrative Quotes from Drill Reports Related to Key Thematic Elements^a,b

Direct Quotes from the After-Action Reports
Screening/Triage
Strengths	• “Triage nurse asked patient screening questions and once identified … travel to Sierra Leone the nurse immediately escorted patient outside triage area and supplied the patient with mask and hand hygiene.”
	• “Staff used Ebola screening tool including travel.”
	• “Reception/registration staff immediately asked the appropriate travel screening questions.”
	• “Security began the incident by recognizing the suspect EVD patient using the screening protocol and reported the incident to responsible leadership”
Challenges	• “Patient picked up notification phone outside waiting room entrance to notify nurse that she had high fever and stomach pains. Directed to sit in waiting room.”
	• “Upon entering ED patient went to security where he was asked to sign in and wait to be called—delay in treatment and increased exposure.”
	• “Attending MD met with patient; did not have or know the 6 questions to ask the patient.”
Infection Control
Strengths	• “Isolation room set up and patient moved via shortest route.”
	• “Greeter gave nurse a mask.”
	• “Nursing staff moves all other patients away from vicinity.”
	• “Patient was provided with a mask, and the triage RN donned a mask, and patient with mask on was moved from the triage area to the private room with a dedicated bathroom.”
	• “Prompt isolation of patient in a single room with bathroom with appropriate signage with companion.”
	• “Gatekeeper assigned to limit room access and who maintained a log of all contact persons.”
	• “Nurse assistant designated to role of gatekeeper/log enter and exit for only those providing patient care.”
	• “The environmental services staff managed the removal and transfer of EVD waste effectively to designated storage area and followed all cleaning protocols for isolation room.”
Challenges	• “Another patient was in isolation room and not vacated, so Ebola patient was placed in resuscitation room on a wheelchair.”
	• “Initially, RN did not mask or glove prior to contact with the patient.”
	• “The RN donned gloves, the MD did not.”
	• “Staff on numerous occasions failed to maintain a safe 3-foot corridor from the patient.”
	• “Greeter nurse left the patient, walked off contaminating the rest of the ED and never triaged the patient.”
	• “Waiting room hand sanitizers require touch.”
	• “No sanitizer dispenser in the isolation rooms.”
	• “Bleach wipes in container inside cabinet—not readily available.”
	• “Counter area in pre-triage was not disinfected with bleach post patient exposure.”
	• “Patient care assistant ‘door guard’ placed late and did not know reason for being stationed.”
Personal Protective Equipment (PPE)
Strengths	• “The door was kept closed and staff members used proper PPE when entering the room. Staff appropriately allocated a cart containing all necessary PPE.”
	• “PPE readily available in triage and isolation room.”
	• “PPE was set up on the cart outside the isolation room.”
	• “Just-in-time training for donning and doffing with buddy system and checklist.”
	• “Appropriate donning and doffing of ED nurse, ED MD, security guard, and housekeeper.”
	• “Front desk RN worked with observer to don PPE.”
	• “Staff donned PPE under the supervision of the charge nurse; charge nurse provided some coaching.”
	• “Staff worked as a team in donning and doffing with buddy system.”
	• “Upon completion of care, PPE was successfully removed. Donning and doffing was supervised by a donning/doffing coach, who provided instructions for each step of the donning/doffing process.”
Challenges	• “Greeter nurse did not don PPE.”
	• “No staff exhibited proficient competency when donning and doffing their PPE.”
	• “PPE box was inside isolation room instead of outside for donning and doffing.”
	• “EVD cart not properly labeled and broken lock. Most EVD items missing.”
	• “Checklist in PPE cart was outdated.”
	• “There was no bleach in the doffing room.”
	• “Donning and doffing instruction sheets were on wall far from ante room.”
	• “Chairs for doffing PPE not properly placed.”
	• “Buddy system not used for donning PPE.”
	• “Triage nurse was rushing to put on PPE before entering the room. Charge nurse told her to take time, start over.”
	• “Staff concerns on lack of head covering, foot covering, and gown not fully covering back.”
	• “Only ‘orange N95 masks’ available. Staff had not been fitted for ‘orange N95 masks’/did not know how to use.”
	• “Visors in PPE cart defective.”
	• “No hazard waste bin in isolation ante room.”
	• “Dedicated Ebola waste receptacle pre-positioned too far away from the isolation rooms at front of ED.”
	• “PPE disposal overflowed containers in isolation.”
Notification
Strengths	• “Greeter nurse notified charge nurse.”
	• “Upon receiving notification, the charge nurse activated a code which notifies the ED staff overhead to prepare the isolation room for a suspected infectious patient.”
	• “Triage nurse escalated to charge nurse, ED Attending, Infectious Diseases, Infection Control, and Nursing Supervisor that there was a symptomatic patient in the private room.”
	• “Notification by charge nurse to ED clerk to make notifications to ‘identified departments.’ ”
	• “Nurse Manager escalated the calls to Infection Control, Environmental Services, administration, and Infectious Diseases.”
	• “Code Yellow used for internal announcement to staff.”
	• “Initiated Hospital Incident Command System (HICS) to notify key leadership and staff of suspected EVD patient.”
	• “Initiated a HICS level 1 to notify key leadership and staff of suspected EVD patient.”
Challenges	• “Security, Respiratory, IC, and administration should have been notified by ED staff but were not.”
	• “No communication protocol exists within the ED to notify other staff that a suspected case has entered the ED.”
	• “Medical provider did not know to call infection control.”
	• “Greeter nurse walked patient to isolation room and never notified security.”
	• “A more robust internal communication plan needs to be created to notify pertinent personnel that there is a suspected Ebola patient in the ED.”
	• “Develop communication call tree to escalate communication of suspect Ebola patient to medical staff, support services, and other critical internal entities.”
	• “Need a code for overhead page that notifies ED staff of a suspected Ebola patient in the ED.”
Communications
Strengths	• “Excellent command and control from key leaders with responding departments, especially nursing and IC.”
	• “HICS Level 2 activation provided the necessary leadership for planning, operations, and response/recovery activities.”
	• “Full coordination and cooperation of all departments.”
	• “Charge nurse notifies security office and assures that isolation room is ready for patient.”
	• “Further communication with the patient was done via intercom phone in isolation room.”
	• “Communication with patient in isolation room via video intercom; video monitor was not functional.”
Challenges	• “It was not explained to the patient that they were being put in an isolation room or the purpose of this.”
	• “Challenges communicating between RNs in isolation room and observer to hear the correct PPE doffing information.”
	• “Communication between nurses outside the room was challenging—phone requested.”
	• “Suggestion made to place a phone with interpreter capability in room.”
	• “Patient unable to make calls from the phone in the room; they were able to receive calls.”
	• “No traffic control instituted; no communications with security.”
	• “Some security personnel did not know what to do.”
	• “Security left stating that he had an emergency, leaving no access control, and did not call for a replacement.”
Setting-Specific Issues
Strengths	• “Security controlled access to ED.”
Challenges	• “Challenges with donning PPE in hallway—consider another space—doffing difficult in small isolation room.”
	• “Cubicle adjacent to isolation room was occupied so donning and doffing took place in hallway—traffic restricted.”
	• “Isolation room 2 does not have ante chamber—ED was not built with ante-chamber rooms.”
	• “Isolation room is OB/GYN with a private bathroom but no anteroom for donning/doffing PPE.”
	• “Code 111 not audible in isolation room.”

Illustrative quotes are representative of topics mentioned in the after-action reports; the table does not include all statements from all reports.

Abbreviations: ED = emergency department; EVD = Ebola virus disease; HICS = hospital incident command system; IC = infection control; MD = medical doctor; OB/GYN = obstetrics and gynecology; PPE = personal protective equipment; RN = registered nurse.

Screening and Triage

Screening and triage refers to statements made in the reports commenting on either screening and registration protocols or their implementation. A strength in this area was that all submitted after-action reports reported that the mystery patient was identified as a person with possible EVD at some point during the drill. Many hospitals highlighted the prompt assessment of travel history, which promoted rapid implementation of appropriate infection control and isolation protocols, with 1 hospital commenting:

Patient presented complaining of symptoms; upon hearing symptoms, travel history was asked within one minute of patient's arrival.

A challenge that was mentioned by a few hospitals was delayed triaging of the patient, which could potentially lead to exposure risks to staff and other patients. One example described that the

… patient picked up notification phone outside waiting room entrance to notify nurse that she had high fever and stomach pains. She was directed to sit in waiting room and was not triaged.

Another hospital described the patient

… entering the walk-in ER entrance, the patient told security he needed to see a doctor. He was asked to sign in and wait to be called.

Infection Control

Infection control refers to statements made in the after-action reports that address initial and subsequent measures taken by staff after identifying a patient with possible EVD to reduce the risk of transmission of disease to staff, other patients, and visitors.

Many strengths in implementation of recommended infection control practices were described in the after-action reports. Hand hygiene and the use of masks were almost universally employed as protective measures once a patient was identified as an EVD risk. Some facilities reported separating the patient from others quickly upon identification. Many had pre-planned the best route from the triage area to the isolation room, limiting the suspected EVD patient's contact with other patients, staff, and visitors by moving the patient out of the general flow of traffic and clearing the shortest route to the isolation rooms. One facility wrote that the

… patient was provided with a mask, and the triage nurse donned a mask, and patient … was moved from the triage area to the private room with a dedicated bathroom.

Additional strengths included limiting access to the isolation room and assigning a “gatekeeper” to control and log staff's entry and exit from the room.

There were a number of challenges around infection control identified in the reports. Some hospitals indicated concerns over where to stage suspected EVD patients while the isolation room was being prepared. In one instance, a facility asked the patient to wait outside until the room was ready. Other challenges included hand sanitizer availability and surface cleaning. As an example, one hospital noted that the “counter area in pre-triage was not disinfected with bleach post patient exposure.”

Personal Protective Equipment

Personal protective equipment (PPE) refers to statements in the after-action reports that reference availability and use of PPE during the drills.

There were several strengths identified around PPE. Many indicated that PPE supplies were readily available and well positioned for easy access, with 1 hospital commenting: “PPE was set up on the cart outside the isolation room.”

Many hospitals highlighted staff use of appropriate PPE based on their role and the scenario, with 1 facility commenting on the “appropriate donning and doffing of ED nurse, ED physician, security guard and housekeeper.” There were also several mentions of use of the “buddy system” to ensure proper donning and doffing of PPE:

Upon completion of care, PPE was successfully removed. Donning and doffing was supervised by a donning/doffing coach, who provided instructions for each step of the donning/doffing process.

Aside from that, there were numerous challenges and opportunities for improvement identified surrounding PPE use. Some staff did not take sufficient time to properly don their equipment. In one instance, a

… triage nurse was rushing to put on PPE before entering the room. Charge nurse told her to take time, start over.

Also, several facilities noted that the “buddy system was not used for donning PPE.” Emergency department staff in many facilities exhibited a general lack of familiarity with proper donning and doffing, including one who observed that

… no staff exhibited proficient competency when donning and doffing their PPE, indicating that more frequent training and practice is necessary.

There were also a few references to quality of equipment, such as with “visors in PPE cart [being] defective” or not fitting, or “isolation gown too short on RN—exposed clothing.” There were additional concerns about PPE disposal when “PPE disposal overflowed containers in isolation” or when “the Ebola waste receptacle [was] pre-positioned too far away from the isolation rooms.”

Notification

Notification refers to statements in the after-action reports describing how staff notifications were carried out once a potential EVD patient had been identified.

A common strength was that triage staff generally notified their ED supervisors, such as the charge nurse and clinical care providers, who then made appropriate notifications to other key leaders and staff members according to the individual organization's notification protocols. Some facilities “initiated the Hospital Incident Command System [HICS] to notify key leadership and staff of suspected EVD patient,” which helped ensure the appropriate leadership staff were alerted. Others mentioned using “a code which notifies the ER staff overhead to prepare the isolation room for a suspected infectious patient.”

However, some hospitals determined that the notification protocols were not clear, and at least 3 facilities mentioned that key individuals such as “security, respiratory, IC, and administration should have been notified by ED staff but were not.” Hospitals that noted difficulty with internal notifications sometimes recommended using their hospital incident command system protocols to improve standardization of notifications during such events. One hospital with notification challenges noted that

… a more robust internal communication plan needs to be created to notify pertinent personnel that there is a suspected Ebola patient in the ED …

and that they might “utilize HICS to communicate with key staff.”

Communication

Communication refers to statements in the reports relating to intra- and interdepartmental staff communication and communication between the staff and patient.

Communication was noted as a strength when all relevant hospital staff were aware of a potential patient with EVD and their role in the response. One hospital commented on their

… excellent command and control from key leaders with responding departments especially nursing and Infection Control.

A few facilities noted communication with the patient as a strength, especially among those who “utilized intercom phone(s) in the isolation room(s).”

There were challenges noted when communication was inadequate between different departments. For example, 1 facility had an issue when the “greeter nurse walked patient to isolation room and never notified security.” Another after-action report noted that “no traffic control [was] instituted; no communications with security.” Others mentioned challenges communicating from the isolation room with 2 facilities, observing that it was

… difficult for staff inside the isolation room to communicate with colleagues outside the room due to noise or improper equipment, …

and that there were

… challenges communicating between RNs in isolation room and observer to hear the correct PPE doffing information.

Setting-Specific Issues

Setting-specific issues include items in the after-action reports that relate to the physical space and/or layout of the emergency departments as they pertained to the conduct of the drill. There were few strengths mentioned in the reports, though 1 facility highlighted the effective use of “Security to control access to ED” and “movement in and out of the triage area.” The challenges identified were primarily related to the isolation rooms, including availability of negative pressure airborne isolation rooms and/or rooms that included the recommended anteroom and private bathrooms. At least 3 hospitals identified that a lack of an anteroom created challenges when donning and doffing PPE, with one noting:

cubicle adjacent to isolation room was occupied so donning and doffing took place in hallway—traffic restricted.

Discussion

This study was unique in that we were able to look at EVD drill performance across a large sample of hospitals to identify recurrent themes related to strengths and challenges experienced through a qualitative review of drill reports. Overall, NYC hospitals reported they had significant success in developing and implementing screening protocols intended to identify febrile patients with travel history and sentinel symptoms suggestive of EVD at registration or triage. Staff almost universally isolated the patient per protocols and used mask and hand hygiene to protect themselves. This success may be partly because NYC had a diagnosed EVD case shortly after the New York State Commissioner of Health Order on Ebola patient screening was issued (late October 2014), which would have likely enhanced ED staff vigilance toward EVD and infection risks.

This is also the first study to evaluate healthcare facility performance using time-based measures (such as time to isolation) for a potential patient with EVD. Though these drills were designed and completed prior to the release of the ASPR/HPP performance measures for Ebola preparedness, the median time from triage to isolation measured in this series of drills met the ASPR Ebola Assessment Hospital goal “to identify and isolate a patient with Ebola or other highly infectious disease” within 5 minutes.¹⁵ However, a potentially more meaningful measure is the total time required from patient entry to being placed in isolation, which takes into account the total time a patient spends in the emergency room waiting room and/or triage area when a patient could be in close contact with and potentially exposing other patients and staff prior to being triaged. In addition, we would also like to consider the utility of other measures in future drills, such as the time required for a patient to be given a mask, which would indicate a crucial step in “source control” for potentially infectious patients that would mitigate exposure risks before placement in isolation occurs.

We identified several best practices in this analysis. One notable success highlighted in many of the drills was the use of hospital incident command systems, which determine how authority and responsibility are distributed in a hospital's incident management team during an event. Integrating hospital incident command system activation along with the associated notification protocols into the Ebola drills appeared to have a positive impact on advancing notifications more efficiently to the appropriate people in charge, who then took the lead in coordinating their respective components of the response. It was notable that many hospitals that identified communication and/or coordination as challenges recommended including hospital incident command system implementation into their protocols. Using the hospital incident command system structure during smaller exercises could provide facilities with additional opportunities to ensure their incident command structure is robust and that designated leaders become more familiar with their roles and responsibilities in that system for types of emergencies other than Ebola.

Among the most significant challenges identified in this analysis were the issues noted with donning and doffing PPE, which could have a significant impact on healthcare worker safety. Multiple hospitals noted that staff appeared rushed when donning the equipment, seemed unfamiliar with protocols, and often failed to use the buddy system. This is concerning, especially because New York State had implemented requirements for monthly competency-based training of clinical staff on donning and doffing of high-level PPE prior to the initiation of these drills. Our findings emphasize the need to maintain standardized PPE protocols and for frontline clinical care providers to demonstrate their proficiency in their facility's PPE protocols on a routine basis.^14,16 There is a clear need for further research into approaches for maintaining healthcare staff competence with Ebola-level PPE protocols,¹⁷ including optimal timing and frequency, in a way that minimizes the burden on staff and maximizes skill retention.^18,19

To our knowledge, this study involves the largest cohort of hospitals performing an infectious disease–based drill that evaluated both initial patient identification as well as implementation of appropriate infection control measures in emergency departments.^6,8-13,20-22 The only other report on the use of Ebola drills was described by Hsu and colleagues.²⁰ They conducted no-notice “mystery patient drills” in 99 hospitals in Taiwan but focused specifically on ED staff compliance with Ebola-related travel and risk-factor screening at the time of triage as part of a national hospital inspection process. They found that 40% of regional hospitals and medical centers failed to ask emergency room patients about travel history despite 90% having self-reported doing so during regular inspections.²⁰ In comparison with our study, all hospitals reported that the ED staff had screened and identified the patient correctly. In order to better evaluate hospital compliance with patient screening protocols under real-world conditions in NYC, it would be beneficial to have an outside party carry out standardized drill scenarios outside of an acute emergency response.

Limitations

There were several limitations to our study. Since hospitals were responsible for the planning and conduct of the drills with few specific guidelines, the exercise design, evaluations, and after-action report format varied considerably between hospitals. Several used the standard Homeland Security Exercise and Evaluation Program (HSEEP) process and template, while others developed brief summaries of the drills. Despite the heterogeneity and the associated challenges with conducting aggregate data analysis, the after-action reports reflected data associated with 1 or more of the defined critical elements. However, given that a specific after-action report format was not provided or specified, at times it was difficult to determine whether specific tasks had been completed.

Another concern was that drills were generally carried out by hospital staff who may have been recognized by the ED personnel. Additionally, most facilities performed multiple mystery patient drills, but they were required to submit only 1 of potentially many after-action reports; this may have introduced a reporting bias toward positive results. Also, if the after-action report submitted happened to be from one of the earlier drills, it may have failed to reflect the evolution toward readiness. Some hospitals implemented their own mystery patient drill programs before the mandate required all hospitals to do so; these hospitals may have submitted an after-action report from one of their own very early drills, possibly increasing the variation in how the drills captured in this analysis were carried out.

Conclusions

Emergency departments, including triage and waiting areas, are vulnerable points of entry for communicable diseases into the healthcare system.²³ Unannounced mystery patient drills can be a beneficial tool for hospitals to test their protocols, assess staff competencies, and identify preparedness gaps in order to rapidly implement necessary corrective measures. To improve their utility as a healthcare preparedness tool, consideration should be given to the further development of specific validated metrics and expected actions to support a more robust evaluation and to allow comparability between multiple facilities. For example, by capturing the required time stamps, facilities can calculate time-based performance measures such as the time required to isolate the patient. Use of such measures could also offer useful benchmarks when comparing overall drill performance at a facility over time.

Going forward, we believe that mystery patient drills, as a complement to planning and training, can improve early recognition of infectious patients and promote the rapid institution of infection control measures to protect staff, patients, and the public at large. In the future, we hope to continue to collect and analyze mystery patient drill reports to look for changes over time. Findings from this study will be used to inform the development of a citywide mystery patient drill program to assess healthcare preparedness for other communicable diseases of public health concern using actors or Medical Reserve Corps volunteers and standardized drill scenarios and formats.

Footnotes

Acknowledgments

We would like to thank the Yale New Haven Health System Center for Emergency Preparedness and Disaster Response (YNHHS-CEPDR) for their contributions. This project was funded through ASPR's Hospital Preparedness Program (CFDA # 93.817).

References

Centers for Disease Control and Prevention. 2014 Ebola outbreak in West Africa—case counts. CDC website. Updated April 13, 2016. http://www.cdc.gov/vhf/ebola/outbreaks/2014-west-Africa/case-counts.html. Accessed November 29, 2016.

Brown

, Aranas

, Benenson

, et al. Airport exit and entry screening for Ebola—August-November 10, 2014. MMWR Morb Mortal Wkly Rep, 2014; 63(49):1163-1167.

Zucker

Commissioner's Order of October 16, 2014 regarding Ebola Virus Disease (EVD). https://www.health.ny.gov/diseases/communicable/ebola/#commissioner_order. Accessed November 29, 2016.

Adini

, Laor

, Hornik-Lurie

, Schwartz

, Aharonson-Daniel

. Improving hospital mass casualty preparedness through ongoing readiness evaluation. Am J Med Qual, 2012; 27(5):426-433.

Adini

, Verbeek

, Trapp

, et al. Continued vigilance—development of an online evaluation tool for assessing preparedness of medical facilities for biological events. Front Public Health, 2014; 2(35):1-8.

Djalali

, Carenzo

, Ragazzoni

, et al. Does hospital disaster preparedness predict response performance during a full-scale exercise? A pilot study. Prehosp Disaster Med, 2014; 29(5):441-447.

Marcozzi

, Lurie

. Measuring healthcare preparedness: an all-hazards approach. Isr J Health Policy Res, 2012; 1(1):42.

Adini

, Goldberg

, Cohen

, Bar-Dayan

. Relationship between standards of procedures for pandemic flu and level of hospital performance in simulated drills. Ann Emerg Med, 2008; 52(3):223-229.

Adini

, Goldberg

, Cohen

, Bar-Dayan

. Relationship between equipment and infrastructure for pandemic influenza and performance in an avian flu drill. Emerg Med J, 2009; 26(11):786-790.

10.

Adini

, Goldberg

, Cohen

, Bar-Dayan

. Impact of pandemic flu training on ability of medical personnel to recognize an index case of avian influenza. Eur J Public Health, 2012; 22(2):169-173.

11.

Cardenosa

, Dominguez

, Carratala

, et al. Usefulness of simulated cases for assessing pandemic influenza preparedness plans. Clin Microbiol Infect, 2010; 16(9):1364-1367.

12.

Leiba

, Goldberg

, Hourvitz

, et al. Lessons learned from clinical anthrax drills: evaluation of knowledge and preparedness for a bioterrorist threat in Israeli emergency departments. Ann Emerg Med, 2006; 48(2):194-199.

13.

Klein

, Atas

, Collins

. Testing emergency medical personnel response to patients with suspected infectious disease. Prehosp Disaster Med, 2004; 19(3):256-265.

14.

NYC DOHMH Guidance Document for Development of Protocols for Management of Patients Presenting to Hospital Emergency Departments and Clinics with Potentially Communicable Diseases of Public Health Concern. New York City Department of Health & Mental Hygiene; October. 2014. http://www.programinfosite.com/oepr/files/2014/10/Guidance-document-final-version-10-06-14.pdf. Accessed August 8, 2016.

15.

Assistant Secretary for Preparedness and Response. The Hospital Preparedness Program (HPP) Measure Manual, Implementation Guidance for the HPP Ebola Preparedness Measures. Washington, DC: U.S. Department of Health and Human Services; July 2015.

16.

Centers for Disease Control and Prevention. Infection prevention and control recommendations for hospitalized patients under investigation (PUIs) for Ebola Virus Disease (EVD) in U.S. hospitals. CDC website. http://www.cdc.gov/vhf/ebola/healthcare-us/ppe/guidance.html Accessed November 29, 2016.

17.

Centers for Disease Control and Prevention. Guidance on personal protective equipment (PPE) to be used by healthcare workers during management of patients with confirmed Ebola or persons under investigation (PUIs) for Ebola who are clinically unstable or have bleeding, vomiting, or diarrhea in U.S. hospitals, including procedures for donning and doffing PPE. CDC website. Updated August 27, 2015. https://www.cdc.gov/vhf/ebola/healthcare-us/ppe/guidance.html. Accessed April 1, 2017.

18.

Hersi

, Stevens

, Quach

, et al. Effectiveness of personal protective equipment for healthcare workers caring for patients with filovirus disease: a rapid review. PLoS One, 2015; 10(10):e0140290.

19.

Verbeek

, Ijaz

, Mischke

, et al. Personal protective equipment for preventing highly infectious diseases due to exposure to contaminated body fluids in healthcare staff. Cochrane Database Syst Rev, 2016; 4:CD011621.

20.

Hsu

, Chien

, Tseng

, Kuo

. A no-notice drill of hospital peparedness in responding to Ebola virus disease in Taiwan. Health Secur, 2015; 13(5):339-344.

21.

Eastwood

, Durrheim

, Merritt

, et al. Field exercises are useful for improving public health emergency responses. Western Pac Surveill Response J, 2010; 1(1):12-18.

22.

Lyons

, Burkle

Jr , Roepke

, Bertz

. An influenza pandemic exercise in a major urban setting, Part I: Hospital health systems lessons learned and implications for future planning. Am J Disaster Med, 2009; 4(2):120-128.

23.

Vetter

, Dayer

J-A

, Schibler

, et al. The 2014-2015 Ebola outbreak in West Africa: hands on. Antimicrob Resist Infect Control, 2016; 5(1):17.