CDC's Flu on Call Simulation: Testing a National Helpline for Use During an Influenza Pandemic

Abstract

During an influenza pandemic, healthcare facilities are likely to be filled to capacity, leading to delays in seeing a provider and obtaining treatment. Flu on Call is a collaborative effort between the US Centers for Disease Control and Prevention and partners to develop a toll-free telephone helpline to reduce the burden on healthcare facilities and improve access to antivirals for people who are ill during an influenza pandemic. This study tested the feasibility of Flu on Call during a 1-day simulation using a severe pandemic scenario. Trained volunteer actors placed calls to the helpline using prepared scripts that were precoded for an expected outcome (“disposition”) of the call. Scripts represented callers who were ill, those calling for someone else who was ill, and callers who were only seeking information. Information specialists and medical professionals managed the calls. Results demonstrated that Flu on Call may effectively assist callers during a pandemic, increase access to antiviral prescriptions, and direct patients to the appropriate level of care. Overall, 84% of calls exactly matched the expected call disposition; few calls (2%) were undermanaged (eg, the caller was ill but not transferred to a medical professional or received advice from the medical professional that was less intensive than what was warranted). Callers indicated a high level of satisfaction (83% reported their needs were met). Because of the high volume of calls that may be received during a severe pandemic, the Flu on Call platform should evolve to include additional triage channels (eg, through internet, chat, and/or text access).

NOTE: The development and testing of Flu on Call was conducted before the emergence of the COVID-19 pandemic. Because telehealth platforms have been activated by multiple providers in the United States to support response measures to the pandemic, a centralized national nurse triage line has not been activated. In March 2020, the US Centers for Disease Control and Prevention (CDC) made a triage decision tree available to direct people to the appropriate level of care, and to provide public health guidance based on a person's exposure, symptoms, and existing medical conditions. This open-source product has been made available through the Open CDC collaboration resource. The CDC also launched a coronavirus self-checker called Clara in March 2020 (https://www.cdc.gov/coronavirus/2019-nCoV/index.html). This electronic triage system uses the CDC decision tree and assists users to make decisions about seeking appropriate medical care and provides public health guidance based on a person's exposure, symptoms, and existing medical conditions, and includes information on testing. Since its launch, approximately 150,000 users access the Clara self-checker every day.

Pandemic influenza planning continues to be a national preparedness priority; a severe influenza pandemic will likely pose a significant public health threat.^1,2 Strategies to decrease the impact of a pandemic include ensuring that people who are ill have access to the appropriate level of clinical care and timely access to antiviral drugs.^3-5 During a severe pandemic, it is likely that medical offices, clinics, and emergency rooms will be filled to capacity, leading to delays in seeing a provider and obtaining antiviral prescriptions.

To address these issues, the US Centers for Disease Control and Prevention (CDC) is exploring ways to decrease the burden on healthcare facilities and improve access to antiviral medicine during a pandemic, including the use of a national nurse triage line call center.^6,7 Call centers have been successfully used for a variety of health issues, including infectious disease outbreaks and during a previous influenza pandemic.^8-14 During the 2009-2010 H1N1 pandemic, both the Minnesota FluLine and the UK National Pandemic Flu Service effectively used call centers to triage callers in their jurisdictions who experienced symptoms of influenza-like illness, and provide access to information, clinical advice, and antivirals.^15-18 Flu on Call was launched in 2013 as a collaborative effort between the CDC and its partners (see Appendix A; www.liebertpub.com/doi/suppl/10.1089/hs.2019.0152) to explore the feasibility and acceptability of launching a national telephone helpline during a future pandemic.⁶

The goals of developing the Flu on Call national helpline for use during a severe pandemic are to: (1) provide an alternative to face-to-face provider encounters to reduce surge on medical facilities; (2) improve access to antiviral prescriptions for people who are ill, when appropriate; (3) provide accurate information to callers; and (4) facilitate the appropriate use of medical resources. Participants include CDC staff, several telephony and customer relationship management system partners (who manage the toll-free telephone number, host a web platform to collect data from callers, and store reference information for those serving callers), data analytics partner, information specialists (who staff the telephone call lines and provide callers with information), and medical professionals (eg, registered nurses and clinical pharmacists).

Although call centers and nurse contact lines are used frequently to provide advice and clinical care to the public,^12,14,19 and several emergency state-based nurse triage lines were successfully activated for their residents during the 2009 influenza pandemic,^20,21 no national programs are designed to serve callers from all parts of the country during a public health emergency, and no studies have been published describing the testing and evaluation of a national emergency nurse triage line. Therefore, as part of the development of Flu on Call, a series of simulations were conducted to test the design, operations, and call outcomes. This manuscript describes the design and findings from the 2018 Flu on Call simulation.

How Does Flu on Call Work?

Flu on Call is designed as a national network, with the ability to serve people from any state when the system is activated by the CDC in a pandemic influenza scenario. The network is structured to serve people who are ill, people who are caring for someone who is ill, and those who are not ill but seeking information.

When the system is activated, state health officials will receive a toll-free number for Flu on Call that they can share with the public in their jurisdictions. Callers will dial a toll-free number, listen to an interactive voice response greeting, and their call will be routed to one of multiple call centers located throughout the country. The information specialist will answer the call, and first ask if the caller is ill or caring for someone who is ill, and then record the information on an intake form designated with a unique identification number generated by the Flu on Call system. If the caller is not ill nor caring for a person who is ill, the information specialist will gather basic demographic information and answer the caller's questions and provide information (using a database of frequently asked questions) or refer them to a public health resource such as a state or local health department or the CDC website. If the caller is ill or caring for a person who is ill, the specialist will ask if the ill person is experiencing a life-threatening emergency and, if they are, advise them to immediately call 911. If they are not experiencing an emergency, the specialist will collect some basic demographic information, and offer to transfer the caller to a medical professional.* The call, along with the intake record started by the information specialist, will be transferred to a medical professional. The medical professional (working under an approved clinical protocol) will first screen for a medical emergency (and advise the patient to call 911 or seek care urgently, if warranted), take a brief medical history (including drug allergies and ongoing medications), inquire about current signs and symptoms, provide clinical advice and patient teaching (eg, about caring for a sick person at home, any medications needed), and if appropriate, send an electronic prescription for an antiviral drug to a nearby pharmacy. The medical professional will also offer to text home care instructions to the caller, when appropriate (Figure 1).

Figure 1.
Call Flow for Calls During the 2018 Flu on Call Simulation

To build the Flu on Call network, evaluate its potential utility, test training methods, evaluate feasibility, and refine operational plans, we have conducted 8 simulations since 2013 (see Supplementary Appendix B online; www.liebertpub.com/doi/suppl/10.1089/hs.2019.0152). After each simulation, we identified areas for improvement and adjusted operational plans; we then tested the updated plans in the next simulation. A severe pandemic influenza outbreak in the United States was used as the scenario for each simulation, which included a large number of people who were ill and needed medical care. The objectives for the 2018 simulation pertained to receiving, triaging, and transferring calls, and providing access to antivirals (when appropriate) during a severe pandemic scenario (Table 1).

Table 1.
Objectives for the 2018 Flu on Call Simulation

1. Conduct the simulation without disruption to real-world calls^a

2. Evaluate the interoperability of the telephony, database, and call metric capabilities^b of the Flu on Call system

3. Evaluate the effectiveness of new training methods for information specialists and medical professionals

4. Evaluate the effectiveness of call transfers between information specialists and medical professionals

5. Improve disposition accuracy and reduce handling time of the medical professionals

6. Assess the feasibility of medical professionals to prepare and transmit electronic antiviral prescriptions

7. Evaluate how the needs of out-of-state callers are managed^c

8. Test the effectiveness of training for call processing times

9. Evaluate accuracy in providing information to callers

a
Some of the information specialists participated in the simulation from their usual call center locations and they only answered calls from Flu on Call during the simulation.

b
Call metrics included call duration, handle times, hold times, and after-call processing times (see Table 3).

c
Because of nurse practice act considerations, if the registered nurse handling the call did not have licensure in the same state as the caller, the nurse referred the caller to a public health clinic to obtain their antivirals (rather than issuing an electronic prescription to a local pharmacy).

Table 3.
2018 Flu on Call Simulation Performance Metrics for Completed Calls (n = 348)

Calls Handled By

Average Call Metrics Information Specialists Medical Professionals

Initial IVR greeting and routing to information specialist 0 min 31 sec N/A

Speed of answer 0 min 16 sec 0 min 45 sec

Time spent waiting in queue initially to speak to information specialist or medical professional

Handle time (across all calls with and without hold time) 6 min 0 sec 15 min 33 sec

Total time an agent or medical professional spent for a call, including actively talking, time caller was on hold (including calls with zero and non-zero hold times), and time to complete required work after hanging up. This does not include the time initially spent in queue.

Talk time 4 min 41 sec 8 min 45 sec

Portion of the total handle time spent actively in conversation

Hold time, all calls 0 min 41 sec 0 min 24 sec

Portion of the total handle time spent on hold, on average, across all calls. This does not include the time initially spent in queue during a transfer.

Hold time, only calls placed on hold 1 min 43 sec 2 min 33 sec

Portion of the total handle time spent on hold between times actively talking for calls placed on a hold. This does not include the time initially spent in queue during a transfer.

After-call wrap-up documentation 0 min 38 sec 6 min 24 sec

Portion of the total handle time spent completing required steps after ending a call and before starting the next.

Total call time experienced by the caller

Includes time from when the call is answered by IVR system to termination of the call. Excludes after-call wrap-up documentation that is not experienced by the caller.

Calls handled only by information specialist 5 min 45 sec

Calls transferred to medical professional after initial intake by information specialist 16 min 9 sec

Abbreviations: IVR, interactive voice response; N/A, not applicable.

Materials and Methods

Before the simulation, all information specialists and medical professionals were required to be trained on how to use the system through a custom-developed online learning management platform. Training components included how to greet callers, answer and transfer calls, identify callers who needed immediate emergency care, document call information and outcomes, and follow specific protocols for managing callers.

Medical professionals received extensive online training and practice sessions to ensure familiarity with the 2018 Flu on Call simulation medical protocol. The medical protocol used for the simulation was based on CDC guidelines for treating people infected with a novel influenza A virus,²² which was adjusted for a severe pandemic scenario with input from expert clinicians. Medical professionals received additional online training about the use of antiviral medications including a rubric for determining if the caller needed medication, selecting which medication to prescribe, calculating the appropriate dosage, locating a convenient pharmacy for the patient, filling out a prescription form to allow for electronic prescription to be sent to a pharmacy, and recording their notes in the system. After training was concluded, all information specialist and medical professional participants had to pass an online competency test in order to take part in the simulation. No actual prescriptions or medications were used during the simulation; mock prescription forms were filled out and emailed to the CDC.

The 2018 Flu on Call simulation was conducted on May 2, 2018, and consisted of 2 call sessions, each lasting 2.5 hours. Helpline participants in the simulation included 55 trained information specialists from 14 dedicated 211 call centers,^† representing 13 states, and 59 trained medical professionals from 4 telehealth companies and 6 poison control centers. Information specialists participated in the simulation from their 211 call centers and medical professionals participated in the simulation from various locations across the country. CDC staff set up a central operations center to lead and coordinate the simulation including 59 trained volunteer actors portraying individual callers who placed calls during the exercise.

Actors were recruited from CDC staff and partner organizations (see Supplementary Appendix A online). During the simulation, the actors placed calls to the Flu on Call telephone line by dialing a unique toll-free number. Actors were trained before the simulation to use predeveloped scripts to portray callers, some representing people with various illness scenarios, and some representing callers who were not ill but needed information. In addition, 2 bilingual actors were designated to make their calls in Spanish to test the ability of the system to manage calls using Spanish prerecorded messages and interactive voice responses and use of third-party telephone language translator services.

Actors used a unique script for each call. The scripts included demographic information (eg, age, birth date, gender, race/ethnicity, address, telephone number), medical history including current medicines and symptoms, and reason for the call (the caller's “story”) (see Supplementary Appendix C online; www.liebertpub.com/doi/suppl/10.1089/hs.2019.0152). For identification purposes, each script also included a unique CDC control number and a place for the actor to record the unique identification number assigned by the telephone system (given by the information specialist during the call).

Intake data recorded by information specialists and medical professionals for each caller, as well as their notes about each call, were stored in the customer relationship management system and then anonymized and transmitted to the data analytics partner. No individual “patient” information was captured by the CDC. All calls were recorded by the telephony system for analytic purposes.

Immediately following each call, the actors evaluated their experience and recorded their findings. To determine how satisfied the actors portraying callers were with the processes and results of each call, we used multiple evaluation methods. Additionally, actors completed an overall evaluation immediately following the simulation to ascertain their overall level of satisfaction with the exercise in general.

To evaluate the accuracy of advice given to callers, the actors were asked about their understanding of the instructions provided by the information specialist or medical professional from the list of 8 mutually exclusive expected outcomes (“dispositions”) (Supplementary Appendix D online; www.liebertpub.com/doi/suppl/10.1089/hs.2019.0152). Each script was precoded with the expected disposition and was based on the 2018 Flu on Call simulation medical protocol. The number of scripts generated for each expected disposition was informed by the proportion of calls by disposition received by a state helpline used during the 2009 H1N1 pandemic (Minnesota FluLine).¹⁸

An analysis was conducted comparing the concordance between the observed and expected dispositions. The observed disposition was recorded by the actor, rather than from call documentation made by the information specialist or medical professional, because it was determined that recollection of the advice given would be the basis for action by the caller. Where observed disposition data were not available, or in cases where actor error in identifying the disposition was suspected (from their notes), a member of the research team reviewed audio recordings of calls for potential recoding of the disposition data, before analysis.

Actors participated in a “hot wash” (or after-action review) after each call session to provide additional feedback on all calls experienced in the session. The actors' evaluations of each call made, aggregated data from post-call “hot wash” sessions, call performance statistics (including call duration and hold times), assessments of call disposition correlations, and feedback from participants were included in the overall simulation analysis. The 2018 Flu on Call simulation was designed to evaluate the overall performance of the network and its ability to serve callers during a severe influenza pandemic emergency scenario, but it was not designed to evaluate individual call centers or participating information specialists or medical professionals, and, therefore, aggregated data were analyzed and will be presented.

Results

A total of 444 unique call records were generated during the 1-day 2018 Flu on Call simulation and 436 of those calls were routed as expected to an information specialist (8 calls “dropped” or were abandoned before connecting with the information specialist). Figure 2 illustrates the flow of these calls through the Flu on Call system based on call outcome.

Figure 2.
Outcome of Calls Placed During the 2018 Flu on Call Simulation ^aSix calls were excluded among those handled solely by information specialists, including calls with an unknown disposition because they were not recorded (n = 3) and calls terminated due to technical issues (n = 3). ^bTwenty-four calls were excluded among those transferred to medical professionals, including calls with an unknown disposition because they were not recorded (n = 7), calls where information was not collected to link the calls with their expected disposition from the script (n = 6), and calls terminated due to technical issues (n = 11). Abbreviations: ASAP, as soon as possible; ER, emergency room.

A total of 96 records were excluded from subsequent analysis of call dispositions (Figure 2). These included calls dropped or abandoned before a connection was made to an information specialist (n = 8), calls with unknown disposition (not specified by the actor and could not be recovered from a recording, n = 10), calls where the CDC control number was missing and, therefore, prevented comparison with the expected script disposition (n = 6), calls terminated due to technical issues before a disposition was given (n = 14), and calls dropped or abandoned in transfer from the information specialist to the medical professional (n = 58). Overall, of the 444 calls, 18% (n = 80) were dropped or terminated due to telephone network technical issues.

Approximately 36% (n = 155) of the calls connected to an information specialist were managed solely by the specialist without transferring to a medical professional. Information specialists transferred 281 callers to medical professionals, but 58 (21%) of these calls were dropped or abandoned during the transfer; 223 calls were ultimately handled by the medical professionals (Figure 2).

A call was considered “complete” and included in the analysis if it did not conclude prematurely, if the actor interacted with an information specialist and/or a medical professional, and if the actor received the requested information from the information specialist or received clinical advice and care instructions from a medical professional (n = 348).

It is important to demonstrate that Flu on Call can deliver the care and advice callers need during a pandemic; therefore, for each call placed, we assessed the correspondence between expected and observed dispositions. Concordance between the precoded “expected” disposition and the “observed” disposition as recorded by the actor for each call required that 3 conditions be met: (1) the call was handled by the appropriate professional, for example, calls from those who were not ill nor caring for an ill person were served by information specialists without transfer to a medical professional, and any call from a person who was ill (or someone calling on behalf of an ill person) was transferred to a medical professional; (2) the call received the appropriate disposition, for example, the caller received the instructions or advice that matched the expected, precoded outcome; and (3) the call was not terminated before a disposition was received by the actor due to technical issues, including disconnections (dropped calls) or cases where the actor decided to hang up (“abandoned”).

Analysis of call outcome concordance (between expected and observed dispositions) was performed on 348 completed calls. As shown in Figure 2, most (84%) of the calls made during the simulation were managed as expected by either the information specialist or medical professional; 94% were managed as expected by the information specialists, and 77% of the calls transferred to a medical professional were managed as expected. Almost all (97% of 180) of the ill callers were appropriately transferred to medical professionals (not shown).

Discordant calls where the advice given by the information specialist or medical professional did not match the expected outcome were further categorized as “overmanaged” or “undermanaged” (Table 2). If the advice given exceeded or was more intensive than was warranted based on the 2018 Flu on Call simulation medical protocol, the call was categorized as “overmanaged” (not shown). Most of the discordant calls (69%) were classified as “overmanaged,” and represented 11% (n = 37) of 348 total calls available for analysis. Examples of overmanagement included callers who were transferred to a medical professional for medical advice or information when they were not ill, but were calling with questions that could have been answered by the information specialists using the frequently asked questions, or callers whose symptoms warranted home care only and no medication but were instead provided with a mock electronic prescription (n = 9) by the medical professional.

Table 2.
Concordance Between Expected and Observed Dispositions from Calls Made During the 2018 Flu on Call Simulation

Expected Disposition Number of Calls for Which Observed Disposition Matched Expected Disposition Number of Calls Precoded with Expected Disposition Percentage of Calls for Which Observed Disposition Matched Expected Disposition Description of the Disposition

Information-only calls 131 140 94% These callers were not ill and had questions about influenza or the influenza pandemic.

Told to call 911 or go to ER now 14 15 93% Ill callers reported they or the ill person was experiencing a life-threatening emergency.

Advised to see healthcare provider or go to clinic today 17 17 100% Ill persons who were under 2 years of age or living in a long-term care facility are not managed by Flu on Call and instead advised to see their usual provider that same day.

Given prescription and advised to see healthcare provider ASAP 34 44 77% Ill persons who had a condition placing them at higher risk for flu complications and who had ILI symptoms for 72 hours or less were provided access to antivirals and advised to see their usual provider as soon as possible. Home care instructions were given.

Given prescription and home care instructions 56 72 78% Ill persons who had no condition placing them at higher risk for flu complications and who had ILI symptoms for 72 hours or less could be managed at home and provided access to antivirals. They were provided with home care instructions.

Was not given prescription for ILI symptoms but instead was provided home care instructions 20 29 69% Ill persons who had ILI symptoms for 72 hours or more and were recovering from the illness were provided with home care instructions and information about danger signals that would warrant contacting a medical provider, but not provided access to antivirals.

Although ill, caller/ill person does not have ILI and was given instructions about what to do 13 17 76% Medical professionals assessed ill persons who did not have ILI symptoms and, based on acuity, provided instructions about if and when to see a medical provider, but did not conduct detailed triage for non-ILI callers.

Not ill, but was transferred to medical professional for information/advice 9 14 64% Some callers had specific questions that were clinical in nature or were about use of medications and needed to be answered by a medical professional.

Total 294 348 84%

Abbreviations: ASAP, as soon as possible; ER, emergency room; ILI, influenza-like illness.

However, 7 of the discordant calls (2% of all calls analyzed) were classified as “undermanaged” (eg, the caller received advice that was less intensive than what was warranted based on the 2018 Flu on Call simulation medical protocol). In one case, the caller's symptoms indicated the need for referral to 911, but instead the medical professional advised them to see their healthcare provider that day. The remaining cases of undermanagement were ill callers who received information from an information specialist but were not transferred to a medical professional (n = 6).

Of the 116 callers that should have received a prescription for an antiviral, almost all (85%) were provided access to the medication by the medical professional. Notably, 6 of the 17 callers who were undermanaged and did not receive a prescription for an antiviral when expected were not appropriately transferred from the information specialist to a medical professional.

Call performance metrics were also measured during the simulation (Table 3). Callers waited a short time on average to initially speak with an information specialist (16 seconds) or a medical professional (45 seconds). The duration of calls that the information specialist managed without transfer to a medical professional averaged 5 minutes and 45 seconds. Callers who were ill or caring for an ill person typically required extensive evaluation, and some calls required the medical professional to retrieve local pharmacy or public health clinic information and electronically compose and send mock prescription forms. Those calls were longer and averaged about 16 minutes, including the time spent talking to the information specialist and medical professional and time spent on hold (Table 3).

The average hold time for all calls that were placed on hold was less than 2 minutes (not shown). Hold times for calls managed by information specialists averaged about 1 minute and 43 seconds and ranged from a few seconds to about 7.5 minutes. Callers interacting with medical professionals had average hold times of approximately 2.5 minutes (ranging from 1 second, which likely represents brief mistakes in activating hold functions, to about 8.5 minutes). Actors infrequently reported experiencing significant hold times during some calls while information specialists or medical professionals looked up information or had technical issues using the input form or the database platform (eg, 5% of the 348 calls analyzed for final disposition concordance had hold times greater than 4 minutes during the call; data not shown).

Although a few of the information specialists were native Spanish speakers, most of the 9 calls placed by the Spanish-speaking callers required third-party translation services. Actors noted particularly long wait times before a Spanish translator could join the call and described technical problems with two-thirds of these calls. Some of these calls were ultimately excluded from the analysis of expected versus observed dispositions because they were abandoned by actors during long wait times to speak to a medical professional (n = 4), terminated for technical issues (n = 1), or terminated before speaking to an information specialist (n = 1).

When callers were asked specifically about their call experience (ie, “Do you feel your needs were met?”), a majority (83%) of callers responded favorably. Additionally, most (81%) callers indicated they “would recommend Flu on Call to family and friends.”

Discussion

Results from the 2018 simulation indicated that Flu on Call is a promising approach for assisting callers during a pandemic, directing patients to the appropriate level of care, and may contribute to reducing healthcare surge during a pandemic. Simulation data show that a high percentage of calls were routed properly, and most connected calls were correctly screened and managed. These findings suggest that triaging ill patients through a national triage line during an emergency may be feasible.

Findings also show that it is feasible to test and evaluate Flu on Call using a simulation that assesses telephone processes, connectivity, user satisfaction, and call dispositions. The approach used for this simulation to test a telephone helpline, which included actors and healthcare professionals, is similar to approaches used in studies to assess preparedness in clinical settings.^23,24 The methods used in Flu on Call simulations, which involved collecting feedback from actors who portrayed callers to assess their satisfaction with the network, are similar to methods used in studies conducted by the UK National Pandemic Flu Service and others.^17,21,25,26 The metric of call dispositions, or the appropriate routing of screened calls during the simulation, has been used previously to assess clinical visit appropriateness when examining referrals made from nurse triage telephone lines to medical visits for people who were ill.²⁷ The 2018 Flu on Call simulation also evaluated connectivity, which included the number of completed calls, dropped calls, disconnected calls, transfer problems, and caller hold/wait times. Because of the lack of studies testing telephony connectivity and transfers within a call center or hotline, this simulation can inform future evaluations of those metrics.

This study demonstrated a number of strengths for Flu on Call. The wait time between the caller hearing the initial interactive voice response and connecting with an information specialist was very fast (16 seconds on average). This is important because a large number of ill callers will likely contact the line during a pandemic and the speed of handling callers will affect the number of callers the system can handle during an emergency.²⁸ The ability of information specialists and medical professionals to appropriately manage most of the calls during the 2018 Flu on Call simulation is promising; one of its goals is to safely manage many ill people without the need for a face-to-face visit with a healthcare provider. A meta-analysis found that approximately 50% of calls managed by doctors or nurses staffing a telephone triage line for ill callers did not require a provider visit and could be sufficiently handled by telephone interaction.²⁹

One of the major goals of Flu on Call is to improve access to antiviral medications during a pandemic. Although approximately 85% of the callers were prescribed antivirals as expected during the simulation, some patients could not be prescribed antiviral medications because the registered nurse answering the call did not have licensure in the same state as the caller; these patients were advised to pick up antivirals at a public health dispensing center instead. This is a major area for development for Flu on Call. During a pandemic emergency, a Flu on Call medical professional should ideally be able to help callers from any state. In the future, expansion of the Nurse Licensure Compact (beyond the current 31 participating states and 3 states with implementation pending), use of gubernatorial executive orders to extend or remove restrictions on prescribing practice for healthcare providers during a declared emergency, and other possible mechanisms may reduce these constraints during a severe pandemic.^30-32

Actors reported a high level of satisfaction with the Flu on Call network. Caller satisfaction metrics for the 2018 Flu on Call simulation were comparable to other emergency call lines.^21,26 Interviews with actors to better understand the factors influencing their satisfaction rankings and suggestions for improvement can be conducted in future Flu on Call simulations.

In addition to the strengths, there are multiple areas for improvement. Some actors reported that it took a significant amount of time for some information specialists to look up and locate the appropriate information from the database to answer callers' questions. Several information specialists and medical professionals also reported having problems finding or saving information in the system. These issues can be addressed through training and hands-on practice in future training sessions, as well as updates to the user system interface.

The duration of calls managed by medical professionals may be shortened through the use of augmented training, providing more real-time decision support to the medical professionals as they are managing calls, including a quicker way of navigating the medical protocol, locating a pharmacy with an available inventory of antivirals (although inventory during a pandemic may fluctuate),^‡ and improving methods used to generate and send an electronic prescription to a pharmacy.

At the start of each call session, the actors reported a number of dropped calls during the transfer process between information specialists and medical professionals. This appeared to be related to training issues among participants; as each session progressed, fewer calls were dropped. Going forward, training will be revised to include additional content and practice sessions to prevent excessive dropped calls.

Although the relative proportion of Spanish-speaking calls in the simulation was small (2% of all calls), actor feedback for most of these calls clearly highlighted that timely provision of Spanish translators was an area in need of improvement. Including more bilingual information specialists and medical professionals (when feasible), securing adequately staffed and rapid-access language translation services, and ensuring better training for transferring calls are high-priority areas for improvement.

The proportion of ill callers who were not managed as expected during the simulation was relatively low, but a detailed review of training methods and content should be conducted to enhance training for scenarios that were commonly mismanaged, and protocols adjusted to reduce these occurrences. Because patient safety is paramount in a system like Flu on Call, instances of undermanagement or overmanagement of callers warrant serious attention going forward. Case studies of these instances will be examined and changes made in the training for Flu on Call information specialists and medical professionals based on evidence-based protocols, current quality measures, and expert advice.³³

The findings from this simulation are subject to several limitations. The scenario, as well as the environment and conditions, were simulated and included a number of artificialities as part of the exercise design that may not reflect the way a future pandemic evolves. First, the simulation used trained actors who may not represent actual callers during a pandemic emergency, and the scripts provided may not accurately reflect the symptoms and concerns of callers to Flu on Call when it is activated. Also, some actors may have portrayed their symptoms or medical history in a way that deviated from the script, which could influence outcomes and exercise results. Third, the information specialists and medical professionals who may ultimately staff Flu on Call during a future pandemic may not have the same training as those participating in the simulation and may require more training. Fourth, because the simulation was conducted during a simulated emergency, personal experiences of the outbreak as well as media coverage messages about the severity and effects of the pandemic, and likely anxiety and concern that would be expected during a real public health emergency were not replicated, and may have affected the actions of the participants. Fifth, although actual systems for electronic prescribing were used in the simulation, the prescriptions were not actually sent, therefore, no data are available on the number of prescriptions that were effectively transmitted to pharmacies. The number of callers who would pick up the prescription and take the medication as prescribed is also not known.

Finally, the simulation did not test the ultimate “load” or challenge the capacity of the Flu on Call system and was not designed to assess call capacity because of the extremely large volume of calls that would be needed to exceed telephony resources. Other means will be used to test call capacity including computerized “load-bearing” exercises. A recently published modeling study of a system like Flu on Call showed that substantial call volumes should be expected if this system is activated during a national emergency.²⁸ Based on this model's findings, the weekly call volume could range from 1.5 million to 7 million calls. Other methods of serving callers will need to be developed. Other successful methods have served the public, for example, an internet interface with UK NHS Direct service¹⁹ and a tobacco quit line that incorporated a web-based approach for triaging and serving persons in combination with its use of a telephone line.³⁴

Implications for Future Research and Conclusions

This study shows that the methods used for telephone triage in Flu on Call may improve access to antiviral prescriptions during a pandemic and reduce surge in healthcare facilities by directing callers to the appropriate level of care (eg, sending callers to face-to-face care only when needed). The study highlighted 2 areas of improvement for the current system: (1) enhancing training methods to ensure patient safety and (2) adding services and translation capabilities so the system may better serve callers who speak Spanish and exploring translation to other languages.^§

Although the impact of a future influenza pandemic is not known, modeling studies predict a large volume of callers to systems like Flu on Call.²⁸ Securing additional staff to answer calls may be feasible in a limited outbreak. However, for a pandemic with high levels of demand for medical care, the Flu on Call platform should evolve to include self-triage and multiple channels including web access, as well as contact by chat and text. These channels would be able to surge more rapidly to serve more people. Expanding triage to include web, chat, and text also aligns the service with communication channels commonly used today.^35,36 Ensuring that these alternative communication platforms can provide accurate information, clinical triage, and predictable outcomes will be critical. Additional research is needed to understand the full utility of evolving technology and the use of such clinical triage tools for use during emergencies.

	Calls Handled By
Initial IVR greeting and routing to information specialist	0 min 31 sec	N/A
Speed of answer	0 min 16 sec	0 min 45 sec
Time spent waiting in queue initially to speak to information specialist or medical professional
Handle time (across all calls with and without hold time)	6 min 0 sec	15 min 33 sec
Total time an agent or medical professional spent for a call, including actively talking, time caller was on hold (including calls with zero and non-zero hold times), and time to complete required work after hanging up. This does not include the time initially spent in queue.
Talk time	4 min 41 sec	8 min 45 sec
Portion of the total handle time spent actively in conversation
Hold time, all calls	0 min 41 sec	0 min 24 sec
Portion of the total handle time spent on hold, on average, across all calls. This does not include the time initially spent in queue during a transfer.
Hold time, only calls placed on hold	1 min 43 sec	2 min 33 sec
Portion of the total handle time spent on hold between times actively talking for calls placed on a hold. This does not include the time initially spent in queue during a transfer.
After-call wrap-up documentation	0 min 38 sec	6 min 24 sec
Portion of the total handle time spent completing required steps after ending a call and before starting the next.
Total call time experienced by the caller
Includes time from when the call is answered by IVR system to termination of the call. Excludes after-call wrap-up documentation that is not experienced by the caller.
Calls handled only by information specialist	5 min 45 sec
Calls transferred to medical professional after initial intake by information specialist	16 min 9 sec

Expected Disposition	Number of Calls for Which Observed Disposition Matched Expected Disposition	Number of Calls Precoded with Expected Disposition	Percentage of Calls for Which Observed Disposition Matched Expected Disposition	Description of the Disposition
Information-only calls	131	140	94%	These callers were not ill and had questions about influenza or the influenza pandemic.
Told to call 911 or go to ER now	14	15	93%	Ill callers reported they or the ill person was experiencing a life-threatening emergency.
Advised to see healthcare provider or go to clinic today	17	17	100%	Ill persons who were under 2 years of age or living in a long-term care facility are not managed by Flu on Call and instead advised to see their usual provider that same day.
Given prescription and advised to see healthcare provider ASAP	34	44	77%	Ill persons who had a condition placing them at higher risk for flu complications and who had ILI symptoms for 72 hours or less were provided access to antivirals and advised to see their usual provider as soon as possible. Home care instructions were given.
Given prescription and home care instructions	56	72	78%	Ill persons who had no condition placing them at higher risk for flu complications and who had ILI symptoms for 72 hours or less could be managed at home and provided access to antivirals. They were provided with home care instructions.
Was not given prescription for ILI symptoms but instead was provided home care instructions	20	29	69%	Ill persons who had ILI symptoms for 72 hours or more and were recovering from the illness were provided with home care instructions and information about danger signals that would warrant contacting a medical provider, but not provided access to antivirals.
Although ill, caller/ill person does not have ILI and was given instructions about what to do	13	17	76%	Medical professionals assessed ill persons who did not have ILI symptoms and, based on acuity, provided instructions about if and when to see a medical provider, but did not conduct detailed triage for non-ILI callers.
Not ill, but was transferred to medical professional for information/advice	9	14	64%	Some callers had specific questions that were clinical in nature or were about use of medications and needed to be answered by a medical professional.
Total	294	348	84%

Footnotes

References

Jester

, Uyeki

, Patel

, Koonin

, Jernigan

. 100 years of medical countermeasures and pandemic influenza preparedness. Am J Public Health. 2018; 108(11):1469-1472.

US Department of Health and Human Services (HHS). Pandemic Influenza Plan:. 2017 Update. Washington, DC: HHS; 2017. Accessed September 8, 2020. https://www.cdc.gov/flu/pandemic-resources/pdf/pan-flu-report-2017v2.pdf

O'Hagan

, Wong

, Campbell

, et al. Estimating the United States demand for influenza antivirals and the effect on severe influenza disease during a potential pandemic. Clin Infect Dis. 2015; 60(suppl 1):S30-S41.

Fiore

, Fry

, Shay

, et al. Antiviral agents for the treatment and chemoprophylaxis of influenza—recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Recomm Rep. 2011; 60(1):1-24.

Koonin

, Patel

. Timely antiviral administration during an influenza pandemic: key components. Am J Public Health. 2018; 108(suppl 3):S215-S220.

Koonin

, Hanfling

. Broadening access to medical care during a severe influenza pandemic: the CDC nurse triage line project. Biosecur Bioterr. 2013; 11(1):75-80.

Fain

, Koonin

, Stoto

, et al. Facilitating access to antiviral medications and information during an influenza pandemic: engaging with the public on possible new strategies. Biosecur Bioterr. 2014; 12(1):8-19.

Jacobs-Wingo

, Ezeoke

, Saffa

, et al. Using a call center to coordinate Zika virus testing—New York City, 2016. J Emerg Manag. 2016; 14(6):391-395.

Shirai

, Fujiyama

, Uchino

, Irie

, Takatoriges

, Iso

. Evaluation of the flu call center and medical system data on pandemic (H1N1) 2009 influenza. Comparison of cases in Kobe city and Ibaraki prefecture. Article in Japanese. Nihon Koshu Eisei Zasshi. 2012; 59(9):684-692.

10.

Lee

, Bulterys

, Martel

, Dahl

. Evaluation of a national call center and a local alerts system for detection of new cases of Ebola virus disease—Guinea, 2014–2015. MMWR Morb Mortal Wkly Rep. 2016; 65(9):227-230.

11.

Miller

, Stanger

, Senesi

, et al. Use of a nationwide call center for Ebola response and monitoring during a 3-day house-to-house campaign—Sierra Leone, September 2014. MMWR Morb Mortal Wkly Rep. 2015; 64(1):28-29.

12.

Stead

, Perera

, Lancaster

. A systematic review of interventions for smokers who contact quitlines. Tob Control. 2007; 16(suppl 1):i3-i8.

13.

Vassilev

, Marcus

, Jennis

, Ruck

, Rego

. Assessment of the utilization of a state AIDS/STD hotline by persons with and without HIV infection and their information needs. AIDS Patient Care STDS. 2005; 19(10):684-689.

14.

Prior

, Woodward

, Hoefkens

, Clayton

, Thirlaway

, Limbert

. Telephone helplines as a source of support for eating disorders: service user, carer, and health professional perspectives. Eat Disord. 2018; 26(2):164-184.

15.

Rutter

, Mytton

, Ellis

, Donaldson

. Access to the NHS by telephone and internet during an influenza pandemic: an observational study. BMJ Open. 2014; 4(2):e004174.

16.

Rubin

, Amlot

, Carter

, Large

, Wessely

, Page

. Reassuring and managing patients with concerns about swine flu: qualitative interviews with callers to NHS Direct. BMC Public Health. 2010; 10:451.

17.

Pandemic Influenza Preparedness Team. The National Pandemic Flu Service: An Evaluation. London: UK Department of Health; 2011. Accessed September 11, 2020. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/215679/dh_125338.pdf

18.

Spaulding

, Radi

, Macleod

, et al. Design and implementation of a statewide influenza nurse triage line in response to pandemic H1N1 influenza. Public Health Rep. 2012; 127(5):532-540.

19.

Eminovic

, Wyatt

, Tarpey

, Murray

, Ingrams

. First evaluation of the NHS Direct online clinical enquiry service: a nurse-led web chat triage service for the public. J Med Internet Res. 2004; 6(2):e17.

20.

Clancy

, Neuwirth

, Bukowski

. Lessons learned in implementing a 24/7 public health call center in response to H1N1 in the state of New Jersey. Am J Disaster Med. 2009; 4(5):253-260.

21.

Spaulding

, Radi

, Macleod

, et al. Satisfaction and public health cost of a statewide influenza nurse triage line in response to pandemic H1N1 influenza. PLoS One. 2013; 8(1):e50492.

22.

US Centers for Disease Control and Prevention. Interim guidance on the use of antiviral medications for treatment of human infections with novel influenza A viruses associated with severe human disease. Updated April 7, 2015. Accessed September 8, 2020. https://www.cdc.gov/flu/avianflu/novel-av-treatment-guidance.htm

23.

Leiba

, Goldberg

, Hourvitz

, et al. Lessons learned from clinical anthrax drills: evaluation of knowledge and preparedness for a bioterrorist threat in Israeli emergency departments. Ann Emerg Med. 2006; 48(2):194-199, 199.e1-2.

24.

Foote

, Daver

, Quinn

. Using “mystery patient” drills to assess hospital Ebola preparedness in New York City, 2014–2015. Health Secur. 2017; 15(5):500-508.

25.

Clinton-McHarg

, Paul

, Boyes

, Rose

, Vallentine

, O'Brien

. Do cancer helplines deliver benefits to people affected by cancer? A systematic review. Patient Educ Couns. 2014; 97(3):302-309.

26.

O'Cathain

, Knowles

, Turner

, Nicholl

. Acceptability of NHS 111 the telephone service for urgent health care: cross sectional postal survey of users' views. Fam Pract. 2014; 31(2):193-200.

27.

Al-Abdullah

, Plint

, Shaw

, et al. The appropriateness of referrals to a pediatric emergency department via a telephone health line. CJEM. 2009; 11(2):139-148.

28.

Adhikari

, Koonin

, Mugambi

, et al. Estimating weekly call volume to a national nurse telephone triage line in an influenza pandemic. Health Secur. 2018; 16(5):334-340.

29.

Bunn

, Byrne

, Kendall

Telephone consultation and triage: effects on health care use and patient satisfaction. Cochrane Database Syst Rev. 2004(4):CD004180.

30.

National Council of State Boards of Nursing (NCSBN). NLC states. Accessed September 8, 2020. https://www.ncsbn.org/NLC_Map_Updated_June_2019.pdf

31.

Sunshine

, Thompson

, Menon

, Anderson

, Penn

, Koonin

. An assessment of state laws providing gubernatorial authority to remove legal barriers to emergency response. Health Secur. 2019; 17(2):156-161.

32.

US Food & Drug Administration (FDA). Pandemic and All-Hazards Preparedness Reauthorization Act of 2013 (PAHPRA) Medical Countermeasure (MCM) Authorities: FDA Questions and Answers for Public Health Preparedness and Response Stakeholders. Silver Spring, MD: FDA; 2014. Accessed September 8, 2020. https://www.fda.gov/media/87718/download

33.

Schmitt

Call Center Metrics for Meeting the Quadruple Aims. Chandler, AZ: Schmitt-Thompson Clinical Content, 2018. Accessed November 10, 2019. https://cdn.cleartriage.com/wp-content/uploads/STCC_Update_201810.pdf

34.

Nash

, Vickerman

, Kellogg

, Zbikowski

. Utilization of a web-based vs integrated phone/web cessation program among 140,000 tobacco users: an evaluation across 10 free state quitlines. J Med Internet Res. 2015; 17(2):e36.

35.

Pew Research Center. Mobile fact sheet. Published June 12, 2019. Accessed November 10, 2019. https://www.pewresearch.org/internet/fact-sheet/mobile/#:~:targetText=The%20vast%20majority%20of%20Americans,range%20of%20other%20information%20devices

36.

Perrin

, Kumar

About three-in-ten U.S. adults say they are ‘almost constantly’ online. Pew Research Center website. Published July 25, 2019. Accessed September 11, 2020. https://www.pewresearch.org/fact-tank/2019/07/25/americans-going-online-almost-constantly/

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

13.48 MB

	Calls Handled By
Average Call Metrics	Information Specialists	Medical Professionals
Initial IVR greeting and routing to information specialist	0 min 31 sec	N/A
Speed of answer	0 min 16 sec	0 min 45 sec
Time spent waiting in queue initially to speak to information specialist or medical professional	0 min 16 sec	0 min 45 sec
Handle time (across all calls with and without hold time)	6 min 0 sec	15 min 33 sec
Total time an agent or medical professional spent for a call, including actively talking, time caller was on hold (including calls with zero and non-zero hold times), and time to complete required work after hanging up. This does not include the time initially spent in queue.	6 min 0 sec	15 min 33 sec
Talk time	4 min 41 sec	8 min 45 sec
Portion of the total handle time spent actively in conversation	4 min 41 sec	8 min 45 sec
Hold time, all calls	0 min 41 sec	0 min 24 sec
Portion of the total handle time spent on hold, on average, across all calls. This does not include the time initially spent in queue during a transfer.	0 min 41 sec	0 min 24 sec
Hold time, only calls placed on hold	1 min 43 sec	2 min 33 sec
Portion of the total handle time spent on hold between times actively talking for calls placed on a hold. This does not include the time initially spent in queue during a transfer.	1 min 43 sec	2 min 33 sec
After-call wrap-up documentation	0 min 38 sec	6 min 24 sec
Portion of the total handle time spent completing required steps after ending a call and before starting the next.	0 min 38 sec	6 min 24 sec
Total call time experienced by the caller
Includes time from when the call is answered by IVR system to termination of the call. Excludes after-call wrap-up documentation that is not experienced by the caller.
Calls handled only by information specialist	5 min 45 sec
Calls transferred to medical professional after initial intake by information specialist	16 min 9 sec