Using Timeliness Metrics to Track Progress and Identify Gaps in Disease Surveillance

Abstract

Timely outbreak detection and response can translate into illnesses averted and lives saved. As such, timeliness is an important criterion for evaluating performance of infectious disease surveillance systems. Through the use of clearly defined outbreak milestones, timeliness metrics can capture the speed of outbreak detection, verification, response, and other key actions across the timeline of an outbreak and evaluate progress over time. In this article, we describe a series of country-level pilot studies designed to assess the feasibility and utility of tracking timeliness metrics and highlight key findings. We then discuss subsequent efforts to develop a timeliness metrics measurement framework through expert consultation and provide recommendations for implementation. National surveillance programs, international agencies, and donor organizations can use timeliness metrics to identify gaps in surveillance performance and track progress toward improved global health security.

Introduction

The frequency of infectious disease outbreaks is increasing globally, posing a substantial risk to both population health and economic activity.¹ In June 2018, for the first time, active outbreaks of Ebola, Middle East respiratory syndrome, Zika, Nipah virus, Lassa fever, and Rift Valley fever occurred simultaneously, all of which are considered World Health Organization (WHO) priority diseases.² The current COVID-19 pandemic illustrates that emerging diseases are a constant threat, resulting in grave consequences when not quickly detected and contained.^3-5 Early detection and rapid response are crucial for mitigating the impact of outbreaks of emerging and reemerging diseases. Rapidly identifying causative pathogens, ascertaining modes of transmission, and implementing control measures can translate into a marked reduction in morbidity and mortality, minimizing social and economic disruption.^6,7 Given this reality, it is imperative to routinely assess the world's ability to quickly find and contain any high-consequence outbreak at its source. Monitoring the timeliness of outbreak detection, verification, notification, and response can provide insight on surveillance capabilities and indicate where performance might be improved.⁸

Evaluations of outbreak detection and response timeliness are difficult without consistent, standardized methods for analyzing timeliness trends across key surveillance functions. Previous studies in this area have focused on the timeliness of reporting (notification) of cases or outbreaks to national notifiable disease surveillance systems to identify areas for improvement.^8,9 Other studies have examined the length of outbreak investigations and the timeliness of analytical studies to characterize those outbreaks.¹⁰ In the Netherlands, a study assessed the impact of reporting delays on outbreaks, noting that realistic improvements in measles and mumps case reporting, when combined with sufficient vaccine coverage, could make notable improvements in outbreak control conditions.¹¹ While each of these studies makes valuable contributions, they do not establish guidance for routine timeliness evaluations of disease surveillance functions at the national or subnational level. In this article, we state the case for tracking the timeliness of outbreak surveillance actions and share guidance for capturing such measures. At the same time, we acknowledge that timeliness is but a single important aspect for evaluation of disease surveillance systems, and should be considered alongside features such as validity, flexibility, acceptability, stability, and cost of implementation.¹²

Tracking timeliness of outbreak surveillance actions complements existing evaluative frameworks currently used at the country level. Health authorities are required by the 2005 revisions to the International Health Regulations (IHR) to define the etiologic agent, source, and mode of spread when an outbreak occurs within their jurisdiction, and to notify the international community of such information.¹³ Country capacity for conducting core surveillance activities identified by the IHR are monitored through State Parties Self-Assessment Annual Reporting to WHO and the Joint External Evaluation (JEE) process. The State Parties Self-Assessment Annual Reporting (SPAR) tool enables countries to report on 24 indicators related to the 13 IHR capacities needed to detect, assess, notify, report, and respond to public health risk and acute events of domestic and international concern.¹⁴ The JEE is a voluntary, collaborative process to periodically assess country capacity to (1) prevent, detect, and rapidly respond to public health risks; (2) assess country-specific progress; and (3) recommend priority actions to be taken across 19 technical areas.¹⁵ As of April 7, 2020, the JEE process had been completed by 113 countries.¹⁶

While the SPAR and JEE tools have proven valuable in identifying gaps and driving preparedness planning and investments, they focus on assessing country capabilities rather than performance. A framework for assessing the timeliness of disease surveillance can complement such efforts by providing a set of quantitative performance measures for tracking progress routinely and consistently, as efforts like the JEE are implemented only on a periodic basis.

Methods

Chan et al¹⁷ measured and analyzed the trends in timeliness using 4 outbreak milestones applied to events from the WHO's Disease Outbreak News from 1996 to 2009. The authors found that the global speed of outbreak discovery and public communication had improved overall by 7.3% and 6.2% per year, respectively. In 2016, a 5-year update to the Chan et al study indicated improvements in global capacity for timely disease detection had continued, but at a slower rate.¹⁸ However, in both studies the ability to stratify by disease type or examine geographic factors at a level more granular than WHO regions was limited. More granular data at the national and subnational levels must be examined to better characterize these trends and provide actionable insights for governments.

Since 2014, Ending Pandemics has focused on developing definitions for outbreak milestones and corresponding timeliness metrics that can be used at the national and subnational level for monitoring performance in disease surveillance. Ending Pandemics partnered with HealthMap of Boston Children's Hospital, the International Society for Infectious Diseases' Program for Monitoring Emerging Diseases, and the Task Force for Global Health's Training Programs in Epidemiology and Public Health Interventions to adapt the methods from Chan et al for use by national health agencies to assess timeliness of outbreak surveillance actions.¹⁷

At the crux of this effort was identifying and defining key outbreak milestones from which timeliness metrics could be calculated. The identified outbreak milestones represent the dates of key surveillance actions, including date of detection, date of verification, and date of response to an outbreak. Timeliness metrics are defined as the time interval between 2 outbreak milestones. For example, timeliness of outbreak detection was calculated as the time interval between the Outbreak Start and the Detect milestones (Figure 1). Following methods established by Chan et al,¹⁷ the median time for each timeliness metric can be calculated for outbreaks during the selected time period and a univariate Cox proportional hazards regression analysis applied to analyze the changes in median timeliness for each metric. The initial version of outbreak milestones used to establish the timeliness metrics for country-level studies are described in Smolinski et al¹⁹ and provided in Table 1.

Figure 1.

Illustration of example outbreak milestones and timeliness metrics.

Table 1.

Pilot Study Outbreak Milestone Definitions

Outbreak Milestones	Definition^a
Outbreak start	Date of the symptoms onset of the index case. If it is not available, first date of hospitalization or medical visit may be used.
Detect	Date that the outbreak threshold^b case presents to a hospital, clinic, laboratory, community health worker, or public health agency.
Report	Date that the outbreak threshold case is reported to public health authorities at local, regional, national, or international level.
Lab confirmation	Date of the first laboratory report of the causative pathogen from an epidemiologically linked case.
Respond	Earliest date when the local public health professionals took actions to stop or control the outbreak in the community.
Public communication	Date that public information about the outbreak appeared in local or international, informal or official, verbal or written reports, including the official press releases or newspapers in the country, news articles, TV or radio broadcasts, internet postings, social media, or informal disease reporting networks such as ProMED or HealthMap.

As described in Smolinski et al¹⁹; ^bOutbreak threshold refers to minimum number of cases and other criteria required to declare an outbreak for a particular disease.

Using the outbreak milestones developed for use at the national level, Ending Pandemics provided financial support and technical assistance for a series of retrospective studies designed to assess the feasibility of measuring the timeliness of key disease surveillance actions. Of primary interest was assessing the availability of outbreak milestones (date variables) within existing disease surveillance systems. These data were identified, extracted, and in many cases supplemented by additional documentation. Available outbreak milestones were then leveraged to assemble outbreak timelines and measure timeliness trends. Studies were initiated by field epidemiology training programs (FETPs) representing 18 countries: Barbados, Haiti, India, Indonesia, Kenya, Pakistan, Paraguay, Senegal, Taiwan, Tanzania, Zimbabwe, and a Central American collaboration through the Council of Ministers of Health of Central America that included Belize, Costa Rica, Dominican Republic, El Salvador, Guatemala, Honduras, and Panama. Two studies were conducted in India, 1 by the Chennai-based FETP (part of the National Institute of Epidemiology) and 1 by the Delhi-based FETP (housed within the Ministry of Health).

Similar pilots were developed with 4 countries in the Mekong Basin Disease Surveillance (MBDS) network (Cambodia, Laos, Myanmar, Vietnam) and 5 countries in the Southeast European Center for Surveillance and Control of Infectious Diseases (SECID) network (Albania, Bosnia and Herzegovina, Bulgaria, Kosovo, Macedonia). Both MBDS and SECID operate as multicountry, regional disease surveillance networks focused on detection and control of disease outbreaks.^20-22

Countries established time periods for inclusion of retrospective outbreak data ranging between 5 and 12 years. Studies conducted by FETPs assessed infectious disease outbreaks during their selected time period, in many cases applying some of the exclusion criteria established by Chan et al,¹⁷ such as excluding foodborne outbreaks, nonnatural cases due to laboratory accidents, or single cases of disease. Further detail on exclusion criteria can be found in Table 2. The MBDS and SECID networks each selected specific priority diseases for evaluation within their respective networks.^21,22 When possible, countries stratified results by pathogen, disease type, and/or subnational geography to assess gaps in surveillance. Some countries also examined timeliness trends before and after key public health interventions to assess their potential impact on timeliness.

Table 2.

Retrospective Pilot Study Results From Field Epidemiology Training Programs

Country	Time Period (# Years)	Diseases Included in Study	# Outbreaks	Timeliness Metrics Captured (Median)^a
Pakistan	2003-2013 (10)	All outbreaks excluding foodborne, nonnatural cases (bioterror/lab accident), isolated/single cases	56	Detection: 8 days Public communication: 11 days
Zimbabwe	2003-2013 (10)	Anthrax, cholera, malaria, measles, rabies, H1N1, typhoid, rotavirus, varicella, hepatitis, plague	134	Median detection time: trended from 6 days in 2003 to 2 days in 2013
Senegal	2003-2013 (10)	All outbreaks excluding foodborne, nonnatural cases (bioterror/lab accident), isolated/single cases	61	Detection: 6 days Reporting: 6 days Lab confirmation: 6.5 days Response: 6 days Public communication: 8 days
India (Chennai)	2003-2014 (10)	All outbreaks excluding nonnatural cases (bioterror/lab accident), isolated/single cases	999	Detection: 2 days Response: 2 days Lab confirmation: 8 days Public communication: 4 days
Kenya	2004-2014 (10)	All outbreaks excluding foodborne, nonnatural cases (bioterror/lab accident), isolated/single cases	460	Detection: 1 day (0-26 range) Reporting: 0 days (0-155 range)
Indonesia	2005-2017 (12)	All outbreaks excluding nonnatural cases (bioterror/lab accident), isolated/single cases	219	Detection: 7 days Response: 1 day Public communication: 6 days
Taiwan	2006-2013 (8)	All outbreaks excluding foodborne, nonnatural cases (bioterror/lab accident), isolated/single cases	1,874	Detection: 6 days (1-96 range)
Paraguay	2006-2015 (10)	All outbreaks excluding nonnatural cases (bioterror/lab accident), isolated/single cases	67	Reporting: decreased 25 days to 8 days post-FETP implementation Response: decreased from 4 days (0-15) to 1 day (0-29)
Belize	2007-2015 (9)	All outbreaks excluding nonnatural cases (bioterror/lab accident), isolated/single cases	24	Detection: 3 days (0-12 range) Public communication: 1 day (0-10 range)
Honduras	2007-2015 (9)	All outbreaks excluding nonnatural cases (bioterror/lab accident), isolated/single cases	41	Detection: 3 days (1-10 range) Response: 7 days (3-15 range)
India (Delhi)	2009-2014 (6)	All outbreaks excluding nonnatural cases (bioterror/lab accident), isolated/single cases	7,092	Detection: 3 days (0-160 range)

Direct cross-country comparisons are not advised due to variations in time period, disease type, and total number of outbreaks examined. Select results from the studies are shown here as examples.

Abbreviations: FETP, field epidemiology training program.

Results

Data collection, analysis, and reported results varied by country due to differing priority diseases selected and wide variation in data availability. We will not attempt to comprehensively detail such a diverse set of studies, but we do share key findings. In total, 19 of the initial 27 countries participating in the retrospective studies were able to complete analyses of timeliness metrics, resulting in 20 studies to draw from due to the participation of 2 India-based FETPs. Of the 8 countries that did not complete analysis of timeliness metrics, most cited lack of data availability as the primary barrier to execution. The remaining 20 studies collectively reviewed 153 years of outbreak data. Individual country studies ranged in size from 56 outbreaks to 7,092 outbreaks over time frames ranging from 5 to 12 years, spanning 2003 through 2017. Results reported here are drawn from a series of published papers, unpublished project reports submitted by FETPs, and workshop presentations. A summary of results from FETP studies can be found in Table 2. Published papers and presentations are cited for the MBDS and SECID network results.^21,22

Data availability for outbreak milestones varied considerably from country to country. No participating country consistently collected structured data for every outbreak milestone, although many were able to calculate trends for multiple timeliness metrics. Identifying dates of the outbreak milestones frequently required extensive data extraction processes for many countries. Missing data and lack of standardized data collection practices were common. Of the 20 studies, 15 (75%) were able to describe results for timeliness of outbreak detection and 12 (60%) described the timeliness of outbreak notification.* In addition, 14 studies (70%) described the timeliness of outbreak response. Results for timeliness of laboratory confirmation and public communication were less common, with 7 (35%) and 9 (45%) studies reporting these metrics, respectively.

Of the 20 studies reported here, 13 included data on timeliness trends for each year included in the study timeline. The 7 countries not reporting yearly results calculated median measures of timeliness for the entire study period to provide a baseline from which further progress can be measured. Of the 13 studies with yearly timeliness data, 10 noted improvements in surveillance timeliness and 3 reported either negligible change or mixed results across specific timeliness metrics. For example, Senegal reported improvement in outbreak detection time, from a median of 10 days in 2003 to a median of 3.5 days in 2013.²³ Zimbabwe also found an overall significant reduction in outbreak detection time, from a median of 6 days in 2003 to a median of 1 day in 2013.²⁴ Furthermore, Honduras and Belize saw an overall improvement in timeliness of outbreak detection, from a median of 12 days in 2007 to a median of 2.5 days in 2016.²⁵

Timeliness metrics were used by some pilot countries to assess the impact of specific interventions designed to improve disease surveillance. Indonesia cited implementation of the Early Warning Alert and Response System as contributing to improvements in the speed of outbreak detection; a Cox hazards regression demonstrated that the speed of outbreak detection had improved (hazard ratio, 1.8; 95% confidence interval [CI], 1.4 to 2.4) from 9 days (range 3 to 27) to 6 days (range 4 to 8).²⁶ Meanwhile a Cox hazards regression indicated that Integrated Disease Surveillance and Response implementation in Senegal was correlated with average annual improvements in timeliness of outbreak detection (15.2%), outbreak reporting (14.8%), laboratory confirmation (12.1%), public health response (14.3%), and public communication (9.8%) over a 10-year period.²³

The establishment of an FETP in some countries was correlated with both improvements in timeliness and an increase in outbreak reporting. For example, in Paraguay, establishment of an FETP in 2011 was correlated with a decrease in median outbreak reporting time from 25 days to 8 days, and a decrease in median response time from 4 days to 1 day.²⁷ Paraguay also noted a statistically significant increase of 87% in the overall number of outbreak reports available for the period after the FETP was established. A similar trend was also noted in Zimbabwe, with 86% of outbreak reports available during the study period following the establishment of an FETP in 2005.

Several countries stratified data at the subnational level to further describe differences in timeliness, including India, Indonesia, Pakistan, Senegal, Taiwan, and Zimbabwe. For example, the Chennai-based FETP in India noted a small but statistically significant gap between urban and rural settings in Tamil Nadu, with outbreak response times in rural communities taking twice as long (median of 2 days) as their urban counterparts (median of 1 day).²⁸ Furthermore, the study conducted by the Indonesia FETP found that timeliness varied across the country, as outbreaks on Java Island (median of 6 days) were detected twice as fast compared with outbreaks in other areas (hazard ratio, 2.1; 95% CI, 1.5 to 2.9).²⁶

In the Mekong Basin, results from studies conducted in Cambodia, Laos, Myanmar, and Vietnam are available in a study by Lawpoolsri et al,²¹ with median reporting times, response times, and time to public communication broken down by year and disease. The countries agreed to examine a shared list of priority diseases that included dengue, food poisoning and diarrhea, diphtheria, measles, H5N1 and H1N1 influenza, rabies, and pertussis. Each country examined 6 years' worth of outbreak data from 2010 to 2015, ranging from 288 outbreaks to 715 outbreaks per country. The timeliness of outbreak notification, response, and communication to the public varied greatly, depending on the country, disease, and year.

An early analysis of SECID country data by Mersini et al,²² presented at the International Meeting on Emerging Diseases and Surveillance in November 2018, examined timeliness trends from Albania, Bosnia and Herzegovina, Bulgaria, Macedonia, and Kosovo. In addition to median time to outbreak detection, reporting, and laboratory confirmation, the 5 SECID countries also reported results on the median time to patient hospitalization and median time to patient treatment. Each country selected priority diseases of particular relevance to them, including anthrax, brucellosis, Crimean-Congo hemorrhagic fever, measles, and tularemia. Across the participating countries, the median detection time was found to be 1 day; median time to laboratory confirmation, 8 days; median time to patient hospitalization, 3 days; and median time to treatment, 4 days. A full analysis of SECID network data is currently in development at the time of this writing.

Discussion

Development of a timeliness metrics framework arose out of a desire to assist countries in monitoring progress toward meeting the requirements of the IHR. When the first round of pilot studies was initiated in 2014, nearly 80% of the countries were either not meeting their IHR obligations or failing to report on their progress.²⁹ At that time, the JEE process was early in development, with the initial Global Health Security Agenda pilot assessments taking place in 2015 and the first JEE tool published by WHO in February 2016.³⁰ Initial planning for the Global Health Security Index, published in 2019, was only just beginning.^31,32 Our goal was to develop and advocate for a set of key indicators that could be measured through existing surveillance infrastructure and robust enough to enable identification of gaps and areas of success.

Timeliness metrics are a critical and complementary evaluation tool for pandemic preparedness. Scores from the Global Health Security Index and the JEEs have not necessarily correlated with a country's ability to respond to the COVID-19 pandemic, emphasizing the need to monitor both capabilities and performance.³³ Hence, timeliness metrics generated by countries for the purpose of improving their own systems may help monitor progress toward meeting the IHR and provide metrics that can feed into larger evaluation efforts.

In the pilot studies highlighted here, the majority of country participants found the timeliness metrics to be useful for evaluating their infectious disease surveillance systems. Examining timeliness trends helped to identify gaps in surveillance capacity and data collection practices. Challenges and limitations, however, surfaced during the retrospective studies, including limited resources for data collection and extraction activities, lack of date variables for establishing outbreak milestones, and incomplete data. Nevertheless, the results reported here demonstrate the feasibility of measuring timelines for outbreak surveillance, which in turn provides a starting point for understanding influencing factors that can be targeted for investment.

Ending Pandemics convened 2 international expert consultations in 2018 and 2019 with more than 60 representatives from 43 health agencies, nongovernmental organizations, universities, and foundations from the human, animal, and environmental health sectors to disseminate lessons learned from the pilots and address the aforementioned challenges. The meetings were conducted in partnership with the Salzburg Global Seminar to spearhead revisions to the outbreak milestone definitions, identify strategies to enable routine data collection, and consider how timeliness trends can be interpreted to inform improvements to surveillance systems across the One Health spectrum. One Health is a collaborative, multisectoral, and transdisciplinary approach that recognizes the interconnection between people, animals, plants, and their shared environment.³⁴

In-depth discussion and rigorous debate at the initial meeting resulted in 8 outbreak milestones with updated definitions for use by both public health agencies and other health organizations. The meeting summary and list of participants can be found in the meeting report, Finding Outbreaks Faster: How Do We Measure Progress?³⁵ The subsequent meeting in 2019 focused on revising the outbreak milestones further to incorporate environmental monitoring for prediction and prevention, as well as for use in animal outbreak surveillance.³⁶ The 11 outbreak milestones found in Table 3 reflect the broadening of definitions to ensure applicability across the One Health spectrum.³⁶

Table 3.

Updated Outbreak Milestone Definitions, 2019

Outbreak Milestone	Definition^a
Predict	Date a reliable and valid predictive alert of a potential outbreak is available (eg, increased rainfall leading to greater density of mosquitoes capable of disease transmission)
Prevent	Date enhanced surveillance or other intervention (eg, mass vaccination in livestock, mosquito abatement) is initiated in response to a predictive alert
Detect	Date symptom onset, death, or other evidence of pathogen circulation is observed or suspected in humans or animals
Notify	Date an outbreak in humans or animals is officially reported to relevant authorities (eg, local to national, national to international, cross-sector)
Verify	Date outbreak is confirmed by field investigation or other valid method
Diagnostic test/lab confirmation	Date outbreak is confirmed by diagnostic or laboratory test in an epidemiologically linked human or animal
Respond	Date an intervention to control or manage the outbreak (eg, mass vaccination, quarantine) is initiated by a responsible authority
Public communication	Date of official release of information to the public by a responsible authority
Outbreak start	Date symptom onset or death occurs in the earliest epidemiologically linked human or animal (most often identified retrospectively or estimated based on available evidence)
Outbreak end	Date outbreak is declared closed by a responsible authority
After-action review^b	Date after-action review is jointly conducted by relevant One Health authorities

Definitions have been updated to reflect a One Health approach and ensure applicability across human, animal, and environmental health sectors. The sequence of the milestones may vary by outbreak. In some cases, a single action may represent more than 1 milestone. The definition of an outbreak may vary by disease, geography, or sector.

The after-action review milestone is included to inspire the necessary collaborations among sectors for operationalizing One Health.

These standardized outbreak milestones enable countries and public health organizations to define and calculate timeliness metrics relevant to them. Importantly, it was agreed that routine collection of the outbreak milestones would alleviate many of the challenges identified in the pilot studies. Modifications to event management systems and after-action report forms provide clear opportunities to routinely capture outbreak milestones where many of these variables may already be collected. Using such opportunities as starting points, countries can explore how to capture outbreak milestones in all relevant surveillance systems to ensure complete and continual data capture.

Guidance to support implementation of these metrics is being developed in partnership with several stakeholders. The outbreak milestones are now included in WHO's country guidance on after action reviews and simulation exercises³⁷ while also being used by the US Centers for Disease Control and Prevention to evaluate event-based surveillance tools.³⁸ Furthermore, WHO has made timeliness of detection, response, and notification to WHO a key component of their methods for measuring progress toward the Thirteenth General Programme of Work goal of “1 billion people better protected from health emergencies.”³⁹ Most recently the WHO Regional Office for Africa published a study examining the dates of onset, detection, notification, and outbreak end for 296 events between 2017 and 2019 using similar methods to those shared here.⁴⁰

Evaluation frameworks that account for the drivers of timeliness trends can help assess the impact of investments in global health security for improving disease surveillance. Outbreak milestones captured by existing disease surveillance systems can be leveraged to identify gaps and highlight areas for improvement. The timeliness metrics can also enable the identification of high-performing programs and spur further exploration into the reasons for success.

Regional disease surveillance networks such as MBDS and SECID provide a platform to examine regional disease surveillance strengths and weaknesses. That said, caution should be exercised when comparing timeliness metrics between or among countries due to variable capabilities, burden of disease, and socioeconomic factors. The timeliness metrics presented in this article are intended to help individual countries assess their progress over time rather than function as comparative metrics across countries.

The outbreak milestones and associated timeliness metrics were developed with the recognition that not all countries or organizations have the capacity or intent to capture each of the outbreak milestones. The pilot studies previously described have demonstrated country-level variance for which outbreak milestones may be captured by existing routine surveillance systems. Countries may consider establishing a minimum subset of outbreak milestones that can be integrated into all routine surveillance systems. Adoption of the entire set of milestones may provide the impetus to revise surveillance systems, outbreak investigations forms, and related systems to capture all necessary data points. As previously noted, the timeliness metrics identified by WHO to measure progress toward the Thirteenth General Programme of Work goal of “1 billion people better protected from health emergencies” may provide a reasonable starting point for many countries.

Due consideration should also be given to the expanded use of timeliness metrics in the animal and environmental health sectors. Embracing a One Health approach within the timeliness metrics framework may further our understanding of how a multisectoral approach better prepares the world to react to emerging health threats. Ending Pandemics will continue to promote the use of timeliness metrics across the human, animal, and environmental health sectors, including efforts to work with partner countries to integrate outbreak milestones into their event management systems, DHIS2 platforms, and other surveillance tools. Efforts also include coordinating with WHO, the Food and Agriculture Organization of the United Nations, and the World Organisation for Animal Health on efforts to capture timeliness metrics in their systems, enabling the use of timeliness metrics through training workshops for FETPs, and linking with workforce development efforts to ensure the inclusion of content that highlights these methods. This work will further inform best practices for capturing these milestones and metrics, and shape future recommendations and guidance. The resulting timeliness data may not only help identify factors influencing surveillance performance, but also have the potential to inform economic and impact analyses that link detection and response times with outcomes.

The pilot studies cited focused on assessing feasibility of acquiring outbreak milestone data and tracking timeliness trends. This article is not intended to be an in-depth assessment of timeliness trends reported by each country, rather it focuses on the potential utility of such an approach and provides suggestions for future use. Pilot study exclusion criteria varied by country, and, therefore, interpretation of timeliness trends should be done with particular consideration for what type of diseases may have been excluded from the study. Pilot studies report a wide range of total number of outbreaks across the study periods. Hence, timeliness trends from studies with a particularly low number of outbreaks should be interpreted with care. The outbreak milestone definitions are intentionally broad to ensure applicability across a wide range of health organizations; further specificity may be needed for individual organizations to operationalize the use of outbreak milestones.

Footnotes

Acknowledgments

The authors would like to acknowledge our many partners who led the pilot studies discussed here, including Aslam Pervaiz (Pakistan FETP), Augustina Silva (Paraguay FETP), Daniella Azor (Haiti FETP), Donewell Bangure (Zimbabwe FETP), David Rodriguez (Council of Ministers of Health of Central America and the Dominican Republic), Hao Yuan-Cheng (Taiwan FETP), Ibrahima Sonko (Senegal FETP), Joan Karanja (Kenya FETP), Nur Aini Kusmayanti (Indonesia FETP), Preeti Padda and Viduthalai Virumbi (India FETPs), Shaba Kilasi (Tanzania FETP), and our colleagues at Training Programs in Epidemiology and Public Health Interventions Network (TEPHINET). In addition we would like to thank our partners in MBDS and SECID for their work in applying this framework in a collaborative, multi-country approach in Southeast Asia and Southeastern Europe, respectively. Finally we would like to thank all of the participants from the “Finding Outbreaks Faster” convenings at the Salzburg Global Seminar in 2018 and 2019 whose contributions helped to refine and advance this framework.

References

Smith

, Goldberg

, Rosenthal

, et al. Global rise in human infectious disease outbreaks. J R Soc Interface. 2014; 11(101):20140950.

Weber

More dangerous outbreaks are happening. Why aren't we worried about the next epidemic? Huffington Post. July 18, 2018. Updated July 19, 2018. Accessed March 2021 . https://www.huffingtonpost.com/entry/outbreaks-epidemic-preparedness_us_5b4f85fbe4b0de86f4892daa

Bambery

, Cassell

, Bunnell

, et al. Impact of a hypothetical infectious disease outbreak on US exports and export-based jobs. Health Secur. 2018; 16(1):1-7.

Sands

, Mundaca-Shah

, Dzau

. The neglected dimension of global security–a framework for countering infectious-disease crises. N Engl J Med. 2016; 374(13):1281-1287.

, Zhao

, Yu

, et al. A new coronavirus associated with human respiratory disease in China. Nature. 2020; 579:265-269.

Zhang

, Lopez-Gatell

, Alpuche-Aranda

, Stoto

. Did advances in global surveillance and notification systems make a difference in the 2009 H1N1 pandemic?–a retrospective analysis. PLoS One. 2013; 8(4):e59893.

Blake

, Chenoweth

, Okayasu

, Donnelly

, Aylward

, Grassly

. Faster detection of poliomyelitis outbreaks to support polio eradication. Emerg Infect Dis. 2016; 22(3):449-456.

Jajosky

, Groseclose

. Evaluation of reporting timeliness of public health surveillance systems for infectious diseases. BMC Public Health. 2004; 4:29.

Yoo

, Park

, et al. Timeliness of national notifiable diseases surveillance system in Korea: a cross-sectional study. BMC Public Health. 2009; 9:93.

10.

van de Venter

, Oliver

, Stuart

. Timeliness of epidemiological outbreak investigations in peer-reviewed European publications, January 2003 to August 2013. Euro Surveill. 2015; 20(6):21035.

11.

Marinović

, Swaan

, van Steenbergen

, Kretzschmar

. Quantifying reporting timeliness to improve outbreak control. Emerg Infect Dis. 2015; 21(2):209-216.

12.

Buehler

, Hopkins

, Overhage

, Sosin

, Tong V

, C Working

Group

. Framework for evaluating public health surveillance systems for early detection of outbreaks: recommendations from the CDC Working Group. MMWR Recomm Rep. 2004; 53(RR-5):1-11.

13.

World Health Organization (WHO). International Health Regulations (2005). 2nd ed. Geneva: WHO; 2008. Accessed March 24, 2021. http://www.who.int/ihr/9789241596664/en/

14.

World Health Organization. e-SPAR: State Party Annual Report. Accessed March 24, 2021. https://extranet.who.int/e-spar

15.

World Health Organization. Joint External Evaluation (JEE) mission reports. Accessed March 24, 2021. http://www.who.int/ihr/procedures/mission-reports/en/

16.

World Health Organization. IHR Country Capacity Assessment, Monitoring, Evaluation, and Planning Update. Published November 13, 2020.

17.

Chan

, Brewer

, Madoff

, et al. Global capacity for emerging infectious disease detection. Proc Natl Acad Sci U S A. 2010; 107(50):21701-21706.

18.

Kluberg

, Mekaru

, McIver

, et al. Global capacity for emerging infectious disease detection, 1996–2014. Emerg Infect Dis. 2016; 22(10):E1-E6.

19.

Smolinski

, Crawley

, Olsen

. Finding outbreaks faster. Health Secur. 2017; 15(2):215-220.

20.

Connecting Organizations for Regional Disease Surveillance. CORDS. Accessed March 24, 2021. https://www.cordsnetwork.org/cords-networks/

21.

Lawpoolsri

, Kaewkungwal

, Khamsiriwatchara

, et al. Data quality and timeliness of outbreak reporting system among countries in Greater Mekong subregion: challenges for international data sharing. PLoS Negl Trop Dis. 2018; 12(4):e0006425.

22.

Mersini

, Sulo

, Agolli

, et al. Evaluation of main infectious diseases outbreak detection and response timeliness in Southeast European Region. Int J Infect Dis. 2019; 79(suppl 1):129.

23.

Sonko

. Mini Grant Status Report: Timeliness of Detection of Infectious Disease Outbreaks in Senegal, 2003-2013. Training Programs in Epidemiology and Public Health Interventions Network; 2016.

24.

Bangure

. Mini Grant Status Report: Faster Detection of Infectious Disease Outbreaks in Zimbabwe, 2003-2013. Training Programs in Epidemiology and Public Health Interventions Network (TEPHINET); 2016.

25.

Rodriguez Araujo

. Mini Grant Status Report: Faster Detection of Infectious Disease Outbreaks in Central America: 2007-2015. Training Programs in Epidemiology and Public Health Interventions Network (TEPHINET); 2017.

26.

Kusmayanti

. Mini Grant Status Report: Timeliness of Communicable Disease Outbreak Milestone in Indonesia During 2005-2017 - A National Picture. Training Programs in Epidemiology and Public Health Interventions Network (TEPHINET); 2017.

27.

Rojas

, Samudio

, Agueda

. Mini Grant Status Report: Evaluation of the Outbreak Investigation Capacity Since the FETP Implementation in Paraguay (2006-2015). Training Programs in Epidemiology and Public Health Interventions Network (TEPHINET); 2017.

28.

Virumbi Balalgurusamy

. Mini Grant Status Report: Assessment of Timeliness in Detection and Reporting of Infectious Disease Outbreaks in Tamil Nadu, India, 2003-2014. Training Programs in Epidemiology and Public Health Interventions Network (TEPHINET); 2017.

29.

World Health Organization (WHO). Implementation of the International Health Regulations (2005). Geneva: WHO; 2013. Accessed March 24, 2021. http://apps.who.int/gb/ebwha/pdf_files/WHA66/A66_16-en.pdf

30.

Kandel

, Sreedharan

, Chungong

, Mahjour

. The Joint External Evaluation tool, second edition: changes, interpretation, and use. Health Secur. 2019; 17(3):248-250.

31.

Moore

, Gelfeld

, Okunogbe

, Paul

. Identifying future disease hot spots: Infectious Disease Vulnerability Index. Rand Health Q. 2017; 6(3):5.

32.

Nuclear Threat Initiative, Johns Hopkins Center for Health Security, The Economist Intelligence Unit. The Global Health Index. Accessed January 3, 2020. https://www.ghsindex.org/

33.

Haider

, Yavlinsky

, Chang

, et al. The Global Health Security index and Joint External Evaluation score for health preparedness are not correlated with countries' COVID-19 detection response time and mortality outcome. Epidemiol Infect. 2020; 148:e210.

34.

US Centers for Disease Control and Prevention. One Health. Updated February 22, 2021. Accessed March 18, 2021. https://www.cdc.gov/onehealth/index.html

35.

Ending

Pandemics

, Salzburg Global

Seminar

. Finding outbreaks faster: how do we measure progress? Published

March 2019

. Accessed

March 24

. 2021. http://endingpandemics.org/wp-content/uploads/2019/06/EP-Salzburg-Global-Seminar-2019.pdf

36.

Ending Pandemics, Salzburg Global Seminar. The Salzburg Statement on Metrics for One Health Surveillance. Washington, DC, and Salzburg, Austria: Ending Pandemics and Salzburg Global Seminar; 2020. Accessed March 24. 2021. https://www.salzburgglobal.org/fileadmin/user_upload/Documents/2010-2019/2019/Session_641/SalzburgGlobal_Statement_641_One_Health.pdf

37.

World Health Organization (WHO).. Country Implementation Guidance: After Action Reviews and Simulation Exercises Under the International Health Regulations 2005 Monitoring and Evaluation Framework (IHR MEF). Geneva: WHO; 2018. Accessed March 24. 2021. https://apps.who.int/iris/handle/10665/276175

38.

Hercik

. Strengthening event-based surveillance in the African

Region

. Paper presented at: EIOS Global Technical Meeting;

November 14

, 2019; Seoul

Republic of Korea

. Accessed

March 24

. 2021. https://www.who.int/docs/default-source/eios-gtm-2019-presentations/37---hercik-eios-gtm-2019.pdf?sfvrsn=2389a7c7_2

39.

World Health Organization (WHO).. Thirteenth General Programme of Work (GPW13): Methods for Impact Measurement. Version 2.1. Geneva: WHO; 2020. Accessed March 24. 2021. https://www.who.int/publications/m/item/thirteenth-general-programme-of-work-(gpw13)-methods-for-impact-measurement

40.

Impouma

, Roelens

, Williams

, et al. Measuring timeliness of outbreak response in the World Health Organization African Region, 2017-2019. Emerg Infect Dis. 2020; 26(11):2555-2564.