Syndromic Surveillance System for Korea–US Joint Biosurveillance Portal: Design and Lessons Learned

Abstract

Driven by the growing importance of situational awareness of bioterrorism threats, the Republic of Korea (ROK) and the United States have constructed a joint military capability, called the Biosurveillance Portal (BSP), to enhance biosecurity. As one component of the BSP, we developed the Military Active Real-time Syndromic Surveillance (MARSS) system to detect and track natural and deliberate disease outbreaks. This article describes the ROK military health data infrastructure and explains how syndromic data are derived and made available to epidemiologists. Queries corresponding to 8 syndromes, based on published clinical effects of weaponized pathogens, were used to classify military hospital patient records to form aggregated daily syndromic counts. A set of ICD-10 codes for each syndrome was defined through literature review and expert panel discussion. A study set of time series of national daily counts for each syndrome was extracted from the Defense Medical Statistical Information System between January 1, 2011, and May 31, 2014. A stratified, adjusted cumulative summation algorithm was implemented for each syndrome group to signal alerts prompting investigation. The algorithm was developed by calculating sensitivity to sets of 1,000 artificial outbreak signals randomly injected in the dataset, with each signal injected in a separate trial. Queries and visualizations were adapted from the Suite for Automated Global bioSurveillance. Findings indicated that early warning of outbreaks affecting fewer than 50 patients will require analysis at subnational levels, especially for common syndrome groups. Developing MARSS to improve sensitivity will require modification of underlying syndromic diagnosis codes, engineering to coordinate alerts among subdivisions, and enhanced algorithms. The bioterrorist threat in the Korean peninsula mandates these efforts.

The Republic of Korea and the United States have constructed a joint military capability, called the Biosurveillance Portal, one component of which is the Military Active Real-time Syndromic Surveillance (MARSS) system to detect and track natural and deliberate disease outbreaks. This article describes the ROK military health data infrastructure and explains how syndromic data are derived and made available to epidemiologists.

The frequency of infectious disease outbreaks since the 2009 influenza pandemic is unprecedented, with each outbreak potentially threatening humanity and highlighting future challenges.^1-3 Beyond pandemics and other naturally occurring global health threats, bioterrorism also remains an enduring hazard to global safety as asymmetric threats increase with development of advanced weaponization and dissemination technologies.⁴ To harmonize national efforts in response to these perceived threats, the Republic of Korea (ROK) and the United States have since 2011 conducted a series of exercises called Able Response, solely dedicated to the bioterrorism threat.⁵ The mission of the exercises is to improve the ROK-US partnership's ability to prepare for and respond to a naturally occurring or intentional biological incident by employing a whole-of-government approach through bilateral cooperative engagements. These activities exemplify a sense of urgency in both nations to strengthen preparedness and response activities for a potential act of bioterrorism.

In addition to the particular emphasis on biodefense, due to the geopolitical context on the Korean peninsula,⁶ global risks of emerging biothreats have increased, including overuse of antibiotics, shifted environmental risk of vectorborne disease due to climate change, and special concerns about the rapid spread of acute respiratory infections through public transportation.⁷ Recent outbreaks of Ebola virus disease in West Africa and Middle East respiratory syndrome coronavirus (MERS-CoV) in the ROK also demonstrated rapid disease transmission across national borders.^8,9 To meet these challenges, the ROK has periodically evaluated and enhanced the plan to respond to various public health emergencies, conducting numerous exercises and gathering strategic national stockpiles of antibiotics, vaccines, and other supplies.¹⁰

Biosecurity has been an essential element of national security for many years in the United States as well, with both domestic and global aims of protecting military and civilian personnel and preventing or mitigating global health threats at the source.¹¹ For this reason, following publication in the United States in 2012 of the National Strategy for Biosurveillance,¹¹ the Joint Program Executive Office for Chemical and Biological Defense planned the Biosurveillance Portal (BSP) as a network of systems to supply information needed for biosurveillance preparedness and particularly for leveraging existing surveillance systems and sharing critical information with strategic allies.¹² It was designed and developed as an unclassified web application in a cloud environment using the Ozone Widget Framework (OWF) architecture.¹³ In view of the perceived plausible threat of bioterrorist attacks in the Korean Peninsula, the ROK Ministry of National Defense (MND) and US Department of Defense (DoD) collaborated to form the first operational BSP late in 2014.¹² This implementation was intended to facilitate international collaboration, communication, and information sharing in support of the detection, management, and mitigation of man-made and naturally occurring biological events. The ROK-US partnership on the BSP represents a key global biosurveillance engagement, and the value of this relationship was demonstrated in the annual joint Able Response exercises.¹⁴

In planning for the Able Response 2014 exercise, the ROK and US planners identified a key need for a disease surveillance system to augment biosecurity efforts in the ROK. The Armed Forces Medical Command (AFMC) under the ROK MND was designated as the lead military agency to integrate current existing capabilities to enhance overall public health planning and actions in the ROK armed forces. To enable this integration, Armed Forces Medical Command and BSP teams, including the Johns Hopkins University Applied Physics Laboratory (JHUAPL), developed a set of analytic tools to detect and track abnormal patterns of infectious disease among armed forces, using capabilities from the Suite for Automated Global bioSurveillance (SAGES).¹⁵ Taking advantage of the current military health information systems, the AFMC designed and built the Military Active Real-time Syndromic Surveillance (MARSS) system for early detection and characterization of health threats. In initiating MARSS, we aimed to integrate a syndromic surveillance capability with other BSP situational awareness components.

Objectives of the study were to (1) determine the syndromes, defined as strategic groupings of diagnosis codes, with emphasis on surveillance for bioterrorism; (2) establish the International Classification of Diseases-10 (ICD-10) code sets for the selected syndromes; and (3) identify a sufficiently flexible and robust detection algorithm for the selected syndromes with syndrome-specific parameters so that we can recommend this syndromic surveillance as an effective tool for force protection.

Methods

The study protocol was approved by the AFMC under the ROK Ministry of National Defense Institutional Review Board (2014-KMMRP-024).

Study Data

The AFMC operates 19 military hospitals across South Korea, and each hospital uses the n-Defense Electronic Medical Information System (n-DEMIS) for collecting and managing electronic medical records. Small-scale military treatment facilities use the simplified version e-DEMIS. Every day at midnight, de-identified n-DEMIS information is transferred to the Defense Medical Statistical Information System (DMSIS), integrating the day's records in a single repository (Figure 1).

Figure 1.

Schematic of information sources and flow of information in the Military Active Real-time Syndromic Surveillance system for Biosurveillance Portal. DEMIS = Defense Electronic Medical Information System; DMSIS = Defense Medical Statistical Information System; MTF = Medical Treatment Facility.

Our study dataset was the collection of n-DEMIS records from January 1, 2012, to May 31, 2014, including encounter date, ICD-10 codes, and patient demographic fields. We categorized these records into syndrome groupings based on literature review and expert panel discussion. Using these categories, each medical encounter record was assigned to 0, 1, or multiple syndrome groups. For example, an encounter resulting from an injury would likely be assigned no infectious disease syndrome, while a febrile respiratory presentation might be assigned to Fever and Respiratory syndromes. For analysis week, we tabulated the number of records assigned to each syndrome to form time series of counts. We divided each series by total number of weekly facility visits, regardless of syndrome classification, and monitored the separate syndromic series adjusted for total visits.

Alerting Method Objectives and Criteria

Many statistical algorithms have been proposed for prospective alerting of syndromic time series to improve situational awareness.¹⁶ For the MARSS system, we used an adjusted cumulative sum control chart (CuSUM).¹⁷ This approach has proved superior to the standard Early Aberration Reporting System methods on simulated data streams.^18,19 We accounted for the day-of-week effects and overdispersion by combining the day-of-week stratification and total-visit baseline adjustment.^20,21 We modified this stratified, adjusted CuSUM implementation to manage typical issues encountered in syndromic data streams, including temporary data dropouts, overall sparseness, and non-normality.^22,23

Alerting Algorithm Evaluation

Our determining factor in choosing algorithm evaluation measures was the MARSS emphasis on bioterrorism detection. Unlike endemic diseases monitored for progression to epidemic state, bioterrorist attacks may be considered to have zero prevalence in the monitored population, and the MARSS designers chose to tolerate a low positive predictive value, with few alerts corresponding to true target events, as long as the background alert rate is manageable. For the background alert rate, we used the recurrence interval or average number of weeks between alerts, so that a large recurrence interval signifies a manageable alert investigation burden.²⁴

For evaluating sensitivity to target signals, the existing ROK military event-based and sentinel surveillance systems had documented a number of respiratory and gastrointestinal (GI) disease outbreaks, but many of these events were at a single facility representing patient groups too small to be detected at the national level, and subpopulation data were unavailable for this study. No documented outbreaks were available for rarer syndromes. Therefore, we adopted a simulation approach using a lognormal epidemic curve model to generate target signals to evaluate detection performance on the syndromic series.^22,25 The best-fitting algorithm was chosen after separately injecting 1,000 randomly generated realistic signals at multiple outbreak sizes into the national level aggregated dataset to measure the tradeoff between background alerting and event sensitivity. Multiple strategies can be applied to choose operational thresholds using these measures; we sought the weakest signal detectable with 95% sensitivity and a recurrence rate of at least 60 days.

Results

Syndrome group formation was a 2-step process. First, ICD-9 codes for syndromic classification previously developed by the US Centers for Disease Control and Prevention (CDC)²⁶ were mapped to ICD-10 codes through the General Equivalent Mapping (GEM) crosswalks.²⁷ Table 1 lists these syndrome groups along with weaponized diseases whose early effects might be detected for each syndrome. Second, an expert panel that included primary physicians and administrative officers reviewed each combination of ICD-10 codes and modified them based on coding practice. Without substantial experience analyzing coding practice and in view of the emphasis on sensitivity, we continued the prior practice of classifying a medical encounter record in a syndrome if the record contained any of that syndrome's codes in the table.²⁶

Table 1.

Syndrome Groupings and Their Target Agents Used in the Military Active Real-time Syndromic Surveillance System

Syndrome Group	Possible Bioterror Agents	ICD-10 Codes
Botulism-like	Botulism	A05.1, G51.0, G51.2, G51.4, G51.8, G51.9, G52.7, G52.9, G61.0, G70.00, G81.90, G82.20, G82.50, G82.54, G83.0, G83.81, G83.84, G83.89, G83.9, H02.409, H49.00, H49.10, H49.20, H49.40, H53.2, H53.8, H53.9, R47.01, R47.02, R47.81, R47.89
Hemorrhagic	Viral hemorrhagic fever	D47.3, D68.8, D68.9, D69.1, D69.2, D69.59, D69.6, D69.8, D69.9, J90, J94.2, J94.8, K92.0, K92.1, K92.2, R04.0, R04.1, R04.2, R04.9, R23.3, R58, R71.0, R79.1
Dermatologic	Cutaneous anthrax, smallpox	A25.0, A25.1, A25.9, A77.0, A79.1, B00.0, B00.89, B00.9, B01.81, B01.89, B01.9, B02.39, B02.8, B02.9, B03, B05.1, B05.4, B05.89, B05.9, B06.81, B06.89, B08.02, B08.04, B08.1, B08.3, B08.4, B09, L25.9, L26, L30.4, L44.4, L52, L53.0, L53.1, L53.2, L53.8, L53.9, L71.0, L71.1, L71.8, L92.0, L93.0, L93.2, L95.1, L98.2, R21
Gastrointestinal	GI anthrax, tularemia	A04.8, A04.9, A05.4, A05.8, A05.9, A08.39, A08.8, A09, A22.2, K29.00, K29.01, K29.6, K29.60, K29.70, K29.71, K29.80, K29.81, K29.90, K29.91, K50.00, K50.10, K50.80, K52.1, K52.2, K52.89, K52.9, K63.9, R11.0, R11.10, R11.11, R11.12, R11.2, R14.0, R14.1, R14.2, R14.3, R19.7
Respiratory	Tularemia, plague, anthrax	A20.2, A21.2, A22.1, J00, J02.9, J03.90, J04.0, J04.10, J04.11, J04.2, J04.30, J04.31, J05.0, J05.10, J05.11, J06.0, J06.9, J12.89, J12.9, J15.9, J16.8, J17, J18.0, J18.9, J20.9, J21.0, J21.8, J39.8, J39.9, J40, J81.0, J85.0, J85.1, J85.2, J85.3, J86.9, J90, J94.1, J94.2, J94.8, J94.9, J96.20, J98.5, R04.2, R04.9, R05, R06.1, R06.82, R06.9, R07.1, R07.81, R09.1, R09.2
Neurologic	Botulism, encephalitis virus	A86, A87.2, A87.8, A87.9, A88.0, A89, F05, F05, G00.9, G02, G03.0, G03.8, G03.9, G04.2, G04.90, G04.91, G44.1, G44.209, R29.1, R40.0, R40.1, R40.4, R47.01, R51, R56.9
Fever	Plague, anthrax, brucellosis	A20.7, A20.8, A20.9, A21.7, A21.8, A21.9, A22.7, A22.8, A22.9, A41.4, A41.50, A41.59, A41.89, A41.9, B33.8, B34.1, B34.2, B34.4, B34.8, B34.9, B97.19, B97.29, B97.89, R50.9, R56.00, R78.81
Influenzalike illness	Various agents	B33.8, B34.1, B34.2, B34.4, B34.8, B97.19, B97.29, B97.89, J00, J02.9, J04.0, J04.10, J04.2, J06.0, J06.9, J10.1, J11.00, J11.1, J11.2, J11.81, J11.89, J12.0, J12.1, J12.2, J12.89, J12.9, J17, J18.0, J18.9, J20.9, J21.0, J21.8, J39.8, J39.9, J40, R05, R07.0

We analyzed the daily counts of these syndrome groups to measure typical noise levels and trend behavior to guide the selection of alerting methods. Table 2 presents the mean and variance of the series for each syndrome on weekends and weekdays. These statistics represent all military hospital visit records from January 1, 2012, through May 31, 2014. Note the scale differences in daily means among the syndromes, ranging from a sparse 1.9 visits per weekday for records classified with botulism-related symptoms to more than 140 visits per day for respiratory syndrome. Also note the sharp weekend drop-off in mean visits for most of the syndromes.

Table 2.

Daily Syndrome Count Statistics for Surveillance Time Series According to Their Syndromic Grouping from January 1, 2012, to May 31, 2014

	Weekday		Weekend
Syndrome Group	Mean	Variance	Mean	Variance
Botulismlike	1.9	2.3	0.1	0.1
Dermatologic	4.9	11.5	0.4	0.5
Fever	3.1	6.4	1.9	6.9
Gastrointestinal	7.4	14.3	1.7	2.9
Hemorrhagic	9.5	26.7	0.5	0.5
Influenzalike illness	133.9	3,284.2	38.2	488.4
Neurologic	41.4	196.6	3.1	3.6
Respiratory	144.4	3,594.6	40.9	523.2

While these syndromes were recently adapted from an ICD-10 translation of the US CDC groupings conceived for bioterrorist threat types, our modest validation measures have been promising: Rates of visits per thousand population members were available from the AFMC sentinel influenzalike illness (ILI) surveillance system and also from the Korean Centers for Disease Control and Prevention (KCDC) ILI sentinel system representative of the entire Korean population.^28,29 We obtained values of approximately 0.7 for the unlagged Pearson correlation coefficient of MARSS ILI proportions with both AMFC and KCDC rates.

Table 3 gives an operational summary of syndromes and detection performance. The rows of this table summarize the results for the distinct syndrome groups based on 2 sets of runs. The first set used stochastic signals of length 1 to 3 days expressing the MARSS epidemiologists' expectation that for outbreaks resulting from bioterrorist attacks, symptoms causing excess hospital visits would likely be seen without long incubation periods, especially for the rarer syndrome groups. For the respiratory, ILI, and fever syndrome groups, we also conducted runs using 7- to 21-day signals with a model peak between days 10 and 11. For the entries for each row, the second and third columns list syndrome groups and median visit counts on the commonest day of the week for each syndrome. The last 2 columns give the signal strength that was needed to attain 95% sensitivity for 1,000 stochastic, simulated signals, assuming a lognormal epidemic curve. The signal strength is expressed 3 ways: in terms of standard deviations above the weekday mean, in terms of additional cases on the peak day, and in terms of total number of cases.

Table 3.

Required Outbreak Sizes and Thresholds for Desired Detection Performance for Each Syndrome Group

	Syndrome Group		Outbreak Size: 95% Detection Probability
Syndromic Data Scales with Service-wide Events Detectable at Manageable Alert Rates	ICD-10 Syndrome Group	Largest Median Day-of-Week Count	SD Above Weekday Mean at Peak	Required Total Outbreak Count	Peak Day Count
	Gastrointestinal	9	6	78	23
	Botulism	2	6	32	9
	Hemorrhagic	12	4.5	80	23
1-3 Day Signals^a	Dermatologic	6	3.5	41	12
	Neurologic	49	3.5	168	49
	Respiratory	148	2.5	512	150
	Fever	3	4.5	39	11
	ILI^b	134	2.5	489	143
	Respiratory	148	1.5	713	90
7-21 Day Signals	Fever	3	4	81	10
	ILI	34	1.5	681	86

Signal: artificial stochastic injection of simulation

ILI = Influenzalike illness

With results from Table 3 and by using the current capability of SAGES, we have developed the MARSS system with a web-based query function that includes type of syndrome, time window, and selected military hospitals. The website visualization allows automatic detection of statistical aberrations based on Table 3 thresholds. Figure 2 is a MARSS system screenshot showing the daily time series of visit counts for the ILI syndrome from October 2013 through February 2014. Note the characteristic day-of-week pattern resulting from the reduced access to care on weekends. The day-of-week stratification and adjustment for total visits were implemented to account for this systematic time series behavior. Alerts are shown as time series markers shaded yellow or red based on threshold levels chosen to indicate increasing levels of significance. For the algorithm version whose outputs are shown, alerts are intermittent, indicating epidemic influenza levels at the beginning of 2014, and they appear consistent later in January. The increase on January 19, following the previous days' upward trend and sustained the following day, provides a signal that could enable a monitor to begin investigation before the high peaks of the following week.

Figure 2.

Screenshot of the Web-Based Military Active Real-Time Syndromic Surveillance System

Discussion

From the study results, the adapted alerting methods applied to time series covering the entire Korean military population require large outbreaks to achieve the desired detection sensitivity, given the randomly generated signals representing a range of plausible real-life data effects. For the respiratory and ILI syndromes, outbreaks of the sizes shown in Table 3 would certainly be recognized early without algorithms. Sensitivity to smaller-scale events is likely desirable for the other syndromes as well. For example, the Korean military sentinel surveillance system collected information on 45 foodborne outbreaks from army, navy, and air force facilities during the study period. The number of patients in these events was fewer than 10 in 17 events, 10 to 50 patients in 21 events, and more than 50 patients in only 7 events. Half of these events were determined to be norovirus outbreaks, and the other half were attributed to assorted other pathogens or unknown causes. Many of these events were restricted to single hospitals, and only a few had effects that could be seen in the national-level study data streams. Among the MARSS syndromes chosen to cover bioterrorist threats in Table 1, only GI and ILI outbreaks were documented in the sentinel system data.

From consideration of these foodborne outbreaks and the signal sizes listed in Table 3, the requirement of high sensitivity to bioterrorist attacks is likely to cause frequent alerting because of small-scale, naturally occurring events. Although facility-level syndromic data were not available for the current study, similar analysis on smaller-scale datasets can be used to refine MARSS and the underlying data to achieve needed detection performance. After initial implementation, evolution of MARSS will be driven by a combination of military priorities, emerging health threats, and available epidemiologic capacities, including human health monitors and technological resources. Potential development directions include the following:

• Surveillance at lower aggregation levels (ie, monitoring of subnational data at geographic or organization levels allowing high sensitivity to outbreaks of desired size and chosen to be convenient for routine and sustainable operations)

• Syndrome group modifications (ie, groupings refined with experience and with diagnosis codes that capture outbreak-related increases with less unexplained variation)

• Target sensitivities and alert rates tailored to threats of interest and to the capability to investigate and respond to alerts, along with the training and communication required for successful implementation (such specifications may be syndrome-dependent)

• Improved separation of routine and bioterrorist health threats, with separate surveillance systems and enhanced measures to promptly rule out bioterrorism, or other efficiencies

• Incorporation of more clinical data sources, such as laboratory test results and ordered prescription medicine, report-based evidence, and novel evidence sources

Alternative and more sophisticated alerting algorithms could also improve the capability to detect bioterrorism. Based on experience with similar algorithms and datasets, algorithm modifications are likely to provide only minor improvements in detection performance, such as a <10% decrease in alert rates; some of the system-level modifications described in the list above are probably necessary to achieve high sensitivity to small-scale outbreaks. One possible approach is to search for small and large spatiotemporal clusters of disease using scan statistics.³⁰ This approach would require relevant location information for each case, and supplying relevant locations may be a difficult obstacle for a military surveillance system with highly classified, mission-specific information.^31,32 The enhanced value of these methods and of complex regression-like models must be weighed against the effort needed for development and maintenance and the additional baseline data required.

Users of biosurveillance systems typically have many other responsibilities, with little time to inspect data plots and make investigation decisions each day. Therefore, a useful system employs statistical algorithms to signal alerts for observations whose algorithm output crosses a threshold, indicating behavior that is statistically aberrant or too far from expected values to be plausible from random variation alone. Algorithm alerts based on syndromic data streams alone cannot make confirmed outbreak detections because statistical aberrations may have many other causes, such as changes in data provider participation. However, if appropriately chosen, calibrated, and applied to data streams engineered with the operational considerations discussed above, the alerts can focus investigations into possible outbreaks at manageable false-alarm rates.

In recent years, there have been numerous efforts to develop internet-based biosurveillance systems to detect health threats by using the open-source online information internationally.³³ Internet news media, social networking services, and government as well as nongovernment reports have been suggested as resources because they can be used to detect any kind of threat.^34,35 Advantages of the BSP compared with other biosurveillance systems are the capabilities to integrate the military and civilian surveillance efforts of the ROK and the United States; to leverage other existing biosurveillance systems, incorporating them into the BSP; and to expand the user interface and analytic tools through an agile development process involving user feedback and implementation of functional exercises.

The capabilities of the BSP have been constantly growing through ROK and US cooperation since the program's inception in November 2013. The MARSS system is one component of the BSP, and its decision-making process and standard operating procedures are being developed to investigate generated alerts and to coordinate with other BSP components to produce biosurveillance reports as well as to generate an immediate public health response. Concurrent with the development of sustainable alert response strategies, education and training of stakeholders are continuing to fill the gap between system performance and user requirements for acceptance as well as routine utility in the military surveillance environment.

Since 2002, the KCDC has operated the Emergency Department Syndromic Surveillance (EDSS) system and the Anti-Bioterrorism Information Network to monitor any bioterrorism-related activity and abnormal spread of infectious disease.³⁶ However, it is reported to have some discrepancies between the data reported and the actual disease outbreak.³⁷ Situational awareness is a prerequisite for successful decision making, which includes not only the comprehensive perception of the current state but also the projection of a given situation to its future state. The MARSS system is only one component of the BSP, and users of the BSP are encouraged to use other capabilities in the system as well as other biosurveillance systems to generate more comprehensive situational awareness and share a common operating picture in different agencies.

Interdependence among humans, animals, and the environment underpins the concept of biosurveillance.^38,39 However, over time, as diagnostic and detecting technologies are being developed, they have become more complex and broader in scope.⁴⁰ Disease surveillance and preparedness for all kinds of hazards requires collaboration among various countries, agencies, and individuals, including first responders, epidemiologists, and laboratory technicians.⁴¹ For the BSP to achieve an international level of situational awareness for all hazards, it is important to involve different government departments and ministries as well as the private sector and academia through constant networking and interactions. As new threats emerge and cutting-edge technologies are developed, the capability of a biosurveillance system must grow accordingly, by facilitating inclusion of advances in data source sharing and rapid incorporation of clinical laboratory technology into the system.^42,43

Conclusion

In conclusion, we have designed and begun implementation of an automated electronic surveillance system, MARSS, as one syndromic component of the comprehensive BSP for joint ROK-US military all-hazard monitoring capability. The design of the MARSS system focuses on the highly plausible threat of bioterrorism in the Korean peninsula. Syndromic classifications in MARSS extend syndrome groups in previous US CDC classifications by mapping diagnosis codes in each group from the ICD-9 system to ICD-10 codes, with modifications from Korean military physicians experienced in coding practices.

Initial statistical analyses of time series of MARSS-derived ILI proportions gave promising correlation with AFMC and KCDC sentinel rates, with indications of earlier warnings from the MARSS series. In more detailed simulations of signals injected into the time series for each syndrome, available only as aggregates over all Korean military forces, we found that more refined organization of data and monitoring operations are required to attain timely high sensitivity to small-scale outbreaks based on syndromic information alone. Parallel to the development of the MARSS system and the BSP, constant efforts are under way to enhance the detection of chemical and biological weapons. These efforts must be developed further as complex evidence sources accumulate and threat types emerge.

Footnotes

Acknowledgments

The authors thank the Joint Program Executive Office for Chemical and Biological Defense (JPEO-CBD) and the Joint Program Manager Information System (JPM-IS). The support of JPEO-CBD for the Biosurveillance Portal (BSP) under contract N00024-13-D-6400 and the support of the JPM-IS staff and use of facilities in the Republic of Korea have been critical enablers for the success of this multinational engagement. The authors also thank the Armed Forces Health Surveillance Center Global Emerging Infections Surveillance division (AFHSC-GEIS) for developing and making available for general use the Suite for Automated Global Electronic bioSurveillance (SAGES). The SAGES tools have played an important role in the development of MARSS.

References

Gostin

, Phelan

. The global health security agenda in an age of biosecurity. JAMA, 2014; 312(1):27–28.

Kalra

, Kelkar

, Galwankar

, et al. The emergence of Ebola as a global health security threat: from ‘lessons learned’ to coordinated multilateral containment efforts. J Glob Infect Dis, 2014; 6(4):164–177.

Horby

, Pfeiffer

, Oshitani

. Prospects for emerging infections in East and southeast Asia 10 years after severe acute respiratory syndrome. Emerg Infect Dis, 2013; 19:853–860.

Roffey

, Tegnell

, Elgh

. Biological warfare in a historical perspective. Clin Microbiol Infect, 2002; 8(8):450–454.

Kim

, Oh

, Jo

, et al. Introduction of the Republic of Korea-the United States of America's joint exercise against biothreats in 2013: Able Response 13. Osong Public Health Res Perspect, 2013; 4(5):285–290.

Cho

, Chu

. How to manage a public health crisis and bioterrorism in Korea. Osong Public Health Res Perspect, 2013; 4(5):223–224.

Lashley

. Emerging infectious diseases: vulnerabilities, contributing factors and approaches. Expert Rev Anti Infect Ther, 2004; 2(2):299–316.

Drazen

, Kanapathipillai

, Campion

, et al. Ebola and quarantine. N Engl J Med, 2014; 371(21):2029–2030.

Korea Centers for Disease Control and Prevention. Middle East Respiratory Syndrome Coronavirus outbreak in the Republic of Korea, 2015. Osong Public Health Res Perspect, 2015; 6(4):269–278.

10.

Lee

, Oh

, Park

, et al. Public health crisis preparedness and response in Korea. Osong Public Health Res Perspect, 2013; 4(5):278–284.

11.

National Strategy for Biosurveillance. Washington, DC: The White House; July 31, 2012. http://www.whitehouse.gov/sites/default/files/National_Strategy_for_Biosurveillance_July_2012.pdf. Accessed March 11, 2016.

12.

Joint United States Forces Korea (USFK) Portal and Integrated Threat Recognition. Biosurveillance Portal (BSP). Aberdeen, MD: Joint Program Executive Office for Chemical and Biological Defense; March 2014. http://cbrnlibrary.apgea.army.mil/documents/BSP%20White%20Paper_26Mar14_v2.pdf. Accessed March 11, 2016.

13.

Anderson

, Rosenzweig

, Roos

, et al. The Global Biosurveillance Portal: biosurveillance for the Department of Defense. Online J Public Health Inform, 2015; 7(1):e63.

14.

2014 Department of Defense Chemical and Biological Defense Annual Report to Congress. Washington, DC: Department of Defense; March 2014. http://www.acq.osd.mil/cp/cbd_docs/home/Final%202014%20DoD%20CBDP%20ARC_signed%2021%20Mar%202014.pdf. Accessed March 11, 2016.

15.

Lewis

, Feighner

, Loschen

, et al. SAGES: a suite of freely-available software tools for electronic disease surveillance in resource-limited settings. PLoS One, 2011; 6(5):e19750.

16.

Unkel

, Farrington

, Garthwaite

, et al. Statistical methods for the prospective detection of infectious disease outbreaks: a review. J R Stat Soc Ser A Stat Soc, 2012; 175:49–82.

17.

Ryan

. Statistical Methods for Quality Improvement. 2d ed. New York: John Wiley & Sons; 2000:218.

18.

Fricker

Jr , Hegler

, Dunfee

. Comparing syndromic surveillance detection methods: EARS' versus a CUSUM-based methodology. Stat Med, 2008; 27:3407–3429.

19.

Hagen

, Fricker

, Hanni

, et al. Assessing the early aberration reporting system's ability to locally detect the 2009 influenza pandemic. Stat Politics Policy, 2011; 2:1–29.

20.

Lucas

, Crosier

. Fast initial response for CUSUM quality-control schemes: give your CUSUM a head start. Technometrics, 1982:24; 199–205.

21.

Burkom

, Murphy

. Data classification for selection of temporal alerting methods for biosurveillance, intelligence and security informatics. Biosurveillance Lecture Notes in Computer Science, 2007; 4506:59–70.

22.

Tokars

, Burkom

, Xing

, et al. Enhancing time-series detection algorithms for automated biosurveillance. Emerg Infect Dis, 2009; 15:533–539.

23.

Burkom

, Elbert

, Magruder

, et al. Developments in the roles, features, and evaluation of alerting algorithms for disease outbreak monitoring. Johns Hopkins APL Tech Dig, 2008; 27:313–331.

24.

Xing

, Burkom

, Tokars

. Method selection and adaptation for distributed monitoring of infectious diseases for syndromic surveillance. J Biomed Inform, 2011; 44:1093–1101.

25.

Burkom

, Elbert

, Guven

, Coberly

. Analytic biosurveillance methods for resource-limited settings. Johns Hopkins APL Tech Dig, 2014; 32:667–678.

26.

Syndrome definitions for diseases associated with critical bioterrorism-associated agents. CDC website. October 23, 2003. http://www.bt.cdc.gov/surveillance/syndromedef/. Accessed December 17, 2015.

27.

2014 ICD-10-CM and GEMs. Centers for Medicare & Medicaid Services website. Updated November 12, 2013. http://www.cms.gov/Medicare/Coding/ICD10/2014-ICD-10-CM-and-GEMs.html. Accessed December 17, 2015.

28.

Lee

, Shin

, Na

, et al. Influenza surveillance in Korea: establishment and first results of an epidemiological and virological surveillance scheme. Epidemiol Infect, 2007; 135:1117–1123.

29.

Disease Web Statistics System. Osong, Korea: Korea Centers for Disease Control and Prevention. http://is.cdc.go.kr/dstat/index.jsp. Accessed December 17, 2015.

30.

Kulldorff

, Heffernan

, Hartman

, et al. A space-time permutation scan statistic for disease outbreak detection. PLoS Med, 2005; 2:216–224.

31.

Xing

, Burkom

, Moniz

, et al. Evaluation of sliding baseline methods for spatial estimation for cluster detection in the biosurveillance system. Int J Health Geogr, 2009; 8:45.

32.

Pavlin

, Burkom

, Elbert

, et al. Combining surveillance systems: effective merging of U.S. veteran and military health data. PLoS One, 2013; 8(12):e84077.

33.

Gajewski

, Peterson

, Chitale

, et al. A review of evaluations of electronic event-based biosurveillance systems. PLoS One, 2014; 9:e111222.

34.

Generous

, Margevicius

, Taylor-McCabe

, et al. Selecting essential information for biosurveillance—a multi-criteria decision analysis. PLoS One, 2014; 9:e86601.

35.

Barboza

, Vaillant

, Le Strat

, et al. Factors influencing performance of internet-based biosurveillance systems used in epidemic intelligence for early detection of infectious diseases outbreaks. PLoS One, 2014; 9:e90536.

36.

Wang

, Jeong

, Cho

, et al. Analysis of early results from the emergency department syndromic surveillance system for bioterrorism [Korean]. J Korean Soc Emerg Med, 2002; 13:513–522.

37.

Ahn

, Lee

, Kim

, Lim

. Analysis of the Korean emergency department syndromic surveillance system: mass type acute diarrheal syndrome. Healthc Inform Res, 2010; 16:177–184.

38.

Gronvall

, Boddie

, Knutsson

, Colby

. One Health security: an important component of the Global Health Security Agenda. Biosecur Bioterror, 2014; 12:221–224.

39.

Rubin

, Myers

, Stokes

, et al. Review of Institute of Medicine and National Research Council recommendations for One Health initiative. Emerg Infect Dis, 2013; 19:1913–1917.

40.

Margevicius

, Generous

, Taylor-McCabe

, et al. Advancing a framework to enable characterization and evaluation of data streams useful for biosurveillance. PLoS One, 2014; 9:e83730.

41.

Stoto

. Biosurveillance capability requirements for the Global Health Security Agenda: lessons from the 2009 H1N1 pandemic. Biosecur Bioterror, 2014; 12:225–230.

42.

Riccardo

, Shigematsu

, Chow

, et al. Interfacing a biosurveillance portal and an international network of institutional analysts to detect biological threats. Biosecur Bioterror, 2014; 12:325–336.

43.

Chute

. Biosurveillance, classification, and semantic health technologies. J Am Med Inform Assoc, 2008; 15:172–173.