Biodefense Policy Analysis—A Systems-based Approach

Abstract

Understanding the overall biosecurity and biodefense policy landscape, the relationships between policies and their effects on each other, and the mechanisms for leveraging advances in science and technology to enhance defensive capabilities is crucial for ensuring that policy strategies address long-standing gaps and challenges. To date, policy analyses have been conducted primarily on single issues, which limits analyses of broader effects of policies, particularly after implementation. Here we describe the first-ever systems-based analysis of the US biosecurity and biodefense policy landscape to analyze functional relationships between policies, including examination of the unintended positive or negative consequences of policy actions. This analysis revealed a striking bifurcation of the US policy landscape for countering biological threats, with one grouping of policies focused on prevention of theft, diversion, or deliberate malicious use of biological sciences knowledge, skills, materials, and technologies (ie, biosecurity) and a second grouping on development of capabilities and knowledge to assess, detect, monitor, respond to, and attribute biological threats (ie, biodefense). An analysis of indirect effects demonstrated that policies within groups may result in mutual benefit, but policies in different groups may counteract each other, limiting achievement of the policy objectives in either group. The current policy landscape predominantly focuses on pathogens and toxins, having limited focus on rapidly changing biotechnologies with potential to positively contribute to biodefense capabilities or introduce unknown and/or unacceptable security risk. Based on our analyses, we present actions for implementing biosecurity and biodefense policy in the United States that intends to harness the benefits of science and technology while also minimizing potential risks. This article synthesizes and highlights the major findings and conclusions from the detailed analyses, which can be found in the full report (http://www.gryphonscientific.com/biosecurity-policy/).

The authors describe a systems-based analysis of the US biosecurity and biodefense policy landscape to analyze functional relationships between policies, which revealed 2 approaches in US policy for countering biological threats: (1) prevention of theft, diversion, or deliberate malicious use of biological sciences knowledge, skills, materials, and technologies, and (2) development of capabilities and knowledge to assess, detect, monitor, respond to, and attribute biological threats. Current policy focuses on pathogens and toxins, having limited focus on rapidly changing biotechnologies with potential to positively contribute to biodefense capabilities or introduce unknown and/or unacceptable security risk.

Historically, biosecurity and biodefense policy initiatives have been reactive and inconsistent in implementation. Biosecurity is defined as the prevention of theft, diversion, or deliberate malicious use of biological sciences knowledge, skills, materials, and technologies, which is consistent with the Biological and Toxins Weapons Convention and World Health Organization's definitions.^1,2 Biodefense is defined as the development of capabilities and knowledge to assess, detect, monitor, respond to, and attribute biological threats. This definition is consistent with the 2018 National Biodefense Strategy, which also describes biosecurity as a subset of biodefense.³

In the aftermath of a major biological incident, the US government typically has responded in an iterative and reactive manner to counter that particular event (Figure 1). For example, the illegal acquisition of plague bacteria by a member of a white supremacist group resulted in the establishment of the Biological Select Agents and Toxins (BSAT) program in 1996. This program changed significantly in 2005 in response to calls for strengthened security around BSAT after 9/11 and the subsequent anthrax letters, and again in 2012 after Dr. Bruce Ivins was identified as the alleged perpetrator of the anthrax letters. Today, this program controls access to and regulates certain research involving dangerous pathogens.⁴ The BSAT program is only one of several policies that have evolved in a reactive manner during the past 30 or more years. Examples of other policies that have evolved include medical countermeasure development, biosurveillance, cooperative biological threat reduction, and dual-use research of concern. This iterative and reactive policymaking process can create inconsistencies in implementation of policies, limit the degree to which existing policies can address emerging risks and threats, and result in broader effects to US national strategic interests.

Figure 1.

Schematic illustrating the reactive nature of US biosecurity and biosafety policies and biodefense investments during the past 50 years. This illustration does not present a comprehensive list of events, regulations, and investments that have occurred during the past 50 years. However, it does include major historical events in biosecurity and biodefense during this period.

In 2018, the US government released the National Biodefense Strategy and Implementation Plan,⁵ the National Security Strategy,⁶ the National Defense Strategy,⁷ and the National Counterterrorism Strategy,⁸ and the National Counter WMD Strategy,^8A all of which include biosecurity and/or biodefense. In addition, the US government is preparing a Global Health Security Strategy⁹ and assisting in the development of a 2024 framework of the Global Health Security Agenda,¹⁰ which involves prevention, detection, and response to natural, accidental, and intentional biological threats. As the US government engages in these current and future policy activities, understanding the existing biosecurity and biodefense policy landscape, the functional relationships between policies (ie, mutual enhancement, no effect, or counteraction), and the mechanisms for leveraging advances in science and technology to enhance defensive capabilities is crucial for ensuring that the new strategies address long-standing gaps and minimize inconsistencies.

Central to all of biosecurity and biodefense policies is the multi-use nature of biotechnology, which is changing rapidly and affects policies in various areas. New funders and practitioners of biology enable convergence of scientific disciplines and adoption of different scientific approaches (eg, using the engineering discipline's design-build-test paradigm to create new biological organisms), which can lead to new fields such as synthetic biology, DNA storage, 3D bioprinting, precision medicine, and precision agriculture. However, many existing policies, including the 2018 National Biodefense Strategy, focus only on technologies affecting pathogens and toxins and do not address advances in biotechnology that fall outside that scope. Although the study and manipulation of pathogens using new technologies may conform to existing biosecurity and biodefense policy frameworks, an overwhelming majority of new tools are used in research that does not involve pathogens and toxins. In addition, new advances in biology and biotechnology may not be covered by policies that apply only to federally funded research. Lastly, the broader effects of enabling biotechnologies may not be directed toward human health or agriculture, which are the primary concern of many biodefense and biosecurity policies, but instead the environment, national security and defense, preparedness and response,^11,12 and commercial services and products, affecting economics, international competitiveness, or diplomacy.

Many efforts have been undertaken to evaluate specific groups of policies—for example, medical countermeasures,^13,14 biosurveillance,^15-17 or dual-use life sciences research.^18-20 In early 2018, a research group at the Pacific Northwest National Laboratories (PNNL) conducted a limited study mapping US government agencies that are responsible for implementing specific activities authorized by 8 major US biosecurity and biodefense policies.²¹ An update of PNNL's tool focuses on agency and subagency responsibilities for these policies. Although these analyses are valuable for gaining insight into the specific strengths and limitations of critical biodefense policy issues, they fall short in identifying policy opportunities and gaps across the entire biosecurity and biodefense system.

Therefore, we undertook an ambitious study to analyze the functional relationships of the current US biodefense and biosecurity policy landscape using a systems-based approach and focusing on potential relationships among policy activities, gaps, and broader effects. Prior to the initiation of our study, no comprehensive relational analysis of the entire biosecurity and biodefense policy landscape had been conducted. Our analysis sought to (1) make fundamental observations about the biodefense policy landscape as it existed in 2017, including the identification of policy and implementation gaps; (2) evaluate performance metrics for assessing implementation outcomes; and (3) analyze opportunity costs associated with the implementation of policies.

This systems-based analysis enabled a deeper level of evaluation of the functional interrelationships between various policies, existing gaps, anticipated implementation challenges, and potential unintended effects (positive or negative) toward achievement of the stated biodefense objectives. The findings of our analyses complement other analyses that were conducted in parallel by PNNL and the US government by highlighting critical issues, such as opportunity costs, policy needs for enhancing and translating biodefense research, and scientific gaps that affect operational realities, all of which will affect implementation of the 2018 National Biodefense Strategy.

Policy Analysis Methodology

Scope of Policy Analysis

Understanding the scope of biosecurity and biodefense included in this article is necessary to understand the purpose of the analysis, results, and conclusions described herein. Policies seeking to prevent theft, diversion, or deliberate malicious use of biological sciences knowledge, skills, materials, and technologies (ie, biosecurity); prevent and mitigate accidental exposure to pathogens studied in high and maximum containment (ie, laboratory biosafety and occupational health and safety policies); and build capabilities and the knowledge base for assessing, detecting, monitoring, responding to, and attributing biological threats (ie, biodefense) were included in the analysis. In addition, health security policies were included, which involve the prevention, detection, and response to natural, accidental, or deliberate biological threats such as Public Health Emergencies of International Concern (PHEIC) as described by the International Health Regulations²² and the Biological Incident Annex to the National Response Framework.²³ However, policies on chronic diseases or infectious diseases such as malaria and human immunodeficiency virus (HIV) were not included in the policy analysis.

In addition to these primary topics, policies that govern biotechnology products and hazardous chemicals derived from synthetic organisms were included, primarily because of security concerns elicited about capabilities in synthetic biology and genome editing.^24-28 Inclusion of these policies enabled a comprehensive, functional evaluation of policies that directly or indirectly affect primary US biodefense objectives.

Our analysis included all US statutes, regulations, international agreements, and executive- and agency-level strategies, and other policies that directly or indirectly affect US biodefense activities and objectives. Statutes were identified through the US House of Representatives online, searchable US Code database (http://uscode.house.gov/). Legislation and regulations were identified through the Government Printing Office, DHHS Public Health Emergency (Phe.gov), our experts working group, US National Research Council reports, the Encyclopedia for Bioterrorism Defense,²⁹ the Risk and Benefit Assessment for Gain-of-Function Pathogens,³⁰ and published scholarly articles. Executive-level strategies were obtained from many of the same sources as the legislation and regulations and from archived White House websites. Agency-level strategies and activities were obtained from individual agency documents and websites. International agreements were obtained through the US Department of State website, the United Nations Office of Disarmament website, the Encyclopedia for Biodefense, and Phe.gov. All other policies were identified from discussions with stakeholders and expert working group members specifically engaged during this project and the in-house expertise of the project team. Because US policies are iterative and often build on past legislation, regulations, and strategies, a full understanding of the present-day policy landscape required inclusion of policies dating back to 1913. To eliminate redundancies between legislation and regulations on the same policy measures and to ensure that the analysis focused on the current policy landscape (as of 2017), US codes were included in the policy analysis along with executive- and agency-level policies, international instruments, and guidances and guidelines. A source list of policies included in our analyses is provided in the supplemental materials (see Supplemental Materials at https://www.liebertpub.com/doi/suppl/10.1089/hs.2018.0082).

The analysis described in this article considered, but does not focus on, implementing agencies and subagencies, in large part because more than 17 agencies and several offices within each agency have documented and assumed responsibilities in biosecurity and/or biodefense. Furthermore, these responsibilities often change because of reorganization of government offices, which alters the results of such analyses. Therefore, the scope of the background, analyses, and conclusions described here focuses on functional aspects of biosecurity and biodefense, particularly with a focus on science and technology.

Creation of the Relational Dataset

A relational dataset of policies was created by associating related policy instruments and leaving unassociated policies separate. For example, Executive Order 13546, which was on enhancing the Biological Select Agents and Toxins (BSAT) regulations, was associated with the US code for animal, human, and plant BSAT. But the Coordinated Framework for Biotechnology Products has not been functionally linked to BSAT policies, so these policies were not associated in our dataset. To ensure that our results reflect the current policy landscape, older policies that were updated or replaced were linked in our relational dataset. Individual policies were tagged based on subject area(s) (eg, nonproliferation, BSAT, medical countermeasures), biodefense objective(s) directly and indirectly addressed, and type of policy instrument. Indirect effects were derived from discussions with stakeholders, working group experts, scholarly literature, and US government reports. The list of all tags, objectives, and policy types are found in Table 1 of the Supplemental material. To analyze the current policy landscape, including the direct and indirect relationships between existing biosecurity and biodefense policies, we used mapping and data visualization software: Gephi, an open source data visualization platform that uses Bayesian statistics to create network maps (https://gephi.org/users/publications/), and Tableau, a data analytics and visualization software (https://www.tableau.com/).

Table 1.

Limitations of the Biosecurity and Biodefense Policy Landscape in the United States

Limitations of the Scope and Relevance of Policies	Limitations to Consistency of Policy Development and Implementation Across the US Government	Limitations to Stakeholder Engagement in Policy Implementation
Expansive policies may lack clarity about what is or is not covered under the policy, which promotes variability in policy implementation at the federal and local levels and risks affecting sectors and activities in unanticipated ways.	The current policy system is not suitable to evaluate the broader consequences of investments or regulations.	Stakeholders do not necessarily understand their roles in achieving biosecurity and biodefense objectives.
Narrow policies, especially those based on defined lists of restricted items, often prevent thorough analysis of research to anticipate and address risks early and to maximize benefits.	Federal and local stakeholders of overlapping policies may not be the same.	Limited or no additional funds are available to assist key stakeholder groups in complying with biosecurity regulations.
Policies that are required only at institutions that receive US government funding do not necessarily cover scientific activities that are not federally funded, regardless of whether they are conducted in the United States or another country, adversely affecting awareness about technological advances and of research oversight.	No consistent or common process exists for reviewing and overseeing research with potential for exploitation by malicious actors. Oversight of research is agency-specific.	Some tools for prioritizing biological threats result in the identification of the same agents regardless of country or situation.

Stakeholder Engagement

We engaged more than 100 stakeholders from the academic, industry, public health, veterinary, health security, intelligence, policy, biosafety, and synthetic biology communities to identify additional policy gaps, implementation challenges, and “opportunity costs” experienced by different stakeholders while implementing or complying with biosecurity policies, which provided insights about broader effects of these policies. Opportunity costs are defined as trade-offs made to support one policy over another policy or activity.

Stakeholders were engaged on a one-on-one basis or in a group setting. Group discussions were convened in cooperation with professional organizations, such as the American Biological Safety Association, the Association of Public Health Laboratories, the American Association for the Advancement of Science, the Engineering Biology Research Consortium, the American Society for Microbiology, and the Association of State and Territorial Health Officials. One-on-one discussions with stakeholders took place at major conferences (eg, the BIO Convention and OIE Biothreats Conference) or via teleconference. Stakeholders were identified through their professional or industry organizations, the expert working group members, policymakers, and recommendations from other stakeholders. These discussions focused on the identification of: (1) policy and implementation gaps broadly (across the full policy landscape), and (2) direct and indirect costs associated with the implementation of the 2012 and 2017 revisions of the BSAT regulations, and 2012 and 2014 US government policies on dual-use life sciences research of concern. We selected the BSAT and dual-use policy instruments because these policies focus on different but overlapping biosecurity activities, and opportunity costs related to their implementation have been discussed in the public domain.

Internal Peer Review

We convened a multidisciplinary working group of experts, who provided ongoing internal peer review of all project analyses, the relational dataset, and final conclusions. The working group members are listed on the project website (http://www.gryphonscientific.com/wp-content/uploads/2018/07/Full-Report_Roadmap-of-US-Biosecurity-and-Biodefense-Policy_2018.pdf).

Policy Analysis

Based on the policy mapping and stakeholder discussions, we identified (1) policy gaps and limitations, highlighting those that affect science and technology; (2) indirect effects of existing policies on US biodefense objectives; and (3) existing approaches for evaluating policy implementation. From these findings, we developed a roadmap for the US government to leverage science and technology while also minimizing biosecurity and biosafety risks.

The core components of the roadmap focus on addressing fundamental activities that currently limit participation in and translation of research and technology development for countering biological threats. This article includes the core components of the roadmap and a synopsis of the policy analysis. As part of the overall project, we developed frameworks for analyzing opportunity costs of biosecurity policies based on the findings from the BSAT and dual-use analyses, and for evaluating successful policy implementation, which involves both activity-based assessments and outcome-based evaluations. These frameworks, the detailed analyses, and the full roadmap are available on the project website (http://www.gryphonscientific.com/biosecurity-policy/).

Biosecurity and Biodefense Landscape

Identification of Biodefense Objectives

To understand the intended and unintended consequences of US biodefense and biosecurity policies, we conducted a relational mapping analysis of all policies that directly or indirectly affect primary US biodefense objectives. First, we sought to define a comprehensive set of biodefense objectives with which to analyze US policies. To identify these objectives, we initially reviewed the suitability of 2004 Homeland Security Presidential Directive (HSPD) 10: Biodefense for the 21st Century,³¹ and the 2009 Presidential Policy Directive (PPD) 2: National Strategy for Countering Biological Threats,³² both of which provided high-level, broad strategic guidance on preventing biological threats. However, neither fully encompassed all the objectives that have been ascribed to biosecurity and biodefense across several national-level policy documents based on funded initiatives and aspirational and operational policy goals.

This finding led us to identify objectives and their scopes that reflect the entire biodefense landscape from situational awareness to response and recovery (Figure 2). These objectives were: (1) situational awareness, which includes threat assessment, risk assessment, and intelligence; (2) prevention, which includes export control, physical security, personnel security, cybersecurity, nonproliferation, and threat reduction; (3) preparedness, which includes preparedness planning, community engagement, research and development of medical countermeasures, and detection and biosurveillance* activities; (4) response; and (5) recovery. Figure 2 highlights these objectives and indicates which objectives can be supported by science and technology.

Figure 2.

A schematic and list of US biodefense objectives to which US biosecurity and biodefense policies were mapped. These objectives were derived from combining the core components of HSPD-10, PPD- 2, and other relevant policies. The science and technology categories were derived from policy documents and funding announcements for research and development for biodefense. Similarly, the policy scope was derived from policy documents and discourse over the past 18 years.

Our analyses focused on policies and programs at the interface between science and technology and biodefense, including instruments that promote enhancement or building of new defensive capabilities and instruments that prevent or deter potential exploitation by adversaries to enhance their capabilities. Science and technology capabilities associated with biodefense include: (1) natural, engineering, and social science research; (2) medical countermeasure research and development; (3) biosurveillance and detection; and (4) forensics. These capabilities were selected because they receive significant policy attention and funding, or they have been identified as clear gaps (eg, forensics^33,34). After completion of the analysis, the 2018 National Biodefense Strategy (NBS) was released, replacing HSPD-10 and PPD-2.³⁵ The biodefense objectives we used in the policy analysis align well with the NBS, demonstrating that the findings of our analyses remain relevant to implementation of the NBS (Figure 3).

Figure 3.

A mapping of biodefense objectives from HSPD-10, PPD-2, the 2018 National Biodefense Strategy (NBS), and our policy analysis. The left column lists the objectives associated with the 4 pillars of HSPD-10 (blue dots and lines) and PPD-2 (red dots and lines), both of which have been replaced by the NBS. The middle column lists the 5 biodefense objectives used in our policy analysis; the scope of each objective is provided in Figure 2. The right column lists the sub-objectives of the NBS (black dots and lines). The lines indicate similarity of objectives listed in the left or right columns with those included in this study.

Systems-based Policy Analysis

Relational mapping of the policies by subject area demonstrated clearly that US biosecurity and biodefense policies cluster into 2 primary groups: actions intended to prevent biothreats, and actions intended to prepare for and respond to biothreats (Figure 4). Only 2 policies are shared between both groups: HSPD-10 and the Public Health Security and Bioterrorism Preparedness and Response Act of 2002.^31,36 The map shows only 1 line connecting the 2 spheres, which corresponds to HSPD-10, because the US codes associated with the statutes set forth in the legislation, not the legislation itself, were included in the visualization. If the analysis were conducted with the NBS objectives in place of HSPD-10 and PPD-2 objectives, only 1 line corresponding to the NBS would connect both spheres, resulting in the same outcomes.

Figure 4.

Relational map of US biosecurity and biodefense policy by policy subject. Each white circle is a unique US Code, international agreement or partnership, executive- or agency-level policy, program activity (if not already associated with a US Code, international partnership, or agency-level policy), guidance, and guidelines. The colored circles are nodes signifying subject area. The size of the nodes reflects the number of policies associated with each subject area, and the distance between nodes reflects the degree to which policies are linked based on the underlying relational dataset. The lines reflect direct relationships between policies and subject areas based only on existing policies. This map does not reflect associations of subject area based on conceptual similarities, but rather associations by direct links between existing policies.

This clustering raised several questions about the mutually beneficial or counteracting effects that policies may have on each other. Indirect or unintended effects of policies were determined using 2 interconnected analyses: (1) through engagement with stakeholders who are responsible for complying with a policy or otherwise affected by a policy, and (2) through association of policies with the activities that they indirectly affect (eg, medical countermeasure policies may enhance response, whereas BSAT regulations may counteract research to create new medical countermeasures). To analyze potential indirect effects, policies were color-coded according to their type (ie, regulation, restriction, requirement, punishment, or capability investment), with restrictive policies coded in shades of red, capability-building policies coded in green, and punishments coded in blue (Figure 5). Also included in the figure is a depiction of policies according to the primary biodefense objective that they are intended to address (the white circles). Inclusion of direct and indirect effects enables objective evaluation of potential positive or negative effects of policies.

Figure 5.

The potential indirect effects of US biosecurity and biodefense policies on US biodefense objectives. Each circle is a unique US Code, international agreement or partnership, executive- or agency-level policy, program activity (if not already associated with a US Code, international partnership, or agency-level policy), guidance, and guidelines. The white circles represent the biodefense objectives (columns) that policies in each of the subject area categories (rows) address directly. The colored circles indicate indirect effects of the policies in the subject area categories on biodefense objectives. The green circles indicate capability-building activities. The pink circles indicate requirements, the red circles indicate regulations, and the burgundy circles indicate restrictions, all of which seek to promote biosecurity activities. The blue circles are policies that criminalize development and/or use of biological weapons or their delivery systems for malicious use. The graph is intended to be viewed by row.

The findings from Figures 4 and 5 suggest that activities within the same domain may benefit each other in intended or unanticipated ways. For example, compliance with the BSAT regulations may have corollary benefits of enhancing laboratory biosafety, in addition to its intended objective of addressing biosecurity. Similarly, efforts toward medical preparedness (eg, research and development of medical countermeasures and hospital preparedness efforts) and pathogen detection enhance response time and effectiveness.

However, policies in different groups may counteract each other. For example, export control regulations may limit, delay, or prevent sharing of information, viral and bacterial samples, and genetic sequences of restricted pathogens with researchers and health professionals. Limited or lack of access to samples, pathogens, and/or genetic sequences may reduce the amount of knowledge gained about the causes of infectious disease outbreaks, development of new medical countermeasures, and detection and monitoring of biological events.

Based on these analyses, we suggest that the US system of biodefense would be served best by an overarching strategy that encompasses all biodefense and biosecurity objectives, providing opportunities to identify and resolve potentially counteracting policies and maintain potentially reinforcing policies. Although the 2018 NBS encompassed a more comprehensive set of critical biodefense objectives than preexisting policies, it still is missing some issues, including support for research and development of forensics and analysis of national security implications of biotechnology advances, especially those that play an insignificant role in modifying or creating pathogens and toxins. In addition, responsibility for implementation of the NBS was assigned to the Assistant Secretary for Preparedness and Response in the Department of Health and Human Services, which raises questions about the roles and responsibilities of other US government stakeholders.

Analysis of Opportunity Costs

To understand the factors leading to counteracting effects of policies, we developed 2 case studies based on stakeholder experiences on compliance with and/or implementation of the 2012 and 2017 revisions to the BSAT regulations and/or the 2012 and 2014 policies on the review and oversight of dual-use life sciences research of concern.^† Through discussions with stakeholders affected by these policies, we elucidated the types of direct costs in implementing or complying with the policies, the indirect effects resulting from the direct costs (ie, trade-offs made by stakeholders), and downstream consequences to US biodefense objectives. Although not intended to generate a comprehensive account of the opportunity costs associated with these policies, the case study findings provided insight into the challenges and stakeholder decisions associated with implementation.

The case study analyses revealed several direct costs incurred by stakeholders involved in implementing the biosecurity policies, including (1) the financial cost of initial and ongoing compliance with new or changing biosecurity policies; (2) time involved in initial and ongoing implementation, including time spent on compliance and delays associated with longer research reviews; and (3) the frustration felt by scientists conducting regulated activities, including concerns about the intrusiveness of personnel security policies, perceived redundancy of dual-use research policies, and stigmatization of regulated research by some members of the public, the media, and biosecurity and policy experts.

Direct costs of implementing policies at the institutional level involved several factors, all of which are critical for evaluating indirect effects. These factors include costs of maintaining biosecurity infrastructure (eg, physical, cyber, and other security measures); the upfront purchase and installation costs of equipment; the level of personnel effort needed for initial and ongoing compliance with policies, including documentation of existing practices or exemptions; and the need for new experiments to satisfy new or updated policies, even if the policies are guidance and not regulation. Furthermore, the direct financial and time costs of complying with a new biosecurity policy were influenced by whether, and to what extent, the policy likely requires changes to the infrastructure or operation of affected institutions. Determining these changes required consideration of 2 additional factors, both of which vary by sector: (1) overlapping requirements established by other policies; and (2) existing laboratory and institutional architectures, workflows, and procedures.

As regulated individuals divert increasing amounts of time and money from research, training, or other biodefense activities to compliance, those activities may be delayed or discontinued at the individual or institutional level. These indirect effects can be grouped into 3 categories: (1) costs to research and other biodefense activities; (2) costs to workforce, including loss of workforce development opportunities and individual capabilities; and (3) loss of institutional capabilities. Costs in each category can have effects on other categories, exacerbating the indirect effects. For example, training opportunities are lost when institutions cease conducting regulated activities, causing secondary effects on the development of a knowledgeable scientific workforce. Collectively, the indirect effects of policy implementation may undermine US capabilities to defend against biological threats. These downstream consequences may limit the United States' ability to meet its biodefense objectives by reducing the number of trained personnel available for critical prevention and response activities and delaying or preventing critical research activities for detection of new zoonotic diseases, characterization of emerging pathogens, development of new medical countermeasures, and microbial forensics. In addition, reduced global competitiveness in biodefense fields arising from the loss of individual and institutional capabilities and the export of biodefense capabilities and knowledge overseas could lead to a reduction in US capabilities to counter biological threats and possibly contribute to a shift in global leadership between the United States and adversary nations.

Opportunity costs can be mitigated by strategies targeting the direct costs or indirect effects of a policy, but only if these costs are evaluated at the individual or institutional level rather than at the population (ie, regulated community) level. One example of a strategy that stakeholders suggested could mitigate direct costs was federal funding for institutions to implement or comply with biosecurity policies. Stakeholders in regulated communities also described strategies for mitigating indirect effects at the individual or institutional level, including division of administrative responsibilities among multiple senior researchers, reducing the burden on a single individual, centralization of compliance activities at an institution, and better communication among scientists, funders, institutional administrators, policymakers, and members of the public about the objective benefits and risks of research.

Stakeholders also highlighted some unanticipated benefits of biosecurity policies, including improved engagement between regulated entities and federal law enforcement to enhance understanding of security concerns and effective biosecurity measures. This benefit was reinforced by discussions with federal stakeholders who described improved biosafety at BSAT-regulated laboratories. These benefits may reinforce the direct, intended objectives of biosecurity policies.

The case study findings suggested that indirect consequences of biosecurity policies on biodefense objectives are likely unavoidable given the restrictive nature of the biosecurity policies. Therefore, evaluating whether the policies are successful in achieving their intended outcomes provides a more thorough understanding of actual benefits afforded by biosecurity policies.

Evaluation of Policy Implementation

Most methods used to evaluate successful implementation of policies measure the achievement of project activities (eg, assessments of project performance) rather than the effectiveness of activities (eg, assessments of policy outcomes), in part because performance can be measured quantitatively. A significant challenge to measuring effectiveness of policy implementation, and therefore the achievement of the intended goals of the policy, is the inherent inability to measure intangible or unquantifiable outcomes, such as building of trusted partnerships, promoting the development of knowledgeable leaders, and preventing the occurrence of deliberate incidents.

Data for measuring these outcomes may be accessible to government staff only or may not exist at all, preventing a systematic evaluation. However, the lack of public access to such data should not prevent government stakeholders, who may have access to non-public data, from assessing whether a policy has achieved its intended biodefense objective(s). Failure to undertake such analyses could result in inaccurate assessments of outcomes and needs of policy and program implementation. Based on an analysis of published methodologies, including documents evaluating policy and program implementation such as assessing cooperative biological engagement programs,³⁷ and environmental health and safety programs such as assessing facility chemical safety and security,^38,39 we concluded that the most informative approach for evaluating implementation of biosecurity and biodefense policies involves both activity- and outcome-based assessments. Within this context, activity-based metrics are quantitative or semi-quantitative measures of successful achievement of discrete activities and milestones. Outcome-based assessments involve a qualitative or semi-quantitative evaluation of the successful achievement of desired end-states, or goals, of the project. By undertaking such analyses, the benefits, broader effects, and shortcomings of individual policies can be identified.

Limitations, Gaps, and Overarching Themes

Analysis of the policy landscape and indirect effects highlighted several limitations and gaps associated with the US biosecurity and biodefense landscape. Table 1 summarizes these limitations, which fall into 3 categories: (1) scope and relevance of policies, (2) consistency of agency-level policies promulgated to achieve government-wide objectives, and (3) stakeholder contributions to implementing policies. Table 2 outlines gaps in biodefense capabilities, policy implementation, and infrastructure that were identified throughout the course of our study. Several key themes emerged from these limitations and gaps, all of which inform the development of a practical path forward for implementing biosecurity and biodefense policy in a manner that leverages advances in science and technology while also minimizing risk. The key themes include:

Table 2.

Gaps in the US Biosecurity and Biodefense Policy

Capability Gaps	Policy Implementation Gaps	Infrastructure Gaps
Microbial forensics is an underinvested field in the United States and internationally. Systems for scanning scientific advances leading to new technologies exist in offices that support or conduct research and development, but generally do not exist at the end-user or operational levels. Despite significant investment in automated biosurveillance systems, the underlying data used to develop effective early warning methods are highly variable and uncertain. The increasing convergence of scientific disciplines, the changing funding paradigm, and expansion of biotechnology practitioners suggests that greater attention is needed on evaluating the broader security implications of advances and applications that are focused not only on pathogens and toxins.	Insufficient funds are available to support local implementers in complying with biosecurity regulations, leading many to choose not to participate in research or diagnostic activities involving restricted agents. The continuous changes to the BSAT regulations resulted in significant challenges and delays in federal implementation and local compliance. Practical resources that enable program managers, research reviewers, and scientists to assess the risks and benefits of research currently are lacking. Annual and inconsistent investment in nonproliferation activities, specifically for cooperative threat reduction programs, limits long-term sustainability of partnerships and outcomes. Effective measures for evaluating biosecurity policy implementation have not been developed. However, measures for evaluating a few biodefense investments do exist, each different from the others. No analytic framework currently exists for assessing opportunity costs of biosecurity policy development.	The regional and national biocontainment laboratories are not considered critical infrastructure, preventing efforts for their protection in case of an emergency. Very few policies and programs exist for enabling or promoting resilience in the biodefense, health, and research sectors. Very little, if any, funding has been provided for research to generate data on the effectiveness of different biosafety and biosecurity measures.

Since 2002, the US government has supported significant research on high-risk, restricted pathogens to increase scientific knowledge, develop medical countermeasures (vaccines, drugs, and diagnostic tools), and develop methods and technologies for pathogen detection. Scientists, technologists, and engineers involved in these studies must comply with US biosecurity regulations and requirements, including the BSAT regulations, dual-use life sciences research of concern policies, and export control requirements. Compliance can require significant investments of money and time, which can limit, delay, or prevent research and development efforts. This situation may create a misalignment between scientific investment and regulation, which can present significant barriers to the United States' reaping the benefits of science and technology advances to address US biodefense needs.

New funders (eg, the public through crowdsourcing platforms and technology venture capital firms), practitioners (eg, amateur biologists, computer scientists, and engineers), countries investing in knowledge-based economies, and societal drivers (eg, precision health care and improved platforms for manufacturing of chemicals) play significant roles in driving the design, development, advancement, and application of biotechnologies. However, these new influences in the biotechnology landscape often are not considered when assessing the implications, trends, and risks or benefits of individual technologies. Domestic and international engagement with nontraditional funders, practitioners, international counterparts of scientific and technological fields and sectors, and end-users (including the public, if appropriate) is needed to promote global support for risk identification and reduction, and governance of the biological sciences.

Advances in biology and biotechnology have the potential to enhance US capabilities for preventing, detecting, and responding to biological threats. These advances are being applied to specific problem-sets, such as the development of bio-based sensors using synthetic biology and early warning systems using advanced biological data analytics (eg, BioSense⁴⁰) and Biosurveillance Ecosystem,⁴¹ the benefits of which remain to be determined. However, the mechanisms used by the US government to scan for promising advances, enable further innovation to address specific defense needs, and transition promising technologies to operational use are few and tend to be field- or discipline-specific. Improving this process would enhance opportunities for promoting creativity and communication among the biodefense policy community, scientists, and technologists, enabling greater harnessing of science and technology advances and their downstream applications.

Promoting an informed and balanced assessment of risk and benefit of research activities is critical at the federal, local, and international levels for ensuring that unsubstantiated fears about risk or blind hope about benefits do not adversely influence the outcomes of these assessments. Furthermore, practical resources are needed to help policymakers, program managers, security experts, research reviewers, and scientists learn from previous assessments and conduct informed and substantiated assessments. This balancing act is a crucial step in enabling creativity and innovation in the scientific and technological communities to design, build or develop, and apply new advances to enhance biodefense, health, agriculture, and other sector-specific capabilities.

Based on this analysis, we conclude that the entire US biodefense and biosecurity landscape should be treated as a dynamic and interconnected ecosystem that leverages the knowledge and capabilities of the science and technology community, while also promulgating practices for preventing malicious development and use of biology and biotechnology.

A Path Forward

The current biosecurity and biodefense policy structure in the United States could benefit from fundamental actions that simultaneously would enhance efforts to leverage biological and biotechnological capabilities and promulgate effective measures to reduce biosafety and biosecurity risks. These actions would enable capability-building, as described in the 2018 NBS and anticipated in the Global Health Security Strategy.

Figure 6 presents 6 primary actions that policy leaders can undertake to address outstanding gaps or limitations as they begin to implement the NBS. Some of these actions reinforce similar recommendations made by august bodies such as the Blue Ribbon Panel on Biodefense, a staff member of which served on our study's expert working group. The 6 actions are:

Figure 6.

The primary actions comprising the roadmap for maximally leveraging science and technology advances for biodefense and minimizing biosafety and biosecurity risks. The placement of the 6 actions correlates with the most relevant biodefense objectives. The science and technology capabilities are listed in the gray circle and placed close to the objective with which they correlate. The capabilities written in blue are discussed further in the policy analysis. The capabilities written in black are included because each is associated with 1 policy document. Other capabilities may exist, even though they are not included in this figure. Various US government agencies have varying degrees of responsibility for each of the actions listed.

Enhance assessments of emerging biotechnologies , which involves the creation of opportunities for security experts and government personnel, more broadly, to learn about new advances, applications, and technical limitations of biotechnology.

Assist the scientific enterprise for research, detection, health security, and forensics , which involves efforts that enable federal and local stakeholders to comply with biosecurity regulations by providing assistance, conducting no-fault risk assessments, advising on securing BSAT agents, and providing dedicated funding to support initial and ongoing compliance with biosecurity policies.

Balance benefit with concern about malicious exploitation of biology and biotechnology , which focuses on the creation and use of resources that help federal and local stakeholders conduct informed, objective risk and benefit assessments, articulate the outcomes of concern to promote proactive identification of technology developments that reduce the time to realization, and share lessons learned from the assessments among stakeholders.

Enable innovative research and development to meet biodefense needs , which enhances US ability to identify unmet or unaddressed capabilities for which science and technology solutions exist, supporting efforts to enhance research capability and workforce development; encourage more scientists and engineers to participate in policy activities; and conduct a cross-cutting analysis of the biodefense budget, as highlighted by the Blue Ribbon Study Panel on Biodefense in 2018,⁴² to understand opportunities for leveraging biotechnology and biological research, behavioral sciences, and social science.

Promote sustainability of activities , which involves efforts to ensure that activities are sustained and desired end-states are promulgated during and after US government funding has ended.

Characterize the biodefense research sector as a critical infrastructure to ensure assistance and guidance on recovery and resilience , which enables research institutions to provide technical support during emergency response to incidents involving biological agents and recovery efforts after an incident ends.

These actions translate to implementation of the 2018 NBS objectives, even though they are based on a policy analysis of prior directives now replaced by the new strategy, as shown in Figure 7. This high-level analysis indicates that several of the suggested actions will be critical during the policy implementation phases of the NBS, which addressed several of the gaps we identified by providing support for engaging stakeholders during policy implementation and conducting research to build biodefense capabilities. However, some aspects of these actions do not match the NBS objectives as easily because the Strategy (1) continues to rely on list-based biosecurity policies that focus on a defined of restricted pathogens and toxins; (2) includes both research-oriented and operational objectives, whereas we focused on actions that focus on science and technology capabilities; and (3) does not include research and development of forensic capabilities, knowledge, and specialized training on the law enforcement process.

Figure 7.

Map of suggested actions and biodefense objectives in the 2018 NBS. The left columns list the NBS sub-objectives (gray circles), and the right column lists our suggested policy actions for addressing gaps and limitations identified in our policy analysis (black circles).

The continued focus on pathogens and toxins as biological threats limits risk and threat assessment of enabling biotechnologies, which reflects another significant gap identified in our analyses. In addition, the NBS does not characterize the biodefense research sector as a critical infrastructure, resulting in a mismatch between that suggested action and the Strategy. Therefore, some of the actions identified in our analyses fall outside the scope of the 2018 NBS, even though they reflect realities of the current state of the biotechnology research landscape. Detailed analyses of the NBS objectives and changing biotechnology landscape are needed to understand fully the extent to which the NBS addresses the current and future biological threats.

Conclusions

The US policy landscape for countering biological threats is split into 2 main groups: (1) biosecurity, which specifically focuses on preventing theft, diversion, or deliberate malicious use of biological sciences knowledge, skills, and technologies; and (2) biodefense, which involves the development of capabilities and knowledge to assess, detect and monitor, treat (or vaccinate against), and respond to biological threats. These 2 groups often affect the same stakeholders, which may result in mutual benefits among policies in the same group or present barriers to achieving either defense or security objectives if in different groups. For example, policies promoting detection or medical countermeasure research and development could enhance capabilities to respond to biological incidents. Conversely, policies restricting research activities associated with certain pathogens may counteract efforts to develop medical countermeasures or pathogen detection methodologies and technologies. This countering effect may be mediated by the indirect effects of policy implementation or compliance on the workforce engaged in regulated activities, institutional capabilities to support regulated activities, and the extent and pace of regulated activities conducted by individuals and institutions. At the same time, the biotechnology landscape is changing dramatically, presenting new opportunities for building technological capabilities while also presenting security vulnerabilities.

The policy analysis undertaken to inform the actions described in this article involved a systematic evaluation of existing policies for harnessing new advances in the biological sciences and biotechnology and for preventing malicious or accidental harms caused by pathogens, toxins, and scientific advances. This systems-based approach enabled identification of limitations and gaps in the current policy landscape, including those emerging from federal and local-level implementation. The 2018 National Biodefense Strategy addresses many, but not all, of our identified gaps. Whether the identified gaps could be addressed during the implementation phase of the NBS is not clear at this time, but systematic evaluation of the outcomes and activities throughout implementation could help to identify early and address potential challenges and highlight or further support successful efforts.

Footnotes

Acknowledgments

This project was supported by a grant (No. FA7000-17-1-0010) from the Defense Threat Reduction Agency's Project on Advanced Systems and Concepts of Countering Weapons of Mass Destruction, which is administered through the US Air Force Academy. The project partners thank the US Air Force Academy and DTRA for supporting this project and all working group members for their contributions, insights, and peer review of all project methodologies, analyses, case studies, and final report. We thank our expert working group for their excellent review of all project analyses, datasets, and conclusions: Dr. Ronald Atlas, University of Louisville; Dr. Carole Baskin, Saint Louis University; Rear Admiral Kenneth Bernard (ret.); Dr. Ellen Carlin, EcoHealth Alliance; Dr. Sarah Carter, Science Policy Consulting; Ms. Eleanor Celeste, Vertex; Mr. Morgan Crafts, Northrop Grumman; Dr. Julie Fischer, Georgetown University; Dr. Matthew Frieman, University of Maryland; Dr. Margaret Kosal, Georgia Institute of Technology; Ms. Mary Beth Koza, University of North Carolina; Dr. Tim Stearns, Stanford University; Dr. Victoria Sutton, Texas Tech University; Dr. Jennifer Weisman, formerly with the Bill and Melinda Gates Foundation. We thank all the stakeholders with whom we spoke to ensure that the analyses and conclusions described in the report are accurate, relevant, and appropriate. We thank US government staff at the White House and Departments of Defense, State, Health and Human Services, and Agriculture, and the Environmental Protection Agency for providing us opportunities to discuss our interim analyses. We thank the American Biological Safety Association, the Association of Public Health Laboratories, the American Society for Microbiology, the Biotechnology Industry Organization, the Engineering Biology Research Consortium, the American Association for the Advancement of Science, the Council on Government Relations, the National Association of County and City Health Officials, the Council of State and Territorial Epidemiologists, the Association of State and Territorial Health Officials, and the National Environmental Health Association for helping us to engage their members during the course of the project.

The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies and endorsements, either expressed or implied, of the US Air Force Academy, the US government, the US Department of Defense, Parsons, or working group members and their affiliated organizations.

References

Biological Toxins Weapon Convention Implementation Support Unit. “Biosafety and Biosecurity.” United National Office at Geneva. Undated. https://www.unog.ch/80256EDD006B8954/(httpAssets)/46BE0B4ACED5F0E0C125747B004F447E/%24file/biosafety%2Bbackground%2Bpaper%2B-%2Badvanced%2Bcopy.pdf. Accessed February 22, 2019.

World Health Organization. Biorisk Management: Laboratory Biosecurity Guidance. Geneva: World Health Organization; 2006. https://www.who.int/csr/resources/publications/biosafety/WHO_CDS_EPR_2006_6.pdf. Accessed February 22, 2019.

U.S. Department of Health and Human Services. Frequently asked questions: national biodefense strategy. https://www.phe.gov/Preparedness/biodefense-strategy/Pages/faqs.aspx. Accessed February 22, 2019.

Centers for Disease Control and Prevention. BioSense platform. National Syndromic Surveillance Program (NSSP). Last reviewed October 31, 2017. https://www.cdc.gov/nssp/biosense/index.html. Accessed February 22, 2019.

The White House. National Biodefense Strategy. 2018. https://www.whitehouse.gov/wp-content/uploads/2018/09/National-Biodefense-Strategy.pdf. Accessed February 22, 2019.

The White House. National Security Strategy of the United States of America. December 2017. https://www.whitehouse.gov/wp-content/uploads/2017/12/NSS-Final-12-18-2017-0905.pdf. Accessed February 22, 2019.

U.S. Department of Defense. Summary of the 2018 National Defense Strategy of the United States of America. Washington, DC: U.S. Department of Defense; 2018. https://dod.defense.gov/Portals/1/Documents/pubs/2018-National-Defense-Strategy-Summary.pdf. Accessed February 22, 2019.

The White House. National Strategy for Counterterrorism of the United States of America. October 2018. https://www.whitehouse.gov/wp-content/uploads/2018/10/NSCT.pdf. Accessed February 22, 2019.

A.The White House. National Strategy for Countering Weapons of Mass Destruction Terrorism. December 2018. https://www.whitehouse.gov/wp-content/uploads/2018/12/20181210_National-Strategy-for-Countering-WMD-Terrorism.pdf. Accessed March 27, 2019.

10.

Consolidated Appropriations

Act

, Pub. L. No. 115-141 (March 23, 2018).

11.

Global Health Security Agenda Steering Committee. Draft Global Health Security Agenda (GHSA) 2024 Framework. June 29, 2018. https://www.ghsagenda.org/docs/default-source/default-document-library/draft-ghsa2024-framework-v2-july2018.pdf. Accessed February 22, 2019.

12.

Berger

KM.

The role of science in preparedness and response. U St Thomas LJ, 2009; 6:622.

13.

Watson

, Sell

, Watson

, et al. Technologies to Address Global Catastrophic Biological Risks. Baltimore, MD: Johns Hopkins Center for Health Security; 2018. http://www.centerforhealthsecurity.org/our-work/publications/technologies-to-address-global-catastrophic-biological-risks. Accessed February 22, 2019.

14.

Aebersold

FDA experience with medical countermeasures under the animal rule. Adv Prev Med, 2012; 2012:507571.

15.

Russell

, Gronvall

. U.S. medical countermeasure development since 2001: a long way yet to go. Biosecur Bioterror, 2012; 10(1):66-76.

16.

Kman

, Bachmann

. Biosurveillance: a review and update. Adv Prev Med, 2012; 2012:301408

17.

Moore

, Fisher

, Stevens

. Toward Integrated DoD Biosurveillance: Assessment and Opportunities. Santa Monica, CA: RAND; 2013. https://www.rand.org/pubs/research_reports/RR399.html. Accessed February 22, 2019.

18.

Nuzzo

JB.

Developing a national biosurveillance program. Biosecur Bioterror, 2009; 7(1):37-38.

19.

Atlas

, Dando

. The dual-use dilemma for the life sciences: perspectives, conundrums, and global solutions. Biosecur Bioterror, 2006; 4(3):276-286.

20.

Frankel

MS.

Regulating the boundaries of dual-use research. Science, 2012; 336(6088):1523-1525.

21.

Selgelid

MJ.

Governance of dual-use research: an ethical dilemma. Bull World Health Organ, 2009; 87(9):720-723.

22.

Bartholomew

, Omberg

Biodefense Policy Landscape Analysis Tool v2.0. Pacific Northwest National Laboratory; 2018. https://bplat.pnnl.gov/. Accessed February 22, 2019.

23.

World Health Organization. International Health Regulations (2005). Geneva, Switzerland: WHO; 2008. http://www.who.int/ihr/9789241596664/en/. Accessed February 22, 2019.

24.

Federal Emergency Management Agency. (2017). Biological Incident Annex to the Response and Recovery of Federal Interagency Operational Plans. Washington, DC. https://www.fema.gov/media-library-data/1511178017324-92a7a7f808b3f03e5fa2f8495bdfe335/BIA_Annex_Final_1-23-17_(508_Compliant_6-28-17).pdf.

25.

Bügl

, Danner

, Molinari

, et al. DNA synthesis and biological security. Nat Biotechnol, 2007; 25(6):627-629.

26.

Cho

, Relman

. Genetic technologies. Synthetic “life,” ethics, national security, and public discourse. Science, 2010; 329(5987):38-39.

27.

Clapper

JR.

Worldwide Threat Assessment of the US Intelligence Community. Washington, DC: Office of the Director of National Intelligence; February 9, 2016. https://www.dni.gov/files/documents/SASC_Unclassified_2016_ATA_SFR_FINAL.pdf. Accessed February 22, 2019.

28.

Garfinkel

, Endy

, Epstein

, Friedman

. Working Papers for Synthetic Genomics: Risks and Benefits for Science and Society. J. Craig Venter Institute; CSIS; MIT; 2007. https://www.jcvi.org/sites/default/files/assets/projects/synthetic-genomics-options-for-governance/Commissioned-Papers-Synthetic-Genomics-Governance.pdf. Accessed February 22, 2019.

29.

Garfinkel

, Endy

, Epstein

, Friedman

. Synthetic genomics: options for governance. Biosecur Bioterror, 2007; 5(4):359-362.

30.

Katz

, Zilinskas

, eds. Encyclopedia of Bioterrorism Defense. 2d ed. Hoboken, NJ: Wiley-Blackwell; 2011.

31.

Risk and Benefit Analysis of Gain of Function Research. Takoma Park, MD: Gryphon Scientific; 2016. https://www.gryphonscientific.com/gain-of-function/. Accessed February 22, 2019.

32.

Homeland Security Presidential Directive (HSPD) 10: Biodefense for the 21st Century. April 28, 2004. https://fas.org/irp/offdocs/nspd/hspd-10.html. Accessed February 22, 2019.

33.

Presidential Policy Directive (PPD) 2: National Strategy for Countering Biological Threats 2009. https://www.hsdl.org/?view&did=31404. Accessed February 22, 2019.

34.

US Government Accountability Office. Bioforensics: DHS Needs to Conduct a Formal Capability Gap Analysis to Better Identify and Address Gaps. Washington, DC: Government Accountability Office; 2017. https://www.gao.gov/assets/690/682007.pdf. Accessed February 22, 2019.

35.

Spinney

Forensic science braces for change. Nature July 22, 2010. https://www.nature.com/news/2010/100722/full/news.2010.369.html. Accessed February 22, 2019.

36.

Presidental Memorandum on the Support for National Biodefense. September 18, 2018. https://www.whitehouse.gov/presidential-actions/presidential-memorandum-support-national-biodefense/. Accessed February 22, 2019.

37.

Public Health Security and Bioterrorism Preparedness and Response Act of 2002, P.L. 107-188, June 12, 2002. https://www.govinfo.gov/content/pkg/PLAW-107publ188/pdf/PLAW-107publ188.pdf. Accessed February 22, 2019.

38.

Young

, Willis

, Moore

, Engstrom

. Measuring Cooperative Biological Engagement Program (CBEP) Performance: Capacities, Capabilities, and Sustainability Enablers for Biorisk Management and Biosurveillance. Santa Monica, CA: RAND; 2014. https://www.rand.org/content/dam/rand/pubs/research_reports/RR600/RR660/RAND_RR660.pdf. Accessed February 22, 2019.

39.

Organization for Economic Co-operation and Development. OECD Guidance on Safety Performance Indicators. 2003. https://www.oecd.org/env/ehs/chemical-accidents/48356891.pdf. Accessed February 22, 2019.

40.

Organization for Economic Co-operation and Development. Guidance on Developing Safety Performance Indicators Related to Chemical Accident Pervention, Preparedness and Response. 2d ed. 2008. http://www.oecd.org/chemicalsafety/chemical-accidents/41269639.pdf. Accessed February 22,2019.

41.

Centers for Disease Control and Prevention. Federal select agent program. https://www.selectagents.gov/. Accessed February 22, 2019.

42.

Pellerin

Watching & warning: DTRA's biosurveillance ecosystem. Armed with Science March 2, 2016. http://science.dodlive.mil/2016/03/02/watching-warning-dtras-biosurveillance-ecosystem/. Accessed February 22, 2019.

43.

Blue Ribbon Study Panel on Biodefense in 2018. https://www.biodefensestudy.org/Holding-the-Line-on-Biodefense. Accessed February 22, 2019.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.32 MB