Behavioral Risk Modeling for Pandemics: Overcoming Challenges and Advancing the Science

Abstract

Pandemic transmission dynamics are influenced by myriad variables, many of which are not captured in classic predictive models.^1-3 COVID-19 nonpharmaceutical interventions have spotlighted the inherently social nature of disease transmission, a particularly challenging area to model. Understanding the complex dynamics of human behavior sufficiently to enable inclusion in pandemic prediction models could improve the practical utility of models for decisionmakers.

How people prepare for and respond to outbreaks is a reflection of societal influences and social determinants of health. Data that reflect these factors, while frequently difficult to acquire,² are essential to allow policymakers and scientists to understand disease etiology and epidemiology, identify hotspots and trends, and engage the public in mitigation.^2-4 Human sociality, however, remains an underappreciated component of pandemic forecasting.^4,5

Incorporating human behavior into transmission modeling is challenging. Modelers often incorrectly assume that contacts are random, populations are homogeneous, and social/political and cultural attributes are irrelevant.^1,6 Traditional compartmental disease models may incorporate variables such as pathogen biology, vaccine and treatment efficacy, and vector seasonality.⁷ Yet they often fail to capture human behaviors, even when known to influence disease transmission locally and globally.^3,5,8 Behavior is correlated with complex variables like culture, social justice, social networks, trust in government, and systemic bias. Social media activity can alter human behaviors and impact the course of an outbreak by rapidly spreading information or misinformation.^9,10 Patterns of social mixing, healthcare-seeking, politically driven decisionmaking, and other behaviors may also influence disease transmission.^2,8,11,12

To date, these complexities have precluded development of forecasting models that meaningfully and reliably incorporate human behavior.^3,8 Behavior may not change in the linear ways implicitly assumed by many transmission models, leading to ineffective policies and interventions with unintended consequences. The COVID-19 pandemic has neatly demonstrated that human behavioral responses (or lack thereof) to policy interventions can change the course of an epidemic and therefore need to be fed back into iterative predictive models.²

In 2018, an informal working group held a workshop at the Smithsonian Institution in Washington, DC, in support of the Biological Defense Research and Development Subcommittee of the National Science and Technology Council Committee on Homeland and National Security. The purpose was to solicit expert input on the technical and policy impediments to better understanding human behavioral risk as it relates to pandemics and how to then incorporate that understanding into forecasting models. An interdisciplinary group of approximately 2 dozen scientists—eg, epidemiologists, ecologists, economists, veterinarians, anthropologists, policy specialists—from the US government, private sector, and academia convened for 2 days to explore scientific and policy solutions for improving the state of behavioral risk modeling for pandemics. The authors issued a report to the Biological Defense Research and Development Subcommittee, which is available upon request, based on the workshop and additional literature review.

Other papers have explored the importance of and challenges to incorporating human behavior into models.^4,5,12-16 Still, the policy prioritization of such work has not kept up with the need. Now, in the context of COVID-19, this commentary offers an updated perspective based on multidisciplinary input on technical and policy challenges to forecasting pandemics with evidence-based and nuanced considerations of human behavior.

Epidemiologic Models Applicable to Behavioral Risk

Classic compartmental epidemiological models assume that contacts between susceptible and infectious individuals depend on their relative frequency in a population—an approach that has proven useful for studying infectious disease transmission but omits behavioral factors that underpin contact rates.¹⁷ While such models can help produce early forecasts during an epidemic, they are unable to capture spatial and social heterogeneity; inadequately account for feedback between social, epidemic, and ecological system components; lack resolution for testing individual-level behavioral interventions; and fail to take full advantage of diverse data sources.¹⁸

The landscape of available epidemiological models has grown in the last decade, and many modeling approaches are now available that could be applied to behavioral risk research. Some are directly useful, and others could be adapted from successful approaches in other disciplines.^19-23 Table 1 outlines 3 model types that with further development could be important drivers of improved behavioral and risk understanding.

Table 1.

Key Modeling Approaches for Social and Behavioral Disease Transmission Dynamics

Model Type	Potential Model Applications	Model Examples
Behavior change Represents elements of human behavioral response to outbreaks	• Decision making for specific behavioral interventions • Complements disease transmission models	• Human movement models in resource poor settings: Ebola case study¹⁹ • Coupling outbreak progression, information access, and impact on behavior change⁸
Agent based Represents behavioral heterogeneity of agents or entities within synthetic populations with defined behaviors	• High-level decision making for societal interventions • Examination of stochastic and local outbreaks • Incorporation of elements like clustering and network analyses	• Simulation of outbreak dynamics using open data: measles case study²² • Transmission dynamics for COVID-19 lockdown effects using smartphones for contact tracing and herd immunity²⁴
Economic Analyzes economic determinants linked to human behavior and decision making; applies economic approaches to epidemiology; identifies economic impacts of outbreaks	• Used to investigate 3rd-order effects of financial markets and revenue or cost driven by an outbreak or bioterror attack • Used to incorporate economic drivers of social mixing along with cost-benefit estimations of contact risks	• Highly pathogenic avian influenza and the cost of control²³ • Cost-benefit perceptions of human interaction for driving transmission and adaptive human behavior²⁵

Behavioral change models are obvious platforms for building behavioral risk modeling capability. These models incorporate elements of behavior response and nonpharmaceutical interventions in disease transmission models.^3,6 In a systematic review covering 2010 to 2015, Verelst et al³ found 178 published behavioral change models for infectious disease transmission, most of which were disease-specific and related either to influenza or influenza-like illness. The study's authors were concerned that these models tend toward the purely theoretical, are reactive to the emergent disease of the day, and lack representative data and validation processes. They cited a need for more empirical research to help base the models on real-life behavior and disease transmission. The kinds of data that could inform and improve such models include elements such as public sources of information, which information the public trusts, and how individuals and groups act upon that information.⁴ In the case of COVID-19, those early “behavioral immunity” responses were unevenly sustained and were not always effective. Theory-driven behavioral change models could not explain why.²⁶

In agent-based infectious disease models, an “agent” may represent individuals, households, governments, or any entity of interest that may impact transmission dynamics.²⁷ Agent-based models are particularly useful for infectious disease analysis since interactions among these entities during an outbreak give rise to population-level patterns of incidence and persistence.²⁷ Agent-based modeling is gaining traction in the behavioral change literature because, unlike standard compartmental modeling, it can incorporate behavioral heterogeneity and clustering to examine stochastic and local outbreaks.³ Agent-based models using known pathogen behavior can apply a disease-specific model to a synthetic population, define risks and behaviors at the individual and societal levels, and simulate transmission dynamics in that population, making them attractive for policymakers faced with intervention decisions.¹⁸ While some diseases, like COVID-19, need only weak social ties for transmission, behavior changes—such as compliance with mandated disease mitigation policies—often require stronger sustained network ties to become established.²⁶Agent-based models that address these complexities of pathogen, human, governmental, population, and other levels of behavior may provide valuable modeling tools, but will require substantially improved datasets to meet their potential.

An emerging class of models that leverages insights and methods from economics is known as economic epidemiology or epidemiological economics. Disease epidemiology and economic analyses are similar in that they seek to understand processes, create forecasts, and estimate policy effects.²⁸ An economics-inspired approach to epidemiology can address feedback between infectious disease dynamics and behavioral decisions.¹⁷ Economic epidemiology models are exquisitely relevant to disease transmission problems because economic drivers of social mixing along with cost-benefit estimations of contact risks are often key influencers of behavior.¹⁷ Economic models can be applied to influential variables, such as the “fear factor” that drives people to avoid potentially contaminated surfaces, mass gatherings, public transportation systems, and other venues. This can be relevant for disease agents that can sporulate or otherwise remain viable environmentally and for bioterrorism attack scenarios. Economic epidemiology models based on incentivizing desired behaviors may be particularly useful for developing policy interventions.¹⁷

Challenges to Optimizing Behavior Risk Modeling for Decisionmaking

These and many other modeling approaches could be developed to maturity and leveraged for the complex task of modeling pandemic risk. The following 4 key challenges must be overcome to realize the potential of pandemic risk models for more accurately informing situational awareness and planning: 1.

Data limitations stymie modeling capabilities. Insufficient availability of standardized, longitudinal datasets is one of the foremost technical challenges precluding model development. It presents a major obstacle for incorporating behavioral risk in the near term. Data that do exist may be cross-sectional rather than longitudinal, have often been captured via methodologies that are inconsistent across research projects and initiatives, and inadequately reflect variability across cultures and regions. Interactions of diverse stressors—eg, comorbidities, environmental toxins, natural disasters, conflict, socioeconomic inequality—are poorly understood in the context of pandemic threats, as are the complex influences of different interventions on behavioral choices.^6,29 A tendency toward population-level compartments in models sidesteps integrating responses of individuals and social groups to outbreaks.¹³

Data demands of advanced modeling require more than traditional disease surveillance paradigms can provide. Traditional data collection methods used to support models include biosurveillance of pathogens and diseases circulating in animal and human populations and primarily quantitative human behavioral surveys. Such approaches are necessary but not sufficient. Some workshop participants offered that because the modeling community and potential partners have not deliberated over this problem, it is not yet clear what the data demands are in terms of type and volume. We have an opportunity to move beyond older paradigms of data collection to construct advanced models for forecasting pandemics.

Social network analytics for health security could be a game-changer, with prioritization and oversight. Emerging network analytic tools could have considerable impact on the efficacy of infectious disease models. However, the development of emerging technologies into mature public health tools is not yet an institutionalized priority for the federal government. Understanding social networks, whether physical or virtual, is requisite for identifying behaviors that create, amplify, or mitigate actual and perceived risk. An improved understanding of contact behavior, including the conditions that drive contact changes and the “corrections” that should be applied to reasonably export contact patterns from one setting to another, could improve our ability to predict the course of a disease.^17,20 Network analysis could enable decisionmakers to identify otherwise unexpected outcomes, such as when conditions that accelerate viral spread can unexpectedly inhibit the spread of behaviors.³⁰ The power of more advanced social media analytics to provide even more powerful tools has not been fully exploited and incentivized.

Interdisciplinary approaches are uncommon. The previously mentioned challenges and the ecological and behavioral layers that characterize infectious disease dynamics necessitate interdisciplinary study and interventions. Yet collaborative methodologies are still largely the subject of discussions at One Health conferences, and less the operating norm. While modelers have made efforts to incorporate social dynamics into disease modeling, the broadly interdisciplinary modeling studies and funding vehicles to support them are limited. Many infectious disease models incorporate vaccination rates, but those rates can be dependent in complex ways on public concerns of vaccine-related side effects, which can be influenced by social networks facilitated by social media and other messaging venues.^15,30,31 Psychologists can use psychological principles to understand personal decisions affecting vaccine coverage, which could in turn be integrated into models. Weston et al³² recommended that as modelers develop infectious disease models, they look to well-established health behavior change theories to identify key constructs that have been undermodeled. Many other dynamics are also at play. For example, human and animal behavior will change in response to climate change, human behavior affects the rate at which antimicrobial resistance develops and alters disease transmission dynamics, and changing land use and land cover patterns narrow or even eliminate the boundary between humans and animals—all of which increase spillover risk. Capturing these many confounders will only be achieved through research and research grants that require multiple fields to collaborate and seek solutions as a unit.

A Path Forward: Advancing Research Priorities

The workshop that informed the authors' findings was conducted before the emergence of SARS-CoV-2; therefore, the key challenges it revealed are not specific to COVID-19. Given the many models that have permeated COVID-19 discussions and response,³³ the findings provide an opportunity for a reinvigorated and timely discussion of the relevance of behavior in pandemic preparedness and response, and the importance of including human behavior variables in modeling.

Models that better incorporate human behavior will improve decisionmaking. Constructive examples are available from other fields that have faced this problem, such as homeland security and natural disaster response modeling.^34-36 Identifying how modeling has evolved during COVID-19 may also be instructive. Many organizations are working to improve COVID-19 predictions to enable more effective mitigation steps. Some innovative modeling efforts have been developed to incorporate the estimated effects of behavior modifications by amending traditional compartmental models or assigning factors to represent reduced contact rates to transmission parameters.^37-39 For example, Google is capturing changes in movement trends by tracking locations that people visit.⁴⁰ The Delphi Research Group at the Carnegie Mellon University took another approach, using the social media platform Facebook to conduct 50,000 daily opt-in surveys of Facebook users in the United States, the Joint Program in Survey Methodology at the University of Maryland undertook a similar effort for international Facebook users.⁴¹ Survey respondents reported health and behavior information for household members, including the extent of mask wearing, number of social contacts, attitudes toward COVID-19 vaccinations, degree of anxiety concerning COVID-19, measures required for in-person classes for household schoolchildren, and type of employment.⁴² A collaborative challenge tasked participants to develop novel analytical approaches using the open-source survey data to enable earlier outbreak detection and incorporation into policy tools and processes that can inform disease control policy decisions and actions by health authorities.⁴³ This incorporation of behavior data into COVID-19 tracking dashboards and forecasts would allow behavioral factors to be taken into account in predictions and trends in transmission over time.

The policy and program implementation communities are particularly incentivized in this COVID-19 moment to help advance scientific efforts to improve decisionmaking. Government and private-sector partners together can address a few items that could bear fruit in the near to medium term. Endeavor in 4 areas now appears particularly critical: (1) identifying the kinds of behavioral and social data most needed to support models, (2) developing a research approach that can realistically address data limitations and standardize data collection, (3) pursuing social media data access and analytics that could be incorporated into infectious disease models, and (4) initiating a governmentwide effort to prioritize and build structures that elevate interdisciplinary modeling science.

Generating enhanced datasets and frameworks to identify the parameters most needed will be required to close information gaps. Standardization of tools for data collection and analysis could assist in building extensive sets of comparable data. Learning from models that do not meet performance expectations (eg, that fail to accurately forecast), as well as those that do, will be a critical practice that is not yet an industry standard. Published models will also need to be transparent about their assumptions when data sources for parameterization or validation are sparse.³

Importantly, investment in infectious disease models without investing in tools to incorporate social network elements misses a major opportunity to improve the predictive capacity of the models on which decisionmakers are becoming dependent. Government should be working in partnership with the commercial sector to support much greater advancement of such tools for public health good. Social media data and individual-level movement and geographic data are available at an unprecedented scale. As we look to advance their use, unresolved privacy and security concerns in using them must also be addressed.

Finally, traditional siloed funding approaches simply do not work to solve scientific problems that require interdisciplinary solutions. How social science is integrated into complex public health problems that require multidisciplinary inputs will determine how predictive any prediction model is by allowing researchers to account for traditional knowledge, human movement and mixing, syndemics or the interacting effects of social and health disparities,²⁹ behavioral changes during epidemics and pandemics, social ecological dynamics, health communication strategies, and the role of media in risk communication and risk perception. Epidemiology, ecology, climatology, microbiology, anthropology, economics, psychology, sociology, modeling, data analytics, and many other disciplines that support modeling science on their own will require impetus and avenues to come together.

Complex systems research requires space to build successful partnerships among scientists. The governmental structures—agencies, programs, and funding vehicles—that can support model development can also benefit from looking to examples from other areas where interdisciplinary constructs have been fruitful. We draw readers' attention to the following structures that could provide the national-level support to dramatically enhance the interdisciplinary science needed to advance behavioral risk modeling for pandemics in the near to medium term:

The National Science Foundation and National Institutes of Health support funding opportunities that advance collaborative efforts. Orienting even more of their research calls toward crosscutting modeling science could spark a new wave of collaborative research to advance modeling science.

The National Institute for Mathematical and Biological Synthesis and the National Center for Ecological Analysis and Synthesis provide an existing forum for interdisciplinary data exchange, partnering, training, and research. These entities can be leveraged directly to support more behaviorally oriented data collection and modeling science.

The National Center for Epidemic Forecasting and Outbreak Analytics, which will be established as directed by National Security Memorandum 1 in 2021,⁴⁴ could offer a centralized forum for idea exchange, scientific advancement, priority setting, and funding to further enable interdisciplinary approaches to behavioral risk modeling and other modeling efforts, supporting improved analytics and decisionmaking for high-consequence infectious diseases.

Conclusion

With each major disease event the global community faces, evidence mounts that human behavior has an enormous influence on disease emergence and transmission. Although much progress has been made in infectious disease modeling to date, the prioritization of interdisciplinary research and the systematic inclusion of behavioral variables into models could dramatically advance the field and improve accuracy and application of risk models for preparedness and response to infectious disease events. Improved models to inform the social-behavioral contours of the COVID-19 pandemic are needed urgently,³³ as are others that can inform future pandemic planning and response. The scientific underpinnings that could support improved models are rapidly progressing and can be leveraged and bolstered by additional attention and prioritization on the part of policymakers and research agencies.

The reliability of most current models is predicated upon the specific qualities of the pathogen or disease under investigation.^7,16 Whether a generally applicable, 1-size-fits-all model can come to fruition remains to be seen. At a minimum, however, it may be possible to combine multiple diseases with the same transmission and prevention properties into behavioral change or other models.³ Doing so may help us not only with responding, but also with developing preventive solutions by improving understanding of the variables that drive pathogen emergence.⁴⁵

Footnotes

Acknowledgments

The authors thank the subject matter experts who participated in an October 2018 behavioral risk modeling workshop coordinated by the Pandemic Prediction and Forecasting Science and Technology Working Group, an informal federal interagency group of which the authors were members at the time. We also thank the Smithsonian Conservation Biology Institute for fiscally supporting the workshop; Sabrina Sholts and Robert Costello of the National Museum of Natural History for providing an excellent forum in which to have the meeting; Elizabeth Ashby and Amanda Burton, who, as student externs with the Smithsonian Conservation Biology Institute Global Health Program, assisted with literature research and other activities in support of this project; Noam Ross for his review of an early version of the manuscript; the National Science Foundation Cultural Anthropology Program for participant travel support; and the anonymous peer reviewers.

References

Poletto

, Scarpino

, Volz

. Applications of predictive modelling early in the COVID-19 epidemic. Lancet Digit Health. 2020; 2(10):e498-e499.

Press

, Levin

. Modeling, post COVID-19. Science. 2020; 370(6520):1015.

Verelst

, Willem

, Beutels

. Behavioural change models for infectious disease transmission: a systematic review (2010-2015). J R Soc Interface. 2016; 13(125):20160820.

Funk

, Salathé

, Jansen

. Modelling the influence of human behaviour on the spread of infectious diseases: a review. J R Soc Interface. 2010; 7(50):1247-1256.

Funk

, Bansal

, Bauch

, et al. Nine challenges in incorporating the dynamics of behaviour in infectious diseases models. Epidemics. 2015; 10:21-25.

Jones

, Hazel

, Almquist

. Transmission-dynamics models for the SARS Coronavirus-2. Am J Hum Biol. 2020; 32(5):e23512.

Grassly

, Fraser

. Mathematical models of infectious disease transmission. Nat Rev Microbiol. 2008; 6(6):477-487.

Perra

, Balcan

, Gonçalves

, Vespignani

. Towards a characterization of behavior-disease models. PLoS One. 2011; 6(8):e23084.

, Wang

, Liu

, Li

. An SEI infection model incorporating media impact. Math Biosci Eng. 2017; 14(5-6):1317-1335.

10.

Roche

, Gaillard

, Léger

, et al. An ecological and digital epidemiology analysis on the role of human behavior on the 2014 Chikungunya outbreak in Martinique. Sci Rep. 2017; 7:5967.

11.

Kim

, Paini

, Jurdak

. Modeling stochastic processes in disease spread across a heterogeneous social system. Proc Natl Acad Sci U S A. 2019; 116(2):401-406.

12.

Anaki

, Sergay

. Predicting health behavior in response to the coronavirus disease (COVID-19): worldwide survey results from early March 2020. PLoS One. 2021; 16(1):e0244534.

13.

Ferguson

Capturing human behaviour. Nature. 2007; 446(7137):733.

14.

Bauch

, Galvani

. Social factors in epidemiology. Science. 2013; 342(6154):47-49.

15.

Manfredi

, D'Onofrio

, eds. Modeling the Interplay Between Human Behavior and the Spread of Infectious Diseases. New York: Springer-Verlag; 2013.

16.

Willem

, Verelst

, Bilcke

, Hens

, Beutels

. Lessons from a decade of individual-based models for infectious disease transmission: a systematic review (2006-2015). BMC Infect Dis. 2017; 17(1):612.

17.

Perrings

, Castillo-Chavez

, Chowell

, et al. Merging economics and epidemiology to improve the prediction and management of infectious disease. Ecohealth. 2014; 11:464-475.

18.

Venkatramanan

, Lewis

, Chen

, Higdon

, Vullikanti

, Marathe

. Using data-driven agent-based models for forecasting emerging infectious diseases. Epidemics. 2018; 22:43-49.

19.

Kraemer

, Golding

, Bisanzio

, et al. Utilizing general human movement models to predict the spread of emerging infectious diseases in resource poor settings. Sci Rep. 2019; 9(1):5151.

20.

Kraemer

, Perkins

, Cummings

, et al. Big city, small world: density, contact rates, and transmission of dengue across Pakistan. J R Soc Interface. 2015; 12(111):20150468.

21.

Funk

, Gilad

, Watkins

, Jansen

. The spread of awareness and its impact on epidemic outbreaks. Proc Natl Acad Sci U S A. 2009; 106(16):6872-6877.

22.

Hunter

, Mac Namee

, Kelleher

. An open-data-driven agent-based model to simulate infectious disease outbreaks. PLoS One. 2018;13(12):e0208775. Published correction appears in PLoS One., 2019; 14(1):0211245.

23.

Backer

, van Roermund

, Fischer

, van Asseldonk

, Bergevoet

. Controlling highly pathogenic avian influenza outbreaks: an epidemiological and economic model analysis. Prev Vet Med. 2015; 121(1-2):142-150.

24.

Shamil

, Farheen

, Ibtehaz

, Khan

, Rahman

. An agent-based modeling of COVID-19: validation, analysis, and recommendations. Cognit Comput. February 19, 2021. doi:10.1007/s12559-020-09801-w

25.

Fenichel

, Castillo-Chavez

, Ceddia

, et al. Adaptive human behavior in epidemiological models. Proc Natl Acad Sci U S A. 2011; 108(15):6306-6311.

26.

Centola

How Behavior Spreads. Princeton, NJ: Princeton University Press; 2018.

27.

Tracy

, Cerdá

, Keyes

. Agent-based modeling in public health: current applications and future directions. Annu Rev Public Health. 2018; 39:77-94.

28.

Avery

, Bossert

, Clark

, Ellison

. An economist's guide to epidemiology models of infectious disease. J Econ Perspect. 2020; 34(4):79-104.

29.

Singer

, Rylko-Bauer

The syndemics and structural violence of the COVID pandemic: anthropological insights on a crisis. Open Anthropol Res. December 25, 2020. doi:10.1515/opan-2020-0100

30.

Centola

Change: How to Make Big Things Happen. New York: Little Brown Spark; 2021.

31.

Brewer

, Chapman

, Rothman

, Leask

, Kempe

. Increasing vaccination: putting psychological science into action. Psychol Sci Public Interest. 2017; 18(3):149-207.

32.

Weston

, Hauck

, Amlôt

. Infection prevention behaviour and infectious disease modelling: a review of the literature and recommendations for the future. BMC Public Health. 2018; 18(1):336.

33.

Gnanvi

, Salako

, Kotanmi

, Glèlè Kakaï

. On the reliability of predictions on Covid-19 dynamics: a systematic and critical review of modelling techniques. Infect Dis Model. 2021; 6:258-272.

34.

Rose

AZ.

A framework for analyzing the total economic impacts of terrorist attacks and natural disasters. J Homel Secur Emerg. 2009; 6(1):9.

35.

Burns

. Modeling the social amplification of risk following a terrorist strike: methodological challenges. 2009. Accessed October 7, 2020. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.689.3500&rep=rep1&type=pdf

36.

Baucum

, Rosoff

, John

, Burns

, Slovic

. Modeling public responses to soft-target transportation terror. Environ Syst Decis. 2018; 38:239-249.

37.

Kim

, Seo

, Jung

. Prediction of COVID-19 transmission dynamics using a mathematical model considering behavior changes in Korea. Epidemiol Health. 2020; 42:e2020026.

38.

Courtemanche

, Garuccio

, Le

, Pinkston

, Yelowitz

. strong social distancing measures in the United States reduced the COVID-19 growth rate. Health Aff (Millwood). 2020; 39(7):1237-1246.

39.

Acuña-Zegarra

, Santana-Cibrian

, Velasco-Hernandez

. Modeling behavioral change and COVID-19 containment in Mexico: a trade-off between lockdown and compliance. Math Biosci. 2020; 325:108370.

40.

Google. See how your community is moving around differently due to COVID-19. Accessed January 26, 2021. https://www.google.com/covid19/mobility?hl=en

41.

Foresman

Carnegie Mellon using Facebook to collect COVID-19 data. EdScoop. Published November 24, 2020. Accessed January 26, 2021. https://edscoop.com/carnegie-mellon-facebook-survey-covid-19/

42.

Delphi Epidata

API

. COVID-19 Trends and Impact Survey. Accessed July 21, 2021. https://cmu-delphi.github.io/delphi-epidata/symptom-survey/

43.

Catalyst @Health 2.0. The COVID-19 symptom data challenge. Accessed January 26, 2021. https://www.symptomchallenge.org/challenge

44.

The White House. National security memorandum on United States global leadership to strengthen the international COVID-19 response and to advance global health security and biological preparedness. Published January 21, 2021. Accessed January 22, 2021. https://www.whitehouse.gov/briefing-room/statements-releases/2021/01/21/national-security-directive-united-states-global-leadership-to-strengthen-the-international-covid-19-response-and-to-advance-global-health-security-and-biological-preparedness/

45.

White

, Forester

, Craft

. Disease outbreak thresholds emerge from interactions between movement behavior, landscape structure, and epidemiology. Proc Natl Acad Sci U S A. 2018; 115(28):7374-7379.