Abstract
Tasked with analyzing the effectiveness of the Department of Defense's (DoD's) global health engagements, the Uniformed Services University of the Health Sciences (USU) used the Measures Of Effectiveness in Defense Engagement and Learning (MODEL) study to conduct a qualitative analysis of the DoD's response efforts to the Ebola pandemic in West Africa. The research aims to summarize the findings of studies that monitor and evaluate the DoD's response to the Ebola pandemic or compare the effectiveness of different DoD response activities; it further aims to identify common themes around positive and negative lessons learned and recommendations that can be applied to future DoD humanitarian assistance and disaster response efforts. The search included documents and observations from PubMed, Disaster Lit: Resource Guide for Disaster Medicine and Public Health, the Joint Lessons Learned Information System, the DoD and US Africa Command websites, and Google Scholar. The records selected from the search were analyzed to provide insights on the DoD's humanitarian assistance and disaster response engagements that could be employed to inform future operations and policy. Furthermore, the research identifies strengths and gaps in military capabilities to respond to disasters, which can be used to inform future training and education courses. Overall, the findings demonstrate the importance of monitoring, evaluating, and assessing disaster response activities and provide new evidence to support the implementation of activities, in accordance with the Global Health Security Agenda, to strengthen all-threat prevention, detection, and response capabilities worldwide.
This article summarizes the findings of studies that monitored and evaluated the DoD's response to the Ebola pandemic or compared the effectiveness of different DoD response activities; it further aims to identify common themes around positive and negative lessons learned and recommendations that can be applied to future DoD humanitarian assistance and disaster response efforts.
T
Operation United Assistance (OUA) began on September 16, 2014, when President Obama announced that, at the request of the Liberian government, US forces would establish a military command center in Liberia to support civilian efforts across the region and would provide support in command and control, logistics, engineering, and training assistance. During the course of the response, the DoD's activities were limited to these 4 lines of effort. 3 As a result of OUA, 1,539 healthcare workers and support staff were trained, a 30-person support team of civilian medical professionals was established, 10 DoD Ebola treatment units were constructed (including a 25-bed medical unit in Monrovia), 7 mobile labs were assembled, more than 4,500 samples were processed, and 1.4 million sets of personal protective equipment (PPE) were provided. 4 Given the unique nature of the response efforts required to manage the outbreak, taking stock of DoD's lessons learned from OUA is critical to ensuring improved preparedness for future global health emergencies.
Tasked with examining the effectiveness of the DoD's global health engagements, the Uniformed Services University of the Health Sciences (USU), using the Measures Of Effectiveness in Defense Engagement and Learning (MODEL) study and funded by the Assistant Secretary of Defense for Health Affairs, has done extensive work analyzing lessons learned from various sources. In collaboration with USU's Department of Preventive Medicine and Biostatistics, the MODEL study group published an article in April 2015 highlighting the results of a summative qualitative content approach used to analyze lessons learned from global health engagements, as recorded in the Peacekeeping and Stability Operations Institute's Stability Operations Lessons Learned and Information Management System. This research highlighted themes addressed by the DoD's Global Health Working Group and identified common capabilities and lessons learned. 5 The MODEL study adapted this approach to analyze lessons learned in the Joint Lessons Learned Information System (JLLIS) as well.
Based on this experience, MODEL used a similar approach to analyzing lessons learned in extant literature focused on the DoD's response to the Ebola pandemic in West Africa. We conducted a qualitative content analysis of literature from multiple databases and incorporated descriptive statistics to track DoD's engagements in West Africa over time. This approach draws themes from a collection of documents around positive and negative lessons learned and recommendations that can be applied to future DoD humanitarian assistance and disaster response efforts. In this article, we aim to provide senior thought and policy leaders with information for making pandemic response policy, advising on global health policy matters, expanding on US government pandemic response capability strengths and closing capability gaps, and demonstrating the importance of publishing monitoring and evaluation activities related to US engagements.
Materials and Methods
Before beginning the analysis of the DoD's lessons learned from OUA, we decided to illustrate how the Ebola virus progressed over time in relation to response efforts implemented by the DoD. We identified key DoD-led events, such as those noted above, and illustrated when these events took place in relation to the number of new Ebola cases by week (Figure 1). The DoD's response efforts began in West Africa shortly after the WHO identified the Ebola outbreak as an “extraordinary event” and as the incidence rate for Ebola began to dramatically increase in Liberia in early August 2014. 6 The DoD continued its response efforts in West Africa—ranging from direct support, research and development, Ebola treatment units, mobile labs, PPE, training exercises, provision of personnel, and the like—until the end of April 2015. While the unique contribution that the DoD made to reducing Ebola cases and deaths in West Africa cannot be determined, one can observe that the DoD's response efforts in Liberia occurred at the peak of the pandemic, and, following their implementation, the number of new Ebola cases started to decline.

DoD Ebola Response Timeline
Overall, 6 databases were used for the analysis, containing documents ranging from grey literature (ie, materials from noncommercial publishing sources) to peer-reviewed articles. The first step in the analytical process itself was the identification of sources from which to search for literature. Initially, we wanted to focus the analysis solely on peer-reviewed articles in PubMed. Since the response effort did not begin until late 2014, we were concerned that the recent nature of the events might preclude the extant literature from having explored the response from an assessment perspective. Therefore, we decided to include additional sources of literature that varied in methodological rigor. Accordingly, we decided to search literature from Disaster Lit: Resource Guide for Disaster Medicine and Public Health, which is the US National Library of Medicine's database of free online resources on disaster medicine and public health. We also selected the Joint Lessons Learned Information System, the joint staff database of lessons learned, as a source of literature, as it contains military- and operations-specific lessons learned observations and documents not found in publicly accessible databases. Finally, Google Scholar, the news section of the DoD website, and the US Africa Command (AFRICOM) website were selected as sources of grey literature.
The literature considered for the analysis were those items available as of December 2013, when patient zero, a 2-year old boy from Guinea, died from an unidentified hemorrhagic fever (confirmed to be Ebola on March 22, 2014). 7 In order to capture as many potential documents and observations as possible, the search terms used to collect relevant literature for the analysis were “Ebola,” “EVD,” and “response.” Using these search terms, a series of 3 searches (October 2015, May 2015, January 2016) was conducted, resulting in 177 items for further review, including 56 peer-reviewed articles, 33 pieces of grey literature, 43 JLLIS observations, and 45 JLLIS documents. (The JLLIS observations and documents are herein referred to collectively as JLLIS records.) Since these items included information on the general US government response, we conducted a second review of the 177 search results (excluding the JLLIS records) by using the search terms “defense” and “DoD” to limit the literature employed for the analysis to only those documents directly pertaining to the DoD response. We reviewed the JLLIS records separately in order to attempt to parse out duplicate entries. At the end of the secondary search, we were left with a total of 24 records for qualitative analysis, of which 14 were from JLLIS, and 10 were from non-JLLIS sources.
We classified records based on methodological elements (publication type, date of publication, country of implementation), strategic elements (leading organization, methods of evaluation for the response), and description of response efforts. Once classified, we coded the response effort descriptions using a qualitative content analysis approach to coding, which involved a series of interconnected steps: reading, coding, displaying, reducing, and interpreting the data. 8 During the primary coding phase, we assigned codes to units of text and organized them into the following themes: positive lessons learned, negative lessons learned, and recommendations. The secondary coding further refined the codes by breaking down the units of text based on the type of positive lesson learned, negative lesson learned, or recommendation described (Table 1). Once we had completed the secondary coding process, we analyzed the resulting codes under each theme to identify central observations, relationships, and patterns in the data.
Lessons Learned and Recommendation Codebook
Results
Lessons Learned
Through the coding process, we identified 7 categories of positive lessons learned and 8 categories of negative lessons learned (Figure 2). The most frequently coded category of positive lessons learned is “coordinated response” (8 times). In terms of what worked well, lessons in this category made specific mention of the importance of liaison officers, whose purpose is to “facilitate communications between elements of a joint task force to ensure mutual understanding and unity of purpose and action.” 9 According to the lessons learned, not only did liaison officers play a critical role in mission success in Liberia, they were also essential to interagency coordination in terms of Coast Guard actions related to the domestic response. In addition to personnel such as liaison officers who served a coordinating function, it was noted that the use of a coordinating tool such as the synchronization matrix, which conveys the status of each element of an activity with respect to other elements and the overall goal, helped to reduce confusion and was shared by the DoD with partners.

Lessons Learned Frequencies
In concert with strong coordination, the results show that other strengths of the DoD response included partnership and cooperation, each of which was coded 4 times. Formulating host nation partnerships was discussed with respect to leveraging existing military-to-military relationships forged prior to the Ebola crisis and ongoing military programs in Liberia, such as Operation Onward Liberty, that enabled the US military to work with the armed forces of Liberia on planning, logistics, and construction of the Ebola treatment units. 10 Engaging the host nation military was also central to ensuring that these construction projects were perceived to have some degree of host nation “ownership,” which is a key element of building trust and public confidence among the host nation population. Additionally, fostering cooperation among partners, such as the WHO and NGOs, helped to bridge coordination and communication gaps and was perceived to have a positive impact on community morale. According to the lessons learned, the robust use of liaison officers to establish these relationships improved cross-organizational communications and established trust among diverse organizations with different operational cultures.
Ebola virus disease response plans is the second most frequently coded positive lesson learned (6 times), though it should be noted that 4 of the 6 coded segments of text were drawn from US Coast Guard observations that refer specifically to Coast Guard activities as a part of domestic Ebola preparedness and response efforts. These lessons indicated that established response processes were effectively used to support interagency cooperation and that transparency and clarity of processes to manage Ebola were essential to building trust and public confidence in domestic preparedness and response efforts.
Despite the inclusion of positive lessons learned, much of the literature used for this article focused more heavily on criticisms of the DoD's response and provided recommendations on how to improve areas of perceived weakness or inefficiency. “Insufficient guidance” is the most commonly employed code and reported weakness in the domestic response (17 times). Given the unprecedented nature of OUA, standards and requirements necessary to undertake certain tasks, such as Ebola treatment unit construction, were reported as not being in place. Furthermore, there was no formal procedure for the “property transfer” of the Ebola treatment units to the Liberian government. On another note, there was a lack of training and acquisition processes required to ensure that the appropriate types and numbers of PPE were made available, which were essential for conducting healthcare worker training sessions. There was also insufficient policy for such an event, which in turn led to confusion between DoD and interagency partners regarding DoD's lines of effort and the issuance of conflicting guidance. One record discussed how, in the absence of timely guidance, units were forced to create and implement their own procedures, which then had to be modified once official guidance was received. 11
Weak coordination at all levels proved to be a challenge, whether at the interagency level, between an agency and military component, or between US forces and international partners. According to lessons learned, poor coordination with regard to reporting requirements, reporting mechanisms, mission planning, and information sharing in some ways inhibited a necessary or appropriate course of action. Coordination is linked to communication, and, as highlighted in one record, communication can also be a major limiting factor. The record points out that in order to get things done, face-to-face relationships must be formed, though this remains a challenge in the joint interagency environment across civilian and military ranks. 12 This in turn posed a barrier to using a common operating language and getting to a mutual understanding of the mission. In terms of physical communications mechanisms, the DoD's propensity to default to the use of the SECRET Internet Protocol Router Network (SIPRNET), whether or not data were classified, imposed a barrier to communication and collaboration both internal and external to the DoD.
Records pertaining to insufficient training as a negative lesson learned discussed how rapid deployment for OUA resulted in little time to provide forces with adequate mission-specific training and preparation. In particular, this lack of mission-specific training included donning and doffing PPE, which was noted several times. Further criticism in this realm was a lack of infectious disease training and experience and, more specifically, the fact that the DoD only had 2 doctors with EVD clinical patient care experience. It was also noted that although the Ebola outbreak was not officially classified as a chemical, biological, radiological, or nuclear (CBRN) event, some of the skill sets required were similar to defense support for civilian authorities and required CBRN-like decontamination methods for cleaning aircraft and equipment.
Additional negative lessons learned highlighted insufficient resources including: PPE (correct type and number); communication tools needed by contracting officer representatives, such as phones, computers, and mobile internet; construction materials required to complete Ebola treatment unit projects; insufficient laboratory equipment; and inter-theater evacuation aircraft, which resulted in having to contract out medevac flights that were unreliable. 13
Recommendations
In concert with the lessons learned, most pieces of literature reviewed for this article, particularly JLLIS records, provided recommendations for improving the DoD's response efforts (Figure 3). The most frequently coded recommendation category was “improved mission planning,” which pertains to a broad spectrum of mission planning deficiencies that may have contributed to complications with mission execution (15 times). A need for improved situational understanding on the ground, especially concerning equipment and personnel requirements, and improved logistics was noted. On the medical front, it was recommended that a medical planner be included with the farthest forward planning unit and that the DoD develop a database and planning primer of its niche medical capabilities. Other recommendations included the integration of NGOs into the planning process and conducting the majority of planning and work in the unclassified environment.

Recommendation Frequencies
As the Ebola outbreak was a first of its kind event in terms of the US government response, a key recommendation made related to guidance was the development of an interagency playbook that would guide a “whole-of-government” approach to response. Such guidance would codify duties and responsibilities of US agency partners for humanitarian assistance operations, including epidemic response, and would enable the formulation of a common operating picture that is shareable with US government and external partners. In the case of OUA, the DoD developed disease response policies, many of which remained in draft form. While the operational need continued to grow, policymaking mechanisms could not respond quickly enough, leading to the suggestion that pre-crisis policy concerning the following areas be established: inter- and intra-theater transport of highly infectious personnel, contaminated remains and material, infectious medical specimens, and personnel redeployment adjusted for risk of infection by disease category. Another sentiment echoed throughout the lessons learned and recommendations was the need for standardized PPE guidance.
As per the benefits of host nation partnerships indicated in the positive lessons learned, the continued use of civil affairs teams to work with local governments to determine local capabilities was recommended. Along these lines, community outreach programs (ie, going to town hall meetings) and face-to-face interactions were identified as being central to building trust and essential to the successful establishment of a training center in a sovereign nation and facilitating construction. Further recommendations in this area included maintaining close contact with the US embassy leadership at all levels of engagement, as well as tying in with the UN cluster system and leveraging NGO support networks, which can be critical during humanitarian assistance and disaster response missions.
Training is another code that appeared in lessons learned and recommendations. A thorough review of training requirements was advised to ensure that training central to expeditionary operations is available to US service members prior to deployment. Maintaining consistency with training standards was also reported as a need. In addition to individual and collective predeployment training, it was recommended that leaders and US service members be provided with cultural, civic, economic, military, and government awareness training and educated on the roles of mission partners and stakeholders in order to fully understand the operating environment.
The use of liaison officers is the most prominent staffing recommendation and is also mentioned as a positive lesson learned from the Ebola response efforts since it is closely tied to relationship building. Liaison officers reduce response time to requests for information and provide critical information on the field activities of US-based NGOs; they are an integral part of the communication and collaboration links between commands and organizations.
Discussion
Given the prominence of the Ebola outbreak and wealth of general Ebola-related literature available, relatively few sources met our final search criteria. The lack of literature examining the DoD's response is an important finding in and of itself. At the outset of conducting the research for this article, we hoped to be able to determine what types of studies had been conducted to monitor and evaluate the DoD's response to the Ebola pandemic. Upon categorizing the literature, it became evident that none of the records employed a method of analysis, and that this article is the first attempt to methodologically examine the lessons learned from the DoD's response at an aggregate level from multiple sources.
Since over half of the records categorized and coded for the article came from the JLLIS database, which is an observational database that derives its content from DoD's own after-action reports and the like, the lack of a methodological rigor is not unexpected with regard to these items. In contrast, it is notable that there is no extant peer-reviewed or grey literature that employs a methodological approach in discussing the DoD's response efforts. Accordingly, it is evident that as the body of available literature continues to grow, the effectiveness of these efforts and their resulting lessons learned and recommendations merit further and more rigorous examination. Although this article does not attempt to determine what the DoD's role should be in humanitarian assistance and disaster response, we hope that this study will spark additional research in this area.
The positive lessons learned surrounding coordination, partnership, and cooperation are a testament to one of the primary reasons why the DoD was charged with engaging in the Ebola outbreak response. The DoD is well known for its logistics capabilities and for being able to operate in austere environments, and it used these skill sets to accomplish tasks in West Africa, such as Ebola treatment units and Monrovia Medical Unit construction, that may not have otherwise been possible and likely played a critical role in controlling the outbreak.3,4 However, while coordination was a theme among positive lessons learned, weak coordination was the second most frequently coded negative lesson learned. One of the key findings from this research is the need to ensure better coordination at all levels of a pandemic response.
The DoD's ability to work with NGO partners and the Liberian military to accomplish many of these tasks, while enabling the Liberian government to have ownership of these efforts, should serve as a model for similar scenarios in the future. Taking into consideration the unique nature of OUA, negative lessons and recommendations concerning the themes of improved planning, augmented training, and development or revision of guidance and procedures in place for such a mission are surprising. A careful review of policy surrounding pandemic response based on the DoD's experience with Ebola will be a critical next step in ensuring that responders are adequately prepared to handle similar events in the future.
While this research has some insightful findings, there are some limitations to note. We were initially concerned with the lack of a robust peer-reviewed literature providing analysis and perspectives on Ebola outbreak response efforts undertaken by the DoD. With West Africa declared Ebola-free as of January 14, 2016, most literature currently available focuses on disease epidemiology or narratives about the spread and severity of the outbreak. 14 This limitation is due in part to the recent nature of the Ebola outbreak itself and the time lapse required to document lessons learned, conduct assessments, and subsequently publish the results. Moreover, collecting data on disaster events in real time is inherently challenging. During disasters, the top concern is to provide rapid and effective relief to affected populations, while navigating challenges such as environmental hazards, security of personnel and resources, political and economic issues, and cultural barriers. 15 This in turn has the potential to contribute to the overall dearth of scholarly literature available for review. For this reason, we opted to expand our search beyond peer-reviewed articles to include grey literature and the DoD's observations and after-action reports from JLLIS.
An additional limitation in terms of the overall count of literature actually used for the analysis pertains to the JLLIS records used, which are comprised of entries of individual JLLIS observations and documents such as after-action reports. Each JLLIS document contains a multitude of observations; however, based on the database structure, it was not possible for us to determine whether any of these coded observations were also included as lessons learned in the JLLIS documents that were reviewed and coded for the analysis.
Conclusion
The devastation caused by the Ebola epidemic in West Africa proved a sobering example of the very real threat posed by infectious diseases and the gaps in global preparedness and capacity to respond to such an event. Unlike natural disasters, infectious diseases are a threat to virtually everyone, everywhere, given their fluidity in a world where trade and travel are constantly increasing interconnectedness. Over the course of 8 months, the DoD played a major role in supporting Ebola response efforts by providing command and control, logistics, engineering, and training support.
During the Ebola pandemic, the DoD was considering its role and responsibilities in relation to the Global Health Security Agenda (GHSA). Led by the CDC and in partnership with other nations and public and private stakeholders, the GHSA seeks to accelerate progress toward a world safe and secure from infectious disease threats, to promote global health security as an international security priority, and to spur progress toward full implementation of major global health security frameworks, such as the 2005 WHO International Health Regulations (IHR).16,17 The use of the DoD's lessons learned from the Ebola epidemic response will be critical to the formulation of GHSA country roadmaps, as well as informing the DoD's response plans for similar threats in the future.
One such endeavor to use the lessons learned from the Ebola response was undertaken in 2015 via the West Africa Disaster Preparedness Initiative (WADPI), which was supported by AFRICOM's Disaster Preparedness Program. WADPI was a regional training program convened by the Economic Community of West African States (ECOWAS), in collaboration with the Kofi Annan Peacekeeping Training Centre and the National Disaster Management Organization of the Republic of Ghana. WADPI brought together representatives from the governments of ECOWAS member states for 3-week sessions focused on strengthening national capacities among West African partner nations and implementing an all-hazards approach to disaster preparedness and response management. These sessions included train-the-trainer activities that brought together US government and civilian personnel and facilitators from West African partner nations, fostering whole-of-government disaster preparedness strategies and plans and ensuring continued regional collaboration, communication, and coordination as well as continuity in planning between and among countries. 18
This article served as a means to provide DoD leaders with greater insight into strengths and weaknesses in terms of the DoD's response efforts that will be used to shape future policy and operations. Furthermore, this article highlighted the current gap in available peer-reviewed literature surrounding monitoring and evaluation of the DoD's response efforts. This research demonstrated the importance of meaningful after-action reports that clearly depict goals, level of effort, capabilities, and specific lessons learned. There may also be the need to understand and identify the process for knowledge integration among the myriad global health engagements stakeholders. The hope is that this research will stress the importance of lessons learned, create more interest in qualitative analysis, and encourage the incorporation of these findings into future humanitarian assistance and disaster response security cooperation engagements (such as WADPI) and the development of related capabilities for the DoD and other global health engagement stakeholders.
Footnotes
Acknowledgments
The views expressed are those of the authors and do not necessarily represent those of the Uniformed Services University or the Department of Defense. The authors have no potential conflicts of interest to declare.
