Conducting the Joint External Evaluation in Uganda: The Process and Lessons Learned

Abstract

Uganda is currently implementing the Global Health Security Agenda (GHSA), aiming at accelerating compliance to the International Health Regulations (IHR) (2005). To assess progress toward compliance, a Joint External Evaluation (JEE) was conducted by the World Health Organization (WHO). Based on this evaluation, we present the process and lessons learned. Uganda's methodological approach to the JEE followed the WHO recommendations, including conducting a whole-of-government in-country self-assessment prior to the final assessment, using the same tool at both assessments, and generating consensus scores during the final assessment. The in-country self-assessment process began on March 24, 2017, with a multisectoral representation of 203 subject matter experts from 81 institutions. The final assessment was conducted between June 26 and 30, 2017, by 15 external evaluators. Discrepancies between the in-country and final scores occurred in 27 of 50 indicators. Prioritized gaps from the JEE formed the basis of the National Action Plan for Health Security. We learned 4 major lessons from this process: subject matter experts should be adequately oriented on the scoring requirements of the JEE tool; whole-of-government representation should be ensured during the entire JEE process; equitable multisectoral implementation of IHR activities must be ensured; and over-reliance on external support is a threat to sustainability of GHSA gains.

To assess progress toward compliance with IHR (2005), a Joint External Evaluation (JEE) was conducted by WHO. Uganda's methodological approach to the JEE followed the WHO recommendations, including conducting a whole-of-government in-country self-assessment prior to the final assessment, using the same tool at both assessments, and generating consensus scores during the final assessment.

The World Health Assembly adopted the revised International Health Regulations (IHR) in 2005; they subsequently came into force in June 2007.^1,2 All 196 World Health Organization (WHO) member states committed to achieving compliance with IHR (2005) by 2012 by developing specified minimum core public health capacities.^3,4 The purpose and scope of the IHR (2005) are “to prevent, protect against, control and provide a public health response to the international spread of disease in ways that are commensurate with and restricted to public health risks, and which avoid unnecessary interference with international traffic and trade.”⁵ Even though IHR regulations had existed since 1964,⁶ the evolving landscape of public health emergencies worldwide made this revision inevitable.⁷ Uganda had earlier been implementing IHR through the Integrated Diseases Surveillance and Response (IDSR) framework since 2000.⁸ Just like most member states, however, Uganda's progress was slow, and the country failed to achieve compliance by 2012.^9,10 Because of this, Uganda subscribed to the Global Health Security Agenda (GHSA), whose ultimate goal is to accelerate compliance with IHR (2005).^11–13

The Uganda Ministry of Health conducted a GHSA demonstration pilot project in 2013 to showcase prevention, detection, and response capabilities resulting from targeted investments in enhanced laboratory systems, real-time surveillance, reporting, and the establishment of a public health emergency operations center (PHEOC). The results of the demonstration project provided critical lessons on the feasibility of global health security, thus laying the foundation for the 2014 GHSA launch.¹⁴ Later, in 2015, Uganda volunteered for a GHSA external assessment, whose results generated baseline scores for the country's GHSA implementation.¹⁵

In December 2016, Uganda voluntarily asked WHO for a Joint External Evaluation (JEE) assessment. Apart from assessing progress in IHR (2005) compliance, the country wished to benefit from the opportunities that may arise from the JEE exercise in evaluating public health emergency prevention, detection, and response capabilities. Uganda is prone to public health emergencies, arising from her proximity to the Congo basin, uncontrolled cross-border movements, refugee influx, and pressure on the ecosystem, among other vulnerabilities. Results of the JEE assessment are available in the public domain.¹⁶ Here, we describe the process and present the lessons learned from this exercise.

Methods

JEE Methodology

Uganda followed the methodological approach to the JEE process as documented by WHO.¹⁷ Here, countries voluntarily and officially request WHO for a JEE assessment. The entire process is designed to assess a country's capacity to prevent, detect, and rapidly respond to public health threats, whether they are occurring naturally, deliberately, or accidentally. The process is characterized by a multisectoral, One Health approach, coupled with transparency and openness of scores, including public release of the findings.

Prior to the actual external assessment, multisectoral teams in the country's key sectors assemble to perform an internal, whole-of-government self-evaluation. These teams include actors from human health, animal health, environment, wildlife, agriculture, security, law enforcement, national defense, border control, finance, communication, disaster management, transportation, customs, civil aviation, academia, research institutions, private sector, parastatals, and community leadership, among others.^18,19 The self-assessment exercise follows similar procedures as the actual external assessment, including use of the same tool.

The JEE tool is divided into 4 broad core capacities: prevent, detect, respond, and other IHR indicators. Each core capacity is further divided into 1 or more technical areas, with specific indicator(s) linked to different score levels.¹⁸ Countries are assessed on each indicator using a scale of 1 (no capacity) to 5 (sustained capacity). Color coding is also used to represent scores, with score 1 being color coded as red, scores 2-3 color coded as yellow, and scores 4-5 as green.¹⁸

Results

Uganda's JEE approach

Uganda formally requested a JEE assessment in December 2016. The process was spearheaded by the office of the prime minister and the ministry of health's public health emergency operations center as the secretariat. The in-country self-evaluation process began on March 24, 2017, with a multisectoral national stakeholders meeting at the Office of the President Conference Center. A total of 203 subject matter experts from 81 institutions participated in the process. Subject matter experts were identified and invited from 16 government ministries, 7 government departments, 7 bodies affiliated with the United Nations, 36 private institutions, 5 academic institutions, and 10 stakeholders' institutions. They were introduced to the objectives of IHR (2005), GHSA, and the JEE tool.

This was followed by training of facilitators, who then took the lead in engaging the subject matter experts through the in-country self-evaluation process. A group of 10 to 15 experts per technical area held separate meetings between April 18 and 27, 2017. The product of these meetings formed the first draft of the in-country self-assessment report, which addressed the technical and contextual questions of the tool, self-scores per indicator, identified gaps, and supporting documentation available to justify the self-score.

The second national stakeholders meeting was held on May 18, 2017, composed of the entire multisectoral team of subject matter experts, under the coordination of the office of the prime minister. Here, different presentations were made for each technical area, and further refinements were generated. The product of this meeting constituted the second draft of the in-country self-assessment report, which was finally compiled and edited by the public health emergency operations center and submitted to the office of the prime minister for approval before being submitted to WHO.

The external JEE by WHO took place between June 26 and 30, 2017, under the leadership of the office of the prime minister. A team of 15 external experts from WHO took the lead in the exercise, in which consensual scores per indicator were reached through open and transparent discussions. Throughout the discussions, there were no significant or irreconcilable disagreements between the external team and the Uganda subject matter experts. In total, 139 delegates from various institutions and sectors attended the final evaluation, and 217 documents were referenced, including policies, guidelines, publications, case studies, and relevant reports. Site visits were also made to health facilities, the atomic energy commission, national laboratories, the public health emergency operations center, points of entry, and isolation centers as part of the evaluation exercise.

Recommendations from this assessment formed the basis of Uganda's National Action Plan for Health Security (NAPHS, 2018-2023). Although implementation activities for the NAPHS commenced, the plan has yet to be officially launched by the Uganda government and thus it has not yet been published in the public domain.

Highlights of the JEE

Results of the Uganda JEE assessment are presented in detail elsewhere.¹⁶ Table 1 presents a comparison between the in-country and the final scores. The 2 finance indicators (P1.3 and P1.4) were not assessed during the in-country self-assessment, since they were not part of the JEE tool 1.0.²⁰ However, Uganda volunteered to pilot their use, and they would later be included in the JEE tool 2.0.²¹ We noticed discrepancies in scores in the in-country and final consensual scores on 27 of the 50 indicators on the JEE tool. These discrepancies were mostly attributed to failure to provide convincing documentation to the external team. Other explanations include presence of advanced public health emergency detections structures in human health, with less developed similar structures in animal and environmental health, and absence of a few but key subject matter experts during the in-country self-assessment process.

Table 1.

Comparison of Uganda's Joint External Evaluation scores at self-assessment and final assessment

Technical Area	Indicator	Consensual Scores
Technical Area	Indicator	In-country Self-Assessment	Final External Assessment
National legislation, policy, and financing	P.1.1 Legislation, laws, regulations, administrative requirements, policies, or other government instruments in place are sufficient for implementation of IHR (2005).	3	3
	P.1.2 The state can demonstrate that it has adjusted and aligned its domestic legislation, policies, and administrative arrangements to enable compliance with IHR (2005).	4	3
	P.1.3 Financing is available for the implementation of IHR capacities.	—	2
	P.1.4 A financing mechanism and funds are available for the timely response to public health emergencies.	—	1
IHR coordination, communication, and advocacy	P.2.1 A functional mechanism is established for the coordination and integration of relevant sectors in the implementation of IHR.	3	2
Antimicrobial resistance	P.3.1 Antimicrobial resistance detection	2	2
	P.3.2 Surveillance of infections caused by antimicrobial-resistant pathogens	2	2
	P.3.3 Health care–associated infection (HCAI) prevention and control programs	2	3
	P.3.4 Antimicrobial stewardship activities	2	3
Zoonotic diseases	P.4.1 Surveillance systems in place for priority zoonotic diseases/pathogens	3	2
	P.4.2 Veterinary or animal health workforce	3	3
	P.4.3 Mechanisms for responding to infectious and potential zoonotic diseases are established and functional	3	2
Food safety	P.5.1 Mechanisms for multisectoral collaboration are established to ensure rapid response to food safety emergencies and outbreaks of foodborne diseases.	3	2
Biosafety and biosecurity	P.6.1 Whole-of-government biosafety and biosecurity system is in place for human, animal, and agriculture facilities.	3	3
Biosafety and biosecurity	P.6.2 Biosafety and biosecurity training and practices	3	3
Immunization	P.7.1 Vaccine coverage (measles) as part of national programme	3	3
Immunization	P.7.2 National vaccine access and delivery	3	4
National laboratory system	D.1.1 Laboratory testing for detection of priority diseases	4	4
	D.1.2 Specimen referral and transport system	3	3
	D.1.3 Effective modern point-of-care and laboratory-based diagnostics	3	3
	D.1.4 Laboratory quality system	3	3
Real-time surveillance	D.2.1 Indicator and event-based surveillance systems	4	4
	D.2.2 Interoperable, interconnected, electronic real-time reporting system	3	3
	D.2.3 Analysis of surveillance data	4	3
	D.2.4 Syndromic surveillance systems	4	3
Reporting	D.3.1 System for efficient reporting to WHO, FAO, and OIE	5	3
Reporting	D.3.2 Reporting network and protocols in country	5	3
Workforce development	D.4.1 Human resources are available to implement IHR core capacity requirements	3	3
	D.4.2 Applied epidemiology training program in place such as FETP	4	4
	D.4.3 Workforce strategy	4	3
Preparedness	R.1.1 National multi-hazard public health emergency preparedness and response plan is developed and implemented.	4	1
Preparedness	R.1.2 Priority public health risks and resources are mapped and utilized.	3	1
Emergency response operations	R.2.1 Capacity to activate emergency operations	4	4
	R.2.2 EOC operating procedures and plans	4	4
	R.2.3 Emergency operations program	4	4
	R.2.4 Case management procedures implemented for IHR relevant hazards.	3	3
Linking public health and security authorities	R.3.1 Public health and security authorities (eg, law enforcement, border control, customs) are linked during a suspect or confirmed biological event.	2	2
Medical countermeasures and personnel deployment	R.4.1 System in place for sending and receiving medical countermeasures during a public health emergency	3	2
Medical countermeasures and personnel deployment	R.4.2 System in place for sending and receiving health personnel during a public health emergency	4	2
Risk communication	R.5.1 Risk communication systems (plans, mechanisms, etc)	1	2
	R.5.2 Internal and partner communication and coordination	5	4
	R.5.3 Public communication	5	4
	R.5.4 Communication engagement with affected communities	3	4
	R.5.5 Dynamic listening and rumor management	3	3
Points of entry	PoE.1 Routine capacities established at points of entry	1	1
Points of entry	PoE.2 Effective public health response at points of entry	3	1
Chemical events	CE.1 Mechanisms established and functioning for detecting and responding to chemical events or emergencies.	2	2
Chemical events	CE.2 Enabling environment in place for management of chemical events.	3	2
Radiation emergencies	RE.1 Mechanisms established and functioning for detecting and responding to radiological and nuclear emergencies.	3	2
Radiation emergencies	RE.2 Enabling environment in place for management of radiation emergencies.	3	2

Note. Joint External Evaluation tool scores: 1 = no capacity, 2 = limited capacity, 3 = developed capacity, 4 = demonstrated capacity, 5 = sustained capacity

Discussion

Uganda followed the approach recommended by WHO to conduct the JEE. Of the 50 indicators assessed using the JEE tool, Uganda scored sustained on no indicators, demonstrated on 10 indicators, developed on 20, limited on 15, and no capacity on 5 indicators.¹⁶ Discrepancies among scores at the self-assessment and final assessment happened on 27 of 50 indicators. Gaps were identified and prioritized in the NAPHS (2018-2023), following the JEE recommendations.

Lessons Learned

We learned 4 major lessons from this exercise. First is the need to ensure whole-of-government representation during the JEE process, including multisectoral and multicultural subject matter expert involvement. Whereas the secretariat (PHEOC) did their best to involve all key actors in health security, a few were either unintentionally omitted or failed to honor the invitation to participate. Most of these were actors from the private and/or public sectors whose line of duty is not concerned directly with public health, including the ministry of public service, local governments, customs, communications and transportation sectors, the private sector foundation, academia, and private veterinary practice, among others. Among the technical areas most affected by lack of appropriate subject matter experts were food safety, antimicrobial resistance, and points of entry. This may further explain the insufficient level of evidence and documentation that were available to the external team—for instance, in the workforce development technical area.

Second is the need for equitable sector involvement while implementing GHSA-related activities. Several discrepancies in scores between the in-country self-assessment and the final scores were due to the presence of advanced public health emergency prevention, detection, and response structures in human health, but less developed similar structures in animal and environmental health. For instance, whereas functionality of human health surveillance systems were found to be well developed by the external team, similar systems in the animal and environmental sectors were less developed in content and context (eg, they were mostly paper-based, did not cover all districts and/or communities, had yet to achieve the desired 80% reporting rates consistently, etc). Only 1 national referral laboratory existed for animal health compared with a hub of 100 referral public health laboratories. There was an absence of sufficient documented capacity to detect neglected zoonotic diseases and monitor antimicrobial resistance patterns in veterinary practice. Whenever subject matter experts based their scores on only 1 sector (eg, human health), the external team scored them downward, based on the principle of “a country being as good as their weakest sector.”

The third lesson is the need to sufficiently orient and train subject matter experts on the scoring requirements of the JEE tool. Although not measured by this study, some of the discrepancies noticed in the scores at self-assessment and final assessment might have been a result of subject matter experts misunderstanding the JEE tool scoring requirements. This was probably more pronounced in both indicators of the preparedness technical area, where subject matter experts scored 4 (demonstrated capacity) and 3 (developed capacity), respectively, yet consensual final scores were 1 (no capacity) for both indicators. Group discussions with the external team revealed a misconception of both indicators by the subject matter experts. Specifically, the National Multi-Hazard Preparedness and Response Plan presented to the external team was assessed to be more of a strategic preparedness document that lacked the appropriate tactical and multisectoral representation requirements of the indicators. Whereas Uganda had prioritized public risks by district, the resource mapping for emergency response according to the hazard profiles requirement of the indicator was yet to be undertaken. Additionally, the lowering of scores in the real-time surveillance technical area between in-country and external assessments resulted after subject matter experts failed to provide documentation for proof of experts and decision makers using surveillance data for policy decisions, and syndromic surveillance systems detecting 3 or more core syndromes indicative of public health emergencies.

Lastly, we learned that among the greatest threats to the sustainability of IHR (2005) activities was heavy reliance on external support. In the financing technical area, public financing for preventing, detecting, and responding to public health emergencies in Uganda was assessed as either unavailable or inaccessible. This may further explain Uganda's failure to achieve any sustained capacity score among all 50 indicators, especially the technical areas of immunization, workforce development, national laboratory systems, real-time surveillance, and emergency operations, where the score came close to 5 (Table 1). In spite of the successes registered, activities in these indicators were assessed to be highly donor-supported and thus would not sufficiently be sustained should such support pull out. However, this situation is not unique to Uganda and is equally present in most low-resourced countries.²²

Conclusions

Uganda's approach to the JEE in 2017 followed guidelines provided by the WHO. The country's capability scores to prevent, detect, and respond to public health emergencies were assessed as being fairly well developed. The assessment also identified several gaps, which were later prioritized in the NAPHS (2018-2023). The lessons learned from the process were: include all relevant actors from the public and private sectors; implementation of GHSA activities should be equitable in all relevant sectors; sufficiently orient subject matter experts to the JEE tool indicators; and that among the greatest threats to sustainability of GHSA gains is heavy reliance on donor support.

Footnotes

Acknowledgments

The authors wish to acknowledge the valuable contribution of the following subject matter experts during the process of developing the in-country self-assessment report: Alex Gisagara, Alex Tumusiime, Anne Nakinsige, Benon Kyokwijuka, Bildard Baguma, Charles Isabirye, Chrisostom Ayebazibwe, Christine Korsah, Deo Ndumu, Derrick Mimbe, Eddy Mukooyo, Edith Nantongo, Edson Katushabe, Emmanuel Othieno, Florence Adongo, Fred Monje, Fred Nghania, Gerald Menhya, Goeffery Kabagambe, Henry Kajumbula, Immaculate Ampaire, John Kissa, Joseph Mbihaye, Joseph Ojwang, Julian Kyomuhangi, Juliet Kyokuhaire, Julius Lutwama, Lillian Idrakua, Mary Akumu, Miriam Lwanga, Miriam Nanyunja, Mohammed Lamorde, Moses Bagyendera, Musa Sekamatte, Nathan Natserie, Nicholas Kauta, Noeline Nantima, Olivia Namusisi, Paul Kagwa, Paul Okware, Peter Obubu, Peter Okwero, Raymond Kirungi, Rhoda Nauda, Robert Downing, Ronald Jaggwe, Ronald Ssegawa, Rose Ademun, Samuel Okurut, Sean Blasche, Seru Morris, Simon Etimu, Sowedi Muyingo, Steven Aisu, Steven Balinandi, Susan Odongo, Vivian Sserwanja Nakaliika, William Lali, and Winyi Kaboyo. We are equally grateful to the team of 15 external experts from WHO whose patience, hard work, and challenging recommendations greatly enriched the JEE exercise, as well as informing the development of the NAPHS.

Composition and review of this manuscript were not funded by any entity. The Office of the Prime Minister of Uganda coordinated the entire Joint External Evaluation (JEE) exercise. Financial assistance to organize and facilitate the various subject matter experts' meetings was received from the World Health Organization, the United States Agency for International Development, and the Centers for Disease Control and Prevention. Representatives of these funding bodies were present throughout the JEE exercise process and the subsequent development of the National Action Plan for Health Security.

References

World Health Organisation. World Health Assembly WHA58.3: Revision of the International Health Regulations. http://www.who.int/ipcs/publications/wha/ihr_resolution.pdf. Accessed May 7, 2019.

Fidler

. From international sanitary conventions to global health security: the new International Health Regulations. Chinese Journal of International Law, 2005; 4(2):325-392.

Rodier

, Greenspan

, Hughes

, Heymann

. Global public health security. Emerg Infect Dis, 2007; 13(10):1447-1452.

Baker

, Forsyth

. The new International Health Regulations: a revolutionary change in global health security. N Z Med J, 2007; 120(1267):U2872.

World Health Organization. International Health Regulations (2005). https://www.who.int/topics/international_health_regulations/en/. Accessed May 8, 2019.

World Health Organization. International Health Regulations (1969). https://www.who.int/ihr/current/en/. Accessed May 8, 2019.

Andrus

, Vicari

, Tambini

, Periago

. The global inter-relatedness of disease control. Lancet Infect Dis, 2007; 7(3):176.

Lukwago

, Nanyunja

, Ndayimirije

, et al. The implementation of Integrated Disease Surveillance and Response in Uganda: a review of progress and challenges between 2001 and 2007. Health Policy Plan, 2013; 28(1):30-40.

Gostin

, Katz

. The International Health Regulations: the governing framework for global health security. Milbank Q, 2016; 94(2):264-313.

10.

World Health Organization. Implementation of the International Health Regulations (2005): Report of the Review Committee on the Role of the International Health Regulations (2005) in the Ebola Outbreak and Response. May 13, 2016. http://apps.who.int/gb/ebwha/pdf_files/WHA69/A69_21-en.pdf. Accessed May 8, 2019.

11.

Katz

, Sorrell

, Kornblet

, Fischer

. Global Health Security Agenda and the International Health Regulations: moving forward. Biosecur Bioterror, 2014; 12(5):231-238.

12.

Inglesby

, Fischer

. Moving ahead on the Global Health Security Agenda. Biosecur Bioterror, 2014; 12(2):63-65.

13.

Global Health Security Agenda. https://www.ghsagenda.org/about. Accessed May 8, 2019.

14.

Borchert

, Tappero

, Downing

, et al. Rapidly building global health security capacity—Uganda demonstration project, 2013. MMWR Morb Mortal Wkly Rep, 2014; 63(4):69-72.

15.

Global Health Security Agenda. Global Health Security Agenda Pilot Assessment of Uganda. February 16-20, 2015. https://www.ghsagenda.org/docs/default-source/jee-reports/uganda-jee-report.pdf. Accessed May 8, 2019.

16.

World Health Organization. Joint External Evaluation of IHR Core Capacities of the Republic of Uganda. Mission Report: June 26-30, 2017. http://apps.who.int/iris/bitstream/handle/10665/259164/WHO-WHE-CPI-REP-2017.49-eng.pdf?sequence=1. Accessed May 8, 2019.

17.

World Health Organization. Development, monitoring and evaluation of functional core capacity for implementing the International Health Regulations (2005). http://www.who.int/ihr/publications/concept_note_201407.pdf. Accessed May 8, 2019.

18.

World Health Organization. International Health Regulations (2005): Country Implementation Guide. Voluntary Joint External Evaluation. 2017. http://apps.who.int/iris/bitstream/handle/10665/259605/WHO-WHE-CPI-2017.62-eng.pdf?sequence=1. Accessed May 8, 2019.

19.

Kandel

, Sreedharan

, Chungong

, et al. Joint external evaluation process: bringing multiple sectors together for global health security. Lancet Glob Health, 2017; 5(9):e857-e858.

20.

World Health Organisation. Joint External Evaluation Tool: International Health Regulations (2005). 2016. http://apps.who.int/iris/handle/10665/204368. Accessed May 8, 2019.

21.

World Health Organization. Technical Framework in Support to IHR (2005) Monitoring and Evaluation. Joint External Evaluation Tool, Second Edition. January 2018. https://extranet.who.int/sph/sites/default/files/document-library/document/9789241550222-eng.pdf. Accessed May 8, 2019.

22.

Kruk

. Emergency preparedness and public health systems lessons for developing countries. Am J Prev Med, 2008; 34(6):529-534.