Grounding evaluation design in the socio-ecological model of health: a logic framework for the assessment of a national routine immunization communication initiative in Kyrgyzstan

Introduction

Routine immunization (RI) is a highly effective public health intervention to prevent infectious diseases, especially among children and adolescents. The role of RI in global health is unequivocal – RI against viral diseases has led to the eradication of smallpox and a decrease in the incidence of polio, measles, mumps, and rubella (1,2). Additionally, the vaccine against bacterial diseases such as Hemophilus influenzae type b (Hib), which causes bacterial meningitis in children under 4 years of age, has led to a dramatic decrease in Hib incidence in developed countries (3,4). Though rates of Hib vaccine coverage have been sub-optimal in developing countries, recent single-country studies show great progress on both rates of Hib immunization and prevention of infection (5). Moreover, the effect of RI is not just restricted to individual-level immunity but has effects on the entire community (also known as herd immunity), whereby high rates of vaccinations may offer protection to unvaccinated individuals due to extremely low incidence of vaccine-preventable diseases (6).

Despite the overwhelming evidence in support of RI effectiveness, many countries and communities are facing increased hesitancy or refusal on the part of caregivers to vaccinate their children (7,8), even when immunization services are available (8). As immunization interventions are essential components of broader health and social systems, parents and caregivers are often influenced by other stakeholders, like family members, clinicians, community outreach workers, policy-makers, media outlets, religious leaders, and other community leaders, in their decision to accept or refuse a vaccine for their children. Communicating about the efficacy and safety of immunization requires a comprehensive community- and system-driven approach that addresses misconceptions, manage hoaxes, and ultimately engages professionals and leaders across sectors as well as the community at large.

Lately, the Kyrgyz Republic (KR) in Central Asia has also been experiencing issues with vaccine hesitancy and refusal within pockets of its population. This is despite a strong immunization system, which has led to a decrease in the incidence of many vaccine preventable diseases (VPDs), including diphtheria, tetanus, whooping cough, and hepatitis. A recent Knowledge, Attitude, and Practice (KAP) study conducted by the United Nations Children’s Fund (UNICEF) revealed a practice of vaccine refusal in many pockets of the population and across different regions in Kyrgyzstan mainly because of misconceptions about the safety of vaccines (9). In addition, data from a 2018 Joint Appraisal Report revealed a decrease in 2017 from 96.1% to 92% in immunization rates among children under 1 year old for the pentavalent vaccine, which protects children again five life-threatening diseases – diphtheria, pertussis, tetanus, Hepatitis B, and Hib – with some cities such as Bishkek reaching rates as low as 88% (10). Compared with 2016, vaccine refusals increased 1.8 times in Bishkek, from 3 to 4.5 times in Naryn, Talass, and Chuya, and from 1.3 to 2 times in Batken, Jalal-Abad, and I-Kul. Increases in vaccine refusal were also observed in the city of Osh and the Osh region (10). Moreover, data from the measles outbreak in 2014–2015 and other surveys also indicate a decreasing trend in immunization coverage. According to the Multiple Indicator Country Survey (MICS), only 80.4% of children surveyed were fully immunized in 2014 (11). Finally, 4% of healthcare providers do not trust that all vaccines have been tested for quality and safety (9,11).

As for all health and social behaviors, immunization behavior (or lack thereof) reflects complex interactions between multiple levels of society (individual, interpersonal, community, organizational, and policymaking) and related groups and stakeholders. For instance, ‘global monitoring of the state of inequality in childhood immunization has demonstrated lower coverage in the poorest and least educated of many low- and middle-income countries’ (12). Establishing strong systems (e.g. information, health, and social support systems), and engaging key influentials in each of these intersecting systems is an important ‘prerequisite to ensuring that policies, programs and practices’ are tailored to – and effective in engaging – the most-disadvantaged groups, ultimately contributing to eliminate inequities in childhood immunization (11).

Moreover, key to communication design in the 21st century is a perspective that takes into account the many social, economic, and environmental determinants of health, promotes health and social equity, and considers the connection between different issues, stakeholders, and behavioral and social results. Socio-ecological theory is a well-established theoretical framework in global health, international development, and communication (13–16). For example, the socio-ecological model (SEM) is at the core of Communication for Development (C4D), UNICEF’s signature approach and planning framework for social and behavior change communication (17). Similarly, the integration of multi-level interventions to engage different groups and stakeholders is a key mantra of other long-standing frameworks for communication planning (17–19). Moreover, the SEM of health has been used successfully for the development of public health, international development, and communication interventions, both for chronic and infectious diseases, in the United States and internationally (20,21).

In order to promote RI in Kyrgyzstan, UNICEF has spearheaded a comprehensive communication initiative to engage all major stakeholders and address key factors that contribute to inequities in childhood immunization. Reflecting a system-driven and multi-level approach to intervention design, the UNICEF RI Communication Strategy and Action Plan 2018–2020 (10) in Kyrgyzstan is grounded in the SEM (15,16), which recognizes that ‘social and individual behavior change does not happen as a result of isolated interventions, but rather through the interaction of social, individual, and structural factors to produce an environment that supports behavior change’ (13).

The UNICEF Communication Initiative incorporates several stakeholder groups impacting RI: community members including parents, grandmothers, and other primary caregivers of young children, and religious and community leaders; the Kyrgyz media; healthcare providers; and policy-makers. The community engagement component aims to give voice to local champions, including community and religious leaders, and parents (vaccine accepters) who understand the importance of, and advocate for, RI; and to provide a platform for information dissemination among vaccine refuters. The media component of the initiative includes, among others, a workshop for key journalists and systematic media outreach, to improve the quality and accuracy of RI media coverage and to promote changes in beliefs, attitudes, and behaviors among journalists on this topic. Finally, the initiative includes an interpersonal communication (IPC) training module for healthcare providers and community influencers/community health workers to better equip them to educate parents and caretakers on immunization-related issues, and promote RI amongst hesitant caregivers. The model also includes a comprehensive outreach to policymakers to gain their support and engagement in the initiative.

Figure 1 depicts the different layers of the SEM and the spheres of influence associated with key stakeholders, as related to the UNICEF initiative. This paper discusses the logical framework that was developed to assess this initiative. Such framework builds upon previous work by one of the authors (22,23), and integrates elements of logic modelling with different types of indicators at all levels of the SEM. The framework seeks to contribute to the current debate on the evaluation of social and behavior change communication initiatives and other kinds of multi-level interventions, and can be applied to a variety of global health interventions. The framework can be implemented both in quantitative and qualitative research settings, and therefore can be tailored to resource-poor settings.

OPEN IN VIEWER

Figure 1.

The socio-ecological model of health and related key stakeholders for the Routine Immunization Communication Initiative by UNICEF-Kyrgyzstan.

Overview of the evaluation framework

The evaluation of social and behavior change communication (SBCC) and other multi-level interventions is an ever-evolving and highly debated topic. Several evaluation frameworks already exist (24,25), but, in many cases, evaluation tends to focus on single- or individual-level factors. This mirrors a critical limitation of many interventions in the intersecting fields of communication, child health, health equity, public health, and international development. While interdisciplinary and multisectoral interventions have shown promising results in many fields, evidence on such integrated approaches is still somewhat scarce, with many interventions and evaluation studies focusing on one single factor, approach, or type of media or activity (26,27).

Given the complexity of the many intersecting health and social issues of our times, ‘we must consider approaches for systematically designing and assessing the efficacy of multilevel interventions’ (26) especially when engaging vulnerable, marginalized, and/or low health literacy populations, such as those where inequalities in childhood immunization tend to be widespread.

Generally, the most widely accepted outcome of immunization interventions is a health-seeking behavior that results in the improvement of overall immunization rates. In other words, the outcome of immunization interventions should be an increase in the number of primary caregivers who comply with the recommended RI schedule, as well as behavioral and social results within groups and systems that have a stake in, and can affect, a decrease in the incidence of vaccine-preventable diseases. Yet, what happens between people’s exposure to a communication intervention and the adoption of immunization-seeking behaviors includes changes in awareness, knowledge, attitudes, participation, skills, self-efficacy, behavior readiness, and other intermediate steps toward behavioral results (27). Alongside the actual behavior, these intermediate indicators are all influenced by many factors and community-specific behaviors among professional and lay groups at different levels of the SEM. In order for behavioral outcomes to be sustainable, immunization interventions need to encourage long-term systems-level change, and promote the engagement of different segments of society.

The proposed evaluation framework reflects such a multilevel approach to communication design, recognizes the importance of the many social, economic, and environmental determinants of health that contribute to immunization behavior, and integrates the SEM with a logic model that connects different components of the communication intervention as related to the monitoring and evaluation of its results. Logic models are the gold standard in evaluation design as they provide a vision tool to link short- and long-term outcomes with the theory, strategies, and activities, and assumptions/contributing factors considered for program development (27–29).

The logic model used here (see Figure 2) was adapted from an earlier evaluation framework developed by the Office of Minority Health (OMH), United States Department of Health and Human Resources (HHS) for improving racial/ethnic minority health and eliminating health inequities (30). This framework was subsequently modified and implemented for the evaluation design of a national intervention on infant mortality prevention in the United States (23), as well as the evaluation of a national communication initiative to eliminate chronic malnutrition in Rwanda (22).

OPEN IN VIEWER

Figure 2.

The logic model framework was adapted from references (23) and (30), Chapters 2 and 14 in reference (27) and references (22) and (31). This figure illustrates the evaluation framework at all levels of the multi-stakeholder intervention, and also includes sample elements.

The framework includes different indicators (progress, outcome, and impact indicators) at all levels of the SEM, which are related to specific key groups and stakeholders. Complementing the model are process indicators, which are activity-specific and are listed in a separate table by intervention area and separated by activity, media, or materials. Table 1 includes a definition for each indicator.

OPEN IN VIEWER

Table 1.

Types of indicators.

○  Process indicators , which ‘compare key steps of the program’s implementation with the original program plan.’ These are activity-specific and ‘refer to parameters such as audience reach, event attendance, short-term message recall, quality and tone of media coverage, number of partnerships, material distribution numbers,’ and similar measures of process ○  Progress indicators , which ‘monitor the intervention’s impact vis-à-vis ‘intermediate steps’ toward behavioral and social change and may include ‘changes in awareness, knowledge, attitudes, skills, community participation levels,’ or other intermediate results ○  Outcome indicators , which measure behavioral, social, policy, and organizational results ○  Impact indicators , which measure the intervention’s impact on disease’s morbidity, mortality, prevalence, incidence, quality of life, and other parameters related to the severity and prominence of a specific health condition

Source: Schiavo R. Health Communication: From Theory to Practice. Second Edition. San Francisco: Jossey-Bass, an imprint of Wiley, 2014, Chapter 14, p.414–418. All rights reserved. Used by permission. (27).

As developed, the framework is intended to help UNICEF, its partners, and other stakeholders, to evaluate the impact of the national initiative in support of children’s right to immunization in Kyrgyzstan, and, ultimately, to contribute to the well-being of our global community by preventing severe or life-threatening childhood infectious diseases. Sample indicators and other information on the framework as applied to the evaluation design of this initiative are highlighted in the Discussion. Implications for the evaluation of other immunization, child health, global health, SBCC, health promotion, and international development interventions are discussed below in an attempt to contribute to the ongoing debate on the evaluation of multi-level interventions.

Discussion

The development of this framework lends itself to strengthening the knowledge base and confidence of different stakeholders (Figure 1) in RI while actively seeking behavioral results. As applied to the UNICEF RI communication initiative, the framework considers key factors that emerged from the KAP as contributing to vaccine hesitancy and refusal in Kyrgyzstan, and integrates strategic objectives of the initiative in developing key indicators at different levels of the SEM. The framework has been generated through analysis of the literature as well as a review of UNICEF’s strategic plan, intermediate evaluation results, media reports, and other documents on immunization issues in Kyrgyzstan.

The KAP study aimed to assess knowledge, attitudes, and practices of mothers and caregivers and practices of healthcare professionals about RI (9,11). Data collected from 2500+ respondents revealed that parental concerns and misconceptions about potential side effects are among the main reasons why children were not vaccinated, or not fully vaccinated (9). The study found that parents received their information about immunization via medical providers, who, in turn, received information from higher-level professional organizations, reiterating the complexity of interactions that influence immunization behavior in Kyrgyzstan.

Our framework builds upon a logic model developed by the Office of Minority Health, United States Department of Health and Human Resources and its subsequent modifications, and incorporates the SEM of health promotion as the conceptual underpinning of the framework (22,23,30). It acknowledges that individual behavior among primary caregivers is influenced by interpersonal, community, organizational, and policy settings. Interaction among key stakeholders in these settings is essential to create a web of social support in favor of immunization within all systems with which primary caregivers interact. Sample measurement criteria and indicators at different levels of the SEM, and specifically for each of the groups and stakeholders in Figure 1, are included in supplementary materials. Hopefully, as the framework is applied in Kyrgyzstan, further criteria will be developed to reflect country- and situation-specific issues.

Examples of progress and outcome indicators associated with each layer of the SEM are in Supplementary Table 2 (online only). While progress and outcome indicators were designed to assess changes at the individual, interpersonal, community, organizational and policy levels, it is assumed that improvements in all indicators will ultimately affect the individual level of the SEM by supporting behavioral results among primary caregivers. In other words, increasing the number of primary caregivers who vaccinate their children can be considered both an outcome indicator at the individual level and the main behavioral outcome of the communication initiative, and is linked to behavioral and social change at different levels of the SEM. Additionally, some of the indicators at the organizational level are not specific to a single organization; they also measure progress and outcomes of interactions across organizations and communities.

Another important component of this evaluation framework is the integration of formative research to consider process indicators, which are activity-specific, so that activities can be adjusted iteratively. Process indicators were deliberately not included in the described logic model, as these indicators are being considered as part of the activity-specific evaluation methodology versus other types of indicators that are likely to be achieved by the combination of multiple components (strategies and activities) of the initiative.

Supplementary Table 3 (online only) provides examples of process indicators as they pertain to the framework and specific activities for different groups. Though we have considered process indicators only for ongoing or upcoming activities, some of these indicators may apply to the measurement of similar future activities.

As for other indicators in the framework, process indicators will be assessed in the initial evaluation phase via qualitative methods (in-depth interviews and panel studies) and mixed methods tools (pre–post event questionnaires). Qualitative methodology is being considered by UNICEF and its partners to meet the goals of initial evaluation efforts, and to predict trends in behavioral change for both program refinement and progress reporting. Qualitative research is often the method of choice for evaluation in global health as it may double as a tool to give voice to unheard populations and to generate information that may increase stakeholder buy-in (32). As the initiative continues, many indicators will be also assessed quantitatively.

Finally, process indicators will be measured in 2019–2020 by using qualitative methods to assess the efficiency of communication activities and understand if expected changes are starting to occur. Progress indicators will be measured in 2021 or 2022 by qualitative and quantitative methods, including in-depth interviews, panel studies, each of which will comprise representatives of a specific key group, focus groups, media monitoring and analysis, and pre–post event qualitative and quantitative assessments. Research questions are being implemented in the field for all indicators to assess and refine clarity, cultural relevance and ease of implementation, and will be included as part of future reporting on the findings of this evaluation effort. Outcome indicators will be measured through the next Multiple Indicator Country Survey (MICS), which is grounded in UNICEF international household survey initiative for data collection and reporting (33). Similar experiences in developing countries in the field of maternal, newborn, infant, and child health, show that behavior results and increased demand for health services tend to be observed within a 3–5 year period from the launch of an integrated communication initiative, including multiple approaches and media (33). Observed changes are usually in the range of a 10–21% increase in the desired behavior, including demand for specific health services. This also depends on the intensity and continuity of all efforts, and other elements that may be intervention- and/or country-specific (34). Moreover, the framework is well suited for use in future evaluation phases to assess and compare results across different groups and regions and related variables (e.g. socio-economic and health literacy levels) and to document the potential association (or lack of thereof) of key findings with changes among key stakeholders and the systems in which they live, work, or operate, at different levels of the framework. This may also help monitor changes in inequities on immunization rates. Similarly, the framework can be adapted to measure health and social inequities as part of the evaluation of other global health, health promotion, and international development interventions.

Conclusion

While immunization coverage has improved throughout the world, immunization disparities still exist across different countries and populations. Though our discussion has revolved on a framework for the assessment of a Routine Immunization Communication Initiative in Kyrgyzstan, vaccine hesitancy and refusal are emerging and timely issues both in economically developing and developed countries. For example, in the U.S., vaccine hesitancy and refusal among parents and other primary caregivers are growing, and have resulted in an increase in children who reach age 2 without receiving any vaccinations from 0.9% for children born in 2011 to 1.3% for children born in 2015 (35). This trend may explain the recent higher rates of vaccine-preventable diseases, such as in the case of the 2019 measles outbreak in New York City (36).

Similar to other health inequities, disparities in immunization coverage and vaccine demand are related to complex factors that require the engagement and participation of multiple levels of society in promoting behavioral and social change, not only among parents and other primary caregivers, but across systems and stakeholders who ultimately influence parental decisions on whether to immunize or not (10). Immunization disparities also have important consequences for population health and well-being. Therefore, it is imperative that society as a whole contributes to protecting children’s right to immunization and survival.

As for other health and social inequities, systems-level change to address immunization disparities can be achieved only via multi-level interventions and cross-sectoral partnerships and engagement (37,38). Efforts to evaluate multi-level interventions must reflect the ‘complex interplay between these levels’ (26), and consider the SEM as an important framework to be integrated in assessments.

Although the evaluation framework discussed here refers to the assessment of the UNICEF RI Communication Initiative in Kyrgyzstan, the framework has broader applicability and transferability for other SBCC interventions to address immunization issues, and across the fields of global health and international development and related disciplines. More specifically, the framework can be applied to other population health interventions for both infectious (e.g., other vaccine-preventable diseases, HIV, etc.) and non-communicable diseases (e.g., diabetes and obesity), as these should aim to build social support for behavioral change among vulnerable and at-risk groups and ultimately promote sustainable results across groups at different levels of the framework. Another strength of the framework is its ease of use and adaptability to international development settings as the framework prompts potential users to take into account the specificity of the local and political contexts of any intervention. The framework also aligns with key principles from the Ottawa Charter for Health Promotion (39) and its emphasis on multi-level interventions and system-thinking, as well as creating supportive environments.

Finally, the framework has implications for future research and practice as it seeks to contribute to the ongoing debate and explorations on setting guidelines and building a body of evidence on methods to assess multi-level interventions that aim to advance immunization equity, address complex issues like vaccine hesitancy, or eliminate other kinds of health disparities, and to systematically integrate intervention and evaluation design.

Supplemental Material