Fidelity and Adaptation of Programs: Does Adaptation Thwart Effectiveness?

Abstract

Evidence-based programs and grassroots programs are often adapted during implementation. Adaptations are often hidden, ignored, or punished. Although some adaptations stem from lack of organizational capacity, evaluators report other adaptations happen in good faith or are efforts to better fit the local context. Program implementers, facilitators who need to adapt during implementation, do not always report adaptations because they fear losing funding if the program is not implemented with fidelity. Program personnel including program evaluators need this information to improve effectiveness of programs, and to determine whether an adaptation is still consistent with the theory of change. Evaluators also need this information for generalizing results to varied settings and populations. Following the PRECEDE–PROCEED model, we recommend a hybrid approach to fidelity and adaptation. We argue in favor of advance planning to accommodate potential adaptations. Such planning also establishes evaluation criteria for determining whether adaptations are helpful, harmful, and appropriate to the context. We illustrate some types of adaptations that can occur, why they may be needed, and how to structure transparent reporting about adaptations to program developers and funding organizations.

Keywords

programming funding fidelity adaptation evaluation

Introduction: The Need to Evaluate Adaptations

One of the key resources for any program is the individuals needed to carry out the program (McKenzie et al., 2017, p. 264). Planners can function as a team or as individuals responsible for overseeing the implementation and evaluation. Practitioners refer to specialists or program facilitators who implement the program at the site or across sites, and evaluators refer to individuals who are internal to an organization implementing a program or hired from an external vendor whose job is to evaluate the program during the planning phase and through to the end of implementation to ensure the program is implemented as planned and/or to track any changes that need to be made. Funders refer to individuals or organizations who are funding the implementation of a program and developers refer to the group or individuals that designed/developed/wrote/published the program. Generally, grassroots programs are developed locally by community members or smaller community organizations and are not necessarily pilot-tested for effectiveness. Evidence-based programs refer to programs that are developed by larger entities with more resources (funding and personnel) like corporate or organizational entities that rigorously pilot test programs developing large datasets that prove effectiveness over time with fidelity to the core components of the program.

Any changes needed to be made to a program that make it more suitable to a population is called adaptation (Connecticut [CT] Department of Health & Human Services, 2022). Fidelity refers to a program being implemented exactly as it was developed and intended to be implemented without adaptation. Program personnel can face a dilemma that could impact the effectiveness of the program. After thorough analysis and discussion and conducting a community needs assessment, with other stakeholders’ input (any person or organization with a vested interest in the program success), the selected program may not fit the population and setting well. This dilemma persists across all phases including program selection, planning, implementation, and evaluation. Sometimes what happens at the moment during implementation will require unplanned adaptation to some of the program components (discussed later in the article). In this article, we focus on the planning, implementation, and evaluation phases of programming in a single setting and on the immediate needs of the program team: planners, practitioners, and evaluators. We will briefly offer suggestions for multi-site evaluation. To mitigate impromptu or unplanned adaptations, the program team needs to employ mindful planning to predict whether the program will work in their context and with the target audience. Evaluators can be invaluable early in the process of selecting, implementing, adapting where necessary, and assessing effectiveness. Evaluating early and often can reduce the potential for impromptu adaptation without reporting which compromises fidelity to program components as designed.

Evidence-based programs have been evaluated, by definition, with their effectiveness being demonstrated in one or more settings and time points. Grassroots programs are not as rigorously tested in settings, however over time effectiveness of grassroot programs can be established through evaluation during each implementation. Evaluators have long expressed concern that adaptation will water down implementation (Leviton & Schuh, 1991; Scheirer, 1981). The practitioner's dilemma mentioned above receives less attention in evaluation literature: the need for reasonable adaptation of the program to better serve diverse participants and communities, and to address setting constraints. The central issue becomes how evaluators distinguish a “reasonable and transparent adaptation” from “adaption that is not reported nor tracked.” If practitioners were asked, they could often give cogent reasons to adapt, including low participant readiness or education, facilitator expertise, or language and cultural barriers. Other barriers have to do with the context, such as limited time or resources to implement the program. A setting could be as innocuous as a mainstream classroom or fraught with potential issues, such as a prison classroom. Does it matter that program sessions were combined to fit the class schedule (a change in dose)? Does it matter that the prison went on lockdown, or the kids had to learn virtually (setting changes)? Changes made to a program that are not documented and reported can thwart the effectiveness of the program.

Examples abound of the need to plan for unexpected adaptation. The practitioner may find that the target audience's literacy level is lower than anticipated. Or the program was not developed for people of color. Therefore, the material may be less relevant to the target audience due to language barriers, different social norms, or life experiences. The practitioner may be of a different race or ethnic background from the target audience and community. These issues can limit reach, also known as coverage (the number or proportion of the target audience served) (Glasgow, et al., 1999; Rossi et al., 2018;). Creating rapport among the program team and stakeholders engenders stronger trust. Further, when practitioners were trained in the program, or when they purchased the manual of operations, the developers may have been silent about adaptation. Given these challenges, how might fidelity be maintained while applying adaptations?

These are issues of external validity: under what conditions (context, target audience) and with what program variations (adaptations) can planners, practitioners, and evaluators achieve the results they desire (effectiveness)? Cronbach and Shapiro (1982) warned evaluators about this challenge 40 years ago; indeed, they saw it as central to evaluation practice. They pointed to two perspectives on external validity: the immediate needs of the planner and practitioner to identify programs that would be effective for their target audience and context; and the needs of the developer and funders to know whether a program is relevant across contexts. They also pointed out that there can be no such thing as exact replication of programs in the social realm. There are simply too many important variations in settings, populations, and the passage of time from the original program test to the current implementation. Moreover, Rogers (2003) points out that when people adopt innovations, some degree of adaptation is almost inevitable. For example, the educator's dictum has long been “assess and adapt.” Evaluators and practitioners cannot rely on a simplistic faith that implementing a program “by the book” will achieve desired results (Cook, 2014). To make this explicit, recent writings on external validity have identified appropriate adaptation as a central component of explicating external validity, along with context, target audience, and measurement (Leviton & Trujillo, 2017; Shadish et al., 2002).

In what follows, we describe program core components, the theory of change, and the Predisposing, Reinforcing, Enabling Constructs in Ecological Diagnosis and Evaluation (PRECEDE)—Policy, Regulatory, and Organizational Constructs in Educational and Environmental Development (PROCEED) or PRECEDE–PROCEED, a model from health behavior that thoughtfully guides planning for adaptation (Green & Kreuter, 2005; see description below). These concepts are important to planning, pre-implementation (transition to next phase), implementation, and evaluation. We then introduce examples from the first author's many lived experiences in adapting an evidence-based program (as opposed to a grassroots program) to the real world. Further, we discuss the evaluator's role during planning, implementation, and outcome phases. We emphasize a hybrid approach to reconciling fidelity and adaptation during this process (Backer, 2001). A hybrid approach (including evaluation) combines transparency in reporting adaptations with assessing whether the core of the program retains fidelity.

Core Components, Underlying Theory, and PRECEDE–PROCEED

While planners and practitioners may be familiar with these concepts, it is part of the evaluator's role to identify the program's core components, apply theory, and assess reasonable adaptation to context, for which the PRECEDE–PROCEED model will be helpful. This section provides an overview of these concepts. Detailed examples are provided later.

Core Components and Underlying Theory

Fidelity requires adherence to the core components of interventions. The core components of a program would be considered an area of the program not to be adapted as they are the parts of the program that are directly related to a program's theory of change, and further described as “essential functions or principles, associated elements, and intervention activities” (Blase & Fixsen, 2013). Fidelity to these components of the program is preferred, but not always possible. To help identify core components, the Federal Family and Youth Services Bureau (2018) provides a general description at the most abstract level for programs (see Table 1). Ideally, the program developer would provide technical assistance to implementers to assure that they fully understand the core components. When presented as principles or mechanisms of change, rather than a recipe, the core components permit thoughtful adaptation. Often, however, developers are silent on how variations in population and setting need to be addressed to adhere to the core components.

Table 1.

A General Description of Core Components.

	Core Component	Description
What	Content	Adherence to program components the way it was written
How	Teaching Strategies	Pedagogy—how the content is taught; what is the interactive process of the delivery and how well is the facilitator delivering (i.e., enthusiasm, preparedness, skilled)
Logistics	Implementation	Program setting—frequency and duration of sessions, participant responsiveness, materials

Source: Department of Health and Human Services, Family & Youth Services Bureau (2018)

If adaptations are needed, evaluators need to monitor them to assess whether they are consistent with the theory underlying program development and whether core components are nevertheless implemented with fidelity. Program theory for many programs will be an application of one or more general behavioral or social theories, such as Bandura's (1986) Social Cognitive Theory, or the Social Ecological Model (Centers for Disease Control and Prevention, 2022b). However, the program itself has a more specific theory of change (TOC) that includes additional core components that are necessary to achieve intermediate program objectives, such as maintaining student attention and connecting with student life experiences. Ideally, these intermediate program objectives will have been identified by the developer. If not, they might be inferred by stakeholders together with the practitioner and evaluator. Evaluation of the progress of meeting these objectives progressively informs impact of the program. If adaptations are needed, evaluators can then make recommendations to improve meeting program objectives and impact. Disseminating careful formative evaluation of adaptations can also inform similar settings and populations for implementation of the same program (see Discussion).

The program's TOC is a guide to three general kinds of adaptation (and measurement for evaluation). (1) Cuts: the TOC helps to identify those activities that cannot be omitted or short-changed (dose), but also those that can be cut in the interest of program cost-effectiveness and spread (Collins, 2016). (2) Appropriateness: the program's prescribed activities are changed to better fit the population and setting, overcoming barriers to achieving intermediate program objectives. (3) Additions: the prescribed activities may be necessary, but not sufficient, to achieve intermediate program objectives. If so, then additional activities may be needed for recruitment, engagement, and other purposes. Alternatively, additions may simply waste time and money, or detract from what is important. Other typologies may also be helpful to the reader (Stirman et al., 2019).

PRECEDE–PROCEED Model

This model, originally developed for health education program design and implementation, structures the tasks to be completed before practitioners implement a program (Green & Kreuter, 2005). The model guides consideration of the predisposing, reinforcing, and enabling factors that contribute to a population's social and health behaviors. Assessing these well in advance will inform practitioners and evaluators about appropriate ways to adapt a program when strict fidelity simply will not fit (Green, 2001). To plan for features of the setting and/or community prior to implementation, evaluators can take an active role in considering the fit between the core components and priority issues identified by school administrators or community leaders (in a community setting, for example). A thoughtful adaptation makes it more likely that the desired outcomes might be attained in any setting. Not only does this add stakeholder confidence in the program but also can build knowledge about where the program can be expanded and used in more diverse settings and populations. Evaluators will find the PRECEDE portion of this model helps to structure those features of implementation that need to be measured. Using this portion of the model as a guide to pre-planning evaluation is recommended.

The PRECEDE portion of the model is structured in four phases. Evaluators can use data from these phases to anticipate the need for potential adaptations. Phases 1 and 2 include a social and epidemiological assessment using qualitative methods such as in-depth interviews and group discussions. Qualitative data collection continues in Phases 3 and 4 with the addition of surveys to quantitatively assess educational, ecological, and administrative and policy influences, respectively (Pourhaji et al., 2020). Data collected during these phases can then assist program planners and evaluators to examine the predisposing, enabling, and reinforcing factors that lead to the social and/or health behavior that is to be targeted or prevented during implementation. Discussion of potential adaptations needed would occur after these assessments have been analyzed and interpreted. Transparency in reporting all adaptations to funders and developers is strongly recommended.

Illustration: ProjectALERT (PA)

PA is a substance use prevention program (evidence-based) administered once a week in middle school beginning in seventh grade. It utilizes an active learning approach to resisting drug use and experimentation. There are 11 lessons to be taught in the first year of implementation and three booster lessons during the second year. Using videos, posters, games and other interactive teaching strategies, PA engages students in identifying the short- and long-term consequences of using drugs; actively learning resistance skills; and learning how to say no to substance use (RAND Corporation, 2022). ProjectALERT (PA) aims to help students develop non-use attitudes and beliefs.

PA illustrates the need for better information to guide adaptation. The RAND Corporation created and evaluated PA over a 10-year period. PA was originally found to be effective when implemented with fidelity; however, a recent review of three large studies found no effect in the total samples and positive effects for certain subgroups that were inconsistent across the studies (Gorman & Conde, 2010). There are at least three possible interpretations for these findings: (1) PA is consistently ineffective, (2) despite guidance, implementation is too variable, and (3) implementation is too strict and does not accommodate variations in the population and setting. To test the third possibility, one might establish whether PA is effective when carefully adapted to setting and population characteristics.

PA uses five assumptions to build the foundation for the program's content. In Table 2, we illustrate how three of these assumptions relate to potential adaptations and map to TOC. Beginning with a program's theory “assumption” can assist in identifying whether adaptations of the program may be needed and may translate to the core components. In Table 2, we further describe the selected core components of PA. An assessment of priority issues for the target audience, school, or community also helps to determine whether the program assumptions apply, and thus whether PA or any program is a good fit to address these priorities. Table 2 illustrates how assessing the program assumptions might work in the case of PA, along with suggested adaptations. Once these are determined and implemented, their effectiveness can be assessed both at the single site and potentially, across sites.

Table 2.

ProjectALERT Assumptions with Suggested Adaptations.

No.	Program Theory	PRECEDE Model Application	Potential Adaptation
1	“Adolescents start using drugs primarily because of social influences (peers, parents, siblings, media) and because they want to emulate behavior they view as socially desirable.”	Example predisposing factor: parents or close friends use substances; use is prevalent in the neighborhood or school yard	Identifying the consequences of use and how they affect personal goals. These are individual so discussion may be adapted to fit these goals
			Reinforce resistance skills to meet personal goals and include a focus on resistance skills to those we know
		Example enabling factor: there is an illicit market in the student's community	Reinforce positive peer influences NOT to use
	Core component mapping: [teaching strategy]
	Identifying consequences of use
2	“Drug prevention programs must help students develop the motivation to resist using drugs. Teaching resistance skills alone is not enough.”	Assess the attitudes and beliefs of the target audience—do most of the target audience believe that “most kids do it”?	The perception of “most kids do it” is addressed in the program using role modeling from older teenagers showing that, in fact, most kids do not use. Adaptations would mean adjusting the implementation to include a social marketing approach that reinforces, through community specific findings, most kids DO NOT use… [fill in substance use here]
		Examining actual use data from within the targeted community and analyzing the data locally (as opposed to nationally) can assist facilitators prior to implementation in understanding if adaptations are needed since it is possible that use in the local community is higher and closer to home than in other communities
	Core component mapping: [teaching strategy] Role modeling, social marketing
3	“Drug prevention programs should target substances that are used first and most widely by young people” Core component mapping: [teaching strategy].	PA addresses drugs used most widely by the target audience based upon the evidence. Like assumption 2 above, assessing actual use and prioritizing for the community or school-based level may be needed	If alcohol is not, in fact, a substance that is highly used in your target audience, adapting this ahead of time through working with the developer and community/school leaders is recommended
3	Address actual use prioritize intervention

Source: www.projectalert.com/research. For a complete list of assumptions, please visit www.projectalert.com/research/programassumptions

Planning and Pre-Implementation Phases

Program planners, practitioners, and evaluators select a program (or increasingly, combine components of various programs; Embry & Biglan, 2008) based upon their assessment of the needs and goals of the target audience's needs and their specific context. Evaluators can take an active role in this decision process. When they do, they can help focus on the potential for adaptation, which can then be integrated explicitly into the program and its evaluation. This needs assessment as described above identifies the priority issues and potential barriers and opportunities to improve program effectiveness. The program selection process considers the needs of the target audience, and their context, to discern the program or programs that would be most effective. A systematic planning approach can yield stronger program outcomes. For example, Chinman et al. (2018) teen sexual health program evaluation found that in communities that undertook systematic planning, adolescents reported more positive attitudes and intentions to use condoms. In a test of Botvin's evidence-based Life Skills Training, a comprehensive adaptation plan—developed as part of the planning phase of programming and prior to implementation and in concert with stakeholders—improved children's outcomes related to substance use (Centers for Disease Control and Prevention, 2021a; Velasco et al., 2017).

Evaluators can also contribute to the planning stage by selecting the most appropriate measures, based upon the existing evidence that informed the development of the program. For example, program activities may involve training students to recognize situations that pressure them into substance use (i.e., peer pressure scenarios). Clearly, increased “skill in recognizing” these situations are an intermediate program objective. Program materials may require adaptation of the scenarios presented if they are not relevant to students’/participants life situations. Prior to implementation, experienced evaluators, using the analysis of data collected, can further assess for the relevance of these scenarios to the audience and prescribe possible adaptations to resolve these issues (relevance may refer to presenting scenarios to people of color using materials not adapted to this population). To demonstrate the effectiveness of the skills taught using adapted activities, it is important to evaluate whether skills did, in fact, increase. Examining variables measured in the development phase of the program can inform formative evaluation: for example, to assess whether adapted materials were more relevant to student lives than originally believed prior to adaptation and therefore helpful to increase student skills (Castro et al., 2004). Evaluators can also select (or if necessary, develop) measures of student skills that take the social or cultural context into account.

In our experiences presented, evaluators, stakeholders, and practitioners conducted assessments, before implementing PA, to mitigate “in the moment” (unplanned) adaptations. Discussing other evaluations of PA and assessing local context revealed patterns that led to the need for adaptation. The setting of the intervention can complicate the adaptation process because the needs for adaptation may appear only during implementation and may not be anticipated. A rehearsal, pilot, or simulation can be helpful to avoid this situation. In our examples below time did not permit an explicit phase of this kind; it was left to the evaluation phase to determine if adaptations were appropriate.

Implementation Phase

ProjectALERT (PA)

What follows are experiences of adaptations needed during implementation of PA. These experiences lead us to emphasize the importance of early assessment (during pre-planning phase) and formative evaluation throughout the cycles of planning and implementation (an iterative process). In the experiences shared below, the program facilitator of PA was trained in evaluation, however, was functioning as the practitioner facilitating the program not evaluating it. A limitation to PA evaluation was that neither the funder in this implementation nor the developer of PA required nor requested reporting of adaptation. Another limitation was that resources and time limited data collection, precluded the assessment of adaptations during the planning phase. Analysis and interpretation of the data collected illustrated the effectiveness of some adaptations used, improving relevance to the audience circumstances, dose, and strength of implementation.

Example 1: Adaptation Related to Predisposing Factors

One of the lessons in the program has an at-home assignment to interview a parent or guardian about past behaviors related to substance use and how they responded to it. Discussion with stakeholders early in this phase revealed that this assignment could place a parent or guardian in the precarious position of being selective in the details of their story, when it was supposed to be an opportunity for honest discussion about experience with substances, the consequences of use, and lessons learned. If a parent(s) still uses substances, which is a predisposing factor (e.g., a father that drinks to excess, or a mother that gets high or takes prescription medications to “manage pain”), we adapted to avoid placing the child in a potentially distressing circumstance. Practitioners will not know if such a child is in the classroom once implementation begins, so planning for this eventuality is needed. Possible teaching strategies might be to instruct a program participant that this assignment will not affect their grade if it is not completed, that they could select any adult they feel safe asking these questions, or that they could skip the assignment and replace it with another assignment. Practitioners should be prepared for this possibility ahead of time and be transparent in reporting even one incident of having to adapt the assignment. Evaluators can predict this as a possible place for adaptation during the early assessment phase described above in the PRECEDE stage of the model.

Example 2: Adaptation for Dose, Relevance, and Reach

A practitioner implemented ProjectALERT in a residential facility for teens. The practitioner, trained and certified to implement PA, had done so many times in mainstream classrooms. Once the program began, the practitioner realized many of the students arrived having taken medication, rendering them sleepy and somewhat nonresponsive. Further, counselors and paraprofessionals were present in case a student experienced a disruptive emotional episode that might place themselves, their classmates, or the practitioner in danger. In the planning phase, the practitioner and the school counselor met frequently, but without an evaluator. They toured classrooms in session and continued discussions about how to implement the program in this setting. The specific student groups to receive the program had not been chosen prior to the implementation date. The practitioner was only aware that this setting was unique and not like the mainstream classroom.

While on the surface, the second example may not seem to be about fidelity and adaptation to the context, it illustrates both program “unknowns” and the need to adapt quickly. It would require a practitioner to be knowledgeable on how to adapt on any given day based upon the disposition of the audience, as well as the likelihood that the lesson would have an impact. The teaching strategy to produce measurable outcomes would likely need to be adapted (e.g., active learning vs. lecture style). This example highlights the importance of several concepts that evaluators would deem necessary for effectiveness including dose or strength (intensity and duration) and conceptual relevance (Yeaton & Sechrest, 1981). Students with disabilities are often excluded from evaluations of effectiveness. They deserve access to the same opportunities as mainstream students—in other words, reach.

Discussion

We recommend a hybrid approach to formative evaluation (pre-implementation and implementation phases) of programs that concurrently addresses both fidelity and adaptation. Challenges to fidelity of implementation can often be addressed with careful consideration of appropriate adaptations well in advance of implementation (Centers for Disease Control and Prevention, 2021b). Other adaptations may not be anticipated, but can still be monitored, assessed for appropriateness, and reported with transparency. Whether anticipated or not, adaptations can be assessed by reference to program theory and core components as set forth by the developer. The entire program team (community/site leaders, program planners, practitioners, and evaluators) plays an active role in the process of assessment during the pre-planning and planning phases. The role of the evaluator has been described throughout this paper. The contribution of the leaders of the site or community cannot be overstated. Their voices can elaborate on assessment findings and help identify priority issues that may lead to the need for adaptation.

When a program aims to change behaviors, the PRECEDE–PROCEED model is particularly helpful, because it predicts and guides adaptation. While PROCEED outlines summative evaluation methods that are familiar to this readership, PRECEDE is vitally important to interpreting the social assessment findings. Assessing the factors that predispose, reinforce, or enable behaviors by the target audience can be predictive of areas where adaptation would be needed. Stakeholders should be included as part of the workgroup sessions that precede implementation, as they are typically more familiar with the target audience than planners, practitioners, and evaluators. Mitigation of on-site, emergent adaptations during implementation can improve effectiveness. Outcome evaluation can validate not only the intermediate program outcomes being measured, but also the appropriateness of the adaptations applied.

The use of the PRECEDE–PROCEED model is helpful because it clarifies context, adaptation, effectiveness of adaptations, and therefore external validity. While some authors offer detailed typologies of adaptation (e.g., Stirman et al., 2019), guidance is limited on how the program theory translates into appropriate adaptation and effectiveness. External validity needs to be assessed based on context, which is a specific local interaction of population, setting (and larger context), treatment variations, and methods (Cronbach & Shapiro, 1982; Leviton & Trujillo, 2017; Shadish et al., 2002). Yet reviews focus primarily on generalizing about variations in population (Findley et al., 2021; Korsan & Crawford, 2014). Applying the PRECEDE–PROCEED model, along with program theory, core components, and other guidance from the stakeholders on context can remedy this situation.

Relevance to Single Site Evaluation

The choice of a program should lead evaluators to report on core components. With transparency and a solid rationale, departures from fidelity may be acceptable. Rigid adherence to fidelity expectations can create disparities between homogeneous, advantaged, capable communities compared to those that are diverse, have low literacy levels, may be economically disadvantaged or have other political and economic burdens that limit their access to resources. Clearly, this situation is not in the best interest of communities steeped in the social issues these programs are intended to address.

Without reporting transparency, however, stakeholders' understanding of the project is impaired, leading to a fundamental disagreement of whether program objectives were met. This misalignment of expectations with outcomes can lead to stakeholders’ frustration and even a loss of funding for a community agency or organization. By addressing adaptations evaluators increase the chances for local improvements in the next implementation cycle; indeed, this is the rationale for a focus on local adaptations in formative evaluation (Geonnotti et al., 2013). Adaptation is part of program adoption and assessing for the need to adapt early in the planning process is imperative (Rogers, 2003).

Relevance to Cross-Site Evaluation

One reason to document these adaptations is that they may benefit other practitioners, who may have encountered similar challenges. These issues arise with regularity in sectors as diverse as mental health, education, health promotion and disease prevention, criminal justice, medical care delivery, and social welfare (for a summary, see Leviton & Trujillo, 2017).

Federal repositories do not systematically collect information about program adaptation (Leviton & Trujillo, 2017). Few developers address the need for appropriate adaptations and rarely provide technical assistance when practitioners and evaluators encounter these challenges. An exception is HIV prevention, in which the Centers for Disease Control and Prevention (2022a) has long recognized that context determines the specific activities consistent with the program theory of change. This practice has demonstrably enabled participation by populations that are otherwise “hard to reach” through a clinic or government office.

Even when developers give considerable attention to identifying the core components of their programs, new queries to practitioners about adaptation can yield surprising results. For example, the Transitional Care Model has shown effectiveness for decades and is one of the most carefully studied programs in health services delivery (Naylor et al., 2009). Nevertheless, one survey of practitioners revealed many situations where the developers had not anticipated the need for adaptation (Naylor et al., 2018). In the same way, Hall et al. (2017) identified 49 separate adaptations of the patient-centered medical home model that were previously unstudied. Adaptations deserve more systematic study, given the hundreds of millions of research dollars that go into the development of tested programs, not to mention the time commitments of practitioners and evaluators and the opportunity cost for needy recipients.

To address this problem, practitioner and evaluator feedback through “crowdsourcing” may represent a golden opportunity to expand the repertoire of approved adaptations of programs; indeed, to test those adaptations for their effectiveness and thus expand the external validity of programs in the areas of highest priority. Evaluators should pinpoint context issues and adaptations that are both (1) frequently encountered and (2) judged important to program theory and core components (Leviton & Trujillo, 2017). In contexts where the core components simply cannot be implemented with fidelity, that needs to be known and shared with other practitioners.

Recommendations for Useful Evaluation by Role and Responsibilities

What follows are recommendations written for each program personnel role discussed in this article. It is possible that recommendations mentioned here with one role may in fact be performed by another program team member given the varied resources available to a community or organization.

Program Planners: During planning and pre-implementation phases, and in collaboration with evaluators:

Assess the situation through an environmental scan of issues facing the target audience.

Conduct interviews or focus groups to gather observations of the priority issues.

Compare the outcomes of evaluation studies on the selected program(s), if possible, to the results of the assessment.

Answer these questions: could this program be effective? What adaptations would be needed?

Once a program is selected, do the legwork prior to implementation to ensure program effectiveness.

Ask [community] stakeholders to join “diving into the weeds” or researching each core component, activity, lesson, and script to ensure fidelity is feasible and to what extent, if at all, the program will require adaptation and when.

Identify the core components as intended by the program developers where adaptation may water down implementation and make it less likely to attain the outcomes sought. The Centers for Disease Control and Prevention (2021) recommends these steps and are also reported in Freire et al. (2015).

Contact the program developers and the publisher of the program for the needed details about research into the reasons the program was developed and what are the core components. This information should be easily found in the program materials as well. If it is not, there may be need for consensus between program developers, [community] stakeholders, evaluators, and practitioners of what the core components should be (CDC, 2021). Alternatively, a different intervention that has the appropriate details may need to be selected.

Practitioners or facilitators:

Implement a program's core components with fidelity and with program's foundational theory in mind, as these are delineated in the evidence supporting the program's original development.

Practitioners or facilitators should take some time to familiarize themselves with the original evidence to understand the dynamics of what initially led to the creation of the program.

For newer practitioners this is especially helpful since this level of clarity can support their training and thereby improve implementation and what we care about most, which is program effectiveness. Be intentional about adaptation, including program changes that are relevant and sensitive to the needs of the target audience.

Balance adaptations with fidelity to the program implementation as it was intended. Mitigating mismatches such as those we have described can improve program effectiveness (Ballard et al., 2016). Should mismatches occur during implementation, the reflective practitioner will attempt to adapt using experience and the assistance of setting staff. Evaluators should carefully note any mismatch and report adaptation for future program improvement.

Evaluators: The role and responsibilities of the evaluator begin in the planning phase. If an evaluator is not employed as part of the program process, then another team member, such as the practitioner, should track adaptations and report them as part of the program reporting.

Interpret and translate findings from a social assessment to predict potential adaptations.

Keep in mind these types of acceptable adaptations: cultural, environmental, intellectual, educational, and medical, among others (Stirman et al., 2019).

Consider the participant worldview; this will challenge implementation to be viewed from “the other end” and will lead to a more equitable and productive relationship among practitioners, community stakeholders and participants (Rural Health Information Hub, 2018).

Following these recommendations will assist everyone to understand where and when it is appropriate to adapt a program.

Conclusion

A balance between fidelity to a program's core components and adaptation to a setting implies that a hybrid approach to implementation and formative evaluation is needed. Such an approach will serve the needs of individuals (communities), practitioners, and evaluators. Rigid fidelity to programming is an extreme requirement and leads practitioners to hide adaptations out of fear of losing funding. This, in turn, fosters an environment of disingenuous reporting and a missed opportunity for learning. We recommend some flexibility in funding criteria, with varying degrees of acceptance of adaptations agreed upon ahead of implementation. This will likely lead to a shift in the paradigm of reporting of adaptations to a more organic, authentic, and transparent practice.

Additional evaluation and crowdsourcing of adaptations are needed in programs and strengthens external validity. Lack of transparent reporting of adaptations keeps practitioners and communities in ignorance about what will work in their contexts. Creating new and improved funding review processes with tools to track adaptations and fidelity will help to eliminate programs that are ineffective in context. Learning and program improvement will follow for practitioners and funders alike, as they identify and track adaptations and fidelity that are appropriate for diverse populations. Ideally, such tracking will open the door to capturing data that improve guidelines for the design, implementation, and evaluation of programs.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Kate L. Nolt

References

Backer

T. E.

(2001). Finding the balance: Program fidelity and adaptation in substance abuse prevention. A state-of-the-art review. http://www.enap.ca/cerberus/files/nouvelles/documents/CREVAJ/Baker_2002.pdf

Ballard

S. M.

Tyndall

Baugh

Bergeson

C.B

. (2016). Framework for best practices in family life education: A case example. Interdisciplinary Studies of Applied Family Studies, 65, 393–406. https://doi.org/10.1111/fare.12200

Bandura

(1986). Social foundations of thought and action: A social cognitive theory. Prentice-Hall.

Blase

Fixsen

(2013) Core intervention components: Identifying and operationalizing what makes programs work. Office for the Assistant Secretary of Planning and Evaluation (ASPE). https://aspe.hhs.gov/reports/core-intervention-components-identifying-operationalizing-what-makes-programs-work-0

Castro

F.G.

Barrera

Jr. Martinez

Jr. . (2004). The cultural adaptation of prevention interventions: Resolving tensions between fidelity and fit. Prevention Science, 5(1), 41–45. https://doi.org/10.1023/B:PREV.0000013980.12412.cd

Centers for Disease Control and Prevention. (2021a). CDC approach to evaluation. https://www.cdc.gov/evaluation/approach/index.htm

Centers for Disease Control and Prevention. (2021b). Indicators - Program Evaluation. https://www.cdc.gov/evaluation/indicators/index.htm

Centers for Disease Control and Prevention. (2022a). Compendium of evidence-based interventions and best practices for HIV prevention. https://www.cdc.gov/hiv/research/interventionresearch/compendium/index.html

Centers for Disease Control and Prevention. (2022b). The social ecological model: A framework for prevention. http://www.cdc.gov/violenceprevention/overview/social-ecologicalmodel.html

10.

Chinman

Acosta

Ebener

Malone

P. S.

Slaughter

. (2018). A cluster-randomized trial of getting to outcomes: Impact on sexual health outcomes in community-based settings. Prevention Science, 19(4), 437–448. https://doi.org/10.1007/s11121-017-0845-6

11.

Collins

L. M

. (2016). Multiphase Optimization Strategy (MOST). https://methodology.psu.edu/ra/most/

12.

Connecticut (CT) Department of Health & Human Services. Making adaptations tip sheet (2022). https://portal.ct.gov/search-results/?q=evidence%20based%20programming#gsc.tab=0&gsc.q=evidence%20based%20programming&gsc.page=1

13.

Cook

T. D

. (2014). Generalizing causal knowledge in the policy sciences: External validity as a task of both multi-attribute representation and multi-attribute extrapolation. Journal of Policy Analysis and Management, 33, 527–536. https://doi.org/10.1002/pam.21750

14.

Cronbach

L. J.

Shapiro

(1982). Designing evaluations of educational and social programs. Jossey-Bass.

15.

Department of Health and Human Services, Family & Youth Services Bureau. (2018). Fidelity monitoring tip sheet. https://www.acf.hhs.gov/sites/default/files/fysb/prep-making-adaptations-ts.pdf

16.

Embry

D. D.

Biglan

. (2008). Evidence-based kernels: Fundamental units of behavioral influence. Clinical Child and Family Psychological Review, 11, 75–113. https://doi.org/10.1007/s10567-008-0036-x

17.

Findley

M. G.

Kikuta

Denly

. (2021). External validity. Annual Review of Political Science, 24, 365–393. https://doi.org/10.1146/annurev-polisci-041719-102556

18.

Freire

K. E.

Perkinson

Morrel-Samuels

Zimmerman

M. A

. (2015). Three Cs of translating evidence-based programs for youth and families to practice settings. New Directions in Child and Adolescent Development, 2015, 25–39. https://doi.org/10.1002/cad.20111

19.

Geonnotti

Peikes

Wang

Smith

(2013). Formative evaluation: Fostering real-time adaptations and refinements to improve the effectiveness of patient-centered medical home models. Agency for Healthcare Research and Quality. AHRQ Publication No. 13-0025-EF

20.

Glasgow

R. E.

Vogt

T. M.

Boles

S. M

. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1323–1327. https://doi.org/10.2105/AJPH.89.9.1322

21.

Gorman

D. M.

Conde

. (2010). The making of evidence-based practice: The case of project ALERT. Children & Youth Services Review, 32(2), 214–222. https://doi.org/10.1016/j.childyouth.2009.08.018

22.

Green

L. W

. (2001). From research to “best practices” in other settings and populations (American academy of health behavior research laureate address). American Journal of Health Behavior, 25, 165–178. https://doi.org/10.5993/ajhb.25.3.2

23.

Green

L. W.

Kreuter

M. W.

(2005). Health program planning: An educational and ecological approach. McGraw-Hill.

24.

Hall

T. L.

Holtrop

J. S.

Dickinson

L. M.

Glasgow

R. E

. (2017). Understanding adaptations to patient-centered medical home activities: The PCMH adaptations model. Translational Behavioral Medicine, 7(4), 861–872. https://doi.org/10.1007/s13142-017-0511-3

25.

Korsan

Crawford

. (2014). How to assess the external validity and model validity of therapeutic trials: A conceptual approach to systematic review methodology. Evidence-based Complementary and Alternative Medicine, 2014, 694804. https://doi.org/10.1155/2014/694804

26.

Leviton

L. C.

Schuh

R. G

. (1991). Evaluation of outreach as a project element. Evaluation Review, 15(4), 420–440. https://doi.org/10.1177/0193841X9101500402

27.

Leviton

L. C.

Trujillo

M. D

. (2017). Interaction of theory and practice to assess external validity. Evaluation Review, 41(5), 436–471. https://doi.org/10.1177/0193841X15625289

28.

McKenzie

J. F.

Neiger

B. L.

Thackeray

(2017). Planning, Implementing, and Evaluating Health Promotion Programs – A Primer (7^th Edition). Pearson Education.

29.

Naylor

M. D

., Feldman

P. H.

Keating

Koren

M. J.

Kurtzman

E. T

. (2009). Translating research into practice: Transitional care for older adults. Journal of Evaluation in Clinical Practice, 15, 11–70. https://doi.org/10.1111/j.1365-2753.2009.01308.x

30.

Naylor

M. D.

Hirschman

K. B.

Toles

M. P.

Jarrin

O. F.

Shaid

Pauly

M. V

. (2018). Adaptations of the evidence-based transitional care model in the U.S. Social Science and Medicine, 213, 28–26. https://doi.org/10.1016/j.socscimed.2018.07.023

31.

Pourhaji

Delshad

M. H.

Tavafian

S. S.

Niknami

Pourhaji

. (2020). Effects of Educational Program Based on Precede-Proceed Model in Promoting Low Back Pain Behaviors (EPPLBP) in Health Care Workers. Shahid Beheshti University of Medical Sciences: Randomized Trial. Heliyon https://doi.org/10.1016/j.heliyon.2020.e05236

32.

RAND Corporation. (2022). ProjectALERT. https://www.projectalert.com/

33.

Rogers

(2003). Diffusion of innovations (5th ed.). Free Press.

34.

Rossi

P. H.

Lipsey

M. W.

Henry

G. T.

(2018). Evaluation: A systematic approach (8th ed). Sage.

35.

Rural Health Information Hub. (2018). “Adapting interventions: Types of adaptations”. https://www.ruralhealthinfo.org/toolkits/rural-toolkit/2/adapting-interventions

36.

Scheirer

M. A.

(1981). Program implementation: The organizational context. Sage.

37.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton-Mifflin.

38.

Stirman

S. W.

Baumann

A. A.

Miller

C. J

. (2019). The FRAME: An expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14, 58. https://doi.org/10.1186/s13012-019-0898-y

39.

Velasco, V., Griffin, K. W., Botvin, G. J., Celata, C & Gruppo LST Lombardia. (2017). Preventing adolescent substance use through an evidence-based program: Effects of the Italian adaptation of life skills training. Prevention Science, 18, 394–405. https://doi.org/10.1007/s11121-017-0776-2

40.

Yeaton

W. H.

Sechrest

. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49(2), 156–167. https://doi.org/10.1037/0022-006X.49.2.156