Abstract
Premised on the idea that evaluators should be familiar with a range of approaches to program modifications, I review several existing approaches and then describe another, less well-recognized option. In this newer option, evaluators work with others to identify potentially needed adaptations for select program aspects in advance. In describing this approach, I note the general steps involved and present alternative techniques for identifying, a priori, adaptations that may come to be needed. In the final section, I discuss implications of the a priori adaptation planning approach for the fidelity–adaptation trade-off, past criticism of logic models as overly fixed and linear, potential research and evaluation questions, the development of more detailed views of programs in evaluation theory and training, and possible resistance to adaptation planning. Discussion also considers the potential future of program adaptations in the wake of the COVID-19 pandemic.
Keywords
Introduction
Among the apparent lessons for evaluators of the COVID-19 pandemic is the need to deal with modifications 1 of original program plans. During the pandemic, some programs—and some evaluations—came to a sudden halt when governments mandated shutdowns. Other programs continued but adapted in various ways, such as by shifting from in-person to remote delivery. Evaluations likewise adapted, often by modifying data collection procedures and timelines. This experience highlights how important is it for evaluators to be familiar with ways in which they can attend to program modifications in their practice. Given the varied circumstances in which program adaptations will occur, evaluators would be well served by being familiar with a range of options for engaging with modifications.
One approach, at least when we are not in the midst of a pandemic or other emergency, would be to strive to avoid deviations from the program plan. The concept of implementation fidelity (Durlak & DuPre, 2008) assumes this as a goal. However, outside of rigidly controlled and monitored implementation settings, this effort might be akin to King Canute’s proverbial command telling the tides to stop. A growing body of research, mostly drawing on interventions in the area of behavioral health, documents that program adaptations take place frequently even during periods of relative equilibrium (e.g., absent a pandemic or other crisis) and in programs that are intended to be relatively stable (e.g., Miller-Day et al., 2013; Hansen et al., 2013; Stirman et al., 2019). Miller-Day and colleagues (2013) documented modifications within a single program, a school-based substance abuse prevention program with a detailed curriculum. Using observer codings as well as teacher self-reports following the lessons, these researchers reported how and why the teacher implementers deviated from the program curriculum. Examples of the observed modifications include sessions being dropped or combined because of school closures (e.g., snow days) or interruptions (e.g., fire drills); material in a session that was deleted or modified as the end of the class period approached; adaptations that were targeted to local student characteristics, such as by the teacher who reported “I adapted the lesson to be more relevant for students living in rural areas”; addition of material intended to increase student attention, such as a popular music video; a school shifting from the planned within-classroom delivery to delivery in the school auditorium for all classrooms at the relevant grade level; and the addition of teachers’ personal experiences and stories, some relevant, but some not—such as the reportedly excellent class discussion on deer hunting. As these and other examples illustrate, program modifications occur in various forms and for various reasons and in varying “degrees of modification” (Resnicow et al., 2000). Typically, such modifications occur with no previous planning or at least no formal planning process.
In the remainder of this article, I first review a selection of existing approaches for evaluators to engage with program modification. In the section following that, I describe another, less familiar approach. This approach involves the advance planning of adaptations which might prove to be needed. A set of general steps for such planning is presented, as are more specific techniques that can be used when engaging in the a priori planning of potential program adaptations. The final section of the article discusses a number of related issues including the potential benefits of, and challenges to, a priori planning of program adaptations.
Existing Options for Engaging With Program Adaptations in Evaluation
Several of the current alternatives for dealing with program modifications are described in the subsections that follow, which present three groupings of options. For the first grouping, the options range from simple description of adaptations that arise during a program’s operations to more recent work on the development and application of classification systems that characterize different types and features of modifications. A second grouping includes options whereby evaluation helps to incorporate program modifications into program design. This stream of thought started with early advice about sequencing types of evaluation and has expanded in recent years with several more advanced ways of incorporating adaptations in program models. A third grouping includes evaluators’ presentation of argument and evidence about the consequences of program modifications, especially with respect to program effectiveness. Although presented here as three distinct grouping of approaches, these at times overlap. In addition, they are better thought of not as exclusive alternatives but as three sections in a menu of options from which evaluators can mix and match in practice.
From Description to Classification
One stream of work started with attempts to describe and represent program modifications. Some of these efforts involve expansions of commonplace program modeling in order to capture adaptations. For example, den Hoyer (2002) introduced the idea of the “temporal logic model.” Drawing on systems theory and the concept of learning organizations, den Hoyer proposed that logic models be recorded at multiple points in time in order to capture program modifications, along with changes in context. Metaphorically, one can imagine shifting from a logic model as a static picture and instead having a series of logic models in a flip-book. In related contributions, Mayne (2015) and Funnell and Rogers (2011) described extensions of simple program models that, among other things, can be used to capture adaptive changes. They suggested the use of nested models, which Mayne (2015) refers to as submodels or subtheories. When used to capture program adaptations, rather than a flip-book the metaphor would be of post-it notes that list adaptations and are stacked on top of the original logic model. Suggestions such as these from den Hoyer, Mayne, and Funnel and Rogers can be helpful when an evaluator wishes to capture and communicate actual program adaptations. More advanced visualization techniques may also be helpful in this task.
Other work in this stream has moved beyond simple description of program modifications (e.g., How frequently do they occur? What do they look like?), shifting to the development, validation, and application of systems for classifying the characteristics and types of adaptations. An example of research that straddles description and classification comes from the previously described Miller-Day et al. (2013) investigation of the kind and range of program modifications within a school-based substance abuse prevention program. Multiple descriptive studies have fed into the development of systematic frameworks which describe the variety of kinds of program modifications that occur in practice as well as a range of related considerations such as who made the modification and why. 2 Alternative coding systems of this sort have been developed (e.g., Hill et al., 2007; Moore et al., 2013), with Stirman and colleagues (2013, 2019) providing the most fully developed framework to date. Labeled Framework for Reporting Adaptations–Expanded (FRAME) in 2019, and with additional revisions by Kirk et al. (2020), this coding system includes attention to (a) when in the program’s implementation process the change was made, (b) who made the modification, (c) at what level of the program (e.g., overall or at the individual service provider level), (d) what was modified (content, context, or training and evaluations), (e) the nature of the modification for a context modification or for (f) a content modifications, (g) one of four codes indicating whether the change was systematic/unsystematic and planned/unplanned, (h) a judgment about the extent to which the change was consistent with core functions (or in other words, consistent with implementation fidelity), and (i) the reason for the change (with options such as feasibility, cost, cultural responsiveness, and sustainability).
The most relevant FRAME category is probably the one describing the nature of modifications to program content. A list of the alternative forms of content change is given in Table 1. Review of the entries in Table 1 provides a sense of the multiple ways in which the content of a program may be modified. The second category of FRAME (Stirman et al., 2019, figure 1) describes alternative forms of context modifications. These involve changes “to the way the overall treatment is delivered.” Types of contextual modifications include changes to format (e.g., from individual to group administration), setting (e.g., from therapist’s office to school-based), personnel (e.g., from certified professional to peer counselor), or population (e.g., predominately middle-class to migrant worker). 3 A third relevant category coded in FRAME involves the level of delivery at which a modification is made. Is a change made for only an individual client or for the entire organization or community? Alternatively, is the modification made for a targeted subgroup or implemented only by a particular practitioner or subgroup of treatment deliverers? Finally, FRAME includes a set of codes representing different reasons for program modifications. Table 2 gives a list of these reasons with slight modifications. See Stirman et al. (2019) for further details of FRAME.
Alternative Forms of Content Modifications.
Source. From Stirman et al. (2019), figure 1.
Reasons for Program Adaptations and Modifications.
Source. Adapted from Stirman et al. (2019), figure 1.
With categories and codes like those in FRAME, the content adaptations that are observed in an evaluation can be classified. Classifying an adaptation in this way is a step beyond the simple description of the kind captured in a temporal logic model or a submodel. A framework such as FRAME enables the use of consistent labeling across evaluations. Classification can also facilitate other approaches for engaging with adaptations. Accordingly, we return to FRAME and its potential uses later in this section and again in the section that follows.
Evaluation as a Tool for Incorporating Adaptations Into Program Design
A second stream within the evaluation literature on program adaptations includes an evolving set of options about how to incorporate modifications into program design. A long-standing version has been advocated by both pioneering thinkers about evaluation (e.g., Cronbach & Associates, 1985) and more recent evaluation scholars and practitioners (e.g., Scheirer, 2012) who have viewed program modifications from the lens of stages or phases. The typical suggestion is that pilot testing, formative evaluation, and the like should take place early in a program’s life history. The learning from this early evaluation phase, including learning about apparently beneficial adaptations, then feeds into the refinement of a stable program model that can be evaluated summatively (Chen, 2004). As an example, in an evaluation of Bright Start, a school-based obesity prevention program, Story et al. (2012) engaged a wide array of stakeholders in a formative stage. Activities during this phase of the evaluation were undertaken to: (i) inform the design and delivery of an obesity prevention intervention program appropriate for Lakota schools and families; (ii) pilot test selected intervention components; and (iii) plan a program that was feasible, culturally appropriate, and sustainable to allow long-term integration into the community. (p. 2242)
An array of options have been developed that update the classical formative–summative sequencing for integrating program modifications into program design. Among these are the multiphase optimization strategy, or MOST, a form of behavioral engineering intended to optimize the design of multiple component interventions (Collins, 2018). Drawing on existing theory and research, and perhaps on additional pilot testing, potential components of an intervention are identified. For example, in a weight loss intervention, potential components might include coaching, informational videos, food tracking, activity tracking, weight tracking, and a buddy messaging system (Lin et al., 2015). While not required, the selection of intervention components and their specific levels can derive from adaptations that emerged during previous experiences with similar interventions. With potential intervention components in hand, optimization criteria are specified, such as a maximum acceptable cost and/or the exclusion of all inactive components. All these feed into an optimization phase, which draws from a selection of research designs to assess which constellation of components best meets the optimization criteria. The simplest design at this stage is the factorial experiment, in which each of the levels of one intervention component (e.g., 0, 1, or 3 hr/week of coaching) is paired with each of the levels of the other component(s). In practice, complete factorials are typically not used when several intervention components are considered, due to the resource demands of such a design, and incomplete factorials are a more efficient alternative. The results of the formative engineering work translate into an intervention that can then be tested summatively in a randomized trial. See Collins (2018) for details on MOST.
Additional options are emerging from the developing field of implementation science. Formative evaluation of the process of implementation is a common activity in implementation science (Bauer et al., 2015). Other specific techniques have been developed. For example, Glasgow and colleagues (2020) describe a procedure in which teams of program staff and investigators used the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) model, which has been widely employed in public health, for the planning of midcourse adaptations. Team members rated progress on each of the five dimensions of RE-AIM, selected dimensions for improvements, developed an action plan for the targeted modifications, implemented the changes, and evaluated the results.
Also related to implementation science and MOST are a family of techniques for building an intervention contingently. While it may sound as though it would cover the entire range of approaches to program modifications, the term adaptive intervention 4 has come to have a more limited meaning. Sometimes referred to as adaptive (or dynamic) treatment strategies, adaptive interventions employ explicit decision rules that specify whether and how the kind and/or dosage of an individual’s treatment should be changed in light of the person’s current state and circumstances. Just-in-time adaptive interventions rely on technologies such as smartphones and mobile sensors to deliver an intervention in real time, with the capacity to change the client’s treatment and perhaps individualize the underlying decision rules. For example, with a mobile phone and motion sensor, an adaptive intervention can employ mini-experiments (sometimes called micro randomized trials) to learn what times of day and which messages or prompts are most effective at increasing physical activity in obese adults. Such mini-experiments can be conducted in early pilot phases or can be built into the entire life of the intervention. Nahum-Shani et al. (2018) describe the nature and design of such interventions.
Patton’s (2011) developmental evaluation also falls within this group of approaches. Developmental evaluation is rooted in work with innovators who were developing social change initiatives in complex environments (Patton, 2006). Drawing on concepts from complexity theory and flexibility with respect to methods, the developmental evaluator strives to provide timely information on what may be an ever-evolving intervention, with program modifications being made in response to complex and changing circumstances. Potential changes even include shifts in program goals, which themselves may evolve over time. In a March 2020 statement inspired by the COVID-19 pandemic, Patton indicated that “All evaluators must now become developmental evaluators” (https://www.utilization-focusedevaluation.org/blog). That statement also characterized much of what developmental evaluation entails. Developmental evaluators, Patton indicated, need to be “capable of adapting to complex dynamics systems, preparing for the unknown, for uncertainties, turbulence, lack of control, nonlinearities, and for emergence of the unexpected” (https://www.utilization-focusedevaluation.org/blog). Developmental evaluation was developed for interventions that continue to evolve in a complex environment. The Discussion section returns to the contention the evaluators will likely be best served by familiarity with a range of options for engaging with program adaptations, with developmental evaluation as one of the options.
Argument and Evidence About the Effects of Adaptations
The third subgroup of work on program modifications examines the consequences of implementation fidelity or, conversely, of adaptations. For programs with relatively detailed plans, such as many school-based substance abuse programs, fidelity refers to the degree of conformity to that plan during implementation. Several studies have shown larger program effects when the treatment is implemented with higher levels of fidelity (see, e.g., Durlak & DuPre, 2008; Dusenbury et al., 2003). Moving beyond a fidelity measure, Hansen et al. (2013) used observation methods to measure how many times the middle school teachers who delivered a drug prevention program modified the planned intervention. Trained raters also coded whether each adaptation seemed positive, neutral, or negative with respect to program goals. Hansen et al. reported that all teachers made adaptations, the students whose teachers made a few positive adaptations had better outcomes, and student outcomes were worst for the teachers who most frequently modified the program. Hansen et al.’s study incorporated description (how often did modifications occur) but went further, probing the association between these modifications and program outcomes.
Kirk et al. (2020) recently pointed a direction for future work in which adaptation features are classified, with the results examined in terms of their relationship to program outcomes. Expanding on FRAME, Kirk and colleagues developed a complementary model, the Model for Adaptation Design and Impact (MADI), to guide such research. MADI treats most of the characteristics from FRAME (e.g., the type of content change, who made the modification) as features that might influence implementation and intervention outcomes. The model includes one mediator, alignment with core function (i.e., relationship to fidelity), and potential moderators can be included. Implementation and intervention outcomes would be treated as outcome variables in analyses that assess the consequences of varying kinds of adaptations. Research of the kind Kirk et al. (2020) suggest, with data on adaptation characteristics and implementation and program outcomes from multiple evaluations, could help clarify the benefits and costs of various kinds of program modifications.
Even before MADI-style research has accumulated, a considerable body of empirical and conceptual work has focused on the relative merits of fidelity and adaptation (e.g., Bopp et al., 2013), especially in relation to contextual and cultural responsiveness (e.g., McIntosh et al., 2013). Contemporary discussions of the fidelity–adaptation trade-off often boil down to the suggestion that it is vital to understand which aspects of the program are “core components” that should not be changed and which are aspects of the “adaptable periphery” that can be modified to enhance cultural and contextual fit (e.g., Damschroder et al., 2009; Movsisyan et al., 2019). In keeping with the core/peripheral distinction, Kendall suggested a principle for the implementation of evidence-based treatment protocols called “flexibility within fidelity” (e.g., Kendall & Frank, 2018). Many others have made similar points, such as when Hansen et al. (2013) indicated that “Programs should define acceptable and unacceptable ways they may be adapted.” The devil in the details involves what kind of evidence, theory, or expert judgment is required in order to treat some aspects of the program as core and others not. Movsisyan et al. (2019), in a systematic review of guidance for adapting behavioral health interventions for new contexts, reported that “while there is a common emphasis on the need to distinguish between core and discretionary components, we did not find guidance on how to identify the core components that need to be maintained during adaptation” (p. 16). Arguably, the line between core component and adaptable periphery is especially likely to be crossed when adaptations are made ad hoc under time pressure, in response to unanticipated problems or negative information about program performance, and especially when changes are made on the fly by program staff with little if any background on the core components of the program. This suggests the potential value of advance planning of possible adaptations for select program elements, which is the focus of the next section.
The Potential Role for Evaluators in A Priori Planning for Program Adaptations
As just noted, this section focuses on yet another way in which evaluators can engage with program modifications: participation in a priori planning for potential adaptations. An example of the need for such planning comes from the previously mentioned Story et al.’s (2011) evaluation of Bright Star. Considerable formative work was conducted on this school-based obesity prevention before the program was evaluated summatively. Even with all the formative evaluation, however, at least one program modification occurred during the summative evaluation (Story et al., 2011). An element of the planned program was dropped mid-implementation. Specifically, the program plan included phone calls to provide motivational encouragement following each of the intervention’s family fun nights. However, this outreach activity was dropped because of the difficulty encountered in reaching the families by phone. As this example illustrates, while a stage model of evaluation may suggest that program adaptations take place only during earlier program stages, the possibility is good that some modifications will occur even in a mature program, and even during summative evaluation. A growing body of evidence supports this assertion (e.g., Hansen et al., 2013; Miller-Day et al., 2013; Stirman et al., 2013, 2019).
In the remainder of this section on a priori planning of potential program adaptations, I first overview the approach, describing four general steps of the process. The overview points out this planning process can be employed at multiple points in the program life cycle and also considers who should be involved in the process. Following the overview, four different techniques are presented that could be employed, alone or in some combination, during the first two steps of the four-step process. Attention then turns to the last two steps, each of which is similar in form regardless of which techniques were used in the earlier steps.
Overview of the Approach
The primary objective of this approach is to have a plan in hand, in advance, if a modification is needed for a select set of the elements of the program. It is of course not possible to predict with certainty which aspects of a program may need to be adapted and how. However, a group process can make informed judgments about the most important candidates.
A priori planning of possible program adaptations involves four general steps, which are summarized in Table 3. First, the part(s) of the program that may need to be modified must be identified, and, if more potential areas for modification are identified than can be addressed during the rest of the process, priorities must be set among the nominated program elements. Second, session participants need to suggest potential adaptations that would address the likely shortcomings. If multiple plausible options are generated for a given program element, one option must be chosen. Alternatively, a decision could be made to offer multiple adaptation options, perhaps with the specification of how each might be enacted contingent on circumstances. In practice, Steps 1 and 2 are likely to be intertwined. Eliciting a potential adaptation will sometimes be at the forefront of the process, with the nature of that adaptation pointing to the part of the program that would be modified. Given the potential intertwining of Steps 1 and 2, a set of techniques are presented shortly that could be employed for the two steps together. In the third step, decision rules are specified that would trigger the implementation of the potential adaptation(s). The topic of decision rules can also lead the process to iterate back to Step 2. In particular, discussion about decisions rules could reveal that a program element may fail in qualitatively different ways, with each kind of failure calling for a different adaptation. The fourth step involves documentation and, ideally, evaluation. The adaptation plan itself should be documented. If any planned adaptations are subsequently implemented, that change should be documented and, if feasible, evaluated. This set of steps bears a general resemblance to many other planning processes including ones discussed by Escoffrey et al. (2019), Glasgow et al. (2020), and Movsisyan et al. (2019).
General Steps in Planning Potential A Priori Program Adaptations.
This four-step process can be applied at various points in the life cycle of an evaluation. It could occur prior to implementation, while the initial program plan is being developed. Conducted at that time, potential adaptations would be available as the program learns to walk, so to speak. Such timing may be relatively infrequent, however, given the common complaint from evaluators that they are brought on the scene too late. Fortunately, the process can also be engaged in later and during either formative or summative evaluation. Moreover, the a priori adaptation planning process can take place whether or not an explicit program plan is already in place. Sometimes no explicit program model will exist. This would occur if an evaluator is participating in initial program development or could occur later in a program’s life cycle when an evaluator is working with staff and stakeholders to surface a program model that has been left implicit. In either case, adaptations would probably be considered after a draft program plan has been developed. Alternatively, the evaluator may be brought in after an explicit program plan exists, whether in the form of a logic model, some other form of program theory, or a program manual. In such cases, the planning process can be conducted as part of the evaluator’s review of the program plan with relevant stakeholders.
Also related to the timing of the proposed planning process, although presented here as a one-time exercise, the process can be repeated and probably should be if pragmatics allow. This may be especially beneficial if the initial a priori planning of adaptations took place early in the program’s life cycle. Some potential threats to successful implementation are more likely to be visible after the experience has been gained with implementation. Further, context may change over time in ways that create new threats to the effectiveness of various parts of the program.
Who participates in the planning process will likely depend in part on where the program is in its life cycle. If the planning takes place prior to the initial program launch, the program planning team may be the only ones involved. If planning for potential adaptations takes place after a program is in operation, implementing staff would likely participate. Knowledge of the program content and delivery is needed, so graduates or longer term clients of an ongoing program may be more appropriate than new or upcoming program clients. Because the discussion about adaptations should be carried out in light of theory, evidence, and plausible assumptions about which aspects of the program plan are core elements, the participation of program developers, and perhaps previous evaluators and/or others familiar with the underlying program theory, seems especially valuable if possible. If these parties cannot participate in the planning process, perhaps they can review the potential planned adaptations that result with an eye toward vetoing or suggesting revisions of any that endanger a core component.
Finally by way of overview, although the present article focuses on the a priori specification of potential future adaptations, the suggested process can also contribute to initial program design or to redesign. For example, the first technique presented for Steps 1 and 2 expands on procedures previously used in business to strengthen an initiative’s implementation plan by considering possible causes of the initiative’s failure. Some evidence suggests that this kind of advance review can increase the likelihood of an initiative’s success. In research on the effectiveness of decisions made in a range of organizations, Nutt (2002) found that consideration of more than one alternative is a strong predictor of success. It is possible that, as in Nutt’s (2002) research, consideration of alternatives may in some instances lead to direct change of a program plan rather than to the selection of a potential adaptation to be implemented if needed. If so, this should not be considered a failure of the planning process!
Steps 1 and 2: Identifying Program Elements and Potential Adaptations
As previously noted, Steps 1 and 2 of the general process are likely to be intertwined in practice during the a priori planning of potential program adaptations. For this reason, techniques are presented for jointly completing these two steps. Four potential techniques are listed. These are not exclusive and might be used together, perhaps in sequence. Nor should this be considered an exhaustive list. Instead, these techniques and the details of their implementation are offered, not as fixed procedures but as suggestions to be considered, possibly adapted, and perhaps replaced by other techniques that prove to be effective.
Technique 1: The premortem
One method for generating ideas about potential program adaptations is to conduct a variation on what the organizational theorist Klein (2007) called a “premortem.” The term is a wordplay on the postmortem that is carried out in hospitals after the loss of a patient. But where a postmortem takes place after a bad outcome has already occurred, a premortem involves thinking about the potential causes of failure before an endeavor kicks off. Applied directly to a program, in a premortem, the program team and perhaps participating stakeholders would be asked—prior to the program’s implementation—to assume that at some point in the future, the program has been shown to be a failure. Each person would be asked to write down every possible reason for the failure, including reasons that might step on someone’s toes. Although variants of the premortem process exist, in the one most relevant to adaptation planning, group discussion occurs after individuals identify possible reasons for program failure. The group discussion leads to the selection of the most serious and/or most likely causes of failure. Further deliberation is then undertaken to identify ways to avoid the potential problems the group has chosen as the most serious threats.
The primary use of the premortem in business settings has been to change the initial plans for an initiative, such as the rollout of a new product. In a program evaluation context, a premortem can also be used to identify backup options to serve as potential, planned adaptations should they prove to be needed. This expanded purpose will require additional steps, the next one being the specification of decision rule about the circumstances that would trigger the implementation of the adaptation, that is, Step 3 as discussed below. When extended to programs, the premortem can also be modified for use after the program has been implemented.
Although the focus of this article is on planned potential program adaptations, as just noted, the premortem and other techniques might also lead to program changes to be made immediately. For example, a lesson in a school-based prevention program session might be restructured so that the key takeaway for program participants does not come at the end, given the possibility of time constraints that will lead to ad hoc revisions or omission of the material slotted for the end of the class period. In other cases, a redundancy might be built into the plan. If, for instance, the initial plan included motivational phone calls to parents, the plan might be revised to include up to three attempts to connect by phone calls, to be followed by an email message if needed, with a hard copy to be sent by mail as a further backup if the previous attempts have been unsuccessful. With this kind of embedded backup, it is as though a potential adaptation is prebuilt into the program plan.
Technique 2: Questioning the game plan
Klein’s (2007) premortem focuses on overall program failure. The program staff and stakeholders participating in the adaptation planning process can also be asked to focus on more specific forms of program failings. Table 4 lists several questions that can be used to elicit ideas about potential program adaptations. In essence, the planning participants are asked to question various portions of the “game plan” embodied in the program theory, logic model, or curriculum. While a few questions in Table 4 are designed to generate thinking about a general lack of success (e.g., “the wheels fall off” or you only get “75% of the way” to the goal line), most ask about more specific forms of program failure. For example, planning participants are asked to imagine that the program is less effective for certain kinds of clients, or in certain kinds of settings, or when delivered by certain kinds of staff. Other questions are designed to ask session participants to reflect on any ideas they had earlier that did not make it into the final program plan or to reflect on practical obstacles that can arise “even despite excellent planning.” These questions are included in part to help avoid a kind of confirmatory bias, whereby session participants may tend to stay committed to the program plan as formulated. Another question (which may or may not be relevant) asks session participants to think about changes that may be called for in future dissemination or scale-up efforts.
Questions to Provoke Consideration of Program Model Modifications and Preplanned Adaptations.
Otherwise, the process would be much like that described for the premortem, including the use of group processes to identify and select potential modifications should the identified problem actually occurs.
Technique 3: Priming the pump
With old-fashioned hand pumps, air pockets could inhibit function. The solution was to “prime the pump,” which involves the counterintuitive action of pouring water into a pump when the goal is to get the pump to provide water. Used widely in economics as a metaphor, the phrase can also apply to adaptation planning. The term “priming” also has an apt meaning in psychology, referring to the phenomenon by which exposure to one concept makes another, related concept or network of ideas more accessible and more quickly retrieved.
In the case of a priori adaptation planning, the pump can be primed, metaphorically speaking, by providing information about other adaptations in order to facilitate session participants’ thoughts about adaptations for their program. This could be done by providing a description of adaptations previously made in other programs such as the list by Miller-Day et al. (2013). Alternatively, Stirman et al.’s (2019) FRAME includes a reasonably comprehensive listing of various kinds of content modifications, alternative forms of context modifications, and various reasons for modifications. These lists were developed based on previous reports that identified changes made to active programs, and they were validated with rater codings of reported adaptations and cross-checks with similar frameworks. Presentation of portions of this framework could prime the pump, stimulating session participants’ thoughts about modifications that might be needed in their program.
Four of the categories/questions in Stirman et al.’s framework are likely to be of greatest interest when thinking about potential program adaptations. One presents different kinds of changes to program content, such as modifications to the curriculum or the exercises in a peer relations intervention. The types of content changes were already listed, in slightly adapted form, in Table 1. FRAME’s codings for alternative types of context modifications may also be useful to prime the pump. The contextual changes involve modifications to format, setting, personnel, or target population. FRAME also included codings for the level of delivery at which an adaptation occurs (e.g., at the level of the individual client, or a single implementer or a subgroup of service providers, or a targeted subgroup of clients, or the entire organization or community). Finally, FRAME includes alternative reasons for program modifications, such as time constraints, the cultural norms of recipients, and staff skills. A list of these reasons, with slight modifications, was presented in Table 2.
To prime the pump, then, participants in the adaptation planning process could receive a list of the type of changes and reasons listed in FRAME. These could be presented in full. More likely, the person leading the process would edit the list in advance. Depending on the program and planning session participants, some items from FRAME might be omitted (e.g., some levels of delivery may not apply for a given program), or condensed (e.g., “resource constraints” would incorporate many of the first set of reasons in Table 2), or rephrased (e.g., translated into local terminology). Some items might not be listed but rather held by the session facilitator to share whether that seems helpful during discussion. After session participants review the types of adaptations and reasons, they would be asked to think individually about which aspects of the program are most likely to need an adaptation. The idea is that the details from FRAME, perhaps supplemented with concrete examples the evaluator provides, would prime the pump and stimulate people’s thinking about where and why adaptations may be needed. Group discussion would then set priorities, review possible adaptations, and move on to Step 3 to discuss decision rules.
Technique 4: Using “old saws” to build an adaptation plan
The participants in some planning sessions may appreciate the comprehensiveness of FRAME. At other times, participants may be put off by the detail and the academic tenor of that framework. The fourth technique may appeal in such instances using more folksy language that can be supplemented with visuals. My suggestion for this technique is to use a set of old saws, that is, long-standing adages or expressions, to help stimulate participants’ thinking about potential program adaptations. In particular, three familiar expressions may help people consider possible program adaptations: (1) “Plan B,” (2) “different strokes for different folks,” and (3) “belts and suspenders.”
The phrase Plan B covers much of the territory of program modifications. In general, Plan B refers to having in advance, or else in real time generating, a fallback option in case one’s original plan falls short. An ad hoc decision to condense a program lesson by skipping a role-playing exercise illustrates a Plan B, albeit a seat-of-the-pants example. For a priori examples of Plan B, imagine a curriculum that specifies in advance which content to leave out if time runs short or a protocol whereby the program will extend its recruitment area if the number of participants does not reach a specified level by a certain date.
The expression of different strokes for different folks conveys the idea that alternative variants may be preferable for different types of people. In an obesity prevention program, the examples or exercises might be different in rural than suburban schools, for example: It’s hard to follow a recommendation to walk around your block three times when there aren’t city blocks! Differentiation across subgroups can occur initially, when participants join the program with varying characteristics. Or it can be implemented later, contingent on poor performance by the members of a subgroup, as discussed further in the context of decision rules.
Multiple terms have been used, somewhat inconsistently, in prior literature to refer to the process by which program variants are matched to individuals or subgroups (including targeting, tailoring, customization, individualization, and market segmentation). Targeting is probably the most relevant for most instances of the proposed planning process. Targeting refers to a subgroup-based approach whereby different procedures are employed for different subgroups. For example, Babar et al. (2010) discuss the development of gender-specific tobacco prevention interventions for boys and girls in India. Targeting is a way of providing different strokes for different folks.
The phrase “belt and suspenders” refers to the simultaneous use of redundant means to a given end. For example, in an intervention designed to teach social skills to youth with autism, McCoy et al. (2016) used video modeling, role-play exercises, and computer-based instruction. Like the person who literally wears a belt and a pair of suspenders, the thought is that good outcomes (like keeping one’s pants up) should be more likely with multiple approaches than if only one is used. A good outcome is not a given in a program context, however, in part because time and other resource constraints may mean program participants receive a low dose of each approach (Squires et al., 2014). Still, it is probably worthwhile during the adaptation planning process to highlight that, rather than only generating adaptations to be held in abeyance (pending the application of some decision rule), the planning group can also identify backups or redundancies to be added immediately to the program plan. The image of belt and suspenders can help convey that.
Under this fourth technique for Steps 1 and 2, the three expressions would be introduced by the evaluator, preferably with visuals (e.g., some nice belt and suspender photos are available online). Session participants would then be asked to think individually how each of the expressions might apply to the current program as planned. They would further consider what implications this has for the parts of the program that might need to be changed and for the kind of changes needed. Note that the three expressions conceptually cover the same territory as the majority of the questions as in Table 4. Those questions could even be used to supplement the old adages as needed. Group discussion would then proceed as in the other procedures.
Comparison of the four techniques for Steps 1 and 2
The advocacy of thoughtful planning of potential program adaptations is based in part on the premise that one size does not always fit all. Having more than one technique available for the initial steps of the adaptation planning process is also consistent with that idea. As noted in introducing the four techniques, they are offered as starting points, tactics that can be mixed and matched, and stimuli for thinking about better alternatives. This is true for the details of procedures offered for Stages 3 and 4 as well.
In selecting among Techniques 1–4, evaluators might consider the possibility that the kind of adaptations elicited may vary across the four techniques. In particular, the techniques might tend to elicit potential adaptations that vary in “size.” Adaptation size can involve two related dimensions: (1) operational scope, with some adaptations changing a big chunk of a program, such as one of three program components, and others modifying only a smaller aspect, such as a specific procedure within a program component; and (2) conceptual significance, with some adaptations being conceptually modest (e.g., replacing phone calls with emails) and others conceptually more substantial (e.g., the one exercise intended to enhance self-efficacy is replaced with a group activity that has another purpose).
Relative to the other techniques, the premortem might elicit suggestions about larger, more conceptually substantial adaptations. In contrast, more modest changes may be elicited by the relatively nuanced focus of most questions about the game plan. Using the old saws may be similar to questioning the game plan in terms of the type and size of adaptations elicited. Even smaller adaptations may be generated when the pump is primed with FRAME, which includes relatively mundane items, such as a portion of program content being omitted because of time constraints. These suggestions are hypotheses, subject to future testing, but they highlight the value of having multiple techniques available. When carrying out a program adaptation planning session, the evaluator might be well advised to start with one technique but be prepared to shift to another.
Step 3: Decision Rules
Regardless of which technique or combination of techniques is used for Steps 1 and 2, the adaptation planning process should turn next to discussion of decision rules. If session participants have agreed to a potential adaptation, what circumstances should trigger its initiation? In some cases, the shift can be triggered by failed assumptions about external conditions. For example, a school-based program might be predicated on the assumption that the program can be offered in 13 weekly lessons of at least 50 min. Short- or longer term school closings would then trigger the use of a planned adaptation that establishes priorities about the lessons or lesson portions to be given during the shortened program schedule.
Beyond external disruptions, the decision rule to initiate a planned adaptation will usually depend on some kind and degree of failure of the original procedures. This might involve an observed implementation failure, such as when recruitment processes result in an unacceptably low client count. In Story and colleagues’ (2012) report, the motivational phone calls were dropped after the second fun night due to difficulty in reaching the families. If participants in an adaptation planning process had considered the challenges of reaching people by phone, they might have planned a possible replacement such as using mailings, texts, or emails. In this case, the decision rule to move to the backup (i.e., the adaptation) might have been something like failure to reach 30% or more of the families on the first or second phone call.
Rather than an indicator of failed implementation (e.g., low recruitment numbers or response rates), an adaptation might alternatively be triggered by poor performance on an outcome of interest or a proxy. That is, the decision rule may be to implement the adaptation if participants on average fail to meet a specified threshold on a measure of interest. In the case of Story et al.’s (2012) evaluation, imagine that the program implementers did not have trouble reaching families by phone but instead that an interim measure of parent motivation remained low despite the calls. With an a priori adaptation planning process, there might have been a backup, that is, a planned adaptation in place of the calls such as (admittedly more labor-intensive) home visits. As another example, program participants might complete activities intended to improve their self-efficacy, but early measures of self-efficacy might show no change. In cases such as this, the specific cutoff point in the decision rule might even be informed by a meta-analysis or other extant evidence indicating how much change in self-efficacy can reasonably be expected.
Alternatively, the planning process could generate a procedural rule whereby the decision to implement an adaptation is based not on a prespecified cutoff on a specific measure but rather on a prespecified process. A procedural rule of this kind should indicate who is to be involved in the decision (e.g., the program manager, a staff representative, and a stakeholder representative), how the process is initiated (e.g., any of the decision-making parties can call for a decision, as can any implementing staff member), and what the decision rule is (e.g., majority rule). Having a procedural rule for initiating a priori planned adaptations may have positive side effects. That is, the use of the procedural rules might carry over when the unexpected occurs, and adaptations are called for that had not already been planned, and this carryover may result in more systematic and more thoughtful responses when responding to the unexpected.
Decision rules can also be designed for the possibility of contingently targeting an adaptation. While targeting can be built into the initial design of a program, it can alternatively be added as an adaptation that is initiated selectively in an ongoing program because of differential performance across subgroups. Suppose, for example, that the initial participant recruitment plan is working well in urban settings but poorly in rural ones or that a similar urban–rural difference is observed on a measure of parent motivation in a school-based intervention. The planned adaptation would then be implemented only in rural sites. As another example, initial program procedures might increase self-efficacy in boys but not girls, so the adaptation may be targeted to girls while boys continue with the original version of the program.
Step 4: Documenting and Perhaps Evaluating Planned Potential Adaptations
The need to document planned potential adaptations arises at two points in time. First, after the planning process, the set of potential adaptations should be documented. How best to do this will depend on how the program model itself is represented. For programs that are detailed in a manual or a curriculum, potential adaptations and their decision rules can be described in the relevant sections (e.g., the chapter presenting the lesson where the adaptation might occur). For a program model displayed in a logic model or theory of change, a few simple planned adaptations could be incorporated directly. For example, the activity “motivational family calls” could be extended to “motivational family calls (w/ emails as backup).” As the number of program activities and potential adaptations grows, however, it might become unwieldy to summarize the adaptations briefly in a visual display. If so, footnotes could give the specifics of potential adaptations.
Second, actual adaptations, that is, the adaptations that have been put into practice, should be documented. These could be acknowledged within a graphical representation, following suggestions by den Hoyer (2002), Funnell and Rogers (2011), and Mayne (2015). Narrative description of the circumstances surrounding an adaptation’s implementation, including the reason for implementing the adaptation and the level at which the change was made (e.g., program-wide or by an individual practitioner), would also be valuable. Such descriptions can aid in future revisions of the program model, in the design of similar programs, and in subsequent descriptive research on adaptations.
The descriptive information about adaptations is in a sense a kind of implementation assessment of the program’s adaptation plan. In addition, interviews with program implementers and others could ask about the adequacy of the adaptation plan, its ease of use, its perceived effectiveness, modifications that were needed but not included in the plan, flaws in the prespecified decision rules, and the like. For some adaptations, simple quasi-experimental comparisons might be possible for a tentative assessment of their effectiveness. For example, recruitment data over time might suggest whether a change in the recruitment process was beneficial. In the Discussion, I review other research questions related to planned adaptations.
Discussion
This article is based on the contention that evaluators will benefit from being familiar with a range of options for engaging with program modifications and adaptations. In service of that idea, several existing approaches, organized into three groupings, were reviewed. Evaluators dealing with actual or potential program modifications should be able to choose from and mix and match aspects of these alternative approaches. Evaluators may also consider the as-yet underrecognized approach described in the previous section of this article. In this approach, evaluators work with program developers, staff, and stakeholders to identify, in advance, potential program modifications that may need to be implemented in the future.
As described in the previous section, a priori planning of potential adaptations consists of four steps, whereby (1) program elements that may come to need adaptation are identified and, if needed, prioritized for consideration in the remaining steps, (2) potential adaptations for these program elements are identified, (3) decision rules are developed that would trigger a planned adaptation, and (4) planned adaptations are documented, as are those adaptations that are actually adapted, with the latter evaluated if possible. The previous section includes suggestions for each of these steps including four techniques that can be considered for carrying Steps 1 and 2 combined. These four are labeled the premortem, questioning the game plan, priming the pump, and using old saws (to build an adaptation plan).
Potential Benefits of Planning for Potential Adaptations
Many benefits may arise from this approach to program adaptations. First, if evaluators facilitate thoughtful review of program plans and of potential adaptations, programs may be more effective. Nutt (2002) studied the effectiveness of decisions made in a range of organizations and found that consideration of more than one alternative is a strong predictor of success. Second, the proposed planning process should reduce—though probably will not eliminate—the occurrence of unplanned adjustments that are made on an ad hoc basis in real time. This change also may have important implications as suggested by Hansen et al.’s (2013) findings that having fewer unplanned program modifications should increase program effectiveness. Third, especially when potential adaptations are planned with sensitivity to a program’s core components, the approach should make it easier to strike a successful balance between fidelity and adaptation. Put differently, the proposed approach offers a pathway toward fidelity with flexibility (Kendall & Frank, 2018). Fourth, adding preplanned adaptations that can respond to predictable challenges might contribute to program sustainment. 5 Scheirer’s (2005) review indicated that the ability to modify programs locally increased the likelihood of continuation. Whether having an a priori adaptation plan increases program sustainment would be an interesting question for future research.
More generally, a fifth benefit of the proposed adaptation process is that it opens, or at least highlights, several fruitful avenues for future research on evaluation. One involves descriptive research on how frequently various kinds of adaptations arise in program areas other than the behavioral health interventions that to date have been the domain for most such research (Miller-Day et al., 2013; Stirman et al., 2019). Research on evaluation could also compare programs with and without planned adaptations in terms of effectiveness, implementer satisfaction, rated fidelity, and sustainment. More specific research questions are also possible, such as the relative merits of different kinds of decision rules for implementing an adaptation.
A sixth benefit of the proposed adaptation planning process is that it would help to address shortcomings of logic models and related representations of program theories. Such models have frequently been criticized as overly fixed and linear (e.g., Funnell & Rogers, 2011; Savaya & Waysman, 2005). The inclusion of potential adaptations clearly adds flexibility to what have traditionally been relatively fixed plans. In addition, with decision rules that take into account implementation failure or poor early outcomes, one form of feedback loop is added to a logic model. Incorporation of planned adaptations will not address all criticisms of logic models as fixed and linear but will help.
Finally, the proposed increase in attention to planned program adaptations might have the side effect of helping to expand attention to the social programming component of evaluation theory. Shadish et al. (1991) indicated that a comprehensive evaluation theory attends to five components: knowledge construction, valuing, use, social programming, and practice. The social programming component addresses how programs operate, what the external forces are that constrain programs, and the role programs play in social change. If more attention is given to the thoughtful, a priori integration of program adaptations in program planning and evaluation, evaluators might broaden their view of the alternative forms that programs can take. Existing evaluation approaches fit well with the polar extremes of fixed interventions, on the one hand, and initiatives that are reformulated in an ongoing manner, on the other. There may be benefits of more explicitly considering a range of program types, including an intermediate form in which programs include planned adaptations.
Challenges and Responses to Them
In addition to potential benefits, the suggested adaptation planning process has costs as well as potential challenges. Costs include the time required of those who participate in the process; the possible perception that much of the time and energy was wasted, in that many planned adaptations will probably not need to be implemented; and the likelihood that the planning process will not foresee all the modifications that ultimately were needed. In response to these criticisms, one could counter that the evaluator–facilitator and planning session participants should be able to constrain and control the amount of time invested; that like insurance policies, one hopes not to need the planned adaptations but will be grateful to have them if needed; and that, even though additional unplanned adaptations are likely to be needed, the a priori adaptation planning process may both reduce how many of these arise and increase relevant skills among those who make those ad hoc modifications in real time.
As to the possible limits of adaptation planning because of humans’ limited ability to predict the future, as is the case more generally, our inability to foresee the future perfectly should not serve as an excuse not to do our best. If participants in the planning process raise concern about their ability to foresee the need for adaptations, it may be helpful to point out that people tend to be better at predicting the likelihood of broader categories of risks rather than predicting the likelihood of each specific risk within that category. Experts did not predict a pandemic due to the specific novel coronavirus that emerged in 2019. However, they strongly warned of the risk of an outbreak due to a novel virus in our lifetimes. Similarly, rather than anticipating a snowstorm will interrupt a school-based program on a particular day, it is safer to think about the risk that something or other will lead to school closures at some point during the course of a school-based program. Additionally, some reasons for program adaptations are easier to see in the crystal ball than are others. For example, the likelihood of at least limited school closures may be especially predictable in regions with wintry weather. It also may be a safe bet that some teachers will run out of time during class periods at least occasionally. And again, human’s inability to predict everything perfectly should not serve as an excuse for failing to try to predict what we can.
Michael Patton (personal communication 07/22/2020) has highlighted another class of possible challenge to adaptation planning, which includes organizations’ desire for “best practices” and the belief these should be adhered to strictly, funding practices that mandate (or are seen as mandating) strict fidelity, and accountability cultures that treat the undoing of plans as a failure. If one encounters resistance to a priori planning based on these considerations, a counterargument along the following lines can be offered. Considerable evidence indicates that program modifications will take place during implementation regardless (e.g., Hansen et al., 2013; Miller-Day et al., 2013; Stirman et al., 2019). Evidence also suggests that adaptations that are consistent with the program model do not reduce, and may increase, program effectiveness (Damschroder et al., 2009; Hansen et al., 2013; Kendall & Frank, 2018) and sustainment (Scheirer, 2005). And fidelity-consistent adaptations are more likely to occur with a thoughtful planning process that takes a program’s core functions into account rather than when modifications are made up on the fly under duress. Indeed, if future research shows that a priori adaptation planning is beneficial, it might come to be identified as a best practice—though, I would argue, one that should be open to future adaptations in its practice.
Pondering Adaptations in the Time of a Pandemic
As noted in the Introduction, the COVID-19 pandemic led to major adaptations in programs and evaluations as in so much else of life. Also as previously noted, the widespread occurrence of program modifications during the pandemic led Michael Patton to state that “All evaluators must now become developmental evaluators” (https://www.utilization-focusedevaluation.org/blog). Increased sensitivity to and thoughtfulness about program modifications seems like a good response to the pandemic on the part of evaluators. But in the long haul, evaluators may benefit most from having a set of alternative ways of thinking about and engaging with adaptations. We need to be careful not to be like the cat Mark Twain described, “The cat, having sat upon a hot stove lid, will not sit upon a hot stove lid again. But he won’t sit upon a cold stove lid, either.” The preferred approach to program adaptations may depend on the temperature of the stove, so to speak. The evolutionary theory concept of punctuated equilibrium, with its expectation that periods of relative stability recur after periods of crisis, has proven useful in other fields including public policy (Baumgartner & Jones, 2009) and the sociology of disasters (Vollmer, 2013). Similarly, programs and evaluation might shift to a period of quasi-equilibrium following the pandemic. To be clear, this position does not contradict Patton’s Spring 2020 declaration that all evaluators needed at that time to be developmental evaluators. He added that this need stemmed from “the current context around the world in general…[which] is the world in which evaluation will exist for the foreseeable future.” It was not a prediction for all evaluations for all time.
All this is not to say that a “new normal” will be identical to an old normal. In his thoughtful discussion of “postnormal evaluation,” Schwandt (2019) offers a list of considerations that might lead evaluation to continue shifting from its modernization-based “normal” form. Expanding on Schwandt’s notions of postnormal evaluation, one possibility is that it might include coproduction (with citizens) of programs, evaluations, and preplanned potential program adaptations. In any case, especially if punctuated equilibrium or something like it applies in the future, the field of evaluation is likely to benefit from having a portfolio of approaches from which to draw in dealing with program adaptations. Adaptation itself calls for the skill of flexibility. For evaluators, this should apply also to our ways of thinking about and engaging with program adaptations.
Footnotes
Acknowledgment
Thanks to Sebastian Lemire and three anonymous reviewers for helpful comments on a previous draft.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
