Implementers’ Perspectives on Fidelity of Implementation

Introduction

Implementation fidelity, the extent to which interventions are carried out as intended, is essential to realizing the potential benefits of evidence-based interventions (EBIs; Elliott & Mihalic, 2004). As promising programs move from efficacy tests to effectiveness trials to dissemination, fidelity of implementation is key to ensuring that the intervention continues to deliver its intended effects (Elliott & Mihalic, 2004; Glasgow, Lichtenstein, & Marcus, 2003; Mrazek & Haggerty, 1994). Because fidelity may moderate the relationship between an EBI and its intended outcomes, assessing fidelity of implementation is essential to interpreting data on an EBI’s effectiveness (Dane & Schneider, 1998; Durlak & DuPre, 2008; Dusenbury, Brannigan, Falco, & Hansen, 2003; Elliott & Mihalic, 2004) and reducing the chances of committing a Type III error—that is, an erroneous conclusion about an EBI’s effectiveness, or lack thereof, because the program was not adequately implemented (Basch, Sliepcevich, Gold, Duncan, & Kolbe, 1985). Evidence reviews in diverse fields support the relationship between fidelity and other implementation factors, and program outcomes (Durlak & DuPre, 2008). Despite long-standing recognition of the importance of fidelity, however, considerable diversity surrounds its definition, interpretation, and application in practice.

Implementation scientists have devoted considerable thought to defining fidelity. Dane and Schneider (1998), using the term program integrity, identified five components of fidelity most often measured within 162 studies of primary and early secondary prevention programs for youth. Adherence is defined as the extent to which the implementer delivers the curriculum or program material as prescribed by program manuals. Exposure is the amount of the program that is received by participants, reflecting the number and length of sessions received or frequency of attendance. Quality of delivery is defined as how well an implementer delivers material in alignment with the theoretical ideals of the intervention. Program differentiation defines the unique features of the intervention that are essential to achieving intended outcomes. Finally, participant responsiveness is defined as how engaged and involved participants are in the activities and content of the program. Recognizing the unique contribution of each dimension, the authors support measuring all of them.

Recent implementation scientists have defined fidelity itself more narrowly and placed it in the context of other implementation factors (Berkel, Mauricio, Schoenfelder, & Sandler, 2011; Carroll et al., 2007; Durlak & DuPre, 2008). In these conceptualizations, fidelity consists of adherence and includes content, frequency, coverage, and duration. Implementation factors that may moderate the relationship between adherence and intended outcomes include quality of delivery, participant responsiveness, intervention complexity, implementer facilitation strategies (Carroll et al., 2007), and monitoring, reach, and adaptation (Durlak & DuPre, 2008).

In addition to definitional inconsistencies, perspectives vary on the degree to which fidelity is desirable. Elliott and Mihalic (2004) argue that fidelity to an EBI should be strictly maintained in order to achieve desired outcomes and are skeptical of deviations from fidelity via local adaptations. Others have advocated a focus on “core elements” or “critical components” of the underlying theoretical logic of an EBI (Backer, 2001; Bauman, Stein, & Ireys, 1991; Galbraith et al., 2011). Prominent among those advocating a more flexible application of fidelity are those arguing that an EBI must respond to communities’ cultural needs in order to elicit participant response (Castro, Barrera, & Martinez, 2004; Lee, Altschul, & Mowbray, 2008; Wingood & DiClemente, 2008).

At the practice level, even implementers committed to implementation with fidelity may encounter confusion. Intervention developers, funders, trainers, and technical assistance providers vary in their guidance to program implementers regarding fidelity (R. L. Miller & Shinn, 2005). Depending on the EBI, curriculum manuals and training materials may present fidelity in terms of rules for adherence and dosage (e.g., this activity may be omitted; names in a role-play may be changed) but less often include guidelines for quality of delivery or participant responsiveness (e.g., how to increase student comprehension; Dane & Schneider, 1998).

Given the crucial role of implementers in delivering EBIs, considerable research has examined relationships between implementers’ characteristics and fidelity of implementation. Implementer characteristics that have been studied include years of experience, confidence in teaching, attitudes toward fidelity, and specific assessment of the EBI in question (Hill, Maucione, & Hood, 2007; Morrison et al., 2009; Ringwalt et al., 2003). However, studies have yet to examine implementers’ understanding of fidelity and its relation to their implementation practice.

The current research focuses on implementers’ understanding of fidelity and its relationship to their implementation behavior. We use qualitative interview data to describe implementers’ perspectives on fidelity, and examine definitions of fidelity vis-à-vis reported implementation behavior. Based on these findings, we suggest strategies for supporting implementers in understanding fidelity and applying it in their implementation of EBIs.

Current Study

In sum, within the field of implementation science, implementation is proposed as the link between evidence-based practices and positive outcomes (Odum, 2009), whereas implementation fidelity is the key source of variation affecting the credibility and utility of research evaluating said outcomes (Carroll et al., 2007). However, among researchers and practitioners within the field, considerable discrepancies and tensions exist pertaining to how implementation fidelity should be defined and operationalized, and whether it is even desirable. Such divergences hamper the field’s ability to establish a unifying set of criteria by which to gauge the credibility of intervention execution and research (Proctor et al., 2009). In this emerging field of implementation science, it is necessary to create “a common lexicon . . . of implementation . . . terminology” and meaning (National Cancer Institute, 2004). This study seeks to advance the field of implementation science by increasing understanding of existing variations in terminology surrounding implementation fidelity among implementers, including implementers’ enactment of implementation fidelity, and by offering recommendations toward a situated conceptualization of implementation fidelity based on study findings.

Method

Procedures and Measures

We distributed an informational memo to coordinators of the seven Start Strong sites, inviting all implementers to participate in key informant interviews. Interviews were conducted in spring 2012, either in person (15) or by telephone (5). Two study team members conducted the interviews, one leading the interviews and one taking near-verbatim notes. With participants’ permission, all interviews were audiotaped as a backup to electronic notes. Before each interview, members of the research team obtained informed consent from participants. Interviews lasted roughly 1 hour. Participants who only taught the curriculum received a $40 gift card for completing the interview; those who were site coordinators did not receive an incentive because participating in evaluation-related activities was considered part of their job. All study procedures were approved by the authors’ institutional review board.

We used a semistructured interview guide to explore implementers’ implementation experiences and beliefs related to fidelity. The interview guide was developed by the study team based on conversations with sites during the Start Strong evaluation that suggested the extent of implementer-initiated adaptation. Introductory questions explored the respondents’ background and experience with the Safe Dates curriculum. A second series of questions asked about changes that the implementers made to any aspect of the curriculum, using the word “adjustments” as a more neutral term than “changes” or “adaptations.” Finally, we asked about participants’ beliefs about fidelity and how they defined fidelity. These questions are shown in Table 1. Questions about fidelity were deliberately introduced following discussions of adjustments to reduce possible reluctance to acknowledge adjustments.

OPEN IN VIEWER

Table 1

Examples of Interview Questions

● Have you made adjustments to how you teach the curriculum? These may be changes needed to fit students’ needs, the school setting, or your own experience and background.

● Were any parts of the curriculum left out of actual implementation?

● Was anything added to the curriculum?

● Was anything changed from how it was described in the manual?

● What have you been told about fidelity to the curriculum? By whom?

● To what extent do you feel that fidelity helps, or interferes with, achieving the goals of Safe Dates? How?

Results

Implementers’ Definitions of Fidelity

Implementers provided four working definitions of fidelity. These were typically framed operationally, in terms of what kinds of changes, if any, were permissible, and why. We have summarized these perspectives in Table 2, presented in descending order of the frequency with which they were offered. The necessity of adjusting to time and logistical constraints was implicit in many implementers’ definitions.

OPEN IN VIEWER

Table 2

Summary of Implementers’ Definitions of Fidelity

Perspective	Definition
Strict Adherence	Everything in the curriculum should be taught as written, without additions, deletions, or modifications in delivery.
True to the Material	Material should be covered in a way that meets the underlying goals of the curriculum and the needs of the students.
Flexible Pedagogy	Everything in the curriculum should be taught, but the implementer can make changes in how it is taught in order to adapt to students’ needs.
Additions Only	Everything in the curriculum should be taught as written, but the implementer can add material to make the lesson more appealing, add depth, or cover additional topics.

The most commonly reported definition of fidelity was one that we have categorized as “Strict Adherence.” This represents the most literal interpretation of fidelity, described as, “Teach every single part, use every single handout” and “Make no changes.” The “Strict Adherence” approach emphasized both covering all curriculum content and delivering material as described in the Safe Dates manual. Implementers espousing this approach were the most likely to describe the importance of fidelity in terms of its contribution to achieving the outcomes reported elsewhere.

Perhaps the most intriguing definition of fidelity was the one that we have categorized as “True to the Material.” This approach was described as doing what was necessary to meet the goals of the curriculum, “Not changing the core messages . . . thinking about the goal of the lesson.” These implementers reported that “Fidelity is to the information, not to how it was delivered.” In one implementer’s words, fidelity consisted of “be[ing] right with the curriculum.” Those ascribing to this definition were all implementers with prior experience teaching TDV prevention, who felt confident in their ability to discern core messages and identify the best way of meeting a lesson’s objectives vis-à-vis their context and participants.

For the definition we have categorized as “Flexible Pedagogy,” implementers reported, “Changing the content is not okay, but changing how it is delivered is okay.” They noted that changes to delivery were appropriate and even necessary if doing so could make the lesson more appropriate to students’ literacy levels, social maturity, or learning styles. The extent of adjustments described within this approach ranged from “tweaking the delivery” to entirely changing an activity’s format. One site coordinator noted that it was necessary to “restructure the activities that are not working,” and that guidance on how to make them would have been helpful.

Finally, only one implementer defined fidelity in a form we have categorized as “Additions Allowed.” This approach was defined as “We need to do everything, not cut anything out,” with the provision that additions were not only allowable but also important. Additions to the curriculum were described as consistent with fidelity when they made the material more relevant to students’ lives or improved the quality of program delivery, as assessed by students’ interest in the lesson.

Implementation Practice

Although all implementers described themselves as having implemented with fidelity, all reported having made adaptations to the curriculum. Implementers’ reasons for making these adaptations, and the process by which they made them, are reported in detail elsewhere (Gibbs, Krieger, Cutbush, Clinton-Sherrod, & Miller, 2016). Adaptations described by implementers included changes to curriculum delivery, additions to the curriculum, and omissions from the curriculum.

Implementers identified student engagement as the predominant motivation for adaptation and as an ongoing source of feedback on whether their adaptations were increasing the fit between the curriculum and students’ literacy levels, learning style, social maturity, and culture. Changes to curriculum delivery included having implementers act out role-plays rather than students, presenting quizzes in a game show format, and reading material aloud rather than asking students to read silently. Additions included material based on relevant news stories involving popular entertainers, movies or music, or anecdotes about young people similar to the students.

Less frequent but more substantive adaptations were made based on implementers’ response to the curriculum’s handling of a specific issue, or to logistical constraints. These consisted of omissions from the curriculum, such as removal of an activity because of concerns by implementers about unintended messages conveyed to students (e.g., perpetuating rather than disproving myths related to sexual assault), or due to the lack of resources required for implementation (e.g., space to stage a play illustrating curriculum concepts).

Reported adaptations were frequently different from, and more extensive than, what would be suggested by implementers’ definitions, as shown in Figure 1. Among the implementers who defined fidelity as “strict adherence,” each made changes in practice. These changes included modifying delivery, adding material, and deleting material. Those who defined fidelity as “being true to the material” were most likely to add material or change curriculum delivery based on their experience and familiarity with other curricula. Implementers who defined fidelity as allowing changes to delivery did so, and they frequently added material as well. Those who defined fidelity as allowing added material also reported making changes to delivery.

OPEN IN VIEWER

Figure 1

Implementers’ Definitions of Fidelity Compared to Reported Practice

Discussion

The field of implementation science has generated a substantial body of literature that considers fidelity in terms of its definitions, components, and value (Backer, 2001; Bauman et al., 1991; Bishop et al., 2014; Blakely et al., 1987; Durlak & DuPre, 2008; Dusenbury et al., 2003). Numerous approaches have been developed for measuring fidelity of implementation (Bishop et al., 2014; Century, Rudnick, & Freeman, 2010; Hill et al., 2007; Mowbray, Holter, Gregory, & Bybee, 2003) and assessing implementers characteristics associated with implementation fidelity (Ennett et al., 2011; Hill et al., 2007; Morrison et al., 2009; Ringwalt et al., 2003). However, despite implementers’ central role in the delivery of EBIs, their perspectives on fidelity are rarely considered. Our findings describe implementers who embraced the value of fidelity but varied substantially in their interpretation of the construct and in its reported application.

Most implementers in this study defined fidelity from an adherence perspective, with some variations among these. The most commonly offered definition considered fidelity to require adherence to delivering the curriculum in precisely the manner described, without additions or deletions. Others combined adherence to curriculum content with adaptation of pedagogy or addition of content selected by the implementer. An alternative perspective was that of implementers who defined fidelity in terms of program theory rather than adherence, inferring a relationship between core messages and achievement of curriculum goals.

Each of the implementers reported that they had implemented the curriculum with fidelity. At the same time, all reported having made adaptations to the curriculum. Even those espousing strict adherence reported having modified pedagogy, added content, or omitted content. Some of the adaptations described may have been allowable based on guidance from the curriculum’s publisher (“Safe Dates Fidelity Checklist,” n.d.); others might not. Taken together, these findings depict implementers who are committed to fidelity but in need of additional guidance on its implementation.

The experience of these implementers suggests the potential value of strategies by which developers can support dissemination into practice settings, as outlined by Freire, Perkinson, Morrel-Samuels, and Zimmerman (2015). Briefly stated, the authors recommend communicating program theory in terms that will be understood by implementers, clarifying expectations for fidelity and permissible adaptations, and codifying what has been learned from implementation.

To better communicate program theory, developers need to open the “black box” and identify core components considered essential to integrity of implementation (Bauman et al., 1991; Morrison et al., 2009). Identifying core components and explicating their relationship to desired outcomes is also a priority for training and ongoing coaching, essential drivers of implementation (Fixsen, Blase, Naoom, & Wallace, 2009). Instead, implementers in this study described numerous situations in which they relied on their own sense of what was essential to the EBI. Not surprisingly, adherence has been shown to be improved when implementers can identify the EBI’s core messages and mechanisms, and how these are believed to contribute to outcomes (Hawe, Shiell, & Riley, 2004).

To improve clarity regarding fidelity, program sponsors and technical assistants need to explicitly define acceptable and unacceptable adaptations. Planned adaptation process, typically involving developers and other experts, are likely to produce modifications that are consistent with developers’ intent (Aarons et al., 2012; Lee et al., 2008; Wingood & DiClemente, 2008). However, researchers have documented the near-universal practice of implementer-initiated adaptations, which may or may not be positive in their impact (Hansen et al., 2013; Lundgren, Amodeo, Cohen, Chassler, & Horowitz, 2011; Morrison et al., 2009; Ringwalt et al., 2003). When implementation circumstances diverge from conditions under which EBIs were tested—as they inevitably will—better information on allowable adaptations would support implementers in responding in ways that are consistent with program theory. This function is a central one within the Prevention Support System (Wandersman et al., 2008), in which intervention-specific capacity building can address potential threats to implementation fidelity.

Finally, developers, sponsors, and technical assistants need to codify implementation lessons to provide implementers with practical strategies for responding to common challenges. Documentation of adaptations made would support a “living” curriculum that is refined in response to real-world challenges (Peters & Wexler, 2005). Importantly, compiling information on adaptations that are consistent with fidelity could be transferred to situations in which implementers may receive less training and support than what is provided when fidelity is monitored and outcomes evaluated.

Several limitations of this study must be acknowledged. First, respondents consisted of 20 participants in 7 sites using a single EBI. This is an acknowledged limitation in that implementers in other settings or using materials from other EBIs might report different interpretations fidelity or different choices with respect to adaptation. At the same time, it is notable that even within this constrained sphere a substantial diversity of interpretations were identified. In addition, implementers’ reported views of fidelity may have been affected by social desirability bias, as a result of having been instructed to implement with fidelity. Finally, descriptions of implementers’ adaptations were based on self-report rather than observation. We did not attempt a comprehensive listing of all adaptations made, and implementers may have avoided reporting adaptations that indicated drift or incomplete implementation. Taken together, these limitations suggest that our findings point to areas worthy of further investigation, rather than generalizable statements about implementation. Larger and more systematic assessment of implementers’ understanding of fidelity in relation to adaptations they made could test our conclusions. Nevertheless, these modest findings support implications for practice that have been proposed elsewhere.

Developers, funders, and researchers are all needed in the process of supporting implementers in understanding and practicing high levels of fidelity. Developers need to share their EBIs’ theory of change and theoretically important core components. Funders need to balance their focus on implementer accountability against the value of judicious flexibility in implementation. Although program drift and implementation failure are valid concerns, check-the-box fidelity measures are unlikely to effectively deter them. Evaluation research could report on adaptations made in terms of positive and negative valence rather than as deviations from fidelity (Durlak & DuPre, 2008; Glasgow et al., 2006). Such changes could take full advantage of the “practice wisdom” that experienced and committed practitioners bring to their work (Dreyfus, 2004), supporting implementation that is truly “right with the curriculum.”