An Assessment of Intervention Fidelity in Published Social Work Intervention Research Studies

Method

Results

A total of N = 32 (2.3%) published articles met all six of these inclusion criteria. These studies included experimental, preexperimental, and minimally quasi-experimental designs. Ranked in descending order: (a) Research on Social Work Practice accounted for 68.75% of the studies analyzed, (b) Social Work Research contributed 12.5%, (c) Social Work and Journal of Social Service Research each contributed 6.25%, and (d) the British of Journal of Social Work and Social Service Review had 3.13% each.

Sample sizes ranged from N = 40 to 974. The number of specified outcomes in the studies ranged from one to seven and included outcomes related to child-rearing practices in a parenting, substance abuse treatment, and sexual risk behaviors for HIV, to name a few. Various statistical methods were used to assess change statistics typically including analysis of variance, multivariate analysis of variance, analysis of covariate, hierarchy linear model, paired sample t-tests, χ² tests, and effect sizes. Table 1 shows the frequency of selected fidelity criteria among the 32 reviewed articles in ranked order. Of the 32 studies examined, 81.3% offered either a partial or full description of intervention.

OPEN IN VIEWER

Table 1.

Frequency of Selected Criteria to Assess Treatment Fidelity in Six Empirical Journals.

Selected Intervention Fidelity Criteria	Yes (%)	No (%)
1. Description of intervention	26 (81.3)	6 (18.7)
2. Use of treatment fidelity term	21 (65.6)	11 (34.4)
3. Manuals/protocols mentioned	20 (62.5)	21 (37.5)
4. Training offered	19 (59.4)	13 (40.6)
5. Use of manuals/protocols	16 (50.0)	16 (50.0)
6. Strategies used to monitor intervention delivery	16 (50.0)	16 (50.0)
7. Supervision offered	13 (40.6)	19 (59.4)
8. Discussion of the tenets of treatment fidelity without using term	3 (9.4)	29 (90.6)

Note. N = 32.

The mention of manuals for the interventions ranked second (62.5%) in the reviewed articles; however, 50% of them stated they actually used the manual in training, supervision, or throughout intervention’s implementation. These articles explicitly indicated that the intervention is both manualized and the manual is being used during interventions by practitioners in order to minimize personal styles to affect results (e.g., see Hall et al., 2009; Kayser, Feldman, Borstelmann, & Daniels, 2010; Striley, Nattala, Abdallah, Dennis, & Cottler, 2013).

In our sample, 59.4% offered training and 40.6% offered supervision to practitioners. A much lower percentage (37.5%) provided both training and supervision.

More than half (65.6%) of the studies reviewed actually used the term intervention fidelity term (or a similar term, such as treatment integrity or adherence). A much smaller proportion (9.4%) of the articles discussed tenets without using these terms. For example, Jones (2009) mentioned that in order to maintain consistency, the principal investigator co-facilitated all group interventions; however, they did not specifically mention fidelity as the reason to ensure consistency.

Finally, 16 out of the 32 articles reviewed utilized strategies to monitor consistency of intervention delivery. The strategies used included using a fidelity checklist, monitoring by an outside research team or audiotapes, and completing standardized fidelity measurements. The most frequently used strategy was quantitatively assessing the intervention using a fidelity checklist by either practitioners or outside members. Some studies reported that intervention practitioners completed the treatment fidelity checklist after each session and discussed it (Conner et al., 2011; Langhorst, Choi, Keyser-Marcus, & Svikis, 2012; Leitch, Vanslyke, & Allen, 2009; Leung et al., 2009), and others reported the intervention was monitored by outside specialists and researchers using a fidelity checklist (Coulter, 2011; Jacobson, Osteen, Sharpe, & Pastoor, 2012; Kayser et al., 2010; Painter, 2009; Timmer et al., 2010; Valdez, Cepeda, Parrish, Horowitz, & Kaplan, 2013). Painter (2009) used standardized scales, Therapist Adherence Measure and Supervisor Adherence Measure, to ensure consistency of delivery intervention. Although quantitative assessments were the most utilized strategy to assess the intervention, few studies reported their results. For example, after conducting a fidelity analysis, Painter (2009) and Parish, Rose, Luken, Swaine, and O’Hare (2012) noted the varying levels to which their respective studies adhered to fidelity.

The top five ranked articles appear in Table 2. All of the listed articles addressed all eight of the selected fidelity criteria.

OPEN IN VIEWER

Table 2.

Top Five Ranked Articles of Eight Selected Fidelity Intervention Criteria in Six Empirical Social Work Journals.

Authors (Years)	Title and Journal Source
Zajicek-Farber (2010)	Building Practice Evidence for Parent Mentoring Home Visiting in Early Childhood, RSWP
Conner and Fraser (2011)	Preschool Social–Emotional Skills Training: A Controlled Pilot Test of the Making Choices and Strong Families Programs, RSWP
Kayser, Feldman, Borstelmann, and Daniels (2010)	Effects of a Randomized Couple-based Intervention on Quality of Life of Breast Cancer Patients and Their Partners, SWR
Lanier et al. (2011)	Parent–Child Interaction Therapy in a Community Setting: Examining Outcomes, Attrition, and Treatment Setting, RSWP
Painter (2009)	Multisystemic Therapy as Community-Based Treatment for Youth With Severe Emotional Disturbance, RSWP

Note. N = 32. RSWP* = Research on Social Work Practice; SWR* = Social Work Research.

Discussion

Adhering to fidelity protocols assists researchers from simply knowing if the intervention worked, to understanding the nuanced particulars (i.e., who, what, when, where, and why) of the intervention’s implementation (Century, Rudnick, & Freeman, 2010). As social work professionals continue to embrace evidence-informed practice, this study’s findings suggested that intervention fidelity is still an area in social work research that requires attention. This research extends previous work in this area by providing a more recent assessment of published social work intervention studies.

Overall, the results from this study report slight improvements in attention to fidelity in social work research. Out of all 8 fidelity items examined in this assessment, studies reported a “description of the intervention” most frequently. Although the majority of the studies (81.3%) did provide description of the intervention, the amount of detail given varied. As H. T. Chen and Rossi (1983) noted, more specificity about the intervention allows for a better understanding and interpretation of its effects. The articles that did provide a description of the intervention only offered scant detail, making it nearly impossible to interpret the intervention’s effects (Summerfelt, 2003). Very few articles went as far to give a weekly detailed description of the intervention. Kayser, Feldman, Borstelmann, and Daniels (2010; see Table 2) and Shin (2009; see Table 1) studies offered clear and well-stated examples about how to provide more clear and concise treatment details using a table. Providing ample details not only assists researchers determine what was actually tested and how the intervention’s components contributed to the outcomes but also assists in replicating the intervention (Maynard, Peters, Vaughn, & Sarteschi, 2013). Social work researchers should strive to include as much detail possible about the intervention in their studies. Doing so helps ensure that the future implementation of the intervention is conducted as effectively and successfully as possible (Summerfelt, 2003).

Training personnel responsible for implementing the intervention was mentioned in 59.4% of studies examined. Adequately training providers to deliver interventions is a critical component in intervention fidelity, as the implementers’ level of education, competence, and skills directly effects if the intervention is being executed as was initially planned (Bellg et al., 2004). Ideally, implementer training, use of manuals, and supervision work synchronically to ensure the intervention is being delivered according to uniform protocols (Naleppa & Cagle, 2010; Waltz, Addis, Koerner, & Jacobson, 1993). Interestingly, however, only 50% of these studies reported an explicit description of how the manual was used. As noted by other authors, the extent to which a treatment manual outlines the study’s purpose, goals, session objectives, instructions for conduct, guidelines for specific problems, and other critical components contributes to the enrichment of provider training and standardization of training procedures (Bellg et al., 2004; Moncher & Prinz, 1991). While 62.5% of the assessed studies mentioned manuals, they did not specify if (or how) the manual was actually used in the treatment’s interventions delivery. It is important that social work researchers report as many nuanced aspects of how the intervention was approached, since variations in manual use can alter the effectiveness of the intervention.

The majority of articles (59.4%) in this assessment did not report supervision of treatment providers. Supervision of providers helps facilitate compliance and adherence to the treatment and reduces deviation (drift) from the original model (Bellg et al., 2004). As Perepletchikova and Kazdin (2005) acknowledged, intervention training should be supplemented with ongoing supervision to ensure its treatment consistency. An intervention is likely to be more successful when trainers are provided with regular feedback, support, and encouragement during implementation (Fixsen et al., 2005; Perepletchikova & Kazdin, 2005). As indicated in the literature, in order to ensure adherence to treatment fidelity accurately, it is critical to provide both training and supervision. The extent to which training and supervision was offered varied considerably, as few authors mentioned detailed context, hours, and times of training and supervision (e.g., see Hall et al., 2009).

A “description of the methods used” to monitor the delivery and progress of treatment was reported in 50% of these articles. The assessed studies used a variety of techniques to address fidelity to the intervention, such as fidelity checklists (or treatment integrity forms) adherence to the treatment manual, quality assurance checks, and review of audio-taped sessions, to name a few. Moncher and Prinz (1991) suggested utilizing outside sources, participant reports, and implementer reports as primary steps for verifying fidelity. As Gearing et al. (2011) noted, intervention monitoring is the “heart” of fidelity, because it involves measuring treatment adherence throughout the course of the intervention. Thus, it is important for researchers to give considerable attention to the specific techniques they will use to help ensure that the treatments are being administered consistently and appropriately.

We also assessed how many articles actually used the term “treatment fidelity” (or a similar term) in their methods section. It was found that 65.6% of the studies used the precise term “treatment fidelity” (or a similar term). This finding was an improvement from Tucker and Blythe’s (2008) study, which noted 10% of their N = 128 studies actually used the term “fidelity.” Similar to their study, we also assessed if the tenets of fidelity were mentioned without specifically using the term fidelity. Here, 9.4% discussed these tenets without specifically using the term. Not using the term treatment fidelity (or its synonyms) may suggest that social work researchers are possibly unaware of how paramount fidelity is to the reliability and validity of their study. Consideration of treatment fidelity prior to the implementation of the study should direct researchers to explicitly include the term in their study. We recommend social work researchers include in their study a subsection of their methods content specifically designated to identifying treatment fidelity. As an example, Conner and Fraser (2011) provides a “treatment fidelity” subsection in their article that clearly reports the methods they used for fidelity. Directly addressing this may assist researchers, practitioners, and educators pinpoint the actual methods employed by the researchers to ensure the intervention was delivered as initially planned.

In addition to attending to the aforementioned fidelity protocols, we suggest that social work researchers pretest the intervention with a subset of the study’s sample of participants, to ensure that the intervention is appropriate for their study’s population. Social work interventions are often designed for use with “vulnerable” populations and should aim to “preserve the dignity, worth, and uniqueness of the individual” (Naleppa & Cagle, 2010, p. 679). Surprisingly, none of the studies in this assessment reported pretesting their interventions on a sample before implementation. While the intervention may have internal reliability and validity without pretesting, one cannot generalize the effects of the intervention from one population to another. As indicated by Wodarski (2011), one of our so-called social research conundrums is applying homogenous interventions to heterogeneous populations. Administering a pretest to a similar sample prior to intervention administration may also assist researchers refine their intervention to meet the needs of that specific group.

Additionally, among this data set, we assumed that 100% of the studies would mention procedures relate to ethical issues and concerns. Surprisingly however, only 75% of the reviewed articles mentioned ethics. Institutional Review Board (IRB) approval, receiving informed consent/assent from participants, or other ethical considerations is “standard operating procedures” for most studies with human subjects and should be reported in the study. We were concerned to find that 25% of the studies were published without providing these assurances, although they actually may have done so, but did not mentioned them.

The intent of this study was to assess how social work intervention research studies noted issues related to intervention fidelity. Most important is to encourage the social work research community to treat intervention fidelity more seriously than it has. While our study does suggest the need for more attention to fidelity in social work research, the data may indicate the underreporting of fidelity measures by authors. In that light, we urge all social work journals to mandate this as a necessary requirement for publication. We conclude by offering a suggested authors’ checklist Table 3, to assist the social work authors in attending to fidelity in their research endeavors. Establishing fidelity protocols should be outlined with as much specificity as possible before intervention delivery, and we hope that using the checklist helps make that process easier.

OPEN IN VIEWER

Table 3.

Recommended Authors’ Brief Checklist for Assessing Treatment Fidelity.

The following guidelines are presented as a recommended checklist of 8 items for assessing intervention fidelity. Items answered with a “No” warrant authors carefully reviewing of intervention fidelity of their treatments
	Yes	No
1. Was the intervention pretested on a subset of the sample population?	□	□
2. Was ethics mentioned?	□	□
3. Were treatment manuals or protocols used?	□	□
4. Was training for practitioners offered?	□	□
5. Was supervision for practitioners offered?	□	□
6. Was a detailed description of intervention provided?	□	□
7. Was the term intervention fidelity used?	□	□
8. Were specific details provided about how intervention fidelity was monitored?	□	□