Support Systems for Treatment Integrity

Abstract

Objective:

This systematic review evaluates the content of effective support provided to practitioners of evidence-based interventions in order to establish and maintain treatment integrity.

Method:

Four articles covering six outcome studies are included in this review, these studies (1) adequately operationalized treatment integrity procedures and (2) showed sufficient effects of the support on levels of treatment integrity.

Results:

The results show that an effective support system consists of a minimum combination of training and ongoing support (supervision, consultation, and coaching), preferably extended with booster sessions.

Conclusions:

The review provides specific information on effective content of this support which practitioners and intervention developers can use in order to establish and maintain treatment integrity of their planned interventions.

Keywords

Support systems evidence-based interventions treatment integrity

Over the last decade, much attention has been paid to the development and implementation of evidence-based intervention programs for troubled youths. One of the main difficulties with these evidence-based interventions are the disappointing treatment outcomes outside the research setting. Researchers frequently conclude that low treatment fidelity may be the cause of disappointing results of these interventions in the practice field (Schoenwald, Chapman, Sheidow, & Carter, 2009; Sexton & Turner, 2010; Tennyson, 2009). Treatment fidelity or integrity refers to the degree to which the practitioner utilizes an interventions prescribed procedures, avoids proscribed procedures (adherence), and proves the right level of skills and clinical acumen (competence) in delivering the components of the treatment (Barber, Triffelman, & Marmar, 2007; Perepletchikova, Treat, & Kazdin, 2007).

Research suggests that frequent and targeted support of practitioners is an effective way to establish and maintain treatment integrity (Kerby, 2006; Mikolajczak, Stals, Fleuren, Wilde, & Paulussen, 2009; Schoenwald et al., 2009). Most evidence-based interventions therefore have incorporated specific demands on the support for practitioners that carry out the intervention. The support systems of these evidence-based interventions, however, differ from each other. Specific knowledge as to what should be the content of a support system or on the standard minimum rules for effective support is lacking. As Beidas and Kendall (2010, p. 26) conclude in their review on the training of practitioners in evidence-based practice, “Despite the importance of EBP, we know less than preferred regarding how to best train therapists in EBP.”

Support Systems

What do we know about (effective elements in) support systems that contribute to establishing and maintaining treatment integrity? Literature suggests a short training of practitioners is not sufficient and it takes ongoing supervision, clinical consultation, and support over an extended period of time to allow for adherent and competent delivery of an intervention (Addis, 2002; Connor-Smith & Weisz, 2003; Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). Research on the optimal focus of the training suggests that it is preferable to focus on intervention principles, rather than teaching the details of manuals (Abramowitz, 2006; Hayes, 2002; Miller, Yahne, Moyers, Martinez, & Pirritano, 2004). When it comes to ongoing support as in supervision, consultation, or coaching, literature suggests that a focus on a practitioners own functioning is the most effective way to enable them to deliver the intervention (Berger & Kleine, 2013; Voogden & Kuyvenhoven, 2010).

In order to gain skills, the training and support of practitioners should consist of direct, active learning methods (Cross, Matthieu, Cerel, & Knox, 2007) as research on effective learning methods suggests that the sole use of indirect, passive, learning methods has limited effects on adapting behavior (El-Tannir, 2002). Distinctive for direct learning methods are the opportunities for practice and feedback, mostly acquired through modelling and role-play (Beidas & Kendall, 2010).

The above described provides a general indication of effective (elements of) support systems for practitioners. The main goal of this review is to provide more specific information on the content of support and provide practical recommendations for intervention developers in order to establish and maintain treatment integrity of their planned interventions. This will be done by selecting and analysing studies that properly measure treatment integrity and show a positive and sufficient effect of the support on levels of treatment integrity. We have formulated three questions:
What type of support is provided in studies with high levels of treatment integrity?

What is the focus of the support and which learning methods are applied?

What are the characteristics of effective supporters?

Method

Literature Search Procedures

To identify studies for this review, we searched the following databases:
Academic Search Premier

ERIC

PsychINFO

Web of Science (social sciences citation index)

The search terms used are (adherence OR competence OR integrity OR fidelity OR dissemination OR implementation OR efficacy) AND (evidence-based practice OR EBP OR research-based OR empirically supported) AND (training OR workshop OR supervision OR education OR feedback) All databases are searched on abstract level.

Inclusion Criteria

Included studies:
evaluate the effects of a support system for practitioners in an evidence-based intervention on dependent measure(s).
◦ An intervention is considered evidence based when it uses replicable procedures (codified in manuals) and has clear empirical support demonstrated in a randomized controlled trial (RCT; Kazdin & Weisz, 2003).
◦ Dependent measures are self-reported and/or independently rated variables measure of adherence and competence¹

are primary studies,

are published in English, Dutch, or German language, and

are published between 2009 and February 2013. Herewith following up on the research of Beidas and Kendall (2010) that included articles published between 1990 and 2008.

Exclusion Criteria

Studies are excluded if they:
have a purpose other than the evaluation of the effects of a support system of practitioners in an evidence-based intervention on dependent measure(s),
◦ Including examination of mediators or moderators of therapeutic process, risk factors, cost effectiveness of the intervention, barriers to treatment implementation, and characteristics of sample and treatment setting.

evaluated interventions that are not delivered by treatment agents (self-help therapies), and

evaluated pharmacological interventions only.

Study Selection

Study selection has been done in a three-step decision-making process; first step was the selection of studies that showed a positive relationship between the support system provided and the measurement of treatment integrity. This selection resulted in 27 articles covering 44 outcome studies on support conditions.

Second step was the rating of the 27 articles on the extent to which treatment integrity was operationalized in a proper way. To judge this, the ITIPS-A (Goense, 2013) was used. The ITIPS-A is an adapted version of the ITIPS developed by Perepletchikova (2006), for details on the adaptations made, see Goense, Boendermaker, van Yperen, Stams, and van Laar (2014). The ITIPS-A consists of 22 items, covering the domains of establishment, assessment, evaluation, and reporting of treatment integrity in outcome studies. Each item is rated on a 4-point scale. The ratings on the scale are based on multiple recommendations in the implementation literature (Goense et al., 2014) for a discussion of these in more detail, see Perepletchikova, Treat, and Kazdin (2007). The ITIPS-A has shown sufficient internal consistency on the domains of establishing (.66), assessing (.65), and evaluating (.64) and marginal consistency for the domain of reporting (.55) of treatment integrity and a sufficient interrater reliability (.633) in the study of Goense et al. (2014). Using the ITIPS-A, the implementation of integrity procedures in outcome studies can be classified as inadequate, approaching adequacy, and adequate. Only studies that could be classified as adequate on the total score were included in this study. This selection resulted in 7 articles covering 13 outcome studies on support conditions.

Last step in the study selection procedure was a selection of the studies in which the treatment integrity scores after implementation of the support systems were considered sufficient by the authors of the articles. This resulted in four articles covering six outcome studies.

Measure

Three trained coders used a manual to evaluate the elements of the support systems in the outcome studies on the evidence-based intervention (key elements, intervention type, and proof of effect); population in the outcome study; the practitioners, supervisors, and consultants in the study (background, education, etc); type of outcome study (RCT, N = 1 etc); the training and/or supervision that was provided (duration, type of training elements, and focus); and the results of the training on levels of treatment integrity.

Results

What type of support is provided in studies with a high level of treatment integrity?

All of the studies provide at least two types of support to the practitioners (Table 1). The extend in which the support systems are provided in the studies differ. The most extensive training was provided in the three studies of Forgatch and DeGarmo (2011). Their training comprehended 21 workshop days. The other trainings in the studies are between 2 and 3.5 days. Two studies provided a booster session of one day. Two studies provided telephone coaching/consultation. Supervision in all cases was provided in a group format.

Table 1.
Type and Extent of Support Systems in Studies.

Forgatch and DeGarmo (2011) Study 1 Forgatch and DeGarmo (2011) Study 2 Forgatch and DeGarmo (2011) Study 3 Hepner, Hunter, Paddock, Zhou, and Watkins (2011) Godley, Garner, Smith, Meyers, and Godley (2011) Lu et al. (2012)

Intervention PMTO PMTO PMTO CBT ACRA CBT

Training 21 days 21 days 21 days 2 days 3.5 days 2 days

Boostersession 1 day 1 day

Supervision/coaching/consultation Live group coaching, individual telephone consultation, monthly group coaching See study 1 See study 1 Weekly group supervision 90–120 minutes Biweekly telephone coaching calls Weekly group supervision

Note. N = 6. PMTO = Parent Management Training Oregon, CBT = Cognitive Behavioral Therapy, ACRA = Adolescent Community Reinforcement Approach.

What is the focus of the support and which learning methods are applied?

In Table 2, the learning methods and content that are applied in the training are described. It shows that the training consists of a combination of indirect and direct learning (see Support Systems section for descriptions) methods with a little more indirect methods that are used. The focus of the trainings has been on the theoretical foundation of the interventions, the techniques/elements to be applied in specific cases and key components and procedures.

Table 2.
Learning Methods and Content Applied in Training.

Indirect Direct

Didactic presentation of information (workbooks, journal articles, chapters, and materials) Modelling of key components and procedures

Lectures on the theoretical foundation of the intervention Role-play with feedback (on treatment integrity)

Directed reading of intervention manual Practice in situations with increasing difficulty

Video demonstrations of techniques/elements/ treatment cases Taking knowledge tests

Presentations

Written feedback (on practice cases)

Note. N = 6.

The focus and the learning methods of the booster sessions are not clearly described in the studies. The study of Hepner, Hunter, Paddock, Zhou, and Watkins (2011) only described that the one-day booster session addressed challenging issues that came up during the initial implementation of the intervention by the practitioners.

Information on the focus of the ongoing support (supervision/consultation/coaching) was provided in all of the studies. The focus was a combination of client progress, previous sessions, and upcoming sessions. Table 3 shows the applied learning methods and content in the ongoing support.

Table 3.
Learning Methods and Content Applied in Ongoing Support.

Indirect Direct

Written feedback on adherence and/or competence Feedback client progress using monitoring data

Reviewing audio-tapes

Feedback/discussion on adherence and/or competence

Discussion of challenges and potential solutions

Role-plays to practice or demonstrate skills for implementing the program and addressing challenges

Note. N = 6.

In the ongoing support, many direct learning methods are applied. There is only one indirect learning method applied and that is the use of written feedback on levels of adherence and/or competence.
What are the characteristics of effective supporters?

Only four studies provide information on the supporters’ characteristics. In the study of Godley, Garner, Smith, Meyers, and Godley (2011), the trainers were the treatment developers. The characteristics in Forgatch and DeGarmo (2011) differ. In the first study, the practitioners (G1) were trained by the intervention developer. In the second study, the practitioners (G2) were trained by selected trainers that were themselves trained by the developer (G1). In the last study, the practitioners were trained by G1 and G2.

All six studies provide some information on the characteristics of the supervisors. In two studies, the supervisors were licensed clinical psychologists who all had experience in the intervention. In one study (Lu, Yanos, Gottlieb, Marcello Duva, Xie, Rosenberg, & Mueser, 2012), the supervisor in the time of the study was being trained in the intervention and treated one case herself. This supervisor was supported by an expert consultant. The coaches in the study of Godley et al. (2011) were model experts.

Discussion

The objective of this review was to provide specific information how to give content to support and thereby establish a framework for an effective support system directed at treatment integrity. This was done by analyzing the support systems of studies that have properly measured treatment integrity and show a positive and sufficient effect of the support on levels of treatment integrity.

The results show that an effective support system consists of a minimum combination of training and ongoing support (supervision, consultation, and coaching), preferably extended with booster sessions. These findings are congruent with the suggestions in the literature that a short training of practitioners (alone) is not sufficient to allow for adherent delivery of the intervention (Addis, 2002; Connor-Smith & Weisz, 2003; Fixsen et al., 2005, in Steinfeld, Coffman, & Keyes, 2009).

The focus of the training was mostly on the theoretical foundation of the interventions and key component and procedures. Thereby the results support other research findings that suggest that an effective training has a focus on the (general) principles of the intervention, rather than teaching the details of manuals (Abramowitz, 2006; Hayes, 2002; Miller et al., 2004). When it comes to supervision, the results show it is recommended to focus on progress of clients, previous sessions, and upcoming sessions.

The results show that effective training consists of a combination of indirect and direct learning methods. This is consistent with the literature that suggests that to gain knowledge can be achieved by indirect methods but to gain skills a combination with direct learning methods is necessary (Cross et al., 2007; El-Tannir, 2002, in Beidas & Kendall, 2010). The merely sole focus on direct learning methods in the ongoing support might be explained by the fact that when practitioners receive these forms of support, they have already sustained knowledge of the intervention in their prior training in the intervention and the focus shifts to the delivery of the intervention. The idea of shift of focus is supported by the results showing the distinction between the content of the direct learning methods in training and the content in the ongoing support. In the training, the focus is on role-play, modelling, and practicing the skills and techniques of the intervention. In ongoing support, feedback on client progress and levels of adherence and/or competence in the delivery of the intervention get more attention.

An interesting finding in the studies was on the characteristics of the persons delivering the support. The study of Forgatch and DeGarmo (2011) showed that all generations of trainers provided a training that showed a positive and sufficient effect on levels of treatment integrity. Looking at the characteristics of the supervisors, one can conclude it is important a supervisor has experience in the intervention but providing the intervention itself at the time of supervision does not seem necessary.

Limitations and Future Suggestions

This is the first review that has looked at the effects of support systems on treatment integrity while taking the operationalization of treatment integrity into account. Only studies that adequately operationalized treatment integrity procedures and showed sufficient effects of the support on adherence levels were included. From the 7 articles and 13 outcome studies on support conditions that adequately implemented treatment integrity procedures, in only 4 articles covering 6 studies the support systems resulted in sufficient treatment adherence in the opinion of the authors of the article. This could be due to the fact that a lot of studies compared effects of different forms of support and therefore proved a lot of support ineffective.

However, four studies to draw an effective framework for a support system is very marginal. This review does however provide specific information on effective content of support (see Tables 2 and 3) which practitioners and intervention developers can use in order to establish and maintain treatment integrity of their planned interventions. It is important to research if the positive results of the studies can be replicated by providing similar support to practitioners of evidence-based interventions. This will not only extend the knowledge on how to best support practitioners in establishing and maintaining treatment integrity. More importantly, it has the potential to put a stop on the disappointing effects of evidence-based interventions in practice settings and result in the promising outcomes for clients they have proven in research settings. Given the influence of support systems on treatment integrity, as a general rule efficacy studies in the practice field should only be performed if support systems are working properly.

	Forgatch and DeGarmo (2011) Study 1	Forgatch and DeGarmo (2011) Study 2	Forgatch and DeGarmo (2011) Study 3	Hepner, Hunter, Paddock, Zhou, and Watkins (2011)	Godley, Garner, Smith, Meyers, and Godley (2011)	Lu et al. (2012)
Intervention	PMTO	PMTO	PMTO	CBT	ACRA	CBT
Training	21 days	21 days	21 days	2 days	3.5 days	2 days
Boostersession				1 day		1 day
Supervision/coaching/consultation	Live group coaching, individual telephone consultation, monthly group coaching	See study 1	See study 1	Weekly group supervision 90–120 minutes	Biweekly telephone coaching calls	Weekly group supervision

Indirect	Direct
Didactic presentation of information (workbooks, journal articles, chapters, and materials)	Modelling of key components and procedures
Lectures on the theoretical foundation of the intervention	Role-play with feedback (on treatment integrity)
Directed reading of intervention manual	Practice in situations with increasing difficulty
Video demonstrations of techniques/elements/ treatment cases	Taking knowledge tests
Presentations
Written feedback (on practice cases)

Indirect	Direct
Written feedback on adherence and/or competence	Feedback client progress using monitoring data
Reviewing audio-tapes
	Feedback/discussion on adherence and/or competence
	Discussion of challenges and potential solutions
	Role-plays to practice or demonstrate skills for implementing the program and addressing challenges

Footnotes

Acknowledgments

We thank Inge Busschers and Sanne Pronk for their contribution to the data collection and analysis.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This research was made possible by Stichting Innovatie Alliantie (SIA), a Dutch public funding body for research done by Applied Universities in the Netherlands, in the form of a grant (2012–14–18P).

Note

References

Abramowitz

(2006). Toward a functional analytic approach to psychologically complex patients: A comment on Ruscio and Holohan. Clinical Psychology: Science and Practice, 13, 163–166.

Addis

M. E.

(2002). Methods for disseminating research products and increasing evidence-based practice: Priorities, obstacles, and future directions. Clinical Psychology: Science and Practice, 9, 367–368.

Barber

Triffelman

Marmar

(2007). Considerations in treatment integrity: Implications and recommendations for PTSD research. Journal of Traumatic Stress, 20, 793–805. doi:10.1002/jts.20295

Beidas

R. S.

Kendall

P. C.

(2010). Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice, 17, 1–30.

Berger

Kleine

(2013). Handreiking reflecteren is leren. Utrecht: Nederlands Jeugdinstituut (NJI).

Connor-Smith

J. K.

Weisz

J. R.

(2003). Applying treatment outcomes research in clinical practice: Techniques for adapting interventions to the real world. Child and Adolescent Mental Health, 8, 3–10.

Cross

Matthieu

Cerel

Knox

(2007). Proximate outcomes of gatekeeper training for suicide prevention in the workplace. Suicide and Life-Threatening Behavior, 37, 659–670.

El-Tannir

(2002). The corporate university model for continuous learning, training, and development. Education & Training, 44, 76–81.

Fixsen

D. L.

Naoom

S. F.

Blasé

K. A.

Friedman

R. M.

Wallace

(2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.

10.

Forgatch

M. S.

DeGarmo

D. S.

(2011). Sustaining fidelity following the nationwide PMTO^TM implementation in Norway. Prevention Science, 12, 235–246. doi:10.1007/s11121-011-0225-6

11.

Godley

S. H.

Garner

B. R.

Smith

J. E.

Meyers

R. J.

Godley

M. D.

(2011). A large-scale dissemination and implementation model for evidence-based treatment and continuing care. Clinical Psychology: Science and Practice, 18, 67–83.

12.

Goense

Boendermaker

van Yperen

Stams

G. J.

Laar

(2014). Implementation of treatment integrity procedures: An analysis of outcome studies of youth interventions targeting externalizing behavioral problems. Zeitschrift für Psychologie, 222, 12–21.

13.

Goense

P. B.

(2013). ITIPS-A, manual for coding studies on implementation of procedures for treatment integrity, adapted version. Amsterdam, the Netherlands: Amsterdam University of Applied Sciences (Internal publication).

14.

Hayes

(2002). Getting to dissemination. Clinical Psychology: Science and Practice, 9, 410–415.

15.

Hepner

K. A.

Hunter

S. B.

Paddock

S. M.

Zhou

Watkins

K. E.

(2011). Training addiction counselors to implement CBT for depression. Administration and Policy in Mental Health, 38, 313–323. doi:10.1007/s10488-011-0359-7

16.

Kazdin

A. E.

Weisz

J. R.

(2003). Evidence-based psychotherapies for children and adolescents. New York, NY: The Guilford Press.

17.

Kerby

N. T.

(Ed.). (2006). Helping others help children; clinical supervision of child psychotherapy. Washington, DC: American Psychological Association (APA).

18.

Yanos

P. T.

Gottlieb

J. D.

Marcello Duva

Xie

Rosenberg

S. D.

Mueser

K. T.

(2012). Use of fidelity assessments to train clinicians in the CBT for PTSD program for clients with serious mental illness. Psychiatric Services, 63, 785–792.

19.

Mikolajczak

Stals

Fleuren

M. A. H.

Wilde

E. J. D.

Paulussen

T. G. W. M.

(2009). Kennissynthese van condities voor effectieve invoering van jeugdinterventies. Leiden / Utrecht: TNO Kwaliteit van Leven / Nederlands Jeugdinstituut (NJI).

20.

Miller

Yahne

Moyers

Martinez

Pirritano

(2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology, 72, 1050–1062.

21.

Perepletchikova

(2006). Retrieved April, 2013, from www.treatmentintegrity.com

22.

Perepletchikova

Treat

T. A.

Kazdin

A. E.

(2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75, 829–841.

23.

Schoenwald

S. K.

Chapman

J. E.

Sheidow

A. J.

Carter

R. E.

(2009). Long-term youth criminal outcomes in MST transport: The impact of therapist adherence and organizational climate and structure. Journal of Clinical Child and Adolescent Psychology, 38, 91–105. doi:10.1080/15374410802575388

24.

Sexton

Turner

C. W.

(2010). The effectiveness of functional family therapy for youth with behavioral problems in a community practice setting. Journal of Family Psychology: JFP: Journal of the Division of Family Psychology of the American Psychological Association (Division 43), 24, 339–348.

25.

Steinfeld

B. I.

Coffman

S. J.

Keyes

J. A.

(2009). Implementation of an Evidence-Based Practice in a Clinical Setting: What Happens When You Get There? Professional Psychology: Research and Practice, 40, 410–416.

26.

Tennyson

H. R.

(2009). Reducing Juvenile Recidivism: A Meta-Analysis of Treatment Outcomes (Doctoral dissertation, Pacific University). Retrieved from: http://commons.pacificu.edu/spp/109

27.

Voogden

Kuyvenhoven

(2010). Onderzoek naar de effecten van supervisie. Tijdschrift Supervisie en Coaching, 27, 3–14.