Abstract
This article provides a very high-level overview of some key considerations in the field of implementation science as it relates to clinical psychology. The article reviews recent findings regarding treatment fidelity, adaptation of treatments, and clinical outcomes. It then details some recent findings on training therapists to provide evidence-based therapies as they were designed to be delivered, with sufficient skill and adaptations if needed to ensure better fit for clients who receive treatments in different treatment settings. Finally, the article considers implications for implementing and sustaining new treatments and supporting their fidelity and adaptation, as well as key directions for future research.
Millions of dollars and hundreds of clinical trials have been devoted to the development and testing of interventions to support and improve mental health. However, the average consumer seeking mental-health services is unlikely to be offered or receive an evidence-based treatment that has been proven to be effective. The reasons for this research-practice gap are numerous and well documented (Stirman et al., 2016). First, information about effective treatments may not have reached consumers: Practicing therapists do not have the time or library access to read the journals in which findings are published and are therefore unlikely to learn about relevant findings or share them with their clients. People who seek their services may not be aware of differences between treatments. Second, factors at the therapist level, including their knowledge, previous training, motivation, attitudes toward evidence-based treatments, and concerns about fit, can also contribute to their decisions regarding whether or not to learn and offer these treatments. Third, many factors at the level of the organization and the systems in which services are provided can affect whether and how evidence-based interventions are made available to clients. These include, but are not limited to, the leadership, climate, and culture within the organization, as well as the availability of resources and personnel (Crowe et al., 2020; Rosen et al., 2016; Stirman et al., 2016). At the system level, sociopolitical factors, policies, and funding affect whether, where, and how these treatments are offered (Stirman et al., 2016).
Implementing, Adapting, and Sustaining Evidence-Based Mental-Health Treatments
Implementation science is a field that has developed to identify and address the challenges associated with bringing evidence-based interventions into routine care. Over time, frameworks have been developed to guide the study of the determinants and process of implementation, as well as the assessment of relevant outcomes, including feasibility, acceptability, appropriateness, cost, implementation fidelity, sustainability, and clinical outcomes (Nilsen, 2015; Stirman et al., 2016). In addition, implementation research focuses on identifying strategies to support efforts to implement and sustain interventions, and on understanding what makes these strategies successful.
Increasing the likelihood of sustained implementation of new interventions often requires changes within an organization or system. These can include changes in policy, staff training (including time allocated for training and consultation), screening and triage, scheduling frequency, and procedures to accommodate weekly time-limited treatments (Crowe et al., 2020; Rosen et al., 2016). Additionally, ensuring fit and effectiveness with the population served often requires adapting interventions to address contextual factors, including literacy levels, cultural norms, available personnel and resources, and accessibility (Stirman et al., 2019).
Although earlier implementation research focused more on training, more recent research has focused on multifaceted strategies to address the complex interactions between the context, the intervention, and the individuals who provide and receive the treatment. This article focuses on research exploring the interplay between training, fidelity, the adaptation of interventions, and clinical outcomes when new interventions are implemented. I review some of the findings in these areas, along with their implications for both clinical practice and future research.
Treatment fidelity, adaptation, and clinical outcomes
Ensuring treatment fidelity (adherence to the treatment as originally designed and specified and competent provision of the treatment) is critical in the development and testing of new interventions. However, as a treatment is transported out of a more controlled study and into contexts where it has not been studied, tension can arise between the need to preserve fidelity and the need to adapt the treatment to address specific needs and clinical realities, such as comorbidities, lack of resources, levels of literacy, and cultural context (Stirman, Gamarra, et al., 2017; Stirman et al., 2019).
In order to be responsive to and effective for the individual who is participating in treatment, treatment components must be tailored to that individual’s unique needs. When an intervention is tailored, it may look somewhat different for different clients because of substitution of a similar exercise or modification of materials, language, analogies, or information and support provided, although the core goals and components remain recognizable. For example, to engage in a process of cognitive restructuring during a therapy session, one client might complete an exercise on a cognitive-therapy worksheet in a session with their therapist, whereas another may learn the “3 Cs” strategy (“Catch, Check, and Change your thoughts”) without the use of a worksheet. However, some adaptations may be more extensive and include integration of other interventions, changes in sequencing or number of sessions, or changes in the personnel who provide the treatment. A distinction can be made between fidelity-consistent modifications, which preserve the core elements and functions of an intervention, and fidelity-inconsistent modifications, which may, for example, remove some elements or integrate different interventions that do not share an underlying theory of change or set of principles (Stirman et al., 2019).
Ideally, adaptations are planned in advance, informed by input and consideration of the needs of the individual or population receiving treatments in a given setting, and refined through a pilot study or by using evaluation data. There are frameworks that can guide this process both for cultural adaptations (Baumann et al., 2014) and for broader adaptations made to address other client needs or factors, such as available resources and staffing (Kirk et al., 2020; Miller et al., 2020). Other times, though, modifications need to be made because of emerging or unexpected circumstances. A recent example at the setting level was the rapid and widespread change in format from face-to-face therapy to telehealth during the COVID-19 pandemic (Beidas & Stirman, 2021).
It is important to understand the impact that such changes have on the effectiveness of treatments as well as on other outcomes that matter to consumers and organizations, such as accessibility, equity, affordability, feasibility, and satisfaction. Because the types of adaptations range from minor tailoring to substantial changes, they may be associated with different outcomes. For example, a fidelity-inconsistent change such as removing a key element of treatment may result in greater feasibility or satisfaction, but less positive clinical outcomes. An example is removing an exposure component from a therapy for anxiety because it is not feasible for a therapist to leave the office with a client to implement this component. In such a case, it might be more feasible and more acceptable to the client to provide a cognitive behavior treatment without exposure, but if exposure is critical to producing change, the treatment may ultimately be less effective than it would be if the exposure component were retained. However, research has shown that other forms of adaptation, such as adding components and making culturally appropriate adaptations, can enhance outcomes (Stirman, Gamarra, et al., 2017; Sundell et al., 2016).
Using a common language to characterize such modifications facilitates understanding the different types of impacts they can have. The Framework for Reporting Adaptations and Modifications-Expanded (Stirman et al., 2019) allows documentation of changes that are made, the levels at which they occur (e.g., individual, cohort, organization), the reasons for the changes (e.g., to address cultural differences or setting constraints), and the goals and desired outcomes (e.g., to increase effectiveness, access, equity, or feasibility). When changes are analyzed in conjunction with outcome data, it is possible to understand whether the changes are in fact associated with the desired outcomes. When changes are examined along with fidelity data, it is possible to understand interrelationships between fidelity, adaptation, and outcomes.
Marques, her colleagues, and I (Marques et al., 2019) used this approach in a community-based clinic that serves many individuals who immigrated to the United States because of violence in their home country and/or who have experienced interpersonal or community violence. We trained therapists in the clinic to conduct an evidence-based trauma-focused therapy (cognitive processing therapy, or CPT) in English or Spanish, after pilot-testing the treatment and interviewing the therapists to understand the types of adaptations that might be necessary (Valentine et al., 2017). As therapists provided the adapted treatment, we reviewed the sessions to assess fidelity and to characterize the additional modifications that therapists made in the sessions with different individuals. We examined associations between fidelity, adaptations, and changes in posttraumatic stress disorder (PTSD) and depression. On average, there were between one and two modifications per session, and more occurred if the sessions were conducted in Spanish. In this study, the fidelity-inconsistent adaptations mostly comprised removing or skipping aspects of CPT, changes that were also captured through adherence ratings. Using statistical models that included language, fidelity-consistent adaptations, adherence, and competence, we found that both greater competence and fidelity-consistent modifications in the first half of therapy were associated with greater decreases in PTSD symptoms. Adherence and fidelity-consistent modifications were associated with greater improvements in depression.
Training and consultation: addressing complexities in practice
Findings of this nature have implications for how new evidence-based treatments should be implemented. The more a therapist understands about whether and how an intervention can be adapted while remaining aligned with the underlying theory of change and core treatment strategies, the better the chances are that the therapist will be able to provide the intervention flexibly and effectively instead of drifting to other approaches when challenges arise. Training and manuals that include guidance on adaptation provide important opportunities for addressing the realities of clinical practice, especially in settings with many individuals who have been underrepresented in research.
Additionally, research suggests that consultation, or follow-up support for therapists, may promote greater fidelity (Herschell et al., 2010) and better treatment outcomes (Monson et al., 2018). In an initial study (Monson et al., 2018), my colleagues and I investigated what elements in consultation may improve clinical outcomes. We compared the following conditions: no consultation and delayed fidelity feedback, standard consultation, and consultation that additionally included review of audio recordings of therapy sessions. Fidelity did not vary meaningfully between the conditions, although therapists who did not receive consultation were less likely to enroll clients and submit fidelity data than were therapists who received consultation. As we had previously hypothesized (Stirman et al., 2013), data from this study revealed that greater therapist competence was associated with greater improvement in symptoms (Keefe et al., 2021), a finding consistent with other work indicating an association between treatment fidelity and outcomes (Power et al., 2022). Therapeutic alliance, the degree to which the therapist and client agree on the goals and tasks of therapy and experience a bond comprising reciprocal positive feelings (Hovarth & Luborsky, 1993), was also associated with symptom change, and there was an interaction between competence and alliance such that when both were high, subsequent symptom scores were especially low (Keefe et al., 2021).
In our initial study, standard consultation without audio review of session content unexpectedly resulted in greater improvement of PTSD symptoms compared with the other two conditions (Monson et al., 2018). In subsequent work, we examined the data more closely to understand this finding, thinking that perhaps the time spent reviewing brief segments of the session was a distraction from the bigger-picture discussion of how to conceptualize the individual case and apply CPT. We reviewed different categories of consultation content, including review of fidelity and provision of feedback, discussion of the application of CPT to specific cases, and off-topic discussion, as well as whether technical difficulties occurred. Specific categories of consultation content were not associated with greater adherence to the treatment as specified or with competence (Swanson et al., 2021), although therapists’ level of adherence was influenced by who their consultant was (competence and clinical outcomes did not differ across consultants, though; C. Johnson et al., 2021). Consultants differed significantly in how much they discussed the application of specific CPT strategies to individual cases (C. Johnson et al., 2021), and this was the only consultation activity that was associated with better treatment outcomes (Swanson et al., 2021).
In recent years, implementation science has begun to expand its focus on effective elements of training and consultation to include potential mechanisms through which training and consultation have their impact, such as the therapist’s knowledge, attitudes, sense of self-efficacy, and skill acquisition (McLeod et al., 2018). Some evidence suggests that skill acquisition, practice, and sense of self-efficacy may be important. For example, even though my colleagues and I found that consultation was not associated with improved fidelity in our initial study (Monson et al., 2018), subsequent analyses of the data revealed that the more consultation sessions therapists attended, the better their clients’ clinical outcomes were (Swanson et al., 2021). As in other research (J. E. Johnson et al., 2019), therapists’ second and subsequent clients did better than their first, which suggests that opportunities to become familiar with and learn a protocol through experience are important. Additional analyses indicated that therapists’ sense of self-efficacy improved significantly over the course of 6 months of CPT training but was not associated with changes in fidelity over the course of consultation (Pace et al., 2021). However, therapists with low self-efficacy who did not receive consultation tended to have poorer client outcomes than therapists with low self-efficacy who received consultation. Taken together, these results suggest that consultation may have a protective effect for therapists with low self-efficacy, and that it may be important to encourage attendance in consultation.
During consultation, feedback on fidelity without comprehensive observation and use of a fidelity measure may not be completely accurate, especially if it is based on self-report or observation of small segments of sessions (C. Johnson et al., 2021). However, group-based consultation with review of audio from treatment sessions may promote sustainment of skill among therapists working within mental-health agencies (Creed et al., 2016; German et al., 2018; Stirman, Pontoski, et al., 2017). Our findings suggest that in the absence of relatively comprehensive observation and feedback, it may be more important to focus on helping therapists plan for upcoming sessions by discussing application of the strategies to individual clients. In other research, therapists’ more positive attitudes about evidence-based treatments during a pretraining assessment were associated with greater fidelity in the treatment they provided, assessed at subsequent points in training and consultation programs (Creed et al., 2021; Sijercic et al., 2020). These findings need to be replicated with other samples, but they add to a literature that can provide guidance about the effective elements and potential mechanisms of consultation and training.
Training and consultation programs alone, though, are not sufficient to ensure successful implementation. There is evidence that organizational culture and climate, for example, are associated with the sustainment of new programs and turnover of providers (Glisson et al., 2010; Williams et al., 2018). Preliminary needs assessment and work with organizations or communities can identify barriers and strengths that can influence the course of implementation. The effectiveness of single implementation strategies and packages of implementation strategies (Kirchner et al., 2020) in addressing barriers and leveraging strengths is being evaluated. To date, for example, there is some evidence that strategies that improve implementation leadership, climate, and culture can support implementation success (Aarons et al., 2015; Glisson et al., 2010) Other approaches, such as learning collaboratives, tool kits, and access to support resources show promise in providing implementation support that goes beyond initial training efforts (LoSavio et al., 2019; Sayer et al., 2021; Weisz et al., 2020; Worley et al., 2020).
Conclusion
Psychologists have developed treatments that can have an impact on clinical outcomes and quality of life of individuals who seek mental-health treatments. The examples provided here illustrate how implementation research can support the translation of these findings to practice. Future research exploring the degree of fidelity needed to achieve good outcomes, adaptations that are most likely to be effective, and how to provide practical, real-time guidance and support to treatment providers could have a substantial impact. Researchers have been investigating more scalable approaches to fidelity assessment and support than the “gold standard” strategy of observing and coding full therapy sessions (e.g., Becker-Haimes et al. 2021). Additional work is being done to see how artificial intelligence (Flemotomos et al., 2021; Imel et al., 2019) and other methods can be applied to data collected over the course of treatment to assess and support fidelity (Stirman et al., 2021).
Also needed is a better understanding of what implementation strategies are most effective in different contexts and circumstances. Learning more about how implementation strategies work and why they have the impact they do will allow the development of even more targeted and efficient strategies. Efficient implementation strategies are essential because implementation often occurs in underresourced settings with many competing priorities and demands. If effective but complex implementation strategies that require substantial time and resources to deploy are developed, this would essentially duplicate already-existing challenges in implementing psychotherapy interventions (Beidas et al., 2022). However, in combination with the effort and resources spent on developing and testing interventions, findings from implementation science can foster progress toward the ultimate goals of improving well-being and ensuring equity and access to effective treatments, thereby realizing the promise and expanding the reach of psychological science.
Recommended Reading
Kirchner, J. E., Smith, J. L., Powell, B. J., Waltz, T. J., & Proctor, E. K. (2020). (See References). A primer on different types of implementation strategies, with guidance on how to select and report them.
Nilsen, P. (2015). (See References). An overview of different types of implementation frameworks and how they can be used to guide implementation research and practice.
Stirman, S. W., Gutner, C. A., Langdon, K., & Graham, J. R. (2016). (See References). A summary of implementation research and theory as it relates to implementation of evidence-based treatments, with a review of promising and effective implementation strategies for mental-health interventions.
Williams, N. J., & Beidas, R. S. (2019). Annual Research Review: The state of implementation science in child psychology and psychiatry: A review and suggestions to advance the field. Journal of Child Psychology and Psychiatry, 60(4), 430–450. https://doi.org/10.1111/jcpp.12960. A review of children’s mental health that discusses recent advances and suggests future directions for implementation research.
