Abstract
Research suggests that a modular approach to treatment and training may be a promising method for promoting evidence-based practices (EBPs). Furthermore, there is growing evidence that the effects of training on specific technique implementation are affected by therapist and youth factors as well as the type of problem area addressed during training. The current study expanded these findings by examining the extent to which youths’ clinical progress improved as a result of therapists’ attendance at modular workshop trainings. Longitudinal, archival data from community mental health providers (n = 48) who participated in a series of anxiety and/or disruptive behavior workshops in modular approaches to EBPs for youth were examined using cross-classified multilevel modeling. Results indicated that youths’ month in treatment and therapists’ practice behaviors following their training significantly and positively influenced clinical progress. However, only therapists who attended a disruptive behavior training reported greater rates of improved youth treatment progress. Therapists’ knowledge of the trained techniques was not a significant moderator of outcomes. Results suggest that therapists may require differential training and implementation supports that vary as function of problem area to positively enhance youth outcomes. Limitations and implications for EBP dissemination and implementation are discussed.
Keywords
Although progress has been made for testing and identifying evidence-based practices (EBPs; Chorpita et al., 2002), there continue to be challenges in bringing these types of interventions into public mental health settings (Weersing et al., 2002). Accordingly, stakeholders have acknowledged the need for large-scale quality improvement initiatives (Flynn & Brown, 2011) and have advocated for community mental health therapists to adopt and implement youth mental health EBPs (Beidas & Kendall, 2010). One potential area for furthering EBP adoption involves expanding our scientific understanding of adapting evidence-based training processes to better suit large public sector mental health systems (Lim et al., 2012).
Influential factors in provider training include therapists’ characteristics, organizational settings, and training designs (Ford & Weissbein, 1997). Although a common training design is that of the time-limited workshop, research suggests that therapists require a balance of didactic, interactive, and ongoing supports for successful adoption and implementation of EBPs (McHugh & Barlow, 2010). At the same time, however, therapists have indicated that a lack of time, training costs, and access to manuals can be obstacles for participating in evidence-based training activities (Pagoto et al., 2007). Recognizing the need for balancing the dialectic between these barriers and evidence-based training strategies, our field has begun to modify and innovate large-scale training efforts (Chorpita & Daleiden, 2009).
Modularized Approach to Training and Treatment
A modular training paradigm is recommended within public sector mental health systems given its focus on core principles of specific interventions rather than complete treatment protocols (cf. Beidas et al., 2011). Treatment research suggests that modularity may be useful for helping balance the prescriptive nature sometimes typical of standard EBP approaches and the flexibility often needed in real-world implementation (Chorpita et al., 2005). Modular therapy involves the guided assembly of numerous discrete therapeutic practices (i.e., practice elements [PEs]), into a coherent treatment approach. A PE has been defined as a discrete clinical technique or strategy used as a component of a larger intervention plan (Chorpita et al., 2005). Furthermore, a PE that is derived from the evidence base (PDEB), is a discrete clinical technique that has been utilized within a sufficient number of EBP study groups (cf. Okamura, 2016). Similar to the concept of modular therapy, McLennan et al. (2019) contend that modular training involves a default sequencing path for smaller units of instruction (e.g., core components for treating youth anxiety), while allowing for some assembly flexibility through the use of decision aids.
Studies that have investigated the effectiveness of modularized training and treatment strategies have found that this approach may be effective for positively influencing therapists’ knowledge and attitudes (Lim et al., 2012), and practice behaviors (McLennan et al., 2019). This type of training approach has also been shown to improve therapists’ treatment adherence/competence and satisfaction (Puspitasari et al., 2013). Furthermore, in the only study to date to examine youth mental health outcomes associated with modular treatment and training, Southam-Gerow et al. (2014) found such approaches useful for significantly and positively changing youth treatment outcomes.
Current Moderators of Treatment Outcomes
As the current study investigated the effects of a training effort on youth outcomes with public mental health therapists as they naturalistically progressed through a statewide training effort, consideration of potential confounds and moderators seemed warranted for informing study design. Given the paucity of research on modular treatment and training, the broader youth mental health outcome literature was examined for identifying possible moderators. Broadly speaking, youth mental health treatment literature seems to highlight three particular areas that may affect youth treatment outcomes: youth or family characteristics, treated problem area, and therapist attributes (Maric et al., 2015). Concerning youth characteristics, outcome research is mixed with regard to the influence of variables such as age, gender, and comorbidity. For example, Compton et al. (2014) found that age did not predict clinical improvement rates, while others found that age was either positively or inversely related to client progress (e.g., Mueller et al., 2010). The literature also suggests that gender (Nilsen et al., 2013) and comorbidity (Beauchaine et al., 2005) do not moderate youths’ clinical outcomes. However, research suggests that some family factors (e.g., lower income, marital status, parental psychopathology) negatively affect treatment outcomes (Leijten et al., 2013; Maric et al., 2015).
Research looking at problem area (e.g., depression) is also mixed, such that some have found that problem type does not significantly affect improvement rates (Weisz et al., 1995), while others have found that it affects both improvement rates and overall posttreatment status (Love et al., 2014). For example, Love et al.’s (2014) findings have suggested that youth with disruptive behavior or depressed mood improve the most over the course of treatment, but that anxious youth may improve faster than disruptive behavior youth.
Regarding therapist characteristics, the literature has focused on constructs such as treatment “adherence” (Webb et al., 2010) and “competence” (Fairburn & Cooper, 2011); however, this line of research is relatively complex. For example, Webb et al.’s (2010) meta-analysis suggested that therapist adherence/competence may not be key variables in determining client outcomes. However, within the substance use treatment area, some have demonstrated a relationship between therapist adherence/competence and positive client outcomes (Barber et al., 2006). In addition, research examining other therapist characteristics such as level of licensure, highest degree earned, professional specialty, and agency have found little to no relationship with youth outcomes (Weisz et al., 1995). Regarding therapists’ practice approaches on clinical progress and outcomes, studies suggest that the specific use of PDEBs with appropriate treatment target populations as well as the greater number and more diverse treatment techniques used may result in better outcomes (Beauchaine et al., 2005; Garland et al., 2014; Milette-Winfree, 2017).
Practices Derived From the Evidence Base (PDEB) Quality Improvement Training Initiative
Recognizing the potential benefit for training public sector mental health clinicians on the appropriate use of PDEBs, the State of Hawai‘i’s Child and Adolescent Mental Health Division (CAMHD) offered three different modular training workshops to their therapists between 2008 and 2009: (a) Introduction to Hawai‘i’s Public Sector Mental Health System, (b) Core Practice Elements Derived from the Evidence-Base for Anxiety and Trauma, and (c) Core Practice Elements Derived from the Evidence-Base for Disruptive Behaviors (Nakamura et al., 2011).
Research studying this CAMHD training endeavor suggests that a targeted approach to training emphasizing technique modularity can result in improving therapists’ EBP knowledge and attitudes (Lim et al., 2012). Moreover, McLennan et al. (2019) expanded upon original work in this area by examining therapists’ changes in practices posttraining. Their results suggested that anxiety trainings were associated with more anxiety PDEB usage while disruptive behavior trainings were associated with less disruptive behavior PDEB usage. However, what remains to be investigated is the extent to which those posttraining therapist behavior changes affected youth clinical progress.
Present Investigation
The present study expanded upon McLennan et al.’s (2019) work by examining the extent to which youths’ clinical progress improved as a result of therapists’ modular workshop training attendance. Study hypotheses are specified by a (effects of anxiety training on anxious youths’ outcomes) or b (effects of disruptive behavior training on disruptive behavior youths’ outcomes). Foremost, given that multiple usual care studies suggest that youth clinical progress generally increases over just time alone (Garland et al., 2014), the current study hypothesized that anxious and disruptive behavior youth who received treatment prior to their therapist attending the training would have clinical progress ratings with significant positive slopes over the course of their treatment episode (Hypotheses 1a and 1b). Furthermore, we hypothesized that anxious and disruptive behavior youths’ clinical progress would have significantly steeper positive slopes (faster rates of change) for clients whose therapist attended the anxiety training and/or disruptive behavior training, and used more of the package of trained PDEBs over the course of the treatment episodes (Hypothesis 1ai and 1bi). We also predicted that the relationship between therapists’ anxiety and disruptive behavior training attendance, and youth clinical progress over the course of their treatment episode, would be positively moderated by the extent to which therapists appropriately used the five trained anxiety (Hypothesis 2a) and seven trained disruptive behavior (Hypothesis 2b) PDEBs posttraining. In addition, given that some research has indicated that therapists’ treatment knowledge and skills may positively affect client outcomes (Barber et al., 2006; Webb et al., 2010), we hypothesized that for anxious and disruptive behavior youth who were seen only after their therapist attended the anxiety and/or disruptive behavior trainings, respectively, their clinical progress would be moderated by therapists’ posttraining knowledge of the specific trained PDEBs. More explicitly, we hypothesized that higher therapists’ posttraining knowledge of the specific trained PDEBs would be associated with higher youth clinical progress ratings at the end of youths’ treatment episodes (Hypotheses 3a and 3b).
Method
Participants
The current study had the following inclusionary criteria. First, therapists had to attend at least one of the anxiety or disruptive behavior state-wide trainings described above. Second, therapist participants were restricted to only CAMHD-contracted therapists from Lim et al.’s (2012) original sample (i.e., Department of Education therapists were excluded), since therapist-report practice data as captured in the current methodology were available for only CAMHD therapists. Client participants were restricted to only those who were seen 2 years prior to the training and 2 years following the training (e.g., November 1, 2006 to November 1, 2011). Therapist-reported practice data were obtained through archival records of the Monthly Treatment Progress Summary (MTPS; CAMHD, 2003) form. Third, each CAMHD therapist was to have treated at least one youth as indicated by MTPS data within the study period.
Concerning youth participant inclusionary criteria, clients were required to have had at least one MTPS form completed during their treatment episode, and the current study utilized up to the first 6 months of practice data (cf. Jackson et al., 2012; Love et al., 2014; Weersing et al., 2002). Youth whose treatment episode occurred while their therapist attended one of the trainings were excluded to best capture therapist pre- and posttraining behavioral changes. In addition, given the cross-classified nature of the data, it was possible that youth were not nested only within one therapist during the entire study period. As such, youth were included if they were seen by up to three therapists. This decision was based on an initial exploration of the data set which indicated that the majority of youth participants had only one therapist and a few having as many as three.
For the therapist sample (n = 48), 26 therapists attended only the disruptive behavior training, 13 attended only the anxiety training, and nine therapists attended both the disruptive and the anxiety trainings. Of these 48 therapists, 72.9% were female (n = 35), with ages ranging from 25 to 66 (M = 42.0, SD = 11.4). Primary ethnicities reported were: White (n = 24; 50.0%), Asian (n = 8; 16.7%), Hawaiian or Pacific Islander (n = 3; 6.3%), and Latino or Hispanic (n = 3; 6.3%). Ten participants (20.8%) did not report a primary ethnicity. Therapists reported varying levels of education, with most therapists reporting earning a master’s level degree (n = 51; 83.4%). Approximately 27.1% (n = 13) of the participants reported holding a state license to practice and reported having on average an active caseload of 7.7 cases (SD = 5.1).
Of the 862 youth participants included in this study, 324 were treated for anxiety-related and 538 were treated for disruptive behavior-related treatment targets. Of the youth treated for anxiety, youths’ ages ranged from 4.4 to 17.8 (M = 13.8, SD = 3.1), while the ages for youth treated for disruptive behavior ranged from 4.4 to 18.8 (M = 14.3, SD = 2.8). Consistent with the general CAMHD youth population, these youths were majority male, multi-racial, generally teenagers, and mainly treated for disruptive behavior concerns (Love et al., 2014).
Measures
Knowledge of Evidence-Based Services Questionnaire
The Knowledge of Evidence-Based Services Questionnaire (KEBSQ; Stumpf et al., 2009) is a 40-item measure that examines participants’ knowledge of various practices derived and not derived from the evidence-base for the youth problem areas of: Anxious/Avoidant (A), Depressed/Withdrawn (D), Disruptive behavior (B), and Attention/Hyperactivity (H). Each participant circles all problem areas for which a particular PE is recognized as being derived from a larger EBP protocol. The 40 items (representing one PE each) are scored on a scale of 1 to 4, with correctly endorsed and omitted responses per problem area each receiving one point, with total possible scores on the KEBSQ ranging from 0 to 160. As such, grouping of these 160 binary decisions can occur in a variety of chunks or categories (Lim et al., 2012; Nakamura et al., 2011). Given that the main objective was to understand the impact that the trainings had on youth outcomes, the present investigation calculated therapists’ posttraining knowledge of the specific trained PDEBs in two different ways. More specifically, therapists’ EBP knowledge was calculated as their (a) KEBSQ Anxiety (ANX) Target score and therapists’ (b) KEBSQ Disruptive Behavior Disorder (DBD) Target score (cf. Lim et al., 2012). The KEBSQ ANX Target score was used to examine therapists’ EBP anxiety knowledge of the five PDEBs (i.e., self-monitoring, psychoeducation for youth [for anxiety], exposure, relaxation, cognitive/coping) taught at the anxiety training workshops. This score was calculated by analyzing the extent to which therapists correctly circled A (indicating that the practice was a PDEB for anxiety) for each of the five trained ANX PDEBs (i.e., the total score for this indicator ranged from 0 to 5). Similarly, the KEBSQ DBD Target score was used to examine the extent to which therapists correctly circled B (indicating that the practice was a PDEB for disruptive behaviors) for each of the seven trained DBD PDEBs (i.e., psychoeducation for caregiver [for disruptive behaviors], commands, tangible rewards, response-cost, praise, monitoring, and attending; the total score for this indicator ranged from zero to seven). The authors of the KEBSQ have demonstrated adequate test–retest reliability in graduate students and community therapists (r = .56) and the ability to discriminate between these two samples in their original study.
Monthly Treatment and Progress Summary
The Monthly Treatment and Progress Summary (MTPS; CAMHD, 2003) is a monthly therapist report surveying service format, service setting, treatment targets, youth clinical progress, and intervention practices used for each client seen by the therapist. Beginning on July 1, 2006, all CAMHD contracted therapists were required on a monthly basis to complete the MTPS for every client in order to receive reimbursement (Nakamura et al., 2007). This investigation paid particular attention to the Treatment Targets, Progress Ratings, and Interventions Strategies sections of the MTPS. In the CAMHD system, treatment targets are used to identify specific areas of focus during the treatment month with a particular youth. For this study, MTPS anxiety data were included when one or more of the treatment targets of anxiety, avoidance, phobia/fears, or trauma (PracticeWise, 2008) were endorsed as a focus of treatment for that reporting month. Additionally, MTPS disruptive behavior data were included when one or more of the treatment targets of aggression, anger, fire setting, oppositional/non-compliant behavior, runaway, sexual misconduct, or willful misconduct/delinquency (PracticeWise, 2008) were endorsed. Therapists also report a subjective rating of progress for each client in regards to each individual target indicated that month. These progress ratings are scored using a 7-point scale, with 0 indicating <0% improvement or deterioration and 6 indicating 91–100% improvement or complete improvement. Monthly progress ratings are calculated by comparing each treatment target to an initial baseline level (CAMHD, 2008). Concerning the intervention strategies section of the MTPS, a therapist can choose from 63 PEs and three additional write-in options. Practice elements include those that were the area of focus during the CAMHD trainings (e.g., exposure, cognitive), but also many others that span a wide array of problems, both derived and not derived from the evidence base. Research has demonstrated evidence for the reliability and validity of the treatment target, progress ratings, and intervention strategies sections of the MTPS (cf. Borntrager et al., 2013).
Therapist Background Questionnaire (Unpublished Measure)
The Therapist Background Questionnaire (TBQ) is an initial instrument designed to assess therapists’ demographics, training, clinical experience, and theoretical orientation. The measure assesses the following: (a) Agency/Affiliation, (b) Age, (c) Gender, (d) Ethnicity/Race, (e) Ethnic Identity (if multiple ethnicities/races are endorsed, the one with which the participant identifies with the most), (f) Degrees Earned, (g) Licensure, (h) Professional Specialty, (i) Position, (j) Level of Care, (k) Years of Clinical Training and Full-time Clinical Experience, (l) Current Caseload, (m) Hours of Supervision per Week, and (n) Theoretical Orientation.
Procedure
All training questionnaire data were collected as participants progressed through as many as three state-sponsored voluntary 1-day trainings: (a) Introduction to Hawai‘i’s Public Sector Mental Health System (INTRO), (b) Core Practice Elements for Anxiety and Trauma (ANX), and (c) Core Practice Elements for Disruptive Behaviors (DBD; more information about training content and procedures on this larger statewide effort can be found in Lim et al. [2012] and McLennan et al. [2019]). Each training was seen as a discrete component that was used as part of a larger training protocol (Lim et al., 2012). Participants were encouraged to complete the INTRO training before attending either the ANX or DBD; however, some therapists attended the INTRO training after attending an ANX or DBD training or did not attend an INTRO training at all, due to scheduling difficulties. A total of 12 INTRO, six ANX, and five DBD trainings were provided by CAMHD expert trainers throughout the state. All trainings were codified, such that each spanned the same duration, were provided by one set of CAMHD trainers, and used the same PowerPoint presentation, outline, and training materials. The CAMHD trainers consisted of two PhD-level clinical psychologists and three master’s level staff with degrees in psychology or related mental health service fields.
The INTRO training served as a broad-level orientation to youth public mental health services within Hawai‘i. The ANX and DBD trainings focused on core PDEBs (rather than brand-named treatment manuals) present across numerous evidence-based protocols, as well as assembling these common elements within a modular approach to treatment. The ANX training focused on five PDEBs: self-monitoring, psychoeducation for youth (for anxiety), exposure, relaxation, and cognitive/coping. The DBD training focused on seven PDEBs: psychoeducation for caregivers (for disruptive behavior), commands, tangible rewards, response-cost, praise, monitoring, and attending. The problem areas of youth anxiety and disruptive behavior were chosen given their generally high base rates of occurrence in the broader psychopathology literature (Merikangas et al., 2010) as well as specifically in CAMHD (Jackson et al., 2012). Trainers provided systematic guidelines adapted from various Practitioner Guide protocols (PracticeWise, 2008) for applying techniques and used traditional didactic methods, along with videos, modeling, and role-playing to provide therapists with fundamental theories and rationale for technique usage. Before attending any of the trainings, therapists were asked to complete a full battery of questionnaires, consisting of the TBQ, Evidence-Based Practice Attitude Scale (EBPAS; Aarons, 2004), KEBSQ (Stumpf et al., 2009), and Modified Practice Attitude Scale (MPAS; Borntrager et al., 2009). The EBPAS, MPAS, and KEBSQ were administered after each of the ANX and DBD workshops. Although the EBPAS and MPAS questionnaires were useful for previous studies on this training effort (e.g., Lim et al., 2012; Nakamura et al., 2011), previous research by McLennan et al. (2019) suggests that these therapists’ attitudes toward EBPs were not a significant predictor of subsequent technique utilization rates for either anxiety or disruptive behavior PDEBs, thus these measures were not leveraged for the current investigation.
Data Structure
To examine youths’ clinical progress patterns following their therapists’ workshop attendance, cross-classified multilevel modeling analyses were used. A cross-classified model allows for exploring clients’ clinical progress by examining multiple therapists and client combinations over a 4-year period (i.e., 2 years pretraining and 2 years posttraining), thus representing a combination of nested and cross-classified relationships (Heck & Reid, 2016; Heck et al., 2014). As an example, within this study, a client could have been seen by more than one therapist at different time points (not simultaneously) during the investigation. However, not all clients were necessarily seen by more than one therapist. Thus, the current investigation’s client and therapist factors are neither fully crossed nor strictly nested in the data structures.
Defining the predictor variables
Within this cross-classified multilevel model, youth’s month in their treatment episode was defined as the Level 1 unit of analysis, youth were defined as the Level 2 unit of analysis, and therapists were defined as the Level 3 unit of analysis. The Level 1 predictor was (a) month in treatment (i.e., monthly MTPS per youth). Youth’s month in treatment was captured through coding each individual MTPS that a youth had within their treatment episode (i.e., a youth’s first MTPS within a treatment episode was coded 0 and their sixth MTPS within the same treatment episode was coded 5).
Regarding youth predictors, (b) therapists’ training attendance, (c) therapists’ appropriate use of the package of trained PDEBs (i.e., therapists’ true-positive utilization rates), and additional interaction and cross-level interaction variables using these terms (further explained below) were included at Level 2. For therapists’ training attendance, youth and their associated MTPS reports were dichotomously coded as occurring either before or after (i.e., 1 = yes, 0 = no) the ANX or DBD trainings. More specifically, youth were coded as having received treatment either before their therapist received training (i.e., pretraining youth) or after their therapist received training (i.e., posttraining youth). Given this codification, therapists’ training attendance was considered a Level 2 predictor. In regards to therapists’ appropriate use of the package of trained PDEBs, McLennan et al. (2019) determined a therapist’s true-positive instance as the usage of workshop-focused PDEBs given the condition that one or more relevant treatment targets were also endorsed on the MTPS. Thus, for instance, a true positive for the intervention of exposure would occur if it were endorsed as occurring in the same month that anxiety was a treatment target. True-positive instances (e.g., also referred to as appropriate usage in this study) of the five workshop-targeted ANX PDEBs were defined as endorsement of these PDEBs when one or more of the treatment targets for anxiety were also checked. Furthermore, appropriate usage of the seven workshop-targeted DBD PDEBs was defined as endorsement of these PDEBs when one or more of the treatment targets for disruptive behaviors were checked. The crosswalk between the MTPS Treatment Targets and Intervention Strategies sections for determining therapists’ true-positive technique utilization was informed through a scoring guide from PracticeWise, LLC (PracticeWise, LLC, 2013) and is consistent with McLennan et al. (2019). Therapists’ use of the total package of trained PDEBs was used to acknowledge that there may be scenarios in which application of workshop-targeted techniques seemed clinically contraindicated for every month of treatment (e.g., utilization of exposure during the first month of treatment for a patient who is not yet ready for such a practice).
Defining therapists’ PDEB utilization was analyzed per treatment episode, through dichotomously coding (i.e., 1 = yes, 0 = no) for the presence or absence of the five ANX and seven DBD workshop-focused PDEBs across all MTPSs comprising treatment episodes. This method allowed for the examination of the extent to which therapists used (i.e., true-positive rate) or failed to use (i.e., false-negative rate) PDEBs at least once throughout the course of a client’s treatment episode. For example, suppose a therapist treated one anxious client before attending the ANX workshop. If that therapist used the ANX workshop-focused PDEBs of exposure, self-monitoring, and relaxation at least once per MTPS during the course of treatment, the trained PDEB true-positive score would be three (out of a possible five ANX workshop-focused PDEBs) for that client’s service episode. However, if the therapist did not use the ANX workshop-focused PDEBs of psychoeducation for anxiety and cognitive/coping even once during the service episode, the trained PDEB false-negative score would be two (out of a possible five ANX workshop-focused PDEBs). As seen in this example then, the maximum trained ANX true-positive and false-negative scores could range from 0 to 5 each, for each youth’s service episode. Similarly, the maximum trained DBD true-positive and false-negative scores could range from 0 to 7. It should be noted that therapists’ true-positive utilization rates are inverses of therapists’ false-negative utilization rates and are fully dependent on each other. As such, the current investigation only used therapists’ true-positive utilization rates (i.e., hereafter referred to as therapists’ appropriate use of the trained PDEBs) as the primary utilization rate predictor variable. For further clarification, this variable could be considered “the PDEBs associated with the training that a client received,” thus it was a client-level variable.
In building the final model for this investigation, additional interaction and cross-level interaction variables were also added to better understand the variation in youths’ clinical progress ratings. An additional Level 2 interaction term that was incorporated within the final model was (d) therapists’ training attendance by their appropriate use of the trained PDEBs (i.e., hereafter referred to as the trained PDEBs used after the training). This variable restricts the predictor to only examining therapists who went to the training and analyzes the effect of their use of the trained PDEBs on youth clinical progress. Moreover, this Level 2 interaction variable serves as a building block for the additional three-way cross-level interaction variable of (e) youth’s month in treatment by therapists’ training attendance and their appropriate use of the package of trained PDEBs (i.e., hereafter referred to as the slope of the trained PDEBs used after the training). This variable examines the effects of therapists’ use of the trained PDEBs after attending the training on youth clinical progress over the course of their treatment episode.
In order to also examine the potential moderating effects of therapists’ posttraining knowledge scores on youth outcomes, the current investigation included a Level 3 variable of (f) therapists’ posttraining KEBSQ ANX/DBD Target scores (i.e., hereafter referred to as therapists’ posttraining knowledge of the trained PDEBs: all clients) and an additional cross-level interaction variable of (g) therapists’ training attendance by their posttraining knowledge of the trained PDEBs (i.e., hereafter referred to as therapists’ posttraining knowledge of the trained PDEBs: posttraining clients). For the therapists’ posttraining knowledge of the trained PDEBs variable, therapists’ EBP knowledge (post-relevant training) was examined using therapists’ post KEBSQ ANX Target scores and post KEBSQ DBD Target scores. Within the final model, therapists’ posttraining knowledge of trained PDEBs: all clients, represented the effect of therapists’ posttraining knowledge on all youths’ progress ratings, regardless of their therapists’ training attendance; while therapists’ posttraining knowledge of the trained PDEBs: posttraining clients examined the effect of therapists’ posttraining knowledge on youths’ progress ratings for only clients seen after the training.
Defining the outcome variables
Youth clinical progress was measured by the slope of each individual MTPS progress rating for up to the first 6 months of youths’ treatment episodes that fell entirely before or after a training. More specifically, youths’ clinical progress was estimated by averaging each youths’ monthly progress ratings of either up to the four anxiety treatment targets checked off by the therapist or up to the seven disruptive behavior treatments targets when appropriate. Youth’s status at the end of their treatment episode defined the intercept, which indicates where the youth finished within the current study. Furthermore, the additional predictors defined above were added to explain the differences in intercepts and the slopes of youth’s progress ratings.
Analytic Strategy
Power analysis
For the final cross-classified models, the major element of analysis was the number of clients (with multiple repeated measures nested within these individuals). The current investigation ran power analyses using Snijders’ (2005) power formula for each parameter utilized within the final model analyses. More specifically, t-ratios and estimated power calculations were run for each individual standardized parameter included within the model to better understand if the current investigation had significant power to detect an effect for each factor. Based on these calculations, it appeared that our sample sizes of 324 anxious youth and 538 disruptive behavior youth (and with time-related measurements on the outcomes within individuals ranging between one and six, respectively) were enough for detecting standardized estimates ranging from as small as .27 for changes in the outcomes over youth’s month in treatment (power > .80) to factor effects on the outcome ranging from as small as .13 (power = .78) to .19 or larger (power > .80). Results also suggested that our Level 3 set of analyses may have been underpowered due to the limited number of therapists (i.e., 21 therapists included for the ANX training and 35 therapists included for the DBD training) at this level. For example, therapists’ level standardized estimates ranged from as small as .10 (power = .50) to as large as .14 (power = .07) for changes in outcomes due to therapists’ posttraining knowledge scores.
Determining model fit
The shapes of the within-subject growth trends for the outcome variables (i.e., slopes of youth progress ratings) were found to be linear in nature. Given the specific way that youth’s month in treatment was coded (i.e., 5, 4, 3, 2, 1, 0) a scaled identity covariance structure at Level 1 was determined to be the best fitting covarying matrix for our repeated measures analyses.
Results
Final Models
Tables 1 and 2 provide the results of the fitted longitudinal fixed-effect analyses, which for the most part generally support the predicted hypotheses.
Final Model With Intercept Placed at the End of Treatment Results for the Anxiety Training.
Note. PDEB = practice elements derived from the evidence base.
This variable is a cross-level interaction variable across Level 1 and Level 2. bThis variable is a cross-level interaction variable across Level 2 and Level 3.
p < .05. ***p < .001.
Final Model With Intercept Placed at the End of Treatment Results for the Disruptive Behavior Training.
Note. PDEB = practice elements derived from the evidence base.
This variable is a cross-level interaction variable across Level 1 and Level 2. bThis variable is a cross-level interaction variable across Level 2 and Level 3.
p < .05. **p < .01. ***p < .001.
ANX final model
Overall, youths’ month in treatment demonstrated that untreated clients’ treatment progress was positive and changed significantly over the course of a youth’s treatment episode (β = 0.19, p < .001), supporting Hypothesis 1a, which suggests that therapists reported that youths’ clinical progress increased over youths’ treatment episodes.
Results of the Level 2 predictors indicated that therapists’ training attendance alone was not a significant predictor of youth treatment progress (β = −2.64, p = .23). However, therapists’ use of the trained ANX PDEBs was a significant predictor of youth outcomes (β = −0.39, p = .05), such that the youth at the end of treatment had lower package of trained ANX PDEB scores. Furthermore, results of therapists’ appropriate use of the trained PDEBs after the training indicated that for those clients whose therapist used more of the trained ANX PDEBs after the ANX training, their therapists reported significantly higher progress ratings at the end of the youth’s treatment episode (β = 0.33, p = .05). Further explained and supporting Hypothesis 1ai, clients whose therapist used more of the ANX trained PDEBs after the ANX training had progress rating scores that were 0.33 higher at the end of their treatment episode than the youth seen before the training (i.e., β = 6.79). Moreover, when examining the primary aim of the study (i.e., the effect of therapists’ appropriate use of the ANX trained PDEBs after attending the training on youth clinical progress over their treatment episodes), the results did not support our Hypothesis 2a predictions and demonstrated that the slope (growth rate) of therapists who attended the training and used more of the package of trained PDEBs did not significantly change youth progress ratings over the course of their treatment episode (β = 0.01, p = .37).
In addition, therapists’ posttraining knowledge of the specific trained ANX PDEBs (i.e., therapists’ posttraining KEBSQ ANX Target score) was not a significant predictor of youth outcomes (β = −0.49, p = .41), before or after the training. Moreover, when examining results related to Hypothesis 3a, therapists’ posttraining knowledge scores of the trained PDEBs specifically for clients seen only after the training (i.e., see Posttraining Knowledge of Trained PDEBs: Posttraining Clients in Table 1), therapists’ posttraining knowledge of the trained PDEBs was still not a significant moderator of youth outcomes (β = 0.55, p = .23).
DBD final model
Similar to the ANX final model, results in Table 2 show that the untreated clients’ treatment progress was positive and changed significantly over the course of their treatment episodes (β = 0.18, p < .001), thus supporting Hypothesis 1b postulating that the slope of youths’ clinical progress significantly and positively changed over their treatment episodes.
Regarding Level 2 predictor results, therapists’ training attendance demonstrated that the training alone was not a significant predictor of youth treatment progress (β = −0.16, p = .78). Furthermore, and in contrast to the ANX final model results, the total number of DBD PDEBs that therapists used from the package of trained PDEBs (i.e., see Trained PDEBs used in Table 2) was not a significant predictor of youth outcomes alone (β = −0.07, p = .39). However, similar to the ANX final results, therapists’ use of the trained DBD PDEBs after the training was a significant predictor of youth outcomes. The results of therapists’ appropriate use of the trained PDEBs after the training supported Hypothesis 1bi, which indicated that for those clients whose therapist used more of the trained DBD PDEBs after the DBD training, their therapists reported significantly higher progress ratings at the end of the youth’s treatment episode (β = 0.13, p < .05). Further explained, clients whose therapist used more of the DBD trained PDEBs after the DBD training had progress rating scores that were 0.13 higher at the end of their treatment episode than the youth seen before the training (i.e., β = 3.78). Contrary to the ANX final model, the effect of therapists who attended the training and used more of the DBD PDEB training package over the course of youths’ treatment episodes (see Slope of the trained PDEBs used after the training in Table 2) was a significant predictor of youth outcomes (β = 0.03, p < .001). These results support Hypothesis 2b, such that these findings suggest that for therapists who attended the DBD training and used more of the package of DBD PDEBs, they reported significant positive slopes (greater rates of change) for disruptive behavior youth clinical progress over the course of their treatment episodes.
Regarding therapist level predictors, therapists’ posttraining knowledge of the specific trained PDEBs (i.e., therapists’ posttraining KEBSQ DBD Target score) was not a significant predictor of youth outcomes (β = 0.13, p = .46) before or after the training. Furthermore, therapists’ posttraining knowledge scores of the specific trained PDEBs, for only those clients seen after the training (i.e., see Posttraining knowledge of trained PDEBs: posttraining clients in Table 2), also was not a significant predictor of youth outcomes (β = −0.02, p = .86). Thus, similar to the ANX results, these findings did not support Hypothesis 3b.
Discussion
The current study aimed to examine the extent to which youths’ clinical progress improved after their therapists’ attendance at appropriately matched workshop trainings. More specifically, the present investigation studied the effects of therapists’ anxiety workshop attendance on anxious youths’ clinical progress and therapists’ disruptive behavior workshop attendance on disruptive behavior youths’ clinical progress. The current study was one of the first of its kind to examine public mental health therapists’ behavioral changes on youth outcomes using modularized approaches to training and treatment.
Youths’ Month in Treatment–Related Hypotheses (1a and 1b)
The results for both the anxiety and disruptive behavior hypotheses (Hypothesis 1a and Hypothesis 1b, respectively) supported the prediction that youths’ clinical progress would significantly and positively change over the course of treatment. Specifically, as predicted in Hypothesis 1a, results indicated that therapists who saw youth prior to attending the ANX training reported significant positive slopes for youth progress ratings over the course of treatment for these anxious clients. Similarly, for those therapists who saw clients before attending the DBD training, they also reported significant positive slopes in youth progress ratings over the course of treatment for their disruptive behavior youth (Hypothesis 1b). These results are in line with recent usual care literature which suggests that youth clinical progress generally improves throughout the course of treatment (Garland et al., 2014; Love et al., 2014), suggesting a potential trend toward youth improvement in response to community mental health care more broadly.
Therapists’ Training Attendance and Use of the Trained Techniques Over Youth’s Month in Treatment–Related Hypotheses (1ai, 1bi, 2a, and 2b)
Results related to the ANX training demonstrated that while therapists’ ANX training attendance alone was not a significant predictor of anxious youths’ clinical progress, significant findings were found for therapists’ use of the trained PDEBs. Results indicated that therapists in general (i.e., without taking into account their training attendance) reported lower progress ratings at the beginning of treatment for clients with which they used more of the trained PDEBs. However, therapists’ specific use of the trained ANX PDEBs after attending the ANX training was a significant positive predictor of youth outcomes. Specifically, therapists who used more of the ANX PDEBs right after attending the ANX training, reported that their anxious clients’ clinical progress was significantly higher at the end of treatment than their anxious clients who were seen prior to their training attendance (Hypothesis 1ai). However, when examining these effects more wholly over time, results demonstrated that therapists’ ANX training attendance and their use of the package of trained ANX PDEBs was not a significant moderator of youth clinical progress changes over the course of their treatment episodes. Thus, Hypothesis 2a was not supported.
Results related to the DBD training demonstrated subtly different outcomes, compared with the ANX training findings. Like the ANX training results, the DBD training was not a significant predictor of youth clinical outcomes. However, therapists’ use of the package of trained DBD PDEBs directly after the DBD training was a significant predictor of youth treatment progress (Hypothesis 1bi). Contrary to the ANX training findings, Hypothesis 2b was supported for the DBD trainings. Results found that therapists’ DBD training attendance and their use of the trained DBD PDEBs moderated youth clinical progress over the course of their treatment episodes. Specifically, therapists who used more of the package of trained DBD PDEBs after attending the DBD training, reported significant positive slopes (i.e., greater rates of change) for youths’ clinical progress ratings throughout treatment for their disruptive behavior clients.
These findings suggest that therapists’ behavioral responses to training can vary depending on the problem type being addressed and, in turn, may affect client treatment progress in a variety of ways. In particular, given the differential moderating effects between therapists’ specific training attendance and their use of the trained PDEBs on youths’ clinical progress, results may be in agreement with previous research which suggests that disruptive behavior youths’ treatment progress may improve the most over the course of time when compared with youth with anxiety and attention-deficit/hyperactivity disorder (Love et al., 2014). Results may also be in line with recent research, which suggests that therapists’ experience with certain problem areas may influence their use of specific practice techniques. Specifically, one study using a similar public sector mental health population to that of the current investigation, found that therapists reported being more comfortable using externalizing PEs with youth rather than internalizing PEs (Milette-Winfree, 2017). Thus, our results could be in part due to CAMHD’s therapists’ experiences. As noted, CAMHD therapists treat more disruptive behavior youth than anxious youth, and they thus may have felt more experienced in using the disruptive behavior PDEBs over time. Further speaking to therapists’ experience potentially affecting their specific use of the PDEBs, the current results may highlight further nuances and complexities to dissemination and implementation training efforts. One speculative explanation for these results may suggest that therapists’ experience with a problem area could influence the types of training designs needed to further help therapists learn and maintain skills over time, and in turn, positively enhance youth treatment outcomes. For example, given that CAMHD therapists largely see an externalizing youth population, and, as McLennan et al. (2019) suggested, the trainings may have clarified the particular PDEB definitions, it may be that CAMHD therapists felt more comfortable using the ANX PDEBs immediately following the ANX training, but that to continue utilizing the ANX PDEBs over time they may have benefited from further consultation and follow-up. Furthermore, public sector mental health therapists may not train equally across problem areas. Thus, training and implementation supports may need to be considered differentially across problem areas (depending on therapists’ baseline experiences with those problem areas). Moreover, training public sector mental health therapists in internalizing techniques may require different and larger allocations of resources and training infrastructures than other techniques (Izmirian et al., 2016; McLennan et al., 2019; Milette-Winfree, 2017).
Therapists’ Posttraining Knowledge of the Trained Technique-Related Hypotheses (3a and 3b)
Results indicated that therapists’ posttraining knowledge was not a significant moderator of youth clinical progress. These results may be in line with the literature, which suggests that therapist adherence and competence are not significant moderators of treatment outcomes (Webb et al., 2010). However, another potential explanation for these non-significant findings could be that the current investigation did not have enough power at the therapist level to properly detect moderating effects of therapists’ knowledge on youth clinical progress. Given the lower number of therapists (n = 48) at Level 3, readers should be mindful of this potential issue when interpreting results. In addition, the particular type of treatment knowledge that the KEBSQ reportedly assesses may not have captured particularly relevant knowledge within this investigation. Moreover, while the KEBSQ measures therapists’ awareness that certain PEs are associated with the treatment outcome literature (Okamura et al., 2014) for multiple PEs, it does not capture therapists’ procedural knowledge in implementing an intervention, which may be an important distinction for the present investigation.
Overall, these results suggest several elements may be influential to therapists’ practice changes and youth clinical progress following participation in trainings on modular approaches to therapy. Although these results should be interpreted with caution, findings potentially demonstrate that youth’s month in treatment, therapists’ training attendance, and therapists’ practice behaviors may influence youth clinical progress. In particular, therapists’ practice behaviors may be differentially affected based on the specific problem area being addressed at a workshop, and in turn, may influence youth outcomes in a variety of ways.
Limitations
Within the present investigation, there are several limitations that warrant discussion. First, this investigation relied exclusively on therapist self-report measures, and while some research has suggested that therapists can self-report accurate descriptions of their behaviors within treatment sessions, other studies have noted the potential for therapists to over-report their in-session technique utilization (Borntrager et al., 2013; Nakamura et al., 2014). The issue of using only therapist-report instruments is also compounded by the nature of the one outcome variable within this study, namely progress ratings for relevant anxiety and disruptive behavior targets. Not only was there the possibility that therapists could have over-reported their clients’ progress, but other dimensions of progress (e.g., functional impairment) were not assessed. Given this limitation, future research may seek to examine youth progress using multiple measures and multiple informants of youth treatment outcomes.
Another limitation is the current study’s inability to determine if self-reported use of a particular PE was specifically for either the problems of anxiety or disruptive behavior. More clearly, therapists were not asked to indicate which PE was used for a particular treatment target on the MTPS, thus making it difficult to parse out therapists’ decision-making when utilizing a PE that is considered to be derived from the evidence bases for more than one treatment target. As an example, for a therapist who indicated addressing the two treatment targets of anxiety and depression, while also endorsing the PE of relaxation on an MTPS, it was unclear if the therapist used relaxation for anxiety and/or depression. Thus, results should be interpreted with some caution, and future research may consider using more specific measurement strategies whereby the crosswalk between specific targets and PEs are more fine-grained in nature.
In addition, while CAMHD’s state-wide public sector mental health training was unique in several ways, the workshop trainings were substantially limited in their ability to fully meet the standard of evidence-based training programs. Although all trainings were codified, incorporated both active and passive learning strategies, focused on a small number of PDEBs and their modular assembly, and allowed attendees to attend trainings in any desired order, the current workshops were limited to 1-day trainings without further consultation from training experts. Future training research may aspire to examine the effects of modular trainings that incorporate further consultation and their influence on therapists’ behaviors and youth clinical progress. At the same time, however, given this limitation in the current investigation’s training paradigm, it may be particularly noteworthy that even some significant findings were revealed. Results seem to potentially suggest that a modular training paradigm that incorporates aspects of active strategies, may be a somewhat fruitful area for influencing therapists’ practices changes, and in turn, youth outcomes. Furthermore, our results also suggest that such practice behaviors may be even further moderated by the problem area addressed at the training.
Although this study is not without limitations, the current findings may aid the field in developing and improving dissemination and implementation of EBPs into community settings. These results, along with previous research examining these workshop trainings (cf. McLennan et al., 2019), support the notion that modular training in modular therapy may influence therapists’ utilization of specific therapeutic PDEBs and in turn, affect youth outcomes in a positive way. However, given the numerous limitations listed above, and this being one of the first demonstrations of modular trainings’ effects on youth outcomes, the only thing that we can say with confidence at this time is that more work is needed, preferably in public mental health settings to maximally help large numbers of youth and families. For example, future work may wish to further examine the relationship between therapists’ behavior changes in response to workshops on specific problem areas (other than anxiety and disruptive behavior as examined in this study) and youth clinical progress. Furthermore, given the differential influences that the anxiety and disruptive behavior trainings had on therapists’ practice behaviors and youth clinical progress, future research may seek to further our understanding of this seemingly moderated effect. Indeed, as previous research has suggested, there may be many different factors that moderate youth treatment outcomes, especially at the patient and family level. As such, future research may aim to examine the components of modular trainings that lead to specific changes in therapists’ behaviors, youth outcomes, and how these effects may be sustained over time.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
