Characterizing therapist delivery of evidence-based intervention strategies in publicly funded mental health services for children with autism spectrum disorder: Differentiating practice patterns in usual care and AIM HI delivery

Abstract

This study identified patterns of therapist delivery of evidence-based intervention strategies with children with autism spectrum disorder within publicly funded mental health services and compared patterns for therapists delivering usual care to those trained in AIM HI (“An Individualized Mental Health Intervention for ASD”). Data were drawn from a randomized community effectiveness trial and included a subsample of 159 therapists (86% female) providing outpatient or school-based psychotherapy. Therapist strategies were measured via observational coding of psychotherapy session recordings. Exploratory factor analysis used to examine patterns of strategy delivery showed that among therapists in the usual care condition, strategies loaded onto the single factor, General Strategies, whereas for therapists in the AIM HI training condition, strategies grouped onto two factors, Autism Engagement Strategies and Active Teaching Strategies. Among usual care therapists, General Strategies were associated with an increase in child behavior problems, whereas for AIM HI therapists, Active Teaching Strategies were associated with reductions in child behavior problems over 18 months. Results support the effectiveness of training therapists in evidence-based interventions to increase the specificity of strategies delivered to children with autism spectrum disorder served in publicly funded mental health settings. Findings also support the use of active teaching strategies in reducing challenging behaviors.

Lay abstract

This study was conducted to identify patterns of therapist delivery of evidence-based intervention strategies with children with autism spectrum disorder receiving publicly funded mental health services and compare strategy use for therapists delivering usual care to those trained to deliver AIM HI (“An Individualized Mental Health Intervention for ASD”), an intervention designed to reduce challenging behaviors in children with autism spectrum disorder. For therapists trained in AIM HI, intervention strategies grouped onto two factors, Autism Engagement Strategies and Active Teaching Strategies, while strategies used by usual care therapists grouped onto a broader single factor, General Strategies. Among usual care therapists, General Strategies were related to an increase in child behavior problems, whereas for AIM HI therapists, Active Teaching Strategies were related with reductions in child behavior problems over 18 months. Findings support the use of active teaching strategies in reducing challenging behaviors in children with autism spectrum disorder and provide support for the effectiveness of training therapists in evidence-based interventions to promote the delivery of targeted, specific intervention strategies to children with autism spectrum disorder in mental health services.

Keywords

autism spectrum disorder evidence-based intervention strategies mental health services

Implementing evidence-based interventions (EBIs) in community service settings is a public health priority to improve the effectiveness of mental health (MH) care (Kilbourne et al., 2007). This priority is fueled by knowledge that empirically supported psychotherapies for youth lead to better outcomes than treatment as usual (Weisz et al., 2013). Training community therapists to deliver EBIs is an essential component of implementation efforts (Beidas & Kendall, 2010; Herschell et al., 2004); however, large variability exists in therapist practices within community MH settings (Becker-Haimes et al., 2019). Characterizing therapist practice patterns is essential for identifying care improvement targets (Garland et al., 2010), informing implementation efforts (Becker-Haimes et al., 2019), testing the effectiveness of implementation and therapist training efforts (Lau & Brookman-Frazee, 2015), and understanding the associations between therapist practice and child outcomes (Brookman-Frazee et al., 2019).

Serving children with autism spectrum disorder within community MH settings

There is increasing recognition that many children with autism spectrum disorder (ASD) receive care in publicly funded MH services (Brookman-Frazee, Drahota, Stadnick, & Palinkas, 2012; Joshi et al., 2010). Challenging behaviors are typically the primary presenting problems for children with and without ASD served in these settings (Brookman-Frazee et al., 2009; Mandell et al., 2005). The majority of children with ASD served in outpatient and school-based MH services meet diagnostic criteria for at least one other non-ASD diagnosis, most commonly attention deficit hyperactivity disorder, and most meet criteria for more than one co-occurring condition (Brookman-Frazee et al., 2018).

In publicly funded MH settings treating a wide variety of diagnostic presentations, previous research has found that although community-based therapists are observed to deliver EBI strategies, the strategies are typically employed with low extensiveness (i.e. frequency and thoroughness) than outlined in EBI protocols (Brookman-Frazee, Haine, et al., 2010; Garland et al., 2010). A similar pattern is found for EBI strategies delivered with children with ASD in the same settings (Brookman-Frazee, Taylor, & Garland, 2010). Although children with ASD are served in MH programs, therapists delivering MH services report limited training or exposure to EBIs for ASD and have requested training and tools to adapt psychotherapy process for this population (Brookman-Frazee, Drahota, Stadnick, & Palinkas, 2012; Williams & Haranin, 2016).

Conceptualizing EBI strategies

There are multiple ways to characterize therapist EBI delivery. The “common elements” approach identifies therapist strategies that are common across EBI protocols (Boustani et al., 2017). Rather than focusing on a specific EBI protocol, this approach considers common elements across a variety of EBI treatment protocols that can be applied to a specific clinical presentation and are organized based on theory to deliver effective treatment. Identifying common treatment elements has been recognized as a helpful tool for characterizing therapists’ practices and training therapists in EBIs (Boustani et al., 2017; Garland et al., 2008). This process can also help to identify intervention mechanism key to therapeutic change, which can result in therapist training that specifies where individual adaptations are welcomed and which core elements must be maintained in order not to compromise intervention effectiveness.

Given the increasing numbers of children with ASD in community MH settings, there is a particular need to use the common elements approach to identify evidence-based strategies used across interventions that specifically target children with ASD receiving MH services. A recent literature review identified 28 focused EBI practices found to have positive effects for children with ASD across treatment domains (Steinbrenner et al., 2020). EBI strategies for ASD are often packaged in conceptually driven modules; however, it is not known how these conceptual combinations of strategies align with therapist practice patterns derived from empirical observationally driven data from psychotherapy sessions with children with ASD.

Training therapists in EBI strategies for ASD

In response to therapists’ request for training to increase the effectiveness of MH services for ASD and for tools to adapt psychotherapy for ASD, Brookman-Frazee and colleagues developed the AIM HI intervention (An Individualized Mental Health Intervention for ASD) (Brookman-Frazee & Drahota, 2010; Brookman-Frazee et al., 2016), a package of parent-mediated and child-directed strategies designed to reduce challenging behaviors in children 5–13 years with ASD (Brookman-Frazee & Drahota, 2010; Brookman-Frazee et al., 2016). AIM HI was designed for delivery in MH services and includes EBI strategies such as teaching parents to track patterns in child behaviors and identify potential external functions, and teaching alternative child skills and complementary parent strategies using active teaching strategies (modeling, behavioral rehearsal, feedback, and between-session practice). AIM HI also includes strategies to adapt psychotherapy sessions to maximize engagement and skill building among children with ASD, such as including the child’s special interests in treatment sessions and maximizing in session predictability through the use of schedules and visuals. The AIM HI protocol categorized strategies into child engagement/motivational strategies and strategies to facilitate skill building (i.e. “active teaching strategies”). Table 1 includes the individual therapist-delivered strategies taught to therapists in AIM HI training, and delineates how these strategies are conceptually divided within AIM HI clinical training. In a randomized community effectiveness trial, children whose therapists participated in ongoing AIM HI training/consultation had greater improvements in child behaviors (Brookman-Frazee et al., 2019) and their caregivers reported increased perceptions of parental self-efficacy (Brookman-Frazee et al., 2020) compared to children whose therapists delivered usual care. Changes in therapist use of increased session structure and treatment continuity, and changes in parental sense of competence mediated the effects of therapist training on child outcomes (Brookman-Frazee et al., 2019, 2020). To date, there has been no detailed examination of differences in therapist practice patterns in response to AIM HI training.

Table 1.

AIM HI clinically conceptualized domains for therapist child strategy use..

Clinically conceptualized domains	Therapist intervention strategies
Within-Session Elements to Increase Child Motivation and Engagement in Session	• Structuring sessions through use of an agenda
	• Using visuals/materials
	• Using ASD-specific motivational strategies (e.g. incorporating special interests)
	• Sharing control
Active Teaching Strategies to Facilitate Child Skill Building	• Providing psychoeducation about child skill
	• Modeling child skill
	• Offering practice
	• Providing feedback
	• Providing positive reinforcement for child use of skill
	• Providing between-session practice

AIM HI: An Individualized Mental Health Intervention for ASD; ASD: autism spectrum disorder.

Current study

Given the variability of strategy use observed during treatment of children with ASD in community-based settings, further understanding of patterns of strategy use between therapists delivering routine care and those trained in evidence-based practices may help to improve EBI measurement as well as training efforts. Rather than only looking at strategy use using a theoretical or clinical framework, research may benefit from using empirically driven approaches to understand therapist EBI strategy use, especially in community MH settings. Within the context of a large-scale community effectiveness trial, this study aims to take an empirical approach to characterize therapists’ child-directed intervention strategies when working with children with ASD in order to examine how strategies are implemented in routine community MH care settings and examine if training in a specific intervention, AIM HI, impacts the clustering of strategies. This study seeks to examine the impact of specific clusters of child-directed intervention strategies on child behavior outcomes. Child strategies, rather than parent strategies, were the focus of current analyses as these strategies involve the most specific ASD-tailored approaches within the AIM HI training. In addition, our previous work found a significant difference in parent attendance at children’s therapy sessions with parents whose therapists received AIM HI training attending a higher percentage of sessions compared to parents whose therapists delivered care as usual. Since parent attendance varied, but children were consistently present in sessions in both conditions, focusing on child-directed strategies allowed for examination of therapist practice patterns across conditions.

The aims of this study were to (1) use exploratory factor analysis (EFA) to examine the patterns of child-directed EBI strategies among the usual care and AIM HI therapists, and (2) examine the associations between the derived factors and changes in child behaviors over time (baseline to 18 months post baseline) for the usual care and AIM HI groups. We hypothesize that there will be different patterns of EBI strategy used based on whether the therapist received training in AIM HI, and that more extensiveness of EBI strategies within a session will be associated with more positive child outcome trajectories.

Method

Study context

Data for this study are drawn from a large-scale community effectiveness trial of AIM HI (“An Individualized Mental Health Intervention for ASD”), a package of EBI strategies designed to reduce challenging behaviors in children with ASD served in community MH settings. The AIM HI protocol was developed in collaboration with community providers and caregivers of children with autism. In the community effectiveness trial, publicly funded clinic and school-based MH programs were randomized to immediate AIM HI training or a wait-list control observation condition (i.e. “usual care”). Therapists from participating programs were recruited and enrolled with an eligible family on their caseload; therapist and client enrolled in the study as a participating “dyad.” This study was approved by the University of California, San Diego Institutional Review Board, and all study participants provided written consent to participate in the study.

AIM HI intervention

The AIM HI intervention is designed to reduce challenging behaviors in school-aged children with ASD by teaching children positive alternative skills and teaching parents antecedent and consequence-based strategies to promote their child’s use of skills. AIM HI is a package of evidence-based strategies that is specifically packaged for delivery in MH settings by therapists with little to no experience with ASD. All individual components of the intervention are considered “well established” for an ASD population (Steinbrenner et al., 2020; Wong et al., 2015). It consists of a series of protocol steps that included treatment planning, active teaching of child and caregiver skills, and evaluation of client progress. Therapists are trained to collaborate with parents to identify the primary functions of the child’s challenging behaviors, identify appropriate child alternative skills and complementary parent skills, and teach child and parent skills in session. Skills are taught to children and parents using “active teaching strategies,” which include therapist modeling of skills, behavioral rehearsal accompanied by targeted feedback and reinforcement, and assignment and review of between-session practice (e.g. homework).

In addition to strategies to teach skills, AIM HI includes “within-session elements” or strategies that therapists use in every AIM HI session to facilitate engagement and participation in skill building. As AIM HI was developed specifically for children with ASD, engagement strategies targeted to this population were built into the intervention to address the engagement challenges that have been reported from community providers working with this population (Brookman-Frazee, Drahota, Stadnick, & Palinkas, 2012). AIM HI within-session elements include strategies for structuring sessions through the use of session schedules and visuals and engaging ASD clients through the use of motivational approaches including child choice, shared control, and strategic integration of preferred interests (see Brookman-Frazee et al., 2020).

AIM HI clinical training

Therapist training occurred over the 6-month study period while therapists implemented AIM HI with their enrolled client. Therapists were trained in AIM HI using a multi-component approach: (1) introductory training workshop, (2) therapist self-study using a intervention manual and a resource website offering video exemplars and electronic copies of intervention materials, (3) 11 consultation meetings (9 group meetings and 2 individual sessions) with an expert AIM HI trainer who provided both dyadic instruction and case-specific feedback, and (4) performance feedback on therapist delivery of AIM HI based on trainer review of video recordings of therapy sessions.

Procedure

Therapists in the training condition received training and consultation for 6 months while they delivered AIM HI to a participating child/parent dyad while therapists in the wait-list condition were observed while they delivered routine care to a participating child/family for 6 months. Therapists in both conditions video recorded their therapy sessions with participating clients and submitted the recordings to the research team. The video recordings were submitted over a 6-month period and a subset of the recordings was coded using observational coding schemes (see description of the coding scheme in section “Measures”). In the effectiveness trial, data collection using surveys and/or interviews occurred at baseline (time of study enrollment) and 6, 12, and 18 months post enrollment for parents and baseline and 6 and 18 months post enrollment for therapists.

AIM HI observational coding system

Session recordings submitted by therapists in both conditions were coded by trained raters using the AIM HI Observational Coding System (Brookman-Frazee & Chlebowski, 2013). The development of the AIM HI Observational Coding System was based on established coding systems designed to measure EBI strategy use, including the Therapeutic Process Observational Coding System for Child Psychotherapy—Strategies scale (TPOCS-S; McLeod & Weisz, 2010) and the modified Practice and Research: Advancing Collaboration Therapy Process Observational Coding System for Child Psychotherapy—Specific Therapy Process Scale (PRAC-TPOCS-S; Garland et al., 2010).

The AIM HI Observational Coding System was designed to document the presence and intensity of 18 clinical strategies used by therapists, with ratings for strategies directed to children and caregivers coded separately. Using a rating system adapted from Garland et al. (2010), coders rated the extensiveness of EBI strategies according to both the frequency and intensity with which the strategy was observed to be used in session. Frequency indicates the number of times the strategy was observed during a session. Intensity reflects both the time in session spent on the strategy and the thoroughness with which the strategy was pursued. An extensiveness rating accounting for both frequency and intensity was rated at the end of each session for each clinical strategy. Ratings used a Likert-type scale ranging from 0 to 6, with a rating of 0 indicating that the clinical strategy was “not observed,” a rating of 1 or 2 denoting that the strategy was “used with low frequency or intensity,” a rating of 3 or 4 indicating that the strategy was “used with medium frequency/intensity,” and a rating of 5 or 6 specifying that the strategy was “used with high frequency/intensity.” This approach is consistent with previous community-based research that focuses on selecting therapeutic strategies that align with essential therapy elements and uses coding of session strategies as an adherence measure for intervention models (McLeod et al., 2015).

Coder training

A team of psychology undergraduate, postbaccalaureate, and postdoctoral coders was trained to reliably code observed session behaviors using the AIM HI Observational Coding System. Training was led by one of the authors of this study (C.C.), who is also one of the developers of the observational coding system and served as the master coder in developing criterion-rated sessions for practice and reliability. Training began with group didactics and group coding sessions to introduce the concepts being coded and to practice documenting and coding in real time. Coders then completed individual practice sessions followed by meetings with the coding trainer to discuss discrepancies with the criterion-rated sessions. Coders were considered trained and eligible to start independent coding when the coder achieved at least 80% agreement for each of the six or more criterion-rated sessions. Coding meetings and “booster” training sessions occurring monthly to prevent coder drift. The coding team was blind to participant condition and study hypotheses when coding.

Inter-rater reliability

In the parent community effectiveness trial (Brookman-Frazee et al., 2020), observer inter-rater reliability was calculated using a one-way random effects ICC(1,k) model based on a mean-rating, absolute-agreement. This model was applicable as a subsection of recordings (25%) was randomly selected for double-coding (Hallgren, 2012; McGraw & Wong, 1996). Inter-rater reliability for ratings of therapist in-session behaviors was good to excellent (Cicchetti, 1994); with one-way random effects ICC(1,k) ranging from 0.62 to 0.89, with the exception of one code (Shared Control) which had fair agreement (0.44). The average intraclass correlation coefficient (ICC) for the 10 child-directed strategies examined in the current analyses was 0.73 (range = 0.44–0.89).

Sampling sessions for coding

In the effectiveness trial, 1769 video recordings of therapy sessions were submitted by participating therapists during the 6 months training/consultation or usual care observation period. A total of 1387 sessions were coded (78% of the submitted recordings). Recordings were randomly selected for coding from 2-month windows representing the beginning (months 1–2), middle (months 3–4), and end (months 5–6) of treatment to allow for a balanced representation of sessions from all therapists (e.g. sessions covering the treatment planning, active teaching, and evaluating progress phases of treatment). The average number of coded sessions per child was 7.5 sessions (SD = 3.27).

Measures

Demographic Questionnaires

Demographics were collected from all participants. Both therapists and parents completed a questionnaire at study baseline that collected demographic information. Child demographic information was reported by parents. Therapists provided information about clinical training experience, MH discipline, and employment history, and parents self-reported their educational, employment, and income information.

Eyberg Child Behavior Inventory

The Eyberg Child Behavior Inventory (ECBI; Eyberg & Pincus, 1999) is a 36-item caregiver-report measure that assesses the frequency and intensity of child disruptive behaviors. The ECBI has two scales: ECBI Intensity, which represents the current frequency of disruptive behaviors and it is rated on a 7-point Likert-type scale, and ECBI Problem, which represents the total number of child behaviors that caregivers endorsed as a problem. Higher scores indicate higher intensity of behavior problems (ECBI Intensity) or increased parental perception of the behavior as problematic (ECBI Problem). The ECBI was completed at baseline and 6, 12, and 18 months post baseline by parents in the usual care and AIM HI conditions.

Participants

Therapists were eligible for participation in the AIM HI effectiveness trial if they were employed as trainee or staff at a participating program and had a child on their caseload meeting the client inclusion criteria. Child client inclusion criteria included aged between 5 and 13 years at time of study enrollment, having an existing diagnosis of an ASD or exhibiting clinically significant ASD symptoms on at least one of two standardized ASD diagnostic measures: the Autism Diagnostic Observation Schedule-2 (ADOS-2; Lord et al., 2012) and the Social Responsiveness Scale-2 (SRS-2, Constantino & Gruber, 2012) and presenting for MH treatment with a challenging behavior. Parents were eligible if they were the parents of a child meeting eligibility criteria being served in community MH settings (e.g. publicly funded outpatient clinic or school-based program).

Data from a subsample of 159 therapists (38 usual care and 121 AIM HI group) and 180 families (41 usual care and 137 AIM HI group) were included in the current analyses; more families were enrolled as a subset of therapists (n = 21) enrolled in the study with more than one family. Family–therapist dyads from the trial were excluded from the current analyses if they did not have session recordings available that included observational coding of child-directed strategies. Table 2 shows the participant characteristics.

Table 2.

Therapist and caregiver participant characteristics..

	Usual care	AIM HI
Therapist characteristics
Therapist age (years), M (SD)	35 years (8.10)	34 years (8.34)
Therapist gender (female), % (n)	87% (33)	85% (103)
Therapist education level
Bachelor’s degree, % (n)	11% (4)	9% (11)
Master’s degree, % (n)	76% (29)	78% (94)
Doctoral degree, % (n)	8% (3)	11% (13)
Primary mental health discipline
Marriage and family therapy, % (n)	50% (19)	44% (53)
Social work, % (n)	26% (10)	28% (33)
Psychology, % (n)	13% (5)	21% (25)
Other discipline^a, % (n)	10% (4)	8% (9)
Trainee % (n)	16% (6)	36% (43)
Caregiver/family characteristics
Child age (years), M (SD)	9 years (2.39)	9 years (2.45)
Child gender (male), % (n)	83% (34)	83% (113)
Child % Hispanic/Latinx, % (n)	66% (27)	57% (78)
Caregiver age (years), M (SD)	38 years (7.02)	41 years (8.75)
Caregiver gender (female), % (n)	98% (40)	93% (128)
Caregiver % Hispanic/Latinx, % (n)	54% (22)	51% (70)
Maternal education level
Less than high school, % (n)	15% (6)	21% (29)
Completed high school, % (n)	50% (20)	33% (45)
Any trade school/college, % (n)	35% (14)	46% (62)
Annual household income
⩽US$25,000, % (n)	59% (24)	42% (57)

AIM HI: An Individualized Mental Health Intervention for ASD; SD: standard deviation.

Other discipline includes psychiatry, licensed professional clinical counselor.

Comparisons were made between the usual care and AIM HI groups to determine if the two groups differed on key therapist or caregiver demographic characteristics. Consistent with previous results reported from the parent trial with a slightly larger sample size (Brookman-Frazee et al., 2019, 2020), overall there were very few significant differences between groups in this subsample. Significantly more therapists were trainees in the AIM HI group than in the usual care group t(80) = −2.66, p < 0.01. There were no differences between therapists in the AIM HI group and therapists in the usual care group on therapist gender, age, education, or primary MH discipline. Furthermore, there were no significant differences between parents in the AIM HI group versus the usual care group on caregiver gender, age, ethnicity, maternal education, or household income. There were also no differences between the two groups in child age, gender, ethnicity, or severity of behavior problem at baseline.

Analytic plan

Aim 1 sought to explore the factor structure of the child-directed strategies among the usual care sample and AIM HI sample, respectively. Multilevel exploratory factor analyses were used to investigate the factor structure of child-directed therapist strategies using geomin rotation. Analyses were completed using Mplus (Eighth edition; Muthén & Muthén, 1998–2017). Factor loadings of greater or equal to 0.4 were used as the criterion for item inclusion (Taherdoost et al., 2014). The current analyses used the Root Mean Square Error of Approximation (RMSEA) and the Comparative Fit Index (CFI) to evaluate model fit. Acceptable model fit was determined by an RMSEA of less than or equal to 0.08 and CFI greater than 0.95 (Hu & Bentler, 1999). Mean values and standard deviations were reported for the derived factors based on observer ratings from 220 videos from the usual care sample and 903 videos from the AIM HI sample.

Aim 2 sought to examine the associations between the derived factors within the usual care and AIM HI groups and the trajectory of child behaviors over time (baseline to 18 months post baseline) as measured by the ECBI Intensity and ECBI Problem scales using multilevel linear regression. The relationships between the factors and the ECBI scales were examined using the mean composite for each factor. Control variables were determined based on univariate analyses showing significant relationships between specific child demographics and ASD severity and the ECBI Intensity and ECBI Problem scales, and differed between the AIM HI and usual care group (control variables for AIM HI: primary setting of service, child gender, and child race/ethnicity; control variables for usual care: child age, child gender, and child ASD severity). Analyses were completed using Mplus (Eighth edition; Muthén & Muthén, 1998–2017).

Results

Aim 1: exploratory factor analyses

Usual care sample

In the usual care sample, results loaded on a single-factor between-level model with unrestricted within-level covariance, which demonstrated excellent model fit χ²(20) = 43.36, p < 0.01; CFI = 0.94, RMSEA = 0.00. The model excluded two strategies (incorporating child interests in session and sharing control with the child in session) which did not load when all items were initially included (see Table 3). The items that loaded onto the single factor were related to teaching child skills in session and using materials and were labeled General Strategies.

Table 3.

Geomin rotated loadings for child-directed therapist strategies.

	Usual care sample	AIM HI sample
	Factor 1: general strategies	Factor 1: autism engagement strategies	Factor 2: active teaching strategies
Used agenda	0.89 *	0.63 *	0.13
Used materials	0.74 *	0.51 **	0.56
Incorporated interests	Did not load	0.86 *	−0.34
Shared control	Did not load	0.80 *	−0.05
Provided psychoeducation	0.85 *	0.08	0.55 *
Modeled	0.88 *	−0.013	0.89 *
Offered in-session practice	1.01 *	0.25	0.82 *
Provided feedback	0.76 *	0.68	0.18
Positive reinforcement	0.80 *	0.70	0.18
Between-session practice	0.64 *	−0.10	0.56 *

AIM HI: An Individualized Mental Health Intervention for ASD.

Factor loadings of greater or equal to 0.4 were used as the criterion for item inclusion. Bold values indicate that strategy was included in final factor.

Loading is significant at the p < 0.05 level; **strategy cross loaded and was determined to conceptually fit on Factor 1.

AIM HI sample

For the AIM HI group, results indicated preference for the two-factor between-level model with unrestricted within-level covariance, which demonstrated excellent model fit χ²(26) = 133.797, p < 0.001; CFI = 0.96, RMSEA = 0.03 (see Table 3).

Six strategies loaded on to the first factor consisted of items relating to optimizing a therapy session specially to engage a child with autism and were labeled Autism Engagement Strategies. The factor items included using a session schedule, using visual materials, incorporating child’s interests into session, sharing control of session activities with the child, providing feedback, and providing positive reinforcement (all strategies directed toward child client). Using this study’s cutoff of 0.4, the Using Visual Materials item cross loaded onto both of the two factors; however, this item was included on the first factor as it was determined by experts in the AIM HI intervention to conceptually “fit” with Autism Engagement Strategies.

The second factor consisted of four strategies related to teaching skills in session and included providing psychoeducation to child, modeling strategies to child, offering the child the opportunity for in-session practice, and making a plan for between-session practice and was labeled Active Teaching Strategies. The autism engagement and active teaching factors were significantly correlated (r = 0.26, p < 0.05).

Characterizing strategy extensiveness using derived factors

Table 4 includes the mean values and standard errors (SEs) for the usual care single factor and the two factors derived for the AIM HI group. Mean values and standard deviations are included for the mean factor score.

Table 4.

Descriptives of factor scores of child-directed therapist strategies.

	Usual care sample	AIM HI sample
	Factor 1: general strategies	Factor 1: autism engagement strategies	Factor 2: active teaching strategies
Mean	2.37	4.16	2.80
SD	0.96	1.14	1.53
Range	0.83–3.96	0.17–6.00	0–6.00

AIM HI: An Individualized Mental Health Intervention for ASD; SD: standard deviation.

Factor scores range from 0 to 6.

Aim 2. Using factors to predict child outcomes

Usual care group

Among the usual care group, when controlling for child age, child gender, and autism symptom severity, the General Strategies factor was associated with increases in the ECBI Intensity scale (B = 0.75, SE = 0.30, p = 0.012, 95% confidence interval (CI) = 0.17, 1.32) and ECBI Problem scale (B = 1.18, SE = 0.45, p = 0.009, 95% CI = 0.30, 2.07) over time, from baseline to 18 months.

AIM HI group

Among the AIM HI group, when controlling for setting, child age, child race/ethnicity, and autism severity, factor one Autism Engagement Strategies was not significantly associated with changes in the ECBI Intensity scale B = −0.05, SE = 0.35, p = 0.89, 95% CI = −0.73, 0.63 or ECBI Problem scale B = −0.21, SE = 0.45, p = 0.64, 95% CI = −1.10, 0.67 over time; however, higher Active Teaching Strategies (factor two) was associated with significant reductions in both the ECBI Intensity B = −0.88, SE = 0.16, p < 0.01, 95% CI = −1.20, −0.56 and ECBI Problem scale B = −0.73, SE = 0.23, p < 0.01, 95% CI = −1.18, −0.28 over time, from baseline to 18 months. The interaction of the two factors (Autism Engagement Strategies and Active Teaching Strategies) predicting child outcomes was explored, but did not significantly predict changes in the ECBI Intensity score (B = 0.07, SE = 0.20, p = 0.74, 95% CI = −0.33, 0.46) or the ECBI Problem score (B = 0.17, SE = 0.24, p = 0.48, 95% CI = −0.30, 0.63).

Discussion

There is a large variability in the EBI strategies therapists use within community MH services and it is important to understand and characterize these patterns of strategy use, both among therapists delivering care as usual and those trained in a specific EBI. This study aimed to characterize patterns of therapist delivery of child-directed EBI strategies when working with children with ASD within publicly funded MH services. This study compared patterns of practice between therapists delivering therapy within usual care and those trained in a specific intervention for children with ASD (AIM HI), and evaluated the associations of specific patterns of strategy delivery on child behavioral outcomes.

Results from exploratory factor analyses highlighted differences in patterns of practice between usual care therapists and those trained in AIM HI who were delivering psychotherapy to children with ASD. Among the usual care therapists, results were consistent with previous research (e.g. Brookman-Frazee, Haine, et al., 2010; Garland et al., 2010) demonstrating that while EBI strategies are employed in usual care, they are used with lower extensiveness than those trained in an EBI. Furthermore, EBI strategy delivery was diffused and clustered into one general evidenced based strategies factor. This factor consisted of both general therapeutic strategies (e.g. providing psychoeducation and offering opportunities for in-session practice) along with some structural elements (e.g. using a session agenda and using materials), but did not include any strategies to target increasing motivation and in-session engagement for their ASD clients specifically (e.g. inclusion of special interests). This finding suggests that although usual care therapists are employing some EBI strategies during their sessions, they do not appear to be tailoring treatment specific to their ASD clients, perhaps due to limited ASD-specialized training available to community-based therapists (Brookman-Frazee, Drahota, Stadnick, & Palinkas, 2012; Williams & Haranin, 2016).

In contrast, in the AIM HI sample, a two-factor pattern emerged, with one group of EBI strategies capturing specific autism engagement strategies, and another set of strategies focusing on specific active teaching techniques used to facilitate acquisition of identified child skills. This difference highlights the impact of AIM HI intervention training on increasing the specificity of child-directed strategies when providing therapy to children with ASD through the use of both child engagement and skill promotion strategies. The AIM HI intervention and training model does conceptualize the intervention to include these two domains (see Table 1); however, in the factor analyses, the clustering of strategies into these domains differed slightly from the clinical model. Specifically, therapist provision of feedback and positive reinforcement to the child, which are presented as active teaching strategies to promote child skill use, clustered together with other engagement strategies, such as incorporating special interests or sharing in-session control. It may be that therapist use of feedback on child in-session behaviors and integration of positive reinforcement in session plays a more important role in engaging a child with ASD than previously conceptualized. This finding can help focus the training and implementation of the AIM HI intervention by encouraging therapists to use feedback and reinforcement strategies throughout session as a potential engagement tool, not only when actively teaching a skill in session.

Follow-up results demonstrated that the relationship between the factors identified in Aim 1 and child outcomes (as measured by the ECBI Intensity and Problem scales) differed between the usual care and AIM HI groups. Among usual care, higher General Strategies scores, on average, were associated with poorer child outcomes (i.e. increases from baseline to 18 months in parent-reported intensity of challenging behaviors and parental perception of child behaviors as problematic). In contrast, among the AIM HI group, higher Active Teaching Strategies scores, on average, were associated with more positive child outcomes (i.e. a reduction over time in parent reported intensity and parental perception of child behaviors as problematic). This is a novel finding and counterintuitive as one might expect that more intensive EBI strategy use would be associated with more positive outcomes. Both the General Strategies and Active Teaching Strategies factor contain intervention strategies that have been shown to have an evidence base (Steinbrenner et al., 2020). This differential impact of EBI strategy use may be explained by their delivery in the context of a structured EBI in which training in adapting psychotherapy for this population is included. The AIM HI intervention focuses on reducing challenging behaviors in children with ASD through the delivery of EBI strategies that are specifically packaged to promote behavior change in children with ASD. The model starts with a systematic process for identifying target behavioral and alternative child and caregiver skills. Specifically, caregivers are taught to identify specific challenging behaviors, track their occurrence, and identify common patterns and functions. Therapists then collaborate with the caregivers to develop a plan for active teaching of specific skills that are directly related to the behavioral patterns. All AIM HI trained therapists must complete this required protocol step and participate in an individual meeting with an AIM HI trainer who confirms both the function of the behavior and the appropriateness of the child skill to be taught in session. In session teaching of child skills does not begin until skills are identified and a plan for teaching is complete. Subsequently, all sessions that follow are focused on practicing child skills identified in the treatment plan through active teaching techniques. The model allows for EBI strategies to be used in a systematic way that is focused on specific, and highly relevant, skills.

While therapists in usual care group may be using active teaching techniques (e.g. modeling or in-session practice) in session, previous data indicate that they have difficulty identifying specific goals/skills (Brookman-Frazee et al., 2012; Brookman-Frazee, Taylor, & Garland, 2010). Thus, we do not know if usual care therapists are developing a focused treatment plan to guide their strategy use, or if the skills they are targeting are appropriately related to the function of the child’s challenging behaviors. As they are not getting the specific training in these domains, unlike their AIM HI counterparts, one hypothesis is that, for children with ASD, delivery of in-session evidence-based strategies is necessary but not sufficient to promote behavior change. Rather, strategy delivery must be a component of a more comprehensive clinical model designed specifically for children with ASD due to the clinical complexity of this population. Children with ASD receiving care as usual in MH clinics may not be receiving this level of targeted intervention, which explain the lack of improvement in their behavioral presentations.

Within the AIM HI group, results showed that the Active Teaching Strategies were significantly associated with positive child outcomes when controlling for the therapists’ Autism Engagement Strategies scores. Autism Engagement Strategies did not independently predict child outcomes. These results highlight the potency of active teaching of identified skills in driving reductions in challenging behaviors among children with ASD are in line with abundant research demonstrating the effectiveness of skill modeling and direct skill practice in teaching a wide variety of skills in children with ASD (Brookman-Frazee, Drahota, & Stadnick, 2012; Leaf et al., 2015; Steinbrenner et al., 2020). Furthermore, while autism engagement strategies were not associated with longitudinal outcomes, it is possible these would be related to short-term outcomes such as in-session behaviors (e.g. reduced in session disruptive behaviors and increased child participation in active teaching) and these strategies may still be a crucial ingredient to engaging children with ASD in treatment. Two primary engagement strategies, incorporating a child’s preferred interests and sharing control in session, did not load onto the factor for the usual care group, likely because these strategies are not being used extensively in usual care settings. It may be that use of these engagement strategies allowed AIM HI therapists to successfully involve their clients in active skill practice in session.

Study strengths and limitations

One limitation of this study is that our analyses did not allow us to compare the extensiveness of individual strategy delivery between the usual care and AIM HI groups. Thus, among the usual care group, we were only able to hypothesize the impact of the extensiveness of strategy usage on child behavior outcomes. Thus, a beneficial next step will be to examine the level of strategy usage needed to bring about behavioral change in children. An additional limitation is that, the usual care group had significantly fewer participants than the AIM HI group, which could impact outcomes given the variability between groups.

A strength of this study was the use of observer ratings to measure therapist strategy use. Observer ratings of therapist EBI delivery are considered gold standard approach to assessing interventions (Schoenwald et al., 2011); however, therapist practice patterns are often measured via therapist self-report methods, that are typically collected retrospectively (e.g. Becker-Haimes et al., 2019). Although self-report has advantages, in particular that it is low burden and cost-effective, self-report methods present a variety of limitations including therapist over estimation of EBI implementation (Hogue et al., 2015). The use of trained coders naive to study condition allowed for a more objective and accurate assessment of in-session strategy use. Despite the benefits of observational ratings, it is not without limitations. When interpreting observational data, one must always consider the potential impact of observation itself (e.g. the act of recording the therapy session) on participant behavior as a potential limitation (Garland et al., 2010). Furthermore, in order for observational ratings to be available, therapists must record their sessions and share the recording with the research team. Thus, for this study results, participants could not be included in analyses if they had not submitted session recordings that included their child client.

An additional limitation to this study was that child behaviors reported on the ECBI were based on parent report, which has inherent limitations (Schwarz, 1999). The use of observational data to complement parent ratings for child behavior problems may be beneficial for future studies. When interpreting the study results, it is also important to consider the generalizability of results to therapists outside of the current participants. Therapists in the study were all from Southern California, and the current results may not reflect the practice patterns of therapists in other geographic regions.

Even with these limitations, this study’s results clearly support the importance of training therapists in interventions based on research-supported strategies in order to improve the specificity of therapist delivery of EBI strategies to ultimately improve child behavior outcomes. Results support the idea that training therapists in an MH intervention for children with ASD may be important for ensuring that sessions are tailored to meet the unique needs of this population. This study supports the importance of emphasizing therapists’ use of active teaching strategies in session in order to reduce problem behaviors in children with ASD.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Research Project Grants R01MH094317 and R01MH094317S from the National Institute of Mental Health.

ORCID iD

Eliana Hurwich-Reiss

References

Becker-Haimes

E. M.

Lushin

Creed

T. A.

Beidas

R. S.

(2019). Characterizing the heterogeneity of clinician practice use in community mental health using latent profile analysis. BMC Psychiatry, 19(1), Article 257. https://doi.org/10.1186/s12888-019-2234-0

Beidas

R. S.

Kendall

P. C.

(2010). Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice, 17(1), 1–30. https://doi.org/10.1111/j.1468-2850.2009.01187.x

Boustani

M. M.

Gellatly

Westman

J. G.

Chorpita

B. F.

(2017). Advances in cognitive behavioral treatment design: Time for a glossary. The Behavior Therapist, 40(6), 199–208.

Brookman-Frazee

L. I.

Baker-Ericzén

Stahmer

Mandell

Haine

R. A.

Hough

R. L.

(2009). Involvement of youths with autism spectrum disorders or intellectual disabilities in multiple public service systems. Journal of Mental Health Research in Intellectual Disabilities, 2(3), 201–219. https://doi.org/10.1080/19315860902741542

Brookman-Frazee

Drahota

(2010). An Individualized Mental Health Intervention for children with autism spectrum disorders (AIM HI): A model to address challenging behaviors in children with ASD –A therapist manual. University of California, San Diego.

Brookman-Frazee

Drahota

Chlebowski

(2016). An Individualized Mental Health Intervention for children with autism spectrum disorders (AIM HI): A model to address challenging behaviors in children with ASD: Therapist manual- Version 2. University of California, San Diego.

Brookman-Frazee

L. I.

Chlebowski

(2013). An Individualized Mental Health Intervention for ASD (AIM HI) observational coding system manual. University of California, San Diego.

Brookman-Frazee

L. I.

Drahota

Chlebowski

Koenig

Williams

Hill

McPherson

(2020). Designing ASD interventions for community implementation: Addressing children’s challenging behaviors in publicly-funded mental health services. In White

S. W.

Maddox

B. B.

Mazefsky

C. A.

(Eds.), Oxford handbook of autism and co-occurring psychiatric conditions (pp. 409–422). Oxford University Press.

Brookman-Frazee

L. I.

Drahota

Stadnick

(2012). Training community mental health therapists to deliver a package of evidence-based practice strategies for school-age children with autism spectrum disorders: A pilot study. Journal of Autism and Developmental Disorders, 42(8), 1651–1661. https://doi.org/10.1007/s10803-011-1406-7

10.

Brookman-Frazee

L. I.

Drahota

Stadnick

Palinkas

L. A.

(2012). Therapist perspectives on community mental health services for children with autism spectrum disorders. Administration and Policy in Mental Health, 39(5), 365–373. https://doi.org/10.1007/s10488-011-0355-y

11.

Brookman-Frazee

L. I.

Haine

R. A.

Baker-Ericzén

Zoffness

Garland

A. F.

(2010). Factors associated with use of evidence-based practice strategies in usual care youth psychotherapy. Administration and Policy in Mental Health and Mental Health Services Research, 37(3), 254–269. https://doi.org/10.1007/s10488-009-0244-9

12.

Brookman-Frazee

L. I.

Roesch

Chlebowski

Baker-Ericzen

Ganger

(2019). Effectiveness of training therapists to deliver An Individualized Mental Health Intervention for children with ASD in publicly funded mental health services: A cluster randomized trial. JAMA Psychiatry, 76, 574–583. https://doi.org/10.1001/jamapsychiatry.2019.0011

13.

Brookman-Frazee

L. I.

Stadnick

Chlebowski

Baker-Ericzén

Ganger

(2018). Characterizing psychiatric comorbidity in children with autism spectrum disorder receiving publicly funded mental health services. Autism, 22(8), 938–952. https://doi.org/10.1177/1362361317712650

14.

Brookman-Frazee

L. I.

Taylor

Garland

A. F.

(2010). Characterizing community-based mental health services for children with autism spectrum disorders and disruptive behavior problems. Journal of Autism and Developmental Disorders, 40(10), 1188–1201. https://doi.org/10.1007/s10803-010-0976-0

15.

Cicchetti

D. V.

(1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284

16.

Constantino

J. N.

Gruber

C. P.

(2012). Social Responsiveness Scale: Second Edition (SRS-2). Western Psychological Services.

17.

Eyberg

S. M.

Pincus

(1999). Eyberg Child Behavior Inventory professional manual. Psychological Assessment Resources, Inc.

18.

Garland

Hawley

Brookman-Frazee

Hurlburt

(2008). Identifying common elements of evidence-based psychosocial treatments for children’s disruptive behavior problems. Journal of the American Academy of Child & Adolescent Psychiatry, 47(5), 505–514. https://doi.org/10.1097/CHI.0b013e31816765c2

19.

Garland

A. F.

Hurlburt

M. S.

Brookman-Frazee

Taylor

R. M.

Accurso

E. C.

(2010). Methodological challenges of characterizing usual care psychotherapeutic practice. Administration and Policy in Mental Health, 37(3), 208–220. https://doi.org/10.1007/s10488-009-0237-8

20.

Hallgren

K. A.

(2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34. https://doi.org/10.20982/tqmp.08.1.p023

21.

Herschell

A. D.

McNeil

C. B.

McNeil

D. W.

(2004). Clinical child psychology’s progress in disseminating empirically supported treatments. Clinical Psychology: Science and Practice, 11(3), 267–288. https://doi-org/10.1093/clipsy.bph082

22.

Hogue

Dauber

Lichvar

Bobek

Henderson

C. E.

(2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health, 42(2), 229–243. https://doi.org/10.1007/s10488-014-0548-2

23.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

24.

Joshi

Petty

Wozniak

Henin

Fried

Galdo

Kotarski

Walls

Biederman

(2010). The heavy burden of psychiatric comorbidity in youth with autism spectrum disorders: A large comparative study of a psychiatrically referred population. Journal of Autism and Developmental Disorders, 40(11), 1361–1370. https://doi.org/DOI10.1007/s10803-010-0996-9

25.

Kilbourne

A. M.

Neumann

M. S.

Pincus

H. A.

Bauer

M. S.

Stall

(2007). Implementing evidence-based interventions in health care: Application of the replicating effective programs framework. Implementation Science, 2(1), Article 42. https://doi.org/10.1186/1748-5908-2-42

26.

Lau

A. S.

Brookman-Frazee

(2015). The 4KEEPS study: Identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implementation Science, 11(1), Article 31. https://doi.org/10.1186/s13012-016-0388-4

27.

Leaf

J. B.

Townley-Cochran

Taubman

Cihon

J. H.

Oppenheim-Leaf

M. L.

Kassardjian

Leaf

McEachin

Pentz

T. G.

(2015). The teaching interaction procedure and behavioral skills training for individuals diagnosed with autism spectrum disorder: A review and commentary. Journal of Autism and Developmental Disorders, 2(4), 402–413. https://doi.org./10.1007/s40489-015-0060-y

28.

Lord

Rutter

DiLavore

P. C.

Risi

Gotham

Bishop

S. L.

Luyster

R. J.

Guthrie

(2012). Autism Diagnostic Observation Schedule: ADOS-2. Western Psychological Services.

29.

Mandell

D. S.

Walrath

C. M.

Manteuffel

Sgro

Pinto-Martin

(2005). Characteristics of children with autistic spectrum disorders served in comprehensive community-based mental health settings. . Journal of Autism and Developmental Disorders, 35(3), 313–321. https://doi.org/10.1007/s10803-005-3296-z

30.

McGraw

K. O.

Wong

S. P.

(1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46. https://doi.org/10.1037/1082-989X.1.1.30

31.

McLeod

B. D.

Smith

M. M.

Southam-Gerow

M. A.

Weisz

J. R.

Kendall

P. C.

(2015). Measuring treatment differentiation for implementation research: The Therapy Process Observational Coding System for Child Psychotherapy Revised Strategies Scale. Psychological Assessment, 27(1), 314–325.

32.

McLeod

B. D.

Weisz

J. R.

(2010). The Therapy Process Observational Coding System for Child Psychotherapy Strategies Scale. Journal of Clinical Child and Adolescent Psychology, 39, 436–443. https://doi.org/10.1037/pas0000037

33.

Muthén

L. K.

Muthén

B. O.

(1998–2017). Mplus user’s guide (7th ed.).

34.

Schoenwald

S. K.

Garland

A. F.

Chapman

J. E.

Frazier

S. L.

Sheidow

A. J.

Southam-Gerow

M. A.

(2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43. https://doi.org/10.1007/s10488-010-0321-0

35.

Schwarz

(1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93–105. https://pdfs.semanticscholar.org/efae/4f3e1558c298017fe02dbd59af91544a9113.pdf

36.

Steinbrenner

J. R.

Hume

Odom

S. L.

Morin

K. L.

Nowell

S. W.

Tomaszewski

Szendrey

McIntyre

N. S.

Yücesoy-Özkan

Savage

M. N.

(2020). Evidence-based practices for children, youth, and young adults with autism. The University of North Carolina at Chapel Hill, Frank Porter Graham Child Development Institute, National Clearinghouse on Autism Evidence and Practice Review Team.

37.

Taherdoost

Sahibuddin

Jalaliyoon

(2014). Exploratory factor analysis: Concepts and theory. Advances in Pure and Applied Mathematics, 27, 375–382. https://pdfs.semanticscholar.org/1bd8/bbd66524ccf605c879982cd35ef3a3d52160.pdf

38.

Weisz

J. R.

Kuppens

Eckshtain

Ugueto

A. M.

Hawley

K. M.

Jensen-Doss

(2013). Performance of evidence-based youth psychotherapies compared with usual clinical care: A multilevel meta-analysis. JAMA Psychiatry, 70(7), 750–761. https://doi.org/10.1001/jamapsychiatry.2013.1176

39.

Williams

M. E.

Haranin

E. C.

(2016). Preparation of mental health clinicians to work with children with co-occurring Autism Spectrum Disorders and mental health needs. Journal of Mental Health Research in Intellectual Disabilities, 9(1–2), 83–100. https://doi.org/10.1080/19315864.2016.1166302

40.

Wong

Odom

S. L.

Hume

K. A.

Cox

A. W.

Fettig

Kucharczyk

Brock

M. E.

Plavnick

J. B.

Fleury

V. P.

Schultz

T. R.

(2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: A comprehensive review. Journal of Autism and Developmental Disorders, 45(7), 1951–1966. https://doi.org/10.1007/s10803-014-2351-z