Provider perspectives and reach of an evidence-based intervention in community services for toddlers

Community involvement: the BRIDGE collaborative

The BRIDGE Collaborative is a group of early intervention stakeholders who came together to address the need for high-quality early intervention within their community. The BRIDGE Collaborative was founded in the principles of community-partnered participatory research (CPPR; Israel et al., 2005), with the goal that community partners and researchers engage as equal partners in the research process. The BRIDGE (Bond, Regulate, Interact, Develop, Guide, and Engage) name is comprised of shared early intervention values and represents the connection the group aims to build between research and practice, as well as between parents and children (see Brookman-Frazee et al., 2012). The group includes clinical providers, funding agency representatives, parents of individuals with autism, and autism services researchers. The goals and activities of the BRIDGE Collaborative, including the research detailed here, involves stakeholders to provide direction, inform methods, and ensure acceptability and appropriateness. Over the last 15 years, this community-academic partnership (Brookman-Frazee et al., 2012) has utilized a range of implementation strategies to improve early intervention for children with social communication challenges and their families in their local community (Rieth, Stahmer, & Brookman-Frazee, 2018).

Project ImPACT for Toddlers

After a discussion of shared goals and values, the initial work of the BRIDGE Collaborative centered on the selection of an intervention model to promote in community practice. The Collaborative conducted local needs assessments and consensus discussions (Powell et al., 2015) to choose an evidence-based practice with potential to support a broad range of providers and families in early intervention in our community. The selection process and guiding values are detailed in Brookman-Frazee et al. (2012). The group used consensus-based decision-making to ultimately identify Project ImPACT (Improving Parents as Communication Teachers; Ingersoll & Dvortcsak, 2019) as the best fit. Project ImPACT is a parent-mediated, naturalistic, developmental, behavioral intervention that is identified as an EBI for children with autism (Steinbrenner et al., 2020). The intervention aims to teach parents strategies to promote their child’s social engagement, communication, imitation, and play skills in daily routines.

After selection, however, initial pilot testing of Project ImPACT by providers and families in the community revealed several areas of needed adaptations to improve its fit for toddlers and the agencies that serve them in service of maximizing community sustainment and uptake. Iterative adaptations to the intervention and training materials were made through pilot testing, expert and clinician feedback, and consultation with Project ImPACT developers to ultimately develop a targeted version of the intervention, known as Project ImPACT for Toddlers (PI^T; Haine-Schlagel et al., 2020; Stahmer et al., 2020). Final products of the adaptation process included both a revised parent coaching manual and full interventionist training materials with flexible delivery options (e.g. in-person and online didactic options, and structured coaching procedures) to match the variable needs across community agencies. Following the adaptations, a quasi-experimental pilot test comparing PI^T to usual care early intervention services demonstrated more positive outcomes after receiving PI^T for both parents and children, including greater changes in parent interaction style and promising impacts on children’s social communication skills (Stahmer et al., 2020). In addition, providers trained in PI^T had moderate to strong adherence to intervention fidelity, and fidelity to the model predicted parent outcomes.

Community uptake of PI^T was initially supported through a series of research studies designed to build local capacity for the intervention, including a train-the-trainer effort implemented across 12 early intervention agencies. To facilitate implementation, the Collaborative designed educational materials and a dynamic training for PI^T involving both didactic instruction and hands-on-practice with feedback that prepared clinical providers to utilize PI^T. Simultaneous with the training effort, a specific contract for funding was first offered to early intervention agencies to deliver PI^T to families by San Diego Regional Center (SDRC), the local funder of public early intervention services. This contract was discussed and developed via the Collaborative, which includes several key members of the Early Start administration. Taken together, these factors represent multiple strategies known to facilitate community EBI implementation (Powell et al., 2015).

Implementation outcomes framework

In the current analyses, we applied the Proctor et al. (2011) taxonomy to guide evaluation of implementation outcomes realized as part of the community-partnered effort to train agencies in PI^T. Implementation outcomes are important to consider as they are a necessary precondition for clinical or treatment outcomes (Proctor et al., 2011). This framework has been successfully applied to interventions for autism within public schools (Locke et al., 2016) as well as mental health interventions used with individuals with autism (Dickson et al., 2021). The Proctor and colleagues (2011) taxonomy specifies eight distinct outcomes: (1) acceptability (satisfaction); (2) adoption (intervention use or intention to try); (3) appropriateness (intervention relevance or usefulness); (4) feasibility (or intervention fit or utility); (5) fidelity (adherence and delivery quality); (6) implementation cost (cost-benefit and cost-effectiveness); (7) penetration (proportion of clients with whom the intervention is used); and (8) sustainability (continued use and routinization). Four of these outcomes are directly addressed through secondary data analyses in this study, including appropriateness, acceptability, feasibility, and sustainability. Intervention reach is examined via the penetration of the intervention at the agency level and the volume of families receiving PI^T in the community 5 years after the training study.

Study aims

The purpose of this study is to: (1) utilize mixed qualitative and quantitative methods to characterize the short-term, provider-level implementation outcomes from the initial community training effort in PI^T, including determinants of outcomes and (2) identify the long-term, system-level sustainment outcomes of PI^T (penetration and reach) in publicly funded early intervention services. The overall goal of these aims is to inform scale-up and further sustainment of evidence-based early intervention.

Method

Implementation outcomes (Aim 1)

Training study context

This study reports implementation outcome data from a PI^T community training effort (Rieth, Stahmer, Dickson, et al., 2018). The training study evaluated a train-the-trainer approach of PI^T via a multiple probe design (Horner & Baer, 1978). Participating agencies identified individuals as “agency trainers,” who then participated in PI^T training with the clinical research team. Each agency trainer then provided training on PI^T content and delivery to direct service providers at their agency, with support from the research team. Feedback was solicited from both agency trainers and service providers on their perspectives via both qualitative (interview) and quantitative (survey) methods. The current analyses combine these sources to examine implementation outcomes of the training effort and initial intervention rollout.

Training study data

Mixed qualitative and quantitative methods were used to examine implementation outcomes related to PI^T and the training process via quantitative feedback surveys and qualitative interviews. Data were solicited from providers after they received training and had delivered PI^T to families for at least 3 months in their usual clinical practice. All data collection procedures were approved by the University of California, San Diego Institutional Review Board and relied on by participating institutions. All study procedures conformed to the US Federal Policy for the Protection of Human Subjects.

The participant sample and training process is briefly described below. A full description of the methods and sample from which participants were drawn is available in Haine-Schlagel et al., (2020).

Participants

Agencies providing early intervention services in two counties were identified based on their involvement and prior participation with the BRIDGE Collaborative or via recommendation of a BRIDGE member. Participants were recruited from 12 community agencies that provide early intervention services through California Early Start, the program for public funding of early intervention (IDEA, 2004). Publicly available data indicate that race/ethnicity for families receiving California Early Start services are as follows: 10% Asian, 5% Black or African American, 58% Hispanic/Latino, 3% two or more races, and 25% White. Although these data are on the demographics of families receiving early intervention services broadly, we expect that these demographics roughly parallel the populations served by participating agencies.

A total of 59 providers enrolled in the train-the-trainer study as either agency trainers (n = 14) or therapists (n = 45), with 38 total providers contributing feedback data in the current sample (agency trainers n = 14, 100% response rate; therapists n = 24, 53% response rate; 64% response rate overall). Given the moderate response rate for therapists, responders and non-responders for the implementation measures reported here were compared on baseline demographic characteristics of race, ethnicity, primary discipline, age, and caseload (chi-square) as well as years of experience with early intervention, autism spectrum disorder (ASD), and parent coaching (t-tests). All tests revealed no significant differences between groups (p < 0.05). Eligibility criteria included (1) a master’s degree or equivalent degree or credentials in a relevant field or specialty; (2) agency employment for at least 6 months; and (3) provision of direct services to infants and toddlers with social communication concerns. Providers invited to participate were selected by each agency and then enrolled on a first come, first serve basis. Participants did not have prior experience with the PI^T intervention specifically. Full demographics for those in the current sample are displayed in Table 1.

OPEN IN VIEWER

Table 1.

Participant demographics.

Characteristic	Agency trainers (n = 14)	Therapists (n = 24)
	n (%)	n (%)
Ethnicity
Hispanic/Latinx	3 (21)	8 (33)
Not Hispanic/Latinx	11 (79)	16 (67)
Race
African American	1 (7)	–
Asian American/Pacific Islander	–	4 (17)
Caucasian/White	9 (64)	12 (50)
Filipino American	2 (14)	–
Multiracial	–	5 (21)
Other	3 (21)	1 (7)
Prefer not to state	–	2 (14)
Primary discipline
Early Childhood Education	3 (21)	11 (46)
Child Development	2 (14)	2 (10)
Applied Behavior Analysis	2 (14)	4 (17)
Physical Therapy	1 (7)	–
Speech and Language Pathology	2 (14)	5 (25)
Psychology	1 (14)	–
Other Specialty	3 (21)	2 (10)
Age
Under 30	2 (14)	5 (21)
31–40	7 (50)	11 (46)
41–50	1 (7)	4 (17)
51–60	3 (21)	4 (17)
Over 60	1 (7)	–
Caseload
Fewer than 10 families	1 (7)	3 (13)
11–20 families	–	5 (21)
21–30 families	2 (14)	3 (13)
Over 31 families	–	4 (17)
Missing	11 (79)	9 (38)
	M (SD)	M (SD)
Years Early Intervention Experience	15.2 (12.4)	11.1 (10.6)
Years ASD Experience	15.1 (12.3)	11.1 (9.93)
Years Parent Coaching Experience	13.7 (13.8)	13.1 (9.2)

M: mean; SD: standard deviation; ASD: autism spectrum disorder.

Measures

Demographics

At baseline, all participants reported on their sociodemographic characteristics, including age, race, ethnicity, and professional history.

Implementation survey

A 41-item questionnaire was utilized to examine implementation outcomes across six subscales including training acceptability, training appropriateness, training feasibility, intervention appropriateness, intervention acceptability, and sustainability. Using a 5-point Likert-type scale (1 = strongly disagree to 5 = strongly agree), participants rated both the training they received in PI^T as well as the intervention itself. This survey was adapted from several sources examining implementation of evidence-based practices in other disciplines (Halliday-Boykins et al., 2005; Haug et al., 2008; Lehman et al., 2002). Example questions included (1) “I am satisfied with the PI^T training materials” (Training Acceptability); (2) “I have space and equipment needed to participate in PI^T Training effectively” (Training Appropriateness); (3) “PI^T training provided me with the skills needed to implement PI^T effectively” (Training Feasibility); (4) “I feel comfortable using the PI^T materials in my program” (Intervention Appropriateness); (5) “I would want to invite other staff from my agency to receive PI^T training” (Intervention Acceptability); and (6) “Therapists at our agency will now be trained in PI^T as part of the general training process” (Sustainability). Alpha values for subscales ranged from 0.36 to 0.87. Table 2 shows the number of items and alpha values for each scale. All subscales with an alpha value below 0.60 were dropped from further analyses (i.e. Training Acceptability and Training Appropriateness), resulting in a total of 23 items utilized in current analyses. Implementation surveys were available for 32 providers (84%).

OPEN IN VIEWER

Table 2.

Implementation survey subscales, item information, and reliability.

Subscale	Number of items	Cronbach’s alpha
Intervention appropriateness	5	0.70
Intervention acceptability	5	0.67
Sustainability	7	0.70
Training acceptability	7	0.38*
Training appropriateness	11	0.36*
Training feasibility	6	0.87

*

Subscale dropped from further analyses due to unacceptably low alpha value.

Semi-structured interview

A brief semi-structured interview was conducted with all interested participants. All interviews were conducted by a trained member of the research team who was not involved in the training of participants. Interviews were guided by a series of questions related to implementation outcomes, including feasibility, acceptability, appropriateness, and sustainability. All interviews were recorded and transcribed for later analyses. The interview guide questions were developed simultaneously with the survey. Interviews were available for 26 providers (68%). The average duration of the interviews was 17.36 min.

Data analyses

The mixed-methods complementarity approach was used in which the qualitative interview data and quantitative survey data were compared side by side to determine commonalities and/or differences across the two methods (Palinkas et al., 2011). Qualitative interview data were analyzed using content analysis concerned with manifest constructs (Bengtsson, 2016). Interview transcripts were entered, coded, and analyzed in QSR-NVivo 2.0, a program frequently used in qualitative research (Tappe, 2002). After transcription, two interviews were randomly selected and examined jointly by two members of the research team to elicit a priori and emergent codes within the interviews and generate a codebook. Segments of the texts, ranging from sentences to paragraphs, were assigned specific codes that enabled members of the research team to consolidate interview data into analyzable units. A senior member of the research team reviewed the codes for accuracy and comprehensiveness relative to the interviews. Two members of the research team then independently coded one additional transcript and codes were checked for consensus and accuracy by a senior team member. Following this process, all interviews were coded, with 27% of interviews coded by both team members and checked for agreement. A review of double coded transcripts was conducted whereby members of the research team reached consensus as to which codes should be applied to specific segments of text. Adjustments were made to independently coded transcripts based on consensus discussions, and any areas of discrepancy or concern were discussed with the team. Emergent themes were identified and assigned a code by considering the frequency of and salience with which (i.e. importance or emphasis) a participant discussed it. Quantitative survey analyses primarily consisted of descriptive statistics to characterize implementation outcomes.

Results

Intervention reach (Aim 2)

In 2016, the first two early intervention agencies were contracted to deliver PI^T with SDRC. As of June 2021, a total of 28 agencies were contracted to provide PI^T to families in the San Diego and Imperial county regions, demonstrating the growing reach of PI^T and PI^T training in the local community. These agencies represent 51% of the total number of agencies contracted to provide services to infants and toddlers with developmental disabilities with SDRC (n = 55).

Figure 1 represents the number of children who have received Project ImPACT across the years 2016–2020 as funded by SDRC. As illustrated by the date, there are a steadily increasing number of children receiving Project ImPACT. Children receiving Project ImPACT were an average of 23.31 months old (SD = 5.14) at the start of service.

OPEN IN VIEWER

Figure 1.

Children receiving Project ImPACT for Toddlers in community services.

Discussion

The implementation outcomes for training efforts in PI^T within the local community thus far appear strong, with expanding reach in terms of the number of children receiving the funded service. A growing number of agencies have contracted with the local regional center to provide PI^T in the community, illustrating the strong fit of the model for the service system. Findings also extend the literature on key components for effective provider training, with results highlighting the importance of user-friendly materials, group-based trainings, and multiple opportunities for feedback to support provider learning.

The facilitators of positive implementation outcomes identified here represent key recommendations for future implementation efforts within the early intervention service system. Specifically, clear, structured materials that providers view as a match for their clients’ needs and adjacent to their existing clinical skills supported their positive view of the training experience and the intervention itself. Efforts to develop professional and high-quality intervention materials that are easy for providers to access and navigate in a variety of modalities (e.g. book/hardcopy and electronic access) will likely support future scale-up efforts. In addition, the opportunity to receive ongoing in vivo supervision on use of the model was highly valued by therapists. The availability of live feedback is often a costly feature of trainings to implement, especially on an ongoing basis, but these results demonstrate the importance of this approach in supporting positive perceptions. Finally, agency support for training and use of PI^T was cited by participants as particularly helpful, supporting recent efforts to approach implementation through intervention at the leadership level (Aarons, Ehrhart, et al., 2017; Green et al., 2014).

It is likely that the iterative, community-informed intervention adaptations and training development greatly facilitated the positive outcomes reported here, and will continue to be important for scale-up of PI^T to new regions. The train-the-trainer model resulted in positive perceptions from both groups involved, the trainers and the therapists. Early intervention agencies involved in the BRIDGE Collaborative have repeatedly indicated that intervention models that require each individual provider to be externally certified and trained, often at high monetary and time costs, are not appropriate or sustainable for their companies. The PI^T model tested here allowed agencies to develop capacity to train their own staff in a way that was positively received. This success is consistent with the need for context-specific adaptations that consider fit across multiple levels to promote strong implementation outcomes. Since the time this study was conducted, the Project ImPACT developers have created a certification model that continues to allow an individual to train others within their agency after meeting training and fidelity requirements, similar to the model reviewed here.

Of the implementation outcomes queried, sustainability scored the lowest quantitatively and the qualitative themes were limited. Providers referred to the generalized use of PI^T elements within broader practice, but did not speak specifically to their continued or ongoing use of PI^T or the PI^T training. However, the reach data indicate strong growth in the number of families who are receiving PI^T, which indicates that therapists must in fact be using the approach with new families and/or that new therapists are being trained. It may be that the interview questions and sustainability focused survey items did not capture the sustainability outcome accurately, and a later or broader exploration of implementation outcomes for PI^T should consider this area more directly and explicitly.

Findings regarding the barriers to PI^T training and intervention highlight the need for continued refinement to try to simplify interventions and trainings to aid learning and delivery. Providers found the amount of information in PI^T overwhelming and difficult to manage. In a system where providers are likely to have high caseloads and burn out is common, perceptions of this complexity could prevent therapists from selecting to engage in training or continued use. Members of the BRIDGE Collaborative have often debated the balance between wanting to ensure that providers have enough information (e.g. foundational understanding of multiple domains of development, detailed knowledge of the intervention strategies themselves, and techniques to engage parents) while also keeping the content streamlined and accessible. Given that providers likely come to the table with varying levels of knowledge and experience, a modular approach to the content that allows providers to tailor their own training may be optimal. As noted by providers here, clear intervention materials can also be a great facilitator in ensuring learning, so time spent developing quality materials is not misplaced. The focus on balancing the amount of information may be especially relevant for EBIs for autism, where a large portion of recommended EBIs are multi-component and data show poor fidelity of these multi-component interventions in community settings (e.g. Mandell et al., 2013).

Differences in perspective between agency trainers and therapists may have been expected based on previous literature reporting differences in opinions between leadership and direct service providers (Beidas et al., 2018). However, agency trainers were identified by individual agencies rather than by specific leadership criteria in this study, and it is likely that the two groups were not fully distinct in the current sample. This is evident in the demographic information regarding the two groups, which reveals a high degree of overlap in terms of parent coaching experience, specifically, as well as other factors. Given the similarity between the two groups, it is not surprising that their perspectives were similar. Despite the positive implementation outcomes thus far, there remain several critical unknowns and areas for future research regarding the use of PI^T in the community. First, the intervention fidelity with which therapists are utilizing PI^T was not assessed in the current analyses. Although data available from the pilot study of PI^T demonstrate the ability of community providers to implement PI^T with fidelity (Stahmer et al., 2020), it could be that some providers feel positively about the training they received and the strategies themselves, but still have difficulty actually implementing the approach with families. It will be important to measure to what extent community providers are able to successfully adhere to the coaching and parent engagement strategies of PI^T, as this is a known area of challenge for the early intervention service system broadly. Future work should also systematically explore the remaining aspects of the Proctor et al.’s (2011) implementation taxonomy that were not available from the measures examined here, including adoption, caseload penetration, and cost.

An additional area for future examination is the successful implementation of PI^T with early intervention providers with various levels of experience. The BRIDGE Collaborative made intentional decisions in the rollout of PI^T training to engage relatively experienced service providers who were likely to be prepared and successful in coaching parents (represented by MA degrees). The average years of experience of the participating sample were over 10 years in EI/ASD/parent coaching, representing a more skilled workforce than many therapists who are providing direct early intervention services to families. While the reach data presented here support the ability of agencies to continue to deliver PI^T by providers with the required level of expertise, future research should focus on whether the approach may also be feasible, well-received, and successful when delivered by members of the early intervention workforce with less experience.

A primary limitation of this study is the fact that implementation outcomes were solicited from participants who participated in a training research study and willing to respond to the survey and interview. Although all participants were employed by community early intervention agencies, those willing to engage in training via research participation may be qualitatively distinct from service providers who do not select to participate in such opportunities, and those that chose to respond may have been only those that felt positively. Implementation outcomes regarding PI^T from a group of providers who are less eager to be involved in research may reveal different results than those identified here, and further exploration of PI^T should attempt to capture perspectives from multiple viewpoints. The early intervention service landscape is comprised of providers with a variety of motivation and skill levels, and it is important that PI^T be a good match across the full range of interventionists. Additional limitations include the short period of time from training to assessment (3 months) and the somewhat limited response rate to the surveys and interviews from the full sample of providers who participated in the training study.

In conclusion, early intervention providers who learned PI^T in a community training study reported the intervention to be acceptable, appropriate, and feasible for their clinical practice. The growing penetration of the intervention in terms of early intervention agencies contracted with SDRC and the expanding reach of the intervention to a growing volume of families each year indicate the sustainability of the approach. Overall, these illustrate success in the efforts of the BRIDGE Collaborative to ensure access to EBI for families of young children with autism.