Examining provider decisions around the delivery and adaptation of a parent-mediated intervention within an Early Intervention system

Abstract

There is growing research aimed at translating parent-mediated interventions into Part C Early Intervention systems to examine the effectiveness and reach of these models. Although research to date suggests that Early Intervention providers deliver parent-mediated intervention near fidelity, current fidelity reporting practices make delivery difficult to discern. Understanding how parent-mediated interventions are delivered and adapted in a more nuanced manner, including fine-grained intervention delivery and adaptation processes, is important and may increase our understanding of how well these models are aligned within Early Intervention systems. The current study examined the delivery of an evidence-based parent-mediated intervention, Project ImPACT, when delivered by Early Intervention providers and examined their intervention fidelity, the decisions they weighed when delivering Project ImPACT, and reported adaptations to Project ImPACT. Results from 24 providers demonstrated, on average, higher fidelity in response to consultative feedback but notable variability across providers. Preliminary qualitative data highlighted that many events arose within sessions that drove providers to augment their delivery of Project ImPACT. Results suggest the importance of carefully examining how and why providers deliver evidence-based interventions within Early Intervention systems, and the impact of these decisions on fidelity metrics and service outcomes.

Lay abstract

Parent-mediated interventions are an evidence-based practice for autism in which providers support caregivers in learning and applying strategies that support their child’s development. Research has begun to study whether parent-mediated interventions can be effectively delivered in Part C Early Intervention systems. This research has been promising; however, it has been difficult to determine how Early Intervention providers deliver and adapt parent-mediated interventions to meet the needs of the families they serve. Examining how parent-mediated interventions are delivered and adapted may help us understand whether parent-mediated interventions are a good fit in these systems. The current study examined the delivery of an evidence-based parent-mediated intervention, Project ImPACT, when delivered by providers within an Early Intervention system. Results from 24 Early Intervention providers demonstrated that, on average, providers delivered Project ImPACT with higher quality during their time in training and consultation. However, there was also variability in how providers delivered Project ImPACT, with some delivering the program inconsistently, some increasing their quality throughout consultation, and others having consistently high-quality delivery. In addition, qualitative data demonstrated that a variety of events arose within Project ImPACT sessions that drove providers to adapt the program. Results suggest the importance of carefully examining how and why providers deliver evidence-based interventions within Early Intervention systems.

Keywords

adaptation autism spectrum disorders Early Intervention system fidelity implementation parent-mediated intervention

Introduction

The last two decades have seen a rise in the detected prevalence of autism spectrum disorder (ASD) from 1 in 150 to current estimates of 1 in 44 (Maenner et al., 2021). Although evidence-based intervention can support the quality of life for autistic people, early access is important for these benefits to be realized (Dawson, 2008). Parent-mediated intervention (PMI) is one example of an early treatment approach for young children with or with an increased likelihood of being autistic. As part of participation in PMIs, caregivers are taught intervention strategies to support their child’s development across home and community activities (Casagrande & Ingersoll, 2017). Recent studies indicate positive effects of PMI on caregivers’ learning and use of intervention strategies (i.e. caregiver fidelity), resulting in gains in child social engagement and communication skills (Brian et al., 2022; Nevill et al., 2018; Yoder et al., 2021). Further, PMIs may represent a practical approach to early intervention given their ability to be delivered at a relatively low dosage and in a manner that aligns with service systems that support toddlers and young children with special health care needs (Adams et al., 2013; Tomasello et al., 2010).

Given growing evidence to support the efficacy of PMIs, there is mounting interest in increasing access to these programs by translating PMIs into community systems that provide a natural entry point into services for young children with a high likelihood of being autistic (Vivanti et al., 2018). The public Early Intervention system has been thought of as an ideal translational setting, as it is federally mandated within the United States to provide therapeutic services to children 0–3 years with developmental delays under Part C of the Individuals with Disabilities Education Act (IDEA, 2004). Federal funds through Part C of IDEA are allocated to each state to implement a statewide Early Intervention system that delivers comprehensive and multidisciplinary services to infants and toddlers and that includes an interagency system to coordinate the delivery of these services (United States (US) Department of Education, 2021). States have autonomy in how they structure and coordinate Early Intervention services and, thus, there is notable variability across states in the coordination and coverage of services (Noyes-Grosser et al., 2018). Upward of 10 percent of children served within Early Intervention systems have an increased likelihood of being autistic (Eisenhower et al., 2021); thus, the system captures many children during a critical window of early development. Further, Early Intervention systems operate using a family-centered and family coaching model, making them appear well suited for the implementation of PMI for autism (Adams et al., 2013; Pellecchia et al., 2022).

In practice, the alignment between Early Intervention systems and PMIs for autism is complicated. Recent research has shown that Early Intervention providers’ use of parent coaching strategies is variable, irrespective of formal training in PMI (Pellecchia et al., 2022). Once trained in PMI, providers’ fidelity to these intervention models also appears mixed; however, these findings are difficult to discern given that fidelity is often reported as an average rating across all providers and all scored sessions. For example, during a trial of the Early Start Denver Model (C-ESDM) within low-resourced Early Intervention systems across several US states in different regions of the country, providers trained in C-ESDM significantly increased their fidelity to the intervention but achieved an average C-ESDM fidelity score of only 7.67 out of a total possible score of 13 (i.e. 59%; Rogers et al., 2022). Early Intervention providers’ fidelity to manualized PMI has been somewhat higher in studies using more intensive training models (Stahmer et al., 2020). For example, during an effectiveness trial of Project ImPACT in California Part C agencies, Early Intervention providers’ coaching fidelity was described as an average of 3.68 points out of a total of five possible points (i.e. 72%; Stahmer et al., 2020). Within this study, trained providers achieved fidelity scores that were significantly higher than the fidelity of providers who did not receive Project ImPACT training (M = 2.17 points or 43.4% for untrained providers; Stahmer et al., 2020). However, the average fidelity score of trained providers was still lower than the 80% threshold that is thought to reflect competence in the intervention (Ingersoll & Dvortcsak, 2019).

The fidelity by which Early Intervention providers deliver PMI raises questions regarding the complexity of the interventions, the feasibility of the training and consultation models that may be necessary to support providers in delivering these interventions, and the overarching alignment of these models within Early Intervention systems. These questions are important given what is known about Part C Early Intervention systems, which are often constrained by funding and significant workforce issues (e.g. Aranbarri et al., 2021; Noyes-Grosser et al., 2018), including turnover and provider experience levels that are more limited than that of providers for whom PMIs were initially developed (Pickard et al., 2023). Further, although reporting fidelity is necessary to establish intervention effectiveness, delivering a manualized PMI at fidelity may not be the only goal of Early Intervention providers when serving diverse families within Part C systems (Pickard et al., 2022).

Taken together, there is much to learn about how PMIs are being delivered within Early Intervention systems. This includes more nuanced information regarding PMI fidelity across the duration of program delivery in addition to adaptations that may be driving seemingly variable fidelity outcomes. Understanding how PMIs are delivered and how PMI delivery relates to key service outcomes is particularly important because Early Intervention providers describe the need to adapt PMIs in response to the diverse needs of families served within Early Intervention systems (Pickard et al., 2021), suggesting that these adaptations are perceived to enhance service outcomes. Outside the autism field, research has demonstrated that intervention adaptation is inevitable (Chambers & Norton, 2016; Stirman et al., 2019), that therapists find value in adapting evidence-based interventions (Barnett et al., 2019), and that therapists perceive that these adaptations may align intervention with a patient’s preferences, cultural values, and needs (Kim et al., 2020). Examining PMI delivery and adaptation within Early Intervention systems may be particularly important given the emphasis on family-centered care within Early Intervention systems, which promotes delivering therapeutic services in a manner that responds to the values and preferences of families (Pickard et al., 2022).

When examining intervention delivery within diverse service systems including Early Intervention systems, it is important to leverage adaptation frameworks that organize planned and ad hoc adaptations to interventions (Stirman et al., 2019). Adaptations included in these frameworks relate to intervention dosage (e.g. extending or reducing intervention length), intervention content (e.g. removing, tailoring, or augmenting of core content), and intervention format (e.g. delivery setting; Stirman et al., 2019). These frameworks can also be used to categorize adaptations as being either consistent or inconsistent with the core components of an intervention (Kirk et al., 2020), thus providing guidance on whether an adaptation may enhance or attenuate service outcomes (Edmunds et al., 2022; Miller et al., 2020). Within the autism field, these frameworks have been used at the conclusion of intervention delivery to categorize adaptations made to the entirety of an intervention (Dickson et al., 2021; Dyson et al., 2019). Although important, there is also value in understanding more fine-grained intervention delivery and adaptation processes. That is, it may be important to understand how and why Early Intervention providers deliver specific PMI sessions and the perceived impact of the decisions made. This type of session-level, process data recognizes that delivering evidence-based interventions includes live clinical decision-making by a provider, in which choices are made in response to specific events and ultimately weighed and selected to enhance or support service outcomes (Sheldrick et al., 2021).

To date, research has yet to examine how PMIs for autism are delivered and adapted within Early Intervention systems, including the specific clinical decisions that providers weigh within their delivery. The current study examined the delivery of an evidence-based PMI, Project ImPACT (Ingersoll & Dvortcsak, 2019), when delivered by providers within a Part C Early Intervention system. Specific objectives were to examine (1) how providers delivered Project ImPACT using traditional fidelity metrics, (2) the decisions providers weighed in their delivery of Project ImPACT, and (3) reported adaptations to Project ImPACT.

Methods

Procedure

This research was conducted as part of an ongoing contract and partnership with Georgia’s Part C Early Intervention system, Babies Can’t Wait. All research procedures were approved by the Emory University Institutional Review Board and the Georgia Department of Public Health. Participants were interdisciplinary providers (e.g. special instructors, speech language pathologists, occupational therapists) working within Babies Can’t Wait who delivered therapeutic services to children 12–36 months of age with a medical diagnosis of ASD or other social communication delays. Of 28 providers who enrolled in this study, 27 completed Project ImPACT training, and 24 completed full consultation. All participating providers identified as Female, were an average of 47.2 years (SD = 12.69, range = 25–79 years) with an average of 9.02 years of experience within Early Intervention systems (SD = 7.95, range = 0–22 years). All providers except two identified as being independent contractors. Independent contractors held contracts directly with the Early Intervention system and worked relatively independently of other providers serving the same children on their caseloads; the remaining two providers were employees of agencies (e.g. Early Head Start programming) that were contracted to provide services for the Early Intervention system. Providers represented diverse racial backgrounds, had varied educational and disciplinary experiences, and worked within 10 health districts representing 74 counties in Georgia. See Table 1 for the demographic information of provider participants.

Table 1.

Provider demographic information (N = 28).

	M (SD)	Range	Percent (N)
Gender (female)			100.00 (28)
Age (years)	47.20 (12.69)	25–79
Independent contractor			92.86 (26)
Experience in EI (years)	9.02 (7.95)	0–22
Ethnicity
Hispanic/Latinx			10.71 (3)
Non-Hispanic/Latinx			89.29 (25)
Race
Black/African American			39.28 (11)
White/Caucasian			53.57 (15)
Asian			3.57 (1)
American Indian			–
Biracial/multiracial			7.14 (2)
Education
Bachelors			21.43 (6)
Some graduate/MS			78.57 (22)
Profession
Special instructor			71.43 (20)
Speech language pathologist			17.86 (5)
Occupational therapist			7.14 (2)
Physical therapist			3.57 (1)

SD: standard deviation; EI: Early Intervention.

Study design

Early Intervention providers were provided with informed consent and completed measures prior to Project ImPACT training participation. Following training (as specified below), providers identified one or more families on their caseload with ASD or social communication delays who consented to video recording for the purpose of weekly group consultation and who completed sociodemographic information (see Table 2). Throughout consultation, providers submitted videos of their Project ImPACT sessions which were scored for fidelity by one of four Project ImPACT certified coaches, two of whom were also certified trainers. After every few sessions, providers were asked to complete an open-ended survey about how they delivered Project ImPACT. At the conclusion of group consultation, providers completed measures about their experience delivering Project ImPACT, their intent to deliver the program in the future, and global adaptations that they made while delivering the program.

Table 2.

Child and caregiver demographic information (N = 25).

	M (SD)	Range	Percent (N)
Child Gender (male)			64.00 (16)
Child Age (months)	25.32 (5.01)	14–34
Child ethnicity
Hispanic/Latinx			24.00 (6)
Non-Hispanic/Latinx			76.00 (19)
Child Race
Black/African American			44.00 (11)
White/Caucasian			44.00 (11)
Asian/Pacific Islander			–
American Indian			–
Biracial/multiracial			12.00 (3)
Participating caregiver
Mother			100.00 (25)
Maternal education
Some high school			8.00 (2)
High school diploma or GED			32.00 (8)
Associate degree			16.00 (4)
Some college, no degree			32.00 (8)
Bachelor’s degree			4.00 (1)
Master’s degree			12.00 (3)

SD: standard deviation.

Intervention

Project ImPACT is a manualized, evidence-based, PMI for ASD (Ingersoll & Dvortcsak, 2019). Project ImPACT was initially developed in partnership with caregivers of autistic children and community providers, and has since been adapted in response to feedback from families and clinicians serving structurally marginalized communities (Pickard et al., 2016). Providers delivering Project ImPACT support caregivers in using a blend of developmental and naturalistic behavioral intervention techniques across daily routines to enhance their child’s social engagement, language, imitation, and play skills. Project ImPACT can be delivered once each week for 1 h over the course of 12 weeks. The caregiver manual provides visuals, written descriptions of the specific techniques, and charted examples of how to use strategies within daily routines or play activities to support adult learning. The program begins with collaborative goal setting with caregivers in the four social communication skill areas targeted by the program. In subsequent sessions, caregivers receive: 1) didactic instruction in intervention strategies; 2) modeling of the intervention techniques or video demonstration of a technique for telehealth sessions; 3) live coaching while practicing; and 4) a practice plan for implementing the targeted strategies with their child in a daily routine or activity.

Project ImPACT training and consultation

Participating providers completed an asynchronous online tutorial that provides an overview of the Project ImPACT strategies (6 h), followed by a live webinar series that was conducted via Zoom and allows for Project ImPACT role-play, video review, and treatment planning (14 h). Emphasis of the webinar series is in how to support caregivers in learning and using the Project ImPACT strategies using best practices in adult learning and family coaching. Following training, providers participated in group consultation 1 h per week for 12 weeks. Providers were encouraged to submit video recording of each session while they participated in group consultation and had the option of completing three booster consultation sessions at the end of the 12 week consultation period. Group consultation was structured such that it embedded active training strategies, including a combination of didactic instruction, role-play to facilitate behavioral rehearsal, video review, joint problem solving, and planning around subsequent sessions to facilitate cognitive rehearsal activities (e.g. McLeod et al., 2018).

Measures

Provider sociodemographic information

Participating providers provided their age, race, ethnicity, gender identity, highest education level, disciplinary background, and years of experience working within Early Intervention systems and with children with an increased likelihood of having ASD.

Caregiver sociodemographic information

Caregivers provided the age, race, ethnicity, gender, and service utilization history of their child receiving Early Intervention services. They also indicated their own age, gender identity, race, ethnicity, educational attainment, household size, and annual household income.

Self-efficacy

Providers completed a 7-item measure of their self-efficacy in delivering Project ImPACT prior to training, immediately following training, and after participating in weekly group consultation. Providers rated their perceived skill in delivering specific Project ImPACT strategies using a 10-point Likert-type scale, with a 1 indicating not having the skill at all, a 5 indicating somewhat having the skill, and a 10 indicating entirely having the skill. Examples of items included: Working collaboratively with parents to develop social communication goals for their child; demonstrating the use of specific Project ImPACT techniques; coaching caregivers in their use of Project ImPACT strategies; helping parents to problem-solve issues to their use of specific strategies.

Enrollment and attendance

Providers’ attendance in group consultation sessions was tracked, along with the number of Project ImPACT videos submitted for review and the number of booster consultation sessions that were requested following the completion of group consultation.

Fidelity

All recorded Project ImPACT sessions were behaviorally coded using the Coaching Fidelity Checklist, which was available in an in-person or telehealth version. Each of the 21 items was rated on a three-point Likert-type scale, where 1 indicated the provider did not use the strategy or did so poorly and 3 indicated that the provider used the strategy fully. The measure includes items about: (1) setting up the coaching environment, reviewing relevant case information, and preparing session materials, (2) using coaching strategies (i.e. setting a session agenda; reviewing prior weeks’ goals and collaboratively troubleshooting barriers to implementation; preparing the caregiver to watch for specific strategies before then demonstrating strategies; supporting caregiver practice of strategies through positive and constructive feedback, reflection on practice; creating a plan for practice in between sessions), and (3) using collaborative and responsive strategies to partner with caregivers.

Total fidelity scores for each session are summed and divided by the total possible fidelity points to achieve a percent rating (0–100%). Provider fidelity was coded by one of four graduate-level clinicians (i.e. psychologists; speech language pathologists) certified in Project ImPACT, two of whom are also certified trainers in Project ImPACT (K.P. and N.H.). All videos were scored for primary fidelity by one of the two trainers. All reliability was then randomly scored by the alternate trainer or one of the certified coaches (K.G. and N.B.). Reliability was calculated on 20% of all recorded sessions. Intraclass correlations (ICCs) for individual fidelity items ranged from 0.68 to 0.84 across raters with an average ICC of 0.75, representing good reliability (Cicchetti, 1994).

Providers’ description of session processes and decision

Immediately following every other Project ImPACT session, providers were asked to complete an online survey with the following open-ended questions: (1) What were your main goals for this session?; (2) Do you feel that those goals were achieved and why?; (3) Was there anything unanticipated that happened during your session?; (4) How did you respond to the situation and why?; and (5) What are your plans for the next session?

Project ImPACT categorical adaptation

At the conclusion of participating in Project ImPACT group consultation, all providers rated the frequency by which they made seven types of adaptations to Project ImPACT based on the Framework for Reporting Adaptations on Modifications-Enhanced (FRAME; Stirman et al., 2019). Providers rated each item using a 5-point Likert-type scale, with 1 indicating that the provider never made the adaptation, 3 indicating that they made the adaptation in about half of sessions, and 5 indicating that they made the adaptation in nearly all sessions. Specific questions included whether providers: (1) tailored, tweaked, or refined intervention content; (2) integrated content from another treatment approach; (3) removed or skipped core modules of Project ImPACT; (4) integrated or substituted extra topics; (5) changed the length of time spent on a certain topic within Project ImPACT; (6) changed the number of weeks that Project ImPACT was delivered; and (7) adjusted the order of the intervention content. After they rated each item, providers were asked to describe any reported adaptations.

Analysis

Quantitative analyses

Quantitative data analyses were conducted in Statistical Package for the Social Sciences (SPSS version 28). Descriptive data was used to examine provider attendance in group consultation, intervention fidelity over the course of study participation, and reported global adaptations to Project ImPACT. Changes in providers’ efficacy delivering Project ImPACT were compared using a one-way ANOVA with Tukey HSD follow-up to determine differences across the three time points (i.e. pre-training, post-training, and post-consultation). Mean fidelity scores from the first and final coaching sessions for providers who submitted more than two of their sessions were also compared using a one-way ANOVA. Providers with greater than two sessions were included in these analyses to ensure that fidelity scores reflected opportunity to respond to consultative feedback. Statistical significance was evaluated at the 0.05 threshold.

Qualitative analysis of session process

Provider responses to open-ended questions about their session delivery were de-identified and imported in MAXQDA for analysis. Qualitative analysis occurred by two research team members (K.P and N.H.), one with a strong background in qualitative analysis. Given that all qualitative data was obtained from open-ended survey data (versus iterative or semi-structured interviewing), content analysis was used to categorize responses as they directly related to the survey questions (Elo et al., 2014; Hsieh & Shannon, 2005). First responses to the question, “Did anything unexpected happen in your session?” were categorized into sessions in which an unexpected event was reported or not reported. For sessions in which an unexpected event was reported, the description of these events was reviewed by the two team members together and to determine codes directly related to the content of the response (Hsieh & Shannon, 2005). This codebook was then applied to each of the responses with consensus coding used for reliability. Following analysis, qualitative findings were presented back to participants as part of round tables and a presentation at Georgia’s Interagency Coordinating Council, which included Early Intervention providers, caregivers, and other state-level administrators. During member checking, participants agreed with the primary qualitative themes and their interpretation.

Community involvement statement

The ideas and results presented in this article were collected in collaboration with providers and administrators within Georgia’s Early Intervention system. In addition to participating in training, consultation, and qualitative research procedures, the specific aims and data collection processes were guided by input from Early Intervention providers and administrators. As stated, the research findings were also presented back to providers and administrators through a series of round tables and during Georgia’s Interagency Coordinating Council meeting. Finally, this article was reviewed by state-level administrators.

Results

Provider enrollment and attendance

Twenty-seven of 28 providers who enrolled in a Project ImPACT training completed the introductory tutorial and live virtual training. Twenty-four providers then participated in weekly group consultation. Group consultation attendance was high, with providers attending an average of 92.38% of consultation sessions (range: 67–100%). Providers reported being satisfied with group consultation participation (M = 6.86 out of 7; SD = 0.47) and reported that they would recommend the consultation process to colleagues (M = 6.82; SD = 0.50). Throughout group consultation, providers submitted an average of 3.87 videos for review (range 0–9; SD = 2.47). Four providers did not submit video. For two providers, this was due to difficulty recording sessions. For the other two providers, video was not shared as the families they selected to practice Project ImPACT with were outside the Early Intervention system (i.e. were receiving private services within the providers’ agencies).

Provider self-efficacy and fidelity

Providers reported significantly higher self-efficacy delivering Project ImPACT over the course of their participation in training and group consultation (F(2, 68) = 12.60; p < 0.001). Follow-up indicated that providers reported significantly higher self-efficacy when comparing ratings before attending Project ImPACT training (M = 5.73; SD = 3.77) to those after attending the training (M = 8.58; SD = 1.41; p < 0.01). Self-efficacy ratings remained relatively stable from the end of training to the end of group consultation (M = 9.10; SD = 1.26). In addition, for the 14 providers who submitted three or greater videos of Project ImPACT coaching sessions for review and feedback, provider fidelity increased from an average of 68.86% for their first coaching session to 77.46% for their final coaching session (F(1, 26) = 2.19, p = 0.12). However, as can be seen in Figure 1, there was significant variability in the extent to which providers delivered Project ImPACT. Of the providers who submitted more than two coaching videos, about a quarter achieved fidelity no higher than 50%, closer to half increased their fidelity but never achieved scores greater than 80%, and another quarter consistently increased or delivered the program at fidelity (i.e. greater than 80%).

Figure 1.

Provider fidelity across Project ImPACT sessions.

Provider descriptions of session processes

A total of 52 Project ImPACT sessions were qualitatively described across 24 providers. Provider responses about how their session unfolded and whether an unanticipated event occurred were categorized into the following: (1) no unexpected event present; (2) a crisis event was disclosed; (3) an unexpected event related to the child occurred; (4) an unexpected event related to the caregiver or family occurred; or (5) an unexpected “other” event occurred. For the sessions in which unexpected events occurred, providers’ description of their response to the event was then categorized into the following: (1) no reported impact on the delivery of Project ImPACT; (2) the event was addressed head-on in a family-centered manner; (3) the session was rescheduled in response to the event; or (4) the session was adapted in another way.

In total, an unexpected event was reported in 52% of Project ImPACT sessions. Figure 2 depicts the frequency to each type of unexpected event and providers’ reported responses.

Figure 2.

Provider-reported response across sessions with unexpected events.

Events categorized as disclosed crises occurred in 9.62% of sessions and included: acute family illness or hospitalization, death of a family member, a car accident, and reported housing insecurity. In describing a crisis event, one provider indicated the following:

The mom’s mother was admitted in the hospital with COPD. They are sending her home to live at the family’s house this week. The mom was trying to get hospice set up. I realize our job is to work with the child. Mom just needed to talk.

In addition to crises, providers indicated that a child’s specific needs were unanticipated in 17.31% of sessions. In many sessions, child-related unexpected events were tied to emotion dysregulation or the presence of behaviors such as tantrums or self-injury. One provider described, “We spent the hour discussing the significant issues that mom was having with tantrums and also how to deal with [child’s] behaviors.” In other instances, these events included child illness or a conflicting nap schedule that interrupted the practice of Project ImPACT intervention strategies.

In 9.68% of sessions, providers described that a non-acute caregiver or family circumstance impacted their delivery of Project ImPACT. This included the need to support caregiver mental health or problem solve around other family members’ needs. In more than one reported session, a provider indicated, “I felt that the mom was really struggling with her own mental health this week. This is in all intervention sessions; we need time to meet the parent needs and the intervention that needs to be covered.” Other family circumstances were described, such as, “The family was staying at another family member’s house this week. There were a lot of people present and no toys available to practice. We needed to problem solve this chaotic environment, and it took time.”

Finally, in 15.38% of sessions, providers indicated other unexpected events that were not related to a crisis, the child, or a family need. For providers delivering services via telehealth, this at times was related to technology challenges, including inconsistent Internet connection or difficulty sharing the screen to show intervention-related materials. In addition to technology challenges, one provider described two sessions in which another therapist was unexpectedly present and the need to adjust to co-treat with this therapist: “There was another therapist present, and I did not know she would be there. She took over the session, and I had to try to insert myself here and there with content.”

When asked about how they responded to unexpected events, in 37% of sessions, providers indicated that there was no impact on the delivery of Project ImPACT. This appeared to be particularly true for “unexpected other events” in which the provider reported, for example, being able to troubleshoot technology issues relatively quickly prior to returning to Project ImPACT content. In an additional 37% of sessions, providers indicated that the unexpected event was addressed in a family-centered manner that resulted in augmenting Project ImPACT by adding content, topics, or responses not typically included in the intervention. Project ImPACT content was sometimes paused for this augmentation. For example, in response to a child-specific event in which challenging behaviors were noted, one provider shared,

I let mom talk and validated what she was saying. I also brainstormed and gave mom some suggestions on how to deal with the behavior issues with her daughter. This took a while, but mom was much calmer at the end of the session.

Although some providers indicated pausing Project ImPACT for the entirety of the session to augment the program, others indicated they were able to integrate content that was responsive to the family’s needs while also delivering Project ImPACT within the same session. For example: “I responded by educating mom and coaching her through his dysregulated moment with sensory-based strategies. We then returned to Project ImPACT during book sharing.”

Other adaptations to Project ImPACT were reported in 18.5% of sessions in which an unexpected event was present. These adaptations involved the repetition of intervention content. For example, when a provider shared that the child was unexpectedly irritable within the session, they reportedly responded by suggesting the caregiver, “move to repeat the bottom of the pyramid and just imitate and focus on [the child] to maintain his engagement.”

In describing how they responded to unexpected events, providers sometimes shared the rationale for why they augmented Project ImPACT and/or why they temporarily paused their delivery of the program. This rationale often included a perceived need to maintain rapport with the caregiver, to respond to a family’s important other priorities, and to maintain caregiver engagement. For example, one provider shared, “I let the mom share about how their family is doing and how they are recovering. I felt that was important to make the mom feel heard and understood.”

Provider-reported categorical adaptation

At the conclusion of group consultation and Project ImPACT delivery, providers reported the extent to which they made adaptations to Project ImPACT, with questions based on the FRAME (Stirman et al., 2019). Providers’ reports of adaptations are summarized in Table 3. On average, providers reported making adaptations infrequently. They indicated most often adapting the length of their Project ImPACT sessions (M = 2.33; SD = 0.91), integrating other intervention strategies into their Project ImPACT sessions (M = 2.33, SD = 1.14), and adapting the duration with which they delivered Project ImPACT (M = 2.29; SD = 1.27).

Table 3.

Provider-reported adaptations to Project ImPACT following delivery.

Description	M	SD	Range
Tailor content	2.19	1.03	1–5
Remove topics	1.48	0.75	1–3
Integrate additional topics	1.81	0.93	1–3
Adapt session length	2.33	0.91	1–4
Adapt program duration	2.29	1.27	1–5
Adjust the order of content	1.45	0.69	1–3
Integrate other interventions	2.33	1.14	1–5

SD: standard deviation.

Scaling for adaptation ratings: 1 indicates that the provider never made the adaptation, 3 indicates that they made the adaptation in about half of sessions, and 5 indicates that they made the adaptation in nearly all sessions.

When asked to describe their endorsed adaptations to Project ImPACT, six providers expressed that their primary adaptation to the program was integrating content that addressed other priorities for the child and family. For example, one provider reported, “We may have been working on the middle of the pyramid [i.e. specific Project ImPACT strategies], and the parent wanted to talk potty-training . . . then we redirected back to the middle of the pyramid.” Another provider described, “I integrated the broader knowledge of ASD, resources, and self-regulation strategies into the work of Project ImPACT with success,” and “Sometimes parents have different concerns that are not related to the project or different strategies will work for the child’s challenges.” In addition to integrating content to the delivery of Project ImPACT, five providers expressed the need to adjust the length of the 12-session Project ImPACT model. Most often they described that this involved repeating content.

Providers were also asked if they ever paused their delivery of Project ImPACT and, if so, to describe the rationale for doing so. Twelve of 24 participating providers reported pausing Project ImPACT at some point during their delivery. Description of these pauses aligned with session-level descriptions in that providers reported pausing delivery in response to a variety of events, including crises, other caregiver priorities, and child needs. For example, one provider noted,

I had to pause it one time because mom had other issues going on in the home. We still discussed Project ImPACT but did not practice the next step of the program with the child.” Another noted that they paused “only for “crisis” days or days to teach sensory supports for dysregulation.

Discussion

The current study examined how 24 Early Intervention providers delivered an evidence-based, PMI, Project ImPACT, within a Part C Early Intervention system. Results demonstrated that providers attended group consultation sessions consistently and were satisfied with their participation in both training and consultation. In addition, providers reported significantly higher self-efficacy delivering Project ImPACT after participating in training and, on average, appeared to deliver Project ImPACT with higher fidelity in response to consultative feedback, although this finding was not significant. Despite positive trends in fidelity ratings, there was significant variability in how Project ImPACT was delivered, with some providers delivering the program with relatively low fidelity, even in response to consultation, some increasing their fidelity throughout consultation, and others demonstrating consistently high fidelity for the duration of participation. In addition to evaluating Project ImPACT fidelity, this study examined how providers reported delivering and adapting Project ImPACT in response to specific events that arose within their sessions. This preliminary qualitative data demonstrated that a variety of events arose that reportedly drove providers to augment or pause their delivery of the program to directly respond to the event. At the conclusion of participation, providers indicated similar types of global adaptations, including the need to integrate other content into Project ImPACT and to pause the intervention in response to family needs and priorities not covered within Project ImPACT.

The findings from this study highlight the importance of examining how and why interventions are delivered within systems that support young children with a high likelihood of having ASD. Consistent with other research, providers in this study had relatively low Project ImPACT parent coaching fidelity at baseline (Pellecchia et al., 2022) and increased their fidelity to Project ImPACT in response to ongoing consultation (Stahmer et al., 2020). However, the findings from this study also suggest that average fidelity scores may mask the significant variability in how evidence-based PMIs are delivered. This variability is important to highlight and may reflect the diverse experience level of providers being trained, the complexity of PMI models, varied administrative support for providers to apply their learning outside of group consultation, and/or the effectiveness of the training models being used to support Early Intervention providers in learning PMIs (Aranbarri et al., 2021; Pickard et al., 2021, 2023). The training model used within this study incorporated elements consistent with adult learning theory, including the use of role play, video review, and direct feedback within both training and group consultation. However, training was not formally tailored to account for the baseline skill and learning needs of individual providers, or administrative support of individual providers. Further, this study was not able to examine the impact of specific training activities on provider fidelity, although this is an important area of future research, particularly given growing interest to characterize the active ingredients of consultation within community systems (McLeod et al., 2018).

It is important to note that providers within this study were almost exclusively independent contractors spread across a variety of health districts representing many counties within the state of Georgia. Thus, although this study occurred within a Part C Early Intervention system, the setting of this study may be somewhat distinct from other ASD implementation research being conducted in Early Intervention systems, which has often occurred in partnership with providers employed or contracted within Part C agencies (Pellecchia et al., 2022; Stahmer et al., 2020), and/or as part of effectiveness research. The lack of agency-level partnership meant that individual providers’ attendance and use of Project ImPACT strategies was not monitored or overseen. This high level of decentralization raises questions regarding the sustainability of intensive training and group consultation models in this system. Specifically, it will be critical to consider the extent to which the models used are able to be sustained within systems that employ a high number of independent contractors, that experience high provider and administrator turnover, and that do not have formal supervision structures in place. In the absence of this infrastructure, it may be difficult for providers to attend, implement, and sustain their use of evidence-based practices (EBPS) (Pickard et al., 2023).

In addition to examining Project ImPACT fidelity, providers were asked to describe how they delivered Project ImPACT immediately following a random sample of their intervention sessions. Providers’ qualitative report of “unexpected” events that influenced their Project ImPACT deliver was largely consistent with other research describing the presence of emergent events within parent-mediated interventions and other mental health interventions delivered in outpatient community settings (Guan et al., 2017; Lau et al., 2018; Lind et al., 2020). However, it is important to note that previous research in this area has primarily focused on evaluating in-session crises that constitute acute stressors (Guan et al., 2017). The preliminary findings from this study suggest that it is important to consider events, beyond just crises, that influence how providers deliver Project ImPACT. A broader range of events, including a child’s complex needs, a family’s other goals for their child, and a caregiver’s own mental health represent other realistic priorities that emerge within intervention delivery (Pickard et al., 2022). Further, although this study focused on provider decisions within Project ImPACT, the families receiving Project ImPACT included those often underrepresented within autism research. Thus, the adaptation reported by providers may represent those critical to align PMI with family priorities or needs not considered within intervention development and efficacy trials (Steinbrenner et al., 2022).

The preliminary qualitative data highlighting how and why providers’ delivered Project ImPACT has at least two implications. First, in addition to describing the events that arose within Project ImPACT sessions, providers indicated the rationale for how they responded to these events. In many instances, providers noted that their decisions were made to support the engagement of caregivers and to align Project ImPACT with the priorities of families. This finding is a common rationale for intervention adaptation (Kim et al., 2020; Stirman et al., 2019), and is consistent with adaptation models such as the model for adaptation design and impact (MADI), which describe how the rationale for adapting an intervention may mediate the relationship between the adaptation itself and service outcomes (Kirk et al., 2020). However, adaptation models such as MADI also suggest that fidelity consistency should mediate the relationship between an adaptation and its impact on service outcomes (Kirk et al., 2020). Within the current study, some reported adaptations were likely fidelity-consistent (i.e. repeating intervention content), whereas others, including pausing the intervention or not providing active coaching, may have reduced Project ImPACT fidelity within that session. However, providers’ perception that these same adaptations enhanced caregiver engagement or caregiver rapport raises some question regarding the core functions of PMIs when delivered in complex systems. It will be important to better understand the priorities of families and providers within Early Intervention systems, to understand how these priorities relate to existing conceptualizations of fidelity, and to consider adaptive interventions that are user-centered and, thus, able to focus on the most common needs expressed by families within these systems.

There are several limitations to note in this study, including small sample size, missing observational coding of session-level adaptation and processes, and missing caregiver perspectives on how providers delivered Project ImPACT. Session-level process data was not collected from every session, limiting the ability to fully estimate the presence of intervention adaptation. Similarly, quantitative adaptation measures were only collected following consultation; thus, it was not possible to examine the relationship between reported adaptation and provider fidelity more systematically. It is also possible that providers adapted how they delivered Project ImPACT in response to events that were not perceived to be unexpected. For example, a caregivers’ request for information unrelated to Project ImPACT or a caregivers’ cultural values and preferences may drive providers to adapt Project ImPACT but were not reported by providers as being unexpected. Finally, this study reports provider fidelity while actively participating in ongoing consultation rather than their fidelity after completing consultation processes. Thus, the inconsistent fidelity may be reflective of how fidelity was examined, which was also not able to look at changes in provider fidelity in response to specific consultative feedback or activities. Given that this study was conducted as part of an ongoing service contract, there are limitations in the research methods used to examine providers’ fidelity trajectories. Future research using more systematic single case study designs or with larger samples using multiple regression modeling would be better equipped to examine changes in provider fidelity over time.

Conclusion

There is mounting interest in increasing access to evidence-based practices for autism, including parent-mediated interventions, by translating them into community systems that provide a natural entry point into services for young autistic children. Although the Early Intervention system has been thought of as an ideal translational setting, provider fidelity in these settings has been variable and not well described. The current study provides preliminary insight regarding the variability in provider fidelity to PMI and the decisions providers weigh within their delivery. Results suggest the importance of carefully examining how providers deliver evidence-based interventions, the impact of these decisions on fidelity metrics, and service outcomes when engaging in implementation efforts within Early Intervention systems.

Footnotes

Acknowledgements

We would like to acknowledge the significant support of administrators, providers, and families within Georgia’s Part C Early Intervention system, Babies Can’t Wait, who contributed to the ideas presented in this commentary and who supported data collection and member checking. We would like to especially acknowledge the effort and support of Synita Griswell, the autism services coordinator within Babies Can’t Wait. Additionally, we would like to thank members of the Marcus Autism Center research team who supported this work including Eileen Kaiser, Jennifer Stapel-Wax, and Naima Bond.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Katherine Pickard

References

Adams

R. C.

Tapia

., & Council on Children with Disabilities. (2013). Early Intervention, IDEA Part C services, and the medical home: Collaboration for best practice and best outcomes. Pediatrics, 132(4), e1073–e1088.

Aranbarri

Stahmer

A. C.

Talbott

M. R.

Miller

M. E.

Drahota

Pellecchia

Barber

A. B.

Griffith

E. M.

Morgan

E. H.

Rogers

S. J.

(2021). Examining U.S. public early intervention for toddlers with autism: Characterizing services and readiness for evidence-based practice implementation. Frontiers in Psychiatry, 12, Article 786138.

Barnett

M. L.

Brookman-Frazee

Gonzalez

J. C.

Zhan

Rodriguez

Stadnick

N. A.

Lau

A. S.

(2019). Qualitative reports of how and when therapists adapt children’s evidence-based practices during community implementation. Journal of Clinical Child & Adolescent Psychology, 48, 894–905.

Brian

Drmic

Roncadin

Dowds

Shaver

Smith

I. M.

Zwaigenbaum

Sacrey

L. R.

Bryson

S. E.

(2022). Effectiveness of a parent-mediated intervention for toddlers with autism spectrum disorder: Evidence from a large community implementation. Autism, 26, 1882–1897.

Casagrande

K. A.

Ingersoll

B. R.

(2017). Parent-mediated interventions for social communication in young children with ASD. In Leaf

J. B.

(Ed.), Handbook of social skills and autism spectrum disorder (pp. 285–312), Springer.

Chambers

D. A.

Norton

W. E.

(2016). The adaptome: Advancing the science of intervention adaptation. American Journal of Preventive Medicine, 51(4), S124–S131.

Cicchetti

D. V.

(1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290.

Dawson

(2008). Early behavioral intervention, brain plasticity, and the prevention of autism spectrum disorder. Development and Psychopathology, 20, 775–803.

Dickson

K. S.

Lind

Jobin

Kinnear

Lok

Brookman-Frazee

(2021). Correction to: A systematic review of mental health interventions for ASD—Characterizing interventions, intervention adaptations, and implementation outcomes. Administration and Policy in Mental Health and Mental Health Services Research, 48(5), 884–908.

10.

Dyson

M. W.

Chlebowski

Brookman-Frazee

(2019). Therapists’ adaptations to an intervention to reduce challenging behaviors in children with autism spectrum disorder in publicly funded mental health services. Journal of Autism and Developmental Disorders, 49(3), 924–934.

11.

Edmunds

S. R.

Frost

K. M.

Sheldrick

R. C.

Bravo

Straiton

Pickard

Grim

Drahota

Kuhn

Azad

Ramos

A. P.

Ingersoll

Wainer

Ibanez

L. V.

Stone

W. L.

Carter

Broder-Fingert

(2022). A method for defining the CORE of a psychosocial intervention to guide adaptation in practice: Reciprocal imitation teaching as a case example. Autism, 26, 601–614.

12.

Eisenhower

Martinez Pedraza

Sheldrick

R. C.

Frenette

Hoch

Brunt

Carter

A. S.

(2021). Multi-stage screening in early intervention: A critical strategy for improving ASD identification and addressing disparities. Journal of Autism and Developmental Disorders, 51(3), 868–883.

13.

Elo

Kääriäinen

Kanste

Pölkki

Utriainen

Kyngäs

(2014). Qualitative content analysis: A focus on trustworthiness. SAGE Open, 4(1), 2158244014522633.

14.

Guan

Levy

M. C.

Kim

R. E.

Brown

T. E.

Reding

M. E.

Rith-Najarian

Sun

Lau

A. F.

Chorpita

B. F.

(2017). Managing in-session “surprises”: Provider responses to emergent life events during evidence-based treatment implementation. Administration and Policy in Mental Health and Mental Health Services Research, 44(2), 164–176.

15.

Hsieh

H. F.

Shannon

S. E.

(2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288.

16.

Individuals With Disabilities Education Act 20 U.S.C. § 1400 2004.

17.

Ingersoll

Dvortcsak

(2019). Teaching social communication to children with autism and other developmental delays (2nd ed.). Guilford Press.

18.

Kim

J. J.

Brookman-Frazee

Barnett

M. L.

Tran

Kuckertz

Lau

A. S.

(2020). How community therapists describe adapting evidence-based practices in sessions for youth: Augmenting to improve fit and reach. Journal of Community Psychology, 48(4), 1238–1257.

19.

Kirk

M. A.

Moore

J. E.

Stirman

S. W.

Birken

S. A.

(2020). Towards a comprehensive model for understanding adaptations’ impact: The model for adaptation design and impact (MADI). Implementation Science, 15(1), Article 56.

20.

Lau

A. S.

Gonzalez

J. C.

Barnett

M. L.

Kim

J. J.

Saifan

Brookman-Frazee

(2018). Community therapist reports of client engagement challenges during the implementation of multiple EBPs in children’s mental health. Evidence-based Practice in Child and Adolescent Mental Health, 3(3), 197–212.

21.

Lind

Lau

A. S.

Gomez

Rodriguez

Guan

Chlebowski

Zhang

Chorpita

Brookman-Frazee

(2020). Emergent life events in the delivery of a caregiver mediated evidence-based intervention for children with autism spectrum disorder in publicly funded mental health services. Autism, 24(5), 1286–1299.

22.

Maenner

M. J.

Shaw

K. A.

Bakian

A. V.

Bilder

D. A.

Durkin

M. S.

Esler

Furnier

S. M.

Hallas

Halle-Lande

Hudson

Hughes

M. M.

Patrick

Pierce

Poynter

J. N.

Salinas

Shenouda

Vehorn

Warren

Constantino

J. N.

Cogswell

M. E.

(2021). Prevalence and characteristics of autism spectrum disorder among children aged 8 years: Autism and Developmental Disabilities Monitoring Network, 11 sites, United States, 2018. Morbidity and Mortality Weekly Report Surveillance Summaries, 70(11), 1–16.

23.

McLeod

B. D.

Cox

J. R.

Jensen-Doss

Herschell

Ehrenreich-May

Wood

J. J.

(2018). Proposing a mechanistic model of clinician training and consultation. Clinical Psychology: Science and Practice, 25(3), Article e12260.

24.

Miller

C. J.

Wiltsey-Stirman

Baumann

A. A.

(2020). Iterative Decision-making for Evaluation of Adaptations (IDEA): A decision tree for balancing adaptation, fidelity, and intervention impact. Journal of Community Psychology, 48(4), 1163–1177.

25.

Nevill

R. E.

Lecavalier

Stratis

E. A.

(2018). Meta-analysis of parent-mediated interventions for young children with autism spectrum disorder. Autism, 22(2), 84–98.

26.

Noyes-Grosser

D. M.

Benham

Osbahr

(2018). Implementing services for children with ASD and their families within state early intervention programs: A view from two states–opportunities, challenges, and future directions. In Siller

Morgan

(Eds.), Handbook of parent-implemented interventions for very young children with autism (pp. 459–481). Springer.

27.

Pellecchia

Mandell

D. S.

Beidas

R. S.

Dunst

C. J.

Tomczuk

Newman

Zeigler

Stahmer

A. C.

(2022). Parent coaching in early intervention for autism spectrum disorder: A brief report. Journal of Early Intervention. Advance online publication. https://doi.org/10.1177/10538151221095860

28.

Pickard

Hendrix

Greenfield

(May, 2023). Examining the infrastructure for the adoption of autism evidence-based practices within an early intervention system [Poster presentation]. International Society for Autism Research Annual Meeting, Stockholm, Sweden.

29.

Pickard

Hendrix

Guerra

Kaiser

Bond

Stapel-Wax

(2022, May). Examining the balance of fidelity and family-centered care within a parent-mediated intervention delivered in a state-funded Early Intervention system [Conference session]. International Society for Autism Research (INSAR) Annual Meeting, Austin, TX, United States.

30.

Pickard

Mellman

Frost

Reaven

Ingersoll

(2021). Balancing fidelity and flexibility: Usual care for young children with an increased likelihood of having autism spectrum disorder within an early intervention system. Journal of Autism and Developmental Disorders, 53, 656–668.

31.

Pickard

K. E.

Kilgore

A. N.

Ingersoll

B. R.

(2016). Using community partnerships to better understand the barriers to using an evidence-based, parent-mediated intervention for autism spectrum disorder in a medicaid system. American Journal of Community Psychology, 57(3-4), 391–403.

32.

Rogers

S. J.

Stahmer

Talbott

Young

Fuller

Pellecchia

Barber

Griffith

(2022). Feasibility of delivering parent-implemented NDBI interventions in low-resource regions: A pilot randomized controlled study. Journal of Neurodevelopmental Disorders, 14(1), 1–14.

33.

Sheldrick

R. C.

Hyde

Leslie

L. K.

Mackie

(2021). The debate over rational decision making in evidence-based medicine: Implications for evidence-informed policy. Evidence & Policy, 17(1), 147–159.

34.

Stahmer

A. C.

Rieth

S. R.

Dickson

K. S.

Feder

Burgeson

Searcy

Brookman Frazee

(2020). Project ImPACT for Toddlers: Pilot outcomes of a community adaptation of an intervention for autism risk. Autism, 24(3), 617–632.

35.

Steinbrenner

J. R.

McIntyre

Rentschler

L. F.

Pearson

J. N.

Luelmo

Jaramillo

M. E.

Boyd

B. A.

Wong

Nowell

S. W.

Odom

S. L.

Hume

K. A.

(2022). Patterns in reporting and participant inclusion related to race and ethnicity in autism intervention literature: Data from a large-scale systematic review of evidence-based practices. Autism, 26(8), 2026–2040.

36.

Stirman

S. W.

Baumann

A. A.

Miller

C. J.

(2019). The FRAME: An expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14(1), Article 58.

37.

Tomasello

N. M.

Manning

A. R.

Dulmus

C. N.

(2010). Family-centered early intervention for infants and toddlers with disabilities. Journal of Family Social Work, 13(2), 163–172.

38.

United States Department of Education. (2021). 43rd Annual Report to Congress on the Implementation of the Individuals with Disabilities Education Act. https://sites.ed.gov/idea/files/43rd-arc-for-idea.pdf

39.

Vivanti

Kasari

Green

Mandell

Maye

Hudry

(2018). Implementing and evaluating early intervention for children with autism: Where are the gaps and what should we do? Autism Research, 11(1), 16-23.

40.

Yoder

P. J.

Stone

W. L.

Edmunds

S. R.

(2021). Parent utilization of ImPACT intervention strategies is a mediator of proximal then distal social communication outcomes in younger siblings of children with ASD. Autism, 25(1), 44–57.