Abstract
This study reports child and family outcomes from a community-based, quasi-experimental pilot trial of Project ImPACT for Toddlers that is a parent-mediated, naturalistic, developmental behavioral intervention for children with or at-risk for autism spectrum disorder developed through a research–community partnership. Community early interventionists delivered either Project ImPACT for Toddlers (n = 10) or Usual Care (n = 9) to families based on Part C assigned provider. Twenty-five families participated, with children averaging 22.76 months old (SD = 5.06). Family and child measures were collected at intake, after 3 months of service, and after a 3-month follow-up. Results indicate significantly greater improvements in positive parent–child interactions for Project ImPACT for Toddlers than usual care families, as well as large, but non-significant, effect sizes for Project ImPACT for Toddlers families in children’s social and communication skills.
Childhood developmental, social, and communicative challenges often indicate possibility of future identification of neurodevelopmental challenges such as autism spectrum disorder (ASD 1 ; Landa, Holman, & Garrett-Mayer, 2007; Zwaigenbaum et al., 2005). These issues represent a significant public health challenge, with one in four children estimated to be at risk for developmental, social, or behavioral delays (Altevogt, Hanson, & Leshner, 2008), which are associated with ongoing linguistic, educational, and social difficulties (Briggs-Gowan & Carter, 2008; Stothard, Snowling, Bishop, Chipchase, & Kaplan, 1998). These toddlers are increasingly likely to receive an ASD diagnosis, which is rising at dramatic rates, with a current incidence of 1 in 59 children (Baio et al., 2018).
As ASD identification improves, communities are struggling to implement appropriate, effective programs for toddlers and their families. Although evidence-based practices for ASD exist (Wong et al., 2014), practices specifically designed for toddlers have not been implemented in community settings where children receive routine care. In addition, in many early intervention programs, children later diagnosed with ASD receive services due to speech–language concerns exclusively, and many agencies wait until age 3 to establish a diagnosis (Yeates & Stahmer, 2017). This means that ASD-specific intervention may not be introduced until later in development than optimal, potentially due to concerns about identifying ASD incorrectly at a very young age (Rossi, Newschaffer, & Yudell, 2013). Furthermore, the effectiveness of most interventions for children with early signs that may be indicative of ASD, but not a confirmed diagnosis, has not been established. On local and national levels, there are calls for capacity building for this population of young children with social communication challenges (Altevogt et al., 2008; Cordero et al., 2006; Zero to Three, 2005).
Addressing issues related to ASD concerns in early invention has the potential to produce positive downstream effects. In fact, intervening at the first signs of ASD, prior to a formal diagnosis, may be a powerful option and may reduce social communication challenges and make learning from social interaction easier for some children (Dawson, 2008; Lavelle et al., 2014). A focus on moving high-quality, evidence-based intervention into community intervention settings where it can be delivered as early as possible may increase provider capacity to serve toddlers with early signs of ASD, improve outcomes for children and families, and reduce overall costs of supporting this population (Cidav et al., 2017).
Need for evidence-based practices in the community
To achieve these goals, evidence-based practices (EBPs) specifically designed to meet the needs of toddlers with social communication delays or ASD and that are feasible to deliver within existing Individuals with Disabilities Education Act (IDEA) Part C early intervention services are required. Part C programs include publicly funded services provided to children under age 3 in the United States who have identified developmental concerns. Eligibility, identification of risk for ASD, and service provision vary widely by state and local agency. Fortunately, during the last decade, we have learned a great deal about EBPs for toddlers with ASD. A review by leading experts in the field led to recommendations for best practices for young children with ASD or early ASD signs. These recommendations include: (1) a combination of developmental and behavioral strategies, delivered as early as possible; (2) active involvement of caregivers as part of the intervention; (3) a focus on social communication needs; and (4) individualization based on family needs and culture in clinical care (Zwaigenbaum et al., 2015). These recommendations are consistent with community-focused recommendations for the use of family-centered help-giving practices (Dunst, Trivette, & Hamby, 2007) and fit with the mission of Part C services broadly, which is to enhance the development of infants and toddlers with disabilities and the capacity of families to meet the needs of their young children with disabilities (Individuals with Disabilities Education Improvement Act of 2004 (IDEA), 2004). In fact, including the caregiver is key to the mission of Part C because expected outcomes of early intervention include: (1) families understand their child’s strengths, abilities, and special needs; (2) families help their child develop and learn; and (3) improvements in family quality of life (Bailey et al., 2006).
Despite wide agreement on how to best support toddlers with ASD, however, meeting these recommendations in practice remains challenging. Family involvement is not widely implemented in community settings, despite being a value and mandate of publicly implemented early intervention systems. For example, data indicate that a majority of Part C early intervention sessions involve the parent playing a passive rather than active role in services (Campbell & Sawyer, 2007). Current training for early intervention providers equips them to work directly with children presenting with general developmental delays, but not to build parent capacity, coach parents effectively (Fleming, Sawyer, & Campbell, 2011), or address learning needs of children with ASD specifically (Ingersoll, Meyer, Bonter, & Jelinek, 2012). Recently, the field of early intervention has begun to focus on how to optimally coach parents (Friedman, Woods, & Salisbury, 2012) as well as how to support children with early signs of ASD in the context of early intervention (Stahmer et al., 2018), but considerable work is needed in these areas to effectively move these shared values to routine community practice.
Current study: Project ImPACT for Toddlers
The purpose of the current pilot study is to test the impact of community early intervention providers delivering a community adapted evidence-based, parent coaching intervention. Specifically, the intervention, Project ImPACT for Toddlers (PI T ), represents an adaptation of Project ImPACT (Improving Parents as Communication Teachers; Ingersoll & Dvortcsak, 2010). Project ImPACT is an evidence-based, naturalistic developmental behavioral intervention (Ingersoll & Wainer, 2013; Schreibman et al., 2015) that focuses on equipping parents to support their child’s social and communication development in children from 18 months to 8 years old, and is soundly supported in the scientific literature (Wong et al., 2014). PI T was collaboratively adapted through a community–academic partnership consisting of a multidisciplinary group of providers, funding agency representatives, parents, and researchers, known as the BRIDGE (Bond, Regulate, Interact, Develop, Guide, Engage) Collaborative (see Brookman-Frazee, Stahmer, Lewis, Feder, & Reed, 2012), to meet the needs of all stakeholders serving the youngest children with ASD and their families. A systematic, mixed-method approach (Creswell & Creswell, 2017) was used to collect data to inform the adaptation of Project ImPACT for the community in a way that would also maintain the integrity of the intervention (see Rieth, Stahmer, & Brookman-Frazee, 2018 for details of the adaptation process). This type of intervention tailoring is recommended as a way to extend the reach of EBP into a community (Curran, Mukherjee, Allee, & Owen, 2008). Specific adaptations were made to address early intervention principles and practice recommendations, to match community values, and to facilitate provider training in active parent coaching (see Table 1). The BRIDGE Collaborative, in coordination with the original Project ImPACT developers, created PI T and associated intervention and training materials specifically for early intervention providers and families in the context of Part C services. See Table 2 for a description of the components of PI T .
Project ImPACT for Toddlers adaptations.
PIT Fidelity checklist: domains, items, and descriptions.
The primary aim of this study (Aim 1) was to examine intervention effects of PI T in community settings on parent and child factors from pre- to post-interventions. The secondary aim (Aim 2) was to examine the relationship between providers’ use of PI T parent coaching strategies and parent fidelity of the intervention. The final exploratory aim (Aim 3) was to examine the sustainment of intervention effects by examining changes in parent and child outcomes 3 months following the end of intervention. We hypothesized that parents in the PI T group would be more likely to use PI T strategies after receiving the intervention than parents in usual care (UC), and that children in the PI T group would make greater progress on social communication assessments than children in UC. We expected that provider use of PI T parent coaching strategies would be associated with parent fidelity and that the intervention changes for both children and parents would sustain after a 3-month follow-up.
Methods
The study is a quasi-experimental pilot test of PI T in community settings. Families enrolled in publicly funded (Part C) early intervention services in a large urban county received either PI T or typically available early intervention services (UC). Part C-funded agencies (n = 8) enrolled in the study and provider (n = 19) and family participants (n = 28) were recruited from within enrolled agencies. Families were assigned to therapists per standard practice in their agencies (i.e. based on provider availability/caseload and language match). Family referral to agencies was determined by the Part C administrator, based on the usual considerations of availability, location, language preference, and other family needs. Intervention group was determined based on training status of the providers. This research was approved by the Institutional Review Board at the University of California, San Diego and relied upon by all other participating institutions.
Participants
Agencies and providers
Agencies were recruited to enroll in the study based on existing relationships with the investigators, agency-initiated contact with the research team to express interest, and/or prior participation in earlier research projects related to early intervention with the same research group. All eight eligible agencies approached by the research team agreed to participate. Agency’s eligibility criteria included: (1) provides Part C-funded early intervention services to children identified as at-risk for ASD under 30 months and (2) employs three or more providers working with parent/child dyads. Agency types included school-based early intervention (N = 2), infant and early childhood services (N = 1), ABA-based services (N = 3), a Children’s Hospital ASD-specific clinic (N = 1), and a federally qualified health center (N = 1). All agencies provided Part C services through California Early Start.
In a partner project (Stahmer et al., 2018), groups of community providers at participating agencies self-selected to receive training in PI T from experienced clinical psychologists who were part of the PI T adaptation team. Detailed information about the training process for PI T as well as the adaptation is available in Rieth et al. (2018). In brief, PI T training for providers involved weekly, 2-h meetings across a period of 12 weeks. The content of the meetings alternated between didactic presentation of intervention content (including direct instruction to therapists, role-play activities among providers, and example videos) and hands-on practice of strategies with feedback from the trainer (i.e. volunteer children/parents attending sessions and providers taking turns in using the PI T strategies with the child, explaining strategies to the parent, and/or coaching the parent directly). Agencies participating in the training project supported their providers time to attend the training, and training was provided free of charge. In the current study, the only requirement for providers was employment by a participating agency, agreement to have sessions filmed, and willingness to refer new, eligible families to the research study as appropriate. Once enrolled, both the providers trained in PI T (n = 10 providers) and providers who had not received training in UC (n = 9 providers) were asked to refer potentially eligible families for whom they were providing services. Agencies that included both trained and untrained providers were eligible to refer families for both the PI T and UC groups (n = 5 agencies, 62.5% of all participating agencies); however, only one agency provided family referrals to both groups. Three agencies referred to UC only (i.e. no PI T trained providers at those agencies). The mean number of families completing treatment with each provider was 2.2 (range 1–4). As self-reported on a demographics questionnaire, providers’ primary discipline was behavior specialist (PI T = 4; UC = 0), child development (PI T = 1; UC = 1), early childhood education (PI T = 0; UC = 4), psychology (PI T = 2; UC = 2), and social work (PI T = 1; UC = 2). Providers’ highest level of education was either a bachelor’s degree (PI T = 2; UC = 4) or master’s degree and higher (PI T = 6; UC = 3). PI T and UC providers were similar in the number of years of experience in early intervention (PI T = 5.46; UC = 3.77) and number of years of experience in coaching parents (PI T = 4.92; UC = 7.27); PI T providers reported more experience working with individuals with ASD (PI T = 6.88; UC = 4.08, t(22) = –2.12, p = 0.05). When considering provider age, ethnicity, and education, no significant differences were found between the PI T and UC groups.
Parent–child dyads
Forty-four potential family participants were referred from PI T providers (n = 19) and UC providers (n = 25; see Figure 1). Inclusion criteria for parent/child dyads were: (1) child age 12–30 months at time of enrollment, (2) clinician has a current concern regarding ASD risk as defined by their Part C provider, (3) score in the “concern” range on one or more scales of the Communication and Symbolic Behavior Scales–Infant Toddler Checklist (CSBS–ITC), (4) English- or Spanish-speaking, and (5) parent consents to regularly participate in sessions with child. Exclusion criteria were (1) parent had not previously received PI T and (2) child was receiving 6 h or more per week of early intervention services overall. This final criterion regarding early intervention service amount was included to keep service intensity similar across the intervention and control groups (i.e. avoid comparing children receiving low-intensity parent-mediated intervention in PI T (3 h per week) with children receiving high-intensity direct service from a clinician (e.g. 15–20 h per week)).
Enrollment flow diagram.
Of the referred families, 16 children were not enrolled in the study: 5 in the PI T group and 11 in the UC group. Four of these families did not consent to video recording (PI T = 1; UC = 3), two families did not respond to multiple attempts to contact them (UC = 2), and five families reported that they did not have time to participate (PI T = 3; UC = 2). In addition, four families (UC = 4) were receiving more than 6 h of intervention service per week and one family (PI T = 1) had started receiving PI T services prior to referral. Enrolled dyads included 28 children and their parents. Three enrolled families did not complete the study. Two families in PI T ended participation early (one moved out of the area and the other initiated intensive intervention services, thus making them no longer eligible for the study). One family in the UC group ended participation early when their provider changed jobs. Overall, 12 child/parent dyads completed participation in the PI T group and 13 completed participation in the UC group. Six of these families who participated from intake to exit of treatment (PI T = 2; UC = 4) did not complete measures at follow-up due being non-responsive or not interested in continuing participation.
Each parent–child dyad was either enrolled in the PI T group (n = 12) or the UC group (n= 13) based on their providers’ training history. The mean age of participating children was 22.76 months (SD = 5.06) and 68.0% were male. Children were ethnically diverse, with 48.0% identifying as Hispanic/Latinx per parent report. The average Early Learning Composite (ELC) on the Mullen Scales was 72.14 (SD = 16.97). Children received an average of 3.7 h per week of intervention during the study. The biological mother was the respondent for 96% of the sample. Table 3 characterizes the sample across groups. There were no significant differences across groups, except that the PI T group included significantly more boys (n = 11) than the UC group (n = 6; χ2 (1, N = 25) = 5.94, p = 0.01).
Characteristics of the sample.
SD: standard deviation; ADOS-T: Autism Diagnostic Observation Schedule, Second Edition–Toddler Module; ADOS ASD Severity: ADOS-2 Module T autism severity score; Mullen ELC: Mullen Scales of Early Learning Composite Standard Score; MBCDI: MacArthur–Bates Communicative Development Inventory; Vineland ABC: Vineland Adaptive Behavior Scales, Second Edition Adaptive Behavior Composite Standard Score.
p < 0.05.
Procedures
Recruitment
Children and families were recruited from eight community Part C agencies. Upon referral, families completed a phone screening. If they met inclusion criteria and the parent consented to participate, the family was enrolled and completed intake assessments (see Assessments). The parent completed a set of standardized questionnaires and video recorded a 10-min play interaction with his or her child. After intake, the participating provider delivered either UC or the intervention (see intervention descriptions, below). Video recordings of these sessions were collected four times during the intervention period. Post-intervention assessments were completed at the conclusion of PI T or after approximately 3 months of participation (M= 3.01; SD = 1.38), to keep the timeline consistent across groups. Follow-up data were collected approximately 3 months after post-data (M = 3.01; SD = 0.58). Each assessment appointment lasted approximately 1–2 h and parents received US$10 for completion of each appointment.
PIT group
Children in the PI T group received early intervention services from providers who had received PI T training within their agency. Agencies delivering PI T had previously worked closely with the BRIDGE Collaborative to prepare one employee as an “Agency Trainer.” The trainer participated in a 12-week training with BRIDGE Collaborative experts on using PI T with families as well as how to effectively train therapists in the model (training structure described above). Additional details on the training model for PI T can be found in the work of Rieth et al. (2018). PI T services are designed to be delivered across 12 weeks and follow the structure of the PI T parent manual, which is provided to parents at the start of the service. The manual describes each strategy and provides examples for how to use the technique with young children. PI T services were provided to families through Part C either as general early intervention services (50% of families) or autism-focused services (50% of families). Two families received PI T in a center-based program (16.67%), and 10 received it in a home-based program (83.33%). The average intensity of PI T services was 2.5 h per week (SD = 0.90).
Community comparison group
UC providers were not trained in PI T but were employed by participating community agencies. Providers delivered whatever intervention techniques and strategies they provided to families in routine care. Services in UC were provided either as general early intervention services (77% of families), autism-focused services (8% of families), or special education early intervention services (15% of families). All services in UC were home-based. The average intensity of UC services was 2 h per week (SD = 1.83).
Assessments
Video coding procedures
For each measure requiring video coding, including Use of PI T Parent Coaching Strategies, Parent Interactions with Children: Checklist of Observations Linked to Outcomes (PICCOLO), and Project ImPACT for Toddlers –Parent Intervention Fidelity, research assistants naïve to the training condition scored the measure based on the video recordings. Coders learned one coding system each. All coders met a reliability criterion of 80% agreement with coding keys across two separate video recordings prior to beginning independent coding. Ongoing agreement was evaluated throughout the coding process. Coders with two consecutive videos below 80% agreement received a didactic review of components or elements for which they were having difficulty and were required to code two reliability videos at 80% agreement or above again before further independent coding. Reliability data are reported for each measure below.
Sample characterization measures
Demographics questionnaire
Parents completed a demographics questionnaire at pre-intervention providing basic family and child information.
Early intervention history interview
Parents were interviewed at pre- and post-interventions regarding the nature and history of early intervention received. The interviewer obtained information regarding the type of intervention, service provider, setting, duration, and frequency of the intervention. This measure has been used by the Autism Centers for Excellence research studies.
Autism Diagnostic Observation Schedule, Second Edition–Toddler Module
The Autism Diagnostic Observation Schedule, Second Edition–Toddler Module (ADOS-2; Lord, Luyster, Gotham, & Guthrie, 2012), completed at pre-intervention, is a standardized, observational measure of social and communication skills. It has been shown to have high reliability and discriminant validity. The ADOS-2 largely operationalizes the process of identifying ASD symptoms. The total score, social affect score, and restricted and repetitive behavior scales were used to characterize the sample.
Mullen Scales of Early Leaning
The Mullen Scales of Early Leaning (MSEL; Mullen, 1995), completed at pre-intervention, measures developmental level across domains. The MSEL can be administered to children from birth to 68 months of age. T scores, percentile ranks, and age equivalents can be computed for five scales separately (gross motor, visual reception, fine motor, expressive language, and receptive language; each scale has a mean of 50 and a standard deviation of 10). An overall standard score, the ELC can also be calculated (M = 100, SD = 15). Construct, concurrent, and criterion validity are all verified by the independent studies and the technical manual for the MSEL (Mullen, 1995). The ELC was used to characterize children in the sample at intake.
Intervention content and process measure
Use of PI T Parent Coaching Strategies was characterized across both groups (PI T and UC) via the video recorded intervention sessions that were recorded three to four times per family (n = 96 total videos, mean of 3.2 videos per family) either directly by the provider or by a member of the research team. All recorded sessions were behaviorally coded using a 22-item Likert-type scale measure to characterize the structure of the session and the provider’s approach to parent coaching. This measure was adapted from the Fidelity of Implementation for Coaching Sessions Form in the original Project ImPACT materials (Ingersoll & Dvortcsak, 2010: 368). Each item was rated on a 1–5 Likert-type scale, where 1 indicated the provider did not use that strategy/element or did so poorly, and 5 indicated that the provider used that strategy/element competently and at an appropriate intensity. The measure includes items around setting up the coaching environment (e.g. limited distractions, availability of materials, presence of developmentally appropriate toys), use of coaching strategies (e.g. explaining the session content, reviewing prior weeks’ goals, explaining key technique accurately, briefly demonstrating strategies competently, facilitating parent’s practice of the technique, providing clear and specific parent’s feedback), use of strategies to promote parent’s engagement (e.g. inviting comments/questions, acknowledging parent’s concerns, assessing parent’s understanding, recognizing parent’s strengths and efforts), and use of general strategies to partner with caregiver (e.g. responsive interaction style; reflective approach). Complete definitions for the measure are available from the authors.
A total of 32% of all video samples (n = 31) were evaluated by two independent coders. Intraclass correlation coefficients (ICCs) were calculated for each item to assess interrater reliability. ICCs ranged from 0.69 to 1.0 across items, with a mean of 0.89 (SD = 0.08). Overall, the codes exhibited good to excellent ICCs according to accepted standards (Cicchetti, 1994).
Dependent measures: parent
Parent Participatory Engagement Measure
Parents completed this 5-item questionnaire at the initial, middle, and final intervention sessions with their provider to examine whether training in parent coaching and engagement strategies in PI T would increase parent’s engagement in the intervention (Haine-Schlagel et al., 2016). The questions, regarding parent comfort with, as well as engagement and participation in intervention, are rated on a 1 (not at all) through 5 (very much) scale. The Parent Participatory Engagement Measure (PPEM) has demonstrated excellent internal consistency (Cronbach’s alpha estimates of 0.86–0.93 across subsamples; Haine-Schlagel et al., 2016). A total mean score was calculated across time points (initial, middle, and final sessions) for each parent and utilized in analyses.
Social Support Index
Parents completed the Social Support Index (SSI) at pre- and post-interventions, which measured their perceived support within the community (McCubbin, Patterson, & Glynn, 1982). This tool was developed to measure family social support, as well as the amount of community-based social support families believe exist in the community which we hypothesized would be higher in the PI T group due to provider training in coaching and engagement. The SSI comprises 17 questions that ask respondents to indicate their agreement or disagreement with a statement on a 5-point Likert-type scale (1 = strongly disagree; 5 = strongly agree). A higher score indicates a higher level of social support (maximum score of 85). The SSI has strong internal consistency, with an alpha across several samples of 0.82; it is also reported to have a test–retest stability correlation of 0.83 (Fischer, Corcoran, & Fischer, 2007). Mean total score was calculated at each time point, with pre- and post-intervention scores utilized to examine differences in social support between groups.
Parent Interactions with Children: Checklist of Observations Linked to Outcomes
The Parent Interactions with Children: Checklist of Observations Linked to Outcomes (PICCOLO; Roggman et al., 2013) conducted at pre- and post-interventions and 3-month follow-up, is an observational measure in which a parent–child interaction is video recorded and trained observers code-specific parenting behaviors known to predict children’s early social, cognitive, and language development. Specifically, the PICCOLO examines four domains of parenting behaviors, including affection, responsiveness, encouragement, and teaching. We hoped that this measure would provide an objective way to determine whether training in PI T would change positive parent skills. Each domain comprises seven to eight individual items that are rated on a 0–2 scale, where 0 = absent, 1 = rarely/briefly, and 2 = frequently. Higher scores are indicative of more developmentally supportive parent–child interactions. ICCs for individual items ranged from 0.73 to 0.82, with an average ICC of 0.77, and are considered good according to current standards (Cicchetti, 1994). The PICCOLO demonstrates strong reliability and both construct and predictive validity. Mean scores for each domain (affection, responsiveness, encouragement, and teaching) were calculated at each designated time point and utilized in analyses.
Project ImPACT for Toddlers–Parent Intervention Fidelity
PI T Fidelity (Project ImPACT for Toddlers–Parent Intervention Fidelity) definitions were used to evaluate parents’ use of PI T components during parent–child interactions. Definitions were adapted from the original Project ImPACT Intervention Fidelity Checklist (Ingersoll & Dvortcsak, 2010) to reflect PI T adaptations. The PI T fidelity measure includes 19 items that comprise seven composites reflective of PI T strategies (see Table 2 for list of components and definitions). Items are rated on a 1–5 Likert-type scale (1 = minimal or no use of the strategy throughout the observation; 5 = correct use of the strategy at least 80% of the time throughout the entire observation). ICCs for individual items ranged from 0.61 to 0.94, with an average ICC of 0.80, and are considered good to excellent according to current standards (Cicchetti, 1994). Mean scores for each of the seven composites and one overall score were calculated; parent post-intervention scores were utilized as predictor of child outcomes in analyses.
Dependent measures: child
Measures examining social communication in toddlers based on independent evaluation and parent report were chosen based on the skills targeted in the intervention.
Communication and Symbolic Behavior Scales–Infant Toddler Checklist
The Communication and Symbolic Behavior Scales–Infant Toddler Checklist (CSBS–ITC; Wetherby & Prizant, 2002) was completed at pre-, post-, and 3-month follow-up of interventions. The CSBS–ITC is a parent questionnaire used to determine risk for developmental disorders, including ASD. The CSBS–ITC includes 24 questions reflecting four composites with three to five choices about developmental milestones of social communication. It is a standardized tool with screening cutoffs and standard scores for children 6–24 months based on a normative sample of 2188 children (Wetherby & Prizant, 2002). It has good reliability and validity as well as high sensitivity and specificity (both 88.9%) for catching toddlers who are later diagnosed with ASD and other developmental delays from a general pediatric sample. Summative scores for the Social, Speech, and Symbolic domains as well as an overall score were calculated for each time point and utilized in analyses.
MacArthur–Bates Communicative Development Inventory
Parents completed the MacArthur–Bates Communicative Development Inventory (CDI; Fenson et al., 2006) at pre-, post-, and follow-up time points. The CDI assesses major features of communicative development, including vocabulary, understanding and use of gesture, and emergence and expansion of grammatical features. The CDI/words and gestures form is a vocabulary checklist structured so the parent can indicate what words the child understands and what words the child understands and says. This form also assesses what phrases the child understands and what play and other imitation skills she or he exhibits. Raw scores for the Words Understood, Words Produced, Early Gestures, and Late Gestures composites were calculated for each time point and utilized in analyses.
Vineland Adaptive Behavior Scales, Second Edition (VABS-II)
This assessment, completed at pre-, post-, and follow-up time points, measures personal and social skills (Sparrow, Cicchetti, & Balla, 2005). It has been validated with children with developmental disabilities and is applicable to children from birth through 18 years, 11 months. Standardization included national samples of children with and without disabilities. The scales yield normative standard scores (M = 100; SD = 15) that indicate level of functioning and can be used for comparison across groups. Standard scores for the communication and socialization domains were utilized in analyses.
Data analysis
Data analyses were conducted in Statistical Package for the Social Sciences (SPSS v 25). Initially, associations between study condition and child and parent demographic and clinical characteristics at baseline were examined to address observed pre-treatment group differences. To compare differences in outcomes by study condition (Aim 1), we conducted linear regressions for end of treatment outcomes with intervention group as the primary predictor. Covariates included pre-treatment scores (for repeated measures) and a variable representing the number of children enrolled for each provider to address issues of nested data (with one or more child nested within provider). The inclusion of this later covariate is recommended to address issues of nesting when sample sizes limit the ability to include these as separate levels (McNeish & Wentzel, 2017). For analyses comparing mean differences in group regardless of time point (e.g. examining group differences in the PPEM), we utilized independent samples’ t-tests. Aim 1 analyses specifically focused intervention effects, whereas Aim 2 examined impact of providers’ use of coaching strategies consistent with PI T on parent outcomes, namely parent fidelity and parent–child interaction (PICCOLO scores). Finally, due to smaller sample size and resulting impact on statistical power, at follow-up (PI T n = 10; UC n = 9), effect sizes were calculated based on changes from pre-intervention to 3-month follow-up for each group (Aim 3). Given the pilot nature of this work and limited statistical power, significant results (p < 0.05), marginal trends (p < 0.10), and non-significant effects with moderate to large effect sizes are discussed.
Results
Aim 1 intervention effects: parent outcomes
See Table 4 for a presentation of all parent outcome results.
Aim 1 intervention effects at baseline (pre) to end of treatment (post): parent outcomes.
PIT: Project ImPACT for Toddlers; UC: usual care; ES: effect size; SSI: Social Support Index; PPEM: Parent Participatory Engagement Measure; PICCOLO: Parent Interactions with Children: Checklist of Observations Linked to Outcomes.
Supportive parenting behaviors
PICCOLO scores indicated a significant effect of condition for all four parent domains: affection (β = 0.47, t(19) = 2.28, p = 0.03), responsiveness (β = 0.84, t(19) = 6.46, p < 0.01), encouragement (β = 0.59, t(19) = 2.93, p = 0.01), and teaching (β = 0.45, t(19) = 2.10, p = 0.05), with larger increases in positive parent behaviors for the PI T group. Large effect sizes were observed for the PI T group across the four domains (Cohen’s d range = 0.83–1.88) compared with the small effects observed for the UC group (Cohen’s d range = –0.66–0.16).
Parent PI T intervention fidelity
Analyses indicate a marginally significant effect of condition on caregiver use of the prompt strategy (β = 0.44, t(19) = 2.09, p = 0.05), with caregivers in the PI T group demonstrating higher fidelity in use of prompt strategies compared with UC group. A large effect size was observed for the prompt strategy (Cohen’s d = 1.19) for the PI T group compared with a small effect size (Cohen’s d = 0.35) for the UC group. Similarly, caregivers in the PI T group demonstrated higher, albeit non-significant (β = 0.21, t(19) = 0.69, p = 0.37), fidelity using the wait strategy compared with those in the UC group, with a large effect size (Cohen’s d = 0.77) for PI T compared the moderate effect for UC (Cohen’s d = 0.36). Results were non-significant and had similar effect sizes across groups for all other PI T strategies (t < 1.52, p > 0.15).
Parent intervention engagement and social support
Independent samples’ t-test analyses indicated significant differences in overall average parent participatory engagement in the intervention (PPEM average across time points; t(21) = –2.66, p < 0.02), with higher overall engagement among families receiving PI T (M = 4.08, SD = 0.66) compared with UC (M = 3.02, SD = 1.12). Linear regressions analyses indicated no significant differences from intake to end of treatment in perceived social support by condition (t(16) = –0.62, p = 0.55).
Aim 1 intervention effects: child outcomes
See Table 5 for full child outcomes results.
Aim 1 intervention effects at baseline (pre), end of treatment (post), and follow-up (F): child outcomes
PIT: Project ImPACT for Toddlers; UC: usual care; ES: effect size; CSBS: Communication and Symbolic Behavior Scales; CDI: Communicative Development Inventory; Vineland: Vineland Adaptive Behavior Scales, Second Edition; SS: standard score; ABC = Adaptive Behavior Composite.
Child communication and social communication
Analyses revealed a marginal trend for a significant impact of condition on CDI words produced (β = –0.18, t(16) = –1.76, p = 0.09), with small effect sizes for both conditions (UC Cohen’s d = 0.35; PI T Cohen’s d = 0.19). A marginal trend was also observed for CSBS symbolic composite (β = 0.20, t(18) = 1.90, p = 0.07), with a large effect size observed for the PI T condition (Cohen’s d = 1.47) compared with a small effect size for the UC condition (Cohen’s d = 0.25). Although further analyses did not reveal any statistically significant condition effects, analyses revealed medium to large effect sizes for the PI T children compared with the small effect sizes for the UC children on several measures, including CSBS social composite (t(18) = 1.11, p = 0.28; PI T Cohen’s d = 0.55, UC Cohen’s d = 0.05), CSBS total score (t(18) = 1.08, p = 0.29; PI T Cohen’s d = 0.91, UC Cohen’s d = 0.24), CDI words understood (t(16) = –0.71, p = 0.49; PI T Cohen’s d = 0.48, UC Cohen’s d = 0.03), CDI early gestures (t(16) = –0.46, p = 0.66; PI T Cohen’s d = 0.54, UC Cohen’s d = 0.07), and CDI late gestures (t(16) = 0.76, p = 0.46; PI T Cohen’s d = 0.66, UC Cohen’s d = –0.02). No group differences were observed for CSBS speech composite (t(18) = –0.45, p = 0.66).
Child adaptive skills
Analyses revealed no significant condition effect for the communication (t(14) = 1.42, p = 0.18) or socialization (t(14) = 0.07, p = 0.95). Composites on the Vineland from pre- to post-intervention. Effect sizes were small and similar across groups.
Aim 2: associations between providers’ use of PIT coaching strategies and parent fidelity
Providers who delivered PI T to families had significantly higher scores on the Use of PI T Parent Coaching Strategies measure than UC providers (PI T M = 3.68, SD = 0.66; UC M = 2.17, SD = 1.01; F (1,87) = 72.77, p < 0.01), indicating a difference between groups in the structure and approach of services delivered to families.
We examined whether providers use of PI T parent coaching strategies predicted overall parent fidelity and/or positive parent–child interaction. Controlling for the number of children enrolled for each provider, analyses indicated that providers’ use of parent coaching strategies was significantly associated with parent fidelity (β = 0.47, t(20) = 2.08, p = 0.05) as well as higher parental teaching behaviors as measured by the PICCOLO (β = 0.49, t(20) = 13, p < 0.05). In addition, there was a marginally significant association between parent fidelity and higher responsiveness behaviors from parents as measured by the PICCOLO (β = 0.40, t(20) = 1.70, p = 0.10). Providers’ use of coaching strategies was not significantly associated with parental affection (t(20) = 0.88, p = 0.39) and encouragement (t(20) = 0.39, p = 0.70), and effect sizes were similar across groups.
Aim 3: exploratory analyses examining intervention effects at follow-up
Parent outcomes at follow-up
See Table 6 for Parent follow-up results. As mentioned, effect sizes were examined based on the small sample sizes and limited statistical power.
Aim 3 intervention effects from baseline (pre) to follow-up (F): parent outcomes.
PIT: Project ImPACT for Toddlers; UC: usual care; PICCOLO: Parent Interactions with Children: Checklist of Observations Linked to Outcomes; F: follow-up; ES: effect size.
Supportive parenting behaviors
Moderate to large effect sizes were observed for the PI T group across the four domains of the PICCOLO (Cohen’s d range = 0.53–1.44) compared with the small effects observed for the UC group (Cohen’s d range = –1.07–0.43), indicating larger changes in affection, responsiveness, encouragement, and teaching parenting behaviors from pre-intervention to 3-month follow-up for parents who received PI T . Differences between the groups were not statistically significant.
Child outcomes at follow-up
See Table 5 for full child outcomes results.
Child communication and social communication
Analyses revealed large effect sizes for the PI T children compared with the small effect sizes for the UC for several measures, including the CSBS social composite (β = 0.17, t(15) = 1.20, p = 0.25; PI T Cohen’s d = 1.19, UC Cohen’s d = 0.15), CSBS symbolic composite (β = 0.13, t(15) = 1.10, p = 0.29; PI T Cohen’s d = 1.71, UC Cohen’s d = 0.45), CSBS total composite (β = 0.08, t(15) = 0.85, p = 0.41, PI T Cohen’s d = 1.40, UC Cohen’s d = 0.41), CDI words understood (β = –0.02, t(13) = –0.10, p = 0.92; PI T Cohen’s d = 1.08, UC Cohen’s d = 0.11), CDI early gestures (β = 0.16, t(13) = 0.96, p = 0.35; PI T Cohen’s d = 1.07, UC Cohen’s d = < –0.01), and CDI late gestures (β = 0.05, t(13) = 0.34, p = 0.74; PI T Cohen’s d = 0.95, UC Cohen’s d = 0.22) from pre-intervention to 3-month follow-up.
Child adaptive skills
Analyses revealed a medium effect sizes on the Vineland Communication Scale for the PI T children (Cohen’s d = 0.49) compared with the small effect sizes for the UC children (Cohen’s d = < –0.01; β = 0.14, t(14) = 1.01, p = 0.33).
Discussion
This study represents one of the first demonstrations of effectiveness for an adapted evidence-based, naturalistic developmental behavioral intervention for toddlers in community early intervention programs. These promising results provide preliminary evidence for the potential benefit of systematically fitting evidence-based interventions into service and community contexts through research–community partnership. Because the intervention was adapted with local providers and funders, implementation of the strategies with fidelity was more likely (see Rieth et al., 2018 for further details regarding training and fidelity) and the sustainment of strategies over time provided the opportunity for this quasi-experimental examination of child and family outcomes of the adapted intervention.
Our data indicate that this intervention shows promise for improving social communication outcomes for young children with early signs of ASD, even if a diagnosis has not yet been made. Although the design was quasi-experimental, the groups were generally equivalent at intake to the intervention. Encouragingly, though all children made progress in early intervention and results were not statistically significant in many areas, there were some promising differences favoring the PI T group. For example, children receiving PI T had strong effect sizes related to progress in symbolic communication and play on a standardized behavioral measure. Parents in that group also reported greater effect sizes for words understood and gesture use after only 3 months of a low-intensity intervention. As may be potentially expected to happen with parent-implemented interventions, even greater differences in effect sizes were evident between the groups after the follow-up period. The reason for the increased differences is posited to be the changes in parents’ use of strategies after completion of the training, thereby increasing intervention intensity and duration for the child.
Parents in the PI T group made greater changes in the way they interacted with their children after receiving intervention. This is important because parents of children with social communication challenges and/or ASD, although just as responsive as parents of typically developing children (Siller & Sigman, 2002), have fewer opportunities to respond because of differences in the way their children learn from their environment (Leezenbaum, Campbell, Butler, & Iverson, 2014). Results are similar to outcomes in community trials of a developmental, relationship-based parent-mediated intervention with older children that has many similar principles to PI T (e.g. Solomon, Van Egeren, Mahoney, Quon Huber, & Zimmerman, 2014). Parent sensitivity and responsivity predicts language outcomes in typical development as well as ASD (Siller & Sigman, 2002). PI T coaching from providers resulted in increased responsivity of parents in all areas, including affection, response to child’s cues, encouragement, and teaching opportunities. These changes may subsequently support child development and have sustained, long-term impacts on child outcomes (Pickles et al., 2016).
Interestingly, although having a provider trained in PI T led to strong differences in more distal parent–child interaction, providers were not effective at increasing parents’ use of specific intervention skills measured on the PI T fidelity rating scale. This was the case even though use of the parent coaching strategies by the provider was associated with overall parent PI T fidelity. This may be related to the fidelity measurement tool itself. Fidelity to an intervention is a complex, multi-dimensional construct that may encompass several areas, including: (1) content—the steps delivered; (2) competence or quality—the skill and judgment used during delivery; (3) quantity—how much of the intervention was delivered; (4) adherence—the degree to which prescribed and not proscribed procedures are utilized; and (5) differentiation−features unique to the intervention (Schoenwald et al., 2011). The PI T fidelity rating scale may be good at capturing content and adherence but less accurate when examining quality and clinical appropriateness of skills use. The relationship between coaching strategies and parent use of the strategies overall may be based on increases in specific skills (e.g. prompting) and not consistent changes in all of the strategies in the intervention. Alternatively, parents may be learning about responsivity through general coaching and the specific skills may or may not be the mechanism of action for the intervention. This is an important limitation for community research seeking to understand the key ingredients of intervention. Ideally, we would see the differences between the groups in their use of the specific strategies taught in the intervention over time (in addition to impacts on more distal measures). More research with larger samples examining the link between coaching in specific strategies and changes in parent behavior over time will help reveal if these types of fidelity measures are capturing what we intend and which strategies truly lead to change in parent and child behavior.
Importantly, this community sample included a diverse group of families who represent typical consumers of early intervention in an urban area. A total of 50% of families receiving PI T were Hispanic/Latinx, and they represented a range of maternal education levels. Although the purpose and design of this study were not to examine differential impact of an intervention across racial and ethnic groups, the large effect sizes for PI T within this representative sample are encouraging as to the cultural appropriateness of the practice. The adaptation process for PI T included many members of the Latinx community, both during broad community feedback phases and as members of the BRIDGE Collaborative. The impact of the intervention in the current trial with a diverse sample is evidence of the utility of involving stakeholders in the early phases of intervention development.
Results should be considered preliminary given several limitations. First, children were not randomly assigned to groups by the research team, but rather were assigned to providers by caseworkers at the local Part C funder as publicly provided service. While the examination of PI T within the context of community care is strength of the current study, this lack of random assignment is a limitation, as it led to apparent, albeit not statistically significant, differences in ASD symptom severity in the PI T group compared with the UC groups. This difference may limit the conclusions that can be drawn from comparison between the two groups of children. In addition, statistical power was limited in all analyses due to the small sample size and pilot nature of the project. Limited power lead us to compare effect sizes across groups as an indicator of the intervention’s impact, but the larger effect sizes on some measures for the PI T group are not equivalent to statistically significant differences in more rigorous analyses. Future work should examine the impact of PI T with a larger group of children in a rigorous experimental design in order to better understand any differential impact of the PI T intervention.
This project was made possible through the development of a research–community partnership that adapted a specific EBP to fit the community context. Community agencies who had received training in PI T through previous interaction with the research team continued to use PI T and to receive funding through Part C to provide this intervention in the community. Participating parents and children obtained the service through public agencies vendored with Part C. Greater change was seen in both parent and child behavior when services were delivered from a PI T trained provider. This pilot project supports the use of research–community partnerships to increase access to care and to support broader research in child and family outcomes for this population. Next steps will include a better understanding of how to measure fidelity of the intervention and conducting a larger, controlled trial of PI T effectiveness in the community.
Footnotes
Acknowledgements
The authors would like to thank the BRIDGE Collaborative for their tireless and rigorous work toward a better understanding of community-based research, and the community providers, children, and families who participated in this project.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a US Department of Education Grant: R324A140004 and an Autism Speaks Grant: 8136, and received infrastructure support through the MIND Institute IDDRC funded by the National Institute of Child Health and Human Development (U54 HD079125).
