Abstract
Children with autism spectrum disorder (ASD) often experience skill deficits that can negatively affect long-term outcomes. Interventions based on applied behavior analysis (ABA) yield improvements in targeted skills. However, families often have difficulty accessing ABA services. The purpose of this study was to evaluate the efficacy of a caregiver coaching program delivered via telehealth. Thirty children with ASD and their caregivers (e.g., parents, grandparents) participated in all phases of the study. The program consisted of therapists providing coaching in English or Spanish to caregivers of children with ASD via synchronous video call telehealth visits, typically provided one to two times per week. Caregivers received coaching in interventions (e.g., functional communication training, discrete trial teaching, total task chaining, and naturalistic teaching) to address individualized goals. We collected data on caregiver treatment fidelity and child outcomes (i.e., Vineland-3, observation, and analysis of time series data). Caregivers implemented intervention procedures with 95% accuracy on average. The single-case effect sizes calculated based on the time series baseline and intervention data yielded medium, large, or very large improvements for 85% of goals addressed. Results indicated that the children improved on appropriate engagement (measured via observation), but there was no statistically significant improvement for the remaining pre-post measures. These results, along with the results of previous studies, provide preliminary support for the use of telehealth to provide ABA services. However, there is a need for additional research evaluating the efficacy of these types of programs.
Individuals with autism spectrum disorder (ASD) often experience difficulties with daily living skills, communication, social skills, and challenging behavior (American Psychiatric Association, 2013; Centers for Disease Control and Prevention, 2021). These difficulties can affect long-term outcomes, such as obtaining a job and living independently (U.S. Department of Health and Human Services, 2017; Roux et al., 2017; Siperstein et al., 2013). For this reason, it is critically important to ensure individuals with ASD have adequate access to evidence-based practices. Many interventions based on applied behavior analysis (ABA) have sufficient research support to be considered evidence-based in improving outcomes for individuals with ASD; nearly all of the currently identified evidence-based practices are ABA interventions (ABA; Steinbrenner et al., 2020; Wong et al., 2015).
ABA interventions involve the therapist arranging the environment to increase a learner’s appropriate behavior and decrease inappropriate behavior (Cooper et al., 2020). Interventions based on ABA have an extensive body of literature supporting their use and ABA is recognized as a billable therapy (Hagopian et al., 2020). ABA-based interventions have been used to improve communication skills (Brown et al., 2000; Fisher et al., 1998; Jones et al., 2007; Kelley et al., 2007), academic skills (Jahr, 2001), and daily living skills (Craig et al., 2021). For example, at least 140 studies have evaluated the use of prompting, an ABA intervention, to support the acquisition of new skills (Steinbrenner et al., 2020). In addition, ABA-based interventions are effective in reducing challenging behavior, such as aggression, self-injury, and property destruction (Gerow et al., 2018). Together, the existing research indicates ABA interventions are highly effective when implemented correctly and consistently. However, the implementation of ABA interventions requires a significant amount of time from a trained implementer.
In order to improve children’s access to ABA interventions and to support caregivers of individuals with ASD, practitioners and researchers often involve caregivers in the delivery of interventions. Caregivers, such as parents or grandparents, and their children often benefit from caregiver implementation of interventions. The effects of caregiver-implemented interventions are often greater and more durable than those implemented by professionals (Bradshaw et al., 2017; Matson et al., 2012). Caregiver-implemented interventions result in reductions in challenging behavior (Gerow, Radhakrishnan, Davis, et al., 2021), improved imitation skills (Wainer & Ingersoll, 2015), and improved daily living skills (Boutain et al., 2020). In addition, when caregivers are trained as interventionists, they gain the skills and confidence to address their child’s individual needs and often experience reduced stress (Patterson et al., 2011; Postorino et al., 2017). Therefore, it is critically important to include caregivers in the delivery of ABA interventions for children with ASD.
Caregivers are often taught to implement interventions through one-on-one coaching; coaching is a process in which the coach works collaboratively with the caregiver to teach an effective practice to improve an outcome for their child (Meadan et al., 2017; Rush & Shelden, 2011). Coaching typically consists of repeated one-on-one consultation with instructions and opportunities to practice with support and feedback (National Association for the Education of Young Children & National Association of Child Care Resource and Referral Agencies [NAEYC & NACCRRA], 2011). This type of coaching can be delivered in-person or via telehealth. Telehealth consists of utilizing telecommunication technologies to distribute services (Institute of Medicine [IOM], 2012). There are a variety of methods for delivering services via telehealth, including live video call, asynchronous messaging, and communication via telephone (American Telemedicine Association, 2020). Therapists can deliver one-on-one coaching to caregivers via synchronous videoconference meetings, resulting in improved outcomes for children (e.g., Wacker et al., 2013). The use of telehealth services can reduce barriers to accessing research-supported interventions, such as the shortage of qualified professionals and the family’s geographic distance from professionals (Boisvert et al., 2010; Lindgren et al., 2016; Neely et al., 2017; Wacker et al., 2013). Telehealth services can lead to lowered costs, enhanced professional collaboration, and positive child outcomes (e.g., Machalicek et al., 2016; Tomlinson et al., 2018; Tsami et al., 2019). For example, Lindgren et al. (2016) demonstrated telehealth delivered services resulted in outcomes and consumer satisfaction levels similar to in-person service delivery models.
Despite the growing literature base, few studies have evaluated the efficacy of telehealth coaching in multi-component interventions for caregivers of children with ASD. Many studies have evaluated the efficacy of telehealth caregiver coaching in one intervention to improve a specific skill or to decrease challenging behavior (e.g., Craig et al., 2021; Machalicek et al., 2016; Tsami et al., 2019; Wacker et al., 2013). For example, Boutain et al. (2020) evaluated the efficacy of caregiver coaching via telehealth to improve daily living skills, such as face washing. Caregivers accurately implemented the intervention—a graduated guidance procedure—and the children’s independence in completing daily living skills improved. This body of work has demonstrated that telehealth coaching is associated with caregivers’ accurate implementation of an intervention and results in improvements in children’s performance on a specific goal (e.g., increased communication, decreased challenging behavior). Children with ASD often experience difficulties across multiple skills and, as a result, benefit from comprehensive intervention programs that target multiple skill domains (Ballaban-Gil et al., 1996; Cohen et al., 2006; Howard et al, 2014; Lovaas, 1987; Smith et al., 2000; Steege et al., 2007). The success of comprehensive programs delivered by a professional suggests the possibility of similar results from caregiver-delivered interventions to address multiple skills. Moreover, caregiver delivery of interventions may yield additional benefits such as increased access to the intervention during the day, across settings, and over time. For this reason, it would be beneficial to evaluate the efficacy of telehealth coaching to teach caregivers multiple interventions across multiple child goals.
To our knowledge, few studies have evaluated a caregiver coaching program delivered via telehealth to address multiple child outcomes. Bearss et al. (2018) evaluated the efficacy of a 24-week telehealth caregiver coaching program in which caregivers of 14 children received coaching at a remote telemedicine site (i.e., clinic-to-clinic). For most of the participants, the caregiver reported the child improved. However, the study did not include a direct measure of child progress and caregivers were trained at a clinic site rather than their homes, which may not be feasible for many families. Ura et al. (2021) evaluated a 12-week caregiver coaching intervention to improve children’s social communication. Caregivers received coaching in their homes on a social communication intervention and the intervention yielded improvements in social communication and other skills. This study involved teaching naturalistic instruction to caregivers, who received coaching in the home. Based on this body of work, there is a need to evaluate the efficacy of coaching caregivers via telehealth in multiple interventions to address multiple child goals. The purpose of the present study was to evaluate the efficacy of a telehealth caregiver coaching program, consisting of teaching caregivers interventions to address multiple child goals, on improvements in skill acquisition and reductions in challenging behavior for children with ASD.
Method
Participants
We recruited families for this project through community providers (e.g., schools, clinics, medical offices) and other organizations (e.g., parent support groups) using printed flyers, email, phone calls, and social media posts in English and Spanish. The inclusion criteria were (a) the family lived in a southwest state in the United States, (b) the child had a medical or educational diagnosis of ASD as reported by the parent, (d) the child was younger than 18 years old, and (d) at least one parent or caregiver was willing to participate as the implementer. Children who engaged in challenging behavior too severe to treat via telehealth were excluded; we recommended the family seek in-person services in these cases. By using power analysis with a medium effect size, we estimated 30 participants would result in adequate statistical power to find effects of the treatment (Faul et al., 2007); we recruited participants until 30 participants completed the study. We obtained informed consent from the caregiver participants and the parent or guardian of the child participants prior to conducting study procedures.
A total of 30 children completed all phases of the project and 33 caregivers received coaching (see Table 1 for participant information). Some of these included participants also participated in published single-case studies (Gerow, Radhakrishnan, Akers, et al., 2021; Gerow, Radhakrishnan, Davis, et al., 2021; Davis et al., 2022). Parents or guardians of 24 child participants provided consent, but did not complete all phases of the project—10 withdrew before or during pre-test procedures, three withdrew during the program, and 11 chose not to participate in the post-test. For the 13 families who withdrew before or during the program, six reported they were too busy or scheduling was too difficult, two did not return contact from the research team, two said they were receiving other services and did not need to participate for that reason, one had a family emergency, and two did not give a specific reason. The remaining 11 families choose not to complete the post-test, which required approximately 2 hours. For the 30 included participants, all of the participants were receiving services outside of the program (e.g., ABA, occupational, physical, and/or speech therapy); six of the participants were receiving ABA services at the onset of the study.
Caregiver and Child Information.
Note. The race and ethnicity categories are based on the U.S. Census Bureau categories (U. S. Census Bureau, 2020). Participants had the option to select more than one category.
Therapists and Therapist Training
Graduate students conducting master’s or doctoral coursework in ABA provided coaching to families throughout the program. The therapists first participated in approximately 25 hours of asynchronous online modules about evidence-based practices, confidentiality, and research ethics and guidelines. Next, the therapists received training in the program procedures, including the use of telehealth, coaching, and specific intervention strategies, using a behavioral skills training (BST) model. These trainings were conducted in-person or via videoconference. The training lasted approximately 40 hours and was conducted across 2 weeks. All therapists were required to demonstrate accurate implementation of (a) telehealth caregiver coaching skills, (b) intervention procedures (e.g., total task chaining), and (c) data collection (with 90% interobserver agreement or above). During training, therapists also received instruction and feedback on engaging in positive, supportive, and collaborative interactions with caregivers. Once therapists demonstrated mastery of these skills during role-play (or recording data from sessions for the data collection competency), they were then required to demonstrate competency in providing caregiver coaching with a family (who was receiving ongoing coaching from a previously trained therapist) with support of a supervisor. After this step, therapists were eligible to serve as a therapist with a family. During their work with families, supervision was systematically reduced from 100% to a minimum of 10% of time with families, for therapists who did not have the Board Certified Behavior Analyst® (BCBA®) credential. An additional training procedure, similar to the procedure described above, was used to train therapists who conducted challenging behavior assessment and treatment. Each of these therapists received extensive training and supervision in the provision of challenging behavior treatment and was a BCBA or a graduate student with one or more years of experience implementing challenging behavior assessment and treatment.
Ongoing Supervision, Collaboration, and Oversight
To provide supervision, support, and oversight, the research team (therapists, supervisors, and data collectors) met weekly via videoconference or in-person. A BCBA-Doctoral® (BCBA-D®) led each meeting. During the meeting, each therapist provided an update related to each participating family, with the support of their supervisors. The therapists shared the child’s goals, presented graphs with updated data, and discussed any recent and relevant successes or difficulties. The meeting allowed supervisors to support each other and the therapists in making treatment decisions and addressing any difficult or unique situations. In addition, the oversight of the BCBA-D served to ensure that the procedures adhered to the project protocol. This portion of the meeting also ensured that all supervisors were aware of procedures for each of the current families, so they could provide appropriate supervision to any of the therapists during visits that week. Following the group meeting, the supervisors met without the therapists. During this meeting, the supervisors discussed topics relevant for only supervisors (e.g., specific feedback provided to a therapist the previous week) and matched supervisors to specific visits with families, based on their availability.
In addition to the weekly team meetings, supervisors conducted weekly one-on-one meetings and frequent supervision of visits with therapists who had not obtained the BCBA credential. In the one-on-one meetings, the supervisor and therapist discussed updates from the previous week (i.e., content of visits with family, data collected) and created the plan for the upcoming week. All treatment decisions, data collection decisions, and problem-solving was led and approved by the supervisor. Therapists often prepared data collection, treatment fidelity sheets, and parent handouts, based on the project protocol prior to the meeting. At the meeting, the supervisor described any needed changes to the materials or procedures and provided final approval on each of these items before they were implemented with the family. The supervisor and therapist also frequently role-played the upcoming discussions with the caregiver. The supervisor ensured the therapist used jargon-free, clear, and concise language and responded appropriately to questions and reactions. During the week, therapists led visits with families. Supervisors often attended these visits and provided additional support when requested by the therapist. Therapists interacted directly with families the most frequently due to leading the visits. However, the families were able to correspond with supervisors via a project phone or email at any time. Much of the communication with families outside of visits (e.g., an email response) was reviewed, edited, and approved by a supervisor.
Setting
All visits occurred in the homes of the families who participated. The caregiver selected the room(s) in which the visits were conducted, based on their preference and the availability of relevant materials. The therapists conducted visits via telehealth from a university campus or from the homes of the therapists (during the COVID-19 pandemic).
Materials
During the program, caregivers primarily used materials available in their homes, such as toys (e.g., blocks, trains, books) and materials for daily living skills goals (e.g., shoes). The therapists also developed and mailed materials to families, such as token economies and picture communication cards. Finally, the therapists used technology to provide telehealth coaching, described below.
Technology
The therapists used a laptop with a video camera and headphones (headphone use was optional) to conduct sessions. Families could choose to use their own technology or to have the research team mail the needed technology to them. For families who chose to use their own technology, we recommended that they select technology that could include the child and caregiver in the frame of the camera without the parent holding the device and that could move with the caregiver and child from room to room, if needed. Families who chose to have the technology mailed to them received a tablet computer (with a protective case and screen cover), Bluetooth® headset (optional for use during sessions), and a stand for the tablet. The tablet computer had the videoconference program on it and had cellular service, so that the family did not need to have their own internet to participate. We used the videoconference software VSee® (VSee, 2020) to conduct all sessions. We selected this software due to the security of the connection, the low bandwidth requirement, and the software’s features (e.g., sharing of documents through the application; VSee, 2013).
Measures
We collected time series data, two pre-post measures, and social validity data. We also collected data on the therapists’ coaching fidelity and the caregivers’ fidelity of implementation. All measures were conducted via telehealth, in English or Spanish, with the therapist or observer collecting information from the caregiver and child via the video call application.
Time series data
During the program, caregivers received coaching on baseline and intervention procedures related to specific child goals. Data were collected on the child’s goals during baseline and intervention sessions. Sessions typically lasted 5 minutes or for a set number of trials (often five trials). The data collection procedures remained consistent across baseline and intervention sessions. The child’s behavior was measured using frequency, percentage of opportunities with the correct response, or interval data collection, depending on the specific goal.
Vineland Adaptive Behavior Rating Scales, Third Edition
We administered the Vineland Adaptive Behavior Rating Scales, Third Edition (Vineland-3; Sparrow et al., 2016) via an interview with the caregiver. The interview consists of a series of open-ended conversation questions; the interviewer uses the caregiver’s response to record scores on items related to each of the domains. The Vineland-3 yields a raw score for each domain which is then converted into a norm-referenced score (i.e., the standard score). The domains evaluated included Communication, Daily Living, Socialization, and Maladaptive Behavior. The Maladaptive Behavior measure reported in this study consists of the Externalizing portion of the Maladaptive behavior domain. The Maladaptive Behavior domain is not administered for children younger than 3 years old. For this reason, three included participants did not have Maladaptive Behavior scores. The Vineland-3 has adequate reliability and validity (Sparrow et al., 2016).
Observation
We conducted a researcher-developed observation. The observation lasted 10 minutes and the caregiver was instructed to play with their child and to encourage the child to communicate and play appropriately. The caregiver chose the play material(s). The caregiver was not given any additional instructions for interacting with the child. The observer collected data on three child behaviors (a) engagement, (b) communication, and (c) challenging behavior. We defined engagement as interacting appropriately with adults or playing with toys/leisure materials in the manner intended or using imaginative play (e.g., pushing cars, dressing doll, painting, reading a book, playing peek-a-boo). Communication consisted of prelinguistic behaviors (defined as indicating interest, wants, or needs without using words; e.g., pointing, leading) or communication using words (e.g., vocal speech, picture exchange, speech generating device, sign language). Challenging behavior consisted of behavior that was inappropriate for the context (e.g., property destruction, crying, screaming, throwing toys, self-injury), repetitive behavior (defined as motor or vocal behavior that occurred more than three times in a row without a pause; e.g., rocking), or behavior that was restrictive in nature (e.g., insisting on playing a certain way).
The observer recorded data on each measure using 10-second partial interval recording (i.e., the percentage of 10-s intervals during which the behavior occurred). Therefore, the observation yielded scores related to engagement, communication, and challenging behavior, ranging from 0-100. All observers received training on the definitions for each of the categories and continued with training until they achieved 80% agreement with the trainer for two consecutive observations. The observers were the therapist, a supervising BCBA/BCBA-D, or another graduate research assistant involved in the project. We recorded interobserver agreement (IOA) data for 100% of observations collected. Average IOA was 97% (range, 79%–100%) for engagement, 90% (range, 52%–100%) for communication, and 95% (range, 68%–100%) challenging behavior.
Social validity
A researcher adapted social validity instrument was used to evaluate the caregivers’ report of the acceptability and effectiveness of the program. The instrument included six items rated on a 4-point Likert-type scale (with 1 indicating “strongly disagree” and 4 indicating “strongly agree”) and three open-ended questions.
Caregiver fidelity
During baseline and intervention sessions, an observer (typically the therapist) recorded the caregiver’s fidelity of implementation of the designated procedures on a researcher-developed checklist. We developed checklists for baseline and intervention procedures for each goal based on the standard protocol. The checklist consisted of the steps for implementing the procedure. The therapist and supervisor individualized the checklist, as needed, based on any modifications for the family (e.g., specific prompting procedure, specific type of reinforcement, number of tokens in a token economy).
During baseline sessions, accurate implementation typically involved the caregiver providing an instruction, not prompting the target skill, and not providing differential reinforcement for correct responding. Therefore, high implementation fidelity for the baseline procedure meant that the parent was not implementing the core features of the intervention, but instead implemented the baseline procedures the therapist asked them to conduct. During intervention, accurate implementation involved adhering to the intervention procedure (see Table 2).
Goal Domains, Intervention, and Example Target Skill with Corresponding Individualized Intervention Steps.
To collect data on caregiver implementation fidelity, the observer recorded the caregiver as correct or incorrect on each step. If the caregiver implemented the step correctly during the entire session, they were counted as correct on that step. If they implemented the step incorrectly for part or all of the session, then they were counted as incorrect. The observer then calculated the percentage of steps implemented correctly for the session. We then calculated the average implementation fidelity for baseline sessions and intervention sessions for each goal for each child.
Coaching fidelity
An observer (often a supervising BCBA) collected data on the therapists’ coaching fidelity during 36% of the visits. The observer recorded the extent to which the therapist implemented the coaching steps correctly, based on a researcher developed checklist. The coaching fidelity checklist consisted of the following steps: (a) introduction to the visit, (b) coaching for each goal, and (c) concluding the visit. The steps for coaching for each goal consisted of: (a) providing written instructions (shared through screen sharing or message in the video call application) and verbal instructions, (b) providing a verbal prompt if the caregiver implemented a step incorrectly or did not initiate the next step within 5 seconds, and (c) providing feedback following the session. Feedback consisted of positive statements for steps implemented correctly and during intervention consisted of an explanation of the correct procedures for any steps implemented incorrectly. The therapist was counted as correct or incorrect on each step. We then calculated the percentage of steps conducted correctly during the visit. We calculated the average coaching fidelity across visits for each family.
Procedures
All procedures were conducted via telehealth, during which the therapist provided instruction to the caregiver via synchronous video and audio communication. Typically, therapists planned to meet with families twice per week for 1.5-hour visits. The program was developed to last approximately 2 months, but the length of program varied based on caregiver preference, the child’s needs, and visit cancellations.
The research team included therapists who spoke English and Spanish. For caregivers who spoke one of these two languages (English or Spanish), we communicated with the caregiver in the language they spoke, including verbal communication and written materials. For caregivers who spoke both English and Spanish, we communicated with the caregiver in the language they preferred.
Parent interview and pre-test
The first visits included the researcher-developed parent interview, Vineland-3, and observation. The researcher-developed parent interview consisted of demographic questions and questions about the child’s preferences (e.g., favorite toys).
Functional analysis
For children with challenging behavior as one of the targeted goals, we conducted a functional analysis to determine the function of the challenging behavior and develop a function-based intervention. Prior to conducting the functional analysis, the therapist completed a safety interview. During the interview, the therapist discussed the child’s challenging behavior with the caregiver and recorded information related to the topography and severity of the challenging behavior. The therapist used the responses to develop any needed safety procedures (e.g., blocking). In addition, the information was used to determine whether assessment and treatment of the challenging behavior over telehealth was appropriate. For children with challenging behavior that was too severe (e.g., had caused injury with only a few instances of the behavior in the past), we recommended the family seek in-person services (see Gerow, Radhakrishnan, Davis, et al., 2021 for additional details).
All functional analyses were conducted over telehealth with caregivers serving as the implementer. The functional analysis methodology used varied by family and consisted of a brief functional analysis with additional assessments as needed (see Gerow, Radhakrishnan, Davis, et al., 2021), trial-based functional analysis (see Davis et al., 2022), or a pairwise functional analysis (with a test and control condition for a child with an idiosyncratic function). Functional analysis methodologies varied due to the child and family and due to specific single-case studies, but the functional analysis often included escape, tangible, attention, and play conditions. For participants in which an automatic function was suggested, the functional analysis included an ignore condition as well.
Program
Following the parent interview and pre-test, we developed individualized goals using a collaborative process with the family and therapist (Behavior Analyst Certification Board [BACB], 2020). First, the therapist and supervisor developed a list of goals based on the Vineland-3 and observation. The therapist used the parent interview and observation to inform and develop specific goals, often related to items on the Vineland-3 from domains with the lowest scores. For example, if a 2- or 3-year-old child scored low in the expressive communication sub-domain, the therapist might identify communicating wants and needs (a topic on the Vineland-3) as a potential goal. The therapist would then use information from the parent interview (e.g., parent priorities, parent preferences, and report about what the child does) and from the observation (e.g., the child did not produce any requests using vocal words, but did make sounds) to develop a specific goal to teach the child one-word requests. The therapist repeated this process until they created a list of goals (often 7–10 goals) and then presented the goals to the caregiver. The caregiver then selected their highest priority goals from the list, or the caregiver suggested different goals. For older children who could communicate preferences, we encouraged input and discussion with the child to select the goals. Each child typically had 3–5 goals and goals that addressed daily living skills (e.g., brushing teeth), socialization (e.g., play skills), communication (e.g., requesting toys), or challenging behavior (e.g., yelling, hitting). The procedures in the study—caregiver implementation and telehealth delivery—influenced the goal selection process in several ways. First, we suggested goals and intervention strategies that were natural or typical for the home environment. Second, we ensured that caregivers were comfortable practicing the goal while on video recording. Third, we used the safety form to confirm that the caregiver could manage challenging behavior, as applicable, without the support of a professional in the home.
Following the selection of goals, the caregiver conducted baseline and intervention procedures for each goal, with coaching from the therapist. Coaching consisted of BST model, including written and verbal instruction, rehearsal (i.e., implementing the procedures with the child) with verbal instruction during the rehearsal, and feedback following rehearsal. Previous research included role-play in BST delivered via telehealth (e.g., Boutain et al., 2020), but the role-play required four adults to attend the telehealth sessions. For this reason, we chose not to include role-play in our approach. During each visit, the therapist typically planned to work on each of the child’s goals. For each goal, the caregiver implemented baseline (i.e., no intervention) procedures prior to implementing the intervention procedures.
We developed a standard protocol for identifying and addressing goals (see Table 2). The protocol was developed by three BCBA-Ds and two BCBAs with experience in assessment and implementing research-supported practices for children with ASD. The protocol included intervention strategies for addressing goals in the following domains: daily living skills, communication, social skills, and challenging behavior. All of the included intervention strategies were behavior analytic evidence-based practices for children with ASD. For communication and social skills, the intervention consisted of naturalistic teaching, including prompting and reinforcement. For daily living skills, the intervention consisted of total task chaining and reinforcement. For some daily living skills and communication goals, the protocol consisted of discrete trial teaching (DTT), often with a token economy. This was the case for two of the 32 daily living skills goals (e.g., identifying coins) and 23 of the 32 communication goals (e.g., identifying letters, echoing a verbal model, responding to questions). For challenging behavior, the intervention consisted of functional communication training (for socially maintained behavior) or differential reinforcement of an alternative behavior with a competing stimulus (for automatically maintained behavior). As described above, a functional analysis was conducted to identify the function of challenging behavior.
Many of the intervention procedures involved the use of reinforcement. For all reinforcement procedures, we selected a reinforcer that was likely to be effective in increasing the child’s responses (based on parent report and/or a preference assessment). We also tried to select reinforcement that was a typical consequence for the response (e.g., a toy corresponding with a request, going outside following putting on shoes) to the extent possible and to the extent that the typical reinforcer was likely to be effective.
The intervention protocol allowed for individualization based on the child and family (e.g., type of prompt, specific reinforcer used). If children required modifications to the standard procedure, the BCBA/BCBA-D led the development of the modifications to the procedure. Modifications typically did not include changes to the core features of the intervention, but rather minor changes related to the child’s current skill level or parent preference (BACB, 2020).
Following the completion of the program, the therapist provided the caregiver with a written summary of the program (i.e., caregiver report), including graphs of child outcomes, a description of each of the interventions, and recommended next steps. This document also included information about how to find additional resources.
Post-Test
After the completion of the program, we administered the Vineland-3 and observation again.
Design and Data Analysis
Single-case effect sizes (Tau or Tau-U) were calculated to evaluate improvement from baseline to intervention, using the time series data for each child participant’s program goals. Tau is a nonparametric effect size that evaluates the extent to which intervention data points indicate improvement compared to baseline data points (Parker et al., 2011). Tau-U extends Tau by also correcting for baseline trend in cases in which a therapeutic baseline trend is observed (e.g., decrease in challenging behavior during the baseline phase; Parker et al., 2011). In this study, Tau-U was calculated for goals with a therapeutic baseline trend (baseline Tau of 0.4 or greater, as recommended by Parker et al. (2011)). We used an online calculator (singlecaseresearch.org) to calculate Tau and Tau-U. We only calculated effect sizes for goals with at least three data points in the baseline and intervention phases. Five goals were not included due to insufficient data. We used the following interpretations of the effect sizes, based on Vannest and Ninci (2015): less than 0.20 = small effect, 0.20–0.60 = medium effect, 0.61–0.80 = large effect, and greater than 0.80 = very large effect.
This study used a one group pre-test post-test quasi-experimental design to evaluate improvement on the pre-post measures (Mills & Gay, 2019). Data analysis was conducted using IBM® SPSS® (IBM Corp., 2020). We conducted a repeated measures analysis of variance (ANOVA) with the pre- and post-test measures to evaluate the effect of the caregiver coaching program on child outcomes. For the Vineland-3 Maladaptive behavior measure, we used a paired samples t-test because we only had data on this measure for 27 of the 30 participants included in the repeated measures ANOVA. For this reason, we removed this measure from the ANOVA to keep the integrity of the full sample and analyzed this measure individually.
Results
Single-Case Effect Sizes based on Time Series Data
For each goal addressed, we calculated effect sizes (see Table 3) and developed line graphs based on the time series data (see examples from included participants in Figure 1). Across the domains, the average effect sizes indicated medium to large effects. The mean effect size for communication goals was 0.58 (range, −0.80 to 1.00), indicating an overall medium effect. Daily living skills goals had a mean of 0.77 (range, −0.78 to 1.00), indicating a large effect. The mean effect size for socialization goals (e.g., social skills, play) was 0.61 (range, −0.56 to 1.00), which was an overall large effect. Finally, goals focused on challenging behavior had a mean of 0.68 (range, 0.04–1.00), indicating an overall large effect.
Program Goal Effect Sizes.
Note. Small = <0.2; medium = 0.20–0.60; large = 0.61–0.80; very large = >0.8 (Vannest & Ninci, 2015).
Example graphs for included participants corresponding to goals described in Table 2.
Vineland-3 and Observation Scores
Descriptive statistics for the Vineland-3 and observation and results of the analysis are presented Tables 4 and 5. The repeated measures ANOVA yielded a significant main effect for the observation engagement measure, F = 6.55 (1, 29), p = 0.016, d = 0.54. The mean engagement score for the observation significantly increased from pre-test (M = 73.82, SD = 29.50) to post-test (M = 87.28, SD = 19.04). There were no significant effects found for the other measures.
Descriptive Statistics and Repeated Measures ANOVA.
Paired Samples T-Test for Vineland Maladaptive Behavior Measure.
Social Validity Survey
Eleven participants responded to the social validity questionnaire. The average response was 3.8 on a scale of 1 to 4, with 4 being more positive (range 2.7–4). These results indicate that the caregivers who chose to respond to the survey rated the treatment as acceptable and effective overall. An example open-ended response was, “I like the one on one interaction we got. I love how patient and understanding they were.” There was only one caregiver with an average of a negative rating (2.7 out of 4); in the open-ended feedback the caregiver said, “The facilitators kept things simple. It was very helpful when they shared printables, task cards, etc.” and “I would like to see a study helping parents to determine, implement, and measure goals with intermittent support.” Overall, responses to open-ended questions on the social validity questionnaire were positive and affirming of the telehealth format and the program procedures.
Caregiver Fidelity and Coaching Fidelity
We collected data on the caregivers’ fidelity of implementation for 99% of baseline and intervention sessions. During baseline, the average implementation fidelity was 94% (range 50%–100%). During intervention, the average implementation fidelity was 95% (range 50%–100%). We also collected coaching fidelity data during 36% of visits. Across families, the average coaching fidelity was 99% (range 87.5%–100%).
Time in Program
We recorded the length of time each child participated in the program, including the time required to conduct the pre- and post-test. On average, families participated in the program for 21.12 weeks (range 5.57–45.57) and participated in 22.33 visits (range 6–53). The length of participation in the program varied due to cancellations, pauses in the programs related to families’ schedules, the need to evaluate additional intervention strategies, the number of goals that could be addressed in a single visit, and caregiver preference. On average, families cancelled 4.47 visits (range 0–19).
Discussion
The purpose of this study was to provide preliminary data regarding the efficacy of a caregiver-implemented ABA interventions delivered via telehealth. Thirty children with ASD and their caregivers participated in the study. The single-case effect sizes indicated that children typically performed better during intervention sessions compared to baseline. The data from pre-post measures indicated that the program yielded improvements in engagement, but there were no statistically significant improvements on the Vineland-3. Overall, the results provided some support for the efficacy of the program, although additional research in the area is needed.
The data from the single-case effect sizes indicated that children tended to improve from baseline to intervention during the program. These results are promising, given the short duration of the program and the delivery of coaching via telehealth technology. However, we did not see improvement for every goal. Interestingly, some domains were associated with larger improvements than others. For example, the average effect size for daily living skills was 0.77 whereas the average effect size for communication goals was 0.58. Importantly, the average indicated children typically improved for all of the domains. It may be the case that some domains are related to faster or larger improvement and future research should continue to evaluate whether children tend to make more progress in some domains than others. However, comparisons between each domain for this study should be interpreted with caution given that the extent to which each domain was addressed was individualized by child, based on the pre-test and caregiver preference. Similarly, there was a wide range of effect sizes within each domain, indicating that while there was large improvement for some children and some goals, there was no improvement on other goals. These differences may have been due to several factors, such as child factors (e.g., age), the difficulty of the goal, and caregivers’ implementation fidelity during and between visits. Future research should evaluate the reasons for the variability in improvements on goals.
The results from the pre-post measures indicated there was improvement in the observation measure of engagement, but there was not statistically significant improvement on any of the other observation measures or on the Vineland-3 measures. The improvements in engagement indicate that the program was correlated with increases in appropriate engagement. Due the quasi-experimental design of this study, these results should be considered preliminary and there is a need to replicate the results. There were not statistically significant improvements on the other measures from the observation (communication and challenging behavior) or in the Vineland-3 domains. However, the single-case effect sizes did show improvements in these domains. The small sample size may have affected the results. It may also be the case that targeting specific skills did not yield improvements in more global measures (e.g., improvement in shoe tying may not be related to large improvements on daily living skills on the Vineland-3). Similarly, it may be the case that the short duration of the program was not sufficient to yield improvements on these measures. Long-term implementation of the program and/or follow-up evaluation of the effects of the program may result in improvements on these types of measures. It is important to continue to investigate the extent to which the findings in this study indicate the type of program, the goals targeted, the program length, the small sample size, or other factors contributed to the lack of identified improvement on the observation and Vineland-3 measures.
We collected demographic information from participants in this study via an interview with the caregiver, including the race/ethnicity of the child (as reported by the caregiver, based on the U. S. Census Bureau (2020) options). In this study, most of the participants selected White as their child’s race/ethnicity (53%), approximately one fourth selected Hispanic (27%), 17% selected multiple races/ethnicities, and 3% selected Black or African American. However, in the state the study occurred, approximately half of parents of children with ASD selected Hispanic and 15% selected Black or African American as their child’s race/ethnicity (U.S. Department of Education, 2020). In previous special education research, a larger portion of the included participants identified as White than in the present study (75%; Robertson et al., 2017). These data indicate that while the race/ethnicity of the participant in this study was more similar to that of the population than previous research, the participants’ race/ethnicity did not fully reflect the diversity of the community. We anticipate that our community-based recruitment strategies (i.e., partnering with community agencies to recruit participants) improved the racial and ethnic diversity of the participants in the study. In addition, the study procedures, consisting of collaborating with caregivers to identify their high-priority goals and to individualize interventions, allowed for individualization based on family priority and preferences. The purpose of these procedures was to improve the fit of the program within each family’s culture, priorities, and daily routines. There is a need for continued work to ensure the participants in special education and applied behavior analytic research reflect the diversity of the community in which the research occurs. Given the lack of racial and ethnic diversity in research, researchers and practitioners need to continue to identify methods to promote equitable access to research and services and ensure participants are meaningfully included throughout the research process.
In the present study, the families of one third of the included children spoke Spanish. In the state in which the study occurred, approximately one fourth of the families of children with ASD speak Spanish (U.S. Department of Education, 2020). These data indicated a similar percentage of families spoke Spanish who participated in this study as the community in which the study occurred. In the present study, we created both English and Spanish recruitment materials (e.g., flyers, social media posts) and conducted recruitment through community partners. In addition, all coaching and program materials were available in English and Spanish and provided to the families based on their language preference. To conduct these procedures, the research team included team members who spoke both English and Spanish (the two most common languages in the state in which the study occurred). For four of the children who participated in the study, their caregivers spoke only Spanish. These families would not have been able to participate without coaching delivered in Spanish. Given the number of families who speak Spanish in the U.S. (U.S. Census Bureau, 2015), it is critically important to ensure there are a sufficient number of Spanish speaking professionals.
We collected data on therapists’ coaching fidelity and caregivers’ implementation fidelity. These data indicated that coaches used the coaching procedures accurately throughout the program and that the coaching procedures were associated with the caregivers’ correct implementation of the baseline and intervention procedures. Each of the therapists received extensive training and supervision on the intervention procedures, the coaching procedures (e.g., instructions, feedback), and the use of telehealth technology, as described in the method section. Therapists received ongoing supervision from a BCBA/BCBA-D through observation of visits and one-on-one meetings. A BCBA-D held weekly meetings with the team to provide support and supervision. As practitioners and researchers use similar models to provide support to families, it is important that they provide adequate training and supervision and continue to identify the critical features that are necessary to provide effective coaching of caregivers via telehealth.
The caregiver implementation fidelity data indicated that caregivers accurately implemented the interventions. The caregiver coaching procedures included written and verbal instructions, rehearsal (i.e., implementing the procedures with the child with ongoing support from the coach), and feedback following rehearsal. Although role-play is often included in a BST model, we chose not to implement this procedure due to the number of adults required. Caregivers often participated in the program without other adults present in their home or room. Future research should continue to evaluate the importance and feasibility of role-play in telehealth coaching. Based on the caregiver implementation fidelity data, the coaching procedure included in this study was associated with caregivers consistently implementing the interventions as intended. Each of the interventions implemented were based on established evidence-based practices (Wong et al., 2015); therefore, the program led to access to evidence-based practices for children with ASD. Along with previous research (e.g., Boutain et al., 2020; Wacker et al., 2013), this research indicates that coaching caregivers via telehealth is likely to lead to high caregiver implementation fidelity, which is likely to increase children’s access to evidence-based practices and yield long-term improvements in child outcomes. However, there is a need for additional research on the extent to which telehealth caregiver coaching yields long-term improvements in caregivers’ implementation fidelity and child outcomes. It is also important to note that the caregivers did not implement every procedure with 100% fidelity. Future research should continue to evaluate factors associated with higher and lower caregiver fidelity.
The social validity of an intervention is addressed through multiple components of the intervention (Horner et al., 2005; Reichow et al., 2008). To address socially significant goals and create an acceptable intervention that fit well within the family routine, we included caregivers and children in goal selection and intervention development. During each visit, therapists collaboratively planned the visit with the caregiver (e.g., order of goals, number of sessions per goal). We also solicited caregiver report about treatment acceptability through a researcher-adapted questionnaire at the end of the study. On the social validity questionnaire, caregivers rated the program highly on average (3.8 out of 4), indicating they found the program acceptable. However, the number of participants who consented but did not complete all study procedures suggests that some component of the study procedures may not have been acceptable or feasible for families. We received consent from caregivers for 24 children who did not complete the project. Many of these families chose not to complete the post-test (11 children), which required 2 hours, or did not initiate or continue participation because they were too busy to schedule visits (six children). These data indicate that the most common reason for not completing all of the study procedures was that the they required too much time for families—especially the pre- and post-test procedures. Researchers and practitioners should consider using measures that require less time for the family.
Limitations and Directions for Future Research
The findings of this study should be interpreted in the context of the limitations. The sample size for the study was relatively small and all participants received the intervention. Therefore, it is difficult to interpret the findings and the results should be considered preliminary. Future research should include a larger sample size and a stronger research design. In addition, several participants withdrew throughout various phases of the study. Future research should conduct additional analyses to assess the extent to which participant withdraw affects the study results and to identify methods to reduce participant withdraw in the context of these types of programs. In the present study, we evaluated the extent to which the intervention yielded improvements on the Vineland-3, but we did not evaluate Vineland-3 results based on the domains targeted for the child (i.e., some children received an intervention to address a socialization goal and some did not, but we evaluated improvement on the Vineland-3 socialization domain across all children). Future research should evaluate the extent to which children improve in targeted and non-targeted domains.
The characteristics of the participants included in this study may have affected the results and there is a need for more research with additional participants. We included participants up to age 17, but the majority of participants were 10 years old or younger. Therefore, the findings of this study are primarily applicable to younger children. Future research should include more participants age 11 to 17 years old and should evaluate the efficacy of similar procedures with participants in that age range. In addition, each of the included participants was receiving a related service at the onset of the study, with six of the included participants receiving ABA services. Access to these services may have affected the results of this study; there is a need to investigate the efficacy of this type of program using a design with a comparable group of participants not receiving the program, to ensure the services did not affect these results.
We did not conduct a cost analysis in the present study. Previous research indicates telehealth delivered challenging behavior interventions are more cost effective than in-person services (Lindgren et al., 2016). Future research should conduct cost analyses for interventions targeting other domains or multiple domains, such as this one. Our telehealth program utilized one coaching procedure; it would be beneficial for future research to evaluate the relative efficacy different procedures and the importance of specific components of telehealth coaching. Finally, the present study did not evaluate maintenance or generalization of caregiver or child behaviors. One potential benefit of caregiver implemented interventions is improved access to evidence-based practices through generalization and maintenance of caregiver implementation fidelity. Therefore, it would be useful to assess maintenance and generalization of both caregiver and child behaviors.
Conclusions and Implications for Practice
The present study consisted of a preliminary investigation of the efficacy of a telehealth caregiver coaching program. The results provided some preliminary support for the efficacy of the intervention, although there was no statistically significant improvement on several of the items measured. Based on the preliminary findings presented in this study, caregivers can implement evidence-based practices with their children with ASD following telehealth coaching and those practices often lead to improvements on individualized goals. However, further research supporting the efficacy of this type of program, with a stronger research design and larger sample size, is needed. The results of this study, along with previous research, provide preliminary support that a telehealth coaching model yields high caregiver implementation fidelity and improvements in individualized goals; practitioners can consider using this model, as appropriate, to provide coaching in research-supported practices to families they work with.
Footnotes
Author’s Note
Stephanie Gerow is also affiliated to Baylor University, Waco, TX, USA.
Marie Kirkpatrick is also affiliated to University of Texas at San Antonio, San Antonio, Texas.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in whole or in part by a grant from the Texas Higher Education Coordinating Board (THECB). The opinions and conclusions expressed in this document are those of the author(s) and do not necessarily represent the opinions or policy of the THECB.
