Abstract
Although paraprofessionals have become an increasingly integral part of special education services, most paraprofessionals lack training in evidence-based instructional strategies. We used a randomized contolled experimental design to examine the efficacy of a professional development training package and its individual components to equip 25 paraprofessionals to implement constant time delay. The effect of the training package on implementation fidelity was statistically significant and large in magnitude (d = 2.67; p < .001). Video modeling and coaching components were effective, although the effect of coaching alone (d = 2.23; p < .01) was larger than video modeling alone (d = .55; p = .18). Recommendations for further refining effective professional development opportunities for special education paraprofessionals are offered along with discussion of future research needs.
Legislative mandates and recommended practices in the field of special education affirm the importance of grounding instruction for students with disabilities in scientific research (Cook, Smith, & Tankersley, 2012; No Child Left Behind Act of 2001). Despite numerous calls for special educators to adopt evidence-based instructional practices when supporting students with disabilities, a concerning gap persists between research and practice (Cook & Schirmer, 2003; Snell, 2003). Special educators report limited use of evidence-based practice (McLean, Snyder, Smith, & Sandall, 2002) and using ineffective educational practices as frequently or more frequently than they use practices validated by research (Burns & Ysseldyke, 2009). Observations of special education programs show that without extensive training and support, special educators implement few evidence-based practices and often with inconsistent fidelity (Odom, Cox, Brock, & National Professional Development Center on ASD, 2013). Indeed, dissemination of evidence-based practice to school-based personnel remains a pressing concern for the field of special education (Cook & Odom, 2013; Klingner, Boardman, & McMaster, 2013).
This gap between research and practice is especially concerning when considering the place and prominence of paraprofessionals in the delivery of special education services. Paraprofessionals have increasingly become an integral part of special education services, especially for students with intellectual disability and/or autism (Giangreco, 2009). More than 400,000 full-time equivalent paraprofessionals currently work with school-aged children who receive special education services in the United States (U.S. Department of Education, 2010). In many states, schools employ more special education paraprofessionals than certified special educators (Giangreco, Hurley, & Suter, 2009). Moreover, these school staff spend considerable time working closely with students with severe disabilities. For example, 97% of special education paraprofessionals report providing one-to-one instruction to students with disabilities either daily or weekly (Carter, O’Rourke, Sisco, & Pelsue, 2009) and 87% report regularly providing behavioral and social support (Fisher & Pleasants, 2012).
Most special education paraprofessionals receive strikingly limited training. Many have no education past high school (Fisher & Pleasants, 2012) and most have never received in-service training on basic instructional strategies (Carter et al., 2009). When surveyed, paraprofessionals indicate they are extremely interested in additional training, including training related to instructional strategies (Breton, 2010; Carter et al., 2009). Descriptive studies suggest that without strong training, paraprofessional support does not appear to improve the learning outcomes of students with disabilities and may actually hinder them (Giangreco, Suter, & Doyle, 2010). For example, untrained paraprofessional support in inclusive settings may limit opportunities for students with disabilities to interact with peers and general educators (Giangreco, Edelman, & Broer, 2001; Malmgren & Causton-Theoharis, 2006).
While it is clear special education paraprofessionals would benefit from more training, it is less clear how such training should be delivered. Training methods for special education paraprofessionals have been experimentally tested in relatively few studies, and most of these studies did not include replicable training procedures (Brock & Carter, 2013). Unfortunately, the most commonly used vehicle for school-based professional development—single-event training workshops (e.g., faculty in-services)—may be ineffective at equipping paraprofessionals to deliver high-quality instructional support. Experimental studies indicate such workshops have a very limited impact on paraprofessional behavior (Barnes, Dunning, & Rehfeldt, 2011; Hall, Grundon, Pope, & Romero, 2010). This lack of effectiveness is not limited to special education; research across disciplines has shown that single-event training sessions have little or no impact on everyday practice (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). The logic of stand-alone training workshops mistakenly presumes practitioners will correctly generalize content from the training to practice context independently without follow-up training or support (Hall et al., 2010).
Several intervention studies involving paraprofessionals have incorporated training approaches that extended beyond the “stand-alone workshop” and directly targeted generalization to daily practice for special education paraprofessionals who serve students with severe disabilities (Brock & Carter, 2013). Nearly all studies in this review involved one-to-one coaching or mentoring where a professional development provider delivered individualized follow-up training to the paraprofessional. Within this context of one-to-one coaching, three components were included in intervention packages associated with paraprofessional acquisition of correct delivery of instruction: modeling, performance feedback, and accountability. Modeling involved live (e.g., Gilligan, Luiselli, & Pace, 2007) or video modeling (e.g., Robinson, 2011) of the targeted intervention as the instructor highlighted key steps of the intervention. Performance feedback involved a mentor observing the paraprofessional implementing the intervention and providing constructive verbal, visual, or video feedback on his or her performance (e.g., Bingham, Spooner, & Browder, 2007; Hall et al., 2010; McDonnell, Johnson, Polychronis, & Risen, 2002). Modeling and performance feedback are associated with improved implementation in the broader coaching literature, including studies involving follow-up training for special educators (Kretlow & Bartholomew, 2010) and early childhood practitioners (Snyder et al., 2012). In addition to modeling and performance feedback, these studies all incorporated some degree of accountability. For example, researchers explicitly instructed paraprofessionals to implement the targeted interventions in daily practice and followed up to ensure implementation actually occurred.
Together, these three components represent three critical training features. Trainers should clearly communicate how to implement an intervention (modeling), ensure participants attempt to implement the intervention in everyday practice (accountability), and then follow up with participants to reinforce what they are doing well and to help them correct their mistakes (performance feedback). In our systematic review of the research literature (Brock & Carter, 2013), we found no studies testing a combination of these components to train special education paraprofessionals in a randomized controlled trial, nor did we identify any studies in which these components were analyzed individually.
In the present study, we combined modeling, performance feedback, and accountability into a flexible and replicable training package called Video Modeling Plus Abbreviated Coaching (VMPAC). VMPAC involves an initial training workshop followed by video modeling and abbreviated on-site coaching. The initial training workshop includes a description and demonstration of the instructional practice, as well as opportunities for practitioners to simulate the instructional practice through role-play. Video modeling involves practitioners comparing their own performance with video exemplars reflecting a range of students. While watching a video exemplar of an instructional practice, practitioners review the steps associated with the practice and plan how they might implement the practice with a student in the classroom. The third component of the training package, abbreviated in-person coaching, involves a single 1-hr meeting in which a coach observes the practitioner implementing the instructional strategy in the natural school setting, provides targeted instructive feedback, models the correct implementation steps, and gives the practitioner additional opportunities for guided practice. During the coaching session, the coach holds the paraprofessional accountable for planning and attempting to implement the instructional strategy by reviewing written plans and discussing issues related to implementation.
VMPAC is designed to capitalize on effective components of training while considering the logistical and resource constraints of public school districts providing professional development. Most individualized coaching models require extensive one-to-one consultation over the course of weeks or months (e.g., Fixsen et al., 2005; Kretlow & Bartholomew, 2010). School systems looking for efficient ways to train large numbers of paraprofessionals are unlikely to invest in approaches that require considerable time and resources while only impacting a single practitioner at a time (Russo, 2004). In contrast, VMPAC only requires 1 hr of one-to-one consultation and utilizes technology to provide low-cost supplemental training through video modeling. If effective, VMPAC could potentially offer school systems a powerful and feasible model of professional development.
In this article, we report findings from a randomized controlled trial testing the efficacy of the VMPAC training package to disseminate evidence-based practice to special education paraprofessionals. While VMPAC is designed to be a flexible training package that could target numerous evidence-based instructional practices, we selected constant time delay as the focus of this evaluation. Constant time delay is a versatile strategy focused on systematically fading instructional prompts (Wolery, Ault, & Doyle, 1992). Constant time delay initially involves errorless learning, where the instructor draws the student’s attention to a stimulus, gives a cue or task direction, and then immediately prompts the student to provide the desired response and provides reinforcement and behavior-specific praise. After several sessions of errorless learning, the instructor pauses for a pre-determined period of time (inserts a delay) to provide the student an opportunity to respond independently before prompting the correct response. The only prompt delivered in time delay is a controlling prompt—the least intrusive prompt that ensures success. We selected constant time delay because (a) numerous studies confirm constant time delay produces positive outcomes for students with intellectual disability and/or autism (Browder, Ahlgrim-Delzell, Spooner, Mims, & Baker, 2009; Odom, Collet-Klingenberg, Rogers, & Hatton, 2010; Spooner, Knight, Browder, & Smith, 2011) and (b) constant time delay is a versatile response prompting strategy that can be used to target a variety of discrete (Wolery, Holcombe, et al., 1992) and chained skills (Schuster et al., 1998).
We tested the efficacy of the VMPAC training package, and its individual components, in a randomized controlled trial. Specifically, we sought to answer the following research questions:
Compared with a stand-alone training workshop, does a combination of a workshop and follow-up VMPAC training package improve paraprofessionals’ implementation of constant time delay?
What is the effect of only the video modeling component on implementation fidelity?
What is the effect of only the coaching component on implementation fidelity?
What is the effect of only the coaching component compared with a combination of coaching and video modeling?
Method
Participants
Participants were 25 paraprofessionals who served children and youth with disabilities within one school district serving rural and suburban communities in the southeastern United States. These 25 paraprofessionals represent 5.9% of the special education paraprofessionals in the school district. All paraprofessionals were more than 40 years old (60.0% were 40–49 years old; 36.0% were 50–59 years old; 4.0% were older than 60), predominately female (96.0% female; 4.0% male), and predominantly Caucasian (88.0% Caucasian; 8.0% Asian–Indian; 4.0% Australian). They served students with disabilities across elementary (n = 15), middle (n = 6), and high (n = 4) school settings. Most (n = 21) reported receiving no previous training in time delay procedures. Participant characteristics are described in more detail by training group in Table 1.
Paraprofessional Characteristics by Training Group.
Australian. bAsian–Indian.
We recruited participants via a mass email sent to all special education paraprofessionals in the school district. This email advertised a professional development opportunity to learn effective strategies to teach students with disabilities new skills and to promote students’ independence. The invitation explained the training would include an initial workshop as well as follow-up video and in-person coaching. Incentives for participation included receipt of district professional development credit contingent on completion of all phases of the professional development package. We screened participants to ensure only special education paraprofessionals who regularly served students with disabilities were admitted to the initial training workshop. Of the 34 paraprofessionals who attended the initial training workshop, 29 consented to the research project and were randomized to training groups for follow-up training. Four paraprofessionals (two from each training group) withdrew from the study prior to completion. The characteristics of the paraprofessionals in the experimental and comparison groups did not differ significantly on any demographic variables except for level of education. Paraprofessionals in the comparison group tended to have higher levels of education, χ2(3, N = 25) = .07, p < .01.
Study Design
This randomized controlled trial tested the efficacy of the VMPAC training package to teach paraprofessionals to implement constant time delay. All paraprofessionals attended a 2-hr training workshop on evidence-based prompting procedures, including constant time delay. At the beginning of the workshop, the first author invited paraprofessionals to participate in a research project by allowing data collected during the training process to be analyzed for research purposes. The first author explained that consent was voluntary, and that participation in the training and receipt of district credit were not contingent on consent to the research project. Paraprofessionals who consented to the research project were randomized to 2 groups for 3 weeks of follow-up training. The experimental group (n = 12) received the VMPAC training package targeting implementation of constant time delay. The comparison group (n = 13) received video training on unrelated strategies and then received in-person coaching on constant time delay. Time delay implementation fidelity data were collected for both groups in their actual classrooms immediately before and immediately after coaching. This design allows for three different contrasts between training groups: a primary contrast between VMPAC versus unrelated follow-up training and two secondary contrasts to gauge the efficacy of the two individual components of the intervention (i.e., video modeling and abbreviated coaching; see Figure 1).
Diagram of research design and procedures.
Training Groups
Pre-randomization training workshop
Prior to randomization to training groups, all paraprofessionals participated in a 2-hr training workshop led by the first author, an experienced special education teacher and school-based trainer. A second trainer circulated the room during role-play activities to support participants. The purpose of this training was to teach participants the implementation steps associated with constant time delay and least-to-most prompting. Activities at the workshop included greeting participants and sign-in (7 min), distribution and explanation of research consent forms (17 min), brief introduction to behavioral terminology and systematic instruction (11 min), lecture on constant time delay implementation steps (Neitzel & Wolery, 2009a; 11 min), video demonstrations and discussion of constant time delay (11 min), instructor demonstration of constant time delay (2 min), and participant role-play of constant time delay with a partner (16 min). Constant time delay implementation steps were divided into simultaneous prompting steps and time delay steps. These implementation steps were targeted explicitly, matched the same implementation steps targeted in subsequent phases of training, and were measured by the primary dependent variable (see Table 2). Additional activities included lecture on least-to-most prompting implementation steps (Neitzel & Wolery, 2009b; 5 min), video demonstration and discussion of least-to-most-prompting (3 min), instructor demonstration of least-to-most prompting (2 min), and participant role-play of least-to-most prompting with a partner (10 min). During training activities, two additional members of the research team randomized all participants into two training groups using a computerized randomization program. During the last 25 min, participants were divided into training groups and each group received an overview of what the rest of the training would entail, including verbal and written directions for participant responsibilities during follow-up training.
Average Fidelity on Implementation Behaviors by Intervention Group and Time Point.
Note. Number of participants varies across steps, as some steps (e.g., 5a and 5b for simultaneous prompting; 5a, 5b, 5c, 6a, and 6b for time delay) are contingent on student response. If the paraprofessional did not correctly implement a controlling prompt at the appropriate time (e.g., Step 4 for simultaneous prompting or Step 5b for time delay trials), then the student did not have an appropriate opportunity to respond, and then the coach was unable to score subsequent steps.
Experimental group
The experimental group received the VMPAC follow-up training package consisting of (a) three weeks of access to video modeling and (b) abbreviated in-person coaching. Each week for three weeks, we sent paraprofessionals an electronic link via email to access the video at an online video sharing website. Each of the three 15-min videos on constant time delay provided an overview of constant time delay, reviewed implementation steps associated with constant time delay addressed at the workshop, provided a video exemplar of a member of the research team implementing constant time delay with a student with a disability, and asked paraprofessionals to complete a worksheet detailing how they might implement each step of time delay with a student at their school. While all three videos addressed the same implementation steps, each included a different video exemplar of a member of the research team implementing time delay to teach a (a) discrete academic skill to an elementary school student with intellectual disability, (b) chained vocational task to a high school student with intellectual disability, or (c) self-help skill to a preschool student with autism. The video narrator directed participants to record on a worksheet how they would carry out each implementation step for a specific student with whom they worked, and reminded participants they would later share these worksheets with a coach during the in-person coaching session.
One-hour, in-person coaching sessions occurred 3 to 4 weeks after the initial training workshop and immediately after video modeling. In-person coaching focused on performance feedback and live modeling. Members of the research team contacted paraprofessionals by email to arrange a 1-hr coaching session when the paraprofessional would have access to a student with a disability. All in-person coaching lasted 60 min or less and occurred in the school environments where paraprofessionals normally worked (e.g., special education resource rooms, general education classrooms, and at workstations in the hallway). An intervention coach (i.e., a graduate student in special education) briefly established rapport with the paraprofessional by introducing himself or herself, thanking the paraprofessional for his or her participation, and providing an overview of what the coaching session would entail. The coach then asked the paraprofessional to demonstrate implementation of time delay with a student with a disability. If paraprofessionals did not identify appropriate instructional materials, the coach provided access to a collection of materials that could be used to teach a range of discrete responses (e.g., math flash cards, sight word sets of variable difficulty, moveable clock faces for telling time). Most paraprofessionals were prepared to target a goal with materials that were already part of ongoing instructional activities. Paraprofessionals targeted a wide range of goals, including identifying the letters of one’s name in the correct order, answering reading comprehension questions, defining vocabulary terms related to a social studies curriculum, matching rhyming words, correctly writing spelling words, and completing long division problems. Coaches advised paraprofessionals to implement time delay in the context of one-to-one or small-group instruction, because these are the contexts where time delay is best supported by experimental research (Wolery, Holcombe, et al., 1992), and because these contexts were the most conducive to providing coaching. Most instruction was provided in a one-to-one format, although two paraprofessionals in the experimental group implemented time delay during small-group instruction.
The intervention coach collected fidelity data using an adapted version of the time delay implementation checklist developed by Neitzel and Wolery (2009a). This checklist aligned with the implementation steps presented to all participants at the training workshop and in the video models viewed by the experimental group. Trials were divided into (a) simultaneous prompting trials and (b) time delay trials. For simultaneous prompting trials, the coach scored six different implementation behaviors dichotomously as correctly implemented or not implemented (see implementation behaviors 1-5b in Table 2). The last two behaviors (5a and 5b) were dependent on the student response. After a correct student response, the coach scored whether the paraprofessional immediately provided reinforcement and stated what the student did. After an incorrect response, the coach scored whether the paraprofessional ignored the response and presented the same stimulus to repeat the trial. For time delay trials, six or nine different implementation behaviors were scored, depending on whether the student provided a correct response within the response interval.
The coach implemented the following steps during each coaching session. First, the coach asked the paraprofessional to implement time delay as if he or she were teaching a new skill to the student for the first time. The coach watched (without scoring) the first instructional trial and took note of the first instructional prompt used by the paraprofessional and the type of stimulus presented. For the remainder of the observation, this instructional prompt was defined as the controlling prompt and the beginning of an instructional trial was defined as the presentation of a similar stimulus. The coach scored 3 to 5 instructional trials (3 trials for 10 paraprofessionals; 4 trials for 6 paraprofessionals; 5 trials for 9 paraprofessionals) using the simultaneous prompting implementation checklist. The coach attempted to score up to five instructional trials, but in some cases student behavior or task characteristics limited the number of instructional trials to three or four. Next, the coach asked the paraprofessional to implement time delay as if he or she had already targeted the skill for several sessions. The coach scored 3 to 5 trials (3 trials for 11 paraprofessionals; 4 trials for 3 paraprofessionals; 5 trials for 11 paraprofessionals) using the time delay implementation checklist. If the paraprofessional did not correctly implement a controlling prompt at the appropriate time (e.g., Step 4 for simultaneous prompting or step 5b for time delay trials; see Table 2), then the coach was unable to score subsequent steps as these steps are contingent on the student’s correct or incorrect response after the controlling prompt. Instead, the coach scored two incorrect implementation behaviors to account for the lack of correct implementation (e.g., equivalent to incorrect implementation of step 5a or 5b for simultaneous prompting).
After collecting implementation fidelity data, the coach entered data into a computer spreadsheet that generated a graph displaying how many times each implementation step had been followed correctly. The coach showed this graph to the paraprofessional and reviewed each implementation step individually (see Table 2). If a step was followed correctly, the coach provided specific praise. If a step was not followed correctly, the coach reviewed the step and modeled correct implementation. Then the coach asked the paraprofessional to restate the intervention steps in his or her own words. Next, the coach prompted the paraprofessional to demonstrate use of time delay a second time on the same skill with the same student. As with the first demonstration, implementation fidelity data were collected and visual feedback was provided. The coach compared the overall fidelity of implementation between the two demonstrations, asked the paraprofessional if he or she would like to review any of the steps a second time, and reviewed and modeled steps as requested. Finally, the coach asked the paraprofessional to share the worksheets he or she had completed during the video modeling sessions. While reviewing these worksheets, the coach provided specific praise for correct examples of implementation, offered suggestions for revising incorrect examples for implementation, and worked with the paraprofessional to identify new opportunities for implementation of time delay.
Comparison group
The comparison group received two components of follow-up training: (a) access to three videos on natural supports, a set of strategies unrelated to time delay that address inclusion of students with disabilities among peers without disabilities; and (b) abbreviated in-person coaching focusing on time delay. Each week, paraprofessionals viewed a different 15-min video that defined and provided examples of a principle associated with natural supports. These principles focused on creating shared activities, promoting valued roles, and fostering real relationships (Carter, Swedeen, & Kurkowski, 2008). At the end of each video, paraprofessionals were prompted to brainstorm ways they might address the presented principle with a specific student, and to record these ideas on a worksheet. Abbreviated in-person coaching focused on time delay, and was identical to the experimental group with one exception. At the end of the coaching session, instead of reviewing time delay worksheets, the coach and the paraprofessional reviewed the ideas related to natural supports that the paraprofessional had recorded on the worksheets and discussed implementation of natural supports.
Dependent Measures
Paraprofessional performance
Intervention coaches collected data on the primary dependent variable—time delay implementation fidelity—for both training groups. This includes data collected immediately before and after the abbreviated coaching session, as described in the previous section. Overall implementation fidelity was calculated for each paraprofessional by (a) computing the percentage of steps implemented correctly for each observed instructional trial, (b) averaging the percentage correct for each trial across simultaneous prompting trials and time delay trials, and (c) averaging the two percentages for simultaneous prompting trials and time delay trials. These steps were followed so each instructional trial would be weighted equally (regardless of student responses that may have increased or decreased the number of opportunities for correct implementation behaviors), and so simultaneous prompting trials and time delay trials would be weighted equally (regardless of variability in the number of instructional trials observed). In addition, correct implementation of each step was calculated for each paraprofessional to analyze differences in error patterns between training groups.
Observer Training and Interobserver Agreement
Three members of the research team, including the first author, observed paraprofessional implementation of time delay. All observers were graduate students in special education, one with coaching experience and two with training in applied behavior analysis. The first author provided the other two observers with written descriptions of observation procedures and operationalized definitions of each implementation behavior. Training included scoring time delay implementation on training videos and continued until all three observers reached 100% agreement on training videos. Training continued during initial data collection until observers reached a minimum of 90% agreement with the first author. All observers reached this criterion after two observations. The observations meeting this criterion, as well as all subsequent observations with a secondary observer, were counted toward calculation of interobserver agreement.
A second observer collected data during coaching sessions for 9 of the 25 participants (36%). Observations were balanced across training groups. Data from the primary and secondary observers were aligned by instructional trial (i.e., both observers coded the same stimulus), and interobserver agreement was calculated as the number of scoring agreements divided by the total number of agreements plus disagreements. Overall interobserver agreement was 95.2% (range = 91.4%−100%) across sessions.
Adherence to Training Protocol
Adherence to training protocol was measured for video modeling and coaching. At the end of each time delay video, the video narrator instructed participants to click a hyperlink below the video. The hyperlink led participants to a webpage where they were prompted to enter their names, which were captured in an electronic database to track which participants had viewed each video. A member of the research team monitored the database twice each week, and sent reminders to those with no record of completion. Of the 12 paraprofessionals in the experimental group, 7 watched all 3 videos, 3 watched 2 videos, 1 watched 1 video, and 1 did not watch any videos. During coaching sessions, a secondary observer collected coaching fidelity data on a 30-behavior coaching fidelity checklist (available from the first author by request). The 30 behaviors were categorized as establishing rapport (3 behaviors), observation of implementation (7 behaviors), performance feedback and modeling (7 behaviors), check for understanding (1 behavior), second observation of implementation (4 behaviors), performance feedback (4 behaviors), and review of completed worksheets (4 behaviors). Implementation fidelity, calculated as correctly implemented coaching behaviors divided by 30 opportunities for correct implementation, was 99.6% (range = 96.6%−100%) across sessions.
Statistical Analysis
Statistical analyses were designed to address the four research questions. First, to assess the efficacy of the workshop plus VMPAC follow-up training package relative to the workshop only, we used a one-way ANOVA to test the difference between the pre-coaching fidelity scores of the comparison group with the post-coaching fidelity scores of the experimental group. Second, to assess the effect of only the video modeling component of the training, we used a one-way ANOVA to test the difference between pre-coaching fidelity scores between training groups. In a separate analysis to examine the relation of videos watched on fidelity outcomes, the number of videos watched was regressed on fidelity of implementation using ordinary least squares (OLS) regression. Because participants in the comparison group did not watch any videos focusing on time delay, they were assigned a value of zero. Third, to assess the effect of the abbreviated coaching component of the training, we used a paired-samples t test to test the difference between the pre-coaching and post-coaching fidelity scores for the comparison group. Fourth, to compare the effects of a combination of video modeling and coaching with coaching alone, we used a one-way ANOVA to test the difference between post-coaching fidelity scores. Effect sizes (Cohen’s d) were computed for each comparison by dividing the difference in means by the pooled standard deviation (Cohen, 1992). Effect sizes were evaluated using the guidelines proposed by Cohen (1988), where a d greater than 0.20 is a small effect, greater than .50 a medium effect, and greater than .80 a large effect.
Results
Efficacy of VMPAC
A one-way ANOVA compared post-coaching scores of the experimental group (M = 96.4%; SD = 7.1%) with the pre-coaching fidelity scores of the comparison group (M = 71.6%; SD = 11.0%). When compared with only the stand-alone workshop, effects attributed to the addition of the VMPAC follow-up training package were statistically significant, F(1, 23) = 43.77, p < .001, and large in magnitude (d = 2.67).
Impact of Video Modeling Alone
A one-way ANOVA tested the difference in pre-coaching fidelity scores between the experimental (M = 78.2%; SD = 12.7%) and comparison (M = 71.6%; SD = 11.0%) groups. When compared with unrelated video modeling, the effect of video modeling alone was not statistically significant, F(1, 23) = 1.92, p = .18, but was moderate in magnitude (d = .55). When a regression analysis took differences in paraprofessional adherence to protocol into account, the number of videos watched was a significant predictor of implementation fidelity, β = 3.63; F(1, 23) = 4.63, p = .04. However, we emphasize this regression analysis does not assess a causal relationship between video modeling and implementation fidelity, because we did not have experimental control of adherence to protocol.
Impact of Abbreviated Coaching Alone
A paired-samples t test tested the difference between the pre-coaching (M = 71.6%; SD = 11.0%) and post-coaching (M = 96.5%; SD = 11.2%) fidelity scores for the comparison group. The effect of coaching alone was statistically significant, t(12) = 6.56, p < .001, and large in magnitude (d = 2.23).
Comparison of Video Modeling and Coaching to Coaching Alone
A one-way ANOVA tested the difference between the post-coaching scores for the experimental (M = 96.4%; SD = 7.1%) and comparison (M = 96.5%; SD = 11.2%) groups. There was no statistically significant difference between the effect of a combination of video modeling and coaching and coaching alone, F(1, 23) < .001, p = .99.
Analysis of Fidelity on Individual Implementation Steps
Fidelity on individual implementation steps is reported in Table 2 by training group and time point. Between the paraprofessionals receiving the complete VMPAC training package and those receiving only workshop training, differences in fidelity were most pronounced for the following implementation steps: immediate delivery of the controlling prompt during simultaneous prompting trials (step 4), delivery of the controlling prompt at the end of the response interval during time delay trials (step 5b), and immediate delivery of reinforcement and stating what the learner did after a correct response (simultaneous prompting step 5a; time delay prompting steps 5a and 6a).
Participant Satisfaction
After the completion of the study, a member of the research team emailed participants a hyperlink to a satisfaction survey to be completed anonymously. Members of the experimental and comparison groups received two different hyperlinks so video ratings could be differentiated by group. Twenty-four of the 25 participants (96.0%) completed the survey. Participants rated how helpful they found each component of the training (workshop, video modeling, in-person coaching) in learning new skills that would help them better serve students with disabilities. Each item was rated on a 5-point Likert-type scale (1 = not helpful at all; 2 = minimally helpful, 3 = somewhat helpful; 4 = very helpful; 5 = extremely helpful). On average, participants rated the initial workshop as very helpful (M = 4.13; SD = 0.68), time delay video modeling as very helpful (M = 4.00; SD = 0.60), natural supports videos as somewhat to very helpful (M = 3.58; SD = 0.88), and in-person coaching as very to extremely helpful (M = 4.46; SD = 0.72).
Discussion
While paraprofessionals have increasingly become an integral part of special education services, most paraprofessionals are not provided training in evidence-based instructional strategies. We investigated the efficacy of a package of promising professional development components compared with a stand-alone training workshop for enabling special education paraprofessionals to implement constant time delay with fidelity. We found that the follow-up training package was effective, and that coaching was the most powerful component of this package. This is the first randomized controlled trial study showing that when provided effective training, special education paraprofessionals can implement an evidence-based practice with high fidelity. The efficacy of the training package, and the relative efficacy of the components of the package, can be explained by two ideas that are familiar to most special educators: generalization and data-based instruction.
The Intersection of Training Methods and Generalization
A key feature of VMPAC—and a critical difference from the stand-alone training workshop—is it directly targets generalization of instructional strategies to the classroom. Other researchers (e.g., Reinke, Stormont, Webster-Stratton, Newcomer, & Herman, 2012) have highlighted generalization as the key to successful professional development. Indeed, training opportunities that promote acquisition of practitioner knowledge—but fail to impact how practitioners provide instruction in the classroom—are of little benefit to students with disabilities. Almost 40 years ago, Stokes and Baer (1977) identified nine different approaches to promoting generalization. Although they created these categories with instruction of students with disabilities in mind, generalization is a ubiquitous concept that can be applied to all learners, including adult practitioners.
The avenues of training tested in this study can be viewed in terms of the degree to which they integrate the generalization strategies outlined by Stokes and Baer (1977). Stand-alone workshops targeting acquisition of intervention skills during training with other adults, but requiring the participants to figure out on their own how to apply interventions to real students in real classrooms, parallel the “train and hope” approach. Without follow-up training, it is unlikely practitioners will implement targeted interventions in daily practice with adequate fidelity. This was exemplified in the present study, as paraprofessionals in the workshop-only condition demonstrated the poorest implementation fidelity. The shortcomings of stand-alone training workshops are especially concerning for paraprofessionals who often have no previous training in special education and depend solely on in-service training for professional development (Fisher & Pleasants, 2012).
The video modeling component of the VMPAC package provided paraprofessionals with several different examples of how one might implement constant time delay to teach different skills to students with diverse characteristics. By sharing multiple exemplars of correct implementation in different contexts (“train sufficient exemplars”), video models may have provided paraprofessionals with a clearer conceptualization of implementation steps and helped them better envision how they might use constant time delay with their own students. To connect these models to implementation in the classroom, the video narrator prompted paraprofessionals to record how they would carry out each implementation step with an actual student on a worksheet. Multiple exemplars might be especially important for special education paraprofessionals who serve multiple students with a diverse range of characteristics and educational goals.
While the number of videos watched correlated with paraprofessional implementation fidelity, the magnitude of the effect was modest when compared with the very large effect produced by the coaching component. Furthermore, coaching alone produced similar effects compared with a combination of video modeling and coaching. The relative superiority of the coaching component is not surprising. By design, the coaching session obviated the need for paraprofessionals to generalize, because the coaching session and the post-assessment focused on the same target student and target skill. Similar to a “sequential modification” approach, in-classroom coaching targets performance in the same context where the skill will be used. When teachers regularly direct paraprofessionals to provide similar kinds of instruction or support to the same student in the same context, coaching might be the most powerful and appropriate training tool.
Data-Based Professional Development
A second critical difference between the VMPAC training package and stand-alone training workshops is a focus on data-based training. Specifically, the most powerful component of the training package—one-to-one coaching with performance feedback—involved observing paraprofessional performance for specific instructional behaviors, and then providing focused support on areas of implementation needing improvement. The other components of this training package—an initial workshop and video modeling—were designed for a general audience and did not use data to target identified weaknesses in implementation.
Data collection is clearly useful when practitioners do not need training on every facet of the intervention. Findings from the present study, as well as previous studies, show practitioners often already implement evidence-based practice partially or inconsistently before receiving focused professional development. For example, Lawton and Kasari (2012) found prior to receiving training on a multi-component joint attention intervention for young children with autism, teachers already used at least some of the strategies associated with the intervention. Similarly, when Sutherland and Wehby (2001) worked with teachers to increase how often they praised their students, they found nearly all teachers already provided at least some praise to their students before participating in professional development. In both cases, the problem was not that teachers completely failed to implement an intervention, but they implemented it only partially or inconsistently. These gaps in implementation impact the potency of the intervention.
Similarly, paraprofessionals in the present study correctly implemented on average 71.6% of the behaviors associated with constant time delay prior to receiving any follow-up training. Many of these behaviors (e.g., securing the student’s attention, delivering the cue or task direction) are not unique to time delay. However, like the aforementioned studies, gaps in implementation were critical and undermined the potency of the intervention. Without follow-up training, the most common implementation error was not delivering the controlling prompt in simultaneous prompting trials. Paraprofessionals inserted a response interval in all trials, failing to distinguish between different teaching procedures for initially teaching a response (simultaneous prompting) and fading prompts over time (time delay). Failing to initially deliver simultaneous prompting trials fundamentally changes the nature of constant time delay, as this is the mechanism for promoting low initial error rates (Schuster, Griffen, & Wolery, 1992). Failing to give behavior-specific praise may also be problematic, as explicitly linking reinforcement to the target behavior is thought to play a role in increasing the likelihood the target behavior will occur in the future (Wolery et al., 1992).
By collecting data on implementation fidelity, the coach was able to directly target these specific errors. This data collection allows for more effective professional development. It is unlikely practitioners need training and support on all implementation behaviors, and data collection allows for targeted and efficient use of valuable training time.
Implications for Practice
We found that given effective training, special education paraprofessionals can implement an evidence-based practice with high fidelity. The findings from this study have implications for public school administrators and special education professionals who make decisions about how to equip and supervise paraprofessionals. The federal law stipulates “paraprofessionals and assistants who are appropriately trained and supervised” may be “used to assist in the provision of special education and related services” (Individuals With Disabilities Education Act, 2004). However, this legislation does not provide any guidelines for what constitutes appropriate training and supervision.
Leaders in the field have suggested the appropriateness of training should be measured by the degree to which it enables paraprofessionals to provide effective instruction and support (Giangreco et al., 2001). In this study, we identified promising training strategies and illustrated how one might embed performance evaluation within the context of professional development opportunities. In a framework where implementation fidelity data are collected naturally within the coaching process, administrators and special educators can adapt promising training methods so they are flexible and feasible, and then adjust these methods based on measures of paraprofessional performance.
The results of this study show a brief coaching session emphasizing modeling and performance feedback can be a powerful tool for promoting implementation fidelity. Instead of depending solely on stand-alone training workshops, special education teachers or special education supervisors could schedule follow-up training through abbreviated coaching sessions throughout the school year to provide more focused support and evaluate paraprofessional performance. Although evidence for this approach is still emerging, using promising training techniques while evaluating paraprofessional performance would seem to be more compelling than business-as-usual stand-alone workshops that have little empirical support.
Limitations
Several limitations to this study suggest avenues for future research. First, we did not measure student outcomes in this study, so the relationship between the implementation fidelity of constant time delay and student performance is unclear. Our goal was not to evaluate the efficacy of constant time delay, as the relationship between implementation of constant time delay and improvement of student outcomes is already well established (Browder et al., 2009; Odom et al., 2010; Spooner et al., 2011). Instead, we sought to examine the efficacy of VMPAC to train paraprofessionals to implement time delay in a manner consistent with the experimental literature. However, future studies measuring practitioner and student outcomes simultaneously could make stronger conclusions about how implementation fidelity mediates the relationship between professional development tools and student outcomes. Second, it is unclear if VMPAC or its components would be as effective for targeting other evidence-based practices. Future studies might test the flexibility and versatility of VMPAC with other instructional strategies, particularly those that are more complicated or have more implementation steps than constant times delay. Third, due to the small sample size in this study, it is unclear if video modeling is an effective professional development strategy. Despite a moderate effect size, video modeling did not have a statistically significant effect on implementation fidelity. Fourth, it is unclear if VMPAC or its components would be effective if implemented by school-based staff. In future studies, researchers might investigate whether these training procedures produce the same effects when implemented by special education teachers or supervisors. Fifth, although the paraprofessionals in our study may not be representative of those in other districts, they are not dissimilar from paraprofessionals in a nationally representative study (Carlson, Brauen, Klein, Schroll, & Willig, 2002). The majority of paraprofessionals in our sample were between 40 and 49 years old and on average worked in their current positions for 6.27 years; paraprofessionals in a nationally representative study were on average 44 years old with 6.5 years of experience (Carlson et al.). Sixth, we compared data collected three weeks after an initial training session with data collected immediately after a coaching session. The timing of data collection may have inflated the effects of the intervention. Finally, we measured implementation fidelity in the same context as the coaching session. Although we demonstrated paraprofessionals had the capacity to correctly implement constant time delay, it is unclear if they actually accurately implemented this strategy on a regular basis outside of the coaching session. In addition, it is unclear whether paraprofessionals would be able to generalize correct implementation behaviors to new students and target skills, or if they would maintain correct implementation behaviors over time. Future studies might include measures of generalization to examine implementation across new contexts or longitudinal designs to explore how VMPAC affects practitioner behavior in the weeks, months, and years after training is complete. To answer these pressing questions, researchers should aim to study professional development with the same experimental rigor applied to other educational practices (e.g., Gersten et al., 2005; Horner et al., 2005).
Conclusion
Paraprofessionals play an important role in serving students with disabilities. However, most paraprofessionals lack training in evidence-based practice. The findings from this study suggest the VMPAC training package may be one effective avenue for disseminating evidence-based practice to paraprofessionals. Within this package, coaching that includes modeling and performance feedback is the most powerful training strategy. Given the lack of evidence to support the use of stand-alone training workshops, future research and practice should focus on more promising professional development tools, particularly strategies that are data-based and promote generalization to everyday practice.
Footnotes
Acknowledgements
We thank Heartley Huber and Michelle Harvey for their assistance with coaching and data collection.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Partial support for this project was provided by a grant from the Tennessee Department of Education to Vanderbilt University (CFDA No. 84.027A).