Efficacy of training job coaches to implement evidence-based instructional strategies

Abstract

BACKGROUND: High-quality job coaching services are pivotal to the vocational success of adults with severe disabilities. However, many job coaches lack formal training in evidence-based instructional practices.

OBJECTIVE: In this study, we test the efficacy of a training package on the degree to which six job coaches implement three evidence-based instructional practices with fidelity (i.e., task analysis, simultaneous prompting, and least-to-most prompting).

METHOD: The training package features promising strategies that are validated in the broader staff training literature: description, modeling, and performance feedback on implementation steps. Six multiple baseline across behavior experiments were conducted.

RESULTS: All participants made progress toward correct implementation with group training, although some participants required individualized coaching to achieve correct implementation of all steps.

CONCLUSIONS: This study provides preliminary evidence that promising training strategies can be used to train job coaches to implement evidence-based instructional practices for adults with severe disabilities. In addition, this training provides one model for how to use both group and one-to-one coaching formats in tandem to promote implementation fidelity.

Keywords

Staff training job coaching modeling performance feedback implementation science severe disabilities

1 Introduction

Although adults with severe disabilities, like all people, desire to make meaningful contributions to society through gainful employment, most struggle to translate this aspiration into reality. The employment rate for adults with any disability in the United States for 2010–2012 was 32.0%, far behind the national average of 72.7% (U.S. Department of Labor, 2015). Findings from the National Longitudinal Transition Study-2 (NLTS2) show that certain disability groups have unemployment rates that are even lower. Specifically, individuals with developmental disabilities such as intellectual disability, autism, deaf-blindness, multiple disabilities, and orthopedic impairments are the least likely to be employed when compared to adults in other disability groups (Newman, Wagner, Knokey, Marder, Nagle, Shaver, & Wei, 2011). Although the NTLS2 does not provide data for the subset of individuals with severe developmental disabilities, findings from the 2010 Survey of Americans with Disabilities demonstrate that compared to other individuals with disabilities, those with the most severe disabilities are the most likely to be unemployed and live in poverty (Kessler Foundation and National Organization of Disability, 2010). Taken together, these statistics show that although having any disability puts one at risk for unemployment, the subset of individuals with developmental disabilities are at greater risk, and the subset of individuals with the most severe developmental disabilities are at the greatest risk of unemployment.

One avenue for improving employment outcomes for individuals with severe disabilities is supported employment. Across research studies, supported employment has enabled individuals with intellectual disability (Wehmeyer & Bolding, 2001), autism (Wehman et al., 2014), traumatic brain injury (Wehman, Targett, West, & Kregel, 2005), and severe mental illness (Burns et al., 2007) to be successful in competitive employment. Wehman et al. (2012) identified four different phases of supported employment intervention, including assessment and development of a jobseeker profile, developing and searching for a job, initial job site training that is faded over time, and long-term supports. Wehman et al. (2014) further identified that the most intensive of these categories is the initial job site training, in which a job coach implements behavioral strategies to analyze and assess job performance, teach new skills, and fade prompting and support to build independence. In addition, job coaches can cultivate natural supports by identifying natural cues in the environment, and training and partnering with coworkers (Wehman et al., 2016). Indeed, initial job site training is a pivotal phase of supported employment, and it is critical that job coaches provide optimal support that facilitates skill acquisition and independence.

Unfortunately, many job coaches are not well prepared to use evidence-based approaches for training jobseekers (Hall, Bose, Winsor, & Migliore, 2014; Migliore, Hall, Butterworth, & Winsor, 2010). Although job coaches in research studies often have a strong background in systematic instructional strategies that are grounded in applied behavior analysis (e.g., Wehman et al., 2012), in everyday practice job coaches often have little or no formal training in evidence-based instructional strategies (Rogan & Held, 1999). Although well-intentioned, job coaches who lack such training are unlikely to provide optimal coaching. In some cases, coaches who lack the skills to effectively train adults with severe disabilities shift their focus from training to task completion and resort to simply completing the job themselves (Parsons, Reid, Green, & Browning, 2001; Towery, Parsons, & Reid, 2014). This approach makes no attempt to teach or promote independence, but instead perpetuates the obstinate myth that this population cannot perform meaningful work (Guess, Benson, & Siegel-Causey, 2008; Wehman & Kregel, 1988).

Fortunately, there is evidence that job coaches can be trained in evidence-based instructional strategies, and that effective implementation of these strategies improves outcomes for adults with severe disabilities. Parsons et al. (2001) worked with job coaches who trained three adults with severe multiple disabilities to prepare printed materials for mailing at a publishing company. During baseline conditions, job coaches provided extremely high rates of work assistance, defined as the job coach completing part of the task for the job trainee. During the intervention condition, the research team created task analyses of each work activity and trained the job coaches to deliver instruction through least-to-most prompting. Across all three participants, job coaches provided substantially less work assistance, and job trainees increased their independence with vocational tasks. Furthermore, two of the job trainees increased their productivity, and the third maintained productivity consistent with baseline conditions when the job coach was performing parts of the task. Although Parsons et al. provide compelling evidence that job coach training can result in implementation of an evidence-based practice and improved outcomes for adults with severe disabilities, the emphasis of this study is on the relationship between job coach and job trainee behavior, and it does not include a description of how job coaches were trained.

Although it is clear that job coaches should be trained to implement evidence-based instructionalstrategies, there is scant evidence about how they can be best trained. We were unable to identify anypublished studies that describe and test the efficacy of training job coaches to implement evidence-based instructional strategies. However, a number of published research studies focus on training school-based paraprofessionals to use systematic instructional strategies to address academic and daily living skills for students with severe disabilities. This literature may be particularly helpful, given that both job coaches and school-based paraprofessionals are often tasked with providing instruction to individuals with severe disabilities without any formal training in evidence-based instructional strategies (Carter, O’Rourke, Sisco, & Pelsue, 2009; Rogan & Held, 1999; Towery et al., 2014). In their systematic review of the paraprofessional training literature, Brock and Carter (2013) identified eight studies in which paraprofessionals were trained to implement systematic instructional strategies including prompting and positive reinforcement (Hall & Macvean, 1997; Licciardello, Harchik, & Luiselli, 2008; Martella, Martella, Macfarlane, & Young, 1993), time delay (McDonnell, Johnson, Polychronis, & Risen, 2002), pivotal response training (Robinson, 2011), and discrete trial training (Dib & Sturmey, 2007; Gilligan et al., 2007; Leblanc, Ricciardi, & Luiselli, 2005). Across these eight studies, the most commonly used training strategies included (a) oral or written description of implementation steps (i.e., all eight studies), (b) performance feedback (i.e., seven studies), and modeling (i.e., six studies used live modeling, and one study also used video modeling).

In this pilot study, we combined these three promising training methods (i.e., description of implementation steps, modeling, and performance feedback) into a training package focused on enabling job coaches to implement systematic instructional practices to teach adults with severe disabilities new vocational skills. Specifically, we addressed two research questions regarding the efficacy of this training. First, what is the effect of group training using promising strategies (i.e., description of implementation steps, modeling, and performance feedback) on implementation fidelity of job coaches implementing systematic instructional practices on an untrained instructional target? Second, for job coaches who do not achieve 100% implementation fidelity after group training, what is the effect of a brief coaching session focused on the previously untrained instructional target?

2 Method

2.1 Participants, setting, and materials

Two male and four female job coaches who provided direct services to adults with developmental disabilities who worked in a sheltered workshop and community settings participated in this study. Highest level of education and years of experience providing job coaching varied greatly across participants. Jason had a bachelor’s degree and 3 years of experience, Sarah had a Master’s degree and 7 years of experience, Heather had a high school diploma and 13 years of experience, Laura had an Associate’s degree and over 30 years of experience, Julie had a Bachelor’s degree and 14 years of experience, and Bryan had a high school diploma and 7 years of experience. Five participants were White; one was African-American. All six job coaches worked for the same employer, who contracted with a large Midwest university to train the job coaches on strategies to teach vocational skills to adults with severe disabilities. The employer selected these particular job coaches because they were perceived as being some of the most skilled and motivated to learn new strategies relative to other staff. The employer believed that these characteristics would increase the likelihood that the selected job coaches would master the new strategies, implement the new strategies when working with adults with severe disabilities, and share the strategies with other job coaches. All trainings took place in a university classroom; all probes took place in the same university classroom, an alternate classroom, or a small break room.

During training sessions, we used materials for washing a table and rolling silverware. For table washing, we used a spray bottle of cleaning solution, a table, paper towels, and a trash can. For rolling silverware, we used cloth napkins and a knife, fork, and spoon. For all probes, we placed a folded paper grocery bag, two canned foods, two boxed foods, a bag of dried fruit, a bag of pretzels, a blank piece of paper, and a pencil on a table in front of the participants. We also had available to each participant upon request pictures that corresponded with the steps of grocery bagging task.

2.2 Dependent measures and measurement

2.2.1 Dependent measures

Dependent variables were the percentage of steps implemented correctly for three systematic instructional practices: task analysis, simultaneous prompting, and least-to-most prompting. Task analysis involves dividing a behavior into component steps in order to assess and teach the skill (Wong et al., 2015). Simultaneous prompting involves delivering a controlling prompt (i.e., the least intrusive prompt that will ensure correct responding) immediately after the target stimulus (Gibson & Schuster, 1992). For chained tasks (e.g., bagging groceries), the target stimulus for the first step is a cue or task direction to begin the task (e.g., cashier scans grocery items and slides them to the bagger), and the target stimulus for each subsequent step is completion of the previous step (e.g., bagging of previous item). Least-to-most prompting (also referred to as the system of least prompts) involves a hierarchy of prompt levels that is sequenced from least intrusive to most intrusive (Alberto & Schofield, 1979). These practices were selected because (a) each has a strong evidence base for teaching a variety of chained skills (for reviews, see Test et al., 2009 and Wong et al., 2015) including vocational skills in particular (e.g., Maciag, Schuster, Collins, & Cooper, 2001; Parsons et al., 2001); (b) each is versatile and could be applied to any vocational task with discrete steps; and (c) together, the three practices would enable job coaches to analyze vocational skills to identify target behaviors, minimize errors during initial teaching, and systematically fade prompts to build independence.

2.2.2 Measurement

For five participants, observations were video recorded and coded afterwards. The sixth participant did not wish to be video recorded, so members of the research team observed and coded live data. For task analysis, we coded whether each step (a) was a component of the larger skill, (b) was listed sequentially, and (c) was matched to a picture or visual representation. A percentage correct was calculated by dividing the number of steps that met all three criteria by the total number of steps.

For simultaneous prompting, we coded each of the five trials in the probe. The beginning of the first trial was defined as the participant providing a task direction (e.g., “Let’s bag the groceries”). Subsequent trials began immediately after the learner completed the previous step from the task analysis. Implementation steps for simultaneous prompting included gesturing toward the step of the task analysis and delivering a controlling prompt immediately after (i.e., within 1 s) the cue (i.e., completion of previous step) or the initial task direction. Observers identified the topography (i.e., verbal, gestural, model, and/or physical) of the first prompt delivered by the participant during the first trial, and defined it as the controlling prompt for all subsequent trials during that observation. Subsequent steps were dependent on learner response. After correct learner responses, implementation steps included immediate reinforcement and stating what the learner did. After incorrect responses, implementation steps included ignoring the learner response and repeating the trial. Percentage of correct implementation steps was calculated across trials.

For least-to-most prompting, we coded each of the five trials in the probe. The beginning of each trial was defined identically as described above for simultaneous prompting. The first implementation step was to wait 3–5 s for a learner response without providing any prompt. If the learner provided no response or an incorrect response, implementation steps included gesturing to the first step of the task analysis and providing a verbal prompt. If the learner provided a second incorrect response, correct implementation involved providing a model prompt. If the learner provided a third incorrect response, correct implementation involved providing a physical prompt. Whenever the learner provided a correct response, implementation steps included immediate reinforcement and stating what the learner did. Each trial ended with a correct learner response, as this was defined as the cue for the subsequent trial.

2.3 Experimental design

This study includes six single-case design experiments, each with a multiple baseline across behaviors design (Gast, Lloyd, & Ledford, 2014). Participants began the experiment in the baseline phase across three behaviors, and the intervention was introduced in a staggered fashion across behaviors. The first tier was introduced when participants demonstrated stable performance in the first tier. Subsequent tiers were introduced when participants either met criterion (i.e., 100% correct implementation) or data showed a trend toward criterion.

2.4 Observer training and reliability

Observers were trained by the first and second authors through provision of a detailed coding manual, practice with coding videos, and repeated feedback. All observers met a criterion of 95% agreement on a complete observation with the first author before collecting data for the study. A secondary observer viewed the video or live observation for 31% of all observations, balanced across participants and experimental phases. Overall point-by-point interobserver agreement was 96% (range: 80–100%) across observations.

2.5 Procedures

2.5.1 Baseline procedures

During the baseline condition, no training or support was provided to job coaches on the instructional strategy. Job coaches provided instruction in the role play activity based on their previous knowledge and skills.

2.5.2 Group training

During the group training phase, the first and second author – both university faculty members with expertise in instructional strategies for individuals with severe disabilities – provided training to the job coaches. On two occasions when the first or second author was unavailable, the fourth author, a doctoral student in special education with expertise in instructional strategies, assisted with the training. Job coaches were divided into two groups that each attended eight one-hour weekly trainings. Training occurred over 10 weeks, with a two week break in between the fourth and fifth training sessions. One training group was comprised of Jason, Sarah, and two other job coaches (who due to poor attendance, were not included in this study). The second training group was comprised of Heather, Laura, Julie, and Bryan.

Each 1-hour group training session consisted of didactic instruction and practice with feedback. First, we provided didactic instruction. Aided by a PowerPoint presentation, we defined the strategy, outlined each implementation step, described two examples of how one might follow the implementation steps, and modeled how to implement the steps with examples (i.e., rolling silverware or washing a table). Second, we directed the job coaches to practice the skill. For the task analysis, job coaches wrote a task analysis for washing a table. For simultaneous prompting and least-to-most prompting, job coaches practiced implementing these strategies to teach either a fellow job coach or one of the instructors how to roll silverware or wash a table. We provided feedback after each role play by pointing out which implementation steps had been followed correctly, and (when necessary) providing corrective feedback for incorrectly implemented steps by describing and modeling those steps. In most cases we provided feedback to job coaches individually, although sometimes for the sake of efficiency we provided feedback to the entire group when multiple job coaches struggled with the same implementation step(s).

When a new strategy was initially introduced, we focused on didactic instruction for the bulk of the training session, followed by a brief opportunity for practice with feedback. In subsequent training sessions that focused on the same strategy, the same didactic instruction was implemented in abbreviated form, and a greater proportion of the training session consisted of role play. In addition to the group training session, we provided brief verbal feedback to job coaches after each generalization probe. Similar to the feedback during training sessions, this feedback focused on pointing out which implementation steps had been followed correctly, and (when necessary) providing corrective feedback for incorrectly implemented steps by describing and modeling those steps.

2.5.3 Coaching

Job coaches who did not reach 100% fidelity on all three skills by the end of the eight weeks received a 5–10 min coaching session focused on the grocery bagging task on the last day of training. We modeled correct use of each strategy for teaching the grocery bagging task through role play with the job coach, switched roles so the job coach practiced implementation, and then delivered performance feedback to the job coach by highlighting correct implementation steps and correcting errors. We repeated this sequence until the job coach implemented all steps with 100% fidelity. One job coach (i.e., Bryan) was absent for the last training session, so we were not able to conduct a coaching session as planned.

2.5.4 Generalization probes

Across all conditions, we conducted role play assessments following the training session to determine if the participants would generalize the targeted skill to a task that had not been practiced (i.e., teaching someone to bag groceries). For all role play generalization assessments, participants were asked to pretend that a member of the research team was an adult learner with a severe disability whom they would be teaching to bag groceries. In the first role play assessment, we told the participants to teach us to bag groceries as if the learner had never bagged groceries before and was learning this skill for the first time. After this task direction, we coded staff behavior on (a) creation of a task analysis and (b) implementation of simultaneous prompting. If participants completed a written task analysis, we asked them if there was anything else they needed. If they indicated that they needed pictures of the steps, we provided them with small photographs that could be paired with each step of the task. After participants completed a task analysis once, we told them that they could request it during subsequent probes. For subsequent probes, we kept both previously created task analyses and pictures out of view of participants, but retrieved them if requested. To keep probes of equal difficulty, and to give participants the opportunity to respond to both correct and incorrect learner responses, the learner responded by providing the correct response to the first two prompts, an incorrect response to the third prompt, and then repeated this pattern of responding. We terminated the probe after the participant had implemented five trials, because five trials provided an adequate sample to gauge fidelity (Brock & Carter, 2015), and allowing probes to continue during the baseline condition would have likely resulted in lengthy probes with increased participant frustration.

Next, we asked participants to pretend that they were no longer teaching the learner to bag groceries for the first time, but that they had been working on this skill for several days. After this task direction, we coded staff behavior on implementation of least-to-most prompting. Again, we standardized learner responses to make probes of equal difficulty and to give participants the opportunity to respond to both correct and incorrect responses. This time, the learner responded by providing a correct response in the first trial; an incorrect and then correct response in the second trial; an incorrect, incorrect, and then correct response in the third trial; an incorrect, incorrect, incorrect, and then correct response in the fourth trial; and a correct response in the fifth trial. Again, we terminated the probe after the participant had implemented five trials.

2.6 Social validity

At the end of the training, participants rated their own perceived proficiency of the targeted evidence-based practices on 5-point scale (1 = Novice; 2 =Beginner; 3 = Competent; 4 = Proficient; 5 = Expert).

3 Results

3.1 Implementation of systematic instructional procedures

Six multiple baseline experiments were conducted across three sets of behaviors, allowing for 18 opportunities to demonstrate experimental effects. Through visual analysis of data from generalization probes (see Figs. 1–6), we identified 15 experimental effects out of 18 opportunities. Below, we describe visual analysis of all opportunities by focusing on the trend, level, and variability of the data.

Jason acquired implementation fidelity of all three practices rapidly, and three experimental effects can be detected through visual analysis (see Fig. 1). For task analysis, his baseline performance was stable at 0%. A change in level was delayed until after the first intervention probe, but then increased to 100% and maintained at this level. For simultaneous prompting, baseline data were somewhat variable with a slight upward trend. After the introduction of group training, the level of the data immediately increased to 100% and maintained for the duration of the study. For least-to-most prompting, Jason’s baseline data were stable at 0%. After the introduction of group training, the level of the data immediately increased to 100% and maintained for the duration of the study. Because Jason performed all three practices with 100% fidelity during the group training condition, the coaching condition was not required.

For Heather, three experimental effects can be detected through visual analysis (see Fig. 2). For task analysis, her performance was stable at 0% in baseline. A change in level was delayed until after the first intervention probe, but then increased to 100% and maintained at this level. For simultaneous prompting, baseline data were somewhat variable with a slight downward trend. After introduction of group training, the level of the data immediately increased to 100%, but data were somewhat variable for the rest of the group training condition (range: 43–100%); however, even with this variability there was no overlap between phases. For least-to-most prompting, Heather’s baseline data were somewhat variable but had no clear upward or downward trend. After the introduction of group training, the level of the data immediately increased to 90%, and trended upward to 100%. Because Heather performed all three practices with 100% fidelity during the group training condition, the coaching condition was not required.

For Laura, three experimental effects can be detected through visual analysis (see Fig. 3). For task analysis, her baseline performance was stable at 0%. A change in level was delayed until after the first intervention probe, but then increased to 100% and maintained at this level. For simultaneous prompting, baseline data were somewhat variable with a slight upward trend. After introduction of group training, the level of the data immediately increased to 89%, but data were somewhat variable for the rest of the group training condition (range: 58–100%). Two of the eight intervention data points (29%) overlapped with baseline data. For least-to-most prompting, Laura’s data were somewhat variable, but had no clear upward or downward trend. After introduction of group training, the level of the data immediately increased to 72%, and ranged between 72% and 100% for the rest of the group training condition. There was no overlap between conditions. Because Laura’s last data point in the group training phase was not 100%, coaching was delivered. She implemented 100% of steps with fidelity for all practices after coaching.

For Julie, only two experimental effects can be detected through visual analysis (see Fig. 4). For task analysis, her baseline performance was stable at 0%. A change in level was delayed until after the first intervention probe, but then increased to 100% and maintained at this level. For simultaneous prompting, baseline data had an upward trend. After the introduction of group training, there was an immediate drop in level followed by a sharp upward trend, and a drop in fidelity in the last probe of the condition. Given the instability of the baseline data, and the irregular pattern in the intervention condition, an experimental effect cannot be detected through visual analysis. For least-to-most prompting, data were somewhat variable with no clear trend in baseline. After the introduction of group training, the level of the data increased immediately to 77% and trended upward to 100%. Because Julie’s last data point in the group training condition was less than 100% for simultaneous prompting, coaching was delivered. She implemented 100% of steps with fidelity for all practices after coaching.

For Sarah, three experimental effects can be detected through visual analysis (see Fig. 5). For task analysis, her baseline performance was stable at 0%. A change in level was delayed until after the first intervention probe, but then increased to 100% and maintained at this level. For simultaneous prompting, baseline data were variable. After the introduction of group training, the level of data increased immediately and variability decreased, with no overlap of data between conditions. For least-to-most prompting, data initially trended downward and then trended upward. After the introduction of the group training, the level of data increased and maintained at 93%. Because Sarah did not implement 100% of steps with fidelity during the group training condition, coaching was delivered. She implemented 100% of steps correctly for all practices after coaching.

For Bryan, only one experimental effect can be detected through visual analysis (see Fig. 6). For task analysis, his baseline performance was stable at 0%. A change in level was delayed until after the first intervention probe, but then increased to 100% and maintained at this level. For simultaneous prompting, baseline data were variable with no clear upward or downward trend. After the introduction of group training, the level of the data dropped, but then trended upward. Because of the variability in the baseline phase and drop in level at the beginning of the group training condition, an effect cannot be detected through visual analysis. For least to most prompting, baseline data were initially variable, and then trended upward. Data continued to trend upward after the introduction of group training. Because the upward trend began before the change between conditions, an effect cannot be detected through visual analysis. Bryan was not present for the final training session. Therefore, we were unable to complete a coaching session despite him not implementing all steps correctly for least-to-most prompting.

3.2 Social validity

When asked to report their perception of their own competence with task analysis, simultaneous prompting, and least-to-most prompting at the end of the training on a 5-point scale, Jason responded that he felt like an expert (5), Heather responded that she felt competent (3), Laura responded that she felt proficient (4), Sarah responded that she felt like a beginner (2), and Bryan responded that he felt competent (3). Julie did not select an individual choice, but wrote that some days she felt like an expert, and on other days she felt like she struggled.

4 Discussion

Despite playing a pivotal role in the job seeking process for individuals with severe disabilities, many job coaches have little or no formal training in evidence-based instructional practices. Although there is evidence that job coaches who implement evidence-based practices can improve outcomes for adults with severe disabilities, there is very little known about how to best train job coaches to implement these practices with fidelity. In this pilot study, we trained six job coaches to design task analyses and implement simultaneous and least-to-most prompting procedures. The training featured description of implementations steps, modeling, and role play with performance feedback — three training strategies that have been validated for training school-based staff. All participants made progress toward correct implementation with group training, although four participants required follow-up coaching in order to implement all steps for all three practices with fidelity. Experimental effects were demonstrated for all six participants, and replicated across practices for four participants. These findings extend the research base for training job coaches in a number of ways.

First, a training package that features description, modeling, and performance feedback on implementation steps can enable job coaches to implement evidence-based instructional practices with fidelity. This finding is an extension of the broader staff training literature in which researchers have reported similar findings when testing these training strategies with school-based paraprofessionals (Brock & Carter, 2013), teachers (Fallon, Collier-Meek, Maggin, Sanetti, & Johnson, 2015), and early childhood providers (Snyder et al., 2012) who serve children and young adults with disabilities. Among these strategies, performance feedback is the most widely replicated and validated in the research literature (Fallon et al., 2015). Given that job coaches in this study tended to acquire correct implementation over time, performance feedback may have played a critical role. Although initial description and modeling of implementation steps were important, most participants only achieved correct implementation after repeated practice with performance feedback. Based on our findings and the validation of these training strategies in the broader literature, we recommend that these three training strategies be used when training job coaches to implement instructional practices, with an emphasis on repeated practice with performance feedback.

Second, job coaches struggled to generalize procedures from the training session to the generalization probe. In particular, no participant created a task analysis during the grocery bagging task without explicit verbal feedback to do so after the first intervention probe. Although they just attended a training during which they heard a rationale for why written task analyses are important, observed examples of how to write task analyses, and practiced writing their own with feedback from trainers, the participants did not connect the training with the probe. Instead, they provided instruction just as they had during the baseline condition. This is concerning, because if job coaches struggle to generalize a practice from a training session to a role play assessment immediately after the training session, it seems unlikely that they would generalize from a training session to an authentic setting in which they deliver instruction to job seekers with severe disabilities. Researchers have reported similar findings when measuring the degree to which paraprofessionals generalize instructional and support strategies in a workshop to a classroom setting (Brock & Carter, 2015; Hall, Grundon, Pope, & Romero, 2010), and the degree to which residential staff implemented basic behavioral techniques with adults with disabilities after extensive training (Smith, Parker, Taubman, & Lovaas, 1992). Although the initial lack of generalization is concerning, our findings also suggest that brief verbal feedback can be a powerful tool to promote generalization. After only verbal feedback from trainers—essentially communicating the expectation for participants to generalize the use of a written task analysis—all participants were successful with this practice. These findings are consistent with the idea that generalization is a ubiquitous concept that applies to all learners. A “train and hope” approach is unlikely to promote generalization (Stokes & Baer, 1977), but brief verbal prompts can make a strong impact.

Another concern with generalization is that once job coaches implement a practice correctly to teach a job seeker to perform one job skill, they may not be able to generalize these implementation steps to a different job skill or a different job seeker. During training sessions for simultaneous and least-to-most prompting, all participants performed all implementation steps correctly on a practice target (i.e., teaching someone to roll silverware). We continued to practice and deliver performance feedback to ensure correct implementation. However, very few quickly generalized all implementation steps to the generalization target (i.e., teaching someone to bag groceries), and four participants did not master all implementation steps on the generalization probe until receiving coaching the focused directly on the grocery bagging skill. This suggests that even if job coaches successfully implement an instructional practice for one target skill, they still may not be able to correctly implement all steps when targeting a different skill or working with a different job seeker. Based on our findings combined with findings from previous studies, we recommend that administrators and trainers who oversee job coaches prioritize performance feedback in authentic settings to promote generalization of strategies targeted in training settings. Furthermore, we recommend that performance feedback continue until job coaches demonstrate correct implementation of the practice across skills and learners.

Third, relative to one another, job coaches in this study performed very differently in the baseline condition, acquired implementation fidelity at very different rates during training, and required different intensities of training to master all implementation steps. For example, one participant acquired and maintained correct implementation of all practices within two training sessions, and could have been successful without the entire group training package. A second participant implemented all practices correctly after only group training, but did not acquire correct implementation for all practices until the final training session. The other four participants made progress with group training, but required individualized coaching that focused specifically on the generalization task in order to achieve correct implementation of all steps. These data patterns raise questions about how staff training opportunities are designed. In the broader staff training literature, training is most often delivered in a one-to-one coaching format, and repeated consultation over time (Kretlow & Barthomew, 2010). Although researchers have demonstrated that this format may be conducive to effective staff training, providing ongoing individualized coaching to all job coaches is neither practical nor efficient (Russo, 2004). This study provides initial evidence that group training that includes promising staff training methods can enable at least some job coaches to implement practices with fidelity, and that other job coaches make considerable progress and are able to meet a training criterion with a brief one-time coaching as a follow-up to the group training. Based on these findings, we recommend that those who design job coaching trainings view coaching as a targeted tool to complement group training. By monitoring implementation fidelity over time, trainers can utilize highly effective but resource-intensive coaching when it is truly needed.

4.1 Limitations and directions for future research

Several limitations of this study suggest possible avenues for future research. First, we measured implementation of evidence-based practices in a role play situation. It is unclear if participants would generalize implementation to actual job seekers with severe disabilities, or how implementation of these practices would impact job seeker outcomes. In future studies, researchers should measure implementation in authentic settings and collect outcomes for job seekers with severe disabilities. Second, experimental effects were not demonstrated for all participants for all opportunities. Although all participants made progress toward correct implementation, this progress could not be attributed to the training in three cases. Third, we chose to focus on three specific evidence-based practices—task analysis, simultaneous prompting, and least-to-most prompting. However, researchers have demonstrated that a number of other practices (e.g., constant time delay, video modeling, and self-monitoring) can improve vocational skills for individuals with severe disabilities (Cannella-Malone & Schaefer, 2015). In future studies, researchers should investigate how to best promote implementation fidelity of these and other instructional practices. Fourth, for some baseline phases, we collected relatively few data points. We made this decision because we had limited time to begin initial training, and because in the case of task analysis data were consistently at zero for all participants and would be unlikely to change with additional probes. Furthermore, repeated probes would have likely increased participant frustration. In future studies, researchers could demonstrate more robust differences between conditions by collecting more data points in each baseline condition. Fifth, this study included a small number of participants who might not be representative of all job coaches, especially given that they their employer perceived them to be more skilled and motivated than other employees. In future studies, researchers might use random sampling methods to obtain a sample that more closely mirrors the larger population. Sixth, we only collected data soon after participants had attended training sessions, making it unclear whether participants would maintain these skills over longer periods of time. In future studies, researchers might include maintenance probes after all training is complete. Sixth, we focused only on systematic instructional strategies in this study, and there are a number of other skills job coaches would need to be optimally effective. In future studies, researchers might focus on other critical job coaching skills such as assessment, job customization, and cultivation of natural supports. Lastly, and perhaps most importantly, this is one of the first studies on this topic. Therefore, findings and recommendations from this pilot study should be interpreted with caution until they are replicated across studies and research groups.

5 Conclusion

Job coaches provide critical services for job seekers with severe disabilities, particularly when they provide initial on-the-job training. Surprisingly, we know very little about how to prepare job coaches to use evidence-based instructional practices that give job seekers with severe disabilities the best chance of learning to perform vocational tasks independently. This study provides preliminary evidence that some of the same training strategies found to be effective for training school-based staff may also be effective for job coaches. Given the pivotal importance of job coaching services in the lives of adults with severe disabilities, they deserve to be taught new job skills using proven strategies by skilled job coaches. Furthermore, it is also unfair to ask job coaches to deliver services without equipping them with evidence-based strategies. The strategies evaluated in this study might contribute to promoting implementation of evidence-based instructional practices, improving the efficacy and confidence of job coaches, and promoting improved outcomes for adults with disabilities whom they serve.

Conflict of interest

The authors have no conflict of interest to report.

References

Alberto

P. S.

, & Schofield

(1979). An instructional interaction pattern for the severely handicapped. Teaching Exceptional Children, 12, 16–19. doi: 10.1177/004005997901200105

Brock

M. E.

, & Carter

E. W.

(2013). A systematic review of paraprofessional-delivered instruction to improve outcomes for students with intellectual and developmental disabilities. Research and Practice for Persons with Severe Disabilities, 38, 211–221. doi: 10.1177/154079691303800401

Burns

, Catty

, Becker

, Drake

R. E.

, Fioritti

, Knapp

, ... Wiermsa

(2007). The effectiveness of supported employment for people with severe mental illness: A randomised controlled trial. The Lancet, 370, 1146–1152. doi: 10.1016/S0140-6736(07)61516-5

Cannella-Malone

H. I.

, & Schaefer

J. M.

(2015). A review of research on teaching people with significant disabilities vocational skills. Career Development and Transition for Exceptional Individuals. Advance online publication. doi: 10.1177/2165143415583498

Carter

E. W.

, O’Rourke

, Sisco

L. G.

, & Pelsue

(2009). Knowledge, responsibilities, and training needs of paraprofessionals in elementary and secondary schools. Remedial and Special Education, 30, 344–359. doi: 10.1177/0741932508324399

Dib

, & Sturmey

(2007). Reducing student stereotypy by improving teachers’ implementation of discrete-trial teaching. Journal of Applied Behavior Analysis, 40, 339–343. doi: 10.1901/jaba.2007.52-06

Fallon

L. M.

, Collier-Meek

M. A.

, Maggin

D. M.

, Sanetti

L. M.

, & Johnson

A. H.

(2015). Is performance feedback for educators an evidence-based practice? A systematic review and evaluation based on single-case research. Exceptional Children, 81, 227–246. doi: 10.1177/0014402914551738

Gast

D. L.

, Lloyd

B. L.

& Ledford

J. R.

(2014). Multiple ine and multiple probe designs. In Gast

D. L.

& Ledford

J. R.

(Eds.), Single subject research methodology in behavioral sciences (pp. 251–296). New York, NY: Routledge.

Gibson

, & Schuster

(1992). The use of simultaneous prompting for teaching expressive word recognition to preschool children. Topics in Early Childhood Education, 12, 247–267. doi: 10.1177/027112149201200208

10.

Gilligan

K. T.

, Luiselli

J. K.

, & Pace

G. M.

(2007). Training paraprofessional staff to implement discrete trial instruction: Evaluation of a practical performance feedback intervention. The Behavior Therapist, 30, 63–66.

11.

Guess

, Benson

H. A.

, & Siegel-Causey

(2008). Concepts and issues related to choice making and autonomy among persons with severe disabilities. Research and Practice for Persons with Severe Disabilities, 33, 75–81. doi: 10.2511/rpsd.33.1-2.75

12.

Hall

A. C.

, Bose

, Winsor

J. E.

, & Migliore

(2014). From research to practice: Knowledge translation in job development. Inclusion, 2, 65–79. doi: 10.1352/2326-6988-2.1.65

13.

Hall

L. J.

, & Macvean

M. L.

(1997). Increases in the communication behaviors of students with cerebral palsy as a result of feedback to, and the selection of goals by, paraprofessionals. Behavior Change, 14, 174–184.

14.

Hall

L. J.

, Grundon

G. S.

, Pope

, & Romero

A. B.

(2010). Training paraprofessionals to use behavioral strategies when educating learners with autism spectrum disorders across environments. Behavioral Interventions, 25, 37–51. doi: 10.1002/bin.294

15.

Kessler Foundation and National Organization of Disability (2010). Survey of Employment of Americans with Disabilities: Executive Summary. Retrieved from: http://nod.org/assets/downloads/2010_Survey_Americans_Disabilities_Employment_executive_summary.pdf

16.

Kretlow

A. G.

, & Bartholomew

C. C.

(2010). Using coaching to improve the fidelity of evidence-based practices: A review of studies. Teacher Education and Special Education, 33, 279–299. doi: 10.1177/0888406410371643

17.

Leblanc

M. P.

, Ricciardi

J. N.

, & Luiselli

J. K.

(2005). Improving discrete trial instruction by paraprofessional staff through an abbreviated performance feedback intervention. Education and Treatment of Children, 28, 76–82.

18.

Licciardello

C. C.

, Harchik

A. E.

, & Luiselli

J. K.

(2008). Social skills intervention for children with autism during interactive play at a public elementary school. Education and Treatment of Children, 31, 27–37.

19.

Maciag

K. G.

, Schuster

J. W.

, Collins

B. C.

, & Cooper

J. T.

(2000). Training adults with moderate and severe mental retardation in a vocational skill using a simultaneous prompting procedure. Education and Training in Mental Retardation and Developmental Disabilities, 35, 306–316.

20.

Martella

R. C.

, Martella

, Macfarlane

C. A.

, & Young

K. R.

(1993). Improving the classroom behaviour of a student with severe disabilities via paraprofessional training. BC Journal of Special Education, 17, 33–44.

21.

McDonnell

, Johnson

J. W.

, Polychronis

, & Risen

(2002). Effects of embedded instruction on students with moderate disabilities enrolled in general education classes. Education and Training in Mental Retardation and Developmental Disabilities, 37, 363–377.

22.

Migliore

, Hall

, Butterworth

, & Winsor

(2010). Job development: What do job developers really do? A study on job development practices. Research and Practice for Persons with Severe Disabilities, 35, 15–23. doi: 10.2511/rpsd.35.1-2.15

23.

Newman

, Wagner

, Knokey

, Marder

, Nagle

, Shaver

, …& Schwarting

The post-high school outcomes of young adults with disabilities up to 8 years after high school. A report from the National Longitudinal Transition Study-2 (NLTS2). Menlo Park, CA: SRI International.

24.

Parsons

M. B.

, Reid

D. H.

, Green

C. W.

, & Browning

L. B.

(2001). Reducing job coach assistance for supported workers with severe multiple disabilities: An alternative off-site/on-site model. Research in Developmental Disabilities, 22, 151–164. doi: 10.1016/s0891-4222(01)00064-6

25.

Rogan

P. M.

, & Held

(1999). Paraprofessionals in job coach roles. Research and Practice for Persons with Severe Disabilities, 24, 273–280. doi: 10.2511/rpsd.24.4.273

26.

Smith

, Parker

, Taubman

, & Lovaas

O. I.

(1992). Transfer of staff training from workshops to group homes: A failure to generalize across settings. Research in Developmental Disabilities, 13, 57–71. doi: 10.1016/0891-4222(92)90040-D

27.

Stokes

T. F.

, & Baer

D. M.

(1977). An implicit technology of generalization. Journal of Applied Behavior Analysis, 10, 349–267.

28.

Test

D. W.

, Mazzotti

V. L.

, Mustian

A. L.

, Fowler

C. H.

, Kortering

, & Kohler

(2009). Evidence-based secondary transition predictors for improving postschool outcomes for students with disabilities. Career Development for Exceptional Individuals, 32, 160–181. doi: 10.1177/0885728809346960

29.

Towery

, Parsons

M. B.

, & Reid

D. H.

(2014). Increasing independence within adult services: A program for reducing staff completion of daily routines for consumers with developmental disabilities. Behavior Analysis in Practice, 7, 61–69. doi: 10.1007/s40617-014-0013-4

30.

U.S. Department of Labor. (2015). Economic Picture of the disability community project. Retrieved from http://www.dol.gov/odep/topics/DisabilityEmploymentStatistics.htm

31.

Wehman

, Brooke

A. M.

, Ham

, Schall

, McDonough

, ... Avellone

(2016). Employment for adults with autism spectrum disorders: A retrospective review of a customized employment approach. Research in developmental disabilities, 53, 61–72. doi: 10.1016/j.ridd.2016.01.015

32.

Wehman

, & Kregel

(1988). Supported competitive employment for individuals with autism and severe retardation: Two case studies. Focus on Autistic Behavior, 33, 1–13. doi: 10.1177/108835768800300301

33.

Wehman

, Lau

, Molinelli

, Brooke

, Thompson

, Moore

, & West

(2012). Supported employment for young adults with autism spectrum disorder: Preliminary data. Research and Practice for Persons with Severe Disabilities, 37, 160–169. doi: 10.2511/027494812804153606

34.

Wehman

P. H.

, Schall

C. M.

, McDonough

, Kregel

, Brooke

, Molinelli

, ... Thiss

(2014). Competitive employment for youth with autism spectrum disorders: Early results from a randomized clinical trial. Journal of Autism and Developmental Disorders, 44, 487–500. doi: 10.1007/s10803-013-1892-x

35.

Wehman

, Targett

, West

, & Kregel

(2005). Productive work and employment for persons with traumatic brain injury: What have we learned after 20 years? The Journal of Head Trauma Rehabilitation, 20, 115–127. doi: 10.1097/00001199-200503000-00001

36.

Wehmeyer

M. L.

, & Bolding

(2001). Enhanced self-determination of adults with intellectual disability as an outcome of moving to community-based work or living environments. Journal of Intellectual Disability Research, 45, 371–383. doi: 10.1046/j.1365-2788.2001.00342.x

37.

Wong

, Odom

S. L.

, Hume

K. A.

, Cox

A. W.

, Fettig

, Kucharczyk

, ... Shultz

T. R.

(2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: A comprehensive review. Journal of Autism and Developmental Disabilities. Advance online publication. doi: 10.1007/s10803-014-2351-z