The generalized impacts of systematic instruction strategy training on the instructional staff of a vocational training center

Abstract

BACKGROUND:

Vocational training centers improve outcomes for people with intellectual and developmental disabilities (IDD), but employment rates remain low. Additionally, vocational direct support professionals (DSPs) often lack the training to implement effective instruction and the administrative procedures needed for change.

OBJECTIVE:

This study examines the effect of behavioral skills training (BST) on the maintained procedural fidelity of evidence-based practice (EBP) instruction in direct support providers with clients who have moderate to extensive support needs (M-ESN).

METHOD:

Three instructor-identified EBPs for teaching individuals with IDD were selected for professional development (PD): most-to-least prompting, time delay procedures, and system of least prompts. Three single case multiple probe across behaviors experiments evaluated the efficacy of the researcher-delivered BST instruction model on trainee accuracy as assessed by trained center administrators.

RESULTS:

A functional relation between the BST instruction and generalized DSP implementation was indicated, despite challenges and limitations related to COVID-19. Social validity measures found the goals, procedures, and outcomes were socially significant, and the intervention was practical and effective.

CONCLUSION:

Practices in vocational training should reflect the high standards for systematic instruction in the K-12 setting. EPB BST training with administration progress monitoring can improve DSPs’ instruction with clients, thus improving learning outcomes for adults with M-ESN.

Keywords

Systematic instruction intellectual and developmental disability vocational education direct support professionals behavior skills training moderate to extensive support needs

1 Introduction

Vocational training centers for adults with intellectual and developmental disabilities (IDD) exist throughout the United States with the ultimate goal of competitive integrated employment (e.g., Employment First, Office of Disability Employment Policy, 2012). These centers offer services such as mentoring, education, training, technical assistance, funds, and impact evaluations, yet poor outcomes continue for adults with IDD. In 2022, the Bureau of Labor Statistics reported 17.9% of individuals with a disability were employed, compared to 63.7% of individuals without a disability. Nationally, adults with disabilities who were employed worked fewer hours and received less pay than those without disabilities (National Core Indicators, 2018). In a moment when regional unemployment rates are at historic lows, and wages are steadily increasing in the recovery from the COVID-19 pandemic (Georgia Department of Labor, 2022), individuals with disabilities deserve shared access to this prosperity. Systemic change within this service-delivery model is needed to improve the disparity issues affecting individuals with disabilities (Winsor et al., 2019). This need is evident in the change of percentage over time concerning adults with moderate to extensive support needs (M-ESN) who attend vocational training centers, which has increased from 39% in 2010 to 52% in 2017.

At many vocational centers, the staff working with adults with M-ESN, often titled Direct Support Providers (DSPs), do not have sufficient training to teach individuals with these particular needs (Johnson et al., 2021). The minimal educational requirement for employment of most DSP positions nationwide is a high school diploma or its equivalent (Hall et al., 2018; Johnson et al., 2021). Staff training in centers typically occurs on the job, is reactionary in nature, and situation-specific (Johnson et al., 2021); thus, there is low generalizability. For this reason, there is a recognized gap in services along the transition spectrum provided to individuals with IDD (Brock et al., 2016; Gormley et al., 2020; Wenzel et al., 2021).

Training for DSPs in evidence-based instructional strategies for individuals with M-ESN is needed to bridge this service gap and improve outcomes for adults with IDD. Systematic instruction includes response prompting such as most-to-least, time delay, and system of least prompts have a strong evidence based on increasing target skills for individuals with M-ESN (Spooner et al., 2011). Per Collins (2022), most-to-least prompting is an EBP for learners with moderate to severe support needs that follows a hierarchy of prompts from most intrusive (e.g., a hand-under-hand prompt) to least intrusive (e.g., an independent response). Constant time delay is a strategy designed to provide a consistent delay (e.g., 3 seconds) between the controlling prompt and a response in order to teach or test the target response. The system of least prompts is a strategy in which the instructor provides the learner only the least intrusive amount of support needed to complete the task, progressing along a least-to-most prompting hierarchy. With each, as an individual begins to demonstrate closer approximations toward the desired behavior, the instructor fades the provided prompt, thus promoting increased independence in the task.

DSPs working with adults with M-ESN often struggle to prepare these individuals to complete vocational skills independently (Gormley et al., 2020; Hall et al., 2018). Brock and colleagues (2016) measured the effectiveness of training six job coaches working with individuals with M-ESN in a group professional development (PD) setting on the use of systematic instruction strategies (e.g., task analysis, simultaneous prompting, and system of least prompting), then measured the fidelity with which those staff taught each other to apply the strategy learned. Although they found a functional relation across participants (staff members) and behaviors, their results did not measure generalization to the workplace setting with adults with M-ESN, nor are there many replications of these findings (e.g., Gormley et al., 2020; Wenzel et al., 2021). A limitation identified by Brock et al. (2016) was the concern that strategies were not used after group PD sessions. This study targeted their concern by observing DSPs after PD sessions in the field.

There is a significant amount of research in the area of preparing paraprofessionals to work with school-aged students, but little regarding the significance of educating staff (such as DSPs) to use evidence-based practice (EBP) instructional strategies with adults with M-ESN (Brock et al., 2016). Hall et al. (2018) emphasized the importance of ongoing training and PD for community rehabilitation agency staff so they may implement best practices to improve inclusive employment opportunities for individuals with IDD. Behavioral Skills Training (BST) is an instructional model that can be easily implemented with DSPs who work with adults with M-ESN. BST consists of (a) direction instruction (typically verbal), (b) modeling of the target behavior, (c) rehearsal or role-play of the target behavior, and (d) feedback on progress (DeGennaro Reed et al., 2018). DiGennaro Reed and colleagues (2018) describe this approach as one of the most effective teaching packages for preparing a diverse group of trainees. Davenport and colleagues (2019) found BST effective for training three teachers to implement a reading strategy with elementary students with fidelity; however, they did not investigate whether the teachers generalized the strategy by observing their behavior with students.

In this study, an informal discussion and needs survey were completed with the staff and administration of a vocational training center for adults with M-ESN in the southeastern United States. Results from the conversation and survey indicated that their clients struggled with skills that involve preparing, assembling, and organizing materials. In addition, they reported their clients needed to demonstrate significant progress in the area of making choices, following multiple steps, and job readiness skills. Response prompting such as most-to-least, time-delay, and system of least prompts are all EBPs that are effective for teaching individuals with IDD how to complete these types of tasks or demonstrate these skills (Collins, 2022). The skills described by DSPs provided within the needs survey were incorporated into the training of participants.

This study seeks to extend the literature in BST by using this method to train DSPs at a vocational training center for adults with M-ESN in the use of three evidence-based response prompting strategies and then evaluate DSP implementation fidelity with their clients. Two research questions were developed to examine the generalization of the training of identified EBPs:

What is the effect of a group EBP BST professional development package on DSPs’ post-intervention implementation fidelity with clients at a vocational training center?

What are the perceived impacts of the intervention on DSPs, administrators, and the skill acquisition of clients with M-ESN?

2 Methods

2.1 Setting

This study occurred at a vocational training center in the southeastern United States. Its mission was to prepare adults with M-ESN within their community for competitive employment and community integration. The researchers taught three PD sessions, one for each strategy, and each was held in the office area of the training center. This space included a conference table at the center with seating around and a large screen for presentation display. The DSPs implemented their instruction in classrooms at the same location, although conditions varied throughout the study (e.g., number of clients per day/activity, accessible materials, etc.).

2.2 Participants

Eligible participants for this study included DSPs who provided direct services to clients with M-ESN who attended this center. All DSPs at the center were asked to attend the PD sessions by their administrator. The sessions were offered during paid employment hours at the end of their workday, and their job responsibilities were adjusted to enable them to attend. Once Institutional Review Board approval was granted, the researchers recruited participants during the meeting, and the researchers were the instructors for all of the PD sessions. Recruitment yielded three staff who gave informed consent. The first participant, Imani, identified as an African American female who had completed courses toward a bachelor’s degree in early childhood education and had five years of experience working at the center. Jasmine and Sophia both had earned bachelor’s degrees but not in an education-related field. Jasmine identified as an African American female who had worked at the center for less than five years. Sophia identified as an African American female who had worked at the center for over five years. All participants were between 30 and 45 years of age.

2.3 Materials

During training sessions, the instructors used Google Slides presentations for direct instruction. Portable hand washing stations were brought into the conference room during modeling and rehearsal to teach the most-to-least prompting strategy. These stations included a sink with a floor press that, when pressed, pumped water out of the faucet. The stations also had a ledge that held liquid soap and hand towels. These stations were used in the center by clients, as well. To teach time delay, three-by-five index cards were provided to participants to teach functional sight words. To teach a system of least prompts, participants were provided with small sandwich bags containing enough Legos to make a small build—a structure with three to five pieces.

In each PD session, the instructors provided participants with two similar handouts. The first contained a brief description of the strategy being taught. It also included examples of when the strategy might be used. The remainder of the handout was broken down into three steps. The first step instructed participants to create a task analysis of the skill being taught. The second prompted staff to (a) get the client’s attention, (b) wait for a response, then (c) give appropriate directions before starting instruction. The third provided participants with the explicit steps involved in the given strategy. The second handout included a shorter overview of the above as well as space to collect data on the client’s performance.

2.4 Dependent variable and measurement

The dependent variable included correct DSP implementation of the steps in the response prompting procedure task analysis, which included most-to-least prompting, constant time delay, and a system of least prompts (Collins, 2022). The first three steps of each of the three strategies were consistent: Did the DSP (a) get the client’s attention, (b) wait for a response from the client, then (c) give the appropriate directions? Then, the task analyses differed depending on the response prompt type. The DSP observation checklist (see Table 1) began by asking which of the three strategies was being observed, then the observer was forwarded to the appropriate block of questions for the selected strategy section (e.g., most-to-least, constant time delay, or system of least prompts). Once in the appropriate section, the observer was able to continue to complete the checklist related to the necessary steps for the strategy. As indicated in Table 1, if most-to-least prompting was chosen, there were eight total possible points that could be earned on the observation form. If time delay was chosen, there were nine possible points; if a system of least prompts was chosen, there were eight possible points. Because the point values were different, they were converted to a percentage for data analysis.

Table 1
(DSP) Instructor observation checklist

1. Your (observer) name (short answer)

2. Name of instructor you are observing (short answer)

3. Date observed

4. Skill/lesson/task being observed (long answer)

5. Are the students being asked to respond by: (check all that apply):

•Identifying

•Comprehending

•Performing a step

•Not asked to respond

6. Did you observe an opportunity to use a system of LEAST-TO-MOST prompts? (ex. making choices, worksheets, introductions, etc. General comprehension tasks that typically do not need to start with physical prompting.) (Yes or No) - If yes, they were sent to the following set of questions:

•Landing page for System of Least Prompts with the steps and prompt hierarchy.

•Did the instructor do the following (check all that apply):

•Get the client’s attention

•Wait for a response from the client

•Give the appropriate directions

•None of these

•Then, did the instructor do the following (check all that apply):

•Provided prompt from hierarchy beginning with least to most intrusive prompt

•Provided reinforcement or error correction

•Recorded data

•Repeated as necessary

•I did not see any of these steps

•Were these prompts provided in the order listed above from top to bottom (specifically from least intrusive to most)?

•Always

•The majority of the time

•About half of the time

•Less than half of the time

•I did not see any of these steps

•Notes?

7. Did you see an opportunity to use a system of MOST-TO-LEAST prompting? (ex. handwashing, following a recipe, planting, etc. This category is for skills in which learners need a great deal of instruction during initial instruction, zipper, writing letter of alphabet, safety issues.) (Yes or No) - If yes, they were sent to the following set of questions:

•Landing page for Most-to-Least Prompts with the steps and prompt hierarchy.

•Did the instructor do the following (check all that apply):

•Get the client’s attention

•Wait for a response from the client

•Give the appropriate directions

•None of these

•Then, did the instructor do the following (check all that apply):

•Provided prompt from hierarchy beginning with most to least intrusive prompt

•Provided reinforcement or error correction

•Recorded data

•Repeated as necessary

•I did not see any of these steps

•Were the steps indicated above observed in the order listed above from top to bottom (specifically from most intrusive to least)?

•Always

•The majority of the time

•About half of the time

•Less than half of the time

•I did not see any of these steps

•Notes?

8. Did you see the opportunity to use TIME DELAY? (ex. vocabulary, identify objects from a set of choices, etc. This method fades a prompt by dimensions of time.) (Yes or No) - If yes, they were sent to the following set of questions:

•Landing page for Time Delay with the steps listed.

•Did the instructor provide the correct response with a zero-second delay during the FIRST session? (Yes or No)

•Then, did the instructor do the following (check all that apply):

•CONSTANT Time Delay - same delay each successive session until fluency

•PROGRESSIVE Time Delay - increased by small increments until fluency

•Provide reinforcement or error correction

•Recorded data

•Repeated as necessary

•I did not see any of these steps

•Were the steps indicated above in the order above (with constant or progressive time delay) observed in the order listed above?

•Always

•The majority of the time

•About half of the time

•Less than half of the time

•I did not see any of these steps

•Notes?

2.5 Experimental design and procedures

This study included three single-case design experiments, each with a multiple probe across behaviors design (Horner et al., 2005). The three strategies were similar but functionally independent behaviors. Prior to the start of the study, it was determined that once the baseline trend was stable or decreasing, intervention would be introduced. Baseline data were collected on all three instructional strategies as conditions allowed. The first strategy to demonstrate a stable baseline was selected for the first intervention. Ongoing probe data were collected on the remaining strategies throughout the first and second intervention phases. During the intervention phases, once DSPs met the criterion of 100%, the next strategy or intervention phase was introduced. Maintenance data were collected for each.

2.5.1 Baseline

Baseline was conducted after initial agreement to participate was collected, so participants knew a study had begun; however, they were not given any training on specific strategies. The agency supervisor observed teaching sessions randomly throughout the day while DSPs taught clients in their typical manner. If an opportunity was evident in which one of the three instructional strategies would be applicable, the supervisor recorded data related to the task analysis of the strategy on the DSP observation checklist. Training was then provided to all participants via a group PD session on the first strategy that demonstrated a stable baseline for all participants, and the first intervention phase began. Baseline data probes were conducted on the remaining strategies while the first was in intervention.

2.5.2 Group training and intervention

The independent variable (labeled IV in Fig. 1) consisted of the DSP training sessions in the targeted response prompting system along with assessment and performance feedback following the training session. Prior to each condition phase, a 1-hour training session was held. BST, an empirically based method of instruction, was used to train the DSPs (DiGennaro Reed et al., 2018; Tarbox & Granpeesheh, 2015). The BST treatment package included (a) verbal presentation of the EBP, (b) modeling of the strategy using common stimuli (e.g., teaching a hand washing task analysis), (c) rehearsal of the strategy taught using the common stimuli, and (d) performance feedback from the instructors (see Table 2). All staff from the center were invited to participate in each training session, so sessions included both those who gave consent to participate in the study and those who did not. All staff in attendance practiced the task analysis via role-play with other staff. The researchers completed the final assessments at the end of each session and provided performance feedback to the DSPs at the end of the training session. Sessions ended when all participants reached 100% accuracy in their demonstration of the EBP task analysis.

Fig. 1

Results across three multiple probe single case design response prompting strategies, all three participants. Note. IV = independent variable intervention data. Data was collected during the training sessions.

Table 2

Training session components

BST framework	Most-to-least (MTL)	Constant time delay (CTD)	System of least prompts (SLP)
1) Verbal instruction	Presentation slides accompanied verbal instruction and description of the steps of most-to-least and when it is most effective	Presentation slides accompanied verbal instruction and description of the steps of CTD and when it is most effective	Presentation slides accompanied verbal instruction and description of the steps of SLP and when it is most effective
2) Modeling	Used video modeling –handwashing demonstrating MTL	Used video modeling –identifying shapes using CTD	Used video modeling –handwashing demonstrating SLP
3) Rehearsal or role-play	Guided participants in creating a TA of handwashing using provided forms and then rehearsing with a partner using MTL	Guided participants through a role-play and rehearsal of identifying given sight words using CTD	Guided participants in creating a TA of handwashing using provided forms and then rehearsing with a partner using SLP
4) Feedback	Gave participants a Lego kit (with 3 blocks and a final assembly format image) and asked to write a TA for assembling. Feedback was given on the analysis. Then they demonstrated MTL using TA with a partner. Feedback was given on the use of a prompting system	Participants asked to break down “Rounding to the next dollar up between $1 and $10” using the provided data sheet. Feedback was given on the breakdown. Then they demonstrated CTD using their breakdown with another partner. Feedback was given on the use of a prompting system	Gave participants a Lego kit (with 3 blocks and a final assembly format image) and asked to write a TA for assembling. Feedback was given on the analysis. Then they demonstrated SLP using TA with a partner. Feedback was given on the use of a prompting system

Note. BST = Behavioral Skills Training; TA = Task Analysis.

Training sessions were purposefully designed to follow similar procedures in each session. The first and third training sessions focused on most-to-least and then a system of least prompting instruction, respectively. To limit confusion for DSPs and data collection mistakes for administration, training on time delay was given in the second intervention phase to interrupt the similar procedures of the other two.

2.5.3 Post-intervention

At the end of the group PD training, participants were asked to indicate when the response prompting strategy would most likely be used in their instruction of adults with M-ESN. The agency supervisor then attended relevant instruction sessions to observe and collect DSP performance checklist data on the strategy efficacy. The agency supervisor/observer did not provide any instruction or feedback during this phase, continued using a phone app to collect the data during typical daily activities discreetly, and the procedures/conditions were comparable to baseline.

2.5.4 Maintenance

Maintenance probes were conducted following each intervention phase. This phase was designed to evaluate whether the DSPs were able to implement the procedures with fidelity in the authentic setting (while working with clients). The maintenance phase provided data for the study’s primary measure (research question 1). Time-lagged maintenance data were also collected following a pause in data collection of more than two weeks at the end of the study. A single additional data point was used to confirm the general maintenance of the intervention.

2.6 Interobserver agreement and procedural integrity

The agency supervisor and chief executive officer (CEO) were trained in the observation data collection procedures. The primary observer was the DSP supervisor, while the CEO completed interrater reliability. Both were trained to recognize the target strategies, problem-solve the data collection process with the researchers and use the data collection tool with 100% fidelity before the study began. In a 1-hour training session, the researchers used the same BTS model from the study’s intervention to model, rehearse, assess fidelity, and give feedback using practice videos for observations. After the first training, a pilot test of the measure was conducted for two months, the data collection tool was adjusted for ease of use and alignment to the research question, and a retraining was conducted following the same format as the original training session.

The supervisor recorded 22 sessions (28%) for interobserver agreement (IOA) purposes. These videos were rated by the CEO and then reviewed between the supervisor and CEO in their morning staff meetings. A total of 12 sessions (31.6%) were reviewed in the baseline phases and the remainder during the intervention (3 sessions, 10%) and maintenance (16 sessions, 67%) phases. Due to COVID-19 limitations, IOA was not conducted on intervention sessions for most-to-least or time-delay conditions but was confirmed at the final maintenance session. IOA was calculated by the total number of agreements divided by the number of agreements plus the number of disagreements multiplied by 100 (Ledford & Gast, 2018). IOA for steps of the instructional strategy completed with efficacy was 100% across all captured sessions and phases.

Procedural integrity for the intervention training session materials was evaluated by comparing the training materials and norms for systematic instruction in Collins (2022) to the BST framework shown in Table 2. The materials were created by two members of the research team and then reviewed by a third. BST framework procedures were followed with 100% accuracy for all three training sessions.

2.7 Social validity

At the end of data collection, the three participants and the two administrators completed a social validity survey. The participants’ survey consisted of nine questions, whereas the administrators’ survey consisted of 10. Each survey had rating scale questions related to the usefulness, cost-effectiveness, feasibility, appropriateness, and ease of implementation as well as open-ended questions regarding the benefits and drawbacks of the intervention and whether they would use it again.

3 Results

The first research question investigated the effect of a group EBP BST package on DSPs’ post-intervention implementation fidelity with clients at a vocational training center. The effect of the training on the implementation of EBPs as well as several limitations were noted. A functional relation was indicated across all three strategies (most-to-least prompting, time delay, and system of least prompting) despite an extended timeline for the implementation, with several interruptions interfering with the re-establishment of baseline. Across all participants, no decreasing trends were noted in any intervention or maintenance phase across all three strategies, and gains made by the participants were maintained (see Fig. 1).

As an example of extended timeline, the first training (most-to-least prompting) was conducted in the beginning of July, and intervention data were collected the following week. Then, due to staff illnesses and planned vacations, data collection did not continue until mid-September. At that time, probes were conducted to establish a consistent trend toward mastery before beginning the next training session and data collection for the time-delay procedure (second condition). However, due to shifting administration responsibilities related to increased client health and safety measures, data were collected on this strategy over the next eight weeks until a clear level and trend were established at the end of November.

3.1 Research question 1: Fidelity of instruction across strategies

Figure 1 is a visual representation of all three single-case design experiments combined—one experiment for Jasmine (square-shaped data points), Sophia (circle-shaped data points), and Imani (triangle-shaped data points)—each with three demonstrations of an effect across strategies. Separately, each experiment demonstrated a clear effect with maintained results. Combined, the resulting measure of the percentage of all completed steps for each strategy procedure demonstrates consistently low-to-moderate baseline levels, a clear immediacy of effect in six of nine demonstrations, and only a single data point of overlap for three of the nine demonstrations. Intervention and maintenance gains showed limited variability with stable trends and levels across all demonstrations of the effect.

3.2 Strategy one: Most-to-least prompting

Jasmine’s data indicated an initial high level of step completion for most-to-least prompting (63%), but three additional data points in baseline established a mildly stable level ranging from 25–50% efficacy (25%, 50%, and 37%). With the introduction of intervention, Jasmine’s data demonstrated an immediacy of effect but overlapped with her initial high level in baseline (63%). Following the first data point in this phase, the data for Jasmine’s efficacy demonstrated an increasing trend that became stable at 88% at the end of intervention and increased further in the maintenance phase. Sophia’s data showed stability in baseline (38%) followed by an immediacy of effect in intervention given a notable change in level (75%). An increasing trend in intervention was maintained at a stable 88% because the DSP was omitting the EBP procedural step to record data. Imani’s data indicated an initial high level of step completion in baseline (63%), but unlike Jasmine, Imani’s data had a clear decreasing trend. In baseline, the immediacy of effect was established but overlapped with her initial high level in baseline (63%). Imani’s data in intervention matched Jasmine’s increasing data path after achieving 88% step completion for the most-to-least strategy. All participants reached 100% step completion in the maintenance phase.

3.3 Strategy two: Time delay

Jasmine’s baseline data for time delay held a low and relatively stable level (range = 11–33%). With the first intervention data point, Jasmine once again demonstrated overlap between her highest level in baseline. Her second data point demonstrated a sharply increasing trend and maintained at the highest point (100%). Sophia’s data indicated an initial high level of variability (range = 0–60%) in baseline and showed a 33% before the start of intervention. With the start of intervention, Sophia showed a significant immediacy of effect to 100%, which remained relatively stable (range = 89–100%) through the maintenance phase. Imani’s baseline data showed some variability (range = 11–33%) but remained at a low level. Following the start of intervention, Imani’s data demonstrated a significant immediacy of effect to 100% for two data points in intervention and maintained at 100% for one session and 89% for the second.

3.4 Strategy three: System of least prompts

In the system of least prompting strategy, all participants achieved 100% step completion in the intervention phase. Jasmine’s data in baseline was very stable at 50% after some initial variability. At the start of the intervention phase, Jasmine demonstrated 100% step completion across two data points. Sophia’s baseline data were moderately low and stable (range = 25–50%). The intervention data showed an immediacy of effect to 100% step completion and ranged from 88–100% in intervention. Imani’s data in baseline demonstrated early low stability, but with two subsequent baseline confirmations, the data showed an increasing trend. The final baseline data point broke the trend and dropped to the initial baseline level before the start of intervention. Intervention data immediately demonstrated a higher level (range = 88–100%) and an increasing trend. All participants maintained 100% in the maintenance phase.

3.5 Research question 2: Social validity

To address the social validity of increased fidelity of EBP instruction on the skill acquisition of adults with M-ESN, surveys collected data from all administrators and study participants. Results indicate providing systematic instruction strategy training to DSPs was relevant to the participants involved and valuable to administration. The participants reported the training was helpful and enabled them to work more effectively with clients with M-ESN. Administration reported both the clients and DSPs made progress on their individual skills and teaching skills, respectively, during this project, and the procedures taught during this training rendered more progress toward skills mastery for clients than prior strategies used by the DSPs. Also, administration felt the instruction was appropriate for DSPs to learn, cost-effective, and feasible to implement.

All three participants either agreed or strongly agreed DSPs should use the strategies taught during the study PD sessions. All three participants either agreed or strongly agreed each of the three strategies was helpful in working effectively with clients, and clients made progress on targeted skills as a result. Two of the three participants reported they had never collected data on target skills before this training, whereas one reported she had recorded data “once or twice.” After the training, one staff said they now collect data frequently, but two staff report collecting data only once or twice beyond the data collection noted in the strategy procedures. This survey was given approximately two weeks after the data point was collected. When asked if the DSPs were using these strategies exactly as practiced in the training sessions, two reported yes, “most of the time,” whereas one reported, “about half of the time.” All three participants either strongly agreed or agreed the delivery of training was appropriate, cost-effective, and feasible to implement. All would recommend this training to others, stating in an open-ended question, “I believe that it’s incredibly beneficial to know how to deliver the same subject in different styles,” “it was great to see how different techniques can open up opportunities to see how you are doing and what you could do now that you know the steps,” and “it was very informative and helpful when put into use with my students.”

When asked what improvements were noticed in client progress over the course of the study, one participant responded, “with training staff on multiple ways to deliver their lessons, I believe that this has benefited our clients’ progress by enabling them to find a learning style that works best for them.” A second participant commented on the decreasing intrusiveness of prompt levels and noted, “One step [in the system of least prompting strategy] was learned with most prompting (full). Now... modeling is only needed half the time.” A third participant responded that clients are “more vocal about their answers.”

4 Discussion

The purpose of this study was to educate DSPs on three evidence-based prompting strategies through three group BST sessions. Strengthened by three separate experiments but weakened by a delayed baseline for one strategy, each demonstration of the effect of the intervention indicates a clear change despite the limitations. The generalization of the noted effect on environments with client learning was indicated. Analysis of the generalized effect of the training was conducted through administrator-led progress monitoring of DSPs in their daily classroom setting with adults with M-ESN. Participating DSPs demonstrated learning through BST sessions, then instructed their clients with increased and sustained fidelity with little variability in the classroom setting. This research extends Davenport et al.’s (2019) findings that BST is effective in training staff. It also extends the literature by demonstrating BST is effective when used with an academic intervention.

Administrators successfully used an online DSP instructor observation checklist to assess the impacts of the training, although one administrator expressed a preference for paper and pencil data collection. Social validity surveys indicate researchers, administrators, and participants agree the response prompting strategies were easy to operationally define, learn, task analyze, and implement, thus increasing the likelihood these results can be replicated. The current study applied Davenport and colleagues (2019) recommendation that social validity results should confirm behavioral observations of the change (e.g., dependent variable of the current study), resulting in qualitative confirmations of the effect. Additionally, DSPs anecdotally reported the use of the strategies extended beyond the study, further highlighting their value to participants. With the high number of currently employed DSPs (Paraprofessional Healthcare Institute, 2019) and the number of competitive employment instructor positions still growing (Johnson et al., 2021), the development of formal training in EBP instruction can rely on the BST model with center-specific adjustments to respond to the demand. Solutions to training barriers will require subsequent inquiry based on study conditions (e.g., the center involved in this study was motivated to invest in their employees’ instructional development, but this will not be universal). A training series such as this can be used with DSPs to help implement EBPs, practice implementation with clients, collect data on individualized goals, and better support adults with M-ESN (Gilson et al., 2017; Johnson et al., 2021).

In consideration of the COVID-19 pandemic, steps were taken to reduce the burden of this study on all involved at every opportunity while maintaining the integrity of the design. Outcomes of the DSP training indicate short sessions within the current responsibilities of a DSPs’ workday produce desired outcomes, which supports Johnson and colleagues (2021) call for training opportunities within reasonable job expectations.

4.1 Limitations

Limitations of a design with a small number of participants include reduced generalizability of results. This study was designed to be responsive to the shifting needs of one facility using a needs assessment measure. Due to this flexibility, the concurrent nature of baseline data collection in the intended methods for single-case design was disrupted: data collection for the baseline of the three strategies (particularly with the time delay strategy) included variable overlap. This changed the design from a multiple baseline to a delayed multiple baseline, opening the results to a risk of intervening variables. Other limitations include (a) the effects of conducting a study during the COVID-19 pandemic, (b) an extended timeline due to administrators’ inconsistent collection of dependent variable data, and (c) DSPs inconsistent collection of client data without retraining.

4.2 COVID-19 pandemic impacts

Data for this study was initially intended to be collected by the research team; however, due to COVID-19 restrictions, the design was adjusted to rely on personnel employed at the center to collect data and conduct procedural fidelity. Ideally, the design would plan to collect data from the clients directly to measure their growth as opposed to teacher perceptions of growth. Due to COVID-19 restrictions, data collected in this study described a general effect on clients, but future inquiry with direct measures of client performance would be more informative. There were limitations with this implementation due to virus-related attendance issues at the center, as well as temporary closures of the facility. This led to data collection gaps and data sharing inconsistencies. The study results were impacted by attrition, as a fourth participant resigned during the baseline phase of the study. IOA was impacted as data were collected across the majority of phases, but due to COVID-19 disruptions within the center, data were only collected in baseline and maintenance phase for two strategies. Future research using remote data collection should include considerations for “consistent data sharing” to protect from disruptions (Collins, 2022, p. 59). Although the results reflect the realities of applied research during and impacted by a pandemic, replications should include procedures to safeguard experimental control.

4.3 Extended timeline

Due to the COVID-19 limitations described above, there were three major timeline interruptions –each interruption occurred between the establishment or reconfirmation of baseline and the start of intervention. Although no indications suggest the immediacy of effect was caused by any other variables, the demonstration of an established baseline was not as strong (e.g., it is possible the baseline was unstable prior to the intervention for most-to-least and least-to-most strategies and it would not be noted in this data). Further replications are necessary to confirm baseline immediately prior to intervention in order to establish a functional relation.

4.4 Retraining necessary due to DSPs inconsistent data collection

The mastery level was not 100% for this study, although all participants were capable of 100% procedural fidelity for the three strategies. The research team determined a demonstration of near mastery would suffice, given data would identify discrepancies that could be addressed directly. However, given COVID-19 impacts, the study was not extended further to address the incongruencies with data collection (the most consistently missed step across all strategies), thus relying on subsequent projects or training efforts to bridge this gap. Other discrepancies may also be addressed through more sensitive measures in future inquiry (e.g., capturing the point at which DSPs were correctly choosing when to use each strategy without support).

5 Conclusion

The effect of a group EBP BST package was an increase in the percentage of steps completed across three strategies, replicated in three separate experiments (one for each participating DSP). This was designed to progress monitor the fidelity of implementation of EBPs in untrained settings and situations. The perceived effects of the intervention matched the effects measured by the dependent variable. Given the extended timeline of the study, the results of the generalized measure (progress monitoring in classrooms with clients) and maintenance of skills are encouraging for putting these results into practice, as research-to-practice efforts typically require flexibility and decreased experimental control. The implementation of this study relied on whole-group training of DSPs, which is typical and practical for PD in the workforce. Performance was assessed by agency administrators, which is practical for ongoing procedural fidelity beyond the scope of this study and by these researchers and in providing employees with effective performance feedback, which could be used for employment evaluative purposes. Future research should consider the potential for participant misconceptions if training is not provided in ongoing PD sessions as this can limit opportunities for reinforcement and repetition of the primary concepts (e.g., participants referring to the response prompting strategies as “lessons,” as opposed to methods within lessons; participants reference to varied “learning styles,” instead of varied learning needs). This confirms even thorough, needs-responsive training with in vivo assessment may be a poor substitute for a comprehensive educator preparation program. Centers and agencies should also consider how their administrators are involved in the training and assessment of EBP implementation. Results indicate replicating these methods may provide administrators with an increased awareness of PD sessions’ impact.

Despite students with IDD receiving years of transition planning and support in the K-12 setting as mandated by federal law, this legislative oversight and demand for transition services do not continue post-graduation (Gilson et al., 2017). If implementing systematic instruction with fidelity results in greater success for students with M-ESN in K-12 classrooms (e.g., Collins et al., 2022), then the provision of the same for adults with M-ESN is a critical step in producing better post-school outcomes for individuals preparing for the workforce.

Footnotes

Acknowledgments

The authors would like to thank the anonymous reviewers who provided excellent constructive feedback on the structure and clarity of this manuscript.

Conflict of interest

The authors declare they have no conflict of interest.

Ethical considerations

Ethical approval for this study was obtained from Georgia Southern University Institutional Review Board (IRB #21102) on December 22, 2020. Informed consent was obtained for all participants, and a letter of support was received from the agency managers. To protect their privacy, pseudonyms were used for participants throughout the manuscript.

Funding

Internal university grant funding (no grant number assigned) was received in the fiscal year 2022 for a graduate assistant to support this research.

Informed consent

All participants were informed of the risks and benefits, study purpose, and voluntary participation prior to the study being conducted. Informed consent documents were reviewed with the participants, and then they had the opportunity to ask questions before opting to sign the form.

References

Brock,

M. E.

, Cannella-Malone,

H. I.

, Schaefer,

J. M.

, Page,

E. J.

, Andzik,

N. R.

, & Seaman,

R. L.

(2016). Efficacy of training job coaches to implement evidence-based instructional strategies. Journal of Vocational Rehabilitation, 45(3), 351–364. https://doi.org/10.3233/JVR-160835

Bureau of Labor Statistics, (2022, February 24). Persons with a disability: Labor force characteristics – 2021 [Press release]. https://www.bls.gov/news.release/pdf/disabl.pdf

Collins,

B. C.

(2022). Systematic instruction for students with moderate and severe disabilities (2nd ed.). Paul H. Brookes.

Davenport,

C. A.

, Alber-Morgan,

S. R.

, & Konrad,

(2019). Effects of behavioral skills training on teacher implementation of a reading racetrack intervention. Education and Treatment of Children, 42(3), 385–408. https://doi.org/10.1353/etc.2019.0018

DiGennaro Reed,

F. D.

, Blackman,

A. L.

, Erath,

T. G.

, Brand,

, & Novak,

M. D.

(2018). Guidelines for using behavioral skills training to provide teacher support. Teaching Exceptional Children50(6), 373–380. https://doi.org/10.1177/0040059918777241

Georgia Department of Labor. (2022, April 14). Georgia Marks Lowest Unemployment Rate Ever as Wages Increase Statewide. [Press Release]. Accessed April 2022. https://dol.georgia.gov/press-releases/2022-04-14/georgia-marks-lowest-unemployment-rate-ever-wages-increase-statewide

Gilson,

C. B.

, Carter,

E. W.

, & Bigs,

E. E.

(2017). Systematic review of instructional methods to teach employment skills to secondary students with intellectual and developmental disabilities. Research and Practice for Persons with Severe Disabilities, 42(2), 89–107. https://doi.org/10.1177/1540796917698831

Gormley,

, Healy,

, Doherty,

, O’Regan,

, & Grey,

(2020). Staff training in intellectual and developmental disability settings: A scoping review. Journal of Developmental and Physical Disabilities, 32(2), 187–212. https://doi.org/10/1007/s10882-019-09683-3

Hall,

A. C.

, Butterworth,

, Winsor,

, Kramer,

, Nye-Lengerman,

, & Timmons,

(2018). Building an evidence-based, holistic approach to advancing integrated employment. Research and Practice for Persons with Severe Disabilities, 43(3), 207–218. https://doi.org/10.1177/1540796918787503

10.

Horner,

R. H.

, Carr,

E. G.

, Halle,

, McGee,

, Odom,

, & Wolery,

(2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71(2), 165–179. https://doi.org/10.1177/001440290507100203

11.

Johnson,

K. E.

, Bailey,

C. E.

, Weiss,

N. R.

, & Eidelman,

S. M.

(2021). Direct support professionals’ perspectives on workplace support: Underappreciated, overworked, stressed out, and stretched thin. Intellectual and Developmental Disabilities, 99(3), 204–216. https://doi.org/10.1352/1934-9556-59.3.204

12.

Ledford,

J. R.

, & Gast,

D. L.

(2018). Single case research methodology: Applications in special education and behavioral sciences. Routledge.

13.

National Core Indicators (2018). Adult Consumer Survey, 2016–2017. Final report. Human Services Research Institute.

14.

Office of Disability Employment Policy. (2012). ODEP accomplishments: 2009–2012. https://www.dol.gov/agencies/odep/about/history/2012#:∼:text=Administration%20on%20Intellectual%20and%20Developmental,for%20individuals%20with%20significant%20disabilities

15.

Paraprofessional Healthcare Institute (2019). Workforce data center. https://phinational.org/policy-research/workforce-data-center

16.

Spooner,

, Browder,

D. M.

, & Mims,

P. J.

(2011). Evidence-based practices. In D. M. Browder & F. Spooner (Eds.), Teaching students with moderate and severe disabilities (pp. 92-122). Guilford.

17.

Tarbox,

, & Granpeesheh,

(2015). Chapter 4 – principles and procedures of acquisition. In D. Granpeesheh, A. C. Najdowski, J. Tarbox, & J. Kornack (Eds.), Evidence-based treatment for children with autism: The CARD Model (pp. 31-74). https://doi.org/10.1016/C2012-0-07276-6

18.

Wenzel,

J. D.

, Fisher,

, & Brodhead,

M. T.

(2021). Preparing job coaches to implement systematic instructional strategies to teach vocational tasks. Career Development and Transition for Exceptional Individuals. https://doi.org/10.1177/21651434211041918

19.

Winsor,

, Timmons,

, Butterworth,

, Migliore,

, Domin,

, Zalewska,

, & Shepard,

(2019). State Data: The national report on employment services and outcomes. University of Massachusetts Boston, Institute for Community Inclusion. https://scholarworks.umb.edu/cgi/viewcontent.cgi?article=1132&context=ici_pubs