Preliminary data from effects of knowledge translation methods on VR counselors providing pre-ETS to youth with significant disabilities: A brief report

1.1 Overview of Pre-ETS

Passage of the Workforce Innovation and Opportunity Act (WIOA) of 2014 placed an emphasis on increasing student education and preparation for employment. Required in WIOA is the use of 15 percent of federal dollars towards the delivery of five Pre-ETS categories: job exploration counseling, work-based learning, postsecondary education counseling, workplace readiness training, and instruction on self-advocacy. Of note, WIOA also changed eligibility requirements for transition services and specified providing services to in-school youth as young as age 14 (WIOA, 2014). While the expansion of eligibility to Pre-ETS services greatly increases the opportunity for youth with disabilities to be better prepared for employment or postsecondary success through earlier involvement with vocational rehabilitation (VR) (Burgess & Cimera, 2014), other studies reveal potential limitations in VR preparation and collaboration (Carter et al., 2011, 2012; Plottner et al., 2019; Wehman, 2013).

Research demonstrates that interagency collaboration between VR and schools are vital to delivering quality Pre-ETS services to students with disabilities (Luecking & Luecking, 2015). Neubert et al. (2018) surveyed VR counselors who rated Pre-ETS services as important, but rated their preparation to deliver these services as significantly lower. A survey of VR counselors in Tennessee identified concerns about ability and time to collaborate, especially with students in general education classes (Awsumb et al., 2020). Concerns about the severity or visibility of a youth’s disability, and parental expectations were also identified as a minor or moderate barrier. This same survey found that VR counselors reported an understanding of Pre-ETS, but half still indicated they had not received sufficient training on implementation or wanted additional training. Additionally, VR counselors in rural areas of Tennessee identified more limited resources than their counterparts in nonrural locations. From an educator’s perspective, 77.9 percent of surveyed educators in Tennessee agreed that they would like more collaboration with VR (Carter, et al., 2021). This same survey indicated a larger difference in current collaboration between high school and middle school staff, with high school educators more likely to collaborate than middle school educators. Similar to VR counselors, educators identified lack of time, training, and resources as barriers. However, they also included providers only servicing certain types of disabilities. For such reasons, the RRTC on Transition chose to focus a study on the delivery of Pre-ETS services to students with significant disabilities aged 14–16.

2.1 Overview

The purpose of the Transition RRTC Study, “Pre-ETS for Youth with Significant Disabilities 14–16 Years of Age” is to use knowledge translation methods to develop and deliver a toolkit to assist VR counselors (and others) providing pre-employment transition services to students 14 to 16 years of age with significant disabilities. The study has defined students with significant disabilities as individuals who meet the Rehabilitation Act of 1973’s definition of significantly disabled, a person with an impairment that will seriously limit one or more functional capacities relevant to employment and will require long-term supports for competitive integrated employment (Rehabilitation Act, 1973). This three-phased study involves collecting information from key stakeholders for the development of an instructional toolkit.

In Phase 1, the study team gathered information from pre-employment counselors, family members/students, and educators across Virginia (VA) and Kentucky (KY). This phase involved the following: (1) interviews with ten pre-employment counselors in each state on the barriers (challenges) and facilitators (positive aspects) of providing Pre-ETS; and (2) interviews with ten family members and their child (optional), as well as ten special education teachers on the barriers and facilitators of receiving Pre-ETS.

Currently in Phase 2, Transition RRTC staff in both states began providing Pre-ETS services. Research facilitated the Pre-ETS process in each of the school systems to obtain student referrals. This phase involves five students referred by VR, through coordination with the schools, to participate in 12 hours of Pre-ETS services. Those services include job exploration (3 one-hour lessons), workplace readiness (6 one-hour lessons) and a community work-based experience (up to 3 hours). Pre- and post-tests are administered and all materials developed by students are shared with families, VR counselors, and educators. At the conclusion of instruction, study staff briefly interview the stakeholders to collect feedback, which will guide the development of a toolkit, to be tested during Phase 3.

Our research questions are as follows:

Phase 1 (already completed):

Research Question 1. What are the barriers and facilitators to providing services on job exploration, workplace readiness and community work-based learning to students 14–16 with significant disabilities?

Phase 2 (currently completing):

Research Question 2. What are the job exploration, workplace readiness, and community work-based learning resource needs reported by 14–16-year-olds with significant disabilities and their families?

Research Question 3. What are the resource needs for Pre-ETS counselors to facilitate job exploration, workplace readiness and community work-based learning with 14–16 year olds with significant disabilities?

Phase 3 (to be completed):

Research Question 4. Is the targeted knowledge translation (KT) strategy superior to the traditional strategy (control) for acquiring and using evidence-based research knowledge (toolkit)?

2.2 Design

Using the Knowledge-to-Action (KTA) method as described by Graham et al. (2006), we will determine the extent of knowledge transfer. Graham and colleagues state that the three phases (with six components) do not necessarily represent a linear procedure. Phase 1 is Knowledge Inquiry, which has been completed. In this phase, all stakeholders participate in providing feedback during the knowledge inquiry process from representatives of all groups, such as Pre-ETS counselors, teachers, students and their families. VCU and UK study staff contacted families via email, and those interviews with students and families were conducted by phone. Students had the option of being included in the interview (they were never interviewed alone), and they were informed that they did not have to answer any questions. Students also had the option of not participating in the interview at all. Interviews with teachers and Pre-ETS counselors were also conducted by phone. VCU and UK study staff have professional relationships with teachers and Pre-ETS counselors across the two respective states, and emailed those individuals to ask for an interview. All email language, interview protocols, and procedures were approved by the institutional review boards.

For Phase 2, termed Knowledge Synthesis by Graham et al. (2006), a maximum of 100 students will be enrolled in the three courses over years two and three. Pre- and post-tests (4-5 items) are given before and after each one-hour lesson to determine if knowledge and skills increased. Responses are dichotomous (yes/no), with a request to answer 2-3 open-ended questions. Descriptive data will be examined, with a paired samples t-test and measure of magnitude to determine differences in knowledge before and after each training. During this phase, the students and their families will be interviewed about the courses in which they are enrolled. Their teachers (a maximum of 2 per region) and the Pre-ETS counselors (a maximum of 2 per region) will provide feedback as well. We will use constant comparative analysis for the qualitative piece.

3.1 Phase 1

In Phase 1, we conducted interviews to hone in on ways to address barriers and enhance facilitators, as well as the KT strategies that Pre-ETS counselors will prefer (Lambert et al., 2023). Demographic information was also collected to ensure representation from a variety of disability categories, as well as representation from rural, suburban, and urban areas. We used thematic analysis (Nowell et al., 2017) to examine the qualitative data. Two researchers examined and analyzed the data separately to reduce bias. The study team also held monthly project staff meetings to discuss findings. The team developed some themes based on expertise, as well as findings in the literature prior to data collection. Table 1 provides a breakdown of the number of families, teachers, and VR counselors interviewed in each state.

The following paragraphs provide information on the results most relevant in development of the lessons.

Across VR counselors, educators, and families, participants felt that providing an interest-driven community-based experience was lacking (Lambert et al., 2023). Participants wanted students with significant disabilities to be exposed to those experiences as early as possible in order to have consistency of exposure, while giving a variety of possibilities. This emerged as one of the most prominent themes in helping students with significant disabilities in future planning, including choosing coursework throughout high school to expand on their interests. Crucially, it was pointed out how important these community relationships or social capital were in rural areas when it came to finding employment. For VR counselors and educators, early exposure was primarily beneficial when it came to informed choice and preparing students for work through resource identification. For families, early exposure to Pre-ETS would provide students with more time to practice skills required for employment and better alignments with academics.

Lambert and colleagues (2023) found that other themes emerged, as well. Those identified by interview participants included time constraints, emphasis on things other than work for students at this age, difficulty engaging students and parents, the need for more training on Pre-ETS and instruction (especially for students with significant disabilities), transportation, collaboration between stakeholders, and limited business partners in the community. Because of the time in which these interviews were occurring, COVID was also identified as a barrier to Pre-ETS implementation.

In terms of resources interviewees believed would help with implementing Pre-ETS services, VR counselors requested more training on how to deliver content to younger students and specific resources and activities for students with significant disabilities. Educators also shared a desire that VR counselors had more professional development on how to work with students with significant disabilities and that they were able to spend more time in schools working directly with students. Educators wished that they themselves knew more about VR and potential services to better prepare students and families to work with VR after graduation. Families also expressed a desire for VR counselors to be more knowledgeable about specific disabilities. Families also wanted to know more about VR services and different providers. They wanted more collaboration and communication, especially during the summer, from schools and VR counselors.

Based on these interview findings, research staff created a curriculum of job exploration and workplace readiness lessons, as well as a community-based work experience in the hope of addressing identified challenges. Starting with students as early as eighth and ninth grades will allow planning of their high school curriculum and IEP development based on career interests and strengths. The curriculum is also designed to be used in a virtual environment or in person, in addition to including clear explanations of alternatives for specific disabilities (vision impairment or non-verbal, for example). The process includes providing feedback to be shared with the student’s family, teacher or case manager, and VR counselor.

At the time this practice brief was submitted, Phase 2 instruction in job exploration, workplace readiness, and completion of a community-based work experience have been completed at four sites and three ongoing in Virginia, one completed and two ongoing sites in Kentucky. Observations shared from instructors across these sites are the need to make the information relevant and engaging to younger students, who may still have a very abstract understanding of work and work expectations. For example, early lesson activities to identify personal interests and strengths were not as straightforward as one may expect. Students were often reliant on instructors to help them identify both personal interests and strengths. In an attempt to engage students, lessons touched on topics that were age appropriate and of high interest. For example, to help students better understand the concept of personal strengths, comic book characters were used to identify individual character strengths, and how they differed from other comic book characters. Superman’s ability to fly was contrasted with Spiderman’s ability to climb. Though not directly work related, introducing the concept in this manner allowed students to relate material to previous knowledge and interests. From there, instruction focused on how strengths and interests could be pulled from academic experiences and related to work environments. Students were given time and support to record in their workbooks personal interests and strengths. Future lessons continue to build on how these identified strengths may be relevant to future jobs and careers.

Similarly, rather than have students generate workplace preferences based on an abstraction of work, lesson material gave specific examples of different preferences and had students identify their preference. Preferences were edited to add accompanying visuals and audio when relevant based on instructor feedback. Examples of preference choices presented to students were working inside or outside, quiet or loud environments, sedentary or active tasks, working alone or in groups, and routine or variable tasks. Students were shown examples of each preference and given time to relate these preferences to their experience at school, at home, and in the community.

Early on, it was observed that some lesson content, pre- and post-tests, and workbook documentation needed to be simplified. Each round of instruction has resulted in a streamlining of lesson activities, and fill-in-the-blank responses were removed from all pre- and post-tests. Research staff are in the process of implementing a pre-lesson technology assessment and guide to introduce basic editing skills to students for use in their workbooks. Table 2 shows preliminary results from the Virginia pre- and post-tests only (Kentucky data are not yet available).

Pre- and post-test scores for both Job Exploration and Workplace Readiness were summed in IBM SPSS Statistics for Windows version 28.0 to calculate the t-test. In addition, Cohen’s d was calculated to determine measure of magnitude, which takes out the effect of the sample size (Cohen, 1988). A calculation of .5 to .7 indicates a medium effect, and .8 or higher indicates a large effect. We hypothesized that students would show improvement from the pre- to the post-tests, after instruction. The results of our two-tailed paired samples t-test supported this. The Cohen’s d for Job Exploration was d = 2.78 and d = 6.02 for Workplace Readiness, indicating a large effect for both sets of lessons.

In addition to making abstract concepts as concrete as possible, staff created lesson activities that engaged students to encourage participation and learning. Each lesson consisted of an icebreaker to both introduce the topic, make the lessons entertaining, and to avoid a purely lecture atmosphere. Example icebreaker activities include scavenger hunts, Hedbanz© inspired guessing games, and team problem solving activities. Lessons also included opportunities for group discussion and personal reflection. Some of the most engaged lessons incorporated class quizzes where students were able to rate examples of work ethic or use interactive slides to work through problem solving activities. Each lesson had accompanying workbook pages for students to complete to review lecture material, and record personal responses to questions related to lesson content. The goal of the workbook was to create a space where students could document their growing career awareness. Lesson material referred back to content recorded in the workbook from previous lessons so students could move from a basic understanding of employment to more individualized interests, preferences, and career clusters. The end goal is that these workbooks can be used by educators and VR professionals to base further Pre-ETS activities by supplying the basis of a job seeker profile.

The added benefit of focusing on class participation was the ability to assess students who did not communicate verbally or proficiently through written responses. In these cases, instructors documented student participation and expressions of interests in case reports in case these items were not documented in individual student workbooks. All students received case notes that were then shared with educators, case managers, and families to supplement information gleaned in student workbooks.

Though the workbook would use specific jobs as examples throughout the instructional series, emphasis was placed on students learning about and discovering potential careers in career clusters. This decision was made to place the emphasis not on learning specific skills for a specific job, but rather to explore potential career paths within a cluster. The reasoning is that limiting a student to a specific job could not only limit their career development, but may also be preparing them for a position that would not be relevant in our future economy. Too frequently, individuals with disabilities are relegated to positions that are quickly becoming obsolete or replaced by advancing technology (World Economic Forum, 2020).