Implementation Preparation Costs of Virtual Interview Training in Pre-Employment Transition Services: A Budget Impact Analysis

Abstract

Virtual Reality Job Interview Training (VR-JIT) and Virtual Interview Training for Transition Age Youth (VIT-TAY) demonstrated initial effectiveness at increasing employment among transition-age youth with disabilities engaged in pre-employment transition services. We characterized activities and estimated the labor and non-labor costs required to prepare schools to implement VR-JIT or VIT-TAY. Implementation preparation and support teams reported labor hours throughout the implementation preparation process. Implementation preparation labor hours at 43 schools cost approximately $1,427 per school, while non-labor costs were $100 per trainee (student). We estimated the replication of implementation preparation labor activities would cost $1,024 per school (range: $841-$1,208). Most costs were spent in delivery planning and teacher training. Given that implementation preparation costs can be barriers to intervention adoption, our results provide critical information for contemplating future implementation of VR-JIT or VIT-TAY.

Keywords

transition-age youth job interview training transition services implementation preparation costs

The Workforce Innovation Opportunity Act (WIOA, 2014) mandates that pre-employment transition services (Pre-ETS) be offered to students with disabilities in the United States. Pre-ETS includes five required services: job exploration counseling, work-based learning experiences, postsecondary education counseling, workplace readiness training, and instruction in self-advocacy. The WIOA also mandates that state level vocational rehabilitation agencies must dedicate a minimum of 15% of their federal funding to Pre-ETS and be offered to “students with disabilities who are eligible or potentially eligible for vocational rehabilitation services” (WIOA, 2014). While efforts have been made to enhance and support collaboration between vocational rehabilitation agencies and educational agencies (Rowe et al., 2021), there is still room to improve the delivery and cost-effectiveness of Pre-ETS.

At a state level, vocational rehabilitation agencies struggle to consistently deliver Pre-ETS. According to the Rehabilitation Services Administration, approximately 60% of students with disabilities received Pre-ETS in 2017 (Honeycutt et al., 2019). However, states varied greatly in whether, how, and by whom Pre-ETS were delivered (Honeycutt et al., 2019). For example, Pre-ETS can be delivered by state level vocational rehabilitation staff, contracted vendors, or other community rehabilitation providers. However, state agencies that do not have dedicated Pre-ETS staff find it easier to track Pre-ETS expenditures when hiring vendors to deliver Pre-ETS (National Council on Disability, 2017). Despite this flexibility, state level vocational rehabilitation counselors and teachers are often tasked with identifying and implementing evidence-based practices to meet students' Pre-ETS goals. Resources such as the National Technical Assistance Center on Transition: The Collaborative (NTACT:C) exist to assist with this task (Rowe et al., 2021). However, there may be delays between the publication of an intervention’s effectiveness and its inclusion in databases such as NTACT:C. Subsequently, counselors and teachers may be unaware of evidence-informed practices which could delay the adoption of effective Pre-ETS interventions by state level vocational rehabilitation agencies and schools.

One such delay is present in the job interview training literature. Specifically, job interview skills have been identified by the United States Department of Education’s Office for Special Education and Rehabilitative Services as an important target for Pre-ETS (United States Department of Education, 2017). However, Pre-ETS resource repositories (such as NTACT:C) only identify a single early-stage job interview intervention based on the findings of one small pilot study of video modeling (Lindsay et al., 2015; Rowe et al., 2021). Meanwhile, existing job interview training in Pre-ETS commonly consists of 1–2 role-plays (for approximately 20 minutes each) between Pre-ETS providers and students with disabilities, which is time- and resource-intensive. Thus, role-play training with providers is cost-prohibitive and recent evidence suggests that completing 1-2 job interview role-plays (within the context of employment services) does not improve interview skill or access to jobs (Smith, Smith et al., 2022). Notably, several recent studies suggest that computerized job interview training simulators may be feasible to implement at scale in special education, may be less resource intensive than in-person methods, and are associated with distal employment outcomes (e.g., Kim et al., 2021; Smith et al., 2021b; Smith et al., 2021c; Smith et al., 2022).

In particular, two of these computerized job interview trainings have been scaled-up in several schools implementing Pre-ETS and their initial efficacy and effectiveness have been evaluated in multiple studies. The first interview training intervention is Virtual Reality Job Interview Training (VR-JIT), which was developed for adults with psychiatric disabilities (Bell & Weintstein, 2011; Smith et al., 2014a) and five randomized controlled trials (RCTs) found that it was efficacious at increasing job interview skills and access to employment among young autistic adults and adults with serious mental illness (e.g., Smith et al., 2014a; Smith et al., 2014b; Smith et al., 2015). Meanwhile, the results of a subsequent quasi-experimental evaluation of VR-JIT suggested that greater VR-JIT engagement was associated with increased odds (OR = 1.63, p < .01) of obtaining competitive employment among 279 transition-age youth with autism or other disabilities engaged in Pre-ETS at 15 special education programs (Smith et al., 2021c).

Specifically, VR-JIT is an internet-delivered job interview simulator that resembles a remote meeting platform experience like Zoom or Microsoft Teams (Figure 1; Smith et al., 2014a). To use VR-JIT, trainees log into a website where they access a point-and-click elearning curriculum to learn about eight job interview skills, complete an online job application (for one of eight jobs) that informs interview questions that are delivered by a virtual hiring manager (i.e., video clips of an actor seamlessly spliced together with an algorithm), and then practice their interviews with the virtual hiring manager. The VR-JIT conversation interface provides trainees with 10–15 scripted options (ranging from helpful to hurtful on your interview performance) to answer questions that range in effectiveness; then trainees read through and speak their selection out loud. Speech recognition software uses the trainee’s response to inform the virtual hiring manager’s ongoing mood and response (or the next question). Trainees receive four levels of feedback to help improve their performance: 1) real-time nonverbal cues from an automated onscreen coach; 2) transcript-level feedback on their specific responses; 3) assessment summary for the eight job interview skills performed; and 4) a numerical score from 1-100.

Figure 1.

VR-JIT user interface (Top) and VIT-TAY user interface (bottom).

Moreover, VR-JIT uses hierarchical learning via easy, medium, and hard interviews that represent the personality of the hiring manager (e.g., friendly, professional, inappropriate). Due to a bank of hundreds of interview questions performed by the virtual hiring manager, VR-JIT can be used to repeatedly practice interviews as trainees progress from easy to hard interviews.

The second interview training intervention is Virtual Interview Training for Transition-Age Youth (VIT-TAY). Specifically, our scientific team partnered with the autism community and community and scientific advisory boards to review the acceptability, usability, and accessibility of VR-JIT and identified a series of adaptations to specifically meet the needs of autistic transition-age youth and youth with other disabilities (Smith et al., 2020b). VIT-TAY maintains the core interface and usability of VR-JIT while integrating additional evidence-based design components (e.g., scaffolding, social story telling), content-focused additions (e.g., adding new interview skills, and increasing diversity among the new virtual hiring managers (Figure 1).

Subsequently, the initial effectiveness of VIT-TAY was evaluated among 71 autistic transition-age youth in a RCT where trainees engaged with Pre-ETS and VIT-TAY. The results revealed that the Pre-ETS with VIT-TAY group improved their interview skills, interview anxiety, and had greater odds (OR = 16.0, p < .05) of competitive employment as compared to the Pre-ETS-as-usual group (Smith et al., 2021b). In addition, we conducted a quasi-experimental evaluation of VIT-TAY among 356 students with autism or other disabilities (e.g., specific learning disability, emotional disturbance) engaged in Pre-ETS within 32 special education programs. This study revealed that greater engagement with VIT-TAY was associated with significantly greater odds (OR = 1.73, p < .001) of obtaining competitive employment within six months. Notably, 48% of 117 non-job-seeking VIT-TAY trainees were competitively employed within six months as compared to 22% of 82 non-job-seeking VR-JIT trainees who were employed within six months (p < .001; Smith et al., 2022).

Moreover, the two aforementioned quasi-experimental studies evaluating the effectiveness of these interventions conducted concurrent initial implementation process outcome evaluations and found that the implementation preparation outcomes (e.g., intervention acceptability, strategies to prepare teachers, implementation feasibility) and teacher-level (e.g., number of sessions per week) and school-level (e.g., types of class room settings) practical strategies used to implement VR-JIT and VIT-TAY did not differ from one another (Smith, Sherwood et al., 2022). However, there were several notable differences in the intervention design between VR-JIT and VIT-TAY. Three examples follow.

First, VIT-TAY was designed to be more accessible than VR-JIT (e.g., lower required reading level (i.e., 4^th grade reading level vs. 6^th grade reading level), shorter virtual interviews (e.g., easy VIT-TAY interviews lasted approximately 10 minutes vs. easy VR-JIT interviews lasting approximately 25 minutes). Second, the newly integrated VIT-TAY social storytelling provided more immersive automated coaching and feedback. Third, VIT-TAY has greater racial diversity of virtual hiring managers (Smith, Pinto et al., 2020b).

Overall, the dissemination of these results has led to the real-world implementation of VIT-TAY (or VR-JIT) in more than 70 high school special education and community post-secondary pre-employment programs across 10 US states (including initial implementation by the State of Illinois’ Department of Health and Human Services Division of Vocational Rehabilitation) as of September 2022. Thus, there is a large-scale initiative to implement VIT-TAY (or VR-JIT) prior to validating the effectiveness of the intervention in a fully powered randomized controlled trial. Given the unexpected emergence of VIT-TAY (or VR-JIT) adoption, schools, school districts, state boards of education, and state vocational rehabilitation agencies will be interested in learning about the budget impact of preparing Pre-ETS facilitators to implement VIT-TAY (or VR-JIT). These findings are especially critical given cost could be a critical barrier to adopting and sustaining this intervention, as noted in previous work with teachers and administrators (Sherwood et al., 2023; McGoey et al., 2014). Specifically, by estimating the costs of the intervention prior to enrollment, administrators can calculate an expected cost before any trainees are enrolled in the intervention. This estimate of implementation preparation costs will allow administrators to understand the magnitude of intervention costs before receiving any return on their investment from VIT-TAY (or VR-JIT).

The field of implementation science defines the “preparation” phase as the period between when a setting (i.e., school) chooses to adopt a new intervention and when that intervention becomes available within the school (Aarons et al., 2011). This implementation preparation phase focuses on several key activities that include identifying communication channels and champions, training administrators and teachers, defining teacher roles for delivering the intervention, identifying and coordinating delivery methods (e.g., hardware and space), and adapting intervention materials. Thus, we sought to characterize the costs of implementation preparation across several schools that were evaluating VIT-TAY (or VR-JIT) effectiveness and implementation. In this study, we define the implementation preparation phase as the activities and resources required for VIT-TAY (or VR-JIT) to be available to special education students engaged in Pre-ETS (i.e., to prepare schools to implement virtual interview training), prior to students actually using the interventions. Notably, the evaluation of an intervention’s implementation preparation labor and non-labor activities is an emerging area of research in primary health care and mental health services settings (Jordan et al., 2019; Smith et al., 2020a). In this paper, we present the results of a budget impact analysis that assessed the expected short-term changes in expenditures for schools preparing to implement VIT-TAY (or VR-JIT). Given this study was the first evaluation of implementation preparation activities associated with VIT-TAY (or VR-JIT) in school settings, time spent on some implementation preparation activities was likely greater than what would be necessary during future implementation preparation activities. Thus, as a secondary exploratory aim, we conducted sensitivity analyses to estimate the costs associated with replicating the activities associated with implementation preparation of VIT-TAY (or VR-JIT) to inform future implementation.

Methods

Study Design

We used an implementation preparation design to estimate the labor and non-labor costs of preparing to implement VIT-TAY (or VR-JIT) in 43 schools. Implementation preparation is a phase of the EPIS (Exploration, Preparation, Implementation, and Sustainment) framework that guides the design and administration of implementation process evaluations (Moullin et al., 2019). Specifically, the implementation preparation phase includes the collection of labor costs incurred during meetings, communication between the schools and research team, training teachers to deliver the interventions, and supervision of these processes.

Participants

To recruit school partners to participate in quasi-experimental studies to evaluate the implementation and initial effectiveness of VIT-TAY or VR-JIT, the research team engaged in networking and relationship building with the Illinois Department of Human Services Division of Rehabilitation Services, Chicago Public Schools’ Office of Diverse Learner Supports and Services, Michigan Rehabilitation Services, and Project SEARCH (i.e., an evidence-informed Pre-ETS program with a network of more than 400 sites in the United States [Rowe et al., 2021]). Selected staff from 76 schools were trained to implement VIT-TAY or VR-JIT within local Pre-ETS programming. Subsequently, 29 schools dropped out due to competing administrative priorities. Forty-seven schools participated in the aforementioned quasi-experimental studies, 43 of which completed an evaluation of implementation preparation activities that are the focus of this study. The 43 schools were from Illinois, Michigan, and Florida, and represented city (n = 15), suburban (n = 22), town (n = 4), and rural locales (n = 2). We used the National Center for Education Statistics (NCES) locale framework to categorize schools into locales based on definitions developed by the United States Census Bureau (Geverdt, 2015). From these schools, 80 school staff members and 635 students participated.

Three participant groups were included in this study: implementation support staff (who were external scientific partners), special education administrative leaders (referenced as leaders) and teachers. Implementation support staff worked with leaders and teachers to guide their implementation of the technology and troubleshoot technology needs (e.g., getting the program to run on local devices) throughout the process. Leaders helped facilitate the implementation process between the implementation support staff and teachers. Teachers were trained by the implementation support team to orient the transition-age youth to use VIT-TAY (or VR-JIT). Some leaders also served as teachers (n = 24). In these instances, they were responsible for engaging in both administrative and teaching duties. All participants worked at educational programs in the United States (i.e., n = 32 public schools; n = 10 Project SEARCH sites; and n = 1 private school). The Project SEARCH model is a partnership between schools, state-level vocational rehabilitation services, and a community employer where students receive class-based instruction at school or an internship site and complete three internships over the course of one year (Rutkowski et al., 2006). While Project SEARCH sites may not always be schools, we refer to them as schools for simplicity. Our calculations weighed each Project SEARCH site equally with school partners; we analyzed their implementation costs the same way that we did for schools.

Time Horizon

Implementation preparation activities across four cohorts began in August 2017 (cohort 1), January 2018 (cohort 2), August 2018 (cohort 3), and January 2019 (cohort 4) and concluded after 4–6 weeks.

Perspective

We completed a budget impact analysis from the perspective of school budget holders. The analysis includes the estimated labor and non-labor costs incurred by schools during the preparation phase, prior to implementation of VIT-TAY (or VR-JIT).

Interventions

VIT-TAY and VR-JIT are computerized job interview simulators that are delivered via the internet and commercially licensed by SIMmersion LLC (www.simmersion.com). As described in the introduction, the simulators include e-learning content and mock job applications to prepare trainees to engage in the virtual interviews. Trainees then repeatedly interview with virtual hiring managers (Figure 1) that vary in mood and personality, which help facilitate a range of virtual hiring managers who can be friendly, professional, or inappropriate. The virtual interviews are facilitated by speech-recognition software and last 10–45 minutes (depending on reading speed and whether interviews occur at an easy, medium, or hard level of difficulty. Overall, VIT-TAY and VR-JIT can provide over 15 hours of unique virtual interviewing experience. See Smith, Pinto and colleagues (2020) for additional details.

Analytic Framework

To calculate implementation preparation labor costs, we used a cost calculator approach, which uses activity descriptions, activity time estimates, and participant hourly rates to calculate costs for each activity (Sullivan et al. 2014). All calculations for this study were completed in Microsoft Excel.

Data Source

Leaders and teachers recorded and emailed their labor hours directly to the research team or submitted their hours electronically via Research Electronic Data Capture (REDCap; Harris et al., 2009). They were also asked to disclose their salaries (Table 1). Annual salaries were used to calculate hourly rates. Table 1 includes the number of participants in each group, their average salary, and hourly salary costs. In some cases where salaries were not disclosed or reported, the research team obtained an estimated salary from a public government website (e.g., https://salary.bettergov.org/) or created an estimate based on employment position title and geographic location of the school. School salaries included their annual wage plus an estimated 34.4% fringe benefit from the national compensation survey collected by the Bureau of Labor Statistics (https://www.bls.gov/web/ecec/ececqrtn.pdf). Additionally, to ensure results were still current, all salaries were adjusted from 2018 and 2019 to 2020 using an inflation estimate from a computer price index inflation calculator (https://www.bls.gov/data/inflation_calculator.htm).

Table 1.

Budget impact analysis input parameters for labor and non-labor costs.

Variables	Average annual salaries¹	Average hourly rate	One time purchase	Reference
Labor cost inputs
External scientific partner (n = 5)	$127,680	$49	—	Actual salaries
High school leaders (n = 37)	$92,177	$44	—
High school teachers (n = 43)	$70,023	$34	—
Nonlabor costs
Software license (n = 635)	—	—	$100²

Average annual salaries include individually reported salary information updated to 2020 estimates and 2020 fringe estimates from the Bureau of Labor Statistics.

²This is the software license price per trainee.

Measures

School staff recorded how much time they spent on each implementation preparation activity. Activity data were collected weekly or biweekly during the 6–8 weeks prior to the school’s implementation of VIT-TAY (or VR-JIT). A priori labor categories were created by adapting existing implementation preparation activity surveys used in mental health settings (Smith, Graham et al., 2020a), which were then reviewed by Pre-ETS teachers and adapted for school settings. Intervention adoption and data collection were conducted in accordance with the ethical standards of the University of Michigan Institutional Review Board, and informed consent was not required.

Analysis

Labor costs were calculated using the cost calculator approach (Sullivan et al. 2014). Labor cost was estimated by multiplying the per-hour salary by the time spent on each activity. This process was completed for each activity and each participant. Collectively, these costs created our labor estimates for the study. The costs of each activity were aggregated by participant type (e.g., teacher, leader). Activities were then collapsed based on implementation preparation activity type: meeting and correspondence, materials, training, technical support, lab space/infrastructure, and supervision.

To estimate replication costs, the implementation support team generated an expected effort to complete each implementation preparation activity in another school setting. Together, the principal investigator (MJS) and project manager (BR) reviewed labor activities and generated replication efforts based on their experience with the study.

Uncertainty Analysis

To provide the schools with more guidance on implementation preparation costs, MJS and BR also discussed and generated effort ranges to use for a sensitivity analysis. The sensitivity analysis was conducted by using the effort ranges for each activity to calculate an expected cost range (Sullivan et al. 2014).

Results

Implementation preparation for all 43 schools included 1662 hours and $61,346 in labor costs. The average labor costs per school was $1,427 (Table 2). Nearly half of labor hours (45%) were spent in meetings and correspondence among school staff and with the implementation support team. One-quarter of the labor hours were devoted to training (orientation, role-playing, and interviewing) with the software. The remaining labor hours were allotted to developing, reviewing, and printing training and orientation materials (13%), providing technical support (9%), coordinating physical infrastructure to support software delivery (6%), and providing supervision for implementation activities (1%). Although many participants were involved in different activities, the total hours were not the same from each participant. For example, four participants reported 65 collective hours spent “leading or co-leading orientation training among implementation support team members”. However, each participant did not spend the same amount of time on this activity. Participant hours ranged from 4 to 11 hours.

Table 2.

Implementation-Preparation Activities.

	Total Number of Unique Staff Involved Across activities	Total Hours	Proportion of Total Hours (%)	Proportion of Hours Accrued by the External Scientific Partner (%)	Total Labor Costs	Proportion of Costs Accrued by the External Scientific Partner (%)
Meetings and correspondence:
Delivery planning and software and device preparation among school staff	75	668	40	0	$23,171	0
Implementation planning with teachers/leaders and implementation support team	4	49	3	100	$2,829	100
Planning among implementation support team	4	35	2	100	$2,595	100
Sub total	79	752	45	11	$28,595	19
Materials:
Developing, reviewing, and printing training and orientation materials among school staff	74	209	12	0	$6,649	0
Developing, reviewing, and printing training and orientation materials among implementation support team members	4	12	1	100	$828	100
Sub total	78	221	13	5	$7,477	11
Training:
Time spent in training (e.g., orientation, role-playing, interviewing with software) among school staff	80	357	21	0	$10,940	0
Time spent leading or co-leading orientation training among implementation support team members	4	65	4	100	$3,070	100
Sub total	84	422	25	15	$14,010	22
Technical support:
Time spent coordinating with school IT team regarding software and device preparation as well as providing technical support among school staff	58	124	8	0	$4,523	0
Time spent coordinating with school IT team regarding software and device preparation as well as providing technical support among implementation support team members	3	18	1	100	$727	100
Sub total	61	142	9	13	$5,250	14
Lab space/Infrastructure:
Time spent coordinating physical infrastructure to support software delivery (i.e. setting up computer lab; access to devices) among school staff	56	78	5	0	$2,917	0
Setup student accounts within software among implementation support team members	4	26	2	100	$874	100
Sub total	60	104	7	25	$3,791	23
Supervision: Implementation support team
Implementation support team providing supervision of overall pre-implementation activities	1	21	1	100	$2,223	100
Sub total	1	21	1	100	$2,223	100
Total		1662			$61,346	100
Price per school					$1,427

In addition to labor costs, VIT-TAY (or VR-JIT) cost approximately $100 per license per trainee (Table 1). This was a one-time fee based on the licensing framework from 2017 that was specific to the scientific evaluation of the interventions and does not represent current pricing.

Table 3 displays the estimated replication effort expected for each implementation preparation activity. All activities were estimated to take fewer labor hours than accrued during the study. We estimated the effort needed to replicate implementation preparation activities of VR-JIT or VIT-TAY would range from 15% to 90% across the different activities (e.g., developing materials estimated at 15% and leading VIT-TAY orientations at 90%). The total estimated labor cost of replicating the implementation preparation activities was 72% ($44,040) of the costs incurred in this study. Based on these replication amounts, we estimated that labor costs for preparing schools to implement VR-JIT or VIT-TAY will be approximately $1,024 per school and could range from $841 to $1,208.

Table 3.

Implementation-preparation estimate.

Implementation-Preparation activity	Estimate (%)	Range (%)	Replication Cost¹	Cost Range
Delivery planning and software and device preparation among school staff	75	60–90	$17,378	$13,903 - $20,854
Implementation planning with teachers/leaders and implementation support team	80	70–90	$2,263	$1980 - $2546
Planning among implementation support team	80	70–90	$2,076	$1,817 - $2,336
Developing, reviewing, and printing training and orientation materials by school staff	30	20–40	$1,995	$1,330 - $2,660
Developing, reviewing, and printing training and orientation materials by implementation support team members	15	5–25	$124	$41 - $207
Time spent in training (e.g., orientation, role-playing, interviewing with software) by school staff	80	65–95	$8,752	$7,111 - $10,393
Time spent leading or co-leading orientation training among implementation support team members	90	80–100	$2,763	$2,456 - $3,070
Time spent coordinating with school IT team regarding software and device preparation as well as providing technical support by school staff	75	65–85	$3,392	$2,940 - $3,845
Time spent coordinating with school IT team regarding software and device preparation as well as providing technical support among implementation support team members	85	75–95	$618	$546 - $691
Time spent coordinating physical infrastructure to support software delivery (e.g., setting up computer lab, access to devices) by school staff	80	70–90	$2,333	$2,042 - $2,625
Setup student accounts within software among implementation support team members	65	50–80	$568	$437 - $699
Implementation support team providing supervision of overall pre-implementation activities	80	70–90	$1,778	$1,556 - $2,001
Total	—	—	$44,040	$36,159 - $51,927
Total per school	—	—	$1,024	$841 - $1208

¹Replication costs are the labor costs from Table 2 multiplied by the estimate % in Table 3.

Figure 2 displays the estimated cost range for each implementation preparation activity. Most activities did not vary more than plus or minus $665. On the other hand, delivery planning and software and device preparation among staff ($3,476) and time spent in training among staff ($1,641) costs may vary widely.

Figure 2.

Estimated cost range per implementation activity.

Discussion

This study determined that the labor costs associated with implementation preparation activities to prepare Pre-ETS programs to deliver VIT-TAY (or VR-JIT) were $1,427 per school. However, our sensitivity analyses determined that the labor costs would be $1,024 per school (ranging from $841 to $1,208) to replicate the implementation preparation activities with future special education programs. Most implementation preparation costs were accrued by school staff from two activities: time spent planning the delivery of software and devices, and time spent in training. Given the wide variation in these activities, we expect there is an opportunity to minimize the labor hours and reduce overall costs. Additionally, the implementation support team identified that some activities, such as setting up student access to VIT-TAY, could be completed by school staff (as opposed to the more expensive costs associated with the implementation support team). This approach could reduce costs to the school as they would otherwise pay additional fees for external implementation support. Costs could also be reduced if school staff would complete the orientation (on how to teach students to use VIT-TAY) via publicly available online resources and troubleshoot any software setup directly with SIMmersion instead of the external implementation support team.

Regarding non-labor costs, VIT-TAY (or VR-JIT) cost $100 per trainee for a one-year license. For our analysis, we assumed that participating schools had computing devices (e.g., computers, laptops, tablets) to run VIT-TAY (or VR-JIT) and headsets (with microphones) to facilitate the speech recognition functions. While most research testing the use of VIT-TAY (or VR-JIT) involves the use of headsets, the software can be used without them. Additionally, although we did not include the cost of devices in our estimates, the cost of over-the-ear headphones with an inline microphone is generally low ($10 to $20 per headset). These costs may be important for schools to consider in their decision making.

Given WIOA mandates that state level vocational rehabilitation agencies dedicate a minimum of 15% of their federal funding to Pre-ETS (approximately $545,247 federally in 2019; United States Department of Education, 2019), our estimates suggest implementation preparation costs may be covered by state grants if delivered by state-level vocational rehabilitation services. For example, the State of Illinois meets part of the 15% reserve requirement of its allotment, under the federal Vocational Rehabilitation Services program, by funding the Secondary Transitional Experience Program or STEP (https://www.dhs.state.il.us/page.aspx?item=35174). STEP helps facilitate access to Pre-ETS for local schools, districts, and special education cooperatives who receive STEP contracts to help fund services. Notably, some of these programs were involved in prior evaluations of VIT-TAY and VR-JIT (Smith et al., 2022; Smith et al., 2021c). After the conclusion of these studies, some programs used their STEP funding to cover the costs of the tools. This knowledge is especially promising given the option to further reduce costs by completing orientation online and shifting some implementation preparation activities from the implementation support team to school staff.

Future research should explore new delivery models (e.g., district-level initiatives) that financially support the needs of several schools in an area to increase the number of recipients while limiting costs. One way to do this would be to compare the implementation preparation costs of VIT-TAY between teachers and vocational rehabilitation counselors and/or vendors. Alternative strategies for funding VIT-TAY in schools could include the Elementary and Secondary School Emergency Relief (ESSER) funding or Individuals with Disabilities Education Act (IDEA, 2004) special education funding. ESSER funds can be spent on evidence-based activities that address learning loss for students including those with disabilities (American Rescue Plan Act, 2021). However, state boards of education and local school districts may have their own limits on how ESSER funds can be used. In terms of IDEA funding, IDEA has historically been underfunded by the federal government and thus puts more strain on state and local education agencies. However, this issue is beyond the scope of this paper. An additional source of funding that one school district is using to support VIT-TAY implementation is through funding obtained from the Strengthening Career and Technical Education for the 21^st Century Act or Perkins V (2018). Perkins V funding is disseminated through state-level grants that in turn funds school districts and postsecondary institutions to support career and technical education programs.

Overall, implementation cost analyses are a major gap in implementation research more broadly (Eisman et al., 2021) and particularly within educational settings (Levin & Belfield, 2015). However, there are a limited number of in-field comparisons which can prove useful in comparing implementation preparation costs of VIT-TAY and VR-JIT to other evidence-based interventions in schools. For example, Positive Behavior Intervention Support (PBIS) supports a broader student audience and has been implemented in over 20,000 schools. Specifically, PBIS is a multi-tiered framework to establish school-wide behavioral support for all students and costs approximately $25,000 per school annually for meetings and training activities (Bradshaw et al., 2020). Notably, school-based coaching provides support to schools to build capacity and facilitate the implementation of school-wide multi-tiered systems of evidence-based practices, which costs schools $8,198 annually (Pas et al., 2020). However, the authors reported that schools spent an average of $984 annually for PBIS preparation and planning activities (Pas et al., 2020). Although the scope of their study used a larger sample and applies to a wider student audience, their estimates are similar to the projected $1,024 one-time cost to prepare schools to implement VIT-TAY (or VR-JIT). Another in-field implementation example, Applied Behavior Analysis (ABA), is one of the most costly autism interventions with average annual full implementation costs of $40,000 per student (Cakir et al., 2020; Chasson et al., 2007). Despite being one of the most commonly funded autism interventions, implementation preparation costs for ABA in a school setting are not yet known as ABA is often not delivered in schools due to its high cost (Rogge & Janssen, 2019). While these comparisons are helpful, PBIS, school-based coaching, and ABA are more often evaluated as comprehensive intervention programs, whereas VIT-TAY and VR-JIT are skill-focused. This discrepancy makes it difficult to conduct implementation preparation cost comparisons.

Overall, the results of this study are a critical addition to the expanding literature highlighting the potential benefits of implementing VIT-TAY within special education Pre-ETS. Notably, prior studies found VIT-TAY demonstrated initial effectiveness at improving interview skills, reducing interview anxiety and increasing access to competitive employment (Smith et al., 2021a). This literature also suggests that special education teachers and students with disabilities perceived VIT-TAY as highly acceptable and feasible to implement within the typical structure of Pre-ETS. The current study expanded this literature by reporting on the estimated amount of labor (and associated costs) that may be required for future sites to prepare to implement VIT-TAY (or VR-JIT) in special education programming. More specifically, special education administrators can reference the reported implementation preparation costs to construct realistic budgets to prepare their schools to deliver VIT-TAY (or VR-JIT). Importantly, the activities and labor required to prepare for VIT-TAY implementation could also serve as a proxy for other technology-based interventions that might be used to enhance Pre-ETS.

Limitations and Implications for Future Research

We note this study has some limitations. First, although our sample included schools from cities, suburbs, towns, and rural settings, schools from towns and rural locales were comparatively underrepresented. Thus, future research should elucidate whether there are any unique implementation preparation costs that may be relevant to schools in town and rural locales. Second, the schools were primarily public schools with only a single private school participating (that served three students). Thus, our results may not generalize to private and charter schools where there could be variation in the required implementation preparation activities. Third, public school partners received support from state-level vocational rehabilitation services. Thus, study results may not generalize to schools that receive less funding or do not have state vocational rehabilitation support; these schools should be included in future research. Fourth, this study estimated school labor costs. However, future studies should examine vocational rehabilitation labor costs or shared labor costs between schools and vocational rehabilitation agencies. Fifth, we used actual salary information to calculate our estimates. While we included schools from three states, salaries may vary beyond our sample and this needs to be considered when applying our results in cost estimations. Lastly, the results of this study may not generalize to schools with limited resources that declined to participate in this study due to understaffing and different priorities. In addition, future research is also needed to evaluate the costs of the actual implementation activities to increase our understanding of the potential cost effectiveness or return on investment of VIT-TAY (or VR-JIT).

Implications for Research, Practice, and Policy

The U.S. Department of Education has identified job interviewing ability as one of several skills to be supported through Pre-ETS (U.S. Department of Education, 2017). Moreover, studies suggest between 40% and 90% of employed youth with disabilities receiving state-supported Pre-ETS interviewed prior to their employment (e.g., Smith et al., 2021b; Smith et al., 2021a). Given the recent emergence of several job interview interventions in the literature (Kim et al., 2021), schools will need to make informed decisions based on the effectiveness and costs of these tools. However, the adoption of novel interventions by state level agencies and their contracted partners may face delays due to the limited knowledge about the upfront costs required to implement new technology-based practices (Bond et al., 2022). Thus, this study provided insight into the costs that administrators may face when preparing an infrastructure to implement VIT-TAY (or VR-JIT).

Specifically, administrators may need to weigh the balance between the labor hours required to prepare their school to implement VIT-TAY (or VR-JIT) with the costs of licensing the use of the tools themselves (e.g., $1000 per 25 student VIT-TAY licenses) while taking into account that several RCTs and quasi-experimental studies of VIT-TAY (or VR-JIT) suggest they significantly increase access to competitive employment. Administrators may also need to balance the difference between paying an external implementation support team to train all teachers to VIT-TAY implementation fidelity and the labor hours associated with training a school staff member to become a designated VIT-TAY trainer to onboard other teachers. Such an approach could reduce out-of-pocket financial costs but increase labor hours for school staff.

Conclusion

Given the emerging effectiveness of VIT-TAY at increasing employment among students engaged in Pre-ETS, the field could benefit from new knowledge on the labor and associated costs needed to prepare special education programs to deliver technology-based enhancements such as VIT-TAY. Accordingly, the estimated labor costs to prepare Pre-ETS programs to implement VIT-TAY were $1024. This estimate provides administrators with a realistic labor requirement within their budget and remove uncertainty that may reduce the likelihood of adoption. Although this estimate may help reduce barriers to adoption, further research is warranted to evaluate whether VIT-TAY provides a return on investment as an increase in the hours worked and wages earned by students with disabilities using VIT-TAY could further inform the adoption of this intervention.

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The University of Michigan will receive royalties from SIMmersion LLC on the sales of virtual interview training for transition age youth (VIT-TAY) that was evaluated in this study. These royalties will be shared with Dr. Matthew Smith and the University of Michigan School of Social Work. Dr. Smith adhered to the University of Michigan’s Conflict Management Plan that was reviewed and approved by a University of Michigan Conflict of Interest Committee. No other authors report any conflicts of interest.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by grants from the Kessler Foundation (1003-1958-SEG-FY2016, PI: Matthew Smith). Marc Atkins was supported by the National Center for Advancing Translational Sciences, National Institutes of Health Grant UL1TR002003. Justin D. Smith was supported by the National Institute on Drug Abuse, National Institutes of Health Grant P30DA027828. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Kessler Foundation and the National Institutes of Health. Elizabeth C. Danielson was supported by funding from the National Institute on Disability, Independent Living, and Rehabilitation Research (#90ARHF0003).

ORCID iDs

Elizabeth C. Danielson

Matthew J. Smith

Kari Sherwood

Author Biography

Elizabeth C. Danielson, PhD is a Research Assistant Professor in the Department of Medical Social Sciences at Northwestern University Feinberg School of Medicine. Dr. Danielson is a health services and user experience researcher focused on utilizing measures of social risk to reduce disparities in policy payment models and improve health outcomes after high-cost, high-volume services, and procedures such as total joint replacement. Her research also includes the development of, and implementation cost analyses related to health technology.

Matthew J. Smith, PhD, MSW, MPE, LCSW is Professor and director of the NIH-Funded Level Up: Employment Skills Simulation Lab at the University of Michigan. Through his research, he develops and evaluates technology-based interventions to help marginalized and underserved communities obtain and sustain employment. Dr. Smith partners with members of the autism, mental health and returning citizen communities to develop employment-focused tools and evaluate them in real world settings such as high schools, community mental health agencies and prisons.

Brittany Ross, MS, is the project manager for the Level Up: Employment Skills Simulation Lab within the School of Social Work at the University of Michigan. Ms. Ross has managed research studies for over six years at the University of Michigan. She currently supports all projects within the Level Up Lab and she is particularly interested in understanding how technology-based interventions can facilitate employment opportunities for transition-age youth with disabilities and other underrepresented populations.

Kari Sherwood, MS, MEd, MSW, LLMSW, is a joint PhD student in Social Work and Developmental Psychology. She has over ten years experience as an instructional designer and is the co-Founder of Pathway Inclusion Center, Inc. She is interested in developing, evaluating, and implementing technology based interventions focused on social cognition for individuals with autism and other developmental and psychiatric disabilities to improve employment outcomes, including employer targeted interventions to reduce unconscious bias in the workplace.

Justin D. Smith, PhD is an Associate Professor in the Department of Population Health Sciences at the University of Utah School of Medicine and adjunct associate professor in the Feinberg School of Medicine at Northwestern University. His work focuses on implementation science methodology and the application of implementation research to improve healthcare quality and reduce health disparities. He has served on several national steering and advisory committees for his expertise in implementation science for the prevention and management of infectious and chronic diseases, such as childhood obesity, hypertension, HIV, mental health, and cancer.

Marc Atkins, PhD is a Professor of Psychiatry and Psychology at the University of Illinois Chicago and director of the Institute for Juvenile Research. He also directs the Community Engagement and Collaboration Core for UIC’s Center for Clinical and Translational Science which provides consultation to investigators on community-based research and dissemination and implementation science. His research examines new models for mental health practice in high poverty urban communities, especially for children with ADHD and aggression, to address long-standing disparities in mental health care.

Neil Jordan, PhD, is a Professor in the Department of Psychiatry & Behavioral Sciences and Preventive Medicine at Northwestern University Feinberg School of Medicine. He is a director of the Mental Health Services & Policy Program. Dr Jordan also leads the Health Economics Program within Northwestern’s Institute for Public Health and Medicine. Dr Jordan’s research focuses on identifying high value services and systems of care for persons with complex chronic illness.

References

Aarons

G. A.

Hurlburt

Horwitz

S. M.

(2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7

American Rescue Plan Act (2021). Public law 117-2. United states congress. § 4089. American Rescue Plan Act. https://www.congress.gov/117/bills/hr1319/BILLS-117hr1319enr.pdf

Bell

M. D.

Weinstein

(2011). Simulated job interview skill training for people with psychiatric disability: feasibility and tolerability of virtual reality training. Schizophrenia Bulletin, 37(Suppl 2), S91–S97. https://doi.org/10.1093/schbul/sbr061

Bond

Pogue

Al-Abdulmunem

Becker

Drake

(2022). State-level barriers and facilitators to individual placement support (IPS) implementation. Westat. https://ipsworks.org/wp-content/uploads/2022/03/22-ASPIRE-IssueBriefState-LevelBarriers-1.pdf

Bradshaw

C. P.

Debnam

K. J.

Player

Bowden

Lindstrom Johnson

(2020). A mixed-methods approach for embedding cost analysis within fidelity assessment in school-based programs. Behavioral Disorders, Advanced online publication. https://doi.org/10.1177/0198742920944850

Cakir

Frye

R. E.

Walker

S. J.

(2020). The lifetime social cost of autism: 1990–2029. Research in Autism Spectrum Disorders, 72, 101502. https://doi.org/10.1016/j.rasd.2019.101502

Chasson

G. S.

Harris

G. E.

Neely

W. J.

(2007). Cost comparison of early intensive behavioral intervention and special education for children with autism. Journal of Child and Family Studies, 16(3), 401–413. https://doi.org/10.1007/s10826-006-9094-1

Eisman

A. B.

Quanbeck

Bounthavong

Panattoni

Glasgow

R. E.

(2021). Implementation science issues in understanding, collecting, and using cost estimates: A multi-stakeholder perspective. Implementation Science: IS, 16(1), 75. https://doi.org/10.1186/s13012-021-01143-x

Geverdt

(2015). Education demographic and geographic estimates program (EDGE): Locale boundaries user’s manual (NCES 2016-012). U.S. Department of Education. Washington, DC: National Center for Education Statistics. https://catalog.data.gov/dataset/locales-2018

10.

Harris

P. A.

Taylor

Thielke

Payne

Gonzalez

Conde

J. G.

(2009). Research electronic data capture (REDCap) - a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377–381. https://doi.org/10.1016/j.jbi.2008.08.010

11.

Honeycutt

Sevak

Rizzuto

(2019). Provision of pre-employment transition services: Vocational rehabilitation agency variation in the first program year of reporting. Mathematica. https://www.mathematica.org/publications/provision-of-pre-employment-transition-services-vocational-rehabilitation-agency-variation

12.

Individuals with Disabilities Education Act . (2004). public law 101-476, 20 U.S.C. § 1400. U.S.C. https://sites.ed.gov/idea/statute-chapter-33/subchapter-i/1400

13.

Jordan

Graham

A. K.

Berkel

Smith

J. D.

(2019). Costs of preparing to implement a family-based intervention to prevent pediatric obesity in primary care: A budget impact analysis. Prevention Science: The Official Journal of the Society for Prevention Research, 20(5), 655–664. https://doi.org/10.1007/s11121-018-0970-x

14.

Kim

J. W.

Park

H. J.

Kim

S. Y.

Kim

H. S.

Baik

G. H.

Park

S. C.

Lee

S. J.

T. H.

(2021). A novel one-step knife approach can reduce the submucosal injection time of endoscopic submucosal dissection: A single-blinded randomized multicenter clinical trials. Gut and Liver, 16(1), 44–52. https://doi.org/10.5009/gnl210103

15.

Levin

H. M.

Belfield

(2015). Guiding the development and use of cost-effectiveness analysis in education. Journal of Research on Educational Effectiveness, 8(3), 400–418. https://doi.org/10.1080/19345747.2014.915604

16.

Lindsay

McDougall

Sanford

Menna-Dack

Kingsnorth

Adams

(2015). Exploring employment readiness through mock job interview and workplace role-play exercises: Comparing youth with physical disabilities to their typically developing peers. Disability and Rehabilitation, 37(18), 1651–1663. https://doi.org/10.3109/09638288.2014.973968

17.

McGoey

K. E.

Rispoli

K. M.

Venesky

L. G.

Schaffner

K. F.

McGuirk

Marshall

(2014). A preliminary investigation into teacher perceptions of the barriers to behavior intervention implementation. Journal of Applied School Psychology, 30(4), 375–390. https://doi.org/10.1080/15377903.2014.950441

18.

Moullin

Joanna C

Dickson

Kelsey S

Stadnick

Nicole A

Rabin

Borsika

Aarons

Gregory A

(2019). Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implement Sci, 1748-590814(1), 1. DOI:10.1186/s13012-018-0842-6. 30611302.

19.

National Council on Disability (2017). Implementation report memorandum "NDC's examination of the early implementation of WIOA’s pre-employment transition services by VR agencies and RSA. National Council on Disability. Retrieved August 20, 2021 from https://ncd.gov/sites/default/files/FINAL_WIOAETSImplementationMemorandum.pdf

20.

Pas

E. T.

Lindstrom Johnson

Alfonso

Y. N.

Bradshaw

C. P.

(2020). Tracking time and resources associated with systems change and the adoption of evidence-based programs: The “hidden costs” of school-based coaching. Administration and Policy in Mental Health, 47(5), 720–734. https://doi.org/10.1007/s10488-020-01039-w

21.

Rogge

Janssen

(2019). The economic costs of autism spectrum disorder: A literature review. Journal of Autism and Developmental Disorders, 49(7), 2873–2900. https://doi.org/10.1007/s10803-019-04014-z

22.

Rowe

D. A.

Mazzotti

V. L.

Fowler

C. H.

Test

D. W.

Mitchell

V. J.

Clark

K. A.

Holzberg

Owens

T. L.

Rusher

Seaman-Tullis

R. L.

Gushanas

C. M.

Castle

Chang

W.-H.

Voggt

Kwiatek

Dean

(2021). Updating the secondary transition research base: Evidence- and research-based practices in functional skills. Career Development and Transition for Exceptional Individuals, 44(1), 28–46. https://doi.org/10.1177/2165143420958674

23.

Rutkowski

Daston

Van Kuiken

Riehle

(2006). Project SEARCH: A demand-side model of high school transition. Journal of Vocational Rehabilitation, 25, 85–96. https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.458.530

24.

Sherwood

Smith

M. J.

Ross

Johnson

Harrington

Blajeski

DaWalt

Bishop

Smith

J. D.

(2023). Mixed methods implementation evaluation of virtual interview training for transition-age autistic youth in pre-employment transition services. Journal of Vocational Rehabilitation, 58(2), 139–154. https://doi.org/10.3233/JVR-230004

25.

Smith

M. J.

Fleming

M. F.

Wright

M. A.

Roberts

A. G.

Humm

L. B.

Olsen

Bell

M. D.

(2015). Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment. Schizophrenia Research, 166(1-3), 86–91. https://doi.org/10.1016/j.schres.2015.05.022

26.

Smith

M. J.

Ginger

E. J.

Wright

Boteler Humm

Olsen

Bell

M. D.

Fleming

M. F.

(2014b). Virtual reality job interview training for individuals with psychiatric disabilities. The Journal of Nervous and Mental Disease, 202(9), 659–667. https://doi.org/10.1097/NMD.0000000000000187

27.

Smith

M. J.

Ginger

E. J.

Wright

M. A.

Taylor

J. L.

Humm

L. B.

Olsen

D. E.

Bell

M. D.

Fleming

M. F.

(2014a). Virtual reality job interview training in adults with autism spectrum disorder. Journal of Autism and Developmental Disorders, 44(10), 2450–2463. https://doi.org/10.1007/s10803-014-2113-y

28.

Smith

M. J.

Graham

A. K.

Sax

Spencer

E. S.

Razzano

L. A.

Smith

J. D.

Jordan

(2020a). Costs of preparing to implement a virtual reality job interview training programme in a community mental health agency: A budget impact analysis. Journal of Evaluation in Clinical Practice, 26(4), 1188–1195. https://doi.org/10.1111/jep.13292

29.

Smith

M. J.

Pinto

R. M.

Dawalt

Smith

J. D.

Sherwood

Miles

Taylor

J. L.

Hume

Dawkins

Baker-Ericzen

Frazier

Humm

Steacy

(2020b). Using community-engaged methods to adapt virtual reality job-interview training for transition-age youth on the autism spectrum. Research in Autism Spectrum Disorders, 71, 101498. https://doi.org/10.1016/j.rasd.2019.101498

30.

Smith

M. J.

Sherwood

Blajeski

Ross

Smith

J. D.

Jordan

Dawalt

Bishop

Atkins

M. S.

(2021a). Job interview and vocational outcomes among transition-age youth receiving special education pre-employment transition services. Intellectual and Developmental Disabilities, 59(5), 405–421. https://doi.org/10.1352/1934-9556-59.5.405

31.

Smith

M. J.

Sherwood

Ross

Smith

J. D.

Dawalt

Bishop

Humm

Elkins

Steacy

(2021b). Virtual interview training for autistic transition age youth: A randomized controlled feasibility and effectiveness trial. Autism: The International Journal of Research and Practice, 25(6), 1536–1552. https://doi.org/10.1177/1362361321989928

32.

Smith

M. J.

Smith

J. D.

Blajeski

Ross

Jordan

Bell

M. D.

McGurk

S. R.

Mueser

K. T.

Burke-Miller

J. K.

Oulvey

E. A.

Fleming

M. F.

Nelson

Brown

Prestipino

Pashka

N. J.

Razzano

L. A.

(2022). An RCT of virtual reality job interview training for individuals with serious mental illness in IPS supported employment. Psychiatric Services, 73(9), 1027–1038. https://doi.org/10.1176/appi.ps.202100516

33.

Smith

M.J.

Smith

J.D.

Jordan

Sherwood

McRobert

Ross

Oulvey

Atkins

M. S.

(2021c). Virtual reality job interview training in transition services: Results of a single-arm, noncontrolled effectiveness-implementation hybrid trial. Journal of Special Education Technology, 36(1), 3–17. https://doi.org/10.1177/0162643420960093

34.

Strengthening Career and Technical Education for the 21st Century Act , Public law 115-224, 20 U.S.C. § 2353 (2018).

35.

Sullivan

S. D.

Mauskopf

J. A.

Augustovski

Jaime Caro

Lee

K. M.

Minchin

Orlewska

Penna

Rodriguez Barrios

J. M.

Shau

W. Y.

(2014). Budget impact analysis—principles of good practice: report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force. Value in Health: The Journal of the International Society for Pharmacoeconomics and Outcomes Research, 17(1), 5–14. https://doi.org/10.1016/j.jval.2013.08.2291

36.

United States Department of Educadtion (2017). A transition guide to postsecondary education and employment for students and youth with disabilities. United States Department of Education. https://sites.ed.gov/idea/files/postsecondary-transition-guide-may-2017.pdf

37.

United States Department of Education (2019). Rehabilitation services fiscal year 2019 budget request. United States Department of Education. https://www2.ed.gov/about/overview/budget/budget19/justifications/i-rehab.pdf.

38.

Workforce Innovation and Opportunity Act , Public Law 113-128, 29 U.S.C. § 3164 (2014).