Abstract
BACKGROUND:
The Rehabilitation Act, as amended in the Workforce Innovation and Opportunity Act of 2014 (WIOA), highlights the need to improve outcomes for transition-age students and youth with disabilities. The amendments require state vocational rehabilitation agencies to allocate 15% of their federal funding to provide pre-employment transition services (Pre-ETS) to an expanded population of transition-age students and youth with disabilities. Rapid implementation of this mandate required state vocational rehabilitation agencies to begin offering the five required Pre-ETS before mechanisms for monitoring and improvement could be created and field-tested. Nearly a decade since the passage of WIOA, many states are still without practical mechanisms for evaluating Pre-ETS despite increasing emphasis on using evidence-based practices in public programs.
OBJECTIVE:
The objective of this article is to present insights from the systematic implementation of The Transition Readiness Toolkit (TRT) within a state vocational rehabilitation agency.
METHODS:
We detail the creation of the TRT, our approach for implementing it across multiple states, and how it supports data-driven decision making for Pre-ETS providers and agencies.
RESULTS:
Initial results of implementation are promising. State VR agencies and Pre-ETS providers are using the data to increase accountability and understand the impact of their services.
CONCLUSION:
We present reflections on use of the TRT in a state agency and implications from an evidenced-based policy framework.
Keywords
Introduction
Underemployment and unemployment rates of transitioning students with disabilities is a perennial problem (Benz et al., 2000; National Council on Disability, 2000; Schutz et al., 2022). Employment disparities are experienced for youth with disabilities during and at exit from high school and do not abate with time (Cheng Shaewitz, 2021; Newman et al., 2011; Winsor et al., 2022). Early disability-employment interventions designed to help transition-age youth with disabilities develop skills and habits for adult life is crucial to increase their chance of (a) future employment, (b) accessing post-secondary education (e.g., apprenticeship programs, skill training certificates, vocational-technical school training, two-and four-year colleges and universities), (c) maximizing lifetime earnings, (d) increasing health and well-being, and (e) feeling increased self-efficacy and stronger self-concept as they move from adolescence to emerging adulthood (Iwanaga et al., 2021). Transition services are provided under the assumption that introducing employment skills, exploration a range of occupations, and facilitating a range of work experiences while in high school are correlated with success after graduation (Mazzotti et al., 2016; Taylor et al., 2022; Wehman, et al., 2015, Wehman et al., 2020). Initial data supports this tenet, showing that earlier and increased engagement with State Vocational Rehabilitation Agency (SVRA) services is predictive of better long-term employment outcomes for youth with disabilities (Carlson et al., 2020; Jun et al., 2015).
WIOA and efforts to improve transition outcomes through Pre-ETS
The Workforce Innovation and Opportunity Act (WIOA) of 2014 and the Rehabilitation Act of 1973, as amended by WIOA, mandated a substantial investment of time, money, and effort in transition-age youth with disabilities receiving vocational rehabilitation (VR) services (U.S. Department of Labor, 2016). Prominent among these mandates is the requirement for SVRAs to allocate 15% of their federal funding to provide a set of core pre-employment transition services (Pre-ETS) including (a) job exploration counseling, (b) self-advocacy instruction, (c) work-based learning, (d) counseling on postsecondary enrollment, and (e) workplace readiness training. The Rehabilitation Act Amendments in WIOA also expanded the target population, specifying that Pre-ETS are available to students who are eligible, or potentially eligible for SVRA services, markedly increasing the number of students who stand to benefit from these services. Only students with disabilities ages 14–21 are eligible to receive Pre-ETS unless superseded by the state to serve those younger or older than the federally established age. WIOA mandates were also designed to facilitate increased collaboration between Local Education Agencies (LEAs) and SVRAs in the shared goal of supporting students in achieving competitive integrated employment post high school. As a result, in the years following WIOA implementation, data indicate that VR counselors provide more outreach to LEAs and serve many more students than prior to WIOA (Fabian et al., 2018; Rehabilitation Services Administration, 2022).
The timeline from WIOA passage to implementation was brief, with SVRAs expected to function under the new mandates within months (National Council on Disability, 2017). SVRAs used “general expectations” provided by federal policy and mandates to implement required Pre-ETS, specify the target populations, and to determine VR counselor responsibilities in coordination (Carlson et al., 2020, p. 44). Many of the specifics of how to provide and evaluate Pre-ETS were left to the SVRAs. Carlson et al. (2020) analyzed state plans and other SVRA-supplied policy documents from 2018 to understand how states were implementing Pre-ETS and the degree to which they were meeting the legislative requirements as specified in WIOA. Nearly all states in the sample (n = 38), offered at least some description of the required Pre-ETS core services. However, authors concluded that SVRAs “varied considerably” in how, and to what extent, state policies addressed requirements dictated in the WIOA policy (Carlson et al., 2020, p. 55). These findings were replicated and extended by a later analysis of state plans where researchers found SVRAs “varied greatly” in their descriptions of Pre-ETS, their plans for service delivery, and the skills targeted (Taylor et al., 2022, p. 67). Ten years after WIOA and the establishment of Pre-ETS, the majority of states lack a clear and operational definition of the purpose and desired outcome of each core service. This lack of operationalization has limited SVRAs in their ability to evaluate and monitor services.
Current evaluation and monitoring of Pre-ETS
Enhanced requirements for coordination and funding contained in WIOA directly address longstanding weaknesses in transition services identified in previous iterations of education and workforce legislation (Baer et al., 2011). However, WIOA did not sufficiently operationalize benchmarks for whether services have the intended outcomes or even what those outcomes should be outside of the long-term outcome of employment. One exception in WIOA came in the reporting of measurable skills gains. Measurable skill gains loosely measure the effectiveness of Pre-ETS and transition services without being tied to a specific service. Measurable skill gains are defined as documented progress in one area including, (a) achievement in educational functioning level, (b) attainment of secondary diploma, (c) post-secondary transcript or report card documenting enrollment, (d) satisfactory or better progress towards established training milestone, or (e) successful passage of an exam required for an occupation or progress toward attaining skills (Performance Accountability Under Title I of the WIOA, 2017). Training milestones and skill gains are both defined as enrollment in a post-secondary training or education program that leads to a recognized post-secondary credential. These measurable skill gains are captured alongside case management data that includes whether a student received Pre-ETS, the date Pre-ETS started, and the date exited. Emphasis is also given to whether students receiving Pre-ETS fall in the category of potentially eligible or have enrolled in VR services. Agencies are also required to show that they are spending no less than 15% of their federal allotment on Pre-ETS.
The Rehabilitation Services Administration (RSA) is mandated to monitor SVRA performance, including Pre-ETS (Rehabilitation Services Administration, n.d.). While SVRAs retain responsibility for Pre-ETS coordination and provision, RSA gathers information from SVRAs, LEAs and contracted providers for Pre-ETS in their monitoring activities. These data are used to determine areas of strength and needed improvement for SVRAs related to Pre-ETS (Rehabilitation Services Administration [RSA], 2023). RSA provides the Monitoring and Technical Assistance Guide that establishes the basis for reporting standards. Values guiding the monitoring process include continuous improvement of VR programming, maximizing outcomes for VR participants, and ensuring efficient and effective use of public funds and personnel time as valuable agency resources (RSA, 2023). The most recent guide highlighted a trend of SVRAs returning increasing amounts of unused funds and RSA responded with added emphasis on financial management and investment.
The Monitoring and Technical Assistance Guide includes multiple questions for SVRAs to address in their agency profile. A section devoted to Pre-ETS includes policies and procedures used to meet WIOA requirements regarding target population, resource allocation, and how SVRAs are providing required services. SVRAs provide information through case review that includes policy documents and evidence of how policies are applied in practice. A major limitation, particularly noted in critiques of attempts to use RSA administrative data for program evaluation, is that RSA 911 data does not include any details of the services or training programs beyond a binary indication of whether or not they were received. Current measures of student change resulting from Pre-ETS are limited to reports of employment outcomes, credential attainment, and measurable skill gains that often occur months to years after a Pre-ETS was provided (WIOA, 2014). There is no requirement for states to identify duration of services, what the services included, or what changed in the immediate aftermath of services. As a result, most SVRAs do not collect this data across all students receiving Pre-ETS. Additionally, employment outcomes, credential attainment and skill gain data are not available for students who do not go on to open a case with VR and receive individualized services. Systematic measurement of change from Pre-ETS has the potential to enhance the quality of services and to improve overall outcomes (Mazurek Melnyk Morrison-Beedy, 2018).
Raising the bar on best practice
Recent federal legislation has increased the focus on program evaluation, quality assurance, and data-driven decision making (Anderson et al., 2021). Through WIOA (2014), SVRAs have seen an increase in reporting requirements focusing on “results-driven outcomes” (Vocational Rehabilitation Technical Assistance Center for Quality Management, 2021, p. 1). Section 116(e)(1) of WIOA encourages evaluation that is rigorous enough to “promote, establish, implement, and utilize methods for continuously improving core program activities in order to achieve high-level performance within and high-level outcomes from, the workforce development system.” Section 116(e)(2 3) goes on to state that “evaluations shall use designs that employ the most rigorous analytical and statistical methods that are reasonably feasible . . . ” with the goal of promoting “the efficiency and effectiveness of the workforce development system.” This language included in the section on evaluation within state agencies invites a level of rigor that far exceeds the basic reporting of what services were received.
This language from Section 116 of WIOA (2014) closely aligns with the more recent Foundations for Evidence-Based Policymaking Act of 2019 (Evidence Act), which was signed into law January 14, 2019. A primary aim of this legislation is to improve the use of evidence and data in the development and implementation of government policies. Among other things, this law mandates (a) each federal agency to designate a Chief Data Officer responsible for managing and promoting the use of data within the agency, (b) the development and implementation of a comprehensive federal data strategy to guide use, management, and sharing of data, and (c) use of rigorous evaluation methodologies that allow agencies to assess the effectiveness of government programs and to facilitate data-driven decision-making. The Evidence Act also encourages the adoption of open data practices where appropriate. The Evidence Act has already increased the emphasis and investment in enhancing equity and employment of people with disabilities through efforts being made both through the Departments of Labor and Education (Vilsack, November 2021).
This trend towards greater accountability and evaluation in government programs has continued since passage of the Evidence Act (2019). For instance, in 2022, the Biden-Harris Administration launched the Year of Evidence for Action with the intent being to strengthen and expand evidence-based government policymaking (White House, 2022). Activities for this Year of Evidence for Action included identifying and creating strategies to promote evidence-based decision-making and involved a partnership between the federal government and several other non-profit organizations and educational institutions (White House, 2022). More information on these more recent efforts can be found at the following website: https://www.evaluation.gov.
In the section that follows we introduce the Transition Readiness Toolkit (TRT) as a mechanism for enhancing transition outcomes for students receiving Pre-ETS. The TRT aligns with actions encouraged by WIOA and the Evidence Act to strengthen the evaluation of services and to provide data necessary for data-driven decision-making. Beyond describing the existing TRT, we describe the processes for its creation that align with the 13 key practices recently recommended by the Government Accountability Office (GAO, 2023), namely to: (a) demonstrate leadership commitment, (b) promote accountability, (c) build and maintain capacity, (d) involve stakeholders, (e) define goals, (f) identify strategies and resources, (g) assess the environment, (h) assess the sufficiency of existing evidence, (i) identify and prioritize evidence needs, (j) generate new evidence, (k) use evidence to learn, (l) apply learning to decision-making, and (m) communicate learning and results.
Methods
The TRT includes three core features to facilitate data-driven decision-making. Specifically, it includes (a) 10 assessment tools that are closely mapped to the five core Pre-ETS, (b) an online software offering real-time data reporting and visualizations, and (c) a simple and intuitive design that facilitates continuous improvement with inclusion and accessibility at the forefront. The combination of these features provides a solution to understanding service effectiveness and developing capacity for evidence-based practice in Pre-ETS that is available to both administrators and providers. We discuss the development and features of the assessment tools and the software below.
Initial development and implementation
The idea for the TRT stemmed from efforts of the Utah State Office of Rehabilitation (USOR) to incorporate pre-post assessment in the evaluation of Pre-ETS being offered throughout the state. USOR, like many SVRAs, demonstrated a commitment to improving evaluation of Pre-ETS and sought to promote accountability for their effectiveness; in doing so, they embody two of the foundational key practices put forth by the GAO (2023). USORs efforts to implement pre-post assessment were motivated by clearly defined goals that led to the assessment of the sufficiency of existing evidence and identification of a need for new evidence (GAO, 2023). A collaboration between researchers and USOR began in early 2020 with an initial focus on the key practice of identifying strategies and resources, assessing the environments where Pre-ETS are provided, and involving stakeholders (initially including stakeholders in Utah and expanding in 2022 to include stakeholders from five additional states). This months-long process resulted in a determination that USOR and other SVRAs would benefit from a centralized approach for evaluating change resulting from Pre-ETS and that no existing system adequately did so (Phillips et al., 2022).
A major challenge for SVRAs seeking to evaluate change resulting from Pre-ETS stemmed from definitions that vary across states and are often not conducive to evaluation (Taylor et al., 2022). Creating greater specificity about (a) what constituted each core service, (b) essential elements of each service, (c) how core services were distinguishable, and (d) the immediate outcome from services was necessary in order to create assessments that aligned with each core service. The assessment tools took over a year to create and involved intensive collaborations with USOR, community providers, and other key stakeholders. After initial piloting, USOR began a statewide use of the assessment tools in November 2021, marking initiation of the key practice the GAO refers to as generating new evidence (GAO, 2023).
Through a NIDILRR funded project, in 2022 we engaged SVRAs and their stakeholders from five additional states representing different regions of the country to ensure transferability beyond the USOR. As part of this project, we reviewed the concept map of the assessments, their target outcomes, and each item created for the assessment tools with agency staff and a panel of parents and transition students from each of the five new states. This critical step helped to refine the instruments while also confirming that the assessments and items matched the target outcomes for Pre-ETS in other states. With priority for continual improvement, we have maintained a high level of engagement with stakeholders to refine the assessments over the past three years. This continued engagement with stakeholders from across six states resulted in a revision and improvement to the assessment tools that was implemented November 1, 2023.
Table 1 provides a brief description of the 10 assessments (nine pre-post and one post-only) and how they align with the WIOA core Pre-ETS. While most core services were more readily defined based on the laws, regulations, and guidance documents, workplace readiness required more assessments to capture the multiple facets of this core service as described by stakeholders. The other core service with multiple assessments is work-based learning; the additional post-only assessment aligns with SVRA priorities to gauge student perceptions of their work experiences and track quality metrics including wages and completion rates.
TRT instrument and alignment with Pre-ETS core services
TRT instrument and alignment with Pre-ETS core services
Originally, we collected data from the assessments using a popular data collection software. This meant that for USOR and the providers to access the data, members of the research team needed to download the data, analyze it, and create custom reports for USOR and each Pre-ETS provider. This process was labor intensive and only allowed for quarterly reporting of the Pre-ETS evaluation data. Even when USOR was the only SVRA, building and maintaining capacity to use and apply the data adequately to facilitate data-driven decision-making, key practices of the GAO (2023), required a new approach. In February 2022, members of the research team enlisted outside resources to create a software for data collection, data analysis, and real-time reporting of Pre-ETS data. This software is now being used by all SVRAs to allow for the type of use, application, and communication that fosters identification and use of evidence-based decision-making that aligns with the aspirations of WIOA (2014) and the Evidence Act (2019). Through the TRT, SVRAs, in-house Pre-ETS providers, community providers, and educators providing Pre-ETS have access to customized reports and visualizations. This real-time reporting provides intuitive insights that can enhance Pre-ETS through the identification of best practices and the improvement of existing services.
Beyond the collection, analysis, and reporting of Pre-ETS change data, SVRAs often struggle with capacity issues related to training and competency; and the systematic adoption of pre-post assessment is a new concept in most if not all SVRAs. Providing training and technical assistance to produce understanding and approval of the TRT has been a critical aspect of building and maintaining capacity. In our training to providers and administrators, we emphasize the value of pre-post assessment, and how users can leverage this data to improve their services. Maintaining a well-informed and approving set of providers is no small undertaking given the amount of turnover that exists across provider agencies. There is often a need to onboard new staff or re-train existing staff, and technical assistance is available as an ongoing support. Despite the challenges of training, provider agencies (including school staff and community Pre-ETS providers) are among the most supportive stakeholders for the enhanced accountability and insights that come through pre-post assessment. The initial rollout across states suggests that Pre-ETS providers take pride in their services and many welcome the opportunity to measure the immediate impact on students.
Accessibility and usability
Accessibility and usability were central in the creation of the TRT (both the assessment tools and the software). Our team worked closely with stakeholders from across the six states to ensure that the environment where the TRT would be used was understood and that the needs of providers and recipients were met. We made multiple revisions to the assessment tools, the software, and to the process of administration to maximize clarity, ease of use, and customization. Some of the primary features included ensuring that assessments were written at or below a 4th grade reading level for greater access. We also created an alternative format that can be completed by an observer when self-report is not possible.
All assessments can be completed through an accessible online software or using a paper and pencil version to meet unique agency and recipient needs. By embedding accessibility into the assessments and providing multiple methods for administration, we sought to optimize availability to students regardless of ability or access to technology. Finally, we capped the number of items for each assessment (ranging from 8 –18 items) to make adoption more practical for busy service providers and to reduce the load on students receiving Pre-ETS. A review of submission data suggests that on average, assessments are completed in less than two minutes. The resulting assessment battery is usable by the population, practical to the setting, and tailored to the service expectations (Phillips et al., 2022).
Results
To date, we have collected more than 12,000 student responses across the 10 TRT assessment tools, with the vast majority of responses coming from USOR. To enhance usability, all TRT assessment tools are rescaled from 0 (lowest possible score) to 100 (highest possible score). When a student or observer completes an assessment, the software is automated to identify and log the entry as a pre- or post-assessment and to provide the results and scoring in real time. When a post-assessment is recorded, a change score is automatically calculated by subtracting the pre-test score from the post-test score. Providers have the option to use the assessments as a screening tool for determining what Pre-ETS are most needed prior to starting a service. Providers also have the option to administer an unlimited number of post-assessments for each service. The most recent post-assessment automatically replaces previous post-assessments in calculations of the change score. This latter feature allows agencies to make data-driven decisions for when to conclude a specific service.
In the spirit of evidence-based practices, we acknowledge there is still much to be learned about the costs, benefits, reliability, and validity of the TRT. Multiple additional analyses are underway to evaluate these aspects and more. That said, the intentional and collaborative three-year process used to develop the TRT combined with strong internal consistency data (Cronbach’s alpha values ranging from .756 to .911), has buttressed initial confidence as more data is collected and analyzed. Among SVRAs using the TRT, we are seeing regular use of the data, specifically the number of students in services and the change scores, for monitoring and technical assistance. This frequent use of the TRT data indicates the GAO key practices of using the evidence to learn and applying learning to decision-making (GAO, 2023).
Initial data from USOR (other SVRA data excluded for the purposes of this article) provide readily available insights about the services being provided and their effectiveness in creating immediate change (see Table 2).
Descriptive statistics for TRT scales
Descriptive statistics for TRT scales
Note. TRT = Transition readiness toolkit; WBLE = Work-based learning experience; CPSE = Counseling on postsecondary enrollment; CSS = Communication and social skills; ILOM = Independent living, orientation, and mobility. 1The difference score is only calculated for students who have both a pre- and post-test. 2The ILOM included seven items in the original assessment tool and was updated to include 14 items on November 1, 2023.
Notably, when aggregated across all providers, students are experiencing positive change from initiation to completion of Pre-ETS across all five core services. The SVRA can filter this data in multiple ways, including by provider agency. Multiple other data views and visualizations are intended to support SVRAs, provider agencies, and the direct Pre-ETS providers in understanding and improving the services they offer students with disabilities.
Perhaps even more impactful to building an evidence-base for Pre-ETS is the ability for providers to search for individual students to evaluate their baseline scoring and trajectory of change. Providers can use this real-time data to evaluate what aspects of each core service to emphasize, when to begin each service, whether services are resulting in the desired change, and when to conclude a service. By making this data readily accessible and easy to interpret, SVRAs have the tools for more effective monitoring and the capacity for customized training and support. Direct service providers and their agencies have the tools necessary to evaluate trends in student outcomes.
The TRT is the result of a multi-year development process, with ongoing and intentional review and feedback from key stakeholders. Nonetheless, instrument development is a lengthy process, and we continue to engage in efforts to validate and strengthen the TRT. Our key efforts to date have focused on content and face validity, primarily through our alignment with WIOA-defined core services and gathering input from key stakeholders to ensure that the instruments are appropriate and useful for the intendent purpose. Our next validation efforts will include applications of item analysis, driven by classical test theory to examine reliability and validity of individual items. We will aim to establish concurrent validity of TRT measures, using a correlational study design against comparable accepted measures and external milestones that correspond with the desired outcome of services.
We are eager to connect this data with case data to understand trends in student outcomes. Merging our TRT data with existing case data (i.e., RSA 911) will allow for understanding trends in pre-post skill gains in Pre-ETS as they relate to accessing post-secondary education and employment. We will test for differential test functioning according to key demographics to ensure usability across our intended population. Finally, we will also be able to explore patterns in service outcomes related to demographic factors, and features of services such as length, ordering, or other elements of training that may elucidate promising practices.
Discussion
We have, to this point, described the need for enhanced evaluation of Pre-ETS and cited recent legislation that affirms this need for government agencies and for SVRAs specifically. We have also described the TRT and how it has facilitated the type of continuous improvement encouraged in recent legislation (i.e., WIOA of 2014 and The Evidence Act of 2019). As noted, the GAO (2023) recently suggested 13 key practices for developing and using evidence to support service provision. Among these 13 practices, four are considered crucial to fostering a culture of learning and continuous improvement, namely: (a) demonstrating leadership commitment, (b) promoting accountability, (c) building and maintaining capacity, and (d) involving stakeholders. We use these four factors as a basis for sharing some of our own reflection on implementing the TRT and policy implications for ensuring continuous improvement of Pre-ETS.
Demonstrated leadership commitment
Demonstrated commitment from agency leadership has been referred to as “the single most important element of successfully managing and improving the performance of federal organizations” (GAO, 2023, p. 39). Commitment can take the form of vocalizing evidence-building as a priority, spending time and resources on evidence-building activities, actively creating buy-in within an agency, and coordinating efforts across teams and agencies.
Reflection
The USOR provided an ideal model for demonstrating leadership commitment from the moment they initiated their own process internally to the present. The five additional SVRAs that have supported development of the TRT and others that have expressed an interest or even attempted to improve the evaluation of Pre-ETS also deserve recognition. SVRAs that evaluate outcomes are not guaranteed to like what they find. This risk did not deter any of the SVRAs that have adopted the TRT. To the contrary, USOR and some of the other SVRAs have made the pre-post assessment a mandatory aspect of service provision. Requiring this kind of engagement had a powerful influence on provider agencies and on their adoption of this form of enhanced evaluation. This approach was particularly important in early implementation when providers had no evidence of benefits to using the TRT and may have viewed it as simply an extra task. USOR and other SVRAs have also demonstrated leadership in their approach to the TRT, with SVRA leadership speaking of the TRT as an opportunity for growth and improvement. As part of this approach, SVRAs have conveyed the messaged that data reports will not be used punitively in cases where providers are attempting to learn and grow from the findings and any accompanying monitoring. This non-punitive method appears to have facilitated buy-in from providers and supports an environment where evidence-based decision making can thrive.
Implications
Many SVRAs have already engaged in processes to enhance the evaluation of Pre-ETS. It is understandable, however, that others have not given the many requirements of the job across the various positions of SVRAs. As previously noted, even among SVRAs with the bandwidth to enhance evaluation, the risk of poor results could create concerns over political and social repercussions. A possible solution to increase the commitment of SVRA leadership is for the federal government to require measures of change resulting from Pre-ETS. The TRT serves as evidence that multiple states can coalesce around agreed upon measures of change and that effective pre-post assessment can be systematically adopted by service providers regardless of training or background.
Promoting accountability
From an evidence-based policy perspective, pursuing accountability demands assignment of responsibility and holding individuals accountable for learning and results (GAO, 2023). SVRAs are already responsible for Pre-ETS delivery and report metrics associated with Pre-ETS to RSA annually (RSA, 2023). As discussed, current data are limited in their utility for understanding quality and impact of services. Pre-post assessment provides the opportunity to garner data that can be used to understand performance in a manner that promotes increased accountability for Pre-ETS.
Reflection
Having access to pre-post assessment data, in the form of change scores, has increased accountability for Pre-ETS at USOR. As noted in USOR contracts, providers are responsible for recording as many pre- and post-assessments for students as possible. Exceptions are made in cases where it is too burdensome for the student, although this is rare. In turn, state administrators, administrators from other provider agencies, and direct service providers (those working in the SVRA and those providing Pre-ETS for the SVRA) have access to real-time reporting of change scores for students receiving Pre-ETS. Data reporting and visualizations are customized to each agency and level of authorization. Administrators tasked with overseeing Pre-ETS for the SVRA have access to aggregate and individualized data across all providers and agencies working directly with the SVRA. Agencies providing Pre-ETS on behalf of a SVRA have access to aggregated data for the entire state that provides a benchmark for services as well as the aggregated and individualized data specific to their agency. Agencies providing Pre-ETS on behalf of SVRA intentionally do not have full access to similar agencies working for a SVRA. Agency staff also have access to the quantity of pre- and post-tests by service, to be able to understand service volume. Data is integrated into site visits and other technical assistance opportunities with providers. In an important first, SVRA leadership can engage in data-informed monitoring based on the degree to which students are achieving targeted outcomes. The data also allows for identifying high-performers and best practices among providers based on empirical measures of change.
Agencies providing services on behalf of SVRAs have also expressed appreciation for and benefits from adopting pre-post assessment to enhance their own accountability. When engaging stakeholders to define the target outcomes of each core service, these provider agencies expressed frustration with the lack of clear outcomes in Pre-ETS. Unlike other VR services where employment is a relatively short-term outcome, providers do not have an immediate target for outcomes of Pre-ETS. These same providers expressed appreciation for the guidance provided by the TRT, where each service has a description and a set of target outcomes associated. Rather than waiting to be monitored with the data, agencies providing services on behalf of SVRAs are using the data to inform curriculum and service design and to customize individual student services (e.g., what services are most needed, when a service is complete). This shift in their practice reflects their commitment to being accountable for their own learning and outcomes (GAO, 2023).
Implications
Language from WIOA and the Evidence Act both demand more rigorous approaches to evaluation of Pre-ETS than are currently possible given the available data. From an accountability standpoint, where data is insufficient for quality decision-making, it is necessary to “improve the availability and quality of the evidence” (GAO, 2023, p. 41). Simple metrics of monetary investment or volume of services do not adequately promote accountability for Pre-ETS because these measures take no account for quality or effectiveness. Individualized pre-post-assessment offers greater rigor in understanding service impact and promotes continuous improvement. Data can be used to highlight providers achieving strong change scores, allowing for celebration, and sharing of promising practices with the hopes of being able to replicate across providers and sites to establish better outcomes for students. Data can also be used for technical assistance, to support providers who are not seeing as much change to consider what they might adjust to improve their services.
In summary, Pre-ETS assessment is an area where current practice is not living up to the accountability ideals set forth in WIOA (2014) or the Evidence Act (2019). SVRAs are responsible to meet monitoring standards set by RSA. There is no need for states to go beyond this reporting, and currently the standards lack any measures of change. Improving accountability for Pre-ETs will necessitate re-examining how outcomes are considered and what methods are used to determine service effectiveness. Only then will we have credible assessments of progress towards identified target outcomes of services, identified as a key indicator of accountability (GAO, 2023).
Building and maintaining capacity
Building and maintaining capacity for evidence-building involves making sure that staff have knowledge, skills, and abilities to carry out the work required. More complex systems or requirements for evidence-building demand greater knowledge, skills, and abilities. Capacity also includes having the time and resources to carry out the necessary activities. Given the significant time constraints experienced by Pre-ETS providers, the resource of time cannot be overlooked. Insufficient funding may also limit capacity to enhance evaluation. Recent research has clearly indicated that limited capacity is a primary challenge among state agencies seeking to build greater evidence for services (Allard et al., 2018).
Reflections
The building and maintaining of capacity for the TRT has been a growth process for the research team, USOR, and the participating providers. In the early stages of development, the research team planned to help SVRAs build capacity to collect, analyze, and report out Pre-ETS change data. However, only reporting the data quarterly did not allow for full utilization of the data in service decisions. We also recognized that building internal SVRA capacity to collect, analyze, and report data so that it could be used to inform services would be challenging, and maybe not feasible, given the multitude of other program evaluation needs and reporting requirements. Automation provided a solution for both issues. External resources were garnered to build the platform currently in use for data collection, real-time data reporting, and easy to use visualizations. Another capacity issue we encountered was training. As the data must be collected by providers, training was needed to help staff understand the value of doing pre- and post-assessment in order to establish buy in and create greater likelihood that the TRT would be implemented across multiple sites. Providing an initial training and onboarding proved to be manageable and ensured that provider staff knew when and how to use the TRT with students. However, as previously noted, high levels of staff turnover in community providers and SVRAs makes the maintenance of well-trained users much more challenging. The solution currently being implemented is to require new users to complete a brief set of online modules and then to sign up for a live follow-up meeting prior to gaining full access to the TRT.
Implications
Quality evaluation comes at a cost. However, we argue that poor or limited evaluation is actually much more costly in the long run. In addition to mandating measures of change resulting from Pre-ETS, we recommend increased resources for development and maintenance of this type of evaluation. We argue, based on our experience with the TRT, that SVRAs would be wise to seek out cross-state mechanisms to support capacity for evaluation rather than each SVRA going through the expenditure to maintain their own evaluation process from scratch. We recommend this because of the potential it creates for SVRAs to benefit from direct comparisons of data across states and for the potential cost-savings of using and maintaining a product across multiple SVRAs. The sudden growth of artificial intelligence and the advancement of statistical technology will surely also play a role in the future of evidence building in a manner that we expect to reduce the overall capacity requirements for public agencies, although not without significant ethical tensions (Madan Ashok, 2023; Pencheva et al., 2020).
Involving stakeholders
The importance of involving multiple stakeholders in transition service delivery has long been reflected in legislation (Oertle Seader, 2015), and is most successful when it occurs early and often (GAO, 2023). The last reauthorization of the Individuals with Disabilities Education Act (IDEA, 2004) defined transition services as a coordinated set of activities that are results-oriented and are designed to facilitate movement of a child with a disability from school to employment, post-secondary education, independent living, and/or participation in the community (20 U.S.C. 1401[34]). The 2014 WIOA places considerable emphasis on transition-age students with disabilities and requires states to submit a plan of rehabilitation services that outlines how the state will meet the provisions outlined in the Act. The state plan must contain policies and procedures for cooperation, collaboration, and coordination between the state rehabilitation and state education agencies developed to facilitate transition students from educational services to rehabilitation services (Rehabilitation Act, Title 1 Section [a] [11] [D]). Despite federal legislation for both special education and rehabilitation that emphasizes coordination among state agencies, the coordination often focuses on state level compliance and data sharing, with little emphasis geared toward capacity building and evaluation.
Reflections
Guidance provided by the TRT change scores allow for more robust collaboration and meaningful stakeholder involvement. We have found the use of a set of common indicators to evaluate student progress (provided through the TRT) creates a shared language and understanding of common goals that supports coordination and collaboration among key stakeholders (e.g., educators, VR counselors, job coaches). Educators and VR counselors, for example, now have a mechanism for aligning common goals and objectives for employment and post-secondary education and use data from the change scores generated by the TRT to facilitate discussions about the extent to which students are achieving desired skills and growth. When key stakeholders understand student performance using a common metric, these stakeholders can engage in more meaningful collaborations regarding student goals and make more effective decisions about the efficacy, quantity, and duration of interventions. For instance, data being generated from the TRT can be included in individualized education programs (IEPs) to meet special education requirements for conducting assessments to develop age-appropriate post-secondary goals and objectives.
Implications
Providing meaningful opportunities for key stakeholders to review assessment data allows SVRAs to move beyond simply meeting monitoring standards established by RSA. Engaging stakeholders in the evaluation process helps build service delivery capacity at both administrative and provider levels. The GAO (2023) recommends that ongoing, data-driven reviews regularly occur, and we recommend that SVRAs dedicate staff time and effort to lead these reviews with providers and LEAs, with the goal of using shared metrics for success wherever possible. Data-driven reviews will give SVRAs and stakeholders unbiased opportunities to review positive and negative change scores that can be used to build a shared understanding of the contextual and mediating factors impacting service delivery and what changes to process and interventions need to occur to build capacity at both the administrative and of provider level.
Conclusion
Recent Federal legislation has raised expectations for SVRAs to engage in high quality program evaluation, quality assurance activities, and data-driven decision making (Anderson et al., 2021). However, the available data currently used for monitoring and compliance is insufficient to build evidence for practice. Our review outlines some concerning issues in Pre-ETS and transition services that remain nearly a decade after the passing of WIOA. The limited guidance on Pre-ETS and approach to monitoring does not allow for understanding the impact of services and whether they are facilitating the kind of student growth that is intended. Current priorities of meeting the spending requirements for services and reaching as many students as possible are important but are not sufficient for meeting accountability standards as described by GAO best practices (GAO, 2023). The TRT, as developed in collaboration with key stakeholders (SVRAs, community providers, and LEA staff) and aligned with core Pre-ETS, is a promising approach to systematizing assessment and building capacity for data-driven service implementation and enhanced collaboration between SVRAs and key partners. USOR and other participating states are leading efforts to understand the impact of Pre-ETS and use resulting data to enhance accountability.
Footnotes
Acknowledgments
The authors would like to express their gratitude to the USOR leadership team, the agency leads involved in the NIDILRR grant, the parent-youth panel, and all providers and educators who have provided input and feedback on the TRT.
Conflict of interest
Drs. Fleming and Phillips have an outside financial interest in an entity that may benefit from the outcome of this research. This outside financial interest has been reviewed by their respective University’s Individual Conflict of Interest Committees and is currently being managed by the Universities.
Ethics statement
This study was approved by the Utah State University Institutional Review Board (approval number 13956).
Funding
This Field Initiated Project (FIP) on Assessing Transition Services at Utah State University is funded by the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR grant number 90IFDV0027). NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS). Products and content stemming from this project do not necessarily represent the policy of NIDILRR, ACL, or HHS, and you should not assume endorsement by the Federal Government.
Informed consent
Not applicable.