Continuous improvement in state vocational rehabilitation: A demonstration and call to action

Abstract

Background

State Vocational Rehabilitation Agencies (SVRAs) are the leading provider of employment services for disabled adults in the United States, with achievement of competitive integrated employment as their main outcome.

Objective

Current data collection policies and practices require counselors to record extensive information for case documentation purposes, however, these data offer limited utility for service planning or for determinations of whether services are having the intended impact. Available data is not structured to support continuous improvement for individual providers or to establish evidence of best practices in day-to-day service provision.

Method

In this article, we present the many potential benefits of including pre-post assessment of services in measuring the impact of SVRA services.

Results

The Transition Readiness Toolkit offers a concrete example of the advantages associated with pre-post assessment and continuous use of data to guide services and quality improvement in SVRAs.

Conclusion

Recommendations and implications for implementing pre-post assessment for other SVRA services are provided.

Keywords

evaluation vocational rehabilitation continuous improvement

The state vocational rehabilitation (VR) services system has one of the most comprehensive administrative datasets of any public program in the United States in the Rehabilitation Services Administration (RSA)-911 data. RSA-911 data is organized by cohort, representing all individuals who closed their VR cases in a given year. It includes more than 300 data elements capturing demographics, case status, services provided, length of services, and conditions of case closure for each customer (U.S. Department of Education, 2023). The collection and maintenance of this extensive dataset requires a significant investment of time and resources from state VR agencies (SVRAs) that demonstrates a strong commitment from both state and federal agencies providing oversight of VR services. This commitment is not surprising given the conspicuous and complementary trends of digitization, data science, data-informed practice, transparency, and accountability that are reflected across government and private institutions. Elements of these trends can be seen in the Government Performance and Results Modernization Act of 2010, Foundation for Evidence-Based Policymaking Act of 2019, and in the Workforce Innovation and Opportunities Act of 2014 (WIOA; See Fernández-Pascual et al., 2024; Fleming et al., 2024; Groomes & Adams, 2013; van Donge et al., 2022 for more description of these influences). However, the vast data elements and resources designated for gathering them have been extremely limited in shaping evidence-based policy and practices specific to how VR services should be conducted.

In this article we echo others in calling for a shift in the evaluation of state VR services from one almost exclusively focused on oversight to one that facilitates continuous improvement through the systematic measure of change resulting from services (Anderson et al., 2021a; Groomes & Adams, 2013; Groomes et al., 2015; Groomes et al., 2017; Sabella et al., 2018; Vandergoot et al., 2009). Such a transformation would benefit all stakeholders by enabling documentation of how customers benefit from VR services. This move would also increase understanding of who benefits from VR services and the degree of equity experienced between VR populations. We start by outlining the limitations of available data elements for identifying and using evidence-based practices and the shortcomings of the current data to inform individual customer services. We then describe an initiative (The Transition Readiness Toolkit), that has been adopted by multiple SVRAs to systematically measure the effectiveness of pre-employment transition services (Pre-ETS). We conclude the article with recommendations for SVRAs to strengthen and modify current evaluation practices across all essential services. We proceed under the strongly held assumption that customers experience multiple benefits from VR services. That said, without measures of change to demonstrate the impact of services, we are limited in our ability to identify best practices and improve the quality and effectiveness of services.

What we can and cannot know from RSA-911 data

Hundreds of published and unpublished studies have been conducted using the RSA-911 dataset (Boeltzig-Brown et al., 2017). These studies have contributed significantly to our knowledge of SVRAs, namely their efficacy, efficiency, equity, and compliance with federal legislation and regulations. For instance, studies have led to insights about whether customers from minoritized racial-ethnic groups, disability populations, and those with prior justice involvement offender statuses receive a similar amount of services to other customers (e.g., Alsaman & Lee, 2017; Ethridge et al., 2020; Ipsen et al., 2014), whether receipt of a specific service correlates with employment outcomes (e.g., Hill et al., 2022; Sprong et al., 2019), and whether SVRAs are compliant with timelines for eligibility determinations or development of an individualized plan for employment (e.g., Crudden et al., 2020; Donahue et al., 2024). A select few more sophisticated studies have paired RSA-911 data with other datasets to demonstrate the effectiveness of SVRA services using a matched comparison group who did not receive VR services (Dean et al., 2014; Leahy et al., 2014; O’Neill et al., 2015). Every year, new studies provide additional insights and updates about SVRAs and the customers receiving their services. That said, there are also many important questions that cannot be answered using RSA-911 data.

The RSA-911 data elements relating to SVRA services focus on (a) whether a service was received, (b) what agency provided it, (c) dates of initiation and completion, and (d) in the case of contracting services, the cost of services provided (U.S. Department of Education, 2023). Outcomes that are measured in connection with these services include measurable skills gains, credential attainment, competitive integrated employment, and earnings. As described more fully by Fleming et al. (2024), measurable skills gains “loosely measure the effectiveness of Pre-ETS and transition services without being tied to a specific service” (p. 199). In summary, RSA-911 data provides a snapshot of what services were received under an Individual Plan for Employment and whether the principal outcome of competitive integrated employment is achieved at their conclusion, with some measure of educational milestones captured in between.

For all of its strengths, RSA-911 data lacks essential information necessary for the identification of best practices and continuous improvement of VR services. Current service definitions do not guide characteristics or content of the service that would be expected, nor the desired gain associated with the specific service. Available data does not inform the dosage of a service that appears most effective in securing the desired gain, nor the level of skill that would best predict successful employment. Without this type of information, it is not possible to accurately evaluate how the results of a specific service relate to an outcome. These issues stunt the development of effective practice while also circumscribing what we can know about the benefits of services provided by SVRAs. Notably absent from the RSA-911 data is any information that directly captures improvement or change resulting from the specific services that were received. Identifying whether a service caused change for an individual customer requires an understanding of the level of skill or ability prior to a specific service and at completion of the service. Collecting more than 300 data elements, none capturing need for services or change resulting from them, means that direct providers experience the primary burden of data collection while experiencing little to no guidance or benefits from that data (Corash et al., 2023).

Method

Proposing systematic measures of change in VR

Pre-post assessment as an evaluation approach allows us to capture customers’ knowledge and skills before and after an intervention. It leverages principles of pre-experimental design, where a person or group is assessed, an intervention is provided, and then the person or group is assessed again to determine whether change occurred (Lutz et al., 2022). The assessment tool(s) align with the most salient indicators of what should change as a result of the intervention, referred to as the intervention target. When administered in close proximity to the intervention, counselors can reduce (although not eliminate) the likelihood that something besides the intervention caused any observed change to the intervention target. A reliable and valid pre-assessment can facilitate greater customization of services to account for the unique strengths and needs of each customer from the initiation of services. Used systematically and in combination with post-assessment, providers and other key stakeholders are able to evaluate the degree of change associated with an intervention and, over time, understand the most ideal dosage and duration of the intervention to achieve desired change. Systematic pre-post assessment can also offer insights about whether individual differences moderate treatment effects.

Pre-post assessment is more commonly leveraged in many related fields of education and allied health. For example, evaluating range of motion, strength, and balance before and after a physical therapy intervention or repeated blood draws to check for reductions in cholesterol or glycated hemoglobin (A1C) levels associated with nutrition counseling. Assessment data is then used to determine if an intervention has had the desired effect (i.e., treatment is complete), and if not, what desired outcome has not been met. Individual-level evaluations allow for modification of direct customer services while program-level evaluation is based on patterns across individuals that allows for understanding of whether the intervention is having the intended effect across a target population. SVRAs have not traditionally included systematic measures of effectiveness through pre-post assessment. However, we believe that meaningful pre-post assessment is possible within the VR system based on the successful implementation of the Transition Readiness Toolkit (TRT) across eight SVRAs representing different regions of the country.

Results

The Transition Readiness Toolkit

The Transition Readiness Toolkit (TRT; Phillips et al., 2022) captures the impact of Pre-ETS delineated by The Workforce Investment and Opportunity Act (2014). The TRT includes 10 pre-post assessment tools closely mapped to the five core Pre-ETS. The online software provides real-time data reporting and visualizations that facilitate data-informed service decisions and programmatic improvement (Fleming et al., 2024). To date, the TRT has captured approximately 21,000 student assessments. See Table 1 for additional information on the content of the 10 assessments. Information about the development of the TRT can be found in Fleming et al. (2024) and Phillips et al. (2022).

Table 1.
TRT instrument and alignment with pre-ETS core services.

WIOA core service Assessment Training description and sample topics

Job exploration counseling TRT job exploration scale Knowledge of available jobs; understanding of general requirements for jobs; knowledge of self, as it relates to job opportunities (interests, strengths); knowledge of what I want in a job (pay, types of tasks / interactions).

Work-based learning experiences TRT work-based learning pre-post scale Experience-based knowledge of general work skills necessary for successful employment. Self-evaluation of performance at work (pace of work, getting along with others, communicating with supervisors).

Work-based learning experiences TRT work-based learning post-only scale Students’ perspective of whether the work-based learning experience was a good fit for them, whether they finished the experience, and whether they were paid (and if so, how much).

Instruction in self-advocacy TRT self-advocacy scale Understanding myself and my disability, knowing my rights and responsibilities when it comes to requesting accommodations, assertiveness, problem solving, negotiation, effective communication, and self-efficacy.

Counseling on post-secondary education opportunities TRT counseling on post-secondary enrollment scale Knowledge of post-secondary opportunities and how to apply. Self-knowledge of interest for post-secondary training and how this relates to future work. Understanding of skills necessary for success in post-secondary education, e.g., study skills, time management, and educational supports / accommodation.

Workplace readiness training TRT communication and social skills scale Connecting, relating with, and respecting others; effective workplace communication; and understanding expectations.

Workplace readiness training TRT employability scale “Soft skills” such as responsibility, responsiveness, and seeking feedback.

Workplace readiness training TRT financial literacy scale Basics of budgeting, consideration of benefits and pay as they relate to employment.

Workplace readiness training TRT independent living, orientation, and mobility scale Wayfinding, safety, following a schedule, hygiene, getting your own food, and keeping your space clean

Workplace readiness training TRT job seeking scale How to find job opportunities, application, and interview skills.

Fleming, A. R., Phillips, B. N., Riesen, T., & Langone, A. (2024). Enhancing transition outcomes: A toolkit to facilitate data-driven Pre-Employment Transition Services. Journal of Vocational Rehabilitation, 60, 197–209. Reprinted with permission.

A few unique features of the TRT are key to its implementation within SVRAs and other transition agencies. First, unlike many other tools being used in transition, the assessments and features of the TRT were created specifically for youth with disabilities to be applied in transition settings. Assessments are brief, taking an average of only 2 to 3 minutes to complete. Multiple formats for administration ensure that all students receiving a Pre-ETS are represented. The data reporting and visualizations are created with busy providers in mind and without any prerequisite data literacy training assumed. The assessments also align with legislation and policy that guide Pre-ETS. Provider agencies using the TRT can access aggregated data that captures overall service effectiveness within their specific agencies as well as data showing the individual responses for each student receiving Pre-ETS. Each SVRA has access to aggregated and disaggregated data for all provider agencies within their jurisdiction, with the ability to view the data across all agencies or within a specific agency. Regular use of these data promotes a culture that is built around continuous improvement and anchored on quality of services rather than exclusively focusing on quantity of services or funding allocated. With the implementation of the TRT, providers and administrators alike are, for the first time, consulting empirical data to determine how to customize and improve services based on individual student strengths and areas for growth. This data also provides guidance that helps providers and administrators determine when services are completed and whether they had the intended impact. See Figure 1 for an example of the data views provided through the TRT.

Figure 1.
TRT instrument data views for an individual student.

TRT Benefits to pre-ETS provision and quality improvement

For most TRT users, this is the first time that they have had clear metrics to guide customization of Pre-ETS services and to assess their impact. Several direct service providers reported during the development phases of the TRT that even the process of clarifying and operationalizing the parameters and intent of the core services was helpful. Many of these providers indicated that, prior to the TRT, they had not thought of the different Pre-ETS as discrete service activities. Now, through the TRT, direct providers, community rehabilitation providers (CRPs), local education agencies (LEAs), and SVRAs using the TRT have a shared understanding of what a core service is and what is expected to change as a result. We describe below a few of the uses and benefits derived from systematic pre-post assessment. These include benefits to the service provider, the agency, the customer, and other key stakeholders. In combination, these benefits in measurement have enormous implications for service monitoring and service quality.

Benefits to service provision

Perhaps the most exciting result of TRT implementation has been the feedback from direct service providers. This group includes community rehabilitation providers (CRPs), local education agency (LEA) staff, and Pre-ETS specialists employed by SVRAs. Providers have indicated benefiting from the ability to (a) identify student strengths and needs at the initiation of services, (b) assess progress and make adjustments to services, and (c) use the data to facilitate collaboration with other key stakeholders.

Providers are able to identify transition strengths and needs for each student and across students. By starting services with a pre-assessment that is closely mapped to the core Pre-ETS services, providers are able to understand students’ existing strengths and needs in the topics or skills included in the training. This means that providers can customize their services to individual students in a manner that is expected to enhance outcomes. For example, if a student starting a financial literacy training identifies that they already track their spending but do not have knowledge of living expenses, this tells the provider which parts of their training they may be able to abbreviate and what topics they may need to emphasize. The provider can also use the post-assessment to see where their training has shifted a student's knowledge or skills and where additional growth may be expected with more training. Over time and with larger sets of data, providers will be able to determine how to sequence their services based on individual student data, taking into account student age and pre-assessment data indicating students’ strengths and needs in a given training area.

Providers can assess services that are in process to determine initial effectiveness and to make adjustments. Students in the 14–21 age range are a diverse set of learners coming into training with a variety of abilities, experiences, and support needs. As a result, students may respond differently to services, even when provided under similar conditions. The ability to assess how services are going before they are completed allows for adjustments and improvements to facilitate optimal outcomes. Providers report that they often use follow-up assessments in combination with their observations and student feedback to determine whether services are having the desired outcome. Providers are encouraged to use these data to check student progress, to adjust services, and to determine whether service goals were achieved. When nearing completion of services, providers are using the TRT post-assessment to verify whether sufficient growth occurred or whether services need to be continued. In the case of service completion, providers have a more precise view of the skills or knowledge areas where students reported growth through TRT data.

Providers can share service-level data with key stakeholders, including collaborating agencies. Transition is a complex subset of VR services due to the number of individuals that are often involved in transition plans and service teams. Collaborating between workforce and education systems to promote post-secondary employment success is mandated through legislation (Parker-Katz et al., 2018; Plotner et al., 2024). Collaboration partners extend to special education teachers and LEA staff, SVRA counselors and staff, and CRPs. Unfortunately, a major impediment to effective collaboration in transition services has been the lack of shared understanding, including shared data metrics, of how each team member can contribute to student outcomes (Oertle et al., 2021; Wehman et al., 2024). Having clear metrics that are closely aligned with the policy-defined skill and knowledge areas provides a shared language and even shared assessments for team members with different roles. This data also opens the potential to integrate student data with other systems employing measures of effectiveness or change as students progress through secondary education (e.g., the individual education plan and measurable postsecondary goals). Students and their transition services teams can revisit student goals over time with the benefit of pre-post data showing their growth.

Benefits to quality improvement

SVRAs and other provider agencies have reported multiple benefits to quality improvement efforts resulting from the TRT. These can be categorized under two areas: data-informed monitoring and focus on impact of services.

Data-informed service monitoring. SVRAs are responsible for Pre-ETS coordination and provision, and often collaborate and contract with multiple agencies and LEAs to meet their mandates to engage students in Pre-ETS. SVRAs, as well as administrators of other provider agencies, have expressed appreciation for the ability to use the TRT for service monitoring across all services and agencies. At this point, we lack evidence supporting any one approach or manualized service in Pre-ETS. Given the diversity of Pre-ETS, having a set of metrics that is agnostic to any specific approach to services allows agencies to evaluate service effectiveness without being required to change what or how they deliver services prior to doing so. Administrators have leaned into the quality improvement opportunities available through TRT data and have used findings as opportunities for growth and improvement rather than a tool for critique or punishment.

With the TRT, SVRA staff can see, at a glance, the degree of change from the start of Pre-ETS until services are completed. They can understand where providers are succeeding and, conversely, areas where technical assistance may be needed to improve outcomes. SVRA administrators who are monitoring multiple agencies are perfectly positioned to use cross-agency data to connect different provider agencies with each other for mentoring and community-of-practice opportunities where data indicates higher or lower than average student growth trends. A similar opportunity exists within any given service agency to collaborate around best practices. That said, the inclusion of a baseline (i.e., pre-assessment) measure allows SVRAs to account for differences in customer populations when interpreting data. For instance, agencies serving a high proportion of students with intensive needs are likely to experience lower post-assessment scores. However, the consideration of baseline (i.e., pre-assessment) scores will convey critical context about where students started from in a comparison across agencies. In this light, it is possible that agencies with lower overall change scores experience more significant improvements after taking baseline scores into account.

Shift in provision of services to impact. Administrators and providers have both appreciated the increased emphasis on student outcomes resulting from the TRT. Traditional monitoring of Pre-ETS has been primarily focused on funding thresholds, with employment outcomes, credential attainment, and measurable skill gains serving as more distal and disconnected measures of effectiveness (Fleming et al., 2024). Due to large quantities of unused service funds, agencies have recently come under increased scrutiny for meeting their 15% mandated threshold of investment in Pre-ETS (RSA, 2023). Without minimizing the importance of providing Pre-ETS to as many eligible or potentially eligible students with disabilities as possible, multiple concerns arise from encouraging spending without any systematic measure of quality. Such concerns have been expressed by SVRAs tasked with delivering Pre-ETS and some practices being implemented to most readily achieve spending thresholds appear to be detrimental to service quality. Increasing the systematic evaluation of services through pre-post assessment has the potential to enhance quality without restraining quantity or investment (Fleming et al., 2024; Mazurek Melnyk & Morrison-Beedy, 2018). This call for accountability in measuring effectiveness is also beneficial in the way that it aligns with others urging SVRAs to re-envision their approaches to quality assurance and evidence-based decision making to keep up with legislative and federal policy demands (Anderson et al., 2021b; Vilsack, 2021). Systematic measures of effectiveness in Pre-ETS through pre-post assessment will, over time, support empirical evidence of best practices in relation to outcomes including employment, credential attainment, and measurable skill gains. Combining change data with VR outcome data, will finally allow for understanding when and for whom Pre-ETS services lead to successful transition outcomes in ways not possible using RSA-911 data alone. Further, there is power in identifying best practices from data that is being systematically collected in the field (in contrast to researcher-initiated studies), because it leaves no questions about generalizability or transferability as many research studies often do (Vandergoot et al., 2009).

Discussion

Potentials for broader implementation in vocational rehabilitation

We have shared the TRT as an example of how measures of meaningful change have the potential to transform Pre-ETS. We believe similar benefits could be derived from implementing systematic measures of change across all essential vocational rehabilitation services— for both youth and adults. We conclude by providing some guidance on how the creation of such measures could be achieved based on our own experience developing the TRT.

Essentials to developing systematic measures of change

Most of the current employment measures were created without a specific agency or the accompanying policy and regulations that guide that agency in mind. Development of the TRT, in contrast, began with a consideration of agency input and the policy and regulations that guide Pre-ETS practices. This guidance was essential in defining each core service and to clarify (a) the essential aspects of each service, (b) how the core services were distinguishable, and (c) the desired immediate outcome from each service (Fleming et al., 2024). Using the information from this process, we developed assessments with ongoing input from SVRA staff and providers as well as disabled students, parents, CRPs and other key stakeholders. Specific attention was given to ensuring that the assessments met the needs of both the students and the providers who would be engaged in their administration. The resulting assessments were brought to life through the creation of an online software that allowed for a simple and seamless process of collecting, analyzing, and reporting the data in real-time. Access to real-time change data proved to be critical to facilitating practical, day-to-day use among service providers. Finally, implementation of the TRT relies on training and technical assistance with educators and community rehabilitation providers due to the significant culture shift necessary for optimal administration and utilization of data that supports continuous improvement.

A similar process of defining, clarifying, creating, training, and implementing would be required to apply systematic measures of change across other SVRA services.

Defining and clarifying

Defining and clarifying the general vocational rehabilitation services would present a greater challenge compared to Pre-ETS. The five core Pre-ETS were created by federal mandate and served as a useful structure for operationalization and construction of corresponding assessments. No similar structure exists for the many traditional VR services around what is core or essential to the function and outcomes of SVRAs (e.g., occupational or vocational training Provided by VR agency staff, basic academic remedial or literacy training provided by VR agency staff, and job readiness training provided by VR agency staff). The Rehabilitation Services Administration provides some definition and instruction in connection with each of the services that are reported for the RSA-911 dataset that would provide a helpful starting point. Variations in how states further define each service would also need to be accounted for.

In addition to guidance of key stakeholders (e.g., SVRA administrators, supervisors, and counselors), empirical data from the RSA-911 dataset around frequency of services and existing studies demonstrating correlations with central outcomes may serve as a resource for determining what services are essential. The defining of services is both part of the identification and clarification process. Some initial definition must be imparted to determine whether it is essential. Once identified as essential, greater definition sufficient to facilitate clarifications about the essential aspects of that service, and how one essential service is distinguishable from all others. Additionally, instrument developers would need to identify an agreed-upon target outcome in the form of a knowledge or skill gain for each of the services identified. To support continuous improvement, this outcome would need to be expected to follow immediately from effective services rather than occurring at much later date or time.

To illustrate, the federal policy guidance defining Job Search Assistance states,
Job search activities support and assist an individual in searching for an appropriate job. Job search assistance may include help in resume preparation, identifying appropriate job opportunities, developing interview skills, and making contacts with companies on behalf of the consumer (U.S. Department of Education, 2023, p. 28).
As previously noted, the first step would be for SVRA staff, counselors, contracted providers, and other key stakeholders to understand and agree on whether it is an essential service. Assuming that it is and that it can be distinguished from other related services, stakeholders would need to come to a shared definition and agreement on the essential elements of job search assistance and the proximal target outcomes that are expected to change from effective services.

Creating

With essential services identified, defined and clarified, it becomes possible to create assessments that accurately represent the targeted outcome of that service. Based on our experience, this is an iterative process of item generation, review, and revision that incorporates multiple sources of information including SVRA leadership and providers, other direct providers, service recipients, and the research literature. To serve as a fair and appropriate assessment, items that are generated must capture only content that is considered to be an indispensable aspect of the target outcome. The assessment items then should not be viewed as the only things to be taught but as necessary aspects of a training that could incorporate other content areas. In the case of the TRT, we engaged individuals from six states in assessing whether the items that represented necessary aspects of training were compatible across contexts.

There are multiple considerations in effective assessment development. In addition to reliability and validity, development must place emphasis on feasibility and usability (Vandergoot et al., 2009). Development of the TRT incorporated many adjustments to meet the practical needs of those who would be providing the assessments as well as those of the people taking them. Potential barriers such as time and resource limitations shaped assessment creation as well as the course of administration, analysis, and reporting of results. For the disabled student population, this included creating items that did not exceed a fourth-grade reading level, providing multiple formats for administration, and extensive instruction and training to administrators on the methods of effective data collection. It also required that assessments be brief enough for practical application in the day-to-day tasks and processes of Pre-ETS providers.

Training and implementing

Training and implementation make up the final elements for adoption of pre-post measurement of change. As noted previously, current reporting requirements of RSA-911 data do not anticipate frequent use or reference by providers to guide their services. The implementation of measures that facilitate data-driven service decisions requires a significant investment in training as well as ongoing technical assistance. Implementation training must be a requirement for engaging with measures of continuous improvement to ensure that quality of data collection and utilization is maintained. To ensure that TRT-users understand the assessments and best practices for implementation, they are unable to access the full features of the TRT software until they complete a minimal set of trainings to become certified for use. This process typically takes 90 to 120 min. Particularly in settings that experience high levels of turnover, a process must be in place for onboarding and training new providers as they are hired that does not place additional burden on agencies.

A major element of training and implementation goes beyond simply imparting procedural knowledge to generating an appreciation of the benefits of collecting high-quality data that can shape continuous improvement and guide service delivery. Providers who understand and appreciate the utility of such data will be more effective in encouraging honest and accurate reporting from those receiving services. In initial adoption, it can be helpful to provide prompts for providers and administrators to access and use the information and insights provided through the data to establish a culture of data literacy and application.

Call to action

Legislative and policy mandates (e.g., WIOA, Evidence Act; Anderson et al., 2021b; Fleming et al., 2024) are increasingly trending towards enforcement of public agencies reporting and using rigorous evaluation approaches that facilitate data-informed decision making and integration of evidence-based practice. Pre-post assessment and its benefits fit within these mandates, and open additional possibilities for meeting these increased standards in SVRAs. This shift towards more systematic measures of continuous improvement would require an investment of time and resources to create and maintain a new culture of evaluation and data utilization (Anderson et al., 2021a; Groomes & Adams, 2013). As stated by Vandergoot et al. (2009), “there is no data-driven system of performance management extant in the VR system” (p. 117). The result, they argue, is that monitoring, oversight, training, and professional development have “no context within which to frame their activities” (p. 117). In such a system, the data casts measures of performance and continuous improvement as an exercise in compliance (Groomes & Adams, 2013), and providers who have primary responsibility for collecting the data have the least to gain (Corash et al., 2023; Vandergoot et al., 2009).

Numerous benefits are likely to flow from a strengthened commitment to continuous improvement that would be experienced by the recipient of services, to service providers, and to administrators. Even more, we believe these efforts to document continuous improvement has the potential to garner trust with collaborating agencies and institutions as well as among the general public. We have used the TRT to demonstrate the feasibility for moving towards systemic continuous improvement measures in SVRAs. Not only is this shift possible, the need for it is pressing, and we argue that SVRAs will be better off if they lead this shift rather than waiting to react to it. In debating the Evidence Act in the House of Representatives (later passed on a vote of 356 to 17), Congressman Blake Farenthold stated, “The driving purpose of this legislation is very simple: we are requiring Federal agencies to prioritize evidence when they are measuring a program's success” (163 Cong Rec H 9281, p. 9289). He continued by saying
Let's just take poverty, for example. Instead of measuring success based on inputs, instead of measuring success based on how many programs we have created, how much money we are spending, how many people are on those programs, let's measure success based on results: Is it working? Are people getting out of poverty? Are the goals and objectives of these bills and these laws actually being achieved or not? By directing agencies to do this, no longer will “we don't know” be an acceptable answer when asked if a program is actually working…so much of what government does, government doesn't actually see whether or not it is actually succeeding at doing it. So, we have got to get off of this input effort-based system, this twentieth century relic, and on to clearly identifiable, evidence-based terms, conditions, data, results, outcomes. With this bill, we are asking the Federal bureaucracy to step up its game. We are asking ourselves: How can we improve the lives of our fellow citizens by better understanding the programs that we put in place? How can we make sure that the money that is being spent on behalf of the hardworking taxpayers, who send the money here in the first place, is being spent wisely, efficiently, effectively? We need results, not just effort. This is just good, commonsense policy, and it is going to mean a real sea change in how we solve problems and how government actually works (p. 9289).
In the context of this federal legislation and the context driving its passage, SVRAs have much to gain from proactively strengthening continuous improvement efforts. Doing so will help establish best practices leading to the cardinal indicators of success (i.e., employment status, wages and earnings, benefits, types of occupations, employment retention, and measurable skill gains and credentials; Rehabilitation Services Administration, 2023). Just as important in an age of increased accountability, the collection of pre-post change will empower SVRAs to report the many other benefits customers are assumed to receive from receipt of VR. In the case of the TRT, this includes the ability to show empirical change across each of the five core Pre-ETS.

The ability to show evidence for change resulting from services is not just an argument that more evidence is better than less. It also allows for SVRAs to mitigate some of the limitations of being judged based upon employment indicators alone. Despite its importance and centrality as an outcome, employment status is a flawed proxy for service effectiveness. This is because (a) the growing proportion of disabled youth receiving services highlights the need for measures of effectiveness for the interim between initiating services and employment, an interim that can span several years and (b) employment status, wages, and benefits are outcomes that are largely outside of the control of SVRAs and their customers. We will briefly discuss each one.

The growing proportion of youth receiving services through SVRAs serves as a limiting factor in accounting for success of services. This is because a significant proportion of youth receiving Pre-ETS services never go on to enroll in VR services and, even among those who do, there is often a multi-year lag between service provision and employment outcomes. These issues are somewhat reduced by the inclusion of measurable skills gains and attainment of credentials, however, much of the effectiveness of services provided to youth remain unmeasured and unknown in the absence of pre-post measures of change. In contrast, with systematic continuous improvement measures in place, SVRAs are able to convey a more accurate picture of the benefits they bring to service recipients and the broader society.

Despite closer proximity of employment for adult populations, employment at exit is shaped by many factors that are outside of the control of the VR counselor or customer. We have been reminded of this fact in recent years through a global pandemic and the more common ebbs and flows of the labor market. Elements that are not fully under SVRA control (e.g., caregiving needs, transportation, changes in family resources, or changes to health status) often shape employment outcomes for better or worse. These confounding factors limit our ability to connect employment with service effectiveness. For instance, in a strong labor market, poor SVRA services may actually drive a customer to withdraw from services and find employment on their own. The same customer experiencing a poor labor market may fail to obtain employment despite even the best of services. Drawing a parallel from a related discipline, this would be like marriage and family therapists publishing client divorce rates as the primary mechanism for the public to judge effectiveness of counseling. To clarify, these issues of controllability and specific caseloads do not warrant removal of employment from a system that was created to this end. Rather, it begs for employment to be combined with measures of change resulting from services that are more within the control of SVRAs and that allow for greater recognition of change that is facilitated by SVRAs in the efforts to support customers in securing competitive integrated employment.

Conclusion

SVRAs maintain extensive data collection and reporting requirements, consistent with legislative and policy mandates for public programs. However, these data have limited applicability in shaping evidence-based policy and practices specific to how SVRA services should be conducted. The TRT provides an example of pre-and post-assessment of Pre-ETS and how data is currently used for enhanced monitoring and quality improvement of Pre-ETS with direct benefits to students. Pre-post assessment approaches could provide similar benefits for other core SVRA services. Agency administrators can leverage these data internally and externally to more accurately understand staff and counselor performance, monitor contracted providers, and provide additional evidence of agency impact. For example, using pre-post data to supplement annual performance reviews of counselors who work with youth, or customers with most significant disabilities and barriers to employment who may show lower employment rates. Further, expanding reports of employment outcomes that are typically provided as evidence of program effectiveness to add pre-post change data as a way to further demonstrate the impact their agency has on the public. If taken on, this initiative would represent a significant system-level change, requiring clarification of the essential aspects of each core service and the desired target outcome in the form of knowledge or skill gains. Access to pre-post assessment data across core SVRA services has high potential to benefit agency leadership and administrators, staff and counselors, and customers.

WIOA core service	Assessment	Training description and sample topics
Job exploration counseling	TRT job exploration scale	Knowledge of available jobs; understanding of general requirements for jobs; knowledge of self, as it relates to job opportunities (interests, strengths); knowledge of what I want in a job (pay, types of tasks / interactions).
Work-based learning experiences	TRT work-based learning pre-post scale	Experience-based knowledge of general work skills necessary for successful employment. Self-evaluation of performance at work (pace of work, getting along with others, communicating with supervisors).
Work-based learning experiences	TRT work-based learning post-only scale	Students’ perspective of whether the work-based learning experience was a good fit for them, whether they finished the experience, and whether they were paid (and if so, how much).
Instruction in self-advocacy	TRT self-advocacy scale	Understanding myself and my disability, knowing my rights and responsibilities when it comes to requesting accommodations, assertiveness, problem solving, negotiation, effective communication, and self-efficacy.
Counseling on post-secondary education opportunities	TRT counseling on post-secondary enrollment scale	Knowledge of post-secondary opportunities and how to apply. Self-knowledge of interest for post-secondary training and how this relates to future work. Understanding of skills necessary for success in post-secondary education, e.g., study skills, time management, and educational supports / accommodation.
Workplace readiness training	TRT communication and social skills scale	Connecting, relating with, and respecting others; effective workplace communication; and understanding expectations.
Workplace readiness training	TRT employability scale	“Soft skills” such as responsibility, responsiveness, and seeking feedback.
Workplace readiness training	TRT financial literacy scale	Basics of budgeting, consideration of benefits and pay as they relate to employment.
Workplace readiness training	TRT independent living, orientation, and mobility scale	Wayfinding, safety, following a schedule, hygiene, getting your own food, and keeping your space clean
Workplace readiness training	TRT job seeking scale	How to find job opportunities, application, and interview skills.

Footnotes

Acknowledgements

The authors wish to thank the participating SVRAs, providers, and students / families who have contributed to the development and evaluation of the TRT instruments.

Ethics statement

Not applicable – this is a non-empirical discussion paper.

Informed consent

Not applicable – this is a non-empirical discussion paper.

Funding

This Field Initiated Project (FIP) on Assessing Transition Services at Utah State University is funded by the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR grant number #90IFDV0027). NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS). Products and content stemming from this project do not necessarily represent the policy of NIDILRR, ACL, or HHS, and you should not assume endorsement by the Federal Government.

Declaration of conflicting interests

Drs Fleming and Phillips have an outside financial interest in an entity that may benefit from the outcome of this research. This outside financial interest has been reviewed by their respective University's Individual Conflict of Interest Committees and is currently being managed by the Universities.

References

Alsaman

M. A.

Lee

C. L.

(2017). Employment outcomes of youth with disabilities in vocational rehabilitation: A multilevel analysis of RSA-911 data. Rehabilitation Counseling Bulletin, 60(2), 98–107. https://doi.org/10.1177/0034355216632363

Anderson

Johnson

Neel

Hoover

(2021a). Improving quality assurance at South Carolina Vocational Rehabilitation Department: Developing a new electronic tool. Rehabilitation Counselors and Educators Journal, 10(2). https://doi.org/10.52017/001c.33103

Anderson

C. A.

Sabella

Groomes

Tansey

T. N.

Homa

Zhu

Connelly

Walker

(2021b). The importance of building capacity for program evaluation and quality assurance within vocational rehabilitation. Rehabilitation Counselors and Educators Journal, 10(2). https://doi.org/10.52017/001c.29113

Boeltzig-Brown

Fleming

A. R.

Heyman

Gauthier

Cully

Foley

S. M.

(2017). A systematic review of state vocational rehabilitation agency-based literature. Rehabilitation Research, Policy and Education, 31(4), 352–371. https://doi.org/10.1891/2168-6653.31.4.352

Corash

Jacobs

Seltzer

May

Warner

Park

(2023). Realigning vocational rehabilitation with its core mission. San Diego State University: Interwork Institute. Retrieved from https://interwork.sdsu.edu/main/nrli/seminar/participant-policy-papers

Crudden

Sui

Lund

(2020). Employed vocational rehabilitation applicants with visual disabilities: Factors associated with timely service delivery. Journal of Visual Impairment & Blindness, 114(1), 31–42. https://doi.org/10.1177/0145482X19900617

Dean

Pepper

J. V.

Schmidt

R. M.

Stern

(2014). State vocational rehabilitation programs and federal disability insurance: An analysis of Virginia’s vocational rehabilitation program. IZA Journal of Labor Policy, 3, 1–19. http://www.izajolp.com/content/3/1/7

Donahue

M. L.

Paquet

Casucci

Nikolaev

(2024). Vocational rehabilitation applicants, the services they receive, and their employment outcomes. Journal of Vocational Rehabilitation, 60(1), 63–85. https://doi.org/10.3233/JVR-230057

Ethridge

Dowden

A. R.

Brooks

Kwan

Harley

(2020). Employment and earnings among ex-offenders with disabilities: A multivariate analysis of RSA-911 data. Journal of Vocational Rehabilitation, 52(3), 279–289. https://doi.org/10.3233/JVR-201077

10.

Fernández-Pascual

Pinto

Garcia Marco

F. J.

(2024). Emergence and evolution of data literacy: Insights from a bibliometric study. Journal of Librarianship and Information Science, Online First. https://doi.org/10.1177/09610006241265102

11.

Fleming

A. R.

Phillips

B. N.

Riesen

Langone

(2024). Enhancing transition outcomes: A toolkit to facilitate data-driven pre-employment transition services. Journal of Vocational Rehabilitation, 60, 197–209. https://doi.org/10.3233/JVR-240006

12.

Groomes

D. A. G.

Adams

L. A.

(2013). Value chain analysis: An implication for improving systems within the vocational rehabilitation program. Journal of Rehabilitation Administration, 37(2), 53–64.

13.

Groomes

D. A. G.

Jones

T. A.

Stoddard

Stevens

(2017). Championing innovation in the field of vocational rehabilitation. Journal of Rehabilitation Administration, 39(2), 79–90.

14.

Groomes

D. A. G.

Shoemaker

Vandergoot

Collins

S. W.

(2015). Excellence is within your reach: The importance of planning for performance in the state-federal vocational rehabilitation program. Journal of Rehabilitation, 81(2), 39–46.

15.

Hill

Mann

D. R.

Gellar

(2022). Predicting program outcomes for vocational rehabilitation customers: A machine learning approach. Journal of Vocational Rehabilitation, 56(2), 107–121. https://doi.org/10.3233/JVR-221176

16.

Ipsen

Swicegood

Colling

Rigles

Asp

(2014). Rural and urban differences in VR caseloads and delivery practices. The University of Montana Rural Institute.

17.

Leahy

M. J.

Chan

Lui

Rosenthal

Tansey

Wehman

Kundu

Dutta

Anderson

C. A.

Del Valle

Sherman

Menz

F. E.

(2014). An analysis of evidence-based best practices in the public vocational rehabilitation program: Gaps, future directions, and recommended steps to move forward. Journal of Vocational Rehabilitation, 41(2), 147–163. https://doi.org/10.3233/jvr-140707

18.

Lutz

Schwartz

Delgadillo

(2022). Measurement-based and data-informed psychological therapy. Annual Review of Clinical Psychology, 18(1), 71–98. https://doi.org/10.1146/annurev-clinpsy-071720-014821

19.

Mazurek Melnyk

B. M.

Morrison-Beedy

(eds) (2018). Intervention research and evidence-based quality improvement: Designing, conducting, analyzing, and funding. Springer Publishing Company.

20.

Oertle

K. M.

Bobroff

Sax

C. L.

(2021). Organizational policies and personnel factors that are incentives for and challenges to collaboration as identified by statewide transition supervisors. Journal of Disability Policy Studies, 32(1), 36–50. https://doi.org/10.1177/1044207320943544

21.

O’Neill

Mamun

A. A.

Potamites

Chan

da Silva Cordoso

(2015). Return to work of disability insurance beneficiaries who do and do not access state vocational rehabilitation agency services. Journal of Disability Policy Studies, 26(2), 111–123. https://doi.org/10.1177/1044207315583900

22.

Parker-Katz

M. B.

Cushing

L. S.

Athamanah

(2018). Fostering collaboration as transition specialists through community conversations. Journal of Disability Policy Studies, 28(4), 244–254. https://doi.org/10.1177/1044207317739407

23.

Phillips

B. N.

Fleming

A. R.

Langone

Baldwin

Thompson

Riesen

(2022). Transition readiness toolkit: A collaborative effort to improve the efficacy of Pre-Employment Transition services. Journal of Rehabilitation Counseling Research and Practice, 43(1), 73–86.

24.

Plotner

A. J.

Walters

C. B.

Gonzalez

(2024). Exploring special education and center for independent living professional beliefs on collaboration and the value-based principles that drive secondary transition service delivery. Journal of Disability Policy Studies, 34(4), 266–277. https://doi.org/10.1177/10442073221131284

25.

Rehabilitation Services Administration (2023). State vocational rehabilitation services and state supported Employment Services Programs Monitoring and Technical Assistance Guide. Author. Retrieved from https://rsa.ed.gov/whats-new/rsa-publishes-mtag-and-announces-monitoring-reviews

26.

Sabella

S. A.

Groomes

D. A.

Markve

M. E.

Shoemaker

M. T.

(2018). Competencies for effective program evaluation in vocational rehabilitation. Journal of Rehabilitation, 84(3), 66–74.

27.

Sprong

M. E.

Dallas

Paul

Xia

(2019). Rehabilitation technology services and employment outcomes among consumers using division of rehabilitation services. Disability and Rehabilitation: Assistive Technology, 14(5), 445–452. https://doi.org/10.1080/17483107.2018.1463400

28.

U.S. Department of Education (2023). Policy Directive 14-01 reporting of data elements. Rehabilitation Services Administration. Retrieved from: https://rsa.ed.gov/performance/rsa-911-policy-directive

29.

Vandergoot

Borden

Stensrud

Bruinekool

(2009). The evolution of quality in the state/federal partnership: Training perspectives on performance management. Journal of Rehabilitation Administration, 33(2), 113–120.

30.

van Donge

Bharosa

Janssen

M. F. W. H. A.

(2022). Data-driven government: Cross-case comparison of data stewardship in data ecosystems. Government Information Quarterly, 39(2). https://doi.org/10.1016/j.giq.2021.101642

31.

Vilsack

(2021, November 10). What does the new evidence-based policymaking scorecard mean for skills advocates? National Skills Coalition. Retrieved from https://nationalskillscoalition.org/blog/future-of-work/what-does-the-new-evidence-based-policymaking-scorecard-mean-for-skills-advocates/

32.

Wehman

Tansey

Taylor

J. P.

Parent-Johnson

Whittenburg

Averill

(2024). Building a foundation for competitive integrated employment: What does the future hold for Pre-employment transition services. Journal of Vocational Rehabilitation, 60(2), 263–272. https://doi.org/10.3233/JVR-240011

33.

Workforce Innovation and Opportunity Act, Pub. L. No. 113-128, Stat (2014). (codified as amended at 29 U.S.C. §§3101-3361).