Abstract
Background
State Vocational Rehabilitation Agencies (SVRAs) are the leading provider of employment services for disabled adults in the United States, with achievement of competitive integrated employment as their main outcome.
Objective
Current data collection policies and practices require counselors to record extensive information for case documentation purposes, however, these data offer limited utility for service planning or for determinations of whether services are having the intended impact. Available data is not structured to support continuous improvement for individual providers or to establish evidence of best practices in day-to-day service provision.
Method
In this article, we present the many potential benefits of including pre-post assessment of services in measuring the impact of SVRA services.
Results
The Transition Readiness Toolkit offers a concrete example of the advantages associated with pre-post assessment and continuous use of data to guide services and quality improvement in SVRAs.
Conclusion
Recommendations and implications for implementing pre-post assessment for other SVRA services are provided.
The state vocational rehabilitation (VR) services system has one of the most comprehensive administrative datasets of any public program in the United States in the Rehabilitation Services Administration (RSA)-911 data. RSA-911 data is organized by cohort, representing all individuals who closed their VR cases in a given year. It includes more than 300 data elements capturing demographics, case status, services provided, length of services, and conditions of case closure for each customer (U.S. Department of Education, 2023). The collection and maintenance of this extensive dataset requires a significant investment of time and resources from state VR agencies (SVRAs) that demonstrates a strong commitment from both state and federal agencies providing oversight of VR services. This commitment is not surprising given the conspicuous and complementary trends of digitization, data science, data-informed practice, transparency, and accountability that are reflected across government and private institutions. Elements of these trends can be seen in the Government Performance and Results Modernization Act of 2010, Foundation for Evidence-Based Policymaking Act of 2019, and in the Workforce Innovation and Opportunities Act of 2014 (WIOA; See Fernández-Pascual et al., 2024; Fleming et al., 2024; Groomes & Adams, 2013; van Donge et al., 2022 for more description of these influences). However, the vast data elements and resources designated for gathering them have been extremely limited in shaping evidence-based policy and practices specific to how VR services should be conducted.
In this article we echo others in calling for a shift in the evaluation of state VR services from one almost exclusively focused on oversight to one that facilitates continuous improvement through the systematic measure of change resulting from services (Anderson et al., 2021a; Groomes & Adams, 2013; Groomes et al., 2015; Groomes et al., 2017; Sabella et al., 2018; Vandergoot et al., 2009). Such a transformation would benefit all stakeholders by enabling documentation of how customers benefit from VR services. This move would also increase understanding of who benefits from VR services and the degree of equity experienced between VR populations. We start by outlining the limitations of available data elements for identifying and using evidence-based practices and the shortcomings of the current data to inform individual customer services. We then describe an initiative (The Transition Readiness Toolkit), that has been adopted by multiple SVRAs to systematically measure the effectiveness of pre-employment transition services (Pre-ETS). We conclude the article with recommendations for SVRAs to strengthen and modify current evaluation practices across all essential services. We proceed under the strongly held assumption that customers experience multiple benefits from VR services. That said, without measures of change to demonstrate the impact of services, we are limited in our ability to identify best practices and improve the quality and effectiveness of services.
What we can and cannot know from RSA-911 data
Hundreds of published and unpublished studies have been conducted using the RSA-911 dataset (Boeltzig-Brown et al., 2017). These studies have contributed significantly to our knowledge of SVRAs, namely their efficacy, efficiency, equity, and compliance with federal legislation and regulations. For instance, studies have led to insights about whether customers from minoritized racial-ethnic groups, disability populations, and those with prior justice involvement offender statuses receive a similar amount of services to other customers (e.g., Alsaman & Lee, 2017; Ethridge et al., 2020; Ipsen et al., 2014), whether receipt of a specific service correlates with employment outcomes (e.g., Hill et al., 2022; Sprong et al., 2019), and whether SVRAs are compliant with timelines for eligibility determinations or development of an individualized plan for employment (e.g., Crudden et al., 2020; Donahue et al., 2024). A select few more sophisticated studies have paired RSA-911 data with other datasets to demonstrate the effectiveness of SVRA services using a matched comparison group who did not receive VR services (Dean et al., 2014; Leahy et al., 2014; O’Neill et al., 2015). Every year, new studies provide additional insights and updates about SVRAs and the customers receiving their services. That said, there are also many important questions that cannot be answered using RSA-911 data.
The RSA-911 data elements relating to SVRA services focus on (a) whether a service was received, (b) what agency provided it, (c) dates of initiation and completion, and (d) in the case of contracting services, the cost of services provided (U.S. Department of Education, 2023). Outcomes that are measured in connection with these services include measurable skills gains, credential attainment, competitive integrated employment, and earnings. As described more fully by Fleming et al. (2024), measurable skills gains “loosely measure the effectiveness of Pre-ETS and transition services without being tied to a specific service” (p. 199). In summary, RSA-911 data provides a snapshot of what services were received under an Individual Plan for Employment and whether the principal outcome of competitive integrated employment is achieved at their conclusion, with some measure of educational milestones captured in between.
For all of its strengths, RSA-911 data lacks essential information necessary for the identification of best practices and continuous improvement of VR services. Current service definitions do not guide characteristics or content of the service that would be expected, nor the desired gain associated with the specific service. Available data does not inform the dosage of a service that appears most effective in securing the desired gain, nor the level of skill that would best predict successful employment. Without this type of information, it is not possible to accurately evaluate how the results of a specific service relate to an outcome. These issues stunt the development of effective practice while also circumscribing what we can know about the benefits of services provided by SVRAs. Notably absent from the RSA-911 data is any information that directly captures improvement or change resulting from the specific services that were received. Identifying whether a service caused change for an individual customer requires an understanding of the level of skill or ability prior to a specific service and at completion of the service. Collecting more than 300 data elements, none capturing need for services or change resulting from them, means that direct providers experience the primary burden of data collection while experiencing little to no guidance or benefits from that data (Corash et al., 2023).
Method
Proposing systematic measures of change in VR
Pre-post assessment as an evaluation approach allows us to capture customers’ knowledge and skills before and after an intervention. It leverages principles of pre-experimental design, where a person or group is assessed, an intervention is provided, and then the person or group is assessed again to determine whether change occurred (Lutz et al., 2022). The assessment tool(s) align with the most salient indicators of what should change as a result of the intervention, referred to as the intervention target. When administered in close proximity to the intervention, counselors can reduce (although not eliminate) the likelihood that something besides the intervention caused any observed change to the intervention target. A reliable and valid pre-assessment can facilitate greater customization of services to account for the unique strengths and needs of each customer from the initiation of services. Used systematically and in combination with post-assessment, providers and other key stakeholders are able to evaluate the degree of change associated with an intervention and, over time, understand the most ideal dosage and duration of the intervention to achieve desired change. Systematic pre-post assessment can also offer insights about whether individual differences moderate treatment effects.
Pre-post assessment is more commonly leveraged in many related fields of education and allied health. For example, evaluating range of motion, strength, and balance before and after a physical therapy intervention or repeated blood draws to check for reductions in cholesterol or glycated hemoglobin (A1C) levels associated with nutrition counseling. Assessment data is then used to determine if an intervention has had the desired effect (i.e., treatment is complete), and if not, what desired outcome has not been met. Individual-level evaluations allow for modification of direct customer services while program-level evaluation is based on patterns across individuals that allows for understanding of whether the intervention is having the intended effect across a target population. SVRAs have not traditionally included systematic measures of effectiveness through pre-post assessment. However, we believe that meaningful pre-post assessment is possible within the VR system based on the successful implementation of the Transition Readiness Toolkit (TRT) across eight SVRAs representing different regions of the country.
Results
The Transition Readiness Toolkit
The Transition Readiness Toolkit (TRT; Phillips et al., 2022) captures the impact of Pre-ETS delineated by The Workforce Investment and Opportunity Act (2014). The TRT includes 10 pre-post assessment tools closely mapped to the five core Pre-ETS. The online software provides real-time data reporting and visualizations that facilitate data-informed service decisions and programmatic improvement (Fleming et al., 2024). To date, the TRT has captured approximately 21,000 student assessments. See Table 1 for additional information on the content of the 10 assessments. Information about the development of the TRT can be found in Fleming et al. (2024) and Phillips et al. (2022).
TRT instrument and alignment with pre-ETS core services.
Fleming, A. R., Phillips, B. N., Riesen, T., & Langone, A. (2024). Enhancing transition outcomes: A toolkit to facilitate data-driven Pre-Employment Transition Services. Journal of Vocational Rehabilitation, 60, 197–209. Reprinted with permission.
A few unique features of the TRT are key to its implementation within SVRAs and other transition agencies. First, unlike many other tools being used in transition, the assessments and features of the TRT were created specifically for youth with disabilities to be applied in transition settings. Assessments are brief, taking an average of only 2 to 3 minutes to complete. Multiple formats for administration ensure that all students receiving a Pre-ETS are represented. The data reporting and visualizations are created with busy providers in mind and without any prerequisite data literacy training assumed. The assessments also align with legislation and policy that guide Pre-ETS. Provider agencies using the TRT can access aggregated data that captures overall service effectiveness within their specific agencies as well as data showing the individual responses for each student receiving Pre-ETS. Each SVRA has access to aggregated and disaggregated data for all provider agencies within their jurisdiction, with the ability to view the data across all agencies or within a specific agency. Regular use of these data promotes a culture that is built around continuous improvement and anchored on quality of services rather than exclusively focusing on quantity of services or funding allocated. With the implementation of the TRT, providers and administrators alike are, for the first time, consulting empirical data to determine how to customize and improve services based on individual student strengths and areas for growth. This data also provides guidance that helps providers and administrators determine when services are completed and whether they had the intended impact. See Figure 1 for an example of the data views provided through the TRT.
TRT instrument data views for an individual student.
TRT Benefits to pre-ETS provision and quality improvement
For most TRT users, this is the first time that they have had clear metrics to guide customization of Pre-ETS services and to assess their impact. Several direct service providers reported during the development phases of the TRT that even the process of clarifying and operationalizing the parameters and intent of the core services was helpful. Many of these providers indicated that, prior to the TRT, they had not thought of the different Pre-ETS as discrete service activities. Now, through the TRT, direct providers, community rehabilitation providers (CRPs), local education agencies (LEAs), and SVRAs using the TRT have a shared understanding of what a core service is and what is expected to change as a result. We describe below a few of the uses and benefits derived from systematic pre-post assessment. These include benefits to the service provider, the agency, the customer, and other key stakeholders. In combination, these benefits in measurement have enormous implications for service monitoring and service quality.
Benefits to service provision
Perhaps the most exciting result of TRT implementation has been the feedback from direct service providers. This group includes community rehabilitation providers (CRPs), local education agency (LEA) staff, and Pre-ETS specialists employed by SVRAs. Providers have indicated benefiting from the ability to (a) identify student strengths and needs at the initiation of services, (b) assess progress and make adjustments to services, and (c) use the data to facilitate collaboration with other key stakeholders.
Benefits to quality improvement
SVRAs and other provider agencies have reported multiple benefits to quality improvement efforts resulting from the TRT. These can be categorized under two areas: data-informed monitoring and focus on impact of services.
With the TRT, SVRA staff can see, at a glance, the degree of change from the start of Pre-ETS until services are completed. They can understand where providers are succeeding and, conversely, areas where technical assistance may be needed to improve outcomes. SVRA administrators who are monitoring multiple agencies are perfectly positioned to use cross-agency data to connect different provider agencies with each other for mentoring and community-of-practice opportunities where data indicates higher or lower than average student growth trends. A similar opportunity exists within any given service agency to collaborate around best practices. That said, the inclusion of a baseline (i.e., pre-assessment) measure allows SVRAs to account for differences in customer populations when interpreting data. For instance, agencies serving a high proportion of students with intensive needs are likely to experience lower post-assessment scores. However, the consideration of baseline (i.e., pre-assessment) scores will convey critical context about where students started from in a comparison across agencies. In this light, it is possible that agencies with lower overall change scores experience more significant improvements after taking baseline scores into account.
Discussion
Potentials for broader implementation in vocational rehabilitation
We have shared the TRT as an example of how measures of meaningful change have the potential to transform Pre-ETS. We believe similar benefits could be derived from implementing systematic measures of change across all essential vocational rehabilitation services— for both youth and adults. We conclude by providing some guidance on how the creation of such measures could be achieved based on our own experience developing the TRT.
Essentials to developing systematic measures of change
Most of the current employment measures were created without a specific agency or the accompanying policy and regulations that guide that agency in mind. Development of the TRT, in contrast, began with a consideration of agency input and the policy and regulations that guide Pre-ETS practices. This guidance was essential in defining each core service and to clarify (a) the essential aspects of each service, (b) how the core services were distinguishable, and (c) the desired immediate outcome from each service (Fleming et al., 2024). Using the information from this process, we developed assessments with ongoing input from SVRA staff and providers as well as disabled students, parents, CRPs and other key stakeholders. Specific attention was given to ensuring that the assessments met the needs of both the students and the providers who would be engaged in their administration. The resulting assessments were brought to life through the creation of an online software that allowed for a simple and seamless process of collecting, analyzing, and reporting the data in real-time. Access to real-time change data proved to be critical to facilitating practical, day-to-day use among service providers. Finally, implementation of the TRT relies on training and technical assistance with educators and community rehabilitation providers due to the significant culture shift necessary for optimal administration and utilization of data that supports continuous improvement.
A similar process of defining, clarifying, creating, training, and implementing would be required to apply systematic measures of change across other SVRA services.
Defining and clarifying
Defining and clarifying the general vocational rehabilitation services would present a greater challenge compared to Pre-ETS. The five core Pre-ETS were created by federal mandate and served as a useful structure for operationalization and construction of corresponding assessments. No similar structure exists for the many traditional VR services around what is core or essential to the function and outcomes of SVRAs (e.g., occupational or vocational training Provided by VR agency staff, basic academic remedial or literacy training provided by VR agency staff, and job readiness training provided by VR agency staff). The Rehabilitation Services Administration provides some definition and instruction in connection with each of the services that are reported for the RSA-911 dataset that would provide a helpful starting point. Variations in how states further define each service would also need to be accounted for.
In addition to guidance of key stakeholders (e.g., SVRA administrators, supervisors, and counselors), empirical data from the RSA-911 dataset around frequency of services and existing studies demonstrating correlations with central outcomes may serve as a resource for determining what services are essential. The defining of services is both part of the identification and clarification process. Some initial definition must be imparted to determine whether it is essential. Once identified as essential, greater definition sufficient to facilitate clarifications about the essential aspects of that service, and how one essential service is distinguishable from all others. Additionally, instrument developers would need to identify an agreed-upon target outcome in the form of a knowledge or skill gain for each of the services identified. To support continuous improvement, this outcome would need to be expected to follow immediately from effective services rather than occurring at much later date or time.
To illustrate, the federal policy guidance defining Job Search Assistance states, Job search activities support and assist an individual in searching for an appropriate job. Job search assistance may include help in resume preparation, identifying appropriate job opportunities, developing interview skills, and making contacts with companies on behalf of the consumer (U.S. Department of Education, 2023, p. 28).
Creating
With essential services identified, defined and clarified, it becomes possible to create assessments that accurately represent the targeted outcome of that service. Based on our experience, this is an iterative process of item generation, review, and revision that incorporates multiple sources of information including SVRA leadership and providers, other direct providers, service recipients, and the research literature. To serve as a fair and appropriate assessment, items that are generated must capture only content that is considered to be an indispensable aspect of the target outcome. The assessment items then should not be viewed as the only things to be taught but as necessary aspects of a training that could incorporate other content areas. In the case of the TRT, we engaged individuals from six states in assessing whether the items that represented necessary aspects of training were compatible across contexts.
There are multiple considerations in effective assessment development. In addition to reliability and validity, development must place emphasis on feasibility and usability (Vandergoot et al., 2009). Development of the TRT incorporated many adjustments to meet the practical needs of those who would be providing the assessments as well as those of the people taking them. Potential barriers such as time and resource limitations shaped assessment creation as well as the course of administration, analysis, and reporting of results. For the disabled student population, this included creating items that did not exceed a fourth-grade reading level, providing multiple formats for administration, and extensive instruction and training to administrators on the methods of effective data collection. It also required that assessments be brief enough for practical application in the day-to-day tasks and processes of Pre-ETS providers.
Training and implementing
Training and implementation make up the final elements for adoption of pre-post measurement of change. As noted previously, current reporting requirements of RSA-911 data do not anticipate frequent use or reference by providers to guide their services. The implementation of measures that facilitate data-driven service decisions requires a significant investment in training as well as ongoing technical assistance. Implementation training must be a requirement for engaging with measures of continuous improvement to ensure that quality of data collection and utilization is maintained. To ensure that TRT-users understand the assessments and best practices for implementation, they are unable to access the full features of the TRT software until they complete a minimal set of trainings to become certified for use. This process typically takes 90 to 120 min. Particularly in settings that experience high levels of turnover, a process must be in place for onboarding and training new providers as they are hired that does not place additional burden on agencies.
A major element of training and implementation goes beyond simply imparting procedural knowledge to generating an appreciation of the benefits of collecting high-quality data that can shape continuous improvement and guide service delivery. Providers who understand and appreciate the utility of such data will be more effective in encouraging honest and accurate reporting from those receiving services. In initial adoption, it can be helpful to provide prompts for providers and administrators to access and use the information and insights provided through the data to establish a culture of data literacy and application.
Call to action
Legislative and policy mandates (e.g., WIOA, Evidence Act; Anderson et al., 2021b; Fleming et al., 2024) are increasingly trending towards enforcement of public agencies reporting and using rigorous evaluation approaches that facilitate data-informed decision making and integration of evidence-based practice. Pre-post assessment and its benefits fit within these mandates, and open additional possibilities for meeting these increased standards in SVRAs. This shift towards more systematic measures of continuous improvement would require an investment of time and resources to create and maintain a new culture of evaluation and data utilization (Anderson et al., 2021a; Groomes & Adams, 2013). As stated by Vandergoot et al. (2009), “there is no data-driven system of performance management extant in the VR system” (p. 117). The result, they argue, is that monitoring, oversight, training, and professional development have “no context within which to frame their activities” (p. 117). In such a system, the data casts measures of performance and continuous improvement as an exercise in compliance (Groomes & Adams, 2013), and providers who have primary responsibility for collecting the data have the least to gain (Corash et al., 2023; Vandergoot et al., 2009).
Numerous benefits are likely to flow from a strengthened commitment to continuous improvement that would be experienced by the recipient of services, to service providers, and to administrators. Even more, we believe these efforts to document continuous improvement has the potential to garner trust with collaborating agencies and institutions as well as among the general public. We have used the TRT to demonstrate the feasibility for moving towards systemic continuous improvement measures in SVRAs. Not only is this shift possible, the need for it is pressing, and we argue that SVRAs will be better off if they lead this shift rather than waiting to react to it. In debating the Evidence Act in the House of Representatives (later passed on a vote of 356 to 17), Congressman Blake Farenthold stated, “The driving purpose of this legislation is very simple: we are requiring Federal agencies to prioritize evidence when they are measuring a program's success” (163 Cong Rec H 9281, p. 9289). He continued by saying Let's just take poverty, for example. Instead of measuring success based on inputs, instead of measuring success based on how many programs we have created, how much money we are spending, how many people are on those programs, let's measure success based on results: Is it working? Are people getting out of poverty? Are the goals and objectives of these bills and these laws actually being achieved or not? By directing agencies to do this, no longer will “we don't know” be an acceptable answer when asked if a program is actually working…so much of what government does, government doesn't actually see whether or not it is actually succeeding at doing it. So, we have got to get off of this input effort-based system, this twentieth century relic, and on to clearly identifiable, evidence-based terms, conditions, data, results, outcomes. With this bill, we are asking the Federal bureaucracy to step up its game. We are asking ourselves: How can we improve the lives of our fellow citizens by better understanding the programs that we put in place? How can we make sure that the money that is being spent on behalf of the hardworking taxpayers, who send the money here in the first place, is being spent wisely, efficiently, effectively? We need results, not just effort. This is just good, commonsense policy, and it is going to mean a real sea change in how we solve problems and how government actually works (p. 9289).
The ability to show evidence for change resulting from services is not just an argument that more evidence is better than less. It also allows for SVRAs to mitigate some of the limitations of being judged based upon employment indicators alone. Despite its importance and centrality as an outcome, employment status is a flawed proxy for service effectiveness. This is because (a) the growing proportion of disabled youth receiving services highlights the need for measures of effectiveness for the interim between initiating services and employment, an interim that can span several years and (b) employment status, wages, and benefits are outcomes that are largely outside of the control of SVRAs and their customers. We will briefly discuss each one.
The growing proportion of youth receiving services through SVRAs serves as a limiting factor in accounting for success of services. This is because a significant proportion of youth receiving Pre-ETS services never go on to enroll in VR services and, even among those who do, there is often a multi-year lag between service provision and employment outcomes. These issues are somewhat reduced by the inclusion of measurable skills gains and attainment of credentials, however, much of the effectiveness of services provided to youth remain unmeasured and unknown in the absence of pre-post measures of change. In contrast, with systematic continuous improvement measures in place, SVRAs are able to convey a more accurate picture of the benefits they bring to service recipients and the broader society.
Despite closer proximity of employment for adult populations, employment at exit is shaped by many factors that are outside of the control of the VR counselor or customer. We have been reminded of this fact in recent years through a global pandemic and the more common ebbs and flows of the labor market. Elements that are not fully under SVRA control (e.g., caregiving needs, transportation, changes in family resources, or changes to health status) often shape employment outcomes for better or worse. These confounding factors limit our ability to connect employment with service effectiveness. For instance, in a strong labor market, poor SVRA services may actually drive a customer to withdraw from services and find employment on their own. The same customer experiencing a poor labor market may fail to obtain employment despite even the best of services. Drawing a parallel from a related discipline, this would be like marriage and family therapists publishing client divorce rates as the primary mechanism for the public to judge effectiveness of counseling. To clarify, these issues of controllability and specific caseloads do not warrant removal of employment from a system that was created to this end. Rather, it begs for employment to be combined with measures of change resulting from services that are more within the control of SVRAs and that allow for greater recognition of change that is facilitated by SVRAs in the efforts to support customers in securing competitive integrated employment.
Conclusion
SVRAs maintain extensive data collection and reporting requirements, consistent with legislative and policy mandates for public programs. However, these data have limited applicability in shaping evidence-based policy and practices specific to how SVRA services should be conducted. The TRT provides an example of pre-and post-assessment of Pre-ETS and how data is currently used for enhanced monitoring and quality improvement of Pre-ETS with direct benefits to students. Pre-post assessment approaches could provide similar benefits for other core SVRA services. Agency administrators can leverage these data internally and externally to more accurately understand staff and counselor performance, monitor contracted providers, and provide additional evidence of agency impact. For example, using pre-post data to supplement annual performance reviews of counselors who work with youth, or customers with most significant disabilities and barriers to employment who may show lower employment rates. Further, expanding reports of employment outcomes that are typically provided as evidence of program effectiveness to add pre-post change data as a way to further demonstrate the impact their agency has on the public. If taken on, this initiative would represent a significant system-level change, requiring clarification of the essential aspects of each core service and the desired target outcome in the form of knowledge or skill gains. Access to pre-post assessment data across core SVRA services has high potential to benefit agency leadership and administrators, staff and counselors, and customers.
Footnotes
Acknowledgements
The authors wish to thank the participating SVRAs, providers, and students / families who have contributed to the development and evaluation of the TRT instruments.
Ethics statement
Not applicable – this is a non-empirical discussion paper.
Informed consent
Not applicable – this is a non-empirical discussion paper.
Funding
This Field Initiated Project (FIP) on Assessing Transition Services at Utah State University is funded by the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR grant number #90IFDV0027). NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS). Products and content stemming from this project do not necessarily represent the policy of NIDILRR, ACL, or HHS, and you should not assume endorsement by the Federal Government.
Declaration of conflicting interests
Drs Fleming and Phillips have an outside financial interest in an entity that may benefit from the outcome of this research. This outside financial interest has been reviewed by their respective University's Individual Conflict of Interest Committees and is currently being managed by the Universities.