How do early process milestones vary before a state vocational rehabilitation agency expands waitlist categories under order of selection? Evidence from Florida

Abstract

BACKGROUND:

State vocational rehabilitation (VR) agencies that lack the resources to serve all eligible applicants are required to place applicants with less significant disabilities on waitlists for services. This system for prioritizing applicants is called order of selection (OOS). OOS status changes might motivate VR counselors and applicants to adjust their behavior in important ways.

OBJECTIVE:

Examine how early milestones in the VR process—including eligibility determinations, determinations of disability significance, and signings of Individualized Plans for Employment (IPEs)—vary around an OOS status change.

METHODS:

We used case-level data from the 2006–2014 RSA-911 files to investigate a 2008 OOS status change at Florida’s general VR agency.

RESULTS:

Our results are consistent with systematic changes in how VR counselors make eligibility and disability-significance determinations, although we cannot rule out unmeasured changes over time in the types of people who apply for VR services. We also found that the likelihood of receiving an IPE before an OOS status change varies with key characteristics.

CONCLUSIONS:

If leaders want to adjust how counselors prioritize IPE completions as an OOS status change approaches, these leaders might consider changing the performance incentives facing their counselors.

Keywords

Vocational rehabilitation program eligibility individualized plan for employment disability Insurance supplemental security income

1 Introduction

State vocational rehabilitation (VR) agencies, which help people with disabilities obtain or maintain employment, sometimes lack the resources to serve all eligible applicants in the state. When this happens, the agencies are required by law to prioritize services to eligible applicants with the most significant disabilities. Eligible applicants with less significant disabilities at these agencies are placed onto waitlists for services. This system of prioritizing applicants is called order of selection (OOS). An agency can change its OOS status by expanding or reducing the disability-significance categories that place new applicants on a waitlist for services.

A pending change in OOS status might cause VR counselors and applicants to adjust their behavior in ways that are important for VR policy. For example, counselors might assess more applicants as having the most significant disabilities, or they might prioritize completing service plans for customers who have certain characteristics. And potential VR applicants might stop applying for services because they do not want to be placed on a waitlist.

These behavioral changes will likely alter an agency’s caseload size and composition, which in turn will affect how agencies use their resources. Furthermore, these behavioral changes might have special consequences for Supplemental Security Income (SSI) and Social Security Disability Insurance (SSDI) beneficiaries, who represent about a third of VR applicants and often have substantial barriers to employment (Mann et al. 2017).

In this study, we used case-level administrative data from Florida’s general VR agency to explore the changes that occurred before an OOS status change that resulted in some new eligible applicants being placed on a waitlist. We sought to answer three research questions:

What is the descriptive evidence that early milestones in the VR process—including eligibility determinations, determinations of disability significance, and signings of Individualized Plans for Employment (IPEs)—changed for people who applied for services in the months before the expansion of OOS waitlist categories?

In the months before this OOS status change, how did determinations of eligibility and disability significance vary (holding applicant characteristics constant)?

In the months before this OOS status change, how did the probability of receiving a signed IPE differ with time and applicant characteristics?

Policymakers and VR administrators could use the answers to these questions to understand how counselors and applicants respond when services become less readily available. With this understanding, administrators could then make adjustments as necessary, such as instructing counselors to maintain consistency in how they assess eligibility or provide services in the months before an OOS status change.

2 Background

The VR program provides supports and services to people with disabilities who want to work and are assessed as eligible to receive services. In federal fiscal year 2016, about 972,000 people were actively receiving these services (Rehabilitation Services Administration [RSA], 2017). Applying for VR is voluntary. Although funded primarily by the federal government, the VR program is administered at the state level by state VR agencies. (RSA in the U.S. Department of Education oversees the program at the federal level.) Each state has one or two VR agencies, depending on whether the state has a stand-alone agency to serve people who are blind.

When a state VR agency does not have enough resources to serve all eligible applicants in the state, the Rehabilitation Act (as amended by the Workforce Innovation and Opportunity Act of 2014) requires that the agency prioritize services to applicants with the most significant disabilities, a process known as OOS.

When someone applies to a state VR agency for services, the agency’s first step is to assess whether the applicant is eligible for services. If an agency does not have enough resources to serve all eligible applicants, each applicant is placed into an appropriate “priority-of-service” category based on the significance of her or his disability. Agencies can choose how many of these categories they have, but most create three: one for people with the most significant disabilities, another for those with significant disabilities, and another for all other applicants. Each category is either “open” or “closed” for the purposes of receiving services under OOS. Eligible applicants assigned to an open category receive services without delay, whereas those assigned to a closed category are placed on a waitlist for services.

An OOS status change occurs when an agency adjusts the set of categories that have a waitlist for services. Each agency chooses which categories are open and closed. We focus in the paper on situations when an agency closes categories. Before closing a category, the agency must perform certain activities, such as consulting with RSA, holding public meetings, and training agency staff. A closed lower-priority category cannot be opened if any higher-priority categories are also closed (Workforce Innovation Technical Assistance Center, n.d.; Silverstein, 2010).

SSI and SSDI beneficiaries make up about a third of VR applicants (Mann et al., 2017). If they pursue VR, these beneficiaries are typically required to assign their ticket from the Social Security Administration’s (SSA’s) Ticket to Work program to the relevant state VR agency. VR agencies are employment service providers eligible for payments under the Ticket to Work program. Because of the seriousness of the medical condition that qualified them for income support, SSI and SSDI beneficiaries are often assigned to agencies’ highest priority-of-service categories.

Between the announcement of an OOS status change and the date the change takes effect, there are several ways in which VR counselors’ and applicants’ behavior might change in anticipation of the change. VR counselors—who often want to help as many people as possible—might change how they conduct eligibility assessments. For example, counselors might assign applicants who are on the cusp of two priority-of-service categories to the higher-priority category, where they will be less likely to be affected by the OOS status change. Counselors might also speed up or change how they prioritize completing IPEs to serve as many customers as possible before the status change. In terms of applicant behavior, some people might decide not to apply for VR because they believe they will be placed on a waitlist for services. Conversely, other potential applicants might apply quickly in hopes of receiving a signed IPE before the OOS status change. 1

Over the past 15 years, macroeconomic conditions and federal legislation have influenced OOS status changes across the United States. Not long after the Great Recession began in late 2007, several state VR agencies responded to sudden decreases in program funding (driven by state budget cuts) by entering OOS or expanding the set of priority-of-service categories that were closed. Conversely, when the American Recovery and Reinvestment Act provided a sudden, short-term boost to VR funding from 2009 through 2011, some agencies reopened select priority-of-service categories (Hill & Mann, 2019). As the economy recovered in the early and mid-2010 s, agencies continued reducing the set of closed categories as resources became available.

New requirements for state VR agencies, as described in the Workforce Innovation and Opportunity Act, might be contributing to more recent expansions of OOS waitlist categories. In particular, the act requires that the agencies allocate at least 15 percent of their federal funding to providing pre-employment transition services to students. But this requirement was created without any additional funding, and as a result, many state VR agencies have had fewer resources to provide services to other applicants and may have closed certain priority-of-service categories.

This study makes a new contribution to our understanding of VR by documenting shifts in processes at a state VR agency in the months before expanding the set of OOS waitlist categories. Despite the extensive literature on VR services, we could not find any studies that empirically explored the potential effects of an OOS status change. Some studies mention how OOS might affect certain groups of applicants (Rumrill et al., 2004; Lamb, 2007). Others, such as Honeycutt et al. (2015), include OOS status as a factor in analyses that focus on different VR topics. However, we found no studies that address how changes in OOS status affect the practices of state VR agencies or the trade-offs that potential VR applicants make.

3 Methods

3.1 Data

Our analysis relied on data from the public use version of the RSA-911 case-service report files. The RSA-911 files provide case information for people who apply for or receive VR services. The files contain information about demographics, employment, and other characteristics at application; disability type and significance; VR service milestones; VR services received; employment status at case closure; and other data elements at program exit.

We analyzed the files from fiscal year 2006 through 2014, which contain data on closed VR cases only. Each part of our analysis included almost all—that is, a nearly 100 percent sample of—cases that closed during our analysis period, except for the few cases that closed after September 2014. 2 The analysis period varied slightly in each part of the analysis but included people who applied for VR in 2006, 2007, or 2008. The public use versions of the RSA-911 files do not contain any personally identifiable information, so we could not link multiple cases during the analysis period to a single applicant. However, evidence from Mann et al. (2017) suggests that few people have more than one record in the data.

Across the several dozen data elements from the RSA-911 files we used in the analysis, we found that some were rarely or inconsistently recorded for VR applicants who were found ineligible for services. Hence, relative to eligible applicants, the RSA-911 files have fewer details on characteristics of ineligible applicants.

3.2 Preliminary analysis

To begin the study, we used RSA-911 data to examine OOS status change events we could identify between 2007 and 2012 for all state VR agencies. We chose 2007 to 2012 because it includes the Great Recession and much of the subsequent recovery—a period when several expansions and reductions in the set of OOS waitlist categories occurred. We identified changes by examining the data for quick, substantive fluctuations over time in case outcomes for applicants to each agency, such as the share who exited from a waitlist or exited before receiving VR services. In some instances, we also contacted agency staff to learn more about a change, such as its exact date.

This preliminary analysis led us to focus on a single event: an OOS status change on August 4, 2008, made by Florida’s general VR agency that expanded the set of applicants who were waitlisted. Our analysis revealed substantive variation across agencies in how OOS status changes occurred and whom they affected. Each of the status changes we identified reflected a particular agency’s unique fiscal situation, service provision and administrative processes, and priority-of-service categories. Because of this inter-agency variation, pooling OOS status changes across agencies into a single analysis might yield little helpful information. Consequently, we decided to focus our analysis on a single agency. The 2008 OOS status change in Florida offered the opportunity to conduct a detailed case study of associated changes in VR process milestones. The change occurred at a fairly large agency, resulting in a relatively strong capacity to detect statistically significant correlations. And because Florida’s 2008 OOS status change affected all categories, we can use all eligible applicants in our analysis; further increasing our ability to detect statistically significant correlations.

The preliminary analysis offered other useful insights about how VR process milestones differed depending on when people apply relative to an OOS status change. People who applied to Florida’s general VR agency in the year before and after the 2008 OOS status change received a signed IPE less frequently than those who applied earlier. (Fig. 1 shows the distribution of eventual case closure outcomes at Florida’s general VR agency by date of application.) Among those who applied to the agency between August 2006 and August 2007, 60 to 65 percent exited the program after signing an IPE (including those who exit successfully with employment). The remaining 35 to 40 percent exited VR before signing an IPE, for reasons such as being ineligible for services or not making adequate progress with their counselor. However, starting with a few summer 2007 applicants and increasing after that, some applicants eventually exited the program while on a waitlist for services. Furthermore, among those who applied in the 12 months before the OOS status change, the share who received an IPE declined. The August 2008 applicant cohort—those who applied the month of the OOS status change—had the lowest proportion of applicants who exited after receiving an IPE. In addition, about 18 percent of applicants from that cohort eventually exited the program while on an OOS waitlist. Among those who applied 6 to 12 months after the change, about half exited after receiving an IPE, around 10 percent exited from a waitlist, and the rest exited before signing an IPE for other reasons.

Fig. 1

Select case closure outcomes for applicants at Florida’s general VR agency, by application month.

Examining trends in VR process outcomes associated with the OOS status change from a different perspective, we observe shifts in both the distribution of eligible applicants and the shares who signed IPEs (Table 1). The eligibility rate was 82 percent for those who applied in the third quarters of 2006 and 2007, but it dipped 6 percentage points by the third quarter of 2008—at which point the Great Recession had begun and Florida’s OOS status change had gone into effect. This decline was concentrated among those with nonsignificant disabilities (as indicated by the RSA-911’s disability-significance measure), for whom the eligibility rate fell from 54 percent to 28 percent. 3 The share of applicants with a significant disability grew by 5 percentage points between the third quarters of 2007 and 2008.

Table 1

Eligibility, disability significance, and IPE outcomes for applicants at Florida’s general VR agency, July through September 2006–2008

	All applicants			Applicants with a significant disability			Applicants with a nonsignificant disability
	July–September 2006	July–September 2007	July–September 2008	July–September 2006	July–September 2007	July–September 2008	July–September 2006	July–September 2007	July–September 2008
N	8,360	9,725	7,168	5,505	6,240	4,972	2,855	3,485	2,196
Has a significant disability (percentage)	65.8	64.2^*	69.4^**	100.0	100.0	100.0	0.0	0.0	0.0
Is eligible for services (percentage)	82.2	82.3	76.3^**	98.1	98.0	97.7	51.7	54.3^*	27.7^**
Signed an IPE (percentage)	64.6	62.6^**	40.2^**	79.4	75.5^**	56.1^**	36.1	39.5^**	4.1^**
Signed an IPE among those eligible for services (percentage)	78.5	76.0^**	52.6^**	80.9	77.1^**	57.4^**	69.9	72.7	14.8^**

Source: RSA-911 files for federal fiscal years 2006–2014. ^*/^**indicates a change from the previous year that is statistically significant at the 0.05 / 0.01 level based on a two-tailed test.

Our preliminary analysis also showed that application trends at Florida’s general VR agency varied in the years before the OOS status change (Table 1). Applications rose between the third quarter of 2006 and 2007 and then declined between the third quarter of 2007 and 2008. Overall, between the third quarters of 2006 and 2008, applications dropped by about 1,200 people (or 14 percent). All of these changes were more pronounced among applicants categorized as having a nonsignificant disability; for example, the overall change in the number of applicants in this group fell by 23 percent between the third quarters of 2006 and 2008.

3.3 Main analysis methods

3.3.1 Methods for analysis of eligibility rates and disability-significance distribution

To examine how the distribution of eligibility and disability-significance determinations changed, we predicted what the distribution would have been if the people applying for services around the OOS status change had the characteristics of those applying 12 months before the change.

We began by inspecting the distribution of eligibility and disability-significance determinations in the year before and the months just after the 2008 OOS status change in Florida (Table 2). The movement in the determinations across months was substantive: in the year before the OOS status change (August 2007 to July 2008), the share of applicants classified as having a nonsignificant disability and found eligible for services fell by 9 percentage points—from 19 percent to 10 percent. This decline was associated with increases in the share of applicants classified as having a significant disability and found eligible (3.9 percentage points) and the share classified as having a nonsignificant disability and found ineligible (4.8 percentage points).

Table 2
Actual eligibility and disability-significance determinations, by application month

Number of applicants Has a significant disability Has a nonsignificant disability

Eligible for services (percentage) Ineligible (percentage) Eligible for services (percentage) Ineligible (percentage)

August 2007 3,472 63.2 1.3 19.1 16.4

September 2007 3,171 64.0 1.4 18.3 16.4

October 2007 4,083 65.2 1.5 17.1 16.2

November 2007 3,411 64.4 1.2 18.0 16.3

December 2007 2,730 59.9 1.4 19.2 19.6

January 2008 3,163 62.9 1.4 16.2 19.6

February 2008 3,337 64.2 1.1 14.7 20.1

March 2008 3,135 62.6 1.4 15.9 20.2

April 2008 3,417 64.8 1.5 13.5 20.2

May 2008 2,875 65.0 1.6 11.9 21.4

June 2008 2,603 66.3 1.6 11.2 20.9

July 2008 2,746 67.1 1.7 10.0 21.2

August 2008 2,000 67.1 1.6 8.7 22.7

September 2008 2,422 69.1 1.4 6.6 22.8

October 2008 2,670 71.6 1.5 5.2 21.7

November 2008 2,005 72.6 2.0 4.6 20.8

	Number of applicants	Has a significant disability	Has a nonsignificant disability
August 2007	3,472	63.2	1.3	19.1	16.4
September 2007	3,171	64.0	1.4	18.3	16.4
October 2007	4,083	65.2	1.5	17.1	16.2
November 2007	3,411	64.4	1.2	18.0	16.3
December 2007	2,730	59.9	1.4	19.2	19.6
January 2008	3,163	62.9	1.4	16.2	19.6
February 2008	3,337	64.2	1.1	14.7	20.1
March 2008	3,135	62.6	1.4	15.9	20.2
April 2008	3,417	64.8	1.5	13.5	20.2
May 2008	2,875	65.0	1.6	11.9	21.4
June 2008	2,603	66.3	1.6	11.2	20.9
July 2008	2,746	67.1	1.7	10.0	21.2
August 2008	2,000	67.1	1.6	8.7	22.7
September 2008	2,422	69.1	1.4	6.6	22.8
October 2008	2,670	71.6	1.5	5.2	21.7
November 2008	2,005	72.6	2.0	4.6	20.8

Source: RSA-911 files for federal fiscal years 2006–2014.

Next, we used a logistic regression model to assess changes in the characteristics of applicants from September 2007 through November 2008—a period that spanned the OOS status change. The resulting estimates revealed how certain characteristics were correlated in a given month versus 12 months before the OOS status change. That is, the sample for each logistic regression included people who applied in a certain month as well as those who applied in August 2007—the 12th month before the status change.

The logistic regressions each had the following functional form: $\begin{matrix} Pr (INMON 12_{i} = 1 | X_{i}) = \frac{1}{1 + e^{- f_{i}}} \\ f_{i} = A X_{i} + ɛ_{i} \end{matrix}$ (1) where i denotes individual, INMON12 is a binary variable indicating whether someone applied in August 2007, and X is a vector containing the characteristics we observed for both eligible and ineligible applicants. The vector X includes a constant, age categories (less than age 25, ages 25 to 39, ages 40 to 54 [reference], and greater than age 54); sex (reference male); employment status at application (employed in an integrated setting without supports, non-employed student, and non-employed nonstudent or other employment [reference]); race and ethnicity indicators (white [reference], black non-Hispanic, Hispanic, and other race non-Hispanic); referral source (self-referral [reference], educational institution, medical provider, government program, or other); SSDI benefit receipt at application; SSI payment receipt at application; receipt of other public assistance at application; and whether the person had another VR case closure in the last three years.

Using logistic regression estimates, we reweighted the pool of applicants to account for changes in the characteristics and assessed the extent of any remaining differences in the eligibility and disability-significance distribution. We predicted the probability that applicants with a given set of characteristics applied in 12 months before the OOS status change versus each actual application month. We then created inverse probability weights for each month that adjusted for any differences between their characteristics and those of applicants in August 2007. After the reweighting, we conducted a chi-square test to determine whether there were meaningful differences in the predicted distributions of eligibility and disability significance—that is, differences not accounted for by the characteristics in the logistic model.

3.3.2 Methods for analysis of receiving a signed IPE

We estimated a linear probability model to explore how participant characteristics and time of application were associated with the probability of receiving a signed IPE. Our model took the following functional form: $Pr ({IPE}_{i} = 1 | Z_{i}) = B Z_{i} + Γ (Z_{i} t_{i}) + υ_{i}$ (2) where IPE is a binary variable indicating whether an eligible applicant received a signed IPE, Z is a vector of explanatory variables, and t is the month of application to VR (relative to the date of the OOS status change). The vector Z contains the same variables as X in equation (1), along with the following additions: disability-significance status; education at application (high school dropout, high school graduate [reference], or some college); and primary impairment type category (sensory/communicative, physical/mobility, other physical, cognitive, psychosocial [reference], other mental, or none). The regression includes all individuals who applied in the year before the OOS status change—August 5, 2007, to August 4, 2008—and were found eligible for services. We excluded from this analysis all those who applied after August 4, 2008, because they were placed on a waitlist the moment they were deemed eligible for services

4 Results

4.1 Eligibility rates and disability-significance distribution analysis

Our findings revealed meaningful differences in eligibility and disability-significance determinations shortly before and after Florida’s expansion of OOS waitlist categories that could not be fully accounted for by applicant characteristics (Table 3). The changes over time that we observed in the predicted distribution of applicants across eligibility and disability-significance categories were smaller in magnitude than we noted previously for the actual distribution. For example, between August 2007 and July 2008, holding characteristics constant, the predicted share of applicants classified as having a nonsignificant disability and found eligible for services fell by 5 percentage points, whereas the actual decline was 9 percentage points. However, the pattern of change for the predicted distribution was qualitatively similar to the actual distribution. The decline over time in the predicted share with a nonsignificant disability and found eligible was associated with increases in the predicted share with a significant disability and found eligible (1.4 percentage points) and the predicted share with a nonsignificant disability and found ineligible (3.5 percentage points). Moreover, the statistic from the chi-square test indicates that, overall, the change over time in the predicted distribution of these determinations, after accounting for characteristics, was statistically significant.

Table 3
Predicted eligibility and disability-significance determinations, by application month

Has a significant disability Has a nonsignificant disability

Eligible Ineligible Eligible Ineligible

Predicted (percentage) Difference from August 2007 (percentage points) Predicted (percentage) Difference from August 2007 (percentage points) Predicted (percentage) Difference from August 2007 (percentage points) Predicted (percentage) Difference from August 2007 (percentage points)

August 2007 63.2 n.a. 1.3 n.a. 19.1 n.a. 16.4 n.a.

September 2007 62.3 –0.8 1.5 0.1 20.5 1.4 15.8 –0.7

October 2007 62.5 –0.7 1.4 0.1 20.4 1.3 15.7 –0.7

November 2007 63.1 0.0 1.2 –0.1 20.4 1.3 15.2 –1.2

December 2007 60.9 –2.2 1.3 0.0 22.1 3.0 15.7 –0.7

January 2008 62.8 –0.3 1.4 0.1 18.7 –0.4 17.1 0.7

February 2008 63.6 0.4 1.1 –0.3 18.7 –0.4 16.6 0.2

March 2008 62.1 –1.1 1.3 0.0 19.6 0.5 17.0 0.5

April 2008 63.8 0.7 1.5 0.1 17.3 –1.8 17.4 1.0

May 2008 64.2 1.1 1.6 0.3 16.8 –2.3 17.3 0.9

June 2008 64.7 1.6 1.5 0.2 15.5 –3.6 18.3 1.9

July 2008 64.5 1.4 1.7 0.3 13.9 –5.2 19.9 3.5

August 2008 67.2 4.0 1.2 –0.1 12.1 –7.0 19.5 3.1

September 2008 69.1 5.9 1.3 0.0 10.4 –8.7 19.2 2.8

October 2008 71.7 8.5 1.4 0.1 9.5 –9.6 17.3 0.9

November 2008 72.7 9.5 1.7 0.3 7.8 –11.3 17.9 1.4

χ² (uncorrected 666.7^**

Pearson) statistic

	Has a significant disability	Has a nonsignificant disability
August 2007	63.2	n.a.	1.3	n.a.	19.1	n.a.	16.4	n.a.
September 2007	62.3	–0.8	1.5	0.1	20.5	1.4	15.8	–0.7
October 2007	62.5	–0.7	1.4	0.1	20.4	1.3	15.7	–0.7
November 2007	63.1	0.0	1.2	–0.1	20.4	1.3	15.2	–1.2
December 2007	60.9	–2.2	1.3	0.0	22.1	3.0	15.7	–0.7
January 2008	62.8	–0.3	1.4	0.1	18.7	–0.4	17.1	0.7
February 2008	63.6	0.4	1.1	–0.3	18.7	–0.4	16.6	0.2
March 2008	62.1	–1.1	1.3	0.0	19.6	0.5	17.0	0.5
April 2008	63.8	0.7	1.5	0.1	17.3	–1.8	17.4	1.0
May 2008	64.2	1.1	1.6	0.3	16.8	–2.3	17.3	0.9
June 2008	64.7	1.6	1.5	0.2	15.5	–3.6	18.3	1.9
July 2008	64.5	1.4	1.7	0.3	13.9	–5.2	19.9	3.5
August 2008	67.2	4.0	1.2	–0.1	12.1	–7.0	19.5	3.1
September 2008	69.1	5.9	1.3	0.0	10.4	–8.7	19.2	2.8
October 2008	71.7	8.5	1.4	0.1	9.5	–9.6	17.3	0.9
November 2008	72.7	9.5	1.7	0.3	7.8	–11.3	17.9	1.4
χ² (uncorrected	666.7^**
Pearson) statistic

Source: RSA-911 files for federal fiscal years 2006–2014. Note: The chi-squared statistic is based on a test for changes in the predicted distribution of eligibility and disability significance across the months listed in the table. Predicted values account for characteristics using the inverse-probability weights described in the text. n.a. = Not applicable. ^**indicates a change in the distribution across months that is statistically significant at the 0.01 level based on a two-tailed test.

4.2 IPE receipt probability analysis

For our IPE analysis, we focused on the coefficient estimates for the interactions between applicants’ characteristics and their date of application.

The interaction coefficient estimates revealed that the probability of receiving a signed IPE based on the time between application date and the expansion of OOS waitlist categories was correlated with several applicant characteristics (Table 4). Hispanics were an additional 0.5 percentage points less likely than white non-Hispanics to receive an IPE for each month closer their application date was to the OOS status change. Conversely, working in an integrated setting without supports (versus not working, working in a non-integrated setting, or working with supports) was associated with a 1.0 percentage point increase in the chances of receiving an IPE for each month closer the application date was to the OOS status change. Examining the referral-source coefficients, people referred to VR by another government program were an additional 0.6 percentage points more likely than people who self-referred to receive an IPE if they applied a month closer to the OOS status change. Three of the disability impairment-type categories—sensory/communicative, physical/mobility, and other mental—were associated with being decreasingly likely (relative to those with psychosocial impairments) to receive an IPE the closer one applied to the OOS status change.

Table 4
Regression estimates of IPE receipt

Variable Noninteracted variables Time-interacted variables

Coefficient Standard error Coefficient Standard error

Female –1.0 1.0 0.0 0.2

Age less than 25 at application –4.5^* 1.8 0.1 0.3

Age 40 to 54 at application 1.2 1.3 0.0 0.2

Age 55 or older at application 5.1^** 1.6 –0.5 0.3

Black, non-Hispanic –0.3 1.2 0.2 0.2

Other race, non-Hispanic 1.5 3.9 0.1 0.7

Hispanic 0.9 1.4 –0.5^* 0.2

Sensory/communicative impairment 24.1^ 1.6 –1.7^ 0.3

Physical/mobility impairment 1.9 1.6 –0.5^* 0.2

Other physical impairment 0.9 1.7 –0.5 0.3

Cognitive impairment 3.7^* 1.8 –0.1 0.3

Other mental impairment 8.3^** 2.3 –0.7^* 0.4

Less than high school education at application 2.0 1.2 –0.4^* 0.2

More than high school education at application 0.7 1.2 –0.0 0.2

Integrated employment, without supports 11.3^ 1.3 1.0^ 0.2

Student 10.4^** 2.3 –0.0 0.4

Receives SSI benefits –3.4^* 1.5 0.7^ 0.2

Receives SSDI benefits –5.5^ 1.4 0.6^ 0.2

Receives other public assistance 13.4^ 1.8 –0.1 0.3

VR: closure with same agency in past three years 0.2 1.1 0.1 0.2

Referred by educational institution –4.8 2.5 0.3 0.4

Referred by medical provider 4.9^** 1.5 0.1 0.2

Referred by government program –4.2^* 2.0 0.6^* 0.3

Other referral source 1.5 1.3 0.2 0.2

Has a significant disability 16.6^ 1.4 0.4 0.2

Reference group 55.7^ 2.1 –1.5^** 0.3

Sample size 30,127

Variable	Noninteracted variables	Time-interacted variables
Female	–1.0	1.0	0.0	0.2
Age less than 25 at application	–4.5^*	1.8	0.1	0.3
Age 40 to 54 at application	1.2	1.3	0.0	0.2
Age 55 or older at application	5.1^**	1.6	–0.5	0.3
Black, non-Hispanic	–0.3	1.2	0.2	0.2
Other race, non-Hispanic	1.5	3.9	0.1	0.7
Hispanic	0.9	1.4	–0.5^*	0.2
Sensory/communicative impairment	24.1^**	1.6	–1.7^**	0.3
Physical/mobility impairment	1.9	1.6	–0.5^*	0.2
Other physical impairment	0.9	1.7	–0.5	0.3
Cognitive impairment	3.7^*	1.8	–0.1	0.3
Other mental impairment	8.3^**	2.3	–0.7^*	0.4
Less than high school education at application	2.0	1.2	–0.4^*	0.2
More than high school education at application	0.7	1.2	–0.0	0.2
Integrated employment, without supports	11.3^**	1.3	1.0^**	0.2
Student	10.4^**	2.3	–0.0	0.4
Receives SSI benefits	–3.4^*	1.5	0.7^**	0.2
Receives SSDI benefits	–5.5^**	1.4	0.6^**	0.2
Receives other public assistance	13.4^**	1.8	–0.1	0.3
VR: closure with same agency in past three years	0.2	1.1	0.1	0.2
Referred by educational institution	–4.8	2.5	0.3	0.4
Referred by medical provider	4.9^**	1.5	0.1	0.2
Referred by government program	–4.2^*	2.0	0.6^*	0.3
Other referral source	1.5	1.3	0.2	0.2
Has a significant disability	16.6^**	1.4	0.4	0.2
Reference group	55.7^**	2.1	–1.5^**	0.3
Sample size	30,127

Source: RSA-911 files for federal fiscal years 2006–2014. Note: This table contains parameter estimates for a linear probability model. This model predicts IPE completion before the date of the OOS status change as a function of the month of application and various other characteristics. Coefficients on noninteracted variables represent the estimated difference in the probability of completing an IPE associated with the given variable in early August 2007. Coefficients on time-interacted variables represent the change in the estimated probability in each subsequent month, up through early August 2008. ^*indicates a 0.05 significance level, two tailed test. ^**indicates a 0.01 significance level, two tailed test. IPE = Individualized Plans for Employment; OOS = order of selection; SSDI = Social Security Disability Insurance; SSI = Supplemental Security Income; VR = vocational rehabilitation.

Both SSI and SSDI benefit receipt at application were associated with being increasingly more likely to receive an IPE the closer one’s application was to the OOS status change. Specifically, SSI recipients were an extra 0.7 percentage points more likely and SSDI beneficiaries were an extra 0.6 percentage points more likely than non-recipients to receive an IPE in each subsequent month during the year leading up to the OOS status change.

5 Discussion

Our study examined early VR process milestones at Florida’s general VR agency to provide insight on whether VR counselors or applicants adjust their behavior in response to a pending closure of the priority-of-service categories. We started by examining whether the distribution of case-closure outcomes shifted substantively for those who applied in the months before this OOS status change. After finding evidence of movement in the share of applicants who closed with an IPE over that period, we conducted two additional analyses. In this section, we discuss our findings from each analysis, place these findings in context, and assess potential implications for policy.

In the first analysis, we examined the predicted the distribution of eligibility and disability-significance determinations, after accounting for certain applicant characteristics. This analysis showed that people who applied for VR services in the months around the OOS status change were more likely than those who applied 12 months before the change to (1) be eligible for services and have a significant disability or (2) be ineligible for services and not have a significant disability. What those two process milestones have in common is that they strongly influence an applicant’s chances of eventually receiving services when a VR agency operates under OOS. In general, those found to be program eligible and have significant disabilities are the most likely to receive program services and avoid being on a waitlist. Conversely, those found to have nonsignificant disabilities are less likely to receive services because of lower eligibility rates or because they are more likely to be on a waitlist when they are found eligible for VR. Increases in the share of these two groups was offset by a decrease in the share of people found to be eligible but who had nonsignificant disabilities—the group most likely to be affected by an OOS status change.

These results suggest that VR counselors might adjust how they initially screen applicants when an expansion of OOS waitlist categories is imminent. Faced with a pending change in OOS status, counselors could sort applicants in a way that aligns eligibility with anticipated waitlist status. If there is some subjectivity in the underlying decisions, counselors could make eligibility and disability-significance determinations so that those deemed eligible will be less likely to be placed on a waitlist—whereas those found ineligible would be more likely to be waitlisted (and less likely to receive timely services), had they been eligible.

Alternatively, the observed changes over time could reflect shifts in applicant characteristics that cannot be measured in the available administrative data for Florida. For example, the Great Recession could have differentially affected workers with disabilities of varying levels of education, types of past training, impairment types, or fields of expertise, resulting in changes to the pool of people who sought VR services.

Either type of sorting would limit the capacity of policymakers and state VR agency administrators to project their future VR caseloads after an OOS status change based on the current distribution of program eligibility and priority of service categorizations. Agency leaders could monitor agency administrative data to determine whether sorting is occurring and consider collecting additional data to help establish why there are changes over time. If the decisions of VR intake counselors turn out to play a role, agency administrators might instruct the counselors to maintain greater consistency in how they determine eligibility and disability significance.

Our second analysis focused on how the chance of receiving an IPE—which essentially marks the beginning of VR service receipt—was correlated with the time of application (relative to the OOS date) and applicant characteristics. The regression estimates revealed that certain characteristics, such as impairment type and receipt of SSI or SSDI benefits, were correlated with a greater or lesser chance of IPE receipt the closer the person’s application date was to the official date for the expansion of OOS waitlist categories.

An examination of which characteristics are associated with increasing an applicant’s chance of receiving an IPE before the OOS status change suggests that policies related to VR performance measures might motivate some of the correlations, although other explanations are possible. The probability of having a successful VR outcome—that is, being employed at the time of case closure—is correlated with various characteristics, especially certain types of disability and an applicant’s initial employment status (Hayward & Schmidt-Davis, 2003; Mann et al., 2017). And so, in the months before an OOS status change, counselors could have prioritized IPEs for applicants who had disabilities that are more often linked to successful program outcomes or who were employed when they applied for VR services. This behavior would make sense from the counselor perspective because the number of successful case closures is a key performance measure for counselors at most agencies. But it could also be that applicants with certain conditions or who are employed can, for whatever reason, more generally get a signed IPE faster and more easily than other applicants.

The main exception to this pattern is that VR applicants who were SSI and SSDI beneficiaries were also more likely to get signed IPEs relative to other applicants in the months before the expansion of OOS waitlist categories. But relative to nonbeneficiaries, SSI and SSDI beneficiaries are also less likely to have successful VR case closures (Hayward & Schmidt-Davis, 2003; Mann et al., 2017). However, when SSI and SSDI beneficiaries achieve substantive sustained earnings after receiving VR services, the agency that provided the services receives payments from SSA, either as cost reimbursement or as payments under the Ticket to Work program (Schimmel Hyde & O’Leary, 2017). Hence, the financial incentives might encourage resource-strained agencies to prioritize IPEs and, therefore, VR services to SSI and SSDI beneficiaries over nonbeneficiaries. Alternatively, disability beneficiaries might also receive IPEs more quickly than other VR applicants for other reasons, such as counselors being able to quickly make initial determinations based on RSA’s presumptive eligibility guidelines.

If policymakers and agency administrators want to adjust how counselors prioritize IPE completions as an OOS status change approaches, these leaders might want to assess the incentives and constraints facing their counselors. For example, if counselors’ performance reviews are based in part on how many successful case closures they have, VR leaders might consider accounting for counselor caseload characteristics when assessing counselor performance so that successfully closing difficult cases receives relatively high praise. The agency administrators should also consider explaining to counselors the budgetary implications of a surge in IPEs just before an OOS status change. Though the IPE surge benefits applicants who apply just before OOS status change and would be affected by the change, an IPE surge increases the agency’s budget shortfall and consequently might increase service delays for those who apply after the change.

Many of the policy recommendations we make require VR leaders to use their case management data in ways that are potentially new to them. For example, if agency administrators instruct the counselors to maintain greater consistency when making eligibility decisions, the administrators should also regularly monitor eligibility decisions in their case management data to assess compliance. That way, the agency could provide targeted assistance to staff who seem to be changing how they make eligibility determinations. Our performance review recommendation would require supervisors to use case management data to review counselor caseload characteristics when assessing counselor performance. VR leaders could also use case management data to frequently monitor signed IPEs before the OOS status change and follow up as needed with staff whose signed IPEs are surging. If VR leaders can learn to use their case management data in these ways, their ability to implement and monitor policy changes at their agency would substantively increase.

Though our results suggest that changes in OOS status might affect the decisions of VR counselors and applicants, our study has limitations that prevent us from drawing further conclusions. Most important, this study was not causal, so we cannot say that the OOS status change is directly responsible for the patterns we saw. Rather, we measured correlations between key characteristics and process milestones of interest. Conducting a causal study would be difficult because there is no source of exogenous variation that we could use to help create causal estimates. In addition to not being a causal study, the study’s external validity could also be limited because we focused on one OOS status change at a single agency. Finally, owing to incomplete or missing data in the RSA-911 files, our analyses may not sufficiently control for applicant characteristics when examining determinations of eligibility and disability significance or probabilities of IPE receipt.

Conflict of interest

None to report.

Footnotes

Acknowledgments

The research reported herein was performed pursuant to a grant from the U.S. Social Security Administration (SSA) that is funded as part of the Disability Research Consortium. The opinions and conclusions expressed are solely those of the authors and do not represent the opinions or policy of SSA or any agency of the federal government. Neither the U.S. government nor any agencies thereof, nor any of their employees, make any warranty, expressed or implied, or assume any legal liability or responsibility for the accuracy, completeness, or usefulness of the contents of this report. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply endorsement, recommendation, or favoring by the U.S. government or any agency thereof.

We originally set out to identify OOS status changes at certain VR agencies that could allow us to estimate VR’s impact on employment and benefit receipt. That study would have added to the literature of quasi-experimental studies that measure how VR affects its customers’ outcomes (see, for example, Dean et al., 1999, Nazarov, 2013, and Dean et al., 2014). But after considering several OOS status changes and different methodological approaches, we were unable to identify a study design whose key assumptions were met. In each case, we found evidence of changes in applicant and counselor behavior that would limit the capacity to measure VR’s causal effects by comparing the outcomes of applicants over time.

Among the 1,790,028 VR cases opened in fiscal years 2006, 2007, and 2008, we observed 1,778,431 (99.4 percent) in the 2006 through 2014 RSA-911 files. Among the 99,621 cases served by Florida’s general VR agency between January 2006 and September 2008, we excluded 154 cases with missing gender information and 2 cases with no impairment information.

The disability-significance categories in the RSA-911 do not necessarily correspond to agencies’ priority-of-service categories. Moreover, the disability-significance measure in the RSA-911 files we examined was binary up through the fiscal year 2013 file (that is, the measure indicated whether an applicant did or did not have a significant disability). Starting with the fiscal year 2014 file, this measure could take one of three values: most significantly disabled, significantly disabled, and not significantly disabled. For our analysis, we recoded the three-category disability-significance measure to align with the binary measure.

References

Dean,

, Dolan,

, & Schmidt,

(1999). Evaluating the vocational rehabilitation program using longitudinal data: Evidence for a quasi-experimental research design, Evaluation Review23(2), 162–189.

Dean,

, Pepper,

J. V

, Schmidt,

R. M

, & Stern,

(2014). State vocational rehabilitation programs and federal disability insurance: An analysis of Virginia’s vocational rehabilitation program, IZA Journal of Labor Policy3(1), 1–19.

Hayward,

B. J.

, & Schmidt-Davis,

(2003, May). Longitudinal study of the vocational rehabilitation services program final report: VR services and outcomes. Research Triangle Institute.

Hill,

, & Mann,

D. R.

(2019). Do short-term changes in funding improve vocational rehabilitation outcomes? Evidence from the ARRA, Journal of Vocational Rehabilitation51(3), 355–367.

Honeycutt,

, Thompkins,

, Bardos,

, & Stern,

(2015). State differences in the vocational rehabilitation experiences of transition-age youth with disabilities, Journal of Vocational Rehabilitation42(1), 17–30.

Lamb,

(2007). Implications of the summary of performance for vocational rehabilitation counselors, Career Development for Exceptional Individuals30(1), 3–12.

Mann,

D. R

, Honeycutt,

, Bailey,

M. S

, & O’Neill

(2017). Using administrative data to explore the employment and benefit receipt outcomes of vocational rehabilitation applicants years after program exit, Journal of Vocational Rehabilitation46(2), 159–176.

Nazarov,

(2013). Can benefits and work incentives counseling be a path to future economic self-sufficiency for SSI/SSDIbeneficiaries? Working Paper No. 17. Center for Retirement Research.

Rehabilitation Services Administration (RSA). (2017). Vocational rehabilitation national summary. https://www2.ed.gov/programs/rsabvrs/resources/fy2016-vr-performance-chart.pdf.

10.

Rumrill,

P. D.

Jr , Roessler,

R. T

, & Fitzgerald,

S. M.

(2004). Vocational rehabilitation-related predictors of quality of life among people with multiple sclerosis, Journal of Vocational Rehabilitation20(3), 155–163.

11.

Schimmel Hyde,

, & O’Leary,

(2017, March). Social Security Administration payments to state vocational rehabilitation agencies for beneficiaries who work: Evidence from linked administrative data. Disability Research ConsortiumWorking Paper No. 2017-02. Mathematica Policy Research.

12.

Silverstein,

(2010). A description and analysis of the federal and selected state policy frameworks regarding order of selection under Title I of the Rehabilitation Act. Institute for Community Inclusion, Rehabilitation Research and Training Center on Vocational Rehabilitation.

13.

Workforce Innovation Technical Assistance Center. (n.d.). Steps and process when state VR agencies implement order of selection. Retrieved June 21, 2019 http://wintac-s3.s3-us-west-2.amazonaws.com/topic-areas/ta04-IntergrationVR/OOS_Steps_and_Process.pdf.

	Has a significant disability				Has a nonsignificant disability
	Eligible		Ineligible		Eligible		Ineligible
	Predicted (percentage)	Difference from August 2007 (percentage points)	Predicted (percentage)	Difference from August 2007 (percentage points)	Predicted (percentage)	Difference from August 2007 (percentage points)	Predicted (percentage)	Difference from August 2007 (percentage points)
August 2007	63.2	n.a.	1.3	n.a.	19.1	n.a.	16.4	n.a.
September 2007	62.3	–0.8	1.5	0.1	20.5	1.4	15.8	–0.7
October 2007	62.5	–0.7	1.4	0.1	20.4	1.3	15.7	–0.7
November 2007	63.1	0.0	1.2	–0.1	20.4	1.3	15.2	–1.2
December 2007	60.9	–2.2	1.3	0.0	22.1	3.0	15.7	–0.7
January 2008	62.8	–0.3	1.4	0.1	18.7	–0.4	17.1	0.7
February 2008	63.6	0.4	1.1	–0.3	18.7	–0.4	16.6	0.2
March 2008	62.1	–1.1	1.3	0.0	19.6	0.5	17.0	0.5
April 2008	63.8	0.7	1.5	0.1	17.3	–1.8	17.4	1.0
May 2008	64.2	1.1	1.6	0.3	16.8	–2.3	17.3	0.9
June 2008	64.7	1.6	1.5	0.2	15.5	–3.6	18.3	1.9
July 2008	64.5	1.4	1.7	0.3	13.9	–5.2	19.9	3.5
August 2008	67.2	4.0	1.2	–0.1	12.1	–7.0	19.5	3.1
September 2008	69.1	5.9	1.3	0.0	10.4	–8.7	19.2	2.8
October 2008	71.7	8.5	1.4	0.1	9.5	–9.6	17.3	0.9
November 2008	72.7	9.5	1.7	0.3	7.8	–11.3	17.9	1.4
χ² (uncorrected	666.7^**
Pearson) statistic