Phase I development of the SGA Model: Use of administrative data and expert opinion to identify key components of the SGA Model

Abstract

BACKGROUND:

The SGA Model Demonstration tested a coordinated team approach that integrated specific components of vocational rehabilitation services to determine if the model would increase earnings outcomes of Social Security Disability income beneficiaries who sought VR services in Kentucky and Minnesota.

OBJECTIVE:

This paper provides details on the first phase of development of the SGA intervention.

METHODS:

Researchers combined a Delphi process, key informant interviews, and administrative data review to identify practices that were high priority for inclusion in the model.

RESULTS:

Researchers reached a consensus on the high priority components to construct a testable intervention under the control of the vocational rehabilitation system.

CONCLUSIONS:

The SGA Project team identified three practice domains to guide an intensive case study for Phase II development of the intervention. These included pacing of services, work incentives counseling, and strategies for employment service delivery.

Keywords

Vocational rehabilitation pacing work incentives counseling SSDI SGA employment services

1 Introduction

Separate bureaucracies with different histories, cultures, rules, authorizing legislation and purposes govern vocational rehabilitation services and public disability insurance programs (Kearney, 2005). The Rehabilitation Services Administration (RSA) has a mandate to serve a wide range of persons with disabilities to enter or stay in the workforce, with a priority for those with the most significant disabilities. The Social Security Administration (SSA) has a critical and challenging mandate to control the costs of a public disability insurance program. Policy tweaks over the decades suggest that public employment services and disability insurance programs have a mutual interest to advance return to work strategies that promote earnings outcomes beyond reliance on public benefits. Yet, there remains considerable uncertainty about which specific combinations of public employment services encourage Social Security Disability Insurance (SSDI) beneficiaries become economically independent.

Large scale model demonstrations, generally funded by the SSA, have tested a range of interventions for SSDI beneficiaries through changes in work incentives (e.g., the Benefits Offset National Demonstration), access to health care (e.g., the Accelerated Benefits Demonstration), and referrals to or use of employment counseling as part of a package of services (e.g. Mental Health Treatment Study). Nearly every demonstration found improvements such as use of health or mental health services and re-entry into employment (c.f., Michaelopoulos, Wittenberg, Israel, et al., 2011; Drake, Frey, Karakus, et al., 2016, Gubits, Geyer, Stapleton, Greenberg et al., 2018). Yet across demonstrations, there was little evidence of substantial improvement in earnings leading to reduced reliance on federal disability benefits within the time window of the evaluation. (Social Security Administration, 2019)

SVRAs may have much to offer in understanding the combination of components necessary to encourage return to work outcomes. The SGA Model Demonstration “SGA Project” was funded by the RSA to identify, implement, and evaluate an employment services intervention that had the potential to increase wage outcomes of SSDI clients of state vocational rehabilitation agencies (SVRAs). RSA required that the model demonstration focus on practices that could be implemented by an SVRA and be limited to non-blind SSDI only beneficiaries. Another requirement was a search for practices already in existence and, if transferable, to test those in another SVRA. This paper describes Phase 1 of intervention development detailing the identification of high performing SVRAs, expert recommendations about service components, and analysis of administrative data to assess the influence of those components on earnings outcomes. We were also interested in how expert opinion could be refined, supported or confirmed by available administrative data.

Hereafter, this paper will refer to persons who receive vocational rehabilitation services and who are also beneficiaries of SSDI based on their own work history as “SSDI clients” and state vocational rehabilitation agencies as “SVRAs.”

2 Methods

Phase I included a heavy use of administrative data sets that could be used to examine recommendations from experts and practitioners. The SGA Project team included researchers, policy fellows, and technical assistance and training personnel working at the Institute for Community Inclusion at the University of Massachusetts Boston and at Mathematica Policy Research. The goal for Phase I was identifying the most viable components for an intervention and a focus for case studies of eight SVRAs (Phase 2). Mathematica Policy Research took the lead on data analysis. Findings reported in this paper are summaries of technical documentation and cited as relevant (Honeycutt & Livermore, 2011a; Honeycutt & Livermore, 2011b).

Phase I was guided by four tasks. The first was to determine SVRA level of variation in earnings outcomes for SSDI clients and identify above average SVRAs through administrative data analysis. The second was to gather the views of SVRA directors about what explains performance rankings. The third was to gather expert opinion on which practices under the control of SVRAs lead to improved earnings outcomes. The final task was analysis of available data to determine if the nominated practices differed by performance rankings.

2.1 Task 1: Identification of high performing SVRAs

Mathematica Policy Research team members took the lead and provided technical documentation cited and available from the author. Identification of high performers included an analysis of case record data publicly available through RSA and an analysis of a matched case record data file of RSA and SSA data.

2.1.1 Research questions

The team addressed the following research questions:

What is the variation in achieving SGA level earnings for SSDI clients?

Which agencies rank as high performers in earnings level outcomes of SSDI clients?

Are rankings stable over time? Do any SVRAs show a stable trend upward?

Do rankings change when matching SSA earnings data with RSA 911 administrative data?

2.1.2 Data sources

We relied upon two administrative datasets to create two samples of SSDI clients: a) those identified in RSA administrative data only; and b) VR clients who are SSDI beneficiaries as identified in SSA data. Other data sources provided SVRA and state level factors (e.g., unemployment rate).

RSA 911 data: SVRAs are required to provide the RSA with individual level data such as demographic, employment status at application and closure, disability, education, beneficiary status, service provision, costs, closure reasons, and wage variables for all persons served. The analysis file included data on closures (i.e., people completing VR services) during program years 2005 through 2009 to examine trends over time.

SSA Ticket Research File (TRF) and matching process: The SGA Project acquired permission to access a matched records file of SSDI clients served by SVRAs and identified in the Ticket Research File (TRF) and for Mathematica Policy Research to receive and analyze the data. The primary outcome was an indication in the TRF of earnings at or above substantial gainful activity (SGA) at any time during the period following VR application and 24 months after VR case closure. Although the RSA-911 data include information about SSDI benefit receipt, matching the data to the TRF provided more accurate information about the receipt of benefits at both application and closure.

Other RSA data sources: The RSA 113 datasets provided the number of clients receiving services, number of applicants and persons on waiting lists. The RSA 2 dataset provided personnel ratios (counseling staff to all staff, administrative staff to all staff, staff supporting counselors to all staff), which entity provided services (SVRA personnel, public community rehabilitation providers, private community rehabilitation providers, other public vendors, other private vendors) and percentage of service expenditures on post-secondary education, job readiness training, vocational training and other training. Combining data from RSA 2 and RSA 113, researchers were able to construct variables that provided ratios of customers to types of SVRA staff and of total expenditures to customers.

Other sources: SSA data provided state level data on the number of SSDI beneficiaries and number terminated due to work. The researchers culled state level population and economic indicators from the Census Bureau (population density), the Bureau of Economic Analysis (per capita income) and Department of Labor (unemployment rate per year).

2.1.3 Sample

We included the 50 SVRAs that were general or combined agencies and excluded the Commissions for the Blind due to study parameters. SVRAs in the territories, Puerto Rico, and the District of Columbia are comparatively small agencies and were excluded because rankings for a small subpopulation could be highly variable from year to year.

SVRA clients were included if their primary disability was not blindness, were between 18 to 62 years of age at application, and if it was their first closure with VR. Clients were excluded if they had died or the record had missing demographic data (gender, race, ethnicity or education). Exclusions reduced the sample by 18.6% resulting in 2,475,946 case records. The matched file was further refined so that comparisons of income beyond VR could be used to identify those that achieved specific earnings outcomes. Using TRF file variables, case records were excluded if the person had died within two years of receiving VR or was receiving SSDI benefits due to blindness (Honeycutt & Livermore, 2011b).

2.1.4 Data analysis

The Mathematica Policy Research partners used hierarchical linear regression models to determine how 50 SVRAs rank in their performance on earnings levels at closure to VR or for earnings outcomes recorded in the TRF file as detailed in Table 1. Client characteristics, agency variables, and state level variables were used to control for factors that may contribute to SVRA differences in performance on earnings, such as unemployment rates, waiting lists, percentage of clients who are not working at application. Table 1 describes the six analytical models. Model 1 is the intended intervention population. Models 2 and 3 extend the sample to people recorded as having SSDI and SSI at application or SSDI at closure. This was out of concern that SVRAs may vary in their ability to identify SSDI only clients at application. Model 4 assessed whether or not high performers for SSDI client earnings were high performers on earnings outcomes in general. Models 5 and 6 use the matched cohort but differ in the degree they were served by an SVRA. Model 5 included all SSDI applicants to an SVRA regardless of whether or not they received services. Model 6 included SSDI clients if they received VR services.

Table 1
Six analysis models for identifying high performing SVRAs

Model Sample Outcome measure Rationale

RSA 911 data only

1 SSDI only at application SGA level earnings at closure Intended intervention population

2 All SSDI at application SGA level earnings at closure SVRAs may not be accurate at application on SSDI or SSI status

3 SSDI only at application or closure SGA level earnings at closure SVRAs may not be accurate at application on SSDI status

4 All clients SGA level earnings at closure High performing SSDI only may be high performing for all

Matched cohort

5 SVRA applicants with SSDI only benefits 2 months prior or after application month Benefits suspended or terminated due to earnings; earnings at or above SGA in any 9 months; or use of the trial work period work incentive All SSDI only beneficiaries that applied to an SVRA even if the SVRA may not have known or was not accurate about the SSDI status.

6 Same as Model 5 Model 5 but limited to only those served (status 26/28) Sensitivity test for Model 5 SVRA agency had a likely role in those outcomes.

Model	Sample	Outcome measure	Rationale
RSA 911 data only
1	SSDI only at application	SGA level earnings at closure	Intended intervention population
2	All SSDI at application	SGA level earnings at closure	SVRAs may not be accurate at application on SSDI or SSI status
3	SSDI only at application or closure	SGA level earnings at closure	SVRAs may not be accurate at application on SSDI status
4	All clients	SGA level earnings at closure	High performing SSDI only may be high performing for all
		Matched cohort
5	SVRA applicants with SSDI only benefits 2 months prior or after application month	Benefits suspended or terminated due to earnings; earnings at or above SGA in any 9 months; or use of the trial work period work incentive	All SSDI only beneficiaries that applied to an SVRA even if the SVRA may not have known or was not accurate about the SSDI status.
6	Same as Model 5	Model 5 but limited to only those served (status 26/28)	Sensitivity test for Model 5 SVRA agency had a likely role in those outcomes.

A full review of SSDI client, agency, and state level characteristics related to SVRA earnings outcomes is beyond the scope of this paper. Readers are referred to Honeycutt & Stapleton (2013) for analysis of the impact of waiting time on SSDI clients. Directors of SVRAs that were identified as high performers or trending upwards in Model 1 were included in the sample for Task 2.

2.2 Task 2: SVRA director views on performance rankings

2.2.1 Method, sample, data collection and analysis

ICI interviewed SVRA leaders representing high performing or improving SVRAs. SVRA interviewees included the leaders of the 20 SVRAs identified as high performers in Model 1 (RSA 911 only). In addition, we included the 7 SVRAs that were showing large gains in earnings outcomes over a five-year period. Interviews were brief (averaging 15 to 30 minutes), tape-recorded with permission and followed a semi-structured interview guide. SVRA interviewees were asked a) if the SVRA had any targeted initiatives for SSDI clients; and b) what practices might explain high performance or improving performance on earnings outcomes. ICI received approval for the interviews through the UMass Boston Institutional Review Board. Interview data was examined to generate a list of practices for Task 3, identify any SVRAs that had a targeted intervention, and identify SVRA invitees for Task 3.

2.3 Task 3: Identify VR practices likely to lead to improved earnings outcomes of SSDI clients

2.3.1 Methods

The SGA team used a modified Delphi panel approach (Fleming, Boeltzig-Brown, and Foley, 2015) and assembled a panel of experts to score and discuss nominated practices.

2.3.2 Panel members

We purposefully weighted the panel to include SVRA personnel for three reasons: a) a key requirement was that intervention components were under SVRA control; b) an interest in identifying SVRA emerging innovations that may not have been studied; and c) understanding the barriers to implementing a randomized controlled trial in an SVRA. We invited 37 experts from eight categories and received consent from 25 persons. The eight categories of invitees include SVRA leadership (n = 10), a representative from the Council on Administrators of Vocational Rehabilitation (n = 1), University-based researchers and training personnel (n = 8), current and former Social Security Administration representatives (n = 2), Rehabilitation Services Administration personnel (n = 5), Vendors of SVRA purchased services (n = 4), representatives of State Departments of Mental Health (n = 3), NIDRR (now known as NIDILRR) personnel (n = 4). NIDRR personnel had to decline the invitation due to rules about participation and potential conflict of interest.

2.3.3 Rounds and data collection

The panel members independently reviewed practices and scored each through an online portal that included instructions, definitions for each practice, and a scoring process (Round 1). The ICI created a master list of 72 service delivery practices identified during key informant calls, in the research literature, document review, or by the SGA team. Each practice was included if it was suggested or potentially related to improved wage outcomes of SSDI clients. Panel members rated each practice for its likeliness to enhance earnings (1 = will not to 4 = very likely to enhance earnings) and whether the practice was under the control of VR agencies (1 = does not control to 5 = full control). The online survey included open-ended questions for any additional relevant input. The SGA Team convened the panel by webinar to review summaries of scores and Task 1 findings and to provide a forum for discussion (Round 2).

2.3.4 Data analysis

Panel member ratings were summarized and grouped according to importance and under SVRA control. Narrative data were grouped into themes. The Round 2 discussion transcript was summarized according to themes and for identifying additional practices that could be used in Task 4.

2.4 Task 4: Investigate nominated practices and whether those practices contribute to Task 1 SVRA performance rankings

2.4.1 Methods and analysis

We attempted to identify variables in RSA data that most closely matched the practices identified by the Delphi Panel members as likely to improve earnings outcomes. We focused on practices that at least one half of the Delphi Panel members rated as important (n = 30). The task at hand was to determine if expert opinion tended to coincide with administrative data. The Delphi panel nominated multiple practices that were agency level capacities such as investing in business relationships or in counselor deployment strategies such as use of specialty counselors. These practices could not be matched to available administrative data. A cluster of practices were focused on prompt delivery of services and were defined by median days to the corresponding vocational rehabilitation process (i.e., eligibility, plan development, and closure). For each variable, an SVRA was given a 1 or a 0 if implementation was above median or below median scores.

3 Results

Results are presented in order by task. Each task informed the other by creating interview samples, identifying practices, constructing variables for analysis, and creating consensus on critical components.

3.1 Task 1: Variation in performance and identification of high performance

Results presented are based upon technical documentation from Honeycutt & Livermore (2011a) for RSA 911 analysis and Honeycutt & Livermore (2011b) for the matched file cohort.

3.1.1 Variation in state level performance

During the program years examined, SVRAs closed 3,041,088 clients and served 288,426 persons who were receiving SSDI only. Of those 34,307 (11.9%) closed out of VR with earnings over SGA levels for the specific program year. SVRA level percentages varied greatly from 22.7% in Alabama to 5.7% in Wisconsin.

3.1.2 High performance in achieving SGA outcomes in RSA 911

Table 2 lists the SVRAs by ranking for Model 1 and the combined Models 5 and 6. For the RSA only model, we ranked SVRAs by three categories: above average (n = 20); average (n = 14); and below average (n = 16). Only minor changes in rankings occurred across the RSA 911 analysis models. Rankings from Model 4 analysis (earnings outcomes for all SVRA clients) indicated that all 16 of the agencies identified as below average were also below average in achieving SGA level earnings outcomes for all clients. 16 of the 20 above average SVRAs were above average on earnings outcomes for all clients served.

Table 2
High performance rankings of SVRAs

Ranking Model 1 Model 5 and 6 combined

Above Average Alabama, Alaska, Arkansas, California, Delaware, Florida, Georgia, Kentucky, Michigan, Mississippi, Nebraska, Nevada, New Jersey, New York, Oregon, Pennsylvania, Rhode Island, South Carolina, Utah, Vermont Alabama, Arkansas, Arizona, California, Nebraska, Nevada, Pennsylvania, South Carolina, Virginia

Potentially Above Average N/A Georgia, Kentucky, Oregon, Vermont, West Virginia

Average Arizona, Idaho, Illinois, Kansas, Maryland, Massachusetts, New Hampshire, New Mexico, North Dakota, Ohio, Texas, Virginia, West Virginia, Wyoming Alaska, Connecticut, Florida, Hawaii, Idaho, Kansas, Mississippi, Montana, New Hampshire, New Jersey, New Mexico, North Dakota, Rhode Island, Texas, Utah, Wyoming

Potentially Below Average N/A Delaware, Illinois, Indiana, Louisiana, Maine, Maryland, Michigan, New York, North Carolina, Oklahoma, South Dakota, Tennessee, Washington

Potentially Below or Below Average Colorado, Connecticut, Hawaii, Indiana, Iowa, Louisiana, Maine, Minnesota, Missouri, Montana, North Carolina, Oklahoma, South Dakota, Tennessee, Washington, Wisconsin Colorado, Iowa, Massachusetts, Minnesota, Missouri, Ohio, Wisconsin

Ranking	Model 1	Model 5 and 6 combined
Above Average	Alabama, Alaska, Arkansas, California, Delaware, Florida, Georgia, Kentucky, Michigan, Mississippi, Nebraska, Nevada, New Jersey, New York, Oregon, Pennsylvania, Rhode Island, South Carolina, Utah, Vermont	Alabama, Arkansas, Arizona, California, Nebraska, Nevada, Pennsylvania, South Carolina, Virginia
Potentially Above Average	N/A	Georgia, Kentucky, Oregon, Vermont, West Virginia
Average	Arizona, Idaho, Illinois, Kansas, Maryland, Massachusetts, New Hampshire, New Mexico, North Dakota, Ohio, Texas, Virginia, West Virginia, Wyoming	Alaska, Connecticut, Florida, Hawaii, Idaho, Kansas, Mississippi, Montana, New Hampshire, New Jersey, New Mexico, North Dakota, Rhode Island, Texas, Utah, Wyoming
Potentially Below Average	N/A	Delaware, Illinois, Indiana, Louisiana, Maine, Maryland, Michigan, New York, North Carolina, Oklahoma, South Dakota, Tennessee, Washington
Potentially Below or Below Average	Colorado, Connecticut, Hawaii, Indiana, Iowa, Louisiana, Maine, Minnesota, Missouri, Montana, North Carolina, Oklahoma, South Dakota, Tennessee, Washington, Wisconsin	Colorado, Iowa, Massachusetts, Minnesota, Missouri, Ohio, Wisconsin

3.1.3 Are the rankings stable over time and which agencies are showing a significant trend upward?

Thirty SVRAs were relatively stable and changed less than 10 points in rank between 2005 and 2009. Of the 20 high performing agencies, Florida, Michigan and Oregon improved rankings ten points or more and Kentucky, Nebraska, New Jersey and Vermont decreased in rank ten points of more. Ten SVRAs showed marked increases in ranking between 2005 and 2009 with seven not in the top 20.

3.1.4 Do the rankings change if including SSA data on earnings outcomes and beneficiary status?

Of those persons that an SVRA identified as having SSDI only, the TRF file indicated that 81% were SSDI only, 10% were also receiving SSI and another 9% were not beneficiaries. Of the 252,000 SSDI only beneficiaries in the TRF file, an SVRA had identified 74% as having SSDI only, 16% as having SSI only or both SSDI and SSI, and 10% as not having either.

For Models 5 and 6, SVRAs were ranked into five subcategories: above average, potentially above average, average, potentially below average, and below average. SVRAs that were identified as above average, average or below average had consistent findings for both Models (positive estimates that were significant at p < . 10). SVRAs that were inconsistent between the two Models were identified as potentially above or below average. Seven SVRAs (displayed in bold) were consistently ranked as above average performers and five were consistently identified as below average performers in all 6 models. The original twenty identified as high performers in the RSA 911 analyses distributed across the rankings in the TRF matched file, though none were identified as below average performers. None of the original 16 below average performers became identified as above or potentially above average on SSDI client earnings outcomes.

3.2 Task 2: High performing SVRA leadership perspectives

Sixteen of the twenty above average and two (Texas and Oklahoma) of the seven improving SVRA directors agreed to an interview. Table 3 provides a summary of the key findings of the interviews. Most SVRAs interviewees reported that there was no special focus on the population or a plan to achieve earnings above SGA as a strategic goal. This is consistent with the Task 1 finding that SVRAs with higher earnings outcomes for SSDI only clients tend to be high performers on earnings outcomes for all clients. Four SVRAs described their efforts to build business relations units and practices to find high paying jobs for all customers (Mississippi, Pennsylvania, Texas, Vermont). Alabama, New York, Mississippi, and Utah described written goals to improve earnings outcomes in state plans, strategic plans, or other leadership guidelines.

Table 3
High performing or improving SVRAs director explanations for earnings outcomes

Reason SVRAs

No special focus other than what SVRAs typically do AK, DE, GA, KY, RI, SC, TX, UT

Purposefully enhancing work incentives capacity or relationship with SSA. FL, MS, NE, NY, OK, OR, PA

Partnership with Departments of Mental Health DE, MI, SC

Partnership with Workforce agency DE

Enhanced relationships with businesses MS, PA, TX, VT

Team approach NE

Written strategic goals to achieve earnings above SGA AL, MS, NY, UT

Involvement with Demonstration Projects FL, NY

Counselor goals written into annual performance plans and data used to track achievement AL, OK

Reason	SVRAs
No special focus other than what SVRAs typically do	AK, DE, GA, KY, RI, SC, TX, UT
Purposefully enhancing work incentives capacity or relationship with SSA.	FL, MS, NE, NY, OK, OR, PA
Partnership with Departments of Mental Health	DE, MI, SC
Partnership with Workforce agency	DE
Enhanced relationships with businesses	MS, PA, TX, VT
Team approach	NE
Written strategic goals to achieve earnings above SGA	AL, MS, NY, UT
Involvement with Demonstration Projects	FL, NY
Counselor goals written into annual performance plans and data used to track achievement	AL, OK

Seven agencies described enhancing options to identify, track, and serve clients receiving benefits from the SSA. These agencies were also pursuing strategies to improve work incentive capacity including building internal resources or through partnering with vendors. Five SVRAs reported building infrastructure in response to Ticket to Work and Work Incentives legislation (Florida, Mississippi, New York, Oklahoma, Vermont). Nebraska VR described a unique team approach in combination with a strategic vendor partnership to provide early and rapid work incentives coordinated with job placement services. Alabama and Oklahoma included earnings outcomes in counselor performance evaluations. Oklahoma described an effort to provide incentive pay for counselors if the agency received increased SSA reimbursements.

Delaware, Michigan, and South Carolina SVRA directors suggested that partnerships with other public systems might be contributing to increased earnings outcomes but expressed that it was not the intent of the partnership. All three SVRAs discussed relationships with Departments of Mental Health to build capacity to deliver evidenced based employment services. The Delaware SVRA director offered that a partnership with their workforce system could be benefitting SSDI only customers but that it was not a focus per se. Florida and New York cited participation in SSA Model Demonstrations as a possible explanation for increased outcomes.

3.3 Task 3: Expert views on critical components

3.3.1 Round 1

All of the panel members scored the 72 nominated practices and gave considerable written responses to open-ended questions. Table 4 lists the practices that at least 50% or more of the members rated as important and whether that factor was thought to be under SVRA control. Most of the factors rated highly were also rated to be under the control of SVRAs. Family Outreach and Involvement, Wrap-Around Service Delivery, a Focus on Wellness Self-Management, Post-Secondary Specialized Supports, and a Focus on Symptom Self-Management were considered important but less under the control of an SVRA.

Table 4
Practices rated as important and under VR control (N of panel members)

Rank Practice Important Under SVRA control

Work-incentive planning and benefits counseling 25 21

2 Rapid job placement 25 20

3 Job accommodations 24 23

4 Rapid service delivery 24 23

5 Use of technology 24 23

6 Developing business relationships 24 22

7 Sustained engagement 24 22

8 Family outreach and involvement 24 13

9 Rapid service planning 23 22

10 Person-centered planning 23 21

11 Specialized training in serving subpopulations 23 21

12 Supported employment 23 21

13 Wrap-around service delivery 23 14

14 Expedited eligibility 22 22

15 Post-secondary training 22 20

16 Vocational services integration with treatment 22 17

17 On-the-job Training (OJT) for jobs 21 21

18 Transitional Employment (TE)/work experience 21 21

19 Asset development and financial planning 21 17

20 Focus on wellness self-management 21 12

21 Specialized counselors for certain disability subpopulations 20 22

22 Motivational interviewing 20 17

23 Employer outreach and education 19 21

24 Post-secondary specialized supports 18 14

25 Focus on symptom self-management 18 13

26 Small business/microenterprise development 17 21

27 Braided funding models 16 18

28 Targeted outreach 14 19

29 Readiness assessment 13 18

30 Links with one-stops 13 18

Rank	Practice	Important	Under SVRA control
	Work-incentive planning and benefits counseling	25	21
2	Rapid job placement	25	20
3	Job accommodations	24	23
4	Rapid service delivery	24	23
5	Use of technology	24	23
6	Developing business relationships	24	22
7	Sustained engagement	24	22
8	Family outreach and involvement	24	13
9	Rapid service planning	23	22
10	Person-centered planning	23	21
11	Specialized training in serving subpopulations	23	21
12	Supported employment	23	21
13	Wrap-around service delivery	23	14
14	Expedited eligibility	22	22
15	Post-secondary training	22	20
16	Vocational services integration with treatment	22	17
17	On-the-job Training (OJT) for jobs	21	21
18	Transitional Employment (TE)/work experience	21	21
19	Asset development and financial planning	21	17
20	Focus on wellness self-management	21	12
21	Specialized counselors for certain disability subpopulations	20	22
22	Motivational interviewing	20	17
23	Employer outreach and education	19	21
24	Post-secondary specialized supports	18	14
25	Focus on symptom self-management	18	13
26	Small business/microenterprise development	17	21
27	Braided funding models	16	18
28	Targeted outreach	14	19
29	Readiness assessment	13	18
30	Links with one-stops	13	18

Panel members were asked to contribute comments beyond scoring 72 practices. Nearly all offered extensive comments generally providing more nuanced reflections. Comments were synthesized into nine themes and presented during the Round 2 discussion.

Access to and match with good quality jobs (n = 20): Comments included use of labor market information, enhanced business relationships, identifying high demand sectors, improving the quality of vendor services, advancing skills of jobseekers to qualify for high paying jobs, and advancing counselor incentives to seek higher wages over job placement.

Benefits counseling and knowledge of work incentives (n = 19): Panel members raised concerns about capacity and the availability of work incentive counselors. Others indicated a need for accurate information across all systems that an SSDI client may encounter. Several suggested specific tools like benefits calculators that SVRAs might want to purchase to meet demand for information. A few panel members raised concerns that the focus should be broadened to include asset development and banking. Comments also indicated a need to integrate financial counseling with vocational counseling.

Customer motivation and relationship to VR staff (n = 13): The majority of comments were about developing a working alliance between the jobseeker and a VR counselor. A few suggested targeting services to SSDI clients who were highly motivated to achieve higher earnings.

VR organization, culture, and allocation of resources (n = 7): Some panel members anchored the solution as SVRA capacity improvement rather than changing counselor behavior or skills. Statements suggested that earnings outcomes would be improved for all clients by investing in critical organizational capacities such as business relationships, cultural change, and referral strategies.

VR client-centered direct services (n = 7): A range of comments fit within this theme to emphasize a more responsive and personalized strategy for interacting with clients including frequent communication, reducing reliance on purchased services, and tailoring employment services to coincide with financial goals.

Partners/service integration (n = 7): Many of the commenters mentioned a relationship with state mental health services as the largest percentage of SSDI clients are persons with mental health disabilities. Other partnerships suggested included with medical providers, Partnership Plus options, and long-term supports.

Time and pacing (n = 5): Comments emphasized the importance of progression through the VR process and stressed that waiting lists and “waiting for services” in general has a very negative influence on the motivation of clients.

Population served (n = 4): Several panel members suggested that SVRAs had more control over which populations are served through strategic outreach and engagement. Panel members raised a concern that SVRAs should improve their ability to work across racial, cultural, and disability subpopulations.

Use of assessments (n = 3): A few comments addressed the use of assessment services and indicated that VR agencies may be too reliant on external or long-term assessments that could discourage momentum.

3.3.2 Round 2

The SGA team convened the Delphi Panel to present a summary of scores and to facilitate discussion. The meeting was tape-recorded and transcribed. Panel members were in agreement that access to high paying jobs, work incentives provision and pacing were the most important factors. SVRA directors in attendance relayed that many SVRAs were developing business relations capacity, improving the ability to identify jobs that pay higher earnings through job matching, increase knowledge of local labor markets, and focus on advancing client education and skills. Panel members were unanimous about the importance of work incentives and financial education beyond a focus on disability benefits. However, many expressed concerns about state and local level capacity. Panel members indicated that SVRA responsiveness (i.e., rapid eligibility, planning, and placement) was an important factor and likely interconnected with SSDI client motivation to pursue earnings above SGA.

3.4 Task 4: Investigate nominated practices and whether those practices contribute to performance rankings and earnings outcomes

3.4.1 Identification of administrative data variables matching Delphi Panel rated practices

Most practices identified by the Delphi Panel did not have a corresponding variable in administrative data. A major gap was the ability to identify variables that linked to the provision or receipt of work incentives counseling. Multiple practices were connected to pacing of services (i.e., expedited eligibility, rapid service planning, rapid service delivery, rapid job placement) and could be defined by days between stages of the rehabilitation process. Other practices could be defined by closure type (i.e., supported employment, self-employment); types of services provided (i.e., post-secondary education service, occupational or vocational training, diagnosis and treatment funded through non-VR sources, and receipt of on-the-job training); and source of referral (e.g., self, One-Stop). Several practices identified as important were agency capacities and not available in administrative data. Examples include developing business relationships, family outreach and involvement, specialized training in serving subpopulations, and use of specialized counselors.

3.4.2 Analysis of available Delphi Panel practices data by SVRA performance ranking

Mathematica Policy Research analyzed the data and provided technical documentation summarized here. Differences between above average and below average agencies on the Delphi Panel factors are displayed in Table 6 (pacing variables), Table 7 (employment services and referral variables) and Table 8 (selected closure variable). The results should be interpreted as clues for whether the expert recommendations were trending similarly in administrative data.

Table 5
RSA 911 variable matches with Delphi practices

Delphi Panel rank Practice RSA 911 variable definition Category

2 Rapid job placement Median time to closure (days) Pacing

4 Rapid service delivery Median time to IPE (days) Pacing

9 Rapid service planning Median time to IPE (days) Pacing

12 Supported employment % closure to supported employment Employment service type

14 Expedited eligibility Median time to eligibility determination (days) Pacing

15 Post-secondary training % Any college service Employment service type

% Any occupational/vocational training

16 Vocational services integration with treatment % Diagnosis and treatment for physical and mental impairments funded through non-VR sources Employment service type

17 On-the-job Training (OJT) for jobs % Receive On-the-job Training Employment service type

26 Small business/microenterprise development % Closed to self-employment Employment service type

Expenditures in small business/microenterprise

28 Targeted outreach % Referral by self Referral

30 Links with one-stops (career centers) % Referral from One Stop Referral

% Any service from One Stop Employment service type

Delphi Panel rank	Practice	RSA 911 variable definition	Category
2	Rapid job placement	Median time to closure (days)	Pacing
4	Rapid service delivery	Median time to IPE (days)	Pacing
9	Rapid service planning	Median time to IPE (days)	Pacing
12	Supported employment	% closure to supported employment	Employment service type
14	Expedited eligibility	Median time to eligibility determination (days)	Pacing
15	Post-secondary training	% Any college service	Employment service type
		% Any occupational/vocational training
16	Vocational services integration with treatment	% Diagnosis and treatment for physical and mental impairments funded through non-VR sources	Employment service type
17	On-the-job Training (OJT) for jobs	% Receive On-the-job Training	Employment service type
26	Small business/microenterprise development	% Closed to self-employment	Employment service type
		Expenditures in small business/microenterprise
28	Targeted outreach	% Referral by self	Referral
30	Links with one-stops (career centers)	% Referral from One Stop	Referral
		% Any service from One Stop	Employment service type

Table 6

Median days to eligibility, plan, closure, and total days by matched SSA/RSA data SVRA performance ranking and closure status (with employment, without employment after services, and before plan)

	Closure status
		With employment				Without employment after services initiated				Before plan
SVRA ranking	N	Elig	Plan	Closure	Total	Elig	Plan	Closure	Total	Elig
Above average	9	17	28	375	420	22	29	722	773	27
Potentially above	4	3	37	322	362	9	55	621	685	7
Average	16	22	35	383	440	26	41	630	697	26
Potentially below	13	20	43	376	439	23	49	566	638	30
Below average	7	25	97	291	413	25	105	680	810	22
Difference between lowest/highest		–22	–69	–92	–78	–3	–76	–43	–37	–23
Difference between above and below average SVRAs		–8	–69	84	7	–3	–76	42	–37	5
Coincides with Delphi Panel		Y	Y	N	N	Y	Y	N	Y	N

Table 7

Differences in rate of above median scores by performance ranking on referral and service delivery Delphi factors

Performance ranking	N	Diagnosis and treatment funded non-VR sources	Self-referrals	Referrals from One-Stop	Any service from a One-Stop	Any On-the-job Training	Any college services	Any occupational or vocational training
Delphi Panel Factor rank		16	28	30	30	17	15	15
Above average	9	0.44	0.44	0.56	0.44	0.56	0.44	0.56
Potentially above average	5	0.40	0.20	0.60	0.80	0.60	0.60	0.60
Average	16	0.44	0.50	0.50	0.38	0.50	0.56	0.50
Potentially below average	13	0.46	0.62	0.54	0.46	0.31	0.46	0.31
Below average	7	0.86	0.57	0.29	0.57	0.71	0.43	0.71
Difference between above/below average		–0.41	–0.13	0.27	–0.13	–0.16	0.02	–0.16
Coincides with Delphi Panel		N	N	Y	N	N	N	N

Table 8

Differences in rates of above median scores for supported employment, self-employment, and total expenditures in small business/microenterprise by SVRA SSA/RSA performance ranking

Performance ranking	N	% Supported employment	Expenditures in small business enterprise or microenterprise	% Self-employment
Delphi Panel Factor rank		12	26	26
Above average	9	0.44	0.56	0.22
Potentially above average	5	0.20	0.20	0.80
Average	16	0.44	0.50	0.63
Potentially below average	13	0.69	0.46	0.31
Below average	7	0.57	0.71	0.71
Difference between above/below average		–0.13	–0.16	–0.49
Coincides with Delphi Panel		N	N	N

Table 6 provides the median days between selected rehabilitation process steps to see if SVRAs varied by their performance ranking for the matched RSA/SSA cohort. We looked at median days for those who had closed with an employment outcome, those who had closed without an employment outcome after services initiated and those who had closed prior to finalizing an individual plan for employment. Median days prior to eligibility were fewer for above average and potentially above average SVRAs than for the other SVRA groups. Four SVRAs identified as potentially above average had markedly fewer days than the other groups. One SVRA in the potentially above average group was excluded due to an issue with IPE dates.

Median days between eligibility and IPE were consistently fewer regardless of employment status at closure for above average SVRAs. Above average SVRAs had a median of 28 days to IPE as opposed to below average SVRAs who had a median of 97 days. This difference was slightly more for SSDI clients who had closed without an employment outcome. Administrative data supported the expert panels rating that rapid eligibility and rapid planning may be related to earnings outcomes. Median days between IPE and closure appeared to have an opposite trend with above average SVRAs having more days to closure than below average SVRAs. For those that closed with an employment outcome, days to closure is an imperfect measure of rapid job placement. Across all SVRA rankings, median days to closure for those who achieved employment was about a year or more shorter than for those that did not achieve employment.

Table 7 clusters factors related to referral and selected employment services that correspond to Delphi factors. Each SVRA received a score of 1 if above median or 0 if below median. A statistic above 0.50 indicates that SVRAs in the category tend to have higher than median scores. The variable that coincided the most with Delphi ratings was referrals from One-Stops (i.e., Career Centers). Below average SVRAs were less likely to have high scores on the percentage of referrals in comparison to the other SVRA categories. Diagnosis and treatment funded from non-VR sources was the closest variable for clinical and vocational integration of services available in administrative data. Below average SVRAs were more frequently above median than other SVRAs rather than as hypothesized by the Delphi Panel.

Table 8 displays above median scores for Delphi Factors related to supported employment and self-employment. Below average and potentially below average SVRAs were more frequently providing more supported employment closures than the other three performance groups. Delphi Panel members rated the use of supported employment very highly. Self-employment expenditures and percentage of people closed into self-employment was inconsistent across the performance rankings. Even in SVRAs that are more active in providing self-employment, a very small percentage of people receive funding or are closed to self-employment in general.

4 Conclusion

SVRAs vary in their earnings outcomes for SSDI only clients. Matching RSA 911 data with SSA data shifts performance rankings slightly. SVRAs with above average performance in achieving earnings outcomes for SSDI clients also tend to be above average performers in earnings outcomes for all clients. Directors of SVRAs seemed to confirm this finding as they expressed that key practices were available to all rather than as a selected service for SSDI clients. A cross-disciplinary panel of experts ranked a wide range of practices and were in agreement that a rapid pace in the rehabilitation process, identifying job opportunities with higher wages and advancement opportunities, and work incentive counseling were critical components. Administrative data that confirms or rejects expert opinion on the rated factors is available but limited.

4.1 Pacing

Expert opinion was supported by administrative data that a faster pace between eligibility and plan development is related to earnings outcomes. SSDI only clients who achieve employment tend to spend about a year receiving services regardless of SVRA performance ranking. Those who are served by high performing SVRAs appear to spend less time in eligibility and planning phases than those served by below average agencies. Efficient service delivery in early phases matters.

4.2 Work incentives counseling

All Delphi Panel members and all directors interviewed indicated that work incentives counseling was one of the most critical services for SSDI only customers looking to improve earnings outcomes. However, at the time, there was no administrative data source for comparing across SVRA performance rankings. Concern about state level capacity was a major topic in Delphi panel discussions. SVRA directors interviewed during Task 2 provided examples of how they were mitigating capacity issues and considering strategic initiatives to set goals, design programs, and improve reimbursement efforts.

4.3 Employment services

Delphi Panel members rated a wide range of employment services as important to improving earnings outcomes. SVRA Directors from Task 2 described new and ongoing efforts to build business relations units, use labor market information, and identify high paying jobs. A disconnect between Delphi Panel members ratings and administrative data was evident. Employment services like supported employment and on-the-job training may have more connection to job placement than to achievement of higher earnings outcomes. Or services like self-employment are supporting a subset of SSDI only clients and do not translate to higher earnings in the time window of the administrative data. Similar to work incentives counseling, there was limited opportunity to match business relations strategies to earnings outcomes in administrative data. Business relations capacities appear to reflect organizational strategies rather than direct services to clients.

5 Discussion

Advancement in SVRA earnings outcomes was achieved in states that had very high unemployment rates during the Great Recession. This suggested that SVRA practices have an influence on earnings even in challenging times. We did not find an SVRA that had designed a package of services unique to SSDI only clients that could be transported and studied in other SVRAs. We sought to narrow in on practices that could combine into a testable intervention and implemented fully by an SVRA. What emerged was consensus to focus further investigation on pacing, work incentive counseling capacity, and understand the complexities of innovations in direct employment services and business relations.

Findings led to multiple methodological questions. If the intervention focuses on early planning periods, then the ability to identify SSDI only clients at application was an important capacity. Task 1 findings indicated that accuracy of early identification could be improved. SVRA Directors described strategic and organizational capacities as explanations for earnings outcomes. Are these capacities that should be in place prior to testing an intervention or are these capacities part of the intervention? SVRAs directors described efforts to bring work incentives counseling in-house either as an SSA grantee, hire work incentives counselors, or to purchase services. Thus, it appeared that some SVRAs defined work incentives counseling as a VR service and within its control.

Combining administrative data analysis with expert opinion presents many opportunities and challenges. Practices the Delphi panel recommended could not be fully defined in administrative data sets. Some variables were proxies for selected factors but may not capture the full meaning of a recommended practice (i.e., client motivation). Pacing was more easily identified in administrative data and the analysis seemed to support expert opinion that rapid service planning was important and under the control of an SVRA. Administrative data are limited in the ability to assess emerging practices or the influence of innovative organizational capacities on outcomes.

Phase I occurred prior to the Workforce Innovation and Opportunity Act of 2014. Pacing, work incentives counseling, and business relations capacities were included in the legislation and subsequent regulations. Findings from the SGA Project Phase I suggest these advances are likely to advance earnings outcomes and that these strategies continue to evolve in both good and challenging economic conditions.

Conflict of interest

None to report.

Footnotes

Acknowledgments

The Institute for Community Inclusion at the University of Massachusetts-Boston and Mathematica Policy Research led the Substantial Gainful Activity Project Demonstration through a grant received from the U.S. Department of Education, Rehabilitation Services Administration. The views and opinions expressed here are those of the authors and do not necessarily reflect the views, opinions, or policies of the Rehabilitation Services Administration. The authors are solely responsible for any errors.

References

Drake,

, Frey,

, Karakus,

, Salkever,

, Bond,

& Goldman,

H. H.

(2016). Policy implications of the Mental Health Treatment Study. Psychiatric Services67(10), 1139–1141. https://doi.org/10.1176.appi.ps.201500336.

Fleming,

, Boeltzig-Brown,

& Foley,

S. M.

(2015). Using the Delphi method for selecting effective rehabilitation practices for case study research: Methods, challenges, and solutions and implications for future research. Rehabilitation Research, Policy, and Education29(4), 391–405. 10.1891/2168-6653.29.4.391.

Gubits,

, Geyer,

, Stapleton,

, Greenberg,

, Bell,

, Nichols,

, Wood,

, McGuirk,

, Hoffman,

, Carroll,

, Croake,

, Kattel,

, Mann,

D. R.

& Judkins,

(2018, October 18), Social Security Administration.https://www.ssa.gov/disabilityresearch/documents/BOND%20Deliv%2024e2%20FER%20Vol%201%2020181018.pdf.

Honeycutt,

, &Livermore, G. (2011, February 28). Employment Outcomes of SSDI-only beneficiaries who sought state vocational rehabiltation agency services. Memorandum. Mathematica Policy Research.

Honeycutt,

, & Livermore,

(2011, October 12). Project SGA - Final estimates of agency-level employment outcomes for SSDI-only beneficiaries who sought state vocational rehabilitation agency services, Memorandum. Mathematica Policy Research.

Honeycutt,

, Stapleton,

(2013). Striking while the iron is hot: The effect of vocational rehabilitation service wait times on employment outcomes for applicants receiving Social Security Disability benefits. Journal of Vocational Rehabilitation39(2), 137–152. 10.3233/JVR-130645.

Kearney,

(2005). Social Security and the “D” in OASDI: The history of a federal program insuring earners against disability. Social Security Bulletin66(3). https://www.ssa.gov/policy/docs/ssb/v66n3/v66n3p1.html.

Michaelopoulus,

, Wittenberg,

, Israel,

, Schore,

, Warren,

, Zutshi,

, Freedman,

S. R.

& Schwartz,

(2011, February), The Accelerated Benefits Demonstration and Evaluation: Impacts on health and employment at twelve months. Volume 1. https://www.mdrc.org/publication/accelerated-benefits-demonstration-and-evaluation-project.

Social Security Administration. (2019, September), Annual Report on Section 234 Demonstration Projects.. https://www.ssa.gov/disabilityresearch/demos.htm.