Abstract
BACKGROUND:
Broadened Horizons, Brighter Futures (BHBF), a transition project in Miami-Dade County, Florida, was the focus of an evaluation of services and work incentives for youth who were receiving disability payments. Notable features of BHBF included paid work experiences for youth and the application of external technical assistance in implementing the program model.
OBJECTIVE:
The evaluation documented the design and implementation of BHBF and assessed its impacts on outcomes for youth three years after they enrolled in the study.
METHODS:
We randomly assigned 859 youth who were receiving disability payments to either a treatment group that was eligible to receive both BHBF services and waivers of certain disability program rules, or to a control group that was not eligible for either the services or the waivers.
RESULTS:
Three years after they enrolled in the study, treatment group youth were more likely than youth in the control group to be employed, their earnings were 50 percent higher, and they were less likely to have had contact with the justice system.
CONCLUSION:
Interventions that include work experiences as a service component can improve outcomes for youth with disabilities. Empirical monitoring of project staff, combined with technical assistance, may be necessary to ensure a consistent focus on helping youth to find jobs.
Keywords
Introduction
As they prepare for and go through the transition from school to employment and adult life, youth with disabilities face persistent, well-documented challenges. Two of these challenges stand out. First, they are less likely to finish school than their non-disabled peers are (Wagner, Newman, Cameto, Garza, & Levine, 2005; Chapman, Laird, Ifill, & KewalRamani, 2011), making them more likely to experience long-term unemployment or underemployment. Second, whether they have completed school or not, their overall employment rate has been persistently low, especially for those with significant disabilities. The National Longitudinal Transition Study-2, for example, revealed that youth with intellectual disabilities, as a group, had a post-school employment rate of just over 20 percent (Wagner et al., 2005). Even this low number is misleading, because in the analysis underlying this finding, the researchers did not distinguish direct-hire integrated community employment from congregate sub-minimum wage employment. For these youth, and for youth with other kinds of significant disabilities, especially those receiving public income support such as Supplemental Security Income (SSI), the likelihood of their experiencing lives of poverty and dependence on public assistance programs is very high (Davies, Rupp, & Wittenberg, 2009; O’Day & Stapleton, 2009; Duggan & Kearney, 2007).
These dismal circumstances exist in a time when evolving interventions in special education, transition, and employment are generating high expectations of better employment outcomes for youth with disabilities. In particular, researchers have identified two conditions that make post-school employment more likely for youth with significant disabilities: the opportunity to work during the secondary school years and the expectations their families have for their employability (Carter, Austin, & Trainor, 2012; Test, Mazzotti, Mustian, Fowler, Kortering, & Kohler, 2009). These findings have bolstered advocacy initiatives to promote the notion that, regardless of the nature of their disabilities, these youth can find and keep jobs if they have the opportunity and the necessary supports for employment and career development (Martinez, 2013). In fact, the presumption of employability is an explicit underpinning of the Workforce Innovation and Opportunity Act of 2014, in which eligibility for vocational rehabilitation services is presumed and prioritized for individuals with the most severe disabilities (U.S. Congress, 2014).
The policy conundrum here is that, on the one hand, there is a growing recognition that people with disabilities should receive employment services that are guided by their presumed employability. On the other hand, the number of SSI recipients, who (ironically) have had to demonstrate their likelihood of being unemployable, has been rapidly and consistently rising. Between December 2005 and December 2015, the number of people receiving federally administered SSI payments rose by 16.8 percent, to 8.3 million (Social Security Administration [SSA], 2017). The consequences for people with significant disabilities are obvious, as long-term dependency on public income support generally goes hand in hand with poverty. The consequences for the public are also clear, as SSI payments in 2015 totaled $55 billion (SSA, 2017).
The federal policy challenge, then, is to identify new policies and intervention strategies that would provide a financial safety net for those whose incomes are adversely affected by the circumstances of disability, while simultaneously discouraging long-term dependence on public income supports. To investigate the size and scope of this challenge, SSA funded Youth Transition Demonstration (YTD) projects starting in 2003 and ending in 2012 (Fraker & Rangarajan, 2009). These projects and the associated evaluation were intended to identify and test strategies that had the potential to increase employment and income for youth who were receiving disability payments under Title XVI (SSI) or Title II (Social Security Disability Insurance or Childhood Disability Benefits) of the Social Security Act. Although the evaluation’s three-year follow-up period was too brief to reasonably expect a manifestation of any reductions in disability payments because of the YTD projects, it was SSA’s hope that such reductions would be realized in the long run.
In this article, we present an analysis of one of the YTD projects, Broadened Horizons, Brighter Futures (BHBF) in Miami-Dade County, Florida. The Florida regional office of ServiceSource, a private nonprofit organization, had overall responsibility for BHBF and delivered most of the project services. ServiceSource had formal relationships with several partners who helped implement BHBF. These included the Human Services Coalition, a Miami-based nonprofit human services organization that gave project participants training on financial literacy, and two other nonprofit organizations that helped participants establish individual development accounts. In addition, ServiceSource had informal relationships with Miami-Dade County Public Schools and the Florida Division of Vocational Rehabilitation, and both gave some participants transition planning and employment services. The BHBF staff at ServiceSource consisted of three individuals who performed managerial and administrative functions and up to 10 frontline staff in two offices, who were employment specialists and benefits specialists.
Our intent in examining BHBF was to determine the impact of a research-supported intervention on a population of youth who are known to have a hard time making the transition to adult employment. This examination also had a focus on how intervention fidelity was impacted by modifications in and improvements to intervention delivery that were supported by expert technical assistance (TA). The responsibility for providing TA to BHBF and the other YTD projects was handled by TransCen, Inc., a nonprofit company that specialized in the design and implementation of employment-focused transition services for youth with disabilities.
In this paper, we address these research questions: What services did BHBF
provide? What challenges did BHBF
confront in implementing those services? How did TA help BHBF work to overcome those challenges? What impacts did BHBF have on outcomes for youth three years after
they entered the study?
Method
The YTD program model at BHBF
The development of the YTD program model drew on the framework presented in Guideposts for Success, developed by the National Collaborative on Workforce and Disability for Youth (2005 & 2009). Guideposts is based on a comprehensive review of research, demonstration projects, and recognized effective practices. It is intended to help practitioners and policymakers conceptualize optimum service delivery for youth with disabilities.
The YTD program model had six components. Most of them were identified in Guideposts, although they were customized to meet the needs of the YTD target population (Luecking & Wittenburg, 2009). Foremost among them were individualized work-based experiences, including worksite tours, subsidized jobs, and, most notably, competitive paid employment in integrated settings. A youth empowerment component enabled youth to acquire the skills and knowledge they needed to chart their own courses and advocate for themselves. YTD fostered empowerment by engaging youth in intensive planning that focused on education, employment, health care, and independent living. Family supports included family-focused training, support for parent networking, and provision of transition-related information. YTD also facilitated system linkages, characterized by connections with service providers that youth may need to access health care, education programs, transportation, and accommodations and assistive technologies for education and employment. SSA’s waivers for YTD made paid employment more rewarding and provided for the temporary continuation of disability payments despite a negative continuing disability review or age-18 medical redetermination. Benefits counseling, which helped youth and parents understand the waivers and the standard SSA program rules, was also central to the YTD model. BHBF implemented all six components.
Two other noteworthy features of YTD were (1) intensive programmatic TA provided by an independent organization and (2) adoption of the Efforts-to-Outcomes management information system. TransCen delivered hands-on assistance and training that helped the staff of the YTD projects maintain the interventions’ fidelity to the program model. Within TransCen, a single individual was the primary point of contact with BHBF. Project staff used Efforts-to-Outcomes to record their delivery of services to individual youth and to record any key outcomes those youth achieved, such as completing an assessment or getting a job.
Evaluation enrollees
SSA contracted with Mathematica Policy Research to evaluate the YTD projects, including BHBF. At six-month intervals, SSA gave Mathematica lists of youth disability recipients in Miami-Dade County. On a rolling basis, Mathematica identified youth on the lists who were in BHBF’s target age range of 16 to 22 and randomly selected 5,573 of them for recruitment into the study (Table 1). The recruitment process began in March 2008 and ran through September 2010, when the target number of youth (880) had completed a baseline interview and provided written affirmative consent (parents or guardians provided consent for youth who were minors) to enroll in the evaluation.
Disposition of the BHBF research sample
Disposition of the BHBF research sample
Source: Mathematica’s YTD survey management system and BHBF’s management information system.
Mathematica randomly assigned 460 of the 880 consenting youth to a treatment group whose members were eligible to enroll in BHBF, and randomly assigned 399 to a control group (Table 1). The remaining 21 youth had siblings who had enrolled in the evaluation already; Mathematica purposefully assigned them to the same groups as their siblings and excluded them from the research sample. Eighty-four percent (388) of the 460 randomly assigned treatment group members met with BHBF representatives, signed project enrollment forms, and thus became BHBF participants.
An examination of the baseline characteristics available in SSA administrative records revealed a number of differences between the 880 youth who enrolled in the evaluation of BHBF and the 4,693 youth who were recruited by Mathematica but did not enroll (Table 2). Relative to non-enrollees, youth who enrolled in the evaluation were 3 percentage points less likely to be male; two-tenths of a year younger, on average; 4 percentage points more likely to have a parent or other relative as a representative payee for disability payments; and 6 percentage points more likely to have a cognitive or developmental disability as the primary disabling condition. All of these differences are statistically significant (p < 0.10).
Baseline characteristics of evaluation enrollees and non-enrollees
Source: SSA and Internal Revenue Service administrative records. Note: *p < 0.10, **p < 0.05, ***p < 0.01. The universe for this table is all youth who were residing in Miami-Dade County and were randomly selected by Mathematica from SSA lists of disability recipients for recruitment into the evaluation of BHBF. Among the enrollees are 21 youth who were excluded from the research sample because they had been purposefully assigned to the treatment or control groups to match the status of their siblings in the research sample. SSI = Supplemental Security Income. SSDI = Social Security Disability Insurance. CDB = Childhood Disability Benefits.
To ensure the intervention’s fidelity to the program model, designers incorporated into YTD the key feature of a continuum of TA and training intended to build the projects’ capacity to effectively implement the program model. The TA was intense, involved a range of delivery modes, and was informed by data from the projects’ management information systems and by emerging findings from the YTD evaluation. Each YTD project was assigned a TA liaison from TransCen. The liaisons were trained and experienced in delivering training and assistance in the implementation of evidence-based practices in secondary transitions. The TA liaison to BHBF had a doctorate in vocational special needs education and was a senior research associate at TransCen. The liaisons gave the YTD projects general support and helped them resolve specific implementationissues.
The design for BHBF’s TA had three basic approaches. First, the TA liaison worked closely with senior managers of the project to develop a service plan consistent with the YTD program model. The liaison visited the project several times during its first months of operation to ensure the plan was being implemented effectively. Second, the TA liaison delivered in-person TA and training to BHBF frontline staff. Early on, this consisted of group trainings on topics such as strategies for engaging families, coordinating with external service providers, and networking with employers. As the project matured, the in-person TA shifted to one-on-one troubleshooting by the liaison with individual frontline staff. For example, the TA liaison accompanied BHBF employment specialists while they were developing jobs with employers and meeting with youth to identify their employment goals. Third, the TA liaison provided just-in-time advice and guidance to BHBF frontline staff to help them quickly resolve issues that typically concerned individual project participants or specific employers.
The quality and intensity of the TA provided to the YTD projects would be difficult to replicate quickly on a broad scale, because there are few professionals with the training and experience to deliver this kind of TA. For this reason, SSA considered YTD a “proof of concept” demonstration. As such, its objectives were to (1) establish whether transition projects could be implemented with both a high degree of fidelity to best practices in serving youth with disabilities and a strong and consistent focus on services designed to promote their employment prospects; and (2) rigorously assess whether such projects could improve employment and other outcomes for those youth in the short run (one year) and medium run (three years). SSA did not expect that positive findings from the YTD evaluation would lead quickly to the wide-scale rollout of YTD-like interventions for youth with disabilities. Instead, the agency anticipated that positive findings would lead to follow-on demonstrations and research on whether effective intervention projects could be implemented on a broader scale, possibly with more limited TA resources.
Design for the implementation analysis
The implementation analysis component of the BHBF evaluation was intended to both measure specific facets of project implementation and gauge the need for and application of advice, assistance, and training to address fidelity to the program model. The analysis examined whether the BHBF intervention included all of the core components of the program model, particularly focusing on the work-basedexperiences. The study’s various sources of qualitative and quantitative data ensured that the implementation analysis would reflect multiple perspectives on these key issues.
The qualitative data for the implementation analysis consisted primarily of information collected on site visits to Miami-Dade County and in telephone calls with BHBF management staff. The evaluation team conducted four site visits: one to support an early assessment of the implementation of BHBF, one to underscore the need for the project’s management and frontline staff to emphasize employment services and outcomes, and two to systematically gather data for assessing project operations. The first two visits included reviews of case files for selected project participants and an assessment of how completely and accurately the services delivered to participating youth were recorded. The last two visits included interviews with BHBF staff at ServiceSource and representatives of partner organizations, observations of the delivery of project services, and focus group discussions with participating youth and their parents. The evaluation team also conducted biweekly conference calls with project management and reviewed project documents, such as monthly management reports.
The quantitative data for the implementation analysis came from BHBF’s management information system. We used those data to answer questions about the efforts of BHBF staff to engage youth in project services and the types and amounts of BHBF services delivered to project participants. Our analysis of service delivery focused on the 388 treatment group youth who had completed the paperwork necessary for them to be classified as BHBF participants; the analysis was based on data for the first 15 months after those youth enrolled in the evaluation. Four service categories are used in the analysis: (1) employment services; (2) benefits planning; (3) education services; and (4) case management, encompassing person-centered planning as well as general case management.
Design for the impact analysis
The impact analysis component of the BHBF evaluation was based on a randomized controlled trial design. Mathematica used a computer algorithm to randomly assign the youth who had agreed to be in the study to either a treatment or control group (Fraker & Rangarajan, 2009). The treatment group members were eligible for both BHBF services and SSA’s waivers for YTD. The control group members could access whatever non-BHBF services happened to be available in Miami-Dade County and were subject to SSA’s standard program rules; they were not eligible for the waivers. Because of random assignment, the two groups were expected to be equivalent, on average, at the beginning of the study. Consequently, any differences in their aggregate outcomes could be attributed to BHBF. The evaluation tracked employment, earnings, disability payments, and other outcomes for the youth for three years. It compared the aggregate outcomes of the treatment and control groups to assess whether BHBF helped youth secure paid employment, increase their incomes, and achieve other transition objectives.
Surveys conducted as part of the BHBF evaluation, along with SSA administrative records, were the data sources for the impact analysis. As part of the enrollment process, Mathematica conducted a baseline survey of youth, collecting data on demographic characteristics, personal and family background, work experience, and attitudes and expectations. Administrative files of the Internal Revenue Service and SSA provided additional baseline data on employer-reported earnings in the calendar year preceding enrollment and on monthly disability payment amounts. Mathematica also surveyed youth at one year and three years after enrollment, achieving response rates of 86.8 percent and 81.5 percent, respectively (Table 1). The follow-up surveys gathered information on the receipt of services, education, employment, earnings, and contact with the justice system. For all enrolled youth, not just those who responded to the follow-up surveys, SSA administrative files were the source of post-enrollment data on monthly disability paymentamounts.
Random assignment worked as expected—youth in the treatment and control groups had statistically equivalent baseline characteristics. We tested for treatment-control differences in 50 baseline characteristics, using chi-square tests for categorical measures and t-tests for continuous measures. In Table 3, we provide the test results for a subset of the baseline characteristics. For 44 of the characteristics, the tests revealed no statistically significant differences between the treatment and control groups. For example, the table shows that treatment group youth were 3.4 percentage points less likely to be male than control group youth were, and one-tenth of a year older, on average; however, neither of these differences is statistically significant at the 0.10 level. For six of the baseline characteristics (job training in the last year; volunteer work in the last year; mother is a high school graduate; father is employed; use of reading, hearing, speaking, or walking aids; and the disability payment amount in the previous 12 months), the tests revealed statistically significant differences (p < 0.10). This is about the number of significant differences we would expect on the basis of random chance in the absence of systematic differences between the two groups.
Baseline characteristics of treatment and control group youth in the research
sample
Baseline characteristics of treatment and control group youth in the research sample
Source: SSA and Internal Revenue Service administrative records and the baseline survey for the BHBF evaluation. Note: *p < 0.10, **p < 0.05, ***p < 0.01. For some characteristics measured through the baseline survey, item nonresponse resulted in effective sample sizes that are smaller than those shown in the last row of the table. The sample includes all youth who enrolled in the evaluation and were randomly assigned to either the treatment group or the control group. The sample does not include 21 youth who enrolled in the evaluation and were purposefully assigned to the treatment or control groups to match the status of their siblings. SSI = Supplemental Security Income. SSDI = Social Security Disability Insurance. CDB = Childhood Disability Benefits. AK = Alaska; HI = Hawaii. aStatistics for these characteristics are based on data from the baseline survey for the BHBF evaluation.
We used multivariate statistical models to estimate the impacts of BHBF at one year and three years after youth enrolled in the evaluation. Within the models, we used data from the baseline survey and administrative files to control for differences between the treatment and control groups that arose by chance during random assignment, and to improve the precision of the impact estimates.
Implementation challenges and a mid-course correction
TransCen’s TA liaison to BHBF visited the project several times during its first year of operation, and those visits, along with findings from tabulating data on services delivered during that year, revealed that the project’s frontline staff were devoting much of their effort to general case management instead of working with employers to develop jobs and helping participants prepare for and obtain work experiences. Many of the participants were not being placed in jobs, nor were they receiving services to quickly move them toward that goal.
Based on this information, the TA liaison concluded that the project’s frontline staff needed to refocus their effort on giving participants employment-related services and placing them in jobs. The TA liaison adjusted the approach to delivering TA to make it more employment-centered, field-based (as opposed to being delivered primarily in workshops and webinars), and delivered in real time directly to frontline staff. The BHBF management team supported these adjustments, thus expediting their implementation and enhancing their effectiveness. In some instances, the TA liaison accompanied project staff in making initial contacts with employers, meeting with participants to discover their skills and interests, and negotiating work experiences for participants with employers. This allowed the liaison to model best practices and observe staff interactions with employers and youth. The TA liaison gave immediate feedback to the staff so they could hone their techniques. Also, the liaison led biweekly case reviews focused on individual participants who were especially challenging to serve. Those reviews allowed the TA liaison to guide the staff in thinking about employment options for those youth and in exploring ways to address the challenges involved in serving them. In short, the TA liaison coached the BHBF staff to correctly and effectively deliver the key component of the program model, work experience.
While TA was being adjusted in the spring and early summer of 2009 (12 to 15 months after the start of project services), the management of BHBF established numeric goals for staff contacts with employers and placements of participants in jobs. These goals were specified both for the project as a whole and for individual frontline staff. Equally important was the systematic quantitative monitoring of progress toward the goals. The Mathematica evaluation team provided BHBF management and the TA liaison with monthly monitoring reports, based on data from the project’s management information system. These monthly reports, whose results were broken down by individual staff member, showed: (1) the employment status of each assigned BHBF participant, (2) the number of initial and follow-up employer contacts, and (3) the number of staff hours by service activity. These reports provided snapshots of how the project was performing relative to its established goals and documented the efforts and participant outcomes for individual frontline staff. The TA liaison and project managers reviewed these reports at a monthly meeting. The TA liaison could use their hard data to adjust training and support in response to emerging issues, and they helped guide project staff in focusing their efforts on job placements.
When the monthly monitoring reports revealed low numbers of employer contacts and job placements for certain staff members, the BHBF management team and the TA liaison worked with those staff to help them improve their delivery of the intervention. In some cases, project management had to replace staff with new employees who were more at ease networking with employers and helping youth find jobs.
Delivery of BHBF services
The BHBF program model emphasized rapidly engaging the treatment group youth who had agreed to participate in the project. Findings from the evaluation’s implementation analysis show that the project achieved this objective. Our analysis of data from the project’s management information system revealed that the median elapsed time between the onset of BHBF participation and the first contact by project staff for delivery of services was four days. Within 30 days, 98 percent of BHBF participants had received a first service contact, and 90 percent had received a second service contact. The services received most often at the first contact were benefits planning (58 percent) and case management (37 percent). In contrast, the services received most often during the last contact were employment services (48 percent) and benefits planning (38 percent).
The intensity of BHBF’s services (of any type) to participating youth was high, whether measured by the number of service contacts or their cumulative duration. These findings are based on data for each service contact, as recorded by project staff. On average, project staff made 48.5 service contacts per participant, lasting a total of 28.5 hours (Table 4). Some of those contacts were with employers and other individuals or organizations on behalf of specific participants. The average cumulative duration of service contacts that directly involved participants was 18 hours (results not shown in the table). The average duration of a single service contact was 26.8 minutes, whereas the median duration was 15.0 minutes, indicating that some contacts were lengthy but many were brief, such as a telephone call to remind a participant of a job interview. Only 17.6 percent of service contacts exceeded 30 minutes in length.
Delivery of BHBF services to project participants
Delivery of BHBF services to project participants
Source: BHBF’s management information system. Note: The sample for this analysis is BHBF participants—youth who were randomly assigned to the evaluation’s treatment group and who formally enrolled in BHBF. We excluded from the analysis contacts of less than two minutes and those made on the day of enrollment. We calculated statistics on the number of services per participant and service time per participant based on those participants who actually received the services in question.
Consistent with the YTD and the BHBF program models, employment services (such as career exploration, job development, job search assistance, and job coaching) were delivered with more intensity than any other type of service. Virtually all BHBF participants received employment services (Table 4) and, on average, BHBF staff made 20 contacts per participant to deliver those services, for a cumulative duration of 13.9 hours. The median cumulative duration of the contacts was about half the average duration, indicating that a few participants received an especially large number of employment service hours. Benefits planning services were also relatively intense in BHBF. Again, virtually all participants received benefits planning services and, on average, project staff made 15.3 contacts per participant to deliver them, for a cumulative duration of 7.9 hours. In contrast, the intensity of case management services and education services was relatively low. Among the 95.9 percent of participants to whom project staff provided case management, the average number of service contacts was 9.4, and the average cumulative duration of those contacts was 3.5 hours. The corresponding statistics for the 83.5 percent of participants to whom BHBF staff delivered education services are 4.0 service contacts for a cumulative duration of 2.1 hours.
During the year after they enrolled in the evaluation, treatment group youth were more likely than their counterparts in the control group to receive services of any type from any source, including but not limited to BHBF. They were also more likely to receive services specifically designed to promote employment, such as career counseling, help preparing resumes, placement in apprenticeships, job search assistance, and counseling on SSA benefits and work incentives. Table 5 shows results from the evaluation’s one-year follow-up survey; 80.5 percent of treatment group youth reported they had received any services, and 58.2 percent reported they had received employment services. These rates reflect statistically significant (p < 0.01) BHBF impacts of 9.9 and 12.5 percentage points, respectively, relative to the rates for control group youth.
BHBF impacts on the receipt of services from any source in the first post-enrollment
year
BHBF impacts on the receipt of services from any source in the first post-enrollment year
Source: The measures of service receipt are from the YTD evaluation’s one-year follow-up survey, sample size 738. Sample weights were used to adjust for survey nonresponse by 112 youth. Item nonresponse reduced the effective sample sizes for some of the measures. Note: *p < 0.10, **p < 0.05, ***p < 0.01. Impact estimates were obtained by using multivariate statistical models to control for baseline characteristics. The units of measurement (%) identified in the first column apply to the results presented in the second and third columns. The results presented in the fourth column (effect size) are units-free.
Note that the data on service receipt by evaluation enrollees (underlying Table 5) are fundamentally different from the data on service delivery by BHBF staff (underlying Table 4). The Table 4 data pertain only to the 388 treatment group youth who participated in BHBF. BHBF staff entered those data into the project’s management information system as they delivered services to those youth. The Table 5 data are based on the 738 treatment and control group youth who responded to the evaluation’s one-year follow-up survey (as documented in Table 1). The youth reported those data, and they included services they received from all sources, not just BHBF.
Because work-based experiences are a key component of the YTD program model, a critical short-term indicator of BHBF services’ strength is whether treatment group youth were more likely than control group youth to have been employed during the recall period for the one-year follow-up survey, which largely overlapped with the time when they were receiving project services. Table 5 shows that treatment group youth were 9.4 percentage points more likely than control group youth (p < 0.01) to have worked in paid or unpaid jobs at some time during the year after they enrolled in the evaluation. BHBF had an identical impact on employment for pay. Among the treatment group youth who worked for pay, about half had hourly earnings of between $7 and $9 in 2008 dollars, slightly more than one-quarter earned less than $7 per hour, and slightly fewer than one-quarter earned more than $9 per hour (results not shown in the table).
BHBF had statistically significant beneficial impacts on a number of outcomes for youth during the third year after they enrolled in the evaluation, which was after BHBF services ended. Most notably, the rate of paid employment at any time during the year was 7.8 percentage points higher (p < 0.05) for treatment group youth than for control group youth (Table 6). This was accompanied by an impact of $615 on mean earnings over the course of the year (p < 0.05). These findings are based on self-reports by the 685 treatment and control group members of the research sample who responded to the evaluation’s three-year follow-up survey (Table 1).
BHBF impacts on outcomes in the third post-enrollment year
BHBF impacts on outcomes in the third post-enrollment year
Source: The outcome measures are from SSA administrative records (disability payment amount), sample size 840; and from the YTD evaluation’s three-year follow-up survey (employment; earnings; education or training; productive activities; arrested or charged), sample size 685. Sample weights were used to adjust for survey nonresponse by 155 youth. Item nonresponse reduced the effective sample sizes for some of the survey-based measures. Note: *p < 0.10, **p < 0.05, ***p < 0.01. Impact estimates were obtained by using multivariate statistical models to control for baseline characteristics. Dollar-denominated outcomes are measured in 2008 dollars. Participation in productive activities is defined as paid or unpaid employment, participation in an education program, or participation in a training program. The units of measurement (% or $) identified in the first column apply to the results presented in the second and third columns. The results presented in the fourth column (effect size) are units-free.
Even though treatment group youth had higher earnings than their control group counterparts, their disability payments were also higher; BHBF had a statistically significant (p < 0.01) impact of $598 on the mean yearly disability payment amount. This difference likely exists because of SSA’s waivers for YTD, which allowed some BHBF participants to continue receiving SSI payments that otherwise would have been discontinued, and to keep more of their payments than would otherwise have been possible given their earnings. Because BHBF increased both earnings and disability payments, it is not surprising that the project had a statistically significant (p < 0.01) impact of $1,266 on the mean income of youth during the third year after they enrolled in the evaluation, with income defined as the sum of earnings and disability payments.
BHBF had no impact on youth’s participation in education or training programs during the third year following their enrollment in the evaluation. However, it did increase participation in a broader set of productive activities that included paid and unpaid employment in addition to education and training. Treatment group youth were 8.4 percentage points more likely than control group youth to have participated in one or more of these productive activities during the third post-enrollment year (p < 0.05).
The project had a beneficial impact on contact by youth with the justice system. Less than 1 percent of treatment group youth reported in the evaluation’s three-year follow-up survey that they had been arrested or charged with delinquency or a criminal complaint during the previous year. This is a statistically significant (p < 0.05) 2.7 percentage points lower than the reported arrest/charge rate for control group youth.
Limitations and implications for future research
As we discussed in Section 2.3, BHBF was implemented with ample amounts of expert TA for frontline staff, and the experts who provided the TA concentrated on helping those staff deliver employment-focused transition services with a high degree of fidelity to the YTD program model. The beneficial impacts of BHBF as revealed by our evaluation, although modest in size, document what a program like this can accomplish under favorable circumstances—that is, when high-quality TA is readily available. These findings suggest a need for more investment in developing TA resources and in implementing and evaluating demonstrations that may be less intensive in their use of those resources than the YTD projects were. Indeed, motivated in part by early findings from the YTD evaluation, SSA undertook a nine-year evaluation of PROMISE demonstration projects in 2013. With funding from the U.S. Department of Education (2013a, b), these projects are providing education-and-employment-focused transition services to 6,000 SSI youth ages 14 through 16 in 11 states. These projects are, in general, less intensive in their use of expert TA resources than the YTD projects were.
Although we estimated that BHBF had beneficial impacts on a number of important outcome measures during the third year after youth enrolled in the evaluation, when expressed as standardized effect sizes, most of the impact estimates are small. For the statistically significant (p < 0.10) impact estimates reported in Table 6, the effect sizes range from 0.17 (paid employment and disability payment amount) to 0.39 (earnings from employment). All of these fall short of the conventional threshold for a medium effect size: 0.50 (Cohen, 1988). But because about 1.2 million adolescents and young adults ages 13–25 received federally administered SSI payments in 2015 with a total value of $8.7 billion (SSA, 2017), an intervention that achieves even modest increases in employment and earnings among those individuals, and carries the potential to lessen their dependency on disability payments in the future, could be a valuable policy tool for managing the size of the SSI program.
The youth who enrolled in the evaluation of BHBF were volunteers who responded to Mathematica’s outreach efforts. Sixteen percent of the youth recipients of disability payments who were targeted for recruitment completed a baseline survey and consented in writing to enroll in the evaluation. Some enrollees already possessed the characteristics that BHBF was designed to affect. For example, the baseline survey revealed that 19 percent had worked for pay in the previous year, and 90 percent expected to do so in the next five years. More intensive or alternative recruiting strategies might have yielded evaluation enrollees with fewer of these baseline characteristics, creating a greater opportunity for BHBF to influence treatment group members and possibly resulting in more and larger impacts on three-year outcomes.
Because the evaluation enrollees were volunteers, the evaluation’s impact estimates should not be extrapolated to the full population of youth disability recipients in Miami-Dade County. More specifically, they should not be used to inform policy decisions about the implementation of hypothetical, mandatory employment-focused interventions for youth disability recipients. However, future interventions for this population are likely to be voluntary, in which case the BHBF impact estimates could be instructive. For example, participation in the PROMISE demonstration projects, funded by the U.S. Department of Education (2013a, b), is voluntary.
SSA’s waivers for YTD participants, which remained in effect for four years after youth enrolled in the evaluation, virtually precluded BHBF from reducing the amount of disability payments the youth received during that period. However, under the terms of Mathematica’s evaluation contract with SSA, data on evaluation enrollees were collected and analyzed for just three years, thus effectively eliminating the possibility that the evaluation would find BHBF reduced disability payments, and increasing the likelihood that it would find the project increased them. SSA plans to conduct analyses of BHBF’s long-term impacts on disability payments and other outcomes using data that it acquires in administering its disability programs.
The randomized controlled trial design for the analysis of the impacts of BHBF was not a significant barrier to the recruitment of youth into the study, nor did it generate many questions or complaints from youth, parents, or other stakeholders. Therefore, designers of future studies should not allow anxiety about random assignment to deter them from specifying rigorous experimental evaluation designs to evaluate interventions for youth with disabilities.
Implications for policy and practice
The findings from this study have two important implications for policy and practice. First, transition interventions that include work experiences as a key service component can make significant differences in the later outcomes of youth with disabilities. The impact analysis of BHBF revealed that, well after the intervention ended, the employment rate of youth in the treatment group exceeded that of their counterparts in the control group. This parallels the results of other studies that suggest strong early work experience and employment-focused interventions positively influence employment outcomes in early adulthood for youth with disabilities (Carter, Austin, & Trainor, 2012; Wehman, Sima, Ketchum, West, Chan, & Luecking, 2014). In addition to improving directly targeted outcomes such as employment and earnings, these interventions may have indirect beneficial impacts, such as decreasing youth’s contacts with the justice system. Federal policy is already moving in the direction of promoting work experiences for youth with disabilities. As a prominent example, the Workforce Innovation and Opportunity Act of 2014 (U.S. Congress, 2014) highlights, among other things, the importance of work experience for youth as they prepare to leave high school and enter the world of work. This study’s findings reinforce the value of this type of intervention, especially when applied to a population with long-standing and seemingly intractable low employmentrates.
Second, project staff who implement employment-focused interventions should be carefully selected, and they will need professional development and support to acquire the skills they will need to help make employment possible for youth with disabilities. It is often challenging for interventions for youth with disabilities to establish and maintain a sharp focus on employment. As documented in this study, when such interventions are staffed with individuals from traditional social service backgrounds, general case management can divert staff from the challenging yet ultimately more productive activities of job development and job placement. The management of BHBF found that the skills and interests of some of its original frontline staff were not ideally matched to the employment focus of the YTD/BHBF program model, particularly with respect to job development. BHBF management adjusted staff retention and hiring policies to achieve a better match between project staff and project goals, ultimately establishing frontline staff who had interest and experience in serving youth with disabilities and conducting outreach to employers. This finding reinforces the need for these kinds of programs to be attentive to key characteristics and traits of potential staff in their recruitment and hiring policies and practices (Tilson & Simonsen, 2013).
This study supports the use of expert TA to ensure intervention fidelity in research projects on transition services. Focused TA can prevent the deviation from program designs and ideal protocols that often happens in such studies; this deviation can negatively influence findings (Duan, Braslow, Weisz, & Wells, 2001). And for BHBF, a critical complement to TA was quantitative monitoring by project managers to ensure frontline staff were devoting enough time to job development and job placement, and were achieving positive employment outcomes for participants. The monitoring was made easier by a management information system that tracked employer contacts, services, and job placements at the individual staff and participant levels. These granular data were the basis for monthly monitoring reports on efforts and outcomes at several different levels of aggregation across project staff and participants. These reports were a powerful performance management tool for BHBF and also informed the delivery of training and support to the project staff.
The field of transition and disability employment services has long recognized the importance of professional development for staff providing employment services. As new disability policies, such as the Workforce Innovation and Opportunity Act, are implemented, attention to professional skill development and support will be critical. To fulfill the mandates of evolving transition and employment policy, transition and employment service staff will need to develop the skills necessary to help youth pursue and succeed in employment. This study suggests that such skills can be reinforced by in-the-field TA and support as a supplement to traditional classroom training and online training. Professionals with the training and experience to deliver such field-based assistance are not widely available, so policymakers should consider ways to expand the field of subject matter experts who are qualified to provide this training and TA.
Conflict of interest
None to report.
Footnotes
Acknowledgments
The research reported in this paper was funded by a contract (number SS00-5-60084) between the Social Security Administration and Mathematica Policy Research. The opinions expressed herein do not necessarily reflect the policy or position of SSA. Nor does the mention of trade names, commercial products, or organizations imply endorsement by SSA.
The preparation of this manuscript was also supported by a grant to TransCen, Inc., from the National Institute of Disability, Independent Living, and Rehabilitation Research (grant number 90DP0007-01-03). The opinions expressed herein do not necessarily reflect the policy or position of the U.S. Department of Health and Human Services.
We are grateful to Joyanne Cobb, Jeffrey Hemmeter, and the late Jamie Kendall (all of SSA) for the guidance they provided to this research.
