Abstract
The authors examined vocational rehabilitation service and outcome differences between youth with disabilities who participated in a model transition intervention and youth with disabilities who did not. The transition model featured research-supported components integrated into a single service model and implemented across 11 Maryland school districts from 2007 to 2013. Using administrative data extracted from the Maryland Division of Rehabilitation Services, several major findings emerged. Model participants experienced a shorter time from eligibility to development of the Individual Plan for Employment, but longer open cases; received more job-related services and less assessment and diagnostic services; cost less to serve; achieved significantly higher employment rates at case closure; and worked slightly fewer hours and earned less per week at closure. These results signify implications for vocational rehabilitation practice with students and youth with disabilities, especially in the context of transition service requirements of the Workforce Innovation and Opportunity Act.
Despite advances in transition service delivery and increased knowledge of research-supported transition strategies, long-standing service and systems issues continue to impede optimal transition outcomes for students and youth with disabilities. These issues are evidenced in challenges that students with disabilities continue to face as they prepare for and experience the transition from school to employment and adult life. Compared with their peers without disabilities, students with disabilities can still expect lower school completion rates (Stark & Noel, 2015), lower adult employment participation (Bureau of Labor Statistics, 2016), higher dependence on public income support (Davies, Rupp, & Wittenburg, 2009), and higher incidence of poverty (DeNavas-Walt & Proctor, 2014). In addition, there remains an ongoing challenge to link students to adult support services that many youth with disabilities need to achieve and sustain postschool employment success (Certo et al., 2009; Clark & Unruh, 2009). The need persists to identify cohesive transition service delivery models that incorporate the multiple factors, supported by a growing body of research, that contribute to successful transition from school to adult employment for these youth.
Given these circumstances, researchers, transition policy makers, and transition service practitioners have identified strategies to mitigate barriers to successful transition to employment for students and youth with disabilities. Since the publication of a compilation of research-supported transition components, called the Guideposts for Success (National Collaborative on Workforce and Disability for Youth [NCWD/Y], 2005), more research has emerged to support several specific factors that contribute to a higher likelihood of postschool employment success. First, several studies continue to reinforce the importance of work experience in secondary school, especially paid employment, in influencing adult employment (e.g., Carter, Austin, & Trainor, 2011; Wehman et al., 2014; Test et al., 2009). Second, family involvement and expectations can influence later employment success of students and youth with disabilities (Carter, Austin, & Trainor, 2011; Wehman et al., 2014). Third, collaborative service coordination between schools, vocational rehabilitation (VR), adult employment service providers, and other community partners has been shown to influence employment success when these collaborations focus on achieving individual youth outcomes (Fabian, Dong, Simonsen, Luecking, & Deschamps, 2016). Finally, recent examination of state VR services to transition-aged youth shows that initiation of VR service well in advance of school exit contributes to successful VR case closures for transition-aged VR service recipients (Honeycutt, Bardos, & McLeod, 2014).
Federal lawmakers have taken several of these factors into account in crafting new legislation as reflected in the Workforce Innovation and Opportunity Act (WIOA; 2014). Title IV of WIOA specifies that 15% of each state’s federal funding allocation for VR services must be designated for transition services for students and youth between the ages of 16 and 24 years. Furthermore, WIOA has established a new category of VR funding, called Pre-Employment Transition Services (Pre-ETS), which can be applied to services before a student exits school. Work-based learning experience is one of the five types of Pre-ETS that VR can authorize for eligible or potentially eligible students as they are preparing for transition from secondary school to adult employment and careers. WIOA also emphasizes interagency collaboration between VR and schools so that information is shared, joint planning occurs, and services from each system are jointly coordinated. Thus, WIOA has the potential to influence the adoption of transition practices that lead to improved employment outcomes for students and youth with disabilities. However, as states and localities endeavor to implement effective transition services, they will need not only to meet the requirements of WIOA but also to organize the delivery of disparate, and often disjointed, transition service elements in a way that is integrated for the best effect and that reflects empirical support for efficacy.
One recent illustrative effort to organize and deliver an integrated model of transition service delivery is represented by the Maryland Seamless Transition Collaborative (MSTC; Luecking & Luecking, 2015). MSTC is a transition service delivery model, implemented in 11 countywide school districts in Maryland, which includes the integration of components represented in the Guideposts for Success (NCWD/Y, 2005), and which have parallels to the recently enacted WIOA. Although the Guideposts are now over a decade old, they present five research-supported components of transition which still have current relevance. They are: school-based preparatory activities that include academic instruction and targeted curricula for transition to employment; career preparatory and work experience, including employer-based work experiences and jobs; youth development and youth leadership, including self-determination in transition planning; family involvement that supports transition to employment; and connecting activities that link students to services that are complementary to their education and that promote the acquisition of employment and development of careers. In effect, WIOA’s inclusion of Pre-ETS work-based experiences, and school/VR collaboration bring selected Guidepost components into the forefront of transition practice.
Luecking and Luecking (2015) previously described the MSTC model and reported selected descriptive outcomes. This article reports on VR service elements, service costs, and VR closure statuses of youth who participated in MSTC in the years between 2007 and 2013. Using an inverse propensity of treatment weighting approach, we compare these factors with those of youth who received VR services in the same Maryland counties, but who did not participate in the service model. The following two research questions guided this inquiry:
Method
The Maryland Division of Rehabilitation Services (DORS) received a transition model demonstration grant in 2007 from the U.S. Department of Education, Rehabilitation Services Administration, to implement and evaluate the MSTC model. Through a competitive process, 11 of the 24 countywide school districts in Maryland were selected to pilot and implement the model. Each participating school district received a small subgrant to plan for implementation of the model (Year 1) in selected high schools and to subsequently work with local partners to implement it (Year 2), with the expectation that the model be sustained beyond the 2-year grant with existing resources. The subgrants also required specific record keeping requirements for project evaluation purposes. Each site received planning and implementation technical assistance (TA) from DORS partner, TransCen, Inc., an organization experienced in transition service implementation and evaluation. The award of the 11 subgrants were staggered over the first 3 years of the 6-year project so that grant and TA resources were evenly available throughout the project and so that each school district would have at least 3 years of project implementation. The participating school districts represented an assortment of urban, suburban, and rural communities.
The MSTC model is intended to be applicable to any high school student with a disability potentially eligible for DORS services, regardless of disability category (Luecking & Luecking, 2015). Due to distinct community demographic and student characteristics, each local school district was given the option to focus model implementation on specific target populations of students for whom the school district wished to achieve better transition outcomes. Consequently, for example, some districts targeted diploma track youth who were at risk of dropping out of school, whereas others targeted students who were on a certificate track and for whom connections to postschool supports and employment had been historically problematic. Some school districts opted to apply the model across populations. Each district was required to enroll up to 20 students per year, depending on the size of the district’s target population. The project’s target total student enrollment across all districts was 400.
Intervention
As stated, the intervention received by each enrolled student was derived from the Guideposts for Success (NCWD/Y, 2005). The intervention was delivered over a 3-year period, beginning in the 10th grade for diploma track students, or third to last year before projected school exit for certificate track students. The components of the intervention included the following:
Discovery, which included the development of a Positive Personal Profile (Luecking & Tilson, 2009; Tilson & Cuozzo, 2001) that guided the development of work experience and jobs, along with self-determination instruction and self-directed plan development.
Individualized work experiences, which potentially included a combination of informational interviews, job site tours, job sampling, unpaid internships, and summer youth employment opportunities.
Individualized paid integrated employment, defined as one where the student is paid directly by the employer and where other workers are primarily individuals without disabilities.
Family supports across all 3 years, including involvement in discovery activities, input into work experience and job search planning, and direct training in benefits management for families of youth receiving Supplemental Security Income (SSI) or other public income benefits.
Early VR case initiation that occurred no later than the second school year prior to projected school exit.
Systems linkages and collaboration represented by an interagency, cross functional team of professionals in each participating community convened on behalf of specific participants to plan for necessary service integration across the 3 years. The team included representatives from DORS, education, mental health services, intellectual/developmental disabilities services, community rehabilitation providers (CRPs), and other community services.
Coordination with teachers and instructional staff so that regular educational services were supplemented and coordinated with services from the team members, with education professionals involved in work experience and job search planning.
To insure intervention fidelity, targeted TA was provided to each site designed to build their capacity to effectively implement the program model. The TA was intensive, involved a range of delivery modes, and was informed by an intervention manual and accompanying fidelity checklist. The latter checklist provided the basis for regular feedback and consequent focus of TA to insure effective program implementation. Each local site was assigned a TA liaison who provided support and assistance in resolving specific implementation issues. The TA included four basic approaches. First, the TA liaison attended the monthly meetings of the local teams (see No. 6 above) to facilitate collaboration in model implementation. Second, the TA liaison delivered in-person TA to front-line staff implementing the model. This consisted of trainings on strategies for discovery processes, engaging families, and networking with employers to facilitate work experiences. It also consisted of just-in-time TA to quickly resolve specific implementation issues that arose. Third, the TA liaisons arranged quarterly meetings attended by representatives from each of the local sites to share information and participate in topical training. Fourth, each local site coordinator met as needed, but no less frequently than once per month, with the assigned TA liaison to review the fidelity checklist and project progress.
Sample
DORS provided the administrative data for the analysis. The files they provided included (a) all MSTC youth and (b) all non-MSTC youth less than age 22 at application and determined eligible between July 1, 2007, and June 30, 2013. We restricted the records to exclude youth who applied before 2007. The treatment group consisted of VR applicants who were identified as participants in the MSTC program. The comparison group consisted of VR applicants who resided in the 11 participating MSTC counties, but who did not themselves participate in MSTC. The final sample included 377 MSTC youth and 6,111 non-MSTC youth.
Measures
The administrative records included a range of variables for demographic characteristics, VR service provision, and VR outcomes. We selected the following demographic characteristics to include in the analysis: age at application, sex, race (White, Black, and Other), disability type (intellectual disability, psychiatric/serious emotional disturbance, specific learning disability, autism, and other), SSDI enrollment at application, and SSI enrollment at application. We considered a diverse set of VR process measures related to assessment, timing, and receipt of services, along with closure outcomes. The full set of VR-specific measures is shown in Table 1.
Process and Outcome Measures Included in the MSTC Evaluation.
Note. An individual was classified as receiving “any service” if he or she had positive expenditures recorded in at least one category. MSTC = Maryland Seamless Transition Collaborative; IPE = individualized plan for employment; VR = vocational rehabilitation.
Service receipt in the listed categories was determined by recorded expenditures greater than US$0 attributed to any of the RSA 911 service codes assigned to the category.
Analytical Approach
The data allow for a nonexperimental impact analysis of the MSTC program. From the VR administrative data, we were unable to discern the criteria that the school districts applied to select students into the MSTC program. Districts varied in the disability and other criteria used to identify potential youth, and we had no records to identify which youth were offered and subsequently did not participate in the program. To address this selection issue and to reduce dissimilarities on observable characteristics between MSTC participants and nonparticipants, we used an inverse probability of treatment (IPT) weighting approach. This approach generates a synthetic sample in which participation in MSTC is independent of observed characteristics. However, if selection involved unobservable characteristics, as is likely, our estimated results could still be biased. Our results should be interpreted with caution.
The IPT weights are derived from a propensity score model that predicted the probability of MSTC participation based on a number of individual-level characteristics related to expected education and employment outcomes. These variables include age at application and its square, sex, race, disability type, Social Security Disability Insurance (SSDI) enrollment at application, SSI enrollment at application, and year of application. These models also included county-level characteristics that we constructed from U.S. Census data for applicant’s application year: log of county median income, log of county population, county unemployment rate, and percent of the county’s population that was White. To eliminate outliers, we trimmed the observations with the highest and lowest one percent of propensity score values. We then generated weights equal to 1 for all MSTC participants and equal to pi / 1 − pi for nonparticipants, where pi is the propensity score for applicant i. We verified that the IPT weighting approach resulted in improved balance on observable characteristics, then compared the outcomes for MSTC and non-MSTC youth using the IPT weights.
The distribution of propensity scores can be seen in Figure 1. Although MSTC participants and nonparticipants had considerable overlap in the support of the distributions, the propensity scores for the comparison group were more heavily concentrated at the lower end of the range. This preponderance of comparison group applicants who bore little resemblance to MSTC participants suggests the IPT weighting approach is appropriate to generate better balance on demographic and disability characteristics between the two groups.
Distribution of propensity scores for MSTC participants and nonparticipants in MSTC counties.
We tested the hypothesis that MSTC participants differed from nonparticipants in the VR application, service, and outcome measures of interest (the null hypothesis being that the two populations had no significant differences). The analysis relied on weighted two-tailed t tests for all outcomes, as the direction of the differences between MSTC participants and nonparticipants could not be predicted for most measures. For categorical variables, we generated binary variables for each category to conduct t tests, but also tested for differences in distribution using chi-square tests. For some measures, it was not possible to obtain an estimate for every member of the sample (e.g., the time between application and closure is not possible to compute for individuals who had not closed at the time we received the data, and wages at closure are not observed for individuals who did not exit with employment). In these cases, statistical tests are inappropriate due to potential differential selection between the two groups. Therefore, we conducted only descriptive analyses for those outcomes.
The effect of the IPT weighting methodology can be seen in Table 2. Whereas the unweighted comparison group differs significantly from the treatment group on age, sex, race, disability, and county characteristics, almost all of the differences disappear after applying the IPT weights.
Summary Statistics by MSTC Participation.
Note. Table provides inverse propensity score weighted means and standard deviations for MSTC participants and non-MSTC participants in MSTC counties. MSTC = Maryland Seamless Transition Collaborative; SSI = Supplemental Security Income; SSDI = Social Security Disability Insurance.
Results of statistical tests of variables for which only a subset of observations have nonmissing values are not reported. We could not observe SSI status (SSDI status) at application for 104 (107) MSTC participants and 2,011 (2,328) non-MSTC participants.
p < .05. **p < .01. ***p < .001. Statistical significance of difference between cell value and MSTC value using two-tailed t test.
Results
Were There Differences in VR Service Patterns and Costs for Youth Who Participated in the Service Model Compared to a Matched Sample of Youth Who Did Not?
Application and assessment
Table 3 compares MSTC participants and non-MSTC participants in three categories of application and assessment characteristics: time between application and eligibility determination, attainment of an Individual Plan for Employment (IPE), and time between eligibility determination and IPE. MSTC participants experienced a shorter time for eligibility determination by an average of a little over 5 days. Ninety-one percent (91%) of the MSTC participants obtained an IPE compared with less than 74% of the non-MSTC participants. Both of these differences are significant at the .00l level. Conversely, compared with non-MSTC participants, a smaller percentage of MSTC participants (9% vs. 26%) did not obtain an IPE. Finally, MSTC participants experienced a much shorter average time between eligibility determination and IPE (132 days vs. 238 days).
DORS Application, Assessment, and Service Receipt by MSTC Status.
Note. Table provides inverse propensity score weighted means and standard deviations for MSTC participants and non-MSTC participants in MSTC counties. DORS = Maryland Division of Rehabilitation Services; MSTC = Maryland Seamless Transition Collaborative.
Results of statistical tests for outcome variables for which only a subset of observations have nonmissing values are not reported.
p < .05. **p < .01. ***p < .001 statistical significance of difference between cell value and MSTC value using two-tailed t tests.
VR service receipt
Table 3 also shows the percent of each group that receive any VR service and the categories of service each group received. There was no statistically significant difference in the percentage of each group who received any VR service. However, there were marked differences in the type of services received. A higher percentage of MSTC participants received job search assistance (37.1% vs. 30.2%), and on-the-job supports (45.9% vs. 21.4%). Non-MSTC participants received more rehabilitation technology (20.7% vs. 10.9%), assessment (66.9% vs. 52.0%), diagnosis and treatment (15.5% vs. 7.4%), college training (9.0% vs. 3.2%), and other services (23.2% vs. 14.1%). The average cost of purchased services was almost US$1,200 higher for non-MSTC participants compared with MSTC participants.
Were There Differences in VR Closure Outcomes for Youth Who Participated in the Service Model Compared to a Matched Sample of Youth Who Did Not?
Employment at closure
Table 4 compares MSTC and non-MSTC participant closure outcomes. Almost 42% of MSTC participants were employed at closure compared with less than 24% of non-MSTC participants. Conversely, fewer MSTC participants (34.5%) experienced case closure without employment than non-MSTC participants (48.6%). The difference in percentage still receiving services at the time of the data extract (almost 24% of MSTC participants and 28% of non-MSTC participants) was not significant.
Closure Outcomes by MSTC Status.
Note. Table provides inverse propensity score weighted means and standard deviations for MSTC participants and non-MSTC participants in MSTC counties. MSTC = Maryland Seamless Transition Collaborative.
Categorical variables also tested for difference in distribution using chi-square test. Difference significant at the .001 level. bResults of statistical tests for outcome variables for which only a subset of observations have nonmissing values not reported.
p < .05. **p < .01. ***p < .001. Statistical significance of difference between cell value and MSTC value using two-tailed t tests.
Employment characteristics at closure
As noted earlier, we cannot statistically compare MSTC and non-MSTC participants on employment-specific variables because of the nature of our evaluation design, but a descriptive comparison of the differences suggests important variation between MSTC and non-MSTC participants. Of those employed at closure, MSTC participants earned an average of US$8.02 per hour, compared with an average of US$8.54 per hour earned by non-MSTC participants. MSTC participants worked slightly fewer hours per week (22.1) than non-MSTC participants (24.8). The resultant weekly earnings at closure yielded a difference of almost US$36 between the two groups (US$218.92 vs. US$182.86).
Other closure variables
As with employment characteristics at closure, we could not statistically compare the differences between MSTC and non-MSTC participants on other closure variables, and present descriptive statistics as suggestive of different aspects of the VR experience. First, MSTC participants had longer times between application and closure (by more than 4 months) and between plan and closure (by more than 5 months) than did non-MSTC participants. Second, for closure reasons other than employment, MSTC youth had lower rates of noncooperation and refusing services, and higher rates of unable to locate or other reasons, than non-MSTC participants.
Discussion
The comparison of MSTC participants to nonparticipants yielded five key findings. First, the MSTC intervention resulted in faster time from eligibility determination to the development of the IPE, but cases remained open longer. Both of these factors reflect the early involvement of VR, well before projected school exit. It is logical that the early collaboration of VR with the schools would accelerate the sharing of information on individual students and early engagement with the students and their families resulting in a shorter time to the development of the IPE. It is also logical that opening the VR case well before school exit would result in cases being open longer, as this would mean that it would take a minimum of 2 years for the student to exit school and for VR to close the case.
Second, VR spent fewer case service dollars on the MSTC participants than on the nonparticipants. This suggests that the integration of resources with schools and other community partners enables the VR service funds to be maximized. It also suggests that work-based experiences and jobs can be accomplished through this type of collaboration with less expenditure than when it does not exist, or when general VR service funds are applied in isolation from other resources and partners. It also suggests that longer case opening does not necessarily correspond to increased use of VR service funds. However, this finding must be interpreted with some caution and qualification. Some, but not all, of the local school districts spent small portions of their implementation grant funds for contracting with local CRPs for student job development and job coaching. This means that such expenditures would have reduced the need for VR to apply service funds to serve these students. It is not possible to pinpoint exactly how much of the grant funds were allocated for this purpose, but the fact that it happened in some circumstances reduces the overall confidence in interpreting this finding as a reflection of the intervention’s impact on cost. We also were unable to measure to what extent school districts may have spent other funds on serving youth and connecting them to work experiences. Such expenditures may increase overall costs to serve youth in the manner prescribed by MSTC. In any case, the findings suggest that VR service costs can be reduced when braided with school and other community resources.
Third, compared with the matched comparison group, the MSTC participants received more work-focused services from VR, such as job search assistance and on-the-job-support, both of which involved the expenditure of VR funds as these services are typically purchased from a CRP. They also received less assessment and diagnostic related services. These patterns reflect the nature of an intervention that emphasized work experiences and paid employment, they also indicate that common assessment and diagnostic services are less necessary and less useful when VR case initiation happens well before school exit. This is also a possible explanation for fewer case service funds being spent on MSTC participants, although it is important to recognize the cautions about interpreting the relative difference in case expenditures discussed above. At the very least it is a reflection of when and how to spend available service funds to be best effective. That is, applying service funds well before school exit, and applying them to work-related activities, has the potential to be an effective approach to achieving a better return on the investment of case service funds.
Fourth, and significantly, the MSTC participants achieved appreciably higher employment rates at VR closure than the comparison group. This outcome offers further support of previous research findings that illustrate the value of work experience in high school (Carter et al., 2011; Wehman et al., 2014), early VR case initiation (Honeycutt et al., 2014), and outcome-oriented collaboration (Fabian et al., 2016). What is new about this evaluation is that it compares outcomes when these features are integrated into a single model of transition service delivery for VR eligible students, regardless of disability category and the demographic characteristics of the communities in which they live. Although it is not possible to determine whether one, or more than one of these features in combination, contributed more or less to the employment outcomes achieved, it can be inferred that the model of service delivery represented by MSTC offers potential guidance for organizing transition collaboration and service delivery in the post-WIOA era.
Finally, although the MSTC participants achieved a higher rate of employment, they worked slightly fewer hours and earned less on average than those who worked in the comparison group. These differences are notable and worthy of further examination in future studies. In both the MSTC group and the comparison group, there were a number of individuals whose cases were still open (24% and 28%, respectively) and were still receiving services at the time of data extract. This means that the ultimate case closure outcomes are not certain, and may have a differential influence on interpreting the differences between the two groups.
Limitations
The present study utilized administrative data provided by DORS. Counselors enter case information at various points of the VR service process and, despite cross-checks to minimize errors, subjective recall bias is possible during data entry. Also, the IPT weight analyses address some, but not all, of the issues associated with the absence of a true control group. Comparisons between MSTC and non-MSTC groups are thus not necessarily purely representative of demographic and service differences. This method of analysis therefore yields correlational, but not causal, results. Finally, external objective indicators of outcome, such as record of postexit earnings, would provide additional support for model effectiveness on youth employment outcomes.
Implications for Rehabilitation Practice
The results of this evaluation of a comprehensive multisite transition intervention offer several implications and recommendations for rehabilitation practice. First, it appears that early case initiation may exert a positive impact on service outcomes for transitioning youth, a finding consistent with new WIOA regulations requiring more active outreach and case referral by VR counselors of youth with disabilities eligible or potentially eligible for VR services while still enrolled in school. Second, VR counselors might want to increase procurement and/or use of vocationally oriented services (such as job search assistance and on the job supports), as these employment-related services appear to lead to more successful case closures for youth, and these services are incorporated in the WIOA description of Pre-ETS services. Although our data do not specifically support the value of paid work-based experiences (such as summer youth employment, or paid internships) alone as a critical service option, prior research strongly substantiates the value of paid employment above and beyond other types of work-based learning activities (Fraker et al., 2016). This study not only provides further support for this as a transition intervention but also suggests that of the five categories of Pre-ETS allowed under WIOA, work-based experience should be a priority as VR counselors authorize Pre-ETS for students with disabilities on their caseloads.
A final and critical implication for rehabilitation practice is interagency collaboration. As described earlier, this activity was a key component of the MSTC model, and VR counselors actively participated on interagency teams at each local implementation site, which were charged with administering all aspects of the model to achieve seamless transition outcomes for all participating youth. One prior study exploring the effect of MSTC interagency collaboration on youth outcomes (Fabian et al., 2016) supported the value of these teams in improving youth outcomes, particularly for those teams with an outcome-oriented focus. Because interagency coordination and collaboration of VR agencies with other state, local, and private entities involved in transition is a hallmark of WIOA, we anticipate an expanded role for VR counselors in active participation on interagency teams as they move forward to implement these regulations, and improve transition-related services to youth with disabilities.
Conclusion
This evaluation study demonstrated the potential effectiveness of a career/work-focused transition intervention on improving employment outcomes for transitioning youth regardless of disability. The study demonstrated that strategic management of existing services and resources (early case initiation, work-focused experiences, and outcome-oriented service collaboration) yielded better service outcomes at less service cost for youth. Future research using a more rigorous research design should explore the long-term benefits of comprehensive transition interventions on youth wages over time, reduction in public benefits use, and increases in financial independence.
Footnotes
Acknowledgements
The authors would also like to acknowledge the cooperation of the Maryland State Department of Education, Division of Rehabilitation Services, in the preparation of this manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was partially supported by the National Institute of Disability, Independent Living, and Rehabilitation Research, U.S. Department of Health and Human Services, Grant H13310007 to TransCen, Inc.