Abstract
Access to effective and comprehensive transition programming is pivotal to transition of youth with disabilities to work and independent living. Successful programs often blend key ingredients, including individualized educational planning, career development, work experiences in secondary school, and interagency partnerships/collaborations. Through a comparative analysis of the New York State transition program data, and the National Longitudinal Transition Study 2 (NLTS2) data, this study empirically demonstrated the simultaneous contributions of various transition program elements to student postsecondary outcomes using mediation models. In addition, the technique of using propensity score analysis for balancing the two comparison cohorts, applied in this study, contributes to the arsenal of analytical techniques for evaluating the impact of transition to adulthood programs.
Engagement in employment and postsecondary education for youth with disabilities continues to be less than optimal in comparison with their peers without disabilities (Newman et al., 2011). Though this difference appears to be shrinking, recent data from the National Longitudinal Transition Study 2 (NLTS2) on young people with disabilities 8 years after their secondary school exit indicate that lower proportions of young people with disabilities were enrolling into postsecondary education, which includes 2- and 4-year colleges, as well as postsecondary vocational schools, compared with their peers without disabilities (60% vs. 67%). Specifically, young people with disabilities were less likely to attend 4-year university degree programs and more likely to attend 2-year and vocational programs compared with their peers without disabilities. More worrisome is that their postsecondary college completion rates were lower than their peers (41% vs. 52%). Furthermore, the differences in the rates of employment between young people with disabilities and without disabilities were not statistically significant (61% vs. 66%). Much of this could be attributed to the growing unemployment rate among youth due to the Great Recession of 2007 rather than improved employment rates for youth with disabilities (Sum, McLaughlin, & Khatiwada, 2008). Data from the Office of Disability Employment and Policy, U.S. Department of Labor indicate distinct differences in the employment numbers for youth with and without disabilities, based on the Current Population Survey data. About 25% of young people with disabilities were employed in contrast with 62% of young people without disabilities in the first quarter of 2012 (U.S. Department of Labor, 2012).
From a social developmental perspective, employment or work-based engagement, in addition to paving pathways to economic self-sufficiency and higher community participation, provide opportunities for greater civic and social participation, especially for individuals with disabilities (Schur, 2002). With an increasingly knowledge-based labor market and higher preference for workers with postsecondary educational degrees, attending college after secondary school continues to be an important element toward ensuring labor market engagement for individuals with disabilities (Fleming & Fairweather, 2012). Fittingly, these two transition outcomes (i.e., postsecondary employment and education) have received sustained attention in policy making in education and secondary educational institutions continue to collect data on these outcomes. In turn, school systems receive funding from Part B of the Individuals With Disabilities Education Act (reauthorized as Individuals With Disabilities Education Improvement Act in 2004) that mandates schools to establish measurable and developmentally appropriate postsecondary goals related to education, employment, and independent living, based on student’s strengths, as well as develop a clear statement of transition services that identifies the resources and services needed for supporting the needs of the transitioning young person. Despite such mandates, less than half of the schools demonstrated full compliance with these requirements (Landmark & Zhang, 2012).
As efforts continue to improve school compliance to these requirements, several model transition demonstration programs continue to be implemented for identifying best practices leading to successful postsecondary employment and education outcomes for these young adults (Nietupski et al., 2004; Rutkowski, Daston, Kuiken, & Riehle, 2006). These model programs, over the years, have also provided important data for supporting the developmental theories of transition to adulthood. Reviewing more than 25 years of research and more than 162 published articles, Test et al. (2009) found moderate degree of evidence for only four practices (i.e., inclusive educational environment, paid work experience in secondary school, vocational education, and work study) leading to improved success in postsecondary employment and education for transitioning youth and young adults. Low sample size, convenient sampling strategies causing issues of selection bias coupled with a lack of rigorous program evaluation data contributes to heterogeneity of impact of interventions on program outcomes, further diminishing the aggregate program effects of many model programs. In addition, lack of comparison or control groups to indicate relative effectiveness of programmatic impact often limit its ability to identify program effects and most transition to adulthood demonstration research studies fail to identify the pathways through which program effects are observed for participating youth and young adults.
The purpose of this research was to address the above stated limitations in evaluating the impact of one transition program. This research compared the postsecondary outcomes for youth and young adults with disabilities served in a model transition program (MTP) implemented across 60 sites in New York State (NY) and a nationally representative sample of youth and young adults from the NLTS2. Utilizing methods of propensity score analysis and mediation analysis, with simultaneously controlling for the effects of sociodemographic variables, as well as selection bias in program participation, the purpose of this study was to identify empirical evidence for critical transition to adulthood program elements contributing to positive postsecondary outcomes for youth with disabilities.
Method
Defining the Context: Brief Description of MTP
Brewer et al. (2011) provided a detailed description and preliminary findings of program outcomes of NY State’s MTP. In brief, the Office of Adult Career and Continuing Education Services—Vocational Rehabilitation (ACCES-VR), the NY-designated VR agency housed within the New York State Education Department, funded 60 MTP sites to improve postsecondary outcomes for youth with disabilities. Funding was awarded (via a request for proposal process) to local education agencies (LEA) that entered into collaborative agreements with community partners, such as Centers for Independent Living, community rehabilitation providers, and postsecondary education. Smaller LEAs were allowed to group themselves into consortia as part of the proposal process. Through the proposal process, ACCES-VR sought representation from all regions of NY, and from urban, suburban, and rural communities. The MTPs were required to (a) use evidence-informed transition to adulthood strategies, (b) establish collaborative service delivery networks with community-based service providers, (c) collaboratively work with the local VR district offices to ensure timely and high-quality referral for students in MTP sites during the final 2 years before their exit from secondary school. The funded MTP programs were provided broad guidelines without specific program or practice recommendations or requirements. This provided opportunities for the funded schools and school districts in NY to adopt existing leading practices in transition and innovate in its implementation of the programs. However, this latitude also resulted in a wide variety of strategies and practices, making the evaluation process challenging.
Utilizing a logic-model framework, Brewer et al. (2011) established a comprehensive data collection system for evaluating impact of the MTP programs on postsecondary outcomes for youth with disabilities. The data collection system provided student-level data on approximately 16,000 students measured quarterly across a 2-year time frame. Collected data were the basis for an in-depth study of program-level factors related to successful transition to adulthood for youth with disabilities. Using methods of conditional logistic regression for estimating aggregated program effects, results indicated MTP students who participated in career development activities, had measurable postsecondary goals in their Individualized Education Plan (IEP), and those who received services from community-based providers were more likely to participate in work experiences in-school and have a successful referral to VR for postsecondary training and education services. The research further highlighted the role of school-VR collaboration as an important ingredient in driving program outcomes.
Comparison Group and Analytical Data Structure
One of the major limitations in the MTP, as with other similar transition program evaluation studies, was the lack of an explicit community-based comparison or control group to assess the relative impact of MTP on postsecondary outcomes for NY youth and young adults with disabilities. This key limitation was addressed by establishing a comparable sample of youth and young adults who participated in the nationally representative longitudinal follow-up study—the NLTS2. The two data sources used in this study included (a) NY MTP data and (b) the comparison sample from NLTS2.
NY MTP data
This data set consisted of merged data from the Transition Impact Data (TID)–online system and the Case Management System (CaMS) data. The TID-online (Karpur, Malzer, & Smith, 2009) is a web-based data collection tool, designed to catalog quarterly information on indicators of progress for students participating across all 60 MTP projects. These indicators were developed based on a review of existing literature and in line with the previously stated goals of the MTP project. In addition to collecting information on student demographics, TID collected information on their secondary school experiences central to MTP project goals (e.g., participation in career development activities, transition planning, work experiences, services received from collaborating agencies). Student transition coordinators, and in some instances, student class teachers, at each of the 60 MTP sites entered and updated student records on a quarterly basis. In addition to serving as a repository of data for program evaluation analysis, the TID system generated systematic reports for each site to facilitate their program improvement and implementation efforts.
The NY CaMS data set, obtained from ACCES-VR, consisted of information on consumer/student characteristics, VR case status, information on services received, employment/work-related engagement information, and closure status for referred consumers (here, MTP students). In the MTP project, students received most of their services from schools while in secondary school and following their secondary school exit, the VR system continued to serve their postsecondary needs in securing a job through a variety of employment training, rehabilitative, and other services. As CaMS data are also utilized to populate the national-level Case Service Report (RSA-911) for VR programs, it served as a reasonable resource to inform on postsecondary employment-related engagement for MTP program participants. It is important to note that almost all the 11th and 12th graders, and some in 10th grade were referred to VR for services and most were determined eligible for receiving VR services.
The NY cohort consisted of the students who exited from the MTP program in October 2008 and had a matching record within the CaMS data set. A total of 4,788 MTP students exited MTP program and among them, 4,063 had matching records in the CaMS data set with 85% match rate. The remaining 725 student exiters did not have matching records in the CaMS data set, as most were not referred to VR as students and/or parents declined VR services. These two groups did not differ significantly on their demographic and other school-based experiences. Thus, the NY cohort consisted of data on secondary school experiences and their postsecondary VR engagement 1 year post exit, reported as of October 2009. Distribution of demographics for NY is presented in Table 1.
Demographic Comparison of Students in NY Transition Program and NLTS2 NS of Secondary School Students With Disabilities.
Note. NY = New York State; NLTS2 = National Longitudinal Transition Study–2; NS = national sample; GED = General Educational Development; SSI = supplemental security income.
n = 4,063. bn = 2,423.
The NLTS2
The NLTS2, sponsored by the U.S. Department of Education, Office of Special Education Programs (OSEP), was a 10-year longitudinal follow-up study (2000–2010) with information on secondary school and postsecondary experiences for 12,000 young people with disabilities receiving special education services who were 13 to 16 years old in December 2000–2001 (Wagner, Cameto, & Newman, 2003). The information in NLTS2 was collected through multiple sources, including parent/youth telephone interviews, direct assessments, and school data collection. The survey was representative of special education students in secondary schools in the United States, and its main purpose was to provide data on the transition experiences for youth and young adults with disabilities. In addition to describing secondary school experiences and outcomes, the NLTS2 also followed up these youth and young adults 2 to 5 years after exiting secondary school. The survey instruments were fielded in several waves across the 10-year data collection period. In summary, the NLTS2 contained five waves of parental interview data, four waves of youth interview data, two waves each of direct student assessment, school program survey, teacher survey, and student transcript analyses.
To replicate the same data structure as the NY cohort, data from the Wave 1 exiters were merged to their postsecondary information in Wave 2, and similarly data from Wave 2 were merged to their postsecondary information in Wave 3. Key variables of secondary school experiences aligning with the TID data elements (see Table 2) were selected in the final data set leading to 2,343 national sample (NS) youth and young adults. Distribution of demographic variables for NS is provided in Table 1.
Comparison of Secondary School Experiences and Work-Engagement Outcomes for Youth With Disabilities in NY Program and NLTS2 NS.
Note. NY = New York State; NLTS2 = National Longitudinal Transition Study–2; NS = national sample; IEP = Individualized Education Plan.
Dependent or Outcome Variable
As the information on postsecondary outcomes for NY and NS students were collected 1 year post exit, it is likely that many students were still engaged in employment-related training programs, including postsecondary education training with VR system and some had postsecondary employment. Because young people with disabilities often take several years to be in a regular full-time employment position (Davis & Vander Stoep, 1997), a composite outcome variable capturing these different facets of transition to work was constructed. Therefore, students who had a paid job after graduation, as well as those who received training (including postsecondary education) to secure employment after secondary school were considered to have positive postsecondary employment-related engagement. Students who had neither postsecondary employment nor were engaged in some type of training leading to employment were not considered to have positive postsecondary employment-related engagement.
Independent Variables
Indicators of participation in school-based transition program were independent variables of interest in this analysis. This included student participation in career development activities, in-school unpaid work and paid work experiences, establishment of postsecondary goals in IEP or transition planning, and services received from community-based partner providers other than the schools the student attended. Specific career development activities included training students to prepare them for work, including career counseling, prevocational training, job search training, and internship or apprenticeship. The sociodemographic variables, including disability classification were treated as independent variables in the context of multivariate analysis.
Selection Bias: Observable Group Differences
Preliminary analysis of demographic and other program participation variables indicated systematic differences between the NY and NS groups. As indicated in Table 1, both the NY and NS cohorts had higher a proportion of males compared with females, and both had higher proportions of Caucasian students. However, NS had a higher proportion of Hispanic/Latino students compared with NY cohort (15 % and 6%, respectively). Also, most of the NY sample students consisted of 11th- and 12th-grade students, whereas most students in NS sample were of 10th or 11th grade. Thus, students in NY were slightly older than the NS cohort. The NS cohort consisted of higher proportions of students with developmental disabilities and sensory disabilities, whereas the NY cohort consisted of higher proportion of students with learning disabilities, emotional disabilities, and multiple disabilities. Furthermore, 20% of NS students received income from the Social Security Administration’s Supplemental Security Income (SSI), where only 8% of NY cohort students indicated they received SSI during the year of their exit.
Table 2 further illustrates the differences in secondary school experiences of students in the two groups. The NY students were twice as likely to have participated in career development activities compared with the NS students (82 % vs. 42 %). A greater proportion of NY students participated in training related to specific jobs and internships compared with their counterparts in the NS. Higher proportions of NY students had postsecondary work and education-related goals on their IEP, and were more likely to have received services from partner provider agencies compared with the NS. However, the NS students were more likely to have in-school unpaid work experiences and paid work experiences during secondary school compared with the NY cohort. As much of these variables have been found to predict successful postsecondary employment and education, differential distribution of these factors across the comparison groups has substantial likelihood to bias the inferences of comparative analysis. Lacking balance among observed factors across the two groups, which is typically overcome through the process of random assignment, a regression-based approach of propensity scores was utilized in this study.
Propensity score analysis attempts to emulate the conditions of a randomized trial where, in theory, the treatment and comparison groups are made to look equal on a set of observable factors that may be related to treatment assignment. Detailed descriptions in theoretical aspects of propensity scores are provided by Joffe and Rosenbaum (1999) and Rosenbaum and Rubin (1985). Empirically, within the logistic regression framework, this is represented as
where pi is the probability that a young person is assigned to NY versus NS groups and Xj is an independent variable that predicts this group assignment.
Using a combination of demographic and disability-related variables, individual-level probabilities of participating in NY compared with NS groups were calculated. Guided by Brookhart et al. (2006) and Austin, Grootendorst, and Anderson (2007), main and interaction effects of variables most likely to affect postsecondary outcomes for transition age youth with disabilities were included in the propensity score model (Haber, Karpur, Deschênes, & Clark, 2008; Karpur, Clark, Caproni, & Sterner, 2005; Murray, 2003). Final propensity score model included Gender, Grade Level, Race/Ethnicity, Disability Classification, Receiving Public Benefits like SSI, Age, and their interaction terms (Race × Gender, Race × Disability, Grade × Disability, Grade × Age, Disability × SSI, Disability × Age).
Once estimated, propensity scores can be used in various ways. Three common methods include (a) matching individuals in treatment and control groups based on propensity scores, (b) stratifying study sample based on propensity scores and calculating treatment effect within each stratum, and (c) using propensity scores as covariate in modeling treatment effects (Heinze & Juni, 2011). Each one of these has its limitations, discussion of which is beyond the scope of this article; however, D’Agostino (1998) suggested classifying the study sample based on propensity scores and using these strata as covariates substantially reduces bias in comparison with matching alone. This study used this strategy as it did not require eliminating extreme values for propensity scores for each group and also provided a better way for addressing selection bias in a modeling framework.
Propensity scores were therefore subclassified into five strata or quintiles to form homogeneous groups with equivalent probabilities of being assigned to NY or NS group. Within each propensity score quintiles, the distribution of each of the demographic and disability classification was studied across the NY and NS group to ensure that proper balance was achieved for these variables. The balance for each binary variable was ascertained using simple logistic regression with each demographic and background variable as a dependent variable, and group assignment (NY vs. NS) and propensity score strata and their interaction effect as independent variables. Nonsignificance of these effects ensured adequate balance achieved and confirmed the appropriateness of the estimated propensity score model (Austin, 2008). Visual examination of cumulative distribution functions of propensity scores across each stratum provided evidence of overlap between the propensity scores for NY and NS (Lanehart et al., 2012).
Regression Modeling
The propensity score strata along with critical program elements of MTP in NY were utilized in building multivariate logistic regression models for studying the impact of participation in MTP project in comparison with the national data. This logistic regression model can be represented as
where po is the probability of employment-related postsecondary engagement, β1 is the coefficient for participating in NY versus NS, β2 is the coefficient for propensity score stratum, and ϵ is the error term.
Furthermore, the analysis also aimed to study the pathways through which the MTP program (NY) affected the postsecondary outcomes for youth and young adults in comparison with the NS group. Mediation analysis was implemented to study the pathways through which the critical elements of MTP program affected the postsecondary outcomes for youth with disabilities. These critical elements included student participation in career development activities, establishment of postsecondary goals of employment and education in IEP, participation in paid and unpaid work, and receipt of services from community-based providers other than the school or school district.
Mediation analysis considers a scenario where the effect of an intervention X on outcome Y is realized by the effect of X on an intermediary variable M and the effect of M on Y (Mackinnon & Dwyer, 1993). This process is illustrated in Figure 1. Furthermore, in this regression-based approach, the estimation of the indirect effect is achieved through the following equations:
where, c represents coefficient for total effects, c′ represents the program effects adjusting for mediator variable M, that is, direct effect, and the difference between c-c′ represents the mediated effects or the indirect effect (ab). Furthermore, Sobel’s test (ab/σab) examines the statistical significance for the indirect effect. As all variables in this analysis were binary/categorical variables, logistic regression methods were used in estimating the coefficients for direct and indirect effects. As the outcome variable, in this case, becomes the logit of outcome probabilities, the scale or variance of outcome is not directly observed, and hence methods of standardization of estimates were applied (Jasti, Dudley, & Goldwater, 2008; Winship & Mare,1983). Besides estimating the indirect effects, its variance, and testing for its statistical significance, an important practical parameter in context of this analysis was the percentage of total effect that is mediated by a given variable. It was estimated as (ab/c). In addition, analysis examined the ratio of indirect effects to direct effects (ab/c′). It is also important to note that the mediation of MTP program effects by each of these critical program elements was assessed, while controlling for the propensity score strata.

Diagrammatic representation of mediation analysis.
Results
Differences in Secondary School Experiences
Table 3 illustrates results of binary logistic regression analysis comparing secondary school program participation of NY and NS samples controlling for the propensity score strata. NY students were 11 times more likely to participate in career development activities in secondary school compared with the NS after controlling for the propensity scores (odds ratio [OR] = 11.2; 95% confidence interval [CI] = [9.4, 13.3]). Similarly, the NY students were 11 times more likely to have an employment goal in their IEP (OR = 11.3; 95% CI = [9.6, 13.0]), nearly 7 times more likely to have a postsecondary education goal in their IEP (OR = 6.7; 95% CI = [5.8, 7.8]), and 3.3 times more likely to have received services from partner agencies (OR = 3.3; 95% CI = [2.9, 3.8]) compared with their peers in the NS after controlling for the propensity scores. Similarly, the NY students were 0.4 times as likely (OR = 0.4; 95% CI = [0.4, 0.5]) or 2.5 times less likely to have participated in in-school work and 0.3 times as likely (OR = 0.3; 95% CI = [0.2, 0.3]) or 3.3 times less likely to have paid work experience in secondary school compared with the NS cohort students.
Summary of Logistic Regression Analysis Examining Differences in Secondary School Experiences Between NY and NLTS2 Samples, Controlling for Propensity Scores.
Note. NY = New York State; NLTS2 = National Longitudinal Transition Study–2; OR = odds ratio; CI = confidence interval; IEP = Individualized Education Plan.
The logistic regression model indicated students participating in NY transition program were approximately 2 times more likely (OR = 1.90; 95% CI = [1.7, 2.2]) to be engaged in work-related postsecondary activities compared with the NS groups after controlling for baseline differences in propensity scores.
Figure 2 illustrates the results from meditational analysis. For convenience, they are presented in the format of a multiple-mediation setting, but each set of dependent variable, mediator variable, and independent variable form one mediation loop examined in this analysis. Furthermore, Table 4 illustrates the mediation effects of NY program elements in terms of the percentage of total effect mediated, as well as percentage of direct effect mediated.

Diagrammatic representation of the mediation analysis examining the impact of NYS on postsecondary work-related engagement through secondary school experiences.
Summary of Simple Mediation Analysis Examining Role of Secondary School Experiences on Postsecondary Work-Related Engagement: NY and NLTS2 Samples, Controlling for Propensity Scores.
Note. NY = New York State; NLTS2 = National Longitudinal Transition Study–2; IEP = Individualized Education Plan.
It can be observed that 17% of the effect of NY transition program on postsecondary work-related outcome was mediated by the impact of NY program on the likelihood of students participating in career development activity. Similarly, 18% of the program effect of NY was mediated by the higher likelihood of NY students having a postsecondary education-related goal in their IEP. Thirteen percent and almost 9% of NY program effect on postsecondary work-related outcome was mediated by the impact of NY program on higher likelihood of transition age young adults participating in in-school work experiences and receiving services from community-based partner providers. Importantly, it can be observed that 25% of lower NY program effect on postsecondary work-related outcomes was due to lower likelihood of NY program students participating in paid work experiences.
Furthermore, it can be noted that participation in career development activities accounted for almost 47% of the direct effect of NY on postsecondary work-related engagement for transition age youth in NY program. Participation in in-school work experiences accounted for 26%, having a postsecondary education goal on IEP accounted for 47% and receiving services from partner providers accounted for 21% of the direct program effects of NY program. Also, the lesser likelihood of participating in paid work contributed to almost 47% lowering of direct program effects of NY program.
Discussion
The purpose of this research was to compare the postsecondary outcomes for youth and young adults with disabilities served in the NY MTP and a nationally representative sample of youth and young adults from the NLTS2. This analysis highlights several important topics for advancing transition to adulthood programming and evaluation. First, by utilizing a NS comparison group, this study was able to highlight the relative impact of the NY MTP. Second, by using a group-balancing approach of propensity score analysis, this research advances evaluation approaches used in examining the impact of transition to adulthood programs for youth with disabilities. Balancing groups on sociodemographic factors, including disability classification, enables all observed factors that are known to affect the outcomes for youth and young adults to be controlled in a regression-based framework. After balancing groups, the evidence revealed that NY youth were twice as likely to have successful postsecondary outcomes compared with the NS, 1 year after their program exit. Third, this research explored the impact of the individual NY MTP program elements in mediating postsecondary outcomes for young people with disabilities.
It is important to note there were key programmatic differences between the NY and NS sample youth and young adults. NY students had significantly higher exposure to career development activities, had significantly higher likelihood of having postsecondary education and employment goals in their IEP, were more likely to participate in in-school work and were more likely to receive services from community-based providers compared with the youth in NS, after controlling for differences in the student background variables. Higher exposure of NY students to these program elements may be attributable to (a) the structure and accountability requirements of the NY MTP and its affiliation to the statewide VR program, (b) approaches and strategies adopted in ongoing continuous evaluation strategies (Brewer et al., 2011; Smith et al., 2012), and (c) the likelihood that community-based providers of services enhanced opportunities for NY MTP students to engage in career development activities such as internships, job skills training, and job shadowing. Also, as data on NS students were collected prior to the federal accountability requirement of inclusion of postsecondary goals in IEP ensuing from the reauthorized IDEA of 2004, it could have potentially contributed to the lower likelihood of postsecondary goals in their IEP compared with NY MTP students. However, the emphasis of this analysis is to understand the mediating role of these secondary school factors on postsecondary outcomes for youth with disabilities.
Transition programming mediators such as participation in career development activities and having postsecondary education goals in IEP, accounted for nearly half of the direct effects of NY MTP program on postsecondary outcomes for students. Furthermore, in-school work experiences without pay accounted for about one quarter and receipt of services from partnering community-based agencies accounting for one fifth of the direct effect of the NY MTP. It is important to note that none of the mediating variables accounted for more than half of the direct NY MTP effects, indicating that there was no single program element that was responsible in driving program outcomes; it was likely that multiple programming elements interacted incrementally to further postsecondary outcomes for NY youth and young adults.
Despite the funding agency requirements, the fidelity of implementation of the various NY MTP components to leading practices and intensity of the programming varied substantially across all the 60 MTP sites. Most NY MTP structures were influenced by the local contextual factors (e.g., urban vs. rural), mix of partnering agencies including partnering schools and school districts, nature and extent of collaboration with regional ACCES-VR offices, as well as school administrator preferences on operationalizing transition programming with schools and school districts. The resulting array of programs, while posing challenges to evaluation, offered an opportunity to examine the impact of each program element on student postsecondary outcome. This further justifies the current approach of developing single-mediation models versus a multiple-mediation model, where the assumption of the latter is that all program elements are operating equally and simultaneously in a given intervention program (Preacher & Hayes, 2008), which was not the case in the NY MTP.
The findings from current research support existing evidence of best practices in postsecondary outcomes for youth with disabilities. Carter, Austin, and Trainor (2012), in their analysis using the NLTS2 data, report paid work experience in secondary school predicted postsecondary employment among youth with significant disabilities in a multivariate model adjusting for their demographic differences. Furthermore, their univariate models indicated a significant relationship between participation in career development activities and postsecondary employment. However, the latter was not substantiated in their multivariate models, which could be due to multicollinearity resulting from a strong correlation between participation in career development activities and paid work during secondary school for study sample. Furthermore, similar to the current analysis, Test et al. (2009), in their systematic review of transition to adulthood literature, identified participation in career development activities, paid employment experience, work study and interagency collaboration among the 16 in-school predictors of successful postsecondary outcomes for youth with disabilities. However, identifying mediation of program effects through multiple in-school variables and quantifying the effect of mediation is the unique contribution of the current analysis. This research supports the need for a multipronged approach to transition programs adding to existing literature (Mazzotti, Test, & Mustian, 2012).
Limitations and Future Research
One of the major drawbacks of propensity scores is that unobserved factors can affect the transition to adulthood outcomes for NY cohort. Past research have shown that factors such as parental and teacher expectations of a student’s positive outcomes (Halpern et al., 1995; Werner, 1993), parental education and socioeconomic status (Svetaz, Ireland, & Blum, 2000), as well as young person social capital affect the transition to adulthood outcomes for youth with disabilities (Trainor, 2008). These factors were not measured in the NY sample, and though NLTS2 captured many of them in its survey, these variables were not used in the current analysis for balancing the two groups. However, studies have documented that young person background characteristics (e.g., disability classification, race/ethnicity, gender) that were used in the balancing techniques of propensity score analysis are systematically related to these unobserved factors (Bates & Davis, 2004; Murray, 2003; Newman, 2005; Wagner et al., 2003), and it is likely that balancing the differences in background characteristics could have provided adequate balance across these unobserved characteristics as well. Lacking information on these variables for NY cohort, it is, however, impossible to verify if adequate balance was achieved on these unobserved factors. Furthermore, the underlying principle of using propensity score is to reduce the multidimensional differences between the intervention and control group by computing a unidimensional measure of probability of program participation and controlling for this factor. However, it is possible that some multidimensional problems could persist causing some misclassification due to the interconnected nature of the various service-receipt and demographic variables. Based on post hoc tests, there were no problems of multicollinearity in the final models in this study.
Another limitation of this analysis is that a large part of the observed effect could be attributed to a renewed focus on transition through NY MTP and that this could have standardized existing practices and motivated establishing of programs focusing on career- and work-based learning leading to program results. One can assume the NS sample may not have such bias as the sampling frame was more extensive for this group and it is likely that students in this cohort may belong to schools and school districts that may not have had such an intense focus on transition programming. Though this could cause a potential treatment artifact from an experimentation perspective (Box, Hunter, & Hunter, 2005), from a practitioner perspective, this renewed focus on transition strengthening existing components is critical to achieving meaningful transition outcomes for youth within prevailing school environment. A more longitudinal follow-up and longer duration of NY MTP implementation could have provided important data to examine such effect and study implementation-related factors. Lack of funding for the project led to its premature termination of financial support for MTP program and activities. Sustainability of NY MTP-like programs, especially in the current fiscal environments of austerity, needs to be systematically studied for further transition to adulthood strategies and program development.
It is also important to note that the outcomes of the NS and NY samples were observed during different time periods. The outcomes for NY students were observed in 2009, and the outcomes for NS sample were observed in 2004–2005. This should potentially affect the difference in outcomes due to differing macroeconomic conditions. However, the outcome observed for NY were during the latter part of the Great Recession that began in 2007, and theoretically, this should have negatively affected postsecondary outcomes for NY students. The higher program impact on postsecondary outcomes for NY compared with NS is, therefore, a conservative estimate of the program effects considering prevailing macroeconomic conditions. However, NY State macroeconomic conditions and access to resources for transition programming could be very different compared with other states and caution must be observed in examining generalizability of program effects. Further research on examining the impact of such macroeconomic conditions on transition outcomes is necessary for practitioners to choose among contextually relevant leading practices.
It is necessary that future transition to adulthood program evaluation research explore the impact of a mixture of transition to adulthood programming strategies within the school’s contextual environment using more rigorous designs of random assignment of schools to intervention groups and adopt rigorous research designs. One such design that has potential for its application include orthogonal design used often in industrial research and now utilized in health care (Zurovac & Brown, 2012). Though a detailed discussion of such a design may be out of scope in the context of this study, it is important to note that orthogonal designs may enable a simultaneous examination of a mix of transition to adulthood programming strategies and their individual components. Furthermore, transition to adulthood program evaluation studies must also utilize the framework of structural equations modeling to examine the critical pathways through which program effects are mediated (Adedokun, Childress, & Burgess, 2011).
Implications for Practice
The findings of mediation analysis have important implications for practitioners, policy makers, and funding agencies, who in some instances have a narrower proclivity in transition programming and focus on solitary programming elements (e.g., enhancing career development activities, providing work experiences for in-school youth and young adults with disabilities, or engendering interagency collaborations).
Model demonstration programs, such as the Youth Transition Program (YTP), have built similar approaches, as NY MTP, of focusing on multiple strategies for achieving higher postsecondary outcomes for youth and young adults with disabilities (Benz, Lindstorm, & Yovanoff, 2000). Furthermore, the Taxonomy for Transition Programming (Kohler, 1996) offers several theoretically informed program development strategies for schools and school districts to develop transition programs for youth with disabilities. However, schools and school districts need to choose strategies that are contextually feasible and relevant to the student needs through a process of mutual adaptation (Lehman, Clark, Bullis, Rinkin, & Castellanos, 2002) and should not simplistically interpret program fidelity as an accurate replication of program elements developed by the inventors of such models. The NY MTP represents a process where schools and school districts adopted a recommended set of strategies within the confines of their contextual environments meeting their unique needs leading to improved outcomes for youth and young adults. Despite variations in programming induced by this process of adaptation, the impact of NY program was positive. These findings indicate that while it is important to strive for high fidelity to principles underpinning the developmental theories in transition to adulthood, fidelity to structural aspects of model programs may not be feasible in a given contextual environment and may not yield the intended program results theorized within model demonstration programs.
Another noteworthy finding of this research is that about one fifth of the direct effect of NY MTP on positive postsecondary outcome was mediated by the fact that greater proportion of NY students received services from the community-based providers of services. Majority of the NY MTP collaborated with these service providers not only as a best practice but also as their way of remaining compliant with the requirements for program funding. Furthermore, the specific impetus of NY MTP for partnering and working closely with the VR offices for ensuring appropriate referrals for students as well as, in some instances, supporting VR office in the process of eligibility determination for referred students, has had significant contributions toward student’s continued postsecondary engagement in VR training and employment. The latter is an important finding in the context that despite necessary policies within the IDEA, as well as the Rehabilitation Act encouraging school-VR collaboration, the collaboration between school systems and VR remains to be tenuous at best (Benz, Lindstorm, & Latta, 1999; The Study Group, 2007). The NY MTP, by mandating this collaboration and providing program resources to schools, engendered this valuable partnership leading to positive postsecondary outcomes for youth with disabilities in NY State. These findings carry substantial implications for policy makers and advocates for exploring ways to strengthen this partnership of two human capital-building institutions (i.e., the schools and VR) for ensuring productive engagement for youth as they transition to adult lives of living independently in their communities.
Footnotes
Authors’ Note
This manuscript was prepared by the Employment and Disability Institute at Cornell University with support from the New York Makes Work Pay initiative, a Comprehensive Employment Services Medicaid Infrastructure Grant funded by the U.S. Department of Health and Human Services, Centers for Medicare and Medicaid Services to the New York State Office of Mental Health.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was conducted by Cornell University’s Employment and Disability under sponsorship from the U.S. Department of Health and Human Services, Center for Medicare and Medicaid Services to the New York State Office of Mental Health for a Comprehensive Employment Systems Medicaid Buy-In Grant.
