Outcomes of Students With Disabilities After Exiting From High School: A Study of Education Data Use and Predictive Analytics

Abstract

We conducted a study of predictive analytics (PA) applied to state data on post-school outcomes (PSO) of exited high-school students with disabilities (SWD). Data analyses with machine learning Random Forest algorithm and multilevel Bayesian ordered logistic regression produced two key findings. One, Random Forest models were accurate in predicting PSO. Two, Bayesian models found high-school graduation was the strongest predictor of higher education and reliably predicted the specific type of outcome relative to other outcomes. Limitations of this study are the data source and small number of predictors. Implications of the study for researchers and educators are discussed in conclusion.

Keywords

post-school outcomes high-school students with disabilities predictive analytics machine learning Bayesian statistics

Introduction

In the United States, special-education research has historically had an in-school focus, in areas such as science and reading instruction. In the most recent decade, another area has garnered increased attention, the post-school outcomes (PSO) of students with disabilities (SWD) (see Lipscomb et al., 2017). At the same time, newer technology has produced new tools, such as predictive modeling with machine learning (see Nassif et al., 2016), to analyze large administrative data, such as PSO, to support decision making (see Larkan-Skinner & Shedd, 2021). Utilizing these data and analytic tools is especially important for educators in closely examining the PSO of SWD (Mandinach, 2012), who continue to experience significant barriers to and challenges in successfully transitioning from high school to young-adult life (Liu et al., 2018). Thus, the focus of this study was to utilize these new tools to analyze the PSO of exited high-school SWD, to assess the modeling performance of these tools, and examine the implications of their utilization for state and local educators and administrators.

Literature Review

Post-School Outcomes

Socioeconomic disparities between people with disabilities and their peers without disabilities have been well documented. In April 2022, the U.S. Department of Labor Office of Disability Employment Policy reported that the unemployment rates of people with and without disabilities aged 16 and older were 8.3% and 3.1%, respectively. Putting those rates into a pre-pandemic historical context, in November 2019 the unemployment rates of people with and without disabilities aged 16 and older were 6.9% and 3.2%, respectively (U.S. Department of Labor, 2019). Even after accounting for the level of educational attainment, people with disabilities had higher unemployment rates (U.S. Department of Labor—Office of Disability Employment Policy, 2022). Moreover, among all people with lower rates of educational attainment and employment regardless of age group, the poverty rate remains higher for people with disabilities (Yin et al., 2014). These employment disparities can also be seen among the younger groups of workers when the data are disaggregated. The unemployment rates of people with and without disabilities aged 16–19 years were 21.1% and 11.4%, respectively. The unemployment rates of people with and without disabilities aged 20–24 were 16.5% and 8.7%, respectively.

The Individuals with Disabilities Education Act, 2004; P.L. 108–446) was enacted to address the educational needs of SWD. Since 2008, one of the requirements of this law is that states have to collect annual data of exited high school SWD, Indicator B14—otherwise known as PSO. This measure refers to the number of SWD in the state who exited high school, had an Individualized Education Program (IEP) when they exited, and within a year of exit were (a) enrolled in higher education, (b) employed in a competitive integrated setting, (c) enrolled in some other post-secondary education or training program, or (d) employed in some other (not competitive) employment (see Individuals with Disabilities Education Act, 2004, 20 U.S.C. 1416[a][3][B]). The exiters who meet criteria for one of these outcomes are counted as being “engaged” in PSO; otherwise they are counted as “not engaged.”

Over the last decade, research in PSO has notably advanced since the seminal work of Test et al. (2009), which identified 16 specific predictors of PSO. Eaves et al. (2012) advanced that work, creating and validating a survey to empirically connect high-school transition services to PSO. Haber et al. (2016) further advanced that work, conducting a meta-analysis on the 16 predictors; and their findings indicated some variance in employment, education, and independent living outcomes based on different in-school predictors and demographic characteristics. Burnes et al. (2018) found that high-school SWD being involved in their IEP, interacting with others (i.e., social skills development), and receiving support from their community were also associated with positive PSO.

Much of the peer-reviewed studies in the PSO research literature have utilized data from the National Longitudinal Transition Study (i.e., NLTS-1, NLTS-2, NLTS-2012) (see Blackorby & Wagner, 1996; Newman et al., 2011), which have been conducted since the 1990s. Over those many years, studies utilizing the NLTS data have consistently reported gender and graduation from high school as significant predictors of post-high school competitive employment and education (e.g., Aud et al., 2010). Carter et al. (2013) also reported that among those with severe disabilities, having paid employment experience during high school, parent expectations, and social skills were significant predictors of post-high school employment.

There are fewer peer-reviewed PSO studies in the research literature that do not utilize NLTS or other nationally-representative samples. A small number of these studies include state-level PSO. Across several states and school districts, McConnell et al. (2015) reported that “student GPA and the percentage of time instruction received in the general education setting cannot indicate a student has all skills needed after high school for employment and further education…more is needed for college and career success” (p. 334). In a Wisconsin pilot study, interventions involving school-led coaching led to increased work experiences in high school, which predicted post-high school paid employment for young adults with intellectual and developmental disabilities (Molfenter et al., 2017). In a study of educators in Indiana, Sprunger et al. (2018) recommended that in order to improve PSO, special-education teachers have to incorporate both academic and work components into the high-school transition plans of SWD, and to receive regular professional development that is focused on transition programming.

The latest and potentially most important development in the PSO literature has been research regarding high-school STEM for SWD and their post-high school outcomes in STEM education and career (see Hwang & Taylor, 2016). This research is also occurring amidst the passage of the STEM Education Act (2015), which places federal priority on STEM education and recent government projections of STEM occupations growing at a faster rate than non-STEM occupations and paying higher wages/salaries over the coming decades (see Bureau of Labor Statistics, 2020). Also, some research indicates that career-technical education and STEM enrichment efforts (e.g., summer, after school programs) in high school may lead to positive PSO for SWD in STEM (Falkenheim et al., 2017; Plasman & Gottfried, 2018). Despite these efforts, however, disparities in STEM access and outcomes between students with and without disabilities remain a persistent problem (National Science Foundation, 2021). More importantly, what is missing from the PSO literature is any federal or state-level study of different types of outcomes (e.g., college, career training, full-time job) analyzed concurrently, rather than as several single-level or single-outcome analyses.

Utilizing Data and New Analytics

Beyond meeting a federal requirement, states can utilize their own annual PSO data to drive and/or support policies, programs, and practices across their local schools and districts (see Gingerich & Crane, 2021). As part of best-practices in data utilization in education decision making, some researchers (e.g., Schildkamp, 2019) have recommended that educators access and utilize more than just assessment data, such as state administrative (e.g., PSO) data. Other researchers have asserted that “decision makers must interpret the data to make informed decisions about how to effectively support students” (Wilcox et al., 2021, p. 2) and called for improving data literacy among educators.

As an increasingly popular approach to data analysis, predictive analytics (PA) is generally described as “statistical or machine learning methods to make predictions about future or unknown outcomes” (Abbasi et al., 2015, p. 35). Machine learning is generally referred to or described as a set of algorithms that use statistics and probabilities in a process of “learning” from the data in order to find patterns and relationships among variables (Meserole, 2018). In recent years, PA have become popularized in mainstream culture, through television shows such as “Numbers” and movies such as “Moneyball.” (see Banerjee, 2018). In research, PA have become more widely used as advances in computing power and data storage have enabled most home computers to run complex algorithms to analyze data (Zhang, 2020). Much of the research application of PA occurs in public health and medical research, for example, to know who may be at greater risk for certain health or medical conditions (see Yperman et al., 2020) and/or to drive targeted interventions and effective treatments (e.g., Desai et al., 2020; Panesar et al., 2019). In these studies, PA application also often includes the use of historical data (see Brown et al., 2015).

Compared to other fields, the application of PA is comparatively newer to education. These studies have focused on assessing the risk of students falling behind academically or dropping out of college (e.g., Herodotou et al., 2019; Wagner & Longanacker, 2016) and K-12 schools (e.g., Porter & Balu, 2016). These studies, however, have not focused on SWD. Studies applying PA in special education and disability/rehabilitation research have centered on the assessment or diagnosis of disability conditions such as autism (e.g., Maenner et al., 2016), multiple sclerosis (e.g., Flauzino et al., 2019), and dyslexia (e.g., Rello et al., 2020). What is missing from the literature are studies regarding PSO of SWD with data national or state level data utilizing PA, specifically machine learning and Bayesian multilevel modeling. These are powerful new analytic tools that are widely used in other fields to support decision makers.

The Current Study

This study seeks to fill two gaps in the PSO research literature. One, educators at the state and local levels need to know whether the predictors of PSO identified through previous studies are also observed when analyzing state-level PSO data. While the studies in the literature have utilized nationally-representative data (i.e., NLTS), local educators and state administrators need real-time and/or annual data of their own populations of SWD and post-high school engagement in further education/training and/or employment after exiting high school. This study utilized state-level, annual PSO data from recent years, taking into account specific school and district contexts in the data analyses. Two, while studies in the PSO research literature have identified variables with statistically significant associations to outcomes, they are missing important aspects of PA—utilizing machine learning and assessing predictive accuracy. In order for local educators and state administrators to reliably use PA, they need to know how accurately these models predict which SWD would and would not become engaged in PSO after exiting from high school. Filling these two gaps in research knowledge is critical for states’ PSO data utilization and effective PA model development to support policies, programs, and practices for SWD. Thus, based on these two knowledge gaps in the literature, this study was framed around two a priori research questions: (1) How did the models of PSO engagement differ in terms of predictive performance across time? (2) Were there differences in the significant predictors of the PSO categories across time?

Method

Data Collection

We received 5 years of de-identified PSO data from a northwest state. The data years represented the exit year of SWD from high school; the outcome data are collected a year after exit. Because the state agreed to provide only de-identified data, other information that the authors would have liked to utilize in their data analyses could not be included.

The frequency counts of the exited high-school SWD and the number of schools and districts in the analyses were tabulated for each year. In 2014, there were N = 2196 exiters from 311 schools and 148 districts. In 2015, there were N = 2937 exiters from 336 schools and 158 districts. In 2016, there were N = 2950 exiters from 318 schools and 150 districts. In 2017, there were N = 3257 exiters from 312 schools and 141 districts. In 2018, there were N = 3102 exiters from 309 schools and 140 districts. The frequency counts differed each year due to annual differences in the number of exiters and the annual PSO response rates. The response rates of the exited high-school SWD for these years ranged from 76% to 79%, which is higher than what is typically observed for this population (i.e., young adults with disabilities) using this method of data collection (i.e., statewide census) (see Dillman et al., 2014).

The state PSO data included one binary outcome variable, “post-school engagement” (i.e., engaged, not engaged in PSO). This outcome is defined by law (Individuals with Disabilities Education Act, 2004) as SWD who, during their first year after high-school exit, had been: (a) enrolled in higher education, (b) competitively employed, (c) enrolled in some other post-secondary education or training program, or (d) employed in some other noncompetitive employment (20 U.S.C. 1416[a][3][B]). Exiters are counted as “engaged” in PSO if they meet any of the four criteria above; otherwise they are counted as “not engaged.” For those who meet criteria for more than one outcome, they are still only counted once in the highest ordered category, higher education, followed by competitive employment, some other postsecondary education or training, and some other employment (Individuals with Disabilities Education Act, 2004). In this study, the state coded 1 if exiters had engaged in PSO (i.e., reference), and 0 (comparator) if they had not engaged in PSO.

The PSO data included five predictors: gender, ethnicity, disability category, least restrictive environment (LRE), and high-school exit status. Research has shown these variables predict PSO (e.g., Eaves et al., 2012; Haber et al., 2016). In this study, the gender predictor was coded with male as the reference (coded as 1) and female as the comparator (coded as 0). Ethnicity was coded with Caucasian/white as the reference and Students of Color as the comparator. Disability was coded with specific learning disability (SLD) as the reference and the other disability categories as the comparator. LRE was coded with the amount of time spent in a general education classroom settings 80% or more per instructional day as the reference and 79% or less per instructional day as the comparator. High-school exit status was coded with graduating from high school as the reference and not graduating as the comparator.

The state data did not contain missing or impossible values. For a couple of predictors, disability category (10 categories) and ethnicity (7 categories), there were little to no data for most of their categories in most schools and districts. These very low counts were concerning, and in order to ensure statistical modeling would produce reliable estimates and an admissible solution (see Kline, 2005), these two predictors were recoded into binary variables. This meant that we coded students with a SLD as the reference and the other disability groups as the comparator. Students with SLD were not only the largest group of SWD in this state but they are also the largest group of high-school SWD in the U.S. served each year in special education (National Center for Education Statistics, 2020). For the ethnicity predictor, Caucasian/white students were coded as the reference and Students of Color as the comparator to reduce problems in model/parameter identifiability (see Gelman & Hill, 2007). Moreover, given the complexity of PA models, our end goal was to be as clear as possible in the interpretations of results for educational stakeholders in describing how these models could be subsequently utilized and/or further refined (see DeCoster et al., 2009).

Data Analysis

All data analyses were conducted using the Stata 17 statistics software (StataCorp, 2021). The decision rule for judging statistical significance of model testing outcomes was set a priori at p < 0.05. Data analyses were conducted in three connected sequential steps: (a) analyses of PSO engagement with machine learning Random Forest algorithm, (b) analyses of Random Forest model predictive performance with the Receiver Operating Characteristic (ROC) curve, and (c) analyses of the outcome types (described in “Data Collection” above) with Bayesian ordered multilevel logistic regression.

The three PA steps taken in the data analysis were necessary to address the knowledge gaps we identified in the PSO literature and to answer the two research questions in this study. The steps were also intentionally sequenced. First, we analyzed whether exited high-school SWD were engaged in PSO, which would represent the states’ main legal reporting requirement for PSO each year. Second, we assessed model predictive performance, which could empirically establish the potential usefulness and usability of PA engagement models for state and local educators. Third, in addition to knowing whether an exited SWD was engaged in PSO, we also analyzed the relationship of the same PSO engagement predictors and each outcome type, which could provide an understanding of why any specific predictor may have been strongly associated with one outcome type but not with another. The three PA steps are each described further.

Machine Learning

The type of machine learning algorithm utilized in this study, “Random Forest” (see Breiman, 2001), is based on an ensemble decision-tree approach and is particularly useful for predicting binary outcomes (Cutler et al., 2011). The algorithm randomly separates data into training and validation samples, refines its predictive modeling on the former and finished modeling on the latter through thousands of iterations of random samples. Thus, the first main result of interest in Radom Forest models is the “out of bag” (OOB) error (i.e., abbreviating “bootstrap aggregating” as “bagging”), indicating the degree of model predictive error out of the training sample iterations. The second main result of interest is the final error, indicating the degree of predictive error out of the validation sample. The difference between the OOB error rate and the final error rate indicates the degree of “learning” by the algorithm (Breiman, 1996). The general expectation is that the predictive algorithm improves or “learns” through this process, and improves model predictive performance (e.g., accuracy in binary-outcome predictions) from the training to validation modeling stages (Couronné et al., 2018).

The specific reason for utilizing the Random Forest model in this study was to determine whether a machine learning algorithm could perform well in model predictive accuracy and be stable across several recent years of moderately-sized state PSO data and predictors. That knowledge would fill a key gap in the PSO literature, as this study would be the first to use machine learning to analyze the socioeconomic outcomes of young adults with disabilities who had recently exited high school. The Random Forest analysis also produced the first part of the answer to the first research question in this study.

Receiver Operating Characteristic Curve

This next PA step in the data analysis provided the other part of the answer to the first research question. We assessed the predictive performances of the Random Forest models for PSO engagement using the ROC Curve analysis. This is a commonly used measure with two key related indices, Sensitivity and Specificity (Metz, 1978). Sensitivity refers to the proportion of people predicted to experience the desired outcome (e.g., engaged in PSO), out of the total number who did in fact experience that outcome. Specificity refers to the proportion of people predicted not to experience the desired outcome (e.g., not engaged in PSO), out of the total number who did not in fact experience that outcome. These two indices also provide important information about the rates of true/false positives, and true/false negatives in these binary predictive models.

The Sensitivity and Specificity indices constitute integral parts of the “area under the curve” or AUC (i.e., two-dimensional area under the ROC Curve), a measure of a binary model’s overall predictive performance (Hosmer et al., 2013; Mason & Graham, 2002). AUC values range from 0 to 1, with 0 indicating 100% inaccurate model prediction and 1 indicating 100% accurate prediction. Although the importance of AUC varies by field (e.g., medicine, business), an AUC value of “0.7 to 0.8 is considered acceptable, 0.8 to 0.9 is considered excellent, and more than 0.9 is considered outstanding” (Mandrekar, 2010, p. 1316).

Bayesian Multilevel Ordered Logistic Regression

The answer to the second research question was produced in this third and final PA step, Bayesian multilevel ordered logistic regression. All Bayesian models are based on the Bayes Theorem of conditional probabilities (Bolstad, 2007; Gelman et al., 2004). In hypothesis testing, Bayesian statistics models the probability of a hypothesis being “true” given the observed data (Stone, 2013). The fundamental and unique aspect of Bayesian statistics is the use of prior information (i.e., “priors”), such as the mean of intercepts or SD of slope parameters (see Van de Schoot et al., 2021). This information (e.g., changes in disease progression) can be derived from previous experiments or the research literature, and then combined with the probability distribution of the newly collected data to produce a new posterior probability distribution, which updates our knowledge of an outcome (e.g., likely prognosis for healthy recovery). The general recommendation is to avoid selecting completely uninformative or completely informative priors (Gelman & Hill, 2007), an especially important consideration for multilevel models given their complexity. Therefore, we chose hierarchical priors, which are between those two types of priors and appropriate given the study design and type of data.

Bayesian statistics also utilizes “Markov Chain Monte Carlo” (MCMC), a large class of algorithms that estimate the posterior distribution of parameters (see Hastings, 1970; Kruschke, 2011). Monte Carlo is a process by which a large number of random samples are drawn from a particular distribution; and these simulations—numbering in the thousands—enable an efficient way to approximate parameters. Markov Chain is a process by which those random samples are sequentially generated, whereby a sample draw depends on the one preceding it and not any other (i.e., Markov property) (Van Ravenzwaaij et al., 2018). A key feature of MCMC is that “the approximate distributions are improved at each step in the simulation, in the sense of converging to the target distribution” (Gelman & Hill, 2007, p. 409). In this study, we used a type of MCMC algorithm called Random-Walk Metropolis-Hastings.

The “ordered” part of the Bayesian modeling begins with an outcome variable (i.e., PSO) containing several categories in a meaningful order, such as lowest to highest or least important to most important (Hedeker, 2015). While similar to a multinomial (i.e., unordered) logistic regression (see Agresti, 2007), the key distinction is that the ordered regression does not assume equal distances, either conceptually or in measurement between the outcome categories (Bauer & Sterba, 2011). The ordered regression is also more parsimonious than fitting the data to several separate models for each individual outcome category and then synthesizing model results post-hoc. Thus, in this study, based on the lowest to highest order of outcome categories (see Post School Outcomes section above), the outcome variable was coded from 0 to 4: (0) no PSO engagement, (1) employed in some other non-competitive employment, (2) enrolled in some other post-secondary education or training program, (3) employed in a competitive integrated setting, and (4) enrolled in higher education (see Individuals with Disabilities Education Act, 2004, 20 U.S.C. 1416[a][3][B]). The analysis was specified as a three-level model with mixed effects (see Austin & Merlo, 2017; Hox, 2002). The fixed effects were each of the five predictors at level-1 (i.e., student), gender, ethnicity, LRE, disability category, and high-school exit status. The interaction effect was gender-by-disability, which was specified in the model based on the research literature. No predictors were available in the state’s data at level-2 (i.e., school) or level-3 (i.e., district).

`The main outputs for our Bayesian multilevel ordered logistic regression included the odds ratios, estimated posterior mean, 95% Credible Interval (95% C.I.), Monte-Carlo standard error (MCSE), and the mean cut points. The odds ratio is an indication, expressed in terms of likelihood, of the magnitude of each predictor’s unique (not shared) effect on the outcome. The 95% Credible Interval assumes that the interval (i.e., Lower − and Upper +) is fixed and the estimated posterior (i.e., mean) parameter is random. This is not to be confused with the more familiar statistical output “95% Confidence Interval” (although it is also abbreviated as “95% C.I.”), which assumes the interval is random and the estimated posterior parameter is fixed. The interpretation of the Credible Interval is also different. It indicates that there is a 95% probability the “true” parameter lies within that interval given the observed data (e.g., PSO state data from 2014 to 2018). The MCSE is an indicator of the degree of imprecision in the Monte-Carlo sampling and estimation of the posterior mean (Gong & Flegal, 2015).

The mean cut points are the mean values representing the estimated threshold at or below which individuals would be classified at the corresponding categories of the outcome variable. In the specified Bayesian model, there were four ordered cut points. Cut Point 1 differentiated “no PSO engagement” (i.e., the lowest ordered outcome) from the other four outcomes. Cut Point 2 differentiated the “no PSO” and “some other employment” outcomes from the other three. Cut Point 3 differentiated “no PSO,” “some other employment,” and “other education or training” outcomes from the other two. Cut Point 4 differentiated the “no PSO,” “some other employment,” “other education or training,” and “competitive employment” outcomes from “higher education,” the highest ordered outcome. With the posterior distribution of parameter estimates, any of the exiters in the 5 years (2014–18) of PSO data could be located within one of these thresholds indicating their most likely outcome. In other words, a model of engagement with a particular outcome could be derived for all exiters, providing information about how likely these SWD are to become engaged in any particular outcome (or not engaged) 1 year after high school, and which predictors (described above) were the most strongly associated with that outcome.

Results

The analyses of data across the 5 years produced results for (a) demographic frequencies (%) of the exited high-school SWD and their PSO, (b) statistical modeling outcomes to answer Research Question 1, and (c) statistical modeling outcomes to answer Research Question 2. Each of these results are described below in turn.

Demographic Characteristics

The demographic characteristics of the exited high-school SWD are presented in Table 1. Across the 5 years of data, most of the exiters were male (64.3%–66.0%), Caucasian/white (62.7%–68.9%), and high-school graduates (66.7%–77.5%). In addition, the largest student-disability group among the exiters were those who had identified in school as having a SLD (44.3%–48.6%). The plurality of exiters who were engaged 1 year after high-school were in competitive employment (32.4%–40.2%), followed by higher education (22.4%–25.2%), some other noncompetitive employment (8.6%–12.4%), and some other postsecondary education or training (4.0%–6.3%). The overall percentage of all exiters who had been engaged in any type of PSO ranged from 71.3% to 75.2%.

Table 1.

Demographic Characteristics (%) of Exited High-School Students With Disabilities.

Characteristics	High-school exit years
Characteristics	2014	2015	2016	2017	2018
Gender
Female	35.6	34.8	35.7	34.0	34.8
Male	64.4	65.2	64.3	66.0	65.2
Race/ethnicity
Hispanic/Latinx	18.9	21.6	21.5	23.2	23.2
American Indian/Alaska Native	2.3	2.6	2.4	2.3	2.4
Black/African American	2.9	3.1	4.2	3.8	3.5
White/Caucasian	68.9	66.0	64.3	63.3	62.7
Asian	2.6	1.9	2.2	2.1	1.7
Pacific Islander^a	—	0.5	0.5	0.6	0.7
Multiple	4.4	4.4	4.9	4.7	5.7
Exit reason
Graduated	66.7	70.3	70.9	75.3	77.5
Received certificate	6.6	7.0	8.0	7.6	7.8
Dropped out	21.2	18.3	16.4	13.7	11.2
Aged out	5.5	4.4	4.7	3.4	3.5
State	0.5	0.2	0.3	—	—
Disability category
Autism spectrum disorder	11.9	10.9	11.3	12.1	12.5
Communication disorder	3.2	3.3	3.4	3.0	3.7
Emotional disturbance	7.7	7.2	7.9	8.2	8.5
Hearing impairment	1.0	1.2	0.8	1.1	1.1
Intellectual disability	7.8	7.5	8.0	7.7	6.9
Orthopedic impairment	0.8	1.2	1.3	0.7	1.0
Other health impairment	17.9	19.3	19.1	21.4	20.8
Specific learning disability	48.6	48.2	47.4	44.8	44.3
Traumatic brain injury	0.7	0.9	0.5	0.7	0.8
Visual impairment	0.5	0.4	0.3	0.3	0.5
Post-school outcome
Higher education	22.4	24.4	24.5	22.8	25.2
Competitive employment	34.8	35.9	37.0	40.2	32.4
Some other postsecondary education or training	5.5	4.5	6.3	4.0	5.3
Employed in some other noncompetitive employment	8.6	9.1	7.2	7.5	12.4
Not engaged in PSO	28.7	26.1	25.0	25.5	24.8

^aAsian/Pacific Islander category was disaggregated starting in 2015–16.

Research Question 1

The first research question asked, “How did the models of PSO engagement differ in terms of predictive performance across time?” Referring to the statistical output in Table 2, the Random Forest machine learning models produced high levels of predictive accuracy based on the AUC values, which ranged from 0.84 in 2017, to 0.88 in 2014. The AUC values were also found to have been reliably estimated in these models based on the small standard errors (0.01). To put these values into context, the research literature describes the performances of predictive models with AUC values at or above 0.80 as “excellent” (Mandrekar, 2010, p. 1316).

Table 2.

Assessment of Machine Learning Random Forest Model Predictive Performance.

Model	Year
Model	2014	2015	2016	2017	2018
Out-of-bag error	0.31	0.28	0.25	0.29	0.27
Final error rate	0.09	0.09	0.08	0.10	0.09
Area under the curve	0.88	0.86	0.87	0.84	0.85
95% CI	0.86, 0.90	0.84, 0.87	0.85, 0.89	0.82, 0.85	0.83, 0.86
Standard error	0.01	0.01	0.01	0.01	0.01

Because the 2014 Random Forest model produced the most accurate predictive model across the 5 years, we analyzed its Sensitivity and Specificity (see Table3). Sensitivity indicates the “true positive” predictive result, and Specificity indicates the “true negative” predictive result (see Method section—ROC Curve). In the 2014 model, Sensitivity was 96%, representing the proportion of exited high-school SWD who had been predicted to be engaged in PSO who were in fact engaged in PSO (1506/1566 = 0.96). Specificity was 80%, representing the proportion of exited high-school SWD who had been predicted to be not engaged in PSO who were in fact not engaged in PSO (503/630 = 0.80). Thus, taken together, there were 2009 (1506 + 503) exited high-school SWD who had been correctly predicted in either outcome (i.e., PSO engaged or not engaged). That total sum comprised 91% (2009/2196) of all exiters in 2014, and 9% “Final Error Rate” for this Random Forest model.

Table 3.

Random Forest Algorithm: Actual Versus Predicted PSO Engagement Frequencies (n) in 2014.

Predicted PSO engagement	Actual PSO engagement		Row total
Predicted PSO engagement	Yes	No	Row total
Yes	1506	127	1633
No	60	503	563
Column total	1566	630	2196

Research Question 2

The second research question asked, “Were there differences in the significant predictors of the PSO categories across time?” In the first part of the answer to this question, the strongest predictor across all 5 years was high school exit status (see Tables 4 –8), with odds ratios ranging from 3.81 to 5.25. These values indicated that SWD who graduated high school were between 3.81 and 5.25 times as likely as SWD who did not graduate to be engaged in the higher education outcome 1 year after exiting high school, compared to the other four outcomes for SWD having the same random effects and controlling for the effects of the other predictors in the model (see Method section for details on interpreting odds ratio values). The second strongest predictor across all 5 years was LRE, with odd ratios ranging from 1.94 to 2.29. This indicated that SWD who had spent 80% or more of their time per instructional day in general education classrooms during high school were 1.94–2.29 times are likely as SWD who spent 79% or less time per instructional day in general education to be engaged in the higher education outcome 1 year after exiting high school, compared to the other four outcomes for SWD having the same random effects and controlling for the effects of the other predictors in the model.

Table 4.

Bayesian Multilevel Ordered Logistic Regression Model 2014.

Variables	Model parameters
	Odds ratio	Mean	SD	MCSE	Median	95% credible interval
	Odds ratio	Mean	SD	MCSE	Median	Lower (−)	Upper(+)
Least restrictive environment	2.0667	0.7207	0.1020	0.0063	0.7208	0.5308	0.9104
High school exit status	5.2467	1.6526	0.1000	0.0082	1.6483	1.4654	1.8521
Disability category	1.3124	0.2680	0.0872	0.0073	0.2656	0.0925	0.4401
Gender	1.1540	0.1398	0.0831	0.0071	0.1358	−.0208	0.3052
Ethnicity	0.7875	−.2433	0.0937	0.0071	−.2462	−.4195	−.0532
Thresholds		Mean	SD	MCSE	Median	95% credible interval
Thresholds		Mean	SD	MCSE	Median	Lower (−)	Upper (+)
Cut point 1		0.4985	0.1377	0.0167	0.4883	0.2570	0.7945
Cut point 2		0.9991	0.1378	0.0163	0.9890	0.7556	1.2905
Cut point 3		1.3057	0.1388	0.0158	1.2929	1.0588	1.6001
Cut point 4		3.2860	0.1491	0.0159	3.2798	3.0252	3.6089

Table 5.

Bayesian Multilevel Ordered Logistic Regression Model 2015.

Variables	Model parameters
	Odds ratio	Mean	SD	MCSE	Median	95% credible interval
	Odds ratio	Mean	SD	MCSE	Median	Lower (−)	Upper(+)
Least restrictive environment	2.1377	0.7561	0.0851	0.0087	0.7588	0.5892	0.9161
High school exit status	4.7739	1.5594	0.0868	0.0064	1.5598	1.3914	1.7309
Disability category	1.3219	0.2766	0.0705	0.0064	0.2799	0.1389	0.4124
Gender	1.1065	0.0983	0.0759	0.0061	0.1018	−.0477	0.2505
Ethnicity	1.0758	0.0701	0.0760	0.0061	0.0730	−.0861	0.2089
Thresholds		Mean	SD	MCSE	Median	95% credible interval
Thresholds		Mean	SD	MCSE	Median	Lower (−)	Upper (+)
Cut point 1		0.6661	0.1235	0.0159	0.6713	0.4287	0.9016
Cut point 2		1.2031	0.1246	0.0163	1.2064	0.9579	1.448
Cut point 3		1.4567	0.1214	0.0153	1.4585	1.2278	1.6928
Cut point 4		3.3962	0.13385	0.0168	3.3957	3.1439	3.6628

Table 6.

Bayesian Multilevel Ordered Logistic Regression Model 2016.

Variables	Model parameters
	Odds ratio	Mean	SD	MCSE	Median	95% credible interval
	Odds ratio	Mean	SD	MCSE	Median		Lower (−)	Upper (+)
Least restrictive environment	2.2890	0.8250	0.0783	0.0065	0.8266	0.6673	0.9739
High school exit status	5.1813	1.6397	0.1034	0.0126	1.6413	1.4408	1.8278
Disability category	1.3398	0.2897	0.0744	0.0071	0.2875	0.1475	0.4422
Gender	1.0199	0.0169	0.0754	0.0060	0.0178	−.1314	0.1624
Ethnicity	1.0183	0.0153	0.0747	0.0074	0.0145	−.1335	0.1578
Thresholds		Mean	SD	MCSE	Median	95% credible interval
Thresholds			SD	MCSE	Median	Lower (−)	Upper (+)
Cut point 1		0.4980	0.1162	0.0135	0.5026	0.2541	0.7260
Cut point 2		0.9654	0.1164	0.0134	0.9695	0.7114	1.1853
Cut point 3		1.3393	0.1157	0.0134	1.3407	1.0879	1.5573
Cut point 4		3.4147	0.1262	0.0150	3.4165	3.1631	3.6599

Table 7.

Bayesian Multilevel Ordered Logistic Regression Model 2017.

Variables	Model parameters
	Odds ratio	Mean	SD	MCSE	Median	95% credible interval
	Odds ratio	Mean	SD	MCSE	Median		Lower (−)	Upper (+)
Least restrictive environment	1.9797	0.6798	0.0788	0.0109	0.6778	0.5222	0.8432
High school exit status	3.8139	1.3348	0.0876	0.0097	1.3291	1.1755	1.5142
Disability category	1.2514	0.2219	0.0695	0.0100	0.2248	0.0814	0.3460
Gender	0.9153	-.0909	0.0692	0.0053	−.0916	−.2291	0.0411
Ethnicity	0.9738	-.0294	0.0753	0.0075	−.0273	−.1807	0.1152
Thresholds		Mean	SD	MCSE	Median	95% credible interval
Thresholds			SD	MCSE	Median	Lower (−)	Upper (+)
Cut point 1		0.1719	0.1210	0.0148	0.1686	−.0575	0.3982
Cut point 2		0.6129	0.1226	0.0153	0.6088	0.3865	0.8570
Cut point 3		0.8335	0.1230	0.0153	0.8309	0.6106	1.0698
Cut point 4		2.9655	0.1279	0.0157	2.9587	2.7142	3.220

Table 8.

Bayesian Multilevel Ordered Logistic Regression Model 2018.

Variables	Model parameters
	Odds ratio	Mean	SD	MCSE	Median	95% credible interval
	Odds ratio	Mean	SD	MCSE	Median		Lower (−)	Upper (+)
Least restrictive environment	1.9442	0.6621	0.0739	0.0072	0.6605	0.5190	0.8056
High school exit status	3.8275	1.3384	0.0868	0.0090	1.3365	1.1642	1.5159
Disability category	1.3245	0.2783	0.0732	0.0095	0.2789	0.1390	0.4190
Gender	1.0218	0.0191	0.0706	0.0069	0.0192	−.1142	0.1564
Ethnicity	0.9538	−.0501	0.0753	0.0050	−.0522	−.2004	0.1003
Thresholds		Mean	SD	MCSE	Median	95% credible interval
Thresholds			SD	MCSE	Median	Lower (−)	Upper (+)
Cut point 1		0.3102	0.1311	0.0160	0.3108	0.0268	0.5750
Cut point 2		1.0197	0.1331	0.0174	1.0201	0.7309	1.2806
Cut point 3		1.2877	0.1328	0.0166	1.2876	1.0126	1.5436
Cut point 4		2.9642	0.1386	0.0168	2.9621	2.6765	3.2310

Adding to the first part of the answer for Research Question 2, we also examined the school and district intercepts (see Table 9), representing the mean log odds of PSO engagement across the 5 years. These are the school/district level log odds when the predictors are at 0 (i.e., the predictors coded as 0, the comparator group). The lowest log odds of PSO engagement for school, 0.2185, occurred in 2015; the highest log odds, 0.3240, occurred in 2014. The lowest log odds of PSO engagement for district, 0.2372, occurred in 2015; the highest log odds, 0.2704, occurred in 2016. Moreover, except in 2015, the effect of school was greater than the effect of district on PSO engagement.

Table 9.

Random Effects for School and District Intercepts 2014 to 2018.

Levels	Model parameters
	Mean	SD	MCSE	Median	95% credible interval
	Mean	SD	MCSE	Median	Lower (−)	Upper(+)
2014
School	0.3240	0.0677	0.0077	0.3154	0.2115	0.4757
District	0.2411	0.0621	0.0043	0.2336	0.1425	0.3805
2015
School	0.2185	0.0471	0.0042	0.2134	0.1380	0.3209
District	0.2372	0.0541	0.0034	0.2332	0.1454	0.3608
2016
School	0.2852	0.0625	0.0065	0.2789	0.1816	0.4236
District	0.2704	0.0644	0.0045	0.2640	0.1677	0.4123
2017
School	0.2537	0.0548	0.0058	0.2496	0.1591	0.3739
District	0.2497	0.0622	0.0049	0.2430	0.1534	0.3982
2018
School	0.2756	0.0618	0.0058	0.2700	0.1746	0.4069
District	0.2375	0.0553	0.0041	0.2313	0.1488	0.3597

The second part of the answer to Research Question 2 produced, perhaps, the most notable finding in this study—using Bayesian statistical modeling to predict the type of outcome for the exited high-school SWD. Among these model results, there were two notable patterns. One, the largest distance (i.e., numerical difference) were between Cut Point 3 and Cut Point 4: 1.98 in 2014, 1.94 in 2015, 2.08 in 2016, 2.13 in 2017, and 1.68 in 2018. This indicated how different “higher education” was as one type of PSO compared to the other types. Two, the smallest distance occurred between Cut Point 2 and Cut Point 3: 0.27 in 2014, 0.22 in 2015, 0.37 in 2016, 0.25 in 2017, and 0.31 in 2018. This indicated that Cut Point 2 (no PSO and some other employment) and Cut Point 3 (no PSO, some other employment, and other education/training) were the least dissimilar among outcomes, and especially compared to the highest ordered outcomes, competitive employment and higher education.

In addition to the distance-differences across outcomes, the threshold values themselves differed from year to year (see Tables 4 –8). In 2014, the values ranged from 0.50 to 3.29. In reference to Cut Point 1, SWD with values of 0.50 or lower would be predicted as not engaged in PSO 1 year after exiting high school. Likewise, for SWD with values of 3.29 or higher would be predicted as engaged in higher education 1 year after exiting high school. SWD with values in between 0.50 and 3.29 would be predicted as engaged in one of the other three outcomes. In 2015, these threshold values ranged from 0.67 to 3.40. In 2016, these values ranged from 0.50 to 3.41. In 2017, these values ranged from 0.17 to 2.97. In 2018, these values ranged from 0.31 to 2.96. These values can be thought of as scores on a latent trait variable. In the output (Tables 4 –9), these values are identified as “Mean”, to specify (a) all of the predictors coded as the comparator group, coded as 0 (e.g., less than 80% per instructional day in general education classrooms, female SWD), (b) fixed effects, and (c) average of random effects (i.e., school- and district-level intercepts) in the model.

Finally, in assessing overall model fit and specification by inspecting the Deviance Information Criterion (DIC), we observed that the best model fit was in 2014, with a DIC of 5707, and the worst model fit was in 2018, with a DIC of 8490. This information criterion does represent a tradeoff between model fit and complexity (see Gelman & Hill, 2007). In addition, across 5 years the Monte Carlo Standard Error (MCSE) for the mean/intercept estimates were also small, indicating a sufficient level of reliability in their posterior estimation.

Discussion

The results of the data analyses not only provide answers to the two research questions but also contribute to the PSO literature and offer potential new tools or approaches to state and local educators for their decision making. In this section, we further elaborate on the results of PA with machine learning and Bayesian modeling, and we also describe the study’s limitations. We conclude this study by describing its implications for education research and practice.

Machine Learning

The first research question focused on the use of machine learning with Random Forest modeling of several recent years of a state’s PSO data. Those results contribute new knowledge to the extant research literature that differs from the current studies in the research literature using data from NLTS, which provide a broadly descriptive and national picture of outcomes for SWD after exiting high-school among student-disability groups (see Blackorby & Wagner, 1996; Liu et al., 2018). This current study takes an important next step. In the U.S., education remains decentralized and mostly managed at the state level, with some autonomy given to local (i.e., district, school) levels. As such, the issue for educators regarding PSO is that much of the important practical decisions for programs, practices, and resourcing have to be made with data of their own students, not with national studies. That is, this study presents, for the first time, an empirical data-analytic approach that researchers and educators could take to examine PSO for an individual state, its local-level Random Forest models and assessing predictive accuracy across several years. With these PA models, states can take the PSO data they collect each year and utilize a set of variables to reliably estimate (e.g., AUC of 80%–90% in this study) exiting high-school SWD that are most likely to be engaged in PSO 1 year after high school. That knowledge could also serve as a valuable resource in educating and supporting current high-school SWD in transition planning for college or career.

As it relates to this study and PA modeling, we acknowledge that there may be misunderstandings or misgivings about its broad application to socioeconomic data. The fact is, PA is not deterministic, and its results should never be considered the “final word” in decision-making for any policies, programs, or practices. It is a tool that needs to be applied carefully and appropriately. There are healthy ongoing debates in research about PA and its appropriate applications. We are also keenly aware of the public’s concerns regarding the over-reliance on technology-heavy and algorithm-driven methods in decision making. There is general agreement across research fields that human judgment remains central, and that these analytic tools should only be used in support of that judgment, not in its place (Chin-Yee & Upshur, 2017). As technology continues to advance, ethical and equitable applications of PA to data (see Palmer & Carpenter-Hubin, 2020) must always be ensured to move these fields forward for everyone.

Bayesian Logistic Regression

The one result in this study that was consistent with findings from other studies in the PSO literature is that high-school graduation was the strongest predictor of SWD engagement in higher education. What this study newly contributes to the literature, however, is that the Bayesian models are also able to predict which type of outcome each SWD is most likely to engage 1 year after high school, relative to the other outcomes. For example, if an exited SWD in 2014 had an overall mean (i.e., “score”) of 3.45, by looking at the corresponding thresholds in Table 4, this clearly met the threshold of the higher education outcome category. Conversely, if that exited SWD in 2014 had a mean of 0.32, this lies within the threshold for the “no PSO” category. On the other hand, if exited SWD have scores that are much closer to the threshold boundary, the interpretations of outcomes would be more nuanced. This would be especially true in the middle categories, such as Cut Point 2 and Cut Point 3, where the differences between threshold values were the smallest in our model results. Overall, when educators at state and local levels have detailed information not only about whether their exited SWD are engaged in any PSO, but also the type of PSO and which set of predictors seem to be driving the outcome, they have more usable information. Their own PSO data (which they collect yearly) and PA can become useful tools in their decision making.

Limitations

There were two limitations in this study. First, the state data we received contained sufficient total numbers of exited high-school SWD across all 5 years, but many schools and districts in the state had small numbers of exiters in their data, or more exiters in one category than in others. In addition, smaller samples (e.g., small numbers of SWD) within many level-2 and level-3 units (i.e., schools and districts) can pose challenges to producing multilevel models that converge to an admissible solution, or models that produce reliable parameter estimates. Grilli and Rampichini (2012) assert that the sample size required for producing reliable estimates in multilevel ordinal models depends on a number of factors, such as the estimation method, cluster variance values, and model complexity. These authors recommend, “In the random intercept case, the estimates are reasonably good with most estimation methods even with 10–15 clusters as long as the average cluster size is at least 10. If the clusters are smaller, more clusters are needed. In the random slope case, the requirement is considerably higher, say 30 clusters of size 30” (p. 6). As described earlier in the Method section, a unique feature of Bayesian statistics is the use of prior information. Thus, some of the challenges with regard to conducting complex multilevel statistical modeling with smaller sample sizes across the school and district levels could be addressed in subsequent studies by utilizing parameter and other distributional information garnered from this study.

The second limitation of this study is the statistical modeling; and it is related to the first limitation. We acknowledge that any statistical model is only as “good” as the limits of the available data being analyzed. This state’s data contained a very small number of predictors due to the state’s need to reduce the identifiability of individual SWD in the PSO data; there certainly could have been some degree of model misspecification as a result. There could have been other variables (not present in these data) that could contribute to exiters becoming engaged in PSO 1 year after high school, for example, individual motivation, access to transportation, and family circumstances. We also recognize that, as this was the first PSO study in the literature to use Bayesian modeling, our selection of “hierarchical priors” in the data analyses could therefore not have been based on previously established studies to identify and select the “right” priors to use. We took a cautious approach, but still could have erred. As Gelman (2007) noted, “All models are wrong, and the purpose of model checking (as we understand it) is not to reject a model but rather to understand the ways in which it does not fit the data” (p. 349).

Implications for Stakeholders

Educators

With an increasing emphasis in U.S. education on using data in decision making for programs, practices, and policies, the application of PA to administrative data, such as PSO of SWD, will become more necessary. Educators can delve into more granular analyses of their PSO data by applying PA modeling approaches from this study to better understand how individual, school, and community factors are connected to PSO across time. Such information could also be shared and utilized at the district and state levels, for example, to identify areas where high-school SWD can obtain additional necessary supports either through their IEP or in community services, mental health, continuing education and job training from local community colleges, leading to better outcomes after exiting high school.

In addition, as state data systems become more complex, the need for real-time actionable information takes on increased significance. Thus the models tested in this study could be considered a template for future model testing, as a starting point for subsequently adding more predictors and addressing this study’s limitations. One possible way to increase the number and types of variables in future analyses is to link a state’s education-agency databases to the relational databases at other state agencies (e.g., health services, workforce/labor) via interagency collaboration. Moreover, the results from Bayesian modeling in this study could be utilized as “priors” (i.e., prior information) in those analyses. Such work could be important for educators to understand how different sets of predictors are more strongly associated with one type of PSO versus another type, and continue to refine their understanding of these analyses with each year’s PSO data. As the economy continues to evolve, the needs of educators, SWD, and families will also have to evolve. This places even greater emphasis on PSO and the need for effective, data-driven transition planning for SWD during high school. For young adults with disabilities already facing obstacles and challenges to post-high school adult life, this need is urgent. For policymakers, supporting educators in the acquisition and application of PA tools is important. They can help to embed these tools and best practices in the everyday work of educators at the district and local levels.

Researchers

The implications of this study for researchers come from the specific modeling techniques we utilized—and within the context of the limitations we noted above. This study was the first in the literature in which PSO data were analyzed using PA with machine learning and ROC for predictive accuracy, and using Bayesian multilevel ordered logistic regression for the different types or categories of outcomes. These models can and should be further refined and tested using state administrative data not only from education but also from state health/human services, labor and employment, and other state agencies. Future studies could also further examine reliability and stability of these models over time, combining state administrative PSO data with other state administrative data. As more states implement longitudinal data systems that link k-12 education data to higher education, employment, and other state data systems, opportunities will expand for researchers to conduct more comprehensive PA modeling of many different predictors over many years. Examining outcomes in the context of students’ in-school experiences, such as participation and completion of career and technical education (CTE), could also further the development of these predictive models. Another important area is to utilize fuller detailed information from students’ IEPs. This would aid researchers in developing models that include the types of special education and related services students are receiving, to the degree they are appropriate, and/or whether the IEP is being implemented as written, or is not being written in such a way as to be able to be effectively implemented.

As described in the implications for educators (above), the Bayesian models developed in this study could also serve as informative “priors” in subsequent state-level PSO research studies. That information could assist researchers in their technical understanding of how well or poorly the predictors within these models perform over time (i.e., empirical testing of reliability and stability of models). This study’s model results could also be used to help researchers develop ways to refine and re-specify models to elaborate where such models were misspecified and how to address such issues. This information could also help researchers further clarify how threshold values near the boundaries of cut points should be understood in terms of probabilities of any of the several PSO categories.

Lastly, although the data analyses in this study were completed using Stata 17 (StataCorp, 2021), it can also be completed using other well-known applications. Commercial options include MPlus, SAS, and SPSS, with varying costs depending on the type of license and features. Open-source options include R, R-Studio, and Python programming platforms (i.e., integrated development environment) such as Jupyter Notebook and Spyder. The authors of this study are not advocating for or recommending the use of any particular program for conducting PA modeling. These programs all provide extensive user support and online resources (e.g., technical specifications and requirements, and programming suggestions) for a range of users, from novice to expert. There are also numerous user-group platforms that offer an array of well-documented, programming advice and step-by step instructions for conducting different PA modeling.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Scott H. Yamamoto

References

Abbasi

Lau

R. Y. K.

Brown

D. E.

(2015). Predicting behavior. IEEE Intelligent Systems, 30(3), 35–43. https://doi.org/10.1109/mis.2015.19

Agresti

(2007). Introduction to Categorical Analysis (2nd ed.). Hoboken, NJ: John C. Wiley and Sons.

Aud

Fox

KewalRamani

(2010). Status and trends in the education of racial and ethnic groups (NCES 2010–015). U.S. Department of Education, National Center for Education Statistics. U.S. Government Printing Office.

Austin

P. C.,

Merlo

(2017). Intermediate and advanced topics in multilevel logistic regression analysis. Statistics in Medicine, 36(20), 3257–3277. https://doi.org/10.1002/sim.7336

Banerjee

(2018). Linear regression: Moneyball – Part 1. Towards Data Science. https://towardsdatascience.com/linear-regression-moneyball-part-1-b93b3b9f5b53

Bauer

D. J.,

Sterba

S. K.

(2011). Fitting multilevel models with ordinal outcomes: Performance of alternative specifications and methods of estimation. Psychological Methods, 16(4), 373–390. https://doi.org/10.1037/a0025813

Blackorby

J.,

Wagner

(1996). Longitudinal postschool outcomes of youth with disabilities: Findings from the national longitudinal transition study. Exceptional Children, 62(5), 399–413. https://doi.org/10.1177/001440299606200502

Bolstad

W. M.

(2007). Introduction to Bayesian statistics (2nd ed.). Wiley.

Breiman

(1996). Bagging predictors. Machine Learning, 24(2), 123–140. https://doi.org/10.1007/bf00058655

10.

Breiman

(2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/a:1010933404324

11.

Brown

D. E.

Abbasi

Lau

R. Y. K.

(2015). Predictive analytics: Predictive modeling at the micro level. IEEE Intelligent Systems, 30(3), 6–8. https://doi.org/10.1109/mis.2015.50

12.

Bureau of Labor Statistics (2020). Employment projections: Employment in STEM occupations. https://www.bls.gov/emp/tables/stem-employment.htm

13.

Burnes

J. J.

Martin

J. E.

Terry

McConnell

A. E.

Hennessey

M. N.

(2018). Predicting post-secondary education and employment outcomes using results from the transition assessment and goal generator. Career Development and Transition for Exceptional Individuals, 41(2), 111–121. https://doi.org/10.1177%2F2165143417705353

14.

Carter

E. W.

Harvey

M. N.

Taylor

J. L.

Gotham

(2013). Connecting youth and young adults with autism spectrum disorders to community life. Psychology in the Schools, 50(9), 888–898. https://doi.org/10.1002/pits

15.

Chin-Yee

B.,

Upshur

(2017). Clinical judgement in the era of big data and predictive analytics. Journal of Evaluation in Clinical Practice, 24(3), 638–645. https://doi.org/10.1111/jep.12852

16.

Couronné

Probst

Boulesteix

A-L.

(2018). Random forest versus logistic regression: A large-scale benchmark experiment. BMC Bioinformatics, 19(270), 1–14. https://doi.org/10.1186/s12859-018-2264-5

17.

Cutler

D. R.

Stevens

J. R.

(2011). Random forests (pp. 1–20). Machine Learning.

18.

DeCoster

Iselin

A. M. R.

Gallucci

(2009). A conceptual and empirical examination of justifications for dichotomization. Psychological Methods, 14(4), 349–366. https://doi.org/10.1037/a0016956

19.

Desai

R. J.

Wang

S. V.

Vaduganathan

Evers

Schneeweiss

(2020). Comparison of machine learning methods with traditional models for use of administrative claims with electronic medical records to predict heart failure outcomes. JAMA Network Open, 3(1), e1918962. http://jamanetwork.com/article.aspx?doi=10.1001/jamanetworkopen.2019.18962

20.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2014). Internet, phone, mail and mixed-mode surveys: The tailored design method (4th ed.). John Wiley & Sons, Inc.

21.

Eaves

R. C.

Rabren

Hall

(2012). The post-school outcomes transition survey: A tool for effective decision making? Career Development for Exceptional Individuals, 38(1), 30–39. https://doi.org/10.1177/1534508412452751

22.

Falkenheim

Burke

Muhlberger

Hale

(2017). Women, minorities, and persons with disabilities in science and engineering: 2017. Special report NSF 17-310. National Science Foundation, National Center for Science and Engineering Statistics. https://www.nsf.gov/statistics/wmpd/

23.

Flauzino

Colado Simão

A. N. C.

de Carvalho Jennings Pereira

W. L.

Alfieri

D. F.

Oliveira

S. R.

Kallaur

A. P.

Lozovoy

M. A. B.

Kaimen-Maciel

D. R.

Maes

Reiche

E. M. V.

(2019). Disability in multiple sclerosis is associated with age and inflammatory, metabolic and oxidative/nitrosative stress biomarkers: Results of multivariate and machine learning procedures. Metabolic Brain Disease, 34(5), 1401–1413. https://doi.org/10.1007/s11011-019-00456-7

24.

Gelman

(2007). Comment: Bayesian checking of the second levels of hierarchical models. Statistical Science, 22(3), 349–352.

25.

Gelman

Carlin

J. B.

Stern

H. S.

Rubin

D. B.

(2004). Bayesian data analysis (2nd ed.). Chapman & Hall/CRC.

26.

Gelman

A.,

Hill

(2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.

27.

Gingerich

J. A.,

Crane

(2021). Transition linkage tool: A systems approach to enhance post-school outcomes. Mathematica. Research funded by U.S. Department of Labor Office of Disability Employment Policy. Contract #1605DC-18-A0020.

28.

Gong

L.,

Flegal

J. M.

(2015). A practical sequential stopping rule for high-dimensional Markov chain Monte Carlo. Journal of Computational and Graphical Statistics, 25(3), 684–700. https://doi.org/10.1080/10618600.2015.1044092

29.

Grilli

L.,

Rampichini

(2012). Multilevel models for ordinal data. In Kennet

Salini

(Eds.), Modern analysis of customer surveys: With applications using R. Wiley.

30.

Haber

M. G.

Mazzotti

V. L.

Mustian

A. L.

Rowe

D. A.

Bartholomew

A. L.

Test

D. W.

Fowler

C. H.

(2016). What works, when, for whom, and with whom: A meta-analytic review of predictors of postsecondary success for students with disabilities. Review of Educational Research, 86(1), 123–162. https://doi.org/10.3102/0034654315583135

31.

Hastings

W. K.

(1970). Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1), 97–109. https://doi.org/10.1093/biomet/57.1.97

32.

Hedeker

(2015). Methods for multilevel ordinal data in prevention science. Prevention Science, 16(7), 997–1006. https://doi.org/10.1007/s11121-014-0495-x

33.

Herodotou

Rienties

Boroowa

Zdrahal

Hlosta

(2019). A large-scale implementation of predictive learning analytics in higher education: The teachers’ role and perspective. Education Technology Research and Development, 67(5), 1273–1306. https://doi.org/10.1007/s11423-019-09685-0

34.

Hosmer

D. W.

Lemeshow

Sturdivant

R. X.

(2013). Applied logistic regression (3rd ed.). John Wiley & Sons.

35.

Hox

J. J.

(2002). Multilevel analysis. Lawrence Erlbaum Associates.

36.

Hwang

J.,

Taylor

J. C.

(2016). Stemming on STEM: A STEM education framework for students with disabilities. Journal of Science Education, 19(1), 39–49. https://doi.org/10.14448/jsesd.09.0003

37.

Individuals with Disabilities Education Act (2004). IDEA 2004, P.L. 108-446, codified as amended at title U.S.C. §20 U.S.C. 1400[d].

38.

Kline

R. B.

(2005). Principles and practices of structural equation modeling (2nd ed.). The Guilford Press.

39.

Kruschke

J. K.

(2011). Doing Bayesian data analysis: A tutorial with R and BUGS. Elsevier, Inc.

40.

Larkan-Skinner

K.,

Shedd

J. M.

(2021). Real-time data predictive analytics: Where does IR fit? New Directions for Institutional Research, 2020(185–186), 11–24. https://doi.org/10.1002/ir.20326

41.

Lipscomb

Haimson

Liu

A. Y.

Burghardt

Johnson

D. R.

Thurlow

M. L.

(2017). Preparing for life after high school: The characteristics and experiences of youth in special education. Findings from the national longitudinal transition study 2012. Volume 2: Comparisons across disability groups: Full report (NCEE 2017–4018). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.

42.

Liu

A. Y.

Lacoe

Lipscomb

Haimson

Johnson

D. R.

Thurlow

M. L.

(2018). Preparing for life after high school: The characteristics and experiences of youth in special education. Findings from the national longitudinal transition study 2012. Volume 3: Comparisons over time (Full report) (NCEE 2018-4007). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.

43.

Maenner

M. J.

Yeargin-Allsopp

Braun

K. V. N.

Christensen

D. L.

Schieve

L. A.

(2016). Development of a machine learning algorithm for the surveillance of autism spectrum disorder. PLoS One, 11(12), e0168224. https://doi.org/10.1371/journal.pone.0168224

44.

Mandinach

E. B.

(2012). A perfect time for data use: Using data-driven decision making to inform practice. Educational Psychologist, 47(2), 71–85. https://doi.org/10.1080/00461520.2012.667064

45.

Mandrekar

J. N.

(2010). Receiver operator characteristic curves in diagnostic test assessment. Journal of Thoracic Oncology, 5(9), 1315–1316. https://doi.org/10.1097/jto.0b013e3181ec173d

46.

Mason

S. J.,

Graham

N. E.

(2002). Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Quarterly Journal of the Royal Meteorological Society, 128(584), 2145–2166. https://doi.org/10.1256/003590002320603584

47.

McConnell

A. E.

Martin

J. E.

Hennessey

M. N.

(2015). Indicators of postsecondary employment and education for youth with disabilities in relation to GPA and general education. Remedial and Special Education, 36(6), 327–336. https://doi.org/10.1177/0741932515583497

48.

Meserole

(2018). What is machine learning? Brookings Institution. https://www.brookings.edu/research/what-is-machine-learning/

49.

Metz

C. E.

(1978). Basic principles of ROC analysis. Seminars in Nuclear Medicine, 8(4), 283–298. https://doi.org/10.1016/S0001-2998(78)80014-2

50.

Molfenter

N. F.

Hartman

Neugart

Webb

(2017). Let’s get to work Wisconsin: Launching youth with intellectual and developmental disabilities into the workforce. Journal of Vocational Rehabilitation, 47, 379–390. https://doi.org/10.3233/JVR-170910

51.

Nassif

A. B.

Azzeh

Banitaan

Neagu

(2016). Guest editorial: Special issue on predictive analytics using machine learning. Neural Computing & Applications, 27(1), 2153–2155. https://doi.org/10.1007/s00521-016-2327-3

52.

National Center for Education Statistics (2020). Students with disabilities – indicator. https://nces.ed.gov/programs/coe/indicator_cgg.asp

53.

National Science Foundation . (2021). Women, minorities, and persons with Disabilities in science and engineering. Special Report NSF 21-321. National Center for Science and Engineering Statistics.

54.

Newman

Wagner

Knokey

A.-M.

Marder

Nagle

Shaver

Wei

Cameto

Contreras

Ferguson

Greene

Schwarting

(2011). The post-high school outcomes of young adults with disabilities up to 8-years after high school. A report from the national longitudinal transition study-2 (NLTS2) (NCSER 2011-3005). SRI International.

55.

Palmer

I.,

Carpenter-Hubin

(2020). Using predictive analytics in the real world: Practice scenarios for ethical decision-making. New Directions for Institutional Research, 2019(183), 47–50. https://doi.org/10.1002/ir.20312

56.

Panesar

S. S.

D’Souza

R. N.

Yeh

F.-C.

Fernandez-Miranda

J. C.

(2019). Machine learning versus logistic regression in a small, heterogeneous glioma database. World Neurosurgery X, 2, 100012. https://doi.org/10.1016/j.wnsx.2019.100012

57.

Plasman

J. S.,

Gottfried

M. A.

(2018). Applied STEM coursework, high school dropout rates, and students with learning disabilities. Educational Policy, 32(5), 664–696. https://doi.org/10.1177/0895904816673738

58.

Porter

K. E.,

Balu

(2016). Predictive modeling of K-12 academic outcomes: A primer for researchers working with education data. MDRC.

59.

Rello

Baeza-Yates

Ali

Bigham

J. P.

Serra

(2020). Predicting risk of dyslexia with an online gamified test. PLoS One, 15(12), e0241687. https://doi.org/10.1371/journal.pone.0241687

60.

Schildkamp

(2019). Data-based decision-making for school improvement: Research insights and gaps. Educational Research, 61(3), 257–273. https://doi.org/10.1080/00131881.2019.1625716

61.

Sprunger

N. S.

Harvey

M. W.

Quick

M. M.

(2018). Special education transition predictors for post-school success: Findings from the field. Preventing School Failure, 62(2), 116–128. https://doi.org/10.1080/1045988X.2017.1393789

62.

StataCorp (2021). Stata statistical software version 17. [Computer Software].

63.

STEM Education Act (2015). Public law (P.L.) 114–59, 42 U.S.C. § 1861.

64.

Stone

J. V.

(2013). Bayes’ rule: A tutorial introduction to Bayesian analysis. Sebtel Press.

65.

Test

D. W.

Mazzotti

V. L.

Mustian

A. L.

Fowler

C. H.

Kortering

Kohler

(2009). Evidence-based secondary transition predictors for improving postschool outcomes for students with disabilities. Career Development for Exceptional Individuals, 32(3), 160–181. https://doi.org/10.1177/0885728809346960

66.

U.S. Department of Labor – Office of Disability Employment Policy (2022). Disability statistics April 2022: Disability employment statistics ages 16 years and over. https://www.dol.gov/agencies/odep

67.

U.S. Department of Labor (2019). Disability statistics – November 2019. Office of Disability Employment Policy. https://www.dol.gov/agencies/odep

68.

Van de Schoot

Depaoli

King

Kramer

Martens

Tadesse

M. G.

Vannucci

Gelman

Veen

Willemsen

Yau

(2021). Bayesian statistics and modeling. Nature Reviews, 1(1), 1–26. https://doi.org/10.1038/s43586-020-00001-2

69.

Van Ravenzwaaij

Cassey

Brown

(2018). A simple introduction to Markov chain Monte-Carlo sampling. Psychonomic Bulletin & Review, 25(1), 143–154. https://doi.org/10.3758/s13423-016-1015-8

70.

Wagner

E.,

Longanacker

(2016). Scaling student success with predictive analytics: Reflections after four years in the data trenches. Change: The Magazine of Higher Learning, 48(1), 52–59. https://doi.org/10.1080/00091383.2016.1121087

71.

Wilcox

Fernandez-Conde

Kowbel

(2021). Using evidence-based practice and data-based decision making in inclusive education. Education Sciences, 11(3), 1–11. https://doi.org/10.3390/educsci11030129

72.

Yin

Shaewitz

Megra

(2014). An uneven playing field: The lack of equal pay for people with disabilities. American Institutes for Research.

73.

Yperman

Becker

Valkenborg

Popescu

Hellings

Van Wijmeersch

Peeters

L.M.

(2020). Machine learning analysis of motor evoked potential time series to predict disability progression in multiple sclerosis. BMC Neurology, 20(105), 1–15. https://doi.org/10.1186/s12883-020-01672-w

74.

Zhang

(2020). Predictive analytics in the era of big data: Opportunities and challenges. Annals of Translational Medicine, 8(4), 68–68. https://doi.org/10.21037/atm.2019.10.97