The Effect of School-Wide Positive Behavior Interventions and Supports on Disciplinary Exclusions: A Conceptual Replication

Abstract

Disciplinary exclusions, particularly out-of-school suspension and expulsions, are a pressing concern for schools, as research demonstrates that they are associated with myriad deleterious outcomes such as increased risk for poor academic achievement, school dropout, and contact with juvenile justice. Research suggests that School-Wide Positive Behavior Interventions and Supports (SWPBIS), a prevention and intervention framework for addressing school-based problem behavior, can have a significant and meaningful impact on reducing the likelihood of student suspensions and expulsions. In this study, we conceptually replicated a series of previous studies conducted in other states and examined the effect of universal SWPBIS on disciplinary exclusions in California. Using propensity score matching, we examine differences in suspension and expulsion rates for 98 schools implementing universal SWPBIS with fidelity and 98 comparison schools not implementing SWPBIS. Results suggest that schools implementing SWPBIS with fidelity have significantly fewer suspensions. No effects were found for expulsions. Implications and recommendations for future research are discussed.

Keywords

schoolwide positive behavior supports disciplinary exclusions suspensions quasi-experimental design propensity score matching

Disciplinary exclusion of students, including suspension and expulsion from school, continues to be a pressing concern for students, school personnel, parents, and policymakers (Losen & Martin, 2018). Research demonstrates the deleterious outcomes associated with disciplinary exclusions, including more incidents of exclusion, poor academic performance, and increased risk for contact with juvenile justice (Noltemeyer et al., 2015). Thus, evidence-based practices and programs are necessary to reduce the likelihood of disciplinary exclusions in schools. One framework for delivering evidence-based behavior prevention and intervention practices, School-Wide Positive Behavior Interventions and Supports (SWPBIS), has a growing evidence base indicating positive impacts on the reduction of disciplinary exclusions (Gage et al., 2019; Gage, Lee, et al., 2018; Gage, Whitford, & Katsiyannis, 2018). The purpose of this study is to conceptually replicate prior studies of the impact of SWPBIS on disciplinary exclusions using data from California. In the following sections, we describe the critical features of SWPBIS and the evidence of SWPBIS impacts on suspensions and expulsions. We then describe the state-level analysis, results of the study, and implications for future research.

SWPBIS

SWPBIS is a multitiered framework for preventing problem behavior before it occurs and implementing evidence-based intervention services, based on data, for students demonstrating school-based behavior problems. As the name suggests, SWPBIS focuses on the use of proactive and preventive discipline practices focusing on positive reinforcement instead of punishments across all prevention and intervention practices. At the universal, or Tier 1, level, schools develop schoolwide behavioral expectations that are explicitly taught in all settings (e.g., classrooms, hallways, cafeteria, playground). Schools also create school-based behavior teams that review behavioral data, such as behavioral screeners or office discipline referrals (ODRs), to identify students in need of evidence-based interventions. Those identified as nonresponsive to Tier 1 prevention receive secondary, or Tier 2, interventions designed for students demonstrating problem behaviors. These evidence-based interventions include schoolwide and classroom-based intervention programs, such as Check-in/Check-out (Crone et al., 2010), group-based social skills instruction, and First Step to Success (Walker et al., 2009). The critical feature of all Tier 2 interventions is that they are targeted at students in need of additional support, but are more efficient, practical, and feasible to implement than Tier 3 support. Students that are nonresponsive to Tiers 1 and 2, or displaying intensive behaviors in need of immediate support, then receive tertiary, or Tier 3, behavioral support, which involves the development and use of functional behavior assessment (FBA)-based interventions.

SWPBIS is currently implemented in more than 25,000 schools and, when implemented with fidelity, has demonstrated positive effects across a number of student and school outcomes (OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports, 2017). For example, Bradshaw et al. (2008) evaluated the effect of SWPBIS on schools’ organizational health, which includes staff affiliation, academic emphasis, collegial leadership, and institutional integrity, using a randomized controlled trial (RCT) design and found statistically significant and positive improvements in organizational health for schools implementing SWPBIS with fidelity. Using student-level data from the same RCT, Bradshaw et al. (2012) found statistically significant decreases in students’ disruptive behavior (d = 0.12) and increases in prosocial behaviors (d = 0.17) in schools implementing SWPBIS. Research has also found positive and statistically significant effects on academic achievement. Gage et al. (2017) examined the effect of universal (Tier 1) SWPBIS on school-level achievement across 10 years for all schools in Florida. Overall, they found that schools implementing SWPBIS with fidelity had significantly more students performing at or above the state benchmark for proficiency in both reading and math (d = 0.12 and d = 0.10, respectively); however, these effects have not been replicated in other states (e.g., Ryoo et al., 2018). Recently, Mitchell et al. (2018) examined the SWPBIS evidence base using the What Works Clearinghouse (WWC, 2015) research quality standards and found (a) four RCT studies that met WWC standards and (b) positive findings for school climate and student outcomes, including ODR. Taken together, SWPBIS has demonstrated significant and positive impacts on a number of school and student outcomes.

Disciplinary Exclusions and SWPBIS

Disciplinary exclusions are defined as consequences for school-based problem behaviors that remove students from instructional settings (Chin et al., 2012). Disciplinary exclusions include an array of disciplinary consequences aimed at increasing safety by removing the student displaying the problem behavior, including timeouts, ODR, in- and out-of-school suspensions, and expulsions (Fabelo et al., 2011). Although widely used, disciplinary exclusions place students at risk for poor academic engagement and performance, increased rates of school dropout, and incarceration (American Academy of Pediatrics, 2013; Fabelo et al., 2011; Losen & Martinez, 2013). Losen and Martin (2018) examined the impact of disciplinary exclusions in California and found that students lost more than 760,000 days of instruction due to those exclusions. Students in Grades 7 and 8 lost the most days of instruction, whereas significant disparities in disciplinary exclusions for Black students were consistently found across all grades. Although these data are concerning, the loss of instruction due to discipline in California decreased almost 50% from the 2011–2012 school year to the 2016–2017 school year. Progress is being made, but there is still an opportunity to continue to reduce loss of instructional time from disciplinary exclusions.

Out-of-school suspensions (OSS) and expulsions are the most severe disciplinary exclusions as they result in the physical removal of students from their schools. Research has consistently demonstrated that both have significant negative impacts on student outcomes. Noltemeyer and colleagues (2015) conducted a meta-analysis of the impact of suspensions on student outcomes and found 25 studies focused on the relationship between OSS and academic achievement, and nine studies focused on the relationship between OSS and dropping out of school. They found a statistically significant, negative relationship between OSS and academic achievement (r = −.24) and a statistically significant, positive relationship between OSS and school dropout (r = .25). Rumberger and Losen (2016) examined outcomes for students that received at least one suspension in California and found that those students with a suspension event were 6.5% less likely to graduate high school, controlling for other potential predictors of school dropout. The authors also conducted an economic analysis of OSS in California and found a negative impact of US$2.7 billion in lifetime costs for only a single graduating class.

More recently, Anderson et al. (2019) examined the effect of disciplinary exclusions on academic achievement using 10 years of data from Arkansas. The authors noted that a number of student- and school-level characteristics may have confounded prior studies, including socioeconomic status, race, and prior achievement, and directly addressed those limitations via unique econometric models. Overall, the authors found that students that receive OSS have significantly lower academic achievement, including test scores, and higher rates of grade retention. Taken together, students receiving OSS demonstrate lower achievement than their peers and are at a greater risk for school dropout. Similarly, expulsions result in harmful outcomes for students, including increased risks for school dropout and later incarceration (American Academy of Pediatrics, 2013). In fact, research suggests that a single expulsion from school doubles the risk for a student repeating a grade, a significant risk factor for dropping out of school (Kang-Brown et al., 2013). These outcomes are particularly concerning for students with disabilities generally, but particularly for students with emotional and/or behavioral disorders (EBD). Sullivan et al. (2014) found that students with EBD were nine times more likely to receive a suspension compared with students with other disabilities.

A series of studies have evaluated the impact of SWPBIS on disciplinary exclusions. Gage et al. (2018) conducted a meta-analysis of studies that (a) used a group quasi-experimental or RCT design, with school as the unit of analysis, and (b) reported disciplinary exclusions for schools in both a treatment and a control group. The authors found four studies, which included 90 schools, with no significant effect on ODR, but a significant and large effect on suspensions (d = −0.86)—suggesting that schools implementing SWPBIS with fidelity demonstrated large reductions in the number of suspensions. Gage, Lee, et al. (2018) evaluated the effect of implementing SWPBIS with fidelity in Georgia by propensity score matching (PSM) 119 elementary schools implementing SWPBIS to 119 covariate-matched comparison schools. The authors found significant and moderate to large effects for ODR (d = −0.64), OSS (d = −0.54), and in-school suspension (ISS; d = −0.71). Gage et al. (2019) replicated and extended prior research using PSM to compare 593 schools implementing SWPBIS with fidelity to 593 covariate-matched comparison schools in Florida. Results confirmed a significant reduction in the use of OSS for schools implementing SWPBIS with fidelity (d = −0.55); however, no effect was found for expulsions.

Recently, Grasley-Boy et al. (2019) evaluated the impact of SWPBIS implemented at the universal level on disciplinary exclusions in California schools during the 2015–2016 school year. The authors replicated the approach used by Gage et al. (2019) by including only schools implementing with fidelity at Tier 1 and data from the U.S. Department of Education’s Office of Civil Rights’ Civil Rights Data Collection (CRDC). The authors propensity score matched 544 schools implementing universal SWPBIS with 544 schools never trained to implement universal SWPBIS. The authors found that students in schools implementing universal SWPBIS with fidelity had significantly fewer OSS (g = −0.25) and that students with disabilities were significantly less likely to be referred to alternative schools for disciplinary reasons (g = −0.65).

Although positive results have been found in Georgia, Florida, and California for OSS, similar studies in South Carolina and Minnesota found no significant effects on disciplinary exclusions (Gage & Stevens, 2018; Ryoo et al., 2018). Furthermore, to date, few studies have examined the relation between SWPBIS implementation and expulsions. Therefore, we designed a conceptual replication of Gage, Lee, et al. (2018) to continue to explore the relation between SWPBIS implementation and disciplinary exclusions.

Purpose

Disciplinary exclusions remain a concern and evidence continues to demonstrate potentially harmful effects from exclusion, including lower achievement and increased risk for dropping out of school. A growing evidence base has found mixed effects on reducing disciplinary exclusions in schools implementing SWPBIS with fidelity; however, those effects have not been replicated in all states. Coyne et al. (2016) noted, “Closely aligned conceptual replications provide evidence about the efficacy of an intervention and help determine whether findings from an initial study are reproducible under very similar conditions” (p. 247). Therefore, in this study, we conceptually replicate Gage, Lee, and colleagues’ (2018) approach using data from California, which has a statewide recognition system similar to the one in Georgia. Schools in California are recognized based on four levels (Bronze, Silver, Gold, and Platinum), which are based on implementing SWPBIS at different fidelity levels. Specific research questions guiding this study were as follows:

Research Question 1: What is the effect of SWPBIS implemented with fidelity at the universal level on OSS?

Research Question 2: What is the effect of SWPBIS implemented with fidelity at the universal level on expulsions?

Research Question 3: Are there differential effects by implementation recognition levels on OSS and expulsions?

Based on the results in Georgia and Florida, we hypothesized that schools implementing SWPBIS with fidelity would have significantly fewer suspensions than comparison schools and no significant differences for expulsions. There is limited research exploring differential effects by recognition levels, and therefore Research Question 3 is exploratory in nature and no a priori hypotheses were made.

Method

Sample

We collected all available demographic and discipline data for all California public schools for the 2016–2017 school year from the California Department of Education website. Data were available for all 10,473 schools in all 1,026 school districts in California. Next, we collected the names of all schools recognized by the California PBIS Coalition for the 2016–2017 school year and their fidelity of implementation scores based on the statewide recognition system. After merging the datasets and removing schools with missing values on the dependent variables (i.e., reported OSS and expulsions), we arrived at our sample of 7,251 public schools used for matching treatment and comparison schools. Almost 60% of the sample are elementary schools; the average enrollment was 622.6 students; and, on average, there were 25.7% White students, 5.9% African American, 53.4% Hispanic, 23.2% English learners, 12.2% students receiving special education services, and 62.6% economically disadvantaged.

Independent Variable: SWPBIS

Statewide implementation of SWPBIS in California is supported by the CA PBIS Coalition (CPC). The Coalition’s purpose is to promote safe and positive social cultures in all CA school communities by sharing effective and evidence-based academic, behavioral, and mental health practices; provide opportunities for networking; and support learning of the PBIS implementation blueprint and sustainable practices. At their inception in 2010–2011, there were 433 schools implementing SWPBIS. By 2014, that number had grown to 956, and in 2017 there were more than 2,500 schools implementing SWPBIS (www.pbisca.org).

The CPC has an annual statewide recognition system for highlighting and celebrating the success of schools implementing SWPBIS in CA. Schools submit evidence of success and are then celebrated across four recognition levels:

Bronze recognition is earned for schools that complete the TFI and an online application.

Silver recognition is earned for schools that implement SWPBIS with 70% or greater fidelity on one or more tiers based on TFI scores completed with an external coach and completing the online application. According to the TFI developers, scores of 70% or higher at each tier indicate implementation with fidelity (Algozzine et al., 2014).

Gold recognition is earned for schools that implement Tier 1 universal SWPBIS with 70% or greater fidelity and 70% or greater on either Tier 2 or 3 according to the TFI, complete an online application, and provide a supporting statement of 100 words or less about the SWPBIS effort of the school.

Platinum recognition is earned for schools that implement all three SWPBIS tiers with 70% or greater fidelity, submit rates of OSS and ODRs, sustain or improve positive academic trends, complete an online application, and provide a supporting statement of 100 words or less about the SWPBIS effort of the school.

For the 2016–2017 school year, 17 schools received Platinum recognition, 91 schools received Gold recognition, 559 schools received Silver recognition, and 222 schools received Bronze recognition. To replicate prior research and ensure that treatment schools implemented universal (Tier 1) SWPBIS with fidelity, we excluded all schools that received Bronze and Silver recognition. We excluded schools with Silver recognition because we could not ensure that they implemented Tier 1 with fidelity (i.e., by definition they could have implemented Tier 2 or 3 with fidelity, but not Tier 1). Therefore, our treatment sample included 108 possible schools. However, after removing all schools without available suspension or expulsion data, the final sample was 98 treatment schools. We used the recognition system to replicate the approach used in Georgia and because school-level fidelity scores are not publicly available.

The recognition system in California is similar to that used in Georgia that served as the treatment indicator in Gage, Lee, et al. (2018). However, there are noteworthy differences. First, in Georgia, recognition is based on the Benchmarks of Quality (BoQ), which only evaluates universal (Tier 1) fidelity. Therefore, the prior study had no information about implementation of Tier 2 or 3. Second, the two categories of recognition in Georgia were based on different universal fidelity cut-scores (i.e., BoQ scores between 70% and 85% were defined as emerging, and BoQ scores above 85% were considered operational).

Measures

SWPBIS Tiered Fidelity Inventory

The SWPBIS Tiered Fidelity Inventory (TFI; Algozzine et al., 2014) is a measure of the extent to which school personnel apply the core features of SWPBIS. There are three parts to the TFI: Tier 1—Universal SWPBIS Features, Tier 2—Targeted SWPBIS Features, and Tier 3—Intensive SWPBIS Features. The TFI is typically completed by the SWPBIS team, but it is strongly recommended that the TFI be completed with an external SWPBIS coach or facilitator familiar with the school (Algozzine et al., 2014). In California, each school’s SWPBIS team completes the TFI. Schools are considered implementing a SWPBIS tier with fidelity if the implementation score is 70% or greater. According to the developers, the TFI can be used (a) for initial assessment to determine if a school is using or needs SWPBIS; (b) as a guide for implementation of Tier 1, 2, and 3 practices; (c) as an index of sustained SWPBIS implementation; or (d) as a metric for identifying schools for recognition within their state implementation efforts (Algozzine et al., 2014). In this study, the TFI is used as a metric for state recognition of implementation. McIntosh and colleagues (2017) evaluated the technical adequacy of the TFI and found an interrater reliability of .99, a test–retest reliability of .99, and an internal consistency of α = .87 for Tier 1, α = .96 for Tier 2, α = .98 for Tier 3, and α = .96 for a combined score.

School demographics

We included 13 school-level covariates in this study. Total enrollment was defined as the total number of students in the school. Then, from the enrollment total, the percentage of African American, Hispanic, and White students was calculated and included. We also included the percentage of students receiving free or reduced-price lunch, English learner services, and special education services. Next, we included a series of categorical predictors, including school type (regular or alternative school), school level (elementary, middle, high, other), urbanicity (city, suburban, town, rural), and Title I status. Last, we included the percentage of students at or above grade level on the California Assessment of Student Performance and Progress (CASPP) in both English language arts and mathematics. All data were reported by each school administration to the state department of education and represented values for the 2016–2017 school year except the CASPP data, which were from the 2015–2016 year. We included prior year achievement because testing occurs at the end of the school year, whereas OSS and expulsions occur throughout the year. Therefore, we used prior year achievement to avoid ambiguous temporal precedence concerns about the relation of predictors to outcomes (Shadish et al., 2002). The Georgia study (Gage, Lee, et al., 2018) included 11 covariates. We included school type, school level, and Title I status as covariates in this study, which were not included in the Georgia study, whereas the Georgia study included the percentage of students proficient or above in science and social studies, which was not included here.

Disciplinary exclusions

California has established clear and consistent criteria for use of OSS and expulsions (U.S. Department of Education, 2018). Students may be suspended or, in extreme cases, expelled, for any of the following behavioral infractions: (a) caused, attempted to cause, or threatened physical injury; (b) willfully used force or violence, except in self-defense; (c) possessed or sold firearms or other weapons; (d) unlawful possession or sale of a controlled substance; (e) robbery or extortion; (f) damage to school property; (g) theft; (h) possessed tobacco products; (i) committed obscene act; (j) disrupted school activities (not applicable to students in Grades K–3); (k) sexual assault; (l) threatened a witness in a school disciplinary proceeding; (m) hazing; or (n) engaged in an act of bullying. The state collects the number of students in a school that receive at least one suspension or are expelled. The rates are calculated as follows: number of students suspended or expelled in the current year divided by cumulative enrollment. Thus, the rates were defined as the unduplicated count of students suspended or expelled, divided by the cumulative enrollment at the school. The rates do not include information about how many times a student is suspended or for how long, only the proportion of students who received a disciplinary exclusion. The study in Georgia (Gage, Lee, et al., 2018) used the raw, unduplicated number of students receiving disciplinary exclusions and controlled for enrollment in the matching and regression models.

Data Analysis

To address the primary research questions in this study, we conducted a quasi-experimental design (QED) comparing schools implementing universal (Tier 1) SWPBIS with fidelity and either Tier 2 or 3 with fidelity (Gold) or all three tiers with fidelity (Platinum) to propensity-score-matched comparison schools not implementing SWPBIS.

Missing data

Prior to reducing the full sample of possible schools to the final analytic sample, we imputed missing data for the available covariates. There was an average of 19.5% missingness across the available covariates used in this study. The most missing was for the academic outcomes (23.1% for the percentage of students at or above benchmark in math and reading), and the least amount of missing was for total student enrollment, with 18.2% missing. We used multivariate imputation by chained equations (MICE) to address missingness. MICE models each of the covariates conditional on the others and the imputations are the predicted values from these regression models with the random error included. The procedure works as follows: (a) The variable with the least missingness is imputed conditional on all variables with no missingness; (b) then the variable with the second least missingness is imputed conditional on the variables with no missingness and the first variable imputed, and so on; (c) once all of the covariates have been cycled through, there are no longer any missing values in the data (Raghunathan et al., 2001). Prior research has found that imputation using MICE can accurately address missingness greater than 20% (Dong & Peng, 2013; Stuart et al., 2009). Imputation was estimated in the MICE package (van Buuren & Groothuis-Oudshoorn, 2011) in R (R Core Team, 2014).

PSM

PSM methods are designed to reduce bias in treatment effect estimates in experimental research design studies where random assignment of participants to conditions is not possible (Leite, 2017). A propensity score is the conditional probability of treatment assignment based on all available covariates (Rosenbaum & Rubin, 1983) and can then be used for one-to-one matching of treatment to comparison schools. PSM is one approach to identify a covariate equivalent comparison group in a QED. In their examination of the effects of charter schools, Fortson et al. (2012) reported that treatment estimates from PSM did not differ significantly from estimates derived from RCT data.

We estimated propensity scores using logistic regression and the school-level covariates described in Table 1. We created a dichotomous dependent variable for all schools as follows: Schools that implemented universal SWPBIS at Platinum and Gold recognition levels were coded as 1, whereas all other schools were coded as 0. Next, we estimated the predicted probability (p), or propensity score, that a school was in the “treatment” (i.e., Platinum and Gold schools) or “control” (i.e., schools not receiving SWPBIS recognition) group based on the included covariates (log[p / (1 − p)]). Then we used the estimated propensity scores to match schools using the one-to-one optimal matching method (Leite, 2017), which minimizes the global propensity score distance between treatment and comparison schools. Unlike other PSM approaches (i.e., propensity score weighting), one-to-one matching identifies a covariate equivalent match for each treatment school. The one-to-one optimal matching algorithm was conducted using the matchit (Ho et al., 2017) and optmatch (Hansen et al., 2018) packages in R (R Core Team, 2014). To confirm covariate equivalence, we examined the standardized mean difference effect sizes (d) reported in the optmatch package and defined equivalence as d ≤ 0.25 standard deviations (WWC, 2015). These procedures were the exact same as those used in Gage, Lee, et al. (2018).

Table 1.

Demographic Characteristics for the Full Sample.

	All possible comparison schools (n = 6,464)		PSM comparison (n = 98)		Treatment schools (n = 98)		d
Demographic	M	SD	M	SD	M	SD	d
Enrollment	699.2	566.9	752.2	596.9	746.8	457.7	−0.02
%AA	5.8	9.2	2.8	3.6	4.7	7.5	0.25
%HI	54.3	27.6	69.8	24.8	70.7	28.1	0.03
%WH	25.2	23.2	16.7	20.2	13.6	19.5	−0.16
%FRL	63.5	27.6	73.9	22.9	76.5	23.7	0.11
%EL	23.0	18.5	36.4	22.1	33.9	20.4	−0.12
%SPED	12.1	10.0	10.0	3.8	10.7	4.3	0.15
%ELA	43.8	20.9	35.7	17.9	37.5	16.1	0.11
%Math	33.1	20.5	30.6	15.3	30.6	17.3	0.00
School Characteristics	%	n	%	n	%	n	d
School type
Regular	91.5	5,915	94.9	93	95.9	94	0.10
Alternative	8.5	549	5.1	5	4.1	4	−0.05
School level
Elementary	58.4	3,775	78.5	77	77.6	76	0.02
Middle	17.7	1,144	9.2	9	12.2	12	−0.09
High	20.6	1,332	12.2	12	10.2	10	−0.07
Other	3.3	213	0.0	0	0.0	0	0.00
Urbanicity
City	38.4	2,480	50.0	49	48.0	47	0.06
Suburban	42.9	2,771	34.7	34	35.7	35	0.02
Town	8.8	569	9.2	9	9.2	9	0.00
Rural	10.0	644	6.1	6	7.1	7	0.04
Title I
No	77.6	5,017	93.9	92	94.9	93	0.05
Yes	22.4	1,447	6.1	6	5.1	5	−0.46

Note. PSM = propensity score matching; AA = African American; HI = Hispanic; WH = White; FRL = free or reduced-price lunch; EL = English learner; SPED = special education; ELA = English language arts.

Statistical models and effect sizes

We used multiple linear regression to address the primary research questions in this study. The primary outcomes for the study were the suspension and expulsion rates reported for each school. The regression models included all covariates to ensure treatment effects controlled for any potential covariate effects. First, we modeled the effect for implementing universal (Tier 1) SWPBIS with fidelity and either Tier 2 or 3 with fidelity (Gold) or all three tiers with fidelity (Platinum) for each outcome. We then conducted exploratory models to determine whether there were differences by recognition level. All four models were estimated as follows:

\begin{array}{l} y_{i} = β_{0} + β_{1} {Treatment}_{i} + \sum_{c = 1}^{13} β_{c} X_{c i} + ε_{i}, \\ ε_{i} ~ N (0, σ^{2}), \end{array}

where $y_{i}$ is the suspension or expulsion rate, $β_{0}$ is the intercept, $β_{1}$ is the treatment effect (0 for comparison and 1 for treatment for the main effects; and 0 for comparison, 1 for Gold, and 2 for Platinum for the exploratory models), and $β_{c}$ is the series of coefficients for the 13 covariates. Last, we calculated covariate adjusted means and standard deviations for both outcomes for treatment groups and estimated conventional standardized mean difference effect sizes (d) (Borenstein et al., 2009). Gage, Lee, et al. (2018) used Poisson regression models because they modeled the number of students with disciplinary exclusions. However, the modeling approach (i.e., fixed-effects models) and the calculation of standardized mean differences were the same.

Results

PSM and Equivalence

As noted previously, we started with a full sample of 7,251 schools and removed all schools that received Bronze or Silver recognition because we could not confirm that they implemented universal (Tier 1) SWPBIS with fidelity (i.e., TFI Tier 1—70% or greater) and either Tier 2 or 3 with fidelity (Gold) or all three tiers with fidelity (Platinum). This left 6,564 possible comparison schools and 98 treatment schools (83 with Gold recognition and 15 with Platinum recognition). Next, we conducted the PSM model with the 13 school-level covariates. Using the one-to-one optimal matching approach, the final analytic data set included the 98 treatment schools and 98 comparison schools with equivalence values less than or equal to 0.25 standard deviation units (see Table 1).

Regression Models

Prior to modeling the treatment effect, we evaluated the distributional characteristics of both dependent variables with the final analytic sample (n = 196). The skew and kurtosis values of 4.7 and 26.9 for OSS and 9.0 and 97.7 for expulsions, respectively, indicated that the dependent variables were positively skewed. Therefore, we transformed both dependent variables for modeling using square root transformations. Next, we evaluated the number of schools within districts to rule out the need for a hierarchical linear model (HLM) with schools nested in school districts. Overall, we found that the 198 schools were nested within 110 school districts, with an average of 1.8 schools per district. Raudenbush and Bryk (2002) recommend at least 10 Level 1 units within each Level 2 unit for accurate HLM estimates; therefore, we relied on multiple linear regression to estimate treatment effects.

We estimated four models, two for each disciplinary exclusion. The models in Table 2 examined the effect of implementing universal (Tier 1) SWPBIS with fidelity and either Tier 2 or 3 with fidelity (Gold) or all three tiers with fidelity (Platinum) on OSS and expulsions. Controlling for the 13 covariates, we found that schools in the treatment group reported statistically significantly fewer OSS than PSM comparison schools. No differences were found for expulsions.

Table 2.

Multiple Linear Regression Model for Treatment Effects on Suspensions and Expulsions.

Term	Suspension		Expulsions
Term	b	SE	b	SE
Intercept	1.65	0.89	−0.15	0.24
Treatment	−0.25*	0.11	−0.03	0.03
Enrollment	0.00	0.00	0.00	0.00
%AA	0.00	0.01	0.00	0.00
%WH	0.00	0.01	0.00	0.00
%HI	−0.01	0.01	0.00	0.00
%FRL	0.01	0.01	0.00	0.00
%EL	0.00	0.00	0.00	0.00
%SPED	0.00	0.01	0.00	0.00
School type: alternative	1.03*	0.40	0.33**	0.11
Urban: suburban	0.03	0.12	0.01	0.03
Urban: town	0.43*	0.20	0.03	0.05
Urban: rural	1.18***	0.25	0.24**	0.07
Title I	0.27	0.27	−0.05	0.07
School level: middle school	1.19***	0.20	0.25***	0.05
School level: high school	0.97**	0.33	0.21*	0.09
%ELA	0.00	0.01	0.00	0.00
%Math	−0.01	0.01	0.00	0.00

Note. AA = African American; WH = White; HI = Hispanic; FRL = free or reduced-price lunch; EL = English learner; SPED = special education; ELA = English language arts.

p < .05. **p < .01. ***p < .001.

Next, we estimated the same models as above, but included recognition level (i.e., PSM comparison, Gold, and Platinum schools) as a categorical variable to compare recognition levels with the PSM comparison schools (see Table 3). There were no statistically significant differences between schools in the Gold recognition category and PSM school comparison schools on suspension rates, but there was a significant difference for the Platinum recognition category. To support the finding, we conducted a post hoc power analysis for the exploratory recognition-level model. Using the degrees of freedom and adjusted R² value (.50), we calculated the power of 0.99, suggesting that the model was adequately powered (Cohen, 1988). Once again, there were no differences in rates of expulsions between schools.

Table 3.

Multiple Linear Regression Model for Recognition Levels on Suspensions and Expulsions.

Term	Suspension		Expulsions
Term	b	SE	b	SE
Intercept	1.68	0.89	−0.15	0.24
Recognition: Gold	−0.21	0.11	−0.03	0.03
Recognition: Platinum	−0.45*	0.23	−0.04	0.06
Enrollment	0.00	0.00	0.00	0.00
%AA	0.00	0.01	0.00	0.00
%WH	0.00	0.01	0.00	0.00
%HI	−0.01	0.01	0.00	0.00
%FRL	0.01	0.01	0.00	0.00
%EL	0.00	0.00	0.00	0.00
%DI	0.00	0.01	0.00	0.00
School type: alternative	0.99*	0.40	0.33**	0.11
Urban: suburban	0.04	0.12	0.01	0.03
Urban: town	0.52*	0.21	0.03	0.06
Urban: rural	1.25***	0.26	0.24**	0.07
Title I	0.30	0.27	−0.05	0.07
School level: middle school	1.16***	0.20	0.25***	0.05
School level: high school	0.97**	0.33	0.21*	0.09
%ELA	0.00	0.01	0.00*	0.00
%Math	−0.01	0.01	0.00	0.00

Note. AA = African American; WH = White; HI = Hispanic; FRL = free or reduced-price lunch; EL = English learner; ELA = English language arts.

p < .05. **p < .01. ***p < .001.

Last, we calculated the covariate adjusted standardized mean difference between treatment and control schools for statistically significant outcomes. The covariate adjusted means were 2.73 (SD = 1.18) square-root-transformed OSS rates for the comparison schools, 2.48 (SD = 0.79) square-root-transformed OSS rates for all treatment schools, and 2.27 (SD = 1.38) square-root-transformed OSS rates for Platinum recognition schools. The effect size for OSS by treatment condition was d = −0.25. The effect size for the Platinum recognition schools compared with PSM comparison schools was d = −0.41.

Discussion

A growing empirical evidence base supports the positive effect SWPBIS can have reducing schools’ use of disciplinary exclusions (Gage et al., 2019; Gage, Whitford, & Katsiyannis, 2018; Grasley-Boy et al., 2019; Pas et al., 2019). We conceptually replicated a prior QED study (Gage, Lee, et al., 2018) identifying schools implementing SWPBIS with fidelity using a state recognition system as the treatment indicator and reproduced those prior results. Specifically, we found that schools implementing universal (Tier 1) SWPBIS with fidelity and either Tier 2 or 3 with fidelity (Gold) or all three tiers with fidelity (Platinum) reported significantly fewer OSS than PSM schools not implementing SWPBIS. More specifically, the effect sizes suggest, based on WWC (2015) criteria, that SWPBIS has a meaningful impact (i.e., d ≥ 0.25) on OSS in California. However, there was no effect on expulsions, and, to date, no studies have found any relation between SWPBIS and expulsions. This may be due to the fact that more elementary schools implement SWPBIS or other unrelated factors (e.g., expulsions are the result of behaviors not directly addressed by SWPBIS, such as bringing a weapon to school). Nationally, data suggest that high school students are almost twice as likely to be expelled as students in Grades K–8 (U.S. Department of Education, National Center for Educational Statistics, 2019).

In comparing the results with those of the prior state-level PSM studies, a number of similarities and differences emerged. First, the effect size for OSS is smaller than those found in Georgia (Gage, Lee, et al., 2018) and Florida (Gage et al., 2019), but statistically significant, unlike South Carolina (Gage & Stevens, 2018) and Minnesota (Ryoo et al., 2018). It is worth noting however that both studies with nonsignificant results included data prior to 2015 and had much smaller sample sizes than any of the states with significant results. Furthermore, Florida and Georgia have well-established statewide support systems for scaling SWPBIS, whereas California’s statewide scale-up efforts have evolved and changed over time and are less established. Minnesota also has state-level support, but the data for the study by Ryoo and colleagues (2018) were taken from only a handful of schools implementing SWPBIS and do not represent a statewide analysis; SWPBIS scale-up in South Carolina has not been fully financially supported by the state, but instead is supported by a loosely connected group of school- and university-based personnel (Gage & Stevens, 2018).

Interestingly, the effect for OSS in this study was similar to that found by Grasley-Boy et al. (2019), also in California. However, that study used data reported by schools to the CRDC during the 2015–2016 school year, only included schools implementing universal SWPBIS with fidelity, and used zero-inflated Poisson regression models to estimate treatment effects, replicating the Florida study. We used the recognition system, replicating the Georgia study, as the treatment indicator and modeled the effects using multiple regression. The consistency of findings between the studies provides additional support for the impact SWPBIS has on OSS in California.

A second important difference is the variety of fidelity of implementation tools used across the PSM studies. Both Florida and Georgia used the BoQ (Kincaid et al., 2005, 2010), whereas South Carolina used the School-wide Evaluation Tool (SET; Sugai et al., 2005), and Minnesota used a combination of the SET and the BoQ. In this study, the TFI was used as the primary indicator of fidelity, which is currently the recommended fidelity tool from the OSEP PBIS Center (www.pbis.org). Differences across studies may be related to the differences in how fidelity is measured across each of the instruments. Mercer et al. (2017) examined the convergent validity among each of the fidelity tools and found moderate correlations between the SET and the BoQ (r = .63), and the BoQ and the TFI (r = .65), and a large correlation between the SET and the TFI (r = .92). Further complicating the convergent validity were clear differences between each measure’s fidelity criterion score. The SET considers that fidelity is present if schools meet 80% of the measure’s items, whereas the TFI requires schools to meet only 70% of items at each tier. It is also worth noting that the effect sizes for schools implementing in the Platinum recognition levels are closer to the effect sizes found in states using the BoQ.

The last important differences are the year the data were collected and how the data are scaled and modeled. First, this study used the most recent available data, the 2016–2017 school year, whereas all other studies used data from previous years. Second, prior studies relied on the number of students receiving one or more OSS (i.e., count data), controlling for enrollment, whereas the data in California for the 2016–2017 school year were reported as a school-level rate. As such, prior studies used Poisson regression analysis, whereas this study used ordinary least squares (OLS) regression. The differences should not have impacted the likelihood of a significant finding or the effect sizes, but are noteworthy for comparing with future state-level analyses.

Implications for Practice

Certainly, the reduction in OSS associated with implementing SWPBIS with fidelity is important based on the numerous negative student-level outcomes associated with suspension events (Noltemeyer et al., 2015). Another benefit is the reduction in long-term economic burden. Using Rumberger and Losen’s (2016) figures and transforming the effect size to Cohen’s U3, we can assume that, if all schools implemented SWPBIS with fidelity, there would be a 9.87% decrease in OSS and, subsequently, a US$264,417,300 reduction in lifetime costs to the state of California for this cohort of students. Of course, these figures are based on assumptions and extrapolations, but the point remains that implementing SWPBIS with fidelity may have additive impacts beyond the school by reducing suspension rates.

Limitations

A number of limitations necessitate discussion. First, this study relied solely on administrative data and the accuracy and reliability of the data cannot be independently confirmed. However, the data used are the same data policymakers at the state and national level use to make policy decisions. This is also the case with the TFI scores used to identify schools for recognition. The TFI was not completed by an external coach; therefore, the reliability of TFI scores cannot be assessed. Second, we could not include prior years’ OSS or expulsions because we cannot confirm how long the schools have been implementing SWPBIS. Anecdotally, we know that some of the schools have been implementing SWPBIS since its inception in California, but without accurate start dates we could not include OSS or expulsions from prior years to truly reflect pre-SWPBIS rates. Third, we used MICE to impute missing data to ensure complete case analysis, but note that there a number of limitations to the approach, including concerns about the conditional distributions applied and ignoring clustering (Azur et al., 2011). However, by imputing missing variables, we were able to maintain our full PSM sample size. Fourth, the suspension rate does not include information about how many times a student is suspended or for how long, only that the student received a disciplinary exclusion. Fifth, we did not conduct a separate PSM for schools in the Platinum group because of the small sample size and exploratory nature of the research question. We examined equivalence between Platinum schools and all PSM schools and did not establish equivalence on all variables (e.g., Platinum schools were more likely to have a greater percentage of White students). Further research should consider focusing exclusively on modeling differences by recognition level, establishing equivalence for each level to the comparison group. Sixth, we do not have direct evidence that the comparison schools were not trained to implement SWPBIS. We only know that they were not recognized by the CPC. It is possible that some schools may have been trained to implement SWPBIS, but did not apply for recognition. Last, future research should consider addressing any potential clustering effects from districts as data become available. As noted, we were unable to address clustering within districts, and therefore district-level characteristics may have impacted the results.

Implications for Future Research

The results of this study, coupled with the prior studies, suggest that the implementation of SWPBIS with fidelity statistically significantly reduces school suspensions. This study replicated prior studies in other states and we hope to continue replicating these analyses in other states. However, the results do not provide information or details on how implementation of SWPBIS reduces suspensions and whether students in treatment schools are more likely not to display the behaviors that would result in suspension, receive different consequences, or receive Tier 2 or 3 supports instead of suspensions. Much work is needed to better understand the mechanisms and components of SWPBIS implementation that directly impact suspensions. In addition, future studies should consider examining student-level data to examine the likelihood of being suspended more than once in a school implementing SWPBIS, number of days missed due to suspensions, and other related outcomes, including academic achievement. Finally, future research should examine the impact of SWPBIS implementation on suspension rates specifically for students with disabilities, particularly students with EBD.

Conclusion

Disciplinary exclusions, including OSS and expulsions, remain a pressing concern for schools. The results of this study suggest that implementing SWPBIS with fidelity significantly and meaningfully reduces schools’ use of OSS, replicating findings from similar studies. This finding, coupled with prior studies, provides empirical support that implementing SWPBIS can reduce suspensions, thus keeping those students in school and increasing their chances of being successful in school. Taken together, the results of this and prior studies provide further evidence that implementing SWPBIS with fidelity matters and can have a lasting impact on the students those schools serve.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Nicholas A. Gage

Nicolette Grasley-Boy

References

Algozzine

Barrett

Eber

George

Horner

Lewis

. . . Sugai

(2014). School-wide PBIS Tiered Fidelity Inventory. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. https://www.pbisapps.org/Resources/SWIS%20Publications/SWPBIS%20Tiered%20Fidelity%20Inventory%20(TFI).pdf

American Academy of Pediatrics. (2013). Policy statement: Out-of-school suspension and expulsion. Pediatrics, 131, e1000–e1007. https://doi.org/10.1542/peds.2012-3932

Anderson

K. P.

Ritter

G. W.

Zamarro

(2019). Understanding a vicious cycle: The relationship between student discipline and student academic outcomes. Educational Researcher, 48(5), 251–262. https://doi.org/10.3102/0013189X19848720

Azur

M. J.

Stuart

E. A.

Frangakis

Leaf

P. J.

(2011). Multiple imputation by chained equations: What is it and how does it work? International Journal of Methods in Psychiatric Research, 20, 40–49. https://doi.org/10.1002/mpr.329

Borenstein

Hedges

L. V.

Higgins

J. P. T.

Rothstein

H. R.

(2009). Introduction to meta analysis. John Wiley.

Bradshaw

C. P.

Koth

C. W.

Bevans

K. B.

Ialongo

Leaf

P. J.

(2008). The impact of school-wide Positive Behavioral Interventions and Supports (PBIS) on the organizational health of elementary schools. School Psychology Quarterly, 23, 462–473. https://doi.org/10.1037/a0012883

Bradshaw

C. P.

Waasdorp

T. E.

Leaf

P. J.

(2012). Effects of school-wide positive behavioral interventions and supports on child behavior problems. Pediatrics, 130, e1136–e1145. https://doi.org/10.1001/archpediatrics.2011.755

Chin

J. K.

Dowdy

Jimerson

S. R.

Rime

W. J.

(2012). Alternatives to suspensions: Rationale and recommendations. Journal of School Violence, 11, 156–173. https://doi.org/doi.org/10.1080/15388220.2012.652912

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.

10.

Coyne

M. D.

Cook

B. G.

Therrien

W. J.

(2016). Recommendations for replication research in special education: A framework of systematic, conceptual replications. Remedial and Special Education, 37, 244–253. https://doi.org/10.1177/0741932516648463

11.

Crone

D. A.

Hawken

L. S.

Horner

R. H.

(2010). Responding to problem behavior in schools: The behavior education program (2nd ed.). Guilford Press.

12.

Dong

Peng

C. Y. J.

(2013). Principled missing data methods for researchers. SpringerPlus, 2, 1–17.

13.

Fabelo

Thompson

M. D.

Plotkin

Carmichael

D. P.

Marchbanks

III Booth

(2011). Breaking schools’ rules: A statewide study of how school discipline relates to students’ success and juvenile justice involvement. Council of State Governments Justice Center. https://csgjusticecenter.org/wp-content/uploads/2012/08/Breaking_Schools_Rules_Report_Final.pdf

14.

Fortson

Verbitsky-Savitz

Kopa

Gleason

(2012). Using an experimental evaluation of charter schools to test whether nonexperimental comparison group methods can replicate experimental impact estimates (NCEE 2012-4019). National Center for Education Evaluation and Regional Assistance.

15.

Gage

N. A.

Grasley-Boy

George

H. P.

Childs

Kincaid

(2019). A quasi-experimental design analysis of the effects of school-wide positive behavior interventions and supports on discipline in Florida. Journal of Positive Behavior Interventions, 21, 50–61. https://doi.org/10.1177/1098300718768208

16.

Gage

N. A.

Lee

Grasley-Boy

George

H. P.

(2018). The impact of school-wide positive behavior interventions and supports on school suspensions: A statewide quasi-experimental analysis. Journal of Positive Behavior Interventions, 20, 217–226. https://doi.org/10.1177/1098300718768204

17.

Gage

N. A.

Leite

Childs

Kincaid

(2017). Average treatment effect of school-wide positive behavioral interventions and supports on school-level academic achievement in Florida. Journal of Positive Behavior Interventions, 19, 158–167. https://doi.org/10.1177/1098300717693556

18.

Gage

N. A.

Stevens

R. N.

(2018). Rigor, replication, and reproducibility: Increasing the relevance of behavioral disorders research. Education and Treatment of Children, 41, 567–588.

19.

Gage

N. A.

Whitford

D. K.

Katsiyannis

(2018). A review of schoolwide positive behavior interventions and supports as a framework for reducing disciplinary exclusions. The Journal of Special Education, 52, 142–151. https://doi.org/10.1177/0022466918767847

20.

Grasley-Boy

Gage

N. A.

Lombardo

(2019). Effects of SWPBIS on disciplinary exclusions for students with and without disabilities. Exceptional Children, 86, 25–39. https://doi.org/10.1177/0014402919854196

21.

Hansen

B. B.

Fredrickson

Buckner

Errickson

Solenberger

(2018). Optmatch: Functions for optimal matching (R package version 0.9-8). https://cran.r-project.org/web/packages/optmatch/

22.

Imai

King

Stuart

Whitworth

(2017). MatchIt: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42(8), 1–28. http://www.jstatsoft.org/v42/i08/

23.

Kang-Brown

Trone

Fratello

Daftary-Kapur

(2013). A generation later: What we’ve learned about zero tolerance in schools. Center on Youth Justice, Vera Institute of Justice. https://storage.googleapis.com/vera-web-assets/downloads/Publications/a-generation-later-what-weve-learned-about-zero-tolerance-in-schools/legacy_downloads/zero-tolerance-in-schools-policy-brief.pdf

24.

Kincaid

Childs

George

H. P.

(2005, 2010). School-Wide Benchmarks of Quality (BoQ) [Unpublished instrument]. Department of Child and Family Studies, University of South Florida. http://flpbs.fmhi.usf.edu/ProceduresTools.cfm

25.

Leite

(2017). Propensity score methods using R. SAGE.

26.

Losen

D. J.

Martin

(2018). The unequal impact of suspension on the opportunity to learn in California: What the 2016-17 rates tell us about progress. Center for Civil Rights Remedies at the UCLA Center for Civil Rights. https://www.civilrightsproject.ucla.edu/research/k-12-education/school-discipline/the-unequal-impact-of-suspension-on-the-opportunity-to-learn-in-ca/Suspension_Impact_California_2018_R6.pdf

27.

Losen

D. J.

Martinez

T. E.

(2013). Out of school and off track: The overuse of suspensions in American middle and high schools. The Center for Civil Rights Remedies at the Civil Rights Project of the University of California at Los Angeles.

28.

McIntosh

Massar

M. M.

Algozzine

R. F.

George

H. P.

Horner

R. H.

Lewis

T. J.

Swain-Bradway

(2017). Technical adequacy of the SWPBIS Tiered Fidelity Inventory. Journal of Positive Behavior Interventions, 19, 3–13. https://doi.org/10.1177/1098300716637193

29.

Mercer

S. H.

McIntosh

Hoselton

(2017). Comparability of fidelity measures for assessing tier 1 school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions, 19, 195–204. https://doi.org/10.1177/1098300717693384

30.

Mitchell

B. S.

Hatton

Lewis

T. J.

(2018). An examination of the evidence-base of school-wide positive behavior interventions and supports through two quality appraisal processes. Journal of Positive Behavior Interventions, 20, 239–250. https://doi.org/1098300718768217

31.

Noltemeyer

A. L.

Ward

R. M.

McLoughlin

(2015). Relationship between school suspension and student outcomes: A meta-analysis. School Psychology Review, 44, 224–240. https://doi.org/10.17105/spr-14-0008.1

32.

OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. (2017). Positive behavioral interventions & supports [Website]. https://www.pbis.org/

33.

Pas

E. T.

Ryoo

J. H.

Musci

R. J.

Bradshaw

C. P.

(2019). A state-wide quasi-experimental effectiveness study of the scale-up of school-wide positive behavioral interventions and supports. Journal of School Psychology, 73, 41–55. https://doi.org/10.1016/j.jsp.2019.03.001

34.

Raghunathan

T. E.

Lepkowski

J. M.

Van Hoewyk

Solenberger

(2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27, 85–96.

35.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). SAGE.

36.

R Core Team. (2014). R: A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.R-project.org/

37.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.

38.

Rumberger

R. W.

Losen

D. J.

(2016). The high cost of harsh discipline and its disparate impact. The Civil Rights Project/Proyecto Derechos Civiles.

39.

Ryoo

J. H.

Hong

Bart

W. M.

Shin

Bradshaw

C. P.

(2018). Investigating the effect of school-wide positive behavioral interventions and supports on student learning and behavioral problems in elementary and middle schools. Psychology in the Schools, 55, 629–643. https://doi.org/10.1002/pits.22134

40.

Shadish

W. R.

Cook

T. G.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Harcourt.

41.

Stuart

E. A.

Azur

Frangakis

Leaf

(2009). Multiple imputation with large data sets: A case study of the children’s mental health initiative. American Journal of Epidemiology, 169, 1133–1139. https://doi.org/10.1093/aje/kwp026

42.

Sugai

Lewis-Palmer

Todd

Horner

R. H.

(2005). School-wide Evaluation Tool, v. 2.1. Educational and Community Supports, University of Oregon.

43.

Sullivan

A. L.

Van Norman

E. R.

Klingbeil

D. A.

(2014). Exclusionary discipline of students with disabilities: Student and school characteristics predicting suspension. Remedial and Special Education, 35, 199–210. https://doi.org/10.1177/0741932513519825

44.

U.S. Department of Education. (2018). California compilation of school discipline laws and regulations. https://safesupportivelearning.ed.gov/sites/default/files/discipline-compendium/California%20School%20Discipline%20Laws%20and%20Regulations.pdf

45.

U.S. Department of Education, National Center for Educational Statistics. (2019). Status and trends in the education of racial and ethnic groups: Indicator 15: Retention, suspensions, and expulsion. https://nces.ed.gov/programs/raceindicators/indicator_RDA.asp

46.

van Buuren

Groothuis-Oudshoorn

. (2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45, 1–67. http://www.jstatsoft.org/v45/i03/

47.

Walker

H. M.

Seeley

J. R.

Small

Severson

H. H.

Graham

B. A.

Feil

E. G.

. . . Forness

S. R.

(2009). A randomized controlled trial of the first step to success early intervention: Demonstration of program efficacy outcomes in a diverse, urban school district. Journal of Emotional and Behavioral Disorders, 17, 197–212. https://doi.org/10.1177/1063426609341645

48.

What Works Clearinghouse. (2015). Procedures and standards handbook, Version 3.0. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_v3_0_standards_handbook.pdf