Relationship Between School-Wide Positive Behavior Interventions and Supports and Academic,Attendance,and Behavior Outcomes in High Schools

Abstract

Attendance, behavior, and academic outcomes are important indicators of school effectiveness and long-term student outcomes. Multi-tiered systems of support (MTSS), such as School-Wide Positive Behavior Interventions and Supports (SWPBIS), have emerged as potentially effective frameworks for addressing student needs and improving student outcomes. Much of the research on SWPBIS outcomes has taken place at the elementary and middle school levels, leaving a need for a more thorough examination of outcomes at the high school level. The purpose of this study was to explore the links between implementation of SWPBIS and academic, attendance, and behavior outcome measures across a large sample of high schools from 37 states. Despite some of the difficulties of SWPBIS implementation at the high school level, evidence suggests positive relationships between SWPBIS implementation and outcomes in behavior and attendance for high schools that implement with fidelity.

Keywords

high school school-wide positive behavioral interventions and supports SWPBIS PBIS attendance behavior

Attendance, behavior, and academic outcomes are important indicators of school effectiveness and long-term student outcomes (Hammond, Linton, Smink, & Drew, 2007). Multi-tiered systems of support (MTSS), such as School-Wide Positive Behavioral Interventions and Supports (SWPBIS), have emerged as potentially effective frameworks for addressing student needs and improving student outcomes.

SWPBIS is a multi-tiered framework that guides the organization of behavior support within a school with the goal of improving both behavior and academic outcomes for all students (Lewis & Sugai, 1999). Schools that are implementing SWPBIS with fidelity (accurately and fluently) clearly define, teach, and reinforce school-wide expectations; make data-based decisions to monitor intervention implementation and student response; differentiate levels of support in response to student need; and establish systems to sustain implementation (Sugai et al., 2010). A leadership team oversees these foundational activities, guiding implementation and monitoring implementation fidelity of the critical features of SWPBIS. Typically, an internal coach or an external evaluator evaluates implementation via an assessment tool (e.g., Benchmarks of Quality [BoQ], School-Wide Evaluation Tool [SET]) to ensure integrity to the SWPBIS model. Implementation has grown from individual schools to districts, states, and regions, and has been supported by a growing base of state- and federal-level policy and funding (e.g. U.S. Department of Education School Climate Transformation Grants).

Behavior interventions are typically organized into three tiers of support. Tier 1, also called universal or school-wide support, is designed for all students and staff and is available across all settings within a school (Sugai & Horner, 2006). Tier 2, or targeted, small group support, addresses the needs of subgroups of students who require support in addition to Tier 1. Targeted support is provided in specific areas such as study skills, social skills, behavior, attendance, or dropout prevention (Crone, Horner, & Hawken, 2004). Tier 3, or individualized support, is provided to students who need intensive individualized supports such as individualized behavior plans or wraparound supports.

According to the National Technical Assistance Center for Positive Behavior Interventions and Supports (PBIS TA Center), SWPBIS has been implemented nationally in more than 20,011 schools across all 50 states and Washington D.C. However, within this network, only 2,606 high schools (or 13% of schools in the database) are implementing SWPBIS. Researchers suggest the unique contextual features in high schools make the adoption of SWPBIS more complex than at lower grade levels (Flannery, Frank, Kato, Doren, & Fenning, 2013). As a result, the adoption and initial implementation process at the high school level may take longer and require adaptations in the typical framework to meet the needs of high schools (Flannery et al., 2013). For example, the larger size of most high schools can make the coordination and implementation of school-wide initiatives, data collection, and monitoring procedures more cumbersome (Bohanon-Edmonson, Flannery, Eber, & Sugai, 2004). Developmentally, students are likely to be more motivated by peer acceptance than adult influence increasing the need for student voice and input into school-wide procedures and initiatives (Murphy, Beck, Crawford, Hodges, & McGaughy, 2001). Student independence creates additional challenges with respect to open campuses and a need for adequate supervision both in school and at extracurricular activities. In addition, high school faculty may be primarily focused on their assigned content area, making it more difficult to carve out time for social skill instruction or intervention (Bohanon, Fenning, Borgmeirer, Flannery, & Malloy, 2009). Finally, high schools may rely more heavily on zero tolerance discipline policies (Skiba & Rausch, 2006) making it more difficult to build faculty and staff support for SWPBIS (Flannery et al., 2013).

Currently, research that explores outcomes of the implementation of the SWPBIS framework in high schools is limited to non-experimental descriptive studies or studies that include small numbers of high schools (Bohanon et al., 2006; Bohanon et al., 2012; Bohanon-Edmonson et al., 2004). A case study in an urban high school documented a decrease in student office discipline referrals and reduced numbers of students requiring Tier 2 or 3 support when the school was implementing Tier 1 with fidelity (Bohanon et al., 2006). This case study was replicated in a second high school with similar results (Bohanon et al., 2009). A more recent study by Flannery, Fenning, Kato, and McIntosh (2014) examined student outcomes across 12 high schools, 8 implementing and 4 not implementing SWPBIS, without random assignment. They documented statistically significant reductions in the number of office discipline referrals in SWPBIS schools. Although available evidence suggests positive outcomes at the high school level, documentation of outcomes across larger samples and experimental studies are critical to determine whether expected student outcomes are similar to elementary and middle school settings.

The purpose of this study was to explore the relationship between exposure to Tier 1 SWPBIS implemented with fidelity and academic, attendance, and behavioral outcome measures across a large sample of high schools. The specific research question that was addressed was as follows:

Research Question 1: To what extent is implementation of SWPBIS (Tier 1) with fidelity at the high school level associated with specific academic, behavioral, or attendance outcomes?

Method

Data Sources and Sample Description

To answer the research question, it was necessary to construct a database with data from multiple sources (see Authors’ Note). We constructed a database across 7 years (2005–2011) by combining data elements from (a) the Office of Special Education Programs (OSEP) National PBIS TA Center database, which is used by schools that have been trained in SWPBIS by affiliates of the PBIS TA Center, and (b) state-level data sets obtained from publicly available data archived at state department websites. The initial sample, identified from the PBIS TA Center’s data set, included 946 high schools that met the following criteria: (a) had a National Center for Educational Statistics (NCES) identification (ID) number, (b) were not listed as an alternative school, and (c) had a reported fidelity score (measured by the SET or BoQ; measures described subsequently) for at least 1 year between 2005–2006 and 2011–2012. We combined these data with select NCES data, verified NCES ID numbers, and removed duplicate schools (e.g., same school but name changed), schools that had closed, and schools listed as alternative in the NCES data set. The resulting final sample included 883 high schools from 37 states. The high schools in this final sample averaged 40% of students on free or reduced-cost lunch and 33% minority students. A total of 22% of schools were from large or mid-sized cities. The average school enrollment was 1,080 students, and the average pupil to teacher ratio was 1 to 16.5.

In addition, we obtained a list of middle schools that were in the same high school district from the PBIS TA Center’s data set. Nine hundred thirty-four middle schools were used in the analysis to control for student exposure to SWPBIS prior to high school.

Research Design

A quasi-experimental interrupted time series design (Shadish, Cook, & Campbell, 2002) was used to explore the relationships between exposure to SWPBIS and selected school-level outcomes. We compared school outcomes after SWPBIS implementation with school outcomes prior to implementation. SWPBIS was implemented at different times across schools, reducing threats to internal validity. Specifically, this study examined the short-term relationship between SWPBIS implemented with fidelity and aggregate outcomes in the areas of academics, attendance, and behavior while controlling for risk factor variables (i.e., school size, percentage free/reduced-cost lunch, percentage minority, pupil–teacher ratio). It is likely that the number of years that a school implements SWPBIS with fidelity may improve the relationship between academic, attendance, and behavior outcomes. However, further analysis of the long-term effect of SWPBIS on these outcome areas was not possible in this study as the number of high schools that had implemented with fidelity for more than 2 years was limited.

Variables

Independent variable

The implementation of SWPBIS (i.e., school-wide universal intervention) with fidelity was the independent variable for this analysis, as measured by two assessments with established psychometric properties: (a) BoQ (Cohen, Kincaid, & Childs, 2007; Kincaid, Childs, & George, 2005), with adequate internal consistency (α = .96), test–retest reliability (r = .94), and inter-rater agreement (M = 89%), and (b) SET (Horner et al., 2004; Sugai, Lewis-Palmer, Todd, & Horner, 2005), also with adequate internal consistency (α = .96), test–retest agreement (r = 97%), and inter-observer agreement (M = 99%). Schools that meet 70% of criteria on the overall BoQ or 80% of criteria on the SET are considered implementing with fidelity (Cohen et al., 2007; Horner et al., 2004).

We used both SET and BoQ measures in this analysis for several reasons. First, both tools are widely used by schools and researchers. Second, many schools have used both measures at different times during their implementation process. Therefore, using both measures in this study provided a more complete understanding of a schools level of implementation across time. We tested a latent fidelity construct as a means of combining the continuous SET and BoQ scores for analysis; however, due to low correlations between these measures (range = 0.241 to 0.668 depending on year and number of each measure reported) and poor fit as a latent construct, this was not possible. Instead, we developed a leveled coding structure (described below) that approximates the use of these measures in practice.

Data coding

Three levels of coding were needed to identify schools that (a) were not yet implementing, (b) had implemented partially but had not yet reached fidelity criteria, and (c) had reached implementation fidelity. We assumed that schools were likely to have some elements of SWPBIS in place before implementing. As such, schools with very low SET or BoQ scores were more similar to non-implementing schools than to schools implementing close to fidelity cutoffs. We determined expected baseline levels through a review of randomized control trial literature where baseline SET scores were reported (Bradshaw, Reinke, Brown, Bevans, & Leaf, 2008; Pas & Bradshaw, 2012) and consultation with SWPBIS experts (Rob Horner, Catherine Bradshaw, Brandi Simonsen, and George Sugai). Although no precedent exists for this type of decision in the literature, a 40% on the SET is halfway to the fidelity cut point of 80% and aligned with baseline SET scores; therefore, 35% on the BoQ was chosen because it is halfway to the BoQ cut point of 70%.

Categorical fidelity variables were developed for each high school and each year (2005–2011). Schools received a code of fidelity = 2 if either the SET score was greater than or equal to 80 or BoQ was greater than or equal to 70 for that year. Schools received a code of fidelity = 1 if a SET score was between 41 and 79 or a BoQ score was between 36 and 69 for that year. Schools received a code of fidelity = 0 if SET scores were less than or equal to 40 or BoQ scores were less than or equal to 35. Frequency of fidelity codes across years is presented in Table 1.

Table 1.

Frequency of SWPBIS Fidelity Codes by Year.

Year	Coded 0 not implementing	Coded 1 partially implementing	Coded 2 fully implementing
2005–2006	802	43	29
2006–2007	760	57	57
2007–2008	716	92	66
2008–2009	678	91	97
2009–2010	591	150	133
2010–2011	478	205	191
2011–2012	432	216	217

Note. Includes manually imputed codes. SWPBIS = School-Wide Positive Behavior Interventions and Supports.

Gaps in reported fidelity

Schools are not required to report annual fidelity measures; thus, the absence of a fidelity measure score did not necessarily indicate that a school was not implementing SWPBIS. Therefore, we developed a set of decision rules to code gaps in fidelity data. First, we never inferred codes for years prior to the first or after the last reported fidelity measure. Second, when 1-year gaps were noted in reported fidelity data and before and after scores were from the same measure, we calculated the mean of the fidelity score immediately prior to and following the gap year, and the resulting code was entered based on that score. We entered a code of 2 for 1-year gaps in reported fidelity measures for which the scores prior to and following the gap were from different measures and if scores before and after the gap indicated that a school was implementing with fidelity (i.e., scores of 70% or higher on the overall BoQ or 80% or higher on the SET). We entered a 1 if before and after measures suggested mixed conclusions about a school’s fidelity level (e.g., SET 40 prior to the gap and BoQ 80 after the gap). We entered a 0 if scores before and after the gap indicated the school was not yet implementing (i.e., SET scores less than 40 and BoQ scores less than 35). Gaps in reported fidelity data longer than 1 year were treated as missing data. Using this procedure, we manually imputed 84 one-year gaps for high schools (9% of sample; 81 high schools).

To account for possible student exposure to SWPBIS implemented with fidelity prior to entering high school, we also developed a categorical variable to indicate middle school implementation prior to high school implementation. We gave high schools a code of 1 if a feeder middle school was implementing with fidelity prior to the high school first reaching fidelity.

Dependent variables

The dependent variables were the school-level average daily attendance rates; a school-level academic variable constructed from average academic performance in reading, language arts, and math; and office discipline referrals (ODRs). Distribution statistics for all dependent variables are shown in Table 2.

Table 2.

Distribution Statistics for Variables.

Variable	M (SD)	Skew (SE)	Kurtosis (SE)	Minimum	Maximum	n
Attendance
2005	91.69 (4.32)	−1.47 (0.17)	2.77 (0.34)	73.00	98.99	203
2006	91.78 (4.00)	−1.29 (0.15)	1.58 (0.31)	78.00	98.10	249
2007	91.82 (4.15)	−2.13 (0.14)	7.59 (0.28)	65.15	97.70	312
2008	92.00 (3.98)	−1.899 (0.14)	4.46 (0.27)	72.39	99.17	320
2009	92.29 (3.55)	−2.74 (0.12)	11.59 (0.24)	67.80	97.90	397
2010	92.38 (3.44)	−2.36 (0.13)	8.00 (0.25)	73.80	97.36	380
2011	92.80 (3.21)	−3.01 (0.13)	15.90 (0.27)	70.20	99.00	331
Behavior
ODR rate 2009	1.63 (1.21)	1.48 (0.23)	2.24 (0.46)	.13	5.91	111
ODR rate 2010	1.51 (1.19)	1.34 (0.20)	1.37 (0.39)	.02	5.38	153
ODR rate 2011	1.94 (1.34)	0.817 (0.17)	−0.01 (0.34)	0.10	6.12	199
Math index
2005	−4.33 (18.18)	−0.74 (0.13)	0.30 (0.26)	−59.10	34.82	354
2006	−6.25 (17.16)	−0.64 (0.13)	0.16 (0.25)	−65.04	30.33	368
2007	−4.38 (16.95)	−0.80 (0.13)	0.70 (0.25)	−65.83	33.40	370
2008	−4.90 (16.69)	−0.76 (0.13)	0.62 (0.25)	−62.63	32.62	370
2009	−4.42 (16.69)	−0.95 (0.13)	1.17 (0.25)	−67.95	31.67	373
2010	−4.43 (16.21)	−0.93 (0.14)	1.12 (0.28)	−66.25	27.32	293
2011	−4.08 (16.17)	−0.70 (0.14)	0.22 (0.28)	−61.53	27.54	293
Reading index
2005	−3.94 (14.50)	−0.84 (0.15)	0.26 (0.29)	−47.81	23.71	284
2006	−4.44 (14.64)	−0.895 (0.14)	0.73 (0.28)	−56.76	22.40	303
2007	−4.70 (16.02)	−0.98 (0.14)	0.80 (0.28)	−60.31	22.55	306
2008	−4.92 (16.04)	−1.09 (0.14)	1.23 (0.28)	−64.90	25.10	307
2009	−4.79 (16.15)	−1.78 (0.14)	1.66 (0.28)	−65.19	24.19	308
2010	−3.78 (14.47)	−1.12 (0.14)	1.24 (0.28)	−56.48	23.22	306
2011	−5.40 (14.91)	−0.92 (0.16)	1.18 (0.32)	−69.59	23.50	228
Language index
2005	−3.94 (15.21)	−0.66 (0.15)	0.64 (0.30)	−58.00	31.02	268
2006	−2.82 (16.22)	−0.49 (0.15)	0.35 (0.29)	−57.01	33.00	284
2007	−4.41 (16.19)	−0.75 (0.15)	0.87 (0.29)	−59.39	32.60	284
2008	−4.77 (16.51)	−0.67 (0.14)	0.65 (0.29)	−60.90	30.02	286
2009	−4.27 (16.74)	−.76 (0.14)	0.56 (0.29)	−57.56	33.50	286
2010	−4.55 (16.21)	−0.87 (0.14)	0.96 (0.29)	−60.39	28.50	287
2011	−4.80 (17.35)	−0.92 (0.21)	0.72 (0.41)	−59.77	29.33	140
Risk variables
Free/reduced-cost lunch	0.40 (0.20)	0.48 (0.09)	−0.26 (0.17)	0	0.92	804
Average % minority	0.33 (0.31)	0.85 (0.08)	−0.52 (0.17)	0	1	842
Average enrollment	1080.06 (738.47)	0.80 (0.08)	0.39 (0.17)	53	4,052.67	841
Pupil–teacher ratio	16.46 (3.49)	0.17 (0.08)	−0.18 (0.17)	7.8	26.75	838

Note. Maryland attendance rates when at or above 95% are reported as ≥95% and are coded 95% in this data set, contributing to slightly elevated kurtosis scores for these variables. SE = standard error; ODR = office discipline referrals.

Attendance

Average daily attendance was calculated by dividing the total number of days in attendance for all students by the total number of school days. We obtained attendance data from 12 states and between 203 and 397 schools, depending on the year.

Academic performance

We used aggregate percentage proficient data in reading, math, and writing/language arts subject areas to create a latent variable (academics) for this analysis. Due to a lack of comparable tests or standardized scoring and reporting across states and across time, the number of states from which we obtained academic performance data was limited to a subsample of the five states with the most schools (Maryland, Oregon, Wisconsin, Colorado, and Illinois). We obtained data from 140 to 373 high schools, depending on year and subject area. Each state test has established psychometric properties that can be found through state department websites (Illinois, ACT and the Illinois Department of Education, 2008; Wisconsin, CTB/McGraw Hill, 2006; Colorado, CTB/McGraw Hill, 2007; Maryland, Educational Testing Service, 2005; Oregon, Oregon Department of Education, 2007).

To compare scores across states, an academic index variable was created by determining the difference between the state mean percentage proficient or above proficient and the school’s percentage proficient or above proficient in each academic area. The index variable then represented the percentage above or below the state mean for a given school. We used these index variables for each subject area to create a combined latent academic performance variable as described below.

Behavioral indicators

ODRs are defined at the local level, making cross-state or cross-district comparisons difficult. The School Wide Information System (SWIS; www.swis.org) provides common definitions of behavior infractions. We collected SWIS behavioral data on ODRs from the PBIS TA Center’s data set. We obtained data from schools within 24 states (California, Colorado, Connecticut, Georgia, Illinois, Kansas, Kentucky, Maryland, Michigan, Minnesota, Missouri, Montana, North Carolina, North Dakota, New York, Ohio, Oregon, Pennsylvania, Rhode Island, South Carolina, Tennessee, Vermont, Washington, Wisconsin) from 111 to 199 schools depending on year. For comparison purposes, we converted reported ODR counts to rate per student by dividing the total count by the NCES student enrollment. Due to low numbers of reporting schools prior to the 2009–2010 school year, ODR rates are only included in the analysis for the years 2009 through 2011.

Risk factors

The following NCES variables were used to account for risk in this analysis because they are associated with student academic attendance or behavioral risk (Suh & Suh, 2007): total students receiving free or reduced-cost lunch, total minority students, pupil–teacher ratio, and school size. To compare the count variables (free/reduced-cost lunch, minority) across schools, we converted total numbers of students receiving free or reduced-cost lunch and minority status with percentages by dividing number of students in each category by the total enrollment. Pupil–teacher ratio did not require conversion for comparison purposes. Distribution statistics for risk variables are provided in Table 2.

Analysis

Descriptive analysis

We performed a descriptive analysis of each variable using IBM SPSS Statistics Version 21. We examined data outliers and univariate normality. We checked all outliers for data entry accuracy against original data sets and school type. All measures of skew and kurtosis (with the exception of attendance rates, see Table 2 note) fell within acceptable ranges (absolute skew statistic < 3, absolute kurtosis statistic < 10) for full maximum likelihood estimation procedures (Kline, 2011). Multivariate normality was assumed because univariate distributions were normal, each pair of variables was normal, and bivariate scatter plots were linear.

Every effort was made to obtain as complete a data set as possible for each outcome area and from as many schools as possible; however, because we used extant data sources, not all data were available for all years, schools, and subject areas. To address missing data, we used full maximum likelihood estimation (FIML). This method does not delete cases or impute missing observations. Instead, relevant statistical information from subsets of the data with similar missing data patterns is used to estimate parameters and standard errors (Enders, 2010).

Structural equation model

We used structural equation modeling (SEM) to examine relationships between SWPBIS implementation and the outcome variables. Analyses were run using IBM SPSS AMOS Graphics Version 20. SEM allows researchers to test the fit of a conceptual model to observed data and allows for the use of latent variables. We measured the fit between our conceptual models and our data with multiple model fit indices. The root mean square error of approximation (RMSEA) is an absolute fit measure. Good or close fit is indicated by a RMSEA score ≤ 0.05. Scores between 0.05 and 0.08 indicate acceptable fit. Scores between 0.08 and 0.1 indicate mediocre fit. Scores above 0.1 indicate a poorly fit model (Little, 2013). The comparative fit index (CFI) and Tucker–Lewis/non-normed fit index (TLI) are relative fit indices that can be used to compare the specified model with the null model (Bentler & Bonett, 1980; Tucker & Lewis, 1973). For both of these fit indices, scores ≥ 0.95 indicate good/close fit. Scores between 0.9 and 0.95 indicate acceptable fit, and scores below 0.9 indicate mediocre or poor fit (Little, 2013).

We constructed and tested latent variables in the areas of academics and risk factors and used confirmatory factor analysis (CFA) procedures to test for model fit and factorial invariance across time. Only academics (math, reading, and language arts) met criteria for good fit as a latent construct as determined by a change in CFI that is 0.01 or less across measurement model adjustments (Cheung & Rensvold, 2002). Results are included in Table 3.

Table 3.

Model Fit for Latent Variable and Final Models in Each Outcome Area.

Latent variable/models	χ²	df	RMSEA	CFI	TLI
CFA for academic latent variable
Null model	9,952.19	231	2.53
Strong factor invariance	359.05	135	0.044	0.977	0.961
Initial growth models
Academic growth	433.79	154	0.046	0.971	0.951
Attendance growth	80.69	23	0.054	0.973	0.967
ODRs 2008–2011	27.93	8	0.053	0.914	0.893
Final models with SWPIBS
Academic with SWPBIS	1,393.27	608	0.038	0.944	0.924
Attendance with SWPBIS	441.63	209	0.036	0.960	0.933
Behavior with SWPBIS	322.52	90	0.054	0.903	0.835

Note. We only report model fit measures for final models. For tested but rejected model fit measures, contact first author. RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker–Lewis/non-normed fit index; CFA = confirmatory factor analysis; ODR = office discipline referrals; SWPBIS = School-Wide Positive Behavior Interventions and Supports.

We constructed and tested separate models in each outcome area (academics, behavior, attendance). We compared model fit for growth curve, autoregressive, piecewise, and latent basis models for each outcome variable model. The final best fitting model for each outcome area is described below, and model fit measures are included in Table 3.

To assess the relationship between Tier 1 SWPBIS fidelity on each outcome variable, we then added SWPBIS fidelity variables to each growth model as time varying covariates, which affect outcomes at each time point, and added mean-centered risk factors and middle-school implementation as time invariant covariates to predict the level and slope across all time points for each outcome variable. The effect of fidelity on outcome measures was then set equal across time to assess the main effects of SWPBIS on each outcome measure. Figure 1 illustrates the model used for this analysis.

Figure 1.

Structural model for outcome variable analysis.

Results

In this section, we report the results for structural models for each outcome area. We provide a summary of important parameter estimates for all models in Table 4.

Table 4.

Parameter Estimates.

Model	Estimate	SE	p value
Academic model
Regression
Slope ← Middle school implementation	0.08	0.16	.602
Intercept ← Middle school implementation	−2.28	1.01	.024
Intercept ← % free or reduced-cost lunch	−36.86	2.88	<.001
Intercept ← % minority	−14.81	1.81	<.001
Academics ← Fidelity1^a	−0.13	0.33	.693
Academics ← Fidelity2^b	0.26	0.36	.536
Intercept
Slope	0.10	0.10	.329
Intercept	−4.19	0.64	<.001
Attendance model
Regression
Slope ← Middle school implementation	0.01	0.06	.875
Intercept ← Middle school implementation	−0.04	0.37	.909
Intercept ← Free or reduced-cost lunch	−5.65	0.95	<.001
Slope ← Pupil–teacher ratio	0.05	0.01	<.001
Intercept ← Pupil–teacher ratio	−0.32	0.05	<.001
Intercept ← % minority	−1.90	0.61	.002
Attendance rate ← Fidelity1^a	0.30	0.11	.009
Attendance rate ← Fidelity2^b	0.51	0.13	<.001
Intercept
Slope	0.07	0.04	.068
Intercept	91.92	0.23	<.001
Behavior model
Regression
Slope ← Middle school implementation	0.00	0.07	.959
Intercept ← Middle school implementation	0.08	0.21	.695
ODR rate ← Fidelity1^a	−0.81	0.09	<.001
ODR rate ← Fidelity2^b	−1.07	0.09	<.001
Slope ← Enrollment	0.00	0.00	.001
Intercept ← Enrollment	−0.00	0.00	<.001
Intercept
Slope	−0.08	0.04	.053
Intercept	2.77	0.13	<.001

Note. Academic refers to the latent construct. We only report main effects of fidelity. For time varying effects, please contact first author. SE = standard error; ODR = office discipline referrals.

Main effects when fidelity = 1. ^bMain effects when fidelity = 2.

Academics

Results from this model indicated that high schools that are not implementing SWPBIS, without implementing middle schools, and with average enrollment, percentage of free or reduced-cost lunch, percentage minority, and pupil–teacher ratios begin with an academic index score of −4.19 (p < .001), or about 4 points below the state average, and grow about 0.10 (p = .33) points per year. The slope is not statistically different from 0, indicating relatively stable scores across time. The correlation between slope and intercept errors is −0.38 and is statistically significant (p < .001), indicating that schools with lower starting scores grow faster than schools with higher starting scores.

High school fidelity estimates were not related to academic outcomes in a statistically significant manner; effects were negative (−0.13, p = .69) for schools that had not reached fidelity and positive (0.26, p = .54) for schools that were implementing with fidelity although neither is statistically different from 0. Middle-school fidelity did not contribute significantly to the slope (0.08, p = .60); however, a negative and statistically significant effect on the intercept −2.28 (p = .02) indicated that high schools with implementing feeder middle schools tended to have lower starting academic scores. Percentage free or reduced-cost lunch and percentage minority had a statistically significant negative effect on the intercept but no statistically significant effect on the slope holding all other covariates constant. In other words, schools with higher levels of risk had significantly lower initial academic scores.

Attendance

Results indicated an overall intercept of 91.92 (p < .001; significant) and slope of 0.07 (p = .07; marginally significant) for high schools that were not implementing SWPBIS, without implementing middle schools, and with average enrollment, percentage minority, percentage free or reduced-cost lunch, and pupil–teacher ratios. The intercept is statistically significant, and the slope is marginally significant, indicating that schools increase their attendance rate about 0.07 per year. The slope and intercept error correlation is statistically significant and negative (−0.38, p < .001), indicating that schools starting with higher attendance rates tend to grow significantly more slowly.

The effect of SWPBIS fidelity on attendance was statistically significant and positive for schools that were implementing with fidelity (fidelity = 2; 0.51, p < .001) and for schools not yet at fidelity (fidelity = 1; 0.30, p = .009). In other words, schools that were approaching or at fidelity had significantly higher levels of attendance across time than those that were not yet at fidelity criteria. Prior middle-school implementation did not contribute to the model. Percentage free or reduced-cost lunch had a negative and statistically significant effect on the intercept (−5.65, p < .001) but a non-statistically significant effect on the slope. In other words, schools with higher percentage of free or reduced-cost lunch had significantly lower attendance rates, but the rate of change across time did not differ significantly from schools with lower rates of free or reduced-cost lunch. Pupil–teacher ratios had a negative and statistically significant effect on the intercept (−0.32, p < .001) but a positive and significant effect on the slope (0.05, p < .001). Higher pupil–teacher ratios were associated with significantly lower initial attendance rates, but significantly more growth across time. Average percentage minority had a negative and statistically significant effect on the intercept (−1.90, p = .03), indicating that schools with higher minority populations had a significantly lower average daily attendance rate.

Behavior

Results indicated an overall intercept of 2.77 (p < .001; significant) and an overall slope of −0.08 (p = .053; marginally significant), indicating that high schools that were not implementing SWPBIS; did not have an implementing middle school; and had average enrollment, minority populations, pupil–teacher ratios, and students eligible for free or reduced-cost lunch had a starting ODR rate of 2.77 and reduced their ODR rate by an average of 0.08 per year. The slope error and intercept error correlation was negative and statistically significant (−0.56, p = .04), indicating that schools with higher starting ODR rates had more negative slopes, or significantly more decline in ODR rates across time.

The effect of fidelity on ODR rates indicated statistically significant decreases (−0.81, p < .001) for schools approaching fidelity and for schools at fidelity (−1.07, p < .001) as compared with the reference group. That is, schools that were approaching or at fidelity had significantly lower ODR rates than schools that were not implementing. The effects of middle-school implementation prior to high school implementation were not statistically significant. Average free or reduced-cost lunch had a positive and statistically significant effect (1.14, p < .001) on the intercept, indicating that schools with that risk factor had significantly higher initial ODR rates.

Discussion

In this study, we investigated the effects of SWPBIS on specific behavior, attendance, and academic outcomes in a large sample of high schools. Implementation of SWPBIS with fidelity was associated with reductions in ODR rates and increases in attendance rates; however, we did not find an effect on academic performance. In prior research across grade levels, SWPBIS has been associated with improvements in behavior (e.g., Bradshaw, Koth, Thornton, & Leaf, 2009) and attendance (e.g., Caldarella, Shatzer, Gray, Young, & Young, 2011). The relationship of SWPBIS to academic performance in previous research has shown mixed results, with some studies indicating increased academic performance in SWPBIS schools (e.g., Horner et al., 2009) and others indicating no relationship between SWPBIS and academic performance (e.g., Gage, Sugai, Lewis, & Brzozowy, 2015). SWPBIS is not expected to affect academic outcomes within first year of fidelity or in the absence of effective instruction (Lassen, Steele, & Sailor, 2006). In addition, given the reduced sample of high schools that have implemented with fidelity for more than 2 years, it was not possible to measure longer-term effects of SWPBIS on academic outcomes with this data set. Future research is needed to further assess the long-term effects of SWPBIS implementation at the high school level.

Study Limitations

The results from this study must be interpreted in light of several limitations related to study design, missing data, measures of fidelity, and school-level data. First, the study design is not a true experimental design; therefore, no causal inferences can be drawn from these results. Although outcome data for attendance and academics were available pre- and post-SWPBIS implementation, behavioral data were only available after schools began using the SWIS database and therefore were involved with the PBIS TA Center network. Risk factors were controlled for in the models; however, complete control for all threats to internal and external validity is not possible without random assignment.

Second, data were not missing at random. Data were obtained from 37 states, but not all data variables were available from all states. Significant effort was put into ensuring that all available data were found from as many states as possible, but some states only make available outcome variables at the district level rather than the school level.

Third, we used both SET and BoQ fidelity measures to ensure adequate sample sizes for this study; however, due to low correlations between SET and BoQ scores, combining these measures using a latent variable construct was not possible. Instead, coding was created to indicate a school’s level of implementation fidelity. Because schools were not required to report scores each year, a set of decision rules was established to address “gap” years. Every attempt was made to base these decisions on prior research and expert opinion; however, this coding may not be completely accurate and could have affected the results. In addition, a confounding effect may exist between measuring and actual implementation fidelity, as the very process of measuring fidelity likely improves implementation processes. Although categorical coding partially addresses this concern, the confounding effect may have affected results.

Fourth, we did not account for implementation interventions at Tiers 2 or 3 or other school initiatives that may have been in place and may have had effects on outcome measures. Measures for assessing SWPBIS implementation at advanced tiers are available (e.g., Anderson et al., 2011) but not widely used or reported at the high school level at the time of this study.

Finally, extant school-level, rather than student-level, data were used for this study. Individual student exposure to SWPBIS may vary as students transfer in or out of schools. However, given that SWPBIS is a school-wide intervention and this study was focused on the effects of the overall school context on outcome measures, school-level outcome data provided an adequate measure of overall school-level effects.

Implications

The purpose of this study was to understand the relationships between Tier 1 SWPBIS, a school-level intervention, and school-level outcomes in a large sample of high schools. Although clear causal conclusions or information about student-level outcomes are not supported, recommendations are indicated for future practice and research in the areas of SWPBIS.

In terms of practice, the outcomes of this study provide a rationale for high schools to consider using the multi-tiered SWPBIS decision-making framework to improve attendance and decrease behavior referrals. Given the effects of status risk factors, results from this study also suggest that schools must consider the cultural and contextual relevance of interventions. Although a full discussion of contextual and culturally relevant implementation is beyond the scope of this article, Sugai, O’Keeffe, and Fallon (2012) provide a number of suggestions for implementing SWPBIS in a contextually and culturally relevant way.

In the area of research, the results from this study provide an overview of relationships between SWPBIS and important outcome measures at the high school level. However, additional research is needed to fully understand these relationships. Randomized controlled trials at the high school level, which look at the effects of SWPBIS on each of these outcome areas, are needed to establish causal relationships. Second, future research studies should include student-level outcomes and fully account for individual student exposure to SWPBIS. School-level indicators provide a general look at relationships, but specific student-level outcomes will be important to guide future implementation efforts.

Research at both the school and student levels that explores the effects of SWPBIS across race/ethnic groups and socioeconomic levels is a critical next step. Some evidence suggests that SWPBIS may lead to improvements for students across all groups but may not be closing the gap between student groups (Vincent, Randall, Cartledge, Tobin, & Swain-Bradway, 2011). To improve our understanding about implementation of SWPBIS in culturally and contextually relevant ways, researchers first must understand how current practices are affecting different groups of students. In addition, research that highlights effective interventions for closing the outcome gap between racial, cultural, and socioeconomic groups in high schools should be a top priority for the field.

Conclusion

Studies that assess the effects of SWPBIS on outcomes at the high school level have been limited in scope and rigor. SWPBIS has been associated with positive outcomes in the areas of attendance, behavior, and in some cases, academics; however, much of this research has been conducted at the elementary and middle-school levels (Flannery et al., 2013). The implementation of SWPBIS at the high school level has been shown to take more time and may require some specific modifications of the SWPBIS framework to fit the unique high school context (Flannery et al., 2013). An understanding of the relationship between SWPBIS implementation and school-level outcome measures across a large sample of high schools is critical for informing and guiding implementers, policy makers, and researchers.

The results from this study provide an overview of the relationship between SWPBIS and academic, attendance, and behavior outcomes in a large sample of schools. Evidence suggests positive relationships between SWPBIS implementation and outcomes in behavior and attendance confirming that despite some of the difficulties of SWPBIS implementation at the high school level, positive outcomes can be expected for schools that implement with fidelity.

Footnotes

Acknowledgements

Authors wish to acknowledge Dr. Renee Bradley, U.S. Office of Special Education Programs, for her contribution to and support of this research.

Action Editor: Lee Kern

Authors’ Note

This data set was also used to test the relationships between SWPBIS implementation and high school dropout rates through a more complex combined model. The text describing the database construction, independent and dependent variables, and coding process is also described in . Opinions expressed herein are the authors’ and do not reflect necessarily the position of the U.S. Department of Education, and such endorsements should not be inferred.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The development of this article was supported in part by a grant from the Office of Special Education Programs, U.S. Department of Education (H029D40055).

References

ACT and the Illinois Department of Education. (2008). Prairie state achievement examination technical manual 2008 testing cycle. Retrieved from http://www.isbe.state.il.us/assessment/pdfs/psae/tech_manual08.pdf

Anderson

Childs

K. E.

Kincaid

Horner

George

H. P.

Todd

A. W.

. . . Spaulding

(2011). Benchmarks for advanced tiers. Eugene: Educational and Community Supports, University of Oregon.

Bentler

P. M.

Bonett

D. G.

(1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606.

Bohanon

Fenning

Borgmeirer

Flannery

K. B.

Malloy

(2009). Finding direction for high school positive behavior supports. In Sailor

Dunlap

Sugai

Horner

(Eds.), Handbook of positive behavior support (pp. 581–602). New York, NY: Springer.

Bohanon

Fenning

Carney

K. L.

Minnis-Kim

Anderson-Harriss

Moroz

K. B.

Pigott

T. D.

(2006). School-wide application of positive behavior support in an urban high school: A case study. Journal of Positive Behavior Interventions, 8, 131–145.

Bohanon

Fenning

Hicks

Weber

Their

Aikins

. . . Irving

(2012). A case example of the implementation of school-wide positive behavior supports in a high school setting using change point test analysis. Preventing School Failure, 56, 91–103.

Bohanon-Edmonson

Flannery

K. B.

Eber

Sugai

(2004). Positive behavior support in high schools: Monograph from the 2004 Illinois High School Forum of Positive Behavioral Interventions and Supports. Unpublished manuscript, University of Oregon, Eugene.

Bradshaw

C. P.

Koth

C. W.

Thornton

L. A.

Leaf

P. J.

(2009). Altering school climate through school-wide positive behavioral interventions and supports: Findings from a group-randomized effectiveness trial. Prevention Science, 10, 100–115.

Bradshaw

C. P.

Reinke

W. M.

Brown

L. D.

Bevans

K. B.

Leaf

P. J.

(2008). Implementation of school-wide positive behavioral interventions and supports (PBIS) in elementary schools: Observations from a randomized trial. Education and Treatment of Children, 31, 1–26.

10.

Caldarella

Shatzer

R. H.

Gray

K. M.

Young

E. L.

(2011). The effects of school-wide positive behavior support on middle school climate and student outcomes. RMLE Online: Research in Middle Level Education, 35, 1–14.

11.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255.

12.

Cohen

Kincaid

Childs

(2007). Measuring school-wide positive behavior support implementation: Development and validation of the benchmarks of quality. Journal of Positive Behavior Interventions, 9, 203–213.

13.

Crone

Horner

R. H.

Hawken

L. S.

(2004). Responding to problem behavior in schools: The behavior education program. New York, NY: Guilford Press.

14.

CTB/McGraw Hill. (2006). Fall 2005 WKCE-CRT technical manual. Retrieved from http://oea.dpi.wi.gov/files/oea/pdf/td-2005-techman.pdf

15.

CTB/McGraw Hill. (2007). Colorado student assessment program technical report 2007. Retrieved from http://www.cde.state.co.us/assessment/CoAssess-AdditionalResources.asp#Techreport

16.

Educational Testing Service. (2005). Maryland high school assessment technical report. Maryland State Department of Education. Retrieved from http://www.marylandpublicschools.org/NR/rdonlyres/713CB109-D8CE-44FC-AC0A-122D47965A52/9862/TechReport_July12FINALwithedits.pdf

17.

Enders

C. K.

(2010). Applied missing data analysis. New York, NY: Guilford Press.

18.

Flannery

K. B.

Fenning

Kato

M. M.

McIntosh

(2014). Effects of school-wide positive behavioral interventions and supports and fidelity of implementation on problem behavior in high schools. School Psychology Quarterly, 29, 111–124. doi:10.1037/spq0000039

19.

Flannery

K. B.

Frank

Kato

Doren

Fenning

(2013). Implementing school-wide positive behavior support in high school settings: Analysis of eight high schools. The High School Journal, 96, 267–282.

20.

Freeman

Simonsen

McCoach

D.B.

Sugai

Lombardi

Horner

(in press). An analysis of the relationship between implementation of school-wide positive behavior interventions and supports and high school attendance and dropout rates. The High School Journal.

21.

Gage

N. A.

Sugai

Lewis

T. J.

Brzozowy

(2015). Academic achievement and school-wide positive behavior supports. Journal of Disability Policy Studies, 25, 199–209.

22.

Hammond

Linton

Smink

Drew

(2007). Dropout risk factors and exemplary programs: A technical report. Clemson, SC: National Dropout Prevention Center.

23.

Horner

R. H.

Sugai

Smolkowski

Eber

Nakasato

Todd

A. W.

Esperanza

(2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior, 11, 133–144.

24.

Horner

R. H.

Todd

A. W.

Lewis-Palmer

Irvin

L. K.

Sugai

Boland

J. B.

(2004). The school-wide evaluation tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6, 3–12.

25.

Kincaid

Childs

George

(2005). School-wide benchmarks of quality. Unpublished instrument, University of South Florida, Tampa.

26.

Kline

(2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.

27.

Lassen

S. R.

Steele

M. M.

Sailor

(2006). The relationship of School-Wide Positive Behavior Support to academic achievement in an urban middle school. Psychology in the Schools, 43, 701–712. doi:10.1002/pits.20177

28.

Lewis

T. J.

Sugai

(1999). Effective behavior support: A systems approach to proactive school wide management. Focus on Exceptional Children, 31, 1–24.

29.

Little

(2013). Longitudinal structural equation modeling. New York, NY: Guilford Press.

30.

Murphy

Beck

Crawford

Hodges

McGaughy

C. L.

(2001). The productive high school: Creating personalized academic communities. Thousand Oaks, CA: Corwin.

31.

Oregon Department of Education. (2007). Oregon’s statewide assessment system technical report: Reliability and validity (Vol. 4). Retrieved from http://www.ode.state.or.us/teachlearn/testing/manuals/2007/asmttechmanualvol4_validity.pdf

32.

Pas

Bradshaw

(2012). Examining the association between implementation and outcomes: State-wide scale up of School-Wide Positive Behavior Intervention and Supports. Journal of Behavioral Health Services and Research, 39, 417–433.

33.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton–Mifflin.

34.

Skiba

R. J.

Rausch

M. K.

(2006). School disciplinary systems: Alternatives to suspension and expulsion. In Bear

G. G.

Minke

K. M.

(Eds.), Childrens’ needs III: Development, prevention, and intervention (pp. 87–102). Bethesda, MD: National Association of School Psychologists.

35.

Sugai

Horner

R. H.

(2006). A promising approach for expanding and sustaining school-wide positive behavior support. School Psychology Review, 35, 245–259.

36.

Sugai

Horner

R. H.

Algozzine

Barrett

Lewis

Anderson

. . . Simonsen

(2010). School-Wide Positive Behavior Support: Implementers’ blueprint and self-assessment. Eugene: University of Oregon.

37.

Sugai

Lewis-Palmer

Todd

A. W.

Horner

R. H.

(2005). School-wide evaluation tool (Version 2.1). Eugene: Educational and Community Supports, University of Oregon. Retrieved from https://www.pbisassessment.org/Evaluation/Surveys

38.

Sugai

O’Keeffe

Fallon

(2012). A contextual consideration of culture and school-wide positive behavior support. Journal of Positive Behavior Supports, 14, 197–208.

39.

Suh

(2007). Risk factors and levels of risk for high school dropouts. Professional School Counseling, 10, 297–306.

40.

Tucker

L. R.

Lewis

(1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10.

41.

Vincent

C. G.

Randall

Cartledge

Tobin

T. J.

Swain-Bradway

(2011). Toward a conceptual integration of cultural responsiveness and school wide positive behavior support. Journal of Positive Behavior Interventions, 13, 219–229.