The Relationship Between Implementation of School-Wide Positive Behavior Intervention and Supports and Performance on State Accountability Measures

Abstract

This study examined data from 96 schools in a Southeastern U.S. state participating in training and/or coaching on School-Wide Positive Behavioral Interventions and Supports (SWPBIS) provided by the State Personnel Development Grant (SPDG) in their state. Schools studied either received training only (“non-intensive” sites) or training and on-site coaching (“intensive” sites). Fidelity of implementation was self-evaluated by both types of schools using the Benchmarks of Quality (BOQ). Some schools were also externally evaluated using the School-Wide Evaluation Tool (SET), with those scoring 80% or higher determined “model sites.” Using an independent sample t-test, analyses revealed statistically significant differences between intensive and nonintensive schools’ Quality of Distribution Index (QDI) scores and between model sites and nonmodel sites on QDI scores. Correlations were performed to determine whether the fidelity of implementation of SWPBIS as measured by the BOQ was related to any of the state’s accountability measures: performance classification, QDI, or growth.

Keywords

positive behavior interventions and supports accountability professional development mentoring

The evolving dynamics in today’s fast-paced life has brought about varied changes, especially in the field of education. The use of a more rigorous curriculum and demanding accountability measures are salient factors educators and school districts across the country consider. Meeting these curricular and accountability demands is negatively impacted by disruptive behavior inside the classroom. This kind of behavior threatens to diminish the quality and the amount of time devoted to the academic instruction students receive inside the classroom (Cotton, 1990; Oliver, Wehby, & Reschly, 2011; Walker, Ramsey, & Gresham, 2005). The loss of instructional time due to recurrent classroom disruptions has been cited as a factor negatively affecting the scores of U.S. students in Reading and Math (Lassen, Steele, & Sailor, 2006; Simonsen et al., 2012).

School-Wide Positive Behavior Interventions and Supports (SWPBIS) have been used in schools for more than a decade to improve student behavior, and ultimately student performance (Sugai & Horner, 2001). With increased attention focused on student outcomes through accountability requirements of laws such as No Child Left Behind (NCLB, P.L. 107-110), many states have adopted policies and practices designed to identify and remediate students’ academic difficulties early (Sugai & Horner, 2006). At the same time, there is movement across the nation to make state standards more uniform, and comparisons among state’s performance more easily done (Common Core State Standards, 2013). In light of these distinct, yet related educational reforms, it is important to investigate the relationships between the implementation of Positive Behavior Interventions and Supports (PBIS) and the accountability measures used in this Southeastern state to evaluate schools and school districts.

The Southeastern state in which this research took place has endured a long cycle of poverty (Noss, 2012) and low academic attainment compared with other states (Mississippi Department of Education, 2010). The socioeconomic composition of the population in the region may make the student population vulnerable to socioeconomic factors (Davis-Kean, 2005; Gassman-Pines & Yoshikawa, 2006). Students who live in poverty have been underserved in educational settings (Duncan & Magnuson, 2005; Evans, 2004; Wamba, 2010), which may impede the chances of educational success. Paolella (2009) noted that children in poverty and minority groups may be more predisposed to exhibit discipline and/or behavioral issues in the classroom.

Increasing student performance is no easy feat, particularly in states where poverty, mobility due to catastrophic natural events, and low academic attainment are prevalent (Smith, Fien, & Paine, 2008). The characteristics of the population in the region, unsatisfactory results of the students in the state tests, high dropout rates, and low graduation rates called for the implementation of strategies to help minimize class disruptions and maximize instructional time (Sugai & Horner, 2006). States in the southern United States have long struggled to overcome circumstances such as these through educational initiatives (Berry & Fuller, 2008). Noss’ (2012) reported the state where this study took place has the lowest household income in the country ($36,919.00), with a poverty rate of 22.6% (Bishaw, 2012).

In 2005, the State Board of Education adopted the Three-Tier Instructional Model (MDE, 2012) to meet students’ needs. Tier 1 refers to the quality of the classroom instruction based on the state’s Curriculum Frameworks (MDE, 2012), Tier 2 refers to the focused supplemental instruction, whereas Tier 3 deals with intensive interventions to meet the students’ individual needs that include instructional and/or behavioral needs. While there are several initiatives taking place in the state, the primary support structure for schools implementing SWPBIS is the State Personnel Development Grant (SPDG).

Originally developed prior to the current accountability mandates and measures, this state’s SPDG was designed to address students’ behavioral needs to decrease dropout rates and increase graduation rates in the state (MDE, 2010). The state’s SPDG goal was to provide the training necessary for teachers to improve classroom management that may lead to a better classroom climate and school climate (Komro, Flay, & Biglan, 2011). There is a plethora of research about the influence of PBIS on academic achievement that may result in higher test scores (Jia et al., 2009; Rowe & Stewart, 2009; Sugai & Horner, 2001, 2006). This study seeks to contribute to the emerging literature on the relationship of SWPBIS implementation fidelity measures and accountability outcomes of participating schools.

Accountability

In 1965, U.S. president Lyndon Johnson declared war on poverty with the implementation of The Elementary and Secondary Education Act (ESEA; Public Law 89-10). The reform provided financial support to local education agencies serving children who came from socioeconomically disadvantaged homes (Irons & Harris, 2007) in an effort to improve academic achievement and thus close the achievement gap (Lassen et al., 2006). When state and local educational agencies accept money from the U.S. Department of Education as authorized by Title I of the ESEA legislation, they also obligate themselves to following regulations imposed by those laws, including increasingly demanding accountability requirements (Manzo, 2000). The 2001 reauthorization of ESEA, also known as No Child Left Behind (NCLB, P.L. 107-110), highlighted school districts’ accountability for students’ achievement, or lack of it. Since it is required to assess students and monitor academic achievement, states have adopted various indicators of educational outcomes in an effort to meet these demands (Vaughn, Bos, & Schumm, 2010).

No Child Left Behind’s required accountability systems not only prompted schools to look more closely at the academic outcomes of their students, but also the underlying factors that supported or impeded academic achievement (NCLB, 2002). The measure promoted the creation of the states’ own accountability systems to evaluate school districts, schools, and teachers (Irons & Harris, 2007). At the same time, the Response to Intervention (RtI) movement gave a way to monitor progress toward accountability targets as stated by Carney and Stiefel (2008). These systems provide a framework for schools to provide interventions and supports to students as they encounter difficulty in the classroom (MDE, 2010). Most states’ RtI models include academic and behavioral components, recognizing that these two aspects are interrelated and must be addressed when attempting to facilitate maximal student achievement (Carney & Stiefel, 2008).

The Southeastern state’s RtI model, where the study took place, is designed to provide students with the academic and behavioral supports required for students to succeed in the classroom (MDE, 2010). The model may help schools and school districts promote academic achievement that may lead to students’ higher scores on the state’s high-stakes test. The test is administered to students toward the end of the school year and the scores are used to determine if schools and/or school districts met the state’s accountability requirements. The state’s accountability system, composed of state and federal components (MDE, 2010), began implementing Quality of Distribution Index (QDI) during the 2008-2009 school year. QDI is calculated using data from the MCT2 (Mississippi Curriculum Test, Second Edition) language arts and mathematics tests, Subject Area Testing Program (SATP) data from the Algebra I, Biology I, English II, and U.S. History tests, and the results from the language arts and mathematics sections of the Mississippi Alternate Assessment of Extended Curriculum Frameworks (MAAECF; MDE, 2013) . The resulting score is then utilized to rank schools and school districts as follows: A, Star School; B, High Performing; C, Successful; D, Academic Watch; and F, Low Performing, At-Risk of Failing, and Failing. The use of both Performance Classifications allows districts, schools, and parents to understand how the former classification, used during the 2011-2012 school year, relates to the letter grades approved by the State Board of Education for the 2012-2013 school year (MDE, 2013). The Performance Classification summarizes the performance of schools and school districts after all the state’s accountability measures have been accounted for.

According to the Southeastern State Department of Education (2013) for the 2011-2012 and 2012-2013 school years, the QDI range for districts and schools without a 12th grade that meet Growth is as follows: A (200-300), B (166-199), C (133-165), D (100-132), and F (0-99). The QDI range for districts and schools without a 12th grade that do not meet Growth is: B (200-300), C (166-199), D (133-165), F (100-132), and F (0-99). The former letter grade is applied to Low-Performing and At-Risk of Failing districts/schools, whereas the latter is applied to Failing districts/schools.

For the 2011-2012 school year, the QDI range for districts/schools with a 12th grade was the same used for districts/schools without a 12th grade as High School Completion Index (HSCI) was factored in separately. For the 2012-2013 school year, the HSCI was included in the QDI calculations: 5-year graduation rate for the state and 4-year graduation rate as mandated by NCLB. The QDI range for districts/schools with a 12th grade and a 5-year graduation rate/HSCI meeting Growth is as follows: A (200-300), B (166-199), C, (133-165), D (100-132), and F (0-99). The QDI range for districts/schools with a 12th and a 5-year graduation rate/High School Completion Index (HSCI) that do not meet Growth is: B (200-300), C (166-199), D (133-165), F (100-132), and F (0-99). The former letter grade is applied to Low-Performing and At-Risk of Failing districts/schools, whereas the latter is applied to Failing districts/schools. The 4-year graduation rate calculation for districts/schools with a 12th grade comprises the districts/schools’ QDI plus the graduation rate. The QDI range for districts/schools that meet Growth is as follows: A (280-400), B (241-279), C (203-240), D (170-202), and F (0-169), whereas the QDI range for districts/schools that do not meet Growth is: B (280-400), C (241-279), D (203-240), F (170-202), and F (0-169). The former letter grade is applied to Low-Performing and At-Risk of Failing districts/schools. The latter is applied to Failing districts/schools.

The accountability model also includes schools’ and school districts’ Growth status, in which a district and/or school’s actual achievement is compared with the expected achievement to determine whether Growth has been met (MDE, 2013). Graduation rates (4-year graduation rate required under NCLB and 5-year graduation rate required for the state component), High School Completion Index (HSCI), Annual Measurable Objectives (AMO) for Reading and Math, and a third indicator referred to as “Other Academic Indicator” that for schools without a Grade 12 is the attendance rate and for schools with a Grade 12 is the graduation rate, complete the state’s accountability measures. The new Performance Classification, A-F, accounts for an increase of districts obtaining higher letter values and a decrease of districts obtaining the lowest letter values (D and F). For the 2011-2012 school year only 57 districts fell in the D and F categories (MDE, 2012).

Satisfactory results on the report card of school districts guarantee the continuous infusion of federal money into the public school systems (Vaughn et al., 2010). Modifications have been made in school districts across the state to meet the requirements imposed by NCLB and the allocation of Title I funds. The increased interest in the state in regard to students’ academic achievement has brought attention to curriculum, instruction, and assessment (English & Steffy, 2001; Irons & Harris, 2007). However, attention is also needed on the behavioral issues negatively impacting classroom instruction (Crone, Horner, & Hawkin, 2004).

School-Wide Positive Behavior Interventions and Supports

Teachers are expected to meet the academic and behavioral needs of their students (Crone et al., 2004; Muscott, Mann, & LeBrun, 2008) to deliver appropriate instruction and to ensure optimal student achievement (Oliver et al., 2011; Sugai & Horner, 2001). Cotton (1990) noted that discipline and behavior disruptions affect the quality and quantity of instruction inside American schools. The author stated that “approximately one-half of all classroom time is taken up with activities other than instruction, and discipline problems are responsible for a significant portion of this lost instructional time.” (p. 1)

In an effort to preserve instructional time in the classrooms, school districts across the country have long tried different approaches for discipline and classroom management. SWPBIS movement has been around since the 1990’s (Sugai & Simonsen, 2012). Tobin, Lewis-Palmer, and Sugai (2002) defined Positive Behavior Interventions and Supports as the measures created and put in place in the classrooms and at schools sites to deal with undesirable behaviors and to promote optimal conditions conducive to learning.

SWPBIS is intended to minimize and/or prevent classroom disruptions to protect instructional time (Sugai & Simonsen, 2012). Researchers believe that the approach might advance students’ performance in the classroom resulting in high scores in the state tests (Jia et al., 2009; Rowe & Stewart, 2009; Sugai & Horner, 2006). Sugai and Horner (2001), the codirectors of the Office of Special Education Program (OSEP) Technical Assistance Center on PBIS, noted the importance of implementing school-wide and district-wide PBIS to create a nurturing, inclusive, and safe learning environment. Jia et al. (2009) and Rowe and Stewart (2009) reported that the school environment affects students’ academic performance in negative or positive ways. Komro et al. (2011) noted that “positive school environments help students feel connected to school, which is associated with improved academic achievement” (p. 120). The benefits of the implementation of PBIS are recurrent in the literature.

Peshak and Kincaid (2008) noted that many schools across the country implement some type of SWPBIS seeking to address students’ behavior at schools. The authors reported that the first step in the implementation of SWPBIS is the establishment of a school leadership team that provides the vision, the leadership, and the resources necessary for the successful execution of the strategies at school level. Research has shown that the appropriate implementation of SWPBIS strategies at schools and school districts might have positive outcomes that in turn might improve the climate inside the classroom (Sugai & Horner, 2001).

Teachers are in the capacity of delivering instruction (Komro et al., 2011; Paolella, 2009; Rowe & Stewart, 2009; Sugai & Horner, 2006) when they have a classroom environment with few distractions in which all the students are able to learn. When teachers have to deal with constant class disruptions not only valuable class time is lost (Cotton, 1990; Walker et al., 2005) solving a behavioral issue, but there is also the risk that this negative behavior might be replicated by other students (Sugai & Horner, 2001, 2006). Unfortunately, many of the approaches to class disruptions at school finalize in the writing of a discipline referral that might get the student In School Suspension (ISS) or Out of School Suspension (OSS). Costenbader & Markson (1998) and Fenning and Rose (2007) argued that the measure might jeopardize students’ return to the educational setting increasing the likelihood of being part of the judicial system.

Rosch and Iselin (2010) also noted that school suspensions may not be the answer to the behavioral problems that teachers encounter at schools. Suspension, as stated by Dupper, Theriot, and Craun (2009), may temporarily alleviate teachers’ and administrations’ frustrations toward the disruptive behavior, but may not provide a permanent solution to the antecedents leading to the misbehavior. The authors asserted that an increase in parental involvement may be a positive consequence of the measure however, Costenbader and Markson (1998) claimed that students need to be in the classroom under the supervision and/or influence of appropriate role models that may impact students’ lives in a positive way. Some researchers have suggested that school suspensions may promote truancy (Fenning & Rose 2007). Dawson (1991) reported that suspended students are more likely to be unsupervised at home, especially students who come from single-parent households. Suspension might not decrease undesired behaviors in the classroom; it may escalate them.

School Improvement Efforts

Some schools in the Southeastern state where the research was implemented have worked collaboratively with the State Department of Education and the SPDG to receive training on the implementation of PBIS on the schools’ sites. The state’s SPDG personnel have multifaceted responsibilities relating to training, coaching, and information dissemination. The SPDG staff develops training content for the Two-Day New Team Training based on the work of the National Center on Positive Behavioral Interventions and Supports, as well as the emerging research-based literature in the area. Materials are then tailored to the needs of audiences in the state, incorporating examples and required processes in the training.

The SPDG staff in the state provides feedback to high-intensity support sites by interpreting the readiness checklist and baseline BOQ (Benchmarks of Quality), reviewing, giving feedback, and providing assistance with the development of action plans, assisting with compiling quarterly data reports, and coaching on interpreting data from quarterly data reports, or earlier reports if available, to use results to update action plans. The SPDG also assists with problem-solving implementation of action plans and in the annual data reporting (BOQ). Team leaders serve as a liaison among SPDG staff and their school/site, use school/site data to monitor progress and effectiveness of interventions (Big 5 for universal, more individualized for advanced tiers), review data with school/site team, and facilitate conversations about program improvement. They also relay data to larger school/site community (teachers, students, families), and serve as a liaison to building-level administration (i.e., principals) to ensure SWPBIS is embedded throughout school improvement strategies. At the time of the data being reported, there were two full-time training and technical assistance providers serving, in addition to a part-time SPDG director.

The SPDG staff and the Southeastern state’s Department of Education collaboratively identify the schools receiving intensive supports. Particular attention is given to ensuring that during any given year, there are schools with higher support needs. Selection is based on school-level data, which may include office discipline referrals; total number of suspensions, for students with and without disabilities; total number of expulsions for students with and without disabilities; attendance; students placed in an alternative school; number of students adjudicated; number of students referred to special education; disproportionate representation of minority students, and so on. In addition to the criteria described above, the SPDG specifically recruits schools not meeting expectations on the state’s monitoring systems whose noncompliance is in areas of the federally required State Performance Plans, which are relevant to the SPDG. These schools are required by the state to implement a Corrective Action Plan (CAP; MDE, 2013) to address identified needs. It is important to note that at Level 2 status districts must engage a consultant to assist with the Corrective Action Plan, and at Level 3 federal funds are withheld. It is the intent of the SPDG to assist districts that may successfully complete their CAP with assistance of the SPDG. It is not the role of the SPDG to remediate all districts out of compliance.

Schools selected to receive intensive supports enter into a Memorandum of Understanding (MOU), which outlines roles and responsibilities of SPDG and participating schools. The MOU outlines expectations for participation and data to be provided to the SPDG, including the development of an annual plan of action, submission of quarterly data reports (suspensions, expulsions, and Office Discipline Referral [ODR] data), and annual evaluations (BOQ). The MOU also describes services and supports provided by the SPDG. Because the SPDG cannot provide intensive supports to all schools in the state, it provides opportunities for other interested schools to receive the same training as the schools receiving intensive supports. Schools not receiving intensive supports are required to complete a Commitment Form, Readiness Checklist, and provide baseline BOQ (SWPBIS, Tier I) prior to attending the free training events offered by the SPDG. During the training, site-based teams develop individualized action plans for implementation. They are asked to submit annual BOQs and quarterly ODR data; however, not all schools follow through with this data request.

All professional development concludes with the development of an action plan based on the critical elements of SWPBIS. Action steps likely to lead to implementation milestones are clearly defined, with timelines and persons responsible for each step. SPDG staff monitors and supports implementation of these plans at intensive support schools; nonintensive support schools, self-monitor implementation. As a prerequisite to registration for training, participating schools complete a pretraining self-assessment (BOQ), complete Commitment Form, and return it to the SPDG, and complete a SWPBIS School Readiness Checklist.

At the beginning of training, participants view a SPDG-produced video on implementation of SWPBIS in the state and review current behavior data, that is, ODRs, suspensions, expulsions, and so on. Then SPDG trainers/coaches review pretraining self-assessment data and alter training content, if necessary, to meet the needs of the audience. During training, each school team participates in learning exercises relating to the critical elements. At the end of each section, the team develops an action plan to implement after the training.

The SPDG anticipated offering 6 Two-Day New Team Trainings during the 2011-2012 year. This included regionally offered trainings that were required for schools receiving intensive supports, but that were also opened to any school wishing to attend that was willing to provide the required prerequisite information already described. The SPDG will also provide training to schools requesting the training if staff is available and if the prerequisite criteria are met. During 2011-2012, the SPDG provided 11 Two-Day New Team Trainings which were attended by 855 individuals. It is important to note, however, that the personnel from schools included in the study may have received training prior to the 2011-2012 school year, with some schools having implemented SWPBIS since 2005.

The Southeastern state’s SPDG provides assistance to schools selected for intensive, on-site assistance for at least 2 years. Participants are engaged in systems improvement, incorporating evidence-based strategies to provide intensive intervention to youth with high levels of behavior support needs. The goal is to successfully transition these students into less-restrictive environments. SWPBIS training and coaching incorporates evidence-based strategies including systemic change/renewal, school–community collaboration, safe learning environments, family engagement, professional development, and individualized instruction. All work with schools, districts, and centers is captured within improvement plans that include specific, measureable outcome data that are analyzed by the school/district and SPDG staff.

The SPDG staff conducts on-site visits (at least monthly), frequent phone conferences, and email exchanges to provide support to intensive schools. SPDG model strategies and support school staff as they implement their individual improvement plans, which are developed annually. The state’s SPDG personnel attend team meetings at high-intensity districts as well as provide formative feedback and guidance. This assistance is faded over time, with more responsibility transferred to the site-based team leader. The SPDG staff coaches high-intensity districts using a team-based model that is guided by site-specific action plans using the critical elements of SWPBIS framework. All SWPBIS training, coaching, and evaluation revolves around the Critical Elements. By using nationally validated instruments (BOQ, SET), sites may self-assess implementation and may also be externally evaluated for fidelity.

Some schools participating in this study were evaluated with the School-Wide Evaluation Tool (SET), while others were not due mainly to financial and time constraints. Because of the cost of completing the SET, all the schools (intensive and nonintensive) that submitted the BOQs with scores of 80% or higher were invited to be externally evaluated using the SET. Schools scoring 80% of higher are considered SWPBIS Model Sites. Model Sites are expected to continue to be evaluated annually using the SET and to submit quarterly ODR data.

The level of training and coaching regarding the implementation of SWPBIS varied in this study with some schools receiving training only (“non-intensive”) and some receiving training and on-site coaching (“intensive”). The schools also differed in the levels of implementation fidelity as measured by the BOQ; an instrument schools used to self-report the fidelity in the execution of SWPBIS.

Training only has shown to be beneficial (Joyce & Showers, 2002). Training and coaching, however, may provide a better structure for the implementation of SWPBIS inside the classrooms (Sugai & Horner, 2006). Joyce and Showers (2002) noted that training and coaching may help teachers not only to change the structure of the classrooms due to a change in teachers’ beliefs, but also to help teachers deal with the discomfort that the new set of procedures may imply. Training and coaching may also provide the emotional support teachers need when implementing the newly learned set of procedures in the classrooms.

Method

This study investigated relationships among the various types of training and coaching received by 96 schools in a Southern state in the United States and their performance on state accountability measures. Training and coaching were supported by the SPDG funded by the U.S. Department of Education’s OSEP. In addition, the data were analyzed to determine if implementation fidelity of SWPBIS was related to performance on those accountability measures. While several program evaluation measures are used for reporting results to OSEP, this study was undertaken to begin to investigate potential relationships among SPDG-specific efforts and the larger accountability measures of the state. The aim of this research is to determine (a) if the level of training and coaching received by the schools was related to the schools’ QDI; (b) if the schools’ classification into “model sites” or “non-model sites” based on the results of the SET instrument was related to the schools’ QDI; (c) if the levels of training and coaching and the results of the SET instrument that classified the schools into “model sites” and “non-model sites” were related to the schools’ QDI, and (d) if the level of SWPBIS implementation fidelity, BOQ, was related to the schools’ performance classifications, QDI, or Growth status.

Participants

The sample drawn for this study came from 96 schools in the targeted state who received training, coaching, or both from the SPDG during the 2011-2012 school year. Primary, Elementary, Lower Elementary, Upper Elementary, Middle Schools, High Schools, and Attendance Centers composed the sample collected from the 2011-2012 school year. Table 1 displays the information related to the participating schools that belong to 41 out of 152 school districts in the area.

Table 1.

Schools Participating in the Study.

Type of school	Frequency
Primary	2
Elementary	42
Lower elementary	1
Upper elementary	6
Middle school	14
High school	18
Achievement center	1
Attendance center	9
Academy	1
Career & technology center	1
Specialty school	1
Total	96

Measures

Data were gathered from several sources for this analysis. First, a list of schools that had participated in training and/or coaching by the SPDG was compiled by its director. Those schools were then coded as either training only or training and on-site coaching. Next, self-reported scores on the BOQ were obtained by the director of the SPDG for each school. These were used to code each school as self-reported implementation fidelity (80% or higher) or no self-reported implementation fidelity. The schools that reported implementation fidelity (80% or higher) were invited to be externally evaluated using the SET. Finally, a list of schools that had scored 80% or higher on the SET and were listed on the SPDGs website as model sites was obtained. The data set was then updated to include coding for external implementation fidelity or no external implementation fidelity. Table 2 displays the mean and standard deviation BOQ and SET scores for each classification group. There is no SET data in regard to the nonmodel and nonintensive schools as they did not report implementation fidelity. .

Table 2.

Schools Classification According to BOQ and SET Scores.

		Total BOQ	SET scores
Model and Intensive	M	0.94	0.93
	SD	0.06	0.03
	n	12	12
Model and nonintensive	M	0.95	0.93
	SD	0.03	0.06
	n	10	9
Nonmodel and nonintensive	M	0.85
	SD	4.05
	n	73
Total	M	0.87	0.93
	SD	3.55	0.04
	n	95	21

Note. BOQ = Benchmarks of Quality; SET = School-Wide Evaluation Tool.

Next, publicly available accountability data for the 2011-2012 school year was obtained from the state’s website and each school’s performance classification, QDI score, and Growth score were added to the data set.

The data collected regarding the BOQ, the intensity of the treatment, and the ranking of the schools into “model site” or “non-model site” came from the information compiled by the SPDG in a local university. Performance classification, QDI, and Growth were calculated by the State Department of Education based on the state’s cut-off points used to determine the improvement of the schools.

Analysis

Different statistical tests were conducted to address the four research objectives posed in this research using an alpha level of significance of α = .05. An independent sample t-test was conducted to determine whether the schools that received training and on-site coaching (“intensive”) differ from the schools that received training only (“non-intensive”) in regard to QDI. A second independent sample t-test was conducted to determine whether the schools that were considered “model sites” differ from the schools that were considered “non-model sites” in regard to QDI. Then, an ANOVA was conducted to determine whether the type of training and coaching (“intensive” or “non-intensive”), and the results of the SET that classified schools into “model sites” and “non-model sites,” were related to the schools’ QDI. The schools were grouped as follows: (a) “model site” and “intensive” (training plus on-site coaching), (b) “model site” and “non-intensive” (training only), and (c) “non-model site” and “non-intensive” (training only). Finally, correlations were performed to establish if the level of SWPBIS implementation fidelity, as determined by the self-administered BOQ, was related to the schools’ performance classifications, QDI, or Growth status.

Out of the 96 schools included in this study, there is accessible data for 91 schools on the number of years of SWPBIS implementation as follows: 10 schools began SWPBIS implementation during the 2006-2007 school year, 1 school during 2007-2008, 2 schools during 2008-2009, 1 school during 2009-2010, 8 schools during 2010-2011, and 69 schools during the 2011-2012 school year. In regard to the schools’ QDI scores prior to SWPBIS implementation, there is QDI information on the last three cohorts as follows: the school that began implementing SWPBIS during 2009-2010 school year had a mean QDI of 167 prior to SWPBIS implementation. The participating schools during 2010-2011 school year had a mean QDI of 160.67 (SD = 28.46) prior to SWPBIS implementation and the participating schools during 2011-2012 had a mean QDI of 147.18 (SD = 36.59) prior to SWPBIS implementation. It is important to note that the state where this study took place adopted QDI as an accountability measure during the 2008-2009 school year.

Results

The first research objective sought to determine whether the intensity of the training was related to the schools’ QDI. The results of the t-test indicated that the schools that received training only (“non-intensive”) had a mean QDI of 151.78 (SD = 32.42) compared with the schools that received training and on-site coaching (“intensive”) with a mean QDI of 171.77 (SD = 20.30). Levene’s F test (p = .017) indicated a violation of homogeneity of variance. Therefore, the Equal variances not assumed, t(25) = −2.904, p = .007, η² = .097, notes there is a significant difference between the means of the two samples. It can be inferred that schools that received training plus on-site coaching (“intensive”) had higher QDIs than the schools that received training only (“non-intensive”).

The second research objective sought to determine whether the schools that were considered “model sites” differ from the schools that were considered “non-model sites” upon completion of the SET in regard to QDI. The results of the t-test indicated that the schools that were considered “model sites” had a mean QDI of 169.36 (SD = 21.72) compared with the schools that were considered “non-model sites” with a mean QDI of 149.59 (SD = 33.12). Levene’s F test (p = .017) indicated a violation of homogeneity of variance. Therefore, the Equal variances not assumed, t(57) = −3.113, p = .003, η² = .110, states there is a significant difference between the means of the two samples. It can be inferred that schools considered “model sites” had higher QDIs than the schools considered “non-model sites.”

For the third research objective, an ANOVA test was conducted to determine whether the levels of training and coaching, and the results of the SET, that classified schools into “model site” or “non-model site”, had any effect on the schools’ QDI. Table 3 displays the statistics for the groups in which it is observed that Group 1 (Model and Intensive) and Group 2 (model and nonintensive) have smaller sample sizes than Group 3 (nonmodel and nonintensive).

Table 3.

Quality of Distribution Index.

				95% CI for mean
	n	M	SD	Lower bound	Upper bound
Model and intensive	13	171.77	(20.30)	159.50	184.04
Model and nonintensive	9	165.89	(24.44)	147.10	184.68
Nonmodel and nonintensive	58	149.59	(33.12)	140.88	158.30
Total	80	155.03	(31.55)	148.00	162.05

Note. M = mean; SD = standard deviation; CI = confidence intervals

Levene’s F test showed a violation in the assumption of homogeneity of variance (p = .035); therefore, Welch’s F(2, 20) = 5.10, p = .016, est. ω² = .093, indicated there is a statistically significant difference between the levels of training and coaching, and the results of the SET, that classified schools into “model site” or “non-model site,” in regard to the schools’ QDI. Due to unequal variances among the groups, the Games-Howell post hoc test was conducted to determine where the difference was between the pair-wise comparisons. The results revealed that the schools that were considered “model sites” and that received training and coaching on-site (“intensive”; M = 171.77, SD = 20.30) had higher QDI than the schools that were considered “non-model sites” and received training only (“non-intensive”) with a mean QDI of 149.59, SD = 33.12, p = .011.

Finally, for the fourth research objective, two-tailed Spearman’s correlations were performed to establish if the level of SWPBIS implementation fidelity, the BOQ, was related to the schools’ performance classifications, QDI, or Growth status. The results indicated there is a positive relationship and a medium effect between the BOQ and performance classifications with r_S(71) = .322, p = .005; there is a positive relationship and a medium effect between the BOQ and QDI with r_S(78) = .365, p = .001, and that there is a positive relationship and a small effect between the BOQ and Growth with r_S(71) = .262, p = .025. The results also revealed there is a positive bivariate relationship among QDI, performance classifications and Growth; the accountability measures used to evaluate schools and school districts in the state.

Discussion

The aim of this research was to determine (a) if the level of training and coaching received by the schools was related to the schools’ QDI; (b) if the schools’ classification into “model sites” or “non-model sites” based on the results of the SET instrument was related to the schools’ QDI; (c) if the levels of training coaching and the results of the SET instrument were related to the schools’ QDI, and (d) if the level of SWPBIS implementation fidelity, BOQ, was related to the schools’ performance classifications, QDI, or Growth status.

The overall findings were consistent with past studies. First, the schools that received training and coaching (“intensive”) had higher QDI than the schools that received training only (“non-intensive”). This finding is consistent with the literature of the benefits of training and coaching when implementing PBIS at schools (Joyce & Showers, 2002; Simonsen et al., 2012; Sugai & Horner, 2006). Second, the schools that were classified as “model sites” based on the results of the SET had higher QDI than the schools that were considered “non-model sites.” This finding provides evidence of the impact that SWPBIS has on academic achievement (Rowe & Stewart, 2009; Sugai & Horner, 2001, 2006). Consequently, the implementation of SWPBIS in the schools in the Southeastern state where the study was conducted may be beneficial to keep students inside the classroom; a measure that may decrease dropout rates and increase graduation rates in the area. Besides, higher QDI for schools in the state may indicate better results on the schools’ and school districts’ report cards, and thus the continuous infusion of federal money into the school districts (Vaughn et al., 2010).

Third, the schools that received training and coaching (“intensive”) and were considered “model sites” based on the results of the SET instrument presented higher QDI than the schools that received training only (“non-intensive”) and were considered “non-model sites.” This illustrates the need for the continuing training and coaching of the schools in the state in regard to the implementation of SWPBIS. It may be that if schools are knowledgeable of the techniques needed to implement SWPBIS and apply those with fidelity, the benefits are exponential. Not only may the schools improve the performance classification obtained in the state’s accountability measure, but in the long run the state may see a decrease in dropout rates and an increase in graduation rates. This may be especially true for at-risk students in the area.

Fourth, the level of SWPBIS implementation fidelity, the BOQ, was related to the schools’ performance classifications, QDI, and Growth status. There was a positive relationship and a medium effect between the BOQ and the schools’ performance levels and between the BOQ and QDI. A possible explanation for these results may lie in the fact that once the schools are knowledgeable and confident on how to implement SWPBIS, the likelihood of fidelity to the measure increases, thus the academic and behavioral issues in the classroom may decrease. The more fidelity to the measure schools exercised, the better the results the schools may obtain in regard to the state’s accountability measures. Therefore, future researchers and/or external coaches for SWPBIS may want to work with school personnel closely to help develop SWPBIS fidelity implementation. It was also observed that the BOQ had a small effect with the Growth status of the schools.

First, contributing to the emerging literature on the relationship of SWPBIS fidelity implementation tools and accountability measures is one of the strengths of the present study. Second, the internal (BOQ) and external (SET) fidelity implementation tools currently used in the state where this research took place were separately analyzed in relation to QDI to determine relationships before being analyzed together in regard to QDI. Third, the scores reported in the BOQ were examined in light of the state’s accountability measures: performance classification, QDI, and Growth. The findings of the study showed the benefits that training and coaching has on the schools that implement SWPBIS in the region. This may encourage the State Department of Education in conjunction with the SPDG to expand the training to other schools and/or school districts in the region.

The results of the study also suggested that improving academic achievement may be possible. Teachers that received training and on-site coaching seemed to be more knowledgeable than teachers that received training only in regard to the behavioral strategies needed to deal with classroom disruptions. The literature has shown that the less time teachers invest in dealing with classroom disruptions the more instructional time is gained and the better results students may obtain when faced with high-stake tests. Districts or states trying to improve performance on state accountability measures may want to consider adding the coaching component to the training school personnel may receive on SWPBIS. Training and coaching has shown to be beneficial for teachers (Joyce & Showers, 2002; Sugai & Horner, 2006) when it comes to SWPBIS implementation.

Although this study contributes to the preliminary investigation of potential relationships among SPDG-specific efforts and the larger accountability measures of the state, it has limitations. The first one is the use of the BOQ, a self-report measure that may or may not be an accurate report on the fidelity of the schools when implementing SWPBIS on site. Second, it would be advisable to collect a larger sample size for future research to have more statistical power in the analysis. The groups identified in the third analysis had uneven sample sizes that may have contributed to the unequal variances in the groups. Sample size may have also accounted for the non-normally distributed data used in this study. Finally, it is important to remember this study does not show causation, only a relationship between SWPBIS implementation fidelity tools and the state accountability measures in which higher levels of implementation fidelity correlated with higher ratings on accountability measures.

Reducing behavioral issues in the classroom in an attempt to improve academic achievement is a major factor in today’s educational reforms. Research has shown that classroom disruptions account for loss of instructional time (Cotton, 1990; Sugai & Horner, 2001; Walker et al., 2005); however, the implementation of measures designed to prevent and/or control disruptions such as SWPBIS may be the key to advance academic achievement in our classrooms. It would be beneficial to keep exploring the relationship of SWPBIS implementation fidelity instruments and accountability measures to contribute to the emerging literature on the topic as well as to explore the benefits of training and coaching when it comes to the implementation of PBIS at schools and inside the classrooms.

Footnotes

Authors’ Note

The views expressed herein do not necessarily represent the positions or policies of the Department of Education. No official endorsement by the U.S. Department of Education of any product, commodity, service, or enterprise mentioned in this publication is intended or should be inferred.

Declaration of Conflicting Interest

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: This research was partially supported by the U.S. Department of Education, Office of Special Education Programs Grant H323A100001.

Author Biographies

Adriana M. Marin is a doctoral candidate in the Department of Curriculum, Instruction, and Special Education at the University of Southern Mississippi. Her research interests are in the areas of Positive Behavior Interventions and Supports, state accountability measures, and academic achievement of at-risk students in K-12 settings.

Hollie Gabler Filce is an Associate Professor and the Coordinator of Special Education Programs at the University of Southern Mississippi. She also serves as the Director of the Mississippi State Personnel Grant, overseeing implementation of Positive Behavioral Interventions and Supports across the state.

References

Berry

Fuller

(2008, January). Final report on the Mississippi project CLEAR Voice Teacher Working Conditions Survey. Center for Teaching Quality. Retrieved from: http://www.teachingquality.org/sites/default/files/Final%20Report%20on%20the%20Mississippi%20Project%20CLEAR%20Voice_0.pdf

Bishaw

(2012). Poverty: 2010 and 2011. American Community Survey Briefs. The United States Census Bureau. Retrieved from http://www.census.gov/prod/2012pubs/acsbr11-01.pdf

Carney

Stiefel

(2008). Long-term results of a problem solving approach to response to intervention. Learning Disabilities: A Contemporary Journal, 6(2), 61-75.

Common Core State Standards. (2013). Implementing the common core state standards. Available from http://www.corestandards.org/

Costenbader

Markson

(1998). School suspension: A study with secondary school students. Journal of School Psychology, 36, 59-82.

Cotton

(1990). Schoolwide and classroom discipline (School improvement and research series, Close-Up No. 9). Portland, OR: Northwest Regional Educational Laboratory.

Crone

D. A.

Horner

R. H.

Hawkin

L. S.

(2004). Responding to problem behavior in schools. New York, NY: Guilford.

Davis-Kean

(2005). The influence of parent education and family income on child achievement: The indirect role of parental expectations and the home environment. Journal of Family Psychology, 19, 294-304. doi:10.1037/0893-3200.19.2.294.

Dawson

(1991). Family structure and children’s health and well-being: Data from the 1988 national health interview survey on child health. Journal of Marriage and Family, 53, 573-584.

10.

Duncan

G. J.

Magnuson

K. A.

(2005). Can family socioeconomic resources account for racial and ethnic score gap? The Future of Children, 15(1), 35-54.

11.

Dupper

Theriot

Craun

(2009). Reducing out-of-school suspensions: Practice guidelines for school social workers. Children & Schools, 31, 6-10.

12.

English

Steffy

(2001). Deep curriculum alignment. Creating a level playing field for all children on high-stakes tests of educational accountability. Lanham, MD: The Scarecrow Press.

13.

Evans

G. W.

(2004). The environment of childhood poverty. American Psychologist, 59, 77-92.

14.

Fenning

Rose

(2007). Overrepresentation of African American students in exclusionary discipline: The role of school policy. Urban Education, 42, 536-559.

15.

Gassman-Pines

Yoshikawa

(2006). The effects of antipoverty programs on children’s cumulative level of poverty-related risk. Developmental Psychology, 42, 981-999. doi:10-1037/0012-1649.42.6.981

16.

Irons

Harris

(2007). The challenges of no child left behind. Understanding the issues of excellence, accountability and choice. Lanham, MD: Rowman & Littlefield Education.

17.

Jia

Way

Ling

Yoshikawa

Chen

Hughes

. . .Lu

(2009). The influence of student perceptions of school climate on socioemotional and academic adjustment: A comparison of Chinese and American adolescents. Child Development, 80, 1514-1530.

18.

Joyce

Showers

(2002). Student achievement through staff development (3rd ed.). Alexandria, VA: Association for Supervision and Curriculum Development.

19.

Komro

Flay

Biglan

(2011). Creating nurturing environments: A science-based framework for promoting child health and development within high-poverty neighborhoods. Clinical Child Family Psychology Review, 14, 111-134.

20.

Lassen

S. R.

Steele

M. M.

Sailor

(2006). The relationship of school-wide positive behavior support to academic achievement in an urban middle school. Psychology In The Schools, 43, 701-712.

21.

Manzo

K. K.

(2000). The state of curriculum. In Staff of Education Week (Eds.), Lessons of a century: A nation’s schools come of age (pp. 122-151). Bethesda, MD: Editorial Projects in Education.

22.

Mississippi Department of Education. (2010). Mississippi statewide accountability system. Retrieved from https://districtaccess.mde.k12.ms.us/Accountability/_layouts/OSSSearchResults.aspx?k=mississippi%20public%20school%20accountability%20standards%2C%202012&cs=This%20List&u=https%3A%2F%2Fdistrictaccess.mde.k12.ms.us%2FAccountability%2FPublic%20Documents

23.

Mississippi Department of Education. (2012). Tier process. Retrieved from http://www.mde.k12.ms.us/curriculum-and-instruction/curriculum-and-instruction-other-links/response-to-intervention-teacher-support-team

24.

Mississippi Department of Education. (2013). Correction action plans. Retrieved from https://www.mde.k12.ms.us/mississippi-board-of-education/board-of-education-policy-manual/policy-5900-probation-(conservatorship)

25.

Mississippi Department of Education. (2013). Mississippi statewide accountability system. Retrieved from https://districtaccess.mde.k12.ms.us/Accountability/_layouts/OSSSearchResults.aspx?k=mississippi%20public%20school%20accountability%20standards%2C%202012&cs=This%20List&u=https%3A%2F%2Fdistrictaccess.mde.k12.ms.us%2FAccountability%2FPublic%20Documents

26.

Mississippi Department of Education. (2013). Recommendations for the 2012-2013 Mississippi statewide accountability system. Retrieved from https://districtaccess.mde.k12.ms.us/Accountability/Public%20Documents/2012-2013%20Performance%20Classification%20Information/2012-2013%20MSAS%20Recommendations%20Revised%204%2018%2013.pdf

27.

Mississippi Department of Education. (2013). Understanding the assignment of 2012-2013 performance classifications for schools and districts . Retrieved from https://districtaccess.mde.k12.ms.us/Accountability/Public%20Documents/2012-2013%20Performance%20Classification%20Information/Webinar%202012-2013%20Performance%20Classifications.pdf

28.

Muscott

Mann

LeBrun

(2008). Positive behavioral interventions and supports in New Hampshire: Effects on large-scale implementation of schoolwide positive behavior support on student discipline and academic achievement. Journal of Positive Behavior Interventions, 10, 190-205. doi:10.1177/1098300708316258

29.

No Child Left Behind (NCLB) Act of 2001, Pub. L. No. 107–110, § 115, Stat. 1425 (2002).

30.

Noss

(2012). Household income for states: 2010-2011. United States Census Bureau. Retrieved from http://www.census.gov/prod/2012pubs/acsbr11-02.pdf

31.

Oliver

Wehby

Reschly

(2011). Teacher classroom management practices: Effects on disruptive or aggressive student behavior. Campbell Systematic Reviews. Advance online publication. doi:10.4073/csr.2011.4

32.

Paolella

(2009). Positive behavior support and student response to the behavior education program (Social Work Student Papers. Paper, 40). Retrieved from http://digitalcommons.providence.edu/socialwrk_students/40

33.

Peshak

Kincaid

(2008). Building district-level capacity for positive behavior support. Journal of Positive Behavior Interventions, 10, 20-32.

34.

Rosch

Iselin

(2010). Alternatives to suspension (North Carolina family impact seminar). Center for Child and Family Policy, Duke University. Retrieved from http://www.childandfamilypolicy.duke.edu/pdfs/familyimpact/2010/Alternatives_to_Suspension.pdf

35.

Rowe

Stewart

(2009). Promoting connectedness through whole-school approaches: A qualitative study. Health Education, 109, 396-413.

36.

Simonsen

Eber

Black

A. C.

Sugai

Lewandownki

Sims

Myers

(2012). Illinois statewide positive behavioral interventions and supports: Evolution and impact on student outcomes across years. Journal of Positive Behavior Interventions, 14, 5-16.

37.

Smith

J. M.

Fien

Paine

S. S.

(2008). When mobility disrupts learning. Educational Leadership, 65(7), 59-63.

38.

Sugai

Horner

(2001). Features of an effective behavior support at the school district level. Positive Behavior, 3, 16-19.

39.

Sugai

Horner

R. H.

(2006). A promising approach for expanding and sustaining school-wide positive behavior support. School Psychology Review, 35, 245-259.

40.

Sugai

Simonsen

(2012). Positive behavioral interventions and support: History, defining features, and misconceptions. OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports. Effective Schoolwide Interventions, Retrieved from http://www.pbis.org/common/pbisresources/publications/PBIS_revisited_June19r_2012.pdf

41.

Tobin

Lewis-Palmer

Sugai

(2002). School-wide and individualized effective behavior support: An explanation and an example. The Behavior Analyst Today, 3(1), 51-75.

42.

Vaughn

Bos

C. S.

Schumm

J. S.

(2010). Teaching exceptional, diverse, and at-risk students in the general education classroom (5th ed.). Boston, MA: Allyn & Bacon.

43.

Walker

H. M.

Ramsey

Gresham

R. M.

(2005). Antisocial behavior in school: Evidence-based practices (2nd ed.). Belmont, CA: Wadsworth/Thomson Learning.

44.

Wamba

N. G.

(2010). Poverty and literacy: An introduction. Reading & Writing Quarterly, 26, 109-114. doi:10.1080/10573560903547429