Abstract
Previous research suggests that Positive Behavioral Interventions and Supports (PBIS) can reduce student disciplinary office referrals and out-of-school suspensions, especially when implemented with fidelity. Existing research is mixed as to whether PBIS also contributes to improvements in student achievement, but at least one study has found that PBIS leads to improvements in teachers’ perceptions of overall organizational health, an effect that may help facilitate improvements in student learning. This study uses the TELL (Teaching, Empowering, Leading, and Learning) Kentucky survey to analyze teacher perceptions of their working conditions between PBIS and non-PBIS schools, and among schools varying in level of PBIS implementation fidelity. Furthermore, because the TELL Kentucky survey has been shown to predict increases in student achievement, this study examined the relationship between PBIS implementation and student test score outcomes. Teachers in PBIS schools reported higher levels of student and faculty understanding of behavioral expectations and a stronger atmosphere of professional trust and respect. Although there were no significant differences in student achievement levels between PBIS and non-PBIS schools, analysis did reveal that student academic outcomes were significantly higher at high- and medium-fidelity PBIS schools than low-fidelity PBIS schools. Significance, limitations, and implications for practice are discussed.
School-Wide Positive Behavior Interventions and Supports (SWPBIS) emphasize unity of staff expectations regarding student behavior, the proactive teaching of positive behaviors, and a tiered, data-driven system of interventions (Sugai & Horner, 2009). Research shows SWPBIS yields improvements in student behavior outcomes like disciplinary office referrals and out of school suspensions (Bradshaw, Mitchell, & Leaf, 2010; Lassen, Steele, & Sailor, 2006; Luiselli, Putnam, Handler, & Fienberg, 2005; Mayer et al., 1993; Nelson, Martella, & Marchand-Martella, 2002; Sadler & Sugai, 2009; Scott & Barrett, 2004; Turnbull et al., 2002). These positive outcomes may be related to the fidelity with which schools implement the various features of SWPBIS (Barrett, Bradshaw, & Lewis-Palmer, 2008; Childs, Kincaid, & George, 2010).
Positive Behavior Support (PBS) emerged from the field of applied behavior analysis (Carr & Sidener, 2002; Dunlap, 2006), representing a move away from punitive strategies toward the proactive teaching and celebration of positive student behavior (Singer & Wang, 2009). SWPBIS seeks to promote positive behavior in schools using multitiered interventions and supports (Sugai & Horner, 2002) and an emphasis on understanding the function of student behaviors (Office of Special Education Programs Center on Positive Behavioral Interventions and Supports [PBIS], 2005). Other features of SWPBIS include schoolwide positive behavior expectations, supports differentiated for both classroom and nonclassroom environments, small group interventions for students who do not respond to universal strategies, and intensive interventions for students with the most significant needs (Lewis & Sugai, 1999). According to Sugai and Horner (2002, 2009), other elements of successful PBIS implementation include schoolwide leadership teams, staff involvement, data-driven decision making, and staff training.
SWPBIS and Student Outcomes
Researchers have identified a range of positive outcomes from implementation of SWPBIS. Mass-Galloway, Panyan, Smith, and Wessendorf (2008) conducted a statewide evaluation of SWPBIS implementation in Iowa and found that SWPBIS schools experienced a substantial decrease in the number of office discipline referrals (ODRs), a metric sometimes associated with other at-risk and disruptive student behaviors (Irvin, Tobin, Sprague, Sugai, & Vincent, 2004). The positive impact of SWPBIS on ODRs has been confirmed by numerous other researchers (Bradshaw et al., 2010; Lassen et al., 2006; Luiselli et al., 2005; Nelson et al., 2002; Sadler & Sugai, 2009; Turnbull et al., 2002; Warren et al., 2006).
Likewise, researchers have found a link between SWPBIS and a reduction in out-of-school suspensions (Bradshaw et al., 2010; Lassen et al., 2006; Luiselli et al., 2005; Mayer et al., 1993; Nelson et al., 2002; Scott & Barrett, 2004; Turnbull et al., 2002; Warren et al., 2006). Other studies have confirmed a positive impact from SWPBIS on drop-out rates (Cheney, Malloy, & Hagner, 1998).
The impact of SWPBIS on student achievement has been less clear (Horner et al., 2009). Nelson et al. (2002) found improvements in reading scores after SWPBIS implementation. Likewise, Muscott, Mann, and LeBrun (2008) found a connection between SWPBIS and math achievement, although their study did not include a comparable non-SWPBIS sample group. In a more sophisticated, 5-year study featuring randomized controls, Bradshaw et al. (2010) found no differences in reading or math achievement between students subjected to SWPBIS and those who were not.
These inconsistent findings suggest more study is warranted. Some researchers found that specific improvements in student behavior, such as reductions in ODRs, were associated with improvements in student achievement (Sadler & Sugai, 2009), suggesting that SWPBIS might create conditions for improved student learning. Those conditions, however, may not, in and of themselves, be sufficient to lead directly to higher test scores as so many other variables may be in play besides student behavior (Horner et al., 2009).
SWPBIS and Teacher Perceptions of Organizational Health
To our knowledge only one study has explored the impact of SWPBIS on teacher perceptions. In a longitudinal, randomized control trial in 37 elementary schools, Bradshaw, Koth, Bevans, Ialongo, and Leaf (2008) examined how SWPBIS implementation led to changes in teacher perceptions of their schools’ organizational health compared with teachers in demographically matched schools that did not implement SWPBIS. The study utilized an instrument called the Organizational Health Inventory (OHI) for Elementary Schools, measuring five dimensions of a healthy-functioning school, including the school’s capacity to deal with destructive outside forces, feelings of warmth and affiliation among staff, the school’s emphasis on academics, the school leader’s level of collegiality, and the school leader’s ability to secure resources. Bradshaw et al. (2008) found that SWPBIS improved teacher perceptions of their school’s overall health, staff affiliation, and the principal’s resource influence. The authors hypothesized that the schoolwide decision-making structures of SWPBIS and its emphasis on shared goals and expectations may have contributed to improvements in organizational health. Bradshaw et al.’s study makes an important contribution to the research by suggesting how SWPBIS might create a stronger and more academically focused culture, which in turn may create a greater capacity for improved student achievement.
SWPBIS and Fidelity of Implementation
Regardless of the outcome variable used to analyze SWPBIS effectiveness, researchers and practitioners alike have expressed a strong interest in the importance of implementation fidelity. Mass-Galloway et al. (2008) found that when schools implemented SWPBIS with fidelity, they experienced large decreases in ODRs. However, when schools did not have the proper supports to implement with fidelity, there was no impact on ODRs. Muscott et al. (2008) examined discipline and academic outcomes in 28 schools after 1 year of high-fidelity SWPBIS implementation. The schools showed decreases in ODRs (28%), out-of-school suspensions (19%), and in-school suspensions (31%), and SWPBIS was associated with gains in math for the majority of schools in the sample; however, gains in reading were evident in less than half the schools achieving SWPBIS fidelity. Barrett et al. (2008) studied more than 400 schools in Maryland and found schools that implemented PBIS with high fidelity had lower ODR and suspension rates than nonimplementing schools. Childs et al. (2010) studied 300 Florida schools implementing SWPBIS. Schools with high fidelity showed substantially different effects on four outcome measures (decreases in ODRs, in-school suspensions, out-of-school suspensions, and increases in the percentage of students scoring proficient in reading). Only the overall percentage decrease in ODRs showed statistical significance.
SWPBIS in Kentucky
Since 2004, the Kentucky Center for Instructional Discipline (KYCID) has provided supports for schools interested in implementation of SWPBIS. KYCID utilizes Benchmarks of Quality (BoQ), voluntarily completed by participating schools, to measure the fidelity with which schools implement SWPBIS elements. Davis (2011) found a high level of integrity of SWPBIS implementation among Kentucky schools that utilized the framework, concluding that, “the PBIS model of training and technical assistance used in Kentucky demonstrates a reliable model for schools to follow to implement sustainable behavior change that likely will lead to improved student outcomes” (p. 141). While Davis found promising trends in SWPBIS implementation in Kentucky, additional studies are warranted that (a) include statewide versus regional data, (b) use a matched comparison group rather than a single group pre–post design, and (c) explore of the impact of SWPBIS implementation on academic achievement in Kentucky schools. In summary, although existing research has established the impact of SWPBIS on student disciplinary behaviors, more research is needed on how SWPBIS may influence student achievement. Furthermore, only one study (Bradshaw et al., 2008) has examined the impact of SWPBIS on teacher perceptions, which is important given the possible link between working conditions and achievement (Daily & Maddock, 2012). This study evaluated working conditions in SWPBIS versus non-SWPBIS schools in Kentucky schools. Furthermore, working conditions and student achievement were compared within SWPBIS schools with varying levels of implementation fidelity.
Three research questions framed this study:
Method
Sample
One hundred fifty-one Kentucky schools participated in SWPBIS in 2010–2011 and completed the BoQ fidelity of implementation self-assessment. Using propensity score matching methods (Rosenbaum & Rubin, 1985), 151 non-SWPBIS schools with matched school demographic variables (total enrollment, dollars spent per student, percentages of White students, percentages of male students, and free/reduced lunch rates) were selected as the comparison group. Researchers reviewed data from all Kentucky non-SWPBIS schools (586 elementary, 216 middle, and 219 high schools) to select a sample of non-SWPBIS for comparison with SWPBIS schools in the sample.
Educators in all schools were asked to complete the TELL (Teaching, Empowering, Leading, and Learning) Kentucky survey measuring teacher perceptions of working conditions. TELL Kentucky respondents in 2011 for SWPBIS schools were 4,308 educators, including 3,829 teachers (88.9%), 100 principals (2.3%), 85 assistant principals (2%), and 294 other education professionals (6.8%); respondents for the non-SWPBIS schools were 5,693 educators, including 5,060 teachers (88.8%), 124 principals (2.2%), 111 assistant principals (2.0%), and 398 other education professionals (7.0%). To merge school or teacher demographics with the TELL Kentucky survey responses and student academic outcome, raw TELL Kentucky data (5-point Likert-type scale at individual level) were aggregated to the school level. In the aggregated data, groups of observations (individual data) were replaced with summary statistics (either average or sum scores) based on those observations. Teacher demographic backgrounds in the SWPBIS schools and non-SWPBIS schools were comparable in terms of average years of teaching and teacher education level.
Measures
This study used existing datasets: PBIS implementation data, TELL Kentucky 2011 survey data, and School Accountability data (student achievement and school demographics).
BoQ
BoQ data were provided by the KYCID in the form of fidelity scale scores by school level. The BoQ instrument is completed by school-based leadership teams made up of teachers and administrators responsible for leading SWPBIS implementation efforts. The BoQ includes 53 items self-assessing SWPBIS implementation (Cohen, Kincaid, & Childs, 2007; Lewis & Sugai, 1999). SWPBIS leadership team members complete individual rating forms, and a SWPBIS coach (a teacher who receives two additional days of training on SWPBIS components, with a special emphasis on data-based decision making) completes a more detailed rating form that provides additional indicators about SWPBIS implementation. A scoring rubric provides clarity and consistency in scoring items. Teams tabulate and discuss results, determining a final score for each item. A score of 70 out of a possible 100 is indicative of an adequate level of fidelity implementation; scores below the cutoff are considered to be not meeting fidelity (Childs et al., 2010). Cohen et al. (2007) found the BoQ to have strong psychometric properties including internal consistency (α = .96), test–retest reliability (.94), and interrater reliability (.87).
TELL Kentucky
In Spring 2011, the Kentucky Department of Education (KDE) administered an online survey to all teachers. Called TELL Kentucky, the survey explored teachers’ perceptions of teaching conditions in their schools. TELL Kentucky contained Likert-type items related to eight constructs, including Time, Facilities and Resources, Community Support and Involvement, Managing Student Conduct, Teacher Leadership, School Leadership, Professional Development, and Instructional Practices and Supports (New Teacher Center, 2011b).
TELL Kentucky 2011 survey data, provided by the New Teacher Center (2011b), contains item responses (5-point Likert-type scale) at the level of individual educators. Exploratory and confirmatory factor analyses commissioned by the New Teacher Center indicated that an eight-factor solution accounted for 63% of the variance in responses, and the authors of the TELL Kentucky survey concluded that the construct validity of the TELL Kentucky survey was satisfactory (New Teacher Center, 2011a, 2011b). New Teacher Center researchers calculated Cronbach’s alphas and found all eight constructs to be reliable with alphas above .848 (New Teacher Center, 2011a, 2011b).
Student academic outcomes and school demographics data
Student academic outcomes and school demographics data were obtained from the School Accountability 2011 Profiles, publicly available from the KDE website. This study used schools’ summative performance scores as a dependent variable for student achievement. The summative score (maximum score of 100) is a composite of accountability components, including achievement in tested subject areas, a measure of gap closure based on achievement scores, and a measure of achieved student growth versus anticipated growth. In addition, at the middle school and high school levels, summative scores include a measure of student progress toward college and career readiness standards. High school summative scores also factor in graduation rates. The authors chose to use summative scores as the dependent variable as achievement measures varied across school grade levels.
Data Analysis
All data analyses were conducted using the school-level data. Propensity Score Matching was conducted using the SPSS R plug-in and custom dialog—Propensity Score Matching. First propensity scores were calculated. The five demographic variables (total enrollment, dollars spent per student, percentages of White students, percentages of male students, and percentage of students receiving free/reduced lunch) were used as covariates. These demographic variables were used as matching variables because they were significantly correlated with student achievement (Niu, Zhang, Chon, Norman, & Miller, 2013). Subclassification strategies were used to select the matched non-SWPBIS schools because creating five propensity score subclasses can remove up to 90% of bias due to all covariates included in the propensity score (Cochran, 1968). The propensity scores were sorted into five subclasses for each grade level (elementary, middle, and high). Within each grade level, one-to-one exact matching without replacement was applied and 151 non-SWPBIS schools were selected. After consulting with the regional SWPBIS program coordinator who provided the data, seven non-SWPBIS elementary schools were removed from the matched comparison group because these schools had partially implemented SWPBIS in 2010–2011. Thus, a total of 151 SWPBIS schools (95 elementary, 31 middle, 25 high) and 144 non-SWPBIS schools (89 elementary, 30 middle, 25 high) were included in the final sample. Among the selected SWPBIS schools, the years of implementation ranged from 1 to 11 with an average of 3 years. Table 1 presents demographics at each school level and significance tests for the group difference between SWPBIS and non-SWPBIS schools. No differences were found between SWPBIS and non-SWPBIS schools on the demographic variables aforementioned.
School Demographics of SWPBIS and Non-SWPBIS Schools.
Note. SWPBIS = School-Wide Positive Behavior Interventions and Supports.
To address the first research question, whether there were significant differences in teacher perceptions regarding their teaching conditions between SWPBIS schools and non-SWPBIS schools, MANOVA tests were performed using item-level responses (5-point Likert-type scale) on each of the eight TELL Kentucky constructs as dependent variables and treatment condition (SWPBIS vs. non-SWPBIS) as the independent variable. If MANOVA showed significant treatment effect on a particular TELL construct, ANOVA tests were performed to examine the treatment difference on each survey item of the construct.
The second research question investigated whether the level of a school’s fidelity of implementing SWPBIS (measured by the BoQ) affected teachers’ perceptions of their teaching conditions. Based on BoQ scores, the SWPBIS schools in the sample were classified as low-fidelity implementers (34 schools implementing 70% of program benchmarks or less), medium-fidelity implementers (67 schools implementing 71%-89% of program benchmarks), or high-fidelity implementers (50 schools implementing 90% or more of benchmarks). Although the BoQ only classifies schools as low- or high-fidelity implementers, the authors hypothesized a link between depth of implementation and teacher satisfaction and opted to include this three-level classification system to tease out differences among schools that had reached acceptable levels of implementation as measured by the BoQ. MANOVAs were conducted using item-level responses (5-point Likert-type scale) on each TELL Kentucky construct as dependent variables and implementation level (high, medium, and low) as the independent variable for all SWPBIS schools.
The third question investigated whether the level of a school’s fidelity of implementing SWPBIS affected student academic outcomes. ANOVAs were conducted using overall scores as dependent variables and implementation status (high-, medium-, and low-fidelity, non-SWPBIS) as the independent variable. To further determine whether fidelity scores (BoQ) made a unique contribution to student academic outcomes after controlling for years of schoolwide PBIS implementation and school demographic variables, hierarchical linear regression was performed with overall scores as the dependent variable. In Step 1, years of SWPBIS implementation and school demographic variables (total enrollment, free, or reduced lunch rates, dollars spent per student, percentages of White students, and percentages of male students) were entered in regression. In Step 2, BoQ scores were entered into the regression.
Results
The first research question explored differences in teacher perceptions of teaching conditions between SWPBIS and non-SWPBIS schools. Results for MANOVA and follow-up ANOVA tests are summarized in Tables 2 and 3. MANOVAs showed significant differences on two constructs: Managing Student Conduct, F(7, 293) = 4.91, Wilks’ Λ = .89, p < .001,
Summary Statistics of All Items on Managing Student Conduct by Program.
Note. SWPBIS = School-Wide Positive Behavior Interventions and Supports.
p < .05. **p < .01.
Summary Statistics of Selected Items on School Leadership by Program.
Note. SWPBIS = School-Wide Positive Behavior Interventions and Supports.
p < .05.
Table 3 displays the summary statistics of these five items on School Leadership. Follow-up ANOVAs on School Leadership found significant differences on two items and marginal significance on three items. Teachers in SWPBIS schools reported more clearly defined expectations, F(1, 235) = 4.51, p < .05,
The second research question examined whether a school’s fidelity of implementation of SWPBIS affected teacher perceptions of their working conditions. MANOVAs indicated significant differences among different levels of implementation on teacher perceptions of Managing Student Conduct, F(14, 284) = 2.78, Wilks’ Λ = .77, p < .01,
Summary Statistics of All Items on Managing Student Conduct by Fidelity Level.
p < .05. **p < .01.
Table 5 presents the summary statistics of teachers’ perception of Community Support and Involvement by fidelity level. Follow-up ANOVAs and multiple comparisons with Bonferroni correction found significant differences between high- and low- fidelity levels on all eight items of Community Support and Involvement, ps < .05. Significant differences were also found between medium- and low- fidelity schools on most items, ps < .05. Multiple comparisons with Bonferroni correction found that teachers in the high- and medium-fidelity PBIS schools reported more positive perceptions of parent–teacher communication, parent involvement, and community support than teachers in the low-fidelity PBIS schools, ps < .05.
Summary Statistics of All Items on Community Support and Involvement by Fidelity Level.
p < .05. **p < .01.
Table 6 presents teachers’ perception of Teacher Leadership by fidelity level. Follow-up ANOVAs and multiple comparisons found significant differences between high- and low-fidelity levels, and between high and medium-fidelity levels on thirteen items of Teacher Leadership, ps < .05. Multiple comparisons with Bonferroni correction found that teachers in the high-fidelity PBIS schools had more positive perceptions of the leadership opportunities and roles teachers have than teachers in the medium- and low-fidelity SWPBIS schools, ps < .05.
Summary Statistics of Significant Items on Teacher Leadership by Fidelity Level.
p < .05. **p < .01.
The third question addressed whether the levels of school-wide PBIS implementation affected student academic outcomes. Table 7 presents descriptive statistics of overall scores by fidelity level. ANOVAs found significant differences among different levels of PBIS implementation on overall student test scores, F(3, 291) = 3.42, p = .01,
Descriptive Statistics of Student Academic Outcomes by Fidelity Level.
Note. SWPBIS = School-Wide Positive Behavior Interventions and Supports.
p < .05. **p < .01.
Hierarchical linear regression was conducted to examine the unique contribution of SWPBIS fidelity (as measured by the BoQ scores) to student academic outcomes after controlling for school demographics. Table 8 summarizes the hierarchical linear regression results for all SWPBIS schools. Results showed that in Step 1, ethnicity (percentages of White students) and free/reduced lunch rates significantly predicted student achievement scores. In Step 2, ethnicity (percentages of White students) and free/reduced lunch rates remained significant predictors of student achievement scores; BoQ scores made a unique contribution to overall scores beyond years of PBIS implementation and school demographic variables, p = .03. These results highlight the importance of program implementation fidelity on student learning outcomes.
Hierarchical Linear Regression Predicting Overall Achievement Scores for SWPBIS Schools (N = 151).
Note. SWPBIS = School-Wide Positive Behavior Interventions and Supports; BoQ = Benchmarks of Quality.
p < .05. **p < .01.
Discussion
SWPBIS implementation positively affects teacher perceptions of Managing Student Conduct and School Leadership, as measured by the 2011 TELL Kentucky statewide teacher survey. High-fidelity SWPBIS implementation promoted positive teacher perceptions of teaching conditions on Managing Student Conduct, Community Support and Involvement, and Teacher Leadership. Significant differences among different levels of SWPBIS implementation on overall achievement scores were also found. High-and medium-fidelity schools had significantly higher overall achievement scores on statewide achievement tests than low-fidelity schools and non-SWPBIS schools. Furthermore, BoQ fidelity scores uniquely predicted academic achievement after controlling for years of SWPBIS implementation and school demographic variables.
The findings of this study make a significant contribution to research literature on the effects of SWPBIS implementation. Previous research found that SWPBIS implementation reduces student discipline infractions and can lead to long-term changes in schoolwide disciplinary practices (Barrett et al., 2008; Luiselli et al., 2005; Mass-Galloway et al., 2008; Mayer et al., 1993; Scott & Barrett, 2004). Similar to Bradshaw et al. (2008), who found that SWPBIS affects teacher perceptions of organizational health, the present study demonstrates that SWPBIS also contributes to greater levels of Kentucky teachers’ satisfaction regarding their teaching conditions. Specifically, teachers in SWPBIS schools reported a stronger sense of shared expectations for student conduct among both teachers and students, an explicit goal of the SWPBIS framework (Lewis & Sugai, 1999; Sugai & Horner, 2002). Moreover, effect sizes for our findings at the construct level were moderate to large (Cohen, 1988), consistent with Bradshaw et al.’s (2008) findings regarding teacher perceptions of organizational health in SWPBIS schools, which also reached practical levels of significance.
Teachers in Kentucky SWPBIS schools also indicated higher levels of satisfaction on the construct of School Leadership than teachers in non-SWBPIS schools. The specific TELL Kentucky survey items that were significantly higher for SWPBIS schools related to district-level, rather than school-level leadership. Teachers in SWPBIS schools report that their district leaders have clear expectations for school performance and establish a clear mission and purpose for schools. Among the sample of SWPBIS schools included in this study, 80% were part of a districtwide SWPBIS initiative. This means that a district leadership team was established to support and monitor SWPBIS implementation, and may have contributed to teachers’ positive views on the School Leadership construct. Marzano and Waters (2009) found that high-performing districts tended to have leaders who engaged teachers in collaborative goal setting and consistently held all district staff accountable for progress toward those goals, both key features of SWPBIS (Sugai & Horner, 2002). Further research into the role of district leaders in SWPBIS schools is warranted.
Another unique contribution of this study was the examination of differences between Kentucky schools implementing SWPBIS at different levels of fidelity. This study revealed significant differences in teacher perceptions between schools at various implementation levels on three TELL Kentucky constructs (Managing Student Conduct, Community Support and Involvement, and Teacher Leadership). Teachers in high-implementation SWPBIS schools were not only more satisfied with student conduct and staff unity around the issue of student discipline but also reported higher levels of community investment in the school, and greater levels of teacher empowerment and involvement in decision making than their counterparts in low-implementation SWPBIS schools.
Moreover, on the construct of Managing Student Conduct, teacher responses on every item were significantly different between high-fidelity and low-fidelity SWPBIS schools. This suggests that as schools improve their implementation of SWPBIS, teacher perceptions of many aspects of student behavior management steadily improve. A practical implication is that schools considering adoption of SWPBIS, or currently involved in implementation, should recognize that fidelity of program implementation can make a significant difference in outcomes (Childs et al., 2010). Likewise, findings related to Community Support and Involvement may suggest the importance of having unity about school purpose, culture, and behavior expectations not only among teachers but also among parents and the larger community (Bradshaw et al., 2008; Lucyshin, Dunlap, & Albin, 2002).
This study found no significant difference in school achievement between the SWPBIS and non-SWPBIS schools. However, Kentucky schools implementing SWPBIS with high and medium fidelity did, indeed, had higher levels of student achievement than low-fidelity implementers. More interestingly, fidelity scores uniquely predicted student academic achievement after controlling for years of SWPBIS implementation and school demographic variables. These results underscore the importance of implementation integrity in schoolwide efforts for improvements in academic areas. Debnam, Pas, and Bradshaw (2012) found that elementary schools implementing SWPBIS with fidelity may still fail to realize its full capabilities by underutilizing a three-tiered system of progressively more intense interventions for students who do not respond to schoolwide expectations. Implemented with such a Multi-Tiered System of Supports (MTSS; also commonly known as Response to Intervention [RtI]), high-fidelity implementation of SWPBIS may provide even greater leverage for improving student academic achievement (Shores & Chester, 2009). Further research is warranted, both in terms of the linkage between SWPBIS and student achievement and the predictive capability of the TELL Kentucky survey to distinguish between high- and low-performing schools.
Limitations and Future Research
Despite the positive findings that teachers in SWPBIS schools report more favorable working conditions as compared with teachers in non-SWPBIS schools, this study features a significant limitation. No data were available to measure teacher working conditions prior to implementation of SWPBIS. It is conceivable that schools with more positive working conditions are more predisposed to adopt or to implement SWPBIS with higher levels of fidelity. In this way, we cannot be confident that SWPBIS actually contributed to the working conditions as reported in those schools.
In a similar way, the findings that high-fidelity SWPBIS schools had higher levels of student achievement must be viewed cautiously. Higher achieving schools may already possess conditions favorable to SWPBIS implementation, including more distributed leadership structures or more focus on data-driven decision making. To confirm this speculation regarding SWPBIS and working conditions, we recommend additional studies using the kinds of longitudinal and randomized control structures found in Bradshaw et al. (2008) and using measures such as the TELL Kentucky survey to assess the impact of SWPBIS on working conditions.
This study also suggests new directions and applications for the TELL Kentucky, as it is the first known research utilizing the survey as an outcome measure of teacher perceptions to evaluate school initiatives. Future research might consider using the TELL Kentucky to measure the impact of other programs and initiatives on teacher perceptions.
Although propensity score methods are useful in estimating causal effects for nonexperimental data, other confounding factors (e.g., school culture, other similar variables) may undermine the equivalence of SWPBIS and non-SWPBIS schools. Another limitation is that, after the propensity score analysis, researchers had to drop seven selected non-SWPBIS schools that had partially implemented SWPBIS according to the SWPBIS district coordinator. These partial implementers were originally included in the set of non-SWPBIS schools. Finally, the TELL Kentucky survey contains limited demographic information of the respondents with only educator roles and years of experience. More detailed demographic information (e.g., gender, age) of respondents is lacking in the current study. Furthermore, this analysis is based on the aggregated responses of all instructional staff who work within the school, including teachers (89%), administrators (4%), and other educational professionals (7%). A more specific analysis of only teacher responses may have yielded different results.
A final limitation lies in the somewhat arbitrary way the researchers classified SWPBIS schools as medium- and high-fidelity implementers. We defined low-fidelity implementers as those meeting 70% or less of the BoQ indicators because this is the cutoff established by KYCID for their own coaching and school support purposes. A score below 70 actually indicates that the school lacks fidelity of implementation. To distinguish between medium- and high-fidelity implementers, however, we had to exercise a judgment call and decided to use the rigorous standard of 90% of BoQ benchmarks to define high-fidelity implementation. It is possible that dividing the medium- and high-fidelity implementers at different cut-points may have produced different results. However, the finding that high-fidelity implementers have higher achievement levels than low-fidelity implementers is encouraging, given the relatively rigorous standard we established for inclusion in the high-fidelity group.
Despite these limitations, this study makes a new contribution to research on SWPBIS through its use of propensity score matching and the use of the TELL Kentucky survey as a measure for program implementation, a finding of interest to researchers of SWPBIS and teacher working conditions. All P-12 practitioners concerned about improving working conditions and promoting strong, learning-focused school cultures may be interested in this study both from what it reveals about the TELL Kentucky survey and implementation of SWPBIS.
Footnotes
Acknowledgements
The authors acknowledge their gratitude to the New Teacher Center, Santa Cruz, California, for providing access to 2011 TELL Kentucky survey data and to the Kentucky Center for Instruction Discipline for access to school-level Benchmarks of Quality (BoQ) data used in the completion of this study. The authors also express appreciation to Professor George Sugai, University of Connecticut, for his thoughtful feedback on the manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
