Critical Incidents in Sustaining School-Wide Positive Behavioral Interventions and Supports

Setting and Participants

We studied a school district in rural British Columbia with schools that had been implementing SWPBIS for more than 15 years. The district enrolled 14,000 students in 32 schools. Specific settings for this study included the district office and three elementary schools, each of which had each been implementing SWPBIS for between 10 and 14 years. At each school, implementation of Tier I SWPBIS was documented as adequate (range: 86%−89%) on the School-Wide Evaluation Tool (SET; Sugai, Lewis-Palmer, Todd, & Horner, 2001), a research validated external evaluation of Tier I SWPBIS fidelity of implementation for the year of the study.

Participants were a convenience sample of 17 educators: 4 current and former administrators, 4 district consultants, 3 special education teachers, and 6 general education teachers familiar with the aim and the daily activities of SWPBIS. Of these, 12 (71%) were female. Minimum criteria for participation included receipt of training or professional development in SWPBIS and at least 2 years of experience implementing Tier I SWPBIS in their current school (or district, for district administrators). Respondents reported an average of 9 years of experience implementing SWPBIS in schools in the district studied, either as a teacher or administrator (range: 5–15 years).

Qualitative Approach

Critical Incident Technique (CIT; Flanagan, 1954) is a phenomenological qualitative research method, from industrial and organizational psychology, that focuses on participants’ lived experiences to draw pragmatic insights concerning real-world problems (Butterfield, Borgen, Amundson, & Maglio, 2005; Woolsey, 1986). Studies suggest that the approach is reliable and valid because of its focus on rich analysis of observable behaviors (Anderson & Nilsson, 1964; Kain, 2004; Ronan & Latham, 1974). CIT has been used in education to examine principals’ change in support for SWPBIS (McIntosh, Kelm, & Canizal Delabra, 2015), teacher collaboration (Kain, 1997), and teacher job satisfaction (Engelking, 1986).

CIT is based on identification and analysis of Critical Incidents (CIs), defined as “any reported occurrence that could be translated into specific, observable, and behavioral terms” (Bedi, Davis, & Williams, 2005, p. 314), with a particular focus on observable, measurable events that are perceived as helping or hindering a process of interest (e.g., regular team meetings). This focus on observable, measurable events mitigates concerns regarding the accuracy of participant recall (Kain, 2004). Moreover, individual interviews and follow-ups with multiple respondents are used to compile and analyze a set of CIs, which allow researchers to identify potential inconsistencies in recall and perform member checks to clarify accounts (Kain, 2004).

Procedure

Data Analysis

Credibility and Trustworthiness of Data and Interpretations

Although the process of coding data into distinctive thematic units is inherently subjective, five steps were taken to assess and secure the trustworthiness of the current findings (as recommended for CIT research by Butterfield et al., 2005). These steps closely align with the credibility checks and quality indicators for interview studies and qualitative data analysis outlined by Brantlinger and colleagues (2005) for special education research. First, expert feedback on the appropriateness and utility of the categorical titles and operational definitions was obtained from the second author, who was blinded to the coding process, through a process known as peer debriefing (Brantlinger et al., 2005). Second, results were compared with categories in the existing literature (Brantlinger et al., 2005; Butterfield et al., 2009). The majority of elements previously articulated in the literature review were reflected in the results obtained. The third verification step involved calculating intercoder agreement for sorting CIs into categories (Anderson & Nilsson, 1964). An independent rater, a published researcher in qualitative research, sorted a randomly selected 25% of the CIs into the 13 categories, using a codebook with the category titles and a brief description of each category. The intercoder agreement was 87%, which is considered to be strong according to Anderson and Nilsson (1964). The fourth accountability procedure, a second-level member check (Brantlinger et al., 2005), was conducted to examine the interpretive validity of the CI extractions and formulation of the categories. Member checks establish the trustworthiness of the researcher’s interpretation according to the intended meaning of the participant (Bedi et al., 2005). The first author provided participants with definitions of all of the final categories and asked them whether the categories captured their experiences. Overall, 73% of the participants completed this member check. Of these participants, all but one indicated that all of the categories resonated with their experiences (this participant suggested a different name for one category, which we adopted). The final step to assess trustworthiness was to set a minimum participation rate in a given category. Flanagan (1954) asserted that a category was valid only if “significant frequencies” of CIs are reported under that category. For this study, categories were deemed valid only if at least 25% of the participants reported a critical incident in that category (Butterfield et al., 2009).

Positionality

Qualitative research requires examining what personal perspectives may influence the research and data analysis process (Brantlinger et al., 2005). The first author, the primary researcher, is an educator with special education teaching experience with a strong belief in data-driven instruction but no previous experience in SWPBIS. The second and third authors are published researchers in the field of positive behavior support with an interest in identifying how SWPBIS can be sustained. Furthermore, the second author provided training and technical assistance to the school district studied. The fourth author is a doctoral student with SWPBIS experience.

Continuous Teaching

Continuous Teaching, cited by 88% of the participants, was perceived as a strong method to enhance SWPBIS sustainability. These CIs refer to consistent reteaching of expectations and social skills through classroom lessons, incidental teaching, assemblies, and presentations. According to participants, the experience of regularly seeing and hearing the expectations through various modes promotes a consistent, unified SWPBIS culture within schools. Participants expressed that the process of continuous teaching focused and operationalized the aim and daily activities of SWPBIS for school personnel, making the expectations “part of our culture. And so if you’re on staff you feel like you need to be a part of that culture.” For them, the explicit and ongoing communication helped their schools form a collective purpose:

For our kids, common language and common understanding are the two biggest things that they need to have. They need to know that when I talk to them at that school about [expected] behavior that we’re all talking about the same thing. One of the major things that we continue to do . . . is a re-teaching of the matrix to our students . . . And quite frankly I think the big piece of it for me was that it was not only for kids. It was for staff.

Not surprisingly, a lack of continuous teaching was described as a hindering event. With a lack of review of core expectations and procedures, staff and student behavior can regress: “When we didn’t [reteach] . . . we saw a greater increase in negative behavior . . . kids running in the hallways . . . teachers reacting . . . it became apparent that staff were contributing to the problem.”

Positive Reinforcement

Fourteen participants (82%) cited Positive Reinforcement as an important factor in sustaining SWPBIS. This category includes (a) a general focus on prosocial behavior, (b) use of school-wide systems for positive reinforcement, and (c) reinforcement of staff behavior. It applies to adults reinforcing student prosocial behavior and adults being reinforced (e.g., through improved student outcomes) for their implementation of SWPBIS:

It’s really giving the kids a double hit for every single [acknowledgment ticket] because the first hit is they got the acknowledgement but then they’re getting another opportunity to be reinforced by being able to trade it in . . . [When students are] not getting positives at home . . . they need a second hit.

Participants reported that using a SWPBIS acknowledgment system occasioned student change, and observing that change occasioned adult implementation of SWPBIS. When asked to recall an event that helped sustainability, the vast majority noted detailed examples of students’ behavior improving on receiving positive acknowledgment, and such improvement served as a stimulus for them. As noted by one administrator about her staff, “they believe that what we’re doing is actually making a difference because we see change in behavior.” Another participant noted,

If people find a very positive environment with children . . . they’re very respectful and so on . . . then you want to . . . It reinforces that what we’re doing is right and we don’t want to lose that, we want to keep going.

Sustainability was hindered when teams did not refresh aspects of the acknowledgment system, which made it less appealing to students and staff alike.

SWPBIS Team Effectiveness

A total of 88% of the participants discussed the importance of SWPBIS Team Effectiveness, the organizational structure within the school to support SWPBIS implementation. Effective teams met consistently and had a broad representation of staff members, with mechanisms for training staff and reporting back to the rest of the school faculty. When SWPBIS teams were well-formed, met regularly, and held a high profile in the school, they were identified as beneficial. As seen in the following excerpt, effective SWPBIS teams provided the organization needed to implement the SWPBIS framework and ensure follow-through for the whole school:

We [meet] before our staff meetings so that we can relay the information to the staff . . . we see what we talked about last time and where we’re going . . . looking at what we’ve discussed last time and our success or, if there’s something that we didn’t get to and making sure that we’re making goals that are actually happening.

Effective teams were able to maintain the conversations about SWPBIS at the school level and allowed people the space to voice concerns and share insights. Hindering CIs in this category identified a lack of organizational structure—such as no regular meetings, no defined roles, poor collaboration, or fragmented team members—as a barrier to sustainability.

Staff Ownership

Staff Ownership, highlighted by 76% of the participants, refers to SWPBIS as a teacher-generated and owned initiative, as opposed to a top–down mandate imposed by administrators. This category centered on teacher buy-in and a high level of involvement in planning and implementation. Participants often identified school champions, lead implementers with passion, credibility, and willingness to keep staff members moving forward with implementation:

If it is given to you from the principal, then there is an automatic reaction, the question is whether I have to do it or I don’t have to do it. I think that a lot of it came from the staff people, the people in the trenches [who] are dealing with some of the behaviors. It was teacher generated . . . there is a reason for why you’re doing it, because it makes my job easier, because I can tell you as a colleague, hey I’m doing this and it’s . . . staff driven. Not being told to do it.

Despite turnover in staff and administrators, the grassroots nature of SWPBIS allowed it to continue even when champions and supportive administrators moved on:

There was enough of us who were dedicated to make sure that the transition happened—that [SWPBIS] would happen no matter what—because our initial core group . . . knew what it was like before [SWPBIS] . . . to have people who, who saw the overall picture and although people have come in and left, you still had to have the people who had the vision of what it could be and maintain it.

Adaptation

Adaptation, noted by 71% of the participants, refers to maintaining core elements while adapting daily practices to make them more efficient and effective. It also includes customizing SWPBIS practices to “fit” local school context (e.g., local cultural translation of expectations, changing practices to reflect student demographics) to keep practices relevant. As one participant reported, “we’re constantly having to re-evaluate both our systems and how those systems match with the student body and the behaviors of those students.” Specific adaptations mentioned included changing acknowledgment systems, revising forms to make them more efficient, and using creative new ideas for teaching expectations. Practitioners explained how these changes both made SWPBIS more relevant for their students and families, and recaptured enthusiasm from teachers and students:

We decide to try different things, not just doing the same things our committees have always done, because kids get tired of it, and the staff get tired of it, and then you lose the power of being inspired, finding things exciting.

Community of Practice

In total, 65% of the participants reported a Community of Practice as a helpful activity toward sustainability of SWPBIS practices. Events in this category describe networking and connections to peers implementing SWPBIS, as opposed to simply receiving training or coaching from an external expert. CIs provided discussion of annual provincial SWPBIS conferences and district networking sessions that engaged implementers in multi-level dialogue. Forums for presenting, sharing, and listening to others’ ideas, resources, and celebrations kept the conversation of SWPBIS alive:

It was more just having the opportunity to collaborate with other SWPBIS schools. I don’t think it was really a matter of what that collaboration looked like. I think it was the event of the collaboration that was valid and important.

Participants reported that networking served several important functions for sustainability. They referred to networking as a helpful buffer against the obstacle of staff turnover. Staff invited new teachers and principals to the meetings and left with strategic plans to focus SWPBIS. Exchanging information and ideas with other schools, sharing data, hearing how other school teams implement, and sharing concrete examples of practices (e.g., matrices, lesson plans) provided a vehicle to implement and refine existing practices.

Involving New Personnel

Involving New Personnel, cited by 65% of the participants, refers to bringing in new staff, with new ideas, energy, and a fresh perspective. Recruitment of new teachers onto the SWPBIS team builds capacity and increases immersion into the school SWPBIS culture, as well as enhances buy-in from staff when champions rotate off of the team. Each school had specific plans for recruiting new teachers and administrators supportive of SWPBIS, such as job postings stating clearly that SWPBIS was a core school practice and experience with SWPBIS was a preferred hiring characteristic. Beyond hiring practices, participants noted the importance of having ongoing mechanisms in place to ensure that new staff understood SWPBIS. Mentor teachers were assigned to help new teachers learn the expectations and forms. These schools also encouraged all new staff to join the SWPBIS team. One participant recalled, “One of the staff members that came up to me when I first came to the school gave me a binder and explained everything in the binder even before I was full time.” Joining the SWPBIS team was seen as a way to introduce new teachers to the positive school culture and help build their teaching skills:

First year teachers, they’re overwhelmed, but we kept saying, you know, we’d love to see you at a meeting, you know, it would really help out. And so when they go there, it just automatically includes, “you need to start understanding how it really works,” and they got involved.

This process of bringing new members on to the teams and cycling veterans off had the additional beneficial effect of refreshing aspects of the SWPBIS systems as well. The fresh perspectives were seen as important to keeping the practices novel and interesting. As one participant noted, “over time . . . new people who had fresh lenses brought in new ideas.”

Use of Data

Use of Data, provided by 65% of the participants, relates to having observable and measureable information to track patterns of both SWPBIS implementation and student outcomes. Alternatively, lack of access to data was a hindrance. Data facilitate communication and accountability between staff, the school team, the school board, and parents, as one participant noted, “Data speaks to the staff, ‘Look what we’ve accomplished; look where we need to go.’ It keeps it impersonal and really clarifies decision making for the future and how to celebrate the present and past.”

Participants noted data collection as supporting high levels of implementation fidelity. Collecting and using data required teachers to focus on critical SWPBIS activities and guiding principles. It also provided opportunities to develop strategies and evaluate their effectiveness:

The fact that at every meeting we’d pull the data out, and we look at it, and we see as the committee where the problem areas, for instance, the hallway, you know, if there’s an increase, okay, what’s contributing to that, how can we make this, the [office discipline referrals] go down? Or what do we need to do to make sure we’re writing [office discipline referrals] to change behavior?

Access to External Expertise

A total of 59% of the participants highlighted the importance of Access to External Expertise and contact with a recognized researcher, consultant, or trained coach with outside information and tools. When systems become too narrowly focused or stagnant, the voice of an outside perspective can keep the system moving forward. As a participant noted, “it’s good to have outside feedback, someone from the outside look in . . . because our vision had become narrow, we get too much into ourselves.” Those interviewed cited the importance of involvement from coaches who were specially trained to provide SWPBIS implementation support. External coaches (i.e., district staff who assist school teams in implementing SWPBIS) helped staff evaluate and troubleshoot daily practices. Coaches also connected district and school administrators and school staff so that they were building systemic district capacity together.

Maintaining Priority

Maintaining Priority, cited by 53% of the participants, refers to SWPBIS having a high profile within the school or district as a valued practice. Participants emphasized the importance of prioritizing SWPBIS through written policy. New, competing initiatives hindered priority. Yet, viewing SWPBIS as an important practice provided validation, affecting daily practice, funding, professional development selection, and accountability, which in turn affected sustainability:

If the school goal was around social responsibility then it formalized the goal setting, the establishing objectives, collecting data, and reporting on it. It became part of the curriculum component . . . [just like] reading, writing, and spelling . . . Before that, it was kind of invisible. Having social responsibility in the provincial curriculum clearly establishes it within the whole educational framework. It was a critical event in validating schools for doing what they were doing . . . and gave them a context . . . it’s part of what we should be doing at schools.

Participants also expressed that having SWPBIS written into the district action plan and goals sets it as a strong district priority, which ensures that it is viewed as important by schools. One district administrator reported, “if it is not in that document, it can easily be dropped.”

School Administrator Involvement

School Administrator Involvement, cited by 76% of the participants, refers to the critical role of principals as agents who can either help or hinder sustainability. Participants described the value of administrators with a clear understanding of and active support for SWPBIS:

They’re the leader of the school, and that’s a really important person to be at the front, spearheading and taking it seriously . . . They’re the ones who are going to be allowing programs and celebrations and assemblies and encouraging all those behaviors, and they’re the ones who are going to find the time for [additional support and data collection]. So if they don’t believe in it, they’re not going to take the time doing it.

Participants also noted CIs in which the principal’s ability to listen and respect what has been done was critical to durability of practices. Teachers expressed that SWPBIS becomes a priority when the principal is actively involved in ground-level implementation, such as consistently using and promoting SWPBIS language themselves, going into the classroom to directly teach expectations, and giving presentations to staff and community agencies.

Staff Turnover

Staff Turnover, noted by 47% of the participants, refers to the hindering event of staff mobility. Staff turnover can diminish staff knowledge and skills in daily SWPBIS practices (e.g., continuous teaching, data-based decision making) and can reduce staff commitment and consistency. For example, one participant noted, “We’ve had so much staff turnover in the last two years . . . and [SWPBIS] hasn’t been explained, and we just kind of assume that people are on board.”

Conflict in Personal Beliefs/Mistaken Beliefs

Conflict in Personal Beliefs/Mistaken Beliefs, noted by 82% of the participants, regarded two types of conflicts. First, they discussed the notions of different personal philosophies, values, and beliefs, which led to inconsistent implementation. For example, divergent beliefs in the areas of equity, social behavior norms, and rewards and consequences were identified as barriers that often led to lack of engagement and poor implementation of practices. The following excerpt is a typical example of CIs cited under this category: “If they have a different philosophy and it doesn’t fit here and they’ll just do their own thing and that hinders the process of sustainability.” From one administrator’s perspective, the belief that teachers should focus on academics while administration deals with behavior can also be a barrier: “They just want to teach their reading, writing, and arithmetic . . . I think we have that feeling in our building—administration should do something . . . aren’t getting it that it is a team approach.” A second aspect of this category regards mistaken beliefs about SWPBIS. One misconception was that writing office discipline referrals is in itself punitive, not realizing that collecting this type of information could help students in the long term and allow teachers to prevent challenges. Another misconception was that SWPBIS is only the use of external rewards: “. . . educators feeling that it’s only about [acknowledgment tickets], tangible reinforcements, and not understanding it’s something way bigger.”

Unique Contributions or Elaborations

Because of CIT’s pragmatic focus on specific events, findings often provide clear implications for practice, including specific strategies to address both common and less understood issues. Often, participants remarked that events in one category were synergistic, as CIs in some categories increased the importance or salience of other categories as well.

Limitations

The findings from this exploratory study should be viewed with specific methodological limitations. First, the study relied on retrospective self-report, which may be selectively or imperfectly recalled. In addition, events were subjectively recalled as personal reconstructions, which may have been biased by availability heuristics, attributions, and cultural artifacts (Gilovich, Keltner, & Nisbett, 2006). However, CIT includes a mechanism designed to protect against this weakness—its reliance on multiple sources of information allows researchers to identify inconsistencies in accounts and “minimize the effects of a particular faulty recall on the part of an individual participant” (Kain, 2004, p. 79). In this study, we analyzed categories across participants, eliminated categories without sufficient representation, and performed separate member checks at different points in time to reduce potential inconsistencies. Nevertheless, these categories might not be replicated in large-scale empirical studies. To address these issues, future studies could include direct behavior observations and analysis of permanent records (e.g., SWPBIS team meeting agendas, coaching logs) to corroborate events cited by participants.

Finally, these results may be most pertinent to elementary schools where Tier I SWPBIS has been sustained. We cannot assume that non-sustaining schools, middle or high schools, or schools implementing SWPBIS at Tiers II and III would identify the same enablers and barriers. Additional research could include a more diverse range of both sustaining and non-sustaining schools implementing all tiers. Future replications, with more individual respondents across roles, will be needed to assess the validity of these findings beyond this particular sample.