Critical Incidents in Sustaining School-Wide Positive Behavioral Interventions and Supports

Abstract

The purpose of this study was to identify, categorize, and describe practitioners’ perspectives regarding factors that help and hinder sustainability of Tier I (universal) systems within School-Wide Positive Behavioral Interventions and Supports (SWPBIS). Seventeen participants involved in sustaining Tier I SWPBIS over several years within a school district were interviewed and asked what events affected its long-term implementation through a qualitative approach called the Critical Incident Technique (CIT). A total of 227 critical incidents were recorded and sorted into emergent unitary clusters based on content analysis. These categories then underwent rigorous reliability and validity checks, including expert analysis, inter-rater agreement, and participant feedback. This process yielded 13 categories that represent the participants’ experience of sustainability: Continuous Teaching, Positive Reinforcement, SWPBIS Team Effectiveness, Staff Ownership, School Administrator Involvement, Adaptation, Community of Practice, Use of Data, Involving New Personnel, Access to External Expertise, Maintaining Priority, Staff Turnover, and Conflict of Personal Beliefs/Mistaken Beliefs.

Keywords

positive behavior support sustainability systems change qualitative implementation science

Educators today are being challenged to meet social behavior needs in addition to academic objectives, in an environment where time and resources are already stretched to capacity. Rising dissatisfaction with reactive and punitive approaches to school discipline has resulted in a paradigm shift toward more broad-based, preventive, and ecological approaches to supporting student behavior needs. School-Wide Positive Behavioral Interventions and Supports (SWPBIS; Horner et al., 2009) is one example of a framework for supporting student behavior that incorporates the principles of applied behavior analysis, contextual validity, systems change, inclusive ethics, and stakeholder collaboration to enhance learning and social environments with a preventive, systems-level approach. SWPBIS has emerged as an effective, efficient, and socially valid approach for improving behavior outcomes and social competencies (Bradshaw, Waasdorp, & Leaf, 2012; Flannery, Fenning, Kato, & McIntosh, 2014; Horner et al., 2009).

The core features of SWPBIS are applied through a three-tiered continuum of service delivery to optimize development and support the range of student needs. At Tier I, universally effective strategies are provided to all students by all school personnel to promote social responsibility. The school develops school-wide expectations, which are brief in number, contextually defined, and positively worded. Expectations are then posted, defined using a matrix that provides specific examples in each setting, taught explicitly, and reinforced strategically. Explicit teaching includes targeted lessons, demonstrations in settings where problem behaviors often occur, and practice with performance feedback. Systematic reinforcement of positive behaviors involves high rates of descriptive feedback, often accompanied by external rewards. For example, students might receive acknowledgment tickets, which can be spent at a school store or to attend a school event. Finally, in Tier I, a continuum of instructional consequences for violations of expectations is developed. A formalized process of collecting, entering, and analyzing discipline data is used to direct decision making and student support. Data are examined regularly to determine patterns, and practices are modified to improve effectiveness and efficiency. Students not successful with Tier I supports are provided additional supports (Tier II or III supports) as needed (Bambara, Nonnemacher, & Kern, 2009).

Sustainability

Although SWPBIS has evidence of effectiveness, sustainability of even the most efficacious practices cannot be assumed (Santangelo, 2009). As SWPBIS scales up, educators, researchers, and policy makers need to know much more about what factors support and hinder the sustained implementation of SWPBIS. Within education, several key elements of sustainability have been highlighted. Studies indicate that practitioner buy-in plays a significant role in continuing a practice (Baker, Gersten, Dimino, & Griffiths, 2004). Hieneman and Dunlap (2001) found that sustainability was related to acceptability, contextual fit, and degree of individual and agency collaboration. Empirical studies have noted the importance of a team-based approach and use of data as powerful predictors of sustainability (Coffey & Horner, 2012; McIntosh et al., 2013). Administrator support has also consistently been highlighted as important (Kincaid, Childs, Wallace, & Blase, 2007; McIntosh et al., 2014). Finally, district support has also been found to have a strong effect on sustainability (Santangelo, 2009). However, there remains little empirical guidance regarding exactly what steps at the school and district level are most likely to maximize sustainability (McIntosh, Filter, Bennett, Ryan, & Sugai, 2010).

The Promise of Qualitative Research in Understanding Sustainability

Research in the field of education has recognized the need for more diverse methods of investigation, including the use of qualitative studies, which “shed light on unanticipated consequences of current practice and lead to a better conceptualization of interventions and reappraisal of practice” (Gersten, Baker, Smith-Johnson, Flojo, & Hagan-Burke, 2004, pp. 330–331). Biglan (2004) noted the importance of understanding complex school and district systems change processes through studying individual examples before turning to experimental trials. Regarding SWPBIS specifically, Singer and Wang (2009) argued that qualitative methodologies can complement the more commonly used experimental research methodologies (e.g., single case design). The subjective dimensions of the process of sustaining SWPBIS implementation can be determined through careful observation and by asking people directly to talk about their lived experiences (Brantlinger, Jimenez, Klingner, Pugach, & Richardson, 2005).

In response to this need, we used a qualitative design to evaluate factors perceived as helping or hindering the long-term sustainability of Tier I SWPBIS in schools in a district that had been implementing SWPBIS for more than 15 years. Qualitative research asks questions regarding “how” or “why” sustainability occurs under real-world conditions, which can help the field more fully understand sustainability, as well as what steps can be taken to enhance it. Elias, Zins, Graczyk, and Weissberg (2003) encouraged researchers to unearth practitioners’ implicit theories of how practices are sustained by documenting and exploring their day-to-day experiences. A better understanding of the process by which sustainability occurs can be beneficial in terms of expanding the knowledge base, but equally important, it can provide a stronger theoretical foundation for specific interventions to enhance program sustainability.

The Present Study

Sustained implementation of educational initiatives is a rare phenomenon (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005), and previous qualitative research has focused on failures to sustain practice rather than on factors that help sustain it (Santangelo, 2009; Sindelar, Shearer, Yendol-Hoppey, & Liebert, 2006). Therefore, in the current study, we used qualitative methods to examine both enablers and barriers to sustaining Tier I SWPBIS. Our specific research questions included the following: (a) What incidents help the sustainability of SWPBIS, and (b) what incidents hinder the sustainability of SWPBIS?

Method

Setting and Participants

We studied a school district in rural British Columbia with schools that had been implementing SWPBIS for more than 15 years. The district enrolled 14,000 students in 32 schools. Specific settings for this study included the district office and three elementary schools, each of which had each been implementing SWPBIS for between 10 and 14 years. At each school, implementation of Tier I SWPBIS was documented as adequate (range: 86%−89%) on the School-Wide Evaluation Tool (SET; Sugai, Lewis-Palmer, Todd, & Horner, 2001), a research validated external evaluation of Tier I SWPBIS fidelity of implementation for the year of the study.

Participants were a convenience sample of 17 educators: 4 current and former administrators, 4 district consultants, 3 special education teachers, and 6 general education teachers familiar with the aim and the daily activities of SWPBIS. Of these, 12 (71%) were female. Minimum criteria for participation included receipt of training or professional development in SWPBIS and at least 2 years of experience implementing Tier I SWPBIS in their current school (or district, for district administrators). Respondents reported an average of 9 years of experience implementing SWPBIS in schools in the district studied, either as a teacher or administrator (range: 5–15 years).

Qualitative Approach

Critical Incident Technique (CIT; Flanagan, 1954) is a phenomenological qualitative research method, from industrial and organizational psychology, that focuses on participants’ lived experiences to draw pragmatic insights concerning real-world problems (Butterfield, Borgen, Amundson, & Maglio, 2005; Woolsey, 1986). Studies suggest that the approach is reliable and valid because of its focus on rich analysis of observable behaviors (Anderson & Nilsson, 1964; Kain, 2004; Ronan & Latham, 1974). CIT has been used in education to examine principals’ change in support for SWPBIS (McIntosh, Kelm, & Canizal Delabra, 2015), teacher collaboration (Kain, 1997), and teacher job satisfaction (Engelking, 1986).

CIT is based on identification and analysis of Critical Incidents (CIs), defined as “any reported occurrence that could be translated into specific, observable, and behavioral terms” (Bedi, Davis, & Williams, 2005, p. 314), with a particular focus on observable, measurable events that are perceived as helping or hindering a process of interest (e.g., regular team meetings). This focus on observable, measurable events mitigates concerns regarding the accuracy of participant recall (Kain, 2004). Moreover, individual interviews and follow-ups with multiple respondents are used to compile and analyze a set of CIs, which allow researchers to identify potential inconsistencies in recall and perform member checks to clarify accounts (Kain, 2004).

Procedure

Recruitment

Schools and participants were recruited by the district SWPBIS coordinator based on his experience in implementing SWPBIS in the district. Informed consent was obtained first from the principals of each school, then from all participants at the interview.

Interview process

A semi-structured interview questionnaire was adapted from Bedi et al. (2005) and Butterfield, Borgen, Maglio, and Amundson (2009). To ensure fidelity to the methodology, the interview protocol was reviewed by independent researchers with specialized expertise in the areas of methodology (CIT interviewing) and content (SWPBIS), as recommended by Bedi et al. (2005).

Data were collected through one face-to-face interview conducted by the first author with each individual participant. All interviews were conducted over a 2-month period by the first author. With the permission of the participants, the interviews were tape recorded and transcribed to facilitate thorough and reliable discourse analysis (Butterfield et al., 2005).

Data Analysis

Transcription and extraction

An independent, professional transcriber completed all transcriptions from the audio-tapes. Following transcription, the first author used a method described by Butterfield et al. (2009) to extract distinct CIs. To ensure that the CIs were extracted accurately (i.e., that the transcription was divided into unique CIs correctly), an independent researcher with qualitative research experience was provided a random sample of 25% of the interviews to independently extract CIs for intercoder agreement. Initial results indicated 96% agreement, which after a consensus discussion was raised to 100%.

Descriptive validity

Multiple methods were used to verify the accuracy of data collection based on procedures from Bedi et al. (2005) and Butterfield et al. (2005). First, to serve as a fidelity check for the interviewing process, an expert in the field of CIT methodology analyzed the transcripts from the first two interviews to confirm adherence to the protocol and lack of leading questions. Second, to ensure saturation (the point at which interviews ceased to generate new categories; Flanagan, 1954), the first author tabulated CIs after every three interviews to monitor category development. Saturation was achieved after the first five interviews. Furthermore, at the end of each interview, the interviewer read CIs back to interviewees in their own words, to allow them the opportunity to add, modify, or clarify them (first-level member checking; Brantlinger et al., 2005).

Coding categories

Initial coding entailed an examination of the similarities and differences among helping CIs (Butterfield et al., 2009) using open coding, in which data are segmented into categories of information best representing the “persistent ideas” (Creswell, 2007). The first author periodically analyzed CIs to determine patterns, categories, and commonalities, and then to form preliminary categories through inductive clustering. CIs containing similar words, phrases, and sentences formed emergent categories. As suggested by Butterfield and colleagues (2009), CIs were coded in sets, consisting of three randomly chosen interviews. Following this first extraction, the next sets of interviews were analyzed.

Credibility and Trustworthiness of Data and Interpretations

Although the process of coding data into distinctive thematic units is inherently subjective, five steps were taken to assess and secure the trustworthiness of the current findings (as recommended for CIT research by Butterfield et al., 2005). These steps closely align with the credibility checks and quality indicators for interview studies and qualitative data analysis outlined by Brantlinger and colleagues (2005) for special education research. First, expert feedback on the appropriateness and utility of the categorical titles and operational definitions was obtained from the second author, who was blinded to the coding process, through a process known as peer debriefing (Brantlinger et al., 2005). Second, results were compared with categories in the existing literature (Brantlinger et al., 2005; Butterfield et al., 2009). The majority of elements previously articulated in the literature review were reflected in the results obtained. The third verification step involved calculating intercoder agreement for sorting CIs into categories (Anderson & Nilsson, 1964). An independent rater, a published researcher in qualitative research, sorted a randomly selected 25% of the CIs into the 13 categories, using a codebook with the category titles and a brief description of each category. The intercoder agreement was 87%, which is considered to be strong according to Anderson and Nilsson (1964). The fourth accountability procedure, a second-level member check (Brantlinger et al., 2005), was conducted to examine the interpretive validity of the CI extractions and formulation of the categories. Member checks establish the trustworthiness of the researcher’s interpretation according to the intended meaning of the participant (Bedi et al., 2005). The first author provided participants with definitions of all of the final categories and asked them whether the categories captured their experiences. Overall, 73% of the participants completed this member check. Of these participants, all but one indicated that all of the categories resonated with their experiences (this participant suggested a different name for one category, which we adopted). The final step to assess trustworthiness was to set a minimum participation rate in a given category. Flanagan (1954) asserted that a category was valid only if “significant frequencies” of CIs are reported under that category. For this study, categories were deemed valid only if at least 25% of the participants reported a critical incident in that category (Butterfield et al., 2009).

Positionality

Qualitative research requires examining what personal perspectives may influence the research and data analysis process (Brantlinger et al., 2005). The first author, the primary researcher, is an educator with special education teaching experience with a strong belief in data-driven instruction but no previous experience in SWPBIS. The second and third authors are published researchers in the field of positive behavior support with an interest in identifying how SWPBIS can be sustained. Furthermore, the second author provided training and technical assistance to the school district studied. The fourth author is a doctoral student with SWPBIS experience.

Results

The interviews generated 227 CIs for analysis, far above the recommended minimum of 100 (Flanagan, 1954). Table 1 provides the 13 categories of CIs, including the number of helping and hindering incidents provided by participants, percent of the total number of CIs, and the number and percent of participants providing the CIs.

Table 1.

Critical Incidents by Category and Contributing Participants.

Category	Number of helping incidents (% of total^a)	Number of hindering incidents (% of total^b)	Number of contributing participants (% of participants^c)
Continuous teaching	21 (12.73)	3 (4.84)	15 (88.24)
Positive reinforcement	14 (8.48)	2 (3.23)	14 (82.35)
SWPBIS team effectiveness	14 (8.48)	5 (8.06)	15 (88.24)
Staff ownership	22 (13.33)		13 (76.47)
Adaptation	20 (12.12)	1 (1.61)	12 (70.59)
Community of practice	17 (10.30)		11 (64.71)
Involving new personnel	13 (7.88)		11 (64.71)
Use of data	13 (7.88)	3 (4.83)	11 (64.71)
Access to external expertise	16 (9.70)		10 (58.82)
Maintaining priority	8 (4.85)	9 (14.52)	9 (52.94)
School administrator involvement	7 (4.24)	12 (19.35)	13 (76.47)
Staff turnover		10 (16.13)	8 (47.06)
Conflict of personal beliefs		17 (27.42)	14 (82.35)

Note. SWPBIS = School-Wide Positive Behavioral Interventions and Supports; CI = Critical Incident.

Total helping CIs = 165. ^bTotal hindering CIs = 62. ^cTotal participants = 17.

Continuous Teaching

Continuous Teaching, cited by 88% of the participants, was perceived as a strong method to enhance SWPBIS sustainability. These CIs refer to consistent reteaching of expectations and social skills through classroom lessons, incidental teaching, assemblies, and presentations. According to participants, the experience of regularly seeing and hearing the expectations through various modes promotes a consistent, unified SWPBIS culture within schools. Participants expressed that the process of continuous teaching focused and operationalized the aim and daily activities of SWPBIS for school personnel, making the expectations “part of our culture. And so if you’re on staff you feel like you need to be a part of that culture.” For them, the explicit and ongoing communication helped their schools form a collective purpose:

For our kids, common language and common understanding are the two biggest things that they need to have. They need to know that when I talk to them at that school about [expected] behavior that we’re all talking about the same thing. One of the major things that we continue to do . . . is a re-teaching of the matrix to our students . . . And quite frankly I think the big piece of it for me was that it was not only for kids. It was for staff.

Not surprisingly, a lack of continuous teaching was described as a hindering event. With a lack of review of core expectations and procedures, staff and student behavior can regress: “When we didn’t [reteach] . . . we saw a greater increase in negative behavior . . . kids running in the hallways . . . teachers reacting . . . it became apparent that staff were contributing to the problem.”

Positive Reinforcement

Fourteen participants (82%) cited Positive Reinforcement as an important factor in sustaining SWPBIS. This category includes (a) a general focus on prosocial behavior, (b) use of school-wide systems for positive reinforcement, and (c) reinforcement of staff behavior. It applies to adults reinforcing student prosocial behavior and adults being reinforced (e.g., through improved student outcomes) for their implementation of SWPBIS:

It’s really giving the kids a double hit for every single [acknowledgment ticket] because the first hit is they got the acknowledgement but then they’re getting another opportunity to be reinforced by being able to trade it in . . . [When students are] not getting positives at home . . . they need a second hit.

Participants reported that using a SWPBIS acknowledgment system occasioned student change, and observing that change occasioned adult implementation of SWPBIS. When asked to recall an event that helped sustainability, the vast majority noted detailed examples of students’ behavior improving on receiving positive acknowledgment, and such improvement served as a stimulus for them. As noted by one administrator about her staff, “they believe that what we’re doing is actually making a difference because we see change in behavior.” Another participant noted,

If people find a very positive environment with children . . . they’re very respectful and so on . . . then you want to . . . It reinforces that what we’re doing is right and we don’t want to lose that, we want to keep going.

Sustainability was hindered when teams did not refresh aspects of the acknowledgment system, which made it less appealing to students and staff alike.

SWPBIS Team Effectiveness

A total of 88% of the participants discussed the importance of SWPBIS Team Effectiveness, the organizational structure within the school to support SWPBIS implementation. Effective teams met consistently and had a broad representation of staff members, with mechanisms for training staff and reporting back to the rest of the school faculty. When SWPBIS teams were well-formed, met regularly, and held a high profile in the school, they were identified as beneficial. As seen in the following excerpt, effective SWPBIS teams provided the organization needed to implement the SWPBIS framework and ensure follow-through for the whole school:

We [meet] before our staff meetings so that we can relay the information to the staff . . . we see what we talked about last time and where we’re going . . . looking at what we’ve discussed last time and our success or, if there’s something that we didn’t get to and making sure that we’re making goals that are actually happening.

Effective teams were able to maintain the conversations about SWPBIS at the school level and allowed people the space to voice concerns and share insights. Hindering CIs in this category identified a lack of organizational structure—such as no regular meetings, no defined roles, poor collaboration, or fragmented team members—as a barrier to sustainability.

Staff Ownership

Staff Ownership, highlighted by 76% of the participants, refers to SWPBIS as a teacher-generated and owned initiative, as opposed to a top–down mandate imposed by administrators. This category centered on teacher buy-in and a high level of involvement in planning and implementation. Participants often identified school champions, lead implementers with passion, credibility, and willingness to keep staff members moving forward with implementation:

If it is given to you from the principal, then there is an automatic reaction, the question is whether I have to do it or I don’t have to do it. I think that a lot of it came from the staff people, the people in the trenches [who] are dealing with some of the behaviors. It was teacher generated . . . there is a reason for why you’re doing it, because it makes my job easier, because I can tell you as a colleague, hey I’m doing this and it’s . . . staff driven. Not being told to do it.

Despite turnover in staff and administrators, the grassroots nature of SWPBIS allowed it to continue even when champions and supportive administrators moved on:

There was enough of us who were dedicated to make sure that the transition happened—that [SWPBIS] would happen no matter what—because our initial core group . . . knew what it was like before [SWPBIS] . . . to have people who, who saw the overall picture and although people have come in and left, you still had to have the people who had the vision of what it could be and maintain it.

Adaptation

Adaptation, noted by 71% of the participants, refers to maintaining core elements while adapting daily practices to make them more efficient and effective. It also includes customizing SWPBIS practices to “fit” local school context (e.g., local cultural translation of expectations, changing practices to reflect student demographics) to keep practices relevant. As one participant reported, “we’re constantly having to re-evaluate both our systems and how those systems match with the student body and the behaviors of those students.” Specific adaptations mentioned included changing acknowledgment systems, revising forms to make them more efficient, and using creative new ideas for teaching expectations. Practitioners explained how these changes both made SWPBIS more relevant for their students and families, and recaptured enthusiasm from teachers and students:

We decide to try different things, not just doing the same things our committees have always done, because kids get tired of it, and the staff get tired of it, and then you lose the power of being inspired, finding things exciting.

Community of Practice

In total, 65% of the participants reported a Community of Practice as a helpful activity toward sustainability of SWPBIS practices. Events in this category describe networking and connections to peers implementing SWPBIS, as opposed to simply receiving training or coaching from an external expert. CIs provided discussion of annual provincial SWPBIS conferences and district networking sessions that engaged implementers in multi-level dialogue. Forums for presenting, sharing, and listening to others’ ideas, resources, and celebrations kept the conversation of SWPBIS alive:

It was more just having the opportunity to collaborate with other SWPBIS schools. I don’t think it was really a matter of what that collaboration looked like. I think it was the event of the collaboration that was valid and important.

Participants reported that networking served several important functions for sustainability. They referred to networking as a helpful buffer against the obstacle of staff turnover. Staff invited new teachers and principals to the meetings and left with strategic plans to focus SWPBIS. Exchanging information and ideas with other schools, sharing data, hearing how other school teams implement, and sharing concrete examples of practices (e.g., matrices, lesson plans) provided a vehicle to implement and refine existing practices.

Involving New Personnel

Involving New Personnel, cited by 65% of the participants, refers to bringing in new staff, with new ideas, energy, and a fresh perspective. Recruitment of new teachers onto the SWPBIS team builds capacity and increases immersion into the school SWPBIS culture, as well as enhances buy-in from staff when champions rotate off of the team. Each school had specific plans for recruiting new teachers and administrators supportive of SWPBIS, such as job postings stating clearly that SWPBIS was a core school practice and experience with SWPBIS was a preferred hiring characteristic. Beyond hiring practices, participants noted the importance of having ongoing mechanisms in place to ensure that new staff understood SWPBIS. Mentor teachers were assigned to help new teachers learn the expectations and forms. These schools also encouraged all new staff to join the SWPBIS team. One participant recalled, “One of the staff members that came up to me when I first came to the school gave me a binder and explained everything in the binder even before I was full time.” Joining the SWPBIS team was seen as a way to introduce new teachers to the positive school culture and help build their teaching skills:

First year teachers, they’re overwhelmed, but we kept saying, you know, we’d love to see you at a meeting, you know, it would really help out. And so when they go there, it just automatically includes, “you need to start understanding how it really works,” and they got involved.

This process of bringing new members on to the teams and cycling veterans off had the additional beneficial effect of refreshing aspects of the SWPBIS systems as well. The fresh perspectives were seen as important to keeping the practices novel and interesting. As one participant noted, “over time . . . new people who had fresh lenses brought in new ideas.”

Use of Data

Use of Data, provided by 65% of the participants, relates to having observable and measureable information to track patterns of both SWPBIS implementation and student outcomes. Alternatively, lack of access to data was a hindrance. Data facilitate communication and accountability between staff, the school team, the school board, and parents, as one participant noted, “Data speaks to the staff, ‘Look what we’ve accomplished; look where we need to go.’ It keeps it impersonal and really clarifies decision making for the future and how to celebrate the present and past.”

Participants noted data collection as supporting high levels of implementation fidelity. Collecting and using data required teachers to focus on critical SWPBIS activities and guiding principles. It also provided opportunities to develop strategies and evaluate their effectiveness:

The fact that at every meeting we’d pull the data out, and we look at it, and we see as the committee where the problem areas, for instance, the hallway, you know, if there’s an increase, okay, what’s contributing to that, how can we make this, the [office discipline referrals] go down? Or what do we need to do to make sure we’re writing [office discipline referrals] to change behavior?

Access to External Expertise

A total of 59% of the participants highlighted the importance of Access to External Expertise and contact with a recognized researcher, consultant, or trained coach with outside information and tools. When systems become too narrowly focused or stagnant, the voice of an outside perspective can keep the system moving forward. As a participant noted, “it’s good to have outside feedback, someone from the outside look in . . . because our vision had become narrow, we get too much into ourselves.” Those interviewed cited the importance of involvement from coaches who were specially trained to provide SWPBIS implementation support. External coaches (i.e., district staff who assist school teams in implementing SWPBIS) helped staff evaluate and troubleshoot daily practices. Coaches also connected district and school administrators and school staff so that they were building systemic district capacity together.

Maintaining Priority

Maintaining Priority, cited by 53% of the participants, refers to SWPBIS having a high profile within the school or district as a valued practice. Participants emphasized the importance of prioritizing SWPBIS through written policy. New, competing initiatives hindered priority. Yet, viewing SWPBIS as an important practice provided validation, affecting daily practice, funding, professional development selection, and accountability, which in turn affected sustainability:

If the school goal was around social responsibility then it formalized the goal setting, the establishing objectives, collecting data, and reporting on it. It became part of the curriculum component . . . [just like] reading, writing, and spelling . . . Before that, it was kind of invisible. Having social responsibility in the provincial curriculum clearly establishes it within the whole educational framework. It was a critical event in validating schools for doing what they were doing . . . and gave them a context . . . it’s part of what we should be doing at schools.

Participants also expressed that having SWPBIS written into the district action plan and goals sets it as a strong district priority, which ensures that it is viewed as important by schools. One district administrator reported, “if it is not in that document, it can easily be dropped.”

School Administrator Involvement

School Administrator Involvement, cited by 76% of the participants, refers to the critical role of principals as agents who can either help or hinder sustainability. Participants described the value of administrators with a clear understanding of and active support for SWPBIS:

They’re the leader of the school, and that’s a really important person to be at the front, spearheading and taking it seriously . . . They’re the ones who are going to be allowing programs and celebrations and assemblies and encouraging all those behaviors, and they’re the ones who are going to find the time for [additional support and data collection]. So if they don’t believe in it, they’re not going to take the time doing it.

Participants also noted CIs in which the principal’s ability to listen and respect what has been done was critical to durability of practices. Teachers expressed that SWPBIS becomes a priority when the principal is actively involved in ground-level implementation, such as consistently using and promoting SWPBIS language themselves, going into the classroom to directly teach expectations, and giving presentations to staff and community agencies.

Staff Turnover

Staff Turnover, noted by 47% of the participants, refers to the hindering event of staff mobility. Staff turnover can diminish staff knowledge and skills in daily SWPBIS practices (e.g., continuous teaching, data-based decision making) and can reduce staff commitment and consistency. For example, one participant noted, “We’ve had so much staff turnover in the last two years . . . and [SWPBIS] hasn’t been explained, and we just kind of assume that people are on board.”

Conflict in Personal Beliefs/Mistaken Beliefs

Conflict in Personal Beliefs/Mistaken Beliefs, noted by 82% of the participants, regarded two types of conflicts. First, they discussed the notions of different personal philosophies, values, and beliefs, which led to inconsistent implementation. For example, divergent beliefs in the areas of equity, social behavior norms, and rewards and consequences were identified as barriers that often led to lack of engagement and poor implementation of practices. The following excerpt is a typical example of CIs cited under this category: “If they have a different philosophy and it doesn’t fit here and they’ll just do their own thing and that hinders the process of sustainability.” From one administrator’s perspective, the belief that teachers should focus on academics while administration deals with behavior can also be a barrier: “They just want to teach their reading, writing, and arithmetic . . . I think we have that feeling in our building—administration should do something . . . aren’t getting it that it is a team approach.” A second aspect of this category regards mistaken beliefs about SWPBIS. One misconception was that writing office discipline referrals is in itself punitive, not realizing that collecting this type of information could help students in the long term and allow teachers to prevent challenges. Another misconception was that SWPBIS is only the use of external rewards: “. . . educators feeling that it’s only about [acknowledgment tickets], tangible reinforcements, and not understanding it’s something way bigger.”

Discussion

This study explored the perspectives of school and district personnel regarding events that affect Tier I SWPBIS sustainability. The CIT methodology, through richly detailed description, provides a comprehensive picture of sustainability as it unfolds in the real world. Data were generated from detailed interviews with 17 participants who were asked to report important events (i.e., specific behaviors or observable happenings) that helped or hindered the sustainability of SWPBIS. Examination of these data revealed 13 categories of CIs, including confirmation of previous findings and some unique contributions. The findings provide additional evidence that staff buy-in, active administrator support, external expertise, teaming, and the use of data are critical variables in sustaining school-based practices (Adelman & Taylor, 2003; Bambara et al., 2009; Coffey & Horner, 2012; Kincaid et al., 2007; McIntosh et al., 2013; McIntosh et al., 2014; Santangelo, 2009). The CIs in this study show the importance of attending to these principles and incorporating them into sustainability action planning.

Unique Contributions or Elaborations

Because of CIT’s pragmatic focus on specific events, findings often provide clear implications for practice, including specific strategies to address both common and less understood issues. Often, participants remarked that events in one category were synergistic, as CIs in some categories increased the importance or salience of other categories as well.

Continuous teaching for continuous regeneration

One cross-category finding was that continuous teaching of expectations and prosocial behavior led to continuous regeneration of the practice. Participants perceived continuous teaching as helpful in reactivating key SWPBIS components. Continuous teaching reminded both staff and students of the guiding principles of SWPBIS. Many staff members recalled that these conspicuous elements of SWPBIS were a regular prompt to use SWPBIS practices. Although educators sometimes consciously abandon programs, they may also slowly neglect practices over time, leading to their demise (Klingner, Vaughn, Hughes, & Arguelles, 1999). Teaching seemed to keep SWPBIS prominent in schools.

Similar to evolutionary principles, practice sustainability depends on the contextual fit of teaching strategies with the ever-changing environmental conditions. When core elements are adapted to address current challenges identified in the school and linked to important outcomes, continued practice is more likely (Han & Weiss, 2005). These ideas are consistent with the finding that flexible initiatives that can be continuously adapted to fit with goals of the community are more sustainable (Pluye, Grad, Dunikowski, & Stephenson, 2005).

Positive reinforcement

Another finding that was highlighted was positive reinforcement as a key mechanism for sustained implementation. In combination, the categories of Positive Reinforcement and Use of Data show the importance of self-sustaining feedback loops that highlight practice effectiveness and motivate future efforts (Han & Weiss, 2005). Participants consistently identified the principle of positive reinforcement as fundamental to sustainability. They reported their personal experiences of observing positive student outcomes (e.g., improved student behavior, achievement, and school climate) as reinforcing their sustained use of SWPBIS. Several studies have also recognized this principle’s importance to sustainability (Baker et al., 2004; Pluye et al., 2005).

CIs from these categories suggest the importance of implementers at the district and school levels using data to highlight the connection between implementation and student success, strengthening the reinforcing properties of continued use. When staff experienced the positive benefits of data collection, their perceptions of the activity improved, as has been shown in other studies (Rogers, 2003). Data systems allowed SWPBIS teams to develop a sense of professional autonomy and become more self-sufficient in directing their own priorities and resources, further increasing positive reinforcement for using data.

Importance of networking

Participants reported that their connection to a community of practice in their district provided not only ongoing information and training but also a sense of a common mission, which strengthened and renewed their commitment (Klingner et al., 1999). According to Baker and colleagues (2004), teachers with experience in the practice reported that their greatest concern was for increased collaboration among professionals, whereas the concerns of teachers new to the practice were primarily related to basic implementation procedures. By encouraging networking across schools, districts can allow implementers at all levels to receive information that is most relevant to them and solve pressing problems.

Bringing new staff into the school culture

According to participants and the literature, staff turnover seriously threatens sustainability (Adelman & Taylor, 2003). In response, participants noted their deliberate, focused efforts and strategic plans to involve new staff members in their SWPBIS community of practice. These strategies included (a) requiring or strongly encouraging new staff to join the school SWPBIS team, (b) pairing them with SWPBIS team veterans for mentoring, (c) encouraging participation in networking sessions and conferences, and (d) changing job descriptions and hiring procedures to require experience or support for SWPBIS, thus institutionalizing the practice into policy (Fixsen et al., 2005).

Involving new staff was also seen as a specific strategy to enhance other categories of sustainability. For example, participants noted that Staff Ownership was strongest when SWPBIS systems were staff-embraced, grassroots systems-change initiatives. This perception was enhanced in part by bringing new staff onto the team and into the fold immediately. This strategy also helped to mitigate the most commonly reported hindering category, Conflict of Personal Beliefs/Mistaken Beliefs. Early immersion of incoming staff in the school’s SWPBIS culture was reported to increase acceptability and dispel misrules about the practice from the start.

School administrator support as barrier

In keeping with the existing literature (Bambara et al., 2009; Kincaid et al., 2007; Santangelo, 2009), school administrator support was identified as important by a majority of participants. However, one interesting finding was the proportion of helping to hindering events. Participants noted more hindering than helping CIs in this category. This pattern may indicate some benefits of a supportive principal, but even stronger drawbacks of an unsupportive principal (Bambara et al., 2009; Pinkelman, McIntosh, Rasplica, Berg, & Strickland-Cohen, 2014). These findings may explain seemingly contradictory findings that school personnel perceive school administrators as the most important factor in promoting sustainability (McIntosh et al., 2014), whereas empirical studies from the same sample indicate administrator support as less influential than effective teaming and use of data for decision making (McIntosh et al., 2013). As a result, school and district teams may want to build action plans not only for recruiting supportive administrators but also for ensuring sustainability through administrative turnover (Strickland-Cohen, McIntosh, & Horner, 2014), as well as enhancing support from skeptical administrators (McIntosh et al., 2015). These results indicate that their active involvement in the team, modeling and reteaching of core features, setting school priorities, and emphasizing data use enhances sustainability.

Limitations

The findings from this exploratory study should be viewed with specific methodological limitations. First, the study relied on retrospective self-report, which may be selectively or imperfectly recalled. In addition, events were subjectively recalled as personal reconstructions, which may have been biased by availability heuristics, attributions, and cultural artifacts (Gilovich, Keltner, & Nisbett, 2006). However, CIT includes a mechanism designed to protect against this weakness—its reliance on multiple sources of information allows researchers to identify inconsistencies in accounts and “minimize the effects of a particular faulty recall on the part of an individual participant” (Kain, 2004, p. 79). In this study, we analyzed categories across participants, eliminated categories without sufficient representation, and performed separate member checks at different points in time to reduce potential inconsistencies. Nevertheless, these categories might not be replicated in large-scale empirical studies. To address these issues, future studies could include direct behavior observations and analysis of permanent records (e.g., SWPBIS team meeting agendas, coaching logs) to corroborate events cited by participants.

Finally, these results may be most pertinent to elementary schools where Tier I SWPBIS has been sustained. We cannot assume that non-sustaining schools, middle or high schools, or schools implementing SWPBIS at Tiers II and III would identify the same enablers and barriers. Additional research could include a more diverse range of both sustaining and non-sustaining schools implementing all tiers. Future replications, with more individual respondents across roles, will be needed to assess the validity of these findings beyond this particular sample.

Conclusion

The events documented in this study illustrate that sustaining a practice among a sea of competing initiatives requires foresight, flexibility, and creativity. Adelman and Taylor (2003) suggested that practices are institutionalized when school personnel actively take on long-term ownership with a plan to address threats to implementation. This study affirms the need for specific strategies to enhance ownership (by both staff and administrators) and counter staff turnover, such as including new personnel as early as possible to improve implementation, enhance continuity, and generate fresh ideas. School staff can be supported through networking meetings, SWPBIS team activities, continuous feedback, and modeling of continuous teaching of core components to foster the establishment of a common language. Because adaptation was also cited as important, school staff can be encouraged to adapt current practices to maintain excitement and meet families’ needs more effectively. Furthermore, because adaptation is crucial to sustainability (Han & Weiss, 2005), adaptation may increasingly become a variable for future research.

Footnotes

Authors’ Note

This study served as the first author’s doctoral dissertation, while the second author was affiliated with the University of British Columbia.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R324A120278 to University of Oregon. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

References

Adelman

Taylor

(2003). On sustainability of project innovations as systemic change. Journal of Educational and Psychological Consultation, 14, 1–25.

Anderson

B. E.

Nilsson

S. G.

(1964). Studies in the reliability and validity of the critical incident technique. Journal of Applied Psychology, 48, 398–403.

Baker

Gersten

Dimino

J. A.

Griffiths

(2004). The sustained use of research-based instructional practice: A case study of peer-assisted learning strategies in mathematics. Remedial and Special Education, 25, 5–24.

Bambara

L. M.

Nonnemacher

Kern

(2009). Sustaining school-based individualized positive behavior support: Perceived barriers and enablers. Journal of Positive Behavior Interventions, 11, 161–176.

Bedi

R. P.

Davis

Williams

(2005). Critical incidents in the formation of the therapeutic alliance from the client’s perspective. Psychotherapy: Theory, Research, Training, 42, 311–323.

Biglan

(2004). Contextualism and the development of effective prevention practices. Prevention Science, 5, 15–21.

Bradshaw

C. P.

Waasdorp

T. E.

Leaf

P. J.

(2012). Effects of school-wide positive behavioral interventions and supports on child behavior problems and adjustment. Pediatrics, 130, e1136–e1145.

Brantlinger

Jimenez

Klingner

Pugach

Richardson

(2005). Qualitative studies in special education. Exceptional Children, 71, 195–207.

Butterfield

L. D.

Borgen

W. A.

Amundson

Maglio

(2005). Fifty years of the critical incident technique: 1954-2004 and beyond. Qualitative Research, 5, 475–497.

10.

Butterfield

L. D.

Borgen

W. A.

Maglio

Amundson

(2009). Using the enhanced critical incident technique in counseling psychology research. Canadian Journal of Counseling, 43, 265–282.

11.

Coffey

Horner

R. H.

(2012). The sustainability of school-wide positive behavioral interventions and supports. Exceptional Children, 78, 407–422.

12.

Creswell

J. W.

(2007). Qualitative inquiry and research design: Choosing among five approaches (2nd ed.). Thousand Oaks, CA: SAGE.

13.

Elias

M. J.

Zins

J. E.

Graczyk

P. A.

Weissberg

R. P.

(2003). Implementation, sustainability, and scaling up of social–emotional and academic innovations in public schools. School Psychology Review, 32, 303–319.

14.

Engelking

J. L.

(1986). Teacher job satisfaction and dissatisfaction. Spectrum, 4, 33–38.

15.

Fixsen

D. L.

Naoom

S. F.

Blase

K. A.

Friedman

R. M.

Wallace

(2005). Implementation research: Synthesis of the literature (FMHI Publications No. 231). Tampa: University of South Florida, the National Implementation Research Network.

16.

Flanagan

J. C.

(1954). The critical incident technique. Psychological Bulletin, 51, 327–358.

17.

Flannery

K. B.

Fenning

Kato

M. M.

McIntosh

(2014). Effects of school-wide positive behavioral interventions and supports and fidelity of implementation on problem behavior in high schools. School Psychology Quarterly, 29, 111–124.

18.

Gersten

Baker

S. K.

Smith-Johnson

Flojo

J. R.

Hagan-Burke

(2004). A tale of two decades: Trends in support for federally experimental research in special education. Exceptional Children, 70, 323–332.

19.

Gilovich

Keltner

Nisbett

R. E.

(2006). Social psychology. New York, NY: Norton.

20.

Han

S. S.

Weiss

(2005). Sustainability of teacher implementation in applied analysis with children. Journal of Abnormal Child Psychology, 33, 665–679.

21.

Hieneman

Dunlap

(2001). Factors affecting the outcomes of community-based behavioral support. Journal of Positive Behavior Interventions, 3, 67–74.

22.

Horner

R. H.

Sugai

Smolkowski

Eber

Nakasato

Todd

A. W.

Esperanza

(2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11, 133–144.

23.

Kain

D. L.

(1997). Critical incidents in teacher collaboration on interdisciplinary teams. Research in Middle Level Education Quarterly, 21, 1–9.

24.

Kain

D. L.

(2004). Owning significance: The critical incident technique in research. In deMarrais

Lapan

S. D.

(Eds.), Foundations for research: Methods of inquiry in education and the social sciences (pp. 69–85). Mahwah, NJ: Lawrence Erlbaum.

25.

Kincaid

Childs

Wallace

Blase

(2007). Identifying barriers and facilitators in implementing school-wide positive behavior support. Journal of Positive Behavior Interventions, 9, 174–184.

26.

Klingner

J. K.

Vaughn

Hughes

M. T.

Arguelles

M. E.

(1999). Sustaining research-based practices in reading. Remedial and Special Education, 20, 263–274.

27.

McIntosh

Filter

K. J.

Bennett

J. L.

Ryan

Sugai

(2010). Principles of sustainable prevention: Designing scale-up of school-wide positive behavior support to promote durable systems. Psychology in the Schools, 47, 5–21.

28.

McIntosh

Kelm

J. L.

Canizal Delabra

(2015). In search of how principals change: Events that help and hinder administrator support for school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions. Advance online publication. doi:10.1177/1098300715599960

29.

McIntosh

Mercer

S. H.

Hume

A. E.

Frank

J. L.

Turri

M. G.

Mathews

(2013). Factors related to sustained implementation of school-wide positive behavior support. Exceptional Children, 79, 293–311.

30.

McIntosh

Predy

L. K.

Upreti

Hume

A. E.

Turri

M. G.

Mathews

(2014). Perceptions of contextual features related to implementation and sustainability of school-wide positive behavior support. Journal of Positive Behavior Interventions, 16, 29–41.

31.

Pinkelman

McIntosh

Rasplica

Berg

Strickland-Cohen

M. K.

(2014). Perceived enablers and barriers related to sustainability of school-wide positive behavioral interventions and supports. Manuscript submitted for publication.

32.

Pluye

Grad

R. M.

Dunikowski

Stephenson

(2005). Impact of clinical information-retrieval technology on physicians: A literature review of quantitative, qualitative, and mixed methods studies. International Journal of Medical Informatics, 74, 745–768.

33.

Rogers

E. M.

(2003). Diffusion of innovations (5th ed.). New York, NY: Free Press.

34.

Ronan

W. W.

Latham

G. P.

(1974). The reliability and validity of the critical incident technique: A closer look. Studies in Personnel Psychology, 6, 53–64.

35.

Santangelo

(2009). Collaborative problem solving effectively implemented, but not sustained: A case for aligning the sun, the moon, and the stars. Exceptional Children, 75, 185–209.

36.

Sindelar

P. T.

Shearer

D. K.

Yendol-Hoppey

Liebert

T. W.

(2006). The sustainability of inclusive school reform. Exceptional Children, 72, 317–331.

37.

Singer

G. H.

Wang

(2009). The intellectual roots of positive behavior support and their implications for its development. In Sailor

Dunlap

Sugai

Horner

(Eds.), Handbook of positive behavior support (pp. 17–48). New York, NY: Springer.

38.

Strickland-Cohen

M. K.

McIntosh

Horner

R. H.

(2014). Sustaining effective practices in the face of principal turnover. Teaching Exceptional Children, 46, 18–24.

39.

Sugai

Lewis-Palmer

T. L.

Todd

A. W.

Horner

R. H.

(2001). School-wide Evaluation Tool (SET). Eugene, OR: Educational and Community Supports.

40.

Woolsey

L. K.

(1986). The critical incident technique: An innovative qualitative method of research. Canadian Journal of Counseling, 20, 242–254.