Examining Behavioral Consultation Plus Computer-Based Implementation Planning on Teachers’ Intervention Implementation in an Alternative School

Abstract

Students who demonstrate the most challenging behaviors are at risk of school failure and are often placed in alternative schools, in which a primary goal is remediating behavioral and academic concerns to facilitate students’ return to their community school. Consistently implemented evidence-based classroom management is necessary toward this goal; yet, many teachers report not having the necessary knowledge or skills to effectively manage behavior in their classroom. Implementation Planning (IP) is a proactive and efficient implementation support that has evidence of being effective when delivered by a consultant during behavioral consultation. The primary aim of this multiple baseline design study was to extend the findings on consultant-mediated IP by evaluating the effect of a self-guided, computer-based version of IP on teacher treatment integrity of classroom management plans. Unlike previous studies of consultant-mediated IP, results from this study indicated a minimal effect across dimensions of teachers’ implementation, and thus also on student disruptions.

Keywords

treatment integrity fidelity classroom management implementation support implementation planning

Students who demonstrate the most challenging behaviors in school and are at risk of school failure are often placed in alternative schools (Lange & Sletten, 2002). An alternative school is defined, at the federal level, as

a public elementary/secondary school that addresses needs of students that typically cannot be met in a regular school, provides nontraditional education, serves as an adjunct to a regular school, or falls outside the categories of regular, special, or vocational education. (Sable, Plotts, & Mitchell, 2010, p. C-1)

The number of students served in these settings has increased significantly in recent years (Lehr, Tan, & Ysseldyke, 2009). A primary goal in alternative schools is reducing problem behavior and remediating academic deficits to facilitate students’ return to their community school (Lehr et al., 2009). To attain this goal, it is essential to adopt and consistently implement evidence-based classroom management strategies as they provide an important foundation for the more intensive interventions required by students in these settings (Simonsen, Myers, & DeLuca, 2010).

Classroom Management Strategies

Building on a systematic review of the classroom management literature (Simonsen, Fairbanks, Briesch, Myers, & Sugai, 2008), a technical assistance document was developed summarizing empirically supported classroom management practices to support teachers’ adoption of best practice (Simonsen et al., 2015). This document overviews practices arranged under three broad domains: foundations, prevention, and responses. These domains of practice align with principles of positive behavioral interventions and supports (PBIS), and can be enhanced through the use of data systems which permit the evaluation of intervention effectiveness. Foundational practices provide structure and predictability for the classroom as well as clear expectations for the students (e.g., optimization of the effectiveness of the physical arrangement of the classroom, use of classroom schedules, and regular teaching and reinforcement of classroom routines and expectations). Preventive practices attempt to reduce the likelihood of problem behavior by consistently encouraging appropriate behavior through active supervision and proximity, as well as delivery of high rates of varied response opportunities, behavior-specific praise, and prompts and precorrections for expected behavior. Finally, response practices encompass methods for responding to inappropriate behavior that range from least (e.g., error corrections, planned ignoring, differential reinforcement) to most intrusive (e.g., response cost, time out). In responding to behavior, Simonsen et al. (2015) emphasized the importance of consistency, specificity, timeliness, and consideration of the function of the behavior. Although it is widely recognized that identification and treatment of the function of an individual’s behavior is an effective intervention practice (Gresham, Watson, & Skinner, 2001), it should be noted that there is a dearth of research on function-based assessment and treatment procedures targeting a class unit. To date, only two studies have investigated descriptive assessment methods to identify functional variables to design treatment for a class (McKerchar & Thompson, 2004; Vanderheyden, Witt, & Gatti, 2001), with only one testing the effectiveness of the treatment following the assessment process. Although preliminary findings are promising, more research and scrutiny are needed before applying these procedures in the field.

Research conducted in general education settings has found that effective use of evidence-based classroom management promotes student engagement and academic outcomes and decreases disruptive behavior (Epstein, Atkins, Cullinan, Kutash, & Weaver, 2008), whereas poor classroom management is associated with increased risk for long-term negative academic, social, and behavioral outcomes (Epstein et al., 2008). Although research on the use of classroom management practices in alternative school settings is more limited, several researchers have called for the extension of universal PBIS practices to alternative schools (e.g., use of classroom schedules, development of a small set of positively stated classroom expectations, teaching and reinforcement of classroom expectations and routines, active engagement of students, and use of a range of behavioral strategies to acknowledge appropriate behavior and respond to inappropriate behavior; Scott et al., 2002). Implementation of such universal practices helps to counter a history of negative school experiences by shifting the focus to reinforcing desired behaviors, as opposed to punishing undesirable ones (Jolivette et al., 2014). Results of recent studies demonstrate that universal PBIS practices can serve an important foundation for intensive interventions as they standardize and enhance the effectiveness of classwide procedures. For instance, the addition of universal PBIS strategies to alternative schools for children with severe emotional/behavioral problems has resulted in fewer serious incidents or discipline referrals as well as improved relationships between staff and students as compared with implementing intensive interventions alone (Jolivette et al., 2014; Simonsen, Britton, & Young, 2010).

Consultation

Consultation is one of the most common approaches for supporting teachers in their identification and delivery of evidence-based interventions (Kratochwill, Altschaefl, & Bice-Urbach, 2014; Kratochwill & Bergan, 1990). In behavioral consultation (BC), a consultant collaborates with a consultee (e.g., teacher) to identify an intervention to support a client (e.g., student, class of students; Kratochwill & Bergan, 1990). Despite decades of research demonstrating the effectiveness of BC (Kratochwill et al., 2014), the model is largely absent of detailed guidance on how to ensure that consultees implement interventions as designed. Research over the past 20 years has made clear that most teacher consultees struggle to implement a wide range of interventions, including classroom management practices, with adequate treatment integrity (TI; for example, Forman et al., 2013; Noell & Gansle, 2014).

Implementation Planning (IP)

These findings make evident that ensuring TI is at the crux of evidence-based practices resulting in improved student outcomes. As such, numerous implementation strategies have been developed to be used by consultants to support teachers’ TI (Long et al., 2016). One proactive method that has demonstrated preliminary efficacy within an educational context is consultant-mediated IP (Sanetti, Collier-Meek, Long, Byron, & Kratochwill, 2015; Sanetti, Collier-Meek, Long, Kim, & Kratochwill, 2014). IP is based on an adult behavior change theory from health psychology, the Health Action Process Approach (HAPA; Schwarzer, 2008), which suggests that initiation and maintenance of new behaviors are predicted in part by the completion of comprehensive IP.

To date, IP has been examined exclusively within the context of a consultative relationship. Following problem identification and analysis, and intervention development, educators and consultants collaboratively complete an implementation plan, which consists of two core components: action and coping planning (Sanetti, Kratochwill, & Long, 2013). The purpose of action planning is twofold. First, the consultant and teacher work together to enhance the fit between each intervention step and the intervention context in a manner that will not compromise intervention effectiveness. Second, the consultant and teacher identify the specific logistics for implementing each intervention step, such as when, where, and how it will be implemented. The purpose of coping planning is to identify potential barriers to implementation and determine how to best overcome them or reduce their impact. In one of the initial empirical evaluations of consultant-mediated IP, Sanetti et al. (2014) found IP increased adherence and quality and decreased variance of teachers’ implementation of individual student behavior support plans. In a follow-up study, Sanetti et al. (2015) replicated these results and found IP increased teacher adherence and quality and decreased variability of behavior support plan implementation. Furthermore, in both studies, when implementation adherence and quality improved following IP, students’ academic engagement increased and disruptive behavior decreased. Despite the encouraging findings related to IP, it requires a consultant to meet in-person one-on-one with an educator to deliver the implementation support. As time is cited as a primary barrier to ensuring TI in schools (Long et al., 2016), utilizing technology to deliver IP may increase its feasibility.

Purpose of the Study

The primary aim of the study is to extend the findings on consultant-mediated IP by evaluating the effect of a self-guided, computer-based version of IP on teacher TI (CB-IP). Based on prior IP research, it is hypothesized that teachers’ classroom management plan (CMP) adherence and quality will increase after completing the self-guided, CB-IP. A secondary aim was to build upon the dearth of research that exists regarding the transportability of universal PBIS practices to alternative school classroom contexts.

Method

Participants and Setting

Participants represent a sample of convenience from the desired target population, teachers in an alternative school setting who requested consultative support regarding classroom management. Participation was voluntary. Participants included four special education, middle/high school teachers (Mr. Carver, Albert, Bart, and Dean) in an out-placement public alternative school in an urban setting in the Northeast. This school provides a small, highly structured environment that offers education and clinical treatment programs for adolescents with learning disabilities and psychiatric challenges not readily met by the regular education setting. The school was selected for inclusion in this study due to its high interest in engaging in PBIS practices. All teachers were male, self-identified as White, held master’s degrees in education, and had an average of 14 years of teaching experience (range = 6–29). The mean number of students per class was six (SD = 2). The most common behavior challenges reported at the onset of the study included inappropriate language (e.g., cursing), off-task/noncompliant behavior, and anger management issues. The most common disabilities documented for students were externalizing disorders (i.e., attention-deficit/hyperactivity disorder [ADHD], oppositional defiant disorder [ODD]) and learning disabilities (i.e., language, developmental, and specific). As a part of their educational programming, each student received a combination of intensive interventions. Specifically, each student received one-on-one therapeutic services weekly and case management and transition planning services, and was on an individual token economy system targeting their unique treatment goals. The token economy system required students to check-in and -out each period regarding their behavioral performance. Student points were added weekly and corresponded to various levels of privilege. The higher the level earned, the more the incentives and privileges that were available to the student.

Two graduate students in school psychology served as the primary consultants. Both consultants had completed didactic training in BC, including assessment and intervention design procedures, and had more than a year of supervised experience in the field. Both consultants received weekly supervision by a licensed psychologist (the first author). Two additional graduate students in school psychology served as secondary observers for interrater agreement. All study consultants and secondary observers received direct training in data collection (i.e., systematic direct observation and use of instrumentation) and study procedures. Mastery was achieved when study consultants and observers obtained a minimum of 80% agreement with a master code on two consecutive practice observations for study variables (i.e., student disruptions and teacher TI). Training first occurred in practice sessions using video clips. Then, mastery from training was confirmed in vivo.

Independent Variable

A CB-IP was created for the purposes of this study. CB-IP involved teachers independently completing intervention planning via use of a dynamic Adobe Acrobat form, guided by an instructional video, as opposed to through one-on-one collaboration with a consultant. When prompted by the researchers, teachers completed CB-IP for the CMP designed for their classrooms. CB-IP was comprised of two parts: action planning and coping planning. Action planning entails having an interventionist plan how they will deliver an intervention in their classroom context. The action planning section of the CB-IP prompted teachers to, first, type in each discrete step of the CMP in the first column of a table. Second, for each discrete CMP step, teachers were prompted to decide if adaptations were needed to improve ease of implementation or fit of the step. If teachers decided an adaptation was necessary, they typed in the revised CMP step in the second column. If no adaptations were needed, the teachers moved to the third step. Third, for each discrete CMP step, teachers were prompted to type in answers to questions about when, where, and how each step would be carried out in the classroom context. For example, for the step “posted and reviewed schedule of activities,” a teacher would first review it and decide whether adaptation to the step is needed or if it should stay consistent with the originally designed plan. Then, the teacher would indicate when it should be implemented (e.g., posted before the start of the class period), where it should be implemented (e.g., visibly on the classroom board), and how it should be implemented (e.g., reviewed daily at onset of the class period). Following completion of action planning, teachers were provided with instructions to complete coping planning. First, teachers were prompted to list up to four barriers to CMP implementation. Second, for each barrier, teachers were prompted to develop specific strategies to reduce or eliminate the barrier. For example, the barrier “students returning from absences or missed classes require my full attention to get them back on track” might be remediated by having those students who miss two or more consecutive classes work one-on-one with the assistant teacher to help them catch up so that the lead teacher can continue to implement the CMP and deliver instruction.

The mean time required to complete the CB-IP was 64.5 min (range = 55–75 min). Notably, none of the teachers adapted the steps in ways that meaningfully changed the original design of the CMP strategies. Instead, they enhanced the specificity of the language to increase fit with the context or clarity regarding step delivery. For example, Mr. Dean enhanced the specificity of the step about posting the class schedule in a visible location by including language about the need for large format print. As another example, Mr. Bart added the need for individualization when implementing the step to review the classroom rule following a transition. On average, teachers made these minor adaptations to four steps of their CMPs; three teachers reported and problem-solved four implementation barriers, except for Mr. Dean, who reported and problem-solved two barriers. A list of implementation barriers reported, and their resolutions are available upon request.

Instrumentation

Instruments to assess classroom management

As an assessment of teachers’ initial classroom management, direct observations of teacher behaviors (i.e., specific and general praise, reprimands, opportunities to respond) and student behaviors (i.e., correct academic responses, class disruptions) were collected using event recording across three, 15-min observations within a 1- to 2-week period. Praise statements were defined as any verbal statement or gesture that indicated the teacher’s approval of a desired academic or social behavior. Praise was classified as specific (i.e., provided feedback identifying the desirable behavior) or general (e.g., displayed approval without identifying a specific behavior). Reprimands included any comments or gestures made by the teacher indicating disapproval of student behavior. An opportunity to respond included any instance the teacher provided an instructional question, statement, or gesture to a student or group of students that sought an academic response. A correct academic response was defined as any time an opportunity to respond was directed toward a student or group of students and the correct response was given by the student or much of the group (≥75%). See below for operational definition of class disruptions.

Instruments to assess dependent variables

CMP TI

Per best practice recommendations, direct observation was used to collect CMP TI data. Individualized TI measures were developed to align with the steps of each CMP. The TI measures’ format was previously established and approved by an expert panel as part of an Institute of Education Sciences, U.S. Department of Education Grant (No. R324A10005). All CMP TI measures were standardized across teachers such that each measure included (a) a column in which each CMP step was listed in operational terms; (b) a column for rating adherence to each step (i.e., 0 = not implemented, 1 = implemented with deviation, 2 = implemented as planned, or NO = no opportunity for implementation of step); (c) a column for rating the quality (or skill) of delivery of each step implemented (i.e., 1 = poor, 2 = fair, 3 = good, 4 = excellent); and (d) a space for the consultant to take notes. Please refer to Sanetti et al. (2015) for the descriptive anchors for each TI rating.

Across the entire 45-min class period, TI was collected for adherence and quality for each step. CMP adherence was calculated as a percentage based on the number of CMP steps “implemented as planned” divided by the total number of steps applicable (i.e., steps that were expected to be implemented per plan) for the observation period. CMP quality was calculated as a percentage based on the number of CMP steps implemented with “good” or “excellent” quality divided by the total number of steps implemented (as planned or with deviation) that period.

Class disruptions

For baseline (i.e., the classroom management assessment period) and each intervention session, data on class disruptions was captured. Class disruptions were defined as any student action that interrupts the regular school or classroom activity. Across the first 15 min of class, event recording procedures were used to collect data on the number of disruptions that occurred across all students in the classroom. The rate of class disruptions per minute was calculated by dividing the total frequency by minutes observed.

Inter-observer agreement (IOA)

Guidelines regarding IOA for single-case design research were strictly adhered to Kratochwill et al. (2010). A second observer was present for at least 20% of the observation sessions by teacher and phase. IOA per instrumentation variable was well above recommended thresholds per phase and overall (M IOA across applicable phases for CMP adherence = 93.8%, CMP quality = 89.8%, specific praise = 99.2%, general praise = 98.0%, reprimands = 98.2%, opportunities to respond = 94.3%, correct academic responses = 95.2%, and class disruptions = 93.0%).

Procedural Integrity

Consultation process

The following instruments were used to measure the extent to which the BC procedures were implemented as planned.

Consultation guide

An adapted version of the Kratochwill and Bergan (1990) BC guide was used to ensure standardization across cases. The guide was used for all three primary interviews (i.e., problem identification, problem analysis, treatment evaluation) conducted by consultants. Similarly, consultation guides were developed and used to standardize brief weekly check-in meetings about CMP implementation.

Consultation process checklists

Consultation interview process checklists, adapted from Kratochwill and Bergan (1990), list the essential components of the three primary interviews. Consultants completed these checklists immediately following each interview as a measure of procedural fidelity. Furthermore, a second rater reviewed 100% of these audiotaped BC interviews. The average percentage of interview components completed across interviews and teachers was 100%; average interrater agreement was also 100%.

IP process

The procedural integrity of teachers’ CB-IP completion was measured by coding the completed CB-IP form using rating forms for action planning and coping planning. These rating forms included a column (a) for detailing the operationally defined steps of action planning and coping planning, (b) for rating the teacher’s adherence to each step of the process (i.e., 0 = none, 1 = limited, 2 = substantial, 3 = complete), and (c) for rating the quality of each step attempted or completed by the teacher (0 = poor, 1 = fair, 2 = good, 3 = excellent). For adherence, integrity was calculated as a percentage based on the number of steps with “complete” implementation divided by the total number of action or coping planning steps. For quality, integrity was calculated as a percentage based on the number of steps performed with “excellent” or “good” quality divided by the total number of action or coping planning steps implemented (completely, substantially, or to a limited degree). Following each teacher’s submission of the permanent product generated through CB-IP, the consultant assessed procedural integrity using the action planning and coping planning fidelity measures. A second rater did the same for 100% of the CB-IP permanent products. Mean adherence and quality for action planning was 93.75% (range = 87.50%–100%) and 90.63% (range = 75%–100%), respectively. Mean adherence and quality for coping planning were both 100%. For action planning and coping planning, the average interrater agreement was ≥97%.

Social Validity

Upon study completion, social validity data were collected on the CMP and CB-IP.

Usage Rating Profile–Intervention Revised (URP-IR)

Teacher participants completed the URP-IR (Chafouleas, Briesch, Neugebauer, & Riley-Tillman, 2011) to provide information about the social validity of the CMPs. The URP-IR is a 29-item, 6-point Likert-type scale questionnaire (1 = strongly disagree to 6 = strongly agree) that covers six domains. Twenty-one items from four domains (i.e., acceptability, understanding, feasibility, and system supports) were used in this study due to their relevance to the CMPs. Subscales have demonstrated acceptable internal consistency reliabilities (α = .80–.95) and structural validity via factor analyses (Briesch, Chafouleas, Neugebauer, & Riley-Tillman, 2013). Total measure Cronbach’s alpha for the present study was .88.

CB-IP Rating Profile

Teachers completed the CB-IP Rating Profile, to provide data on their perceptions of the acceptability of the computer-based implementation support. The CB-IP Rating Profile was created by adapting the 21 items from the URP-IR to reflect CB-IP, as opposed to the CMP intervention. Total measure Cronbach’s alpha for the present study was .93.

Design

A multiple treatment embedded within a dual-randomized multiple-baseline design across teachers was used to evaluate the effects of CB-IP and, for Mr. Albert, in-person (consultant-mediated) IP, delivered within BC dyads, on teachers’ CMP TI (Koelher & Levin, 1998; Kratochwill & Levin, 2010). In this type of design, the basic integrity of the conventional multiple-baseline design is maintained (i.e., the systematic staggering of introduction of the intervention across cases). However, the design negates a response-guided approach to using visual analysis as it is enhanced by further controlling for threats to internal validity through a priori randomization of (a) cases to the order in which they will receive the intervention and (b) the timing of the introduction of the intervention (for detailed explanation, see Koelher & Levin, 1998; Kratochwill & Levin, 2010). In the present study, following the Baseline phase, teachers were randomly assigned to intervention order and the Standard BC phase began. Each teacher was then prompted to complete CB-IP, initiating the CB-IP phase, based on his predetermined randomized phase start point. To determine start points, the researcher identified the total number of CMP intervention sessions from which data could be collected before a major interruption in the school calendar. Next, a start point for each teacher was randomly selected from three time points which permitted appropriate staggering for methodological rigor. In-person IP was also delivered to Mr. Albert due to ethical considerations (see in-person IP phase). To minimize threats to internal validity, observers were blind to study phase (i.e., when teachers completed CB-IP) but not study purpose.

Procedure

Baseline phase

During this phase of the study, consultants gathered data on existing classroom management practices and levels of class disruptions, and developed the CMPs. An uncontrolled baseline was used such that there was no manipulation of existing classroom behavior management practices.

Problem identification interview (PII) and classroom assessment

Assessment of existing classroom management practices was completed through completion of a PII and systematic direct observations of teacher and student behavior. First, the consultants conducted PIIs with each teacher. The PII focused on identifying and operationalizing top classroom behavior management concerns, characterizing environmental events surrounding these concerns (e.g., setting events, antecedents, consequences), and gathering detailed information related to teachers’ existing classroom management. Following the PII, observations of teacher behaviors (i.e., specific and general praise, reprimands, opportunities to respond) and student behaviors (i.e., correct academic responses, class disruptions) were collected across three, 15-min observations within a 1- to 2-week period. Preimplementation data were organized by the five critical features of classroom management (i.e., maximizing structure and predictability, teaching classroom expectations and routines, actively engaging students, regularly applying empirical strategies to promote appropriate behavior, and consistently applying empirical strategies to respond to inappropriate behavior; Simonsen et al., 2008) as an initial step to link assessment data to intervention. All intervention sessions occurred during the class period deemed by the teacher as the most challenging, when the teacher was present. All four teachers selected either the second or third morning period. Class content largely centered on social studies or English for Mr. Bart, Carver, and Albert, and technology/computing for Mr. Dean.

CMP development

CMPs were designed based on results of the data collected during the problem identification stage of BC. Data gathered were compared with guidelines provided in the classroom management literature to identify areas of strength and weakness (e.g., desired student time spent academically engaged is ≥90%; Reinke, Herman, & Sprick, 2011). Based on the work of Simonsen and colleagues (2008) and Simonsen and colleagues (2015), the research team created a menu of empirically proven classroom management strategies, categorized under each of the five critical features of effective classroom management. This menu was used by consultants to link teachers’ assessment results with specific classroom management strategies. This systematic process ensured CMPs were comprised of appropriate and research-supported strategies. As well, it resulted in all CMPs including behaviorally based strategies that attended to teachers’ areas of weakness, aligned with current recommendations for classroom management, and were similar with respect to content and total number of steps (range = 19–20).

Problem analysis interview (PAI)

After CMP development, the teacher and consultant completed a PAI during which they discussed the results of the consultant’s classroom assessment and reviewed a draft CMP. Once the consultant and the teacher reached consensus regarding CMP details, the consultant provided direct training on CMP implementation and supplied the teacher with materials needed for CMP implementation. Direct training occurred at the end of the PAI session, lasted an average of 56 min (SD = 15 min), included modeling by the consultant and role play with feedback, and concluded when the teacher felt comfortable with CMP implementation.

Standard BC phase

After training, teachers implemented the CMP daily during the target class period. Twice per week, observers collected data for 45 min; class disruptions were collected for the first 15 min only whereas CMP adherence and quality data were collected across the total 45 min. In addition, the consultant met with the teacher briefly once per week to answer questions regarding CMP implementation or student concerns.

CB-IP phase

Each teacher completed CB-IP during this phase based on his predetermined randomized phase start point. Within this phase, data on CMP TI and class disruptions continued to be collected.

In-person IP phase

Mr. Albert displayed a declining trend and drop in adherence level immediately following CB-IP; therefore, traditional in-person IP was also delivered. Within this phase, data on Mr. Albert’s CMP TI and class disruptions continued to be collected.

Treatment evaluation phase

Following ongoing CMP implementation, the consultants met with their teachers to complete the Treatment Evaluation Interview (TEI). During the TEI, the consultants and teachers discussed whether the goals of consultation were met, and determined the most appropriate steps for moving forward with classroom management. Teachers completed the social validity measures immediately following this meeting.

Results

Single-case design visual analysis procedures were used to analyze study data. Although the sophistication of the study design permits the use of randomization tests, strengthening the scientific credibility of findings, these tests were not conducted given the clear null effects. Likewise, calculation of single-case effect sizes was also deemed unwarranted.

Effect of CB-IP on Teacher TI

Adherence

Mr. Albert, Bart, and Dean demonstrated low-to-moderate, somewhat variable CMP adherence during the Standard BC phase, with all three demonstrating adherence levels consistently well below the recommended level of 80% (M level range across teachers = 41.5%–52.4%). In contrast, Mr. Carver demonstrated moderate-to-high, somewhat variable CMP adherence prior to CB-IP (M level = 74.1%). Across teachers, no increasing or decreasing trends were evident during the Standard BC phase. During the CB-IP phase, no teachers displayed an immediate change in level of CMP adherence, variability in adherence data did not improve, and there was significant overlap in data points across the Standard BC and CB-IP phases. Mr. Albert demonstrated a decreasing trend in adherence in addition to a decline in his mean level of adherence. Overall, Mr. Carver’s CMP adherence stayed at the same moderate-to-high level, while Mr. Bart’s and Mr. Dean’s both modestly increased post CB-IP. After receiving in-person IP, a distinct and immediate increase in Mr. Albert’s CMP adherence was displayed such that adherence levels were similar to Mr. Carver’s (moderate-to-high) and no overlapping data with the previous phase were noted (see Figure 1 and Table 1).

Figure 1.

Percentage of CMP steps with adherence ratings of implemented as planned and quality ratings of excellent or good (for steps implemented) across teachers and phases.

Table 1.

Mean CMP Adherence and Quality and Class Disruptions Data Across Teachers and Phases.

Teacher/class	Baseline	Standard behavioral consultation	Computer-based IP	In-person IP
Adherence
Mr. Carver	—	74.1	79.1	—
Mr. Albert	—	52.4	42.2	76.8
Mr. Bart	—	41.5	60.4	—
Mr. Dean	—	47.7	75.8	—
Quality
Mr. Carver	—	86.2	88.8	—
Mr. Albert	—	82.6	78.3	80.5
Mr. Bart	—	71.5	71.8	—
Mr. Dean	—	73.6	78.3	—
Class disruptions
Mr. Carver	1.1	0.7	0.3	—
Mr. Albert	0.8	1.1	0.9	.6
Mr. Bart	3.8	0.9	0.6	—
Mr. Dean	1.8	0.7	0.4	—

Note. Adherence is expressed as a percentage of CMP steps implemented as planned. Quality is expressed as the percentage of CMP steps implemented with excellent or good quality. Class disruptions is expressed as the rate of student disruptions per minute. CMP = classroom management plan; IP = implementation planning.

Quality

During the Standard BC phase, all teachers demonstrated high mean levels of CMP quality; Mr. Bart and Dean demonstrated significant variability in quality. During the CB-IP phase, all teachers maintained high mean levels of CMP quality; Mr. Bart’s quality levels continued to be variable whereas Mr. Dean’s quality levels were less variable. During the in-person IP phase, Mr. Albert maintained high mean levels of CMP quality. Overall, from Standard BC phase to the CB-IP phase, there was little change in mean implementation quality and significant overlap in data (see Figure 1 and Table 1).

Changes in Class Disruptions

During baseline, levels of class disruptions were moderate-to-high and variable in Mr. Carver’s, Bart’s, and Dean’s classrooms (M disruptions per minute all >1), with Mr. Bart’s class displaying substantially higher levels of disruptions compared with the other classrooms (M disruptions per minute = 3.8). In contrast, Mr. Albert’s class displayed relatively low and consistent levels of class disruptions (M disruptions per minute = 0.8). Across the remaining study phases, class disruptions were low-to-moderate and less variable in Mr. Carver’s, Bart’s, and Dean’s classrooms. Notably, a clear and immediate decrease in the level and variability of class disruptions was visible after Mr. Bart received standard consultation. This change was maintained during the CB-IP phase. Mr. Albert’s class demonstrated an immediate increase in level and variability in class disruptions after receipt of standard consultation. This pattern of class disruptions remained constant during the CB-IP phase but declined to baseline levels during the in-person IP phase (see Figure 2 and Table 1).

Figure 2.

Number of student disruptions per minute across sessions.

Teachers Social Validity Ratings

URP-IR ratings indicate teachers found CMPs to be acceptable and understandable (M = 4.8), somewhat feasible (M = 4.3), and requiring many system supports (M = 4.2). CB-IP Rating Profile ratings indicate teachers found CB-IP slightly acceptable (M = 4) and understandable (M = 4.4), not particularly feasible (M = 3.5), and requiring a few system supports (M = 3.4).

Discussion

Alternative schools aim to reduce problem behavior and remediate academic deficits toward returning students to their community schools. Consistently implemented evidence-based classroom management is necessary for achieving this goal; yet, many teachers struggle to manage problem behavior. Research has shown that even when teachers are provided with intervention training, they are not able to successfully implement interventions with adequate TI (e.g., Forman et al., 2013; Noell & Gansle, 2014). Although highly efficacious implementation supports, such as performance feedback, exist to promote TI, these methods are too time or resource intensive to be scaled-up to the vast number of teachers requiring assistance. Thus, the primary purpose of this study was to extend the literature on an efficacious implementation support, consultant-mediated IP, that might lend itself to scale-up by adapting the delivery procedure from consultant-mediated support to self-guided, computer-based completion. Contrary to the stated hypothesis, results of this study found that CB-IP did not result in substantive changes to teachers’ TI and, as an extension, targeted student outcomes.

There are likely three main reasons for these null effects: (a) limitations of the CB-IP delivery format, (b) differences between the study contexts and designs in which CB-IP and IP were tested, and (c) a potential lack of program differentiation. Regarding the delivery format, to date, education research on IP has taken place exclusively within the context of a consultative relationship. This consultant-mediated delivery of IP has resulted in enhanced implementation adherence and quality, followed by corresponding improvements in student behaviors (e.g., Sanetti et al., 2015). It is highly probable that one of the active ingredients of IP is the collaborative interchange between the consultant who has intervention expertise and the teacher who has intervention setting expertise. Thus, removal of the consultant may have been a fatal flaw thwarting treatment effectiveness. There is some research to support this hypothesis. First, there is preliminary evidence for the positive influence of consultant–teacher collaboration on teacher TI (see Long et al., 2016). Second, technology-based methods that have resulted in adequate educator TI, to date, have employed telepractice technologies which maintain the active role of the consultant (e.g., videoconferencing; Neely, Rispoli, Gerow, & Hong, 2016). It is likely that none of the teachers adapted CMP steps to enhance fit in this study, as has been done in previous research, because they lacked the expertise to do so on their own. Thus, the removal of consultant may have been particularly problematic in this alternative school context where the need for plan precision and fit are heightened. Given the more dynamic nature of alternative classrooms and intensity of student needs, an ongoing implementation support may be necessary.

Regarding differences in the study contexts and designs, prior research has examined the effect of IP on teacher TI in regular education classrooms only and using response-guided logic for determining start points in the multiple baseline, as opposed to randomization. Therefore, there were significant differences across studies in (a) the challenges the intervention contexts presented (e.g., the alternative school classrooms had to deal with student mobility/absenteeism, explosive outbursts posing safety risks, and possible countercontrol behaviors) and (b) the researchers ability to adapt to the needs of teachers and students (e.g., the dual-randomization design required strict adherence to the original design procedures). Although response-guided procedures can help to reduce ambiguities in the data pattern and enhance researcher responsiveness to participants, it should be stated that they may also increase the chance of incorrectly concluding that an intervention has an effect (Ferron, Joo, & Levin, 2017). Thus, for early investigations of intervention efficacy, designs that incorporate randomization may be of benefit as they better control for threats to internal validity and reduce the chance of making a Type I error.

Finally, there may have been a lack of sufficient program differentiation between pretraining and either the Standard BC or CB-IP phases in teacher practices. Program differentiation is the extent to which the selected intervention differs from the practices already in place (Dane & Schneider, 1998). Program differentiation may be particularly relevant for Tier 1 interventions as teachers are being asked to adjust their universal practices, not replace them completely. Thus, the larger the shift in teacher practices, the more likely an improvement in student behavior will be observed. In this study, immediately following CMP training, teachers displayed low-moderate to moderate adherence and high levels of quality that sustained. These data suggest teachers likely continued to implement with high quality the classroom management strategies they were using, but did not consistently incorporate many new ones despite having the skills to. A review of teacher adherence to individual CMP steps across phases may support this hypothesis. As an example, in the present study, notable improvements in student behavior were observed only in Mr. Bart’s classroom where there was a meaningful change recorded in his rates of specific praise and praises to reprimand ratio from baseline to Standard BC.

Limitations

As with all research, there were some limitations to this study. First, the Standard BC phase always preceded the CB-IP, potentially creating an ordering effect. However, the ordering of implementation supports was deliberate as this order is most consistent with conventional BC practice. Second, the restricted baseline duration was problematic as it limited the comparison that could be made between the teachers’ and students’ baseline behaviors and their behaviors after CMP training. However, this limitation was unavoidable due to contextual constraints and the primary focus of the study was to examine the phase contrast between the Standard BC and CB-IP phases. A third limitation was the lack of opportunity for in vivo CMP training. Although direct training procedures were used, skill acquisition was checked, and teachers were highly skilled, the possibility of inadequate training cannot be fully ruled out.

Conclusion

Current results highlight the need of conducting effectiveness studies to learn the specific conditions under which an intervention is and is not effective, and can be adapted without compromising its effectiveness. In this study, changing the delivery method of IP appeared to have a disadvantageous effect as the consultant may be an active ingredient of its success. A secondary takeaway may be the need to attend to program differentiation for universal practices and intervention component adherence, beyond overall adherence. To achieve desired intervention outcomes, changes in teacher practices must shift enough to create a meaningful experiential change for students.

Footnotes

Acknowledgements

The authors thank Deirdre Byrne for her important contributions to this research.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R324A100051 to the University of Connecticut (UConn) and the UConn Research Foundation.

ORCID iD

Anna C. J. Long

References

Briesch

A. M.

Chafouleas

S. M.

Neugebauer

S. R.

Riley-Tillman

T. C.

(2013). Assessing influences on intervention implementation: Revision of the Usage Rating Profile-Intervention. Journal of School Psychology, 51, 81–96. doi:10.1016/j.jsp.2012.08.006

Chafouleas

S. M.

Briesch

A. M.

Neugebauer

S. R.

Riley-Tillman

T. C.

(2011). Usage Rating Profile –Intervention (Revised). Storrs: University of Connecticut.

Dane

A. V.

Schneider

B. H.

(1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45.

Epstein

Atkins

Cullinan

Kutash

Weaver

(2008). Reducing behavior problems in the elementary school classroom: A practice guide (NCEE No. 2008-012). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Ferron

J. M.

Joo

S.-H.

Levin

J. R.

(2017). A Monte Carlo evaluation of masked visual analysis in response-guided versus fixed-criteria multiple-baseline designs. Journal of Applied Behavior Analysis, 50, 701–716. doi:10.1002/jaba.410

Forman

S. G.

Shapiro

E. S.

Codding

R. S.

Gonzales

J. E.

Reddy

L. A.

Rosenfield

S. A.

. . . Stoiber

K. C.

(2013). Implementation science and school psychology. School Psychology Quarterly, 28, 77–100. doi:10.1037/spq0000019

Gresham

Watson

T. S.

Skinner

C. H.

(2001). Functional behavioral assessment: Principles, procedures, and future directions. School Psychology Review, 30, 156–172.

Jolivette

Patterson

D. P.

Swoszowski

N. C.

McDaniel

S. C.

Kennedy

Ennis

R. P.

(2014). School-wide positive behavioral interventions and supports in a residential school for students with emotional and behavioral disorders: First years of implementation and maintenance follow-up focus groups. Residential Treatment for Children & Youth, 31, 63–79. doi:10.1080/0886571X.2014.878584

Koelher

M. J.

Levin

J. R.

(1998). Regulated randomization: A potentially sharper analytical tool for the multiple-baseline design. Psychological Methods, 2, 206–217. doi:10.1037/1082-989X.3.2.206

10.

Kratochwill

T. R.

Altschaefl

M. R.

Bice-Urbach

(2014). Best practices in school-based problem-solving consultation: Applications in prevention and intervention systems. In Harrison

P. L.

Thomas

(Eds.), Best practices in school psychology: Data-based and collaborative decision making (pp. 461–482). Bethesda, MD: National Association of School Psychologists.

11.

Kratochwill

T. R.

Bergan

J. R.

(1990). Behavioral consultation in applied settings: An individual guide. New York, NY: Plenum Press.

12.

Kratochwill

T. R.

Hitchcock

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2010). Single-case designs technical documentation. Retrieved from https://ies.ed.gov/ncee/wwc/Document/229

13.

Kratochwill

T. R.

Levin

J. R.

(2010). Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods, 15, 124–144.

14.

Lange

C. M.

Sletten

S. J.

(2002). Alternative education: A brief history and research synthesis (Cooperative Agreement No. H159K70002). Alexandria, VA: National Association of State Directors of Special Education.

15.

Lehr

C. A.

Tan

C. S.

Ysseldyke

(2009). Alternative schools: A synthesis of state-level policy and research. Remedial and Special Education, 30, 19–32.

16.

Long

A. C. J.

Sanetti

L. M. H.

Collier-Meek

M. A.

Gallucci

Altschaefl

Kratochwill

T. R.

(2016). An exploratory investigation of teachers’ intervention planning and perceived implementation barriers. Journal of School Psychology, 55, 1–26. doi:10.1016/j.jsp.2015.12.002

17.

McKerchar

P. M.

Thompson

R. H.

(2004). A descriptive analysis of potential reinforcement contingencies in the preschool classroom. Journal of Applied Behavior Analysis, 37, 431–444.

18.

Neely

Rispoli

Gerow

Hong

E. R.

(2016). Preparing interventionist via telepractice in incidental teaching for children with autism. Journal of Behavioral Education, 25, 393–416. doi:10.1007/s10864-016-9250-7

19.

Noell

G. H.

Gansle

K. A.

(2014). Research examining the relationships among consultation procedures, treatment integrity, and outcomes. In Erchul

W. P.

Sheridan

S. M.

(Eds.), Handbook of research in school consultation: Empirical foundations for the field (2nd ed., pp. 386–408). New York, NY: Routledge.

20.

Reinke

W. M.

Herman

K. C.

Sprick

(2011). Motivational interviewing for effective classroom management: The classroom check-up. New York, NY: Guilford Press.

21.

Sable

Plotts

Mitchell

(2010). Characteristics of the 100 largest public elementary and secondary school districts in the United States: 2008–09 (NCES 2011–301). Washington, DC: U.S. Department of Education, National Center for Education Statistics.

22.

Sanetti

L. M. H.

Collier-Meek

M. A.

Long

A. C. J.

Byron

J. R.

Kratochwill

T. R.

(2015). Increasing teacher treatment integrity of behavior support plans through consultation and implementation planning. Journal of School Psychology, 53, 209–229. doi:10.1016/j.jsp.2015.03.002

23.

Sanetti

L. M. H.

Collier-Meek

M. A.

Long

A. C. J.

Kim

Kratochwill

T. R.

(2014). Using implementation planning to increase teachers’ adherence and quality to behavior support plans. Psychology in the Schools, 51, 879–895. doi:10.1002/pits.21787

24.

Sanetti

L. M. H.

Kratochwill

T. R.

Long

A. C. J.

(2013). Applying adult behavior change theory to support mediator-based intervention implementation. School Psychology Quarterly, 28, 47–62. doi:10.1037/spq0000007

25.

Schwarzer

(2008). Modeling health behavior change: How to predict and modify the adoption and maintenance of health behaviors. Applied Psychology: An International Review, 57, 1–29. doi:10.1111/j.1464-0597.2007.00325.x

26.

Scott

T. M.

Nelson

C. M.

Liaupsin

C. J.

Jolivette

Christle

C. A.

Riney

(2002). Addressing the needs of at-risk and adjudicated youth through positive behavior support: Effective prevention practices. Education and Treatment of Children, 25, 532–551.

27.

Simonsen

Britton

Young

(2010). School-wide positive behavior support in an alternative school setting: A case study. Journal of Positive Behavior Interventions, 12, 180–191.

28.

Simonsen

Fairbanks

Briesch

Myers

Sugai

(2008). A review of evidence-based practices in classroom management: Considerations for research to practice. Education & Treatment of Children, 31, 351–380.

29.

Simonsen

Freeman

Goodman

Mitchell

Swain-Bradway

Flannery

. . . Putman

(2015). Supporting and responding to behavior: Evidence-based classroom strategies for teacher. OSEP Technical Assistance Brief. Retrieved from www.pbis.org/common/cms/files/pbisresources/Supporting%20and%20Responding%20to%20Behavior.pdf

30.

Simonsen

Myers

DeLuca

(2010). Teaching teachers to use prompts, opportunities to respond, and specific praise. Teacher Education and Special Education, 33, 300–318. doi:10.1177/0888406409359905

31.

Vanderheyden

A. M.

Witt

J. C.

Gatti

(2001). Descriptive assessment method to reduce overall disruptive behavior in a preschool classroom. School Psychology Review, 30, 548–567.