Abstract
In areas of emerging research, such as supporting teachers’ classroom management, replication of research is critical to ensuring that recommendations for the field are based on sound science and appropriate for the contexts to which they are being applied. This article describes a replication of research on efficient professional development supports for teachers’ classroom managements in a new context: high school classrooms. Data did not support a functional relation between teachers’ use of specific praise and the targeted professional development and self-management in the high school setting. Results of this study highlight the critical importance of replication in education research. Based on our findings and our experience conducting this study, we suggest several possible adaptations may be necessary for successful replication at the high school level.
Keywords
Replication is essential to the practice of science, providing strength to an argument that experimental results are not due solely to chance (Cooper, Heron, & Heward, 2007; Schmidt, 2009). Therefore, replication studies serve many important functions, from controlling for coincidental results to generalizing results to different settings or populations (Schmidt, 2009). In other words, replication studies not only provide necessary validation that an intervention will work again but also provide evidence that an intervention may work in different contexts or with different samples than those with which they have been previously tested. Despite the importance of replication studies, this type of research is rarely conducted in the field of special education. Makel et al. (2016) found that only 0.5% of all special education articles specifically define themselves as a replication study. Cook, Collins, Cook, and Cook (2016) examined the prevalence of replication in special education intervention studies using broader definitions and suggested that 31% of published studies both built upon a specific previously published study and compared the results of the new study with the previously published results. However, they noted that no direct replications were identified in their review. Especially in areas of emerging research, like supporting teachers’ classroom management, replication is critical to ensuring that recommendations for the field are based on sound science and appropriate for the contexts to which they are being applied. This article describes a replication of research on efficient professional development supports for teachers’ classroom managements in a new context: high school classrooms.
Effective Classroom Management Practices
Simonsen, Fairbanks, Briesch, Myers, and Sugai (2008) identified five critical features of research supported classroom management practice: (a) maximize structure; (b) post, teach, review, monitor, and reinforce expectations; (c) actively engage students in observable ways; (d) use a continuum of strategies for providing contingent reinforcement for appropriate behaviors; and (e) use a continuum of strategies to decrease the intensity and frequency of inappropriate behaviors. By proactively establishing a classroom environment that supports appropriate academic and social behaviors and consistently recognizing and reinforcing those behaviors, teachers may help to reduce inappropriate student behavior (Partin, Robertson, Maggin, Oliver, & Wehby, 2010). Unfortunately, much of this research has been conducted at the elementary and middle school level; therefore, it is critical to carefully examine the current use and effectiveness of these strategies at the high school level.
Specific Praise in High Schools
The importance of specific praise and the effects of supporting teachers’ implementation of this skill have yet to be fully established in a high school setting. Specific, contingent praise is a positive statement directed at a student or group of students that directly follows the occurrence of a desired behavior and identifies for the student/s what she or he did well (Partin et al., 2010; Simonsen et al., 2008). Specific praise from teachers is associated with increases in students’ prosocial behavior and time on task (Chalk & Bizo, 2004; Simonsen et al., 2008; Sutherland, Wehby, & Copeland, 2000).
Although researchers have identified multiple benefits associated with specific praise, observed rates of teachers’ use of the strategy are low. Pas, Cash, O’Brennan, Debnam, and Bradshaw (2015) found the average rate of high school teacher approval was 0.12 “approval” interactions per minute (e.g., providing verbal, nonverbal, or tangible items to acknowledge behavior). Hirn and Scott (2014) reported an average of 0.03 overall positive statements per minute in observations of high school teachers. Scott, Alter, and Hirn (2011) reported an average rate of 0.06 overall positive statements per minute in a sample that included both elementary and high school classrooms. In ninth-grade inclusive classrooms, McKenna, Muething, Flower, Bryant, and Bryant (2015) observed an average of 0.24 specific praise statements per minute.
Opinions about the use of specific praise at the high school level are mixed. Historically, teacher praise of high school students has been described as not only ineffective but potentially even punishing (e.g., Ward, 1976). However, more recent studies have found that high school students value teacher praise and perceive it to be an important element in their academic and social behaviors (Elwell & Tiberio, 1994; Fefer, DeMagistris, & Shuttleton, 2016). Some evidence suggests that adolescents may have a preference for private praise (Elwell & Tiberio, 1994; Fefer et al., 2016) and process, effort, or product-related praise (Burnett & Mandel, 2010; Corpus & Lepper, 2007). However, both private and public praise are linked with improved student behavior (Blaze, Olmi, Mercer, Dufrene, & Tingstom, 2014) and motivation (Deci, Koestner, & Ryan, 2001) in high schools. This emerging evidence suggests that praise is likely to be both socially acceptable and effective as a classroom management practice at the secondary level; however, we need to know more about precisely when, where, and under what conditions teachers can use praise effectively at the high school level.
The High School Context
A number of factors make the high school setting unique, including a focus on higher level academics, the developmental age of the students, physical size of the campus, a large student population, compartmentalized operations, and typically a zero-tolerance approach to discipline (Bohanon, Fenning, Borgmeier, Flannery, & Malloy, 2009; Flannery, Frank, Kato, Doren, & Fenning, 2013; Skiba & Rausch, 2006). The impact of these factors on high school teachers’ use of empirically supported classroom management practices and the development of professional development supports to increase their use must be examined empirically. Replication of elementary school studies that demonstrate improvement in teachers’ use of classroom management practices would help to establish whether these same practices would be effective for teachers in the high school setting. Indeed, regardless of results, such studies would provide valuable information about whether established professional development supports could translate to the high school level.
Supporting High School Teachers’ Use of Classroom Management Strategies
Although researchers have identified specific evidence-based classroom management practices (Oliver, Wehby, & Reschley, 2011; Simonsen et al., 2008) and are developing professional development supports to improve teachers’ use of these practices at the elementary school level (e.g., Simonsen et al., 2014; Simonsen et al., 2016; Sutherland & Wehby, 2001; Wills, Iwaszuk, Kamps, & Shumate, 2014), less is known about strategies for improving teachers’ use of effective practices or typical rates of their use in high schools. High school teachers cite behavioral issues as a major challenge that they are underprepared to address (Boyd et al., 2011; Chesley & Jordan, 2012; Feng, 2006; Henke, Zahn, & Carroll, 2001; Ingersoll, 2001; Ingersoll, Merril, & May, 2012; Johnson & Birkeland, 2003; Kukla-Acevedo, 2009; Lane, Wehby, & Barton-Arwood, 2005; Luekens, Lyter, Fox, & Changler, 2004; Stough, 2006; Torres, 2012; Zabel & Zabel, 2002). However, research-based classroom management strategies are not consistently taught in high school teacher preparation programs (Freeman, Simonsen, Briere, & MacSuga-Gage, 2014), suggesting a critical need for effective professional development in this area.
In a comprehensive literature review of professional development in classroom management, researchers concluded that multi-component professional development including didactic training, coaching, and performance feedback leads to observed changes in teachers’ use of classroom management practices (Freeman et al., under review). Although effective, such multi-component professional development approaches are resource intensive and may not be feasible for most school settings (Martens, Hiralall, & Bradley, 1997; Myers, Simonsen, & Sugai, 2011; Sanetti, Fallon, & Collier-Meek, 2013; Simonsen, Myers, & DeLuca, 2010; Solomon, Klein, & Politylo, 2012). Given the scale and complexities of high school settings, identifying efficient and effective supports for teachers is critical.
Using Self-Management to Improve Professional Development
Initial research suggests that professional development approaches that emphasize teacher self-management strategies may be an effective and less resource-intensive approach to improving classroom teachers’ use of effective practices (Briere, Simonsen, Sugai, & Myers, 2015; Oliver, Wehby, & Nelson, 2015; Simonsen et al., 2016; Simonsen et al., 2014; Sutherland & Wehby, 2001). Self-management is the personal use of behavior change tactics (e.g., manipulating antecedents and consequences) to yield a desired change in one’s own behavior (Cooper, Heron, & Heward, 2007). For example, Simonsen et al. (2014, Simonsen et al., 2016) investigated the effects of a targeted professional development approach to increase teachers’ specific praise rates, in which teachers received a brief, explicit skill-based training on specific praise and implemented a self-management package. As part of the self-management package, teachers set a goal for rate of skill performance, self-monitored skill use, self-evaluated their rate of skill use relative to their goal, and self-reinforced when their goal was met. Initial results were promising: Teachers consistently increased their specific praise rate while self-managing. This targeted professional development package has since been tested for use with small groups of teachers and with other classroom management skills (i.e., opportunities to respond [OTRs], prompts; Simonsen, Freeman, Dooley, et al., in preparation; Simonsen, Freeman, Meyers, et al., in preparation). However, this initial research was conducted in elementary or middle school settings.
The primary purpose of this study is to examine the impact of targeted professional development plus self-management on teachers’ use of specific praise in the high school setting, replicating research on a promising intervention previously studied only at lower grade levels. A secondary purpose is to examine current rates of effective classroom management practices at the high school level. Specifically, we address the following research questions:
Research Question 1, the primary research question, guided the design of the study and is addressed experimentally. Research Questions 2 and 3 are supplemental and are answered descriptively.
Method
Participants and Setting
This study took place at a suburban high school in the northeastern United States. Of the approximately 2,250 students enrolled in the school, 8.2% were eligible for free/reduced-price meals, 7% came from homes where English was not the primary language, and 10% were identified as having a disability. Of students at the school, 80% were White, 0.2% were American Indian, 8.4% were Asian American, 3.2% were Black, 6.3% were Hispanic, and 1.9% were two or more races.
After listening to a brief overview of the study given by the lead researcher at an all-staff meeting, interested teachers completed a handout indicating their interest and providing their contact information. The research team contacted all teachers who indicated interest in participation in random order until five teachers had responded, completed consent forms, and were selected for the study. Although initially five teachers enrolled, only four teachers completed the study; one teacher dropped out during the baseline phase when a student teacher took on most direct instruction in her classroom. We removed her data from all analysis. Our institutional review board approved all recruitment and study procedures to ensure compliance with ethical and legal obligations.
Teacher 1
Teacher 1 was a female science teacher with certification in general science and biology (seventh–12th grades). We observed Teacher 1 teaching a year long freshman course for 16 students who have experienced poor academic success in the past. She reported that almost all of the students in this class had an Individualized Educational Plan (IEP), 504 plan, or student intervention plan. Teacher 1 had 12 years of teaching experience, but did not recall any prior training in classroom management. Her highest degree earned was an MBA.
Teacher 2
Teacher 2 taught an art course for students in ninth to 12th grades. She did not report her years of teaching experience, highest degree earned, certification, prior training in classroom management, or demographic information. We observed Teacher 2 teach an art class of about 20 students.
Teacher 3
Teacher 3 was a male teacher who taught health and physical education (PE) to students in ninth to 12th grades. We observed him teach health during the first semester and PE during the second semester. Both classes included about 25 ninth-grade students. He had 7 years of teaching experience, a master’s degree in educational leadership, and certification in K–12 health and PE. He stated that he had no prior training in classroom management.
Teacher 4
Teacher 4 was a male music teacher who taught a chorus elective for students in ninth to 12th grades that included students receiving special education. He had a doctoral degree, certification in K–12 music, and 13 years of teaching experience, but no formal previous training in classroom management.
Dependent Measures
We collected data on teacher rates of classroom management practices (i.e., specific praise, OTRs, correctives, and social prompts), student behaviors, fidelity of implementation, and social validity.
Specific praise rates
The primary dependent variable was teachers’ use of specific praise statements. That is, we calculated the rate with which teachers delivered specific praise statements to individuals or groups of students during teacher-directed instruction. Trained doctoral student observers conducted daily direct observations. Observations occurred during the same time period each day and lasted for 15 min. Observation times were scheduled to align with regular teacher direct instruction. Observers recorded the frequency of specific praise statements and divided by the number of minutes observed to obtain a daily rate per minute. The same procedures were used to record rates of general praise, OTRs, correctives, and social prompts. See Table 1 for operational definitions of each of these skills.
Observed Classroom Management Skills and Their Definitions.
Source. Definitions taken from Simonsen, Fairbanks, Briesch, Myers, and Sugai (2008).
Observer training
Observers were trained to collect data across a series of training activities. Training included one meeting to introduce the tool and operational definitions of the behaviors included on the form, several practice sessions using the form and the definitions with video recordings of direct instruction, and a minimum of two sessions of in-vivo training (i.e., observing educators and children in the classroom with the form). In-vivo training continued until the observers reach the predetermined criterion (i.e., 85%) of interrater reliability.
Interobserver agreement (IOA)
Throughout the study, IOA observations were conducted for 24.79% of observations. These observations were spread across study phases and participants to prevent observer drift. We calculated IOA using the mean count-per-interval method (Cooper et al., 2007). IOA for specific praise rates remained high throughout the study with an average of 99.7% in baseline (range, 96.7%–100%) and 98.2% in intervention (range, 92.4%–100%). In addition, we monitored IOA across teacher skills. IOA averaged 99.7% (range, 96.7%–100%) across all teacher skills and averaged 99.8% (range, 97.8%–100%) for OTRs.
Fidelity of implementation
To ensure fidelity of implementation of training procedures, we recorded adherence to scripted training sequences using a checklist based on the training script. Scripted training sequences were delivered with 100% adherence. We also monitored teacher adherence and accuracy with self-monitoring and self-evaluation procedures. Observers recorded teacher adherence as fully implementing (e.g., holding and actively using the counter), partially implementing (e.g., holding but not actively using or not using the whole observation), or not implementing (e.g., not using the counter at all) the intervention. Teacher 1 fully implemented the intervention during 93.94%, partially implemented 3.03%, and did not implement 3.03% of observations. Teacher 2 demonstrated the lowest levels of adherence to the intervention, fully implementing only 22.22%, partially implementing 20%, and not implementing 57.78% of observations. Teacher 3 fully implemented 81.82% and did not implement 18.18% of observations. Teacher 4 was rated as fully implementing during 83.33% of observations, partially implementing 8.33% of observations, and not implementing 8.33% of observations. Observers recorded the teacher’s accuracy by noting and comparing the number of specific praise statements recorded by the teacher using the golf counter with the number recorded by observers. Overall, teacher accuracy was as follows: Teacher 1 = 40.7%, Teacher 2 = 45.2%, Teacher 3 = 58.3%, Teacher 4 = 73.8%. We monitored adherence to the student behavior evaluation component by asking participating teachers to share their completed rating form with the research team during the intervention phase. We calculated the rate of adherence by dividing the number of opportunities (days in intervention) by the number of submissions. The rate of adherence for Teacher 1 was 86.6%; Teacher 2, 80%; Teacher 3, 100%; and Teacher 5, 88.9%. Finally, we monitored adherence to the self-evaluation and spreadsheet use by asking teachers to submit their completed spreadsheets to the research team. Only Teacher 2 submitted a completed spreadsheet.
Social validity
At the conclusion of the study, each teacher was asked to complete a Targeted Professional Development Acceptability Questionnaire Intervention Rating Profile–15 (TPD-AQ), adapted from Martens, Witt, Elliot, and Darveaux (1985). The original 15-item Intervention Rating Profile (IRP-15) has a one-factor structure, which has been called “general acceptability,” with high internal consistency (Cronbach’s α = .98; Martens et al., 1985). Like the IRP-15, the TPD-AQ prompts teachers to rate 15 items related to the acceptability of targeted PD on a scale from 1 (strongly disagree) to 6 (strongly agree). The TPD-AQ was used to assess the social validity of this teacher-focused training and intervention; the psychometric properties of the TDP-AQ have not been established.
Research Design and Procedures
We used a single-case multiple baseline design across participants (Horner et al., 2005; Kratochwill et al., 2010) with two phases: baseline and intervention. In addition, Teacher 1 entered a third, more intensive intervention phase after several weeks of failing to respond to the initial intervention. We randomly selected participant order for entering the intervention condition and staggered the intervention systematically across participants to demonstrate experimental control. A follow-up or maintenance phase was not used as there was little teacher response to the intervention and the study approached the end of the school year.
Baseline
During the baseline phase, a team of trained doctoral students completed structured observations, using the procedures outlined above. No changes were made to the teachers’ typical routines and procedures. The baseline phase lasted until all teachers had stable data for specific praise. Observed rates of teacher skills use are summarized in Table 2 and discussed further in the “Results” section below.
Rate per Minute of Teacher Use of Classroom Management Practices.
Intervention
Following baseline, we introduced the targeted professional development and self-management intervention (TPD+SM). All intervention components were consistent with previous studies of TPD+SM; however, we modified the examples and nonexamples provided during the training to ensure a contextual fit in high school classrooms. In addition, to ensure that participating teachers made connections between their use of specific praise and student behavior and to promote maintenance, we added the teacher reflection on student behavior component to the intervention phase.
We determined the order in which teachers entered the intervention phase through random selection. After data indicated a clear pattern and data for all teachers still in baseline remained stable, we introduced the intervention to the next randomly selected teacher. We repeated this procedure until all teachers were in the intervention phase. To ensure that participants were not exposed to the intervention before they were phased in, we asked teachers not to share information about study procedures with others.
When each teacher entered the TPD+SM intervention phase, they received explicit training in how to reinforce appropriate student behavior with specific, contingent praise. The training consisted of (a) defining specific praise, (b) rationale for using specific praise (including a rationale for use at the secondary level), (c) examples of specific praise use (including examples consistent with secondary classroom use), and (d) critical features of effective specific praise. This direct instruction was followed by an application activity in which the teacher developed and wrote down three examples of specific praise statements they might use in their classroom. Then teachers received training in the self-management strategy. During this phase of the training, the researcher defined self-management and explained that for the purposes of this study, the participants would (a) arrange their environments to increase the likelihood that they would use praise (e.g., set goals and give themselves reminders), (b) self-monitor (i.e., count) their use of specific praise during a 15-min period of instruction, (c) graph their specific praise rates daily and self-evaluate whether they met their daily goal, (d) evaluate student behavior, and (e) self-reinforce. The lead researcher delivered all trainings following a detailed training script, with a doctoral student checking fidelity of training. Training took place during one session per teacher at a time that was convenient for the participating teacher (e.g., during their planning period) and lasted approximately 25 to 35 min. Teachers received a copy of the training script and completed a brief self-management plan activity (e.g., developing their goal, identifying reinforcers) to promote implementation fidelity.
During the intervention phase, teachers monitored their use of specific praise use using a golf counter while teaching. Teachers’ use of a golf counter for self-monitoring was identified as the preferred method for self-monitoring in an alternating treatment design (Simonsen, MacSuga, Fallon, & Sugai, 2013) and has been used successfully by teachers across previous studies (Simonsen, Freeman, Dooley, et al., in preparation; Simonsen, Freeman, Meyers, et al., in preparation; Briere et al., 2015; Simonsen et al., 2014; Simonsen et al., 2016). Observers continued to monitor teacher use of specific praise and the other classroom management behaviors using the procedures outlined above. In addition, observers collected the specific praise count from the teacher’s golf counter (i.e., the number of times the teacher believed they had used specific praise) at the end of each 15-min observation. Teachers tracked their own progress through use of a researcher-designed excel spreadsheet. To generate a graph of progress, teachers entered the count from their golf counter into the spreadsheet. To evaluate student behavior, teachers completed a modified two item (i.e., academically engaged, disruptive) Direct Behavior Rating (DBR) for the whole class at the end of the instructional period during which the observation took place (modified from Chafouleas, Riley-Tillman, Christ, & Sugai, 2009). Finally, the lead researcher sent a weekly email to each teacher in intervention, reminding him or her of their goal and the definition of specific praise. At the study’s conclusion, teachers completed demographics and social validity questionnaires.
Intervention 2 (IV2)
After all participants entered the initial intervention phase, we implemented a second more intensive intervention phase for Teacher 1, who had not responded adequately to the initial intervention. In addition to the procedures described above, during the IV2 phase, we sent a daily email at the end of the school day to Teacher 1 including specific performance feedback and coaching. These emails included her daily specific praise rate as recorded by observers and one to three examples of specific praise statements that she made during that day’s observation. The email also included one to two suggestions for specific praise statements that she could have used during the observation period (e.g., general praise statements that could be made specific, positive student behaviors that were observed but not acknowledged).
Analysis
We used visual analysis to examine the specific praise data collected through direct observation and to determine if there was a functional relation between the intervention and dependent variable. In keeping with traditional single-case design standards (Kratochwill et al., 2013), visual analysis included (a) examining the level, trend, and variability, within each phase; (b) documenting the immediacy of effect; (c) examining consistency of the data between phases; and (d) examining external factors and anomalies. We also calculated individual and weighted Tau-U, which provides a standardized measure of intervention effectiveness within and across participants that accounts for both phase nonoverlap and trend (Vannest, Parker, & Gonen, 2011), using the Single Case Research™ web-based calculator (Parker & Vannest, 2009). Descriptive statistics were calculated for the other classroom management strategies on data collected during the extended baseline phase (see Table 2).
Results
Research Question 1: Functional Relation Between Intervention and Specific Praise
Overall, rates per minute of specific praise used by teachers were low at baseline (M = 0.23, range, 0.00–3.07), but increased during intervention (M = 0.43, range, 0.00–3.07). The following sections summarize results for each teacher’s data related to specific praise. Figure 1 illustrates teachers’ use of specific praise across study phase. The weighted Tau-U for the TPD+SM intervention was small (.517) and statistically significant (p < .01).
Teachers’ use of specific praise statements.
Teacher 1
During baseline, Teacher 1 demonstrated the lowest levels of specific praise (M = 0.05, range, 0.00–0.31), most often providing no specific praise statements for the duration of the observation. Following intervention, there was no meaningful change in level or trend (M = 0.14, range, 0.00–0.67). However, a slight increase in variability is noted. Following the more intensive performance feedback and coaching intervention, there was an initial increasing trend; however, this did not sustain (M = 0.22, range, 0.00–0.67). Unfortunately, the implementation of this second intervention phase coincided with the end of the school year, so further data collection was not possible. Tau-U for Teacher 1 was .491and statistically significant (p < .001).
Teacher 2
Teacher 2 demonstrated low levels of specific praise at baseline with low variability or trend (M = 0.12, range, 0–0.47). A very slight increase in level and variability is noted following intervention with no change in trend; however, her mean number of specific praise statements per minute increased (M = 0.21, range, 0.00–1.00). Tau-U for Teacher 2 was .254 and statistically significant (p < .041).
Teacher 3
Baseline data collection began at the end of one semester and continued into the next. This did not impact most teachers, who continued to teach the same class with the same students. However, Teacher 3 taught a very different type of course when the semester changed and this is reflected in the data. In the second semester, Teacher 3’s baseline use of specific praise increased dramatically. Compared with the other participants, he had the highest rates of specific praise statements in baseline (M = 0.52, range, 0−3.07) and after intervention (M = 1.39, range, 0–3.07). However, his specific praise rates were highly variable at baseline and remained so following intervention. Variability was not notably different across phases. Tau-U for Teacher 3 was calculated using only baseline data after the semester break and was .560 and statistically significant (p < .001).
Teacher 4
Teacher 4 demonstrated modest effects of the intervention. He exhibited low and stable rates of specific praise (M = 0.18, range, 0.00–0.87) at baseline. Immediately following the intervention, Teacher 4’s rate per minute of specific praise statements increased (M = 1.0, range, 0.60–1.42) and stayed above baseline rates for the remainder of the study. There was a slight decreasing trend at the end of the study and school year. No changes in variability are noted across phases. Tau-U for Teacher 4 was .987 and statistically significant (p < .001).
Research Question 2: Social Validity
Social validity data were obtained from only three participants as the study concluded at the end of the school year and we were not able to schedule follow-up meetings with all teachers. All participants were given a copy of the TPD-AQ and asked to complete it and return it to the lead researcher via mail. The overall teacher rating on the intervention is 4.3, with the lowest rating (2.5) occurring on Item 10 indicating that the targeted professional development presented in this study was different from past trainings these teachers had participated in (see Table 3).
Social Validity Ratings on the Targeted Professional Development Acceptability Questionnaire.
Ratings on a scale from 1 (strongly disagree) to 6 (strongly agree).
Research Question 3: Relative Levels of Classroom Skills
Throughout the extended baseline in this study, we collected data on other classroom management strategies (e.g., OTR, correctives, prompts) to gather more information about the relative use of these strategies at the high school level. We summarize teachers’ use of these classroom management techniques in Table 2. Prior to intervention, the average observed specific praise rate per minute was .23 (range, 0–3.07) and the average general praise rate was .32 (range, 0–2.0). Of all the strategies, teachers most frequently used OTR (M = 1.15, range, 0–6.73). Across teachers, social prompts were the least frequently used classroom management strategy (M = 0.01, range, 0–0.20). Corrective statements were observed at an average rate per minute of .20 (range, 0–0.87) for specific corrective statements and .02 (range, 0–2.15) for general corrective statements.
Discussion
In this study, we sought to replicate research examining the use of a TPD+SM strategy to help teachers increase their use of specific praise in the high school setting. We also wanted to learn more about the typical rates of empirically supported classroom management strategies by high school teachers.
Research Question 1: Functional Relation Between Intervention and Specific Praise
Data did not support a functional relation between teachers’ use of specific praise and the TPD+SM in the high school setting. Although minor increases in the level of teacher use of specific praise were noted for three teachers, these changes were not markedly different from baseline. Given the repeated success of this model at the elementary level, these results were unexpected.
Results from this study provide an important caution regarding the application of evidence-based practices in untested settings without further study. We, as a field, often assume that empirically supported practices will be equally effective across settings. This is particularly the case in under-studied areas, such as professional development for classroom management in high school settings, where we may rely on using empirically supported practices that have only been tested in elementary and middle school settings. Differences in high school teacher training, student developmental age, and school characteristics (e.g., enrollment, organization) may impact the implementation or effectiveness of interventions at the high school level. Results of this study are a reminder that caution is needed when generalizing across settings. More research is needed to understand (a) why this attempt to replicate was not successful, as there may be unique variables in this study (see “Limitations” section), and (b) if and how this practice should be altered to be more successful in the high school setting.
Based on our experiences with this study, there appear to be several aspects of the high school context that impacted results of this replication. Secondary teachers receive less preservice training in classroom management practices than their elementary school counterparts (Freeman et al., 2014) and then enter a school environment that is typically more focused on academics and in which specific praise may be perceived as an ineffective strategy (Bohanon et al., 2009; Flannery et al., 2013; Skiba & Rausch, 2006). This may result in a need for more intensive professional development and training. For example, in this study, one teacher was initially confused about the idea of “catching” students who were on task rather than looking for students who were off task and indicated that this was very different from past practice and training. The training provided with this intervention may need to be intensified to meet the needs of high school teachers.
Second, unlike most elementary schools, many high schools operate on a rotating block schedule and students may change their course enrollment each semester. Given longer academic blocks and attention spans of older students, teachers may structure academic instruction differently than in elementary settings. A teacher may engage in direct instruction and modeling one day and then give the students guided practice or lab time during the next scheduled block which may be a day or two later. This variability in the schedule may make it hard to establish stable baseline data and may result in changes in dependent variables, which are not due to intervention, as experienced in this study. Classroom observations for future research may need to be longer or on a more varied schedule to fully capture the range and variability of teacher skill use across instructional approaches.
Research Question 2: Social Validity
Despite the fact that teachers may have had little prior experience using behavioral strategies and this intervention did not produce desired effects across teachers, participants in this study indicated that they felt the intervention package was generally acceptable and felt it would help them increase their use of specific praise. The intervention was rated as different from training the teachers had previously experienced. Future attempts at replication or adaptation of this intervention should consider the fact that the targeted professional development including self-management intervention were well received.
Research Question 3: Relative Levels of Classroom Skills
Little is known about the typical rates of empirically supported classroom management strategies at the high school level. This study contributes to the literature by providing information about baseline rates of use of several classroom management strategies. More research is needed to determine if the rates identified in this study are typical of teacher practices in high school settings. Although we collected data on a wider range of skills, we elaborate on teachers’ use of OTRs and praise as these skills were the most frequently observed proactive skills in this study. Social prompts are the other proactive skills that we collected data on; however, they were almost never observed in this study.
OTR
Research has demonstrated that increased teacher use of OTRs is associated with decreases in disruptive student behavior, increases in on-task behavior, improvements in academic engagement, and increases in the number of correct responses (Carnine, 1976; Sutherland, Alder, & Gunter, 2003; Sutherland & Wehby, 2001; West & Sloane, 1986). In particular, for students with emotional or behavior disorders, increased rates of OTRs has been found to positively impact reading and math performance and task engagement, and to decrease inappropriate and disruptive behaviors (Sutherland & Wehby, 2001). The Council for Exceptional Children (CEC; 1987) suggests that OTRs take place at a rate of 4 to 6 responses per minute with 80% accuracy in student responses when students with mild disabilities are learning new material, and 8 to 12 responses per minute with 90% accuracy in student responses for independent practice. Despite CEC’s recommendation and evidence that high rates of OTRs are associated with positive student outcomes, teachers’ use of OTRs appear to be lower than recommended. It is important to note that these recommendations are intended to guide instruction for students with disabilities and this study was conducted in general education classrooms. In this study, teachers provided an average of 1.15 (range, 0.00–6.73) OTRs per minute.
Specific praise
Previous research has identified low rates of teachers’ specific praise (Hirn & Scott, 2014; McKenna et al., 2015; Pas, Cash, O’Brennan, Debnam, & Bradshaw, 2015; Scott, Alter, & Hirn, 2011). Although the average baseline rate in this study was low (M = 0.23, range, 0.00–3.07), it was higher than some averages reported in prior research (M = 0.12, Pas et al., 2015; M = 0.03, Hirn & Scott, 2014; M = .06, Scott et al., 2011) and consistent with rates reported for ninth-grade inclusive classrooms (M = 0.24; McKenna et al., 2015).
More research is needed to understand if rates identified in this study are typical of and beneficial in the high school setting as well as to clearly establish best practices for both students with and without disabilities. Future studies should examine potential barriers to teachers’ use of these skills and the implementation of an intervention designed to increase their use such as targeted professional development with self-management.
Limitations
Several limitations to this study should be considered when interpreting results. First, the lack of diversity (i.e., both ethnicity and socioeconomic [SES] status) of the setting in which the study was implemented may limit what we can learn about contexts in which this intervention may be successfully implemented.
Second, there was no maintenance phase implemented in this study. The fact that teachers did not respond meaningfully to the intervention made a maintenance phase, in which we would withdraw the intervention to determine if behavior rates maintained in the absence of intervention, irrelevant. In addition, the end of the school year made further intervention manipulations impossible. Therefore, a functional relation could not be tested for the more intensive performance feedback and coaching intervention.
Third, because targeted professional development has been conceptualized as support that could be effective for all teachers in past studies, we did not screen teachers prior to admitting them to the study. This resulted in a wide range of teacher comfort with and use of classroom management strategies potentially impacting study results.
Fourth, the changes in content taught before and after the semester break directly impacted our ability to collect continuous data across that time. We waited to introduce the intervention until data had stabilized in the second semester. In general, we are not able to draw comparisons between teacher behaviors before and after this change as course topics and student groups varied. In addition, three of the four participants in this study taught elective course content rather than traditional core academic content. Previous studies conducted at the elementary level involving the targeted PD intervention included observations of instruction in both academic and elective classes with no consistent differences in teacher outcomes across content areas. It is possible, however, that at the high school level instructional differences across subject areas increase and may have impacted outcomes in this study.
Finally, the logistics of data collection in large classrooms was also a limitation. To be minimally disruptive, observers generally tried to stay out of the way of the teacher and student activities. At times, this made it challenging to hear what teachers were saying to students in noisy, busy classrooms in which students were engaged in independent or group work and when teachers spoke to students in small groups or individually. It is possible that some specific praise statements were not heard and therefore not recorded because of this logistical limitation. In future studies, researchers may consider using a teacher microphone to record or amplify teacher interactions with students.
Future Directions
Results of this study highlight the critical importance of replication in education research. Specifically, in this case, they stress the importance of testing established methods for training teachers to use classroom management strategies in a new setting: high school. Based on our findings and our experience conducting this study, we suggest several possible adaptations may be necessary for successful replication at the high school level.
First, we suggest accounting for teachers’ previous experience and training with classroom management practices and instructional design. In this study, participating teachers had a wide range of backgrounds, training, and professional development experiences. Given this variability and what we know about the lack of training in classroom management overall for secondary teachers, some teachers may have benefited from a more intensive or comprehensive training and self-management package to successfully increase their use of specific praise. Future studies may consider the use of a more intensive performance feedback and coaching phase to establish initial skill levels prior to asking teachers to self-manage skill use (similar to the IV2 phase attempted here). Providing intensive performance feedback for a short period of time prior to asking teachers to self-manage would ensure that teachers had a clear understanding of the critical features of the skill they were working to implement and strategies for applying that skill in the specific context of secondary classrooms. The intervention we tested in this study has been successfully applied in elementary settings for both individuals and groups of teachers and was designed to be feasible for school-wide implementation. Successfully adapting teacher support for the secondary level must adequately consider the possibility of teachers’ limited training and experiences with classroom management.
Second, the varied instructional formats used in the secondary setting may necessitate adaptations in both the data collection procedures and the guidance provided to teachers regarding the application of classroom management skills. Lengthening the observations from 15 to 30 mins or even a full class period would allow researchers to capture a more complete range of the instructional and classroom management strategies used. In addition, teachers may need more guidance about how to apply strategies such as specific praise across different instructional formats (e.g., lecture, small group work, individual work time). Future researchers in this area may consider adapting training materials to include explicit training and examples on the use of classroom management skills across instructional formats and extending observation times to adequately capture this variety.
We present this study as an urgent call for more research on classroom management practices and teacher supports in the high school setting. As this study makes clear, we need further evidence to (a) support that similar practices work across settings and (b) suggest the specific modifications or adaptations that are needed to increase the effectiveness as we generalize strategies into the high school setting. Results of this study remind us to be cautious when making generalizations across settings and reinforce the importance of replication studies in different settings. High school teachers frequently report a need for support in the area of classroom management and the findings of this study suggest that more work is needed to identify effective and feasible interventions to support them.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was supported in part by a grant from the US department of education office of special education programs. #H326S130004