Teacher Praise and Reprimands: The Differential Response of Students at Risk of Emotional and Behavioral Disorders

Abstract

In the United States, many teachers feel underprepared to manage student classroom behavior positively. Such management is crucial for students to learn effectively, especially those with or at risk of emotional and behavioral disorders (EBD). Although increasing teacher praise and decreasing teacher reprimands may be research-based practices, more empirical evidence is required for them to be considered evidence based. The current study of 65 elementary school teachers and 239 students across three states contrasted the effects of these teacher behaviors on engagement and disruptions of students who were and were not at risk of EBD. Using structural equation modeling, we examined how the engagement and disruptions of students at risk were more sensitive to teacher praise and reprimand than the behavior of their typical peers. These results support the literature and invite teachers to consider that who they praise and reprimand is just as important as how.

Keywords

praise reprimands elementary school emotional and behavioral disorders

Students with emotional and behavioral disorders (EBD) face numerous obstacles to learning. They typically rank very low on teacher desirability (Soodak, Podell, & Lehman, 1998), receive very little teacher praise (Rathel, Drasgow, Brown, & Marshall, 2014; Rathel, Drasgow, & Christle, 2008), and are more likely to fail academically than students with a learning disability or with no disability (Nelson, Benner, Lane, & Smith, 2004). Students with EBD demonstrate unsuccessful peer relationships, antisocial behavior, internalizing behavior, aggression, and attention problems (Conley, Marchant, & Caldarella, 2014). Such students commonly use argumentative language, make disruptive statements during classroom instruction, and leave their seats often (Weeden, Wills, Kottwitz, & Kamps, 2016). A meta-analysis including 2,486 participants with EBD (Reid, Gonzalez, Nordness, Trout, & Epstein, 2004), of whom 80% were male, found academic performance was affected across all subjects (particularly for students younger than 12 years) in both resource and general education classrooms. In a study of 422 students with disabilities (Doren, Bullis, & Benz, 1996), researchers found that individuals with severe emotional disturbance were 13 times more likely to be arrested than peers with other disabilities.

Researchers and practitioners have attempted to help students both with and at risk of EBD overcome behavioral and academic challenges by studying the relationship between teacher praise or reprimands and subsequent student behavior (Partin, Robertson, Maggin, Oliver, & Wehby, 2009; Reinke, Herman, & Stormont, 2013). Despite promising findings, little is known about the relationship between natural rates of teacher praise and student behavior (Floress, Jenkins, Reinke, & McKown, 2018). For example, no clear patterns regarding when or for whom teacher praise might be effective were revealed during a recent review of praise literature (Moore et al., 2018), and only one study has attempted to explore the relationship between differential rates of teacher classroom management skills and student behavior (Gage, Scott, Hirn, & MacSuga-Gage, 2018). Gaps in the literature also show the need for studies with large samples using consistent operational definitions (Jenkins, Floress, & Reinke, 2015).

Teacher Praise

Teacher praise has been referred to as encouragement (Abramowitz, O’Leary, & Rosen, 1987) or positive verbal reinforcement (Kennedy & Jolivette, 2008). Some consider praise as a construct to reinforce student behavior (Weeden et al., 2016) or simply to recognize student engagement (Embry & Biglan, 2008). Praise can be written (Caldarella, Christensen, Young, & Densley, 2011) or spoken (Kennedy & Jolivette, 2008), or it may be conveyed with gestures or tangible objects (Floress, Beschta, Meyer, & Reinke, 2017).

Praise has been effective in general education (Partin et al., 2009; Reinke, Lewis-Palmer, & Merrell, 2008), particularly for students with EBD (Conroy, Sutherland, Snyder, Al-Hendawi, & Vo, 2009) or at risk of EBD (Sutherland, Wehby, & Copeland, 2000). It has been considered the simplest classroom management strategy to implement (Gable, Hester, Rock, & Hughes, 2009) and has shown evidence of positive effects on student academic and social outcomes (Simonsen, Fairbanks, Briesch, Myers, & Sugai, 2008). Using Council for Exceptional Children guidelines, Royer, Lane, Dunlap, and Ennis (2018) considered behavior-specific praise to meet criteria as a potentially evidence-based practice. Unfortunately, praise is not implemented as often or as well as it could be (Gage et al., 2018).

In elementary classrooms, rates of verbal and nonverbal praise appear to be low for general education students (M = 0.38–0.75 per minute, Floress, Jenkins, et al., 2018; M = 0.38 per minute, Reddy, Fabiano, Dudek, & Hsu, 2013) as well as for students exhibiting disruptive behaviors (M = 0.46 per minute; Reinke, Lewis-Palmer, & Martin, 2007) and those with EBD (0.33–1.37 per minute; Rathel et al., 2014; Rathel et al., 2008). Unfortunately, students tend to receive less praise as they become older (Floress, Jenkins, et al., 2018; Reddy et al., 2013).

Reviewing 40 years of classroom praise research, Jenkins et al. (2015) found limitations: less focus on students who are at risk of or have EBD in general education, small sample sizes, and single subject or correlational designs. Moore and colleagues (2018) have called for research specifically addressing when or for whom teacher praise is effective. In response, the current study used a large student sample (with and without EBD risk) across three states, from multiple grade levels, during various classroom activities, using clear operational definitions.

Teacher Reprimands

Teacher reprimands have been called correction statements (Allday et al., 2012), negative communication (Rathel et al., 2008), and contingent punishment (Merrell, Ervin, & Gimpel Peacock, 2012). Reprimands have included both verbal statements (Caldarella, Williams, Hansen, & Wills, 2015) and gestures (Weeden et al., 2016). Reprimand rates in schools are generally low for students with behavior problems (M = 0.01–0.03 per minute, Shores, Jack, et al., 1993) and without behavior problems (M = 0.67 per minute, Reinke et al., 2013). However, reprimands are more prevalent than praise in many elementary classrooms (Van Acker, Grant, & Henry, 1996), and the praise–reprimand ratio tends to worsen (more reprimands than praise) with increasing grade level (Reddy et al., 2013).

Relatively little research has addressed teacher reprimands. In one study, 206 elementary students screened to be at high risk of aggression received more reprimands than their peers, which predicted increasing negative behavior and noncompliance (Van Acker et al., 1996). Harsh reprimands positively correlate with teacher emotional exhaustion (Reinke et al., 2013). These potentially negative outcomes indicate a need to better understand the relationship of teacher reprimands to student engagement and disruptions.

Student Engagement

Student engagement has been studied in both general education (Aitken et al., 2011; Germer et al., 2011) and special education classrooms (Bock & Erickson, 2015). Proactive interventions can improve the engagement of students with or at risk of EBD (Allday et al., 2012). Results of 1,197 direct observations of teacher and student behaviors (Scott, Hirn, & Alter, 2014) showed a significant positive correlation between total instructional time and student engagement. Despite promising intervention outcomes, observed student engagement can vary. For example, in private schools or emotional support classrooms (Grades 1–12), students at risk of EBD were found to be engaged between 68% and 74% of the time (Hayling, Cook, Gresham, State, & Kern, 2008), whereas similar students in general education early childhood (K–2) settings were found to be between 82% and 92% engaged (Caldarella et al., 2015). Variability across ages and settings suggests engagement alone may not be very descriptive of EBD.

Student Disruptions

Student disruptions have been studied in general education classrooms (Shores, Gunter, & Jack, 1993) and alternative schools (Denune et al., 2015) by observing general education students (Reinke et al., 2008), students at risk (Kamps et al., 2011), and students with EBD (Sutherland, Alder, & Gunter, 2003). One study (Reinke et al., 2008) involved giving feedback to four teachers regarding their rates of praise. As teacher praise rates increased, student disruptions decreased. In a study including 294 elementary general education students, the average rate of disruptions was 0.03 per minute (Scott et al., 2014). In contrast, another study (Caldarella et al., 2015) found disruptions made by students at risk in general education (K–2) settings ranged from 0.54 to 1.61 per minute. Considering the large variability in the engagement of students at risk discussed earlier, rates of disruptions help to create a more detailed profile of what EBD looks like in elementary school.

Study Purpose

The purpose of the current study was to understand the differential relationship between naturally occurring rates of teacher behavior and various behaviors of students who are and are not at risk in elementary classrooms. The following research questions guided the study:

Research Question 1: How are teacher praise and reprimand rates related to the engagement of students at risk when compared with the engagement of peer-comparison students?

Research Question 2: How are teacher praise and reprimand rates related to disruption rates of students at risk when compared with the disruption rates exhibited by peer-comparison students?

Method

Setting and Participants

Data were previously gathered from 18 elementary schools across Kansas, Tennessee, and Utah as part of a 3-year, multisite efficacy trial of a group contingency intervention, Class-Wide Function-Related Intervention Teams (CW-FIT), measuring teacher and student behaviors (H. P. Wills et al., 2010). As part of the efficacy trial, classrooms were randomly assigned to either treatment or control conditions. Teachers in control classrooms conducted academic instruction using their typical classroom management practices, including behavior charts, token economies, and praise. Demographic information was gathered from school records. All other data for the current study were gathered from efficacy trial participants in control classrooms by observing naturally occurring teacher and student behavior.

Teachers

Districts referred schools for participation, and principals allowed researchers to request voluntary teacher participation. Teachers completed approved informed consent forms following requirements of institutional review boards at universities and school districts. Teachers were asked which time of the day they experienced the most challenging student behavior, and all data were collected at these times. The 65 participating teachers were 97% female and 86% White/Caucasian. Of these participants, 45% had a master’s degree and 44% had a bachelor’s degree; 18% had been teaching for 1 year, but years of teaching ranged from 0 to 34 years (M = 10.86 years, SD = 9.69 years). Observations across Grades K–6 included math (28%), language arts (13%), reading (15%), writing (11%), social studies (4%), science (1%), and other (2%). Approximately 4% of the classrooms were special education settings.

Students

Among the sample included in the current study, 64% were male, 48% were White/Caucasian, and 31% were Black/African American. English was the primary language of 71%. Free/reduced-price lunch (FRL) ranged from 34% to 98% (M = 68.27, SD = 20.77). Table 1 displays student demographic information by group. The 8% who required an individualized education plan were classified with specific learning disability (50%), autism spectrum disorder (28%), specific language impairment (17%), and intellectual disability (6%). The current study was part of a larger conceptual replication and, accordingly, used the same screening procedures as the study being replicated (H. Wills, Kamps, Fleming, & Hansen, 2016).

Table 1.

Student Demographics: Gender, Grade, and Ethnicity by Group.

Variable	Students at risk (n = 130)		Peer-comparison students (n = 109)
Variable	Frequency	Percentage	Frequency	Percentage
Gender
Male	99	76.2	53	48.6
Female	31	23.8	52	47.7
Missing	0	0	4	3.7
Grade
K	20	15.4	16	14.7
1	22	16.9	18	16.5
2	21	16.2	18	16.5
3	25	19.2	24	22.0
4	16	12.3	13	11.9
5	15	11.5	12	11.0
6	5	3.8	4	3.7
SPED	6	4.6	4	3.7
Missing	0	0	0	0
Ethnicity
Black/African American	51	39.2	22	20.2
Hispanic/Latino	17	13.1	16	14.7
White/Caucasian	60	46.2	54	49.5
Asian/Pacific Islander	1	0.8	5	4.6
Other	1	0.8	1	0.9
Missing	0	0	11	10.1
FRL	70	81.4	43	67.4

Note. The range of FRL by school was 34% to 98% (M = 68.27; SD = 20.77) with no statistically significant difference between groups (p > .05). FRL = free/reduced-price lunch. SPED = special education.

Identification of students at risk

Of the 239 participating students, 130 (54%) were identified as at risk of EBD using the following process. Teachers first completed Stage 1 of the Systematic Screening for Behavior Disorders (SSBD; Walker & Severson, 1992) to nominate students with significant behavior problems. The parents of the three top ranked students on teacher-generated lists of externalizing and internalizing categories were contacted to obtain informed consent and student assent. The Social Skills Improvement System (SSIS; Gresham & Elliott, 2008) was used to further assess nominated students, who were included in the at-risk group only if their scores on the Problem Behavior subscale were in the above average range (standard score ⩾ 115). Behavior of students at risk was verified with the SSIS and confirmed by three to five direct observations (as defined below) during a 3-week period. Students with engagement levels below 75% or having more than 10 disruptions in at least two out of five 15-min observations were considered at risk.

Identification of peer-comparison students

Of the 239 students, 109 (46%) were identified as peer-comparison students using the following process. After completing SSBD Stage 1, teachers identified and ranked peer-comparison students. Teachers listed up to four students whose classroom behavior was appropriate and cooperative to be observed to compare with students at risk. Informed consent was obtained from the parents of these students, along with student assent. As with the at-risk nominees, direct observation data (e.g., engagement, disruptions) were reviewed to confirm peer-comparison status over the course of the study.

Measures and Procedures

Data in the current study were collected over 4 to 6 months. Measures consisted of direct observations of student and teacher behavior, as well as classroom management ratings. Each student was observed between 3 and 9 times, and classroom management ratings were collected during each observation. Raw data were entered and stored electronically.

Direct observations

Student (engagement and disruptions) and teacher (praise and reprimands) data were collected by direct observations using the Multi-Option Observation System for Experimental Studies (MOOSES; Tapp, Wehby, & Ellis, 1995). This computer-based tool allows researchers to record both frequency and duration events (Reinke et al., 2013; Smith, Lewis, & Stormont, 2011). Functions of teacher and student behaviors were not identified. Students who were at risk were the focus of the efficacy trial; as such, they were observed more often, an average of 9 times (M = 9.32, SD = 1.37) during 15-min sessions (M = 14.69 min, SD = 0.32 min), compared with peer-comparison students who were observed an average of 3 times (M = 3.28, SD = 0.97) also during 15-min sessions (M = 14.76 min, SD = 0.45 min). The total observation minutes were 1,924 for students at risk and 1,609 for peer-comparison students; 20% of observations included a reliability observer.

To ensure accuracy, researchers trained all observers to recognize and record engagement, disruption, praise, and reprimand behaviors by (a) memorizing definitions found in Table 2, (b) practicing with videotaped classrooms to achieve 90% reliability (with a master code file) over three sessions, and (c) practicing in nonstudy classrooms to achieve 90% reliability across three sessions with the research coordinator. The training was complete when all observers reached 90% accuracy in training sessions. To calculate interobserver agreement (IOA), the number of agreements was divided by the total number of agreements plus disagreements. IOA percentage was acceptable for student engagement (M = 98.39, SD = 3.21), teacher reprimands (M = 91.87, SD = 23.42), teacher praise (M = 91.72, SD = 26.18), and student disruptions (M = 89.48, SD = 22.20).

Table 2.

Definitions of Student and Teacher Variables.

Variable	Definition	Examples	Measurement	Equation
Teacher praise	Verbal statements indicating approval of appropriate behavior (beyond the correct response to a question), to individuals or groups, as indicated by tone of voice or content	“Sarah, give yourself a pat on the back for finishing all your homework.” “Everyone in class made it back from the break on time, way to go.”	15-min observations, recorded via frequency codes using MOOSES	Praise rate per minute = teacher praises given to student (averaged) / 15 min
Teacher reprimands	Verbal statements to individuals or groups indicating disapproval of inappropriate behavior (including threats or scolding) or desire that a specific behavior be stopped	“Robyn, I told you to stop throwing pencils.” “Everyone needs to stop talking right now.”	15-min observations, recorded via frequency codes using MOOSES	Reprimand rate per minute = teacher reprimands given to student (averaged) / 15 min
Student engagement	Student action in response to assigned/approved task or student focused on assigned/approved task	Student responding verbally following teacher request for comments, silently working on math problems during independent work time Student listening during silent reading, quietly waiting for other students to finish math problems	15-min observations, recorded via duration codes using MOOSES	Engagement = percentage of time engaged during observation
Student disruptions	Voluntary inappropriate physical/motor or verbal behavior, including gestures, intended to self-stimulate, gain attention, or escape, which may or may not detract from the learning of peers	Making inappropriate gestures, shouting in class, engaging in physical violence against a peer or a teacher	15-min observations, recorded via frequency codes using MOOSES	Disruption rate per minute = student disruptions (averaged) / 15 min

Note. MOOSES = Multi-Option Observation System for Experimental Studies.

Classroom Management Rating Form (CMRF)

This nine-item rating form (copy available from the corresponding author) was based, in part, on the Classroom Atmosphere Rating Scale (Wehby, Dodge, & Greenberg, 1993), a seven-item instrument measuring classroom climate. The CMRF includes statements such as “students follow rules appropriate to setting” and “praise (points) ratio to reprimands approximately 4:1,” which considered together represent important constructs previously discussed (e.g., teacher praise, teacher reprimands). The rating scale options are from 1 (40% of students or time) to 4 (90% of students or time), with higher scores indicating better classroom management. For a total percentage, item scores were summed and divided by 36 (total points possible). Across the 3 years of data collection, CMRF scores ranged from 25% to 85% (M = 57.89, SD = 12.51). We chose to include CMRF latent variables in structural equation models (SEMs) to control for the potential relationship between classroom environment and teacher or student behaviors. (Excluding these latent variables would have likely biased statistical analyses.) Because no psychometric data for the CMRF existed prior to this study, psychometric analyses were conducted (see “Data Analysis Plan” section).

Data Analysis Plan

We began with preliminary descriptive analyses to understand the nature of the data, then examined differences between at-risk and peer-comparison students across demographic and target variables. We first conducted a psychometric evaluation of the CMRF to assess internal consistency and factor structure. We then created two SEMs to answer the research questions regarding the differential relationship between teacher and student behaviors, one with student engagement and the other with student disruptions as the dependent variable.

Nested data required use of the clustering variable student identification number in the analysis (Mplus syntax “CLUSTER=studentID,” “TYPE=COMPLEX”). Fit indices produced by Mplus enabled good model fit. SPSS 24 was used to calculate Cronbach’s alpha. For all SEMs, as with regression, the assumptions of linearity, independence, normality, variance equality, and multicollinearity were considered. Independence was violated, as participating students were nested in classrooms. Residual plots showed no departure from normality or linearity.

Preliminary analyses

We used chi-square analyses to examine the proportion of students assigned to at-risk and peer-comparison groups by gender and ethnicity. We chose independent samples t tests to explore potential differences between groups.

Psychometric analyses

We assessed the psychometric properties of the CMRF in Mplus 8 and SPSS 24. We performed an exploratory factor analysis (EFA) on a random selection of 50% of the data (n = 120 students) and a confirmatory factor analysis (CFA) with the remaining data (n = 119 students) in Mplus (Worthington & Whittaker, 2006). Assumptions checked in the CFA included correct model, missing data, independent observations, and item linearity, with absence of multivariate outliers and extreme collinearity. See “Results” section for additional information.

SEM analyses

SEM is appropriate for observational and measurement analyses (MacCallum & Austin, 2000). Although the design was insufficient to prove causation, a secondary data analysis using SEM can build on previous research that addressed group differences between similar variables (Cook et al., 2017; Hayling et al., 2008) by portraying how those variables interact, and in what ways a variable might be related to one group in different ways than it is to another.

We ran a series of SEMs in Mplus 8 to answer the research questions. Initial models included grade level, site, and class subject, along with covariates listed below, but due to problems with multicollinearity and a finding of no significant change in outcome, we chose a more parsimonious model. Covariates of this clearer model included student status (at-risk = 1, peer = 0), CMRF score (latent variables: student classroom behavior and teacher classroom management), student gender (male = 1, female = 0), student ethnicity (Black/African American = 1, Other = 0), and interactions (student status with praise, student status with reprimands).

Results

Results of Preliminary Analyses

Preliminary analyses revealed significant differences between students at risk and their peers (see Tables 1 and 3). Rates of praise, reprimand, and disruptions were reported per minute, as in previous research (Reinke, Stormont, Herman, Wachsmuth, & Newcomer, 2015), with CMRF scores and engagement as percentages. Considering all students, reprimands (M = 0.07, SD = 0.07) occurred significantly (t = −5.54, p < .001) more often than praises (M = 0.04, SD = 0.05). Praise and reprimand rates were nearly equal for peer-comparison students, but students at risk received significantly more reprimands (t = −6.80, p < .001). Students at risk were less engaged by approximately 20% (t = −12.31, p < .001) and disruptive approximately 3 times as often (t = 9.34, p < .001). Differences between praise rates and CMRF scores were not significant. Chi-square analyses revealed a greater proportion of males, χ² (1) = 16.76, p < .001, and Black/African American students, χ²(4) = 10.24, p < .05, in the at-risk group.

Table 3.

Descriptive Statistics by Student Group.

Variables	Students at risk (n = 130)			Peer-comparison students (n = 109)			t	p
Variables	M	SD	Minimum–maximum	M	SD	Minimum–maximum	t	p
Praise rate	0.05	0.06	0–0.30	0.04	0.05	0–0.29	0.79	.43
Reprimand rate	0.10	0.07	0–0.33	0.04	0.05	0–0.20	7.19	.001*
CMRF score	57.58	11.08	27.88–77.50	58.17	14.08	25–84.73	−0.36	.72
Engagement (%)	72.98	14.72	31.48–96.25	91.70	8.39	60.33–100.00	−12.31	.001*
Disruption rate	0.68	0.51	0.03–3.75	0.22	0.23	0–1.10	9.34	.001*

Note. All rates are per 1 min. CMRF = Classroom Management Rating Form.

p < .001.

Results of Psychometric Analyses

We performed psychometric analyses to examine the factor structure and model fit of the CMRF when used in elementary schools. We used appropriate cross-validation techniques (Worthington & Whittaker, 2006), in which we randomly divided the data sample in half, performing an EFA on the first half (120 students) and a CFA on the second half (119 students). Missing data (4%, due to insufficient time to complete the rating form during original data collection) were addressed using the full information maximum likelihood method in Mplus. A two-factor model had the best fit. Residual errors were allowed to correlate between Items 7 and 8 at .78 (p < .001), which appeared to be accessing the same construct of teacher behavior. Item 9 was deleted because it did not relate specifically to student or teacher classroom behavior.

The final factor solution had acceptable fit statistics, root mean square error of approximation (RMSEA) = 0.13, comparative fit index (CFI) = 0.96, Tucker–Lewis index (TLI) = 0.93, standardized root mean square residual (SRMR) = 0.04, with cutoffs for RMSEA < 0.08 (Byrne, 2013), CFI > 0.95 (Hu & Bentler, 1999), and TLI > 0.90 (Wang & Wang, 2012). These latent variables (student classroom behavior and teacher classroom management) were correlated with a value of .83 (p < .001). See Table 4 for factor loadings. Cronbach’s alpha was .96 for student classroom behavior and .83 for teacher classroom management. With these acceptable results, we used SEM, including latent variables as covariates.

Table 4.

Factor Loadings for CMRF Factors (N = 239).

CMRF items	Standardized factor loadings
Student classroom behavior
1. Students are compliant during academic instruction.	0.96**
2. Students follow rules appropriate to setting.	0.94**
4. Students are focused and on task.	0.93**
Teacher classroom management
3. Transitions are short with only minor disruptions.	0.81**
5. Level of lesson structure includes organized clear directions and sufficient work to keep students busy.	0.93**
6. Teacher ignores minor inappropriate behaviors.	0.85**
7. Frequent and specific praise are given (points count toward frequency).	0.26**
8. Praise (points) ratio to reprimands are approximately 4:1.	0.50**
Deleted items
9. Three to five clearly and positively stated classroom expectations/rules are visibly posted.

Note. Researchers allowed residual errors to be correlated between Items 7 and 8; correlation was .78 (p < .001). CMRF = Classroom Management Rating Form.

p < .01. **p < .001.

Results of SEM

Student outcome variables were regressed on student status (at risk = 1, peer = 0), teacher praise rate, teacher reprimand rate, CMRF score for latent variables, student gender (male = 1, female = 0), student ethnicity (Black/African American = 1, Other = 0), and interactions of student status with teacher praise and teacher reprimands. Significant interactions of student status with teacher praise or reprimands were graphed with a method commonly used: considering one standard deviation below and one above the mean to represent the lower (praise, 0 per minute; reprimand, 0.01 per minute) and higher (praise, 0.10 per minute; reprimand, 0.14 per minute) values in this sample. SEM results can be found in Table 5.

Table 5.

Betas, SEs, and Standardized Betas for SEMs (N = 239).

Variable	Engagement			Disruptions			Disruptions without A/P
Variable	B	SE	β	B	SE	β	B	SE	β
Target	−15.42***	1.49	−0.50	5.01***	0.64	0.36	4.99***	0.64	0.36
Praise rate	−1.10	1.01	−0.06	0.20	0.38	0.02	0.49	0.54	0.06
Reprimand rate	−1.14	1.18	−0.07	0.63	0.67	0.09	0.67	0.67	0.10
CMRF–SCB	9.58***	2.55	0.40	−4.31**	1.28	−0.40	−4.41***	1.22	−0.41
CMRF–TCM	−5.55	3.49	−0.21	4.56*	1.81	0.38	4.65*	1.80	0.39
Male	−1.93	1.38	−0.06	0.85	0.65	0.06	0.81	0.67	0.06
Black/African American	1.96	1.85	0.06	0.91	0.93	0.06	0.91	0.94	0.06
A/P	5.60***	0.99	0.23	0.48	0.89	0.04	—	—	—
A/R	−4.88**	1.46	−0.25	2.24*	1.05	0.25	2.20*	1.08	0.25
R ²	.59			.41			.41

Note. B = unstandardized beta; SE = standard error of beta; β = standardized beta; SEM = structural equation model; A/P = interaction—at-risk status and praise rate; CMRF = Classroom Management Rating Form; SCB = student classroom behavior; TCM = teacher classroom management; A/R = interaction—at-risk status and reprimand rate.

p < .05. **p < .01. ***p < .001.

Research Question 1 asked how teacher praise and reprimand rates related to the engagement of students at risk when compared with the engagement of peer-comparison students. With student engagement as the dependent variable, the model yielded an R² of .59. Interaction terms were both significant (see Figure 1). Higher praise rates (B = 5.60, p < .001) were associated with more engagement of students at risk, but engagement of peer-comparison students appeared relatively stable. Higher reprimand rates (B = −4.88, p < .01) were associated with less engagement of students at risk and of peer-comparison students (but with smaller magnitude). Student status (B = −15.42, p < .001) and the student classroom behavior factor of the CMRF (B = 9.58, p < .001) were significant; all other variables were not.

Figure 1.

Interaction of student status and teacher praise and reprimand rates on student engagement.

Research Question 2 asked how teacher praise and reprimand rates related to disruption rates of students at risk when compared with the disruption rates exhibited by peer-comparison students. With student disruptions as the dependent variable, the model yielded an R² of .41. A significant interaction (B = 2.24, p < .05) indicated that a higher rate of teacher reprimands was associated with more disruptions by students at risk and that disruptions by their comparison peers also increased, but with a much smaller magnitude (see Figure 2). This specific interaction (β = 0.25) also met criteria to be considered influential in educational settings according to What Works Clearinghouse (2014). Student status (B = 5.01, p < .001) and both student classroom behavior (B = −4.31, p < .01) and teacher classroom management (B = 4.56, p < .05) factors were significant. No other variables were significantly predictive. The interaction between student status and teacher praise was not significant, and a second application of the model without the interaction term showed no change in outcome. However, student status (B = 4.99, p < .001) and both student classroom behavior (B = −4.41, p < .01) and teacher classroom management (B = 4.65, p < .05) CMRF factors remained significant.

Figure 2.

Interaction of student status and teacher reprimand rate on student disruptions.

Discussion

The current study was undertaken to examine relationships between naturally occurring rates of elementary teacher and student behaviors. The unclear relationship between natural rates of teacher praise and student behavior (Floress, Jenkins, et al., 2018; Gage et al., 2018), along with gaps in praise literature (Jenkins et al., 2015), provided direction for the study. Considering the negative academic (Nelson et al., 2004) and social (Doren et al., 1996) outcomes associated with EBD, proactive efforts to mitigate these outcomes are particularly valuable for students at risk. Results are discussed according to research questions, grouped by teacher behavior.

Praise

Research has illustrated the effectiveness of increasing teacher praise on the task engagement of students with and at risk of EBD (Allday et al., 2012; Sutherland et al., 2000). Behavior-specific praise has shown a significant negative correlation with student off-task behavior (Floress, Jenkins, et al., 2018). Our results support both findings: Higher rates of teacher praise were associated with increased engagement of students at risk, whereas engagement of peer-comparison students remained relatively stable (see Figure 1). The particular impact of teacher praise on students at risk may be explained, in part, by the small amount of praise many students with EBD receive from teachers (Rathel et al., 2014; Rathel et al., 2008). Similarly low amounts of praise at home or in the community may result in few positive interactions with adults, making the praise these students do receive more salient and effective. Students not at risk may receive more positive adult interaction across settings, making additional praise less impactful (e.g., possible satiation effects). The differential results from this study may be applicable to antipraise literature (e.g., Dweck, 1999): Whereas some students may appear to respond counterintuitively to teacher praise, others (e.g., students at risk) appear quite sensitive to it.

Disruptions by at-risk and peer-comparison students were not correlated differently with teacher praise in this sample. The interaction between teacher praise and at-risk status was not significant. However, at-risk status and classroom management were significant factors, clarifying how classroom environment in elementary school may influence student behavior in later years (Kellam, Ling, Merisca, Brown, & Ialongo, 1998). Such findings suggest both student characteristics and classroom context must be considered in efforts to improve student outcomes. Furthermore, the finding of no significant interaction between teacher praise and at-risk status does not necessarily mean these variables do not interact with student disruptions. Factors such as teacher reprimands could be obscuring the relationship between praise and disruptions, as previous research and analyses with the current sample have found reprimands to be more prevalent than praise (Van Acker et al., 1996). In a previous study (Floress, Jenkins, et al., 2018), teacher praise did not correlate significantly with student disruptions, but observed ranges of data may not have been large enough to observe a correlation; the same may have been true in the current study. Teachers must increase praise to help students (who are at risk) succeed (Cambone, 1990)—avoiding academic failure, decreasing the likelihood of arrest, and improving social and behavioral outcomes.

Reprimands

Use of reprimands has been studied less often than other corrective techniques (Van Houten, Nau, MacKenzie-Keating, Sameoto, & Colavecchia, 1982), possibly due to ethical concerns over increasing teacher reprimand rates to study their effects. If teacher reprimands correlate with increased noncompliance and negative student behavior (Van Acker et al., 1996), manipulating those rates would mean deliberately producing harmful student outcomes. The current study has helped to clarify the relationship between teacher and student variables without deliberate distortion, and the results relate directly to the relationship between teacher reprimands and student engagement, which has been studied very little.

Higher rates of teacher reprimands were associated with lower rates of student engagement in this sample. Classroom management and student at-risk status were significant factors in the SEM model, and the interaction between teacher reprimands and at-risk status was significant when measuring engagement. A significant interaction suggests the engagement of students at risk was more sensitive to teacher reprimands (decreasing with greater magnitude) than engagement of peer-comparison students. The sensitive nature of the engagement of students at risk in this sample emphasizes how participant characteristics may affect outcomes.

Although classroom reprimand rates are generally low (Gage et al., 2018; Reinke et al., 2013), reprimands have been found to be more prevalent than praise in elementary schools (Van Acker et al., 1996). The results of the current study support both the low rates and higher prevalence of reprimands found in other research. Levels of disruptions of students at risk were found to be more sensitive to teacher reprimands than those of peer-comparison students: increasing at a much faster rate as reprimands increased. Gage and colleagues (2018) did not find significant correlations between rates of reprimands and student disruptions, but this may have resulted from their random selection of all student participants. Gage and colleagues also did not report information regarding EBD status, which appears to have contributed substantially to the reprimand–disruption relationship found in the current study.

Disruptions observed across ages and settings highlight a vulnerability of students at risk, who may become more disruptive because they are reprimanded more often. Educators need to be aware of reprimand–disruption relationships early in a child’s education if they want to mitigate potential negative outcomes associated with EBD, especially because the teacher praise–reprimand ratio worsens with increasing grade levels (Reddy et al., 2013). If sensitivity to teacher variables could be replicated as a distinguishing characteristic of students at risk, researchers could develop ways of identifying these students as candidates for early intervention.

Limitations and Directions for Future Research

Various limitations in the current study suggest opportunities for future research. The statistical analyses were performed on an existing data set, which limited the study’s design, its generalizability, and the applicable statistical methods for data analysis. For example, MOOSES measured only verbal praise and reprimands of teachers. Many previous studies have measured both verbal and nonverbal/gestured behavior (see, for example, Reddy et al., 2013; Reinke et al., 2013). This limitation could be one reason why rates in this data set were lower than many observed in other studies. Also, all the observations were completed in control classrooms, where rates of praise and reprimands are traditionally low. Students at risk were observed more frequently than peer-comparison students, producing a potentially limited range of data. Promising correlations were found between teacher and student variables, but this might have been different if the data range had been larger.

The identification process for peer-comparison students was less detailed than screening for students at risk. Teachers used multiple screening measures and observations to confirm student inclusion in the at-risk group after nomination, but peer-comparison students were simply recommended by their teachers as exhibiting appropriate and cooperative behaviors and included without further screening beyond review of MOOSES data (e.g., engagement, disruptions). To consider these peer-comparison students as representative of average elementary students would require further information. Statistical analyses were also limited because the study was a secondary data analysis. Although results are promising, they can only illustrate relationships between variables, not causal contingencies.

Future research could address many of these limitations. Replications could use a stronger research design, such as a randomized controlled trial (RCT), and could be expanded to include a larger sample, which could increase generalizability. Further research might benefit from continuing into middle schools and high schools to examine whether teacher behavior is similarly related to student behavior across developmental stages. A replicated study using an RCT design would allow researchers to include a screening measure for peer-comparison students and to manipulate rates of teacher praise and reprimands while measuring the effects of those respective rates on student engagement and disruptions.

Implications

Some researchers have argued the way praise is given determines how effective it is (Brophy, 1981; Collins & Cook, 2016). For example, praise should be contingent (Royer et al., 2018). However, data from the current study suggest first considering who to praise and reprimand rather than how. Students at risk in this sample appeared to be more sensitive to teacher praise and particularly more sensitive to teacher reprimands than peer-comparison students (see Figures 1 and 2). This finding is a “potential idiosyncratic difference” that Moore and colleagues (2018) encouraged educators to look for (p. 11). It also highlights needs of students at risk because they may experience negative outcomes throughout their education and beyond. Such a finding also affirms the application of positive behavior support theory with elementary students, especially those at EBD risk, by considering specific factors such as at-risk status and classroom management.

Understanding correlations of classroom praise and reprimands with student engagement and disruptions, particularly what those relationships mean for individuals (e.g., avoiding academic failure, decreasing the likelihood of arrest, and improving social and behavioral outcomes), teachers may be able to intervene early and help students avoid developing EBD. The current study has yielded promising results, but future replication is needed to confirm their validity and extend generalizability. Future research could focus on rates of teacher praise and reprimand as well as ideal ratios to use with students at risk.

Footnotes

Action Editor: Wendy Oakes

Authors’ Note

The opinions presented in this article are those of the authors, and no endorsement by the agency is intended or implied.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported in this article was supported in part by a grant from the Institute of Education Sciences and the U.S. Department of Education (R324A120344) awarded to the University of Kansas in cooperation with Brigham Young University and Vanderbilt University. The opinions presented in this article are those of the authors and no endorsement by the agency is intended or implied.

ORCID iD

Paul Caldarella

References

Abramowitz

A. J.

O’Leary

S. G.

Rosen

L. A.

(1987). Reducing off-task behavior in the classroom: A comparison of encouragement and reprimands. Journal of Abnormal Child Psychology, 15, 153–163. doi:10.1007/BF00916345

Aitken

Harlan

Hankins

Michels

Moore

T. C.

Oakes

W. P.

Lane

K. L.

(2011). Increasing academic engagement during writing activities in an urban elementary classroom. Beyond Behavior, 20(3), 31–43.

Allday

R. A.

Hinkson-Lee

Hudson

Neilsen-Gatti

Kleinke

Russel

C. S.

(2012). Training general educators to increase behavior-specific praise: Effects on students with EBD. Behavioral Disorders, 37, 87–98.

Bock

A. K.

Erickson

K. A.

(2015). The influence of teacher epistemology and practice on student engagement in literacy learning. Research and Practice for Persons With Severe Disabilities, 40, 138–153. doi:10.1177/1540796915591987

Brophy

(1981). Teacher praise: A functional analysis. Psychological Review, 88, 93–134. doi:10.2307/1170249

Byrne

B. M.

(2013). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. Mahwah, NJ: Lawrence Erlbaum.

Caldarella

Christensen

Young

K. R.

Densley

(2011). Decreasing tardiness in elementary school students using teacher-written praise notes. Intervention in School and Clinic, 47, 104–112. doi:10.1177/1053451211414186

Caldarella

Williams

Hansen

B. D.

Wills

(2015). Managing student behavior with class-wide function-related intervention teams: An observational study in early elementary classrooms. Early Childhood Education Journal, 43, 357–365. doi:10.1007/s10643-014-0664-3

Cambone

(1990). Teachers and teaching: Tipping the balance. Harvard Educational Review, 60, 217–236. doi:10.17763/haer.60.2.2776711615220k33

10.

Collins

L. W.

Cook

(2016). Never say never: The appropriate and inappropriate use of praise and feedback for students with learning and behavioral disabilities. Advances in Learning and Behavioral Disabilities, 29, 153–173. doi:10.1108/S0735-004X20160000029007

11.

Conley

Marchant

Caldarella

(2014). A comparison of teacher perceptions and research-based categories of student behavior difficulties. Education, 134, 439–451.

12.

Conroy

M. A.

Sutherland

K. S.

Snyder

Al-Hendawi

(2009). Creating a positive classroom atmosphere: Teachers’ use of effective praise and feedback. Beyond Behavior, 18(2), 18–26.

13.

Cook

C. R.

Grady

E. A.

Long

A. C.

Renshaw

Codding

R. S.

Fiat

Larson

(2017). Evaluating the impact of increasing general education teachers’ ratio of positive-to-negative interactions on students’ classroom behavior. Journal of Positive Behavior Interventions, 19, 67–77. doi:10.1177/1098300716679137

14.

Denune

Hawkins

Donovan

Mccoy

Hall

Moeder

(2015). Combining self-monitoring and an interdependent group contingency to improve the behavior of sixth graders with EBD. Psychology in the Schools, 52, 562–577. doi:10.1002/pits.21846

15.

Doren

Bullis

Benz

M. R.

(1996). Predicting the arrest status of adolescents with disabilities in transition. Journal of Special Education, 29, 363–380. doi:10.1177/002246699602900401

16.

Dweck

(1999). Caution: Praise can be dangerous. American Educator, 23(1), 4–9.

17.

Embry

D. D.

Biglan

(2008). Evidence-based kernels: Fundamental units of behavioral influence. Clinical Child and Family Psychology Review, 11, 75–113. doi:10.1007/s10567-008-0036-x

18.

Floress

M. T.

Beschta

S. L.

Meyer

K. L.

Reinke

W. M.

(2017). Praise research trends and future directions: Characteristics and teacher training. Behavioral Disorders, 43, 227–243. doi:10.1177/0198742917704648

19.

Floress

M. T.

Jenkins

L. N.

Reinke

W. M.

McKown

(2018). General education teachers’ natural rates of praise: A preliminary investigation. Behavioral Disorders, 43, 411–422. doi:10.1177/0198742917709472

20.

Gable

R. A.

Hester

P. H.

Rock

M. L.

Hughes

K. G.

(2009). Back to basics: Rules, praise, ignoring, and reprimands revisited. Intervention in School and Clinic, 44, 195–205. doi:10.1177/1053451208328831

21.

Gage

N. A.

Scott

Hirn

MacSuga-Gage

A. S.

(2018). The relationship between teachers’ implementation of classroom management practices and student behavior in elementary school. Behavioral Disorders, 43, 302–315. doi:10.1177/0198742917714809

22.

Germer

K. A.

Kaplan

L. M.

Giroux

L. N.

Markham

E. H.

Ferris

G. J.

Oakes

W. P.

Lane

K. L.

(2011). A function-based intervention to increase a second-grade student’s on-task behavior in a general education classroom. Beyond Behavior, 20(3), 19–30.

23.

Gresham

F. M.

Elliott

S. N.

(2008). Social Skills Improvement System: Rating Scales. Bloomington, MN: Pearson Assessments.

24.

Hayling

C. C.

Cook

Gresham

F. M.

State

Kern

(2008). An analysis of the status and stability of the behaviors of students with emotional and behavioral difficulties. Journal of Behavioral Education, 17, 24–42. doi:10.1007/s10864-007-9059-5

25.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. doi:10.1080/10705519909540118

26.

Jenkins

L. N.

Floress

M. T.

Reinke

(2015). Rates and types of teacher praise: A review and future directions. Psychology in the Schools, 52, 463–476. doi:10.1002/pits.21835

27.

Kamps

Wills

H. P.

Heitzman-Powell

Laylin

Szoke

Petrillo

Culey

(2011). Class-wide function-related intervention teams: Effects of group contingency programs in urban classrooms. Journal of Positive Behavior Interventions, 13, 154–167. doi:10.1177/1098300711398935

28.

Kellam

S. G.

Ling

Merisca

Brown

C. H.

Ialongo

(1998). The effect of the level of aggression in the first grade classroom on the course and malleability of aggressive behavior into middle school. Development and Psychopathology, 10, 165–185. doi:10.1017/S0954579498001564

29.

Kennedy

Jolivette

(2008). The effects of positive verbal reinforcement on the time spent outside the classroom for students with emotional and behavioral disorders in a residential setting. Behavioral Disorders, 33, 211–221.

30.

MacCallum

R. C.

Austin

J. T.

(2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51, 201–226.

31.

Merrell

K. W.

Ervin

R. A.

Gimpel Peacock

(2012). School psychology for the 21st century: Foundations and practices (2nd ed.). New York, NY: Guilford Press.

32.

Moore

T. C.

Maggin

D. M.

Thompson

K. M.

Gordon

J. R.

Daniels

Lang

L. E.

(2018). Evidence review for teacher praise to improve students’ classroom behavior. Journal of Positive Behavior Interventions. Advance online publication. doi:10.1177/1098300718766657

33.

Nelson

J. R.

Benner

G. J.

Lane

Smith

B. W.

(2004). Academic achievement of K-12 students with emotional and behavioral disorders. Exceptional Children, 71, 59–73. doi:10.1177/001440290407100104

34.

Partin

T. C. M.

Robertson

R. E.

Maggin

D. M.

Oliver

R. M.

Wehby

J. H.

(2009). Using teacher praise and opportunities to respond to promote appropriate student behavior. Preventing School Failure, 54, 172–178. doi:10.1080/10459880903493179

35.

Rathel

J. M.

Drasgow

Brown

W. H.

Marshall

K. J.

(2014). Increasing induction-level teachers’ positive-to-negative communication ratio and use of behavior-specific praise through e-mailed performance feedback and its effect on students’ task engagement. Journal of Positive Behavior Interventions, 16, 219–233. doi:10.1177/1098300713492856

36.

Rathel

J. M.

Drasgow

Christle

C. C.

(2008). Effects of supervisor performance feedback on increasing preservice teachers’ positive communication behaviors with students with emotional and behavioral disorders. Journal of Emotional & Behavioral Disorders, 16, 67–77. doi:10.1177/1063426607312537

37.

Reddy

L. A.

Fabiano

G. A.

Dudek

C. M.

Hsu

(2013). Instructional and behavior management practices implemented by elementary general education teachers. Journal of School Psychology, 51, 683–700. doi:10.1016/j.jsp.2013.10.001

38.

Reid

Gonzalez

J. E.

Nordness

P. D.

Trout

Epstein

M. H.

(2004). A meta-analysis of the academic status of students with emotional/behavioral disturbance. Journal of Special Education, 38, 130–143. doi:10.1177/00224669040380030101

39.

Reinke

W. M.

Herman

K. C.

Stormont

(2013). Classroom-level positive behavior supports in schools implementing SW-PBIS: Identifying areas for enhancement. Journal of Positive Behavior Interventions, 15, 39–50. doi:10.1177/1098300712459079

40.

Reinke

W. M.

Lewis-Palmer

Martin

(2007). The effect of visual performance feedback on teacher use of behavior-specific praise. Behavior Modification, 31, 247–263. doi:10.1177/0145445506288967

41.

Reinke

W. M.

Lewis-Palmer

Merrell

(2008). The classroom check-up: A classwide teacher consultation model for increasing praise and decreasing disruptive behavior. School Psychology Review, 37, 315–332.

42.

Reinke

W. M.

Stormont

Herman

K. C.

Wachsmuth

Newcomer

(2015). The brief classroom interaction observation-revised: An observation system to inform and increase teacher use of universal classroom management practices. Journal of Positive Behavior Interventions, 17, 159–169. doi:10.1177/1098300715570640

43.

Royer

D. J.

Lane

K. L.

Dunlap

K. D.

Ennis

R. P.

(2018). A systematic review of teacher-delivered behavior-specific praise on K-12 student performance. Remedial and Special Education. Advance online publication. doi:10.1177/0741932517751054

44.

Scott

T. M.

Hirn

R. G.

Alter

P. J.

(2014). Teacher instruction as a predictor for student engagement and disruptive behaviors. Preventing School Failure, 58, 193–200. doi:10.1080/1045988x.2013.787588

45.

Shores

R. E.

Gunter

P. L.

Jack

S. L.

(1993). Classroom management strategies: Are they setting events for coercion? Behavioral Disorders, 18, 92–102.

46.

Shores

R. E.

Jack

S. L.

Gunter

P. L.

Ellis

D. N.

DeBriere

T. J.

Wehby

J. H.

(1993). Classroom interactions of children with behavior disorders. Journal of Emotional and Behavioral Disorders, 1, 27–39. doi:10.1177/106342669300100106

47.

Simonsen

Fairbanks

Briesch

Myers

Sugai

(2008). Evidence-based practices in classroom management: Considerations for research to practice. Education and Treatment of Children, 31, 351–380. doi:10.1353/etc.0.0007

48.

Smith

S. C.

Lewis

T. J.

Stormont

(2011). The effectiveness of two universal behavioral supports for children with externalizing behavior in Head Start classrooms. Journal of Positive Behavior Interventions, 13, 133–143. doi:10.1177/1098300710379053

49.

Soodak

L. C.

Podell

D. M.

Lehman

L. R.

(1998). Teacher, student, and school attributes as predictors of teachers’ responses to inclusion. Journal of Special Education, 31, 480–497. doi:10.1177/002246699803100405

50.

Sutherland

K. S.

Alder

Gunter

P. L.

(2003). The effect of varying rates of opportunities to respond to academic requests on the classroom behavior of students with EBD. Journal of Emotional and Behavioral Disorders, 11, 239–248. doi:10.1177/10634266030110040501

51.

Sutherland

K. S.

Wehby

J. H.

Copeland

S. R.

(2000). Effect of varying rates of behavior-specific praise on the on-task behavior of students with EBD. Journal of Emotional and Behavioral Disorders, 8, 2–826. doi:10.1177/106342660000800101

52.

Tapp

J. T.

Wehby

J. H.

Ellis

(1995). A multiple option observation system for experimental studies: MOOSES. Behavior Research Methods, Instruments, & Computers, 27, 25–31. doi:10.3758/bf03203616

53.

Van Acker

Grant

S. H.

Henry

. (1996). Teacher and student behavior as a function of risk for aggression. Education and Treatment of Children, 19, 316–334.

54.

Van Houten

Nau

P. A.

MacKenzie-Keating

S. E.

Sameoto

Colavecchia

. (1982). An analysis of some variables influencing the effectiveness of reprimands. Journal of Applied Behavior Analysis, 15, 65–83. doi:10.1901/jaba.1982.15-65

55.

Walker

H. M.

Severson

H. H.

(1992). Systematic screening for behavior disorders. Longmont, CO: Sopris West.

56.

Wang

(2012). Structural equation modeling: Applications using Mplus. West Sussex, UK: John Wiley.

57.

Weeden

Wills

H. P.

Kottwitz

Kamps

(2016). The effects of a class-wide behavior intervention for students with emotional and behavioral disorders. Behavioral Disorders, 42, 285–293. doi:10.17988/bd-14-12.1

58.

Wehby

J. H.

Dodge

K. A.

Greenberg

. (1993). Classroom atmosphere rating scale. (Unpublished technical manual). Nashville, TN: Vanderbilt University.

59.

What Works Clearinghouse. (2014). Procedures and standards handbook (Version 3.0). Washington, DC: Institute of Education Sciences.

60.

Wills

Kamps

Fleming

Hansen

(2016). Student and teacher outcomes of the class-wide function-related intervention team (CW-FIT) program. Exceptional Children, 83, 58–76. doi:10.1177/0014402916658658

61.

Wills

H. P.

Kamps

Hansen

Conklin

Bellinger

Neaderhiser

Nsubuga

(2010). The classwide function-based intervention team program. Preventing School Failure, 54, 164–171. doi:10.1080/10459880903496230

62.

Worthington

R. L.

Whittaker

T. A.

(2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34, 806–838. doi:10.1177/0011000006288127