Effects of Targeted Professional Development on Teachers’ Specific Praise Rates

Abstract

Classroom management continues to be a concern for educators, administrators, and policymakers. Although evidence-based classroom management practices exist, teachers often receive insufficient training and support to implement these practices successfully. Schools need reliable and efficient ways to support teachers’ classroom management. This study employed a multiple baseline design across elementary teachers to investigate the effect of targeted professional development (TPD), an efficient approach that incorporated self-management and email prompts, on teachers’ rates of specific praise. We replicated this study at a second elementary school to provide additional evidence of the efficacy of TPD. Across teachers in both schools, data support a functional relation between TPD and an increase in teachers’ use of specific praise.

Keywords

classroom management teacher training self-management professional development specific praise

Students experience positive outcomes when teachers implement evidence-based classroom management practices (Simonsen, Fairbanks, Briesch, Myers, & Sugai, 2008). Unfortunately, teachers routinely cite classroom management as an area in which they need support, as classroom management is largely ignored during pre- and in-service professional development (Begeny & Martens, 2006; Freeman, Simonsen, Briere, & MacSuga-Gage, 2014; Wei, Darling-Hammond, & Adamson, 2010). This lack of support contributes to the high rate of teacher attrition; nearly half of teachers leave the field within their first 5 years of teaching, and teachers who leave due to job dissatisfaction identify problems with student discipline and motivation as contributing factors (Ingersoll & Smith, 2003; Smith & Ingersoll, 2004).

Furthermore, in a recent survey, 97% of teachers reported concerns with disruptive or acting out behaviors, only 56% had heard of “evidence-based practices,” and 21% reported having no or minimal training in behavioral interventions (Reinke, Stormont, Herman, Puri, & Goel, 2011, pp. 6–7). Consequently, researchers have consistently demonstrated that teachers implement evidence-based classroom management practices at lower levels than recommended (e.g., Reinke, Herman, & Stormont, 2013; Scott, Alter, & Hirn, 2011). As a result, many students receive instruction from teachers who struggle with classroom management.

Leaders in research, policy, and practice need to identify efficient and effective strategies to support teachers’ classroom management. Traditional “sit and get” professional development approaches promote few changes in the classroom (e.g., National Joint Committee on Learning Disabilities, 2000), as research shows that training alone does not produce sustained change in implementation (e.g., Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). Instead, systematic reviews of the literature demonstrate that multi-component approaches, which include direct training, coaching, and performance feedback (often driven by an expert or researcher performing in vivo observation, support, or both), result in desired increases in teachers’ classroom management behaviors (e.g., Allen & Forman, 1984).

Comprehensive training approaches are effective but resource-intensive (e.g., Martens, Hiralall, & Bradley, 1997; Myers, Simonsen, & Sugai, 2011; Sanetti, Fallon, & Collier-Meek, 2013; Simonsen, Myers, & DeLuca, 2010; Solomon, Klein, & Politylo, 2012). The field needs to identify similarly effective, but more efficient, approaches to supporting teachers’ classroom management. Self-management approaches offer promise as an efficient and effective alternative to expert-driven comprehensive training, as they may be designed to include many of the same elements and implemented in a more efficient manner. Self-management occurs when individuals manage their own behavior as they would manage someone else’s: altering antecedents, learning new behaviors, and arranging consequences to increase the likelihood of a desired behavior (e.g., Cooper, Heron, & Heward, 2007; Skinner, 1953). Using this approach, a teacher self-monitors, self-evaluates, and self-delivers feedback contingent on performance.

One critical classroom management skill that may be increased with self-management is specific praise—a positive statement delivered contingent on desired behavior that names the desired behavior (e.g., teacher saying, “Thanks for raising your hand” immediately after a student raises her hand to get attention). Across decades, researchers have targeted specific praise as a critical classroom management skill because (a) observed rates of specific praise are typically lower than desired (e.g., Reinke et al., 2013; Scott et al., 2011) and (b) higher rates of specific praise have a positive impact on students’ social behavior and academic outcomes (e.g., Chalk & Bizo, 2004; Simonsen et al., 2008; Sutherland, Wehby, & Copeland, 2000). Decades ago, researchers explored using self-management to increase teachers’ specific praise (Workman, Watson, & Helton, 1982), but findings were inconclusive and self-management strategies initially did not gain traction.

More recently, researchers (e.g., Keller, Brady, & Taylor, 2005; Sutherland & Wehby, 2001; Wright, Ellis, & Baxter, 2012) focused on training pre-service, in-service, and early-childhood teachers, respectively, to self-evaluate their use of specific praise. Researchers trained teachers to use specific praise, audio- or video-record portions of their instruction, and review recordings to self-evaluate their use of specific praise. Across studies, results indicate that teachers increased their use of specific praise during self-evaluation. However, participating teachers dedicated additional time outside of school for daily review of recordings, which may limit the feasibility of this approach. A more efficient method could be to use in vivo self-monitoring in conjunction with regular data-based consultation (e.g., Briere, Simonsen, Sugai, & Myers, 2015; Kalis, Vannest, & Parker, 2007), but this still requires an additional time commitment for consultant and consultee.

To explore self-monitoring in the absence of consultation support, Simonsen, MacSuga, Fallon, and Sugai (2013) examined the effects of various self-monitoring techniques (i.e., using a counter, tallying on paper, and completing a rating), employed during instruction, on five middle school teachers’ specific praise rates. Results suggest that frequency count methods (counter or tally) were optimal (associated with highest levels of specific praise, greatest accuracy, or adherence to self-monitoring), and teachers preferred to use the counter; however, not all teachers responded consistently to self-monitoring. Subsequently, Simonsen et al. (2014) worked with four middle school teachers to test the effects of self-monitoring with a counter on teachers’ specific praise rates across class periods and again found that teachers responded inconsistently: Two teachers responded favorably to self-monitoring, but two teachers required further support. For the two teachers who required additional support, Simonsen et al. piloted (but did not experimentally manipulate) targeted professional development (TPD), which included the following empirically supported components: (a) initial goal setting (Duncan, Dufrene, Sterling, & Tingstrom, 2013; Martens et al., 1997); (b) daily self-monitoring (Simonsen et al., 2013), self-evaluation (Keller et al., 2005; Sutherland & Wehby, 2001; Wright et al., 2012), and opportunities for self-reinforcement (Mahoney, Moura, & Wade, 1973); and (c) weekly email prompts (e.g., Greaney et al., 2012). Although this pilot suggested that TPD might result in increases in specific praise, a functional relation was not tested or established.

In sum, researchers have demonstrated the efficacy of intensive professional development approaches in supporting teachers’ classroom management; however, researchers have yet to document the efficacy of more efficient approaches, like TPD. The purpose of the present study was to explore whether TPD results in an immediate and sustained increase in teacher’s use of specific praise. In particular, we addressed the following research question:

Is there a functional relation between the implementation of TPD and teachers’ use of specific praise?

Method

Settings

We initially conducted this study in a suburban New England elementary school (School 1) located in a district with an established relationship with our school of education. School 1 served 319 students in Grades K-5 in a 13:1 student to teacher ratio. The reported ethnicity of students was 80% White, 10% Asian/Pacific Islander, 3% Hispanic, 2% Black, and 5% multiple races; 1% of students were eligible for free or reduced-price lunch; and the school was not eligible for Title I assistance (http://nces.ed.gov/ccd/schoolsearch). During the study year, School 1 scored 94.6 on the school performance index (SPI; scored 0–100 based on average performance on state assessments), and it was identified as an “excelling” school of distinction (School Performance Report [SPR], available on the state’s department of education website).

To determine if similar results would be found in a more diverse setting, we replicated the study in a second New England suburban elementary school (School 2) from a different district with a more diverse demographic profile, which also had an established relationship with our school of education. School 2 served 355 students in Grades K-5 in a 13:1 student to teacher ratio (http://nces.ed.gov/ccd/schoolsearch). The reported ethnicity of students was 32% Black, 25% Asian/Pacific Islander, 23% Hispanic, 15% White, 1% American Indian/Alaskan, and 4% multiple races; 52% of students were eligible for free or reduced-price lunch; and the school was eligible for Title I assistance (http://nces.ed.gov/ccd/schoolsearch). During the study year, this school received an SPI score of 71.3 and was identified as “transitioning” (SPR, available on the state’s department of education website).

Participants

School 1

The school principal initially contacted teachers at School 1 to determine their interest in participating in the present study and shared a list of interested teachers’ names with the first author. We emailed each listed teacher and met individually with each teacher who replied; three teachers provided written and informed consent to participate (i.e., teachers signed a consent form that described procedures, potential benefits and risks, and the US$50 gift card research incentive). During baseline, data collectors confirmed that teachers delivered low rates of specific praise (i.e., less than once per minute), which was the criterion for participation.

Teacher 1 was a female teacher with 15 years of teaching experience; she had a master’s degree and was certified to teach general education kindergarten through eighth grade. She taught all subjects to 19 kindergarten students. Data collectors observed Teacher 1 during whole-group teacher-directed literacy instruction from 9:30 to 9:45 a.m.

Teacher 2 was a female teacher with a bachelor’s degree who was certified to teach general elementary education (Grades K-6). She had 1 year of teaching experience in a private school setting and 1 year in a public school setting. She taught all subject areas to her 17 third-grade students. Data collectors observed during whole-group teacher-directed math instruction from 10:40 to 10:55 a.m.

Teacher 3 was a female teacher with a master’s degree who was certified to teach general education Grades pre-K-3 and special education. At the time of the study, she had been teaching for 11 years in public school and had previously taught for 2 years in a private preschool. She taught all subjects to an inclusive class of 17 third-grade students. Data collectors observed during whole-group teacher-directed literacy instruction from 9:05 to 9:20 a.m.

School 2 participants

In School 2, we presented the research opportunity to grade-level teams and followed up by email and brief meetings with interested teachers. Four teachers provided written and informed consent to participate. One teacher was exited from the study because her baseline rates of specific praise were already high, leaving three teachers to participate in the replication study.

Teacher 4 was a female third-grade teacher with a master’s degree and 7 years of teaching experience. She was certified to teach Grades K-5, and she taught approximately 20 students. Data collectors observed during literacy instruction from 9:45 to 10:00 a.m.

Teacher 5 taught all subjects to a class of 20 kindergarten students. She had 5 years of teacher experience, a master’s degree, and held certification in the area of elementary education. Data collectors observed during small group literacy instruction from 11:35 to 11:50 a.m.

Teacher 6 had 11 years of teaching experience; she held certifications in elementary education and remedial reading/language arts and has a master’s degree. She taught a class of 20 third graders. During the study, three new students arrived and three students left her classroom. Data collectors observed during small group literacy instruction from 10:00 to 10:15 a.m.

Dependent Measures

We collected data on teachers’ specific praise rates (primary dependent variable), fidelity of implementation, and social validity of the TPD intervention.

Specific praise rates

The primary dependent variable was the rate at which teachers delivered specific praise (i.e., specific positive feedback, provided to one or more students, contingent on behavior). Trained behavioral observers conducted daily direct observations of each teacher during selected 15-min segments of teacher-directed instruction (identified by each teacher as his or her most structured literacy or math instruction) and recorded the frequency of specific praise statements during each minute; the observer then converted frequency to rate by dividing the total number of praise statements recorded by the number of minutes observed.

Observer training

The lead data collector, an advanced doctoral candidate with prior experience conducting direct observations, led all training activities for the other doctoral-level data collectors prior to the start of the study. Training consisted of (a) one meeting to introduce the data collection tool and discuss operational definitions, including positive and negative examples, of the behaviors included on the form; (b) several practice sessions with video clips; and (c) several sessions of in vivo training (i.e., observing teachers and children in the classroom with the form) until all behavioral observers exceeded the predetermined criterion (i.e., 85.0%) of inter-observer agreement (IOA) with the lead data collector and with each other.

IOA

Throughout the project, IOA was computed for 36.73% (36/98) of observations in School 1 and 43.33% (39/90) of observations in School 2. These observations were spread throughout baseline (37.65%), TPD (37.93%), and follow-up (66.67%) conditions. We calculated IOA using the mean count-per-interval method (Cooper et al., 2007); that is, we computed IOA within each minute interval (dividing the smaller count by the larger count and multiplying by 100%) and then averaged IOA across intervals for each observation. Across both schools, IOA remained high throughout baseline, TPD, and follow-up conditions, respectively, for Teacher 1 (Mdn = 100.00%, range = 100.00%–100.00%; Mdn = 96.81%, range = 91.67%–100.00%; and Mdn = 100.00%, range = 100.00%–100.00%), Teacher 2 (Mdn = 98.89%, range = 93.33%–100.0%; Mdn = 98.58%, range = 95.33%–100.00%; and Mdn = 100.00%, range = 100.00%–100.00%), Teacher 3 (Mdn = 98.92%, range = 96.67%–100.00%; Mdn = 100.00%, range = 100.00%–100.00%; and Mdn = 96.67%, range = 93.33%–100.00%), Teacher 4 (Mdn = 97.77%, range = 93.30%–100%; Mdn = 98.23%, range = 92.70%–100%; and Mdn = 100.00%, range = 100.00%–100.00%), Teacher 5 (Mdn = 100.00%, range = 100.00%–100.00% for all conditions), and Teacher 6 (Mdn = 96.75%, range = 87.00%–100%; Mdn = 99.05%, range = 94.70%–100%; and Mdn = 100.00%, range = 100.00%–100.00%).

Fidelity of implementation

We measured several dimensions of fidelity (e.g., Dane & Schneider, 1998; Power et al., 2005) throughout this study, including adherence to scripted training, adherence to and accuracy of self-monitoring, and adherence to and accuracy of self-evaluation. Specific indicators are described in the design and procedures section.

Social validity

Teachers completed the TPD Acceptability Questionnaire (TPDAQ), based on the Intervention Rating Profile–15 (IRP-15; Martens, Witt, Elliott, & Darveaux, 1985), to provide descriptive data on the social validity of TPD at the end of the study. Martens et al. (1985) designed the IRP-15, based on the longer IRP (Witt, Martens, & Elliott, 1984), to measure teachers’ acceptability of student-focused behavior interventions. The original 15-item IRP-15 has a one-factor structure, which has been called “general acceptability,” with high internal consistency (Cronbach’s α = .98; Martens et al., 1985). We adapted each student-focused item of the IRP-15 to reflect a teacher-focused intervention targeting classroom management. Like the IRP-15, the TPDAQ prompts teachers to rate 15 items (see Table 1) related to acceptability of TPD on a scale from 1 (strongly disagree) to 6 (strongly agree). The psychometric properties of the TPDAQ have not been established.

Table 1.

Social Validity Ratings on the Targeted Professional Development Acceptability Questionnaire.

Item	Teacher ratings^a
	School 1				School 2
	T1	T2	T3	M	T4	T5	T6	M
1. Targeted professional development was an acceptable intervention for increasing use of specific classroom management skills (i.e., specific praise).	6.0	5.0	6.0	5.7	6.0	6.0	6.0	6.0
2. Most teachers would find targeted professional development appropriate for increasing use of specific classroom management skills (i.e., specific praise).	6.0	5.0	6.0	5.7	6.0	6.0	5.0	5.7
3. Targeted professional development proved effective in increasing use of specific classroom management skills (i.e., specific praise).	4.0	4.0	5.0	4.3	5.0	5.0	5.0	5.0
4. I would recommend the use of targeted professional development to other teachers.	6.0	6.0	5.0	5.7	6.0	6.0	5.0	5.7
5. The classroom management challenges were severe enough to warrant use of targeted professional development.	1.0	3.0	3.0	2.3	6.0	6.0	2.0	4.7
6. Most teachers would find targeted professional development appropriate for increasing use of specific classroom management skills (i.e., specific praise).	5.0	5.0	5.0	5.0	6.0	6.0	5.0	5.7
7. I would be willing to continue using the targeted professional development in the classroom setting.	6.0	6.0	6.0	6.0	5.0	5.0	3.0	4.3
8. Targeted professional development would not result in negative side effects for teachers.	6.0	5.0	6.0	5.7	5.0	6.0	5.0	5.3
9. The targeted professional development would be appropriate for a variety of teachers.	6.0	6.0	6.0	6.0	5.0	6.0	5.0	5.3
10. The targeted professional development is consistent with trainings I have had before in the school setting.	1.0	5.0	3.0	3.0	3.0	2.0	6.0	3.7
11. Targeted professional development is a fair way to increase use of specific classroom management skills (i.e., specific praise).	6.0	6.0	6.0	6.0	5.0	5.0	6.0	5.3
12. Targeted professional development is reasonable for increasing use of specific classroom management skills (i.e., specific praise).	6.0	6.0	6.0	6.0	5.0	6.0	5.0	5.3
13. I liked the procedures used in the targeted professional development.	5.0	6.0	6.0	5.7	5.0	5.0	5.0	5.0
14. Targeted professional development is a good way to increase use of specific classroom management skills (i.e., specific praise).	6.0	5.0	6.0	5.7	5.0	6.0	5.0	5.3
15. Overall, targeted professional development was beneficial for increasing use of specific classroom management skills (i.e., specific praise).	5.0	6.0	6.0	5.7	5.0	6.0	5.0	5.3

Source. Adapted from the Intervention Rating Profile–15; Martens, Witt, Elliott, and Darveaux (1985).

Ratings on a scale from 1 (strongly disagree) to 6 (strongly agree).

Design and Procedures

Within each school, we employed a single-case, multiple baseline design across participants (Horner et al., 2005; Kratochwill et al., 2010). We conducted the initial multiple baseline study in School 1 during fall and winter, and we conducted a replication in School 2 during the winter and spring of the same academic year. The participating teachers in each school progressed through three conditions: baseline, TPD (intervention), and follow-up. We randomly assigned intervention order at each school by drawing numbers out of an opaque container, and we systematically staggered the introduction of the TPD across teachers. We employed the following procedures across both schools.

Baseline condition

During the baseline condition, trained observers recorded the frequency of specific praise during the selected 15-min segment of teacher-directed instruction. No changes were made to the teachers’ typical instructional strategies or routines. Each teacher remained in baseline condition for a minimum of 5 days and until a clear pattern of specific praise was established (i.e., three or more consecutive data points with visual evidence of stability or a counter-therapeutic trend observed during visual analysis).

TPD condition

After we documented a stable pattern of behavior for the three teachers participating within each school, we introduced TPD to the first teacher randomly selected to enter intervention. After her data demonstrated a stable pattern of responding in intervention and data from two teachers in baseline remained stable, we introduced TPD to the second selected teacher. We repeated this process to introduce the intervention to the third selected teacher. We asked teachers not to share information about study procedures with other study participants.

As each teacher entered the TPD condition, the first author conducted a 15- to 20-min meeting to introduce the intervention and provide a scripted training on (a) specific and contingent praise (i.e., definition, summary of supporting research, multiple examples, and written praise statements to use in class) and (b) self-management (i.e., definition and review of specific procedures employed in this study). The first author provided a copy of the training script to the teacher (available upon request from first author), who followed along and wrote her own specific praise statements in the space provided. In addition, the first author explained that the teacher would receive weekly email prompts to remind her to use specific praise and submit her data (either via email or handing a printout to the lead data collector) on a weekly basis.

Then, the first author worked with the teacher to develop a self-management plan based on a template provided at the end of the training materials. The self-management plan prompted the teacher to (a) estimate her current specific praise rate; (b) set a goal (i.e., criterion for self-reinforcement) for her specific praise rate; (c) identify a reinforcer she would self-deliver on days she met her goal; (d) document when she would enter data, determine if her goal was met, and self-reinforce; and (e) identify when she would check email to receive weekly reminders and how she would submit her data weekly (i.e., by emailing or submitting a printout of her spreadsheet). The first author shared that teachers in previous studies had achieved approximately two specific praise statements per minute during intervention conditions. Each teacher selected two as her goal specific praise rate and chose a reinforcer for meeting her goal (e.g., treat or special coffee, soda at lunch, 15-min of reading for pleasure at the end of the day).

Once the self-management plan was developed, the first author demonstrated how to (a) advance and reset the counter (self-monitor); (b) enter data (date, daily count, and minutes) in an Excel spreadsheet, which automatically calculated a rate and updated a graph; and (c) self-evaluate to determine whether her goal was met for the day (i.e., look at value and font of the rate and examine the graph, which included a goal line). In School 1, a scheduling conflict resulted in the second author conducting this portion of the training for Teacher 3. To ensure that each training event was conducted with fidelity, a second member of the research team attended the training and noted whether each element in the script was delivered fully, partially, or not at all. In each case, 100.00% of elements of the script were delivered fully.

Following the training, the teacher implemented the self-management strategies daily (i.e., used the counter to self-monitor, entered data, and self-evaluated), and observers continued to collect data on each teacher’s specific praise rate. In addition, observers noted the fidelity with which each teacher implemented the self-management strategy during the same, daily 15-min segment of instruction observed during baseline. Observers rated the teacher’s adherence to intervention using the following scale: 2 = fully (kept the counter with her and was seen clicking it), 1 = partially (used the counter during some, but not all, of the 15-min segment), or 0 = not at all (was not observed using the counter). Across both schools, Teacher 1 (M = 1.86, range = 0.00–2.00), Teacher 2 (M = 2.00, range = 2.00–2.00), Teacher 3 (M = 1.86, range = 1.00–2.00), Teacher 4 (M = 2.00, range = 2.00–2.00), Teacher 5 (M = 1.94, range = 1.00–2.00), and Teacher 6 (M = 1.78, range = 1.00–2.00) adhered to self-monitoring during most of the TPD condition.

Observers also recorded the number of specific praise statements the teacher counted on the golf counter at the end of each 15-min observation. We calculated the accuracy of self-monitoring using the total percent agreement method (Gast, 2010); that is, we divided the number of agreements (lower frequency count) by the number of opportunities for agreement (higher frequency count) and multiplied by 100%. Across the TPD condition, Teacher 1 (M = 77.51%, range = 37.50%–96.88%), Teacher 2 (M = 80.15%, range = 33.33%–95.65%), Teacher 3 (M = 75.75%, range = 46.88%–96.00%), Teacher 4 (M = 73.81%, range = 50.00%–100.00%), Teacher 5 (M = 83.86%, range = 65.85%–96.77%), and Teacher 6 (M = 77.92%, range = 22.50%–93.75%) achieved acceptable, but variable, agreement with observer-collected data.

In addition, the first author sent teachers scripted weekly email prompts (which included a brief greeting, a reminder about one or more features of specific praise, and a request for teachers to submit their data) at the beginning of each week during the TPD condition. Each teacher shared her data (either via email or by printing out her spreadsheet) throughout her participation in the intervention condition; however, there was some variability in submitting requested data. In School 1, Teachers 1, 2, and 3 submitted data on 83.33% (5 out of 6), 50.00% (2 out of 4), and 50.00% (1 out of 2) of opportunities, respectively. All teachers failed to submit data prior to winter break, and Teacher 2 also missed one other submission immediately after winter break. In School 2, Teachers 4, 5, and 6 submitted data on 66.67% (4 out of 6), 100.00% (5 out of 5), and 66.67% (2 out of 3) of opportunities, respectively. Furthermore, although teachers self-monitored daily (with counts recorded by observers, as previously described), there was some variability in adherence to daily data entry. In School 1, Teachers 1, 2, and 3 entered daily data on 90.48%, 92.86%, and 100.00% of opportunities, respectively. In School 2, Teachers 4, 5, and 6 entered daily data on 41.18%, 100.00%, and 86.67% of opportunities, respectively. Teacher 4 did not share an updated spreadsheet for the last month of data collection.

The accuracy of daily data entry also varied across teachers. To examine the accuracy of daily data entry, we calculated IOA between (a) the teachers’ actual count recorded on her counter at the end of each observation and (b) the count teachers entered into their spreadsheet using the total percent agreement method described previously (Gast, 2010). Agreement was high, but slightly variable across Teachers 1, 2, and 3 in School 1 (M = 93.32%, range = 71.05%–100.00%; M = 98.46%, range = 80.00%–100.00%; and M = 96.53%, range = 85.71%–100.00%; respectively) and Teachers 4, 5, and 6 in School 2 (M = 100.00%, range = 100.00%–100.00%; M = 95.28%, range = 50.00%–100.00%; and M = 76.71%, range = 33.33%–100.00%; respectively). Variable IOA may have resulted from errors related to the counter (e.g., counter advancing accidentally prior to data entry), data entry (e.g., typing the incorrect numerals), or memory (e.g., teacher recording from memory rather than counter).

Follow-up condition

Once we observed stable patterns of responding for all teachers during the intervention phase, all teachers entered the follow-up phase. At the beginning of this phase, the first author (a) emailed each teacher; (b) communicated that she was entering the final follow-up phase of the study, which meant that she would no longer receive weekly email reminders and observers would be coming weekly (instead of daily); and (c) asked her to use the self-management strategy at her own discretion for the remainder of the study (4 weeks in School 1 and 2 weeks in School 2, due to the end of the school year), stating that the teacher could continue to use the self-management strategies daily, periodically (e.g., every other day, once or twice per week), or not at all. The first author also reminded teachers that researchers would schedule a closing meeting and request a final copy of their spreadsheet at that time.

Observers conducted weekly observation probes during the same 15-min segment of teacher-directed instruction. During probes, observers continued to collect data on each teacher’s specific praise rates and record fidelity data for self-monitoring. In addition, observers asked each teacher whether she had been using self-management strategies during the previous week. During the four weekly follow-up probes in School 1, observers noted that Teachers 1 and 3 fully implemented self-monitoring during the first two observations, with moderate agreement between teacher and observer (M = 69.26%, range = 47.62%–90.91%; and M = 69.89%, range = 62.50%–77.27%, respectively), but did not use it during the last two observations. In contrast, observers noted that Teacher 2 used self-monitoring across the four probes with acceptable accuracy (M = 83.07%, range = 64.71%–93.33%). Following each probe, Teacher 1 reported that she did not engage in ongoing self-monitoring; Teacher 2 reported that she did not use the counter during the first week (she reported counting in her head), but she did report using it during the final 3 weeks; and Teacher 3 reported ongoing self-monitoring during the first week, but not the remaining 3 weeks. During the two weekly follow-up probes in School 2, observers noted that Teacher 4 did not use the counter, Teacher 5 used self-monitoring during both probes with acceptable accuracy (M = 81.10%, range = 80.95%–81.25%), and Teacher 6 used self-monitoring during both probes with variable accuracy (M = 56.25%, range = 37.50%–75.00%). When asked at the end of each probe, Teacher 4 reported that she self-monitored during both weeks, Teacher 5 reported that she did not self-monitor during either week, and Teacher 6 reported that she did not self-monitor the first week but did during the second week.

Closing meetings

At the conclusion of the study in each school, the first author met with each teacher for a closing meeting. At this meeting, the teacher (a) completed the social validity measure (the TPDAQ) and a brief demographic questionnaire and (b) received a US$50 gift card to acknowledge her participation. At the end of the meeting, the first author presented a report that summarized all data collected and provided tips for maintaining or improving use of specific classroom management skills. We also requested a final copy of each teacher’s Excel spreadsheet (self-collected data). Although each teacher reported using self-monitoring to varying degrees, only Teacher 6 entered data into her spreadsheet during follow-up.

Analysis

Traditionally, the results of single-case design studies are analyzed visually (Horner et al., 2005; Kratochwill et al., 2010). To establish a functional relation, researchers visually examine the stability, level, and trend within and across phases to determine whether clear improvements (or changes) in the dependent variable occur with the introduction (or manipulation) of the independent variable at three points in time (Horner et al., 2005; Kratochwill et al., 2010). To supplement visual analysis, we calculated the percentage of non-overlapping data (PND; Scruggs, Mastropieri, & Castro, 1987), a non-parametric estimate of effect size. To interpret PND, Scruggs and Mastropieri (1998) suggested that PND scores above 90% represent very effective, between 70% and 90% represent effective, between 50% and 70% represent questionable, and below 50% represent ineffective interventions.

Results

Teachers’ Specific Praise Rates Across Conditions

In general, participating teachers in both schools demonstrated low and fairly stable specific praise rates during baseline and increased their specific praise rates during TPD. In School 1 (see Figure 1), the school’s winter break occurred during the TPD phase, prior to the final three data points for each teacher (represented by a hash mark on the abscissa and continuity break in each data path), and teachers maintained or increased praise rates after the break. During follow-up, teachers’ specific praise rates were somewhat lower, and two teachers’ data were somewhat variable; however, one teacher’s specific praise became more consistent. In School 2 (see Figure 2), specific praise rates also increased during TPD, relative to baseline, but decreased during the follow-up phase (during final weeks of the school year).

Figure 1.

School 1 teachers’ specific praise rate across baseline, targeted professional development, and follow-up phases.

Figure 2.

School 2 teachers’ specific praise rate across baseline, targeted professional development, and follow-up phases.

School 1

Teacher 1

Teacher 1’s specific praise rates were somewhat variable during her 8 days of baseline (Mdn = 0.27, range = 0.13–0.93 specific praise statements per min). When she entered TPD, her specific praise rates demonstrated an immediate and sustained level change (Mdn = 1.50, range = 1.07–2.33). No data points during the TPD phase overlapped with baseline data (100.00% PND), indicating a very effective intervention. Teacher 1 moved to follow-up after 22 days of TPD; her data showed a moderate decrease in level and some variability (Mdn = 1.00, range = 0.67–1.33).

Teacher 2

Teacher 2’s specific praise rates were low and stable throughout her 16 days in the baseline condition (Mdn = 0.13, range = 0.00–0.33 specific praise statements per min). Her specific praise rates increased in trend immediately after her introduction to TPD and remained high but somewhat variable throughout the TPD condition (Mdn = 1.33, range = 0.27–2.27). There was only one data point during the TPD phase that overlapped with baseline data (92.86% PND), indicating a very effective intervention. After 14 days of TPD, Teacher 2 moved to follow-up, and her specific praise rates decreased, but became more stable; however, the final three data points demonstrated a decreasing trend (Mdn = 0.90, range = 0.73–1.13).

Teacher 3

During 19 days of baseline, Teacher 3 demonstrated low but somewhat variable specific praise rates (Mdn = 0.40, range = 0.00–1.00 specific praise statements per min). Upon entering TPD, her praise rates immediately increased in level and her data remained stable throughout (Mdn = 1.40, range = 1.00–1.67). There was one data point during the TPD phase that overlapped with baseline data (85.71% PND), indicating an effective intervention. When Teacher 3 moved into follow-up, after 7 days of TPD, her data generally decreased in level and became less stable (Mdn = 0.83, range = 0.27–1.13).

School 2

Teacher 4

Teacher 4’s specific praise rates were low with a stable to slightly increasing trend during her 8 days in baseline (Mdn = 0.43, range = 0.33–0.60). Teacher 4 demonstrated a moderate immediate and sustained change in level upon entering TPD; her specific praise rates were moderately variable during this phase (Mdn = 0.93, range = 0.47–1.40). There was only one overlapping data point between intervention and baseline phases (88.24% PND), indicating an effective intervention. Teacher 4 moved to follow-up after 17 days of intervention; her use of specific praise decreased and data remained variable (Mdn = 0.50, range = 0.27–0.73).

Teacher 5

Teacher 5’s specific praise rates were low and somewhat variable during her 15 days in baseline (Mdn = 0.33, range = 0.00–0.73). Upon entering TPD, she demonstrated an immediate and sustained increase in her use of specific praise (Mdn = 2.00, range = 0.87–2.73). There were no overlapping data points between intervention and baseline phases (100.00% PND), indicating a very effective intervention. Teacher 5 received 16 days of TPD. During follow-up, Teacher 5’s use of specific praise declined but remained above baseline levels (Mdn = 1.13, range = 0.87–1.40).

Teacher 6

During her 19-day baseline, Teacher 6 demonstrated a low and slightly declining trend of specific praise statements per minute (Mdn = 0.20, range = 0.00–0.40). She increased her use of specific praise statements immediately upon entering TPD and sustained that increase throughout (Mdn = 0.87, range = 0.60–1.13). There were no overlapping data points between baseline and intervention phases (100.00% PND), indicating a very effective intervention. She moved to follow-up after 9 days of TPD and her use of specific praise returned to baseline rates (Mdn = 0.20, range = 0.20–0.20)

In sum, visual analysis supports a functional relation between implementation of TPD and increases in teachers’ specific praise rates, with three demonstrations of effect demonstrated at three points in time in the first school and a replication of effects in the second school. Furthermore, there was minimal overlap of data between TPD and baseline conditions, with PND indicating an effective or very effective intervention for all teachers. Effects did not maintain during follow-up phase; specific praise rates were lower and more variable across teachers.

Acceptability of TPD

All teachers completed the TPDAQ, adapted from the IRP-15, to provide ratings, on a scale of 1 (strongly disagree) to 6 (strongly agree), to describe their perceptions of intervention acceptability (see Table 1). In general, teachers rated all items pertaining to acceptability high (5 or 6), with the exception of Item 3 (related to whether the intervention “proved effective”), which received a slightly lower rating (i.e., 4) from two teachers in School 1. When asked whether classroom management challenges were significant enough to warrant TPD, teachers in School 1 provided relatively neutral ratings (3) and one teacher strongly disagreed (1); in School 2, one teacher disagreed (2). Perhaps, because we did not recruit teachers on the basis of classroom management difficulties, some teachers did not believe they had significant challenges with classroom management prior to participation. Furthermore, although two teachers (2 and 6) rated TPD as consistent with previous trainings (ratings of 5 or 6 on Item 10), four teachers (1, 3, 4, and 5) indicated that TPD was inconsistent with prior trainings and rated the item 1, 2, or 3 (strongly disagree to neutral).

Discussion

Discussion of Study Results

Across two elementary schools with different demographic profiles, participating teachers increased their specific praise rates when receiving TPD. Previous researchers had documented (a) inconsistent effects when teachers engaged in self-monitoring in the absence of additional support (Simonsen et al., 2014; Simonsen et al., 2013) and (b) positive effects when teachers employed self-management strategies that required additional time or support outside of instruction, including self-evaluating video or audio recordings (Keller et al., 2005; Sutherland & Wehby, 2001; Wright et al., 2012) or in vivo self-monitoring with data-based consultation (Briere et al., 2015; Kalis et al., 2007). The present study highlights the potential efficacy of an efficient TPD approach (self-management with weekly email reminders), which required minimal time outside of instruction. Furthermore, teachers found TPD acceptable; for example, Teacher 2 commented, “This study was extremely helpful in terms of modifying my instruction. As a first-year teacher, I felt both my students and I benefited.” In sum, data indicate TPD may be an effective and user-friendly intervention to increase teachers’ use of evidence-based practices.

Although teachers’ specific praise rates increased while they implemented TPD, most teachers’ specific praise deteriorated during the follow-up phase: follow-up data overlapped with baseline data for four (i.e., Teachers 1, 3, 4, and 6) of six teachers. This may have been due to lack of time spent in intervention, rapid withdrawal of reinforcement, or other competing factors. First, teachers spent between 7 (Teacher 3) and 22 (Teacher 1) days in intervention, and teachers may have required more time (or opportunities to practice) to build fluency with the self-management strategy, use of specific praise, or both skills than was allotted in the current study. Descriptively, teachers who spent more time in intervention (e.g., Teachers 1, 2, 4, and 5) may have maintained effects at slightly higher levels than teachers who spent less time in intervention (e.g., Teachers 3 and 6). It is also possible that withdrawing self-management supports, including self-delivered reinforcement, without careful fading, resulted in ratio strain and related decreases in specific praise (e.g., Cooper et al., 2007). Finally, it is possible that other factors (e.g., transition back from winter break, end of the school year, schedule of prompting, and level of “buy in” for higher specific praise rates) interfered with maintenance of effects. Future research is needed to explore maintenance and generalization of intervention effects.

Limitations

There are two main limitations to the present study. First, although TPD was designed to be an efficient professional development (PD) approach, researchers, rather than natural implementers, provided the brief training and weekly prompts (external supports) during the TPD phase and collected direct observation data, which may have resulted in reactivity. Different results may have occurred if natural implementers (e.g., mentors, coaches, administrators) had provided supports or collected data. Second, although data collectors made efforts to observe in a consistent environment throughout study conditions, schedule disruptions and other variations typical in an applied environment may have affected study results.

Implications

Although preliminary, this study suggests that TPD may be an efficient and effective approach to increase teachers’ use of specific praise and may be one way to augment existing PD efforts at the pre- and in-service levels. However, given that teachers’ specific praise rates decreased when they moved into follow-up phases, research is needed to develop and study strategies to promote maintenance and generalization of effects (e.g., careful fading of supports). In addition, further research is needed to explore (a) whether similar effects would be found across contexts (e.g., types of educators, locations, school sizes, school levels) and teacher practices (e.g., opportunities to respond, prompts), (b) impacts of natural implementers providing TPD in lieu of researchers, and (c) effects of changes in teacher behavior(s) on student outcomes.

Footnotes

Action Editor: Dan Maggin

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The development of this paper was supported in part by a grant from the Office of Special Education Programs, U.S. Department of Education (H029D40055). Opinions expressed herein are the author’s and do not reflect necessarily the position of the US Department of Education or the University of Connecticut, and such endorsements should not be inferred.

References

Allen

C. T.

Forman

S. G.

(1984). Efficacy of methods of training teachers in behavior modification. School Psychology Review, 13, 26–32.

Begeny

J. C.

Martens

B. K.

(2006). Assessing pre-service teachers’ training in empirically-validated behavioral instruction practices. School Psychology Quarterly, 21, 262–285. doi:10.1521/scpq.2006.21.3.262

Briere

D. E.

Simonsen

Sugai

Myers

(2015). Increasing new teachers’ specific praise rates using a within-school consultation intervention. Journal of Positive Behavior Interventions, 17, 50–60. doi:10.1177/1098300713497098

Chalk

Bizo

L. A.

(2004). Specific praise improves on-task behavior and numeracy enjoyment: A study of year four pupils engaged in numeracy hour. Educational Psychology in Practice, 20, 335–351.

Cooper

J. O.

Heron

T. E.

Heward

W. L.

(2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Prentice Hall.

Dane

A. V.

Schneider

B. H.

(1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45. doi:10.1016/S0272-7358(97)00043-3

Duncan

N. G.

Dufrene

B. A.

Sterling

H. E.

Tingstrom

D. H.

(2013). Promoting teachers’ generalization of intervention use through goal setting and performance feedback. Journal of Behavioral Education, 22, 325–347. doi:10.1007/s10864-013-9173-5

Fixsen

D. L.

Naoom

S. F.

Blase

K. A.

Friedman

R. M.

Wallace

(2005). Implementation research: A synthesis of the literature (FMHI Publication #231). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.

Freeman

Simonsen

Briere

D. E.

MacSuga-Gage

A. S.

(2014). Pre-service teacher training in classroom management: A review of state accreditation policy and teacher preparation programs. Teacher Educational and Special Education, 37, 106–120. doi:10.1177/0888406413507002

10.

Gast

D. L.

(Ed.). (2010). Single subject research methodology in behavioral sciences. New York, NY: Routledge.

11.

Greaney

M. L.

Sprunck-Harrild

Bennett

G. G.

Puleo

Haines

Viswanath

K. V.

Emmons

K. M.

(2012). Use of email and telephone prompts to increase self-monitoring in a web-based intervention: Randomized controlled trial. Journal of Medical Internet Research, 14(4), 76–86. doi:10.2196/jmir.1981

12.

Horner

R. H.

Carr

E. G.

Halle

McGee

Odom

Wolery

(2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179.

13.

Ingersoll

R. M.

Smith

T. M.

(2003). The wrong solution to the teacher shortage. Educational Leadership, 60, 30–33.

14.

Kalis

T. M.

Vannest

K. J.

Parker

(2007). Praise counts: Using self-monitoring to increase effective teaching practices. Preventing School Failure, 51, 20–27.

15.

Keller

C. L.

Brady

M. P.

Taylor

R. L.

(2005). Using self-evaluation to improve student teacher interns’ use of specific praise. Education and Training in Developmental Disabilities, 40, 368–376.

16.

Kratochwill

T. R.

Hitchcock

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2010). Single-case designs technical documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf

17.

Mahoney

M. J.

Moura

N. G.

Wade

T. C.

(1973). Relative efficacy of self-reward, self-punishment, and self-monitoring techniques for weight loss. Journal of Consulting and Clinical Psychology, 40, 404–407. doi:10.1037/h0034565

18.

Martens

B. K.

Hiralall

A. S.

Bradley

T. A.

(1997). A note to teacher: Improving student behavior through goal setting and feedback. School Psychology Quarterly, 12, 33–41. doi:10.1037/h0088945

19.

Martens

B. K.

Witt

J. C.

Elliott

S. N.

Darveaux

D. X.

(1985). Teacher judgments concerning the acceptability of school-based interventions. Professional Psychology: Research and Practice, 16, 191–198. doi:10.1037/0735-7028.16.2.191

20.

Myers

Simonsen

Sugai

(2011). Increasing teachers’ use of praise with a response to intervention approach. Education & Treatment of Children, 34, 35–59.

21.

National Joint Committee on Learning Disabilities. (2000). Professional development for teachers. Learning Disability Quarterly, 23, 2–6.

22.

Power

T. J.

Bloom-Hoffman

Clarke

A. T.

Riley-Tillman

Kelleher

Manz

P. H.

(2005). Reconceptualizing intervention integrity: A partnership-based framework for linking research with practice. Psychology in the Schools, 42, 495–507.

23.

Reinke

W. M.

Herman

K. C.

Stormont

(2013). Classroom-level positive behavior supports in schools implementing SW-PBIS: Identifying areas for enhancement. Journal of Positive Behavior Interventions, 15, 39–50. doi:10.1177/1098300712459079

24.

Reinke

W. M.

Stormont

Herman

K. C.

Puri

Goel

(2011). Supporting children’s mental health in schools: Teacher perceptions of needs, roles, and barriers. School Psychology Quarterly, 26, 1–13. doi:10.1037/a0022714

25.

Sanetti

L. M. H.

Fallon

L. M.

Collier-Meek

M. A.

(2013). Increasing teacher treatment integrity through performance feedback provided by school personnel. Psychology in the Schools, 50, 134–150. doi:10.1002/pits.21664

26.

Scott

T. M.

Alter

P. J.

Hirn

R. G.

(2011). An examination of typical classroom context and instruction for students with and without behavior disorders. Education & Treatment of Children, 34, 619–641. doi:10.1353/etc.2011.0039

27.

Scruggs

T. E.

Mastropieri

M. A.

(1998). Summarizing single-subject research: Issues and applications. Behavior Modification, 22, 221–242.

28.

Scruggs

T. E.

Mastropieri

M. A.

Castro

(1987). The quantitative synthesis of single subject research: Methodology and validation. Remedial and Special Education, 8, 24–33.

29.

Simonsen

Fairbanks

Briesch

Myers

Sugai

(2008). A review of evidence based practices in classroom management: Considerations for research to practice. Education & Treatment of Children, 31, 351–380.

30.

Simonsen

MacSuga

A. S.

Fallon

L. M.

Sugai

(2013). Teacher self-monitoring to increase specific praise rates. Journal of Positive Behavior Interventions, 15, 3–13.

31.

Simonsen

MacSuga-Gage

A. S.

Briere

D. E.

Freeman

Myers

Scott

Sugai

(2014). Multi-tiered support framework for teachers’ classroom management practices: Overview and case study of building the triangle for teachers. Journal of Positive Behavior Interventions, 16, 179–190. doi:10.1177/1098300713484062

32.

Simonsen

Myers

DeLuca

(2010). Providing teachers with training and performance feedback to increase use of three classroom management skills: Prompts, opportunities to respond, and reinforcement. Teacher Education in Special Education, 33, 300–318. doi:10.1177/0888406409359905

33.

Skinner

B. F.

(1953). Science and human behavior. New York, NY: MacMillan.

34.

Smith

T. M.

Ingersoll

(2004). What are the effects of induction and mentoring on beginning teacher turnover. American Education Research Journal, 41, 681–714.

35.

Solomon

B. G.

Klein

S. A.

Politylo

B. C.

(2012). The effect of performance feedback on teachers’ treatment integrity: A meta-analysis of the single case literature. School Psychology Review, 41, 160–175.

36.

Sutherland

K. S.

Wehby

J. H.

Copeland

S. R.

(2000). Effects of varying rates of behavior-specific praise on the on-task behavior of students with EBD. Journal of Emotional and Behavioral Disorders, 8, 2–8.

37.

Sutherland

K. S.

Wehby

J. H.

(2001). The effect of self-evaluation on teaching behavior in classrooms for students with emotional and behavioral disorders. The Journal of Special Education, 35, 2–8.

38.

Wei

R. C.

Darling-Hammond

Adamson

(2010). Professional development in the United States: Trends and challenges. Dallas, TX: National Staff Development Council.

39.

Witt

J. C.

Martens

B. K.

Elliott

S. N.

(1984). Factors affecting teachers’ judgments of the acceptability of behavioral interventions: Time involvement, behavior problem severity, and type of intervention. Behavior Therapy, 15, 204–209.

40.

Workman

E. A.

Watson

P. J.

Helton

G. B.

(1982). Teachers’ self-monitoring of praise vs. praise instructions: Effects on teachers’ and students’ behavior. Psychological Reports, 50, 559–565.

41.

Wright

M. R.

Ellis

D. N.

Baxter

(2012). The effect of immediate or delayed video-based teacher self-evaluation on Head Start teachers’ use of praise. Journal of Research in Childhood Education, 26, 187–198. doi:10.1080/02568543.2012.65774