Abstract
This study examined the effects of in-service training with performance feedback on preschool teachers’ use of classroom preventive practices. Three practices designed to prevent challenging behavior were selected: transition preparations, rule reminders, and social-emotional teaching strategies. Following a brief training on each practice, teaching teams received email feedback on their use of the practice and the effects on child behavior. A multiple baseline design across practices replicated across two teaching teams was used. There were increases in all three preventive practices following training and feedback for both teaching teams. The effects on child behavior were mixed.
As more children with challenging behavior enter early childhood settings, educators describe preventing and responding to challenging behavior as an important training priority (Hemmeter, Corso, & Cheatham, 2006). The exact prevalence of the behaviors eliciting this need has been difficult to determine. Estimates of the number of children with persistent, ongoing challenging behavior have ranged from 10% to nearly 40% of the preschool population (Campbell, 1995; Kaiser, Cai, Hancock, & Foster, 2002; Lavigne et al., 1996). Furthermore, preschool children are expelled from early care and education settings for behavior problems at a rate 5 times that of school-aged children (Gilliam, 2005).
Several strategies have been identified in the early childhood literature to address and prevent these negative outcomes. One promising approach, the Pyramid model, offers a three-tiered system of support for children in early childhood settings (Fox, Dunlap, Hemmeter, Joseph, & Strain, 2003; Hemmeter, Ostrosky, & Fox, 2006). This model consists of universal strategies, secondary supports, and intensive individualized interventions. Universal strategies are designed to promote positive outcomes for all children. At this level, teachers ensure (a) nurturing and responsive relationships with children, families, and colleagues and (b) high-quality, supportive environments. Secondary supports are designed to systematically teach social skills and support children who are at risk for problem behavior. Finally, intensive individualized supports are geared toward providing function-based intervention for the small number of children with severe, persistent challenging behavior. Table 1 shows specific practices associated with each level of the Pyramid.
Examples of Pyramid Model Practices.
Source. Adapted from Fox, Hemmeter, Snyder, Binder, and Clarke (2011). The final, definitive version of this table has been published by the Hammill Institute on Disabilities and Sage Publications Ltd. All rights reserved. © 2011.
In the current study, three strategies were drawn from the universal and secondary levels of the Pyramid: transition preparations, rule reminders, and social-emotional teaching strategies. Although social-emotional teaching strategies can be considered a secondary support in the Pyramid model, early childhood teachers are encouraged to teach, recognize, reinforce, and encourage social skills for all children. For this reason, social-emotional teaching strategies can also be considered a preventive practice (Greenberg et al., 2003; Webster-Stratton & Taylor, 2001).
These three strategies represent only a subset of the Pyramid model. They were chosen for four reasons. First, each of these strategies is a recommended practice with a strong research base. Transition preparation strategies help children follow directions, understand the flow of the day, and engage in common classroom routines (Dettmer, Simpson, Myles, & Ganz, 2000; Sainato, 1990; Schmit, Alper, Raschke, & Ryndak, 2000). Rule reminders are a way of intentionally teaching and reinforcing a small number of classroom expectations. Teaching, reinforcing, and precorrecting a small number of rules is a major component of all positive behavior support systems (PBS) and has been evaluated with children of all ages within the PBS framework (Sugai & Horner, 2002). Social-emotional teaching strategies are effective ways to address deficits in language and social skills that are common in young children who engage in challenging behavior (Birch & Ladd, 1998; Webster-Stratton & Reid, 2004; Webster-Stratton & Taylor, 2001). Second, the strategies are applicable to a variety of situations across the day. Third, they can be used with large groups, small groups, or individual children. Finally, there is evidence from a number of studies that teachers are unlikely to use these strategies in the absence of training on the Pyramid model (Artman, 2010; Hemmeter, Snyder, & Fox, 2009, 2010).
To respond to teachers’ professional development needs around these promising strategies, systematic training and support is necessary. Pyramid model training materials have been widely disseminated through the Center on the Social and Emotional Foundations for Early Learning (CSEFEL; www.vanderbilt.edu/csefel). There is very little research, however, on how best to support teachers and teams as they learn to use these new strategies. Much of the research around the Pyramid model has focused on training teachers to use a single classwide strategy (e.g., praise in Hemmeter, Snyder, Kinder, & Artman, 2011) or individualized behavior support plans (e.g., Duda, Dunlap, Fox, Lentini, & Clarke, 2004). Fox, Hemmeter, Snyder, Binder, and Clarke (2011) provided the only published demonstration of professional development around multiple components of the Pyramid model. They used the CSEFEL modules to provide 3 days of training to three Early Childhood Special Education teachers. This was followed by up to 14 biweekly observation and feedback sessions in the classroom. Using a multiple baseline across participants design, these researchers found a functional relation between the professional development intervention and teachers’ use of Pyramid practices across all levels of the Pyramid.
The professional development model provided by Fox and colleagues (2011) exemplified the call for professional development that is sustained, intensive, and focused on practice (Snyder, Hemmeter, & McLaughlin, 2011). It was built around several decades of research on providing in-classroom performance feedback about early childhood teaching practices (e.g., Bruder & Nikitas, 1992; Casey & McWilliam, 2008; Kaiser, Ostrosky, & Alpert, 1993; Mudd &Wolery, 1987; Schepis, Reid, Ownbey, & Parsons, 2001; Venn & Wolery, 1992). It also built on performance feedback studies designed to improve teachers’ use of specific behavior support practices like precorrections (Benedict, Horner, & Squires, 2007; Stormont, Smith, & Lewis, 2007), attention to children’s positive behaviors (Cooper, Thomson, & Baer, 1970; Fullerton, Conroy, & Correa, 2009; Hemmeter et al., 2011), positive social interactions (Hendrickson, Gardner, Kaiser, & Riley, 1993), appropriate directions (Hirallal & Martens, 1998), and implementation of function-based interventions (Noell et al., 2005). Despite the evidence that job-embedded performance feedback is effective for improving early childhood teaching practices, such models can be difficult to implement when time, financial, or personnel resources are limited. Research is needed to explore efficient and effective forms of professional development.
As technology becomes more available in early care and education settings and web-based communication options grow, the Internet is becoming a forum for professional development and a mechanism through which performance feedback might be delivered. To date, however, only four studies were found that examined the use of technology for providing performance feedback to early childhood teachers. Two of these studies provided feedback via email (Barton & Wolery, 2007; Hemmeter et al., 2011). Barton and Wolery (2007) used email feedback to increase preservice teachers’ use of verbal behaviors such as descriptive praise and expansions. A multiple baseline design across teachers was used. Each email was delivered weekly and contained specific elements: (a) verbatim examples of the target behavior observed during the session, (b) total frequency of target behaviors, and (c) an embedded response prompt. Email feedback was associated with increases in expansions and descriptive praise for preservice teachers. No data were collected on child outcomes. Hemmeter and colleagues (2011) used a multiple probe across participants design to evaluate the effects of email feedback on Head Start teachers’ use of descriptive praise during large group activities. Following a brief training, email feedback was delivered an average of 3 times per week. Each email followed a five-step protocol and included a link to watch a video example of teachers using descriptive praise. Email feedback was associated with increases in teachers’ use of descriptive praise, but effects on child challenging behavior were mixed.
Two additional studies used larger systems of web-mediated performance feedback to evaluate the use of technology in professional development. Pianta, Mashburn, Downer, Hamre, and Justice (2008) provided public preschool teachers with online professional development around specific dimensions of teacher–child interactions. Teachers viewed videos of their own practice, received specific written feedback related to their interactions with children, and participated in web chats with trained coaches. Relative to a control group that only had access to video examples of high-quality interactions, these teachers showed significantly more improvement in the Teacher Sensitivity, Instructional Learning Format, and Language Modeling dimensions of the Classroom Assessment Scoring System (Pianta, LaParo, & Hamre, 2008).
A similar coaching protocol was used to increase teachers’ use of literacy practices (Powell, Diamond, Burchinal, & Koehler, 2010). For 15 weeks, teachers videotaped their literacy instruction and mailed the tapes to a trained coach. Like the My Teaching Partner (MTP) intervention, the coach selected segments to share with the teacher and provided written feedback to accompany the clips. Videos and accompanying feedback were burned to a compact disc and mailed to the teacher every 2 weeks. The project compared the effects of this distance coaching model with weekly live coaching in the classroom. There were statistically significant improvements in general classroom environments as measured by the Early Language and Literacy Classroom Observation (Smith, Dickinson, Sangeorge, & Anastasopoulos, 2002) and on teachers’ code-focused instruction for both groups. There were not, however, statistically significant differences between the distance and live coaching groups. This indicates that web-mediated professional development can be an efficient alternative to intensive, on-site professional development. Additional research is necessary, however, to help “unpack” the active ingredients (Snyder et al., 2011) and to help understand the circumstances in which it can be most effective.
This study explored the effects of email feedback on teachers’ use of preventive practices and children’s challenging behavior. It was designed to address several gaps in the literature. First, no studies have evaluated alternatives to the traditional 3-day training on the Pyramid model. Such alternatives are already in use by practitioners in the field (i.e., “train-coach-train” models), but they have not been studied. Second, no studies have examined the effects of classwide prevention strategies on the behavior of target children. This is critical because there is evidence that these strategies may reduce challenging behavior and thereby reduce the need for intensive, tertiary interventions (Fullerton et al., 2009; Hemmeter et al., 2011; Stormont et al., 2007; Webster-Stratton, Reid, & Stoolmiller, 2008). The purpose of this study was to investigate the effects of a brief in-service training program followed by email feedback on preschool teachers’ use of three classroom preventive practices. A multiple baseline across behaviors design was used to address the following research questions:
Research Question 1: Does in-service training with email feedback increase preschool teachers’ use of classroom preventive practices?
Research Question 2: Are targeted preventive practices associated with a decrease in instances of challenging behavior for a preschool child identified with mild problem behavior?
Method
Participants and Setting
Two preschool teaching teams participated in this project. They were recruited from a university-based inclusive child care setting in the southern United States. The child care setting served children with and without disabilities from 10 months to 5 years. All children in the participating classrooms were between 3 and 4 years of age. Both classrooms had 13 children. Approximately 25% of the children in each class had Individualized Education Programs (IEPs). Early interventionists and therapists provided services inside the classroom. The student population of both classrooms was approximately 54% European American, 23% African American, and 23% Asian American. Students varied widely in socioeconomic background. Some students were children of university faculty and staff; others were enrolled through a large metropolitan school district.
Teaching teams
Each teaching team consisted of a lead teacher and an assistant. Helen was the lead teacher in Teaching Team 1. She was an African American female with 5 years teaching experience and a bachelor’s degree in psychology. She had worked in her current position for 3 years. Troy, the assistant teacher, was an African American male with 11 years teaching experience and a bachelor’s degree in theology. He had worked in the child care center for 4 years. Although Helen and Troy had both worked in the center for several years, this was their 1st year working together as a teaching team.
Teaching Team 2 consisted of Lucy and Ramona. Lucy, the lead teacher, was a European American female with 13 months of teaching experience. She had a bachelor’s degree in early childhood education and was pursuing a master’s degree in early childhood special education. Ramona was an African American female with 25 years of teaching experience. She had been working in the child care center for 13 of those years. Lucy and Ramona had been working together as a teaching team for 1 year.
Children
Each teaching team was asked to identify one child who exhibited persistent, mild challenging behavior. Mild behaviors included crying, not following directions, using objects inappropriately, and wandering around the room. Children with more intense or severe challenging behavior were not included because these children typically require more individualized and systematic support than this study was designed to provide. Parents were asked permission for their child to participate. Helen and Troy identified Lily, a 3.5-year-old African American girl diagnosed with Down’s syndrome. Lucy and Ramona identified Ricky. Ricky was a 4-year-old European American boy diagnosed with a speech and language delay with characteristics of autism. Both children had Individualized Education Plans and received special education services including speech and language therapy and occupational therapy.
Characteristics of classrooms
The Teaching Pyramid Observation Tool (TPOT; Hemmeter, Fox, & Snyder, 2008) was administered to determine the extent to which teaching teams were using positive behavior support strategies at baseline. It was a descriptive measure only; it was not used to select participants or intervention targets. The TPOT was collected live during a 2-hr observation and brief interview with each teaching team. It contains 141 indicators related to Pyramid model practices. A total of 21 indicators provided specific data on the strategies targeted in this study: teacher’s use of rule reminders, transition preparations, and social-emotional strategies prior to beginning the study. Teaching Team 1 (Helen and Troy) implemented 59.8% of all TPOT indicators prebaseline. They implemented 4 of 8 indicators associated with transitions, 0 of 7 indicators associated with behavior expectations and rules, and 2 of 8 indicators associated with teaching social skills. Teaching Team 2 (Lucy and Ramona) implemented 64.2% of all TPOT indicators prebaseline. They implemented 4 of 8 indicators associated with transitions, 0 of 7 indicators associated with behavior expectations and rules, and 2 of 8 indicators associated with teaching social skills.
Experimental Design
A multiple baseline design across behaviors and replicated across teaching teams was used to evaluate the effects of a brief training with email feedback. The multiple baseline design was used to examine the effects of training and feedback on three types of classroom preventive practices: transition preparation, rule reminding, and social-emotional teaching. The experiment was replicated with a second preschool team. Condition changes were made based on the teaching team’s use of each classroom preventive practice. During formative data analysis, a team total of preventive practices was used for making condition change decisions. Therefore, data decisions were based on data summed across teachers within a classroom for each preventive practice. The stability of baseline data was assessed through visual analysis. Stability decisions were made when there was an absence of shifts in level or trend of graphed data. For convenience, preventive practices were introduced in the same order for each classroom: Training on transition preparations occurred first, training on rule reminders occurred second, and training on social-emotional teaching strategies occurred third.
Procedure
Behavior recording system
With the exception of pre- baseline descriptive measures, which were collected live in the classroom, all data were collected via video recording on a Sony SR85 digital camcorder. The camcorder was mounted to a tripod and placed in a quiet corner of the room between the cubbies (where children hung up their belongings) and the listening center. This position allowed the entire classroom to be captured on film. The research assistant or first author set up the camera and remained in the room at all times while filming was occurring. Filming occurred only during the target activity and lasted throughout the entire target activity. When necessary, the videographer adjusted the angle of the camera so the target child in each classroom was visible at all times. Occasionally the adults were not visible, but their voices were audible at all times. Reactivity to filming procedures was minimized by acclimating teachers and children to the video recording equipment before the study began. The first author introduced the camera to the children prior to baseline data collection. The first author or research assistant set up the camera in each classroom for 1 to 2 days before data collection began.
Timed event recording was used to collect observational data of teacher and teaching assistant behavior. Each instance of one of the three preventive practices was marked in the data stream. A 10-s partial interval system was used to estimate the frequency of target children’s challenging behavior. The coder indicated the presence of child challenging behavior at any time during a 10-s interval. Data were collected from videotaped observations of the classroom. A research assistant or the first author conducted all videotaping session. Once the data were collected on digital videotapes, a research assistant uploaded the videos to a computer. The first author conducted all coding using ProcoderDV software (Tapp & Walden, 1993). This software allowed multiple viewings, so each teacher and child behavior could be coded on separate passes through the video. A second research assistant coded 33% of videotapes. The total frequency/occurrence of teacher behavior and the estimated frequency of challenging behavior were assessed.
Baseline
Baseline observations took place during typical classroom activities and routines. Teachers were asked to identify one time during the day when problem behaviors were likely to occur. Teaching Team 1, Helen and Troy, chose center time as their primary data collection activity. Center time occurred 45 min each day. Children were assigned to groups that rotated between two teacher-led small group activities and a third activity that was free play in a center (blocks, home living).
Teaching Team 2, Lucy and Ramona, chose large group circle time as the primary data collection activity. Circle time lasted approximately 25 min each day. It was led by Ramona, whereas Lucy prepared center-time activities and supported the group activity as needed. Both teachers were present throughout all observations and contributed data. Circle time consisted of a welcome song, calendar, choice of finger play songs, story, and transition.
Training
Following baseline, each teaching team participated in a brief training meeting. Trainings occurred separately for each teaching team. The first author conducted the training for both teaching teams. The first author was a licensed early interventionist with training and experience in positive behavior support. The first author met with the teacher and teaching assistant in their classroom or a small conference room at a time that was convenient to them. Training occurred separately for transition preparations, rule reminders, and social-emotional strategies. For convenience, the trainings were provided in the same order for each team. On average, trainings were 9 school days apart (range = 5–14). Trainings were not scheduled until data in the previous condition (i.e., baseline or previously trained skill) were stable for at least three sessions. At each training, the first author presented a brief PowerPoint presentation about the targeted preventive practice. These presentations were adapted from existing CSEFEL modules. Each training meeting followed a protocol. First, the presentation included a definition of the practice, rationale, examples and nonexamples of the practice, and sample materials. Second, the teaching team brainstormed ways to integrate the practice into their routines. Throughout the meeting, the first author answered any questions the teaching team had about using the practice in their classroom. Finally, the teaching team made a plan for how the strategies would be used in the classroom. The first author provided resources the teachers needed, including visual rule posters, visual schedules, and scripted stories for each teaching team as per their plans. Each training lasted approximately 1 hr.
Email feedback
On the morning following each training meeting and an average of 2.2 times per week thereafter, the first author or research assistant collected video footage of each classroom. The first author reviewed the footage and provided the teaching teams with feedback on their use of the preventive practices and on child behavior. This feedback was provided via email and was sent within 8 hr after each observation. All emails followed a five-step protocol used in a previous study (Hemmeter et al., 2011). Emails contained a friendly greeting, data on the teaching team’s use of preventive practices and the target child’s challenging behavior, supportive feedback describing positive examples of the practice, corrective feedback offering suggestions to improve the practice, and an embedded response prompt. The purpose of the response prompt was to determine whether teachers had accessed the email messages. Reading and responding to the email feedback took teaching teams approximately 15 min per email. On one occasion when preventive practice use had been unusually low during the previous sessions, the first author also provided feedback face-to-face immediately following the observation session. This occurred one time during the rule reminders condition for Teaching Team 1 and is indicated on the graph in Figure 1. Teachers received emails for a minimum of three sessions and a maximum of five sessions for each preventive practice.
Rate of Teaching Team 1’s use of classroom preventive practices and effects on child behavior.
Dependent Measures
Teacher Preventive Practice
Teacher preventive practice was defined as a teacher verbal utterance, physical gesture, signal, or use of visual aide with the intention of assisting, prompting, promoting, or reminding a child of appropriate classroom behavior. Three types of preventive practices were coded: transition preparation, rule reminders, and social-emotional teaching. Data were collected on teachers’ and assistants’ use of these strategies toward the target child or toward the class as a whole.
Transition preparation
This strategy was defined as any teacher utterance, physical gesture, signal, or visual aide that was directed to a target child or nontarget child with the intention of warning him or her that a change was about to occur or helping the child identify the next step to take. Examples of preparing for transitions included—but were not limited to—directing the child to notice a posted visual schedule, the setting of a timer that went off before transitions, providing a verbal warning when transitions were about to occur (i.e., “We’re going to start cleaning up in one more minute.”), making “first-then” statements (i.e., “First we wash our hands. Then we have snack”), and singing songs or using games to keep the child engaged in the transition. Nonexamples included visual aides (such as schedules or “first-then” charts) that were available in the classroom but not referred to at any time during the session.
Rule reminders
A rule reminder was any teacher utterance, physical gesture, signal, or visual aide that was directed to a target or nontarget child with the intention of making the classroom expectations clear by precorrecting the child’s behavior, reminding a child of the classroom rules, or commenting on a child’s use of a rule. Examples of preventive practices included—but were not limited to—verbally reminding the child of behavior expectations, reading a social story or reviewing the rules prior to an activity, referring the child to posted visual rule displays with or without verbal directions, and checking in with the child prior to an activity to affirm understanding (i.e., before the class starts to walk out of the room, ask, “Joey, can you show me how we walk in the hallway?”). Nonexamples included any rules that were posted in the room but never referred to by the teacher and telling the child what was expected of him or her after a problem behavior had already occurred (i.e., saying “Joey, we walk in the hall” after he had run ahead of the class).
Social-emotional strategies
These strategies were defined as any teacher utterance, physical gesture, signal, or visual aide that was directed to a target child or nontarget child with the intention of helping him or her recognize and deal with emotions or interact with peers. Examples of social-emotional strategies included—but were not limited to—acknowledging a child’s feelings verbally (i.e., “I can tell you are frustrated about leaving Centers”), assisting a child with conflict resolution or problem solving by directing the child to use classroom problem-solving materials such as “Tucker the Turtle” or “The Solution Kit,” prompting the child to interact with a peer or prompting a peer to assist or play with the target child, providing the child with a leadership role during activities (calendar helper, “Clean Up Foreman,” etc.), and engaging in conversations about the child’s interests and ideas.
Child behavior
Challenging behavior was defined as any behavior that (a) impeded the successful completion of activities for a child or children; (b) was disruptive to instruction or classroom routines; (c) limited a child’s interactions with peers, materials, or activities; or (d) was harmful to self, others, or property. Examples included (a) not following instructions or specific rules; (b) talking to a peer inappropriately during group instruction; (c) taking a toy from another child; (d) accessing off-limits materials by picking up a teacher’s book, turning on the cassette player inappropriately, going to a closed center, and so on; (e) physical outbursts that included hitting, scratching, biting, kicking, or using objects to hit others; (f) physically pulling away from a teacher when she was providing physical guidance; (g) tantruming by using a combination of crying, falling to floor, flailing limbs, stomping feet, physical resistance; (h) property destruction; or (i) elopement.
Observer Training and Interobserver Agreement
Interobserver agreement (IOA) data were collected on two groups of variables: (a) teacher use of the three preventive practices (rule reminders, transition preparation, and social-emotional strategies) and (b) target child challenging behavior. Prior to beginning data collection, a graduate assistant was trained to at least 80% accuracy across three consecutive occasions with the primary coder on all dependent variables. Training was completed by coding full-length sample videos of classrooms using varying levels of preventive practices.
All videos were coded by the first author. The trained graduate assistant independently coded 33% of the videotaped sessions per teaching team with at least one session per phase. The IOA sessions were randomly selected, and the primary coder did not know which sessions would be assessed for agreement. The graduate assistant was naive to the treatment condition of all video sessions. Exact agreements were scored when both coders marked the occurrence of a teacher or child behavior within the same 10-s interval. Agreement was computed by dividing the number of agreements of occurrence by the total number of agreements and disagreements and multiplying by 100%. IOA results by dependent variable and teaching team are shown in Table 2. IOA data were lowest for social-emotional teaching strategies and child challenging behavior. The large standard deviations for social-emotional teaching strategies reflect one occasion of low agreement for each teaching team. Because these low agreements occurred on videos from different phases of the study, they likely reflect the complexity of coding social behavior rather than observer drift or bias.
Interobserver Agreement Results by Dependent Variable and Teaching Team.
Treatment Integrity
A 15-item treatment integrity protocol was completed on all training meetings. Sample items on the treatment integrity protocol included clearly outlining the goals of the session, defining the preventive practice, giving at least five general examples of the practice, giving at least two specific examples of the practice related to the target child, and soliciting participants’ comments about the practice. An independent observer attended 50% of the training meetings and independently completed a treatment integrity protocol. For all trainings across both teaching teams, 100% of steps of the training were completed as planned.
Treatment integrity was also assessed on all email messages. Sample items on the email treatment integrity form included opening the email with a friendly greeting, providing positive descriptive feedback, providing the number of preventive practices used during the session, and offering ideas or suggestions for using additional practices. An independent reader coded all email messages against an eight-item treatment integrity checklist. For Teaching Team 1, treatment integrity for email messages averaged 96% across phases (range = 83%–100%). One email did not begin with a positive greeting; rather, it opened with an apology for filming on a day when many therapists were present. In addition, the coach failed to ask for a reply via email in one message. For Teaching Team 2, treatment integrity was 100% for all email messages.
Results
Visual analysis of graphed data was used to interpret the effects of training with email feedback on teachers’ use of preventive practices. The rate of teachers’ use of preventive practices is shown in Figures 1 (Teaching Team 1) and 2 (Teaching Team 2). Data are presented as rate per 10 min to account for unequal observation lengths. Data are presented by teaching team. Ten-minute increments were chosen because of the low frequency of preventive practices. For example, it would be unexpected and undesirable for teachers to provide rule reminders every minute during center time. For Teaching Team 1, the data represent an average of four 10-min segments per observation (range = 3–5). For Teaching Team 2, the data represent an average of three 10-min segments (range = 2–4). To calculate rate, the duration of each observation (in minutes) was divided by 10 and rounded to the nearest 10th. The total frequency of each preventive practice was then divided by this number. Data are presented by teaching team.
Rate of Teaching Team 2’s use of classroom preventive practices and effects on child behavior.
Teaching Team 1
Following baseline, training and email feedback were provided to Teaching Team 1 on the use of Transition Preparations. There was a clear effect of training and feedback for Team 1. During baseline, Helen and Troy used one or fewer transition preparation strategies per session (average rate = 0.2 strategies per 10 min; total frequency range across 45 min observation = 0–2). As shown in Figure 1, Helen and Troy increased their use of transition preparation strategies relative to baseline. Their average rate of strategy use after training with performance feedback was 2.5 transition preparations per 10 min (total frequency range = 2–17). It is notable, however, that a downward trend began at the third data point.
During baseline, Teaching Team 1 used very low levels of rule reminders. Teaching Team 1 used an average of 0.2 reminders per 10 min (total frequency range = 0–2) during baseline. Following training, there was no immediate change in use of rule reminders. After two sessions with little use of rule reminders, immediate face-to-face feedback was provided to the team. Prior to the third session, the first author, who was filming the session, approached the team and gave two suggestions for adding rule reminders to the activity for the day. Team 1 immediately began using the strategies. The team’s use of rule reminders showed an increasing trend across 3 days and reached a total frequency of 18 rule reminders during center-time activities. When training occurred on social-emotional strategies after Session 12, however, use of rule reminders dropped dramatically. Their average rate of rule reminders following intervention was 1.4 strategies per 10 min (total frequency range = 0–18). Although use of rule reminders remained above baseline levels, Teaching Team 1 used the strategy much less when feedback was not provided on the strategy.
There were clear effects of training with feedback on Teaching Team 1’s use of social-emotional strategies. During baseline, the team used an average of 0.5 social-emotional strategies per 10 min (total frequency range = 0–5). Following training and feedback on social-emotional strategies, Team 1 showed a clear increase in level and trend. Their average use of strategies following training was 4.4 per 10 min (total frequency range = 5–23).
When a new preventive practice was introduced during training, feedback was discontinued on the previously trained practice. This resulted in an unanticipated third phase of the study for the transition preparation and rule reminder strategies. When feedback was discontinued on each of these preventive practices, there was a clear decrease in use of that practice for Teaching Team 1.
Teaching Team 2
Teaching Team 2 also showed a marked effect of treatment on transition preparations. During baseline, the team used an average of 0.7 transition preparation strategies per 10 min (total frequency range = 0–6).There was a change in level immediately following training, and use of transition preparations continued to increase until Session 9. After Session 9, there was a marked downward trend in use of transition preparations. This decrease coincides with the introduction of training and feedback on the final preventive practice: social-emotional teaching strategies. During the training plus feedback condition, Teaching Team 2 used an average of 2.8 strategies per 10 min (total frequency range = 1–12).
Results for rule reminders were similar for Teaching Team 2. During baseline, Lucy and Ramona used low levels of rule reminders. As a team, they used an average of 0.4 rule reminders per 10 min during baseline (total frequency range = 0–3). Following training with feedback, the team’s use of rule reminders showed a marked increase. Lucy and Ramona used an average of 3.1 reminders per minute (total frequency range = 3–16).
There was a clear effect of training with feedback for Teaching Team 2’s use of social-emotional strategies. During baseline, Team 2 used an average of 0.5 strategies per 10 min (total frequency range = 0–6). Following training with feedback, there was a clear increase in level. The team used an average of 5.3 social-emotional strategies per 10 min following training and feedback (total frequency range = 9–19).
Challenging Behavior
Data were collected on target child challenging behavior to measure the effects of the intervention (see Figures 1 and 2). The effects on target child challenging behavior were variable for both children. Although Lily, in Teaching Team 1’s classroom, had variable data, she showed successively lower levels of challenging behavior across phases. All sessions during the social-emotional strategies condition had lower levels of target child challenging behavior than baseline.
Ricky’s use of challenging behavior in Teaching Team 2’s classroom was also variable. The phase change from baseline to transition preparation was associated with 100% nonoverlapping data points. This was the only demonstration of effect for Ricky. The remaining phases were associated with higher levels of challenging behavior. These phases were highly variable, however. During rule reminders, Ricky’s challenging behavior was between 9% and 13% of intervals for two sessions. On the final session for rule reminders, his behavior spiked to 27% of intervals. A similar pattern was evident when teachers began using social-emotional strategies. He used challenging behavior for 10% of intervals for the first 2 days. On the 3rd day, his behavior spiked to 17% of intervals.
Discussion
Following training and performance feedback delivered via email, two teaching teams increased their use of transition preparations, rule reminders, and social-emotional teaching strategies. There was a functional relation between performance feedback and preventive practices for nearly all skills. Notably, there was a downward trend in use of most practices when feedback was discontinued. This supports the premise that professional development must be sustained to foster lasting change (Snyder et al., 2011).
This study extends a line of research on helping preschool teachers implement the Pyramid model with fidelity. Much of the work around the Pyramid model has focused on training teachers to use a single classwide strategy (e.g., Hemmeter et al., 2011), individualized behavior support plans (e.g., Duda et al., 2004), or the comprehensive Pyramid model (Fox et al., 2011). No studies to date have taught teachers to sequentially introduce a series of Pyramid practices. This type of brief, “train-coach-train” professional development model could be practical and beneficial for schools and child care programs. The successful use of email for providing feedback also adds to a growing body of literature on that practice and offers another practical strategy for the field (Barton & Wolery, 2007; Hemmeter et al., 2011).
Furthermore, no studies to date have evaluated the effects of classwide preventive practices on target children’s challenging behavior. Hemmeter et al. (2011) collected data on classwide challenging behavior, but changes in the data could not be associated with individual children. In many positive behavior support studies, teacher strategies are selected based on the child’s specific needs. This study took a broader approach and selected global preventive practices that were easy to implement and could be used across children and settings. This is an important consideration given the effort and resources necessary to design and implement individualized behavior support plans. Certain targeted preventive practices may help eliminate the need for more intensive planning.
The effects of the intervention on children’s challenging behavior were mixed. In Teaching Team 1, the target child’s behavior decreased cumulatively across the span of the study. In Teaching Team 2, a clear functional relationship was not established between the intervention and changes in target child behavior. Although this child was diagnosed with a speech and language disorder, he exhibited characteristics of autism. Many of the transition preparation strategies involved visual supports such as a schedule, visual cue cards, timers, and so on. Visual supports have been associated with improved behavior for children with autism (Dettmer et al., 2000; Odom, Brown, Frey, Karasu, Smith-Canter, & Strain, 2003), and Ricky’s behavior declined during this phase. It is possible that the subsequent preventive practices (rule reminders and social-emotional teaching strategies) did not include as many visual supports and therefore were not adequate for changing Ricky’s behavior. Further studies of the relationship between specific teaching strategies and specific target behaviors should be undertaken. It is possible that these global preventive practices are more effective for children with milder presenting issues. It is important that this study be replicated with more children to determine the conditions under which classwide interventions work for specific types of children.
Limitations
The first limitation of this study involved the target children. This study included as participants children with persistent, mild challenging behavior. Children with more intense or severe challenging behavior were not included because these children typically require more individualized and systematic support than this study was designed to provide. In Classroom 2, there were nontarget children with severe and persistent challenging behavior who disrupted the flow of the day. It was not unusual for these behaviors to affect the target child’s behavior. This study did not measure the effects of the classwide prevention strategies on nontarget children, and we did not measure when and to what extent the target child’s behavior was “sparked” by another child with persistent challenging behavior. Given the needs of the children in this classroom, more systematic supports may have been necessary.
Second, the teaching teams recruited for this study differed from one another in many ways. This could improve the generalizability of the study, but it could also have important implications for data collection and intervention. The teams chose different target activities and had very different roles within those activities. The length of observations and the opportunities for strategy use both varied considerably across teams. The teams also responded differently to email feedback. For example, Teaching Team 1 required a face-to-face reminder about using a strategy. Therefore, changes in the team’s behavior cannot be attributed solely to email feedback. Future research should examine the influences of classroom and teacher characteristics on responses to professional development interventions.
Third, no generalization or social validity data were collected in the current study. This limits the external validity of the findings. It would be valuable to know whether teachers used these strategies in different settings across the day. Anecdotal evidence indicated this was the case. After the study ended, teachers independently created and used a rule reminder book for a new playground and posted their own classwide visual schedules. It was unclear in this study whether teachers would maintain the use of preventive strategies after the study concluded. Due to time constraints with upcoming holidays in the school calendar, formal maintenance data were not collected. The decrease in preventive practice use when feedback was discontinued for one practice and applied to the next speaks directly to the issue of maintenance, however. There are a few possible explanations for the decrease in strategy use. One is that they did not learn the strategies with enough fluency to maintain them in the absence of feedback. An alternative explanation is that they thought they were supposed to only use the strategies while the coach was providing feedback on them. They may have wanted to demonstrate the skill they knew the researcher was watching. Regardless, it is possible that the use of the strategies would have continued to decrease in the absence of feedback. Follow-up data would have provided a more compelling case that the teachers had, in fact, acquired the trained skills.
Finally, it is impossible to rule out the effects of sequencing and treatment contamination across phases. Each teaching team received training on transition preparations, rule reminders, and social-emotional teaching strategies in sequential order. It is possible that the strategies are not independent and a teacher’s performance on one skill influences the other skills. For example, many early childhood classrooms have rules related to social skills (“Be a friend”). The study would have been strengthened by counterbalancing the treatment conditions across teaching teams.
Implications for Research and Practice
This study offers several implications for research and practice. First, distance learning is a viable and promising direction for research and practice. In this study, follow-up interactions between the trainer/coach and the teachers took place via email. This greatly reduced the time required of the coach and the teacher. In the current study, video review and email feedback took approximately 45 to 60 min per session for the coach. Teachers spent approximately 10 to 15 min reading the email. This is in contrast to an ongoing study of the Pyramid intervention in which teachers received in-person performance feedback (Hemmeter, Snyder, & Fox, 2010). During live coaching visits in the Hemmeter et al.’s (2010) study, coaches spent an average of 144 min per week in the classroom and an additional 44 min per week in debrief sessions. Video observations and distance feedback could be an efficient way to provide quality professional development. Future studies could streamline this process even more by offering web-based training as well. The increasing ease with which teachers can access technology makes this a worthwhile avenue for investigation.
The second implication involves providing feedback to teachers. This study used a low-cost strategy for supporting teachers’ use of recommended practices. The only expenses were initial costs for camcorders and supplies. Research associates were paid to collect the video footage, but video collection could be integrated into program responsibilities (e.g., Pianta, Mashburn, et al., 2008). Supervisors and mentors could use similar procedures as they work with practicing teachers. For example, researchers have examined the ways supervisors and mentors could incorporate video review into face-to-face meetings and professional development groups (Sherin & van Es, 2009; Summers, Funk, Twombly, Waddell, & Squires, 2007) or web-mediated consultation experiences (Pianta, Mashburn, et al., 2008).
Future research should also focus on examining the characteristics of effective feedback. Research should investigate the optimum frequency, intensity, and duration of the feedback relationship. It should also investigate the elements of effective feedback. This study included data-based feedback, supportive feedback, and corrective feedback. Future researchers should evaluate these types of feedback singly and in combination.
In conclusion, this study provided additional support for the use of email-based feedback in early childhood professional development. Teachers received data-based, supportive, and corrective feedback following each observation. This was associated with increases in both teaching teams’ use of rule reminders, transition preparations, and social-emotional strategies. There was a concurrent decrease in target child challenging behavior in one of the classrooms.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by Grant R324A070212 from the Institute of Education Sciences and the Doctoral Leadership Training Grant awarded to Vanderbilt University from the U.S. Department of Education.