Impact of Video Self-Monitoring With Graduated Training on Implementation of Embedded Instructional Learning Trials

Abstract

We used a multi-component single-subject experimental design across three preschool teachers to examine the effects of video self-monitoring with graduated training and feedback on the accuracy with which teachers monitored their implementation of embedded instructional learning trials. We also examined changes in teachers’ implementation of learning trials. In each self-monitoring condition, teachers observed and recorded their implemented learning trials using video and a coding form. Conditions differed in the specificity of prompts on the coding form and the type of training and feedback provided. The combination of training, coding forms with specific prompts for learning trial components, and external feedback generally resulted in more accurate self-monitoring for two of three participants and increases in the fidelity of implementation of learning trials. Findings suggest self-monitoring can be effective for increasing the fidelity with which teachers implement embedded instructional learning trials, but systematic training and feedback are important for ensuring self-monitoring accuracy.

Keywords

self-monitoring self-coaching embedded instructional trials single-subject design

A fundamental indicator of quality instruction in preschool classrooms is accurate implementation of instructional learning trials embedded across naturally occurring activities and routines to support children’s acquisition of individualized learning targets (Barton, Bishop, & Snyder, 2014; Snyder, Hemmeter, McLean, Sandall, & McLaughlin, 2013). Embedded instruction involves teachers implementing three-term contingency instructional learning trials. These trials include an adult or environmental antecedent (A) to set the occasion for the child to practice a targeted behavior, the child’s response (B) to the antecedent, and an adult- or environmentally delivered consequence (C) or error correction procedure immediately following the child’s behavior.

Researchers have emphasized the importance of quantifying the accuracy of instructional learning trials as a basic unit of instruction (Albers & Greer, 1991; B. F. Skinner, 1968; C. H. Skinner, Fletcher, & Hennington, 1996; VanDerHeyden, Snyder, Smith, Sevin, & Longwell, 2005; Vargus & Vargus, 1991). Complete learning trials (CLTs) is a term used to describe embedded learning trials in which the A and C components of the three-term contingency are implemented by a teacher around a selected child behavior (B) component (Barton et al., 2014; Snyder et al., 2013). Research has shown a positive relationship between the frequent implementation of CLTs and children’s engagement and learning (VanDerHeyden et al., 2005). Despite research supporting the importance of teachers understanding and implementing embedded instruction trials that involve the three-term contingency, obtaining fidelity of implementation in preschool classrooms during ongoing activities has been challenging (McBride & Schwartz, 2003).

One potential explanation for this lack of fidelity might be that embedding instructional trials requires careful planning and support (Snyder et al., 2013). Teachers often have limited access to implementation supports that might help them understand the three-term contingency in relation to instructional trials or to evaluate their implementation of CLTs (Barton et al., 2014). Implementation supports teachers might access include materials or assistance from external agents (e.g., training, mentoring, coaching) or use of internal processes (e.g., self-monitoring, self-reflection, self-coaching).

Coaching is an implementation support demonstrated to be effective for improving teachers’ implementation of instructional practices (e.g., Fox, Hemmeter, Snyder, Binder, & Clarke, 2011; Hemmeter, Snyder, Kinder, & Artman, 2011). Coaching in many published studies has been delivered by an expert (Snyder et al., 2012). There is growing interest in identifying alternative delivery formats for effective coaching. Effective coaching includes opportunities for goal-setting and action planning, focused observation of teachers implementing evidence-based teaching practices, and reflection and feedback (National Center on Quality Teaching and Learning [NCQTL], 2014).

Self-coaching is a format that includes self-guided goal-setting and self-monitoring of one’s implementation of effective teaching practices (NCQTL, 2014). Self-monitoring is characterized by self-observation and self-recording, in which an individual observes for a specific behavior, records the occurrence of that behavior, and self-evaluates regarding aspects of one’s performance (Kazdin, 1994; Maag, 1999). Explicitly teaching self-monitoring might be an efficient strategy for supporting implementation of effective teaching practices, including CLTs.

Self-monitoring has been shown to be efficacious for improving teachers’ implementation of a variety of instructional strategies, including instructional learning trials (Belfiore, Fritts, & Herman, 2008; Lylo & Lee, 2013; Pelletier, McNamara, Braga-Kenyon, & Ahearn, 2010), components of token economy interventions (Petscher & Bailey, 2006; Plavnick, Ferreri, & Maupin, 2010), and praise statements (Kalis, Vannest, & Parker, 2007; Wright, Ellis, & Baxter, 2012). Only two of these studies were conducted in early childhood settings (Plavnick et al., 2010; Wright et al., 2012). Methods used to support teachers’ self-monitoring ranged from checklists completed based on teacher recall (Petscher & Bailey, 2006; Plavnick et al., 2010), in situ monitoring via hand-held counters (Kalis et al., 2007), and reviewing audio or video recordings and noting the occurrence of targeted teaching behaviors (Belfiore et al., 2008; Lylo & Lee, 2013; Pelletier et al., 2010). Although training to implement the self-monitoring intervention was included in every study, the intensity of training varied based on the method used to self-monitor.

Of particular note across these studies is the varied use of feedback procedures to enhance implementation of the self-monitoring intervention. In four of the seven studies (Belfiore et al., 2008; Lylo & Lee, 2013; Pelletier et al., 2010; Wright et al., 2012), feedback on participants’ self-monitoring accuracy was provided by a researcher during initial training to implement the self-monitoring intervention but not after participants began self-monitoring independently. Two studies included feedback from a researcher on participants’ self-monitoring accuracy during training and for the duration of the self-monitoring intervention (Kalis et al., 2007; Petscher & Bailey, 2006), and one study did not include a feedback component (Plavnick et al., 2010). None of the studies examined relationships between variations in training and feedback on self-monitoring behaviors and the accuracy of self-monitoring. In addition, no studies examined relationships between self-monitoring accuracy and participants’ implementation of the monitored instructional practices (e.g., components of the token economy, discrete learning trials).

The purpose of the present study was to extend the research on self-monitoring as a component of self-coaching, with a focus on early childhood teachers’ instructional practices. We used a multi-component single-subject experimental design across three teachers to evaluate three video self-monitoring conditions. Each self-monitoring condition involved teachers viewing video of themselves implementing embedded instructional learning trials in their classrooms and completing a self-monitoring coding form with prompts corresponding to the components of instructional learning trials (i.e., antecedent, behavior, consequence/error correction). Two research questions were addressed:

Research Question 1: What are the functional relationships between graduated training and feedback and teachers’ accuracy of self-monitoring their embedded instruction learning trials?

Research Question 2: How does the completeness of teachers’ learning trials (CLTs) change across the experimental conditions?

Method

Participants

We recruited teachers working in an inclusive, university-based early care and education center who were interested in using video to self-monitor their instructional practices. To be eligible for the study, teachers had to (a) be the lead teacher of children between the ages of 24 and 60 months, (b) have at least one study-eligible child in their classroom, and (c) be able to view and code up to five, 10- to 20-min videos of themselves providing instructional trials to a child during a classroom activity, routine, or transition each week.

Teacher participants

All three teachers were female and identified themselves as White/non-Hispanic. Natalie was 24 years old when the study began. She held a bachelor’s degree in special education and a master’s degree in early childhood education, with a director’s credential and teaching endorsements in pre-kindergarten, pre-kindergarten disabilities, and English for Speakers of Other Languages (ESOL). She had 2 years experience working in early childhood settings. Rhonda was 39 years old when the study began, was a mentor teacher, and had 10 years 7 months experience working in early childhood settings. She held an associate’s degree in elementary and early childhood education and was also a full-time student working toward a bachelor’s degree in early childhood education. She had a director’s and a Child Development Associate credential. Brenda was 24 years old when the study began and held a bachelor’s degree in psychology. She had 5 years 2 months experience working in early childhood settings and had no teaching certifications or endorsements. Both Natalie and Rhonda shared teaching responsibilities with a co-teacher. Brenda was the only teacher in her classroom.

Natalie was the only teacher in the study who reported familiarity with the terminology for the components of embedded instruction learning trials prior to learning about them in the study. None of the teachers indicated they were planning for or implementing embedded instruction learning trials as part of targeted or individualized instruction for children in their classrooms. We conducted observations using the Classroom Assessment Scoring System (CLASS; Pianta, La Paro, & Hamre, 2008) to characterize the types of classroom organization and instructional supports for embedded instruction. CLASS scores indicated low-range scores on quality domains of classroom organization (range = 3.1–3.5) and instructional support (range = 2.1–2.5), which include dimensions related to instructional learning formats, productivity, and the quality of feedback provided to children.

Child participants

Each teacher nominated a child to participate in the study. To be eligible for the study, children had to (a) be between the ages of 24 and 60 months when the study began, (b) not be transitioning to a new classroom during the course of the study, and (c) be identified by the teacher as being at-risk for learning challenges.

All teachers in the study used the Ages and Stages Questionnaires, Third Edition (ASQ-3; Squires, Twombly, Bricker, & Potter, 2009). Information from these assessments and from observations of children was used to determine whether there were children who needed more targeted instruction in a particular developmental domain because they were at risk for learning challenges. This information was also used to inform children’s eligibility for the present study.

Three children participated in the study: David, Amanda, and Allison. None of the children were dual language learners. All three children were eligible for participation based on ongoing observational assessment that they were at risk for learning challenges. David was 57 months old when the study began and was White/non-Hispanic. He had been attending preschool for 32 months and had been in Natalie’s classroom for 12 months. Natalie expressed concerns on the ASQ-3 that David had delays in some drawing skills and noted from observations that he showed limited engagement in early literacy activities that involved drawing and writing. Amanda was 26 months old when the study began and was White/non-Hispanic. She had been attending preschool for 18 months and had been in Rhonda’s classroom for 2 months. Rhonda reported Amanda’s expressive communication during interactions with peers was of concern. Allison was 43 months old when the study began and was multiracial. She had been attending preschool for 26 months and had been in Brenda’s classroom for 3 months. Allison was reported by her teacher and parent to have difficulty with receptive language skills, including following multiple-step directions.

Settings

Classrooms in the center in which the study was conducted were organized in groups of three according to child age. Classrooms were arranged in an L-shape around two restrooms, which were shared among the classrooms. Natalie and Brenda were in adjoining classrooms. They followed the same daily schedule but conducted scheduled activities and instruction independently. There were 14 children in Natalie’s classroom and 9 in Brenda’s classroom. Rhonda’s classroom adjoined to two other classrooms. There were a total of 30 children across the three classrooms, with 9 children in Rhonda’s classroom. Rhonda and one other teacher followed the same daily schedule. These two classrooms sometimes combined for morning circle, and they always combined for learning centers.

Procedures

Teachers were recruited via an email announcement sent by the center director. Teachers who were interested in participating in the study contacted the first author, and eligibility was determined based on the inclusion criteria. Each teacher communicated with the child’s parent to obtain informed consent for his or her participation in the study.

CLASS (Pianta et al., 2008) observations were conducted according to the administration protocol by two trained observers during the time when teachers were obtaining consent for child participants. Inter-observer agreement was calculated at the dimension-item level for one of the CLASS observations. Inter-observer agreement was 86%.

Pre-intervention

After children were enrolled, each teacher completed a child demographic form to indicate child’s gender, age, number of months enrolled in preschool, and number of months in the teacher’s classroom. We obtained permission from both the center and the child’s parents to retain a copy of the child’s ASQ-3 completed prior and most proximal to the start of the study.

Each teacher met with the first author to review information from the ASQ-3 and their observations of the children they nominated to participate in the study. Teachers discussed their concerns about the children they nominated to participate in the study, and the first author observed the children to confirm they were eligible to participate in the study due to learning challenges. Information provided by the teacher and documented in the first author’s observations of the children was used to determine an individualized learning target for the child. The term learning target was defined for the teacher as an instructional goal for the child. The learning target specified an observable child behavior the teacher wanted the child to demonstrate. After the learning target was determined, the teacher and first author identified one classroom activity, routine, or transition in which the teacher would teach the target behavior and which could be videotaped for self-monitoring and data collection purposes. Throughout the study, the teacher and first author monitored the child’s demonstration of the target behavior and modified the child’s learning target if the child demonstrated mastery or if the target behavior appeared to be too difficult for the child. Each child’s learning target was focused on the same developmental domain from the ASQ-3 throughout the study. The activity during which videotaping occurred did not change. Information about the activities and children’s learning targets is shown in Table 1.

Table 1.

Child Learning Target Behaviors and Target Activities.

Child	Initial learning target behavior	Final learning target behavior	Target activity	Activity duration in minM (Range)
David	Use a writing tool to make letters or drawings with straight, curved, or angled markings	Write uppercase letters and numbers with a model	Literacy learning center facilitated by Natalie	10 (4–25)
Amanda	Use a three-word phrase with at least one describing word to make requests, describe objects/actions, or answer questions (e.g., my red ball, he runs fast, I want blue)	Use a three-word phrase with at least one describing word to make requests, describe objects/actions, or answer questions (e.g., my red ball, he runs fast, I want blue)	Learning center activity facilitated by Rhonda	12 (5–21)
Allison	Complete multi-step transition tasks (e.g., brush teeth, wash hands, clean up lunch) independently	Complete multi-step transition tasks (e.g., brush teeth, wash hands, clean up lunch) independently within 3 min of starting the task	Transition from lunch to nap	17 (12–27)

During pre-intervention, study data collectors gathered video of the teachers implementing learning trials in the target activities shown in Table 1, but these videos were not shared with the teachers. Teachers did not receive information on the components of embedded instruction learning trials nor were they given access to materials used in the self-monitoring conditions.

Video self-monitoring: Condition I

Following pre-intervention, teachers were given a digital video camera and asked to begin self-monitoring their implementation of learning trials using a self-monitoring coding form with general prompts corresponding to the components of a CLT. The Self-Monitoring Coding Form had the following series of open-ended questions to help teachers identify and describe the components of instructional learning trials: (a) At what time in the video was there an intentional learning opportunity provided or arranged by you? (b) What happened to provide an opportunity for the child to practice the learning target behavior? (c) Did the target behavior occur? (d) What did you do after the target behavior occurred, if anything? (e) If the target behavior did not occur, what did you do next, if anything?

At naptime on the first day of this video self-monitoring condition, the teacher engaged in a 15- to 20-min orientation meeting in which the first author (a) described how to operate the digital video camera, (b) provided the teacher with Self-Monitoring Coding Forms, and (c) described how to complete the Self-Monitoring Coding Form. This orientation did not include any training on the components of CLTs or opportunities for the teacher to practice coding and receive feedback on her use of the coding form. At the end of the meeting, the first author gave the teacher her video from earlier in the day and instructed her to review the video and complete the Self-Monitoring Coding Form before the next scheduled data collection. Teachers used video collected by study data collectors during the established target activity and the Self-Monitoring Coding Form 3 to 5 times a week to enter answers to the series of questions related to their implementation of embedded instruction learning trials.

Video self-monitoring: Condition II

Following the first self-monitoring condition, teachers received systematic training by the first author about the components of CLTs and began using the Embedded Instruction Observation System–Teacher Version (EIOS-T; Bishop, Snyder, Crow, Mullen, & Embedded Instruction for Early Learning Project, 2011) to self-monitor their implementation of embedded instruction learning trials from video collected by study data collectors. The EIOS-T is a system for monitoring implementation of CLTs that includes a coding form, coding manual, and training materials. The EIOS-T coding form has a series of coding prompts with precise terminology for teachers to record the occurrence and accuracy of the antecedent, child behavior, and consequence or error correction (presented as “extra help”) components of instructional learning trials rather than the open-ended questions that appeared on the Self-Monitoring Coding Form. The EIOS-T coding manual includes information on (a) the components of an instructional learning trial, (b) how to determine when a trial occurs, and (c) how to determine whether each component of the trial is implemented with fidelity. The EIOS-T training was a 2-hr, interactive training that reviewed and supplemented coding manual content.

Participants received the EIOS-T coding manual and participated in the EIOS-T training session before they began self-monitoring coding using the EIOS-T. The training was delivered by the first author and occurred outside of work hours at a time and location convenient for the teacher. The first 90 min of the training was devoted to familiarizing the teacher with the components of instructional trials and EIOS-T coding procedures. For each component of a trial, the trainer (a) gave the teacher a definition of the component, (b) explained coding rules to help the teacher identify when the component was occurring and whether it was implemented with fidelity, (c) showed a video clip of the component being implemented correctly and a video clip of the component being implemented incorrectly or not at all, (d) modeled how to record the occurrence and accuracy of the component on the EIOS-T coding form, (e) brainstormed with the teacher what the component might look like relative to her child’s target behavior, and (f) answered questions the teacher had about coding the component. The trainer also discussed how to summarize EIOS-T data and use this information to evaluate whether implemented trials were CLTs.

In the final 30 min of the training, the teacher practiced coding independently. Each teacher viewed and coded four video clips from her pre-intervention sessions. Each teacher coded one video clip of herself implementing (a) a trial in which the target behavior occurred without extra help and a consequence was provided (i.e., CLT), (b) a trial in which the target behavior occurred without extra help and no consequence was provided (i.e., incomplete learning trial), (c) a trial in which the target behavior occurred following extra help and a consequence was provided (i.e., CLT), and (d) a trial in which the target behavior did not occur and no extra help was provided (i.e., incomplete learning trial). Each teacher also coded three, 1- to 2-min videos of a preschool teacher unfamiliar to her implementing multiple trials during a classroom activity or routine. The trainer provided immediate feedback about the teachers’ coding of the occurrence and accuracy of learning trial components for each trial. Teachers were required to demonstrate at least 80% coding accuracy for learning trial occurrences and learning trial components on three of the seven practice videos (i.e., four pre-intervention video clips and three clips of an unfamiliar teacher) before continuing with the self-monitoring intervention.

After teachers demonstrated 80% coding accuracy on three practice videos, they used the videos collected by study data collectors and the EIOS-T coding form to self-monitor their implementation of learning trials 3 to 5 times per week. No additional training on CLT components was provided nor did teachers receive feedback about their coding accuracy.

Video self-monitoring: Condition III

In the final intervention condition, teachers continued to use the EIOS-T to self-monitor their implementation of trials, and they received ongoing feedback from the first author about their self-monitoring accuracy. We used correspondence between the teacher’s self-monitoring data and the researcher’s data on the occurrence of trials and the components of CLTs implemented as a measure of self-monitoring accuracy. On the first day of this condition, the first author videotaped the teacher implementing trials on the child’s identified target behavior during the targeted routine or activity. The first feedback session was a 30- to 45-min interactive discussion that occurred at naptime of the same day. The goal of this session was for the teacher and first author to gain consensus on how the child’s target behavior would be recognized. The first author presented an EIOS-T Coding Feedback Form, which highlighted the teacher’s coding strengths and explained discrepancies between the teacher’s and first author’s coding that were due to differences in their definitions of the target behavior. Feedback was based on comparisons of the teacher’s and first author’s coding from sessions in the video self-monitoring II condition. The teacher and first author completed a worksheet together to reach consensus about how they would identify when the target behavior occurred. At the end of the session, the teacher was instructed to view the video collected by the first author earlier that day and use the EIOS-T coding form to self-monitor her implementation of trials and CLTs on the revised target behavior.

For each subsequent session, the first author videotaped the teacher and child in the target activity and collected the teacher’s EIOS-T Self-Monitoring Coding Form from the previous session. After filming, the first author reviewed the teacher’s coding form from the previous session and completed the EIOS-T Coding Feedback Form. At naptime of the same day, the first author and teacher met for 10 to 20 min to discuss the teacher’s coding. Every feedback session included (a) opportunities for the teacher to ask questions about her coding, (b) a presentation of a summary of coding agreements and disagreements between the teacher and first author, (c) written and verbal positive feedback regarding the teacher’s use of the EIOS-T to monitor her implementation of trials and CLTs, (d) written and verbal corrective feedback regarding the teacher’s use of the EIOS-T to monitor her implementation of learning trials, and (e) confirmation of the child’s target behavior. Supportive and corrective feedback were provided in a variety of ways, including (a) graphs showing changes in the teacher’s coding accuracy, (b) reviewing video of the teacher implementing trials and discussing coding discrepancies, and (c) modeling EIOS-T coding procedures for the teacher.

Maintenance

A data collector videotaped the teacher 1 to 3 times per week during the target activity and provided the video to the teacher. Each teacher was given the option to self-monitor her implementation of trials using either the Self-Monitoring Coding Form or the EIOS-T. No training or feedback about self-monitoring accuracy was provided.

Instrumentation

Project-developed measures were used to assess the (a) fidelity with which the researcher implemented the components of the self-monitoring training and feedback sessions, (b) correspondence between teachers’ self-monitoring data and the researcher’s data about the occurrence of trials and components of CLTs, (c) percentage of CLTs implemented by the teacher, and (d) social validity.

Training and feedback fidelity

Data were collected for at least 33% of all trainings and feedback sessions. Project-developed checklists were used to assess procedural fidelity for all intervention conditions. Each checklist outlined the steps to be completed in the training session for each condition and provided spaces for an observer to indicate whether or not each step occurred. A graduate assistant familiar with the training components of the study listened to an audio recording of each training or feedback session and completed the corresponding checklist by indicating whether or not each step of the protocol occurred. Procedural fidelity was calculated by dividing the number of steps completed by the total number of steps and multiplying the result by 100.

Self-monitoring accuracy

We calculated the agreement between the teachers’ and first author’s recordings of trial occurrences and CLT components. Occurrence agreement was defined as the level of agreement between the teacher and researcher on the occurrence of trials to elicit the child’s target behavior. To examine occurrence agreement, we compared the time codes recorded by the teacher and researcher to indicate the occurrence of a trial. Time codes within 3 s of one another were considered agreements. If two time codes were close to one another but more than 3 s apart, we reviewed the video to determine whether both time codes corresponded to the same trial. If both time codes corresponded to the same trial, these codes were counted as occurrence agreements. Time codes more than 3 s apart that did not correspond to the same trial were counted as disagreements. To calculate the percentage of occurrence agreement, we divided the total number of agreements by the sum of agreements and disagreements and multiplied the result by 100.

Component agreement was defined as the extent to which the teacher and researcher agreed on the fidelity with which each CLT component was implemented. To examine component agreement, we compared the antecedent, child behavior, consequence, and error correction codes for all trial occurrences recorded by both the teacher and researcher. The percentage of component agreement was calculated by dividing the total number of agreements by the sum of agreements and disagreements and multiplied by 100. Because the Self-Monitoring Coding Form only provided teachers with an opportunity to describe learning trial components but did not include EIOS-T codes, component agreement was only calculated for the video self-monitoring II and video self-monitoring III conditions.

Implementation of trials and CLTs

We used the EIOS-T coding form to record teachers’ occurrence of trials and accuracy of CLTs. The first author was the primary coder for the study and coded every video collected. A graduate assistant coded 33% of all videos collected in the study for calculating inter-observer agreement. Both coders helped develop the EIOS-T coding system and so were familiar with the components of instructional learning trials. Each coder was required to demonstrate 80% inter-observer trial occurrence and component agreement on three consecutive training videos of teachers providing embedded instruction learning trials within the context of ongoing preschool activities before coding study video footage. Each training video was a minimum of 10 min.

Throughout the study, the two coders met each time a child’s individualized learning target changed to gain consensus on how an occurrence of the child’s target behavior would be recognized for coding. To record the occurrence of learning trials, coders played the videotape and documented the time code for every intentional antecedent the teacher provided or arranged in the environment to elicit the child’s target behavior. Coders indicated whether the target behavior occurred after the antecedent and whether an appropriate consequence or error correction procedure was provided. In trials where an error correction procedure was necessary to elicit the target behavior, the coder indicated whether the target behavior occurred after error correction and whether the teacher provided an appropriate consequence or feedback.

For each session, coders summarized the percentage of CLTs. A trial was considered complete if all teacher-delivered or environmentally arranged components (i.e., antecedent, consequence, error correction) were implemented appropriately before and after the targeted child behavior. These data were graphed for visual analysis to examine teachers’ implementation of CLTs across all experimental conditions. The percentage was calculated by dividing the number of CLTs by the total number of trials attempted and multiplying the result by 100.

Inter-observer agreement

Occurrence and component agreement were examined for 33% of all coded sessions (n = 37). Mean occurrence agreement was 87% (range = 43–100) overall and 92% (range = 71–100), 81% (range = 50–100), and 89% (range = 43–100) for Natalie, Rhonda, and Brenda, respectively. Mean component agreement was 87% (range = 50–100) overall and 91% (range = 75–100), 84% (range = 50–100), and 88% (range = 62–100) for Natalie, Rhonda, and Brenda, respectively.

Social validity

To assess teacher acceptability of the self-monitoring instruments used in the study, we asked teachers to complete Intervention Rating Profiles (IRP) for the Self-Monitoring Coding Form and for the EIOS-T. The IRPs were adapted versions of an available instrument measuring the acceptability of behavioral interventions implemented by teachers in classroom settings (Martens, Witt, Elliott, & Darveaux, 1985). Both the IRP for the Self-Monitoring Coding Form (IRP-SM) and the IRP for the EIOS-T (IRP-EIOST) are 12-item scales. Teachers responded to items regarding (a) the usefulness of the form to evaluate instructional practices (v = 3), (b) the value of the form with respect to the time needed to use it to self-monitor instruction (v = 2), (c) practicality of using the form in terms of time and technical skill required (v = 2), (d) usefulness of the form for evaluating the implementation of instructional learning trials (v = 4), (e) and overall benefit of the form to teachers (v = 1). Responses were recorded on a 6-point Likert-type scale ranging from 1 (strongly disagree) to 6 (strongly agree). The IRP-SM was administered at the end of the first self-monitoring intervention condition. The IRP-EIOST was administered at the end of the third self-monitoring intervention condition. Teachers were asked not to identify themselves on the forms, so their answers could remain anonymous.

Experimental Design and Data Collection

We used a multi-component single-subject experimental design with probes (Horner & Baer, 1978) across teachers. Consistent with multiple probe procedures described by Kennedy (2005), data collection occurred intermittently during the execution of the experimental series, with at least three probes occurring in the same week before a condition change. Data collection and analysis procedures are described for each phase of the study.

Pre-intervention

Pre-intervention data collection began after the teacher and first author identified an individualized learning target for the child and occurred during the target activity identified by the teacher and first author (see Table 1). To determine whether teachers were engaging in other types of self-monitoring, data collectors observed whether teachers (a) had access to or used video cameras during the target activities, (b) used checklists or other forms to document their implementation of embedded instruction learning trials, or (c) reported using other procedures to self-monitor their implementation of learning trials.

Data collectors used project video cameras to collect video data to examine teachers’ implementation of trials during the target activities shown in Table 1. On each day of videotaping, the data collector (a) confirmed the child’s target behavior with the teacher, (b) asked the teacher for information about how she planned to provide instruction to elicit the target behavior, and (c) videotaped the teacher and child during the target activity. Data collection occurred 1 to 5 times per week.

Video self-monitoring I

On the first day of this condition, the first author used the teacher’s video camera to videotape the teacher during the target activity and provided the teacher with a copy of her video during the orientation session described previously. Occurrence agreement was calculated for this session.

Following the orientation session, each teacher was videotaped 3 to 5 times per week using her own video camera. The same videotaping procedures outlined for the pre-intervention condition were used, and the teacher’s Self-Monitoring Coding Form was collected for the previous session. The teacher was provided with a copy of her video to review, so she could self-monitor and record her implementation of learning trials. Data from these sessions were used to analyze learning trial occurrence agreement and teachers’ implementation of CLTs.

Video self-monitoring II

Data collection for this condition began the day following the EIOS-T training. Each teacher was videotaped 3 to 5 times per week following the same procedures used in the video self-monitoring I condition. Data from these sessions were used to analyze trial occurrence and CLT component agreement between the teacher and the researcher as well as teachers’ implementation of learning trials.

Video self-monitoring III

Data collection for this condition began the day of the first feedback session. Data collection and feedback sessions occurred 3 to 5 times per week. Video collection procedures were the same as the previous two intervention phases, and the data analysis procedures were the same as those described for the video self-monitoring II condition.

Maintenance

Videotaping continued 1 to 3 times per week following the same procedures used in the intervention conditions. When teachers submitted coding forms during the maintenance phase, we calculated the type(s) of agreement appropriate for the coding form submitted by the teacher (i.e., occurrence and component for EIOS-T, occurrence for Self-Monitoring Coding Form). Data from these sessions were also used to analyze teachers’ implementation of learning trials.

Results

Training and Feedback Fidelity

A fidelity checklist was completed for 50% of all researcher-implemented orientation, training, and feedback sessions (n = 15). In total, one Self-Monitoring Training Fidelity Checklist, two EIOS-T Training Fidelity Checklists, three EIOS-T Feedback Orientation Checklists, and nine EIOS-T Feedback Checklists were completed. Each step on the fidelity checklists was implemented for the orientation to the Self-Monitoring Coding Form, the EIOS-T training, and the feedback orientation sessions. Across the nine EIOS-T feedback sessions, an average of 97% steps were implemented (range = 87–100).

Self-Monitoring Accuracy

There was no evidence that teachers were self-monitoring their implementation of embedded instruction learning trials during pre-intervention; therefore, occurrence and component agreement were not recorded for these sessions. As shown in Figure 1, none of the teachers achieved the criterion of 80% agreement with the researcher’s data until the video self-monitoring III condition. Natalie and Brenda consistently demonstrated 80% or higher occurrence agreement with the researcher’s data during this condition. Component coding agreement levels increased and became less variable for all three teachers in this condition.

Figure 1.

Correspondence between teachers’ self-monitoring data and researcher’s data on the occurrence and components of embedded instruction learning trials.

Natalie’s mean occurrence agreement during video self-monitoring I, II, and III was 12% (range = 0–20), 62% (range = 38–75), and 90% (range = 75–100), respectively. Her occurrence agreement varied by approximately 20% within in each condition, but she demonstrated increases in occurrence agreement across each condition, with 100% non-overlapping data in Conditions II and III compared with Condition I, and only one overlapping data point across Conditions III and II. Following her fifth feedback session, Natalie’s occurrence agreement with the first author was consistently 100%. Similar findings were observed with respect to component agreement. Natalie’s mean component agreement for Conditions II and III was 66% (range = 40–83) and 88% (range = 80–100), respectively, with only one non-overlapping data point across conditions.

Rhonda’s mean occurrence agreement during Conditions I, II, and III was 31% (range = 9–50), 36% (range = 9–50), and 47% (range = 18–76), respectively. Her occurrence agreement was highly variable throughout the study. When she began receiving feedback, Rhonda showed some improvement in occurrence agreement for specific sessions, but these changes were not consistent. This is shown by completely overlapping data between Conditions I and II, and only three of eight non-overlapping data points across Condition III and Conditions I and II. Rhonda’s mean component agreement for Conditions II and III was 66% (range = 33–83) and 83% (range = 69–100), respectively. In the final four sessions of the feedback condition, she consistently showed component agreement between 80% and 100%. Rhonda was the only teacher to submit a coding form during maintenance, and her agreement with the researcher’s data approximated the final sessions of the feedback condition.

Brenda’s mean occurrence agreement during Conditions I, II, and III was 33% (range = 0–66), 41% (range = 20–63), and 90% (range = 78–100), respectively. Her occurrence agreement was relatively stable in the beginning of Condition I and then varied in the final three sessions of this condition. This variability continued throughout Condition II. Although her mean occurrence agreement was higher in Condition II, all data points in this condition overlapped with data points in Condition I. In the feedback condition, Brenda’s occurrence agreement increased and became less variable, with 100% non-overlapping data between this condition and Conditions I and II. Brenda’s mean component agreement during the video self-monitoring II and III conditions was 67% (range = 39–89) and 82% (range = 61–83), respectively. Component agreement for all except two of her coded sessions in the feedback condition was above 80%, compared with only three sessions above 80% in Condition II. Although there was less variability in her component agreement during Condition III, only one of seven data points in this condition was non-overlapping with those in Condition II.

Teachers’ Implementation of CLTs

Changes in percentage of CLTs are shown in Figure 2. All three teachers demonstrated increases in the percentage of CLTs during at least one of the video self-monitoring conditions. Natalie’s mean percentage of CLTs implemented during Conditions I, II, and III was 19% (SD = 22.3), 72.4% (SD = 11.5), and 98% (SD = 3.6), respectively. All of the data points in Condition I overlapped with those in pre-intervention, and percentage of CLTs decreased to 0 just before receiving training on the EIOS-T. Positive changes in Natalie’s implementation of CLTs were immediate following EIOS-T training. Her mean percentage of CLTs was substantially higher in Conditions II and III than in pre-intervention or Condition I, with no overlapping data points between pre-intervention and each of these two conditions. Natalie’s implementation of CLTs was highest and most stable after she began receiving feedback on her EIOS-T coding accuracy. Her implementation of CLTs became slightly more variable when self-monitoring stopped, but she generally maintained implementing a similar percentage of CLTs as in Condition III.

Figure 2.

Changes in percentage of CLTs across self-monitoring conditions.

David’s target behavior was modified 3 times during the study (see Figure 2). Modifications were made because he consistently demonstrated the previous target behavior. The modifications involved increasing the complexity of the target skills (see Table 1). Natalie’s implementation of CLTs around the time of the first target behavior modification in Condition II increased compared with the previous two sessions but was within the range of observed percentages (57%–88%) of CLTs observed in Condition II. Her implementation of CLTs around the time of the second target behavior modification in Condition III was higher than any previous sessions (95%). Subsequent observations showed variability around this level of implementation (91%–100%), and her implementation was consistent with previous sessions at the time of the third learning target modification.

Rhonda’s mean percentage of CLTs during Conditions I, II, and III was 54.7% (SD = 9.7), 55.9% (SD = 15.7), and 74.9% (SD = 12.7), respectively. Her implementation of CLTs was variable within and across conditions. Although her mean percentage of CLTs was higher in Conditions I, II, and III compared with pre-intervention, the only condition in which there was 100% non-overlapping data compared with pre-intervention was Condition III. During maintenance, Rhonda had similar percentages of CLTs across sessions to those observed in each of the video self-monitoring conditions, and on average, these levels were higher than those in pre-intervention.

Brenda’s mean percentage of CLTs during Conditions I, II, and III was 10.4% (SD = 14.5), 28.7% (SD = 22.2), and 75.7% (SD = 13.7), respectively. Her implementation of CLTs varied within and across conditions. Nevertheless, she consistently demonstrated the highest percentages of CLTs during video self-monitoring with feedback (Condition III), as shown by six of seven non-overlapping data points across this condition and pre-intervention. In addition, she had only one non-overlapping data point between Condition III and Condition II and completely overlapping data between Condition II and pre-intervention. Brenda’s implementation of CLTs was generally higher and less variable in maintenance than in pre-intervention or Conditions I and II, but she did not continue to implement CLTs with the same level of accuracy as she did during video self-monitoring with feedback.

Allison’s target behavior was modified 4 times during the study (see Figure 2). Modifications involved breaking down the number of steps in the transition task she was required to complete and the duration in which she was expected to complete them. Brenda’s implementation of CLTs varied around the time of target behavior modifications. However, there was no apparent deviation in variability of implementation compared with other observations in the conditions in which modifications were made (pre-intervention = 0%–63%, Condition II = 67%, Condition III = 60%–100%).

Social Validity

Teacher ratings on the IRP-SM and the IRP-EIOST showed teachers found both self-monitoring instruments acceptable. The overall mean rating for the Self-Monitoring Coding Form was 4.8 (range = 4.5–5.2), and for the EIOS-T, it was 4.8 (range = 3.8–5.4). Average ratings for items on both IRP instruments were nearly identical across teachers. Average IRP ratings for the item pertaining to the overall benefit of the Self-Monitoring Coding Form were slightly higher than ratings for the overall benefit of the EIOS-T (5.3 and 4.7, respectively).

Discussion

Findings from the present study show the combination of training, self-monitoring forms that explicitly defined CLTs, and external feedback resulted in more accurate video self-monitoring for two of the three teachers in the study. As teachers’ accuracy of video self-monitoring increased, researcher-observed implementation of CLTs increased. The training provided in the video self-monitoring II condition had components similar to the training provided in previous studies examining relationships between self-monitoring and teachers’ implementation of instructional learning trials (e.g., Belfiore et al., 2008; Lylo & Lee, 2013). These components include definitions of behaviors teachers will self-monitor, modeling self-monitoring procedures, feedback from a trainer on self-monitoring accuracy, and required demonstration of self-monitoring accuracy before engaging in independent self-monitoring.

Our data show that teachers met the required self-monitoring accuracy criteria during training but demonstrated modest and variable levels of self-monitoring accuracy during Condition II. Their accuracy generally improved when they received feedback in video self-monitoring Condition III. Our findings differ from those of previous studies, which reported teachers reliably met self-monitoring accuracy criteria in the initial training and generally continued to self-monitor at or above the established criteria without receiving additional supports. One explanation for this finding might be that in previous studies, teachers monitored their implementation of discrete or massed trials during structured direct instruction activities. It is possible teachers in the present study required more support to self-monitor reliably because self-monitoring occurred in relatively unstructured activities, and learning trials were embedded throughout the activities. In addition, teachers in the present study might not have been as familiar with the components of instructional learning trials before participating in the study as those in previous studies.

Our findings also show positive relationships between video self-monitoring with training and feedback and the fidelity with which teachers implemented CLTs. Changes in the percentages of CLTs implemented across conditions in the present study were similar to those reported in previous studies (e.g., Belfiore et al., 2008; Lylo & Lee, 2013; Pelletier et al., 2010), but the intensity of support provided from an external agent to achieve these changes differed. In addition, the magnitude of change varied across teachers, which suggests a need for additional research to determine under what conditions and for whom self-coaching, including self-monitoring, might be most effective. For example, the primary foci of the training and feedback in the present study were to identify the occurrence and evaluate the accuracy of instructional trials rather than increase the number of trials implemented. Video self-monitoring with training and feedback was most effective for Natalie, who had the highest level of training, which included a bachelor’s degree in special education. Natalie also reported familiarity with the components of embedded instruction learning trials before intervention began. It is possible that findings for Rhonda and Brenda might have been more robust if they had received training on embedded instruction learning trials before implementing the self-monitoring interventions evaluated in the present study.

During maintenance, only one teacher showed CLT implementation percentages similar to those shown when she was receiving feedback on her self-monitoring accuracy. Belfiore and colleagues (2008) observed similar trends in teachers’ implementation of discrete learning trials, but when the researchers re-initiated the treatment phase (i.e., use of self-monitoring forms without additional training), implementation levels increased. These findings suggest a potential need for longer periods of supported self-monitoring. In addition, data from the present study suggest a potential need for occasional “booster” sessions, in which teachers receive feedback on their self-monitoring accuracy. Additional research is needed to examine the relative effects of different durations of supported self-monitoring on teachers’ instructional practices and to identify the frequency and types of supports, including feedback from an external agent, needed to sustain these effects.

Although teachers generally found both self-monitoring coding forms acceptable, the maintenance data show they did not continue to self-monitor with either tool. The implementation data show use of the coding form with explicit descriptions of learning trial components was more effective at increasing the percentage of CLTs than the general coding form. Teachers did not continue to use this form, however, to self-monitor their implementation of trials. This finding suggests a need for external supports, such as reminders or incentives to continue self-monitoring.

Two primary limitations of the present study are noted. First, we did not collect data about whether teachers’ self-monitoring generalized to other activities or targeted child behaviors. Another limitation is that teachers did not collect their own video data. We provided teachers with personal digital video cameras to collect their own data, but conditions in their classrooms (e.g., staff–child ratios, room arrangements) prevented them from doing so. Future studies could explore feasible ways for teachers to gather video for use in self-monitoring and identify programmatic and infrastructure supports that are necessary to self-monitoring.

Findings from the present study contribute toward understanding which training and coaching components are most likely to result in accurate self-monitoring and improved implementation of instructional practices. Self-monitoring, as used in the present study, included several key components of effective coaching (i.e., defined set of teaching practices, self-observation, self-evaluation) identified by Snyder and colleagues (2012). Face-to-face expert coaching might not be feasible or necessary to implement with all teachers. Alternative coaching formats, such as self-coaching, might be efficient and effective to support practice implementation for some teachers under certain conditions, provided the key components of effective coaching are used (NCQTL, 2014). This includes identifying the self-monitoring practices most important for implementing self-coaching. In the context of implementation science, continuing to explore the utility and effectiveness of self-monitoring as a competency driver of evidence-based practice implementation seems warranted.

Footnotes

Acknowledgements

The authors would like to acknowledge the teachers and children who participated in this study, as well as graduate assistants Dana Kasian and Cathleen Pasia for their contributions to this research.

Authors’ Note

The opinions expressed are those of the authors, not the funding agency.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Work reported in this article was supported, in part, by a grant from the National Center for Special Education Research in the Institute of Education Sciences to the University of Florida (R324A070008).

References

Albers

A. E.

Greer

(1991). Is the three-term contingency trial a predictor of effective instruction? Journal of Behavioral Education, 1, 337–354.

Barton

Bishop

Snyder

(2014). High quality instruction through CLTs: Blending intentional teaching with embedded instruction. In Pretti-Frontczak

Grisham-Brown

Sullivan

(Eds.), Young Exceptional Children Monograph Series No. 16: Blending practices for all children (pp. 73–96). Los Angeles, CA: The Division for Early Childhood of the Council for Exceptional Children.

Belfiore

P. J.

Fritts

K. M.

Herman

B. C.

(2008). The role of procedural integrity: Using self-monitoring to enhance discrete trial instruction. Focus on Autism and Other Developmental Disabilities, 23, 95–102. doi:10.1177/1088357607311445

Bishop

C. C.

Snyder

Crow

Mullen

, & Embedded Instruction for Early Learning Project. (2011). EIOS-T: Embedded instruction for early learning observation system—Teacher version [Manual and training videos]. Unpublished instrument. College of Education, University of Florida, Gainesville.

Fox

Hemmeter

M. L.

Snyder

Binder

D. P.

Clarke

(2011). Coaching early childhood special educators to implement a comprehensive model for promoting young children’s social competence. Topics in Early Childhood Special Education, 31, 178–192. doi:10.1177/0271121411404440

Hemmeter

M. L.

Snyder

Kinder

Artman

(2011). Impact of performance feedback delivered via electronic mail on preschool teachers’ use of descriptive praise. Early Childhood Research Quarterly, 26, 96–109.

Horner

R. H.

Baer

D. M.

(1978). Multi-probe technique: A variation of the multiple baseline. Journal of Applied Behavior Analysis, 11, 189–196.

Kalis

T. M.

Vannest

K. J.

Parker

(2007). Praise counts: Using self-monitoring to increase effective teaching practices. Preventing School Failure: Alternative Education for Children and Youth, 51(3), 20–27. doi:10.3200/PSFL.51.3.20-27

Kazdin

A. E.

(1994). Behavior modification in applied settings (5th ed.). Belmont, CA: Wadsworth.

10.

Kennedy

C. H.

(2005). Single-case designs for educational research. Boston, MA: Pearson Education.

11.

Lylo

B. J.

Lee

D. L.

(2013). Effects of delayed audio-based self-monitoring on teacher completion of learning trials. Journal of Behavioral Education, 22, 120–138. doi:10.1007/s10864-012-9166-9

12.

Maag

J. W.

(1999). Behavior management: From theoretical implications to practical applications. San Diego, CA: Singular.

13.

Martens

B. K.

Witt

J. C.

Elliott

S. N.

Darveaux

D. X.

(1985). Teacher judgments concerning the acceptability of school-based interventions. Professional Psychology: Research and Practices, 16, 191–198.

14.

McBride

B. J.

Schwartz

I. S.

(2003). Effects of teaching early interventionists to use discrete trials during ongoing classroom activities. Topics in Early Childhood Special Education, 23, 5–17.

15.

National Center on Quality Teaching and Learning. (2014). Practice-based coaching. Seattle, WA: U.S. Department of Health and Human Services, Administration on Children and Families, Office of Head Start, National Center on Quality Teaching and Learning. Retrieved from http://eclkc.ohs.acf.hhs.gov/hslc/tta-system/teaching/development/coaching.html

16.

Pelletier

McNamara

Braga-Kenyon

Ahearn

W. H.

(2010). Effect of video self-monitoring on procedural integrity. Behavioral Interventions, 25, 261–274. doi:10.1002/bin.316

17.

Petscher

E. S.

Bailey

J. S.

(2006). Effects of training, prompting, and self-monitoring on staff behavior in a classroom for students with disabilities. Journal of Applied Behavior Analysis, 39, 215–226.

18.

Pianta

R. C.

La Paro

Hamre

(2008). Classroom Assessment Scoring System—PreK [CLASS]. Baltimore, MD: Brookes.

19.

Plavnick

J. B.

Ferreri

S. J.

Maupin

A. N.

(2010). The effects of self-monitoring on the procedural integrity of a behavioral intervention for young children with developmental disabilities. Journal of Applied Behavior Analysis, 43, 315–320.

20.

Skinner

B. F.

(1968). The technology of teaching. New York, NY: Meredith Corporation.

21.

Skinner

C. H.

Fletcher

P. A.

Hennington

(1996). Increasing learning rates by increasing student response rates: A summary of research. School Psychology Quarterly, 11, 313–325.

22.

Snyder

Hemmeter

M. L.

Artman

Kinder

Pasia

McLaughlin

(2012). Characterizing key features of the early childhood development literature. Infants & Young Children: An Interdisciplinary Journal of Special Care Practices, 25, 188–212. doi:10.1097/IYC.0b013e31825a1ebf

23.

Snyder

Hemmeter

M. L.

McLean

M. E.

Sandall

McLaughlin . (2013). Embedded instruction to support early learning in response to intervention frameworks. In Buysse

Peisner-Feinberg

E. S.

(Eds.), Handbook of response to intervention in early childhood (pp. 283–300). Baltimore, MD: Brookes.

24.

Squires

Twombly

Bricker

Potter

(2009). Ages and Stages Questionnaires [ASQ-3] (3rd ed.). Baltimore, MD: Brookes.

25.

VanDerHeyden

A. M.

Snyder

Smith

Sevin

Longwell

(2005). Effects of complete learning trials on child engagement. Topics in Early Childhood Special Education, 25, 81–94.

26.

Vargus

E. A.

Vargus

J. S.

(1991). Programmed instruction: What it is and how to do it. Journal of Behavioral Education, 1, 235–251. doi:10.1007/BF00957006

27.

Wright

M. R.

Ellis

D. N.

Baxter

(2012). The effect of immediate or delayed video-based self-evaluation on Head Start teachers’ use of praise. Journal of Research in Childhood Education, 26, 187–198. doi:10.1080/02568543.2012.657745