Evaluating the Evidence Base of Performance Feedback in Preservice Special Education Teacher Training

Abstract

Performance feedback is commonly used during field experiences to improve desired teaching behaviors in preservice teachers. The authors identify eight single-subject studies examining the effects of performance feedback in preservice teachers to determine the evidence base for this practice. These eight studies are reviewed using quality indicators specific to single subject. Positive findings are shown across seven of the eight studies. Targeted behaviors showing the most change are those focused on implementation fidelity of evidence-based instructional strategies. Teacher-specific behaviors do improve after receiving performance feedback. However, the change in these behaviors was not consistent across all studies. Five studies met all rigorous standards of quality research and surpass additional requirements pertaining to evaluating single-subject research. The authors conclude that performance feedback is an evidence-based practice when used to increase implementation fidelity of instructional practices. Implications to the field of special education teacher preparation are discussed.

Keywords

performance feedback preservice teacher teacher quality quality indicators

Teacher quality is a national concern (U.S. Department of Education, Office of Postsecondary Education, 2011). Educational researchers share this concern and cite teachers as the most influential schooling factor affecting student achievement (Sass, Hannaway, Xu, Figlio, & Feng, 2010). Therefore, it would seem a logical conclusion that American students’ futures depend on a quality teaching force (Feiman-Nemser, 2001). Unfortunately, it remains uncertain how to define quality and thus unclear how to prepare quality teachers (Boe, Shin, & Cook, 2007; Darling-Hammond & Sykes, 2003; Harris & Sass, 2007). It is important that teacher preparation researchers determine effective ways to develop quality teachers. However, this will not be an easy task until we agree on the defining characteristics of a quality special educator and how to measure those characteristics.

Government and private agencies address educational concern of “quality” by attempting to conceptualize teacher quality through the lens of student achievement (U.S. Department of Education, Office of Postsecondary Education, 2011). Measuring teacher quality based on student outcomes can be especially problematic for special educators because many students with disabilities consistently under perform on standardized testing and methods for appropriately interpreting accommodated test scores are limited. Additionally, students with more severe disabilities are excluded from value-added models all together due to alternative assessments (Holdheide, Goe, Croft, & Reschly, 2010). In their policy brief, Holdheide and her colleagues (2010) described one promising method to measure special education teacher quality. The proposed formula recognizes the unique role of special educators by breaking the evaluation into five areas: (a) 50% teacher practice, (b) 30% quality of the Individualized Education Program (IEP) and meeting compliance timelines, (c) 10% measure of student growth, (d) 5% overall school-wide achievement, and (e) 5% commitment to the school community. Accordingly, it is important to use evidence-based practices when training special education teachers to develop desired behaviors of teacher practice as this is important within any construct of teacher quality.

Effective special education teacher preparation programs should deliver a “conscious blending of theory, disciplinary knowledge, subject-specific pedagogical knowledge and practice” (Brownell, Ross, Colon, & McCallum, 2005, p. 243). Field experiences allow preservice special educators to apply theoretical course work and implement evidence-based instructional practices they have been taught in authentic settings before they are alone in their own classroom (Darling-Hammond & Sykes, 2003). Therefore, it is important during field experiences that preservice teachers receive support from veteran mentor teachers and feedback from a representative of their university program to help hone skills and develop effective teaching practices (Brownell et al., 2005; Feiman-Nemser, 2001).

Using evidence-based practices for training special education teachers will ensure preservice teachers learn methods that have shown to be effective through empirical research. Single-subject research is often used in special education research to study the effectiveness of interventions because this type of research lends well to the individualized nature of special education (Horner et al., 2005). Rigorous single-subject research is structured to examine change in a dependent variable, over time by systematic introduction or manipulation of a well-defined independent variable (Kazdin, 2011). Quality research is the key to identifying teacher training techniques shown to be effective in developing quality special education teachers. Accordingly, teachers who are trained with evidence-based techniques will be better equipped to demonstrate effective practices and thereby positively impact student achievement. Therefore, it is the intention of this study to determine whether performance feedback is one such evidence-based practice.

The purpose of feedback is goal oriented. Teachers give feedback to students in an effort to close the gap between the current performance and the desired performance (Hattie & Timperley, 2007; Scheeler, 2008; Scheeler, Ruhl, & McAfee, 2004). As preservice teachers are students developing their knowledge of desired teacher behaviors, preparation programs should apply this training technique during field experiences. Therefore, if performance feedback is to be used as a training technique during teacher preparation the evidence base supporting performance feedback must be evaluated (Scheeler et al., 2004).

Before researchers begin to study effects of evidence-based practices, it is critical to develop a common understanding and construct of the practice. Hattie and Timperley (2007) reviewed 12 meta-analyses that assessed the influence of feedback on performance. They defined feedback as “information provided by an agent regarding aspects of one’s performance or understanding” (Hattie & Timperley, 2007, p. 81). Feedback draws attention to the performance and leads the trainee to reflect and engage with three questions: (a) What is the objective of my task? (b) How am I doing? and (c) How do I get to the next level? Scheeler and colleagues (2004) conducted a meta-analysis of preservice teacher feedback studies to determine what dimensions of feedback impacted preservice teacher behaviors and to operationally define feedback as it pertained to teacher preparation. Ten studies were reviewed to understand the content and delivery medium of the feedback offered to preservice teachers. We found that the medium (electronic, in person, peer, or supervisor) did not have as much impact on the preservice teachers as the content and immediacy of the feedback.

Hattie and Timperley (2007) reviewed 74 meta-analyses on feedback and found that under ideal conditions performance feedback had a combined .79 effect size. Ideal conditions occurred when the performance feedback was immediately offered during task acquisition, included specific goals, and was a constructive review of the current practice. Similarly, Scheeler and her colleagues (2004) indicated that the most influential component of performance feedback was immediacy, while specific, positive, and corrective were offered as promising components. Combining the findings of these two reviews allows for a clear understanding of performance feedback and enables us to identify a framework by which special education teacher preparation programs can provide consistent feedback during the field training experience.

Simply stated, performance feedback should include four essential components: immediacy, specificity, constructiveness, and purposefulness (Hattie & Timperley, 2007; Scheeler et al., 2004). Immediacy refers to the timeliness in which feedback is given. Researchers have found positive effects with real-time feedback using electronic receivers (ear buds; Scheeler, McAfee, Ruhl, & Lee, 2006), conferences held directly after an observation (Auld, Belfiore, & Scheeler, 2010; Capizzi, Wehby, & Sandmel, 2010; Mallette, Maheady, & Harper, 1999; Morgan, Gustafson, Hudson, & Salzberg, 1992), as well as emails within the same day (Barton & Wolery, 2007). For the purpose of this review, immediate is then defined as “the day of the observation.” Performance feedback has the greatest impact when recipients hear specific descriptions and analyses immediately following their performance, including how to correct their performance when necessary (Hattie & Timperley, 2007).

Specificity and constructiveness refer to the detail and manner of the supervisor’s statements. Specific praise or correction describes the desired behavior and provides clear examples for the recipient (Scheeler et al., 2004). Specificity helps the person receiving the feedback perform, replicate, and extend the desired behavior (Brophy, 1981). Performance feedback should be delivered in a positive manner and used to encourage the trainees to extend their knowledge or correct their misconceptions (Scheeler et al., 2004). In addition, performance feedback should directly relate to the behavior observed by a supervisor or trainer that leads to reflection of goals (Hattie & Timperley, 2007). Therefore, for the purposes of this evaluation, performance feedback is defined as a critique of observed behavior that is immediate, specific, positive, and corrective when needed, designed to move the recipient toward a desired performance.

In summary, special education teacher preparation programs should use best practices to prepare professionals who will greatly impact student learning by improving targeted teacher behaviors and increasing implementation fidelity. Performance feedback has resulted in increased targeted teacher behaviors (Auld et al., 2010; Barton & Wolery, 2007; Capizzi et al., 2010; Morgan et al., 1992) and improved implementation fidelity of instructional practices (Auld et al., 2010; Capizzi et al., 2010; Mallette et al., 1999; Morgan et al., 1992; Scheeler et al., 2006) during special education preservice teacher field experiences, but this research has never been evaluated for quality using rigorous research standards. Therefore, the purpose of this evaluation was to determine whether providing performance feedback to preservice special education teachers during their field experience is an evidence-based practice that improves targeted teacher-specific behaviors and increases implementation fidelity of evidence-based instructional practices. To further advance special education teacher preparation practices, we evaluated the use of performance feedback within special education single-subject research to determine its evidence base using Horner et al.’s (2005) quality indicators.

Method

Search and Selection Procedures

A comprehensive search for articles to review was conducted using four steps. First, an extensive search of four electronic databases, Education Resources Information Center (ERIC), Education Full Text, PsychINFO, and ProQuest Central, was conducted. Search terms included various combinations of the following: teacher, training, intern, student, behaviors, characteristics feedback, task specific, performance, corrective, observation, university, supervisor, mentor, peer, impact, and effect. Second, reference lists of articles obtained from the database search were reviewed to locate additional studies that might have been overlooked by database platforms. Third, a hand search of Teacher Education and Special Education and Journal of Teacher Education was conducted, as these journals are directly related to teacher education and likely included research on performance feedback. Finally, the reference lists of articles compiled after the initial search were also reviewed to locate any study that may have been overlooked in the preliminary steps. The comprehensive search resulted in 617 articles. Examining the abstracts to exclude the literature outside the field of education limited the number of relevant articles to 95, and eliminating duplicates reduced the pool of articles to 45.

Once we identified the relevant articles, we applied the following criteria to isolate studies suitable for an analysis of performance feedback as an evidence-based practice:

The article reported on a research study in which performance feedback was the focus of improving an identified teacher behavior.

The study was published in a peer-reviewed journal.

The study included preservice teachers as the target population.

The independent variable (intervention) examined was performance feedback as defined by immediate, specific, positive, and corrective when needed.

The dependent variable was measurable desired teacher behavior (i.e., increased fidelity of evidence-based practice, decreased ineffective practice).

The study used an experimental or quasi-experimental research design to show a causal relationship of performance feedback on teacher behaviors.

We reviewed the articles independently and then discussed each study to reach full agreement on the inclusion of studies based on the criteria listed above. Seven of the 45 articles met all inclusion criteria and represented 8 single-subject studies. Thirty-eight articles were excluded primarily because they did not include an experimental research design, including studies on performance feedback that used a descriptive qualitative approach. While this methodology is valuable for providing rich descriptions of preservice teacher’s experiences and providing an understanding of feedback techniques that preservice teacher think are effective, it does not allow researchers to examine whether there is a causal relationship between performance feedback and desired teacher behaviors. Consequently, these articles were useful in framing our definition of performance feedback, but were not used to examine the relationship between performance feedback and teacher practice.

After agreeing on the seven articles, we carefully reviewed each of the studies with respect to purpose and design by examining the settings, participants, dependent and independent variables, and results as shown in Table 1. As a whole, the studies all aimed to measure the improvements preservice teachers made after receiving performance feedback regarding their field experience, yet there were three major differences between studies. The first major difference between studies was the academic level of the participants. Thirty-four preservice teachers were represented in this review, 76.5% (n = 26) were undergraduates and 23.5% (n = 8) were graduate students. The second major difference involved the persons giving the feedback, which varied between supervisors, experts, researchers, and peers. The third main difference was the way feedback was delivered to the preservice teachers. In one study, a researcher provided instantaneous feedback from an ear bud; in two studies, researchers emailed feedback generated by observers; and in the remaining five studies, feedback was delivered via personal conferences. Within the five studies where conferences occurred, one conference included video evidence and feedback from an expert, two conferences consisted of feedback via peer coaching, and two conferences were university personnel providing feedback to preservice teachers. Although these differences were noted, no further exclusions were made because the intervention itself, providing performance feedback to preservice teachers, remained constant based on our definition of immediate, specific, positive, and corrective when needed.

Table 1.

Overview Studies.

Study	Design	Participants	Nature of feedback	Instructional target	Targeted behavior	Results
Auld, Belfiore, and Scheeler (2010)	Multiple baseline	7 UG	Face-to-face conference with supervisor using frequency data	Implantation fidelity	DRA—Planned ignoring for 10 seconds followed by corrective feedback and then acknowledging correct behavior (e.g., hand raising)	Following intervention workshop and performance feedback sessions, all participants demonstrated implementation fidelity of the DRA procedure
Barton and Wolery (2007)	Two separate multiple baseline	3 UG/3 G	Email feedback from supervisor with list of targeted behaviors observed: Both examples and nonexamples provided	Teacher-specific behaviors	Expansions—A verbal response by the teacher based on the child’s utterance and current focus of attention that expands or elaborates through information provided or in grammatical usage	Teachers all demonstrated some increase in use of expansions at the beginning of the intervention but then all began to decrease again toward the end of the intervention
Capizzi, Wehby, and Sandmel (2010)	Multiple baseline	3 G	Face-to-face conference with expert using video of teacher observation as evidence	Implementation fidelity and teacher-specific behaviors	Increased % of teacher fidelity of lesson components, teacher behavior–specific praise, and OTR	Mixed results—The intervention had positive impact on two behaviors across participants. One behavior, OTR, did not increase
Mallette, Maheady, and Harper (1999)	Multiple baseline	6 UG (3 pairs)	Face-to-face conferences with peer coaches using reciprocal coaching to deliver performance feedback	Implementation fidelity	PALS—preservice teachers and student dyads engaged in four components of PALS	All participants increased their accuracy of PALS. From 63%-80% at baseline to 81%-96% after intervention—Mean increase of 15%
Morgan, Gustafson, Hudson, and Salzberg (1992)	Multiple baseline	5 UG (and 3 UG coaches)	Face-to-face conferences between participants and “High Achiever” peer coaches	Implementation fidelity and teacher-specific behaviors	Six effective behavior (i.e., presentation of instruction, correct “signal system”) and seven ineffective practices (i.e., unwarranted feedback, negative verbal feedback)	Frequency of effective behaviors increased across participants, especially when coach attended to participants’ grade performance
Rathel, Drasgow, and Christle (2008)	Multiple baseline	2 G	Face-to-face supervisor conferences	Teacher-specific behaviors	Teacher behavior–specific praise	Inconclusive
Scheeler, McAfee, Ruhl, and Lee (2006)	Multiple baseline	5 UG	Immediate praise and corrective feedback provided through “Bug in Ear” electronic receiver	Implementation fidelity	Three-term contingency trials—Antecedent: Teacher asks question, Response: Student answer, Consequence: Teacher responds	Percentage of completion increased for all teachers, with a range from 30 to 92

Note. UG = undergraduate students; DRA = differential reinforcement of alternative behavior; G = graduate students; OTR = opportunities to respond; PALS = peer assisted learning strategies.

Identifying Quality Indicators

Intervention research aims to demonstrate that the desired change in participants’ behavior was caused by the intervention and not because of any other plausible explanation. Kratochwill and colleagues (2010) stated that the goal of single-subject research is to determine the effectiveness of an intervention to change behavior under specific conditions. Kazdin (2011) asserted that causal inferences can be made in multiple-baseline research if the intervention is implemented at different times across subjects, and a visual difference in the dependent variable is displayed at or immediately following implementation. Accordingly, it is fitting to use single-subject research in determining the evidence base of performance feedback.

To systematically evaluate the quality of single-subject research, Horner et al. (2005) recommended quality standards, which were chosen for this review because they are widely accepted in the field of special education as they have been successfully used to determine the evidence base of several interventions (see Browder, Ahlgrim-Delzell, Spooner, Mims, & Baker, 2009; Chard, Ketterlin-Geller, Baker, Doabler, & Apichatabutra, 2009; Montague & Dietz, 2009). Horner et al. provided 18 quality indicators within seven domains for evaluating single-subject studies (see Table 2).

Table 2.

Quality Indicators of Single-Subject Design.

	Auld, Belfiore, and Scheeler (2010)	Barton and Wolery (2007); Exp I	Barton and Wolery (2007); Exp II	Capizzi, Wehby, and Sandmel (2010)	Mallette, Maheady, and Harper (1999)	Morgan, Gustafson, Hudson, and Salzberg (1992)	Rathel, Drasgow, and Christle (2008)	Scheeler, McAfee, Ruhl, and Lee (2006)
Participants	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Described sufficiently to replicate	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Selection described sufficiently to replicate	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Setting described sufficiently to replicate	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Dependent variable	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes
Dependent variable described sufficiently to replicate	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Quantifiable	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Measurement valid and described sufficiently to replicate	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Measurement occurred repeatedly	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
IOA data reported and met minimal standards	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes
Independent variable	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes
Independent variable described sufficiently to replicate	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Systematically manipulated	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Procedural fidelity measured and described	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes
Baseline	Yes	Yes	No	Yes	Yes	Yes	No	Yes
Conditions described sufficiently to replicate	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes
Baseline provided evidence of pattern prior to intervention	Yes	Yes	No	Yes	Yes	Yes	No	Yes
Experimental control/internal validity	Yes	No	No	Yes	Yes	Yes	No	Yes
Three demonstrations of experimental effect	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes
Design controlled threats to internal validity	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Pattern that demonstrates experimental control (as judged by visual analysis)	Yes	No	No	Yes	Yes	Yes	Yes	Yes
External validity	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes
Effects replicated across participants	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes
Social validity	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes
Dependent variable socially important	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes
Magnitude of change in dependent variable due to intervention	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Independent variable is cost-effective	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Independent variable implemented over time, in typical contexts, and by typical intervention agents	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Note. Quality Indicators were used as proposed by Horner et al. (2005). IOA = interobserver agreement.

Exp I = The first study reported within Barton and Wolery (2007); Exp II = The second study reported within Barton and Wolery (2007).

Applying Quality Indicators

The eight studies were coded using Horner et al.’s (2005) standards and the following guiding questions adapted from Table 1:

Participants and setting addresses three questions: (a) are the participants described sufficiently to be replicated by the reader, (b) is the selection of participants described sufficiently to be replicated by the reader and (c) is the setting of the study described sufficiently to be replicated by the reader.

Dependent Variable addresses five questions: (a) was it described well enough to be replicated, (b) was it observable, (c) how was it measured, (d) how often it was measured, and (e) was interobserver reliability reported and did it meet minimum levels of acceptability (e.g. IOA= 80%; Kappa = 60%). For the purposes of this review the Dependent Variable was judged against the description of desired teacher behaviors of interest.

Independent Variable addresses three questions: (a) was it described sufficiently enough to be replicated, (b) was it systematically manipulated, and (c) was procedural fidelity described and measured.

Baseline addresses two questions: (a) was the condition of baseline described sufficiently as to be replicated by the reader and (b) was evidence provided regarding baseline patterns and trends.

Experimental Control/Internal Validity addresses three questions: (a) were there three demonstrations of experimental effect, (b) did the design control for threats to internal validity, and (c) did the results indicate a pattern that demonstrated experimental control, as judged by visual analysis.

External Validity addresses if experimental effects were replicated across participants.

Social Validity addresses four questions: (a) was the dependent variable socially important, (b) was the magnitude of change in the dependent variable caused by the intervention, (c) was the intervention cost effective, and (d) was the independent variable feasible to implement. (p. 174)

Each domain was rated as Yes only when all the guiding questions within that domain received a Yes. If any one indicator within a domain was missing, the domain was rated as No. We independently reviewed all eight studies for quality and then together reviewed individual scores to calculate interrater reliability. Our reliability score reflects the percentage of unanimous agreement on the quality indicators for each study. We reached 100% agreement on five studies, 94.4% agreement on two studies, and 88.9% agreement on one study, with an average interrater reliability score of 97.2%. When we did encounter disagreements, we referred back to the Horner et al. (2005) model of quality standards and conferenced to reach consensus.

To guide our decision for Domain 5, Experimental Control/Internal Validity, we applied Kratochwill et al.’s (2010) rules for conducting visual analysis for single-subject research as outlined for What Works Clearinghouse. Their rules consist of four steps: (a) verify a documented predictable pattern during baseline, (b) determine there is sufficient consistent data within each phase to project subjects’ response patterns, (c) compare data patterns in similar phases to determine that the independent variable was “effecting” the change, and (d) integrate data across phases to determine that the effect is replicated three or more times. Furthermore, to establish data patterns, six variables were used: (a) level, the mean score of the data points within each phase; (b) trend, the slope of best-fit line; (c) variability, the range of data points within a phase; (d) immediacy of effect, the change in “level” between last three data points and first three data points in adjoining phases; (e) overlap, the proportion of data that is represented in adjoining phases; and (f) consistency of data patterns across similar phases.

Through reviewing each study, we discovered an important distinction in the outcomes measuring desired teacher behaviors. As shown in Table 1, the targeted outcomes fell into two mutually exclusive categories, either implementation fidelity of evidence-based instructional strategies or teacher-specific behaviors. Implementation fidelity included components of direct instruction, differential reinforcement of alternative behaviors (DRAs), three-term contingency trials, and peer assisted learning strategies (PALS). Teacher-specific behaviors included opportunities to respond (OTR), verbal expansions, behavior-specific praise (BSP), and providing student corrections. The determination was made to evaluate the two categories individually because this would allow for a more clear understanding of what specific changes occurred for the preservice teachers, if any, as a result of receiving performance feedback. The two outcome categories were mutually exclusive, but the studies were not discrete because two studies measured outcomes in both categories. In these two studies, researchers provided separate narrative and visual analysis for each outcome. The detailed reporting enabled us to better understand the effect performance feedback had on each desired teacher behavior.

In addition, Horner et al. (2005) provided three supplemental standards to measure the quality of research for a group of studies. These three group standards must be met for the body of research to warrant the intervention as evidence-based. Horner et al. (2005) provided three group standards:

A practice may be considered evidence based when (a) minimum of five single-subject studies that meet minimally acceptable methodological criteria and documented experimental control have been published in peer-reviewed journals, (b) the studies are conducted by at least three different researchers across at least three different geographical locations, and (c) the five or more studies include a total of at least 20 participants. (p. 176)

The group standards were applied to the body of research for each category separately after the individual studies were rated using the quality indicators. We agreed prior to the review that “minimally acceptable” would apply when studies met 100% of the quality indicators. There were no disagreements when applying the three group standards.

Results

Research Findings

Overall, there were 34 participants across the eight studies. However, Mallette et al. (1999) presented their findings by dyads because the intervention was reciprocal peer coaching, resulting in 31 measured participant outcomes. The results of the eight studies demonstrated positive outcomes for the majority of participants. Considering the researchers’ reported findings and using Kratochwill et al.’s (2010) rules for visual analysis, we determined that 90% (n = 28) of preservice teachers across all eight studies increased desired teaching behaviors in some capacity during the intervention phase.

The use of performance feedback to increase desired teachers behaviors demonstrated more positive outcomes for implementation fidelity of evidence-based instructional strategies as compared with the effects of performance feedback on teacher-specific behaviors (see Figure 1). Specifically, 95% (n = 21) of the participants in the five studies that targeted implementation fidelity of evidence-based instructional strategies displayed a positive trend throughout the intervention phase. Participants all increased their accuracy and consistency of performance when measuring implementation fidelity of DRA (Auld et al., 2010), lesson components of direct instruction (Capizzi et al., 2010; Morgan et al., 1992), PALS (Mallette et al., 1999), and three-term contingency trials (Scheeler et al., 2006).

Figure 1.

Outcomes of reviewed studies.

On the other hand, 53% (n = 10) of the participants from the five studies that targeted teacher-specific behaviors displayed a positive trend throughout the intervention phase. Participants experienced inconsistent improvement across studies or no meaningful gains in desired performance when measuring teacher-specific behaviors such as expansions (Barton & Wolery, 2007), BSP (Barton & Wolery, 2007; Capizzi et al., 2010; Rathel, Drasgow, & Christle, 2008), and OTR (Capizzi et al., 2010). The research findings do support performance feedback effects positive changes in desired teacher behaviors in both categories. As stated, more consistent change was demonstrated with implementation fidelity. However, providing performance feedback to preservice teachers led to positive change in both categories of desired teacher behaviors. Therefore, we proceeded to review the quality of all eight studies.

Research Quality

All eight studies were reviewed using the seven domains and corresponding quality indicators. Five of the eight studies (Auld et al., 2010; Capizzi et al., 2010; Mallette et al., 1999; Morgan et al., 1992; Scheeler et al., 2006) met 100% of the rigorous standard of quality single-subject research (Table 2). These were the same five studies where the effects of performance feedback were measured through changes in implementation fidelity of evidence-based instructional strategies. Two of these studies also included a measure of change in teacher-specific behaviors, but the other three studies that measured the effects of performance feedback on teacher-specific behaviors did not meet 100% of the standards for quality research as measured by Horner et al.’s (2005) quality indicators.

The five studies that measured changes in implementation fidelity were reviewed using the three supplementary group standards. These studies met the first group standard because individually they rated Yes in all seven domains of the quality standards. The five studies measuring changes in implementation fidelity also met the second group standard because there were at least three different researchers (n = 17) in at least three different geographical locations (n = 4). The third group standard was also met because the preservice teacher participants surpass the minimum requirement of 20 (n = 26). Taken together, the five studies demonstrating the effects of performance feedback as measured by changes in implementation fidelity met 100% of the group standards required for determining a practice to be evidence-based using single-subject research.

The supplementary group standards were then applied to the five studies that measured changes in teacher-specific behaviors. These studies did not meet the first group standard because three studies within the group did not rate Yes in all seven domains of the quality standards. The second standard was met because there were at least three different researchers (n = 11) in at least three different geographical locations (n = 4). In addition, the third group standard was not met because the preservice teacher participants did not meet the minimum requirement of 20 (n = 16). Subsequently, only one of the three supplementary group standards was met for the five studies that measured changes in teacher-specific behaviors.

Discussion

In reviewing the located research on performance feedback delivered to preservice teachers, we found a range in quality of the reported studies. Scheeler and colleagues (2006) conducted the most rigorous study of those targeting teacher-specific behaviors. They systematically replicated positive trends within the intervention phase across all five participants as well as demonstrated the sustainability of changes during the maintenance phase. This study not only demonstrated the high-quality methodological standard set by Horner and colleagues (2005) but also provided evidence of continued levels of desired teaching behaviors without the intervention, demonstrated in a maintenance phase, this study made the strongest causal argument for performance feedback in relation to changes in teacher-specific behaviors. Scheeler and colleagues’ (2006) high-quality study likely demonstrated the true promise of performance feedback to positively impact teacher-specific behaviors.

Studies targeting change in teacher-specific behaviors should be reviewed to determine whether the research could be attempted again with reported weaknesses in mind for future improvement. For example, one common shortcoming was the failure to meet Domain 5, Experimental Control.

Barton and Wolery (2007) did not clearly demonstrate experimental control, as evidenced by a lack of baseline trend for one participant, unstable improvements of the targeted teacher behavior during the intervention phase, and a weak narrative explaining their methods for controlling threats to validity. Similarly, Rathel and colleagues (2008) analyzed the effects of performance feedback on only two participants, did not demonstrate a stable baseline prior to the intervention, and only reported interobserver agreement during the baseline of one participant limiting the implications of their findings. Horner et al. (2005) specifically stated that interobserver agreement should be established during each phase to ensure that the dependent variable is consistently measured.

Rathel and colleagues (2008) had inconclusive findings because the positive changes were not replicated across the two participants in the study. The first participant did demonstrate increased use of BSP, however a stable baseline was not established limiting conclusions about the direct effect of the intervention on the instructional targets. The second participant initially increased the use of BSP during the intervention phase, but the increases were not sustained throughout the intervention phase. Similarly, Barton and Wolery (2007) reported initial increases in the use of the teacher-specific behavior, expansions, at the beginning of the intervention, but half of the participants did not sustain these increases throughout the intervention phase. Positive changes were initially reported and replicated for two of the three behaviors across all participants within the study, but the addition of a third behavior likely overloaded participant focus, as evidence by flat lined trends. Similarly, Capizzi et al. (2010) also noted that the addition of a third behavior did not provide an increase in changed teacher-specific behaviors. Morgan and his colleagues (1992) were able to circumvent this problem by grouping behaviors together as either effective or ineffective behaviors. However, due to this grouping, it is difficult to determine whether one or more behaviors were observed during the targeted lesson. The reported positive outcomes could also be due to the measured effectiveness of protocols associated more with implementation fidelity than with teacher-specific behaviors.

Determination of Evidence Base

Increasing implementation fidelity of evidence-based instructional strategies

The findings in this review provide valuable information about the effects of performance feedback in training special education teachers. The five studies that targeted implementation fidelity of evidence-based instructional strategies (Table 1) displayed positive trends throughout the intervention phases across all five studies for all but one participant. These findings strongly support previous research on performance feedback suggesting that performance feedback during field experience can lead to increased implementation fidelity of evidence-based instructional strategies. The same five studies also met the rigorous quality standards and additionally met the standards recommended for evaluating a group of research (Figure 1). Therefore, due to the research findings and quality, performance feedback provided to preservice teachers for the purpose of increasing implementation fidelity can be supported as an evidence-based practice.

Improving teacher-specific behaviors

The majority of participants in the five studies targeting teacher-specific behaviors also exhibited positive trends throughout the intervention phase. However, the figures provided by researchers indicated that the change in teacher-specific behaviors was not consistent across studies. Possible causes for the inconsistent findings include (a) the construct of the behavior may not have been clear enough to determine it as present or not (see Rathel et al., 2008), (b) the behavior may not have lent itself to observation in every lesson (see Capizzi et al., 2010), or (c) focusing on too many behaviors could have been a distraction for the preservice teacher or supervisor (see Barton & Wolery, 2007; Capizzi et al., 2010). Furthermore, only two of the five studies that targeted teacher-specific behaviors met all the rigorous quality standards required to deem performance feedback and evidence-based practice to improve targeted teacher behaviors (Figure 1). Due to the research findings and missing quality indicators in three of the five studies, performance feedback provided to preservice teachers for the purpose of improving teacher-specific behaviors cannot be supported as an evidence-based practice. Nonetheless, based on the positive change in the majority of participants and the quality research that was presented, we do consider this a promising practice that warrants further investigation.

Summary and Implications for Future Research

Eight experimental studies on performance feedback were reviewed to determine whether providing performance feedback to preservice teachers during their field experience is an evidence-based practice. All the studies included in this review supported the use of performance feedback in preservice teacher training. The five studies that measured changes in implementation fidelity met rigorous guidelines for high-quality research, and can support that performance feedback is an evidence-based practice when training special education preservice teachers. This is important for special education teacher preparation because evidence-based instructional strategies provide the most promise for educational growth of students with disabilities, and it is imperative that teachers implement these practices correctly and consistently (Burns & Ysseldyke, 2009). Performance feedback that is immediate, specific, positive, and corrective when needed should be included in every special education teacher training program to increase educators correct implementation of evidence-based practices while teaching.

The review did not support performance feedback as an evidence-based practice in improving teacher-specific behaviors because of inconsistent findings and missing quality indicators within the group of studies reviewed even though two studies from this group (Capizzi et al., 2010; Morgan et al., 1992) presented positive outcomes of improved teacher-specific behaviors and met all of the quality indicators. Had the reporting from these five studies met all quality indicators, we would better understand the effect performance feedback had on improving teacher-specific behaviors of preservice special education teachers. Therefore, we recommend teacher preparation programs continue to implement, study, and report the effects of performance feedback in the training of teacher-specific behaviors using Horner et al. (2005) as a guide. Specifically, we urge researchers to conduct quality studies measuring the effects of performance feedback on specific teacher behaviors including OTR, verbal expansions, BSP, and providing student corrections.

We also recommend studying performance feedback as a training tool used by mentor teachers in addition to university supervisors, as mentor teachers have daily contact with preservice teachers throughout the field placement experience. The mentor–student teacher relationship plays a significant role in training preservice teachers (O’Brian, Stoner, Appel, & House, 2007). The mentor teacher is instrumental in bridging the preservice teacher’s theoretical knowledge with their application of evidence-based instructional strategies. By training mentor teachers to use performance feedback in the same consistent manner, it may strengthen the field placement experience for preservice teachers.

In this review, we found mixed results in determining performance feedback an evidence-based practice. Performance feedback is a fundamental method used by university faculty to train and develop preservice teachers and should be studied further. Just as identifying practices that are evidence-based is necessary for improving the outcomes of students with disabilities, it is also essential to identify evidence-based practices in training and developing the professionals that teach these students. Consequently, we urge all researchers to use quality standards such as those recommended by Horner et al. (2005) as a guide for designing future research studies that will inform the field of the true potential of performance feedback.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Kyena E. Cornelius is a doctoral student at Johns Hopkins Univeristy in the Deparment of Special Education. She is currently researching special education teacher preparation and the induction of novice special educators focusing on the relationship between early career teachers and their mentors.

Sarah A. Nagro is a doctoral student at Johns Hopkins Univeristy in the Deparment of Special Education. She is currently researching special education teacher preparation including methods for teaching content areas to students with high incidence disabilities and the use of video in teacher development.

References

*Auld

R. G.

Belfiore

P. J.

Scheeler

M. C.

(2010). Increasing pre-service teachers’ use of differential reinforcement: Effects of performance feedback on consequences for student behavior. Journal of Behavioral Education, 19, 169-183.

*Barton

E. E.

Wolery

(2007). Evaluation of e-mail feedback on the verbal behaviors of pre-service teachers. Journal of Early Intervention, 30, 55-72.

Boe

E. E.

Shin

Cook

L. H.

(2007). Does teacher preparation matter for beginning teachers in either special or general education? The Journal of Special Education, 41, 148-170. doi:10.1177/00224669070410030201

Brophy

(1981). Teacher praise: A functional analysis. Review of Educational Research, 51, 5-32.

Browder

Ahlgrim-Delzell

Spooner

Mims

P. J.

Baker

J. N.

(2009). Using time delay to teach literacy to students with severe developmental disabilities. Exceptional Children, 75, 343-364.

Brownell

M. T.

Ross

D. D.

Colon

E. P.

McCallum

C. L.

(2005). Critical features of special education teacher preparation: A comparison with general teacher education. The Journal of Special Education, 38, 242-252.

Burns

M. K.

Ysseldyke

J. E.

(2009). Reported prevalence of evidence-based instructional practices in special education. The Journal of Special Education, 34, 3-11.

*Capizzi

A. M.

Wehby

J. H.

Sandmel

K. N.

(2010). Enhancing mentoring of teacher candidates through consultative feedback and self-evaluation of instructional delivery. Teacher Education and Special Education, 33, 191-212.

Chard

D. J.

Ketterlin-Geller

L. R.

Baker

S. K.

Doabler

Apichatabutra

(2009). Repeated reading interventions for students with learning disabilities: Status of the evidence. Exceptional Children, 75, 263-281.

10.

Darling-Hammond

Sykes

(2003). Wanted, a national teacher supply policy for education: The right way to meet the “Highly Qualified Teacher” challenge. Education Policy Analysis Archives, 11, Article 33. Retrieved from http://epaa.asu.edu/ojs/article/view/261

11.

Feiman-Nemser

(2001). From preparation to practice: Designing a continuum to strengthen and sustain teaching. Teachers College Record, 103, 1013-1055.

12.

Harris

D. N.

Sass

T. R.

(2007). Teacher training, teacher quality and student achievement (Working Paper 3). Washington, DC: National Center for Analysis of Longitudinal Data in Education Research.

13.

Hattie

Timperley

(2007). The power of feedback. Review of Educational Research, 77, 81-112.

14.

Holdheide

L. R.

Goe

Croft

Reschly

D. J.

(2010). Challenges in evaluating special education teachers and English language learner specialists. Washington, DC: National Comprehensive Center for Teacher Quality Research and Policy Brief.

15.

Horner

R. H.

Carr

E. G.

Halle

McGee

Odom

Wolery

(2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165-179.

16.

Kazdin

A. E.

(2011). Single-case research designs: Methods for clinical and applied settings (2nd ed.). New York, NY: Oxford University Press.

17.

Kratochwill

T. R.

Hitchcock

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2010). Single-case designs technical documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_scd.pdf

18.

*Mallette

Maheady

Harper

G. F.

(1999). The effects of reciprocal peer coaching on preservice general educators’ instruction of students with special learning needs. Teacher Education and Special Education, 22, 201-216.

19.

Montague

Dietz

(2009). Evaluating the evidence base for cognitive strategy instruction and mathematical problem solving. Exceptional Children, 75, 285-302.

20.

*Morgan

R. L.

Gustafson

K. J.

Hudson

P. J.

Salzberg

C. L.

(1992). Peer coaching in a preservice special education program. Teacher Education and Special Education, 15, 249-258.

21.

O’Brian

Stoner

Appel

House

J. J.

(2007). The first field experience: Perspectives of preservice and cooperating teachers. Teacher Education and Special Education, 30, 264-275.

22.

*Rathel

J. M.

Drasgow

Christle

C. C.

(2008). Effects of supervisor performance feedback on increasing preservice teachers’ positive communication behaviors with students with emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 16, 67-77.

23.

Sass

Hannaway

Figlio

Feng

(2010). Value added of teachers in high-poverty schools and lower-poverty schools (Working Paper 52). Washington, DC: National Center for Analysis of Longitudinal Data in Education Research.

24.

Scheeler

M. C.

(2008). Generalizing effective teaching skills: The missing link in teacher preparation. Journal of Behavior Education, 17, 145-158.

25.

*Scheeler

M. C.

McAfee

J. K.

Ruhl

K. L.

Lee

D. L.

(2006). Effects of corrective feedback delivered via wireless technology on preservice teacher performance and student behavior. Teacher Education and Special Education, 29, 12-25.

26.

Scheeler

M. C.

Ruhl

K. L.

McAfee

J. K.

(2004). Providing performance feedback to teachers: A review. Teacher Education and Special Education, 27, 396-407.

27.

U.S. Department of Education, Office of Postsecondary Education. (2011). Preparing and credentialing the nation’s teachers: The secretary’s eighth report on teacher quality based on data provided for 2008, 2009 and 2010. Washington, DC: U.S. Government Printing Office.