Abstract
The studies in the special series focus on the measurement of teacher practices and approaches for facilitating and evaluating school personnel’s use of data to inform instructional decisions that promote student achievement. In this commentary, the importance of conducting research on professional development and coaching is highlighted. The need to consider a broader framework within which professional development and coaching are contextualized and directions for future research are discussed.
The Institute of Education Sciences (IES) within the U.S. Department of Education is one of the primary sources of federal funding to support educational research. In March 2018, Dr. Mark Schneider was appointed as the director of IES. If you have read his blog or heard him speak, you will know that he is deeply concerned that many teachers do not believe federally funded research has a tangible impact on classrooms and schools. His aim as director is to guide the agency in solving what he calls the “last mile” problem—getting information into the hands of people who need it the most. To do this, he is prioritizing that IES-supported researchers focus on making research findings useful, usable, and used (Schneider, 2018).
Research focused on professional development (PD) and coaching is one promising line of inquiry that has the potential to improve researchers’ abilities to develop effective tools that teachers may value and desire. Kraft, Blazar, and Hogan (2018) recently conducted a meta-analysis in which they examined the effect of coaching on instruction and student achievement. Combining results across 60 studies that used causal designs, they found pooled effect sizes of 0.49 standard deviations (SD) on instruction and 0.18 SD on achievement. Although these results are encouraging, the authors noted that substantial variability exists across programs and they highlighted several limitations that should be addressed by future research. For example, education researchers need (a) to understand which aspects of coaching are associated with better outcomes (for both teachers and students), (b) to know whether coaching needs to differ across content areas (e.g., do effective coaching models in literacy work as well in math or social studies), and (c) more knowledge about components of PD and coaching that are conducive to bringing these approaches to scale (Kraft et al., 2018).
Research on these topics is important because when products from education research (i.e., interventions, assessments) are implemented in school contexts, they enter a dynamic, complex system. Understanding the multifaceted interactions between instructional coaches, teachers, and students has great potential to expand educational researchers’ ability to design interventions that can span the “last mile” and contribute to closing the perennial research-to-practice gap. Furthermore, as Kraft et al. (2018) emphasized, a conservative estimate of the amount K–12 schools currently spend on PD puts the amount in the tens of billions of dollars. Clearly, our society needs to better understand this investment.
The articles in the March 2019 special series (Assessment for Effective Intervention, Volume 44, Issue 2) are initial studies designed to extend understanding of PD and coaching. The special series is aligned with Schneider’s aims for increasing the impact of educational research. The series included research focused on methods to enhance PD and related coaching to improve teachers’ abilities to impact student outcomes. In this commentary, we first describe the studies in the special series within the context of a framework that depicts the possible interactions that could be examined in schools and in educational research. We follow this with future directions for research.
Studying the Complex Systems of Teacher Practice
Many of the complex interactions in research focused on understanding how to support teachers in improving student outcomes are represented in Figure 1. Although we acknowledge that this framework is likely incomplete and that numerous other features of the education system could be integrated, this represents a starting point for elements that could be considered during the design of research investigating PD and coaching models. The primary foci of this work often include evaluations of coaches’ and teachers’ knowledge and skill, fidelity or quality, and their perceptions and values; student outcomes and dosage are often included, with fewer studies examining students’ perceptions or values. Interactions with other key stakeholders (e.g., researcher, supervisor/district administration, parents) are less frequently considered.
Framework representing interactions in professional development and coaching research.
In the studies included in the special series, Kettler and Reddy (2019) focused their work on Path A (Figure 1) by demonstrating that a novel scoring method for the Framework for Teaching (Danielson, 2013), a frequently used teacher evaluation rubric, generated composite that were internally consistent, stable across time, and predictive of academic outcomes. Dudek, Reddy, Lewka, Hua, and Fabiano (2019) demonstrated that instructional coaching that was supported by a teacher formative assessment was associated with changes in teachers’ behavior management and high levels of satisfaction with the coaching. This study primarily explores Path B (see Figure 1) and includes feedback associated with Path C (see Figure 1).
Glover, Reddy, Kurz, and Elliott (2019) described an online coaching platform and reported on initial data examining Paths A and B (see Figure 1). Specifically, the authors shared findings in which the coaching behaviors of “facilitating teacher practice” and “providing feedback” were associated with reductions in the instructional gap in teachers’ classrooms; “facilitating teacher practice” was also predictive of fidelity of implementation and quality. Furthermore, they reported data in which specific coaching actions were predictive of student outcomes on end-of-year assessments. Reddy, Glover, Kurz, and Elliott (2019) focused their study on Path C (see Figure 1) by conducting the initial investigation of the utility and validity of a teacher completed assessment of instructional coaching (i.e., Instructional Coaching Assessments–Teacher Forms). Reliability indicated that the assessment demonstrated high internal consistency and data suggested the measure is relatively free from item bias. Furthermore, teachers and coaches reported that the measure was easy to use and helpful; teachers suggested the tool could be used to support positive experiences during coaching. The authors described the teacher measure as one component of a 360-degree or comprehensive multirater instructional coaching system. The authors are currently conducting research with two remaining assessment forms that will capture feedback from supervisors and coaches to complete the comprehensive assessment.
The work reported in the special series is important in that it increases understanding of critical features of PD, instructional coaching, and student outcomes. More specifically, the studies focused in on a set of tools to measure and support coaching and teachers’ instruction with the aim of using data to inform instruction. However, as represented in Figure 1, there are numerous relationships between critical features that are not captured in this body of work. Next, we outline three potential areas for future research that would extend the foundational science captured in the special series.
Future Directions
Measurement
As in many areas of research, measurement related to PD and coaching poses a critical challenge. Kettler and Reddy (2019) and Dudek et al. (2019) both contributed work in this area. Kettler and Reddy evaluated a novel scoring method for a measure to capture teacher practices. Although the measure demonstrated technical adequacy, it predicted only a portion of growth in achievement, indicating a need for additional measures to fully understand the interactions between teachers’ instruction and student outcomes. Dudek et al. demonstrated that coaches’ use of data was associated with changes in teachers’ behavior management but not in academic instruction. In both studies, the measures included are promising, but insufficient in that they assess only a limited number of potentially important factors.
Supplementary measures are needed to allow researchers to capture additional elements included in Figure 1. For example, many researchers successfully design and implement fidelity of implementation rubrics for coaches and instructors. However, these infrequently integrate direct measures of knowledge or perceptions and values. The scales are often blunt instruments that do not allow for a critical analysis of which aspects (e.g., knowledge and skill, fidelity and quality, perceptions and values) are most associated with effective delivery of an intervention or improvements in student outcomes. And, as studies are often not conducted across multiple years, elements associated with sustainability are not well understood.
There are at least three potential areas to focus on related to measurement. First, researchers could devote time to developing, piloting, and validation for measures that capture aspects of the framework that are infrequently studied (i.e., student perceptions and values). Second, future studies should include larger samples of participants so that researchers can analyze component or subscale scores on measures. This work would extend understanding of which aspects of coaching or instruction are most associated with change in student outcomes. Third, education researchers need measures that are sensitive to change in expertise over time and that capture variables that are associated with sustained implementation. Changing teacher behavior is difficult work and it is likely that multi-year studies will provide better insight in how to improve PD and coaching compared with single-year studies.
Relationships and Interactions in Schools
Another realm of future work relates to understanding which coaching behaviors and interactions are most supportive of changing teachers’ behaviors. Glover et al. (2019) advanced research in this area by exploring the use of an online platform to facilitate instructional coaching. Interestingly, their tool allowed the research team to analyze which specific coaching behaviors (i.e., providing feedback, structuring practice) were associated with change in teacher and student outcomes. Relatedly, Reddy et al. (2019) explored the technical adequacy of an assessment that allows researchers to collect data on teachers’ perceptions on the effectiveness of instructional coaching. Combined, both studies contribute to understanding of important aspects of the coach–teacher relationship that are critical to understanding teachers’ implementation of interventions and associated gains in student outcomes. However, as with measurement, additional work is needed to understand how other characteristics of the context in which teachers and coaches are working influence their relationship and student outcomes. For example, researchers could conduct studies to further understanding of the impact of school climate or administrator support on the effectiveness of coaches to support teachers.
When examining the interactions between instructional coaches and school staff, there are many “soft skills” that are critical to success. Coaches often have to play the roles of cheerleader, encourager, comforter, and enforcer—push the individuals receiving coaching hard, but not so hard that they cease to participate. These types of interactions are challenging to measure but capturing them could broaden education researchers’ understanding of which coaching behaviors are essential for success (and, perhaps, for which type of school staff member). Relatedly, those of us who are involved in intervention research in schools know that school leadership (i.e., campus administrators) impacts teachers’ fidelity to research protocols. Furthermore, when studying students with disabilities, education researchers are often interacting with general and special education teachers and perhaps other school staff (e.g., paraprofessional, speech language pathologist). It is likely that there are important aspects of these relationships (e.g., mutual respect, shared vision) that could extend understanding of the context in which interventions are most likely to succeed.
Education researchers also need studies that take into consideration a “bigger picture” view—incorporating more elements from the framework proposed in Figure 1. To understand how interventions fit into the complex systems of schools, studies will have to advance in their complexity. Jacobson, Levin, and Kapur (2019) recently proposed that educational researchers are limited by traditional quantitative and qualitative approaches because these methods are generally appropriate for studying linear aspects of educational systems. The authors suggest that computational modeling approaches, which are used in other fields to study nonlinear characteristics of complex systems, could advance educational research. Their conceptual framework focuses on two areas: “collective behaviors of a system and behaviors of individual agents in a system” (p. 113). The authors argue that incorporating new methodological tools that allow for complex systems analysis can improve educational researchers’ understandings of the dynamics of the systems education researchers study and that this work can inform educational policy by offering better predictions related to various approaches to systematic educational reform.
Changing (and Sustaining Change in) Teacher Behaviors
Another aspect of studying PD and coaching that needs additional attention is studying factors impacting teachers’ implementation. In a systematic review examining implementation studies across a range of fields (e.g., medicine, social sciences), Durlak and DuPre (2008) identified as many as 23 contextual factors that could impact implementation; however, they demonstrated that most studies only focused on a single dimension of implementation, primarily fidelity of implementation. In the field of education, we need to understand factors including why teachers decide to implement an intervention or not, how they determine whether they will make adaptations to interventions, and how their practice is influenced by factors that are not under their control (work climate, organizational norms).
Consider this anecdote: the first author was recently in an elementary school and re-encountered a teacher who had been a high-quality implementer of an early elementary reading program in a randomized controlled trial conducted several years earlier. The teacher remembered the author’s involvement with the project and shared how much she had enjoyed the intervention, how easy it was to implement, how much her students enjoyed the program, and how data she had collected indicated that it was effective for her students. The teacher was informed that the teacher’s manual had been updated and asked if she was still using the program. When she indicated “no,” she was asked for her rationale. Her reason was that she had not been invited to be in another study. Although the teacher had many positive reflections on the intervention, these were not sufficient to drive her to continue to use the program. Education researchers need a deeper understanding of reasons why teachers decide to sustain implementation, how school and district leaders’ decisions may supersede (or simply fail to reinforce) teacher decision making, and information researchers can provide to teachers to encourage sustained use of the interventions they perceive to be effective.
Changing practice, however, includes not only the learning of new practices but also discarding or amending old and outdated knowledge. Conceptual frameworks of implementation include behavior change at their core, but as the anecdote above demonstrates, education researchers have a limited understanding of how this change occurs in education. In the field of medicine, implementation researchers rely on use of behavior change theories to explain adoption and sustainability of new practices. The Theoretical Domains Framework (TDF) was developed by behavioral scientists and implementation researchers through identification and synthesis of 33 psychological theories related to behavior change (Cane, O’Connor, & Michie, 2012; Michie et al., 2005). Cane et al. (2012) conducted the initial validation study in which experts used closed and open sort tasks and then tested replication using discriminant content validation and fuzzy cluster analysis. This led to the development of a refined TDF that includes 14 domains, each coupled with an evidence-based behavior change technique—knowledge; skills; professional role and identity; beliefs about capabilities; optimism; beliefs about consequences; reinforcement; intentions; goals; memory, attention, and decision processes; environmental context and resources; social influences; emotion; and behavioral regulation. Some of these domains are familiar to educational researchers focused on teacher PD and coaching; however, others are generally not considered in our work. Each of these could likely be incorporated into an expanded Figure 1. For example, educational researchers could evaluate the influence of coach and teacher emotions on implementation, the impact of environmental context and resources on sustainability, or on the role that beliefs about consequences guide decisions made by supervisors and district administrators.
Determining the factors that influence a given behavior is essential for any successful attempt to change existing behavior. The TDF has been used to inform and address a wide range of implementation issues in health care practice (e.g., Dyson, Lawton, Jackson, & Cheater, 2011; Francis et al., 2009). This framework serves as a planning tool to help identify barriers (and potential targets) to successful implementation of certain practices. Adapting the TDF for use in educational research could extend understanding of malleable targets for changing teachers’ behaviors and it could provide a roadmap for factors to consider when designing instructional supports and coaching models. Furthermore, when low (or questionable) fidelity of implementation is observed, the TDF could provide guidance on critical questions to ask during an evaluation.
Conclusion
To solve the “last mile” problem, the field of education sciences needs to be bold and innovative. The success of U.S. schools is directly reflected in society as our students learn, grow, and engage in new pursuits. As such, more comprehensive research on PD and coaching should be designed to help teachers with the challenging task of meeting the needs of an increasingly diverse group of learners in a rapidly changing world. Furthermore, the field of education needs to ensure that our science is at the forefront of policy and decision making in education—and ultimately, that teachers adopt a scientific approach to their own practices.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.