Perverse incentives? A cautionary tale about coaching evaluation

Study context and the Evaluation Tool

Our study took place in a midsize urban district — which we will call Midtown — serving a socioeconomically and racially diverse student population (35% White/non-Latinx, 36% Black, 12% Latinx, 9% Asian, and 8% other). One full-time instructional coach was assigned to each school, and their primary responsibility was to provide individual as well as group professional development that would help teachers improve their instruction.

During the 2016-17 school year, the district implemented the new ET process. As part of this process, coaches and teachers were required to work with a group of students for a specified time frame (four or nine weeks), pick an area of instructional focus based on student data, and then gather pre- and post-student assessment data to determine the extent to which students met their growth goals. Teachers and coaches had to enter all ET-related information, such as individual student growth goals and pre-/post-assessment data, into an online portal that principals and district-level administrators could access. Depending on the amount of growth the students made, the coaches and teachers were assigned a certain level of proficiency (Excellent, Proficient, Needs Improvement, and Unsatisfactory). Given that the process required classroom data, it was most efficient for the coaches to complete it while engaging in a coaching cycle with a teacher. Furthermore, the coach and teacher could choose to complete their evaluation together, with identical data for student growth used for both.

We launched our study of coaches Meg and Claire while the Evaluation Tool was being implemented. Overall, we observed 15 planning or reflection meetings and 23 modeled or cotaught lessons that the coaches conducted with five elementary teachers, and we conducted 27 interviews with administrators, coaches, and teachers. By chance, two of the coaching cycles we observed involved the coaches completing their evaluation. In particular, Coach Meg worked with Teacher Michelle, and Coach Claire worked with Teacher Cathy to gather student assessment data and complete their ET requirements together. It became apparent to us that the ET seemed to have a strong hand in shaping coach-teacher interactions in these particular coaching cycles. What, specifically, did we observe?

Tensions

The purpose of the Evaluation Tool was to encourage teachers and coaches to use data to inform instructional decisions and to focus their attention on student growth. These are certainly important goals. However, we noticed two key ways in which the ET detracted from the coaches’ efforts to foster teacher improvement.

Coaches engaged in abnormally long coaching cycles. Although Midtown’s coaching cycles were typically two to four weeks long, the ET process was designed to take four to nine weeks. The longer nine-week period would give coaches and teachers more time to help students meet growth goals, thereby enhancing their evaluation scores.

While Coach Meg completed a six-week ET with Teacher Michelle, Coach Claire and Teacher Cathy agreed to complete a nine-week ET together. However, Claire said that she would rarely decide to engage in a nine-week coaching cycle as part of her everyday practice, particularly given that Cathy was already an exceptional educator — which is why Claire chose to complete the ET with Cathy in the first place. Claire ultimately felt frustrated the ET prompted her to spend more time in a coaching cycle with Cathy than she needed to, limiting the time she could spend with other teachers who might benefit more from her support.

Coaches did too much for teachers. Because the student data produced from ET-driven coaching cycles would be directly tied to the coaches’ own evaluations, Meg and Claire did much more for their teachers when engaged in the ET than they typically did. For example, Meg modeled whole-group instruction in Michelle’s classroom to complete her ET. This, in itself, was not unusual. But, contrary to what is considered best practice for modeling (Clarke, Triggs, & Nielsen, 2014; West & Staub, 2003), Meg never released the classroom to Michelle to let her teach on her own. Instead, Meg modeled for the entire duration of the ET before giving the post-test, enabling her to teach lessons precisely as she felt they should be taught. Meg herself was conscious of this choice: “I want to actually model for the first couple of weeks to almost even three because this is my Evaluation Tool, so I am responsible for the scores.”

Similarly, when completing her ET with Cathy, Claire acknowledged that she took on more of the work than she typically would have, by, for example, independently creating the small groups and schedule rather than cocreating them with the teacher. Claire said that she needed to take on this work because her evaluation was linked to the coaching cycle: ”I do feel like the Evaluation Tool really drove that for me a lot more with Cathy because I wouldn’t have normally done that, but I felt like I had to because I needed to do my part…and…it was tied to my evaluation.”

Given that Meg and Claire did so much for, rather than with, Michelle and Cathy during the coaching cycles, both teachers expressed concerns at the end of the coaching cycle about their ability to continue to implement what their coaches initiated. For example, Cathy shared that it would be difficult for her to continue with the guided mathematics groups Claire modeled without having Claire in the room to assist.

Perverse incentives

The Evaluation Tool seems appropriate for evaluating teachers, as their primary role is teaching students. Coaches, on the other hand, are supposed to help teachers improve their practice. Imposing the teacher evaluation system on coaches created perverse incentives for the coaches we studied. Specifically, it prompted the coaches to spend more time than was appropriate in their ET-related coaching cycles and to do much of the work for teachers instead of with them. Because the ET focuses on student, instead of teacher, growth, the coaches were motivated to sidestep the teachers and focus on generating strong student data, rather than ensuring that teachers were consistently engaged in substantive learning opportunities. The evaluation system made it possible for the coaches to receive a very positive evaluation without having made much, if any, impact on the teachers.

Evaluating coaches’ multifaceted work

Coaches support teaching and learning in various ways across an entire school or even district. In a given day, a coach may begin by modeling a number-sense activity in a kindergarten classroom, then facilitate a grade-level team meeting for 5th-grade teachers, then co-teach with a 3rd-grade teacher during guided reading, and end the day by leading a schoolwide professional development session on student sense-making in mathematics.

Instead of evaluating coaches on the basis of student performance, the coaching evaluation system should attend to these varied components of coaches’ work. That is, coaches should receive feedback on how well they provide learning opportunities for teachers in one-on-one settings (such as modeling and co-teaching), as well as in small- and large-group professional development settings across various grade levels and content areas.

How can this be done? There are several options:

Administrator observations of coaches. Administrators could observe coaches as they provide professional development for individuals and groups of teachers. But what should administrators look for? In their 2018 book Systems for Instructional Improvement: Creating Coherence from the Classroom to the District Office, Paul Cobb and his colleagues outline five principles that should be followed for professional development to support teacher learning. Below, we outline each principle and then provide questions for administrators to consider as they observe coaches:

Principle 1: Professional development should be coherent and connected. Does the coach connect the focus of group professional development with the focus of one-on-one professional development? For example, if the coach provides group professional development on how to effectively launch a mathematical task, does the coach also seek opportunities to support individual teachers in their classrooms as they launch mathematical tasks with their own students?

As part of the evaluation process, the administrator should meet with the coach to debrief observations and set goals for the coach’s future work. Furthermore, just as teachers are typically observed several times each year, coaches should be observed at least as often, with observations taking place in both one-on-one and group settings.

Teacher surveys about coaches. Administrators could ask teachers to complete brief, confidential surveys about their interactions with coaches. For example, coaches could provide the names of teachers they have recently worked with in individual and group settings, and then those teachers could be surveyed using a combination of Likert-scale items and open-ended responses. Likert-scale items could ask teachers to rate the extent to which they agree with several statements, such as:

Working with my coach has helped me improve my teaching.

The open-ended questions could ask teachers to describe the ways in which the coach worked with them and to identify areas of strength and improvement for their coach. After reading through the survey data, the administrator could then debrief with the coach, summarizing the main survey findings to protect teacher confidentiality.

Student data as evidence of teacher improvement. A central idea behind coaching is that coaches should help teachers improve their practice, which should ultimately affect student achievement. Hence, coach evaluations could be partially based on the improvement of teachers’ instruction, as measured by student growth. Admittedly, doing this would require both access to and careful analysis of meaningful student achievement data.

To provide a concrete example, suppose that a coach provided yearlong, whole-group professional development for K-2 teachers on how to implement number talks, which are activities designed to enhance students’ number sense (e.g., “How many pairs of numbers can you find that add up to 15?”). Furthermore, suppose that this coach also worked individually with teachers to support their integration of number talks into their mathematics instruction. Thus, to evaluate the effectiveness of the coach at enhancing teachers’ number-sense instruction, student achievement data specific to number sense could be examined.

Specifically, the number-sense gains of participating teachers’ students could be compared with: (1) those same students’ gains in other topics during the current year, (2) those students’ gains in number sense in prior years, (3) the number-sense gains of students taught by the participating teachers in previous years and/or (4) the number-sense gains of teachers who did not work with the coaches. Although no single analysis should be considered definitive, a pattern of student improvement could emerge to reveal the influence of coaches on teacher and student learning.

Coach self-study. One last method that could supplement any of the methods described above is a coach self-study of their work with teachers. As part of a self-study, a coach might consider asking teachers to complete a brief questionnaire before and after a coaching cycle in which teachers state what they hoped to learn and evaluate whether and how the coaching cycle helped them reach these learning goals. The coach could also consider asking teachers to keep an informal reflective journal to document their learning during a coaching cycle. Finally, the coach could conduct formal or informal observations of the teacher’s instruction before and after the coaching cycle to identify any changes in instruction.

Measuring what matters

Effectively evaluating the work of instructional coaches is complex. Although the ultimate goal of coaching is to improve student learning, solely relying on student performance data may lead coaches to focus their work on students, rather than teachers. To avoid this, we recommend that district leaders take a more holistic view of coaches’ work by observing coaches in action, asking both teachers and coaches to reflect on their work together, and carefully using student data to examine evidence of teacher and student improvement. The Evaluation Tool we studied rewarded coaches for sidestepping teachers, but the approaches we’ve suggested above are intended to help coaches keep their focus where it should be — on fostering teachers’ instructional improvement.