Abstract
To support continuous professional development model in the teaching and learning of mathematics, many districts and schools have begun hiring elementary mathematics coaches and/or specialists (MCSs). However, limited large-scale empirical research exists that determines how the use of MCSs affect student learning and achievement. Kristin E. Harbour and Evthokia Stephanie Saclarides begin to fill in this gap by using data from the National Assessment of Educational Progress to explore the relationship between the presence and responsibilities of elementary MCSs and 4th-grade student achievement in mathematics. Based on their findings, they share practical implications for districts and administrators to consider.
An analysis of data from the National Assessment of Educational Progress provides insight into the relationships between mathematics coaches and/or specialists and student achievement.
Given the ineffectiveness of traditional, whole- group, one-shot professional development workshops (Ball & Cohen, 1999), many school districts have, in recent years, turned to instructional coaches to support the teaching and learning of mathematics (McGatha, Davis, & Stokes, 2015). However, coaching is expensive (Knight, 2012), especially when experienced teachers are removed from the classroom to serve as coaches (Mangin, 2009), and additional research is needed to better understand whether the benefits of coaching outweigh the costs.
In theory, coaching should be an effective professional development model, since coaches can meet with teachers multiple times throughout the school year, closely observe their teaching practice, and help them address specific problems and challenges they face (Desimone & Pak, 2017; Gibbons & Cobb, 2017). As yet, though, little large-scale empirical research has been conducted into how well this model actually improves student achievement. Most studies of this topic have been small-scale and qualitative in nature, and few researchers have conducted large-scale, quantitative studies of the relationships between coaching and student achievement (Hjalmarson et al., in press; Saclarides et al., in press).
One group of studies has shown some promising results, suggesting that math coaching was related to student gains (Balfanz, MacIver, & Byrnes, 2006; Campbell, 1996; Foster & Noyce, 2004). In these studies, however, the coaching was part of a larger school reform initiative, making it difficult to tease out the extent to which the coaching itself — and not some other part of the reform —made a difference.
To get a clearer picture of how effective mathematics coaching really is — as well as the effects of mathematics specialists — we and other researchers have developed a small body of research that attempts to zero in on the effects of coaching itself, apart from any other variables that might influence student achievement (Campbell & Malkus, 2011; Ellington, Whitenack, & Edwards, 2017; Harbour, Adelson, & Karp, 2016; Harbour et al., 2018). As part of this work, we conducted an empirical study that drew upon restricted-use data from the National Assessment of Educational Progress (NAEP), administered by the National Center for Education Statistics (NCES), to explore relationships between the presence and responsibilities of mathematics coaches and specialists and student achievement (Harbour et al., 2016, 2018). To our knowledge, this is the only research to date that has measured the influence of mathematics coaches and specialists on student achievement using both large-scale and nationally representative data.
Delving into the data
While the words coach and specialist can be defined in a number of ways (National Mathematics Advisory Panel, 2008), a common definition distinguishes a coach as someone who directly supports teachers by providing them with professional learning opportunities, whereas a specialist is someone who works directly with students (McGatha, Davis, & Stokes, 2015). However, NAEP combines the terms coach and specialist in data collection, using the abbreviation MCS to refer to both mathematics coaches and specialists. Thus, our findings and discussions below pertain to both coaches and specialists, as it is not possible to tease out the unique influence of each role.
We analyzed a data sample that included roughly 191,000 students, from 7,500 elementary schools, who participated in the 2011 4th-grade NAEP in mathematics (including data from the accompanying student, teacher, and school surveys; NCES, 2011a, 2011b). Because NAEP is a large-scale, nationally representative dataset, our findings can be generalized to a larger population than is possible for smaller-scale studies (Schneider et al., 2007). Also, NAEP is the only large-scale assessment dataset that makes it possible to analyze variables pertaining to MCS’s.
Not only are NAEP’s data-collection techniques well-respected and widely regarded to be valid, its dataset is particularly rich and varied, including data on overall mathematics achievement as well as achievement in specific mathematical content domains, such as geometry, data analysis, and measurement. It also includes a wealth of other data for each participating student, teacher, and school (for example, on student demographics, teachers’ years of experience, school size, and more), allowing us to analyze various relationships among these variables.
Thus, in addition to analyzing how the presence of MCS’s in schools, and their specific job responsibilities, related to students’ mathematics achievement, we were also able to see how these related to variables such as students’ opinions of mathematics, the mathematics content that teachers emphasized in their classes, the percentage of students receiving free or reduced-price lunch, whether a school was public or private, and many more (see Harbour et al., 2018 for a complete list of variables and controls).
To gauge whether the mere presence of MCS’s was related to student achievement, we looked to see whether an elementary school’s policy of employing an MCS (full time, part time, or not at all) connected to students’ overall mathematics achievement score and their scores in five specific content domains (number properties and operations; measurement; geometry; data analysis, statistics, and probability, and algebra). Of the elementary schools in our sample, 62% reported that they did not employ an MCS and 28% reported having an MCS available (full time at 50% of the schools, and part time at 50%).
Next, because the work of MCS’s can vary widely from school to school, we looked for connections between mathematics achievement (both overall and in specific areas) and principals’ reports as to how much time those coaches and specialists spent on six specific responsibilities: (1) providing technical assistance/support to individual teachers about mathematics content or the teaching of mathematics, (2) conducting professional development for groups of teachers about mathematics content or the teaching of mathematics, (3) providing mathematics instruction to students on various topics, (4) providing mathematics instruction to students at various grade levels, (5) providing mathematics remediation/intervention to some student groups, and (6) providing mathematics enrichment to some student groups (NCES, 2011b).
Using a large-scale nationally representative dataset such as NAEP has many benefits for this sort of research, but it also has some limitations that must be considered when interpreting the results. For instance, NAEP represents student achievement at only one particular point in time, so it doesn’t allow for analyses across time. Also, while NAEP collects data on a rich range of variables, it leaves some things out — for example, it doesn’t use a precise definition of full-time and part-time employment, making it impossible to know exactly how much time the MCS’s spent in their schools. Also, while it might have been more accurate for MCS’s themselves to report how much time they spent engaged in specific job responsibilities, principals were asked to give this estimate. And most important, while these data allow us to see how specific variables are correlated with each other, they don’t allow us to say for sure that one factor caused the other.
What we learned
When we analyzed the data, two important findings emerged that have implications for schools and districts. First, we found a statistically significant relationship between elementary schools that had full-time MCS’s and 4th-grade students’ mathematics achievement, both overall and across the five specific mathematics content domains. In other words, we can tentatively say that elementary schools with a full-time MCS are likely to have higher 4th-grade mathematics achievement compared to schools without them, after controlling for many variables, including school and student demographic data and teaching practices. (For details on these variables and how they affect our findings, see Harbour et al., 2018). This is consistent with prior research (e.g., Campbell & Malkus, 2011; Ellington, Whitenack, & Edwards, 2017; Foster & Noyce, 2004). In contrast, our findings indicated that schools having a part-time MCS did not have significantly different 4th-grade mathematics achievement scores (overall and across content domains) than schools without one.
Second, we noted significant relationships between the responsibilities of full-time MCS’s (as reported by principals) and achievement outcomes. Interestingly, results revealed statistically significant positive relationships between the time MCS’s spent on the two responsibilities related to supporting teachers (i.e., supporting both individuals and groups of teachers) and student achievement, but there was either no relationship or a negative relationship noted for their work with students. (See Table 1 for details on specific relationships with overall mathematics achievement and across the five content domains.)
Relationships between coaches’ and specialists’ responsibilities and 4th-grade students’ mathematics achievement
Again, because the data do not indicate a causal relationship, the negative results when MCS’s work more with students must be interpreted with caution. These results do not necessarily mean that having coaches and specialists work directly with students results in lower mathematics achievement scores; it is likely that full-time MCS’s in schools with lower achievement spend more time providing instruction, remediation, and/or interventions to students than they do in other schools.
What this means for schools
Given that a significant and positive relationship was only found between full-time coaches and specialists and 4th-grade students’ mathematics achievement, the most pressing implication for district administrators and principals to consider is to prioritize hiring full-time, not part-time, MCS’s. Given that decades of research stress that effective professional development is ongoing and time-intensive (Darling-Hammond et al., 2009; Desimone, 2009; Hawley & Valli, 1999), our findings make sense. And yet, given the high cost of coaching, it is understandable that some school districts will decide that they can only afford to have part-time MCS’s in individual schools, whether that involves sharing MCS’s among several different schools or having an MCS spend half of their time coaching and the other half of their time as an interventionist or classroom teacher. However, if, as our results indicate, schools can expect to see little to no return on the coaching investment if the MCS is only working part time, district administrators and principals would do well to be aware of potential limitations with part-time coaching when making spending decisions.
District administrators and principals should also think carefully about how they define the responsibilities of their MCSs, particularly the amount of time they spend coaching teachers and/or working directly with students. As they write job descriptions, they should strongly consider emphasizing having MCSs work with teachers in either one-on-one or group settings. Although NAEP does not explicitly define the meaning of one-on-one and group coaching, the literature provides some examples of what it could look like. One-on-one coaching activities may include modeling, co-teaching, and engaging in the coaching cycle (goal setting, working, and reflecting); group coaching may include engaging in lesson study, analyzing classroom videos, examining student work, and studying disciplinary content with other teachers (Gibbons & Cobb, 2017). By focusing coaches’ work on learning opportunities for teachers, administrators can tentatively expect to see a return on their coaching investment.
This research is a significant step forward in understanding the relationship between MCSs and student achievement. Such research is important, given how quickly schools have invested in hiring MCSs in recent years, as well as the high cost of this professional development model. The evidence of the positive influence of MCSs, particularly when they work with teachers, will be important to leaders as they set professional development priorities. But it is just the beginning — additional research is needed to establish a causal relationship between the work of MCSs and improved student performance and to show how their work relates to student achievement in other subject areas and grade levels.