The Sensitivity of Teacher Value-Added Scores to the Use of Fall or Spring Test Scores

Abstract

Papay (2011) noticed that teacher value-added measures (VAMs) from a statistical model using the most common pre/post testing timeframe–current-year spring relative to previous spring (SS)–are essentially unrelated to those same teachers’ VAMs when instead using next-fall relative to current-fall (FF). This is concerning since this choice–made solely as an artifact of the timing of statewide testing–produces an entirely different ranking of teachers’ effectiveness. Since subsequent studies (grades K/1) have not replicated these findings, we revisit and extend Papay’s analyses in another Grade 3–8 setting. We find similarly low correlations (.13–.15) that persist across value-added specifications. We delineate and apply a literature-based framework for considering the role of summer learning loss in producing these low correlations.

Keywords

correlational analysis econometric analysis educational policy regression analyses school/teacher effectiveness validity/reliability

During the past 10 years, many U.S. states have revised their teacher evaluation systems to incorporate value-added measures (VAM) scores. Federal Race to the Top grants initially spurred these changes, and the more recent Every Student Succeeds Act—although not requiring states to link teacher evaluations specifically to test scores—codified the expectation that districts distinguish teachers based on effectiveness (Berg-Jacobson, 2016). As of 2019, 26 states require teacher evaluations to include student growth data based on standardized tests (Ross & Walsh, 2019), and Steinberg and Donaldson (2016) report that 61% of the nation’s largest school districts include VAM scores in teacher evaluations. At their core, value-added (VA) models attempt to isolate the causal effect of individual teachers on student achievement by predicting students’ end-of-year test scores while taking into account baseline scores from the previous spring.¹ Yet estimating teachers’ causal effects in this way is far from straightforward. As VAM usage expanded, a large body of research² emerged to consider whether the statistical properties of VAM scores (e.g., precision, unbiasedness) could support high-stakes teacher talent management decisions such as teacher hiring, retention, compensation, or tenure.

In one such study, Papay (2011) briefly shows that teacher VAM scores from models using the most common pre/post testing timeframe—that is, based on changes in test scores from last spring to current spring—are essentially orthogonal (correlation of -.10) to those same teachers’ VAM scores from a model using test score changes from current fall to next fall.³ This finding should be of great concern: There is no principled reason to use spring-to-spring over fall-to-fall pre/post timeframes to construct VAMs, and according to Papay’s results, this choice—made solely as an artifact of the timing of statewide testing systems—would lead to an entirely different ranking of teachers’ effectiveness.

This troubling finding warrants further study for several reasons. First, because VAM sensitivity to the choice of pre/post timeframe was not his focus, Papay (2011) did not explore what potential mechanisms might be at work. It is also possible the finding was idiosyncratic to the assessment at hand, the specific district or decade, or the specification of the model used to produce the VAM scores. In fact, a handful of subsequent studies have used Early Childhood Longitudinal Study–Kindergarten Class (ECLS-K) data to explore this question in kindergarten and first grade but have not found similarly low correlations (Gershenson & Hayes, 2018; Hayes & Gershenson, 2018; Palardy & Peng, 2015). Given that Papay’s findings would have serious implications for policies that use VAM scores, but results from later studies substantively differ, we take up this line of inquiry in a new school district with data from a more recent period in the tested subjects and grades in which VAM scores are typically implemented.

Research Questions

Because the current district administered the same assessment for all Grade 3 through 8 students in both the fall and spring for at least 3 consecutive years, we can do what is typically not possible with state standardized test score data: For each teacher, we estimate three VAM scores using spring-to-spring (S→S), fall-to-fall (F→F), and fall-to-spring (F→S) test timeframes.⁴ We then examine whether VAM rankings are sensitive to choice of pre/post timeframe and whether findings are specific to only certain VA model specifications. We also consider the role summer learning loss (SLL) may play in producing this result. We pose the following four questions:

(1) Using a VA model as similar as possible to that used by Papay (2011), do we observe the same no-correlation finding now in a different district, during a different time period, and using a different test?

(2) Does the correlation remain low when using updated versions of the VA model?

(3) Do patterns of variation in SLL in the current district help us understand this finding?

(4) Is there heterogeneity in the results across teachers, grades, and schools?

As a preview, our basic results are similar to Papay’s: We find very low correlations between SS- and FF-based VAM scores (.13 for English language arts [ELA], .17 for math). This finding—now apparent in two educational contexts—raises concerns about the expansion of using VAM scores for teacher evaluation during the past decade.

Conceptual Framework

An ideal way to isolate teachers’ causal effects on student test scores would require administration of a reliable assessment at the very start of fall and end of spring in each school year. This FS timing would most directly capture a student’s achievement gains that occurred while assigned to a given teacher. An artifact of administering standardized tests only once annually is that a summer period—either the summer before (SS) or the summer after (FF) a given school year—will be attributed to the teacher, even though the teacher and student do not interact during that time (see Figure 1). Although neither summer misattribution is desirable, incorporating the presummer (SS) time seems more problematic because the teacher has not yet met the student. In sum, we can think of FS-based VAMs as ideal but impractical, SS as pervasive but problematic (includes the prior summer), and FF as an uncommon, but more desirable, alternative to SS.

Figure 1.

Simplified illustration of the three pre/post test score timeframes FF, FS, & SS used to estimate the effect of a Grade 4 teacher on one student’s test scores.

Figure 1 illustrates this pre/post timeframe issue. First, note that estimating all three VAMs for the same teacher requires data across a minimum of 3 years (here Grades 3, 4, and 5). Although nine basic patterns of SLL preceding and following a given school year are possible,⁵ we highlight two scenarios that would lead SS-, FS-, and FF-based VAMs for the same teacher all to be different from one another. Consider the simplified Scenario 1 (left) in which we estimate the impact of a Grade 4 teacher on one student’s test scores (zig-zagging line, dashed and shaded during the summers). The horizontal lines at the bottom depict SS (black dashed), FS (gray dashed), and FF (gray solid) timeframes and the summers they encompass. The corresponding vertical lines on the right of the Scenario 1 graph illustrate that the student’s growth appears smaller for this teacher when using the SS timeframe and larger when using FF. We also can see that summer gains/losses will affect the ordering of these VAM scores: Spring scores are the same on both the left and right panels of Figure 1; however, Scenario 1 (left) shows that the child lost ground in the summer before Grade 4 but nearly sustained her school year growth rate in the summer after Grade 4. Scenario 2 (right) shows the opposite pattern (gain in presummer, loss in postsummer), and now SS attributes the most growth to the Grade 4 teacher and FF the least. We therefore need to think carefully not only about the unintended inclusion of summers but about how the patterns of summer learning will affect teacher VAMs.

Potential Role of SLL

Three conditions are necessary for SLL⁶ to produce low correlations among SS-, FS-, and FF-based VAMs: (1) There must in fact be SLL ( $A c h_{y}^{f a l l} \neq A c h_{y - 1}^{s p r}$ for all students). (2) All students must not exhibit essentially the same SLL (var(SLL) ≠ 0. SLL research has focused less on estimating this variance, but researchers have identified correlates of SLL, which implies some variation. We estimate var(SLL) herein and find incredible SLL variability across students. (3) The variation in SLL must not be random. SLL must be correlated with either the sociosystemic sorting of students to classes/schools, the (true) effectiveness of the teachers to whom students are assigned (call this $δ_{p}$ for teacher p), or the sequencing of the effectiveness of a student’s teachers across grades. Next, we consider how this nonrandom allocation of SLL to teachers could arise.

Summer before current-year teacher

At first, it seems intuitive that SLL in the summer before being assigned to a given teacher (the presummer) will be unrelated to that teacher’s effectiveness—indeed, the teacher and student have not yet met. Based on this logic, we would hypothesize that SS-based VAMs that incorporate the prior summer would be less correlated with FS-based VAMs than would FF-based VAMs. However, if SLL in the presummer is correlated with factors associated with the sorting of students to teachers or schools, then this correlation may arise indirectly. For example, student race/ethnicity and socioeconomic status (SES) predict access to high-quality teachers and schools (Boyd et al., 2005; Goldhaber et al., 2019; Isenberg et al., 2016; Lankford et al., 2002). Do these factors also predict SLL? Below, we bring to bear SLL literature and empirical evidence on this question.

Summer after current-year teacher

Because students no longer interact with their teachers or schools in the postsummer, this SLL could be unrelated to teacher effectiveness. However, in contrast to presummer SLL—for which an as-yet-unmet teacher’s effectiveness can have no direct impact—a teacher’s influence could persist into the postsummer. If so, then $c o r r (δ_{p}, S L L^{p o s t})$ would not be zero. The postsummer SLL could even be thought of as part of the teacher’s effect.

Teacher effectiveness sequencing

A final complication arises if a student’s teachers are sequenced across grades in a manner that induces a correlation between effectiveness and summer learning. For example, if school leaders recognize that some students were assigned to a weaker teacher in Grade 3 (so presumably they exhibit lower $A c h_{y - 1}^{s p r}$ and no positive impact on SLL), they might compensate by assigning those students to a stronger teacher in Grade 4, which would lead to $c o r r (S L L^{p r e}, δ_{p}^{4 t h}) \neq 0$ . Alternatively, if weaker teachers tend to be concentrated in a given school, students moving through grades in that school might experience a sequence of relatively ineffective teachers who do not counteract learning losses in the postsummers. In sum, these sequences could induce nonrandom variation in SLL but may do so in unpredictable ways.

SLL Literature

A comprehensive review of the SLL literature is beyond the scope of the current article (see Gershenson, 2013, for an overview). However, several studies are particularly relevant to the three necessary conditions outlined above. For example, Downey et al. (2004) document significant correlations between learning rates in the school year (in kindergarten or Grade 1) and the intervening summer. Assuming school year learning rates are in part produced by teachers, then $c o r r (S L L, δ_{p})$ may not be zero. Gershenson’s (2013) work with time-diary survey data shows that the variability of children’s summer environments and activities is greater than the variability during school, suggesting there will be nontrivial variation in SLL. Finally, several studies using ECLS-K:99 and/or ECLS-K:11 cohorts find that SLL is correlated with SES and—in some cases—race/ethnicity,⁷ suggesting demographic sorting may be relevant (Burkam et al., 2004; Downey et al., 2004; Quinn & Le, 2018; Quinn et al., 2016; von Hippel et al., 2018). It is interesting to note, however, Burkam et al. (2004) also find that a full vector of student demographics, home characteristics, prior achievement, and a list of summer activities together account for no more than 13% of the variation in SLL after kindergarten. In other words, SLL appears to vary greatly for reasons we have not yet identified.

Existing Research on Pre/Post Timeframes for VAM

A handful of studies have reported on the correspondence among teachers’ FF-, FS-, and SS-based VAMs (see Table 1 for an overview). We are particularly motivated by the remarkable finding in Papay (2011) of virtually no correlation between teachers’ FF-based and SS-based scores: $C o r r ({\hat{δ}}_{p}^{F F}, {\hat{δ}}_{p}^{S S}) = - . 10$ . He also finds that FF- and SS-based teacher VAMs are not strongly correlated with more ideal (but impractical) FS-based VAMs: $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{F F})$ = .19 and $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{S S})$ = .66.

Table 1

Correlations Between FF-, FS-, and SS-Based Value-Added Measures (VAMs) Across Known Studies

									ELA VAM			Math VAM
Author(s)	Year	Dataset	Geography	Data Years	Grade Levels	No. of Students	Test Used	High Stakes?	Corr(FF, SS)	Corr(FS, FF)	Corr(FS, SS)	Corr(FF, SS)	Corr(FS, FF)	Corr(FS, SS)
School Level
McEachin & Atteberry	2017	Anonymous school district	Southern state	2008-2011	2-8	45,000	MAP	No			0.85			0.85
Palardy & Peng	2015	ECLS-K:99	Nationally representative	1999-2004	K-1	2,251	ECLSK⁺	No			0.82			0.89
Teacher* Level
Palardy & Peng	2015	ECLS-K:1999	Nationally representative	1999-2004	K-1	2,251	ECLSK⁺	No			0.84			0.91
Gershenson & Hayes	2018	ECLS-K:99	Nationally representative	1999-2004	K-1	2,000	ECLSK⁺	No		0.80	0.92		0.45	0.80
Hayes & Gershenson	2018	ECLS-K:11	Nationally representative	2011-2013	1	1,800	ECLSK⁺	No	0.64	0.71	0.89	0.46	0.60	0.81
Papay	2011	Large anonymous urban district	Northeast U.S.	2002-2008	3-8	55,000	SRI	No	-0.10	0.19	0.66
Atteberry & Mangan (current)	2020	Large anonymous district	Southeast (both & and urban)	2012-2015	3-8	50,000	MAP	No	0.13(0.14)	0.26(0.20)	0.59(0.68)	0.17(0.23)	0.36(0.38)	0.60(0.67)

Note. Data years are reported based on the spring of the school year. All estimates reported from other studies are from the available model most similar to our preferred model, including vectors of student characteristics and, when possible, classroom characteristics and school effects.

ECLS-K:99 and ECLS-K:11 administered direct cognitive assessments designed to measure children’s knowledge and skills in reading and mathematics, partly modeled after more commonly known assessments, including the Peabody Picture Vocabulary Test (PPVT), the Woodcock Johnson, and/or the Test of Early Math Ability (TEMA).

Teacher-level VAMs are sometimes time invariant but sometimes are allowed to vary by year. ECLSK:99 data observe each teacher in only 1 year, so this 1-year teacher effect could be framed as a classroom effect (the language used by Gershenson & Hayes, 2018) or simply as a teacher effect (used by Palardy & Peng, 2015, when using the same data). In the current article, we estimate both teacher time-invariant effects across multiple years (akin to Papay, 2011) and teacher-by-year estimates (shown in parentheses below the teacher effect correlations). However, we do not use the term “classroom” for these latter effects because the two terms are not synonymous in higher grades.

Although our focus is on teacher VAMs, two recent studies examine this question for school VAMs. The issue of summer misattribution is substantively different for school impacts: Because teachers typically work with each student for only 1 year, it seems much clearer that the summer before they meet should not be considered part of the teacher’s effect. Schools, on the other hand, serve many of the same students year after year as they return after each summer to move across grades. A student’s intervening summers, then, are typically situated between two exposures to the same school (except in the case of cross-school transitions). Unlike teacher VAMs, school VAMs based on annual testing do not fully misattribute summers to a school that a student has not yet attended. We therefore would anticipate higher correlations among FF-, FS-, and SS-based school VAMs.⁸ Indeed, McEachin and Atteberry (2017) use data from an anonymous southern state and find for school VAMs a $C o r r ({\hat{δ}}_{s}^{F S}, {\hat{δ}}_{s}^{S S})$ = $. 85$ .85 (math and ELA). Palardy and Peng (2015) estimate school-level VAMs using ECLS-K:99 data and find similar correlations of .82 and .89 for ELA and math, respectively.

Shifting back to teacher⁹ VAMs, three studies employ ECLS-K data to estimate $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{S S})$ . Using ECLS-K:99, Palardy and Peng (2015) report a teacher-level correlation of .84 in ELA (.91 in math). In a second article, Gershenson and Hayes (2018) instead use ECLS-K:99 and find similar estimates (.92 in ELA, .80 in math). Hayes and Gershenson (2018) leverage the more recent ECLS-K:11 data, and again results are similar: The $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{S S})$ is .89 in ELA and .81 in math. These authors also estimate the $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{F F})$ and find correlations of .80 (ELA) and .45 (math) when using ECLS-K:99 (Gershenson & Hayes, 2018) and .71 (ELA) and .60 (math) when using ECLS-K:11 (Hayes & Gershenson, 2018).

Although ECLS-K:99 cannot be used¹⁰ to replicate the troubling $C o r r ({\hat{δ}}_{p}^{F F}, {\hat{δ}}_{p}^{S S})$ that Papay (2011) found to be -.10, ECLS-K:11 does permit such analysis. When Hayes and Gershenson (2018) do so, however, they find much stronger correlations of .64 in ELA and .46 in math. Far from suggesting that teachers’ FF- and SS-based VAMs are orthogonal to one another, these results suggest a moderate, positive correspondence between the two.

The correlations reported from studies using ECLS-K:99 or ECLS-K:11 are notably higher than those from Papay (2011) or—as we will see—the current study. The different findings across data sets could be due to any of several factors. First, the ECLS-K studies follow a single cohort of students, and thus each teacher is observed only once. Second, none of the three data sets contains the same vertically scaled assessments, all of which may differ from one another in terms of scaling, content covered, or intended usage (see von Hippel & Hamrock, 2019, for a particularly relevant discussion of how measurement artifacts can affect inferences about SLL). Third, the VAM scores estimated in the ECLS-K studies are based off an average of 3 students per teacher. In the current study, each teacher is linked to an average of 23 students per year in elementary schools (100 students per year in middle schools). Although smaller samples might be expected to attenuate correlations, one possibility is that empirical Bayes shrinkage on teacher VAM scores with low reliability could overshrink estimates toward a grand mean and make the correlations appear higher. Another speculation for those ECLS-K analyses that do not shrink estimates is that, whereas the correlations across teachers’ VAMs are precisely estimated in ECLS-K (because there are around 750 classrooms), the underlying VAM scores’ being correlated could be somewhat idiosyncratic to the approximately 3 students (on average) with both spring and fall scores in each classroom.¹¹

Finally, the biggest difference when using ECLS-K studies is that these correlations can be estimated only for kindergarten and Grade 1 teachers. In practice, districts use teacher VAMs almost exclusively in Grades 4 and above, and it is unclear whether results from the earliest grades also would apply to these later grades. The current data set thus provides a unique opportunity to replicate and extend Papay’s (2011) findings in policy-relevant grades.

Data and Analytic Sample

The current study uses 2010–2011 through 2014–2015 administrative student- and school-level data from one anonymous district located in the southeastern United States. The district covers one of the state’s largest cities and its surrounding area, and it consists of 57 schools serving about 50,000 students annually. It spans a large geographic area—about 250 square miles—which includes a central urban area surrounded by both suburban and rural communities.

The data set includes typical demographic data for its students, including race/ethnicity, sex, free/reduced-price lunch program (FRPL) eligibility, limited English proficiency (LEP) status, and special education (individualized education program [IEP]) status. Roster data link students to their classrooms, teachers, and schools in each grade and school year. Unfortunately, teacher covariates like years of experience are not available. Finally, the data set contains students’ state standardized test scores and—the key to the current study—fall and spring Measures of Academic Progress (MAP) assessment scores from the Northwest Evaluation Association (NWEA), which was administered to students in Grades 3 through 8¹² for up to 5 consecutive years.

About the MAP Assessment

Scaling, intended use, and content

The MAP assessment has different scaling properties and purposes than the state’s standardized achievement test,¹³ which would typically be used to construct SS-based teacher VAM scores. Unlike the state standardized assessment, the MAP test is scored to follow a vertical and interval scale, to produce what NWEA calls RIT (Rasch Unit) scale score points. One could hypothesize that MAP’s vertical scaling explains the patterns we observe across FF-, FS-, and SS-based VAM scores. However, we present main analyses on both the original RIT scale scores and a standardized version and find nearly identical results.

The intended use of the statewide end-of-year exam differs from that of MAP, which is designed to be used as a supplementary tool to aid schools in improving their instruction, not as the high-stakes test of record. Moreover, the MAP test is given to students across the United States and in this sense is not curriculum based, although NWEA does attempt to tailor the assessment to a state’s content standards. NWEA has conducted a MAP linking study within the past 3 years with the current state’s assessment and found high proficiency classification consistency rates at or greater than 81% across subjects and grades.¹⁴ Although scaling, intended use, and content may differ, students’ standardized state test scores and spring MAP scores in this district are strongly correlated with one another (.82 in ELA, .78 in math). Despite their differences, these two assessments appear to provide similar information about students’ math and ELA achievement.

Differential effort

One must acknowledge the possibility that MAP testing could be approached differently in the fall than in the spring in ways that could introduce some bias (e.g., certain teachers give it greater focus than others) or imprecision. For instance, MAP tests could be taken less seriously in the fall than in the spring. If, at the extreme, fall scores are essentially noise, then low correlations could be an artifact of that noise.¹⁵ Although it is not possible with this data set, we explore this issue indirectly using a separate, national data set¹⁶ of MAP scores that contains time spent on each test and the number of items attempted. There, we see that students do spend about 6 fewer seconds per item on fall tests relative to spring tests. This is consistent with a possible fall-spring effort differential; however, the magnitude of this difference suggests that fall tests are not generally disregarded. In addition, fall MAP scores in the current district are strongly correlated with both spring MAP scores ( $. 82$ .82 ELA, .87 math) and spring scores on the statewide exam (.76 in ELA, .78 in math), again suggesting that fall scores do contain signal.

Unknown test dates

Finally, MAP tests are not taken on the very first and last day of school. Therefore, some of the time between the prior spring test and current fall test—what we would label summer—in fact takes place during the end and start of the school year. We use publicly available district calendar data to identify likely MAP testing window dates and then project RIT scores to the first and last day of school.¹⁷ Consistent with projected scores in the national MAP data set, projected and observed scores in this district are correlated at greater than .96. We present analyses using both the observed and projected RIT scores. The pattern of results remains quite consistent, especially among models producing empirical Bayes shrunken teacher estimates.

Analytic Sample

We limit the current analytic sample to students observed with MAP test scores in Grades 3 through 8 and their linked math and ELA teachers. To provide context for the kinds of districts to which our findings might generalize, Table 2 contains basic student- and school-level demographics for the analytic sample in one example school year (2011–2012).

Table 2

Sample Descriptives at the Student and School Level, Academic Year 2011–2012

	M	SD
Student-level characteristics (N = 22,438)
% male	.52	—
% Asian	.08	—
% Black	.38	—
% Hispanic	.07	—
% White	.47	—
% LEP	.06	—
% FRPL	.36	—
% special education	.13	—
School-level characteristics (N = 57)
Average % male	.50	.07
Average % Asian	.08	.08
Average % Black	.38	.32
Average % Hispanic	.07	.06
Average % White	.46	.28
Average % LEP	.07	.08
Average % FRPL	.38	.25
Average % special education	.13	.05
Average number of students	409	307

Note. Descriptive statistics are for the 2011–2012 school year. LEP = limited English proficiency; FRPL = free/reduced-price lunch program. Dashes indicate the SD of a binary, student-level variable (not applicable).

The analytic sample includes about 22,000 students annually, of whom about half are identified as non-White and more than one third are eligible for the federal FRPL. Black students constitute the largest non-White racial/ethnic group (38%); another 8% are Asian and 7% are Hispanic. See Appendix Table A1 for annual analytic sample sizes for students by grade, teachers by subject, and schools. Overall, the 5-year analytic sample includes 44,793 unique students, 1,459 unique teachers, and 57 unique schools.

Analytic Approach

In Equation 1, we first specify a VA model as similar as possible to the one used by Papay (2011) to estimate time-invariant teacher VAM scores, separately for math and ELA:

Y_{t i c p g s} = α_{g} f (Y_{t - 1, i}) + (X_{(t) i}) λ + ({\bar{X}}_{t c}) ζ + δ_{p} + φ_{s} + θ_{g} + μ_{t} + ε_{t i c p g s} .

(1)

Like Papay, we allow teacher effects, $δ_{p}$ , to vary randomly in a model with end-of-period test scores on the left-hand side, and on the right-hand side a polynomial function of a same-subject prior test score, student and classroom covariates, as well as school ( $φ_{s}$ ), grade ( $θ_{g}$ ), and year (µ_t) fixed effects.¹⁸ First using a traditional SS test timing in ELA, $Y_{t i c p g s}$ represents standardized¹⁹ ELA MAP scores at time t = end-of-spring for student i in classroom c of teacher p in grade g in school s. This outcome is modeled as a grade-specific, fifth-order polynomial of the student’s baseline ELA MAP score ( $Y_{t - 1, i,}$ ) at time t–1, which—in the SS-based VAM—is the prior spring. The student characteristics vector $X_{(t) i}$ includes race/ethnicity dummies, gender, FRPL status, LEP status, and IEP status.²⁰ These also are aggregated to the classroom level ( ${\bar{X}}_{t c})$ . The teacher random effects themselves become the VAM scores.

We estimate the above model three times, each time changing only how we define t–1 = baseline and t = end-of-period (from SS, to FS, to FF). We generate for each teacher three VAM-based rankings: ${\hat{δ}}_{p}^{' F S}$ (ideal but impractical), ${\hat{δ}}_{p}^{' S S}$ (typically available), and ${\hat{δ}}_{p}^{' F F}$ (theoretically preferable to SS). We then estimate teacher-level Spearman rank correlations across the three test administration timings: $C o r r ({\hat{δ}}_{p}^{' F F}, {\hat{δ}}_{p}^{' S S})$ , $C o r r ({\hat{δ}}_{p}^{' F S}, {\hat{δ}}_{p}^{' F F})$ , and $C o r r ({\hat{δ}}_{p}^{' F S}, {\hat{δ}}_{p}^{' S S})$ . Strong correlations would imply that the choice of pre/post timeframe has little influence on which teachers are labeled high- versus low-performing and thus should not be a source of much concern.

We also examine whether the strength of these correlations differs across other specifications of the VA model (e.g., with school covariates in place of school fixed effects). We rerun the analysis using the RIT scores (not standardized) to see whether vertical scaling affects our findings. We also respecify the VA model to produce year-specific teacher VAM scores (as opposed to teacher time-invariant scores). This allows us to explore several sources of potential heterogeneity: We examine how results change when teachers’ scores are allowed to vary from one year to another. We also explore whether those correlations vary based on teachers’ current-year grade assignment or current-year school’s student composition.

Results

Research Question 1: Replication

Like Papay (2011), we find that switching to a different test administration timing dramatically alters the ranking of teachers based on the VA model in Equation 1. In Table 3, we find that teachers’ ELA VAM rankings that are produced utilizing SS versus FF test administration timings are only correlated at .17 (compare this to Papay’s estimate of -.10 for ELA). We extend the analysis into math and find a $C o r r ({\hat{δ}}_{p}^{' F F}, {\hat{δ}}_{p}^{' S S})$ of .15. The low correlations (approaching zero) indicate that test timing—a mere artifact of how statewide standardized assessments are typically administered—is a highly influential factor in how teachers are ranked according to VAMs.

Table 3

Replication of Papay (2011) Spearman Rank Correlations Across Value-Added Measures (VAMs) Using FF, FS, and SS Pre/Post Timeframes

	English Language Arts Correlations			Math Correlations
	(δ’_p^FF, δ’_p^SS)	(δ’_p^FS, δ’_p^FF)	(δ’_p^FS, δ’_p^SS)	(δ’_p^FF, δ’_p^SS)	(δ’_p^FS, δ’_p^FF)	(δ’_p^FS, δ’_p^SS)
Papay (2011)	−.10(624)	.19(713)	.66(644)	n/a	n/a	n/a
Current Article	.17(579)	.32(777)	.61(745)	.15(580)	.38(783)	.58(739)

Note. Deltas $({\hat{δ}}_{p}^{'})$ represent the VAM score ranking of teachers (p). Teacher sample sizes are shown in parentheses. These VAM scores are derived from the Equation 1 value-added model, which includes student and classroom controls as well as grade, year, and school fixed effects. VAM scores are shrunk. The model also includes an up to fifth-order polynomial function of students’ prior score in the same subject (allowed to vary by grade). Test scores are standardized at the subject-grade-semester-year level. For the reader’s reference, we also include the correlations reported in Table 9 of Papay (2011), alongside our replication of those correlations. FF = current fall → next fall; SS = prior spring → current spring; FS = current fall → current spring; n/a = not available.

We also consider the use of the more ideal FS-based timing for VAM scores, in which end-of-year spring test scores are modeled as a function of start-of-year fall test scores, thus isolating both the summer before and the summer after the school year from the teacher effects. We are interested in whether this ideal scenario ( ${\hat{δ}}_{p}^{' F S}$ ) aligns more with the typically available option ( ${\hat{δ}}_{p}^{' S S}$ ), or the theoretically preferred ${\hat{δ}}_{p}^{' F F}$ alternative.

In Table 3, we find the $C o r r ({\hat{δ}}_{p}^{' F S}, {\hat{δ}}_{p}^{' F F})$ is also quite low— .32 in ELA and .38 in math. Although these correlations are slightly higher than Papay’s (2011) corresponding estimate of .19 in ELA, they are still quite low. We also find that the typically available option ( ${\hat{δ}}_{p}^{' S S})$ exhibits a stronger correlation of .61 with the ideal estimates ${\hat{δ}}_{p}^{' F S}$ than does the theoretically preferred ${\hat{δ}}_{p}^{' F F}$ option (.32). Recall that the ${\hat{δ}}_{p}^{' F F}$ incorporates the summer following a teacher’s time with the student, whereas ${\hat{δ}}_{p}^{' S S}$ incorporates the summer before they meet. Because a student’s learning in the summer before cannot be directly affected by the teacher’s true effectiveness, we anticipated a stronger $C o r r ({\hat{δ}}_{p}^{' F S}, {\hat{δ}}_{p}^{' F F})$ than $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{S S})$ . It is noteworthy that our findings suggest the opposite to be the case, although Papay observes the same pattern in his study. In general, the fact that we replicate Papay’s pattern of results quite closely supports the hypothesis that his findings were not idiosyncratic to that setting.

Research Question 2: Updated Model Specifications

We also consider the possibility that the low correspondence of teacher VAM scores across test timings is an artifact of the specific VA model chosen by Papay (2011) (represented by our Equation 1). For instance, his model includes school fixed effects, and whereas there often are reasons to include these for research purposes, districts are unlikely to use them because they undermine their ability to compare teachers across schools. In addition, the literature on VA modeling specifications has progressed since 2011. We adopt a preferred model based on both these practical considerations and the literature (Chetty et al., 2014a; Guarino et al., 2015; Kane et al., 2013; Koedel et al., 2015; Sass et al., 2012). We therefore make the following changes to Equation 1 for our preferred model: We replace school fixed effects with a vector of time-varying and time-invariant school covariates,²¹ use a third-order polynomial in place of the fifth-order polynomial function,²² include prior scores in both subjects rather than only the same subject, and analyze correlations among the estimates (rather than rankings).

Results from this preferred model are presented in Table 4 (row 1 [preferred]). The general pattern of correlations holds, and if anything, correlations from the preferred model are slightly lower than those from the replication model. The $C o r r ({\hat{δ}}_{p}^{F F}, {\hat{δ}}_{p}^{S S})$ is still very low at .13 in ELA (.17 in math); the $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{F F})$ is .26 in ELA (.36 in math), and the $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{S S})$ is .59 in ELA (.60 in math).

Table 4

Teacher Value-Added Measure (VAM) Correlations for FF, FS, and SS Pre/Post Timeframes, Across 12 Specifications of the Value-Added Model

Model	VA Model Specification Permutations			English Language Arts Correlations			Math Correlations
Model	Score Scaling	Empirical Bayes Shrinkage	School Controls	(δ’_p^FF, δ’_p^SS)	(δ’_p^FS, δ’_p^FF)	(δ’_p^FS, δ’_p^SS)	(δ’_p^FF, δ’_p^SS)	(δ’_p^FS, δ’_p^FF)	(δ’_p^FS, δ’_p^SS)
1 (preferred)	Standardized	Yes	S & SY Covs	.13	.26	.59	.17	.36	.60
2	Standardized	Yes	FEs & SY Covs	.11	.24	.59	.12	.34	.59
3	Standardized		S & SY Covs	−.04	.19	.54	.17	.39	.54
4	Standardized		FEs & SY Covs	.04	.25	.54	.09	.23	.53
5	RIT scale score	Yes	S & SY Covs	.12	.26	.60	.15	.36	.61
6	RIT scale score	Yes	FEs & SY Covs	.10	.25	.58	.09	.33	.59
7	RIT scale score		S & SY Covs	−.07	.21	.56	.19	.46	.59
8	RIT scale score		FEs & SY Covs	.03	.22	.54	.07	.26	.57
9	Projected RIT	Yes	S & SY Covs	.14	.25	.57	.19	.33	.58
10	Projected RIT	Yes	FEs & SY Covs	.10	.24	.58	.14	.31	.58
11	Projected RIT		S & SY Covs	−.04	.21	.38	.10	.39	.50
12	Projected RIT		FEs & SY Covs	.11	.28	.35	−.07	.17	.25

Note. Correlations across VAM estimates generated from the preferred model are shown in the first row (1 [preferred] row). All models in this table include the vector of time-varying and time-invariant student demographics and the vector of time-varying classroom controls. All models also include an up to third-order polynomial function of prior test scores in both math and English language arts, and these functions are allowed to vary by grade. Models differ from one another on three dimensions: test scaling, whether teacher effect estimates have been shrunk, and school controls. FF = current fall → next fall; SS = prior spring → current spring; FS = current fall → current spring; S & SY Covs = the model includes the vector of both time-invariant and time-varying school covariates; FEs & SY Covs = the model substitutes school fixed effects in place of the time-invariant school covariates; RIT = Rasch Unit (NWEA scale units).

We again draw the reader’s attention to the lowest correlations in the first row of Table 4 between FF- and SS-based VAMs. To illustrate the troubling nature of these results, we represent this $C o r r ({\hat{δ}}_{p}^{F F}, {\hat{δ}}_{p}^{S S})$ finding as a quartile-quartile transition matrix in Table 5. We categorize teachers into one of four quartiles (Quartile 1 is highest; Quartile 4 is lowest) according to, first, their SS-based VAM scores (rows) and, then, their FF-based VAM scores (columns). The cells of Table 5 contain the number of teachers in each combination of quartiles as well as the row percentages. For example, 55 teachers (38%) of the 144 top-quartile ELA teachers according to ${\hat{δ}}_{p}^{S S}$ are also in the top quartile according to ${\hat{δ}}_{p}^{F F}$ .

Table 5

Quartile-Quartile Teacher Transition Matrix for SS Versus FF Test Timings, by Subject

	English Language Arts
	FF—Q1 (Top)	FF—Q2	FF—Q3	FF—Q4 (Bottom)	Row Total
SS—Q1 (top)	55(0.38)	31(0.22)	36(0.25)	22(0.15)	144(1.00)
SS—Q2	30(0.21)	50(0.35)	34(0.24)	30(0.21)	144(1.00)
SS—Q3	33(0.23)	39(0.27)	34(0.24)	38(0.26)	144(1.00)
SS—Q4 (bottom)	26(0.18)	24(0.17)	40(0.28)	54(0.38)	144(1.00)
Column total	144(0.25)	144(0.25)	144(0.25)	144(0.25)	576(1.00)
	Math
	FF—Q1 (Top)	FF—Q2	FF—Q3	FF—Q4 (Bottom)	Row Total
SS—Q1 (top)	41(0.28)	41(0.28)	27(0.19)	36(0.25)	145(1.00)
SS—Q2	48(0.33)	31(0.21)	39(0.27)	27(0.19)	145(1.00)
SS—Q3	32(0.22)	46(0.32)	41(0.28)	26(0.18)	145(1.00)
SS—Q4 (bottom)	24(0.17)	27(0.19)	38(0.26)	55(0.38)	144(1.00)
Column total	145(0.25)	145(0.25)	145(0.25)	144(0.25)	579(1.00)

Note. Quartiles are created for value-added measure scores from the preferred specification of the value-added model shown in the first row of Table 3 (1 [preferred] row). Rows show quartiles based on SS timing, and columns show quartiles based on FF timing. Cells contain counts of teachers in a given quartile transition (e.g., for English language arts, 22 teachers were in Q1 according to SS timing but in Q4 according to FF timing). Row percentages are included in parentheses in cells. We also present row and column totals at the margins. SS = prior spring → current spring; FF = current fall → next fall; Q = quartile.

Only about 33% of all ELA teachers appear along the table diagonal—that is, in the same quartile using their SS-based and FF-based VAM score rankings. In addition, about 30% of teachers appear in an FF-based category two or more quartiles away from their SS-based category. In fact, 15% of top-quartile ELA teachers according to SS timing would be considered bottom-quartile teachers according to the FF timing (25% for math). In essence, a nontrivial portion of the seemingly strongest teachers according to the traditional SS timing would be categorized among the weakest teachers using the FF timing. Results are similar for math. This underscores that FF-based teacher VAM score rankings bear little resemblance to those based on the traditional SS-based timing. There is no principled reason to use SS-based VAMs over FF-based VAMs; however, this choice would lead to very different conclusions about teachers’ effectiveness.

Robustness to other model specifications

In Table 4, we reproduce these three correlations— $C o r r ({\hat{δ}}_{p}^{F F}, {\hat{δ}}_{p}^{S S})$ , $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{F F})$ , and $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{S S})$ —across 11 additional permutations of the VA model for both ELA (left) and math (right). The permutations include one of three scalings of the test scores, the use of empirical Bayes teacher estimates versus teacher fixed effects without shrinkage, and choices of school-level controls.²³ Although our preferred specification does not include school fixed effects, these models can be quite useful in research contexts, and for those models in Table 4 we include both school fixed effects and a vector of time-varying school covariates.

Overall, we find that the general takeaway holds across model specifications. In every model, the $C o r r ({\hat{δ}}_{p}^{F F}, {\hat{δ}}_{p}^{S S})$ is the weakest of the three, the $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{F F})$ is in the middle, and the $C o r r ({\hat{δ}}_{p}^{F S}, {\hat{δ}}_{p}^{S S})$ is the strongest of the three. The average $C o r r ({\hat{δ}}_{p}^{F F}, {\hat{δ}}_{p}^{S S})$ value across all models (Columns 1 and 4) is .10 (and never higher than .19 $. 19$ ), suggesting that this low correspondence persists across modeling choices. Regardless of model specification, SS test timing will provide a very different ranking of teachers from those based on FF test timing.

Although the primary takeaway from Table 4 is one of consistency, we note some patterns in the variability across specifications. Looking across all 12 models, results from our preferred model (Model 1) are among the least problematic (i.e., somewhat higher correspondence across FF, FS, and SS timings). We also see that estimates produced without shrinkage tend to exhibit lower correlations. This makes sense if the shrunk estimates have eliminated some noise that would otherwise attenuate correlations. For instance, whereas Models 3, 7, and 11 all produce negative ELA correlations between FF- and SS-based scores—similar to Papay’s (2011) -.10 estimate—our correlations are never negative in models with shrinkage.

Between two otherwise similar models, the one that includes school fixed effects in lieu of time-invariant school covariates often leads to slightly lower correlations. Although this is not always the case, it is always true among models with shrinkage. Finally, correlations are often lower in models using the projected RIT scores, which attempt to separate the time between the prior spring test and current fall test into the actual summer and school year periods. This projection approach, however, does not overturn the basic findings.

Research Question 3: SLL Patterns in District

To understand why FF-, FS-, and SS-based timings produce such different teacher VA scores, we provide a brief analysis of SLL in the current district, and we find results that are consistent with the theoretical discussion earlier in the narrative. Because MAP is vertically scaled and growth norms differ across grades, we conduct this SLL analysis separately by grade. For each student, we estimate a summer learning gain/loss by comparing their fall score in a given grade to their previous spring score. For each grade, we use an unconditional, three-level random effects model to partition the variation in SLL across students, teachers, and schools. This model allows us to estimate mean SLL and student-level variation in SLL. It also allows us to explore the extent to which SLL is correlated with student demographics, sorted across teachers in a given school, or clustered across schools. The greater the (nonrandom) variation in SLL, the lower we think the correlations across test administration timings should be.

The SLL findings are summarized in Table 6, using both observed RIT scores (upper panel) and RIT scores we projected to the first and last day of school (lower panel). Not surprisingly, mean SLL looks more negative when using projected scores because the projection attends to the fact that students may continue to learn after the spring test in April and before the fall test in September. Although estimates of mean SLL are somewhat different based on this choice, the subsequent examination of SLL variation is quite consistent across the two. We therefore focus on results using the original RIT scale. As a point of reference for what follows, NWEA reports that the standard deviation of math RIT scores in Grade 5 is about 16 points, and the Grade 5 NWEA growth norm is 9.9 RIT points (Thum & Hauser, 2015).

Table 6

Statistical Exploration of Variation in Summer Learning Loss (SLL) Using a Three-Level Hierarchical Model, by Score Scaling, Subject, and Grade

Scaling	Subject	Grade	Mean SLL	Student-Level SD(SLL)	% of Variation in SLL at			95% Plausible Value Range	% of Student-Level Var(SLL) Explained by Demographics
Scaling	Subject	Grade	Mean SLL	Student-Level SD(SLL)	Student Level	Teacher Level	School Level	95% Plausible Value Range	% of Student-Level Var(SLL) Explained by Demographics
RIT scale	ELA	4	0.52	8.6	95.4	1.8	2.8	(–16.3, 17.4)	0.3
		5	0.03	8.1	94.4	2.5	3.1	(–15.9, 16.0)	0.3
		6	−0.98	8.8	91.1	1.8	7.1	(–18.2, 16.3)	2.0
		7	0.61	8.6	95.3	1.8	2.9	(–16.3, 17.6)	2.4
		8	0.70	8.5	96.3	1.3	2.3	(–16.0, 17.4)	1.5
	Math	4	0.45	6.9	93.0	2.8	4.2	(–13.1, 14.0)	0.9
		5	0.10	7.0	93.3	3.3	3.4	(–13.6, 13.8)	1.0
		6	−2.83	8.0	90.7	1.7	7.5	(–18.4, 12.8)	2.1
		7	−0.48	7.6	95.2	2.0	2.8	(–15.4, 14.5)	2.6
		8	0.22	7.7	94.0	2.6	3.5	(–14.9, 15.3)	1.0
Projected	ELA	4	−3.22	10.0	93.9	3.0	3.1	(–22.8, 16.4)	0.1
		5	−3.19	10.1	93.7	3.5	2.9	(–23.1, 16.7)	0.4
		6	−2.98	10.4	94.0	1.1	4.9	(–23.3, 17.4)	0.0
		7	−0.88	10.2	95.9	1.8	2.3	(–20.8, 19.0)	2.2
		8	−0.48	9.9	97.1	1.0	1.9	(–19.9, 19.0)	1.8
	Math	4	−4.72	8.0	92.8	3.2	4.0	(–20.4, 10.9)	0.3
		5	−5.41	8.7	92.6	4.9	2.6	(–22.5, 11.6)	0.3
		6	−7.02	9.8	94.2	2.2	3.6	(–26.1, 12.1)	0.6
		7	−2.54	9.0	95.6	2.0	2.4	(–20.1, 15.0)	2.1
		8	−1.59	9.0	95.2	2.1	2.6	(–19.3, 16.1)	1.4

Note. Results are from an unconditional, three-level random effects model used to partition the variation in SLL in a given subject/grade across students, teachers, and schools—see Columns 6, 7, and 8. The standard deviation of RIT scale score points ranges between 13 and 17, depending on the subject and grade. This model allows us to estimate mean SLL (Column 4) and student-level variation in SLL (Column 5). We translate student-level variation into a 95% plausible value range for expected SLL across students in a given school (Column 9). In Column 10, we add to the model the vector of student-level demographics (identical to the vector in the value-added models) and report the reduction in Level 1 residual variance, relative to the unconditional model. Analyses are conducted for SLL estimates based on observed RIT scores from the data set, as well as RIT scores we projected to the first and last day of school. RIT = Rasch Unit (NWEA scale units); ELA = English language arts.

Summer learning patterns in the current district mirror findings from prior SLL research and meet the necessary conditions for test administration timing to affect teacher VA estimates. For instance, although students exhibit, on average, very little change in math test scores in the summer after fifth grade (0.10 RIT points)—consistent with prior work—this average masks incredible variability across students in SLL (Atteberry & McEachin, 2020). The 95% plausible value range for math SLL after Grade 5 spans from -13.6 to +13.8 RIT score points. This means that some students return to school in the fall of sixth grade with test scores that are much higher than their observed scores from the prior spring, whereas others return having lost much of what they gained over the course of the previous grade. There is a potential, then, for a teacher to encounter a substantial range in SLL among his or her students at the start of each school year.

We also find that the large majority of variation in SLL occurs at the student level (about 90% to 96%). In some ways, it makes sense that out-of-school, summer learning is less clustered by school than learning that occurs during the school year. There is some sorting of SLL across teachers in the same schools and across schools, but the main takeaway here is the incredible variation among students.

In an effort to explore the systematic sorting of SLL, we add a full vector of student-level demographics to the student level of the model. Again, results mirror findings documented in prior work: Although we observe a remarkable spread of summer gains/losses across students, the sum of student demographics does little to account for this variation. The set of race/ethnicity, sex, IEP, and LEP status indicators—although sometimes correlated with SLL—together account for less than 3% of the student-level variance in SLL (final column of Table 6).

Taken together, this brief SLL analysis suggests that there is real variation in SLL that could cause FF-, FS-, and SS-based VAM scores to diverge. Given the low correspondence among VAMs using those three timings, this SLL variation must be systematic in some way. However, the sorting among teachers in the same school and across schools appears limited. This may explain why VA models with school fixed effects yield only slightly different correlations. Moreover, although demographics play a small part in accounting for this notable variation across students in SLL, the research community has yet to identify the features of students, families, or their summer experiences that tell us why so much variation exists.

Research Question 4: Heterogeneity in Correlations Across Years, Grades, and Schools

Finally, we explore the potential for heterogeneity in the FF-, FS-, and FF-based VAM correlations by year, by grade, and by school setting. For this research question, we modify the VA model to estimate teacher-by-year VAM scores to more directly consider how absorbing the preceding summer instead of the following summer affects how a teacher is evaluated in a given year. Consistent with the theoretical framework, a teacher’s year-specific VAM scores may be more affected by pre/post timeframes when there is some effectiveness-based sequencing across students’ teachers from one grade to the next. The greater the differential in latent effectiveness between a given teacher and the teacher(s) who taught their students in the preceding year, the more one might expect FF-based, SS-based, and FS-based VAM scores to diverge.

We therefore rerun the preferred specification of the VA model but now substitute teacher-by-year effects ( $δ_{p t}$ ) for the teacher effects ( $δ_{p}$ ), reproduce the correlations, and disaggregate correlations by the teacher’s modal grade²⁴ taught in each year. Results are presented in Table 7. We continue to find that the correspondence between FF-based and SS-based VAM score rankings tends to be low: $C o r r ({\hat{δ}}_{p t}^{F F}, {\hat{δ}}_{p t}^{S S})$ for teachers across all grades is .14 for ELA and .23 for math. In fact, all six correlations across timings and subjects with the teacher-year effects are quite similar to those correlations using the teacher time-invariant effects (summarized in Table 1).

Table 7

Teacher-by-Year Value-Added Measure (VAM) Score Correlations: Overall and Separately by the Teacher’s Grade Assignment and the Socioeconomic Status of the Teacher’s School

	English Language Arts Correlations			Math Correlations
	(δ’_pt^FF, δ’_pt^SS)	(δ’_pt^FS, δ’_pt^FF)	(δ’_pt^FS, δ’_pt^SS)	(δ’_pt^FF, δ’_pt^SS)	(δ’_pt^FS, δ’_pt^FF)	(δ’_pt^FS, δ’_pt^SS)
Overall correlations using teacher-year VAMs
All teacher-years	.14	.20	.68	.23	.38	.67
Heterogeneity across teacher’s year-specific grade assignment
Grade 4	.29	.40	.68	.28	.51	.68
Grade 5	.09	.23	.71	.11	.32	.71
Grade 6	.06	−.03	.63	.52	.39	.84
Grade 7	.18	.35	.73	.42	.65	.67
Heterogeneity across socioeconomic status of teacher’s year-specific school assignment
Low % FRPL schools	.27	.25	.79	.26	.42	.68
High % FRPL schools	.09	.19	.64	.22	.33	.69

Note. Deltas $(δ_{p t}^{'})$ represent the time-varying VAM scores of teacher (p) at time (t). These VAM scores are produced by the preferred specification of the value-added model shown in the first row of Table 4 (1 [preferred] row), with the exception that teacher-by-year random effects replace the teacher effects. Sample sizes for grade-specific correlations presented in the table range from about 100 to 300. In low % FRPL schools, 25% or fewer of students are eligible for federal FRPL; in high % FRPL schools, more than 50% of students are FRPL eligible. These cutoffs correspond closely to the bottom and top thirds of the % FRPL distribution. FRPL = free/reduced-price lunch program.

Turning to the grade-specific results in Table 7, there do appear to be substantive differences across grades: In ELA, whereas $C o r r ({\hat{δ}}_{p t}^{F F}, {\hat{δ}}_{p t}^{S S})$ is .06 for Grade 6 teachers, that same correlation is $. 29$ .29 for Grade 4 teachers. In math, the $C o r r ({\hat{δ}}_{p t}^{F F}, {\hat{δ}}_{p t}^{S S})$ varies across grades, reaching a surprising .52 in Grade 6. In general, there does not appear to be a clear trend in the strength of these associations from lower to higher grades. The disaggregation by grade shows some surprising results that would be worthy of more in-depth investigation than possible with the current data. For instance, in ELA, these correlations are consistently lowest in Grade 6—a common middle school transition year. However, the same is not true for math. Overall, the results in Table 7 generally corroborate the main finding that measures of teaching effectiveness depend on whether the underlying VA model utilizes FF, SS, or FS test administration timing.

In Table 7 we also explore whether the low correspondence among the FF-, FS-, and SS-based VAM scores differs across teachers in schools serving high- and low-SES students. We see some evidence that correlations are slightly less problematic (i.e., higher) for teachers assigned to high-SES schools (less than 25% of students are FRPL eligible) than for teachers in low-SES schools (more than 50% of students are FRPL eligible). We see this pattern for five of the six correlations reported, with an exception in the math $C o r r ({\hat{δ}}_{p t}^{F S}, {\hat{δ}}_{p t}^{S S})$ , which is about equal in high- and low-SES schools. The differences by school SES are intriguing because they suggest that this measurement issue stemming from the pre/post timeframes is somewhat more severe for teachers who choose to work in schools with larger historically underserved student populations. In fact, we see that the $C o r r ({\hat{δ}}_{p t}^{F S}, {\hat{δ}}_{p t}^{S S})$ in ELA reaches nearly .80 among teachers in high-SES schools. This particular correlation is quite policy relevant because it concerns the SS timing that is widely used in practice.

Discussion

We find a very weak relationship (.13 ELA, .17 math) between teachers’ VAM scores produced by an otherwise identical VA model that estimates student growth either from prior spring to current spring (SS—the pragmatically feasible option) or current fall to next fall (FF—a theoretically better alternative to SS). The majority of teachers—about 70%—would be categorized into a different quartile of performance by FF- and SS-based VAMs. Moreover, neither of these options exhibits a strong correlation with VAM score rankings based on a current-fall to current-spring (FS—ideal but impractical) test timing.

Results from the current study generally align with findings from a similar analysis embedded in Papay (2011) and hold up against the additional robustness checks we are able to bring to bear. The results are less well aligned with studies that have used ECLS-K data that find much stronger teacher-level correlations (Gershenson & Hayes, 2018; Hayes & Gershenson, 2018; Palardy & Peng, 2015). As discussed in the literature review, however, there are a number of possible explanations for the discrepancy, most notably the difference in grade level.

Findings have implications for how VAM scores are used to characterize teacher performance in both policy and research contexts. These measures of teacher effectiveness often play a role in high-stakes decisions about teacher retention, placement, and compensation. Low correlations between models that—at least intuitively—should produce somewhat similar rankings call into question the fairness of using SS-based VAM scores in ways that can dramatically affect the lives of teachers. For example, districts that use VAMs as a factor in teacher compensation may not be correctly identifying the teachers who have the strongest positive impact on student achievement outcomes.

A clear candidate for explaining the discrepancies in teacher VAM scores based on FF- versus SS-based test timings lies in the potential misattribution of summer periods. We document incredible variability across students in summer learning gains/losses, which we think is a key to understanding why these different VAM scores diverge. Several studies show that SLL appears to disproportionately occur for low-income and historically marginalized students (Allington et al., 2010; Burkam et al., 2004; Cooper et al., 1996; Downey et al., 2004; Entwisle & Alexander, 1992; Quinn & Le, 2018; Quinn et al., 2016; von Hippel et al., 2018). In our heterogeneity analyses based on school SES, we find some evidence that the VAM scores for teachers working in low-SES schools may be slightly more susceptible to these timing-based VAM discrepancies.

Given that a student’s learning in the summer before a given school year cannot be affected by the current-year teacher’s true effectiveness, one might be somewhat surprised to find that the typically available SS-based rankings exhibit a stronger correlation of .61 with the more “ideal” FS-based rankings than do the theoretically preferred FF-based option (correlation of .32). Such a pattern could arise if SLL effects carry forward into the subsequent school year in ways that influence a teacher’s effectiveness or if teacher assignment patterns are correlated with prior-SLL. In practice, the fact that SS-based teacher VAMs are more correlated with FS-based VAMs can be of some comfort because SS-based VAMs are typically the only available option to districts.

Our work also highlights how much we do not yet understand about SLL. Although we clearly can see that students’ summers are very different from one another, some of our most powerful predictors of school year learning inequalities (e.g., student demographics) are only a small part of the story for summer learning rates. As one might expect, SLL is also much less clustered by teachers and schools than is school year learning because students are not in school when these summer gains/losses take place. Nonrandom variation in SLL must exist if SLL is the cause of the discrepancies among FF-, FS-, and SS-based teacher VAM scores. However, the field has not yet been able to explain why SLL varies. Researchers could explore this issue using more detailed data on teacher and school contexts as well as students’ summer experiences.

The current findings contribute to the existing literature base that considers the many factors that drive the same teacher to have somewhat different VA scores (e.g., year-to-year instability, in math versus ELA, different tests, controlling for various sets of observed covariates, including student or school fixed effects, using a random effects framework, a student growth percentile approach). In broad terms, the literature documents correlations across these choices in the general range of .30 to .70 (see Koedel et al., 2015, for a recent, excellent synthesis of points of consensus and contention in the VA literature). In contrast, many of the correlations documented in the current study are in the .10 to .20 range. If these findings truly reflect an unprincipled artifact of test administration timing, then it would suggest that inferences about teacher effectiveness made using SS-based VAMs should be questioned.

Future research could push this topic forward by addressing several important limitations of the current study. A key concern is differential effort on fall MAP administrations, which could lead to greater imprecision in fall scores. If fall scores are simply unreliable or inaccurate measures of students’ math or ELA skills at that point in time, then incorporating them into a VA model would not be advisable. We are unable to fully address this limitation in the current context, although we do observe that fall MAP scores are strongly correlated with students’ high-stakes test scores on the statewide assessment, which occurs just a few months earlier in the preceding spring (e.g., .79 for math). It would be useful to conduct these analyses in settings either where there are compelling reasons to believe the fall and spring test administrations are of equal importance to teachers or where information about the reliability of individual test scores—for example, test score standard errors, seconds spent per item, total minutes spent by each student—is available to the researcher.

The finding of a very low correlation between FF- and SS-based teacher VAMs has now been documented in both a large, urban school district in the northeast (Papay, 2011) and a southeastern school district of similar size that includes urban, suburban, and rural areas. How do these results fit with other research on FF, FS, and SS pre/post timeframes? We can see in Table 1 that much higher correlations have been found when using ECLS-K data or for school-level VAMs. Though we posit some reasonable explanations for these varied results across studies, more work is needed to truly understand the source of these discrepancies. If future studies corroborate the low correlations documented herein, then researchers and policymakers should carefully consider the implications of relying on SS-based VAM scores to characterize teachers’ relative effectiveness. This is particularly troubling given that the theoretically more appropriate approach of estimating teacher VAM scores based on FS or FF timings is simply not an option for most districts.

Footnotes

Appendix Table A1

Student, Teacher, and School Sample Sizes by Year, Grade, and Subject

	2010–2011	2011–2012	2012–2013	2013–2014	2014–2015	Total Unique
Students
Grade 3	3,861	3,850	3,929	3,901	4,154	19,649
Grade 4	3,811	3,880	3,842	3,948	3,928	19,382
Grade 5	3,921	3,844	3,846	3,822	3,981	19,405
Grade 6	3,826	3,969	3,929	3,955	3,959	19,511
Grade 7	3,901	3,870	3,964	3,963	4,041	19,561
Grade 8	3,713	3,922	3,888	4,055	3,991	19,347
All grades	23,033	23,335	23,398	23,644	24,054	44,793
Teachers
English language arts only	141	134	135	135	138	268
Math only	133	139	138	138	131	266
Both subjects	559	557	556	537	531	967
All teachers	833	830	829	810	800	1,459
Schools
Total unique	56	57	56	56	56	57

ORCID iD

Allison Atteberry

Notes

Authors

ALLISON ATTEBERRY, PhD, is an assistant professor of research and evaluation methodology at the University of Colorado–Boulder School of Education, 249 UCB, Boulder, CO 80309; allison.atteberry@colorado.edu. Her research focuses on policies and interventions that are intended to help provide effective teachers to the students who need them most.

DANIEL MANGAN is a PhD student at the University of Colorado–Boulder School of Education, 249 UCB, Boulder, CO 80309; daniel.mangan@colorado.edu. His research interests are focused on teacher labor markets and school reforms that promote supportive and persistent student-teacher relationships.

References

Allington

R. L.

McGill-Franzen

Camilli

Williams

Graff

Zeig

Zmach

Nowak

(2010). Addressing summer reading setback among economically disadvantaged elementary students. Reading Psychology, 31(5), 411–427. https://doi.org/10.1080/02702711.2010.505165

American Educational Research Association. (2015). American Educational Research Association statement on use of value-added models (VAM) for the evaluation of educators and educator preparation programs. Educational Researcher, 44, 448–452.

Atteberry

Allison

McEachin

Andrew

. (in press). School’s out: The role of summers in understanding achievement disparities. American Educational Research Journal.

Atteberry

Loeb

Wyckoff

. (2017). Teacher churning: Reassignment rates and implications for student achievement. Educational Evaluation and Policy Analysis, 39(1), 3–30.

Berg-Jacobson

(2016). Teacher effectiveness in the Every Student Succeeds Act. https://gtlcenter.org/sites/default/files/TeacherEffectiveness_ESSA.pdf

Boyd

D. J.

Lankford

Loeb

Wyckoff

(2005). Explaining the short careers of high-achieving teachers in schools with low-performing students. American Economic Review, 95(2), 166–171.

Briggs

D. C.

Dadey

(2017). Principal holistic judgments and high-stakes evaluations of teachers. Educational Assessment, Evaluation and Accountability, 29(2), 155–178.

Burkam

D. T.

Ready

D. D.

Lee

V. E.

LoGerfo

L. F.

(2004). Social-class differences in summer learning between kindergarten and first grade: Model specification and estimation. Sociology of Education, 77(1), 1–31.

Chetty

Friedman

J. N.

Rockoff

J. E.

(2011). The long-term impacts of teachers: Teacher value-added and student outcomes in adulthood. National Bureau of Economic Research.

10.

Chetty

Friedman

J. N.

Rockoff

J. E.

(2014a). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593–2632. https://doi.org/10.1257/aer.104.9.2593

11.

Chetty

Friedman

J. N.

Rockoff

J. E.

(2014b). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood. American Economic Review, 104(9), 2633–2679. https://doi.org/10.1257/aer.104.9.2633

12.

Cooper

Nye

Charlton

Lindsay

Greathouse

(1996). The effects of summer vacation on achievement test scores: A narrative and meta-analytic review. Review of Educational Research, 66, 227–268.

13.

Downey

D. B.

von Hippel

P. T.

Broh

B. A.

(2004). Are schools the great equalizer? Cognitive inequality during the summer months and the school year. American Sociological Review, 69(5), 613–635.

14.

Entwisle

D. R.

Alexander

K. L.

(1992). Summer setback: Race, poverty, school composition, and mathematics achievement in the first two years of school. American Sociological Review, 57, 72–84.

15.

Gershenson

(2013). Do summer time-use gaps vary by socioeconomic status? American Educational Research Journal, 50, 1219–1248. https://doi.org/10.3102/0002831213502516

16.

Gershenson

Hayes

M. S.

(2018). The implications of summer learning loss for value-added estimates of teacher effectiveness. Educational Policy, 32, 5–85.

17.

Goldhaber

Quince

Theobald

(2019). Teacher quality gaps in US public schools: Trends, sources, and implications. Phi Delta Kappan, 100(8), 14–19.

18.

Guarino

C. M.

Reckase

M. D.

Wooldridge

J. M.

(2015). Can value-added measures of teacher performance be trusted? Education Finance and Policy, 10(1), 117–156.

19.

Hayes

M. S.

Gershenson

(2018). Spring or fall annual tests? Implications for value-added models. Teachers College Record. https://hayes.camden.rutgers.edu/files/accepted_manuscript-1.pdf

20.

Isenberg

Max

Gleason

Johnson

Deutsch

Hansen

(2016). Do low-income students have equal access to effective teachers. Evidence from 26 districts. U.S. Department of Education.

21.

Jacob

B. A.

Lefgren

(2008). Can principals identify effective teachers? Evidence on subjective performance evaluation in education. Journal of Labor Economics, 26(1), 101–136.

22.

Kane

T. J.

McCaffrey

D. F.

Miller

Staiger

D. O.

(2013). Have we identified effective teachers? Validating measures of effective teaching using random assignment. [Research Paper. MET Project]. Bill & Melinda Gates Foundation.

23.

Koedel

Betts

J. R.

(2011). Does student sorting invalidate value-added models of teacher effectiveness? An extended analysis of the Rothstein critique. Education Finance and Policy, 6(1), 18–42.

24.

Koedel

Mihaly

Rockoff

J. E.

(2015). Value-added modeling: A review. Economics of Education Review, 47, 180–195.

25.

Lankford

Loeb

Wyckoff

(2002). Teacher sorting and the plight of urban schools: A descriptive analysis. Educational Evaluation and Policy Analysis, 24(1), 37–62.

26.

McCaffrey

D. F.

Sass

T. R.

Lockwood

Mihaly

(2009). The intertemporal variability of teacher effect estimates. Education Finance and Policy, 4(4), 572–606.

27.

McEachin

Atteberry

(2017). The impact of summer learning loss on measures of school performance. Education Finance and Policy, 12(4), 468–491.

28.

Palardy

G. J.

Peng

(2015). The effect of summer on value-added assessments of teacher and school performance. Education Policy Analysis Archives, 23(92), 1–27.

29.

Papay

J. P.

(2011). Different tests, different answers: The stability of teacher value-added estimates across outcome measures. American Educational Research Journal, 48, 163–193.

30.

Quinn

D. M.

Cooc

McIntyre

Gomez

C. J.

(2016). Seasonal dynamics of academic achievement inequality by socioeconomic status and race/ethnicity: Updating and extending past research with new national data. Educational Researcher, 45, 443–453.

31.

Quinn

D. M.

Q. T.

(2018). Are we trending to more or less between-group achievement inequality over the school year and summer? Comparing across ECLS-K cohorts. AERA Open, 4(4). https://doi.org/10.1177/2332858418819995

32.

Ross

Walsh

(2019). State of the states 2019: Teacher and principal evaluation policy. https://www.nctq.org/pages/State-of-the-States-2019:-Teacher-and-Principal-Evaluation-Policy#TStudentGrowth

33.

Rothstein

(2009). Student sorting and bias in value-added estimation: Selection on observables and unobservables. Education Finance and Policy, 4(4), 537–571.

34.

Sass

T. R.

Hannaway

Figlio

D. N.

Feng

(2012). Value added of teachers in high-poverty schools and lower poverty schools. Journal of Urban Economics, 72(2/3), 104–122.

35.

Steinberg

M. P.

Donaldson

M. L.

(2016). The new educational accountability: Understanding the landscape of teacher evaluation in the post-NCLB era. Education Finance and Policy, 11(3), 340–359.

36.

Thum

Y. M.

Hauser

C. H.

(2015). NWEA 2015 MAP norms for student and school achievement status and growth. https://www.nwea.org/content/uploads/2018/01/2015-MAP-Norms-for-Student-and-School-Achievement-Status-and-Growth.pdf

37.

von Hippel

P. T.

Hamrock

. (2019). Do test score gaps grow before, during, or between the school years? Measurement artifacts and what we can know in spite of them. Sociological Science, 6, 43–80.

38.

von Hippel

P. T.

Workman

Downey

D. B

. (2018). Inequality in reading and math skills forms mainly before kindergarten: A replication, and partial correction, of “Are schools the great equalizer?” Sociology of Education, 91(4), 323–357.