The Reliability of Linear Gain Scores as Measures of Student Growth at the Classroom Level in the Presence of Measurement Bias and Student Tracking

Abstract

The evaluation of teachers’ performance in the classroom is an important application of educational testing and psychometrics. School districts are using gain scores, such as the average difference between pre- and posttest scores, to characterize teacher performance in the classroom. Consequently, additional research is needed to understand factors that affect the reliability of gain scores to ensure reliable teaching evaluations. This article examines a class of linear gain scores (LGS) with a modified common factor model to understand the effect of latent variable characteristics on the reliability of observed gain scores. The analytic results derive an upper bound for the reliability of a class of LGS and compare simple difference scores and residualized scores. The results suggest that simple difference scores tend to be more reliable than residualized gain scores whenever strong invariance is satisfied where latent intercepts and loadings are equal over classrooms. However, residualized gain scores are more similar to the optimal reliability in instances when classrooms differ in latent measurement intercepts and loadings. In addition, in contrast with previous conjectures, the results imply that student tracking artificially inflates LGS reliability. The results in this article serve as a guide for researchers who develop and refine methods for measuring student learning gains and evaluating teachers.

Keywords

gain scores growth models latent variable models value-added modeling

Standardized tests are increasingly used to calculate gain scores in an effort to measure student learning and evaluate teacher performance. Prior research has examined the reliability of measures of growth and change (e.g., Bond, 1979; Cronbach & Furby, 1970; Tucker, Damarin, & Messick, 1966; Zimmerman & Williams, 1998). Some special cases of a broader class of linear gain scores (LGS) have become popular measures of student learning as researchers and decision makers attempt to quantitatively evaluate the impact of teachers on students (Battauz, Bellio, & Gori, 2011; Lockwood, McCaffrey, Mariano, & Setodji, 2007; McCaffrey, Lockwood, Koretz, Louis, & Hamilton, 2004; Woodhouse, Yang, Goldstein, & Rasbash, 1996). The reliability of gain scores is particularly important given that the popular press has echoed concerns about using statistical models as a means to rate teachers (Garland, 2012; Watanbe, 2011; Winerip, 2011). For instance, many teachers do not understand the statistical models that are used in their evaluation and are concerned that highly reliable, but imperfect measures of student content knowledge, such as standardized test scores, may distort ratings that are used for high-stakes decisions such as teacher retention or compensation. Researchers have voiced numerous methodological issues with models that employ LGS, which range from attenuation attributed to test score measurement error (Battauz et al., 2011; Culpepper & Aguinis, 2011; Woodhouse et al., 1996), vertical scaling issues (Briggs & Weeks, 2009), threats to internal validity such as the persistence of school-level effects (Briggs & Weeks, 2011) and inability to estimate the causal effect of teachers from observational designs (Raudenbush, 2004; Reardon & Raudenbush, 2009; Rubin, Stuart, & Zanutto, 2004), or the consistency of teacher rankings when using different outcome variables (Papay, 2011).

Applied researchers are relying on gain scores to measure student learning and growth and this study provides a framework for understanding factors that affect the reliability of classroom-level gain scores. In a manner similar to previous investigations on the connection between prediction and measurement bias (Culpepper, 2012; Millsap, 1997, 1998, 2007), this article uses a modified common factor model (CFM) to highlight important theoretical results pertaining to a class of LGS. This study examines the reliability of LGS in the scenario where differences between classrooms are a function of the underlying measurement model, common factor moments, and latent student growth.

It is important to delineate the contributions of this study to previous research on the reliability of gain scores. Schochet and Chiang (2013) examined classification errors associated with least squares and empirical Bayes estimators of classroom gain scores. Schochet and Chiang made several simplifying assumptions, such as random assignment of teachers to classrooms, perfectly reliable student gain scores, and measurement invariance between classrooms, and recommended that future research should examine how these factors impact the reliability of student growth measures. Vautier, Steyer, and Boomsma (2008) proposed a structural equation model (SEM) to measure student growth between two periods and over multiple methods. Vautier et al. derived the reliability of simple difference scores under the assumption of strong invariance where latent intercepts and loadings are constant over time. The current investigation addresses the aforementioned assumptions. This article defines a general class of LGS for investigating the relative reliability of simple difference and residual scores and uses a latent framework to understand how latent variable model characteristics and the presence of student tracking into pathways with differential growth potential impact the reliability of observed LGS. The results in this article also extend the work of Schochet and Chiang by examining the effect of measurement bias (i.e., classroom differences in the relationship between latent and observed variables) on the reliability of observed student gains.

This article includes four sections. The first section introduces the definition of a class of LGS, in addition to relevant theoretical results. The second section provides an overview of the modified CFM for latent change and derives general expressions used to compute the reliability of observed LGS. The third section examines how issues such as student tracking and measurement bias impact the reliability of observed LGS. The last section provides a discussion of the findings, directions for future research, and concluding remarks.

Measuring Student Growth With LGS

The first subsection introduces the general class of LGS and discusses several special cases that will be examined throughout the remainder of the article. The second subsection includes theoretical information regarding the upper bound to reliability for any LGS.

The General Class of LGS

Researchers have proposed evaluating teachers’ contribution toward student learning by examining the average gain that students make in their test scores over the academic year. Several measures of observed change fall within a general class of LGS. Let $X_{0 j}$ and $X_{1 j}$ represent the pre- and posttest scores for students, where $j$ indexes classrooms (student subscripts are omitted throughout the manuscript). This article assumes that $X_{0 j}$ is recorded prior to instruction and $X_{1 j}$ is collected at the conclusion of instruction. Decision makers and prior researchers have compared posttest scores with a function of pretest scores. Let $g_{j}$ indicate the expected observed difference between posttest scores and a function of students’ pretest scores for classroom $j$ ,

g_{j} = E_{j} (X_{1 j} - f (X_{0 j})),

where $E_{j}$ represents the expectation within classroom $j$ whereas $E$ will be used to denote expectations across classrooms. A larger $g_{j}$ indicates that student growth is, on average, larger in classroom $j$ . Decision makers use $g_{j}$ to make normative comparisons of teacher performance and it is important that $g_{j}$ provides a reliable measure of student growth.

For the general class of LGS, let $f (X_{0 j})$ be a linear function of pretest scores,

f (X_{0 j}) = a + δ X_{0 j},

with constants $a$ and $δ$ representing a shift and slope parameter, respectively. Based on Equation 1, let $g_{δ j} = μ_{1 j} - δ μ_{0 j} - a$ be the expected LGS in classrooms for a given $a$ and $δ$ , where $μ_{0 j} = E_{j} (X_{0 j})$ and $μ_{1 j} = E_{j} (X_{1 j})$ .

Let $Γ_{j}$ represent the true latent growth for individual students within classroom $j$ , and let $γ_{j}$ be the expected latent growth in classroom $j$ (i.e., $γ_{j} = E_{j} (Γ_{j})$ ). $μ_{0 j}$ and $μ_{1 j}$ will vary if classrooms consist of students with different levels of latent achievement and growth. Let $Σ_{01}$ be the variance–covariance matrix for $μ_{0 j}$ and $μ_{1 j}$ with elements $σ_{0}^{2}$ , $σ_{1}^{2}$ , and $σ_{0, 1}$ . The covariances between latent classroom growth and observed classroom means are $σ_{0, γ}$ and $σ_{1, γ}$ . Let $σ_{δ, γ}$ denote the covariance between $g_{δ j}$ and $γ_{j}$ and $σ_{δ}^{2}$ the variance of $g_{δ j}$ :

\begin{array}{l} σ_{δ, γ} = σ_{1, γ} - δ σ_{0, γ} \\ σ_{δ}^{2} = δ^{2} σ_{0}^{2} + σ_{1}^{2} - 2 δ σ_{0, 1} . \end{array}

The reliability of $g_{δ j}$ for a given $a$ and $δ$ is defined as

ρ_{δ}^{2} = \frac{σ_{δ, γ}^{2}}{σ_{δ}^{2} σ_{γ}^{2}} = \frac{{(σ_{1, γ} - δ σ_{0, γ})}^{2}}{(σ_{1}^{2} + δ^{2} σ_{0}^{2} - 2 δ σ_{0, 1}) σ_{γ}^{2}} .

Applied researchers employ two special cases of Equations 1 and 2. Simple difference scores arise when $f$ is the identity function with $a = 0$ and $δ = 1$ . A significant body of prior research has studied the reliability of gain scores based on simple differences, that is, $g_{1 j}$ . Early research (Cronbach & Furby, 1970; Lord, 1956) concluded that difference scores are less reliable than their composite scores and should be avoided. Additional investigations examined situations where gain scores are reliable (Rogosa & Willett, 1983; Williams & Zimmerman, 1996; Zimmerman & Williams, 1998) and offered new methods for modeling growth and change using SEM (Mun, von Eye, & White, 2009; Raykov, 1999; Vautier et al., 2008), item response theory (Verguts & De Boeck, 2000; Verhelst & Glas, 1993; Wang & Chyi-In, 2004), and generalizability theory (Brennan, Yin, & Kane, 2003).

One alternative to simple difference scores are residualized scores, which compares posttest scores to a predicted value based on pretest scores (Cronbach & Furby, 1970; Malgady & Colon-Malgady, 1991). Let $μ_{0}$ and $μ_{1}$ be the average test scores across classrooms at pre- and posttest, respectively (i.e., $μ_{0} = E (X_{0 j})$ and $μ_{1} = E (X_{1 j})$ ) and note that residualized gain scores are represented by the case where $a = μ_{1} - δ μ_{0}$ and $δ = β$ , which is the pooled slope across classrooms that relates observed pre- and posttest scores. The residualized gain scores use the overall pooled slope to calculate predicted scores, which is the ratio of the covariance between $X_{0 j}$ and $X_{1 j}$ across students and classrooms to the total pretest variance and can be expressed using properties of finite mixtures. Specifically, if $π_{j}$ represents the proportion of students in classroom $j$ , the pooled slope is

\begin{array}{l} β = \frac{\sum_{j} π_{j} σ_{0,1 j} + \sum_{j} π_{j} μ_{0 j} μ_{1 j} - μ_{0} μ_{1}}{\sum_{j} π_{j} σ_{0 j}^{2} + \sum_{j} π_{j} μ_{0 j}^{2} - μ_{0}^{2}} = \frac{E (σ_{0,1 j}) + σ_{0,1}}{E (σ_{0 j}^{2}) + σ_{0}^{2}} \end{array},

where $E (σ_{0, 1 j})$ is the expected within-classroom covariance between pre- and posttest scores and $E (σ_{0 j}^{2})$ is the expected within-classroom pretest score variance. This article extends prior investigations (Malgady & Colon-Malgady, 1991; McCaffrey et al., 2004) by considering the relative reliability of $g_{1 j}$ and $g_{β j}$ in the presence of student tracking and classroom-level measurement bias.

The results in this article are valid for any LGS and two additional cases are studied for $a$ and $δ$ . Some researchers or decision makers may naively make judgments concerning student learning and growth using posttest scores. $δ = 0$ is included as a comparison and, as discussed in the results section, tends to be less reliable than both the $δ = 1$ and the $δ = β$ cases. One benefit of using a definition for LGS is that it is possible to identify the value of $δ$ that yields an upper bound for reliability, which is denoted by $δ^{★}$ . The results section compares all of the previously mentioned gain scores to the optimal situation where $δ = δ^{★}$ and $δ^{★}$ will be helpful for understanding the upper bound of reliability for different combinations of student tracking and classroom measurement bias. Let $g_{0 j}$ , $g_{1 j}$ , $g_{β j}$ , and $g_{★ j}$ be the LGS for posttest scores (i.e., $δ = 0$ ), simple difference scores (i.e., $δ = 1$ ), residualized scores (i.e., $δ = β$ ), and the optimal gain scores (i.e., $δ = δ^{★}$ ), respectively.

An Upper Bound for the Class of LGS

The previous subsection introduced the LGS and several special cases. This section includes two theorems. The first theorem shows that $δ^{★}$ yields the maximum reliability and the second theorem reports results used to compare reliability for different values of $δ$ .

Theorem 1: If the variance–covariance matrix among observed classroom means, $Σ_{01}$ , is positive definite, the value of $δ$ that maximizes $ρ_{δ}^{2}$ is

δ^{⋆} = \frac{A_{1}}{A_{2}} = \frac{σ_{0,1} σ_{1, γ} - σ_{0, γ} σ_{1}^{2}}{σ_{0}^{2} σ_{1, γ} - σ_{0, γ} σ_{0,1}} .

Proof. Technical details are included in the Online Appendix.

Theorem 1 implies that the maximum reliability for a given scenario is

ρ_{⋆}^{2} = \frac{σ_{0}^{2} σ_{1, γ}^{2} + σ_{1}^{2} σ_{0, γ}^{2} - 2 σ_{0,1} σ_{0, γ} σ_{1, γ}}{(σ_{0}^{2} σ_{1}^{2} - σ_{0,1}^{2}) σ_{γ}^{2}} .

Theorem 1 demonstrates that $ρ_{δ}^{2}$ is a downward-facing function of $δ$ with a single global maximum. Later results in this article use the fact that $ρ_{δ}^{2}$ has a global maximum to identify a range of values for $δ$ where $ρ_{1}^{2}$ is relatively larger or smaller than $ρ_{β}^{2}$ .

Theorem 2: For any $δ \neq δ^{★}$ , there exists a $δ' \neq δ$ such that $ρ_{δ}^{2} = ρ_{δ'}^{2}$ .

Proof. Technical details are included in the Online Appendix.

Theorem 2 will be useful for establishing conditions where a value of $δ$ yields a relatively more reliable $g_{δ j}$ . The relative reliability of $g_{β j}$ and $g_{δ j}$ for $δ \neq β$ is determined by

\begin{matrix} ρ_{β}^{2} > ρ_{δ}^{2} & \min (δ^{'}, δ) < β < \max (δ^{'}, δ) \\ ρ_{β}^{2} \leq ρ_{δ}^{2} & otherwise \end{matrix} .

The results below compare the reliability of $g_{1 j}$ to $g_{β j}$ , where $δ' = δ_{1}$ for simple difference scores (i.e., $δ = 1$ ) is defined as

δ^{'} = δ_{1} = 2 (\frac{σ_{0,1} ρ_{1}^{2} σ_{γ}^{2} - σ_{1, γ} σ_{0, γ}}{σ_{0}^{2} ρ_{1}^{2} σ_{γ}^{2} - σ_{0, γ}^{2}}) - 1 .

LGS Under the Modified CFM

The purpose of this section is to discuss the general class of LGS in the context of the modified CFM. This section first introduces the modified CFM as a latent variable framework for describing the change between pre- and posttest scores within classrooms and, second, discusses $ρ_{δ}^{2}$ with the aid of CFM parameters.

The CFM With Student Growth

Let $X_{j} = (X_{1 j}, X_{0 j})$ , $W_{j}$ represents the unobserved common factor with mean $κ_{j}$ and variance $ϕ_{j}$ within classroom $j$ , and $Γ_{j}$ is the latent growth for students in classroom $j$ . Let $σ_{Γ j}^{2}$ be the within-classroom variance of student growth and recall that $γ_{j} = E_{j} (Γ_{j})$ . Figure 1 illustrates the modified CFM with student growth where a triangle is used to denote a constant variable for latent intercepts. Figure 1 shows that the CFM assumes that the observed scores (i.e., $X_{j}$ ) relate linearly to $W_{j}$ and $Γ_{j}$ as shown below:

X_{j} = [\begin{matrix} τ_{1 j} \\ τ_{0 j} \end{matrix}] + [\begin{matrix} λ_{1 j} & 1 \\ λ_{0 j} & 0 \end{matrix}] [\begin{matrix} W_{j} \\ Γ_{j} \end{matrix}] + [\begin{matrix} u_{1 j} \\ u_{0 j} \end{matrix}] = τ_{j} + Λ_{j} [\begin{matrix} W_{j} \\ Γ_{j} \end{matrix}] + u_{j},

where $τ_{j}$ is a vector of latent intercepts with elements $τ_{1 j}$ and $τ_{0 j}$ , $Λ_{j}$ is a matrix that includes the common factor loadings $λ_{1 j}$ and $λ_{0 j}$ , and $u_{j}$ is a vector of unique factors such that $u_{j} = (u_{1 j}, u_{0 j})$ . The unique factors represent other characteristics that are independent of $W_{j}$ and $Γ_{j}$ , but impact students’ academic performance. For instance, $u_{j}$ includes other variables that relate to test scores, such as environmental variables like poverty (Sass, Hannaway, Xu, Figlio, & Feng, 2012), the effect of schools (Briggs & Weeks, 2011), other important student characteristics (Hill, Kapitula, & Umland, 2011), and measurement error. $u_{j}$ also includes factors discussed by Schochet and Chiang (2013), such as the specific effects of policies and practices within schools and districts, the effect of school leadership, and classroom factors (e.g., relatively more gifted or disruptive students). Let $E_{j} (u_{j}) = 0$ and the expected value and variance of $X_{j}$ within classroom $j$ are defined as

E_{j} (X_{j}) = [\begin{matrix} μ_{1 j} \\ μ_{0 j} \end{matrix}] = τ_{j} + Λ_{j} [\begin{matrix} κ_{j} \\ γ_{j} \end{matrix}], Σ_{j} = Λ_{j} [\begin{matrix} ϕ_{j} & σ_{W, Γ j} \\ σ_{W, Γ j} & σ_{Γ_{j}}^{2} \end{matrix}] {Λ^{'}}_{j} + Θ_{j},

where $Σ_{j}$ is the covariance matrix of $X_{j}$ and $Θ_{j}$ is a covariance matrix of $u_{j}$ , which includes zeros in the off-diagonal elements (i.e., $u_{0 j}$ and $u_{1 j}$ are independent) and $θ_{1 j}$ and $θ_{0 j}$ on the diagonal. $σ_{W, Γ j}$ denotes the within-classroom covariance between $W_{j}$ and $Γ_{j}$ .

Figure 1.

Common factor model with a latent gain score, $Γ_{j}$ , for classroom $j$ .

Equation 10 presents the general case where classrooms vary in CFM parameters such as $τ_{j}$ , $Λ_{j}$ , and $Θ_{j}$ . In the general case, students with the same latent scores who attend different classes may differ in observed scores, which is referred to as measurement bias as opposed to measurement invariance where observed test scores have the same interpretation for all students regardless of classroom membership (Meredith, 1993; Millsap, 1995, 1997, 1998, 2007; Millsap & Kwok, 2004). Results discussed in this article examine the effect that deviations from measurement invariance have on the reliability of $g_{δ j}$ .

It is important to briefly define additional parameters that are used throughout the article. This article examines how variability and dependence among the CFM parameters impacts the reliability of $g_{δ j}$ . Let the CFM parameters for classroom $j$ be denoted by $Ω_{j} = (τ_{0 j}, τ_{1 j}, λ_{0 j}, λ_{1 j}, κ_{j}, γ_{j})$ and denote the $E (Ω_{j})$ and $Var (Ω_{j})$ as

E (Ω_{j}) = [\begin{matrix} τ_{0} \\ τ_{1} \\ λ_{0} \\ λ_{1} \\ κ \\ γ \end{matrix}], Var (Ω_{j}) = [\begin{matrix} σ_{τ_{0}}^{2} & σ_{τ_{0}, τ_{1}} & σ_{τ_{0}, λ_{0}} & σ_{τ_{0}, λ_{1}} & σ_{τ_{0}, κ} & σ_{τ_{0}, γ} \\ σ_{τ_{1}}^{2} & σ_{τ_{1}, λ_{0}} & σ_{τ_{1}, λ_{1}} & σ_{τ_{1}, κ} & σ_{τ_{1}, γ} \\ σ_{λ_{0}}^{2} & σ_{λ_{0}, λ_{1}} & σ_{λ_{0}, κ} & σ_{λ_{0}, γ} \\ σ_{λ_{1}}^{2} & σ_{λ_{1}, κ} & σ_{λ_{1}, γ} \\ σ_{κ}^{2} & σ_{κ, γ} \\ σ_{γ}^{2} \end{matrix}],

respectively. Several parameters in Equation 12 should be zero in practice. For instance, the average student growth within classroom $j$ should be independent of pretest measurement properties (i.e., $σ_{τ_{0}, γ} = σ_{λ_{0}, γ} = 0$ ). It is also reasonable for latent intercepts in one period to be independent of loadings in another (i.e., $σ_{τ_{0}, λ_{1}} = σ_{τ_{1}, λ_{0}} = 0$ ).

The parameters in Equation 12 have important substantive interpretations. First, $σ_{κ}^{2} > 0$ implies that classrooms differ in underlying common factor means. Furthermore, $σ_{γ}^{2} > 0$ implies that classrooms differ in the expected amount of student growth. Teacher performance affects $γ_{j}$ and $σ_{γ}^{2}$ can be thought of as the upper bound for the degree of true variation in latent teacher performance.

It is important to consider the effect that a relationship between $κ_{j}$ and $γ_{j}$ could have on LGS reliability. A relationship between $κ_{j}$ and $γ_{j}$ exists whenever students are sorted or tracked into pathways that are associated with differential growth potential. For instance, $κ_{j}$ could relate to $γ_{j}$ whenever students are tracked into pathways that consist of better or worse teachers. There is empirical evidence of nonrandom assignment of teachers to classrooms (Hanushek & Rivkin, 2010; Koedel & Betts, 2011). Rothstein (2010) found evidence of a relationship between current teachers and students’past performance and other research suggests that gain scores are related to student characteristics (Hill et al., 2011). Given existing empirical research, this article examines the influence of student tracking on the reliability of $g_{δ j}$ by manipulating the relationship between $κ_{j}$ and $γ_{j}$ , $σ_{κ, γ}$ . That is, student tracking occurs if student gains are related to classroom common factor means (i.e., $σ_{κ, γ} \neq 0$ ), whereas $σ_{κ, γ} = 0$ represents a random process of assigning classrooms to environmental conditions that affect aggregate student growth.

Another issue to examine is the impact of using test scores that satisfy strong invariance (Meredith, 1993; Millsap & Kwok, 2004), which requires $τ_{j} = τ$ and $Λ_{j} = Λ$ , and weak invariance, which only requires that classroom loadings are equal (Millsap & Kwok, 2004). The violation of strong invariance, which is defined as measurement bias (Millsap, 1997, 1998, 2007), exists whenever $τ_{0 j}$ or $τ_{1 j}$ differ across classrooms, which using the parameters in Equation 12 implies that $σ_{τ_{0}}^{2} > 0$ and $σ_{τ_{1}}^{2} > 0$ . Violating strong invariance implies that classrooms with a larger $τ_{1 j}$ will have an artificially larger $E_{j} (X_{1 j})$ and $g_{δ j}$ .

Similarly, violating weak invariance implies that $Λ_{j} \neq Λ$ , $σ_{λ_{0}}^{2} > 0$ , and $σ_{λ_{1}}^{2} > 0$ . In general, classrooms differ in loadings in cases where the relationship between the common factor and observed scores is a function of classroom membership. The loadings can also be interpreted in the context of latent growth curves where researchers constrain loadings to estimate a latent intercept (i.e., a latent variable with loadings set equal to one) and a linear slope (i.e., a latent variable with loadings equal to 0, 1, 2, etc.). In traditional latent curve analyses, researchers interpret individual differences in latent slopes as subject differences in linear growth over time and latent intercepts indicate overall level differences (Culpepper, 2009; Ding, Davison, & Petersen, 2005; Kim, Frisby, & Davison, 2004). More generally, unconstrained loadings represent the typical growth/change trajectories for a group of students. For the modified CFM, the loadings for $Γ_{j}$ are constrained to zero and one in a similar manner as for latent slopes in growth curve models. In contrast, the loadings for the common factor are not constrained to equal one and instead represent a basis function for classroom $j$ where larger values of $W_{j}$ , “ means that there is high similarity between the individual’s curve and the overall group curve” and smaller values indicate “a poor match between the individual’s curve and the group curve” (McArdle & Epstein, 1987, p. 116). $λ_{0 j}$ and $λ_{1 j}$ indicate the pattern of growth that is attributed to the common factor between the pre- and posttests for students in classroom $j$ and $λ_{1 j} λ_{0 j}^{- 1} - 1$ is the percent of change between the pre- and posttests (McArdle & Epstein, 1987). Weak invariance (i.e., $λ_{0 j} = λ_{0}$ and $λ_{1 j} = λ_{1}$ ) implies that the common factor predicts all classrooms to experience the same average growth rate of $λ_{1} λ_{0}^{- 1} - 1$ . In practice, classrooms may differ in student growth rates, which under the modified CFM suggests that weak invariance is violated with $σ_{λ_{0}}^{2} > 0$ and $σ_{λ_{1}}^{2} > 0$ . In short, researchers can expect loadings to vary across classrooms whenever the common factor predicts different growth trajectories and the results in this article examine the reliability of LGS when weak invariance is violated.

The Reliability of LGS Under the CFM

$E_{j} (X_{j})$ implies that the relationship between classroom means and latent growth are generally defined as

σ_{0}^{2} = σ_{τ_{0}}^{2} + σ_{λ_{0} κ}^{2} + 2 σ_{τ_{0}, λ_{0} κ},

σ_{1}^{2} = σ_{τ_{1}}^{2} + σ_{λ_{1} κ}^{2} + σ_{γ}^{2} + 2 (σ_{τ_{1}, λ_{1} κ} + σ_{τ_{1}, γ} + σ_{λ_{1} κ, γ}),

σ_{0,1} = σ_{τ_{0}, τ_{1}} + σ_{τ_{0}, λ_{1} κ} + σ_{τ_{0}, γ} + σ_{τ_{1}, λ_{0} κ} + σ_{λ_{0} κ, λ_{1} κ} + σ_{λ_{0} κ, γ},

σ_{0, γ} = σ_{τ_{0}, γ} + σ_{λ_{0} κ, γ},

σ_{1, γ} = σ_{τ_{1}, γ} + σ_{λ_{1} κ, γ} + σ_{γ}^{2} .

Equations 13 through 17 include variances of products and covariances involving products of random variables (Bohrnstedt & Goldberger, 1969). For instance, $σ_{λ_{0} κ}^{2}$ is the variance of $λ_{0 j} κ_{j}$ and $σ_{λ_{0} κ, λ_{1} κ}$ is the covariance between $λ_{0 j} κ_{j}$ and $λ_{1 j} κ_{j}$ . The results in the next section consider special cases of Equations 13 through 17 for scenarios under the modified CFM.

An expression for the pooled slope as a function of common factor parameters is available by noting that the expected within-classroom pretest variance and pre- and posttest covariance are generally defined as

E (σ_{0,1 j}) = σ_{λ_{0} λ_{1}, ϕ} + ϕ (σ_{λ_{0}, λ_{1}} + λ_{0} λ_{1}) + E (λ_{0 j} σ_{W, Γ j}),

E (σ_{0 j}^{2}) = σ_{λ_{0}^{2}, ϕ} + ϕ (σ_{λ_{0}}^{2} + λ_{0}^{2}) + θ_{0},

where $σ_{λ_{0} λ_{1}, ϕ}$ is the covariance between $λ_{0 j} λ_{1 j}$ and the within-classroom common factor variance, $ϕ_{j}$ , and $σ_{λ_{0}^{2}, ϕ}$ is the covariance between $λ_{0 j}^{2}$ and $ϕ_{j}$ . $E (λ_{0 j} σ_{W, Γ j}) = λ_{0} E (σ_{W, Γ j})$ if pretest loadings are assumed independent of the within-classroom covariance between the common factor and student growth. Equations 13, 15, 18, and 19 show that $β$ is a function of CFM parameters and whether the parameters vary and covary across classrooms.

The Reliability of Observed Teacher Effects

This section discusses the reliability of $g_{δ j}$ (i.e., $ρ_{δ}^{2}$ ) in two scenarios. The first scenario examines the reliability of $g_{δ j}$ in circumstances where classrooms differ in common factor means (i.e., $σ_{κ}^{2} > 0$ ) and typical growth levels (i.e., $σ_{γ}^{2} > 0$ ), classrooms are nonrandomly assigned to different growth possibilities (i.e., the presence of student tracking, $σ_{κ, γ} \neq 0$ ), and test scores satisfy strong invariance at the classroom level (i.e., $τ_{j} = τ$ and $Λ_{j} = Λ$ , which implies that $σ_{τ_{0}}^{2} = σ_{τ_{1}}^{2} = σ_{λ_{0}}^{2} = σ_{λ_{1}}^{2} = 0$ ). The second scenario introduces measurement bias (i.e., $τ_{j} \neq τ$ and/or $Λ_{j} \neq Λ$ ) by manipulating $σ_{τ_{0}}^{2}$ , $σ_{τ_{1}}^{2}$ , $σ_{λ_{0}}^{2}$ , and $σ_{λ_{1}}^{2}$ .

The figures discussed in the following two scenarios examine reliability of $g_{δ j}$ with the expected values of $τ_{1 j}$ , $τ_{0 j}$ , $κ_{j}$ , and $γ_{j}$ across classrooms equal to zero (i.e., $τ_{0} = τ_{1} = κ = γ = 0$ ), $λ_{0} = λ_{1} = 0.8$ , and the expected pretest unique variance is $θ_{0} = 0.2$ . The expected latent common factor variance is the sum of within- and between-classroom variance and $ϕ = 1 - σ_{κ}^{2}$ to set the overall latent variance equal to 1. Consequently, $σ_{κ}^{2}$ represents the share of total latent variance that is attributed to between-classroom differences. The expected relationship between growth and the common factor within classrooms was set to zero (i.e., $E (σ_{W, Γ j}) = 0$ ). Hanushek and Rivkin (2010) summarized 10 value-added studies and presented evidence that the standard deviation of student gains attributed to teacher effects in relation to a standard deviation of latent student achievement ranged from .08 to .36. Given that $ϕ + σ_{κ}^{2} = 1$ , each scenario fixes $σ_{γ}^{2} = 0 . 2^{2}$ , which is about the average value from empirical research (Hanushek & Rivkin, 2010).

Scenario 1: Strong Invariance and Student Tracking

This subsection explores the relative reliability of $g_{δ j}$ in cases where classroom-level growth is related to the average common factor, such as situations where teachers are nonrandomly assigned to classrooms. For instance, $σ_{κ, γ} > 0$ captures cases where better teachers are assigned to classrooms with larger average common factors. In fact, this section demonstrates that reliability declines as $σ_{κ, γ} \to 0$ , which contradicts prior assertions pertaining to gain scores (Koedel & Betts, 2011; Schochet & Chiang, 2013).

In the case where $σ_{κ, γ}$ is unrestricted, Equations 13 through 17 simplify to

\begin{array}{l} σ_{0}^{2} = λ_{0}^{2} σ_{κ}^{2}, σ_{1}^{2} = λ_{1}^{2} σ_{κ}^{2} + σ_{γ}^{2} + 2 λ_{1} σ_{κ, γ}, σ_{0,1} = λ_{0} λ_{1} σ_{κ}^{2} + λ_{0} σ_{κ, γ} \\ σ_{0, γ} = λ_{0} σ_{κ, γ}, σ_{1, γ} = λ_{1} σ_{κ, γ} + σ_{γ}^{2} . \end{array}

The equation for the reliability of $g_{δ j}$ in this scenario is

ρ_{δ}^{2} = 1 - \frac{{(λ_{1} - δ λ_{0})}^{2} (1 - ρ_{κ, γ}^{2}) σ_{κ}^{2}}{Var ((λ_{1} - δ λ_{0}) κ_{j} + γ_{j})} .

Equation 21 demonstrates that student tracking affects $ρ_{δ}^{2}$ . For example, considering teacher performance prior to the assignment of teachers to classrooms results in more reliable LGS. Specifically, $ρ_{δ}^{2} \to 1$ as $ρ_{κ, γ}^{2} \to 1$ .

Equation 21 and Theorem 1 imply that $δ^{★} = λ_{1} / λ_{0}$ if strong invariance is satisfied.

Corollary 1: If pre- and posttest scores satisfy strong invariance where $τ_{j} = τ$ and $Λ_{j} = Λ$ , the slope associated with the optimal reliability is $δ^{★} = λ_{1} / λ_{0}$ .

Proof. Technical details are included in the Online Appendix.

We can examine general conditions where residualized scores outperform gain scores when factors that systematically promote aggregate growth are nonrandomly assigned to classrooms. Clearly, if strong invariance is satisfied, simple difference scores will be optimal whenever $λ_{0} = λ_{1}$ . The slope for calculating residualized scores in this scenario is

β = \frac{E (σ_{0,1 j}) + σ_{0,1}}{E (σ_{0 j}^{2}) + σ_{0}^{2}} = δ^{⋆} C_{0} + \frac{λ_{0} (σ_{κ, γ} + E (σ_{W, Γ j}))}{λ_{0}^{2} (ϕ + σ_{κ}^{2}) + θ_{0}},

where $C_{0} = λ_{0}^{2} (ϕ + σ_{κ}^{2}) {[λ_{0}^{2} (ϕ + σ_{κ}^{2}) + θ_{0}]}^{- 1}$ is the communality of the pretest across all classrooms and represents the ratio of total true variation to observed variation, $ϕ = E (ϕ_{j})$ , and $θ_{0} = E (θ_{0 j})$ . Teacher quality is a resource that could vary over classrooms to create a relationship between $κ_{j}$ and $γ_{j}$ . Equation 22 implies that $β$ is directly affected by the nature of teacher assignment. That is, assigning better teachers to classrooms with lower common factor means (i.e., $σ_{κ, γ} < 0$ ) will tend to reduce $β$ whereas placing better teachers in classrooms with higher achieving students tends to increase $β$ . Note that the within-classroom relationship between students’ common factors and growth has a similar effect as $σ_{κ, γ}$ . Furthermore, $β = δ^{★}$ if the second term in Equation 22 equals $δ^{★} (1 - C_{0})$ . Stated differently, in most applied settings residualized gain scores can only be optimal if better teachers are assigned to higher achieving students (i.e., $σ_{κ, γ} > 0$ ) and/or student growth is positively related to the common factor within classrooms (i.e., $E (σ_{W, Γ j}) > 0$ ).

Figure 2 examines the reliability of $g_{δ j}$ for different combinations of $ρ_{κ, γ}$ and $λ_{1} / λ_{0}$ under the assumption that 20% of variance in latent achievement occurs between classrooms (i.e., $σ_{κ}^{2} = 0.2$ ) and the within-classroom relationship between the common factor and learning gains are independent (i.e., $E (σ_{W, Γ j}) = 0$ ). Note that under strong invariance that $ρ_{★}^{2} = 1$ and contour plots are omitted from Figure 2.

Figure 2.

Contours of impact of $σ_{κ, γ}$ and $λ_{1} / λ_{0}$ on $ρ_{δ}^{2}$ for $δ = β$ , 1, and 0.

As noted in Equation 21, $ρ_{δ}^{2} \to 1$ as $ρ_{κ, γ}^{2} \to 1$ . It is important to note that $ρ_{δ}^{2} \to 1$ steadily if $ρ_{κ, γ} > 0$ whereas $ρ_{δ}^{2}$ initially declines for $ρ_{κ, γ} < 0$ , but quickly approaches 1 as $ρ_{κ, γ} \to - 1$ . Figure 2 shows that $ρ_{1}^{2} \to 1$ as $λ_{1} / λ_{0} \to 1$ regardless of whether students are tracked. However, Figure 2 shows combinations of $ρ_{κ, γ}$ and $λ_{1} / λ_{0}$ , where $ρ_{1}^{2}$ falls below 0.8. Specifically, $ρ_{1}^{2}$ sharply declines as $λ_{1} / λ_{0}$ deviates from 1. $ρ_{β}^{2}$ is larger when $λ_{0} > λ_{1}$ (i.e., the common factor predicts a declining growth pattern) and falls below 0.8 when $λ_{1} > λ_{0}$ (i.e., the common factor predicts an increasing growth pattern) and $ρ_{κ, γ} = 0$ . In short, Figure 2 illustrates that $ρ_{1}^{2}$ is primarily determined by $λ_{1} / λ_{0}$ , whereas $ρ_{β}^{2}$ is affected by both student tracking and loading differences.

Scenario 2: Measurement Bias and Student Tracking

This scenario examines the reliability of $g_{δ j}$ for the case where $κ_{j}$ , $τ_{j}$ , $Λ_{j}$ , $ϕ_{j}$ , and $θ_{0 j}$ are independent random variables and the typical classroom-level student growth (i.e., $γ_{j}$ ) relates to $κ_{j}$ . Classroom latent intercepts and loadings are allowed to relate between periods (i.e., as indicated by $σ_{τ_{0}, τ_{1}}$ and $σ_{λ_{0}, λ_{1}}$ ). In this scenario, manipulations of Equations 13 to 17 (see the Online Appendix for details) imply that $ρ_{δ}^{2}$ is,

ρ_{δ}^{2} = \frac{Var ((λ_{1} - δ λ_{0}) κ_{j} + γ_{j}) - {(λ_{1} - δ λ_{0})}^{2} (1 - ρ_{κ, γ}^{2}) σ_{κ}^{2}}{Var ((λ_{1} - δ λ_{0}) κ_{j} + γ_{j}) + Var (τ_{1 j} - δ τ_{0 j}) + (κ^{2} + σ_{κ}^{2}) Var (λ_{1 j} - δ λ_{0 j})} .

It is immediately clear from Equation 23 that $ρ_{δ}^{2}$ declines in the presence of measurement bias and increases as $ρ_{κ, γ}^{2} \to 1$ . In contrast to the previous scenarios, $ρ_{★}^{2} < 1$ whenever weak or strong invariance is violated. Note that the expressions for $β$ and $δ^{★}$ are more cumbersome in this scenario and are reported in the Online Appendix.

Figure 3 shows the effect of $σ_{τ}^{2} / σ_{κ}^{2}$ and $σ_{λ}^{2} / σ_{κ}^{2}$ on $ρ_{δ}^{2}$ (for $δ$ = 0, 1, $β$ , and $δ^{★}$ ) for $σ_{κ}^{2} = 0.2$ , $σ_{κ, γ} = 0.5$ , $σ_{τ_{0}}^{2} = σ_{τ_{1}}^{2}$ , and $σ_{λ_{0}}^{2} = σ_{λ_{1}}^{2}$ . Figure 3 demonstrates that even modest degrees of measurement bias relative to between-classroom latent variance severely reduces the optimal reliability $ρ_{★}^{2}$ . Classroom differences in $τ_{j}$ has a stronger effect on reliability than differences in loadings. $ρ_{β}^{2}$ more closely resembles the surface of $ρ_{★}^{2}$ than $ρ_{1}^{2}$ . The findings suggest that $ρ_{1}^{2} > ρ_{β}^{2}$ for small degrees of measurement bias, but $ρ_{1}^{2}$ declines more quickly as latent intercepts deviate between classes. Figure 3 shows that $ρ_{0}^{2} < 0.6$ and is too unreliable for practical purposes whenever measurement bias exists between classrooms.

Figure 3.

Contours of impact of $σ_{τ}^{2} / σ_{κ}^{2}$ and $σ_{λ}^{2} / σ_{κ}^{2}$ on $ρ_{δ}^{2}$ for $δ = β$ , 1, 0, and $δ^{★}$ .

Discussion

This article examined the accuracy of commonly used measures of student growth, which are used to evaluate teachers. This article included new theoretical results for LGS and evaluated the reliability of traditional indicators of student learning with the aid of a latent variable framework. The remainder of the manuscript summarizes key findings, discusses the practical implications of the results, and offers ideas for future research.

It is important to describe the contributions of this study in the context of existing research. First, this article offers new theoretical results to existing research on the reliability of gain scores. Prior research examined the reliability of residual and simple difference scores. This article framed gain scores in terms of a general class of LGS and identified a pretest slope $δ^{★}$ that is associated with the optimal reliability $ρ_{★}^{2}$ . The results in this article identified general conditions where $g_{δ j}$ is preferred for different values of $δ$ . New results were reported for situations where researchers should prefer residualized scores to difference scores, and both observed measures of student growth were compared with $ρ_{★}^{2}$ .

Second, this article builds on prior research that used the CFM to clarify issues related to measurement bias and prediction bias (Culpepper, 2012; Millsap, 1997, 1998, 2007). In fact, the findings in this study show that classroom-level differences in test score properties have significant implications for the interpretation of gain scores and for measures of teacher performance that arise from those scores. Furthermore, the use of the CFM as a latent variable framework offered an examination of the reliability of $g_{δ j}$ that was unavailable with previous approaches. For instance, Koedel and Betts (2011) and Schochet and Chiang (2013) conjectured that nonrandom assignment of teachers into classrooms would decrease the reliability of difference scores. The results in this article demonstrated that, contrary to previous conjectures, the presence of student tracking inflates the reliability of $g_{δ j}$ . Recall that the inflation in LGS reliability arises when tracking occurs in either direction (i.e., $ρ_{κ, γ} = - 1$ or $ρ_{κ, γ} = 1$ ). One explanation is that the process of nonrandom student assignment inherently uses teacher performance information, so observed measures of classroom growth simply reflect the information used to assign students to classrooms. It is important to emphasize that the increase in LGS reliability attributed to nonrandom assignment of students to classrooms is an artifact of student tracking and should not be considered a strength of LGS as a method for measuring classroom-level growth. Instead, LGS are most valuable for decision makers in cases where an analysis of pre- and posttest scores provides a unique contribution to a profile of information that is available on student growth and teacher performance.

Third, this article contributes to the dialogue pertaining to teacher evaluation. Namely, school districts are increasingly adopting statistical modeling techniques that rely on gain scores or residualized gain scores as a way to judge teacher quality. The results in this article suggest that more research is needed to understand classroom measurement bias. If strong invariance is satisfied $g_{1 j}$ is nearly as reliable as $g_{★ j}$ . However, LGS reliability is compromised if measurement bias exists between classrooms. Figure 3 showed that $g_{★ j}$ is a substandard indicator of student learning at the classroom level when there are modest amounts of measurement bias variance relative to common factor variance. In fact, $ρ_{★}^{2} < 0.75$ when $σ_{λ}^{2} / σ_{κ}^{2} = 0.02$ and $σ_{τ}^{2} / σ_{κ}^{2} = 0.05$ . One area for future research is to assess the extent to which latent intercepts and loadings differ across classrooms. Given that neither of the standard measures of teacher performance can be counted on when tests are biased, researchers should develop new alternatives (e.g., Vautier et al., 2008) that model relationships among latent variables rather than observed scores.

Fourth, this article offers some guidance about the relative performance of residual and gain scores in a variety of situations. Researchers currently use $δ = β$ and/or $δ = 1$ as a measure of student learning and/or growth. The most important distinction identified between $g_{1 j}$ and $g_{β j}$ is that using a pooled slope tends to yield larger reliabilities when classroom latent intercepts and loadings are more variable (e.g., see Figure 3). $g_{1 j}$ is most reliable in cases where strong invariance is satisfied and pre- and posttest loadings are nearly equal.

Another direction for future research is to extend the approach discussed by Vautier et al. (2008). For instance, some models incorporate test scores from several years and several content areas. A natural extension of this article would be to examine measures of classroom growth when student performance over time is a function of the CFM and one or more teachers or schools (Briggs & Weeks, 2011). Past teachers likely impact students’ performance on future exams and the CFM could be used to understand the reliability of observed growth in instances where learning effects decay over time. Similarly, previous research examined student learning across more than one content domain in addition to over time. Future research could expand the CFM by including additional latent and observed variables for other content areas and examine how changes in student characteristics impacts multivariate measures of growth.

In conclusion, this study continues an important discussion about technical issues surrounding the reliability of gain scores, which are increasingly used to evaluate teachers. As noted earlier, the popular press has tried to make sense of highly technical measurement and psychometric issues and interest in teacher evaluations will continue to grow with the use of test scores to rate teachers. Some school districts seem poised to use statistical models to evaluate teachers. The stakes are high for educators to ensure that statistical tools yield reliable teacher ratings, especially given that decision makers are using, or planning to use, model estimates to reward and/or punish teachers. The findings in this study shed light on the relative reliability of different types of gain scores for measuring student growth and ranking teachers’ performance. In short, the nature of measurement bias and student tracking are important factors to understand regarding LGS reliability and the results in this article can serve as a guide as researchers, statisticians, and psychometricians develop and refine methods for teacher evaluation.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

The online appendix is available at .

References

Battauz

Bellio

Gori

(2011). Covariate measurement error adjustment for multilevel models with application to educational data. Journal of Educational and Behavioral Statistics, 36, 283-306.

Bohrnstedt

Goldberger

(1969). On the exact covariance of products of random variables. Journal of the American Statistical Association, 64, 1439-1442.

Bond

(1979). On the base-free measure of change proposed by Tucker, Damarin and Messick. Psychometrika, 44, 351-355.

Brennan

Yin

Kane

(2003). Methodology for examining the reliability of group mean difference scores. Journal of Educational Measurement, 40, 207-230.

Briggs

D. C.

Weeks

J. P.

(2009). The sensitivity of value-added modeling to the creation of a vertical score scale. Education Finance and Policy, 4, 384-414.

Briggs

D. C.

Weeks

J. P.

(2011). The persistence of school-level value-added. Journal of Educational and Behavioral Statistics, 36, 616-637.

Cronbach

Furby

(1970). How we should measure “change”—Or should we? Psychological Bulletin, 74, 68-80.

Culpepper

S. A.

(2009). A multilevel nonlinear profile analysis model for dichotomous data. Multivariate Behavioral Research, 44, 646-667.

Culpepper

S. A.

(2012). Using the criterion-predictor factor model to compute the probability of detecting prediction bias with ordinary least squares regression. Psychometrika, 77, 561-580.

10.

Culpepper

S. A.

Aguinis

(2011). Using analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16, 166-178.

11.

Ding

C. S.

Davison

M. L.

Petersen

A. C.

(2005). Multidimensional scaling analysis of growth and change. Journal of Educational Measurement, 42, 171-191.

12.

Garland

(2012, February 8). Tennessee teacher evaluations: Using value-added analysis. The Huffington Post. Retrieved from http://www.huffingtonpost.com

13.

Hanushek

Rivkin

(2010). Generalizations about using value-added measures of teacher quality. American Economic Review, 100, 267-271.

14.

Hill

Kapitula

Umland

(2011). A validity argument approach to evaluating teacher value-added scores. American Educational Research Journal, 48, 794-831.

15.

Kim

S.-K.

Frisby

C. L.

Davison

M. L.

(2004). Estimating cognitive profiles using Profile Analysis via Multidimensional Scaling (PAMS). Multivariate Behavioral Research, 39, 595-624.

16.

Koedel

Betts

(2011). Does student sorting invalidate value-added models of teacher effectiveness? An extended analysis of the Rothstein critique. Education Finance and Policy, 6, 18-42.

17.

Lockwood

J. R.

McCaffrey

D. F.

Mariano

L. T.

Setodji

(2007). Bayesian methods for scalable multivariate value-added assessment. Journal of Educational and Behavioral Statistics, 32, 125-150.

18.

Lord

(1956). The measurement of growth. Educational and Psychological Measurement, 16, 421-437.

19.

Malgady

Colon-Malgady

(1991). Comparing the reliability of difference scores and residuals in analysis of covariance. Educational and Psychological Measurement, 51, 803-807.

20.

McArdle

Epstein

(1987). Latent growth curves within developmental structural equation models. Child Development, 58, 110-133.

21.

McCaffrey

D. F.

Lockwood

J. R.

Koretz

Louis

T. A.

Hamilton

(2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29, 67-101.

22.

Millsap

R.E.

(1995). Measurement invariance, predictive invariance, and the duality paradox. Multivariate Behavioral Research, 30(4), 577-605.

23.

Millsap

R. E.

(1995). Invariance in measurement and prediction: Their relationship in the single-factor case. Psychological Methods, 2, 248-260.

24.

Millsap

R. E.

(1997). Invariance in measurement and prediction: Their relationship in the single-factor case. Psychological Methods, 2, 248-260.

25.

Millsap

R. E.

(1998). Group differences in regression intercepts: Implications for factorial invariance. Multivariate Behavioral Research, 33, 403-424.

26.

Millsap

R. E.

(2007). Invariance in measurement and prediction revisited. Psychometrika, 72, 461-473.

27.

Millsap

R. E.

Kwok

(2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93-115.

28.

Mun

von Eye

White

(2009). An SEM approach for the evaluation of intervention effects using pre-post designs. Structural Equation Modeling: A Multidisciplinary Journal, 16, 315-337.

29.

Papay

(2011). Different tests, different answers: The stability of teacher value-added estimates across outcome measures. American Educational Research Journal, 48, 163-193.

30.

Raudenbush

(2004). What are value-added models estimating and what does this imply for statistical practice. Journal of Educational and Behavioral Statistics, 29, 121-129.

31.

Raykov

(1999). Are simple change scores obsolete? An approach to studying correlates and predictors of change. Applied Psychological Measurement, 23, 120-126.

32.

Reardon

Raudenbush

(2009). Assumptions of value-added models for estimating school effects. Education Finance and Policy, 4, 492-519.

33.

Rogosa

Willett

(1983). Demonstrating the reliability of the difference score in the measurement of change. Journal for Educational Measurement, 20, 335-343.

34.

Rothstein

(2010). Teacher quality in educational production: Tracking, decay, and student achievement. Quarterly Journal of Economics, 125, 175-214.

35.

Rubin

Stuart

Zanutto

(2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29, 103-116.

36.

Sass

Hannaway

Figlio

Feng

(2012). Value added of teachers in high-poverty schools and lower poverty schools. Journal of Urban Economics, 72, 104-122.

37.

Schochet

Chiang

(2013). What are error rates for classifying teacher and school performance using value-added models? Journal of Educational and Behavioral Statistics, 38, 142-171.

38.

Tucker

Damarin

Messick

(1966). A base-free measure of change. Psychometrika, 31, 457-473.

39.

Vautier

Steyer

Boomsma

(2008). The true-change model with individual method effects: Reliability issues. British Journal of Mathematical and Statistical Psychology, 61, 379-399.

40.

Verguts

De Boeck

(2000). A Rasch model for detecting learning while solving an intelligence test. Applied Psychological Measurement, 24, 151-162.

41.

Verhelst

N. D.

Glas

C. A.

(1993). A dynamic generalization of the Rasch model. Psychometrika, 58, 395-415.

42.

Wang

Chyi-In

(2004). Gain score in item response theory as an effect size measure. Educational and Psychological Measurement, 64, 758-780.

43.

Watanbe

(2011, March 28). “Value-added” teacher evaluations: L.A. unified tackles a tough formula. Los Angeles Times. Retrieved from http://www.latimes.com

44.

Williams

Zimmerman

(1996). Are simple gain scores obsolete? Applied Psychological Measurement, 20, 59-69.

45.

Winerip

(2011, March 6). Evaluating New York teachers, perhaps the numbers do lie. The New York Times. Retrieved from http://www.nytimes.com

46.

Woodhouse

Yang

Goldstein

Rasbash

(1996). Adjusting for measurement error in multilevel analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society), 159, 201-212.

47.

Zimmerman

Williams

(1998). Reliability of gain scores under realistic assumptions about properties of pretest and posttest scores. British Journal of Mathematical and Statistical Psychology, 51, 343-351.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.50 MB