Impact Evaluation Using Analysis of Covariance With Error-Prone Covariates That Violate Surrogacy

Abstract

Background:

Analysis of covariance (ANCOVA) is commonly used to adjust for potential confounders in observational studies of intervention effects. Measurement error in the covariates used in ANCOVA models can lead to inconsistent estimators of intervention effects. While errors-in-variables (EIV) regression can restore consistency, it requires surrogacy assumptions for the error-prone covariates that may be violated in practical settings.

Objectives:

The objectives of this article are (1) to derive asymptotic results for ANCOVA using EIV regression when measurement errors may not satisfy the standard surrogacy assumptions and (2) to demonstrate how these results can be used to explore the potential bias from ANCOVA models that either ignore measurement error by using ordinary least squares (OLS) regression or use EIV regression when its required assumptions do not hold.

Results:

The article derives asymptotic results for ANCOVA with error-prone covariates that cover a variety of cases relevant to applications. It then uses the results in a case study of choosing among ANCOVA model specifications for estimating teacher effects using longitudinal data from a large urban school system. It finds evidence that estimates of teacher effects computed using EIV regression may have smaller bias than estimates computed using OLS regression when the data available for adjusting for students’ prior achievement are limited.

Keywords

measurement error errors-in-variables regression selection on observables analysis of covariance surrogacy teacher value-added

Knowledge of the causal effects of interventions, programs, treatments, and even individuals (e.g., teachers or physicians) is of great value to policy and decision makers. Learning about such effects from data often involves observational studies, in which naturally occurring variation in exposure to treatments¹ is used to estimate the effects of those treatments on outcomes of interest. A well-known limitation of such studies is the potential for units with differing levels of the exposure also to differ on other factors that are related to the outcomes. These other factors can confound the estimated causal effects. A common method to reduce the potential for confounding is to make adjustments for “covariates,” observed measures of the factors that are related to the outcomes and that might vary among units with different exposures.

Historically, a primary method for adjusting for covariates has been ordinary least squares (OLS) linear regression of the outcome on the covariates and indicator (dummy) variables for treatment status. This method is commonly referred to as “analysis of covariance” (ANCOVA). Although there has been substantial research on alternatives to ANCOVA for the estimation of causal effects (e.g., Imbens & Rubin, 2015; Pearl, Glymour, & Jewell, 2016; Shadish, Cook, & Campbell, 2002), it remains widely used. For example, it is commonly used to estimate treatment effects from data arising from pretest–posttest designs (e.g., Yang & Tsiatis, 2001). It is also an appealing choice in settings with a potentially large number of treatment groups, each with a relatively small number of units, which poses practical difficulties for alternatives to ANCOVA such as matching or weighting estimators. A recent, high-profile example of such an application is so-called value-added modeling for estimating the effects of teachers or schools on student achievement using longitudinal test score data (Harris, 2011; McCaffrey, Lockwood, Koretz, Louis, & Hamilton, 2004).

A key assumption required for ANCOVA to yield unconfounded estimates of treatment effects is the assumption of “selection on observables” or “no unobserved confounders” (Imbens, 2004; Rosenbaum & Rubin, 1983). This assumption means that conditional on the covariates included in the ANCOVA model, there is no variable that both predicts the outcomes and has different distributions across treatment groups. Provided this assumption holds, and provided the functional form assumptions of the ANCOVA model are correct and other standard regularity conditions hold, then treatment effect estimators from the ANCOVA model will be consistent.

Since as early as 1959, measurement error in covariates has been recognized as a complication for making accurate inferences from ANCOVA models (Scheffe, 1959). Specifically, if the selection-on-observables assumption would hold conditional on a set of adjustment variables $H$ , but the observed covariates $\tilde{H}$ measure any of the components of $H$ with error, then the selection-on-observables assumption generally will not hold conditional on $\tilde{H}$ (e.g., Steiner, Cook, & Shadish, 2011; Yi, Ma, & Carroll, 2012). Thus, an ANCOVA model in which observed covariates $\tilde{H}$ are used in place of their corresponding latent variables $H$ can lead to confounded treatment effect estimators (e.g., Culpepper & Aguinis, 2011; Lockwood & McCaffrey, 2014).

A standard method for restoring consistent estimation in linear models with error-prone covariates is errors-in-variables (EIV) regression (Carroll, Ruppert, Stefanski, & Crainiceanu, 2006; Fuller, 2006). If information about the variances and covariances of the measurement errors in error-prone covariates is known, or can be estimated, then EIV regression can yield consistent estimators of treatment effects when covariates are measured with error, provided that other assumptions hold. One of the critical assumptions involves what is called “surrogacy” in the measurement error literature (Buonaccorsi, 2010; Carroll et al., 2006; Lockwood & McCaffrey, 2016). In ANCOVA models, the surrogacy assumption is that measurement errors are uncorrelated with the latent values of error-prone covariates, any other error-free covariates in the model, the treatment group indicators, and the residual error term of the model.

Unfortunately, practical settings pose various opportunities for failure of the surrogacy assumption. First, measurement errors may be related to the levels of treatment individuals receive. For example, test scores from achievement assessments measure latent achievement with error (Lord, 1980), and school staff might use test scores, along with other variables correlated with achievement, to select students for a reading intervention or to assign students to advanced or remedial classes. This can result in correlations between measurement errors and treatment status. Second, outcomes might depend on the measurement errors. For example, blood pressure is known to be routinely measured with error (Burkard et al., 2018), and a particularly high reading might encourage people to change their diets and increase exercise in ways that could affect their future health status. Both of these scenarios violate the surrogacy assumption and generally will cause EIV regression to yield biased estimates of treatment effects. Although there are multiple sources for analytic derivations of the bias in ANCOVA using error-prone covariates without a correction for measurement error (e.g., Culpepper & Aguinis, 2011), we are unaware of the corresponding results for the bias when using EIV regression in ANCOVA models when the required assumptions do not hold.

This study has two objectives. First, we derive asymptotic results for ANCOVA using EIV regression when measurement errors may not satisfy the standard surrogacy assumptions. We show that the magnitude of asymptotic bias in estimated treatment effects depends on a combination of true model coefficients, the degree of sorting of units to treatment groups with respect to covariates, and the relationships of measurement errors to various quantities. Second, using a case study of estimating teacher effects on student achievement, we demonstrate a method for using data to explore the potential bias from using either OLS or EIV to estimate ANCOVA models when the assumptions for both methods may be violated. The results of the method can help to guide which estimation approach is likely to have smaller bias in a given setting. We conclude with a discussion of the findings, their limitations, and areas for future work.

Asymptotic Results for EIV Regression With Surrogacy Violations

This section first defines the structural model, which allows for the possibility that measurement errors in error-prone covariates may not satisfy the standard surrogacy assumptions. It then defines the EIV regression estimator of the parameters of the structural model, derives the probability limit for this estimator, and derives a scalar summary of the asymptotic bias across treatment groups.

Structural Model

Consider the ANCOVA model:

Y_{i} = θ' T_{i} + β' H_{i} + λ G_{i} + ε_{i}

for independent and identically distributed (IID) samples $i = 1, \dots, n$ from some target population. The variable $Y_{i}$ is an observed outcome of interest. The vector $T_{i}$ contains dummy variables for a fixed number $T$ of groups, with component $t$ equal to 1 if unit $i$ is in group $t$ and 0 otherwise. We assume that each unit $i$ is linked to exactly one group, so that $1' T_{i} = 1$ for each $i = 1, \dots, n$ . Thus, the model excludes an intercept, making the parameters $θ$ for the groups equal to adjusted group means. In typical settings, “treatment effects” would be defined by contrasts among the components of $θ$ , such as the comparison of each group to a baseline group or to the average of all groups. Our focus on the adjusted group means directly rather than contrasts simplifies the analytical results and does not affect the substantive conclusions.

The variable $H_{i}$ in Equation 1 is a vector of length $P$ . The scalar $G_{i}$ in Equation 1 represents a variable that may belong in the model, and generally may be correlated with elements of $T_{i}$ and/or $H_{i}$ , but is not used in the estimation and thus is a potential source of bias. We refer to $G_{i}$ as an “omitted variable” since it is not included in the analysis. There is no loss of generality by assuming $G_{i}$ is a scalar. If there were multiple omitted variables, $G_{i}$ could equal the inner product of the variables and their associated model coefficients, and $λ$ would then equal to 1. The error term $ε_{i}$ in Equation 1 is assumed to satisfy $E [ε_{i} | T_{i}, H_{i}, G_{i}] = 0$ . Without the loss of generality, we assume that $E [H_{i}] = 0$ and $E [G_{i}] = 0$ because this simplifies some later expressions. We assume that standard regularity conditions apply, so that if $H_{i}$ were observed and if $λ E [G_{i} | T_{i}, H_{i}] = 0$ , then OLS regression of $Y_{i}$ on $T_{i}$ and $H_{i}$ would consistently estimate $(θ', β')'$ . One such regularity condition that is important for our derivations is that the probability $π_{t}$ that component $t$ of $T_{i}$ is equal to 1 satisfies $π_{t} > 0$ for all $t = 1, \dots, T$ .

This article focuses on the case where $H_{i}$ is not directly observed² but rather is measured with error by ${\tilde{H}}_{i} = H_{i} + U_{i}$ . We assume that the vector of measurement errors $U_{i}$ satisfies $E [U_{i}] = 0$ . To make the notation more compact, we allow components of $U_{i}$ to be identically equal to 0, corresponding to components of $H_{i}$ that are measured without error by the corresponding components of ${\tilde{H}}_{i}$ . Thus, ${\tilde{H}}_{i}$ generally may consist of both error-free and error-prone components.

The assumptions about the measurement errors $U_{i}$ are the focus of this study. The surrogacy assumption made in standard EIV regression is that $E [U_{i} | T_{i}]$ , $E [U_{i} | H_{i}]$ , $E [U_{i} | G_{i}]$ , and $E [U_{i} | ε_{i}]$ all equal $0$ . This assumption means that $U_{i}$ is uncorrelated with all quantities determining the outcomes $Y_{i}$ under Model 1, including the treatment assignments, the latent covariates $H_{i}$ , and all other unobserved or unmodeled variables that affect the outcomes. This is a strong assumption, and as previously noted, there are often reasons to doubt this assumption in practical applications. The primary purpose of this study is to explore the implications of the failure of the assumption. We focus on the case where $E [U_{i} | H_{i}]$ , $E [U_{i} | G_{i}]$ , and $E [U_{i} | ε_{i}]$ equal $0$ , but where $E [U_{i} | T_{i}]$ may not equal $0$ .³ This assumption corresponds to the case where measurement errors have no direct relationship to outcomes but may be related to treatment assignments. Lockwood and McCaffrey (2016) use the term “weak surrogacy” to refer to such a case. One mechanism for a relationship between measurement errors and treatment assignments would be if ${\tilde{H}}_{i}$ directly influenced the assignments of units to treatment groups, which can induce correlation between elements of $U_{i}$ and elements of $T_{i}$ . As noted previously, such a situation is plausible in many applications with error-prone covariates, so that understanding estimator behavior in this situation is valuable.

We denote the variance–covariance matrix of $U_{i}$ for unit $i$ by $Σ_{U, i}$ , a $(P \times P)$ positive semidefinite matrix that is allowed to vary across units $i$ to account for the possibility of heteroscedastic measurement errors. Diagonal and corresponding off-diagonal elements of $Σ_{U, i}$ may be 0 due to degeneracy of some components of $U_{i}$ for error-free covariates. We assume that $E [Σ_{U, i}] : = Σ_{U}$ exists and is positive semidefinite. Note that because we are assuming that $E [U_{i}] = 0$ for all $i$ , the matrix $Σ_{U}$ is the marginal variance–covariance matrix of the measurement errors in the population of units. Let $Σ_{H}$ be the $(P \times P)$ positive definite variance–covariance matrix of $H_{i}$ , let $Σ_{H G}$ be the $(P \times 1)$ vector of covariances between $H_{i}$ and $G_{i}$ , and let $Σ_{G}$ be the variance of $G_{i}$ . Then, the marginal variance–covariance matrix of $({U'}_{i}, {H'}_{i}, G_{i})'$ for a unit $i$ randomly sampled from the population is the $(2 P + 1) \times (2 P + 1)$ matrix:

Σ_{U H G} = [\begin{matrix} Σ_{U} & 0 & 0 \\ 0 & Σ_{H} & Σ_{H G} \\ 0 & {Σ'}_{H G} & Σ_{G} \end{matrix}] .

The zero blocks follow from the assumption that $U_{i}$ is uncorrelated with both $H_{i}$ and $G_{i}$ . This implies that the marginal variance–covariance matrix $Σ_{\tilde{H}}$ of ${\tilde{H}}_{i}$ for a unit $i$ randomly sampled from the population is $Σ_{H} + Σ_{U}$ .

The EIV Regression Estimator

We assume that the goal is to estimate the parameters $(θ', β')'$ in Equation 1 from observed data ${(Y_{i}, T_{i}, {\tilde{H}}_{i})}_{i = 1}^{n}$ using EIV regression of $Y_{i}$ on $T_{i}$ and ${\tilde{H}}_{i}$ . Let $Y_{n} = (Y_{1}, \dots, Y_{n})'$ , and let $Q_{n}$ denote the $n \times (T + P)$ design matrix with rows $({T^{'}}_{i}, {\tilde{H}}_{i}^{'})$ . Let $C_{n}$ be a $(T + P) \times (T + P)$ matrix with all zero entries, except for the lower right-hand $(P \times P)$ block equal to $n {\tilde{Σ}}_{U}^{(n)}$ , where ${\tilde{Σ}}_{U}^{(n)}$ is a working variance–covariance matrix of the measurement errors, discussed in additional detail below. Then, the EIV regression estimator of $(θ', β')'$ is

(\hat{θ}', \hat{β}')' = ({Q'}_{n} Q_{n} - C_{n})^{- 1} {Q'}_{n} Y_{n} .

The EIV estimator differs from the OLS estimator only through the subtraction of $C_{n}$ , which is intended to correct the elements of ${Q'}_{n} Q_{n}$ that are distorted by the measurement errors. Additional details on the EIV regression estimator can be found in Fuller and Hidiroglou (1978) as well as in texts such as Buonaccorsi (2010), Carroll, Ruppert, Stefanski, and Crainiceanu (2006), and Fuller (2006).

Specifying the working variance–covariance matrix of the measurement errors, ${\tilde{Σ}}_{U}^{(n)}$ , is the critical step of applying EIV regression. The specification of ${\tilde{Σ}}_{U}^{(n)}$ depends on what is known about the magnitude of measurement errors, and the superscripting by $n$ is used to indicate that ${\tilde{Σ}}_{U}^{(n)}$ may be a function of the observed data. This accommodates several scenarios that commonly arise in applications involving error-prone covariates, and we itemize some of those scenarios here. First, when $Σ_{U, i}$ is known for $i = 1, \dots, n$ , then ${\tilde{Σ}}_{U}^{(n)}$ could be $\frac{1}{n} \sum_{i = 1}^{n} Σ_{U, i}$ . This also covers the case of homoscedastic measurement error where $Σ_{U, i}$ is a known constant matrix $Σ_{U}$ and $\frac{1}{n} \sum_{i = 1}^{n} Σ_{U, i} \equiv Σ_{U}$ . Second, in some applications, such as those involving achievement tests, $Σ_{U, i}$ may equal $v (H_{i})$ for a known function $v$ (e.g., Lord, 1980). In such applications, even though $v (H_{i})$ is not observed for any $i$ because $H_{i}$ is latent, ${\tilde{Σ}}_{U}^{(n)}$ may be specified as an estimator of $Σ_{U} = E [v (H_{i})]$ computed from the observed data using deconvolution methods (Lockwood & McCaffrey, 2014, 2017). Third, in many applications, analysts may know the “reliability” of each error-prone covariate (e.g., Crocker & Algina, 1986). Letting $H_{i} = (H_{i 1}, \dots, H_{i P})'$ , $U_{i} = (U_{i 1}, \dots, U_{i P})'$ , and ${\tilde{H}}_{i} = ({\tilde{H}}_{i 1}, \dots, {\tilde{H}}_{i P})'$ , the reliability $r_{j}$ of ${\tilde{H}}_{i j}$ for $j = 1, \dots, P$ is $Var [H_{i j}] / (Var [H_{i j}] + Var [U_{i j}])$ . Note that $0 < r_{j} \leq 1$ , with $r_{j} = 1$ for any component $j$ of $H_{i}$ that is measured without error. Also note that $Var [U_{i j}] = (1 - r_{j}) Var [{\tilde{H}}_{i j}]$ for each $j$ , under the assumption that $E [U_{i} | H_{i}] = 0$ . Thus, when $(r_{1}, \dots, r_{P})$ is known and it is assumed that measurement errors for unit $i$ are uncorrelated across components $j$ , it is common to specify ${\tilde{Σ}}_{U}^{(n)}$ as a diagonal matrix with diagonal element $j$ equal to $(1 - r_{j}) \hat{Var} [{\tilde{H}}_{i j}]$ , where $\hat{Var} [{\tilde{H}}_{i j}]$ is an estimator of $Var [{\tilde{H}}_{i j}]$ (e.g., Fuller & Hidiroglou, 1978). Finally, note that setting ${\tilde{Σ}}_{U}^{(n)} \equiv 0$ corresponds to OLS regression, so that EIV regression can be viewed as a generalization of OLS regression, and the asymptotic results presented later cover both estimators.

Probability Limits

We now derive the probability limit of $(\hat{θ}', \hat{β}')'$ in Equation 3. Some additional notation is needed. We assume that $plim {\tilde{Σ}}_{U}^{(n)}$ exists and denote this limit by ${\tilde{Σ}}_{U}$ . Note that ${\tilde{Σ}}_{U}$ may or may not equal the true population variance–covariance matrix of the measurement errors $Σ_{U}$ . Different combinations of $Σ_{U}$ and ${\tilde{Σ}}_{U}$ cover four situations: (1) OLS regression when all covariates are measured without error $(Σ_{U} = {\tilde{Σ}}_{U} = 0)$ , (2) OLS regression with measurement error that is ignored $(Σ_{U} \neq 0, {\tilde{Σ}}_{U} = 0)$ , (3) EIV regression using an asymptotically correct error variance–covariance matrix $(Σ_{U} = {\tilde{Σ}}_{U} \neq 0)$ , and (4) EIV regression using a misspecified error variance–covariance matrix $(Σ_{U} \neq 0, {\tilde{Σ}}_{U} \neq 0, Σ_{U} \neq {\tilde{Σ}}_{U})$ .

We denote the conditional means of $H_{i}$ , $U_{i}$ , and $G_{i}$ for groups $t = 1, \dots, T$ by $η_{t}$ , $μ_{t}$ , and $γ_{t}$ , respectively. We use $η$ to denote the $(T \times P)$ matrix with row $t$ equal to ${η'}_{t}$ , and analogously use $μ$ to denote the $(T \times P)$ matrix with row $t$ equal to ${μ'}_{t}$ . We let $γ = (γ_{1}, \dots, γ_{T})'$ . We let $Π = diag (π_{1}, \dots, π_{T})$ , where $π_{t}$ is the probability that component $t$ of $T_{i}$ is equal to 1. Assuming the necessary conditions so all the limits exist, algebraic manipulations yield that:

plim [({Q'}_{n} Q_{n} - C_{n}) / n] = [\begin{matrix} Π & Π (η + μ) \\ (η + μ)' Π & Σ_{H} + (Σ_{U} - {\tilde{Σ}}_{U}) \end{matrix}],

and

plim [{Q'}_{n} Y_{n} / n] = [\begin{matrix} Π (θ + η β + λγ) \\ (η + μ)' Π θ + Σ_{H} β + λ Σ_{H G} \end{matrix}] .

Additional details on this derivation, and other derivations in this section, are provided in the Appendix. Provided the inverse of Equation 4 exists, $plim (\hat{θ}', \hat{β}')'$ is equal to the product of this inverse and Equation 5. This can be evaluated using the standard formula for the inverse of a block matrix. After some simplification, this yields that $plim (\hat{θ}', \hat{β}')'$ equals:

[\begin{matrix} (Π^{- 1} + (η + μ) S^{- 1} (η + μ)') & - (η + μ) S^{- 1} \\ - S^{- 1} (η + μ)' & S^{- 1} \end{matrix}] [\begin{matrix} Π (θ + η β + λγ) \\ (η + μ)' Π θ + Σ_{H} β + λ Σ_{H G} \end{matrix}],

where

S = Σ_{H} + (Σ_{U} - {\tilde{Σ}}_{U}) - (η + μ)' Π (η + μ) .

Further algebraic manipulation then yields:

\tilde{β} : = plim \hat{β} = S^{- 1} [Σ_{H} β + λ Σ_{H G} - (η + μ)' Π (η β + λ γ)],

\tilde{θ} : = plim \hat{θ} = θ + η (β - \tilde{β}) + λγ - μ \tilde{β} .

There are some special cases where consistency holds, that is, where $\tilde{β} = β$ and $\tilde{θ} = θ$ .

Case 1: The estimator is consistent when there is no measurement error, OLS is used, and $λ = 0$ , so that there is no omitted variable. Under these conditions, $Σ_{U} = {\tilde{Σ}}_{U} = 0$ , $μ \equiv 0$ , and $S$ reduces to $Σ_{H} - η' Π η$ . Then Equation 8 reduces to $S^{- 1} S β = β$ , and Equation 9 reduces to $θ$ .

Case 2: The estimator is consistent when there is measurement error $(Σ_{U} \neq 0)$ , EIV regression is applied with a working value ${\tilde{Σ}}_{U}^{(n)}$ satisfying $plim {\tilde{Σ}}_{U}^{(n)} = {\tilde{Σ}}_{U} = Σ_{U}$ , the required surrogacy assumptions hold so that $μ \equiv 0$ and $λ = 0$ . Under these conditions, expressions for the probability limits reduce as in Case 1.

Case 3: The estimator is consistent when there is measurement error, measurement error is ignored by using OLS (i.e., ${\tilde{Σ}}_{U}^{(n)}$ is set equal to a matrix of zeros), $E [H_{i} | {\tilde{H}}_{i}, T_{i}] = L {\tilde{H}}_{i}$ for some matrix $L$ , and $λ = 0$ . This is a special case where selection on observables holds given ${\tilde{H}}_{i}$ and where $Y_{i}$ is linear in ${\tilde{H}}_{i}$ and $T_{i}$ . Thus, the fact that ${\tilde{H}}_{i}$ measures $H_{i}$ with error is inconsequential because ${\tilde{H}}_{i}$ itself is sufficient to account for all relevant differences among treatment groups.⁴

Outside of these special cases, consistency generally will not hold. It is clear from Equations 8 and 9 that even when $λ = 0$ , the asymptotic bias in $\hat{θ}$ depends on a combination of the true model coefficients, the degree of sorting of units to treatment groups with respect to the latent variables, and the extent to which violations of surrogacy lead to $μ \neq 0$ . The asymptotic behavior can be nonintuitive when surrogacy fails to hold. For example, it is possible for $μ \neq 0$ , but $\tilde{β} = β$ . This would occur if the groups were balanced on the observed, error-prone variable but not the latent variable, that is, the group means for ${\tilde{H}}_{i}$ all equal 0 but the group means for $H_{i}$ do not. In such cases, it is possible to have no omitted variables, correctly estimate the coefficients for all covariates, yet still have $\tilde{θ} \neq θ$ due to the term $μ \tilde{β}$ in Equation 9.

Asymptotic Bias in Group Effects

In this study, we are primarily interested in the bias in $\hat{θ}$ because this translates directly into bias in estimated treatment effects. A scalar summary of the magnitude of the asymptotic bias of $\hat{θ}$ across the ensemble of groups can be constructed as follows. From Equation 9, the asymptotic bias of $\hat{θ}$ is $b = η (β - \tilde{β}) + λγ - μ \tilde{β}$ , which can be expressed as the product of the $(T \times (2 P + 1))$ matrix $M = [μ, η, γ]$ and the vector $a$ of length $(2 P + 1)$ given by:

a = (- \tilde{β}', β' - \tilde{β}', λ)' .

Thus, the mean of the squared asymptotic bias is

b' Π b = a' M' Π M a = a' Σ_{U H G}^{(b)} a,

where $Σ_{U H G}^{(b)}$ is the $(2 P + 1) \times (2 P + 1)$ “between-group” variance–covariance matrix of $({U'}_{i}, {H'}_{i}, G_{i})'$ under a probability distribution for the groups $T_{i}$ with probabilities $π$ . The matrix $Σ_{U H G}^{(b)}$ can be expressed as:

Σ_{U H G}^{(b)} = [\begin{matrix} μ' Π μ & μ' Π η & μ' Π γ \\ η^{'} Π μ & η' Π η & η' Π γ \\ γ^{'} Π μ & γ^{'} Π η & γ^{'} Π γ \end{matrix}],

which follows from the assumption that the marginal means of $U_{i}$ , $H_{i}$ , and $G_{i}$ are each a conforming $0$ , implying that $π' μ$ , $π' η$ , and $π' γ$ are each a conforming $0$ .

Not all of the elements of the matrix $Σ_{U H G}^{(b)}$ in Equation 12 can be identified from the observed data without additional assumptions. For example, the elements corresponding to $U_{i}$ and $H_{i}$ are not identified without additional assumptions about how ${\tilde{H}}_{i}$ was used to assign units to groups. However, $Σ_{U H G}^{(b)}$ is subject to three constraints without additional assumptions. First, $Σ_{U H G}^{(b)}$ must be positive semidefinite. Second, $Σ_{U H G} - Σ_{U H G}^{(b)}$ , the “within-group” variance–covariance matrix of $({U'}_{i}, {H'}_{i}, G_{i})'$ , must be positive semidefinite. The third constraint follows from the fact that ${\tilde{H}}_{i} = H_{i} + U_{i}$ . Specifically, define $A$ to be the $(P \times 2 P)$ matrix $[I_{P}, I_{P}]$ , so that ${\tilde{H}}_{i} = A ({H'}_{i}, {U'}_{i})'$ . Denote the between-group variance–covariance matrix of ${\tilde{H}}_{i}$ by $Σ_{\tilde{H}}^{(b)}$ , and denote the between-group variance–covariance matrix of $({H'}_{i}, {U'}_{i})'$ by $Σ_{U H}^{(b)}$ , the $(2 P \times 2 P)$ upper left block of $Σ_{U H G}^{(b)}$ in Equation 12. Then, $Σ_{U H}^{(b)}$ and $Σ_{\tilde{H}}^{(b)}$ are related by the constraint that $A Σ_{U H}^{(b)} A' = Σ_{\tilde{H}}^{(b)}$ .

In the following section, using a case study of estimating teacher effects on student achievement, we demonstrate how these constraints can be used to explore the potential magnitude of bias in estimates of treatment effects obtained using either OLS or EIV ANCOVA models.

Application to Exploring Bias in Teacher Value-Added

As noted previously, variants of ANCOVA are commonly used to estimate the effects of individual teachers on student achievement (e.g., Clotfelter, Ladd, & Vigdor, 2006; Goldhaber & Hansen, 2010; Goldhaber, Quince, & Theobald, 2018; Harris & Sass, 2011; Isenberg et al., 2013; Sass, Hannaway, Xu, Figlio, & Feng, 2012). For example, a common approach to estimating teacher effects uses ANCOVA where the outcome variable is a student’s standardized test score at the end of a given school year, the “treatment” is the teacher who taught the student that year, and the covariates include test scores from prior school years as well as other student background variables such as race or ethnicity, gender, free or reduced-price meal eligibility, special education status, or indicators for participation in other programs (McCaffrey et al., 2004).

Test scores measure student achievement with error, and the literature on estimating teacher effects contains conflicting recommendations about whether to account for this measurement error when using prior test scores as adjustment variables in ANCOVA models. Current achievement is likely to depend on students’ prior achievement, not students’ prior test scores, which motivates adjusting for measurement error in prior test scores using EIV regression or related methods (Andrabi, Das, Khwaja, & Zajonc, 2011; Lockwood & McCaffrey, 2014). On the other hand, prior test scores may be among the variables used to assign students to current-year teachers. This would violate the surrogacy assumptions required for consistency of the EIV regression estimator and could cause estimators that ignore measurement error, such as OLS regression, to have smaller bias than methods that try to account for it (Sengewald, Steiner, & Pohl, 2019; Steiner & Kim, 2016). Lockwood and McCaffrey (2019) develop empirical tests about what information is used to assign students to teachers and find evidence that assignments are influenced by both prior test scores and unobserved variables related to prior achievement. Such assignment mechanisms are likely to violate the assumptions required for consistent estimation of teacher effects using either EIV regression or OLS regression. However, because these methods are straightforward to apply using standard software, they are likely to remain common in research and policy applications of teacher value-added estimation. Thus, finding ways to facilitate the choice among these estimators in settings where both may be biased is valuable.

In this section, we use the analytical results derived in the previous section to explore the average magnitude of the asymptotic bias in teacher effects estimated using different ANCOVA models applied to data from a large urban school district. We first consider whether EIV or OLS regression appears to be preferred, that is, which method tends to yield smaller asymptotic bias, when there are multiple prior years of test score data available for adjustment. We then consider whether the preference changes as the data available for adjustment become more limited. Specifically, both research and policy applications demand the ability to estimate teacher effects for elementary-grade teachers (e.g., at least teachers in Grades 4 and 5) where there may be test scores from only a single prior year available for adjustment. Guarino, Reckase, Stacy, and Wooldridge (2013) propose modeling with a single prior year, but other authors (e.g., Lockwood & McCaffrey, 2014; Rothstein, 2009) suggest more prior years are beneficial for reducing bias. Practitioners are likely to use whatever data are readily available to estimate teacher effects, even if only a single prior-year score is available, so it is useful to provide guidance about the choice among estimators under different assumptions about what data are available for adjustment.

Data

Our analyses use longitudinal test score data from a large urban school system. The data are from groups of students who were in one of three grades (Grades 6, 7, and 8) and in one of four school years (2008–2009 to 2011–2012) for a total of 12 grade/year groups. Each group of students has math and English language arts (ELA) test score data from its corresponding school year (the “target year”) and three previous school years (the “Lag-1,” “Lag-2,” and “Lag-3” years). For example, for students in the grade/year group corresponding to Grade 6 in 2008–2009, the target year is the 2008–2009 school year, the Lag-1 year is the 2007–2008 school year when the students were in Grade 5, the Lag-2 year is the 2006–2007 school year when the students were in Grade 4, and the Lag-3 year is the 2005–2006 school year when the students were in Grade 3. Test scores are linked to students across years by a unique student identifier.

For each student, the data also contain background characteristics consisting of gender, race/ethnicity, home language, English language learner status, special education status, disability status, free and reduced-price lunch status, and an indicator of excessive absence ( $\geq 10 %$ of school year). Gender, race/ethnicity, and home language are time-invariant in our data, whereas the remaining background variables can vary across years within students. Finally, the students are linked to their schools and teachers of core math courses in the target year. Our analyses consider estimating the effects of math teachers on math achievement in the target year.

Table 1 summarizes the data for the 12 grade/year groups, providing the numbers of students, teachers, and schools⁵ as well as the reliability (coefficient $α$ ; Cronbach, 1951) of each of the math and ELA tests from the Lag-1, Lag-2, and Lag-3 years. These values were obtained from the technical reports for the state’s testing program and will be used to compute EIV regression estimators of teacher effects, discussed later in this section.

Structural Model for Teacher Effects

We conducted analyses separately for each of the 12 grade/year groups. For each grade/year group, we follow Rothstein (2009) and Lockwood and McCaffrey (2019) by focusing on estimating the effects of math teachers within schools, that is, comparing the effects of math teachers in the target grade and year relative to other math teachers in the same school, grade, and year.⁶ Let $i = 1, \dots, N$ index students in a given grade/year group. For each student $i$ , let $Y_{i}$ denote the math test score in the target year, and let $W_{i}$ denote the vector of length 6 consisting of math and ELA test scores for the Lag-1, Lag-2, and Lag-3 years. We assume that $W_{i}$ is an error-prone measure of a corresponding vector of unobserved prior achievement attributes $X_{i}$ . Let $Z_{i}$ be the vector of other background variables which are assumed to be measured without error. To reduce the dimension of the space of unknown parameters in our explorations, we used $Z_{i}$ to create a scalar composite $Z_{i}$ equal to a linear combination of all of the time-invariant student background variables and the time-varying background variables from the target and Lag-1 years in $Z_{i}$ , standardized to have mean 0 and variance 1, and oriented such that it is positively correlated with achievement. The weights in the linear combination were taken as the estimated coefficients on the components of $Z_{i}$ in a OLS regression of $Y_{i}$ on $Z_{i}$ , $W_{i}$ , and teacher dummy variables. The replacement of $Z_{i}$ by $Z_{i}$ is likely to have minimal consequence on the substantive results because in the regression for current year scores, $Z_{i}$ has extremely small incremental $R^{2}$ (less than .01) conditional on prior-year test scores.

Let $H_{i} = ({X'}_{i}, Z_{i})'$ , a vector of length $P = 7$ , and let ${\tilde{H}}_{i} = ({W'}_{i}, Z_{i})'$ be the corresponding vector of observed variables. Assume that ${\tilde{H}}_{i} = H_{i} + U_{i}$ where $U_{i}$ is a vector of mean-zero measurement errors with fixed values of zero for the final component corresponding to $Z_{i}$ . Let $T_{i}$ be a vector of teacher dummy variables, equal to 1 for the student’s target-year math teacher and 0 otherwise, and let $D_{i}$ be a vector of school dummy variables indicating the student’s target-year school.⁷ We then assume the following linear structural model for the student outcomes, similar to models commonly used in applications of estimating teacher effects on student achievement:

Y_{i} = θ' T_{i} + δ' D_{i} + β' H_{i} + ε_{i} .

We assume that $ε_{i}$ are IID with mean 0, finite variance, and uncorrelated with the remaining terms in the model. Note that the model assumes that current achievement $Y_{i}$ depends on the latent prior achievement attributes $X_{i}$ because $H_{i} = ({X'}_{i}, {Z'}_{i})'$ . Thus, outside of special cases, OLS regression in which the error-prone measures $W_{i}$ are used in place of $X_{i}$ (i.e., where ${\tilde{H}}_{i}$ is plugged in for $H_{i}$ ) will lead to biased estimators of the teacher parameters $θ$ . Alternatively, EIV regression that accounts for the error in ${\tilde{H}}_{i}$ as a measure of $H_{i}$ using the test reliabilities will lead to bias if test scores contributed to the target-year teacher assignments, violating surrogacy. Lockwood and McCaffrey (2019) show that the assumptions required for either OLS or EIV to yield consistent estimation for the model in Equation 13 do not hold for these data. Thus, it is valuable to evaluate which method might yield smaller bias. In the following sections, we use the data and asymptotic results to explore the possible magnitudes of the biases for OLS and EIV regression.

Our analyses assume that the model in Equation 13 is properly specified, so that there is no relevant omitted variable $G_{i}$ , as appears in Model 1. That is, we assume that in a hypothetical situation in which $H_{i}$ of dimension $P = 7$ , consisting of $Z_{i}$ as well as three prior years of math and ELA latent achievement attributes $X_{i}$ , were observed for each student without measurement error, then the parameters of Model 13 could be estimated consistently using OLS regression with $H_{i}$ . In practice, ${\tilde{H}}_{i}$ rather than $H_{i}$ is observed, and either OLS or EIV regression using ${\tilde{H}}_{i}$ may lead to biased estimates of teacher effects even when three prior years of test scores are available. We explore the potential magnitudes of these biases in this section. We also explore the potential magnitudes of the biases that may arise with either OLS or EIV regression when some of the covariates are omitted from the model, as would be the case if only a single year of prior data were available or if only math scores were used as covariates. In those cases, $G_{i}$ is determined by the subset of the seven covariates that we omit when fitting the model. For instance, if we omit ELA scores, then $G_{i}$ will equal a linear combination of the three prior years of ELA achievement.

Procedures

In this section, we describe the procedures that we used to explore the average asymptotic bias in teacher effects estimated with either OLS or EIV regression. Separately for each of the 12 grade/year groups, we executed the steps below using code written for the R environment (R Core Team, 2019), available from the authors by request.

Step 1

We set $Σ_{U}$ to be a $(7 \times 7)$ diagonal matrix of measurement error variances, computed by multiplying one minus the reliabilities in Table 1 by the marginal variance of each component of $W_{i}$ , and a fixed zero for the diagonal element corresponding to $Z_{i}$ . We further assume that the test measurement errors have mean 0 in each school, so that $Σ_{U}$ is the variance–covariance matrix of measurement errors in each school. This assumption is based on the fact that the vast majority of students in our data attended neighborhood schools, so that family housing choices largely determine school assignments. For this process to create nonzero school means of measurement errors of test scores used as covariates, it would be necessary for a nonnegligible number of individual families to make housing decisions on the basis of their own children’s prior test scores in such a way that the group of students who ended up in school in a given year would have measurement errors of their prior test scores with a nonzero mean. It seems implausible that selection of this form would be widespread enough to create more than negligible differences in measurement error means by school. This is in contrast to teacher-within-school assignments, where the decisions of one or a small number of people can simultaneously affect the assignments of many students and where there are substantive reasons to believe that prior test scores of individual students may influence the assignment decisions.

Table 1.

Summary of Student, Teacher and School Counts, as well as Reliabilities of Prior Test Scores, for the 12 Grade/Year Groups Used in the Empirical Explorations.^a

School Year	Grade	# Students	# Teachers	# Schools	Reliabilities
School Year	Grade	# Students	# Teachers	# Schools	M1	M2	M3	E1	E2	E3
2008–2009	6	37,188	1,333	360	.90	.94	.89	.84	.89	.85
2008–2009	7	39,086	1,189	304	.92	.90	.94	.88	.84	.88
2008–2009	8	43,046	1,183	300	.90	.91	.90	.88	.88	.82
2009–2010	6	44,333	1,424	370	.90	.94	.88	.83	.90	.86
2009–2010	7	44,288	1,289	315	.91	.90	.94	.86	.84	.89
2009–2010	8	45,655	1,252	322	.90	.92	.90	.88	.88	.84
2010–2011	6	46,033	1,484	396	.90	.94	.89	.83	.88	.86
2010–2011	7	44,534	1,272	333	.90	.90	.94	.87	.83	.90
2010–2011	8	45,023	1,226	311	.90	.91	.90	.88	.86	.84
2011–2012	6	26,583	751	221	.91	.94	.88	.90	.86	.86
2011–2012	7	25,663	693	205	.93	.90	.94	.92	.83	.88
2011–2012	8	25,317	620	176	.92	.90	.90	.92	.87	.83

^a Reliabilities for the math tests from the Lag-1, Lag-2, and Lag-3 years are labeled “M1,” “M2,” and “M3,” respectively. “E1,” “E2,” and “E3” denote the analogous reliabilities for the English language arts tests.

The matrix $Σ_{U}$ is treated as the known, true measurement error variance–covariance matrix given the large student samples available, and the working measurement error variance–covariance matrix ${\tilde{Σ}}_{U}^{(n)}$ for EIV regression is set equal to $Σ_{U}$ .

Step 2

We estimate $β$ in Model 13 using EIV regression. The resulting estimated value of $β$ , which we denote by $β^{*}$ , is used as the working value of $β$ in subsequent evaluations of average squared bias for the group effects. Of course, the average squared bias for the group effects depends on the unknown $β$ . We show that substantive findings are not sensitive to using $β^{*}$ obtained from OLS rather than EIV regression.

Step 3

We use the data to compute three variance–covariance matrices: (a) ${\hat{Σ}}_{\tilde{H}}$ , an estimate of the within-school variance–covariance matrix $Σ_{\tilde{H}}$ of ${\tilde{H}}_{i}$ ; (b) ${\hat{Σ}}_{\tilde{H}}^{(b)}$ , an estimate of the within-school, between-teacher variance–covariance matrix $Σ_{\tilde{H}}^{(b)}$ of ${\tilde{H}}_{i}$ ; and (c) ${\hat{Σ}}_{U H}$ , an estimate of the within-school variance–covariance matrix $Σ_{U H}$ of $({U'}_{i}, {H'}_{i})'$ . The matrix ${\hat{Σ}}_{U H}$ has upper left block $Σ_{U}$ from Step 1 corresponding to $U_{i}$ , lower-right block ${\hat{Σ}}_{\tilde{H}}$ − $Σ_{U}$ corresponding to $H_{i}$ , and a block of zeros corresponding to the covariances between elements of $U_{i}$ and $H_{i}$ . The matrices ${\hat{Σ}}_{\tilde{H}}^{(b)}$ and ${\hat{Σ}}_{U H}$ computed in this step are used in Step 4 to create bounds on unknown variance–covariance parameters and also in Step 5 to evaluate asymptotic bias.

Step 4

We generate feasible values of $Σ_{U H}^{(b)}$ , the within-school, between-teacher variance–covariance matrix of $({U'}_{i}, {H'}_{i})'$ , using empirical versions of the three constraints on $Σ_{U H}^{(b)}$ noted previously. Those constraints are (1) $Σ_{U H}^{(b)}$ is positive semidefinite, (2) $(Σ_{U H} - Σ_{U H}^{(b)})$ is positive semidefinite, and (3) $A Σ_{U H}^{(b)} A' = Σ_{\tilde{H}}^{(b)}$ . We apply empirical versions of these constraints where the estimates ${\hat{Σ}}_{\tilde{H}}^{(b)}$ and ${\hat{Σ}}_{U H}$ from Step 3 are plugged in place of their corresponding unknown parameters, so that a potential value of $Σ_{U H}^{(b)}$ is considered to be feasible if (1) $Σ_{U H}^{(b)}$ is positive semidefinite, (2) ( ${\hat{Σ}}_{U H} - Σ_{U H}^{(b)}$ ) is positive semidefinite, and (3) $A Σ_{U H}^{(b)} A' = {\hat{Σ}}_{\tilde{H}}^{(b)}$ . An important consideration in this step is that in general, $Σ_{U H}^{(b)}$ contains $P (2 P + 1)$ unknown parameters subject to multiple linear and nonlinear constraints. With $P = 7$ , this equals 105 parameters. It is slightly lower in our case because $Z_{i}$ is assumed to have no measurement error but is still too large to feasibly explore or visualize without additional restrictions. To make the exploration of feasible values of $Σ_{U H}^{(b)}$ manageable, we placed restrictions on the parameters of the matrix. We considered three different sets of restrictions each corresponding to a different set of assumptions about how prior scores contributed to student assignments to teachers and the correlations of teacher-level means of $H_{i}$ and $U_{i}$ . By making these restrictions, the $Σ_{U H}^{(b)}$ matrices depend on two or three parameters and we then conducted a grid search of these parameters to explore potential values for $Σ_{U H}^{(b)}$ . Details are provided in the Appendix.

Step 5

For each feasible $Σ_{U H}^{(b)}$ , we compute the average asymptotic squared bias for the adjusted teacher means estimated with OLS, and estimated with EIV, applying the formulas derived earlier. These formulas depend on $Σ_{U H}^{(b)}$ , the estimated matrices from Step 3 and $β^{*}$ from Step 2. We then repeated these calculations using selected subsets of the covariates to address the issue of how bias for these estimators changes as fewer prior years of data are available for modeling. In models where fewer than the full set of $P = 7$ variables are used, the remaining variables combine to form the omitted variable $G_{i}$ defined in the analytical derivations. Specifically, the components of $β^{*}$ corresponding to subsets of the $P = 7$ variables that are omitted define a linear combination of omitted variables that can be treated as $G_{i}$ with coefficient $λ = 1$ .

Results

Per Step 2 above, for each of the 12 grade/year groups, the working value $β^{*}$ of dimension $P = 7$ obtained using EIV regression was used in place of the unknown true parameter $β$ . Figure 1 presents the values of $β^{*}$ for each of the 12 grade/year groups, showing that most of the coefficients are positive. The coefficients on prior math achievement decay with further lags but are positive even at Lag 3. The corresponding coefficients for prior ELA achievement also tend to be positive at Lags 1 and 2. Thus, there is the potential for omitted variable bias when some of the variables are excluded from the model, making our comparisons between OLS and EIV regression meaningful as we consider restricting the amount of data available for the adjustment in the ANCOVA model.

Figure 1.

Estimated standardized coefficients on lagged achievement and the composite background variable from errors-in-variables (EIV) regression, for each of the 12 grade/year groups. Estimates from the same group are connected by line segments.

Figure 2 summarizes the average squared asymptotic bias for the adjusted teacher means estimated using OLS and EIV regression, for each feasible value of $Σ_{U H}^{(b)}$ and configuration of the $P = 7$ adjustment variables used in the estimation. On the vertical axis in each frame is the percentage of feasible scenarios in which the average squared bias for the teacher effects is larger for OLS than for EIV. Specifically, the denominator of the percentage is the number of feasible values of $Σ_{U H}^{(b)}$ that we considered across the three selection scenarios and grids of unknown parameters noted previously (with details in the Appendix), and the numerator is the number of these feasible scenarios in which the average squared asymptotic bias for OLS regression exceeds that of EIV regression. The upper-left frame presents results for models that use one, two, or three lags of only math scores, ignoring both ELA scores and $Z_{i}$ in the model. The upper-right frame is for models that use one, two, or three lags of math scores as well as $Z_{i}$ . The lower-left frame is for models that use one, two, or three lags of both math and ELA scores, not using $Z_{i}$ . Finally, the lower-right frame is for models that use one, two, or three lags of both math and ELA scores as well as $Z_{i}$ . Each frame has 12 triplets of points connected by a line, one for each of the 12 grade/year groups. This configuration of models where either $Z_{i}$ , ELA scores, or both are excluded was motivated by our experiences working with states and school districts to specify growth models, where depending on the purposes and intended audience, there may be substantive motivations to exclude either or both of these types of adjustment variables.

Figure 2.

Summary of percentages of feasible scenarios where the average asymptotic bias in group effects estimated by ordinary least squares exceeds that for errors in variable (EIV). The four frames indicate different configurations of which adjustment variables are included in the models, and within each frame, results are presented for models that use only Lag-1 data, Lag-1 and Lag-2 data, and data from all three lags. Each frame presents results for all 12 grade/year groups, with the results for the same group connected by line segments.

The results show that feasible values of the unknown parameters contain cases where EIV would be preferred and cases where OLS would be preferred. EIV is preferred over more of the feasible space, particularly when only 1 year of prior data are used, but the results are mixed. The choice between OLS and EIV becomes less clear when more variables are included in the model. One possible explanation for this finding is that including additional covariates may reduce omitted variable bias⁸ and the potential impact of uncontrolled measurement error, in which case the potential benefits of EIV over OLS are reduced, but the risks of using EIV when surrogacy fails remain. The basic patterns in Figure 2 are replicated using an alternative value of $β^{*}$ obtained using OLS regression rather than EIV regression (Appendix Figure A1). Thus, the findings that OLS tends to have larger asymptotic bias than EIV over more of the feasible space of unknown parameters is not a consequence of using EIV to compute $β^{*}$ .

The results to this point favor EIV, but only modestly. It is thus useful to consider ways of reducing the space of unknown parameters, with the hope that comparisons among different estimators may be less ambiguous over a more restricted space. To develop a way to restrict the space, we begin with the observation that many of the feasible scenarios consist of cases where $plim \hat{β}$ from EIV regression differs substantially from $β^{*}$ . That is, there are many feasible scenarios where the surrogacy violations are severe enough to cause the probability limit of $\hat{β}$ estimated from EIV regression to deviate substantially from the assumed working value of $β^{*}$ . This arises when student–teacher assignments are strongly dependent on prior test scores, causing the average measurement error to vary substantially across teachers within schools. If there were evidence to suggest that such extreme selection mechanisms are not likely, it would provide a way to restrict the space of unknown parameters and perhaps sharpen the comparison of the bias for different estimators.

To explore this, we return to the assumption that the average measurement error in prior test scores should be approximately zero for each school. Under this assumption, prior test scores satisfy surrogacy with respect to dummy variables for school groupings, so that surrogacy violations should not be a source of bias for EIV regression that includes dummy variables for schools but excludes teacher dummy variables. We thus used EIV regression to fit a version of the model in Equation 13 that includes school dummy variables but excludes teacher dummy variables. We did this for each of the 12 grade/year groups. Figure 3 plots the estimated coefficients on the $P = 7$ variables from the models that include only school dummy variables versus the estimated coefficients from the models that include both teacher and school dummy variables. The coefficients are remarkably insensitive to whether dummy variables for teacher groupings are included or excluded from the model. The simplest explanation for these results is that the average measurement error does not vary much across teachers; if it did, the resulting surrogacy violations for teacher dummy variables should cause larger discrepancies in estimated coefficients depending on whether teacher dummies are included or excluded from the model. While other explanations for the insensitivity of coefficients to model specification are possible, they involve distinct sources of bias happening to compensate in exactly the right ways to make the coefficients from the two model specifications line up. These results suggest that values of the unknown parameters determining the bias in estimated teacher effects that would also lead to very large distortions in $\hat{β}$ from EIV regression do not appear to be likely with our data.

Figure 3.

Comparison of $β^{*}$ estimated using errors-in-variables (EIV) regression using only school dummy variables (vertical axis), versus those obtained from EIV regression using school and teacher dummy variables (horizontal axis). Each point in the figure is for one of the seven model predictors, for one of the 12 grade/year groups, for a total of 84 points.

We thus can consider how the results in Figure 2 change if we restrict to scenarios that do not involve $plim \hat{β}$ differing substantially from $β^{*}$ . This restriction is analogous to using an informative prior distribution over the space of unknown parameters, where we restrict attention to a subset of the space consisting of values that we believe to be more likely. For each of the 12 grade/year groups, let $β_{sch}^{*}$ denote the coefficients on the adjustment variables obtained using EIV regression that includes school dummy variables but excludes teacher dummy variables. Figure 4 is analogous to Figure 2 but is restricted to feasible scenarios where the difference between $plim \hat{β}$ and $β^{*}$ is about the same magnitude as we observe between $β_{sch}^{*}$ and $β^{*}$ . Specifically, we computed the median Euclidean distance between $β_{sch}^{*}$ and $β^{*}$ across the 12 grade/year groups and then restricted the feasible scenarios to those where the Euclidean distance between $plim \hat{β}$ and $β^{*}$ did not exceed this median discrepancy. Under these restrictions, the conclusions are somewhat more clearly in favor of EIV regression, particularly when only Lag-1 data are used in the adjustment. This provides some evidence that EIV may be preferred over OLS in settings where only limited prior data are available for adjustment. There is more ambiguity when more lagged data are used in the model, although EIV is still favored in a majority of scenarios.

Figure 4.

Analogous to Figure 2, but for a subset of feasible scenarios in which the difference between plim $\hat{β}$ and $β^{*}$ is restricted to be about the same magnitude as we observe between $β_{sch}^{*}$ and $β^{*}$ .

Discussion

While EIV regression, and latent variable models more generally, are popular methods for estimating causal effects with observational data, the asymptotic results make clear that assessing the magnitude and direction of biases resulting from surrogacy violations can be difficult due to dependence on a potentially large number of unidentified parameters. Although the use of the analytical results in the empirical application seemed to provide some insights regarding the choice of estimators for teacher effects, the findings are still limited by the need to reduce the number of free, unknown parameters arising from potential surrogacy violations, the need to condition on values of unknown structural parameters, and the need to decide how to aggregate results across the many feasible configurations of unknown parameters. The findings are also contingent on the specific distributions of variables and measurement errors in our data and thus may not generalize. Our application also benefited from having multiple prior years of data, which allowed us to study the impacts of explicitly defined omitted variables $G_{i}$ when teacher effects were estimated using fewer prior years of data for adjustment. In other applications of the empirical methods, it may be necessary to make assumptions about the properties of omitted variables that are not informed by the data in order to examine the trade-offs of various EIV versus OLS regression model specifications, which introduces even more unknown parameters. Also, if there are large discrepancies between estimates of the coefficients from OLS and EIV regression, then the results about bias may be sensitive to which is used to compute $β^{*}$ . In these cases, additional sensitivity analyses may be necessary to explore the impact of different values for $β^{*}$ .

In pretest–posttest design settings where the pretest and posttest are measured on the same scale, a common alternative to ANCOVA is the analysis of change or “gain scores” (e.g., Allison, 1990). A standard application of this type of analysis would regress the difference between the outcomes measured before and after the intervention on treatment group indicators and possibly other covariates. Situations in which the resulting OLS regression estimator will or will not be consistent could be assessed using developments analogous to those used here. Violations of the surrogacy assumption, such as when treatment assignments are in part determined by the error-prone pretest score, can cause treatment effect estimators from gain score models to be inconsistent.

We focused on the impacts of surrogacy violations in linear models. Recent research has considered how to adapt other common estimators of causal effects, such as weighting, matching, and doubly robust estimation, to the case of error-prone confounders (Kuroki & Pearl, 2014; Lockwood & McCaffrey, 2015, 2016; McCaffrey, Lockwood, & Setodji, 2013; Sengewald et al., 2019; Webb-Vargas, Rudolph, Lenis, Murakami, & Stuart, 2017; Yi et al., 2012). Most of these methods also rely on the assumption of surrogacy of the error-prone measures. While the analytical results derived here apply only to the linear model, it may be possible to extend the basic logic of using data to explore potential biases to these alternative estimators. Another worthwhile extension may be to linear models with multiple levels of nesting. For example, while we focused estimating teacher effects within schools, many policy applications of teacher value-added estimation compare teachers both within and among schools. Extending the analytical results to that case would require considering a three-level variance decomposition for $H_{i}$ , $U_{i}$ , and $G_{i}$ into within-teacher, between-teacher within-school, and between-school components. This case would be more complicated to evaluate than the case considered here due to both additional moment constraints and additional unknown parameters but would be worth future consideration, given its relevance to teacher evaluation policies.

This article focused on EIV regression as a method for addressing measurement error in covariates in ANCOVA models. EIV regression requires that the variance–covariance matrix of the measurement errors is known or can be estimated. There are settings in which such information may not be easily obtained. An alternative to EIV regression that is commonly applied in the case of error-prone covariates is instrumental variables (IVs) estimation (e.g., Greene, 2003; Wooldridge, 2002). IV estimation does not require knowing the variance–covariance matrix of the measurement errors and can yield consistent treatment effect estimators with error-prone covariates provided appropriate IVs are available and other assumptions hold. IV can yield consistent estimators of treatment effects under some violations of surrogacy considered in this article. The primary limitation is finding IVs that satisfy the appropriate assumptions required for consistent estimation, which generally can be difficult, and may be particularly difficult in the setting considered here where error-prone measures may directly influence treatment assignments. For example, in cases where there is concern that an error-prone measure may have been used in treatment assignments, a variable that provided an alternative measurement of the same latent variable but was known not to be used in treatment assignments could potentially meet the requirements to be an IV. However, situations in which such a variable exists and can be obtained for estimation may be rare.

The analytical results derived here cover two distinct types of surrogacy violations: one in which measurement error influences treatment assignments and one in which measurement error influences outcomes (in the Appendix). A third type of violation can occur when measurement errors are related to other covariates in the ANCOVA model. For example, there is evidence that people in different demographic groups have different proclivities to choose extreme values on survey items even when they have equal values of the latent construct(s) being measured by the survey (Weech-Maldonado, Elliott, Oluwole, Schiller, & Hays, 2008). More generally, “differential item functioning” (e.g., Holland & Wainer, 1993) exists when people with the same value of a latent construct being measured by a given item, but who are in different demographic groups, do not have the same distribution of item responses. These situations can lead to correlation between measurement errors and indicator variables for demographic groups that may be included in the ANCOVA model. Extensions to the analytical results presented here to cover this case would be straightforward. However, the methods for exploring the asymptotic bias with real data would be more tenuous because the correlations between measurement errors and other covariates introduce additional unknown parameters that affect the asymptotic bias of different estimators.

Finally, our asymptotic results and associated empirical methods focused on bias, but in applications, errors in treatment effect estimates due to variance in parameter estimates also are relevant. Thus, the choice among estimators may involve criteria such as mean squared error (MSE) for true treatment effects that combine bias and variance. Model misspecification for either OLS or EIV regression may lead to inconsistent variance estimators under some circumstances, and the bias may depend on unknown parameters just as it does for the bias in estimators of the regression model coefficients. Future work could consider analytical results for variance estimators and associated empirical methods for comparing MSE for different model specifications.

Footnotes

Appendix

Authors’ Note

The opinions expressed are those of the authors and do not represent views of ETS, the Institute of Education Sciences, or the U.S. Department of Education.

Acknowledgments

We thank James Carlson, Katherine E. Castellano, Matthew S. Johnson, the Editor, and two anonymous referees for helpful comments on earlier drafts.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported here was supported in part by the Institute of Education Sciences, U.S. Department of Education, through Grant No. R305D140032 to ETS.

ORCID iD

J. R. Lockwood

Notes

References

Allison

P. D.

(1990). Change scores as dependent variables in regression analysis. Sociological Methodology, 20, 93–114. doi:10.2307/271083

Andrabi

Das

Khwaja

Zajonc

(2011). Do value-added estimates add value? Accounting for learning dynamics. American Economic Journal: Applied Economics, 3, 29–54. doi:10.1257/app.3.3.29

Ballou

Sanders

Wright

(2004). Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics, 29, 37–66. doi:10.3102/10769986029001037

Buonaccorsi

(2010). Measurement error: Models, methods, and applications. Boca Raton, FL: Chapman and Hall/CRC Interdisciplinary Statistics.

Burkard

Mayr

Winterhalder

Leonardi

Eckstein

Vischer

A. S.

(2018). Reliability of single office blood pressure measurements. Heart, 104, 1173–1179. doi:10.1136/heartjnl-2017-312523

Carroll

Ruppert

Stefanski

Crainiceanu

(2006). Measurement error in nonlinear models: A modern perspective (2nd ed.). London, England: Chapman and Hall.

Clotfelter

C. T.

Ladd

H. F.

Vigdor

J. L.

(2006). Teacher-student matching and the assessment of teacher effectiveness. Journal of Human Resources, 41, 778–820. doi:10.3368/jhr.XLI.4.778

Crocker

Algina

(1986). Introduction to classical and modern test theory. Orlando, FL: Holt, Rinehart and Winston.

Cronbach

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. doi:10.1007/bf02310555

10.

Culpepper

Aguinis

(2011). Using analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16, 166–178. doi:10.1037/a0023355

11.

Fuller

(2006). Measurement error models (2nd ed.). New York, NY: John Wiley.

12.

Fuller

Hidiroglou

(1978). Regression estimation after correcting for attenuation. Journal of the American Statistical Association, 73, 99–104. doi:10.1080/01621459.1978.10480011

13.

Goldhaber

Hansen

(2010). Using performance on the job to inform teacher tenure decisions. American Economic Review, 100, 250–255. doi:10.1257/aer.100.2.250

14.

Goldhaber

Quince

Theobald

(2018). Has it always been this way? Tracing the evolution of teacher quality gaps in us public schools. American Educational Research Journal, 55, 171–201. doi:10.3102/0002831217733445

15.

Greene

(2003). Econometric analysis (5th ed.). Upper Saddle River, NJ: Prentice Hall.

16.

Guarino

Ham

E. H.

Reckase

Stacy

Wooldridge

. (2013). Sending teacher value-added into a tailspin: A simulation study of measurement error and nonrandom sorting. Unpublished manuscript.

17.

Harris

(2011). Value-added measures in education: What every educator needs to know. Cambridge, MA: Harvard Education Press.

18.

Harris

Sass

(2011). Teacher training, teacher quality and student achievement. Journal of Public Economics, 95, 798–812. doi:10.1016/j.jpubeco.2010.11.009

19.

Holland

Wainer

(1993). Differential item functioning. Hillsdale, NJ: Lawrence Erlbaum.

20.

Imbens

G. W.

(2004). Nonparametric estimation of average treatment effects under exogeneity: A review. The Review of Economics and Statistics, 86, 4–29. doi:10.1162/003465304323023651

21.

Imbens

G. W.

Rubin

D. B.

(2015). Causal inference in statistics, social, and biomedical sciences. Cambridge, England: Cambridge University Press.

22.

Isenberg

Max

Gleason

Potamites

Santillano

Hock

Hansen

. (2013). Access to effective teaching for disadvantaged students (Technical Report). Washington, DC: Institute of Education Sciences, U.S. Department of Education, NCEE 2014-4001. Retrieved from https://ies.ed.gov/ncee/pubs/20144001/pdf/20144001.pdf

23.

Johnson

M. T.

Lipscomb

Gill

(2015). Sensitivity of teacher value-added estimates to student and peer control variables. Journal of Research on Educational Effectiveness, 8, 60–83. doi:10.1080/19345747.2014.967898

24.

Kuroki

Pearl

(2014). Measurement bias and effect restoration in causal inference. Biometrika, 101, 423–437. doi:10.1093/biomet/ast066

25.

Lockwood

J. R.

McCaffrey

D. F.

(2014). Correcting for test score measurement error in ANCOVA models for estimating treatment effects. Journal of Educational and Behavioral Statistics, 39, 22–52. doi:10.3102/1076998613509405

26.

Lockwood

J. R.

McCaffrey

D. F.

(2015). Simulation-extrapolation for estimating means and causal effects with mismeasured covariates. Observational Studies, 1, 241–290.

27.

Lockwood

J. R.

McCaffrey

D. F.

(2016). Matching and weighting with functions of error-prone covariates for causal inference. Journal of the American Statistical Association, 111, 1831–1839. doi:10.1080/01621459.2015.1122601

28.

Lockwood

J. R.

McCaffrey

D. F.

(2017). Simulation-extrapolation with latent heteroskedastic error variance. Psychometrika, 82, 717–736. doi:10.1007/s11336-017-9556-y

29.

Lockwood

J. R.

McCaffrey

D. F

. (2019). Using hidden information and performance-level boundaries to study student-teacher assignments: Implications for estimating teacher causal effects. Unpublished manuscript.

30.

Lord

(1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum Associates.

31.

McCaffrey

Lockwood

Koretz

Louis

Hamilton

(2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29, 67–101. doi:10.3102/10769986029001067

32.

McCaffrey

Lockwood

Setodji

(2013). Inverse probability weighting with error-prone covariates. Biometrika, 100, 671–680. doi:10.1093/biomet/ast022

33.

Papay

Murnane

Willett

. (2016). The impact of test score labels on human-capital investment decisions. Journal of Human Resources, 51, 357–388. doi:10.3368/jhr.51.2.0713-5837R

34.

Pearl

Glymour

Jewell

N. P.

(2016). Causal inference in statistics: A primer. Hoboken, NJ: Wiley.

35.

R Core Team. (2019). R: A language and environment for statistical computing [Computer software manual] . Vienna, Austria. Retrieved from https://www.R-project.org/

36.

Rosenbaum

Rubin

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55. doi:10.1093/biomet/70.1.41

37.

Rothstein

(2009). Student sorting and bias in value-added estimation: Selection on observables and unobservables. Education Finance and Policy, 4, 537–571. doi:10.1162/edfp.2009.4.4.537

38.

Sass

T. R.

Hannaway

Figlio

D. N.

Feng

(2012). Value added of teachers in high-poverty schools and lower poverty schools. Journal of Urban Economics, 72, 104–122. doi:10.1037/e721792011-001

39.

Scheffe

(1959). The analysis of variance. New York, NY: Wiley.

40.

Sengewald

M.-A.

Pohl

(2019). Compensation and amplification of attenuation bias in causal effect estimates. Psychometrika, 84, 589–610. doi:10.1007/s11336-019-09665-6

41.

Sengewald

M.-A.

Steiner

Pohl

(2019). When does measurement error in covariates impact causal effect estimates? Analytical derivations of different scenarios and an empirical illustration. British Journal of Mathematical and Statistical Psychology, 72, 244–270. doi:10.1111/bmsp.12146

42.

Shadish

Cook

Campbell

(2002). Experimental and quasi-experimental designs for generalized causal inference. New York, NY: Houghton Mifflin.

43.

Steiner

Cook

Shadish

(2011). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics, 36, 213–236. doi:10.3102/1076998610375835

44.

Steiner

Kim

. (2016). The mechanics of omitted variable bias: Bias amplification and cancellation of offsetting biases. Journal of Causal Inference, 4, 20160009. doi:10.1515/jci-2016-0009

45.

Webb-Vargas

Rudolph

Lenis

Murakami

Stuart

(2017). An imputation-based solution to using mismeasured covariates in propensity score analysis. Statistical Methods in Medical Research, 26, 1824–1837. doi:10.1177/0962280215588771

46.

Weech-Maldonado

Elliott

M. N.

Oluwole

Schiller

K. C.

Hays

R. D.

(2008). Survey response style and differential use of CAHPS rating scales by hispanics. Medical Care, 46, 963. doi:10.1097/mlr.0b013e3181791924

47.

Wooldridge

(2002). Econometric analysis of cross section and panel data. Cambridge: MIT Press.

48.

Yang

Tsiatis

(2001). Efficiency study of estimators for a treatment effect in a pretest-posttest trial. The American Statistician, 55, 314–321. doi:10.1198/000313001753272466

49.

Carroll

(2012). A functional generalized method of moments approach for longitudinal studies with missing responses and covariate measurement error. Biometrika, 99, 151–165. doi:10.1093/biomet/asr076