Abstract
The AERA Statement on the use of value-added models (VAM) is very welcome (American Educational Research Association, 2015) but only alludes to the principal problem with them—misspecification. The Statement argues that to “isolate the contributions of teachers and leaders to student learning” is “very difficult.” It is not—it is impossible—even if all the technical requirements in the Statement are met. For proper specification of any form of regression analysis, three conditions must hold: All confounding variables must be in the equation, all must be measured correctly, and the correct functional form must be used.
As the 40-year literature on input-output functions that use student test scores as the dependent variable make clear, we never even come close to meeting these conditions. Literally, dozens, perhaps hundreds, of variables may be relevant to explaining student gain score variations; we have multiple, competing measures for most; and we have no idea of the proper functional interrelationships (see Klees, 2016). Such studies always find significant variables and residuals can easily be used to rank order teacher and principal “contributions.” But adding relevant variables to the model, changing how you measure them, or using alternative functional forms will always yield significant differences in the rank ordering of teachers’ and principals’ contributions.
I would argue that with any VAM process that made its data available to competent researchers, those researchers would find that reasonable alternative specifications would yield major differences in rank ordering. Misclassification is not simply a “significant risk”—major misclassification is rampant and inherent in the use of VAM. The bottom line is that regardless of technical sophistication, the use of VAM is never “accurate, reliable, and valid” and will never yield “rigorously supported inferences.”