Abstract
Until recently, legal challenges to the use of value-added models (VAMs) in evaluation and teacher employment decisions in federal court had been unsuccessful. However, in May 2017 a federal court in Texas ruled that plaintiff-teachers established a viable federal constitutional claim to challenge the use of VAMs as a means for their termination in Houston Federation of Teachers v. Houston Independent School District. Houston represents a significant departure from prior federal court rulings that upheld the constitutionality of VAMs to terminate teachers on the basis of poor performance. The Houston court found that the districts’ refusals to release the underlying data of VAM ratings used to terminate those teachers violated the teachers’ procedural due process rights. By denying access to the code, teachers could not protect against the government’s making a mistaken deprivation of their property right to continued right to employment. The authors discuss Houston and its potential impact, limitations, and significance.
Keywords
Over the past decade, numerous education policymakers (Duncan, 2009, 2011; Rhee, 2011; U.S. Department of Education, 2006a, 2006b, 2014) believed that they had found a teacher reform measure to help solve the United States’ seemingly intractable problem of improving teacher quality. The idea was this: statistically link a teacher’s contributions to students’ growth on standardized tests, predominantly in mathematics and English/language arts, and hold teachers accountable for their students’ collective growth over time or lack thereof. Teachers deemed effective would be rewarded, and those deemed ineffective would be incentivized to improve their professional practice or leave the classroom, through termination, tenure, or contract nonrenewal. Although observations of teacher practice in classrooms were still to be used to evaluate teachers, despite criticisms about said observational measures that helped spur these reforms at the same time (Weisberg, Sexton, Mulhern, & Keeling, 2009; see also Kraft & Gilmour, 2017), the linchpin of these reforms was the introduction and use of value-added models (VAMs) or growth models (hereafter referred to more generally as VAMs 1 ). VAMs were to help more objectively hold teachers accountable in the name of educational reform.
VAMs are sophisticated statistical models that estimate teachers’ impact on their students’ growth in achievement over time while controlling for many outside variables that also influence student achievement. In 2009 the federal government effectively required states to adopt and use VAMs as a condition to receive $4.35 billion in Race to the Top funds (Race to the Top Act, 2011). In addition, states that implemented teacher evaluation systems that included a VAM component were eligible to receive waivers, excusing them from not meeting the prior requirement of the No Child Left Behind Act (2002) that all students within their states would be proficient in mathematics and English/language arts by 2014 (U.S. Department of Education, 2014). Because of these federal efforts, school districts in almost all states (and the District of Columbia) implemented teacher evaluation systems that used VAMs as primary components of their systems, often with high-stakes consequences attached (e.g., merit pay, tenure, and contract nonrenewal).
The Houston Independent School District (HISD)—the seventh largest district in the country and the largest and “worst” performing school district in Texas (Leicht, 2014)—was at the forefront of adopting VAMs for such high-stakes employment decisions, even before the aforementioned federal initiatives. The district aggressively used a proprietary VAM, the Education Value-Added Assessment System (EVAAS), 2 upon the arrival of a new superintendent in 2007. Four years later, in 2011, HISD declined to renew 221 teachers’ contracts because they demonstrated “a significant lack of student progress attributable to the educator” or “insufficient student academic growth reflected by [their EVAAS] value-added scores” (Houston Independent School District, 2011). HISD continued to terminate and make other meaningful employment decisions about teachers (e.g., merit pay) until 2016 (Z. Capo, personal communication, June 2, 2019; see also Corcoran, 2010; Harris, 2011; Houston Independent School District, 2015, 2016).
Throughout this period of time (2007–2016), many Houston teachers objected to the use of VAMs for such important employment decisions, echoing concerns documented in the literature (Amrein-Beardsley & Collins, 2012; Collins, 2014; Kappler Hewitt, 2015). Concerns, for example, matched those cited by the American Statistical Association (2014), which cautioned against the use of VAMs and, rather, encouraged the “wise use” of VAM estimates (p. 1), especially in instances in which someone’s livelihood could be at risk. Teachers’ concerns also matched those expressed by the American Educational Research Association (AERA), which wrote that VAMs should “not be used without sufficient evidence that [a high] technical bar has been [set and] met in ways that support all claims, interpretative arguments, and [especially high-stakes] uses” (American Educational Research Association Council, 2015, p. 4). Others around this same period of time prospectively warned of looming legal issues, especially when consequential decisions were to be attached to VAM output (Baker, Oluwole, & Green, 2013; Pullin, 2013; Superfine, 2016).
Ultimately, many teachers in Houston and elsewhere sought judicial remedy to stop the use of VAMs on the basis of various legal theories. 3 In fact, at least 15 lawsuits had been filed by 2015 (“Teacher Evaluation Heads to the Courts,” 2015) across federal and state courts. In federal courts, which potentially have a multistate jurisdictional reach on an issue (as opposed to state courts, whose jurisdictional reach is limited to defining the law in particular states), teachers brought claims under the Fourteenth Amendment to the Constitution. Yet until Houston Federation of Teachers v. Houston Independent School District (2017), federal constitutional claims protesting the use of VAMs for termination were generally rejected. For example, the Eleventh Circuit Court of Appeals (which has jurisdiction over federal matters in Alabama, Florida, and Georgia) in Cook v. Bennett (2011) dismissed substantive due process and equal protection claims brought by teachers. Other federal courts in jurisdictions outside of the Eleventh Circuit have generally followed Cook since (see, e.g., Trout v. Knox County Board of Education, 2016; Wagnar v. Haslam, 2015) and dismissed teachers’ constitutional claims. Taken together, these decisions appeared to foreclose federal courts as a venue for teachers seeking to advance constitutional claims to overturn termination decisions on the basis of the use of VAMs. 4
The Houston case, decided on May 4, 2017, however, reopened the door to the federal courts. Indeed, the Houston court agreed that HISD’s refusal to release the coding data underlying VAM ratings used to terminate teachers violated the teachers’ procedural due process rights under the Constitution. Although the perceived flexibility in the Every Student Succeeds Act 5 (ESSA) may have reduced federal and state attention toward VAMs, it is worth noting that many states and school districts continue to use VAMs (including the EVAAS®) for high-stakes employment decisions. As such, the legal issue pertinent to this review continues to warrant attention.
Purpose of This Review
In this review, we explore why HISD’s use of the EVAAS® was susceptible to a procedural due process claim. We highlight the case’s significance and note its limitations. To be sure, there are indicators that the use of VAMs across states and school districts is on the decline, overall (Close, Amrein-Beardsley, & Collins, 2020; Ross et al., 2017) and at least in part as explained by ESSA (2016). However, current evidence also indicates that many states and school districts continue to use VAMs state- and districtwide (Close et al., 2020; see also Ross et al., 2017). Moreover, given the human and financial capital already invested in their use (e.g., via Race to the Top [2011] and No Child Left Behind waivers [U.S. Department of Education, 2014]), it stands to reason the policy inertia is providing for their continued use.
Beyond reason, evidence more specifically suggests that educational policymakers and leaders in some states and school districts continue to promote the use of VAM data to make high-stakes personnel decisions (e.g., merit pay, tenure, and contract nonrenewal), precisely the types of decisions at issue in Houston. For example, although states have granted school districts more local control over districts’ teacher evaluation systems since ESSA, 12 states still allow or encourage districts to make teacher termination decisions solely 6 or primarily on the basis of their VAM data, and 23 states allow or encourage such decisions per a combination of districts’ VAM and other evaluative (e.g., observational) data (Close et al., 2020). In addition, states that were using the EVAAS before ESSA have continued to use it since (e.g., EVAAS in Ohio, North Carolina, and South Carolina; the Pennsylvania Value-Added Assessment System in Pennsylvania; the Tennessee Value-Added Assessment System in Tennessee; the Texas Value-Added Assessment System in Texas). Many districts within states also continue their EVAAS use (e.g., districts in Arkansas, Georgia, Indiana, and Virginia). But these policymakers and leaders are ignoring (perhaps at their peril) Houston, which represents a cautionary tale for states and local districts currently using or still contemplating using any VAM as the sole (see note 6) or primary criterion for high-stakes employment decisions.
This review proceeds as follows. First, we highlight the major concerns of VAMs in the research, particularly those related to VAM use in high-stakes employment decisions because these themes were echoed in Houston and other federal cases. Although states and school districts were (and still are; Close et al., 2020) using VAM output alongside other teacher evaluation measures such as observations, and these measures can be similarly criticized for their statistical and pragmatic shortcomings (see, e.g., Gill, Shoji, Coen, & Place, 2016; Weisberg et al., 2009), the lawsuits detailed in Education Week (“Teacher Evaluation Heads to the Courts,” 2015), including Houston, directly pertain to states’ and school districts’ uses of VAMs only. 7 As such, we recognize but do not review the literature surrounding the systemic observations of teachers (see, e.g., Anderson, 2013; Goldring et al., 2015; Jiang, Sporte, & Luppescu, 2015; Steinberg & Garrett, 2016; Winerip, 2011). However, it is also important to note, again, that in Houston as well as in several other cases, VAM ratings effectively trumped (see note 5) observations in terms of the weighted subcomponents of teachers’ overall evaluations.
Similarly, and given space limitations in this review, we do not take a position as to the relative merit of VAMs in educational policy. Rather, we recognize other scholars who have also taken up this task (see, e.g., Bill & Melinda Gates Foundation, 2013; Chetty, Friedman, & Rockoff, 2014a, 2014b; Goldhaber, 2015; Kane, McCaffrey, Miller, & Staiger, 2013). Similarly, we bypass discussions regarding the use of VAMs to assess the impacts of larger scale educational policies and interventions, to which others have also attended (see, e.g., American Educational Research Association Council, 2015; American Statistical Association, 2014; Chiang, McCullough, Lipscomb, & Gill, 2016; Foundation for Excellence in Education, 2018, n.d.; Harris & Herrington, 2015). Our discussion of the criticisms of VAMs herein is for the limited purpose of illustrating the extent to which the Houston court may have addressed the issues recognized in the literature.
Second, we discuss the facts and ruling of Houston, emphasizing the court’s decision in favor of the teachers on procedural due process grounds because the school district did not allow the teachers access to underlying codes that generated teacher VAM ratings. Importantly, and by way of background, procedural due process is a constitutional guarantee that a teacher with a continuing contract has notice and a hearing to contest the underlying reasons for a termination decision before that termination occurs (e.g., cross-examine witnesses, rebut assumptions, and review evidence presented by the district). Procedural due process ensures fundamental fairness and is intended to reduce the risk for error in a government’s decision to terminate a teacher’s employment. As the court noted, without access to the EVAAS codes, teachers could not adequately defend against their termination and, therefore, would be deprived of their procedural due process rights.
Third, we draw several conclusions about what Houston means for the continued use of VAMs. Again, we note that there are many states and districts that are still approving, endorsing, using, or contemplating the use of VAMs (including EVAAS) for consequential decision-making purposes, despite the fact that ESSA lessened federal requirements pertaining to the use of VAMs. We also contemplate the impact of this case if HISD (and SAS Institute) had in fact made their proprietary codes public.
The use of VAMs has been widely criticized in the extant literature, and these criticisms center predominantly on statistical and pragmatic issues. A complete overview of the technical criticisms surrounding VAMs is beyond the scope of this review and can be found elsewhere (see, e.g., American Educational Research Association Council, 2015; American Statistical Association, 2014; Amrein-Beardsley, 2008, 2014; Baker et al., 2010; Harris & Herrington, 2015), as can literature reflecting the potential benefits of using VAMs for teacher evaluation purposes (American Educational Research Association Council, 2015; American Statistical Association, 2014; Bill & Melinda Gates Foundation, 2013; Chetty et al., 2014a, 2014b; Goldhaber, 2015; Harris & Herrington, 2015), but a brief sketch of the concerns regarding VAMs is warranted because the concerns reviewed do appear in the Houston decision. These concerns relate primarily to the following terms, as defined in Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014): reliability (see, e.g., Martínez, Schweig, & Goldschmidt, 2016; Schochet & Chiang, 2013; Yeh, 2013), validity (see, e.g., Hill et al., 2011; Kane et al., 2012; Martínez et al., 2016); bias (see, e.g., Koedel, Mihaly, & Rockoff, 2015; Newton, Darling-Hammond, Haertel, & Thomas, 2010; Rothstein, 2009), transparency (see, e.g., Eckert & Dabrowski, 2010; Goldring et al., 2015; Graue, Delaney, & Karch, 2013), and fairness (see, e.g., Gabriel & Lester, 2013; Harris, 2011; Jiang et al., 2015). Although the court did not overturn the district’s use of VAMs on the basis of statistical concerns as a matter of substantive due process, the court’s analysis on procedural due process notes these concerns, especially as related to transparency and fairness.
The Facts and Legal Theory in Houston: Procedural Due Process as a Viable Theory to Challenge VAMs
The relevant facts of Houston are as follows. In 2007, HISD contracted with a private third-party vendor (SAS Institute) to use its EVAAS to evaluate HISD teachers, at a cost of $500,000 to $680,000 per year (Z. Capo, personal communication, May 10, 2016). EVAAS generates a particular score for a teacher using SAS’s proprietary software. The EVAAS score is then converted into a “teacher gain index,” which sorts teachers into five effectiveness categories: well above, above, no detectable difference, below, and well below average levels of effectiveness. Starting in 2011, as also mentioned earlier, HISD did not renew 221 of its teachers’ contracts, because they demonstrated “a significant lack of student progress attributable to the educator” or “insufficient student academic growth reflected by [their EVAAS] value-added scores” (Houston Independent School District, 2011). Although at the time the EVAAS component of HISD’s teacher evaluation systems was to carry only 30% of the weight of a Houston teacher’s effectiveness rating, compared with 50% for observed performance, the VAM measure trumped its accompanying observational measure if the observational measure contradicted or did not correlate with the VAM estimate (see note 5). This, effectively, turned the EVAAS measure’s weight to 100%, not the 30% allotted.
As such, Houston teachers sought to enjoin the district from using the EVAAS, as well as a judgment that HISD’s use of the EVAAS was unconstitutional. The teachers alleged that HISD’s use of EVAAS violated the Fourteenth Amendment’s equal protection clause, substantive, and procedural due process clauses. In particular, the teachers contended that they were deprived of their constitutional right to procedural due process because SAS Institute would not allow them access to the underlying algorithms (on the often named “black box” VAM, see, e.g., Amrein-Beardsley, 2008; Ballou & Springer, 2015; Collins, 2014; Eckert & Dabrowski, 2010; Kappler Hewitt, 2015) used to calculate their ratings. Without the codes, teachers could not determine if there were errors that may have led to mistakes in their evaluations and, therefore, the decisions to terminate. This inaccessibility, they argued, violated fundamental principles of fairness and guards against mistaken deprivation of property rights that procedural due process is intended to guarantee.
Equal Protection and Substantive Due Process Claims
The court dismissed the teachers’ equal protection and substantive due process claims, noting that the school district’s decision (to use VAMs for its employment decision) need only satisfy a rational basis test, a low bar of constitutional scrutiny applied to reviewing a state law or policy (Houston, 2017, p. 1183). Under this test, a government’s decision to adopt a policy or law need only have some rational basis to support a legitimate government end. Courts rarely overturn a law or policy when the rational basis level of scrutiny is applied and defer to the government’s policy choice (United States v. Carolene Products, 1938). Here, the court agreed that there was some rational basis that the district’s use of EVAAS to terminate teachers may ultimately improve student performance and teacher quality. Importantly, the rational basis argument (that EVAAS will improve student performance) may be debatable among educational experts, but it is strong enough to survive an equal protection or substantive due process challenge.
The Houston court borrowed from other federal courts’ analyses of the equal protection and substantive due process claims (see, e.g., Cook v. Bennett, 2011; Trout v. Knox County Board of Education, 2016; Wagnar v. Haslam, 2015). The Eleventh Circuit Court of Appeals (a court one level below the Supreme Court), for example, held that the state of Florida’s reliance on VAMs to make evaluations and employment decisions passed the rational basis test (Cook v. Bennett, 2011). To be sure, the Cook court expressed skepticism about the wisdom of using VAMs, but it could not, as a matter of law, find that the use of VAMs failed the rational basis test. In Trout v. Knox County Board of Education (2016), the court had substantial questions about the reliability and validity of the Tennessee Value-Added Assessment System (the original version of the EVAAS, implemented in Tennessee), but the court determined that the state satisfied the very low threshold of the “rational basis test.” Similarly, in Houston, the court held that although the EVAAS may be a “blunt tool” that “may produce only marginal results,” it also satisfied the rational basis test (Houston, 2017, p. 1183). Put another way, the statistical concerns education researchers and plaintiffs raised in these cases as well as in Houston, as we outlined earlier, were insufficient to convince the Houston court that the use of EVAAS to terminate teachers was unconstitutional.
Procedural Due Process Claims
However, the court upheld the teachers’ procedural due process claims, marking the first time a federal court found in favor of teachers challenging the use of VAMs for their termination as unconstitutional. Procedural due process is a guarantee of certain procedural protections before states (or their actors, like school districts) attempt to deny a property right, like a continuing contract to teach (see, e.g., Cleveland v. Loudermill, 1985). 8 In its simplest form, it is a constitutional guarantee to fairness if the government intends to take away a property interest. As also noted earlier, it requires that the government (e.g., a school district) must provide notice and a hearing when it threatens to deny a person that interest. To succeed on a procedural due process claim, then, a plaintiff must first demonstrate the plaintiff has a recognized property interest and the plaintiff did not receive adequate process when the government proposed to take away that property.
The teachers in Houston had a property interest in continued employment (i.e., they had continuing contracts), and therefore, they were constitutionally guaranteed a fair process that included “notice” and a “hearing” before they could be terminated. More specifically, a teacher was to “be advised of the cause of termination in sufficient detail [italics added] so as to (a) enable [the teacher] to show any error that may exist” (Houston, 2017, p. 1177, citing Ferguson v. Thomas, 1970), (b) hear evidence upon which a proposed termination is based, (c) present evidence on his or her behalf (Texas Education Code §§ 21.255–21.256), and (d) rebut evidence presented by the school district. Houston teachers contended that these minimal due process requirements were violated when they were denied access to the computer algorithms and data (i.e., the underlying EVAAS algorithms and computer codes) necessary to verify the accuracy of their scores. Put another way, because Houston teachers could not review the underlying information (and were consequently foreclosed from rebutting that evidence at a hearing), they were denied the constitutional protections of procedural due process.
The court agreed with the teachers’ position. Indeed, the denial of access to the EVAAS proprietary decision was dispositive to its ruling. The court emphasized the need for the teachers to be able to verify their scores’ accuracy. For example, noted the court, “any effort by teachers to replicate their own scores, with the limited information available to them, [would] necessarily fail” (Houston, 2017, p. 1177). This was also confirmed by the one of the plaintiffs’ expert witnesses, who was “unable to replicate the scores despite being given far greater access to the underlying computer codes than [was] available to an individual teacher” (p. 1177). Hence, and according to the unrebutted testimony of [one of the] plaintiffs’ expert [witnesses], without access to SAS’s proprietary information—the value-added equations, computer source codes, decision rules, and assumptions—EVAAS® scores [would] remain a mysterious “black box,” impervious to challenge. (p. 1179)
Thus, the court concluded that the teachers had “no meaningful way to ensure correct calculation of their EVAAS® scores, and as a result [were] unfairly subject to mistaken deprivation of constitutionally protected property interests in their jobs” (Houston, 2017, p. 1181). The court added that a simple data-entry mistake or glitch in the computer code could also produce an incorrect score (p. 1177). Even HISD acknowledged that mistakes can occur (p. 1178). And, as the court found, one mistake would also lead to many others and “could alter the scores of every other teacher in the district.” The court subsequently framed the district’s scoring system as built on a “house of cards fragility” (p. 1178; see also Ballou & Springer, 2015). In sum, because SAS Institute refused to release that information and HISD teachers could not review (and potentially refute the underlying evidence for their terminations), they were denied procedural due process.
What would have happened had EVAAS and HISD released the code and continued to use EVAAS for high-stakes decisions such as termination? In our view, this would have remedied one procedural due process problem but created another, at least from the district’s perspective. On the one hand, the procedural due process concern identified by the Houston court (access to the code) would have been rectified; the teachers would have had the codes and underlying information that was the gravitas of the complaint. On the other hand, the procedural due process issue would persist. Significantly, as the Houston court noted, due process is not simply to ensure access to the underlying decisions, but is “designed to foster government decision-making that is both fair and accurate [italics added]” (Houston, 2017, p. 1176). If underlying codes were accessible, inaccuracies in the code (if they existed), in fact, could be identified, raising questions of fairness and accuracy. Moreover, this would have opened the door for teachers to raise issues at termination hearings concerning the statistical sensitivity of the EVAAS, an issue that has also been well documented by researchers (Ballou & Springer, 2015).
It is quite possible, therefore, that the procedural due process issues would have persisted if the codes were accessible, as long as HISD continued to use those codes for adverse employment decisions. At the very least, HISD would invariably be mired in litigation with the proverbial “battle of the experts” debating the statistical merits and accuracy of the codes. This would not mean, necessarily, that teachers could defeat the termination decision, but it is to say that the risk for litigation would continue had the district proceeded to use EVAAS for high-stakes decision-making purposes. Thus, as we also note in the final section, VAMs such as EVAAS may simply be better suited for use in lower stakes situations.
In the end, though, the Houston teachers’ claims survived a motion to dismiss (the court made its ruling in the context of a motion for summary judgment) and were allowed to go forward. The case never proceeded to trial, and the sides ultimately settled. The terms of the settlement effectively granted the relief the teachers sought: that HISD would no longer use the EVAAS, or any other VAM, as a basis to terminate the employment of a term or probationary contract teacher during the term of that teacher’s contract, or to terminate a continuing contract teacher at any time, so long as the value-added score assigned to the teacher remain[ed] unverifiable. (State of Texas, 2017, p. 2; see also American Federation of Teachers, 2017)
Additionally, HISD agreed to pay plaintiffs’ attorneys’ fees and costs associated with bringing the case to federal court, amounting to approximately $237,000 (State of Texas, 2017, p. 2; see also American Federation of Teachers, 2017).
Now, HISD no longer uses a VAM. Instead, it uses a homegrown comparative growth model that is calculated internally and only used “to identify areas of strength or concern” in terms of teacher effectiveness. No high-stakes personnel decisions are attached to these data (Z. Capo, personal communication, June 2, 2019). We are otherwise generally unaware of how the Houston decision has since influenced other states or school districts in this regard. For those still using or contemplating the use of a VAM, including the EVAAS, for such purposes, however, the issues presented in Houston should be addressed, so as to also reduce the risk and costs of potential litigation elsewhere.
Conclusions: Legal and Policy Consequences of Houston
State and school district lawmakers, policymakers, leaders, administrators, and teachers should recognize the significance of Houston. Although the decision is subject to some limitations (noted below), it is important for several reasons.
First, because 12 states still allow or encourage districts to make teacher termination decisions solely (see note 6) or primarily on the basis of their VAM data, and 23 states allow or encourage such decisions per a combination of districts’ VAM and other evaluative (e.g., observational) data (Close et al., 2020), and the states and districts that were using the EVAAS before ESSA are still using the EVAAS today, this implies that the Houston ruling may not have had the impact it should have (especially since May 2017). This could be because state and school district policymakers, leaders, administrators, and teachers are unaware of this important case and ruling, made critical, again, given that this was the first time that challenges to VAMs succeeded under a federal constitutional theory. Likewise, although ESSA does not prohibit the use of VAMs, and VAMs (including the EVAAS) continue to be used and contemplated for use, given the resources devoted to VAMs through the federal government’s initiatives (e.g., Race to the Top), it is fair to suggest that policy inertia will promote their continued use. However, those who continue to use VAMs in general should also be aware.
Second, Houston suggests that the federal courts are not entirely foreclosed as a venue for teachers seeking remedy for perceived unfairness of VAMs. Until Houston, plaintiff-teachers had generally not succeeded in federal court in challenging school districts’ termination decisions on the basis of VAM ratings. It had appeared that the Eleventh Circuit’s Cook v. Bennett decision and those that followed it, discussed earlier, effectively closed the federal courts as a receptive venue for remedy. However, Houston represents one potentially successful line of attack for those pursuing relief in federal courts, in addition to other venues, such as state courts or administrative law proceedings (Paige, 2016). To be sure, Houston’s facts may be unique in that teachers were denied access to the code, and this was a dispositive fact to the ruling. Houston’s facts may also generalize, though, even if such codes and algorithms were made more accessible, 9 wherein procedural due process issues may still persist. Notwithstanding, the Houston court also called attention to the fact that procedural due process requires a hearing to determine if a district’s decision to terminate employment is both fair and accurate. Thus, even if the codes were released (satisfying a fairness prong), teachers still could contest the accuracy of the underlying decision to terminate an individual teacher by raising the host of statistical issues also identified in the literature. 10 It does not mean that teachers would succeed, but it does create a viable path (and costs and risks for both sides) for a procedural due process claim regarding a proposed termination even where codes may be accessible.
Third, the Houston decision raises a question of government transparency and control of private, for-profit corporations engaged in providing a public good, such as education. This concern has also been noted in other contexts with regard to public oversight of charter schools (see, e.g., Mead, 2003). In those circumstances, scholars have noted that public tax dollars are spent on private, for-profit enterprises, but there is a lack of public oversight and control. Similar concerns are reflected in Houston. In this case, teachers and the school district itself were prohibited from accessing important information (i.e., the underlying formulas and computer codes used to terminate teachers) that had significant impact on the public school workforce.
Fourth, Houston raises questions regarding the ability of EVAAS ability to improve teacher quality, insofar as the underlying algorithms remain out of public inspection. Even assuming the premise that VAMs can be linked to improved teacher quality, the Houston court identified numerous areas where errors could occur. More troubling is that a simple error (e.g., even as mundane as a data-entry mistake) has a ripple effect on all other teacher ratings. This could have significant consequences, not the least of which might be that a well-performing teacher would be erroneously misclassified as a poor performer, or the inverse. If EVAAS (and VAMs) remains removed from public inspection, those errors go undetected. Thus, a school district may be taking negative employment action against people whose performance does not warrant such action or, again, the inverse. Without some public scrutiny to review for such errors, these risks remain. Alternatively, if the data were open to review, it would incentivize school districts to develop and improve the integrity of other metrics used in evaluation, such as principal observations, so that the accuracy of any VAM categorization could be better assessed or “triangulated” against other information.
Finally, this case, and several others at the state level (Paige, 2016), should bring policymakers and practitioners to an inflection point to consider the role, if any, VAMs should play in employment decisions. Houston ultimately demonstrates some of the costs of using a particular VAM in high-stakes situations; HISD engaged in a protracted, costly legal battle, ultimately abandoned the use of VAMs, and paid plaintiffs’ attorneys’ fees. This is not to say that districts should ignore VAMs in attempting to improve teacher quality. But it is to say that their most efficient use may be in lower stakes situations (i.e., removing them as a component of termination decisions) that recognizes their imperfections but attempts to also explore how they might be useful. This would reduce the risk exposure of school districts as a legal matter. But more important, this may also better situate VAMs to be an ingredient of teacher quality rather than a perceived obstacle. We leave it for others to see how, or if, the former can be achieved.