Improving Balance in Educational Measurement: A Legacy of E. F. Lindquist

Technique Versus Practice in Educational Measurement

Much of the material in Lindquist’s (1951) chapter illustrates what I am calling issues of practice. The terms practice and technique are ambiguous, so additional clarification is needed.

I use technique and technical work to refer to the mathematical and procedural aspects of test construction, scaling, linking, and reporting. I use practice to refer to issues that arise from the actual use of tests, including the purposes and goals of testing, the limitations of testing, the trade-offs between different approaches to testing, the diverse effects of testing, and the implications of these issues for test design, test use, and inference.

These two categories clearly overlap, and the defining difference between them is often focus and motivation rather than substance. For example, Lindquist devoted a substantial portion of his 1951 chapter to test construction, but his focus was not procedural concerns, such as maintaining appropriate item-test correlations. His focus was on the practical considerations constraining test construction. The is also true of his much briefer discussion of task formats. Moreover, addressing practical issues often requires technical work. In contrast, issues of technique can be, and often are, discussed without reference to practical problems or implications.

Inappropriate test preparation and the score inflation that it can create provide a clear illustration. These are clearly issues of practice, that is, they arise from the particular uses of tests in schools, admissions, and certification, and they create severe practical problems of inference. This is not to say, however, that they lack technical elements. Evaluating score inflation entails technical work, and addressing and ameliorating these problems entail difficult technical work in test design, linking, and validation (e.g., Haertel, 2013; Koretz, 2015; Koretz & Beguin, 2010; Morley et al., 2004). However, the motivation for this technical work is to address a practical problem that arises from test use. It is not motivated purely by an interest in technique per se, that is, by an interest in technical advancement or refinement.

These definitions help clarify the balance in E. F. Lindquist’s work. His stature stems in large part from his work in the realm of technique, in particular, from his exceptional productivity as a developer of achievement tests. Yet throughout his career, he maintained a strong and unflinching focus on issues of practice, including some that were—and remain today—serious problems for standardized assessment.

It is easy to find other eminent scholars in the field whose work has shown a balance of this sort. I would include many of the scholars who previously received the E. F. Lindquist Award established in his honor. I cite numerous other examples in the following. Nonetheless, the balance that Lindquist and other scholars exemplified is atrophying in the field as a whole, to the detriment of both the field and the people it should serve.

The Evidence

This shift to reliance on standards-based reporting has occurred despite a recognition in the field that the results of standard-setting are arbitrary and often dramatically inconsistent across alternatives. This fundamental threat to valid inference was the focus of debate decades ago but has largely faded from professional discussion, replaced by copious discussions of procedural issues.

This lack of robustness was one focus of the special issue of JEM 45 years ago that I noted above. In that issue, Glass (1978) wrote that standard-setting is unavoidably arbitrary and maintained that

the consequences of the arbitrary decisions are so varied that it is necessary either to reduce their arbitrariness, and hence the unpredictability, or to abandon the search for criterion levels altogether in favor of ways that are less arbitrary and, hence, safer. (p. 237)

Glass’s choice of wording was unfortunate, because capriciousness is a red herring. The threat to valid inference arises from two factors: the arbitrariness of procedural decisions and the magnitude of the resulting variation in performance. Capriciousness is not essential. If a state can choose arbitrarily between one method that labels 15% of students as “proficient” and another than labels 60% as “proficient,” users of scores clearly cannot have confidence in either estimate.

Unfortunately, the inconsistency among standards-setting results is sometimes even larger than this hypothetical example. This was made clear by additional research over the two decades following that 1978 debate.

A decade after that special issue, Richard Jaeger, then one of the nation’s leading experts on performance standards, published a review in the third edition of Educational Measurement (Jaeger, 1989). He included a table showing 32 contrasts between methods drawn from 12 studies. In each case, he tabulated the ratio of the higher percentage passing to the lower percentage passing. These ratios ranged from 1 to 42, with a mean of 5.3 and a median of 1.5. A decade and a half later, Linn (2003) evaluated the results of three standard-setting methods (bookmark, contrasting groups, and Jaeger-Mills) for elementary-school tests in five subject areas in a new Kentucky assessment (Table 1). The ratios of highest to lowest percentage proficient ranged from 1.8 in mathematics to 10.8 in social studies, with a mean across subjects of 5.2.

OPEN IN VIEWER

Table 1.

Percent of Students Labeled “Proficient” Using Three Methods for Applying Standards in Kentucky

Method	Reading	Math	Science	Social Studies	Arts and Humanities
Bookmark	57	35	35	48	15
Jaeger-Mills	15	21	5	5	11
Contrasting groups	29	20	25	27	25
Multiple, high/low	3.7	1.8	7.5	10.8	2.3

Note. Adapted from Linn (2003).

Worse, inconsistencies across methods are not the only source of a lack of robustness. It has been documented for at least 30 years that procedural differences within a given method can create substantial inconsistencies in results. For example, an evaluation of the 1992 NAEP Achievement Levels (NAEP’s terminology for performance standards) conducted by a panel of the National Academy of Education (Shepard et al., 1993) found that the backgrounds of the panelists, item format, and item difficulty all affected the cut scores set using the modified Angoff method. And the problem of within-method inconsistencies is not limited to the Angoff method. In the bookmark method, the predominant method in U.S. K–12 education, panelists are given definitions of the performance standards, an ordered item booklet (OIB) that presents test items in ascending order of difficulty, and a response probability (RP), which is the proportion of students at the cut score that should answer the target item correctly. They must then imagine students at the cut score and select the item from the OIB that these students will answer correctly with a probability equal to the RP. The RP is a largely arbitrary number, and researchers working in the area have argued for different values (see, e.g., Lewis et al., 2012; National Research Council, 2005). If panelists were able to perform this task well, the choice of RP shouldn’t matter; changing the RP should simply lead the panelists to select a different item from the OIB, and the same students would be identified as reaching the cut score. However, a number of studies have shown that panelists cannot do this, and changing the RP results substantially different cut scores (e.g., National Research Council, 2005). ²

The Responses in the Field

Inconsistencies of this magnitude create a fundamental—I would claim unsurmountable—barrier to valid inference unless there is either clear justification for choosing one specific method and one specific approach for implementing that method or other data supporting the validity of the inference. There is generally neither. As Linn (2003) aptly stated:

The variability in the percentage of students who are labeled proficient or above due to the context in which the standards are set, the choice of judges, and the choice of method to set the standards is, in each instance, so large that the term proficient becomes meaningless. (p. 13)

Linn’s suggestion that we not rely on performance standards has been almost universally ignored, despite his eminence in the field. To a substantial degree, the blame for this lies with policy decisions, not decisions by people in the field, but in recent years, the field also has largely ignored the truly fundamental problems that Jaeger, Linn, Hoover, Shepard, and others documented. To be fair, a modest number of people have tried to address the lack of robustness. For example, both Linn (2003) and Jaeger (1989) suggested that if cut scores are used, the results of more than one method should be presented. Unfortunately, this isn’t a practical suggestion; it is hard to image a department of education releasing conflicting estimates. Another example is the work of Haertel (2002) and Haertel and Lorié (2004), who proposed a new approach to setting standards, labeled the briefing book method, that might tie cut scores more clearly to the inferences they are intended to support. But for the most part, the field has ignored these issues and has devoted its energies to copious additional technical work on standard-setting. Some years ago, I had a conversation with a leading researcher in the area of performance standards in which I lamented the lack of a thorough review of the robustness of performance standards more recent than Jaeger’s (1989) chapter. Their answer was that people working in the area were no longer very interested in the question of robustness and instead were focusing more on procedural issues. That seemed to me to be a damning indictment of the field’s atrophying attention to practical issues, even when they are fundamental.

The Evidence

During the 1980s, as high-stakes testing gradually spread in the United States, a number of prominent scholars in the measurement field warned about the risk of score inflation (e.g., Koretz, 1988; Linn, 1987; Madaus, 1988a, 1988b; Shepard, 1988). Empirical evidence of both inappropriate test preparation practices and score inflation began to accumulate shortly afterward.

The types of behavioral responses to testing and their relationship to score inflation have been clarified by work over the past several decades. Some “teaching to the test” is of course desirable—for example, finding areas in which instruction needs to be strengthened and identifying students’ weaknesses for remediation—but a number of responses can bias scores. One is cheating. While data on the incidence of cheating are sparse, it is not rare (Koretz, 2017). Another response to testing can be called gaming, such as manipulating the sample of students tested (e.g., Figlio & Getzler, 2002; Jennings & Beveridge, 2009). Two responses that are particularly relevant to Lindquist’s and Campbell’s concerns and that have been the focus of considerable subsequent research have been labeled reallocation and coaching (Koretz & Hamilton, 2006; Koretz et al., 2001). Reallocation refers to shifting time and other instructional resources to better match the specific content of a particular test. Reallocation from untested to tested subjects does not in itself inflate test scores, although it can be undesirable for other reasons and can distort some inferences based on scores. For example, it remains unclear how much if any of the much discussed rapid rise in fourth-grade mathematics scores on the NAEP that began in 1990 stemmed from instructional time shifted from other subjects to mathematics rather than improved mathematics instruction or other factors (Koretz, 2017). Reallocation within a tested subject, which has been the focus of numerous studies, inflates scores when the de-emphasized material is relevant to the inferences based on scores.

Although it can inflate scores, reallocation is often encouraged by education authorities, as illustrated by a PowerPoint presentation placed online by a Massachusetts school district (Quincy Public Schools, 2004). Each slide showed a chapter of the algebra I text used in the district, broken into numbered sections. Each section showed the numbers of all of the released state test items from the last several years that had measured content from that section, which made it immediately apparent which sections—most of them—had not been tested at all.

Coaching refers to focusing on specifics of a test that are not germane to the inference but that can nonetheless affect performance, such as item format, other aspects of item presentation, and the specifics of scoring rubrics. Test-taking tricks, such as the process of elimination for multiple-choice items, are also examples of coaching. A clear illustration of coaching is provided by a test-preparation book for the tenth-grade Massachusetts MCAS test published by Princeton Review (Rubinstein, 2000). The tests at that time typically included an item about the Pythagorean Theorem. The preparation for these items offered in the book was of two types. One simply stated the formula so that students could memorize it. While this may be, in the words of one of my students, “garden-variety lousy teaching,” it is not coaching, and there is no reason to expect it to bias scores. The second type of preparation, however, capitalized on the substantively irrelevant fact that test items about the theorem typically have simple integer solutions because students do not know how to calculate square roots without a calculator. Many students who know the theorem would not be able to demonstrate that knowledge if the item required a noninteger solution. In response, the Princeton Review materials instructed students that one of two “common” or “popular” ratios, 3:4:5 or 5:12:13, or multiples of these, would solve the item (Rubinstein, 2000, p. 56). These ratios are not particularly common in the real world, but they are popular among item writers. This coaching permits students to answer the item correctly without learning the theorem, thereby inflating their scores.

A considerable number of studies over more than 40 years have documented behavioral responses of educators that can inflate scores, in particular reallocation (e.g., Hamilton et al., 2007, 2008; Koretz, Barron, et al., 1996; Koretz, Mitchell, et al., 1996; Luna & Turner, 2001; Pedulla et al., 2003; Romberg et al., 1989; Shepard & Dougherty, 1991; Smith, 1991; Smith & Rottenberg, 1991; Stecher, 2002; Stecher & Barron, 2001).

The first empirical evidence of score inflation may have been two papers by John Jacob Cannell, a West Virginia physician who began investigating when young patients of his who were experiencing academic difficulty nonetheless reported that they scored well on achievement tests (Cannell, 1987, 1989). He found that although his state of West Virginia scored very poorly on other educational indicators, the state’s students scored at the 65th percentile rank in grade three and at the 62nd percentile rank in grade six (Cannell, 1987, p. 1; subject not specified). He then began collecting data nationwide and found that:

No state scores below the publisher’s “national norm” at the elementary level on any of the six major nationally normed, commercially available tests. A few states had one or two elementary grade levels or an occasional subject that was below average, but most states had all subject areas and all elementary grade levels well above the publisher’s “national norm” of the 50th percentile. (1987, p. 2)

Unambiguous evidence of score inflation began accumulating a few years later. The first of these studies was a cluster-randomized administration of multiple tests carried out in a district that had by today’s standards only moderately high stakes (Koretz et al., 1991). The study included two primary comparisons: the difference in performance between a new test administered by the district and a quite similar test administered the previous year, and the difference in performance on the old test in its last operational year and when readministered by the researchers four years later. Both comparisons suggested that that performance in third-grade mathematics was inflated by roughly half an academic year.

Additional confirmation of score inflation continued to accumulate, usually from studies that compared the performance of the same students or randomly equivalent groups of students on a high-stakes test and a lower stakes audit test of similar content—often, the NAEP (e.g., Fuller et al., 2006; Haladyna et al., 1991; Haney, 2000; Ho, 2007, 2009; Ho & Haertel, 2006; Jacob, 2005, 2007; Jennings & Bearak, 2014; Klein et al., 2000). Two early studies found severe inflation in Kentucky’s KIRIS program, which relied largely on constructed-response items and performance tasks (Hambleton et al., 1995; Koretz & Barron, 1998). These were particularly important at the time in showing that score inflation is not limited to multiple-choice tests.

Moreover, the magnitude of score inflation is often very large. For example, a number of studies have found gains on high-stakes tests two to six times as large as those on NAEP (e.g., Jacob, 2007; Klein et al., 2000; Koretz & Barron, 1998). In the first three years that New York State administered a new test introduced in 2006, the mean score in eighth-grade mathematics increased by 0.58 standard deviation, seven times the gain the state’s students showed on NAEP, 0.08 standard deviation (Figure 1). In one case, the gains of 0.76 standard deviation on a fourth-grade high-stakes reading test were accompanied by no gains whatever on the NAEP (Hambleton et al., 1995).

OPEN IN VIEWER

Figure 1.

Trends in Eighth Grade Mathematics Scores in New York State, State Test and National Assessment of Educational Progress (Koretz, 2017).

The Responses in the Field

The accumulation of empirical evidence of score inflation and inappropriate test preparation generated substantial discussion in the field of educational measurement and, to a much lesser degree, economics (e.g., Neal, 2011). Recall that Lindquist raised the issue in the first edition of Educational Measurement (Lindquist, 1951). Fifty-five years later, inflation, the behaviors producing it, and methods for evaluating it were all discussed in detail in the fourth edition of Educational Measurement (Koretz & Hamilton, 2006). Perhaps the single most striking indication of the extent to which the problem of test preparation and inflation was recognized is the aforementioned 2003 paper by Popham, who had been one of the most influential proponents of measurement-driven instruction, that is, high-stakes testing. In that paper, entitled “Seeking redemption for our psychometric sins,” Popham (2003) excoriated the field for failing to respond to the degradation of instruction caused by high-stakes testing, including reduced attention to untested curricular content and the use of “seemingly interminable test preparation practice sessions” (p. 46) in an effort to boost scores. While some in the field of measurement might disagree about the magnitude and frequency of inappropriate test preparation and score inflation, it seems that recognition these problems is nearly ubiquitous in the field of educational measurement.

Inappropriate test preparation and score inflation have critically important implications for several core aspects of measurement practice. Yet despite the very long time that these problems have been recognized—nearly a century, if one goes back to the Iowa Brain Derby, and three-quarters of a century if one starts the clock with Lindquist’s (1951) chapter—standard measurement practice has not changed appreciably in response.

Perhaps most obvious, the risk of inflation requires augmenting conventional validation. Haertel (2013) noted that “Using high-stakes tests to focus the system on specific subjects or specific kinds of knowledge and skills carries substantial risks of unintended consequences, including score inflation and distortion of curriculum coverage” (p. 13) and argued that validation must evaluate these effects. Clearly, traditional validation methods cannot accomplish this. Koretz (2015) clarified that

The empirical evidence adduced in traditional validation is typically collected before high stakes have influenced educator and student behavior—that is, before there has been an opportunity for score inflation to occur. This evidence is also primarily cross-sectional and correlational, so it is insensitive to changes in level even if collected repeatedly. (p. 20)

Inappropriate test preparation and score inflation also have critically important implications for test design (Koretz, 2015). Reallocation and coaching can only inflate scores when there is unnecessary predictability of content or of substantively irrelevant details of item and test construction. One can see this predictability documented in test-preparation materials, but for a systematic exploration of these recurrences in two testing programs, see Holcombe et al. (2013). The unnecessary recurrences that afford the opportunity for coaching improve item-level performance (Morley et al., 2004).

The consequences of test preparation and score inflation for test design have received even less attention from the field. I am aware of only two suggestions for new approaches in test design to lessen the opportunities for inappropriate test preparation and score inflation, and only one of these is from the field of educational measurement. The economists Barlevy and Neal (2012) suggested an approach that they labeled pay for percentile in which the test is entirely replaced for each administration with a test that lacks unnecessary predictability. However, this approach is intended for cases where only rank order matters and is not useful for applications where scaling is necessary, for example, for description of trends. Koretz and Beguin (2010) suggested self-monitoring assessments that would include embedded audit items that do not share unnecessary characteristics with other items.

These two approaches have had no appreciable impact on the practice of educational measurement. The pay for percentile approach has had limited application in development economics, but some of these studies do not implement the key feature of Barlevy and Neal’s approach, that is, the elimination of predictable elements (Taylor, 2023). Moreover, to my knowledge, this approach has not been implemented in the American educational system. Self-monitoring assessments have been tried in a few pilot studies in a single state (Koretz et al., 2016) but have not been otherwise used in operational assessments.

Test preparation and score inflation also have major, troubling implications for linking. Most large-scale assessments are linked over time using some form of nonequivalent groups anchor test (NEAT) linking, in which a small number of anchor items are repeated from one administration to the next, and performance on those items is used to estimate aggregate change. In testing programs that use NEAT linking, which includes most assessments in which score inflation has been evaluated, inflation can only arise when NEAT linking fails, that is, if the estimated difficulty of the anchor items in a new administration is biased (Koretz, 2015). This need not require that the anchor items themselves become familiar; it may be sufficient for the anchor items to share attributes with other items that are familiar. Either way, the fact of score inflation necessarily indicates that linking has failed. Therefore, in the absence of other data, NEAT linking cannot be trusted under high-stakes conditions.

Nonetheless, NEAT linking is still generally conducted as if this problem did not exist. Standard methods of screening out individual linking items, which eliminate anchor items that lie far from the equating line that relates performance on the prior and current administrations, do not address the problem of inflation, which entails a displacement of the entire equating line.

Thus, to a large degree, large-scale tests are still designed, linked, and validated as if inappropriate test preparation and score inflation were not an issue, even though empirical studies have shown that this bias is not rare and can be extremely large.