Strive for Measurement,Set New Standards,and Try Not to Be Evil

AI and Educational Measurement: Definitions and Intersections

I have previously argued that the term measurement (and for that matter, the term equity) has become so encompassing that its boundaries are getting lost, and this can have unfortunate repercussions for deliberative discourse (Briggs, 2022). The same is true for the term “AI.” In the DET Responsible AI Standards, “the term ‘AI’ refers to AI systems and AI-adjacent methods and disciplines, such as computational psychometric methods” (Burstein, 2023, p. 5). In contrast, the definition of AI in the AI Principles released by the Association of Test Publishers (ATP) is strongly influenced by regulatory text passed by the European Union that focuses more narrowly on AI systems: “an AI system is one that perceives its environment and takes actions through ‘learning, reasoning, and modeling’ of data that maximizes its chances of success.” More formally, the Guidelines for Technology-Based Assessment define AI Systems as

Software and/or hardware systems designed by humans that, given a complex goal, act in the physical or digital dimension by perceiving their environment through data acquisition, interpreting the collected structured or unstructured data, reasoning on the knowledge or processing the information derived from this data, and deciding the best action(s) to take to achieve the given goal. AI systems can be used in technology-based assessments to assist humans in making test administration and scoring decisions or to make automated decisions in place of humans. (International Test Commission and Association of Test Publishers, 2022, p. 138)

What role can generative AI and AI-adjacent methods play in educational measurement? I follow Mari et al. (2023) in defining measurement as a process that provides information about a property of an object or event in the world, uses instrumentation designed to be sensitive to differences in this property, produces information in the form of values related to a scale or unit for that same property, and is qualified by information about uncertainty. Briggs et al. (2024) elaborate this definition with respect to four interacting components that are explicitly or implicitly part of any measurement process: theory, instrumentation, scales and units, and modeling (also see Briggs, 2022, and for a related NCME task force conceptualization of educational measurement competencies, see Ackerman et al., 2023).

The use of methods, such as adaptive testing, AIG, and automatic scoring, provide a classic example of an interaction between instrumentation and mathematical modeling that has long characterized the psychometrics of educational testing. An innovative feature of the DET is the use of construct theory to guide AIG, such that it becomes possible to generate items with known psychometric properties (e.g., difficulty) on the basis of manipulable variables that define item construction. To the extent this is possible, it leads to an interaction between on-the-fly item generation and test adaptation, which would seem to bring the testing experience more in line with the generative AI systems conceptualization in the ATP-ITC guidelines. Just how fully and successfully this is being realized in the current DET implementation is not entirely clear, but it seems some intriguing innovations are in the works (c.f., Attali et al., 2022; Cardwell, Naismith, et al., 2023; Maris, 2020).

To the extent that generative AI or AI-adjacent methods are being employed in the context of an assessment for learning activity (e.g., Duolingo’s language learning app), since they do not produce scores—at least not ones that get expressed on a formal measuring scale—it does not need to represent an instance of educational measurement. For this and other reasons outside the scope of this commentary (but see an excellent discussion in Swiecki et al., 2022), the prospective use of truly generative AI to support assessment for learning should be separated from its use in a more general sense (i.e., AI-related methods) to support the assessment of learning. Importantly, the assessment of learning requires measurement; the use of assessment for learning may not. What matters most in the latter case is whether the incorporation of AI into assessment activities has a demonstrable and significant effect on learning, and the evidence for this is often thin and/or equivocal. ²

Standards, Principles, Guidelines, or Goals?

Standards can age fast. Although 2014 does not seem that long ago, within about 5 years of their publication, it was already clear that the AERA/APA/NCME Standards for Educational and Psychological Testing (hereafter referred to as the AERA/APA/NCME Standards) contained limited discussion and guidance regarding the applications of AI in testing specifically, and new digital technologies in testing more generally. This has left a void that testing practitioners have taken increasingly active steps to fill. One starting point was the publication of AI Principles by the ATP in January 2022. This was followed in October 2022 by the release of the Guidelines for Technology-Based Assessment by the ITC and the ATP, and then most recently by the DET Responsible AI Standards. The ATP AI Principles are brief and general; in the ITC-ATP Guidelines, AI and AI-related methods are cast as instances of technology-based assessments, and the related guidance is fairly extensive. The DET Responsible AI Standards—which claims in its title to present standards but is actually written with respect to “goals” and concrete activities that support these goals—sit somewhere in between.

The elephant in the room is the need for an immediate amendment to the AERA/APA/NCME Standards, which historically has had the most authoritative standing as a consensus document to guide and evaluate the quality of testing practices. Until this happens, I worry that the proliferation of principles, guidelines, and goals (many of which may be saying the same things with different labels) could lead to considerable confusion. It seems important to appreciate that the AERA/APA/NCME Standards, even in their current form, should still apply to tests that incorporate AI into their design, administration, and reporting. As such, the DET Responsible AI Standards could be improved by more clearly articulating how its collection of goals and activities are substantively different from the activities that should be undertaken for a test that does not incorporate AI-related methods (this is particularly true for the general topics in the DET Responsible AI Standards characterized as “Validity and Reliability” and “Privacy and Security”).

The two areas where the use of AI in a testing content has the most obvious implications not addressed (at least not in sufficient depth) in the AERA/APA/NCME Standards are in regard to fairness, social responsibility, and transparency. In essence, this is the entire basis for the ATP’s Principles: transparency, human-in-the-loop, balanced utilization, fair, and unbiased. A real strength of the DET Responsible AI Standards is that they take these general principles and express them in terms of more concrete and actionable goals and activities. The document would be even better if it gave activities done to foster and evaluate fairness, responsibility, and transparency even greater prominence by providing a crosswalk to how this is being enacted in the DET. The DET Responsible AI Standards appear to be intended as a complement to the AERA/APA/NCME Standards (Cardwell, Naismith, et al., 2023, p. 3), but if viewed in isolation, it would be easy to come away with the misconception that they are intended as a replacement. At a minimum, they serve as a useful conversation starter about key intersections between AI and educational testing. To further this conversation, it would be helpful to see DET staff make more explicit connections between the goals and activities listed in the DET Responsible AI Standards and the organization and information presented in the DET Technical Report. For example, what activities are easy to realize and lead to a desired goal, what activities are easy to realize but are insufficient to reach a goal, and what activities are for all intents and purposes more aspirational than actionable?

Scientific Advance and Transparency in a Competitive Marketplace

English language proficiency is a complex idea, one that should evolve over time as we come to a better understanding of it through experience and research. But should our understanding of it depend on the test that is used to measure it? On many prominent tests of English language proficiency administered to students who are nonnative English speakers (e.g., Test of English as a Foreign Language [TOEFL], WIDA, ELPA21), a total score is produced that represents a composite of four language modalities: speaking, listening, reading, and writing. Interestingly, on the DET, English language proficiency has been conceptualized somewhat differently in terms of four constructs that represent targeted two-way interactions of these modalities: literacy (reading and writing), comprehension (reading and listening), production (writing and speaking), and conversation (listening and speaking). Empirically speaking (Cardwell, Nydick, et al., 2023), scores from the DET appear to be positively correlated with scores on the tests of two of its principal competitors, TOEFL and International English Language Testing System (IELTS). But the correlations are not as strong as one might have anticipated: .71 with TOEFL and .65 with IELTS. By way of contrast, the SAT and ACT composites tend to have a correlation of about .90 (see, e.g., University of California, 2020). This raises some interesting questions. Ultimately, aren’t all these tests attempts to measure the same latent variables? To what extent can these relatively low correlations be explained by DET’s use of AI and/or AI-adjacent methods for its instrumentation and modeling (as opposed to challenges in collecting data to compute concurrent correlations)?

The ability to answer these questions and to come to a better (and perhaps more generalizable) understanding of language proficiency and how it develops, would require a degree of transparency and cooperation that I worry runs counter to the financial incentives of companies trying to thrive in a competitive marketplace. Duolingo (and for that matter, Educational Testing Service [ETS]) makes technical reports and research studies related to its tests publicly available, but they are not presently curated in a way that gives a complete and coherent picture of how AI and AI-adjacent methods are being used, so they are only transparent to a point. ³

When Google was an upstart company, its motto was “don’t be evil”; now, it is being sued by the U.S. Justice Department for monopolizing digital advertising technologies. The story of OpenAI—founded as a nonprofit with a promise of keeping its generative AI open source—is another cautionary tale (https://www.vice.com/en/article/5d3naz/openai-is-now-everything-it-promised-not-to-be-corporate-closed-source-and-for-profit). Duolingo’s mission is “to develop high-quality education and provide universal access” and it espouses a belief in “the principles of AI for good.” The DET Responsible AI Standards most certainly talk the talk—but is it truly possible for any testing company to walk the walk? Or will this become a fraught instance of the fox guarding the henhouse (Briggs, 2022)?