AI and Psychometrics: Epistemology,Process,and Politics

Psychometrics

Despite its eugenics origins, psychometrics has moved from being logical positivist to becoming more postmodern in theory and practice (Mislevy, 2018). Bob Mislevy’s work has been significant in these efforts. In earlier work, Mislevy (1997) argued for the utility of measurement models that are situated in discourse communities and interpreted as provisional and not fixed. Building on this, he later developed his sociocognitive framework for educational measurement. By integrating insights from cognitive science, psychology, and sociology, Mislevy’s sociocognitive framework for measurement calls for more holistic and contextually sensitive approaches to educational measurement that account for the complex interplay between individual cognition, social dynamics, and learning environments. As an important and noteworthy perspective in contemporary psychometrics, the postmodern turn in sociocoginitive approaches to educational measurement is a significant development that begins to approach the tenets of complex systems theories.

This is a far cry from where psychometrics began let alone how it was weaponized. Yet, even this theoretical turn in measurement maintains assumptions about the dynamics of systems and information. More specifically, even though there is a recognition of the interplay of cognitive, social, and environmental factors and the nonstatic nature of learning and the learner, what continues to be missed is the self-generating process of systems, what Chilean cyberneticians Maturana and Varela (1972) called autopoiesis. Additionally, while psychometrics has a theory of evidence, it does not draw from an information theory that not only assumes infinite information but also that there are always model indeterminacies that are not a “nuisance”, “error”, or “disturbance” but, as Claude Shannon (1948) posits, noise that is fundamentally information too; that is, what psychometrics needs is a different epistemology of noise (i.e., error). While computational psychometrics introduces methods based in complex systems theories of cybernetics, it seems as though this effort is done so in the service of traditional practices of psychometrics (e.g., automated item generation or scoring), but the epistemological and ontological assumptions of psychometrics remain intact. What might a full integration of AI do to psychometrics? What might (unrestricted) autopoiesis and indeterminacy do to the constructs of psychometrics? How might there be potential for accounting for fairness concerns of situated knowledges and multiplicity of know-how? In what ways is AI already psychometrics-in-the-wild?

AI and Theories of Cybernetics

As Pasquinelli (2023) documents, AI is made up of Norbert Weiner’s feedback loops or recursive systems, Claude Shannon’s information theory, and logical mathematics initially developed by Warren McCulloch and Walter Pitts as well as John von Neumann. In its inception, AI was built on the complex systems theories of cybernetics, which became formalized in its logic and mathematical forms of computation. Cybernetics began as systems theories, from biological, cognitive, to computational, that were always interested in self-generating systems and the inseparability of interiority (i.e., thought or reason) and exteriority (i.e., body and the external world). This meant that the assumed separation between subject and object that logical positivism held in order to maintain objective measurement, cybernetics not only complicated but also sought to develop systems that traversed such dualisms, understanding the interconnectedness between them. In order to meet this aim, cybernetic theories of AI have long been based on recursivity (including human-in-the-loop) and learning through contingency.

The relationship between recursivity and contingency is understood to be central to the dynamic reconfiguration of complex systems, which is parallel to a dynamic form of technology. Drawing from cultural theorist of technology (Yuk Hui, 2019), the logic of technology can be theorized on the basis of the logic of recursivity. Because of a metaphysics that does not separate subject and object, interior and exterior, organic and inorganic, and culture and technology…, cybernetics holds that each of these … are mutually constitutive. As much as computation is not simply a tool, but a form of instrumentalized reason, it importantly brings forward the sensibility of a culture whose perceptual and cognitive matrix is transformed by and through technology. Moreover, recursivity enables this techno-cultural reciprocity insofar as the logic of feedback, adaptation, learning, and change is generated by their assemblage. Yet, what enables recursivity to change is its need for contingency or indeterminacies; thus, there can be no recursivity without contingency. Contingency is what conditions technocultural learning and change (Amaro, 2023; Byrd, 2011; Chun, 2022, Keeling, 2019). Both of these concepts bring a significant challenge and opportunity for the radical advancement of psychometrics.

AI and Psychometrics

Arguably, computational psychometrics began in 1958. Although psychometrics did not seek to design intelligent machines, it did inaugurate the quantification of human intelligence paving the way for later influences in how to model interior processes of perception. As Pasquinelli recently captures, it was from the statistical modeling of psychometrics that provided the key to the modeling of perception in the first machine learning algorithm, Frank Rosenblatt’s perceptron. Rosenblatt was informed by Weiner’s recursivity and McCulloch and Pitts logical mathematics of neurophysiological processes and sought to formalize this in an intelligent machine. Rosenblatt was a psychologist who had previously employed the multidimensional modeling of factor analysis to study personality profiles in his dissertation work. As an inductive method designed to analyze the proximity and distance of variables in variance/covariance structures, factor analysis became a plausible route for the pattern recognition of visual images.

Rosenblatt also drew from Claude Shannon’s (1948) information theory. As we learn from information theory, wave patterns and intensities carry information—messages communicated in various modes—across space, time, and matter. Information can be analog or digital and can be transmitted across a myriad of mediums. The transmission of information via a signal always contains a degree of noise. Although itself a signal, noise is any undesired interference or disturbance to the original signal. This means that the received message does not completely match the original message sent; there is a distortion, and the original signal cannot be reconstructed with certainty. It is opaque. As Claude Shannon formulates it mathematically, a received signal E = transmits signal S + noise N, where some degree of E is a function of S and N. This should look familiar to psychometrics as it is the same mathematical model of classical test theory: observed score = true score + error. Both models are based on theories of measurement in classical physics being applied to human social processes, that is, communication and psychology, and both models maintain modernity’s presupposition of the transcendental S or “true score.” I also think it was, in part, this mathematical analog where Rosenblatt made connections between the statistical model of factor analysis and Claude Shannon’s (1948) information theory.

What I want to draw our attention to is how in both models is the attempt to mathematically partition out and account for the opaque, the disturbance, the interference, the error, or the noise. While both psychometrics and the information theory of AI have substantially advanced since these initial models I present here, what is of most importance is that this is the point of their epistemological departure. While psychometrics maintains specters of logical empiricist tenets of evidence, where validity and reliability are a function of the model error, AI has long concerned itself with a focus on how to learn from noise. The relationship and interplay of recursivity and contingency is crucial to this process. It is through the recursive feedback loop where the model seeks to account for and learn from noise as contingency. This is a radically different epistemology of error or noise that I do not think can be easily reconciled or overlooked, as noise, error, contingency, or indeterminacy is where we find the ontoepistemologies of the dispossessed, the uprooted, the socially precarious, and the racially subjugated or subjected. Thus, not only do I want to push a speculative intervention by taking up AI’s epistemology of noise in psychometrics, what potentially might lead to a complex and dynamic system that could become more responsive to the multiplicities of situated knowledges and know-hows but also here is where I raise my concerns with Burstein’s (2023)Responsible AI Standards. For Burstein (2023), human expertise and the human-in-the-loop recursive design are important components to responsible AI practices. While I recognize the intention and even ethical practice that is being called for here, what I worry about is the “human” and its capacity to not police or discipline the boundaries of knowledges and know-hows.

Since the Enlightenment, the project of the “human” has long been a contested and fraught category. In fact, the human was formed based on a discourse of dualisms: interiority/exteriority, subject/object, included/excluded, and culture/nature. This also was inseparable from, and in many ways in the service of, the interest of colonial capitalism and racialized authority. The theological rationality of trans atlantic enslavement became reconfigured in the Age of Reason and its developments of science, technology, and political philosophy. While in other work I do go after the ways in which colonial reason can be traced to technoscientific practices of today, my concern here is how the “human expertise” of the human-in-the-loop becomes overdetermined, possessing great authority, and managing and preserving the hegemony of knowledge and know-how. I am not talking about a politics of inclusion that will not challenge and invariably will reify the overdetermined logic of the “human expert,” rather I am questioning to what extent we can trust the human. The “human” has committed significant violence in the world and under the authorized authority of “expert.” In the human-in-the-loop recursive model, the gaze on the expected outcomes will ultimately manage and preserve an overdetermined construct validity while violently negating those who are other-wise. What is needed are articulated practices, processes, and logics of what is in the loop, ones that are not averse to difference as actualized in noise as contingency but rather seeking to learn from the indeterminacies of xeno-, alien, or negative ways of knowing and know-how. In other words, what is needed are a set of AI standards that are response-able to the epistemologies of the uprooted and dispossessed, the socially precarious and subjected, and will bring transformations to the logic of the system.