Psychometrics versus Representational Theory of Measurement

2.1. Representational Theory of Measurement

According to RTM, measurement involves “the construction of homomorphisms (scales) from empirical relational structures of interest to numerical structures that are useful” (Krantz et al. 1971, 9; henceforth, FOM, from Foundations of Measurement). Homomorphisms are many-to-one mappings, and in RTM, these mappings are from the empirical relational structures to numerical ones. To measure, one needs to prove two types of theorems. First, one needs a representation theorem, which establishes that if a given empirical relational structure of interest satisfies certain (non-contradictory) axioms, then a homomorphism φ to a certain numerical structure can be established. Second, a uniqueness theorem establishes the permissible transformations of φ that also yield a homomorphism to the same numerical structure. Usually one distinguishes between four types of homomorphisms, that is, scales: ratio, interval, ordinal, and nominal. Ordinal scales, such as IQ, allow monotonic increasing transformations of the form ϕ → f ( ϕ ) . Interval scales, for example, temperature measured in Celsius or Fahrenheit, are such that they represent equality and inequality of intervals of the target attribute. For such scales, the permissible transformations are of the form ϕ → α ϕ + b , α > 0 . Ratio scales, such as length and weight, represent equality and inequality of intervals and have a non-arbitrary zero point. They allow for multiplicative transformation of the form ϕ → α ϕ , α > 0 .

In the RTM approach, measurement is based on empirical (relational) structures. In order to measure, we have to investigate empirically the relations between targeted objects, and establish that the empirical structure of interest satisfies the axioms that guarantee the existence of the mapping from the empirical structure to the numerical one. For example, the conditions that an empirical structure has to fulfill in order for it to be meaningfully represented on an ordinal scale are as follows:

Let A be a set of objects, and ≽ a binary relation on A. The relational structure (≽,A) can be meaningfully represented on an ordinal scale, iff for all a , b , c ∈ A ,

For example, Set A of objects denotes a set of commodity bundles, and the relation ≽ denotes a preference relation, that is, a ≽ b is interpreted as a is at least as preferred as b . If the testing of preferences reveals that the empirical relation ≽ satisfies connectedness and transitivity, then one can prove a representation theorem: there is a function ϕ from A to the set of real numbers such that for all commodity bundles a and b in A, a ≽ b iff ϕ ( a ) ≥ ϕ ( b ) , that is, in informal terms, the preference relation ≽ holds between a and b if and only if the number associated with a is greater than or equal to the number associated to b . Another function ϕ ′ has the same property and thus constitutes a homomorphism to the same numerical structure as ϕ if there is a strictly increasing function f such that for all a in A, ϕ ′ ( a ) = f [ ϕ ( a ) ] . In informal terms, in this case ϕ ′ is a permissible transformation of ϕ as long as it preserves the order of the numbers assigned to the objects.

2.2. Psychometric Validation

The problem with describing psychometric validation is that the concept has several meanings in contemporary methodological literature as well as in practice (Markus and Borsboom 2013). Here, I shall operate with a focus on reliability and so-called construct validity, because that is how Angner (2011, 2013) describes psychometric validation, and because it is a prominent way to go about psychometric validation.

On the psychometric approach, you start off by characterizing the target construct, that is, the latent variable of interest, such as well-being or intelligence, and by proposing a measure (usually in the form of a questionnaire) of that construct. You then administer the test and run a series of statistical tests on the response data to check whether the measure is reliable and has construct validity. Reliability amounts to the testing of the stability and consistency of the results the measure yields. There is a multitude of ways of doing this in practice, but it is common to check whether the test yields the same (or reasonably similar) result for a test taker when she or he takes it on another occasion (test-retest reliability) and to check whether the individual test items correlate with each other to a sufficient degree (internal consistency reliability) (Angner 2011, 128; Kline 1998, 29-30). ²

Construct validation, on the other hand, is thought of as the test of the degree to which the measure captures the construct it is supposed to capture (Angner 2011; Kline 1998). In line with Cronbach and Meehl’s (1955) seminal characterization of the process, the researcher should begin construct validation by formulating theoretical expectations of how the target construct relates to other constructs and measures, and then proceed to check whether the expected associations between these measures do indeed emerge. For example, suppose that our target construct, the unobservable latent variable, is well-being, and we have devised a questionnaire that purportedly captures this variable. If we have a theory that links well-being with mental health, then the construct validity of the new measure can be (partly) investigated by checking whether the purported well-being measure correlates to a sufficient degree with relevant measures of mental health. ³

Consider, as an example, the famous Satisfaction with Life Scale (SWLS) that was devised to capture a particular target concept, that is, unobservable latent variable: subjective well-being. SWLS consists of five questionnaire items, and asks subjects to rate their agreement with each of the items on a scale from 1 to 7. According to its authors Diener and colleagues (1985), the measure was validated, that is, it was shown to capture the target construct subjective well-being, when researchers compared responses on the SWLS to responses on other existing measures of subjective well-being and related constructs such as affect intensity, happiness, and mental health. The results confirmed their expectation that SWLS scores correlate highly with those measures that also elicit a judgment on subjective well-being, but less so with measures that are intended to capture other related but distinct notions.

2.3. Radically Different Approaches

Angner (2011, 123) claims that RTM and psychometrics are “radically different.” It is hard to disagree, given that RTM focuses on proving theorems while psychometrics focuses on relationships between different measurement instruments—two very different kinds of activities. One way to interpret this radical difference is to say that the two approaches deal with different, independently sufficient conditions for measurement. While Angner does not describe the two measurement approaches in terms of necessary and sufficient conditions of measurement, it is clear from the way he writes that he treats them as independent and self-contained approaches, that is, if you have one you do not need the other. For example, he argues (Angner 2011, 147) that under some circumstances in which RTM is inapplicable, psychometrics is “the only game in town” and therefore the only option for those who are keen on measurement. I take this claim to manifest the assumption that the two approaches are independent and self-contained. Such an interpretation is admittedly tempting in the context in which Angner makes his inconsistency claim. Angner (2008, 2009, 2011, 2013) argues that well-being researchers from economics and psychology have tended to rely on different approaches: orthodox welfare economists have often relied on RTM, whereas psychologists (and some heterodox economists) have validated their measures in terms of the psychometric approach. If that is indeed the case, it can be taken as evidence that in practice psychometrics and RTM are treated as two different full-fledged ways to go about measurement.

I believe there is a more fruitful way to think about the radical difference between RTM and psychometrics: they are partial approaches that deal with different non-sufficient but necessary aspects of measurement. ⁴ I call these aspects representational interpretability and procedural validity. These aspects are in many ways intertwined in practice; in particular, when procedural validation is done with extreme care, the result is the fulfillment of the condition of representational interpretability. But the analytic distinction between the two aspects is nonetheless helpful for understanding what RTM and psychometrics can and cannot do. I argue that RTM focuses on representational interpretability but is silent about procedural validity, while the reverse is true for psychometrics.

3.1. RTM and Representational Interpretability

Measurement is widely and almost invariably considered to involve numerical representation. The need for representational interpretability arises from the further observation that when it comes to measurement, not all numerical representations of empirical properties are created equal. The requirement of representational interpretability reflects the intuition that given certain empirical relations, some numerical representations are more appropriate than others to represent these relations, in the sense that they are appropriately interpretable in terms of the target system. But is representational interpretability a necessary aspect of measurement?

Consider a simple example. Lena has transitive strict preferences over slices of cakes: Black Forest ≻ Sacher ≻ Baked Alaska. If any numerical assignment would do, we could assign numbers to cakes as follows: Black Forest would be represented by −1, Sacher by 100, and Baked Alaska by 50. But assigning 3 to Black Forest, 2 to Sacher, and 1 to Baked Alaska is informative about an interesting property of Lena’s preferences, namely, order, which the former assignment fails to account for. If you agree that an adequate approach to measurement should be able to weed out the former assignment because it does not lend itself to a meaningful interpretation of the target system (preferences), you should agree that some kind of representational interpretability is crucial for measurement. That measurement requires representational interpretability may even seem obvious to you. But existing measurement practices show that its importance is not always recognized: psychometricians are often accused of arbitrary and uninterpretable numerical representation of their target systems (FOM, 33; Section 3.3 below). ⁵ It is therefore important to consider what representational interpretability amounts to, and how RTM helps attain it.

What exactly does it take for a numerical representation to be interpretable in terms of the targeted empirical system? At the very least, a criterion of interpretability should weed out arbitrary numerical assignments. To achieve this, we could take our cue from S.S. Stevens’ definition of measurement (e.g., Stevens 1975) and assign numbers to processes according to a rule. The trouble is that we need to specify the concept of a rule for it to get rid off arbitrary assignments. Stevens does not provide much help here, for he states that “[t]he only rule not allowed would be random assignment” (p. 47). But that just begs the crucial question, namely, how should we specify the notion of a rule so that random assignments are excluded. Stevens also speaks about the importance of matching operations as the basis of measurement, for example, when people match numbers to sensations. But it is legitimate to compare the informativeness of numerical representations that different matching operations yield, suggesting that there is more to the interpretability of a numerical representation than just that it results from matching. (One could, after all, ask people to match numbers −1, 100, and 50 to objects so that −1 is assigned to the most preferred, 100 to second best option, and 50 to the least preferred one.)

There is another criterion for representational interpretability that is well-defined, intuitive and discriminates between alternative representations: a numerical structure has to mirror (or map onto) the empirical structure it is supposed to represent in order for it to be a useful representation. ⁶ This criterion immediately gets rid of the previous troublesome representation, because the suggested assignment of −1, 100, and 50 does not mirror the relevant empirical structure, namely, order. Here’s another example: if we have four rods a , b , c , and d such that when they are set side by side, the difference between the length of a and b is equal to that between c and d , a useful numerical representation mirrors this, so that ϕ ( a ) − ϕ ( b ) = ϕ ( c ) − ϕ ( d ) , and so on for other relational structures. Representations that capture these mirroring relations are intuitive and useful exactly because they tell us how to interpret the numerical structure, and arithmetical operations on the assigned numbers, in terms of the target objects and their relations, when these objects and relations are examined in light of a given attribute. Thus the notion of mirroring grounds representational interpretability better than the other candidates, that is, rules and matching.

RTM builds on precisely this conception of representational interpretability, and studies meticulously the conditions under which such mirrorings or mappings can be said to hold. The whole point of the representation theorem is to establish the conditions under which a given relation between numbers that are used to represent the target system has a parallel relation in the realm of the objects, so that a given empirical relation exists between objects if and only if the numbers assigned to those objects have the corresponding relation to each other. The uniqueness theorem, in turn, establishes how the numbers in the numerical representation can be transformed without breaking the mapping between the empirical relations and the numerical ones. As indicated by the complexity of some of the axiomatic systems considered in FOM, as well as the serious intellectual effort that goes into proving the representation and uniqueness theorems, the conditions for the existence of such appropriately interpretable representations are not at all self-evident. RTM thus provides the conditions under which there is a rationale for a specific kind of numerical representation of an empirical structure and the transformations that preserve a representation of that structure. In doing so it deals with a crucial aspect of measurement: representational interpretability.

3.2. Psychometrics and Procedural Validity

The problem of validity of procedures stems from the observation that there is often a discrepancy between our best characterizations of the concept we want to measure and the reach of the empirical procedures that we want to use as tools for capturing that concept. This is because a target concept, that is, the aspect of an empirical system that we want to study, such as temperature, happiness, or intelligence, has meaning independent of the procedures that are supposed to capture that concept. Although operationalism has been offered as a measurement strategy in the past, it is widely regarded as inadequate to simply define the target concept in terms of a procedure by stipulation. Recognizing that (many) concepts are not defined in terms of procedures gives rise to the problem of procedural validity: how do we know that a suggested procedure captures the concept we want it to capture, and how do we know it tracks empirical manifestations of this concept reliably across conditions. I take it to be evident that these are questions that a full-fledged approach to measurement should be able to address. It is hard to imagine measurement without adequate procedures.

In recent historico-philosophical literature on measurement, the proposed solution to the difficulty in validating procedures has been appeal to coherence (e.g., Chang 2004; van Fraassen 2008). The idea is, roughly, that claims about the appropriateness of a given procedure for the measurement of a given concept require multiple determinations of the same concept via different fallible methods, or multiple determinations of related concepts via different methods. If the different determinations agree with each other, that is evidence in favor of the assumption that the procedures indeed capture the target concept. In other words, the hypothesis that a measure is capturing what it is intended to capture gains evidence from the fact that multiple determinations of that construct (or theoretically related constructs) cohere with each other.

It is easy to see that the psychometric approach, in particular the strategy of construct validation, is an instance of a coherentist approach to the validation of a measurement procedure (Alexandrova and Haybron 2016). Construct validation starts from a web of theoretical assumptions that entail that a proposed measure of the target concept correlates with certain other measures (and across contexts). The hypothesis is tested by checking whether such correlations emerge, and if they do, that is taken as evidence for the claim that the proposed measure indeed captures the correct construct. In other words, the claim that the proposed measurement instrument captures the correct construct is supported by appeal to coherence between relevant measures and theoretical expectations. Under the above characterization of coherentism in measurement, construct validation counts as a coherentist solution to the problem of validity of procedures.

3.3. Limits

We have seen that RTM deals with representational interpretability and psychometrics deals with procedural validity. Let me now discuss how each of these approaches is silent about the task that the other one focuses on. Start with RTM and procedural validity. Many scholars think that RTM is unhelpful when it comes to finding an appropriate measurement procedure (Boumans 2005; Reiss 2008). Julian Reiss (2008, 67) puts the point as follows:

[RTM] tells us what kind of structure an attribute or a phenomenon must have in order to be measurable given we have a reliable measurement instrument, but it does not tell us where and how to look for a reliable instrument in the first place.

Reading through the authorative statement of RTM supports this observation: FOM says virtually nothing about procedures. Even when the authors consider empirical examples of applications of the axiomatic conditions that FOM explores (e.g., transitivity), they do not identify how the applicability of a certain axiomatization in a given empirical context can be established. ⁷ As the authors of FOM themselves note, their empirical examples have the modest role of “motivating” and “illuminating” the axiomatic foundations the book is primarily concerned with (FOM, xvii).

To get a clearer grasp of this, consider the paradigmatic example of RTM: measurement of length with rigid rods. Even this case does not tell enough about procedures to guarantee that the axioms apply. This is because simple observations of differences between lengths of rigid rods cannot be considered reliable, when these differences are extremely small. We need more subtle procedures to state that the axiomatic conditions hold in extreme circumstances. Similarly for another seemingly simple case of RTM, namely, weight. Suppose we place two objects on an equal-arm pan balance and the arms remain horizontal. How do we judge that the two objects are indeed equal in weight, rather than that our balance is broken or simply not sensitive enough to detect the difference between the two? Although the first chapter of FOM mentions some of these issues, the axiomatizations that the rest of the three volumes deal with do not solve, or purport to solve them. Note that none of this is to say that tracking the axiomatic conditions that RTM lays down does not require procedures but rather that RTM only gives advice on what the relevant conditions are, not how they can be captured by means of measurement procedures. RTM can hardly be an approach to validating procedures if it is virtually silent about procedures.

How does the psychometric approach deal with representational interpretability? Several authors have noted that in psychometrics, the appropriateness of a given type of numerical representation (usually interval level) is often assumed rather than established (Borsboom and Zand Scholten 2008; see also Hobart et al. 2007; Kristoffersen 2010). This points to neglect for the question of representational interpretability. Such neglect is implicit in practices where subjects are asked to rate their standing on some attribute (e.g., well-being), and these scores ⁸ are taken to constitute an interval scale of the target attribute, implying (by definition of interval scale) that differences between the assigned numbers mirror equality and inequality of distances between objects on the measured attribute. But it is not trivial that differences between the assigned numbers represent distances between objects on the measured attribute. Subjects may or may not be using the rating instrument so that this assumption is fulfilled, and usually they are not (Hobart et al. 2007). By taking the scores as interval-level measurements, psychometricians assume that the emerging numerical representation mirrors certain relations between the objects of study when they are compared in terms of the target attribute. Thus they operate under the assumption that the numbers have a mirroring relation to manifestations of the underlying attribute, but neglect rigorous study of the conditions for the existence of such mirroring relations, and empirical testing of whether or not those conditions are fulfilled by the target system.

The assumption concerning interval-level measurement is left unsubstantiated, because the validation techniques described in our specification of the psychometric approach (Section 2.2) are not apt for ensuring evidence of representational interpretability. Test-retest reliability and internal consistency reliability give us evidence of how test results correlate across time and how individual items correlate with each other, but they do not tell us anything about how we should interpret equalities and inequalities in manifest scores in terms of the latent attribute. Similarly, while construct validation tells us that the proposed measurement procedure in some sense captures the correct construct (rather than a related one), it does not tell us whether the emerging numerical representation is interpretable in the sense that the appropriate mirroring relation exists, at least when it comes to interval and ratio scales. ⁹ Knowing that two measures are related in some way and to some extent does not (and is not meant to) ensure that either measure yields interval-level measurement. In fact there is a substantial psychometric literature (starting with Stevens 1951) that argues that interval-level measurement is a precondition for a meaningful interpretation of many of the statistical tests that are commonplace in the construct validation exercise. (About the permissibility of statistical tests, see Luce 1959; Stevens 1951 cf. Hobart et al. 2007; Kristoffersen 2010.) The psychometric approach therefore builds on assumptions about representational interpretability but does not help establish the truth of those assumptions.

3.4. Consistency, Finally

We have seen that RTM gives guidance on representational interpretability but is silent about procedures, while the psychometric method of construct validation gives a coherentist response to procedural validity but is silent about conditions for representational interpretability. I have also argued that both representational interpretability and procedural validity are crucial aspects of measurement. Thus we can conclude that RTM and construct validation are only partial approaches to measurement.

This reveals that there is in principle no reason to believe that “the simultaneous endorsement of the two approaches to measurement would lead to inconsistency” as Angner (2011) claims. To say that a measure has been validated in terms of psychometrics but not in terms of RTM does not mean that the measure both is and is not validated. It means that one condition for full-fledged measurement has been satisfied via psychometrics and that another aspect has either not been studied or has been appropriately dealt with relying on some other approach than RTM. Furthermore, the fact that the psychometric approach deals with non-sufficient conditions of measurement means that the psychometric approach does not entail that the axiomatic conditions RTM outlines are not necessary for measurement. Rather, the psychometric approach is just silent about an aspect of measurement that RTM deals with. It is therefore consistent to simultaneously endorse psychometrics and RTM.