The Legacies of R. A. Fisher and K. Pearson in the Application of the Polytomous Rasch Model for Assessing the Empirical Ordering of Categories

Abstract

Assessments in response formats with ordered categories are ubiquitous in the social and health sciences. Although the assumption that the ordering of the categories is working as intended is central to any interpretation that arises from such assessments, testing that this assumption is valid is not standard in psychometrics. This is surprising given that it has been known for some 35 years that this assumption can be checked routinely using the psychometric Rasch model for more than two ordered categories. The purpose of this article is twofold. First, to demonstrate three distinct but related legacies of R. A. Fisher that have contributed to the use of the Rasch model to assess the empirical ordering of categories: (a) his construction of sufficient statistics, (b) his recognition that the ordering of categories should be an empirical property of the data, and (c) his integration of the design of empirical studies with statistical analyses of data. Second, to suggest two reasons behind both the indifference, and even the rejection, of both the need and possibility of testing the assumption of the empirical ordering of categories: (a) the lack of recognition of the problem before it was understood that it could be solved using the Rasch model and (b) the legacy of K. Pearson that legitimized the atheoretical modeling of data with parameters that have no substantive meaning.

Keywords

polytomous Rasch model rating scale partial credit ordered category formats empirical category order

Introduction and Background

Assessments in response formats with ordered categories, formats such as ratings and partial credit (R&PC), are ubiquitous in the social and health sciences. Central to such formats is the assumption that the successive categories imply increasing amounts of the property that is being assessed. If this assumption is satisfied, it validates the operation of the response format as an expression of increasing amounts of the property; otherwise, it invalidates the operation of the format. Despite its importance, testing the assumption that the empirical ordering of the categories in data analyzed is working as intended has not been a standard part of psychometric analyses of R&PC data. Not only has it not been standard practice, but since the demonstration that the polytomous Rasch model (PRM) for ordered categories can be used to check the empirical ordering of categories in R&PC format routinely (Andrich, 1978, 1979), reactions have included both indifference (e.g., Embretson & Reise, 2000) and rejection and controversy (e.g., Adams, Wu, & Wilson, 2012).

The specific contentions of Adams et al. (2012) have been rebutted in Andrich (2013). However, because the rationale for assessing the empirical ordering of categories using the PRM is mathematical and not a matter of opinion, it seems instructive to try to understand the broader context that engenders such indifference or rejection. The purpose of this article is to contribute to this understanding.

In anticipation, two factors are suggested for this purpose. First, because the solution to the problem of testing that the ordering of categories is an empirical property of the responses using the PRM was discovered before the problem was recognized, the problem was not in the public domain and therefore was not considered relevant. The recognition of the problem after it is solved, leads to showing three specific overlapping roles of R. A. Fisher, not only on his general impact on Rasch’s work but also on the specific development and interpretation of the PRM for ordered categories. These roles include (a) the specific application of Fisher’s formulation of sufficiency, (b) Fisher’s awareness of the need that ordered categories are operating as required empirically, and (c) his integration of empirical and experimental design with statistical analyses.

Second, because the typical statistical test of fit between the data and the model, which is irrelevant to testing the empirical ordering of the categories in the data, is so pervasive in psychometrics, it has been invoked to ignore the evidence of the empirical ordering of categories. The application of the test of fit as a justification for ignoring evidence of the violation of the empirical ordering of categories leads to the role of K. Pearson in statistical analyses and his influence in psychometrics. This role concerns Pearson’s justification for modeling data descriptively without apparent concern for the substantive and theoretical rationale behind the data, and by extension, the substantive interpretation of parameters in the model used to model the data.

The rest of the article is structured as follows. The next section summarizes the mathematical result central to the PRM and its implication for the empirical ordering of categories. The subsequent section summarizes the development of the PRM in its current form, highlighting Fisher’s identification of sufficient statistics and the solution of the problem of testing the empirical ordering of categories before this problem was recognized. This is followed by a section that introduces Fisher’s drawing attention to the idea that the ordering of categories is an empirical matter. This in turn is followed by a section that emphasizes Fisher’s integration of statistical analysis and experimental design, and the legacy it has had in the argument that the empirical ordering of categories could and should be assessed using the PRM. Next is a section that describes the initial reactions to publication of the PRM. It includes Pearson’s legacy of finding models to describe data that has led many current researchers to either ignore or reject the argument that the empirical ordering of categories could and should be assessed using the PRM. The penultimate section summarizes the current status of the assessment of the empirical ordering of categories, and the final section is a conclusion.

Summary of the Mathematics of the Polytomous Rasch Model and the Empirical Ordering of Categories

Rasch’s simple logistic model (SLM) for a response in one of two ordered categories can be expressed in the form

P_{nix} = P {X_{ni} = x; β_{n}, δ_{i}} = [\exp x (β_{n} - δ_{i})] / γ_{ni},

where $x \in {0, 1}$ , β_n and δ_i are, respectively, the locations of person n and item i on an implied continuum, and $γ_{ni} = \sum_{k = 0}^{1} \exp k (β_{n} - δ_{i})$ . In proficiency assessment, x = 1 characterizes the successful response and β_n and δ_i are referred to, respectively, as proficiency and difficulty. In this article, δ_i is referred to as a threshold of item i—it is the point on the continuum at which β_n = δ_i, that is, where the probability of each response is .50. Figure 1 shows the conceptualization of two ordered categories on a latent continuum defined generically as Fail (that is, Not a Pass) and Pass, where $δ_{i} = δ_{\tilde{P} / P}$ .

Figure 1.

Response format for two ordered categories and the dichotomization of the latent continuum

The most convenient generalization of Equation (1) to m_i+ 1 ordered categories for this article (Andrich, 1978; Wright & Masters, 1982) is

P_{nix} = P {X_{ni} = x; m_{i}, β_{n}, (ψ_{i})} = [\exp (ψ_{ix} + x β_{n})] / γ_{ni},

where $x \in {0, 1, 2, \dots, m_{i}}$ is an integer variable; β_n remains the location of person n; (ψ_i) is an m_i+ 1 vector of category coefficients defined by

ψ_{ix} = - \sum_{k = 0}^{x} δ_{ik},

where $(δ_{i}) = δ_{ik}, (k = 1, 2, \dots, m_{i})$ is a vector of m_i ordered thresholds that partitions the continuum of item i into m_i+ 1 categories, where for notational convenience, $δ_{i 0} \equiv 0$ ; and where $γ_{ni} = \sum_{k = 0}^{m_{i}} [\exp (ψ_{ki} + k β_{n})]$ . In a dichotomous item $m_{i} = 1$ and the only threshold is $δ_{i 1} = δ_{i}$ .

It is stressed that the distribution in Equation (2) is not of individuals among the categories of an item but that of the response of a single person to an item. Accordingly, unless required because multiple persons and items are considered, the person and item subscripts n and i are dropped in the rest of the article.

Figure 2 shows the conceptualization of four ordered categories on a latent continuum defined generically as Fail, Pass, Credit, and Distinction, where the intended ordering is obvious and where $δ_{1} = δ_{\tilde{P} / P}, δ_{2} = δ_{\tilde{C} / C}, δ_{3} = δ_{\tilde{D} / D}$ .

Figure 2.

Response format for four ordered categories and the successive dichotomization of the latent continuum

From Equation (2), the conditional probability

Pr {X_{n} = x; m, β, δ_{x} | (X_{n} = x ​ or X_{n} = x - 1)} = P_{x} / (P_{x} + P_{(x - 1)}) = [\exp (β - δ_{k})] / γ_{n x}; k = x,

is once again the SLM with each threshold δ_k, k = x referenced to pairs of successive categories. Given the intended ordering of the categories, the implied responses x and x− 1 in Equation (4) are deemed, respectively, the successful and unsuccessful responses at threshold δ_k, k = x. Although Equation (4) expresses a dichotomous response, the response is latent and inferred, not observed, and as evident in Equation (2), the probability of the observed response in a category is a function of all thresholds.

The Polytomous Rasch Model, the Simple Logistic Model, and the Rationale of Threshold Order

The PRM can be applied to data from items with a range of formats. Here we are concerned only with the case where there is a single response in an R&PC format in the assessment of proficiency, and ratings in the assessment of attitude, for example, Likert (1932) scales.

The thresholds in Figure 2 are analogous to marks on a standard ruler that indicate centimeter locations from an origin. There are, however, five differences. First, the number of categories is finite; second, the successive thresholds must be estimated from data and cannot be assumed equidistant; third, the thresholds do not have a natural, substantive origin; fourth, given the location of a person on the continuum, the response in any category is explicitly probabilistic.

The fifth difference, the core of the PRM, is that although the thresholds are required to be ordered, their empirical estimates from the data may take any order. Therefore, the threshold order in the data being analyzed is taken as a hypothesis to be tested. The relationship between the thresholds and the implication for the ordering of the categories is described in detail, with examples and simulations, in Andrich (1979, 2013) and only a summary of the interpretation, relevant for the present article, is presented here.

For assessments on one item with m+ 1 ordered categories, and arising from the identity of Equations (1) and (4), the relevant mathematical result is the following:

From responses to an ordered category item, the PRM estimates the same thresholds as the SLM estimates from a set of compatible, experimentally independent, dichotomous responses made at the same thresholds.

Thus, a single judge making an assessment in one of m+ 1 ordered categories, and the assessments analyzed according to the PRM, is equivalent to having m independent judges making compatible, experimentally independent assessments of success or failure at the same thresholds, and analyzed by the SLM. This result is proved and illustrated in Andrich (2013).

The relevant inference is the following

If responses at the thresholds were experimentally independent and analyzed according to the SLM, then it would be required that the threshold estimates showed increasing difficulty, that is, in Figure 2, that ${\hat{δ}}_{\tilde{P} / P} < {\hat{δ}}_{\tilde{C} / C} < {\hat{δ}}_{\tilde{D} / D}$ . Therefore, because a PRM analysis of responses in ordered categories provides estimates of the same thresholds as an SLM analysis of compatible, experimentally independent dichotomous responses, the threshold estimates from the PRM are also required to be ordered. Otherwise, the ordering of the categories is not operating empirically as required.

To elaborate the above inference, for experimentally independent dichotomous responses of success or failure at each threshold, the ordering $δ_{\tilde{P} / P} < δ_{\tilde{C} / C} < δ_{\tilde{D} / D}$ is specified a priori. However, with experimentally independent decisions, there is nothing to force the threshold estimates to be in this or any other order; therefore, their order is a property of the data and the estimated order is a test of the hypothesis that ${\hat{δ}}_{\tilde{P} / P} < {\hat{δ}}_{\tilde{C} / C} < {\hat{δ}}_{\tilde{D} / D}$ . Then even if the data fit the SLM according to traditional criteria, if the required ordering is not present in the data analyzed, the hypothesis of empirical order is rejected. The implication is that there is an anomaly in the responses, perhaps resulting from the definition of the performances at the thresholds, or some such reason. The analysis provides no explanation as to the source of the anomaly (Andrich, 2004; Kuhn, 1961)—the anomaly would need to be addressed substantively and qualitatively and corrected empirically. Moreover, if the data already fitted the SLM, it would be irrelevant to search for a different model.

Therefore, because of the relationship between a PRM analysis of responses in ordered categories and an SLM analysis of compatible, independent dichotomous responses at thresholds summarized above, the threshold estimates from the PRM are also required to be ordered. And again, if they are not ordered, an anomaly is revealed that needs to be addressed substantively and qualitatively and corrected empirically. The anomaly in the data cannot be corrected simply by choosing a different model for the responses.

It is the above inference and implication that is the source of indifference and controversy.

Figure 3 shows the PRM response probabilities of four ordered categories where (a) the thresholds are ordered and (b) a pair of thresholds is disordered. The graphs also show the latent, dichotomous probability curves of Equation (2). Two anomalies of disordered thresholds, not present in the top graph, are evident in the bottom graph of Figure 3. First, there is no point on the continuum at which the response of Credit is more likely than responses in its adjacent categories Pass and Distinction. Even where Credit is most likely, Pass and Distinction on either side are simultaneously more likely than Credit. Second, the threshold at Distinction is easier than the threshold at Credit.

Figure 3.

Category probability curves for ordered thresholds (top) and disordered thresholds ${\hat{δ}}_{\tilde{D} ID} < {\hat{δ}}_{\tilde{C} IC}$ (bottom)

Fisher’s Sufficient Statistics, the Identification of Disordered Threshold Estimates in the Polytomous Rasch Model, and Recognizing the Problem After It Is Solved

In interpreting the distinctive epistemological work of G. Rasch (1901-1980) in social measurement, including its controversial aspects, Andrich (2004) invoked two aspects of Kuhn’s (1961, 1970) work: first, Kuhn’s analysis of paradigms and intellectual revolutions and second, his articulation of the function of measurement in science. The first aspect included recognizing a problem after it is solved and controversy as part of resistance to the new paradigm. Recognizing a problem after it is solved featured in Rasch’s understanding of the class of models that lead to a Rasch measurement theory (Rasch, 1960). The class of models within Rasch measurement theory is based on the requirement of invariant comparisons, a requirement specified independently of any data set. Recognizing a problem after it is solved suggests that, because the problem solved is not immediately obvious, even to the researcher who solves it, it is not yet in the public domain. It also suggests that in attempting to place the solution to the problem in the public domain, the major challenge might be to communicate the existence of the problem in the first place.

The second aspect of Kuhn’s work invoked was his explanation that the role of measurement in science was to disclose anomalies, anomalies that are to be understood substantively from a qualitative analysis. In the application of Rasch models, misfit and other evidence against theoretical requirements were taken as anomalies that needed to be understood substantively from a qualitative analysis. Such an analysis would result not only in an improved instrument but also in a better understanding of the variable of measurement.

This article provides another illustration of both recognizing a problem after it is solved and implications for qualitative analyses with the disclosure of anomalies in relation to the PRM. The problem solved, the subject of this article, is having a necessary and sufficient mathematical condition for assessing routinely whether or not response categories of items structured to be ordered were operating as required in the data analyzed. Then if there is evidence to the contrary, an anomaly, qualitative analysis, and further experimentation are implied.

The PRM (Andrich,1978) was derived in 1975 and operationalized for analyzing contingency tables with a dependent, ordered category variable (Andrich, 1979), where, for the purpose of this article, no distinction is made between an ordered category response in a contingency table and a response to an item in a psychometric context. Disordered threshold estimates, observed in some data sets but not others, were not understood immediately. The understanding followed when the derivation of the model was related to Andersen (1977).¹ There Andersen began with sufficiency of a unidimensional person parameter in Rasch’s (1961) general form of the model for ordered categories, and studied its consequences. Thus, interpreting disordered threshold estimates in the PRM involved specifically Fisher’s concept of a sufficient statistic in relation to a hypothesized unidimensional variable. This is the first role of Fisher’s work for the purposes of this article.²

Rasch’s (1961) general form for the ordered category model can be expressed as

P_{nix} = P {X = x; m, β, (δ)} = [\exp (ψ_{x} + φ_{x} β)] / γ_{ni} .

Andersen (1977) established two conditions for the ϕ_x, known as scoring functions of the categories: first, for categories $x = 1, 2, 3, \dots, m$

φ_{x + 1} - φ_{x} = φ_{x} - φ_{x - 1};

second, frequencies in adjacent categories x and x− 1 could, and should, be combined to a single category only if $φ_{x} = φ_{x - 1}$ .

No interpretation of either ϕ_x or of the category parameter equivalent to ψ_x in Equation (5), was provided by either Rasch or Andersen. Andrich (1978) showed that (a) $δ_{k}, k = 1, 2, \dots, m$ were thresholds on the continuum as defined in Equation (3); (b) Equation (6) arises from equal discriminations at the thresholds from which ϕ_x = x; and (c) categories x and x− 1 could be combined only if the discrimination at the threshold between the categories is 0. This explanation for combining categories did not satisfy Jansen and Roskam (1986) and it, too, resulted in some controversy (Andrich, 1995b, 1995c; Roskam, 1995). From the perspective of the argument of the article, it is relevant to note that Rasch (1961), Andersen (1977), and Andrich (1978) did not have an empirical example. The derivations were made on theoretical grounds of invariance of comparisons reflected in a model with sufficient statistics for a unidimensional parameter. The argument was not based on describing any data set.

Recognizing the Problem After It Is Solved: The Interpretation of Disordered Thresholds

It was only in 1977, when the model derived in 1975 and operationalized for contingency table contexts (noted earlier) was identified with Andersen’s (1977) formalization of Equation (6) in the general Rasch model, that it became clear how the required ordering of the values of the thresholds was built into the model as a hypothesis that could be accepted or rejected in data, and not as a specification of the values that had to be ordered. Thus, the puzzle of disordered threshold estimates in some data sets was solved, and it became clear how a PRM analysis could provide a routine check on the empirical ordering of categories.

There are three reasons for recounting the above development. First, the structure of the PRM did not arise from any attempt to find a solution to the problem of assessing the empirical ordering of categories. Second, the problem and solution to the empirical ordering of categories were only understood some 2 years after the PRM was written and operationalized, and only after that form of the model was connected to Andersen’s work exploiting sufficiency of a unidimensional person parameter in Rasch’s (1961) general form of the model for ordered categories.

Third, and perhaps most important, the solution revealed that the empirical ordering of categories is a property of the data and that it could be a problem. The author, and presumably many other researchers at the time, had not known of this possibility.

Thus, in practice up to that time, and to a large degree still, the empirical ordering of categories was not considered something that could and should be checked. For example, texts on constructing questionnaires based on Likert’s (1932) principles do not deal with this issue. Likert did consider the weighting of the categories and concluded that integer scoring was satisfactory. The standard procedure for weighting categories, based on Thurstone’s work (Edwards & Thurstone, 1952), which used cumulative probabilities across the categories, did not deal with their empirical ordering. Dubois and Burns (1975) considered the meaning of a particular category designated “?” in bipolar Likert-style questionnaires with response categories Strongly Agree, Agree, “?”, Disagree, and Strongly Agree. Although they stressed that the relative meaning of categories needs to be studied rather than taken for granted as self-evident, they also concluded that the typical successive integer scoring was satisfactory. (As seen above, integer scoring is consistent with the PRM, though not because the categories as such are equidistant, but because discriminations at the thresholds are equal.) Furthermore, psychometric curves of the kind shown on the right in Figure 3 appear in both the statistical (e.g., Anderson, 1984) and psychometric (e.g., Embretson & Reise, 2000; Samejima, 1997) literature with no reference to such curves implying any problems in the category order.

It is suggested that one reason the problem and solution of the empirical ordering of categories generates controversy is that recognition of the problem would expose standard theory and practice as deficient in some way.

Fisher’s Understanding of the Relevance of the Empirical Ordering of Categories

In Andrich (1979), an example of a PRM analysis in a contingency table context with a dependent variable of assessment with four ordered categories showed disordered threshold estimates. On closer examination of the data, it was evident that the responses between the two middle categories were no more than random assignments (zero discrimination). Accordingly, from a perspective of modeling using the PRM, the responses in these two categories should be collapsed into one category. However, the lack of discrimination disclosed was a property of the data no matter how identified. Therefore, and more important from the perspective of the substantive variable being assessed, it was evident that judges could not distinguish between the middle two categories and that these classifications could be misleading. This suggested the need for further experimental work with the definitions of the categories. Although the model as such was part of a member of a class of models being developed with ordinal data (Goodman, 1981), showing how it could be used to investigate the empirical ordering of categories seemed new.

Andrich (1979) did receive some attention that is commented on later in the article, but the point regarding the empirical ordering of categories was received mostly with indifference. However, the article did provide a lead to a method that R. A. Fisher had already used for analyzing ordered categorical data, a method based on maximizing the variance between objects relative to the variance within them. This led to Fisher’s second role for the purposes of this article. Fisher’s comment on the analysis of a particular data set was that

It will be observed that the numerical values . . . lie . . . in the proper order of increasing reaction. This is not a consequence of the procedures by which they have been obtained, but a property of the data examined. (Fisher, 1958, p. 294)

The implication that the ordering (or disordering) of the relevant threshold estimates was relevant seems clear. A reanalysis of Fisher’s data with the PRM shows that its threshold estimates are also ordered (Andrich, 1996).

That Fisher’s statement above is effectively an aside to his example and his method does not diminish its power. In an interview, parts of which are reported in Rasch (1960) and Andersen and Wøhlk (2001), Rasch (1979) states,

. . . hidden in the derivation of the characteristic function of the probability distribution for the sufficient statistic when it exists, I found the conditions in a passage which was easily overlooked. He mentioned it just in passing. He pointed out effectively that when there are sufficient statistics then, of course, they arise from a distribution with some kind of exponential connotation. You know sufficient statistics are available when the distribution belongs to what was a little later called the exponential family. That I had realized, but he had just mentioned that in passing, between two commas in his derivation of what the characteristic function of sufficient statistic is, in the case that it does exist. Fisher was a remarkable mathematician.

The above quote from Rasch illustrates something apparently well known (Fisher Box, 1978)—that Fisher made relatively cryptic commentaries to his deep insights. It adds legitimacy to Fisher’s aside which implies that the intended category ordering may be violated in data, and that the empirical ordering of the categories should be a property of the data and not obscured by a property of the model by which they are analyzed. It will be shown that the standard approach to analyzing R&PC data uses a model in which the ordering of the categories is a property of the model, irrespective of any property of the data.

The Epistemology of R. A. Fisher

We now consider Rasch’s distinctive approach more closely, which, for the purpose of this article, involves R. A. Fisher’s third role.

Rasch and the Model–Data Relationship

Rasch raises the question whether, in the case of misfit between the data and the model, “. . . it is the model or the test that has gone wrong” (Rasch, 1960, p. 51). One example where it is taken that the data “has gone wrong” arose from his analysis of a test scored dichotomously using the SLM. Note that he had derived the SLM for dichotomous responses as a model with the property of sufficiency (Rasch, 1960) and that he did not derive it to describe any particular set of data.

Rasch then applied the SLM to responses to Raven’s (1940) nonverbal test of reasoning and to a Danish military intelligence test. He concluded that the Raven’s data fitted the SLM, but that the military test data did not: He had one success and one failure with his new model. However, instead of modifying the model to account for the data (in particular by adding a discrimination parameter in the first instance) he studied the patterns of misfit and concluded that the test seemed to be composed of four different kinds of items. As a result, the head of the military psychologists had four tests constructed, one for each class of items, where each test would conform to Rasch’s new model. Thus, it was the data from set of intelligence items that was seen to fail, not the model. More important, and distinctively, however, this failure led to further experimentation in test development leading to the improvement in the assessment of the intended proficiencies, rather than to the application of models with more parameters which would have better modeled the original data. The test was required to be substantively valid, but in addition, and contributing to the validity, it was required to conform to the model, that is, conform to invariant comparisons of items and persons.

Further examples of concern with experimentation, rather than model modification, can be found in Rasch (1960). Such experimentation does not preclude using different models that are germane to the experimental conditions and results, but the class of model would still be one with sufficient statistics that operationalized invariance of comparisons. In a concluding statement to these investigations, Rasch writes,

Altogether these experiences—limited as they are to intelligence tests and attainment tests—suggest that once items have been constructed with an eye to uniformity of content, but variance in difficulty – which may even cover “complexity”—then there is a fair chance that they on the whole fit well into the model of simple conformity. (Rasch, 1960, p. 125)

The theme of Rasch’s (1960) exposition is explanation from empirical studies designed to conform to a model that characterises invariant comparisons, a model, which in turn has properties compatible with those of measuring instruments constructed in the natural sciences. The same theme is present in Rasch (1977). Wøhlk Olsen (2003) summarizes Rasch’s contribution to statistics in Denmark, illustrating the range of substantive areas of his application of models, which if conformed to, provide invariant comparisons.

Rasch’s emphasis on substantive explanation and experimentation, rather than finding a model to describe the data, is compatible within the epistemology of R. A. Fisher. As will be seen, this contrasts with the epistemology of K. Pearson.

Fisher and the Relation Between Research Design and Statistical Analysis

Although making unique contributions to statistical theory, Fisher’s appointment was as Professor of Eugenics at University College London after Pearson’s departure, not of statistics. Thus, Fisher was not only a statistician, but he was also a substantive researcher. He combined both roles in his own work and provided extensive support to other substantive researchers (Fisher Box, 1978). His contribution here is characterized by the publications Statistical Methods for Research Workers (Fisher, 1925, 1934a) and The Design of Experiments (Fisher, 1935), both of which had many new and expanded editions.

The interaction of observation, experimentation, statistical analysis and explanation seems to be exemplified by Fisher’s attitude to the use of mice in genetic studies.

Fisher had a great faith in the sort of intimacy that daily observation and handling forged between mice and men; it gave the investigator constantly renewed opportunities to notice new facts, . . . When new styles of cage were devised which allowed of the rearing of mice “untouched by human hand,” he resisted the idea as being quite unsuitable for genetic research. (Fisher Box, 1978, p. 174)

For the purpose of this article, MacKenzie (1981) summarizes the distinctive role of Fisher expressed in the Statistical Methods for Research Workers:

. . . the book incorporated an effectively new concept of the statistician’s role, and therefore a new function for statistical theory. The message was that the statistician should get involved in the practical business of experimentation. . . . It was not even enough that the scientist should hand their results to the statistician for analysis: experiments (especially large-scale applied experiments that were difficult to “control”) had to be designed by those with statistical expertise. (pp. 212-213)

This “new concept of the statistician’s role” will be shown in the next section to be in sharp contrast to the role established by K. Pearson, which was finding models to describe data free from substantive, theoretical considerations and complementary experimentation and substantive explanation.

Before he developed the class of models that now goes under his name, as a mathematician, Rasch had been asked to help with the analysis of data sets in the medical and biological areas. Thus he already had an opportunity to work with substantive researchers. To improve his skills and knowledge in statistics, he was subsequently supported to study with R. A. Fisher at University College London (Rasch, 1960). Studying with Fisher would have at least reinforced and further justified working with substantive researchers. In particular, Rasch was involved in the design of the experiments in reading which lead to his recognition (a) that the idea of invariant comparisons was a realization of measurement and (b) that in a probabilistic context, this invariance can be characterized by models with sufficient statistics (Rasch, 1960, 1961). It is evident in Rasch (1960), where he interpreted problems with fit as matters to be studied experimentally, and where his analyses could contribute to the design of the studies, that Rasch’s approach was consistent with, and arguably influenced by, the epistemology of R. A. Fisher.

Influence of Rasch on Problematizing the Ordering of Categories

In the above section it was noted that Rasch had made a case for a class of models for measurement based on the requirement of invariant comparisons, and that if the responses did not fit this class of model, there is a problem with the data, not the model. In categories intended to be ordered, there may be a problem with the ordering even when data do fit the model. However, specifying a priori that the responses should fit a particular, relevant model, and in addition requiring that threshold estimates be ordered, is an extension of the same principle—namely that the data should meet a priori specifications.

The article now considers the influences that led to recognizing the opportunity the PRM gives for the study of the empirical ordering of categories.

As part of analyzing the contingency table data (Andrich, 1979), the author had programmed in 1975, not only the model that became the PRM described above but also the model explicated by McCullagh (1980) and Bock (1975, Chap. 8),³ which is summarized shortly and which obviates any concern with the empirical ordering of categories. As indicated earlier, it was not for a further 2 years, when the author synthesized his work with that of Andersen, that the understanding of the implications of disordered thresholds became clear. The question is why did the author not continue with the alternate model and ignore the opportunity for studying the empirical ordering of categories?

The answer seems to be the influence of Rasch and his theory of measurement, a theory compatible with Fisher’s epistemology, which integrated statistical analysis with empirical and experimental design.⁴ Two features seemed important to the reasoning. First, because of the sufficient statistics in the PRM, the estimates of the thresholds were independent of the distribution of the person parameters and therefore inherently provided structural properties of the categories. Therefore, the disordered threshold estimates could not be attributed to the distributional properties in the sample of data. In particular, it could not be attributed to some simplistic aspect of the relative frequencies in the categories, such as low frequencies, which are affected by the distribution of persons. Second, because the model had embedded in it the requirement of invariant comparisons, if when applied to relevant data the model provided puzzling results, it seemed necessary that these results be understood in the terms of the implications of the model, not simply ignored or the model abandoned. Thus, despite the initial difficulty in understanding the disordered threshold estimates, it seemed necessary to understand the implications of the disordered threshold estimates within the framework of the PRM itself. In doing so, the author was helped by being able to refer to the rationale with which he had derived the model, a rationale that considered hypothetical, independent dichotomous decisions at thresholds with a substantive definition, and where these decisions had to be constrained by the ordering of the thresholds which reflected the intended ordering of the categories (Andrich, 1978, 1979).

The implication is that to overcome the disordered threshold estimates, the response format or some other aspect of the empirical set up, or some combination of these, needs to be dealt with empirically and perhaps experimentally, and not by choosing a different model. For example, it is possible to combine categories after the fact and obtain a smaller number of ordered thresholds, but such combining must also be seen as generating hypotheses to be followed up empirically and experimentally. Because of the property of the PRM that combining categories after the data are collected is not the same as combining them before they are collected, a property already known to Rasch (1966), this exploratory work must be considered to provide hypotheses that are to be tested. Examples of such a sequence of actions is provided in Andrich and Styles (2004) and Andrich, de Jong, and Sheridan (1997).

The implication that when the threshold estimates are disordered, the category format and all aspects of the data collection design reconsidered, and that categories might be combined to generate possible hypotheses for further study, are entirely compatible with the theme of Rasch (1960) summarized above, and are clearly influenced by that theme. In turn this theme is compatible with Fisher’s epistemology also summarized above. The further implication is that the responsibility for this experimental and empirical work rests with the substantive researcher, working perhaps in conjunction with the psychometrician. Moreover, also consistent with the theme is that it is not the responsibility of the psychometrician simply to find a model that describes data better or to find models that disguise the problem in the empirical ordering of the categories.

To conclude this section, we note that Bradley & Katti (1962) did follow up on Fisher’s method for analyzing ordered categories but, recognizing that the relevant parameters could be disordered, constrained them to be ordered in the estimation:

In view of this difficulty, the work of this paper was undertaken, and the principle of maximizing the variance between objects after scaling relative to within-objects variance was adopted, but with the difference that the maximization was carried out subject to the imposed order restrictions on the scale points. (Bradley & Katti, 1962, p. 356)

Thus, and reflecting a focus on the model rather than the data in the first instance, their first reaction was to counter the reversing of the estimates simply by constraining them to be ordered. This, of course, would have created greater misfit to the model. However, on further reflection, they did go on to note that when these parameters estimates are disordered, perhaps a reconsideration of the definition of the categories might be relevant.

One might argue that, where the intended natural ordering is not achieved, reconsideration of the scale used and the descriptive terminology related to it should be reconsidered. (Bradley & Katti, 1962, p. 368)

Thus, Bradley and Katti reached the conclusion argued in this paper providing evidence that the argument in this article is not unique. However, it seems that no further development of Fisher’s perspective on the empirical ordering of categories was pursued, either by Bradley and Katti, or others, and certainly was not mainstreamed in either psychometrics or statistics. Finding a model to fit the data was seen as sufficient.

Experimental Design and the Polytomous Rasch Model

Where applicable, designs with experimentally independent responses are generally considered the most powerful for assessing some empirical hypothesis. We have seen that the implications of a PRM analysis of data collected in putatively ordered categories, (e.g., R&PC formats), is that although each response is a function of all the thresholds, an inference can be made as if the data came from a hypothetical design of compatible, experimentally independent, dichotomous Bernoulli response variables at the same thresholds and which are analyzed according to the SLM.

This result is not only surprising, but because of the inference to a design with experimentally independent responses, it is as powerful as it can be for testing the relevant hypothesis that the thresholds are ordered correctly. However, in addition to the inference to a hypothetical, experimental design in the case of data collected in ordered categories, the result also has direct implications for carrying out actual experiments with the rubrics for ordered categories. Thus, using balanced designs that ensure experimental independence, different judges may be asked to make dichotomous decisions at thresholds defined to be of increasing levels on the variable, and the responses analysed using the SLM. Not only can the ordering of the thresholds be studied to ensure that the required ordering is working as intended, but the distances between thresholds can also be examined with a view to having some minimum, and perhaps approximately homogeneous, distances between successive thresholds. Then it might be relevant to collect data in the standard ordered category format with the same definition of thresholds and compare the threshold estimates from the two designs. There is of course nothing to force the threshold estimates from the two designs to be statistically equivalent. However, any differences between the two designs would give insight into the way each design generates responses. The differences in threshold estimates would disclose how the hypothetical design of experimentally independent variables was different from a real design. Then, for example, if the thresholds are disordered in the standard design (the inferred, hypothetical, experimentally independent design with dichotomous response variables), but not in the actual experimentally independent design with dichotomous response variables, it would be a focus for considering how to overcome the problems of disordered thresholds in the standard design.

That, first an inferred, hypothetical design of experimentally independent dichotomous variables from responses in putatively ordered categories, and second, that an implication for conducting parallel real designs of experimentally independent dichotomous variables, arise from the mathematics of a model with sufficient statistics, are consistent with all three of Fisher’s three legacies described in this article, legacies advanced further by Rasch.

Karl Pearson and Finding a Model to Fit the Data

In contrast to the application of the PRM, in a paper read before the Royal Statistical Society, McCullagh (1980) elaborates a model for a dependent variable with ordered categories for contingency tables. In this model, the ordering of its threshold estimates, defined differently from those in the PRM, is a property of the model, irrespective of any feature of the data. Essentially the same response model was used by Thurstone (Edwards & Thurstone, 1952), Samejima (1969) and Bock (1975), to provide what is referred to in psychometrics as the graded response model (GRM). This model is used extensively in psychometrics. Briefly, let the cumulative probability $P_{x}^{*}$ , $x = 0, 1, 2, \dots, m$ of a response in one of m+ 1 putatively ordered categories be defined as

P_{x}^{*} = \sum_{k = x}^{m} P {X_{ni} = x; m, β, (δ^{*})}, P_{0}^{*} = 1,

where (δ*) is a vector of thresholds defined differently from those in the PRM. Thus, in contrast to the PRM and exemplified in Figure 2, instead of the continuum being dichotomized by thresholds where each threshold generates a dichotomous response (latent), a single probability distribution for all responses is successively dichotomized to classify the response. Then from Equation (7),

P {X_{ni} = x; m, β, (δ^{*})} = P_{x}^{*} - P_{x - 1}^{*} .

Specifying the same number of parameters as in the PRM, let

P_{x}^{*} = \exp (β - δ_{x}^{*}) / γ^{*},

where $γ^{*} = 1 + \exp (β - δ_{x}^{*})$ . Eq. (9) is the GRM in psychometrics. Then because $P_{x}^{*} \leq P_{x - 1}^{*}$ , and unlike the ${\hat{δ}}_{x}$ in the PRM, which can take any order, ${\hat{δ}}_{x}^{*} \geq {\hat{δ}}_{x - 1}^{*}$ is a property of the GRM. Therefore, the threshold estimates ${\hat{δ}}_{x}^{*}$ cannot be disordered, irrespective of any property of the data.

A PRM analysis of a contingency table example in McCullagh’s paper showed that, as the example in Andrich (1979), it too had disordered threshold estimates. In invited comments to the reading of McCullagh’s paper, Andrich (1980) raised the evidence of disordered threshold estimates as follows:

Unfortunately, because of the strict ordering of the categories . . ., the models of (4.1) . . . do not expose any anomalies in the category ordering. The ordering of the categories is a property of these models, whether or not it is a property of the data. (p. 135)

McCullagh’s response, which seemed inconsistent with his recommendation in other parts of his paper that models should be chosen on the basis of the interpretation of parameters rather than primarily on model fit, is shown below:

Dr Andrich points out an apparent anomaly in the data of Table 6. If the response is in one of the two middle categories there is no relation between the conditional response and the covariate. However this is not an anomaly but is predicted by the model. (McCullagh, 1980, p. 141)

The model prediction McCullagh refers to is the recovery of the data given the estimates of the parameters from the data, taking into account the degrees of freedom. Therefore, successful recovery of the data by a model in the above sense, including the PRM, does not address the hypothesis that the category ordering is a property of the data.

Koch (1980) references Andrich (1979) and notes the issue of threshold ordering but there seems to have been no follow up to it in the statistical literature. Goodman (1981) also references Andrich (1979) in relation to threshold ordering, but only descriptively, without indicating that disordered threshold estimates require further examination. Anderson (1984) references Andrich (1979) but pays no attention to the threshold order. He does, however, distinguish between the processes effectively underpinning the PRM and GRM, stating that the former is relevant for assessment type data, the latter for what he termed group continuous type data—data of the kind found with distributions of incomes. The differences between these processes leading to the two models are elaborated in Andrich (1978, 1995a, 1996).

McCullagh’s response, however, is instructive in its reliance on model fit to justify not addressing the empirical ordering of categories. Like McCullagh (1980), Adams et al. (2012) also rely on tests of fit for the same purpose:

When the data fit the Rasch rating model the response categories are ordered regardless of the (order of the) values of the parameter estimates. (p. 547)

Although reliance on a statistical test of model fit, even when it is irrelevant, can be seen as a mathematical oversight, this oversight seems to have arisen, not because of a lack of sophistication with mathematical statistics, but because of deeper epistemological views of the function of tests of fit in describing data with a model. The article now turns to the statistical epistemology that places so much store on the use of models to describe data—that of K. Pearson.

The Legacy of K. Pearson

It has been argued that the polymath K. Pearson (1857-1936), “. . . established the foundation of modern statistics” (Magnello, 2005-2006), and similarly that he is the “father” of modern statistics (Norton, 1978). Furthermore, “His originality, his real transformation rather than re-ordering knowledge, is to be found in his statistical biology, where he took Galton’s insights and made out of them a new science” (MacKenzie, 1981, p. 88). MacKenzie quotes Pearson from an early paper on statistics from which it can be inferred that Pearson saw the application of statistics as descriptive, rather than explanatory:

Personally I ought to say that there is, in my opinion, considerable danger in applying the methods of exact science to problems of descriptive science …the grace and logical accuracy of the mathematical processes are apt to so fascinate the descriptive scientist that he seeks for sociological hypotheses which fit his mathematical reasoning. (Pearson, 1889, quoted in MacKenzie, 1981, p. 88).

Although the caveat of going beyond the data through mathematical reasoning is well founded, the above quote is considered because of its emphasis on the use of statistics to describe data.

Pearson argued against assuming that all distributions of natural populations measured on some variable were normal, an assumption that had become popular in his time. Instead, he established how nonnormal distributions could be modeled with their first four moments (a term Pearson coined by analogy to moments in physics with which he was familiar), the mean, standard deviation, skewness, and kurtosis. To assess the fit of data to these nonnormal distributions, he developed the chi-square test of fit. Among his many contributions, this was some kind of culmination of his work (Magnello, 2005-2006). This test of fit is used in many contexts, but in general, essentially in two ways. First, it is used to assess an a-priori hypothesis, for example as in the one-way contingency table where the hypothesis of equal frequencies in the cells can be assessed by assuming equal expected frequencies or, for example, in testing independence in a two-way contingency table where, given marginal frequencies, the expected frequencies under independence can be specified.

Second, and relevant for this article, it can be used to assess the recovery of the data given estimates from a small number of parameters using a model. If the model does fit the data according to this criterion, it can be said that the model describes or characterizes the data. This is the test of fit referred to by McCullagh (1980) and Adams et al. (2012) noted above. An elaboration of this second use of the chi-square test is to compare two parameterizations of a model where one set of parameters is a natural subset of the other. If a smaller number of parameters can recover the data, then the model with the smaller number is preferred as a description of the data.

This second application of describing a data set with a model seems to be a K. Pearson legacy to a statistical epistemology. MacKenzie (1981) summarizes this legacy:

. . . one aspect of Pearson’s approach—the construction of models to fit the data—has if anything gained importance since his day . . . (p. 180)

This modeling could be carried out without a theory behind the data:

Using statistics, the biologist could (apparently) measure without theorising, summarise facts without going beyond them, describe without explaining. (pp. 88-89)

MacKenzie’s analysis of Pearson’s descriptive epistemology is not unique. Norton (1978) also describes the same perspective:

Galton . . . always linked his statistical investigations with exercises in theorising . . . about the underlying biological mechanisms that might be responsible for the patterns of correlation and regression he observed. Pearson had absolutely no time for such a combined approach. (p. 11)

Pearson’s goal was a phenomenal theory of heredity lacking any theoretical mediation (such as Galton’s ideas on heredity particles). (p. 11)

Other aspects of his work also arose in a biometric context, and it is not too much to say that they reflect an approach to science with a massive emphasis on the production of mathematical ways of describing observable phenomena, and on the ways of checking up on the goodness of that description. (p. 13).

His second paper developed the series of Pearson curves as a way of describing non-symmetrical and unresolvable distributions of (biological data). And, generally, if the correlational part of Pearson’s work stemmed from a desire to find theory-free connections between different sets of data, then the aim in this other part of his work seems to have been to find ways of accurately describing any given set of data—notably by fitting a curve to it. (p. 13)

Perhaps consistent with MacKenzie’s observation, modeling data as an end in itself has been amplified to a level beyond that which would be supported by Pearson himself. This amplification, even in psychometrics when instruments are being developed and where it cannot be assumed that the data being modeled arise from measurements as found in the natural sciences, may have arisen in part because finding a model to account for data can be an interesting statistical challenge, and in part because to make inferences based on a model of data, it is necessary that the model does describe (fit) the data.

However, rather than a model fitting the data being seen as a necessary condition for the purpose of using the model to make inferences about the data, it seems to have become viewed as a sufficient condition, with one use being to justify ignoring the empirical ordering of categories. Ironically, in this case of the hypothesis of empirical ordering of categories using the PRM, not only is a test of fit irrelevant, but inferences regarding the empirical ordering of categories as such are best made when the data do fit the model. This is because when the data do not fit the model, other vagaries in the data can interact with the threshold estimates. For example, multidimensionality among the items may lead, not only to misfit, but to disordered threshold estimates.

Pearson’s legacy of modeling data can also be seen in the broader psychometric field of item response theory. For example,

Normally, an assumption is made when fitting an IRT [item response theory] model to a set of data . . . (Hambleton, 2000, p. 71)

and

First, there is the difficulty of finding a model that fits the available data and estimating model parameters. (Hambleton, 2000, p. 73)

More explicitly,

. . . when the criterion indicates nonrandomness, an examination of residuals may suggest how the model should be modified to improve fit. (Bock & Jones, 1968, p. 5)

and

. . . if the proportion of misfitted items is large, the reasonable solution is to discard the Rasch model and try a model that includes discrimination . . . in a systematic manner. (Divgi, 1986, p. 295).

In addition, in none of these cases is there a suggestion that the problem is with the data rather than the model. Instead, finding a model that fits the data is the focus. As a result, description using a statistical model appears, not only without explanation (to paraphrase MacKenzie above), but in the case of the PRM and ordered categories, description masquerades as explanation. The epitome of description without explanation is demonstrated in Masters and Wright (1997) and Adams et al. (2012) where they simply refer to the item parameters by their Greek name, δ or τ (depending on the form of the PRM), and give them no substantive meaning—the function of the parameters in the model is simply to help characterize, not to explain, the data.

In addition to the influence of Pearson’s epistemology of atheoretical model fitting, the disposition to describe, and not to problematize, the empirical ordering of categories has taken other related manifestations. These manifestations arose, not only to justify ignoring but explicitly to reject the argument that the PRM can be used to assess the empirical ordering of categories. Three of these manifestations are now considered briefly.

Misinterpreting the Model to Accommodate Disordered Threshold Estimates

The coefficients in Equation (2) were resolved and connected to Andersen (1977) in the first half of 1977. This work was communicated personally to Wright and Masters⁵ in the second half of 1977 and it resulted in the papers Masters (1982) and Wright and Masters (1982). Wright and Masters derived the model by beginning with Equation (4), and reversing the derivation to obtain Equation (2), a derivation that produces the identical model with identical interpretations shown formally in Luo (2005) and Andrich (2013).

However, Wright and Masters introduced the first manifestation, which they thought would justify rejecting concern for the empirical ordering of categories—they interpreted the thresholds as sequential “steps” in which threshold values of any order would be substantively meaningful. Although this interpretation was not compatible with the model (Ostini & Nering, 2006), it became for a time the standard interpretation. Molenaar (1983) described the interpretation as “seductive.” Masters and some research colleagues no longer use the term step, though it can still be found among writings of other researchers.

Second, disordered threshold parameters were simply considered too difficult to interpret (Glas & Verhelst, 1989). Of course, if one does not recognize that the values of the estimates reflect ordering or disordering of the categories in the data, they are impossible to interpret. Given apparent difficulties in interpreting disordered threshold estimates in the PRM, a related manifestation has been simply to assert that there is no relation between the threshold estimates and the ordering of the categories. For example, “We . . . provide two formal definitions of order and show that the Rasch rating model satisfies these definitions regardless of the values of the parameters” (Adams et al., 2012)⁶ and “Therefore, in adjacent-category . . . models the boundaries are not necessarily ordered in the same way as the item categories” (Mellenbergh, 1995).

Given the apparent difficulties in interpreting disordered threshold estimates in the PRM, the third manifestation is the most astonishing. This is to estimate the PRM parameters, and then to abandon the PRM and to interpret the parameters in terms of the GRM of Equations (7) and (9) (Masters & Wright, 1997; Wilson, 2005). That is, having estimated the parameters δ_x in the PRM, they form the cumulative probabilities $P_{x}^{*}$ of Equation (7) and then calculate thresholds $δ_{x}^{*}$ of Equation (9). These authors seem not to appreciate that, in principle, this presentation is identical to having used Equations (7) and (9) of the GRM to analyze the data in the first place. The values of the $δ_{x}^{*}$ in the two analyses would be virtually identical. This shift to the GRM creates a second surprise, namely, that even though it has been well documented that summing the probabilities of adjacent categories in the PRM, and in particular dichotomization (as done by these authors), destroys the PRM (Andrich, 1995c; Jansen & Roskam, 1986; Rasch, 1966), these authors seem to continue to assume that they are dealing with a Rasch model. Seemingly to find a reason to reject the question of the empirical ordering of categories as revealed by the PRM, they do not see that this move to the GRM thresholds abandons the PRM and all its properties as a Rasch model. Wilson (2005) continues with the term step for the thresholds δ_x in the PRM and the term threshold for $δ_{x}^{*}$ and Masters and Wright (1997) make it appear as if the two presentations are simply part of the so called “partial credit model,” a Rasch model!

In none of these three manifestations that eschew the problematizing of the empirical ordering of categories is there any concern for differences in processes behind the PRM and the GRM, a difference described by Anderson (1984) and noted earlier. Nevertheless, it is very telling that in abandoning the PRM and moving to the GRM, Masters and Wilson must sense that the results from the PRM show that something is amiss. If they did not, then they would simply retain the PRM and interpret its parameter estimates. However, rather than seeing any potential problem with the data as disclosed by the PRM, they conclude that it is a problem with the PRM. The GRM, in which its thresholds are always ordered, and to which they then shift, hides any problem with the empirical ordering of categories; presumably, that is exactly the reason they make that shift. And this is contrary to Fisher’s indication that the ordering of the categories should be a property of responses and not of the model used to analyze them.

The Current Status of Assessing the Empirical Ordering of Categories

As indicated earlier, in most of modern psychometrics the question of the empirical ordering of categories is not an issue. In general, the PRM and the GRM are seen simply as alternatives and the choice is a matter of taste. For example, Thissen and Steinberg (1986) provide a descriptive taxonomy of models for ordered categories, and call the class of models to which the PRM belongs as “divide by total models” and the models to which the GRM belongs as “difference” models. They, too, do not consider the possible different processes that might make them relevant for different kinds of situations as indicated in Anderson (1984).

Among the researchers who apply the Rasch models, there is a range of reactions. Some adopt the argument that the PRM provides an empirical testing of ordering of categories (e.g., Von Davier & Rost, 1995; Zhu, 2003), whereas others are hostile to the idea (e.g., Adams et al., 2012).

There is also a different kind of reaction that accepts the proposition that the empirical ordering of categories may be a problem, but then suggests a range of other evidence that might be used. From this kind of reaction, the threshold ordering is only one possible requirement, but it is not a necessary requirement (Linacre, 1999). At least debating and recognizing that there may be a problem with the empirical ordering of categories is a major advance in general, and an advance toward understanding the unique value of the PRM in assessing the empirical ordering of the categories, in particular.

Not surprisingly, given the properties of the PRM, researchers who have most engaged in problematizing the empirical ordering of categories are users of Rasch models. In addition, and again not surprisingly, given the influence of Wright, many of these were students of Wright⁷—for example, Masters, Linacre, Adams, and Wilson (e.g., Adams et al., 2012; Linacre, 1999; Masters, 1982). These latter authors essentially take the position of Masters (1982) and Masters and Wright (1997), embracing various or all aspects of their three misconceptions described above, and if there is any concern with disordered threshold estimates, then it is seen as primarily, if not exclusively, a problem with the PRM.

The maintenance of these misconceptions by these researchers seems curious given that B. D. Wright, while still referring to the thresholds as “steps,” wrote to the author indicating that he (Wright) had abandoned his original position:

I have to admit after all these years, I have ended up on your side with respect to partial credit. Unless the steps go up in difficulty—I am always forced to reconsider the response categories, and in the end, combine them until I get something I can make sense of. But of course that is unsatisfactory because it makes me wish I had asked my questions differently. (B. D. Wright, personal communication, 1989)

The most important part of the change of perspective of Wright comes from his last sentence where he correctly concludes that simply combining categories post hoc is not satisfactory. However, rather than merely “wishing” that the questions had been asked differently, the implication is that for the next round of data collection, either the questions do have to be asked differently, or some other aspect of the data collection may need to be changed. How differently, or what needs to be changed, depends on a qualitative analysis based on all features of the data as revealed by an analysis with the PRM. Thus, 12 years after being shown the ordering properties of the PRM, Wright came to appreciate it. There is not the space in this article to indicate that in his promotion of Rasch measurement theory in general, Wright, like Rasch, did subscribe to the need for empirical and experimental investigation, both in constructing items before data collection and in following up when data showed misfit to the PRM.

Conclusion

Response formats that involve categories that are intended to be ordered are common in the educational, psychological, health and economic sciences. Because the ordering of the categories reflects on the understanding of what it means to have more of the property assessed, and decisions assume that ordering is operating as intended, evidence that such is the case seems critical. Nevertheless, the question of whether the categories are working empirically as intended has been largely ignored in psychometrics and more generally in statistics.

This article describes some of the theoretical development of the PRM, which can be used routinely to check the empirical ordering of categories. The main theme of the article is the contribution of R. A. Fisher leading to this property of the model: first, Fisher’s concept of sufficiency in operationalizing Rasch’s requirement of invariant comparisons for models for measurement and Andersen’s application of sufficiency in the case of a unidimensional model for ordered categorical responses; second, Fisher’s compelling observation that the ordering of categories should be an empirical property of the data and not of the model that is used to analyze the data; and third, Fisher’s epistemology of integrating empirical and experimental design with statistical analysis to explain, not simply to describe, data. This theme contrasts the position that at present seems most common in psychometrics in the analysis of data in ordered categories—that of relying on tests of fit to justify ignoring the study of the empirical ordering of categories. It is suggested that the use of a test of fit in this way arises from a legacy of K. Pearson’s epistemology of finding models which, free from any theoretical and substantive explanation, describe or fit data. From this brief historical analysis, it is suggested that in analyzing data from response formats in ordered categories, Pearson’s legacy needs to be replaced with Fisher’s, and that not only should the empirical ordering of categories be tested routinely, but that if the ordering is not working as intended, then qualitative analyses and further experimentation to correct the problem need to be carried out. The PRM provides necessary and sufficient evidence when this needs to be done.

Finally, the direct link between the PRM analysis of ordered categorical data with both hypothetical and real designs with experimentally independent judgments at thresholds was explained. This link can be exploited in helping define ordered categories that work as required.

Footnotes

Acknowledgements

G. Marcoulides encouraged that this article be written and Josh McGrane, Pender Pedler, Joe Ryan, Barry Sheridan, and Irene Styles made constructive suggestions for the present version of the article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was funded in part by grants from the Australian Research Council and by Pearson.

Notes

References

Adams

R. J.

M. L.

Wilson

(2012). The Rasch rating model and the disordered threshold controversy. Educational and Psychological Measurement, 72, 547-573. doi:10.1177/0013164411432166

Andersen

E. B.

(1977). Sufficient statistics and latent trait models. Psychometrika, 42, 69-81.

Andersen

E. B.

Wøhlk

(2001). The life of Georg Rasch as a mathematician and as a statistician. In Boomsma

van Duijn

M. A. J.

Sniders

T. A. B.

(Eds.), Lecture notes in statistics: Vol. 157. Essays in item response theory (pp. 3-24). New York, NY: Springer.

Anderson

J. A.

(1984). Regression and ordered categorical variables. Journal of the Royal Statistical Society Series B, 46, 1-30.

Andrich

(1978). A rating formulation for ordered response categories. Psychometrika,43, 561-574.

Andrich

(1979). A model for contingency tables having an ordered response classification. Biometrics,35, 403-415.

Andrich

(1980). Invited contribution to the discussion on: Regression models for ordinal data, P. McCullagh. Journal of the Royal Statistical Society Series B,42, 135.

Andrich

(1995a). Distinctive and incompatible properties of two common classes of IRT models for graded responses. Applied Psychological Measurement. 19, 101-119.

Andrich

(1995b). Further remarks on the non-dichotomization of graded responses. Psychometrika, 60, 37-46.

10.

Andrich

(1995c). Models for measurement, precision and the non-dichotomization of graded responses. Psychometrika, 60, 7-26.

11.

Andrich

(1996). Measurement criteria for choosing among models for graded responses. In von Eye

Clogg

C. C.

(Eds.), Analysis of categorical variables in developmental research (pp. 3-35). Orlando, FL: Academic Press.

12.

Andrich

(2004). Controversy and the Rasch model: A characteristic of incompatible paradigms? Medical Care, 42, 7-16.

13.

Andrich

(2013). An expanded derivation of the threshold structure of the polytomous Rasch rating model which dispels any “threshold disorder controversy”. Educational and Psychological Measurement,73, 78-124.

14.

Andrich

de Jong

J. H. A. L.

Sheridan

B. E.

(1997). Diagnostic opportunities with the Rasch model for ordered response categories. In Rost

Langeheine

(Eds.), Applications of latent trait and latent class models in the social sciences (pp. 58-68). GMBH: Münster, Germany: Waxmann Verlag.

15.

Andrich

Styles

I. M.

(2004). Final report on the psychometric analysis of the Early Development Instrument (EDI) using the Rasch model: A technical paper commissioned for the development of the Australian Early Development Instrument (AEDI). Retrieved from http://www.rch.org.au/aedi/studies/index.cfm?doc_id=14792

16.

Bock

R. D.

(1975). Multivariate statistical methods in behavioral research. New York, NY: McGraw-Hill.

17.

Bock

R. D.

Jones

L. V.

(1968). The measurement and prediction of judgment and choice. San Francisco, CA: Holden Day.

18.

Bradley

R. A.

Katti

S. K.

(1962). Optimal scaling for ordered categories. Psychometrika, 27, 355-374.

19.

Divgi

D. R.

(1986). Does the Rasch model really work for multiple choice items? Not if you look closely. Journal of Educational Measurement, 23, 283-298.

20.

Dubois

Burns

J. A.

(1975). An analysis of the meaning of the question mark response category in attitude scales. Educational and Psychological Measurement, 35, 869-884.

21.

Edwards

A. L.

Thurstone

L. L.

(1952). An internal consistency check for scale values determined by the method of successive intervals. Psychometrika, 17, 169-180.

22.

Embretson

S. E.

Reise

S. P.

(2000). Item response theory for psychologists. London, England: Erlbaum.

23.

Fisher

R. A.

(1925). Statistical methods for research workers. Edinburgh, Scotland: Oliver & Boyd.

24.

Fisher

R. A.

(1934a). Statistical methods for research workers (5th ed.). Edinburgh, Scotland: Oliver & Boyd.

25.

Fisher

R. A.

(1934b). Two new properties of mathematical likelihood. Proceedings of the Royal Society A, 144, 285-307.

26.

Fisher

R. A.

(1935). The design of experiments. Oxford, England: Oliver & Boyd.

27.

Fisher

R.A.

(1958). Statistical methods for research workers (13th ed.). New York, NY: Hafner.

28.

Fisher Box

(1978). R. A. Fisher: The life of a scientist. New York, NY: Wiley.

29.

Glas

C. A. W.

Verhelst

N. D.

(1989). Extensions of the partial credit model. Psychometrika, 54, 635-659.

30.

Goodman

L. A.

(1981). Three elementary views of log linear models for the analysis of cross-classifications having ordered categories. In Leinhardt

(Ed.), Sociological methodology (pp. 193-239). San Francisco, CA: Jossey-Bass.

31.

Hambleton

R. K.

(2000). Emergence of item response modeling in instrument development and data analysis. Medical Care, 38 (Suppl. II), II-28-II-42.

32.

Jansen

P. G. W.

Roskam

E. E.

(1986). Latent trait models and dichotomization of graded responses. Psychometrika, 51, 69-91.

33.

Koch

G. G.

(1980). Invited contribution to the discussion on regression models for ordinal data, P. McCullagh. Journal of the Royal Statistical Society Series B, 42, 137.

34.

Kuhn

T. S.

(1961). The function of measurement in modern physical science. Isis, 52, 161-190.

35.

Kuhn

T. S.

(1970). The structure of scientific revolutions (2nd ed., enlarged.) Chicago, IL: University of Chicago Press.

36.

Likert

(1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 5-55.

37.

Linacre

J. M.

(1999). Investigating rating scale category utility. Journal of Outcome Measurement, 3, 103-122.

38.

Luo

(2005). The relationship between the rating scale and partial credit models and the implication of disordered thresholds of the Rasch models for polytomous responses. Journal of Applied Measurement, 6, 443-455.

39.

MacKenzie

D. A.

(1981). Statistics in Britain, 1865–1930. Edinburgh, Scotland: Edinburgh University Press.

40.

Magnello

M. E.

(2005-2006). Karl Pearson and the origins of modern statistics: An elastician becomes a statistician. Rutherford Journal—The New Zealand Journal for the History and Philosophy of Science and Technology, 1. Retrieved from http://www.rutherfordjournal.org/current.html

41.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.

42.

Masters

G. N.

Wright

B. D.

(1997). The partial credit model. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of item response theory (pp. 101-121). New York, NY. Springer.

43.

McCullagh

(1980). Regression models for ordinal data. Journal of the Royal Statistical SocietySeries B, 42, 109-142.

44.

Mellenbergh

G. J.

(1995). Conceptual notes on models for discrete polytomous item responses. Applied Psychological Measurement, 19, 91-100.

45.

Molenaar

I. W.

(1983). Item steps (Heymans Bulletin HB-83-63). University of Groningen, Groningen, Netherlands.

46.

Norton

B. J.

(1978). Karl Pearson and statistics: The social origins of scientific innovation. Social Studies of Science, 8(3), 3-34.

47.

Ostini

Nering

M. L.

(2006). Polytomous item response models (Sage University Paper Series on Quantitative Applications in the Social Sciences, 07-144). Thousand Oaks, CA: Sage.

48.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests (Reprint, with Foreword and Afterword by B. D. Wright, Chicago, IL: University of Chicago Press, 1980). Copenhagen, Denmark: Danmarks Paedogogiske Institut. Available from www.rasch.org/books.htm

49.

Rasch

(1961). On general laws and the meaning of measurement in psychology. In Neyman

(Ed.), Proceedings of the fourth Berkeley Symposium on mathematical statistics and probability (Vol. 4, pp. 321-334). Berkeley: University of California Press.

50.

Rasch

(1977). On specific objectivity: An attempt at formalising the request for generality and validity of scientific statements. Danish Yearbook of Philosophy, 14, 58-94.

51.

Rasch

(1966). An individualistic approach to item analysis. In Lazarsfeld

P. F.

Henry

N. W.

(Eds.), Readings in mathematical social science (pp. 89-108). Chicago, IL: Science Research Associates.

52.

Rasch

(1979). Series of interviews recorded with D. Andrich. Laesoe, Denmark.

53.

Raven

J. C.

(1940). Matrix tests. Mental Health, 1, 10-18.

54.

Roskam

E. E.

(1995). Graded responses and joining categories: A rejoinder to Andrich’s “Models for measurement, precision, and the non-dichotomization of graded responses”. Psychometrika, 60, 27-35.

55.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores. Psychometric Monographs, 34(2, No. 17). Retrieved from http://www.psychometrika.org/journal/online/MN17.pdf

56.

Samejima

(1997). Graded response model. In Van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory (pp. 139-152). New York, NY: Springer.

57.

Thissen

Steinberg

(1986). A taxonomy of item response models. Psychometrika, 51, 567-577.

58.

Von Davier

Rost

(1995). Polytomous mixed Rasch models. In Fischer

G. H.

Molenaar

I. W.

(Eds.), Rasch models: Foundations, recent developments, and applications (pp. 371-379). New York, NY: Springer.

59.

Wilson

(2005). Constructing measures. New York, NY. Psychology Press.

60.

Wøhlk Olsen

(2003). Essays on Georg Rasch and his contributions to statistics. University of Copenhagen, Institute of Economics, Copenhagen, Denmark.

61.

Wright

B. D.

Masters

G. N.

(1982). Rating scale analysis: Rasch measurement. Chicago, IL: MESA Press.

62.

Zhu

(2003). A confirmatory study of Rasch-based optimal categorization of a rating scale. Journal of Applied Measurement, 3, 1-15.