An Improved Inferential Procedure to Evaluate Item Discriminations in a Conditional Maximum Likelihood Framework

Abstract

A modified and improved inductive inferential approach to evaluate item discriminations in a conditional maximum likelihood and Rasch modeling framework is suggested. The new approach involves the derivation of four hypothesis tests. It implies a linear restriction of the assumed set of probability distributions in the classical approach that represents scenarios of different item discriminations in a straightforward and efficient manner. Its improvement is discussed, compared to classical procedures (tests and information criteria), and illustrated in Monte Carlo experiments as well as real data examples from educational research. The results show an improvement of power of the modified tests of up to 0.3.

Keywords

item discrimination conditional maximum likelihood hypothesis tests Rasch model

1. Introduction

A crucial question in psychometrics is the problem of evaluating the discriminations of items. In applications of psychometric models ranging from factor analysis and structural equation modeling to item response theory, dealing with it is almost inevitable. Some item response models allow for different item discriminations, and others do not. The Rasch model (Fischer & Molenaar, 1995; Rasch, 1960) assumes equal item discriminations. Testing this assumption on the basis of data is central to applications of the model. Inductive inference on assumptions and parameters of the Rasch model is often based on a conditional maximum likelihood (CML) approach (Andersen, 1970; Molenaar, 1995; Pfanzagl, 1993). A discussion and justification of this approach in a wider research context has recently been given by Skrondal and Rabe-Hesketh (2022). In the CML framework, there is essentially only one well-established approach suitable to investigate differences in item discriminations. It is the comparison of item parameters between groups of persons yielding different scores, that is, numbers of correct responses, which are sufficient statistics for the person parameters of the model. Rasch (1960) suggested a purely descriptive and graphical check comparing the estimates of item parameters. Andersen (1973) derived the first hypothesis testing procedure serving this purpose, that is, a likelihood ratio test of the hypothesis of invariance of item parameters across different person score groups. Both are contained in the R (R Core Team, 2022) package eRm (Mair & Hatzinger, 2007). Other asymptotically equivalent test statistics are straightforward as pointed out by Glas and Verhelst (1995) and Draxler and Alexandrowicz (2015), that is, the Rao score or Lagrange multiplier (Rao, 1948; Silvey, 1959) and the Wald test (Wald, 1943). Recently, Draxler et al. (2022) showcased the application of another relatively new test statistic in this context, that is, the gradient test (Lemonte, 2016; Terrel, 2002). All four test statistics are easily accessible in the R package tcl (Draxler & Kurz, 2023). The comparison of item parameters between person score groups makes sense immediately. For example, if an item was easier relative to the other items for persons with a high score (persons with high ability or proficiency) and at the same time the respective item was more difficult for persons with a low score, the item would have higher discrimination than the other items.

This article suggests a new modified procedure to evaluate item discriminations in the CML framework. It discusses four statistical tests of the hypothesis of invariance of item parameters across person score groups. The modification implies the specification of a linearly restricted alternative hypothesis, that is, an appropriately chosen subset of probability distributions that represents the scenario of different item discriminations in a more straightforward and more efficient manner. It discusses the underlying theory and illustrates in an extensive Monte Carlo study its improvement over the original Andersen likelihood ratio test (ALRT) showing up to $0.3$ higher observed power rates. The new tests are available in the R package tcl.

The remainder of this text is organized as follows. In Section 2, the ideas and concepts are formalized, and the respective test statistics are derived. Section 3 describes the design of a Monte Carlo investigation into relevant properties of the tests (like type I and II error rates), and Section 4 shows its results. Section 5 presents real data examples from educational research. Section 6 gives a discussion and final remarks.

2. Theoretical Foundation

Assume that the true unknown probability distribution generating the data in a sample space is a member of a parametric family of distributions that is specified by a psychometric model indexed by parameters taking values in parameter space $Θ$ . The latter is assumed to be an open subset of Euclidean space. The data are obtained by the binary responses, for example, correct, or incorrect, of a number (sample) of persons to a number of items $k > 2$ (k being an integer of 3 or greater). Roughly speaking, the psychometric model is given by assuming different Rasch models for different groups of persons, where the groups are defined by the person score, that is, the number of correct responses. Denote by n_r the observed number of persons that obtain r correct responses, where $\sum_{r = 1}^{k - 1} n_{r} = n$ . Note that persons with a score $r = 0$ and $r = k$ are excluded since the responses of these persons are completely uninformative in the CML context. Thus, n denotes the total sample, but it is considered to consist only of persons with an integer score from 1 to $k - 1$ . Note also that both n_r and n are basically random variables, whose distributions depend on the unknown parameters of the model, but they are treated as given constants in the present framework.

The psychometric model is given by

P (Y_{i j} = y_{i j}) = \frac{exp (y_{i j} (τ_{i} + β_{j r}))}{1 + exp (τ_{i} + β_{j r})}, i = 1, \dots, n_{r}, j = 1, \dots, k, r = 1, \dots, k - 1,

where $Y_{i j} \in {0, 1}$ is the binary response of person i to item j (1 standing for a correct response), $τ_{i} \in ℝ$ is a person parameter usually interpreted as an ability, proficiency, or attitude, and $β_{j r} \in ℝ$ is the item parameter of item j in the group of persons with score r. It is usually interpreted as easiness or attractiveness. Thus, this model allows the parameters to differ between groups of persons with different numbers of correct responses. Setting the item parameters equal between all person score groups yields the Rasch model as a special case. The parameters are identified when, for example, $β_{k r} = 0$ , or $\sum_{j} β_{j r} = 0 \forall r$ , and the conditions derived by Fischer (1981) are met in each person score group r. Further assume that the k responses of every single person are independent, that is, local independence, and the persons are drawn independently from a population of interest. Thus, the k-vectors of responses of the persons in the total sample are considered as a sequence of n independent k-vector-valued random variables. Let the binary responses of all persons to all items be arranged in an $n \times k$ matrix denoted by Y, that is, the response matrix. Then, the joint distribution of all these responses is (by using matrix multiplication) obtained by

P (Y = y) = \prod_{r = 1}^{k - 1} \prod_{i = 1}^{n_{r}} \prod_{j = 1}^{k} P (Y_{i j} = y_{i j}) = \underset{C (τ, β)}{\underset{︸}{\frac{1}{\prod_{r} \prod_{i} \prod_{j} (1 + exp (τ_{i} + β_{j r}))}}} exp (\sum_{r} \sum_{i} \sum_{j} (y_{i j} (τ_{i} + β_{j r})))

= C (τ, β) exp (τ^{⊤} r + β^{⊤} s),

where $τ$ denotes an $n \times 1$ matrix (i.e., a column vector of length n) containing as elements all person parameters of the total sample n (which itself is composed of the numbers of persons in all $k - 1$ person score groups) and $β^{⊤} = (β_{1}^{⊤}, \dots, β_{k - 1}^{⊤})$ with $β_{r}^{⊤} = (β_{1 r}, \dots, β_{k r})$ as a $k \times 1$ matrix of the k item parameters in person score group r. It can immediately be seen from the factorization criterion that the statistics R (with realization r) and S (with realization s) are sufficient for the class of distributions. It is a member of a multiparameter exponential family with $(τ, β) \in Θ$ as its natural parameter space since the first factor $C (τ, β)$ is a normalizing constant (that does not depend on $Y = y$ ) and the second factor $exp (\cdot)$ depends on the observations only through sufficient statistics. The statistic R is an $n \times 1$ matrix containing the numbers of correct responses (person scores) of all n persons, that is,

R_{i} = \sum_{j = 1}^{k} Y_{i j}, i = 1, \dots, n_{r}, r = 1, \dots, k - 1,

and S a $k (k - 1) \times 1$ matrix with elements given by

S_{j r} = \sum_{i = 1}^{n_{r}} Y_{i j}, j = 1, \dots, k, r = 1, \dots, k - 1.

Hence, further considerations can be restricted to the distributions of the sufficient statistics. Since the parameter $τ$ is treated as a vector of nuisances and the vector $β$ as the parameter of interest, the marginal distribution $P (R = r)$ of the sufficient statistic for the nuisance parameters and the joint distribution of both vectors of sufficient statistics $P (R = r, S = s)$ are considered to obtain the conditional distribution

P (S = s | R = r) = \frac{P (R = r, S = s)}{P (R = r)} .

It does not depend on the nuisance parameter vector $τ$ . Treating it as a function of $β$ and taking the logarithm yields the conditional log likelihood denoted by $ℓ (β)$ . A further treatment of fundamentals of conditional likelihood theory is given in Online Appendix A.

2.1. Andersen’s Likelihood Ratio Test

Let $f : ℝ^{k (k - 1)} \to ℝ^{k (k - 2)}$ ,

f (β) = A β = (\begin{matrix} β_{1} - β_{2} \\ ⋮ \\ β_{r - 1} - β_{r} \\ ⋮ \\ β_{k - 2} - β_{k - 1} \end{matrix}),

where

A = \frac{\partial f}{\partial β^{⊤}} = (\begin{matrix} I_{k} & - I_{k} & 0_{k, k} & \dots & 0_{k, k} \\ ⋮ & ⋱ & ⋮ \\ 0_{k, k} & \dots & 0_{k, k} & I_{k} & - I_{k} & 0_{k, k} & \dots & 0_{k, k} \\ ⋮ & ⋱ & ⋮ \\ 0_{k, k} & \dots & 0_{k, k} & I_{k} & - I_{k} \end{matrix}),

with I _k denoting the identity matrix and $0_{k, k}$ denoting a zero matrix with k rows and k columns. Thus, A is the Jacobian of the transformation f, which is a $k (k - 2) \times k (k - 1)$ matrix of the first-order partial derivatives with respect to all item parameters that transforms the vector $β$ in a vector of differences (between successive person score groups). Consider the subsets of distributions or (equivalently) parameter spaces defined by

Θ_{0} = {(τ, β) | (τ, β) \in Θ, A β = 0},

and

Θ_{1} = {(τ, β) | (τ, β) \in Θ, A β \neq 0},

$Θ_{0} \cup Θ_{1} = Θ$ . Of interest is to test the hypothesis that the true unknown distribution or (equivalently) the true unknown parameter vectors satisfy

(τ, β) \in Θ_{0}

(as assumed by the Rasch model) against the alternative

(τ, β) \in Θ_{1} .

The test statistic is based on the likelihood ratio or, equivalently, the log likelihood difference. The key to it is to maximize $ℓ (β)$ twice: on the one hand, by subjecting it to the condition $A β = 0$ (equality of item parameters across person score groups), and on the other hand, without restrictions. The closer the ratio of both is to 1 (and equivalently, the log likelihood difference to 0), the more likely or more plausible is the scenario of equal item parameters across person score groups given the data. Denote by ${\hat{β}}_{0}$ the vector maximizing $ℓ (β)$ subject to $A β = 0$ (subscript 0 indicating this restriction). Then, the test statistic is obtained by

T_{1} = - 2 (ℓ ({\hat{β}}_{0}) - ℓ (\hat{β})) .

Andersen (1973) proved that the limiting distribution of T ₁ is the central $χ^{2}$ with $d f = (k - 1) (k - 2)$ when $(τ, β) \in Θ_{0}$ holds true and $n \to \infty$ . Note that one item parameter of the k parameters for each person score group r is not free and all item parameters of one of the $k - 1$ person score groups are also not free.

In practice of data analysis, usually only two groups of persons are considered instead of all $k - 1$ , for example, one group of persons with a score r that is below (or equal) a specified value, that is, a cutoff, and the other group of persons with a score r that is above. The cutoff value is often chosen as the median of the observed frequency distribution of the person score. Considering only two groups of persons reduces the degrees of freedom or the number of free parameters to $k - 1$ . This has basically two beneficial consequences provided $k > 3$ . One is that the parameters can be estimated more accurately, that is, estimates with lower standard errors, and the test attaining higher power. The other is that a practically sufficient approximation of the distribution of the test statistic to the limiting $χ^{2}$ distribution can already be achieved for smaller sample sizes.

2.2. A Modified Procedure

The suggested modification implies the specification of a proper subset of $Θ$ by considering $β$ as a linear function as follows. Let $β : ℝ^{2 k} \to ℝ^{(k - 1) k}$ ,

β = (\begin{matrix} β_{1} \\ ⋮ \\ β_{r} \\ ⋮ \\ β_{k - 1} \end{matrix}) = (\begin{matrix} α \\ ⋮ \\ α + δ (r - 1) \\ ⋮ \\ α + δ (k - 2) \end{matrix}),

β_{r} = α + δ (r - 1) = (\begin{matrix} α_{1} + δ_{1} (r - 1) \\ ⋮ \\ α_{k} + δ_{k} (r - 1) \end{matrix}) .

Instead of $k - 1$ parameters per item (as many as person score groups), only two parameters per item are considered. Obviously, this is a more parsimonious parametrization provided $k > 3$ . The $α$ parameter as the additive constant of the linear function represents a baseline, that is, the easiness or attractiveness of the item in person score group 1, which is a nuisance parameter in the present problem. The $δ$ parameter is the gradient (or slope) of the linear function and, thus, represents the constant change of the easiness or attractiveness of the item between successive person score groups. It is the only parameter of interest. If $δ_{j - 1} < δ_{j}$ , then item $j - 1$ will less discriminate than item j. If the $δ$ parameters of two items are equal, they will have the same discrimination. Obviously, the $δ$ parameters have a linear effect on the log odds (i.e., logits) of the response probabilities (probabilities of correct responses). The greater a $δ$ parameter (relative to the others) is, the steeper is the increase of the probability of a correct response with increasing person score r, and thus, the higher is the discrimination of the respective item.

The restricted model is then given by

P (Y_{i j} = y_{i j}) = \frac{exp (y_{i j} (τ_{i} + α_{j} + δ_{j} (r - 1)))}{1 + exp (τ_{i} + α_{j} + δ_{j} (r - 1))},

and the joint distribution of the responses of all persons to all items by

P (Y = y) = \prod_{r = 1}^{k - 1} \prod_{i = 1}^{n_{r}} \prod_{j = 1}^{k} P (Y_{i j} = y_{i j})

= \underset{C (τ, α, δ)}{\underset{︸}{\frac{1}{\prod_{r} \prod_{i} \prod_{j} (1 + exp (τ_{i} + α_{j} + δ_{j} (r - 1)))}}} exp (\sum_{r} \sum_{i} \sum_{j} (y_{i j} (τ_{i} + α_{j} + δ_{j} (r - 1))))

= C (τ, α, δ) exp (τ^{⊤} r + α^{⊤} t + δ^{⊤} u) .

For identifiability, let $α_{k} = δ_{k} = 0$ . Again, the model is a member of a multiparameter exponential family with $(τ, α, δ) \in Θ_{2}$ as its natural parameter space. It holds that $Θ_{2} \subset Θ$ given $k > 3$ . In case of $k = 3$ , one obtains $Θ_{2} = Θ$ (i.e., $Θ_{2}$ has the same elements as $Θ$ ) with $α = β_{1}$ and $δ = β_{2} - β_{1}$ . The factorization criterion immediately shows that the statistics R (with realization r), T (with realization t), and U (with realization u) are sufficient for the restricted family of distributions, where

T^{⊤} = (T_{1}, \dots, T_{k}), T_{j} = \sum_{r = 1}^{k - 1} \sum_{i = 1}^{n_{r}} Y_{i j},

and

U^{⊤} = (U_{1}, \dots, U_{k}), U_{j} = \sum_{r = 1}^{k - 1} \sum_{i = 1}^{n_{r}} Y_{i j} (r - 1) .

A conditional distribution and conditional likelihood can again be derived by conditioning on $R = r$ to eliminate the effect of $τ$ . As already noted, the parameter α is also not of interest. Principally, one may also eliminate it by additionally conditioning on $T = t$ but, practically, this is computationally (much) more intensive, nevertheless doable (with today’s machines). A comment on this is given in Section 6. The joint distribution of R, T, and U, the marginal distribution of R, and the conditional distribution of T and U given $R = r$ are obtained by

P (R = r, T = t, U = u) = C (τ, α, δ) \prod_{r = 1}^{k - 1} \prod_{i = 1}^{n_{r}} \prod_{j = 1}^{k} exp (τ_{i} r_{i} + α_{j} t_{j} + δ_{j} u_{j}) K (r, t, u)

= C (τ, α, δ) exp (τ^{⊤} r + α^{⊤} t + δ^{⊤} u) K (r, t, u),

P (R = r) = C (τ, α, δ) \prod_{r = 1}^{k - 1} \prod_{i = 1}^{n_{r}} exp (τ_{i} r_{i}) γ_{r}^{n_{r}} (α, δ)

= C (τ, α, δ) exp (τ^{⊤} r) \prod_{r = 1}^{k - 1} γ_{r}^{n_{r}} (α, δ),

and

P (T = t, U = u | R = r) = \frac{P (R = r, T = t, U = u)}{P (R = r)}

= \frac{1}{\prod_{r = 1}^{k - 1} γ_{r}^{n_{r}} (α, δ)} exp (α^{⊤} t + δ^{⊤} u) K (r, t, u) .

All three of them are also the members of multiparameter exponential families (as can immediately be seen). The function $γ_{r} (\cdot)$ denotes an elementary symmetric function of order r, that is

γ_{1} = exp (α_{1}) + \dots + exp (α_{k}),

γ_{2} = exp (α_{1} + α_{2} + δ_{1} + δ_{2}) + \dots + exp (α_{k - 1} + α_{k} + δ_{k - 1} + δ_{k}),

γ_{3} = exp (α_{1} + α_{2} + α_{3} + (δ_{1} + δ_{2} + δ_{3}) 2)

+ \dots + exp (α_{k - 2} + α_{k - 1} + α_{k} + (δ_{k - 2} + δ_{k - 1} + δ_{k}) 2),

⋮

γ_{k - 1} = exp (α_{1} + \dots + α_{k - 1} + (δ_{1} + \dots + δ_{k - 1}) (k - 2))

+ \dots + exp (α_{2} + \dots + α_{k} + (δ_{2} + \dots + δ_{k}) (k - 2)) .

Thus, $γ_{1}$ is composed of a sum over all items, each summand being a function of the single parameter ( $α$ ) associated with the respective item, $γ_{2}$ is composed of a sum over all (potential) pairs of items, each summand being a function of the four parameters (two $α$ par., one for each item, and two $δ$ par., one for each item) associated with the respective pair, $γ_{3}$ is composed of a sum of all (potential) triples of items, each summand being a function of the six parameters associated with the respective triple, and so on. The factor $K (r, t, u)$ is another combinatorial function. Its exact determination is complicated and computationally demanding (Miller & Harrison, 2013). It is the number of potential $n \times k$ matrices for which one obtains that $R = r$ , $T = t$ , and $U = u$ . Fortunately, in the present inferential problem, it can be ignored since it does not depend on any of the parameters. As can be seen in (2), $τ$ is eliminated by conditioning on the observed person scores. Treating (2) as a function of the remaining parameters (item parameters) and taking the logarithm yields the conditional log likelihood function

ℓ (α, δ) = α^{⊤} t + δ^{⊤} u - \sum_{r = 1}^{k - 1} n_{r} log (γ_{r} (α, δ)),

where the additive constant $log (K (r, t, u))$ is omitted.

The score function is given by

V_{1} = (\begin{matrix} \frac{\partial ℓ}{\partial α_{1}} \\ ⋮ \\ \frac{\partial ℓ}{\partial α_{k - 1}} \end{matrix}) = (\begin{matrix} t_{1} - E (T_{1}) \\ ⋮ \\ t_{k - 1} - E (T_{k - 1}) \end{matrix}) = (\begin{matrix} t_{1} - \sum_{r} n_{r} γ_{r}^{- 1} \frac{\partial γ_{r}}{\partial α_{1}} \\ ⋮ \\ t_{k - 1} - \sum_{r} n_{r} γ_{r}^{- 1} \frac{\partial γ_{r}}{\partial α_{k - 1}} \end{matrix}),

V_{2} = (\begin{matrix} \frac{\partial ℓ}{\partial δ_{1}} \\ ⋮ \\ \frac{\partial ℓ}{\partial δ_{k - 1}} \end{matrix}) = (\begin{matrix} u_{1} - E (U_{1}) \\ ⋮ \\ u_{k - 1} - E (U_{k - 1}) \end{matrix}) = (\begin{matrix} u_{1} - \sum_{r} n_{r} γ_{r}^{- 1} \frac{\partial γ_{r}}{\partial δ_{1}} \\ ⋮ \\ u_{k - 1} - \sum_{r} n_{r} γ_{r}^{- 1} \frac{\partial γ_{r}}{\partial δ_{k - 1}} \end{matrix}),

V = (\begin{matrix} V_{1} \\ V_{2} \end{matrix}) .

Note that the score function V for the restricted model has only length $2 (k - 1)$ , where V ₁ refers to the nuisance parameters and V ₂ to the parameters of interest. The elements referring to the fixed parameters that are not free are omitted, that is, the elements referring to the kth $α$ and $δ$ parameters. Again, the score function is composed of the differences between the (first $k - 1$ ) elements of r and u (observed values of the sufficient statistics for α and $δ$ ) and their respective expected values as a function of α and $δ$ given $R = r$ .

The Fisher information matrix can be obtained as follows. Let

J = \frac{\partial β}{\partial (α_{1}, \dots, α_{k - 1}, δ_{1}, \dots, δ_{k - 1})}

be the Jacobian matrix of the linear function $β$ , which has $k (k - 1)$ rows and $2 (k - 1)$ columns. Then, the information matrix of the restricted model is obtained by

F (α, δ) = J^{⊤} F (β) J .

It is simply a linear function of the information matrix $F (β)$ of the general model given in Online Appendix A.

The CML estimate of $(α, δ)$ is defined by

(\hat{α}, \hat{δ}) : = \underset{(α, δ) \in ℝ^{2 k}}{argmax} ℓ (α, δ),

which is obtained by solving $V = 0_{2 (k - 1)}$ for α and $δ$ . To obtain asymptotic properties of the CML estimator, general results of likelihood theory and exponential families as well as particular results for the CML case established by Andersen (1970) and Pfanzagl (1993) can be applied. For the CML case, one has specifically to consider a mild regularity condition that the values of the person parameters ( $τ$ parameters) must not be too extreme, that is, in asymptotic theory, when $n \to \infty$ , one considers an infinite sequence of $τ$ parameters. Then, for fixed k, it holds that

({\hat{α}}_{1}, \dots, {\hat{α}}_{k - 1}) \overset{P}{\to} (α_{1}, \dots, α_{k - 1}),

({\hat{δ}}_{1}, \dots, {\hat{δ}}_{k - 1}) \overset{P}{\to} (δ_{1}, \dots, δ_{k - 1}),

and

\sqrt{n} (({\hat{α}}_{1}, \dots, {\hat{α}}_{k - 1}, {\hat{δ}}_{1}, \dots, {\hat{δ}}_{k - 1}) - (α_{1}, \dots, α_{k - 1}, δ_{1}, \dots, δ_{k - 1})) \overset{D}{\to} N (0_{2 (k - 1)}, F^{- 1} (α, δ)),

when the number of persons $n \to \infty$ , where $F^{- 1} (α, δ)$ is the asymptotic covariance matrix of the CML estimate $(\hat{α}, \hat{δ})$ .

2.3. Statistical Tests

Consider the two subclasses of distributions or (equivalently) subsets of the restricted parameter space $Θ_{2}$ :

Θ_{3} = {(τ, α, δ) | (τ, α, δ) \in Θ_{2}, δ = 0},

which is the same as $Θ_{0}$ , and

Θ_{4} = {(τ, α, δ) | (τ, α, δ) \in Θ_{2}, δ \neq 0},

$Θ_{3} \cup Θ_{4} = Θ_{2}$ . Note that $Θ_{4} \subset Θ_{1}$ for $k > 3$ , and $Θ_{4} = Θ_{1}$ for $k = 3$ , with $α = β_{1}$ and $δ = β_{2} - β_{1}$ . Of interest is to test the hypothesis that the true unknown distribution (generating the data) or the true unknown parameter vectors satisfy

(τ, α, δ) \in Θ_{3}

(as assumed by the Rasch model) against the alternative

(τ, α, δ) \in Θ_{4},

which represents the scenario of a constant difference of the easiness or attractiveness of at least one item between successive person score groups and thus a different discrimination of at least one item relative to the others.

Four test statistics derived from asymptotic theory may be used. Denote by $({\hat{α}}_{0}, δ = 0)$ the argument of the maximum of $ℓ (α, δ)$ subject to $δ = 0$ (subscript 0 indicating this restriction). In this case, the elements of ${\hat{α}}_{0}$ are simply the CML estimates of the item parameters of the Rasch model. A likelihood ratio test statistic (as a modification of the ALRT) is then given by

T_{2} = - 2 (ℓ ({\hat{α}}_{0}, δ = 0) - ℓ (\hat{α}, \hat{δ})) .

A Rao score test statistic can be obtained by

T_{3} = V^{⊤} ({\hat{α}}_{0}, δ = 0) F^{- 1} ({\hat{α}}_{0}, δ = 0) V ({\hat{α}}_{0}, δ = 0),

where both the score function V and the information matrix F are evaluated at the restricted estimate $({\hat{α}}_{0}, δ = 0)$ . Thus, $δ$ need not be estimated at all. This is a quite remarkable feature of the score test, which sets it apart from others. A Wald test statistic is given by

T_{4} = {\hat{δ}}_{*}^{⊤} Σ (\hat{α}, \hat{δ}) {\hat{δ}}_{*},

where the notation ${\hat{δ}}_{*}$ is used to indicate that the respective vector is assumed to contain only the estimates of the free $δ$ parameters, that is, the first $k - 1$ since $δ_{k} = 0$ (for identifiability). The matrix $Σ$ is composed of the last $k - 1$ rows and the last $k - 1$ columns of the complete covariance matrix $F^{- 1}$ , that is, $Σ$ is the asymptotic covariance matrix of ${\hat{δ}}_{*}$ . It is evaluated at the unrestricted CML estimates $(\hat{α}, \hat{δ})$ . Finally, a gradient test statistic is obtained by

T_{5} = V_{2}^{⊤} ({\hat{α}}_{0}, δ = 0) {\hat{δ}}_{*},

where V ₂ is evaluated at the restricted estimates. The four test statistics have a common limiting distribution when $(τ, α, δ) \in Θ_{3}$ holds true and $n \to \infty$ . It is the central $χ^{2}$ distribution with $d f = k - 1$ .

Note that the gradient test is a relatively recent development and thus is still little known in the context of psychological and educational research. To the best of the authors’ knowledge, a discussion of the gradient test in respect of psychometric problems can only be found in two papers. Draxler et al. (2022) discuss it in a CML and Zimmer et al. (2022) in a marginal maximum likelihood framework. It shows similar asymptotic and finite sample size properties and performance to the other three likelihood-based tests not only in a psychomeric but also in a much broader context (Lemonte, 2016; Terell, 2002). The test statistic can be derived from a combination of Rao score and Wald test statistics. One of the most striking features is that, unlike Rao score and Wald test statistics, it does not depend on an information matrix, which can entail computational and practical advantages in several scenarios and models. A peculiarity of it is that the test statistic is not necessarily nonnegative in finite sample sizes (since the test statistic is composed of the scalar product of two vector-valued random variables) but it is asymptotically.

All four test statistics of the modified approach are accessible through the R package tcl (Draxler & Kurz, 2023).

2.4. Rationale for the Restricted Model

The more parsimonious parametrization in the restricted model implies more information concerning the parameters of interest, that is, the $δ$ parameters. The linear restriction that allows the logit of a correct response to differ between successive person score groups only by a constant (which is the $δ$ parameter) models the concept of the discrimination of an item more efficiently compared to the general model. The general model that assumes for each person score group a separate item parameter ( $β$ parameter) considers all potential differences in the logit of a correct response between successive person score groups, for instance, the difference between score groups 1 and 2 can be greater than between 2 and 3. Thus, unlike the restricted model, the general model does not only consider the logit to be linear in r. The price for this generality is less information in the sample and thus less power of statistical tests against the (specific) alternative of different item discriminations (but, of course, it has power against a more general alternative).

If the total sample is divided only into two person score groups (e.g., low vs. high), as it is often the case in practice of data analysis as noted in Section 2.1, and thus, one obtains only two vectors of $β$ parameters, of which only one is free (because they are compared and one is used as a baseline) potential differences (in the $β$ parameters) between different person scores within each of the two groups of persons will be ignored and not adequately modeled for. This yields again a loss of information compared to the restricted model, where a constant difference per item between all successive person score groups is allowed for.

An obvious question concerns the similarity of the restricted model given by (1) and the well-known two parameter logistic (2PL) model (Birnbaum, 1968), which is typically applied in scenarios of different item discriminations. The 2PL model assumes not only a parameter for each person and a parameter for each item but a discrimination parameter for each item as well. A brief discussion on the similarity of the two models is as follows. Consider the linear predictor for a person with parameter $τ$ and an item with parameter $β$ in the 2PL model

ψ (τ + β),

where $ψ > 0$ is the discrimination parameter of that item, and the respective linear predictor in the restricted model

τ + α + δ (r - 1) .

Consider the difference in the linear predictor between two arbitrary persons with person parameters $τ_{1}$ and $τ_{2}$ , say, that obtain the person scores r ₁ and r ₂. Assume that $r_{1} < r_{2}$ and $ψ \neq 1, δ \neq 0$ . It is obvious that in case of $ψ = 1, δ = 0$ , both models must be equivalent since both reduce to the Rasch model. Then, one obtains for the 2PL model

ψ (τ_{2} + β) - ψ (τ_{1} + β) = ψ (τ_{2} - τ_{1}),

and for the linearly restricted model

τ_{2} + α + δ (r_{2} - 1) - τ_{1} - α - δ (r_{1} - 1) = τ_{2} - τ_{1} + δ (r_{2} - r_{1}) .

Setting these two differences equal yields a condition for the two models to be equivalent, that is

\frac{(ψ - 1)}{δ} (τ_{2} - τ_{1}) = r_{2} - r_{1} .

Hence, if the difference between the scores of any two persons is proportional to the difference in their person parameters, the two models will be equivalent. In case of $r_{2} - r_{1} = 1$ , the corresponding difference in person parameters must be $δ / (ψ - 1)$ ; in case of $r_{2} - r_{1} = 2$ , it must be $2 δ / (ψ - 1)$ ; and so on. For successive person score groups, the person parameters must increase in discrete steps given by $δ / (ψ - 1)$ . Note that the constant $δ / (ψ - 1)$ must be the same for every item, even though items differ in their $δ$ and $ψ$ parameters from other items. Thus, the relationship between the person parameters and the person scores must be perfectly linear in order to obtain exact equivalence of the two models. Otherwise, the restricted model can be viewed as an approximation of the 2PL model.

2.5. Remarks on Generalizations

The extension of the modified procedure introduced so far to more general psychometric models, which consider the responses of persons to items in more than two ordered (or unordered, i.e., nominal) categories, is basically straightforward. An example of one of the most popular and frequently applied models is the partial credit model (Masters, 1982). It assumes that all response categories have the same discrimination for all items. This assumption can be tested along the same lines as in the binary Rasch model.

Let $(ϑ, θ) \in Θ$ , where $ϑ$ is a vector of nuisance parameters and $θ$ a vector of parameters of interest. In psychometrics, the former usually represents the characteristics of the persons, that is, person parameters, and the latter is a vector of item parameters associated with characteristics of items and/or their response categories. Let Q be a vector of sufficient statistics for $ϑ$ and W a vector of sufficient statistics for $θ$ , both being functions of the data $Y = y$ , that is, responses of persons to items which are assumed to be nonnegative integers. Consider the following general form of a psychometric model from a multiparameter exponential family given by

P (Y = y) = C (ϑ, θ) exp (ϑ^{⊤} q + θ^{⊤} w) h (y) .

Assume a division of the sample of persons according to the potential values an element of Q can take. Let q denote any of these potential values. Thus, q is equivalent to r in the binary case and therefore may also be called person score. Uninformative values are again excluded. In the partial credit model, for example, the person score $q = 0$ as well as the maximum value of the person score (i.e., when a person responds in the highest response category for all items) are uninformative. Assume the vector of the parameters of interest to differ between the groups of persons defined this way, that is, one yields a person score group specific vector of item/category parameters denoted by $θ_{q} \forall q$ . The complete vector $θ$ is accordingly composed of all person score group specific vectors. Let

θ_{q} = η + λ (q - 1) \forall q

be the linear decomposition that allows the person score group specific item/category parameters to differ between successive person score groups only by constants represented by the vector $λ$ . Let W ₁ and W ₂ denote the vectors of sufficient statistics for $η$ and $λ$ . Then, the distribution of Y, the joint distribution of all sufficient statistics, the marginal distribution of the sufficient statistics for the nuisance parameters, the conditional distribution of W ₁ and W ₂ given $Q = q$ , and the conditional log likelihood function are obtained by

P (Y = y) = C (ϑ, η, λ) exp (ϑ^{⊤} q + η^{⊤} w_{1} + λ^{⊤} w_{2}) h (y),

P (Q = q, W_{1} = w_{1}, W_{2} = w_{2}) = C (ϑ, η, λ) exp (ϑ^{⊤} q + η^{⊤} w_{1} + λ^{⊤} w_{2}) h (q, w_{1}, w_{2}),

P (Q = q) = C (ϑ, η, λ) exp (ϑ^{⊤} q) h (q),

P (W_{1} = w_{1}, W_{2} = w_{2} | Q = q) = \frac{P (Q = q, W_{1} = w_{1}, W_{2} = w_{2})}{P (Q = q)}

= C (η, λ) exp (η^{⊤} w_{1} + λ^{⊤} w_{2}) h (w_{1}, w_{2} | q),

ℓ (η, λ) = η^{⊤} w_{1} + λ^{⊤} w_{2} + log (C (η, λ)) .

Of interest is to test the hypothesis $λ = 0$ against the alternative $λ \neq 0$ (i.e., equality of person score group specific item/category parameters between those groups), and the four test statistics are obtained along the same lines as in the binary case.

3. Monte Carlo Design

To investigate finite sample size properties of the four tests of the modified procedure, Monte Carlo experiments are carried out. This includes the approximation of the limiting distribution and type I and type II error or power rates. Compared to the classical ALRT, an improvement of the power of the tests is expected. Various practically relevant scenarios are considered to get an idea of how much of an increase of power can be expected with sufficient accuracy.

3.1. Fundamentals of the Design

The design considers different numbers of persons and items, that is

n_{*} = 100, 200, 300, 500, 750, 1, 000, 1, 500, 2, 000,

k = 5,7,9,11,15,21,31.

Note that $n_{*}$ denotes the number of all persons, also the ones with a person score $r = 0, k$ (number of correct responses). Recall that these two have been excluded in Section 2 since they are completely uninformative (they do not influence the value of the test statistic). Thus, the total informative sample sizes (from which the test statistics are computed) are likely to be smaller than the respective $n_{*}$ used to generate the data in each scenario (since the numbers of persons with a particular r, except for one r, are random variables given $n_{*}$ ).

The data have been generated using the 2PL model. The item parameters have been chosen from an interval of $- 2$ to 2. Concerning the discrimination parameters, the following choices have been made: The first assumes (for every combination of values of $n_{*}$ and k) all discr. parameters equaling 1 (for all items), the second assumes one discr. parameter deviating from 1, where the deviating parameter is referring either to an extreme item (i.e., one with the smallest or largest item par.) or a middle item with parameter being 0, and the third considers two discr. parameters deviating simultaneously from 1 in opposite directions, where the two parameters are referring to an extreme and a middle item. The absolute deviations from 1 are increased in $0.1$ steps until the power of the tests is close to 1 (which, of course, depends on the respective sample size scenario), that is, $0.9, 0.8, 0.7$ , and so on and $1.1, 1.2, 1.3$ , and so on.

Additionally, scenarios are considered, in which all discrimination parameters vary in four different ranges: from 0.9 to 1.1, from 0.8 to 1.2, from 0.7 to 1.3, and from 0.6 to 1.4. Thus, in these cases, each single item differs in its discrimination from each other item, which are very natural and realistic scenarios. The exact values of the discrimination parameters are shown in Online Appendix B.

The person parameters have been drawn from the standard normal or Gaussian distribution $N (0, 1)$ in most scenarios. Exceptions are the consideration of a rectangular distribution with support from $- 3$ to 3, that is, $U (- 3, 3)$ , and a mixed Gaussian distribution assuming for half of the persons $N (- 1.25, 0.5)$ and for the other half $N (1.25, 0.5)$ (the second parameter being the standard deviation). These two cases of person parameter distributions have only been considered in scenarios with all discrimination parameters set to 1. The number of Monte Carlo replications has been set to $10^{4}$ in every scenario.

The test statistics computed for every replication in each scenario are the classical ALRT T ₁, and the four based on the suggested modified approach: T ₂ (LRT), T ₃ (Rao score), T ₄ (Wald), and T ₅ (gradient). Note that in respect of the computation of the ALRT T ₁, the sample is divided only into two score groups, that is, below (or equal to) the median of the frequency distribution of the person scores and above.

Additionally, a likelihood ratio test based on a marginal likelihood procedure that (directly) compares Rasch and 2PL models is considered. The CML procedure is not applicable in this case. Instead, a marginal ML approach is suitable, which requires the predetermination of a parametric family of distributions for the person parameters. Hence, such a marginal likelihood is not a function of the individual person parameters but the parameters of the assumed distribution and the item and discrimination parameters (individual person parameters need not be estimated). In this work, the standard normal distribution (Gaussian) is assumed (for the person parameters). Thus, the marginal likelihood is only a function of the k item and k discrimination parameters. In respect of the Rasch model, which is a special case of the 2PL by setting all discrimination parameters equal, the marginal likelihood is then a function of the k item parameters and a common discrimination parameter for all items, that is, it is a function of $k + 1$ parameters. The respective likelihood ratio test has therefore $2 k - (k + 1) = k - 1$ degrees of freedom. Thus, its asymptotic distribution is the same as for all the other tests when all discrimination parameters are equal. This test is an obvious alternative outside of the conditional likelihood framework. The motivation to include it is to get an idea of the relative suitability of the CML-based approach of modeling different item discriminations in comparison to the 2PL model in realistic scenarios, in which the two models are not exactly equivalent (i.e., when the condition derived in the section on the rational of the restricted model is not met). The question is also whether the difference even matters in practice of psychometric analyses.

Furthermore, the so-called information criteria or indices that are derived from information theory are considered. Such an approach does not provide a hypothesis test with error probabilities. The indices are widely used in general statistical modeling problems, that is, in selecting an (empirically) appropriate model. They are also popular in psychometrics. Thus, its worthwhile to consider them too. Some specifics and technical details on these are provided in the next subsection.

An overview of the Monte Carlo study design is given in Online Appendix B, and the complete code used for all computations can be downloaded from an online repository https://github.com/canguerer/JEBS_item_discrimination_cml.

3.2. Specifics on Information Criteria

Two well-known and well-established indices are the so-called Akaike information criterion (AIC; Akaike, 1974) and Bayes’s information criterion (BIC; Schwarz, 1978). They are given by

AIC = - 2 ℓ (α, δ) + 2 p, BIC = - 2 ℓ (α, δ) + log (n) p,

where p denotes the number of free parameters of the model. They add to the conditional likelihood a penalizing term that is a function of the number of parameters and, in case of the BIC, additionally a function of the sample size. These indices are computed for both the model allowing for different item discriminations given by (1) and the Rasch model, which is a special case by setting $δ = 0$ . In case of the former, the conditional likelihood function $ℓ (\cdot, \cdot)$ is evaluated at the CML estimate $(\hat{α}, \hat{δ})$ and $p = 2 (k - 1)$ . In case of the Rasch model, $ℓ (\cdot, \cdot)$ is evaluated at the restricted estimate $(\hat{α}, δ = 0)$ and $p = k - 1$ . That model that yields a smaller value of the respective information criterion is considered to be empirically more appropriate. For instance, when the difference (of the respective information criterion) between the model given by (1) and the Rasch model is $< 0$ , model (1) is selected and the Rasch model is rejected. In respect of the AIC, one obtains

\overset{AIC for model(1)}{\overset{︷}{- 2 ℓ (\hat{α}, \hat{δ}) + 2 \cdot 2 (k - 1)}} - \overset{AIC for Rasch model}{\overset{︷}{(- 2 ℓ (\hat{α}, δ = 0) + 2 (k - 1))}} < 0

- 2 ℓ (\hat{α}, \hat{δ}) + 2 ℓ (\hat{α}, δ = 0) + 2 (k - 1) < 0

- 2 (ℓ (\hat{α}, δ = 0) - ℓ (\hat{α}, \hat{δ})) - 2 (k - 1) > 0

T_{2} - 2 (k - 1) > 0,

T_{2} > 2 (k - 1) .

Hence, the difference in AIC values between the two models can be expressed as a function of the likelihood ratio test statistic T ₂ and the number of items k. When T ₂ is greater than the number of free or estimated parameters of model (1), the AIC decides in favor of (1) and rejects the Rasch model. In respect of the BIC, one obtains

T_{2} > log (n) (k - 1) .

Thus, the BIC selects model (1) when T ₂ is greater than $log (n) (k - 1)$ .

When comparing the likelihood ratio test T ₂ with the AIC in respect of selection rates of model (1), the following can immediately be derived. The test T ₂ selects model (1), that is, it rejects the hypothesis of equal item discriminations, when the test statistic T ₂ is greater than the $100 (1 - α)$ th percentile of the $χ^{2}$ distribution with $d f = k - 1$ . The AIC makes the same choice when T ₂ is greater than $2 (k - 1)$ . Setting, for instance, the probability of the error of the first kind $α = 0.05$ , the respective $95$ th percentile is greater than $2 (k - 1)$ as long as $k < 9$ . Hence, the AIC chooses model (1) more frequently (with higher probability) than the test T ₂. Expressed in terms of type I error and power rates, this means that both are higher for the AIC. When $k \geq 9$ , the $95$ th percentile is smaller than $2 (k - 1)$ , and thus, the AIC has lower probability of rejecting the hypothesis of equal item discriminations or choosing model (1) than the test T ₂. Accordingly, it then has smaller type I error and power rates.

In respect of the BIC, it follows from analogous considerations that it is, in all scenarios considered (regarding n and k), a more conservative procedure than the test T ₂ and the AIC, that is, lower selection rates of model (1). It simply penalizes much more than the AIC.

All these considerations apply in the same manner when analyzing selection rates of the 2PL compared to the Rasch model by AIC and BIC based on the marginal likelihood function. Hence, these selection rates are also computed and compared with type I error and power rates of the likelihood ratio test based on the marginal likelihood (that directly compares Rasch and 2PL models).

4. Results

The complete results are uploaded in an online repository and can be downloaded from https://github.com/canguerer/JEBS_item_discrimination_cml. The main results are threefold.

The first part refers to the observation that the common limiting distribution of the test statistics assuming all item discriminations as equal is approximated fairly well already for sample sizes from about $300$ . In particular, no indications of a worse performance of the four tests based on the modified approach compared to the classical ALRT are found. As expected from asymptotic theory, the approximation improves with an increasing sample size. Accordingly, observed type I error rates of the tests are practically sufficiently close to any predetermined levels of the probability of the error of the first kind $α$ . An exception is the likelihood ratio test statistic based on the marginal ML approach in those scenarios assuming the true distribution of person parameters not being $N (0, 1)$ in the data generation process. This is, of course, also an expected result. Tables 1 and 2 show some typical examples of these results in more detail, that is, means, variances, and a number of selected quantiles of the observed distributions compared to the respective parameters of the $χ^{2}$ distribution with $d f = k - 1$ .

Table 1.

Observed Distributions of the Six Test Statistics Compared to the Limiting Chi Square Distribution With $df = 30$ in Case of $k = 31$ Items and Equal Item Discriminations

Sample Size	Test Statistic	Mean	Variance	90th Perc.	95th Perc.	99th Perc.
300	χ²	30.00	60.00	40.26	43.77	50.89
	ALRT	30.29	61.65	40.78	43.95	50.99
	Mod.LRT	30.24	60.83	40.54	43.93	50.99
	Score	30.09	59.97	40.43	43.65	50.75
	Wald	29.69	56.87	39.77	42.91	49.72
	Gradient	30.50	62.98	40.91	44.45	51.98
	Marg.LRT	30.36	61.24	40.73	44.14	51.11
750	ALRT	30.01	59.86	40.30	43.91	50.40
	Mod.LRT	29.96	61.31	40.34	43.70	51.49
	Score	29.89	60.83	40.09	43.66	51.30
	Wald	29.73	59.55	39.81	43.40	50.85
	Gradient	30.06	62.22	40.48	43.93	51.86
	Marg.LRT	30.00	61.51	40.25	43.85	51.71
1,000	ALRT	30.18	60.48	40.45	43.99	50.77
	Mod.LRT	30.13	59.99	40.41	44.13	50.97
	Score	30.09	59.68	40.32	44.08	50.80
	Wald	29.97	58.75	40.13	43.83	50.50
	Gradient	30.20	60.63	40.56	44.32	51.04
	Marg.LRT	30.14	59.92	40.40	44.06	50.85
2,000	ALRT	30.16	60.61	40.48	43.79	51.33
	Mod.LRT	30.10	60.73	40.39	43.76	51.84
	Score	30.09	60.59	40.29	43.70	52.03
	Wald	30.04	60.19	40.18	43.70	51.90
	Gradient	30.15	61.05	40.45	43.83	51.79
	Marg.LRT	30.12	60.88	40.38	44.05	51.73

Note. ALRT is used as an abbreviation for Andersen’s likelihood ratio test; Mod.LRT for the likelihood ratio test based on the modified approach; Score, Wald, and Gradient for the respective other tests of the modified approach; Marg.LRT for the likelihood ratio test based on the marginal maximum likelihood procedure; and Perc. for percentile.

Table 2.

Observed Distributions of the Six Test Statistics Compared to the Limiting χ² Distribution With $df = 14$ in Case of $k = 15$ Items, $n_{*} = 1,500$ Persons, and Equal Item Discriminations

Test Statistic	Mean	Variance	90th Perc.	95th Perc.	99th Perc.
χ²	14.00	28.00	21.06	23.68	29.14
ALRT	14.06	28.67	21.23	23.85	29.35
Mod.LRT	14.07	28.65	21.15	23.83	29.52
Score	14.05	28.50	21.11	23.71	29.41
Wald	13.99	28.07	21.00	23.59	29.20
Gradient	14.11	28.98	21.25	23.88	29.54
Marg.LRT	17.26	42.41	25.95	29.20	35.72
ALRT	14.05	28.14	21.22	23.89	28.86
Mod.LRT	13.94	27.64	21.00	23.44	28.93
Score	13.93	27.60	20.98	23.47	28.96
Wald	13.88	27.23	20.90	23.35	28.78
Gradient	13.97	27.88	21.04	23.50	28.95
Marg.LRT	20.41	53.33	30.15	33.41	40.86

Note. The upper block refers to results obtained from the assumption of a rectangular distribution for the true distribution of person parameters and the lower block to results referring to a mixed Gaussian distribution. ALRT is used as an abbreviation for Andersen’s likelihood ratio test; Mod.LRT for the likelihood ratio test based on the modified approach; Score, Wald, and Gradient for the respective other tests of the modified approach; Marg.LRT for the likelihood ratio test based on the marginal maximum likelihood procedure; and Perc. for percentile.

In scenarios with sample sizes of $100$ and $200$ rather conservative results have been observed, that is, rejection rates of the hypothesis of equal item discriminations being below the assumed level $α$ . It is worse in scenarios where the deviating item is an extreme item. The Wald test is more affected than the other tests of the modified procedure. This is, of course, at least partly to be in line with and expected from research literature on the Wald test. The gradient test and the likelihood ratio test based on the marginal likelihood seem to perform best in this regard. The ALRT also delivers too low rejection rates. It seems that the approximation of the respective limiting $χ^{2}$ distribution is rather poor in these scenarios. More specifically, the observed distributions (from the $10^{4}$ replications) of the test statistics have smaller means and smaller variances compared to the respective $χ^{2}$ . These problems disappear with larger sample sizes.

The second part of the results refers to observed type I error and power rates of the tests. The comparison of the modified tests and the ALRT unveils an explicit picture. The modified tests outperform the ALRT in all scenarios considered. When the power of the tests is neither close to 0 nor 1, a gain of power of up to $0.3$ is observed. Table 3 gives a typical example of these results. No remarkable differences are found between the four different test statistics (T ₂, T ₃, T ₄, and T ₅) based on the modified approach. The observed selection rates (type I error and power rates) by AIC are in line with the theoretical considerations in Section 3.2. They are all lower than those for the tests since $k = 21$ (and thus, greater than 8). The selection rates of the BIC are not shown in Table 3 because they are all very close to 0.

Table 3.

Observed Type I Error and Power Rates in Percentages Referring to Scenarios of $k = 21$ Items

Sample Size	Discr. Par.	Test Statistics						AIC Selection Rates
		ALRT	Mod.LRT	Score	Wald	Gradient	Marg.LRT	model (1)	2PL
500	1.0, 1.0	5.63	5.51	5.50	5.01	5.76	5.65	0.7	0.7
	0.9, 1.1	6.60	7.82	7.53	6.92	8.15	7.59	1.0	1.0
	0.8, 1.2	10.08	13.87	13.84	12.88	14.28	14.14	2.2	2.5
	0.7, 1.3	17.61	29.71	30.16	28.53	30.16	30.28	8.2	8.2
	0.6, 1.4	31.31	54.28	55.64	53.76	54.31	54.89	23.2	23.7
	0.5, 1.5	49.55	79.08	80.59	79.53	78.70	79.85	50.3	51.2
750	1.0, 1.0	5.06	5.13	5.01	4.70	5.29	5.14	0.5	0.5
	0.9, 1.1	6.67	8.42	8.27	7.83	8.68	8.58	1.1	1.1
	0.8, 1.2	12.56	18.93	19.19	18.22	19.36	19.51	3.9	4.0
	0.7, 1.3	25.36	45.32	46.01	44.88	45.26	45.86	16.7	17.2
	0.6, 1.4	47.46	76.48	77.82	76.63	75.97	77.38	46.5	47.3
	0.5, 1.5	72.94	95.32	95.89	95.46	95.09	95.53	80.5	81.3
1,000	1.0, 1.0	5.41	5.37	5.27	5.03	5.53	5.34	0.6	0.6
	0.9, 1.1	7.08	8.76	8.61	8.35	8.95	9.07	1.2	1.2
	0.8, 1.2	15.62	25.36	25.51	24.84	25.60	25.95	6.3	6.6
	0.7, 1.3	34.76	60.34	61.26	60.44	60.23	61.01	27.9	28.6
	0.6, 1.4	64.40	90.38	91.12	90.76	90.06	90.72	68.3	69.4
	0.5, 1.5	86.95	99.15	99.24	99.18	99.09	99.19	93.9	94.2

Note. The discrimination parameters of two items (middle and extreme) are varied. All other discrimination parameters are set to 1. The assumed $α$ -level of the tests is $0.05$ . ALRT is used as an abbreviation for Andersen’s likelihood ratio test; Mod.LRT for the likelihood ratio test based on the modified approach; Score, Wald, and Gradient for the respective other tests of the modified approach; and Marg.LRT for the likelihood ratio test based on the marginal maximum likelihood procedure. AIC = Akaike information criterion; 2PL = two parameter logistic.

The third part of the results refers to a comparison of the four modified tests (based on CML) with the LRT based on marginal ML (directly comparing Rasch and 2PL models). As has been expected, the marginal LRT performs slightly better in respect of observed power rates in case of small numbers of items. For larger item numbers, hardly any differences that are of practical relevance are observed any more. Furthermore, Table 2 shows the results of scenarios, in which the marginal likelihood-based LRT is severely biased and practically useless, that is, when the distribution of the person parameters is misspecified. In such scenarios, the CML-based tests are undoubtedly preferable.

5. Real Data Examples

Finally, two real data examples from educational research have been considered, which are contained in the R package sirt (Robitzsch, 2022). The names of the files are data.pisaMath.rda and data.pisaRead.rda. The former contains binary responses of 565 students to 11 mathematics items and the latter binary responses of 623 students to 12 reading items. The results shown in Table 4 confirm the main observations from the Monte Carlo investigation and what is to be expected from theory. The p values are all $< .001$ except for the ALRT in the reading example ( $χ^{2} = 25.29 and d f = 11$ ), where it is $.008$ . As can be seen, only the BIC being the most conservative procedure (i.e., preferring models with less parameters) and only in the reading example selects the Rasch model as the one to be empirically more appropriate (in both frameworks CML and marginal ML).

Table 4.

Results of Real Data Examples

	ALRT	Mod.LRT	Score	Wald	Gradient	Marg.LRT
Math data	46.25	73.03	76.72	73.47	72.43	74.21
Reading data	25.29	34.81	36.05	34.46	34.77	43.55
	${AIC}_{(1)}$	${AIC}_{RM}$	${BIC}_{(1)}$	${BIC}_{RM}$
Math data	4,800.45	4,853.48	4,885.91	4,896.21
Reading data	3,521.84	3,534.65	3,618.90	3,583.18
	${AIC}_{Marg . (1)}$	${AIC}_{Marg.RM}$	${BIC}_{Marg . (1)}$	${BIC}_{Marg.RM}$
Math data	7,519.44	7,573.64	7,614.85	7,625.69
Reading data	6,265.65	6,287.20	6,372.08	6,344.85

Note. AIC and BIC based on conditional likelihood computed for both model (1) and Rasch model. AIC and BIC based on marginal likelihood computed for both 2PL and Rasch model. ALRT is used as an abbreviation for Andersen’s likelihood ratio test; Mod.LRT for the likelihood ratio test based on the modified approach, Score; Wald, and Gradient for the respective other tests of the modified approach; and Marg.LRT for the likelihood ratio test based on the marginal maximum likelihood procedure. AIC = Akaike information criterion; 2PL = two parameter logistic; BIC = Bayes’s information criterion.

Regarding the mathematics example, the estimates of the $δ$ parameters reveal that items 4, 7, and 8 have distinctly higher discriminations than the other items. Concerning the reading example, the $δ$ estimates of items 1 and 5 are farthest from 0, where the estimate of item 1 is negative and that of item 5 is positive. Thus, item 1 has lower and item 5 higher discrimination than the rest of items. The estimates of the other items are closer to 0. All estimates and their standard errors are shown in Table 5.

Table 5.

Estimates With Standard Errors of Real Data Examples

Item	Math Data	Reading Data
1	.045 (.069)	$-$ .223 (.104)
2	$-$ .052 (.073)	$-$ .164 (.093)
3	$-$ .150 (.081)	.054 (.144)
4	.304 (.065)	.137 (.134)
5	.004 (.071)	.207 (.080)
6	.098 (.071)	$-$ .126 (.096)
7	.204 (.064)	$-$ .052 (.089)
8	.206 (.064)	$-$ .051 (.091)
9	.009 (.071)	.040 (.085)
10	.082 (.068)	$-$ .013 (.092)
11	0	.089 (.107)
12		0

Note. In both data examples, the last $δ$ parameter is set to 0 for identifiability.

6. Final Remarks

The linearly restricted model and the modified approach of investigating item discriminations in the CML framework presented in this work can be viewed as a special case of a model already utilized on a number of occasions, for example, by Gürer and Draxler (2022) as well as Draxler et al. (2022), when the person score (number of correct responses of a person) is considered as a covariate (or predictor or explanatory variable) and is assumed to have a linear effect on the log odds of response probabilities. As already noted in Section 2.2, the modified procedure is compatible with conditioning not only on person but also on item scores, that is, column sums of the response matrix. This procedure eliminates both $τ$ and $α$ parameters (all nuisance parameters). The conditional likelihood is then only a function of the parameters of interest, that is, the $δ$ parameters. Such an approach has already been discussed by Draxler and Zessin (2015), Draxler and Nolte (2018), Draxler and Dahm (2020), Draxler and Kurz (2021), and Draxler (2018), where the latter also incorporates Bayesian considerations. A practical drawback with such a procedure is the computational burden. It requires either the exact or approximate determination of (huge) numbers of matrices and respective conditional distributions (Miller & Harrison, 2013; Verhelst, 2008). Therefore, it is rather recommended for smaller sample sizes and item numbers, a few hundreds of persons and less than 30 items, say.

The case of missing data may also briefly be discussed. Basically, one can distinguish between data not missing at random, for example, by design of the investigation, and data missing at random. The latter entails scenarios in which data are missing completely at random and missing at random (conditional on covariates). Generally, the modified approach can be applied in missing data scenarios in so far as the classical approach of comparing different person score groups by Rasch (1960) and Andersen (1973) is appropriate. In case of data missing completely at random or missing data depending on any covariates unrelated to the person parameters and thus missing at random conditional on the covariates, its application is straightforward. If the frequencies of missing responses depend on the person parameters, one may yield biased results and/or a reduction of the power of the tests. Consider, for instance, that persons with higher ability tend to have more missing responses and consequently obtain lower person scores than persons with lower ability but fewer missing responses. In such a case, the dependence between person parameters and person scores may not be monotone any more. A further discussion on the suitability of the modified approach of investigating item discriminations in missing data scenarios (also missing by design) shall be kept for another occasion.

The outperformance of the modified procedure compared to the classical approach (ALRT) by Andersen (1973) in terms of power rates of the statistical tests is undoubtedly impressive. Nevertheless, a brief word of caution in respect of the practice of psychometric analyses is in order. Unlike the ALRT, the modified procedure is only suggested in case of evaluating item discriminations, that is, testing the hypothesis of equal discriminations for all items against the alternative that at least one item possesses a different discrimination (than the others). The higher power rates are achieved by specifying a subset of probability distributions in respect of the alternative hypothesis. In case of the ALRT, the alternative hypothesis is more general. Hence, one can expect a reasonable power of the ALRT against other violations of assumptions of the Rasch model as well, not only a violation of equal item discriminations. Therefore, it is reasonable to suggest both the modified tests (or at least one of the four) and the ALRT (or another asymptotically equivalent $χ^{2}$ test) in a typical application of the Rasch model, which usually not only deals with the assumption of equal item discriminations but other assumptions like unidimensionality as well.

When the discriminations of the items are not equal (and thus, the Rasch model is not applicable), one has several choices of models considering a discrimination parameter. Most popular are the well-known 2PL and 3PL models (Birnbaum, 1968) or, in case of ordinal responses in more than two categories, the generalized partial credit model (Muraki, 1992). The CML approach is not compatible with these examples of models since they are not of a multiparameter exponential family. As an alternative, one may consider a less common model, that is, the so-called one parameter logistic model (OPLM) for binary and ordinal responses (Verhelst & Glas, 1995; Verhelst et al., 1994). The only difference to the 2PL model and the generalized partial credit model is that the discrimination parameters are treated as given (or known) constants called discrimination indices. Thus, they need not be estimated. In the binary case, the OPLM has only one parameter per person and one parameter per item just like the Rasch model, and consequently, the sufficient statistic for the person parameters is obtained as a weighted sum of correct responses for each person, where the given discrimination indices are used as weights. Thus, one can condition on the observed values of these weighted scores, eliminate the person parameters from the conditional distribution, and use CML. The practical problem is the specification of the discrimination indices, which are usually unknown. Information on the items’ discriminations can be obtained from the score function V ₂, for instance. It has length $k - 1$ , where each element represents an item indicating the relative discriminations of the items. It may be used to specify the discrimination indices for the OPLM.

The final remark is in respect of the conditional approach to statistical inference, which is at the core of this work. The strongest philosophical and theoretical argument for conditioning on a statistic is met when the statistic is ancillary (e.g., Lehmann & Romano, 2005), that is, when its probability distribution does not depend on the parameters of interest (in this case, the $δ$ parameters). Obviously, this is not the case in the Rasch modeling framework. The distribution of R does depend on the $δ$ parameters. Another argument is more of a technical nature. It is concerned with the asymptotic properties of the CML estimates. It is the well-known incidental parameter (in the present case, the $τ$ parameters) problem. Neyman and Scott (1948) showed that the maximum likelihood estimates of structural parameters (in the present case, the $α$ and $δ$ parameters) need not be consistent when the likelihood is a function of both incidental and structural parameters. One of a number of solutions to this problem is CML, of course. Marginal ML also provides a solution but at the cost of an assumption on the true distribution of the incidental parameters. A third argument on the problem of conditioning has extensively been discussed in the psychometric and Rasch modeling literature. It implies that the item parameters are estimated independently of the person parameters, that is, one obtains consistent and asymptotically unbiased estimates. A further extensive discussion on the CML approach has been given by Skrondal and Rabe-Hesketh (2022).

Supplemental Material

Supplemental Material, sj-pdf-1-jeb-10.3102_10769986231183335 - An Improved Inferential Procedure to Evaluate Item Discriminations in a Conditional Maximum Likelihood Framework

Supplemental Material, sj-pdf-1-jeb-10.3102_10769986231183335 for An Improved Inferential Procedure to Evaluate Item Discriminations in a Conditional Maximum Likelihood Framework by Clemens Draxler, Andreas Kurz, Can Gürer and Jan Philipp Nolte in Journal of Educational and Behavioral Statistics

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Can Gürer

References

Akaike

(1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705

Andersen

E. B.

(1970). Asymptotic properties of conditional maximum-likelihood estimators. Journal of the Royal Statistical Society. Series B (Methodological), 32(2), 283–301. https://doi.org/10.1111/j.2517-6161.1970.tb00842.x

Andersen

E. B.

(1973). A goodness of fit test for the Rasch model. Psychometrika, 38(1), 123–140. https://doi.org/10.1007/BF02291180

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–424). Addison Wesley.

Draxler

(2018). Bayesian conditional inference for Rasch models. AStA Advances in Statistical Analysis, 102, 245–262. https://doi.org/10.1007/s10182-017-0303-6

Draxler

Alexandrowicz

R. W.

(2015). Sample size determination within the scope of conditional maximum likelihood estimation with special focus on testing the Rasch model. Psychometrika, 80(4), 897–919. https://doi.org/10.1007/s11336-015-9472-y

Draxler

Dahm

(2020). Conditional or pseudo exact tests with an application in the context of modeling response times. Psych, 2(4), 198–208. https://doi.org/10.3390/psych2040017

Draxler

Kurz

(2021). Conditional inference in small sample scenarios using a resampling approach. Stats, 4(4), 837–849. https://doi.org/10.3390/stats4040049

Draxler

Kurz

(2023). tcl: Testing in conditional likelihood context (0.2.0) [Computer software]. https://CRAN.R-project.org/package=tcl

10.

Draxler

Kurz

Lemonte

A. J.

(2022). The gradient test and its finite sample size properties in a conditional maximum likelihood and psychometric modeling context. Communications in Statistics—Simulation and Computation, 51(6), 3185–3203. https://doi.org/10.1080/03610918.2019.1710193

11.

Draxler

Nolte

J. P.

(2018). Computational precision of the power function for conditional tests of assumptions of the Rasch model. Open Journal of Statistics, 8(6), 873–884. https://doi.org/10.4236/ojs.2018.86058

12.

Draxler

Zessin

(2015). The power function of conditional tests of the Rasch model. AStA Advances in Statistical Analysis, 99(3), 367–378. https://doi.org/10.1007/s10182-015-0249-5

13.

Fischer

G. H.

(1981). On the existence and uniqueness of maximum-likelihood estimates in the Rasch model. Psychometrika, 46(1), 59–77. https://doi.org/10.1007/BF02293919

14.

Fischer

G. H.

Molenaar

I. W.

(1995). Rasch models: Foundations, recent developments, and applications. Springer. https://doi.org/10.1007/978-1-4612-4230-7

15.

Glas

C. A. W.

Verhelst

N. D.

(1995). Testing the Rasch model. In Fischer

G. H.

Molenaar

I. W.

(Eds.), Rasch models: Foundations, recent developments, and applications (pp. 69–95). Springer. https://doi.org/10.1007/978-1-4612-4230-7_5

16.

Gürer

Draxler

(2022). Penalization approaches in the conditional maximum likelihood and Rasch modelling context. British Journal of Mathematical and Statistical Psychology, 00, 1–38. https://doi.org/10.1111/bmsp.12287

17.

Lehmann

E. L.

Romano

J. P.

(2005). Testing statistical hypotheses. Springer. https://doi.org/10.1007/0-387-27605-X

18.

Lemonte

A. J.

(2016). The gradient test: Another likelihood-based test. Academic Press. https://doi.org/10.1016/C2015-0-00195-2

19.

Mair

Hatzinger

(2007). Extended Rasch modeling: The eRm package for the application of IRT models in R. Journal of Statistical Software, 20(9), 1–20. https://doi.org/10.18637/jss.v020.i09

20.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. https://doi.org/10.1007/BF02296272

21.

Miller

J. W.

Harrison

M. T.

(2013). Exact sampling and counting for fixed-margin matrices. The Annals of Statistics, 41(3), 1569–1592. https://doi.org/10.1214/13-AOS1131

22.

Molenaar

I. W.

(1995). Estimation of item parameters. In Fischer

G. H.

Molenaar

I. W.

(Eds.), Rasch models: Foundations, recent developments, and applications (pp. 39–52). Springer. https://doi.org/10.1007/978-1-4612-4230-7_3

23.

Muraki

(1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16(2), 159–176. https://doi.org/10.1177/014662169201600206

24.

Neyman

Scott

E. L.

(1948). Consistent estimates based on partially consistent observations. Econometrica, 16(1), 1–32. https://doi.org/10.2307/1914288

25.

Pfanzagl

(1993). On the consistency of conditional maximum likelihood estimators. Annals of the Institute of Statistical Mathematics, 45(4), 703–719. https://doi.org/10.1007/BF00774782

26.

R Core Team. (2022). R: A language and environment for statistical computing (4.2.0) [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/

27.

Rao

C. R.

(1948). Large sample tests of statistical hypotheses concerning several parameters with applications to problems of estimation. Mathematical Proceedings of the Cambridge Philosophical Society, 44(1), 50–57. https://doi.org/10.1017/S0305004100023987

28.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research.

29.

Robitzsch

(2022). sirt: Supplementary item response theory models (3.12-66) [Computer software]. https://CRAN.R-project.org/package=sirt

30.

Schwarz

(1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. https://doi.org/10.1214/aos/1176344136

31.

Silvey

S. D.

(1959). The Lagrangian multiplier test. The Annals of Mathematical Statistics, 30(2), 389–407.

32.

Skrondal

Rabe-Hesketh

(2022). The role of conditional likelihoods in latent variable modeling. Psychometrika, 87(3), 799–834. https://doi.org/10.1007/s11336-021-09816-8

33.

Terrell

G. R.

(2002). The gradient statistic. Computing Science and Statistics, 34(34), 206–215.

34.

Verhelst

N. D.

(2008). An efficient MCMC algorithm to sample binary matrices with fixed marginals. Psychometrika, 73(4), 705–728. https://doi.org/10.1007/s11336-008-9062-3

35.

Verhelst

N. D.

Glas

C. A. W.

(1995). The one parameter logistic model. In Fischer

G. H.

Molenaar

I. W.

(Eds.), Rasch models: Foundations, recent developments, and applications (pp. 215–237). Springer. https://doi.org/10.1007/978-1-4612-4230-7_12

36.

Verhelst

N. D

Glas

C. A. W.

Verstralen

H. H. F. M.

(1994). One parameter logistic model [Computer program and manual]. CITO.

37.

Wald

(1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society, 54(3), 426–482. https://doi.org/10.2307/1990256

38.

Zimmer

Draxler

Debelak

(2022). Power analysis for the Wald, LR, score, and gradient tests in a marginal maximum likelihood framework: Applications in IRT. Psychometrika. https://doi.org/10.1007/s11336-022-09883-5

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.06 MB