A Didactic Presentation of Snijders’s l z * Index of Person Fit With Emphasis on Response Model Selection and Ability Estimation

Abstract

This paper focuses on two likelihood-based indices of person fit, the index l_z and the Snijders’s modified index l_z *. The first one is commonly used in practical assessment of person fit, although its asymptotic standard normal distribution is not valid when true abilities are replaced by sample ability estimates. The l_z * index is a generalization of l_z , which corrects for this sampling variability. Surprisingly, it is not yet popular in the psychometric and educational assessment community. Moreover, there is some ambiguity about which type of item response model and ability estimation method can be used to compute the l_z * index. The purpose of this article is to present the index l_z * in a simple and didactic approach. Starting from the relationship between l_z and l_z *, we develop the framework according to the type of logistic item response theory (IRT) model and the likelihood-based estimators of ability. The practical calculation of l_z * is illustrated by analyzing a real data set about language skill assessment.

Keywords

person fit asymptotic distribution ability estimation logistic model

1. Introduction

One of the most important problems in educational measurement is to ensure that the responses of an examinee to some specified tests are in agreement with the design of that test. This issue is often called appropriateness measurement or person fit. Two particular person fit patterns should ideally be detected. The first pattern comes up when the examinee tries to perform better than his or her actual ability level, for instance, by cheating during the examination. The other pattern is actually the opposite behavior: It might happen that some examinees with high ability levels are attempting to answer incorrectly to easy items, in order to get lower scores. This astonishing tendency can occur if, for instance, the examinees are classified into groups of various difficulty levels, and their interest is to be classified into an “easier” group to get better chances to perform very well at the graduation. This situation was detected, among others, by Dodeen (2003), Meijer (1998), Raîche and Blais (2003a, 2003b, 2005), and Zickar and Drasgow (1996).

Other abnormal patterns might be detected, such as those of examinees who reply randomly to multiple choice questionnaires or experiencing distraction during the test. Several classifications exist to distinguish between the different types of misfit (e.g., Cronbach, 1946; Ro, 2001; Smith, 1982). However, one is usually interested in detecting these abnormal response patterns first, before a more detailed examination of each particular case (Levine & Drasgow, 1980).

A vast literature about person fit detection methods is available, but most of these refer to the same principle. It consists in computing a person fit index for each examinee and to compare this index with some preestablished threshold to draw a conclusion about the possible misfit. There are two main categories of person fit index: Those based on parametric probability distribution of item responses, usually item response theory (IRT) modeling of the item response and those based on summary statistics from the data set without requiring an IRT solution. We further refer to these categories as the parametric and the nonparametric indices, respectively. For a broad overview of the different indexes, parametric or not, see Karabatsos (2003), Klauer (1995), Levine and Drasgow (1982; 1988), and Meijer and Sijtsma (2001), as well as the first issue of the 1996 volume of Applied Measurement in Education.

The aim of this article is to concentrate on one well-known parametric index of person fit, called the l_z index (Drasgow, Levine & Williams, 1985) and its extended version, the l_z * index (Snijders, 2001). The l_z index is very popular because it is derived from the log likelihood function and is easy to compute. Moreover, l_z has an asymptotic standard normal distribution when the true ability levels are considered (Molenaar & Hoijtink, 1990, 1996), so the detection of person fit can be performed by comparing the values of l_z to some appropriate quantiles of the N(0,1) distribution. In fact, l_z seems to be very powerful to detect aberrant response patterns, with respect to other person fit indices (Armstrong, Stoumbos, Kung, & Shi, 2007; Drasgow, Levine, & McLaughlin, 1987; Li & Olejnik, 1997; Nering & Meijer, 1998).

Unfortunately, the main drawback of l_z is that its asymptotic distribution is not standard normal anymore when the true ability levels are replaced by sample ability estimates (Molenaar & Hoijtink, 1990; Nering, 1995, 1997; Reise, 1995). To overcome this problem, Snijders (2001) proposed a slight modification of the l_z index to obtain the desired asymptotic distribution with sample estimates of ability instead of the true (unknown) values. His index is nowadays referred to as the l_z * index, since it is an extension of l_z . Moreover, Snijders established that l_z * could be computed with various estimators of ability and different IRT models and that the process for correcting the l_z index can actually be adapted to other person fit indices. In the following, and unless explicitly stated, we simply mention Snijders to refer to his 2001 paper.

In this context, it is very surprising that the l_z * index did not become more popular in the last years. For instance, the l_z * index is not considered by Karabatsos (2003) in the comparison of 36 person fit statistics. A brief review of the literature indicated only a few papers that make reference to Snijders’s approach, either by explicitly describing the index (de la Torre & Deng, 2008; van Krimpen-Stoop & Meijer, 1999) or by simply citing it (Emons, 2009; Ferrando, 2004; Meijer & Sijtsma, 2001). Hence, despite of its appealing properties, Snijders’s approach remains largely unknown in the psychometric and educational assessment community.

To our opinion, the main reason for such an ignorance of l_z * is due to the way Snijders’s article was written. This is a rather sophisticated technical article and, although the developments are concise and well explained, they remain hardly accessible for a wide range of applied researchers. A concrete illustration is that the choice of both an IRT model and an estimator of ability imply different calculations for l_z *, and although it is clearly stated by Snijders, it becomes difficult for the practitioner to understand the precise form of the index. Moreover, some texts referring to Snijders’s work are also confusing. For instance, van Krimpen-Stoop and Meijer’s 1999 paper suggests that the distribution of l_z * is established in the particular framework of the two-parameter logistic model and with Warm’s weighted likelihood estimator of ability, while it is only a particular case that was (partly) discussed by Snijders’s. The presentation of l_z * was actually based on a more general framework, where both the IRT model and the estimator of ability, though unspecified, are constrained to fulfill some hypotheses (this is discussed later on in this article).

In sum, the corrected index l_z * is a promising indicator of person fit, because it incorporates the estimator of ability instead of the true ability level, but it is still largely ignored. The purpose of this paper is to palliate the lack of diffusion of l_z *, mostly by proposing a didactic review of Snijders’s approach. First, we set the IRT framework of person fit, by fixing the notations and briefly reviewing the l_z index, using the notations of van Krimpen-Stoop and Meijer (1999) for sake of simplicity. Then, we present the l_z * index from Snijders’s general framework and with connection to the notations for l_z . In the next section, we discuss some possible choices of item response models and estimators of ability, and their relationship with the form of l_z* index. We end with the practical analysis of an example about language skill assessment.

2. IRT Assessment of Person Fit and the l_z Index

Since our main goal is to present some parametric person fit indices, we consider the following framework throughout the paper. A sample of examinees from a targeted population was assigned a test of n items. The underlying model that describes the probability of answering correctly to item i (i = 1, . . . , n) takes the form of the three-parameter logistic (3PL) model (Birnbaum, 1968):

P_{i} (θ) = Pr (X_{i} = 1 | θ, a_{i}, b_{i}, c_{i}) = c_{i} + (1 - c_{i}) \frac{exp [a_{i} (θ - b_{i})]}{1 + exp [a_{i} (θ - b_{i})]} .

The probability function P_i (θ) is also called the item characteristic curve of item i. In Equation 1, X_i is the response of the examinee to item i, coded as zero for an incorrect response and 1 for a correct response, θ is the ability level of the examinee, and a_i , b_i and c_i denote, respectively, the discrimination, the difficulty, and the pseudo-guessing parameters of the item. When the pseudo-guessing parameter c_i is fixed to zero, the 3PL model reduces to the two-parameter logistic (2PL) model. In addition, fixing the discrimination level a_i to unity yields the one-parameter logistic (1PL) or Rasch model.

An important assumption in the person fit framework is that the knowledge about the item parameters is sufficiently good so that they can be considered as fixed values. For instance, this assumption is meaningful if prior study or research was carried out in order to get very precise estimates of these parameters. In other words, the item characteristic curves are completely determined as simple functions of the ability parameter θ only.

Many indices have been proposed to detect person fit, but as already explained, we restrict on the l_z index of Drasgow et al. (1985). To present this index in the simplest and clearest way, we make use of the notations by van Krimpen-Stoop and Meijer (1999). Given a test of n items with fixed item parameters and item characteristic curves described by Equation 1, the likelihood function of any response pattern (X₁ , . . ., X_n ) is given by

L (θ) = \prod_{i = 1}^{n} P_{i} (θ)^{X_{i}} Q_{i} (θ)^{1 - X_{i}},

where Q_i (θ) = 1 − P_i (θ) stands for the probability of an incorrect response. The ML estimator

{\hat{θ}}_{M L}

of θ is obtained by determining the value of θ, which maximizes the likelihood function (Equation 2). For mathematical commodity, one often prefers to maximize the logarithm of the likelihood function, which is subsequently called the log likelihood function and is written as

l_{0} (θ) = log L (θ) = \sum_{i = 1}^{n} \{X_{i} log P_{i} (θ) + (1 - X_{i}) log Q_{i} (θ)\} .

Levine and Rubin (1979) proposed to use the log likelihood function l ₀(θ) as a person fit statistic, which was subsequently denoted by l ₀:

l_{0} = l_{0} (θ) .

Although they refer to the same function, we use the distinct notations l ₀ and l ₀(θ) to refer to the person fit statistic and to the log likelihood function, respectively. In practice, l ₀ is evaluated at some estimate

\hat{θ}

of θ, for instance, the ML estimator.

However, l ₀ is not standardized and its distribution thus depends on θ. Drasgow et al. (1985) proposed the following standardized version of l ₀:

l_{z} = \frac{l_{0} - E (l_{0})}{V (l_{0})^{1 / 2}},

where

E (l_{0})

and

V (l_{0})

are respectively the mean and the variance of l ₀:

E (l_{0}) = \sum_{i = 1}^{n} \{P_{i} (θ) log P_{i} (θ) + Q_{i} (θ) log Q_{i} (θ)\},

and

V (l_{0}) = \sum_{i = 1}^{n} P_{i} (θ) Q_{i} (θ) {(log \frac{P_{i} (θ)}{Q_{i} (θ)})}^{2}

(van Krimpen-Stoop & Meijer, 1999, p. 328). The expectation (Equation 6) and the variance (Equation 7) can be obtained directly using the fact that X_i is a Bernoulli variable with success probability P_i (θ). Drasgow et al. (1985) argued that the l_z index has an asymptotic standard normal distribution. In this framework, “asymptotic” refers to the fact that the test length n is infinite, but in practice, it was shown that the standard normal distribution is accurate for tests of at least 80 items (Drasgow et al., 1985).

However, the asymptotic normality of the l_z index is only valid when the true ability values θ are used to compute this index, but unfortunately the latter are usually unknown. Considering the use of $\hat{θ}$ in the response probability function $P_{i} (\hat{θ})$ , and consequently into the computation of $l_{0}$ , its expectation $E (l_{0})$ and its variance $V (l_{0})$ , it is not easy to predict the shape of the probability distribution of l_z . Several studies, however, showed that the distribution of l_z is negatively skewed and the variance of l_z is smaller than the expected value under the standard normal distribution (Molenaar & Hoijtink, 1990; Nering, 1995, 1997; Reise, 1995). This usually turns to an underdetection of inadequate response patterns, especially when the test length is small to moderate (see also Meijer & Sijtsma, 2001). Moreover, and independently of that issue, Meijer and Nering (1997) and Molenaar and Hoijtink (1990) showed that the asymptotic standard normal distribution is not suitable for tests of 20 to 60 items. However, none of these studies provided a suitable analytical method to circumvent this issue. Only Raîche and Blais (2003a, 2003b, 2005) proposed a computer-intensive, simulation-based approach that is strictly dependent on the item parameters from a specific test to establish the threshold of l_z .

3. The *l_z** Index

In his 2001 article, Snijders discusses first the inadequacy of the distribution of a general class of person fit indices (including l_z ) with estimated ability levels. Then, he proposes a correction for these indices such that, once computed with some convenient estimate of ability, they are asymptotically standard normal distributed. This class of indices is built up on a general framework that incorporates the choice of an IRT model and the selection of an ability estimator. In particular, the corrected version of l_z is nowadays referred to as the l_z* index (van Krimpen-Stoop & Meijer, 1999) since it represents an extension of l_z . To present it, we start from Snijders’s general notations. Whenever appropriate, the connection with the l_z index will be stated.

We start by considering that the l_z index is actually a particular case of the broader class of standardized person fit statistics of the following form:

\frac{W_{n} (θ)}{V [W_{n} (θ)]^{1 / 2}},

where

W_{n} (θ) = \sum_{i = 1}^{n} [X_{i} - P_{i} (θ)] w_{i} (θ),

and where w_i (θ) are suitable weight functions of θ (Snijders, 2001, p. 332). “Suitable” means here that a particular choice of w_i (θ) implies a particular person fit index. Because X_i is a Bernoulli variable with success probability P_i (θ), the statistic W_n (θ) has expectation and variance equal to

E [W_{n} (θ)] = 0 and V [W_{n} (θ)] = n σ_{n}^{2} (θ),

with

σ_{n}^{2} (θ) = \frac{1}{n} \sum_{i = 1}^{n} w_{i} (θ)^{2} P_{i} (θ) Q_{i} (θ)

(Snijders, 2001, pp. 332–335). To establish that l_z takes the form as in Equation 8, the following choice of the weights is appropriate:

w_{i} (θ) = log \frac{P_{i} (θ)}{Q_{i} (θ)} .

Indeed, using respectively Equations 6, 9, and 10 and 7, 10, and 11, the following results are obtained

l_{0} - E (l_{0}) = W_{n} (θ) and V (l_{0}) = n σ_{n}^{2} (θ) = V [W_{n} (θ)] .

Snijders pointed out that, using the true ability level θ, the statistic W_n (θ) is asymptotically normally distributed, so that the person fit statistic in Equation 8 follows a standard normal distribution for large sets of items. This is in agreement with the findings of Drasgow et al. (1985) about l_z , but the form (Equation 8) proposed by Snijders is in fact more general because of the flexibility of the choice of weights w_i (θ). However, the main focus of Snijders’s approach was to study the distribution of W_n (θ) when θ is replaced by some estimate

\hat{θ}

. Such an estimator is constrained to satisfy the following relationship:

r_{0} (\hat{θ}) + \sum_{i = 1}^{n} [X_{i} - P_{i} (\hat{θ})] r_{i} (\hat{θ}) = 0.

The functions r ₀(θ) and r_i (θ) (i = 1, . . ., n) must be chosen in order for the estimator

\hat{θ}

to ensure Equation 14. As it will be shown later, r_i (θ) depends on the selected model, while r ₀(θ) is partly determined by the type of estimator. In addition, for any ML-based estimator, r ₀(θ) corresponds to the derivative of a specific informative or noninformative prior. In sum, any suitable estimator satisfying Equation 14 can be considered in the following developments. For the moment, we simply mention that several well-known estimators fulfill the condition in Equation 14: the ML estimator

{\hat{θ}}_{ML}

, the maximum a posteriori (or Bayes model) estimator

{\hat{θ}}_{MAP}

and Warm’s weighted likelihood estimator

{\hat{θ}}_{WLE}

(Warm, 1989).

In order to derive the distribution of $W_{n} (\hat{θ})$ , several intermediate notations must be introduced. Set first ${P_{i}}^{'} (θ)$ as the first derivative of P′_i (θ) with respect to θ. Define then

c_{n} (θ) = \frac{\sum_{i = 1}^{n} P_{i^{'}} (θ) w_{i} (θ)}{\sum_{i = 1}^{n} P_{i^{'}} (θ) r_{i} (θ)},

and the modified weight functions

{\tilde{w}}_{i} (\hat{θ}) = w_{i} (\hat{θ}) - c_{n} (\hat{θ}) r_{i} (\hat{θ}) (i = 1, \dots, n) .

The basic idea of Snijders consists in replacing the weights in Equation 9 by the modified weights in Equation 16. By doing so, he established that

W_{n} (\hat{θ})

is asymptotically normally distributed with mean and variance given by

E [W_{n} (\hat{θ})] = - c_{n} (\hat{θ}) r_{0} (\hat{θ}) and V [W_{n} (\hat{θ})] = n τ_{n}^{2} (\hat{θ}),

where

τ_{n}^{2} (θ) = \frac{1}{n} \sum_{i = 1}^{n} {\tilde{w}}_{i} (θ)^{2} P_{i} (θ) Q_{i} (θ) .

In other words,

\frac{W_{n} (\hat{θ}) + c_{n} (\hat{θ}) r_{0} (\hat{θ})}{n^{1 / 2} τ_{n} (\hat{θ})}

has an asymptotic standard normal distribution. The variance of

W_{n} (\hat{θ})

has the same form as that of W_n (θ) in Equations 10 and 11, except for the weight functions. Moreover, the mean value of

W_{n} (\hat{θ})

is not necessarily equal to zero, as it is the case for W_n (θ).

In particular, by setting the weights w_i (θ) equal to those in Equation 12, Equation 19 is nothing else than the standardized version of l_z , in other words the l_z* index. At this step, it is useful to rewrite Equation 19 with the notations of van Krimpen-Stoop and Meijer (1999). We make use of the notation $l_{0} (\hat{θ})$ because the function l ₀ is now computed with an estimate $\hat{θ}$ of ability that satisfies Equation 14. Using Equations 11, 13, and 19, the l_z* index is obtained by

l_{z} * = \frac{l_{0} (\hat{θ}) - E [l_{0} (\hat{θ})] + c_{n} (\hat{θ}) r_{0} (\hat{θ})}{\tilde{V} [l_{0} (\hat{θ})]^{1 / 2}},

where

\tilde{V} [l_{0} (θ)] = \sum_{i = 1}^{n} {\tilde{w}}_{i} (θ)^{2} P_{i} (θ) Q_{i} (θ) .

Thus, l_z* is derived from l_z by adding

c_{n} (\hat{θ}) r_{0} (\hat{θ})

to the numerator and replacing the weights

w_{i} (\hat{θ})

by the modified weights

{\tilde{w}}_{i} (\hat{θ})

in the denominator. The modified variance

\tilde{V} [l_{0} (θ)]

, given in Equation 21, is obtained from this weight replacement, and it is straightforward to notice the similarity with the variance V(l ₀) in Equations 11 and 13, since only the weights differ. Furthermore, according to Snijders’s developments, the modified variance

\tilde{V} [l_{0} (θ)]

is smaller than the usual variance V(l ₀), so the variance of l_z* is actually larger than that of l_z .

In sum, the correction introduced by Snijders modifies both the expectation and the variance of the statistic W_n (θ) to take the sampling variability of $\hat{θ}$ into account. The l_z* index can thus be seen as a rescaled version of l_z by adjusting both its mean and its variance. Because the distribution of l_z* is supposed to fit closer the standard normal distribution than that of l_z , the proportion of response patterns which indicate misfit is much closer to the nominal significance level α with l_z* than with l_z , as shown by Snijders (2001) and de la Torre and Deng (2008).

4. Model Selection and Ability Estimation

The previous section outlined the different steps involved in the calculation of the l_z* index with any suitable ability estimator $\hat{θ}$ . By “suitable,” we meant that for an appropriate choice of functions r ₀(θ) and r_i (θ) (i = 1, . . . , n), the condition in Equation 14 was fulfilled with such an estimator. In this section, we establish that the functions r_i (θ) are actually independent of the likelihood-based method of ability estimation, only the function r ₀(θ) is set up by the method itself. The three aforementioned methods (maximum likelihood, Bayes model, and weighted likelihood) are discussed separately. On the other hand, we highlight that the functions r_i (θ), which are intrinsically defined by the IRT model, can take very simple forms with some logistic models.

4.1. Estimation of Ability

First, we consider the ML estimator ${\hat{θ}}_{ML}$ . By definition, it is obtained by maximizing the log likelihood function (Equation 3), which is equivalent to equate its first derivative to zero:

{\frac{d l (θ)}{d θ}|}_{θ = {\hat{θ}}_{ML}} = 0.

Combining Equations (3) and (22) implies that

{\hat{θ}}_{ML}

must fulfill the relationship:

\sum_{i = 1}^{n} [X_{i} - P_{i} (θ)] \frac{P_{i}^{'} (θ)}{P_{i} (θ) Q_{i} (θ)} = 0.

In fact, it is equivalent to consider a constant prior distribution. Hence, the ML estimator is suitable for computing the l_z* index, since it satisfies the condition in Equation (14) with

r_{0} (θ) = 0 and r_{i} (θ) = \frac{P_{i}^{'} (θ)}{P_{i} (θ) Q_{i} (θ)} (i = 1, \dots, n) .

Let us focus now on the maximum a posteriori (or Bayes model) estimator

{\hat{θ}}_{MAP}

. In this framework, the function to maximize is the posterior distribution, given as the product of the likelihood 2 times some prior distribution for the abilities, say f(θ). Thus,

{\hat{θ}}_{MAP}

is the ability value that maximizes f (θ) L(θ), or equivalently its logarithm g(θ) = log f(θ) + l(θ). Hence,

{\hat{θ}}_{MAP}

must verify

{\frac{d g (θ)}{d θ}|}_{θ = {\hat{θ}}_{MAP}} = {(\frac{d log f (θ)}{d θ} + \frac{d l (θ)}{d θ})|}_{θ = {\hat{θ}}_{MAP}} = 0,

or using Equation 23,

\frac{d log f (θ)}{d θ} + \sum_{i = 1}^{n} [X_{i} - P_{i} (θ)] \frac{P_{i}^{'} (θ)}{P_{i} (θ) Q_{i} (θ)} = 0.

Hence, the maximum a posteriori estimator is also an acceptable method for computing the l_z* index, since it satisfies the condition in Equation 14 with

r_{0} (θ) = \frac{d log f (θ)}{d θ} and r_{i} (θ) = \frac{P_{i}^{'} (θ)}{P_{i} (θ) Q_{i} (θ)} (i = 1, \dots, n) .

In particular, one often chooses the standard normal distribution as a prior density f(θ). This comes from the more general normal distribution N(μ,σ²) with mean μ and variance σ², defined as

f (θ) = \frac{1}{\sqrt{2 π σ^{2}}} exp (- \frac{1}{2} \frac{(θ - μ)^{2}}{σ^{2}}) .

With this normal distribution, it comes directly that

r_{0} (θ) = \frac{μ - θ}{σ^{2}},

which eventually reduces to

r_{0} (θ) = - θ

in the case of the standard normal N(0,1) distribution.

The last estimator to be considered is the weighted likelihood estimator ${\hat{θ}}_{WLE}$ (Warm, 1989). The basic idea beyond this method is to maximize a function of the form h(θ) L(θ), but h(θ) does not represent a prior distribution anymore. Instead, the aim of weighted likelihood estimation is to set h(θ) such that it almost cancels the estimation bias of the method. To achieve this goal, Warm established that h(θ) must then be chosen such as

\frac{d log h (θ)}{d θ} = \frac{J (θ)}{2 I (θ)},

where

I (θ) = \sum_{i = 1}^{n} \frac{[P_{i}^{'} (θ)]^{2}}{P_{i} (θ) Q_{i} (θ)} and J (θ) = \sum_{i = 1}^{n} \frac{P_{i}^{'} (θ) P_{i}^{''} (θ)}{P_{i} (θ) Q_{i} (θ)},

and

{P_{i}}^{''} (θ)

stands for the second derivative of

{P_{i}}^{''} (θ)

with respect to θ. In sum, following the same argument as for Bayes model estimation,

{\hat{θ}}_{WLE}

must satisfy the relationship

\frac{J (θ)}{2 I (θ)} + \sum_{i = 1}^{n} [X_{i} - P_{i} (θ)] \frac{P_{i}^{'} (θ)}{P_{i} (θ) Q_{i} (θ)} = 0,

which corresponds to Equation 14 with

r_{0} (θ) = \frac{J (θ)}{2 I (θ)} and r_{i} (θ) = \frac{P_{i}^{'} (θ)}{P_{i} (θ) Q_{i} (θ)} (i = 1, \dots, n) .

Interestingly, with the definition (Equation 33) of the r_i (θ) functions, one can rewrite I(θ) and J(θ) as

I (θ) = \sum_{i = 1}^{n} r_{i} (θ) P_{i}^{'} (θ) and J (θ) = \sum_{i = 1}^{n} r_{i} (θ) P_{i}^{''} (θ),

hence

r_{0} (θ) = \frac{\sum_{i = 1}^{n} r_{i} (θ) P_{i}^{''} (θ)}{2 \sum_{i = 1}^{n} r_{i} (θ) P_{i}^{'} (θ)} .

Note eventually that Hoijtink and Boomsma (1995, p. 57) have established that in the case of the 1PL model, the function h(θ) used in Equation 30 is nothing else than the so-called Jeffreys' prior density (Jeffreys, 1939, 1946).

In conclusion, these three likelihood-based estimators can be considered as suitable methods for computing the l_z* index and ensuring its asymptotic standard normal distribution. They differ by setting the function r ₀(θ) but share the same forms of functions r_i (θ), the latter being set up by the logistic model under consideration. The maximum a posteriori method can be considered with any type of prior distribution, although the usual normal density yields a simple form for r ₀(θ).

It is important to notice that these estimators are maybe not the only one to be accurate in this framework. As already mentioned, any estimator that satisfies the condition in Equation 14 is a potential estimator for using with l_z*. For instance, the expected a posteriori (EAP) estimator ${\hat{θ}}_{EAP}$ is also often used for ability estimation, especially in the framework of computerized adaptive testing (Bock & Mislevy, 1982). Basically, this method aims at computing the expected value of the posterior distribution g(θ), instead of finding its maximum to determine ${\hat{θ}}_{MAP}$ . However, computing ${\hat{θ}}_{EAP}$ requires the numerical estimation of integrals, which complicates the task of assessing that the condition in Equation 14 is fulfilled with ${\hat{θ}}_{EAP}$ .

4.2. Model Selection

Since the functions r_i (θ) in Equations 22, 27, and 33 have the same form for the three estimators studied previously, it is of primary interest to look at their particular form with some specific IRT models. Only logistic models are considered in this presentation.

We start from the 3PL model, given by Equation 1. To simplify the notations, set e_i (θ) = exp [a_i (θ − b_i )]. One has then

P_{i} (θ) = c_{i} + \frac{(1 - c_{i}) e_{i} (θ)}{1 + e_{i} (θ)}, Q_{i} (θ) = \frac{1 - c_{i}}{1 + e_{i} (θ)}, P_{i}^{'} (θ) = \frac{(1 - c_{i}) a_{i} e_{i} (θ)}{[1 + e_{i} (θ)]^{2}},

36a

and

P_{i}^{''} (θ) = \frac{(1 - c_{i}) a_{i}^{2} e_{i} (θ) [1 - e_{i} (θ)]}{[1 + e_{i} (θ)]^{3}},

36b

so that

r_{i} (θ) = \frac{P_{i}^{'} (θ)}{P_{i} (θ) Q_{i} (θ)} = \frac{a_{i} e_{i} (θ)}{c_{i} + e_{i} (θ)} = \frac{a_{i} exp [a_{i} (θ - b_{i})]}{c_{i} + exp [a_{i} (θ - b_{i})]} .

Thus, the functions r_i (θ) can be calculated very easily with the 3PL, without even requiring the first derivative of P_i (θ). Let us turn now to the 2PL model, which is obtained from Equation 1 by setting c_i to zero. We get directly from Equation 37 that

r_{i} (θ) = a_{i} .

And with the 1PL model, the simplification is even stronger since the item discriminations are all equal to one, so that r_i (θ) = 1 with the 1PL model.

If we consider the 2PL model, some calculation steps of l_z* can also be simplified. For instance, since it comes from Equations 37 and 38 that ${P_{i}}^{'} (θ) = a_{i} P_{i} (θ) Q_{i} (θ)$ , one can plug this simplification into c_n (θ) and the first derivative ${P_{i}}^{'} (θ)$ is not required anymore:

c_{n} (θ) = \frac{\sum_{i = 1}^{n} a_{i} P_{i} (θ) Q_{i} (θ) w_{i} (θ)}{\sum_{i = 1}^{n} a_{i}^{2} P_{i} (θ) Q_{i} (θ)} .

Moreover, one gets directly that

{P_{i}}^{''} (θ) = a_{i}^{2} P_{i} (θ) Q_{i} (θ) [P_{i} (θ) - Q_{i} (θ)]

, which means that for the weighted likelihood estimator, Equation 35 simplifies in this case to

r_{0} (θ) = \frac{\sum_{i = 1}^{n} a_{i}^{3} P_{i} (θ) Q_{i} (θ) [P_{i} (θ) - Q_{i} (θ)]}{2 \sum_{i = 1}^{n} a_{i}^{2} P_{i} (θ) Q_{i} (θ)}

and can be computed without requiring the first and second derivatives of P_i (θ). If the 1PL is considered instead, Equations 38 and 39 are simpler since a_i = 1. Finally, equivalent forms can be found with the 3PL, but they are less mathematically tractable and are therefore skipped from the discussion. However, it is important to recall that c_n (θ) and r ₀(θ) (for the weighted likelihood estimator) can be computed without requiring the first and second derivatives of P_i (θ).

In sum, the selection of a logistic IRT model has some impact on the functions r_i (θ), which are required to compute the person fit index l_z*. Moreover, except for weighted likelihood estimation, these models do not influence the function r ₀(θ), which is entirely set up by the method of ability estimation.

5. *l_z** in Practice

It is time now to summarize the developments of The l_z* Index and Model Selection and Ability Estimation sections and to propose a schematic overview of the computational process of l_z*. This process is sketched in Figure 1 .

Figure 1.

Schematic process to compute l_z*.

First, the selection of an IRT model determines the form of the r_i (θ) functions. This initial step is actually not to be realized for computing l_z*, because the model is selected and the item parameters are estimated prior to person fit investigation. The second step consists in selecting a suitable estimator of ability, which yields the estimate $\hat{θ}$ and determines the form of the r₀ (θ) function.

Once these steps are performed, and the functions r₀ (θ) and r_i (θ) are set, the next steps are straightforward and hierarchically ordered. Using the weight functions w_i (θ) in Equation 17, one successively computes $c_{n} (\hat{θ})$ , ${\tilde{w}}_{i} (\hat{θ})$ , and $τ_{n}^{2} (\hat{θ})$ to end up with l_z*. Eventually, the comparison between the value of l_z* and the quantile z_α of the standard normal distribution (with lower tail α) gives the final classification of the response pattern as indicating misfit or not.

The latter point merits some additional discussion. First, Equation 20 indicates that l_z* is simply a shifted and rescaled version of l_z , when the latter is computed with sample estimates of θ. Thus, extremely negative values of l_z* also correspond to extremely negative values of l_z and the same rule applies for extreme positive values. This is the reason why the same rule for detecting person fit applies for both l_z and l_z*. In addition, small negative values of l_z refer to aberrant patterns for which misfit obviously occurs, while large positive values of l_z indicate an almost perfect fit of the response pattern to the test characteristics.

To highlight this trend, consider an examinee with true ability level θ. The variance of l ₀ (Equation 7) is then a positive constant value, and the sign of l_z is determined by the numerator of Equation 5, which can be rewritten as

l_{0} - E (l_{0}) = \sum_{i = 1}^{n} [X_{i} - P_{i} (θ)] \{log P_{i} (θ) - log Q_{i} (θ)\},

using Equations 3 and 6. Now, let S ₁ be the subset of items such that P_i (θ) ≥ 1/2 and S ₂ the subset of items such that P_i (θ) < 1/2. By this way, the success probability P_i (θ) is larger than Q_i (θ) in S ₁ and strictly smaller than Q_i (θ) in S ₂. Equation 41 can now be rewritten as

l_{0} - E (l_{0}) = \sum_{i \in S_{1}} [X_{i} - P_{i} (θ)] {\log P_{i} (θ) - \log Q_{i} (θ)} - \sum_{i \in S_{2}} [X_{i} - P_{i} (θ)] {\log Q_{i} (θ) - \log P_{i} (θ)} .

Thus, the largest positive value of

l_{0} - E (l_{0})

will be observed when

X_{i} = \{\begin{matrix} 1 & i f i \in S_{1} \\ 0 & i f i \in S_{2} \end{matrix},

while the smallest negative value of

l_{0} - E (l_{0})

will be reached when

X_{i} = \{\begin{matrix} 0 & i f i \in S_{1} \\ 1 & i f i \in S_{2} \end{matrix} .

The response pattern described in Equation 43 is actually the most perfect one in terms of test fitting, since the examinee succeeds all easy items for which the success probability is larger than one half, and fails the other, “difficult” ones. Similarly, the response pattern described in Equation 44 is the most abnormal one, because the examinee fails all “easy” items and succeeds all “difficult” items. In conclusion, the detection of person fit is mainly concerned with the very small negative values of l_z index. This rule remains valid for l_z*, as it was highlighted earlier.

6. An Example

We shall illustrate the computation of l_z* by making use of a real test about English-language aptitude. The TCALS-II (Test de Classement en Anglais, Langue Seconde - Version II [Placement Test of English as a Second Language]) questionnaire aims at evaluating the level of English knowledge from French-speaking students entering into college graduation, in order to assign them to appropriate groups of difficulty level (Laurier, Froio, Paero, & Fournier, 1998). The test consists of 85 multiple-choice items, divided into eight subgroups of questions, and is identical for all French-speaking colleges of the province of Quebec (Canada). The data under consideration are the responses of 1,373 students (749 females and 624 males) entering into the College of Outaouais (Gatineau, Quebec, Canada) in the academic year 1998.

In the present analysis, we will start by giving the detailed calculation of both l_z and l_z* indices for one randomly selected examinee among the 1,373 students of the study. Then, the full distribution of l_z and l_z* will be displayed to provide a global overview of the TCALS-II data set.

First, the items were calibrated by marginal ML, using the 2PL model. The R package ltm (Rizopoulos, 2006) was considered for such an item calibration. The estimated discrimination and difficulty parameters are reported in Table 1 . It can be noticed that the TCALS-II is an easy test: the average difficulty level is −1.34 (SE = 0.83), with minimum and maximum difficulties of −4.49 and 0.66, respectively. Moreover, the average discrimination level is 1.63 (SE = 0.62), with minimum discrimination of 0.14 and maximum discrimination of 4.07. It is therefore expected that many examinees will respond correctly to many test items, as noticed by Raîche (2002).

Table 1.

Item Discrimination and Difficulty Estimates of the TCALS-II Data Set and One Randomly Selected Response Pattern

Item	a_i	b_i	X_i	Item	a_i	b_i	X_i	Item	a_i	b_i	X_i	Item	a_i	b_i	X_i
1	2.23	−2.01	1	23	2.90	−0.71	1	45	2.16	−0.75	1	67	2.03	−1.10	1
2	1.02	−2.90	1	24	1.40	−0.31	1	46	1.29	−1.04	1	68	2.33	−0.81	1
3	2.59	−2.09	1	25	1.38	0.59	1	47	1.21	−2.13	1	69	1.49	−0.60	1
4	2.37	−1.67	1	26	0.92	−1.21	1	48	1.47	−1.60	0	70	1.31	−0.42	1
5	1.09	−1.93	1	27	0.58	−0.11	1	49	2.49	−1.67	1	71	1.65	−1.60	1
6	1.23	−1.87	1	28	1.34	−1.39	1	50	2.30	−1.56	1	72	0.95	−1.60	1
7	1.80	−1.48	1	29	1.39	−1.50	0	51	1.74	−1.35	1	73	0.90	−1.26	1
8	2.58	−0.75	1	30	1.64	−0.80	1	52	0.88	−1.95	1	74	1.12	−0.75	0
9	2.15	−0.65	1	31	1.40	−0.47	1	53	2.69	−1.32	1	75	1.00	−0.90	1
10	2.11	−0.40	1	32	0.14	−4.49	1	54	2.28	−1.40	1	76	1.03	−0.77	1
11	1.39	−0.16	1	33	1.25	−1.50	1	55	1.65	−1.38	1	77	0.86	0.62	1
12	1.65	−0.29	1	34	1.04	−3.57	1	56	1.31	−1.80	1	78	0.87	−2.05	1
13	1.83	−1.66	1	35	1.75	−2.01	1	57	1.00	−1.60	1	79	1.01	−1.76	1
14	1.96	−2.52	1	36	2.37	−1.97	1	58	1.54	−1.66	1	80	1.92	0.66	1
15	1.99	−1.51	1	37	1.74	−2.12	0	59	1.96	−0.72	1	81	1.62	−0.68	1
16	1.13	−2.16	1	38	1.99	−1.66	1	60	1.91	−0.68	1	82	1.22	−1.30	1
17	1.52	−1.67	1	39	0.79	−2.88	1	61	1.18	−0.61	1	83	1.58	−1.55	1
18	1.90	−1.44	1	40	2.27	−1.61	1	62	1.99	−0.26	1	84	1.44	−1.04	1
19	4.07	−1.17	1	41	1.23	−1.69	1	63	2.82	0.05	1	85	1.33	−1.53	1
20	1.38	−1.57	1	42	2.14	−1.99	1	64	1.39	−2.23	1
21	0.74	−1.25	1	43	1.76	−1.23	1	65	1.94	−1.77	1
22	1.86	−1.00	1	44	2.42	−1.18	1	66	1.59	−1.70	1

Table 1 also holds the complete response pattern of the randomly selected examinee who was assigned the TCALS-II questionnaire. This particular pattern will be considered in the next calculations.

Since the 2PL model is considered, the r_i (θ) functions reduce to the discriminations a_i displayed in Table 1. Moreover, the weighted likelihood estimator was used to get an estimation of the examinee’s proficiency levels. All forthcoming calculations were performed using an implementation of the person fit indices into the R (R Development Core Team, 2009) software. The main commands of the R code are displayed in the Appendix.

By solving Equation 14, using Equations 38 and 40 due to the 2PL model, it turned out that $\hat{θ} = 0.98$ was obtained as weighted likelihood estimation of ability for this examinee. With this value, the success probabilities $P_{i} (\hat{θ})$ and the weights $w_{i} (\hat{θ})$ were evaluated. Then, the value of $c_{n} (\hat{θ})$ was calculated using its particular form (Equation 39), and we obtained $c_{n} (\hat{θ}) = 1.55$ . Together with the value $r_{0} (\hat{θ})$ , calculated with Equation 40 and equal to $r_{0} (\hat{θ}) = - 0.62$ , the modified weights ${\tilde{w}}_{i} (\hat{θ})$ were computed. Table 2 lists the values $P_{i} (\hat{θ})$ , $w_{i} (\hat{θ})$ , and ${\tilde{w}}_{i} (\hat{θ})$ for each item.

Table 2.

Probability Estimates Pi(θˆ), Weights wi(θˆ) and Modified Weights wi˜(θˆ) of the TCALS-II Data Set

Item	$P_{i} (\hat{θ})$	$w_{i} (\hat{θ})$	${\tilde{w}}_{i} (θ)$	Item	$P_{i} (\hat{θ})$	$w_{i} (\hat{θ})$	${\tilde{w}}_{i} (θ)$	Item	$P_{i} (\hat{θ})$	$w_{i} (\hat{θ})$	${\tilde{w}}_{i} (\hat{θ})$	Item	$P_{i} (\hat{θ})$	$w_{i} (\hat{θ})$	${\tilde{w}}_{i} (\hat{θ})$
1	1.00	6.68	3.22	23	0.99	4.92	0.42	45	0.98	3.75	0.40	67	0.99	4.24	1.09
2	0.98	3.95	2.37	24	0.86	1.82	−0.36	46	0.93	2.61	0.61	68	0.98	4.18	0.57
3	1.00	7.96	3.95	25	0.63	0.55	−1.59	47	0.98	3.76	1.89	69	0.91	2.35	0.05
4	1.00	6.29	2.61	26	0.88	2.02	0.59	48	0.98	3.79	1.51	70	0.86	1.83	−0.20
5	0.96	3.19	1.50	27	0.65	0.64	−0.26	49	1.00	6.63	2.76	71	0.99	4.25	1.70
6	0.97	3.50	1.60	28	0.96	3.17	1.10	50	1.00	5.86	2.29	72	0.92	2.45	0.98
7	0.99	4.43	1.64	29	0.97	3.45	1.29	51	0.98	4.05	1.36	73	0.88	2.01	0.62
8	0.99	4.48	0.47	30	0.95	2.92	0.37	52	0.93	2.59	1.22	74	0.88	1.95	0.20
9	0.97	3.52	0.19	31	0.88	2.03	−0.14	53	1.00	6.20	2.02	75	0.87	1.88	0.33
10	0.95	2.91	−0.36	32	0.68	0.77	0.55	54	1.00	5.44	1.89	76	0.86	1.82	0.21
11	0.83	1.59	−0.57	33	0.96	3.11	1.17	55	0.98	3.90	1.34	77	0.58	0.31	−1.02
12	0.89	2.09	−0.46	34	0.99	4.72	3.11	56	0.97	3.63	1.60	78	0.93	2.64	1.29
13	0.99	4.84	2.00	35	0.99	5.24	2.53	57	0.93	2.60	1.04	79	0.94	2.76	1.20
14	1.00	6.86	3.82	36	1.00	7.01	3.33	58	0.98	4.05	1.67	80	0.65	0.63	−2.34
15	0.99	4.96	1.87	37	1.00	5.39	2.70	59	0.97	3.33	0.29	81	0.94	2.70	0.18
16	0.97	3.54	1.79	38	0.99	5.26	2.17	60	0.96	3.18	0.21	82	0.94	2.78	0.89
17	0.98	4.04	1.68	39	0.95	3.05	1.82	61	0.87	1.88	0.05	83	0.98	4.00	1.55
18	0.99	4.60	1.65	40	1.00	5.90	2.37	62	0.92	2.46	−0.62	84	0.95	2.92	0.68
19	1.00	8.78	2.46	41	0.96	3.31	1.39	63	0.93	2.62	−1.75	85	0.97	3.35	1.28
20	0.97	3.54	1.39	42	1.00	6.35	3.03	64	0.99	4.47	2.31
21	0.84	1.64	0.50	43	0.98	3.90	1.17	65	1.00	5.34	2.33
22	0.98	3.68	0.80	44	0.99	5.24	1.48	66	0.99	4.26	1.79

Eventually, successive calculations with the previous findings led to the following values: $W_{n} (\hat{θ}) = - 4.85$ , $σ_{n}^{2} (\hat{θ}) = 0.28$ , and $τ_{n}^{2} (\hat{θ}) = 0.06$ . The values of l_z and l_z* are finally obtained with Equations 8 and 19: one gets l_z = −0.99 and l_z* = −2.57.

Thus, after this detailed calculation of both person fit indices, one may conclude that the response pattern of the selected examinee is not that abnormal with respect to the l_z index, since the quantile z _α with lower tail α = 5% is equal to −1.64. On the other hand, when correcting for the ability estimate, the l_z* index is equal to −2.57, which is largely under the acceptable threshold. In conclusion, the examinee has an aberrant response pattern with respect to the l_z* criterion.

For completeness, the indices l_z and l_z* were computed for all examinees of the TCALS-II data set. Figure 2 displays the density estimates of the distributions of l_z (solid line) and l_z* (dashed line) together with the standard normal density (dotted line). The density estimates are obtained by kernel density estimation (Silverman, 1986), using the Gaussian kernel and an optimal bandwidth selection (Sheather & Jones, 1991). One can observe that the variance of l_z* (Equation 21) is much larger than that of l_z (Equation 7), as expected. None of the distributions seem to be quite close to the standard normal density, which means for l_z* that there is much more misfit than expected under the normal distribution. This fact has indeed been validated by Raîche and Blais (2002). On the other hand, according to Drasgow et al. (1985), the test is sufficiently long (85 items) to consider the asymptotic framework as valid. Therefore, one possible explanation is that the 2PL model is not suitable for this real data set.

Figure 2.

Density estimates of l_z and l_z * (2PL model, weighted likelihood estimator) and standard normal density, TCALS-II data set.

7. Conclusion

In this article, we tried to provide an overview of Snijders’s approach to correct the l_z index of person fit. This correction is necessary when ability estimates are used instead of the true proficiency levels. Although being a promising approach, this corrected index l_z * is not yet popular in the field of appropriateness measurement, possibly because of the technical features in Snijders’s 2001 article. Our presentation will hopefully provide some clearer insight on that topic and be helpful for interested researchers in the future.

The analysis of the TCALS-II data set not only illustrated the differences between the distributions of l_z and l_z* but also highlighted that l_z* may have a somewhat different distribution than the expected standard normal density. It would be useful then to perform a simulation study to control for examinees which follow the item response model perfectly. In addition, the distributions of l_z and l_z * could be compared for various choices of item response models and ability estimators, and this would constitute an extension of Snijders’s (2001) simulated results.

One of the most striking advances of Snijders’s article is that other person fit indices could be adjusted similarly to l_z in order to get a better asymptotic framework for identifying misfit. Some promising indices are the Zeta index (Tatsuoka, 1984, 1996), U (Wright & Stone, 1979), W (Wright & Masters, 1982), and the class of Extended Caution Indices (ECI Tatsuoka & Linn, 1983). With an appropriate selection of the weight functions, these indices could be adjusted to take the replacement of θ by $\hat{θ}$ into account. The correction process described in Figure 1 remains unchanged, except for the definition of the weights. This was pointed out by Snijders’s (2001) but apparently not yet investigated further. Once these corrected indices are available, it will be of most interest to study their efficiency in providing better person fit detection, in particular with respect to their original, uncorrected version. This is the main topic of an ongoing research project. In addition, an R package is under development to provide a software solution for computing l_z*. This package will merge most of the standard (parametric and nonparametric) indices of person fit and also provide the option of computing the corrected l_z* index instead of l_z . The appended R code will obviously be integrated to the package. It is hoped that other corrected indices will be added to the package in the future.

Recently, some person fit indexes and tests for misfit have been introduced for polytomous items (Emons, 2009; Glas & Dagohoy, 2007). In this framework, the l_z index can be naturally extended as the standardized log likelihood function from a polytomous item response model (Ro, 2001), for example, the graded response model (Samejima, 1969) or the partial credit model (Masters, 1982). Until now, however, Snijders’s correction has not been developed for polytomous items. Such a correction would also probably improve the identification of person fit, as it is the case for dichotomous items.

Finally, very little has been done to adapt parametric person fit indexes in the Bayesian paradigm of model and parameter estimation. Most of the developments related to misfit measurement are made under the usual likelihood framework. However, the increase in Bayesian methods and software for solving complex issues should not be discarded. Instead of computing person fit indexes such as l_z or l_z* and comparing them to theoretical cutoff scores, one could imagine to compute posterior fit (or misfit) probabilities based on posterior estimates of the item and subject parameters. This field of application seems promising and will hopefully receive more attention in the years to come.

Footnotes

Acknowledgments

The authors wish to thank the editor and the referees for their helpful comments and suggestions. This work was supported by a research grant “Chargé de recherches” from the National Funds for Scientific Research (FNRS), Belgium, the grant 122750 from the Social Sciences and Humanities Research Council of Canada (SSHRC), and the grant 103426 from the Fonds Québécois de Recherche sur la Société et la Culture (FQRSC).

Appendix: R Code for Computing lz and lz*

## Response probabilities, first and second derivatives under the 3PL model (equation 36)

## th: ability value

## it: matrix of item parameters: one row per item, three columns

## (discrimination, difficulty, pseudo-guessing)

Pi<-function(th,it){

res1<-res2<-res3<-NULL

for (i in 1:nrow(it)) {

res1[i]<-it[i,3]+(1-it[i,3])*exp(it[i,1]*(th-it[i,2]))/(1+exp(it[i,1]*(th-it[i,2])))

res2[i]<-(1-it[i,3])*it[i,1]*exp(it[i,1]*(th-it[i,2]))/(1+exp(it[i,1]*(th-it[i,2])))ˆ2

res3[i]<-(1-it[i,3])*it[i,1]ˆ2*exp(it[i,1]*(th-it[i,2]))*(1-exp(it[i,1]*(th-it[i,2])))/(1+exp(it[i,1]*(th-it[i,2])))ˆ3

}

RES<-list(Pi=res1,dPi=res2,d2Pi=res3)

return(RES)}

## Functions ri and r0 (equations 24, 27 and 33)

## method: « ML » for maximum likelihood, « BM » for Bayesian modal (or MAp), « WL » for weighted likelihood

## mu, sigma: prior mean and standard deviation parameters of the normal distribution

ri<-function(th,it) Pi(th,it)$dPi/(Pi(th,it)$Pi*(1-Pi(th,it)$Pi))

r0<-function(method=”ML”,th,it,mu=0,sigma=1){

res<-switch(method,

ML=0,

BM=(mu-th)/sigmaˆ2,

WL=sum(ri(th,it)*Pi(th,it)$d2Pi)/(2*sum(ri(th,it)*Pi(th,it)$dPi)))

return(res)}

## Ability estimation (constrained to range [-4; 4]) (equation 14)

thetaEst<-function(x,it,method=”ML”,mu=0,sigma=1){

f<-function(th) r0(method=method,th,it=it,mu=mu,sigma=sigma)+sum((x-Pi(th,it)$Pi)*ri(th,it))

if (f(-4)<0 & f(4)<0) res<--4

else{

if (f(-4)>0 & f(4)>0) res<-4

else res<-uniroot(f,c(-4,4))$root

}

return(res)}

## Weight function wi (equation 12)

wi<-function(th,it) log(Pi(th,it)$Pi/(1-Pi(th,it)$Pi))

## Function Wn(theta) (equation 9)

## x: response pattern (same length as nrow(it), zeros and ones as entries)

Wn<-function(x,th,it) sum((x-Pi(th,it)$Pi)*wi(th,it))

## Function sig2n (equation 11)

sig2n<-function(th,it)

sum(wi(th,it)ˆ2*Pi(th,it)$Pi*(1-Pi(th,it)$Pi))/nrow(it)

## Function cn (equation 15)

cn<-function(th,it) sum(Pi(th,it)$dPi*wi(th,it))/sum(Pi(th,it)$dPi*ri(th,it))

## Function wi ‘tilde’ (equation 16)

wiTilde<-function(th,it) wi(th,it)-cn(th,it)*ri(th,it)

## Function tau2n (equation 18)

tau2n<-function(th,it)

sum(wiTilde(th,it)ˆ2*Pi(th,it)$Pi*(1-Pi(th,it)$Pi))/nrow(it)

## Indexes lz and lz*

## snijders: logical argument: FALSE returns lz, TRUE returns lz*

Lz<-function(x,it,method=”ML”,mu=0,sigma=1,snijders=FALSE){

th<-thetaEst(x=x,it=it,method=method,mu=mu,sigma=sigma)

if (snijders==TRUE){

EWn<--cn(th,it)*r0(method=method,th=th,it=it,mu=mu,sigma=sigma)

VWn<-nrow(it)*tau2n(th,it)

}

else{

EWn<-0

VWn<-nrow(it)*sig2n(th,it)

}

res<-(Wn(x,th,it)-EWn)/sqrt(VWn)

return(res)}

References

Armstrong

R. D.

Stoumbos

Z. G.

Kung

M. T.

Shi

(2007). On the performance of the Lz person-fit statistics. Practical Assessment, Research and Evaluation, 12, 16.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 395–479). Reading, MA: Addison-Wesley.

Bock

R. D.

Mislevy

R. J.

(1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431–444.

Cronbach

L. J.

(1946). Response set and test validity. Educational and Psychological Measurement, 6, 475–494.

de la Torre

Deng

(2008). Improving person fit assessment by correcting the ability estimate and its reference distribution. Journal of Educational Measurement, 45, 159–177.

Dodeen

(2003, April). The use of person-fit statistics to analyse placement tests. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.

Drasgow

Levine

M. V.

McLaughlin

M. E.

(1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59–79.

Drasgow

Levine

M. V.

Williams

E. A.

(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67–86.

Emons

(2009). Detection and diagnosis of person misfit from patterns of summed polytomous item scores. Applied Psychological Measurement, 33, 599–619.

10.

Ferrando

P. J.

(2004). Person reliability in personality measurement: An item response theory analysis. Applied Psychological Measurement, 28, 126–140.

11.

Glas

C. A. W.

Dagohoy

A. V. T.

(2007). Person fit test for IRT models for polytomous items. Psychometrika, 72, 159–180.

12.

Hoijtink

Boomsma

(1995). On person parameter estimation in the dichotomous Rasch model. In Fischer

G. H.

Molenaar

I. W.

(Eds.), Rasch models. Foundations, recent developments, and applications (pp. 53–68). New York, NY: Springer Verlag.

13.

Jeffreys

(1939). Theory of probability. Oxford, UK: Oxford University Press.

14.

Jeffreys

(1946). An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London, Series A, Mathematical and Physical Sciences, 186, 453–461.

15.

Karabatsos

(2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16, 277–298.

16.

Klauer

K. C.

(1995). The assessment of person fit. In Fischer

G. H.

Molenaar

I. W.

(Eds.), Rasch models: Foundations, recent developments, and applications (pp. 97–110). New York, NY: Springer.

17.

Laurier

M. D.

Froio

Paero

Fournier

(1998). L’élaboration d’un test provincial pour le classement des étudiants en anglais langue seconde au collégial [The elaboration of a provincial English, as a second language, test to classify college students]. Québec, Canada: Direction générale de l’enseignement collégial, ministère de l’Éducation du Québec.

18.

Levine

M. V.

Drasgow

(1980). Appropriateness measurement: Validating studies and variable ability models. In Weiss

D. J.

(Ed.), New horizons in testing (pp. 110–131). New York, NY: Academic Press.

19.

Levine

M. V.

Drasgow

(1982). Appropriateness measurement: Review, critique, and validating studies. British Journal of Mathematical and Statistical Psychology, 35, 42–56.

20.

Levine

M. V.

Drasgow

(1988). Optimal appropriateness measurement. Psychometrika, 53, 161–176.

21.

Levine

M. V.

Rubin

D. B.

(1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4, 269–290.

22.

M. F.

Olejnik

(1997). The power of Rasch person-fit statistics in detecting unusual response patterns. Applied Psychological Measurement, 21, 215–231.

23.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.

24.

Meijer

(1998). Consistency of test behaviour and individual difference in precision of prediction. Journal of Occupational and Organizational Psychology, 71, 147–160.

25.

Meijer

Nering

M. L.

(1997). Trait level estimation for nonfitting response vectors. Applied Psychological Measurement, 21, 321–336.

26.

Meijer

Sijtsma

(2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107–135.

27.

Molenaar

I. W.

Hoijtink

(1990). The many null distributions of person fit indices. Psychometrika, 55, 75–106.

28.

Molenaar

I. W.

Hoijtink

(1996). Person-fit and the Rasch model, with an application to knowledge of logical quantors. Applied Measurement in Education, 9, 27–45.

29.

Nering

M. L.

(1995). The distribution of person fit using true and estimated person parameters. Applied Psychological Measurement, 19, 121–129.

30.

Nering

M. L.

(1997). The distribution of indexes of person-fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21, 115–127.

31.

Nering

M. L.

Meijer

R. R.

(1998). A comparison of the person response function and the lz person-fit statistic. Applied Psychological Measurement, 22, 53–69.

32.

Raîche

(2002). Le dépistage de sous-classement aux tests de classement en anglais, langue seconde, au collégial [The detection of under-performance to college aptitude tests of English, as a second language]. Gatineau, Quebec, Canada: Collège de l’Outaouais.

33.

Raîche

Blais

J.-G.

(2003a). Efficacité du dépistage des étudiantes et des étudiants qui cherchent à obtenir un résultat faible au test de classement en anglais, langue seconde, au collégial [Accuracy of the detection of college under-performing students at the English, as a second language, aptitude test]. In Blais

J.-G.

Raîche

(Eds.), Regards dur la modélisation de la mesure en éducation et en sciences sociales [Horizons in measurement modelling in education and social sciences]. Ste-Foy, Québec, Canada: Presses de l’université Laval.

34.

Raîche

Blais

J.-G.

(2003b, July). The distribution of person-fit indices conditional on the estimated proficiency level and the detection of underachievement at a placement test. Paper presented at the International Meeting of the Psychometric Society, Cagliari, Italy.

35.

Raîche

Blais

J.-G.

(2005, July). Characterization of the distribution of the Lz index of person fit according to the estimated proficiency level. Paper presented at the International Meeting of the Psychometric Society, Tilburg, the Netherlands.

36.

R Development Core Team. (2009). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

37.

Reise

S. R.

(1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19, 213–229.

38.

Rizopoulos

(2006). ltm: An R package for latent variable modeling and item response theory Analyses. Journal of Statistical Software, 17, 1–25.

39.

(2001). Characteristics of a likelihood-based person-fit index under the graded response model. Unpublished doctoral dissertation. Minneapolis: University of Minnesota.

40.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement 17, Richmond, VA: Psychometric Society.

41.

Sheather

S. J.

Jones

M. C.

(1991). A reliable data-based bandwidth selection method for kernel density estimation. Journal of the Royal Statistical Society Series B, 53, 683–690.

42.

Silverman

B. W.

(1986). Density estimation. London, England: Chapman and Hall.

43.

Smith

R. M.

(1982). Detecting measurement disturbances with the rash model. Unpublished doctoral dissertation. Chicago, IL: University of Chicago.

44.

Snijders

T. A. B.

(2001). Asymptotic null distribution of person fit statistics with estimated person parameter. Psychometrika, 66, 331–342.

45.

Tatsuoka

K. K.

(1984). Caution indices based on item response theory. Psychometrika, 49, 95–110.

46.

Tatsuoka

K. K.

(1996). Use of generalized person-fit indexes, zetas for statistical pattern classification. Applied Measurement in Education, 9, 65–76.

47.

Tatsuoka

K. K.

Linn

R. L.

(1983). Indices for detecting unusual patterns: Links between two general approaches and potential applications. Applied Psychological Measurement, 7, 81–96.

48.

van Krimpen-Stoop

Meijer

(1999). The null distribution of person-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23, 327–344.

49.

Warm

T. A.

(1989). Weighted likelihood estimation of ability in item response models. Psychometrika, 54, 427–450.

50.

Wright

B. D.

Masters

G. N.

(1982). Rating scale analysis. Chicago, IL: MESA Press.

51.

Wright

B. D.

Stone

M. H.

(1979). Best test design. Rasch measurement. Chicago, IL: Mesa Press.

52.

Zickar

M. J.

Drasgow

(1996). Detecting faking on a personality instrument using appropriateness measurement. Applied Psychological Measurement, 20, 71–87.

A Didactic Presentation of Snijders’s l z * Index of Person Fit With Emphasis on Response Model Selection and Ability Estimation

Abstract

Keywords

1. Introduction

2. IRT Assessment of Person Fit and the lz Index

3. The lz* Index

4. Model Selection and Ability Estimation

4.1. Estimation of Ability

4.2. Model Selection

5. lz* in Practice

6. An Example

7. Conclusion

Footnotes

Acknowledgments

Appendix: R Code for Computing lz and lz*

References

2. IRT Assessment of Person Fit and the l_z Index

3. The *l_z** Index

5. *l_z** in Practice