On the Generalized S − X 2 –Test of Item Fit: Some Variants,Residuals,and a Graphical Visualization

Abstract

The generalized $S - X^{2}$ –test is a test of item fit for items with polytomous responses format. The test is based on a comparison of the observed and expected number of responses in strata defined by the test score. In this article, we make four contributions. We demonstrate that the performance of the generalized $S - X^{2}$ –test depends on how sparse cells are pooled. We propose alternative implementations of the test within the framework of limited information testing. We derive the distribution of the $S - X^{2}$ –residuals that can be used for post hoc analyses. We suggest a diagnostic plot that visualizes the form of the misfit. The performance of the alternative implementations is investigated in a simulation study. The simulation study suggests that the alternative implementations are capable of controlling the Type-I error rate well and have high power. An empirical application concludes this article.

Keywords

item response theory item fit Fear of Negative Evaluation Scale

Item response theory is a field of psychology that deals with the development of item response models. Item response models specify how a latent trait is related to the responses of a test taker in a series of test items that are supposed to be manifest indicators of the trait. The models are used for deductive statements about the test taker’s trait levels and provide a mathematical basis for trait inference, adaptive testing, and test linking. Such applications, however, are only justified in case the item response model used for assessment is capable of representing the true relation between the responses and the latent trait well. The guidelines of the American Psychological Association (2014) demand that evidence is given for the validity of an item response model before it is applied in psychological assessment.

The fit of an item response model to specific data is often assessed with a test of model fit. Tests of model fit assess whether certain aspects of the distribution of the responses in a sample conform to predictions from the model. The tests are sometimes distinguished between tests of global fit and tests of item fit. The tests of global fit are omnibus tests that assess whether the item response model fits to the observed data in all items. The tests of item fit assess whether the model is capable to represent the distribution of the responses in a single item. Overviews of tests of model fit have been given by Swaminathan et al. (2006), Mavridis et al. (2007), Maydeu-Olivares (2013), Glas (2016), and Ranger and Much (2020). In this article, we focus on one specific test of item fit, the $S - X^{2}$ –test proposed by Orlando and Thissen (2000). The $S - X^{2}$ –test is among the most popular tests of item fit. It adheres to the nominal Type-I error rate well and has high power (Orlando & Thissen, 2003; Sinharay & Ying, 2007). Although originally proposed for the three-parameter logistic model, the $S - X^{2}$ –test has been extended to polytomous (Kang & Chen, 2008; LaHuis et al., 2011) and multidimensional item response models (Zhang & Stone, 2008; Zhang et al., 2018).

In this article, we make four additions to the knowledge in the field. First, we investigate the performance of the $S - X^{2}$ –test in polytomous item response models. We demonstrate that the performance of the test crucially depends on how sparse table cells are pooled. Pooling too many cells or pooling too few cells increases the Type-I error rate of the test beyond the intended level $α$ . Second, we propose two alternative implementations of the test that adhere to the nominal Type-I error rate even when no cells are pooled. Third, we derive the distribution of the residuals the $S - X^{2}$ –test is based on. This can be used for exploratory post hoc tests in case the $S - X^{2}$ –test indicates a misfit. Fourth, we suggest a graphical check of item fit that is similar to the residual plots used in regression analysis. The graphical check allows for an exploratory analysis of the form of misfit.

The outline of this article is as follows. In the first part of this article, we review the logistic model for graded response data, describe the $S - X^{2}$ –test, and give an overview of the literature in the field. Then, we discuss the alternative implementations of the test and derive their asymptotic distribution. We also describe the graphical check of item fit. In the second part of this article, we investigate the performance of the tests in a simulation study. The third part contains an empirical application.

1. The Logistic Model for Graded Response Data

Item response models represent the distribution of the responses in the items of a test for a population of test takers. In this article, we focus on one specific item response model, the logistic model for graded response data (GRM model) of Samejima (1997).

Let a psychological test consist of $i = 1, \dots, I$ items with graded response options coded from 0 to M_i . The response x_i of a randomly drawn test taker in item i is regarded as a realization of the discrete random variate X_i with probability mass function $p_{X_{i}} (x)$ and sample space $Ω_{X_{i}} = {0, 1, \dots, M_{i}}$ . Define by $x = (x_{1}, \dots, x_{I})$ the response pattern of the test taker in all items of a test. The response pattern is the realization of the discrete random vector $X = (X_{1}, \dots, X_{I})$ with probability mass function $p_{X} (x)$ and sample space $Ω_{X} = Ω_{X_{1}} \times \dots \times Ω_{X_{I}}$ . In item response theory, each test taker can be characterized by a latent trait. The trait value $θ$ of a randomly chosen test taker is the realization of the continuous random variate $Θ$ with density function $f_{Θ} (θ)$ and sample space $Ω_{Θ} = ℝ$ . Here, we consider $Θ$ as standard normally distributed. Conditionally on the latent trait, the responses and the response pattern have conditional distributions with probability mass functions $p_{X_{i} | Θ} (x | θ)$ and $p_{X | Θ} (x | θ)$ . The GRM model specifies the functional relation between the values of the conditional probability mass functions and the latent trait. Its integral parts are the cumulative category response function and the category response function.

The cumulative category response function ${CCF}_{i} (θ, x)$ , $θ \in ℝ$ , $x \in Ω_{X_{i}}$ , $i = 1, \dots, I$ maps the latent trait and the response to the value of the conditional probability $Pr (X_{i} \geq x | Θ = θ)$ of responding with graded response option x or higher in item i. In the GRM model, the cumulative category response function is

{CCF}_{i} (θ, x) = \{\begin{array}{l} 1, x = 0 \\ \frac{exp [a_{i} (θ - b_{i x})]}{1 + exp [a_{i} (θ - b_{i x})]}, x = 1, \dots, M_{i} . \end{array}

The CCF function depends on two kinds of item parameters. Item parameter a_i ( $a_{i} \in ℝ$ ) is the item discrimination that determines the amount of influence the latent trait has on the response in item i. Item parameters $b_{i x}$ ( $b_{i x} \in ℝ$ ; $x = 1, \dots, M_{i}$ ) determine the value of the function at $θ = 0$ . Note that the item parameters $b_{i x}$ are assumed to be ordered ( $b_{i 1} < b_{i 2} < \dots < b_{i M_{i}}$ ). The category response function ${CRF}_{i} (θ, x)$ , $θ \in ℝ$ , $x \in Ω_{X_{i}}$ , $i = 1, \dots, I$ links levels of the latent trait and the responses to the conditional probability $Pr (X_{i} = x | Θ = θ)$ of responding with response option x in item i. The category response function can be derived from the CCF function as follows:

{CRF}_{i} (θ, x) = \{\begin{array}{l} {CCF}_{i} (θ, x) - {CCF}_{i} (θ, x + 1), x = 0, \dots, M_{i} - 1 \\ {CCF}_{i} (θ, x), x = M_{i} . \end{array}

In the GRM model, the responses to different items are assumed to be independent when conditioning on the latent trait. This assumption determines—in combination with the CCF functions—the probability mass function of the discrete random vector X representing the response pattern. The GRM model implies the probability mass function

Pr (X = x) = p_{X} (x) = \int \prod_{i = 1}^{I} p_{X_{i} | Θ} (x_{i} | θ) f_{Θ} (θ) dθ = \int \prod_{i = 1}^{I} {CRF}_{i} (θ, x) f_{Θ} (θ) dθ, x \in Ω_{X} .

2. The Generalized $S - X^{2}$ –Test of Item Fit

One aspect of model fit is the question whether the category response function of an item response model is representative of the true conditional mass functions of the items. Statistical tests intended to assess this aspect of model fit are usually denoted as tests of item fit, although in a strict sense, it is the whole model that fits or not. The first tests of item fit date back to Bock (1972), Yen (1981), and McKinley and Mills (1985). The general proceeding in these tests is very similar. First, the unknown latent traits of the test takers are replaced by estimates (e.g., maximum likelihood estimates or empirical Bayes estimates). The test takers are then grouped into strata of increasing trait level. For each stratum characterized by a typical trait level, the observed frequencies of the responses are determined as well as the frequencies that are expected according to the assumed category response function and the typical trait level. Item fit is then assessed by quantifying the correspondence of the observed and the expected frequencies over all strata. This is accomplished with one of the test statistics originally proposed for the analysis of frequency tables. The test statistics are usually compared to the quantiles of a $χ^{2}$ distribution. This proceeding, however, lacks a mathematical justification. More important, the $χ^{2}$ distribution does not work well as a reference distribution in practice. This is—at least partly—due to the fact that the test takers are grouped on the basis of an error prone trait estimate. For this reason, later implementations of the tests approximate the distribution of the test statistic via a parametric bootstrap (Rizopoulos, 2006) or account for the measurement error in the trait estimates explicitly (Stone, 2003). In an influential paper, Orlando and Thissen (2000) proposed an alternative test of item fit that does not require the grouping of the test takers into strata on the basis of an error prone trait estimate. The strata are formed on the basis of the observed test scores instead. The test was originally proposed for tests with binary responses but later generalized to tests with graded responses (Kang & Chen, 2008). This extended test—the generalized $S - X^{2}$ –test of item fit—is considered in the following.

Let the data consist of the response patterns of N randomly chosen test takers in a test with I items. The response patterns of the test takers are considered as independent realizations of the random vector X whose probability mass function is given in Equation 3 in case the GRM model holds. The number of possible response patterns is $| Ω_{X} |$ . We assume that the different response patterns have been numbered in an arbitrary order with index $g = 1, \dots, | Ω_{X} |$ indicating a specific response pattern. For simplicity, we will use the place holder G for the cardinality of the sample space $| Ω_{X} |$ in the following. Denote by $o (x_{g})$ the absolute frequency of response pattern x_g in the sample. The vector of all frequencies is $o (x) = (o (x_{1}), \dots, o (x_{G}))$ . The vector of all frequencies is the realization of the random vector $O (x)$ ( $Ω_{O (x)} = {0, \dots, N}^{G}$ ) with a multinomial distribution whose parameters are denoted as N and p. The values of p are the values of the probability mass function given in Equation 3 provided that the GRM model holds.

The theoretical counterparts to the observed frequencies of the response patterns $o (x)$ are their model-based expectations $e (x)$ . As the frequencies of the response patterns have a multinomial distribution, the model-based expectations are just $e (x) = N p$ where p is given in Equation 3. In practice, the model-based expectations are not available as the values of the item parameters (Equation 1) are not known. The model-based expectations, however, can be estimated on the basis of Equation 3 when the unknown values of the item parameters are replaced by marginal maximum likelihood estimates (Baker & Kim, 2004). We denote these estimated model-based expectations in the sample as $\hat{e} (x)$ with corresponding random variate $\hat{E} (x)$ ( $Ω_{\hat{E} (x)} = {0, \dots, N}^{G}$ ). Here, as in the following, we do not explicitly indicate the dependency of the estimated model-based expectations on the marginal maximum likelihood estimates of the item parameters.

In the $S - X^{2}$ –test, the observed frequencies $o (x)$ are compared to the model-based expectations $e (x)$ . In order to avoid sparse tables, the data are aggregated as follows. First, the frequency distribution of the test scores is determined. The test score is simply the sum of the graded responses in a response pattern; note that in a test with I items, the possible test scores range from 0 to $\sum_{i = 1}^{I} M_{i}$ . We denote a specific test score as s and the maximal test score as S in the following. Test score s occurs in the sample with frequency $o (s)$ , $s = 0, \dots, S$ . The frequency $o (s)$ is determined by summing the frequencies of all response patterns with a test score of s as $o (s) = \sum_{g = 1}^{G} o (x_{g}) I (x_{g}^{'} 1 = s)$ . Here, function $I (x)$ is the indicator function and vector $1$ a vector of ones. Additionally, the frequency $o_{i} (x, s)$ of the joint occurrence of a test score s and a response x in a selected target item i is determined by summing the frequencies of all response patterns with test score s and response x in item i as $o_{i} (x, s) = \sum_{g = 1}^{G} o (x_{g}) I (x_{g}^{'} 1 = s) I (x_{g [i]} = x)$ , $s = 0, \dots, S$ , $x = 0, \dots, M_{i}$ . With $x_{g [i]}$ , we refer to the response to item i in response pattern g. The corresponding model-based expectations are determined similarly by $\hat{e} (s) = \sum_{g = 1}^{G} \hat{e} (x_{g}) I (x_{g}^{'} 1 = s)$ , $s = 0, \dots, S$ and ${\hat{e}}_{i} (x, s) = \sum_{g = 1}^{G} \hat{e} (x_{g}) I (x_{g}^{'} 1 = s) I (x_{g [i]} = x)$ , $s = 0, \dots, S$ , $x = 0, \dots, M_{i}$ . Note that the frequency $o_{i} (x, s)$ and its estimated model-based expectation ${\hat{e}}_{i} (x, s)$ is zero whenever $x > s$ or $M_{i} - x > S - s$ . The sample statistics are realizations of the random variates $O (s)$ ( $Ω_{O (s)} = {0, \dots, N}$ , $s = 0, \dots, S$ ), $O_{i} (x, s)$ ( $Ω_{O_{i} (x, s)} = {0, \dots, N}$ , $x = 0, \dots, M_{i}$ , $s = 0, \dots, S$ , $i = 1, \dots, I$ ), $\hat{E} (s)$ ( $Ω_{\hat{E} (s)} = {0, \dots, N}$ , $s = 0, \dots, S$ ), and ${\hat{E}}_{i} (x, s)$ ( $Ω_{{\hat{E}}_{i} (x, s)} = {0, \dots, N}$ , $x = 0, \dots, M_{i}$ , $s = 0, \dots, S$ , $i = 1, \dots, I$ ).

The $S - X^{2}$ –test is an item specific test that evaluates the correspondence of the observed and expected frequencies. Its test statistic is defined for item i as

s x_{i}^{2} = \sum_{s = M_{i}}^{S - M_{i}} \sum_{x = 0}^{M_{i}} \frac{{\{o_{i} (x, s) - [{\hat{e}}_{i} (x, s) / \hat{e} (s)] \cdot o (s)\}}^{2}}{[{\hat{e}}_{i} (x, s) / \hat{e} (s)] \cdot o (s)}, i = 1, \dots, I .

In Equation 4, the sum excludes those levels of the test scores that imply at least one observed and one expected frequency of zero. In practice, some estimated model-based expectations ${\hat{e}}_{i} (x, s)$ are small. In this case, it is recommended to pool the frequencies over responses or test scores (Kang & Chen, 2008; Orlando & Thissen, 2000).

The $S - X^{2}$ –test has similarities to the standard X ²–tests for frequency tables (Moore, 1978). In order to notice this, the observed frequencies have to be arranged in a table where columns denote the test score and rows the response in item i. The quantity ${\hat{e}}_{i} (x, s) / \hat{e} (s)$ is the predicted probability of choosing response x in item i conditionally on receiving test score s. This is multiplied by $o (s)$ , the observed frequency of the test score. The $S - X^{2}$ –test, thus, compares the observed frequencies of the responses to their model-based predictions for strata defined by the test score. Despite this similarity, the $S - X^{2}$ –test and the standard X ²–tests differ fundamentally with respect to the underlying sampling models, such that the asymptotic theory for X ²–tests cannot be applied to the $S - X^{2}$ –test. On the basis of heuristic considerations, the $s x_{i}^{2}$ statistic is evaluated with respect to a $χ^{2}$ distribution with $(M_{i}) (S - 2 M_{i} + 1) - (M_{i} + 1)$ degrees of freedom. The degrees of freedom are the number of test scores considered in Equation 4 multiplied by the number of response categories minus one reduced by the number of the item parameters in item i. In practice, some test scores have a frequency of zero. In this case, the test score is not considered when determining the degrees of freedom. Pooling cells within a test score or merging rows of adjacent test scores reduces the degrees of freedom further. The $S - X^{2}$ –test usually controls the Type-I error well. It, however, does not perform well in small samples and tests with many items (Kang & Chen, 2008).

3. Alternative Implementations of the Test and the $S - X^{2}$ –Residuals

In this section, we present two alternative versions of the $S - X^{2}$ –test. The first version—denoted as LI test in the following—is a linear function of limited information statistics. The second test—denoted as $S - R^{2}$ –test—is a modification of the original $S - X^{2}$ –test.

3.1. LI Test

The formula of the $S X_{i}^{2}$ statistic given in Equation 4 can be modified as follows. The statistic contains the factor $o (s) / \hat{e} (s)$ with corresponding random variate $O (s) / \hat{E} (s)$ . The factor $O (s) / \hat{E} (s)$ converges to 1 in probability provided that the GRM model holds. For this reason, it suggests itself to consider simply the differences $o_{i} (x, s) - {\hat{e}}_{i} (x, s)$ . Note that the components of the differences are linear combinations of the observed frequencies $o (x)$ and their estimated model-based expectations $\hat{e} (x)$ in the sample. They are, thus, limited information statistics and can be interpreted as generalized residuals (e.g., Haberman & Sinharay, 2013; Maydeu-Olivares & Joe, 2005). This is beneficial as all theoretical findings for limited information statistics can be applied; see Online Appendix A for more details. As an alternative to the $S X_{i}^{2}$ statistic, we suggest assessing item fit by simply adding the squared differences over the test scores and the response options:

l i_{i}^{2} = 1 / N \cdot \sum_{s = M_{i}}^{S - M_{i}} \sum_{x = 0}^{M_{i}} {[o_{i} (x, s) - {\hat{e}}_{i} (x, s)]}^{2}, i = 1, \dots, I .

Similar to the $S X_{i}^{2}$ statistic, we exclude test scores where observed frequencies are necessarily zero. The value of the $L I_{i}^{2}$ statistic in the sample is the realization of a random variate whose distribution can be approximated as follows. The differences $o_{i} (x, s) - {\hat{e}}_{i} (x, s)$ are realizations of the random variates $O_{i} (x, s) - {\hat{E}}_{i} (x, s)$ that have a multivariate normal distribution in the limit when divided by $\sqrt{N}$ and the GLM model holds. The expected values of the differences are zero. This follows from findings for limited information statistics; see Online Appendix A for more details. The value of the $L I_{i}^{2}$ statistic in the sample is, thus, the realization of a sum of squared centered multivariate normally distributed random variates. The asymptotic distribution of such sums is a mixture of $χ^{2}$ distributions when scaled properly (Yuan & Bentler, 2010). The asymptotic distribution can be approximated as follows. Denote by $Σ_{D_{i}}$ the variance covariance matrix of all $(M_{i} + 1) (S - 2 M_{i} + 1)$ differences that are considered in Equation 5. A description of how $Σ_{D_{i}}$ can be estimated is given in Online Appendix A. Determine then the two quantities $c_{i 1} = \sum_{k} λ_{k i}^{2} / \sum_{k} λ_{k i}$ and $c_{i 2} = (\sum_{k} λ_{k i})^{2} / \sum_{k} λ_{k i}^{2}$ , where $λ_{k i}$ ( $k = 1, \dots, (M_{i} + 1) (S - 2 M_{i} + 1))$ are the eigenvalues of $Σ_{D_{i}}$ . With $c_{i 1}$ and $c_{i 2}$ , the distribution of the $L I_{i}^{2}$ statistic can be approximated by that of the scaled χ² random variate $c_{i 1} \cdot χ_{c_{i 2}}^{2}$ with $c_{i 2}$ degrees of freedom. The scaling factor and degrees of freedom are chosen in order to match the moments of $c_{i 1} \cdot χ_{c_{i 2}}^{2}$ with those of $L I_{i}^{2}$ ; see Yuan and Bentler (2010) for more details. Similar approximations are of wide use in categorical data analysis and date back to Welch (1938).

As a reviewer pointed out, a scaled χ² distribution is a special case of the gamma distribution. Hence, the asymptotic distribution of $L I_{i}^{2}$ can also be approximated by a Gamma distribution with shape parameter $c_{i 2} / 2$ and scale parameter $2 c_{i 1}$ :

L I_{i}^{2} \sim Gamma (c_{i 2} / 2, 2 c_{i 1}) i = 1, \dots, I .

This approximation is more natural as the degrees of freedom $c_{i 2}$ are usually not a natural number. Although the components of the $S - X^{2}$ –test and the LI–test appear to be similar and $O (s) / \hat{E} (s)$ converges to one in probability, their asymptotic distribution is different. This is due to the fact that the $S - X^{2}$ –test and the LI–test differ in how the distribution of the test score $O (s)$ affects the test. In the $S - X^{2}$ –test, the sum of the predicted frequencies $[e_{i} (x, s) / e (s)] \cdot o (s)$ over x is necessarily identical to $o (s)$ . This is not the case in the LI–test; for further differences, see Online Appendix A.

3.2. $S - R^{2}$ –Test

The $S - R^{2}$ –test is based on the $S - X^{2}$ –residuals that appear in the numerator of the $S X_{i}^{2}$ statistic:

r_{i} (x, s) = o_{i} (x, s) - {\hat{e}}_{i} (x, s) \cdot [o (s) / \hat{e} (s)], x = 0, \dots, M_{i}, s = 0, \dots, S, i = 1, \dots, I .

All components of $S - X^{2}$ –residuals are limited information statistics that result from aggregating the frequencies of certain response patterns (Maydeu-Olivares & Joe, 2005). The components can, thus, be interpreted as generalized residuals (Haberman & Sinharay, 2013); see Online Appendix A for more details. The observed residual $r_{i} (x, s)$ is a realization of the random variate $R_{i} (x, s)$ ( $Ω_{R_{i}} = [- N, N]$ ). This random variate is a function of the random variates $O_{i} (x, s)$ , ${\hat{E}}_{i} (x, s)$ , $O (s)$ , and $\hat{E} (s)$ . As these random variates are linear functions of the frequencies of the response patterns, they have a multivariate normal distribution in the limit provided they are properly scaled; see the paragraph above. In contrast to the LI-test, the $S - X^{2}$ –residuals are not a linear function of their constituents. The asymptotic distribution of $1 / \sqrt{N} \cdot R_{i} (x, s)$ can be derived by an application of the Delta method where Equation 7 is approximated by a truncated Taylor expansion (Agresti, 1990, p. 424). We refer to Online Appendix A for more details. In summary, it can be shown that the residuals $R_{i} (x, s)$ are asymptotically distributed according to a multivariate normal distribution with expectation of zero and variance of $Σ_{R_{i}}$ when divided by $\sqrt{N}$ .

As an alternative to the $S X_{i}^{2}$ statistic, we suggest summing the squared $S - X^{2}$ –residuals:

s r_{i}^{2} = 1 / N \cdot \sum_{s = M_{i}}^{S - M_{i}} \sum_{x = 0}^{M_{i} - 1} {[r_{i} (x, s)]}^{2}, i = 1, \dots, I .

In Equation 8, we exclude the same residuals as in the original $S X_{i}^{2}$ statistics. We additionally exclude the residuals in the highest response option (M_i ) as they are linearly dependent on the remaining residuals. The value of the $S R_{i}^{2}$ statistic is the realization of a random variate whose asymptotic distribution is a mixture of $χ^{2}$ random variates when scaled properly for the same reasons as given above. We suggest the same approximation for this distribution. Denote by $Σ_{R_{i}}$ the variance covariance matrix of all $(M_{i}) (S - 2 M_{i} + 1)$ differences that enter the calculation of the test statistic and let $λ_{k i}$ ( $k = 1, \dots, (M_{i}) (S - 2 M_{i} + 1))$ be the eigenvalues of $S_{R_{i}}$ . With $c_{i 1} = \sum_{k} λ_{k i}^{2} / \sum_{k} λ_{k i}$ and $c_{i 2} = (\sum_{k} λ_{k i})^{2} / \sum_{k} λ_{k i}^{2}$ , the distribution of the $S R_{i}^{2}$ statistic is approximated as

S R_{i}^{2} \sim Gamma (c_{i 2} / 2, 2 c_{i 1}) i = 1, \dots, I,

where $c_{i 2} / 2$ is the shape parameter and $2 c_{i 1}$ the scale parameter. The $S - R^{2}$ –test differs from the $S - X^{2}$ –test in how the discrepancy is measured. There is, however, another difference. The $S - R^{2}$ –test is based on the interpretation of the test scores as multinomial distributed random variates. Our framework is, thus, the framework of multinomial sampling. The $S - X^{2}$ –test, on the other hand, is based on a different sampling model as the distribution of the test score is considered as fixed.

4. A Graphical Check of Item Fit

Tests of item fit can be used in order to identify items that require a closer inspection. The $S - X^{2}$ –residuals can be used for post hoc tests that might clarify for which test scores the observed frequencies and the model-based expectations diverge. Such analyses are sometimes useful in order to clarify at what trait level the assumed category response function is invalid. For a further analysis of the source of the misfit, we suggest a graphical plot. The plot is based on the interpretation of the graded responses as realizations of originally continuous but then categorized random variates. For the plot, the values of the originally continuous random variates—the so-called augmented residuals—are recovered. The augmented residuals are then used for graphical checks of model fit in a similar way as the residuals in regression analysis. The idea of using augmented residuals was introduced by Liu and Zhang (2018) for the visualization of model fit in the ordinal logit model. Here, we adapt this approach to item response theory. Graphical checks of model fit have been proposed before; see, for example, the suggestions of Sijtsma (1998), Douglas and Cohen (2001), Haberman et al. (2013), or Sinharay (2006). The latter two suggestions even allow for inferential statements. Our plot is not supposed to replace these suggestions. We consider the plot as a complement that gives additional insight into the source of misfit. The proposed plot is simple to interpret as it is not based on estimates of the different category response functions. Furthermore, it allows for a distinction of different sources of misfit. This is similar to the residual plots in regression analysis where linearity and variance homogeneity can be evaluated separately.

The GRM model can be interpreted as a linear factor model with logistic distributed residuals where the originally latent continuous responses are categorized at thresholds (e.g., Takane & de Leeuw, 1987). In this interpretation, the latent response of a test taker in an item is characterized as $X_{i}^{*} = a_{i}^{*} + b_{i}^{*} Θ + U$ ( $Ω_{X_{i}^{*}} = ℝ$ ) where random variate $Θ$ is the latent trait, random variate U ( $Ω_{U} = ℝ$ ) a residual with a standard logistic distribution, and $a_{i}^{*}$ ( $a_{i}^{*} \in ℝ$ ) and $b_{i}^{*}$ ( $b_{i}^{*} \in ℝ$ ) are item parameters. The observable graded responses result from the categorization of the latent continuous responses. Assume that the response options in item i determine M_i ordered thresholds $τ_{i 1}$ <…< $τ_{i M_{i}}$ ( $τ_{i 1}, \dots, τ_{i M_{i}} \in ℝ$ ). The observed graded response x_i is just the number of thresholds that are exceeded, that is $x_{i} = \sum_{j = 1}^{M_{i}} I (x_{i}^{*} > τ_{i j})$ . It can be shown that the conditional probability of a response $Pr (X_{i} = x | Θ = θ) = Pr (\sum_{j = 1}^{M_{i}} I (X_{i}^{*} > τ_{i j} | Θ = θ)$ can be represented by the CRF function given in Equation 2 when adequate item parameter values and thresholds are chosen; see Online Appendix B for more details. For the plot, we recover values of U—the so-called augmented residuals—for all test takers in an item.

The proceeding is as follows. For each test taker in the sample, we draw a latent trait value $θ$ from the standard normal distribution and augmented residuals u_i ( $i = 1, \dots, I$ ) from the standard logistic distribution. Then, the corresponding response pattern of graded responses is determined by categorizing the implied continuous responses $x_{i}^{*} = a_{i}^{*} + b_{i}^{*} θ + u_{i}$ at the thresholds $τ_{i 1}$ ,…, $τ_{i M_{i}}$ . The parameters $a_{i}^{*}$ and $b_{i}^{*}$ and the thresholds $τ_{i 1}$ ,…, $τ_{i M_{i}}$ are replaced by the values that are implied by the marginal maximum likelihood estimates of the GRM model parameters; more details on this are given in Online Appendix B. When the generated data are coherent with the observed response pattern of the test taker, the augmented residual in target item i is retained. With coherent we denote the case that the generated response in the target item is identical to the response of the test taker and the sum of the remaining generated responses is identical to the test score minus the response in the target item. When the generated data are not coherent, new augmented residuals are simulated. This proceeding is repeated for all test takers in the sample. In order to reduce the randomness, one can generate several replications of augmented residuals for each test taker.

The augmented residuals of the test takers in a target item i are plotted against the test score of the test takers in the remaining items. Here, in contrast to the tests of model fit discussed above, the target item i is excluded when summing the responses. In case the model holds (and the real item parameters are used when generating the augmented residuals), the augmented residuals are distributed according to a standard logistic distribution in all strata of different test scores; a proof of this is given in Online Appendix B. Whether this distribution holds for the augmented residuals can be checked graphically with any plot developed for the evaluation of model fit in linear regression models. We suggest plotting the augmented residuals and the squared augmented residuals against the test score. The residuals should scatter randomly around zero and the squared residuals randomly around $π^{2} / 3$ .

5. Simulation Study

In order to compare the performance of the tests, we conducted a small-scale simulation study consisting of two scenarios. In the first scenario, we assessed the size of the tests, in the second scenario their power.

5.1. Simulation Scenario I

In the first simulation scenario, the data were generated and analyzed with the GRM model. The data consisted of the responses to items with five graded response options. The data were generated as follows. The latent traits of the fictitious test takers were drawn from a standard normal distribution. Then, the conditional probabilities of choosing one of the five response options were determined according to Equation 2. Having determined the probabilities, a response was generated for all combinations of a test taker with an item. Three different sample sizes were considered, namely, sample sizes of $N = 125$ , $N = 250$ , and $N = 1,000$ . The length of the test was either $I = 5$ , $I = 10$ , or $I = 20$ . These test lengths and sample sizes are typical for data in personality assessment and were chosen in accordance to previous simulation studies (Kang & Chen, 2008; LaHuis et al., 2011). Item parameter values were chosen such that the distribution of the responses varied systematically, being right skewed, symmetric, or left skewed; the exact values of the item parameters are reported in Online Appendix C. Fully crossing the three sample sizes with the three lengths of the test defined $3 \times 3$ simulation conditions for which $1,000$ simulation samples were generated.

The data were analyzed with the GRM model. The item parameters were estimated by marginal maximum likelihood estimation using the mirt package (Chalmers, 2012) from the statistical software environment R (R Development Core Team, 2009). Three tests were performed, namely, the original $S - X^{2}$ –test, the $LI$ –test, and the $S - R^{2}$ –test. For the $S - X^{2}$ –test, we used the implementation in the mirt package (Chalmers, 2012). Three different versions of the $S - X^{2}$ –test were requested. In the first version, no cells were pooled, that is, the observed frequencies and model-based expectations were compared for all test scores that occurred at least once. In the second version, cells with small model-based expectations were combined such that all combined cells had an expected frequency of at least one. This is the default implementation of the $S - X^{2}$ –test in the mirt package. In the third version, cells were combined such that all combined cells had an expected frequency of at least five. This is the size that is usually recommended for the analysis of frequency tables (Agresti, 1990, p. 246). The $LI$ –test and the $S - R^{2}$ –test were implemented as described above. The marginal probabilities of the response patterns (Equation 3) were determined with Gauss-Hermite quadrature using $20$ nodes (Stroud, 1971). The simulation study was implemented in the software environment R.

The results for the different tests and the different simulation conditions are reported in Table 1. Here, the empirical rejection rates $P_{α}$ of the tests are tabulated for different nominal Type-I error rates $α$ . We also report the Kolmogorov–Smirnov (KS) statistic that measures the difference between the empirical distribution function of the p values and the uniform $U [0, 1]$ distribution for the different tests. A small value of the KS statistic implies that a test works as expected. In Table 1, the results have been averaged over the items. Results were similar for the individual items.

Table 1.

Type-I Error Rates ( $P_{α}$ ) of the Three Tests of Item Fit for Different Amounts of Aggregation, Nominal Type-I Error Rates (α), Sample Sizes (N), and Test Lengths (I) as Well as the Statistic of the Kolmogorov–Smirnov (KS) Test Assessing Uniformity

Test	Agg.	$α$	I = 5			I = 10			I = 20
Test	Agg.	$α$	N = 125	N = 250	N = 1,000	N = 125	N = 250	N = 1,000	N = 125	N = 250	N = 1,000
$S - X^{2}$	None	$P_{0.10}$	.120	.123	.116	.137	.144	.141	.150	.158	.162
		$P_{0.05}$	.088	.088	.068	.103	.117	.107	.110	.121	.126
		$P_{0.01}$	.053	.049	.026	.065	.080	.070	.064	.071	.084
		KS	.167	.134	.060	.197	.218	.218	.126	.159	.216
	e = 1	$P_{0.10}$	.125	.107	.100	.134	.111	.101	.157	.116	.093
		$P_{0.05}$	.066	.049	.050	.064	.056	.051	.076	.057	.045
		$P_{0.01}$	.016	.012	.009	.012	.011	.011	.014	.011	.009
		KS	.112	.050	.030	.144	.073	.031	.177	.094	.032
	e = 5	$P_{0.10}$	.572	.228	.108	.469	.274	.120	.419	.254	.135
		$P_{0.05}$	.383	.125	.052	.303	.159	.062	.258	.144	.070
		$P_{0.01}$	.127	.028	.010	.093	.040	.013	.075	.034	.014
		KS	.598	.250	.051	.512	.305	.068	.464	.285	.097
LI	None	$P_{0.10}$	.098	.099	.104	.117	.105	.112	.104	.112	.103
		$P_{0.05}$	.052	.048	.052	.061	.053	.060	.054	.058	.055
		$P_{0.14}$	.010	.011	.014	.012	.012	.014	.014	.013	.011
		KS	.041	.027	.028	.065	.056	.053	.042	.042	.033
$S - R^{2}$	None	$P_{0.10}$	.090	.093	.092	.094	.104	.114	.086	.098	.102
		$P_{0.05}$	.046	.050	.047	.046	.051	.059	.043	.052	.051
		$P_{0.01}$	.010	.012	.013	.010	.014	.012	.009	.012	.012
		KS	.034	.029	.028	.035	.041	.049	.043	.029	.034

Note. Results are based on 1,000 simulation samples. Results have been averaged over the different items. $S - X^{2}$ = original $S - X^{2}$ –test given in Equation 4; $LI$ = LI–test given in Equation 5; $S - R^{2}$ = modified $S - X^{2}$ –test given in Equation 8; Agg. = aggregation of sparse cells.

The results in Table 1 suggest that the $LI$ –test and the $S - R^{2}$ –test achieve to control the Type-I error rate well under all conditions. The distribution of the p values is also close to the uniform distribution. Note that for 1,000 replications, the critical value of the KS test is $1.36 / \sqrt{1,000} = 0.043$ for $α = .05$ . Findings are different for the $S - X^{2}$ –test whose performance depends on the sample size, the length of the test, and the amount of pooling. In samples of $125$ test takers, the empirical rejection rates are too high irrespective of the amount of pooling. In the samples of $250$ and $1,000$ test takers, the $S - X^{2}$ –test controls the Type-I error rate well provided that cells are pooled to an expected frequency of one. The KS statistics of the $S - X^{2}$ –test are still rather large in longer tests and often larger than the KS statistic of the $S - R^{2}$ –test. The $S - X^{2}$ –test performs less well when cells are not pooled at all or when cells are pooled to an expected frequency of five. This is remarkable as the last finding contradicts the typical recommendation for the standard X ²–test.

In addition to the performance of the tests, we analyzed the distribution of the $S - X^{2}$ –residuals given in Equation 7; note that the $S - X^{2}$ –residuals could be used for a post hoc analysis in order to determine the region where the CRF functions are misspecified. For the analysis, we tested whether the residuals’ expectations deviated from zero on the basis of Wald’s tests; note that an estimate of the covariance matrix of the $S - X^{2}$ –residuals is a by-product of the $S - R^{2}$ –test. The empirical rejection rates are visualized in Figure 1 for $α = .05$ separately for test scores of 4 to $4 I - 4$ and the five response categories. For lack of space, only the results for one prototypical item are reported (Item 3). Results were similar for the remaining items. Figure 1 also contains a 95% confidence band within each ofthe empirical rejection rates should be located.

Figure 1.

Empirical rejection rates of a test whether the expectation of the $S - X^{2}$ –residuals is zero in Item 3, first simulation scenario for $α = .05$ and different test scores s ranging from 4 to $4 I - I$ . Results are reported separately for the different combinations of the test length I and the sample size N. The straight line indicates the level of $α$ , and the dotted lines indicates a 95% confidence band within the empirical rejection rates should be located. Different plotting symbols refer to different response options x ( $□ = 0$ , $o = 1$ , $▵ = 2$ , $♦ = 3$ , $▾ = 4$ ).

Figure 1 implies that the empirical rejection rates are close to the intended nominal Type-I error rate of $α = .05$ . Only in low and high test scores with low frequency, the Wald test is not capable of controlling the Type-I error rate well. Overall, these findings suggest that the $S - X^{2}$ –residuals can be used for post hoc analyses.

5.2. Simulation Scenario II

In the second simulation scenario, we investigated the power of the tests. For this purpose, we generated data that violate the GRM model. As the model can be interpreted in terms of a linear factor model with originally continuous, but then categorized indicators—see the section on the augmented residuals above and Online Appendix B—we studied model misspecification in terms of a misspecified linear factor model. Two forms of misspecification were considered, variance heterogeneity (Condition A) and nonlinearity (Condition B).

5.2.1. Condition A

The GRM model can be derived from a linear factor model with standard logistic distributed responses when the continuous responses are categorized (e.g., Takane & de Leeuw, 1987). In factor analysis, the conditional expectation of the latent continuous responses is a linear function of the latent trait. The conditional variance is constant. In Condition A, we misspecified the GRM model as we related both, the conditional expectation and the conditional variance of the latent continuous response to the latent trait. This is similar to variance heteroscedasticity in a linear regression model. Such violations are due to unexplained heterogeneity.

Data were generated for a test of 10 items and 1,000 test takers. Other sample sizes and numbers of items were not considered anymore. We chose this setting as in 10 items and 1,000 test takers all tests adhered to the nominal Type I error rate well; see Table 1. This is necessary in order to compare the power of the tests. The data were generated as follows. We first drew a latent trait from the standard normal distribution for each test taker. We then generated a latent continuous response to each item by a linear combination of the trait and a residual that was drawn randomly from the logistic distribution. The residuals were drawn from a standard logistic distribution in eight of the $10$ items. These were the regular items. In two items, Item 3 and Item 8, respectively, the residuals were drawn from a logistic distribution with expectation of zero and standard deviation of $π / \sqrt{3} \cdot exp (0.25 θ)$ . These were the irregular items; note that $π / \sqrt{3}$ is the standard deviation of the standard logistic distribution. The responses were then categorized at thresholds that corresponded to the graded response options in the items. When generating the data, we ensured that the implied category response functions were identical to the category response functions of the first simulation scenario in the regular items. The true category response functions as well as their best approximations in the Kullback–Leibler sense (White, 1982) that are provided by the GRM model are visualized in Online Appendix C for the irregular items. Altogether 1,000 simulation samples were generated.

Data were analyzed as in the first simulation scenario. A GRM model was fitted to the data via marginal maximum likelihood estimation. Then, the tests of item fit were performed, namely, the $S - X^{2}$ –test, the $LI$ –test, and the $S - R^{2}$ –test. For the $S - X^{2}$ –test, the cells were aggregated in order to achieve an expected frequency of one. As we noted that the power of the tests increased by aggregation, we also aggregated cells for the $LI$ –test and the $S - R^{2}$ –test for the sake of comparability. We, however, also performed the tests without aggregation. The simulation study was implemented in the software environment R as in the first simulation scenario. All scripts are available from the authors on request.

The empirical rejection rates of the tests are reported in Table 2 for different nominal Type-I error rates $α$ . Results are reported for the irregular items (Power) and the regular items (Type-I error). Results have been aggregated over the different items.

Table 2.

Empirical Rejection Rates of Different Tests in the Second Simulation Scenario and Condition A Reported Separately for Regular and Irregular Items, Different Levels $α$ , and Different Amounts of Aggregation (Agg./Com)

Items		Irregular			Regular
$α$		.10	.05	.01	.10	.05	.01
$S - X^{2}$	Agg.	.439	.324	.140	.106	.055	.010
$LI$	Com	.301	.190	.063	.121	.068	.016
$LI$	Agg.	.309	.206	.074	.103	.052	.010
$S - R^{2}$	Com	.264	.172	.052	.123	.065	.016
$S - R^{2}$	Agg.	.342	.232	.098	.086	.041	.009

The tests have moderate power to detect the irregular items. The $S - X^{2}$ –test is most powerful. The $LI$ –test and the $S - R^{2}$ –test without aggregation have the lowest power. Aggregating cells increase the power slightly. In the regular items, the empirical rejection rate is close to $α$ . This suggests that the test is capable of distinguishing between regular and irregular items; note, however, that although we classify items as regular or irregular, this is not fully justified. It is always the whole model that fits or not. So, in a strict sense, one cannot interpret the rejection frequencies in the regular items as Type-I error rates and in the irregular items as power; note also that in practice, one probably would use a correction for multiple testing which would decrease the power of the tests further.

In addition to the global tests of item fit, we performed local tests of item fit on the basis of the $S - X^{2}$ –residuals. For this purpose, we tested whether the $S - X^{2}$ –residuals have an expectation of zero. We used Wald’s tests and a nominal Type-I error rate of $α = .05$ . The empirical rejection rates are visualized in Figure 2 for the different response categories, test scores and four items. Item 3 and Item 8 are irregular items. Item 2 and Item 9 are regular items with similar item parameters. Figure 2 also contains a 95% confidence band within the observed rejection rates should fall when their expectation is $α = .05$ .

Figure 2.

Empirical rejection rates of a Wald test assessing whether the expectation of the $S - X^{2}$ –residuals is zero as a function of the test score s and the response category in four items of a test for $α = .05$ and simulation scenario II, Condition A. The test score ranges from 4 to $36$ . The straight line denotes the level of $α$ , and the dotted lines a 95% confidence band within the observed rejection rates should fall when their expectation is $0.05$ . Different plotting symbols refer to different response options x ( $□ = 0$ , $o = 1$ , $▵ = 2$ , $♦ = 3$ , $▾ = 4$ ).

For the regular items, the empirical rejection rates are close to $α = .05$ . Similar to the first simulation scenario (Figure 1), there is some deviation in small and large test scores that occur infrequently. In the irregular items, the empirical rejection rates exceed $α = .05$ . The highest rejection rates are achieved in test scores from $25$ to $40$ and response category $x = 0$ .

We also investigated the misfit graphically with the augmented residual plots suggested in the previous section. Figure 3 contains a plot of the augmented and the squared augmented residuals against the test score in the regular Item 2 and the irregular Item 3 for an exemplary data set. The plots also include a curve (red line) depicting an estimate of the conditional expectation that was determined with a loess smoother. Any deviation of the curve from its expectation under model fit (black line) indicates a form of misspecification. The plots are interpreted in the same way as the residual plots in regression analysis. The plots reveal that the expected values of the squared augmented residuals depend on the test score in Item 3. This accords with the way the misspecification was induced.

Figure 3.

Plot of augmented residuals U (left side) and squared augmented residuals U ² (right side) against the test score s in regular Item 2 and irregular Item 3 of simulation scenario II, Condition A. The red line visualizes an estimate of the conditional expectation as a function of the test score that was determined with a loess smoother, and the black line the expected value under correct model specification.

5.2.2. Condition B

In Condition B, we violated the linearity assumption of the GRM model with respect to the underlying continuous, but then categorized latent response. In two items, Item 3 and Item 8, we assumed that the expected values decreased quadratically in the range $θ \leq 0$ , but then increased linearly for $θ > 0$ ; for evidence for nonmonotonous relations between the trait $θ$ and the response tendency in personality and attitudinal scales, see Meijer and Baneke (2004) and Stark et al. (2006). The nonmonotonous relation in the latent continuous response implies nonmonotonous CCF functions.

Apart from the form of misspecification, the proceeding in Condition B was similar to the proceeding in Condition A. Data were generated for a test of 10 items and a sample size of 1,000 test takers. Data were simulated as follows. We first drew a latent trait from the standard normal distribution for each test taker. We then generated a latent continuous response to each item by using a linear combination of the trait and a logistically distributed residual in the regular items or by adding a quadratic function of the trait and a logistic distributed residual in the irregular items. The responses were than categorized at thresholds that corresponded to the graded response options in the items. When generating the data, we ensured that the implied category response functions of the correctly specified items were identical to the ones of the first simulation scenario. The true category response function as well as their best approximations in the Kullback–Leibler sense (White, 1982) provided by the GRM model are visualized in Online Appendix C for the irregular items. Altogether 1,000 simulation samples were generated.

Data were analyzed as in the first simulation scenario. A GRM model was fitted to the data via marginal maximum likelihood estimation. Then, the tests of item fit were performed, namely, the $S - X^{2}$ –test, the $LI$ –test, and the $S - R^{2}$ –test. For the $S - X^{2}$ –test, the cells were aggregated in order to achieve an expected frequency of one. As we noted that the power of the tests increased by aggregation, we also aggregated cells for the $S - R^{2}$ –test and the $LI$ –test for comparability, but also performed the tests without aggregation.

The empirical rejection rates of the tests are reported in Table 3 for different nominal Type-I error rates $α$ . Results are reported for the irregular items (Power) and the regular items (Type-I error). Results have been aggregated over the different items.

Table 3.

Empirical Rejection Rates of Different Tests in the Second Simulation Scenario and Condition B Reported Separately for Regular and Irregular Items, Different Levels $α$ , and Different Amounts of Aggregation (Agg./Com)

Items		Irregular			Regular
$α$		.10	.05	.01	.10	.05	.01
$S - X^{2}$	Agg.	.242	.146	.037	.110	.057	.011
$LI$	None	.214	.126	.042	.185	.110	.034
$LI$	Agg.	.528	.418	.216	.177	.103	.031
$S - R^{2}$	None	.146	.091	.030	.126	.067	.016
$S - R^{2}$	Agg.	.393	.283	.120	.113	.058	.012

The power of the $S - X^{2}$ –test, the power of the $LI$ –test without aggregation, and the power of the $S - R^{2}$ –test without aggregation are very low. The power increases when sparse cells are aggregated. In this case, the $LI$ –test detects the irregular items with a power of $.418$ at $α = .05$ . The $S - R^{2}$ –test has a power of $.393$ . The rejection rates in the regular items are slightly above $α$ in all tests, an exception being the $LI$ –test. This implies that the $LI$ –test is not very specific. Although it is sensitive to the presence of misspecification, it cannot identify the items that are misspecified. The $S - R^{2}$ –test with aggregation is better in this respect.

In addition to the global tests of item fit, we performed local tests of item fit on basis of the $S - X^{2}$ –residuals. We tested whether the $S - X^{2}$ –residuals have an expectation of zero with a Wald test. The nominal Type-I error rate was set to $α = .05$ . The empirical rejection rates are visualized in Figure 4 for the different response categories, test scores, and four items. Item 3 and Item 8 are irregular items, and Item 2 and Item 9 are regular items. Figure 4 also contains a 95% confidence band within the observed rejection rates should fall when their expectation is $α = .05$ .

Figure 4.

Empirical rejection rates of a Wald test assessing whether the expectation of the $S - X^{2}$ –residuals is zero as a function of the test score s and the response category in four items of a test for $α = .05$ and simulation scenario II, Condition B. The test score ranges from 4 to $36$ . The straight line denotes the level of $α$ , and the dotted lines denote a 95% confidence band within the observed rejection rates should fall when their expectation is $0.05$ . Different plotting symbols refer to different response options x ( $□ = 0$ , $o = 1$ , $▵ = 2$ , $♦ = 3$ , $▵ = 4$ ).

The power to detect deviations from the assumed category response function is rather low. In the irregular items, the empirical rejection rates are highest around a test score of $s = 32$ where it reaches a value of about $0.20$ . The empirical rejection rates are close to $α = .05$ for test scores below $20$ .

We also investigated the type of misfit graphically with the augmented residual plot suggested in the previous section. Figure 5 contains a plot of the augmented and the squared augmented residuals in the regular Item 7 and the irregular Item 8 for an exemplary data set. The plots also contain a curve (red line) representing the estimated conditional expectation that was determined with a LOESS smoother. The conditional expectation under model fit is represented by a black line. The plots are interpreted in the same way as the standard residual plots in regression analysis. Any deviation of the two curves indicates a misspecification of the model. The plots reveal that the relation between the latent trait and the conditional expectation of the augmented residuals is not linear in the irregular items.

Figure 5.

Plot of augmented residuals U (left side) and squared augmented residuals U ² (right side) against the test score s in regular Item 7 and irregular Item 8 of simulation scenario II, condition B. The red line visualizes an estimate of the conditional expectation as a function of the test score that was determined with a loess smoother. The black line visualizes the expected value of the augmented residual under correct model specification.

6. Empirical Example

The use of the proposed tests is illustrated by means of a real-data application. The data set contained the real-life responses of $N = 280$ participants to the 20-item German-language version of the Fear of Negative Evaluation Scale (FNES; Watson & Friend, 1969; German: Vormbrock & Neuser, 1983). The FNES is a frequently used instrument to assess individual differences in the anxiety to be disparagingly judged by others. Responses are given on a 5-point Likert-type scale (0 = almost never; 4 = almost always) and a sample item is “I am afraid that others will not approve of me.” The raw data are available from the authors upon request. Test scores ranged from 0 to $75$ . Extreme test scores, however, were very infrequent. In the sample, 79% of all test scores were between $20$ and $60$ .

Results are summarized in Table 4. Table 4 contains the parameter estimates and the results of the $S - X^{2}$ -test, the $LI$ -test, and the $S - R^{2}$ -test. In all tests, the cells were pooled. Reversed items were recoded before performing the tests; note that results are different when reversed items are not recoded. Due to the different degrees of freedom, the original values of the test statistics cannot be compared directly. For this reason, we report the p values of the tests and quantiles that correspond to the tests’ p values for a reference $χ^{2}$ distribution with $292$ degrees of freedom. These allow the comparison of the quantiles over the different tests.

Table 4.

Item Parameter Estimates, p Values, and Corresponding Quantiles Q of a Reference $χ^{2}$ Distribution in the Items of the Fear of Negative Evaluation Scale

	Item Parameter					$S - X^{2}$ –test		$LI$ –test		$S - R^{2}$ –test
Item	a_i	$b_{i 1}$	$b_{i 2}$	$b_{i 3}$	$b_{i 4}$	p	Q	p	Q	p	Q
1	1.04	−0.92	0.57	2.04	3.99	.49	292.05	.01	351.08	.23	309.83
2	2.85	−1.23	−0.31	0.56	1.56	.08	325.79	.02	344.37	.09	324.36
3	2.20	−1.51	−0.51	0.47	1.68	.17	315.24	.00	361.16	.05	332.83
4	0.53	−5.52	−2.63	−0.16	2.64	.06	331.19	.00	358.40	.21	311.20
5	2.35	−2.07	−1.00	0.29	1.36	.57	286.84	.23	309.53	.95	253.58
6	2.23	−1.63	−0.73	0.29	1.56	.04	334.60	.00	383.37	.02	343.83
7	0.67	−4.85	−1.20	0.33	2.63	.04	336.45	.00	363.33	.08	327.10
8	2.27	−1.08	−0.01	0.98	1.90	.58	286.37	.20	311.81	.91	259.68
9	3.66	−1.30	−0.36	0.45	1.43	.06	330.90	.12	320.16	.72	277.78
10	2.11	−1.04	0.09	0.89	1.84	.73	276.58	.08	327.27	.55	288.36
11	0.60	−4.14	−1.09	0.85	3.51	.04	337.20	.00	396.64	.00	381.33
12	2.13	−1.73	−0.70	0.20	1.54	.16	315.79	.02	346.38	.20	312.41
13	2.71	−0.94	−0.05	0.74	1.74	.13	318.94	.12	320.91	.47	293.24
14	3.17	−0.88	0.10	0.87	1.78	.01	353.65	.07	329.37	.20	311.88
15	1.46	−2.42	−0.93	0.06	1.51	.16	316.03	.00	365.01	.05	333.25
16	1.89	−1.74	−0.67	0.33	1.54	.00	367.06	.01	351.96	.20	311.69
17	0.99	−3.40	−1.69	−0.49	1.09	.01	357.61	.09	325.57	.48	292.68
18	2.67	−0.82	0.03	0.64	1.33	.41	296.98	.18	314.05	.44	295.13
19	2.91	−1.08	−0.27	0.42	1.52	.34	301.68	.18	313.98	.59	285.71
20	2.04	−1.64	−0.61	0.21	1.48	.52	290.06	.02	344.84	.22	310.18

The $S - X^{2}$ –test flags six items, the $LI$ –test 11 items, and the $S - R^{2}$ –test two items as misfitting at $α = .05$ . Two items, Item 6 and Item 11, are flagged by all tests. The p values of the $S - X^{2}$ –test were moderately correlated to the p values of the $LI$ –test and the $S - R^{2}$ –test (Kendall $τ$ : $τ \approx 0.33$ ). The p values of $LI$ –test and the $S - R^{2}$ –test were more similar (Kendall $τ$ : $τ \approx 0.73$ ). This suggests that the two tests agree which items are relatively more suspicious. We evaluated the fit of Item 11 that had the highest Q value further. We performed local tests of item fit on basis of the $S - X^{2}$ –residuals. Findings, however, were not clear. Wald tests on whether the expected residual is zero were significant for some combinations of test score and response. There was, however, no interpretable pattern. This is not really surprising as some forms of misfit have heterogeneous effects on the category characteristic curves; see Figure C1 and Figure C2 in Online Appendix C. We finally determined the augmented residual plot for Item 11; see Figure 6. The plot suggests that there is a form of variance heterogeneity. The variance seems to be too small in average test scores.

Figure 6.

Plot of augmented residuals U (left side) and squared augmented residuals U ² (right side) against the test score s in Item 11 of the Fear of Negative Evaluation Scale. The red line visualizes an estimate of the conditional expectation as a function of the test score that was determined with a loess smoother, and the black line visualizes the expected value under correct model specification.

In addition to Item 11, the tests indicated that Item 6 and, to a lesser degree, Items 3, 7, and 15 show statistically significant deviations from the underlying model. When analyzing the items’ content, there was no overlap between all items, in the wording or meaning, that would clearly explain why those items were flagged. However, three of the five flagged items (i.e., 7, 11, and 15) are worded negatively. Evidence that reversed items produce ambiguous results by eliciting changes in response behaviors, acquiescence or misunderstandings of negations (e.g., Lindwall et al., 2012; Roszkowski & Soven, 2010; Weijters et al., 2013), and prior findings on the FNES negative items (Carleton et al., 2007; Rodebaugh et al., 2004) might contribute to understand why the tests flag negatively worded items as deviating from the GRM model. One also might speculate that the flagged items were not adequate for the nonclinical sample we used. The latter would indicate that researchers should be cautious when using the FNES for comparing clinical and nonclinical samples in studies.

7. Discussion

Identifying items that undermine the fit of the data to the employed item response model is of great importance for the development and validation of psychological tests. One of the most popular tests of item fit is the $S - X^{2}$ –test that was proposed by Orlando and Thissen (2000) and extended to polytomous item response models by Kang and Chen (2008). This test compares the observed frequencies of responses with the corresponding model-based expectations in strata defined by the test scores. Numerous findings suggest that the $S - X^{2}$ –test works very well in tests with binary items (Orlando & Thissen, 2003; Sinharay & Ying, 2007) but is liberal in tests with graded response format (Kang & Chen, 2008; LaHuis et al., 2011). In this article, we have expanded the previous findings on the $S - X^{2}$ –test in several ways. We compared the $S - X^{2}$ –test to two alternative variants, we derived the distribution of the associated residuals, and we proposed a graphical check of item fit that can be used as a supplement to the statistical tests.

As the $S - X^{2}$ –test does not have the same mathematical foundation as, for example, the limited information tests (e.g., Haberman & Sinharay, 2013; Maydeu-Olivares & Joe, 2005), we considered two alternative tests, the LI–test and the $S - R^{2}$ –test. The two alternative tests differ from the $S - X^{2}$ –test in several aspects. The first alternative, the LI–test, is based on the joint distribution of the test score and the response in an item. As it is based on limited information statistics, its asymptotic distribution can be derived within the limited information framework developed by Maydeu-Olivares and Joe (2005). The $LI$ –test differs from the $S - X^{2}$ –test in the way the model-based expectations are determined. In the $S - X^{2}$ –test, the sum of the model-based expectations $\hat{e} (x, s)$ over the responses x is enforced to be identical to the frequency $o (s)$ of the test score in the data. This is not the case in the $LI$ –test where the expected frequencies are not restricted. The $LI$ –test, therefore, taps two aspects, the congruence of the theoretical and the empirical distribution of the test score and the congruence of the theoretical and empirical distribution of the response in an item when conditioning on the test score. This small modification reduces the specificity of the test. The second test, the $S - R^{2}$ –test, is based on the same quantities than the $S - X^{2}$ –test. It differs from the $S - X^{2}$ –test in the way the test statistic is implemented and in the statistical model. When deriving the sample distribution of the test statistic, we do not condition on the test score. The test score’s random fluctuation from one sample to the other is explicitly considered. Contrary to the $S - X^{2}$ –test, the $S - R^{2}$ –test controls the Type-I error rate well even when samples are small and tests consist of up to $20$ items.

As a by-product, we derived the distribution of the $S - X^{2}$ –residuals. This result can be used for post hoc checks in order to locate the source of misfit. One can also test for a linear or quadratic trend by using polynomial contrasts. Whether such more specific tests can be used in order to increase the power to detect misspecification requires future research. We finally suggested a graphical check of item fit that is intended to supplement the statistical tests. The graphical check of item fit is similar to the residual plots used in regression analysis. The plot visualizes how the data deviates from the model.

Supplemental Material

Supplemental Material, sj-docx-1-jeb-10.3102_10769986211050304 - On the Generalized $S - X^{2}$ –Test of Item Fit: Some Variants, Residuals, and a Graphical Visualization

Supplemental Material, sj-docx-1-jeb-10.3102_10769986211050304 for On the Generalized $S - X^{2}$ –Test of Item Fit: Some Variants, Residuals, and a Graphical Visualization by Jochen Ranger and Kay Brauer in Journal of Educational and Behavioral Statistics

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Kay Brauer

References

Agresti

(1990). Categorical data analysis. Wiley.

American Psychological Association. (2014). Standards for educational and psychological testing. AERA Publications.

Baker

Kim

S.-H.

(2004). Item response theory: Parameter estimation techniques. Marcel Dekker. https://doi.org/10.1201/9781482276725

Bock

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29–51. https://doi.org/10.1007/BF02291411

Carleton

Collimore

Asmundson

(2007). Social anxiety and fear of negative evaluation: Construct validity of the BFNE-II. Journal of Anxiety Disorders, 21, 131–141. https://doi.org/10.1016/j.janxdis.2006.03.010

Chalmers

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 1–29. https://doi.org/10.18637/jss.v048.i06

Douglas

Cohen

(2001). Nonparametric item response function estimation for assessing parametric model fit. Applied Psychological Measurement, 25, 234–243. https://doi.org/10.1177/01466210122032046

Glas

(2016). Frequentist model-fit tests. In van der Linden

(Ed.), Handbook of item response theory (Vols. 2, Statistical Tools, pp. 343–361). Chapman and Hall/CRC Press.

Haberman

Sinharay

(2013). Generalized residuals for general models for contingency tables with application to item response theory. Journal of the American Statistical Association, 108, 1436–1444. https://doi.org/10.1080/01621459.2013.835660

10.

Haberman

Sinharay

Chon

(2013). Assessing item fit for unidimensional item response theory models using residuals from estimated item response functions. Psychometrika, 78, 417–440. https://doi.org/10.1007/s11336-012-9305-1

11.

Kang

Chen

(2008). Performance of the generalized S-X 2 item fit index for polytomous IRT models. Journal of Educational Measurement, 45, 391–406. https://doi.org/10.1111/j.1745-3984.2008.00071.x

12.

LaHuis

Clark

O’Brien

. (2011). An examination of item response theory item fit indices for the graded response model. Organizational Research Methods, 14, 421–434. https://doi.org/10.1177/1094428109350930

13.

Lindwall

Barkoukis

Grano

Lucidi

Raudsepp

Liukkonen

Thogersen-Ntoumani

(2012). Method effects: The problem with negatively versus positively keyed items. Journal of Personality Assessment, 94, 196–204. https://doi.org/10.1080/00223891.2011.645936

14.

Liu

Zhang

(2018). Residuals and diagnostics for ordinal regression models: A surrogate approach. Journal of the American Statistical Association, 113, 845–854. https://doi.org/10.1080/01621459.2017.1292915

15.

Mavridis

Moustaki

Knott

(2007). Goodness-of-fit measures for latent variable models for binary data. In Lee

S.-Y.

(Ed.), Handbook of latent variable and related models (pp. 135–161). Elsevier.

16.

Maydeu-Olivares

(2013). Goodness-of-fit assessment of item response theory models. Measurement, 11, 71–101. https://doi.org/10.1080/15366367.2013.831680

17.

Maydeu-Olivares

Joe

(2005). Limited- and full-information estimation and goodness-of-fit testing in 2 n contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009–1020. https://doi.org/10.1198/016214504000002069

18.

McKinley

Mills

(1985). A comparison of several goodness-of-fit statistics. Applied Psychological Measurement, 9, 49–57. https://doi.org/10.1177/014662168500900105

19.

Meijer

Baneke

(2004). Analyzing psychopathology items: A case for nonparametric item response theory modeling. Psychological Methods, 9, 354–368. https://doi.org/10.1037/1082-989X.9.3.354

20.

Moore

(1978). Chi-square tests. In Hogg

(Ed.), Studies in statistics (pp. 66–106). Mathematical Association of America.

21.

Orlando

Thissen

(2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50–64. https://doi.org/10.1177/01466216000241003

22.

Orlando

Thissen

(2003). Further investigation of the performance of s-x2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289–298. https://doi.org/10.1177/0146621603027004004

23.

Ranger

Much

(2020). Analyzing the fit of IRT models with the Hausman test. Frontiers in Psychology, 11, 149. https://doi.org/10.3389/fpsyg.2020.00149

24.

R Development Core Team. (2009). R: A language and environment for statistical computing [Computer software manual] . http://www.R-project.org

25.

Rizopoulos

(2006). ltm: An R package for latent variable modeling and item response theory analysis. Journal of Statistical Software, 17, 5. https://doi.org/10.18637/jss.v017.i05

26.

Rodebaugh

Woods

Thissen

Heimberg

Chambless

Rapee

(2004). More information from fewer questions: The factor structure and item properties of the original and brief fear of negative evaluation scale. Psychological Assessment, 16, 169–181. https://doi.org/10.1037/1040-3590.16.2.169

27.

Roszkowski

Soven

(2010). Shifting gears: Consequences of including two negatively worded items in the middle of a positively worded questionnaire. Assessment & Evaluation in Higher Education, 35, 113–130. https://doi.org/10.1080/02602930802618344

28.

Samejima

(1997). Graded response model. In van der Linden

Hambleton

(Eds.), Handbook of modern item response theory (pp. 237–266). Springer. https://doi.org/10.1007/978-1-4757-2691-6

29.

Sijtsma

(1998). Methodological review: Nonparametric IRT approaches to the analysis of dichtomous item scores. Applied Psychological Measurement, 22, 3–31. https://doi.org/10.1177/01466216980221001

30.

Sinharay

(2006). Bayesian item fit analysis for unidimensional item response theory models. British Journal of Mathematical and Statistical Psychology, 59, 429–449. https://doi.org/10.1348/000711005X66888

31.

Sinharay

Ying

(2007). The correlation between item parameters and item fit statistics. ETS Research Report Series, 2007, 1–19. https://doi.org/10.1002/j.2333-8504.2007.tb02078.x

32.

Stark

Chernyshenko

Drasgow

Williams

(2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring? Journal of Applied Psychology, 91, 25–39. https://doi.org/10.1037/0021-9010.91.1.25

33.

Stone

(2003). Empirical power and type I error rates for an IRT fit statistic that considers the precision of ability estimates. Educational and Psychological Measurement, 63, 566–583. https://doi.org/10.1177/0013164402251034

34.

Stroud

(1971). Approximate calculation of multiple integrals. Prentice-Hall.

35.

Swaminathan

Hambleton

R. K.

Rogers

H. J.

(2006). Assessing the fit of item response theory models. In Rao

Sinharay

(Eds.), Handbook of statistics: Vol. 26. Psychometrics (pp. 683–718). Elsevier.

36.

Takane

de Leeuw

(1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 53, 393–408. https://doi.org/https://doi.org/10.1007/BF02294363

37.

Vormbrock

Neuser

(1983). Konstruktion zweier spezifischer Trait-Fragebogen zur Erfassung von Angst in sozialen Situationen (SANB und SVSS). [Construction of two specific trait questionnaires to assess anxiety in social situations (SANB and SVSS)]. Diagnostica, 29, 165–182.

38.

Watson

Friend

(1969). Measurement of social-evaluative anxiety. Journal of Consulting and Clinical Psychology, 33, 448–457. https://doi.org/10.1037/h0027806

39.

Weijters

Baumgartner

Schillewaert

(2013). Reversed item bias: An integrative model. Psychological Methods, 18, 320–334. https://doi.org/10.1037/a0032121

40.

Welch

(1938). The significance of the difference between two means when the population variances are unequal. Biometrika, 29, 350–362. https://doi.org/10.2307/2332010

41.

White

(1982). Maximum likelihood estimation of misspecified models. Econometrica, 50, 1–25. https://doi.org/10.2307/1912526

42.

Yen

(1981). Using simulation results to choose a latent trait model. Applied Psychological Measurement, 5, 245–262. https://doi.org/10.1177/014662168100500212

43.

Yuan

K.-H.

Bentler

(2010). Two simple approximations to the distributions of quadratic forms. British Journal of Mathematical and Statistical Psychology, 63, 273–291. https://doi.org/10.1348/000711009X449771.

44.

Zhang

Stone

(2008). Evaluating item fit for multidimensional item response models. Educational and Psychological Measurement, 68, 181–196. https://doi.org/10.1177/0013164407301547

45.

Zhang

Wang

Tao

(2018). Assessing item-level fit for higher order item response theory models. Applied Psychological Measurement, 42, 644–659. https://doi.org/10.1177/0146621618762740

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.25 MB

On the Generalized S − X 2 –Test of Item Fit: Some Variants,Residuals,and a Graphical Visualization

Abstract

Keywords

1. The Logistic Model for Graded Response Data

2. The Generalized S − X 2 –Test of Item Fit

3. Alternative Implementations of the Test and the S − X 2 –Residuals

3.1. LI Test

3.2. S − R 2 –Test

4. A Graphical Check of Item Fit

5. Simulation Study

5.1. Simulation Scenario I

5.2. Simulation Scenario II

5.2.1. Condition A

5.2.2. Condition B

6. Empirical Example

7. Discussion

Supplemental Material

Supplemental Material, sj-docx-1-jeb-10.3102_10769986211050304 - On the Generalized S − X 2 –Test of Item Fit: Some Variants, Residuals, and a Graphical Visualization

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iD

References

Supplementary Material

2. The Generalized $S - X^{2}$ –Test of Item Fit

3. Alternative Implementations of the Test and the $S - X^{2}$ –Residuals

3.2. $S - R^{2}$ –Test

Supplemental Material, sj-docx-1-jeb-10.3102_10769986211050304 - On the Generalized $S - X^{2}$ –Test of Item Fit: Some Variants, Residuals, and a Graphical Visualization