Testing the Local Independence Assumption of the Rasch Model With Q 3 -Based Nonparametric Model Tests

Abstract

Local independence is a central assumption of commonly used item response theory models. Violations of this assumption are usually tested using test statistics based on item pairs. This study presents two quasi-exact tests based on the $Q_{3}$ statistic for testing the hypothesis of local independence in the Rasch model. The proposed tests do not require the estimation of item parameters and can also be applied to small data sets. The authors evaluate the tests with three simulation studies. Their results indicate that the quasi-exact tests hold their alpha level under the Rasch model and have higher power against different forms of local dependence than several alternative parametric and nonparametric model tests for local independence.

Keywords

Rasch model local independence IRT model fit

Introduction

Many commonly used item response theory (IRT) models, such as the Rasch model (Rasch, 1960), but also the more general two- and three-parametric logistic (2PL and 3PL) models (Birnbaum, 1968) assume that the probability of a positive response of a person to an item does not depend on this person’s response to any other item, conditional on the person’s ability. This assumption is generally known as the local independence assumption (De Ayala, 2009).

Violations of this assumption can be observed if learning effects, testlet effects (Wainer, Bradlow, & Wang, 2007), multidimensionality, and similar phenomena are not accounted for by the IRT model that is used to describe the data (Levy, Mislevy, & Sinharay, 2009). In practical evaluations of psychological tests, the assumption of local independence is usually tested for pairs of items (Kim, De Ayala, Ferdous, & Nering, 2011). For this purpose, numerous approaches have been described in the literature and evaluated in several simulation studies (Chen & Thissen, 1997; Edwards, Houts, & Cai, 2018; Kim et al., 2011; Yen, 1984), often testing the local independence assumption in the 2PL and 3PL models. For the special case of the Rasch model, practical suggestions have been presented by Koller and Hatzinger (2013).

This article aims at evaluating two new quasi-exact nonparametric methods for testing the local independence assumption for the Rasch model, which can also be applied to small data sets. The proposed methods can be seen as quasi-exact variations of the $Q_{3}$ statistic (Yen, 1984). The rest of the article is structured as follows: In the next two sections, several parametric methods for testing local dependence and a framework of quasi-exact tests for the Rasch model are reviewed. In the “Simulation Studies” section, these methods are compared by the means of three simulation studies. The “Empirical Data Analyses” section illustrates the application of the new tests in an empirical data set. Finally, the practical implications of the results are discussed in the “Discussion” section.

Parametric Model Tests for Testing Local Independence in the Rasch Model

Numerous statistics have already been proposed to detect violations of local independence in the Rasch model. Among them, one may discern statistics that aim at detecting model violations on the level of the item set from statistics that aim to detect violations for specific item groups or item pairs. Test statistics that aim at detecting violations of local independence on the level of the item set include the $Q_{2}$ (van den Wollenberg, 1982) and $R_{2}$ (Glas, 1988) statistics. These statistics are not the focus of this article, but evaluations of these tests were provided by Suárez-Falcón and Glas (2003) and Maydeu-Olivares and Montaño (2013).

Instead, this study focuses on statistics that are calculated for item pairs. Examples include the $Q_{3}$ statistic (Yen, 1984), the $X^{2}$ and $G^{2}$ statistics of Chen and Thissen (1997) and the Power-Divergence Statistics of Cressie and Read (1984). There are also several approaches developed for more general IRT models, such as the item-level jackknife resampling procedure of Edwards et al. (2018), the $R_{2}$ statistic by Liu and Maydeu-Olivares (2013), the $z$ statistic by Maydeu-Olivares and Liu (2015), the score test proposed by Glas and Suárez-Falcón (2003), and methods that are not based on IRT models, such as the approaches based on logistic regression reviewed by Kim et al. (2011) and an approach based on the Mantel–Haenszel test (Ip, 2001). Alternative approaches provide IRT models that allow the modeling of local dependence as part of the parameter estimation. An important class of models of this type are testlet models (Wainer et al., 2007; Wang & Wilson, 2005).

Recent overviews on tests for the detection of local dependence were provided by Kim et al. (2011) or Edwards et al. (2018). Of these procedures, the authors will describe the $Q_{3}$ statistic of Yen (1984) and the $X^{2}$ and $G^{2}$ statistics of Chen and Thissen (1997) in this section. For $Q_{3}$ , they further describe a method proposed by Christensen, Makransky, and Horton (2017) to calculate critical values.

Chen and Thissen (1997) proposed two asymptotically equivalent statistics that allow testing the hypothesis that the distribution of the responses observed for each item pair does not differ significantly from that expected under a specific IRT model. Using the notation of Kim et al. (2011), let $O_{pq}$ denote the observed frequency of response $p$ in item $i$ and of response $q$ in item $j$ , and $E_{pq}$ the corresponding expected frequency, then the two statistics $X^{2}$ and $G^{2}$ are given as follows:

X^{2} = \sum_{p = 0}^{1} \sum_{q = 0}^{1} \frac{{(O_{pq} - E_{pq})}^{2}}{E_{pq}}

and

G^{2} = - 2 \sum_{p = 0}^{1} \sum_{q = 0}^{1} O_{pq} \log (\frac{E_{pq}}{O_{pq}}) .

Both of these statistics can be assumed to follow a $χ^{2}$ distribution for sufficiently large samples, with larger values indicating a violation of the assumption of local independence. If this assumption is not violated, a uniform distribution of p values between 0 and 1 can be expected for each item pair. A relationship between two items that is stronger or weaker than predicted by the Rasch model leads to small p values for these statistics. These statistics were evaluated in a series of simulation studies that showed satisfactory results (Chen & Thissen, 1997; Kim et al., 2011).

Another widely applied method for detecting local dependence is the $Q_{3}$ statistic of Yen (1984). In its original form, this statistic is calculated for an arbitrary item pair $(i, j)$ in three steps: In a first step, all item and person parameters are calculated. In a second step, the probability of a positive response $P_{vi}$ and $P_{vj}$ to items $i$ and $j$ is determined for each respondent $v$ based on the model parameters. As a third step, the authors determine the residuals $d_{vi} = X_{vi} - P_{vi}$ and $d_{vj} = X_{vj} - P_{vj}$ for both items and each respondent $v$ based on the observed responses X_vi and X_vj. The statistic $Q_{3}$ for this item pair is calculated as the correlation between the residuals for both items over the sample of respondents.

The practical application of the $Q_{3}$ statistic poses several challenges, with the most important being that its distribution under local independence is usually unknown (Chen & Thissen, 1997). A possible solution to this problem was recently proposed by Christensen et al. (2017). Their general idea is a three-step procedure. As a first step, all item and person parameters are estimated. Based on these estimates, a large number of artificial response matrices (e.g., 10,000) are simulated based on the chosen IRT model (which is the Rasch model in this case) in a second step, and the $Q_{3}$ statistics are calculated for each response matrix. By determining the maximum values of $Q_{3}$ in each matrix, one obtains a reference distribution for the maximum of $Q_{3}$ under the null model. In a third step, the observed values for $Q_{3}$ in the original response matrix are compared with specific values of the reference distribution (e.g., the 95th or 99th percentile) to detect unusually large values of $Q_{3}$ . These authors further investigated this approach for an adjusted $Q_{3}$ statistic that is calculated by subtracting the mean of all $Q_{3}$ values from $Q_{3}$ . In this paper, the authors will refer to this statistic by $Q_{3}^{*}$ . Overall, this approach leads to critical values for the $Q_{3}$ statistic, is very flexible, and can be easily adapted to test the fit of item subsets and even single item pairs. However, it is computationally intensive and also based on the point estimates of the item and person parameters, which can be inaccurate in small data sets. Another important aspect follows from that the 95th percentile of the maximum value of the $Q_{3}$ statistic is used as critical value; as a consequence, most response matrices generated under the true model will not contain any item pair with a $Q_{3}$ value that is considered as critical, making this approach conservative. This article aims at presenting a related nonparametric approach for detecting local dependence that will be outlined below.

Quasi-Exact Model Tests for the Rasch Model

Ponocny (2001) described a framework for nonparametric, quasi-exact tests for testing the Rasch model in small samples. This framework, which does not require the estimation of item or person parameters, was further evaluated by Koller and colleagues in several simulation studies (Koller & Hatzinger, 2013; Koller, Maier, & Hatzinger, 2015; Koller, Wiedermann, & Glück, 2015).

The general idea of this framework is to test the Rasch model against a more general IRT model, which can be described as an exponential family. A distinct feature of the proposed model tests is that they are uniformly most powerful tests for the Rasch model against these alternative models (Ponocny, 2001). The test consists in comparing the observed value of a statistic, whose form can be deduced from the IRT model against which the Rasch model is tested, with its distribution in a bootstrap sample of data matrices with equal marginal sums (i.e., the sum of positive responses in every row and every column in the observed data set). This comparison leads to the calculation of p values. A p value near 0 or 1 indicates a violation of the Rasch model.

To obtain the bootstrap sample of response matrices, a Markov-Chain Monte Carlo (MCMC) algorithm developed by Verhelst (2008) is used. This algorithm is available in the software package eRm (Mair, Hatzinger, & Maier, 2018) for the R software environment (R Core Team, 2017). This algorithm generates artificial response matrices with identical marginal sums from the original response matrix using a stepwise procedure; for a more detailed description, see Koller and Hatzinger (2013). It is important that the generated response matrices are neither too similar to the original response matrix nor to each other, as both could lead to a bias in the calculation of the p values. To this end, not all response matrices generated by the algorithm are used. By discarding response matrices at the beginning of the sequence, which is also named the burn-in phase, a high similarity with the original response matrix is avoided. After the burn-in phase, every $k$ th element of the sequence is retained for calculating the p values, and the rest is discarded. $k$ is named the step size. A large step size ensures that the artificial response matrices are dissimilar to each other.

Ponocny (2001) and Koller et al. (2015) already described several test statistics for this nonparametric framework, which may be used to detect specific violations of the Rasch model. For testing against local independence, Ponocny (2001) described the $T_{1}$ statistic, which is calculated for each item pair. For a given item pair $(i, j)$ , the $T_{1}$ statistic counts the number of respondents, which gave identical responses to both items. To test the hypothesis of local independence, it can be tested whether the test statistic is unusually high or low for response matrices generated under the Rasch model. To discern between these two possibilities, Koller et al. (2015) named the test statistic for the first possibility $T_{1}$ and for the second possibility $T_{1 m}$ , with $m$ standing for multidimensionality. In summary, the underlying statistic for testing the assumption of local independence between items $i$ and $j$ in a test given to a sample of $N$ respondents is given as follows (Koller, Alexandrowicz, & Hatzinger, 2012):

T_{1} = \sum_{v = 1}^{N} δ_{vij}, with δ_{vij} = {\begin{matrix} 1 & for X_{vi} = X_{vj} \\ 0 & otherwise \end{matrix} .

Koller and Hatzinger (2013) described another variation of the $T_{1}$ statistic to assess the presence of a learning effect. This variation, which they labeled $T_{1 l}$ , is also calculated for each item pair $(i, j)$ and counts the number of respondents that gave a positive response to both items. The resulting test statistic is thus given as follows (Koller et al., 2012):

T_{1 l} = \sum_{v = 1}^{N} δ_{vij}, with δ_{vij} = {\begin{matrix} 1 & for X_{vi} = 1 = X_{vj} \\ 0 & otherwise \end{matrix} .

Further simulation results were presented by Koller, Wiedermann & Glück (2015), who investigated quasi-exact tests in the context of change measurement.

The authors now discuss how the nonparametric testing framework of Ponocny (2001) can be used for testing the local independence assumption based on the $Q_{3}$ statistic. As was already noted, the calculation of $Q_{3}$ for an item pair $(i, j)$ is based on $P_{vi}$ and $P_{vj}$ , which are the probabilities that respondent $v$ gives a positive response on item $i$ and $j$ , respectively. In this nonparametric framework, it is possible to estimate $P_{vj}$ for any respondent-item pair $(v, j)$ by averaging $X_{vj}$ over all bootstrap samples. The calculation of the proposed quasi-exact variation of the $Q_{3}$ statistic for two items $(i, j)$ thus consists of the following steps: First, generate a sufficiently large bootstrap sample of data matrices (e.g., 500 or more matrices) with the same marginal sums as the original data. Based on this sample, estimate $P_{vi}$ and $P_{vj}$ by averaging over the bootstrap samples, as outlined above. In a second step, determine the residuals $d_{vi} = X_{vi} - P_{vi}$ and $d_{vj} = X_{vj} - P_{vj}$ for both items and each respondent $v$ , using the entries $X_{vi}$ and $X_{vj}$ of the original response matrix. Like $Q_{3}$ , the test statistic is again calculated as the correlation coefficient between the residuals for both items over the sample. As the other statistics in this framework, this statistic does not require the calculation of item or person parameters and is, therefore, nonparametric.

Comparable to the aforementioned $T_{1}$ and $T_{1 m}$ statistics, it is possible to define two variations of this nonparametric statistic. In the first variation, small p values correspond to a high correlation between the residuals, whereas in the second variation, small p values correspond to a low correlation. To discriminate between these two variations, the first one is denoted as $Q_{3 h}$ and the second one as $Q_{3 l}$ . The small letters stand for “high” and “low,” respectively, and indicate what kind of relationship between the items in an item pair is indicated by small p values: In $Q_{3 h}$ , small p values indicate an unusual high rate of identical responses between the two items, whereas the reverse is true for $Q_{3 l}$ . The resulting two statistics are also closely related to the detection of positive and negative local item dependence. If the responses between two test items over the population are more similar than predicted by an IRT model, this is named positive local item dependence, whereas the contrary is named negative local item dependence (Habing & Roussos, 2003). Therefore, positive local independence leads to small p values in $Q_{3 h}$ , whereas negative local item dependence leads to small p values in $Q_{3 l}$ .

Aim of This Study

This study aimed at comparing several methods for detecting violations of local independence with regard to their Type I error rate and their power against local dependence. Based on the review of the literature, the authors chose to evaluate the following statistics besides $Q_{3 l}$ and $Q_{3 h}$ : $T_{1}$ , $T_{1 l}$ , $T_{1 m}$ , $X^{2}$ , and $G^{2}$ . The study further included $Q_{3}$ and $Q_{3}^{*}$ , for which critical values were calculated using the parametric bootstrap approach of Christensen et al. (2017).

Previous work on tests for detecting local dependence considered at least two violations of this assumption. The first violation occurs when more than one latent ability is necessary to describe the observed person–item interactions (i.e., the items measure more than one trait) and includes multidimensionality. The second type can be expected if specific item pairs are unusually similar with regard to their content (e.g., solving the first item helps in solving the second one). Both types of model violations are closely related to the concepts of trait and response dependence, as they were proposed by Andrich and Kreiner (2010) for the Rasch model.

All test statistics were evaluated with regard to their power against these two forms of local dependence. In the following section, the design of the simulation studies used for this evaluation is discussed.

Simulation Studies

Overall, three simulation studies were carried out, of which the first aimed at evaluating the Type I error rate, the second one aimed at testing the sensitivity against response dependence, and the third one aimed at evaluating the sensitivity against testlet effects.

General Settings

The three simulation studies shared the following general characteristics:

The number of replications: Under all conditions, 1,000 data sets were simulated.

The number of respondents: In all simulation studies, data sets with 100, 150, 200, 500, and 1,000 respondents were simulated. These numbers resemble the sample sizes typically used for the evaluation of psychological tests and scales.

The number of items: The authors simulated test lengths of 10, 20, 30, and 40 items.

The person parameter distribution: The person parameters were drawn from a standard normal distribution. They were drawn anew for each simulated dataset.

The item parameter distribution: The item parameters were fixed for all items over all generated datasets and drawn from a standard normal distribution.

Settings of the bootstrap algorithm for the nonparametric tests: For the nonparametric tests, the authors used the R package eRm (Mair et al., 2018) to generate the bootstrap samples using the MCMC algorithm of Verhelst (2008). The settings of the bootstrap algorithm were the default settings of this software package: Of the generated response matrices, the first 3,200 were discarded as a burn-in phase. After the burn-in phase, every 16th generated response matrix was used for the calculation of p values. Overall, the p values of all nonparametric tests based on $Q_{3 h}, Q_{3 l}, T_{1}, T_{1 l}$ , and $T_{1 m}$ were calculated based on 500 generated response matrices.

Implementation of the parametric bootstrap algorithm of Christensen et al. (2017): This algorithm was carried out in three steps: For each simulated data set, the item and person parameters were estimated in a first step; for the estimation of the item parameters, a marginal maximum likelihood (MML) algorithm was used, whereas the person parameters were estimated using a weighted maximum likelihood algorithm (Warm, 1989). In a second step, 1,000 data sets were generated based on the Rasch model, using the parameter estimates obtained in the first step as true parameter values. For each of these data sets, the maximum of the $Q_{3}$ and $Q_{3}^{*}$ matrices were calculated. Therefore, a reference distribution for each generated data set was obtained. In a third step, $Q_{3}$ and $Q_{3}^{*}$ were calculated for the original response matrix and the rate of values in the respective reference distribution that was larger than the observed value was obtained. It should be noted that the use of the 95th percentile of the reference distribution to label values of $Q_{3}$ or $Q_{3}^{*}$ as critical, as it was suggested by Christensen et al. (2017), is analogous to labeling values as critical where this rate is below 0.05. It was found that both statistics led to very similar results. To keep the presentation of results concise, only the results for $Q_{3}^{*}$ are presented and detailed results for $Q_{3}$ are omitted.

Simulation Study I: Type I Error Rate

Aim of Simulation Study I

Simulation Study I aimed at investigating the Type I error rate of the various tests. Tests with a high Type I error rate typically also have higher power against model violations, therefore, the results of this investigation also affect the interpretation of the remaining simulation studies.

Design of Simulation Study I

In this simulation study, all data were generated according to the Rasch model. As it is commonly known, this model uses the following item response function for describing the probability of a response of $x_{vj}$ of respondent $v$ with person parameter $θ_{v}$ to an item $j$ with difficulty parameter $β_{j}$ :

P (X_{vj} = x_{vj} | β_{j}, θ_{v}) = \frac{\exp (x_{vj} (θ_{v} - β_{j}))}{1 + \exp (θ_{v} - β_{j})} .

(1)

For all test statistics, the rate of p values lower than 0.01 and 0.05 for all item pairs were evaluated.

Results and conclusions of Simulation Study I

Overall, all tests were found to have Type I error rates at or below the nominal alpha level. For interpreting the sensitivity of statistical tests against various model violations, it is important to state that the Type I error rate is not inflated. Under this perspective, the results were satisfactory. For the $X^{2}$ and $G^{2}$ statistics, results agree with findings in the literature (Chen & Thissen, 1997; Kim et al., 2011). For the parametric bootstrap approach of Christensen et al. (2017), the authors found that $Q_{3}$ and $Q_{3}^{*}$ are almost always below their respective critical values, making this approach comparatively conservative. Detailed results are presented in the Online Appendix for this article.

Simulation Study II: Response Dependence

Aim of Simulation Study II

Simulation Study II aimed at evaluating the power of the test statistics to detect violations of local dependence on the level of single item pairs. This type of local dependence can be commonly observed in the presence of learning effects, similar item content, and similar causes of model violations.

Design of Simulation Study II

This study simulated a violation of local independence by generating items with response dependence (Marais & Andrich, 2008). Similar designs were used in a number of studies investigating local dependence in the Rasch model (Andrich & Kreiner, 2010; Marais & Andrich, 2008). In the data generating model, there are item pairs that violate the assumption of local independence. Let $(i, j)$ be one of these item pairs, then the probability of a correct response to item $j$ increases or decreases in dependence of the response to item $i$ (Andrich & Kreiner, 2010; Marais & Andrich, 2008):

P (X_{vj} = x_{vj} | θ_{v}, β_{j}) = \frac{\exp (x_{vj} (θ_{v} - β_{j} - (1 - 2 x_{vi}) d))}{1 + \exp (θ_{v} - β_{j} - (1 - 2 x_{vi}) d)} .

(2)

This model can be compared with a Rasch model in which the item parameter of item $j$ depends on the response that was given to another item $i$ . If item $i$ was correctly solved by respondent $v$ , that is, $x_{vi} = 1$ , the term $θ_{v} - β_{j} - (1 - 2 x_{vi}) d$ in the item response function becomes $θ_{v} - β_{j} + d$ . Therefore, the item parameter of item $j$ is decreased by $d$ in this case, making the item easier. If item $i$ was worked on incorrectly by person $v$ , the same term becomes $θ_{v} - β_{j} - d$ , and the item parameter is increased for item $j$ , making the item more difficult. For item pairs for which the assumption of local independence was not violated, the data generating model was again the Rasch model.

Depending on the simulated condition, $d$ was set to 0.3 or 0.5.

There were two additional conditions in this simulation study, which differed in the number of item pairs for which local independence was violated. Depending on the simulated condition, the violation of the item parameters affected only one item pair $(1, 2)$ or three item pairs $(1, 2), (3, 4)$ , and $(5, 6)$ . These conditions were combined with the different values for $d$ .

Results and conclusions of Simulation Study II

This type of model violation leads to a higher correlation between specific item pairs than it is predicted by the Rasch model. Therefore, it is of central interest whether statistics such as $Q_{3 h}$ , $T_{1}$ , and $T_{1 l}$ are sensitive against this model violation. For brevity, the authors only present the main findings here and report detailed results in the Online Appendix.

Overall, the nonparametric test based on $Q_{3 h}$ has the highest power against this model violation, whereas other model tests such as $T_{1}$ , $T_{1 l}$ , $G^{2}$ , and $X^{2}$ are less powerful. For tests such as $Q_{3 l}$ and $T_{1 m}$ , the authors observed a smaller rate of p values below 0.05 in item pairs with response dependence than they had observed under the Rasch model (Simulation Study I). Given that these tests aim at detecting item pairs for which the responses are more dissimilar than expected under the Rasch model, this finding is not surprising. In item pairs for which no model violation was simulated, all tests tend to have a rate of significant test results that is close to the nominal alpha level.

The authors conclude this section with results on the parametric bootstrap approach. $Q_{3}^{*}$ was less sensitive against response dependence than the other statistics reported here. The results obtained for $Q_{3}$ were comparable to those of $Q_{3}^{*}$ . This finding is in line with the small rate of critical $Q_{3}$ and $Q_{3}^{*}$ statistics that was found in Simulation Study I. However, the differences in the underlying rationale and in the Type I error rate between this approach and the other tests make a comparison difficult.

Simulation Study III: Rasch Testlet Model

Aim of Simulation Study III

As was already stated, the test statistics evaluated in this article aim at detecting local dependence on the level of item pairs. The presence of multidimensionality affects all item pairs to some degree, and, therefore, can be regarded as a model violation that these statistics were originally not aimed to detect. Such model violations are fairly common and include cases where groups of items measure an additional trait, as it is also modeled in testlet models (Wainer et al., 2007). Therefore, the authors chose to investigate the power of all tests also under a testlet model in Simulation Study III.

It seems important to note that multidimensionality was found to lead to a pattern of positive and negative local dependence in previous studies. A heuristic explanation for this observation was provided by Yen (1984), whereas a formal proof for the special case of normally distributed person ability parameters in multidimensional compensatory IRT models was provided by Habing and Roussos (2003). Therefore, the authors expected all test statistics to be sensitive against this model violation. Comparable simulation designs, which were based on multidimensional IRT models, were used in the studies of Edwards et al. (2018), Kim et al. (2011), and others for evaluating tests of local independence.

Design of Simulation Study III

This simulation study evaluated the power of the tests for local dependence in data sets generated from the Rasch testlet model (Wang & Wilson, 2005). The response function of this model is given by the following:

P (X_{vi} = x_{vi} | β_{i}, θ_{v}, γ_{vd (i)}) = \frac{\exp (x_{vi} (θ_{v} - β_{i} + γ_{vd (i)}))}{1 + \exp (θ_{v} - β_{i} + γ_{vd (i)})} .

As can be seen from a comparison with Equation 1, this model contains an additional parameter $γ_{vd (i)}$ , which is specific for each respondent $v$ and a set of items (or testlet) $d (i)$ . Setting this parameter to 0 results in the Rasch model. Under the Rasch testlet model, all items measure the trait $θ_{v}$ , whereas all responses to items in the testlet $d (i)$ are further affected by a person-specific random effect $γ_{vd (i)}$ , which leads to a form of multidimensionality. The resulting model is conceptually very similar to the trait dependence model (Marais & Andrich, 2008), which also defines a general ability parameter and additional ability parameters which are specific to certain item groups.

The data sets simulated in Simulation Study III contained five item testlets of equal size, which each contained one fifth of the overall item set. The corresponding five random effects $γ_{vd (i)}$ were drawn from a normal distribution $N (0, σ_{γ_{v (i)}}^{2})$ . As in Simulation Studies I and II, the person parameters were drawn from a standard normal distribution. As a measure of the size of each testlet effect, the parameters $σ_{γ_{v (1)}}^{2}$ , $σ_{γ_{v (2)}}^{2}$ , $σ_{γ_{v (3)}}^{2}$ , $σ_{γ_{v (4)}}^{2}$ , and $σ_{γ_{v (5)}}^{2}$ were set to 0.5 or 0.75, indicating a small or moderate testlet effect (Wang & Wilson, 2005).

Results and conclusions of Simulation Study III

As this simulation study analyzed item sets that measured multiple traits, the authors could discern between item pairs in which both items measured the same latent traits from item pairs in which both items measured distinct traits. Again, the authors only present a summary here, whereas detailed results are found in the Online Appendix.

A comparison of the test results for the two different types of item pairs indicates that the p values tend to decrease in both types of item pairs for the tests based on $χ^{2}$ statistics, which are $X^{2}$ and $G^{2}$ . For the quasi-exact tests, the means of the p values differ for the two types of items. For the statistics, which aim to detect a high correlation between the two items of an item pair (which are $Q_{3 h}$ , $T_{1}$ , and $T_{1 l}$ ), the authors observed a high rate of p values below .05 for items which measure the same latent traits, but not for the remaining items. For the other nonparametric tests ( $Q_{3 l}$ , $T_{1 m}$ ), the inverse pattern was observed.

These effects can be explained by considering how these test statistics are calculated: $X^{2}$ and $G^{2}$ provide a general comparison of the observed response pattern with that expected under a (unidimensional) Rasch model, with any deviation leading to a decrease of their p values. In data sets generated from testlet models, there is an obvious discrepancy between the observed and expected response patterns, which leads to a decrease of the p values.

In statistics like $T_{1}$ , $T_{1 l}$ , and $Q_{3 h}$ , a high rate of identical responses to both items corresponds to a lower p value. For $T_{1 m}$ and $Q_{3 l}$ , the opposite is true. In multidimensional data sets, such as those generated from testlet models, this rate is increased for item pairs assessing the same latent trait, whereas it is decreased for item pairs assessing different traits, which leads to the observed pattern. Overall, $Q_{3 l}$ and $Q_{3 h}$ were most powerful against this model violation.

As was already the case for response dependence, the parametric bootstrap of Christensen et al. (2017) led to a lower rate of item pairs that were considered as critical than the other approaches. As in the previous studies, the results obtained for $Q_{3}$ were comparable with those of $Q_{3}^{*}$ .

The authors also found that the distinct behavior of the nonparametric tests in the data sets with trait dependence can be illustrated graphically. The first option are QQ plots, which compare the distribution of the p values for all item pairs against an uniform distribution. A second option are plots that present the matrix of p values as a matrix of circles, in which p values near 1 are represented by full and p values near 0 as empty circles. Similar plots were presented by Sinharay (2005) and Bechger and Maris (2015). Figure 1 provides two examples for each of these plots, which were generated using the car (Fox & Weisberg, 2011) and corrplot (Wei & Simko, 2017) packages for R. The right-hand plots were generated based on the results of the $Q_{3 l}$ statistic in a data set with 500 respondents working on 20 items from the Rasch testlet model, and the left-hand plots on the results of the same statistic from a data set of the same size generated from the Rasch model (i.e., in Simulation Study I). As can be seen, a testlet effect leads to a distinct pattern in each plot, which is not visible when no model violation is simulated.

Figure 1.

Graphical model tests (top row: graphical illustration of p values; bottom row: QQ plots) based on the patterns of p values for the $Q_{3 l}$ statistic in two simulated data sets with 500 respondents working on 20 items.

Empirical Data Analyses

In this section, the authors briefly illustrate the application of the quasi-exact $Q_{3}$ tests in an empirical data set by reporting a reanalysis of a data set used by Koller and Hatzinger (2013) and Koller and Alexandrowicz (2010) for demonstrating the quasi-exact model tests. The analyzed data set consists of the responses of $n = 221$ second- to third-grade children (second grade: n = 122, males = 51; third grade: n = 99, males = 44) working on five items of the “perceptive quantity estimation” (k = 5 items) subtest of the Neuropsychological Test Battery for Number Processing and Calculation in Children (ZAREKI-R; von Aster, Zulauf, & Horn, 2006).

In this subtest, the first two items ask the children to estimate a number of black dots, which are presented separately for 2 s. In Items 3 and 4, the numbers in two sets of three-dimensional objects, which are presented for 5 s, should be estimated. The fifth item requires a quantitative comparison of the objects presented in Items 3 and 4 (Question: “Were there more . . .?”). Based on the item content, the authors further investigate these two item pairs $(1, 2)$ and $(3, 4)$ . For each of these item pairs, the authors test the null hypothesis that the responses are accurately described by the Rasch model against the alternative hypothesis that there is a higher rate of similar responses than it is predicted by the Rasch model. Therefore, a p value based on $Q_{3 h}$ was calculated for both null hypotheses and also correct for multiple testing by a Bonferroni–Holm correction (Holm, 1979). For the null hypothesis of local independence between Items 1 and 2, the authors found a corrected p value of .002, for the null hypothesis of local independence between Items 3 and 4, they found a corrected p value below .001. In summary, these results indicate a violation of local independence. This corresponds to the results reported by Koller and Hatzinger (2013). The corresponding graphical model tests are presented in the following Figure 2.

Figure 2.

Graphical model tests (top row: graphical illustration of p values; bottom row: QQ plot) based on the patterns of p values for the $Q_{3 h}$ statistic in the empirical data example.

As a comparison with Figure 1 shows, these results may indicate a multidimensionality of the data set. The authors do not investigate this hypothesis further.

Discussion

The results indicate that the proposed tests based on $Q_{3 l}$ and $Q_{3 h}$ have higher power than the tests based on $T_{1}$ , $T_{1 l}$ , $T_{1 m}$ , $X^{2}$ , and $G^{2}$ to detect violations of local independence. If no such model violations are present in the data, the rate of significant p values is close to the nominal alpha level. All tests were compared with $Q_{3}$ and $Q_{3}^{*}$ , for which critical values were determined using the parametric bootstrap approach of Christensen et al. (2017). In contrast to the other tests considered in this study, this approach does not aim at the calculation of p values, which are uniformly distributed under the Rasch model, but leads to critical values for the $Q_{3}$ and $Q_{3}^{*}$ statistics. When compared with the other tests, the rate of item pairs that are labeled as violating the Rasch model is overall the lowest for this approach under all conditions considered here. It may, therefore, be particularly interesting for applications in which the avoidance of false-positive results is important.

Another distinct feature of $Q_{3 l}$ and $Q_{3 h}$ is that their calculation avoids the estimation of model parameters and is not based on asymptotic results, which makes them particularly interesting for small data sets. However, the authors point out again that, because of the underlying statistical framework, they can only be applied with the Rasch model for dichotomous items. They further require the absence of missing data. To allow the application of $Q_{3 l}$ and $Q_{3 h}$ in empirical research, it is planned to implement them in the eRm package (Mair et al., 2018).

There are two additional points that should be further considered in practical application of these statistics: First, like the original $Q_{3}$ statistic, $Q_{3 l}$ and $Q_{3 h}$ are calculated for item pairs. When multiple item pairs are tested for a violation of local independence, this leads to a multiple testing problem. The authors recommended to develop hypotheses which item pairs might be affected by local dependence based on the item content or previous data, and to test these hypotheses only for these item pairs. To prevent an inflated Type I error rate (i.e., an increased rate of false-positive test results), one might apply an alpha correction for these hypotheses tests, for instance, by using the Bonferroni–Holm (Holm, 1979) method. This procedure was illustrated by the empirical example study reported in the previous section. A similar suggestion was given by Koller and Hatzinger (2013). The authors do not recommend to test the local independence assumption in all item pairs simultaneously, as the resulting alpha correction would drastically reduce the power of the model tests.

A second point concerns the detection of multidimensionality. As was reported in Simulation Study III, the p values of the tests based on $Q_{3 l}$ and $Q_{3 h}$ are approximately uniformly distributed when the analyzed data set is generated from the Rasch model. Multidimensionality typically leads to a larger number of extreme p values, which can be easily detected by comparing the distribution of the observed p values with a uniform distribution $U (0, 1)$ . Although this type of model violation cannot be detected by analyzing single item pairs, graphical model tests like those presented in Figure 1 may help in the detection of this model violation.

If local dependence has been detected on the level of specific item pairs, possible causes for the model violation should be examined (Kim et al., 2011). To address local dependence, the item set may be altered (e.g., by changing or removing items), or, to avoid the loss of information, by using an IRT model that allows the modeling of local dependence. Examples for such models, which are closely related to the Rasch model are the Rasch testlet model (Wang & Wilson, 2005), the multidimensional Rasch model, and the locally dependent Rasch model (Kelderman, 2007).

Software Used in This Study

All simulation studies were carried out in the R statistical software environment (R Core Team, 2017). The test statistics $X^{2}$ and $G^{2}$ were calculated using the TAM software package (Robitzsch, Kiefer, & Wu, 2018), whereas the nonparametric tests based on $T_{1}$ , $T_{1 m}$ , and $T_{1 l}$ were calculated using the eRm package (Mair et al., 2018). The tests based on $Q_{3 h}$ and $Q_{3 l}$ and the parametric bootstrap of Christensen et al. (2017) were applied using R code that was written specifically for this study. The graphical model tests reported in Figures 1 and 2 were created using the R packages car (Fox & Weisberg, 2011) and corrplot (Wei & Simko, 2017).

Supplemental Material

Appendix_PDF – Supplemental material for Testing the Local Independence Assumption of the Rasch Model With Q3-Based Nonparametric Model Tests

Supplemental material, Appendix_PDF for Testing the Local Independence Assumption of the Rasch Model With Q3-Based Nonparametric Model Tests by Rudolf Debelak and Ingrid Koller in Applied Psychological Measurement

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Rudolf Debelak

Supplemental Material

Supplemental material for this article is available online.

References

Andrich

Kreiner

(2010). Quantifying response dependence between two dichotomous items using the Rasch model. Applied Psychological Measurement, 34, 181-192. doi:10.1177/0146621609360202

Bechger

T. M.

Maris

(2015). A statistical test for differential item pair functioning. Psychometrika, 80, 317-340. doi:10.1007/s11336-014-9408-y

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 392-479). Reading, MA: Addison-Wesley.

Chen

W.-H.

Thissen

(1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289. doi:10.2307/1165285

Christensen

K. B.

Makransky

Horton

(2017). Critical values for Yen’s Q3: Identiﬁcation of local dependence in the Rasch model using residual correlations. Applied Psychological Measurement, 41, 178-194. doi:10.1177/0146621616677520

Cressie

Read

T. R.

(1984). Multinomial goodness-of-ﬁt tests. Journal of the Royal Statistical Society: Series B (Methodological), 46, 440-464.

De Ayala

. (2009). The theory and practice of item response theory. New York, NY: Guilford Press.

Edwards

M. C.

Houts

C. R.

Cai

(2018). A diagnostic procedure to detect departures from local independence in item response theory models. Psychological Methods, 23, 138-149. doi:10.1037/met0000121

Fox

Weisberg

(2011). An R companion to applied regression (2nd ed.). Thousand Oaks CA: SAGE. Retrieved from http://socserv.socsci.mcmaster.ca/jfox/Books/Companion

10.

Glas

C. A. W.

(1988). The derivation of some tests for the Rasch model from the multinomial distribution. Psychometrika, 53, 525-546.

11.

Glas

C. A. W.

Suárez-Falcón

J. C.

(2003). A comparison of item-ﬁt statistics for the three-parameter logistic model. Applied Psychological Measurement, 27, 87-106. doi:10.1177/0146621602250530

12.

Habing

Roussos

L. A.

(2003). On the need for negative local item dependence. Psychometrika, 68, 435-451. doi:10.1007/BF02294736

13.

Holm

(1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65-70.

14.

H.-S.

(2001). Testing for local dependency in dichotomous and polytomous item response models. Psychometrika, 66, 109-132. doi:10.1007/BF02295736

15.

Kelderman

(2007). Loglinear multivariate and mixture Rasch models. In von Davier

Carstensen

C. H.

(Eds.), Multivariate and mixture distribution Rasch models (pp. 77-97). New York, NY: Springer.

16.

Kim

De Ayala

Ferdous

A. A.

Nering

M. L.

(2011). The comparative performance of conditional independence indices. Applied Psychological Measurement, 35, 447-471. doi:10.1177/0146621611407909

17.

Koller

Alexandrowicz

R. W.

(2010). Eine psychometrische Analyse der ZAREKI-R mittels Rasch-Modellen [A psychometric analysis of the ZAREKI-R by the means of Rasch models]. Diagnostica, 56, 57-67. doi:10.1026/0012-1924/a000003

18.

Koller

Alexandrowicz

R. W.

Hatzinger

(2012). Das Rasch Modell in der Praxis: Eine Einführung mit eRm [The Rasch model in practical applications: An introduction with eRm]. Vienna, Austria: facultas.wuv, UTB.

19.

Koller

Hatzinger

(2013). Nonparametric tests for the Rasch model: Explanation, development, and application of quasi-exact tests for small samples. InterStat. Retrieved from http://interstat.statjournals.net/YEAR/2013/abstracts/1311002.php

20.

Koller

Maier

M. J.

Hatzinger

(2015). An empirical power analysis of quasi-exact tests for the Rasch model: Measurement invariance in small samples. Methodology, 11, 45-54. doi:10.1027/1614-2241/a000090

21.

Koller

Wiedermann

Glück

(2015). Item response models for dependent data: Quasi-exact tests for the investigation of some preconditions for measuring change. In Stemmler

von Eye

Wiedermann

(Eds.), Dependent data in social sciences research (pp. 263-279). Cham, Switzerland: Springer.

22.

Levy

Mislevy

R. J.

Sinharay

(2009). Posterior predictive model checking for multidimensionality in item response theory. Applied Psychological Measurement, 33, 519-537. doi:10.1177/0146621608329504

23.

Liu

Maydeu-Olivares

(2013). Local dependence diagnostics in irt modeling of binary data. Educational and Psychological Measurement, 73, 254-274. doi:10.1177/0013164412453841

24.

Mair

Hatzinger

Maier

M. J.

(2018). eRm: Extended Rasch Modeling (0.16-1) [Computer software manual]. Available from https://cran.r-project.org/web/packages/eRm/index.html

25.

Marais

Andrich

(2008). Formalizing dimension and response violations of local independence in the unidimensional Rasch model. Journal of Applied Measurement, 9, 200-215.

26.

Maydeu-Olivares

Liu

(2015). Item diagnostics in multivariate discrete data. Psychological Methods, 20, 276-292. doi:10.1037/a0039015

27.

Maydeu-Olivares

Montaño

(2013). How should we assess the ﬁt of Rasch-type models? Approximating the power of goodness-of-ﬁt statistics in categorical data analysis. Psychometrika, 78, 116-133. doi:10.1007/s11336-012-9293-1

28.

Ponocny

(2001). Nonparametric goodness-of-ﬁt tests for the Rasch model. Psychometrika, 66, 437-459. doi:10.1007/BF02294444

29.

Rasch

(1960). Probabilistic models for some intelligence and achievement tests. Copenhagen, Denmark: Danish Institute for Educational Research.

30.

R Core Team. (2017). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Available from https://www.R-project.org/

31.

Robitzsch

Kiefer

(2018). TAM: Test analysis modules (R package version 2.12-18) [Computer software manual]. Available from https://CRAN.R-project.org/package=TAM

32.

Sinharay

(2005). Assessing ﬁt of unidimensional item response theory models using a Bayesian approach. Journal of Educational Measurement, 42, 375-394. doi:10.1111/j.1745-3984.2005.00021.x

33.

Suárez-Falcón

J. C.

Glas

C. A.

(2003). Evaluation of global testing procedures for item ﬁt to the Rasch model. British Journal of Mathematical and Statistical Psychology, 56, 127-143. doi:10.1348/000711003321645395

34.

van den Wollenberg

A. L

. (1982). Two new test statistics for the Rasch model. Psychometrika, 47, 123-140. doi:10.1007/BF02296270

35.

Verhelst

N. D.

(2008). An efficient MCMC algorithm to sample binary matrices with ﬁxed marginals. Psychometrika, 73, 705-728. doi:10.1007/s11336-008-9062-3

36.

von Aster

Zulauf

M. W.

Horn

. (2006). Neuropsychologische Testbatterie für Zahlenverarbeitung und Rechnen bei Kindern (ZAREKI-R) [Neuropsychological Test Battery for Number Processing and Calculation in Children]. Frankfurt, Germany: Harcourt Test Services.

37.

Wainer

Bradlow

E. T.

Wang

(2007). Testlet response theory and its applications. Cambridge, UK: Cambridge University Press.

38.

Wang

W.-C.

Wilson

(2005). The Rasch testlet model. Applied Psychological Measurement, 29, 126-149. doi:10.1177/0146621604271053

39.

Warm

T. A.

(1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427-450. doi:10.1007/BF02294627

40.

Wei

Simko

(2017). R package “corrplot”: Visualization of a correlation matrix (Version 0.84) [Computer software manual]. Retrieved from https://github.com/taiyun/corrplot

41.

Yen

W. M.

(1984). Effects of local item dependence on the ﬁt and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145. doi:10.1177/014662168400800201

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.13 MB