Rasch Mixture Models for DIF Detection

Abstract

Rasch mixture models can be a useful tool when checking the assumption of measurement invariance for a single Rasch model. They provide advantages compared to manifest differential item functioning (DIF) tests when the DIF groups are only weakly correlated with the manifest covariates available. Unlike in single Rasch models, estimation of Rasch mixture models is sensitive to the specification of the ability distribution even when the conditional maximum likelihood approach is used. It is demonstrated in a simulation study how differences in ability can influence the latent classes of a Rasch mixture model. If the aim is only DIF detection, it is not of interest to uncover such ability differences as one is only interested in a latent group structure regarding the item difficulties. To avoid any confounding effect of ability differences (or impact), a new score distribution for the Rasch mixture model is introduced here. It ensures the estimation of the Rasch mixture model to be independent of the ability distribution and thus restricts the mixture to be sensitive to latent structure in the item difficulties only. Its usefulness is demonstrated in a simulation study, and its application is illustrated in a study of verbal aggression.

Keywords

mixed Rasch model Rasch mixture model DIF detection score distribution

Introduction

Based on the Rasch model (Rasch, 1960), Rost (1990) introduced what he called the “mixed Rasch model,” a combination of a latent class approach and a latent trait approach to model qualitative and quantitative ability differences. As suggested by Rost (1990), it can also be used to examine the fit of the Rasch model and check for violations of measurement invariance such as differential item functioning (DIF). Since the model assumes latent classes for which separate Rasch models hold, it can be employed to validate a psychological test or questionnaire: if a model with two or more latent classes fits better than a model with one latent class, measurement invariance is violated and a single Rasch model is not suitable because several latent classes are present in the data that require separate Rasch models with separate sets of item difficulties. These classes are latent in the sense that they are not determined by covariates.

As the model assesses a questionnaire—or instrument as it will be referred to in the following—as a whole, it works similar to a global test like the likelihood ratio (LR) test (Andersen, 1972; Gustafsson, 1980), not an itemwise test like the Mantel–Haenszel test (Holland & Thayer, 1988). Hence, it is the set of item parameters for all items, which is tested for differences between groups rather than each item parameter being tested separately.

The mixed Rasch model—here called Rasch mixture model to avoid confusion with mixed (effects) models and instead highlight its relation to mixture models—has since been extended by Rost and von Davier (1995) to different score distributions and by Rost (1991) and von Davier and Rost (1995) to polytomous responses. The so-called “mixed ordinal Rasch model” is a mixture of partial credit models (PCM; Masters, 1982) and includes a mixture of rating scale models (RSM; Andrich, 1978) as a special case.

The original dichotomous model as well as its polytomous version have been applied in a variety of fields. Zickar, Gibby, and Robie (2004) use a mixture PCM to detect faking in personality questionnaires, while Hong and Min (2007) identify three types/classes of depressed behavior by applying a mixture RSM to a self-rating depression scale. Another vast field of application is tests in educational measurement. Baghaei and Carstensen (2013) identify different reader types from a reading comprehension test using a Rasch mixture model. Maij-de Meij, Kelderman, and van der Flier (2010) also apply a Rasch mixture model to identify latent groups in a vocabulary test. Cohen and Bolt (2005) use a Rasch mixture model to detect DIF in a mathematics placement test.

Rasch mixture models constitute a legitimate alternative to DIF tests for manifest variables such as the LR test or the recently proposed Rasch trees (Strobl, Kopf, & Zeileis, 2013). These methods are usually used to test DIF based on observed covariates, whereas Maij-de Meij et al. (2010) show that mixture models are more suitable to detect DIF if the “true source of bias” is a latent grouping variable. The simulation study by Preinerstorfer and Formann (2011) suggests that parameter recovery works reasonably well for Rasch mixture models. While they did not study in detail the influence of DIF effect size or the effect of different ability distributions, they deem such differences relevant for practical concern but leave it to further research to establish just how strongly they influence estimation accuracy.

As the Rasch model is based on two aspects, subject ability and item difficulty, Rasch mixture models are sensitive not only to differences in the item difficulties—as in DIF—but also to differences in abilities. Such differences in abilities are usually called impact and do not infringe on measurement invariance (Ackerman, 1992). In practice, when developing a psychological test, one often follows two main steps. First, the item parameters are estimated, for example, by means of the conditional maximum likelihood (CML) approach, checked for model violations and problematic items are possibly excluded or modified. Second, the final set of items is used to estimate person abilities. The main advantage of the CML approach is that, for a single Rasch model, the estimation and check of item difficulties are (conditionally) independent of the abilities and their distribution. Other global assessment methods like the LR test and the Rasch trees are also based on the CML approach to achieve such independence. However, in a Rasch mixture model, the estimation of the item difficulties is not independent of the ability distribution, even when employing the CML approach. DeMars and Lau (2011) find that a difference in mean ability between DIF groups affects the estimation of the DIF effect sizes. Similarly, other DIF detection methods are also affected by impact, for example, inflated Type I error rates occur in the Mantel–Haenszel and logistic regression procedures if impact is present (DeMars, 2010; Li, Brooks, & Johanson, 2012).

When using a Rasch mixture model for DIF detection, an influence of impact alone on the mixture is undesirable as the goal is to uncover DIF groups based on item difficulties, not impact groups based on abilities. To avoid such confounding effects of impact, we propose a new version of the Rasch mixture model specifically designed to detect DIF, which allows for the transfer of the crucial property of CML from a single Rasch model to the mixture: estimation and testing of item difficulties is independent of the abilities and their distribution.

A simulation study is conducted to illustrate how previously suggested versions and this new version of the Rasch mixture model react to impact, either alone or in combination with DIF, and how this affects the suitability of the Rasch mixture model as a DIF detection method.

In the following, we briefly discuss the Rasch model and Rasch mixture models to explain why the latter are sensitive to the specification of the score distribution despite employing a conditional maximum likelihood approach for estimation. This section is concluded with our suggested new score distribution. We illustrate and discuss the behavior of Rasch mixture models with different options for the score distribution in a Monte Carlo study in the next section. Then, the suggested approach for DIF detection via Rasch mixture models is illustrated through an empirical application to a study on verbally aggressive behavior. Concluding remarks are provided in the last section.

Theory

The Rasch Model

The Rasch model, introduced by Georg Rasch (1960), models the probability for a binary response $y_{i j} \in {0, 1}$ by subject $i$ to item $j$ as dependent on the subject’s ability $θ_{i}$ and the item’s difficulty $β_{j}$ . Assuming independence between items given the subject, the probability for observing a vector $y_{i} = {(y_{i 1}, \dots, y_{im})}^{T}$ with responses to all $m$ items by subject $i$ can be written as

P (Y_{i} = y_{i} | θ_{i}, β) = \prod_{j = 1}^{m} \frac{\exp {y_{i j} (θ_{i} - β_{j})}}{1 ​ + ​ \exp {θ_{i} - β_{j}}},

depending on the subject’s ability $θ_{i}$ and the vector of all item difficulties $β = {(β_{1}, …, β_{m})}^{T} .$ Capital letters denote random variables, and lower case letters denote their realizations.

Since joint maximum likelihood (JML) estimation of all abilities and difficulties is not consistent for a fixed number of items $m$ (Molenaar, 1995), conditional maximum likelihood (CML) estimation is employed here. This exploits that the number of correctly scored items, the so-called raw score $R_{i} = \sum_{j = 1}^{m} Y_{ij}$ , is a sufficient statistic for the ability $θ_{i}$ (Molenaar, 1995). Therefore, the answer probability from Equation 1 can be split into two factors where the first factor is conditionally independent of $θ_{i}$ :

\begin{array}{l} P (Y_{i} = y_{i} | θ_{i}, β) = P (Y_{i} = y_{i} | r_{i}, θ_{i}, β) P (R_{i} = r_{i} | θ_{i}, β) \\ = \underset{h (y_{i} | r_{i}, β)}{\underset{︸}{P (Y_{i} = y_{i} | r_{i}, β)}} \underset{g (r_{i} | θ_{i}, β)}{\underset{︸}{P (R_{i} = r_{i} | θ_{i}, β)}} . \end{array}

Due to this separation, consistent estimates of the item parameters $β$ can be obtained by maximizing only the conditional part of the likelihood $h (\cdot)$ :

h (y_{i} | r_{i}, β) = \frac{\exp {- \sum_{j = 1}^{m} y_{ij} β_{j}}}{γ_{r_{i}} (β)},

with $γ_{j} (\cdot)$ denoting the elementary symmetric function of order $j$ . The resulting CML estimates $\hat{β}$ are consistent, asymptotically normal, and asymptotically efficient (Molenaar, 1995).

If not only the conditional likelihood but the full likelihood is of interest—as in Rasch mixture models—then the score distribution $g (\cdot)$ needs to be specified as well. The approach used by Rost (1990) and Rost and von Davier (1995) is to employ some distribution for the raw scores $r_{i}$ based on a set of auxiliary parameters $δ .$ Then the probability density function for $y_{i}$ can be written as

f (y_{i} | β, δ) = h (y_{i} | r_{i}, β) g (r_{i} | δ) .

Based on this density, the following subsections first introduce mixture Rasch models in general and then discuss several choices for $g (\cdot)$ . CML estimation is used throughout for estimating the Rasch model, that is, the conditional likelihood $h (\cdot)$ is always specified by Equation 2.

Rasch Mixture Models

Mixture models are essentially a weighted sum over several components, that is, here over several Rasch models. Using the Rasch model density function from Equation 3, the likelihood $L (\cdot)$ of a Rasch mixture model with $K$ components for data from $n$ respondents is given by

\begin{matrix} L (π^{(1)}, \dots, π^{(K)}, β^{(1)}, \dots, β^{(K)}, δ^{(1)}, \dots, δ^{(K)}) & = Π_{i = 1}^{n} \sum_{k = 1}^{K} π^{(k)} f (y_{i} | β^{(k)}, δ^{(k)}) \\ = Π_{i = 1}^{n} \sum_{k = 1}^{K} π^{(k)} h (y_{i} | r_{i}, β^{(k)}) g (r_{i} | δ^{(k)}), \end{matrix}

where the $(k)$ -superscript denotes the component-specific parameters: the component weight $π^{(k)}$ , the component-specific item parameters $β^{(k)}$ , and the component-specific score parameters $δ^{(k)}$ for $k = 1, \dots, K .$

This kind of likelihood can be maximized via the expectation-maximization (EM) algorithm (Dempster, Laird, & Rubin, 1977), which alternates between maximizing the component-specific likelihoods for obtaining parameter estimates and computing expectations for each observations belonging to each cluster.

More formally, given (initial) estimates for the model parameters ${\hat{π}}^{(k)}$ , ${\hat{β}}^{(k)}$ , ${\hat{δ}}^{(k)}$ for all components $k = 1, \dots, K$ , posterior probabilities of each observation $i$ belonging to a component, or latent class, $k$ are calculated in the E-step. This is simply i’s relative contribution to component $k$ compared to the sum of all its contributions:

{\hat{p}}_{ik} = \frac{{\hat{π}}^{(k)} f (y_{i} | {\hat{β}}^{(k)}, {\hat{δ}}^{(k)})}{\sum_{ℓ = 1}^{K} {\hat{π}}^{(ℓ)} f (y_{i} | {\hat{β}}^{(ℓ)}, {\hat{δ}}^{(ℓ)})} = \frac{{\hat{π}}^{(k)} h (y_{i} | r_{i}, {\hat{β}}^{(k)}) g (r_{i} | {\hat{δ}}^{(k)})}{\sum_{ℓ = 1}^{K} {\hat{π}}^{(ℓ)} h (y_{i} | r_{i}, {\hat{β}}^{(ℓ)}) g (r_{i} | {\hat{δ}}^{(ℓ)})} .

In the M-step of the algorithm, these posterior probabilities are used as the weights in a weighted ML estimation of the model parameters. This way, an observation deemed unlikely to belong to a certain latent class does not contribute strongly to its estimation. Estimation can be done separately for each latent class. Using CML estimation for the Rasch Model, the estimation of item and score parameters can again be done separately. For all components $k = 1, \dots, K$ :

\begin{matrix} ({\hat{β}}^{(k)}, {\hat{δ}}^{(k)}) = \underset{β^{(k)}, δ^{(k)}}{argmax} \sum_{i = 1}^{n} {\hat{p}}_{ik} \log f (y_{i} | β^{(k)}, δ^{(k)}) \\ = {\underset{β^{(k)}}{argmax} \sum_{i = 1}^{n} {\hat{p}}_{ik} \log h (y_{i} | r_{i}, β^{(k)}); \underset{δ^{(k)}}{argmax} \sum_{i = 1}^{n} {\hat{p}}_{ik} \log g (r_{i} | δ^{(k)})} . \end{matrix}

Estimates of the class probabilities can be obtained from the posterior probabilities by averaging:

{\hat{π}}^{(k)} = \frac{1}{n} \sum_{i = 1}^{n} {\hat{p}}_{ik} .

The E-step (Equation 5) and M-step (Equations 6 and 7) are iterated until convergence, always updating either the weights based on current estimates for the model parameters or vice versa.

Note that the above implicitly assumes that the number of latent classes $K$ is given or known. However, this is typically not the case in practice and $K$ needs to be chosen based on the data. As $K$ is not a model parameter—regularity conditions for the likelihood ratio test are not fulfilled (McLachlan & Peel, 2000)—it is often chosen via some information criterion that balances goodness of fit (via the likelihood) with a penalty for the number of model parameters. Since the various information criteria differ in their penalty term, the decision which model is considered “best” may depend on the information criterion chosen. In the following, the BIC (Bayesian information criterion; Schwarz, 1978) is used, which Li, Cohen, Kim, and Cho (2009) found to be a suitable model selection method for dichotomous mixture item response theory models. Note that this is not a formal significance test because one does not control a Type I error rate.

Score Distribution

In a single Rasch model, the estimation of the item parameters is invariant to the score distribution because of the separation in Equation 3. In the mixture context, this invariance property holds only given the weights in Equation 6. However, these posterior weights depend on the full Rasch likelihood, including the score distribution (Equation 5). Therefore, the estimation of the item parameters in a Rasch mixture model is not independent of the score distribution for $K > 1$ , even if the CML approach is employed. Hence, it is important to consider the specification of the score distribution when estimating Rasch mixture models and to assess the consequences of potential misspecifications.

Saturated and Mean-Variance Specification

In his introduction of the Rasch mixture model, Rost (1990) suggests a discrete probability distribution on the scores with a separate parameter for each possible score. This requires $m - 2$ parameters per latent class as the probabilities need to sum to $1$ (and the extreme scores, $r = 0$ and $r = m$ , do not contribute to the likelihood).

Realizing that this saturated specification requires a potentially rather large number of parameters, Rost and von Davier (1995) suggest a parametric distribution with one parameter each for mean and variance.

Details on both specifications can be found in Rost (1990) and Rost and von Davier (1995), respectively. Here, the notation of Frick, Strobl, Leisch, and Zeileis (2012) is adopted, which expresses both specifications in a unified way through a conditional logit model for the score $r = 1, \dots, m - 1$ :

g (r | δ^{(k)}) = \frac{\exp {z_{r}^{T} δ^{(k)}}}{\sum_{j = 1}^{m - 1} \exp {z_{j}^{T} δ^{(k)}}},

with different choices for $z_{r}$ leading to the saturated and mean-variance specification, respectively. For the former, the regressor vector is $(m - 2)$ -dimensional with

z_{r} = {(0, \dots, 0, 1, 0, \dots, 0)}^{T}

and the 1 at position $r - 1$ . Consequently, if $r = 1$ , $z_{r}$ is a vector of zeros. For the mean-variance specification, the regressor vector is two-dimensional and given by

z_{r} = {(\frac{r}{m}, \frac{4 r (m - r)}{m^{2}})}^{Τ} .

Restricted Specification

In the following we suggest a new specification of the score distribution in the Rasch mixture model, which aims at obtaining independence of the item parameter estimates from the specification of the score distribution and therefore enabling the Rasch mixture model to distinguish between DIF and impact. Other global DIF detection methods like the LR test and Rasch trees are able to make this distinction (Ankenmann, Witt, & Dunbar, 1999; Strobl et al., 2013) because they are based only on the conditional part of the likelihood (Equation 2). Analogously, we suggest a mixture of only this conditional part rather than the full likelihood (Equation 3) of the Rasch model so that the mixture model will only be influenced by differences in the item parameters.

Mixing only the conditional likelihood $h (\cdot)$ means that the sum over the $K$ latent classes in the likelihood of the Rasch mixture model in Equation 4 only applies to $h (\cdot)$ but not to the score distribution $g (\cdot)$ . The mixture is then only based on latent structure in the item difficulties, not on latent structure in both difficulties and scores. Moreover, such a Rasch mixture model based only on the conditional likelihood without any score distribution is equivalent to a Rasch mixture model where the score distribution is independent of the latent class $k = 1, \dots, K$ :

g (r | δ^{(k)}) = g (r | δ) (k = 1, \dots, K),

because then the factor $g (r | δ)$ is a constant that can be moved out of the sum over the components $k$ in Equation 4. Consequently, compared to the case without any score distribution, the log-likelihood just changes by an additional constant without component-specific parameters. In either case, the estimation of the component-specific parameters item parameters as well as the selection of the number of components $K$ is independent of the specification of the score distribution.

This equivalence and independence from the score distribution can also be seen easily from the definition of the posterior weights (Equation 5): If restricted, $g (\cdot)$ can be moved out of the sum and then cancels out, preserving only the dependence on $h (\cdot)$ . Thus, the ${\hat{p}}_{ik}$ depend only on ${\hat{π}}^{(k)}$ and ${\hat{β}}^{(k)}$ but not ${\hat{δ}}^{(k)} .$ Therefore, the component weights and component-specific item parameters can be estimated without any specification of the score distribution.

Subsequently, we adopt the restricted perspective rather than omitting $g (\cdot)$ completely, when we want to obtain a mixture model where the mixture is independent of the score distribution. From a statistical point of view this facilitates comparisons of the restricted Rasch mixture model with the corresponding unrestricted counterpart.

Overview

The different specifications of the score distribution vary in their properties and implications for the whole Rasch mixture model.

The saturated model is very flexible. It can model any shape and is thus never misspecified. However, it needs a potentially large number of parameters, which can be challenging in model estimation and selection.

The mean-variance specification of the score model is more parsimonious as it only requires two parameters per latent class. While this is convenient for model fit and selection, it also comes at a cost: since it can only model unimodal or U-shaped distributions (see Rost & von Davier, 1995), it is partially misspecified if the score distribution is actually multimodal.

A restricted score model is even more parsimonious. Therefore, the same advantages in model fit and selection apply. Furthermore, it is invariant to the latent structure in the score distribution. If a Rasch mixture model is used for DIF detection, this is favorable as only differences in the item difficulties influence the mixture. However, it is partially misspecified if the latent structure in the scores and item difficulties coincides.

Monte Carlo Study

The simple question DIF or no DIF? leads to the question whether the Rasch mixture model is suitable as a tool to detect such violations of measurement invariance.

As the score distribution influences the estimation of the Rasch mixture model in general, it is of particular interest how it influences the estimation of the number of latent classes, the measure used to determine Rasch scalability.

Motivational Example

As a motivation for the simulation design, consider the following example: The instrument is a knowledge test that is administered to students from two different types of schools and who have been prepared by one of two different courses for the knowledge test. Either of the two groupings might be the source of DIF (or impact). If the groupings are available as covariates to the item responses of the students, then a test for DIF between either school types or course types can be easily carried out using the LR test. However, if the groupings are not available (or even observed) as covariates, then a DIF test is still possible by means of the Rasch mixture model. The performance of such a DIF assessment is investigated in our simulation study for different effects of school and course type, respectively.

In the following we assume that the school type is linked to ability difference (i.e., impact but not DIF) while the course type is the source of DIF (but not impact). This can be motivated in the following way (see also Figure 1): When the students from the two school types differ in their mean ability, this is impact between these two groups. The courses might be a standard course and a new specialized course. While the standard course covers all topics of the test equally, the specialized course gives more emphasis to a relatively advanced topic and due to time constraints less emphasis to a relatively basic topic. This may lead to DIF between the students in the standard and the specialized course. See the left panel of Figure 2 for illustrative item profiles of the standard course (in dark gray) and the specialized course (in light gray).

Figure 1.

Grouping structure in the motivational example.

Figure 2.

Scenario 2. Left: Item difficulties with DIF ( $Δ = 2$ ). Right: Stacked histogram of unimodal score distribution with homogeneous abilities ( $Θ = 0$ ).

Finally, the ability groups by school and the DIF groups by course can either coincide or not. If all students in the first school type are being taught the standard course while all students in the second school type are being taught the specialized course, the DIF groups coincide with the ability groups. The DIF and ability groups do not coincide but only overlap partly if both course types are taught in both school types: each DIF group (based on the type of course taught) consists of a mix of students from both schools and therefore from both ability groups. An illustration of coinciding and not coinciding ability and DIF groups is provided in the upper and lower rows of Figure 1, respectively. Ability groups, based on school type, are shown in the columns, while DIF groups, based on course type, are illustrated with dark and light gray for the standard course and specialized course, respectively. This difference of coinciding or not coinciding DIF and ability groups might have an influence on the Rasch mixture model’s ability to detect the DIF because in the former case the score distributions differ between the two DIF groups while in the latter case they do not.

Subsequently, a Monte Carlo study is carried out to investigate how the Rasch mixture model performs in situations where such groupings are present in the underlying data-generating process but are not available as observed covariates. Moreover, we vary whether or not all students come from the same school type (i.e., from the same ability distribution), whether or not all students receive the standard course (i.e., whether there is DIF), and whether both school types use the same or different courses (i.e., whether the groupings coincide or not). For all computations, the R system for statistical computing (R Core Team, 2013) is used along with the add-on packages psychomix (Frick et al., 2012) and clv (Nieweglowski, 2009).

Simulation Design

The simulation design combines ideas from the motivational example with aspects from the simulation study conducted by Rost (1990). Similar to the original simulation study, the item parameters represent an instrument with increasingly difficult items. Here, 20 items are employed with corresponding item parameters $β^{I}$ which follow a sequence from $- 1.9$ to 1.9 with increments of 0.2 and hence sum to zero.

\begin{array}{l} β^{I} = {(- 1.9, - 1.7, \dots, 1.7, 1.9)}^{Τ} \\ β^{I I} = {(- 1.9, - 1.7, \dots, - 1.1 + Δ, \dots,1.1 - Δ, \dots, 1.7, 1.9)}^{Τ} \end{array}

To introduce DIF, a second set of item parameters $β^{II}$ is considered where Items 5 and 16 are changed by $\pm Δ$ . This approach is similar in spirit to that of Rost (1990)—who reverses the full sequence of item parameters to generate DIF—but allows for gradually changing from small to large DIF effect sizes. Subject abilities are drawn with equal weights from two normal distributions with means $- Θ / 2$ and $+ Θ / 2$ and standard deviation 0.3, thus creating a sample with two groups of subjects: one group with a lower mean ability and one with a higher mean ability.

In the simulations below, the DIF effect size $Δ$ ranges from 0 to 4 in steps of 0.2

Δ \in {0, 0.2, \dots, 4}

while the impact $Θ$ covers the same range in steps of 0.4:

Θ \in {0, 0.4, \dots, 4} .

Impact and DIF, or lack thereof, can be combined in several ways. Table 1 provides an overview and Figures 2, 3, and 4 show illustrations. In the following, the different combinations of impact and DIF are explained in more detail and connected to the motivational example:

If the simulation parameter $Δ$ for the DIF effect size is set to zero, both sets of item parameters, $β^{I}$ and $β^{II}$ , are identical and no DIF is present. Since CML is employed, model selection and parameter estimation is typically expected to be independent of whether or not an impact is present (Scenarios 1 and 3 in Table 1).

In the example: Only the standard course is taught and hence no DIF exists.

If $Δ > 0$ , the item parameter set $β^{II}$ is different from $β^{I}$ . Hence, there is DIF and two latent classes exist (Scenarios 2, 4, and 5). Both classes are chosen to be of equal size in this case. For an illustration, see the left panel of Figure 2.

In the example: Both courses are taught, thus leading to DIF. The standard course corresponds to the straight line as the item profile while the specialized course corresponds to the spiked item profile with relatively difficult Item 16 being easier and the relatively easy Item 5 being more difficult for students in this specialized course than for students in the standard course.

If the simulation parameter $Θ$ for the impact is set to zero (Scenarios 1 and 2), then the resulting score distribution is unimodal. For an illustration of such a unimodal score distribution see the right panel of Figure 2. This histogram illustrates specifically Scenario 2 where no impact is present but DIF exists. The histogram is shaded in light and dark gray for the two DIF groups and thus to be read like a “stacked histogram.”

In the example: All students are from the same school and hence there is no impact. However, both types of courses may be taught in this one school, thus leading to DIF as in Scenario 2.

If $Θ > 0$ , subject abilities are sampled with equal weights from two normal distributions with means ${- Θ / 2, + Θ / 2}$ , thus generating impact. When no DIF is included (Scenario 3), the resulting score distribution moves from being unimodal to being bimodal with increasing $Θ$ . The two modi of high and low scores represent the two groups of subjects with high and low mean abilities, respectively. However, only a medium gray is used to shade the illustrating histogram in Figure 3 as no DIF groups are present.

In the example: Only the standard course is taught in both school types. Hence no DIF is present but impact between the school types.

If there is DIF (i.e., $Δ > 0$ ) in addition to impact (i.e., $Θ > 0$ ), subjects can be grouped both according to mean ability (high vs. low) and difficulty (straight vs. spiked profile in $β^{I}$ and $β^{II}$ , respectively).

These groups can coincide: For subjects with low mean ability $- Θ / 2$ , item difficulties $β^{I}$ hold, while for subjects with high mean ability $+ Θ / 2$ , item difficulties $β^{II}$ hold. This is simulated in Scenario 5 and labeled Impact and DIF, coinciding. The resulting score distribution is illustrated in the right panel of Figure 4. Subjects for whom item difficulties $β^{I}$ hold are shaded in dark gray, and as they also have lower mean abilities, their scores are all relatively low. Conversely, subjects for whom item difficulties $β^{II}$ hold are shaded in light gray, and as they also have higher mean abilities, their scores are all relatively high.

Additionally, the DIF groups and ability groups can also not coincide: Subjects in either DIF group may stem from both ability groups, not just one. This is simulated in Scenario 4 and labeled Impact and DIF, not coinciding. The resulting score distribution is illustrated in the left panel of Figure 4. Again, subjects for whom item difficulties $β^{I}$ and $β^{II}$ hold are shaded in dark and light gray, respectively. As subjects stem from both ability groups (high vs. low abilities), both score distributions are bimodal.

In the example: Students from both school types and from both course types are considered, thus leading to both impact and DIF. Either both courses are taught at both schools (Scenario 4, not coinciding) or the standard course is only taught in the first school and the specialized course is only taught at the second school (Scenario 5, coinciding).

Table 1.

Simulation Design. The Latent-Class-Specific Item Parameters $β^{I}$ and $β^{II}$ Differ by $Δ$ for Two Elements and Thus Coincide for $Δ = 0$ , Leaving Only a Single Latent Class.

	Latent Class I		Latent Class II
Scenario	Mean abilities	Difficulties	Mean abilities	Difficulties
No impact $(Θ = 0)$
1. No DIF $(Δ = 0)$	${0}$	$β^{I}$	—	—
2. DIF $(Δ > 0)$	${0}$	$β^{I}$	${0}$	$β^{II}$
Impact $(Θ > 0)$
3. No DIF $(Δ = 0)$	${- Θ / 2, + Θ / 2}$	$β^{I}$	—	—
4. DIF $(Δ > 0),$ not coinciding	${- Θ / 2, + Θ / 2}$	$β^{I}$	${- Θ / 2, + Θ / 2}$	$β^{II}$
5. DIF $(Δ > 0),$ coinciding	${- Θ / 2}$	$β^{I}$	${+ Θ / 2}$	$β^{II}$

Figure 3.

Scenario 3. Left: Item difficulties without DIF ( $Δ = 0$ ). Right: Histogram of bimodal score distribution with impact ( $Θ = 2$ ).

Figure 4.

Stacked histograms of score distributions for Scenarios 4 (left) and 5 (right) with DIF ( $Δ = 2$ ). Left: impact and DIF, not coinciding ( $Θ = 2$ ). Right: impact and DIF, coinciding ( $Θ = 2$ ). For item difficulties see Figure 2 (left).

Note that Scenario 1 is a special case of Scenario 2 where $Δ$ is reduced to zero as well as a special case of Scenario 3 where $Θ$ is reduced to zero. Therefore, in the following, Scenario 1 is not inspected separately but included in both the setting of No impact with DIF (Scenario 2) and the setting of Impact without DIF (Scenario 3) as a reference point. Similarly, Scenarios 4 and 5 both can be reduced to Scenario 3 if $Δ$ is set to zero. It is therefore also included in both the setting of Impact and DIF, not coinciding (Scenario 4) and the setting of Impact and DIF, coinciding (Scenario 5) as a reference point.

For each considered combination of $Δ$ and $Θ$ , 500 data sets of 500 observations each are generated. Larger numbers of data sets or observations lead to very similar results. Observations with raw scores of 0 or $m$ are removed from the data set as they do not contribute to the estimation of the Rasch mixture model (Rost, 1990). For each data set, Rasch mixture models for each of the saturated, mean-variance, and restricted score specifications are fitted for $K = 1, 2, 3$ .

False Alarm Rate and Hit Rate

The main objective here is to determine how suitable a Rasch mixture model, with various choices for the score model, is to recognize DIF or the lack thereof.

For each data set and type of score model, models with $K = 1, 2, 3$ latent classes are fitted and the $\hat{K}$ associated with the minimum BIC is selected. Choosing one latent class ( $\hat{K} = 1$ ) then corresponds to assuming measurement invariance while choosing more than one latent class ( $\hat{K} > 1$ ) corresponds to assuming violations of measurement invariance. While Rasch mixture models do not constitute a formal significance test, the empirical proportion among the 500 data sets with $\hat{K} > 1$ corresponds in essence to the power of DIF detection if $Δ > 0$ (and thus two true latent classes exist) and to the associated Type I error of a corresponding test if $Δ = 0$ (and thus only one true latent class exists). If the rate corresponds to power, it will be referred to as hit rate, whereas if it corresponds to a type I error it will be referred to as false alarm rate.

In the following subsections, the key results of the simulation study will be visualized. The exact rates for all conditions are included as a data set in the R package psychomix; for details, see the section on computational details.

Scenario 2: No Impact With DIF

This scenario is investigated as a case of DIF that should be fairly simple to detect. There is no impact as abilities are homogeneous across all subjects so the only latent structure to detect is the group membership based on the two item profiles. This latent structure is made increasingly easy to detect by increasing the difference between the item difficulties for both latent groups. In the graphical representation of the item parameters (left panel of Figure 2) this corresponds to enlarging the spikes in the item profile.

Figure 5 shows how the rate of choosing a model with more than one latent class ( $\hat{K} > 1$ ) increases along with the DIF effect size $Δ$ . At $Δ = 0$ , this is a false alarm rate. It is around 7% for the saturated model and very close to zero for the mean-variance and the saturated score model (<1%). With increasing $Δ > 0$ , the rate is a hit rate. For low values of $Δ$ the two more parsimonious versions of the Rasch mixture model (with mean-variance and restricted score distribution) are not able to pick up the DIF, but at around $Δ = 3$ the hit rate for the two models increases and almost approaches 1 at $Δ = 4$ . Not surprisingly, the restricted score specification performs somewhat better because in fact the raw score distributions do not differ between the two latent classes. The baseline hit rate of the saturated model for low values of $Δ$ is the same as the false alarm rate for $Δ = 0$ . It only increases beyond the same threshold ( $Δ = 3$ ) as the hit rate of the other two models. However, its rate is much lower compared to the other two score model (only around 30%). The reason is that it requires 18 additional score parameters for an additional latent class, which is “too costly” in terms of BIC. Hence, $\hat{K} = 1$ is chosen for most Rasch mixture models using a saturated score distribution.

Figure 5.

Rate of choosing a model with $\hat{K} > 1$ latent classes for data from Scenario 2 (DIF without impact, i.e., $Θ = 0$ ).

The number of iterations in the EM algorithm that are necessary for the estimation to converge is much lower for the mean-variance and the restricted model than for the saturated model. Since the estimation of the saturated model is more extensive due to the higher number of parameters required by this model, it does not converge in about 10% of the cases before reaching the maximum number of iterations which was set to 400. The mean-variance and saturated model usually converge within the first 200 iterations.

Brief summary

The mean-variance and restricted model have higher hit rates than the saturated model in the absence of impact.

Scenario 3: Impact Without DIF

Preferably, a Rasch mixture model should not only detect latent classes if the assumption of measurement invariance is violated but it should also indicate a lack of latent structure if indeed the assumption holds. In this scenario, the subjects all stem from the same class, meaning each item is of the same difficulty for every subject. However, subject abilities are simulated with impact resulting in a bimodal score distribution as illustrated in Figure 3.

Here, the rate of choosing more than one latent class can be interpreted as a false alarm rate (Figure 6). The restricted score model is invariant against any latent structure in the score distribution and thus almost always (≤0.2%) suggests $\hat{K} = 1$ latent class based on the DIF-free item difficulties. The rate does not approach any specific significance level as the Rasch mixture model, regardless of the employed score distribution, is not a formal significance test. The saturated model also picks $\hat{K} = 1$ in most of the simulation. This might be due to its general reluctance to choose more than one latent class as illustrated in Figure 5 or the circumstance that it can assume any shape (including bimodal patterns). However, the mean-variance score distribution can only model unimodal or U-shaped distributions as mentioned above. Hence, with increasing impact and thus increasingly well-separated modes in the score distribution, the Rasch mixture model with this score specification suggests $\hat{K} > 1$ latent classes in up to 53% of the cases. Note, however, that these latent classes do not represent the DIF groups (as there are none) but rather groups of subjects with high versus low abilities. While this may be acceptable (albeit unnecessarily complex) from a statistical mixture modeling perspective, it is misleading from a psychometric point of view if the aim is DIF detection. Only one Rasch model needs to be estimated for this type of data, consistent item parameter estimates can be obtained via CML and all observations can be scaled in the same way.

Figure 6.

Rate of choosing a model with $\hat{K} > 1$ latent classes for data from Scenario 3 (impact without DIF, i.e., $Δ = 0$ ).

Brief summary

If measurement invariance holds but ability differences are present, the mean-variance model exhibits a high false alarm rate while the saturated and restricted model are not affected.

Scenario 4: Impact and DIF, Not Coinciding

In this scenario, there is DIF (and thus two true latent classes) if $Δ > 0$ . Again, Scenario 3 with $Δ = 0$ (and thus without DIF) is included as a reference point. However, unlike in Scenario 2, the abilities within the latent classes are not homogeneous but two ability groups exist, which do not coincide with the two DIF groups. Nonetheless, the score distribution is the same across both latent classes (illustrated in the left panel of Figure 4).

Figure 7 again shows the rate of choosing $\hat{K} > 1$ for increasing DIF effect size $Δ$ for two levels of impact ( $Θ = 2.4$ and $3.6$ ), exemplary for medium and high impact. If impact is small (e.g., $Θ = 0.4$ ), the rates are very similar to the case of completely homogeneous abilities without impact (Figure 4 with $Θ = 0$ ) and thus not visualized here. While the rates for the restricted and the saturated score model do not change substantially for an increased impact ( $Θ = 2.4$ and $3.6$ ), the mean-variance model is influenced by this change in ability differences. While the hit rate is increased to around 20% over the whole range of $Δ$ , the false alarm rate at $Δ = 0$ is increased to the same extent. Moreover, the hit rate only increases noticeably beyond the initial false alarm rate at around $Δ = 3$ , that is, the same DIF effect size at which the restricted and mean-variance specifications have an increasing hit rate given homogeneous abilities without impact. Thus, given rather high impact ( $Θ = 3.6$ ) the hit rate is not driven by the DIF detection but rather the model’s tendency to assign subjects with high vs. low abilities into different groups (as already seen in Figure 6).

Figure 7.

Rate of choosing a model with $\hat{K} > 1$ latent classes for data from Scenario 4 (impact and DIF, not coinciding).

As Rasch mixture models with $K = 1, 2, 3$ classes are considered, selecting $\hat{K} > 1$ classes can either mean selecting the correct number of $K = 2$ or overselecting $\hat{K} = 3$ classes. For the saturated and restricted specifications overselection is rare (occurring with rates of less than 9% or less than 1%, respectively). However, similar to Scenario 3 overselection is not rare for the mean-variance specification. Figure 8 depicts the rates of selecting $\hat{K} = 2$ and $\hat{K} = 3$ classes, respectively, for increasing $Δ$ at $Θ = 3.6$ . If the chances of finding the correct number of classes increase with the DIF effect size $Δ$ , the rate for overselection ( $\hat{K} = 3$ ) should drop with increasing $Δ$ . For Scenario 4, denoted with hollow symbols, this rate stays largely the same (around 25%) and even slightly increases beyond this level, starting from around $Δ = 3$ . This illustrates again the pronounced tendency of the mean-variance model for overselection in cases of high impact.

Figure 8.

Rates of choosing the correct number of classes ( $\hat{K} = 2$ ) or overselecting the number of classes ( $\hat{K} = 3$ ) for the Rasch mixture model with mean-variance score specification in Scenarios 4 (hollow, impact within DIF groups) and 5 (solid, impact between DIF groups).

Brief summary

If impact is simulated within DIF groups, the mean-variance model has higher hit rates than the saturated and restricted models. However, the latent classes estimated by the mean-variance model are mostly based on ability differences if the DIF effect size is low. If the DIF effect size is high, the mean-variance model tends to overestimate the number of classes.

Scenario 5: Impact and DIF, Coinciding

In Scenario 5, there is also DIF (i.e., $Δ > 0$ ) and impact. However, in contrast to Scenario 4 the ability and DIF groups coincide (see the right panel of Figure 4). Furthermore, Scenario 3 is included also here as the reference point without DIF ( $Δ = 0$ ).

Again, small ability differences do not strongly influence the rate of choosing more than one latent class (rates for low levels of impact, such as $Θ = 0.4$ , are similar to those for $Θ = 0$ as depicted in Figure 5). Recall, both mean-variance and restricted specification have comparable hit rates for DIF detection starting from around $Δ = 3$ while the saturated specification has lower hit rates.

As impact increases (Figure 9), the hit rates of all models increases as well because the ability differences contain information about the DIF groups: separating subjects with low and high abilities also separates the two DIF groups (not separating subjects within each DIF group as in the previous setting). However, for the mean-variance model these increased hit rates are again coupled with a highly increased false alarm rate at $Δ = 0$ of 26% and 50% for $Θ = 2.4$ and $3.6$ , respectively. The restricted score model, on the other hand, is invariant to latent structure in the score distribution and thus performs similarly as in previous DIF scenarios, suggesting more than one latent class past a certain threshold of DIF intensity, albeit this threshold being a bit lower than when ability groups and DIF groups do not coincide (around $Δ = 2$ ). The saturated model detects more than one latent class at a similar rate to the restricted score model for medium or high impact but its estimation converges more slowly and requires more iterations of the EM algorithm than the other two score models.

Figure 9.

Rate of choosing a model with $\hat{K} > 1$ latent classes for data from Scenario 5 (impact and DIF, coinciding).

Finally, the potential issue of overselection can be considered again. Figure 8 (solid symbols) shows that this problem disappears for the mean-variance specification if both DIF effect size $Δ$ and impact are large and coincide. For the restricted model overselection is again very rare throughout (occurring in less than 1% of all cases) while the saturated model overselects in up to 29% of the datasets.

Brief summary

If abilities differ between DIF groups, the mean-variance model detects the violation of measurement invariance for smaller DIF effect sizes than the saturated and restricted model. While the mean-variance model does not overselect the number of components in this scenario, the high hit rates are connected to a high false alarm rate when no DIF is present but impact is high. This does not affect the other two score models.

Quality of Estimation

Although here the Rasch mixture model is primarily used analogously to a global DIF test, model assessment goes beyond the question whether or not the correct number of latent classes is found. Once the number of latent classes is established/estimated, it is of interest how well the estimated model fits the data. Which groups are found? How well are the parameters estimated? In the context of Rasch mixture models with different score distributions, both of these aspects depend heavily on the posterior probabilities ${\hat{p}}_{ik}$ (Equation 5) as the estimation of the item parameters depends on the score distribution only through these. If the ${\hat{p}}_{ik}$ were the same for all three score specifications, the estimated item difficulties were the same as well. Hence, the focus here is on how close the estimated posterior probabilities are to the true latent classes in the data. If the similarity between these is high, CML estimation of the item parameters within the classes will also yield better results for all score models.

This is a standard task in the field of cluster analysis and we adopt the widely used Rand index (Rand, 1971) here: Each observation is assigned to the latent class for which its posterior probability is highest yielding an estimated classification of the data which is compared to the true classification. For this comparison, pairs of observations are considered. Each pair can either be in the same class in both the true and the estimated classification, in different classes for both classifications, or it can be in the same class for one but not the other classification. The Rand index is the proportion of pairs for which both classifications agree. Thus, it can assume values between 0 and 1, indicating total dissimilarity and similarity, respectively.

In the following, the Rand index for models with the true number of $K = 2$ latent classes in Scenarios 4 and 5 (with DIF) is considered. Thus, the question of DIF detection (or model selection) is not investigated again but only the quality of latent class recovery (assuming the number of classes $K$ to be known or correctly selected). The top row of Figure 10 depicts the average Rand index for data from Scenario 4 (impact and DIF, not coinciding). Here, all three score specifications find similarly well-matching classifications, while the Rand index generally decreases with increasing impact (left to right panel). In particular, while the mean-variance score model has problems finding the correct number of latent classes in this scenario, it only performs slightly worse than the other two specifications in determining the correct classes if the number were known. Similarly, if it is provided with the correct number of classes, the saturated model also identifies the correct classes equally well compared to the other models—despite its difficulties with convergence for higher DIF effect sizes.

Figure 10.

Average Rand index for models with $K = 2$ latent classes. Top row: Scenario 4 (impact and DIF, not coinciding). Bottom row: Scenario 5 (impact and DIF, coinciding).

However, in Scenario 5 where the score distribution contains information about the DIF groups, the three score specifications perform very differently as the bottom row of Figure 10 shows. Given the correct number of classes, the mean-variance model is most suitable to uncover the true latent classes, yielding Rand indices close to 1 if both DIF effect size and impact are large. The saturated specification follows a similar pattern albeit with poorer results, reaching values of up to 0.87. However, the classifications obtained from the restricted score specification do not match the true groups well in this scenario, remaining below 0.52 if impact is high. The reason is that the restricted score model is partially misspecified as the score distributions differ substantially across DIF groups.

Summary and Implications for Practical Use

Given various combinations of DIF and ability impact, the score models are differently suitable for the two tasks discussed here—DIF detection and estimation of item parameters in subgroups. Starting with a summary of the results for DIF detection:

The saturated score model has much lower hit rates than the other two specifications, that is, violation of measurement invariance remains too often undetected. Only if high impact and high DIF effect sizes coincide does the saturated model perform similarly well as the restricted model.

The mean-variance model has much higher hit rates. However, if impact is present in the abilities, this specification has highly inflated false alarm rates. Hence, if the mean-variance model selects more than one latent class it is unclear whether this is due to DIF or just varying subject abilities. Thus, measurement invariance might still hold even if more than one latent class is detected.

The restricted score model also has high hit rates, comparable to the mean-variance model if abilities are rather homogeneous. But unlike the mean-variance specification, its false alarm rate is not distorted by impact. Its performance is not influenced by the ability distribution and detecting more than one latent class reliably indicates DIF, that is, a violation of measurement invariance.

Hence, if the Rasch mixture model is employed for assessing measurement invariance or detecting DIF, then the restricted score specification appears to be most robust. Thus, the selection of the number of latent classes should only be based on this specification.

DeMars (2010) illustrates how significance tests based on the observed (raw) scores in reference and focal groups suffer from inflated Type I error rates with an increased sample size if impact is present. This does not apply to the false alarm rate of Rasch mixture models because not a significance test but rather model selection via BIC is carried out. The rate of the BIC selecting the correct model increases with larger sample size if the true model is a Rasch mixture model. Since consistent estimates are employed, a larger sample size also speeds up convergence, which is particularly desirable for the saturated model if the number of latent classes and thus the number of parameters is high.

Given the correct number of classes, the different score models are all similarly suitable to detect the true classification if ability impact does not contain any additional information about the DIF groups. However, if ability impact is highly correlated with DIF groups in the data and the ability groups thus coincide with the DIF groups, this information can be exploited by the unrestricted specifications while it distracts the restricted model.

Thus, while the selection of the number of latent classes should be based only on the restricted score specification, the unrestricted mean-variance and saturated specifications might still prove useful for estimating the Rasch mixture model (after $\hat{K}$ has been selected).

We therefore recommend a two-step approach for DIF detection via a Rasch mixture model. First, the number of latent classes is determined via the restricted score model. Second, if furthermore the estimation of the item difficulties is of interest, the full selection of score models can then be utilized. While the likelihood ratio test is not suitable to test for the number of latent classes, it can be used to establish the best fitting score model, given the number of latent classes. If this approach is applied to the full range of score models (saturated and mean-variance, both unrestricted and restricted), the nesting structure of the models needs to be kept in mind.

Empirical Application: Verbal Aggression

We use a data set on verbal aggression (De Boeck & Wilson, 2004) to illustrate this two-step approach of first assessing measurement invariance via a Rasch mixture model with a restricted score distribution and then employing all possible score models to find the best fitting estimation of the item difficulties.

Participants in this study are presented with one of two potentially frustrating situations (S1 and S2)

S1: A bus fails to stop for me

S2: I miss a train because a clerk gave me faulty information

and a verbally aggressive response (cursing, scolding, shouting). Combining each situation and response with either “I want to” or “I do” leads to the following 12 items:

S1WantCurse	S1DoCurse	S1WantScold	S1DoScold	S1WantShout	S1DoShout
S2WantCurse	S2DoCurse	S2WantScold	S2DoScold	S2WantShout	S2DoShout

First, we assess measurement invariance with regard to the whole instrument: we fit a Rasch mixture model with a restricted score distribution for $K = 1, 2, 3, 4$ and employ the BIC for model selection. Note that the restricted versions of the mean-variance and saturated model only differ in their log-likelihood by a constant factor and therefore lead to the same model selection. Results are presented in Table 2.

Table 2.

DIF Detection by Selecting the Number of Latent Classes $\hat{K}$ Using the Restricted Rasch Mixture Model.

Model	k	#df	log L	BIC
Restricted (mean-variance)	1	13	$- 1900.9$	3874.6
Restricted (mean-variance)	2	25	$- 1853.8$	3847.8
Restricted (mean-variance)	3	37	$- 1816.9$	3841.4
Restricted (mean-variance)	4	49	$- 1792.0$	3858.8

The BIC for a Rasch mixture model with more than one latent class is smaller than the BIC for a single Rasch model, thus indicating that measurement invariance is violated. The best fitting model has $\hat{K} = 3$ latent classes. Given this selection of $K$ , we want to gain further insight in the data and thus want to establish the best fitting Rasch mixture model with $K = 3$ latent classes. Four different models are conceivable: either using a restricted or unrestricted score model, and either using a saturated or mean-variance specification. The results for all four options are presented in Table 3. Note that the models with restricted saturated score distribution and restricted mean-variance score distribution lead to identical item parameter estimates. However, it is still of interest to fit them separately because each of the restricted specifications is nested within the corresponding unrestricted specification. Furthermore, the mean-variance distribution is nested within the saturated distribution.

Table 3.

Selection of the Score Distribution Given the Number of Latent Classes $\hat{K} = 3$ .

Model	k	#df	log L	BIC
Saturated	3	65	$- 1795.2$	3955.1
Restricted (saturated)	3	45	$- 1814.1$	3880.6
Mean-variance	3	41	$- 1812.2$	3854.4
Restricted (mean-variance)	3	37	$- 1816.9$	3841.4

As $K = 3$ is identical for all of these four models, standard likelihood ratio tests can be used for comparing all nested models with each other. Testing the most parsimonious score model, the restricted mean-variance model, against its unrestricted version and the restricted saturated model at a 5% level shows that a more flexible score model does not yield a significantly better fit. The p values are .051 and .685, respectively. Hence, the restricted mean-variance distribution is adopted here which also has the lowest BIC.

To visualize how the three classes found in the data differ, the corresponding item profiles are shown in Figure 11.

Figure 11.

Item profiles for the Rasch mixture model with $\hat{K} = 3$ latent classes using a restricted mean-variance score distribution for the verbal aggression data.

The latent class in the right panel (with 108 observations) shows a very regular zig-zag pattern where for any type of verbally aggressive response actually “doing” the response is considered more extreme than just “wanting” to respond a certain way as represented by the higher item parameters for the second item, the “do-item,” than the first item, the “want-item,” of each pair. The three types of response (cursing, scolding, shouting) are considered increasingly aggressive, regardless of the situation (first six items vs. last six items).

The latent class in the left panel (with 112 observations) distinguishes more strongly between the types of response. However, the relationship between wanting and doing is reversed for all responses except shouting. It is more difficult to agree to the item “I want to curse/scold” than to the corresponding item “I do curse/scold.” This could be interpreted as generally more aggressive behavior where one is quick to react a certain way rather than just wanting to react that way. However, shouting is considered a very aggressive response, both in wanting and doing.

The remaining latent class (with 53 observations considerably smaller), depicted in the middle panel, does not distinguish that clearly between response types, situations or wanting versus doing.

Therefore, not just a single item or a small number of items have DIF but the underlying want/do relationship of the items is different across the three classes. This instrument thus works differently as a whole across classes.

In summary, the respondents in this study are not scalable to one single Rasch-scale but instead need several scales to represent them accurately. A Rasch mixture model with a restricted score distribution is used to estimate the number of latent classes. Given that number of classes, any type of score model is conceivable. Here, the various versions are all fairly similar and the restricted mean-variance specification is chosen based on likelihood ratio tests. Keep in mind that the resulting fits can be substantially different from each other as shown in the simulation study, in particular for the case of impact between DIF classes. The latent classes estimated here differ mainly in their perception of the type and the “want/do” relationship of a verbally aggressive response.

Conclusion

Unlike in a single Rasch model, item parameter estimation is not independent of the score distribution in Rasch mixture models. The saturated and mean-variance specifications of the score model are both well established. A further option is the new restricted score specification introduced here. In the context of DIF detection, only the restricted score specification should be used as it prevents confounding effects of impact on DIF detection while exhibiting hit rates positively related to DIF effect size. Given the number of latent classes, it may be useful to fit the other score models as well, as they might improve estimation of group membership and therefore estimation of the item parameters. The best fitting model can be selected via the likelihood ratio test or an information criterion such as the BIC. This approach enhances the suitability of the Rasch mixture model as a tool for DIF detection as additional information contained in the score distribution is only employed if it contributes to the estimation of latent classes based on measurement invariance.

Computational Details

An implementation of all versions of the Rasch mixture model mentioned here is freely available under the General Public License in the R package psychomix from the Comprehensive R Archive Network. Accompanying the package at http://CRAN.R-project.org/package=psychomix is a data set containing all the simulation results and a vignette with a replication of the verbal aggression example.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Austrian Ministry of Science BMWF as part of the UniInfrastrukturprogramm of the Focal Point Scientific Computing at Universität Innsbruck.

References

Ackerman

T. A.

(1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29:67-91.

Andersen

E. B.

(1972). A goodness of fit test for the Rasch model. Psychometrika, 38, 123-140.

Andrich

(1978). A rating formulation for ordered response categories. Psychometrika, 43, 561-573.

Ankenmann

R. D.

Witt

E. A.

Dunbar

S. B.

(1999). An investigation of the power of the likelihood ratio goodness-of-fit statistic in detecting differential item functioning. Journal of Educational Measurement, 36, 277-300.

Baghaei

Carstensen

C. H.

(2013). Fitting the mixed Rasch model to a reading comprehension test: Identifying reader types. Practical Assessment, Research & Evaluation, 18(5), 1-13.

Cohen

A. S.

Bolt

D. M.

(2005). A mixture model analysis of differential item functioning. Journal of Educational Measurement, 42, 133-148.

De Boeck

Wilson

(Eds.). (2004). Explanatory item response models: A generalized linear and nonlinear approach. New York, NY: Springer-Verlag.

DeMars

C. E.

(2010). Type I error inflation for detecting DIF in the presence of impact. Educational and Psychological Measurement, 70, 961-972.

DeMars

C. E.

Lau

(2011). Differential item functioning detection with latent classes: How accurately can we detect who is responding differentially? Educational and Psychological Measurement, 71, 597-616.

10.

Dempster

Laird

Rubin

(1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B, 39(1), 1-38.

11.

Frick

Strobl

Leisch

Zeileis

(2012). Flexible Rasch mixture models with package psychomix. Journal of Statistical Software, 48(7), 1-25.

12.

Gustafsson

J.-E.

(1980). Testing and obtaining fit of data in the Rasch model. British Journal of Mathematical and Statistical Psychology, 33, 220.

13.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Lawrence Erlbaum.

14.

Hong

Min

S.-Y.

(2007). Mixed Rasch modeling of the self-rating depression scale: Incorporating latent class and Rasch rating scale models. Educational and Psychological Measurement, 67, 280-299.

15.

Cohen

A. S.

Kim

S.-H.

Cho

S.-J.

(2009). Model selection methods for mixture dichotomous IRT models. Applied Psychological Measurement, 33, 353-373.

16.

Brooks

G. P.

Johanson

G. A.

(2012). Item discrimination and type 1 error in the detection of differential item functioning. Educational and Psychological Measurement, 72, 847-861.

17.

Maij-de Meij

A. M.

Kelderman

van der Flier

(2010). Improvement in detection of differential item functioning using a mixture item response theory model. Multivariate Behavioral Research, 45, 975-999.

18.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.

19.

McLachlan

Peel

(2000). Finite mixture models. New York, NY: John Wiley.

20.

Molenaar

I. W.

(1995). Estimation of item parameters. In Fischer

G. H.

Molenaar

I. W.

(Eds.), Rasch models: Foundations, recent developments, and applications (pp. 39-51). New York, NY: Springer-Verlag.

21.

Nieweglowski

(2009). clv: Cluster validation techniques (R package version 0.3-2.1). Retrieved from http://CRAN.R-project.org/package=clv

22.

Preinerstorfer

Formann

A. K.

(2011). Parameter recovery and model selection in mixed Rasch models. British Journal of Mathematical and Statistical Psychology, 65, 251-262.

23.

R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

24.

Rand

W. M.

(1971). Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66, 846-850.

25.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Chicago, IL: University of Chicago Press.

26.

Rost

(1990). Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement, 14, 271-282.

27.

Rost

(1991). A logistic mixture distribution model for polychotomous item responses. British Journal of Mathematical and Statistical Psychology, 44, 75-92.

28.

Rost

von Davier

(1995). Mixture distribution Rasch models. In Fischer

G. H.

Molenaar

I. W.

(Eds.), Rasch models: Foundations, recent developments, and applications (pp. 257-268). New York, NY: Springer-Verlag.

29.

Schwarz

(1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.

30.

Strobl

Kopf

Zeileis

(2013). Rasch trees: A new method for detecting differential item functioning in the Rasch model. Psychometrika. Advance online publication. doi:10.1007/s11336-013-9388-3

31.

von Davier

Rost

(1995). Polytomous mixed Rasch models. In Fischer

G. H.

Molenaar

I. W.

(Eds.), Rasch models: Foundations, recent developments, and applications (pp. 371-379). New York, NY: Springer-Verlag.

32.

Zickar

M. J.

Gibby

R. E.

Robie

(2004). Uncovering faking samples in applicant, incumbent, and experimental data sets: An application of mixed-model item response theory. Organizational Research Methods, 7, 168-190.