Detecting Differential Item Functioning of Polytomous Items for an Ideal Point Response Process

Abstract

There has been growing use of ideal point models to develop scales measuring important psychological constructs. For meaningful comparisons across groups, it is important to identify items on such scales that exhibit differential item functioning (DIF). In this study, the authors examined several methods for assessing DIF on polytomous items generated by an ideal point process. Two paradigms (i.e., null hypothesis significance testing [NHST] and effect size quantification) were utilized, and three test statistics (i.e., the log-likelihood ratio [LR], the Akaike information criterion [AIC], and Lord’s chi-square) and two approaches to DIF testing (i.e., the constrained and free baseline methods) were evaluated. In addition, the authors investigated three levels of impact. The results revealed that DIF effect sizes were moderately large for the .50 uniform DIF conditions and small for nonuniform DIF; moreover, the LR test in general yielded the best results. When there was small to moderate impact, the free baseline approach combined with an item linking implementation produced the most satisfactory results.

Keywords

item response theory ideal point model differential item functioning the generalized graded unfolding model (GGUM)

The measurement of individual differences is fundamentally important for psychological research, educational and clinical decision making (R. J. Cohen & Swerdlik, 2002), and personnel selection (Hough & Oswald, 2000). Accurately measuring and comparing individual differences underpins individual difference theory (Ackerman & Humphreys, 1990), and these processes are vital to the study of domains such as job performance (Motowidlo, Borman, & Schmit, 1997) and work motivation (Mitchell & Daniels, 2003). Recent advancements in measurement science have converged on the notion that self-reported typical behaviors (e.g., personality, attitudes, and vocational interests) are most appropriately characterized by an ideal point response process (Drasgow, Chernyshenko, & Stark, 2010a, 2010b; Roberts, Laughlin, & Wedell, 1999; Stark, Chernyshenko, Drasgow, & Williams, 2006; Tay & Drasgow, 2012; Tay, Drasgow, Rounds, & Williams, 2009).

Ideal point response processes, also termed the unfolding technique (Coombs, 1964), contrast sharply with dominance response processes and their associated traditional psychometric models that include classical test theory, factor analysis, and logistic item response theory (IRT). Dominance models assume that the probability of a correct or affirmative response to an item is positively related to (θ_j−b_i) whereas ideal point models assume that the probability is negatively related to | θ_j−b_i |, where θ_j denotes the standing of person j on the latent trait and b_i denotes the extremity of item i. Thus, ideal point models posit that individuals are most likely to endorse items that are closest to their latent trait standing, and that the probability of endorsement is nonmonotonic and decreasing as the item’s location becomes more discrepant from the person’s standing on the latent trait.

The rationale and importance of using ideal point models to characterize self-reported typical behaviors converge from both theoretical and empirical perspectives. Theoretically, Thurstone (1928) argued that individuals use introspective cognitive processes when answering items, asking themselves, “Does this statement closely describe me?” Therefore, the closer an item’s location on the latent trait continuum to the individual’s standing, the greater the probability the individual endorses the item, and the probability of endorsing an item decreases as items’ locations are further away from an individual’s ideal point (Thurstone, 1929). For example, the item “Compulsory military training in all countries should be reduced but not eliminated” from the Thurstone–Droba War scale (cited by Likert, 1932, p. 34) might be disagreed with by both a strong militant and a strong pacifist. Empirically, Chernyshenko, Stark, Chan, Drasgow, and Williams (2001) fit several IRT models to data from the Sixteen Personality Factor (16PF) Questionnaire and found the best model was not monotonic, which is the hallmark of dominance models, but, instead, found nonmonotonicities in the probabilities of endorsement, suggesting an ideal point response process. In addition, a study by Carter and Dalal (2010) has demonstrated that, because of its unique flexibility, an ideal point model fits work satisfaction data better than dominance models. Tay, Ali, Drasgow, and Williams (2011) further conducted a simulation study and found that ideal point models are not simply more flexible (e.g., fitting many types of response processes): A goodness-of-fit index had substantial power to detect model misspecification when fitting an ideal point model to dominance data and vice versa.

If an ideal point model is required to appropriately characterize responses to assessments of psychological constructs using introspection, it is crucial to devise methods to assess whether items measure similarly across relevant groups. For legal and scientific reasons, measurement equivalence has been on the center stage of psychological measurement research (Thissen & Wainer, 2001). Common measurement equivalence procedures—such as multigroup confirmatory factor analysis (CFA) and differential item functioning (DIF)—are often used to ensure that group differences on individual difference measures are attributable to actual differences in the trait assessed (Stark, Chernyshenko, & Drasgow, 2004, 2006) rather than a flaw in the assessment instrument.

A recent study by Carter and Zickar (2011) initiated research on the evaluation of DIF for ideal point models (see also Roberts & Gordon, 2008). These authors examined a log-likelihood ratio (LR) method and Raju’s differential functioning of items and tests (DFIT; Raju, van der Linden, & Fleer, 1995) and found the LR approach to function more effectively.

Many fundamental questions about DIF detection with ideal point models remain unanswered. For example, how do different approaches to DIF assessment (e.g., the constrained baseline and free baseline approaches) and different statistical testing methods (e.g., the log-likelihood test, Akaike’s information criterion [AIC], and Lord’s chi-square) influence the efficacy of DIF detection? Are different forms of DIF (uniform and nonuniform DIF) detectable? What is the effect of impact (i.e., mean differences in the trait across reference and focal groups)? What is the magnitude of DIF when it is quantified by a statistic like Cohen’s d?

To begin answering these important questions, the authors utilized two paradigms: (a) the null hypothesis significance testing (NHST) paradigm and (b) the effect size quantification paradigm. NHST has been the most common approach to the study of DIF. Here two models—a null model and an alternative model—are constructed and compared, and a test statistic is computed. If the test statistic is statistically significant, the null hypothesis is rejected and the studied item is flagged for DIF. The authors adopted two approaches to constructing NHST models, the constrained baseline approach and free baseline approach, and used three test statistics,¹ the LR test, the AIC (Akaike, 1973, 1974), and Lord’s (1980) chi-square test. In addition, their DIF detection performance was examined in situations where there was impact.

An alternative to NHST is the effect size paradigm where the magnitude of differences is quantified (J. Cohen, 1992). For DIF, an effect size measure indexes the magnitude of differential functioning between reference and focal populations (Meade, 2010). A compelling virtue of the use of DIF effect size is that it is independent of sample size, and thus, large Ns do not inevitably lead to many significant differences.

In the following, the authors begin by introducing the most commonly used ideal point IRT model—the generalized graded unfolding model (GGUM)—and then discuss the approaches to DIF testing with the NHST paradigm. Next, they elaborate on the effect size paradigm and introduce the DIF quantification method utilized in this study. Then these approaches are used in a simulation study.

The GGUM and DIF Testing Models

Although multiple ideal point IRT models have been proposed, including the hyperbolic cosine latent trait model (Andrich & Luo, 1993) and the normal probability density function model (Maydeu-Olivares, Hernandez, & McDonald, 2006), the GGUM (Roberts, Donoghue, & Laughlin, 2000, 2002) is perhaps the most widely used. The GGUM has the form,

P [Z_{i} = z | θ_{j}] = \frac{\exp (α_{i} [z (θ_{j} - δ_{i}) - \sum_{k = 0}^{z} τ_{ik}]) + \exp (α_{i} [(M - z) (θ_{j} - δ_{i}) - \sum_{k = 0}^{z} τ_{ik}])}{\sum_{w = 0}^{C} \exp (α_{i} [w (θ_{j} - δ_{i}) - \sum_{k = 0}^{w} τ_{ik}]) + \exp (α_{i} [(M - w) (θ_{j} - δ_{i}) - \sum_{k = 0}^{w} τ_{ik}])},

where i denotes the ith item; Z_i represents a random variable denoting the response to the ith item; z = 0, 1, 2, … , C, where z is the observed response, 0 represents the strongest level of disagreement, and C represents the strongest level of agreement; θ_j denotes the location of jth individual on the latent continuum; δ_i represents the location of the ith item on the latent continuum; α_i represents the discrimination of the ith item; τ_ik denotes the kth subjective category threshold parameter associated with the ith item; C is the number of observable response categories minus 1; and M = 2 ×C+ 1.

The authors discuss two approaches—the constrained baseline approach and free baseline approach—to constructing the LR test and the AIC test. Previous research has revealed that these approaches can yield very different results (Stark, Chernyshenko, & Drasgow, 2006).

The Constrained Baseline Approach

The constrained baseline approach has been commonly used for DIF analysis. This approach constrains the parameters of all items to be equal across reference and focal groups in its baseline model. The alternative model freely estimates the parameters of the studied item while holding the parameters of all other items equal across groups. Then, a test statistic (e.g., the LR) is computed from the baseline and the alternative models. If a given item truly has DIF, the difference of log-likelihood chi-square statistics is expected to be greater than a critical value, and the AIC value from the alternative model will be smaller than that from the baseline model.

The Free Baseline Approach

The free baseline approach assumes there may be DIF on some items and begins by freely estimating item parameters in the reference and focal groups. Its alternative model assumes no DIF on the studied item and constrains its parameters across the reference and focal groups. Then the log-likelihood chi-square statistic and AIC indices are calculated as in the constrained baseline approach (note that DIF is indicated when the AIC is smaller for the baseline model). Statistical theory (Maydeu-Olivares & Cai, 2006) indicates that the free baseline approach should be preferred. Moreover, a simulation study (Stark, Chernyshenko, & Drasgow, 2006) found much lower Type I error rates and higher power with the free baseline approach for dominance IRT and CFA models. However, no study has compared the two approaches for ideal point models.

Three Statistical Testing Methods for DIF Detection

LR Test

The LR test is a popular testing method for comparing models. This test is

\begin{matrix} l (\underline{x}) = - 2 log (\frac{{sup}_{γ_{0} \in Γ_{0}} f (X | γ_{0})}{{sup}_{γ_{A} \in Γ_{A}} f (X | γ_{A})}) = - 2 log (\frac{f (X | {\hat{γ}}_{0})}{f (X | {\hat{γ}}_{A})}) \\ = - 2 (log (f (X | {\hat{γ}}_{0})) - log (f (X | {\hat{γ}}_{A}))), \end{matrix}

where X denotes the data, f is the likelihood, γ is a vector of parameters with maximum likelihood estimate $\hat{γ}$ , and “−2” is included so that the distribution of the test statistic approximates a chi-square distribution with degrees of freedom (df) equal to the difference of the number of parameters between the two models. If the calculated test statistic is greater than its critical value, the studied item is considered to exhibit DIF. The LR test has been demonstrated to have high power for DIF detection in previous research (Thissen, Steinberg, & Wainer, 1993) and proved to be effective in DIF analysis (Edelen, Thissen, Teresi, Kleinman, & Ocepek-Welikson, 2006).

AIC

The AIC is an index of model fit based on information theory. According to Akaike (1973, 1974),

AIC = - 2 \ln (L_{m}) + 2 h .

Here, L_m is the maximum likelihood of the estimated model and h is the number of parameters estimated for the model. The AIC index is typically used to evaluate the fit of alternative models. In this study, the authors utilized the AIC index for NHST: The AIC indices of two models (i.e., a null model and an alternative model) were compared to determine whether a model assuming DIF was a better fit. Because a smaller AIC index indicates better fit, the studied item was identified as containing DIF if the AIC index of the DIF model was smaller. To the best of our knowledge, there has been little research examining the use of the AIC index to detect DIF and its effectiveness has not previously been explored. The authors included this method because the AIC index is conveniently generated by GGUM and it has a computational advantage if it is proven to be effective for DIF analysis.

Lord’s Chi-Square

Lord’s (1980) chi-square method is well known and widely used for assessing DIF. For this method, item parameters are first estimated separately for each group, and the latent trait scales are linked so that all the parameters are put on the same metric. Then item parameter estimates for different groups are compared; a significant difference between estimates indicates DIF. Lord’s chi-square method has been extensively studied for dominance models and has been found to be robust and powerful. For example, by conducting simulation studies, McLaughlin and Drasgow (1987) and Lim and Drasgow (1990) found that Lord’s chi-square produced accurate DIF results when marginal maximum likelihood or Bayes modal estimates were used. In an empirical study with a 45-item vocabulary test, Raju, Drasgow, and Slinde (1993) found that Lord’s chi-square test yielded DIF detection results that closely agreed with the results of other methods such as the Mantel–Haenszel (MH), the signed area measure, and the unsigned area measure. Indeed, Donoghue and Isham (1998) reported Lord’s chi-square test as the most effective in identifying DIF items. However, little is known about the robustness and power of this method for ideal point models. One of the objectives was to compare Lord’s chi-square test with the LR test and AIC method.

The Impact Issue for Ideal Point Models

The impact issue is an important consideration when assessing DIF detection. It occurs when there is a mean difference between the reference and focal groups. Donoghue, Holland, and Thayer (1993) and Mullis, Dossey, Owen, and Phillips (1993) reported that it is quite common for the reference and focal groups to have a mean discrepancy of one standard deviation. The unequal latent means between two groups may pose a problem for DIF detection. This problem has been carefully studied for dominance models. For example, Chang, Mazzeo, and Roussos (1996) simulated a reference sample from a N(0,1) distribution and a focal sample from a N(−1,1) distribution. Their Simultaneous Item Bias Test (SIBTEST) procedure yielded appropriately small Type I error rates for the impact conditions. Using similar manipulations, but with the MH method, Wang and Su (2004a) found small Type I error rates in the impact conditions for the one-parameter logistic (1PL) model but inflated Type I error rates for the two-parameter logistic (2PL) and three-parameter logistic (3PL) models. It has been found that under impact conditions, Type I error rates increase as the number of parameters increases, and the common DIF assessment methods (e.g., MH, SIBTEST, and LR test) perform better when there is no impact than when there is impact (Finch, 2005; Wang & Su, 2004a, 2004b).

The authors suspect that the impact issue is prevalent for constructs that require ideal point models. These constructs include a variety of personality traits, attitudes, and vocational interests, which may vary substantially among subpopulations (e.g., genders, cultures, socioeconomic status [SES] classes, age subgroups). For example, Westerners are significantly more extraverted than non-Westerners, and Hong Kong Chinese and Japanese are significantly more neurotic than mainland Chinese and South Koreans, indicating that subcultures can differ even in the same geographic area (McCrae et al., 2010). However, the common DIF detection methods have not been examined under impact situations for ideal point models. By incorporating impact conditions in this study, the authors sought to examine DIF detection under a range of conditions commonly encountered.

DIF Effect Size

Although the NHST paradigm has been widely used to detect DIF, it does not come without criticism. One criticism is that significance tests only provide a yes-or-no result and thereby fail to provide useful information such as the magnitude, value, or importance of an effect (Kirk, 2006). Importantly, the result of a significance test is highly dependent on the sample size: Even a trivial effect can reach a statistical significance if the sample size is big enough.

The effect size paradigm overcomes these limitations by quantifying the magnitude of effect. In this study, two concerns were taken into consideration when the authors selected a measure of DIF effect size. First, the DIF areas for uniform DIF of ideal point models are almost completely symmetric with opposite signs. The signed area and related effect size measures (e.g., A. S. Cohen, Kim, & Baker, 1993; Penfield, 2010; Raju, 1988) will be close to zero due to cancellation. Second, because this study examined several forms of DIF, a calculation method is preferred that generates a standardized effect size that can be compared across conditions.

With these two considerations in mind, the authors selected a new method developed by Nye (2011). The Nye method first squares the difference between conditional expected scores, which avoids the cancellation problem and thus it is sensitive to different forms of DIF. More importantly, the Nye method standardizes the DIF effect size by dividing it by the pooled standard deviation of the reference and focal groups’ responses to a given item. This standardization puts all the DIF effect sizes on the same metric and this statistic—like Cohen’s d—is comparable across different DIF conditions and studies. This new method has been empirically tested in both dichotomous and polytomous dominance IRT contexts and has proved effective.

Nye’s (2011) DIF effect size² is

d_{DIF} = \frac{1}{S D_{iP}} \sqrt{{\int [E S_{iR} (θ) - E S_{iF} (θ)]}^{2} f_{F} (θ) dθ .}

Here, f_F(θ) is the normal distribution latent trait density function for the focal group; ES_iR(θ) and ES_iF(θ) are the expected scores for item i given a latent trait value of θ in the reference and focal groups, respectively, and are given by

E S_{iR} (θ) = \sum_{k = 0}^{C} x P_{iR} (x = k | θ)

and

E S_{iF} (θ) = \sum_{k = 0}^{C} x P_{iF} (x = k | θ)

for polytomous items that are scored x = 0, 1, … , C, and SD_iP is the pooled standard deviation of reference and focal groups and is used to standardize the effect. SD_iP is computed as

S D_{i P} = \sqrt{\frac{(N_{R} - 1) V a r_{i R} + (N_{F} - 1) V a r_{i F}}{N_{R} + N_{F} - 2}} .

Here, N_R and N_F are the sample sizes of the reference and focal groups, respectively; Var_iR and Var_iF are the variances in the reference and focal groups, respectively, and they are given by

Va r_{iR} = \int \sum_{k = 0}^{C} [{(k - E S_{iR} (θ))}^{2} P_{iR} (x = k | θ)] f_{R} (θ) dθ

and

Va r_{iF} = \int \sum_{k = 0}^{C} [{(k - E S_{iF} (θ))}^{2} P_{iF} (x = k | θ)] f_{F} (θ) dθ,

where P_iR(x = k|θ) and P_iF(x = k|θ) are the probabilities of an examinee in the reference and focal groups, respectively, with ability level θ selecting category k of item i. In this study, these probabilities were calculated using the GGUM model in Equation 1. In practice, this integral may be approximated numerically.

The Present Study

Using simulation procedures, the present study was designed to systematically understand DIF detection for ideal point IRT models under the NHST and effect size paradigms. Under the NHST paradigm, the authors examined three DIF detection methods and compared two approaches with DIF testing model construction. In addition, they investigated DIF detection under various conditions by manipulating five data characteristics: (a) type of DIF, (b) level of impact, (c) scale length, (d) percentage of DIF items, and (e) sample size per comparison group. The DIF effect size paradigm was used to overcome limitations of the NHST paradigm and help better understand DIF detection by offering a precise measure of DIF effect magnitude.

Method

Basic Simulation Setup for the GGUM Model

Item Parameter Generation

The authors followed the procedures by Roberts et al. (2002) for the basic model generation in the simulation, because they proposed the GGUM model and their simulation procedures have been followed in other studies (e.g., Tay et al., 2011), which facilitate comparisons across studies. Specifically, α_i was generated from a uniform distribution U(0.5,2). The location parameters, δ_I, were evenly distributed in the interval of [−2,2] with the first item and nth values always −2 and 2, respectively. The responses to each item had five categories: 0 to 4, where 0 represented the strongest disagreement and 4 represented the strongest agreement. Therefore, each item required four thresholds (τ_i1, τ_i2, τ_i3τ_i4); τ_i4 was generated from a uniform (−1.4,−0.4) distribution and the other thresholds (i.e., τ_i1, τ_i2, τ_i3) were calculated using Equation 10:

τ_{ik - 1} = τ_{ik} - 0.25 + e_{ik - 1}, for k = 4, 3, 2,

where e_ik_{− 1} represents a random error term generated from a N(0,0.04) distribution.

DIF Item Manipulations

The number of DIF items for the focal group was determined by the product of the scale length (10 or 20 items) and the percentage of DIF items in the scale (0%, 20%, or 40%). DIF items were randomly selected for each replication. For large and small uniform DIF conditions, δ_i values of the selected DIF items were incremented by .50 and .25, respectively; for nonuniform DIF conditions, α_i values of the selected DIF items were increased by .50.

Theta Generation and Impact Manipulation

In this study, all latent trait values for the reference group were sampled from a N(0,1) distribution. For the conditions of no impact, the latent trait values for the focal group were also sampled from a N(0,1) distribution. For the conditions of .25 SD and .50 SD impact, the latent trait values for the focal group were sampled, respectively, from N(−0.25,1) and N(−0.50,1) distributions.

Response Data Generation

After item parameters were generated for the reference and focal groups, Equation 1 was used to compute the probability that each category of a given item was endorsed by a respondent with a simulated theta value (θ_j). Then, the cumulative probability from the first category to the fifth category of an item (i.e., a vector $c = (P_{i 0}, \sum_{k = 0}^{1} P_{ik}, \sum_{k = 0}^{2} P_{ik}, \sum_{k = 0}^{3} P_{ik}, 1)$ ) was calculated and a number was randomly generated from a uniform distribution U(0,1). The simulated response to an item was determined by the location of the randomly generated cumulative probability vector.

Constructing DIF Tests

Constrained Baseline Analysis

Here, item parameter estimates were constrained across the reference and focal groups. The corresponding alternative model was similar to the baseline model except that the studied DIF item was freely estimated. Each item in turn was compared with the baseline model to determine whether the studied item contained DIF by using the LR test and the AIC (Roberts & Gordon, 2008).

Free Baseline Analysis

In this analysis, the responses were separated from the reference and focal groups, and item parameters were freely estimated across the two groups. Next, each studied item was constrained to be equal across the two groups and DIF was evaluated by the LR test and the AIC (Stark, Chernyshenko, and Drasgow, 2006). This worked appropriately for conditions with no impact, because fitting models with the latent trait distributions specified as N(0,1) place item parameter estimates on the same metric. However, when there was impact between the reference and focal groups, it was necessary to take steps to place parameter estimates on the same scale. To do so, the authors used linking items (i.e., anchor items) in the estimation process. Specifically, 1, 2, or 4 non-DIF items were constrained to be equal across groups to link the latent trait metrics.

The linking items were randomly selected from non-DIF items. Random selection was preferred because a preliminary study showed that the DIF effect size was largely independent of theta location, and in practice, it is difficult to know a priori the characteristics of the linking items; thus, random selection improves the results’ generalizability. To implement 1-item linking, one item was randomly selected from the items with positive delta (δ_i) values (because the deltas were symmetric around 0, the results would be the same regardless of whether the item was selected from the negative or positive delta items).³ For 2- and 4-item linkings, half of the items were randomly selected from the negative delta items and randomly selected the other half of the items from the positive delta items. After selecting the linking item(s), a certain number (i.e., the product of the DIF percentage and scale length) of items were then randomly selected from the remaining items for the DIF manipulation.

Lord’s Chi-Square Method

After generating response data for the reference and focal groups, the authors first ran GGUM2004 (Version 1.1; Roberts & Shim, 2008) separately for each group. Then they used GGUMLINK (Version 1.0; Roberts, 2002) to link item parameters of the focal group to the reference group. Then Lord’s chi-square statistic was computed.

Calculating DIF Effect Sizes

The calculation of DIF effect size followed Equation 4. A preliminary study showed that the item-level DIF effect size was independent of scale length and sample size. Therefore, the authors only report DIF effect sizes for the 10-item scale in nine conditions: three types of DIF (i.e., .50 uniform DIF, .25 uniform DIF, and .50 nonuniform DIF) × three levels of impact (i.e., 0 SD, .25 SD, and .50 SD).

Summary

For the NHST approach, the authors investigated six factors: type of DIF (no DIF, nonuniform DIF, large uniform DIF, and small uniform DIF) × level of impact (no impact, .25 SD impact, and .5 SD impact) × scale length (10 and 20 items) × percentage of DIF items (0%, 20%, and 40%) × sample size (250, 500, and 1,000) × testing method (LR, AIC, and Lord’s chi-square). In addition, the constrained and free baseline approaches were nested in the log-likelihood test and AIC methods, and three forms of anchor items (1-, 2-, and 4-linking items) were nested under the .25 SD and .5 SD impact levels for the log-likelihood test and AIC methods. For the effect size paradigm, the authors examined two factors: type of DIF (no DIF, nonuniform DIF, large uniform DIF, and small uniform DIF) × level of impact (no impact, .25 SD impact, and .5 SD impact). The authors replicated 100 times for each condition in the NHST paradigm and replicated 1,000 times in the effect size paradigm.

A shell script was written in the R programming language (R Development Core Team, 2011) to automate the entire processes, including data generation, DIF model construction, parameter estimation, output extraction, and DIF effect size calculation. In the item estimation process, the initial signs—which were determined by the simulated delta values of each item before the DIF manipulation—were added to improve estimation accuracy (see Roberts & Shim, 2008).

Analysis

Analysis for the NHST

The analyses for the NHST primarily focused on Type I error rates and power. The Type I error rate was calculated as the percentage of items that were falsely flagged as DIF items among the non-DIF items across 100 replications for each condition. Similarly, power was calculated as the percentage of items that were correctly flagged as DIF items among all the truly DIF items over the 100 replications.

LR Test

In this study, the df for the χ² statistic was 6. For a significance level of .05, the critical value of the χ² statistic is $χ_{. 95}^{2} (6)$ = 12.59.

Akaike Information Criterion (AIC)

A smaller AIC value indicates a better data-model fit when jointly considering fit and parsimony. For the constrained baseline approach, a studied item was flagged for DIF when the AIC value from the baseline/null model was greater than its corresponding alternative model (i.e., AIC_baseline− AIC_alternative > 0); for the free baseline approach, a studied item was flagged for DIF when the AIC value from the baseline/null model was smaller than its corresponding alternative model (i.e., AIC_baseline− AIC_alternative < 0).

Lord’s Chi-Square Analysis

Calculation of the chi-square statistic for each item strictly followed Lord (1980) and an item was flagged for DIF if the obtained chi-square statistic was greater than 12.59.

Inferential Statistical Analysis

Besides the descriptive analyses, the authors also conducted inferential statistical analyses of Type I error rates and power to better understand the effects of different factors on the dependent variables. Specifically, they adopted De Ayala’s (De Ayala & Sava-Bolesta, 1999) method to analyze Type I error rates and power to examine the proportion of the variability accounted for by different factors.

Analysis for the DIF Effect Size

Across 1,000 replications, effect sizes were computed when each item was subjected to a DIF manipulation. Means and standard deviations of the effect sizes were then computed for each item in each condition. For each condition, the effect sizes for each item were about the same (as suggested by the small standard deviations of the 10 DIF effect sizes), so the authors computed the grand mean of all the 10 items and used the grand mean as the DIF effect size index for each condition.

Results

The results of the significance testing methods are summarized in this section and partly presented in Tables 1 to 3.⁴ DIF effect sizes were examined for nine DIF conditions, and each condition was based on 1,000 replications. The results for DIF effect sizes are presented in Table 4.

Table 1.

Type I Error Rates and Power of DIF Detection When There Was No Impact.

Type of DIF	Scale length	DIF%	n	Three DIF detection methods
				Log-likelihood ratio				AIC				Lord’s chi-square
				Type I		Power		Type I		Power		Type I	Power
No DIF	10	0 (n = 0)	250	.06	(.06)	—	—	.07	(.06)	—	—	.04	—
			500	.05	(.03)	—	—	.06	(.04)	—	—	.05	—
			1,000	.03	(.04)	—	—	.04	(.06)	—	—	.05	—
	20	0 (n = 0)	250	.05	(.05)	—	—	.07	(.08)	—	—	.05	—
			500	.04	(.06)	—	—	.05	(.06)	—	—	.05	—
			1,000	.05	(.05)	—	—	.06	(.05)	—	—	.05	—
Uniform DIF^a	10	20 (n = 2)	250	.14	(.06)	0.96	(0.75)	.16	(.07)	0.97	(0.77)	.06	.40
			500	.26	(.06)	0.99	(0.97)	.28	(.06)	0.99	(0.97)	.07	.56
			1,000	.38	(.04)	1.00	(1.00)	.41	(.05)	1.00	(1.00)	.16	.71
		40 (n = 4)	250	.44	(.05)	0.83	(0.68)	.48	(.06)	0.85	(0.70)	.12	.23
			500	.69	(.04)	0.95	(0.97)	.72	(.04)	0.95	(0.97)	.26	.46
			1,000	.89	(.06)	0.99	(1.00)	.90	(.07)	1.00	(1.00)	.44	.61
	20	20 (n = 4)	250	.14	(.07)	0.92	(0.58)	.15	(.08)	0.93	(0.59)	.06	.41
			500	.23	(.04)	1.00	(0.94)	.25	(.05)	1.00	(0.94)	.11	.62
			1,000	.39	(.04)	1.00	(1.00)	.42	(.05)	1.00	(1.00)	.20	.74
		40 (n = 8)	250	.45	(.05)	0.80	(0.61)	.48	(.05)	0.82	(0.64)	.18	.31
			500	.68	(.05)	0.96	(0.94)	.70	(.05)	0.96	(0.94)	.33	.48
			1,000	.90	(.03)	0.99	(1.00)	.91	(.04)	0.99	(1.00)	.51	.67
Nonuniform DIF	10	20 (n = 2)	250	.05	(.07)	0.28	(0.30)	.07	(.09)	0.31	(0.32)	.03	.08
			500	.05	(.05)	0.55	(0.57)	.06	(.07)	0.57	(0.58)	.04	.33
			1,000	.07	(.06)	0.76	(0.78)	.08	(.08)	0.80	(0.80)	.07	.58
		40 (n = 4)	250	.05	(.05)	0.37	(0.31)	.06	(.06)	0.39	(0.33)	.04	.10
			500	.06	(.05)	0.59	(0.50)	.08	(.07)	0.61	(0.52)	.04	.32
			1,000	.07	(.05)	0.84	(0.75)	.08	(.07)	0.85	(0.76)	.05	.55
	20	20 (n = 4)	250	.06	(.05)	0.32	(0.38)	.07	(.06)	0.36	(0.41)	.03	.17
			500	.05	(.06)	0.58	(0.54)	.06	(.07)	0.61	(0.57)	.05	.44
			1,000	.05	(.06)	0.85	(0.83)	.07	(.07)	0.86	(0.85)	.06	.71
		40 (n = 8)	250	.05	(.06)	0.38	(0.34)	.07	(.07)	0.42	(0.37)	.03	.18
			500	.06	(.05)	0.62	(0.58)	.07	(.07)	0.64	(0.61)	.05	.36
			1,000	.06	(.05)	0.87	(0.84)	.07	(.07)	0.89	(0.85)	.06	.67

Note: DIF = differential item functioning; AIC = Akaike information criterion. The numbers without parentheses were produced by the constrained baseline approach, and the numbers in parentheses were produced by the free baseline approach.

The small uniform DIF with .25 manipulation is omitted here due to space limitation. The complete table is available from the first author upon request.

Table 2.

Results of DIF Detection When the Impact Was .25 SD^a.

Type of DIF	Scale length	DIF%	n	Three DIF detection methods
				Log-likelihood ratio				AIC				Lord’s chi-square
				Type I		Power		Type I		Power		Type I	Power
No DIF	10	0 (n = 0)	250	.05	(.07)	—	—	.06	(.09)	—	—	.04	—
			500	.06	(.06)	—	—	.07	(.07)	—	—	.05	—
			1,000	.06	(.11)	—	—	.07	(.12)	—	—	.06	—
	20	0 (n = 0)	250	.06	(.05)	—	—	.08	(.07)	—	—	.04	—
			500	.06	(.06)	—	—	.07	(.07)	—	—	.04	—
			1,000	.05	(.08)	—	—	.05	(.10)	—	—	.06	—
Uniform DIF^a	10	20 (n = 2)	250	.10	(.07)	0.97	(0.97)	.13	(.08)	0.98	(0.97)	.06	.31
			500	.20	(.08)	1.00	(1.00)	.21	(.10)	1.00	(1.00)	.09	.51
			1,000	.29	(.10)	1.00	(1.00)	.30	(.11)	1.00	(1.00)	.15	.61
		40 (n = 4)	250	.35	(.04)	0.85	(0.98)	.37	(.06)	0.85	(0.98)	.14	.19
			500	.60	(.08)	0.98	(1.00)	.61	(.09)	0.98	(1.00)	.25	.35
			1,000	.81	(.07)	1.00	(1.00)	.83	(.10)	1.00	(1.00)	.42	.51
	20	20 (n = 4)	250	.12	(.09)	0.93	(0.97)	.14	(.09)	0.93	(0.98)	.07	.43
			500	.19	(.09)	0.99	(1.00)	.22	(.11)	0.99	(1.00)	.10	.54
			1,000	.32	(.10)	1.00	(1.00)	.35	(.11)	1.00	(1.00)	.18	.69
		40 (n = 8)	250	.41	(.07)	0.80	(0.98)	.44	(.08)	0.82	(0.98)	.16	.28
			500	.68	(.07)	0.96	(1.00)	.71	(.09)	0.96	(1.00)	.32	.44
			1,000	.86	(.09)	1.00	(1.00)	.87	(.11)	1.00	(1.00)	.50	.59
Nonuniform DIF	10	20 (n = 2)	250	.09	(.06)	0.40	(0.35)	.10	(.07)	0.44	(0.36)	.03	.12
			500	.07	(.06)	0.58	(0.60)	.08	(.07)	0.59	(0.65)	.04	.28
			1,000	.06	(.08)	0.84	(0.81)	.07	(.10)	0.85	(0.83)	.06	.57
		40 (n = 4)	250	.03	(.07)	0.39	(0.33)	.04	(.08)	0.41	(0.35)	.04	.11
			500	.05	(.07)	0.62	(0.58)	.07	(.09)	0.65	(0.60)	.04	.31
			1,000	.06	(.06)	0.81	(0.82)	.09	(.07)	0.84	(0.83)	.08	.59
	20	20 (n = 4)	250	.05	(.06)	0.35	(0.39)	.07	(.07)	0.38	(0.42)	.04	.19
			500	.04	(.07)	0.61	(0.66)	.05	(.08)	0.63	(0.69)	.06	.40
			1,000	.04	(.09)	0.85	(0.90)	.05	(.10)	0.87	(0.90)	.07	.71
		40 (n = 8)	250	.05	(.08)	0.44	(0.40)	.07	(.09)	0.47	(0.44)	.04	.20
			500	.06	(.07)	0.65	(0.61)	.07	(.09)	0.69	(0.65)	.05	.42
			1,000	.08	(.09)	0.87	(0.88)	.10	(.10)	0.89	(0.90)	.07	.71

Note: DIF = differential item functioning; AIC = Akaike information criterion. The complete table is available from the first author upon request.

The results were based on 4-item linking conditions; the numbers without parentheses were produced by the constrained baseline approach, and the numbers in parentheses were produced by the free baseline approach.

Table 3.

Results of DIF Detection When the Impact Was .50 SD^a.

Type of DIF	Scale length	DIF%	n	Three DIF detection methods
				Log-likelihood ratio				AIC				Lord’s chi-square
				Type I		Power		Type I		Power		Type I	Power
No DIF	10	0 (n = 0)	250	.08	(.07)	—	—	.09	(.09)	—	—	.02	—
			500	.07	(.14)	—	—	.09	(.17)	—	—	.05	—
			1,000	.09	(.25)	—	—	.09	(.28)	—	—	.06	—
	20	0 (n = 0)	250	.06	(.11)	—	—	.07	(.12)	—	—	.05	—
			500	.06	(.11)	—	—	.07	(.14)	—	—	.05	—
			1,000	.07	(.26)	—	—	.08	(.29)	—	—	.06	—
Uniform DIF^a	10	20 (n = 2)	250	.07	(.10)	.94	(.95)	.09	(.11)	.94	(.96)	.06	.24
			500	.10	(.15)	1.00	(.99)	.12	(.17)	1.00	(.99)	.10	.40
			1,000	.21	(.29)	1.00	(1.00)	.24	(.31)	1.00	(1.00)	.15	.49
		40 (n = 4)	250	.29	(.12)	.85	(.96)	.31	(.15)	.85	(.96)	.14	.22
			500	.47	(.11)	.95	(1.00)	.48	(.13)	.96	(1.00)	.27	.35
			1,000	.72	(.27)	1.00	(1.00)	.74	(.30)	1.00	(1.00)	.42	.46
	20	20 (n = 4)	250	.11	(.11)	.91	(.93)	.13	(.13)	.92	(.93)	.06	.35
			500	.18	(.17)	1.00	(1.00)	.21	(.18)	1.00	(1.00)	.11	.48
			1,000	.30	(.24)	1.00	(1.00)	.33	(.25)	1.00	(1.00)	.16	.69
		40 (n = 8)	250	.37	(.12)	.81	(.96)	.39	(.14)	.83	(.97)	.18	.23
			500	.63	(.16)	.95	(.98)	.66	(.18)	.95	(.98)	.30	.41
			1,000	.85	(.27)	1.00	(1.00)	.86	(.30)	1.00	(1.00)	.46	.53
Nonuniform DIF	10	20 (n = 2)	250	.04	(.09)	.31	(.41)	.06	(.09)	.35	(.45)	.03	.14
			500	.06	(.12)	.65	(.57)	.08	(.14)	.69	(.60)	.05	.28
			1,000	.11	(.23)	.87	(.86)	.12	(.24)	.88	(.88)	.07	.58
		40 (n = 4)	250	.04	(.09)	.38	(.39)	.05	(.10)	.41	(.42)	.02	.10
			500	.10	(.12)	.67	(.66)	.12	(.13)	.69	(.68)	.04	.29
			1,000	.11	(.23)	.87	(.91)	.12	(.26)	.88	(.92)	.07	.55
	20	20 (n = 4)	250	.07	(.12)	.37	(.44)	.09	(.12)	.39	(.46)	.04	.16
			500	.05	(.12)	.66	(.70)	.06	(.14)	.71	(.73)	.05	.43
			1,000	.06	(.24)	.84	(.92)	.08	(.27)	.87	(.93)	.07	.72
		40 (n = 8)	250	.07	(.09)	.40	(.44)	.08	(.10)	.44	(.46)	.04	.17
			500	.07	(.12)	.62	(.70)	.08	(.14)	.65	(.73)	.05	.41
			1,000	.06	(.23)	.85	(.91)	.08	(.26)	.87	(.92)	.06	.69

Note: DIF = differential item functioning; AIC = Akaike information criterion. The complete table is available from the first author upon request.

Table 4.

DIF Effect Sizes for Different DIF Conditions for a Scale Length of 10 Items.

Impact	DIF types
Impact	.50 uniform M_grand (SD_grand, SD_item)	.25 uniform M_grand (SD_grand, SD_item)	.50 nonuniform M_grand (SD_grand, SD_item)
0 SD	.640 (.169, .017)	.324 (.086, .010)	.169 (.065, .004)
.25 SD	.640 (.172, .033)	.322 (.087, .016)	.169 (.064, .005)
.50 SD	.626 (.174, .047)	.319 (.087, .022)	.168 (.064, .004)

Note: DIF = differential item functioning; M_grand = mean of DIF effect sizes based on the 1,000 replications; SD_grand = standard deviation based on all the 10 items over the 1,000 replications; SD_item = standard deviation of the means of DIF effect sizes of the 10 items.

Results From the NHST

Comparison of the Constrained and Free Baseline Approaches and Influence of Impact

The results revealed that the free baseline approach generally outperformed the constrained baseline approach when there was no impact. For example, when uniform DIF was simulated (Table 1), the Type I error rates were generally well controlled at around .05 with the free baseline approach, but they were highly inflated with the constrained baseline approach, especially when the sample size was large. Specifically, as shown in Table 1, for large uniform DIF conditions, the Type I error rates provided by the free baseline approach ranged from .03 to .07 (M = .05). In contrast, the Type I error rates provided by the constrained baseline approach ranged from .14 to .90 (M = .47). Power was high for the free baseline approach (M = .87). Because it failed to control the Type I error rate, the power values for the constrained baseline approach are not interpretable.

However, when there was moderate impact (e.g., .25 SD), the constrained baseline approach outperformed the free baseline approach for the 1- and 2-item linking conditions, but the free baseline approach performed better than the constrained baseline approach for the 4-item linking conditions; when there was high impact (e.g., .50 SD), neither the free baseline approach nor the constrained baseline approach performed well, regardless of the number of linking items. One important finding was that the constrained baseline approach seemed to be more resistant to impact than the free baseline approach: The results for the constrained baseline approach were almost unchanged across Tables 1 to 3, whereas the results for the free baseline approach changed dramatically from no impact to large impact.

Overall, the results from Tables 1 to 3 showed that impact exerted a dramatic effect on Type I error rates and power. When there was no impact (Table 1), low Type I error rates and high power were generally obtained with the free baseline approach. However, it became more difficult to obtain satisfactory results with increased level of impact. When the impact increased to .25 SD (Table 2), Type I error rates dramatically increased, especially for the 1- and 2-item linking conditions. When the impact was .50 SD, the authors found that none of the methods provided satisfactory results (i.e., low Type I error rates and high power), especially for the large uniform DIF conditions: The log-likelihood test with the constrained baseline approach provided varying results for different DIF types and the constrained baseline approach produced unacceptably high Type I error rates for conditions of 20% and 40% DIF.

Comparison of the Three Significance Testing Methods

Looking through Tables 1 to 3, it appears that the LR test method performed the best, although for some conditions the AIC method performed as well as the LR test. For the constrained baseline approach, the power of the AIC for all the conditions was similar to the power of the LR test. However, Type I error rates by the AIC for all the conditions were slightly higher than those of the log-likelihood methods. The performance by the free and constrained baseline approaches under the AIC method showed the same pattern that was observed for the log-likelihood method.

The results for Lord’s chi-square method are presented in the right two columns in Tables 1 through 3. The results indicated that this method produced satisfactorily low Type I error rates for nonuniform DIF conditions but inflated Type I error rates for the uniform DIF conditions. The Type I error rates for uniform DIF conditions varied based on sample sizes, DIF percentages, and scale lengths. The pattern tended to be very similar to the pattern that was observed for the log-likelihood test and AIC methods with the constrained baseline approach; that is, Type I error rates tended to increase with sample size, scale length, and percentage of DIF item in the scale. Even with inflated Type I error rates, the power of Lord’s chi-square was substantially lower than the free baseline log-likelihood method.

Results From the Inferential Statistical Analysis

The inferential analysis examined the variability of the Type I error rates and power by the LR test method accounted for by six factors: level of impact, baseline (constrained and free), DIF type, DIF percentage, scale length, and sample size. All the two-way interaction terms were also included in the analysis. The results showed that impact accounted for the biggest proportion for both Type I error rates and power ( $ω_{impact}^{2} = 91.5 %$ for Type I error rate; $ω_{impact}^{2} = 44.2 %$ for power). DIF type and the baseline × DIF type interaction also accounted for relatively large proportions of Type I error rates ( $ω_{DIF type}^{2} = 12.3 %$ ; $ω_{baseline \times DIF type}^{2} = 11.1 %$ ), but not for power.⁵

DIF Effect Size

The results of the DIF effect sizes for the nine DIF conditions are presented in Table 4.⁶ The grand means (M_grand) and grand standard deviations (SD_grand) are presented for DIF effect sizes for each condition, followed by the standard deviations of the 10 items’ means of DIF effect sizes (SD_item). The grand means and standard deviations are based on all 10 items over the 1,000 replications (i.e., 10,000 DIF effect sizes).

The findings of the DIF effect sizes showed several patterns. First, uniform DIF had a much bigger DIF effect size than nonuniform DIF for the same DIF manipulation (e.g., .50); the DIF effect size for .50 nonuniform DIF was even smaller than the .25 uniform DIF (M_{.50 uniform} = .640; M_{.25 uniform} = .324; M_{.50 nonuniform} = .169). According to J. Cohen (1992), an effect size around .20 is considered a small effect size, around .50 is medium, and around .80 is large. Thus, the results indicated that the DIF effect sizes in this study were moderately large for the .50 uniform DIF conditions and were small for nonuniform DIF conditions. Second, interestingly, impact has almost no influence on DIF effect size. For each DIF type, increasing impact even decreased the DIF effect size very slightly (e.g., for .50 uniform conditions: M_{0 SD impact} = .640, M_{.25 SD impact} = .640, M_{.50 SD impact} = .626). Third, the standard deviations of the 10 items’ mean DIF effect sizes were quite small, ranging from .004 to .047, suggesting that the DIF effect size was largely independent of the items’ extremity and consistent across the entire latent trait continuum. This result also confirmed the appropriateness of randomly selecting items for DIF manipulations.

Discussion

As more and more researchers have seen the value of ideal point IRT models for psychological measurement, DIF assessment has become increasingly important. In light of this background, this study systematically assessed DIF for polytomous responses generated by an ideal point process. This study contributes to the existing DIF literature in several ways. This is the first study to quantify DIF effect size for unfolding models. The authors overcame the challenges of traditional method of calculating DIF effect size in the ideal point models and used a new method to investigate the DIF effect size under various conditions. By examining the quantification of DIF effect sizes for various data conditions, this study advances the understanding of DIF and its detection. Second, this is the first study that has investigated the impact problem for the GGUM model. As discussed previously, the impact problem is quite prevalent for psychological attributes that require ideal point measurement models. This study examined three levels of impact and investigated its influence on both DIF detection and DIF effect sizes. The authors found that impact severely affected DIF detection under the NHST paradigm, especially when the impact was bigger than .25 SD, although it had almost no influence on the DIF effect size. They also discovered that, when there was impact, the free baseline approach combined with an item linking implementation produced the most satisfactory results. These findings not only increase the knowledge regarding DIF phenomena for the GGUM model but also have great practical implications. Third, the multiple approaches to DIF testing and the 126 DIF conditions examined in the study extend the findings of the initial work of Carter and Zickar (2011). In their study, they examined 12 conditions through two popular DIF testing methods, the LR test and Raju’s DFIT, and found that the LR method was superior to DFIT. The LR method was compared with the AIC and Lord’s chi-square method using free and constrained baseline approaches under different levels of impact conditions with varying numbers of linking items. The authors found the AIC method was almost as good as the LR method when there was no impact or the impact was small. When the impact was .50 SD, none of the three methods performed well. In addition, the authors used De Ayala’s method (De Ayala & Sava-Bolesta, 1999) for further analyzing the effects of data characteristics (e.g., sample size, DIF types, DIF percentage, scale length) on DIF detection. DIF types and sample size are found to exert considerable greater influence than DIF percentage and scale length on DIF detection. These new findings have improved the understanding of DIF detection for ideal point models.

Applied Implications and Recommendations

Checking Ideal Point Models and Verifying Testing Assumptions

The present simulations were entirely based on ideal point IRT models for DIF detection. However, in reality, the most appropriate model may not be clear in advance and thus the authors recommend that researchers and practitioners check model fit before adopting the current methods for DIF analysis. Tay et al. (2011) presented procedures for assessing the relative fit of both ideal point and dominance models.

In addition to the parametric form, the ideal point model (or, at least, GGUM) assumes a unidimensional latent trait and, consequently, local independence. Checking the assumption of unidimensionality is difficult because factor analysis of ideal point data can produce artificial factors (Davison, 1977; Tay & Drasgow, 2012). At the very least, the survey items should be examined, making sure that all the items are answered independently without influence from the preceding items, that is, there are no order effects or priming effects.

The Impact Problem

One of the important findings of this study is the impact problem for DIF detection with GGUM. When there is impact, accurately identifying DIF items is especially critical because the group may result from measurement nonequivalence.

Findings of this study indicate that although impact did not affect the magnitude of the DIF effect size, it did undermine DIF detection. Especially when the impact was as large as .50 SD, none of the testing methods or linking strategy worked very effectively to detect DIF, which poses a crucial challenge for practitioners. Although this problem clearly requires further research, this study provides some recommendations before an effective solution is found. First, the authors recommend calculating the standardized mean difference of latent trait estimates between the two groups of interest. If the calculated impact is substantially greater than .25 SD, at present, there seems to be no effective method for DIF detection. However, if the impact is found to be of a more modest size, researchers may proceed by carefully selecting a few (e.g., 4) non-DIF items, using them as anchor items, and conducting a DIF analysis with the LR method and free baseline approach.

Of course, selecting non-DIF items is another challenge. However, this study seems to suggest using the constrained baseline approach is a good start to find non-DIF items. Note that the constrained baseline approach produced high power and inflated Type I error rates, therefore items classified as non-DIF by this approach are likely to be truly non-DIF. Thus, the authors recommend first utilizing the constrained baseline approach to identify a few non-DIF items, and then using the non-DIF items as linking items with the free baseline approach to detect DIF.

DIF Model Construction and Testing Methods

The present findings revealed that both the free baseline approach and the constrained baseline approach have strengths and weaknesses, although the free baseline approach seems generally superior to the constrained baseline approach, which confirms previous findings by Stark, Chernyshenko, and Drasgow (2006). Although the free baseline approach outperformed the constrained baseline approach when there was no impact, it seemed to be more susceptible to impact than the constrained baseline approach, which highlights an important advantage of the constrained baseline approach. Therefore, the constrained baseline approach might be useful for the situations where there exists large impact. Similarly, Lord’s chi-square method showed less susceptibility to impact. Thus, it may be also useful to utilize Lord’s chi-square method for DIF detection when there is substantial impact.

Limitations and Future Directions

Detecting DIF for ideal point models is a fairly new topic, and clearly more research is needed for researchers and practitioners to better understand this topic. In this study, the authors only investigated the GGUM model. It is possible that results may be different for other models, such as the normal probability density function model (Maydeu-Olivares et al., 2006). Moreover, although simulation is useful for assessing DIF detection methods, the authors acknowledge that not all the assumptions of the psychometric model may be satisfied in real situations. Thus, research on the robustness of DIF detection methods under assumption violation is clearly needed.

Having said that, the authors believe this study has identified important questions for future research. Perhaps the most important is the impact problem. This problem can be approached from several perspectives. First, as one of the reviewers speculated, GGUM2004 may have some metric indeterminacy issues, as it appeared to encounter problems in simultaneously estimating parameters of reference and focal groups when there was impact. To address this concern, the authors assigned delta signs for the items as advised by Dr. James Roberts and checked model fit. Nonetheless, further research into metric indeterminacy is needed. A second issue for further research concerns the number of item parameters estimated. Previous research has shown that Type I error rates increase as the number of parameters in the model increases when there is impact (e.g., Finch, 2005; Wang & Su, 2004a, 2004b). The GGUM involved six parameters per item in this study, which may have contributed to the impact problem. This speculation can be tested by fitting GGUM models with different numbers of response categories. A third topic for research focuses on the ideal point model itself. It is possible that the impact problem is not unique to the GGUM model but is a problem for other unfolding IRT models as well. Thus, DIF detection is needed to be examined with other ideal point models, such as the normal probability density function model (Maydeu-Olivares et al., 2006). Fourth, perhaps some strategy for different testing methods and linking procedure may work effectively for DIF detection under impact conditions. For example, iteratively linking the latent trait metrics (Candell & Drasgow, 1988) might be considered for Lord’s chi-square method.

Conclusion

Impact perhaps is the biggest barrier to DIF detection for ideal point models. When there was impact, the authors observed many false positives and consequently a significant DIF statistic may not truly indicate differential functioning. Therefore, it is important to calculate the standardized mean difference of latent trait estimates as a first step to gain insight about how subsequent analyses should be interpreted. If impact is close to zero, the LR test with the free baseline approach should prove effective; and if impact is small to moderate (e.g., .25 SD), the authors recommend using the free baseline approach with a few anchor items. Detecting DIF when impact is .50 SD or greater is problematic for the methods that were examined. Clearly, more research is needed, and it is believed this is an important direction for future directions.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Ackerman

P. L.

Humphreys

L. G.

(1990). Individual differences theory in industrial and organizational psychology. In Dunnette

M. D.

Hough

L. M.

(Eds.), Handbook of industrial and organizational psychology (pp. 223-282). Palo Alto, CA: Consulting Psychologist Press.

Akaike

(1973). Information theory and an extension of the maximum likelihood principle. In Petrov

B. N.

Csaki

(Eds.), Proceedings of the second international symposium on information theory (pp. 267-281). Budapest, Hungary: Akademiai Kiado.

Akaike

(1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716-723.

Andrich

Luo

(1993). A hyperbolic cosine latent trait model for unfolding dichotomous single-stimulus responses. Applied Psychological Measurement, 17, 253-276.

Candell

Drasgow

(1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253-260.

Carter

N. T.

Dalal

D. K.

(2010). An ideal point account of the JDI work satisfaction scale. Personality and Individual Differences, 49, 743-748.

Carter

N. T.

Zickar

(2011). A comparison of the LR and DFIT frameworks of differential functioning applied to the generalized graded unfolding model. Applied Psychological Measurement, 35, 623-642.

Chang

Mazzeo

Roussos

(1996). Detecting DIF for polytomously scored items: An adaptation of the SIBTEST procedures. Journal of Educational Measurement, 33, 333-353.

Chernyshenko

O. S.

Stark

Chan

Drasgow

Williams

(2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523-562.

10.

Cohen

A. S.

Kim

S.-H.

Baker

F. B.

(1993). Detection of differential item functioning in the graded response model. Applied Psychological Measurement, 17, 335-350.

11.

Cohen

(1992). A power primer. Psychological Bulletin, 112, 155-159.

12.

Cohen

R. J.

Swerdlik

(2002). Psychological testing and assessment: An introduction to tests and measurement. New York, NY: McGraw-Hill.

13.

Coombs

C. H.

(1964). A theory of data. New York, NY: Wiley.

14.

Davison

M. L.

(1977). On a metric, unidimensional unfolding model for attitudinal and developmental data. Psychometrika, 42, 523-548.

15.

De Ayala

R. J.

Sava-Bolesta

(1999). Item parameter recovery in the nominal response model. Applied Psychological Measurement, 23, 3-20.

16.

Donoghue

J. R.

Holland

Thayer

(1993). A Monte Carlo study of factors that affect the Mantel-Haenszel and standardization measures of differential item functioning. In Holland

P. W.

Wainer

(Eds.), Differential item functioning (pp. 137-166). Hillsdale, NJ: Erlbaum.

17.

Donoghue

J. R.

Isham

S. P.

(1998). A comparison of procedures to detect item parameter drift. Applied Psychological Measurement, 22, 33-51.

18.

Drasgow

Chernyshenko

O. S.

Stark

(2010a). Improving the measurement of psychological variables: Ideal point models rock!Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 515-520.

19.

Drasgow

Chernyshenko

O. S.

Stark

(2010b). 75 years after Likert: Thurstone was right!Industrial and Organizational Psychology: Perspectives on Science and Practice, 3, 465-476. doi:10.1111/j.1754-9434.2010.01273.x

20.

Edelen

M. O.

Thissen

Teresi

J. A.

Kleinman

Ocepek-Welikson

(2006). Identification of differential item functioning using item response theory and the likelihood-based model comparison approach: Application to the Mini-Mental State Examination. Medical Care, 44, S134-S142.

21.

Finch

(2005). The MIMIC model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29, 278-295.

22.

Hough

L. M.

Oswald

F. L.

(2000). Personnel selection: Looking toward the future—Remembering the past. Annual Review of Psychology, 2000, 631-664.

23.

Kirk

R. E.

(2006). Effect magnitude: A different focus. Journal of Statistical Planning and Inference, 137, 1634-1646. doi:10.1016/j.jspi.2006.09.011

24.

Likert

(1932). A technique for the measurement of attitudes. Archives of Psychology, 22, 1-55.

25.

Lim

R. G.

Drasgow

(1990). Evaluation of two methods for estimating item response theory parameters when assessing differential item functioning. Journal of Applied Psychology, 75, 164-174.

26.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.

27.

Maydeu-Olivares

Cai

(2006). A cautionary note on using G2 (dif) to assess relative model fit in categorical data analysis. Multivariate Behavioral Research, 41, 55-64.

28.

Maydeu-Olivares

Hernandez

McDonald

R. P.

(2006). A multidimensional ideal point item response theory model for binary data. Multivariate Behavioral Research, 41, 445-471.

29.

McCrae

R. R.

Terracciano

De Fruyt

De Bolle

Gelfand

M. J.

Costa

P. T.

(2010). The validity and structure of culture-level personality scores: Data from ratings of young adolescents. Journal of Personality, 78, 815-838.

30.

McLaughlin

M. E.

Drasgow

(1987). Lord’s chi-square test of item bias with estimated and with known person parameters. Applied Psychological Measurement, 11, 161-173.

31.

Meade

A. W.

(2010). A taxonomy of effect size measures for the differential functioning of items and scales. Journal of Applied Psychology, 95, 728-743.

32.

Mitchell

T. R.

Daniels

(2003). Motivation. In Borman

W. C.

Ilgen

D. R.

Klimoski

R. J.

(Eds.), Handbook of industrial and organizational psychology (pp. 225-254). Hoboken, NJ: John Wiley.

33.

Motowidlo

S. J.

Borman

W. C.

Schmit

M. J.

(1997). A theory of individual differences in task and contextual performance. Human Performance, 10, 71-83.

34.

Mullis

Dossey

Owen

Phillips

(1993). NAEP 1992 mathematics report card for the nation and the states. Washington, DC: National Center for Education Statistics.

35.

Nye

C. D.

(2011). The development and validation of effect size measures for IRT and CFA studies of measurement equivalence (Doctoral dissertation). University of Illinois at Urbana-Champaign, Champaign.

36.

Penfield

R. D.

(2010). Distinguishing between net and global DIF in polytomous items. Journal of Educational Measurement, 47, 129-149.

37.

R Development Core Team. (2011). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-project.org/

38.

Raju

N. S.

(1988). The area between two item characteristic curves. Psychometrika, 53, 495-502.

39.

Raju

N. S.

Drasgow

Slinde

J. A.

(1993).An empirical comparison of the area methods, Lords’ chi-square test, and the Mantel-Haenszel technique for assessing differential item functioning. Educational and Psychological Measurement, 53, 301-314.

40.

Raju

N. S.

van der Linden

W. J.

Fleer

P. F.

(1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19, 353-368.

41.

Roberts

J. S.

(2002). GGUMLINK version1.0 user’s manual. Atlanta: Georgia Polytechnic University.

42.

Roberts

J. S.

Donoghue

J. R.

Laughlin

J. E.

(2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24, 3-32.

43.

Roberts

J. S.

Donoghue

J. R.

Laughlin

J. E.

(2002).Characteristics of MML/EAP parameter estimates in the generalized graded unfolding model. Applied Psychological Measurement, 26, 192-207.

44.

Roberts

J. S.

Gordon

J. D.

(2008, March). Identification of differentially functioning items in the generalized graded unfolding model. Paper presented at the American Educational Research Association Meeting, New York, NY.

45.

Roberts

J. S.

Laughlin

J. E.

Wedell

D. H.

(1999). Validity issues in the Likert and Thurstone approaches to attitude measurement. Educational and Psychological Measurement, 59, 211-233.

46.

Roberts

J. S.

Shim

H. S.

(2008). GGUM2004 technical reference manual (Version 1.1). Atlanta: Georgia Polytechnic University.

47.

Stark

Chernyshenko

O. S.

Drasgow

(2004). Examining the effects of differential item (functioning and differential) test functioning on selection decisions: When are statistically significant effects practically important? Journal of Applied Psychology, 89, 497-508.

48.

Stark

Chernyshenko

O. S.

Drasgow

(2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91, 1292-1306.

49.

Stark

Chernyshenko

O. S.

Drasgow

Williams

B. A.

(2006). Examining assumptions about item responding in personality assessment: Should ideal point models be considered for scale development and scoring? Journal of Applied Psychology, 91, 25-39.

50.

Tay

Ali

U. S.

Drasgow

Williams

B. A.

(2011). Fitting simulated dichotomous and polytomous data: Examining the difference between ideal point and dominance models. Applied Psychological Measurement, 35, 280-295.

51.

Tay

Drasgow

(2012). Theoretical, statistical, and substantive issues in the assessment of construct dimensionality: Accounting for the item response process. Organizational Research Methods, 15, 363-384.

52.

Tay

Drasgow

Rounds

Williams

(2009). Fitting measurement models to vocational interest data: Are dominance models ideal? Journal of Applied Psychology, 94, 1287-1304.

53.

Thissen

Steinberg

Wainer

(1993). Detection of differential item functioning using the parameters of item response models. In Holland

P. W.

Wainer

(Eds.), Differential item functioning (pp. 67-113). Hillsdale, NJ: Erlbaum.

54.

Thissen

Wainer

(2001). Test scoring. Mahwah, NJ: Erlbaum.

55.

Thurstone

L. L.

(1928). Attitudes can be measured. American Journal of Sociology, 33, 529-554.

56.

Thurstone

L. L.

(1929). Theory of attitude measurement. Psychological Review, 36, 222-241.

57.

Wang

W.-C.

Y.-H.

(2004a). Effects of average signed area between two item characteristic curves and test purification procedures on the DIF detection via the Mantel-Haenszel method. Applied Measurement in Education, 17, 113-144.

58.

Wang

W.-C.

Y.-H.

(2004b). Factors influencing the Mantel and generalized Mantel-Haenszel methods for the assessment of differential item functioning in polytomous items. Applied Psychological Measurement, 28, 450-480.