How Does Polytomous Item Bias Affect Total-group Survey Score Comparisons?

Abstract

The growing use of scales in survey questionnaires warrants the need to address how does polytomous differential item functioning (DIF) affect observed scale score comparisons. The aim of this study is to investigate the impact of DIF on the type I error and effect size of the independent samples t-test on the observed total scale scores. A simulation study was conducted, focusing on potential variables related to DIF in polytomous items, such as DIF pattern, sample size, magnitude, and percentage of DIF items. The results showed that DIF patterns and the number of DIF items affected the type I error rates and effect size of t-test values. The results highlighted the need to analyze DIF before making comparative group interpretations.

Keywords

differential item functioning polytomous items type I error rates survey questionnaire item bias

Item bias and Differential Item Functioning (DIF) have been widely studied in psychological and educational testing since the 1980s. As indicated in the last edition of the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education 2014), there is a broad consensus about how DIF can undermine the validity of test score interpretations. As it is considered a threat to the validity of group comparisons based on test scores, psychometricians and statisticians have worked intensively to develop statistical tests, effect size measures, and criteria that not only identify DIF items but also help users make decisions on whether to keep or remove these DIF items from tests. For recent reviews on DIF research, readers should refer to the work of Hidalgo and Gómez-Benito (2010), Osterlind and Everson (2009), Sireci and Rios (2013), and Zumbo (2007).

The interest in item bias and DIF in survey research has been quite varied. In general, scant attention has been paid to how DIF can affect survey data quality, with the exception of some studies, most of them in the health survey field aimed at detecting DIF in particular scales included in survey questionnaires. Even within the total survey error (Biemer 2011) paradigm, no systematic efforts have been carried out to evaluate the impact of DIF. This could minimize the impact of DIF in later statistical decisions based on survey data from DIF items.

The growing use of psychological scales in education, public opinion, health, and quality-of-life surveys at a national and international level warrants the need to address how DIF can affect statistical decisions, based on scale scores included in survey questionnaires. For example, let us imagine that a survey researcher is interested in whether there are gender differences in, for instance, one of the versions of the PROMIS scales (e.g., Cella et al. 2010) frequently used in health surveys. One of the research questions could be what is the impact of gender-based DIF on the eventual statistical decision if the group means (men or women) of the observed scores—or trait estimates—on the PROMIS scales are equal? This could also be applied to some of the scales generally included in health surveys, such as the KIDSCREEN questionnaires (Ravens-Sieberer et al. 2001). For example, Robitail et al. (2007) found 5 items showing uniform DIF and another 10 items flagged with nonuniform DIF in a cross-cultural study of KIDSCREEN-27 across 13 countries. Due to the wide usage of KIDSCREEN and similar scales in the health context and in other fields, it would be useful to know how much DIF can be tolerated without affecting statistical conclusions about real differences between groups.

While some recent studies have been conducted in testing of how DIF affects statistical decisions (e.g., Li and Zumbo 2009; Oliveri et al. 2014), the previously mentioned issues have not been systematically studied in survey research. For example, Teresi et al. (2008) reviewed DIF concepts and methods for detecting DIF in patient-report outcome measures. These authors noted the need to develop methods that would allow researchers to analyze DIF magnitude and its effects.

The existence of DIF in scales included in any survey questionnaire could be due to the fact that (a) DIF analyses are not often performed when survey questions and scale data are analyzed or (b) scales are considered a “valid instrument” for which no additional research is needed. The second reason often leads, in survey research, to the parallel version of the decision to keep DIF items within educational tests, considering content validity issues. For example, Varni et al. (2015) found a small number of DIF items in PROMIS Parent Proxy Report Scales that were retained with the advice to avoid their use when comparing younger and older children.

This study will address these issues by aiming to answer the general research question of “how does polytomous DIF affect observed scale score comparisons?” The objective is to discover the variables related to the presence of DIF (e.g., group sizes, number of DIF items, or DIF patterns), which can affect the type I error and effect size of the independent samples t-test on the observed total scale scores. In order to attain this goal, the present study follows the framework developed by Li and Zumbo (2009) for analyzing the impact of dichotomous DIF on statistical conclusions, based on observed test scores. Li and Zumbo’s framework was adapted to cover the specific characteristics of polytomous items like different DIF patterns that cannot be explored in dichotomous items, and more simulation conditions were added to increase the relevance of the study in survey research.

Method

A simulation study was conducted using R software version 2.15.2 (2012). Different conditions were manipulated in order to examine the DIF effect on between-group mean differences. In this study, data were simulated based on a polytomous item response theory (IRT) model.

Data Generation

Item responses were generated using the IRT graded response model (Samejima 1969). One of the assumptions of this model establishes that the item response to a polytomous item is a continuous variable divided into m ordered categories, one for each response option. For each category, it is possible to specify an item category characteristic curve (ICCC), which describes the probability that a respondent with a particular level of “ability” (θ), that is, the underlying construct measured (e.g., opinion, attitude, self-reported health outcome, etc.), chooses the k response option (P_jk (θ)) to item j as a function of θ. As the ICCCs for polytomous items have different forms for each response category, Samejima (1969) defined the “boundary” category characteristic curves (BCCCs), or boundary response functions, to represent the cumulative probability (P*_jk (θ)) of a response above category k. For a polytomous item having, for instance, four response categories, there are three BCCCs. The logistic form of the BCCC is given as:

P_{j k}^{*} (θ) = \frac{\exp (a_{j} (θ - b_{j k}))}{1 + \exp (a_{j} (θ - b_{j k}))} .

Each of the BCCCs describes the probability that a respondent advances to a response category equal to or greater than k. Where a_j is the discrimination parameter for item j, which is constant across all categories for the same item, b_jk is the location parameter for item j in the boundary category k, and θ is the ability parameter. An item j may be characterized by a vector of location parameters b _j , and when the boundaries of the categories are considered, this vector would be as follows:

b_{j} = (\begin{matrix} b_{1} \\ b_{2} \\ b_{3} \\ ⋮ \\ b_{m - 1} \end{matrix}) .

In this study, the simulated test consisted of 20 items with five categories, where the b parameters of boundary categories and the a parameters were taken from Raju et al. (2009). The item length and format for the simulated test were typical of scales included in survey questionnaires. For example, the length of PROMIS instruments for different domains varies from 20 to 39 items (Cella et al. 2010), there is a KIDSCREEN version with 27 items (Ravens-Sieberer et al. 2001), and one of the versions of the General Health Questionnaire includes 20 items (Goldberg 1972).

Data were generated by the following different steps: (a) Latent trait parameters were randomly generated from a Gaussian distribution using R software and (b) item response data were generated for two groups using a computer program written by the authors. This program implemented the procedure described by Hambleton and Cook (1983) but modifying the input, which was the latent trait in this study, and the defined item parameters.

For simplicity, the terms “reference” and “focal” groups are used below to describe the experimental conditions. Both terms are common in DIF testing literature and can also mean “majority” and “minority” group, or the mainstream group and possible cultural–linguistic groups in the target population, respectively.

Experimental Conditions

Four independent variables were manipulated in this study: sample size, amount of DIF, DIF patterns, and the percentage of DIF items in the test.

Sample size

Different sample sizes were used for the reference and focal groups: 100/100, 250/250, 500/500, and 1,000/1,000. These conditions reflected situations that are more likely to occur in survey practice and involve small and large sample sizes for group comparison. However, it is also frequent to find unbalanced sample sizes across reference and focal groups. In these cases, equal sample sizes can be reached by extracting a random subsample from the majority group, which allows the cross validation of DIF results. Moreover, the mean “abilities” of the reference (μ_R) and focal (μ_F) groups were settled as equal (no impact condition), thus the mean difference was 0 (μ_d = μ_R − μ_F = 0).

Magnitude of DIF

Two levels of differences in location parameters (b _j ) were manipulated (.2 and .4), which indicated that the magnitudes of DIF were simulated to be small or moderate for each item. The discrimination parameters of the DIF items were not manipulated, thus this study did not directly investigate the nonuniform DIF.

Percentage of DIF items

The proportion of DIF items in the test was also manipulated. Four levels were considered: 0 percent, 10 percent (2 DIF items, items 1 and 2 in Table 1), 30 percent (4 DIF items, items 1–4 in Table 1), and 40 percent (8 DIF items, specifically items 1–8 in Table 1). The condition exhibiting DIF in 10 percent of the items shows a possible common situation for scales included in survey questionnaires, given that it is common in achievement or aptitude tests to find between 10 percent and 15 percent of DIF items (Narayanan and Swaminathan 1994), this being even higher in questionnaire adaptations with percentages of DIF items above 20 percent when comparing different linguistic or cultural versions of scales (Gierl, Gotzmann, and Boughton 2004). Following Li and Zumbo’s (2009) work, it was expected that a larger magnitude of DIF would increase the differences in item response between groups, and, hence, the combined DIF effect across items might result in greater type I error rates. Thus, the type I error rate might be affected by the proportion of DIF items in the scale: the more the DIF items, the higher the type I error rate in t-test statistics. In this study, the magnitudes of DIF in the test were indirectly manipulated through the percentage of DIF items and DIF direction, “favoring” the focal or the reference group, depending on each specific condition.

Table 1.

Parameters for DIF Items in Reference and Focal Groups When the Amount of DIF Was 0.4 and the Percentage of DIF Items Was 40 Percent.

Item	Reference Group						Focal Group
Item	a	b _R1	b _R2	b _R3	b _R4	DIF Pattern	a	b _F1	b _F2	b _F3	b _F4
1	0.73	−1.80	−0.60	0.60	1.80	Constant	0.73	−1.40	−0.20	1.00	2.20
2	1.00	−2.32	−1.12	0.08	1.28		1.00	−1.92	−0.72	0.48	1.68
3	1.00	−1.28	−0.08	1.12	2.32		1.00	−0.88	0.32	1.52	2.72
4	1.00	−1.80	−0.60	0.60	1.80		1.00	−1.40	−0.20	1.00	2.20
5	0.73	−1.80	−0.60	0.60	1.80		0.73	−1.40	−0.20	1.00	2.20
6	1.00	−2.32	−1.12	0.08	1.28		1.00	−1.92	−0.72	0.48	1.68
7	1.00	−1.28	−0.08	1.12	2.32		1.00	−0.88	0.32	1.52	2.72
8	1.36	−1.80	−0.60	0.60	1.80		1.36	−1.40	−0.20	1.00	2.20
1	0.73	−1.80	−0.60	0.60	1.80	Balanced	0.73	−1.40	−0.60	0.60	1.40
2	1.00	−2.32	−1.12	0.08	1.28		1.00	−1.92	−1.12	0.08	0.88
3	1.00	−1.28	−0.08	1.12	2.32		1.00	−0.88	−0.08	1.12	1.92
4	1.00	−1.80	−0.60	0.60	1.80		1.00	−1.40	−0.60	0.60	1.40
5	0.73	−1.80	−0.60	0.60	1.80		0.73	−1.40	−0.60	0.60	1.40
6	1.00	−2.32	−1.12	0.08	1.28		1.00	−1.92	−1.12	0.08	0.88
7	1.00	−1.28	−0.08	1.12	2.32		1.00	−0.88	−0.08	1.12	1.92
8	1.36	−1.80	−0.60	0.60	1.80		1.36	−1.40	−0.60	0.60	1.40
1	0.73	−1.80	−0.60	0.60	1.80	DIF low	0.73	−1.40	−0.60	0.60	1.80
2	1.00	−2.32	−1.12	0.08	1.28		1.00	−1.92	−1.12	0.08	1.28
3	1.00	−1.28	−0.08	1.12	2.32		1.00	−0.88	−0.08	1.12	2.32
4	1.00	−1.80	−0.60	0.60	1.80		1.00	−1.40	−0.60	0.60	1.80
5	0.73	−1.80	−0.60	0.60	1.80		0.73	−1.40	−0.60	0.60	1.80
6	1.00	−2.32	−1.12	0.08	1.28		1.00	−1.92	−1.12	0.08	1.28
7	1.00	−1.28	−0.08	1.12	2.32		1.00	−0.88	−0.08	1.12	2.32
8	1.36	−1.80	−0.60	0.60	1.80		1.36	−1.40	−0.60	0.60	1.80
1	0.73	−1.80	−0.60	0.60	1.80	DIF high	0.73	−1.80	−0.60	0.60	2.20
2	1.00	−2.32	−1.12	0.08	1.28		1.00	−2.32	−1.12	0.08	1.68
3	1.00	−1.28	−0.08	1.12	2.32		1.00	−1.28	−0.08	1.12	2.72
4	1.00	−1.80	−0.60	0.60	1.80		1.00	−1.80	−0.60	0.60	2.20
5	0.73	−1.80	−0.60	0.60	1.80		0.73	−1.80	−0.60	0.60	2.20
6	1.00	−2.32	−1.12	0.08	1.28		1.00	−2.32	−1.12	0.08	1.68
7	1.00	−1.28	−0.08	1.12	2.32		1.00	−1.28	−0.08	1.12	2.72
8	1.36	−1.80	−0.60	0.60	1.80		1.36	−1.80	−0.60	0.60	2.20
1	0.73	−1.80	−0.60	0.60	1.80	Cancellation	0.73	−1.40	−0.20	1.00	2.20
2	1.00	−2.32	−1.12	0.08	1.28		1.00	−2.72	−1.52	−0.32	0.88
3	1.00	−1.28	−0.08	1.12	2.32		1.00	−0.88	0.32	1.52	2.72
4	1.00	−1.80	−0.60	0.60	1.80		1.00	−2.20	−1.00	0.20	1.40
5	0.73	−1.80	−0.60	0.60	1.80		0.73	−1.40	−0.20	1.00	2.20
6	1.00	−2.32	−1.12	0.08	1.28		1.00	−2.72	−1.52	−0.32	0.88
7	1.00	−1.28	−0.08	1.12	2.32		1.00	−0.88	0.32	1.52	2.72
8	1.36	−1.80	−0.60	0.60	1.80		1.36	−2.20	−1.00	0.20	1.40

Note: DIF = differential item functioning.

DIF patterns

Five DIF patterns were manipulated: constant, balanced, shift-low, shift-high, and constant-item/balanced-test (Su and Wang 2005).

Constant DIF pattern

The item parameters of the two groups depicted the magnitude of DIF, which were b_Fjk = b_Rjk + s, where s is a positive number. Thus, in this condition, all the location parameters within a DIF item of the focal group were larger than those of the reference group by s, which takes values of .2 or .4, depending on the condition. Larger location parameters for the focus group mean that reference and focus group members, who have the same level of ability, do not have equal probability of advancing in each response category and that difference between groups is the same across all-boundary categories. The constant DIF pattern can provoke an amplification effect at test level (Li and Zumbo 2009). Previous studies provide evidence on how constant DIF can come from an item-level property such as unfamiliar content, translation problems, or contextual difference between groups (Benítez and Padilla 2014; Penfield and Lam 2000).

Balanced DIF pattern

The location parameters of the two groups were b_Fj ₁ = b_Rj ₁ + s, b_Fj ₂ = b_Rj ₂, b_Fj ₃ = b_Rj ₃, and b_Fj ₄ = b_Rj ₄ − s. This pattern inserted DIF on extreme categories leaving intermediate boundary categories free of DIF. Overall, the magnitudes of DIF within items were balanced between groups, so that contaminations within items were cancelled out between groups.

Shift-low DIF pattern

The location parameters of the two groups were b_Fj ₁ = b_Rj ₁ + s and had the same values for the other categories. That means that DIF was added only in the lowest extreme boundary category.

Shift-high DIF pattern

The location parameters were set b_Fj ₄ = b_Rj ₄ + s, while being equal for the other categories, similar to the previous pattern but including DIF in the highest category. There is evidence supporting that when DIF affects one isolated boundary category (as in the balanced, shift-low, and shift-high DIF patterns), the proper category is the responsible factor for DIF (Penfield, Alvarez, and Lee 2009). Such factors could be associated with translation problems with the category labels, different relations between item stems and response categories, and so on.

Constant-item/balanced-test pattern (cancellation)

DIF was constant within items as in the first pattern, but the effect of the items was cancelled at the test level by changing the DIF effect across items. Specifically, half of the DIF items had a positive s value, and the other half had a negative s value, so that the contaminations within the test were cancelled out between groups.

In terms of type of DIF patterns, the balanced, shift-low, and shift-high patterns represented situations of nonconstant DIF, since some boundaries between categories require more ability from the focal group than from the reference group in order to advance to the next response category, while for others (constant and constant-item/test balanced), the ability level would be equal across groups.

A total of 4 × 2 × 3 × 5 DIF conditions plus four non-DIF conditions were manipulated and 1,000 replications were made under each one. Table 1 shows the specific parameters for each simulated DIF pattern.

As Table 1 shows, items for the reference group maintained the values of location parameters across conditions, while DIF manipulations were applied for item location parameters in the focal group, as explained above.

Data Analysis

The effect of DIF in the observed total scores was analyzed by comparing total scores of reference and focal groups computing independent samples t-test analyses. Also, the standardized mean difference was calculated by computing Cohen’s d (Cohen 1988) to assess the effect size of the difference between means, as follows:

d = \frac{{\bar{X}}_{R} - {\bar{X}}_{F}}{s},

where

s = \sqrt{\frac{(n_{R} - 1) S_{R}^{2} + (n_{F} - 1) S_{F}^{2}}{n_{R} + n_{F} - 2} .}

Two dependent variables were calculated: (a) type I error rates, as the proportion of rejected t-tests for 1,000 replications in each of the manipulated conditions and (b) typified mean difference (d) effect size measure for each replication. For this effect size measure, the mean and standard deviation for each replication were obtained. Moreover, intervals at the 95 percent confidence level were also calculated for the d effect size means in each condition through all replications (Kelley and Preacher 2012). For type I error rates, a value of .05 was used at the nominal significance level, which meant that the type I error was defined as the proportion of times that a true null hypothesis was falsely rejected at the .05 level. Following Bradley’s (1978) indications when interpreting the type I error rates, in the present study, a fairly stringent criterion for robustness was assumed, which requires the empirical type I error rate to lie between .045 and .055. However, the results obtained were also compared with the moderate criterion (type I error rate between .040 and .060) and the liberal criterion (type I error rate between .025 and .075).

Results

Type I Error Rate

Table 2 shows type I error rates for t-test of mean differences for independent samples in each one of the conditions. Regarding the non-DIF item condition, the type I error rates were, as expected, close to the nominal level across all sample sizes, ranging from .04 to .05.

Table 2.

Type I Error Rate for Independent Samples t-Test across DIF Conditions.

Sample Size (RG/FG)	DIF Items	Amount of DIF	DIF Patterns
Sample Size (RG/FG)	DIF Items	Amount of DIF	Constant	Balanced	Shift- low	Shift- high	Cancellation
100/100	0	0.0	0.041
	2	0.2	0.044	0.043	0.053	0.056	0.040
		0.4	0.049	0.037	0.044	0.033	0.042
	4	0.2	0.064	0.046	0.043	0.044	0.032
		0.4	0.162	0.046	0.044	0.048	0.048
	8	0.2	0.164	0.049	0.058	0.055	0.040
		0.4	0.363	0.048	0.067	0.047	0.036
250/250	0	0.0	0.046
	2	0.2	0.055	0.048	0.040	0.041	0.044
		0.4	0.078	0.032	0.061	0.031	0.047
	4	0.2	0.109	0.046	0.040	0.050	0.042
		0.4	0.290	0.026	0.056	0.045	0.029
	8	0.2	0.310	0.039	0.058	0.037	0.049
		0.4	0.687	0.034	0.088	0.055	0.051
500/500	0	0.0	0.046
	2	0.2	0.067	0.037	0.044	0.045	0.057
		0.4	0.112	0.028	0.062	0.050	0.043
	4	0.2	0.181	0.045	0.053	0.059	0.062
		0.4	0.590	0.045	0.081	0.043	0.044
	8	0.2	0.527	0.039	0.051	0.036	0.084
		0.4	0.961	0.048	0.164	0.064	0.051
1,000/1,000	0	0.0	0.054
	2	0.2	0.080	0.039	0.034	0.035	0.043
		0.4	0.165	0.037	0.105	0.043	0.058
	4	0.2	0.339	0.039	0.047	0.056	0.054
		0.4	0.814	0.043	0.106	0.059	0.038
	8	0.2	0.860	0.037	0.079	0.039	0.072
		0.4	0.999	0.050	0.283	0.102	0.066

Note: DIF = differential item functioning; RG = reference group; FG = focus group.

As Table 2 shows, the effect of the DIF presence on mean differences was affected by the manipulated DIF pattern as well as by the number of DIF items and the amount of DIF in these items. For example, when the manipulated pattern was “constant,” type I error rates were higher than .05 in all conditions, except when the sample size was 100 in both groups. In this condition, differences were observed only when 10 percent of items or more showed DIF regardless of the amount of DIF.

In terms of the percentage of DIF items, type I error rates were above the nominal level of significance when the percentage of DIF items was equal to or greater than 20 percent, even with small sample sizes. More specifically, type I error rates ranged from .06 to .81 when the percentage of DIF items was 20 percent, being closer to 1 when the sample size was higher (1,000/1,000). In general, type I error rates were closer to 1 when the percentage of DIF items was higher, that is, 40 percent, and the larger sample size.

Results for the balanced DIF pattern (the pattern exhibiting DIF only in extreme categories) analysis showed that the type I error rates were lower than or equal to .05 for all conditions. Furthermore, when the simulated DIF pattern was “cancellation,” that is, constant across all item categories (within items) but balanced in test, the type I error rates were lower or slightly higher than the nominal level. This fact was evident with higher sample sizes (1,000/1,000) and with higher percentage of DIF items (40 percent). In any case, type I error rates were lower when a liberal criterion was considered.

Finally, when DIF was manipulated across categories, similarly to what Penfield et al. (2009) called constant nonpervasive DIF, and specifically when DIF was in low categories (shift-low pattern), it was found that type I error rates were between .040 and .058 when the amount of DIF was low for sample sizes from 100 to 500. In this situation, none of the type I error rates were inflated, using Bradley’s (1978) moderate criterion. However, when the sample size was increased to 1,000, the type I error rates were between .034 (lower than the moderate criterion) and .079 (higher than the liberal criterion). In addition, when the sample size was larger than 100 examinees per group, the type I error rate did not reach the moderate degree of robustness in the condition of 40 percent of DIF items, as the type I error rate was .067. Therefore, the type I error rate was inflated when higher percentages of DIF items and larger sample sizes were analyzed, reaching values over the liberal criterion.

On the other hand, when DIF was manipulated in high categories (shift-high pattern), the stringent criterion for the nominal level was not reached in 16 of the 24 manipulated conditions, as type I error rates were lower than .045 (in nine cases) and higher than .055 (in the rest of the situations). Nevertheless, the liberal criterion was satisfied in all the conditions except in the condition with the largest sample size (1,000/1,000), the largest amount of DIF (.4), and the highest percentage of DIF items (40 percent). In this case, the type I error rate was .102.

Effect Size Measure

Means and standard deviations of the standardized mean differences as well as confidence intervals are reported in Table 3. Regarding the measure of effect size for the total scores, it was found that d increased in the constant DIF pattern (d̄ = .109), followed by the shift-low DIF pattern (d̄ = .028), while the lowest values were obtained in the balanced DIF pattern (d̄ = .001). The effect size of the differences between both groups (reference and focal) was not affected by sample size, although it was affected by the percentage of DIF items. It was observed that the higher the number of DIF items in the test, the higher the difference in the typified mean between groups. With respect to the DIF patterns, for the constant DIF pattern, d̄ was .039 when only 10 percent of the items had DIF, but it increased to .104 when 20 percent of the items in the test showed DIF, and .185 for 40 percent. For the other DIF patterns, small effects on d̄ values were found.

Table 3.

Mean, Standard Deviation (SD), and Confidence Interval (CI) for d Effect Size Measure across Different Conditions.

Sample Size (RG/FG)	DIF Items	Amount of DIF	DIF Patterns
			Constant			Balanced			Shift-low			Shift-high			Cancellation
			Mean	SD	CI 95%	Mean	SD	CI 95%	Mean	SD	CI 95%	Mean	SD	CI 95%	Mean	SD	CI 95%
100/100	0	0.0	.009	.141	−.000, .017
	2	0.2	.028	.138	.019, .036	−.003	.144	−.012, .006	−.008	.146	−.017, .001	−.001	.146	−.011, .008	.012	.144	.003, .021
		0.4	.054	.145	.045, .063	.008	.137	−.001, .016	.036	.140	.002, .044	.001	.140	.001, .018	−.021	.139	−.030, −.013
	4	0.2	.069	.140	.060, .078	.012	.146	.003, .021	.013	.144	.004, .022	.014	.143	.005, .023	−.013	.138	−.022, −.005
		0.4	.145	.143	.136, .154	−.004	.144	−.013, .005	.034	.144	.025, .043	.014	.149	.005, .023	−.005	.146	−.005, .013
	8	0.2	.139	.144	.130, .148	.006	.144	−.003, .015	.030	.151	.021, .039	−.004	.147	−.013, .005	−.026	.137	−.035, −.018
		0.4	.240	.141	.231, .249	−.012	.143	−.021, −.003	.059	.144	.050, .067	.041	.143	.032, .050	−.026	.131	−.034, −.018
250/250	0	0.0	.004	.089	.004, .089
	2	0.2	.023	.091	.017, .029	−.007	.092	−.013, −.001	−.008	.092	−.014, −.002	.006	.087	.000, .011	.009	.090	.003, .014
		0.4	.050	.088	.044, .055	.003	.088	−.002, .009	.038	.090	.032, .043	.014	.086	.009, .020	−.015	.089	−.021, −.010
	4	0.2	.078	.086	.073, .083	.007	.090	.001, .012	.014	.087	.008, .019	.019	.091	.013, .024	−.012	.089	−.018, −.007
		0.4	.133	.093	.127, .139	−.003	.083	−.009, .002	.038	.090	.032, .043	.017	.087	.011, .022	.001	.087	−.004, .007
	8	0.2	.138	.089	.132, .143	.001	.089	−.005, .006	.026	.092	.020, .032	−.002	.088	−.008, .003	−.032	.089	−.037, .026
		0.4	.230	.089	.224, .235	−.009	.087	−.014, −.004	.062	.087	.056, .067	.035	.091	.029, .040	−.026	.088	−.032, −.021
500/500	0	0.0	.007	.064	.003, .011
	2	0.2	.029	.065	.025, .033	−.006	.061	−.010, −.002	−.004	.064	−.008, −.000	.003	.063	−.001, .007	.016	.064	.012, .020
		0.4	.052	.064	.048, .056	.000	.062	−.004, .004	.034	.061	.030, .037	.015	.065	.011, .019	−.015	.062	−.019, −.012
	4	0.2	.075	.060	.072, .079	.008	.063	.004, .012	.013	.064	.009, .017	.013	.064	.009, .017	−.015	.066	−.019, −.016
		0.4	.131	.062	.127, .134	−.005	.065	−.009, −.001	.032	.065	.028, .036	.015	.062	.011, .019	.004	.065	−.000, .008
	8	0.2	.135	.062	.131, .139	.003	.063	−.001, .007	.025	.062	.022, .029	−.002	.063	−.006, .002	−.029	.066	−.033, −.025
		0.4	.230	.061	.226, .234	−.004	.063	−.008, −.001	.065	.065	.060, .069	.034	.061	.031, .036	−.022	.065	−.026, −.018
1000/1000	0	0.0	.005	.046	.002, .008
	2	0.2	.025	.045	.022, .027	−.006	.044	−.008, −.003	−.003	.043	−.003, .000	.002	.044	−.001, .005	.013	.043	.011, .016
		0.4	.048	.045	.045, .051	.000	.044	−.002, .003	.035	.045	.032, .038	.012	.045	.010, .015	−.021	.043	−.023, −.018
	4	0.2	.072	.044	.069, .075	.052	.044	.003, .008	.013	.044	.011, .016	.013	.045	.010, .015	−.012	.046	−.015, −.009
		0.4	.132	.044	.130, .135	−.007	.046	−.009, −.004	.038	.043	.035, .040	.017	.044	.015, .020	.005	.044	.002, .007
	8	0.2	.138	.044	.135, .140	.003	.045	.000, .006	.026	.046	.023, .029	−.000	.043	−.003, .002	−.027	.045	−.030, −.025
		0.4	.230	.046	.227, .232	−.006	.044	−.009, −.003	.062	.046	.060, .065	.033	.045	.031, .036	−.023	.044	−.026, −.021

Note: DIF = differential item functioning; RG = reference group; FG = focus group.

Variations in the typified mean differences were found related to the magnitude of DIF across DIF patterns. In this line, higher values in the typified mean were observed when higher magnitudes of DIF were included in the constant, shift-low, and shift-high patterns, but these effects were not obtained for both the balanced and the cancellation patterns. The effect was the same for all sample sizes whose increment was related to the drop in standard deviations, as expected.

Discussion

The aim of the study was to investigate conditions under which the presence of DIF affected observed scale score comparisons. The objective behind the study was to find out under what circumstances the DIF effect can be disregarded, given how difficult removing DIF items from scales in surveys can be. In survey research, group comparisons are often made without conducting DIF analyses, and when DIF is analyzed, there is rarely the option of removing DIF items due to content validity reasons (Langer et al. 2008; Reeve et al. 2007; Teresi et al. 2007). Therefore, this study provides arguments based on simulated results for survey researchers to better interpret DIF effects on scales included in survey questionnaires.

The results of the study suggest that the presence of DIF items affects statistical conclusions based on comparing means, but this effect is linked to specific aspects. The most relevant of them are DIF patterns that increase the DIF effect, followed by the number of DIF items in the test, and the magnitude of DIF in these items. Thus, when the manipulated DIF pattern was constant, the type I error rates exceeded the nominal level and an amplification effect occurred at the test level. However, when the sample size was small (100/100) and the percentage of DIF items was low (10 percent), type I error rates were in the expected range, that is, DIF did not affect the mean comparisons between groups. Therefore, the presence of DIF was modulated by reduced sample sizes and small percentages of DIF items. The situation changed when the percentage of DIF items was 20 percent or higher (often found when assessing DIF between different cultural or linguistic versions). In these cases, the presence of DIF would threaten the validity of group comparisons based on total scale scores. Therefore, in terms of the effect on group comparisons, the worst scenario seems to be when the pattern of DIF is constant across the localization parameters, unidirectional (“favoring” only the reference group), the percentage of DIF items is 20 percent or higher, and sample size is above 100 respondents per group.

As previously mentioned, results from the present study are in line with those reported by Li and Zumbo (2009) for tests consisting of dichotomous items. These authors found that, when the manipulated DIF was unidirectional (amplification effects), the inflation of the type I error rates increased as the number of DIF items and the sample size augmented. The results obtained in the present study also indicated that the effect size differences increased as the percentage of DIF items in the test increased, which means that the observed mean differences were spuriously inflated by the presence of more DIF contamination in the test. Finally, the amount of DIF and the percentage of DIF items provoke undesired differences across groups, but clear criteria cannot be established considering both variables separately, as it seems both are affecting the scale together. Therefore, decisions about using the DIF items should take into account both content validity considerations and equivalence issues when comparing different linguistic and cultural groups.

Thinking of survey researchers and practitioners, some guidelines can be proposed based on the main results of the simulation study: (a) before making comparisons across groups, DIF analysis should be performed together with empirical item characteristic curve representations to describe possible DIF patterns; (b) if the DIF patterns found are constant, look at the item stems to identify unfamiliar contents or stimuli, translation problems, and so on; and (c) when the DIF pattern found is not constant, possible different understanding of the category meaning (labels, numbers, etc.) should be investigated.

The present study has some limitations, mainly related to the conditions examined. Future research should focus on overcoming these flaws by addressing the issues that remain unclear, such as the impact of the length of the test on the DIF effect, unbalanced sample sizes, the impact between groups (i.e., when mean group differences are higher than zero (μ _d > 0)), or the extension of results to other magnitudes of DIF and sample sizes (i.e., determining the approximate number after which the DIF effect is clear). However, despite such limitations, this study provides a useful framework for the conditions behind the increase in the impact of DIF, which improves our knowledge of how DIF works.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was partially funded by the Andalusia Regional Government under the Excellent Research Fund (Project n° SEJ-6569) and by Seneca Foundation, Agency for Science and Technology in the Region of Murcia under Research Projects in the Humanities and Social Sciences Fund (Project 11917/PHCS/09).

Supplemental Material

The online appendices are available at .

References

American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. 2014. Standards for Educational and Psychological Tests. Washington, DC: American Educational Research Association.

Benítez

Padilla

J. L.

. 2014. “Analysis of Non-equivalent Assessments across Different Linguistic Groups Using a Mixed Methods Approach: Understanding the Causes of Differential Item Functioning by Cognitive Interviewing.” Journal of Mixed Methods Research 8:52–68.

Biemer

P. P.

2011. “Total Survey Error: Design, Implementation, and Evaluation.” Public Opinion Quarterly 74:817–48.

Bradley

J. V.

1978. “Robustness?” British Journal of Mathematical and Statistical Psychology 34:144–52.

Cella

Riley

Stone

Rothrock

Reeve

Yount

Amtmann

Bode

Buysse

D. J.

Choi

S. W.

Cook

K. F.

DeVellis

DeWalt

Fries

J. F.

Gershon

Hahn

Pilkonis

Revicki

Rose

Weinfurt

Hays

R. D.

, on behalf of the PROMIS Cooperative Group. 2010. “Initial Item Banks and First Wave Testing of the Patient–Reported Outcomes Measurement Information System (PROMIS) Network: 2005–2008.” Journal of Clinical Epidemiology 63:1179–94.

Cohen

1988. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum.

Gierl

Gotzmann

Boughton

. 2004. “Performance of SIBTEST When the Percentage of DIF Items Is Large.” Applied Measurement in Education 17:241–64.

Goldberg

1972. The Detection of Psychiatric Illness by Questionnaire. London, UK: Oxford University Press.

Hambleton

R. K.

Cook

L. L.

. 1983. “Robustness of Item Response Models and Effects of Test Length and Sample Size on the Precision of Ability Estimates.” Pp. 31–49 in New Horizons in Testing: Latent Trait Test Theory and Computerized Adaptive Testing, edited by Weiss

D. J.

. New York: Academic Press.

10.

Hidalgo

M. D.

Gómez-Benito

. 2010. “Education Measurement: Differential Item Functioning.” Pp. 36–44 in International Encyclopedia of Education, edited by Peterson

Baker

McGaw

. Oxford, UK: Elsevier—Science & Technology.

11.

Kelley

Preacher

K. J.

. 2012. “On Effect Size.” Psychological Methods 17:137–52.

12.

Langer

M. M.

Hill

C. D.

Thissen

Burwinkle

T. M.

Varni

J. W.

DeWalt

D. A.

. 2008. “Item Response Theory Detected Differential Item Functioning between Healthy and Ill Children in Quality-of-life Measures.” Journal of Clinical Epidemiology 61:268–76.

13.

Zumbo

B. D.

. 2009. “Impact of Differential Item Functioning on Subsequent Statistical Conclusions Based on Observed Test Score Data.” Psicológica 30:343–70.

14.

Narayanan

Swaminathan

. 1994. “Performance of the Mantel-Haenszel and Simultaneous Item Bias Procedures for Detecting Differential Item Functioning.” Applied Psychological Measurement 18:315–28.

15.

Oliveri

M. E.

Ercikan

Zumbo

B. D.

Lawless

. 2014. “Uncovering Substantive Patterns in Student Responses in International Large-scale Assessments-comparing a Latent Class to a Manifest DIF Approach.” International Journal of Testing 14:265–87.

16.

Osterlind

S. J.

Everson

H. T.

. 2009. Differential Item Functioning. London, UK: Sage.

17.

Penfield

R. D.

Alvarez

Lee

. 2009. “Using a Taxonomy of Differential Step Functioning to Improve the Interpretation of DIF in Polytomous Items: An Illustration.” Applied Measurement in Education 22:61–78.

18.

Penfield

R. D.

Lam

T. C. M.

. 2000. “Assessing differential item functioning in performance assessment: Review and recommendations.” Educational Measurement: Issues and Practice 19:5–15.

19.

Raju

N. S.

Fortmann-Johnson

K. A.

Kim

Morris

S. B.

Nering

M. L.

Oshima

T. C.

. 2009. “The Item Parameter Replication Method for Detecting Differential Functioning in the DFIT Framework.” Applied Measurement in Education 33:133–47.

20.

Ravens-Sieberer

Gosch

Abel

Auquier

Bellach

B. M.

Bruil

Dür

Power

Rajmil

, and The European KIDSCREEN Group. 2001. “Quality of Life in Children and Adolescents: A European Public Health Perspective.” Sozial- und Präventivmedizin 46:294–302.

21.

R Development Core Team. 2012. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.

22.

Reeve

B. B.

Hays

R. D.

Bjorner

J. B.

Cook

K. F.

Crane

P. K.

Teresi

J. A.

Cella

. 2007. “Psychometric Evaluation and Calibration of Health-related Quality of Life Item Banks: Plans for the Patient-reported Outcomes Measurement Information System (PROMIS).” Medical Care 45:S22–31.

23.

Robitail

Ravens-Sieberer

Simeoni

M. C.

Rajmil

Bruil

Power

Duer

Cloetta

Czemy

Mazur

Czimbalmos

Tountas

Hagquist

Kilroe

Auquier

, and KIDSCREEN Group. 2007. “Testing the Structural and Cross-cultural Validity of the KIDSCREEN-27 Quality of Life Questionnaire.” Quality of Life Research 16:1335–45.

24.

Samejima

1969. “Estimation of Latent Ability Using a Response Pattern of Graded Scores.” Psychometrika Monograph Supplement 34:100–14.

25.

Sireci

S. G.

Rios

J. A.

. 2013. “Decisions That Make a Difference in Detecting Differential Item Functioning.” Educational Research and Evaluation 19:170–87.

26.

Y. H.

Wang

W. C.

. 2005. “Efficiency of the Mantel, Generalized Mantel-Haenszel, and Logistic Discriminant Function Analysis Methods in Detecting Differential Item Functioning for Polytomous Items.” Applied Measurement in Education 18:313–50.

27.

Teresi

J. A.

Ocepek-Welikson

Kleinman

Cook

K. F.

Crane

P. K.

Gibbons

L. E.

Cella

. 2007. “Evaluating Measurement Equivalence Using the Item Response Theory Log-likelihood Ratio (IRTLR) Method to Assess Differential Item Functioning (DIF): Applications (with Illustrations) to Measures of Physical Functioning Ability and General Distress.” Quality of Life Research 16:43–68.

28.

Teresi

J. A.

Ramirez

Lai

J. S.

Silver

. 2008. “Occurrences and Sources of Differential Item Functioning (DIF) in Patient-reported Outcome Measures: Description of DIF Methods, and Review of Measures of Depression, Quality of Life and General Health.” Psychology Science Quarterly 50:538–601.

29.

Varni

J. W.

Thissen

Stucky

B. D.

Liu

Magnus

DeWalt

D. A.

. 2015. “Item-level informant discrepancies between children and their parents on the PROMIS^® pediatric scales”. Quality of Life Research 24:1921–1937.

30.

Zumbo

B. D.

2007. “Three Generations of DIF Analyses: Considering Where It Has Been, Where It Is Now, and Where It Is Going.” Language Assessment Quarterly 4:223–33.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB