Establishing Effect Size Guidelines for Interpreting the Results of Differential Bundle Functioning Analyses Using SIBTEST

Overview of SIBTEST

SIBTEST (Shealy & Stout, 1993) uses a nonparametric statistical procedure to detect unidirectional DIF or DBF in dichotomous items. Unidirectional DIF occurs when the direction of DIF is constant across the ability scale. However, it should be noted that a version of SIBTEST known as Crossing-SIBTEST can be used if it is hypothesized that there is an interaction between ability and DIF such that an item functions differentially in favor of one group at lower levels of ability and functions differentially in favor of the other group at higher levels of ability.

DIF detection methods compensate for impact, or true mean ability differences between the two groups for which an item is thought to function differentially, by matching examinees on some conditioning variable and summarizing differences in item or bundle performance across the different levels of the conditioning variable. However, the presence of impact can still inflate the Type I error rates of DIF detection methods because the expected value of the reference group’s target ability will tend to differ from the corresponding expected value for the focal group (Shealy & Stout, 1993). SIBTEST uses a nonlinear regression correction to correct for this inflated Type-I error rate (Jiang & Stout, 1998).

Specifically, SIBTEST is based on the assumption that DIF occurs when

E R [ Y | T ] ≠ E F [ Y | T ] ,

where Y denotes the score on the studied item or bundle of items, T denotes the true score on the conditioning variable, and the subscript R or T refers to either the reference or the focal group, respectively. The conditioning variable used in SIBTEST is a matching subtest that consists of a set of items hypothesized to be unidimensional and therefore only measuring the primary construct of interest. The studied item is hypothesized to be multidimensional, measuring a secondary dimension on which the two matched groups are believed to differ, in addition to the primary construct of interest. The assumption outlined in Equation (1) implies that if E_R [Y|T]−E_F [Y|T] ≈ 0, then the studied item or set of items does not exhibit DIF or DBF. Denoting this difference by B(T), a global index of DIF can be obtained by

β uni = ∫ B ( T ) f F ( T ) d T ,

where f_F(T) denotes the probability density function of T in the focal group. Therefore, the statistical procedure used by SIBTEST is H₀: β = 0 versus H₁: β ≠ 0. The global index of DIF is approximated by

β ^ = ∑ k = 0 n p k d k * ,

where n is defined to be the number of items in the matching subtest, p_k denotes the proportion of examinees who obtained a score of k on the matching subtest, and

d k * = Y ¯ R k * − Y ¯ F k * ,

where Y ¯ G K * is defined to be the average performance on the studied item for examinees in group G (i.e., the reference or the focal group) who obtained a score of k on the matching subtest, corrected for bias. The matching subtest is corrected for bias by calculating an estimate of reliability, separately for both the reference and focal groups, and then using this estimate of reliability as the slope of a nonlinear regression of true scores on observed scores to estimate, Y ¯ G K , the average performance on the studied item for group G. The average of the true scores for each of the two groups, Y ¯ R K * and Y ¯ F K , is obtained using a first-order Taylor series approximation.

The test statistic corresponding to this global DIF index is defined by

B = B ^ s β ^ ,

where

s β ^ = ∑ k = 0 n k p k 2 ( s Y | k , R 2 N R k + s Y | k , F 2 N F k ) ,

where S Y | G K 2 is the sample variance of the studied item for group G who obtained a score of k on the matching subtest. This test statistic, commonly referred to as beta-uni (β_uni), is approximately normally distributed with a mean of 0 and a standard deviation of 1 and is easily generalized to test for DBF.

There has been extensive prior research investigating the influence of sample size on power and Type I error rates when using SIBTEST to detect DIF/DBF. However, it should be noted that this line of research is not particularly relevant to the current study because effect size measures should not be influenced by sample size. Whereas previous research, pertaining to the impact of sample size on power and Type I error rates, has focused on counting up the proportion of times the test statistic correctly (or incorrectly) rejected the null hypothesis, the present study focuses on how the presence of DIF/DBF affects the actual magnitude of β ^ , not how it affects the p value. Clearly, the p value may be affected by the magnitude of β ^ . However, it may also be affected by the standard error of β ^ and, when considering the impact of sample size on Type I error rates and power, it is likely that the standard error of β ^ is affected more than the magnitude of β ^ , because the computation of the standard error includes terms pertaining to the sample size of the reference and focal group (see Equation 6).

Roussos and Stout (1996) found that even with as few as 250 examinees in the reference and focal groups, Type I and Type II errors were not inflated to a large degree, when impact was not present. With such small sample sizes, the Type I error rate was inflated slightly only in the presence of impact; however, this issue was only found with certain types of items (e.g., those with high discrimination and low difficulty). In terms of the effect of sample size on power, previous research has demonstrated that SIBTEST displays adequate power (slightly less than 0.8) when there are 500 examinees present in each group (Gierl, Gotzmann, & Boughton, 2004). Finch (2005) found that the greatest influence on power, when using SIBTEST to detect DIF, was the size of the focal group; smaller focal groups yielded lower power than focal groups with larger sample sizes.

Method

Given the fact that the focus of this study was on creating effect size guidelines for DBF detection when using SIBTEST, as opposed to studying Type I error rates and power when doing DBF analyses, ideal conditions were simulated. Specifically, as previous research has demonstrated that impact may increase the Type I error rate, no impact was simulated. This ensured that the most ideal conditions were simulated and that the results would not be influenced by the well-known effect of impact on increasing the Type I error rate. No impact was simulated by randomly generating the true ability for both reference- and focal-group examinees from a standard normal distribution.

Similarly, sample size was also not considered in this study because prior research has shown that an increased sample size may increase the power of SIBTEST, whereas a decreased sample size may decrease the power of SIBTEST (Furlow, Ross, & Gagne, 2009; Gierl et al., 2004; Roussos & Stout, 1996). Likewise, the ratio of focal-group examinees to reference-group examinees was not considered in this study because prior research has clearly indicated that having an unequal ratio of focal- and reference-group examinees is only an issue when the overall sample size is small, less than 250 in the reference and focal groups (Furlow et al., 2009). Therefore, this study only included ideal sample size conditions of 500 examinees in both the reference and focal groups. A sample size of 500 in each group is considered more than adequate for controlling the Type I error and power rates (Clauser & Mazor, 1998).

Factors in this simulation study included the manipulation of (a) the number of items in a bundle (1, 3, or 5) that were simulated to function differentially, (b) test length (20 or 40 items), and (c) the magnitude of uniform DIF against the focal group in each item in a bundle. Of particular interest was the sum of the magnitude of DIF across items when testing for DBF. It was hypothesized that the magnitude of β ^ would be larger for a bundle of items, when compared with a single item analysis, if the sum of the magnitude of DIF across items in a bundle was comparable with that of a single item, given the well-known attributes of amplification and cancellation that can occur when using SIBTEST (Nandakumar, 1993). For example, it was expected that the results obtained from testing three items for DBF, in which the magnitude of DIF for all three items was 0.2, would be larger than the results obtained when testing one item for DIF, in which the magnitude of DIF for that one item was 0.6. Therefore, all possible combinations, ignoring item order, which resulted in the same sum of the magnitude of DIF were considered. The design of this study, which was replicated for a test that consisted of 20 items as well as one that consisted of 40 items, involved 116 conditions and is depicted in Table 1. Each condition was replicated 200 times.

OPEN IN VIEWER

Table 1.

Design of Simulation Study Conducted

No. of DIF items	Magnitude of DIF for five items studied					Sum of DIF
1	0	0	0	0	0	0
	0.1	0	0	0	0	0.1
	0.2	0	0	0	0	0.2
	0.3	0	0	0	0	0.3
	0.4	0	0	0	0	0.4
	0.5	0	0	0	0	0.5
	0.6	0	0	0	0	0.6
	0.7	0	0	0	0	0.7
	0.8	0	0	0	0	0.8
	0.9	0	0	0	0	0.9
	1.0	0	0	0	0	1.0
	1.2	0	0	0	0	1.2
	1.4	0	0	0	0	1.4
	1.6	0	0	0	0	1.6
	1.8	0	0	0	0	1.8
	2.0	0	0	0	0	2.0
	2.2	0	0	0	0	2.2
	2.4	0	0	0	0	2.4
	2.6	0	0	0	0	2.6
	2.8	0	0	0	0	2.8
	3.0	0	0	0	0	3.0
3	0.2	0.2	0.2	0	0	0.6
	0.2	0.2	0.4	0	0	0.8
	0.2	0.2	0.6	0	0	1.0
	0.2	0.4	0.4	0	0	1.0
	0.2	0.4	0.6	0	0	1.2
	0.2	0.6	0.6	0	0	1.4
	0.4	0.4	0.4	0	0	1.2
	0.4	0.4	0.6	0	0	1.4
	0.4	0.6	0.6	0	0	1.6
	0.6	0.6	0.6	0	0	1.8
	0.6	0.6	0.8	0	0	2.0
	0.6	0.8	0.8	0	0	2.2
	0.8	0.8	0.8	0	0	2.4
	0.8	0.8	1.0	0	0	2.6
	0.8	1.0	1.0	0	0	2.8
	1.0	1.0	1.0	0	0	3.0
5	0.2	0.2	0.2	0.2	0.2	1.0
	0.2	0.2	0.2	0.2	0.4	1.2
	0.2	0.2	0.2	0.2	0.6	1.4
	0.2	0.2	0.2	0.4	0.4	1.4
	0.2	0.2	0.2	0.4	0.6	1.6
	0.2	0.2	0.2	0.6	0.6	1.8
	0.2	0.2	0.4	0.4	0.4	1.6
	0.2	0.2	0.4	0.4	0.6	1.8
	0.2	0.2	0.4	0.6	0.6	2.0
	0.2	0.2	0.6	0.6	0.6	2.2
	0.2	0.4	0.4	0.4	0.4	1.8
	0.2	0.4	0.4	0.4	0.6	2.0
	0.2	0.4	0.4	0.6	0.6	2.2
	0.2	0.4	0.6	0.6	0.6	2.4
	0.2	0.6	0.6	0.6	0.6	2.6
	0.4	0.4	0.4	0.4	0.4	2.0
	0.4	0.4	0.4	0.4	0.6	2.2
	0.4	0.4	0.4	0.6	0.6	2.4
	0.4	0.4	0.6	0.6	0.6	2.6
	0.4	0.6	0.6	0.6	0.6	2.8
	0.6	0.6	0.6	0.6	0.6	3.0

Note. DIF = Differential item functioning.

Item parameters were randomly generated for each replication to ensure that the results were as generalizable as possible. Specifically, the discrimination parameter (a) was randomly generated from a log-normal distribution with a mean of 1.0 and a standard deviation of 0.5, the difficulty parameter (b) was randomly generated from a standard normal distribution, and the lower asymptote (c) was randomly generated from a beta distribution with a mean of approximately 0.16 and a standard deviation of approximately 0.004. When an item was simulated to function differentially, the item was always simulated to be more difficult for the focal group by adding the magnitude of the DIF to the difficulty parameter used for the reference group (e.g., b_foc = b_ref + magnitude of DIF).

SIBTEST was used to test for DIF/DBF. To prevent from biasing the results, the matching subtest was always created by using only those items that were not simulated to function differentially. In other words, the matching subtest was always DIF-free (Ackerman & Evans, 1994; Gierl, 2005). One-tailed tests were conducted in all cases to examine whether the studied items were functioning differentially against the focal group, which is reflective of the direction for which DIF was simulated.

An additional step was included in this study. The additional step consisted of obtaining estimates of ability for the reference and focal groups in an effort to better understand the effect of DIF/DBF on ability estimation. This step was taken to determine how much DIF or DBF an item or bundle of items needed to contain before the ability estimates for the focal examinees became seriously biased. The ability estimates were obtained for the reference and focal group members simultaneously. This ensured that the estimates were on the same scale for each group. T tests were conducted to determine if the ability estimates were statistically significantly different because of the presence of DIF/DBF. The t tests were conducted on the average ability estimates obtained for the reference and focal groups for each of the various conditions examined. In addition, bias, defined as the average signed difference between true and estimated ability, was used to summarize the impact of DIF/DBF on ability estimation.

Results

One-Item DIF Analyses

Figure 1 illustrates the results obtained from the simulations conducted to examine the behavior of β ^ when conducing one-item DIF analyses. As the figure illustrates, there is almost a perfectly linear relationship between the magnitude of DIF and the average β ^ obtained from SIBTEST. In addition, the results do not appear to depend on test length, as the lines are almost identical, regardless of test length. There are several exceptions to this finding, particularly when the magnitude of DIF exceeds 1.5, in which case the average β ^ is slightly larger when there are fewer test items. As theoretically expected, when no DIF occurs, the expected value of β ^ is zero. Interpreting these results with respect to the standard DIF effect size guidelines, in which large DIF is said to occur if | β ^ | ≥ 0 . 088 (Roussos & Stout, 1996), large DIF would typically be found by SIBTEST if the magnitude of the DIF was 0.5. Moreover, as Figure 2 illustrates, the power obtained from SIBTEST, approximately 0.85, is adequate when the magnitude of the DIF is 0.5, which is to be expected given the ideal conditions that were simulated and is consistent with the findings of Roussos and Stout (1996). Once again, as theoretically expected, when the magnitude of the DIF increases, so does the power. Figure 2 also illustrates that the Type I error rate associated with SIBTEST is well within the nominal level of 0.05, as would be expected given the ideal conditions that were simulated.

OPEN IN VIEWER

Figure 1.

Average β ^ when using SIBTEST to test for DIF for all one-item conditions studied

OPEN IN VIEWER

Figure 2.

Power when using SIBTEST to test for DIF for all one-item conditions studied

Multi-Item DBF Analyses

Figure 3 illustrates the average β ^ for all conditions studied, broken down by test length and sum of DIF across all items in a bundle. As the figure illustrates, the relationship between the sum of DIF and the average β ^ is almost perfectly linear, and there is little difference for the two test lengths considered, which is similar to what was observed when only looking at the results for one-item DIF analyses. However, the original hypothesis that the magnitude of the average β ^ would be larger for a bundle of items than for a single-item analysis, if the sum of the magnitude of DIF was comparable, is only partially supported. On average, larger β ^ are obtained when testing a bundle of items, as opposed to a single item, only as the sum of DIF increases. Specifically, in the current study, the effect of amplification is not observed until the sum of DIF/DBF reaches 2.0 for a test length of 20 and when it reaches 1.6 for a test length of 40.

OPEN IN VIEWER

Figure 3.

Average beta-uni statistics obtained for all DIF/DBF sums studied, grouped by two different test lengths studied

Impact of DIF/DBF on Ability Estimation

Figure 4 illustrates ability estimation bias as a function of the sum of DIF, under all conditions studied. As expected, test length has an effect on ability estimation bias, with more bias associated with shorter tests. However, what are more interesting are the different patterns of ability estimation bias obtained for the reference and focal groups. It was expected that as the presence of DIF or DBF increased there would be increased negative ability estimation bias for focal-group examinees. However, what Figure 4 illustrates is that as the sum of DIF increased there was increased negative ability estimation bias for focal-group examinees, as well as increased positive ability estimation bias for reference-group examinees. In other words, whereas the ability of the focal group was expectedly underestimated, the ability of the reference group was surprisingly overestimated. This effect is more pronounced for shorter length tests and for larger bundles of items that are functioning differentially.

OPEN IN VIEWER

Figure 4.

Estimation bias obtained when using SIBTEST to test for DIF and DBF across all conditions studied

So, does this estimation bias result in statistically significant differences in ability estimation for reference and focal groups? This question seems to get at the heart of establishing DIF or DBF effect size guidelines for SIBTEST because, after all, it is overall performance differences on achievement tests which initiated the development of DIF detection procedures (Angoff, 1993) and, despite all our attempts to ensure that tests are fair to all examinees through DIF and DBF analyses, the achievement gap still persists today. Figure 5 shows the rejection rates obtained when comparing the average estimated ability for the reference and focal groups under various DIF/DBF conditions studied using simple t tests. As the figure illustrates, the relationship between finding statistically significant differences in the average ability estimates for the reference and focal groups is highly dependent on two things: (a) the sum of DIF or DBF present in the items and (b) test length. In other words, it is the proportion of DIF or DBF, defined as the sum of DIF divided by the number of test items, that is indicative of whether mean ability estimates will statistically differ for reference- and focal-group examinees, not simply the presence of DIF or DBF. In addition, as the sum of DIF increases, having more items in a bundle will lead to finding statistically significant differences in average ability estimates for the reference and focal groups more often, as long as the proportion of DBF is high.

OPEN IN VIEWER

Figure 5.

T test rejection rates obtained when comparing the average estimated ability for the reference and focal groups under various DIF/DBF conditions studied

So how can these results be used to establish effect size guidelines for SIBTEST? In the conditions explored in this study, β ^ was almost perfectly related to the sum of DIF or DBF (r = .97). In practice, although one would know β ^ , one would not know the sum of DIF or DBF. Therefore, a regression model was fit to predict the sum of DIF/DBF, given β ^ and the number of items in a bundle, and these results are depicted in Table 2. With a little algebraic manipulation, this model can be used in reverse, after running a SIBTEST analysis, to estimate the sum of DIF. Specifically, since

β ^ = − 0 . 011 + 0 . 155 ( Sum of DIF ) + 0 . 015 ( Number of Items in Bundle ) ,

then

Sum of DIF = ( β ^ + 0 . 011 − 0 . 015 ( Number of Items in Bundle ) ) / 0 . 155

OPEN IN VIEWER

Table 2.

Regression Model Obtained When Regressing Sum of DIF and Number of DIF Items on Beta-Uni Statistic From SIBTEST

	Unstandardized coefficients		Standardized coefficients
	B	SE	β	t	Significance
Constant	−.011	.005		−2.065	.041
Sum of DIF	.155	.003	.902	53.666	<.001
No. of DIF items	.015	.001	.187	11.140	<.001

Note. DIF = differential item functioning; SIBTEST = simultaneous item bias test; SE = standard error. Dependent variable: Beta-uni statistic.

Dividing the Sum of DIF by the total number of items on the test will provide an estimate of the proportion of DIF which can be used as an estimate of how much the DIF or DBF will affect ability estimation. However, this estimate needs to be interpreted according to the number of items that are found to be functioning differentially, given that greater ability estimation bias is associated with having more items in a bundle (see Figure 3). According to the results obtained from this study, when the proportion of DBF is 0.15 (i.e., 3/20, where 3 = the sum of DIF and 20 = the total number of items on the test, which is the highest proportion of DIF/DBF considered in this study) then statistically significant ability differences will be found between reference- and focal-group examinees about 60% of the time for a five-item bundle and about 40% of the time for a three-item bundle, which is likely cause for concern. Moreover, this trend seems to increase exponentially, as illustrated in Figure 5.

How do these results compare with the Roussos and Stout (1996) effect size guidelines for SIBTEST which state that | β ^ | ≥ 0 . 088 is indicative of large DIF? It is clear that using these guidelines will typically correspond to statistically significant DIF findings when using SIBTEST. However, assuming this occurs for a one-item bundle, this value corresponds to a sum of DIF of only about 0.54, that is, 0.088 + 0.011 − 0.014(1)/0.155 = 0.54. Taking this one step further and dividing 0.54 by 0.15, to determine the test length at which this would result in a statistically significant difference in average ability between reference- and focal-group examinees about 60% of the time, indicates that this would be of concern only if a test consisted of a mere 3.6 items.

Real-Data Examples

To demonstrate how these results might be applied in a practical situation, the results of two previous studies that used a DBF analytic framework were considered. In the first study, published by Walker and Beretvas (2001), Poly-SIBTEST was used to determine if open-ended items on a mathematics test that required examinees to explain their answers in writing functioned differentially in favor of examinees who were proficient in writing. In this study, the average β ^ , over 10 replications, was β ^ = 1.18 for fourth-grade students and β ^ = 2.31 for seventh-grade students. For both grades, there were 16 items included in the bundle. Therefore, for fourth-grade students the Sum of DIF = (1.1.8 + 0.011 − 0.015(16))/0.155 = 6.14, whereas for seventh-grade students the Sum of DIF = (2.31 + 0.011 − 0.015(16)/0.155 = 13.43. The fourth-grade test consisted of a total of 40 items, whereas the seventh-grade test consisted of a total of 46 items. Therefore, for the fourth-grade test, the proportion of DBF is 6.14/40 = 0.15, whereas for the seventh-grade test, the proportion of DBF is 13.43/46 = 0.29. Interpreting these results in terms of this study, 0.15 was the highest proportion of DBF considered in this study. Moreover, statistically significant differences in ability estimates were more likely to be found between the reference and focal groups when a greater number of items were included in a bundle. In fact, the highest number of items considered in a bundle was only 5 and when the proportion of DBF was 0.15 for a five-item bundle, statistically significant differences were found between reference- and focal-group examinees approximately 60% of the time. Therefore, both these findings should be a cause of concern from an effect size perspective.

In the second study, published by Walker et al. (2008), SIBTEST was used to determine if mathematics items that required a higher reading load functioned differentially in favor of proficient readers. In this study, the highest β ^ (0.695) was found when comparing fifth-grade students who were classified as basic readers with fifth-grade students who were classified as advanced readers. For this analysis, the bundle consisted of 15 items. Therefore, Sum of DIF = (0.695 + 0.011 − 0.015(15)/0.155 = 0.481. The total number of items on the fifth-grade test was 32. Therefore, the proportion of DBF for this analysis is only 0.481/32 = 0.015, which according to the results of the simulation study is likely not a cause for concern.

Discussion

The primary purpose of this study was to determine effect size guidelines when using SIBTEST to conduct DBF analyses. This was accomplished by linking the original impetus for conducting DIF analyses—the fact that many believed that large-scale tests were unfair to minority students—to ability estimation since the achievement gap has lingered despite attempts by psychometricians to ensure that tests are fair to all by conducting DIF/DBF analyses. Obviously, there is not one definitive reason as to why the achievement gap persists, but the results of this study clearly demonstrate that only with very high levels of DIF or DBF, in terms of the proportion of DIF/DBF with respect to the total number of items on the test, can these disparities be caused by a series of items that are functioning differentially against the disadvantaged groups. This finding suggests that scholars must begin to look outside the statistical properties of the test in an effort to develop explanations for the lingering achievement gap. Ability estimation bias can only be attributed to DIF or DBF when a large number of items in a bundle are functioning differentially against focal examinees in a small way or a small number of items are functioning differentially against focal examinees in a large way. In either of these situations, the presence of DIF or DBF should be a cause for concern because it would lead one to erroneously believe that distinct groups differ in ability when in fact they do not. Given that large-scale test items are routinely examined for DIF, it is unlikely that this would occur in practice. It may be that the differential test performance between distinct groups is signaling a much broader and dire problem with the American educational system that extends far beyond the test.

It should be noted that the findings of this study are somewhat contingent on an adequate sample size and equal reference- and focal-group examinees. Future research is needed, in terms of investigating how overall sample size and the ratio of reference- and focal-group examinees affect the magnitude of β ^ . Previous research indicates that power would decrease, in terms of finding DIF or DBF with smaller sample sizes and unequal ratios. However, it is unclear if smaller sample sizes and unequal ratios would affect the magnitude of β ^ , which would in turn affect a practitioner’s estimate of the Sum of DIF/DBF, or if these conditions simply affect the standard error associated with β ^ .

Finally, it is unclear why the well-known effects of amplification were only partially supported. The results of this study suggest that amplification only occurs with a large magnitude of DIF/DBF. Although this was unexpected, it was not the primary focus of this study. Therefore, future research is needed in this area to determine under what conditions amplification will occur in practice.