The Power of Crossing SIBTEST

Abstract

Crossing SIBTEST or CSIB is designed to detect crossing differential item functioning (DIF) as well as unidirectional DIF. A theoretical formula for the power of CSIB is derived based on the asymptotic distribution of the test statistic under the null and alternative hypotheses. The derived power formula provides insights on the factors that influence the CSIB power, including DIF effect size, standard error, and sample size. The power formula and those influencing factors are further discussed in the context of the item response theory (IRT) three parameter logistic model (3PL) model. Simulation results show the consistency between the theoretical power and the observed rejection rate. The power of CSIB is compared with the unidirectional SIBTEST in theory and through simulation.

Keywords

DIF Crossing SIBTEST SIBTEST power

Introduction

In item response theory, an item with differential item functioning (DIF) or item bias will have different item characteristic curves (ICCs) for examinees from different groups (usually a reference group and a focal group). DIF describes the fact that examinees with the same ability may have different probabilities of getting the item correct, depending on which group they come from. When the item favors one group over the other over the ability range of interest, the DIF is unidirectional (Figure 1A). When the item favors one group in one ability range and favors the other group in another ability range, crossing in ICCs occurs, and there is crossing DIF (Figure 1B). Some DIF procedures such as the Mantel–Haenszel (MH) test (Holland & Thayer, 1988; Mantel &Haenszel, 1959) and SIBTEST (Shealy & Stout, 1993) are designed for detecting unidirectional DIF, but not designed for crossing DIF. Other DIF procedures such as logistic regression DIF test (Swaminathan & Rogers, 1990) and Crossing SIBTEST(Li & Stout, 1996) can handle crossing DIF.

Figure 1.

Unidirectional DIF and crossing DIF: (A) ICCs for a unidirectional DIF item, (B) ICCs for a crossing DIF item, (C) difference in ICCs for a unidirectional DIF item, and (D) difference in ICCs for a crossing DIF item.

In this article, a theoretical formula for the power of Crossing SIBTEST (CSIB) will be derived. A power formula for CSIB can help in planning the design of a DIF study that uses the CSIB procedure by providing the means for sample size calculation. It also provides insight on the factors that influence the power of CSIB. The derivation of the power formula is based on the large sample distribution of the test statistic under the null hypothesis and under the alternative hypothesis. In the power formula, there are three components that determine the power of a statistical test: effect size, standard error, and critical value. Based on the power formula, our focus will be on the discussion of factors including item characteristics (difficulty, discrimination, guessing, etc.) and examinee characteristics (ability distribution) that may affect the power of the CSIB, and how they affect the three components: effect size, standard error and the critical value. We also would like to draw a comparison with the unidirectional SIBTEST (USIB), for which the power formula was derived in Z. Li(2014). There are similarities between CSIB and USIB in effect size and the standard error; however, the critical value for CSIB is more complicated. Although CSIB was designed for detecting crossing DIF where USIB may lack power, CSIB can nonetheless be used for unidirectional DIF detection (Li & Stout, 1996). With the power formulas for CSIB and USIB, the question whether it is justifiable to use CSIB for unidirectional DIF in preference to USIB can be answered. Recently, Chalmers (2018) provided a new development in crossing SIBTEST (CSIB2). In this article, the power formula for CSIB will be focused on. For CSIB2, the issues related to the power calculation will be discussed, but a full development of power formula for CSIB2 is out of the scope of this article.

The rest of the article is organized as follows. The CSIB procedure is introduced, and then the null and alternative distributions of the CSIB statistic are discussed. The power formula for CSIB test is then derived based on the asymptotic distribution of the CSIB statistic. The formula is then applied to the item response theory (IRT) model and the factors that affect the power are discussed. Simulation studies are conducted to compare the power values from the formula and the actual rejection rates observed in the simulation.

Crossing SIBTEST

According to Shealy and Stout (1993), SIBTEST defines the average amount of (unidirectional) DIF for a dichotomous studied item by

\begin{array}{l} β_{u n i} = \int_{- \infty}^{\infty} [E_{R} (Y | θ) - E_{F} (Y | θ)] f (θ) d θ \\ = \int_{- \infty}^{\infty} [P_{R} (θ) - P_{F} (θ)] f (θ) d θ, \end{array}

(1)

where $P_{R} (θ)$ and $P_{F} (θ)$ are the ICCs for the reference and the focal groups, respectively, and $f (θ)$ is the density function for the pooled latent distribution. It is the weighted total signed area between the two ICCs where the weight is $f (θ)$ . Note that the area between ICCs is a signed-area where the part above 0 is positive and the part below 0 is negative. Although the signed area is useful for describing the unidirectional DIF, it is not suited for crossing DIF because positive and negative areas may cancel out.

On the contrary, Crossing SIBTEST defines the DIF effect by the weighted absolute area between ICCs (Li & Stout, 1996). Assuming there is one crossing point for the two ICCs at $θ = θ_{c}$ , the crossing DIF size is

\begin{matrix} β_{cro} = \int_{- \infty}^{θ < θ_{c}} [E_{R} (Y | θ) - E_{F} (Y | θ)] f (θ) d θ - \int_{θ > θ_{c}}^{\infty} [E_{R} (Y | θ) - E_{F} (Y | θ)] f (θ) d θ \\ = \int_{- \infty}^{θ < θ_{c}} [P_{R} (θ) - P_{F} (θ)] f (θ) d θ - \int_{θ > θ_{c}}^{\infty} [P_{R} (θ) - P_{F} (θ)] f (θ) d θ . \end{matrix}

(2)

Note that $| β_{cro} | = \int_{- \infty}^{\infty} | P_{R} (θ) - P_{F} (θ) | d θ$ . Therefore by definition, it is guaranteed that $β_{cro}$ has an absolute value no smaller than $β_{uni}$ . The two-sided test for DIF is considered where the null and the alternative hypotheses are

H_{0} : β = 0 vs . H_{1} : β \neq 0,

where $β$ is $β_{cro}$ for CSIB and $β_{uni}$ for USIB.

CSIB and USIB use the total score of the matching subtest (consisting of only non-DIF items) to stratify examinees in different ability groups, where examinees having the same matching score are considered to be in the same ability level. In each stratum, the difference in probability of correct response to the item between the reference and focal groups is estimated by difference in the proportions of correct response.

USIB estimates the DIF effect by

{\hat{β}}_{uni} = \sum_{k = 0}^{m} p_{k} ({\bar{Y}}_{Rk}^{*} - {\bar{Y}}_{Fk}^{*}),

(3)

where ${\bar{Y}}_{Rk}^{*}$ and ${\bar{Y}}_{Fk}^{*}$ are the adjusted average scores of the studied item(s) for the reference and the focal groups that achieve the same total matching score $k$ , which ranges from 0 to the total number of items $m$ , and $p_{k}$ is the proportion of examinees in the stratum with matching score $k$ . The average scores are adjusted by a regression correction method to remove the inflating effect of impact due to the measurement error introduced by using the matching score instead of the unknown true ability to match the examinees (Jiang & Stout, 1998; Shealy & Stout, 1993). For hypothesis testing, a standardized z-score statistic is formulated by dividing ${\hat{β}}_{uni}$ by its standard error:

B_{uni} = \frac{{\hat{β}}_{uni}}{{\hat{σ}}_{β}},

(4)

and the standard error is estimated by

{\hat{σ}}_{β} = {[\sum_{k = 0}^{m} p_{k}^{2} (\frac{{\hat{σ}}_{R}^{2} (Y | k)}{N_{Rk}} + \frac{{\hat{σ}}_{F}^{2} (Y | k)}{N_{Fk}})]}^{\frac{1}{2}},

(5)

where ${\hat{σ}}_{R}^{2} (Y | k)$ and ${\hat{σ}}_{F}^{2} (Y | k)$ are the sample variances of the scores for examinees in stratum $k$ from the reference group and the focal group; $N_{Rk}$ and $N_{Fk}$ are the corresponding sample sizes. For large sample size, the standard error estimate converges to the population parameter (Z. Li, 2014):

σ_{β} = {[\int_{- \infty}^{\infty} f {(θ)}^{2} (\frac{σ_{R}^{2} (Y | θ)}{N_{R} f_{R} (θ)} + \frac{σ_{F}^{2} (Y | θ)}{N_{F} f_{F} (θ)}) d θ]}^{\frac{1}{2}},

(6)

where $σ_{R}^{2} (Y | θ)$ and $σ_{F}^{2} (Y | θ)$ are the population variances of the scores for examinees with latent trait value $θ$ ; $N_{R}$ and $N_{F}$ are total sample sizes for the reference and focal groups; $f_{R} (θ)$ and $f_{F} (θ)$ are the density for latent trait distributions for reference and focal groups, so pooled distribution density $f (θ) = (N_{R} / N) f_{R} (θ) + (N_{F} / N) f_{F} (θ)$ where $N = N_{R} + N_{F}$ . Under the null hypothesis of no DIF, $B_{uni}$ has a large-sample distribution of the standard normal distribution: $B_{uni} | H_{0} ~ N (0, 1)$ . Therefore, the null hypothesis is rejected if $| B_{uni} | > z_{α / 2}$ , where the critical value $z_{α / 2}$ is the $(1 - α / 2) th$ quantile for the standard normal distribution (e.g., for $α = 0.05$ , $z_{α / 2} = 1.96$ ).

In CSIB, the first step is to obtain an estimate of the crossing score $k_{c}$ , and after that the DIF effect is estimated by

{\hat{β}}_{cro} = \sum_{k = 0}^{k_{c} - 1} p_{k} ({\bar{Y}}_{Rk}^{*} - {\bar{Y}}_{Fk}^{*}) - \sum_{k = k_{c} + 1}^{m} p_{k} ({\bar{Y}}_{Rk}^{*} - {\bar{Y}}_{Fk}^{*}) .

(7)

And the CSIB statistic is given by

B_{cro} = \frac{{\hat{β}}_{cro}}{{\hat{σ}}_{β}},

(8)

where ${\hat{σ}}_{β}$ is the same as in the USIB statistic. However, unlike $B_{uni}$ in USIB, there is no easily derived large sample distribution for the CSIB statistic $B_{cro}$ under the null hypothesis, as pointed out in Li and Stout (1996). Therefore, randomization tests are used to obtain the p-value, and the null hypothesis of no DIF is rejected if the p-value is less than the level of the test, for example, $p < . 05$ . In the case that the null is rejected and the estimated crossing score is outside of the range of matching score, it is further determined the DIF is unidirectional, whereas a crossing score within the range would indicate a crossing DIF. This way both unidirectional and crossing DIF can be detected by CSIB. Note that the rule for rejecting the null hypothesis is determined based on the null distribution for the CSIB statistic, which will be studied next.

The Null and Alternative Distributions of $B_{cro}$ Statistic

Let $d_{k}^{*} = {\bar{Y}}_{Rk}^{*} - {\bar{Y}}_{Fk}^{*}$ , CSIB estimates the crossing point by fitting a line to the points $(k, d_{k}^{*})$ by the weighted least squares method where the weight is the sample size in each matching score group,

d_{k}^{*} = \overset{\land}{β_{0}} + \overset{\land}{β_{1}} k k = 0, 1, \dots, m

and the point at which the regression line crosses the x-axis is the estimated crossing point:

k_{c} = - \frac{\overset{\land}{β_{0}}}{\overset{\land}{β_{1}}},

and for convenience the nearest integer of $- \overset{\land}{β_{0}} / \overset{\land}{β_{1}}$ is used as the estimate for $k_{c}$ , and if the crossing point is below 0 or above $m$ , the estimate is set to $0$ or $m$ , respectively.

Figure 2A, 2C, and 2E shows how this algorithm works for a non-DIF item, a crossing DIF item, and a unidirectional DIF item, respectively. For a non-DIF item, $β_{1} = 0$ and $β_{0} = 0$ , so $k_{c}$ is a ratio of two independent normal variable with mean 0, which results in a Cauchy random variable. Figure 2B shows the empirical distribution of the CSIB statistic obtained from 10,000 replicated datasets simulated for a non-DIF item. One can immediately see that the null distribution for the CSIB statistic is not the standard normal distribution, but a bimodal distribution that is symmetric to 0. A mixture of normal distribution with two symmetric components $(1 / 2) N (μ, σ^{2}) + (1 / 2) N (- μ, σ^{2})$ is fitted to the simulated values, and the fitted distribution density curve agrees well with empirical density (i.e., the histogram). Based on this null distribution for CSIB statistic, the critical value $c_{α}$ for a Type I error $α$ is the value such that the tail area is $α$ . And this critical value $c_{α}$ is larger than the critical value obtained under the standard normal distribution. For example, $α = 0.05$ , the critical value obtained from the empirical null distribution for $| B_{cro} |$ in Figure 2B is 2.3, which is larger than critical value 1.96 for $B_{uni}$ . The null distribution of the CSIB statistic does not have a closed form but can be simulated using the Monte Carlo method as long as the generating IRT model for the null model is known. Data simulated with typical parameter values for the three-parameter logistic model (3PL) model show that the critical values for $| B_{cro} |$ are close to 2.3.

Figure 2.

Distributions of CSIB statistics for a non-DIF, a crossing DIF, and a unidirectional DIF item: (A) identification of the crossing point k_c under the null hypothesis, (B) distribution of CSIB statistic under the null hypothesis, (C) identification of the crossing point k_c for a crossing DIF item, (D) distribution of CSIB statistic for a crossing DIF item, (E) identification of the crossing point k_c for a unidirectional DIF item, and (F) distribution of CSIB statistic for a unidirectional DIF item.

As the CSIB statistic does not follow the standard normal distribution, Li and Stout (1996) used a randomization procedure to determine the significance of the CSIB test. In the randomization procedure, each $d_{k}^{*}$ is multiplied by $+ 1$ or $- 1$ randomly, and a CSIB statistic $B_{cro}^{(t)}$ is calculated from the values with signs randomly shuffled. A randomization null distribution is constructed by the CSIB statistics calculated from many iterations of the randomization (Li and Stout recommended 1,000 iterations), and the randomization p-value is obtained by the proportion of the iterations that have $| B_{cro}^{(t)} |$ greater than the $| B_{cro} |$ value obtained from the original data. The null hypothesis is rejected if the randomization p-value is less than the user specified level of significance, for example, .05. Simulation studies in Li and Stout (1996) showed that the randomization tests control the Type I error of CSIB procedure. This is not surprising because in essence the randomization tests construct a null distribution for the CSIB statistic, which approximates the true null distribution, and identify the critical value from the randomization null distribution. Under the null hypothesis of no DIF, this randomization null distribution is equivalent to the empirical null distribution described in previous paragraph, thus the corresponding critical value for CSIB statistic $| B_{cro} |$ would be the same. (However, as revealed later in simulation study section, this equivalence does not hold under the alternative hypothesis, which causes the randomization test lose some power.)

Under the alternative hypothesis, the asymptotic distribution for $B_{cro}$ for a crossing DIF item follows a normal distribution with standard deviation 1(Figure 2D). For a unidirectional DIF item, the asymptotic distribution for $B_{cro}$ follows a normal distribution with standard deviation 1(Figure 2F).

Power Formula for Crossing SIBTEST

For the two-sided test of the hypotheses $H_{0} : β_{cro} = 0 versus H_{1} : β_{cro} \neq 0$ , the null hypothesis is rejected if the CSIB statistic is in the rejection region $| B_{cro} | > c_{α}$ , where $c_{α}$ is the critical value. By definition, the Type I error is the probability of rejecting the null given that the null hypothesis is true: $Type I error = P (| B_{cro} | > c_{α} | H_{0}) = α$ , and the power is the probability of rejecting the null given that the alternative hypothesis is true: $power = P (| B_{cro} | > c_{α} | H_{1}) .$ Under the alternative hypothesis, the CSIB statistic asymptotically follows a normal distribution: $B_{cro} | H_{1} ~ N (β_{cro} / σ_{β}, 1)$ , so the power is

\begin{matrix} power = P (B_{cro} > c_{α}) + P (- B_{cro} < - c_{α}) \\ = Φ (\frac{| β_{cro} |}{σ_{β}} - c_{α}) + Φ (\frac{- | β_{cro} |}{σ_{β}} - c_{α}), \end{matrix}

(9)

where $Φ (.)$ is the cumulative distribution function for the standard normal distribution. Figure 3 illustrates the Type I error and power for the CSIB statistic. The tail areas formulated by the rejection region under the null distribution are symmetric to zero because the null distribution is symmetric, and the total area is the Type I error $α$ . The same rejection region forms the tail areas under the alternative distribution that are not symmetric and the total area is the power. When the DIF size $| β_{cro} |$ is large, the tail area opposite to the sign of $β_{cro}$ is very small and can often be ignored.

Figure 3.

Null and alternative distributions for CSIB statistic.

The following equation is derived by substituting the values for $β$ and $σ_{β}$ as shown in Equations (2) and (6)

\begin{matrix} power = Φ (\frac{\int_{- \infty}^{\infty} | E_{R} (Y | θ) - E_{F} (Y | θ) | f (θ) d θ}{{[\int_{- \infty}^{\infty} f {(θ)}^{2} (\frac{σ_{R}^{2} (Y | θ)}{N_{R} f_{R} (θ)} + \frac{σ_{F}^{2} (Y | θ)}{N_{F} f_{F} (θ)}) d θ]}^{\frac{1}{2}}} - c_{α}) \\ + Φ (\frac{- \int_{- \infty}^{\infty} | E_{R} (Y | θ) - E_{F} (Y | θ) | f (θ) d θ}{{[\int_{- \infty}^{\infty} f {(θ)}^{2} (\frac{σ_{R}^{2} (Y | θ)}{N_{R} f_{R} (θ)} + \frac{σ_{F}^{2} (Y | θ)}{N_{F} f_{F} (θ)}) d θ]}^{\frac{1}{2}}} - c_{α}) . \end{matrix}

(10)

In the case that $Y$ is the score of a dichotomous item, $E_{g} (Y | θ) = P (Y = 1 | θ, g) = P_{g} (θ),$ $g = R$ or $F$ , where $P_{g} (θ)$ is the IRF for group $g$ ; and $σ_{g}^{2} (Y | θ) = Var (Y | θ, g) = P_{g} (θ) Q_{g} (θ)$ , where $Q_{g} (θ) = 1 - P_{g} (θ)$ . Also, let $N = N_{R} + N_{F}$ be the total sample size of the combined groups, and $γ_{R} = N_{R} / N$ and $γ_{F} = N_{F} / N$ be the sample proportions for the reference and focal groups. Then the power formula for CSIB with a single studied dichotomous item is

\begin{matrix} power = Φ (\frac{\int_{- \infty}^{\infty} | P_{R} (θ) - P_{F} (θ) | f (θ) d θ}{{[\int_{- \infty}^{\infty} f {(θ)}^{2} (\frac{P_{R} (θ) Q_{R} (θ)}{N γ_{R} f_{R} (θ)} + \frac{P_{F} (θ) Q_{F} (θ)}{N γ_{F} f_{F} (θ)}) d θ]}^{\frac{1}{2}}} - c_{α}) \\ + Φ (\frac{- \int_{- \infty}^{\infty} | P_{R} (θ) - P_{F} (θ) | f (θ) d θ}{{[\int_{- \infty}^{\infty} f {(θ)}^{2} (\frac{P_{R} (θ) Q_{R} (θ)}{N γ_{R} f_{R} (θ)} + \frac{P_{F} (θ) Q_{F} (θ)}{N γ_{F} f_{F} (θ)}) d θ]}^{\frac{1}{2}}} - c_{α}) . \end{matrix}

(11)

The power formula for CSIB is similar to the formula for USIB. In Equation (9), if $β_{cro}$ is replaced by $β_{uni}$ , and $c_{α}$ by the critical value for standard normal distribution, the formula for SIBTEST can be derived (Z. Li, 2014).

Factors That Influencing CSIB Power

From the power formula, it can be seen that the power is determined by DIF effect size, standard error and critical value:

1. The size of the DIF effect $| β_{cro} | = \int_{- \infty}^{\infty} | P_{R} (θ) - P_{F} (θ) | f (θ) d θ$ .

2. The standard error parameter $σ_{β}$ , which is determined by the following factors:

a. The total sample size $N$ ;

b. The proportion of the focal group and reference group $γ_{F}$ and $γ_{R} = 1 - γ_{F}$ in the sample;

c. The distributions of the latent trait $f_{R} (θ)$ and $f_{F} (θ)$ ;

d. The IRF of the studied item, $P_{R} (θ)$ and $P_{F} (θ)$

3. The Type I error rate $α$ which determines the critical value $c_{α}$ .

Of these factors, only sample size $N$ and the proportion of composition $γ_{F}$ can be manipulated. Other factors are the item characteristics or examinee characteristics that are considered fixed once the study item and the target populations are determined. The power always increases as total sample size $N$ increases. If total sample size $N$ is fixed, a balanced design with $γ_{F} = γ_{R} = 0.5$ will always have a larger power than an unbalanced design. If one of the group sizes, for example, $N_{F}$ , is fixed, increasing the other group size $N_{R}$ will always increase the power because the total sample size is larger.

Note that in deriving the CSIB power formula (Equation 11), no assumption was made about the form of the item response functions, $P_{R} (θ)$ and $P_{F} (θ)$ , the latent trait distributions, $f_{R} (θ)$ and $f_{F} (θ)$ . Therefore, the formula is applicable to any parametric or nonparametric IRT models and latent trait distributions. In general, the integrals in the formula do not have a closed form but can always be evaluated numerically. The numerical solution is good enough for conducting the power and sample size calculation in planning a DIF study. However it is hard to get much insight into how the effect size, the standard error, and the power are influenced by the item parameters and person parameters in an IRT model. Next, a specific IRT 3PL model should be considered where a closed-form approximate formula can be derived, and such insight can be seen from the closed-form solution.

Closed-Form Formulas Under IRT 3PL Model With $f_{R} (θ) = f_{F} (θ) ~ N (μ, σ^{2})$

Consider the IRT 3PL model for dichotomous items where the IRF is given by

P (θ; a, b, c) = c + (1 - c) \frac{\exp (1.7 a (θ - b))}{1 + \exp (1.7 a (θ - b))},

(12)

where $θ$ is the latent trait, $a$ is the discrimination parameter, $b$ is the difficulty parameter, and $c$ is the guessing parameter. Assuming that the guessing parameters are the same for the two groups, $c_{R} = c_{F} = c$ , the ICCs for the two groups cross at

θ_{c} = \frac{a_{F} b_{F} - a_{R} b_{R}}{a_{F} - a_{R}} .

(13)

When there is no difference in discrimination parameter and a difference in difficulty parameter, the ICCs do not intersect and the DIF is uniform. Also, it is possible that even when there is a difference in discrimination parameter, the intersection point $θ_{0}$ is outside the range of interest of the latent trait distribution and the DIF is essentially considered as unidirectional and not crossing. If it is further assumed that the two groups both have the same normal latent trait distribution with mean $μ$ and standard deviation $σ$ : $f_{R} (θ) = f_{F} (θ) = f (θ) ~ N (μ, σ^{2})$ , the effect size $β$ can been shown (proof online in Supplemental Appendix A) to have a closed-form approximate formula

\begin{matrix} β_{cro} = \int_{- \infty}^{\infty} | P_{R} (θ) - P_{F} (θ) | f (θ) d θ \\ \approx (1 - c) \frac{1}{\sqrt{a^{- 2} + σ^{2}}} ϕ (\frac{b - μ}{\sqrt{a^{- 2} + σ^{2}}}) \times {Δ b \times B + (\frac{Δ a}{a}) \times A}, \end{matrix}

(14)

where

\begin{matrix} B = 2 Φ (\frac{θ_{0} - μ_{P}}{σ_{P}}) - 1, \\ A = \frac{b - μ}{a^{2} σ^{2} + 1} - 2 (b - μ_{P}) [1 - Φ (\frac{θ_{0} - μ_{P}}{σ_{P}})] + 2 σ_{P}^{2} \frac{1}{σ_{P}} ϕ (\frac{θ_{0} - μ_{P}}{σ_{P}}), \\ μ_{P} = \frac{σ^{2} b + a^{- 2} μ}{a^{- 2} + σ^{2}} and σ_{P} = \sqrt{\frac{a^{- 2} σ^{2}}{a^{- 2} + σ^{2}}} . \end{matrix}

In the formula, $a,$ $b,$ and $c$ are the item parameters for the reference group, and $Δ b = b_{F} - b_{R}$ and $Δ a = a_{F} - a_{R}$ are differences of the item parameters between the focal and the reference groups.

From Equation (14), how the DIF effect size $β_{cro}$ is influenced by the IRT parameters can be seen. The presence of guessing parameter $c$ reduces the DIF effect size by a factor of $(1 - c)$ . The crossing DIF effect can be decomposed into two parts in the bracket: the uniform DIF which is proportional to the difference in difficulty $Δ b$ , and the nonuniform DIF which is proportional the relative difference in discrimination $Δ a / a$ . The DIF effect is also influenced by the relative location of the item difficulty and the latent distribution. Effect size $β_{cro}$ is large when an item’s difficulty is close to the mean ability of the population, that is, $| b_{R} - μ |$ is small. When an item is too easy or too difficult relative the target population, the effect size $β_{cro}$ becomes very small and it is hard to detect DIF.

When the reference and focal groups have the same latent trait distribution, the standard error $σ_{β}$ can be approximated by its value evaluated under the null hypothesis

σ_{β} \approx {[\frac{1}{N γ_{R} γ_{f}} \int_{- \infty}^{\infty} P_{R} (θ) Q_{R} (θ) f (θ) d θ]}^{\frac{1}{2}} .

Besides the sample size $N$ and the proportion $γ_{F}$ , other factors that influence the standard error are reflected in the integral $\int_{- \infty}^{\infty} P_{R} (θ) Q_{R} (θ) f (θ) d θ .$ The integrand has two parts: $P_{R} (θ) Q_{R} (θ)$ is related to the item characteristics and $f (θ)$ is the latent trait distribution.

For the 3PL model with item parameters $a$ , $b$ , and $c$ , and assuming $f (θ) ~ N (μ, σ^{2})$ , a closed-form approximation for the standard error has been derived (see online Supplemental Appendix):

σ_{β} \approx {[\frac{1 - c}{N γ_{R} γ_{F}} {c Φ (\frac{b - μ}{\sqrt{a^{- 2} + σ^{2}}}) + (\frac{1 - c}{1.7 a}) \frac{1}{\sqrt{a^{- 2} + σ^{2}}} ϕ (\frac{b - μ}{\sqrt{a^{- 2} + σ^{2}}})}]}^{\frac{1}{2}} .

(15)

If the closed-form formulas for $β_{cro}$ and $σ_{β}$ are substituted into Equation (9), a closed-form power formula can be derived.

Power Formulas for Decision Rule Based on DIF Effect Size ${\hat{β}}_{cro}$

When the p-value is less than .05, the null hypothesis is rejected and the item is declared to have DIF. However, when sample size is very large, the test may give a small p-value even if the observed DIF size is small and of no practical significance. In practice, when making a decision whether an item has DIF, it is often useful to also consider the actual DIF effect size ${\hat{β}}_{cro}$ . For example, Dorans and Kulick (1986) developed standard p-difference, which can be considered as a special case of SIBTEST statistic. Standard p-difference uses a decision rule only based on the DIF effect size, where an item is declared as a DIF item if the DIF effect size $| \hat{β} |$ is greater than some prespecified value, for example $| \hat{β} | > 0.05$ .

Consider the decision rule that declares a DIF item if $| {\hat{β}}_{cro} | > b_{c}$ , where $b_{c}$ is the prespecified critical value for ${\hat{β}}_{cro}$ (e.g., $b_{c} = 0.05)$ . As the asymptotic distribution for ${\hat{β}}_{cro}$ is $N (β_{cro}, σ_{β}^{2})$ , the power is derived by calculating probability to reject the null hypothesis:

\begin{matrix} power (rule : | {\hat{β}}_{cro} | > b_{c}) \\ = P ({\hat{β}}_{cro} > b_{c}) + P (- {\hat{β}}_{cro} < - b_{c}) \\ = Φ (\frac{| β_{cro} | - b_{c}}{σ_{β}}) + Φ (\frac{- | β_{cro} | - b_{c}}{σ_{β}}) . \end{matrix}

(16)

Then the formulas for $β_{cro}$ and $σ_{β}$ (Equations 2 and 6) to calculate the power can be substituted.

The power depends on the sample size only through the standard error $σ_{β}$ , which is inversely proportional to $\sqrt{N}$ . Consider the two decision rules: (a) reject null if $p < . 05$ , or equivalently $| B_{cro} | > 2.3$ , and (b) reject null if $| {\hat{β}}_{cro} | > 0.05$ . In the situation that the sample size $N$ is very large and the population DIF size $| β_{cro} |$ is positive, even if $| β_{cro} |$ is small, the power for decision rule $| B_{cro} | > 2.3$ $(p < . 05)$ will be close to 1 and the null is almost always rejected. On the other hand, when the population value $| β_{cro} |$ is less than 0.05, the power for decision rule $| {\hat{β}}_{cro} | > 0.05$ will be close to 0 for very large sample size, which means the null is almost always not rejected.

It is advisable that practitioners make decision rules based on both p-value and the DIF effect size to accommodate both the statistical significance and practical significance. For the decision rule that requires both $| B_{cro} | > c_{α}$ and $| β_{cro} | > b_{c}$ , the power formula is

\begin{matrix} power (Rule : | B_{cro} | > c_{α} and | {\hat{β}}_{cro} | > b_{c}) \\ = P ({\hat{β}}_{cro} > max (b_{c}, c_{α} σ_{β})) + P (- {\hat{β}}_{cro} < - max (b_{c}, c_{α} σ_{β})) \\ = Φ (\frac{| β_{cro} |}{σ_{β}} - max (\frac{b_{c}}{σ_{β}}, c_{α})) + Φ (- \frac{| β_{cro} |}{σ_{β}} - max (\frac{b_{c}}{σ_{β}}, c_{α})) . \end{matrix}

(17)

Simulation Study

Simulation Setup

In the simulation study, a test form of 75 items is considered. All the items follow the IRT 3PL model with guessing parameter $c = 0.2$ . Items 1 to 25 are items having no DIF $(a_{R} = a_{F}, b_{R} = b_{F})$ , Items 26 to 50 are items having crossing DIF $(a_{R} \neq a_{F}, b_{R} = b_{F})$ , and Items 51 to 75 are items having unidirectional DIF $(a_{R} = a_{F}, b_{R} \neq b_{F})$ . For each item, the reference group discrimination parameter value $a_{R}$ was chosen by a positive random draw from a normal distribution with mean 1.2 and variance 0.1 (i.e., SD = 0.32); the reference group difficulty parameter $b_{R}$ was drawn from a normal distribution with mean 0 and standard deviation 1. For non-DIF Items 1 to 25, the focal group difficulty parameter $b_{F}$ was set equal to $b_{R}$ , and the discrimination parameter $a_{F}$ was set equal to $a_{R}$ . For crossing DIF Items 26 to 50, $a_{F}$ was set equal to $a_{R} + Δ a$ , where $Δ a$ was chosen from the following five values: 0.1, 0.2, 0.3, 0.5, 1, with five items per value. For unidirectional DIF items 51 to 75, $b_{F}$ was set equal to $b_{R} + Δ b$ , where $Δ b$ is chosen from the following five values: −0.1, 0.1, −0.2, 0.2, −0.3, with five items per value. The item parameter values were recorded and the same items were used in 10,000 replicated simulation runs. In each run, a sample of $N_{R} = 1, 000$ examinees from the reference group and $N_{F} = 1, 000$ examinees from the focal group were simulated by drawing their ability parameter $θ$ from a $N (0, 1)$ distribution. Then for each examinee in the reference and focal groups, the dichotomous response to each of the 75 items was simulated from the IRT 3PL model. The simulated dataset was then analyzed by CSIB to detect DIF in each item, where the total score of the first 25 non-DIF items was used as the matching variable. Two ways of deciding statistical significance with level 0.05 were conducted and results compared: (a) randomization test as described in Li and Stout (1996), and rejecting the null if $p < . 05$ ; (b) rejecting the null if $| B_{cro} | > 2.3$ , the cutoff value as suggested by the empirical null distribution. In the analysis, USIB were also conducted and results were reported alongside the CSIB results. The decision rule for USIB was to reject the null if $p < . 05$ . Based on 10,000 replicated simulation runs, the rejection rate (= the number of times rejecting $H_{0}$ /10,000) was summarized for each item. This study used the source code for Crossing SIBTEST and SIBTEST written by the original authors of Li and Stout (1996), Shealy and Stout (1993), and Jiang and Stout (1998), which was free to download for public (DIF-Pack, n.d.); the source code written in FORTRAN was linked in R (R Core Team, 2017), and the simulation study and analysis were conducted in R.

Simulation Results

For Items 1 to 25 with no DIF, the rejection rate is an estimate of the Type I error rate, which is expected to be close to 0.05. For the DIF Items 26 to 50 (crossing DIF) and 51 to 75(unidirectional DIF), the rejection rate is an estimate of the true power, which will be used to check the validity of the power formulas. Figure 4A shows that the rejection rates for the non-DIF items (Type I error) are close to or below 0.05 which indicates the Type I error is well controlled by both the randomization test and the $| B_{cro} | > 2.3$ rule. Figure 4B shows the rejection rates, that is, the power, of the CSIB for each of the crossing DIF and unidirectional DIF items. It is notable that the $| B_{cro} | > 2.3$ rule consistently has higher power than the randomization test in those DIF items. This suggests that the randomization procedure somehow loses some power for DIF items as compared with the rejection rule based on Monte Carlo null distribution. A closer examination of the null distribution constructed by randomization procedure on a DIF item reveals that it does have longer tails (i.e., larger variability) than the Monte Carlo null distribution simulated under the null. The rejection region defined by randomization reference distribution ( $| B_{cro} |$ >97.5-percentile for $p < . 05$ ) would be outside the cutoff value of $2.3$ , thus it is less likely to reject the null. It is also revealed that the difference between randomization null distribution and Monte Carlo null distribution disappears when the randomization is conducted on a non-DIF item. Recall that in the CSIB randomization procedure, randomization samples are generated by randomly changing the signs of the difference scores $d_{k}^{*}$ . For a non-DIF item, the randomization does not change the variability of the $β_{cro}$ estimates in randomization samples, as given by Equation (5) which can be considered “within-matching-score” variability, since each $d_{k}^{*}$ already has an expected value 0. For a DIF item, however, the randomization would introduce an additional “between-matching-score” variability. Because the trend along the matching scores (i.e., the expected value of $d_{k}^{*}$ ) is non-zero, but the values are become now randomly shuffled above and below 0, and this part is not accounted for by the “within-matching-score” variability in Equation (5). This explains why the randomization distribution obtained from a DIF item have longer tails than the true null distribution.

Figure 4.

Observed rejection rate by CSIB in simulation data: (A) CSIB Type 1 error and (B) CSIB power.

For the DIF items, a comparison is made between the mean of estimated DIF effect size $| {\hat{β}}_{cro} |$ over the 10,000 replicates and theoretical value for $| β_{cro} |$ calculated from Equation (2), and there is a good agreement (Figure 5A). Theoretical values of the power calculated from Equation (11) are compared with observed rejection rates for the randomization test (Figure 5B) and the $| B_{cro} | > 2.3$ rule (Figure 5C). The observed rejection rates for the randomization test are slightly lower than the theoretical power values, which is due to the additional variability introduced by randomization that causes the loss of the power. On the contrary, CSIB test based on the $| B_{cro} | > 2.3$ rule has rejection rates that are close to theoretical values given by the power formula.

Figure 5.

Comparisons of Monte Carlo results with theoretical values in CSIB test: (A) DIF effect size $| β_{cro} |$ , (B) CSIB power (randomization test), and (c) CSIB power $(| β_{cro} | > 2.3)$ .

A Comparison Between CSIB and USIB

From the simulation results, it can be seen that CSIB is capable of detecting both crossing DIF and unidirectional DIF. Figure 6 shows the difference between the rejection rates of CSIB and USIB on the DIF items. For most of the items with crossing DIF (Items 26–50), CSIB has higher power than USIB, and in many cases, USIB lacks the power to detect the DIF. For the items with unidirectional DIF (Items 51–75), both CSIB and USIB have power to detect the DIF, but the power of CSIB is consistently lower than USIB. In the case of crossing DIF, CSIB has a larger effect size β than USIB because it accounts the unsigned total area difference between the two ICCs, thus lead to larger power than USIB to detect the DIF. In the case of unidirectional DIF, CSIB and USIB have the same population parameter for the effect size and the B statistic, but the critical value for CSIB is larger than that of USIB, which causes the probability of rejecting the null to be smaller for CSIB.

Figure 6.

A comparison of rejection rates: CSIB versus USIB.

Discussion

In this article, a formula for calculating the power of the Crossing SIBTEST procedure is derived. The derivation is based on the asymptotic distributions of the CSIB statistic under the null and alternative hypotheses, assuming the sample size is large and the test length is long. A comparison of the theoretical power from the formula and the observed rejection rate in the simulation study with moderate length (25 matching items) and moderate sample size (1,000 for each group) showed that the formula provides a good approximation to the power of CSIB. The factors that influence the power of the CSIB statistic in general and under a specific IRT 3PL model have been discussed. Power formulas for decision rules based on DIF effect size ${\hat{β}}_{cro}$ have been derived in a similar fashion, and the formulas clearly explain why the introduction of the effect size based rule helps the robustness in DIF detection when the sample size is extremely large.

The power formula helps explain the difference between the CSIB and the USIB. CSIB improves the power of DIF detection when there is crossing DIF by estimating the crossing DIF effect $β_{cro}$ which is larger in absolute value than the unidirectional DIF effect $β_{uni}$ . This improvement comes with the price of less power for unidirectional DIF than USIB. Because the null distribution for CSIB is more variable than the null distribution for USIB, the critical value for CSIB has to be larger than that for USIB in order to achieve the same level for Type I error, which results in less power for detecting unidirectional DIF by CSIB. A combined procedure using both CSIB and USIB is possible and cautions should be made to control the Type I error (see the online Supplement Appendix B on discussions on the combined procedure).

There is no closed form for the null distribution for CSIB statistic, so Li and Stout (1996) used a randomization test to calculate the p-value. The reference distribution constructed by the randomization test tends to have longer tails than the null distribution from direct simulation. This leads to a slight loss of power for the randomization test as compared with the value from the power formula. One possible solution to this problem is to use the empirical null distribution from Monte Carlo simulation to calculate the p-value, or equivalently, to use the cutoff value on CSIB test statistic (e.g., $| B_{cro} | > 2.3$ for the level .05) from the empirical null distribution.

The power formulas developed in this article are based on asymptotic distributions where large sample size and long test are assumed. It is not expected the formulas will work for short tests (e.g., test with five matching items).

Chalmers (2018) introduced new ideas in the crossing SIBTEST method and proposed a new method (CSIB2). In CSIB2, the p-value is calculated conditional on the location of the crossing point $k_{c}$ : (a) when $k_{c}$ is in range $(0 < k_{c} < m)$ , suggesting crossing DIF, calculate $B_{cro}$ statistic in two regions $(< k_{c} and \geq k_{c})$ separately: $B_{1}$ and $B_{2}$ , then use statistic $B^{2} = B_{1}^{2} + B_{2}^{2}$ , and use $χ_{2}^{2}$ as the null distribution to obtain the p-value; (b) if $k_{c}$ is outside of score range, suggesting unidirectional DIF, calculate $B_{cro}$ statistic in the whole region, then use statistic $B^{2}$ , and use $χ_{1}^{2}$ as its null distribution to obtain the p-value. For a level .05 test, the rejection region based on test statistic is (a) for $0 < k_{c} < m$ , $B^{2} > χ_{2, (α = 0.05)}^{2} = 5.99$ $(i . e ., | B | > 2.45)$ ; and (b) for $k_{c} \leq 0$ or $k_{c} \geq m$ , $B^{2} > χ_{1, (α = 0.05)}^{2} = 3.84$ $(i . e ., | B | > 1.96)$ . By using $χ_{1}^{2}$ and $χ_{2}^{2}$ as the null distribution in different conditions, CSIB2 seems to have avoided the difficulty of null distribution of in CSIB, thus no need to use the randomization test to calculate the p-value. However, the simulation study (in online Supplemental Appendix C) shows that the empirical null distribution of the CSIB2 test statistic is not $χ_{2}^{2}$ or $χ_{1}^{2}$ in those two conditions. Interestingly, despite the discrepancy between the nominal null distribution and the empirical null distribution, the empirical Type I error of CSIB2 is close to 0.05. Some explanations for this are given based on observations in simulation data (see online Supplemental Appendix C), but further investigations are needed to understand the null distribution of CSIB2 and development of a power formula for CSIB2 is possible.

Supplemental Material

Supplemantary_Appendix – Supplemental material for The Power of Crossing SIBTEST

Supplemental material, Supplemantary_Appendix for The Power of Crossing SIBTEST by Zhushan Li in Applied Psychological Measurement

Footnotes

Acknowledgements

The author thanks two anonymous reviewers and Associate Editor Dan Bolt for constructive comments and suggestions that helped improve the work.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Zhushan Li

Supplemental Material

Supplemental material for this article is available online.

References

Chalmers

R. P.

(2018). Improving the crossing-SIBTEST statistic for detecting non-uniform DIF. Psychometrika, 83(2), 376–386.

DIF-Pack. (n.d.). Source code for SIBTEST, POLY-SIBTEST and Crossing SIBTEST [Computer software]. https://psychometrics.onlinehelp.measuredprogress.org/tools/dif/

Dorans

N. J.

Kulick

(1986). Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the Scholastic Aptitude Test. Journal of Educational Measurement, 23(4), 355–368.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

(Eds.), Test validity(pp. 129–145). Lawrence Erlbaum.

Jiang

Stout

(1998). Improved Type I error control and reduced estimation bias for DIF detection using SIBTEST. Journal of Educational Statistics, 23(4), 291–322.

H.-H.

Stout

(1996). A new procedure for detection of crossing DIF. Psychometrika, 61(4), 647–677.

(2014). A power formula for the SIBTEST procedure for differential item functioning. Applied Psychological Measurement, 38(4), 311–328.

Mantel

Haenszel

(1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22(4), 719–748.

R Core Team. (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/

10.

Shealy

Stout

(1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159–194.

11.

Swaminathan

Rogers

H. J.

(1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.45 MB

The Power of Crossing SIBTEST

Abstract

Keywords

Introduction

Crossing SIBTEST

The Null and Alternative Distributions of B cro Statistic

Power Formula for Crossing SIBTEST

Factors That Influencing CSIB Power

Closed-Form Formulas Under IRT 3PL Model With f R ( θ ) = f F ( θ ) ~ N ( μ , σ 2 )

Power Formulas for Decision Rule Based on DIF Effect Size β ^ cro

Simulation Study

Simulation Setup

Simulation Results

A Comparison Between CSIB and USIB

Discussion

Supplemental Material

Supplemantary_Appendix – Supplemental material for The Power of Crossing SIBTEST

Footnotes

Acknowledgements

Declaration of Conflicting Interests

Funding

ORCID iD

Supplemental Material

References

Supplementary Material

The Null and Alternative Distributions of $B_{cro}$ Statistic

Closed-Form Formulas Under IRT 3PL Model With $f_{R} (θ) = f_{F} (θ) ~ N (μ, σ^{2})$

Power Formulas for Decision Rule Based on DIF Effect Size ${\hat{β}}_{cro}$