A Power Formula for the Mantel–Haenszel Test for Differential Item Functioning

DIF and MH Test

Consider a test consisting of I items, where the response to the items is binary and coded as Y i = 1 if the answer to the item i is correct, and Y i = 0 if incorrect. The test-takers are from two population groups, one is the focal group as coded by G = F , and the other is the reference group as coded by G = R . The problem is to test whether an item of interest (the studied item) is a DIF item. The remaining I − 1 items are usually considered as non-DIF items (anchor items). An item is a non-DIF item if the probability of getting the studied item correct is the same regardless of the test-takers’ group membership after the ability is matched; otherwise, the item is a DIF item. By matching on the total score V , the examinees are grouped into I + 1 strata by the scores 0 , 1 , … , I . Within each stratum, a 2 × 2 cross table between the group membership and the response to the studied item can be constructed (Table 1).

OPEN IN VIEWER

Table 1.

2×2 Table for the jth Matched Set of Members of Reference and Focal Groups.

		Response to the studied item
		Correct ( Y = 1 )	Incorrect ( Y = 0 )
Group	Reference ( G = R )	A j	B j	n Rj
	Focal ( G = F )	C j	D j	n Fj
		m 1 j	m 0 j	N j

In the table, A j , B j , C j , and D j are the observed counts of examinees in each cell in the j th stratum; N j = A j + B j + C j + D j is the number of examinees in the stratum; n Rj = A j + B j and n Fj = C j + D j are the numbers of examines in the reference and focal groups in the stratum, respectively; and m 1 j = A j + C j and m 0 j = B j + D j are the numbers of examinees who answer the studied item correctly and incorrectly, respectively. The null hypothesis of no DIF states that conditional on the matching variable, there is no association between the group membership and the response to the studied item.

The MH chi-square test statistic (Mantel & Haenszel, 1959) is given by,

MH − χ C 2 = ( | ∑ j A j − ∑ j E A j | − 0 . 5 ) 2 ∑ j V A j ,

where E A j = ( A j + B j ) ( A j + C j ) / N j , and

V A j = n Rj n Fj m 1 j m 0 j N j 2 ( N j − 1 ) ,

are the estimates of the mean and the variance, respectively, of the cell count A j under the null hypothesis and the hypergeometric sampling model (Agresti, 2002). Under the null hypothesis, the MH chi-square statistic has a chi-square distribution with degree of freedom 1. The term “−0.5” in the MH chi-square test statistic is a continuity correction, which is supposed to improve accuracy of Type I error (Holland & Thayer, 1988). It is possible to define the MH chi-square statistic without the continuity correction as follows:

MH − χ 2 = ( ∑ j A j − ∑ j E A j ) 2 ∑ j V A j .

Previous simulation studies suggest that the MH chi-square with the continuity correction tends to be more conservative than the MH chi-square without the continuity correction (Paek, 2010). For example, at the level of 0.05, it is often observed in simulated data with smaller test length that the actual Type I error rate for the MH chi-square test with continuity correction is about 0.03 to 0.04, whereas for the MH-chi-square test without continuity correction on the same data set, it is about 0.05.

The MH chi-square statistics in Equations 1 or 2 do not directly give the direction of the DIF effects if they are significant. Holland and Thayer (1988) use the following formula to estimate the common odds ratio:

α ^ MH = R S = ∑ j R j ∑ j S j = ∑ j A j D j N j ∑ j B j C j N j .

The MH log odds ratio estimate ψ ^ MH = log α ^ MH has an asymptotic normal distribution around the true value ψ MH = log α MH . A formula for the variance of the MH log odds ratio estimate that was derived by Phillips and Holland (1987) is given by

Var ^ PH ( ψ ^ MH ) = 1 2 R 2 ∑ j N j − 2 ( A j D j + α ^ MH B j C j ) [ A j + D j + α ^ MH ( B j + C j ) ] .

Therefore, a MH-Z statistic can be defined by dividing the log odds ratio estimate by its standard error:

MH - Z = ψ ^ MH Var ^ ( ψ ^ MH ) .

The MH-Z statistic is asymptotically equivalent to the MH- χ 2 statistic. Under the null hypothesis, the MH-Z statistic has an asymptotic standard normal distribution, so (MH-Z)² has an asymptotic distribution of chi-square with 1 degree of freedom. Under the alternative hypothesis, (MH-Z)² and MH- χ 2 have the same asymptotic distribution of non-central chi-square with 1 degree of freedom. The proof of this equivalence is shown in the online appendix. In this article, the asymptotic power formula is derived based on the MH-Z statistic. Because of the asymptotic equivalence, the same formula is also applicable for the MH- χ 2 statistic. Later in simulation studies, it is also shown that the rejection rates of the test based on MH-Z statistic is very similar to those of the MH- χ 2 test.

Asymptotic Formula for ^ α _MH and the Variance Estimate of Log ^ α _MH

Let P R ( θ ) and P F ( θ ) be the IRF of the studied item for the reference group and the focal group, respectively: P R ( θ ) = P ( Y = 1 | θ , G = R ) and P F ( θ ) = P ( Y = 1 | θ , G = F ) ; and let Q R ( θ ) = 1 − P R ( θ ) and Q F ( θ ) = 1 − P F ( θ ) .

Define the local odds ratio α ( θ ) be the odds ratio given θ ,

α ( θ ) = P R ( θ ) Q F ( θ ) P F ( θ ) Q R ( θ ) .

Roussos, Schnipke, and Pashley (1999) gave a general asymptotic formula for the MH odds ratio estimate α ^ MH . As test length tends to ∞ and the sample size tends to ∞ , matching on total score tends to matching on θ . Then, the MH odds ratio will tend to,

α ^ MH → α ∞ = ∫ P F ( θ ) Q R ( θ ) f R ( θ ) f F ( θ ) γ F f F ( θ ) + γ R f R ( θ ) α ( θ ) d θ ∫ P F ( θ ) Q R ( θ ) f R ( θ ) f F ( θ ) γ F f F ( θ ) + γ R f R ( θ ) d θ ,

where γ R and γ F are the proportion of reference group and focal group test-takers in their pooled population. Thus, γ R = P ( G = R ) and γ F = P ( G = F ) = 1 − γ R . Furthermore, f R ( θ ) and f F ( θ ) are the probability density functions (PDFs) for θ in the reference and focal group populations: f R ( θ ) = p ( θ | G = R ) and f F ( θ ) = p ( θ | G = F ) .

To derive a power formula for the MH-Z statistics, a general formula is also necessary for the variance of the log odds ratio estimates under the null hypothesis, where P R ( θ ) = P F ( θ ) . Starting from the Phillips and Holland (1987) formula (Equation 4), under the null, α ^ MH → α MH = 1 , α ^ j = A j D j / B j C j → α MH = 1 , therefore,

Var ^ PH ( ψ ^ MH | H 0 ) ≈ 1 2 R 2 ∑ j 2 S j = S R 2 ≈ 1 R → 1 n ∫ P R ( θ ) Q R ( θ ) f R ( θ ) f F ( θ ) ( 1 p ( θ ) ) γ R γ F d θ .

Deriving the Power Formula for MH Test

Under the null hypothesis, the MH-Z statistic follows the N(0,1) distribution. So for the one-sided test, H 0 : ψ = 0 versus H 1 : ψ > 0 , the null hypothesis is rejected if MH − Z > z α for Type I error α, where the critical value, z α = Φ − 1 ( 1 − α ) and Φ , is the distribution function for the standard normal distribution (e.g., for α = 0.05, z α = 1.645). The power is the probability of rejecting the null under the alternative hypothesis. Under H 1 , MH-Z follows asymptotic distribution N ( n ψ / τ ( ψ ) , 1 ) . So the power is,

power = Pr ( reject H 0 | H 1 ) = Φ ( n ψ τ ( ψ ) − z α ) .

For details on the theory on asymptotic power, see Lehmann (1999).

Therefore, the asymptotic power for the one-sided DIF test ( H 1 : ψ > 0 or H 1 : ψ < 0 ) is given by

power = Φ ( n | ψ | τ ( ψ 0 ) − z α ) = Φ ( n | ψ MH | [ ∫ − ∞ ∞ P R ( θ ) Q R ( θ ) f R ( θ ) f F ( θ ) γ R γ F γ R f R ( θ ) + γ F f F ( θ ) d θ ] 1 2 − z α ) .

And for the two-sided DIF test ( H 1 : ψ ≠ 0 ), the power is given by

power = Φ ( n ψ τ ( ψ 0 ) − z α / 2 ) + Φ ( − n ψ τ ( ψ 0 ) − z α / 2 ) = Φ ( n ψ MH [ ∫ − ∞ ∞ P R ( θ ) Q R ( θ ) f R ( θ ) f F ( θ ) γ R γ F γ R f R ( θ ) + γ F f F ( θ ) d θ ] 1 2 − z α / 2 ) + Φ ( − n ψ MH [ ∫ − ∞ ∞ P R ( θ ) Q R ( θ ) f R ( θ ) f F ( θ ) γ R γ F γ R f R ( θ ) + γ F f F ( θ ) d θ ] 1 2 − z α / 2 ) .

The integrals in the power formulas can be evaluated by numeric integration methods. The author has used the numeric integration procedure in R (R Core Team, 2012) to obtain the numeric results in this article.

Factors That Influence the Power of the MH Test

From Equations 8 and 9, it can be shown that the power of the MH test when the number of items is large is related to

A detailed discussion is presented on the components in the integral

∫ − ∞ ∞ P R ( θ ) Q R ( θ ) f R ( θ ) f F ( θ ) γ R γ F γ R f R ( θ ) + γ F f F ( θ ) d θ ,

and how they influence the power. The reciprocal of the integral is τ ( ψ 0 ) 2 , and τ ( ψ 0 ) / n is the standard error of the log odds ratio estimate ψ ^ = log α ^ MH under the null hypothesis. The integrand is the product of two parts: the first part P R ( θ ) Q R ( θ ) is only related to the IRF; and the second part, which is denoted by H ( θ ) = f R ( θ ) f F ( θ ) γ R γ F / [ γ R f R ( θ ) + γ F f F ( θ ) ] , is only related to the distributions of the latent traits.

IRF P(θ)

The most often used IRT models are the one-parameter logistic (1PL), 2PL, and 3PL models. Consider the IRF for the 3PL model:

P 3 PL ( θ ) = c + ( 1 − c ) 1 1 + exp ( − 1 . 7 a ( θ − b ) ) ,

where b is the difficulty parameter, a is the discrimination parameter, and c is the guessing parameter. The 2PL model is a special case of the 3PL model with c = 0 , and the Rasch model is a special case of 2PL model with a = 1 . Figure 1 shows how the product P ( θ ) Q ( θ ) changes at different values of a, b, and c. In Panels (a) and (c), the item follows 2PL, and in Panels (b) and (d), the item follows the 3PL model.

OPEN IN VIEWER

Figure 1.

P ( θ ) Q ( θ ) for IRT model at different values of a, b, and c.

If the studied item follows the 2PL model, the product P ( θ ) Q ( θ ) is a bell-shaped function symmetric about θ 1 / 2 , where θ 1 / 2 is the value of θ where P(θ) equals 1/2 (Figure 1, Panels (a) and (c)). It is well known that the 2PL IRF is also the cumulative distribution function (CDF) of a logistic distribution with location parameter b , and scale parameter 1 / 1 . 7 a . The derivative of the 2PL IRF is P ′ 2 PL ( θ ) = 1 . 7 a P 2 PL ( θ ) Q 2 PL ( θ ) . Therefore, the product P ( θ ) Q ( θ ) for the 2PL model can be seen as a PDF of the logistic distribution multiplied by (1/1.7 × a ), and thus has a bell-shaped curve.

However, by including a non-zero guessing parameter c in the 3PL model, the product P ( θ ) Q ( θ ) is no longer symmetric (Figure 1, Panels (b) and (d)). As P 3 PL ( θ ) = c + ( 1 − c ) P 2 PL ( θ ) , it can be derived that P 3 PL ( θ ) Q 3 PL ( θ ) = c ( 1 − c ) Q 2 PL ( θ ) + ( 1 − c ) 2 P 2 PL ( θ ) Q 2 PL ( θ ) . Therefore, on Panels (b) and (d) of Figure 1, as θ → − ∞ , the function tends to c ( 1 − c ) (in the figure, 0.2 × 0.8 = 0.16), while as θ → ∞ , the function tends to 0.

Panels (a) and (b) of Figure 1 show that the difficulty parameter b only affects the location of the curve and does not affect the shape of the curve. Panels (c) and (d) of Figure 1 show that by increasing the discrimination parameter a , the shape of the curve gets narrower.

Latent Trait Distributions

Now consider the term

H ( θ ) = f R ( θ ) f F ( θ ) γ R γ F γ R f R ( θ ) + γ F f F ( θ ) ,

which has three components: the proportion of the population composed of the focal group γ F (reference group, γ R = 1 − γ F ), the reference group latent trait distribution f R ( θ ) , and the focal group latent trait distribution f F ( θ ) . Assuming that for the reference group f R ( θ ) = N ( 0 , 1 ) , consider three situations for the focal group f F ( θ ) : (a) same distribution as reference group, p ( θ ) = γ R N ( 0 , 1 ) + γ F N ( 0 , 1 ) = N ( 0 , 1 ) ; (b) same variance, different mean, p ( θ ) = γ R N ( 0 , 1 ) + γ F N ( d θ , 1 ) ; and (c) same mean, different variance, p ( θ ) = γ R N ( 0 , 1 ) + γ F N ( 0 , s 2 ) .

Figure 2 shows the shape of the function H ( θ ) under the different situations. When the reference and focal groups have the same population distribution for the latent trait, f R ( θ ) = f F ( θ ) = p ( θ ) , then H ( θ ) = γ R γ F p ( θ ) . Figure 2, Panel (a) shows the curve H ( θ ) when γ R = γ F = 0 . 5 , and p ( θ ) is the PDF of N(0,1). Figure 2, Panel (b) shows the curves when the focal group proportion γ F takes a range of reasonable values, namely, 0.5, 0.3, 0.2, and 0.1. As the proportions of focal and reference groups get more imbalanced, H ( θ ) gets smaller, leading to a larger standard error and smaller power. Figure 2, Panel (c) shows the H ( θ ) curve when the reference and the focal groups have different means but the same variance. Specifically, the reference has a N(0, 1) distribution, and the focal group has a N ( d θ , 1 ) distribution, where the curves for d θ = 0 , 1 , 2 , 3 are plotted. As d θ increases, the H ( θ ) curve has a smaller area under curve, which means the integral tends to get a smaller value, leading to larger standard error and smaller power. Figure 2, Panel (d) shows the curves where the reference and the focal have the same mean but different variances. In this situation, the reference group has a N(0,1) distribution, whereas the focal group has a N ( 0 , s 2 ) distribution, where curves for s = 0 . 5 , 1 , 2 , 3 are plotted. As s increases, the curve gets flatter. The area under the curve obtains its highest value when the focal group has the same SD as the reference ( s = 1 ).

OPEN IN VIEWER

Figure 2.

The shape of H ( θ ) under different situations.

Combining Item Characteristic Curve and Latent Trait Distribution

For the integral ∫ P R ( θ ) Q R ( θ ) H ( θ ) d θ , combining the effects of the item characteristic curve and the test-taker latent trait distribution, one would expect higher power if P R ( θ ) Q R ( θ ) is well matched with the latent trait distribution, as characterized by H ( θ ) , in terms of the location and the shape of the curves. That means it will be easier to detect DIF for an item with difficulty similar to the mean ability of the population than for those items that are too easy or too difficult.

Table 2 summarizes the factors that influence the power of the MH test as discussed so far. These factors were also previously studied and reported by various authors through simulation studies. Here, they were identified from the theoretical power formula. These factors may influence the power in complicated ways. Therefore, the power formulas are important tools to quantify how the power is influenced by these factors. It should be pointed out that the power formulas are very general and their applicability should not be considered as only restricted to the situations already discussed. There is no restriction that the IRF has to be 2PL or 3PL IRT models; actually, it can be of any form, for example, a normal ogive model, even a nonparametric IRT model. Also, the distribution of the latent trait may take forms other than the normal distribution.

OPEN IN VIEWER

Table 2.

A Summary Table of Factors Influencing the Power of MH Test.

Factor	Change	| α MH |	SE of log α MH	Power
DIF effect: b F − b R	b F − b R increases by f Δ b times	Increases by f Δ b times	Not much change	Increases
Difficulty b R	| b R − M ( θ ) | increases	No change for Rasch and 2PL	Increases	Decreases
Discrimination a	a R increases by f a times	Increases by f a times	Decreases	Increases
Guessing c	c increases	Decreases	Decreases	Decreases
Proportion of focal group in the sample γ F	| γ F − 0 . 5 | increases from 0 to 0.5	No change	Increases	Decreases
Difference in population mean d θ = μ F − μ R	| d θ | increases	No change for Rasch and 2PL	Increases	Decreases

Note. MH = Mantel–Haenszel; DIF = differential item function; 2PL = two-parameter logistic.

Sample Size Formula

Based on the power formula, the formula for sample size can be readily solved. Generally, for a one-sided test, H 0 : ψ = ψ 0 versus H 1 : ψ > ψ 0 , the power formula has the general form β = Φ ( μ − z α ) , where μ = n ψ / τ ( ψ 0 ) . Solving for μ , μ = z α + z 1 − β . Note that here α is the Type I error, and β is the power, so 1 − β is the Type II error. And z α and z 1 − β correspond to the Z scores in the standard normal distribution with right tail areas α and 1 − β , respectively; namely, z α = Φ − 1 ( 1 − α ) and z 1 − β = Φ − 1 ( β ) . For example, for a test with significance level α = . 05 , and desired power β = . 80 , the corresponding z 0 . 05 = 1 . 64 and z 0 . 20 = 0 . 84 , and ( z 0 . 05 + z 0 . 20 ) 2 = 6 . 15 . Solving for the sample size n ,

n = [ τ ( ψ 0 ) ψ ( z α + z 1 − β ) ] 2 .

By substituting in ψ = ψ MH = log α MH , and the formula for τ ( ψ 0 ) , the formula for the sample size for the one-sided MH test can be obtained as

n = ( z α + z 1 − β ) 2 ψ MH 2 [ ∫ P R ( θ ) Q R ( θ ) f R ( θ ) f F ( θ ) γ R γ F γ R f R ( θ ) + γ F f F ( θ ) d θ ] .

For the two-sided MH test, H 0 : ψ = ψ 0 versus H 1 : ψ ≠ ψ 0 , there is no closed form solution for the sample size. By noting that the power is a monotonic increasing function of the sample size, numeric methods, such as a binary search algorithm, can be used to find the minimal sample size to achieve a desired level of power, and are straightforward to implement.

As an example, for the two-sided MH test with a Type I error rate of 0.05, Table 3 shows the results of minimum sample size needed to achieve 0.80 power when the studied item follows the Rasch model or a model modified from the Rasch model by changing one of the factors. In the default setup (the bolded row in each section of the table), the studied item follows the Rasch model: the reference difficulty b R is 0, the latent trait distributions for both the focal and reference groups are N(0,1), and the group sizes are balanced ( N R = N F = N / 2 ) . Consider two values for Δ b = b F − b R : a medium effect value of Δ b = 0.426, and a large effect value of Δ b = 0.638. Under the default setup, if the focal group difficulty is 0.426 (medium effect), the required sample size is 838 ( N R = N F = 419). If the focal group difficulty is 0.638 (large effect), then the required sample size is 374 ( N R = N F = 187). The table shows sample sizes in the cases when one of the following factors changes while keeping the rest fixed at default values: the reference difficulty b R , discrimination a , guessing c , mean ability μ , mean difference in ability d θ = μ R − μ F , and SD of the ability σ . In each case, the asymptotic MH odds ratio α ∞ and its logarithm log α ∞ , the standard error of log α ∞ multiplied by n (which is τ ( ψ 0 ) in the power formula) are also reported.

OPEN IN VIEWER

Table 3.

Sample Size Needed to Reach Power 0.80 for the Two-Sided MH Test on the Rasch Model or a Model Modified From the Rasch Model.

	Medium effect					Large effect
	Δ b = b F − b R = 0 . 426					Δ b = b F − b R = 0 . 638
Difficulty for the reference	α ∞	log α ∞	S E log α × n	n	N R = N F = n / 2	α ∞	log α ∞	S E log α × n	n	N R = N F = n / 2
b R = 0	1.53	0.426	4.40	838	419	1.89	0.638	4.40	374	187
1	1.53	0.426	4.74	974	487	1.89	0.638	4.74	434	217
2	1.53	0.426	5.88	1,496	748	1.89	0.638	5.88	668	334
3	1.53	0.426	8.18	2,892	1,446	1.89	0.638	8.18	1,290	645
Discrimination
a = 0.5	1.24	0.213	4.12	2,932	1,466	1.38	0.319	4.12	1,308	654
1	1.53	0.426	4.40	838	419	1.89	0.638	4.40	374	187
2	2.34	0.852	5.14	286	143	3.58	1.276	5.14	128	64
3	3.59	1.278	5.90	168	84	6.78	1.914	5.90	76	38
Guessing
c = 0	1.53	0.426	4.40	838	419	1.89	0.638	4.40	374	187
0.15	1.41	0.341	4.33	1,268	634	1.65	0.502	4.33	586	293
0.2	1.38	0.320	4.34	1,446	723	1.60	0.469	4.34	672	336
0.25	1.35	0.301	4.36	1,650	825	1.55	0.440	4.36	772	386
Mean ability ( μ = μ R = μ F )
0	1.53	0.426	4.40	838	419	1.89	0.638	4.40	374	187
1	1.53	0.426	4.74	974	487	1.89	0.638	4.74	434	217
2	1.53	0.426	5.88	1,496	748	1.89	0.638	5.88	668	334
3	1.53	0.426	8.18	2,892	1,446	1.89	0.638	8.18	1,290	645
Mean ability difference ( d θ = μ R − μ F )
0	1.53	0.426	4.40	838	419	1.89	0.638	4.40	374	187
1	1.53	0.426	4.96	1,066	533	1.89	0.638	4.96	476	238
2	1.53	0.426	6.93	2,076	1,038	1.89	0.638	6.93	926	463
3	1.53	0.426	11.62	5,842	2,921	1.89	0.638	11.62	2,604	1,302
SD of ability ( σ = σ R = σ F )
0.5	1.53	0.426	4.12	734	367	1.89	0.638	4.12	328	164
1	1.53	0.426	4.40	838	419	1.89	0.638	4.40	374	187
2	1.53	0.426	5.14	1,144	572	1.89	0.638	5.14	510	255
3	1.53	0.426	5.90	1,508	754	1.89	0.638	5.90	672	336

Note. The default values of parameters are b R = 0 , a = 1 , μ R = 0 , d θ = 0 , σ = 1 . The results for the default setup are bolded. In each section, one parameter’s value is changed, while the rest stay at the default values.

Simulation Studies

In this section, the purpose of the simulation studies is to confirm that the power formula is correct, in the sense that the theoretical power is close to the power obtained by Monte Carlo simulation. Here, the theoretical values are compared with the results from two previously published simulation studies. The first was from Paek and Guo (2011), and the second was Uttaro and Millsap (1994). For the Paek and Guo study, their simulation results were used directly, whereas for the Uttaro and Millsap study, their simulation setup was used and the simulation study was re-conducted.

Simulation Study 1: Paek and Guo (2011)

Paek and Guo (2011) examined the power of the MH procedure when the focal and reference groups have notably unbalanced sample size. They focused on the situation where the focal group has a fixed small sample size, which may not satisfy the minimum DIF sample size requirement. They showed through simulation studies that the MH procedure still may have enough power to detect DIF in such situations if the sample size of the reference group is large enough. Here, the results could be reexamined by comparing the DIF detection rates reported in their simulation studies with the theoretical power as calculated by Equation 9.

Paek and Guo (2011) conducted two simulation studies, where in Study 1, they studied the impact of the sample size ratio at a fixed total, and in Study 2, the focal group is fixed at a small sample size whereas the reference group sample size increases. The DIF data were generated from the 3PL model with a scaling factor D = 1.7. In each test form, there were 40 anchor items and 1 studied DIF item. For the anchor items, the item parameters were randomly selected from the following distributions: a ~ N ( 1 , 0 . 3 ) , b ~ N ( 0 , 1 ) , and c ~ Uniform ( 0 , 0 . 35 ) . For the studied DIF item: a F = a R = 1 . 086 , c F = c R = 0 . 203 , b F − b R = 0 . 35 , and b R may take values −0.338 or 0.338. For person ability, if the group ability differences d θ = 0 , then for both reference and focal group θ ~ N ( 0 , 1 ) ; if d ( θ ) = 0 . 5 , then θ R ~ N ( 0 . 25 , 1 ) and θ F ~ N ( − 0 . 25 , 1 ) . By varying b R , d θ , N R , and N F , in Study 1, there were 44 conditions, and in Study 2, there were 100 conditions. In each condition, 4,000 replicated data sets were simulated. And for each replicated data set, three MH DIF tests: MH- χ c 2 , MH- χ 2 , and MH-Z were applied and the decisions of statistical significance were recorded. The power of the tests in each condition was estimated by the rejection rate (number of significant iterations/total number of iterations). The Monte Carlo error for the power estimates is small (<0.008) due to the large number of iterations. Table 4 (online appendix) reproduces the parameters and the rejection rates as reported in Paek and Guo (2011; Tables 1-2 for Study 1, and Tables 3-6 for Study 2). Based on the parameters used to simulate the data sets in each condition, the theoretical power is calculated from Equation 9, and the values are reported in the last column of Table 4.

The observed rejection rates from the simulated data agree very well with the power calculated from the asymptotic power formula in Equation 9. Panel (a) of Figure 3 shows the scatter plot of the rejection rate from the MH-Z statistic versus the rejection rate from the MH- χ 2 (without continuity correction) statistic, where all the points are very close to the 45° line, reflecting the fact that MH-Z and MH- χ 2 have the same power because they are asymptotically equivalent. Panels (b) and (c) of Figure 3 compare the observed rejection rates from the MH-Z statistic and the MH- χ 2 statistic, respectively, to the theoretical power values calculated by Equation 9. It is clear that the observed rejection rates from the simulation data are close to the theoretical power as all the points are close to the 45° line. Out of the 144 simulation conditions, the differences between the theoretical power and the simulation rejection rate of the MH- χ 2 test have a mean of −0.004 and a SD of 0.022; whereas the differences for the MH-Z test have a mean of 0.012 and a SD of 0.021. Panel (d) of Figure 3 compares the rejection rates from MH- χ 2 statistics with continuity correction to the theoretical power. With the continuity correction, the MH test is conservative in rejecting the null hypothesis and its actual Type I error is lower than the nominal value 0.05 (Paek, 2010). The scatter plot shows that all the observed rejection rates for the continuity-corrected MH- χ 2 statistic are slightly lower than the theoretical power values. The differences between the theoretical power and the rejection rate of the MH- χ c 2 test have a mean of 0.043 and a SD of 0.023.

OPEN IN VIEWER

Figure 3.

Comparisons of the rejection rate from the Peak and Guo (2011) simulation studies to the theoretical power.

Simulation Study 2: Uttaro and Millsap (1994)

Uttaro and Millsap (1994) conducted a comprehensive simulation study of a variety of factors and their interactions on the detection of DIF by the MH procedure. They designed 20-item and 40-item tests with 36 conditions for the studied item by varying the following four factors: item parameters a, b, c for the focal group in the IRT 3PL model, and the group difference in mean θ level: d θ . Specifically, they used a 3 × 3 × 2 × 2 factorial design with (a) three levels of a: 0.5, 1.0, 1.5; (b) three levels of b: 0.0, 0.3, 0.5; (c) two levels of c: 0.0, 0.2; and (d) two levels of d θ : 0.0, −1.0. The item parameters for the studied item in the reference group were a = 1.0, b = 0.0, and c = 0.2. The numbers of examinees in the focal and the reference groups were equal: N R = N F = 500. The latent trait values of the examinees were drawn from θ R ~ N ( 0 , 1 ) and θ F ~ N ( d θ , 1 ) .

In the original article, the number of replicates for each condition was 200, and only the MH-χ² test with continuity correction was considered. Instead of directly using the results reported in Uttaro and Millsap (1994), this study re-conducted the simulation studies under the same design, and increased the number of replicates to 2000, and also obtained the results for the MH-χ² test without continuity correction and the MH-Z test.

Figure 4 show the scatter plots of rejection rates of the MH tests versus the theoretical power. In the scatter plots, the points are close to the 45° line, indicating the theoretical power values are close to the rejection rates. The differences between the theoretical power value and the rejection rate for 40-item test have a root mean square difference (RMSD) of 0.032 for MH-χ² and MH-Z and an RMSD of 0.041 for MH- χ c 2 . For the case of 20 items, the corresponding RMSD values are 0.044 (MH-Z), 0.045 (MH- χ c 2 ), and 0.052 (MH-χ²). Based on RMSD, the theoretical power is closer to the 40-item rejection rate than to the 20-item rejection rate. This is expected, because the formula is asymptotic and thus assumes a very long test with a large number of items. However, the increase in RMSD from 40-item (moderate) to 20-item (short) test is quite small from a practical perspective, which indicates that the power formula can still be used for short tests.

OPEN IN VIEWER

Figure 4.

Comparison of rejection rates and the theoretical power in re-conducting the Uttaro and Millsap (1994) simulation studies.

Conclusion

In this article, a general formula for calculating the power of the MH test is derived. The formula is asymptotic in the sense that it assumes the number of examinees is large, and the number of the items is large so that by matching the total scores it is very close to the ideal situation that the underlying latent traits are matched. A comparison of the theoretical values of the power and the observed rejection rate in the simulation studies have shown that the formula provides a close approximation to the true power and thus can be used in practical applications. In particular, for example, it can be used in the design of DIF analyses in regard to determining the level power for detecting DIF effect of certain size under current sample size or in determining the sample size needed in each group for detecting a given DIF effect.

The power formula also helps understanding the factors that influence the power of the MH test, as seen by the discussion in this article. There have been many previous simulation studies on the factors that influence the power of the MH test, and the power formula explains precisely why these factors influence the power.

Practitioners should bear in mind that the use of the power formula involves certain limitations. The formula is derived under the assumptions that matching on the total score is equivalent to matching the latent trait. This assumption may not hold if the test is short and the study item is not included as a part of the matching score, and the distributions of the latent trait in the reference and focal group differ, one could see inflated Type I error if the sample size is large (DeMars, 2010; Roussos & Stout, 1996). In such situations, the power is not meaningful because the Type I error is not controlled and, thus, the test itself is not valid.

An R statistical package for conducting the power and sample size calculations for the MH DIF test, using the methods presented in the article, and the package is available upon request.