Influences on the Mantel-Haenszel Chi-Square in Detection of Differential Item Functioning under Rasch Conditions

Abstract

The Mantel-Haenszel chi-square (χ²MH) is widely used to detect differential item functioning (item bias) between ethnic and gender-based subgroups on educational and psychological tests. The empirical behavior of χ²MH has been incompletely understood; previous research is inconclusive. The present simulation study explored the effects of sample size, number of items, and trait distributions on the power of χ²MH to detect modeled differential item functioning. A significant effect was obtained for sample size with unacceptably low power for 250 subjects each in the focal and reference groups. The discussion supports the 1990 recommendations of Swaminathan and Rogers, opposes the 1993 view of Zieky that a sample size of 250 for each group is adequate.

References

Breslow

(1981) Odds ratio estimates when the data are sparse. Biometrika, 68, 73–84.

Donner

Hauck

W. W.

(1986) The large sample relative efficiency of the Mantel-Haenszel estimator in the fixed strata case. Biometrics, 42, 537–545.

Holland

P. W.

Thayer

D. T.

(1988) Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity. Hillsdale, NJ: Erlbaum. Pp. 129–145.

Meredith

Millsap

R. E.

(1992) On the misuse of manifest variables in the detection of measurement bias. Psychometrika, 57, 289–311.

Schulz

E. M.

Perlman

C. P.

Rice

W. K.

Wright

B. D.

(1989) Empirical comparison of Rasch and Mantel-Haenszel procedures. Presented at the annual meeting of the American Educational Research Association, Boston, MA.

Swaminathan

Rogers

H. J.

(1990) Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361–370.

Uttaro

Millsap

R. E.

(1994) Factors affecting the Mantel-Haenszel procedure in the detection of differential item functioning. Applied Psychological Measurement, 18, 15–25.

Zieky

(1993) Practical questions in the use of DIF statistics in item development. In Holland

P. W.

Wainer

(Eds.), Differential item functioning: theory and practice. Hillsdale, NJ: Erlbaum. Pp. 337–364.

Zwick

(1990) When do item response function and Mantel-Haenszel definitions of differential item functioning coincide? Journal of Educational Statistics, 15, 185–197.