The Effect of Sample Size on the Functioning of the Mantel-Haenszel Statistic

Abstract

The Mantel-Haenszel (MH) procedure has become one of the most popular procedures for detecting differential item functioning. Valid results with relatively small numbers of examinees are one of the advantages typically attributed to this procedure. In this study, examinee item responses were simulated to contain differentially functioning items, and then were analyzed at five sample sizes to compare detection rates. Results showed the MH procedure missed 25 to 30% of the differentially functioning items when groups of 2000 were used. When 500 or fewer examinees were retained in each group, more than 50% of the differentially functioning items were missed. The items most likely to be undetected were those which were most difficult, those with a small difference in item difficulty between the two groups, and poorly discriminating items.

Get full access to this article

View all access options for this article.

References

Hambleton, R. K. and Rovinelli, R. (1973). A FORTRAN IV program for generating examinee response data from logistic test models. Behavioral Science, 18, 73-74.

Hills, J. (1990). Screening for potentially biased items in testing programs. Educational Measurement: Issues and Practice, 8, 5-11.

Holland, P. and Thayer, D. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer and H. I. Braun (Eds.), Test validity. Hillsdale, NJ: Lawrence Erlbaum Associates.

Kingston, N. , Leary, L. , and Wightman, L. (1988). An exploratory study of the applicability of item response theory methods to the Graduate Management Admission Test (GMAC Occasional Papers). Princeton, NJ: Graduate Management Admission Council.

Rogers, H. J. (1989). Item bias investigations with logistic regression. Unpublished doctoral dissertation, University of Massachusetts.

Wise, L. L. (1987, April). Differential item difficulty indicators in small samples. Paper presented at the meeting of the American Educational Research Association, Washington.