Influence of the Criterion Variable on the Identification of Differentially Functioning Test Items Using the Mantel-Haenszel Statistic

Abstract

This study investigated the effectiveness of the Mantel-Haenszel (MH) statistic in detecting dif ferentially functioning (DIF) test items when the internal criterion was varied. Using a dataset from a statewide administration of a life skills examina tion, a sample of 1,000 Anglo-American and 1,000 Native American examinee item response sets were analyzed. The MH procedure was first applied to all the items involved. The items were then cate gorized as belonging to one or more of four subtests based on the skills or knowledge needed to select the correct response. Each subtest was then analyzed as a separate test, using the MH pro cedure. Three control subtests were also established using random assignment of test items and were analyzed using the MH procedure. The results revealed that the choice of criterion, total test score versus subtest score, had a substantial influence on the classification of items as to whether or not they were differentially functioning in the American and Native American groups. Evidence for the convergence of judgmental and statistical procedures was found in the unusually high proportion of DIF items within one of the classifications and in the results of the reanalysis of this group of items.

Keywords

Index terms: differential item functioning item bias Mantel-Haenszel statistic,test bias.

Get full access to this article

View all access options for this article.

References

Ackerman, T.A. (1989). Unidimensional IRT calibration of compensatory and noncompensatory multidimensional items. Applied Psychological Measurement, 13, 113-127.

Bennett, R.E. , Rock, D.A. , & Novatkoski, I. (1989). Differential item functioning on the SAT-M Braille Edition. Journal of Educational Measurement , 26, 67-79.

Berk, R. A. (Ed.). (1982). Handbook of methods for detecting test bias. Baltimore MD: Johns Hopkins University Press.

Hambleton, R.K. , Bollwark, J. , & Rogers, H.J. (1990). Factors affecting the stability of the Mantel-Haenszel item bias statistic (Rep. No. 203). Amherst: University of Massachusetts, School of Education, Laboratory of Psychometric and Evaluative Research.

Hambleton, R.K. , & Rogers, H.J. (1989). Detecting potentially biased test items: Comparison of IRT area and Mantel-Haenszel methods. Applied Measurement in Education, 2, 313-334.

Hattie, J.A. (1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9, 139-164.

Holland, P.W. , & Thayer, D.T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test Validity (pp. 129-145). Hillsdale NJ: Erlbaum.

Mellenbergh, G.J. (1989). Item bias and item response theory. International Journal of Educational Research, 13, 127-143.

Raju, N.S. , Bode, R.K. , & Larsen, V.S. (1989). An empirical assessment of the Mantel-Haenszel statistic for studying differential item performance. Applied Measurement in Education, 2, 1-13.

10.

Scheuneman, J. , & Bleistein, C.A. (1989). A consumer's guide to statistics for identifying differential item functioning. Applied Measurement in Education, 2, 255-275.

11.

Zwick, R. , & Ercikan, K. (1989). Analysis of differential item functioning in the NAEP history assessment. Journal of Educational Measurement , 26(1), 55-66.