Through simulations, this study investigates the effects of anchor item methods on Type I error and power of detecting differential item functioning (DIF) using the likelihood ratio test within the framework of item response theory. Four anchor item methods were compared: the all-other, 1-item, 4-item, and 10-item methods. The results showed that it is the average signed area between the reference and focal groups rather than the percentage of DIF items in a test that determines the Type I error of the all-other method. The all-other method yields good control over Type I error and reasonable power only when the average signed area approaches zero. The all-other method is not recommended for practical DIF analysis because it is only adequate under very stringent conditions. The other three methods perform appropriately under all the simulated conditions. The more anchor items are used, the higher the power of DIF detection.
Get full access to this article
View all access options for this article.
References
1.
Ankenmann, R. D., Witt, E. A., & Dunbar, S. B. (1999). An investigation of the power of the likelihood ratio goodness-of-.t statistic in detecting differential item functioning. Journal of Educational Measurement, 36, 277-300.
2.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
3.
Bock, R. D., & Lieberman, M. (1970). Fitting a response model for ndichotomously scored items. Psychometrika, 35, 179-197.
4.
Candell, G. L., & Drasgow, F. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253-260.
5.
Cohen A. S., Kim, S., & Wollack, J. A. (1996). An investigation of the likelihood ratio test for detection of differential item functioning. Applied Psychological Measurement, 20, 15-26.
6.
Fidalgo, A. M., Mellenbergh, G. J., & Muñniz, J. (2000). Effects of amount of DIF, test length, and puri.cation type on robustness and power of Mantel-Haenszel procedures. Methods of Psychological Research Online, 5, 43-53.
7.
Hidalgo-Montesinos, M. D., & Lopez-Pina, J. A. (2002). Two-stage equating differential item functioning detection under the graded response model with the Raju area measures and the Lord statistic. Educational and Psychological Measurement, 62, 32-44.
8.
Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. Braun (Eds.), Test validity(pp. 129-145). Hillsdale, NJ: Lawrence Erlbaum.
9.
Jodoin, M. G.,& Gierl, M. J. (2001). Evaluating Type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14, 329-349.
10.
Kim, S.-H., & Cohen, A. S. (1995). A comparison of Lord’s chi-square, Raju’s area measures, and the likelihood ratio test on detection of differential item functioning. Applied Measurement in Education, 8, 291-312.
11.
Kim, S.-H., & Cohen, A. S. (1998). Detection of differential item functioning under the graded response model with the likelihood ratio test. Applied Psychological Measurement, 22, 345-355.
12.
Kim, S.-H., Cohen, A. S., & Kim, H.-O. (1994). An investigation of Lord’s procedure for detection of differential item functioning. Applied Psychological Measurement, 18, 217-228.
13.
Li, H.,& Stout, W. (1996). Anewprocedure for detection of crossing DIF. Psychometrika, 61, 647-677.
14.
Lord, F. M. (1968). An analysis of theVerbal Scholastic Aptitude Test using Birnbaum three-parameter logistic model. Educational and Psychological Measurement, 28, 989-1020.
15.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
16.
Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22, 719-748.
17.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
18.
McLaughlin, M. E., & Drasgow, F. (1987). Lord’s chi-square test of item bias with estimated known person parameters. Applied Psychological Measurement, 11, 161-173.
19.
Miller, M. D., & Oshima, T. C. (1992). Effect of sample size, number of biased items and magnitude of bias on a two-stage item bias estimation method. Applied Psychological Measurement, 16, 381-388.
20.
Narayanan, P., & Swaminathan, H. (1994). Performance of the Mantel-Haenszel and simultaneous item bias procedures for detecting differential item functioning. Applied Psychological Measurement, 18, 315-328.
21.
Narayanan, P., & Swaminathan, H. (1996). Identi.cation of items that shownonuniform DIF. Applied Psychological Measurement, 20, 257-274.
22.
Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference. Biometrika, 20A, 174-240, 263-294.
23.
Raju, N. S. (1988). The area between two item characteristics curves. Psychometrika, 53, 495-502.
24.
Rao, C. R. (1973). Linear statistical inference and its applications. New York: John Wiley.
25.
Rasch, G. (1960). Probabilistic models for some intelligent and attainment tests. Copenhagen: Institute of Educational Research.
26.
Reise, S. P., & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27, 133-134.
27.
Rogers, H. J.,& Swaminathan, H. (1993). Acomparison of logistic regression and Mantel-Haenszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17, 105-116.
28.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometric Monograph, no. 17.
29.
Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361-370.
Thissen, D., Steinberg, L., & Gerrard, M. (1986). Beyond group mean differences: The concept of item bias. Psychological Bulletin, 99, 118-128.
32.
Thissen, D., Steinberg, L., & Wainer, H. (1988). Use of item response theory in the study of group differences in trace lines. In H. Wainer& H. Braun (Eds.), Test validity(pp. 147-169). Hillsdale, NJ: Lawrence Erlbaum.
33.
van der Flier, H., Mellenbergh, G. J., Adér, H. J., & Wijn, M. (1984). An iterative item bias detection method. Journal of Educational Measurement, 21, 131-146.
34.
Wald, A. (1943). Tests of statistical hypotheses concerning several parameters when the number of observations in large. Transactions of the American Mathematical Society, 54, 426-482.
35.
Wang, W.-C. (2001, September). Effects of anchor item methods on the detection of differential item functioning within the family of Rasch models.Paper presented at the annual meeting of the Chinese Psychological Association, Chia-Yi, Taiwan. Manuscript submitted for publication.
36.
Webb, N. L. (1994). Wisconsin performance assessment development project: Analysis and technical report for scal year 1993-93. Madison: University of Wisconsin, Wisconsin Center for Educational Research.
