Alternative Matching Scores to Control Type I Error of the Mantel–Haenszel Procedure for DIF in Dichotomously Scored Items Conforming to 3PL IRT and Nonparametric 4PBCB Models
Restricted accessOtherFirst published online May, 2010
Alternative Matching Scores to Control Type I Error of the Mantel–Haenszel Procedure for DIF in Dichotomously Scored Items Conforming to 3PL IRT and Nonparametric 4PBCB Models
When the matching score is either less than perfectly reliable or not a sufficient statistic for determining latent proficiency in data conforming to item response theory (IRT) models, Type I error (TIE) inflation may occur for the Mantel—Haenszel (MH) procedure or any differential item functioning (DIF) procedure that matches on summed-item score, but primarily on short tests. Alternative matching scores were developed based on sufficient statistics, reliability, and explicit corrections for measurement error. Manipulated factors were tests (20, 24, 26 items), reference/focal sample sizes (1,000/1,000, 800/200), proficiency distributions (identical, means differed, variances differed, means and variances differed), and simulation technique (three-parameter logistic IRT model and four-parameter beta compound-binomial model with nonparametric nonmonotonic item-true score step functions). Outcomes were as follows: TIE of MH chi-square test at the .05 nominal level; and the bias, standard error, and root mean square error of the MH delta-DIF statistic under null-DIF conditions. Of eight categorized alternative matching scores, four scores controlled TIE as well as or better than traditional summed-item score in almost all items for all conditions: (a) estimated latent proficiency from a 3PL IRT model, (b) the sum of weighted item scores where the weight was the item— total score biserial correlation coefficient excluding the item from total score, (c) the sum of weighted item scores where the weight was the item loading on the single common factor from factor analysis of tetrachoric correlation coefficients, and (d) Kelley’s linear regressed true score estimate.
Birch, M.W. ( 1964). The detection of partial association I: The 2 x 2 case . Journal of the Royal Statistical Society, B26, 313-324.
2.
Birnbaum, A. ( 1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 395-479). Menlo Park, CA: Addison-Wesley.
3.
Brennan, R.L. ( 2004). Manual for BB-CLASS: A computer program that uses the beta-binomial model for classification consistency and accuracy. Version1 (Research Report No. 9). Iowa City, IA: Center for Advanced Studies in Measurement and Assessment. Retrieved from http://www.education.uiowa.edu/casma/computer_programs.htm
4.
Camilli, G., & Shepard, L.A. ( 1994). Methods for identifying biased test items (Vol. 4). Thousand Oaks, CA: Sage.
5.
Clauser, B., Mazor, K.M., & Hambleton, R.K. (1993). The effects of purification of the matching criterion on the identification of DIF using the Mantel-Haenszel procedure. Applied Measurement in Education, 6, 269-279.
6.
Clauser, B., Mazor, K.M., & Hambleton, R.K. (1994). The effects of score group width on the Mantel- Haenszel procedure. Journal of Educational Measurement, 31, 67-78.
7.
DeMars, C.E. ( 2009). Modifications of the Mantel-Haenszel and logistic regression DIF procedures to incorporate the SIBTEST regression correction. Journal of Educational and Behavioral Statistics, 34, 149-170.
8.
Donoghue, J.R., & Allen, N.L. ( 1993). Thin versus thick matching in the Mantel-Haenszel procedure for detecting DIF. Journal of Educational Statistics, 18, 131-154.
9.
Fidalgo, A.M., Ferreres, D., & Muniz, J. ( 2004). Utility of the Mantel-Haenszel procedure for detecting differential item functioning in small samples. Educational and Psychological Measurement, 64, 925-936.
10.
Finch, H. ( 2005). The MIMIC model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST and the IRT likelihood ratio. Applied Psychological Measurement, 29, 278-295.
11.
Fleiss, J.L. ( 1981). Statistical methods for rates and proportions (2nd ed.). New York, NY: John Wiley.
12.
Gulliksen, H.O. ( 1950). Theory of mental tests. New York, NY: John Wiley.
13.
Hanson, B.A. ( 1991). Method of moments estimates for the four-parameter beta compound binomial model and the calculation of classification consistency indexes (Research Report No. ACT-RR-91-5). Iowa City, IA: American College Testing. Retrieved from http://www.act.org/research/reports/91index.html
14.
Holland, P.W., & Thayer, D.T. ( 1985). An alternate definition of the ETS scale of item difficulty (Research Report No. RR-85-43). Princeton, NJ: Educational Testing Service.
15.
Holland, P.W., & Thayer, D.T. ( 1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Lawrence Erlbaum.
Kelley, T.L. ( 1923). Statistical methods. New York, NY : Macmillan.
18.
Lord, F.M. ( 1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
19.
Lord, F.M., & Novick, M.R. ( 1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
20.
Lu, S.-M. ( 1996). The relationship between item statistics and the Mantel-Haenszel and standardization DIF statistics when comparison groups differ in ability . Dissertation Abstracts International, A57, 2987.
21.
Mantel, N., & Haenszel, W.M. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22, 719-748.
22.
Meredith, W., & Millsap, R.E. ( 1992). On the misuse of manifest variables in the detection of measurement bias. Psychometrika, 57, 289-311.
23.
Millsap, R.E., & Meredith, W. ( 1992). Inferential conditions in the statistical detection of measurement bias. Applied Psychological Measurement, 16, 389-402.
24.
Monahan, P.O. ( 2002). The Mantel-Haenszel procedure for DIF: Alternative matching scores to control Type I error and improve distributional properties. Dissertation Abstracts International, A63, 1313. Retrieved from http://www.biostat.iupui.edu/Faculty/monahan.aspx
25.
Monahan, P.O. ( 2004, April). Examining the assumption of linearity of the logit in the logistic regression procedure for detecting DIF. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA. Retrieved from http://www.biostat.iupui.edu/Faculty/monahan.aspx
26.
Monahan, P.O., & Ankenmann, R.D. (2005). Effect of unequal variances in proficiency distributions on Type I error of the Mantel-Haenszel chi-square test for differential item functioning. Journal of Educational Measurement, 42, 101-131.
27.
Monahan, P.O., Lee, W.-C., & Ankenmann, R.D. (2007). Generating dichotomous item scores with the four-parameter beta compound binomial model. Journal of Educational Measurement, 44, 211-225.
28.
Monahan, P.O., McHorney, C.A., Stump, T.E., & Perkins, A.J. ( 2007). Odds ratio, delta, ETS classification, and standardization measures of DIF magnitude for binary logistic regression. Journal of Educational and Behavioral Statistics, 32, 92-109.
29.
Parshall, C.G., & Miller, T.R. ( 1995). Exact versus asymptotic Mantel-Haenszel DIF statistics: A comparison of performance under small-sample conditions. Journal of Educational Measurement, 32, 302-316.
30.
Rasch, G. ( 1966). An item analysis which takes individual differences into account. British Journal of Mathematical and Statistical Psychology , 19, 49-57.
31.
Roussos, L., & Stout, W. ( 1996). Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel-Haenszel Type I error performance . Journal of Educational Measurement, 33, 215-230.
32.
Shealy, R., & Stout, W. ( 1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58, 159-194.
33.
Shepard, L.A., Camilli, G., & Williams, D.M. (1984). Accounting for statistical artifacts in item bias research. Journal of Educational Statistics , 9, 93-128.
34.
Swaminathan, H., & Rogers, J. ( 1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement , 27, 361-370.
35.
Thissen, D. ( 1991). MULTILOG: Multiple, categorical item analysis and test scoring using item response theory (Version 6.0) [Computer program]. Chicago, IL: Scientific Software International.
36.
Uttaro, T., & Millsap, R.E. ( 1994). Factors influencing the Mantel-Haenszel procedure in the detection of differential item functioning. Applied Psychological Measurement, 18, 15-25.
37.
Wang, W.-C., & Su, Y.-H. ( 2004). Effects of average signed area between two item characteristic curves and test purification procedures on the DIF detection via the Mantel-Haenszel method. Applied Measurement in Education, 17, 113-144.
38.
Zwick, R. ( 1990). When do item response function and Mantel-Haenszel definitions of differential item functioning coincide? Journal of Educational Statistics, 15, 185-197.
39.
Zwick, R., Thayer, D.T., & Mazzeo, J. ( 1997). Descriptive and inferential procedures for assessing differential item functioning in polytomous items. Applied Measurement in Education, 10, 321-344.
