The purpose of this simulation study is to investigate the effects of anchor subtest composition on the accuracy of item response theory (IRT) likelihood ratio (LR) differential item functioning (DIF) detection (Thissen, Steinberg, & Wainer, 1988). Here, the IRT LR test was implemented with a free baseline approach wherein a baseline model was formed by freeing all items except a referent or anchor subset and examining the changes in fit with respect to a series of models wherein 1 item at a time was constrained in addition to the referent(s). The results clearly indicated that the composition of the anchor subtest is important for accurate DIF detection. It was found that using a single highly discriminating rather than a low discriminating referent greatly enhanced the power of the procedure. Moreover, in conditions involving small DIF or smaller sample sizes or both, power appeared to improve when a group of highly discriminating referents was used. These findings have implications for applied research involving short scales and small sample sizes.
American Psychological Association. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-472). Reading, MA : Addison-Wesley.
3.
Bolt, D.M. (2002). A Monte Carlo comparison of parametric and nonparametric polytomous DIF detection methods. Applied Measurement in Education , 15, 113-141.
4.
Camilli, G., & Shepard, L.A. (1994). Methods for identifying biased test items. Thousand Oaks, CA: Sage.
5.
Candell, G.L., & Drasgow F. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253-260.
6.
Chernyshenko, O.S., Stark, S., Chan, K.-Y., Drasgow, F., & Williams, B. (2001). Fitting item response models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523-562.
7.
Cheung, G.W., & Rensvold, R.B. (1999). Testing factorial invariance across groups: A reconceptualization and proposed new method. Journal of Management, 25, 1-27.
Drasgow, F., & Kang, T. (1984). Statistical power of differential validity and differential prediction analyses for detecting measurement nonequivalence . Journal of Applied Psychology, 69, 498-508.
10.
Finch, H. (2005). The MIMIC model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio test . Applied Psychological Measurement, 29, 278-295.
11.
Harwell, M., Stone, C.A., Hsu, T.-C., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20, 101-125.
Kim, S.-H., & Cohen, A.S. (1995). A comparison of Lord's chi-square, Raju's area measures, and the likelihood ratio test on detection of differential item functioning. Applied Psychological Measurement, 8, 291-312.
14.
Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
15.
Ones, D.S., & Anderson, N. (2002). Gender and ethnic group differences on personality scales in selection: SomeBritish data. Journal of Occupational and Organizational Psychology, 75, 255-276.
16.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement , 34, 100-114.
17.
Singelis, T., & Triandis, H. (1995). Horizontal and vertical dimensions of individualism and collectivism: A theoretical and measurement refinement. Cross-Cultural Research, 29, 240-275.
18.
Smith, P.C., Kendall, L.M., & Hulin, C.L. (1969). The measurement of satisfaction in work and retirement . Skokie IL: Rand McNally.
19.
Sörbom, D. (1974). A general method for studying differences in factor means and factor structure between groups. British Journal of Mathematical and Statistical Psychology, 27, 229-239.
20.
Stark, S. (1999). SGRGEN: A computer program for polytomous data generation [Computer program]. Urbana-Champaign: University of Illinois at Urbana-Champaign.
21.
Stark, S. (2000). 3PLGEN: A computer program for dichotomous data generation [Computer program]. Urbana-Champaign: University of Illinois at Urbana-Champaign.
22.
Stark, S., Chernyshenko, O.S., & Drasgow F. (2004). Examining the effects of differential item functioning and differential test functioning on selection decisions: When are statistically significant effects practically important? Journal of Applied Psychology, 89, 497-508.
23.
Stark, S., Chernyshenko, O.S., & Drasgow, F. (2006). Detecting differential item functioning with CFA and IRT: Toward a unified strategy. Journal of Applied Psychology , 19, 1292-1306.
Thissen, D., Steinberg, L., & Wainer, H. (1988). Use of item response theory in the study of group differences in trace lines. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 147-169). Hillsdale, NJ: Erlbaum.
26.
Timbrook, R.E., & Graham, J.R. (1994). Ethnic differences on the MMPI-2? Psychological Assessment, 6, 212-217.
27.
Wang, W.-C., & Yeh, Y.-L. (2003). Effects of anchor item methods on differential item functioning detection with the likelihood ratio test. Applied Psychological Measurement, 27, 479-498.
28.
Wang, W.-C. (2004). Effects of anchor item methods on differential item functioning detection within the family of Rasch models. Journal of Experimental Education, 72, 221-261.
