The current study examined the Type I error rates and power of several item response theory (IRT) item fit indices used in conjunction with the graded response model (GRM). Specifically, S — χ2, χ2*, and adjusted χ 2 degrees of freedom ratios (χ2/dfs) were examined. Model misfit was introduced by manipulating item parameters and by using a different IRT model to generate item data. Results indicated lower than expected Type I error rates for S — χ2 and χ 2*. Adjusted χ2/dfs resulted in large Type I error rates when used with cross validation and very low Type I error rates when used without cross validation. χ2* and adjusted χ 2/dfs without cross validation were the most powerful overall.
Bjorner, J.B., Smith, K.J., Stone, C., & Sun, X. ( 2007). IRTFIT: A macro for item fit and local dependence tests under IRT models [Computer software]. Retrieved June, 2009 from http://outcomes.cancer.gov/areas/measurement/irt_model_fit.html
2.
Bock, R.D. ( 1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika , 37, 29-51.
3.
Brown, M.W., & Cudeck, R. ( 1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230-258.
4.
Chernyshenko, O.S., Stark, S., Chan, K.Y., Drasgow, F., & Williams, B. ( 2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523-562.
5.
Chernyshenko, O.S., Stark, S., Drasgow, F., & Roberts, B.W. ( 2007). Constructing personality scales under assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19, 88-106.
6.
Donovan, M.A., Drasgow, F., & Probst, T.A. ( 2000). Does comparing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight. Journal of Applied Psychology, 85, 305-313.
7.
Drasgow, F., Levine, M.V., Tsien, S., Williams, B., & Mead, A.D. ( 1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143-165.
8.
Embretson, S.E., & Reise, S.P. ( 2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
9.
Goldberg, L.R., Johnson, J.A., Eber, H.W., Hogan, R., Ashton, M.C., Cloninger, C.R., et al. (2006). The International Personality Item Pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84-96.
10.
Kang, T., & Chen, T.T. ( 2008). Performance of the generalized S - χ2 item fit index for polytomous IRT models. Journal of Educational Measurement , 45, 391-406.
11.
Koenig, J.A., & Roberts, J.S. ( 2007). Linking parameters estimated with the generalized graded unfolding model: A comparison of characteristic curve methods. Applied Psychological Measurement, 31, 504-524.
12.
LaHuis, D.M., & Copeland, D.A. (2009). Investigating faking using a multilevel logistic regression approach to measuring person fit. Organizational Research Methods, 12, 396-319.
13.
Meade, A.W., Lautenschlager, G.J., & Johnson, E.C. ( 2007). A Monte Carlo examination of the sensitivity of the differential functioning of items and tests framework for tests of measurement invariance with Likert data. Applied Psychological Measurement, 31, 430-455.
14.
McKinley, R.L., & Mills, C.N. ( 1985). A comparison of several goodness-of-fit statistics. Applied Psychological Measurement, 9, 49-57.
15.
Mislevy, R.J., & Bock, R.D. ( 1990). BILOG: Item analysis and test scoring with binary logistic models [computer program]. Chicago, IL: Scientific Software, Inc.
16.
Orlando, M. ( 1997). Item fit in the context of item response theory. Dissertation Abstracts International, 58/04-B, 2175.
17.
Orlando, M., & Thissen, D. ( 2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement , 24, 50-64.
18.
Orlando, M., & Thissen, D. ( 2003). Further investigation of the performance of S - χ2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289-298.
19.
Reise, S.P. ( 1990). A comparison of item- and person-fit methods of assessing model-data fit in IRT. Applied Psychological Measurement, 14, 127-137.
20.
Roberts, J.S., Donoghue, J.R., & Laughlin, J.E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24, 3-32.
21.
Robie, C., Zickar, M.J., & Schmit, M.J. ( 2001). Measurement equivalence between applicant and incumbent groups: An IRT analysis of personality scales. Human Performance , 14, 187-207.
22.
Samejima, F. ( 1969). Estimation of latent ability using a response pattern of graded scores. Psychometrica Monographs, 34, 139.
23.
Seong, T.J. ( 1990). Sensitivity of marginal maximum likelihood estimation of item and ability parameters to the characteristics of prior ability distributions . Applied Psychological Measurement, 14, 299-311.
24.
Stark, S. ( 2007). Modfit version 2.0 [computer software]. Retrieved March, 2007, from http://work.psych.uiuc.edu/irt/mdf_modfit.asp
25.
Stark, S., Chernyshenko, O.S., Drasgow, F., & Williams, B.A. (2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring? Journal of Applied Psychology , 91, 25-39.
26.
Stone, C.A. ( 1992). Recovery of marginal maximum likelihood estimates in the two-parameter logistic response model: An evaluation of MULTILOG. Applied Psychological Measurement, 16, 1-16.
27.
Stone, C.A. ( 2000). Monte Carlo based null distribution for an alternative goodness-of-fit test statistic in IRT models. Journal of Educational Measurement, 37, 58-75.
28.
Stone, C.A. ( 2003). Empirical power and Type I error rates for an IRT fit statistic that considers the precision of ability estimates. Educational and Psychological Measurement, 63, 566-583.
29.
Stone, C.A., & Yin, Y. ( 2006, April). Accounting for item and ability parameter estimation in a goodness-of-fit statistic for IRT models. Paper presented at the Annual Meeting of the National Council on Measurement in Education, San Francisco.
30.
Stone, C.A., & Zhang, B. ( 2003). Assessing goodness-of-fit of IRT models: A comparison of traditional and alternative procedures. Journal of Educational Measurement, 40, 331-352.
Thissen, D., Pommerich, M., Billeaud, K., & Williams, V. ( 1995). Item response theory for scores on tests including polytomous items with ordered responses. Applied Psychological Measurement , 19, 39-49.
33.
Yen, W.M. ( 1981). Using simulation results to choose a latent trait model . Applied Psychological Measurement, 5, 245-262.
