The random-effects facet model that deals with local item dependence in many-facet contexts is presented. It can be viewed as a special case of the multidimensional random coefficients multinomial logit model (MRCMLM) so that the estimation procedures for the MRCMLM can be directly applied. Simulations were conducted to examine parameter recovery and effects of model misspecification. Results indicated that the item and person parameters as well as the random effects could be recovered very accurately when the data were analyzed using the generating random-effects facet model. When local item dependence was ignored and the standard facet model was used, the parameter estimates were shrunken and the test reliability was overestimated. Two empirical examples about journal quality ratings and rater effects are given to illustrate implications and applications of the proposed random-effects facet model.
Get full access to this article
View all access options for this article.
References
1.
Adams, R. J., & Wilson, M. R. (1996). Formulating the Rasch model as a mixed coefficients multinomial logit. In G. Englhard & M. R. Wilson (Eds.), Objective measurement: Theory into practice (Vol. 3, pp. 143-166). Norwood, NJ: Ablex.
2.
Adams, R. J., Wilson, M. R., & Wang, W.-C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.
3.
Adams, R. J., Wilson, M. R., & Wu, M. (1997). Multilevel item response models: An approach to errors in variable regression. Journal of Educational and Behavioral Statistics, 22, 47-76.
4.
Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561-573.
5.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-479). Reading, MA: Addison-Wesley.
6.
Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443-459.
7.
Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.
8.
Bond, T., & Fox, C. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.
9.
Bradlow, E. T., Wainer, H., & Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153-168.
10.
Chen, W., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265-289.
11.
Congdon, P., & McQueen, J. (1997, March). The stability of rater severity estimates in large performance assessment programs. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
12.
De Boeck, P., & Wilson, M. R. (Eds.). (2004). Explanatory item response models: A generalized linear and nonlinear approach. New York: Springer-Verlag.
13.
Douglas, J., Kim, H. R., Habing, B., & Gao, F. (1998). Investigating local dependence with conditional covariance functions. Journal of Educational and Behavioral Statistics, 23, 129-151.
14.
Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7, 1-26.
15.
Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
16.
Ferrara, S., Huynh, H., & Baghi, H. (1997). Contextual characteristics of locally dependent open-ended item clusters in a large-scale performance assessment. Applied Measurement in Education, 10, 123-144.
17.
Fischer, G. H. (1973). The linear logistic test model as instrument in educational research. Acta Psychologica, 37, 359-374.
18.
Fox, J.-P., & Glas, C. A. W. (2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika, 66, 269-286.
19.
Glas, C. A. W., Wainer, H., & Bradlow, E. T. (2000). MML and EAP estimation in testletbased adaptive testing. In W. van der Linden &C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 271-287). London: Kluwer.
20.
Hoijtink, H., Rooks, G., & Wilmink, F. W. (1999). Confirmatory factor analysis of items with a dichotomous response format using the multidimensional Rasch model. Psychological Methods, 4, 300-314.
21.
Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Methods & Research, 26, 329-367.
22.
Hoskens, M., & De Boeck, P. (1997). A parametric model for local dependence among test items. Psychological Methods, 2, 261-277.
23.
Hoskens, M., & De Boeck, P. (2001). Multidimensional componential item response theory models for polytomous items. Applied Psychological Measurement, 25, 19-37.
24.
Lee, P. M. (1989). Bayesian statistics: An introduction. New York: Oxford University Press.
25.
Linacre, J. M. (1989). Many-facet Rasch measurement. Chicago: MESA Press.
26.
Linacre, J. M., & Wright, B. D. (2002). Understand Rasch measurement: Construction of measures from many-facet data. Journal of Applied Measurement, 3, 486-512.
27.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
28.
McCullagh, P., & Nelder, J. A. (1989). Generalized linear models (2nd ed.). London: Chapman & Hall.
29.
McCulloch, C. E., & Searle, S. R. (2001). Generalized, linear, and mixed models. New York: John Wiley.
30.
Mislevy, R. J., Beaton, A. E., Kaplan, B., &Sheehan, K. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29, 133-161.
31.
Mislevy, R. J., & Bock, R. D. (1989). A hierarchical item-response model for educational testing. In R. D. Bock (Ed.), Multilevel analysis of educational data (pp. 57-74). San Diego: Academic Press.
32.
Nelder, J. A., & Wedderburn, R. W. M. (1972). Generalized linear models. Journal of the Royal Statistical Society A, 135, 370-384.
33.
Patz, R. J., & Junker, B. W. (1999). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24, 342-366.
34.
Patz, R. J., Junker, B. W., Johnson, M. S., &Mariano, L. T. (2002). The hierarchical rater model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics, 27, 341-384.
35.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Institute of Educational Research.
36.
Rijmen, F., & De Boeck, P. (2002). The random weights linear logistic test model. Applied Psychological Measurement, 26, 269-283.
37.
Rijmen, F., Tuerlinckx, F., De Boeck, P., &Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8, 185-205.
38.
Rosenbaum, P. R. (1988). Item bundles. Psychometrika, 53, 349-359.
39.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 17, 1-100.
40.
SAS Institute. (1999). The NLMIXED procedure [Computer software]. Cary, NC: Author.
41.
Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28, 237-247.
42.
Skrondal, A., & Rabe-Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal and structural equation models. Boca Raton, FL: Chapman & Hall/CRC Press.
43.
Smith, E. V., Jr., & Smith, R. M. (Eds.). (2004). Introduction to Rasch measurement. Maple Grove, MN: JAM Press.
44.
StataCorp. (2003). Stata statistical software (release 8) [Computer software and manual]. College Station, TX: Stata Press.
45.
Thissen, D., Steinberg, L., & Mooney, J. A. (1989). Trace lines for testlets: A use of multiple-categorical-response models. Journal of Educational Measurement, 26, 247-260.
46.
Tuerlinckx, F., De Boeck, P., & Lens, W. (2002). Measuring needs with the Thematic Apperception Test: A psychometric study. Journal of Personality and Social Psychology, 82, 448-461.
47.
Volodin, N., & Adams, R. J. (1995, April). Identifying and estimating a D-dimensional Rasch model. Paper presented at the International Objective Measurement Workshop, University of California at Berkeley, California.
48.
Wainer, H. (1995). Precision and differential item functioning on a testlet-based test: The 1991 Law School Admissions Test as an example. Applied Measurement in Education, 8, 157-186.
49.
Wainer, H., Bradlow, E. T., & Du, Z. (2000). Testlet response theory: An analog for the 3PL model using in testlet-based adaptive testing. In W. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 245-269). London: Kluwer.
50.
Wainer, H., & Kiely, G. (1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24, 185-202.
51.
Wainer, H., & Lukhele, R. (1997). How reliable are TOEFL scores?Educational and Psychological Measurement, 57, 749-766.
52.
Wainer, H., & Thissen, D. (1996). How is reliability related to the quality of test scores?What is the effect of local dependence on reliability? Educational Measurement: Issues and Practice, 15(1), 22-29.
53.
Wainer, H., & Wang, X. (2000). Using a new statistical model for testlets to score TOEFL. Journal of Educational Measurement, 37, 203-220.
54.
Wang, W.-C. (2003). Ranking academic journals of psychology in Taiwan (NSC Research Report No. NSC91-2413-H-194-032). Taipei, Taiwan: National Science Council.
55.
Wang, W.-C., Chen, P.-H., & Cheng, Y.-Y. (2004). Improving measurement precision of test batteries using multidimensional item response models. Psychological Methods, 9, 116-136.
56.
Wang, W.-C., & Wilson, M. R. (2005). The Rasch testlet model. Applied Psychological Measurement, 29, 126-149.
57.
Wang, X., Bradlow, E. T., & Wainer, H. (2002). A general Bayesian model for testlets: Theory and applications. Applied Psychological Measurement, 26, 109-128.
58.
Wilson, M. R., & Adams, R. J. (1995). Rasch models for item bundles. Psychometrika, 60, 181-198.
59.
Wilson, M. R., & Hoskens, M. (2001). The rater bundle model. Journal of Educational and Behavioral Statistics, 26, 283-306.
60.
Wolfinger, R. D., & SAS Institute. (n.d.). Fitting nonlinear mixed models with the new NLMIXED procedure. Retrieved August 17, 2003, from http://support.sas.com/rnd/app/papers/nlmixedsugi.pdf
61.
Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. Chicago: MESA Press.
62.
Wu, M. L., Adams, R. J., & Wilson, M. R. (1998). ConQuest [Computer software and manual]. Camberwell, Victoria, Australia: Australian Council for Educational Research.
63.
Yen, W. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145.
64.
Yen, W. (1993). Scaling performance assessment:Strategies for managing local item dependence. Journal of Educational Measurement, 30, 187-213.
