The usefulness of multidimensional adaptive testing (MAT) for the assessment of student literacy in the Programme for International Student Assessment (PISA) was examined within a real data simulation study. The responses of N = 14,624 students who participated in the PISA assessments of the years 2000, 2003, and 2006 in Germany were used to simulate MAT with different restrictions (unrestricted, treatment of link items, treatment of open items, content balance, unitwise item selection, all restrictions). Compared with conventional testing based on the booklet design of PISA 2006, unrestricted MAT increases measurement efficiency by 74% and reduces the average number of presented items from 55 to 26 without a loss in measurement precision. The incorporation of restrictions reduces the advantages of MAT. MAT is recommended for the assessment of newly introduced constructs but not for the assessment of the literacy domains in PISA.
Adams, R.J. ( 2005). Reliability as a measurement design effect. Studies in Educational Evaluation, 31, 162-172.
2.
Adams, R. J., & Wu, M. (Eds.). (2002). PISA 2000 technical report . Paris, France: OECD.
3.
Bartram, D., & Hambleton, R.K. (2005). Computer-based testing and the Internet: Issues and advances. New York, NY: Wiley.
4.
Cheng, Y., & Chang, H.H. ( 2009). The maximum priority index method for severely constrained item selection in computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 62, 369-383.
5.
Frey, A. ( 2007). Adaptives Testen [Adaptive testing]. In H. Moosbrugger & A. Kelava (ed.), Testtheorie und Fragebogenkonstruktion [Test theory and construction of questionnaires].(pp. 261-278). Berlin, Germany: Springer.
6.
Frey, A., Hartig, J., & Rupp, A. ( 2009). Booklet designs in large-scale assessments of student achievement: Theory and practice. Educational Measurement: Issues and Practice,28, 39-53.
7.
Frey, A., & Seitz, N.N. ( 2009). Multidimensional adaptive testing in educational and psychological measurement: Current state and future challenges. Studies in Educational Evaluation, 35, 89-94.
8.
Frey, A., & Seitz, N.N. ( 2010). Multidimensionale adaptive Kompetenzdiagnostik: Ergebnisse zur Messeffizienz [Multidimensional adaptive testing of competencies: Results regarding measurement efficiency]. Zeitschrift für Pädagogik , 56, 40-51.
9.
Hartig, J., & Hoehler, J. ( 2008). Representation of competencies in multidimensional IRT models with within-item and between-item multidimensionality. Zeitschrift für Psychologie, 216, 89-101.
10.
Masters, G.N., & Wright, B.D. ( 1997). The partial credit model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 101-122). New York, NY: Springer.
11.
Oecd. (2001). Knowledge and skills for life: First results from PISA 2000. Paris, France: Author.
12.
Oecd. (2004). Learning for tomorrow’s world: First results from PISA 2003. Paris, France: Author.
Sands, W. A., Waters, B. K., & McBride, J. R. (Eds.). (1997). Computerized adaptive testing: From inquiry to operation. Washington, DC: American Psychological Association.
Segall, D.O. ( 2001). General ability measurement: An application of multidimensional item response theory. Psychometrika, 66, 79-97.
22.
Segall, D.O. ( 2005). Computerized adaptive testing. In K. Kempf-Leonard (Ed.), Encyclopedia of social measurement . New York, NY: Academic Press.
23.
van der Linden, W.J. ( 1999). Multidimensional adaptive testing with a minimum error-variance criterion. Journal of Educational and Behavioral Statistics , 24, 398-412.
24.
van der Linden, W. J., & Hambleton, R. K. (Eds.). (1997). Handbook of modern item response theory. New York, NY: Springer.
25.
Veldkamp, B.P., & van der Linden, W.J. (2002). Multidimensional adaptive testing with constraints on test content. Psychometrika, 67, 575-588.
26.
Veldkamp, B.P., & van der Linden, W.J. (2010). Designing item pools for adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Elements of adaptive testing (pp. 231-245). New York, NY: Springer.
