In a randomized experiment (n = 515), a computerized and a computerized adaptive test (CAT) are compared. The item pool consists of 24 polytomous motivation items. Although items are carefully selected, calibration data show that Samejima's graded response model did not fit the data optimally. A simulation study is done to assess possible consequences of model misfit. CAT efficiency was studied by a systematic comparison of the CAT with two types of conventional fixed length short forms, which are created to be good CAT competitors. Results showed no essential administration mode effects. Efficiency analyses show that CAT outperformed the short forms in almost all aspects when results are aggregated along the latent trait scale. The real and the simulated data results are very similar, which indicate that the real data results are not affected by model misfit.
Andrich, D. (1978). Application of a psychometric rating model to ordered categories which are scored with successive integers. Applied Psychological Measurement, 2, 581-594.
2.
Baek, S.G. (1995). Computerized adaptive attitude testing using the partial credit model. Dissertation Abstracts International-A,55(7-A), 1922 (UMI No. AAM9430378). Retrieved March 10, 2003, from PsychInfo database.
3.
Balasubramanian, S.K., & Kamakura Wagner, A. (1989). Measuring consumer attitudes toward the marketplace with tailored interviews. Journal of Marketing Research, 26, 311-326.
4.
Bock, R.D., & Lieberman, M. (1970). Fitting a response model for n dichotomously scored items. Psychometrika, 35, 179-197.
5.
Chang, H.H., Qian, J., & Ying, Z. (2001). a-Stratified multistage computerized adaptive testing with b blocking. Applied Psychological Measurement , 25, 333-341.
6.
Chernyshenko, O.S., Stark, S., Chan, K.Y., Drasgow, F., & Williams, B. (2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523-562.
7.
Dodd, B.G., Koch, W.R., & de Ayala, R.J. (1989). Operational characteristics of adaptive testing procedures using the graded response model. Applied Psychological Measurement, 13, 129-143.
8.
Drasgow, F., Levine, M.V., Tsien, S., Williams, B., & Mead, A.D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement , 19, 143-165.
9.
Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
10.
Finger, M.S., & Ones, D.S. (1999). Psychometric equivalence of the computer and booklet forms of the MMPI: A meta-analysis. Psychological Assessment , 11, 58-66.
11.
Gibbons, J.D. (1985). Nonparametric statistical inference. New York: Marcel Dekker.
12.
Hambleton, R.K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff .
13.
Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75, 800-802.
14.
Hol, A.M., Vorst, H.C.M., & Mellenbergh, G.J. (2001). Toepassing van een computergestuurde adaptieve testprocedure op persoonlijkheidsdata [Application of a computerized adaptive test procedure on personality data]. Nederlands Tijdschrift voor de Psychologie en haar Grensgebieden, 56, 119-133.
15.
Hol, A.M., Vorst, H.C.M., & Mellenbergh, G.J. (2005). A randomized experiment to compare conventional, computerized, and computerized adaptive administration of ordinal polytomous attitude items. Applied Psychological Measurement, 29, 159-183.
16.
Koch, W.R., Dodd, B.G., & Fitzpatrick, S.J. (1990). Computerized adaptive measurements of attitudes. Measurement and Evaluation in Counseling and Development, 23, 20-30.
17.
Lima Passos, V., Berger, M.P.F., & Tan, F.E.S. (2004). D-optimality item selection criterion in CAT early stage—A study with the Graded Response Model. Manuscript submitted for publication.
18.
MacDonald, P.L. (2003). Computer-adaptive test for measuring personality factors using item response theory. Dissertation-Abstracts-International-B , 64(2-B), 999. (UMI No. AAINQ77103)
19.
Masters, G.N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
20.
Mead, A.D., & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. Psychological Bulletin , 114, 449-458.
21.
Mokken, R.J. (1971). A theory and procedure of scale analysis: With applications in political research. The Hague, the Netherlands: Mouton.
22.
Mokken, R.J. (1997). Nonparametric models for dichotomous responses. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 351-367). New York: Springer-Verlag.
23.
Molenaar, I.W. (1997). Nonparametric models for polytomous responses. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 369-380). New York: Springer-Verlag.
24.
Molenaar, I.W., Van Schuur, W.H., Sijtsma, K., & Mokken, R.J. (2000). Mokken scale analysis for polytomous items (Version 5.0) [computer software] . Groningen, the Netherlands: Pro Gamma.
25.
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
26.
Reckase, M.D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics , 4, 207-230.
27.
Reise, S.P., & Henson, J.M. (2000). Computerization and adaptive administration of the NEO PI-R. Assessment,7, 347-364.
28.
Reise, S.P., & Henson, J.M. (2003). A discussion of modern versus traditional psychometrics as applied to personality assessment scales. Journal of Personality Assessment, 81, 93-103.
29.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement , 34, 100.
30.
Samejima, F. (1997). Graded response model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 85-100). New York: Springer-Verlag.
31.
Sijtsma, K., & Molenaar, I.W. (2002). Introduction to nonparametric item response theory. Thousand Oaks, CA: Sage.
32.
Singh, J., Howell, R.D., & Rhoads, G.K. (1990). Adaptive designs for Likert-type data: An approach for implementing marketing surveys. Journal of Marketing Research , 27, 304-321.
33.
Thissen, D. (2003). Multilog for Windows (Version 7.0) [computer software] . Lincolnwood, IL: Scientific Software International.
34.
van der Ark, L.A. (2001). Relationships and properties of polytomous item response theory models. Applied Psychological Measurement , 25, 273-282.
Waller, N.G. (1999). Searching for structure in the MMPI. In S. E. Embretson & S. L. Hershberger (Eds.), The new rules of measurement: What every psychologist and educator should know (pp. 185-217). Mahwah, NJ: Lawrence Erlbaum.
37.
Waller, N.G. (2003). MicroFACT: A microcomputer factor analysis program for ordered polytomous data and mainframe size problems (Version 2.1) [computer software]. St. Paul, MN: Assessment Systems Corporation.
38.
Waller, N.G., & Reise, S.P. (1989). Computerized adaptive personality assessment: An illustration with the Absorption scale. Journal of Personality and Social Psychology, 57, 1051-1058.
39.
Weiss, D.J. (2004). Computerized adaptive testing for effective and efficient measurement in counseling and education. Measurement and Evaluation in Counseling and Development, 37, 70-84.
