Impact of Test Design,Item Quality,and Item Bank Size on the Psychometric Properties of Computer-Based Credentialing Examinations

Abstract

Computer-based testing by credentialing agencies has become common; however, selecting a test design is difficult because several good ones are available—parallel forms, computer adaptive (CAT), and multistage (MST). In this study, three computerbased test designs under some common examination conditions were investigated. Item bank size and item quality had a practically significant impact on decision consistency and accuracy. Even in nearly ideal situations, the choice of test design was not a factor in the results. Two conclusions follow from the findings: (a) More time and resources should be committed to expanding the size and quality of item banks, and (b) designs that individualize an exam administration such as MST and CAT may not be helpful when the primary purpose of the examination is to make pass-fail decisions and conditions are present for using parallel forms with a target information function that can be centered on the passing score.

Get full access to this article

View all access options for this article.

References

Bock, B. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.

Chang, H. H., & Ying, Z. (1999). A-stratified multi-stage computerized adaptive testing. Applied Psychological Measurement, 23, 211-222.

Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.

Jodoin, M., Zenisky, A., & Hambleton, R. K. (2002, April). Comparison of the psychometric properties of several computer-based test designs for credentialing exams. Paper presented at the meeting of the National Council on Measurement in Education, New Orleans.

Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

Luecht, R. M. (1998). Computer-assisted test assembly using optimization heuristics. Applied Psychological Measurement, 22, 224-236.

Luecht, R. M.,& Clauser, B. E. (2002). Test models for complex computer-based testing. In C. N. Mills, M. T. Potenza, J. J. Fremer, & W. C. Ward (Eds.), Computer-based testing: Building the foundation for future assessments (pp. 67-88). Hillsdale, NJ: Lawrence Erlbaum.

Pitoniak, M. (2002). Automatic item generation methodology in theory and practice (Center for Educational Assessment Research Report No. 444). Amherst: University of Massachusetts, School of Education.

Robin, F. (2000). CBTS: Computer-based testing simulation and analysis [Computer software]. Amherst: University of Massachusetts, School of Education.

10.

van der Linden, W. J., & Glas, C. A. W. (2000). Computer adaptive testing: Theory and practice. Boston: Kluwer Academic.

11.

Veldkamp, B. P., & van der Linden, W. J. (2000). Designing item pools for computerized adaptive testing. In W. J. van der Linden & C. A. Glas (Eds.), Computer adaptive testing: Theory and practice. Boston: Kluwer Academic.

12.

Wainer, H., Dorans, N. J., Eignor, D., Flaugher, R., Green, B. F., Mislevy, R. J., et al. (Eds.). (2000). Computerized adaptive testing: A primer (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.