Mutual Information Item Selection in Adaptive Classification Testing

Abstract

A general approach for item selection in adaptive multiple-category classification tests is provided. The approach uses mutual information (MI), a special case of the Kullback-Leibler distance, or relative entropy. MI works efficiently with the sequential probability ratio test and alleviates the difficulties encountered with using other local- and global-information measures in the multiple-category classification setting. Results from simulation studies using three item selection methods, Fisher information (FI), posterior-weighted FI (FIP), and MI, are provided for an adaptive four-category classification test. Both across and within the four classification categories, it is shown that in general, MI item selection classifies the highest proportion of examinees correctly and yields the shortest test lengths. The next best performance is observed for FIP item selection, followed by FI.

Keywords

computerized adaptive testing item selection mutual information sequential probability ratio test classification

Get full access to this article

View all access options for this article.

References

Chang, H.-H. , & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213-229.

Chen, S.-Y. , Ankenmann, R. D. , & Chang, H.-H. (2000). A comparison of item selection rules at the early stages of computerized adaptive testing. Applied Psychological Measurement, 24, 241-255.

Cheng, P. E. , & Liou, M. (2000). Estimation of trait level in computerized adaptive testing. Applied Psychological Measurement, 24, 257-265.

Cover, T. M. , & Thomas, J. A. (1991). Elements of information theory. New York: John Wiley.

Eggen, T. J. H. M. (1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement, 23(3), 249-261.

Eggen, T. J. H. M. , & Straetmans, G. J. J. M. (2000). Computerized adaptive testing for classifying examinees into three categories. Educational and Psychological Measurement, 60(5), 713-734.

Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

McGlohen, M. K. , Chang, H.-H. , & Wills, J. T. (2004, April). Combining computer adaptive testing technology with cognitively diagnostic assessment. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.

Reckase, M. D. (1983). A procedure for decision making using tailored testing. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 237-255). New York: Academic Press.

10.

Rudner, L. M. (2002, April). An examination of decision-theory adaptive testing procedures. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

11.

Spray, J. A. , & Reckase, M. D. (1994, April). The selection of test items for decision making with a computerized adaptive test. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.

12.

Spray, J. A. , & Reckase, M. D. (1996). Comparison of SPRT and sequential Bayes procedures for classifying examinees into two categories using a computerized test. Journal of Educational and Behavioral Statistics, 21(4), 405-414.

13.

Stone, C. A. , Weissman, A. , & Lane, S. (2005). Consistency of student proficiency classifications under competing IRT models for a state assessment program. Educational Assessment, 10(2), 125-146.

14.

van der Linden, W. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63(2), 201-216.

15.

Veerkamp, W. J. J. , & Berger, M. P. F. (1997). Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22, 203-226.

16.

Wald, A. (1947). Sequential analysis. New York: John Wiley.

17.

Xu, X. , Chang, H.-H. , & Douglas, J. (2003, April). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.