The selection of items from a calibrated item bank for fixed-form tests is an optimal test design problem; this problem has been handled in the literature by mathematical programming models. A similar problem, however, arises when items are selected for an adaptive test or for testlets. This paper focuses on the similarities of optimal design of fixed-form tests, adaptive tests, and testlets within the framework of the general theory of optimal designs. A sequential design procedure is proposed that uses these similarities. This procedure not only enables optimal design of fixed-form tests, adaptive tests, and testlets, but is also very flexible. The procedure is easy to apply, and consistent estimates for the trait level distribution are obtained. Index terms: adaptive tests, consistency, efficiency, optimal test design, sequential procedure, test design, testlets.
Get full access to this article
View all access options for this article.
References
1.
Adema, J.J. (1990). Models and algorithms for the construction of achievement tests. Doctoral dissertation, University of Twente, Enschede.
2.
Atkinson, A.C. (1982). Developments in the design of experiments. International Statistical Review, 50, 161-177.
3.
Baker, F.B., Cohen, A.S., & Barmish, B.R. (1988). Item characteristics of tests constructed by linear programming. Applied Psychological Measurement, 12, 189-199.
4.
Berger, M.P.F. (1991). On the efficiency of IRT models when applied to different sampling designs. Applied Psychological Measurement , 15, 293-306.
5.
Berger, M.P.F. (1992a). Generation of optimal designs for nonlinear models when the design points are incidental parameters. In Y. Dodge & J. Whittaker (Eds.), Computational statistics (Vol. 2; pp. 200-208). New York: Springer-Verlag.
6.
Berger, M.P.F. (1992b). Sequential sampling designs for the two-parameter item response theory model. Psychometrika, 57, 521-538.
7.
Berger, M.P.F. (1994). D-optimal sequential sampling designs for item response theory models. Journal of Educational Statistics , 19, 43-56.
8.
Berger, M.P.F., & Van der Linden, W.J. (1992). Optimality of sampling designs in item response theory models. In M. Wilson (Ed.), Objective measurement: Theory into practice (Vol. 1; pp. 274-288). Norwood NJ : Ablex.
9.
Berger, M.P.F., & Veerkamp, W.J.J. (in press). A review of selection methods for optimal test design. In M. Wilson & G. Engelhard (Eds.), Objective measurement: Theory into practice (Vol. 3). Norwood NJ: Ablex.
10.
Boekkooi-Timminga, E. (1989). Models for computerized test construction. Doctoral dissertation, University of Twente.
11.
Chaudhuri, P., & Mykland, P.A. (1993). Nonlinear experiments: Optimal design and inference based on likelihood. Journal of the American Statistical Association,88, 538-546.
12.
Ford, I., Kitsos, C.P., & Titterington, D.M. (1989). Recent advances in nonlinear experimental design. Technometrics, 31, 49-60.
13.
Ford, I., Titterington, D.M., & Wu, C.F.J. (1985). Inference and sequential design. Biometrika, 72, 545-551.
14.
Grambsch, P. (1989). Sequential maximum likelihood estimation with applications to logistic regression in case-control studies. Journal of StatisticalPlanning and Inference, 22, 355-369.
15.
Hambleton, R.K., Jones, R.W., & Rogers, H.J. (1993) Influence of item parameter errors in test development . Journal of Educational Measurement, 30, 143-156.
16.
Lord, F.M. (1971). Robbins-Monro procedures for tailored testing . Educational and Psychological Measurement, 31, 3-31.
17.
Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale NJ: Erlbaum.
Theunissen, T.J.J.M. (1986). Binary programming and test design. Psychometrika, 50, 411-420.
20.
Thissen, D., & Mislevy, R.J. (1990). Testing algorithms. In H. Wainer (Ed.), Computerized adaptive testing: A primer (pp. 103-135). Hillsdale NJ: Erlbaum.
21.
Tsay, J.Y. (1976). On the sequential construction of D-optimal designs . Journal of the American Statistical Association,71, 671-674.
22.
Van derLinden, W.J., & Boekkooi-Timminga, E. (1989). A maximin model for test design with practical constraints. Psychometrika, 53, 237-247.
23.
Veerkamp, W.J.J., & Berger, M.P.F. (1993). Some new item selection criteria for adaptive testing (Research Report). Enschede , The Netherlands: University of Twente, Faculty of Educational Science and Technology.
24.
Wainer, H. (1990). Computerized adaptive testing: A primer. Hillsdale NJ: Erlbaum.
25.
Wainer, H., Kaplan, B., & Lewis, C. (1992). A comparison of the performance of simulated hierarchical and linear testlets. Journal of Educational Measurement, 29, 243-251.
26.
Wainer, H., & Kiely, G.L. (1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24, 185-202.
27.
Wainer, H., & Lewis, C. (1990). Toward a psychometrics for testlets. Journal of Educational Measurement, 27, 1-14.
28.
Weiss, D.J. (1976). Adaptive testing research in Minnesota: Overview, recent results, and future directions. In C. L. Clark (Ed.), Proceedings of the First Conference on Computerized Adaptive Testing (pp. 24-35). Washington DC: United States Civil Service Commission.
29.
Weiss, D.J. (1978). Proceedings of the 1977 Computerized Adaptive Testing Conference. Minneapolis: University of Minnesota, Department of Psychology, Computerized Adaptive Testing Laboratory.
30.
Wu, C.F.J. (1985). Asymptotic inference from sequential design in nonlinear situation. Biometrika, 72, 533-558.
31.
Wu, C.F.J., & Wynn, H.P. (1978). The convergence of general step-length algorithms for regular optimum design criteria. The Annals of Statistics , 6, 1273-1285.
32.
Wynn, H.P. (1970). The sequential generation of D-optimum experimental designs. Annals of Mathematical Statistics, 41, 1655-1664.
