Maximum Information per Time Unit Designs for Calibrating Multidimensional Items Online

Abstract

In multidimensional computerized adaptive testing (MCAT), the existing online calibration designs do not account for the time spent on items, potentially resulting in reduced calibration efficiency. To improve the calibration efficiency for MCAT, response time was used as auxiliary information for the online calibration of items. Drawing upon the D-optimal design for MCAT (M-D design) and A-optimal design for MCAT (M-A design), this study introduced the D-optimal per expectation time unit design (M-D-ET design) and A-optimal per expectation time unit design (M-A-ET design) for MCAT. These two new designs prioritized the selection of new items that yield the greatest increment in calibration information per time unit for examinees. Simulation studies were conducted to compare the performances of the M-D, M-A, M-D-ET, and M-A-ET designs across various scenarios. The results indicate that, prior to the calibration of approximately 70% of the new items, the M-D-ET/M-A-ET designs outperformed the M-D/M-A designs in terms of calibration information per time unit and average response time. In the continuous testing scenario, the M-D-ET/M-A-ET designs outperformed the M-D/M-A designs by generating more information per time unit, reducing response time, and maintaining approximately equivalent calibration accuracy. A study based on real response data yielded comparable findings.

Keywords

multidimensional computerized adaptive testing online calibration design response time multidimensional two-parameter logistic model

Get full access to this article

View all access options for this article.

References

Ali

U. S.

Chang

H. H.

(2014). An item-driven adaptive design for calibrating pretest items. ETS Research Report Series, 2014(2), 1–12.

Berger

M. P.

(1991). On the efficiency of IRT models when applied to different sampling designs. Applied Psychological Measurement, 15(3), 293–306.

Chalmers

Pritikin

Robitzsch

Zoltak

Kim

Falk

C. F.

Meade

(2015). mirt: Multidimensional item response theory. R package version 1.14. http://CRAN.R-project.org/package=mirt

Chen

Wang

(2016). A new online calibration method for multidimensional computerized adaptive testing. Psychometrika, 81(3), 674–701.

Chen

Wang

Xin

Chang

H. H.

(2017). Developing new online calibration methods for multidimensional computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 70(1), 81–117.

Chen

Zhang

Xin

(2013). Application of online calibration technique in computerized adaptive testing. Advances in Psychological Science, 21(10), 1883.

Cheng

Yuan

K. H.

(2010). The impact of fallible item parameter estimates on latent trait recovery. Psychometrika, 75(2), 280–291.

Choe

E. M.

Kern

J. L.

Chang

H. H.

(2018). Optimizing the use of response times for item selection in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 43(2), 135–158.

Entink

R. K.

van der Linden

W. J.

Fox

J. P.

(2009). A Box–Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62(3), 621–640.

10.

Fan

Wang

Chang

H. H.

Douglas

(2012). Utilizing response time distributions for item selection in CAT. Journal of Educational and Behavioral Statistics, 37(5), 655–670.

11.

Gulliksen

(2013). Theory of mental tests. Routledge.

12.

Guo

Wang

(2003). Online calibration and scale stability of a CAT program. Annual meeting of the National Council on Measurement in Education, Chicago, IL.

13.

Guo

Shang

Xia

(2017). Advantages and illustrations of application of response time model in psychological and educational testing. Advances in Psychological Science, 25(4), 701–712.

14.

Hassan

M. U.

Miller

(2019). Optimal item calibration for computerized achievement tests. Psychometrika, 84(4), 1101–1128.

15.

Chen

(2020). New efficient and practicable adaptive designs for calibrating items online. Applied Psychological Measurement, 44(1), 3–16.

16.

Chen

(2021). Maximum information per time unit designs for continuous online item calibration. British Journal of Mathematical and Statistical Psychology, 74, 24–51.

17.

Y. Y.

(2023). Using response time in multidimensional computerized adaptive testing. Journal of Educational Measurement, 60(4), 697–738.

18.

Jones

D. H.

Jin

(1994). Optimal sequential designs for on-line item estimation. Psychometrika, 59(1), 59–75.

19.

Kang

H. A.

Zheng

Chang

H. H.

(2020). Online calibration of a joint model of item responses and response times in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 45(2), 175–208.

20.

Kern

J. L.

Choe

(2021). Using a response time–based expected a posteriori estimator to control for differential speededness in computerized adaptive test. Applied Psychological Measurement, 45(5), 361–385.

21.

Lee

Y. H.

E. H.

Fuh

C. D.

(2008). A strategy for controlling item exposure in multidimensional computerized adaptive testing. Educational and Psychological Measurement, 68(2), 215–232.

22.

Man

Harring

J. R.

Jiao

Zhan

(2019). Joint modeling of compensatory multidimensional item responses and response times. Applied Psychological Measurement, 43(8), 639–654.

23.

Ranger

Kuhn

J. T.

(2012). A flexible latent trait model for response times in tests. Psychometrika, 77(1), 31–47.

24.

Reckase

M. D.

(1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9(4), 401–412.

25.

Reckase

M. D.

(2009). Multidimensional item response theory models. In Reckase

M. D.

(Ed.), Multidimensional item response theory (pp. 79–112). Springer.

26.

Ren

van der Linden

W. J.

Diao

(2017). Continuous online item calibration: Parameter recovery and item utilization. Psychometrika, 82(2), 498–522.

27.

Robitzsch

Kiefer

George

C. A.

Uenlue

(2015). CDM: Cognitive diagnosis modeling. R package version 4.6-0. http://CRAN.R-project.org/package=CDM

28.

Samejima

(1974). Normal ogive model on the continuous response level in the multidimensional latent space. Psychometrika, 39(1), 111–121.

29.

Segall

D. O.

(1996). Multidimensional adaptive testing. Psychometrika, 61(2), 331–354.

30.

Stocking

M. L.

(1990). Specifying optimum examinees for item parameter estimation in item response theory. Psychometrika, 55(3), 461–475.

31.

van der Linden

W. J.

(2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204.

32.

van der Linden

W. J.

(2008). Using response times for item selection in adaptive testing. Journal of Educational and Behavioral Statistics, 33(1), 5–20.

33.

van der Linden

W. J.

Guo

(2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365–384.

34.

van der Linden

W. J.

Ren

(2015). Optimal Bayesian adaptive design for test-item calibration. Psychometrika, 80(2), 263–288.

35.

van der Linden

W. J.

van Krimpen-Stoop

E. M.

(2003). Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika, 68(2), 251–265.

36.

Veldkamp

B. P.

(2016). On the issue of item selection in computerized adaptive testing with response times. Journal of Educational Measurement, 53(2), 212–228.

37.

Wainer

Mislevy

R. J.

(1990). Item response theory, item c3alibration, and proficiency estimation. Computerized Adaptive Testing: A Primer, 4, 65–102.

38.

Wang

(2015). On latent trait estimation in multidimensional compensatory item response models. Psychometrika, 80, 428–449.

39.

Wang

Chang

H. H.

Boughton

K. A.

(2011). Kullback–Leibler information and its applications in multi-dimensional adaptive testing. Psychometrika, 76(1), 13–39.

40.

Wang

Weiss

D. J.

Shang

(2019). Variable-length stopping rules for multidimensional computerized adaptive testing. Psychometrika, 84(3), 749–771.

41.

Zheng

(2014). New methods of online calibration for item bank replenishment. University of Illinois at Urbana-Champaign.

42.

Zheng

Chang

H. H.

(2017). A comparison of five methods for pretest item selection in online calibration. International Journal of Quantitative Research in Education, 4(1/2), 133–158.