Jointly Modeling Omitted and Not-Reached Items in Time-Limit Tests: A Survival Analysis Approach

Abstract

Item response theory models are widely used and play an important role in psychological measurement and educational assessment. In its practical application, missing data is unavoidable, especially the not-reached and omitted items. For such missingness, if not handled properly, a biased parameter estimation would be derived, leading to an inaccurate estimation of examinees. To address this issue, this paper develops a joint model allowing for missing responses resulting from not-reached and omitted items. The proposed model simultaneously incorporates responses, response time, and omission time, using a survival analysis framework. Under the proposed model, the missing mechanism of not-reached items (NRIs) is interpreted by the right censoring of potential response times for examinees’ first NRIs. Furthermore, the missing mechanism of omitted items is explained by the competing relationship between response time and omission time. The simulation study confirms the consistent recovery of the proposed method. In addition, the comparison of the proposed method and alternative methods showed that only the method accounting for both missing patterns could recover model parameters well. An empirical study based on the Program for International Student Assessment 2015 Science Test was further conducted.

Keywords

not-reached item omitted item response time omission time item response theory

Get full access to this article

View all access options for this article.

References

Baker

F. B.

Kim

S.-H.

(2004). Item response theory: Parameter estimation techniques. CRC Press.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley.

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29–51.

Dai

(2021). Handling missing responses in psychometrics: Methods and software. Psych, 3(4), 673–693.

Dempster

A. P.

Laird

N. M.

Rubin

D. B.

(1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 39(1), 1–22.

Deribo

Kroehne

Goldhammer

(2021). Model-based treatment of rapid guessing. Journal of Educational Measurement, 58(2), 281–303.

Glas

C. A.

Pimentel

J. L.

(2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68(6), 907–922.

Guo

Fang

Ying

Zhang

(2025). Jointly modeling responses and omitted items by a competing risk model: A survival analysis approach. British Journal of Mathematical and Statistical Psychology. Advance online publication. https://doi.org/10.1111/bmsp.12382

Guo

Ying

Zhang

(2022). Modeling not-reached items in timed tests: A response time censoring approach. Psychometrika, 87(3), 835–867.

10.

Holman

Glas

C. A.

(2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58(1), 1–17.

11.

Johnson

Allen

(1992). The 1990 NAEP technical report (no. 21–tr–20). https://files.eric.ed.gov/fulltext/ED396903.pdf

12.

Kendall

M. G.

(1938). A new measure of rank correlation. Biometrika, 30(1–2), 81–93.

13.

Lord

F. M.

(1974). Estimation of latent ability and item parameters when there are omitted responses. Psychometrika, 39(2), 247–264.

14.

Wang

(2020). A response time process model for not-reached and omitted items. Journal of Educational Measurement, 57(4), 584–620.

15.

Wang

Shi

(2023). A mixture response time process model for aberrant behaviors and item nonresponses. Multivariate Behavioral Research, 58(1), 71–89.

16.

Martin

M. O.

von Davier

Mullis

I. V.

(2020). Methods and procedures: TIMSS 2019 technical report. International Association for the Evaluation of Educational Achievement.

17.

Mislevy

R. J.

Wilson

(1996). Marginal maximum likelihood estimation for a psychometric model of discontinuous development. Psychometrika, 61(1), 41–71.

18.

Mislevy

R. J.

P.-K.

(1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing. Wiley Online Library.

19.

Moustaki

Knott

(2000). Weighting for item non-response in attitude scales by using latent variable models with covariates. Journal of the Royal Statistical Society: Series A (Statistics in Society), 163(3), 445–459.

20.

Muraki

(1992). A generalized partial credit model: Application of an EM algorithm. Wiley Online Library.

21.

O’Muircheartaigh

Moustaki

(1999). Symmetric pattern models: A latent variable approach to item non-response in attitude scales. Journal of the Royal Statistical Society: Series A (Statistics in Society), 162(2), 177–194.

22.

Organisation for Economic Co-operation and Development (OECD). (2009). PISA 2006 technical report.

23.

Organisation for Economic Co-operation and Development (OECD). (2013). Technical report of the survey of adult skills (PIAAC).

24.

Organisation for Economic Co-operation and Development (OECD). (2021). PISA 2018 technical report.

25.

Organisation for Economic Co-operation and Development (OECD). (2024). PISA 2022 technical report.

26.

Pohl

Becker

(2020). Performance of missing data approaches under nonignorable missing data conditions. Methodology, 16(2), 147–165.

27.

Pohl

Haberkorn

Hardt

Wiegand

(2012). NEPS technical report for reading–scaling results of starting cohort 3 in fifth grade. National Educational Panel Study Bamberg, University of Bamberg.

28.

Pohl

Ulitzsch

von Davier

(2019). Using response times to model not-reached items due to time limits. Psychometrika, 84(3), 892–920.

29.

Robitzsch

(2023). Regularized Mislevy-Wu model for handling nonignorable missing item responses. Information, 14(7), 368.

30.

Rose

von Davier

Nagengast

(2017). Modeling omitted and not-reached items in IRT models. Psychometrika, 82(3), 795–819.

31.

Rose

von Davier

(2010). Modeling nonignorable missing data with item response theory (IRT). Wiley Online Library.

32.

Sahin

Anil

(2017). The effects of test length and sample size on item parameters in item response theory. Educational Sciences: Theory & Practice, 17(1), 321–335.

33.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(1), 1–97.

34.

Thissen

Wainer

(1982). Some standard errors in item response theory. Psychometrika, 47(4), 397–412.

35.

Ulitzsch

von Davier

Pohl

(2020a). A multiprocess item response model for not-reached items due to time limits and quitting. Educational and Psychological Measurement, 80(3), 522–547.

36.

Ulitzsch

von Davier

Pohl

(2020b). Using response times for joint modeling of response and omission behavior. Multivariate Behavioral Research, 55(3), 425–453.

37.

Ulitzsch

Zhang

Pohl

(2024). A model-based approach to the disentanglement and differential treatment of engaged and disengaged item omissions. Multivariate Behavioral Research, 59, 599–619.

38.

van der Linden

W. J.

(2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204.

39.

van der Linden

W. J.

(2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308.

40.

van Rijn

P. W.

Ali

U. S.

(2018). A generalized speed–accuracy response model for dichotomous items. Psychometrika, 83(1), 109–131.

41.

Yin

Bezirhan

Fishbein

Foy

(2021). Implementing the PIRLS 2021 achievement scaling methodology. In Methods and procedures: PIRLS (pp. 1–83). https://pirls2021.org/wp-content/uploads/2023/05/P21_MP_Ch11-scaling-implementation.pdf

42.

Yuan

K.-H.

Cheng

Patton

(2014). Information matrices and standard errors for MLEs of item parameters in IRT. Psychometrika, 79(2), 232–254.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.26 MB