The Rank-2PL IRT Models for Forced-Choice Questionnaires: Maximum Marginal Likelihood Estimation with an EM Algorithm

Abstract

The rank two-parameter logistic (Rank-2PL) item response theory models refer to a set of models applying the 2PL model in a sequential ranking process that occurs in forced-choice questionnaires. The multi-unidimensional pairwise preference with 2PL model (MUPP-2PL) is a Rank-2PL model for items with two statements. Focusing on items with three statements, we develop a maximum marginal likelihood estimation with an expectation-maximization algorithm to estimate item parameters and their standard errors. A simulation study is conducted to check parameter recovery, and then the model is applied to a real dataset. Finally, the findings are summarized and discussed, and future research is suggested.

Keywords

forced-choice questionnaire rank two-parameter logistic model item response theory maximum marginal likelihood estimation expectation-maximization algorithm

Get full access to this article

View all access options for this article.

References

Atkinson

K. E.

(1989). An introduction to numerical analysis (2nd ed.). John Wiley.

Birkeland

S. A.

Manson

T. M.

Kisamore

J. L.

Brannick

M. T.

Smith

M. A.

(2006). A meta-analytic investigation of job applicant faking on personality measures. International Journal of Selection and Assessment, 14(4), 317–335. https://doi.org/10.1111/j.1468-2389.2006.00354.x

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley Pub. Co.

Bock

R. D.

Aitkin

(1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443–459. https://doi.org/10.1007/BF02293801

Brown

(2016). Item response models for forced-choice questionnaires: A common framework. Psychometrika, 81(4), 135–160. https://doi.org/10.1007/s11336-014-9434-9

Brown

Maydeu-Olivares

(2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502. https://doi.org/10.1177/0013164410375112

Brown

Maydeu-Olivares

(2012). Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods, 44, 1135–1147. https://doi.org/10.3758/s13428-012-0217-x

Broyden

C. G.

(1970). The convergence of a class of double-rank minimization algorithms 1. General considerations. IMA Journal of Applied Mathematics, 6(1), 76–90. https://doi.org/10.1093/imamat/6.1.76

Bürkner

P.-C.

Schulte

Holling

(2019). On the statistical and practical limitations of Thurstonian IRT models. Educational and Psychological Measurement, 79(5), 827–854. https://doi.org/10.1177/0013164419832063

10.

Cai

(2010). High-dimensional exploratory item factor analysis by a Metropolis-Hastings Robbins-Monro algorithm. Psychometrika, 75(1), 33–57. https://doi.org/10.1007/s11336-009-9136-x

11.

Cai

(2023). flexMIRT (Version 3.6): Flexible multilevel multidimensional item analysis and test scoring [Computer software]. Vector Psychometric Group. https://vpgcentral.com/software/flexmirt/

12.

Cai

Hansen

(2013). Limited-information goodness-of-fit testing of hierarchical item factor models. British Journal of Mathematical and Statistical Psychology, 66(2), 245–276. https://doi.org/10.1111/j.2044-8317.2012.02050.x

13.

Chalmers

R., P.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/1.18637/jss.v048.i06

14.

Chalmers

R. P.

(2018). Numerical approximation of the observed information matrix with Oakes’ identity. British Journal of Mathematical and Statistical Psychology, 71(3), 415–436. https://doi.org/10.1111/bmsp.12127

15.

Chen

W. H.

Thissen

(1997). Local dependence indices for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289. https://doi.org/10.2307/1165285

16.

Chen

Zhang

(2021). Estimation methods for item factor analysis: An overview. In Zhao

Chen

D. G.

(Eds.), Modern statistical methods for health research: Emerging topics in statistics and biostatistics (pp. 329–350). Springer, Cham. https://doi.org/10.1007/978-3-030-72437-5_15

17.

Davis

P. J.

Rabinowitz

(1984). Methods of numerical integration. Academic Press. https://doi.org/10.1016/B978-0-12-206360-2.50012-1

18.

de la Torre

Ponsoda

Leenen

Hontangas

(2012, April). Examining the viability of recent models for forced-choice data [Paper presentation]. American Educational Research Association.

19.

Drasgow

Levine

M. V.

Williams

E. A.

(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86. https://doi.org/10.1111/j.2044-8317.1985.tb00817.x

20.

Forero

C. G.

Maydeu-Olivares

(2009). Estimation of IRT graded response models: Limited versus full information methods. Psychological Methods, 14(3), 275–299. https://doi.org/10.1037/a0015825

21.

(2019). Maximum marginal likelihood estimation with an expectation-maximization algorithm for multigroup/mixture multidimensional item response theory models. (ETS Research Report No. 19–35). Educational Testing Service. https://doi.org/10.1002/ets2.12272

22.

Kyllonen

P. C.

Tan

(2024): From Likert to forced choice: Statement parameter invariance and context effects in personality assessment. Measurement: Interdisciplinary Research and Perspectives. Advance online publication. https://doi.org/10.1080/15366367.2023.2258482

23.

Garnier-Villarreal

Merkle

E. C.

Magnus

B. E.

(2021). Between-item multidimensional IRT: How far can the estimation methods go? Psych, 3, 404–421. https://doi.org/10.3390/psych3030029

24.

Gilks

W. R.

Richardson

Spiegelhalter

D. J.

(1996). Introducing Markov Chain Monte Carlo. In Gilks

W. R.

Richardson

Spiegelhalter

D. J.

(Eds.), Markov Chain Monte Carlo in practice. (pp. 1–20). Chapman and Hall. https://doi.org/10.1201/b14835

25.

Haberman

S. J.

(2013). A general program for item-response analysis that employs the stabilized Newton-Raphson algorithm (Research Report No. RR-13-32). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.2013.tb02339.x

26.

Hontangas

P. M.

de la Torre

Ponsoda

Leenen

Morillo

Abad

F. J.

(2015). Comparing traditional and IRT scoring of forced-choice tests. Applied Psychological Measurement, 39(8), 598–612. https://doi.org/10.1177/0146621615585851

27.

Kim

(2012). A note on the reliability coefficients for item response model-based ability estimates. Psychometrika, 77(4), 153–162. https://doi.org/10.1007/s11336-011-9238-0

28.

Lee

Joo

S. H.

Stark

Chernyshenko

O. S.

(2019). GGUM-RANK statement and person parameter estimation with multidimensional forced choice triplets. Applied Psychological Measurement, 43(3), 226–240. https://doi.org/10.1177/0146621618768294

29.

Louis

T.A.

(1982). Finding the observed information matrix when using the EM algorithm. Journal of the Royal Statistical Society. Series B, 44(2), 226–233. https://www-jstor-org.web.bisu.edu.cn/stable/2345828.

30.

Luce

R. D.

(2005). Individual choice behavior. Dover. (Original work published in 1959.)

31.

Maydeu-Olivares

(1999). Thurstonian modeling of ranking data via mean and covariance structure analysis. Psychometrika, 64(3), 325–340. https://doi.org/10.1007/BF02294299

32.

Maydeu-Olivares

Brown

(2010). Item response modeling of paired comparison and ranking data. Multivariate Behavioral Research, 45(6), 935–974. https://doi.org/10.1080/00273171.2010.531231

33.

Maydeu-Olivares

Joe

(2014). Assessing Approximate Fit in Categorical Data Analysis. Multivariate Behavioral Research, 49(4), 305–328. https://doi.org/10.1080/00273171.2014.911075

34.

McDonald

R. P.

(1997). Normal-Ogive multidimensional model. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory (pp. 258–270). Springer.

35.

Meng

X.-L.

Schilling

(1996). Fitting full-information item factor models and an empirical investigation of bridge sampling. Journal of the American Statistical Association, 91(435), 1254–1267. https://doi.org/10.1080/01621459.1996.10476995

36.

Morokoff

W. J.

Caflisch

R. E.

(1995). Quasi-Monte Carlo integration. Journal of Computational Physics, 122(2), 218–230. https://doi.org/10.1006/jcph.1995.1209

37.

Morillo

Abad

F. J.

Kreitchmann

R. S.

Leenen

Hontangas

Ponsoda

(2019). The journey from Likert to forced-choice questionnaires: Evidence of the invariance of item parameters. Journal of Work and Organizational Psychology, 35(2), 75–83. https://doi.org/10.5093/jwop2019a11

38.

Morillo

Leenen

Abad

F. J.

Hontangas

De la Torre

Ponsoda

(2016). A dominance variant under the multi-unidimensional pairwise-preference framework: Model formulation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 40(7), 500–516. https://doi.org/10.1177/0146621616662226

39.

R Core Team (2021). R: A language and environment for statistical computing (Version 4.1.2). R Foundation for Statistical Computing. https://www.R-project.org/.

40.

Roberts

J. S.

Donoghue

J. R.

Laughlin

J. E.

(2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24(1), 3–32. https://doi.org/10.1177/01466216000241001

41.

Salgado

J. F.

Anderson

Tauriz

(2015). The validity of ipsative and quasi-ipsative forced-choice personality inventories for different occupational groups: A comprehensive meta-analysis. Journal of Occupational and Organizational Psychology, 88(4), 797–834. https://doi.org/10.1111/joop.12098

42.

Schilling

Bock

R. D.

(2005). High-dimensional maximum marginal likelihood item factor analysis by adaptive quadrature. Psychometrika, 70(3), 533–555. https://doi.org/10.1007/s11336-003-1141-x

43.

Stark

Chernyshenko

O. S.

Drasgow

(2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise preference model. Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988

44.

Thurstone

L. L.

(1927). A law of comparative judgment. Psychological Review, 34(4), 273–286. https://doi.org/10.1037/h0070288

45.

von Davier

(2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287–307. https://doi.org/10.1348/000711007X193957

46.

Wang

W.-C.

Qiu

X.-L.

Chen

C.-W.

Jin

K.-Y

(2017). Item response theory models for ipsative tests with multidimensional pairwise-comparison items. Applied Psychological Measurement, 41(8), 600–613. https://doi.org/10.1177/0146621617703183

47.

Yuan

K.-H.

Cheng

Patton

(2014). Information matrices and standard errors for MLEs of item parameters in IRT. Psychometrika, 79(1), 232–254. https://doi.org/10.1007/S11336-013-9334-4

48.

Zhang

Angrave

Zhang

Sun

Tay

(2023). The generalized Thurstonian unfolding model (GTUM): Advancing the modeling of forced-choice data. Organizational Research Methods. Advance online publication. https://doi.org/10.1177/10944281231210481

49.

Zhang

Chen

Liu

(2020). An improved stochastic EM algorithm for large-scale full-information item factor analysis. British Journal of Mathematical and Statistical Psychology, 73(1), 44–71. https://doi.org/10.1111/bmsp.12153

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.22 MB

6.23 MB