Detecting Guessing Effects for the Nominal Categories Model Using a Conditional Likelihood Ratio Test With Data-Dependent Degrees of Freedom

Abstract

Item response theory models have been extended by including guessing parameters to represent the response tendencies of individuals with low trait levels. Particular cases of these models arise when the guessing parameters are fixed at boundary values. For example, the two-parameter logistic model is derived from the three-parameter logistic model by fixing the guessing parameter to zero. However, the likelihood ratio test statistic for nested pairs of models asymptotically fails to follow a chi-square distribution when some parameters take boundary values, resulting in an unreliable p-value. This article proposes two solutions to this problem: a chi-square distribution with estimated degrees of freedom and a score test to evaluate fit at the item level. Guessing parameters are incorporated into the nominal categories model, and models with and without guessing are compared using the proposed chi-square statistic. The special case of the comparison between the three-parameter logistic and two-parameter logistic models is also considered. A simulation study was conducted to evaluate the performance of the proposed chi-square statistic with dichotomous and polytomous data. The results indicate that it is a viable alternative for detecting significant guessing effects. The solution is further illustrated with an empirical example to demonstrate its practical application.

Keywords

conditional likelihood ratio test nominal categories model three parameter logistic model guessing multiple choice items item response theory

Get full access to this article

View all access options for this article.

References

Andrich

(2010). Understanding the response structure and process in the polytomous Rasch model. In Nering

M. L.

Ostini

(Eds.), Handbook of polytomous item response theory models (pp. 123–152). Routledge.

Baker

F. M.

Kim

S. H.

(2004). Item response theory: Parameter estimation techniques (2nd ed.). Marcel Dekker.

Barton

M. A.

Lord

F. M.

(1981). An upper asymptote for the three-parameter logistic item-response model. RR-81-20. NJ Educational Testing Service.

Bech

(2012). Clinical psychometrics. Wiley Blackwell. https://doi.org/10.1002/9781118511800

Bishop

Y. M.

Fienberg

S. E.

Holland

P. W.

(2007). Discrete multivariate analysis: Theory and applications. Springer.

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29–51. https://doi.org/10.1007/BF02291411

Bock

R. D.

Aitkin

(1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443–459. https://doi.org/10.1007/BF02293801

Brown

Templin

Cohen

(2015). Comparing the two- and three-parameter logistic models via likelihood ratio tests: A commonly misunderstood problem. Applied Psychological Measurement, 39(5), 535–348. https://doi.org/10.1177/0146621614563326

Buse

(1982). The likelihood ratio, Wald, and Lagrange multiplier tests: An expository note. The American Statistician, 36(3), 153–157. https://doi.org/10.2307/2683166

10.

Casella

Berger

R. L.

(2002). Statistical inference (2nd ed.). Brooks/Cole Cengage Learning.

11.

Chalmers

R. P.

(2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06

12.

Culpepper

S. A.

(2016). Revisiting the 4-parameter item response model: Bayesian estimation and application. Psychometrika, 81(4), 1142–1163. https://doi.org/10.1007/s11336-015-9477-6

13.

Drasgow

Levine

M. V.

Mead

A. D.

(1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19(2), 143–165. https://doi.org/10.1177/014662169501900203

14.

Embretson

S. E.

Reise

S. P.

(2000). Item response theory for psychologists. Psychology Press. https://doi.org/10.4324/9781410605269

15.

Glas

C. A. W.

(1999). Modification indices for the 2-PL and the Nominal Response Model. Psychometrika, 64(3), 273–294. https://doi.org/10.1007/BF02294296

16.

Glas

C. A. W.

(2016). Frequentists model-fit tests. In van der Linden

W. J.

(Ed.), Handbook of item response theory: Vol. 2. Statistical tools (pp. 343–361). CRC Press. https://doi.org/10.1201/b19166

17.

Hambleton

R. K.

Swaminathan

(1985). Item response theory: Principles and applications. Springer.

18.

Han

K. T.

(2012). Fixing the c parameter in the three-parameter logistic model. Practical Assessment, Research, and Evaluation, 17(1), 1. https://doi.org/10.7275/f0gz-kc87

19.

Held

Bové

D. S.

(2014). Applied statistical inference. Springer.

20.

Hutchinson

T. P.

(1991). Ability, partial information and guessing: Statistical models applied to multiple-choice tests. Rumsby Scientific Publishing.

21.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates.

22.

Luce

R. D.

(1959). Individual choice behavior. Wiley.

23.

McFadden

(1974). Conditional logit analysis of qualitative choice behavior. In Zarembka

(Ed.), Advances in econometrics (pp. 105–142). Academic Press.

24.

McFadden

D. L.

(1984). Econometric analysis of qualitative response models. In Griliches

Intriligator

M. D.

(Eds.), Handbook of econometrics (Vol. 2, pp. 1395–1457). Elsevier.

25.

Mellenbergh

G. J.

(2011). A conceptual introduction to psychometrics: Development, analysis, and application of psychological and educational tests. Eleven International Publishing.

26.

Mislevy

R. J.

(1986). Bayes modal estimation in item response models. Psychometrika, 51(2), 177–195. https://doi.org/10.1007/BF02293979

27.

Monroe

(2019). Estimation of expected Fisher information for IRT models. Journal of Educational and Behavioral Statistics, 44(4), 431–447. https://doi.org/10.3102/1076998619838240

28.

Paek

Lin

Chalmers

R. P.

(2022). Investigating confidence intervals of item parameters when some item parameters take priors in the 2PL and 3PL models. Educational and Psychological Measurement, 83(2), 375–400. https://doi.org/10.1177/00131644221096431

29.

R Core Team. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/

30.

Revuelta

(2010). Estimating difficulty from polytomous categorical data. Psychometrika, 75(2), 331–350. https://doi.org/10.1007/s11336-009-9145-9

31.

Revuelta

(2005). An item response model for nominal data based on the rising selection ratios criterion. Psychometrika, 70(2), 305–324. https://doi.org/10.1007/s11336-002-0975-y

32.

Rizopoulos

(2006). ltm: An R package for latent variable modelling and item response theory analyses. Journal of Statistical Software, 17(5), 1–25. https://doi.org/10.18637/jss.v017.i05

33.

San Martín

del Pino

de Boeck

(2006). IRT models for ability-based guessing. Applied Psychological Measurement, 30(3), 183–203. https://doi.org/10.1177/0146621605282773

34.

Suh

Bolt

D. M.

(2010). Nested logit models for multiple-choice item response data. Psychometrika, 75(3), 454–473. https://doi.org/10.1007/s11336-010-9163-7

35.

Susko

(2013). Likelihood ratio test with boundary constraints using data-dependent degrees of freedom. Biometrika, 100(4), 1019–1023. https://doi.org/10.1093/biomet/ast032

36.

Thissen

Cai

Bock

R. D.

(2010). The nominal categories model. In Nering

M. L.

Ostini

(Eds.), Handbook of polytomous item response theory models (pp. 43–75). Routledge.

37.

Thissen

Steinberg

(1984). A response model for multiple choice items. Psychometrika, 49(4), 501–519. https://doi.org/10.1007/BF02302588

38.

Waller

N. G.

Reise

S. P.

(2010). Measuring psychopathology with nonstandard item response theory models: Fitting the four-parameter model to the Minnesota Multiphasic Personality Inventory. In Embretson

S. E.

(Ed.), Measuring psychological constructs: Advances in model-based approaches (pp. 147–173). American Psychological Association. https://doi.org/10.1037/12074-007

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB