Increase of Uncertainty in Summed-Score-Based Scoring in Non-Rasch IRT

Abstract

Summed-score (SS)-based scoring in non-Rasch IRT allows for pencil-and-paper administration and is used in the Patient-Reported Outcomes Measurement Information System (PROMIS) alongside response-pattern-based scoring. However, this convenience comes with an increase in uncertainty (the increase) associated with SS scoring. The increase can be quantified through the relationship between Bayesian SS and RP scoring. Given an SS of s, the SS posterior is a weighted sum of RP posteriors, with weights representing the marginal probabilities of RPs. From this mixture, the SS score (SS posterior mean) is a weighted sum of RP posterior means, and its uncertainty (variance of the SS posterior) is decomposed into the uncertainty of RP scoring (the weighted sum of RP posterior variances) and the increase (variance of RP posterior means). Without quantifying the increase, PROMIS recommends RP scoring for greater accuracy, suggesting SS scoring as a second option. Using variance decomposition, we quantified the increases for two short forms (SFs). In one, the increase is very small, making SS scoring as accurate as RP scoring, while in the other, the increase is large, indicating SS scoring may not be a viable second option. The increase varies widely, influencing scoring decisions, and should be reported for each SF when SS scoring is used.

Keywords

summed score item response theory patient-reported outcomes

Get full access to this article

View all access options for this article.

References

Ayala

R. J. D.

(2009). The theory and practice of item response theory. The Guilford Press.

Cai

(2015). Lord–Wingersky algorithm version 2.0 for hierarchical item factor models with applications in test scoring, scale alignment, and model fit testing. Psychometrika, 80(2), 535–559. https://doi.org/10.1007/s11336-014-9411-3

Cella

Riley

Stone

Rothrock

Reeve

Yount

Amtmann

Bode

Buysse

Choi

Cook

Devellis

DeWalt

Fries

J. F.

Gershon

Hahn

E. A.

Lai

J. S.

Pilkonis

Revicki

PROMIS Cooperative Group . (2010). The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194. https://doi.org/10.1016/j.jclinepi.2010.04.011

Cella

Yount

Rothrock

Gershon

Cook

Reeve

Ader

Fries

J. F.

Bruce

Rose

PROMIS Cooperative Group . (2007). The patient-reported outcomes measurement information system (PROMIS): Progress of an NIH roadmap cooperative group during its first two years. Medical Care, 45(5 Suppl 1), S3–S11. https://doi.org/10.1097/01.mlr.0000258615.42478.55

Chalmers

R. P.

(2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06

Forrest

C. B.

Devine

Bevans

K. B.

Becker

B. D.

Carle

A. C.

Teneralli

Moon

J. H.

Tucker

C. A.

Ravens-Sieberer

(2018). Development and psychometric evaluation of the PROMIS pediatric life satisfaction item banks, child-report and parent-proxy editions. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 27(1), 217–234. https://doi.org/10.1007/s11136-017-1681-7

Frühwirth-Schnatter

(2006). Finite mixture and markov switching models. Springer.

HealthMeasures . (2024). Northwetern university. https://www.healthmeasures.net/index.php

Lord

F. M.

Wingersky

M. S.

(1984). “Comparison of IRT true-score and equipercentile observed-score” equatings. Applied Psychological Measurement, 8(4), 453–461. https://doi.org/10.1177/014662168400800409

10.

Pilkonis

P. A.

Choi

S. W.

Reise

S. P.

Stover

A. M.

Riley

W. T.

Cella

PROMIS Cooperative Group . (2011). Item banks for measuring emotional distress from the patient-reported outcomes measurement information system (PROMIS): Depression, anxiety, and anger. Assessment, 18(3), 263–283. https://doi.org/10.1177/1073191111411667

11.

R Core Team . (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

12.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(S1), 1-97.

13.

Segawa

Benjamin

Cella

(2020). A comparison of computer adaptive tests (CATs) and short forms in terms of accuracy and number of items administrated using PROMIS profile. Quality of Life Research, 29(1), 213–221. https://doi.org/10.1007/s11136-019-02312-8

14.

Steinberg

Thissen

(2013). Item response theory. In Comer

J. S.

Kendall

P. C.

(Eds.), The oxford handbook of research strategies for clinical psychology. Oxford University Press.

15.

Thissen

Maria

(2001). Item response theory for items scored in two categories. In Test scoring (pp. 85–152). Routledge.

16.

Thissen

Nelson

Rosa

McLeod

L. D.

(2001). Item response theory for items scored in more than two categories. In Test scoring (pp. 153–198). Routledge.

17.

Thissen

Pommerich

Billeaud

Williams

V. S. L.

(1995). Item response theory for scores on tests including polytomous items with ordered responses. Applied Psychological Measurement, 19(1), 39–49. https://doi.org/10.1177/014662169501900105

18.

Wickham

Averick

Bryan

Chang

McGowan

L. D. ’A.

Francois

Garrett

Hayes

Henry

Hester

Kuhn

Pedersen

Miller

Bache

Müller

Ooms

Robinson

Seidel

Spinu

Yutani

(2019). Welcome to the tidyverse. Journal of Open Source Software, 4(43), 1686. https://doi.org/10.21105/joss.01686

19.

Zhu

(2024). Create Awesome LaTeX Table with knitr:: kable and kableExtra.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.22 MB