A measure of association for ordered categorical data in population-based studies

Abstract

Ordinal classification scales are commonly used to define a patient’s disease status in screening and diagnostic tests such as mammography. Challenges arise in agreement studies when evaluating the association between many raters’ classifications of patients’ disease or health status when an ordered categorical scale is used. In this paper, we describe a population-based approach and chance-corrected measure of association to evaluate the strength of relationship between multiple raters’ ordinal classifications where any number of raters can be accommodated. In contrast to Shrout and Fleiss’ intraclass correlation coefficient, the proposed measure of association is invariant with respect to changes in disease prevalence. We demonstrate how unique characteristics of individual raters can be explored using random effects. Simulation studies are conducted to demonstrate the properties of the proposed method under varying assumptions. The methods are applied to two large-scale agreement studies of breast cancer screening and prostate cancer severity.

Keywords

Agreement association crossed random effects generalized linear mixed model ordinal classifications weighted kappa

Get full access to this article

View all access options for this article.

References

Kallman

Wigley

Scott

et al.

New radiographic grading scales for osteoarthritis of the hand. Reliability for determining prevalence and progression. Arthritis Rheum 1989; 32: 1584–1591.

Faust

Gonin

Chuang

et al.

Reliability testing of the dermatology index of disease severity (DIDS) - An index for staging the severity of cutaneous inflammatory disease. Arch Dermatol 1997; 133: 1443–1448.

Gonin

Lipsitz

Fitzmaurice

et al.

Regression modelling of weighted kappa by using generalized estimating equations. J Roy Stat Soc Ser C 2000; 49: 1–18.

Brumback

Jones

. Interobserver agreement in the classification of open fractures of the tibia - the results of a survey of 245 orthopedic surgeons. J Bone Joint Surg 1994; 76A: 1162–1166.

Sickles

D’Orsi

Bassett

. ACR BI-RADS mammography. ACR BI-RADS atlas, breast imaging reporting and data system, Reston, VA: American College of Radiology, 2013.

Gleason

The Veteran’s Administration Cooperative Urologic Research Group: Histologic grading and clinical staging of prostatic carcinoma. In: Tannnbaum

(ed). Urologic pathology: The prostate 1977, pp. 171–198.

Epstein

Allsbrook

Amin

et al.

The 2005 International Society of Urological Pathology (ISUP) consensus conference on gleason grading of prostatic carcinoma. Am J Surg Pathol 2005; 29: 1228–1242.

Holmquist

McMahan

Williams

. Variability in classification of carcinoma in situ of the uterine cervix. Arch Pathol 1967; 84: 334–345.

Beam

Conant

Sickles

. Association of volume and volume-independent factors with accuracy in screening mammogram interpretation. J Natnl Cancer Inst 2003; 95: 282–290.

10.

Elmore

Jackson

Abraham

et al.

Variability in interpretive performance at screening mammography and radiologists’ characteristics associated with accuracy. Radiology 2009; 253: 641–651.

11.

Onega

Smith

Miglioretti

et al.

Radiologist agreement for mammographic recall by case difficulty and finding type. J Am Coll Radiol 2012; 9: 788–794.

12.

Miglioretti

Smith-Bindman

Abraham

et al.

Radiologist characteristics associated with interpretive performance of diagnostic mammography. J Ntnl Cancer Inst 2007; 99: 1854–1863.

13.

Frandsen

Andersen

Madsen

et al.

Garden classification of femoral-neck fractures - an assessment of inter-observer variation. J Bone Joint Surg 1988; 70: 588–590.

14.

Bloch

Kraemer

. 2x2 kappa-coefficients - measures of agreement or association. Biometrics 1989; 45: 269–287.

15.

Kraemer

. Measurement of reliability for categorical data in medical research. Stat Meth Med Res 1992; 1: 183–199.

16.

Graham

Jackson

. The analysis of ordinal agreement data - beyond weighted kappa. J Clin Epidemiol 1993; 46: 1055–1062.

17.

Svensson

Holm

. Separation of systematic and random differences in ordinal rating-scales. Stat Med 1994; 13: 2437–2453.

18.

Ibrahim

Molenberghs

. Missing data methods in longitudinal studies: A review. Test 2009; 18: 1–43.

19.

Cohen

. Weighted kappa - nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 1968; 70: 213–220.

20.

Svensson

. Different ranking approaches defining association and agreement measures of paired ordinal data. Stat Med 2012; 31: 3104–3117.

21.

Coull

Agresti

. Generalized log-linear models with random effects, with application to smoothing contingency tables. Stat Model 2003; 3: 251–271.

22.

Uebersax

Grove

. A latent trait finite mixture model for the analysis of rating agreement. Biometrics 1993; 49: 823–835.

23.

Tanner

Young

. Modeling agreement among raters. J Am Stat Assoc 1985; 80: 175–180.

24.

Agresti

. A model for agreement between ratings on an ordinal scale. Biometrics 1988; 44: 539–548.

25.

Becker

Agresti

. Log-linear modeling of pairwise interobserver agreement on a categorical scale. Stat Med 1992; 11: 101–114.

26.

Cohen

. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960; 20: 37–46.

27.

Nelson

Pepe

. Statistical description of interrater variability in ordinal ratings. Stat Meth Med Res 2000; 9: 475–496.

28.

Shrout

Fleiss

. Intraclass correlations - uses in assessing rater reliability. Psychol Bull 1979; 86: 420–428.

29.

Berry

Johnston

Mielke

. Weighted kappa for multiple raters. Percept Motor Skills 2008; 107: 837–848.

30.

Mielke

Willett

. Unweighted and weighted kappa as measures of agreement for multiple judges. Int J Manage 2009; 26: 213–223.

31.

Maclure

Willett

. Misinterpretation and misuse of the kappa-statistic. Am J Epidemiol 1987; 126: 161–169.

32.

Williamson

Lipsitz

Manatunga

. Modeling kappa for measuring dependent categorical agreement data. Biostatistics 2000; 1: 191–202.

33.

Johnson

. On Bayesian analysis of multirater ordinal data: An application to automated essay grading. J Am Stat Assoc 1996; 91: 42–51.

34.

Johnson

Albert

. Ordinal data modeling (Statistics for Social Science and Public Policy), New York: Springer, 1999.

35.

Hsiao

Chen

Kao

. Bayesian random effects for interrater and test-retest reliability with nested clinical observations. J Clin Epidemiol 2011; 64: 808–814.

36.

R Core Team (2015). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/.

37.

Hedeker

Gibbons

. A random-effects ordinal regression-model for multilevel analysis. Biometrics 1994; 50: 933–944.

38.

Liu

Agresti

. The analysis of ordered categorical data: An overview and a survey of recent developments. Test 2005; 14: 1–30.

39.

Breslow

Clayton

. Approximate inference in generalized linear mixed models. J Am Stat Assoc 1993; 88: 9–25.

40.

Agresti

. Analysis of ordinal categorical data, 2nd ed. New York: Wiley, 2010.

41.

Nelson

Edwards

. On population-based measures of agreement for binary classifications. Can J Stat 2008; 36: 411–426.

42.

Nelson

Edwards

. Measures of agreement between many raters for ordinal classification. Stat Med 2015; 34: 3116–3132.

43.

Shun

McCullagh

. Laplace approximation of high-dimensional integrals. J Roy Stat Soc Ser B 1995; 57: 749–760.

44.

Gueorguieva

. Multivariate generalized linear mixed model for joint modeling of clustered outcomes in the exponential family. Stat Model 2001; 1: 177–193.

45.

Capanu

Gonen

Begg

. An assessment of estimation methods for generalized linear mixed models with binary outcomes. Stat Med 2013; 32: 4550–4566.

46.

Fleiss

Cohen

. Equivalence of weighted kappa and intraclass correlation coefficient as measures of reliability. Educ Psychol Meas 1973; 33: 613–619.

47.

Fleiss

. Measuring nominal scale agreement among many raters. Psychol Bull 1971; 76: 378–382.

48.

Light

. Measures of response agreement for qualitative data - some generalizations and alternatives. Psychol Bull 1971; 76: 365–377.

49.

Conger

. Integration and generalization of kappas for multiple raters. Psychol Bull 1980; 88: 322–328.

50.

Landis

Koch

. Measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–174.

51.

Allsbrook

Mangold

Johnson

et al.

Interobserver reproducibility of Gleason grading of prostatic carcinoma: General pathologist. Human Pathol 2001; 32: 81–88.

52.

Allsbrook

Mangold

Johnson

et al.

Interobserver reproducibility of Gleason grading of prostatic carcinoma: urologic pathologists. Human Pathol 2001; 32: 74–80.

53.

Tagliafico

Tosto

et al.

Mammographic density estimation: Comparison among BI-RADS categories, a semi-automated software and a fully automated one. Breast 2009; 18: 35–40.

54.

Banerjee

. Beyond kappa: A review of interrater agreement measures. Can J Stat 1999; 27: 3–23.