Comparing Three-class Diagnostic Tests by Three-way ROC Analysis

Abstract

Three-way ROC surfaces are based on a generalization of dichotomous ROC analysis to three-class diagnostic tests. The discriminatory power of three-class diagnostic tests is measured by the volume under the ROC surface. This measure can be given a probabilistic interpretation similar to the equivalence of the c-index to the area under the ROC curve. This article presents a method to calculate nonparametric estimates of the variance of the volume under the surface using Mann-Whitney U statistics. As a simple extension of this result, it is possible to calculate covariance estimates for the volume under the surface. This allows the statistical comparison of two tests used for diagnostic tasks with three possible outcomes. The formulas derived are validated on synthetic data and applied to a three-class data set of pigmented skin lesions. It is shown that a neural network algorithm trained on clinical data and lesion features performs better than one trained on only the lesion features. Key words: Receiver operating characteristic curves; trichotomous ROC analysis. (Med Decis Making 2000; 20:323-331)

Get full access to this article

View all access options for this article.

References

Mossman D

Three-way ROCs

Med Decis Making 1999,19 78-89

Swets JA , Pickett RM Evaluation of Diagnostic Systems Methods from Signal Detection Theory New York: Academic Press, 1982.

Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol. 1975,12:387-415.

Hoeffding W.

A class of statistics with asymptotically normal distribution

Annals of Mathematical Statistics 1948;19:293-325.

Lehmann EL Consistency and unbiasedness of certain nonparametric tests. Annals of Mathematical Statistics. 1951,22: 165-79

Metz CE , Kronman HB

Statistical significance tests for bi-normal ROC curves

J Math Psychol 1980;22 218-43

Hanley JA , McNeil BJ

The meaning and use of the area under the receiver operating characteristic (ROC) curve

Radiology 1982,143 29-36

Hanley JA , Hajian-Tilaki KO Sampling variability of nonparametric estimates of the areas under receiver operating characteristic curves. an update. Acad Radiol 1997,4.49-58

Hanley JA , McNeil BJ A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148.839-43

10.

DeLong ER , DeLong DM , Clarke-Pearson DL

Comparing the areas under two or more correlated receiver operating characteristic curves a nonparametric approach

Biometrics 1988,44 837-45

11.

Swaving M , VanHouwelingen H , Ottes FP , Seerneman T

Statistical comparison of ROC curves from multiple readers

Med Decis Making 1996:16:143-52

12.

Metz CE , Herman BA , Roe CA

Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets

Med Decis Making 1998;18:110-21

13.

Randles RH , Wolfe DA. Introduction to the Theory of Nonparametric Statistics Malabar, FL. Krieger, 1979

14.

Pehamberger H , Binder M , Steiner A , Wolff K.

In vivo epiluminescence microscopy improvement of early diagnosis of melanoma

J Invest Dermatol 1993,100 356-62

15.

Binder M , Schwarz M , Winkler A , et al Epiluminescence microscopy A useful tool for the diagnosis of pigmented skin lesions for formally trained dermatologists Arch Dermatol. 1995;131 286-91

16.

Neal RM Bayesian Learning for Neural Networks New York Springer , 1996

17.

Lehmann EL. Nonparametrics. Statistical Methods Based on Ranks New York. McGraw-Hill, 1975.