A minimal net reclassification improvement to assess predictions of intensive care mortality

Abstract

Objective

In assessing the improved discrimination of a new prognostic score, the “Net Reclassification Improvement” from reclassification methods appears of interest. We propose a measure that takes into account improvements in predicted probabilities to assess and allows testing the additional predictive ability of a new scoring system Y compared to a reference score X.

Study design and settings

To assess and test the improvement in mortality prediction of (X + Y) compared to X, we defined a minimal net reclassification improvement that restricted improvements in predicted probabilities according to some positive threshold δ. Both absolute and relative improvements were considered. A simulation study was performed to assess its performances in a range of practical situations. We then applied our measures to real intensive care unit data.

Results

Expectedly, minimal net reclassification improvement increased with the effect size of Y and decreased with the value of δ. Using relative improvements allowed erasing the influence of the population mortality. For given effect sizes of X and Y, the difference in all measures of reclassification decreased when a correlation between X and Y was introduced.

Conclusion

Reclassification methods, particularly the minimal net reclassification improvement, seem to be clinically relevant when used with continuous clinical data with no known threshold.

Keywords

Reclassification methods mortality prediction severity scores

Get full access to this article

View all access options for this article.

References

Hanley

. Receiver operating characteristic (ROC) methodology: the state of the art. Crit Rev Diagn Imaging 1989; 29(3): 307–335.

Harrell

Jr FE

Shih

. Using full probability models to compute probabilities of actual interest to decision makers. Int J Technol Assess Health Care 2001; 17(1): 17–26.

Cook

. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007; 115(7): 928–935.

Janes

Pepe

. Adjusting for covariates in studies of diagnostic, screening, or prognostic markers: an old concept in a new setting. Am J Epidemiol 2008; 168(1): 89–97.

Vickers

. Prediction models in urology: are they any good, and how would we know anyway? Eur Urol 2010; 57(4): 571–573; discussion 4.

Hosmer

Lemeshow

. Applied logistic regression, 2nd ed. New York: Wiley, 2000.

Cook

. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008; 54(1): 17–23.

Pencina

D’Agostino

Sr RB

D’Agostino

Jr RB

. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27(2): 157–172. discussion 207–212.

Steyerberg

Vickers

Cook

. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21(1): 128–138.

10.

Cook

Paynter

. Performance of reclassification statistics in comparing risk prediction models. Biom J 2011; 53(2): 237–258.

11.

Pencina

D’Agostino

Sr Steyerberg

. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 2011; 30(1): 11–21.

12.

Pencina

. Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvements for normal variables and nested models. Stat Med 2012; 31: 101–113.

13.

Tzoulaki

Liberopoulos

Ioannidis

. Use of reclassification for assessment of improved prediction: an empirical evaluation. Int J Epidemiol 2011; 40(4): 1094–1105.

14.

Azoulay

Mokart

Lambert

. Diagnostic strategy for hematology and oncology patients with acute respiratory failure: randomized controlled trial. Am J Respir Crit Care Med 2010; 182(8): 1038–1046.

15.

DeLong

Clarke-Pearson

. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837–845.

16.

Le Gall

Lemeshow

Saulnier

. A new simplified acute physiologic score (SAPS II) based on an European/North American multicenter study. JAMA 1993; 270: 2957–2963.

17.

Le Gall

Neumann

Hemery

. Mortality prediction using SAPS II: an update for French intensive care units. Crit Care 2005; 9(6): R645–R652.

18.

Metnitz

Moreno

Almeida

. SAPS 3–From evaluation of the patient to evaluation of the intensive care unit. Part 1: Objectives, methods and cohort description. Intensive Care Med 2005; 31(10): 1336–1344.

19.

Moreno

Metnitz

Almeida

. SAPS 3–From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med 2005; 31(10): 1345–1355.

20.

van Buuren

Boshuizen

Knook

. Multiple imputation of missing blood pressure covariates in survival analysis. Stat Med 1999; 18(6): 681–694.

21.

Noda

Iso

Yamashita

. Risk stratification based on metabolic syndrome as well as non- metabolic risk factors in the assessment of carotid atherosclerosis. J Atheroscler Thromb 2011; 18(6): 504–512.

22.

Pencina

D’Agostino

Sr RB

Steyerberg

. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 2010; 30(1): 11–21.

23.

Schuetz

Suter-Widmer

Chaudri

. nostic value of procalcitonin in community-acquired pneumonia. Eur Respir J 2011; 37(2): 384–392.

24.

Calfee

Ware

Glidden

. Use of risk reclassification with multiple biomarkers improves mortality prediction in acute lung injury. Crit Care Med 2011; 39(4): 711–717.

25.

Greenland

. The need for reorientation toward cost-effective prediction: comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al. Stat Med 2008; 27(2): 199–206.

26.

Pepe

. Problems with risk reclassification methods for evaluating prediction models. Am J Epidemiol 2011; 173(11): 1327–1335.

27.

Vickers

Cronin

. Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: towards a decision analytic framework. Semin Oncol 2010; 37(1): 31–38.

28.

Steyerberg

Pencina

Lingsma

. Assessing the incremental value of diagnostic and prognostic markers: a review and illustration. Eur J Clin Invest 2012; 42(2): 216–228.