A Comprehensive Multimorbidity Index for Predicting Mortality in Intensive Care Unit Patients

Abstract

Background:

Accurate prediction of mortality for patients admitted to the intensive care units (ICUs) is an important component of medical care. However, little is known about the role of multimorbidity in predicting end of life for high-risk and vulnerable patients.

Objective:

The aim of the study was to derive and validate a multimorbidity risk model in an attempt to predict all-cause mortality at 6 and 12 months posthospital discharge.

Methods:

This is a retrospective, observational, clinical cohort study. Data were collected on 442,692 ICU patients who received care through the Veterans Administration between January 2003 and December 2013. The primary outcome was all-cause mortality at 6 and 12 months posthospital discharge. We divided the data into derivation (80%) and validation (20%) sets. Using multivariable logistic regression models, we compared prognostic models based on age, principal diagnosis groups, physiological markers, immunosuppressants, comorbidity categories, and a newly developed multimorbidity index (MMI) based on 5695 comorbidities. The cross-validated area under the receiver operating characteristic curve (AUC) was used to report the accuracy of predicting all-cause mortality at 6 and 12 months of hospital discharge.

Results:

The average age of patients was 68.87 years (standard deviation = 12.1), 95.9% were males, 44.9% were widowed, divorced, or separated. The relative order of accuracy in predicting mortality was the MMI (AUC = 0.84, CI = 0.83–0.84), VA Inpatient Evaluation Center index (AUC = 0.80, CI = 0.79–0.81), principal diagnosis groups (AUC = 0.74, CI = 0.73–0.74), comorbidities (AUC = 0.69, CI = 0.68–0.70), physiological markers (AUC = 0.65, CI = 0.64–0.65), age (AUC = 0.60, CI = 0.60–0.61),and immunosuppressant use (AUC = 0.59, CI = 0.58–0.59).

Conclusions:

The MMI improved the accuracy of predicting short- and long-term all-cause mortality for ICU patients. Further prospective studies are needed to validate the index in different clinical settings and test generalizability of results in patients outside the VA system of care.

Introduction

Accurate prediction of prognosis for patients admitted to the intensive care units (ICUs) is important for the clinical management of these patients, assessing quality of care, anticipating the likelihood of readmission, and identifying patients who are eligible to benefit from hospice and palliative care services.^1,2

Currently, large number of prognostic indicators are available. Some, such as indices developed by Mazzaglia et al.,³ Gagne et al.,⁴ Carey et al.,^5,6 Lee et al.,⁷ Schonberg et al.,⁸ Porock et al.,^9,10 Mitchell et al.,¹¹ Flacker and Kiely,¹² Levine et al.,¹³ Walter et al.,¹⁴ Rozzini et al.,¹⁵ Di Bari et al.,¹⁶ Inouye et al.,¹⁷ Pilotto et al.,^18,19 Teno et al.,²⁰ Dramé et al.,²¹ and Fischer et al.,²² rely on physiological markers. Others such as Charlson et al.,²³ Deyo et al.,²⁴ Romano et al.,²⁵ Roos et al.,²⁶ D'Hoores et al.,²⁷ Elixhauser et al.,²⁸ and van Walraven et al.²⁹ rely on diagnoses.

Most existing ICU models have focused on a combination of physiological and diagnostic categories to predict prognosis.^30–38 A few available models highlighted the role of select comorbidities, in particular organ failure, in improving the accuracy of predicting in-hospital and postdischarge mortality.^36,39–43 However, these existing models are mostly focused on hemodynamic instability⁴⁴ and a selection of few comorbidities.^45–48 The effects of other comorbidities have been overlooked.⁴⁹ This study compares these selective approaches with a comprehensive approach to measuring comorbidities, named the multimorbidity index (MMI). In contrast to indices that are based on select physiological markers or select comorbidities, the MMI is a comprehensive approach that focuses on all patient diagnoses. The existing models use 30–160 diagnosis categories, while MMI uses thousands of individual diagnoses. This article is an extension to our previously published work,^50–52 which evaluated the performance of the MMI in predicting mortality for heart failure and nursing home patients.

Methods

Source of data

This was a retrospective data analysis of 442,692 ICU patients admitted to 87 medical ICUs throughout the Veterans Affairs Medical Centers (VAMCs). The study included only medical ICU patients and hence surgical ICUs and mixed ICUs (medical and surgical patients) were excluded from the study. Some medical ICU patients may have received surgical interventions such as tracheostomies and insertion of a feeding tube. The study did not include any surgical variables or, for that matter, any treatment variables. Treatment variables were excluded so that the index can be used to evaluate the effectiveness of treatment.

The study examined patients admitted from 2003 through 2013. Data were obtained from the VA Informatics and Computing Infrastructure Corporate Data Warehouse.⁵³ For patients with multiple admissions, one admission was selected randomly to avoid correlated data issues.

Dependent variables

The primary outcomes were survival at 6-month postdischarge mortality and 12-month postdischarge mortality. The secondary outcome was in-hospital survival. Because our findings at these time intervals are not different, this article reports in detail the 6-month and 12-month data and provides a brief summary of in-hospital survival.

The date of death (DOD) was extracted from the VA Informatics and Computing Infrastructure Corporate Data Warehouse. The limitations for DOD are mentioned,⁵⁴ including coverage and delay.

Independent variables

The potential predictors of mortality were selected based on the VA Inpatient Evaluation Center (IPEC) index.^55,56 The IPEC index includes a large number of independent predictors of mortality selected from the following:

(1) Demographics: Patient demographics such as age, gender, marital status, and race.

(2) Physiological measures: The study relied on 13 physiological measures previously used by both IPEC and Acute Physiology and Chronic Health Evaluation (APACHE) index. These included sodium, blood urea nitrogen, creatinine, glucose, albumin, bilirubin, white blood cell count, hematocrit, PaO₂, PaCO₂, pH, eGFR, and lactic acid. The missing laboratory data were replaced with the mean for all patients. If a patient had multiple values for a test in the time period 24 hours before and 24 hours after admission, the most abnormal test value was used. Version 4 of APACHE uses 116 categories of diagnoses⁵⁷ and these were not examined in the current study.

(3) Medications: The study relied on 13 immunosuppressant medications known to affect mortality rates.⁵⁶ These included prednisone, cyclosporine, cyclophosphamide, adriamycin, methotrexate, cisplatin, etoposide, velban, vincristine, bleomycin, azathioprine, dexamethasone, or hydrocortisone medications.

(4) Elixhauser index: The study examined the predictive accuracy of Elixhauser comorbidity categories.²⁸ The IPEC index also relies on the Elixhauser comorbidity categories. These included the following select categories of diseases: congestive heart failure (CHF), valvular, pulmonary circulation, peripheral vascular, hypertension, paralysis, other neurological, chronic pulmonary, uncomplicated diabetes, complicated diabetes, hypothyroidism, renal failure, liver, peptic ulcer excluding bleeding, AIDS, lymphoma, metastatic cancer, solid tumor without metastasis, rheumatoid arthritis/collagen vascular, coagulopathy, obesity, weight loss, fluid and electrolyte disorders, blood loss anemia, deficiency anemia, alcohol abuse, drug abuse, psychoses, and depression disorders. Patients' comorbidity categories were based on diagnoses during the hospital admission and coded using the International Classification of Disease, version 9. The quality of coding diagnoses within VA medical records has been demonstrated to be reasonably accurate.^58,59

The IPEC index comprises variables borrowed from several other indices. This study compared the predictive ability of the IPEC constituent parts as well as the overall index with that of the MMI.

Multimorbidity index

In addition to the predictors listed above, the study examined an index comprising a comprehensive set of diagnoses, the MMI.^50,51 The purpose of the MMI was to account comprehensively for all patients' diagnoses and comorbidities. Existing indices categorize diagnoses into broad categories that reduce the accuracy of predictions. These broad categories include diagnoses that differed significantly in their association with mortality and therefore are not homogeneous. For example, there are 141 cancers that fit the metastatic cancer category and all are scored in the same manner. Obviously, metastatic cancers are not equally fatal. In addition to the problem with use of nonhomogeneous broad categories, there are also concerns that selective inclusion of diseases could lead to missing important diseases. The MMI separately scored each diagnosis and therefore does not rely on broad disease categories. The MMI expanded the number of diagnoses scored from 17 broad categories in the Charlson index and 30 broad categories in the Elixhauser to 5695 separate diagnoses that typically occur among ICU patients. Instead of scoring broad categories, MMI estimated the impact of each diagnosis separately. The overall score for the MMI was calculated using the following 2 steps:

Step 1: Calculate a likelihood ratio (LR) for each diagnosis ( \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} $${ \rm{L}}{{ \rm{R}}_{{ \rm{Diagnosis}}}}$$ \end{document} ): \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} {\rm {LR}_{\rm {Diagnosis}}} & = { \frac {Prevalence \ of \ diagnosis \ among \ dead \ patients} {Prevalence \ of \ diagnosis \ among \ alive \ patients}} \\ & = { \frac {\rm p(Diagnosis|Dead)} {\rm p(Diagnosis|Alive)}} \end{align*} \end{document}

A diagnosis changes the odds of mortality proportional to its LR. An LR of 2 indicates that the odds of mortality for those with the disease are double that of the entire cohort. An LR of 0.5 indicates that the odds of mortality are reduced by half.

Step 2: MMI assumes that the impact of each diagnosis on mortality is independent. Then, the MM score is calculated as follows: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*} {\rm {MM \ score}} = { \frac { \mathop \prod \nolimits_ { Dx } { \rm { L } } { { \rm { R } } _ { { \rm { Dx } } } } } { 1 + \mathop \prod \nolimits_ { Dx } { \rm { L } } { { \rm { R } } _ { { \rm { Dx } } } } } } \end{align*} \end{document}

The MM score assumes equal priors, which several studies have shown does not matter when a large number of pieces of information are used to make the prediction.⁶⁰ Given the assumption of equal priors, the MM score can be interpreted as the probability of mortality. It ranges from 0 to 1; the higher the score, the more likely the patient will die.

Methods of statistical analyses

Multivariable logistic regression models were developed using age, principal diagnoses (grouped into the 74 diagnosis-related groups based on the VA IPEC guidelines), Elixhauser comorbidity index, MM score, immunosuppressant status, and laboratory variables. Separate mortality models were developed for the in-hospital mortality, 6-month postdischarge mortality, and 12-month postdischarge mortality.

Data were randomly divided into training (80%) and validation (20%) sets and used the training set to create risk prediction models and the validation set to assess model accuracy. Model accuracy was assessed using the C-statistic, also known as area under the receiver operating characteristic (AUC) curves. The pROC function in R was used to calculate 95% confidence interval for the AUC.⁶¹ This function calculated AUC using 2000 stratified bootstrap replicates. Statistical analyses were performed by using R v3.1.2.

Results

Between 2003 and 2013, the number of individual ICU patients at 87 medical ICU VAMCs was 442,692 patients (Table 1). The average age was 68.9 (standard deviation = 12.1) years. The majority of the ICU patients were males (95.9%) and older, with an average age of 60 years (77.5%). The top six comorbidity classifications were hypertension (52.1%), diabetes, uncomplicated (28.1%), fluid and electrolyte disorders (18.9%), CHF (15.8%), anemia (14.9%), and renal failure (12.9%). Table 2 shows sample diagnoses within various body systems and their associated LRs. The diagnoses found to be highly predictive of mortality in 6 months included cardiac arrest (LR 15.2) and anoxic brain damage (LR 13.0).

Table 1.

Characteristics of Patients in Medical Intensive Care Units (n = 442,692)

Characteristic	n (%)
Age (years)
≤40	8505 (1.9)
41–60	91,250 (20.6)
61–80	258,554 (58.4)
>80	84,361(19.1)
Gender
Male	424,472 (95.9)
Female	18,220 (4.1)
Marital status
Married	194,622 (44)
Widowed/divorced/separated	198,609 (44.9)
Never married	48,331 (10.9)
Other marital status	1130 (0.25)
Ethnicity
Hispanic origin	16,798 (3.8)
Not of Hispanic origin	176,042 (39.8)
Other ethnic origins	249,852 (56.4)
Race
American Indian	939 (0.21)
Asian	781 (0.18)
Black	47,587 (10.4)
White	153,496 (34.7)
Other races or race missing/unknown	239,889 (54.2)
Top principal diagnostic group
Angina	36,474 (8.2)
Cardiac dysrhythmias	19,497 (4.4)
Acute myocardial infarction	19,264 (4.3)
Other neuro disease	18,036 (4.0)
Drug overdose	16,629 (3.8)
GI bleed, Ulc/Lac	15,095 (3.4)
Occlusion or stenosis of precerebral arteries	13,181 (3.0)
Congestive heart failure	13,056 (2.9)
Other medical disease	12,675 (2,9)
Peripheral vascular disease	11,889 (2.7)
Other GI disease	11,771 (2.7)
Top immunosuppressants
Hydrocortisone	1406 (26.5)
Prednisone	1121 (12.1)
Cyclosporine	299 (5.6)
Methotrexate	294 (5.5)
Dexamethasone	226 (4.3)

GI, gastrointestinal; Ulc/Lac, ulcer/laceration.

Table 2.

Sample Diagnoses Predictive of Mortality 6 Months Postdischarge from Intensive Care Unit

Likelihood ratio	Circulatory system	Digestive system	Nervous system	Kidney system
1–2	Dissection aorta, Unsp	Dieul lesion of intestine	Late Eff CV-Dysarthria	Bladder neoplasm NOS
2–4	Left heart failure	Bacterial enteritis NEC	Bacterial meningitis NOS	Malig Neo bladder NOS
4–8	Shock NOS	Mal Neo esophagus NEC	Metabolic encephalopathy	Sec Malig Neo Urin NEC
>8	Cardiac arrest	Sec Malig Neo Sm Bowel	Anoxic brain damage	Tumor lysis syndrome

NEC, not elsewhere classified; NOS, not otherwise specified; Unsp, unspecified; mal, malignant; neo, neoplasm.

Figure 1 demonstrates the performance of the MMI at different scores. The MMI ranges from 0 to 1 and was stratified into five risk quintiles: 0.0–0.2, 0.2–0.4, 0.4–0.6, 0.6–0.8, and 0.8–1.0. In Figure 1, the X-axis shows the score from MMI. The Y-axis demonstrates the observed mortality rate. The shaded histogram bars demonstrate the number of patients within each category. The higher scores were associated with increased risk of mortality. Patients in the highest risk group (MMI >0.8) had a higher risk of the 12-month death than patients in lower scores. Figure 1 provides the calibration between MMI and observed mortality rates. The Brier calibration score for this index was 0.21.

FIG. 1.

MMI and 12-month mortality risk (n = 442,692). MMI, multimorbidity index.

The performances of the various indices were evaluated using the validation data set. For the IPEC index, the validation data sets were 0.85, 0.81, and 0.80 for in-hospital, 6-, and 12-month mortality, respectively. For the comprehensive MMI model, the AUC values were 0.92, 0.86, and 0.84 for in-hospital, 6-, and 12-month mortality, respectively (Table 3). The MMI provided better prediction of in-hospital, 6-, and 12-month mortality compared with IPEC index or any components of the IPEC index.

Table 3.

Area Under Receiver Operating Curve for Different Models (Validation Data n = 88,538)

Models	6-month mortality (CIs)	12-month mortality (CIs)
Age only (years)	0.60 (0.59–0.61)	0.60 (0.60–0.61)
74 Principal diagnoses only	0.75 (0.75–0.75)	0.74 (0.73–0.74)
Elixhauser (29 categories of comorbidities only)	0.69 (0.69–0.70)	0.69 (0.68–0.70)
74 Principal and 29 comorbidity diagnosis categories	0.78 (0.78–0.79)	0.77 (0.77–0.78)
13 immunosuppressants only	0.59 (0.58–0.59)	0.59 (0.58–0.59)
APACHE laboratories (13 laboratory values only)	0.65 (0.64–0.65)	0.65 (0.64–0.65)
VA IPEC index^a	0.81 (0.80–0.81)	0.80 (0.79–0.81)
Multimorbidity index based on 5695 diagnoses^b	0.86 (0.85–0.86)	0.84 (0.83–0.84)
All variables	0.87 (0.86–0.87)	0.85 (0.84–0.85)

Includes the above six variables, which together comprise the IPEC.

Multimorbidity index was built from 5695 unique ICD9 diagnoses.

APACHE, Acute Physiology and Chronic Health Evaluation; IPEC, In-Patient Evaluation Center.

The receiver operating curves for 6- and 12-month mortality are shown in Figure 2. The X-axis shows specificity and the Y-axis shows the 1- sensitivity of the predictions. The diagonal line shows the performance of random prediction. The distance of the curved lines from the diagonal line shows the AUC or accuracy of the predictions. The two panels demonstrate the comparison of different models. These include (in order of accuracy) all sources, MMI, IPEC, Elixhauser, APACHE laboratories, medications, and age. The MMI demonstrates better predictive value than Elixhauser, suggesting that categorization of diagnoses may decrease the accuracy of predictions. The MMI is more accurate than any single source of data. Adding the nondiagnostic information to the MMI improved the MMI marginally. Figure 2 compares the performance of MMI with the IPEC index, which includes Elixhauser, APACHE laboratories, and immunosuppressant medications. This suggests that a comprehensive approach to diagnoses may be preferred to a selective combination of various sources of data.

FIG. 2.

Receiver Operating Curves for (a) the 6-month and (b) 12-month mortality (validation data n = 88,538). MMI is based on individual diagnoses. IPEC index includes Elixhauser diagnostic categories, physiological markers, medications, and age. Elixhauser index is based on diagnostic categories. Physiological markers include 13 common laboratory values available in ICU. ICU, intensive care unit. IPEC, in-patient evaluation center.

Discussion

In a large cohort of US veterans with an ICU admission, the inclusion of additional comorbidities improved the accuracy of short- and long-term mortality prediction. The study compared a newly established MMI with the IPEC index and its components. The MMI outperformed the IPEC index and its components, including indices based on Elixhauser categories, physiological markers, or immunosuppressant medications. To the best of our knowledge, this is the first study to report an association between the comprehensive inclusion of comorbidities and mortality prediction for critically ill patients.

This study also found that patients' diagnoses were more predictive than physiological markers or select immunosuppressant medications. Christensen et al. examined the performance of the Charlson comorbidity categories and three physiology-based scores (APACHEs version II, Simplified Acute Physiology score version II and III). The study found patients' diagnosis categories were as predictive as physiological markers.⁶ However, clinicians and investigators are typically critical of relying on patients' diagnoses in administrative data. Many consider these diagnoses as unreliable and insufficient.^62–64 Thus, the finding that patients' diagnoses are the most accurate predictors of mortality would need to be studied in prospective trials. Clinicians evaluate patients' hemodynamic status based on physiological markers^65,66; these markers, patients admitted into ICUs, typically have abnormal markers and clinicians monitor these critical laboratory values and rapidly correct them with appropriate management and treatment. The existing models that include abnormal laboratory values within the first 24 hours may overlook the data on subsequent days of hospitalization or may reflect the managed value, as opposed to the original value of these laboratory tests. Physiological markers may be predictive of in-hospital mortality (see the fit and performance of APACHE, SAPS, or the MPM model). Presumably, for patients who are stable and are discharged alive from the ICU, the day 1 physiology in the ICU may be less predictive. Some prognostic indices distinguish between early physiologic derangement so as to not reward bad medical care. For example, if a patient comes in with shock and is not treated appropriately, his physiological values will deteriorate over time and he would eventually die. One does not want to reward poor care by claiming that the patient condition was severe and death was expected. In the MMI, a similar situation occurs with iatrogenic diagnoses (e.g., patient falling in the hospital). Since this index is used to predict long-term survival, iatrogenic diagnoses are just as predictive of mortality as diagnosis on hospital admission. Nevertheless, it is important to keep in mind that not all diagnoses reflect patient condition as some may be the side effect of poor care.

There are at least three ways to explain the superior performance of MMI compared with other measures in the study. The physiological markers may not be as accurate in predicting mortality over time because many of these measures are corrected as a consequence of good care. MMI is based on comorbid conditions that may not be masked through better management and are not affected by typical variation in physiological variables. It may also be that chronic comorbidities listed in Elixhauser index are not as predictive as MMI, with its focus on both chronic and acute predictors. Finally, MMI uses more variables than all other indices combined; so the differences in cross-validated performance may be due to a comprehensive versus a selective approach. In essence, this latter explanation says that more data lead to better predictions, not a very novel insight, but nevertheless a possible explanation for the performance of MMI. No matter what explains the added accuracy of MMI, the degree of improvement is notable. The improvement in accuracy is not small, but large, and likely to be clinically relevant.

A comprehensive index, such as the MMI, would be difficult to use without relying on electronic health records. Given the widespread availability of the electronic health records, the MMI score can be automated and provided to clinicians. The index and the scoring method are in the public domain and can be incorporated into any healthcare system. A comprehensive approach to score patient-specific diagnoses may improve the clinical utility of these prediction models if such an approach can be programmed and designed as a clinical decision support tool within the electronic health record system. In managing complex patients, it is crucial to expand from focusing on one disease to simultaneous examination of multiple diseases. Multimorbidity is common. Its prevalence increases with age and substantial number of patients under the age of 65 suffer from multiple conditions. Patients with mental illness, coupled with lacking social support, are most likely to suffer from multiple morbidities. Unitizing the MMI as a decision support tool embedded in the electronic medical records may prove to be valuable as it includes diagnoses ranging from medical to mental to social. Furthermore, a comprehensive approach may improve continuity and coordination of care for patients with complex set of diseases.

Limitations

This study had several limitations. The MMI assumed that each diagnosis independently increases the odds of mortality, which clearly is not accurate. Diseases interact and change likelihood of death. For example, the observed effects of hypertension may be partially accounted for by CHF, so in cases where these diagnoses occur together, the LR for hypertension may be overestimated.

Our models also did not include do-not-resuscitate orders. Clearly, these orders affect risk of mortality and should be considered when reporting patients' prognosis.

The study cohort was predominantly male and representative of the VA population; therefore, the results may not be readily generalized to other nonveteran populations where patients may be more likely to be treated for other conditions such as sepsis. The study focused on medical ICU patients only. The results may not be applicable to surgical ICU patients. The assignment of comorbid diagnoses is at best incomplete and is subject to variation in coding practices. Finally, this study chose the predictors based on the existing VA IPEC index. Many important ICU predictors, including number of previous hospitalizations, mean arterial pressure, mechanical ventilation, dialysis, and medications (e.g., vasopressors, antibiotics, and inotropes), were not investigated here. Treatment variables were by design excluded from predictors of prognosis. Further prospective studies are needed to validate the index in different clinical settings and test generalizability of results in patients outside the VA system of care.

Conclusion

In a large cohort of US veterans with an ICU admission, a comprehensive inclusion of comorbidities improved the accuracy of mortality prediction.

Footnotes

Acknowledgment

This study was supported by a contract from the District of Columbia Veteran Affairs Medical Center. The opinions are the authors' and do not reflect those of the U.S. Department of Veterans Affairs.

Author Disclosure Statement

No competing financial interests exist.

References

Campbell

, et al.: Predicting death and readmission after intensive care discharge. Br J Anaesth, 2008; 100:656–662.

Power

, Harrison

: Why try to predict ICU outcomes?. Curr Opin Crit Care, 2014; 20:544–549.

Mazzaglia

, et al.: Screening of older community-dwelling people at risk for death and hospitalization: The Assistenza Socio-Sanitaria in Italia project. J Am Geriatr Soc, 2007; 55:1955–1960.

Gagne

, et al.: A combined comorbidity score predicted mortality in elderly patients better than existing scores. J Clin Epidemiol, 2011; 64:749–759.

Carey

, et al.: Prediction of mortality in community-living frail elderly people with long-term care needs. J Am Geriatr Soc, 2008; 56:68–75.

Carey

, et al.: Development and validation of a functional morbidity index to predict mortality in community-dwelling elders. J Gen Intern Med, 2004; 19:1027–1033.

Lee

, et al.: Development and validation of a prognostic index for 4-year mortality in older adults. JAMA, 2006; 295:801–808.

Schonberg

, et al.: Index to predict 5-year mortality of community-dwelling adults aged 65 and older using data from the National Health Interview Survey. J Gen Intern Med, 2009; 24:1115–1122.

Porock

, et al.: Predicting death in the nursing home: Development and validation of the 6-month Minimum Data Set mortality risk index. J Gerontol A Biol Sci Med Sci, 2005; 60:491–498.

10.

Porock

, et al.: The MDS Mortality Risk Index: The evolution of a method for predicting 6-month mortality in nursing home residents. BMC Res Notes, 2010; 3:200.

11.

Mitchell

, et al.: Prediction of 6-month survival of nursing home residents with advanced dementia using ADEPT vs hospice eligibility guidelines. JAMA, 2010; 304:1929–1935.

12.

Flacker

, Kiely

: Mortality-related factors and 1-year survival in nursing home residents. J Am Geriatr Soc, 2003; 51:213–221.

13.

Levine

, et al.: A prognostic model for 1-year mortality in older adults after hospital discharge. Am J Med, 2007; 120:455–460.

14.

Walter

, et al.: Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA, 2001; 285:2987–2994.

15.

Rozzini

, Sabatini

, Trabucchi

: Prediction of 6-month mortality among older hospitalized adults. JAMA, 2001; 286:1315–1316.

16.

Di Bari

, et al.: Prognostic stratification of older persons based on simple administrative data: Development and validation of the “Silver Code,” to be used in emergency department triage. J Gerontol A Biol Sci Med Sci, 2010; 65:159–164.

17.

Inouye

, et al.: Burden of illness score for elderly persons: Risk adjustment incorporating the cumulative impact of diseases, physiologic abnormalities, and functional impairments. Med Care, 2003; 41:70–83.

18.

Pilotto

, et al.: Development and validation of a multidimensional prognostic index for one-year mortality from comprehensive geriatric assessment in hospitalized older patients. Rejuvenation Res, 2008; 11:151–161.

19.

Sancarlo

, et al.: Validation of a Modified-Multidimensional Prognostic Index (m-MPI) including the Mini Nutritional Assessment Short-Form (MNA-SF) for the prediction of one-year mortality in hospitalized elderly patients. J Nutr Health Aging, 2011; 15:169–173.

20.

Teno

, et al.: Prediction of survival for older hospitalized patients: The HELP survival model. Hospitalized Elderly Longitudinal Project. J Am Geriatr Soc, 2000; 48(5 Suppl):S16–S24.

21.

Drame

, et al.: Derivation and validation of a mortality-risk index from a cohort of frail elderly patients hospitalised in medical wards via emergencies: The SAFES study. Eur J Epidemiol, 2008; 23:783–791.

22.

Fischer

, et al.: A practical tool to identify patients who may benefit from a palliative approach: The CARING criteria. J Pain Symptom Manage, 2006; 31:285–292.

23.

Charlson

, et al.: A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. J Chronic Dis, 1987; 40:373–383.

24.

Deyo

, Cherkin

, Ciol

: Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol, 1992; 45:613–619.

25.

Romano

, Roos

, Jollis

: Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: Differing perspectives. J Clin Epidemiol, 1993; 46:1075–1079; discussion 1081–1090.

26.

Roos

, et al.: Short-term mortality after repair of hip fracture. Do Manitoba elderly do worse?. Med Care, 1996; 34:310–326.

27.

D'Hoore

, Sicotte

, Tilquin

: Risk adjustment in outcome assessment: The Charlson comorbidity index. Methods Inf Med, 1993; 32:382–387.

28.

Elixhauser

, et al.: Comorbidity measures for use with administrative data. Med Care, 1998; 36:8–27.

29.

van Walraven

, et al.: A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data. Med Care, 2009; 47:626–633.

30.

Knaus

, et al.: APACHE II: A severity of disease classification system. Crit Care Med, 1985; 13:818–829.

31.

Le Gall

, Lemeshow

, Saulnier

: A new simplified acute physiology score (SAPS II) based on a European/North American multicenter study. JAMA, 1993; 270:2957–2963.

32.

Knaus

, Wagner

, Draper

: APACHE III study design: Analytic plan for evaluation of severity and outcome in intensive care unit patients. Development of APACHE. Crit Care Med, 1989; 17(12 Pt 2):S181–S185.

33.

Lemeshow

, et al.: Mortality probability models for patients in the intensive care unit for 48 or 72 hours: A prospective, multicenter study. Crit Care Med, 1994; 22:1351–1358.

34.

Cook

, et al.: Prospective independent validation of APACHE III models in an Australian tertiary adult intensive care unit. Anaesth Intensive Care, 2002; 30:308–315.

35.

Knaus

, et al.: The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest, 1991; 100:1619–1636.

36.

Marshall

, et al.: Multiple organ dysfunction score: A reliable descriptor of a complex clinical outcome. Crit Care Med, 1995; 23:1638–1652.

37.

Christensen

, et al.: Comparison of Charlson comorbidity index with SAPS and APACHE scores for prediction of mortality following intensive care. Clin Epidemiol, 2011; 3:203–211.

38.

Quach

, et al.: A comparison between the APACHE II and Charlson Index Score for predicting hospital mortality in critically ill patients. BMC Health Serv Res, 2009; 9:129.

39.

Knaus

, et al.: Variations in mortality and length of stay in intensive care units. Ann Intern Med, 1993; 118:753–761.

40.

, et al.: The effect of comorbidities on risk of intensive care readmission during the same hospitalization: A linked data cohort study. J Crit Care, 2009; 24:101–107.

41.

, et al.: Combining multiple comorbidities with Acute Physiology Score to predict hospital mortality of critically ill patients: A linked data cohort study. Anaesthesia, 2007; 62:1095–1100.

42.

Umegaki

, et al.: An in-hospital mortality equation for mechanically ventilated patients in intensive care units. J Anesth, 2013; 27:541–549.

43.

Braber

, van Zanten

: Unravelling post-ICU mortality: Predictors and causes of death. Eur J Anaesthesiol, 2010; 27:486–490.

44.

Rosenberg

, Watts

: Patients readmitted to ICUs: A systematic review of risk factors and outcomes. Chest, 2000; 118:492–502.

45.

Mukhopadhyay

, et al.: Risk factors for hospital and long-term mortality of critically ill elderly patients admitted to an intensive care unit. Biomed Res Int, 2014; 2014:960575.

46.

Baldwin

, et al.: A prognostic model for 6-month mortality in elderly survivors of critical illness. Chest, 2013; 143:910–919.

47.

Lagu

, et al.: Development and validation of a model that uses enhanced administrative data to predict mortality in patients with sepsis. Crit Care Med, 2011; 39:2425–2430.

48.

Higgins

, et al.: Assessing contemporary intensive care unit outcome: An updated Mortality Probability Admission Model (MPM0-III). Crit Care Med, 2007; 35:827–835.

49.

Ongel

, et al.: How do COPD comorbidities affect ICU outcomes?. Int J Chron Obstruct Pulmon Dis, 2014; 9:1187–1196.

50.

Kheirbek

, Alemi

, Fletcher

: Heart failure prognosis: Comorbidities matter. J Palliat Med, 2015; 18:447–452.

51.

Levy

, et al.: Predictors of six-month mortality among nursing home residents: Diagnoses may be more predictive than functional disability. J Palliat Med, 2015; 18:100–106.

52.

Alemi

, Levy

, Kheirbek

: The multi-morbidity index: A tool for assessing the prognosis of patients from their history of illness, generating evidence and methods to improve patient outcomes. 2016; 4(1).

53.

VA Informatics and Computing Infrastructure CDW. www.hsrd.research.va.gov/for_researchers/vinci/cdw.cfm. (Last accessed November 19, 2015).

54.

VIReC Database and Methods Cyber Seminar Series.. 2013. www.hsrd.research.gov/for_researchers/vinci/colw.cfm (Last accessed November 23, 2016).

55.

Render

, Deddens

, Freyberg

, Almenoff

, Connors

Jr , Wagner

, Hofer

: Veterans Affairs intensive care unit risk adjustment model: Validation, Updating, Recalibration. Crit Care Med, 2008; 36:1031–1032.

56.

Render

, et al.: Veterans Affairs intensive care unit risk adjustment model: Validation, updating, recalibration. Crit Care Med, 2008; 36:1031–1042.

57.

Zimmerman

, et al.: Acute physiology and chronic health evaluation (APACHE) IV: Hospital mortality assessment for today's critically ill patients. Crit Care Med, 2006; 34:1297–1310.

58.

Render

, et al.: Automated computerized intensive care unit severity of illness measure in the Department of Veterans Affairs: Preliminary results. SISVistA Investigators. Scrutiny of ICU Severity Veterans Health Sysyems Technology Architecture. Crit Care Med, 2000; 28:3540–3546.

59.

Singh

, Ayub

: Accuracy of VA databases for diagnoses of knee replacement and hip replacement. Osteoarthritis Cartilage, 2010; 18:1639–1642.

60.

Lopes

, Tobias

: Confronting prior convictions: on issues of prior sensitivity and likelihood robustness in Bayesian analysis. Annu Rev Econ, 2011; 3:107–131.

61.

Robin

, et al.: pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 2011; 12:77.

62.

Chrischilles

, et al.: Beyond comorbidity: Expanding the definition and measurement of complexity among older adults using administrative claims data. Med Care, 2014; 52 Suppl 3:S75–S84.

63.

Iezzoni

, et al.: Predicting who dies depends on how severity is measured: Implications for evaluating patient outcomes. Ann Intern Med, 1995; 123:763–770.

64.

Iezzoni

, et al.: Predicting in-hospital mortality for stroke patients: Results differ across severity-measurement methods. Med Decis Making, 1996; 16:348–356.

65.

Que

, et al.: Prognostication of mortality in critically ill patients with severe infections. Chest, 2015; 148:674–682.

66.

Xiao

, et al.: Prognostic value of different scoring models in patients with multiple organ dysfunction syndrome associated with acute COPD exacerbation. J Thorac Dis, 2015; 7:329–336.