Development and Cross-Validation of the In-Hospital Mortality Prediction in Advanced Cancer Patients Score: A Preliminary Study

Abstract

Purpose:

Acute palliative care units (APCUs) provide intensive symptom support and transition of care for advanced cancer patients. Better understanding of the predictors of in-hospital mortality is needed to facilitate program planning and patient care. In this prospective study, we identified predictors of APCU mortality, and developed a four-item In-hospital Mortality Prediction in Advanced Cancer Patients (IMPACT) predictive model.

Methods:

Between April and July 2010, we documented baseline demographics, the Edmonton Symptom Assessment Scale (ESAS), 80 clinical signs including known prognostic factors, and 26 acute complications on admission in consecutive APCU patients. Multivariate logistic regression analysis was used to identify factors for inclusion in a nomogram, which was cross-validated with bootstrap analysis.

Results:

Among 151 consecutive patients, the median age was 58, 13 (9%) had hematologic malignancies, and 52 (34%) died in the hospital. In multivariate analysis, factors associated with in-hospital mortality were advanced education (odds ration [OR]=11.8, p=0.002), hematologic malignancies (OR=8.6, p=0.02), delirium (OR=4.3, p=0.02), and high ESAS global distress score (OR=20.8, p=0.01). In a nomogram based on these four factors, total scores of 6, 10, 14, 17, and 21 corresponded to a risk of death of 10%, 25%, 50%, 75%, and 90%, respectively. The model has 92% sensitivity and 88% specificity for predicting patients at low/high risk of dying in the hospital, and a receiver-operator characteristic curve concordance index of 83%.

Conclusions:

Higher education was associated with increased utilization of the interdisciplinary palliative care unit until at the end of life. Patients with higher symptom burden, delirium, and hematologic malignancies were also more likely to require APCU care until death.

Introduction

Patients with advanced cancer frequently develop significant symptoms and acute complications requiring hospitalization at the end of life. Some of these admissions involve a stay at an acute palliative care unit (APCU) for intensive symptom management and/or transition to end-of-life care.^1,2 During this admission, many important therapeutic and discharge decisions are made, such as cessation of aggressive therapies, and hospice and home care referral. Many of these decisions are dependent on the patient's prognosis, and whether he/she is expected to be discharged alive.³

The issue of in-hospital mortality has been addressed by a number of prognostic models. In the critical care setting, the Acute Physiology and Chronic Health Evaluation (APACHE) system has been used to provide information regarding the expected length of stay and mortality rate.⁴ A number of in-hospital mortality models are also available in the general medicine/oncology⁵ and postoperative settings.^6,7 The use of these models can facilitate both program planning and clinical decision making by allowing clinicians to identify high-risk patients and to initiate proper interventions.⁸

There is currently no established in-hospital mortality model for APCU. In a retrospective study, we examined a number of factors associated with in-hospital mortality among 2568 advanced cancer patients admitted to an APCU using data available from institutional administrative databases, and found that male gender, hematologic malignancies, and admissions from other oncology units were associated with death in the APCU.¹ In another retrospective study involving 500 advanced cancer patients, younger age, admission from other oncology units, hyponatremia, hypernatremia, high blood urea nitrogen, high heart rate, and supplemental oxygen use were identified as indicators of in-hospital mortality.⁹ The lack of prospective studies means that many important signs and symptoms were not included in these analyses. A better understanding of the factors associated with APCU mortality would allow clinicians to identify patients at risk for dying in the hospital, and to provide appropriate therapeutic and discharge recommendations for patients and their families. An in-hospital mortality prediction model for the APCU similar to the APACHE system in the intensive care unit (ICU) setting could have significant implications clinically, administratively, and academically. In this prospective study, we identified predictors of APCU mortality, and developed a four-item prognostic model for in-hospital death.

Patients and Methods

Study setting and criteria

The Institutional Review Board at The University of Texas M.D. Anderson Cancer Center approved this study. All clinicians who participated in this study signed the informed consent prior to enrollment. The Institutional Review Board provided waiver of consent for patient participation.

Consecutive patients with advanced cancer who were ≥18 years of age and admitted to the APCU at M.D. Anderson Cancer Center between April and July 2010 were included in this study. This APCU is a dedicated 12-bed unit that provides intensive symptom support and transition of care for patients with advanced cancer and their families.¹⁰ It is staffed by an interdisciplinary team of nurses, physicians, physiotherapists, an occupational therapist, a pharmacist, a social worker, a chaplain, and other allied health professionals.

Data collection

We collected baseline patient demographics, including age, sex, ethnicity, religion, marital status, education, cancer diagnosis, and source of admission. Within the first 12 hours of admission, the bedside nurse documented the Edmonton Symptom Assessment Scale (ESAS), the palliative performance scale (PPS), and 80 signs and symptoms of prognostic significance. These included all the vital signs and various abnormalities related to the neurological (e.g., level of consciousness, motor/sensory changes, seizures, myoclonus), cardiovascular (e.g., decreased heart sounds, pulselessness of radial artery, peripheral edema), respiratory (e.g., dyspnea, respiration with mandibular movement, grunting, death rattle, apnea, Cheyne Stokes breathing, use of respiratory devices), gastrointestinal (GI) (e.g., dysphagia, hematemesis, fecal incontinence), genitourinary (e.g., urine output, hematuria, urinary incontinence), and integumentary (e.g., skin coolness, cyanosis, mottling, jaundice) systems, as well as other signs associated with the dying process (e.g., hyperextension of neck, drooping of nasolabial fold, inability to clear sections, unable to close eyelids). All nurses who participated in this study had specialized training in palliative care, and had been working at the APCU for at least one month. The attending physician on the unit was responsible for documenting the presence or absence of 26 acute complications within 24 hours of admission. Vital status (dead or alive) at the end of hospital stay was also recorded.

ESAS measures 10 symptoms (pain, fatigue, nausea, depression, anxiety, drowsiness, appetite, feeling of well-being, shortness of breath, and sleep) in the previous 24 hours using an 11-point numeric rating scale from 0 (no symptom) to 10 (worst symptom),¹¹ and has been validated in cancer patients.¹² The ESAS was completed by patients with assistance from responsible clinicians. Patients who were delirious did not complete the ESAS. A global symptom distress score was subsequently calculated based on the sum of nine aforementioned symptoms (all except sleep), with a total score from 0 to 90.¹³

The Palliative Performance Scale (PPS) is an 11-point functional assessment scale modified from the Karnofsky Performance Scale.¹⁴ It is designed for palliative care patients, and provides a score between 0% and 100% (0%=death, 100%=completely asymptomatic) based on patient's ambulation, activity level, disease severity, ability to care for self, oral intake, and level of consciousness. It has been validated for palliative care patients¹⁵ and has prognostic utility.¹⁶

In addition to ESAS and PPS, we systematically documented 80 signs and symptoms within the first 12 hours of admission, including abnormalities regarding the vitals, neurological, head and neck, respiratory, cardiovascular, gastrointestinal, genitourinary, and integumentary systems, and any psychosocial concerns.

Acute complications documented by our attending APCU physicians included bowel obstruction, bowel perforation, cachexia, cerebral hemorrhage, delirium, fracture(s), heart failure, hemoptysis, hypercalcemia, hyperkalemia, hypernatremia, hyponatremia, ischemic stroke, lower GI bleed, metabolic acidosis, myocardial infarction, peritonitis, pneumonia, pressure ulcer, pulmonary embolism, renal failure, retroperitoneal bleed, sepsis, tamponade, upper GI bleed, and urinary tract infection. Delirium was diagnosed by the attending APCU physician based on the Diagnostic and Statistical Manual of Mental Disorders IV, Text Revision (American Psychiatric Association [2000]). criteria to accommodate for uncertainty in establishing this clinical diagnosis. The last two choices were coded as “Yes” during the analysis. All physicians who participated were board certified in palliative care and provided patient care based on standardized clinical protocols.

All eight physicians and 20 nurses who participated in the study had a 30-minute orientation to review the study objectives and data collection forms. All standardized forms were filled out independently without reference to what was completed the day before.

Statistical analysis

We summarized the baseline demographics using descriptive statistics, including medians, ranges, frequencies, and 95% confidence intervals (CIs).

We initially completed univariate analyses to determine which factors were related to death in an APCU as opposed to being discharged alive. Only 77 variables with p<0.10 were further examined in our model. A more liberal value for statistical significance was used in the initial building stage of our model because we did not want to exclude for further analysis any variables that might truly impact in-hospital mortality but might have had elevated statistical significance due to confounding. We then further selected 15 of the 38 variables that exhibited at least a threefold increase or decrease in odds of dying for multivariate analysis using backward selection (i.e., education, hematologic tumor, delirium, ESAS, PPS, and multiple signs including tachypnea, hypertension, disorientation, disorganized thinking, drooping of nasal labial fold, death rattle, Cheyne-Stokes breathing, dysphagia, metabolic acidosis, and urinary incontinence). The threefold threshold was chosen to ensure that only factors with the strongest predictive odds ratio (OR) were included for further analysis. ESAS was found to have a linear risk response with in-hospital death, so it was divided into three equal categories (0–30, 31–60, 61–90) to make application of the nomogram simpler. This did not impact the efficacy of the final model. Our final In-hospital Mortality Prediction in Advanced Cancer Patients (IMPACT) model included only four variables—education, delirium, hematologic malignancy, and ESAS; all were statistically significant with p<0.05 as a whole. The score ranged from 0 (lowest risk of in-hospital mortality) to 30 (highest risk). Similar to other prognostic scoring systems such as the Palliative Prognostic Score¹⁷ and Palliative Prognostic Index,¹⁸ we divided the IMPACT score into three categories—low risk (≤6), moderate risk (7–14) and high risk (≥15)—to facilitate application of this model.

The nomogram was then internally validated using bootstrapping techniques, in which 500 random samples with replacement were generated that had the same sample size as the cohort used to create our final IMPACT model. We then fit the model from the original cohort onto the bootstrap samples to estimate how well the model would perform in future data.

Performance of the IMPACT model was characterized by (1) how well it was able to differentiate between groups (i.e., discrimination) and (2) how well the predicted outcome correlated with the actual outcome (i.e., calibration).¹⁹ We assessed discrimination using the area under the curve (AUC) of the receiver-operating characteristic (ROC) curve, with a concordance index of 0.5 indicating no discrimination, and 1.0 indicating perfect discrimination. Calibration was examined using the Hosmer-Lemeshow statistic, where a pvalue of >0.05 indicates good fit.

The IMPACT model had limited discrimination for patients with moderate scores (7–14). Consequently, these individuals were excluded from determination of sensitivity and specificity for the model. Sensitivity was calculated by dividing the number of patients with scores ≥15 who died by the total number of in-hospital death. Specificity was determined by dividing the number of patients with scores ≤6 and discharged alive by the total number of patients discharged alive.

STATA (Stata/SE version 11.1, StataCorp LP, College Station, TX) and R (version 2.12.2) software were used for statistical analysis.

Results

Patient characteristics

A total of 151 patients were enrolled in this study. The patient demographics on APCU admission are summarized in Table 1. The median age was 58 (range 18–85), and the majority were Caucasian. GI, lung, and gynecologic malignancies were the most common primary oncologic diagnoses. The median duration of APCU admission was 6 days (interquartile range 4–8 days). A total of 52 (34%) patients died during the admission.

Table 1.

Patient Characteristics on Acute Palliative Care Unit Admission

Patient characteristics	N (%)^a
Age, median (range)	58 (18–85)
Female sex	95 (63)
Ethnicity
Caucasian	97 (64)
African American	21 (14)
Hispanic	27 (18)
Other	6 (4)
Christian	130 (86)
Married	92 (62)
Education
High school or below	60 (40)
College education	45 (30)
Post graduate education	17 (11)
Missing	29 (19)
Cancer
Breast	20 (13)
Dermatologic	9 (6)
Gastrointestinal	30 (20)
Genitourinary	12 (8)
Gynecologic	23 (15)
Head and neck	(5)
Hematologic	13 (9)
Respiratory	27 (18)
Sarcoma	6 (4)
Other	3 (2)
Months between cancer diagnosis and APCU admission, median (interquartile range)	19 (9–43)
Total ESAS
0–30	25 (17)
31–60	83 (55)
61–90	11 (7)
Missing^b	32 (21)

Unless otherwise specified.

Patients with significant delirium were not able to complete ESAS.

APCU, acute palliative care unit; ESAS, Edmonton Symptom Assessment Scale.

Predictors associated with in-hospital mortality

Table 2 shows the univariate and multivariate analysis for in-hospital mortality. We found that advanced education, hematologic malignancies, delirium, and high ESAS global symptom distress score were significantly associated with in-hospital death. The hematologic malignancies included non-Hodgkin lymphoma (n=5), Hodgkin lymphoma (n=1), multiple myeloma (n=1), acute myeloid leukemia (n=2), acute lymphocytic leukemia (n=1), chronic myeloid leukemia (n=1), and chronic lymphocytic leukemia (n=2).

Table 2.

Predictors of In-Hospital Mortality

	Univariate analysis					Multivariate analysis
	Alive N=99 (%)	Died N=52 (%)	Fisher's exact P value	Odds ratio (95% CI)	P value	Odds ratio (95% CI)	P value
Education			0.005				0.008
High school or less	48 (59)	12 (30)		1.0	–	1.0	–
College education	27 (33)	18 (45)		2.7 (1.1-6.4)	0.027	2.1 (0.6-6.9)	0.24
Post graduate education	7 (9)	10 (25)		5.7 (1.8-18.1)	0.003	11.8 (2.5-56.2)	0.002
Hematologic malignancy	5 (5)	8 (15)	0.035	3.4 (1.1-11.0)	0.040	8.6 (1.3-55.2)	0.02
Delirium	48 (49)	43 (83)	<0.0001	5.1 (2.2-11.5)	<0.0001	4.3 (1.3-14.8)	0.02
Total ESAS			0.006				0.038
0–30	23 (27)	2 (6)		1.0	–	1.0	–
31–60	56 (67)	27 (77)		5.5 (1.2-25.3)	0.027	3.7 (0.7-20.5)	0.13
61–90	5 (6)	6 (17)		13.8 (2.1-89.5)	0.006	20.8 (2.0-214.5)	0.01

CI, confidence interval; ESAS, Edmonton Symptom Assessment Scale.

Nomogram development and validation

Based on the four predictive factors identified above, we constructed a nomogram (Table 3). This was used to generate an IMPACT score, which ranged from 0 to 30, with a higher score indicating an increased risk of in-hospital mortality (Table 3).

Table 3.

IMPACT Scoring and Interpretation

Scoring
Variables	Nomogram score
Education
High school	0
College or less	2
Advanced	8
Hematologic malignancy	7
Delirium	5
Total ESAS
0–30	0
31–60	4
61–90	10

Interpretation
Total score	Risk of in-hospital death
6	0.10
10	0.25
14	0.5
17	0.75
21	0.9

ESAS, Edmonton Symptom Assessment Scale; IMPACT, In-hospital Mortality Prediction in Advanced Cancer Patients.

We assessed how well the predicted outcome correlated with the actual outcome (i.e., calibration). As shown in Figure 1, our model provided a good fit, particularly for low scores. It slightly underestimates the risk of death with intermediate scores and slightly overestimates the risk of death at higher scores. Furthermore, the Hosmer-Lemeshow goodness-of fit-statistic was 5.0 (p=0.66), indicating good calibration.

FIG. 1.

Calibration of nomogram. This diagram illustrates how far the predictions are from the actual risk of death. The x-axis shows the predicted probability of death at discharge based on the logistic regression model. The y-axis represents the actual probability of death at discharge, which is calculated by (1) dividing patients into subgroups according to their predicted probability of death at discharge in ascending order and then (2) determining the proportion of actual death at discharge for each subgroup. There are two predicted curves in the figure—apparent and bias corrected. The apparent method assesses the performance of the logistic regression model using the same data that were fit to the model, whereas bias-corrected uses data generated from 500 bootstrap samples and is used to estimate how well the model will perform for future data. Perfect calibration would result in both curves falling on the 45-degree line.

Figure 2 shows the ROC curve for our model. The concordance index was 83%, suggesting good discrimination.

FIG. 2.

Receiver operator characteristic (ROC) curve. The IMPACT model's ability to discriminate between groups was assessed by an ROC curve created using data from all 151 patients. The vertical axis represents sensitivity, which is the proportion of predicted death among patients who actually died, and the horizontal axis represents the proportion of predicted death among patients who were actually discharged alive. The more concave the curve, the better the prediction ability. Area under the curve (AUC) is a numerical assessment of concavity. An AUC of 50% indicates that prediction is based purely on random chance, and an AUC of 100% indicates that the outcome can be perfectly predicted. The AUC for the authors model was 0.83, suggesting good discrimination.

This model was used to build the nomogram scoring system and we applied this to our derivation cohort to calculate each subject's total score. Among 37 patients with IMPACT scores ≤6, 1 (2.7%) died; among 46 patients with scores of 7 to 14, 15 (32.6%) died; among 16 patients with scores ≥15, 11 (69%) died. Thus, the sensitivity of the model was 92% (95% CI, 62-100), and the specificity was 88% (95% CI, 74-96).

Discussion

We examined a comprehensive list of signs and symptoms related to poor prognosis, and identified four predictive factors for in-hospital mortality. We found that higher education was associated with increased utilization of the interdisciplinary APCU until the end of life. Patients with higher symptom burden, delirium, and hematologic malignancies were also more likely to require APCU care until death. Based on these findings, we constructed the IMPACT model and conducted an internal validation.

Given the significant morbidities associated with cancer progression and its treatments, optimal supportive care is essential to alleviate symptom burden and distress. The physical presence of an APCU within a tertiary care cancer center promotes simultaneous oncologic and palliative care for hospitalized patients with particularly high symptom expression.^20,21 The APCU also plays a critical role in providing family support, and facilitates transition to end-of-life care and complex discharge planning.¹ In-hospital mortality is dependent on the length of survival; however, the two outcomes differ in that in-hospital mortality is also affected by myriad other factors, such as dischargablity, psychosocial concerns, various logistical issues, and institutional policy. Successful development of an in-hospital mortality algorithm would facilitate benchmarking and enable quality improvement efforts at the national and institutional level, providing information at a higher resolution than APCU mortality rate alone. It would also assist patients and health professionals in end-of-life decision making and resource allocation.²² Despite the need for accurate prediction of in-hospital death, there is a paucity of research on APCU mortality models. This could be explained by the small number of academic centers equipped with such facilities.²³ As APCUs become more common, the use of in-hospital mortality prediction tools will likely become more widespread.

Interestingly, we identified advanced education as an important predictor in hospital mortality. Patients with postcollege eduction had an OR of 12 for dying at the APCU. There are several possible explanations for this observation. First, higher education is a marker of higher socioeconomic status and thus access to health care resources.^24,25 Meropol and colleagues demonstrated that patients with advanced degrees tend to be more likely to want to focus on quality of life.²⁶ Given the limited level of support in hospices, highly educated patients may prefer to die in the APCU where they have access to intensive symptom control and monitoring while their families receive psychosocial support in an interdisciplinary holistic environment. This hypothesis challenges the “Holy Grail,” including our own belief, that a majority of cancer patients wish to die at home.^27–29 Second, previous studies have shown that patients with higher socioeconomic status were more likely to be in agreement with their physicians in decision making preferences.³⁰ Physicians may find it easier to empathize with highly educated individuals, and agree more with their choices. Third, higher education may potentially be associated with higher frequency of antineoplastic therapy use, which may delay referral to the palliative care service leading to higher in-hospital mortality. Finally, there may be institutional pressure to discharge patients with a low level of education and/or socioeconomic status due to insurance issues. Clearly, further research is required to understand the interplay between education level, socioeconomic status, and discharge outcome.

High symptom burden was identified as a novel predictor for in-hospital mortality. A previous study examining the ESAS only analyzed individual symptoms, rather than the combined symptom burden.⁹ A high symptom burden confers a poor prognosis, which is associated with high in-hospital mortality. This finding may explain why effective management of symptoms by a palliative care team was associated with improved survival in a recent randomized controlled trial.³¹ A high global distress score could also be indicative of high symptom expression as a manifestation of suffering. Patients with severe pain and other symptoms tend to be more difficult to treat,³² and thus would be more challenging to discharge.

Related to symptom burden, we also found delirium to be an important predictor of in-hospital mortality. Delirium is a well-established prognostic factor,³³ and is associated with higher symptom expression³⁴ and higher distress in families and caregivers, potentially leading to increased caregiver burden, making such patients less dischargable.^35,36 Some patients may require high-dose neuroleptics and/or palliative sedation for agitation, hampering discharge to home or inpatient hospices. Consequently, these patients were more likely to die in the hospital.

Hematologic malignancy was another factor associated with increased in-hospital mortality. Our finding is consistent with previous studies examining predictors of in-hospital mortality.¹ Indeed, patients with hematologic malignancies tend to have a higher symptom burden,³⁷ and receive more aggressive care at the end of life, including antineoplastic therapies,¹⁰ critical care unit admissions, and ICU deaths.³⁸ They were also referred very late in the disease trajectory, contributing to the higher in-hospital mortality rate.³⁹ Finally, many patients with hematologic malignancy require blood product support at the end of life, making them less likely to be discharged to hospices.

This is the first prospective study that proposes an in-hospital mortality model for APCU patients. We identified four simple factors that are amenable to bedside assessment, with a high sensitivity and specificity for patients at lower and higher risk for death at discharge. The discrimination of our four-variable model (AUC=0.83) resembles that of the APACHE IV score (AUC=0.88) that requires 48 variables. Our findings will hopefully stimulate other groups to independently validate and improve this predictive model.

The IMPACT score has potential implications for patients with either a low or a high score. A patient with an IMPACT score of 6 or lower is highly likely to be discharged, and discharge planning should start at admission. Conversely, an IMPACT score of 15 or higher indicates a high likelihood of in-hospital death, and an active effort to discharge the patient could result in unnecessary distress for the patient, the family, and the health care team.

This preliminary study has several important limitations. First, we only had a small sample size, and examined a large array of signs and symptoms during the development of this predictive model. Thus, we restricted the number of factors included in the multivariate analysis. Second, the missing data in ESAS (related to delirium) and education limited the sample size further. Third, the small sample size also means that we were not able to validate the model with an independent cohort. Cross-validation with bootstrapping was provided instead. Further studies are necessary to confirm our findings. Third, the threefold cutoff for variable selection may have inadvertently excluded some important variables from the final model. Finally, this study was conducted in a tertiary care cancer center with a unique patient population and referral pattern. The four variables identified in this study may only be applicable to our inpatient unit. It is unclear whether our findings can be generalized to APCUs in other hospitals. External validation is needed before our findings can be applied in routine clinical practice.

In summary, we developed and validated a predictive model for APCU mortality. Higher education was identified as a novel factor contributing to increased risk of APCU death, leading us to hypothesize that patients and families may actually prefer death in a controlled environment, providing the appropriate resources are available. The other factors including total symptom burden, delirium, and diagnosis of hematologic malignancies are all markers of poor prognosis, higher care needs, and more difficult discharges. Further validation of the IMPACT model, which would include using data from other centers, is warranted to determine its impact on program planning and patient care.

Footnotes

Acknowledgments

We would like to thank all physicians and nurses who participated in the data collection for this study.

Author Disclosure Statement

This research is supported in part by the National Institutes of Health through M.D. Anderson's Cancer Center Support Grant CA016672. Dr. Bruera is in part supported by the National Institutes of Health grants RO1NR010162-01A1, RO1CA122292-01, and RO1CA124481-01. The funding source/sponsor was not involved in the conduction of the study or development of the manuscript. The authors have full control of all primary data.

References

Hui

, Elsayem

, Palla

, De La Cruz

, Li

, Yennurajalingam

, Bruera

. Discharge outcomes and survival of patients with advanced cancer admitted to an acute palliative care unit at a comprehensive cancer center. J Palliat Med, 2010; 13:49–57.

Elsayem

, Swint

, Fisch

, Palmer

, Reddy

, Walker

, Zhukovsky

, Knight

, Bruera

. Palliative care inpatient service in a comprehensive cancer center: Clinical and financial outcomes. J Clin Oncol, 2004; 22:2008–2014.

Hui

, Con

, Christie

, Hawley

. Goals of care and end-of-life decision making for hospitalized patients at a Canadian tertiary care cancer center. J Pain Symptom Manage, 2009; 38:871–881.

Zimmerman

, Kramer

, McNair

, Malila

. Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality assessment for today's critically ill patients. Crit Care Med, 2006; 34:1297–1310.

Bozcuk

, Koyuncu

, Yildiz

, Samur

, Ozdogan

, Artaç

, Coban

, Savas

. A simple and accurate prediction model to estimate the intrahospital mortality risk of hospitalised cancer patients. Int J Clin Pract, 2004; 58:1014–1019.

Chandra

, Mangam

, Marzouk

. A review of risk scoring systems utilised in patients undergoing gastrointestinal surgery. J Gastrointest Surg, 2009; 13:1529–1538.

Ren

, Upadhyay

, Wang

, Li

, Lu

, Fu

. Mortality rate prediction by Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity (POSSUM), Portsmouth POSSUM and Colorectal POSSUM and the development of new scoring systems in Chinese colorectal cancer patients. Am J Surg, 2009; 198:31–38.

Paterson

, MacLeod

, Thetford

, Beattie

, Graham

, Lam

, Bell

. Prediction of in-hospital mortality and length of stay using an early warning scoring system: clinical audit. Clin Med, 2006; 6:281–284.

Elsayem

, Mori

, Parsons

, Munsell

, Hui

, Delgado-Guay

, Paraskevopoulos

, Fadul

, Bruera

. Predictors of inpatient mortality in an acute palliative care unit at a comprehensive cancer center. Support Care Cancer, 2010; 18:67–76.

10.

Hui

, Elsayem

, Li

, De La Cruz

, Palmer

, Bruera

. Antineoplastic therapy use in patients with advanced cancer admitted to an acute palliative care unit at a comprehensive cancer center: A simultaneous care model. Cancer, 2010; 116:2036–2043.

11.

Bruera

, Kuehn

, Miller

, Selmser

, Macmillan

. The Edmonton Symptom Assessment System (ESAS): A simple method for the assessment of palliative care patients. J Palliat Care, 1991; 7:6–9.

12.

Richardson

, Jones

. A review of the reliability and validity of the Edmonton Symptom Assessment System. Current Oncol (Toronto, Ont.), 2009; 16:55.

13.

Zimmermann

, Burman

, Bandukwala

, Seccareccia

, Kaya

, Bryson

, Rodin

, Lo

. Nurse and physician inter-rater agreement of three performance status measures in palliative care outpatients. Support. Care Cancer, 2010; 18:609–616.

14.

Anderson

, Downing

, Hill

, Casorso

, Lerch

. Palliative performance scale (PPS): A new tool. J Palliat Care, 1996; 12:5–11.

15.

, Lau

, Downing

, Lesperance

. A reliability and validity study of the Palliative Performance Scale. BMC Palliat Care, 2008; 7:10.

16.

Downing

, Lau

, Lesperance

, Karlson

, Shaw

, Kuziemsky

, Bernard

, Hanson

, Olajide

, Head

, Ritchie

, Harrold

, Casarett

. Meta-analysis of survival prediction with Palliative Performance Scale. J Palliat Care, 2007; 23:245–252discussion 252–244.

17.

Maltoni

, Nanni

, Pirovano

, Scarpi

, Indelli

, Martini

, Monti

, Arnoldi

, Piva

, Ravaioli

, Cruciani

, Labianca

, Amadori

. Successful validation of the palliative prognostic score in terminally ill cancer patients. Italian Multicenter Study Group on Palliative Care. J Pain Symptom Manage, 1999; 17:240–247.

18.

Morita

, Tsunoda

, Inoue

, Chihara

. The Palliative Prognostic Index: A scoring system for survival prediction of terminally ill cancer patients. Support Care Cancer, 1999; 7:128–133.

19.

Hadorn

, Keeler

, Rogers

, Brook

. Assessing the performance of mortality models. www.rand.org/pubs/monograph_reports/MR181.html. 2011 March 29.

20.

Bruera

, Hui

. Integrating supportive and palliative care in the trajectory of cancer: Establishing goals and models of care. J Clin Oncol, 2010; 28:4013–4017.

21.

Byock

, Twohig

, Merriman

, Collins

. Promoting excellence in end-of-life care: A report on innovative models of palliative care. J Palliat Med, 2006; 9:137–151.

22.

Keegan

, Gajic

, Afessa

. Severity of illness scoring systems in the intensive care unit. Crit Care Med, 2011; 39:163–169.

23.

Hui

, Elsayem

, De la Cruz

, Berger

, Zhukovsky

, Palla

, Evans

, Fadul

, Palmer

, Bruera

. Availability and integration of palliative care at US cancer centers. JAMA, 2010; 303:1054–1061.

24.

Herndon

2nd , Kornblith

, Holland

, Paskett

. Patient education level as a predictor of survival in lung cancer clinical trials. J Clin Oncol, 2008; 26:4116–4123.

25.

Hussain

, Lenner

, Sundquist

, Hemminki

. Influence of education level on cancer survival in Sweden. Ann Oncol, 2008; 19:156–162.

26.

Meropol

, Egleston

, Buzaglo

, Benson

3rd , Cegala

, Diefenbach

, Fleisher

, Miller

, Sulmasy

, Weinfurt

. CONNECT Study Research Group. Cancer patient preferences for quality and length of life. Cancer, 2008; 113:3459–3466.

27.

Bruera

, Russell

, Sweeney

, Fisch

, Palmer

. Place of death and its predictors for local patients registered at a comprehensive cancer center. J Clin Oncol, 2002; 20:2127–2133.

28.

Bruera

, Sweeney

, Russell

, Willey

, Palmer

. Place of death of Houston area residents with cancer over a two-year period. J Pain Symptom Manage, 2003; 26:637–643.

29.

Cantwell

, Turco

, Brenneis

, Hanson

, Neumann

, Bruera

. Predictors of home death in palliative care cancer patients. J Palliat Care, 2000; 16:23–28.

30.

Bruera

, Willey

, Palmer

, Rosales

. Treatment decisions for breast carcinoma: Patient preferences and physician perceptions. Cancer, 2002; 94:2076–2080.

31.

Temel

, Greer

, Muzikansky

, Gallagher

, Admane

, Jackson

, Dahlin

, Blinderman

, Jacobsen

, Pirl

, Billings

, Lynch

. Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med, 2010; 363:733–742.

32.

Fainsinger

, Fairchild

, Nekolaichuk

, Lawlor

, Lowe

, Hanson

. Is pain intensity a predictor of the complexity of cancer pain management? J Clin Oncol, 2009; 27:585–590.

33.

Lawlor

, Gagnon

, Mancini

, Pereira

, Hanson

, Suarez-Almazor

, Bruera

. Occurrence, causes, and outcome of delirium in patients with advanced cancer: A prospective study. Arch Intern Med, 2000; 160:786–794.

34.

Lawlor

, Bruera

. Delirium in patients with advanced cancer. Hematol Oncol Clin North Am, 2002; 16:701–714.

35.

Bruera

, Bush

, Willey

, Paraskevopoulos

, Li

, Palmer

, Cohen

, Sivesind

, Elsayem

. Impact of delirium and recall on the level of distress in patients with advanced cancer and their family caregivers. Cancer, 2009; 115:2004–2012.

36.

Hui

, Bush

, Gallo

, Palmer

, Yennurajalingam

, Bruera

. Neuroleptic dose in the management of delirium in patients with advanced cancer. J Pain Symptom Manage, 2010; 39:186–196.

37.

Fadul

, El Osta

, Dalal

, Poulter

, Bruera

. Comparison of symptom burden among patients referred to palliative care with hematologic malignancies versus those with solid tumors. J Palliat Med, 2008; 11:422–427.

38.

McGrath

. Are we making progress? Not in haematology! Omega, 2002; 45:331–348.

39.

Fadul

, Elsayem

, Palmer

, Zhang

, Braiteh

, Bruera

. Predictors of access to palliative care services among patients who died at a Comprehensive Cancer Center. J Palliat Med, 2007; 10:1146–1152.