Abstract
Abstract
Objective:
To determine whether a prognostic index could predict one-week mortality more accurately than hospice nurses can.
Method:
An electronic health record-based retrospective cohort study of 21,074 hospice patients was conducted in three hospice programs in the Southeast, Northeast, and Midwest United States. Model development used logistic regression with bootstrapped confidence intervals and multiple imputation to account for missing data. The main outcome measure was mortality within 7 days of hospice enrollment.
Results:
A total of 21,074 patients were admitted to hospice between October 1, 2008 and May 31, 2011, and 5562 (26.4%) died within 7 days. An optimal predictive model included the Palliative Performance Scale (PPS) score, admission from a hospital, and gender. The model had a c-statistic of 0.86 in the training sample and 0.84 in the validation sample, which was greater than that of nurses' predictions (0.72). The index's performance was best for patients with pulmonary disease (0.89) and worst for patients with cancer and dementia (both 0.80). The index's predictions of mortality rates in each index category were within 5.0% of actual rates, whereas nurses underestimated mortality by up to 18.9%. Using the optimal index threshold (<3), the index's predictions had a better c-statistic (0.78 versus 0.72) and higher sensitivity (74.4% versus 47.8%) than did nurses' predictions but a lower specificity (80.6% versus 95.1%).
Conclusions:
Although nurses can often identify patients who will die within 7 days, a simple model based on available clinical information offers improved accuracy and could help to identify those patients who are at high risk for short-term mortality.
Introduction
Although patients are seeking hospice care in increasing numbers, they are still enrolling in hospice very late in the course of illness. For instance, half of hospice patients are still referred in the last 3 weeks of life. 2 Moreover, one-third of patients are referred in the last week and 10% are referred in the last 24 hours. Patients with shorter lengths of stay, 3 and particularly those whose families say they were referred too late, 4 may have more unmet needs for care.
It is possible that improved advance care planning and other interventions could encourage patients to consider hospice earlier. However, it also will be be essential to explore ways in which hospices can more effectively meet the needs of patients who are referred to hospice very late. That is, it will be important to ensure that these patients receive aggressive “front-loaded” services. Moreover, some patients who are at high risk for dying very soon and who have severe symptoms could receive care in an inpatient unit that is optimally designed to provide high-intensity palliative care.
It is not known how accurately hospice nurses—the de facto “gold standard”—are able to identify patients who are likely to die within 7 days of enrollment. Nor is it known whether a prognostic model can improve the accuracy of their predictions. Therefore, the goals of this study were to determine how well hospice providers are able to identify patients at high risk for dying within one week, and to determine whether a simple prognostic index could improve these predictions.
Method
Setting and sample
The three participating hospices (Suncoast Hospice, Clearwater, FL; Agrace HospiceCare, Madison WI; Hospice of Lancaster County, Lancaster, PA) are pilot members of the CHOICE network (Coalition of Hospices Organized to Investigate Comparative Effectiveness). CHOICE hospices all use Suncoast Solutions Inc.'s Electronic Health Record (EHR) Software, and all agree to permit use of their data for research projects that are defined by the coalition's members. Participating hospices range in size (daily census) from 500 to 2700 patients/day. Data elements for this study were defined a priori and then extracted from participating hospices' EHRs. Extracted data were then stripped of identifiers to create a Health Insurance Portability and Accountability Act (HIPAA)-compliant limited dataset, and transferred as an encrypted file to the University of Pennsylvania for analysis.
Data analysis
We used logistic regression models to identify variables that were associated with an increased likelihood of dying within 7 days. In selecting candidate variables, we focused on those that would be readily available on admission, to develop a model that hospice providers and referring physicians could apply at the time of a patient's enrollment in hospice. We used a limited set of demographic variables, including age, gender, race (white versus nonwhite), site of care (e.g., hospital, nursing home) as well as marital status, which has been described as a predictor of hospice survival. 5
As a measure of functional status, we included the Palliative Performance Scale (PPS), which has been widely studied as a predictor of survival in hospice and palliative care populations.5–8 The PPS is scored from 0 to 100 in 10-point increments to create an 11-point scale in which higher numbers correspond to better functional status. We also considered clinical variables that have not been studied as predictors of prognosis in this population, but which are plausibly associated with poor survival. These included the presence of a wound (venous stasis, pressure ulcer, or malignant), the use of medical devices (e.g., intravenous or urinary catheters), and the location of care at the time of hospice enrollment (hospital versus other).
All continuous variables were coded as is, to preserve predictive power, 9 with two exceptions. First, because of small cell sizes at the higher (better) end of the PPS scale, we recoded the PPS into six categories (0–10, 20, 30, 40, 50, and 60–100) based on previous studies. 7 Second, the presence of many uncommon diagnoses made it more efficient to group diagnoses into the top six categories (five diagnoses plus “other”).
We had decided a priori to exclude variables for which more than 10% of values were missing or out of range. However, rates for all variables under consideration were well below this limit (range: 0.02%–4.1%, interquartile range: 0.4%–3.2%). Therefore, we restricted prognostic models by using a casewise deletion of observations with missing variables.10–13
We developed a model and tested the resulting survival index in sequential steps.14,15 First, we created two independent samples of approximately equal size by selecting the two smaller hospices for model training (n=8496). We then used the remaining hospice for model validation (n=12,578). This is consistent with current recommendations of prognostic model development that suggest development and validation in distinct and independent populations.14,16 In the model development sample, we examined each of the variables in Table 1 as potential predictors of death within 7 days.17,18
Includes: patient's own home, friend/family member's home, assisted living, group home, prison, homeless shelter.
Because the number of variables under consideration was small, we used backwards selection to maximize the predictive value of the resulting model.15,19 For each version, we used bootstrapped confidence intervals to provide internal validation of the c-statistic. 20 We reasoned that an index with as few variables as possible would be most clinically useful, so we restricted candidate models to those in which all variables were significant. From these, we selected the final model based on its c-statistic and optimal Bayes Information Criterion (BIC), a modification of the Akaike Information Criterion.15,21
Next, we repeated the steps above using multiple imputation to assess the impact of missing data on the model's fit. We then compared the imputed model derived from the entire training sample with the model derived from the subset of patients with complete data. To assess the model's optimism, or potential for inflated estimates of accuracy, we tested the final model in the validation sample, again using bootstrapped confidence intervals. We then compared the original and validation c-statistics. In general, a large reduction in the c-statistic in the validation sample suggests that its predictions may not be generalizable.
For each patient in the validation sample with complete data, we calculated a prognostic score corresponding to the absolute value of the sum of his/her coefficients (the prognostic index). We assessed the index's predictive validity in three ways. First, we calculated the predicted and actual one-week mortality rates for each index level. Second, we selected the index score with the best c-statistic and calculated its sensitivity, specificity, and positive and negative predictive values. Third, to examine the index's calibration, we used the binomial test of proportions to examine differences in the number of observed versus expected deaths among the index groups.
Finally, we examined the accuracy of nurses' predictions of a patient's survival. To do this, we used responses to a question embedded in the EHR admission form that asks the admitting nurse whether death is “imminent.” Although this term is not defined in the EHR, we reasoned that it would offer a good assessment of a patient's short-term prognosis. We calculated a c-statistic and test characteristics of nurses' predictions, and compared these predictions with actual results using the binomial test. We also tested a hybrid model that incorporated nurses' predictions and the prognostic index, comparing its accuracy with either alone.
We estimated that a training sample of at least 8000 patients with a one-week mortality rate of 25% would provide adequate power to detect a 5% increase (e.g., between 25% and 30%) in the risk of one-week mortality attributable to a single variable with a prevalence of 10% (α=0.05). Although no standards exist for the estimation of necessary sample sizes for logistic regression models, a rough rule of thumb is to allow at least 10 events for each variable under consideration.22,23 Therefore, this sample would provide adequate power to detect even small effects of a candidate predictor variable on the risk of one-week mortality.
The University of Pennsylvania's Institutional Review Board approved the use of secondary data for this study. Stata statistical software (Stata MP2 11.0 for Mac, StataCorp., College Station, TX) was used for all statistical analysis.
Results
A total of 21,074 patients were admitted between October 1, 2008 and May 31, 2011. Hospices contributed 4475 patients (Lancaster), 4021 patients (Madison), and 12,578 patients (Clearwater). Of all patients admitted during the study period, 5562 (26.4%) died within 7 days. An additional 122 patients who were discharged alive from hospice in the first 7 days were assumed to have lived at least 7 days. Seven-day mortality rates were similar across the three hospices (Lancaster 27.3%, Madison 25.4%, Clearwater 26.4%).
The characteristics of the sample are described in Table 1. Compared with national descriptions of the U.S. hospice population, 2 this sample had a slightly longer length of stay (median 25 days versus 21 days), and patients in this sample were less likely to die within 7 days (26.4% versus 33%). This sample resembled the national hospice population with respect to gender (56% versus 54% female), the proportion with cancer (37% versus 41%), and age (84% versus 83% >65). The sample had a higher prevalence of white patients compared with the national hospice population (96% versus 80%), but was typical of the populations of these three regions.
Model development and validation
Of 21,074 patients, 8496 (40.3%) were from the hospices used for model training, and 12,578 (59.7%) were from the remaining hospice used for model validation. One-week mortality rates were identical in the development (2242/8496; 26.4%) and validation samples (9258/12,578; 26.4%).
Most patients (19,867; 94.3%) had a PPS score assessed within 24 hours of admission, but 345 (1.6%) had a score assessed after 24 hours, and 862 (4.1%) never had a score documented. Patients without a PPS score were more likely to die within 7 days (289/862; 33.5%) compared with patients with an early PPS score (5265/19,867; 26.5%). Conversely, patients with a late PPS score were less likely to die within 7 days (8/345; 2.3%). Because these late PPS patients necessarily had a longer survival, we excluded them from analysis.
Backwards selection using all variables in Table 1 generated an optimal model that was chosen based on the inclusion of only significant variables and an optimum c-statistic (0.83) and BIC (6427.47). Variables included in the model are displayed in Table 2. This model was based on 7425 patients in the development sample with complete data, out of a possible 8496 (87%). We followed the same process using multiple imputation, but because the resulting model was virtually identical, the nonimputed (casewise deletion) model was used. The final model was run again with bootstrapping on the validation sample with a c-statistic of 0.84, compared with a c-statistic in the development sample of 0.83, indicating no optimism of the original model.
Next, we calculated a survival index for each patient in the training sample, using the sum of the absolute value of each patient's coefficients. For instance, a female patient admitted from home with a PPS score of 20 had a score of 1.93 (1.02 [PPS score]+0.48 [home]+0.43 [gender]), which would be rounded to an index score of 2. The resulting scores ranged from 0 to 5 and higher numbers corresponded to a better prognosis (lower probability of 7-day mortality). The index's overall c-statistic was 0.86, and the best c-statistic was for an index <3 (0.75). This was considerably better than the most informative variable in the model (PPS score), which had a c-statistic of only 0.74 overall, and 0.67 for a score of <40.
We then calculated an index for each patient in the validation sample. The index had a c-statistic in the validation sample of 0.84 for those with complete data (n=12,442). As in the development sample, the optimal c-statistic corresponded to a score of <3 (0.78). This threshold had a sensitivity of 74.4%, a specificity of 80.6%, and positive and negative predictive values of 57.9% and 89.8%, respectively.
Because diagnosis was not included in the final model, we examined the index's performance across the most common diagnoses (Table 1). The c-statistic was highest for pulmonary disease (0.89). Other diagnoses had lower c-statistics, including cardiovascular disease (0.85), debility (0.85), stroke (0.82), cancer (0.80), and dementia (0.80).
Nurses' predictions
Of 12,578 patients in the validation sample, nurses predicted an imminent death for 2074 (16.5%). The c-statistic for predictions of death at various time periods between one and 10 days ranged from 0.65 (one day), 0.64 (3 days), 0.72 (7 days), 0.63 (10 days), and 0.61 (14 days). Patients for whom death was believed to be imminent were much more likely to die within 7 days (1617/2074 versus 1703/10,504; 78.0% versus 16.2%; odds ratio: 18.28; 95% confidence interval: 16.3–20.5; p<0.001). A prediction of imminent death had a c-statistic of 0.72, a sensitivity of 48.7%, a specificity of 95.1%, and positive and negative predictive values of 78.0% and 83.8%, respectively. Thus compared with the model's predictions at a threshold of <3, nurses' predictions offered a lower sensitivity (47.8% versus 74.4%) but a higher specificity (95.1% versus 80.6%) and overall worse c-statistic (0.72 versus 0.78).
Compared with nurses' predictions, the index offered better calibration across all index levels (Fig. 1). For a score of 5, the 7-day mortality (45/1242; 3.6%) was greater than nurses' predictions (1.3%) but less than the index's prediction (4.6%). For lower (worse) index scores, nurses' predictions increasingly underestimated the probability of 7-day mortality. For instance, for patients with an index of 4 (261/3351; 7.8%), the predicted mortality was 10.5% but nurses' was 2.3%. For an index score of 3 (533/3634; 14.7%), the predicted mortality was 19.7% but nurses' was 4.7%. For an index score of 2 (684/1867; 36.6%), the predicted mortality was 36.8% and nurses' was 17.9%. For patients with an index score of 1 (843/1241; 67.9%) the predicted mortality (63.1%) was higher than the nurses' predicted 49.0% mortality. Finally, for patients with an index score of 0 (912/1107; 82.4%), the predicted mortality (81.4%) was also higher than the nurses' predictions (75.4%).
Seven-day mortality of hospice patients in the validation sample (n=12,578), by index score (0–5).
Because the model's and nurses' predictions offered greater sensitivity and specificity, respectively, we developed a hybrid index that incorporated both. We created five versions by adding between 1 and 10 points if death was not coded as being imminent. In the validation sample, none of these hybrid models offered a c-statistic that was greater than that of the original index. The optimal index was obtained when an additional 5 points were assigned when death was not noted to be imminent. This index had a c-statistic of 0.84, which was the same as that of the original index.
Discussion
Despite rapid growth in the hospice industry, patients still enroll very late in the course of illness. Therefore, hospices and patients' health care providers will need to put systems in place that allow them to meet the needs of patients who enroll in the last days of life. This study provides two key findings that will guide these efforts.
First, this study found that hospice nurses are reasonably accurate in predicting 7-day mortality. This is unexpected because previous studies have found that health care providers are poor at predicting prognosis,24,25 and even palliative care providers tend to be optimistic. 26 However, these results are more in line with reports of providers working in specialized settings such as the intensive care unit (ICU), where predictions may be more accurate.27,28 It is noteworthy, though, that the hospice nurses in this study were subject to the same optimism bias that has been described elsewhere for physicians24,25 and that they consistently underestimated 7-day mortality.
Second, this study found that a prognostic model predicts 7-day mortality more accurately than nurses do. This is surprising because the nurses in this sample had access to a wealth of clinical information (e.g., blood pressure, respiratory rate, overall appearance) that was not included in the model. More importantly, the survival index appears to offer a useful stratification tool that could help hospices and providers to be able to identify those patients at highest risk of dying within days of hospice admission, so that the necessary services can be provided quickly. In addition, the variables included in the model described here are all available at the time of hospice admission. This will make it possible for EHR-based decision support systems to calculate a patient's risk of 7-day mortality at the time of admission, creating a flag for hospice clinicians and referring physicians.
This study has three limitations that should be noted. First, although the development and validation samples described here were distinct, the hospice programs from which they were drawn are somewhat different than other hospices in the United States. For instance, they are larger than most, 2 they use an EHR system, and they have a low prevalence of nonwhite patients. Therefore, further research is needed to determine how well this index predicts survival in a broader sample of hospices.
Second, nurses' predictions were based on their assessments that death was “imminent,” not that a patient would die within a week. It is possible that if this question had been asked directly, nurses' responses might have more closely resembled actual mortality rates. However, the current wording of the question provided the best c-statistic for predictions of death at one week, and therefore this timeframe is a reasonable test of nurses' predictions.
Third, we did not have information about the nurses whose predictions are reported here. For instance, we were unable to determine whether some were more accurate than others, or whether accuracy was associated with nurses' characteristics (e.g., experience, training). Future research is needed to define variation in accuracy among nurses, and to identify characteristics that are associated with more accurate predictions.
As the hospice industry grows and evolves, it will be essential to meet the needs of the patients who are referred very late. There may be opportunities to promote earlier referrals to hospice, 29 but gains are likely to be modest. Therefore, tools such as the index described here are needed to help health care providers meet patients' and families' needs.
Footnotes
Author Disclosure Statement
Three authors (TC, SF, and RB) have support from Suncoast Solutions Inc. for the submitted work. No other authors have had relationships with companies that might have an interest in the submitted work in the previous 3 years; no authors' spouses, partners, or children have financial relationships that may be relevant to the submitted work; and no authors have nonfinancial interests that may be relevant to the submitted work.
This study was funded by a National Institutes of Health grant (1KM1CA156715-01) to Dr. Casarett. The study funder had no role in the collection, analysis, and interpretation of data, in the writing of the report, or in the decision to submit the article for publication. The investigators maintained independence from funders throughout all stages of research. All authors, external and internal, had full access to all of the data (including statistical reports and tables) in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.