Abstract
Background:
Early referral of cancer patients for palliative care significantly improves the quality of life. It is not clear which patients can benefit from an early referral, and when the referral should occur. A Delphi Panel study proposed 11 major criteria for an outpatient palliative care referral.
Objective:
To operationalize major Delphi criteria in a cohort of lung cancer patients, using a prospective approach, by linking health administrative data.
Design:
Population-based observational cohort study.
Setting/Subjects:
The study population comprised 38,851 cases of lung cancer in the Ontario Cancer Registry, diagnosed from January 1, 2012, to December 31, 2016.
Measurements:
We operationalized 6 of the 11 major criteria (4 diagnosis or prognosis based and 2 symptom based). Patients were considered eligible (index event) for palliative care if they qualified for any criterion. Among eligible patients, we identified those who received palliative care.
Results:
Twenty-eight thousand one hundred sixty-four patients were eligible for palliative care by qualifying for either the diagnosis- or prognosis-based criteria (n = 21,036, 76.5%), or for symptom-based criteria (n = 7128, 23.5%). A total of 23,199 (82.4%) patients received palliative care. The median time from palliative care eligibility to the receipt of first palliative care or death or maximum study follow-up was 56 days (range = 17–348).
Conclusions:
We operationalized six major criteria that identified the majority of lung cancer patients who were eligible for palliative care. Most eligible patients received the palliative care before death. Future research is warranted to test these criteria in other cancer populations.
Introduction
It is estimated that the number of cancer patients requiring palliative care will increase by 45% by the year 2040. 1 A timely referral for palliative care significantly improves quality of life2–5 and may also increase survival.3,4,6,7 Studies show that early referral of cancer patients for palliative care results in lesser use of ambulatory services as well as less aggressive end-of-life care.3,8 The American Society of Clinical Oncology (ASCO) guidelines recommend an early integration of specialist palliative care into the routine cancer care 9 ; however, both the identification of patients who could benefit most from an early referral and appropriate timing of referral have been subjects of debate.10–12
At present, there is a lack of universal criteria for palliative care referral among cancer patients. Hui et al. conducted a systematic review of 21 studies and identified 20 unique criteria for referral of cancer patients to an outpatient specialty palliative care clinic. 12 However, the authors reported significant heterogeneity between studies in regard to the choice of referral criteria and the timing of referral. To address this issue, Hui et al. conducted a Delphi study to reach a consensus on criteria that the panel members deemed most useful for outpatient palliative care referral. 13 The Delphi panel proposed 11 major criteria where panel members had a priori agreement of 70% or higher and 36 minor criteria, where the agreement was less than 70%. 13
The primary objective of this study was to operationalize the major Delphi criteria for palliative care referral prospectively in a cohort of lung cancer patients, using the population-based linked health care data. Our specific objectives were twofold: (1) to identify lung cancer patients who could benefit from palliative care and (2) to quantify the gap between those patients who could benefit from palliative care and those who receive it.
Methods
Study design, setting, and data sources
We conducted an observational population-based cohort study, using the linked provincial administrative health care datasets at our research institute, ICES, in Toronto, Ontario, Canada. These datasets were linked using unique encoded identifiers. The datasets were as follows: Ontario Cancer Registry (OCR), Discharge Abstract Database (DAD), National Ambulatory Care Reporting System (NACRS), Activity Level Reporting (ALR), New Drug Funding Program (NDFP), and Edmonton Symptom Assessment Scale (ESAS), Same Day Surgery (SDS), Immigration, Refugees and Citizenship Canada (IRCC)'s Permanent Resident Database, Ontario Health Insurance Plan (OHIP), Registered Persons Database (RPDB), and Ontario Registrar General—Death (ORGD). Further details are available at https://www.ices.on.ca/Data-and-Privacy/ICES-data/Data-dictionary
Ethical considerations
ICES is an independent, nonprofit research institute whose legal status under Ontario's health information privacy law allows it to collect and analyze health care and demographic data, without consent, for health system evaluation and improvement. The use of data in this project was authorized under section 45 of Ontario's Personal Health Information Protection Act, which does not require review by a Research Ethics Board (https://www.ices.on.ca/Data-and-Privacy).
Study population
From January 1, 2012, to December 31, 2016, we identified all lung cancer cases in the OCR, using the ICD-O-3 topography code C34 (International Classification of Diseases for Oncology). We excluded cases for the following reasons: missing information on a unique identifier number or sex; date of death before the date of lung cancer diagnosis; stage 0; and palliative care received at any time within two years before the date of lung cancer diagnosis.
For each patient in the study population, we retrieved information on the following: age at lung cancer diagnosis (years), sex (male, female), local health integration network area, immigration status (immigrants and nonimmigrants), time since immigration (0–5, 6–10, and >10 years), date of lung cancer diagnosis and stage at diagnosis, comorbidity using The Johns Hopkins ACG System® Aggregated Diagnosis Groups (ADGs), 14 and date of death (if they died before study end date December 31, 2017).
Palliative care eligibility (for determining the index date)
Using database linkage, we operationalized 6 of the 11 major Delphi criteria. 13 Four criteria were diagnosis based (or prognosis based), and two criteria were symptom based. We considered the date of palliative care eligibility as our main anchor point and defined it as the date when a patient qualified any of the six referral criteria; we called this date the index date. Figure 1 shows the steps to determine palliative care eligibility.

Steps to determine palliative care eligibility in lung cancer patients. The SEER-based median survival of patients with stages III and IV lung cancer was less than a year, and therefore, those patients contributed directly to the palliative care eligibility pool. SEER, Surveillance, Epidemiology, and End Results.
Diagnosis based (or prognosis based) criteria
Within three months of diagnosis of advanced or incurable cancer with a predicted median survival of one year or less: we used the Surveillance, Epidemiology, and End Results (SEER) 18 registries research database (November 2017 Submission) of the National Cancer Institute (NCI) to estimate the median survival with lung cancer. 15 In the SEER database, the median survival with stage III and IV lung cancer was less than a year, and therefore, we considered all stage III and IV lung cancer patients in our study cohort eligible for palliative care. The SEER-based median survival with stage I, II, and the unknown stage was greater than a year, and therefore, for all those patients in our study cohort, we used a Mortality Risk Score model. 16 The model takes into account the age, sex, and ADGs, and predicts survival of individuals from the date of cancer diagnosis. 16 Patients whose predicted survival time was one year or less contributed to the palliative care eligibility pool.
Diagnosis of advanced cancer with progressive disease despite second-line systemic chemotherapy: using the NDFP database, we identified patients who received the second-line systemic chemotherapy. The last date of second-line chemotherapy was considered the date when patients became eligible for palliative care due to this criterion.
Brain or leptomeningeal metastasis: we used ICD-10-CA codes (International Classification of Diseases, 10th Revision, Canada) to identify patients with brain or leptomeningeal metastasis at cancer diagnosis (Supplementary Table S1).
Spinal cord compression or cauda equina: we used a previously developed algorithm (Loblaw et al.) to define spinal cord compression or cauda equina (Supplementary Table S1). 17
Symptom-based criteria
Severe physical symptoms: we used the Edmonton Symptom Assessment System (ESAS) to measure the severity of symptoms. A score of 7–10 on a 10-point scale for pain, shortness of breath, nausea, drowsiness, lack of appetite, tiredness, or wellbeing was considered severe physical symptoms.
Severe emotional symptoms: when a score of 7–10 for anxiety or depression was reported on ESAS.
Patients who did not qualify for any of the eligibility criteria during the first iteration entered in the successive iterations for palliative care eligibility at six-monthly time intervals until the study end date (December 31, 2017).
Palliative care receipt
After establishing palliative care eligibility, we determined the receipt of first palliative care, starting from the date of palliative care eligibility (index date). We searched for ICD-10-CA codes indicating a hospitalization or an emergency department visit for palliative care, outpatient physician billing codes for palliative care, and any flags indicating palliative care receipt at home or hospice (Supplementary Table S1). We excluded patients who had a record of palliative care receipt anytime between cancer diagnosis and index date. Among individuals who were eligible for palliative care, we used the Kaplan-Meier method to determine median time from their index date to the receipt of first palliative care (the event of interest). We censored patients on their death date, or at study end date (December 31, 2017), whichever happened first.
Statistical analysis
We examined baseline characteristics of the study population using descriptive statistics. We calculated mean (standard deviation) and median (interquartile range [IQR]) to describe continuous variables; and frequencies and proportions for categorical variables. Among patients who were eligible for palliative care, we calculated the proportion that received care and described the type of palliative care service received.
Among patients who were eligible for palliative care, we used the cumulative incidence function (CIF) approach, starting from the index date, to calculate the cumulative incidence of receiving palliative care against the cumulative incidence of dying without receiving palliative care, where dying was considered a competing risk.
Due to the low competing probability of death in this cohort, as seen by the CIF curve, we implemented a cause-specific Cox proportional hazards regression model to estimate the hazard of receiving palliative care. We used univariable and multivariable regression models. The multivariable models included age, sex, region, income quintile, stage at diagnosis, the time between cancer diagnosis and palliative care eligibility, type of Delphi criteria, comorbidity, and immigration status. We then implemented a Cox regression model for examining time to overall mortality (that is death, with or without the receipt of palliative care).
We considered a p-value of less than 0.05 statistically significant, and all p-values were two tailed. The SAS Enterprise Guide 7.1 was used for analyses.
Results
Descriptive statistics
From January 1, 2012, to December 31, 2016, we identified 41,932 individuals in the OCR who were diagnosed with lung cancer. We excluded 36 cases (0.82%) due to missing information on sex or a unique identifier number, date of death before the date of diagnosis, or stage 0. We also excluded 3045 (7.26%) cases because they received palliative care at any time within two years before their cancer diagnosis. After exclusions, a total of 38,851 (92.6%) cases of lung cancer qualified for testing the Delphi criteria for palliative care eligibility.
Figure 2 shows the palliative care eligibility of 38,851 patients. A total of 21,610 (55.6%) patients had stage III or IV lung cancer, and those patients contributed directly to the palliative care eligibility pool by meeting the first eligibility criterion (median survival one year or less). Of the remaining 17,241 patients, 8758 (22.5%) became eligible by qualifying for one or more of the remaining criteria; 3694 (9.5%) patients died before becoming eligible for palliative care; and 4789 (12.3%) patients did not meet any criterion by the maximum study follow-up (December 31, 2017).

Palliative care eligibility in lung cancer patients, by months after cancer diagnosis.
In total, 30,368 (78.2%) cases were eligible for palliative care. Of those, we excluded 2204 (7.6%) cases who had received palliative care at any time between lung cancer diagnosis and palliative care eligibility (the index date), leaving 28,164 cases available for the final analyses.
Table 1 presents characteristics of 28,164 cases of lung cancer, who were eligible for palliative care. The mean age of patients at lung cancer diagnosis was 69.9 years (median = 70; IQR = 63–78). The mean time from lung cancer diagnosis to palliative care eligibility was 1.45 months (median = 0; IQR = 0–0). Of patients with staging data, stage IV cancer was the largest group (n = 15,105; 53.6%), followed by stage III (n = 6290; 22.3%).
Characteristics of Lung Cancer Cases Who Were Eligible for Palliative Care (N = 28,164)
IQR, interquartile range; SD, standard deviation.
A total of 23,199 (82.4%) of 28,164 patients received palliative care by the maximum study follow-up time; 3546 (12.6%) patients did not receive palliative care and were alive; and 1419 (5.04%) patients did not receive palliative care and were dead. Among palliative care-eligible patients (n = 28,164), the median time from eligibility to the event (i.e., first receipt of palliative care, death, or maximum follow-up) was 56 days (IQR = 17–348).
In total, 15,089 (65%) of 23,199 patients received palliative care through outpatient physician service (indicated by physician billing codes); 25% (n = 5799) received care at home; 9.5% (n = 2193) received care through inpatient or emergency services; and 0.4% (n = 98) received palliative care at a long-term or chronic care facility. The median time from receipt of first palliative care to death was 80 days (IQR = 22–225).
CIF curves and regression models
The probability of receiving palliative care within one year after the date of palliative care eligibility was 72.1%, whereas the risk of dying without receiving palliative care within one year was 3.5% (Fig. 3).

Cumulative incidence of receiving palliative care first versus dying first among palliative care eligible lung cancer patients (N = 28,164).
Table 2 presents the multivariable cause-specific hazard ratio (HR) of receiving palliative care. Comparing diagnosis- or prognosis-based criteria (referent), patients qualifying for the symptom-based criteria were more likely to receive palliative care (HR = 0.44; 95% confidence interval [CI] 0.40–0.48). Comparing the highest income quintile group (referent), patients in the lowest income quintile group were more likely to receive palliative care (HR = 1.13; 95% CI 1.08–1.18). Comparing nonimmigrants (referent), immigrants were less likely to receive palliative care (HR = 0.84; 95% CI 0.79–0.89). Comparing Toronto Central region (referent), patients in the Waterloo Wellington region were more likely to receive palliative care (HR = 1.39; 95% CI 1.28–1.50).
Multivariable Cause-Specific Cox Proportional Hazards Regression Model of Receiving Palliative Care, among Patients Who Were Eligible for Palliative Care (N = 28,164)
Values in bold are statistically significant (p < 0.05).
CI, confidence interval.
Discussion
Key results
In this population-based cohort study of lung cancer patients, we operationalized six major Delphi criteria to determine palliative care eligibility. We used patient-specific data and determined palliative care eligibility in a forward-looking manner, instead of creating a decedent cohort and looking backward, which allowed us to create a more meaningful denominator. We observed that Delphi criteria identified a majority of lung cancer patients who could benefit from a palliative approach. Furthermore, among patients who were eligible for palliative care, most received palliative care before death. An advanced cancer stage at diagnosis, presence of diagnosis- or prognosis-based criteria, and living in a lower-income area were associated with a higher likelihood of receiving palliative care first rather than dying first.
Interpretation
There is a consensus that palliative care is most beneficial for patients when offered early-on and alongside the standard treatment.18,19 Most patients in our study population were identified by Delphi criteria for palliative care referral early in the course of their disease. For example, 75% of total palliative care-eligible patients were picked up by Delphi criteria at the time of cancer diagnosis because those patients had advanced cancers; an additional 20% were picked up within one year of diagnosis. Furthermore, the majority (76%) of patients qualified for one or more prognosis-based criteria. These observations imply that an acute trajectory of lung cancer patients, indicated by advanced cancer stage at diagnosis and a shorter median overall survival, maybe a leading factor in driving the efficiency of Delphi criteria. These criteria may be useful as an additional tool for early planning and subsequent referral for palliative care in lung cancer patients. Further research should be directed at testing these criteria in other cancer populations with different disease trajectories.
Equitable access to palliative care is an essential measure of cancer care system performance. 20 In our study population, most patients who were eligible for palliative care received the care before death; however, we noted sociodemographic differences in the receipt of palliative care. For example, when compared with the Toronto Central region, patients from North West region were less likely, and from the Waterloo Wellington region were more likely to receive palliative care. To some extent, these regional variations may be due to differences in available resources or difficulties in providing services to patients spread out over a wide geographic area. We also observed that patients living in low-income neighborhoods were more likely than patients in high-income neighborhoods to receive palliative care before death. This finding was unexpected and is in contrast with other studies from the United Kingdom and United States, which reported that low-income patients were less likely to have access to palliative care.21–23 An earlier study from Ontario also reported that low-income patients were less likely to receive palliative care; however, the study population included all decedent in Ontario. 24 We also observed that patients who were immigrants were less likely than nonimmigrants to receive palliative care before dying. These observations are consistent with reports from the Canadian Institute for Health Information 20 and the Canadian Partnership Against Cancer, 25 which highlighted inequalities in the access to palliative care. When compared to these reports, the extent of inequity was lesser in our study population.
Limitations and strengths
Our study has some limitations. First, we were unable to operationalize 5 (of the total of 11) major criteria because the required information was not available in the health care databases. The criteria that we could not operationalize were (1) request for hastened death, (2) spiritual or existential crisis, (3) assistance with decision making or care planning, (4) patient request, and (5) delirium. Second, we used a mortality risk score model to predict survival times of patients in the study cohort. While the model has been tested and validated in general population-based cohorts, its performance in the disease-specific cohorts, such as lung cancer, has not been evaluated. Third, we determined the receipt of palliative care by the presence of codes indicating either physician or nursing care. However, these codes do not indicate the quantity or quality of palliative care. Fourth, underreporting of palliative care is a possibility with our data sources, and therefore, we may not have captured all patients who received palliative care. However, we used extensive codes to look for palliative care receipt, and we feel that the magnitude of missing information, if any, would be small.
The major strength of our study is the robustness of our population-based data, which is linked at the individual level and includes information on all patients and hospitals in Ontario. Therefore, the results of our study can be generalized to lung cancer patients outside of Ontario. Unlike previous studies that used decedent cohorts and looked backwards to determine palliative care need,1,26 we determined palliative care eligibility in a forward-looking manner. To our knowledge, this is the first study that operationalized Delphi criteria for palliative care referral, using population-based health data.
In conclusion, we operationalized six major Delphi criteria, which identified the majority of lung cancer patients who could benefit from palliative care; most patients who were deemed eligible received the care before death. It is imperative to expand the early provision of palliative care to a larger population of cancer patients. The ability to estimate the number of patients who may benefit from a palliative approach would be a powerful planning tool, given the anticipated increasing need. Further research is warranted to test these criteria in other patient populations. Using a cancer-specific model may be a valuable tool to estimate the number of patients who might benefit from a palliative care approach to care.
Footnotes
Acknowledgments
Parts of this material are based on data and information provided by Cancer Care Ontario (CCO). The opinions, results, view, and conclusions reported in this article are those of the authors and do not necessarily reflect those of CCO. No endorsement by CCO is intended or should be inferred.
Parts of this material are based on data and/or information compiled and provided by CIHI. However, the analyses, conclusions, opinions, and statements expressed in the material are those of the author(s), and not necessarily those of CIHI.
We acknowledge IRCC (Immigration, Refugees and Citizenship Canada) for permission to use their database.
Funding Information
This study was conducted with the support of the Ontario Institute for Cancer Research through funding provided by the Government of Ontario. This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The opinions, results, and conclusions reported in this article are those of the authors and are independent of funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred.
Author Disclosure Statement
No competing financial interests exist.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
