Abstract
Context:
Early palliative care can improve end-of-life outcomes, but referrals to palliative care specialists can be delayed in the primary care setting.
Objective:
This pilot study assessed the effect of a machine-learning algorithm on time to palliative care in a primary care population.
Methods:
Patients (aged ≥18 years) were eligible if they were empaneled with a primary care provider (PCP) from July 20, 2020, through May 30, 2021. The algorithm evaluated their health records and presented patients who were predicted to have the greatest need for palliative care. Records were then reviewed by palliative care specialists, and patients were randomized in a stepped-wedge fashion to have a referral notification sent to their PCPs if unmet palliative care needs were verified. Time-to-event outcomes were evaluated with Poisson regression models.
Results:
Of the 127,080 patients evaluated, 934 had their health records presented for review. Some patients were repeatedly presented by the algorithm (total presentations: 1592). In the intervention arm, PCPs were prompted to order a palliative care consultation for 142 patients. The time to 0.1% of the population receiving a palliative care consultation was 60.9 days for the intervention arm versus 71.8 days for the control arm (probability of a shorter time with the intervention, 0.88).
Conclusion:
A machine-learning algorithm to identify palliative care needs was successfully integrated into a primary care practice. More work is needed to improve the workflow.
Introduction
Palliative care improves quality of life for patients and caregivers and improves health care utilization metrics.1,2 A recent, large clinical trial of hospitalized patients showed that an artificial intelligence/machine-learning decision-support tool for predicting palliative care needs increased inpatient palliative care consultations and reduced hospital readmissions. 3 Studies of palliative care often focus on hospital settings or subspecialty populations4,5 or on prognostication or mortality outcomes.6–8 Consequently, less is known about the benefit of decision-support tools in primary care settings. A predictive model for palliative care is needed in the context of primary care.
Primary care providers (PCPs) are often best placed to identify patients who would benefit from palliative care. Patients with complex disease trajectories and/or multiple comorbid conditions are commonly seen by PCPs, and because the PCP typically has an established relationship with the patient, they are usually contacted first when symptoms or care needs arise. 9 Timing of the palliative care consultation is important, with earlier interventions being associated with better end-of-life outcomes, 10 but palliative care often is delayed. Reasons for these delays at the patient and clinician level include limited knowledge about palliative care, uncertainty about when to refer the patient, cultural beliefs, and lack of access.7–9,11
Given the potentially large numbers of patients in primary care populations, large-scale efforts are needed to identify patients who would benefit most from palliative care. A machine-learning algorithm may be able to compile data at a population level to help identify community patients living with serious illness who have unmet palliative care needs, including symptom management and advance care planning, that are otherwise not being addressed. In this trial, we applied an existing machine-learning algorithm to a primary care population to identify patients who might benefit from palliative care.
Methods
All study activities were approved by the Mayo Clinic Institutional Review Board (protocols 20-005977 and 21-007274). In accordance with Minnesota law, patients who did not authorize review of their medical records for general research studies were excluded from the analysis.
Study design
This study was conducted at Mayo Clinic (Rochester, Minnesota). Mayo Clinic provides primary care to community members at 5 locations. Within the adult primary care practice are 42 care teams nested within 9 care-team clusters.
We conducted a stepped-wedge pragmatic randomized clinical trial that piloted implementation of a machine-learning algorithm to identify primary care patients with unmet palliative care needs. This pilot study was conducted to assess the feasibility and application process within our primary care population. The stepped-wedge design enabled gradual addition of care teams to the intervention arm, with the first wedge having all units on the control arm and the last wedge having all units on the treatment arm.
The study protocol, including the setting, algorithm development, trial design, outcome measures, and data analysis plan, has been described previously. 12 Briefly, adult patients (≥18 years) were eligible if they were empaneled with a PCP from July 20, 2020, through May 30, 2021. We excluded patients who had a palliative care consultation within the previous 75 days, were current hospice enrollees, or lived in a skilled nursing facility (which uses a different model of primary care). The machine-learning algorithm was developed and trained on our primary care population using historical data, as previously outlined. 12 In this model, covariates included age, sex, laboratory test values, diagnosis codes, clinical note metadata, opioid use, pain scores, prior care use, and time since last appointment. 12 These covariates were included in the machine-learning algorithm but were not part of the statistical analysis to determine the efficacy of the intervention. We rolled out use of the algorithm in a randomized (computer-generated), 7-cluster, stepped-wedge design matrix every 42 days. Hence, the nine care-team clusters were randomized into seven wedges, with several clusters grouped together, based on the distribution of predicted palliative scores that was identified in historical data during algorithm development. 12
During the pilot, we ran the algorithm on a weekly basis to identify patients with the highest predicted need for palliative care. Each week, the 40 patients with the highest palliative scores were presented for review (in the control and treatment arms). Their electronic health records (EHRs) were reviewed by one of two board-certified palliative medicine specialists to determine the appropriateness of a palliative care consultation. If a consultation was appropriate and the patient was in the treatment arm, then the intervention was implemented. The palliative care specialist sent a standardized in-basket message to the PCP (and oncologist, if appropriate) recommending an order for a palliative care consultation. Control patients who were deemed appropriate for palliative care referral did not have the intervention but still may have palliative care through usual care pathways (typically PCP or specialist referral). Therefore, the patient’s status would not be blinded because the clinician would be aware of whether the patient was in the control or treatment arm.
The palliative care specialist could delay (“snooze”) an EHR review for either 7 days (e.g., if the patient was hospitalized) or 90 days (e.g., if the patient had a recent goals-of-care conversation or if symptoms were already well managed). If patients continued to meet the algorithm’s identification criteria, then they were presented again at the later date. Patients could be excluded from being presented if the EHR showed that they had moved, were enrolled in hospice, or were in the palliative care homebound program.
Outcomes
The primary outcome was time to palliative care consultation, measured as time to record an outpatient palliative care consult in the EHR. We modeled the population rate of palliative care (how many consults or orders out of the population), the inverse of which is time to palliative care and easier to conceptualize. We selected an arbitrary threshold of time to 0.1% of the population having the consultation ordered to give reasonably scaled numbers and to put the results into context for the reader (i.e., to show whether any change was big or small). Secondary outcomes were number of advance care planning notes documented in the EHR, number of International Statistical Classification of Diseases, Tenth Revision (ICD-10) diagnosis codes for palliative care, positive predictive value (PPV) of screening (i.e., expressing the percentage of patients presented to the clinician for review who were accepted), and performance metrics regarding the palliative care specialist and PCP handoff, including time to placement of the referral. 12 Both the uptake of the advanced care planning note and use of the diagnostic code were considered proxies for palliative care engagement by the provider.
To assess the PCPs’ satisfaction with the process, we surveyed those who had received alerts recommending a palliative care referral during the pilot. Specifically, we asked them to rate the helpfulness of the alert and to describe what about it was helpful. If a consultation was not ordered, we asked them to indicate why. We asked them to rate the likelihood that the alert enabled them to address their patient’s palliative care needs earlier than they might have otherwise and to rate their satisfaction with the alert system. We provided free-text areas for comments and recommendations.
Analysis
To assess the various time-to-event outcomes in the stepped-wedge trial, we followed the analysis plan previously described. 12 This method models the time to event by using a heterogeneous Poisson process, with adjustments to the event rate due to time and unit clustering. The time to event was estimated using the Poisson rate for each unit at the end of the study period. Posterior probabilities were based on 24,000 realizations sampled from the model’s posterior distribution. Credible intervals are reported using the 2.5 and 97.5 percentiles.
Results
Of 127,080 potentially eligible patients, 9031 were excluded because they declined research authorization. The algorithm was applied to 118,049 patients; of these, 1592 cases (934 unique patients [0.8%]) were presented and reviewed by a palliative care specialist (Fig. 1). The mean (SD) age of patients at first presentation by the algorithm was 71.8 (15.4) years (range, 19.5–98.6 years); 509 (54.5%) of patients were male and 425 (45.5%) were female. Most were self-described as White (91.2%) and not Hispanic or Latino (97.1%).
Consolidated Standards of Reporting Trials (CONSORT) Diagram. The diagram is a simplified representation of the allocation process because the statistical analysis accounted for multiple presentations of a given patient. In addition, the control and intervention arms were more balanced than the diagram suggests.
Of the 934 patients presented, 230 (24.6%) were accepted at least once, 631 (67.6%) were snoozed, 72 (7.7%) were excluded, and 1 (0.1%) was never reviewed (lack of provider time). The median (range) number of times that a patient was presented for review was 1 (1–7). In total, 142 patients (15.2%) were deemed appropriate for a palliative care consultation, and a message was sent to their PCPs (intervention). Consultations were ordered for 72 patients (50.7%), and 37 (26.1%) completed the consultation during the pilot. Of all patients presented by the algorithm (control and intervention arms), 112 (12.0%) had a palliative care consultation within 12 weeks of the patient’s presentation in the app.
For the primary outcome of time to a palliative care consultation, the median (95% credible interval) time to 0.1% of the primary care adult population receiving the consultation was 60.9 (47.8–79.4) days for the intervention, as compared with 71.8 (50.9–103.8) days for the control (posterior probability of a shorter time with the intervention, 0.88; Fig. 2). The median (95% credible interval) time to 0.1% of the population having the consultation ordered was 42.8 (34.5–54.2) days for the intervention arm and 63.9 (46.8–90.1) days for the control arm (posterior probability of a shorter time with the intervention, > 0.99; Fig. 3). Time to generation of an advance care planning note was not significantly changed, with a median intervention effect of 1.04 (95% credible interval, 0.81–1.35; posterior probability of a shorter time with the intervention, 0.64). Time to recording a palliative care diagnosis (ICD-10 Z51.5) was unchanged, with a median intervention effect of 1.03 (95% credible interval, 0.87–1.24; posterior probability of a shorter time with the intervention, 0.64).
Palliative Care Consultations. The figure shows the incidence rate for completed consultations. Solid lines indicate the median value for each treatment arm; shaded areas indicate the 95% credible interval (2.5 and 97.5 percentiles).
Palliative Care Referral Orders. The figure shows the incidence rate of consultations ordered. Solid lines indicate the median value for each treatment arm; shaded areas indicate the 95% credible interval (2.5 and 97.5 percentiles).
Algorithm performance
As noted by Heinzen et al., 12 the algorithm’s cross-validated concordance in the training set was 0.94. In this trial, the algorithm had a PPV of 0.18, with the palliative care specialist accepting 273/1512 cases presented across both arms.
PCP survey results
Surveys were completed by 28 of 92 PCPs (response rate, 30%). Most respondents (n = 21 [75%]) found the alert helpful: 4 (14%) rated it as “extremely” helpful, 7 (25%) as “very” helpful, and 10 (36%) as “moderately” helpful. Six respondents (21%) rated the alert as “slightly” helpful, and 1 (4%) rated it as “not at all” helpful (this respondent did not remember receiving an alert). Several free-text comments described how the in-basket message reminded them to consider palliative care for their patient. One PCP commented, “It helped me recognize a palliative patient when I was not realizing it myself.” Others noted that it saved time by facilitating their review of the patient’s EHR to determine eligibility for palliative care. One indicated that some patients and families found it “more palatable for them to accept the referral” if they were told an algorithm had identified palliative care needs.
Survey respondents could choose from multiple reasons to explain why a palliative care referral ultimately was not sought. Most commonly, the patient was uninterested in palliative care (n = 10 [36%]). In some instances, alert timing did not align with patient care (n = 6 [21%]) or the PCP disagreed with the recommendation (n = 2 [7%]). Most reported that the alert led to the patient’s palliative care needs being addressed earlier: “extremely” likely, n = 10 (36%); and “somewhat” likely, n = 10 (36%). The remaining respondents (n = 8 [29%]) gave a “neutral” response.
When asked to provide general comments about their experience with the pilot, one PCP asked for training on how to introduce the concept of palliative care to patients still seeking active treatment, another noted the need for improved access to in-person palliative care visits, and another requested information on how their patients would benefit specifically from palliative care.
Discussion
Our trial applied an existing palliative care machine-learning algorithm 12 to a primary care population and had two important findings. First, we successfully integrated the use of this algorithm into an outpatient primary care practice and identified patients with unmet palliative care needs. Second, by alerting PCPs to their patients with unmet palliative care needs, we markedly reduced the time to placing an order for a specialist consultation. However, the intervention had a smaller effect on the time to the palliative care consultation, indicating that considerable barriers to timely consultations still existed. Two secondary outcomes, time to documentation of an advance care planning note and time to recording a palliative care ICD-10 diagnosis code, showed no change, but our institution does not consistently use the advance care planning note template or the palliative care diagnosis code, reflecting the challenges of examining clinical documentation practices in real time.
Our institution has previously implemented a palliative care algorithm in the inpatient setting. 3 We successfully adapted and expanded that approach in this outpatient pilot by integrating palliative care, primary care, and health informatics. Although we were able to successfully integrate the algorithm into our primary care practice, its performance in this patient population needs more refinement because the PPV was 0.18. We acknowledge that this PPV is low, but it does not necessarily mean that the selected patients were not in need of palliative care. Possibly, a specialist consultation was not required (e.g., the PCP was already adequately addressing needs) or the alert recommending a consultation was untimely (e.g., patients may have been hospitalized and did not have upcoming appointments scheduled); such factors may have influenced the level of agreement between the palliative care clinician and the algorithm.
To our knowledge, little has been reported to date about the use of machine-learning algorithms in primary care to identify palliative care needs that can be directly compared with this study’s findings. Nevertheless, we can apply what we know about barriers to palliative care to future modifications. Often, barriers include appointment scheduling challenges, such as requiring coordination with other visits or relying on the patient to schedule the visit. Such factors are unique to the outpatient setting and may have affected the time to receipt of a palliative care consultation. Other barriers pertain to the patients’ reluctance to receive palliative care and the possibility that they conflated palliative care with hospice care. For instance, Enguidanos et al. 11 explored health care professionals’ perceptions of patient barriers to palliative care; identified barriers included financial concerns, attitudes toward home visits, and cultural or language differences. Additionally, the clinicians noted challenges with the referral process, visit time constraints, and lack of knowledge about palliative care. A 2018 national survey indicated that only 28.8% of adults know about palliative care 13 and that misconceptions about palliative care were common, including equating palliative care with hospice enrollment or death and believing that palliative care meant stopping other treatments or “giving up.” The prevalence and scope of these misunderstandings is further validated by important results from our PCP survey. The most common reason for not referring patients was the lack of patient interest. We hypothesize that this lack of interest is likely multifactorial and may include the above-described barriers, including misconceptions about palliative care. Hence, ongoing work that improves public and clinician education about palliative care is important to successfully apply machine-learning algorithms such as ours to influence actual patient outcomes.
This study has strengths and limitations. To our knowledge, this study is the first to use a machine-learning algorithm in an outpatient primary care practice to identify patients with unmet palliative care needs. Because this tool requires stakeholder (i.e., PCP) involvement, we qualitatively assessed their opinions, which is important for future design and successful implementation in clinical practice. However, we acknowledge that, as a pilot, this study had a small sample from a single study site, which suggests that the findings may not be generalizable outside the Midwest. This study was also limited by many of the previously described disparities that may influence palliative care access, including race and ethnicity (our study cohort was predominantly White). The analysis did not account for competing events (e.g., death, hospice). Practically, we also recognize that this intervention as it currently stands requires time and clinical review, which may present challenges and limit application in its current form because it requires additional chart reviews in busy primary care practices. Finally, our study period overlapped with the COVID-19 pandemic. The pandemic-associated statewide restrictions may have influenced access to primary care clinic visits, willingness to attend medical appointments, and potential palliative care referrals.
Further work is needed to refine this tool and the referral process into a more streamlined and time-efficient procedure for the primary care setting, which may help overcome some of these limitations. Updates might include having a nurse care coordinator review EHRs to reduce or eliminate the need for clinician reviews; this approach has been successful in the inpatient setting at our institution. Additionally, integrating the machine-learning algorithm into the EHR could reduce administrative burden and facilitate more complex data collection, but the algorithm must be further refined to improve its PPV.
In conclusion, we successfully implemented this algorithm in our primary care practice to identify patients with unmet palliative care needs. Although further adjustments are needed to streamline the process, this tool has the potential to help identify the palliative care needs of our primary care patients living with serious illness.
Authors’ Contributions
M.M.B.: Investigation, resources, writing—original draft, writing—review and editing. J.C.K.: Writing—review and editing. R.M.W.: Writing—review and editing. E.P.H.: Data curation, formal analysis, methodology, writing—review and editing. P.M.W.: Data curation, formal analysis, writing—review and editing. G.O.D.: Data curation, formal analysis. S.W.A.: Data curation, formal analysis. C.B.S.: Data curation, formal analysis, supervision. R.D.H.: Conceptualization, data curation, investigation, supervision, writing—review and editing.
Footnotes
Data availability statement
The data contain confidential patient information and are not available for sharing.
Author Disclosure Statement
The authors declare no conflicts of interest. June Oshiro, PhD, ELS, Mayo Clinic, substantively edited the article. The Scientific Publications staff at Mayo Clinic provided proofreading, administrative, and clerical support.
Funding Information
Dr. Havyer’s effort was supported by the Mayo Clinic Department of Medicine Catalyst for Advancing in Academics grant. The authors also acknowledge the Mayo Clinic’s Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery.