Using Electronic Health Records for Quality Measurement and Accountability in Care of the Seriously Ill: Opportunities and Challenges

Abstract

Background:

As our population ages and the burden of chronic illness rises, there is increasing need to implement quality metrics that measure and benchmark care of the seriously ill, including the delivery of both primary care and specialty palliative care. Such metrics can be used to drive quality improvement, value-based payment, and accountability for population-based outcomes.

Methods:

In this article, we examine use of the electronic health record (EHR) as a tool to assess quality of serious illness care through narrative review and description of a palliative care quality metrics program in a large healthcare system.

Results:

In the search for feasible, reliable, and valid palliative care quality metrics, the EHR is an attractive option for collecting quality data on large numbers of seriously ill patients. However, important challenges to using EHR data for quality improvement and accountability exist, including understanding the validity, reliability, and completeness of the data, as well as acknowledging the difference between care documented and care delivered. Challenges also include developing achievable metrics that are clearly linked to patient and family outcomes and addressing data interoperability across sites as well as EHR platforms and vendors. This article summarizes the strengths and weakness of the EHR as a data source for accountability of community- and population-based programs for serious illness, describes the implementation of EHR data in the palliative care quality metrics program at the University of Washington, and, based on that experience, discusses opportunities and challenges. Our palliative care metrics program was designed to serve as a resource for other healthcare systems.

Discussion:

Although the EHR offers great promise for enhancing quality of care provided for the seriously ill, significant challenges remain to operationalizing this promise on a national scale and using EHR data for population-based quality and accountability.

The field of palliative care has grown substantially over the past three decades.¹ High-quality specialty palliative care has resulted in improved quality of life and symptoms for patients with serious illness and their families, as well as decreased healthcare costs attributed to lower intensity treatments at the end of life—treatments that match patients' goals of care.^2–5 Despite these successes, effective implementation of high-quality primary and specialty palliative care has been challenging.⁶ Some well-designed palliative care interventions have not resulted in significant improvements in patient or family outcomes and, in some cases, have been associated with worsened outcomes.^7–9 Given the increasing interest in developing interventions, payment plans, and policies that enhance palliative care, it is important to document that such interventions, plans, and policies improve quality of care without unintended consequences.¹⁰ There is also increasing interest in ensuring high-quality, population-based care for seriously ill patients and their families across settings, including hospital, clinical, long-term care facility, and home, and using quality measurement to ensure accountability for this care.^11–13 In this context, it is critical that we develop and validate quality metrics that facilitate implementation and evaluation of clinical and policy programs that improve palliative care and enhance accountability for serious illness care across diverse settings.

Quality metrics for palliative care have been developed and implemented for specialty palliative care programs, primarily using data entry completed by palliative care specialists during their clinical work.^14–18 Although these programs have tremendous potential, they are limited to the minority of seriously ill patients seen by palliative care specialists and sometimes require that busy clinicians input quality data in addition to their clinical workload. Quality metrics for palliative care have also been developed and implemented in the Veterans Health Administration, but require manual chart abstraction and may be challenging to scale to other healthcare systems.¹⁹

In the search for feasible, reliable, and valid palliative care quality metrics, automated methods of using the electronic health record (EHR) are an attractive option. However, the EHR has often not delivered on its potential.²⁰ In this article, we identify strengths and weaknesses of using EHR data for quality metrics for serious illness care and review our experience developing and implementing an EHR-based quality metrics program for primary and specialty palliative care, that is, care for seriously ill patients and their families delivered by all clinicians in the healthcare system (primary) as well as palliative care specialists (specialty). We also describe the key domains and individual metrics that we focused on after feedback from diverse stakeholders, including patients, families, clinicians, administrators, quality improvement staff, and information technology experts. Finally, we describe lessons learned through this program and identify opportunities and challenges for moving this work forward.

Strengths and Weaknesses of the EHR for Palliative Care Quality Metrics

As the organized repository for information about both intended healthcare and actual healthcare, the EHR has strengths that are important to understand and capitalize on. The EHR is a comprehensive record of clinical and administrative data on all individuals cared for within our healthcare systems. As such, it offers a repository with sufficient size and scope to support analyses of detailed clinical care and assessments of important subgroups of patients, such as those with serious illness, including “high-cost, high-need” patients who particularly benefit from palliative care.^13,21–23 In addition, EHR data are recorded directly from documentation of healthcare delivery and utilization without requiring additional data entry beyond that accompanying routine care. The EHR also captures social determinants of health, which are important to the delivery of high-quality, accountable care for the seriously ill.²⁴

Alongside these strengths, the EHR has important limitations. First, information obtained from the EHR reflects care documented rather than care delivered or care perceived by patients and families. Goals-of-care discussions offer a good example of this potential mismatch: clinicians may report having these conversations with patients, but either fail to document them or document them in ways that are not easily retrievable.^25,26 Conversely, aspects of care may be documented in the EHR but not actually delivered; the frequency of duplicative documentation by “copy-and-paste” makes it difficult to determine how many goals-of-care conversations really occurred.^27,28

Second, the diverse range of purposes for which EHR data are collected may compromise its validity for use in quality and accountability. Purposes include documentation of care delivered, communication between clinicians, billing, care coordination, quality improvement, and administrative reporting. Each of these purposes introduces idiosyncrasies in the data that may undermine reliability and validity for quality metrics. One example is the widespread use of flowsheets documenting that patients were asked about advance directives, as mandated by the Patient Self-Determination Act.²⁹ At the University of Washington, these forms were often erroneously uploaded into the same data field that records actual advance directives, making it difficult to distinguish what the record contains. Since the EHR was not designed for assessment of palliative care quality metrics, such metrics need to be adapted to the information available and then psychometric properties need to be validated. Alternatively, modifications to the EHR may help capture data relevant to high-quality palliative care.

A third limitation of the EHR is the absence of key outcomes essential to high-quality serious illness care. The EHR generally lacks systematic, structured assessment of outcomes such as patients' and families' experiences of care, physical and psychological symptoms, quality of life, spiritual needs, or functional status. As these represent essential domains of serious illness care, any EHR-based metrics program must be supplemented by patient and family reports. Novel approaches to include systematic assessment of patient-reported outcomes in the EHR are in development and offer promise for addressing this shortcoming.^30,31

Fourth, the lack of interoperability between EHRs from different healthcare organizations and different vendors creates challenges to developing quality metrics that span multiple EHRs and can be used to assess population-based care or accountability.³² Efforts to unify data from different EHR systems are hampered by both technical challenges in aggregating data from multiple platforms and idiosyncrasies in how the same data are stored in different EHRs.^32,33 Without access to data with consistent meaning, metrics from different healthcare systems are difficult to compare.¹² Although standards and incentive programs have improved the utility of EHR data for population health, substantial disincentives to the creation of interoperable metrics remain, including incentives to sell and individualize separate modules for particular metrics (e.g., advance care planning) and incentives to allow platforms to import, but not export, data to other systems.²⁰

Finally, modifications to a healthcare system's EHR may pose ongoing challenges for the collection of quality metrics. Updates to an EHR system, or a change in vendor, may change the way data are collected or represented, resulting in disruptions in previously successful quality metrics. Periodic validation of EHR-based metrics is essential.

Identifying the Key Domains and Individual EHR-Based Palliative Care Metrics

Key to any quality metrics program is agreement on a set of measures and guidance on how to collect and report the data. A systematic review of palliative care quality metrics for cancer identified 284 metrics.³⁴ Smaller sets of metrics have been promoted by national programs, including Measuring What Matters (by the American Academy of Hospice and Palliative Medicine),³⁵ the National Quality Forum,³⁶ and others.^{15–17,37–39} These programs provide important insights into the development of specific metrics and permit benchmarking across programs and settings, but none of these programs focused on using the EHR.

The Cambia Palliative Care Center of Excellence at UW Medicine has developed a series of palliative care quality metrics across UW Medicine's four hospitals, diverse outpatient clinic network, and comprehensive cancer center. The project systematically evaluates primary and specialty palliative care for patients with serious illness in inpatient and outpatient settings, and has developed a users' manual to document our process as a resource for other healthcare systems.⁴⁰ We are engaging with stakeholders from diverse healthcare systems to guide the implementation and dissemination of this project.

We began the process of identifying palliative care quality metrics by reviewing recommendations from systematic reviews^34,41 and synthesizing these into a list of 17 metrics, which we presented at a stakeholder meeting in May 2015. The meeting involved 105 participants, including patient representatives, family members, clinicians, administrators, information technology specialists, quality improvement staff, and payers. The group reviewed a draft list of metrics, recommended one additional metric, and prioritized each metric. This resulted in 18 metrics in four domains: (1) end-of-life healthcare utilization; (2) circumstances of death; (3) assessing symptoms and social and spiritual needs; and (4) documenting patients' goals and preferences (Table 1).

Table 1.

Domains and Metrics Identified by Stakeholders for the Cambia Palliative Care Metrics Project and Representing Stakeholders' and Investigators' Opinions

Domain/measure no.	Numerator	Denominator	Importance ^a (%)	Feasibility ^b (%)	Difficulty ^c
Service utilization
1	No. of emergency room visits	Serious illness and last month of life	32	68	Low
2	No. of hospitalizations	Serious illness and last month of life	36	66	Low
3	Admitted to the ICU	Serious illness and last month of life	24	56	Low
4	Received chemotherapy	Metastatic cancer and last two weeks of life	18	44	Low
5	Readmissions in last 90 days of life (30 days; 7 days)	Serious illness	n/a	n/a	Low
Circumstances of death
6	Died in an acute care hospital	Serious illness decedents	24	56	Low
7	Died in ICU	Serious illness decedents	24	50	Low
8	Hospitalized patients who die an expected death with an ICD deactivated	Inpatient deaths with an ICD	8	18	Medium
9	Not admitted to hospice	Metastatic cancer decedents	20	30	Medium
10	Fewer than three days in hospice	Serious illness decedents	26	32	Medium
Screening and assessment
11	Received bowel regimen	Inpatients with serious illness who receive opiates	26	28	Medium
12	Screened for the presence and intensity of pain using a numeric pain score	Serious illness	52	40	Medium
13	Had comprehensive assessment completed (includes prognosis, functional assessment, screening for physical and psychological symptoms, and assessment of social and spiritual concerns)	Serious illness	54	14	High
14	Screened for shortness of breath	Serious illness	22	24	High
Patient needs and preferences
15	Have an advance care plan or surrogate decision maker documented in the EHR or documentation that an advance care plan was discussed but the patient did not wish or was not able to name a surrogate decision maker or provide an advance care plan	Serious illness	72	18	Medium
16	Documentation of patient's preferences for care (may include code status, preferences for general aggressiveness of care) has been considered or an attempt was made to identify them.	Serious illness	80	16	Medium
17	Discussion of emotional or psychological needs	Serious illness	48	12	High
18	Documentation in the clinical record of a discussion of spiritual/religious concerns or documentation that the patient/caregiver did not want to discuss	Inpatients with serious illness	46	14	High

Stakeholders included patients, patient representatives, families, clinicians, administrators, quality improvement staff, and information technology experts.

Importance = Percentage of stakeholders marking this as having high importance, n = 50.

Feasibility = Percentage of stakeholders marking this as having high feasibility, n = 50.

Difficulty = Expectation of difficulty anticipated by team developing metrics.

ICD, implantable cardioverter-defibrillator.

Example of Specific Metrics Across a Multihospital Healthcare System

We began implementation of our program at UW Medicine with the domains of healthcare utilization and circumstances of death for a decedent sample with chronic life-limiting illnesses. This initial focus represented a logical starting place because these metrics were relatively easy to obtain and viewed as important by healthcare system leadership. We chose decedents as our initial denominator population because the methodology for analyzing patterns of end-of-life care among decedents is well established.^42–44 Finally, to understand the quality of palliative care received by patients with chronic illnesses, we focused on decedents with at least one of the nine Dartmouth Atlas chronic conditions: cancers with poor prognoses, chronic pulmonary disease, coronary artery disease (CAD), heart failure, severe chronic liver disease, chronic renal failure, dementia, diabetes with end-organ damage, and peripheral vascular disease.⁴² We report on some of these metrics here to highlight the potential and the limitations of this approach.

We linked Washington State death certificate data to our EHR data to identify patients who met our criteria of attribution to the UW Medicine system. These criteria were adapted from Dartmouth Atlas: at least one nonsurgical inpatient visit at an affiliated hospital in the two years before death, or at least two outpatient visits from the same site in the last 32 months of life, with at least one visit occurring during the last 24 months of life.⁴² Decedents were excluded if they were younger than 18 years or if the cause of death on the death certificate was “injury or poisoning emanating from an accident, suicide, homicide, or an undetermined source.”

Using these criteria, we identified 23,096 decedents with life-limiting illness attributable to UW Medicine who died between January 1, 2010, and December 31, 2015. Overall, 57% of decedents were male and the average age at death was 66.5 years (Table 2). The most common chronic illness was cancer (51%), followed by chronic pulmonary disease (26%) and CAD (26%). The mean number of diagnoses per decedent was 1.9.

Table 2.

Characteristics of Decedents with One or More Chronic Conditions Who Were Cared for by UW Medicine, Death between 2010 and 2015

Characteristics	Total (n = 23,096^a)
Comorbidities, n (%)
Cancer	11,891 (51.4)
Chronic pulmonary disease	6052 (26.3)
Coronary heart disease	6001 (26.2)
Heart failure	5306 (23.0)
Peripheral vascular disease	2954 (12.6)
Severe chronic liver disease	2681 (12.8)
Diabetes with end-organ damage	1983 (8.6)
Chronic renal failure	4379 (19.0)
Dementia	2124 (9.2)
Insurance status, n (%)
Private	7988 (34.6)
Medicare	7892 (34.2)
Medicaid	5183 (22.4)
Military	829 (3.6)
Other insurance type	518 (2.2)
Uninsured	686 (3.0)
Male, n (%)	13,163 (57.0)
Race, n (%)
White	17,248 (82.8)
Black	1190 (5.7)
Native American/Alaska Native	355 (1.7)
Asian	1299 (6.2)
Native Hawaiian/other Pacific Islander	131 (0.6)
Hispanic	385 (1.8)
Mixed race	226 (1.1)
Age, mean (SD)	65.5 (15.1)
Education, n (%)
8th Grade or less	918 (4.5)
9–12 Years, no diploma	1422 (7.0)
High school graduate or equivalent	6823 (33.4)
Some college, no degree	4079 (20.0)
Associate's degree	1665 (8.1)
Bachelor's degree	3456 (16.9)
Master's degree	1418 (6.9)
Doctorate or professional degree	659 (3.2)

The group and total sample sizes were reduced for two variables for which death certificates provided incomplete information. Race was known for 20,834 decedents; education for 20,440 decedents.

SD, standard deviation.

From 2010 to 2015, we found decreasing intensity of hospital and ICU care for patients with hospitalizations at UW Medicine's two largest hospitals during decedents' last 30 days of life that included a lower proportion with acute care hospitalizations (29% to 24%; p < 0.001; Fig. 1) and with ICU care (21% to 17%; p < 0.001; Fig. 2). In addition, a lower proportion of decedents had 30-day hospital readmissions in the last 90 days at UW Medicine hospitals (10% to 8%; p = 0.02; Fig. 3).

FIG. 1.

Trends in inpatient care for patients hospitalized at the two largest UW Medicine hospitals during the last 30 days of life, from 2010 through 2015, patients with one or more chronic conditions. Significant negative linear effect—fewer people with hospitalizations as time progressed (p < 0.001). Curvilinear effect was not statistically significant (p = 0.084 for the unadjusted model, as shown in the graph below; p = 0.780—after adjustment for age at death, racial/ethnic minority status, and number of diagnoses).

FIG. 2.

Trends in ICU use for patients hospitalized at the two largest UW Medicine hospitals during the last 30 days of life, from 2010 through 2015, patients with one or more chronic conditions. Significant negative linear effect—fewer people with hospitalizations as time progressed (p < 0.001). Curvilinear effect was not statistically significant (p = 0.075 before adjustment, as shown in the graph below; p = 0.066 after adjustment for number of diagnoses).

FIG. 3.

Trends in inpatient readmission for patients hospitalized at the two largest UW Medicine hospitals within 30 days of discharge during the last 90 days of life, from 2010 through 2015, patients with one or more chronic conditions. Significant linear effect: fewer people with readmits as time progressed (p = 0.020). No significant curvilinear effect.

We also explored specific predictors of hospital and ICU utilization at the end of life, identifying decreased hospital utilization for patients with comorbid psychiatric illness raising concerns about inequities in serious illness care.⁴⁵ We found also that ∼40% of patients with chronic illness died in a hospital, with almost half of hospital deaths occurring outside the UW Medicine system, highlighting the importance of ensuring that advance care planning is transmitted to other healthcare systems.⁴⁶ In addition, we found that the decrease in hospital utilization during the last 30 days of life was consistent for patients older and younger than 65 years, while the decrease in ICU utilization in the last 30 days of life was more prominent for patients older than 65 years.⁴⁷ Importantly, one of the limitations of EHR-based metrics is that utilization data are limited to the healthcare system. For example, regional changes in hospital use (such as changes in market share) could invalidate these findings. These findings need to be validated with population-based data across healthcare systems.

Our stakeholders gave top priority to metrics that reflect documentation of patients' goals of care and advance care planning (Table 1). We began this work by using available data that recorded the presence of advance directives (including healthcare directives and durable power of attorney for healthcare) and Physician Orders for Life Sustaining Treatment (POLST) forms, as well as upload dates. We found that the proportion of decedents who had either an advance directive or a POLST form in their EHR at the time of death increased significantly from 21% in 2010 to 52% in 2015 (p < 0.01; Fig. 4). We validated these utilization and advance directive measures and found strong agreement between manual chart review and our automated methods (with agreement between 75% and 100%). However, these metrics fall short of the charge from the stakeholder group: identification of discussions of advance care planning and goals of care.

FIG. 4.

Trends in EHR documentation of advance directives, from 2010 through 2015, patients with one or more chronic conditions. Significant positive linear effect—more documentation as time progressed (p < 0.001). There was also a significant curvilinear effect with the overall positive effect (p = 0.003) becoming even more positive as time progressed (p < 0.001). EHR, electronic health record.

Challenges and Lessons Learned from EHR-Based Metrics

In our efforts to use the EHR to assess primary and specialty palliative care, we uncovered specific challenges and lessons. One of our most important challenges was to define the population of interest, or “denominator.”^48,49 This is a particular challenge for assessing population-based care, since only some healthcare systems provide comprehensive population-based care through accountable care networks or other population-based programs. In addition, the denominator of interest may differ for different stakeholders. The use of EHR-based metrics for accountability and value-based purchasing will also need to ensure that metrics cannot be easily gamed.¹² EHR-based metrics will be difficult to use for population-based programs and accountability purposes until EHRs achieve interoperability.¹²

We started with a focus on decedents with at least one of the Dartmouth Atlas chronic conditions for three reasons: (1) diagnostic codes were readily available in the EHR; (2) methods to use these data were well developed; and (3) these conditions account for 90% of deaths among the Medicare population, suggesting many of these patients receive serious illness care.^42,50 Examining decedents allows a clear focus on the quality of end-of-life care, but this approach creates a second challenge in that it cannot be used to identify individual patients whose care can be improved, since patients have already died. We are working to define other patient populations with unmet palliative care needs, such as those with psychiatric illness⁴⁵ or adult congenital heart disease.⁵¹

A third challenge is that of benchmarking quality and the related consequence of adjudicating accountability for low or high quality. The most appropriate benchmark may vary by stakeholder group, hospital, or healthcare system because of differences in patient populations. Furthermore, improvements in metrics over time can be difficult to attribute to specific programs or providers. Accountability can be challenging with all forms of quality metrics, but is particularly important for rewarding success in improving palliative care or holding providers or programs accountable for shortcomings.

A fourth challenge is to integrate champions of serious illness care into institution-wide quality improvement efforts, to ensure their efforts are both durable and supported by institutional leadership and system-level quality improvement. Similarly, these champions should be engaged when EHR systems are updated, as such changes can create erroneous data and lead to unexplained variation in quality reports.

A fifth challenge was associated with ensuring that metrics of importance to stakeholders are discoverable in the EHR. Our stakeholders prioritized the assessment of advance care planning and goals-of-care discussions. Although written advance directives may be a marker for these discussions, capturing advance care planning and goals-of-care discussions is more important. Unfortunately, these measures are more difficult to capture.^52,53 Advance care planning is documented in diverse ways by different clinicians and can be stored in various locations in the EHR that may include a specific EHR module, a specific note type, or a standard clinic or hospital progress note with no overt flags to signal this content.⁵² Widespread use of well-designed, comprehensive, and retrievable templates would facilitate documentation of advance care planning and goals-of-care conversations, and could also provide a guide for these discussions.⁵⁴ Natural language processing (NLP) and machine learning also hold promise for facilitating quality metrics related to advance care planning and goals-of-care discussions. Recent studies have shown that NLP approaches can effectively extract meaningful information, such as adverse drug reactions, cancer staging, and disease progression from clinical notes.^55–60 Preliminary results suggest that NLP can identify documentation of advance care planning.⁶¹ Several groups are currently working on using NLP or machine learning to identify and evaluate these discussions.⁶²

Summary

The EHR offers important opportunities to identify seriously ill patients and assess palliative care quality metrics across large numbers of patients with serious illness. Currently, the opportunities are most achievable for assessing intensity of care and some easily captured metrics such as advance directives, POLST forms, and pain assessments. Other metrics, such as documentation of advance care planning and goals-of-care discussions, are not as readily available in most healthcare systems, but the use of EHR modules specific for advance care planning and approaches such as NLP offer potential solutions. However, EHR-based palliative care metrics pose important challenges that must be addressed: lack of interoperability across healthcare systems and, at times, within a single system; absence of systematic documentation of patient- and family-centered outcomes; and difficulties in capturing quality metrics information from data that were collected for other purposes. In this report, we describe a palliative care metrics program to assess serious illness care that can serve as a model and resource for others. Although the EHR offers great promise for measuring, benchmarking, and improving palliative care, these promises are largely unrealized on a national or population-based scale at this time. As EHR-based quality measures are developed, validated, and used to improve serious illness care, their strengths and shortcomings must be assessed and addressed, with adjustments made to ensure that the metrics are feasible, reliable, valid, and meaningful for quality improvement and accountability.

Footnotes

Acknowledgments

This article was supported by the Cambia Health Foundation, the Gordon and Betty Moore Foundation, and UW Medicine.

Author Disclosure Statement

No competing financial interests exist.

References

Dumanovsky

, Augustin

, Rogers

, et al.: The growth of palliative care in U.S. Hospitals: A status report. J Palliat Med, 2016; 19:8–15.

Temel

, Greer

, Muzikansky

, et al.: Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med, 2010; 363:733–742.

Bakitas

, Lyons

, Hegel

, et al.: Effects of a palliative care intervention on clinical outcomes in patients with advanced cancer: The Project ENABLE II randomized controlled trial. JAMA, 2009; 302:741–749.

Morrison

, Dietrich

, Ladwig

, et al.: Palliative care consultation teams cut hospital costs for Medicaid beneficiaries. Health Affairs, 2011; 30:454–463.

Morrison

, Penrod

, Cassel

, et al.: Cost savings associated with US hospital palliative care consultation programs. Arch Intern Med, 2008; 168:1783–1790.

Meier

, Back

, Berman

, et al.: A National Strategy for palliative care. Health Aff (Millwood), 2017; 36:1265–1273.

Curtis

, Back

, Ford

, et al.: Effect of communication skills training for residents and nurse practitioners on quality of communication with patients with serious illness: A randomized trial. JAMA, 2013; 310:2271–2281.

Carson

, Cox

, Wallenstein

, et al.: Effect of palliative care-led meetings for families of patients with chronic critical illness: A randomized clinical trial. JAMA, 2016; 316:51–62.

Kentish-Barnes

, Chevret

, Champigneulle

, et al.: Effect of a condolence letter on grief symptoms among relatives of patients who died in the ICU: A randomized clinical trial. Intensive Care Med, 2017; 43:473–484.

10.

Tinetti

: The retreat from advanced care planning. JAMA, 2012; 307:915–916.

11.

Smith

, Bernacki

, Block

: The role of palliative care in population management and accountable care organizations. J Palliat Med, 2015; 18:486–494.

12.

Conway

, Mostashari

, Clancy

: The future of quality measurement for improvement and accountability. JAMA, 2013; 309:2215–2216.

13.

Teno

, Price

, Makaroun

: Challenges of measuring quality of community-based programs for seriously ill individuals and their families. Health Aff (Millwood), 2017; 36:1227–1233.

14.

Kamal

, Hanson

, Casarett

, et al.: The quality imperative for palliative care. J Pain Symptom Manage, 2015; 49:243–253.

15.

Kamal

, Harrison

, Bakitas

, et al.: Improving the quality of palliative care through National and Regional Collaboration Efforts. Cancer Control, 2015; 22:396–402.

16.

Kamal

, Kavalieratos

, Bull

, et al.: Usability and acceptability of the QDACT-PC, an electronic point-of-care system for standardized quality monitoring in palliative care. J Pain Symptom Manage, 2015; 50:615–621.

17.

Pantilat

, O'Riordan

, Bruno

: Two steps forward, one step back: Changes in palliative care consultation services in California hospitals from 2007 to 2011. J Palliat Med, 2014; 17:1214–1220.

18.

Pantilat

, Marks

, Bischoff

, et al.: The Palliative Care Quality Network: Improving the quality of caring. J Palliat Med, 2017; 20:862–868.

19.

Walling

, Tisnado

, Asch

, et al.: The quality of supportive cancer care in the veterans affairs health system and targets for improvement. JAMA Intern Med, 2013; 173:2071–2079.

20.

Rosenbaum

: Transitional chaos or enduring harm? The EHR and the disruption of medicine. N Engl J Med, 2015; 373:1585–1588.

21.

Blumenthal

, Abrams

: Tailoring complex care management for high-need, high-cost patients. JAMA, 2016; 316:1657–1658.

22.

Bynum

JPW

, Austin

, Carmichael

, Meara

: High-cost dual eligibles' service use demonstrates the need for supportive and palliative models of care. Health Aff (Millwood), 2017; 36:1309–1317.

23.

Aldridge

, Bradley

: Epidemiology and patterns of care at the end of life: Rising complexity, shifts in care patterns and sites of death. Health Aff (Millwood), 2017; 36:1175–1183.

24.

Adler

, Stead

: Patients in context—EHR capture of social and behavioral determinants of health. N Engl J Med, 2015; 372:698–701.

25.

Moody

, Slocumb

, Berg

, Jackson

: Electronic health records documentation in nursing: Nurses' perceptions, attitudes, and preferences. Comput Inform Nurs, 2004; 22:337–344.

26.

Mack

, Cronin

, Taback

, et al.: End-of-life care discussions among patients with advanced cancer: A cohort study. Ann Intern Med, 2012; 156:204–210.

27.

Dimick

: Documentation bad habits. Shortcuts in electronic records pose risk. J AHIMA, 2008; 79:40–43; quiz 5–6.

28.

Hammond

, Helbig

, Benson

, Brathwaite-Sketoe

: Are electronic medical records trustworthy? Observations on copying, pasting and duplication. AMIA Annu Symp Proc, 2003:269–273.

29.

Teno

, Lynn

, Wegner

, et al.: Advance directives for seriously ill hospitalized patients: Effectiveness with the Patient Self-Determination Act and the SUPPORT Intervention. J Am Geriatr Soc, 1997; 45:500–507.

30.

Berry

, Trigg

, Lober

, et al.: Computerized symptom and quality-of-life assessment for patients with cancer part I: Development and pilot testing. Oncol Nurs Forum, 2004; 31:E75–E83.

31.

Berry

, Wolpin

, Lober

, et al.: Actual use and perceived usefulness of a web-based, decision support program for men with prostate cancer. Stud Health Technol Inform, 2006; 122:781–782.

32.

Mandl

, Khorasani

, Kohane

: Meaningful use of electronic health records. Health Aff (Millwood), 2012; 31:1365; author reply 6.

33.

Duftschmid

, Wrba

, Rinner

: Extraction of standardized archetyped data from Electronic Health Record systems based on the Entity-Attribute-Value Model. Int J Med Inform, 2010; 79:585–597.

34.

Kamal

, Gradison

, Maguire

, et al.: Quality measures for palliative care in patients with cancer: A systematic review. J Oncol Pract, 2014; 10:281–287.

35.

Unroe

, Hickman

, Torke

, Group

ARCW

: Care consistency with documented care preferences: Methodologic considerations for implementing the “measuring what matters” quality indicator. J Pain Symptom Manage, 2016; 52:453–458.

36.

Attribution–Principles and Approaches: National Quality Forum. www.qualityforum.org/Publications/2016/12/Attribution_-_Principles_and_Approaches.aspx 2016. (Last accessed March 25, 2017).

37.

Mularski

, Curtis

, Billings

, et al.: Proposed quality measures for palliative care in the critically ill: A consensus from the Robert Wood Johnson Foundation Critical Care Workgroup. Crit Care Med, 2006; 34:S404–S411.

38.

Clarke

, Curtis

, Luce

, et al.: Quality indicators for end-of-life care in the intensive care unit. Crit Care Med, 2003; 31:2255–2262.

39.

Earle

, Park

, Lai

, et al.: Identifying potential indicators of the quality of end-of-life cancer care from administrative data. J Clin Oncol, 2003; 21:1133–1138.

40.

Palliative Care Quality Metrics User's Guide: Cambia Palliative Care Center of Excellence. www.uwpalliativecarecenter.com. 2017. (Last accessed September 8, 2017).

41.

Lorenz

, Lynn

, Dy

, et al.: Quality measures for symptoms and advance care planning in cancer: A systematic review. J Clin Oncol, 2006; 24:4933–4938.

42.

The Dartmouth Atlas of Healthcare. www.dartmouthatlas.org. 2016. (Last accessed March 25, 2017).

43.

Wennberg

, Bronner

, Skinner

, et al.: Inpatient care intensity and patients' ratings of their hospital experiences. Health Aff (Millwood), 2009; 28:103–112.

44.

Wennberg

, Fisher

, Stukel

, et al.: Use of hospitals, physician visits, and hospice care during last six months of life among cohorts loyal to highly respected hospitals in the United States. BMJ, 2004; 328:607.

45.

Lavin

, Davydow

, Downey

, et al.: Effect of psychiatric illness on acute care utilization at end of life from serious medical illness. J Pain Symptom Manage, 2017; 54:176–185 e1.

46.

Hicks

, Downey

, Engelberg

, et al.: Predictor of death in the hospital for patients with chronic serious illness. J Palliat Med, 2017 [Epub ahead of print; DOI: 10.1089/jpm.2017.0127].

47.

Sathitratanacheewin

, Engelberg

, Downey

, et al.: Temporal trends between 2010 and 2015 in intensity of care at end-of-life for patients with chronic illness: Influence of age under versus over 65 years. J Pain Symptom Manage, 2017 [Epub ahead of print]. DOI: 10.1016/j.jpainsymman.2017.08.032

48.

Kelley

, Bollens-Lund

, Covinsky

, et al.: Prospective identification of patients at risk for unwarranted variation in treatment. J Palliat Med, 2017 [Epub ahead of print]. DOI: 10.1089/jpm.2017.0063

49.

Kelley

, Covinsky

, Gorges

, et al.: Identifying older adults with serious illness: A critical step toward improving the value of health care. Health Serv Res, 2017; 52:113–131.

50.

Wennberg

: Unwarranted variations in healthcare delivery: Implications for academic medical centres. BMJ, 2002; 325:961–964.

51.

Steiner

, Kirkpatrick

, Heckbert

, et al.: Identification of adults with congenital heart disease of moderate or great complexity from administrative data. Congenital Heart Disease, 2017 [Epub ahead of print]. DOI: 10.1111/chd.12524

52.

Turley

, Wang

, Meng

, et al.: Impact of a care directives activity tab in the electronic health record on documentation of advance care planning. Perm J, 2016; 20:43–48.

53.

Turley

, Wang

, Meng

, et al.: An information model for automated assessment of concordance between advance care preferences and care delivered near the end of life. J Am Med Inform Assoc, 2016; 23:e118–e124.

54.

Bernacki

, Block

: Serious illness communications checklist. Virtual Mentor, 2013; 15:1045–1049.

55.

Cheng

, Zheng

, Savova

, Erickson

: Discerning tumor status from unstructured MRI reports—Completeness of information in existing reports and utility of automated natural language processing. J Digit Imaging, 2010; 23:119–132.

56.

Danforth

, Early

, Ngan

, et al.: Automated identification of patients with pulmonary nodules in an integrated health system using administrative health plan data, radiology reports, and natural language processing. J Thorac Oncol, 2012; 7:1257–1262.

57.

Imler

, Morea

, Imperiale

: Clinical decision support with natural language processing facilitates determination of colonoscopy surveillance intervals. Clin Gastroenterol Hepatol, 2014; 12:1130–1136.

58.

Imler

, Morea

, Kahi

, Imperiale

: Natural language processing accurately categorizes findings from colonoscopy and pathology reports. Clin Gastroenterol Hepatol, 2013; 11:689–694.

59.

Carrell

, Halgrim

, Tran

, et al.: Using natural language processing to improve efficiency of manual chart abstraction in research: The case of breast cancer recurrence. Am J Epidemiol, 2014; 179:749–758.

60.

Sarker

, Gonzalez

: Portable automatic text classification for adverse drug reaction detection via multi-corpus training. J Biomed Inform, 2015; 53:196–207.

61.

Using natural language processing to develop quality measures in palliative surgery through the use of electronic health record data: National Quality Forum. http://nqf.commpartners.com/se/Meetings/Playback.aspx?meeting.id=337177. 2016. (Last accessed July 26, 2017).

62.

Barrett

, Weber-Jahnke

, Thai

: Engineering natural language processing solutions for structured information from clinical text: extracting sentinel events from palliative care consult letters. Stud Health Technol Inform, 2013; 192:594–598.