Abstract
Background:
Electronic health records (EHRs) may help enable reliable, rapid data management for many uses, such as facilitating communication of advance care planning (ACP). However, issues with validity and accuracy of EHRs hinder the use of ACP information for practical applications.
Design:
We present a cross-sectional pilot study of 433 older adults with cancer from three large health care systems, participating in an ongoing multisite pragmatic trial (4UH3AG060626-02). We compared data extracted from dedicated structured EHR fields for ACP to a chart review of corresponding ACP documentation contained in the medical chart.
Results:
Structured ACP data existed for 43.2% of patients and varied by site (25.7% −48.9%). Of the identified structured ACP data elements, 59.2% of recorded elements were correct, 23.7% were incorrect, and 17.1% were duplicates with heterogeneity across sites.
Conclusion:
Structured ACP data in EHRs were frequently incorrect. This represents a problem for patients and their families, as well as quality improvement and research efforts. Clinical Trials Registration: NCT03609177.
Background
Electronic health records (EHRs) may help enable reliable, rapid data management for clinical care, quality improvement, regulatory compliance, and research. The EHR may also assist with implementation of advance care planning (ACP), a process aimed at understanding patients' goals, values, and preferences and using these to guide future medical care. 1 The intention of ACP is to align medical care in serious illness with those stated preferences and the focus is on iterative conversation and preparation of patients and their family and friends for shared decision making rather than on completion of legal forms.1–3
Experts recommend expanding ACP to reach more patients and in response, policy makers have created financial and other incentives to drive its growth in practice.4–9 As highlighted by the COVID-19 pandemic, ACP is particularly important for older adults; however, only half of them have EHR ACP documentation.6,10–13 Successful ACP requires easy access to reliable data about patient preferences, which depend upon EHR functionality.5,14 However, documented shortcomings in accessibility, reliability, validity, and completeness of existing EHR ACP information threaten its utility.9,15 Furthermore, ACP documentation can only help further patients' goals if clinicians can quickly access accurate data about their preferences and transform wishes into goal-concordant action in moments of clinical need. 16 Accordingly, EHR designers have established dedicated structured locations, where clinicians can quickly record and retrieve this information.17–19
Structured locations are those created specifically for storing data in the EHR that follows a prescribed format, often imposed by the given data entry software (e.g., Epic). Typical examples of structured data include patient demographic data, laboratory values, and billing codes. The structured data in this study came from site-defined specific data fields created for holding ACP documents (e.g., advance directives, templated ACP storage locations). Unstructured data, on the other hand, do not have such strict rules imposed on them. This allows for more freedom when entering the data; however, the lack of standardization makes it more difficult for the data to be utilized by computers for extraction and measurement. Examples of unstructured data fields in the EHR include clinical text notes regarding a patient's symptoms, X-ray images, and a clinician's ACP documentation in text notes of a conversation they had with the patient regarding goals and preferences.
In this study of older adults with serious illness, we identified structured EHR data fields that label the presence of ACP documentation and examined the documents within those structured data fields to determine if the content of these documents accurately reflected the structured EHR data field element.
Methods
We present data from three large health care systems that use structured EHR ACP data fields. This analysis arises from an ongoing multisite pragmatic trial (4UH3AG060626-02) aimed at improving ACP for older adults with cancer, which was approved by the Institutional Review Board at Dana-Farber Cancer Institute. 20
We analyzed one year of data (2018–19) from a convenience sample of three disease-based oncology clinics, one drawn from each site. We included older adults, 65 years and older, with advanced cancer identified pragmatically using International Classification of Diseases codes. 20 Local research and data teams from each site, including experts in palliative care, oncology, and informatics, identified the site-specific location used to store structured data for ACP elements (e.g., structured data tab for an advance directive or health care proxy form) in their EHR. Next, each site extracted all data contained in preidentified, locally defined structured fields for known ACP elements. Two of the sites use the Epic EHR and one uses a combination of Allscripts in the outpatient setting and Sunrise in the inpatient setting.
To explore the accuracy of all data extracted from structured ACP EHR locations, we then classified the contents of each data element contained in the structured fields into categories in a three-step approach. First, local research staff reviewed chart documentation represented by each structured data point in the medical chart and described its content in narrative form (e.g., “Health Care Power of Attorney”). Second, one coordinating center investigator classified narrative forms into a subset of nine exhaustive categories common across all sites (e.g., “Health Care Proxy/Durable Power of Attorney for Healthcare,” see Table 1 for complete list). Last, local research staff reviewed these categories for each data element and, when necessary, reclassified each document accordingly.
Documentation Categories for Structured Data Advance Care Planning Documents
ACP, advance care planning.
For analysis, we then grouped the nine categories into three areas: (1) “Correct” ACP documents accurately reflecting ACP information in structured data (e.g., a completed Advance Care Plan or Health Care Proxy), (2) “Incorrect” documents that do not represent completed ACP (e.g., Surgical Consent Form, Blank or Incomplete Form), and (3) “Duplicate” documents, those that are exactly the same as another document, such as those that have been scanned twice (Table 1). We present data as descriptive statistics with counts, proportions, and medians with interquartile ranges.
Results
Across all three sites, 433 patients met inclusion criteria. Overall, structured ACP data were present for 43.2% of these patients, and prevalence varied by site (25.7% Site 1, 48.9% Site 2, and 42.2% Site 3). Patients with structured data had a median age of 72.7 and were 65% male, 82.4% white, and predominantly English speaking; patient characteristics varied between sites. During the one-year study period, the mortality rate for patients with structured ACP data was 19.8% (ranging from 0% to 22.2% across sites) (Table 2). Of the identified structured ACP data, 59.2% of recorded data elements were correct, 23.7% were incorrect, and 17.1% were duplicates. The most common category of correct ACP data was Advance Directive/Description of End-of-Life Wishes (32.8% of all structured data elements). The most common reason for incorrect documents were those that were reported in structured data to be available documents, but were not discoverable on chart review (8.8% of all structured data elements).
Patient Characteristics for Patients with Advance Care Planning Data Present in Structured Locations by Site
Statistics presented: median (IQR); n (%).
Patients >89 years coded as 90 due to institutional deidentification rules.
IQR, interquartile range.
The level of accuracy varied by site. At Site 1, 49.1% of identified structured data elements were correct, 36.4% were incorrect, and 14.5% were duplicates. The most commonly identified correct ACP category was Health Care Proxy/Durable Power of Attorney for Healthcare (23.6% of total Site 1 structured data elements). Documents that were reported in structured data to be available, but were not discoverable on chart review were the most common cause of incorrect structured data elements (32.7% of total Site 1 structured data elements).
At Site 2, 83.5% of identified structured data elements were correct, 9.1% were incorrect, and 7.4% were duplicates. The most frequent correct ACP category was Advance Directive/Description of End-of-Life Wishes (59.1% of total Site 2 structured data elements). Documents that do not represent ACP, such as surgical consent forms or safety checklists, were the most common cause of incorrect data elements at Site 2 (6.2% of total Site 2 structured data elements).
At Site 3, 31.1% of identified structured data elements were correct, 37.9% were incorrect, and 31.1% were duplicates. The most common category for correct ACP was Health Care Proxy/Durable Power of Attorney for Healthcare (25.0% of total Site 3 structured data elements). Documents that were reported in structured data to have been requested, but not completed when examined on chart review were the most common cause of incorrect data elements at Site 3 (22.0% of total Site 3 structured data elements) (Table 3).
Chart Review Content of Structured Data Advance Care Planning Documents by Classification
Statistics presented: n (%).
DPOA, durable power of attorney; EOL, end of life; HCP, health care proxy; HIPAA, Health Insurance Portability and Accountability Act; MOLST, Medical Orders for Life Sustaining Treatment.
Discussion
This three-site cross-sectional study of older adults with cancer found that 43.2% of older patients with advanced cancer had ACP data in structured data fields for ACP in the EHR, similar to rates of ACP completion in prior studies.6,13 However, on deeper chart review, only 59.2% of identified ACP data elements were accurate and many were incorrect or duplicates, with notable heterogeneity across three large systems. These two findings together, demonstrating that around half of older adults with cancer have ACP documentation in readily retrievable locations and notable inaccuracy with duplication in that documentation, point toward an ongoing deficit in ACP for older adults. The key findings of this study, the inaccuracy and duplication of documentation elements in EHR storage locations chosen specifically to store ACP information, indicate an ongoing need for systems that can reliably and accurately capture and organize this information. These preliminary findings sound a note of caution for clinical care, legal compliance, quality improvement, and research.
While structured EHR documentation may be necessary to improve ACP practice, current implementation that uses unvalidated ACP data may provide false reassurance, given the levels this study identified of incomplete and inaccurate documents contained in structured data fields. Effectively, in many cases, structured data elements constitute a mislabeling of the source documents in a manner that can lead to inappropriate reliance for extremely impactful decisions. This pilot work suggests that estimates using structured EHR data to evaluate clinical ACP practice using pragmatic methods could be erroneous at rates approaching 40%–50%. In addition, the differences in findings between sites also raise concerns that comparisons between different health centers that rely on structured data elements may be flawed. Duplicate documentation, especially if scanned out of chronological order, can hinder the speed and confidence with which clinicians can use ACP documentation during times of acute need, such as those highlighted by COVID-19. However, a detailed accounting of the difference between the dates of structured data elements and the dates of the source documentation was beyond the scope of this analysis.
This study has important limitations to consider. First, it is a small pilot study completed by retrospective chart review, conducted at three oncology clinics across the central and eastern United States, and this limits generalizability. Second, we did only examine ACP data that are contained in the identified and intended location for storage at each setting, which means we did not include ACP information that is contained within clinical notes or other unstructured locations in the EHR. However, one of the key reasons driving the creation of structured, central, easily discoverable locations for ACP information in health records is because ACP information is difficult to find in the large volume of clinical notes and documentation.
Legally, health care systems may expose themselves to liability with incomplete or inaccurate forms that purport to officially represent patients' documented preferences and surrogate decision makers. In research and quality improvement, variability and inaccuracy of structured ACP documentation data at the levels observed in this study point toward significant bias and error inherent in using structured EHR variables for large pragmatic evaluations. For policy makers, legislation and incentive plans that depend on structured EHR data for efficiency may be hindered by inaccuracies and duplications in collective and comparative data. If replicated, these results suggest greater need for scrutiny, rigor, and accountability of ACP data in the EHR.
Footnotes
Authors' Contributions
Study concept and design: J.R.L., D.A.G., M.K.P.-O., J.A.T., and A.V. Acquisition of subjects and/or data and analysis and interpretation of data: all authors. Preparation of article: all authors.
Acknowledgment
The ACP-PEACE Investigators: S. Yousuf Zafar, MD; Maria Torroella Carney, MD; Diana Martins-Welch, MD; Camille Chan; Michael Qiu, MD, PhD; Craig E. Devoe, MD; Jon C. Tilburt, MD; Charles L. Loprinzi, MD; Parvez A. Rahman, MHI; Aretha Delight Davis, MD, and JD; Areej El-Jawahri, MD.
Disclaimer
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Funding Information
Research reported in this publication was supported within the National Institutes of Health (NIH) Health Care Systems Research Collaboratory by cooperative agreement UH3AG060626 from the National Institute on Aging. This work also received logistical and technical support from the NIH Collaboratory Coordinating Center through cooperative agreement U24AT009676. The sponsor had no role in the design, data collection, and conduct of this work and did not participate in interpretation of data or preparation of this article.
Author Disclosure Statement
Dr. Tulsky is a Founding Director of VitalTalk, a nonprofit organization focused on clinician communication skills training, from which he receives no compensation. Dr. Volandes has a financial interest in the nonprofit foundation Nous Foundation (d/b/a ACP Decisions, 501c3). The nonprofit organization develops ACP video decision aids and support tools. Dr. Volandes' interests were reviewed and are managed by Massachusetts General Hospital and Partners HealthCare in accordance with their conflict of interest policies. None of the other authors have any conflict of interests to disclose.