Development of the Quality Data Collection Tool for Prospective Quality Assessment and Reporting in Palliative Care

Abstract

Background:

Assessing and reporting the quality of care provided are increasingly important in palliative care, but we currently lack practical, efficient approaches for collection and reporting.

Objective:

In response, the Global Palliative Care Quality Alliance (“Alliance”) sought to create a Quality Data Collection Tool for Palliative Care (QDACT-PC).

Methods:

We collaboratively and iteratively developed QDACT-PC, an electronic, point-of-care quality monitoring system for palliative care that supports prospective quality assessment and reporting in any clinical setting. QDACT-PC is the web-based data collection and reporting interface. Quality measures selected to be used in QDACT-PC were derived from a systematic review summarizing all published palliative care quality measure sets; Alliance clinical providers prioritized measures to be included in QDACT-PC to ensure maximal clinical relevance. Data elements and variables required to ascertain conformance to all selected quality measures were included in the QDACT-PC data dictionary. Whenever possible, variables collected in QDACT-PC align with validated surveys and/or nationally recognized common data elements. QDACT–PC data elements and software programmed business rules inform real-time assessments of conformance to selected quality measures. Data are deposited into a centralized registry for future analyses.

Results:

QDACT-PC can be used to report on >80% of all published palliative care quality measures and 100% of high-priority measure.

Conclusion:

Electronic methods for collecting point-of-care quality monitoring data can be developed using collaborative partnerships between community and academic palliative care providers. Feasibility testing and creation of feedback reports are ongoing.

Background

The ability to perform regular, standardized, quality assessment is rapidly becoming integral to palliative care program viability and growth.¹ On the horizon are several impending policy changes that will press every palliative care organization to develop capacity for regular quality measurement. The recently developed set of quality measures for palliative care, endorsed by the National Quality Forum (NQF), has set the stage for near-term changes that will be the foundation for future quality measure-based reimbursement. Lack of collecting data on such measures would result in a significant physician reimbursement penalty, threatening the viability of many palliative care programs.

To meet these demands, we need a practical data collection and reporting solution applicable to academic, community-based, and rural settings. Requirements include the following: (1) standardized, replicable methods for quality assessment; (2) processes that can be broadly applied across diverse organizations and populations; and (3) structures that generate aggregated population-based and site-level data to define palliative care quality and benefit. These must integrate within the usual reimbursement challenges,² workforce issues,³ and diverse documentation techniques encountered by organizations every day. Furthermore, a quality assessment system must seamlessly integrate data collection into clinical workflows of diverse providers and organizations, and integrate with information technology (IT) solutions being put into place. The ideal solution aligns with healthcare processes,⁴ facilitates quality-based payment, and integrates data collection to inform clinical, quality, and research needs simultaneously.⁵

Herein, we report the development of the Quality Data Collection Tool for Palliative Care (QDACT-PC), an electronic, prospective quality assessment, and reporting solution. Building on the experience derived from a regional palliative care database (PCD) initiative,^6,7 QDACT-PC is intended to be a point-of-care quality monitoring tool that is (1) feasible and useful in everyday practice in palliative care, (2) applicable across provider types, organizations, and geographic regions, (3) aligned with quality improvement initiatives in palliative care, and (4) able to generate a registry database with reliable data that can be used for current and future analyses.

Methods

Background

An academic/community collaboration called the Carolinas Palliative Care Consortium (“Carolinas Consortium”) was created in 2007 between Duke University Medical Center (Durham, North Carolina) and three community palliative care organizations (Four Seasons, Flat Rock, NC; Forsyth Palliative Care, Winston-Salem, NC; Hospice of Wake and Horizons Palliative Care, Raleigh, NC. As the Consortium expanded to organizations outside the state, it was renamed the “Global Palliative Care Quality Alliance” (hereafter the “Alliance”). Patient-level data were collected on paper, entered into a local database, and intermittently transmitted to a centralized dataset for analysis and quality reporting. The information contributed to a growing data resource, with more than 32,000 patient encounters—total of 6957 unique patients collected from June 2008 through October 2011. Data analyses supported organizational quality improvement objectives and descriptive research about the population served by the Alliance. This provided proof of principle that collecting data on quality is feasible in community settings⁸ and that these data can inform clinical practice and institutional priorities in community-based palliative care.^9,10 However, data collection processes were inefficient and data collected did not always map to emerging quality measures; a contemporary informatics-based solution was needed, aligned with expectations for quality monitoring in palliative care.

Stakeholder recommendations for QDACT-PC

As a first step in developing QDACT-PC, we surveyed the Alliance clinicians about the strengths and weaknesses of the paper-based processes and solicited suggestions for improvement. We supplemented the survey with open-ended questions and a group discussion during a two-day retreat that included all clinicians and administrative stakeholders. Twenty-two clinicians participated in the survey; 45% noted that QDACT-PC should provide users with direct feedback at point-of-care to inform clinical care and practice improvement, and 74% indicated that QDACT-PC must contain a core, mandatory set of questions augmented by optional, site-specific questions. Survey respondents and participants in the retreat recommended that we (1) make the tool web based; (2) incorporate data collection instruments drawn from validated sources familiar to palliative care providers; and (3) demonstrate QDACT-PC usability in “normal” clinical environments. When asked which NQF¹¹ domains should inform the data dictionary and questions embedded in QDACT-PC, respondents endorsed all eight domains (Structure and Processes of Care, Physical Aspects of Care, Psychiatric and Psychological Aspects of Care, Spiritual and Existential Aspects of Care, Social Aspects of Care, Cultural Aspects of Care, Care of the Imminently Dying, Ethical and Legal Aspects of Care), supplemented by other domains such as Independence and Functionality, Prognostic Understanding, and existing PQRS measures from other disciplines such as geriatrics.

Team approach to designing QDACT-PC

Starting in August 2010, design features were defined in biweekly teleconferences and three in-person meetings involving community palliative care providers in the Alliance and an interdisciplinary team of palliative care clinicians, clinical researchers, software programmers, database architects, data analysts, a data security specialist, a graphic designer, and a statistician. Since many of the terms used in quality improvement, research, and IT overlap with differing definitions by discipline, we developed an agreed nomenclature (Table 1). The team focused on the following: needs, as perceived by clinicians; usability and security of data collection practices; sustainability of processes across diverse settings; and value added for clinicians. These informed development of the questions to be asked, data dictionary, QDACT-PC user interface, how data collection should be integrated into care processes, and reports needed.

Table 1.

Nomenclature and Definitions Used in the Development of QDACT-PC

Term	Definition
Carolinas Consortium for Palliative Care (“Carolinas Consortium”)	A four-organization academic/community collaborative in North Carolina with a mission to collectively explore issues of quality assessment and improvement.
Global Palliative Care Quality Alliance	The updated name for the Carolinas Consortium, reflecting the geographically diverse nature of members.
PCD	The previous (2007–2011) PCD used by the Carolinas Consortium; based on paper and electronic pen data submission.
QDACT-PC	The user interface for electronic quality monitoring; aligns with queries required to address quality measures in palliative care. QDACT-PC is defined as the user interface, the data dictionary, and the business rules required to support this project.
Data elements	A unit of data for which the definition, identification, representation, and permissible values are specified by means of a set of attributes and codified in the data dictionary. The QDACT-PC data elements reflect the units of information collected and stored by QDACT-PC to inform planned questions and queries. Most QDACT-PC data elements map directly to one or several quality measures.
Common data element	Whenever possible, QDACT-PC uses common data elements aligned with national standards or seeks to publish the data elements used so that they can become adopted as the national standard.
Quality measures	A standardized assessment that quantifies the extent to which an individual unit (e.g., healthcare organization, provider, patient) within a predefined population meets some criterion for quality of care; conformance is assessed within a prespecified population (cohort) by evaluating the number complying with the measure (numerator) as a proportion of the cohort (denominator).
Clinical Decision Support	Electronic solutions that assist health professionals with decision-making tasks, such as choosing evidence-based symptom-directed interventions in palliative care.

PCD, palliative care database; QDACT-PC, Quality Data Collection Tool for Palliative Care.

Development of the data dictionary

Each question asked in QDACT-PC maps to a predefined data element in the QDACT-PC data dictionary. A data element is the atomic unit of data (Table 1), and a variable used in an analysis comprising one or more data elements. Attributes such as how the question is asked in QDACT-PC to generate the data, the permissible values, how the information is stored in the database, and how the data element maps to other similar data elements are all characteristics of the metadata about a data element; metadata are standardized characteristics describing the data (aka, “data about data”) that are stored in the data dictionary.

We started with a detailed review of data elements needed to report conformance with well-recognized quality measures in palliative care. We performed a systematic review of all published quality measures relevant to palliative care, supportive oncology, and end-of-life care (manuscript submitted). This systematic review produced 282 measures. Alliance clinical members, all physicians and advance practice providers, with a primary clinical practice in hospice or palliative care, reviewed identified measures, prioritizing those most relevant to community-based clinical practice, most likely to be evaluable from patient report during usual clinical encounters, those not directly impacting clinical decision making (such as omitting measures involving patient satisfaction), those relevant to general, consultative palliative care practice, and most likely to align with pay-for-performance initiatives. Our goal was to generate a data dictionary that contained all numerator and denominator variables to facilitate assessment of conformance for at least 75% of quality measures identified in the systematic review and 100% of those measures the Alliance deemed “high” or “medium” priority. Alliance members categorized 152 of the 282 identified measures as “high” or “medium” priority; remaining measures were set aside for future consideration during iterative updates of QDACT-PC.

We recorded the definitions, numerators, and denominators of published quality metrics, generating a list of variables needed in the QDACT-PC dataset. Additional variables were added to support characterization of the patient population, palliative care service, and patient outcomes (e.g., demographics, site of care, caregiver availability, data of death). Our goal was to choose highly regarded survey questions to generate data elements and to harmonize QDACT-PC data elements with national standards and other palliative care datasets whenever possible. Through a second literature review, we created a candidate list of validated instruments for collecting palliative care data that, in turn, could be used to respond to the selected list of quality metrics (e.g., symptom data collected via the Edmonton Symptom Assessment Scale).^12,13 To ensure that items included in QDACT-PC would be familiar to palliative care clinicians, we drew from well-recognized instruments whenever available. Where validated and clinically feasible instruments were not available, we developed new questions through extensive discussions within the Alliance and by consultation with outside, national experts; queries were refined through three rounds of critique. We also looked to standardized common data element repositories such as the National Cancer Institute's caDSR¹⁴ to ensure that QDACT-PC aligns with national standards whenever possible; similarly, we aligned with the Palliative Care Research Cooperative Group's¹⁵ data elements and the Australian Palliative Care Outcomes Collaboration¹⁶ database to maximize opportunities for data harmonization and integration of datasets in the future. The original PCD data dictionary informed the development of the new QDACT-PC data dictionary to maximize our capabilities to combine PCD and QDACT-PC data for future analyses; however, whenever PCD data elements were performing poorly (e.g., a question was hard to answer in the clinic) or needed to be replaced to meet a national standard (e.g., so QDACT-PC data about constipation aligns with national norms), then we made an intentional decision to retire the PCD data element and refresh it in QDACT-PC.

The result was a revised data dictionary for QDACT-PC with a total of 92 data elements organized into eight clinical quality domains (Table 2). In its current form, the QDACT-PC data elements inform point-of-care conformance assessment of 100% of the 152 high- or medium-priority quality measures, and 82% of all published quality measures identified in the systematic review (Table 3).

Table 2.

Choice of Data Elements in QDACT-PC

Clinical domain	Quality domain	No. of data elements in QDACT-PC	Sources of QDACT-PC question items that align with the data elements in the data dictionary and the questions asked via the user interface
Demographics	Cultural Aspects of Care, Structure, and Processes of Care	20	Consortium-developed, Palliative Care Research Cooperative,¹⁵ National Cancer Institute Cancer Bioinformatics Grid and Cancer Data Standards Registry and Repository, Australian Palliative Care Outcomes Collaborative¹⁶
Symptom Assessment and Management	Physical Aspects of Care, Structure, and Processes of Care	50	Edmonton Symptom Assessment Scale,²⁸ Memorial Symptom Assessment Scale,²⁹ Two-question depression assessment,^30,31 Consortium-developed
Advance Care Planning	Ethical and Legal Aspects of Care	3	Consortium-developed
Psychosocial	Psychiatric and Psychological Aspects of Care, Social Aspects of Care	4	Consortium-developed, UNISCALE QOL,^32,33 “Burden to Family,”³⁴ LASA³⁵
Independence/Functionality	Consortium-Developed	2	AKPS,³⁶ PPS³⁷
Spirituality	Spiritual and Existential Aspects of Care	3	LASA,³⁵ Johnson et al.³⁸ “Are you at peace?” question,³⁹ Consortium-developed
Prognosis	Consortium-Developed	2	Consortium-developed
Transitions/Discharge	Consortium-Developed	7	Consortium-developed
Physician Quality Reporting System	None	4	Center for Medicare and Medicaid Services (CMS) PQRS 2011 list of measures

Table 3.

Quality Measures Informed by QDACT-PC

Quality domain	Quality measure sources ^a	Percent of quality measures informed by QDACT-PC	Quality measures excluded from QDACT–PC
Structure and Processes of Care	3–11	13/14 = 93%	Structural measures involving team structure and competencies
Physical Aspects of Care	1–6, 8, 10, 11	69/85 = 81%	Measures specific to chemotherapy or radiation treatments in cancer patients, those specific to diarrhea and skin rash, workup for anemia, and invasive interventions for pleural effusion, causes, and treatment of delirium
Psychiatric and Psychological Aspects of Care	2–6, 8, 10, 11	12/13 = 92%	Caregiver grief, bereavement, and satisfaction with care
Spiritual and Existential Aspects of Care	3–6, 10, 11	2/4 = 50%	Caregiver satisfaction, value of life
Social Aspects of Care	3–6, 8, 10, 11	1/3 = 33%	Family structure, caregiver preference, caregiver satisfaction with patient life stance
Cultural Aspects of Care	4–6, 10, 11	1/1 = 100%	Most not measured by patient response
Care of the Imminently Dying	3–6, 10, 11	1/1 = 100%	Most involve information sharing with family
Ethical and Legal Aspects of Care	1–6, 10, 11	26/31 = 84%	Patient preferences for locations of care, informed decision making regarding chemotherapy
All domains/measures		125/152 = 82%

Source names and reference:

1. Cancer Quality ASSIST.⁴⁰

2. Acute Care of the Vulnerable Elderly.⁴¹

3. Claessen et al.⁴²

4. Miyashita et al.⁴³

5. National Consensus Project for Quality Palliative Care.⁴⁴

6. National Quality Forum.¹¹

7. Center to Advance Palliative Care.^45,46

8. Twaddle et al.⁴⁷

9. Grunfeld et al.⁴⁸

10. Robert Wood Johnson Foundation Critical Care Workgroup.⁴⁹

11. Hospice-PEACE.⁵⁰

Additional data elements can be added as single data elements or in groups (“modules”) to meet the needs of individual organizations or ongoing pilot projects. For example, a specific palliative care program might want to conduct a deeper evaluation of a question like pain management medications prescribed, or to conduct a continuous quality improvement project focused on optimal management of dysphagia. By maintaining the standard at the data dictionary level, different projects embedded in QDACT-PC can be initiated, modified, terminated, and combined in a modular manner.

Developing the user interface, business rules, and reports

Next, we developed the QDACT-PC user interface and business rules. During the biweekly teleconferences, Alliance clinicians and the QDACT-PC development team defined a “look and feel” of the QDACT-PC interface that would be attractive, intuitive to clinicians, and would fit into workflow. A graphic designer presented color schemes, layouts, and information flow options; clinicians provided iterative feedback until the optimal user interface was identified. A set of business rules was defined. This required agreement within the team as to the goal, intended rule, and expected outcomes, followed by programming the software to meet the rule based on the data collected via the QDACT-PC interface.

Using these specifications, the QDACT-PC software was programmed using an agile design approach, iteratively updating the software in response to input received during the biweekly teleconferences. Since QDACT-PC is intended to be implemented in diverse clinical settings, we purposefully designed a web-based platform, as opposed to an application (“app”), so that QDACT remains compatible with a bevy of current hardware platforms (e.g., Windows, Macintosh, Apple iPad^®, Blackberry^®), Internet browsers (e.g., Microsoft Internet Explorer^®, Google Chrome^®, Apple Safari^®), and Internet protocols (e.g., WiFi, 3G, 4G), while also remaining poised to integrate with future technologies. QDACT-PC was programmed on a software platform called ASQ using open-source Python and Java Script tools with a relational PostgreSQL database. Since the model is based on metadata, it is possible to create various versions of the QDACT-PC interface suited for the differing needs of providers and organizations.

The main user interface is a data dictionary-driven set of quality measure-based needs assessment questions coupled with an intelligent conditional logic and rules engine. In its current iteration, all data are provider entered, including patient-reported variables such as pain, so that providers are asked to document the care they delivered in a quality measure-minded way. Future iterations of QDACT-PC will incorporate a patient and caregiver data collection interface. To meet organizations' needs for quality reporting, QDACT-PC includes a real-time, quarterly, and ad-hoc, graphics-based feedback and reporting tool. This includes immediate feedback on unmet needs identified in real time as providers enter data. For example, a color-coded system reflects whether responses meet an “alarm threshold,” that is, an evidence- or consensus-based parameter (e.g., pain score >7 out of 10) that, when crossed, is brought to the clinician's attention on current and subsequent visits. Other aggregate reports include longitudinal summaries that can be customized to provider, organization, and Alliance requests. Descriptive statistics on patient needs, conformance to quality measures, comparative performance, and changes over time can be numerically and graphically displayed.

Hosting, confidentiality, security, and intellectual property

The QDACT-PC software and data are hosted on a third-party server in a secure data center with multisite, multicontinent backup. To prevent protected health information (PHI) loss, QDACT-PC dynamically sends information to the database through a secure Internet connection. We confirmed compliance with Health Insurance Portability and Accountability Act standards, among others, for data provenance and PHI data protection. For those times when Internet connectivity is not available, such as in rural geographies, a paper version that includes a core set of questions identified by the Consortium for current activities has also been developed; site-based standard operating procedures for paper-based data management are used and information is entered into the QDACT-PC interface by administrators or clinicians as soon as practically possible. Finally, all Consortium members have attested to adhere to privacy and security standards.

We further conducted a data security risk assessment to ensure that QDACT-PC conforms to contemporary data security standards, including data collection, transmission, storage, analysis, and management. We hired a third-party technology security firm to actively challenge the system (e.g., penetration testing) to assess any weaknesses in the server, its network configuration, and the QDACT-PC software; iterative refinements were made to the system to resolve issues identified.

Since the central database includes PHI, we developed Business Associate Agreements with participating organizations. Although data are stored in a central database, each participating organization owns its own data. Data use agreements specify the appropriate uses of site-specific and aggregated data for quality monitoring and research purposes. A Registry Agreement between participating organizations specifies standards for data management, access, and reporting using the centralized database. A collaborative ownership agreement specifies relative ownership of the software. We intentionally used open-sourced data elements, selected from previously published and validated tools.

Discussion

QDACT-PC was developed in a novel manner—through ongoing collaborative discussion between community-based palliative care clinicians, who provided clinical content expertise; academicians, with analytic and registry design expertise; and software developers and data analysts who provided technical support.⁷ The resulting, provider-entered, web-based, quality assessment, and reporting tool is designed to be scalable, open-source, and compatible with diverse IT platforms.

Traditionally, healthcare quality monitoring programs have used paper-based, clinician-conducted, retrospective chart review. In addition to the significant time and effort burden placed by retrospective review on clinicians or their staff, studies have raised concerns about the validity and accuracy of data on quality collected through chart abstraction.^17,18 Major criticisms of retrospective quality data collection focus on lack of information immediacy and analysis in aggregate, and this impairs the ability to affect care of individual patients. QDACT-PC responds to these concerns and is one of the first examples of an electronic, prospective, data collection, and storage system specific to palliative care quality assessment and reporting. It is being implemented in the Alliance and subjected to formal human factors testing. For example, we recently reported the results of usability testing, which revealed time to completion of less than 15 minutes and error rates not inferior to paper-based methods.¹⁹

One of the largest evident benefits²⁰ of an electronic data system featuring prospective data collection and real-time feedback is its support of personalized medicine, allowing clinicians to make patient-specific decisions based on timely, individual-level data. In addition, a robust reporting process that gives both point-of-care and aggregate feedback, identifying, for example, areas of inadequate symptom management reflected in unacceptably high/stable symptom scores, could impact provider behavior²¹ leading to improvements in local standards of care. Furthermore, recent evidence suggests that conformance to quality measures improves patient outcomes, including quality of life²² and survival.²³ These evidences highlight the imperative for practices to monitor quality and address performance gaps.

We have implemented the QDACT-PC tool in the Global Palliative Care Quality Alliance, a subscription-based quality measurement and improvement collaborative for specialty palliative care (www.gpcqa.org). The Alliance allows any specialty palliative care organization in the world to benchmark data with other practices, while also supporting outcomes research needed in the field. For example, QDACT-PC data have informed baseline adherence to the Measuring What Matters set of measures,²⁴ alongside identifying how these measures perform across settings of care usual to our discipline.²⁵ In addition, QDACT-PC data have highlighted the differences in the unmet palliative care needs of various populations^26,27 that will assist palliative care organizations in deploying scarce resources (e.g., social workers) or assessment tools (e.g., patient-reported outcomes) in the targeted populations who may benefit the most.

We have identified several important limitations and concordant steps moving forward in QDACT-PC development. First, to reduce provider dual entry of data, QDACT-PC must be transitioned from a stand-alone system to one that integrates with documentation techniques used currently, including electronic medical records. This also includes developing a patient-reported version, which first required establishing that infrastructure for quality monitoring can be built and implemented in community palliative care. Second, addition of processes through which QDACT-PC data automatically prefill clinical notes or dictation templates will enhance clinical utility and improve data quality. Third, provider self-report introduces response bias, especially when measure conformance may be tied to provider compensation. This underlines the need for data validation steps, such as medical record abstraction and sampling, to authenticate data reported in QDACT-PC. Fourth, provider-facing decision support to provide guidance on evidence-based practices is a necessary step to increase the value of QDACT-PC from a quality monitoring tool to a care improvement solution. All of these issues are currently being addressed.

Conclusions

QDACT-PC is an electronic, point-of-care, clinician-driven approach to quality monitoring and reporting in palliative care. The tool can be used by providers to collect data for reporting on >80% of published quality measures applicable to palliative care and all NQF-nominated clinically important measures.

Acknowledgment

Funding for this work has been provided by The Duke Endowment.

Footnotes

Author Disclosure Statement

A.A. is Chief Medical Officer of Flatiron. Each of the Carolinas Consortium sites owns the data in the Registry generated at that particular site. J.B. is on the Scientific Advisory Board of Archimedes and Meda Pharmaceuticals and Speakers Bureau of Pfizer and Meda. In 2012, Martha Adams received funding for consulting with Med-IQ, a continuing medical education company partnered with Duke, with Sanofi, and with CustomID, a clinical decision support tool for antimicrobial stewardship, licensed to hospitals in the Netherlands. All other authors have no disclosures.

References

Kamal

, Currow

, Ritchie

, et al.: The value of data collection within a palliative care program. Curr Oncol Rep, 2011; 13:308–315.

Meier

, Beresford

: Outpatient clinics are a new frontier for palliative care. J Palliat Med, 2008; 11:823–828.

Lupu

; American Academy of Hospice, Palliative Medicine Workforce Task Force: Estimate of current hospice and palliative medicine physician workforce shortage. J Pain Symptom Manage, 2010; 40:899–911.

Kamal

, Swetz

, Dy

, et al.: Integrating technology into palliative care research. Curr Opin Support Palliat Care, 2012; 6:525–532.

Abernethy

, Kamal

, Wheeler

, Cox

: Management of dyspnea within a rapid learning healthcare model. Curr Opin Support Palliat Care, 2011; 5:101–110.

Bull

, Zafar

, Wheeler

, et al.: Establishing a regional, multisite database for quality improvement and service planning in community-based palliative care and hospice. J Palliat Med, 2010; 13:1013–1020.

Abernethy

, Wheeler

, Bull

: Development of a health information technology-based data system in community-based hospice and palliative care. Am J Prev Med, 2011; 40(5 Suppl 2):S217–S224.

Kamal

, Bull

, Stinson

, et al.: Collecting data on quality is feasible in community-based palliative care. J Pain Symptom Manage, 2011; 42:663–667.

Kavalieratos

, Kamal

, Abernethy

, et al.: Comparing unmet needs between community-based palliative care patients with heart failure and patients with cancer. J Palliat Med, 2014; 17:475–481.

10.

Kamal AH

, Kavalieratos

, Taylor

, et al.: Palliative care needs of patients with cancer living in the community. J Oncol Pract, 2011; 7:382–388.

11.

National Quality Forum: A National Framework for Preferred Practices for Palliative and Hospice Care Quality. Washington D.C., 2006.

12.

Selby

, Chakraborty

, Myers

, et al.: High scores on the edmonton symptom assessment scale identify patients with self-defined high symptom burden. J Palliat Med, 2011; 14:1309–1316.

13.

Chang

, Hwang

, Feuerman

: Validation of the edmonton symptom assessment scale. Cancer, 2000; 88:2164–2171.

14.

Institute NC. NCI Wiki caDSR: https://wiki.nci.nih.gov/display/caDSR/caDSR+Wiki+Home+Page. 2013 (Last accessed February 1, 2016).

15.

Abernethy

, Aziz

, Basch

, et al.: A strategy to advance the evidence base in palliative medicine: Formation of a palliative care research cooperative group. J Palliat Med, 2010; 13:1407–1413.

16.

Eagar

, Watters

, Currow

, et al.: The Australian Palliative Care Outcomes Collaboration (PCOC)—measuring the quality and outcomes of palliative care on a routine basis. Aust Health Rev, 2010; 34:186–192.

17.

Dresselhaus

, Luck

, Peabody

: The ethical problem of false positives: A prospective evaluation of physician reporting in the medical record. J Med Ethics, 2002; 28:291–294.

18.

Luck

, Peabody

, Dresselhaus

, et al.: How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record. Am J Med, 2000; 108:642–649.

19.

Kamal

, Kavalieratos

, Bull

, et al.: Usability and acceptability of the QDACT-PC, an electronic point-of-care system for standardized quality monitoring in palliative care. J Pain Symptom Manage, 2015; 50:615–621.

20.

Beckmann

, Bohringer

, Carless

, et al.: Evaluation of two methods for quality improvement in intensive care: Facilitated incident monitoring and retrospective medical chart review. Crit Care Med, 2003; 31:1006–1011.

21.

Hillman

, Ripley

, Goldfarb

, et al.: The use of physician financial incentives and feedback to improve pediatric preventive care in medicaid managed care. Pediatrics, 1999; 104(4 Pt 1):931–935.

22.

Kamal

AH BJ

, Zhong

, Abernethy

: Linking quality metrics to outcomes through prospective, point-of-care quality data collection in community-based palliative cancer care Paper presented at: Chicago Supportive Oncology Conference; October 27, 2011, 2011; Chicago.

23.

Lennes

IT TJ

, Greer

, Jacobson

, et al.: ASCO Quality Oncology Practice Initiative (QOPI) quality measures for smoking cessation and survival in metastatic NSCLC. Paper presented at: American Society of Clinical Oncology (ASCO) 2011 Annual Meeting2011; Chicago.

24.

, Kiley

, Ast

, et al.: Measuring what matters: Top-ranked quality indicators for hospice and palliative care from the american academy of hospice and palliative medicine and hospice and palliative nurses association. J Pain Symptom Manage, 2015; 49:773–781.

25.

Kamal

, Bull

, Ritchie

, et al.: Adherence to measuring what matters measures using point-of-care data collection across diverse clinical settings. J Pain Symptom Manage, 2016; 51:497–503.

26.

Hochman

, Wolf

, Zafar

, et al.: Comparing unmet needs to optimize quality: Characterizing inpatient and outpatient palliative care populations. J Pain Symptom Manage, 2016. PMID: 27046299.

27.

Kamal

, Bull

, Wolf

, et al.: Unmet needs of African Americans and Whites at the time of palliative care consultation. Am J Hosp Palliat Care, 2016. PMID: 26888883.

28.

Bruera

, Kuehn

, Miller

, et al.: The Edmonton Symptom Assessment System (ESAS): A simple method for the assessment of palliative care patients. J Palliat Care, 1991; 7:6–9.

29.

Portenoy

, Thaler

, Kornblith

, et al.: The memorial symptom assessment scale: An instrument for the evaluation of symptom prevalence, characteristics and distress. Eur J Cancer, 1994; 30A:1326–1336.

30.

Mitchell

: Are one or two simple questions sufficient to detect depression in cancer and palliative care? A Bayesian meta-analysis. Br J Cancer, 2008; 98:1934–1943.

31.

Payne

, Barry

, Creedon

, et al.: Sensitivity and specificity of a two-question screening tool for depression in a specialist palliative care unit. Palliat Med, 2007; 21:193–198.

32.

Sloan

, Loprinzi

, Kuross

, et al.: Randomized comparison of four tools measuring overall quality of life in patients with advanced cancer. J Clin Oncol, 1998; 16:3662–3673.

33.

Spitzer

, Dobson

, Hall

, et al.: Measuring the quality of life of cancer patients: A concise QL-index for use by physicians. J Chronic Dis, 1981; 34:585–597.

34.

Steinhauser

, Bosworth

, Clipp

, et al.: Initial assessment of a new instrument to measure quality of life at the end of life. J Palliat Med, 2002; 5:829–841.

35.

Bretscher

, Rummans

, Sloan

, et al.: Quality of life in hospice patients. A pilot study. Psychosomatics, 1999; 40:309–313.

36.

Abernethy

, Shelby-James

, Fazekas

, et al.: The Australia-modified Karnofsky Performance Status (AKPS) scale: A revised scale for contemporary palliative care clinical practice [ISRCTN81117481]. BMC Palliat Care, 2005; 4:7.

37.

Anderson

, Downing

, Hill

, et al.: Palliative performance scale (PPS): A new tool. J Palliat Care, 1996; 12:5–11.

38.

Johnson

, Piderman

, Sloan

, et al.: Measuring spiritual quality of life in patients with cancer. J Support Oncol, 2007; 5:437–442.

39.

Steinhauser

, Voils

, Clipp

, et al.: “Are you at peace?”: One item to probe spiritual concerns at the end of life. Arch Int Med, 2006; 166:101–105.

40.

, Lorenz

, O'Neill

, et al.: Cancer quality-ASSIST supportive oncology quality indicator set: Feasibility, reliability, and validity testing. Cancer, 2010; 116:3267–3275.

41.

Wenger

, Shekelle

: Assessing care of vulnerable elders: ACOVE project overview. Ann Int Med, 2001; 135(8 Pt 2):642–646.

42.

Claessen

, Francke

, Belarbi

, et al.: A new set of quality indicators for palliative care: Process and results of the development trajectory. J Pain Symptom Manage, 2011; 42:169–182.

43.

Miyashita

, Nakamura

, Morita

, Bito

: Identification of quality indicators of end-of-life cancer care from medical chart review using a modified Delphi method in Japan. Am J Hosp Palliat Care, 2008; 25:33–38.

44.

National Consensus Project for Quality Palliative Care: Clinical practice guidelines for quality palliative care, executive summary. J Palliat Med, 2004; 7:611–627.

45.

Weissman

, Meier

: Center to advance palliative care inpatient unit operational metrics: Consensus recommendations. J Palliat Med, 2009; 12:21–25.

46.

Weissman

, Morrison

, Meier

: Center to advance palliative care palliative care clinical care and customer satisfaction metrics consensus recommendations. J Palliat Med, 2010; 13:179–184.

47.

Twaddle

, Maxwell

, Cassel

, et al.: Palliative care benchmarks from academic medical centers. J Palliat Med, 2007; 10:86–98.

48.

Grunfeld

, Lethbridge

, Dewar

, et al.: Towards using administrative databases to measure population-based indicators of quality of end-of-life care: Testing the methodology. Palliat Med, 2006; 20:769–777.

49.

Mularski

, Curtis

, Billings

, et al.: Proposed quality measures for palliative care in the critically ill: A consensus from the Robert Wood Johnson Foundation Critical Care Workgroup. Crit Care Med, 2006; 34(11 Suppl):S404–S411.

50.

Schenck

, Rokoske

, Durham

, et al.: The PEACE Project: Identification of quality measures for hospice and palliative care. J Palliat Med, 2010; 13:1451–1459.