Measuring Palliative Care Quality for Seriously Ill Hospitalized Patients

Abstract

Objective:

Hospice and palliative care providers need ways to measure and improve care processes. We tested feasibility, usability, reliability, and validity of Prepare, Embrace, Attend, Communicate, Empower (PEACE) quality measures for palliative care.

Methods:

Trained research nurses abstracted data from medical records to generate quality measures for a random sample of 460 seriously ill patients without, and 102 patients with, specialty palliative care (SPC) services.

Results:

Patient age ranged from 16 to 99 years, 50% were women, and 24% were African American. Of 34 PEACE quality measures, 17 were feasible for hospital palliative care. Inter-rater reliability was high (κ>0.80) for all but two quality measures. Face validity was endorsed by clinical service leaders, and construct validity was established by higher scores for patients receiving SPC. Comprehensive palliative care assessment was completed for only 10% of seriously ill hospitalized patients, compared with 56% of patients with SPC (p<0.001). Patients with moderate or severe pain were more likely to have a clinical assessment with SPC (67% versus 42%, p=0.002). Patients with SPC more often received attention for their emotional and spiritual needs (64% versus 40%, p<0.001) and documentation of preferences for life-sustaining treatments (91% versus 59%, p>0.001). Usability was endorsed by service leaders, who initiated two practice improvement projects.

Conclusion:

PEACE quality measures are feasible and reliable, and may be useful to examine and improve the quality of palliative care for seriously ill hospitalized patients as well as for patients in hospice. Research is needed to test measures for actionability and responsiveness to intervention.

Introduction

Seriously ill patients desire relief from pain and symptoms, attention to emotional and spiritual needs, and informed decision making. The Institute of Medicine recommended developing measurement for palliative care quality improvement and research.^1,2 In 2004, the National Consensus Project for Quality Palliative Care (NCP) defined practice guidelines for palliative care.³ The National Quality Forum (NQF) subsequently reviewed and endorsed NCP guidelines and preferred practices.^4,5

To understand when practices do or do not meet these guidelines, providers need quality measures that are feasible, reliable, and valid. Over 100 published quality measures address palliative care domains; however, few have been tested with adequate rigor to justify use.^6,7 In preparation for hospice quality requirements, the Centers for Medicare & Medicaid Services (CMS) contracted with the Carolinas Center for Medical Excellence to conduct the Prepare, Embrace, Attend, Communicate, Empower (PEACE) project, to recommend quality measures and data collection procedures for hospice and palliative care patients. Although PEACE quality measures were initially tested in hospice care, they were designed to address care for the broader palliative care population.^8,9 The aims of the current study were to examine the feasibility, inter-rater reliability, validity, and usability of the PEACE quality measures for seriously ill hospitalized patients.

Methods

Study sites

The study site was a public academic medical center with 799 inpatient beds in North Carolina, with an integrated electronic medical record inclusive of medical, nursing, pharmacy, and care management. Physicians, nurse practitioners, and social workers with added training and board certification provided interdisciplinary palliative care consultation. All study procedures were approved by the Institutional Review Board for the University of North Carolina at Chapel Hill School of Medicine.

PEACE quality measures

Each measure has a numerator that defines an evidence-based palliative care process, and a denominator that defines the patient population for which this care process is expected. Potential quality measures were identified from scientific publications and quality improvement projects, and all potential quality measures were pilot tested by volunteer hospice organizations.^10–16 Data on potential measures was reviewed by a 14-member Technical Expert Panel (TEP) with extensive experience in hospice and palliative care, measurement and quality improvement. Measures receiving highest ratings for importance, scientific soundness, usability and feasibility formed the final set of 34 PEACE quality measures.^8,17

Data collection

Medical record abstractions were completed for a random sample of 460 seriously ill patients without, and 102 patients with, specialty palliative care (SPC) services. Patients without SPC were admitted between December 2007 and March 2010 for at least 1 day to one of four inpatient services with high proportions of seriously ill patients: medical and surgical intensive care units, an acute care for the elderly unit, and the medical oncology service. All intensive care and geriatric care patients, and oncology patients with stage IV carcinoma were eligible to be sampled. Because palliative care domains become more relevant closer to death, patients dying in hospital were oversampled to ensure a final ratio of one decedent to one live discharge.

All data collection was completed by two trained research nurses using a structured abstraction tool that separates numerator from denominator data. Demographic data were collected, including diagnoses, age, ethnicity, and gender of the patient, and presence or absence of communication barriers, including confusion or sedation, history of dementia, or primary language other than English. Abstractors were given a detailed codebook with highly specified data definitions. For example, evidence for a screen for pain was confirmed by a numeric score or by designation of none, mild, moderate, or severe as recorded in a narrative note. The presence or absence of other symptoms, including dyspnea, nausea and constipation, was accepted without severity scoring. Pain assessment was “comprehensive” if it included at least five elements: pain severity, duration, frequency, location, character, what relieves or worsens, and effect on function. Any reference by any provider to psychological stressors, emotional distress, or family dynamics was accepted as evidence for attention to emotional needs. Any reference to religious or spiritual concerns or beliefs, or presence of a chaplaincy visit, was accepted for evidence of attention to spiritual needs. Evidence for advance care planning required recording the name and contact information for a surrogate decision maker, and evidence of a discussion of treatment options. Simple statement of code status was not accepted as evidence for discussion of patient preferences.

Feasibility of PEACE quality measures for hospital care

Feasibility was assessed by testing operational definitions for use in hospital care of seriously ill patients. Five PEACE measures were excluded from all data collection because they were organizational rather than patient level, addressed adverse events only, or utilized data rarely available in hospital records. Excluded measures addressed procedures for symptom screening, adverse event rates, weekly assessment of bowel function, and two measures derived from post-death surveys.

Reliability of PEACE quality measures

Two research nurses independently abstracted quality measures data on 20 patient records. Intra-rater reliability between the two research nurses was assessed using κ statistics.

Validity

Face validity of PEACE quality measures for in hospital use was addressed using review and feedback from physician and nurse stakeholders on each service. Investigators shared operational definitions and preliminary data reports, and, clinicians responded to questions on the meaningfulness, appropriateness and actionability of measures. They were asked about the accuracy of measures, and inclusion or exclusion criteria for their patient population.

Construct validity was tested by comparing the PEACE quality measures for patients with versus those without specialty interdisciplinary palliative care.

Usability

To evaluate the usability of quality measures, nursing and physician leaders from each of the four clinical services received feedback of the final, full data report of all quality measures for their service, compared with other participating services. They were asked about the potential usefulness of the quality measure data, and to consider a voluntary quality improvement initiative in one domain of palliative care based on the data report. Usability was defined by the number and depth of quality initiatives selected.

Statistical analysis

All quality measures were calculated as percentages of patients in the denominator population who received the recommended care. Kappa statistics were used to test for inter-rater reliability, and χ² statistics were used to test for significance when comparing the percent of patients with and without SPC for whom each quality measure was met.

Results

Sample characteristics

The study sample was 460 patients without SPC and 102 patients with SPC (Table 1). Consistent with oversampling of decedent records, 55% of patients without SPC died in hospital. Age ranged from 16 to 99 years, with the mean age being 61. Although patients were predominantly Caucasian (65%), 24% were African American and 4% were Hispanic. The most common diagnoses were infections (37%), cancer (34%), pulmonary diseases (29%), and neurologic diseases (21%). Decedents were not oversampled for the 102 sequential patients receiving SPC, but 38% of these patients died in hospital.

Table 1.

Patient Sample Characteristics

	All patients		Decedents only
Characteristics	Combined services (n=460)^a	Palliative care (n=102)	Combined services (n=255)^a	Palliative care (n=39)
Age	61	65	62	65
Female	49%	57%	49%	59%
Race/Ethnicity
African American	24%	25%	22%	23%
Caucasian	65%	64%	66%	56%
Hispanic/Latino	4%	6%	3%	13%^**
Other	6%	4%	8%	5%
Deaths in hospital	55%	38%^*	100%	100%
Major diagnoses
Cardiac	19%	23%	24%	28%
Cancer	34%	47%^*	32%	44%
Neurologic	21%	28%	24%	31%
Pulmonary	29%	19%^*	36%	28%
Trauma	10%	1%^*	9%	3%
Infection	37%	28%	44%	31%
Other	55%	44%^*	57%	23%^**
Dementia	7%	13%^*	8%	15%^**
Non-English speaking	1%	4%^*	2%	8%^**
Confused, sedated, or non verbal	43%	54%^*	44%	67%^**
Treatment in the last week of life
CPR			11%	3%^**
Ventilator			25%	18%^**
ICU			71%	41%^**
Dialysis			11%	5%^**
Chemotherapy			2%	5%

Excludes patients who received a palliative care consultation.

p value is <0.05 when all patients from the combined services were compared with all patients from the palliative care service.

p value is <0.05 when decedent patients from the combined services were compared with decedent patients from the palliative care service.

CPR, cardiopulmonary resuscitation; ICU, intensive care unit.

Feasibility

After 60 pilot record abstractions, investigators found 17 PEACE measures feasible for this population, as defined in Table 2. Complete data collection for all 17 measures averaged 45 minutes per record.

Table 2.

Inter-Rater Reliability for Palliative Care Quality Measures

Quality measures by domain	Kappa
Structure and process
1. Percent of all patients admitted for >1 day who had a screening for physical symptoms (pain, dyspnea, nausea, and constipation) during the admission visit	0.74
2. Percent of all patients admitted for >1 day who had comprehensive assessment (pain, dyspnea, nausea, constipation, emotional/social screen, spiritual needs) completed within 5 days of admission	1
Care for physical symptoms: pain
3. Percent of all patients screened for pain during the admission visit	1
4. For patients who screened positive for moderate to severe pain on admission, the percent with any physician clinical assessment within 24 hours of screening	0.94
5. For patients who screened positive for moderate to severe pain on admission, the percent with medication or non-medication treatment, within 24 hours of screening	1
6. For patients who screened positive for moderate to severe pain on admission, the percent who had an order for pain medication within 24 hours of screening	1
Care for physical symptoms: dyspnea
7. Percent of all patients who were screened for shortness of breath during the admission visit	0.91
8. For patients who screened positive for dyspnea, the percent who receive medication or non-medication treatment within 24 hours of screening	0.89
9. For patients who screened positive for dyspnea, the percent who had a second assessment within 24 hours	0.71
Care for physical symptoms: other conditions
10. Percent of patients on opioids who had a bowel regimen ordered	0.86
Emotional and spiritual aspects of care
11. Percent of patients with chart documentation of a discussion of spiritual or religious concerns	1
12. Percent of patients with chart documentation of a discussion of emotional or psychological needs	1
Care of the imminently dying: last week of life
13. For patients who died, the percent who had moderate to severe pain at any time in the last week of life	1
Communication and decision making: ethical and legal aspects of care
14. Percent of patients with chart documentation of their preference for life- sustaining treatments	1
15. Percent of patients with chart documentation of an advance directive or discussion that there is no advance directive	1
16. Percent of patients with name and contact information for their surrogate	1
17. Percent of patients with impaired decision making (dementia, coma or other altered mental status) with name and contact information for their surrogate	1

Inter-rater reliability

In independent abstractions for 20 randomly selected records, two trained nurse abstractors achieved very good inter-rater reliability. Kappa statistics were 0.91–0.93 for identification of the two denominator subgroups (patients with moderate or severe pain, patients with dyspnea). Kappas statistics were 0.71-1.0, and all but two quality measures had very good reliability (κ>0.80) (Table 2).

Construct validity: Structure and process of care

Comprehensive palliative care assessment on admission, including screening for pain, dyspnea, nausea, constipation, and emotional and spiritual needs, was completed for only 10% of seriously ill hospitalized patients, compared with 56% of patients who received SPC (p<0.001) (Table 3). Screening for four common physical symptoms was more often completed for patients who received SPC (86% versus 23%, p<0.001).

Table 3.

Palliative Care Quality Measures with and without Specialized Palliative Care

Quality measures (n)	Combined services ^a	Palliative care	p value
Structure and process
1. Screen for symptoms	23% (460)	86% (102)	<0.001
2. Comprehensive assessment	10% (460)	56% (102)	<0.001
Pain
3. Screen for pain	100% (460)	99% (102)	0.494
Patients with moderate/severe pain	51% (458)	51% (101)	0.977
4. Clinical assessment for pain	42% (232)	67% (51)	0.002
5. Received treatment	96% (232)	100% (51)	0.131
6. Order for pain medication	95% (232)	100% (51)	0.113
Order for opioid	89% (232)	94% (51)	0.676
Dyspnea
7. Screen for shortness of breath	95% (460)	100% (102)	0.016
Patients with dyspnea	25% (435)	28% (102)	0.514
8. Received treatment	93% (110)	96% (29)	0.479
9. Second assessment	85% (110)	83% (29)	0.815
Other conditions
10. Bowel regimen if opioid	44% (206)	44% (48)	0.957
Emotional and spiritual
11. Spiritual/religious concerns addressed	55% (460)	55% (102)	0.986
12. Emotional/psychological needs	40% (460)	64% (102)	<0.001
Last week of life
13. Moderate/severe pain	61% (255)	72% (39)	0.206
Severe pain	54% (155)	79% (28)	0.004
Mostly asleep/unconscious	58% (255)	46% (39)	0.163
Communication and decision making
14. Life-sustaining treatments	59% (460)	91% (102)	<0.001
Advance care planning note	11% (460)	21% (102)	0.009
15. Advance directive or discussion	78% (460)	89% (102)	0.009
DNR documented	55% (460)	71% (102)	0.004
Comfort or palliative care goals	33% (460)	64% (102)	<0.001
Hospice at discharge	3% (460)	25% (102)	<0.001
16. Contact info for surrogate	93% (460)	92% (102)	0.691
17. Impaired with documentation of surrogate	94% (213)	93% (58)	0.718

Excludes patients who received a palliative care consultation.

DNR, Do not resuscitate order.

Construct validity: Physical symptoms

Screening for pain was nearly universal for all seriously ill patients, and half had moderate or severe pain. Patients with moderate or severe pain were more likely to have a clinical assessment of pain if seen by SPC (67% versus 42%, p=0.002). Medication was prescribed for nearly all patients in pain, and opioids were prescribed for 89% of patients without SPC. Screening for dyspnea was done for all patients receiving SPC (100% versus 95%, p=0.016). One quarter of patients screened had dyspnea, and nearly all of these patients received some form of treatment.

Construct validity: Emotional and spiritual needs

Spiritual needs were acknowledged for 55% of all patients. Emotional or psychological needs were more often acknowledged if SPC was provided (64% versus 40%, p<0.001).

Construct validity: Ethical and legal

Patients with SPC were more likely to have documentation of their preferences for life-sustaining treatments (91% versus 59%, p>0.001), and were somewhat more likely to have a documented discussion about advance directives (89% versus 78%, p=0.009). The study site has a specialized advance care planning section of the medical record to facilitate rapid review in emergency care; patients with SPC were more likely to have their treatment preferences in this designated section (21% versus 11%, p=0.009).

Construct validity: Care for actively dying patients

We compared quality measures for patients who died with or without SPC. Similar differences in care during active dying were found for symptom screens, comprehensive assessment, pain assessment, attention to emotional needs, and approaches to decision making (Table 4).

Table 4.

Palliative Care Quality Measures with and without Specialized Palliative Care - Decedents

Quality measures (n)	Combined services ^a	Palliative care	p value
Structure and process
1. Screen for symptoms	19% (255)	79% (39)	<0.001
2. Comprehensive assessment	9% (255)	62% (39)	<0.001
Pain
3. Screen for pain	99% (255)	100% (39)	0.579
Patients with moderate/severe pain	46% (253)	49% (39)	0.773
4. Clinical assessment	29% (117)	74% (19)	<0.001
5. Received treatment	97% (117)	100% (19)	0.4133
6. Order for pain medication	96% (117)	100% (19)	0.359
Order for opioid	92% (117)	95% (19)	0.655
Dyspnea
7. Screen for shortness of breath	96% (255)	100% (39)	0.186
Patients with dyspnea	29% (244)	41% (39)	0.120
8. Received treatment	94% (70)	100% (16)	0.328
9. Second assessment	81% (70)	88% (16)	0.564
Other conditions
10. Bowel regimen if on opioid	48% (108)	39% (18)	0.466
Emotional and spiritual
11. Spiritual/religious concerns addressed	79% (255)	74% (39)	0.529
12. Emotional/psychological needs	46% (255)	72% (39)	0.003
Last week of life
13. Moderate/severe pain	61% (255)	72% (39)	0.206
Severe pain	54% (155)	79% (28)	0.004
Mostly asleep/unconscious	58% (255)	46% (39)	0.163
Communication and decision making
14. Life-sustaining treatments	91% (255)	100% (39)	0.046
Advance care planning note	18% (255)	31% (39)	0.063
15. Advance directive or discussion	74% (255)	90% (39)	0.033
DNR documented	81% (255)	95% (39)	0.031
Comfort or palliative care goals	56% (255)	85% (39)	0.001
Hospice at discharge	0% (255)	0% (39)	1
16. Contact info for surrogate	92% (255)	92% (39)	0.974
17. Impaired with documentation of surrogate	95% (121)	93% (28)	0.644

Excludes patients who received a palliative care consultation.

DNR, do not resuscicitate order.

Face validity and usability

Medical intensive care unit (MICU), surgical intensive care unit (SICU), geriatrics, and oncology medical and nursing leaders reviewed quality measures for face validity and usability. Service leaders broadly endorsed face validity, while discussing potential additional exclusion criteria for denominator populations. For example, intensive care providers cared for many patients, such as those with diarrheal illnesses or bowel obstruction, who could be harmed by reflexive use of bowel medication with opioids. They discussed lack of clinical confidence in pain scores when patients are ventilated or delirious, and recommended these patients be excluded until research improves nonverbal pain scores.

When asked to consider the usability of measures for quality improvement, service leaders gave varied responses. Geriatrics service leaders began a quality improvement project to enhance documentation of advance care planning. Intensive care unit leaders endorsed strengthening sedation protocols that incorporate opioids as a standard of intensive care, and are now collaborating on a research study of the validity of nonverbal pain assessment instruments. Oncology service leaders did not provide broad endorsement for this measure set, and expected their quality improvement efforts to be defined by quality metrics developed for specialty cancer care. Overall, two of the four services initiated efforts to improve quality of care based on data feedback.

Discussion

The PEACE quality measures are feasible, reliable, and valid for use with seriously ill hospitalized patients with palliative care needs. The most promising quality measures are screening for pain, dyspnea, and other symptoms, comprehensive assessment, assessment of moderate to severe pain, addressing emotional or psychological needs, and documentation of preference for life-sustaining treatments.

In a recent systematic review, Lorenz found strong evidence for effective assessment and treatment of cancer pain, dyspnea from chronic lung disease, depression associated with cancer, and routine advance care planning.¹⁸ This study provides initial evidence that quality measures are reliable and valid for these domains of practice. Quality measures have many potential uses: in research, clinical quality improvement, benchmarking, public reporting, or payment approaches to reward quality of care. Uses other than research and internal quality improvement may demand further study to ensure that these practices are actionable by providers and show responsiveness to practice innovation, and that efforts to improve a specified practice do not have unintended adverse consequences on patient care.

The description of palliative care processes provided by the PEACE quality measures suggests opportunities for improvement, but does not define benchmarks or optimal levels for these care processes. SPC providers met quality measures more often than other providers serving seriously ill patients, and quality could improve through expanded access to palliative care consultation. However, palliative care is highly responsive to patient preferences, and for some care processes the optimal level of performance may not be 100% adherence.

PEACE quality measures show good concordance with measures developed for other seriously ill patients, particularly Assessing Care of Vulnerable Elders (ACOVE) and Cancer Quality-Addressing Symptoms, Side Effects, and Indicators of Supportive Treatment (ASSIST).^19–22 These investigators have also concluded that screening for pain, initiating a bowel regimen with opioid treatment, performing a comprehensive assessment, and documenting presence of a surrogate decision maker are essential components of high quality care. PEACE and ACOVE include quality measures for performing a comprehensive assessment for pain, treating pain in a timely manner, documenting pain in the last week of life, screening for dyspnea, and documenting the patients' preference for life-sustaining treatments.

Limitations

Results should be considered in light of some limitations. First, as patients were not randomly assigned to receive SPC, differences between patients with and without SPC may be related to unmeasured patient characteristics. This effect is real, but moderated by the selection of process measures, which collect data on care practices, rather than outcomes. Second, high inter-rater reliability was achieved after careful training of abstractors on operational definitions. We utilized approaches to enhance reproducibility of abstraction, including very simple operational definitions for target populations and use of inclusive operational definitions for the source of documentation sought. Third, quality measures reliant on chart documentation are a proxy for the actual care providers deliver and patients experience; some care may have been given but not documented, whereas other palliative care practices were purposefully omitted when judged in the best interests of an individual patient. Fourth, these data were collected at a single institution; further reliability and validity data from other acute palliative care populations is desirable. Finally, these patients were often cared for by more than one medical team or were transferred between teams as their needs changed. In some cases, the attribution of care to specific medical providers was unclear, and judgment was required. Quality measures for seriously ill patients may be difficult to attribute at the individual provider level, although they may be applicable at the team or organizational level.

Conclusion and Implications

This study provides initial evidence for the feasibility, validity, and reliability of the PEACE quality measures for use in hospital-based palliative care. These quality measures may be useful as organizational metrics to track and improve hospice and palliative care practice. The numbers of seriously ill patients far exceed current workforce capacity for SPC in the United States. The PEACE quality measures may also be used to examine unmet palliative care needs, to motivate improved palliative care by primary providers, or to track the effectiveness of adding specialty palliative care. As hospice and palliative care develops its evidence base, these measures of the quality of care processes should be further tested for their relationship to patient and family-reported outcomes.

Footnotes

Acknowledgment

Supported by an Investments for the Future award from the UNC Health System.

Author Disclosure Statement

No competing financial interests exist.

References

Field

, Cassel

. Approaching Death: Improving Care at the End of Life. Washington, DC: National Academy Press, 1997.

Lunney

, Foley

, Smith

, Gelband

. Describing Death in America: What We Need to Know. Washington, DC: National Academy Press, 2003.

Clinical Practice Guidelines for Quality Palliative Care, 2004. National Consensus Project: Pittsburgh, Pennsylvania www.nationalconsensusproject.org. 2009 August 9.

National Quality Forum (NQF). National Priorities for Healthcare Quality Measurement and Reporting: A Consensus Report. Washington, DC: NQF, 2004.

National Quality Forum (NQF). National Framework and Preferred Practices for Palliative and Hospice Care. Washington, DC: NQF, 2007.

Pasman

HRW

, Brandt

, Deliens

, Francke

. Quality indicators for palliative care: a systematic review. J Pain Symptom Manage, 2009; 38:145–156.

Lorenz

, Lynn

, Dy

, Wilkinson

, Mularski

, Shugarman

, Hughes

, Asch

, Rolon

, Rastegar

, Shekelle

. Quality measures for symptoms and advance care planning in cancer: a systematic review. J Clin Oncol, 2006; 24:4933–4938.

Hanson

, Scheunemann

, Zimmerman

, Rokoski

, Schenck

. The PEACE project review of clinical instruments for hospice and palliative care. J Palliat Med, 2010; 13:1253–1260.

Schenck

, Rokoske

, Durham

, Cagle

, Hanson

. The PEACE project: identification of quality measures for hospice and palliative care. J Palliat Med, 2010; 13:1451–1459.

10.

Centers for Medicare & Medicaid Services. CMS Hospice Rural Demonstration Domains, Indicators, and Resources for Measurement. 2010. http://www.cms.hhs.gov/DemoProjectsEvalRpts/downloads/RMBH_QA.pdf. 2012 May 28.

11.

Chodosh

, Ferrell

, Shekelle

, Wenger

. Quality indicators for pain management in vulnerable elders. Ann Intern Med, 2001; 135:731–735.

12.

Wenger

, Rosenfeld

. Quality indicators for end–of-life care in vulnerable elders. Ann Intern Med, 2001; 135:677–685.

13.

National Quality Forum. Compendium 2000–2005. 2006. www.qualityforum.org/Publications/2006/05/Compendium_2000–2005.aspx. 2010 March 22.

14.

Lorenz

, Lynn

, Dy

, Hughes

, Mularski

, Shugarman

, Wilkinson

. Cancer care quality measures: symptoms and end-of-life care. Evid Rep Technol Assess, 2006; 137:1–77.

15.

Twaddle

, Maxwell

, Cassel

, Liao

, Coyne

, Usher

, Amin

, Cuny

. Palliative care benchmarks from academic medical centers. J Palliat Med, 2007; 10:86–98.

16.

National Quality Measures Clearinghouse. Agency for Healthcare Research and Quality. 2010. www.qualitymeasures.ahrq.gov/. 2010 March 22.

17.

Carolinas Center for Medical Excellence: The Hospice PEACE Project. www.thecarolinascenter.org/default.aspx?pageid=24. 2010 November 29.

18.

Lorenz

, Lynn

, Dy

, Shugarman

, Wilkinson

, Mularski

, Morton

, Hughes

, Hilton

, Maglione

, Rhodes

, Rolon

, Sun

, Shekelle

. Evidence for improving palliative care at the end of life: a systematic review. Ann Intern Med, 2008; 148:147–159.

19.

Wenger

, Solomon

, Roth

, MacLean

, Saliba

, Kamberg

, Rubenstein

, Young

, Sloss

, Louie

, Adams

, Chang

, Venus

, Schnelle

, Shekelle

. The quality of medical care provided to vulnerable community-dwelling older patients. Ann Intern Med, 2003; 139:740–747.

20.

Walling

, Asch

, Lorenz

, Roth

, Barry

, Kahn

, Wenger

. The quality of care provided to hospitalized patients at the end of life. Arch Intern Med, 2010; 170:1057–1063.

21.

, Lorenz

, ONeill

, Asch

, Walling

, Tisnado

, Antonio

, Malin

. Cancer quality ASSIST supportive oncology quality indicator set. Cancer, 2010; 116:3267–3275.

22.

, Asch

, Lorenz

, Weeks

, Sharma

, Wolff

, Malin

. Quality of end of life care for patients with advanced cancer in an academic medical center. J Palliat Med, 2011; 14:451–457.