Quality of End-of-Life Care for Patients with Advanced Cancer in an Academic Medical Center

Abstract

Purpose:

We assessed key aspects of the quality of end-of-life care using validated explicit process quality measures in an academic medical center (hospital and cancer center) before expanding to a broader palliative care initiative.

Methods:

We evaluated 21 indicators most relevant to end-of-life care from the Cancer Quality-ASSIST supportive oncology indicator set for 238 patients with advanced/metastatic solid tumors who died between 2–15 months after diagnosis. These included outpatient and hospital indicators for cancer symptoms and information and care planning that met criteria for feasibility, reliability, and validity. We abstracted detailed information from medical records to specify the necessary data elements.

Results:

Overall adherence was 53% (95% confidence interval [CI], 50%–56%); this varied widely among indicators. Adherence was highest for pain indicators; in particular, 97% of eligible subjects' hospitalizations had documented screening for pain, and, after an outpatient pain medication was changed, 97% of patients had a pain assessment at the subsequent visit. For other symptoms, adherence ranged from 0% for documentation of life expectancy for patients starting parenteral or enteral nutrition to 87% for assessment of nausea or vomiting on hospital admission. For information and care planning, results ranged from 6% for documentation of ventilation preferences prior to intubation to 68% for documented communication of risks and benefits or prognosis prior to starting chemotherapy.

Conclusion:

In conclusion, Cancer Quality-ASSIST indicators are useful for practical quality assessment of cancer end-of-life care in an academic medical center. These results will serve as useful data for targeting areas for quality improvement and measuring progress.

Introduction

Patients with advanced cancer benefit greatly from supportive care to reduce symptoms and ensure that care is consistent with their goals. Good symptom management is critical to improving quality of life in patients with advanced cancer,¹ and good communication and care planning are associated with better end-of-life outcomes.² While comprehensive data are lacking, patients' supportive oncology needs may often not be well met in oncology practice.³ For example, in a study of audiotaped initial oncology consultations for patients with terminal cancer, fewer than half of oncologists offered alternatives to chemotherapy as an option. In addition, only 58% of patients were informed of their life expectancy and only 60% were aware of uncertainty about the benefit of chemotherapy.⁴

Good supportive oncology care in pain and other domains requires detailed assessment, treatment, and followup.⁵ Indicators to measure quality across domains relevant to end-of-life care and across the spectrum of care are needed to assess where quality improvement efforts should be targeted and evaluate their impact. Although well-developed measures now exist to measure potential overutilization of cancer interventions, family and patient perceptions, and other aspects of care,⁶ there have been no studies using comprehensive, well-validated process measures to evaluate the quality of oncology end-of-life care.

Although outcome measures are the ultimate goal of improving quality of care, process measures are necessary to identify whether care is meeting national standards (such as the National Quality Consensus Project for Palliative Care guidelines)⁷ and the areas where quality improvement efforts should be targeted. Process measures have the advantage of focusing on discrete actions taken by clinicians; measuring those actions for quality assessment requires the documentation of those actions. Although that is sometimes problematic, in the case of supportive care, documentation may reflect a critical aspect of good quality. For example, care planning cannot be effective if preferences for future management are not available to other clinicians who may be called upon to fulfill them. The lack of valid process measures for evaluating end-of-life care has made it challenging for programs to identify targets for quality improvement or the impact of their interventions.

We developed the Cancer Quality-ASSIST (Assessing Symptoms Side Effects and Indicators of Supportive Treatment) quality indicators to address this gap in quality measurement. The ASSIST indicators provide a systematic way to measure the quality of the process of care for managing symptoms and informational needs of cancer patients and represent the most comprehensive toolkit available for evaluating supportive cancer practice.⁸ ASSIST addresses symptoms of cancer, including pain, dyspnea, depression, nausea and vomiting, fatigue, and anorexia; symptoms of cancer treatment; and information and care planning. The feasibility, reliability, and validity of the indicators were previously evaluated in a joint project at an academic medical center and a VA medical center. We determined interrater reliability using a 4% sample of charts, and excluded indicators when the κ statistic was <0.8 for eligibility (denominator) or <0.6 for care specified by the QI (numerator), or when there was insufficient face validity in the abstraction process. Results were similar in the two centers for all indicators included in this project: the overall κ score was 0.87 for eligibility and 0.86 for specified care.⁹

We report here the results of a pilot quality assessment effort at the academic medical center and comprehensive cancer center using the ASSIST indicators. The objective of this study was to evaluate key aspects of the quality of end-of-life care, focusing on the symptoms of cancer and information and care planning, prior to a planned expansion of the well-established cancer center pain program into a broader palliative care initiative.

Methods

This article reports on the quality of end-of-life care measured at a large academic medical center (AMC), The Johns Hopkins Hospital (JHH) and The Johns Hopkins Sidney Kimmel Comprehensive Cancer Center (SKCCC). At the time of the study, the Department of Medicine had a palliative care program, while the SKCCC had a long-established cancer pain program and was in the process of developing a larger palliative care initiative. The study protocol was approved by the Johns Hopkins Institutional Review Board.

ASSIST quality indicators

Details of the quality indicators⁷ and the reliability, validity, and feasibility evaluation⁸ are described elsewhere. Of the total of 41 indicators that passed criteria for reliability, validity, and feasibility,⁸ we targeted 21 that best addressed the goals of palliative care in this analysis, by including only indicators related to the cancer itself, not those related to cancer treatment (e.g., we did not include indicators for nausea related to chemotherapy). Indicators addressed both inpatient and outpatient care (including longitudinal indicators that crossed both settings), and included 4 for pain, 2 for depression, 4 for dyspnea, 3 for nausea and vomiting, 1 for fatigue, 2 for anorexia, and 5 for information and care planning.

Study sample

We developed the goals, target population, and data sources for the study based on the needs of practical quality assessment for the palliative care initiative and on local data sources and availability, with consultation with a variety of oncology and palliative care providers and those involved in quality assurance. Since our initiative would focus initially on patients with life-limiting cancers and those who would receive chemotherapy through the cancer center, we used our hospital cancer registry to consecutively sample patients with the most common cancers (lung, pancreatic, breast, and colorectal) diagnosed with stage IV (and stage IIIB for lung cancer) disease in 2003–2005 who met the inclusion criteria. To maximize the efficiency of identifying patients who would have been eligible for the end-of-life indicators, we included only patients who died between 2 and 15 months after diagnosis. To ensure that patients had sufficient visits to abstract the data needed for the indicators, we included patients with at least three outpatient visits at Johns Hopkins or at least two outpatient visits and a cancer-related hospitalization between 3–30 days in length after the diagnosis (only 20 of the included patients did not have an eligible hospitalization). Our goal sample size was approximately 240 patients, to give sufficient sample size to assess most of the indicators of interest.

Data sources

We obtained data from the following sources to construct the ASSIST quality indicators: (1) electronic outpatient pain vital sign scores available through the Cancer Center information system, (2) electronic medical records, and (3) paper medical records. Nursing notes were generally captured in a structured electronic inpatient nursing documentation system and physician and social worker documentation were generally relatively unstructured documentation in paper or electronic format (formats varied throughout the hospital and between the inpatient and outpatient setting). The electronic pain scores were used for one indicator, for pain screening. We investigated the possibility of identifying some patients or conditions or evaluating some indicators through electronic abstraction, but since the electronic medical records were not uniformly or easily searchable, we manually abstracted all data from both paper and electronic formats. To collect the needed data elements for scoring indicators from the medical records, we developed a Microsoft Access-based abstraction tool with extensive supporting abstraction guidelines (details available upon request). After undergoing training on the guidelines and test medical records, nurses experienced in medical record abstraction used the tool.

Our data abstraction strategy was guided by the need to be efficient in our data collection strategy, to target our data collection to best capture sufficient data for the relevant indicators while providing a valid characterization of patients' end-of-life care. Since the greatest opportunities to address end-of-life issues may be at the beginning and end of care, we selected the first two and the last outpatient visits, as well as the patients' last hospitalization. We abstracted the first 3 days of the hospitalization for symptom management, and the entire hospitalization for information and care planning. We calculated the mean percent adherence for each QI as well as an overall summary measure of quality of care.

Results

Sample characteristics

The characteristics of the 238 study participants are shown in Table 1. The mean age was 60.2 years (SD 12.2 years) and just over half of the sample was female (53%). The majority of patients were Caucasian (63%) with African Americans constituting another 32% of the sample. Over half of the patients had lung cancer and another third had pancreatic cancer. There were relatively few patients with colorectal cancer (n = 24) or breast cancer (n = 7). The mean survival time from the initial visit for advanced cancer was 7.5 months. All but 20 of the included patients also had an eligible hospitalization.

Table 1.

Characteristics of the Study Sample

Patient characteristics	No. of patients n = 238	%
Age, mean (SD)	65.9 (9.9)
Female	127	53
Race/Ethnicity
Caucasian	151	63
African American	76	32
Other or unknown	11	5
Cancer Type
Breast	7	3
Colorectal	24	10
Lung	131	55
Pancreas	76	32
Survival from initial visit with advanced disease, months (SD)	7.5 (3.6)

SD, standard deviation.

Quality indicators

The results for the quality indicators (QIs) are shown in Table 2. Overall adherence to recommended care was 53% (95% confidence interval [CI], 50%–56%). There was considerable variability in adherence across indicators and domains of care. Adherence was highest in the pain domain, with 79% of 467 eligible outpatient visits and 97% of 148 eligible hospitalizations having documentation on screening for the absence or presence of pain. In addition, after an outpatient pain medication was changed, 97% of the 33 eligible events had a pain assessment at the subsequent visit. However, only half of patients who were initiated on opiates had chart documentation that they were either offered a prescription or nonprescription bowel regimen or had contraindications to a bowel regimen.

Table 2.

Adherence to ASSIST Quality Indicators for End-of-Life Care in the Domain of Pain (n = 238)

Quality Indicator	Number of eligible events ^a	Adherence (%)	95% confidence interval	Notes
Pain
IF a patient with cancer has a cancer-related outpatient visit THEN there should be screening for the presence or absence and intensity of pain using a numeric pain score	467	79%	75%, 83%	Calculated from electronic pain vital sign scores (only cancer center visits where pain scores electronically collected were included)
IF a patient with cancer is admitted to a hospital THEN there should be screening for the presence or absence of pain	148	97%	93%, 99%	Abstracted within 48 hours, from nursing notes
IF a patient with cancer pain is started on long-term opioid treatment THEN he/she should be offered either a prescription or nonprescription bowel regimen within 24 hours or there should be documented contraindication to a bowel regimen	39	51%	35%, 68%	Mention of a bowel regimen or any appropriate medications in the physician note or medication list met criteria for this indicator
IF a patient's outpatient cancer pain regimen is changed, THEN there should be an assessment of the effectiveness of treatment at or before the next outpatient visit with that provider or at another cancer-related outpatient visit	33	97%	84%, 100%	Defined as a pain assessment on the next visit; first visit only included

Overall n = 238. n for indicators varies—for some patients, they were eligible for the indicator multiple times, such as at each of the 2–3 outpatient visits specified—these are the cases where n > 238. For inpatient symptoms, only 148 hospitalizations were abstracted because all hospitalizations were eligible for these indicators. Other indicators would only apply in selected circumstances when the denominator criteria were met on one of the abstracted visits (e.g., for outpatient visits when the patient was started on chronic opioids, a visit would have been eligible only if chronic opioids were started on one of the 2–3 visits abstracted for each patient).

One indicator was not reported because of issues with how it was collected in the abstraction form and electronic tool: IF a patient with cancer pain is started on a long-acting opioid formulation, THEN a short-acting opioid formulation for breakthrough pain should also be provided.

ASSIST, Assessing Symptoms Side Effects and Indicators of Supportive Treatment.

The results from indicators on depression, dyspnea, and other symptoms are in Table 3. Compared to other domains, there were relatively fewer events which were eligible for assessment in the two domains of depression and dyspnea (n ranged from 6 to 51). Adherence rates for the individual indicators in these two domains showed considerable variability, ranging from 14% to 86%. There were more eligible events in the domain of other symptoms (n ranged from 10 to 396), and adherence rates were also variable, ranging from 0% to 87%. At one end of the spectrum, no patients who received enteral or parenteral nutrition (n = 10) had documentation that their life expectancy was at least a month. At the other end, 87% of the 67 patients admitted to the hospital with advanced cancer affecting the gastrointestinal tract or abdomen had documented assessment of nausea or vomiting within 24 hours. Assessment of these symptoms in the outpatient setting, however, occurred less frequently (45% of 396 eligible visits). Moreover, fatigue was only assessed in 25% of initial outpatient visits.

Table 3.

Adherence to ASSIST Quality Indicators for End-of-Life Care in the Domains of Depression, Dyspnea, and Other Symptoms

Quality indicator	Number of eligible events ^a	Adherence (%)	95% confidence interval	Notes
Depression
IF depression is diagnosed in a patient with cancer, THEN a treatment plan for depression should be documented	13	69%	39%, 91%
IF a patient with cancer is treated for depression, THEN response to therapy should be documented within 6 weeks	6	17%	0%, 64%
Dyspnea
IF a patient with cancer reports new or worsening dyspnea, THEN there should be documentation of cause or of investigation of at least one of the following: hypoxia, anemia, bronchospasm or chronic obstructive pulmonary disease, pleural effusion, tumor obstruction of bronchi or the trachea, pneumonia, or pulmonary embolism	46	80%	66%, 91%	Defined as physician documentation of cause of dyspnea on admission for the hospitalization
IF an outpatient with primary lung cancer or advanced cancer reports new or worsening dyspnea, THEN they should be offered symptomatic management or treatment directed at an underlying cause within one month	7	57%	18%, 90%	Abstracted as new dyspnea
IF an inpatient with primary lung cancer or advanced cancer has dyspnea on admission, THEN they should be offered symptomatic management or treatment directed at an underlying cause within 24 hours	51	14%	6%, 26%
IF a patient with cancer in the hospital is treated for dyspnea, THEN there should be an assessment within 24 hours that the treatment was effective in relieving dyspnea OR that a change in treatment for dyspnea was made	7	86%	42%, 100%
Other Symptoms
IF a patient (with cancer undergoing moderately or highly emetic chemotherapy) or with advanced cancer affecting the gastrointestinal tract or abdomen is seen for a visit in a cancer-related outpatient setting, THEN the presence or absence of nausea or vomiting should be assessed at every visit	396^a	45%	40%, 50%	Defined as patients with gastrointestinal cancer or cancer metastatic to liver or peritoneum; chemotherapy issues not addressed
IF a patient with advanced cancer affecting the gastrointestinal tract or abdomen is admitted to a hospital, THEN the presence or absence of nausea or vomiting should be assessed within 24 hours	68	87%	76%, 94%	Calculated from nursing notes
IF an outpatient with cancer not receiving chemotherapy or radiation is treated for nausea or vomiting with an antiemetic medication, THEN the effectiveness of treatment should be evaluated before or on the next visit to the same outpatient site	14	57%	29%, 82%
IF a cancer patient is seen for an initial visit (or any visit while undergoing chemotherapy) at a cancer-related outpatient site, THEN there should be an assessment of the presence or absence of fatigue	238	25%	20%, 31%
IF a patient presents for an initial visit for cancer affecting the oropharynx or gastrointestinal tract or advanced cancer at a cancer-related outpatient site, THEN there should be an assessment for the presence or absence of anorexia or dysphagia	238	45%	39%, 52%
IF a cancer patient is treated with enteral or parenteral nutrition, THEN there should be an assessment prior to starting nutrition that there was difficulty maintaining nutrition due to significant gastrointestinal issues and that expected life expectancy was at least one month	10	0%	0%, 31%

One indicator was not reported because of issues with how it was collected in the abstraction form and electronic tool: IF an outpatient with cancer not receiving chemotherapy or radiation has an initial episode of nausea or vomiting, THEN the clinician should assess for underlying causes.

ASSIST, Assessing Symptoms Side Effects and Indicators of Supportive Treatment.

Finally, in the domain of information and care planning, adherence rates to the individual quality indicators were generally relatively low (Table 4). Only 19% of patients had documentation of an advance directive or a surrogate decision maker in the medical record. Furthermore, only 1 of the 17 patients who received mechanical ventilation had documentation of preferences regarding mechanical ventilation or documentation of why this information was unavailable. However, discussion of the risks and benefits or the intent of treatment for chemotherapy was documented at a much higher frequency (68% of 178 eligible visits). Lastly, referral to hospice was documented in the medical record for 45% of the 238 patients in the study.

Table 4.

Adherence to ASSIST Quality Indicators for End-of-Life Care in the Domain of Information and Care Planning

Quality indicator	Number of eligible events	Adherence (%)	95% confidence interval	Notes
Information and care planning
IF a patient with advanced cancer dies an expected death, THEN there should be documentation of an advance directive or a surrogate decision maker in the medical record	235	19%	14%, 25%	Expected death defined as all patients in this sample; 12% had advance directive and 12% surrogate
IF a patient with advanced cancer dies an expected death, THEN s/he should have been referred for palliative care prior to death (hospital-based or community hospice) or there should be documentation why there was no referral	238	45%	39%, 52%	Documentation of why there was no referral difficult to operationalize; 8% referred to palliative care and 45% to hospice Patient did not have to accept hospice or palliative care to meet this indicator
IF a patient with advanced cancer is admitted to the ICU and survives 48 hours, THEN within 48 hours of ICU admission, the medical record should document the patient's preferences for care or attempt to identify them	22	9%	1%, 29%	On oncology wards, patients receive critical care in place rather than ICU transfer; abstracted as preferences for ICU care
IF a patient with advanced cancer is mechanically ventilated in the ICU, THEN within 48 hours of admission to the ICU, the medical record should document the patient's preference for mechanical ventilation or why this information is unavailable	17	6%	0%, 29%	Abstracted as mechanical ventilation
IF a patient with cancer undergoes chemotherapy, THEN prior to chemotherapy, s/he should be informed about the risks and benefits of treatment, including likely symptoms and side effects, and whether the treatment intent is curative or palliative	174	68%	60%, 75%	Abstracted as “risks and benefits, or whether the treatment intent.…”

ICU, intensive care unit.

Discussion

In conclusion, in this focused application of 21 end-of-life oncology indicators in the domains of pain, other symptoms, and information and care planning, the care specified by the indicators was documented in only half of eligible encounters. Documentation of care ranged widely, from 0% for documenting prognosis before starting enteral or parenteral nutrition, to nearly universal adherence for inpatient pain screening and for documentation of a pain score on the second visit if the pain regimen was changed at the initial cancer-related outpatient visit. However, most indicators were within a range where at least some patients had documentation that they received recommended care, but there was significant room for improvement. Performance on some of these indicators may reflect specific aspects of the setting—for example, the SKCCC had a long-established pain program with electronic collection of outpatient pain scores, as well as structured electronic medical record nursing documentation for pain and nausea/vomiting, which may have contributed to high rates of screening. Lack of focused efforts on specific symptoms (e.g., fatigue) or lack of guidelines and structured documentation (e.g., communication about chemotherapy and advance care planning) may have contributed to lower scores in these areas. In comparison with evaluation of these indicators in a VA medical center with a robust palliative care program,¹⁰ scores in the academic center were substantially higher in the domain of pain and substantially lower for information and care planning.

Other studies including similar indicators in larger cancer quality initiatives have found some similar results. Implementation of the American Society for Clinical Oncology's Quality Oncology Practice Initiative (QOPI), a comprehensive oncology indicator set that includes some measures relevant to end-of-life care, found similar results in areas such as documenting the effectiveness of opioids. The study also found that measure performance was much higher when documentation was structured into the electronic medical record, and that presentation of performance data to physicians was helpful in changing behavior.¹¹ An evaluation of pain documentation in patients with bone metastases in comprehensive cancer centers also found that nurses more frequently recorded pain scores than physicians.¹² Another study found that performance on an indicator in the psychosocial domain, the percentage of patients with a documented problem in emotional well-being who had evidence that action was taken to address it, was 50% or less in 5 of 7 outpatient oncology sites.¹³

This pilot study focusing on practical, efficient quality assessment has a number of limitations. First, the cancer registry was able to identify only a few patients with advanced breast cancer. This is likely because most of these patients are diagnosed at early stages and are not identified in the registry when they develop advanced disease. Although we evaluated multiple potential methods for identifying these patients, including electronic searches of the cancer center information system, none would have been effective in our setting and data systems. The lack of searchable medical records at our facility also made it infeasible for us to identify sufficient patients with relatively uncommon conditions, such as pleural effusions, for quality assessment. Although the precision of the overall assessment of quality was high, selected indicators had low denominators and wide confidence intervals and results should be interpreted with caution. There are limitations of measuring quality for conditions where the denominator is incompletely documented (such as dyspnea) or conditions that are relatively rare. Finally, as in most quality measurement, these indicators are limited by what is documented and easily interpretable in the medical record; for example, whether a nurse screened a patient for pain is far easier to discern reliably than whether a physician made a change in a patient's care in response to that pain score.

In conclusion, the variation in performance among the Cancer Quality-ASSIST quality indicators targeting end-of-life care supports the usefulness of these indicators and the need for quality measurement and improvement efforts. In our setting, these results are useful as pilot assessment data for further developing quality improvement programs in pain management in areas beyond screening and in communication about chemotherapy and care planning, as well as for development and evaluation of palliative care initiatives in other areas of the hospital, including documentation templates to better capture the necessary data for relevant patients. These medical record-based indicators for processes of care are being used in conjunction with other quality assessment efforts, including clinician surveys, patient and family feedback, and outcome measures such as hospice utilization. Further research is ongoing on the generalizability of these indicators to other cancers, less advanced disease, and other settings, as well as how these indicators can best be applied to guide performance improvement efforts and associations between performance on these indicators and patient outcomes.

Footnotes

Acknowledgments

We acknowledge the contributions of Christine Weston, Ph.D., Robert Herbert, M.S., Joyce Kane, R.N., Carol Roth, R.N., M.P.H., Peg Barnaba, R.N., John Fetting, M.D., Michael Carducci, M.D., Paul Han, M.D., M.A., M.P.H., William Lawrence, M.D., M.S., Eric Bass, M.D., M.P.H., and Albert Wu, M.D., M.P.H.

The research reported here was supported under Contract No. 290-2005-0034I from the Agency for Healthcare Research and Quality, US Department of Health and Human Services as part of the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) program. The authors of this paper are responsible for its content. Statements in this paper should not be construed as endorsement by the Agency for Healthcare Research and Quality.

Author Disclosure Statement

No competing financial interests exist.

References

Lorenz

, Lynn

, Dy

, Shugarman

, Wilkinson

, Mularski

, Morton

, Hughes

, Hilton

, Maglione

, Rhodes

, Rolon

, Sun

, Shekelle

. Evidence for improving palliative care at the end of life: A systematic review. Ann Intern Med, 2008; 148:147–159.

Wright

, Zhang

, Ray

, Mack

, Trice

, Balboni

, Mitchell

, Jackson

, Block

, Maciejewski

, Prigerson

. Associations between end-of-life discussions, patient mental health, medical care near death, and caregiver bereavement adjustment. JAMA, 2008; 300:1665–1673.

Snyder

, Dy

, Carducci

, Brahmer

, Carducci

, Wolff

, Wu

. Asking the right questions: Investigating patient-reported outcomes questionnaires for use in oncology clinical practice. J Support Care Cancer, 2007; 15:1075–1085.

Gattellari

, Voigt

, Butow

, Tattersall

. When the treatment goal is not cure: are cancer patients equipped to make informed decisions? J Clin Oncol, 2002; 20:503–513.

Seow

, Snyder

, Mularski

, Shugarman

, Kutner

, Lorenz

, Wu

, Dy

. A framework for assessing quality indicators for cancer care at the end of life. J Pain Symptom Manage, 2009; 38:903–912.

Lorenz

, Lynn

, Dy

, Wilkinson

, Mularski

, Shugarman

, Hughes

, Asch

, Rolon

, Rastegar

, Shekelle

. Quality measures for symptoms and advance care planning in cancer: A systematic review. J Clin Oncol, 2006; 24:4933–493.

National Consensus Project for Quality Palliative Care. Clinical Practice Guidelines for Quality Palliative Care, Second 2009. www.nationalconsensusproject.org. 2011 February 19.

Lorenz

, Dy

, Naiem A Walling

, Sanati

, Smith

, Shanman

, Roth

, Asch

. Quality Measures for Supportive Cancer Care: The Cancer Quality-ASSIST (Assessing Symptoms, Side Effects, and Indicators of Supportive Treatment) Project. J Pain Symptom Manage, 2009; 37:943–964.

, Lorenz

, ONeill

, Asch

, Walling

, Tisnado

, Antonio

, Malin

. Cancer Quality-ASSIST Supportive Oncology Quality Indicator Set: Feasibility, Reliability, and Validity Testing. Cancer, 2010; 116:3267–3275.

10.

Malin

, ONeill

, Asch

et al. Quality of supportive care for patients with advanced cancer in a VA medical center(in press).

11.

Blayney

, McNiff

, Hanauer

, Miela

, Markstrom

, Neuss

. Implementation of the Quality Oncology Practice Initiative at a university center. J Clin Oncol, 2009; 27:3802–3807.

12.

Weinstein

, Romanus

, Lepisto

, Reyes-Gibby

, Cleeland

, Greene

, Muir

, Niland

. Documentation of pain in comprehensive cancer centers in the United States: A preliminary analysis. J Natl Compr Canc Netw, 2004; 2:173–180.

13.

Jacobsen

, Shibata

, Siegel

, Lee

, Alemany

, Brown

, Cartwright

, Levine

, Smith

, Abesada-Terk

Jr , Malafa

. Initial evaluation of quality indicators for psychosocial care of adults with cancer. Cancer Control, 2009; 16:328–334.