Association between Patient-Reported Symptoms and Nurses' Clinical Impressions in Cancer Patients Admitted to an Acute Palliative Care Unit

Abstract

Background:

Clinicians typically rely on their own or the nurses' clinical impression (NI) of symptoms rather than patient self-reports. It is unclear whether these means of assessment yield similar results.

Objective:

To prospectively compare patient-reported symptoms on a modified Edmonton Symptom Assessment System (ESAS) with NI scores.

Methods:

Consecutive patients with advanced cancer admitted to our acute palliative care unit between April and July 2010 were studied. We collected the results of the ESAS on the day of admission (D1) to the unit and 2 days later (D3). We also collected the NI of each patient's physical and psychological distress on D1 and D3.

Results:

One hundred eighteen patients completed the ESAS on D1 and 116 on D3. On D1 there was no significant association between NI score and ESAS assessment except for dyspnea, which was weakly correlated with NI score for physical distress (r=0.22, p=0.02). The median ESAS physical and psychosocial scores were 31 and 12 in patients with NI of low or no physical distress, versus 34 (p=0.07) and 15 (p=0.18) in patients with NI of moderate or severe distress, respectively. On D3, we found a significant association between ESAS and NI only for pain (r=0.32, p<0.001) and anxiety (r=0.30, p=0.001). Sensitivity and specificity of the NIs for ESAS scores were low on both days.

Conclusion:

The clinical impression of highly trained palliative care nurses showed poor association with patient-reported symptom intensity. Validated symptom assessment tools are needed for bedside clinical assessment.

Introduction

Inpatients with advanced cancer develop multiple severe symptoms.^1–5 Effective treatment is only possible if appropriate symptom assessment take place.^6,7 Physical symptoms such as pain, fatigue, dyspnea, anorexia, nausea, and sleep disorders are common, particularly at advanced stages of the illness.⁸ These symptoms are often associated with psychological distress, including depression and anxiety.^9–11

Use of the Edmonton Symptom Assessment Scale (ESAS)¹² in daily clinical practice has been reported to result in physicians' identifying symptoms earlier and more efficiently.¹³ However, most clinicians do not routinely use such symptom assessment tools in everyday practice and treat according to their or the nurse's clinical impression (NI) only.^14–17 One of the reasons might be that some clinicians believe that it burdens patients to ask them to complete questionnaires and similar reporting instruments,^18,19 and that the patient can make errors when responding as a result of misinterpreting the question.²⁰

Therefore, in most clinical settings, decisions regarding the administration of rescue medication for pain, nausea, dyspnea, or the request for psychosocial support services are based on the clinical impression of the bedside nurse.

Patient-reported outcomes are self-reports by the patient (i.e., data are collected directly from the patient). The prevalence and severity of symptoms according to patient-reported outcomes compared with symptoms assessed by a third party (e.g., health care providers) tend to vary significantly,^21–27 which can make management decisions more complicated.

Our group had previously measured the association among ESAS done by the physicians, the patient, and the nurses, and we found that there were significant differences among all three, with consistent under-finding by the physician.⁹

In clinical practice, most physicians and nurses do not ask all ESAS questions to patients before making changes in pharmacological and/or nonpharmacological symptom management. Therefore in this prospective study, we decided to compare patient-reported symptoms on the ESAS with the overall NI of physical and emotional distress because it better reflects what is regularly done in bedside care.

Patients and Methods

The study was conducted at The University of Texas M. D. Anderson Cancer Center, a comprehensive cancer center in Houston, Texas. The Institutional Review Board at The University of Texas M. D. Anderson Cancer Center approved this study. All nurses who participated in this study signed the informed consent prior to enrollment. The Institutional Review Board provided waiver of consent for patient participation as we were using only the usual assessment used in daily practice in our unit.

The ESAS is used daily to assess all patients admitted to the M. D. Anderson Acute Palliative Care Unit (APCU), a 12-bed unit for patients experiencing acute symptoms related to their cancer or cancer treatment. Patients admitted in the unit have advanced cancer (i.e., metastatic solid tumor, locally/regionally advanced, or hematologic with refractory or relapse).

All patients admitted to the APCU are placed under the primary care of a team of board-certified palliative medicine specialists, specially trained bedside nurses, and an interdisciplinary team consisting of counselors, social workers, a chaplain, occupational therapists, physiotherapists, and a pharmacist. All patients are assessed by all members of the team during their inpatient hospital stay.

All adults patients admitted to the APCU were potentially eligible for the study. The only exclusion criteria were delirium and unresponsiveness that precluded completion of ESAS.

Data collection

To standardize documentation of clinical data, we provided training sessions for all APCU clinical staff nurses who agreed to participate in this study. All the nurses who participated in this study had specialist orientation to palliative care nursing and were full-time palliative nursing staff. To minimize missing data, our study coordinators brought the APCU nurses study forms daily to ensure that they were completed in a timely fashion. We collected various nurses' characteristics including age, sex, years of clinical experience, and years of palliative care experience.

We only assessed the nurses' impression about the physical and emotional distress because the nurses at the bedside make many decisions regarding contacting the physicians, administering rescue medications for different symptoms, calling others disciplines for counseling, and so forth. NI was not collected using a specific tool but by scoring distress as follows: no distress=0, low distress=1, moderate distress=2, and severe distress=3.

We collected the NI of each patient's physical and psychological distress on day of admission (D1) and 2 days later (D3). The nurses gave two scores, one for physical distress and the other for psychological distress.

We decide to collect NI even though it's not a validated tool because it is what guides the care, including the use of as-needed medications at the bedside. These NI scores were collected at the end of the nurses' shifts at 4–6 a.m. or 4–6 p.m.

For each patient, we documented demographic characteristics, including age, sex, ethnicity, religion, marital status, education, and cancer diagnosis. Patients completed ESAS forms on D1 and on the third day of hospitalization in the APCU (D3), if possible.

The ESAS is a self-report tool that allows patients with advanced cancer to document the intensity of nine common¹² symptoms (pain, nausea, tiredness or fatigue, drowsiness, loss of appetite, shortness of breath or dyspnea, depression, anxiety, and diminished feeling of well-being). For this study, we added a 10th symptom, sleep disturbance. The modified ESAS, like the original, is designed to document changes in symptom profiles over time by repeated quantitative measurements of symptom intensity. It has been shown to have good test–retest reliability.^28–30 The ESAS asks patients to rate the intensity of these symptoms over the past 24 hours using an 11-point numeric rating scale, which runs from 0 (no symptom) to 10 (worst possible symptom).

The ESAS yields a total score and two subscale scores. These scores have been validated in advanced cancer patients with good reliability when compared with others tools.^28–32 The total symptom distress score (TSDS) is the sum of the scores for the 10 symptoms for a total score of 0 to 100.¹² The physical distress subscore (PHS) was the sum of scores for seven symptoms (pain, nausea, tiredness or fatigue, drowsiness, appetite, shortness of breath or dyspnea, and sleep), and the psychological distress subscore (PSS) was the sum of scores for three symptoms (depression, anxiety, and feeling of well-being).³³ The results of the ESAS were recorded separately by our research coordinator. The assessment was completed either by the patients unassisted or by the patients assisted by the research coordinator. The nurses asked for an NI rating were blinded to the patients' responses.

D1 assessment was done independently, without the nurses' having knowledge of the results of the ESAS. Similarly, D3 assessment was done independently by the nurse who was in charge of the patient, who did not have knowledge of the results of the D1 assessment or of the day's score on the ESAS.

The assessment was completed by the nurse in charge of the patient that day. The nurse was not always the same person because the study was designed to reflect standard daily practice.

Statistical analysis

Standard descriptive statistics, including mean, median, standard deviation, range, proportion, and frequency, together with 95% confidence intervals, were calculated for patient and nurse characteristics and survey outcomes. A nonparametric Mann-Whitney test (two-sided) and Spearman's rank correlation analysis (two-sided) were used to assess the association between ESAS score and NI.

In our analysis of the sensitivity and specificity of the NI to ESAS items scores, we used optimal ESAS item cut scores reported in the literature, ranging from 2 to 5, to define classes. For most of the symptoms, we used a cut score of 3; however, for depression and anxiety the cut score was 2, and for fatigue, the cut score was 5.^34–36

We used the SPSS statistical package version 17.0.2 for Windows (SPSS, Inc., Chicago, IL) for all statistical analyses. For all our statistical analyses, a p value of <0.05 was considered to be statistically significant.

Results

Nurses characteristics

Twenty nurses participated to this study with a median age of 44 years. Seventeen (85%) of them were women. The median years of clinical practice was 9 (interquartile range 3 to 18 years) and the median years of palliative care experience was 2 (interquartile range 1 to 7 years).

Patient characteristics

One hundred fifty-one patients were admitted to the APCU between April and July 2010, and a total of 118 patients were enrolled in this study. The other 33 patients were not able to complete the ESAS on D1 because they were unresponsive and so were excluded. Patient demographics are reported in Table 1. The patients' median age was 58 years (range, 18 to 85 years), and most were non-Hispanic white. Gastrointestinal, respiratory, breast, and gynecologic malignancies were the most common diagnoses.

Table 1.

Patient Characteristics on APCU Admission (n=118)

Patient characteristics	N (%)
Female sex	78 (66)
Ethnicity
Caucasian	77 (65)
African American	13 (11)
Hispanic	23 (20)
Other	5 (4)
Christian religion	102 (86)
Married	69 (59)
Highest education level
High school or below	51 (43)
Any College undergraduate education	33 (28)
Any advanced postgraduate education	14 (12)
Age (in years), median (range)	58 (18–85)
Cancer diagnosis
Breast	18 (15)
Dermatologic	7 (6)
Gastrointestinal	26 (22)
Genitourinary	7 (6)
Gynecologic	18 (15)
Head and neck	5 (4)
Hematologic	13 (11)
Respiratory	19 (16)
Other	5 (4)

APCU, acute palliative care unit.

ESAS score and NI

One hundred eighteen patients were able to complete the ESAS on D1, and 116 completed it on D3 (two patients developed cognitive impairment that precluded completion of the second assessment).

On D1 there was no significant association between NI scores of physical and psychological symptom distress and any ESAS item score, with the exception of dyspnea score. This score was weakly correlated with NI score for physical distress (r=0.22, p=0.02); the group with an NI physical distress score of 0 or 1 had a median ESAS dyspnea score of 3, and the group with an NI physical distress score of 2 or 3 had a median ESAS dyspnea score of 5 (Table 2).

Table 2.

ESAS Score Categorized According to NI of Physical Distress and Correlation between NI and ESAS on D1

	NI of physical distress
ESAS item/subscale/total score	None – mild distress (0–1) N=50 median ESAS score	Moderate – severe distress (2–3) N=68 median ESAS score	P value^*	Correlation between NI and ESAS ^a
Fatigue	6	7	0.07	r=0.14; p=0.14
Drowsiness	4	4	0.14	r=0.39; p=0.68
Appetite	5	5	0.80	r=0.15; p=0.10
Nausea	2	1	0.67	r=−0.74; p=0.43
Pain	4.5	5	0.41	r=0.66; p=0.49
Dyspnea	3	5	0.02	r=0.22; p=0.18
Sleep	5	5	0.40	r=0.09; p=0.34
PHS	31	34	0.07	r=0.18; p=0.07
TSDS	44	50	0.10	r=0.15; p=0.10

ESAS, Edmonton Symptom Assessment System; NI, nurses' clinical impressions; PHS, physical distress subscore; TSDS, total symptom distress score.

Mann-Whitney U test.

Pearson's correlation.

The median PHS and PSS were 31 and 12, respectively, in patients with NI scores of 0 or 1 for physical and psychological distress and were 34 (p=0.07) and 15, respectively, in patients with NI scores of 2 or 3 for physical and psychological distress. These data and the results of correlation analysis are shown in Tables 2 and 3.

Table 3.

ESAS Score Categorized According to NI of Psychological Distress and between NI and ESAS on D1

	NI of psychological distress
ESAS item/subscale/total score	None – mild distress N=52 median ESAS score	Moderate – severe distress N=66 median ESAS score	P value^*	Correlation between NI and ESAS ^a
Depression	4	5	0.34	r=0.09; p=0.34
Anxiety	5	5	0.29	r=0.10; p=0.29
Feeling of well-being	5	6	0.20	r=0.12; p=0.20
PSS	12	15	0.18	r=0.13; p=0.19
TSDS	45	48	0.22	r=0.11; p=0.22

ESAS, Edmonton Symptom Assessment System; NI, nurses' clinical impressions; PSS, psychological distress subscore; TSDS, total symptom distress score.

Mann-Whitney U test.

Pearson's correlation.

There was no association between NI score of physical and psychological distress and TSDS (p=0.1), with median TSDS score of 44 and 45 in patients with NI scores of 0 or 1 for physical and psychological distress and of 50 and 48 in patients with NI scores of 2 or 3 for physical and psychological distress, respectively.

On D3 there were significant associations between pain and NI scores for physical distress (r=0.32, p<0.001; Table 4), anxiety and NI scores for psychological distress (r=0.30, p=0.001, Table 5), anxiety and NI scores for physical distress (r=0.32, p=0.001), TSDS and NI score for physical distress (r=0.27, p=0.003) (Table 4), and PHS and NI score for physical distress (r=0.22, p=0.02) (Table 4).

Table 4.

ESAS Score Categorized According to NI of Physical Distress and Correlation between NI and ESAS on D3

	NI of physical distress
ESAS item/subscale/total score	None – mild distress (0–1) N=46 median ESAS score	Moderate – severe distress (2–3) N=70 median ESAS score	P value^*	Correlation between NI and ESAS ^a
Fatigue	5	6	0.07	r=0.14; p=0.13
Drowsiness	4	5	0.27	r=0.09; p=0.30
Appetite	5	5	0.14	r=0.17; p=0.07
Nausea	1	1	0.30	r=−0.09; p=0.34
Pain	2	4	0.001	r=0.32; p<0.001
Dyspnea	3	3.5	0.85	r=0.02; p=0.85
Sleep	4	5	0.24	r=0.14; p=0.14
PHS	27	30	0.02	r=0.22; p=0.02
TSDS	39	45	0.003	r=0.27; p=0.003

ESAS, Edmonton Symptom Assessment System; NI, nurses' clinical impressions; PHS, physical distress subscore; TSDS, total symptom distress score.

Mann-Whitney U test.

Pearson's correlation.

Table 5.

ESAS Score Categorized According to NI of Psychological Distress and Correlation between NI and ESAS on D3

	NI of psychological distress
ESAS item/subscale/total score	None – mild distress (0–1) N=47 median ESAS score	Moderate – severe distress (2–3) N=69 median ESAS score	P value^*	Correlation between NI and ESAS ^a
Depression	3	4	0.37	r=0.06; p=0.53
Anxiety	6	7	0.001	r=0.30; p=0.001
Feeling of well-being	5	5	0.18	r=−0.08; p=0.40
PSS	14	16	0.20	r=0.13; p=0.18
TSDS	42	42	0.85	r=0.02; p=0.85

ESAS, Edmonton Symptom Assessment System; NI, nurses' clinical impressions; PSS, psychological distress subscore; TSDS, total symptom distress score.

Mann-Whitney U test.

Pearson's correlation.

NI score sensitivity and specificity to ESAS item scores

On D1, the NI score had low sensitivity to ESAS item score, ranging from 0.57 to 0.65, and low specificity, ranging from 0.42 to 0.57. The results were similar on D3.

Discussion

In this study, we compared bedside NIs of advanced cancer patients' symptom distress with the patients' own reports (as captured in ESAS scores) for multiple symptoms. We observed no significant association between NI and ESAS for the vast majority of symptoms on the day of admission and only slightly better performance/correlation on the patient's third day in the unit, probably related to a larger amount of clinical information collected by the team at D3 (Tables 2 –5.) In addition, there was no significant difference between the ESAS scores of patients considered to be in no or low physical and psychological distress by nurses and patients considered to be in moderate to severe distress at the time of admission (Tables 1 –3). These data suggest poor association between nurses' impressions of patient symptoms and the patients' own experiences of those symptoms. This finding is particularly striking because the participating nurses had been specially trained in palliative care; in other settings, nurses' impressions of patient symptoms might be expected to be even less accurate.

The implications of these findings reflect an important unmet (and heretofore largely unrecognized) need in palliative care—the possibility that suboptimal care is being provided because of erroneous assumptions about the accuracy of third parties' impressions of cancer patients' distress, both physical and psychological. NIs have a strong impact on patients' receipt of pharmacological interventions, such as extra doses of opioids, antiemetics, or other agents for the management of symptoms such as pain, dyspnea, and nausea; receipt of nonpharmacological interventions such as counseling; and access to other members of the interdisciplinary team such as counselors, chaplains, and rehabilitation specialists.

Nurses' evaluations also have great influence on physicians' use of pharmacological and nonpharmacological interventions. Our findings suggest that decisions based on clinical impression alone have the potential to be biased and less effective than they could be. Thus, overall impression has very low accuracy to detect patients' distress, and we should probably ask questions regarding specific symptoms. Our findings suggest that from the perspective of patient care, regular symptom assessment using formal validated tools is very important for determining the levels of physical and psychosocial distress in patients admitted to an APCU. In most inpatient settings in the United States there is no regular patient-reported symptom assessment. Instead, treatment decisions are made based on nurses' and physicians' clinical impressions. Our group and others^{9,15,21,22,24,25} previously found that there is limited association between symptoms ratings by patients and by health professionals, but in those previous studies, the sample size was smaller and the patient group was mixed (age, type of cancer, cancer extension). Another difference with previous studies is that they have used the same tools for the symptoms ratings by patients and by health professionals. This is not what is done in clinical practice, and this can also explain some of the discrepancies founnd between our results and the previous study we have conducted on this topic.⁹ In our previous study, nurses were able to rate the ESAS closer to the patients' rating than did the physician, but still there were differences when compared with patients' rating. In this study, we chose the global impression rather than the ESAS, because asking every element of the ESAS would have been more artificial to what is the real clinical scenario. In the clinical setting, what guides our interventions is clinical impression, and so we wanted to have an assessment of the value of the clinical impression rather than an ESAS scenario. Our current results strongly confirm those data in a prospective study with consecutive patients.

Many studies have shown that the ESAS is a simple tool that can be completed by patients or their families in a couple of minutes.^12,30 However, it is still not used in daily practice in the vast majority of clinical settings.³⁷ A recent paper explored the journey of the ESAS from initial publication to its yet-unrealized widespread uptake and its use in patient assessment.³⁷ That paper clearly explained the negative consequences of delays in the adoption of clinical tools such as the ESAS. The low level of nurses' impression sensitivity is particularly concerning because it's more serious that nurses fail reporting to the team existing problems (sensitivity) than reporting problems that are not present (specificity). Future research should address ways in which the sensitivity of nurses' assessment can be improved, likely by the use of regular tools rather than clinical impression.

Power consideration were less important in the case of this study because we observed differences, and power might have been a consideration of more importance if we had observed that there were no difference between patients' assessment and NI. However, given information from 118 patients, we would have been able to declare as significant correlation coefficients that were 0.26 or higher, assuming 80% power and a two-sided significance level of 0.05. Given the two group category sizes for patients (e.g., regarding NI for psychological distress on D3; n=47 and n=69), we would have been able to detect differences between groups that were at 0.53 standard deviations or greater (effect size 0.53) for continuous variables, assuming 80% power and a two-sided significance level of 0.05. Then, our sample size would have been sufficient to find, if there was one, an association between the ESAS and NI.

The use of systematic patient-reported assessment is not enough to ensure improvements in palliative care for patients with advanced cancer. Improving symptom management of cancer patients will also require training of health care professionals and regular documentation of assessment findings.¹³ These changes may be challenging for some already overburdened clinical teams.³⁸ Nevertheless, it is likely that such changes can be made: our findings suggest that efforts toward incorporating symptom assessment in daily practice should be done because there is very poor association between patient-reported outcomes and the clinical impression of a specialized palliative care nurse. More research is necessary to identify the tools that are adequate for making these assessments in the context of different symptoms and settings and to develop the training needed by health care providers to integrate these tools and the information they yield into their clinical practice.

Conclusion

The clinical impressions of palliative care nurses showed poor association with patient-reported symptom intensity. Therefore, validated tools are needed for daily clinical assessment of physical and psychosocial distress in advanced cancer patients.

Footnotes

Acknowledgments

We would like to thank Ms. Kathryn Carnes and Mr. Zach Bohannan for manuscript review. Both are supported by M. D. Anderson Cancer Center Support Grant CA016672.

Author Disclosure Statement

This research is supported in part by the National Institutes of Health through M. D. Anderson Cancer Center's Support Grant CA016672. Dr. Bruera's research is supported in part by National Institutes of Health grants R01NR010162-01A1, R01CA122292-01, and R01CA124481-01. No competing financial interests exist. This paper was partially reported at the annual meeting of the American Society of Clinical Oncology, Chicago, June 4, 2011.

References

Fainsinger

, Miller

, Bruera

, Hanson

, Maceachern

. Symptom control during the last week of life on a palliative care unit. J Palliat Care, 1991; 7:5–11.

Fadul

, El Osta

, Dalal

, Poulter

, Bruera

. Comparison of symptom burden among patients referred to palliative care with hematologic malignancies versus those with solid tumors. J Palliat Med, 2008; 11:422–427.

Bruera

, Hui

. Integrating supportive and palliative care in the trajectory of cancer: Establishing goals and models of care. J Clin Oncol, 2010; 28:4013–4017.

Hui

, Parsons

, Nguyen

, Palla

, Yennurajalingam

, Kurzrock

, Bruera

. Timing of palliative care referral and symptom burden in phase 1 cancer patients: A retrospective cohort study. Cancer, 2010; 116:4402–4409.

Kirkova

, Davis

, Walsh

, Tiernan

, O'Leary

, LeGrand

, Lagman

, Russell

. Cancer symptom assessment instruments: A systematic review. J Clin Oncol, 2006; 24:1459–1473.

Grossman

. Assessment of cancer pain: A continuous challenge. Support Care Cancer, 1994; 2:105–110.

Bruera

, Michaud

, Vigano

, Neumann

, Watanabe

, Hanson

. Multidisciplinary symptom control clinic in a cancer center: A retrospective study. Support Care Cancer, 2001; 9:162–168.

Walsh

, Donnelly

, Rybicki

. The symptoms of advanced cancer: relationship to age, gender, and performance status in 1,000 patients. Support Care Cancer, 2000; 8:175–179.

Nekolaichuk

, Bruera

, Spachynski

, MacEachern

, Hanson

, Maguire

. A comparison of patient and proxy symptom assessments in advanced cancer patients. Palliat Med, 1999; 13:311–323.

10.

Lloyd-Williams

, Dennis

, Taylor

. A prospective study to determine the association between physical symptoms and depression in patients with advanced cancer. Palliat Med, 2004; 18:558–563.

11.

Delgado-Guay

, Parsons

, Li

, Palmer

, Bruera

. Symptom distress in advanced cancer patients with anxiety and depression in the palliative care setting. Support Care Cancer, 2009; 17:573–579.

12.

Bruera

, Kuehn

, Miller

, Selmser

, Macmillan

. The Edmonton Symptom Assessment System (ESAS): A simple method for the assessment of palliative care patients. J Palliat Care, 1991; 7:6–9.

13.

Dudgeon

, Harlos

, Clinch

. The Edmonton Symptom Assessment Scale (ESAS) as an audit tool. J Palliat Care, 1999; 15:14–19.

14.

Breivik

, Cherny

, Collett

, de Conno

, Filbet

, Foubert

, Cohen

, Dow

. Cancer-related pain: A pan-European survey of prevalence, treatment, and patient attitudes. Ann Oncol, 2009; 20:1420–1433.

15.

Sollner

, DeVries

, Steixner

, Lukas

, Sprinzl

, Rumpold

, Maislinger

. How successful are oncologists in identifying patient distress, perceived social support, and need for psychosocial counselling? Br J Cancer, 2001; 84:179–185.

16.

Eder

, Sloan

, Todd

. Documentation of ED patient pain by nurses and physicians. Am J Emerg Med, 2003; 21:253–257.

17.

Bainbridge

, Seow

, Sussman

, Pond

, Martelli-Reid

, Herbert

, Evans

. Multidisciplinary health care professionals' perceptions of the use and utility of a symptom assessment system for oncology patients. J Oncol Pract, 2011; 7:19–23.

18.

Little

, Dionne

, Eaton

. Nursing assessment of depression among palliative care cancer patients. J Hospice Palliat Nurs, 2005; 7:98–105.

19.

Lloyd-Williams

, Payne

. Nurse specialist assessment and management of palliative care patients who are depressed—a study of perceptions and attitudes. J Palliat Care, 2002; 18:270–274.

20.

Garyali

, Palmer

, Yennurajalingam

, Zhang

, Pace

, Bruera

. Errors in symptom intensity self-assessment by patients receiving outpatient palliative care. J Palliat Med, 2006; 9:1059–1065.

21.

Nekolaichuk

, Maguire

, Suarez-Almazor

, Rogers

, Bruera

. Assessing the reliability of patient, nurse, and family caregiver symptom ratings in hospitalized advanced cancer patients. J Clin Oncol, 1999; 17:3621–3630.

22.

Palma

, Del Rio

, Bonati

, Tupper

, Villarroel

, Olivares

, Nervi

. [Frequency and assessment of symptoms in hospitalized patient with advanced chronic diseases: Is there concordance among patients and doctors?] Rev Med Chil, 2008; 136:561–569.

23.

Higginson

, McCarthy

. Validity of the support team assessment schedule: Do staffs' ratings reflect those made by patients or their families? Palliat Med, 1993; 7:219–228.

24.

Ewing

, Rogers

, Barclay

, McCabe

, Martin

, Campbell

, Todd

. Palliative care in primary care: A study to determine whether patients and professionals agree on symptoms. Br J Gen Pract, 2006; 56:27–34.

25.

Brunelli

, Costantini

, Di Giulio

, Gallucci

, Fusco

, Miccinesi

, Paci

, Peruselli

, Morino

, Piazza

, Tamburini

, Toscani

. Quality-of-life evaluation: When do terminal cancer patients and health-care providers agree? J Pain Symptom Manage, 1998; 15:151–158.

26.

Mantyselka

, Kumpusalo

, Ahonen

, Takala

. Patients' versus general practitioners' assessments of pain intensity in primary care patients with non-cancer pain. Br J Gen Pract, 2001; 51:995–997.

27.

Puntillo

, Neighbor

, O'Neil

, Nixon

. Accuracy of emergency nurses in assessment of patients' pain. Pain Manag Nurs, 2003; 4:171–175.

28.

Moro

, Brunelli

, Miccinesi

, Fallai

, Morino

, Piazza

, Labianca

, Ripamonti

. Edmonton symptom assessment scale: Italian validation in two palliative care settings. Support Care Cancer, 2006; 14:30–37.

29.

Chang

, Hwang

, Feuerman

. Validation of the Edmonton Symptom Assessment Scale. Cancer, 2000; 88:2164–2171.

30.

Carvajal

, Centeno

, Watson

, Bruera

. A comprehensive study of psychometric properties of the Edmonton Symptom Assessment System (ESAS) in Spanish advanced cancer patients. Eur J Cancer, 2011; 47:1863–1872.

31.

Richardson

, Jones

. A review of the reliability and validity of the Edmonton Symptom Assessment System. Curr Oncol, 2009; 16:55.

32.

Bruera

, MacMillan

, Hanson

, MacDonald

. The Edmonton staging system for cancer pain: Preliminary report. Pain, 1989; 37:203–209.

33.

Cheung

, Barmala

, Zarinehbaf

, Rodin

, Le

, Zimmermann

. The association of physical and psychological symptom burden with time to death among palliative cancer outpatients. J Pain Symptom Manage, 2009; 37:297–304.

34.

Butt

, Wagner

, Beaumont

, Paice

, Peterman

, Shevrin

, Von Roenn

, Carro

, Straus

, Muir

, Cella

. Use of a single-item screening tool to detect clinically significant fatigue, pain, distress, and anorexia in ambulatory cancer practice. J Pain Symptom Manage, 2008; 35:20–30.

35.

Vignaroli

, Pace

, Willey

, Palmer

, Zhang

, Bruera

. The Edmonton Symptom Assessment System as a screening tool for depression and anxiety. J Palliat Med, 2006; 9:296–303.

36.

Selby

, Cascella

, Gardiner

, Do

, Moravan

, Myers

, Chow

. A single set of numerical cutpoints to define moderate and severe symptoms for the Edmonton Symptom Assessment System. J Pain Symptom Manage, 2010; 39:241–249.

37.

Cummings

, Biondo

, Campbell

, Stiles

, Fainsinger

, Muise

, Hagen

. Can the global uptake of palliative care innovations be improved? Insights from a bibliometric analysis of the Edmonton Symptom Assessment System. Palliat Med, 2011; 25:71–82.

38.

Watanabe

, McKinnon

, Macmillan

, Hanson

. Palliative care nurses' perceptions of the Edmonton Symptom Assessment Scale: A pilot survey. Int J Palliat Nurs, 2006; 12:111–114.