Development of a French Version of the Edmonton Symptom Assessment System-Revised: A Pilot Study of Palliative Care Patients' Perspectives

Abstract

Background:

The Edmonton Symptom Assessment System-revised (ESAS-r) is a nine-item self-report symptom intensity tool developed for palliative care patients, with the option of adding a 10th patient-specific symptom. Due to growing international uptake, the ESAS-r has been translated into different languages. There has not been agreement, however, regarding a standard process for translation into multiple languages, which also includes patients' perspectives.

Objective:

The purpose of this study was to develop a French version of the ESAS-r, using a standardized translation protocol, and to obtain palliative care patients' perspectives regarding this translated tool.

Design:

We developed a French version of the ESAS-r, using a standard translation method, involving both professional translators (n = 2) and bilingual palliative care experts (n = 3). Fifteen Francophone participants recruited from palliative care sites in two urban centers in Canada completed the ESAS-r and provided feedback on the translation, in the presence of a trained interviewer. Descriptive statistics and thematic analysis were used to analyze the quantitative and qualitative data, respectively.

Setting/Subjects:

Fifteen Francophone participants were recruited from palliative care sites in two urban centers in Canada.

Measurements:

Participants completed the ESAS-r and provided feedback on the translation in the presence of a trained interviewer. Descriptive statistics and thematic analysis were used to analyze the quantitative and qualitative data, respectively.

Results:

Based on participants' concerns, translations for four of the nine symptoms were revised: drowsiness, nausea, lack of appetite, and shortness of breath. Concerns expressed for three additional symptoms (depression, anxiety, and well-being) were related to overall difficulty rating these symptoms, not specific to the translation.

Conclusion:

The French version of the ESAS-r is a credible tool for symptom assessment in Francophone patients. The study findings provide a vital step in the development of a standardized translation protocol, including patients' perspectives, which can be applied to other languages.

Introduction

The Edmonton Symptom Assessment System (ESAS) is a self-report tool for assessing symptom intensity of nine specific symptoms, with the option of adding a 10th patient-specific symptom.¹ It was designed to capture multidimensional symptom profiles over time by obtaining repeated quantitative measurements of symptom intensity with minimal patient burden. The ESAS was initially developed for advanced cancer patients in hospitals and outpatient clinic settings, though it has been applied in patients with noncancer illnesses, such as AIDS,² renal diseases,³ and cardiopulmonary diseases.⁴

Since its original inception by Bruera et al. in 1991,¹ the ESAS has been adopted in diverse palliative care programs and countries. A bibliometric analysis of the ESAS highlighted a rapid and multinational uptake: Between 1991 and 2006, 222 documents from 25 countries around the world made reference to the original ESAS publication, where it has been used for clinical, research, and administrative purposes.⁵ A series of validation studies have complemented the rapid clinical uptake of the ESAS, providing further evidence for its psychometric properties and clinical utility.^6,7

Despite its broad clinical uptake, there are some challenges in using the ESAS in clinical practice, including potential errors in symptom ratings.^8–10 A revised version of the ESAS-revised (ESAS-r) was developed in response to these concerns. The ESAS-r retains the core elements of the ESAS, with key revisions focusing on symptom assessment time frame, terminology, item order, and format.¹¹ In a multicenter study comparing the ESAS and the ESAS-r, patients rated both versions as very easy to understand and complete, though the ESAS-r was significantly easier to understand than the ESAS because of its additional definitions, clarity, and revised format.¹¹ Similar to the ESAS, the ESAS-r is a screening tool for capturing patients' perspectives on symptom severity at a point in time, as opposed to a comprehensive symptom assessment in itself. Repeated assessments may help track changes in symptom severity over time. Detailed guidelines for use of the ESAS-r may be accessed through the Edmonton Zone Palliative Care Program website.¹²

Due to the growing international interest in and need for common symptom assessment tools in palliative care settings, the ESAS has been translated into different languages, including Spanish,^13,14 Thai,¹⁵ and Italian.¹⁶ More recently, the ESAS-r has also been translated into other languages, including Spanish,^17,18 Brazilian Portuguese,¹⁹ Japanese,²⁰ and Icelandic.²¹

Despite this widespread international interest, a single standard process for translation into multiple languages, which also includes patients' perspectives, does not currently exist. For translations into multiple languages, it is essential to produce accurate translations, which are not only conceptually equivalent to the original language version, but are also culturally relevant to the target population. Few published methodologies have been available in producing global translations of patient-reported outcome measures.^22–24

The primary purpose of this study was to develop a French version of the ESAS-r, using a rigorous standardized translation method that included patients' perspectives. A secondary purpose was to develop a standardized translation protocol that could be used in future translation studies. Ultimately, the intention of this study was to provide palliative care clinicians with a simple symptom assessment tool that could potentially be used routinely with French-speaking patients and family members.

Methods

Study design

A standardized translation protocol, developed specifically for this study, was adapted from three sources: (1) European Organization for Research and Treatment of Cancer (EORTC) Quality of Life Group translation procedure²⁵; (2) European Palliative Care Research Centre (PRC) translation guidelines for the European Palliative Care Cancer Study (EPCCS)²⁶; and (3) cross-cultural adaptation of the Disabilities of the Arm, Shoulder and Hand (DASH) Questionnaire in French.²⁷ As shown in Figure 1, the protocol consisted of five steps (see Appendix for detailed description of protocol):

Step 1: Forward translation from English to French: The ESAS-r was initially translated into French by two research team members, who were native speakers of the Canadian French language, had a high level of fluency in English and were also palliative care professionals with experience using the ESAS or ESAS-r clinically, resulting in two independent forward translations (FWT1 and FWT2).

Step 2: Reconciliation of the two forward translations: These two forward translations (FWT1 and FWT2) were merged into a single provisional forward translation (FWT12), coordinated by a third research team member, who was fluent in Canadian French and had an expertise in palliative care.

Step 3: Backward translation from French to English: Two professional translators, who were native English speakers with a high level of fluency in Canadian French, independently and without knowledge of the original English version of the ESAS-r, translated the provisional forward translation (FWT12) back into English. This process resulted in two backward translations: backward translation version 1 (BWT1) and backward translation version 2 (BWT2).

Step 4: Review and revision by expert panel: An expert panel, consisting of all of the research team members, compared the two backward translations (BWT1 and BWT2) with the original English version of the ESAS-r, resulting in a prefinal French version.

Step 5: Pilot-testing of prefinal French version: The prefinal French version of the ESAS-r was pilot-tested with a convenience sample of 15 patients. The aim of this step was to identify and solve any potential problems in translation (e.g., wording which was confusing or difficult to understand) and to ensure that the French version was clear and understandable from the patients' perspectives, using structured interview questions.

FIG. 1.

Stepwise standardized translation protocol. BWT1, backward translation version 1; BWT2, backward translation version 2.

The study received ethical approval from the appropriate research ethics boards in the two collaborating centers that were used to recruit patients: (1) Edmonton: University of Alberta Health and Research Ethics Board (Pro00026732); and (2) Ottawa: Ottawa Hospital Research Ethics Board (2012006-01H), Bruyère (M16-11-039), and Monfort (CN-14-02-12).

Sample

A convenience sample of 15 patients with progressive incurable illness (cancer and noncancer) was recruited for the study, using the following eligibility criteria:

1. Patients with advanced cancer or progressive nonmalignant diseases;

2. Native speakers of the French language, who were fluent in French and used it as their principal language;

3. 18 Years of age or older;

4. Patients receiving palliative care;

5. Cognitively intact [based on cognitive screening tool, such as Mini Mental State Examination (MMSE) adjusted for age and education, or Short Orientation Memory Concentration Test (SOMCT)].

6. Able to complete the French version of the ESAS-r and provide feedback.

The sample size for this study was based on previous cognitive probing or think aloud studies conducted by our study group¹⁰ and others,^28–30 as well as general guidelines for conducting qualitative research.³¹ Sample sizes tend to be quite small, ranging from a few cases to >10. The determination of sample size is based on the quality of participant characteristics and experiences, representing the number needed to achieve data saturation (i.e., when there are no new themes emerging from the data).

Measures

Edmonton Symptom Assessment System-revised

The ESAS-r is a revised version of the original ESAS,¹⁰ with some modifications, as previously discussed. Each symptom is rated along a numerical rating scale, ranging from 0 (not at all or no symptom) to 10 (worst possible symptom), based on current symptom intensity (“right now”). Each scale is interpreted independently of the other scales, although it is also possible to calculate a total symptom distress score¹⁰ (Fig. 2).

FIG. 2.

Edmonton System Assessment System-revised (ESAS-r).

Patient demographics

The team member conducting the interview documented the following demographics from the patient's medical record: age, gender, marital status, education level (Edmonton sites only), diagnosis, time since initial diagnosis, current cancer treatment, MMSE (Edmonton sites only),³² SOMCT (Ottawa sites only),³³ and Palliative Performance Scale (version 2).³⁴

Structured interview questionnaire

The research team members developed a structured questionnaire to assess the translated items for the following concerns: (1) difficult to answer, (2) confusing, (3) difficult to understand, (4) upsetting/offensive, and (5) whether the patient would have used different terminology. Additional questions focused on (1) difficulty understanding instructions, (2) ease of understanding tool, (3) ease of completing tool, and (4) previous exposure to ESAS/ESAS-r.

Data collection

Patients were recruited from the following palliative care settings: (1) Palliative Care Consultation Service, Montfort Hospital (Ottawa) (n = 11); (2) Tertiary Palliative Care Unit, Grey Nuns Community Hospital (Edmonton) (n = 2); and (3) Symptom Control and Palliative Care Service, Cross Cancer Institute, Edmonton (n = 2). A clinical healthcare provider who was part of the patient's primary care team initially informed patients about the study. If a patient was interested, then a research assistant (Ottawa site) or bilingual research team member (A.H., Edmonton sites) informed the patient about the study and obtained written consent. Participants independently completed the prefinal French version of the ESAS-r and responded to a 30-minute (approximate) structured interview. The team member documented the participants' verbal responses in writing. All interview sessions were also audiotaped.

Before initiating the study, the research assistant was trained in administering the ESAS-r, using the ESAS-r administration guidelines.¹¹ The research team member who was involved in data collection at the Edmonton sites (A.H.) was a palliative care physician who had routinely used the ESAS-r in clinical practice. All research team members participated in a teleconference training session to review the study procedures. Data collection occurred from September 2012 to April 2013.

Data analysis

Descriptive statistics [frequencies, means (M), and standard deviations (SDs)] and thematic analysis³¹ were used to analyze the quantitative and qualitative data, respectively. The trustworthiness of the qualitative findings was addressed by maintaining a detailed audit trail to document the research process.

Results

A summary of the participant characteristics and pain intensity scores appears in Tables 1 and 2, respectively. As shown in Table 1, most participants were cancer patients (n = 14, 93%), with a median age of 72 years. The majority of patients were male (n = 10, 67%), and married (n = 10, 67%). Almost half (n = 6, 40%) had previously been exposed to the ESAS or ESAS-r. As shown in Table 2, the two highest rated symptoms were tiredness (M = 4.8, SD = 2.3) and well-being (M = 4.2, SD 2.8).

Table 1.

Patient Demographics

Characteristic	Frequency, n (%)
Gender
Male	10 (67)
Female	5 (33)
Marital status
Married/common law	10 (67)
Widowed	2 (13)
Divorced	1 (7)
Other	2 (13)
Primary cancer diagnosis
Gastrointestinal	5 (33)
Genitourinary	3 (20)
Lung	2 (13)
Other cancer	4 (27)
Noncancer	1 (7)
Current cancer treatment
Radiotherapy	2 (13)
Other	3 (20)
None	10 (67)
Education level (years)
5–8	1 (7)
9–12	2 (13)
>12	6 (40)
Unknown	6 (40)
Location of recruitment
Ottawa	11 (73)
Edmonton	4 (27)
Previous exposure to ESAS and/or ESAS-r
Yes	6 (40)
No	8 (53)
Unknown	1 (7)
Patient type
Inpatient	13 (87)
Outpatient	2 (13)

Characteristics	Median (range)	Mean (SD)
Age (years)	72 (34–83)	70 (13)
Palliative performance scale (0–100%)	60 (30–80)	55 (14)

ESAS-r, Edmonton Symptom Assessment System-revised; SD, standard deviation.

Table 2.

Edmonton Symptom Assessment System-Revised Symptom Ratings

ESAS-r symptoms ^a	Mean	SD
Pain	3.1	2.4
Tiredness	4.8	2.3
Drowsiness	4.0	2.3
Nausea	0.8	1.5
Appetite	3.2	3.0
Shortness of breath	3.5	3.0
Depression	3.1	3.2
Anxiety	2.8	3.0
Well-being	4.2	2.8

Other symptoms/problems (10th patient specific symptom) discluded due to missing data.

In terms of using the tool, one-third of the participants (n = 5, 33.3%) reported that the instructions for completing the tool were difficult to understand. However, most participants (n = 9, 60%) reported that, overall, the tool was very easy to understand and complete. On a scale from 1 (very easy) to 5 (very hard), all participants rated these two questions (i.e., ease of understanding and ease of completion) as three or less.

Table 3 provides a summary of the participants' comments and final recommendations proposed by the research team. As shown in Table 3, based on participants' concerns, translations for four of the nine symptoms were revised: drowsiness, nausea, lack of appetite, and shortness of breath. Concerns expressed for three additional symptoms (depression, anxiety, and well-being) were related to overall difficulty rating these symptoms, not specific to the translation. There were no concerns for pain and tiredness. Based on these comments and recommendations, a final French version of the ESAS-r was created (Fig. 3).

FIG. 3.

ESAS-r final French version.

Table 3.

Qualitative Responses

Question	Summary of patients' comments	Final recommendations of changes
Pain	Question difficult, but clear.	No changes
Tiredness	Question difficult, subjective, varies over time, but clear.	No changes
Drowsiness	Drowsiness because Somnolence has no specific meaning for someone with grade 10 education;	Use Somnolence (drowsiness) as main line with brackets: “Se sentir très éveillé” (feel very awake) [left]
	Explanation in brackets was helpful.	“Se sentir très endormi” (feel very sleepy) [right]
		Add additional descriptors in guidelines
Nausea	Abstract question because it depends on some factors (e.g., The gender…);	Use Envie de vomir (want to vomit) as main line with “Envie de vomir = nausée” in brackets;
	Keep Envie de vomir instead of nausea, which is not commonly used and difficult to understand	Add additional descriptors in guidelines
Lack of appetite	Could be replaced by: good appetite or no desire to eat;	Use Très bon appétit (very good appetite) instead of “aucun manque appétit” (no lack of appetite) and Aucun appétit (no appetite) instead of “Le pire manque d'appétit possible” (the worst possible lack of appetite)
	Replace poor appetite by difficulty swallowing (or include difficulty swallowing in the item)
	Difficulty to swallow instead of absence of
Shortness of breath	Could be replaced by: difficulty to breathe or normal breathing;	Add “Manquer de souffle” (shortness of breath) in brackets underneath Aucun essoufflement (no shortness of breath)
	Gasping for air
Depression	Do not consider depression = “tristesse”(sad), because someone can be sad without depression;	No changes. Consider further descriptions in guidelines
	Separate depression and sadness;
	Depression and anxiety are two similar terms and difficult to rate separately
Anxiety	Difficulty rating anxiety because difficult to define even with explanations in brackets. Anxiety can also mean suffocation, feeling caught in a net;	No changes. Consider further descriptions in guidelines
	Nervous is too vague and general;
	Depression and anxiety are two similar terms and difficult to rate separately
Well-being	Difficulty with well-being. Better phrase would be: how do you feel in general? Not useful to say “Sentiment de bien-être”;	No changes. Keep same order as original translation
	Sense of well-being was upsetting because asks about something very personal
Other symptoms	Could be replaced by: No other problem? All is good?	Add Autre problème (p.e. constipation) [Other problems (e.g., constipation)] on first line before rating scale

Discussion

The ESAS-r is a widely used screening tool for capturing the patients' perspectives on various symptoms at a point in time. Although it has been translated into other languages^17–21 a standardized translation process has not been used across the studies. Hence, this is the first study focusing on the translation of the ESAS-r into French, using a standardized process and patients' perspectives. For most of the symptoms of the ESAS-r, we found no substantial difficulty in the translation process to French. However, in common with the Spanish,^13,14,17,18 Brazilian Portuguese,¹⁹ Italian,¹⁶ and Thai¹⁵ translations, some terms had to be modified or definitions added to better reflect their meaning.

Based on the patients' input, the translation for pain to douleur was not associated with any difficulty, similar to the translations in Spanish,^13,14,17,18 Italian,¹⁶ or Brazilian Portuguese.¹⁹ Interestingly, the authors of the Thai version¹⁵ noted that the term, pain, was too broad a concept, reflecting some cultural and religious differences in this population.

In our study, the translation of the symptom, tiredness, into fatigue caused no difficulty. However, in the Spanish translation^17–18 the term agotado (exhaustion) was accompanied by two definitions, causancio (fatigue) and debilidad (weakness), to clarify the meaning.

The translation of three of the symptoms led to some confusion for some of the participants: drowsiness translated to somnolence, nausea translated to nausée, and shortness of breath translated to essoufflement. Therefore, we added a definition for each symptom, respectively: “se sentir très éveillé/endormi” (feel very awake/sleepy), “envie de vomir” (want to vomit), and “manquer de souffle” (shortness of breath). It is also worth mentioning that we used the definition for nausea, envie de vomir, as the main term to improve patients' understanding, since the word, nausea, is not frequently used in the Francophone population. Some Francophones may spontaneously use colloquial terms, such as “mal au coeur” (heart ache), to express nausea. All of the participants in this study, however, understood the term, envie de vomir, and did not mention “mal au coeur” as an option. Since envie de vomir is a better description of the actual symptom, it was chosen as the main term. When using the tool in clinical practice, however, it would be important to consider using colloquial terms to provide further explanation, if patients are having difficulty understanding the standard terminology.

Similar to the Brazilian-Portuguese version,¹⁹ the symptom, lack of appetite, was causing some difficulty due to the double negative (no lack of appetite), which was difficult to translate in French. As a consequence, we changed the wording by using an affirmative statement on the left (très bon appetit, very good appetite) and a single negative statement on the right (aucun appetit, no appetite).

The symptom, depression, has been previously discussed in the Spanish version^13,14,17 and the Spanish term, desanimado (discouraged), was chosen as better reflecting the meaning, instead of deprimido (depressed) or triste (sad). In our version, we opted to use dépression (depression) with the definition tristesse (sadness), same as in the original English version. Some of our patients commented that depression and sadness were not equivalent and were recommended keeping only the word, sadness. However, we chose to keep the format the same, based on the original ESAS-r version, even though depression and sadness do not necessarily have the same meaning.

The symptom anxiety was translated to anxiété, with a definition provided (feeling nervous, “se sentir nerveux”). Participants did not report any difficulty with this translation, whereas in the Spanish version,^13,14,17,18 three terms—ansioso (anxious), intranquilo (disquiet), or nervioso (nervous)—were recommended to screen for anxiety.

The term, well-being, was difficult to translate in our study, similar to other translations. For example, as pointed out in the Italian translation of the ESAS,¹⁶ well-being has a wide meaning and can underlie several domains, including perception of terminal stage, care in hospital or at home, social difficulties or body image. Subsequently, it may not directly correlate with worsening of symptom scores. In our study, it was also a source of confusion for some patients, who preferred the definition “comment vous sentez en général” (how you feel overall). In addition, worst possible well-being was difficult to translate in French due to lack of clarity, so we opted for aucune sensation de bien-être (no sense of well-being). The use of the term, aucune sensation de bien- être, represents the extreme end of the scale opposite to meilleure sensation de bien-être (best possible sense of well-being). This term is more meaningful than the direct translation from English, “pire sensation de bien-être” (worst possible sense of well-being). The participants did not have difficulty understanding this translation. It was further clarified by the definition included in parentheses (bien-être = comment vous vous sentez en général), which translates to “How do you feel in general?.” In the Thai version,¹⁵ the authors encountered the same situation, so a phrase was added to better describe the condition (“Sabai Dee Tang Kai Lae Jai,” an English translation of this phrase was not provided). The complexity of the concept, well-being, is further illustrated by its potential use as an overall quality-of-life measure,³⁵ as a concept associated with suffering,³⁶ and as one of the primary endpoints for intervention studies.^37,38 Further validation studies, focusing specifically on this symptom, are warranted.

This study represents a fundamental step in the development of a French version of the ESAS-r, providing preliminary validity evidence for this tool. We used a standardized translation method that involved professional translators, palliative care experts and patients' perspectives, which enhanced the rigor and credibility of the translation process. We adapted our translation method from three reliable sources and conducted our pilot study with 15 Francophone patients. The use of palliative care experts as translators and as the reconciliation coordinator ensured that the translation process would be clinically relevant (step 1) and that the translated terms would be adapted to familiar words for the general French-speaking population (step 2). In step 3, the newly translated version in French was translated back in English to confirm that it was accurately reflecting the original intention of the ESAS-r. This method is easy to use and can be applied to any language. It is also important to note that the three symptoms that patients had difficulty rating—depression, anxiety, and well-being—were similar to the findings from our think aloud study involving patients' perspectives of the English version of the ESAS.¹⁰ These difficulties were related to the overall complexities of rating these symptoms due to their subjective nature and potentially confusing terminology, and not directly related to language translation.

Future research to gather further reliability and validity evidence is necessary. Using Messick's³⁹ validity framework as a guide, additional evidence regarding the internal structure of the tool could be obtained using correlational studies, such as factor analysis, or the assessment of its internal consistency using Cronbach's alpha. The nature of its external structure could be assessed using correlational studies that focus on the relationship of the tool with relevant variables, such as functional status, quality-of-life, and symptom burden. Longitudinal studies, as well as comparisons across different groups (e.g., cancer vs. noncancer) and settings (e.g., inpatient vs. outpatient) would be effective in gathering evidence regarding potential differences in processes and structures over time or across groups and settings. Finally, validity evidence regarding the appropriateness of changes in the tool scores in response to interventions could be gathered using intervention studies.

Limitations

There are several limitations associated with our study. Our sample was small (15 patients). We were unable to obtain the level of education for 6/15 patients (40%), which limited our interpretation of patients' education and their understanding of the terms on the ESAS-r. The study was conducted in Canada, where there are some national differences in the use of French. Thus, the final version of the ESAS-r might vary in other countries. It would, therefore, be useful to repeat the same study in another francophone country, such as France or Belgium, where it is known that some differences are present in the vocabulary.

Conclusions

The development and availability of the ESAS-r in various languages enables diverse populations to complete a version of the tool that is culturally sensitive and adapted to their own vocabulary. Since its inception in 1991, the ESAS and now the ESAS-r have been widely used for clinical, research, and administrative reasons in numerous countries. By using a standardized translation protocol, access to the ESAS-r and collaborations between various palliative care teams around the world will be improved. The importance of providing adequate education for healthcare experts is also needed to ensure that ESAS-r guidelines are followed so patients are able to appropriately express their symptom experience.

Footnotes

Acknowledgments

This work was supported by an Interface Project Grant awarded to Dr. Cheryl Nekolaichuk, which was a collaborative initiative provided by Alberta Health Services, the Canadian Cancer Society, Alberta/N.W.T. Division, and the Alberta Cancer Foundation. The remaining authors had no conflicts of interest to declare. The authors would like to acknowledge the research assistance of Salomon Fotsing, the administrative support of Samantha Zinkie, the coordination of the study at multiple sites by Viki Muller, the analytical and editorial support of Crystal Beaumont and Lisa Fainsinger, and the assistance received from the Covenant Health Palliative Institute. The authors would also like to express a special acknowledgement to our colleague, Dr. Pierre Allard, who is unfortunately no longer with us, but who contributed to the study design and the translation process.

Author Disclosure Statement

No competing financial interests exist.

Appendix: Standardized Translation Protocol

References

Bruera

, Kuehn

, Miller

, et al.: The Edmonton Symptom Assessment System (ESAS): A simple method for the assessment of palliative care patients. J Palliat Care, 1991; 7:6–9.

Brechtl

, Murshed

, Homel

, Marilyn

: Monitoring symptoms in patients with advanced illness in long-term care: A pilot study. J Pain Symptom Manage, 2006; 32:168–174.

Davison

, Jhangri

, Johnson

: Cross-sectional validity of modified Edmonton Symptom Assessment System in dialysis patients: A simple assessment of symptom burden. Kidney Int, 2006; 69:1621–1625.

Walke

, Gallo

, Tinetti

, Fried

: The burden of symptoms among community-dwelling older persons with advanced chronic disease. Arch Intern Med, 2004; 164:2321–2324.

Cummings

, Biondo

, Campbell

, et al.: Can the global uptake of palliative care innovations be improved? Insights from a bibliometric analysis of the Edmonton Symptom Assessment System. Palliat Med, 2011; 25:71–82.

Nekolaichuk

, Watanabe

, Beaumont

: The Edmonton Symptom Assessment System: A 15-year retrospective review of validation studies (1991–2006). Palliat Med, 2008; 22:111–122.

Richardson

, Jones

: A review of the reliability and validity of the Edmonton Symptom Assessment System. Curr Oncol, 2009; 16:53–64.

Watanabe

, McKinnon

, Macmillan

, Hanson

: Palliative care nurses' perceptions of the Edmonton Symptom Assessment Scale: A pilot survey. Int J Palliat Nurs, 2006; 12:111–114.

Garyali

, Palmer

, Yennurajalingam

, et al.: Errors in symptom intensity self-assessment by patients receiving outpatient palliative care. J Palliat Med, 2006; 9:1059–1065.

10.

Watanabe

, Nekolaichuk

, Beaumont

, Mawani

: The Edmonton Symptom Assessment System—What do patients think?. Support Care Cancer, 2009; 17:675–683.

11.

Watanabe

, Nekolaichuk

, Beaumont

, et al.: A multicenter study comparing two numerical versions of the Edmonton Symptom Assessment System in palliative care patients. J Pain Symptom Manage, 2011; 41:456–468.

12.

Edmonton Zone Palliative Care Program: Edmonton Symptom Assessment System–revised (ESAS-r) guidelines for administration. 2010. Available at www.palliative.org/search-results.html?q=Edmonton%20Symptom%20Assessment%20System%20%E2%80%93Revised%20(ESAS-r)%20guidelines%20for%20administration (Last accessed June 25, 2016).

13.

Noguera

, Centeno

, Carvajal

, et al.: “Are you discouraged? Are you anxious, nervous or uneasy?”: In Spanish some words could be better than others for depression and anxiety screening. J Palliat Med, 2009; 12:707–712.

14.

Carvajal

, Centeno

, Watson

, Bruera

: A comprehensive study of psychometric properties of the Edmonton Symptom Assessment System (ESAS) in Spanish advanced cancer patients. Eur J Cancer, 2011; 47:1863–1872.

15.

Chinda

, Jaturapatporn

, Kirshen

, Udomsubpayakul

: Reliability and validity of a Thai version of the Edmonton Symptom Assessment Scale (ESAS-Thai). J Pain Symptom Manage, 2011; 42:954–960.

16.

Moro

, Brunelli

, Miccinesi

, et al.: Edmonton symptom assessment scale: Italian validation in two palliative care settings. Support Care Cancer, 2006; 14:30–37.

17.

Carvajal

, Martinez

, Centeno

: The ESAS in patients with advanced cancer: A review of studies of the Spanish version. Eur J Palliat Care, 2012; 19:77–81.

18.

Carvajal

, Hribernik

, Duarte

, et al.: The Spanish version of the Edmonton Symptom Assessment System-Revised (ESAS-r): First psychometric analysis involving patients with advanced cancer. J Pain Symptom Manage, 2013; 45:129–136.

19.

Monteiro

, Almeida

, Kruse

MHL

: Translation and cross-cultural adaptation of the Edmonton Symptom Assessment System for use in palliative care. Rev Gaúcha Enferm, 2013; 34:163–171.

20.

Yokomichi

, Morita

, Nitto

, et al.: Validation of the Japanese version of the Edmonton Symptom Assessment System-revised. J Pain Symptom Manage, 2015; 50:718–723.

21.

Gretarsdottir

, Fridriksdottir

, Gunnarsdottir

: Psychometric properties of the Icelandic version of the revised Edmonton Symptom Assessment Scale. J Pain Symptom Manage, 2016; 51:133–137.

22.

Two

, Vergee-Lorenz

, Clayson

, et al.: A methodology for successfully producing global translations of patient reported outcome measures for use in multiple countries. Value Health, 2010; 13:128–131.

23.

Bullinger

, Alonso

, Apolone

, et al.: Translating health status questionnaires and evaluating their quality: The IQOLA project approach. J Clin Epidemiol, 1998; 51:913–923.

24.

Maneesriwongul

, Dixon

: Instrument translation process: A methods review. J Advanced Nursing, 2004; 48:175–186.

25.

Kuliś

, Bottomley

, Velikova

, et al.: EORTC quality of life group translation procedures. 2016. Available at http://groups.eortc.be/qol/sites/default/files/img/newsletter/translation_manual_2016_compressed.pdf (Last accessed June 25, 2016).

26.

Hjermstad

, Aass

, Aielli

, et al.: Characteristics of the case mix, organisation and delivery in cancer palliative care: A challenge for good-quality research. BMJ Support Palliat Care, 2016:1–12. DOI:10.1136/bmjspcare-2015-000997.

27.

Durand

, Vachon

, Hong

, Loisel

: The cross-cultural adaptation of the DASH Questionnaire in Canadian French. J of Hand Therapy, 2005; 18:34–39.

28.

Nielsen

: Estimating the number of subjects needed for a thinking aloud test. IJHCS, 1994; 41:385–397.

29.

Smith

, Vart

, Wolf

, et al.: How do people interpret information about colorectal cancer screening: Observations from a think-aloud study. Health Expect, 2015; 18:703–714.

30.

Lundgrén-Laine

, Salanterä

: Think-aloud technique and protocol analysis in clinical decision-making research. Qual Health Res, 2010; 20:565–575.

31.

Patton

: Qualitative Research & Evaluation Methods, 3rd ed. Thousand Oaks, CA: Sage Publications, Inc., 2002.

32.

Folstein

, Folstein

, McHugh

: “Mini-Mental State” a practical method for grading the cognitive state of patients for the clinician. J Psychiat Res, 1975; 12:189–198.

33.

Katzman

, Brown

, Fuld

, et al.: Validation of a short Orientation-Memory-Concentration Test of cognitive impairment. Am J Psychiatry, 1983; 140:734–739.

34.

Anderson

, Downing

, Hill

, et al.: Palliative Performance Scale (PPS): A new tool. J Palliat Care, 1996; 12:5–11.

35.

Bush

, Parsons

, Palmer

, et al.: Single- vs. multiple-item instruments in the assessment of quality of life in patients with advanced cancer. J Pain Symptom Manage, 2010; 39:564–571.

36.

Best

, Aldridge

, Butow

, et al.: Conceptual analysis of suffering in cancer: A systematic review. Psychooncology, 2015; 9:977–986.

37.

Wai

, Finkelstein

, Tangente

, et al.: Quality of life in surgical treatment of metastatic spine disease. SPINE, 2003; 28:508–512.

38.

Chow

, Davis

, Holden

, et al.: Prospective assessment of patient-rated symptoms following whole brain radiotherapy for brain metasteses. J Pain Symptom Manage, 2005; 30:18–23.

39.

Messick

: Validity. In: Linn

(ed): Educational Measurement, 3rd ed. New York: Macmillan, 1989, pp. 13–103.