Comparison of Patient-Reported Outcome Measures for Use as Performance Metrics in Adolescent and Young Adult Psychosocial Cancer Care

Abstract

Purpose:

To compare the Cancer Distress Scales for Adolescents and Young Adults (CDS-AYA)—Emotional and Impact scales—with the Edmonton Symptom Assessment Scale–revised (ESAS-r), Hospital Anxiety and Depression Scale (HADS), and the National Comprehensive Cancer Network Distress Thermometer (NCCN-DT) for use as a patient-reported outcome—performance measure (PRO-PM) for AYA cancer care.

Methods:

Data were collected as part of the field test study of CDS-AYA. Assessment criteria were based on a National Quality Forum report. Internal and test-retest (TRT) reliability was assessed using Cronbach's alpha and intraclass correlation coefficients, respectively. A content sort of items in each scale was performed to determine AYA-specific content. Two predefined hypotheses for gender and treatment status were assessed using t-test.

Results:

Four hundred twenty-one participants were included in the analyses. Cronbach's alpha ranged from 0.79 to 0.94, with only the two scales of CDS-AYA achieving >0.90. TRT reliability for HADS was considered excellent (>0.90), with remaining scales having moderate to good reliability (>0.70). Only the Impact scale of CDS-AYA had items that addressed the specific concerns of AYAs (e.g., romantic relationships). In all scales, active therapy patients had higher levels of distress than patients not receiving treatment (p ≤ 0.05). Distress was greater for females than males for all scales, but nonsignificant for both the ESAS-r (p = 0.07) and the HADS depression subscale (p = 0.13). The proportion of AYAs screening positive for distress varied by instrument.

Conclusion:

The Impact scale of CDS-AYA met the most criteria for selection of a PROM for use in performance measurement for the AYA cancer population.

Introduction

Adolescents and young adults (AYAs, 15–39 years of age) with cancer require an approach to psychosocial care, unlike that of children or adults, which considers their developmental stage and associated unique needs.¹ Important psychosocial concerns in the AYA cancer population can include: increased reliance on family after gaining independence, isolation from peers, and interference of cancer with education or career development.^1,2 To address these concerns, AYA-specific health care programs that are flexible with AYA-specific support and resources are needed.¹ AYA psychosocial care plans are also necessary to enable routine psychosocial screening and to help patients gain access to AYA-specific psychosocial experts and community resources.³

To ensure that newly developed programs and treatments are improving outcomes in this population, it is necessary to have appropriate metrics. A recent report identified a lack of suitable metrics for performance measurement in psychosocial care of AYAs with cancer.⁴ Performance measures (PMs) provide a quantification of health care quality for health care entities such as hospitals or long-term care homes (e.g., proportion of patients meeting a target wait time for diagnostics). These measures can capture many different types of outcomes, including patient-reported outcomes (PROs) such as quality of life.⁵ A PRO-PM is a type of metric captured using a patient-reported outcome measure (PROM) that provides the patient's perspective on aspects of care and well-being that are important to them (e.g., proportion of patients reporting high levels of satisfaction with care).⁵ The use of PRO-PMs may be particularly important in the area of psychosocial care where many outcomes such as depression, anxiety, and distress are not readily observable. Distress has been recognized as an important issue in the area of psychosocial care of all cancer patients, and screening for distress is recommended.⁶ The importance of measuring distress as an outcome in cancer care has been previously described^7,8 along with its significance to AYA-aged cancer patients and survivors.^1–3,9

There are many different PROMs that measure distress in the cancer population¹⁰; however, many have not been evaluated for use in AYAs.¹¹ In Canada, the Edmonton Symptom Assessment Scale–revised (ESAS-r) has been selected for use in cancer distress screening.^7,8 Although the ESAS-r is extensively used and validated, most work has been done in adult cancer populations.^12,13 The National Comprehensive Cancer Network (NCCN) Distress Thermometer (DT) is another instrument commonly used internationally.^14,15 Recklitis et al.¹⁶ compared the NCCN-DT with a psychiatric diagnostic interview in a young adult cancer survivor population. This study concluded that the NCCN-DT should not be used as a stand-alone screening tool in this population because it did not meet the criteria for acceptable sensitivity or specificity.¹⁶

An AYA-specific tool to measure distress, developed in Australia, comprises a detailed problem checklist (PCL) and modified distress thermometer. This tool served as the basis for the recently developed Cancer Distress Scales (CDS)-AYA, a set of scales designed specifically for this population.^11,17 The CDS-AYA have not yet been extensively used in research or clinical practice.¹⁷ Given the number of PROMs used to measure distress in cancer patients that are not AYA specific, it is important to evaluate PROMs for use in the development of PRO-PMs for the AYA cancer population.

The National Quality Forum (NQF) uses four criteria to endorse a PRO-PM, including importance, scientific acceptability, feasibility, and usability.¹⁸ Patient-centered measurement is a key recommendation from the NQF for developing PRO-PMs to ensure that the selected PROM is both meaningful and valued by the target population.^18–20 In an article commissioned by NQF, Cella et al.²¹ further describe the characteristics to consider when selecting an instrument to develop a PRO-PM, including (1) a conceptual and measurement model; (2) reliability (internal consistency and test-retest [TRT]); (3) validity (content, construct, criterion, and responsiveness); (4) interpretability of scores (minimally important differences, reference means, and interpretation guide); (5) burden (time and effort for both respondent and administrator); (6) alternative modes of administration (e.g., phone and touch screen); (7) cultural and language adaptations; and (8) electronic health records (e.g., automated real-time scoring).

The purpose of this study was to assess the CDS-AYA, ESAS-r, Hospital Anxiety and Depression Scale (HADS), and NCCN-DT for use in performance measurement for AYA cancer care.

Methods

Participants

AYAs were defined as persons aged 15–39 years consistent with the LIVESTRONG Young Adult Alliance²² and Canadian Partnership Against Cancer (CPAC)⁴ definitions. Participants were included if they had been diagnosed with cancer and were either undergoing or had completed treatment.

Data collection

This study used data collected for the field test of CDS-AYAs (see Tsangaris et al.¹⁷ for detailed methods). Ethics approval was obtained from all institutions. Questionnaires were completed during a clinic visit and the package included self-complete demographic and clinical information, CDS-AYAs, NCCN-DT, ESAS-r, and HADS.

Measurement instruments

Cancer Distress Scales for Adolescents and Young Adults

CDS-AYA consist of 5 scales: Impact of Cancer (Impact-12 items), Physical (13 items), Emotional (11 items), Cognitive (8 items), and Cancer Worry (5 items).¹⁷ Respondents are asked to indicate how much distress they experienced in the past week for each scale using four response options (none, mild, moderate, and severe). Item scores are summed within each scale and transformed onto a scale of 0–100, with higher scores reflecting more distress. Each scale functions independently (no total score across the five scales). Cut-points for the Impact of Cancer and Emotional scales have the best sensitivity and specificity²³ and were selected for evaluation in this study. The presence of distress is defined in the Impact of Cancer scale as scores ≥24 and the Emotional scale as scores ≥27.²³

Edmonton Symptom Assessment Scale–Revised

The ESAS-r is a 9-item symptom scale with 11 response options ranging from 0 (none) to 10 (severe). The questionnaire asks respondents to circle one number based on how they feel at the time of completion. The ESAS along with the Canadian PCL²⁴ is recommended for distress screening of cancer patients in Canada.⁶ According to implementation guidelines, any score ≥4 on the anxiety or depression symptom item requires action to address psychosocial distress.⁸

Distress Thermometer

The NCCN-DT²⁵ asks patients to rate their level of distress in the past week on a scale from 0 (no distress) to 10 (highest level of distress). In the screening process, a response of ≥4 should prompt clinicians to ask additional questions.²⁶ The NCCN-DT is administered along with the PCL, which was not used in this study.

Hospital Anxiety and Depression Scale

The HADS^27,28 was selected because of its use as a screening tool for cancer patients.²⁹ The HADS has 14 items (7 depression and 7 anxiety items), with 4 response options. The total score ranges from 0 to 21 for each scale, with higher scores indicating more symptoms of depression or anxiety.^27,28 The questionnaire asks participants to answer based on the past week. Recommended categories for identifying anxiety or depression in the subscales are <8 = normal, 8 through 10 = borderline abnormal, and 11 through 21 = abnormal.^27,28 For this study, both the subscales and overall scores were examined. The cut-point for the HADS total score varies in the literature; in this study, a score of >16 was considered to be distressed.³⁰

Analyses

Analyses were completed using SPSS 25.0 (IBM SPSS Statistics, Version 25; IBM Corp); statistical significance was p < 0.05. Only participants with complete clinical information and questionnaire data for the CDS-AYA (Impact and Emotional scales), ESAS-r, imputed HADS, and NCCN-DT were included in analyses. The HADS was imputed using the half-rule, where the mean of the subscale was used if less than 50% of the data were missing. Participants were classified as either distressed or not distressed, according to identified scale cut-points.

Criteria for comparing PROMs for use in performance measurement were adapted from the study by Cella et al. (Table 4).¹⁸ Five of the 8 characteristics presented by Cella et al.²¹ are considered required evidence for a PRO-PM.³¹ For the purposes of this article, four of these five criteria were assessed, including reliability, validity, interpretability, and burden.²¹ The fifth criterion, conceptual and measurement model, refers to the definition and description of concepts and how they are organized into a conceptual framework. This criterion was not evaluated because a detailed literature review would have been required, which was beyond the scope of this study.

Reliability

Internal consistency and TRT reliability were assessed using Cronbach's alpha and the intraclass correlation coefficient (ICC), respectively. ICC values were interpreted as follows: <0.5 poor, 0.5–0.75 moderate, 0.75–0.90 good, and ≥0.9 excellent.³²

Validity

Participant characteristics for development of each scale were collected from the literature. A concept sort was conducted by two researchers to assess content validity for AYA cancer patients. The items in each scale were categorized by major and minor themes. Examples of major themes included emotional, impact of cancer, and physical.¹⁷ These themes were elicited from the cognitive development interview process of CDS-AYA.¹⁷ Minor themes were developed from scale items. AYA-relevant content was based on concepts described in the literature relating to psychosocial care.^1,33,34 The following concepts were considered to be AYA relevant: appearance changes, relationships (peer, family, and romantic), participation (returning to social roles and normalcy), identity, independence, health literacy, fertility, education, and finance. The proportion of AYA-relevant content in each scale was calculated.

To assess construct validity, differences between factors known to be associated with distress, including gender (male or female) and treatment status (on or off treatment), were assessed using a t-test. Based on the literature, it is expected that females³⁵ and those currently on treatment should have poorer outcomes on the measurement scales.^36–40

Interpretability and burden

Literature values for cut-points were reported for interpretability of the scale. The number of items in each scale was also summarized to provide an indication of respondent burden.

Comparison of PROMs

The proportion of patients screening positive for distress was calculated by gender and treatment status for each scale, and differences were assessed using a chi-square test. Variables for stratification were chosen because of known differences in distress by gender³⁵ and treatment status.^36–40

Results

There were 515 participants in the field test of CDS-AYA, of which 86 completed the TRT. From the field test participants, 421 had complete clinical and scale data and were included in analyses. Cancer diagnoses of participants were heterogeneous with most classified as carcinoma, leukemia, lymphoma, or sarcoma. For TRT, 81% participants had complete data for all the scales and were included in reliability analyses. Clinical and demographic statistics are displayed in Table 1.

Table 1.

Demographic and Clinical Statistics

	Field test (n = 515)		Analysis sample (n = 421)
	N	%	n	%
Age at survey
15–19 years	127	24.7	103	24.5
20–29 years	226	43.9	194	46.1
30–39 years	162	31.4	124	29.5
Gender
Male	289	56.1	234	55.6
Female	225	43.7	187	44.4
Treatment status
Active treatment	277	53.8	202	48.0
Treatment complete or not required	229	44.5	219	52.0

A summary of characteristics for evaluation of PROMs for use in performance measurement is provided in Table 2.

Table 2.

Characteristics for Evaluation of Patient-Reported Outcomes for Use in Performance Measurement^a

Criteria (N = 421)		CDS-AYA Emotional (0–100)^b	CDS-AYA Impact of cancer (0–100)^b	ESAS-r (0–100)	NCCN DT (0–10)	HADS overall (0–42)	HADS anxiety (0–21)	HADS depression (0–21)
Reliability—Internal consistency	Cronbach's alpha ≥0.90 individual level	0.936	0.940	0.880	N/A	0.876	0.851	0.793
Reliability—Test retest	ICC (95% confidence intervals)	0.853^*** (0.763, 0.908)	0.855^*** (0.768, 0.910)	0.836^*** (0.736, 0.898)	0.728^*** (0.562, 0.831)	0.910^*** (0.856, 0.944)	0.862^*** (0.769, 0.916)	0.864^*** (0.773, 0.917)
Content validity—How well the scale captures all aspects of a construct for the target population of AYA cancer patients	Age range in the development process (years)	15–39¹⁷	15–39¹⁷	Median age	52–88²⁵	16–65²⁸	16–65²⁸	16–65²⁸
				56—interview
				61—field test²⁴
	Target condition in development	Cancer active treatment and survivorship¹⁷	Cancer active treatment and survivorship¹⁷	Advanced cancer²⁴	Prostate cancer²⁵	General outpatient clinics²⁸	General outpatient clinics²⁸	General outpatient clinics²⁸
	AYA cancer patients involved in content development	Exclusively	Exclusively	AYAs not excluded, mostly older adults involved	Older adults in the original phase of development	AYAs involved not cancer patients	AYAs involved not cancer patients	AYAs involved not cancer patients
	Patient-centered questions (proportion of items relevant to AYAs)	0%	75%	0%	0%	0%	0%	0%
Construct Validity—Support of predefined hypotheses for differences in scores between known groups	Females score poorer than males [mean diff (SE)] Males–females	−6.2 (1.9)^***	−5.1 (1.9)^**	−2.6 (1.4)	−0.55 (0.24)^*	−1.6 (0.67)^*	−1.1 (0.41)^*	−0.52 (0.35)
		−6.2 (1.9)^***	−5.1 (1.9)^**	(p = 0.068)	−0.55 (0.24)^*	−1.6 (0.67)^*	−1.1 (0.41)^*	(p = 0.133)
	Active treatment patients score poorer than patients who have completed treatment	9.0 (1.9)^***	16.6 (1.8)^***	6.9 (1.4)^***	0.67 (0.24)^**	2.7 (0.66)^***	0.84 (0.41)^*	1.8 (0.33)^***
	Active-completed	9.0 (1.9)^***	16.6 (1.8)^***	6.9 (1.4)^***	0.67 (0.24)^**	2.7 (0.66)^***	0.84 (0.41)^*	1.8 (0.33)^***
Interpretability—Are there ways to interpret what scores and differences mean?	Cut-points	≥27	≥24	≥4 on the anxiety or depression symptom item	≥4	>16	>8	>8
Burden—How much time and effort does it take for respondents or administrators to complete?	Number of items	11	12	10	1	14	7	7

Adapted from Table 4 of Cella et al.¹⁸

Slight deviations may occur from values reported in the development article¹⁷ due to the use of a smaller sample size to ensure complete data for all scales.

***

p < 0.001.

p < 0.01.

p < 0.05.

HADS, hospital anxiety and depression scale; SD, standard deviation; SE, standard error; N/A, not applicable; CDS, cancer distress scale; AYAs, adolescents and young adults; ESAS-r, edmonton symptom assessment scale-revised; NCCN, national comprehensive cancer network; DT, distress thermometer.

Reliability

For internal consistency, Cronbach's alpha ranged from 0.79 to 0.94, with only the two scales of CDS-AYA, Emotional and Impact of Cancer, achieving above the recommended >0.90 cut-point. The TRT reliability for the HADS overall was considered excellent (>0.90), with remaining scales having good reliability (>0.75), except for the NCCN-DT (0.73) with an ICC slightly below 0.75.

Validity

A summary of scale content by major and minor themes is displayed in Table 3. The majority of items in the scales fell under the emotional theme with most under the minor theme of anxiety. These included items such as feeling worried (CDS-AYA—Emotional) and sudden feeling of panic (HADS—anxiety). The ESAS-r primarily focused on the physical theme with items associated with the minor theme, symptoms, such as pain, nausea, and tiredness. Only the Impact scale of CDS-AYA and HADS depression subscale had items that were associated with the impact theme such as enjoyment of things (HADS—depression). All scales, except the Impact scale of CDS-AYA, lacked items considered relevant to unique concerns of AYAs. In the Impact scale of CDS-AYA, items relevant to AYAs included the impact of cancer on identity, level of independence, family relationships, keeping up with peer achievements, ability to do normal activities, participating fully in life, making new friends, and romantic relationships. Many of the scales captured more general concepts such as feeling tense or wound up (HADS—anxiety), pain (ESAS-r), and feeling sad (CDS-AYA—Emotional), which are more broadly applicable.

Table 3.

Summary of Scale Content by Major and Minor Themes

HADS-Anx, HADS Anxiety; HADS-Dep, HADS Depression.

For construct validity, all the scales supported the hypothesis that active therapy patients have higher levels of distress than patients not receiving treatment (p ≤ 0.05). More distress was observed in female than male participants for all scales; however, differences were not significant for the ESAS-r (p = 0.068) and HADS depression (p = 0.133) scales.

Interpretability and burden

For interpretability, all scales had defined cut-points. The length of the scales ranged from 1 to 14 items, with all scales viewable on a single page.

Comparison of PROMs

The proportion of AYAs who were on treatment and screened positive for distress ranged from 20.9% to 61.8% and 23.9% to 70.6% in males and females, respectively (Fig. 1). For off-treatment patients, the proportion that screened positive for distress ranged from 9.7% to 27.4% for males and 10.5% to 43.2% for females. The CDS-AYA (Emotional and Impact scales) classified a larger proportion of on-treatment patients as distressed compared with the ESAS-r, NCCN-DT, and HADS for both females and males (p ≤ 0.001; Fig. 1). For off-treatment males and females, the HADS overall score and HADS depression subscale classified a smaller proportion of patients as distressed compared with CDS-AYA (Emotional and Impact scales), ESAS-r, and NCCN-DT (p ≤ 0.006; Fig. 1).

FIG. 1.

Proportion of AYAs who screened positive for distress by instrument. AYAs, adolescents and young adults.

Discussion

Overall, the majority of scales examined in this study met the Cella et al.²¹ characteristics for selection of a PROM for PM. The scales had good internal consistency and reliability and were generally brief and interpretable. The greatest difference among the scales was related to content validity. The CDS-AYA scales were the only metrics explicitly designed for the AYA cancer population,¹⁷ using patient interviews to ensure content validity.

AYA-relevant psychosocial concerns^1,32,33 were not predominant in the scales examined. Many areas of distress important to AYAs with cancer were not represented, such as distress related to cancer worry, cognitive problems, employment, and education.¹⁷ For the three major themes (emotional, impact, and physical) observed in this article, most of the scale's items were limited to just one major theme. By not capturing all areas of distress experienced by patients, it is possible that some patients with distress may be missed in screening. Clinically, this gap could be overcome using checklists such as those included in the ESAS-r and NCCN-DT. However, checklists have important limitations and do not have the same psychometric properties as scales. An important aspect of a scale is that it can be used to examine change over time in varying degrees of distress rather than just the presence or absence of symptoms of distress. CDS-AYA have additional scales¹⁷ covering other areas of distress that are important to AYAs, although administration of all the scales may be burdensome for screening purposes.

Many of the items in the scales described concepts such as worry, fatigue, and pain that could be considered more broadly applicable in the cancer population. Most issues in the psychosocial care of AYAs with cancer relate to their stage of development. During the AYA period, individuals are reaching key milestones, including developing an identity, self-image, independence, increased involvement in relationships and dating, and important decisions regarding education, career, and family.²

The Impact of cancer scale of CDS-AYA was the only scale that specifically captured some of these key concepts. This included items on keeping up with achievements of peers and normalcy. These are both important issues to AYAs with cancer who are missing out on the experiences of their peers such as going to the prom, attending higher education, and starting careers and families.^1,32,33 Aspects of identity and level of independence captured by the Impact scale are also very important to AYAs who become more reliant on family during their illness and are missing out on key experiences that help them form their self-identity (going away to university).^1,32,33 The Impact scale of CDS-AYA also has relationship items (e.g., impact of cancer on romantic relationships). This is important to AYAs because cancer can interfere or change relationships with both family and peers. Relationship concerns can include making friends, protecting parents, and developing romantic relationships (disclosure of cancer or fertility).^2,10,41 Having specific content that captures the unique concerns of AYAs with cancer may help to better measure distress in this population.

There was substantial variation in the prevalence of distress identified when using different PROMs in this study. Literature reports of distress in AYAs with cancer have been previously described and are also highly variable.⁴² Enskar and von Essen³⁸ reported that more than 50% of AYA cancer patients and survivors reported psychosocial distress in 12 of 13 aspects that were measured. Recklitis et al.¹⁶ found that 49.4% of AYA cancer survivors had moderate or high distress levels on the NCCN-DT. Clinical levels of distress in AYA cancer, where a patient meets the criteria for diagnosis of a mental disorder such as post-traumatic stress disorder, have also been found to be variable, ranging from 5.4% to 56.5% in a 2013 review.⁴³ The variations between scales in this study are likely related to differences in the distress construct measured. Given the variability in estimates produced by different scales, it is unlikely that distress as a performance measure would be comparable across institutions, provinces, or countries using different PROMs. In the selection of a scale for a distress PRO-PM, it is important to consider the patient-centered nature of the PROM to ensure that it captures the aspects of distress most important to the target population.

A limitation of this study is that clinical interviews were not conducted to definitely diagnose distress, therefore scales could not be compared with gold standard diagnostics. Further work is needed to compare screening scales used in the AYA cancer population with gold standard interviews to determine how accurately these instruments classify distress in this age group. In addition, examining changes in distress is an important aspect for a performance measure and the inability to assess responsiveness is a limitation. This study also did not include the PCL, a component of the NCCN-DT. A protocol for use of the NCCN-DT with a modified PCL in the AYA cancer population has been developed and is currently being validated in Australia for use in distress screening.⁴⁴ However, the PCL does not contribute to the scoring of the NCCN-DT.

Given that the Impact scale content of CDS-AYA is highly relevant to AYA cancer patients, this scale met the most criteria for selection of a PROM for use in performance measurement. However, this scale has limitations because it does not capture aspects of distress outside of the impact of cancer theme. Domains such as physical and emotional distress are also highly relevant to AYAs with cancer, and the effect on screening within a single theme is unknown. Further work could look at the application of all the CDS-AYA in distress screening through the use of computer adaptive testing (CAT). A recent study of a PROM for patients with a cleft lip and/or palate was able to use CAT to reduce the 12-scale 110-item CLEFT-Q to a mean of 43 items.⁴⁵ Using CAT with CDS-AYAs may allow administration of all five scales using fewer items to greatly reduce respondent burden. Development of benchmarks for CDS-AYA is also necessary before it can be used in performance measurement.

Footnotes

Acknowledgments

This study was conducted with support from the C17 Council and funded by the Childhood Cancer Canada Foundation and the Kids With Cancer Society. The authors would also like to acknowledge Abha Gupta, Dilshan Pieris, Kate Wang, Seline Tam, Nivetha Ramachandran, Colleen Fitzgerald, Sarah McKillop, and Linda Churcher for their contributions. The authors would also like to acknowledge Pandora Patterson, Michael Osborn, and Fiona McDonald—developers of the Australian DT AYA on-treatment and survivorship tools.

Author Disclosure Statement

The authors have no conflicts of interest.

Funding Information

Support provided by the C17 Council through funding from Childhood Cancer Canada Foundation and the Kids with Cancer Society.

References

D'agostino

, Penney

, Zebrack

. Providing developmentally appropriate psychosocial care to adolescent and young adult cancer survivors. Cancer. 2011; 117(10 Suppl):2329–34.

Zebrack

. Psychological, social, and behavioral issues for young adults with cancer. Cancer. 2011; 117(10 Suppl):2289–94.

Fernandez

, Fraser

, Freeman

, et al. Principles and recommendations for the provision of healthcare in Canada to adolescent and young adult–aged cancer patients and survivors. JAYAO. 2011; 1(1):53–9.

Adolescent & Young Adults with Cancer: Technical Report. Toronto (ON): Canadian Partnership Against Cancer; April 2017. Accessed June 21, 2019 from www.systemperformance.ca/report/adolescents-young-adults-cancer

Cella

, Hahn

, Jensen

, et al. Patient-Reported Outcomes in Performance Measurement. Research Triangle Park (NC): RTI Press; 2015 Sep. Table 1, Definitions and key concepts for patient-reported outcomes and measures. Accessed June 21, 2019 from: www.ncbi.nlm.nih.gov/books/NBK424379/table/t1

Bultz

, Groff

, Fitch

, et al. Implementing screening for distress, the 6th vital sign: a Canadian strategy for changing practice. Psychooncology. 2011; 20(5):463–9.

Howell

, Olsen

. Distress—the 6th vital sign. Curr Oncol. 2011; 18(5):208.

Howell

, Hack

, Green

, Fitch

. Cancer distress screening data: translating knowledge into clinical action for a quality response. Palliat Support Care. 2014; 12(1):39–51.

Bellizzi

, Smith

, Schmidt

, et al. Positive and negative psychosocial impact of being diagnosed with cancer as an adolescent or young adult. Cancer. 2012; 118(20):5155–62.

10.

Wakefield

, Patterson

, McDonald

, et al. Assessment of psychosocial outcomes in adolescents and young adults with cancer: a systematic review of available instruments. Clin Oncol Adolesc Young Adults. 2013; 2013(3):13–27.

11.

Palmer

, Patterson

, Thompson

. A national approach to improving adolescent and young adult (AYA) oncology psychosocial care: the development of AYA-specific psychosocial assessment and care tools. Palliat Support Care. 2014; 12(3):183–8.

12.

Selby

, Cascella

, Gardiner

, et al. A single set of numerical cutpoints to define moderate and severe symptoms for the Edmonton Symptom Assessment System. J Pain Symptom Manage. 2010; 39(2):241–9.

13.

Hughes

, Holland

, Zanino

, et al. Prevalence and intensity of pain and other physical and psychological symptoms in adolescents and young adults diagnosed with cancer on referral to a palliative care service. JAYAO. 2015; 4(2):70–5.

14.

Donovan

, Grassi

, McGinty

, Jacobsen

. Validation of the distress thermometer worldwide: state of the science. Psychooncology. 2014; 23(3):241–50.

15.

Mitchell

. Pooled results from 38 analyses of the accuracy of distress thermometer and other ultra-short methods of detecting cancer-related mood disorders. J Clin Oncol. 2007; 25(29):4670–81.

16.

Recklitis

, Blackmon

, Chang

. Screening young adult cancer survivors for distress with the distress thermometer: comparisons with a structured clinical diagnostic interview. Cancer. 2016; 122(2):296–303.

17.

Tsangaris

, D'Agostino

, Rae

, et al. Development and psychometric evaluation of the cancer distress scales for adolescent and young adults. JAYAO. 2019 [Epub ahead of print]; DOI: 10.1089/jayao.2019.0005.

18.

Patient Report Outcomes (PROs) in Performance Measurement Washington, DC: National Quality Forum (NQF); 2013. Accessed June 21, 2019 from: www.qualityforum.org

19.

Basch

, Spertus

, Dudley

, et al. Methods for developing patient-reported outcome-based performance measures (PRO-PMs). Value Health. 2015; 18(4):493–504.

20.

Basch

, Abernethy

, Reeve

. Assuring the patient-centeredness of patient-reported outcomes: content validity in medical product development and comparative effectiveness research. Value Health. 2011; 14(8):965–6.

21.

Cella

, Hahn

, Jensen

, et al. Methodological issues in the selection, administration and use of patient-reported outcomes in performance measurement in health care settings.Washington, DC: National Quality Forum (NQF); 2012. From: www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2%7C Accessed July 8, 2019.

22.

National Institutes of

Health

, National Cancer

Institute

, LiveStrong Young Adult

Alliance

. Closing the gap: research and care imperatives for adolescents and young adults with cancer: a report of the Adolescent and Young Adult Oncology Progress Review Group. Accessed July 8, 2019 from: www.cancer.gov/types/aya/research/ayao-august-2006.pdf

23.

Rae

, Klassen

, Tsangaris

, et al. Distress screening in adolescents and young adults (AYA) with cancer: development of cut-points for the cancer distress scales (CDS)–AYA. JAYAO.2019; [Epub ahead of print]; DOI: 10.1089/jayao.2019.0032.

24.

Watanabe

, Nekolaichuk

, Beaumont

. The Edmonton Symptom Assessment System, a proposed tool for distress screening in cancer patients: development and refinement. Psychooncology. 2012; 21(9):977–85.

25.

Roth

, Kornblith

, Batel-Copel

, et al. Rapid screening for psychologic distress in men with prostate carcinoma: a pilot study. Cancer. 1998; 82(10):1904–8.

26.

National Comprehensive Cancer Network (NCCN). NCCN Clinical Practice Guidelines in Oncology: Distress Management. Fort Washington, PA: NCCN; 2018. Version 2.2018.

27.

Snaith

. The hospital anxiety and depression scale. Health Qual Life Out. 2003; 1:29.

28.

Zigmond

, Snaith

. The hospital anxiety and depression scale. Acta Psychiat Scand. 1983; 67(6):361–70.

29.

Carroll

, Kathol

, Noyes Jr

, et al. Screening for depression and anxiety in cancer patients using the Hospital Anxiety and Depression Scale. Gen Hosp Psychiat. 1993; 15(2):69–74.

30.

Vodermaier

, Linden

, Siu

. Screening for emotional distress in cancer patients: a systematic review of assessment instruments. J Natl Cancer Inst, 2009; 101(21):1464–88.

31.

Deutsch

, Smith

, Gage

, et al. Pateient-Reported Outcomes in Performance Measurement. commissioned paper on PRO-based performance measures for healthcare accountable entities. Washington, DC: National Quality Forum;. 2012.

32.

Koo

, Li

. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016; 15(2):155–63.

33.

Zebrack

, Isaacson

. Psychosocial care of adolescent and young adult patients with cancer and survivors. J Clin Oncol. 2012; 30(11):1221–6.

34.

Evan

, Zeltzer

. Psychosocial dimensions of cancer in adolescents and young adults. Cancer. 2006; 107(S7):1663–71.

35.

Strong

, Waters

, Hibberd

, et al. Emotional distress in cancer patients: the Edinburgh Cancer Centre symptom study. Br J Cancer. 2007; 96(6):868–74.

36.

Hedstrom

, Ljungman

, von Essen

. Perceptions of distress among adolescents recently diagnosed with cancer. J Pediatr Hematol Oncol. 2005; 27(1):15–22.

37.

, Robison

, Jenney

, et al. Assessment of health-related quality of life of adolescent cancer patients using the Minneapolis-Manchester Quality of Life Adolescent Questionnaire. Pediatr Blood Cancer. 2007; 48(7):678–86.

38.

Enskar

, von Essen

. Prevalence of aspects of distress, coping, support and care among adolescents and young adults undergoing and being off cancer treatment. Eur J Oncol Nurs. 2007; 11(5):400–8.

39.

von Essen

, Enskar

, Kreuger

, et al. Self-esteem, depression and anxiety among Swedish children and adolescents on and off cancer treatment. Acta Paediatr. 2000; 89(2):229–36.

40.

Kwak

, Zebrack

, Meeske

, et al. Trajectories of psychological distress in adolescent and young adult patients with cancer: a 1-year longitudinal study. J Clin Oncol. 2013; 31(17):2160–6.

41.

D'Agostino

, Edelstein

. Psychosocial challenges and resource needs of young adult cancer survivors: implications for program development. J Psychosoc Oncol, 2013; 31(6):585–600.

42.

Zebrack

, Corbett

, Embry

, et al. Psychological distress and unsatisfied need for psychosocial support in adolescent and young adult cancer patients during the first year following diagnosis. Psychooncology. 2014; 23(11):1267–75.

43.

Sansom-Daly

, Wakefield

. Distress and adjustment among adolescents and young adults with cancer: an empirical and conceptual review. Transl Pediatr. 2013; 2(4):167–97.

44.

Patterson

, McDonald

, Anazodo

, et al. Validation of the distress thermometer for use among adolescents and young adults with cancer in Australia: a multicenter study protocol. Clin Oncol Adolesc Young Adults. 2015; 2015(5):51–62.

45.

Harrison

, Geerards

, Ottenhof

, et al. Computerised adaptive testing accurately predicts CLEFT-Q scores by selecting fewer, more patient-focused questions. J Plast Reconstr Aesthet Surg. 2019. [Epub ahead of print]; DOI: 10.1016/j.bjps.2019.05.039.