Validation of a Health Literacy Measure for Adolescents and Young Adults Diagnosed with Cancer

Abstract

Health literacy can influence long-term health outcomes. This study aimed to validate an adapted version of the Functional, Communicative and Critical Health Literacy measure for adolescent and young adult (AYA) cancer patients and survivors (N = 105; age 12–24 years). Exploratory factor analysis was used to validate the measure, and indicated that a slightly modified item structure better fit the results. Furthermore, item response theory analysis highlighted location and discrimination parameter differences among items. Acceptability of the measure was high. This is the first validation of a health literacy measure among AYAs with an illness such as cancer.

Adolescence and young adulthood is a period of change where individuals take greater responsibility for their own health¹ and become more engaged with the healthcare system. Health literacy (HL) during adolescence is particularly important, as it can have a lifelong preventative impact,² with low levels of HL associated with many poor health behaviors and outcomes.^3,4 Among adolescents, high HL is associated with better self-reported health status and increased levels of health-promoting behaviors.² Conversely, adolescents who do not understand basic health concepts and find health information difficult to understand have less interest in learning about health and are less likely to adhere to health-promoting messages.⁵

These issues are compounded for the approximately 900 adolescents and young adults (AYAs) between 15 and 25 years of age who are diagnosed with cancer each year in Australia.⁶ They typically receive large amounts of complex information related to their diagnosis and treatment that they must understand and assimilate into their already complex lives in order to navigate their care and manage their health into the survivorship phase. Though further investigation of the impact of HL on AYA cancer patient outcomes is needed, research has shown that aspects of HL such as low levels of engagement in health information seeking are associated with negative long-term health behaviors such as smoking among childhood and young adult cancer survivors.⁷ Given that the majority of AYAs survive cancer,⁶ it is clearly essential that they develop appropriate skills to manage both their immediate diagnosis and their long-term healthcare.⁸ Health promotion and risk behavior education during adolescence can establish and consolidate these skills.² However, the HL needs and abilities of this age group must be understood and accounted for in order for education to be effective.⁹ Measuring these needs and abilities with validated age- and illness-appropriate instruments is thus crucial in order to understand fully the impact of HL on outcomes for AYAs with cancer and appropriately target health-promotion messages and education.

Recent reviews have highlighted many different approaches to the concept of HL and how it may be quantified.^10,11 Furthermore, instruments differ widely in their content and underlying constructs, and many are of questionable validity and reliability.^10,11 Traditionally, HL measures have focused on functional components such as basic abilities to read and write, and familiarity with health-related terms¹² (e.g., the Rapid Estimate of Adult Literacy in Medicine [REALM]¹³ and the Test of Functional Health Literacy in Adults [TOFHLA]¹⁴). Recently, instruments have been developed that measure a broader concept of HL,^15–18 including higher cognitive abilities. Such measures are based on a three-factor model described by Nutbeam, which includes an individual's ability to derive meaning from health information (communicative literacy) and analyze information sources (critical literacy) in addition to functional literacy.^9,12,19 New instruments include the Functional, Communicative and Critical HL measure (FCC-HL),¹⁶ the Health Literacy Questionnaire²⁰ (an 8-item measure developed for young adults¹⁵), and Massey et al.'s multidimensional measure of adolescent HL.¹⁷ Recently, HL measures have also been developed or adapted for use with AYAs.^2,3,15,18,21 While some address only functional HL,^3,21 several also consider critical and communicative HL.^15,17–19

While there are several HL measures for either adolescents or young adults, no measure has been validated across this whole AYA age range, or with AYAs with an illness such as cancer.¹¹ The purpose of the current study was to select and adapt a measure of HL for use with AYAs diagnosed with cancer and to conduct an initial validation of the new scale. The study also sought to explore the ease of understanding of the measure and whether it caused any distress for participants.

Methods

Measure selection

In selecting a measure to adapt and validate with AYAs diagnosed with cancer, the authors considered both conceptual and structural criteria. Measurement of a broad, multidimensional concept of HL as described by Nutbeam¹⁹ was chosen over functional HL alone. Additionally, a short instrument with items relevant for the target population that was easy to administer and score was considered desirable. Existing measures developed for AYAs had a different focus, such as interaction and satisfaction with healthcare providers,¹⁷ were difficult to administer and score,¹⁸ contained items irrelevant to the target population,¹⁵ or were restricted to functional HL.^3,21 Also, many adult scales had limitations in terms of administration difficulty and a narrow HL conceptualization.

The Functional, Communicative, and Critical Health Literacy—AYA Cancer measure development

The 14-item FCC-HL¹⁶ was originally validated with Japanese adults with diabetes. Items load across three HL domains: functional (items 1–5), communicative (items 6–10), and critical (items 11–14), reflecting a multidimensional conceptualization of HL. Additionally, it met length and ease of administration and scoring criteria. It was thus chosen as the basis for adaptation.

The Functional, Communicative, and Critical Health Literacy—AYA Cancer (FCCHL-AYAC) was adapted from the reported version of the FCC-HL¹⁶ to be suitable for AYAs diagnosed with cancer. Though the FCC-HL was developed and validated in Japanese, the scale was originally published in English and so no translation was necessary for the present study. A previous exploration of the FCC-HL in a Dutch population expressed some reservations about the degree of conceptual abstraction skill needed to understand and respond to the scale.²² To minimize the potential that this would impact on the ability of individuals with lower literacy to respond to the FCCHL-AYAC, the language of each item was revised to use simplified, first-person prose and an active voice to improve the clarity of the instrument, while maintaining item content. Phrases that could be difficult for AYAs to understand were modified (e.g., using the term “pull out” instead of “extracted” and “lots of people and places” instead of “various sources”). References to “diabetes” were also changed to “cancer.” The Flesch–Kincaid Readability Ease²³ score of the FCC-HL-AYAC was 92.9 (School Grade 2—age 7–8 years). The revised items are presented in Table 2.

Participants

Participants were recruited from CanTeen,^a the Australian Organization for Young People Living with Cancer. Eligible participants were current CanTeen Members, aged 12–24 years at the time of participation, who had any cancer diagnosis and (a) were being treated for cancer, or (b) had completed treatment. Potential participants were identified from the CanTeen database according to these eligibility criteria.

Four hundred and sixteen members were either sent a link to the survey via email (n = 346) or were mailed the survey if an email address was not available (n = 70). Two reminders were sent at 3–4-week intervals following initial contact. The survey contained demographic and cancer-related questions, the FCCHL-AYAC, and measure acceptability questions. Those who were mailed questionnaires were provided with postage-paid reply envelopes. Capacity to consent was assumed for participants aged 18 years and over, with consent implied by return/completion of the survey. Parent/guardian consent was required for participants aged 12–17 years. The study received ethical clearance from the CanTeen Ethics and Young People Committee.

Analysis

Exploratory factor analysis

Exploratory factor analysis (EFA) was conducted using SPSS v21 (IBM Corp., Armonk, NY) to establish the factor structure of the FCCHL-AYAC (principal axis factoring with promax rotation). A scree plot was examined to determine the number of factors to extract. Inter-factor correlations and the internal consistency of each factor (Cronbach's α) were examined.

Item response theory analysis

In item response theory (IRT), the probability of an individual's response on an item is determined by its value on an underlying (latent) variable and properties of the item. Samejima's two-parameter graded response model was employed,²⁴ in which observed responses to polytomous items (i.e., items with more than two response options) are assumed to reflect a latent variable (in the present case, each literacy domain). The probability of responding with a higher response option increases as the level of the latent variable increases, according to a logistic function. The two item parameters estimated for each item in this model are (a) location along the continuum of values of the latent variable, and (b) discrimination, or the ability to differentiate between those scoring high and low on each latent variable.

IRT analysis was conducted using the graded response model [grm()] function of the latent trait model [ltm()] package²⁵ in R, and was performed separately for the factors identified using EFA. Location and discrimination parameters of each item were examined. The fit of the two-parameter model was assessed by comparing this model to the one-parameter model (where the discrimination parameter is held constant between items) using the likelihood ratio test, where p < 0.01 indicates significantly better fit of the two-parameter model.

Measure acceptability

After completing the FCCHL-AYAC measure, participants were asked “I found the questions above easy to understand” and “I felt distressed answering the questions above,” with a 4-point Likert response scale (1 = “strongly disagree,” 2 = “disagree,” 3 = “agree,” and 4 = “strongly agree”).

Results

Participants

One hundred and five AYAs completed the questionnaire (age at participation M = 18.7 years, standard deviation [SD] = 3.3; age range 12–24 years, males = 35 [33.3%]; response rate = 25.2%). The mean time since their cancer diagnosis was 42.3 months (SD = 40.9 months, range 1 month–20 years^b). Further details are given in Table 1.

Table 1.

Participant Demographic and Cancer Information (N = 105)

	n (%)
Current or highest level of education
Year 7 or 8	12 (11)
Year 9 or 10	14 (13)
Year 11 or 12	48 (46)
Vocational training	9 (9)
University	22 (21)
Country of birth (missing: n = 1)
Australia	95 (91)
Other (n = 1 for each): Canada, Indonesia, New Zealand, Pakistan, Philippines, South Africa, Switzerland, United Kingdom	9 (9)
Mother's country of birth (missing: n = 2)
Australia	78 (74)
United Kingdom	9 (9)
New Zealand	4 (4)
Indonesia	2 (2)
Philippines	2 (2)
Other (n = 1 for each): Canada, China, Germany, Italy, Pakistan, Singapore, United States	8 (8)
Father's country of birth (missing: n = 3)
Australia	75 (71)
United Kingdom	9 (9)
New Zealand	4 (4)
Germany	2 (2)
Philippines	2 (2)
South Africa	2 (2)
Sri Lanka	2 (2)
Other (n = 1 for each): Bermuda, China, India, Italy, Pakistan, Poland	6 (6)
Language spoken at home
English only	99 (94)
English and another language(s): Other language: Afrikaans (n = 1), Indonesian (n = 1), Tagalog (n = 2), Urdu (n = 1), Zulu and French together (n = 1)	6 (6)
Cancer diagnosis^a
Leukemia	26 (25)
Hodgkin lymphoma	25 (24)
Bone	17 (16)
Brain	13 (12)
Non-Hodgkin lymphoma	6 (6)
Reproductive	6 (6)
Spinal	2 (2)
Other: breast, head and neck, kidney, liver, lung, other lymphoma, pancreatic, soft tissue, thyroid, unknown primary	22 (21)
Treatment stage
Recently diagnosed	1 (1)
On active treatment	14 (13)
On maintenance treatment	14 (13)
Finished treatment	70 (67)
No further treatment/palliative care	2 (2)
Unsure	4 (4)

More than one type of cancer possible.

HL measure

Descriptive results

Means and standard deviations for each item of the FCCHL-AYAC are given in Table 2. Independent samples t-tests indicated that there were no significant differences (at p = 0.05) in FCCHL-AYAC total scores according to sex, whether the measure was completed online or on paper, or whether the participant was on or off treatment.

Table 2.

Results of Exploratory Factor and Item Response Theory Analyses

			Exploratory factor analysis			Item response theory analysis
Item #	Item	Mean (SD; missing n)^a	Factor 1	Factor 2	Factor 3	Threshold 1	Threshold 2	Threshold 3	Discrimination
When reading things about cancer…
1	The printing is too small to read	1.90 (0.82; 1)	0.66	0.01	0.08	−0.46	0.87	2.86	1.89
2	There are words that I do not know	2.65 (0.94; 1)	0.81	0.20	−0.05	−1.40	−0.26	0.93	2.81
3	The content is too difficult	2.34 (0.88; 1)	0.94	0.17	0.10	−1.02	0.20	1.36	3.99
4	I need a long time to read and understand it	2.47 (0.94; 2)	0.78	−0.12	0.10	−1.10	−0.02	1.24	2.79
5	I need someone to help me read it	1.75 (0.92, 2)	0.64	−0.07	−0.09	0.06	0.99	2.23	1.85
Since I was diagnosed with cancer…
6	I get my information about cancer from lots of people and places	3.13 (0.84, 2)	0.13	0.41	0.04	−3.72	−1.55	0.62	0.97
7	I can pull out the information I want	3.17 (0.84, 1)	−0.29	0.22	0.31	—	—	—	—
8	I understand the information I've found	3.38 (0.66, 1)	–0.50	0.30	0.16	−0.14	2.56	4.55	1.14
9	I have expressed my thoughts about my cancer with someone	3.08 (0.98, 1)	0.04	0.81	−0.12	−1.63	−0.71	0.19	2.57
10	I have used the information I've found	3.02 (0.77, 2)	−0.08	0.49	0.14	−2.33	−1.17	0.91	1.80
11	I think about whether what I read is relevant to me	3.30 (0.81, 3)	−0.02	0.07	0.64	−2.57	−1.23	0.08	2.01
12	I think about where the information comes from and if I trust that source	3.07 (0.97, 2)	0.03	−0.25	1.06	−1.28	−0.68	0.28	4.52
13	I check information to see if it is correct	2.75 (0.99, 2)	0.03	0.05	0.77	−1.26	−0.16	0.74	3.06
14	I collect information to help make decisions about my cancer	2.64 (0.93, 2)	0.10	0.23	0.57	−1.48	−0.12	1.15	2.11

Note. Items in bold are significant at p = 0.05.

Factor 1: Functional Literacy; Factor 2: Communicative Literacy; Factor 3: Critical Literacy.

Possible range of scores for all items is 1–4. For all items, the full range was used. The first five items are reverse scored.

SD, standard deviation.

Exploratory factor analysis

The scree plot suggested the extraction of three factors (53.1% variance explained). Factor loadings for the pattern matrix are shown in Table 2. In summary, the present data replicated the factor structure found by Ishikawa et al.¹⁶ with two exceptions: (1) item 8 loaded on functional literacy, which is unsurprising, as its content (whether respondents understood the information) has more in common with functional than communicative literacy, and (2) item 7 loaded most strongly (albeit weakly) on critical literacy, which again is unsurprising, as it involves respondents understanding what information is desired. Cronbach's alphas and inter-factor correlations are shown in Table 3.

Table 3.

Cronbach's Alpha for Each Factor (FCC-HL and FCCHL-AYAC) and Inter-Factor Correlations

	FCC-HL α^a	FCCHL-AYAC α	Factor 1—Functional	Factor 2—Communicative
Factor 1—Functional (items 1–5 and 8)	0.84	0.73^b
Factor 2—Communicative (items 6, 9 and 10)	0.77	0.63	−0.13
Factor 3—Critical (items 11–14)	0.65	0.85	−0.05	0.40

Cronbach's α for original measure.¹²

With item 8 omitted, α = 0.87.

FCC-HL, Functional, Communicative and Critical Health Literacy measure; FCCHL-AYAC, Functional, Communicative and Critical Health Literacy Adolescent and Young Adult Cancer measure.

IRT analysis

The threshold and discrimination parameters for each item in the IRT analysis are shown in Table 2. Based on the EFA results, IRT analysis was conducted separately for three factors: (1) items 1–5 and 8; (2) items 6, 9, and 10; and (3) items 11–14. The two-parameter model had significantly better fit than the one-parameter model for factors 1 (functional) and 3 (critical), but not 2 (communicative), as reflected by the discrimination parameters in Table 2, which exhibit a larger range of values for factors 1 and 3 than for factor 2.

Measure acceptability

For the item “I found the questions easy to understand,” 58 (55%) strongly agreed, 43 (41%) agreed, and 4 (4%) disagreed (96% positive responses). For the item “I felt distressed answering the questions,” 66 (63%) strongly disagreed, 34 (32%) disagreed, 3 (3%) agreed, 1 (1%) strongly agreed, and there was one missing response (95% positive responses).

Measure revision

The factor structure of the measure was revised such that items 1–5 and 8 loaded onto the functional literacy domain; items 6, 9, and 10 loaded onto the communicative literacy domain subscale; and items 11–14 loaded onto the critical literacy domain. While moving item 8—“I understand the information I've found”—into the functional domain did not change the item, the scoring procedure is now different for this domain, as item 8 is not reverse scored like the other items in the domain. As item 7 did not significantly load onto any subscale, it was removed, leaving a final 13-item measure. The full text of the revised measure is included in Appendix 1 (online only).

Discussion

The factor and IRT analyses found that the structure of the FCCHL-AYAC measure was similar to the original FCC-HL.¹⁶ One item from the communicative domain was found to load better onto the functional domain, and another item from the communicative domain did not load clearly on any factor. This is consistent with previous research validating a Dutch translation of the FCC-HL, which found that while the functional and critical domains had good internal consistency, the communicative domain did not.²² Compared to the FCC-HL, Cronbach's alpha for the FCCHL-AYAC was (1) lower for functional literacy, perhaps attributable to the inclusion in this factor of item 8 (“Since I was diagnosed with cancer … I understand the information I've found”) in the FCCHL-AYAC (α with this item excluded increased to 0.87 from 0.84); (2) lower for communicative literacy, possibly because this factor contained fewer items in the FCCHL-AYAC; and (3) higher for critical literacy, possibly because the AYA participants tended to score consistently higher on these items than those in Ishikawa et al.'s study.¹⁶ IRT analysis facilitates the assessment of how precisely items within a scale measure a given construct. The current analysis indicated that item 3 (“The content is too difficult”), item 9 (“I have expressed my thoughts about my cancer with someone”), and item 12 (“I think about where the information comes from and if I trust that source”) were the most precise indicators of functional, communicative, and critical HL, respectively.

The measure had very high acceptability among AYAs diagnosed with cancer; the overwhelming majority found it easy to complete and not distressing. This contrasts with the Dutch study, which reported difficulties with understanding.²² The authors of that study suggested this was due to items being too abstract, rather than translation issues. It is likely that the small changes made to items to make them clearer to AYAs in the present study impacted on the understandability of the items.

Several HL measures have been developed or modified for adolescents or young adults that assess different aspects of HL. The FCCHL-AYAC will be a useful addition to this suite of HL measures for AYAs affected by cancer, as it is both minimally time consuming and targets a broad conceptualization of HL; functional, communicative, and critical elements. Alternative measures such as the TOFHLA or REALM-Teen^3,21 are similarly minimally time-consuming, making them suitable for screening or assessment of large populations in clinical or research contexts. However, both scales target functional HL alone. Likewise, measures involving interpretation of paragraphs of health information, such as Wu et al.'s performance-based measure,¹⁸ may be better suited to contexts where clinicians need to gain a nuanced understanding of an AYA's HL limitations, as they target both understanding and evaluation of information but require a longer administration time. The FCCHL-AYAC may provide the benefits of this nuanced understanding of AYA HL abilities paired with a short administration time.

Limitations and future directions

While the sample was representative in terms of cancer diagnoses, most participants were Australian born to Australian-born parents, and only spoke English at home, which does not reflect the cultural and linguistic diversity of the Australian population. Additionally, one-third of the sample was male, less than one-third were on active treatment, and the response rate for the study was 25%. While this response rate could be considered low, recruitment has been noted as a substantial challenge in this population, and such response rates are not uncommon.^26,27 It is possible that young people who were underrepresented in this sample may find the measure more difficult to complete or less reflective of the HL challenges they face. The sample was also self-selected and drawn from a support organization with voluntary membership. However, given how closely the factor structure previously found for the FCC-HL was mirrored in the present study, it seems unlikely that the factor structure or relevance of the FCCHL-AYA would be substantially different with a different sample. It would nonetheless be beneficial to validate the measure further with a more diverse and representative sample. Furthermore, although there is no widely accepted guidance on adequacy of sample size for IRT, future research should attempt to replicate the current findings with a larger sample in order to establish their stability. Such studies should be part of wider efforts to validate the FCCHL-AYAC scale and current factor structure including comparison of the scale with other measures to establish construct validity and exploration of the test–retest reliability.

This new scale does not tap interactions with healthcare providers, or patient rights and responsibilities as in other AYA measures such as Massey et al.'s multidimensional measure of adolescent HL.¹⁷ Items such as this could be considered for inclusion in a more comprehensive HL measure for AYAs with illnesses such as cancer in the future. Additionally, Manganello¹ has developed a framework for adolescent HL that considers the impact of a range of variables, and the development or modification of future HL measures could benefit from reflection on this model.

Conclusion

The importance of HL is becoming increasingly apparent, as is the complexity of this concept. There is a need to continue to develop robust measures for use with AYAs, particularly those with illnesses such as cancer. This study has found that the FCCHL-AYAC adapted from the FCC-HL is suitable for use with Australian AYAs diagnosed with cancer.

Footnotes

Acknowledgments

The authors acknowledge the young people who gave their time to participate in the study. We also acknowledge Adam Walczak and Kimberley Allison's assistance with manuscript preparation and thank Dr. Hirono Ishikawa for her permission to use this measure in a new population.

Author Disclosure Statement

No competing financial interests exist.

Appendix 1: Functional,Communicative,and Critical Health Literacy—AYA Cancer measure

Scoring:

To obtain the functional literacy subscale score, reverse score the participant's responses for items 1–5 and add these values to the participant's response for item 7.

To obtain the communicative literacy subscale score, sum the participant's responses for items 6, 8, and 9.

To obtain the critical literacy subscale score, sum the participant's responses for items 10–13.

To obtain the total FCCHL-AYAC score, sum the functional, communicative, and critical literacy subscale scores.

References

Manganello

. Health literacy and adolescents: a framework and agenda for future research. Health Educ Res. 2008; 23(5):840–7.

Chang

L-C

. Health literacy, self-reported status and health promoting behaviours for adolescents in Taiwan. J Clin Nurs. 2010; 20(1–2):190–6.

Chisolm

, Buchanan

. Measuring adolescent functional health literacy: a pilot validation of the test of functional health literacy in adults. J Adolesc Health. 2007; 41(3):312–14.

Berkman

, Sheridan

, Donahue

, et al. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011; 155(2):97–107.

Brown

, Teufel

, Birch

. Early adolescents perceptions of health and health literacy. J Sch Health. 2006; 77(1):7–15.

Australian Institute of Health and Welfare. Cancer in adolescents and young adults in Australia. Cancer series No. 62. Cat. No. CAN 59. 62 ed. Canberra: AIHW; 2011.

Nagler

, Puleo

, Sprunck-Harrild

, et al. Health media use among childhood and young adult cancer survivors who smoke. Supp Care Cancer. 2014; 22(9):2497–507.

DeRosa

, Kazak

, Doshi

, et al. Development and validation of the health competence beliefs inventory in young adults with and without a history of childhood cancer. Ann Behav Med. 2011; 41(1):48–58.

Nutbeam

. The evolving concept of health literacy. Social Sci Med. 2008; 67(12):2072–8.

10.

Jordan

, Osborne

, Buchbinder

. Critical appraisal of health literacy indices revealed variable underlying constructs, narrow content and psychometric weaknesses. J Clin Epidemiol. 2011; 64(4):366–79.

11.

Perry

. Health literacy in adolescents: an integrative review. J Spec Pediatr Nurs. 2014; 19(3):210–8.

12.

Berkman

, Davis

, McCCormack

. Health literacy: what is it?. J Health Commun. 2010; 15(S2):9–19.

13.

Davis

, Long

, Jackson

, et al. Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med. 1993; 25(6):391–5.

14.

Parker

, Baker

, Williams

, Nurss

. The test of functional health literacy in adults: a new instrument for measuring patients' literacy skills. J Gen Intern Med. 1995; 10(10):537–41.

15.

Abel

, Hofmann

, Ackermann

, et al. Health literacy among young adults: a short survey tool for public health and health promotion research. Health Promot Int. 2015; 30(3):725–35.

16.

Ishikawa

, Takeuchi

, Yano

. Measuring functional, communcative and critical health literacy among diabetic patients. Diabetes Care. 2008; 31(5):874–9.

17.

Massey

, Prelip

, Calimlim

, et al. Findings towards a multidimensional measure of adolescent health literacy. Am J Health Behav. 2013; 37(3):342–50.

18.

, Begoray

, MacDonald

, et al. Developing and evaluating a relevant and feasible instrument for measuring health literacy of Canadian high school students. Health Promot Int. 2010; 25(4):444–52.

19.

Nutbeam

. Health literacy as a public health goal: a challenge for contemporary health education and communication strategies into the 21st century. Health Promot Int. 2000; 15(3):259–67.

20.

Osborne

, Batterham

, Elsworth

, et al. The grounded psychometric development and initial validation of the Health Literacy Questionnaire (HLQ). BMC Public Health. 2013; 13:658.

21.

Davis

, Wolf

, Arnold

, et al. Development and validation of the Rapid Estimate of Adolescent Literacy in Medicine (REALM-Teen): a tool to screen adolescents for below-grade reading in health care settings. Pediactrics. 2005; 118(6):e1707–e14.

22.

Fransen

, Van Schaik

, Twickler

, Essink-Bot

. Applicability of internationally available health literacy measures in the Netherlands. J Health Commun. 2011; 16(supp 3):134–49.

23.

Kincaid

, Fishburne

Jr , Rogers

, Chissom

. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. DTIC Document, 1975.

24.

Samejima

. Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph. 1969; 34(4 Pt 2):100.

25.

Rizopoulos

. Itm: an R package for latent variable modelling and item response theory analyses. J Stat Softw. 2006; 17(5):1–25.

26.

Aubin

, Bingen

, Carpentier

, et al. Challenges conducting qualitative psychosocial research for adolescent and young adult patients and survivors. J Adolesc Young Adult Oncol. 2011; 1(2):71–6.

27.

Cantrell

, Conte

, Hudson

, et al. Recruitment and retention of older adolescent and young adult female survivors of childhood cancer in longitudinal research. Oncol Nurs Forum. 2012; 39(5):483.