Reading Grade Level and Completeness of Freely Available Materials on Thyroid Nodules: There Is Work to Be Done

Abstract

Background:

Patient education is one of the key components required for patients to participate in decision making for their care. Sources of information should be complete and understandable. The objective of this study was to assess the quality and reading grade level of freely available materials pertaining to the evaluation and treatment of thyroid nodules.

Methods:

All available materials were gathered through a comprehensive Internet search or direct contact with each identified organization associated with thyroid care. Materials meeting inclusion criteria were assessed for completeness of content and reading grade level using the SMOG readability formula. Then, the DISCERN instrument was utilized to assess the quality of the five materials which had the best (lowest) and the five materials that had the worst (highest) SMOG reading grade levels.

Results:

A total of 63 materials on thyroid nodule evaluation and management were identified. A minority of materials presented readers with a complete picture of the topic, describing both benefits and cautions of thyroid nodule evaluation, benefits and cautions of fine-needle thyroid nodule biopsy, or risks, benefits, and long-term effects of thyroidectomy. The SMOG reading grade level assessment was well above recommended levels. The range was eighth grade to college level; the mean and median reading grade level were 13. Lastly, the DISCERN instrument revealed that most of the materials with both the best and the worst reading grade levels assessed had “extensive or serious shortcomings” that would impede their usefulness as high-quality sources of information regarding management options.

Conclusions:

Existing thyroid nodule evaluation and management materials have serious shortcomings. Future work should focus on the development of patient materials that are both grade level appropriate and meet the DISCERN criteria for topic completeness and accuracy.

Introduction

The U.S. healthcare system is moving toward more patient-centered care. Patient education, along with opportunities for explicit deliberation and determination when making healthcare decisions, are key to patients assuming the more active role required for this model of care to work (1,2). Ideally, all commonly addressed medical decisions would be represented by formally developed and tested decision aids—tools specifically designed to impart information clearly, facilitate clarification of personal values and priorities related to the decision, and promote active discussion with the healthcare provider. Failing that, available information should at least be provided that is complete, accurate, and understandable. Unfortunately, even the most straightforward of these criteria—reading grade level—have not historically been met (3).

In today's society, a significant portion of health information is accessed from the Internet. In 2012, 81% of U.S. adults used the Internet, and 72% of these individuals used the Internet to search for health information (4). This broad access to Internet-based healthcare information greatly expands accessibility to patient education materials, but the usefulness of these patient materials is still limited by their understandability. In the United States, just 12% of the adult population has “proficient” health literacy skills, defined as being able to obtain, process, and understand basic health information. Proficient health literacy is necessary to fill out medical forms, provide a personal history, engage in self-care for chronic illnesses, and calculate probabilities and risks. More than a third of U.S. adults have only “basic” health literacy (able to understand common prose or simple documents) or “below basic” health literacy (nonliterate in English) (5).

Because U.S. health literacy is limited, the National Institutes of Health recommends that patient materials be written at or below the sixth grade reading level in order to be most effective and understood (6). Beyond simple assessments of grade level, the quality of patient materials matters, and a measurement tool (DISCERN) has been created for this purpose. A high-quality patient material is defined by the creators of the DISCERN criteria as one that is: clear about its purpose, complete, accurate, unbiased, and includes references that can be fact checked (7).

The need for high-quality materials is especially true for the problem of thyroid nodules. They are rarely malignant, but become more and more common with advancing age—by the age of 50, half the population has nodules; by the age of 90, virtually everyone has nodules (8). In recent years, thyroid findings and thyroid cancers have been uncovered at dramatically increasing rates, without changes in the number of people dying from the cancer, raising questions about appropriate treatment (9,10). High-quality patient information is imperative for patients to participate in decision making about their own thyroid findings. In this study, what was contained in materials freely available to people searching for information about thyroid nodules and their management is described. The completeness of content, reading grade level, and overall quality of widely available materials are assessed. The findings and recommendations for future work are reported.

Materials and Methods

Available materials on thyroid nodule evaluation and management were analyzed using three strategies: content analysis to summarize commonly included and omitted information, assessment of reading grade level using the Simple Measure of Gobbledygook Readability Formula (SMOG) (11), and assessment of quality through the DISCERN instrument (7).

Materials identification

Two methods were used to identify all freely available materials related to thyroid nodule evaluation and management. The first was a search engine driven approach used to gather materials published in digital form. The second was a search of organizations known to provide care for thyroid disorders through direct telephone contact to request all patient materials available that might not have been published electronically. Each printed and digital source of material was saved electronically in an individual PDF file for the content analysis portion of the study.

In the search engine driven approach, Google was employed in June 2013 because it was the most frequently used search engine at that time. Specifically, two thirds of all Internet searches conducted within the United States were done using Google (12). To remove the influence of prior searches and Internet activity, the browsing history, download history, cookies, and other files documenting browsing information and preferences were cleared. The phrases “nodule in thyroid,” “thyroid cancer,” “I have a thyroid nodule,” and “cancer in thyroid nodule” were used to conduct the search. The first 20 unique results for each search phrase were saved for analysis. Results that led to open forums for patients to discuss their conditions and treatments, videos and celebrity interviews, or news-related pieces were excluded. The data from the search engine driven approach were used to create the list of organizations for contact in the direct telephone search.

The direct telephone search was conducted in order to ensure that any printed materials from associations involved in thyroid care not available online, though freely available in clinic offices, were not missed and omitted from this study. Three categories of organization were contacted by telephone: medical/professional, governmental, and patient support. The following medical organizations were contacted: The American Cancer Society, The National Comprehensive Cancer Network, The American Association of Clinical Endocrinologists, The American Thyroid Association, The American College of Endocrinology, The American Academy of Otolaryngology—Head and Neck Surgery, and The American Association of Endocrine Surgeons. The government organization contacted was The National Cancer Institute. Last, the following patient support groups were contacted: ThyCa and The Light of Life Foundation.

Content analysis for risk and benefit information

A sentence-by-sentence evaluation of each material was completed to assess for the presence and absence of the following content: benefits of and cautions about thyroid nodule evaluation and fine-needle thyroid nodule biopsy; and risks, benefits, and long-term effects of thyroidectomy. Table 1 outlines the concepts identified iteratively through close reading of the documents to categorize the data related to benefits, risks, and long-term effects for each topic. Initial coding was done by one author (J.A.B.) with audit and checking for agreement by (L.D.). Disagreements were resolved by consensus and, when necessary, discussion with the senior author's research team.

Table 1.

All Concepts Identified in Materials During Content Analysis

Benefits	Cautions
Thyroid nodule evaluation:
• “… thyroid cancers can be found early …”• ‘… objective of evaluation is to exclude malignancy …’ • ‘… Can lead to a more prompt diagnosis and therefore a better outcome …’	• ‘… as we age, most of us will develop nodules …’ • ‘… after evaluation, most nodules will be found to be benign …’ • “… nodules less than 1 cm can be followed with observation …”
Fine needle thyroid nodule biopsy:
• “… if positive, significantly reduces the need for more elaborate and expensive testing …”• ‘… most accurate test for diagnosing thyroid nodules …’ • ‘… simple, quick, in-office procedure …’ • “… fast, safe, and associated with few risks …”• ‘… most cost effective approach …’	• “… sampling errors occur in very large or very small nodules …”• ‘… test is frequently inconclusive and requires additional testing …’ • ‘… accuracy is dependent upon physician and pathologist skill and experience …’ • “… nodules less than 1 cm can be followed without the need for fine needle aspiration …”

Benefits	Risks	Long-term effects
Thyroidectomy:
• ‘Surgery with radioiodine therapy may be sufficient to completely treat cancer’ • ‘… high survival rate following surgery …’ • “… best chance for survival and the lowest risk of recurring cancer …”• ‘Beneficial for nodules or cancers that are causing compression of surrounding structures’	• ‘… surrounding structures can be damaged during surgery …’ • ‘… risk of blood loss or infection during or after surgery …’ • ‘… risk of hypoparathyroidism following surgery …’	• ‘… will require life-long thyroid hormone replacement …’ • ‘… risk of life-long hypoparathyroidism requiring close monitoring of calcium levels …’

Direct quotes are in italics within double quotation marks; paraphrases are in roman within single quotation marks.

Reading grade level assessment

Reading grade level of each material was assessed by utilizing the SMOG formula (11). To do this, 30 sentences were randomly selected from each source, and the number of words with three or more syllables was determined. The SMOG reading grade level was calculated with the following equation: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$${\rm SMOG} = 3 + {\sqrt {\#\ of \ polysyllabic \ words}}$$ \end{document} . The SMOG formula has been found to be the best readability formula for healthcare applications due to its simplicity and ease of use. Furthermore, the SMOG formula demonstrates greater consistency as commonly applied when compared to other readability formulas such as the Flesch–Kincaid and FOG formulas (13). Once the reading grade level for each source was calculated using the SMOG formula, the average reading grade level score was calculated by type of source (i.e., professional society, hospital, health information Web site, government, or patient support group).

Quality of information assessment

The five materials with reading grade levels that most closely approximated the National Institutes of Health's recommended sixth grade reading level (6) and the five materials with the highest reading grade level were assessed using the DISCERN instrument (7). The DISCERN instrument allows for the assessment of the quality of written patient information by taking into account more qualitative aspects of the written materials (7). For example, the DISCERN instrument does not rely on objective reading grade level calculations, but rather focuses on benchmarks of quality such as clarity of purpose of the document, relevance of information presented, and balanced or unbiased delivery of information. The DISCERN instrument has been designed to be used by anyone involved in the development or use of health educational materials. It consists of 15 questions that enable the user to assess the quality of those materials. Once the assessment questions are completed, publications are given an overall numerical score ranging from 1 to 5. A score of 5 indicates that a publication has minimal shortcomings, is of high overall quality, and is a useful source of information about health treatment options. A score of 3 to 4 indicates a publication has potentially important shortcomings and is a limited source of information regarding treatment options. Finally, a score of 1 to 2 indicates that a publication has extensive shortcomings and is not a useful source of information regarding treatment options (7). Ideal patient material by DISCERN standards represents current levels of evidence, is not biased, refers to a range of choices for treatment, and provides links or access to key information sources to allow for fact checking (14).

Results

Document identification

The comprehensive search yielded an initial total of 72 freely available materials. At initial screening, nine sources were excluded because they did not meet our definition—they were news articles, celebrity interviews, or blog forums. This yielded 63 materials for analysis (15 –77).

Content analysis of documents

Thyroid nodule evaluation

Nine materials covered evaluation of thyroid nodules. Seven of these sources thoroughly described the benefits, but only two mentioned cautions or reasons why thyroid nodule evaluation might not be necessary (Fig. 1). One benefit listed was not accurate—that prompt evaluation of thyroid nodules leads to earlier diagnoses of malignancy and a higher likelihood of surgical cure (fallacy of early detection). Earlier identification of thyroid malignancy (or any malignancy) increases length of survival, but does not necessarily reduce mortality (78 –80).

FIG. 1.

Thyroid nodule evaluation. Information on benefits and cautions provided in assessed materials.

Cautions or reasons not to evaluate a thyroid nodule were not described fully in either of the materials that listed them—the two documents each listed one point. The first patient material mentioning cautions of thyroid nodule evaluation explained that thyroid nodules become much more common with advancing age and, following evaluation, most of these are found to be benign (21). The other source stated that the majority of nodules <1 cm in diameter could simply be monitored over time (20). While this might be generally true, some thyroid management guidelines suggest criteria other than 1 cm are most appropriate for determining whether to biopsy a nodule. Neither of these materials described the potential harms specifically associated with unnecessary workups or treatments of thyroid nodules.

Thyroid needle biopsy evaluation

Of the 27 materials that discussed needle biopsy of thyroid nodules, 21 provided their readers with only benefits, four provided both benefits and cautions of this procedure, and two provided only cautions pertaining to this procedure (Fig. 2). While seven of the 27 materials mentioned that the biopsy might need to be repeated if the results returned inconclusive, this was not categorized as a caution for thyroid needle biopsy specifically because it is a concept relevant to needle biopsy testing in general, and our interest was specifically in thyroid-related benefits and cautions.

FIG. 2.

Needle biopsy of thyroid nodules. Information on benefits and cautions provided in assessed materials.

Thyroidectomy

Of the 37 materials that provided information on thyroidectomy, only three covered all three areas considered as important: risks, benefits, and long-term effects of the procedure (Fig. 3). Seventeen of the 36 materials outlined only the risks associated with the surgery, without listing benefits of thyroidectomy or long-term effects. While the majority of the 36 materials indirectly presented thyroid removal as beneficial by using phrases such as “your doctor might recommend you have your thyroid removed” or “most thyroid cancers are treated with thyroidectomy,” only 15 of these materials made explicit statements describing the benefits associated with this procedure. The explicit benefits described were high cure rates associated with thyroidectomy, decompression of surrounding structures following tumor removal, and good survival rates associated with surgery. Twenty-nine of the 36 materials specifically mentioned the life-long need for thyroid hormone replacement as a long-term effect of thyroidectomy.

FIG. 3.

Thyroid surgery. Information on risks, benefits, and long-term effects provided in assessed materials.

Reading grade level

Table 2 demonstrates the average reading level when the materials were stratified by source type, thus indicating that patient information produced by the government had the lowest average reading level (11th grade), while those produced by professional societies had the highest (college level).

Table 2.

Reading Grade Level of Included Materials

Source type	Professional society (n=16)	Hospital (n=18)	Health Information Web site (n=21)	Government (n=3)	Patient support group (n=5)
Organization name	• American Cancer Society • American Academy of Otolaryngology • American Thyroid Association • American Association of Clinical Endocrinologists • Empower Your Health • National Comprehensive Cancer Network • American Association of Endocrine Surgeons • American Academy of Family Physicians • Medical Journal of Australia	• University of Michigan • Mayo Clinic • UC Denver • Stanford • Cedars-Sinai • Cleveland Clinic • Washington Hospital • Columbia • MD Anderson Cancer Center • Sloan-Kettering • US San Francisco • Invision Sally Jobe	• WebMD.com • eMedicineHealth.com • MedlinePlus.com • Cancer.net • About.com • MedicineNet.com • EndocrineWeb.com • New York Times Health Guide • Cleveland.com • Medical News Today • UpToDate.com • Wikipedia.com • Cigna	• WomensHealth.gov • National Cancer Institute • CDC	• ThyCa • The Light of Life Foundation
Reading grade level Mean, median (range)	14, 14 (10 –16)	13, 13 (10 –18)	12, 12 (8 –15)	11, 9 (9 –13)	13, 14 (10 –15)

Score calculated using the Simple Measure of Gobbledygook (SMOG) Readability Formula.

Individually, the reading grade level assessment of the 63 materials revealed a spectrum of reading grade levels ranging from eighth grade to college level. The median grade level was grade 13, and the overall average was also grade 13, both of which fall within the college level reading range. More than 30 of the materials assessed were written at or above the college reading level. In fact, the most common reading grade level observed (the mode) was grade 14, which is equivalent to a college reading level. The five materials with the lowest (best) reading grade levels were: WebMD's Thyroid Cancer Overview (eighth grade) (16), WomensHealth.gov's Thyroid Disease FAQ (ninth grade) (18), The National Cancer Institute's About Thyroid Cancer booklet (ninth grade) (19), eMedicineHealth's webpage on thyroid nodules (ninth grade) (15), and WebMD's webpage on thyroid nodules (ninth grade) (17). It should be noted that WebMD and eMedicineHealth are owned by the same company. This company buys its medical content from one company, Healthwise. Healthwise is a privately owned company that produces content for a myriad of uses by hospitals, physician clinics, and insurance companies, among others. The five materials with the highest (worst) reading grade levels, all of which were college levels were: American Family Physician's “Thyroid Nodules” (22), The American Thyroid Association's “Thyroid Surgery” (34), Medical Journal of Australia's “Thyroid Nodules and Thyroid Cancer” (49), American Association of Clinical Endocrinologists' “Thyroid Carcinoma” (28), and the Mayo Clinic's “Thyroid Nodule Clinic” (52).

Information quality

The DISCERN instrument was applied to the five materials with the best and the five materials with the worst reading grade levels (Table 3). Four of the five materials with the best reading grade levels received an overall score of 2–3—having “serious or extensive shortcomings.” Only one of the sources, the National Cancer Institute's “What You Need to Know About Thyroid Cancer” (19), was given an overall score of 4—having “potentially important but not serious shortcomings.” Of the materials with the worst (highest) reading grade levels, the overall scores range from a high of 3 with “important shortcomings” to a low of 1–2 with “serious or extensive shortcomings.”

Table 3.

Overall Document Quality of the Freely Available Materials with the Highest and Lowest Reading Grade Levels

Reading grade level	Patient material and date of last revision	DISCERN score
Lowest reading grade level	• WebMD—Thyroid Cancer Overview—4/2011	2–3
	• Women's Health.gov—Thyroid Disease FAQ—4/2009	2–3
	• National Cancer Institute—About Thyroid Cancer—1/2012	4
	• eMedicineHealth—Thyroid Nodules—4/2011	2–3
	• WebMD—Thyroid Nodules—3/2011	2–3
Highest reading grade level	• Mayo Clinic—Thyroid Nodule Clinic—2012	1–2
	• American Association of Clinical Endocrinologists—Thyroid Carcinoma—No date of revision	2
	• Medical Journal of Australia—Thyroid Nodules and Thyroid Cancer—2004	3
	• American Thyroid Association—Surgery—2012	3
	• American Family Physician—Thyroid Nodules—2/2003	3

Scores calculated using the DISCERN instrument for the five materials with the highest reading grade levels and for the five materials with the lowest reading grade levels.

While each of the 10 materials had their own successes and shortcomings when assessed with the DISCERN instrument, there was some commonality in regards to the areas that scored poorly and were deemed inadequate. Collectively, 8 of the 10 materials failed to provide both in-text citations and full bibliographies in order to make it clear what sources were used in the compilation of information. Additionally, each of these sources failed to highlight adequately the areas of uncertainty within the field of thyroid nodules and thyroid cancer. Also, these sources did not explicitly describe what would happen if a monitoring approach rather than treatment was chosen.

Evaluation of selected materials by both DISCERN score and reading grade level

Among the 10 materials that encompassed the five highest and lowest reading grade levels, there was little variation in DISCERN scores. Materials were most commonly found to have scores in the range of 2–3. One material performed best overall, having the lowest reading grade level (9) and the highest DISCERN score (4): the National Cancer Institute's “What You Need to Know About Thyroid Cancer” (19).

Aside from the one factual problem noted earlier about the fallacy of early detection, one additional factual inaccuracy was present in two of these documents (both documents were content from Healthwise): exposure to any radiation was listed as a risk factor for thyroid cancer. High-quality epidemiologic data show that the risk of thyroid cancer is only elevated in people who have significant radiation exposure prior to the age of 20—those exposed after that time have a risk no greater than the risk of people not exposed to radiation (81).

Discussion

This literature scan revealed no grade level appropriate, complete, high-quality materials on thyroid nodule evaluation, needle biopsy, or surgery. Most materials emphasized only one aspect, such as benefits or harms, without complete information in both areas. Most materials were written at levels that far exceeded the sixth-grade reading level recommended by the NIH (6), and none of the materials described the current state of the science regarding controversies surrounding strategies for management other than immediate intervention. The conclusions from the DISCERN instrument, which allows assessment of qualities such as bias and relevance, agreed with the findings of the content analysis: even materials with the best (lowest) reading grade level were mostly rated as having “important or serious shortcomings” (rated 2–3 out of 5, where 5 is the best). Out of the 63 materials identified, just one material was close to ideal: the National Cancer Institute's “What You Need to Know About Thyroid Cancer,” written at a reading grade level of 9 and having a DISCERN score of 4 (19).

Other studies of materials related to thyroid care have arrived at similar conclusions. A study assessing the reading grade level of materials concerning thyroid surgery found that all sources assessed had reading grade level scores higher than the recommended reading levels (82). In other areas of medicine, materials examined have been found to be commonly written at reading grade levels higher than recommended, including facial plastic and reconstructive surgery (83), endoscopic sinus surgery (84), vascular and interventional radiology (85), gastroenterology (86), neurosurgery (87), and orthopedic surgery (88,89). The problem occurs across source type—professional societies, hospitals, health information Web sites, government sources, and patient support groups. All of them consistently provide patient information at reading levels higher than those that are recommended.

The problem of incompleteness of content regarding management strategies other than immediate surgical intervention for thyroid nodules is also not unique to this field. In a similar study on prostate cancer patient materials—a disease with diagnostic issues similar to those of thyroid cancer, and for which the problems of subclinical detection have been recognized far longer—the same shortcomings in state of the science information were identified (90).

The findings in this study have limitations. First, while a systematic Internet search was conducted and each major organization was contacted by phone during the acquisition phase of this study to make sure all relevant materials were obtained, it is possible some materials were missed. For instance, if a local hospital or organization has materials they use internally but do not post publicly, that would be missed. Loss of these data could potentially decrease our estimates of the quality of patient education materials, if those materials were of better quality than other published materials. In addition, it is likely that the exact search results may have changed since June 2013. However, the overall quality of materials available as a whole on the Internet is unlikely to have changed significantly over that time. Also, materials found that had revision dates listed on them were typically a year or older at the time they were accessed, suggesting that overall document age may be at least as relevant as where a source appears in a search listing. Lastly, the SMOG algorithm for assessing reading grade level uses a randomly selected set of sentences to assess the document as a whole. Therefore, the certainty of the SMOG scores calculated is subject to some random variation.

Moving forward, how could materials on thyroid nodule evaluation and surgery be improved? There are lessons to be found in the most currently available materials on prostate and breast cancer screening from the U.S. Preventive Services Task Force (91,92). While there are important caveats, because thyroid cancer is not a screening detected disease, there are two major things that could be learned from their approach. First, in their materials, the goals of the intervention are made very clear. For example, in prostate cancer, the goal of screening is to lower the number of men dying of the disease. Since that benefit has not been shown in large-scale studies, and treatment of identified cancers has significant side effects, the screening is no longer uniformly recommended. The second communication principle observed in breast and prostate cancer materials is the clear outlining of potential harms that can come from interventions. Thyroid materials could be clearer about the goal of each procedure, whether it is needle biopsy or surgery, and the likelihood of each possible outcome. Lastly, future thyroid materials could add information on the areas where the DISCERN instrument revealed our greatest weaknesses: being clear about the areas of scientific uncertainty and adding information about what might happen if no treatment is pursued.

In conclusion, available materials on thyroid nodule evaluation, needle biopsy, and surgery do not equip patients with the knowledge necessary to be informed participants in decisions regarding their thyroid care because they omit key topics such as benefits and downsides of nodule evaluation or risks, long-term consequences of treatment, and information on the state of the science for management options for subclinical disease. If the progress in the patient materials for prostate cancer is any indication, it will require much time, work, and consensus building among physicians involved in thyroid care to reach the goal of presenting unbiased, relevant, up-to-date, and complete patient information. Future work should focus on the development of materials that provide information at the appropriate reading grade level, and meet the DISCERN criteria for accuracy and completeness: the materials represent current levels of evidence and the state of the science, are not biased, refer to a range of choices for treatment, and provide links or access to key information sources to allow for fact checking (14). Improvement in patient materials will help both patients and physicians navigate the complexities of thyroid nodule management.

Footnotes

Acknowledgment

The views expressed do not necessarily represent the views of the Department of Veterans Affairs or the United States Government.

Author Disclosure Statement

No competing financial interests exist.

References

Joseph-Williams

, Elwyn

, Edwards

. 2014. Knowledge is not power for patients: a systematic review and thematic synthesis of patient-reported barriers and facilitators to shared decision making. Patient Educ Couns, 94:291–309.

Elwyn

, Miron-Shatz

. 2010. Deliberation before determination: the definition and evaluation of good decision making. Health Expectations, 13:139–147.

Berland

, Elliott

, Morales

, Algazy

, Kravitz

, Broder

, Kanouse

, Muñoz

, Puyol

, Lara

, Watkins

, Yang

, McGlynn

. 2001. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA, 285:2612–2621.

Fox

, Duggan M 2013 Health online 2013. Pew Research Center's Internet and American Life Project. Available at: www.pewinternet.org/2013/01/15/health-online-2013/ (accessed December 22, 2014).

Kutner

, Greenberg

, Jin

, Paulsen

. 2006. The health literacy of America's adults: results from the 2003 National Assessment of Adult Literacy (NCES 2006–483). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Available at: http://eric.ed.gov/?id=ED493284

The National Institutes of Health 2013 How to write easy-to-read health materials. Available at: www.nlm.nih.gov/medlineplus/etr.html (accessed December 23, 2013 ).

Charnock

, Shepperd

, Needham

, Gann

. 1999. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health, 53:105–111.

Mortensen

, Woolner

, Bennett

. 1955. Gross and microscopic findings in clinically normal thyroid glands. J Clin Endocrinol Metab, 15:1270–1280.

Davies

, Welch

. 2014. Current thyroid cancer trends in the United States. JAMA Otolaryngol Head Neck Surg, 140:317–322.

10.

Ito

, Miyauchi

, Inoue

, Fukushima

, Kihara

, Higashiyama

, Tomoda

, Takamura

, Kobayashi

, Miya

. 2010. An observational trial for papillary thyroid microcarcinoma in Japanese patients. World J Surg, 34:28–35.

11.

Laughlin

GHM

. 1969. SMOG grading—a new readability formula. J Reading, 12:639–646.

12.

comScore 2013 comScore releases July 2013 U.S. search engine rankings. Available at: www.comscore.com/Insights/Press-Releases/2013/8/comScore-Releases-July-2013-US-Search-Engine-Rankings (accessed September 19, 2014 ).

13.

Wang

, Miller

, Schmitt

, Wen

. 2013. Assessing readability formula differences with written health information materials: application, results, and recommendations. Res Social Adm Pharm, 9:503–516.

14.

The DISCERN Project Background to DISCERN. Available at: www.discern.org.uk/background_to_discern.php (accessed April 13, 2014 ).

15.

Healthwise Staff 2011 eMedicineHealth—Thyroid nodules. Available at: www.emedicinehealth.com/thyroid_nodules-health/article_em.htm (accessed June 17, 2013 ).

16.

Healthwise Staff 2011 WebMD—Thyroid cancer. Available at: www.webmd.com/cancer/tc/thyroid-cancer-topic-overview (accessed June 17, 2013 ).

17.

Healthwise Staff 2013 WedMD—Thyroid nodules. Available at: www.webmd.com/a-to-z-guides/thyroid-nodules-topic-overview (accessed June 17, 2013 ).

18.

Jonklass

, Helfand

. 2010 WomensHealth.gov Thyroid disease—FAQ. Available at: www.womenshealth.gov/publications/our-publications/fact-sheet/thyroid-disease.pdf (accessed June 17, 2013 ).

19.

National Cancer Institute 2012 What you need to know about thyroid cancer, Internet Edition ed. U.S. Department of Health and Human Services. Available at: www.cancer.gov/publications/patient-education/wyntk-thyroid-cancer (accessed December 17, 2014 ).

20.

The Light of Life Foundation 2009 Thyroid cancer Q&A. Available at: www.checkyourneck.com/About-Thyroid-Cancer/What-is-it#nodules (accessed June 18, 2013 ).

21.

Norman

. 2012. Thyroid cancer. Available at: www.endocrineweb.com/conditions/thyroid-cancer/thyroid-cancer (accessed June 17, 2013 ).

22.

American Academy of Family Physicians 2003 Thyroid nodules. Available at: www.aafp.org/afp/2003/0201/p573.html (accessed June 17, 2013 ).

23.

American Academy of Otolaryngology—Head and Neck Surgery 2010 Thryoid disorders and surgery. Available at: www.entnet.org/HealthInformation/Thyroid-Disorders.cfm (accessed June 13, 2013 ).

24.

American Academy of Otolaryngology—Head and Neck Surgery 2011 Fine needle aspiration. Available at: www.entnet.org/HealthInformation/fineNeedleAspiration.cfm (accessed June 13, 2013 ).

25.

American Association of Clinical Endocrinologists. About thyroid cancer. Available at: http://empoweryourhealth.org/endocrine-conditions/thyroid/about_thyroid_cancer (accessed June 13, 2013 ).

26.

American Association of Clinical Endocrinologists. About thyroid nodules. Available at: http://empoweryourhealth.org/endocrine-conditions/thyroid/about_thyroid_nodules (accessed June 13, 2013 ).

27.

American Association of Clinical Endocrinologists. Radioiodine therapy. Available at: http://thyroidawareness.com/sites/all/files/Radioiodine-Therapy.pdf (accessed June 18, 2013 ).

28.

American Association of Clinical Endocrinologists. Thyroid carcinoma. Available at: http://thyroidawareness.com/sites/all/files/Thyroid-Carcinoma.pdf (accessed June 18, 2013 ).

29.

American Association of Clinical Endocrinologists. Thyroid nodules. Available at: http://thyroidawareness.com/sites/all/files/Thyroid-Nodules.pdf (accessed June18, 2013).

30.

American Cancer Society 2014 Thyroid cancer. Available at: www.cancer.org/acs/groups/cid/documents/webcontent/003144-pdf.pdf (accessed June 13, 2013 ).

31.

American Cancer Society 2014 Thyroid cancer overview. Available at: www.cancer.org/acs/groups/cid/documents/webcontent/acspc-030369-pdf.pdf (accessed June 18, 2013 ).

32.

American Thyroid Association 2012 Thyroid cancer. Available at: www.thyroid.org/wp-content/uploads/patients/brochures/ThyroidCancer_brochure.pdf (accessed June 13, 2013 ).

33.

American Thyroid Association 2012 Thyroid nodules. Available at: www.thyroid.org/wp-content/uploads/patients/brochures/Nodules_brochure.pdf (accessed June 13, 2013 ).

34.

American Thyroid Association 2012 Thyroid surgery. Available at: www.thyroid.org/wp-content/uploads/patients/brochures/ThyroidSurgery.pdf (accessed June 13, 2013 ).

35.

Cancer.net Editorial Board 2013 Thyroid cancer. Available at: www.cancer.net/cancer-types/thyroid-cancer (accessed June 17, 2013 ).

36.

Cedars-Sinai.. Thyroid cancer. Available at: www.cedars-sinai.edu/Patients/Health-Conditions/Thyroid-Cancer.aspx (accessed June 17, 2013 ).

37.

Cedars-Sinai. Thyroid nodules. Available at: www.cedars-sinai.edu/Patients/Health-Conditions/Thyroid-Nodules.aspx (accessed June 17, 2013 ).

38.

Centers for Disease Control and Prevention 2012 Thyroid cancer and the environment. Available at: http://ephtracking.cdc.gov/showCancerThyroidEnv.action (accessed June 17, 2013 ).

39.

Columbia University Medical Center 2014 Thyroid nodules. Available at: http://columbiathyroidcenter.org/nodules.html (accessed June 17, 2013 ).

40.

Conrad Stöppler

MedicineNet—Thyroid cancer. Available at: www.medicinenet.com/thyroid_cancer/article.htm (accessed June 17, 2013 ).

41.

Ferry

. 2013. eMedicine Health—Thyroid nodules. Available at: www.emedicinehealth.com/thyroid_nodules/article_em.htm (accessed June 17, 2013 ).

42.

Ferry

. 2013. MedicineNet—Thyroid nodules. Available at: www.medicinenet.com/thyroid_nodules/article.htm (accessed June 17, 2013 ).

43.

Haugen

. University of Colorado Denver—Thyroid nodules: what you should know. Available at: www.ucdenver.edu/academics/colleges/medicalschool/departments/medicine/EndocrinologyMetabolismDiabetes/Programs/Neoplasms/Documents/Thyroid_Nodules_Brochure.pdf (accessed June 17, 2013 ).

44.

Healthwise Staff 2013 Cigna—thyroid nodules. Available at: ww.cigna.com/individualandfamilies/health-and-well-being/hw/medical-topics/thyroid-nodules-hw169627.html (accessed June 17, 2013 ).

45.

Invision Sally Jobe. Thyroid cancer and thyroid nodules. Available at: www.riainvision.com/conditions/thyroid_cancer_nodules.aspx (accessed June 17, 2013 ).

46.

Kraft

. 2010. What are thyroid nodules? What causes thyroid nodules? Available at: www.medicalnewstoday.com/printerfriendlynews.php?newsid=185672 (accessed June 17, 2013 ).

47.

Landry

C Banner

. MD Anderson Cancer Center—My doctor told me I have a thyroid nodule. What are the chances that I have thyroid cancer? Available at: www.bannerhealth.com/Locations/Arizona/Banner+MD+Anderson+Cancer+Center/News/Ask+the+Expert/_My+doctor+told+me+I+have+a+thyroid+nodule.+What+are+the+chances+that+I+have+thyroid+cancer.htm (accessed June 17, 2013 ).

48.

Light of Life Foundation. About the thyroid gland. Available at: www.checkyourneck.com/About-Thyroid-Cancer/About-the-Thyroid-Gland (accessed June 13, 2013 ).

49.

Mackenzie

, Mortimer

. 2004. The Medical Journal of Australia—Thyroid nodules and thyroid cancer. Available at: www.mja.com.au/journal/2004/180/5/6-thyroid-nodules-and-thyroid-cancer (accessed June 17, 2013 ).

50.

Mathur

. 2014. MedicineNet—Fine needle aspiration biopsy of the thyroid. Available at: www.medicinenet.com/fine-needle_aspiration_biopsy_of_the_thyroid/article.htm (accessed June 17, 2013 ).

51.

Mayo Clinic Staff. Thyroid cancer. Available at: www.mayoclinic.org/thyroid-cancer/ (accessed June 17, 2013 ).

52.

Mayo Clinic Staff. Thyroid nodule clinic. Available at: www.mayoclinic.org/medicalprofs/thyroid-nodule-clinic.html (accessed June 17, 2013 ).

53.

Mayo Clinic Staff 2014 Thyroid cancer basics. Available at: www.mayoclinic.com/health/thyroid-cancer/DS00492 (accessed June 17, 2013 ).

54.

Mayo Clinic Staff 2014 Thyroid nodules. Available at: www.mayoclinic.com/health/thyroid-nodules/DS00491 (accessed June 17, 2013 ).

55.

MD Anderson Cancer Center. The University of Texas—Thyroid nodule clinic. Available at: www.mdanderson.org/patient-and-cancer-information/care-centers-and-clinics/care-centers/endocrine/thyroid-nodule-clinic/index.html (accessed June 17, 2013 ).

56.

MedStar Washington Hospital Center Thyroid nodules. Available at: www.whcenter.org/blank.cfm?print=yes&id=558195 (accessed June 17, 2013 ).

57.

Memorial Sloan Kettering Cancer Center. About thyroid cancer. Available at: www.mskcc.org/cancer-care/adult/thyroid/about-thyroid (accessed June 17, 2013 ).

58.

Murphy

. 2013. Thyroid nodules usually not cancerous, but should be treated. Available at: http://blog.cleveland.com/health_impact/print.html?entry=/2013/05/thyroid_nodules_usually_not_ca.html (accessed June 17, 2013 ).

59.

National Comprehensive Cancer Network. Thyroid cancer: an overview. Available at: www.nccn.com/component/content/article/54-cancer-basics/1742-thyroid-cancer-overview.html (accessed June 13, 2013 ).

60.

Norman

. 2010. Fine needle biopsy of thyroid nodules: is it cancer or just a benign nodule? Available at: www.endocrineweb.com/conditions/thyroid/fine-needle-biopsy-thyroid-nodules (accessed June 17, 2013 ).

61.

Norman

. 2014. Thyroid nodules: hyperthyroidism and thyroid cancer. Available at: www.endocrineweb.com/conditions/thyroid/thyroid-nodules (accessed June 17, 2013 ).

62.

Oertel

. 2010. ThyCa: Thyroid Cancer Survivors' Association, Inc. Fine needle aspiration of the thyroid. Available at: http://thyca.org/fna.htm (accessed June 17, 2013 ).

63.

Ross

. 2014. UpToDate Patient information: thyroid nodules (beyond the basics). Available at: www.uptodate.com/contents/thyroid-nodules-beyond-the-basics (accessed June 17, 2013 ).

64.

Sayre

. 2013. New York Times—The rising incidence of thyroid cancer. Available at: http://health.nytimes.com/health/guides/disease/thyroid-cancer (accessed June 17, 2013 ).

65.

Shomon

. 2003. Thyroid nodules/lumps and goiter (enlargement): thyroid nodules, thyroid lumps, and the thyroid enlargement known as a goiter. Available at: http://thyroid.about.com/cs/nodulesgoiters/a/nodules.htm (accessed June 17, 2013 ).

66.

Shomon

. 2007. Thyroid cancer symptoms: a look at the signs and symptoms of thyroid carcinoma. Available at: http://thyroid.about.com/od/thyroidcancer/a/symptoms.htm (accessed June 17, 2013 ).

67.

Stanford University Cancer Institute. Information about endocrine cancers: thyroid tumors. Available at: http://cancer.stanford.edu/endocrine/thyroid/ (accessed June 17, 2013 ).

68.

The American Association of Endocrine Surgeons. Thyroid nodule: fine needle aspiration biopsy (FNA). Available at: http://endocrinediseases.org/thyroid/nodule_fna.shtml (accessed June 17, 2013 ).

69.

The Cleveland Clinic Foundation 2012 Thyroid nodules. Available at: http://my.clevelandclinic.org/disorders/thyroid_nodule/hic_thyroid_nodules.aspx (accessed June 17, 2013 ).

70.

ThyCa: Thyroid Cancer Survivors' Association, Inc.. 2012. Thyroid cancer basics. Available at: http://thyca.org/download/document/350/TCBasics.pdf (accessed June 17, 2013 ).

71.

ThyCa: Thyroid Cancer Survivors' Association, Inc.. 2013. Lab tests and nodules. Available at: http://thyca.org/nodules.htm (accessed June 17, 2013 ).

72.

Topiwala

. 2012. MedlinePlus—thyroid nodule. Available at: www.nlm.nih.gov/medlineplus/ency/article/007265.htm (accessed June 17, 2013 ).

73.

University of California San Francisco. Goiter and thyroid nodules. Available at: http://endocrine.surgery.ucsf.edu/conditions–procedures/thyroid-nodules.aspx (accessed June 17, 2013 ).

74.

University of Michigan Health System, Department of Internal Medicine. Thyroid nodules. Available at: www.med.umich.edu/intmed/endocrinology/patients/Thyroid-nodules.htm (accessed June 17, 2013 ).

75.

University of Michigan Health System, Department of Surgery. Thyroid nodules. Available at: http://surgery.med.umich.edu/general/endocrine/patient/conditions/thyroid/thyroid_nodules.shtml (accessed June 17, 2013 ).

76.

Wikipedia 2014 Thyroid cancer. Available at: http://en.wikipedia.org/wiki/Thyroid_cancer (accessed June 17, 2013 ).

77.

Wikipedia 2014 Thyroid nodule. Available at: https://en.wikipedia.org/wiki/Thyroid_nodule (accessed June 17, 2013 ).

78.

Feinstein

, Sosin

, Wells

. 1985. The Will Rogers Phenomenon. N Engl J Med, 312:1604–1608.

79.

Ito

, Miyauchi

, Kihara

, Higashiyama

, Kobayashi

, Miya

. 2014. Patient age is significantly related to the progression of papillary microcarcinoma of the thyroid under observation. Thyroid, 24:27–34.

80.

Welch

, Schwartz

, Woloshin

. 2000. Are increasing 5-year survival rates evidence of success against cancer?. JAMA, 283:2975–2978.

81.

Furukawa

, Preston

, Funamoto

, Yonehara

, Ito

, Tokuoka

, Sugiyama

, Soda

, Ozasa

, Mabuchi

. 2013. Long-term trend of thyroid cancer risk among Japanese atomic-bomb survivors: 60 years after exposure. Int J Cancer, 132:1222–1226.

82.

Patel

, Cherla

, Sanghvi

, Baredes

, Eloy

. 2013. Readability assessment of online thyroid surgery patient education materials. Head Neck, 35:1421–1425.

83.

Misra

, Agarwal

, Kasabwala

, Hansberry

, Setzen

, Eloy

. 2013. Readability analysis of healthcare-oriented education resources from the American academy of facial plastic and reconstructive surgery. Laryngoscope, 123:90–96.

84.

Cherla

, Sanghvi

, Choudhry

, Liu

, Eloy

. 2012. Readability assessment of Internet-based patient education materials related to endoscopic sinus surgery. Laryngoscope, 122:1649–1654.

85.

Hansberry

, Kraus

, Agarwal

, Baker

, Gonzales

. 2014. Health literacy in vascular and interventional radiology: a comparative analysis of online patient education resources. Cardiovasc Intervent Radio, 37:1034–1040.

86.

Tian

, Champlin

, Mackert

, Lazard

, Agrawal

. 2014. Readability, suitability, and health content assessment of web-based patient education materials on colorectal cancer screening. Gastrointest Endosc, 80:284–290.

87.

Agarwal

, Hansberry

, Singh

, Heary

, Goldstein

. 2014. Quality assessment of spinal cord injury patient education resources. Spine (Phila Pa), 2014 Apr 8. [Epub ahead of print].

88.

Polishchuk

, Hashem

, Sabharwal

. 2012. Readability of online patient education materials on adult reconstruction Web sites. J Arthroplasty, 27:716 (abstract).

89.

Ganta

, Yi

, Hussein

, Frank

. 2014. Readability of sports medicine-related patient education materials from the American Academy of Orthopaedic Surgeons and the American Orthopaedic Society for Sports Medicine. Am J Orthop (Belle Mead NJ), 43:E65–68.

90.

Walling

, Maliski

, Bogorad

, Litwin

. 2004. Assessment of content completeness and accuracy of prostate cancer patient education materials. Patient Educ Couns, 54:337–343.

91.

U.S. Preventive Services Task Force 2012 Screening for Prostate Cancer. Are you considering being screened? Available at: www.uspreventiveservicestaskforce.org/Tools/ConsumerInfo/Index/information-for-consumers (accessed December 22, 2014 ).

92.

U.S. Preventive Services Task Force 2014 Screening for breast cancer—questions and answers. Available at: www.uspreventiveservicestaskforce.org/Page/Name/screening-for-breast-cancer—questions-and-answers (accessed October 7, 2014 ).