Capturing health literacy assessment in the electronic health record through evidence-based concept creation: A review of the literature and recommendations for action

Abstract

Health literacy is the capacity to understand and act upon health-related information and navigate the healthcare system. Published evidence demonstrates a relationship between health literacy and health status. Because of this, there are increasingly calls for a health literacy assessment to be collected and stored in the electronic health record for use by the healthcare team. This article describes the results of a literature review of health literacy assessment instruments with the goal of formulating semantically interoperable concepts that may be used to store the interpretation of the health literacy assessment in the electronic health record. The majority of health literacy instruments could be stored in the electronic health record using a three-concept solution of inadequate, marginal and adequate health literacy. This three-concept solution fully supports semantic interoperability needs across the patient care spectrum.

Keywords

electronic health record evidence-based health literacy interoperability literature review

Background

Since 2000, health literacy (HL) has been defined as “the capacity to obtain, process and understand basic health information and services needed to make appropriate health decisions.”¹ More recently, Baker² and Rudd³ suggest that “literacy is the product of individuals’ capacities and the health literacy-related demands and complexities of the healthcare system.” The Institute of Medicine (IOM)⁴ purports that HL measures should assess both patients’ skills and abilities as well as the demands and complexities of the healthcare systems.

Persons with limited HL are those individuals who read at a sixth-grade level or less, while those with marginal HL read between the seventh- and eighth-grade levels.⁴ In 2003, the US Department of Education National Assessment of Adult Literacy reported that only 12% of adults surveyed possessed proficient HL, while 36% fell at or below basic HL levels.⁵ Nearly a decade later, the 2012 Program for the International Assessment of Adult Competencies (PIAAC) demonstrated relatively little change with only 12% of adults showing proficient literacy skills.⁶

Low HL can hinder health communication, which is the exchange of health-related information that occurs between the patient and provider during a patient encounter in the clinical setting.⁴ Policy leaders suggest that HL assessment could become a standard for healthcare organizations.⁷ Both the Healthy People 2020 goals and the Communication and Health Information Technology (IT) standards address HL⁸ by advocating for an increase in the proportion of persons who report that their healthcare provider always provides easy-to-understand instructions about self-management of their health condition.

Low HL impacts a person’s ability to understand aspects of the medical regimen. Patients with low HL experience more serious medication errors,⁹ have higher rates of medication nonadherence,^10,11 take medications incorrectly and misinterpret labels and health communications.¹² Finally, populations with low literacy HL experience problems navigating the healthcare system, completing medical forms, following medical instructions, communicating with providers and reading appointment slips.⁴

Research has documented an association between limited HL and poorer health status.¹² Overall, patients with limited HL report poorer health status for themselves^12–14 and for their children.¹⁵ Moreover, an increased prevalence and severity of chronic disease¹⁶ and morbidity and mortality¹² have been reported in low HL populations.

Populations with low HL use more healthcare services and have higher healthcare costs. Patients with limited HL have more emergency department (ED) visits,¹⁶ increased ED recidivism,¹⁷ and more inpatient hospital stays^16,18,19 and receive fewer preventive screening services for themselves¹² and their children.¹⁵ Cost estimates of low HL in the United States vary from US$75 billion²⁰ to US$236 billion annually.²¹ In summary, patients who do not understand and cannot act on the medical information or instructions provided are more likely to have poor health status resulting in avoidable consumption of healthcare services and related costs.²²

HL is a critical factor when engaging patients in chronic disease self-management, transitions of care (TOCs), taking medication correctly²³ and health promotion and disease prevention.^24,25 Patients who are engaged in their health have heightened awareness of why preventive care is important and are more likely to practice healthy behaviors and disease-specific health management behaviors.²⁶ Patients with HL skills can self-manage their health to reduce inpatient hospital stays and use prescribed medications appropriately.

HL also plays an important role in care coordination. TOC can pose a risk to patients due to communication challenges that occur between patients and their providers and among providers. The lack of semantic interoperability, the degree to which systems and devices can exchange and interpret data to support comprehension by a user,²⁷ adds an additional barrier to effective communication when a patient transitions from one setting to another. Patients’ responsibilities in the TOC process include comprehending their discharge plans, managing their medication and understanding how to communicate with their providers when needed. For example, among patients discharged from EDs, 78% do not adequately understand important information or instructions. To further complicate the issue, most of these patients do not realize that they lack an understanding of the information presented to them.²⁸

HL measurement

According to Baker,²⁹ there are two approaches to HL measurement, performance-based tests of reading capacities and self-reported difficulty to understand healthcare professionals and written healthcare-related materials. A recent systematic literature review found no difference between the results of performance-based assessment versus self-reported HL assessment.³⁰ Performance-based assessment instruments include the Test of Functional Health Literacy in Adults (TOFHLA), Shortened Test of Functional Health Literacy in Adults (S-TOFHLA), the Rapid Estimate of Adult Literacy in Medicine (REALM), the Rapid Estimate of Adult Literacy in Medicine-Revised (REALM-R) and Newest Vital Sign (NVS). Self-reported assessments vary from single-item to four-item measures and may assess different aspects of an individual’s HL skills. One assessment instrument measures the ability to read the newspaper, medical forms or written healthcare materials, while another measures the use of a surrogate reader to help understand health information,³¹ and separate assessment items measure confidence in completing medical forms and problems learning about health conditions due to lack of comprehension of written information.⁴

In a systematic literature review of 85 studies from 1963 to 2004, Paasche-Orlow et al.³² reported 14 literacy skills assessment instruments that measured prose and document literacy, subdomains of reading capacity and numeracy. Of the 85 studies reviewed, the majority (68%) used the REALM or a version of the TOFHLA. The rates of low literacy reported in these studies were comparable regardless of the instrument used (REALM 22%; 95% confidence interval (CI): 17–27; TOFHLA 28%; 95% CI: 22–34). In a cross-sectional study of 400 emergency patients, Kiechle et al.³³ used several instruments to assess HL, including the S-TOFHLA, the NVS, the Single Item Literacy Screener (SILS), brief screening questions, REALM-R and the Medical Term Recognition Test (METER). The authors reported that all instruments were significantly correlated, and instruments targeting similar skills were more strongly correlated. A recent systematic literature review of HL and cancer screening from 1990 to 2011³⁴ included 10 articles and revealed the S-TOFHLA as the most frequently used instrument. Similarly, Altin et al.³⁵ reported that nearly one-third of the 17 instruments studied were based on the REALM or TOFHLA.

By performing HL assessment during a clinical encounter, providers could better identify patients with limited HL, address HL deficits and tailor communication to the needs of patients.³⁶ Consequently, HL assessment will likely be stored in the electronic health record (EHR). Although storing the HL assessment in the EHR has many benefits, the ability to capture, store, retrieve and synthesize HL instruments into usable information via the EHR is limited. In addition, rigor in the representation of knowledge is needed for meaningful interoperability and the exchange and reuse of EHR data.³⁷

Health information that is stored electronically can be retrieved in a meaningful way if it is mapped to standard terminologies. Terminologies are the basic words or terms that have specific meanings that represent concepts and can be used to gauge the health of individuals, groups and populations.³⁸ According to the Healthcare Information and Management Society (HIMSS), “semantic interoperability is the ability of two or more systems or elements to exchange information and to use the information that has been exchanged.”²⁷ Semantic interoperability leverages data exchange structures and data codification, including vocabulary, which enables the receiving IT systems to interpret the data. When this level of interoperability is achieved, it permits disparate EHR systems, such as business-related information systems, medical devices and mobile technologies, to share data and impact health and healthcare delivery.²⁷

Within SNOMED CT (Systematic Nomenclature of Medical and Clinical Terms (http://browser.ihtsdotools.org/)) International Edition, there is only one HL concept, health literacy enhancement, which is a regime/therapy code (SCTID: 430253004) associated with learning facilitation. In the US Edition, there is an HL concept for an assessment to document the procedure (SCTID: 431531000124101). Yet, no concepts exist for mapping the interpretation of an HL assessment in SNOMED CT nor in LOINC (an international standard for tests, measurements and observations). Without the ability to map an interpretation of an HL assessment to a standard terminology, the scored HL categories derived by an instrument’s cut points cannot be retrieved. Although there are concepts that represent literacy-level findings, which include fully literate literacy level (SCTID: 707842005) and illiterate literacy level (SCTID: 707843000), both are based upon social or personal history but are not derived from a valid and reliably scored instrument. The other related SNOMED CT concept related to literacy findings is literacy problems (SCTID: 161138004) and describes an educational problem acquired through history taking versus an HL assessment using an instrument. Clearly, there is a deficit in the ability to capture, store and retrieve an HL assessment in the EHR. This article describes the results of a literature review of HL assessment instruments with the goal of formulating semantically interoperable concepts that may be used to store the interpretation of an HL assessment in the EHR, which, in turn, would facilitate clinical decision support.

Methods

The objectives of this literature review were to (1) confirm HL assessments frequently used in a clinical setting that are scored and have documented cut points that delineate categories of HL, (2) assess how the composite assessment score is converted into HL categories, (3) translate categories of HL into interoperable concepts appropriate for storage and use in the EHR and (4) recommend concepts for submission to SNOMED CT. Healthcare and Health IT are quickly evolving industries that require incorporating recent practice into solution development; therefore, literature searches were limited to papers published between 2005 and 2016. Using PubMed, CINAHL and PsychINFO search engines, three HL key words were searched including HL assessment, HL assessment tool and HL assessment instrument, which yielded 1952, 130 and 252 articles, respectively.

The review methodology followed a three-step process that involved filtering for appropriate terminology first by title, second by abstract and third by a full-text review. Peer-reviewed articles that included a valid measure of health literacy, encompassed instruments designed for an adult population (aged 18 years and older), were written and tested in English, were scored and had cut points defining categories of HL and had been used in a clinical setting and were reviewed. Instruments that assess health numeracy only were excluded in addition to opinion pieces, editorials or letters to the editor. Any articles that addressed a specific clinical setting or condition (e.g. nutrition, oral or condition-specific, such as cancer) were excluded. Of particular interest were publications involving comparison of HL assessments and systematic literature reviews, which were hand-searched and followed the reference list of relevant articles.

Results

After consideration of exclusion criteria, 28 articles were retained for full review and tabulation. Nearly 43.0% of the articles were literature reviews: nine were systematic reviews, one critical review, one integrative review and one environmental scan. The results of the literature review are shown in Table 1, which describes the number of HL instruments cited within each article reviewed and the percent representation of each HL instrument (N = 108).

Table 1.

Frequency of instrument citation in literature review.

	BHLS	BRIEF	HALS/IALSS	HLSI	METER	NVS	REALM	REALM-R-SF-7	SILS	TILS	S-TOFHLA	TOFHLA	Total
Kilfoyle et al.^39,a	×					×	×	×			×
Mackey et al.⁴⁰						×	×			×	×	×
Duell et al.^41,a			×		×	×	×		×	×	×	×
Guzys et al.^42,b						×	×	×				×
De Oliverira, et al.^43,a						×	×		×		×	×
Kiechle^30,a		×							×
Mantwill et al.^44,a	×		×		×	×	×	×		×	×
Smith, O’Conor, et al.⁴⁵						×	×					×
Altin, et al.^35,a				×		×
Haun, et al.^46,c					×	×	×	×	×	×		×
Sand-Jecklin and Coyle⁴⁷	×										×
Carpenter et al.⁴⁸						×	×	×	×		×
Nemmers and Milagros³⁶						×	×	×	×	×	×	×
Stagliano and Wallace⁴⁹	×					×
Collins et al.^50,a						×	×	×	×	×	×	×
Wolf et al.⁵¹						×	×					×
Berkman et al.^16,a						×	×		×			×
Zamora and Clingerman^52,a							×				×
Patel et al.⁵³						×					×	×
Haun, et al.⁵⁴		×					×				×
Mancuso^55,d						×	×		×		×
Rawson et al.⁵⁶					×		×
Morris et al.⁵⁷									×		×
Wallace et al.⁵⁸							×		×
Paasche-Orlow et al.⁵⁹							×				×	×
Weiss et al.⁶⁰						×						×
	4	2	2	1	3	19	21	8	12	5	16	15	102
Percent representation	3.9	2.0	2.0	1.0	3.9	17.6	18.6	6.9	10.8	5.9	14.7	12.7	100

BHLS: Brief Health Literacy Screener; BRIEF: Brief Health Literacy Screening Tool; HALS/IALSS: Health Activity Literacy Scale/International Adult Literacy and Skills Survey; HLSI: Health Literacy Survey Instrument; METER: Medical Term Recognition Test; NVS: Newest Vital Sign; REALM: Rapid Estimate of Adult Literacy in Medicine; REALM-R-SF-7: Rapid Estimate of Adult Literacy in Medicine-Revised-Short Form-7; SILS: Single Item Literacy Screen; TILS: Two-Item Literacy Screen; S-TOFHLA: Shortened Test of Functional Health Literacy in Adults; TOFHLA: Test of Functional Health Literacy in Adults.

Systematic review.

Critical review.

Environmental scan.

Integrative review.

Figure 1 visually describes the most frequently cited instruments, which included the TOFHLA/S-TOFHLA (28.4%), the REALM/REALM-R (25.9%), the NVS (17.6%) and SILS (11.8%). Each of the remaining instruments represented 5.0% or less of the instruments cited.

Figure 1.

Percent Representation of Frequently Cited Health Literacy Assessment Instruments.

Instruments

TOFHLA

As an objective, HL assessment, the TOFHLA, is composed of both numeracy and literacy items that assess comprehension of information (e.g. a prescription label, an appointment slip, a chart describing eligibility for financial aid, example results from a medical test), whereas the reading assessment applies the cloze procedure in which every fifth to seventh word of a text string is missing and must be supplied by the participant. Scores are classified as low, marginal or adequate HL.⁴⁶ The S-TOFHLA, an abbreviated version of the TOFHLA, requires 7–12 min versus up to 22 min to administer.

REALM

The REALM, a word-recognition test containing 66 medically relevant words, assesses HL by having the individual pronounce a list of words that are increasingly difficult. This instrument is highly correlated with the TOFHLA and S-TOFHLA. HL scores are classified as low, marginal or adequate.⁴⁶ The REALM-R (revised)⁶¹ and REALM-SF (Short Form) are abbreviated forms of the REALM.⁶²

Brief item screeners (single- to four-item screens)

The SILS,⁵⁸ developed in 2006 using a population of Veterans Affairs patients, comprises three questions using a 5-point Likert scale to identify patients with low HL. More specifically, items assess confidence in filling out medical forms, assistance in comprehending hospital materials and perceived difficulty understanding written information when attempting to learn more about a medical condition.^33,63 The Two-Item Literacy Screen (TILS) poses two questions, “What was the last grade you completed?” and “Can you estimate your reading ability with one of the following: I frequently read complete books, I read the newspaper, I occasionally need help with the newspaper, or I frequently need help with the newspaper.”⁶³

Brief Health Literacy Screening Tool

The Brief Health Literacy Screening Tool (BRIEF), an extension of the brief screening questions (BSQ),⁶⁴ adds an oral HL question (difficulty understanding what is told to patient) to create a four-question scale. The SILS, also referred to as the Short Literacy Survey (SLS) and Brief Health Literacy Screen (BHLS),⁴⁸ has been referred to as a “single-item literacy test.”^49,55,65 These instruments use adequate, marginal and inadequate HL for composite scoring.

NVS

The NVS (English version) is a brief, objective, HL assessment that involves reading a nutrition label and a label on a container of ice cream, which is followed by six questions to assess comprehension.⁶¹ The 6-point scale is scored into three levels, high likelihood (50% or more) of limited literacy, possibility of limited literacy and adequate literacy.^46,49

METER

The METER, a self-administered instrument patterned after the REALM, includes 40 medical words and 40 non-words and requires only 2 min to complete. The patient is asked to mark only those items they recognize as actual words. Categories of HL using the METER are low, marginal and functional.⁵⁷

Health Literacy Survey Instrument

The Health Literacy Survey Instrument (HLSI) and HLSI-SF (Short Form) assess HL through self-assessment as well as print, oral, quantitative and Internet-based information-seeking skills. The full instrument contains 25 items, while the shortened version contains 10 items and requires 5–10 min to complete. HL scores are captured in three categories, proficient, basic and below basic literacy.⁶⁶

Health Activity Literacy Scale

The Health Activity Literacy Scale (HALS) was developed in the United States but has been used in many countries including Canada, Australia and Switzerland (International Adult Literacy and Skills Survey (IALSS)).⁶⁷ As a more comprehensive and somewhat lengthy assessment (191 items), the HALS distinguishes health-related abilities among five domains (health promotion, health protection, disease prevention, healthcare and maintenance and systems navigation) and between different health literacy tasks and skills. The HALS scoring ranges from 0 to 500 and reflects a progression of health-related literacy skills ranging from levels 1 to 5. Individuals scoring at or above 276 have higher health literacy versus below 276 having low health literacy.⁴⁷

Categories of health literacy

Figure 2 displays the scored HL categories derived by cut points of each instrument. The majority of instruments (9 of 10) use three categories to define HL. Of those 10 instruments, 7 (70.0%) use low/inadequate, marginal, adequate/functional; 1 (10.0%) uses proficient, basic, below basic; and 1 (10.0%) uses limited, possibly limited, adequate (10%). One instrument (10%) uses a dichotomous categorical description, high versus low HL. The three-category HL instrument labels were similar and could be considered as representing comparable concepts. “Proficient” or “Functional” could be interpreted as “adequate” health literacy, “possibly limited” or “basic” HL as “marginal” HL and “below basic” or “limited” as “inadequate” HL. Likewise, low HL would align with “inadequate.” Using this logic, 9 of 10 HL assessments reviewed could be stored in the EHR as a concept, regardless of their scales or cut points, using the categories of inadequate, marginal and adequate HL. Additionally, dichotomous HL label, “at risk” of low HL, could use the labels “inadequate” and “adequate” to store the composite HL assessment score.

Figure 2.

Categories Capturing Health Literacy Assessment.

Discussion

HL is a critical factor in patient engagement²³ and has been identified as a core metric for health and healthcare progress.⁷ In the IOM report, Vital signs: core metrics for health and healthcare progress, individual engagement has been defined as an “index of personal involvement in health-related behaviors, self-care, caregiving, and social activities that reflect a personal health orientation.”⁶⁸ Moreover, the IOM has identified health literacy as one of the best current measures for individual engagement related to health and lifestyle choices.⁷

As previously suggested, the electronic retrieval of the interpretation the HL assessment score is needed to enhance the exchange of health information between patient and provider, as well as provider to provider. An HL assessment interpretation stored electronically and mapped to a standard terminology facilitates actionable knowledge across healthcare settings. Therefore, with semantic interoperability, the highest level of interoperability, HL assessment data can be exchanged across providers in different settings, with different devices, on different systems.

One of the goals of this literature review was to use an evidence-based methodology to create interoperable HL interpretation concepts appropriate for storage and use in the EHR. This review of the literature revealed that multiple HL instruments with established validity and reliability exist. The TOFHLA/S-TOFHLA and REALM/REALM-R were the most frequently reported instruments, which parallel previously reported findings.⁶⁰ Screening instruments with single, two or three items were represented as well. Wallace et al.⁵⁷ suggest that one screening question may be sufficient for detecting limited and marginal HL skills in clinic populations. To be practical in a busy clinical setting, an ideal self-report measure for HL would (1) rapidly and correctly identify patients with limited HL, (2) be readily administered with minimal training, (3) pose marginal discomfort to patients and (4) be acceptable to patients.⁴ Additionally, assessments in a clinical setting may be used for triage, enabling providers to identify patients at greatest risk and for allocating resources to those patients.⁴

HL is part of a person-centered care process and essential to the delivery of cost-effective, safe and high-quality health services. Healthcare assessments, whether laboratory, radiological, physical, mental or otherwise, are performed to trigger activity. Assessment drives clinical decision-making and, in a healthcare setting, informs which intervention is suitable based upon assessment information. However, the average clinical setting poses challenges to assessing and addressing HL deficits. Despite the important implications of limited HL for patient care, healthcare providers are often unaware of patients’ reading abilities⁶² or misjudge the HL of their patients.⁶⁹ Shorter and less stigmatizing screening tools would help to remove barriers⁷⁰ and make widespread screening implementation in a resource and time-limited practice setting possible.⁷¹

Many instruments measure similar aspects of HL. In their study, Paasche-Orlow et al.³² report that rates of low HL were comparable, regardless of the instrument used. Additionally, Kiechle et al.³³ indicated that the instruments employed in their study were significantly correlated. Therefore, the majority of HL instruments, regardless of scoring mechanisms, could be stored in the EHR using a three-concept solution of inadequate, marginal and adequate HL, which fully supports semantic interoperability needs across the patient’s care spectrum.

Limitations

Despite following an evidence-based search strategy using MESH terms and key words, relevant literature may have been missed. Likewise, the exclusion of non-English instruments narrowed the review of instruments. Furthermore, instruments that did not provide scoring or cut points for categorization of HL were not included. However, employing systematic guidelines to perform the search and analysis of the articles included in this study provided the basis to identify possible HL concepts for use in the EHR.

Recommendations

Using the results of the systematic literature review, the following concepts were proposed for storing the interpretation of an HL assessment using SNOMED CT, as listed in Table 2. The parent concept Health Literacy (finding) would be related to HL level. The children of this concept would include inadequate HL level, marginal HL level and adequate HL level. The synonyms for inadequate HL would include limited HL level and below basic HL level, marginal HL level would include possibly limited HL level and basic HL level and adequate HL level would include proficient HL level or functional HL level. In the seminal publication, Health literacy: a prescription to end confusion, the IOM⁴ recommends,

health literacy assessment should be a part of healthcare information systems and quality data collection. Public and private accreditation bodies, including Medicare, the National Committee for Quality Assurance, and the Joint Commission on Accreditation of Healthcare Organizations, should clearly incorporate health literacy into their accreditation standards.¹

Table 2.

Proposed SNOMED CT concepts for the interpretation of a health literacy assessment.

Parent	Children	Synonym
Finding related to health literacy level	Inadequate health literacy level	Limited health literacy levelBelow basic health literacy level
	Marginal health literacy level	Possibly limited health literacy levelBasic health literacy level
	Adequate health literacy level	Proficient health literacy levelFunctional health literacy level

SNOMED CT: Systematic Nomenclature of Medical and Clinical Terms.

The assessment of HL has become increasingly important to effective, patient-centered care and to the practice of population health. HL assessment should be a part of the data collection process during a patient encounter and documented in the EHR. If the interpretation of an HL assessment is mapped to a standard terminology, it will be semantically interoperable, and therefore, useful across systems and settings. Currently, no concepts in a standard terminology exist to capture the interpretation of an HL assessment, and therefore, concepts in the clinical finding axis of SNOMED CT are needed. This review provides an evidence-based recommendation for the addition of concepts to store the interpretation of an HL assessment, regardless of the instrument used.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

References

Institute of Medicine (US) Committee on Health Literacy; Nielsen-Bohlman

Panzer

Kindig

(eds). Health literacy: a prescription to end confusion: executive summary. Washington, DC: The National Academies Press, 2004, p. 16.

Baker

DW.

The meaning and the measure of health literacy. J Gen Intern Med 2006; 21(8): 878–883.

Rudd

Objective 11–12: improvement of health literacy. In: Communicating health: priorities and strategies for progress. Washington, DC: Office of Disease Prevention and Health Promotion, US Department of Health and Human Services, 2003.

Institute of Medicine. State of the USA health indicators: letter report. Washington, DC: The National Academies Press, 2009.

Kutner

Greenberg

Jin

, et al. The health literacy of America’s adults: results from the 2003 national assessment of adult literacy (NCES 2006–483). Washington, DC: US Department of Education, 2006.

Goodman

Finnegan

Mohadjer

, et al. Literacy, numeracy, and problem solving in technology-rich environments among U.S. adults. Results from the program for the international assessment of adult competencies 2012: first look (NCES 2014–008). Washington, DC: US Department of Education, National Center for Education Statistics; 2013, https://nces.ed.gov/pubs2014/2014008.pdf (accessed 7 February 2017).

Institute of Medicine. Vital signs: core metrics for health and health care progress. Washington, DC: The National Academies Press, 2015.

US Department of Health and Human Services, Office of Disease Prevention and Health Promotion. Healthy People 2020. Washington, DC, https://www.healthypeople.gov/2020/topics-objectives/topic/health-communication-and-health-information-technology (accessed 20 July 2015).

Schillinger

Grumbach

Piette

, et al. Association of health literacy with diabetes outcomes. JAMA 2002; 288: 475–482.

10.

Kalichman

Ramachandran

Catz

Adherence to combination antiretroviral therapies in HIV patients of low health literacy. J Gen Intern Med 1999; 14(5): 267–273.

11.

Dewalt

Berkman

Sheridan

, et al. Literacy and health outcomes: a systematic review of the literature. J Gen Intern Med 2004; 19(12): 1228–1239.

12.

Berkman

Sheridan

Donahue

, et al. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med 2011; 155(2): 97–107.

13.

Baker

Parker

Williams

, et al. The relationship of patient reading ability to self-reported health and use of health services. Am J Public Health 1997; 87(6): 1027–1030.

14.

Weiss

Hart

McGee

, et al. Health status of illiterate adults: relation between literacy and health status among persons with low literacy skills. J Am Board Fam Pract 1992; 5(3): 257–264.

15.

Sanders

Federico

Klass

, et al. Literacy and child health: a systematic review. Arch Pediatr Adolesc Med 2009; 163(2): 131–140.

16.

Berkman

Sheridan

Donahue

, et al. Health literacy interventions and outcomes: an updated systematic review. Evid Rep Technol Assess (Full Rep) 2011; (199): 1–941.

17.

Griffey

Kennedy

D’Agostino McGowan

, et al. Is low health literacy associated with increased emergency department utilization and recidivism? Acad Emerg Med 2014; 21(10): 1109–1115.

18.

Baker

Parker

Williams

, et al. Health literacy and the risk of hospital admission. J Gen Intern Med 1998; 13(12): 791–798.

19.

Baker

Gazmararian

Williams

, et al. Functional health literacy and the risk of hospital admission among Medicare managed care enrollees. Am J Public Health 2002; 93(8): 1278–1283.

20.

Friedland

New estimates of high costs of inadequate health literacy. In: Proceedings of Pfizer conference “promoting health literacy: a call to action,” 7–8 October 1998, pp. 6–10. Washington, DC: Pfizer Inc.

21.

Vernon

Trujillo

Rosenbaum

, et al. Low health literacy: implications for national health policy. Report, Department of Health Policy, School of Public Health and Health Services, The George Washington University, Washington, DC, 2007.

22.

Institute of Medicine (IOM). Measures of health literacy: workshop summary. Washington, DC: The National Academies Press, 2009.

23.

America’s Health Insurance Plans. Health literacy and America’s Health Insurance Plans: laying the foundation and beyond (3rd edn), Report, November 2013. https://www.ahip.org/health-literacy-and-americas-health-insurance-plans-laying-the-foundation-and-beyond (accessed 10 October 2016).

24.

Barry

D’Eath

Sixsmith

Interventions for improving population health literacy: insights from a rapid review of the evidence. J Health Commun 2013; 18(12): 1507–1522.

25.

National Quality Forum. Health literacy: a linchpin in achieving national goals for health and healthcare. NQF Issue Brief, 13 March 2009, http://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=17654 (accessed June 10, 2016).

26.

Hibbard

Mahoney

Stock

, et al. Do increases in patient activation result in improved self-management behaviors? Health Serv Res 2007; 42(4): 1443–1463.

27.

Healthcare Information and Management Systems Society (HIMSS). What is interoperability? http://www.himss.org/library/interoperability-standards/what-is-interoperability (2016, accessed 12 January 2016).

28.

Engel

Heisler

Smith

, et al. Patient comprehension of emergency department care and instructions: are patients aware of when they do not understand? Ann Emerg Med 2009; 53(4): 454–461.e15.

29.

Baker

DW.

The meaning and the measure of health literacy. J Gen Intern Med 2006; 21(8): 878–883.

30.

Kiechle

Bailey

Hedlund

, et al. A systematic review of performance-based versus self-reported measures of health literacy and numeracy. J Gen Intern Med 2015; 30(10): 1538–1546.

31.

Williams

Parker

Baker

, et al. Inadequate functional health literacy among patients at two public hospitals. JAMA 1995; 274(21): 1677–1682.

32.

Paasche-Orlow

Parker

Gazmararian

, et al. The prevalence of limited health literacy. J Gen Intern Med 2005; 20(2): 175–184.

33.

Kiechle

Hnat

Norman

, et al. Comparison of brief health literacy screens in the emergency department. J Health Commun 2015; 20(5): 539–545.

34.

Oldach

Katz

ML.

Health literacy and cancer screening: a systematic review. Patient Educ Couns 2014; 94(2): 149–157.

35.

Altin

Finke

Kautz-Freimuth

, et al. The evolution of health literacy assessment tools: a systematic review. BMC Public Health 2014; 14: 1207.

36.

Nemmers

Milagros

Clinical assessment of health literacy. Topics Geriat Rehabil 2013; 29(2): 89–97.

37.

Monsen

Finn

Fleming

, et al. Rigor in electronic health record knowledge representation: lessons learned from a SNOMED CT clinical content encoding exercise. Inform Health Soc Care 2016; 41(2): 1–15.

38.

Coiera

Guide to Health Informatics. Boca Routon, FL: CRC Press, 2015.

39.

Smith

O’Conor

Curtis

, et al. Low health literacy predicts decline in physical function among older adults: findings from the LitCog cohort study. J Epidemiol Community Health 2015; 69(5): 474–480.

40.

Mackey

Doody

Werner

, et al. Self-management skills in chronic disease management: what role does health literacy have? Med Decis Making 2016; 36(6): 741–759.

41.

Duell

Wright

Renzaho

, et al. Optimal health literacy measurement for the clinical setting: a systematic review. Patient Educ Couns; 2015; 98(11): 1295–1307.

42.

Guzys

Kenny

Dickson-Swift

, et al. A critical review of population health literacy assessment. BMC Public Health 2015; 15: 215.

43.

De Oliveira

Jr. McCarthy

Wolf

, et al. The impact of health literacy in the care of surgical patients: a qualitative systematic review. BMC Surg 2015; 15: 86.

44.

Mantwill

Monestel-Umana

Schulz

PJ.

The relationship between health literacy and health disparities: a systematic review. PLoS ONE 2015; 10(12): e0145455.

45.

Smith

O’Conor

Curtis

, et al. Low health literacy predicts decline in physical function among older adults: findings from the LitCog cohort study. J Epidemiol Communty Health 2015; 69(5): 474–480.

46.

Haun

Valerio

McCormack

, et al. Health literacy measurement: an inventory and descriptive summary of 51 instruments. J Health Commun 2014; 19(Suppl. 2): 302–333.

47.

Sand-Jecklin

Coyle

Efficiently assessing patient health literacy: the BHLS instrument. Clin Nurs Res 2014; 23(6): 581–600.

48.

Carpenter

Kaphingst

Goodman

, et al. Feasibility and diagnostic accuracy of brief health literacy and numeracy screening instruments in an urban emergency department. Acad Emerg Med 2014; 21(2): 137–146.

49.

Stagliano

Wallace

LS.

Brief health literacy screening items predict newest vital sign scores. J Am Board Fam Med 2013; 26(5): 558–565.

50.

Collins

Currie

Bakken

, et al. Health literacy screening instruments for eHealth applications: a systematic review. J Biomed Inform 2012; 45(3): 598–607.

51.

Wolf

Curtis

Wilson

, et al. Literacy, cognitive function, and health: results of the LitCog study. J Gen Intern Med 2012; 27(10): 1300–1307.

52.

Zamora

Clingerman

EM.

Health literacy among older adults: a systematic literature review. J Gerontol Nurs 2011; 37(10): 41–51.

53.

Patel

Joel

Rovena

, et al. Testing the utility of the newest vital sign (NVS) health literacy assessment tool in older African-American patients. Patient Educ Couns 2011; 85(3): 505–507.

54.

Haun

Noland-Dodd

Varnes

, et al. Testing the BRIEF health literacy screening tool. Fed Pract; 26: 24–31.

55.

Mancuso

JM.

Assessment and measurement of health literacy: an integrative review of the literature. Nurs Health Sci 2009; 11(1): 77–89.

56.

Rawson

Gunstad

Hughes

, et al. The METER: a brief, self-administered measure of health literacy. J Gen Intern Med 2010; 25(1): 67–71.

57.

Morris

MacLean

Chew

, et al. The Single Item Literacy Screener: evaluation of a brief instrument to identify limited reading ability. BMC Fam Pract 2006 Mar 24; 7: 21.

58.

Wallace

Rogers

Roskos

, et al. Brief report: screening items to identify patients with limited health literacy skills. J Gen Intern Med 2006; 21(8): 874–877.

59.

Paasche-Orlow

Parker

Gazmararian

, et al. The prevalence of limited health literacy. J Gen Intern Med 2005; 20(2): 175–184.

60.

Weiss

Mays

Martz

, et al. Quick assessment of literacy in primary care: the newest vital sign. Ann Fam Med 2005; 3(6): 514–522.

61.

Bass

III Wilson

Griffith

CH.

A shortened instrument for literacy screening. J Gen Intern Med 2003; 18(12): 1036–1038.

62.

Arozullah

Yarnold

Bennett

, et al. Development and validation of a short-form, rapid estimate of adult literacy in medicine. Med Care 2007; 45(11): 1026–1033.

63.

Brice

Foster

Principe

, et al. Single-item or two-item literacy screener to predict the S-TOFHLA among adult hemodialysis patients. Patient Educ Couns 2014; 94(1): 71–75.

64.

Ohl

Harris

Nurudtinova

, et al. Do brief screening questions or provider perception accurately identify persons with low health literacy in the HIV primary care setting? AIDS Patient Care STDs 2010; 24(10): 623–629.

65.

McNaughton

Rothman

Storrow

, et al. Measuring numeracy and health literacy in the emergency department. Acad Emerg Med 2014; 21(8): 944–945.

66.

McCormack

Bann

Squiers

, et al. Measuring health literacy: a pilot study of a new skills-based instrument. J Health Commun 2010; 15(Suppl. 2): 51–71.

67.

Nutbeam

Defining and measuring health literacy: what can we learn from literacy studies?

Int J Public Health 2009; 54(5): 303–305.

68.

Blumenthal

McGinnis

JM.

Measuring vital signs: an IOM report on core metrics for health and health care progress. JAMA 2015; 313(19): 1901–1902.

69.

Ryan

Leguen

Weiss

, et al. Will patients agree to have their literacy skills assessed in clinical practice? Health Educ Res 2008; 23(4): 603–611.

70.

DeWalt

Broucksou

Hawk

, et al. Developing and testing the health literacy universal precautions toolkit. Nurs Outlook 2011; 59(2): 85–94. p. 1902.

71.

Green

Mor

Shields

, et al. Associations of health literacy with dialysis adherence and health resource utilization in patients receiving maintenance hemodialysis. Am J Kidney Dis 2013; 62(1): 73–80.

72.

Kilfoyle

Vitko

O’Conor

, et al. Health literacy and women’s reproductive health: a systematic review. J Women’s Health 2016; 25(12): 1237–1255.