Piloting an Educational Approach to Assess eHealth Literacy and Evidence-Based Medicine in Integrative Health: A Feasibility and Validation Study

Abstract

Introduction:

Low rates of online health literacy put consumers at risk of misinformation, but this could be mitigated through practitioner engagement. Integrative health (IH) crosses health care disciplines, so it is well-positioned to improve health information sharing. IH practitioners require evidence-based medicine (EBM) and electronic health literacy (eHL) competencies to make such impact. Several EBM assessments exist, but none are IH-specific. The Fresno Test of EBM FEBM is a validated, performance-based assessment used in medical education. We sought to assess feasibility of incorporating eHL and EBM assessments into graduate coursework while adapting and validating the FEBM for an IH audience (FEBM-IH).

Methods:

A pilot observational design was used to adapt, evaluate, administer, and validate the FEBM-IH. Revalidation of the FEBM-IH began with a discipline-focused adaptation, which was reviewed by an expert panel. The FEBM-IH was then administered to IH students and faculty. Independently scored assessments determined inter-rater reliability, internal consistency, item discrimination, and item difficulty. eHL assessments (eHEALS and General Health Numeracy Test-6) were also embedded in the online course.

Results:

Outcome completion rates suggest the FEBM-IH and eHL assessment tools are feasible to include in online courses, with 68.9% (102/148) eligible participants joining and 76.5% (78/102) completing all questions in all measures. The FEBM-IH demonstrated excellent assessor agreement (kappa = 0.97, p < 0.001), high internal consistency (α = 0.799), and acceptable item discrimination (0.26–0.68). Median self-perceived eHL scores increased from 30/40 to 33/40 points by course’s end, suggesting some increase in eHL.

Conclusions:

Tools were feasible to integrate; FEBM-IH maintains acceptable validity; and further exploration of the relationship between EBM and eHL is warranted.

Introduction

In an information and technology-based world, high-level competencies in electronic health literacy (eHL) (including language/numeric, informational, computer, health, scientific, and media literacies) are imperative for health care practitioners and consumers. Failure to demonstrate eHL—defined as the skills required to find, evaluate, and apply evidence found online to health care decision making—erodes the quality of health information published and shared online.^1
–3 A lack of understanding in scientific process results in mistrust of medicine.⁴ As a result, consumers of health information may turn to less credible information sources.

Integrative Health (IH) practitioners take a whole-person, multimodal approach to health care and collaborate with a variety of therapies, often incorporating traditional or complementary approaches. IH forms the crossroads where multiple disciplines within health care meet, which is an ideal space to combat medical misinformation online.^5
–7 IH practitioners with eHL competency would be better equipped to inform those they work with and serve as a trusted source to correct misinformation found online.

Increased interest in the field increases demand for both practitioners and research in IH.^6,8 As more IH practitioners join the health care industry, they’ll have opportunities to contribute to the online health information landscape. Evidence-based medicine (EBM) is a critical component of IH training where future practitioners should learn the skills to critically evaluate and apply evidence to practice and patient education.^9
–11 Including a validated evidence-based skills assessment within EBM courses helps to ensure that future IH professionals are meeting core competencies.^12,13

The Fresno EBM assessment tool (FEBM) was developed for standard biomedical education to assess student understanding and mastery of EBM skills such as the abilities to ask clinical questions, acquire relevant evidence, appraise the evidence found, and apply findings to practice when appropriate.¹³ Though the Fresno tool is validated for use in medical education, and primarily in the US, it has been adapted to be discipline-specific in several non-IH fields (e.g.,: nursing, physical therapy, and pharmacy education) and in locations outside of the United States.^13

–18 To date, there are no validated EBM skill assessment tools available specifically for IH professions.

With the advancement of internet capabilities, especially related to health and health care information access, we must recognize overlap between eHL and EBM competencies. IH practitioners need to be eHealth literate to maintain a current EBM practice.^1,19,20 Though medical texts and other offline resources help form a clinical knowledge base, they cannot keep pace with the growth of medical research available online.²¹ In fact, IH practitioners for years have cited online resources and databases as a facilitator for EBM practice.^22
–24

This study aimed to first assess the feasibility of incorporating assessments for both eHL and EBM into the IH classroom. In addition, we also sought to adapt and revalidate a commonly used assessment of EBM skills to an IH audience (FEBM-IH).

Methods

Study design

A pilot longitudinal descriptive design was employed in four phases to assess feasibility of using eHL and FEBM-IH assessments in graduate coursework and revalidate the revised FEBM to integrative health. This study was approved by the university Institutional Review Board (08.NAU.IS9.08.20.01) prior to initiation.

Phase I: Discipline focused adaptation

Validation of the FEBM-IH was assessed through reliability and validity measures that were guided by similar research.^13

–18 The initial format and revisions made to convert the FEBM to the FEBM-IH are detailed in the Supplementary Data S1. The study team consisted of faculty who teach EBM courses designed specifically for IH programs. Study team members with experience/expertise in IH approaches, such as nutrition, health promotion, yoga therapy, and herbal medicine, iteratively drafted scenario adaptations and met to discuss them until general group consensus was formed.

Phase II: Content validity

An expert panel of three nonstudy team faculty reviewed the FEBM-IH before its administration to establish content validity.^17,18 These faculty members held expertise that spanned IH fields, including herbal medicine, nutrition, acupuncture, and health coaching. Once the study team drafted initial adaptations, the panel was asked to review the changes and leave commentary for improvements that were then implemented prior to administration.

Phase III: Administration

During this phase, eHL and FEBM-IH assessments were administered online to students in an IH-specific research literacy course designed for IH programs. Feasibility was determined through completion rates of the course’s requisite and optional questions, as well as student consent to data use.

Assessments of eHL were given at two time points (start and end of the course). Throughout the course, students were given exams that included questions from the FEBM-IH. Rather than administering all FEBM-IH questions at a single time point, questions were distributed throughout unit exams.

Phase IV: Validation of FEBM-IH

The last phase of this study consisted of testing the revised FEBM-IH for validity and reliability. The course instructors scored according to the FEBM-IH rubrics. After each assessment was scored, student data was exported from the learning platform and secured for de-identification by research personnel with no role in the course.

Participants and raters

University faculty that taught research courses were invited to complete the revised FEBM-IH through an anonymous online form. Email requests were sent out with the link to the online form version of the FEBM-IH with instructions for completion.

All students in the online version of a graduate-level research literacy course (January 2021–December 2021) were invited to participate regardless of age, gender, or location. Student participants were limited to enrolled graduate students (over the age of 18 who could read and communicate in English). As of 2019, the student population was majority female (90%), under the age of 50 (79%), and 33% minority race/ethnicity.

Any student who enrolled in the online version of the course during this time was eligible for inclusion, irrespective of course outcome or completion. All students were given the same resources and assessments. Regardless of consent, all students were expected to participate in the same assessments and meet the same standards to pass the course. The instructors were blinded to student participation in the study.

Raters consisted of two lecturers with over 5 years of experience designing, teaching, and tutoring students through EBM coursework. Both raters were study team members and took part in the adaptation phase of the study.

Outcome measures

One goal of this study was to assess the feasibility of incorporating both eHL and EBM tools into a graduate level course. Feasibility was determined by completion rates of three tools: the FEBM-IH, the eHealth Literacy Scale(eHEALS), and the General Health Numeracy Test-6 (GHNT-6).^1,25,26 The combined eHEALS and GHNT-6 formed our eHL assessment.

FEBM is a 12-question performance-based evaluation designed for medical education settings. FEBM measures a range of skills using open-ended questions scored by standardized grading rubrics.^12,13 The FEBM has been adapted and validated for multiple biomedical fields, these guided the analyses performed here.^15

–18 The FEBM has shown good inter-rater reliability (IRR), internal reliability, and construct validity when used to evaluate the effectiveness of EBM instruction.^12,13

The eHEALS has been validated in multiple languages and consists of 8-scored questions. Using a 5-point Likert scale, respondents can choose ratings from “strongly agree” to “strongly disagree” for each question.²⁵ These are multiplied by each other to produce a score range of 5–40 points when summed. Though eHEALS does not have the same skills-based components as similar tools, it has often been compared with them with a high degree of alignment, and maintains one of the lowest costs in respondents’ time and effort.^20,27

–30 Higher scores on the eHEALS indicate a higher self-perceived aptitude for eHL.

The eHEALS doesn’t specifically address the eHL facet of traditional literacy, so we also elected to include the GHNT-6. The GHNT-6 is a 6-question assessment that evaluates health-related numeracy skills, which are relevant to EBM practice. First published in 2013, the GHNT-6 has been validated in a U.S. population and assessed for validity and reliability. Higher scores on the GHNT-6 indicate a higher degree of health numeracy.²⁶

Analysis

Feasibility

The methods used for evaluating feasibility included enrollment in the course, completion of outcome measures, and consent for data use. Student enrollment and participation ultimately determined the final sample size obtained. Similar research suggested that approximately 65% of students were likely to consent to participate, so this was used as our threshold for establishing feasibility.³¹ General feasibility results are presented as counts and percentages along with the outcome measure results themselves, presented as descriptive summary statistics, using means, medians, standard deviation, and interquartile range (IQR) as appropriate.

Validation

To determine IRR, evaluators independently scored a sample of 10 FEBM-IH tests, which were compared to obtain intra-class correlations (ICCs). ICC scores range from 0 to 1, with larger scores indicating higher consistency.^16
–18

Cronbach’s alpha was used to assess internal consistency. An acceptable range of internal consistency is achieved with Cronbach’s alpha between 0.7 and 0.95.^16
–18 Item difficulty was assessed according to pass rates. On the original FEBM, passing scores were defined as achieving an average “strong” score (equivalent to a score of 72% or more of the points earned) for each item.¹⁷ For item discrimination index, each FEBM-IH question’s total scores were separated into quartiles. The proportion of participants in the bottom quartile was subtracted from the proportion of participants in the top quartile. From this, an item discrimination index was formed, with a score greater than 0.2 being considered acceptable.¹⁶

To assess construct validity and understand any differences between novice and expert scores, student and faculty mean overall scores were to be compared using a t-test. It was expected that, if valid, expert scores would be significantly higher than novice scores.^16
–18

Changes to initial protocol

This study was initially planned to run between late 2019 and early 2020. The COVID-19 pandemic required unforeseen adaptations. We altered the periods of data collection from one to three trimesters for students and added a second round of query for faculty. The second round of query only yielded 10 submissions. Faculty cited time for completion and increased workload as reasons for nonparticipation. The resulting lack of data meant the planned construct validity analysis could not be completed.

Missing data

The analysis included all scores in the last observation carried forward approach to account for those with incomplete data. Students withdrawing from the course before completion were considered incomplete data since they no longer were included in the course roster for assessments that took place after they left.

Results

Participants

In total, 117 of 148 (79.0%) potentially eligible students consented to data use. Of those 117 students, 15 did not provide course data, bringing the total participants to 102. (Fig. 1).

FIG. 1.

Study Flow Diagram showing the number of the students involved at each timepoint.

As Table 1 shows, 43.1% of student participants taking this course in the fall, while 22.5% and 34.3% were in the summer and spring trimesters, respectively. Of the 102 students completing outcomes, 91 students (89.2%) cited English as their primary language. All students had English proficiency since this is a university admissions requirement. At baseline, 60.8% (n = 62/102) of participating students agreed that they felt confident interpreting data tables and graphs. Finally, 82.4% (n = 84/102) felt confident in their study skills as online learners at baseline.

Table 1.

Participant Characteristics at Baseline

	Baseline
n	102
Trimester n (%) response rate	102 (100.0)
Fall	44 (43.1)
Spring	23 (22.5)
Summer	35 (34.3)
Primary language for communication (written/speaking/reading) n (%) response rate	100 (98.0)
NonEnglish primary language	9 (8.8)
English only	78 (76.5)
English primary language, plus other fluencies	13 (12.7)
Felt comfortable working with numbers and reading quantitative tables or graphs n (%) Response Rate	102 (100.0)
Strongly disagree/disagree	23 (22.6)
Unsure	17 (16.7)
Agree/strongly agree	62 (60.8)
Felt confident in study skills as an online learner n (%) response rate	101 (99.0)
Strongly disagree/disagree	5 (4.9)
Unsure	12 (11.8)
Agree/strongly agree	84 (82.4)

Feasibility

Of the 148 students who enrolled in the course, 102 (68.9%) both produced course data and consented to its use in analysis, meeting the 65% threshold for general feasibility of integration based on the total number of enrolled students (Table 2). Of those who consented to use of their coursework data, 86.3% (88/102) completed all eHEALS, while only 81.4% (83/102) completed the GHNT-6 at both time points. A majority of those who consented to participate (92/102, 90.2%) completed all FEBM-IH questions.

Table 2.

Feasibility, Outcome Measure Completion, and Scale Score Summary Statistics

	Baseline	Post
eHEALS response rate n (%) of 148	100 (67.6)	88 (59.5)
eHEALS response rate n (%) of 102	100 (98.0)	88 (86.3)
eHEALS score (median [IQR].)	30.00 (27.00, 32.00)	33.00 (31.00, 37.75)
GHNT response rate n (%) of 148	97 (65.5)	83 (56.1)
GHNT response rate n (%) of 102	97 (95.1)	83 (81.4)
GHNT score (median [IQR].)	5.00 (4.00, 6.00)	5.00 (4.00, 6.00)
FEBM-IH response rate n (%) of 148		92 (62.2)
FEBM-IH response rate n (%) of 102		92 (90.2)
FEBM-IH score out of 100% (median [IQR].)		86.11 (74.19, 92.47)
FEBM-IH score out of 216 points (median [IQR].)		186.00 (160.25, 199.75)

At baseline, eHEALS scores had a median of 30 (out of 40), with an IQR of 27–32. At post assessment, scores on the eHEALS increased to a median of 33, with an IQR of 31–37.75. The skill-based assessment, GHNT-6, saw the same median and IQR (5, 4–6 out of a possible 6) respectively, from baseline to post. Finally, the FEBM-IH scores saw an overall median of 86.11% (when converted to a percentage out of 100 points), which corresponded with 186/216 possible points. Similarly, the FEBM-IH scores were mostly spread between 74.19% (160.25 points) and 92.47% (199.75 points).

The initial protocol failed to consider the risk that faculty involvement would be lacking. A total of just 10 faculty chose to participate out of the 40 invited.

Validation

To adapt the FEBM-IH to the course structurally, the questions were divided and administered when the course content aligned conceptually. Since Question 1 of the FEBM-IH contained two different scenarios, it was split-up to ease the cognitive load and clarify scoring with the rubric. Rubrics were altered to align with the question adaptations; these can be obtained by contacting the corresponding author.

Content validity was established via faculty experts, who reviewed the questionnaire and offered feedback prior to administration. The final version administered, along with the original versions, can be found in the Supplementary Data S2. IRR between assessors rating the scores of 10 individuals was “excellent” using the two-way random effect models and “single rater” unit, kappa = 0.97, p < 0.001.³²

Internal consistency was estimated at α = 0.799, (Table 3) which fell within an acceptable range (between 0.7 and 0.95). Item difficulty was assessed according to pass rates.¹⁷ Questions 8 and 10 (which scored at 70% and 64% on average, respectively) scored below “strong” ratings. All other items scored at or above the “strong” passing score of 72%.

Table 3.

FEBM-IH Validation Internal Consistency, Item Difficulty, and Item Discrimination

Administered question	Mean score % of points earned	% Scoring “strong” or better	Item difficulty	Item discrimination
q1.1: PICO (Scenario 1)	86.19%	88.2%	0.86	0.39
q1.2: PICO (Scenario 2)	87.66%	89.2%	0.88	0.55
q2: Evidence-based resources	82.31%	87.3%	0.82	0.54
q3: Search strategy	89.3%	87.3%	0.89	0.51
q4: Study design	84.15%	83.3%	0.84	0.34
q5: Relevancy	83.29%	80.4%	0.83	0.56
q6: Validity	86.68%	88.2%	0.87	0.47
q7: Magnitude and significance	86.93%	89.2%	0.87	0.68
q8: Ratio calculations	70.14%	62.7%	0.70	0.47
q9: Clinical calculations	82.07%	70.6%	0.82	0.61
q10: Ratios and Confidence Intervals	64.17%	54.9%	0.64	0.39
q11: Study design	74.59%	67.6%	0.75	0.26
q12: Study design	81.7%	77.5%	0.82	0.41

IRR kappa = 0.97; Cronbach’s α = 0.799.

Fewer than 75% of the participants scored at or above “strong” for Questions 8 through 11. Item discrimination ranged between 0.26 (Question 11) and 0.68 (Question 7), which is considered acceptable.¹⁶ The mean overall scores of students and faculty could not be compared to establish whether there was a difference between expert and novice scores due to faculty participation outlined in the feasibility reporting.^16
–18

Discussion

Principal results

The eHL measures and FEBM-IH, were feasible to incorporate into the course. Percent completion exceeded the study goal of 65%. Participation in this study among IH graduate students in the course was high, with almost 70% of eligible students consenting to share their data and over 80% of consenting students completing all assessment items. Overall, students’ scores from baseline to post on the eHEALS increased, indicating that self-perceived eHL was higher after the course. Students generally scored well on the GHNT-6 in both assessment points too (median 83% converted score), further supporting their self-perceptions with measurable eHL numeracy skills.

FEBM-IH questions were well aligned with preexisting course content and were easily integrated within the EBM course. The adapted version of the FEBM-IH was deemed valid, with internal consistency falling within an acceptable range, as seen in prior validation studies of the original Fresno EBM tool (0.7–0.95).^14,16
–18 Apart from two items relating to statistical ratios, item difficulty was regarded as “strong” (0.72 or greater) by original FEBM standards. There may be several explanations for this. One such reason is that IH students enter graduate school with diverse backgrounds, while other health care professionals tend to begin their training with a stronger foundation in math and science. The source of this difference might be further explored with EBM assessment taking place before and after training.

Limitations

To protect student identity, participant demographics were not collected, so it is impossible to determine if the sample represents IH students more broadly. The questions were interspersed throughout the course to reduce the burden on students. Both factors could limit the application of these results. Due to insufficient faculty participation, we could not establish construct validity. The completion and scoring of the tool are both time-consuming, which may have contributed to these issues. Further revision to a short form version of the FEBM-IH, such as the short-version produced by Buchanan, et al. (2010) may be worth consideration.

One of the responses used for IRR calculations also contained multiple missing responses, which may have inflated the IRR. More conservative calculations were explored to address this concern. Even with reduction to nine participants, there was still “good” absolute agreement between the assessors, using the two-way random effect models and “single rater” unit, kappa = 0.87, p < 0.001. It should also be noted, that the two raters were part of the study team and had a history of collaborative teaching and grading.

The analysis was planned as a last observation carried forward, which allowed inclusion of data for those who skipped questions or did not continue in the course. Withdrawal and skipping questions became more frequent with the pandemic, so conservative estimates were also calculated using only complete cases (n = 92) for validity measures of internal consistency, item difficulty, and item discrimination. Internal consistency and item discrimination both dropped below acceptable ranges with this approach, but item difficulty did not deviate from the main results.

Conclusions

These findings suggest that not only is it feasible to incorporate such tools into online education, but that there may be some relationship between self-perceived eHL and EBM skills attainment.

The intersection of eHL and EBM skillsets should be further examined to identify whether these overlaps could inform EBM education in integrative health. Future research should consider whether eHL and/or EBM (or possibly their interaction) enhances the overall quality of online integrative health information sharing.

Footnotes

Acknowledgments

Thank you to Camille Freeman, Elizabeth Ahmann, and Alex York for serving as our expert faculty in content revisions. The authors would also like to thank the students of the course who agreed to share their data for the analysis. This work has been deposited on a preprint server for the Journal of Medical Internet Research (JMIR) Preprints: Nault D, Abatemarco A, Missenda M, et al. Piloting an Educational Approach to Assess eHealth Literacy and Evidence-Based Medicine in Integrative Health: A Feasibility and Validation Study (Preprint). 2024; doi: https://doi.org/10.2196/preprints.56928.. Available from:

Authors’ Contributions

D.N.: Conceptualization, methodology, formal analysis, investigation, data curation, writing—original draft, writing—review and editing, visualization, project administration. S.M.: Conceptualization, methodology, writing—original draft, writing—review and editing, visualization, supervision. A.A.: Conceptualization, investigation, writing—original draft, writing—review and editing. M.M.: Conceptualization, data curation, writing—original draft, writing—review and editing. C.C.-C.: Conceptualization, data curation, writing—original draft, writing—review and editing.

Author Disclosure Statement

The authors have no conflicts of interest to report.

Funding Information

No funding was obtained for this study.

Supplementary Material

Supplementary Data S1

Supplementary Data S2

References

Norman

. eHealth literacy 2.0: Problems and opportunities with an evolving concept. J Med Internet Res, 2011; 13(4):e125; doi: 10.2196/jmir.2035

Norman

, Skinner

. eHealth literacy: Essential skills for consumer health in a networked world. J Med Internet Res, 2006; 8(2):e9; doi: 10.2196/jmir.8.2.e9

Pluye

, El Sherif

, Granikov

, et al. Health outcomes of online consumer health information: A systematic mixed studies review with framework synthesis. J Assoc Inf Sci Technol, 2019; 70(7):643–659; doi: 10.1002/asi.24178

Nasr

. Overcoming the discourse of science mistrust: How science education can be used to develop competent consumers and communicators of science information. Cult Stud Sci Educ, 2021; 16(2):345–356; doi: 10.1007/s11422-021-10064-6

Seetharaman

, Krishnan

, Schneider

. The future of medicine: Frontiers in integrative health and medicine. Medicina (Kaunas), 2021; 57(12):1303; doi: 10.3390/medicina57121303

Witt

, Chiaramonte

, Berman

, et al. Defining health in a comprehensive context: A new definition of integrative health. Am J Prev Med, 2017; 53(1):134–137; doi: 10.1016/j.amepre.2016.11.029

Burns

, Cowan

, Fleming

. Breaking through the wellness noise with legitimacy. J Integr Complement Med, 2024; 30(9):814–816; doi: 10.1089/jicm.2024.0231

Song

, Ang

, Lee

. Increasing trends and impact of integrative medicine research: From 2012 to 2021. Integr Med Res, 2022; 11(4):100884; doi: 10.1016/j.imr.2022.100884

Dorsch

, Aiyer

, Meyer

. Impact of an evidence-based medicine curriculum on medical students’ attitudes and skills. J Med Libr Assoc, 2004; 92(4):397–406.

10.

Draaisma

, Maggio

, Bekhof

, et al. Impact of deliberate practice on evidence-based medicine attitudes and behaviours of health care professionals. Perspect Med Educ, 2021; 10(2):118–124; doi: 10.1007/s40037-020-00634-9

11.

West

, Jaeger

, McDonald

. Extended evaluation of a longitudinal medical school evidence-based medicine curriculum. J Gen Intern Med, 2011; 26(6):611–615; doi: 10.1007/s11606-011-1642-8

12.

Buljan

, Jerončić

, Malički

, et al. How to choose an evidence-based medicine knowledge test for medical students? Comparison of three knowledge measures. BMC Med Educ, 2018; 18(1):290–210; doi: 10.1186/s12909-018-1391-z

13.

Ramos

, Schafer

, Tracz

. Validation of the Fresno test of competence in evidence-based medicine. BMJ, 2003; 326(7384):319–321; doi: 10.1136/bmj.326.7384.319

14.

Buchanan

, Jelsma

, Siegfried

. Measuring evidence-based practice knowledge and skills in occupational therapy—a brief instrument. BMC Med Educ, 2015; 15(1):191; doi: 10.1186/s12909-015-0475-2

15.

Mccluskey

, Bishop

. The adapted Fresno test of competence in evidence-based practice. J Contin Educ Health Prof, 2009; 29(2):119–126; doi: 10.1002/chp.20021

16.

Tilson

. Validation of the modified Fresno test: Assessing physical therapists’ evidence-based practice knowledge and skills. BMC Med Educ, 2010; 10(1):38; doi: 10.1186/1472-6920-10-38

17.

Coppenrath

, Filosa

, Akselrod

, et al. Adaptation and validation of the Fresno test of competence in evidence-based medicine in doctor of pharmacy students. Am J Pharm Educ, 2017; 81(6):106; doi: 10.5688/ajpe816106

18.

Halm

. Evaluating the impact of EBP education: Development of a modified Fresno test for acute care nursing. Worldviews Evid Based Nurs, 2018; 15(4):272–280; doi: 10.1111/wvn.12291

19.

Chan

, Kaufman

. A framework for characterizing ehealth literacy demands and barriers. J Med Internet Res, 2011; 13(4):e94; doi: 10.2196/jmir.1750

20.

Giudice

, Bravo

, Poletto

, et al. Correlation between ehealth literacy and health literacy using the ehealth literacy scale and real-life experiences in the health sector as a proxy measure of functional health literacy: Cross-sectional web-based survey. J Med Internet Res, 2018; 20(10):e281; doi: 10.2196/jmir.9401

21.

Zhao

, Jiang

, Yin

, et al. Changing trends in clinical research literature on PubMed database from 1991 to 2020. Eur J Med Res, 2022; 27(1):95; doi: 10.1186/s40001-022-00717-9

22.

Leach

, Aucoin

, Cooley

. Canadian naturopathic doctor engagement, preparedness, and perceptions of evidence-based practice: A national cross-sectional study. J Integr Complement Med, 2022; 28(12):955–964; doi: 10.1089/jicm.2022.0637

23.

Snow

, Leach

, Clare

. Attitudes, skill and use of evidence-based practice among U.S. Western herbal medicine providers: A national survey. J Complement Integr Med, 2017; 14(1); doi: 10.1515/jcim-2015-0101

24.

Sullivan

, Leach

, Snow

, et al. Understanding North American yoga therapists’ attitudes, skills and use of evidence-based practice: A cross-national survey. Complement Ther Med, 2017; 32:11–18; doi: 10.1016/j.ctim.2017.03.005

25.

Norman

, Skinner

. eHEALS: The eHealth literacy scale. J Med Internet Res, 2006; 8(4):e27; doi: 10.2196/jmir.8.4.e27

26.

Osborn

, Wallston

, Shpigel

, et al. Development and validation of the General Health Numeracy Test (GHNT). Patient Educ Couns, 2013; 91(3):350–356; doi: 10.1016/j.pec.2013.01.001

27.

Neter

, Brainin

. Perceived and performed ehealth literacy: Survey and simulated performance test. JMIR Hum Factors, 2017; 4(1):e2; doi: 10.2196/humanfactors.6523

28.

Stellefson

, Paige

, Tennant

, et al. Reliability and validity of the telephone-based ehealth literacy scale among older adults: Cross-sectional survey. J Med Internet Res, 2017; 19(10):e362; doi: 10.2196/jmir.8481

29.

Britt

, Hatten

. The development and validation of the eHealth competency scale: A measurement of self-efficacy, knowledge, usage, and motivation. Tech Commun Q, 2016; 25(2):137–150; doi: 10.1080/10572252.2016.1149621

30.

Sudbury-Riley

, FitzPatrick

, Schulz

. Exploring the measurement properties of the eHealth literacy scale (eHEALS) among baby boomers: A multinational test of measurement invariance. J Med Internet Res, 2017; 19(2):e53; doi: 10.2196/jmir.5998

31.

Blom-Hoffman

, Leff

, Franko

, et al. Consent procedures and participation rates in school-based intervention and prevention research: Using a multi-component, partnership-based approach to recruit participants. School Ment Health, 2009; 1(1):3–15; doi: 10.1007/s12310-008-9000-7

32.

Koo

, Li

. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med, 2016; 15(2):155–163; doi: 10.1016/j.jcm.2016.02.012

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

0.13 MB