Teaching Evidence-Based Medicine at Complementary and Alternative Medicine Institutions: Strategies,Competencies,and Evaluation

Abstract

Background:

As evidence-based medicine (EBM) becomes a standard in health care, it is essential that practitioners of complementary and alternative medicine (CAM) become experts in searching and evaluating the research literature. In support of this goal, the National Institutes of Health (NIH) National Center for Complementary and Alternative Medicine (NCCAM) provided R25 funding to nine CAM colleges to develop individual programs focused on teaching EBM. An overarching goal of these research education grants has been to provide CAM faculty and students with the skills they need to apply a rigorous evidence-based perspective to their training and practice.

Methods/Results:

This paper reviews the competencies and teaching strategies developed and implemented to enhance research literacy at all nine R25-funded institutions. While each institution designed approaches suitable for its research culture, the guiding principles were similar: to develop evidence-informed skills and knowledge, thereby helping students and faculty to critically appraise evidence and then use that evidence to guide their clinical practice. Curriculum development and assessment included faculty-driven learning activities and longitudinal curricular initiatives to encourage skill reinforcement and evaluate progress.

Conclusion:

As the field of integrative medicine matures, the NIH-NCCAM research education grants provide essential training for future clinicians and clinician-researchers. Building this workforce will facilitate multidisciplinary collaborations that address the unique needs for research that informs integrative clinical practice.

Introduction

As public demand for integrative medicine has increased,¹ so has the need for collaboration between practitioners of complementary and alternative medicine (CAM) and conventional health care providers. Integrative medicine combines conventional biomedicine with complementary and alternative therapies to form coordinated treatment plans. A major challenge to integrative medicine is the ability of providers of CAM and conventional care to fully understand and appreciate each other's disciplines.

To promote better collaborative care, the National Institutes of Health (NIH) National Center for Complementary and Alternative Medicine (NCCAM) funded 15 CAM education grants at conventional health professions' schools from 2000 to 2006² by using an R25 education grant mechanism. This initiative was followed by nine R25 education grants at CAM institutions from 2005 to 2011 (Table 1). The first set of R25 grants emphasized knowledge of CAM disciples with the allopathic curricula. The second set of R25 grants, termed CAM Practitioner Research Education grants, was designed to increase the quality and quantity of research and evidence-based medicine (EBM) in the curriculum at CAM institutions and is the focus of the current study. Each institutional applicant met specific criteria for eligibility to apply for these 4-year R25 research education program project grants. NIH-NCCAM also required formal collaborations between CAM and conventional, research-intensive medical institutions to strengthen CAM practitioners' ability to critically evaluate biomedical literature and incorporate the best evidence in clinical decision making.

Table 1.

R25 Education Grants at Complementary and Alternative Medicine Institutions from 2005 to 2011

Institution City, state	Total enrollment (n)	Graduate and professional programs of study	Collaborating institution(s)
AT Still University Kirksville, MO	3624	Osteopathic medicine (n=680)^a	Penn State
Bastyr University Kenmore, WA	1018	Naturopathic medicine (n=409),^a acupuncture and Oriental medicine, nutrition, counseling	University of Washington
National College of Natural Medicine Portland, OR	570	Naturopathic medicine, acupuncture and classical Chinese medicine (n=570)^a	Oregon Health & Science University
National University of Health Sciences Lombard, IL	985	Naturopathic medicine, chiropractic medicine (n=646),^a acupuncture and Oriental medicine, massage therapy	University of Illinois Chicago, School of Public Health
Northwestern Health Sciences University Bloomington, MN	859	Chiropractic medicine, acupuncture and Oriental medicine, massage therapy (n=670)^a	University of Minnesota
Oregon College of Oriental Medicine Portland, Oregon	270	Acupuncture and Oriental medicine (n=225)^a	Oregon Health & Science University
Palmer College of Chiropractic Davenport, IA	2106	Chiropractic medicine (n=1102)^a, clinical research	University of Iowa and Thomas Jefferson University
University of North Texas Health Science Center Texas College of Osteopathic Medicine Fort Worth, TX	1200	Osteopathic medicine (n=685)^a	University of Texas Medical Branch at Galveston
University of Western States Portland, OR	519	Chiropractic medicine (n=462),^a massage therapy	Oregon Health & Science University

Enrollment figures included for R25 target programs only; undergraduate programs not included.

CAM disciplines cover a wide array of medical systems, including chiropractic, naturopathy, osteopathic manipulative medicine, Asian and Oriental medicine, and massage therapy, among others. Diagnosis and treatment in these fields have often relied on individual expertise and intuition and have been responsive to patients' cultural values.³ Training of medical providers has often been conducted through apprenticeships rather than formalized education.⁴ The R25 grants promoted the implementation or enhancement of an evidence-based approach to CAM medical education, including use of the evidence pyramid to incorporate traditional forms of evidence along with critical thinking skills and clinical best practices.

The incorporation of training in EBM is one of the most important advances in medical education of the past 20 years.⁵ Medical education at accredited CAM institutions is similar to that at conventional medical schools; basic science is taught early in training, followed by clinical education. The faculty consists of clinicians and basic scientists who have expertise in their subject area but may not be expert educators. Although traditional medical schools and graduate medical training programs have long struggled with how best to teach the principles of EBM,^6

–9 until recently less attention has been paid to EBM skills in CAM schools.¹⁰ However, the quality and quantity of research studies on CAM therapies have increased over the past 25 years, allowing CAM practitioners the opportunity to use this research base to guide an evidence-based approach to medical education.

Each R25 institution was free to define the level of proficiency expected of students in each generally accepted EBM competency category found in the literature. EBM training encompasses a wide range of competencies, including acquisition, appraisal, and application to clinical practice of scientific information; understanding of general research methods; and skills for the conduct of discovery research.^11,12 “EBM competencies” is used in this paper as an umbrella term to cover the breadth and depth of competencies that were ultimately incorporated into each of the R25 institution's curricula. Further, it is important to note that implementing a competency-based EBM curriculum requires addressing concepts in both the classroom and the clinic.¹³

The purpose of this paper is to describe the curricular and culture-changing strategies that emerged as best practice models across these nine R25-funded CAM colleges. It reviews common strategies for developing curricular goals, achieving curriculum revision, and evaluating EBM competency and learning outcomes, as well as some of their unintended outcomes. Individual institutions have described their specific curriculum and teaching strategies in other publications.^{14

–30}

Materials and Methods

Data Collection

An electronic survey was administered to principal investigators of the nine R25 education grants. The survey consisted of 36 closed- and open-ended questions. Follow-up questions were sent via email to clarify responses as needed. Data were compiled for review and content was analyzed for common themes among institutions.

This project was determined to fall under the category of exempt research by the institutional review board at the National College of Natural Medicine (IRB #091608B).

From 2007 to 2011, the nine principal investigators and co-investigators from collaborating conventional institutions held annual meetings in Bethesda, Maryland, to share programmatic initiatives, accomplishments, and challenges. The reports from those meetings were used to enrich the survey data.

Data analysis

The data were analyzed and grouped into four categories: needs assessments, curricular revisions, research and evidence-based practice competency assessments, and faculty development programs. A qualitative analysis was performed using three independent reviewers. This team identified the most successful strategies that the individual institutions used, in addition to the most substantial challenges they encountered.

Results

Needs assessment

Each institution began the project with a curriculum inventory and needs assessment to identify current knowledge, skills, and attitudes and behaviors of students, faculty, and administration (Table 2). Several institutions identified current institutional cultures highlighting research activity. Three of the institutions identified pre-existing strong partnerships with research-intensive universities. A few institutions identified “research,” “research utilization” and/or “use of evidence” in institutional mission statements and professional curricular competencies before the R25 initiatives.

Table 2.

Initial Strategies Used to Incorporate Evidence-Based Medicine into Medical Curricula

Needs assessment and gap analysis

Curriculum mapping process to establish optimum placement of research and evidence-based practice competencies, including faculty participation

Stakeholder participation in defining EBM/research outcomes or competencies

Presence of EBM/research champions (in addition to the project principal investigator and staff)

Input from curriculum development expert(s)

Use of a train-the-trainer model for faculty development

Alumni assessed for long-term behavior changes

EBM, evidence-based medicine.

Nonetheless, initial assessments revealed a lack of consensus on the definition of EBM and EBM process in the clinic. Faculty and students lacked knowledge of research and EBM processes. They also expressed the strong desire to maintain a professional health care provider focus versus “being researchers.” The needs assessments and gap analyses identified lack of research opportunities and exposure for faculty and students, lack of confidence in knowledge and skills, lack of time for the EBM process in both the curriculum and clinical practice, lack of research and evidence from the CAM perspective, and limited resources and support for teaching and using EBM (tutorials, training, library subscriptions, and access).

Some faculty members at CAM institutions already included research literature in their courses; however, they often did not use the hierarchy of evidence or focus on developing strong critical appraisal skills. Three institutions reported that basic science faculty members were unclear about how to integrate clinical relevance into teaching about research. Five institutions reported that many courses contained poor-quality research materials and non–peer-reviewed articles. Other faculty members, especially those in Chinese medicine programs, tended to use historical clinical texts as substitutes for research literature. Many faculty reported that they “used EBM” if they listed references on course materials. In fact, many of those references had been superseded by more current research or were never specifically incorporated into the curricular content.

Every institution noted opportunities to restructure competencies to incorporate research and EBM and to highlight or initiate EBM skill-building and knowledge learning in the curricula. The needs assessment often identified which courses and faculty members to target for grant initiatives. The needs assessment process also generated interest among faculty with no research involvement, who became curricular champions or early adopters, voluntarily inspiring change and capacity-building in the EBM competencies.

Curricular revision

Finding the optimum placement of research-related materials in the curriculum required extensive teamwork and administrative support. Onsite instructional design staff were available at only two of the nine R25 institutions. As the R25 grantees integrated into curriculum committees, research education initiatives were gradually accepted because other faculty became better informed about the R25 goals. This process created opportunities for culture change. One institution commented that it was more effective to let those who are enthusiastic about integrating research and EBM into their classroom just do it (and help them do it effectively) than to plan it out in sequential order across curricula and coordinate across courses. Librarians emerged as leaders in several institutions and supported student and faculty understanding and use of EBM resources. They also facilitated building and implementing systems for teaching EBM content.

As a result of the R25 initiative, the number of required research courses at the nine CAM colleges increased from eight courses in six institutions to 21 courses in eight institutions (Table 3). These courses all provided foundational skills in identifying and critically evaluating research literature. In addition to required courses, the R25 programs stimulated development of many elective courses, ranging from journal clubs to advanced statistics. Recognizing the importance of integrating the targeted learning objectives or competencies throughout the curriculum, institutions generally identified two key strategies for success: (1) Course content was conducive to reinforcing EBM competencies using spiral learning strategies, and (2) faculty were willing to learn and teach EBM skills. Through use of these strategies, the overall percentage of curricular offerings, which included EBM content, rose substantially in eight of the nine institutions (Table 2).

Table 3.

Research and Evidence-Based Medicine Curriculum

	R25 Institution
Variable	1	2	3	4	5	6	7	8	9
No. of required research courses before R25	0	2	1	1	1	1	2	0	0
No. of required research courses following R25	2	3	2	4	3	2	4	0	1
No. of elective research courses before R25	2	0	0	0	0	0	0	0	0
No. of elective research courses following R25	5	0	2	5	0	0	0	6	3
Curriculum that includes EBM or research content before R25 (%)	None	≤10	51–60	21–30	None	≤10	≤10	≤10	≤10
Curriculum that includes EBM or research content following R25 (%)	60–70	30–40	80–90	50–60	40–50	20–30	50–60	≤10	50–60

Early in the project, the R25 institutions began to define targeted learning outcomes and competencies, beginning with research literacy. To infuse EBM principles and practices into the curriculum, all R25 institutions reported including the competencies shown in Table 4. One institution described creating a specific list of competencies with the cooperation of research or EBM course instructors. This allowed them to “rapidly see what we were already teaching versus what our ‘ideal curriculum’ required.”

Table 4.

Individual and Shared Evidence-Based Medicine Competencies

Shared competencies

Articulate an answerable clinical question

Effectively gather information, including conducting an evidence‐based literature search

Critically evaluate scientific literature and different forms of evidence

Synthesize and interpret the evidence

Effectively apply relevant evidence to clinical practice

Individual school competencies

Understand, define, and discuss the importance of research and evidence-informed practice and describe current research in one's field

Describe fundamental principles of research, including study designs, sources of bias, and concepts of biostatistics

Be familiar with the principles and regulations of human subjects protection in the conduct of research and maintain ethical standards of practice

Test and contextualize new insights against a variety of complex systems and theories of medical care

Effectively use evidence-informed to communicate with patients, other health care providers, third-party payers, and others

Engage in personal and professional development and self-evaluation or reflective practice

Be able to describe nonresearch forms of evidence and differentiate which information to consider as evidence

Participates in the culture of research and contributes to the research base

Some institutions established additional research literacy competencies to explicitly reflect the layers of knowledge involved in mastering EBM skills. For example, one institution included both critical appraisal skills and biostatistics expertise as separate competencies. Other institutions assumed an understanding of research principles and biostatistics concepts as part of the critical appraisal process and did not specify them as separate competencies. One institution emphasized research as different from EBM and increased the opportunities for students to participate in clinical research to strengthen their grounding in how scientific evidence emerges.

Although one school integrated research and EBM concepts only into existing clinical courses, the other eight introduced EBM concepts early in the form of stand-alone foundational courses. Online resources were used at most institutions for augmenting lecture materials, thereby permitting students and faculty to acquire additional explanations of certain specialized EBM skills. These included microskills modules, online discussion boards, and PubMed learning groups tailored to specific clinical courses. At all institutions, acquired competencies were later reinforced at a higher level in advanced courses through case studies, journal clubs, and clinical experience. Introducing EBM competencies at each developmental stage reinforced learning and helped students move through knowledge acquisition, knowledge application, and demonstration of competence.

Learning outcomes

Faculty reported using a wide variety of EBM assessments and outcome measures to evaluate learning activities (Table 5).^21,27,31 Many of the measures were aimed at simple knowledge acquisition (e.g., quizzes, exams, and grading rubrics). Less commonly used measures targeted the students' ability to synthesize information. Creative approaches to evaluation included, for example, the development of best-evidence tables as one-page summaries of the best available evidence related to a clinical topic that answered a specific, patient-based, clinical question. Preclinical and comprehensive exams were also aimed at assessing higher levels of knowledge.

Table 5.

Evidence-Based Medicine Activities and Assessments

Learners	EBM activities	Method of assessment	Outcomes measured
Students	✓ Lectures ✓ Readings ✓ Small or large group discussions ✓ Interactive exercises ✓ Projects or assignments ✓ Online tutorials ✓ Journal clubs	✓ Quizzes ✓ Exams in courses ✓ Projects ✓ Simulated patients ✓ Grading rubrics ✓ Comprehensive exams ✓ Preclinical exams	✓ Knowledge ✓ Skills ✓ Attitudes ✓ Confidence ✓ Behaviors
Alumni		✓ Competency-based exit interview✓ Survey	✓ Behavior change

EBM, evidence-based medicine.

In addition to integrating EBM concepts into the classroom, five institutions also integrated them into clinical training. At these sites, students performed literature searches specific to clinical cases before, during, or after clinic shifts. Furthermore, when early adopters modeled the use of EBM practices in the clinic, other faculty and students reported learning from these early adopters. Strategies specific to clinical learning involved referencing literature summary databases and using research literature in making clinical decisions. Learning outcomes focused on research literacy and the critical appraisal and judicious use of evidence to guide clinical practice. Clinical journal club exercises also enhanced EBM skills. These exercises were valuable because they allowed students to practice developing questions and use evidence to support clinical decisions, but they also were useful to instructors as evaluation tools.

The R25 institutions agreed that three key domains for assessing EBM training should be (1) acquisition of knowledge, (2) acquisition of skills, and (3) changed attitudes about EBM. Several institutions used published, validated EBM assessment instruments, such as the Fresno or Berlin scales.^32,33 To avoid “assessment fatigue,” many institutions added EBM skills measures to other assessment tools already in use at the institution.

At several institutions, students and faculty self-reported a lower degree of knowledge after EBM or research training than they had reported before the training, possibly because the training made them aware of the limitations of their previous knowledge.³⁰ As one investigator noted, “Faculty don't know what they don't know. Until they got a chance to really talk about research with their peers and weigh different perspectives, they really didn't realize some of the things they were missing.” This is similar to what has been previously described regarding self-perceived EBM competence with medical students.³⁴

Faculty development programs

All institutions recognized the need for faculty development as a key strategy for successful culture change and curricular revision. As shown in Table 2, a common approach used for faculty development was a train-the-trainers model, in which key faculty members (often called “early adopters”) learned methods for teaching research literacy and EBM principles and practices. In turn, these faculty members trained others interested in becoming involved in the project. Faculty at some institutions conducted workshops or assumed the role of mentor in a peer-to-peer team process. While two sites continued to focus on training their entire faculty, most institutions eventually adopted a train-the-trainers model and reported it to be more effective. Five of the R25 projects used offsite EBM resources for faculty development, such as having planners and faculty attend week-long intensive courses at Oxford University, McMaster University or the Rocky Mountain Workshop on How to Practice Evidence-Based Health Care to refine competencies and teaching methods. One institution reported early adopters “packing up the university van and taking other faculty on a two day field trip to an EBM course.”

Unintended benefits

Adding research literacy and EBM competencies to the curricula at these CAM colleges has yielded a variety of unintended benefits, the most commonly reported being changes in the institutional culture. Because CAM historically has not had a substantial evidence base, introducing EBM principles exposed students and faculty to unfamiliar territory. Adding EBM and research content to the curriculum required technical assistance in curriculum development, as well as a change in perspective about the role of research in CAM medical education. This change is reflected in many of the unintended benefits, or “ripple effects,” reported on the CAM campuses. In some situations synergy developed between basic and clinical science faculty. For example, one institution developed a “case study initiative” that substantially increased the number of case studies written by their faculty for use in the classroom, thereby increasing research publications. Another site developed a case study elective course, co-taught by research and clinical faculty, that has students writing and publishing case studies. Another institution reported that affiliated colleges within their university requested that CAM faculty teach clinical research literacy skills in guest lectures.

In addition to spontaneous faculty-driven initiatives, students at some institutions took the initiative of developing their own research clubs. Other R25 institutions saw an increase in the number of students pursuing dual-degree masters or doctoral programs. One R25 institution established a new optional Masters of Science degree in integrative medicine research.

Discussion

This paper identifies emerging best practices for curricular and culture change at CAM institutions aimed at improving future practitioners' and current faculty skills in using principles and practices of EBM. The challenge of implementing an EBM program is not unique to CAM institutions. Schools of nursing, dentistry, podiatry, physical therapy, and other health professions are also working to incorporate EBM principles and practices.^35

–38 There is widespread agreement that EBM competencies include five skills: (1) asking an answerable clinical question, (2) accessing information, (3) appraising the evidence, (4) applying evidence to clinical practice, and (5) assessing the results as effective outcomes and for quality improvement.^11,13 Generally the published instructional designs for teaching EBM focus on teaching practicing professionals how to use specific elements in the EBM process, including asking a clinical question or critical appraisal,³⁹ and measuring the mastery of specific knowledge and skills.^40,41 Few graduate health professions programs have incorporated all five competencies into the curriculum.

This study reports four aspects of curricular revision and cultural change: curricular revision, learning outcomes, faculty development, and unintended benefits. By examining approaches taken at nine separate institutions with 4-year NIH-NCCAM R25 research education grants, several successful strategies have been identified for developing, introducing, incorporating, and measuring the outcomes of research education and EBM training in the health professions curriculum. These strategies include assessing baseline curricular strengths, weaknesses, and opportunities for incorporating EBM principles; integrating research education content into the medical education courses; introducing EBM competencies early in the education process; inspiring curricular and culture change champions and early adopters to model new behaviors and attitudes; and providing enriching faculty development activities, such as “train-the-trainer” model.

Conclusion

Incorporating research literacy and EBM competencies into the curriculum requires both new and expanded content and faculty development.^16,22 Curricular revision and enhanced faculty teaching skills produce changes in the academic culture²⁰ and in behaviors.⁴² Only with changes in attitudes and behaviors can the element of time constraints be effectively addressed. Time constraints in curriculum and in clinical training and practice can discourage students and faculty when faced with having to learn and use EBM tools.⁴³ These NIH-NCCAM R25 grantee colleges were obliged to find innovative ways to work within time constraints. Although several of these R25 CAM institutions created additional required courses for first-year students, they incorporated learning activities into existing coursework in the second year and follow-up application learning activities in clinical practice training years.

These nine CAM institutions faced multiple challenges and developed similar and dissimilar strategies for success. This paper summarizes those strategies with the aim of guiding other institutions contemplating similar curricular and cultural changes. An enriched, EBM-infused CAM curriculum can better prepare future CAM practitioners for communicating effectively with their conventional medicine colleagues. Practitioners in the 21st century will need to understand how research and evidence-based practice are related and support one another in order to truly bring about optimal patient care.

Footnotes

Acknowledgments

The authors would like to thank Roni Evans, DC, MS, PhD, Director of Research Education at the Northwestern Health Sciences University, Wolfe-Harris Center for Clinical Studies for providing editorial review.

Funding for this work was provided by the National Institutes of Health, National Center for Complementary and Alternative Medicine (NIH-NCCAM). The authors wish to acknowledge the NIH-NCCAM R25 grants and to thank NIH-NCCAM for support of the grant recipients referenced in this article.

Author Disclosure Statement

No competing financial interests exist.

References

Barnes

, Bloom

, Nahin

. Complementary and alternative medicine use among adults and children: United States, 2007. Natl Health Stat Rep, 2008;:1–23.

Pearson

, Chesney

. The CAM Education Program of the National Center for Complementary and Alternative Medicine: an overview. Acad Med, 2007; 82:921–926.

Mennin

, Krackov

. Reflections on relevance, resistance, and reform in medical education. Acad Med, 1998; 73(9 Suppl):S60–64.

Epstein

. Learning from the problems of problem-based learning. BMC Med Educ, 2004; 7:1–7.

Peterson

. Health professions education: a bridge to quality. Tar Heel Nurse., 2003; 65:12.

Green

. A train-the-trainer model for integrating evidence-based medicine training into podiatric medical education. J Am Podiatr Med Assoc, 2005; 95:497–504.

Chichester

, Wilder

, Mann

, Neal

. Utilization of evidence-based teaching in U.S. dental hygiene curricula. J Dent Hyg., 2001; 75:156–164.

Khan

, Coomarasamy

. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ, 2006; 6:59.

Sackett

, Straus

, Richardson

, Rosenberg

, Haynes

. Evidence-Based Medicine: How to Practice and Teach EBM. 2nd ed. London: Churchill Livingstone, 2000.

10.

Mills

, Hollyer

, Guyatt

, Ross

, Saranchuk

, Wilson

. Teaching evidence-based complementary and alternative medicine: 1. A learning structure for clinical decision changes. J Altern Complement Med, 2002; 8:207–214.

11.

Del Mar

, Glasziou

, Mayer

. Teaching evidence based medicine. BMJ, 2004; 329:989–990.

12.

Sackett

, Rosenberg

. On the need for evidence-based medicine. J R Soc Med, 1995; 88:620–624.

13.

Maggio

, Tannery

, Chen

, ten Cate

, O'Brien

. Evidence-based medicine training in undergraduate medical education: a review and critique of the literature published 2006–2011. Acad Med., 2013; 88:1022–1028.

14.

Allen

, Connelly

, Morris

, Elmer

, Zwickey

. A train the trainer model for integrating evidence-based medicine into a complementary and alternative medicine training program. Explore (N Y), 2011; 7:88–93.

15.

Banzai

, Derby

, Long

, Hondras

. International web survey of chiropractic students about evidence-based practice: a pilot study. Chiropr Man Therap, 2011; 19:6.

16.

Connelly

, Elmer

, Morris

, Zwickey

. The Vanguard Faculty program: research training for complementary and alternative medicine faculty. J Altern Complement Med, 2010; 16:1117–1123.

17.

Cruser

, Dubin

, Brown

, et al. Biomedical research competencies for osteopathic medical students. Osteopath Med Prim Care, 2009; 3:10.

18.

Cruser

, Brown

, Ingram

, et al. Learning outcomes from a biomedical research course for second year osteopathic medical students. Osteopath Med Prim Care., 2010; 4:4.

19.

Evans

, Delagran

, Maiers

, Kreitzer

, Sierpina

. Advancing evidence informed practice through faculty development: the Northwestern Health Sciences University model. Explore (N Y), 2011; 7:265–268.

20.

Evans

, Maiers

, Delagran

, Kreitzer

, Sierpina

. Evidence informed practice as the catalyst for culture change in CAM. Explore (N Y), 2012; 8:68–72.

21.

Haas

, Leo

, Peterson

, LeFebvre R

. Outcomes from the UWS evidence-based practice R25 grant: knowledge, attitudes, skills, and behaviors. J Manipulative Physiol Ther, 2012; 35:701–709.

22.

Hammerschlag

, Lasater

, Salanti

, Fleishman

. Research scholars program: a faculty development initiative at the Oregon College of Oriental Medicine. J Altern Complement Med, 2008; 14:437–443.

23.

Hodsdon

, Nygaard

, Zwickey

. The sugar study?: a monograph for in-class research with medical students. Med Sci Educ, 2013; 23:159–164.

24.

Kreitzer

, Sierpina

, Fleishman

. Teaching research literacy: a model faculty development program at Oregon College of Oriental Medicine. Explore (N Y), 2010; 6112–114.

25.

Laird

, George

, Sanford

. NIH public access. Methods, 2011; 110:593–601.

26.

Lasater

, Salanti

, Fleishman

, et al. Learning activities to enhance research literacy in a CAM college curriculum. Altern Ther Health Med, 2009; 15:46–54.

27.

Lefebvre

, Peterson

, Haas

, et al. Training the evidence-based practitioner: University of Western States document on standards and competencies. J Chiropr Educ, 2011; 25:30–37.

28.

McCarty

, Fenn

, Gaster

, Weber

, Guiltinan

. Building bridges: qualitative assessment of a clinical faculty exchange between a naturopathic and an allopathic medical training program. Explore (N Y)., 2011; 7:249–253.

29.

Taylor

, Delagran

, Baldwin

, et al. Advancing integration through evidence informed practice: Northwestern Health Sciences University's integrated educational model. Explore (N Y), 2011; 7:396–400.

30.

Wayne

, Hammerschlag

, Savetsky-German

, Chapman

. Attitudes and interests toward research among students at two colleges of acupuncture and Oriental medicine. Explore (N Y)., 2010; 6:22–28.

31.

Leo

, Peterson

, Haas

, LeFebvre

, Bhalerao

. Development and psychometric evaluation of an assessment tool for a chiropractic college evidence-based practice curriculum. J Manipulative Physiol Ther, 2012; 35:692–700.

32.

Ramos

, Schafer

, Tracz

. Validation of the Fresno test of competence in evidence based medicine. BMJ, 2003; 326:319–321.

33.

Fritsche

, Greenhalgh

, Falck-Ytter

, Neumayer

H-H

, Kunz

. Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ, 2002; 325:1338–1341.

34.

Lai

, Teng

. Self-perceived competence correlates poorly with objectively measured competence in evidence based medicine among medical students. BMC Med Educ, 2011; 11:25.

35.

Coomarasamy

, Khan

. Learning in practice: What is the evidence that postgraduate teaching in evidence based medicine changes anything. BMJ, 2004; 329:1017.

36.

Hadley

, Wall

, Khan

. Learning needs analysis to guide teaching evidence-based medicine: knowledge and beliefs amongst trainees from various specialities. BMC Med Educ, 2007; 7:11.

37.

Malick

, Hadley

, Davis

, Khan

. Is evidence-based medicine teaching and learning directed at improving practice?. J R Soc Med., 2010; 103:231–238.

38.

Shuval

, Shachak

, Linn

, Brezis

, Feder-Bubis

, Reis

. The impact of an evidence-based medicine educational intervention on primary care physicians: a qualitative study. J Gen Intern Med., 2007; 22:327–331.

39.

Chakraborti

. Teaching evidence-based medicine using team-based learning in journal clubs. Med Educ, 2011; 45:516–517.

40.

Tilson

, Kaplan

, Harris

, et al. Sicily statement on classification and development of evidence-based practice learning assessment tools. BMC Med Educ, 2011; 11:78.

41.

Khader

, Batayha

, Al-Omari

. The effect of evidence-based medicine (EBM) training seminars on the knowledge and attitudes of medical students towards EBM. J Eval Clin Pract, 2011; 17:640–643.

42.

Bowe

, Lahey

, Armstrong

, Kegan

, Bowe

. Instituting change. Questioning the “big assumptions”. Part I: addressing personal contradictions that impede professional development. Med Educ., 2003; 37:715–722.

43.

Van Dijk

, Hooft

, Wieringa-de Waard

. What are the barriers to residents' practicing evidence-based medicine? A systematic review. Acad Med., 2010; 85:1163–1170.