Faculty Survey to Assess Research Literacy and Evidence-Informed Practice Interest and Support at Pacific College of Oriental Medicine

Abstract

Context:

Educating healthcare practitioners to understand, critically evaluate, and apply evidence to the clinical practice of complementary and alternative medicine has been an important initiative for the National Institutes of Health National Center for Complementary and Alternative Medicine.

Objective:

To determine the self-assessed research skills and interest of faculty at Pacific College of Oriental Medicine (New York campus) and their likely support of, and participatory interest in, an evidence-based medicine (EBM) training program.

Design:

The survey was administered in Survey Monkey. All questions were close-ended with 5-point Likert answers, except for one open-ended question at the end of the survey.

Setting:

One of three campuses of Pacific College of Oriental Medicine (PCOM), the largest Chinese medicine college in the United States.

Participants:

102 faculty employed at PCOM.

Results:

The response rate was 88.7%. Responses illustrated a generally high degree of interest and support for research, EBM, and institutional participation in research activities. Faculty who responded to the open-ended question (19.6% of respondents) expressed concerns about the relevance of research to Chinese medicine and the possibility of co-option by biomedicine.

Conclusions:

While faculty were overall supportive and interested in research and EBM, the results are consistent with the hypothesis that success of EBM training programs could be enhanced by soliciting and addressing faculty concerns and by being inclusive of approaches that honor the traditions of Chinese medicine and its own forms of clinical evidence.

Introduction

Applying evidence to clinical practice has been the focus of several national initiatives in the United States. These initiatives have generated evidence and evidence reports and educated healthcare practitioners to understand, critically evaluate, and apply evidence to clinical practice. In 1999, the National Institutes of Health (NIH) National Center for Complementary and Alternative Medicine (NCCAM) established the R25 CAM Education Project Grant Program aimed at accelerating the integration of CAM and biomedicine.¹ These R25 grants focused on the incorporation of CAM information into biomedical and residency training programs and on continuing education courses. Later, NCCAM awarded a series of R25 grants to CAM institutions² through the CAM Practitioner Research Education Project Grant Partnership, which was aimed at increasing the quality and quantity of research education, especially evidence-based medicine (EBM), in the CAM institutions. These grants involved a partnership between a CAM and research-intensive institution.

This paper presents data from a new project at Pacific College of Oriental Medicine (New York campus), which builds on one of the R25 awards to develop an educational research initiative, entitled “Evidence Informed Practice: Faculty and Curriculum Development.” This new project is supported by an NCCAM K07 grant and is being undertaken through collaboration with Albert Einstein College of Medicine (Einstein, Yeshiva University) and Northwestern Health Sciences University (NWHSU). NWHSU, in partnership with the University of Minnesota, received NCCAM R25 grant funding and has developed a very successful evidence-informed practice (EIP) faculty training and curriculum development project in all of their CAM degree programs.^3,4 PCOM-NY has a well-developed relationship with Einstein, supporting an interprofessional student education exchange program,⁵ research activities, and reciprocal faculty appointments.

In preparation for undertaking the EIP project at PCOM-NY, surveys were administered to the PCOM-NY faculty and students. The aim of the faculty survey was to gain insight into faculty self-assessed research skills and interest and to gauge their support of the EIP project as well as their interest in participating. This paper presents the outcomes of the faculty surveys.

Materials and Methods

Survey development and implementation

The survey instrument was developed using previously published surveys^6

–9 with questions tailored to the PCOM-NY faculty and the proposed EIP project. Following PCOM institutional review board approval, the survey was implemented in Survey Monkey via emails sent by the PCOM-NY academic dean to 115 faculty members. Survey participation was voluntary, with no penalty for nonparticipation. An initial email and four reminder emails were sent between June 21 and July 3, 2012. The survey consisted of questions about demographic characteristics, qualifications, years of service at PCOM, and prior research education and experience. Fourteen questions addressed opinions regarding research literacy and interest, importance of research to PCOM, interest in participating and using research, and attitudes and beliefs about research and EBM. Responses used a 5-point Likert scale (5=strongly agree; 4=agree; 3=neutral; 2=disagree; 1=strongly disagree). There was one open-ended question at the end of the survey: “Please add any additional comments below.”

Statistical analysis

Demographic characteristics of respondents were described with proportions for categorical variables and for continuous variables with mean±standard deviation or, if not meeting normality assumptions, median (interquartile range). Responses on the Likert scale were reported as proportion expressing agreement (“agree” and “strongly agree” combined). To facilitate interpretation, scores for broad survey question topic areas were constructed by adding the Likert scale value (1–5) for all questions related to that topic area. For example, an area with 5 questions could have a range of possible scores from 5 to 25, whereas an area with 3 questions could have a range from 3 to 15. Main topic areas were literacy, EBM value, funding support, and research participation. Bivariate associations between the topic area scores were assessed with Spearman rank correlations. A Spearman rho (ρ) and p-value for that ρ were reported for each bivariate correlation. Statistical significance was ascribed on the basis of a two-tailed α value of 0.05. Statistical analyses were performed with SPSS for Windows software, version 20 (SPSS, Inc., Chicago, IL). Qualitative theme analysis¹⁰ was used to assess the open-ended question responses.

Results

Survey response rate

The survey was emailed to 115 current teaching faculty in both the graduate and undergraduate programs at PCOM-NY. Of these 115 faculty, 10 teach exclusively in the undergraduate programs and do not have Chinese medicine qualifications. The survey was completed by 102 faculty (88.7% response rate).

Demographic characteristics

Table 1 presents the sociodemographic data of the survey responders. Fifty-three percent of the faculty were female, with a mean age of 47.6±9.1 years (range, 34–70 years). Faculty reported a median of 5 years of service at the College (interquartile range, 2–10; range, <1–17 years). Half of the faculty had prior research education, and of these, 41% of the prior research education was at the graduate level. Research experience was reported by 29% of the faculty, and 40% of these had the experience at the level of principal investigator or co–principal investigator, 64% as research assistant, 25% as research coordinator, and 25% performing data entry. The breakdown for research experience fields was as follows: 70% in the biological sciences, 37% in the social sciences, and 16% in the physical sciences.

Table 1.

Faculty Demographic Characteristics

Characteristic	Value
Women (%)	52.9
Mean age±standard deviation (range) (y)	47.6±9.05 (34–70)
Education: highest degree (%)
Undergraduate	9.6
Masters	55.7
Doctorate	26
Medical doctorate	8.7
Ethnic background (%)^a
White	63.5
Asian	25
African-American/black	4.8
Hispanic	4.8
Other	5.9
Time at college (y)
Median (IQR)	5.0 (2–10)
Range	0–17
Prior research education (%)
Yes	50
At graduate level	41.3
Prior research experience (%)^a
Yes	28.8
PI/Co-PI	39.3
Research assistant	64.3
Research coordinator	25
Data entry	25
Biological sciences	69.8
Social sciences	36.5
Physical sciences	15.9

The proportions sum to over 100% because respondents were asked to check all that apply.

IQR, interquartile range; PI, principal investigator.

Research literacy and the value of EBM

Table 2 presents the proportions agreeing with statements aimed at gauging research literacy and the extent to which faculty placed value on the role of EBM as it is applied to the clinical practice of CAM, and more specifically to acupuncture. Few answers were “disagree” or “strongly disagree,” so most of the “nonagree” responses were “neutral.” Overall, most faculty self-reported research literacy; 78% indicated that they are comfortable searching the web for relevant scientific literature, and 72% claimed to be comfortable with critically appraising scientific literature. However, the number of faculty who reported reading journal articles on a weekly basis was lower, at 58%.

Table 2.

Faculty Survey Responses to Questions About Research Literacy and the Value of Evidence-Based Medicine

Survey question	Agree (%) ^a
Research literacy
I am comfortable searching the Web for relevant scientific literature	77.5
I am comfortable critically appraising scientific literature	71.6
I read journal articles weekly (i.e. scientific, biomedical, Oriental medicine, etc.)	57.8
Value of evidence-based medicine
Critical evaluation of biomedical research methodology and outcomes is necessary to be able to assess the value and application of biomedical research to the practice of CAM	77.5
Evidence from well-designed biomedical research clinical trials, where the acupuncture intervention appears appropriate, is useful to an acupuncturist's clinical practice	74.8
Defining what successful outcomes look and feel like with patients allows practitioners and patients to agree on what “successful” treatment is, and facilitates patient retention and satisfaction	82.4
The quality of patient care is improved when practitioners seek out evidence about effective approaches to treating patients	76.5
The quality of patient care is improved when practitioners assess the outcomes of their treatments and use this knowledge to decide on further treatment strategies	93.1

Agree+strongly agree.

CAM, complementary and alternative medicine.

Overall, the faculty perceived value in EBM. Faculty were most likely to agree (93%) that the quality of patient care is improved when practitioners assess the outcomes of their treatments and use this knowledge to decide upon further treatment strategies. Most faculty also agreed (82%) that defining successful outcomes with patients allows practitioners and patients to agree on what successful treatments are and facilitates patient retention and satisfaction. Somewhat less agreement was shown for the other three questions, which referred explicitly to “biomedical” or “evidence.” Of these questions, the least extent of agreement (75%) was shown for the notion that evidence from well-designed biomedical research clinical trials, where the acupuncture intervention appears appropriate, is useful to an acupuncturist's clinical practice. Seventy-seven percent agreed that the quality of patient care is improved when practitioners seek out evidence about effective approaches to treating patients, and 78% of the faculty agreed that critical evaluation of biomedical research methods and outcomes is necessary to assess the value and application of biomedical research to the practice of CAM.

Research training and implementation of EIP

Most faculty (89%) agreed that it is important for the institution to develop a research agenda and apply for external research funding (Table 3). Furthermore, they overwhelmingly felt (95%) that with appropriate training, Oriental medicine practitioners and professors could make a substantial contribution to CAM research. Many (79%) agreed that as CAM becomes more integrated with biomedicine, CAM practitioners would benefit from having a strong foundation of training in research methods and outcomes.

Table 3.

Faculty Survey Responses to Questions About Engaging in Research Training and Implementation of Evidence-Informed Practice

Survey question	Agree (%) ^a
Faculty support for research funding and training at PCOM
It is important that PCOM develop a research agenda and apply for external research funding	89.2
As complementary and alternative medicine (CAM) becomes more integrated with biomedicine, CAM practitioners will benefit from having a strong foundation of training in research methodology and outcomes.	78.8
Oriental medicine trained practitioners and professors, if appropriately trained, can make a significant contribution to complementary and alternative medicine research.	95.1
Participation in research training and EIP implementation
If PCOM-NY were successful in receiving NIH funding to support the development of research curriculum for faculty professional development, and offered this to faculty for NCCAOM approved CEU's at a significantly discounted rate, I would be interested in taking this training.	75.5
Having received research training I would be interested in being a paid member of a PCOM committee (working with experts at other institutions) whose aims are to introduce research curriculum into the Master's degree, in order to train students to understand research and apply it to clinical practice.	68.7
After I have received research training, I would be interested in incorporating research-focused material into the courses and clinics that I instruct at PCOM.	74.5

Agree+strongly agree.

PCOM, Pacific College of Oriental Medicine; EIP, evidence-informed practice; NIH, National Institutes of Health; NCCAOM, National Certification Commission for Acupuncture and Oriental Medicine; CEU, continuing education units.

Faculty expressed general enthusiasm for receiving research training; 76% indicated they would be interested in taking the training if offered at a discounted rate and with eligibility for continuing education credit. A total of 75% indicated they would be interested in incorporating research-focused material into their courses and clinic shifts after receiving such training. Slightly fewer (69%) expressed interest in being a paid member of a committee aimed at developing research curriculum for student training.

Correlation between survey responses and prior research education and experience

Correlations between the focus categories of the survey questions, as defined by the four categories shown in Tables 2 and 3 and prior research education and experience were examined with Spearman rank correlation coefficients (Fig. 1). Panel A of Figure 1 lists the four categories of survey question focus: literacy, EBM value, funding support, and research participation. Panel B of Figure 1 shows the correlation coefficients (with p-values) between the four categories of survey question focus along with survey responses of faculty regarding prior research education and experience. Research literacy was significantly correlated with EBM value, funding support, and research participation, suggesting that research knowledge leads to interest in, and enthusiasm for, research. Valuing EBM was also significantly correlated with support for PCOM getting external research funding and for engaging in research activities. Support for research funding was also significantly correlated with engaging in research activities. There was almost no correlation between prior research education and experience with any of the survey question categories, except that respondents with prior research education were slightly more interested in research participation.

FIG. 1.

Correlation between survey question categories and prior research education and experience. (A) Survey question topic areas. (B) Correlation coefficients and probability significance. EBM, evidence-based medicine.

Qualitative outcomes

Of the 102 faculty who completed the survey, 20 (19.6%) responded to the open-ended question. Across all faculty comments, four broad themes were apparent: (1) Research is important for the profession; (2) Biomedical research paradigms are not compatible with Chinese medicine; (3) Chinese medicine could become co-opted by biomedicine; and (4) Research isn't relevant to the practice of Chinese medicine. Table 4 presents faculty comments divided among these four themes. Nine of the 20 faculty made strong statements in support of the usefulness of research, evidence, research education, and EBM. However, 15 of the 20 faculty made statements that were less supportive of the value of research; they highlighted issues associated with paradigm differences and concerns around the loss of authenticity of Chinese medicine and questioned the relevance of research to the practice of Chinese medicine.

Table 4.

Theme Analysis of Open-Ended Question (19.6% Response Rate - 20 Faculty Responded of 102 that Completed the Survey)

Theme 1: Research is important to the profession

“All clinicians should utilize EBM, clinical and empirical judgment”

“All forms of evidence are important in treatment decisions”

“I actually agree to the concept of incorporating scientific research into TCM developments”

“I am of the view that research is essential to gain inroads into the practice of acupuncture as integrative medicine”

“I would be very interested in seeing more basic research in CAM”

“Overall I am positive that CAM research is beneficial and important to the profession”

“I believe that incorporating research into the curriculum, as well as educating students to enable them to critically evaluate research articles will strengthen the entire profession as well as elevate it in the eyes of traditional allopathic medicine”

Theme 2: Biomedical research paradigms are not compatible with Chinese medicine

“TCM has a long history and does not need to be “validated” by western research methodologies whose assumptions do not match those of TCM”

“The primary obstacle to successful acupuncture research is the demand that studies follow the standard model of theorem, experiment or study, control, and blinding”

“Current models of research (i.e., how we are asking the questions) are the major impediment in CAM research”

“Research and assessment of Chinese medicine should be done with paradigms and markers common to the medical tradition”

“There is no question that assessment of treatment outcomes is important but it must be done in the context of the medical paradigm one is practicing under”

Theme 3: Chinese medicine could get co-opted by biomedicine

“I do not believe integration is possible, more likely absorption into this corrupt and limited paradigm”

“TCM practitioners must work to defend our medicine from those who would seek to limit our scope of practice”

“Modern trend of integration is to westernize Chinese medicine”

“There is still a danger zone where integration can mean assimilation for us. We also are in danger of having methods co-opted”

Theme 4: Research isn't relevant to the practice of Chinese medicine

“Research cannot validate our medicine, it is already been validated through centuries of practice”

“I think some people will excel as researchers but I do not believe it should be an integral element in Chinese Medicine training”

“The medical community ignores classic literature and lends no precedence or validity to ancient scholars who developed the medicine”

“I do not think that research outcomes have any significant effect on the actual clinical practice”

“Research is value, nonetheless, it is possible to be a strong practitioner without knowledge of research methodology”

Discussion

Robust faculty research and EBM training programs have been central to the development of EBM curricula at NWHSU and other CAM institutions that received NCCAM funding for EBM education projects.^{3,7,11

–14} Introducing a focus on biomedical research as a determinant of clinical practice into environments where clinical choices are often based on anecdotal evidence, textbooks, and opinions of respected clinicians has inherent challenges, which can be significantly lessened through faculty training and support.¹⁵

The results from this survey are encouraging. The high response rate (88.7%) suggests that faculty were enthusiastic to share their perspectives, although some of this may have been attributable to the academic dean's role in sending survey emails. The answers to the survey questions showed a generally high degree of interest and support for research, EBM, and College participation in research activities. Of the 102 faculty who responded to the survey, 20 contributed comments in the open-ended section of the survey, some expressing support for research and EBM, along with a comparatively greater number of responses expressing concern and resistance. It is critical that all potential barriers to EBM are fully understood. To this end, we intend to conduct further systematic evaluation of faculty readiness for change (using Davis's AVICTORY model^16
–18) and barriers to implementation (using Rogers' diffusion of innovation theory¹⁹) as part of our ongoing educational research initiative.

These results highlight several important aspects of the interface between CAM and EBM. First is the fact that the inherent challenges associated with CAM and the perspectives of CAM practitioners may make incorporating EBM difficult.^20,21 Second is that a significant cultural change is necessary to successfully implement EBM faculty training and curriculum development projects within CAM institutions.^15,22 Chinese medicine may be uniquely challenging among the CAM disciplines for EBM integration due to its substantial paradigm differences in comparison to biomedicine.²³

In addition to paradigm issues, the often skeptical attitude of CAM practitioners toward biomedicine and research methods^19,20,24 is an important challenge for EBM integration. CAM practitioners tend to view biomedicine as reductionist and disempowering for patients.^21,24,25 The view of the body and its physiologic and pathologic mechanisms held by Chinese medicine is very different from that of biomedicine.²⁶ A further challenge to integration is the fact that the vast and complex nature of the theories that underlie the practice of Chinese medicine tends to be under-recognized outside the profession, as does the incredible wealth of 2000 years of empirical and anecdotal experience, much of which has not been translated. Despite the well-established educational and licensing infrastructure and the rigorous training required to become a licensed acupuncturist, many practitioners are often confronted with the need to validate their training and approach. The lack of familiarity of the biomedical world with the knowledge base and theory of Chinese medicine can engender a perspective among practitioners that their medicine is little understood and undervalued, which leads to the expectation that Chinese medicine could be co-opted by biomedicine and modified in ways that would likely be deleterious to the efficacy and integrity of the medicine.²⁷ Such sentiments were clearly evident in many of the responses to the open-ended question in this survey.

The application of research results, especially those from randomized controlled trials (RCTs), has had a very large effect on the attitude of Chinese medicine practitioners toward research and the use of evidence to inform clinical practice. Practitioners tend to assume that EBM relies solely on research evidence¹² and are often unaware that it also includes evidence from a wide range of study designs, as well as textbooks, respected clinical scholars and practitioners, along with patient preferences and clinician experience.¹¹ Given that the quantity, quality, and clinical applicability of RCTs has, up until more recently, been quite poor,^{8,24,28

–31} RCT evidence is often viewed by Chinese medicine practitioners as inappropriate to clinical practice and very limiting.¹² The development of STRICTA (Standards for Reporting Interventions in Clinical Trials of Acupuncture) and other reporting guidelines^32,33 have improved the quality and clinical relevance of RCTs. However, these guidelines do not address many issues of concern to Chinese medicine practitioners; many practitioners feel that research, and especially RCTs, oversimplify real-world practice and lessen the clinical applicability of the findings.^34,35 These issues were reflected in our survey, with many faculty indicating that they were less enthusiastic about the notion that research training improves a Chinese medicine practitioner's clinical practice. Future training efforts that illustrate how a range of study designs and other nonresearch evidence can work in concert to inform clinical practice will likely prove helpful.

The term “evidence-informed practice” was adopted by the CAM community²² to engender greater receptivity from CAM students and practitioners and to better emphasize the role of the other factors, such as patient preferences and clinician experience. This new term implies the notion that all forms of evidence, along with patient and practitioner preferences, should determine treatment choice. Evidence is presented in the role of guide not dictator of practice and can be generated by the practitioners themselves as they treat patients over time and collect their own data.^36,37 As seen in this survey, most faculty agreed with the notion that patient care is improved when practitioners use ongoing treatment outcomes to determine further treatment. Fundamentally what is really being encouraged and taught in EIP education is critical thinking, or that reflective moment when a clinician asks internally “Why am I doing this treatment?” When the concept of EIP is expressed in this context, CAM practitioners are more receptive and interested.²²

Despite all of the influences discussed here, outcomes of this survey suggest that faculty are generally supportive and enthusiastic regarding research, EBM, and the possibility of PCOM receiving research funding. Similar trends were seen in other U.S. Chinese medicine colleges that received NIH funding.^11,12 PCOM has several distinguishing factors that may have contributed to this outcome: a strong biomedical curriculum, a focus on integrated medicine, and a doctoral program that is not required for acupuncture licensure. PCOM-NY faculty reported high levels of self-rated research literacy, and many had prior research education and experience. These characteristics of the College and faculty could suggest that the level of enthusiasm for research and EBM observed at PCOM in this survey may be higher than the average for all Chinese medicine colleges in the United States. The outcomes reported in this study are encouraging for the success of the NIH-funded project to develop EIP faculty training and curriculum development. The levels of support expressed by faculty were above the benchmark goals of the grant, which aims to have 65% of the faculty undertake the EIP training.

This study does have some limitations. The survey was administered by the academic dean, and thus faculty may have felt obligated to respond in a favorable manner. Only 20% of the faculty responded to the open-ended question, so the shared perspectives presented represent only a subset of all faculty; there are likely other valuable insights not expressed. Results related to the associations between survey topic areas and research experience and education should be interpreted with caution and do not imply causal relationships. All self-reporting has inherent inaccuracy, and previous studies have shown that EBM surveys are sometimes associated with inaccurately elevated self-reported EBM literacy.³⁸ Given that this survey was administered before any formal EIP training has taken place, it is possible that some faculty are not sufficiently knowledgeable of research and EBM to be able to accurately assess their own literacy. Finally, the survey used in this study represented a modification of previously used instruments and was not psychometrically tested. However, it provides important preliminary data to inform future faculty development in research literacy and EBM.

Conclusions

The reports from the NIH NCCAM R25-funded EBM training programs at CAM institutions indicated that adoption of EBM was facilitated by accompanied institutional cultural changes. This concept is consistent with the outcomes of this survey, which suggest the hypothesis that faculty will probably need to be encouraged to engage in critical thinking and reflective practice and to appreciate the benefits of EIP in improving patient care. Future research should explore the possibility that faculty EIP training programs are likely to benefit from factoring in the concerns that CAM faculty may have around the relevancy of clinical trial evidence to clinical practice. Emphasizing the three pillars of EIP (research evidence, patient preferences, and clinician experience) could be beneficial. Faculty receptivity might be enhanced when research evidence is interpreted broadly and includes the primary forms of evidence that exist for many CAM traditions, including traditional texts, opinions of respected clinicians, and experience through regular patient treatments. With such approaches, faculty EIP training programs may be well received by faculty and could result in treatment approaches that are better informed by evidence with improved patient outcomes.

Footnotes

Acknowledgments

We thank Eric Patel for assistance with survey preparation and administration and Roni Evans for reviewing the manuscript. Research reported in this publication was supported by the NCCAM of the NIH under award number K07AT007186. This publication was also supported in part by the Clinical and Translational Science Award grant 1 UL1 TR001073-01, 1 TL1 TR001072-01, 1 KL2 TR001071-01 from the National Center for Advancing Translational Sciences (NCATS), a component of the NIH. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Author Disclosure Statement

No competing financial relationships exist.

References

Pearson

, Chesney

MA.

The CAM Education Program of the National Center for Complementary Alternative Medicine: an over view. Acad Med., 2007; 82:921–926.

Kreitzer

, Sierpina

. NCCAM awards grants to CAM institutions to enhance research education. Explore (NY), 2008; 4:74–76.

Evans

, Delagran

, Maiers

, et al. Advancing evidence informed practice through faculty development: the Northwestern Health Sciences University model. Explore (NY), 2011; 7:265–268.

Taylor

, Delagran

, Baldwin

, et al. Advancing integration through evidence informed practice: Northwestern Health Sciences University's integrated educational model. Explore (NY)., 2011; 7:396–400.

Anderson

, Herron

, Downie

, et al. Interprofessional Student Education: exchange program between Albert Einstein College of Medicine and Pacific College of Oriental Medicine. Explore (NY)., 2012; 8:377–381.

McAlister

, Graham

, Karr

, Laupacis

. Evidence-based medicine and the practicing clinician. J Gen Intern Med., 1999; 14:236–242.

Wayne

, Hammerschlag

, Savetsky-German

, Chapman

. Attitudes and interests toward research among students at two colleges of acupuncture and Oriental medicine. Explore (NY). 2010; 6:22–28.

Allen

, Connelly

, Morris

, et al. A train the trainer model for integrating evidence-based medicine into a complementary and alternative medicine training program. Explore (NY)., 2011; 7:88–93.

Hendricson

, Rugh

, Hatch

, et al. Validation of an instrument to assess evidence-based practice knowledge, attitudes, access, and confidence in the dental environment. J Dent Educ., 2011; 75:131–144.

10.

Guest

, MacQueen

, Namey

. Applied Thematic Analysis. Thousand Oaks, CA: Sage Publications, 2012.

11.

Hammerschlag

, Lasater

, Salanti

, Fleishman

. Research scholars program: a faculty development initiative at the Oregon College of Oriental Medicine. J Altern Complement Med., 2008; 14:437–443.

12.

Wayne

, Buring

, Davis

, et al. Increasing research capacity at the New England School of Acupuncture through faculty and student research training initiatives. Altern Ther Health Med., 2008; 14:52–58.

13.

Lasater

, Salanti

, Fleishman

, et al. Learning activities to enhance research literacy in a CAM college curriculum. Altern Ther Health Med., 2009; 15:46–54.

14.

Kreitzer

, Sierpina

, Fleishman

. Teaching research literacy: a model faculty development program at Oregon College of Oriental Medicine. Explore (NY)., 2010; 6:112–114.

15.

Evans

, Maiers

, Delagran

, et al. Evidence informed practice as the catalyst for culture change in CAM. Explore (NY)., 2012; 8:68–72.

16.

Davis

. Change and innovation. In: Feldman

, ed. Administration and Mental Health. Springfield, IL: Thomas, 1973.

17.

Davis

. Management of innovation and change in mental health services. Hosp Commun Psychiatry., 1978; 29:649–658.

18.

Davis

, Salasin

. The utilization of evaluation. In: Struenig

, Guttentag

, eds. Handbook of Evaluation Research. Vol. 1. Beverley Hills, CA: Sage Publications, 1975.

19.

Rogers

. Diffusion of Innovations. 4th ed. New York: The Free Press, NY: 1995.

20.

Wilson

, Mills

. Closing comment: Evidence-based complementary and alternative medicine: is it a viable concept?. J Altern Complement Med., 2002; 8:875–876.

21.

Barrett

, Marchand

, Scheder

, et al. Themes of holism, empowerment, access, and legitimacy define complementary, alternative, and integrative medicine in relation to conventional biomedicine. J Altern Complement Med., 2003; 9:937–947.

22.

Kreitzer

, Sierpina

, Maiers

, et al. Ways of knowing: integrating research into CAM education and holism into conventional health professional education. Explore (NY)., 2008; 4:278–281.

23.

Kaptchuk

. Acupuncture: theory, efficacy, and practice. Ann Intern Med., 2002; 136:374–383.

24.

Mills

, Hollyer

, Guyatt

, et al. Teaching evidence-based complementary and alternative medicine: 1. A learning structure for clinical decision changes. J Altern Complement Med., 2002; 8:207–214.

25.

Kaptchuk

, Eisenberg

. The persuasive appeal of alternative medicine. Ann Intern Med., 1998; 129:1061–1065.

26.

Kaptchuk

. The Web That Has No Weaver: Understanding Chinese Medicine. Chicago: Congdon & Weed, 2000.

27.

Kaptchuk

, Chen

, Song

. Recent clinical trials of acupuncture in the West: responses from the practitioners. Chin J Integr Med., 2010; 16:197–203.

28.

Nahin

, Straus

. Research into complementary and alternative medicine: problems and potential. BMJ., 2001; 322:161–164.

29.

Langevin

, Hammerschlag

, Lao

, et al. Controversies in acupuncture research: selection of controls and outcome measures in acupuncture clinical trials. J Altern Complement Med., 2006; 12:943–953.

30.

Walji

, Boon

. Redefining the randomized controlled trial in the context of acupuncture research. Complement Ther Clin Pract., 2006; 12:91–96.

31.

Jobst

. Becoming and growing—what does integration mean?. J Altern Complement Med., 2009; 15:v–vi.

32.

MacPherson

, Altman

, Hammerschlag

, et al. Revised STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA): Extending the CONSORT statement. J Evid Based Med., 2010; 3:140–155.

33.

Hammerschlag

, Milley

, Colbert

, et al. Randomized controlled trials of acupuncture (1997–2007): an assessment of reporting quality with a CONSORT- and STRICTA-based instrument. Evid Based Complement Alternat Med., 2011; 2011. pii: 183910.

34.

Smith

, Zaslawski

, Zheng

, et al. Development of an instrument to assess the quality of acupuncture: results from a Delphi process. J Altern Complement Med., 2011; 17:441–452.

35.

Anderson

, Rosenthal

. Acupuncture and in vitro fertilization: critique of the evidence and application to clinical practice. Complement Ther Clin Pract., 2013; 19:1–5.

36.

Maiers

, McKenzie

, Evans

, McKenzie

. Patient outcomes at a traditional Chinese medicine teaching clinic: a prospective data collection project. J Altern Complement Med, 2008; 14:1083–1088.

37.

Marx

, Rubin

, Milley

, et al. A prospective patient-centered data collection program at an acupuncture and oriental medicine teaching clinic. J Altern Complement Med., 2013; 19:410–415.

38.

Khan

, Awonuga

, Dwarakanath

, Taylor

. Assessments in evidence-based medicine workshops: loose connection between perception of knowledge and its objective assessment. Med Teach., 2001; 23:92–94.