Instruments for Evaluating Undergraduate Medical Education in Complementary and Integrative Medicine: A Systematic Review

Abstract

Objectives:

The provision of courses in complementary and integrative medicine (CIM) varies widely between medical schools. To effectively improve CIM education, it is essential to use robust evaluation instruments that measure the impact of different educational interventions. This review aimed to identify and critically appraise qualitative and quantitative instruments used to evaluate CIM courses in undergraduate medical education.

Methods:

A systematic review was conducted in PubMed/MEDLINE, LIVIVO, CINAHL/EBSCO, Scopus, Web of Science, and Ovid/Embase in January 2023. Eligible studies included complete evaluation instruments for medical students and reported learning outcomes. Data extraction included information on the study design, the educational intervention, the evaluation instrument, and the outcome measure (e.g., Kirkpatrick levels: 1 reaction, 2a attitudes, 2b knowledge/skills, 3 behavioral change, 4 results). Instruments were categorized as validated, nonvalidated, or qualitative and analyzed using descriptive statistics. Validated instruments were assessed for quality using standardized criteria.

Results:

Of the 1909 records identified, 263 were subjected to a full-text review and 100 studies met the inclusion criteria. Twenty-seven studies reported on 14 validated instruments, 7 studies reported on qualitative, and 66 reported on nonvalidated instruments. Most were conducted in the United States (31) and Europe (28), 51 were cross-sectional studies, and 42 were intervention studies. Most of the instruments were self-administered (50), addressed general aspects of CIM (53), and assessed student attitudes (74). None of the validated instruments covered Kirkpatrick level 1, one covered level 3. Measurement of levels 2b and 3 was usually based on subjective self-assessment. Qualitative instruments covered the widest range of outcomes overall. Validated instruments often had good content validity and internal consistency, but lacked reliability and responsiveness. Revalidation of translated or modified instruments was mostly inadequate.

Discussion:

This structured and comprehensive set of existing instruments provides a starting point for the further development of CIM course evaluation in undergraduate medical education. Future studies should prioritize the measurement of higher-level learning outcomes, such as behavioral change and impact on patient care. Comparative intervention studies between medical schools or with pre–post designs and follow-up evaluations are needed to assess the effectiveness of different teaching approaches. Regular revalidation of both existing and newly developed instruments is essential to ensure their applicability to different audiences and settings. Their structured and standardized use would promote evidence-based CIM training and understanding of its impact on student competencies and patient outcomes.

Introduction

The use of complementary medicine has increased significantly in recent decades and has recently stagnated at a high level of 45%–70% of the population.^1

–5 A number of positive effects on health care have been shown, for example, on chronic diseases⁶ or on the use of antibiotics.⁷ It is therefore important to bring together conventional and complementary approaches in a coordinated way.⁸ The potential benefits of integrative health have already been demonstrated in a variety of settings, including pain management, symptom relief in cancer patients and survivors, and programs to promote healthy behaviors.⁹ Complementary medicine is when a nonmainstream approach is used alongside conventional medicine.⁹ Integrative medicine goes beyond this, as “Integrative medicine and health reaffirms the importance of the doctor-patient relationship, targets the whole person, is informed by evidence, and employs all appropriate treatment, preventive, health-promoting, or lifestyle approaches, health care professionals and disciplines to achieve optimal health and healing; equally emphasizing the art and science of healing. It is based on a social and democratic as well as a natural and healthy environment.”¹⁰

A review by Joyce et al. shows that medical students perceive that the teaching of complementary and integrative medicine (CIM) is not adequately covered in their curriculum and that sufficient CIM knowledge and skills are important for their future as doctors to be able to advise and refer patients. Therefore, CIM content should be introduced or expanded upon.¹¹ Soliman and Bilszta conducted a scoping review and found an inconsistent inclusion of CIM teaching, even within medical schools in the same country. Medical curricula that include CIM teaching and learning vary widely in both content and delivery.^12,13 Although there is a need to equip medical students with CIM-related knowledge and practice, there is no research on the optimal way to impart this knowledge.¹⁴ In the absence of such evidence, it is difficult to define clear objectives for a CIM teaching program within a medical curriculum and to develop strategies for its implementation.^15,16 For example, while interprofessional education is a promising approach in many countries for training health professionals together to improve patient-centered collaboration, little is known about how teaching CIM can promote both: interprofessional collaboration and CIM practice.^17,18 Well-designed and validated program evaluation instruments allow medical researchers to assess whether a program or teaching format has worked, how it has worked, why it has worked, and to gather more information about the implementation process of the educational intervention. By engaging in this process, the researcher will develop a sound understanding of the relationships between programs, the contexts in which they operate, and the outcomes that result.¹⁹ There is currently no overview of the instruments available to evaluate CIM teaching. A systematic review from 2012 shows that high-quality program evaluations available at that time were scarce.¹⁴

In order to be able to further develop teaching in the field of CIM, it is necessary to identify suitable evaluation instruments for CIM training, which, in the best case, can be used across faculties. Quantitative instruments, such as closed-ended questionnaires, allow the measurement of antecedents, learning gains, and impact on patient care, for example, by identifying differences and correlations. The use of validated instruments can significantly increase the reliability and validity of the data collected. Qualitative tools, such as interview guidelines, can be used to explore influencing factors and underlying patterns and attitudes, even in small groups.²⁰

Objectives

This study aims to investigate what qualitative and quantitative instruments are available to evaluate CIM courses and programs for undergraduate medical education and to assess the quality of validated instruments.

Materials and Methods

The review was conducted by members of the Committee for Integrative Medicine and Pluralism of Perspectives of the German Association for Medical Education in cooperation with the Forum of University Working Groups on Naturopathic Treatment and Complementary Medicine. The authors conducted a systematic review in accordance with the Preferred Reporting Items for abstract Systematic reviews and Meta-Analyses (PRISMA) 2020 Checklist for abstracts (see Supplementary Table S1) and the guidelines detailed in the PRISMA checklist (see Supplementary Table S2).²¹ The review protocol was prospectively registered with PROSPERO, number CRD42022354606.

Data sources and search strategy

The following databases were searched: PubMed/MEDLINE, LIVIVO, CINAHL/EBSCO, Scopus, Web of Science, and Ovid/Embase on January 5, 2023. The search term was composed of the following elements: “undergraduate education,” “complementary medicine,” and “questionnaire,” and was adapted to each database. The search strategies are listed in Supplementary Table S3. The findings were imported into the Rayyan research collaboration platform (https://www.rayyan.ai/). Duplicates were deleted.

Eligibility criteria

All studies that met the following criteria were included:

The study describes or provides an instrument for the evaluation of CIM training or CIM aspects.

The instrument described may be either qualitative (e.g., an interview guide) or quantitative (e.g., a questionnaire) or both.

The target population of the instrument is undergraduate medical students or interprofessional student groups that include medical students.

Studies describing or providing instruments for examinations or accreditation procedures were excluded, as were studies in which the instrument described could not be sufficiently reconstructed (not all questions listed, measurement method not specified) or they were only abstract publications or study protocols. There were no restrictions on the types of studies included.

In the review protocol, we had originally specified that only studies directly related to CIM training would be included. However, at the beginning of the search process, we realized that this meant that psychometric studies and instruments for recording current training or knowledge would not be included, although they might be suitable for use in teaching evaluation, and we therefore changed this specification.

Selection of studies

The screening and selection of studies took place in two steps. In the first step, the titles and abstracts were checked for suitability. Studies were sorted alphabetically by first author. The first 20 studies were reviewed by all researchers (A.H., B.S.-S., G.R., K.F., M.T.) in a blinded mode. Any conflicting reviews were then discussed by the whole group in order to refine the previously defined screening strategy. The remaining studies were screened blindly by at least two researchers each (title and abstract). Any discrepancies and/or conflicts were resolved through discussion by the whole group. If discrepancies remained, the study was retained. In a second step, all remaining studies were included in the full-text screening. Two researchers independently screened each full text to decide on final inclusion or exclusion in the review. Any discrepancies and/or conflicts were resolved through discussion in the research group meetings. To check the objectivity of the rating in both steps, interrater reliability was calculated using Cohen’s kappa.²²

Foreign language (not German, English, or Spanish) studies were translated into English using DeepL Pro, and a person with the relevant language skills was consulted if there were any uncertainties.

Data extraction

A Microsoft Excel spreadsheet was created containing the following information:

Identification/demographic information (DOI, first author)

Study information (country, study design, sample size)

Educational intervention, if indicated (educational approach, duration)

Instrument information (approach, target group, subject/domains, if indicated: the name of the questionnaire)

Quality of the instrument (development process, validation status)

Outcome measures (interest in learning, personal experiences, learning experiences, didactic issues, Kirkpatrick levels of learning outcome,²³ individual benefits)

The following rules were defined to specify the outcome measures prior to data extraction:

Interest in learning: Respondents are asked if they are generally interested in the subject of CIM or if they would like to be taught in this subject area.

Personal experience: Respondents are asked if they have any experience with CIM, for example, if they have visited alternative practitioners or know CIM applications from their family or use them for self-treatment.

Learning experience: Respondents are asked if they have received CIM training, what was taught and how much.

Didactic issues: Respondents are asked about CIM teaching methods and questions about specific delivery or the choice of content, for example, whether they prefer particular teaching methods or formats or content, or whether they find the program useful at a particular stage in their studies.

Reaction or satisfaction (Kirkpatrick level 1): Respondents are asked how satisfied they are with the CIM training and how good they think the training was (only if a course was actually offered).

Attitudes or perceptions (Kirkpatrick level 2a): Respondents are asked about their attitudes, opinions, and evaluations of CIM in general or of individual procedures.

Knowledge and skills (Kirkpatrick level 2b): Respondents assess their own knowledge or skills, answer knowledge questions, or are observed performing.

Behavioral change or intention to change (Kirkpatrick level 3): Respondents are asked if they are already using what they have learned, or if they expect to use it in the future or if they think it will benefit them in their professional practice.

Results (Kirkpatrick level 4): Captures changes in patient care, for example, in health or well-being organizational change, for example, in work units or institutions, or community change.

Individual benefits: Respondents are asked whether they feel that they have personally benefited and learned something for their own health or personal development.

Data extraction was carried out by one researcher. A second researcher performed a quality control of extracted data. All missing data were marked as such in the Excel spreadsheet.

Data analysis and synthesis

The identified studies are divided into the following three categories: Studies with validated instruments, studies with nonvalidated instruments, and studies with qualitative instruments. An instrument is considered validated if at least one psychometric test has been carried out. All studies and the instruments described in them are analyzed using descriptive statistics and summarized in a table. In addition, the data extracted from each study are also summarized in tables. As we do not analyze the data collected in each study as such, we do not measure the effects or synthesize these results.

Quality assessment

As we refer to the instruments provided, the quality of the studies themselves and the data collected in them will not be subject to any structured quality assessment, and no reporting or safety assessment will be carried out in relation to the data collected. Instead, we include instruments from the category “validated” in the quality assessment using the following established criteria for measure properties for questionnaires: content, criterion and construct validity, internal consistency, reproducibility, responsiveness, floor/ceiling effects, and interpretability.²⁴ Our approach followed the steps and scoring strategy described by Orfanos et al.²⁵ Each item within the criterion was assessed by two researchers (A.H., B.S.-S.) for adequate design and statistical reporting. If the criteria for a particular psychometric property were met, a score of 2 was assigned. If the design of the criterion was questionable, if there was no clear description of the aspects outlined, or if the criteria for a particular psychometric property were only partially met, a score of 1 was assigned. If no information was provided for a particular criterion, or if the criterion was explicitly not met, a score of 0 was given. The total score for each measure and each category of quality criteria is given for all measures. The two researchers initially evaluated each criterion independently. If they disagreed, an agreement was reached through argumentative discussion. The results are tabulated including the total score achieved.

Results

Search process and study selection

Our electronic search identified a total of 2329 records. After reviewing the titles and abstracts, we identified 263 potentially relevant studies. We obtained two further studies from the reference lists of the identified studies and finally included 265 studies in the full-text screening, resulting in 100 included studies. Thirty-five studies described an evaluation instrument, but lacked information on the type, sequence, or completeness of the questions and could therefore not be fully reconstructed. These studies were excluded. The search process is shown in Figure 1.

FIG. 1.

Flowchart. Adapted from Page et al.²¹

Interrater reliability was at the level of substantial agreement in both steps (step 1: kappa = 0.671, standard error 0.025, 95% confidence interval [0.623–0.72]; step 2: kappa = 0.657, standard error 0.043, 95% confidence interval [0.572–0.742]).

A total of 14 different instruments were included in the 27 studies that reported validated instruments. Seven of the included studies reported a qualitative instrument, whereas 66 described nonvalidated instruments.

Characteristics of the included studies

From the 100 studies identified, 31 were conducted in the United States and 28 in European countries. Fifty-one studies were cross-sectional studies and 42 refer to an educational intervention. Fourteen studies used a single-arm pre–post design, three studies included a control group, and four were randomized. The studies that included other students as well as medical students mostly included students from other health professions, such as pharmacy or nursing. Exceptions are the studies by Iktidar 2022 and Xie 2020, which also included students who had not studied medicine, health care, or any other patient-related subjects.^26,27 Of the 10 studies that include more than just students, some included teachers^28

–33 or professionals,^34
–36 and one included patients.³⁷

The majority of the quantitative studies (n = 48 out of 93) were conducted with 100–499 participants. The four large studies with 1784,³⁸ 2292,²⁷ 2004,³⁹ and 2839⁴⁰ participants were national surveys. International studies were conducted by Schmidt et al. 2005 and Rees et al. 2009.^41,42

Table 1 provides an overview of the characteristics of the included studies.

Table 1.

Characteristics of the Included Studies

Studies with	Validated instruments (n = 27)	Qualitative instruments (n = 7)	Nonvalidated instruments (n = 66)	Total (n = 100)
Country
United States	7	3	21	31
United Kingdom	4	—	4	8
Germany	4	1	2	7
Saudi Arabia	1	—	6	7
China	1	—	5	6
Turkey	1	—	5	6
India	1	1	3	5
Israel	—	—	4	4
Australia	—	1	1	2
Brazil	1	—	1	2
Ireland	1	—	1	2
Japan	—	—	2	2
Switzerland	—	—	2	2
International	2	—	—	2
Others	4 (Bangladesh, Denmark, Serbia, Sierra Leone)	1 (Hong Kong)	8 (Canada, Ghana, Iran, Kuwait, Libya, Netherlands, Palestine, Spain)	13
Study type
Cross-sectional	24	2	25 (2 RT)	51
Intervention	2 (1 CT)	5	35 (2 RCT, 14 pre–post)	42
Combined	1	—	—	1
Cohort	—	—	6	6
Population^a
Medical students	16	7	55	78
Other students	8	—	6	14
Staff/teacher/ professionals	4	—	5	9
and patients	—	—	1	1
Number of participants
<50	—	4	10	14
50–99	1	1	13	15
100–499	16	2	32	50
500–999	8	—	8	16
>1000	2	—	2	4

Multiple assignments possible.

CT, controlled trials; RT, randomized trial; RCT, randomized controlled trial.

Characteristic of the evaluation instruments

Half of the authors (50 out of 100) stated that the instruments had been developed by themselves; 35 based on previously published questionnaires. Five studies with validated instruments also consulted a panel of experts,^{35,38,43
–45} of the other studies it was indicated that the process was iterative^46,47 with internal revision⁴⁸ or with a multidisciplinary team.⁴⁹

Most of the identified instruments related to CIM in general. Some combined CIM with other aspects, such as health issues or conventional medicine. Evaluation instruments on specific topics dealt with, for example, osteopathy, Chinese Medicine, kampo medicine, or acupuncture.

Most of the instruments, especially the validated ones, measure students’ attitudes (Kirkpatrick level 2a), for example, their attitudes toward specific therapeutic methods and concepts in relation to CIM in general. None of the validated questionnaires measured Kirkpatrick level 1. This is mainly because there are no intervention studies in this category and therefore reaction to or satisfaction with an educational intervention could not be measured. In contrast, of the seven studies describing qualitative instruments, five are intervention studies, four of which measure satisfaction with the intervention.^50

–53 Kirkpatrick level 2b (knowledge and skills) was usually assessed by subjective self-assessment of the students in relation to their existing knowledge or knowledge acquisition. Only the unvalidated questionnaire by Karpa et al.⁵⁴ asked objective questions about CIM knowledge before and after the training program. Kirkpatrick level 3 (behavioral change) was also addressed more frequently in the qualitative instruments and referred mainly to the intention to change rather than actual changes. For example, the validated Furnam et al. 1999 questionnaire⁴⁵ included questions about treatment preferences. Other validated questionnaires also asked Kirkpatrick level 3 questions, that is, outside the validated range, to assess intended behavioral change, for example, whether students would recommend CIM methods or think they would use the knowledge gained in their future careers.^{27,34,43,55,56} Similarly, five of the seven qualitative instruments only assessed whether students intended to recommend or practice CIM, or thought the gained knowledge would contribute to their work as physicians.^{46,51
–53,57} None of the instruments identified measured Kirkpatrick level 4 (results). Overall, the qualitative studies are the most comprehensive in terms of outcome measurement.

An overview of the characteristics of the evaluation instruments is given in Table 2.

Table 2.

Characteristics of the Evaluation Instruments

Studies with	Validated instruments^a (n = 27)	Qualitative instruments (n = 7)	Nonvalidated instruments (n = 66)	Total (n = 100)
Development^b
Self-administered	6	4	40	50
Literature based	9	—	26	35
Expert/consensus process	5	1	3	9
Retained tool	3	—	—	3
Modified tool	10	—	2	12
Not reported	1	3	13	17
Area
CAM/CIM	16	2	35	53
Combined with other areas	8	—	5	13
Special topics	3	5	26	34
Outcomes
Interest in learning	4 (+3)	5	30	39 (+3)
Personal experience	7 (+9)	3	34	44 (+9)
Learning experience	- (+1)	2	18	20 (+1)
Didactic issues	1 (+3)	4	21	26 (+3)
Individual benefit	—	2	12	14
Kirkpatrick level
1: Reaction	- (+1)	4	6	10 (+1)
2a: Attitudes	26	4	44	74
2b: Knowledge/skills	4 (+4)	4	44	52 (+4)
3: Behavioral change	1 (+4)	5	25	31 (+4)
4: Results	—	—	—	0

Data in brackets: additional questions, not within the validated scale.

As stated by the authors, multiple assignments possible.

CAM/CIM, complementary and alternative medicine/complementary and integrative medicine.

Table 3 lists the validated instruments, Table 4 lists the qualitative instruments. The nonvalidated instruments are provided in Supplementary Table S4, instruments regarding CIM training in general, and Supplementary Table S5, instruments for specific topics in CIM training.

Table 3.

Validated Evaluation Instruments for Complementary and Integrative Medicine Training

Name of the questionnaire, language if not English, (extended)^a	First author, publication year	Area^b	Country	Population	Study design	Number of participants (enrolled)	Outcome measures (extended)^c				Instrument development	Validity status
	First author, publication year	Area^b	Country	Population	Study design	Number of participants (enrolled)	Interest in learning	Personal experience	Learning experience/didactic issues	Learning outcome^d	Instrument development	Validity status
CAIMAQ (extended)	Abbott, 2011³⁸	CAIM	USA	Medical students	National survey, cross-sectional	(68,001) 1,784	(Yes)	(Yes)	No/no	(1), 2a	Literature based on Schneider 2003,³⁵ expert group	Content validity, confirmatory and exploratory factor analyses
CAIMAQ	Singh, 2021⁵⁸	CAIM	India	Medical students and teaching stuff (physicians)	Cross-sectional	(541) 138	No	No	No/no	2a	Retained	Not revalidated
CAMs inventory	Ahmad, 2017⁵⁹	CAM	Saudi Arabia	Medical and pharmacy students	Cross-sectional	(369, calculated sample size) 269	Yes	Yes	No/no	2a, 2b	n.r.	Content validity, sample adequacy, total reliability
CAM survey	Frye, 2006⁶⁰	CAM	USA	Medical students	Cross-sectional	(n.r.) 152	No	Yes	No/yes	2a, 2b	Literature based	Exploratory factor analyses, score reliability
CHBQ (IMAQ)	Lie, 2004⁶¹	CAM	USA	Medical students	Cross-sectional	(282) 272	No	No	No/no	2a	Literature based, expert group	Score reliability, internal consistency reliability (comparison with IMAQ)
CHBQ (extended)	Lie, 2006²⁹	CAM	USA	Medical students, interns, and faculty	Cross-sectional and longitudinal (intervention)	(n.r.) 667	No	(Yes)	No/no	2a	Conducted by adapting items from Hopper 1998⁶² and Schneider 2003,³⁵ additional questions	Not revalidated
CHBQ	Riccard, 2008⁶³	CAM	USA	Medical students	Cross-sectional	(355) 95	No	No	No/no	2a	Retained	Not revalidated
CHBQ (extended)	James, 2016⁴³	CAM	Sierra Leone	Medical, pharmacy, and nursing students	Cross-sectional	(944) 874	(Yes)	(Yes)	No/no	2a, (2b, 3)	Additional questions: literature-based, expert group	Not revalidated
CHBQ, Danish (modified and extended)	Damgaard-Mørch, 2008⁵⁵	CAM	Denmark	Medical students	Cross-sectional	(508) 470	No	(yes)	No/(yes)	2a, (2b, 3)	Modified for Danish environment, additional questions	Not revalidated
CHBQ, Serbian (modified and extended)	Jakovljevic, 2013³⁴	CAM	Serbia	Medical, dentistry, and pharmacy students and professionals	Cross-sectional	(n.r.) 707	No	(Yes)	No/no	2a, (3)	Literature research, CHBQ translated into Serbian, adjusted to local environments	Not revalidated
CHBQ, Chinese (extended)	Xie, 2020²⁷	CAM	China	Medical and nonmedical students (subjects n.r.)	Cross-sectional	(n.r.) 2,292	No	(Yes)	No/(yes)	2a, (3)	Translation into Chinese, additional questions	Not revalidated
CHBQ, German	Rotter, 2021⁶⁴	CAM	Germany	Medical students	Cross-sectional	(515) 278	No	No	No/no	2a	Translation into German (back-translation to English), addition of frequently used CAM methods	Revalidated⁶⁵
HCAMQ Turkey (extended)	Kirsoy, 2021⁶⁶	Holistic CAM (TCM)	Turkey	Medical students	Cross-sectional	(912) 825	(Yes)	(Yes)	(Yes/yes)	2a, (2b)	Developed by Hyland, 2003⁶⁷ via literature research and earlier pilot studies, additional questions	(⁶⁷: exploratory factor analysis, re-test reliability, internal consistency, divergent validity) score reliability
IAPSU, Brazilian	Teixeira, 2019⁴⁴	Health promotion inclusive CAM practice	Brazil	Health professional students	Cross-sectional	(544, stratified sample) 512	No	Yes	No/no	2b	Developed by Fontenelle Catrib 2015⁶⁸ via expert group	(⁶⁸: interrater-reliability) not revalidated
IMAQ	Schneider, 2003³⁵	CAM/IM combined with openness and relationship	USA	Medical students, health care providers, and physicians	Cross-sectional	(870) 196	No	No	No/no	2a	Literature based, expert group	Exploratory factor analysis and score reliability
IMAQ (modified and extended)	Schmidt, 2005⁴¹	CAM/IM combined with holism, introspection and doctor-patient relationship	Australia, Canada, China, Hong Kong, New Zealand, UK, USA	Medical students	Cross-sectional	(1154) 639	No	No	No/no	2a	Modified for international survey based on a pilot study	Confirmatory factor analysis, internal consistency, score reliability
IMAQ (modified and extended)	Rees, 2009⁴²		New Zealand, UK, USA	Medical students	Cross-sectional	(487) 154	No	No	No/no	2a	Modified version, Schmidt 2005⁴¹	Confirmatory factor analysis, score reliability
IMAQ (modified and extended)	Flaherty, 2015⁶⁹		Ireland	Medical students	Cross-sectional	(468) 308	No	(Yes)	No/no	2a	Additional questions	Not revalidated
IMAQ and CHBQ (modified and extended)	Kanadiya, 2012⁵⁶		USA	Osteopathic medical students	Cross-sectional	(4003) 635	No	(Yes)	No/no	2a (2b, 3)	Both instruments were modified, additional questions	Score reliability
QACAM	Ditte, 2011⁷⁰	CAM	Germany	Medical and psychology students	Intervention	(392) 233	No	No	No/no	2a	Literature based on QAPT: Ditte 2006, Schmitt-Ott 2003^71,72 and FAN: Augustin 1995⁷³	Factor analyses, score reliability
Treatment preference questionnaire	Furnham, 1999⁴⁵	CAM (combined with: Health beliefs and lifestyle, Scientific beliefs, Attitudes to CM practitioners questionnaire)	UK	Medical, social science and arts students	Cross-sectional	(n.r.) 248	No	No	No/no	2a, 3	Literature based	Exploratory factor analysis
n.r.	Furnham, 1993⁷⁴	CAM combined with health issues	UK	Medical students	Cross-sectional	(n.r.) 209	No	No	No/no	2a	Literature based, expert group, patient interviews	Exploratory factor analysis
n.r.	Furnham, 1995⁷⁵	CAM	UK	Medical students	Cross-sectional	(n.r.) 180	No	Yes	No/no	2a	Modified version of Furnham’s 1986⁷⁶, additional questions	Content analysis, cluster analysis, exploratory factor analysis, score reliability
n.r.	Furnham, 2003⁷⁷	CAM	UK	Medical students	Cross-sectional	(n.r.) 311	No	Yes	No/no	2a, 2b	Self-administered	Exploratory factor analysis
n.r.	Iktidar, 2022²⁶	CAM	Bangladesh	Medical and nonmedical students (e.g., literature, engineering, social law)	Cross-sectional	(n.r.) 576	Yes	Yes	No/no	2a	Self-administered	Content validity, reliability and internal consistency (pretest)
n.r.	Jocham, 2017a⁷⁸	Homeopathy and acupuncture	Germany	Medical students	National survey, cross-sectional	(n.r.) 333	Yes, + qual^e	Yes + qual	No/no	2a	Self-administered	Confirmatory factor analysis of the attitude measure, score reliability
n.r.	Jocham, 2017b⁷⁹				Controlled intervention	Intervention group: (n.r.) 333 control group (344) 315	Yes	No	No/no	2a

Extended: additional questions were added.

Area: as stated by the authors, areas in brackets not within a validated scale.

Data in brackets: additional questions, not within a validated scale.

Learning outcome described with Kirkpatrick level: 1 = reaction, satisfaction, 2a = attitude, 2b = knowledge and skills, 3 = behavioral change or intention to change.

+ qual: additional open questions.

CAIMAQ, Complementary Alternative and Integrative Medicine Attitude Questionnaire; CAM, complementary and alternative medicine; CHBQ, CAM Health Belief Questionnaire; FAN, Freiburg Questionnaire on Attitudes toward Alternative Medical Procedures; HCAMQ, Holistic Complementary and Alternative Medicine Attitude Scale; IAPSU, Instrumento de Avaliação da Promoçao da Saúde na Universidade; IM, Integrative Medizine; IMAQ, Integrative Medicine Attitude Questionnaire; n.r., not reported; TCM, Traditional Chinese Medicine; QACAM, Questionnaire on Attitudes toward Complementary Medical Treatment; QAPT, Questionnaire on Attitudes Toward the Use of Psychotherapeutic Help.

Table 4.

Qualitative Evaluation Instruments for Complementary and Integrative Medicine Training

First author, publication year	Area^a	Country	Population	Study design	Number of participants	Type of educational approach/ duration	Data collection	Outcome measures					Instrument development
First author, publication year	Area^a	Country	Population	Study design	Number of participants	Type of educational approach/ duration	Data collection	Interest in learning	Personal experience	Learning experience/ didactic issues^b	Learning outcome^a	Individual benefit	Instrument development
Lehmann, 2014⁵⁰	Homeopathy	Germany	Medical students	Intervention	30	Elective course/56h	Focus group interviews	Yes	Yes	No/no	1	Yes	Self-administered
Lie, 2008⁴⁶	CAM	USA	Medical students	Cross-sectional	49	Integrated CAM curriculum/20h required, plus elective courses	Focus group interviews	No	Yes	No/yes	2b, 3	No	Self-administered, process of internal revision in a consensus process
Frenkel, 2007⁸⁰	CIM	USA	Medical students	Intervention	4	Longitudinal courses within the curriculum/n.r.	Individual interviews	No	No	Yes/no	2a, 2b	No	Self-administered
McKenzie, 2012⁸¹	Mindfulness	Australia	Medical and psychology students	Cross-sectional	171	Course with a series of lectures and 6 tutorials/n.r.	Questionnaire	Yes	No	No/no	2a, 3	No	Self-administered
Saunders, 2007⁵¹	MBM	USA	Medical students	Intervention	82	Elective mind-body-skills-course / 22 h (11 weeks)	Questionnaire	Yes	Yes	No/yes	1, 2a, 2b, 3	Yes	n.r.
Sultana, 2021⁵²	Alternative Systems of Medicine	India	Medicine and surgery Bachelor students (MBBS)	Intervention	100	General lectures of alternative systems of medicine /n.r.	Questionnaire	Yes	No	No/yes	1, 3	No	n.r.
Wong, 2006⁵³	TCM	Hong Kong	Medical students	Intervention	28	TCM clinic visit/ afternoon session	Focus group interviews	Yes	No	Yes/yes	1, 2a, 2b, 3	No	n.r.

Area as stated by the authors.

Kirkpatrick level 1 = reaction, satisfaction, 2a = attitude, 2b = knowledge and skills, 3 = behavioral change or intention to change.

CAM, complementary and alternative medicine; CIM, complementary and integrative medicine; MBM, mind body medicine; n.r., not reported; TCM, Traditional Chinese Medicine.

Quality of the validated evaluation instruments

For the assessment of the quality of the validated instruments using a standardized list,²⁴ the following two instruments were not included because they used a validated instrument from another study without revalidation, which was not originally aimed at medical students: Kirsoy⁶⁶ used the English-language questionnaire developed by Hyland,⁶⁷ which has been validated for interviewing patients, for medical students in Turkish. Teixeira et al.⁴⁴ used the Assessment Tool for Health Promotion at the University (IAPSU)⁶⁸ without revalidation. This questionnaire is self-administered and used by Brazilian researchers after data collection in nursing and physiotherapy courses.

The analysis of the 12 included instruments showed that only two achieved more than half of the total number of points.^35,77 While most of the instruments scored well in terms of content validity and internal consistency and had no floor/ceiling effects, there were considerable shortcomings in the areas of criterion validity, reproducibility, and responsiveness. The results of the analysis are shown in Supplementary Table S6. The lack of retesting is also evident in the translated instruments. The validated Complementary and Alternative Medicine Health Belief Questionnaire (CHBQ)⁶¹ was translated into Danish, Serbian, Chinese, and German. The German translation was revalidated by Hinse et al.⁶⁵ using exploratory factor analysis, score, and split half reliability, but no further validation was carried out for the other translated questionnaires. Even when untranslated questionnaires were used repeatedly, they were often not revalidated, even though some questions were added and/or the target group was changed.^{29,43,58,63,69}

Discussion

The aim of this systematic review was to provide an overview of the quantitative and qualitative instruments available to evaluate CIM courses and programs in undergraduate medical education and to assess the quality of the validated instruments. A total of 100 studies were identified that included currently available evaluation instruments. Of these, 14 were validated, 7 were qualitative, and 66 were nonvalidated, primarily quantitative. Twelve validated original instruments were assessed in detail for quality. Both the low proportion of validated instruments overall and the results of the quality assessment show that there is a paucity of high-quality instruments.

Limitation

Due to database restrictions our results may be biased. We primarily searched English-language databases, so it is possible that additional instruments exist in other regions or have not been published. This may also explain the preponderance of studies from English-speaking countries. However, it is notable that there are only a few publications from Australia and Canada, and we found relatively few duplicates in five databases, despite similar search strategies. It is possible that the inclusion of other databases would have resulted in even more hits. When selecting the literature, there were many studies that did not include the complete evaluation tool or did not provide it as an appendix. In this case, the individual parts of the tool were reconstructed from the article, in particular from the tables and figures. It is possible that some aspects were overlooked. Many of the instruments were not primarily used to evaluate teaching. Their suitability for this purpose should be further investigated. In assessing the quality of the instruments, we made only a limited assessment of the quality of the underlying studies. This could have added further aspects to the quality assessment.

Study design

The studies identified are mainly cross-sectional studies that have been used to measure attitudes. Kirkpatrick emphasizes that the assessment of attitudes, knowledge, and behavioral changes requires a pre–post design and the use of a control group.²³ None of the studies met these criteria. Some of the studies were conducted as national surveys with a very large number of respondents. The aim of these studies was to determine the baseline situation rather than to evaluate a course. Nevertheless, we assume that the instruments could also be used for a pre–post study. In the case of the many nonvalidated cross-sectional and intervention studies with more than 100 participants, the opportunity for validation was missed.

Outcome measures of the instruments

Most of the instruments identified focused on assessing students’ attitudes and perceptions, and none of the validated instruments captured actual changes in behavior or patient care. This finding is consistent with other studies that have examined outcomes in CIM education. For example, Quartley et al.¹⁴ also found a lack of studies that objectively assessed behavioral change and longer term objective outcomes. Soliman and Bilszta¹² noted that the instruments used mainly assessed student satisfaction, learning success through self-assessment, but rarely effectiveness in terms of changing clinical practice. The qualitative instruments included in this review covered the widest range of outcomes. In general, qualitative research aims to understand the full experiences of research participants,⁸² so qualitative evaluation can provide insight into a fuller understanding of the educational intervention, including leading to a wider range of targeted outcomes. Qualitative research is often criticized for a lack of validity or rigour.^82,83 However, qualitative evaluation methods have been found to use a number of best practices through the use of direct data sources, structured data collection instruments, nonleading questions, and expert evaluators.⁸² The analysis of best practices in qualitative evaluation in ethics education revealed that other best practices were missing: Data should be collected using multiple sources, methods, raters, and time points, and triangulation analyses should be used to assess convergence.⁸² These weaknesses need to be addressed by qualitative evaluation methods in CIM education to include multiple sources (students, patients or standardized patients, faculty), multiple methods (e.g., free-text options in questionnaires, short interviews, focus groups), multiple raters (e.g., faculty, examiners, students themselves), and time points (e.g., mid-course, at the end of the course, months or years later). Qualitative evaluation methods make an important contribution to identifying the advantages and disadvantages of different educational interventions.⁸⁴ However, evaluation methods need to be chosen according to their appropriateness for the purpose, particularly in relation to the resources available and the intended use.⁸⁵

Quality assessment

The quality assessment of the 12 validated questionnaires showed that most of them were of poor to medium quality. Relatively high scores were obtained for internal consistency. This may be because only a small proportion of the total number of instruments were categorized as validated and therefore included in this quality assessment. The lack of criterion validity, reproducibility, responsiveness, and interpretability may indicate that even many of the validated instruments described were developed to assess the quality of individual courses at the respective medical faculty and therefore do not claim general validity. The focus of some studies may not have been primarily on developing a valid instrument that could be used universally. As a result, there are few gold standards, group comparisons, or references to the general interpretation of the results. This would be necessary if instruments are to be used in medical education research to provide cross-cutting evidence, for example, by comparing the effectiveness of different educational interventions. Other systematic reviews evaluating questionnaires also show low scores, possibly for the same reason. For example, Orfanos et al.²⁵ used the same quality assessment tool²⁴ to rate the quality of eight therapeutic group process questionnaires. Overall scores ranged from 2 to 6 out of 18, with an average score of 3. Quartey et al. conducted a systematic review of CIM training for health professionals¹⁴ and assessed the overall quality of the 10 relevant studies identified using the Medical Education Research Study Quality Instrument.⁸⁶ They found only moderate study quality, with the lowest mean scores for validity of evaluation tools and sampling. Another reason may be that further validation of some instruments is planned, but has not yet been carried out or published. However, the validation process of some of the validated instruments we identified was not continued in further applications in the assessment of CIM teaching.^11,12 Zhao et al.⁸⁷ used an instrument designed for the assessment of cross-sectional studies for their systematic review on CIM teaching among nursing students. The assessment tool did not include items to assess the survey instrument. However, in all 26 included studies also no follow-up was obtained.

Almost all identified instruments were originally developed and published in English. Some of the validated instruments were translated. Translation requires revalidation,⁸⁸ which has only be done in the German version of the CHBQ.⁶⁵ Validation in general is not about properties that can be determined once and for all; even a series of studies cannot achieve this. Reliability, and especially validity, are incremental, never-ending processes, as scales are continually used with different groups of people and in different settings, and their psychometric properties need to be established with them.⁸⁹ It is therefore important for researchers and teachers to embark on this journey and to develop their instruments in a theory-based and informed way. The evaluation instruments identified in our review were often not used a second time, and when they were, they were not revalidated despite other target groups and modifications.

Future research

Based on our findings, future research should develop high-quality and comprehensive instruments for evaluating undergraduate medical education in CIM. The currently validated questionnaires, which mainly capture general attitudes toward CIM, contribute little to the development of CIM teaching. However, they can help to identify learning needs. Currently, the courses offered in the faculties vary widely in scope and content, so it is important to find out which teaching methods and content are useful and actually contribute to students’ skills development and improve patient care. This will require more intervention studies, ideally with a before and after survey and follow-up. To demonstrate the benefits of CIM teaching, it is necessary to include questions which not only cover the intended change in behavior but also the actual change in behavior and patient care outcomes. In addition, comparative studies could be conducted to examine the effects of different didactic implementations and settings. This also requires researchers to look beyond their own teaching responsibilities and (further) develop and validate instruments that can be used across faculties. Structured multiple use of these newly developed instruments could contribute to a deeper knowledge of CIM teaching programs and their impact on student competencies and patient care.

Conclusions

This systematic review shows that there are still no reliable, comprehensive instruments for the evaluation of CIM programs. Although many instruments have been identified that can be used in undergraduate medical education, there are still no validated instruments to assess higher learning outcomes such as behavioral changes. The structured tabulation of the 100 identified studies with evaluation instruments and the identification of existing gaps is a first milestone for the structured development of high-quality and comprehensive questionnaires. The identified qualitative instruments cover the broadest range of outcomes and could be used for the development of new instruments. A systematic evaluation of CIM training and its impact is necessary to further develop CIM training and to demonstrate what works for whom and under what conditions.

Footnotes

Acknowledgment

The authors would like to thank Katrin Schüttpelz-Brauns for her methodological advice on conducting the systematic review.

Authors’ Contributions

A.H.: Conceptualization, data curation, investigation, methodology, validation, visualization, and writing—original draft. G.R.: Data curation, investigation, methodology, and writing—review and editing. M.T.: Data curation, investigation, methodology, and writing—review and editing. K.F.: Data curation, investigation, methodology, and writing—review and editing. B.S.-S.: Conceptualization, investigation, methodology, supervision, validation, and writing—review and editing.

Author Disclosure Statement

The authors declare that they have no conflicts of interest.

Funding Information

Language editing was supported by the German Association for Medical Education. The authors received no funding.

Supplementary Material

Supplementary Table SA1

Supplementary Table SA2

Supplementary Table SA3

Supplementary Table SA4

Supplementary Table SA5

Supplementary Table SA6

References

Kemppainen

, Kemppainen

, Reippainen

, et al. Use of complementary and alternative medicine in Europe: Health-related and sociodemographic determinants. Scand J Public Health, 2018; 46(4):448–455; doi: 10.1177/1403494817733869

Kristoffersen

, Quandt

, Stub

. Use of complementary and alternative medicine in Norway: A cross-sectional survey with a modified Norwegian version of the international questionnaire to measure use of complementary and alternative medicine (I-CAM-QN). BMC Complement Med Ther, 2021; 21(1):93; doi: 10.1186/s12906-021-03258-6

Møller

, Ekholm

, Christensen

. Trends in the use of complementary and alternative medicine between 1987 and 2021 in Denmark. BMC Complement Med Ther, 2024; 24(1):23; doi: 10.1186/s12906-023-04327-8

Fjær

, Landet

, McNamara

, et al. The use of Complementary and Alternative Medicine (CAM) in Europe. BMC Complement Med Ther, 2020; 20(1):108; doi: 10.1186/s12906-020-02903-w

Jeitler

, Ortiz

, Brinkhaus

, et al. Use and acceptance of traditional, complementary and integrative medicine in Germany-an online representative cross-sectional Study. Front Med (Lausanne), 2024; 11:1372924; doi: 10.3389/fmed.2024.1372924

Guillaud

, Darbois

, Allenet

, et al. Predictive factors of complementary and alternative medicine use in the general population in Europe: A systematic review. Complement Ther Med, 2019; 42:347–354; doi: 10.1016/j.ctim.2018.12.014

Tan

, Guo

, Chow

. Complementary and alternative medicine (CAM) use and its association with antibiotic usage practices: A nationally representative population study. Int J Antimicrob Agents, 2023; 61(6):106796; doi: 10.1016/j.ijantimicag.2023.106796

Hoenders

, Ghelman

, Portella

, et al. A review of the WHO strategy on traditional, complementary, and integrative medicine from the perspective of academic consortia for integrative medicine and health. Front Med (Lausanne), 2024; 11:1395698; doi: 10.3389/fmed.2024.1395698

National Center for complementary and Integrative Health. Complementary, Alternative, or Integrative Health: What’s in a Name? Available from: https://www.nccih.nih.gov/health/complementary-alternative-or-integrative-health-whats-in-a-name

10.

Brinkhaus

, Esch

. Integrative Medizin und Gesundheit. MWV Medizinisch Wissenschaftliche Verlagsgesellschaft: Berlin; 2021.

11.

Joyce

, Wardle

, Zaslawski

. Medical student attitudes towards complementary and alternative medicine (CAM) in medical education: A critical review. J Complement Integr Med, 2016; 13(4):333–345; doi: 10.1515/jcim-2014-0053

12.

Soliman

, Bilszta

. Teaching complementary and alternative medicine in undergraduate medical education: A scoping review. Int J Med Educ, 2021; 12:140–149; doi: 10.5116/ijme.60e2.f3ed

13.

Stock-Schröer

, Huber

, Joos

, et al. Evaluation of the current status of Rehabilitation, Physical Medicine and Naturopathy education 10 years after the reform of the Medical Licensure Act - a nationwide survey of German Medical Universities. GMS J Med Educ, 2017; 34(1):Doc3; doi: 10.3205/zma001080

14.

Quartey

, Ma

, Chung

, et al. Complementary and alternative medicine education for medical profession: Systematic review. Evid Based Complement Alternat Med, 2012; 2012:656812; doi: 10.1155/2012/656812

15.

Homberg

, Scheffer

, Brinkhaus

, et al. Naturopathy, complementary and integrative medicine in medical education – position paper by the GMA Committee Integrative Medicine and Perspective Pluralism. GMS J Med Educ, 2022; 39(2):Doc16; doi: 10.3205/zma001537

16.

Homberg

, Krug

, Klafke

, et al. Consensus views on competencies and teaching methods for an interprofessional curriculum on complementary and integrative medicine: A Delphi study. J Integr Med, 2021; 19(3):282–290; doi: 10.1016/j.joim.2021.03.001

17.

Keene

, Heslop

, Sabesan

, et al. Knowledge, attitudes and practices of health professionals toward complementary and alternative medicine in cancer care – a systematic review. J Communication in Healthcare, 2020; 13(3):205–218; doi: 10.1080/17538068.2020.1755202

18.

Prill

, Steinhäuser

, Herchenröder

, et al. Determinants of interprofessional collaboration in complementary medicine to develop an educational module “complementary and integrative medicine” for undergraduate students: A Mixed-Methods Study. J Interprof Care, 2022; 36(3):390–401; doi: 10.1080/13561820.2021.1935810

19.

Haji

, Morin

, Parker

. Rethinking programme evaluation in health professions education: Beyond ‘did it work?’. Med Educ, 2013; 47(4):342–351; doi: 10.1111/medu.12091

20.

Cleland

. Exploring versus measuring: considering the fundamental differences between qualitative and quantitative research. In: Researching Medical Education. John Wiley & Sons: 2015, pp. 1–14.

21.

Page

, McKenzie

, Bossuyt

, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 2021; 372:n71; doi: 10.1136/bmj.n71

22.

Cohen

. A coefficient of agreement for nominal scales. Educ Psychol Meas, 1960; 20(1):37–46; doi: 10.1177/001316446002000104

23.

Kirkpatrick

. Great ideas revisited. Techniques for evaluating training programs. Revisiting Kirkpatrick’s Four-Level Model. Training and Development, 1996; 50(1):54–59.

24.

Terwee

, Bot

, de Boer

, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol, 2007; 60(1):34–42; doi: 10.1016/j.jclinepi.2006.03.012

25.

Orfanos

, Burn

, Priebe

, et al. A systematic review and quality assessment of therapeutic group process questionnaires. Int J Group Psychother, 2020; 70(3):425–454; doi: 10.1080/00207284.2020.1755292

26.

Iktidar

, Roy

, Liza

, et al. Knowledge, attitude, perceived effectiveness and self-practice of complementary and alternative medicine: A cross-sectional comparison between medical and non-medical students of Bangladesh. BMC Complement Med Ther, 2022; 22(1):342; doi: 10.1186/s12906-022-03797-6

27.

Xie

, Sang

, Li

, et al. A survey on perceptions of complementary and alternative medicine among undergraduates in China. Evid Based Complement Alternat Med, 2020; 2020:9091051; doi: 10.1155/2020/9091051

28.

Singh

, Boudville

, Corderoy

, et al. Impact on the dermatology educational experience of medical students with the introduction of online teaching support modules to help address the reduction in clinical teaching. Australas J Dermatol, 2011; 52(4):264–269; doi: 10.1111/j.1440-0960.2011.00804.x

29.

Lie

, Boker

. Comparative survey of Complementary and Alternative Medicine (CAM) attitudes, use, and information-seeking behaviour among medical students, residents & faculty. BMC Med Educ, 2006; 6:58.

30.

, Lu

, Ruan

. Application of PBL method and LBL method in the teaching of Acupuncutre and Moxibustion. Zhongguo Zhen Jiu, 2015; 35(8):831–834.

31.

Owen

, Lewith

. Complementary and Alternative Medicine (CAM) in the undergraduate medical curriculum: The Southampton experience. Med Educ, 2000; 35(1):73–77.

32.

Kreitzer

, Mitten

, Harris

, et al. Attitudes toward CAM among medical, nursing, and pharmacy faculty and students: A comparative analysis. Altern Ther Health Med, 2002; 8(6):44–47, 50–53.

33.

Zhu

, Wu

, Zhang

, et al. A national survey on how to improve traditional Chinese medicine learning internationally: Perceptions from both teachers and students. Integr Med Res, 2022; 11(4):100895; doi: 10.1016/j.imr.2022.100895

34.

Jakovljevic

, Djordjevic

, Markovic

, et al. Cross-sectional survey on complementary and alternative medicine awareness among health care professionals and students using CHBQ questionnaire in a Balkan country. Chin J Integr Med, 2013; 19(9):650–655; doi: 10.1007/s11655-013-1434-6

35.

Schneider

, Meek

, Bell

. Development and validation of IMAQ: Integrative medicine attitude questionnaire. BMC Med Educ, 2003; 3:5; doi: 10.1186/1472-6920-3-5

36.

Shani-Gershoni

, Freud

, Press

, et al. Knowledge and attitudes of internists compared to medical students regarding acupuncture. Isr Med Assoc J, 2008; 10(3):219–223.

37.

Steele

, Baker

, Boxwell

, et al. Community-based osteopathic manipulative medicine student clinic: Changes in curriculum and student confidence levels. J Am Osteopath Assoc, 2005; 105(11):503–513.

38.

Abbott

, Hui

, Hays

, et al. Medical student attitudes toward complementary, alternative and integrative medicine. Evid Based Complement Alternat Med, 2011; 2011:985243; doi: 10.1093/ecam/nep195

39.

Kolkman

, Visser

, Vink

A-M

, et al. CAM-education in the medical curriculum: Attitude towards and knowledge of CAM among Dutch medical students. European J Integrative Med, 2011; 3(1):17–22; doi: 10.1016/j.eujim.2011.02.005

40.

Liu

, Huynh

N-T

, Broukhim

, et al. Determining the attitudes and use of complementary, alternative, and integrative medicine among undergraduates. J Altern Complement Med, 2014; 20(9):718–726; doi: 10.1089/acm.2014.0041

41.

Schmidt

, Rees

, Greenfield

, et al. Multischool, international survey of medical students’ attitudes toward “holism. Acad Med, 2005; 80(10):955–963; doi: 10.1097/00001888-200510000-00017

42.

Rees

, Wearn

, Dennis

, et al. Medical students’ attitudes to complementary and alternative medicine: Further validation of the IMAQ and findings from an international longitudinal study. Med Teach, 2009; 31(2):125–132; doi: 10.1080/01421590802139724

43.

James

, Bah

, Kondorvoh

. Exploring self-use, attitude and interest to study Complementary and Alternative Medicine (CAM) among final year undergraduate medical, pharmacy and nursing students in Sierra Leone: A Comparative Study. BMC Complement Altern Med, 2016; 16:121; doi: 10.1186/s12906-016-1102-4

44.

Teixeira Medeiros

, Fontenelle Catrib

, Anchieta Mendes Melo

, et al. Academic education in health profession programs, knowledge and use of Complementary and Alternative Medicine (CAM) by university students. Complement Ther Med, 2019; 44:189–195; doi: 10.1016/j.ctim.2019.03.006

45.

Furnham

, Yardley

, Fahmy

, et al. Health beliefs and preferences for medical treatment: A comparison between medical and social science students. Complement Ther Med, 1999; 7(2):101–109; doi: 10.1016/s0965-2299(99)80088-0

46.

Lie

, Shapiro

, Pardee

, et al. A focus group study of medical students_ views of an integrated Complementary and Alternative Medicine (CAM) Curriculum”, “Students Teaching Teachers. Med Educ Online, 2008; 13(3):1–13.

47.

Dossett

, Kohatsu

, Nunley

, et al. A medical student elective promoting humanism, communication skills, complementary and alternative medicine and physician self-care: An evaluation of the HEART program. Explore (NY), 2013; 9(5):292–298; doi: 10.1016/j.explore.2013.06.003

48.

Bailey

, Chudgar

, Engle

, et al. The impact of a mandatory immersion curriculum in integrative medicine for graduating medical students. Explore (NY), 2015; 11(5):394–400; doi: 10.1016/j.explore.2015.07.003

49.

Samuels

, Shaham

, Schiff

, et al. From the “what” to the “how”: Teaching integrative medicine-related skills to medical students during COVID-19. Patient Educ Couns, 2022; 105(7):2256–2263; doi: 10.1016/j.pec.2021.10.022

50.

Lehmann

, Krémer

, Werwick

, et al. Homeopathy as elective in undergraduate medical education—An opportunity for teaching professional core skills. GMS Z Med Ausbild, 2014; 31(1):Doc7; doi: 10.3205/zma000899

51.

Saunders

, Tractenberg

, Chaterji

, et al. Promoting self-awareness and reflection through an experiential Mind-Body Skills course for first year medical students. Med Teach, 2007; 29(8):778–784; doi: 10.1080/01421590701509647

52.

Sultana

, Sadanandam

, Roopa

, et al. Effectiveness of including “alternative systems of medicine” in foundation course of first MBBS students according to CBME (New curriculum) for 2019 batch. Int J Toxicological Pharmacological Res, 2021; 11(3):6–14.

53.

Wong

, Lee

, Wong

, et al. Strengths, weaknesses, and development of Traditional Chinese Medicine in the health system of Hong Kong: Through the eyes of future Western doctors. J Altern Complement Med, 2006; 12(2):185–189; discussion 191–192; doi: 10.1089/acm.2006.12.185

54.

Karpa

. Development and implementation of an herbal and natural product elective in undergraduate medical education. BMC Complement Altern Med, 2012; 12:57; doi: 10.1186/1472-6882-12-57

55.

Damgaard-Mørch

, Nielsen

, Uldall

. Knowledge and perceptions of complementary and alternative medicine among medical students in Copenhagen. Ugeskr Laeger, 2008; 170(48):3941–3945.

56.

Kanadiya

, Klein

, Shubrook

Jr . Use of and attitudes toward complementary and alternative medicine among osteopathic medical students. J Am Osteopath Assoc, 2012; 112(7):437–446.

57.

McKenzie

, Hassed

, Gear

. Medical and psychology students’ knowledge of and attitudes towards mindfulness as a clinical intervention. Explore (NY), 2012; 8(6):360–367; doi: 10.1016/j.explore.2012.08.003

58.

Singh

, Kamath

. Attitude of medical students and doctors towards complementary, alternative and integrative medicine: A single-center, Questionnaire-Based Study. J Pharmacopuncture, 2021; 24(2):84–90; doi: 10.3831/KPI.2021.24.2.84

59.

Ahmad

, Naqvi

, Ahmad

, et al. Awareness, perception, attitude, and knowledge regarding Complementary and Alternative Medicines (CAMs) among the pharmacy and medical students of a public university in Saudi Arabia. Arch Pharma Pract, 2017; 8(2):51–63.

60.

Frye

, Sierpina

, Boisaubin

, et al. Measuring what medical students think about Complementary and Alternative Medicine (CAM): A pilot study of the complementary and alternative medicine survey. Adv Health Sci Educ Theory Pract, 2006; 11(1):19–32; doi: 10.1007/s10459-005-1587-y

61.

Lie

, Boker

. Development and validation of the CAM Health Belief Questionnaire (CHBQ) and CAM use and attitudes amongst medical students. BMC Med Educ, 2004; 4:2.

62.

Hopper

, Cohen

. Complementary therapies and the medical profession: A study of medical students’ attitudes. Altern Ther Health Med, 1998; 4(3):68–73.

63.

Riccard

, Skelton

. Comparative analysis of 1st, 2nd, and 4th year MD students’ attitudes toward Complementary Alternative Medicine (CAM). BMC Res Notes, 2008; 1:84; doi: 10.1186/1756-0500-1-84

64.

Rotter

, Jerzynski

, Hinse

, et al. The attitude of medical students toward complementary medicine: Results of a Cross-Sectional Study. J Altern Complement Med, 2021; 27(12):1124–1130; doi: 10.1089/acm.2021.0181

65.

Hinse

, Jerzynski

, Binting

, et al. German translation and validation of the complementary and alternative medicine health belief questionnaire (CHBQ). BMC Med Educ, 2024; 24(1):99; doi: 10.1186/s12909-023-04985-9

66.

Kirsoy

, Pirincci

, Kurt

. Knowledge, attitude and behavior of medical faculty students about traditional and complementary. Ahi Evran Med J, 2021; doi: 10.4633/2/aemj.972230

67.

Hyland

, Lewith

, Westoby

. Developing a measure of attitudes: The holistic complementary and alternative medicine questionnaire. Complement Ther Med, 2003; 11(1):33–38; doi: 10.1016/s0965-2299(02)00113-9

68.

Fontenelle Catrib

, Bitar da Cunha Olegario

, Gardano Bucharles Mont’Alverne

, et al. Desenvolvimento e reprodutibilidade do instrumento de avaliação da promoção da saúde na universidade - IAPSU. Revista Brasileira em promoção da Saúde, 2015; 28(3):305–317; doi: 10.5020/18061230.2015.p305

69.

Flaherty

, Fitzgibbon

, Cantillon

. Attitudes of medical students toward the practice and teaching of integrative medicine. J Integr Med, 2015; 13(6):412–415; doi: 10.1016/s2095-4964(15)60206-0

70.

Ditte

, Schulz

, Ernst

, et al. Attitudes towards complementary and alternative medicine among medical and psychology students. Psychol Health Med, 2011; 16(2):225–237; doi: 10.1080/13548506.2010.532559

71.

Ditte

, Schulz

, Schmid-Ott

. Attitude towards psychotherapy in the Russian population and in the population with a Russian/Soviet cultural background in Germany. A Pilot Study. Nervenarzt, 2006; 77(1):64–72; doi: 10.1007/s00115-005-1889-6

72.

Schmid-Ott

, Reibold

, Ernst

, et al. Development of a questionnaire to assess attitudes towards psychotherapeutic treatment. Dermatol Psychosom, 2003; 4(4):187–193; doi: 10.1159/000075905

73.

Augustin

, Zschocke

, Buhrke

. Attitudes and prior experience with respect to alternative medicine among dermatological patients: The Freiburg questionnaire on attitudes to naturopathy (FAN). Forsch Komplementarmed, 1999; 6 (Suppl 2):26–29; doi: 10.1159/000057144

74.

Furnham

. Attitudes to alternative medicine: A study of the perceptions of those studying orthodox medicine. Complement Ther Med, 1993; 1(3):120–126.

75.

Furnham

, Hanna

, C

. Medical students’ attitudes to complementary medical therapies. Complement Ther Med, 1995; 3(4):212–219.

76.

Furnham

. Medical students’ beliefs about nine different specialties. Br Med J (Clin Res Ed), 1986; 293(6562):1607–1610; doi: 10.1136/bmj.293.6562.1607

77.

Furnham

, McGill

. Medical students’ attitudes about complementary and alternative medicine. J Altern Complement Med, 2003; 9(2):275–284; doi: 10.1089/10755530360623392

78.

Jocham

, Berberat

, Schneider

, et al. Why do students engage in elective courses on acupuncture and homeopathy at medical school? A Survey. Complement Med Res, 2017; 24(5):295–301; doi: 10.1159/000468539

79.

Jocham

, Kriston

, Berberat

, et al. How do medical students engaging in elective courses on acupuncture and homeopathy differ from unselected students? A survey. BMC Complement Altern Med, 2017; 17(1):148; doi: 10.1186/s12906-017-1653-z

80.

Frenkel

, Frye

, Heliker

, et al. Lessons learned from complementary and integrative medicine curriculum change in a medical school. Med Educ, 2007; 41(2):205–213.

81.

McKenzie

, Hassed

, Gear

. Medical and psychology students’ knowledge of and attitudes towards mindfulness as a clinical intervention. Comparatove Study, 2012.

82.

, Pyo

, Lee

, et al. Qualitative research in healthcare: Data analysis. J Prev Med Public Health, 2023; 56(2):100–110; doi: 10.3961/jpmph.22.471

83.

Watts

, Todd

, Mulhearn

, et al. Qualitative evaluation methods in ethics education: A systematic review and analysis of best practices. Account Res, 2017; 24(4):225–242; doi: 10.1080/08989621.2016.1274975

84.

Rortveit

, Saetre Hansen

, Joa

, et al. Qualitative evaluation in nursing interventions-A review of the literature. Nurs Open, 2020; 7(5):1285–1298; doi: 10.1002/nop2.519

85.

Patton

. Qualitative Research & Evaluation Methods : Integrating Theory and Practice. SAGE: Los Angeles; London; New Delhi; Singapore; Washington DC; 2015.

86.

Reed

, Cook

, Beckman

, et al. Association between funding and quality of published medical education research. JAMA, 2007; 298(9):1002–1009.

87.

Zhao

, Kennedy

, Cleary

, et al. Knowledge about, attitude toward, and practice of complementary and alternative medicine among nursing students: A systematic review of cross-sectional studies. Front Public Health, 2022; 10:946874; doi: 10.3389/fpubh.2022.946874

88.

Fourie

, Feinauer

. The quality of translated medical research questionnaires. Southern African Linguistics and Applied Language Studies, 2005; 23(4):349–367; doi: 10.2989/16073610509486395

89.

Streiner

, Kottner

. Recommendations for reporting the results of studies of instrument and scale development and testing. J Adv Nurs, 2014; 70(9):1970–1979; doi: 10.1111/jan.12402

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.03 MB

0.02 MB