Game-Based Learning Interventions to Foster Cross-Cultural Care Training: A Scoping Review

Abstract

Objective:

Differences in cultural background between health providers and patients can reduce effective access to health services in multicultural settings. Health sciences educators have recently suggested that game-based learning may be effective for cross-cultural care training. This scoping review maps published knowledge on educational games intended to foster cross-cultural care training and highlights the research gaps for future research.

Materials and Methods:

A scoping review searched PubMed, Eric, Embase, Lilacs, PsycINFO, and Google Scholar for theoretical and empirical research, using terms relevant to cross-cultural care and game-based learning. A participatory research framework engaged senior medical students and participatory research experts in conducting and evaluating the review.

Results:

Forty-one documents met the inclusion criteria, all from developed countries. The most common source of publication was nursing and medicine (39%; 16/41) and used the cultural competence approach (44%; 18/41). Around one-half of the publications (51%; 21/41) were theoretical and 39% (16/41) were empirical. Empirical studies most commonly used mixed methods (44%; 7/16), followed by strictly quantitative (31%; 5/16) or qualitative (25%; 4/16) approaches. There were no randomized controlled trials and only one study engaged end-users in the design. Empirical studies most frequently assessed role-play-related games (44%; 7/16) and used game evaluation-related outcomes or learning-related outcomes. None used patient-oriented outcomes. Findings suggest that educational games are an effective and engaging educational intervention for cross-cultural care training.

Conclusions:

The paucity of studies on educational games and cross-cultural care training precludes a systematic review. Future empirical studies should focus on randomized counterfactual designs and patient-related outcomes. We encourage involving end-users in developing content for educational games.

Introduction

Cultural diversity is an asset to societies that can also pose challenges to health care.¹ Without proper training, differences in cultural background between service providers and patients hinder access to health services in multicultural settings.² At worst, these differences can lead to confrontation with and discrimination against patients, accentuating racial/ethnic health disparities.³

The nursing sciences have advanced work on cultural diversity in health sciences education. In the 1960s, Madeleine Leininger, a nurse theorist and anthropologist, noticed the missing relationship between culture and health outcomes in nursing care practice and education.⁴ She coined the term culturally congruent care to call attention on the need to reconsolidate emic/indigenous and etic/outsider care phenomena.⁵

Later concepts and lexicon included cultural competence, cultural awareness, cultural sensitivity, cultural humility, and cultural safety.^6,7 Some inconsistencies and debates continue about the exact meaning, boundaries, and application of each of these terms, and some authors use them interchangeably.⁸ There is growing agreement, however, about the need to train medical students to provide care that is congruent with the cultural needs of patients.^9,10 The 2015 Standards for Accreditation of Medical Education Programs in Canada¹¹ calls for cultural competence training and the Royal College of Physicians and Surgeons of Canada will require all residency programs to include cultural safety training in their curricula.¹²

Many medical schools include cross-cultural care education, although with variation in the content and teaching approaches.¹³ There is no agreement about the most effective way to provide cross-cultural care training for health professionals.¹⁴ Health sciences educators have recently suggested that game-based learning may be effective for cross-cultural care training.¹⁵

Early evidence of games for cross-cultural care education occurred out of the context of health sciences education. Since the 1970s, simulation games such as BaFá BaFá and Barnga¹⁶ allowed players to experience intercultural tensions and thus to foster cross-cultural skills. Yet, contemporary use of educational games for cross-cultural training in health sciences education is not widespread.¹⁶

To the best of our knowledge, no previous literature reviews using systematic methods have explored the evidence on educational games to foster cross-cultural care training. This scoping review maps current knowledge on educational games intended to foster cross-cultural care training and to highlight the research gaps that should be addressed in future research. For the purposes of this review, the term cross-cultural care encompasses cultural competence, cultural awareness, cultural sensitivity, cultural safety, cultural humility, cross-cultural training, and other aligned concepts.

Methods

Our scoping review followed methods proposed by Arksey and O'Malley¹⁷ and Levac et al.¹⁸ to (1) identify the research question; (2) identify relevant studies; (3) select the studies; (4) chart the data; and (5) collate, summarize, and report the results. The review sought to answer the question: what is the extent and nature of the literature on educational games to foster cross-cultural care training of health sciences students? We developed a protocol for this scoping review in advance (available from the authors on request).

Inclusion/exclusion criteria

Our inclusion criteria were as follows: (1) publication theme was health sciences education; (2) publication examined a game-based learning intervention; (3) learning topic was cross-cultural care training (or related terms); and (4) language of publication was English, Spanish, Italian, Portuguese, or French.

There is no standard definition for games¹⁹; we followed the categories proposed by the 2006 Horizon Report²⁰ used in related systematic reviews^21,22: role-playing or simulations (strategies to replicate real-life situations); virtual environments (interactive computer-based systems); social and cooperative games (based on social interaction such as board games); and alternative reality games, which blend gameplay and real life. We were especially interested in exploring role-playing and simulations since they often trigger conflictive emotions that favor cross-cultural training.^15,23

Search strategy

The search included PubMed, Eric (EBSCO), Embase (OVID), Lilacs, PsycINFO (OVID), and Google Scholar. An experienced health sciences librarian based at McGill University reviewed, modified, and approved the search strategy. Appendix Table A1 shows our search strategy.

The review included research in the fields of medicine, nursing, psychology, social work, occupational therapy, and physical therapy. We included terms of cultural safety, cultural competence, cultural sensitivity, cultural humility, cultural awareness, and cross-cultural medicine. We also included the aligned concepts proposed by Horvat,¹⁴ such as patient-centered care, person-centered care, family-centered care, patient engagement, equitable health care, and patient participation.

We included theoretical and empirical research, published or not, as well as different sources of data, including online databases, references lists, and conferences, as suggested by Arksey and O'Malley¹⁷ and Levac et al.¹⁸

Study selection and data extraction

Using the open-source systematic review web application Rayyan,²⁴ two independent reviewers (A.M. and D.R.) performed the initial title and abstract screening. These research assistants solved discrepancies by discussion involving a third party (J.P.) in case of no resolution. Subsequently, we retrieved the full-text articles of all the selected references and removed the duplicates using Endnote X8.2.

Two independent reviewers (A.A. and J.P.) performed the final selection of studies using an eligibility format based on the inclusion criteria. We calibrated this format on 10% of the retrieved studies to ensure clarity of inclusion criteria.

The next step involved reviewing and charting the included studies. Charting is a “technique for synthesizing and interpreting qualitative data by sifting, categorizing, and sorting material according to key issues and themes.”(p15)¹⁷ Using Google Sheets, the research team designed the data charting form based on the variables that would answer the research question. This step involved regular meetings to discuss and update the data charting form.

We followed the “descriptive-analytical method,” which involves applying a common analytical framework and collecting standard information on each document.¹⁷ First, two independent reviewers (A.M. and D.R.) calibrated the data of 5% of the studies to determine whether our approach to data extraction was appropriate to answer the research question. Subsequently, the reviewers extracted all data and a third reviewer (J.P.) verified the accuracy of data extraction.

We extracted the following data when available: basic study information (title, year of publication, country, authors, and journal); type of document (journal article, book chapter, book, thesis, conference abstract, or technical report); study type (qualitative, quantitative, mixed-methods, or theoretical); reported study design (observational, experimental, systematic review, comment/reflection, case study, or qualitative descriptive); discipline (medicine, psychology, occupational therapy, physical therapy, nursing, or social work); type of game (e.g., board game, videogame, simulation, and role-play); and basic characteristics of each game.

Synthesis and presentation of results

We present the results of the scoping review following the categories proposed by Grudniewicz et al.²⁵: (1) a summary of the characteristics and distribution of included studies and (2) a narrative synthesis and mapping of results. We generated tables to display an overview of the included studies and created a narrative synthesis describing the characteristics of game-based interventions to foster cross-cultural care training. In this article, we followed the PRISMA extension for reporting scoping reviews (PRISMA-ScR).²⁶

Participatory research

Our participatory research framework²⁷ engaged knowledge users through all stages of the study. Our team included three senior medical students interested in cross-cultural care training (A.A., D.R., and A.M.) along with four participatory research experts (J.P., A.-M.C., A.C., and N.A.). The stakeholders participated by framing the research question, codesigning the study, collecting and analyzing data, and reporting the results of the study. We held six 2-hour online meetings to discuss each stage of the study. Engaging end-users ensured the research design was aligned with their interests and needs.²⁸

Results

Publication statistics

The final list of publications included 41 documents (Fig. 1 and Appendix Table A2). The national affiliation of the first author included 24 from the United States, eight from Canada, two from Sweden, and one each from Australia, Finland, Germany, Greece, Israel, Saudi Arabia, and the United Kingdom. Twenty documents were published between 2013 and 2017, eight documents between 2007 and 2012, seven documents between 1998 and 2005, and six documents between 1988 and 1997. All the documents were in English.

FIG. 1.

Flow diagram of the scoping review.

Table 1 depicts the basic characteristics of included publications. Some 90% (37/41) of the documents were related to cross-cultural care training and the remainder to patient-centered care. Similarly, 44% (18/41) of the documents used the cultural competence approach, while 24% (10/41) used a nonspecific cultural framework.

Table 1.

Characteristics of Included Publications

Characteristics	Categories	n = 41	%
Topic	Cross-cultural care training	37	90
Topic	Patient-centered care	4	10
Conceptual framework	Cultural competence	18	44
	Cultural awareness	3	7
	Cultural sensitivity	1	2
	Acculturation	1	2
	Cultural—nonspecific	10	24
	More than one	4	10
	Patient-centered care	4	10
Type of document	Journal article	30	73
	Conference abstract	4	10
	Thesis	4	10
	Book chapter	2	5
	Technical report	1	2
Study type	Theoretical	21	51
	Empirical
	Quantitative	5	12
	Qualitative	4	10
	Mixed methods	7	17
	Other
	Literature review	2	5
	Theses	2	5
Discipline of first author's affiliation	Nursing	8	20
	Medicine	8	20
	Business and information technology	5	12
	Public health and health sciences	5	12
	Education	4	10
	Psychology	2	5
	Social work	2	5
	Management	1	2
	Not specified	6	15

Most publications (73%; 30/41) were journal articles, followed by conference abstracts, thesis, book chapters, and technical reports. About half of included documents (51%; 21/41) were theoretical publications, and 39% (16/41) were empirical publications. Empirical publications most commonly used mixed methods (44%; 7/16), followed by strictly quantitative (31%; 5/16) or qualitative (25%; 4/16) studies.

We analyzed theoretical and empirical publications separately. We included four additional publications, two literature reviews, and two graduate theses that we report in a separate study-type category because they do not fit into either theoretical or empirical categories.

Regarding the discipline of the first author's affiliation, the majority of the documents came from nursing and medicine (39%; 16/41), followed by business and information technology, public health and health sciences, education, psychology, social work, and management.

Empirical studies

Table 2 shows the characteristics of the 16 published empirical studies. About one-third (31%; 5/16) reported before-and-after designs, while others reported experiments, mixed-methods design, and qualitative descriptive studies (two studies each category). Five studies (31%) included a third time point or follow-up measurement. None of the studies used a randomized controlled trial design and none of them used a multivariate approach to explore potentially confounding variables.

Table 2.

Characteristics of Empirical Publications

Reference, game type, and game name (when available)	Study type/reported design	Statistical test/qualitative method	Participants	Goal of the game	Main findings
Bai et al. (2016)^a	Qualitative	Not mentioned	Eight undergraduate and graduate students in social work and nursing	“engage students and demonstrate cultural difference.”(p76)	“The pedagogical strategies were found to be effective in increasing cross-cultural learning, as reflected in the course evaluations.”
Role-play	Qualitative	Not mentioned		“engage students and demonstrate cultural difference.”(p76)
Barber-Madden and Glanz (1989)	Quantitative	Not conducted	25 occupational health nurses	“increase participant sensitivity to sociocultural dimensions of lifestyle and health care.”(p129)	Increased sensitivity to sociocultural dimensions of health care: mean = 6.1 SD = 0.9 (Likert scale 7 = highest level of attainment; 1 = lowest level of attainment)
Board game
Trivia-Public Health and Medicine - USSR Edition
Brown et al. (1992)^a	Qualitative	Not mentioned	Five family physicians	“learn the patient-centered method of care which requires behavioral change and skill acquisition”(p183)	“The results of the course evaluation indicated that the teaching methods were generally viewed by the participants as acceptable and effective.”
Role-play	Qualitative	Not mentioned	Five family physicians
Ellman et al. (2012)	Mixed methods	Kruskal–Wallis test and Mann–Whitney U. Content analysis	205 medical, 65 nursing, 39 divinity students.	“identify spiritual and cultural needs of patients and understand how to meet these needs.”(p1241)	Greater understanding of the role of culture in a patient's experience at the end of life: mean = 4.16 (divinity); 4.1 (med); 4.36 (nursing); P = 0.179 (Likert scale 5 = strongly agree; 1 = strongly disagree).
Simulation	Mixed methods	Kruskal–Wallis test and Mann–Whitney U. Content analysis	205 medical, 65 nursing, 39 divinity students.
Harding and D'Eon (2001)^a	Mixed methods/before-and-after	Paired t-test and MANOVA. Qualitative analysis method not reported	Medical undergraduates (number of students not mentioned)	“Introduce medical students to patient-centered interviewing.”(p1)	Pieces of information recalled after 7 months: mean = 2.74 (Lego Details) 1.03 (Details of Anecdotes) P < 0.001; 1.5 (Lego Concepts) 0.33 (Concepts from Anecdotes) P < 0.001.
Simulation
Lego-Based Communication Simulation
Khan (2015)	Quantitative/experiment	Not conducted	Five volunteer health care professionals and nine students (not specified)	“low-fidelity, cultural competence web-based serious game intended to serve as a teaching tool for medical students, and practitioners.”(p24)	Interaction with the system was easy (7.3), satisfying (6.5), and flexible (4.8) (scale 1 = not at all; 10 = to a great extent). 21% of the participants believed to be immersed into the game.
Virtual patient simulation^b
Fydlyty
Kiosses et al. (2017)^a,c	Quantitative/controlled before-and-after	Cohen's d	87 medical undergraduates	“empathy training for medical undergraduates, based on the principles of Person-Centered Approach.”(p20)	Jefferson Scale of Physician Empathy mean score and SD before, after, and at 6-month follow-up was 109.3 (12.7), 121.1 (9), and 121.1 (9.5), respectively. Before-after and before-follow-up difference was significant in both cases.
Role-play
Empathize with me, Doctor!
Koskinen et al. (2008)	Mixed methods	Mann–Whitney U and Wilcoxon signed-rank test and content analysis	23 nursing, 40 public health nursing, and 40 dental hygienist students	“provoke emotional confusion in learners, thus sensitising them to perceive the world from the perspective of a culturally different person.”(p55)	Playing: gave me new insights into different cultures 1.83 (Barnga) 1.57 (BaFá BaFá) P = 0.000; changed my thinking about cultures 2.28 (Barnga) 2.07 (BaFá BaFá) P = 0.002; enabled me to become a member of another culture 1.89 (Barnga) 2.93 (BaFá BaFá) P = 0.000; Confirmed my own self-awareness 2.26 (Barnga) 2.35 (BaFá BaFá) P = 0.097; provided me with new personal insights 2.56 (Barnga) 2.54 (BaFá BaFá) P = 0.747. (Likert scale 1 = strongly agree; 4 = strongly disagree).
Simulation^b
Barnga and BaFá BaFá
Kratzke and Bertolo (2013)	Qualitative/descriptive qualitative study	Content analysis	11 community health students	Not mentioned	“The results show that students may enhance their cultural competence in the classroom in academic preparation.”
Simulation
BaFá BaFá
Mao et al. (1988)	Quantitative/before-and-after	Paired t-test	415 medical undergraduates	“improve the students' ability to provide medical care appropriate to the ethnic and sociocultural backgrounds of their patients”(p625)	68% of the students found the content useful in 1986 and 93% describe it as outstanding/very good in 1987. 49% found the role-playing exercises useful in 1986 and 94% rated interactions among group members outstanding/very good in 1987. In 1986, 94% approved the use of student discussion leaders and 85% enjoyed the use of video-taped scenarios. 70% felt that the workshop achieved its objectives in 1986 and 1987.
Role-play	Quantitative/before-and-after	Paired t-test	415 medical undergraduates
Odreman (2016)^c	Quantitative/quasiexperimental study	Paired t-test	196 health care students from nursing, paramedic, dental assisting, and occupational/physical therapy assisting	“allow observation of how a healthcare professional uses cultural knowledge to provide transculturally congruent care.”(p11)	Learners' Transcultural Self-Efficacy: mean (SD): pretest 9.2 (61.2) posttest 33.6 (49.5). P < 0.05.
Role-play^b	Quantitative/quasiexperimental study	Paired t-test
Ong-Flaherty et al. 2017	Qualitative—descriptive qualitative study	Thematic analysis	34 graduate nursing students and 11 doctoral clinical psychology students	“[Create] cultural awareness among graduate students in nursing and clinical psychology”(p151)	“Participants experienced heightened emotions, and with guided reflection, were able to empathize with others “different” from themselves, an indication of cultural humility.”
Simulation
BaFá BaFá
Pantziaras et al. (2012)	Mixed-methods/mixed-methodological approach	Inductive content analysis and statistical analysis not conducted	Eight residents in psychiatry and one general practitioner	“enhance clinical, interpersonal, social and cultural competence”(p133)	Virtual patient system's realistic nature score (median): 5; Virtual patient's ability to mirror the course of a real clinical investigation score: 5. Likert scale (1 = highly disagree to 7 = highly agree)
Virtual patient simulation^b
Virtual Mrs. K
Pantziaras et al. (2014)	Mixed methods/mixed methodology	Inductive content analysis and sign test	32 residents in psychiatry	“enhance clinical, interpersonal and intercultural competence”(p109)	Experienced realism: 4 (median value); engagement 4.5 (median value); concentration 5 (median value). Likert scale: (1 = highly disagree; 5 = highly agree).
Virtual patient simulation^b
Virtual Mrs. K
Pruegger and Rogers (1994)^a,c	Mixed methods/controlled before-and-after	Adjusted F statistic and content analysis of students' reflections	71 undergraduate psychology students	“promote cross-cultural sensitivity and to develop cross-cultural awareness”(p375)	Cross-Cultural Sensitivity Scale mean and SD: pretest 43.61 (6.59); posttest 43.61 (6.06); follow-up 43.79 (6.52). P > 0.05
Simulation^b
BaFá BaFá
Zamboanga et al. (2016)	Mixed methods/before-and-after	Paired t-test and thematic analysis	39 undergraduate psychology students	“simulate the process of acculturation and some of the cultural adjustment challenges associated with it”(p244)	Mean and SD. Students' awareness that acculturation is a real-world phenomenon: pretest 6.79 (1.69) posttest 8.46 (1.45) P < 0.001; their understanding of how acculturation can impact people's lives: pretest 3.97 (0.78) postest 4.56 (0.55) P < 0.001 (Scale: 1 = strongly disagree; 5 = strongly agree); and their sensitivity (pretest 7.54 (1.86) postest 8.51 (1.47) P < 0.001) and empathy for people who face some of the challenges associated with acculturation: pretest 8.33 (1.53) posttest 9.08 (1.20) P < 0.001. Scale: 1 = unaware, no empathy, no sensitivity; 10 = extremely aware, high empathy, high sensitivity.
Role-play	Mixed methods/before-and-after	Paired t-test and thematic analysis	39 undergraduate psychology students

Included follow-up assessment.

Game-based learning intervention exclusively.

Included control group.

MANOVA, multivariate analysis of variance; SD, standard deviation.

Four studies (25%) included students from more than one health profession and the others included students from a single health profession. More than one-third of interventions involved nursing students (37%; 6/16), followed by medical students or residents (31%; 5/16), social work students (12%; 2/16), and psychology students (12%; 2/16). Other students included in the studies were paramedic, dental hygiene, community health, divinity, occupational therapy, and physical therapy students.

We found only one publication using a participatory research framework to develop content for a cross-cultural care training intervention. In Mathew's study,²⁹ a community advisory board that comprised cultural, clinical, and educational experts led the development of the intervention's content.

The most common game type assessed by empirical studies was role-play-related games (44%; 7/16), followed by simulation (five publications), video patient simulation (three publications), and board games (one publication). Six studies (37%) used a game-based learning intervention exclusively, while 10 mixed the game with other interventions, such as theory lectures, in-class discussions, field trips, reading assignments, workshops, and online modules.

Researchers reported a wide range of outcome types. We organized the studies' outcomes in two broad categories. Seven studies used course or game evaluation-related outcomes and seven studies used learning-related outcomes. No studies used patient-oriented outcomes.

All mixed-methods or quantitative methods (12/12) studies used at least one Likert-type questionnaire, alone or combined with another method, to assess the impact of the intervention. One-half of these studies (6/12) reported validation of their instrument.

Researchers used a variety of statistical tests to analyze their data: the most frequent was a paired t-test (three publications), followed by Mann–Whitney U, Kruskal–Wallis, multivariate analysis of variance, Wilcoxon signed-rank, f-adjusted, and Cohen's d tests. None used more complex modeling to account for potential confounding effects. Three studies used content analysis, and one study used thematic analysis.

Of the studies using mixed or quantitative methods (n = 12), the total number of participants was 1300 (mean = 118.2; median = 71; standard deviation = 133.8; range = 9–415). Two-thirds (8/12) of these publications measured the effect before and after the intervention and the remaining measured the effect exclusively after the intervention. Only three empirical studies used a control group (without randomization).

Publications on theory

We identified 21 publications on conceptual or theoretical aspects of cross-cultural care. Ten of these (48%) used the cultural competence approach and seven (33%) used a nonspecific cultural approach. The publications discussed aspects of the games BaFá BaFá, Barnga, Fydlyty, HealthCare DIVERSOPHY, Take a risk? Virtual patient system, Ecotonos, and Multi-player cultural competence serious game, OWARE, MOSHI, Game-Based Cognitive Behavioral Group Therapy, and Breast Care Bingo. We describe the majority of these games in Table 3.

Table 3.

Basic Description of Selected Games

Game name/type	Game objective/no. of players	Story/narrative	Game dynamics	Rewards/incentives	Penalties	Targeted group
BaFá BaFá^a/simulation	Stimulate self-reflection of personal responses to intercultural differences/>11	There are two different cultures: Alpha and Beta. Alpha values sharing and physical contact, while Beta is individualistic and competitive.	Members of one culture visit the other one in pairs to try to make sense of it. They report back to their team after each interaction. The game elicits strong emotional reactions among participants.	Communication improves when cultural norms are respected.	Penalties are imposed when cultural norms are violated.	Originally targeted naval personal. Currently used in multiple professions.
Barnga^a/simulation	Reflect on communication problems in intercultural situations/>7	NA	Tournament-like card game. Participants are not allowed to speak. They switch tables at the end of each game. They do not know that the rules at each table are different, thus experiencing conflictive emotions.	Highest card wins a trick. The player who wins more tricks wins the game and moves to the next table (clockwise).	Losers move to the next table (counterclockwise).	Multiple professions.
Fydlyty^a/virtual patient simulation	Improve cultural competence skills through simulation/single player	The player, who takes the role of a physician, examines an 83-year-old retired nurse (a virtual patient), Li Chen. She complains of “toothache- like” chest pain.	The objective of the player is to complete a dialogue script with the virtual patient, focusing on the cultural background and responsive mood of the patient. The system uses a set of predefined responses to the player, who must select the most appropriate options to conduct the medical interview successfully.	Selecting the appropriate response allows the player to earn points and move ahead in the conversation until the interaction ends.	The virtual patient exhibits upset and angry moods based on the player's wrong decisions.	Educators/curriculum designers, and students/trainees. Mostly physicians.
HealthCare DIVERSOPHY^a/board game	Develop cultural competence of health professionals who interact with major U.S. ethnic groups/>2	Players go from ethnocentricity to wisdom about other cultures as they complete the game successfully.	Players roll dice and move around a board. They must overcome five challenges described on the cards: DiversiSMARTS cards develop players' knowledge of specific ethnic groups; DiversiCHOICE cards teach players how to handle intercultural tensions; DiversiSHARE cards promote self-reflection and cultural awareness of players; DiversiRISK cards allow players to experience the challenges of cross-cultural medicine; DiversiGUIDE cards offer advice to handle communication issues with multicultural patients.	Successful players collect dividends (diversiCOINS).	Players can fall in four traps: ethnocentricity, stereotypes, bias, and assimilation.	Health professionals and health care administrators.
Virtual Mrs. K^a/virtual patient simulation	Enhance clinical, interpersonal, and intercultural competence/single player	A female Bosnian refugee patient, “Mrs. K,” comes to see the doctor (represented by the player) with severe mental trauma, post-traumatic stress disorder, and depression.	The system uses prerecorded video clips and allows user interaction in five areas: medical interview, physical examination, screening instruments, and laboratory and imaging tests.	Automated feedback by the virtual patient and by a virtual advisor.	Patient's reactions such as leaving the examination room, crying, and coughing. Automated feedback by the virtual patient and by a virtual advisor.	Health professionals (residents in psychiatry).
Take a risk?/simulation	Prepare students for cultural competence practice with members of oppressed populations/groups of five to seven students	Instructors can use scenarios related to gender, religious belief, physical ability, and race/ethnicity, and they can create other situations.	Groups of players are confronted with situations including (1) the background and setting; (2) the challenging situation; and (3) the question, “Do you take action by … or would you remain silent?” Students use two index cards labeled “Take Action” and “Remain Silent.” If players take action, they roll the dice.	A roll of one to four represents a positive outcome and the group earns four points. This rewards their decision to challenge oppression. Neutral outcomes (roll of three to six) receive two points.	A roll of two to five represents a negative outcome (players lose two points).	Social work students.
Virtual patient system/virtual patient simulation	Allow medical students to develop interview skills when interacting with multicultural patients/single player	Students knock on the examination room door, enter, and start interacting with the virtual patient who is projected life-size on the room wall. The system uses virtual patients with diverse backgrounds.	Medical students interact with virtual patients in standard examination rooms. The system uses a projector, two cameras, a microphone, and speech recognition software. It tracks interaction differences such as empathy, body lean, interruptions, posture, and jargon use. Instructors can use this information to provide feedback to the students.	The system allows for assessing the effect of the patient's background on the interaction. Students can reflect on successful interactions.	Students can identify interaction biases based on the patients' background.	Medical students.
ECOTONOS/simulation	Explore expectations, group processes, and contradictions that may arise in intercultural interactions/multiplayer	Using a set of cards, players create a myth that explains the origins of its culture, their values and beliefs, and the way their culture supports survival and harmony.	Players internalize their culture and then are asked to solve a problem (case study or physical task). They are later blended into three different multicultural groups: majority/minority, balanced, and a complex mix of people from each culture. As participants bring their own way of addressing the issue, intercultural tensions arise.	Students reflect on the positive aspects of their way of dealing with intercultural tensions.	Students reflect on negative effects of their way to interact with other cultures.	Originally targeted corporations. Currently used in multiple professions.
Multi-player CCSG/videogame	Cultural competence education and training/supports online multiplayer	The game places the player in a hospital with several rooms. Each room/patient offers a specific cultural competence scenario. Educators can create or modify scenarios to simulate several culturally sensitive situations.	Players (taking the role of health professionals) interact with patients. The system uses prescripted dialogues. The player can choose from a list of responses/actions and must choose the correct answer at each step.	Correct answers will trigger a high-frequency sound effect and the player receives feedback.	An incorrect answer will anger, sadden, or make the patient upset. The player receives feedback.	Students in medical and nursing schools and health professionals.

Games mentioned in at least two publications. The rest of the games are mentioned in only one publication.

CCSG, cultural competence serious game; NA, not applicable.

Other publications

We identified four additional publications. Two of these were literature reviews (one systematic and one scoping review) that included two of the studies that we included in this scoping review. The two remaining publications, an MSc thesis and a PhD thesis, described the development of content for a future game-based learning intervention.

Game-based interventions

Several review publications described as many as five games. Selected by reviewer consensus, we included four additional innovative games that might be of interest for researchers and educators. Table 3 describes the dynamics of each game.

Discussion

This is the first review using systematic methods to collate evidence about educational games in cross-cultural care training. No impact assessment used a randomized controlled trial design and none reported patient-orient outcomes. Only one reported engagement of end-users in the design. All publications come from high-income developed countries, possibly prompted by immigration and the consequent increase in cultural diversity of these regions.¹⁰ In developing countries, game-based interventions and research to promote cross-cultural care are much less prominent, indicating a need to advance in this direction.

Educational games to foster cross-cultural care training

The majority of games in our review were role-playing and simulation, emulating cultural conflicts to make the players more aware of their own cultural backgrounds and prejudices.²³ Pyburn suggests that simulation provides learners with a safe environment for practicing cross-cultural situations that may be emotionally charged.¹⁵ Several games used virtual patient simulation systems (e.g., Fydlyty and Virtual Mrs K), an approach that is gaining attention due to its reported effectiveness, appropriateness, and the increasing use of technology in health care sciences education.³⁰ Our study highlights the advantages of using role-playing and simulation for cross-cultural care training.

Most studies reported a complementary teaching method such as theory lecture or in-class discussion in addition to game-learning to foster cross-cultural care training. This is in line with earlier recommendations that, to be most effective, educational games must be embedded in a multistep program comprising background knowledge on the topic, face-to-face training, and the game itself.³¹

Two games, BaFá BaFá and Barnga, accumulated most evidence in health sciences education research. BaFá BaFá was originally developed for creating cultural awareness in the U.S. Navy, while Barnga was developed to reflect on normative assumptions and cross-cultural communication. The games have been used by sectors other than health care and their incorporation into health professional training is recent.³² Due to their reported effectiveness and availability, these games could be a good starting point for educators interested in exploring this type of training.

Cultural approaches: need for engaging end-users

Cultural competence—sensitizing students or health professionals about another culture—is the most frequent cultural focus of the publications reviewed. Some authors criticized the concept of cultural competence, widespread in the United States, as it deals with beliefs and behaviors of heterogeneous cultural blocks such as African American, Latino, and indigenous patients. Betancourt argued that cultural competence “can lead to stereotyping and oversimplification of culture without respect for its complexity.”(p145)²

The relatively newly popularized concept of cultural safety is an opportunity to approach complexity of culture in cross-cultural care training. Cultural safety starts with recognition that the patient's ways of knowing have validity; the patient is a partner in the health care decision-making process; and the patient determines whether the approach to care is culturally safe or not.³³ Cultural safety encourages codesign of culturally safe interventions by engaging end-users early in the research process.³⁴

End-users are better resources in matters of their own cultures and therefore better positioned to know what a cross-cultural care intervention should include. None of the game-based interventions identified in our scoping review used the cultural safety approach, and only one used a participatory framework.

Our participatory research approach helped the medical students involved to deepen their research interests and refine their research skills. The three students who contributed to this review (A.A., D.R., and A.M.) participated in an international conference³⁵ to share the protocol of the scoping review, and they prepared and submitted the article to a peer-reviewed journal. Earlier scoping reviews have used participatory research to include the perspectives of stakeholders throughout the study, to obtain additional sources of information, and to spearhead end-of-study knowledge translation.²⁵

Gaps in empirical research

Our review highlights the urgent need for more empirical research on the effectiveness of educational games in training for cross-cultural care, particularly for unbiased counterfactual studies. We found two broad types of outcomes in the studies included in our review: course evaluation outcomes and learning-related outcomes. Experts in cross-cultural care training, however, recommend use of patient-related outcomes to evaluate whether cross-cultural care interventions actually benefit patients.¹⁴

Several authors of the studies included in this review suggested that educational games are effective and engaging in cross-cultural care training. Empirical studies included in the review report only P-values exploring differences between the means of groups of participants, precluding meta-analysis.

Limitations

This study shares common limitations of scoping reviews. We did not assess the quality of empirical publications nor did we do a quantitative data synthesis, both of which are outside the objectives of scoping reviews.¹⁷ Future studies should appraise the quality of publications and perform a synthesis of quantitative data.

One limitation of using Google Scholar is that search results may not be replicable.³⁶ The value of using this database is, however, that it collects gray literature.³⁷

The study generated a considerable quantity of data. Since we used a participatory research framework, our analysis focused on the interests and needs of our stakeholders. This approach may have left out some information of interest for other groups of readers. Interested researchers can consult the list of included publications.

We dealt with a large range of study designs, methodologies, and concepts. Although our tables provide transparency in aggregated findings, others concerned with educational games or cross-cultural care training may have taken a different analytical approach. We hope that this study can pave the way for future scoping reviews, and eventually systematic reviews and meta-analyses, exploring the topic.

Conclusion

Educational games for cross-cultural care training are in their infancy. The current state of the literature underlines the need for research in developing countries that lack cross-cultural care training despite traditional health care systems being prominent. Our review encourages exploration of methods that invite patients and end-users to codesign educational games, embracing a cultural safety approach. This will help to avoid oversimplification of culture and ensure that the interventions are better aligned with the patients' expectations and needs. Our review indicates a need for more empirical research to determine the effectiveness and acceptability of educational games for cross-cultural care training. Future studies should include a control group, randomization, and patient-related outcomes.

Our experience using a participatory research approach to conduct this scoping review was positive. Researchers conducting scoping reviews might consider a participatory research framework to improve the impact, appropriateness, and ownership of their research.

Footnotes

Acknowledgments

Germán Zuluaga, Andrés Isaza, Carolina Amaya, Andrés Cañón, and Camilo Correal provided methodological advice on cultural safety and medical education. Genevieve Gore provided advice for conducting the systematic literature search. Cassandra Laurie helped proofread the final version of the article and supported its write-up.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This study was funded by the CEIBA Foundation (Colombia), the Fonds de recherche du Québec—Santé (Canada), and Building 21 at McGill University. This did not influence the design and execution of the study.

Appendix

Appendix Table A2.

Publications Included in This Scoping Review

1.	Akakpo TF. Empowering clinicians to work with African immigrants through game playing. In: Bean RA, Davies SD, Davey MP, editors. Clinical supervision activities for increasing competence and self-awareness. Hoboken, NJ: John Wiley & Sons Inc; US; 2014. p. 219–224.
2.	Akl EA, Pretorius RW, Sackett K, Erdley WS, Bhoopathi PS, Alfarah Z, et al. The effect of educational games on medical students' learning outcomes: A systematic review: BEME Guide No 14. Med Teach. 2010;32(1):16–27. DOI: 10.3109/01421590903473969.
3.	Bai J, Larimer S, Riner ME. Cross-Cultural Pedagogy: Practical Strategies for a Successful Interprofessional Study Abroad Course. J Scholarsh Teach Learn. 2016;16(3):72. DOI: 10.14434/josotl.v16i3.19332.
4.	Barber-Madden R, Glanz K. Participatory learning in continuing nursing education: Soviet edition of a public health and medicine trivia game. J Contin Educ Nurs. 1989/05/01. 1989;20(3):128–131.
5.	Beig M, Mayer A, Chan C, Kapralos B, Dubrowski A. A Serious Game for Medical-Based Cultural Competence Education and Training. In: 2014 IEEE 14th International Conference on Advanced Learning Technologies. IEEE; 2014. p. 211–212. DOI: 10.1109/ICALT.2014.68.
6.	Brown JB, et al. Teaching the Patient-Centered Clinical Method in Primary Care: A Program for Community Physicians. J Contin Educ Health Prof. 1992;12(3):181–186.
7.	Conley CL, Deck SM, Miller JJ, Borders K. Improving the Cultural Competency of Social Work Students With a Social Privilege Activity. J Teach Soc Work. 2017;37(3):234–248. DOI: 10.1080/08841233.2017.1313804.
8.	Dearnley C, McClelland GT, Irving D. Innovation in teaching and learning in health higher education. The Higher Education Academy. London, England; 2013.
9.	Ellman MS, Schulman-Green D, Blatt L, Asher S, Viveiros D, Clark J, et al. Using Online Learning and Interactive Simulation To Teach Spiritual and Cultural Aspects of Palliative Care to Interprofessional Students. J Palliat Med. 2012;15(11):1240–1247. DOI: 10.1089/jpm.2012.0038.
10.	Ferdig RE, Coutts J, DiPietro J, Lok B, Davis N. Innovative technologies for multicultural education needs. Multicult Educ Technol J. 2007;1(1):47–63. DOI: 10.1108/17504970710745201.
11.	Flessa S. MOSHI: A Culture-Tailored Management Game for African Hospital Managers. Trop Doct. 2001/07/11. 2001;31(3):144–146. DOI: 10.1177/004947550103100309.
12.	Fowler SM, Pusch MD. Intercultural Simulation Games: A Review (of the United States and Beyond). Simul Gaming. 2010;41(1):94–115. DOI: 10.1177/1046878109352204.
13.	Gary R, Marrone S, Boyles C. The use of gaming strategies in a transcultural setting. J Contin Educ Nurs. 1998;29(5):221–227.
14.	Graham I, Richardson E. Experiential gaming to facilitate cultural awareness: its implication for developing emotional caring in nursing. Learn Heal Soc Care. 2008;7(1):37–45. DOI: 10.1111/j.1473-6861.2008.00168.x.
15.	Harding SR, D'Eon MF. Using a Lego TM -Based Communications Simulation to Introduce Medical Students to Patient-Centered Interviewing. Teach Learn Med. 2001;13(2):130–135. DOI: 10.1207/S15328015TLM1302_8.
16.	Hummel F, Peters D. Bafa’ Bafa’: a cultural awareness game. Nurse Educ. 1994/03/01. 1994;19(2):8.
17.	Khan ZT. A low-fidelity serious game authoring tool and educational network to facilitate medical-based cultural competence education. University of Ontario Institute of Technology (Canada); 2015.
18.	Khan Z, Kapralos B. A scenario and dialogue editor for a cultural competence serious game. In: 2014 IEEE Games Media Entertainment. IEEE; 2014. p. 1–2. DOI: 10.1109/GEM.2014.7048112.
19.	Khan Z, Kapralos B. A Scenario Editor and Authoring Tool for a Low Fidelity Cultural Competence Serious Game. In: CEUR Workshop Proceedings. 2015. p. 1–4.
20.	Khan Z, Kapralos B. A low-fidelity serious game for medical-based cultural competence education. Health Informatics J. 2017/07/22. 2017;146045821771956. DOI: 10.1177/1460458217719562.
21.	Kiosses VN, Tatsioni A, Dimoliatis ID, Hyphantis T. “Empathize with me, Doctor!” Medical Undergraduates Training Project: Development, Application, Six-months Follow-up. J Educ Train Stud. 2017;5(7):20. DOI: 10.11114/jets.v5i7.2418.
22.	Koskinen L, Abdelhamid P, Likitalo H. The simulation method for learning cultural awareness in nursing. Divers Heal Soc Care. 2008;5(1):55–63.
23.	Kratzke C, Bertolo M. Enhancing students' cultural competence using cross-cultural experiential learning. J Cult Divers. 2013;20(3):107–111.
24.	Lockhart JS, Resick LK. Teaching cultural competence. The value of experiential learning and community resources. Nurse Educ. 1997;22(3):27–31.
25.	Mao C, et al. A Workshop on Ethnic and Cultural Awareness for Second-Year Students. J Med Educ. 1988;63(8):624–628.
26.	Mathew L. Developing Content for an Online Virtual Interactive Simulation Case for Cultural Competency of Nursing Students in Caring for Puerto Ricans in New York City: A Community Based Participatory Research Approach. The University of Arzizona; 2015.
27.	Misurell JR, Springer C. Developing Culturally Responsive Evidence-Based Practice: A Game-Based Group Therapy Program for Child Sexual Abuse (CSA). J Child Fam Stud. 2013;22(1):137–149. DOI: 10.1007/s10826-011-9560-2.
28.	O'Connor BB, Rockney R, Alario A. BaFá BaFáTM: a cross-cultural simulation experience for medical educators and trainees. Med Educ. 2002;36(11):1102.
29.	Odreman HA. Videotaped role-play simulation in teaching transcultural self-efficacy to interprofessional healthcare students. Walden disertations and doctoral studies. Walden University; 2016.
30.	Ong-Flaherty C, Valencia-Garcia D, Martinez DA, Borges W, Summers L. Effectiveness of gaming in creating cultural awareness. Learn Cult Soc Interact. 2017;12:149–158. DOI: 10.1016/j.lcsi.2016.12.005.
31.	Pantziaras I, Courteille O, Mollica R, Fors U, Ekblad S. A pilot study of user acceptance and educational potentials of virtual patients in transcultural psychiatry. Int J Med Educ. 2012;3:132–140. DOI: 10.5116/ijme.5004.7c78.
32.	Pantziaras I, Fors U, Ekblad S. Virtual Mrs K: The learners expectations and attitudes towards a virtual patient system in transcultural psychiatry. J Contemp Med Educ. 2014;2(2):109. DOI: 10.5455/jcme.20140627042240.
33.	Pruegger VJ, Rogers TB. Cross-cultural sensitivity training: Methods and assessment. Int J Intercult Relations. 1994;18(3):369–387. DOI: 10.1016/0147-1767(94)90038-8.
34.	Pyburn R, Bauman EB. Striving for cultural competency by leveraging virtual reality and game-based learning. In: Game-based teaching and simulation in nursing and healthcare. New York, NY: Springer Pub. Co.; 2013. p. 147–176.
35.	Roubidoux MA, Hilmes M, Abate S, Burhansstipanov L, Trapp MA. Development of Computer Games to Teach Breast Cancer Screening to Native American Patients and Their Healthcare Providers. J Women's Imaging. 2005;7(2):77–86. DOI: 10.1097/01.jwi.0000168675.22467.f6.
36.	Salimbene S. HealthCare DIVERSOPHY—a cultural competence training tool for nurse executives. Aspens Advis Nurse Exec. 1998;13(10):10–12.
37.	Shearer R, Davidhizar R. Using Role Play To Develop Cultural Competence. J Nurs Educ. 2003;42(6):273–276.
38.	Takhsha M. Incorporating cultural content in nursing simulation scenarios. California State University, Stanislaus; 2015.
39.	Williams J, Rogers S. The multicultural workplace: preparing preceptors. J Contin Educ Nurs. 1993;24(3):101–104.
40.	Yeheskel A, Biderman A, Borkan JM, Herman J. A Course for Teaching Patient-Centered Medicine to Family Medicine Residents. Acad Med. 2000;75(5):494–497.
41.	Zamboanga BL, Ham LS, Tomaso CC, Audley S, Pole N. “Try Walking in Our Shoes”: Teaching Acculturation and Related Cultural Adjustment Processes Through Role-Play. Teach Psychol. 2016;43(3):243–249. DOI: 10.1177/0098628316649484.

References

Kirmayer

. Rethinking cultural competence. Transcult Psychiatry, 2012; 49:149–164.

Betancourt

, Corbett

, Bondaryk

. Addressing disparities and achieving equity: Cultural competence, ethics, and health-care transformation. Chest, 2014; 145:143–148.

Institute of Medicine (U.S.).. Committee on understanding and eliminating racial and ethnic disparities in health care. In: Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. Smedley

, Stith

, Nelson

, editors. Washington, DC: National Academies Press; 2003; 243 pp.

Leininger

. Some anthropological issues related to community mental health programs in the United States. Community Ment Health J, 1971; 7:50–61.

Leininger

. Culturally congruent care: Visible and invisible. J Transcult Nurs, 1994; 6:23.

Schim

, Doorenbos

, Benkert

, Miller

. Culturally congruent care: Putting the puzzle together. J Transcult Nurs, 2007; 18:103–110.

Brascoupe

, Waters

. Cultural safety: Exploring the applicability of the concept of cultural safety to aboriginal health and community wellness. J Aborig Heal, 2009; 5:6–41.

Cai

D-Y

. A concept analysis of cultural competence. Int J Nurs Sci, 2016; 3:268–273.

Liaison Committee on Medical

Education

. Functions and Structure of a Medical School: Standards for Accreditation of Medical Education Programs Leading to the MD Degree. Washington, DC; 2017. https://med.virginia.edu/ume-curriculum/wp-content/uploads/sites/216/2016/07/2017-18_Functions-and-Structure_2016-03-24.pdf (accessed December 17, 2019).

10.

General Medical Council. Tomorrow's Doctors—Outcomes and standards for undergraduate medical education. Manchester, United Kingdom; 2009. http://www.ub.edu/medicina_unitateducaciomedica/documentos/TomorrowsDoctors_2009.pdf (accessed December 17, 2019).

11.

Committee on Accreditation of Canadian Medical Schools. Standards for Accreditation of Medical Education Programs Leading to the MD Degree. Ottawa, ON; 2019. https://cacms-cafmc.ca/sites/default/files/documents/CACMS_Standards_and_Elements_-_AY_2019-2020.pdf (accessed December 17, 2019).

12.

Vogel

. Residency programs grapple with new Indigenous cultural safety training requirement. Can Med Assoc J, 2018; 190:E778–E779.

13.

Green

, Chun

MBJ

, Cervantes

, et al. Measuring medical students' preparedness and skills to provide cross-cultural care. Heal Equity, 2017; 1:15–22.

14.

Horvat

, Horey

, Romios

, Kis-Rigo

. Cultural competence education for health professionals. Cochrane Database Syst Rev, 2014; CD009405:1–100.

15.

Pyburn

, Bauman

. Striving for cultural competency by leveraging virtual reality and game-based learning. In: Game-Based Teaching and Simulation in Nursing and Healthcare, edited by Bauman

E.B.

, New York, NY: Springer Pub. Co.; pp. 147–176, 2013.

16.

Fowler

, Pusch

. Intercultural simulation games: A review (of the U.S. and beyond). Simul Gaming, 2010; 41:94–115.

17.

Arksey

, O'Malley

. Scoping studies: Towards a methodological framework. Int J Soc Res Methodol, 2005; 8:19–32.

18.

Levac

, Colquhoun

, O'Brien

. Scoping studies: Advancing the methodology. Implement Sci, 2010; 5:69.

19.

Plass

, Homer

, Kinzer

. Foundations of game-based learning. Educ Psychol, 2015; 50:258–83.

20.

Johnson

, Laurence

, Smith

. The 2006 Horizon Report. Stanford, CA: Horizon; 2006. https://library.educause.edu/-/media/files/library/2006/1/csd4387-pdf.pdf (accessed December 17, 2019).

21.

Akl

, Pretorius

, Sackett

, et al. The effect of educational games on medical students' learning outcomes: A systematic review: BEME Guide No 14. Med Teach, 2010; 32:16–27.

22.

Akl

, Kairouz

, Sackett

, et al. Educational games for health professionals. Cochrane Database Syst Rev, 2013; CD006411.

23.

Koskinen

, Abdelhamid

, Likitalo

. The simulation method for learning cultural awareness in nursing. Divers Heal Soc Care, 2008; 5:55–63.

24.

Ouzzani

, Hammady

, Fedorowicz

, Elmagarmid

. Rayyan—A web and mobile app for systematic reviews. Syst Rev, 2016; 5:210.

25.

Grudniewicz

, Nelson

, Kuluski

, et al. Treatment goal setting for complex patients: Protocol for a scoping review. BMJ Open, 2016; 6:e011869.

26.

Tricco

, Lillie

, Zarin

, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med, 2018; 169:467.

27.

Green

, George

, Daniel

, et al. Study of Participatory Research in Health Promotion: Review and Recommendations for the Development of Participatory Research in Health Promotion in Canada. Ottawa, ON: Royal Society of Canada; 1995. http://www.worldcat.org/title/study-of-participatory-research-in-health-promotion-review-and-recommendations-for-the-development-of-participatory-research-in-health-promotion-in-canada/oclc/32094281 (accessed December 17, 2019).

28.

Cargo

, Mercer

. The value and challenges of participatory research: strengthening its practice. Annu Rev Public Health, 2008; 29:325–350.

29.

Mathew

Developing Content for an Online Virtual Interactive Simulation Case for Cultural Competency of Nursing Students in Caring for Puerto Ricans in New York City: A Community Based Participatory Research Approach. Tucson, Arizona, United States. The University of Arizona; 2015.

30.

Liaw

, Chan

SW-C

, Chen

F-G

, et al. Comparison of virtual patient simulation with mannequin-based simulation for improving clinical performances in assessing and managing clinical deterioration: Randomized controlled trial. J Med Internet Res, 2014; 16:e214.

31.

Nagelvoort-Schuit

. Serious Gaming: The ABCDEsim. Serious Gaming—Erasmus University Rotterdam. 2017. https://www.coursera.org/learn/serious-gaming/lecture/ZgLRK/the-abcdesim (accessed December 17, 2019).

32.

Ong-Flaherty

, Valencia-Garcia

, Martinez

, et al. Effectiveness of gaming in creating cultural awareness. Learn Cult Soc Interact, 2017; 12:149–58.

33.

Allan

, Smylie

. First Peoples, Second Class Treatment: The Role of Racism in the Health and Well-Being of Indigenous Peoples in Canada. Toronto, ON: Wellesley Institute; 2015. http://www.wellesleyinstitute.com/wp-content/uploads/2015/02/Summary-First-Peoples-Second-Class-Treatment-Final.pdf (accessed December 17, 2019).

34.

Nursing Council of New Zealand. Guidelines for Cultural Safety, the Treaty of Waitangi and Maori Health in Nursing Education and Practice. Wellington, New Zealand; 2005. https://www.nursingcouncil.org.nz/Public/Nursing/Standards_and_guidelines/NCNZ/nursing-section/Standards_and_guidelines_for_nurses.aspx?hkey=9fc06ae7-a853-4d10-b5fe-992cd44ba3de (accessed December 17, 2019).

35.

North American Primary Research Group (NAPCRG). 46th North American Primary Research Group Annual Meeting. Chicago, IL; 2018. https://www.napcrg.org/conferences/annual/about-the-conference (accessed December 17, 2019).

36.

Giustini

, Kamel Boulos

. Google Scholar is not enough to be used alone for systematic reviews. Online J Public Health Inform, 2013; 5:1–10.

37.

Haddaway

, Collins

, Coughlin

, Kirk

. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS One, 2015; 10:e0138237.