Appraisal of Face and Content Validity of a Serious Game Improving Situational Awareness in Surgical Training

Abstract

Background:

Equipment-related malfunctions during minimally invasive surgery (MIS) are common and threaten patient safety. As they occur in the periphery of the surgeon's vision, the surgical team requires a high level of situational awareness in order to intercept these errors timely. A serious game has been developed to train surgical residents to deal with equipment-related errors. This study investigates to what extent surgical educators and trainees would accept a serious game as a training method.

Materials and Methods:

A cross-sectional survey was conducted among 45 surgeons, surgical residents, and medical students who played the serious game at a scientific convention. The questionnaire contained statements on perceived realism, usefulness, teaching capability, user experience and application toward surgical training. Results were analyzed according to participants' MIS experience (“expert,” “intermediate,” and “novice”).

Results:

The majority found that important medical constructs are represented realistically (64.4%–88.9%) and indicated the game to be particularly useful for training operating room nurses and surgical residents (75%–86%). Both educators and trainees found the game to be useful for surgical training (53%). Serious gaming was viewed as positive (78%) and challenging (60%), and 66% would play the game in their leisure time. Licensed surgeons perceived the game more frequently as boring than the intermediate-level and trainee groups (23.5% versus 6.7% and 8.3%; P=.045).

Conclusions:

This is the first study to show acceptance of a serious game as a training format in surgical training by educators and trainees. Future research should investigate whether the serious game indeed improves problem-solving and situational awareness in the operating room.

Introduction

The use of complex technological and electronic equipment has allowed for great progression in minimally invasive surgery (MIS), but has also resulted in a heightened mental workload for the surgeon.¹ High mental workload may lead to errors as the untrained human brain has limited capability of recognizing alterations or unexpected events under stressful circumstances, even if they occur in plain sight. Perception is a selective process that focuses mainly on potentially interesting objects in the visual field while largely ignoring the uninteresting or unexpected, rightfully or not. These processes are referred to as “inattention”² and “change blindness.”³ This accounts for a delayed or inaccurate recognition of potentially harmful events in the operating room (OR). As a result, equipment failure may either be ignored or misinterpreted, as well as changes in patients' physiological parameters. Studies show that equipment failure occurs frequently in MIS, leading to delays in operating time and complications for patients with clinical consequences.^4,5

The adaptive coupling between humans and their environment during the performance of a complex task is referred to as “situational awareness.” An observational study showed that surgeons with high levels of situational awareness are less likely to make technical errors during laparoscopic cholecystectomies.⁶ Training surgical residents in dealing with equipment-related problems and other nonroutine events is thought to reduce their mental workload during their first procedures,^7,8 when they need to use most of their mental capacity to focus on the procedure itself. This will most likely result in better recognition and identification of relevant changes in the periphery of their focus, reducing inattention and change blindness.

Serious or applied games are computer applications that offer a challenging and fun experience to the player, while simultaneously providing educational content in a subtle “stealthy” fashion.^9,10 A serious game mimics a simulation in that it provides a simulated experience of reality. An important surplus of serious gaming is the abstracted “game layer.” The game layer aims at keeping players engaged and immersed in the serious game in order to increase their voluntary adherence to training.¹¹ Because educational content such as situational awareness is out of direct surgical focus, it could be perceived as uninteresting or irrelevant by trainees and even by surgical educators. A serious game was designed to train surgical trainees in recognizing and dealing with equipment-related problems in MIS.

To date, no evidence has been reported on the acceptance of serious gaming by surgical trainees and surgical educators with regard to surgical training.⁹ This study assessed to what extent educators and trainees would accept serious gaming to improve situational awareness as useful and relevant to MIS training.

Materials and Methods

Participants

A consecutive cohort of 45 surgeons, surgical residents, and medical students with an interest in surgery with no previous exposure to the serious game were recruited on a voluntary base during the annual convention of the Dutch Surgical Society (Nederlandse Vereniging voor Heelkunde), May 30–31, 2013. In total, 50 persons played the serious game, and 5 participants were excluded because they did not have a medical background.

The participants were introduced to the serious game on a laptop computer and received a hands-on instruction by trained instructors to familiarize them with the gameplay, after which they played four 3-minute sessions. Then, the participants completed a questionnaire. The participants' opinions were compared among the expert group (defined as having performed >100 MIS procedures as primary surgeon), intermediate-level group (defined as having performed 1–99 MIS procedures as primary surgeon), and novice group (defined as having no experience with MIS). A sample size calculation was not performed because of the nature of the study.

Serious game

The serious game was developed specifically for surgical residents,¹² aiming to educate them (1) to identify important elements from the surroundings in the OR unrelated to the procedure itself and (2) to solve the problems correctly and efficiently. The serious game (Weirdbeard Inc., Amsterdam, The Netherlands) is designed for use on smartphones and tablets. This game itself appears to have little to do with surgery at first sight and presents itself to the player as a popular, easy-to-play entertainment game. This is thought to trigger the learner's intrinsic motivation to play and support the user to keep playing. The educational content includes the laparoscopic tower's screen and lighting problems, gas transport problems, electrosurgical problems, and specific complications related to MIS (Table 1). This content is virtually embedded in the entertainment game (Fig. 1, left). While the player plays the game, signals occur signifying specific equipment failure scenarios (Fig. 1, middle). The game's screen reacts to monitor and lighting, the visibility reacts to the insufflation, moving the blocks reacts to the electrosurgical unit, and realistic auditory alarms could signify pathophysiological complications or malfunctions of equipment.

FIG. 1.

The serious game (screenshots). (Left) Main screen, with mini-game (below), the patient's vital signs, and a supervising surgeon (above). (Middle) During the mini-game, the player deals with problem scenarios that resemble real-life problems in minimally invasive surgery, such as a darkened screen. (Right) After the player recognizes the problem scenario, he or she can solve it by selecting the correct action on a simulation of the minimally invasive surgery equipment.

Table 1.

List of Problem Scenarios Included in the Serious Game Content, Grouped by Equipment

Screen/lighting	Gas transport/pneumoperitoneum	Electrosurgery	Pathophysiology
Blurred screen	Intraabdominal pressure too high	Electrosurgery alarm	Desaturation
Condensation on screen	Insufflation insufficient	Electrosurgery does not function	Hypotension
Flashing screen	Obstructed gas chain	Electrosurgery insufficient
Moving image	Empty gas supply
Yellow coloring
Green coloring
Red coloring
Darkened screen
Light screen
Black screen
“No signal” sign
Smoke on screen

Problems can have multiple causes.

As soon as the player suspects a malfunction or complication, he or she stops the game by pressing “stop,” after which he or she enters the trouble-shooting mode (Fig. 1, right). This depicts a simulated laparoscopic tower. The player should diagnose the problem and correct the issue at hand. The players' performance in problem recognition and problem solving is assessed (proportions of problems recognized and solved, time required, and amount of correct and incorrect diagnostic steps). The player receives feedback on his or her actions.

By embedding equipment failure scenarios in an alternate activity, the “real” OR situation is simulated, in which trouble occurs outside the visual field itself from the surgeon. Educational content was derived from the Fundamentals of Laparoscopic Surgery course.¹³ The scenarios were checked and corrected by two independent laparoscopic surgeons and five MIS equipment specialists. They were given in written form with correct solutions, leaving content experts to choose between “valid” and “invalid.”

Questionnaire

The questionnaire was accessed through Google Docs (Google Inc., Mountain View, CA) and contained 12 items on demographic characteristics and 7 statements on realism of important medical aspects (MIS equipment, problem scenarios). Six statements questioned the serious game's educational value, and six statements concerned its usefulness for different user groups. Seven statements questioned user experience, and seven statements appeared on use in surgical curricula.

The statements were evaluated on a 5-point Likert scale, in which 1 equaled “fully disagree,” 3 equaled “neutral,” and 5 equaled “fully agree.” A median value of >3.0 was viewed as a positive response to the statement, versus <3.0 as a negative response. Additionally, participants could clarify their opinion through an open textbox per topic.

Statistical analysis

Measurements were recorded and analyzed using the IBM Statistical Package for Social Sciences (SPSS version 19; IBM Corp., Armonk, NY). Nonparametric tests were used to calculate differences among user groups; statistical significance was considered at P<.05.

Results

Demographic characteristics

Fourteen licensed surgeons, 25 surgical residents, and 6 medical students with an interest in surgery participated in the study (Table 2). Participants were based at different hospitals in The Netherlands. Of the surgeons, 13 specialized in general surgery, and 1 specialized in vascular surgery. The mean number of years recorded was 12.5 years (standard deviation=8.4). Of the residents, 88% specialized in general surgery, 8% in urology, and 4% in plastic surgery. The participants were grouped according to their experience with MIS using the above-mentioned criteria. All were included in the analysis, although one participant from the expert group was lost to follow-up because of technical failure of the questionnaire.

Table 2.

Demographic Characteristics per Study Group

Characteristic	Novice (no experience in MIS)	Intermediate (1–99 MIS procedures)	Expert (>100 MIS procedures)
Group size	12	15	18
Sex
Male (%)	50	66	89
Female (%)	50	33	11
Age (years)	25.8±0.7	31.9±0.8	44.6±2.3
Function
Student	6	0	0
Resident	6	14	5
Specialist	0	1	13
Videogame experience (%)
Current	66	53	22
Past (%)	17	7	17
Laparoscopic equipment training (%)
Basic laparoscopic course	8	87	61
Advanced laparoscopic course	0	0	6

Data are number of individuals, mean±standard error, or percentage as indicated.

MIS, minimally invasive surgery.

Representation of medical constructs

Table 3 refers to the participants' opinions on realism of medical constructs that were incorporated in the serious game. In total, 88.9% found the MIS equipment representation to be realistic, 84.4% the displays and parameters on the equipment, 75.9% the auditory signals, and 66.7% the visual signals. The problem scenarios were viewed to be realistic by 64.4%, and feedback was appreciated by 64.4%. Of the participants, 48.9% found solving problems to be realistic, versus 24.4% who did not. There were no significant differences among the groups. The open comments indicated that two participants viewed lack of realism as a problem, and two participants indicated that the displays were not adjusted correctly during a problem scenario.

Table 3.

Opinions on Representation of Important Medical Constructs

	Novice (n=12)			Intermediate (n=15)			Expert (n=18)
Realistic representation	Median	P25	P75	Median	P25	P75	Median	P25	P75	P^a
Laparoscopic tower	4.00	4.00	4.00	4.00	4.00	4.00	4.00	4.00	4.00	NS
Display parameters	4.00	3.25	4.00	4.00	4.00	4.00	4.00	4.00	4.00	NS
Auditory cues	4.00	4.00	4.00	4.00	3.00	4.00	4.00	3.00	4.00	NS
Visual cues	4.00	4.00	4.00	4.00	3.00	4.00	3.50	2.00	4.00	NS
Problem scenarios	4.00	3.00	4.00	4.00	4.00	4.00	3.50	2.00	4.00	NS
Solving problems	4.00	3.00	4.00	3.00	2.00	4.00	3.00	2.00	4.00	NS
Feedback	4.00	3.00	4.00	4.00	3.00	4.00	4.00	2.75	4.00	NS

By Kruskall–Wallis test.

NS, not significant; P, percentile.

Teaching capability

Table 4 refers to the perceived teaching capability of the serious game. Of the participants, 48.9% found the serious game to be useful for functioning in the laparoscopic environment, versus 28.9% did not. Furthermore, 93.4% found that the serious game enhanced the players' awareness of equipment malfunctions, 86.6% their problem recognition capabilities, and 71.1% their problem-solving capabilities. In total, 48.9% agreed that the game enhanced the players' environment perception, versus 22.2% who disagreed. The game was thought to enhance overall situational awareness by 62.2%. No significant differences between the groups existed.

Table 4.

Opinions on the Serious Games Usefulness for Teaching Different Skills and Different User Groups

	Novice (n=12)			Intermediate (n=15)			Expert (n=18)^a
	Median	P25	P75	Median	P25	P75	Median	P25	P75	P ^b
Usefulness for learning
To function in laparoscopic environment	3.00	2.00	4.00	4.00	3.00	4.00	3.50	2.00	4.00	NS
Problem awareness	4.00	4.00	5.00	4.00	4.00	4.00	4.00	4.00	4.25	NS
Problem recognition	4.00	4.00	4.75	4.00	4.00	4.00	4.00	4.00	4.00	NS
Problem solving	4.00	4.00	4.00	4.00	3.00	4.00	4.00	3.75	4.00	NS
Perception of environment	3.50	2.00	4.00	3.00	3.00	4.00	3.50	2.75	4.00	NS
Situation awareness	4.00	3.00	4.00	4.00	3.00	4.00	4.00	4.00	4.00	NS
Usefulness for teaching
Students	4.00	4.00	4.00	4.00	4.00	4.00	4.00	3.50	4.00	NS
OR nurses	4.00	4.00	4.00	4.00	4.00	4.00	4.00	3.50	4.00	NS
Residents (not-in-training)	4.00	2.25	4.00	4.00	4.00	4.00	4.00	3.50	4.00	NS
Residents (in-training)	4.00	3.25	4.00	4.00	4.00	4.00	4.00	3.00	4.50	NS
Fellows (MIS)	3.00	2.25	4.00	3.00	2.00	4.00	3.00	2.00	4.00	NS
Surgeons	3.00	2.25	4.00	3.00	2.00	4.00	3.00	2.00	4.00	NS

One participant was lost to follow-up because of technical problems.

By Kruskal–Wallis test.

MIS, minimally invasive surgery; NS, not significant; OR, operating room; P, percentile.

Of the participants, 80% viewed the serious game as useful for teaching medical students, 84.4% for OR nurses, 75.6% for interns, and 75.6% for surgical residents. In total, 35.6% viewed it to be useful for fellows in MIS, and 33.3% did so for licensed surgeons. There were no significant differences among the groups. Three participants indicated in the open comments box that lack in realism is problematic to the game's teaching capability. One participant indicated the game to be too specific for students.

User experience

Table 5 refers to the user experience of the participants when playing the serious game. In total, 82.2% found the game to be pleasant, 77.8% funny, 60.0% challenging, and 17.8% addictive. Of the participants, 33.3% found it frustrating, versus 51.1% who did not. None found the experience to be repulsive, and 13.3% found the serious game boring. Experts found it significantly more boring than the intermediate-level and trainee groups (23.5% versus 6.7% and 8.3%; P=.045). Open comments revealed two participants who explicitly indicated it was an attractive way to learn about malfunctioning equipment.

Table 5.

User Game Play Experience

	Novice (n=12)			Intermediate (n=15)			Expert (n=17)^a
User experience	Median	P25	P75	Median	P25	P75	Median	P25	P75	P ^b
Pleasant	4.00	3.25	4.00	4.00	4.00	4.00	4.00	4.00	4.00	NS
Funny	4.00	4.00	4.00	4.00	4.00	4.00	4.00	3.00	4.00	NS
Frustrating	2.50	2.00	4.00	2.00	2.00	4.00	3.00	2.00	4.00	NS
Challenging	3.50	3.00	4.00	4.00	3.00	4.00	4.00	2.50	4.00	NS
Repulsive	2.00	1.00	3.00	2.00	2.00	3.00	2.00	1.00	2.50	NS
Addictive	3.00	2.00	4.00	3.00	2.00	3.00	2.00	1.50	3.00	NS
Boring	2.00	1.25	2.00	2.00	2.00	2.00	2.00	2.00	3.50	.045^c

One participant was lost to follow-up because of technical problems.

By Kruskal–Wallis test.

Statistically significant difference between novice and expert groups.

NS, not significant; P, percentile.

Surgical training

Table 6 refers to the opinions on the applicability of the serious game in regular surgical teaching. Of the participants, 13.3% considered the difficulty level too low, versus 6.7% who thought it too high. Furthermore, 53.3% of the participants found it to fit into the regular surgical curriculum, versus 77.8% who thought that it could be played outside the official curriculum. Only 6.6% thought that it should be an obligatory part of the curriculum. In total, 66.6% would download the game. There were no significant differences among the groups.

Table 6.

Opinions on the Applicability in Surgical Teaching

	Novice (n=12)			Intermediate (n=15)			Expert (n=16)^a
Applicability	Median	P25	P75	Median	P25	P75	Median	P25	P75	P ^b
Level too low	2.00	2.00	3.00	2.00	2.00	3.00	2.00	2.00	3.00	NS
Level too high	2.00	2.00	3.00	2.00	2.00	2.00	2.50	2.00	3.00	NS
Improves functioning in laparoscopic surgery environment	4.00	2.00	4.00	4.00	3.00	4.00	3.00	2.25	4.00	NS
Fits into regular surgical curriculum	3.50	2.00	4.00	4.00	3.00	4.00	4.00	3.00	4.00	NS
Fits outside surgical curriculum	4.00	4.00	4.00	4.00	4.00	4.00	4.00	3.25	4.00	NS
No interest in playing the SG	2.00	2.00	3.00	2.00	2.00	3.00	3.50	2.00	4.00	NS
Should be part of surgical curriculum	2.00	2.00	3.00	2.00	2.00	2.00	2.00	1.00	3.00	NS
Would download SG for
Free	4.00	3.25	4.00	4.00	4.00	5.00	4.00	2.00	4.00	NS
1 Euros	2.50	2.00	4.00	4,00	2.00	4.00	2.00	1.00	3.75	NS
5 Euros	2.00	2.00	2.00	2.00	2.00	3.00	2.00	1.00	3.00	NS

One participant was lost to follow-up because of technical problems during assessment.

By Kruskal–Wallis test.

NS, not significant; P, percentile; SG, serious game.

Discussion

This cross-sectional study shows that surgeons and surgical residents from multiple institutions in The Netherlands have a clear positive opinion towards applying serious games in surgical residency training. Serious gaming is believed to be “the next big thing” in medical education. Residents frequently access applications and games on mobile devices in order to gain knowledge or merely to spend their leisure time. Over half the physicians and almost 70% of the residents use smartphone applications and mobile technology in clinical decision-making, a number that is likely to increase over the next few years.¹⁴ Results are in conformity with other studies that generally show postgraduate medical trainees to have a positive opinion regarding virtual reality applications in postgraduate education.^15,16 This study is the first to report on the acceptance of serious games in surgical training,⁹ revealing positive opinions. Although it represents a novel teaching modality in surgery, the system's face validity is an important step.¹⁷

The usefulness of videogames to education has gained acknowledgement in recent decades¹¹ and has found its way into surgical training.⁹ Rosser et al.¹⁸ showed that visuomotor skills in laparoscopy are correlated to playing commercially available videogames on well-known game consoles. New generations are used to the advantages of gaining knowledge through digital portals. Residents currently spend more time playing videogames than reading books.¹⁹ This correlates with the high percentage of videogame experience in the novice and intermediate groups in this study (83% and 60%, respectively, versus 39% in the expert group). These habits therefore call for a different instructional approach.¹⁹ To date, no scientific results are available on the issue of embedded use of smart strategies such as serious gaming in surgical education.^9,20

Our study describes a novel stand-alone offsite instructional tool to improve the situational awareness of inexperienced surgical trainees. Problems related to equipment occur often during MIS,⁴ mostly outside the surgeon's direct area of focus, which is the surgical field. To improve situational awareness in the OR, the operating team should train to recognize and deal with “nonroutine events” during routine procedures.^7,8 To date, no official training programs have focused on situational awareness in surgery. This situational awareness is currently gained “on the job,” during which patients are at unnecessary risk. Moreover, significant knowledge deficiencies on resolving issues with MIS equipment and surroundings frequently exist after regular skills training, even in experienced laparoscopic surgeons.²¹ Whereas situation awareness is part of a team's performance in the OR,⁶ curriculum development on nontechnical skills in the OR should include the complete OR team.

These results show that both residents and educators are likely to accept serious gaming as a format for training skills in surgery. The traditional view in surgical training is that well-designed training environments require full representations of the real OR. Several expert participants in our study, in fact, shared this view. Evidence, however, indicates that the level of realism of the construct in the simulation (referred to as “physical fidelity”) is not a necessary precondition to allow skills transfer. As long as the relevant elements in the decision-making process of the educational construct are represented realistically (referred to as “functional fidelity”), the simulation can still induce learning in trainees.^22,23 The level of immersion (players' feeling of presence and control in the simulation), challenge, and acceptance of the instrument are more important for skills transfer than physical representation.²³

Limitations

Limitations and potential sources of bias to the study include the following. First, the participants' voluntary participation could have introduced a selection bias. However, the sample's demographic characteristics are in conformity with the target population. Gender differences corresponded to gender differences between specialists and medical students, whereas in The Netherlands over two-thirds are female.²⁴ Age differences are as may be expected because of stratification on experience. Selection bias does not appear to have influenced intergroup opinion differences, but may have positively influenced the cohort as a whole. However, results correspond to the positive opinions of surgical postgraduate trainees on other types of virtual reality–based training methods.^15,16

Next, mere attention toward the subject could have positively influenced participants' opinions (the Hawthorne effect). In particular, lesser-experienced participants are susceptible to this phenomenon. It does not appear to be of great influence to intergroup differences, as no significant differences are seen between groups for all items but one.

Third, participants could be influenced by survey questioners' enthusiasm (the Pygmalion effect). To reduce this, questioners were not affiliated with the game developer, and the survey was completed online anonymously.

Finally, the application of Likert scales could introduce lack of clarity and ambiguity concerning individual questionnaire items.²⁵ In the survey's design, many different definitions were therefore given per construct (e.g., “situational awareness,” “problem awareness,” and “perception of environment”), as well as conflicting items (“challenging” versus “boring”). However, this source of bias is hard to exclude.

Conclusions

Results of our study show positive attitudes from educators and trainees from different surgical centers regarding acceptability and use of serious gaming in surgical curricula. These results strengthen legitimacy of serious games in surgical educational curricula. Research is required on novices' learning curves on the serious game and transfer of problem-solving abilities of game-trained residents in the reality before its use is justified, which is customary for validation standards for instructional tools.

Footnotes

Acknowledgments

The authors wish to thank the organizing committee of the 2013 Annual Convention of the Dutch Surgical Society for facilitating this study. This study received funding (grant reference number PID 101060) from the “Pieken in de Delta” program of the Ministry of Economic Affairs, Agriculture and Innovation, the City and Province of Utrecht (The Netherlands). The funding agency had no role in design and conduct of the study, data collection, management, analysis, and interpretation, or preparation, review, or approval of the manuscript.

Disclosure Statement

No competing financial interests exist.

References

Zheng

, Cassera

, Martinec

D V

, Spaun

, Swanstrom

. Measuring mental workload during the performance of advanced laparoscopic tasks. Surg Endosc, 2010; 24:45–50.

Drew

, Võ

ML-H

, Wolfe

. The invisible gorilla strikes again: Sustained inattentional blindness in expert observers. Psychol Sci, 2013; 24:1848–1853.

Simons

, Rensink

. Change blindness: Past, present, and future. Trends Cogn Sci, 2005; 9:16–20.

Verdaasdonk

EGG

, Stassen

LPS

, van der Elst

, Karsten

, Dankelman

. Problems with technical equipment during laparoscopic surgery. An observational study. Surg Endosc, 2007; 21:275–279.

Courdier

, Garbin

, Hummel

, Thoma

, Ball

, Favre

, et al. Equipment failure: Causes and consequences in endoscopic gynecologic surgery. J Minim Invasive Gynecol, 2009; 16:28–33.

Mishra

, Catchpole

, Dale

, McCulloch

. The influence of non-technical performance on technical outcome in laparoscopic cholecystectomy. Surg Endosc, 2008; 22:68–73.

Endsley

, Robertson

. Training for situation awareness in individuals and teams. In: Endsley

, Garland

(eds). Situation awareness analysis and measurement. Mahwah, NJ: Lawrence Erlbaum Associates; 2000:1–13.

Moorthy

, Munz

, Adams

, Pandey

, Darzi

. A human factors analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Ann Surg, 2005; 242:631–639.

Graafland

, Schraagen

, Schijven

. Systematic review of serious games for medical education and surgical skills training. Br J Surg, 2012; 99:1322–1330.

10.

Sharp

. Stealth learning: Unexpected learning opportunities through games. J Instr Res., 2012; 1:42–48.

11.

Michael

, Chen

. Serious Games: Games That Educate, Train, and Inform. Boston: Thomson Course Technology, 2006.

12.

Graafland

, Schijven

. A serious game to improve situation awareness in laparoscopic surgery. In: Schouten

, Fedtke

, Bekker

, Schijven

, Gekker

(eds). Games for Health. Wiesbaden: Springer Fachmedien Wiesbaden, 2013:173–182.

13.

Peters

, Fried

, Swanstrom

, Soper

, Sillin

, Schirmer

, et al. Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery, 2004; 135:21–27.

14.

Franko

, Tirrell

. Smartphone app use among medical providers in ACGME training programs. J Med Syst, 2012; 36:3135–319.

15.

Dankbaar

MEW

, Storm

, Teeuwen

, Schuit

SCE

. A blended design in acute care training: Similar learning results, less training costs compared with a traditional format. Perspect Med Educ, 2014; 3:289–299.

16.

Cowan

, Sabri

, Kapralos

, Porte

, Backstein

, Cristancho

, et al. A serious game for total knee arthroplasty procedure, education and training. J Cyberther Rehabil Cyberther, 2010; 3:285–298.

17.

Schijven

, Jakimowicz

. Validation of virtual reality simulators: Key to the successful integration of a novel teaching technology into minimal access surgery. Minim Invasive Ther Allied Technol, 2005; 14:244–246.

18.

Rosser

Jr , Lynch

, Cuddihy

, Gentile

, Klonsky

, Merrell

, et al. The impact of video games on training surgeons in the 21st century. Arch Surg, 2007; 142:181–186.

19.

El-Shamy

. Training for the new and emerging generations. San Francisco: John Wiley & Sons, 2004.

20.

Akl

, Kairouz

, Sackett

, Erdley

, Mustafa

, Fiander

, et al. Educational games for health professionals. Cochrane Database Syst Rev, 2013; 3(3):CD006411.

21.

Menezes

, Birch

, Vizhul

, Shi

, Sherman

, Karmali

. A deficiency in knowledge of basic principles of laparoscopy among attendees of an advanced laparoscopic surgery course. J Surg Educ, 2011; 68:3–5.

22.

Mania

, Wooldridge

, Coxon

, Robinson

. The effect of visual and interaction fidelity on spatial cognition in immersive virtual environments. IEEE Trans Vis Comput Graph, 2006; 12:396–404.

23.

Alexander

, Brunyé

, Sidman

, Weil

. From gaming to training: A review of studies on fidelity, immersion, presence, and buy-in and their effects on transfer in PC-based simulations and games. In: Proceedings from the 2005 Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC). Arlington, VA: 2005:1–14.

24.

Arts en Auto. Feminisation in the workplace [in Dutch]. 2012:16–9. Available at www.artsenauto.nl/wp-content/uploads/2012/07/AA08-p016-019.pdf (accessed January 8, 2015).

25.

Johns

. Likert Items and Scales. University of Strathclyde. 2010:1–11. Available at surveynet.ac.uk/sqb/datacollection/likertfactsheet.pdf (accessed January 8, 2015).