Mapping the Maze of Minimally Invasive Surgery Simulators

Abstract

Background:

Conforming to, among other considerations, legal and ethical concerns for patient safety, there is an increasing demand to assess a surgeon's skills prior to performance in the operating room in pursuit of higher-quality treatment. Training in minimally invasive surgery (MIS) must therefore be intensified, including team training. New methods to train and assess minimally invasive surgical skills are gaining interest. The goal of this review is to provide instructors with an overview of available MIS training tools. In this review, we discuss currently available simulators for MIS training. Applicability, validity, and construction of simulators are reviewed. Also, some of the leading training programs and assessment methods in MIS are reviewed.

Methods:

A literature search was performed on studies evaluating surgical task performance on a simulator, reviewing satisfaction with laparoscopic training programs, or validating simulators or assessment methods.

Results:

Simulators may be divided into simple box trainers and computer-based systems, such as virtual and augmented simulators. All have advantages and disadvantages. An overview is provided of currently available training systems, validity, trainee assessment, and the importance of training programs in MIS.

Conclusions:

No simulator yet provides the ability to train the entire set of required psychomotor skills or procedures for MIS. A multiyear training program combining various simulators for multiple-level training, including team training, should be constructed.

Introduction

The introduction of minimally invasive surgery (MIS) has reduced the surgical burden for many patients because of the use of minimal incisions and videoscopic technology. MIS has changed the landscape of surgery during the past two decades; postoperative pain has been diminished, patients mobilize earlier after MIS, and hospital stays are shorter, beside obvious cosmetic advantages. Because of these advantages, many open procedures are being replaced by MIS.

At the onset of MIS in the early 1990s, comprehensive training in MIS was not available. Many of the early adapters of MIS were their own teachers, associated with long operating hours. Complications such as bile duct injury during laparoscopic cholecystectomy were more common in the early phase of MIS compared to open surgery because optimal MIS operative approaches and technologies had not yet been fully established.¹ The transition from a three-dimensional working field to a two-dimensional image projected on a TV monitor with consequent loss of accurate palpation of vital structures and pathology was one of the largest transformations in surgical techniques this past century. Additionally, impaired tactile feedback, loss of joint dexterity, and counterintuitive instrument movement render MIS a different technique compared to open surgery.²

To ensure patient safety, adequate training of MIS is mandatory.³

A spectrum of training scenarios—inanimate training, training on live or cadaveric animal tissues, box trainers, virtual reality (VR) and augmented reality (AR) simulators, and training on cadaveric human tissues—is available.^4–7

The aim of this study is to provide an overview of available simulators in MIS. We offer an aid to surgical educators in choosing a simulator for MIS training. We present recommendations for optimal use of specific MIS simulators within different stages of a surgical curriculum.

Methods

Eligible studies were identified using an extensive and systematic search conducted in February 2011 using MEDLINE/PubMed (Table 1). We included all articles focused on surgical task performance on a simulator, reviewing satisfaction with laparoscopic training programs, or validating simulators or assessment methods.

Table 1.

Ovid MEDLINE Search Strategy (Search Date, February 28, 2011)

Number	Search history	Results
1	(“Education”[Mesh] OR “education” [Subheading])	596,824
2	(“Laparoscopy”[Mesh] OR “Minimal Invasive Surgery, Video-Assisted”[Mesh]) OR “Video-Assisted Surgery”[Mesh]	59,899
3	Numbers 1 and 2	2500
4	Videotrain^* OR box OR boxes OR boxtrain^* OR simulat^* OR virtual OR augmented	557,333
5	Numbers 2 and 4	1530
6	Numbers 3 and 4, limits: English	615
7	Numbers 5 not 6, limits: English	838
8	“Laparoscopy”[Mesh]	44,883
9	Numbers 8 and 6	601
10	Numbers 8 and 7	767

Title and abstract of English articles identified by the database were scanned to assess inclusion eligibility. When information necessary for the assessment of eligibility was lacking, the full text article was retrieved for review. Additionally, related articles and reference lists of selected articles were scanned.

Knowledge of key features is essential in choosing a training or assessment device. Evidence suggests that simulators are valid instruments in the acquisition of MIS skills.^6,8–10 Hamilton et al.¹¹ established that laparoscopic skills developed outside the operating room are transferrable to actual performance of laparoscopic surgery.

Teaching, rehearsal, and assessment in simulator-based training always occur simultaneously. Feedback should provide residents with an indication of present performance. Furthermore, the collected data allow evaluation of resident progression.¹²

Simulators may provide objective data that allows assessment of technical skills. Performance of surgical tasks improves by standardized repetition.¹³ Improvement is exponential at the start. Subsequently, it levels off over time until a steady state of performance is established.¹⁴ A learning curve is the graphic representation of the relationship between experience and outcome. Outcome may be described as mortality or morbidity or towards scientific criteria such as procedure duration or instrument path length.¹⁵

Reliability

Reliability is the reproducibility and consistency of results or examination in a simulator. Test–retest reliability is the commonly used indicator for instrumental reliability; it is estimated by performing the same survey with the same respondents on a comparable level at different moments in time. The greater the agreement of results, the greater the test–retest reliability of the survey instrument.¹⁶

Validity

Validity is “the property of being true, correct, and in conformity with reality.” Validity indicates whether a simulator measures the skill it is intended to measure. According to the European Association of Endoscopic Surgeons consensus guidelines drawn from Carter et al.,¹⁷ there is a need for a validation of the MIS simulators and their assessment methods, before their inclusion into training curricula.

A simulator must be evaluated vigorously and objectively regarding its reliability and validity.^16,18

Validation contains several principles, and to assess the validity of a training device or assessment method several benchmarks have been developed. These include construct, face, predictive, content, concurrent, and discriminate validity.^17,19

Construct validity is defined as correctly simulating or measuring the skills the device intends to train or assess.²⁰

Face validity illustrates adequate simulation and adequate resemblance to a task and whether a simulator is considered useful for training.^21–23 Predictive validity is the extent to which a score predicts scores on defined criterion measures in the future. This is the extent to which scores on a test are predictive of actual performance in the operating room.

Content validity is an indication of the appropriateness of a simulator as a teaching tool.

Correlation of test scores between two devices or a device and the gold standard—which are assumed to measure the same variable—is defined as concurrent validity.

Discriminate validity describes the translation of simulator skill to skill in the operating room.¹⁹

Validation may take a subjective and objective approach.^16,17,19 Face and content validity both are subjective approaches to validity. Content validity requires two groups of different skill levels to perform a procedure or assessment, after which both groups are asked their opinion of whether the tested device and procedures resembles the real-life experience. Face validity involves a questionnaire exploring two groups of different skill level's opinion on the training or assessment method. Objective approaches to validity include construct, discriminative, concurrent, and predictive validity. These validities generally involve studies investigating whether a simulator provides metrics for performance or discriminates among different levels of expertise. When investigating training programs a key question to objective approaches of validity is whether training on inanimate systems is transferable to operating room conduct. Simulator metrics display predictive validity when they show correlation with objective assessment of in vivo skill. Predictive validity actually provides here the only meaningful assessment of clinical skills, whereas the other validity measures focus on the training methods and outcomes.¹⁹

Training Systems

Several studies have examined the training capacities of various surgical simulators.^23,24

Training on simulators increases psychomotor skills, which translates into improved performance in MIS before performing on animate models and patients.²⁵ MIS simulators are computerized VR simulators, traditional video box trainers, or a combination of these.²⁶ The use of MIS simulators results in faster acquisition of endoscopic technical skills compared with ample observation of an expert or practicing parts of procedures under supervision in the operating room.²⁷ However, each simulator has its own advantages and disadvantages (Table 2). Several simulators have been developed and examined. Validated simulators available up to February 2011 are listed in Table 3. We describe published data on efficacy, reliability, and validity of available MIS training devices.

Table 2.

Basic Advantages and Disadvantages of Various Minimally Invasive Surgery Simulators

Simulator	Advantage	Disadvantage
Box trainer	Inexpensive	Lack of (data on) automatic and objective skill assessment tools Faculty required for training and exams
	Genuine haptic feedback
	Actual laparoscopic instruments
	Practice on genuine cadaveric tissue
Virtual reality simulator	Simulation of specific surgical procedures	Relatively expensive
	Automatic and objective skill assessment	No realistic haptic feedback
	No faculty required	No realistic anatomy
		Frequent maintenance/updates
		Possibility of routine practice
Augmented reality simulator	Genuine haptic feedback	Relatively expensive
	Actual laparoscopic instruments	Vision-tracking systems sensitive to malfunction
	Practice on genuine cadaveric tissue	Possibility of routine
	Simulation of specific surgical procedures	Lack of assessment protocols
	Automatic and objective skill assessment
	No faculty required

Table 3.

Minimally Invasive Surgery Simulators and Main Features

		Virtual reality						Augmented reality
	Box trainer ^a	MIST-VR	LapSim	SIMENDO	CEA (Endo Tower and LapVR ^b )	LapMentor	SimSurgery SEP²⁸	ProMis	SurgicalSIM	TrEndo
Validity^c	FLS^d	1, 3, 5	1, 2, 3, 6	1, 2, 3, 5	1, 3	1, 3	3	1, 3	3, 5	–
Reliability	+	+	+	–	–	+	–	+	+	+
Assessment method	OSATS	+	+	–	–	+	+	+	+	+
P	–	+	+	–	+	+	+	+	–	–
Haptic feedback	+	Optional (Procedicus)	Optional	+	+	+	–	+	+	+
CRM	+	–	+	+	–	–	–	+	–	–
Decision-making	+	–	+	-	+	+	–	+	–	+
Instructions	DVD	+	+	+	+	+	+	+	+	–
T/P	T/P	T	T/P	T	T/P	T/P	T/P	T/P	T	T/P
Knowledge	–	–	–	–	+	+	–	–	–	–
Cost indication	€2500	€25,500	€45,000	€2500	€66,000	€21,650	€33,000	€45,000	€53,000	€2500

Box trainers

Real-life surgical conditions are probably best obtained by box trainers (video trainers) as box trainers use conventional laparoscopic equipment inserted through trocars, include a video monitor and camera, and provide the possibility of the introduction of a range of targets for practice such as synthetic inanimate models or real cadaver tissue.³¹ It thereby provides an ideal environment for training different surgical scenarios. The box typically measures the same size as an adult human abdominal cavity. Main exercises in a box trainer include movement, coordination, and procedural training. The use of real tissue has many advantages, such as haptic feedback and an acquaintance with different types of tissue. Haptic feedback is described as the experience of force feedback when manipulating tissue.^32,33 Training in the absence of haptic feedback significantly decreases the amount of skill transferred to the operating room when compared with training in the presence of haptic feedback.³² As such, we feel it is essential to provide haptic feedback when training MIS skills. Furthermore, several key aspects of learning skills involve a trainee's subtle interaction with tissue and suturing materials (haptic feedback). Visual feedback alone does not suffice.^4,34,35

Several box and video trainers including various validated exercises are available. An overview is given in Table 4. The use of a laparoscopic box trainer is particularly well documented in the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS).⁴⁸ The MISTELS system was designed to objectively assess basic laparoscopic skills through a series of structured tasks performed under video guidance in a box trainer.^49,50 MISTELS has been shown to discriminate between competent and non-competent laparoscopic surgeons and may be used to evaluate individual skill levels.⁵¹ These box trainers have been studied and validated extensively and have progressed into the Fundamentals of Laparoscopic Surgery training course and evaluation system (FLS).⁵²

Table 4.

Box and Video Trainers

Reference	Type	Validity ^a
Yale laparoscopic skills and suturing program³⁶	Laparoscopic surgical trainer	3
Lentz et al.³⁷ (6 tasks)	Mirrored box trainer	3
Risucci et al.³⁸	Box trainer	3
McGill Inanimate System for Training and Evaluations of Laparoscopic Skills (MISTELS)⁸⁹	Box trainer	1, 3, 4
Southwestern video trainer stations⁴⁰	Video trainer	1, 3, 4
Black and Gould⁴¹ (5 tasks)	Video trainer	3
Fundamentals of Laparoscopic Surgery (FLS)⁴²	Box trainer	1, 3, 4
SIMULAB⁴³	Lap trainer with SimuVision LTS-10	1, 3
Legacy Inanimate System for Laparoscopic Team Training (LISETT)⁴⁴	Ethicon Laptrainer	3
Pelv–Sim⁴⁵	Pelv–Sim box trainer	3
Kolkman et al.⁴⁶ (5 tasks)	Box trainer	3
Clevin and Grantcharov⁴⁷ (5 tasks)	Box trainer	3

1, face; 2, content; 3, construct; 4, predictive; 5, concurrent; 6, discriminative.

Box trainers do not instantly provide feedback on performance. Therefore, educators (most often experienced surgeons) must evaluate skill parameters when using a box trainer. Mostly the objective structured assessment of technical skills (OSATS) is used for the objectively assessing skills on a traditional box trainer. The OSATS was first described by Martin et al.⁵³ and Reznick et al.⁵⁴ and is based on a structured clinical examination format, using a combination of checklists and global rating scores to judge performance. Consequently, metrics for box trainers are subject to inter- and intra-observer variation, and an—often—expensive faculty member must always be present for examination purposes. However, compared with most other training systems, the box trainer is still relatively inexpensive.^3,36

Example of box trainer implementation: FLS program

A laparoscopic training device should be integrated into laparoscopic training programs, and a training curriculum should be designed before choosing a training system. The FLS program, designed in 2004 by a committee of the Society of American Gastrointestinal Endoscopic Surgery and based on the laparoscopic box trainer, is an excellent example of a curriculum covering the basics in laparoscopic surgery. The FLS contains comprehensive coverage of cognitive (declarative knowledge) and psychomotor (procedural skill) components. The FLS includes educational material, mechanisms for assessment, and a didactic instruction.⁵² Within the FLS, five training modules are combined, including peg transfer, pattern cutting, ligating loop, and intracorporeal and extracorporeal knot tying.^48,49

The FLS exam is a combination of a written exam and a timed and scored laparoscopic skills evaluation in the box trainer. All tasks are scored according to pre-established standards using time and error measurements.⁵¹ Exams in the FLS are conducted by trained examiners using standardized criteria, enabling trainees to undergo a certification process.⁵⁵ The MISTELS is used within the FLS to assess technical skill.^50,56 The FLS exams are only available to a few centers in the United States and Canada. Consequently participation in the FLS system is expensive for participants following an individual residency program outside these two countries.

In numerous studies, the FLS program has been found to be a valid teaching and assessment tool for laparoscopic knowledge and skills. Performance in the FLS has also been shown to correlate with operative performance.^{20,48,50,52,57}

VR simulators

Recent advances in VR technology have led to the development of VR simulators for acquiring surgical skills and a measurement of performance or skill. VR simulators are computer-based. VR technology has developed software that attempts to replicate skills required for entire MIS procedures, such as cutting, grasping, and suturing, and thereby enabling the operator to acquire the psychomotor skills necessary to perform these procedures. VR software generates three-dimensional images on a two-dimensional monitor, which may appear unrealistic. It is important for a trainee to practice an entire procedure including decision-making options and not only basic suturing and knot-tying tasks, including theory lessons.⁵⁸ Haptic feedback provided on VR simulators is often absent or relatively poor compared with haptic feedback provided in box trainers.

VR simulators provide an instant, unbiased, reliable, and valid assessment of technical MIS skills.^59–61 Most of the currently validated VR simulators use time, errors, and psychomotor-related parameters for assessment such as path length traveled by instruments.⁶² The assessment and feedback provided by a VR simulator aim to only improve the psychomotor skills of a trainee. As a result, VR simulators do not require the presence of an educator to assess the trainee.

Several VR simulators have been developed and examined. Validated simulators available up to February 2011 are listed in Table 4. Below we discuss some of most commonly used VR simulators.

One of the first and most commonly used VR simulators developed as a task trainer is the Minimally Invasive Surgical Trainer-Virtual Reality (MIST-VR) (Mentice AB, Göteborg, Sweden). The device contains three modules: MIST Core Skills 1 and 2 contain simple tasks designed to acquire psychomotor skills, and the MIST suturing module is designed to train needle handling, suturing, and knot tying. The original MIST system does not provide haptic feedback or a VR abdominal environment. The Procedicus system, an optional modular simulation environment built by SimSurgery A/S (Oslo, Norway), additionally provides haptic feedback and different modules in an abdominal environment to the MIST-VR. The MIST-VR is the most extensively studied and validated system regarding construct, face, and concurrent validity and the assessment of basic laparoscopic skills in MIS.^35,61–65 The MIST-VR does not assess cognitive knowledge or complete laparoscopic procedures, including intraoperative problem solving. The system provides real-time feedback regarding skill-based errors based on an electromagnetic field. A web-enabled platform allows for remote performance monitoring and administration.

The LapSim VR laparoscopic simulator (Surgical Science, Göteborg) has a high degree of realism regarding graphics and tissue–instrument interaction.²⁴ The system provides nine realistic tasks that closely resemble an operative field. Objects are deformable and may “bleed.” The LapSim also features a scoring game module that integrates different skills at various levels coupled to a scoring system. Haptic feedback is optional. Various studies have proven construct, face, and content validity of the LapSim and its ability to distinguish between novice and experienced laparoscopic surgeons.^66–68

The SIMENDO (DelltaTech, Delft, The Netherlands) is a laparoscopic simulator designed to train hand–eye coordination motor skills. It provides an easy-to-use plug-and-play system for surgical trainees. The system does not provide haptic feedback. Verdaasdonk et al.⁶⁹ established content, face, and concurrent validity of the SIMENDO and found construct validity for the simulator training program. To produce objective assessments, the SIMENDO includes the parameters task time, instrument collisions with non-target objects, and total path length for the right and left instruments.⁷⁰ The SIMENDO is able to participate in a serious online gaming environment, creating an online competition for VR simulation training. This stimulates voluntary skills training.⁷¹

The LapMentor VR laparoscopy simulator (Simbionix, Cleveland, OH) is an adapted version of the LS500 surgical simulator with added haptic feedback. The device may be used to practice basic laparoscopic skills as well as complex skills and total surgical laparoscopic procedures (e.g., a laparoscopic cholecystectomy). Face validation of the Simbionix LapMentor VR training module was demonstrated by Ayodeji et al.⁵⁹ A virtual instructor guides the trainee and provides feedback during the simulation. Parameters used in assessments include total time, motion analysis parameters, and safety parameters (e.g., complications such as perforation and blood loss). Didactic parameters are included as decision-making options such as conversion.

VR simulators provide rapid and precise results on many measures of skill and appear to be good instruments for the objective assessment of surgical skills.^54,61,72,73 Various parameters are utilized for feedback and assessment purposes. Most VR systems provide objective feedback on time, path length, and motion efficiency. However, most commercially available VR simulators are offered with a broad range of options and no predefined criteria or information regarding the intensity or duration of training needed to achieve surgical competence. Beside the expensive hardware, VR simulator software is often expensive and requires frequent maintenance and updating.⁷

AR simulators

AR combines physical reality (such as in a box trainer) and VR into one system. Haptic feedback is maintained, using original laparoscopic instruments and tactile tasks. Additionally, objective measures of performance are generated.⁶⁶ AR devices are equipped with modules that simulate a laparoscopic environment and allow performance of tasks related to the box trainer tasks within the construct of the simulator.

Seven AR simulators are currently available. These simulators vary from relatively simple augmented box trainers with a separate assessment module to more advanced simulators including demo videos, projection of a realistic environment during performance, and assessment of performance.⁷⁴ Hardware and software costs of an AR simulator are comparable to those of a VR simulator but depend on the modules purchased.

A widely used AR simulator is the ProMis AR laparoscopic simulator (Haptica Inc., Boston, MA). The ProMis measures movements of marked instruments by a passive vision-tracking system. Three separate cameras capture the internal movements of marked laparoscopic instruments from three different angles. The tracking system is situated in a large mannequin, and its design allows measurement of motions in the x, y, and z directions. The distal end of a laparoscopic instrument shaft is covered with yellow electrical tape to serve as a reference point for the tracking systems.³⁴ Construct and face validation for the ProMis was found by Van Sickle et al.³⁴

Another widely used AR simulator is the SurgicalSIM (LTS3E) (METI, Sarasota, FL), a video-laparoscopic training system based on an integrated Microsoft Windows (Redmond, WA) computer and software. The simulator incorporates a sensor carousel to computerize scoring of performance. The system provides a set of 10 skill and coordination tasks that are scored on speed and precision. Construct and concurrent validation of the SurgicalSIM was shown by Soyinka et al.⁷⁵ and Mathis et al.⁷⁶ Task performance scoring is based on MISTELS. A unique feature of this device is the optional addition of a “tensiometer,” which electronically verifies knot integrity. Knot integrity is defined as a tensile strength of at least 25 N. The SurgicalSIM has multiple training programs for general surgery, gynecology, and urology. A drawback includes the dependency of the SurgicalSIM on an electromagnetic field required for motion tracking. Consequently any metallic object used may disturb the electromagnetic field and therefore potentially invalidate any recorded tracking data. Only surgical instruments supplied with the simulator may be used.

A device that allows free manipulation of a standard MIS instrument and tracks its movements in any box trainer without using a surrounded infrared system is the TrEndo, developed by Chmarra et al.⁷⁷ at the Delft University of Technology. The TrEndo system is a relatively new tracking device constructed as a trocar through which any MIS instrument may be inserted. The TrEndo consists of a two-axis gimbal mechanism with three optical eyes. The TrEndo aims to provide detailed performance assessment on inanimate box trainers. Tracked and recorded motions include path length, insertion distance, angular area, volume, and time as an objective quantitative measure of performance (motion analysis parameters). The TrEndo is a low-cost device that may be interesting for institutes already using box trainers.

Discussion

This review provides an overview of currently available MIS simulators and their established validity, reliability, and other central attributes.

Simulator training provides an opportunity for repetitive practice and trial and error in the acquisition of new laparoscopic skills without pressure or consequence from clinical reality. Simulators are a solution to current working hour restrictions and scarce operating room resources and may bypass an early learning curve, the latter of which is associated with an increased complication rate and thus injury and discomfort to patients.^78–80

All systems show a faster acquisition of basic psychomotor skills. However, no system is superior to any other.⁷ Low-cost box trainers utilize authentic laparoscopic instruments, provide haptic feedback, and provide an opportunity to train numerous individuals simultaneously.⁸¹ Validated box-trainer tasks can be used to measure performance based on a single assessment.⁵¹ A disadvantage of box trainers is obligatory use of the OSATS assessment method, rendering it subject to subjective supervisors' interpretation and scoring. Motion tracking (e.g., by use of the TrEndo) may address this problem, as an automatic objective feedback and assessment instrument for box trainers. Motion-analyzing parameters may be combined with real-time assessment of knot quality using a tensiometer.⁸² Recruitment of supervisors and faculty time present the major expenses in using box trainers. Available VR and AR systems demand less faculty attendance during practice; however, limitations include a longer “warming up” for the trainee to fully comprehend the system and a lack of standard validated assessment methods.⁶⁵ Most VR and AR simulators have no validated predictive validity. Performance scores based on construct validity cannot be used for proficiency-based training because an expert-derived performance score may be inaccurate.

It is imperative that key measurements such as time and path length are integrally evaluated because a non-integrative evaluation per variable is not sufficient to improve on overall MIS skills. Trainees must to be trained to attain a certain level of proficiency and not merely be assessed on the amount of training time completed.

None of the current simulators and assessment methods can provide all data required to make an all-inclusive evaluation of physician's MIS skills possible. All feedback provided by current available simulators is (still) less when not combined with an expert's feedback.⁸³

We believe that VR will play an increasing role in the future, taking into account current VR software development (e.g., the integration of haptic feedback, anatomy derived from computed tomography or magnetic resonance imaging datasets, and serious gaming).

For a skills lab to invest in simulators, a validated, multiyear training program on multiple-level training should be developed. Specialists, residents, educationalists, and industrial designers should collaborate in such development. We should learn from similar difficulties regarding the use of simulators in training encountered in other domains such as the military and aviation.⁸⁴ It is imperative that training be structured within a standardized curriculum.⁸⁵ Such a curriculum should include more than one training modality, ranging from a low-fidelity training suited for novices to high-fidelity simulators that may simulate entire procedures. As observed by experts in aviation, novices are more likely to effectively acquire skills when situated in a simple simulation training environment than when directly placed in a “real-life” cockpit where the complexity of instrumentation combined with the pressure to perform may be overwhelming.⁸⁴ This makes the box trainer a good candidate for inclusion in a curriculum for MIS training. Once basic laparoscopic skills have been attained and laparoscopic residents tend to become equally dextrous and confident with both hands, the training program may be extended with more complex VR simulations and include entire procedures (e.g., cholecystectomy). It is important to also regard anatomic variations, for seldom in real life will a “stereotypical” situation be encountered. If such variations are not programmed into a simulator, there will be a risk of awkward creativity, in which the trainee will design his or her own techniques to face such challenges. Performance will then be conforming to the optimal learning curve of the simulator itself, thereby defeating the goals of the simulator and also patient treatment and safety.

If valid metrics are not used, reliable assessment is not possible. So far, reliability and validity studies have focused mainly on technical skills. Skill acquistion, of course essential, is not the only competency when performing surgery on patients. Surgeons also require a core knowledge base, clinical decision-making and communicative skills, and the ability to think and work under stress in a team setting.⁸⁶ All these aspects must be trained to perform a given task correctly. Therefore, it may be better to design and evaluate a comprehensive training program instead—as in most existing training—of only validating one aspect or part of a procedure that can be performed on a simulator. As Satava⁸⁷ recently suggested, “Simulators are only of value within the context of a total educational curriculum, the technology must support the training goals.”

Given the ubiquity of threat and error, effective teamwork is required to ensure safety. For choosing the right simulator for each concerned skill, more rigorous studies with longer follow-up of trainees and assessment of the influence of more peripheral aspects such as team training are needed.

Further research is needed to combine team training (crew resource management [CRM]) with single simulator training. As is known in aviation, CRM is required for a team, including recording and evaluation of errors. Participants can be effectively taught using briefing and debriefing approaches.^88,89 CRM is defined as the mental processes used for gaining and maintaining situational awareness, for solving problems, and for making decisions such as conversion to open surgery if needed.⁹⁰ The potential value of possible future simulators combining CRM, complete procedural performance, and automated direct feedback is immense. A laparoscopic box trainer is easy to deploy in an operating room for team training sessions. Only three VR and AR simulators include some form of CRM (Table 3). So far, there are no validation studies to evaluate these simulators' utility for CRM.

A limitation of present review includes a lack of available structured data. Because of a lack of standardization in published studies, outcomes were not easily comparable. No study had defined cutoff points to differentiate competent and non-competent surgeons. Meta-analysis was therefore not possible. We chose to provide an overview of the most commonly used and completely validated simulators.

Unfortunately, most included validation studies did not estimate concurrent validity. Concurrent validity is difficult to measure as operating room conditions often differ from those in the laboratory and therefore are impossible to standardize. As most validation studies focused on the use of a simulator as a training device, more research is needed to confine validity concepts to the use of a simulator as an objective assessment method.

In conclusion, no existing simulator may teach the whole range of skills required for competent psychomotor skills or procedures in MIS.

None of the methods of simulation examined in this study has shown better results than other forms of surgical training. Single simulators will not replace the commitment of surgical educators, but because of restricted working hours and ethical discussions, surgical residents will be forced to spend an increasing amount of time with MIS simulators. This is amplified by an increasing number of laparoscopic procedures compared with open surgery, in which the resident could study the required anatomy prior to performance of the similar procedure laparoscopically.

There is therefore a need for integrated standardized training programs specifically designed to improve MIS competence during residence training, complemented by a mentored program including cognitive and communication competence tailored to individual needs.^91–93 This program should constitute knowledge-based learning, a stepwise technical skills pathway, provision of ongoing feedback as based on validated metrics, and measurable progression toward proficiency goals, thereby enabling transfer to a realistic environment. Scott and Dunnington⁹³ described the National Skills Curriculum designed by the American College of Surgeons; the College stated that “distributed, deliberate and structured practice using performance-based endpoints is an ideal method for teaching many technical skills using simulators to ensure “operating room readiness” for residents.”

In our opinion a comprehensive training program ideally combines the following multiple stages:

1. Training in MIS should be introduced in a surgical curriculum as early as possible.⁹⁴ First, attention should be paid to the acquisition of basic laparoscopic skills, such as suturing to improve hand–eye coordination. This could be achieved by a basic course such as the FLS on a low-cost box trainer or AR system. Feedback on time, smoothness, and path length in a box trainer could be provided by using a tracking system. Examination at the end of the first year can be executed by experts using the OSATS scoring system.

2. Second, the trainee should participate in a theoretical course regarding steps in MIS procedures (e.g., the critical view of safety during a laparoscopic cholecystectomy).⁹⁵ Practice should here include performance of entire procedures on a box trainer, a VR system, or an AR system. Key steps are preferably trained in this way. Goals must be set and achieved prior to progression to the next stage.

3. Third, a team training program should be developed and implemented to improve cognitive and interpersonal skills needed to manage a procedure within an organized operating room team. Team training has become an essential driving force in reducing medical errors and increasing communication in the operating room. Some of the new-generation VR and AR simulators made the first steps toward team training (Table 4), replacing the need for animal models and cadavers.⁹⁶

In our opinion, the above proposal for a curriculum will provide the trainee with a comprehensive training program that provides full attention to not only technical skill but also to critical steps such as procedural decision-making and interpersonal management, which integrally will allow performing entire MIS procedures on patients in the operating room. A completely validated curriculum embedding technical skills training, assessments, and team training will still be subject to an ever-changing environment. Taking into consideration that an AR system provides realistic haptic feedback but is considerably more expensive than other devices and that CRM, including serious gaming, is still in its infancy, we suggest a laparoscopic box trainer including a low-cost motion-tracking device would at this moment fit all stages. Continuous development and adaptation to new training devices and simulators will still be required.

Footnotes

Disclosure Statement

No competing financial interests exist.

References

Deziel

, Millikan

, Economou

, Doolas

, Ko

, Airan

. Complications of laparoscopic cholecystectomy: A national survey of 4,292 hospitals and an analysis of 77,604 cases. Am J Surg, 1993; 165:9–14.

Moore

, Bennett

. The learning curve for laparoscopic cholecystectomy. The Southern Surgeons Club. Am J Surg, 1995; 170:55–59.

Scott

, Bergen

, Rege

, Laycock

, Tesfay

, Valentine

, Euhus

, Jeyarajah

, Thompson

, Jones

. Laparoscopic training on bench models: Better and more cost effective than operating room experience? J Am Coll Surg, 2000; 191:272–283.

Grantcharov

, Kristiansen

, Bendix

, Bardram

, Rosenberg

, Funch-Jensen

. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg, 2004; 91:146–150.

Schijven

, Jakimowicz

, Broeders

IAMJ

, Tseng

LNL

. The Eindhoven laparoscopic cholecystectomy training course—Improving operating room performance using virtual reality training: Results from the first E.A.E.S. accredited virtual reality trainings curriculum. Surg Endosc, 2005; 19:1220–1226.

Sutherland

, Middleton

, Anthony

, Hamdorf

, Cregan

, Scott

, Maddern

. Surgical simulation: A systematic review. Ann Surg, 2006; 243:291–300.

Munz

, Kumar

, Moorthy

, Bann

, Darzi

. Laparoscopic virtual reality and box trainers: Is one superior to the other? Surg Endosc, 2004; 18:485–494.

Naylor

, Hollett

, Valentine

, Mitchell

, Bowling

, Ma

, Dineen

, Bruns

, Scott

. Can medical students achieve skills proficiency through simulation training? Am J Surg, 2009; 198:277–282.

Larsen

, Soerensen

, Grantcharov

, Dalsgaard

, Schouenborg

, Ottosen

, Schroeder

, Ottesen

. Effect of virtual reality training on laparoscopic surgery: Randomised controlled trial. BMJ, 2009; 338:b1802.

10.

Ahlberg

, Heikkinen

, Iselius

, Leijonmarck

, Rutqvist

, Arvidsson

. Does training in a virtual reality simulator improve surgical performance? Surg Endosc, 2002; 16:126–129.

11.

Hamilton

, Scott

, Kapoor

, Nwariaku

, Bergen

, Rege

, Tesfay

, Jones

. Improving operative performance using a laparoscopic hernia simulator. Am J Surg, 2001; 182:725–728.

12.

Goff

, Lentz

, Lee

, Fenner

, Morris

, Mandel

. Development of a bench station objective structured assessment of technical skills. Obstet Gynecol, 2001; 98:412–416.

13.

Schijven

, Jakimowicz

. The learning curve on the Xitact LS 500 laparoscopy simulator: profiles of performance. Surg Endosc, 2004; 18:121–127.

14.

Ramsay

, Wallace

, Garthwaite

, Monk

, Russell

, Grant

. Assessing the learning curve effect in health technologies. Lessons from the nonclinical literature. Int J Technol Assess Health Care, 2002; 18:1–10.

15.

Wanzel

, Ward

, Reznick

. Teaching the surgical craft: From selection to certification. Curr Probl Surg, 2002; 39:573–659.

16.

McDougall

. Validation of surgical simulators. J Endourol, 2007; 21:244–247.

17.

Carter

, Schijven

, Aggarwal

, Grantcharov

, Francis

, Hanna

, Jakimowicz

. Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc, 2005; 19:1523–153.2

18.

Bullock

, Kovacs

, Macdonald

, Story

. Evaluating procedural skills competence: inter-rater reliability of expert and non-expert observers. Acad Med, 1999; 74:76–78.

19.

Gallagher

, Ritter

, Satava

. Fundamental principles of validation, and reliability: Rigorous science for the assessment of surgical education and training. Surg Endosc, 2003; 17:1525–1529.

20.

Derossis

, Antoniuk

, Fried

. Evaluation of laparoscopic skills: A 2-year follow-up during residency training. Can J Surg, 1999; 42:293–296.

21.

Feldman

, Sherman

, Fried

. Using simulators to assess laparoscopic competence: Ready for widespread use? Surgery, 2004; 135:28–42.

22.

Schijven

, Jakimowicz

. Face, expert, and referent validity of the Xitact LS500 laparoscopy simulator. Surg Endosc, 2002; 16:1764–1770.

23.

Schreuder

, van Dongen

, Roeleveld

, Schijven

, Broeders

. Face and construct validity of virtual reality simulation of laparoscopic gynecologic surgery. Am J Obstet Gynecol, 2009; 200:540–548.

24.

Schijven

, Jakimowicz

. Virtual reality surgical laparoscopic simulators. Surg Endosc, 2003; 17:1943–1950.

25.

Goff

, Mandel

, Lentz

, Vanblaricom

, Oelschlager

, Lee

, Galakatos

, Davies

, Nielsen

. Assessment of resident surgical skills: is testing feasible? Am J Obstet Gynecol, 2005; 192:1331–1338.

26.

Cosman

, Hugh

, Shearer

, Merrett

, Biankin

, Cartmill

. Skills acquired on virtual reality laparoscopic simulators transfer into the operating room in a blinded, randomised, controlled trial. Stud Health Technol Inform, 2007; 125:76–81.

27.

Aggarwal

, Grantcharov

, Eriksen

, Blirup

, Kristiansen

, Funch-Jensen

, Darzi

. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg, 2006; 244:310–314.

28.

Buzink

, Goossens

RHM

, De Ridder

, Jakimowicz

. Training of basic laparoscopy skills on SimSurgery SEP. Minim Invasive Ther Allied Technol, 2010; 19:35–41.

29.

Haluck

, Gallagher

, Satava

, Webster

, Bass

, Miller

. Reliability and validity of Endotower, a virtual reality trainer for angled endoscope navigation. Stud Health Technol Inform, 2002; 85:179–184.

30.

Iwata

, Fujiwara

, Kodera

, Tanaka

, Ohashi

, Nakayama

, Koike

, Nakao

. Construct validity of the LapVR virtual-reality surgical simulator. Surg Endosc, 2011; 25:423–428.

31.

Madan

, Frantzides

, Tebbit

, Quiros

. Participants' opinions of laparoscopic training devices after a basic laparoscopic training course. Am J Surg, 2005; 189:758–761.

32.

Aggarwal

, Moorthy

, Darzi

. Laparoscopic skills training and assessment. Br J Surg, 2004; 91:1549–1558.

33.

Bholat

, Haluck

, Murray

, Gorman

, Krummel

. Tactile feedback is present during minimally invasive surgery. J Am Coll Surg, 1999; 189:349–355.

34.

Van Sickle

, McClusky

, Gallagher

, Smith

. Construct validation of the ProMIS simulator using a novel laparoscopic suturing task. Surg Endosc, 2005; 19:1227–1231.

35.

Seymour

, Gallagher

, Roman

, O'Brien

, Bansal

, Andersen

, Satava

. Virtual reality training improves operating room performance: Results of a randomized, double-blinded study. Ann Surg, 2002; 236:458–463.

36.

Rosser

, Rosser

, Savalgi

. Skill acquisition and assessment for laparoscopic surgery. Arch Surg, 1997; 132:200–204.

37.

Lentz

, Mandel

, Lee

, Gardella

, Melville

, Goff

. Testing surgical skills of obstetric and gynecologic residents in a bench laboratory setting: Validity and reliability. Am J Obstet Gynecol, 2001; 184:1462–1468.

38.

Risucci

, Cohen

, Garbus

, Goldstein

, Cohen

. The effects of practice and instruction on speed and accuracy during resident acquisition of simulated laparoscopic skills. Curr Surg, 2001; 58:230–235.

39.

Fried

. Simulators for laparoscopic surgery: A coming of age. Asian J Surg, 2004; 27:1–3.

40.

Korndorffer

JRJ

, Clayton

, Tesfay

, Brunner

, Sierra

, Dunne

, Jones

, Rege

, Touchard

, Scott

. Multicenter construct validity for southwestern laparoscopic videotrainer stations. J Surg Res, 2005; 128:114–119.

41.

Black

, Gould

. Measuring laparoscopic operative skill in a video trainer. Surg Endosc, 2006; 20:1069–1071.

42.

Ritter

, Scott

. Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov, 2007; 14:107–112.

43.

Dayan

, Ziv

, Berkenstadt

, Munz

. A simple, low-cost platform for basic laparoscopic skills training. Surg Innov, 2008; 15:136–142.

44.

Zheng

, Denk

, Martinec

, Gatta

, Whiteford

, Swanstrom

. Building an efficient surgical team using a bench model simulation: Construct validity of the Legacy Inanimate System for Endoscopic Team Training (LISETT) Surg Endosc, 2008; 22:930–937.

45.

Arden

, Hacker

, Jones

, Awtrey

. Description and validation of the Pelv-Sim: A training model designed to improve gynecologic minimally invasive suturing skills. J Minim Invasive Gynecol, 2008; 15:707–711.

46.

Kolkman

, Van de Put

MAJ

, Van den Hout

, Trimbos

JBMZ

, Jansen

. Implementation of the laparoscopic simulator in a gynecological residency curriculum. Surg Endosc, 2007; 21:1363–1368.

47.

Clevin

, Grantcharov

. Does box model training improve surgical dexterity and economy of movement during virtual reality laparoscopy? A randomised trial. Acta Obstet Gynecol Scand, 2008; 87:99–103.

48.

Fried

, Feldman

, Vassiliou

, Fraser

, Stanbridge

, Ghitulescu

, Andrew

. Proving the value of simulation in laparoscopic surgery. Ann Surg, 2004; 240:518–525.

49.

Derossis

, Fried

, Abrahamowicz

, Sigman

, Barkun

, Meakins

. Development of a model for training and evaluation of laparoscopic skills. Am J Surg, 1998; 175:482–487.

50.

Vassiliou

, Ghitulescu

, Feldman

, Stanbridge

, Leffondre

, Sigman

, Fried

. The MISTELS program to measure technical skill in laparoscopic surgery: Evidence for reliability. Surg Endosc, 2006; 20:744–747.

51.

Fraser

, Klassen

, Feldman

, Ghitulescu

, Stanbridge

, Fried

. Evaluating laparoscopic skills: Setting the pass/fail score for the MISTELS system. Surg Endosc, 2003; 17:964–967.

52.

Peters

, Fried

, Swanstrom

, Soper

, Sillin

, Schirmer

, Hoffman

. Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery, 2004; 135:21–27.

53.

Martin

, Regehr

, Reznick

, MacRae

, Murnaghan

, Hutchison

, Brown

. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg, 1997; 84:273–278.

54.

Reznick

, Regehr

, MacRae

, Martin

, McCulloch

. Testing technical skill via an innovative “bench station” examination. Am J Surg, 1997; 173:226–230.

55.

Keyser

, Derossis

, Antoniuk

, Sigman

, Fried

. A simplified simulator for the training and evaluation of laparoscopic skills. Surg Endosc, 2000; 14:149–153.

56.

Derossis

, Bothwell

, Sigman

, Fried

. The effect of practice on performance in a laparoscopic simulator. Surg Endosc, 1998; 12:1117–1120.

57.

McCluney

, Vassiliou

, Kaneva

, Cao

, Stanbridge

, Feldman

, Fried

. FLS simulator performance predicts intraoperative laparoscopic skill. Surg Endosc, 2007; 21:1991–1995.

58.

Abraham

, Abdelshehid

, Lee

, Alipanah

, Andrade

, Sargent

, Box

, Deane

, McDougall

, Clayman

. LapED 4-In-1 silicone training aid for practicing laparoscopic skills and tasks: A preliminary evaluation. J Endourol, 2008; 22:1351–1357.

59.

Ayodeji

, Schijven

, Jakimowicz

, Greve

. Face validation of the Simbionix LAP Mentor virtual reality training module and its applicability in the surgical curriculum. Surg Endosc, 2007; 21:1641–1649.

60.

Schijven

, Jakimowicz

. Construct validity: Experts and novices performing on the Xitact LS500 laparoscopy simulator. Surg Endosc, 2003; 17:803–810.

61.

Gallagher

, Satava

. Virtual reality as a metric for the assessment of laparoscopic psychomotor skills. Learning curves and reliability measures. Surg Endosc, 2002; 16:1746–1752.

62.

Gallagher

, Richie

, McClure

, McGuigan

. Objective psychomotor skills assessment of experienced, junior, and novice laparoscopists with virtual reality. World J Surg, 2001; 25:1478–1483.

63.

McNatt

, Smith

. A computer-based laparoscopic skills assessment device differentiates experienced from novice laparoscopic surgeons. Surg Endosc, 2001; 15:1085–1089.

64.

Taffinder

, Sutton

, Fishwick

, McManus

, Darzi

. Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: Results from randomised controlled studies using the MIST VR laparoscopic simulator. Stud Health Technol Inform, 1998; 50:124–130.

65.

Maithel

, Sierra

, Korndorffer

, Neumann

, Dawson

, Callery

, Jones

, Scott

. Construct and face validity of MIST-VR, Endotower, and CELTS: Are we ready for skills assessment using simulators? Surg Endosc, 2006; 20:104–112.

66.

Duffy

, Hogle

, McCarthy

, Lew

, Egan

, Christos

, Fowler

. Construct validity for the LAPSIM laparoscopic surgical simulator. Surg Endosc, 2005; 19:401–405.

67.

Hyltander

, Liljegren

, Rhodin

, Lonroth

. The transfer of basic skills learned in a laparoscopic simulator to the operating room. Surg Endosc, 2002; 16:1324–1328.

68.

Sherman

, Feldman

, Stanbridge

, Kazmi

, Fried

. Assessing the learning curve for the acquisition of laparoscopic skills on a virtual reality simulator. Surg Endosc, 2005; 19:678–682.

69.

Verdaasdonk

, Stassen

, Monteny

, Dankelman

. Validation of a new basic virtual reality simulator for training of basic endoscopic skills: The SIMENDO. Surg Endosc, 2006; 20:511–518.

70.

Verdaasdonk

, Stassen

, Schijven

, Dankelman

. Construct validity and assessment of the learning curve for the SIMENDO endoscopic simulator. Surg Endosc, 2007; 21:1406–1412.

71.

Verdaasdonk

EGG

, Dankelman

, Schijven

, Lange

, Wentink

, Stassen

LPS

. Serious gaming and voluntary laparoscopic skills training: A multicenter study. Minim Invasive Ther Allied Technol, 2009; 18:232–238.

72.

Taffinder

, Smith

, Huber

, Russell

, Darzi

. The effect of a second-generation 3D endoscope on the laparoscopic precision of novices and experienced surgeons. Surg Endosc, 1999; 13:1087–1092.

73.

Torkington

, Smith

, Rees

, Darzi

. The role of simulation in surgical training. Ann R Coll Surg Engl, 2000; 82:88–94.

74.

Botden

, Jakimowicz

. What is going on in augmented reality simulation in laparoscopic surgery? Surg Endosc, 2009; 23:1693–1700.

75.

Soyinka

, Schollmeyer

, Meinhold-Heerlein

, Gopalghare

, Hasson

, Mettler

. Enhancing laparoscopic performance with the LTS3E: A computerized hybrid physical reality simulator. Fertil Steril, 2008; 90:1988–1994.

76.

Mathis

, Wiegmann

. Construct validation of a laparoscopic surgical simulator. Simul Healthc, 2007; 2:178–182.

77.

Chmarra

, NH , Grimbergen

, Dankelman

. TrEndo, a device for tracking minimally invasive surgical instruments in training setups. Sensors Actuators A, 2006; 126:328–334.

78.

Aggarwal

, Undre

, Moorthy

, Vincent

, Darzi

. The simulated operating theatre: Comprehensive training for surgical teams. Qual Saf Health Care, 2004; 13,Suppl 1:i27–i32.

79.

Aggarwal

, Tully

, Grantcharov

, Larsen

, Miskry

, Farthing

, Darzi

. Virtual reality simulation training can improve technical skills during laparoscopic salpingectomy for ectopic pregnancy. BJOG, 2006; 113:1382–1387.

80.

Aggarwal

, Ward

, Balasundaram

, Sains

, Athanasiou

, Darzi

. Proving the effectiveness of virtual reality simulation for training in laparoscopic surgery. Ann Surg, 2007; 246:771–779.

81.

Furnee

EJB

, van Empel

, Mahdavian Delavary

, van der Peet

, Cuesta

, Meijerink

WJHJ

. Evaluation of a technical skills training program in surgical residents. J Laparoendosc Adv Surg Tech A, 2009; 19:615–621.

82.

Ritter

, McClusky

, Gallagher

, Smith

. Real-time objective assessment of knot quality with a portable tensiometer is superior to execution time for assessment of laparoscopic knot-tying performance. Surg Innov, 2005; 12:233–237.

83.

Stefanidis

, Korndorffer

JRJ

, Heniford

, Scott

. Limited feedback and video tutorials optimize learning and resource utilization during laparoscopic simulator training. Surgery, 2007; 142:202–206.

84.

Farmer

, Rooij van

, Riemersma

, Jorna

, Moraal

. Handbook of Simulator-Based Training, 2nd. Aldershot, UK: Ashgate Publishing Ltd., 1999.

85.

Anastakis

, Wanzel

, Brown

, McIlroy

, Hamstra

, Ali

, Hutchison

, Murnaghan

, Reznick

, Regehr

. Evaluating the effectiveness of a 2-year curriculum in a surgical skills center. Am J Surg, 2003; 185:378–385.

86.

Wetzel

, Kneebone

, Woloshynowych

, Nestel

, Moorthy

, Kidd

, Darzi

. The effects of stress on surgical performance. Am J Surg, 2006; 191:5–10.

87.

Satava

. Historical review of surgical simulation—a personal perspective. World J Surg, 2008; 32:141–148.

88.

Helmreich

. On error management: Lessons from aviation. BMJ, 2000; 320:781–785.

89.

Karl

. Staying safe: Simple tools for safe surgery. Bull Am Coll Surg, 2007; 92:16–22.

90.

Lingard

, Reznick

, DeVito

, Espin

. Forming professional identities on the health care team: Discursive constructions of the 'other' in the operating room. Med Educ, 2002; 36:728–734.

91.

Stefanidis

, Acker

, Heniford

. Proficiency-based laparoscopic simulator training leads to improved operating room skill that is resistant to decay. Surg Innov, 2008; 15:69–73.

92.

Aggarwal

, Grantcharov

, Darzi

. Framework for systematic training and assessment of technical skills. J Am Coll Surg, 2007; 204:697–705.

93.

Scott

, Dunnington

. The new ACS/APDS Skills Curriculum: Moving the learning curve out of the operating room. J Gastrointest Surg, 2008; 12:213–221.

94.

Hogle

, Chang

, Strong

VEM

, Welcome

AOU

, Sinaan

, Bailey

, Fowler

. Validation of laparoscopic surgical skills training outside the operating room: A long road. Surg Endosc, 2009; 23:1476–1482.

95.

Avgerinos

, Kelgiorgi

, Touloumis

, Baltatzi

, Dervenis

. One thousand laparoscopic cholecystectomies in a single surgical unit using the “critical view of safety” technique. J Gastrointest Surg, 2009; 13:498–503.

96.

Kinoshita

, Kanehira

, Matsuda

, Okazumi

, Katoh

. Effectiveness of a team participation training course for laparoscopy-assisted gastrectomy. Surg Endosc, 2010; 24:561–566.