Face,Content,and Construct Validity of the Take-Home EoSim Augmented Reality Laparoscopy Simulator for Basic Laparoscopic Tasks

Abstract

Background:

The eoSim^® laparoscopic augmented reality (AR) simulator has instrument tracking capabilities that may be suitable for implementation in laparoscopic training. The objective is to assess face, content, and construct validity of this simulator for basic laparoscopic skills training.

Methods:

Participants were divided into three groups: novices (no training), intermediates (<50 laparoscopic procedures), and experts (>50 laparoscopic procedures). Three basic tasks were completed on the simulator: thread transfer (1), cyst dissection (2), and tube ligation (3). A questionnaire was completed on realism, didactic value, and usability of the simulator. Measured outcome parameters were as follows: time, distance, time off screen, average speed, acceleration, and smoothness.

Results:

Mean ± standard deviation scores on realism were positive (Task 1 or T1; 3.9 ± 0.7, P = .13, T2; 3.7 ± 0.7, P = .07, T3; 3.7 ± 0.07), as well as didactic value (T1; 3.9 ± 0.8, P = .71, T2; 3.9 ± 0.8, P = .31, T3; 4.0 ± 0.8, P = .40). Usability was valued the highest, with mean scores between 3.9 and 4.3 (T1; P = .71, T2; P = .80, T3; P = .85). Scores did not differ significantly between groups. Experts were significantly faster (Task 1; P < .001, Task 2; P = .042, Task 3: P < .001) with higher handling speed for tasks 2 and 3 (Task 1; P = .20, task 2; P = .034, task 3; P = .049). Results for other outcome parameters also indicated experts had better instrument control and efficiency than novices, although these differences did not reach statistical significance.

Conclusions:

The eoSim laparoscopic AR simulator is regarded as a realistic, accessible, and useful tool for the training of basic laparoscopic skills, with good face validity. Construct validity of the eoSim AR simulator was demonstrated on several core variables, but not all.

Introduction

Minimally invasive surgery (MIS) is a frequently used approach for surgical procedures and is currently the gold standard for most gastrointestinal procedures. Laparoscopic surgery requires a specific skill set, including the ability to handle laparoscopic instruments with dexterity, reduced haptic feedback, and the two-dimensional representation of a three-dimensional environment.^1–4

Simulation techniques replace or amplify real experiences with controlled experiences that evoke or replicate relevant aspects of the real world.⁵ The wide spread of MIS created the need for accessible, easy to use simulation techniques to develop and train the required skills, to enhance patient safety. In addition, it allows for frequent, consistent, and deliberate practice, with objective assessment and feedback.

Simulators became commercially available in the mid-1990's.⁶ Several different types of simulation modalities have since been developed, such as traditional box trainers, virtual reality (VR) and augmented reality (AR) simulators. The costs of these simulators increase with the advancement of the technology used.⁷ The technologically of the most basic type, box trainers, provides realistic haptic feedback, but an expert is required to observe and assess the performance. VR simulators mostly lack realistic haptic feedback, but they do provide instructions for the exercises and objective assessment of the performance.⁸ Objective performance assessment can guide training and skills development, and provide preoperative warm-up.^9–11 Instrument movement metrics are mainly incorporated in VR and AR simulators with motion detection hardware, which likely has a negative influence on the cost-effectiveness of such a simulator.^12,13 AR simulators incorporate the combination of an objective assessment and feedback, while providing realistic haptic feedback, as real tissue and instruments are used in the simulator.^8,14,15

There is a need for an easily accessible and user-friendly simulator, as the importance of frequent, persistent training using a standardized module is widely acknowledged among surgeons.¹⁶ Unfortunately, there appears to be limited access to a simulator in their own environment.⁸ The eoSim^® laparoscopic AR simulator (eoSurgical Ltd., Edinburgh, United Kingdom) is a relatively low-cost simulator, which could meet these needs. However, to prove this, a validation study has to be performed.^17–19 Therefore, the aim of this study is to validate the eoSim laparoscopic AR simulator, for the training of basic laparoscopic skills in terms of face, content, and construct validity.

Methods and Materials

Participants

For this study, participants were recruited and tested during a period of 5 months (February to June 2017) at the Radboudumc Nijmegen, Rijnstate hospital Arnhem en Catharina hospital Eindhoven in the Netherlands. There were also participants recruited during two national congresses (congress for Dutch endoscopic surgeons, March 2017, Amsterdam and Dutch annual surgical congress, May 2017, Veldhoven), who were tested on site.

The participants were divided into three groups according to their surgical training. Novices were defined as participants who had no relevant surgical experience, but were familiar with the concept of laparoscopic surgery (medical students and first-year residents). Participants who had some form of surgical training with laparoscopic experience were defined as intermediates (performed <50 laparoscopic procedures, the target group for this simulator) and participants with more extensive laparoscopic training were included as experts (performed >50 laparoscopic procedures). Participants were aware they were evaluated and were informed of the goal of the task, although no specific explanation concerning the measurement criteria was provided. Participants were also informed that all data were processed anonymously.

Materials

The eoSim laparoscopic AR simulator was used in a standard setup. This simulator ranges in price from £699 to £1298, depending on the included number of instruments and training modules.²⁰ It consists of a light-weight case that can hold the materials and instruments (Fig. 1). It has a high definition camera mounted on the inside of the case, which can be rotated in a horizontal and a vertical plane. The position of the camera in the case and the distance to the allocated workspace on the opposite side of the box are fixed. The camera was connected to a laptop using USB 2.0 cable. The laptop specifications were in accordance with the specifications as stipulated by Surgtrac. The laparoscopic instruments were provided with the EoSim: laparoscopic instruments, needle holders, suturing pad, thread transfer platform and materials for standard exercises. Colored markings on the instruments allowed for instrument tracking with the Surgtrac tracking software (right = red and left = blue). The software also provides instruction videos for all exercises and includes a playback function. The laptop screen placement was adjusted to the right height for each participant. The simulator itself was placed on a standard height table.

FIG. 1.

Setup of the eoSim^® laparoscopic AR simulator. AR, augmented reality.

Tasks

The first task, “thread transfer,” involved picking up a cord and maneuvering it through six plastic rings attached to a plastic platform that can be attached to the bottom of the box (Fig. 2.1). For the second task, “cyst dissection,” an inflated white balloon had to be shelled out intact from a second noninflated green balloon that was attached to the provided three-clip board. For this task, the white balloon was placed inside the green balloon and then inflated. This was to simulate a double-layered cyst, or cyst with an overlying remaining ovarian tissue or peritoneum. The task was completed once the outer balloon was cut off the inner balloon, preferably without damaging the inner balloon (Fig. 2.2). The third task, tube ligation, involved applying two preformed surgical extracorporeal knots to a tube (full-length finger cut of a latex glove), at two premarked points, and then dividing in between the knots with a scissor. The knots were tied in advance and inserted in a knot pusher, which was then used by the participant to put the loop in place and secure the knot. Hence, the more advanced technique of (intracorporeal) knot tying was omitted from this basic task. This task simulates the ligation of the appendix (Fig. 2.3).

FIG. 2.

Setup of the eoSim laparoscopic AR simulator for each of the three basic tasks. AR, augmented reality.

Protocol

The participants commenced by completing the first part of the questionnaire. Before starting each task, the participants were briefed by the observer on the purpose and goals of the task and they watched the instruction video provided by eoSurgical™. The participants were then asked to practice each task once to familiarize themselves with the eoSim laparoscopic AR simulator. Next, they performed the task again and the measurements were recorded by the program software. The tasks were performed with the instruments of the eoSim as instructed in the task information and instruction videos. After performing the tasks, the participant completed the questionnaire concerning their opinion on realism, didactic value, and usability of the eoSim laparoscopic AR simulator.

Outcomes

To assess face and content validity, each participant was asked to complete a questionnaire on realism, didactic value, and usability of the eoSim laparoscopic AR simulator after each task. This questionnaire has been used previously in validation studies of laparoscopic simulators.^14,15 The items were scored on a five-point Likert scale, in which a score of “1” was defined as strong disagreement and “5” as strong agreement. A score of “3” indicated a “neutral” response; hence, a mean ≥ 3.5 was considered a positive evaluation. The questionnaire included a section on demographics and (laparoscopic) surgical experience. The construct validity was assessed using the outcome parameters generated by the tracking software, by tracking the color-coded tips of each instrument separately. The measurements recorded were as follows: time needed to complete a task (speed; seconds), distance between instruments (instrument control and efficiency; centimeters), distance covered by instruments (precision and instrument control; meters), time the instruments were off screen (precision and instrument control; percentage), and average speed of instruments (millimeter per second). Two derivatives from the “speed” parameter were also included: acceleration (millimeter per square second [mm/s²]) and motion smoothness (millimeter per second to the third exponentiation [mm/s³]).

Statistical analysis

Face, content, and construct validity were determined by analyzing the mean outcomes. The questionnaire scores and outcomes of task performances were compared between the novices, intermediates, and experts group. Parametric statistics were used for comparisons between the groups for each task. All analyses were performed using SPSS version 22.0. A P value of <.05 was considered statistically significant.

Results

Demographics

A total of 102 participants were included in this study, of which 48 were untrained (novices), 20 were intermediates (performed <50 laparoscopic procedures; intermediates), and 34 were experts (performed >50 basic and >20 advanced laparoscopic procedures; experts). The intermediates were the most heterogeneous in terms of years of experience: with 15% 0–1 years, 35% 1–5 years, and 50% 5–10 years of experience. The participants had a mean age of 30.3 (standard deviation [SD] 8.2) years, with a mean age of 24.4 (SD 2.8) for novices, 32.1 (SD 4.7) for intermediates, and 37.1 (SD 9.0) for experts. Novices were 21% female, compared to 9% in both the intermediates and experts groups.

Face and content validity

General consensus existed among all participants on realism, didactic value, and usability of the eoSim laparoscopic AR simulator for all three tasks, as questionnaire scores did not differ significantly between the groups. The simulated tasks were scored positive (overall mean scores: 3.6–4.0 for three tasks) (Table 1), although this mean (SD) difference was not statistically significant (Task 1: 3.9 [0.6], 3.6 [0.9], 4.0 [0.6], P = .13; Task 2: 3.8 [0.6], 3.4 [0.8], 3.7 [0.7], P = .08; and Task 3: 3.8 [0.6], 3.3 [0.8], 3.7 [0.8], P = .05). “Behavior of the tissue” received the lowest scores (2.6–3.5) and was the only item that received scores of <3.0, while the “on-screen response” and “physical manipulation of the instruments” were valued best in terms of realism (Tables 1 –3) (average scores 4.0–4.3 for all three tasks). The simulator was deemed as a useful didactic tool for all three tasks, both as a training tool for novices and for trainee assessment with positive scores for all three tasks (3.9 [0.8], 3.9 [0.8], 4.0 [0.8], Task 1–3, respectively). These scores did not differ significantly between the groups (Tables 1 –3). The usability was the highest valued aspect of the eoSim, with all mean scores between 3.9 and 4.3 (Table 2), with a uniform opinion among the groups (Tables 1 –3). The target group, intermediates, considered it a useful training tool for all three tasks.

Table 1.

Questionnaire Scores on Realism, Didactic Value, and Usability for Thread Transfer Task

Task 1: thread transfer	Novices (n = 48)	Intermediates (n = 20)	Experts (n = 34)	Total (n = 92)	P
Realism
On-screen response of instruments	4.1 ± 0.8	3.8 ± 1.1	4.2 ± 1.0	4.1 ± 0.9
Physical manipulation graspers	4.0 ± 0.6	3.8 ± 1.2	4.0 ± 0.9	4.0 ± 0.9
Force feedback graspers	3.9 ± 0.8	3.7 ± 1.2	4.2 ± 0.8	4.0 ± 0.9
Task sufficiently realistic	4.0 ± 0.8	3.6 ± 0.8	4.0 ± 0.6	3.9 ± 0.8
Behavior of tissue	3.5 ± 0.9	2.9 ± 1.0	3.3 ± 1.1	3.3 ± 1.0
Overall	3.9 ± 0.6	3.6 ± 0.9	4.0 ± 0.6	3.9 ± 0.7	.13
Didactic value
To train novices	4.1 ± 0.8	4.4 ± 0.7	4.3 ± 0.9	4.2 ± 0.8
Assessment skills trainee	3.7 ± 1.0	3.7 ± 1.0	3.5 ± 0.9	3.6 ± 1.0
Overall	3.9 ± 0.8	4.1 ± 0.7	3.9 ± 0.8	3.9 ± 0.8	.71
Usability
Selecting and changing instruments	4.3 ± 0.6	4.6 ± 0.9	4.4 ± 0.7	4.4 ± 0.7
Simulator interface	3.9 ± 0.8	4.2 ± 1.0	3.7 ± 1.2	3.9 ± 0.9
Training tool for this task	4.1 ± 0.8	3.9 ± 1.0	3.9 ± 1.0	4.0 ± 0.9
Overall	4.1 ± 0.6	4.2 ± 0.8	4.0 ± 0.8	4.1 ± 0.7	.71

All measurements are presented as mean ± SD unless otherwise specified. Overall mean scores were compared using one-way ANOVA analysis.

ANOVA, analysis of variance; SD, standard deviation.

Table 2.

Questionnaire Scores on Realism, Didactic Value, and Usability for Cyst Dissection

Task 2: cyst dissection	Novices (n = 48)	Intermediates (n = 20)	Experts (n = 34)	Total (n = 92)	P
Realism
On-screen response of instruments	4.1 ± 0.8	3.8 ± 1.0	4.2 ± 0.9	4.1 ± 0.9
Physical manipulation graspers	4.1 ± 0.7	3.7 ± 1.1	4.1 ± 0.9	4.0 ± 0.9
Force feedback graspers	3.9 ± 0.8	3.8 ± 1.0	4.1 ± 0.9	3.9 ± 0.9
Task sufficiently realistic	3.7 ± 0.8	3.3 ± 1.2	3.1 ± 0.9	3.4 ± 1.0
Behavior of tissue	3.3 ± 0.8	2.6 ± 0.8	2.8 ± 1.2	3.0 ± 1.0
Overall	3.8 ± 0.6	3.4 ± 0.8	3.7 ± 0.7	3.7 ± 0.7	.08
Didactic value
To train novices	4.1 ± 0.9	4.2 ± 0.8	3.9 ± 0.8	4.1 ± 0.8
Assessment skills trainee	3.9 ± 0.9	3.5 ± 1.2	3.6 ± 0.8	3.7 ± 0.8
Overall	4.0 ± 0.8	3.8 ± 0.8	3.7 ± 0.8	3.9 ± 0.8	.39
Usability
Selecting and changing instruments	4.2 ± 0.6	4.1 ± 1.1	4.3 ± 0.6	4.2 ± 0.8
Simulator interface	3.9 ± 0.7	4.1 ± 0.9	3.7 ± 1.0	3.9 ± 0.8
Training tool for this task	4.1 ± 0.7	3.7 ± 1.3	3.8 ± 1.0	3.9 ± 1.0
Overall	4.1 ± 0.6	4.0 ± 1.0	4.0 ± 0.8	4.0 ± 0.7	.80

All measurements are presented as mean ± SD unless otherwise specified. Overall mean scores were compared using one-way ANOVA analysis.

ANOVA, analysis of variance; SD, standard deviation.

Table 3.

Questionnaire Scores on Realism, Didactic Value, and Usability for Tube Ligation

Task 3: tube ligation	Novices (n = 48)	Intermediates (n = 20)	Experts (n = 34)	Total (n = 92)	P
Realism
On-screen response of instruments	4.2 ± 0.8	3.6 ± 1.0	4.0 ± 1.1	4.3 ± 1.0
Physical manipulation graspers	4.1 ± 0.6	3.4 ± 1.1	4.0 ± 1.1	3.9 ± 0.9
Force feedback graspers	4.0 ± 0.8	3.3 ± 1.1	4.0 ± 1.0	3.9 ± 1.0
Task sufficiently realistic	3.6 ± 0.7	3.1 ± 1.0	3.5 ± 0.9	3.5 ± 0.9
Behavior of tissue	3.3 ± 0.9	3.1 ± 1.0	3.1 ± 1.1	3.2 ± 0.9
Overall	3.8 ± 0.6	3.3 ± 0.8	3.7 ± 0.8	3.7 ± 0.7	.05
Didactic value
To train novices	4.3 ± 0.8	4.1 ± 0.9	4.1 ± 0.8	4.2 ± 0.8
Assessment skills trainee	3.9 ± 0.9	3.7 ± 1.1	3.6 ± 0.9	3.8 ± 0.9
Overall	4.1 ± 0.7	3.9 ± 0.9	3.9 ± −0.7	4.0 ± 0.8	.40
Usability
Instructions
Selecting and changing instruments	4.0 ± 0.8	4.1 ± 1.0	4.3 ± 0.8	4.1 ± 0.8
Simulator interface	4.0 ± 0.8	3.9 ± 1.0	3.8 ± 1.1	4.0 ± 0.9
Training tool for this task	4.1 ± 0.8	3.7 ± 1.1	3.8 ± 1.2	4.0 ± 1.9
Overall	4.1 ± 0.6	3.9 ± 0.9	4.0 ± 0.9	4.0 ± 0.8	.85

All measurements are presented as mean ± SD unless otherwise specified. Overall mean scores were compared using one-way ANOVA analysis.

ANOVA, analysis of variance; SD, standard deviation.

Construct validity

As shown in Table 4, expert participants performed all three tasks significantly faster than intermediates and novices (Task 1: P < .001, Task 2: P = .042, Task 3: P < .001). Also, experts demonstrated higher precision and better instrument control on the first task as the distance covered by the instruments to complete this task thread transfer was significantly lower compared to novices (P < .001, Table 4). For tube ligation, the distance covered was similar among all three groups; however experts were significantly faster with higher handling speed (of note, for cyst dissection [Task 2], the distance covered by instruments was significantly lower in novices group compared to the other two groups [P = .006, Table 4]). It was observed, however, that novices took much longer to initiate the first cut to the balloon, leaving the instruments in the same position for a relatively longer time than more experts, while not progressing in their task. The results for “distance between instruments,” “instrument handling,” “speed,” “acceleration,” and “smoothness” demonstrated a trend that indicates experts performed better than novices in terms of instrument control and efficiency. Novices appeared to hold their instruments further apart, while their movements were slower overall, compared to experts (Table 4). Novices managed to keep the percentage of time the instruments were out of view significantly lower than the other two groups when performing the cyst dissection task (P = .013, Table 4).

Table 4.

Construct Validity

Thread transfer	Novices (N = 48)	Intermediates (N = 20)	Experts (N = 26)	P
Time (seconds)	161 (110–231)	94 (111–163)	75 (63–104)	<.001
Distance between instruments (cm)	5.4 (4.2–5.8)	5.7 (4.8–6.9)	5.1 (3.9–5.7)	.490
Distance covered by instruments (m)	3.1 (1.7–4.3)	2.6 (1.8–4.3)	1.6 (1.2–2.0)	.001
Speed (mm/s)	2.4 (1.6–4.6)	3.0 (2.4–6.8)	2.8 (2–3.2)	.154
Time off screen (%)	9.3 (5.6–18.3)	10.9 (3.9–18.8)	10.8 (6.8–20.8)	.737
Acceleration (mm/s²)	1.2 (0.9–1.9)	1.6 (1.1–2.2)	1.3 (0.9–1.5)	.244
Smoothness (mm/s³)	0.02 (0.01–0.10)	0.04 (0.02–0.08)	0.05 (0.03–0.08)	.142
Cyst dissection
Time (seconds)	243 (141–302)	242 (134–398)	170 (79–239)	.042
Distance between instruments (cm)	4.7 (4.0–5.6)	4.6 (3.3–5.0)	4.5 (3.7–5.2)	.428
Distance covered by instruments (m)	2.8 (1.7–4.8)	4.6 (3.2–13.6)	3.5 (1.9–5.7)	.006
Speed (mm/s)	1.9 (1.2–4.4)	3.9 (2.1–7.9)	2.3 (1.8–3.5)	.034
Time off screen (%)	3.5 (1.9–9.5)	11.8 (4.0–22.1)	8.2 (3.9–25.0)	.013
Acceleration (mm/s²)	0.8 (0.5–2.0)	1.3 (0.9–2.7)	1.3 (1.0–1.6)	.060
Smoothness (mm/s³)	0.02 (0.0–0.7)	0.08 (0.0–1.0)	0.08 (0.01–2.00)	.861
Tube ligation
Time (seconds)	209 (180–275)	157 (135–197)	149 (123–173)	<.001
Distance covered by instruments (m)^a	3.7 (2.4–5.9)	3.3 (2.3–4.8)	3.8 (2.6–4.6)	.562
Speed (mm/s)^a	6.5 (4.3–9.8)	5.2 (3.2–8.2)	4.8 (3.4–6.1)	.049
Time off screen (%)^a	1.7 (0.9–5.6)	1.6 (0.5–4.2)	1.6 (0.4–3.5)	.460
Acceleration (mm/s²)^a	0.6 (0.4–1.1)	0.9 (0.7–1.2)	0.6 (0.5–0.7)	.008
Smoothness (mm/s³)^a	0.03 (0.01–0.10)	0.02 (0.01–0.08)	0.02 (0.02–0.03)	.430

Data from right-handed instrument only.

All measurements are presented as median (P25–P75) unless otherwise specified. Median scores were compared using nonparametric testing.

For tube ligation, the instruction as provided with the eoSim laparoscopic AR simulator set required using two instruments that were both equipped with red tracking markers. These were clearly shown in the instructional video. The tracking software, however, requires a red and blue marker to differentiate between left and right. By following the instructions as intended, the resulting data concern the right-handed instrument only and several metrics that require input from both hands could not be incorporated.

Considering the heterogeneity within the included groups in terms of laparoscopic training experience, post hoc subanalyses were performed for different variations of cutoff values for the novices, intermediates, and experts groups based on years of experience. These variations in group classification did not provide significantly different results (not shown) when comparing the same outcome parameters of the tracking software.

Discussion

The eoSim laparoscopic simulator is an AR simulator that is portable, relatively low in cost, and easy to use, while providing objective performance metrics to assess the technical skills. In this study, good-to-excellent face and, particularly, content validity were demonstrated regarding realism, didactic value, and usability features of the simulator. In general, novices valued the eoSim simulator the most, although these differences were not statistically significant.

This study also showed that the eoSim laparoscopic AR simulator is able to distinguish between the skill level of the participants for the following measurements: time, speed and instrument distance, which establishes the construct validity of the simulator for these parameters. However, significant differences between the groups could not be demonstrated for all variables. This could, in part, be due to the heterogeneity of the intermediates group.

The eoSim AR simulator provides a certain degree of haptic feedback during use and also direct assessment of the used material and tissue handling as opposed to a VR simulator. This provides further insight into the quality of the performance to both the observer and the trainee. These data significantly contribute to a meaningful interpretation of the software-acquired data. Furthermore, it has been shown that the addition of haptic feedback in early surgical training may improve sensory perception of the trainee, facilitating the transfer of skills from the simulator to the operating room, which may be distorted or less authentic in VR simulators.^21–23 In addition, the mobile suitcase setup of this simulator allows it to be used virtually everywhere and requires only a laptop to install it. It is therefore very accessible for surgeons in training to practice laparoscopic skills in a safe setting before attempting such maneuvers on a patient.

There are several other low-cost laparoscopic simulators available, yet many of these have not been properly validated.²⁴ Previous studies have demonstrated the construct and concurrent validity of the eoSim laparoscopic AR simulator for intracorporal suturing.^25,26 These studies did not evaluate the basic tasks that were included in this study. In addition, face and content validity were extensively investigated in this study, showing the usability and additive value of this simulator. Construct validity of the eoSim laparoscopy AR simulator was investigated in a recent study.²⁵ However, a compound score was used in this study that does not allow the investigation of the separate parameters contributing to the construct validity. Therefore, no meaningful comparison of those results can be performed. In contrast, this study included a larger number of participants, with a separate target group, and the face and content validity were established for individual as well as compound variables.

Construct validity was shown for the variable time and handling speed. The majority of the other parameters were correlated to the level of experience, but were not significantly different. Interestingly, experts did not score as well as expected for some parameters. For example, results showed experts were off screen with their instruments during a higher percentage of time compared to less experienced participants. Possibly, experience allows experts to execute a basic laparoscopic task without having their instruments perfectly in view all the time. Time off screen is a relevant parameter as increased time spent out of view could increase the risk of collateral damage to tissue that is not visible. On the other hand, novices appeared less successful in completing the task correctly, despite appearing more careful in their movements. It would be interesting to know if increased handling speed and less time required to fulfill the task also lead to better results and fewer mistakes on the eoSim laparoscopic AR simulator. Therefore, it is important to investigate the concurrent validity of the eoSim simulator for basic laparoscopic tasks in future research.

Several limitations of this study need to be considered. Participants were aware they were evaluated, which could affect their performance. However, it is likely that this effect would have affected all participants, not exempting a particular group. All results were processed anonymously, which was also communicated to the participants. During exercises, the participants noticed the changing focus of the installed webcam, which could not be manually adjusted to the users' preference. This could affect their opinion on the simulator and their performance while using it. Neutral scores were frequently encountered, which may be due to the basic nature of the exercises, making it more difficult to imagine their applicability in an actual operative field and value the simulator for these tasks. Of note, for tube ligation, the data could not be fully included as tracking software was only partially functional when performing the task as instructed. The color coding demonstrated in the instructional video includes red markers on both instruments, which do not allow it to differentiate properly between the left and right instrument. It was decided to include the available data, because all groups were affected similarly by this. Because smoothness and accelerations were measures that were derived from the “speed” and “distance” variable, they were not validated as primary outcome parameters. Finally, the heterogeneity of the intermediates group possibly made it more difficult to achieve significant differences between all three groups. The wide range in experience level could translate to a high interparticipant variability on performance outcomes. The most inexperienced participants in the intermediates group probably bear more similarities to the novices than to the more experienced participants in their own group. Post hoc subanalyses were performed to investigate the effect of experience level heterogeneity, but these did not yield significantly different results.

Conclusion

The eoSim laparoscopic AR simulator is regarded as a realistic, valuable, easy-to-use, accessible tool for the training of basic laparoscopic skills, with good-to-excellent face validity. Construct validity of the eoSim AR simulator was demonstrated on several, but not all, variables. The eoSim being an AR simulator also allows for observer-acquired assessments, which may provide additional information on participants' performance and training progress. The relatively low-cost, its mobility, and the ability to track the performance over time could make it an attractive choice for implementation in laparoscopic training curricula. Hence, future research is needed to further investigate construct and concurrent validity of the eoSim AR simulator, also including more advanced training exercises. It would be of interest to investigate the value of this simulator in assessing and tracking participants' training progression over time, also evaluating its discriminative ability.

Footnotes

Acknowledgment

We would like to thank eoSurgical for providing accessibility to their simulators.

Statement of Informed Consent

Informed consent was obtained from all individual participants included in the study.

Disclosure Statement

The authors of this article would like to declare there were no financial belongings in eoSurgical ltd. and have received no financial support for this study.

Funding Information

No funding was received for this article.

References

Pearson

, Gallagher

, Rosser

, Satava

. Evaluation of structured and quantitative training methods for teaching intracorporeal knot tying. Surg Endosc, 2000; 16:130–137.

Buzink

, Goossens

RHM

, Jakimowicz

. Image-based surgical proficiency: Reflection on human factors In: Meeting Diversity in Ergonomics. 16th World Congress on Ergonomics of the International Ergonomics Association, Maastricht, 2006, pp. 10–14.

Eyal

, Tendick

. Spatial ability and learning the use of an angled laparoscope in a virtual environment. Stud Health Technol Inform, 2001; 81:146–152.

Ballantyne

. The pitfalls of laparoscopic surgery: Challenges for robotics and telerobotic surgery. Surg Laparosc Endosc Percutan Tech, 2002; 12:1–5.

Gaba

. The future vision of simulation in health care. Qual Saf Health Care, 2004; 13:i2–i10.

Partridge

MAH

, Brennan

, Hennessey

IAM

. There is a worldwide shortfall of simulation platforms for minimally invasive surgery. J Surg Simul, 2015; 2:12–17.

Nguyen

, Braga

, Hoogenes

, Matsumoto

. Commercial video laparoscopic trainers versus less expensive, simple laparoscopic trainers: A systemic review and meta-analysis. J Urol, 2003; 190:894–899.

Botden SMBI. Augmented reality simulation for laparoscopic training. Realistic haptic feedback & meaningful assessment. Technische Universiteit Delft, Delft, 2009.

Aggarwal

, Moorthy

, Darzi

. Laparoscopic skills training and assessment. Br J Surg, 2004; 91:1549–1558.

10.

McGaghi

, Issenberg

, Cohen

, Barsuk

, Wayne

. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med, 2011; 86:706–711.

11.

Chen

, Green

, Colbert-Getz

, Steele

, Chou

, Lawson

, et al. Warm-up on a simulator improves residents' performance in laparoscopic surgery: A randomized controlled trial. Int Urogynecol J, 2013; 24:1615–1622.

12.

Pellen

, Horgan

, Barton

, Attwood

. Construct validity of the ProMIS laparoscopic simulator. Surg Endosc, 2009; 23:130–139.

13.

Zendejas

, Wang

, Brydges

, Hamstra

, Cook

. Cost: The missing outcome in simulation-based medical education research: A systematic review. Surgery, 2013; 153:160–176.

14.

Botden

, Berlage

, Schijven

, Jakimowicz

. Face validity study of the ProMIS augmented reality laparoscopic suturing simulator. Surg Technol Int, 2008; 17:26–32.

15.

Botden

, Buzink

, Schijven

, Jakimowicz

. ProMIS augmented reality training of laparoscopic procedures face validity. Simul Healthc, 2008; 3:97–102.

16.

Ganni

, Botden

SMBI

, Hamilton

, Bedi

, Lomanto

, Rao

, et al. Current state of training and evaluation of laparoscopic surgical skills. J Surg Open Access, 2017; 3:133–141.

17.

Schout

, Hendrikx

, Scheele

, Bemelmans

, Scherpbier

. Validation and implementation of surgical simulators: A critical review of present, past, and future. Surg Endosc, 2010; 24:536–546.

18.

Carter

, Schijven

, Aggarwal

, Grantcharov

, Francis

, Hanna

, et al. Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc, 2005; 19:1523–1532.

19.

Bradley

. The history of simulation in medical education and possible future directions. Med Educ, 2006; 40:254–262.

20.

EoSim models. www.eosurgical.com/collections/simulators-individual (accessed January 23, 2019).

21.

Ström

, Hedman

, Särnå

, Kjellin

, Wredmark

, Felländer-Tsai

. Early exposure to haptic feedback enhances performance in surgical simulator training: A prospective randomized crossover study in surgical residents. Surg Endosc, 2006; 20:1383–1388.

22.

Lamata

, Gómez

, Sánchez-Margallo

, Lamata

, del Pozo

, Usón

. Tissue consistency perception in laparoscopy to define the level of fidelity in virtual reality simulation. Surg Endosc, 2006; 20:1368–1375.

23.

Chamarra

, Dankelman

, van den Dobbelsteen

, Jansen

. Force feedback and basic laparoscopic skills. Surg Endosc, 2008; 22:2140–2148.

24.

, George

. A systematic review of low-cost laparoscopic simulators. Surg Endosc, 2017; 31:38–48.

25.

Hennessey

, Hewett

. Construct, concurrent, and content validity of the eoSim laparoscopic simulator. J Laparoendosc Adv Surg Tech A, 2013; 23:855–860.

26.

Retrosi

, Cundy

, Haddad

, Clarke

. Motion analysis-based skills training and assessment in pediatric laparoscopy: Construct, concurrent, and content validity for the eoSim simulator. J Laparoendosc Adv Surg Tech A, 2015; 25:944–945.