Face and Construct Validity of the SimSurgery SEP VR Simulator for Salpingectomy in Case of Ectopic Pregnancy

Abstract

Objective: To secure patient safety, skills needed for laparoscopy are preferably obtained in a non-patient setting. Therefore, we assessed face and construct validity of performance of a salpingectomy in case of ectopic pregnancy on the SimSurgery SEP VR simulator. Materials and Methods: Fifteen experienced gynecologists (≥ESGE level 2) and 17 novices (no laparoscopy experience) performed the Place Arrow (PA), Inspect Abdomen (IA), and Ectopic Pregnancy (EP) tasks and evaluated realism and didactic value of the simulator on 5-point scales. Their task performance was assessed according to the time needed to complete the tasks, total instrument path length, and parameters that indicated quality of performance. Results: The experienced gynecologists performed the PA task significantly faster (p=0.003, Mann–Whitney U-test) and with a shorter total instrument path length (p=0.001) compared to novices. The experienced gynecologists performed the EP task significantly better on parameters that indicate quality of performance, such as amount of blood loss (p=0.019), time to react to blood loss (p=0.020), and time of suction in the air (p=0.007) compared to novices. Between both groups, no significant differences were found at all for the IA task. Data from the questionnaire revealed that, in general, all participants had a favorable opinion toward the EP module on the SimSurgery SEP. Conclusions: This study demonstrates that the SimSurgery SEP simulator offers a realistic representation of the salpingectomy procedural task according to both experienced gynecologists as well as novices (face validity), and that the simulator can discriminate between different levels of expertise (construct validity) for the PA and EP tasks. The simulator is also perceived as an important additional training tool for gynecological residents. (J GYNECOL SURG 28:411)

Introduction

Over the years, laparoscopy has become the surgical procedure of choice for the management of ectopic pregnancy.^1–3 The laparoscopic approach has several potential advantages over laparotomy, such as faster recovery, shorter hospital admission, less impact on reproductive health, and better cosmetic results.^3,4 However, the relevance of these advantages are strongly dependent on the laparoscopic skills of the surgeon. Far fewer complications occur during laparoscopy when the procedure is performed by an experienced surgeon, in comparison to when the procedure is performed by a novice surgeon.^5–7 Tackacs et al. found that surgeon inexperience was significantly associated with unintended laparotomy during laparoscopic management of an ectopic pregnancy.⁷

Compared to open procedures, laparoscopy is more technically demanding and less efficient due to reduced dexterity, impaired vision, and tactile feedback.^8,9 The learning curve to reach the desired level of proficiency in laparoscopic skills is extensive. Therefore, it is preferable to train basic skills for laparoscopy in a preclinical setting, which can be done on virtual reality (VR) simulators.^10–14 Several VR systems are available that provide training in basic component skills.^12,15,16 Recently, systems also became available that focus on procedural training as well, such as the Ectopic Pregnancy (EP) module for the SimSurgery SEP VR simulator (SimSurgery AS, Oslo, Norway). The objective of this study was to assess the face and construct validity* of the simulated performance of a tubectomy in case of ectopic pregnancy on this simulator.¹⁶

Methods

Subjects

Fifteen experienced gynecologists from eight Dutch hospitals participated in this study. Their experience was at least at the second standard level of laparoscopy defined by the European Society for Gynaecological Endoscopy (ESGE). The first or basic level covers diagnostic laparoscopy, sterilization, needle aspiration of simple cysts, and ovarian biopsy. The second or intermediate level is reached by finishing normal training during specialization in obstetrics and gynecology and consists of five salpingotomies for ectopic pregnancy, five salpingectomies, five salpingecto-oophorectomies, eight ovarian cystectomies, five adhesiolysis including moderate bowel adhesions, and five treatments of mild or moderate endometriosis including salpingostomy and salpingo-ovariolysis. The third or advanced level requires extensive training and consists of hysterectomy, myomectomy, treatment of incontinence, surgery for severe endometriosis, extensive adhesiolysis including bowel and ureter, and repair of simple intestinal or bladder injuries. The fourth level comprises repair of pelvic-floor defects, recto-vaginal nodules, oncologic procedures like lymphadenectomy, radical hysterectomy and axilloscopy, and other procedures not yet described.¹⁷

A second group was formed by 17 novices, who were medical students with no laparoscopy experience. All subjects participated on a voluntary basis. Before starting the first task on the VR simulator, all participants were informed about the aims and design of the study and gave their informed consent.

Simulator

All tasks were performed on the SimSurgery VR Simulator, using the SimSurgery Educational Platform (SEP) Basic and Ectopic Pregnancy modules. Before each task, the simulator software provided instructions and a demo video. Haptic feedback was not provided by this simulator. However, during the tasks, the participants received additional visual feedback, that is, color changes to indicate both success and error. Results (e.g., total time, path length, blood loss, number of errors, etc.) were recorded and presented to the trainee as numerical data and in graphs to visualize the learning curve after each exercise.

Tasks

All individual participants received standardized information (verbal and software instructions) prior to commencing the tasks on the simulator. Following the instructions, participants performed the Place Arrow (PA) task to become acquainted with the simulator. This task was additionally used to verify the level of performance expected by their previous live laparoscopy experience. Next, the participants performed the tasks of the Ectopic Pregnancy module—the Inspect Abdomen (IA) and Ectopic Pregnancy (EP) tasks—twice (see Fig. 1A,B). The EP task focuses on procedural skills, while the PA task trains bimanual tissue manipulation and the IA task inspection of the abdominal cavity, which are both basic skills. Overall performance on the PA task was assessed by the time needed to accomplish the task, total instrument path length (Fig. 2), and errors made. Overall performance on the IA task was assessed according to the time needed to accomplish the task, total instrument path length used (Fig. 2), and errors made. For the EP task, the SEP simulator provides feedback on parameters reflecting economy of surgery such as total time and path length, as well as on items that indicate quality of performance such as amount of blood loss, time to react to blood loss, excessive traction, quality of salpingectomy with minimal damage to related structures such as mesosalpinx, and time of suction in the air.

FIG. 1.

(A) Overview of study protocol. (B) Photo of actual screen during maneuver in basic skill tasks (top and middle), and photo of actual screen during instrument change in procedural task (bottom).

FIG. 2.

Time needed to accomplish the tasks (left) and total path length of the instrument tips covered upon completion of the tasks (right). Circles represent outliers within 1.5 times the interquartile range, whereas asterisks represent extreme or far outliers within 3 times the interquartile range.

Questionnaire

All participants filled out the first part of the questionnaire prior to performing the tasks. This contained questions related to demographics and individual laparoscopic experience. After completing the tasks on the VR simulator, the participants were asked to answer the remaining questions and judge the realism, practicality, and didactic value of the simulator and training modules on 5-point scales (where 1=“fully disagree” and 5=“fully agree”).

Data analysis and statistics

SPSS v17.0 was used to analyze all data. As the simulator performance data are non-parametric, differences between groups were assessed using the Mann–Whitney U-test. We analyzed the performance data with a one-tailed Mann–Whitney U-test, while the two-tailed Mann–Whitney U-test was used for assessing the questionnaire data. A p≤0.05 was considered statistically significant. For the questionnaire data, the median and range are presented.

Results

Demographics

Thirty-two participants were recruited for this study. One person was excluded from the experienced group because fewer than 100 gynecologic laparoscopic surgeries had been performed. The mean age of the experienced participants was 47 years (range 33–59 years), compared to 24 years for the novice participants (range 21–33 years; p<0.001, two-tailed Mann–Whitney U-test). Furthermore, 27% of the experienced participants were female compared to 65% of the novice participants (p=0.034).

Of all experienced participants, 60% were performing laparoscopy at the second standard level defined by the ESGE, while 33% were performing laparoscopy at the third and 7% at the fourth standard level. None of the novice participants had performed laparoscopy before. Regarding previous experience on surgical simulators for laparoscopy, the experienced group stated they had had significantly more trials with laparoscopic simulators overall (p=0.010) and VR-trainers in particular compared to the novice group (p=0.051).

Place Arrow task performance

The experienced group performed the PA task significantly better than the novice group (time: p=0.003; path length: p=0.001; Fig. 2). No significant differences between the novice and experienced group were found on errors made.

We also carried out subgroup analyses comparing the novice group to a gynecologist performing at the second and third or higher standard level of laparoscopy defined by the ESGE respectively. Again we found significant differences on time taken and path length used upon completion of the task (time: p=0.038 and p=0.002 respectively; path length: p=0.029 and p<0.001 respectively) but not for errors made. When we compared gynecologists performing at the second standard level with gynecologists performing at the third or fourth standard level defined by the ESGE, no significant differences were found.

Face validity

After confirming that all participants performed at their expected level of performance, we assessed the face validity of the Ectopic Pregnancy module on the SimSurgery SEP. In general, all participants had a favorable opinion toward this module. There were no significant differences in opinion between the novice and experienced group (Table 1). However, the movement of organs and tissue upon manipulation was perceived to be not very realistic by both the experienced participants and the novices.

Table 1.

Face Validity of the Ectopic Pregnancy Module on the SimSurgery SEP VR Simulator

		Total group (n=31)	Novice group (n=17)	Experienced group (n=14)
Camera navigation in the IA task resembles clinical practice	Median (min–max)	4.00 (1–5)	4.00 (2–5)	3.50 (1–5)
The procedure trained (tubectomy for ectopic pregnancy) resembles clinical practice	Median (min–max)	4.00 (2–5)	4.00 (2–4)	4.00 (2–5)
Movement of organs/tissue upon manipulation appears real	Median (min–max)	3.00 (1–5)	3.00 (2–5)	3.00 (1–5)
Trocars are positioned correctly	Median (min–max)	4.00 (2–5)	4.00 (2–5)	3.50 (2–5)
Dexterity is genuine	Median (min-max)	4.00 (2–5)	3.00 (2–5)	4.00 (2–5)

Rating: 1=“I fully disagree” to 5=“I fully agree.” No significant differences were found between the novice and experienced group using the two-tailed Mann–Whitney U-test.

Construct validity

No significant differences were found between the performances of the novice and experienced group for the IA task. Again we carried out subgroup analyses and found that novices lost their target out of view more often than experienced participants performing laparoscopy at the second standard level (p=0.026). They also had more frequent camera–target collisions than experienced participants performing at the third and fourth standard level of laparoscopy defined by the ESGE (p=0.026).

For the EP task, the performance of the experienced group did not differ significantly from the novice group with regard to the time taken to accomplish the task and the total instrument path length (Fig. 2), nor when subgroup analyses were performed. However, on several parameters that indicate quality of performance, significant differences were found: amount of blood loss (p=0.019), time to react to blood loss (p=0.020), and time of suction in the air (p=0.007; Fig. 3). Subgroup analyses showed that the novice group spent significantly more seconds of suction in the air compared to the experienced participants performing laparoscopy at the second standard level (p=0.033) and third or fourth standard level (p=0.018). Compared to the latter group, the novices needed more time to react on blood loss (p=0.002) and ended the procedure with a higher amount of blood loss (p=0.001). We also found significant differences within the experienced group for amount of blood loss and reaction to blood loss, parameters more adequately covered by the gynecologists performing laparoscopy on the third and fourth standard level (p=0.009 and p=0.006 respectively).

FIG. 3.

Additional parameters providing insight into the quality of the performance on the Ectopic Pregnancy task. Circles represent outliers within 1.5 times the interquartile range, whereas asterisks represent extreme or far outliers within 3 times the interquartile range.

Didactic value

Both the novice and experienced participants had a favorable opinion toward the Ectopic Pregnancy module on the SimSurgery SEP (Table 2). However, both novice and experienced participants agreed that haptic feedback is necessary to perform the EP task as well as to train with the simulator. All participants strongly agreed that VR training is important additional training for gynecological residents. Even though haptic feedback could not be provided with this simulator, both groups still felt that experience gained at the SimSurgery SEP VR simulator can directly be used in daily practice and that the simulator measures the necessary values for determining the level of proficiency of the user. The experienced group regarded background or patient information important for training laparoscopic skills on the VR simulator.

Table 2.

Opinion Toward Contents and Didactic Value of the Ectopic Pregnancy Module on the SimSurgery SEP VR Simulator

		Total group (n=31)	Novice group (n=17)	Experienced group (n=14)
VR training is important additional training for gynecological residents	Median (min–max)	5.00 (3–5)	5.00 (4–5)	5.00 (3–5)
Background (patient) information is important for training on the VR simulator	Median (min–max)	4.00 (2–5)	4.00 (2–5)	4.00 (3–5)
Localization of targets ensured a relevant abdominal inspection in the IA task	Median (min–max)	4.00 (1–5)	4.00 (2–5)^*	3.50 (1–5)^*
Camera navigation is a relevant procedure before tubectomy on the simulator	Median (min–max)	4.00 (1–5)	4.00 (2–5)	4.00 (1–5)
Haptic feedback is necessary to perform the EP task	Median (min–max)	4.00 (2–5)	4.00 (3–5)	4.00 (2–5)
Haptic feedback is necessary for training with the SimSurgery SEP	Median (min–max)	4.00 (2–5)	4.00 (2–5)	4.00 (2–5)
Experience gained on the VR simulator can directly be used in daily practice	Median (min–max)	4.00 (2–5)	4.00 (2–5)	4.00 (2–5)
SimSurgery SEP can reliably determine the level of proficiency of the user	Median (min–max)	4.00 (2–5)	4.00 (2–5)	4.00 (3–5)

Rating: 1=“I fully disagree” to 5=“I fully agree”; 3=neutral. ^*p=0.032 (two-tailed Mann–Whitney U-test).

There were no significant differences in opinion between the two groups except for the item “localization of targets ensured a fully relevant inspection of the abdomen in the IA-task,” which the novice group rated significantly higher than the experienced group (p=0.032).

Discussion

The first studies done with laparoscopic VR simulators focused on the training of general surgical skills by means of component tasks. The added value of this type of preclinical training has been proven.¹⁸ In recent years, additional laparoscopic modules have been developed, focusing on training in other specialties such as gynecology, for example the SEP Ectopic Pregnancy Learning Module.¹⁹ Because these modules follow a different approach, their added value should be assessed separately.

This study demonstrates that the SimSurgery SEP simulator offers a realistic representation of laparoscopic salpingectomy for ectopic pregnancy according to experienced gynecologists (face validity) and that the simulator can discriminate between different levels of expertise (construct validity) for the PA task as well as for the EP task. Furthermore, participants in this study perceived the SimSurgery SEP ectopic pregnancy module as an important additional training tool for gynecologic residents.

Consistent with results published by Buzink et al. in 2009, we found significant differences in performance between the experienced and novice group for time and path length in the PA task, confirming that all participants performed at their expected level of performance.²⁰ Based on our hypothesis that the SimSurgery SEP can discriminate between different levels of expertise, we expected similar results for the IA and EP tasks. For the EP task, the experienced group performed significantly better than the novices on items that reflected quality of performance, but, against our expectations, not significantly faster or with shorter path length.

There are only a few studies available that have assessed construct validity for laparoscopic salpingectomy in case of ectopic pregnancy. These studies show that experienced gynecological surgeons perform a simulated laparoscopic salpingectomy significantly faster and with shorter instrument path length compared to novice gynecological surgeons.^10,14,21 These findings are consistent with published literature on the construct validity of VR simulator training in general surgery.^15,22,23 However, Moore et al. demonstrated that increased laparoscopic experience for gynecologists was associated with poorer performance on a simulated laparoscopic task.²⁴ They suggest that gynecologists may have certain expectations about the feedback given by the simulator that are not met and therefore have difficulty adapting to the parameters measured. In our study, the gynecologists agree that haptic feedback is necessary for training with the SimSurgery SEP overall (mean 4.00; 2–5), and to perform the EP task in particular (mean 4.00; 2–5), and therefore support this suggestion. Although VR training without haptic feedback appears to be not realistic, research proves that this does not affect the effectiveness of the training.²⁵

Apart from the lack of haptic feedback, which the experienced group is accustomed to when in the operating room (OR), possible other explanations for the fact that performance on the PA task did not correlate with performance on the EP task include novices hypothetically having more videogame experience because they are younger and therefore acquire the skills needed to perform the simulated task faster.^26,27 In addition, from our data on quality of performance for the EP task, we hypothesized that novices might have performed faster than expected because they have limited insight into the possible complications of the procedure and therefore work less accurately.

Several studies show that the differences between novices and experienced participants also become more apparent when more difficult tasks are performed.^28,29 This poses the question of whether the IA and EP task were difficult enough to distinguish properly between novice and experienced gynecologic laparoscopists. Another study conducted with SimSurgery SEP again showed no significant differences between novice and experienced groups when it came to total time taken or path length used to accomplish the camera navigation task, although both novice and experienced participants perceived the camera navigation task, which is similar to our IA task, as hard work and more challenging than anticipated.³⁰

At this moment, no studies are done as to whether training with SimSurgery SEP really improves performance of novice laparoscopists specifically in the OR. Most studies done to investigate whether skills gained through training with a VR simulator transfer to a clinical setting show a positive effect of VR training.^18,31–33 However, little is known as to what training scheme results in acquisition of the most relevant skills and thus leads to the best preparation before going into the OR. Inevitably, more research needs to be done in this area.

Footnotes

Acknowledgments

We would like to thank SimSurgery for providing equipment support. This work was presented at the 38th AAGL Global Congress of Minimally Invasive Gynecology, Orlando, Florida, November 16–19, 2009.

Disclosure Statement

No competing financial interests exist.

*

The terminology of the European Association for Endoscopic Surgery (EAES) consensus guidelines for validation of virtual reality surgical simulators is followed.

References

Howard

. Breaking new ground or just digging a hole? An evaluation of gynecologic operative laparoscopy. J Gynecol Surg, 1992; 8:143.

Tackacs

, Chakhtoura

. Laparotomy to laparoscopy: Changing trends in the surgical management of ectopic pregnancy in a tertiary care teaching hospital. J Minim Invasive Gynecol, 2006; 13:175.

Zhu

, Wong

, Bai

. Operative laparoscopy versus laparotomy for the management of ectopic pregnancy. Chin Med J, 2000; 113:810.

Murphy

, Nager

, Wujek

, Kettel

, Torp

, Chin

. Operative laparoscopy versus laparotomy for the management of ectopic pregnancy: A prospective trial. Fertil Steril, 1992; 57:1180.

Crist

, Gadacz

. Complications of laparoscopic surgery. Surg Clin North Am, 1993; 73:265.

Sokol

, Chuang

, Milad

. Risk factors for conversion to laparotomy during gynecologic laparoscopy. J Am Assoc Gynecol Laparosc, 2003; 10:469.

Takacs

, Latchaw

, Gaitan

, Chakhtoura

, De Santis

. Risk factors for conversion to laparotomy during laparoscopic management of an ectopic pregnancy. Arch Gynecol Obstet, 2005; 273:32.

Cuschieri

. Whither minimal access surgery: tribulations and expectations. Am J Surg, 1995; 169:9.

Rapport

IGZ

. ‘Risico's minimaal invasieve chirurgie onderschat, Kwaliteitssysteem voor laparoscopische operaties ontbreekt’ November 2007. www.igz.nl. 2011 April .

10.

Aggarwal

, Tully

, Grantcharov

et al. Virtual reality simulation training can improve technical skills during laparoscopic salpingectomy for ectopic pregnancy. Br J Obstet Gynaecol, 2006; 113:1382.

11.

Buzink

, Goossens

, de Ridder

, Jakimowicz

. Training of basic laparoscopy skills on SimSurgery SEP. Minim Invasive Ther Allied Technol, 2010; 19:35.

12.

Dunkin

, Adrales

, Apelgren

, Mellinger

. Surgical simulation: A current review. Surg Endosc, 2007; 21:357.

13.

Hasson

. Core competency in laparoendoscopic surgery. J Soc Laparoendosc Surg, 2006; 10:16.

14.

Larsen

, Grantcharov

, Aggarwal

et al. Objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator. Surg Endosc, 2006; 20:1460.

15.

Aggarwal

, Moorthy

, Darzi

. Laparoscopic skills training and assessment. Br J Surg, 2004; 91:1549.

16.

Carter

, Schijven

, Aggarwal

et al. Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc, 2005; 19:1523.

17.

ESGE. www.esge.org/education/endoscopic-training/esge-laparoscopy-standard. 2011 April .

18.

Gurusamy

, Aggarwal

, Palanivelu

, Davidson

. Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery. Br J Surg, 2008; 95:1088.

19.

Mettler

, Dewan

. Virtual reality simulators in gynecological endoscopy: a surging new wave. J Soc Laparoendosc Surg, 2009; 13:279.

20.

Buzink

, Botden

, Heemskerk

, Goossens

, de Ridder

, Jakimowicz

. Camera navigation and tissue manipulation: Are these laparoscopic skills related? Surg Endosc, 2009; 23:750.

21.

Schreuder

, van Dongen

, Roeleveld

, Schijven

, Broeders

. Face and construct validity of virtual reality simulation of laparoscopic gynecologic surgery. Am J Obstet Gynecol, 2009; 200:540.e1–8.

22.

Aggarwal

, Crochet

, Dias

, Misra

, Ziprin

, Darzi

. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg, 2009; 96:1086.

23.

Schijven

, Jakimowicz

. Construct validity: Experts and novices performing on the Xitact LS500 laparoscopy simulator. Surg Endosc, 2003; 17:803.

24.

Moore

, Grow

, Bush

, Seymour

. Novices outperform experienced laparoscopists on virtual reality laparoscopy simulator. J Soc Laparoendosc Surg, 2008; 12:358–362.

25.

Grober

, Hamstra

, Wanzel

et al. The educational impact of bench model fidelity on the acquisition of technical skill: The use of clinically relevant outcome measures. Ann Surg, 2004; 240:374.

26.

Grantcharov

, Bardram

, Funch-Jensen

, Rosenberg

. Impact of hand dominance, gender, and experience with computer games on performance in virtual reality laparoscopy. Surg Endosc, 2003; 17:1082.

27.

Shane

, Pettitt

, Morgenthal

, Smith

. Should surgical novices trade their retractors for joysticks? Videogame experience decreases the time needed to acquire surgical skills. Surg Endosc, 2008; 22:1294.

28.

Brunner

, Korndorffer

Jr , Sierra

et al. Determining standards for laparoscopic proficiency using virtual reality. Am Surg, 2005; 71:29.

29.

Van Dongen

, Tournoij

, van der Zee

, Schijven

, Broeders

. Construct validity of the LapSim: Can the LapSim virtual reality simulator distinguish between novices and experts? Surg Endosc, 2007; 21:1413.

30.

Buzink

, Christie

, Goossens

, de Ridder

, Jakimowicz

. Influence of anatomic landmarks in the virtual environment on simulated angled laparoscope navigation. Surg Endosc, 2010; 24:2993.

31.

Hart

, Doherty

, Karthigasu

, Garry

. The value of virtual reality-simulator training in the development of laparoscopic surgical skills. J Minim Invasive Gynecol, 2006; 13:126.

32.

Larsen

, Soerensen

, Grantcharov

et al. Effect of virtual reality training on laparoscopic surgery: Randomised controlled trial. Br Med J, 2009; 338:b1802.

33.

Seymour

. VR to OR: A review of the evidence that virtual reality simulation improves operating room performance. World J Surg, 2008; 32:182.