Transfer of Flexible Ureteroscopic Stone-Extraction Skill from a Virtual Reality Simulator to the Operating Theatre: A Pilot Study

Abstract

Aim:

To assess the learning curve of flexible ureteroscopic stone extraction using the UroMentor™ simulator and transfer of flexible ureteroscopic stone extraction skills to the operating theatre.

Materials and Methods:

After obtaining ethics approval, urology Post-Graduate Trainees (PGTs) from Post-Graduate Years (PGYs) 1 to 4 were recruited. During phase I, participants completed 3 weekly 1-hour training sessions on the UroMentor simulator practicing task 10, where two stones from the left proximal ureter and renal pelvis were extracted using a basket. Objective assessments by the simulator and subjective assessments using the validated Ureteroscopy-Global Rating Scale (URS-GRS) were used to establish the learning curve. During phase II, the URS-GRS tool was used to assess performance of participants in the operating theatre. URS-GRS scores obtained on the simulator and in the operating theatre were correlated.

Results:

In phase I, eight urology PGTs (PGY1–4) with mean age of 27.8 ± 2 (25–31) years participated in the study. PGTs practiced a total of 52 times, with a mean operative time of 14.6 ± 4.3 minutes and a mean fluoroscopy time of 10.4 ± 12 seconds. Competency in task 10 was achieved after seven trials on the UroMentor simulator. In phase II, 5 PGTs were assessed during 55 consecutive flexible ureteroscopic stone extraction in the operating theatre. The mean operative time was 51.4 ± 15.2 minutes and the mean fluoroscopy time was 29 ± 6 seconds. There was a significant positive correlation between URS-GRS scores obtained on the simulator and in the operating theatre (r = 0.9, p = 0.03), thus establishing predictive validity of performance on the UroMentor simulator.

Conclusions:

Introduction

Nephrolithiasis is a prevalent medical problem, affecting 1 in 11 individuals in the population.¹ While percutaneous nephrolithotomy, ureteroscopy, and extracorporeal shockwave lithotripsy are different minimally-invasive options for management of nephrolithiasis, the latest American Urological Association guidelines recommend ureteroscopic lithotripsy as the gold standard for management of most ureteral and some renal stones.² In addition to being used in the management of nephrolithiasis, flexible ureteroscopes are also used in the management of ureteropelvic junction obstruction and upper tract urothelial carcinomas.^3,4

Although flexible ureteroscopic lithotripsy is a minimally invasive surgery, it carries potential risks of major complications, such as ureteral perforations and avulsions.⁵ In addition, fluoroscopy is used intraoperatively; thus the risk of excessive radiation exposure to patients and operating theatre personnel. Finally, flexible ureteroscopy is associated with a steep learning curve requiring as much as 50 cases to achieve competency.⁶ Therefore, the Accreditation Council for Graduate Medical Education requires that Post-Graduate Trainees (PGTs) perform at least 60 ureteroscopies before finishing urology training program.⁷ However, given the latest reduction in work hours of PGTs, there are concerns regarding whether this number of ureteroscopic cases could be reached.⁸ It is difficult for surgical educators to rely exclusively on the operating theatre for teaching technical skills. Therefore, virtual reality simulators were introduced and validated for training and assessment of technical skills.^9

–14 These simulators give the opportunity for PGTs to obtain the early phase of the learning curve of certain procedures before performing them in the operating theatre. Thus, these simulators compensate for the deficiency in training hours for PGTs and improve patient safety.

The UroMentor™ simulator has been validated for training cystoscopy and ureteroscopy.^8,11 However, there are no data on how many times ureteroscopic stone extraction should be performed on the UroMentor simulator before performing this procedure in the operating theatre. In addition, it is still unknown whether skills acquired from practicing flexible ureteroscopic stone extraction on the UroMentor simulator are transferable to the operating theatre. There is paucity of high quality studies demonstrating the predictive validity of performance on the UroMentor simulator to performance in the operating theatre. Therefore, the aim of the present study was to assess the learning curve of flexible ureteroscopic stone extraction using the UroMentor simulator. The second aim was to assess the transfer of this skill to the operating theatre, thus establishing predictive validity of performance on the UroMentor simulator.

Materials and Methods

This study was conducted at McGill University Health Center after obtaining McGill University ethics approval (No. A03-E20-12B) and informed consents from all participants. Urology residency training in Canada is 5 years. At McGill University Health Center, there are three urology PGTs (one PGY1–2, one PGY3–4, and one PGY5) at a given time rotating on the endourology service. Before implementation of this training program on the UroMentor simulator, flexible ureteroscopic stone extraction was performed by PGTs in the PGY4 level. Regardless of their Post-Graduate Year (PGY) level, PGTs who wanted to participate in this study were recruited to be trained on the simulator immediately before starting their clinical endourology rotation and, thus, before performing ureteroscopic stone extraction in the operating theatre. Since each endourology rotation is 12 weeks, consecutive PGTs from PGY1 to 4 rotating on the endourology service were recruited for the study between September 2015 and May 2016. Each participant filled out a questionnaire regarding age, gender, handedness, PGY level, previous practice on virtual reality simulators, and previous experience in performing endourologic procedures, such as cystoscopy and semirigid and flexible ureteroscopy.

This study was conducted in two phases. The aim of phase I was to define the learning curve of flexible ureteroscopic stone extraction on the UroMentor simulator (Simbionix, Cleveland, OH) and the aim of phase II was to assess transfer of this skill to the operating theatre, thus establishing the predictive validity of performance on this simulator. During phase I, participants were trained on the UroMentor simulator task 10 for 3 consecutive weekly sessions lasting 1 hour each. Three hours of practice was arbitrarily chosen since there are no previous publications on the learning curve of ureteroscopic stone extraction on this simulator. Before the first training session, participants were oriented to the simulator. Task 10, or ureteroscopic stone extraction procedure, was chosen since it is the most complex task on the UroMentor simulator. This task required the use of a rigid cystoscope to enter the bladder and place a guidewire in the left ureteral orifice. Then a flexible ureteroscope and a stone basket were used to extract two stones from the left proximal ureter and renal pelvis under fluoroscopic guidance. At the end, a systematic examination of the left renal calices was performed (Fig. 1).

FIG. 1.

Description of task 10 on the UroMentor™ simulator. (A) A rigid cystoscope is used to enter the bladder. (B) A guidewire is inserted on the left side under fluoroscopic guidance. (C) A flexible ureteroscope is used to find the left proximal and renal pelvic stones. (D) The stones are extracted using a basket.

Following completion of the task, the simulator generated performance reports with several objective parameters, such as operative time, fluoroscopy time, number of ureteral wall traumas, and number of ureteral perforations. In addition, a single assessor (M.A.) used the validated Ureteroscopy-Global Rating Scale (URS-GRS) tool to subjectively assess the performance of participants performing successive trials of task 10 during the 3 weekly sessions.¹⁵ This assessment tool contained seven parameters, each scored on a Likert scale from 1 to 5 bringing the maximum score to 35. These parameters included respect for tissue, instrument handling, endoscope handling, time and motion, forward planning, use of assistants, and knowledge of the procedure.¹⁵ Competency on the learning curve of flexible ureteroscopic stone extraction using task 10 on the UroMentor simulator was determined when the URS-GRS scores reached a plateau.

In phase II, participants, who received training on the simulator, were observed performing consecutive flexible ureteroscopic stone extraction in the operating theatre during their endourology rotation. Participants were subjectively assessed by one assessor (MA) using the same URS-GRS tool. Furthermore, operative time, fluoroscopy time, and intraoperative complications were collected. The procedure and fluoroscopy times included all of the required components for ureteroscopic stone extraction, including retrograde pyelography, possible balloon dilatation, possible usage of ureteral access sheaths, and insertion of indwelling ureteral stents. Correlation between the URS-GRS scores on the simulator and inside the operating theatre was performed to assess the transfer of flexible ureteroscopic stone extraction skill from the simulator to the operating theatre, thus establishing predictive validity.

Statistical analysis

Data gathered from the questionnaires, the UroMentor simulator, and intraoperative variables were tabulated and analyzed. The Statistical Package of Social Sciences for Windows (SPSS, Chicago, IL) software version 20 was used. Descriptive data were presented in terms of numbers and percentages, median, or mean and standard deviation. Spearman's correlation coefficient was used to determine associations between continuous variables, and significance was considered whenever the two tailed p-value was <0.05.

Results

During phase I, eight urology PGTs (one PGY1, three PGY2, three PGY3, and one PGY4) with a mean age of 27.8 ± 2 (25–31) years participated in the study. All PGTs were right handed and two were female. Only one participant had practiced on a virtual reality simulator and none of them had previous practice on the UroMentor simulator before this study. On average, participants had performed 45 cystoscopies, 36 semirigid and flexible ureteroscopies, and 23 transurethral resections.

During the 3 weekly practice sessions, PGTs performed task 10 on the UroMentor simulator 52 times (mean: 6.4 ± 3.1; range: 3–10) with a mean operative time of 14.6 ± 4.3 minutes and mean fluoroscopy time of 10.4 ± 12 seconds. The learning curve on the UroMentor simulator reached a plateau after seven trials of task 10 (Fig. 2A). The mean practice time required to reach the plateau was 100 minutes. Furthermore, fluoroscopy time decreased reaching a plateau at the third trial (Fig. 3A). Operative time and number of traumas from the scope, guidewire, and basket decreased reaching a plateau after the fifth trial (Figs. 2B and 3B). There were no ureteral perforations. Furthermore, there was a positive significant correlation between the mean URS-GRS scores obtained on the first trial on the simulator and the number of previously performed semirigid and flexible ureteroscopies (r = 0.809, p = 0.015) (Fig. 4A). Previous semirigid and flexible ureteroscopic experience also had a negative correlation with fluoroscopy time (r = −0.299, p = 0.471) and operative time (r = −0.228, p = 0.588) of the first trial on the simulator. However, there was no correlation between previous experience and the number of traumas (r = −0.055, p = 0.898).

FIG. 2.

(A) Learning curve of flexible ureteroscopic stone extraction on the UroMentor simulator. (B) Mean URS-GRS score and mean operative time on the simulator. URS-GRS, Ureteroscopy-Global Rating Scale.

FIG. 3.

(A) Mean URS-GRS scores and fluoroscopy time on the simulator. (B) Mean URS-GRS scores and the mean number of traumas from the scope and instruments on the simulator.

FIG. 4.

(A) Correlation between the first URS-GRS score obtained on the simulator and mean number of previously performed semirigid and flexible ureteroscopies in the operating theatre. (B) Correlation of mean URS-GRS scores obtained on the simulator and in the operating theatre.

During phase II, five PGTs (two PGY2, two PGY3, and one PGY4) out of the eight who practiced on the UroMentor simulator during phase I were assessed to perform flexible ureteroscopic stone extraction in the operating theatre. They performed 55 consecutive flexible ureteroscopic stone extractions with an average of 9.1 ± 3.9 (range: 5–15) procedure per PGT. The average operative time was 51.4 ± 15.2 minutes, and average fluoroscopy time was 29 ± 6 seconds. Regression analysis showed that there was a strong correlation of URS-GRS scores obtained on the simulator and in the operating theatre (r = 0.9, p = 0.03) (Fig. 4B), thus establishing the predictive validity of performance on the UroMentor simulator.

Discussion

The Halstedian model of training relies on the introduction of trainees in the operating theatre from early phases of their learning curve to achieve competency in performing a particular surgical procedure.⁸ This training provides theoretical and practical knowledge of the procedure. It also satisfies several crucial surgical elements such as the experience of operating room environment and development of cognitive aspects of surgery, including decision-making, communication, and collaboration skills. However, concerns about patient safety in addition to restrictions in trainee work hours have resulted in dramatic reductions to operating theatre exposure necessitating the incorporation of validated virtual-reality simulators into the curricula of training programs to teach PGTs the early phases of the learning curve.^16,17 Interestingly, both psychomotor and cognitive skills, such as risk assessment could be acquired by appropriate training on the simulators.¹⁸ For instance, it has been shown that training on simulators improves dexterity during ureteroscopy. In a study by Watterson and Denstedt, manual dexterity was enhanced following training of 26 urology PGTs on a low-fidelity model while performing semirigid ureteroscopy.¹⁷

The UroMentor simulator is a validated computer-based virtual-reality simulator^8,19,20 offering semirigid and flexible ureteroscopy modules. Two prospective randomized controlled trials on the UroMentor simulator showed educational impact of this model in training semirigid and flexible ureteroscopy for medical students using a global rating scale.^17,21 They reported improved acquisition of ureteroscopic skills in novice trainees following training on the UroMentor simulator. However, there is no data regarding the learning curve of flexible ureteroscopic stone extraction on the UroMentor simulator. In addition, there is an absence of high quality studies regarding the predictive validity of this simulator indicating whether performance on this simulator predicts performance in the operative theatre.

Learning curve is used as a means of assessing surgical expertise and the number of procedures needed to gain surgical competence.⁸ However, the learning curve may differ according to the selected outcome criteria. For instance, when operative time was used as an outcome for percutaneous nephrolithotomy, the estimated case load of 60 patients was necessary to reach a plateau.²² However, when the stone-free rates were considered as an endpoint, the plateau was achieved at the initial cases.²³ In the current study, URS-GRS scores were used as the outcome to establish the learning curves since the simulator does not provide an overall objective score. Using the URS-GRS tool, the learning curve of ureteroscopic stone extraction, while performing task 10, reached a plateau after performing seven trials on the UroMentor simulator. The estimated time to complete seven trials of task 10 was 100 minutes. Therefore, on average, PGTs need to spend 100 minutes on the UroMentor simulator to achieve competency in performing ureteroscopic stone extraction on the simulator before performing ureteroscopic stone extraction in the operating theatre.

This study showed that the mean operative time on the simulator was 14.6 minutes and the operative time reached a plateau after the fifth trial. Similarly, Knoll and colleagues reported that untrained PGTs performed flexible ureteroscopy as rapid as trained residents on the UroMentor simulator by the fifth trial.⁵ In addition, in the current study, the increase in URS-GRS score was associated with a decrease in fluoroscopy time and the number of mucosal traumas (Fig. 3). Furthermore, the first trial URS-GRS scores on the simulator correlated with the number of previously performed semirigid and flexible ureteroscopies in the operating room. This is similar to other studies on the GreenLight and PERC Mentor™ simulators where performance on these simulators correlated with previous intraoperative experience.^11,12

Previous studies have demonstrated the validity of the UroMentor simulator.^{5,20,21,24
–26} While Ogan and colleagues reported the transfer of diagnostic flexible ureteroscopy skills from the UroMentor simulator to cadavers, Schout and colleagues demonstrated transfer of cystoscopy skills from the UroMentor simulator to the operating theatre.^24,25 However, there is an absence of high quality studies on the predictive validity of the UroMentor simulator regarding the transfer of ureteroscopic stone extraction skill to the operative theatre. In the present study, transfer of flexible ureteroscopic stone extraction skill from the UroMentor simulator to the operating theatre, or predictive validity, was confirmed by finding a correlation between the URS-GRS scores on the simulator and in the operating theatre.

Despite the various advantages of incorporating simulators in the training of PGTs, there are some limitations for using these simulators. For instance, the high cost of these simulators. Moreover, in a study by Chou and colleagues, the ability to perform ureteroscopic stone extraction was independent of whether the training method was on a virtual reality simulator or an ureteroscopy training model.²⁷

This study has some limitations. First, the absence of randomization in the study could have introduced bias. However, ethically we were not able to deny access to the simulator as an educational tool to PGTs. Moreover, there was some consideration regarding patient safety and ethical issue of placing patients at higher risk with PGTs performing flexible ureteroscopy without having them first practice on the simulator. Second, despite recruiting all PGTs rotating at the endourology service, the sample size was small. Third, there were variations in PGY level, previous ureteroscopic experience, number of practice sessions on the UroMentor simulator, and ureteroscopic stone extraction cases among participants. Although this could have introduced bias, this is the reality where PGTs with various technical aptitudes and learning skills are trained on the same simulator. Fourth, all evaluations were performed by one assessor. However, the URS-GRS is a validated assessment tool. Finally, the evaluations were not blinded; therefore, the halo effect could have introduced bias. Nonetheless, this is the first prospective study to assess the learning curve of flexible ureteroscopic stone extraction on the UroMentor simulator and to assess the predictive validity of this simulator in assessing flexible ureteroscopic stone extraction skills in the operating theatre. Future studies with larger sample size are needed to confirm these results.

Conclusions

Competency in task 10 on the UroMentor simulator (flexible ureteroscopic stone extraction) was achieved after seven trials. Since there was a strong positive correlation between URS-GRS scores on the simulator and in the operating theatre, the skills obtained on the simulator could be transferred to the operating theatre. Therefore, predictive validity of performance on this simulator was demonstrated. Future studies with larger sample size are needed to confirm these results.

Footnotes

Acknowledgments

This work was partially sponsored by Fonds de la Recherche Santé du Québec (FRSQ) grants to Dr. Sero Andonian and Dr. Mehdi Aloosh and by a grant from the Urology Care Foundation Research Scholars Program and the Boston Scientific Corporation, The Endourological Society, the “Friends of Joe,” and the Canadian Urological Association-Société Internationale d'Urologie (CUA-SIU) Scholarship to Dr. Yasser Noureldin.

Author Disclosure Statement

No financial interests with Simbionix. The UroMentor™ simulator was purchased by funds from the Montreal General Hospital Foundation.

Abbreviations Used

References

Scales

, Tasian

, Schwaderer

, et al. Urinary stone disease: Advancing knowledge, patient care, and population health. Clin J Am Soc Nephrol, 2016; 11:1305–1312.

Assimos

, Krambeck

, Miller

, Monga

, Murad

, Nelson

, et al. Surgical Management of Stones: American Urological Association/Endourological Society Guideline, PART I. J Urol, 2016. DOI: 10.1016/j.juro.2016.05.091.

Sofer

, Watterson

, Wollin

, et al. Holmium: YAG laser lithotripsy for upper urinary tract calculi in 598 patients. J Urol, 2002; 167:31–34.

Gettman

, Segura

. Indications and Outcomes of Ureteroscopy for Urinary Stones. Urinary Stone Disease. Totowa, New Jersey: Humana Press, 2007, pp. 571–588.

Knoll

, Trojan

, Häcker

, et al. Validation of computer based training in ureterorenoscopy. BJU Int, 2005; 95:1276–1279.

Botoca

, Bucuras

, Boiborean

, et al. The learning curve in ureteroscopy for the treatment of ureteric stones. How many procedures are needed to achieve satisfactory skills?. Eur Urol, 2004; 3:138.

Accreditation Council for Graduate Medical Education (ACGME). Case Log Information for Urology Programs Review Committee for Urology [Internet]. 2013 [cited May 1, 2016]. Available at: www.acgme.org/portals/0/pfassets/programresources/480-urology-case-log-info_.pdf Accessed September 6, 2016 .

Skolarikos

, Gravas

, Laguna

, et al. Training in ureteroscopy: A critical appraisal of the literature. BJU Int, 2011; 108:798–805.

Noureldin

, Aloosh

, Andonian

. How to use virtual-reality simulators to assess competency in basic endourologic and robotic skills?. J Endourol, 2016 [Epub ahead of print]; DOI: 10.1089/vid.2016.0014.

10.

Noureldin

, Elkoushy

, Aloosh

, et al. Objective Structured Assessment of Technical Skills for the Photoselective Vaporization of the Prostate Procedure (PVP-OSATS): A pilot study. J Endourol, 2016; 30:923–929.

11.

Noureldin

, Elkoushy

, Andonian

. Assessment of percutaneous renal access skills during Urology Objective Structured Clinical Examinations (OSCE). Can Urol Assoc J, 2015; 9:E104–E108.

12.

Noureldin

, Elkoushy

, Fahmy

, et al. Assessment of photoselective vaporization of prostate skills during Urology Objective Structured Clinical Examinations (OSCE). Can Urol Assoc J, 2015; 9:e61–e66.

13.

Noureldin

, Fahmy

, Anidjar

, Andonian

. Is there a place for virtual reality simulators in assessment of competency in percutaneous renal access?. World J Urol, 2016; 34:733–739.

14.

Noureldin

, Stoica

, Kassouf

, et al. Incorporation of the da Vinci Surgical Skills Simulator at urology Objective Structured Clinical Examinations (OSCEs): A pilot study. Can J Urol, 2016; 23:8160–8166.

15.

Matsumoto

, Hamstra

, Radomski

, Cusimano

. A novel approach to endourological training: Training at the Surgical Skills Center. J Urol, 2001; 166:1261–1266.

16.

Bahnson

. Residency training: Where do we go from here. Urol Clin North Am, 2009; 36:73–77.

17.

Watterson

, Denstedt

. Ureteroscopy and cystoscopy simulation in urology. J Endourol, 2007; 21:263–269.

18.

Gallagher

, Ritter

, Champion

, et al. Virtual reality simulation for the operating room: Proficiency-based training as a paradigm shift in surgical skills training. Ann Surg, 2005; 241:364–372.

19.

Schout

, Hendrikx

, Scherpbier

, Bemelmans

. Update on training models in endourology: A qualitative systematic review of the literature between January 1980 and April 2008. Eur Urol, 2008; 54:1247–1261.

20.

Matsumoto

, Pace

, Honey

, John

. Virtual reality ureteroscopy simulator as a valid tool for assessing endourological skills. Int J Urol, 2006; 13:896–901.

21.

Wilhelm

, Ogan

, Roehrborn

, et al. Assessment of basic endoscopic performance using a virtual reality simulator. J Am Coll Surg, 2002; 195:675–681.

22.

Allen

, O'Brien

, Tiptaft

, Glass

. Defining the learning curve for percutaneous nephrolithotomy. J Endourol, 2005; 19:279–282.

23.

Tanriverdi

, Boylu

, Kendirci

, et al. The learning curve in the training of percutaneous nephrolithotomy. Eur Urol, 2007; 52:206–212.

24.

Ogan

, Jacomides

, Shulman

, et al. Virtual ureteroscopy predicts ureteroscopic proficiency of medical students on a cadaver. J Urol, 2004; 172:667–671.

25.

Schout

, Ananias

, Bemelmans

, et al. Transfer of cystourethroscopy skills from a virtual reality simulator to the operating room: A randomized controlled trial. BJU Int, 2010; 106:226–231.

26.

Williams

Jr. , Satiani

, Thomas

, Ellison

. The impending shortage and the estimated cost of training the future surgical workforce. Ann Surg, 2009; 250:590–597.

27.

Chou

, Abdelshehid

, Clayman

, McDougall

. Comparison of results of virtual-reality simulator and training model for basic ureteroscopy training. J Endourol, 2006; 20:266–271.