Association of Suturing Technical Skill Assessment Scores Between Virtual Reality Simulation and Live Surgery

Abstract

Introduction:

Robotic surgical performance, in particular suturing, has been linked to postoperative clinical outcomes. Before attempting live surgery, virtual reality (VR) simulators afford opportunities for training surgeons to learn fundamental technical skills. Herein, we evaluate the association of suturing technical skill assessments between VR simulation and live surgery, and functional clinical outcomes.

Materials and Methods:

Twenty surgeons completed a VR suturing exercise on the Mimic™ Flex VR simulator and the anterior vesicourethral anastomosis during robot-assisted radical prostatectomy (RARP). Three independent and blinded graders provided technical skill scores using a validated assessment tool. Correlations between VR and live scores were assessed by Spearman's correlation coefficients (ρ). In addition, 117 historic RARP cases from participating surgeons were extracted, and the association between VR technical skill scores and urinary continence recovery was assessed by a multilevel mixed-effects model.

Results:

A total of 20 (6 training and 14 expert) surgeons participated. Statistically significant correlations for scores provided between VR simulation and live surgery were found for overall and needle driving scores (ρ = 0.555, p = 0.011; ρ = 0.570, p = 0.009, respectively). A subanalysis performed on training surgeons found significant correlations for overall scores between VR simulation and live surgery (ρ = 0.828, p = 0.042). Expert cases with high VR needle driving scores had significantly greater continence recovery rates at 24 months after RARP (98.5% vs 84.9%, p = 0.028).

Conclusions:

Our study found significant correlations in technical scores between VR and live surgery, especially among training surgeons. In addition, we found that VR needle driving scores were associated with continence recovery after RARP. Our data support the association of skill assessments between VR simulation and live surgery and potential implications for clinical outcomes.

Introduction

Surgeon technical skill has been linked to postoperative clinical outcomes.¹ In particular, suturing performance during robot-assisted radical prostatectomy (RARP) has been linked to urinary continence recovery after surgery.^2
–4 Before attempting live surgery, virtual reality (VR) simulators afford opportunities for training surgeons to practice with surgical instruments and learn fundamental technical skills.

Simulators have been developed for use in many different surgical fields, including general surgery,⁵ spine surgery,⁶ neurosurgery,⁷ colorectal surgery,⁸ and urology.⁹ Procedure-specific simulators have also been created. For example, simulated exercises exist for laparoscopic cholecystectomy,¹⁰ appendectomy,⁵ antromastoidectomy,¹¹ carotid artery stenting,¹² and transurethral resection of bladder tumors,¹³ as well as those specific to steps of surgical procedures such as the vesicourethral anastomosis (VUA) during RARP.^14
–16

VR simulators have been found to be more cost-effective and efficient compared to live animal training models.^17,18 Validation assessments show that surgical residents feel they would benefit from practicing on simulators and they would increase their comfort in the operating room.¹⁹ In addition to skill practice, VR simulators afford opportunities for training surgeons to receive formative feedback on their technical skills.²⁰ Furthermore, adjunctive training utilizing a VR simulator has been shown to be effective in reducing surgical operating times.¹⁶

Our previous work has assessed associations of automated performance metrics (APMs) between two surgical settings. That study found many computer-generated measures of surgeon performance could be translated between a VR simulator and laboratory-based surgical robot.²¹ These computer-generated metrics provide valuable insight into hand movements during suturing (measure of efficiency) but are not readily translated into direct actionable feedback.

Given the practical utility and proven benefit of VR simulators, we were interested in studying how suturing technical skills assessed in the VR simulation environment and live surgery correlated with each other. Herein, we evaluate the ability to transfer suturing technical skill assessments between VR simulation and live surgery. We further assess the ability of VR performance to anticipate a functional clinical outcome after RARP.

Materials and Methods

Training and expert urologic surgeons at our institution completed a VR suturing exercise on the Mimic™ Flex VR simulator. A priori, training surgeons were defined as those having robotic surgery caseload ≤100 (e.g., junior urologic residents), and expert surgeons were defined as having a robotic surgery caseload >100 (e.g., senior urologic residents, urologic robotic surgery fellows, urology faculty). A console caseload cutoff of 100 was based on a learning curve meta-analysis and previous study from our laboratory.^22,23

The simulator exercise, “Tubes,” consisted of 2 opposed open-ended tubular segments with 16 alternating targets, 8 on each segment end (Fig. 1). A surgeon progressed through the exercise by throwing stitches at illuminating targets, which could appear from within the lumen of the tubes segment or from the exterior. Surgeons completed one trial attempt followed by the graded study attempt. Participants had no prior experience on this VR simulator.

FIG. 1.

A representative image of the simulator exercise. Color images are available online.

Historical videos of the same surgeons performing the VUA during RARP were also obtained. To ensure consistency between participants, the last eight stitches (four urethral and four bladder) of VUA were included for assessment. The most contemporary video sample with associated clinical data from a surgeon was utilized. For training surgeons, where clinical data were not available, we collected the videos within 2 months before completing the VR exercise.

After collection, live surgery and VR simulation video samples were de-identified and time annotated for each substep of a suturing motion (needle positioning, needle entry angle, needle driving, and needle withdrawal) as previously described.^24,25 Three independent and blinded graders received standardized training and provided technical skill scores guided by the validated assessment tool Robotic Anastomosis Competency Evaluation (RACE).²⁶ Scores were assigned for each substep of the suturing process. Each substep score was then summated to provide an overall score. Discrepant scores were discussed until group consensus was reached. Each skill domain score ranged from 0 to a maximum of 5; three domains were assessed allowing for a maximum overall score of 15. A high RACE skill score was defined as a 4 or 5, whereas a low score was a 3 or below.

RARP cases from 2016 to 2019, which were performed by participating surgeons in this study, were extracted from a prospectively collected database. Follow-up data at 3, 6, 12, and 24 months were obtained by chart review or telephone by an independent research coordinator utilizing patient-reported outcomes, including urinary continence recovery status. Continence was defined as zero pads or one safety pad per day.²⁷

Suturing technical skill scores between training and expert surgeons were compared by Wilcoxon rank sum test. Correlations between VR and live scores were assessed by Spearman's correlation coefficients (ρ). A multilevel mixed-effects model was used to test the association between technical skill and urinary continence recovery, although adjusting for the clustering of surgeon data.

Our study complied with protocols approved by the University of Southern California's Institutional Review Board. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This article does not contain any study with animals performed by any of the authors. Informed consent was obtained from all individuals included in the study.

Results

A total of 6 training (median [IQR] caseload 18 [2–45]) and 14 expert surgeons (400 [150–725]) completed a VR simulation exercise and a live VUA.

In live surgery, expert surgeons averaged greater skill scores in all assessed domains, individually and overall. Statistically significant differences were seen for overall RACE score (median [IQR] 14 [14–15] vs 14 [10–14], p = 0.045) as well as the needle driving domain (5 [4–5] vs 4 [3–4], p = 0.009) (Fig. 2A). In VR simulation, experts again averaged greater individual and overall scores. We found that needle driving scores were significantly higher for experts (4 [4–5] vs 3 [3–4], p = 0.045) (Fig. 2B).

FIG. 2.

(A) Differences between expert (caseload ≥100) and training (caseload <100) surgeons for overall gradings and by skill domain in live surgery. (B) Differences between expert (caseload ≥100) and training (caseload <100) surgeons for overall gradings and by skill domain in VR simulation. *p < 0.05, **p < 0.01, ^T p < 0.10. VR = virtual reality.

Across all experience groups (training and experts), statistically significant correlations between VR simulation and live surgery were found for overall and needle driving scores (ρ = 0.555, p = 0.011; ρ = 0.570, p = 0.009, respectively) (Fig. 3A). A subanalysis performed using only our training surgeon cohort found significant correlations for overall scores between VR and live surgery (ρ = 0.828, p = 0.042) (Fig. 3B). Other RACE skill domains did not reach significance, although needle driving was trending significance (ρ = 0.783, p = 0.066) (Fig. 3B). When studied alone, association between VR and live surgery in the expert surgeon cohort did not reach significance (ρ = 0.367, p = 0.178).

FIG. 3.

(A) Correlation of technical skill assessment scores between VR simulation and live surgery for overall gradings and for the needle driving domain in the entire participating cohort (training and expert surgeons combined). (B) Correlation of technical skill assessment scores between VR simulation and live surgery for overall gradings and for the needle driving domain for the training surgeon cohort (caseload ≤100). Correlations were assessed by Spearman's correlation coefficients (ρ).

One hundred seventeen RARP cases with 2-year patient continence recovery data from 10 expert surgeons were available for analysis of continence recovery. Cases were not available from the other four expert surgeons as they were urologic fellows with limited case load at our institution. Average caseloads per surgeon available for analysis were 12.2 (IQR 3.5–16). Median follow-up time was 517 days (IQR 335–784). At every interval after RARP, we observed that High VR needle driving scores had greater continence recovery rates with a significant difference observed at 24 months compared to low RACE scores (98.5% vs 84.9%, p = 0.028) (Table 1). For this cohort of surgeons, other skill domains in VR and live surgery did not reveal a similar relationship with continence recovery.

Table 1.

Continence Recovery at 3, 6, 12, and 24 Months After Robot-Assisted Radical Prostatectomy

Continence recovery, months	Needle driving low	Needle driving high	p
3	38.5% (15/39)	52.6% (41/78)	0.152
6	66.7% (24/36)	70.7% (53/75)	0.669
12	77.8% (28/36)	81.9% (59/72)	0.607
24	84.9% (28/33)	98.5% (65/66)	0.028

A high skill score was defined as a 4 or 5, whereas a low score was a 3 or below.

Discussion

VR simulators are an increasingly utilized surgical training tool. They provide opportunities for training surgeons to practice with surgical instruments and learn fundamental technical skills before attempting live surgery. With studies showing that VR simulators are more cost-effective and efficient compared to live animal models, along with surgical residents supporting their implementation for surgical training, VR simulation is a technology that will undoubtedly continue to expand in its use.^16

–20 This foundational study further demonstrates the potential benefit VR may offer, showing that the same skill assessments made in VR simulation correlate with those made in live surgery. We also demonstrate a correlation between VR performance and a functional clinical outcome.

Not surprisingly, our study found that expert surgeons outperformed training surgeons in both assessed surgical settings. In live surgery, we saw significant differences for overall scores as well as in the needle driving domain. Other domains had a greater average score for expert surgeons, but the differences were trending or did not reach significance. In the VR setting, we again demonstrated superior performance in our expert surgeon cohort with differences reaching significance in the needle driving domain. Our findings align with previously published results demonstrating experts outperforming training surgeons on VR simulators.²⁴

Recently, greater emphasis has been placed on investigating suturing at the more granular substep level (i.e., needle positioning, needle entry, needle driving).²⁸ This increased level of detail allows for more precision when providing feedback to training surgeons as well as better accuracy when anticipating urinary continence recovery after RARP.^3,24 Our finding that expert surgeons outperformed training surgeons in specifically the needle driving domain aligns with previous studies' findings demonstrating greater expert performance overall, but more specifically, in the needle driving skill.²⁴

In addition, our study found technical scores provided in VR and live surgery were significantly correlated. We saw moderately strong correlations for overall scores as well as in the needle driving domain. We were particularly interested in assessing correlations in our training surgeon cohort as we envision training surgeons utilizing VR simulators during their surgical education, although preparing for live surgery. We demonstrated that training surgeons had a particularly strong association in overall scores provided between the two surgical settings. When assessed, expert surgeons' scores did not significantly correlate between surgical settings. Expert surgeons, with greater live surgical experience, may have a cognitive dissonance between the simulated (less real) and live environments, detracting from their performance in VR. This may explain the lack of correlation between surgical settings for our expert cohort, whereas our training cohort with limited experience in both environments showed a strong overall correlation.

To our knowledge, this is the first article to study the transfer of suturing technical skill assessments between VR simulation and live surgery. We saw significant correlations in the needle driving domain, which has been demonstrated as one of the robust factors that can be used in the prediction of continence recovery after RARP.³ A previous article from our group assessed the transfer of APMs (measures of efficiency) between two surgical settings. That study found many computer-generated measures of surgeon performance could be translated between a VR simulator and laboratory-based surgical robot.²¹ This study compares two similar platforms, but a key difference is our use of manually derived skill assessments.

In comparison to APMs, manually derived assessments can be more informative and a source of directly actionable feedback. Knowing that skill assessments can be transferred between a training and real-life surgical setting can allow for development of training curricula that incorporate VR simulation training and workshops alongside live surgical training in the operating room.

This study also investigated how technical skill assessments made in live surgery and VR correlate with functional clinical outcomes. Prior work has found that suturing performance during RARP, and specifically the VUA, has been linked to urinary continence recovery after surgery.^2,3 Furthermore, suturing studied at the substep level was found to be more predictive of continence recovery.³ Using urinary continence recovery data from our expert surgeon cohort, we found that VR needle driving scores correlated with continence recovery after RARP. As previously discussed, our findings suggest that this substep (needle driving) of suturing is particularly relevant to assess and to train for. VR simulation exercises can focus on specific aspects of the suturing process (i.e., wrist angulation in needle driving) to help surgeons improve their skill. This initial correlative finding in our foundational study warrants further evaluation in a larger multi-institutional cohort.

It should be noted that there are two distinct, but equally important, ways of interpreting our data. The first is that one common assessment tool demonstrated construct validity (i.e., detecting differences in expert and learning surgeon performance) in two different surgical settings for assessment of technical skill. We found that expert surgeons outperformed training surgeons in both live surgery and VR simulation as graded using our clinically validated assessment tool. The second is that a surgeon demonstrated similar skills in both settings. Our data show technical skill for participants, particularly training surgeons, strongly correlates between VR simulation and live surgery. These data support the ability to implement a common assessment tool to provide skill assessments to surgeons, although supporting the ability to use VR simulation as a potential directly translatable modality for early surgery training.

Our foundational study is not without limitations. Our analyses of clinical outcomes were limited by a relatively small training surgeon cohort from which we had no clinical data. The training surgeons included in the study performed live surgical cases within the last year (2021), preventing us from studying long-term continence recovery at this time. In addition, to ensure standardization, our study only assessed part of a VUA, which may limit predictions of urinary continence recovery attributable to a single surgeon. Further research can expand our training and expert surgeon cohorts, include a full VUA with multiple examples from each surgeon, and allow greater time to collect data on clinical outcomes. Our study is also limited by not finding a correlation between live surgery RACE scores and urinary continence recovery.

A potential explanation is that we did not have enough variation when using RACE scores from only our expert surgeons (average was 14/15 [IQR 14–15]). With similar scores, we were not able to effectively correlate RACE scores with continence recovery. We predict that if we had greater variance in RACE scoring (i.e., training surgeons included), we could have achieved better accuracy in predicting continence recovery. Finally, it should be noted that our results demonstrate associations and not causative effects. We do not necessarily believe that suturing skills improve urinary continence outcomes. More likely, suturing skills are a good global assessment of surgeon performance, and urinary continence recovery is, in part, determined by surgeon performance. Our findings in this introductory study warrant a large-scale, multi-institutional study to further evaluate the associations demonstrated in this article.

Although these limitations exist, to our knowledge, this study is the first to correlate manually derived ratings of technical skill between VR simulation and live surgery. Future work can expand this study to further evaluate whether using VR can actually improve surgeon performance in real surgery and ultimately improve patient outcomes.

Conclusions

Our data support the transferability of skill assessments between VR simulation and live surgery and potential implications for clinical outcomes. We also show the skill of a surgeon, specifically training surgeons, strongly correlates between two different surgical settings. Our work strengthens the value of VR simulation as a tool for preparing training surgeons for live surgery. We support the further development and implementation of training curricula utilizing VR simulators to shorten the learning curve of training surgeons.

Footnotes

Authors' Contributions

D.I.S. was in charge of project development, data collection and preparation, and article writing and editing.

R.M. performed data collection, data analysis, and article editing.

A.G. performed data collection and article editing.

T.F.H. performed data collection and article editing.

J.H.N. did article editing and project management.

A.J.H. was in charge of project development, data management, and article writing and editing.

Acknowledgments

An abstract from this article was presented at the 2021 American Urologic Association meeting, which was held virtually. The abstract can be accessed using the following link: doi.org/10.1097/JU.0000000000002092.07.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

Research reported in this publication was supported, in part, by the National Cancer Institute under award number R01CA251579-01A.

Abbreviations Used

References

Birkmeyer

, Finks

, O'Reilly

, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med, 2013; 369(15):1434–1442; doi: 10.1056/NEJMsa1300625.

Hung

, Chen

, Ghodoussipour

, et al. A deep-learning model using automated performance metrics and clinical features to predict urinary continence recovery after robot-assisted radical prostatectomy. BJU Int, 2019; 124(3):487–495; doi: 10.1111/bju.14735.

Trinh

, Mingo

, Vanstrum

, et al. Survival analysis using surgeon skill metrics and patient factors to predict urinary continence recovery after robot-assisted radical prostatectomy. Eur Urol Focus, 2022; 8(2):623–630; doi: 10.1016/j.euf.2021.04.001.

Goldenberg

, Goldenberg

, Grantcharov

. Surgeon performance predicts early continence after robot-assisted radical prostatectomy. J Endourol, 2017; 31(9):858–863; doi: 10.1089/end.2017.0284.

Sinitsky

, Fernando

, Potts

, et al. Development of a structured virtual reality curriculum for laparoscopic appendicectomy. Am J Surg, 2020; 219(4):613–621; doi: 10.1016/j.amjsurg.2019.04.020.

Sumdani

, Aguilar-Salinas

, Avila

, et al. Utility of augmented reality and virtual reality in spine surgery: A systematic review of the literature. World Neurosurg, 2022; 161:e8–e172021; doi: 10.1016/j.wneu.2021.08.002.

Clark

, Barone

, Candy

, et al. The effect of 3-dimensional simulation on neurosurgical skill acquisition and surgical performance: A review of the literature. J Surg Educ, 2017; 74(5):828–836; doi: 10.1016/j.jsurg.2017.02.007.

Sankaranarayanan

, Parker

, De

, et al. Simulation for colorectal surgery. J Laparoendosc Adv Surg Tech A, 2021; 31(5):566–569; doi: 10.1089/lap.2021.0096.

Aydin

, Raison

, Khan

, et al. Simulation-based training and assessment in urological surgery. Nat Rev Urol, 2016; 13(9):503–519; doi: 10.1038/nrurol.2016.147.

10.

Ikonen

, Antikainen

, Silvennoinen

, et al. Virtual reality simulator training of laparoscopic cholecystectomies—A systematic review. Scand J Surg, 2012; 101(1):5–12; doi: 10.1177/145749691210100102.

11.

Gawecki

, Wegrzyniak

, Mickiewicz

, et al. The impact of virtual reality training on the quality of real antromastoidectomy performance. J Clin Med, 2020; 9(10):3197; doi: 10.3390/jcm9103197.

12.

Van Herzeele

, Aggarwal

, Neequaye

, et al. Experienced endovascular interventionalists objectively improve their skills by attending carotid artery stent training courses. Eur J Vasc Endovasc Surg, 2008; 35(5):541–550; doi: 10.1016/j.ejvs.2007.12.017.

13.

Neumann

, Mayer

, Russo

, et al. Transurethral resection of bladder tumors: Next-generation virtual reality training for surgeons. Eur Urol Focus, 2019; 5(5):906–911; doi: 10.1016/j.euf.2018.04.011.

14.

Shim

, Noh

, Kim

, et al. Predictive validation of a robotic virtual reality simulator: The tube 3 module for practicing vesicourethral anastomosis in robot-assisted radical prostatectomy. Urology, 2018; 122:32–36; doi: 10.1016/j.urology.2018.08.013.

15.

Kang

, Cho

, Kang

, et al. The Tube 3 module designed for practicing vesicourethral anastomosis in a virtual reality robotic simulator: Determination of face, content, and construct validity. Urology, 2014; 84(2):345–350; doi: 10.1016/j.urology.2014.05.005.

16.

Pinheiro

EFM

, Barreira

, Moura Junior

, et al. Simulated training of a laparoscopic vesicourethral anastomosis. Acta Cir Bras, 2018; 33(8):713–722; doi: 10.1590/s0102-865020180080000007.

17.

Berry

, Hellstrom

, Gothlin

, et al. Endovascular training with animals versus virtual reality systems: An economic analysis. J Vasc Interv Radiol, 2008; 19(Pt 1):233–238; doi: 10.1016/j.jvir.2007.09.004.

18.

Mishra

, Kurien

, Ganpule

, et al. Percutaneous renal access training: Content validation comparison between a live porcine and a virtual reality (VR) simulation model. BJU Int, 2010; 106(11):1753–1756; doi: 10.1111/j.1464-410X.2010.09753.x.

19.

Keith

, Hansen

, Johannessen

. Perceived value of a skills laboratory with virtual reality simulator training in arthroscopy: A survey of orthopedic surgery residents. J Am Osteopath Assoc, 2018; 118(10):667–672; doi: 10.7556/jaoa.2018.146.

20.

Tronchot

, Berthelemy

, Thomazeau

, et al. Validation of virtual reality arthroscopy simulator relevance in characterising experienced surgeons. Orthop Traumatol Surg Res, 2021; 107(8):103079; doi: 10.1016/j.otsr.2021.103079.

21.

Cowan

, Chen

, Mingo

, et al. Virtual reality vs dry laboratory models: comparing automated performance metrics and cognitive workload during robotic simulation training. J Endourol, 2021; 35(10):1571–1576; doi: 10.1089/end.2020.1037.

22.

Hung

, Chen

, Jarc

, et al. Development and validation of objective performance metrics for robot-assisted radical prostatectomy: A pilot study. J Urol, 2018; 199(1):296–304; doi: 10.1016/j.juro.2017.07.081.

23.

Abboudi

, Khan

, Guru

, et al. Learning curves for urological procedures: A systematic review. BJU Int, 2014; 114(4):617–629; doi: 10.1111/bju.12315.

24.

Sanford

, Der

, Haque

, et al. Technical skill impacts the success of sequential robotic suturing sub-steps. J Endourol, 2022; 36(2):273–278; doi: 10.1089/end.2021.0417.

25.

Der

, Sanford

, Hakim

, et al. Efficiency and accuracy of robotic surgical performance decayed among urologists during COVID-19 shutdown. J Endourol, 2021; 35(6):888–890; doi: 10.1089/end.2020.0869.

26.

Raza

, Field

, Jay

, et al. Surgical competency for urethrovesical anastomosis during robot-assisted radical prostatectomy: Development and validation of the robotic anastomosis competency evaluation. Urology, 2015; 85(1):27–32; doi: 10.1016/j.urology.2014.09.017.

27.

Ficarra

, Novara

, Rosen

, et al. Systematic review and meta-analysis of studies reporting urinary continence recovery after robot-assisted radical prostatectomy. Eur Urol, 2012; 62(3):405–417; doi: 10.1016/j.eururo.2012.05.045.

28.

Chen

, Liang

, Nguyen

, et al. Machine learning analyses of automated performance metrics during granular sub-stitch phases predict surgeon experience. Surgery, 2021; 169(5):1245–1249; doi: 10.1016/j.surg.2020.09.020.