Validation,Correlation,and Comparison of the da Vinci Trainer ™ and the da Vinci Surgical Skills Simulator ™ Using the Mimic ™ Software for Urologic Robotic Surgical Education

Abstract

Purpose:

Virtual reality simulators with self-assessment software may assist novice robotic surgeons to augment direct proctoring in robotic surgical skill acquisition. We compare and correlate the da Vinci Trainer™ (dVT) and da Vinci Surgical Skills Simulators (dVSSS) in subjects with varying robotic experience.

Materials and Methods:

Students, urology residents, fellows, and practicing urologists with varying robotic experience were enrolled after local institutional review board approval. Three virtual reality tasks were preformed in sequential order (pegboard 1, pegboard 2, and tubes)—initially on the dVSSS and then on the dVT. The Mimic™ software used on both systems provides raw values and percent scores that were used in statistical evaluation. Statistical analysis was performed with the two-tailed independent t-test, analysis of variance, Tukey, and the Pearson rank correlation coefficient where appropriate.

Results:

Thirty-two participants were recruited for this study and separated into five groups based on robotic surgery experience. In regards to construct validity, both simulators were able to differentiate differences among the five robotic surgery experience groups in the tubes suturing task (p≤0.00). Sixty-seven percent (4/6) robotic experts thought that surgical simulation should be implemented in residency training. The overall cohort considered both platforms easy to learn and use.

Conclusions:

Although performance scores were less in the dVT compared with the dVSSS, both simulators demonstrate good content and construct validity. The simulators appear to be equivalent for assessing surgeon proficiency and either can be used for robotic skills training with self-assessment feedback.

Introduction

W hile advances in technology have allowed surgery to move into more laparoscopically based approaches, the educational process to incorporate these skills into clinical practice has been challenging. With emphasis on improved surgical outcomes and heightened awareness of patient safety, development of these unique surgical skills is moving away from the operative suite to allow completion of the learning curve in a simulated, risk-free environment away from the patient.^1

–4 To simulate surgical skills needed during robotic surgery, virtual reality programs provide an opportunity for repetitive practice to predetermined proficiency levels with immediate feedback for the learner.

Virtual reality surgical simulation has been shown to improve basic skills, decrease operative times, and reduce medical errors.^5,6 As such, surgical simulation has been encouraged in urologic residency robotic training and may be incorporated into preclinical phases of future robotic surgical training curricula.^7,8 As the da Vinci Surgical System^™ (Intuitive Surgical, Inc., Sunnyvale, CA) is the only commercially available clinical robotic platform with current widespread use, this system has been the focus of developing a training platform for basic skills testing.

Herein we focus on two previously validated virtual reality robotic surgery simulators: the da Vinci Trainer™ (dVT) (Mimic Technologies, Inc., Seattle, WA) and da Vinci Surgical Skills Simulator™ (dVSSS) (Intuitive Surgical, Inc.) (Fig. 1). Both use the same virtual training software, which makes their comparison appropriate. However, the dVT is a smaller, tabletop sized, stand-alone simulator using foot pedals that replicate the actual da Vinci Robot. In contrast the dVSSS integrates the Mimic virtual reality tasks using the actual da Vinci Si surgeon console as the user interface. These two robotic trainers have been tested by ourselves and other researchers for face, content, and construct validity.^9

–13 These studies have shown that the skill tasks on these teaching platforms can differentiate the novice from the expert surgeon based on performance scores, and can help train learners with limited or no robotic experience to acquire and improve their robotic skills. However, these two systems have not previously been compared with the same test subjects using both systems. The purpose of this study was to evaluate the basic skill task assessment program of the dVT and dVSSS using the same study subjects with varying degrees of robotic experience to provide comparative validity testing of both systems.

FIG. 1.

(A) dV Trainer™, (B) dV Trainer foot pedals, (C) dV Surgical Skills Simulator™, and (D) dV Surgical Skills Simulator foot pedals. dV, da Vinci.

Materials and Methods

After obtaining institutional review board approval, subjects were recruited for the study. Subjects were categorized by robotic surgery experience as student (0 hours of robotic surgery), junior resident (R1–R4,<5 hours of console time), senior resident and fellow (R4–5 or F1–2 with 5–50 hours of robotic console time), or attending surgeons. Attending surgeons were also categorized by robotic novice (<5 hours of robotic console time) or by fellowship trained surgeons with a current use of robotics in their practice (>50 hours of robotic console time).

Each subject was instructed to complete three tasks on the dVSSS, which included Pegboard Transfer 1, Pegboard Transfer 2, and Tubes Suturing, in order respectively (Fig. 2). The pegboard 1 and pegboard 2 tasks were chosen successively in order to provide a warm-up and allow the subjects to familiarize themselves with the specifics of the training device in regards to pedals, clutching, master controls, and working space. All subjects then were tested on the tubes suturing task because this task simulates the technique of the urethrovesical anastomosis (UVA) suturing during the robotic radical prostatectomy. The two less-challenging pegboard tasks were used as practice warm-up tasks prior to attempting the more technically difficult UVA skill task; there were no other practice trials.

FIG. 2.

Three Mimic™ software virtual reality tasks completed on both the dV Trainer and dV Surgical Skills Simulator, which include (A) Pegboard 1, (B) Pegboard 2, and (C) Tubes suturing.

Using the built-in scoring algorithm provided by the Mimic™ software, each subject was assessed on overall score, economy of motion, time to completion, excessive instrument force, instrument collisions, instruments out of view, master workspace range, and drops. These domains have previously been validated in other studies.^9,10,12 Each score had a raw value and associated percent score. After completing the dVSSS robotic tasks, each subject was then asked to perform the same sequence of tasks on the dVT (Fig. 2).

To provide face and content validity comparisons of the two simulators, all subjects answered a questionnaire regarding realism, usefulness as a training tool, functionality as a trainer for both the dVT and dVSSS, and which simulator they preferred using. Questions were answered on a 5-point Likert scale. Information was also collected on the study subject's age, handedness, gender, and robotic experience.

Statistical analysis was performed with the two-tailed independent t-test, analysis of variance (ANOVA), Tukey correction, and the Pearson rank correlation coefficient using SPSS^®, version 20. Ninety-five percent confidence intervals are given when appropriate and p-values<0.05 were considered significant.

Results

Of the 32 participants recruited for this study, 6 were students with no surgical or robotic experience, 7 were attending urologic surgeons without prior robotic experience, 7 were junior urology residents with ≤5 hours of robotic console time, 6 were senior urology residents or fellows with a range of >5–50 hours of robotic console time, and 6 were fellowship-trained urologic robotic surgeons with >50 hours of robotic console time experience (Table 1).

Table 1.

Study Subject Demographics

Characteristics	N	(%)
Gender
Male	26	81
Female	6	19
Handedness
Right	24	75
Left	2	6
Ambidextrous	3	9
Did not respond	3	9
Age
<35	16	50
35–45	7	22
45–55	4	12.5
55–65	4	12.5
>65	1	3
Surgical training
None	6	18.8
Junior resident (R1–R3)	7	21.9
Senior resident (R4–R5) or fellow	6	18.8
Attending surgeon (robotic novice)	6	18.8
Expert robotic urologic surgeon	7	21.9
Robotic experience
Novice	13	40.6
Intermediate	9	28.1
Expert	10	31.3

Construct validity

Both the dVT and dVSSS simulators were able to differentiate all five groups on overall score using the tubes suturing task (ANOVA: p=0.001 and p<0.001, respectively) (Fig. 3). The dVT also showed significant differences among the five surgeon groups in the areas of critical errors, economy of motion, and missed targets (all p values<0.02). The dVSSS showed significant differences among all five groups in areas of time to complete task, economy of motion, and instrument collisions (all p-values<0.02). To examine the relationships between specific groups, the Tukey method was used to evaluate the overall tubes suturing score. We noted that the senior resident/fellow group and expert surgeon group distinctly had improved scores compared with the student population for both the dVT (p=0.001 and p=0.027, respectively) and the dVSSS (p=0.004 and p=0.001). In addition, the dVSSS demonstrated improved differentiation of the experienced robotic surgeons compared with the students, novice robotic attending surgeons, and junior residents (p=0.001, p=0.007, and p<0.001, respectively).

FIG. 3.

(A) dVSSS Overall Tubes and (B) dVT Overall Tubes percent score on the tubes suturing task grouped by overall surgical experience. dVSSS, da Vinci Surgical Skills Simulator; dVT, da Vinci Trainer.

Correlation

After examining all factors in the Mimic software analysis, the overall score was the most significant for differentiating between surgeon experience and robotic skill task performance on both simulators and therefore was used for the correlation factor. We also used the paired t-test in order to test for a learning effect from performing the skill tasks on the dVSSS first and then the same tasks on the dVT. There were significant differences in each category but all study subjects demonstrated poorer performance on the dVT (paired t-test, all p-values<0.001). Comparing the pegboard 1 and pegboard 2 task overall scores, the dVT and dVSSS were highly congruent with Pearson correlation values of r=0.44 (p=0.001) and r=0.55 (p=0.001). The Pearson correlation demonstrated that the most technically challenging and clinically applicable tubes suturing task was in good agreement between the dVT and dVSSS, with an R-value of 0.52 (p=0.002) (Fig. 4).

FIG. 4.

Pearson correlation of the overall score on the tubes suturing task demonstrating the agreement between the dV Trainer and the dV Surgical Skills Simulator.

Face and content validity

Overall, 67% (4/6) of robotic experts thought that surgical simulation should be implemented in residency training compared with 80% (20/25) of the remaining study subjects. Only 50% of experts and 44% of novice/intermediate experience subjects thought simulators should be part of qualification testing for robotic surgery (Table 2). The study subjects uniformly considered both simulators to be easy to learn and had good realism. However, in general the subjects felt that the dVSSS was more realistic, easier to learn, and had better features, such as foot controls, haptic feedback, 3D visualization, and movement of masters, compared with the dVT (all p-values<0.001).

Table 2.

Face and Content Validity Comparison of the da Vinci Trainer and da Vinci Surgical Skills Simulator

	N	dVT Mean (SD)	dVSSS mean (SD)	p Value
User realism (face validity)
Likert scale: 1=disagree to 5=agree
Did the simulator look realistic?	31	3.5 (1.1)	4.6 (0.62)	<0.001
Training tool (content validity)
Likert scale: 1=disagree to 5=agree
It was easy to learn.	31	4 (1.0)	4.74 (0.57)	<0.001
It would be useful to use this simulator for residency training	31	3.81 (1.22)	4.65 (0.88)	<0.001
Reporting score: Yes (1), No (2), Don't know (3)
Do you think simulators would be useful in training people to use a da Vinci robot?	32	1.3 (0.69)		Yes (81%)
Do you think simulators should be implemented in residency training?	32	1.37(0.75)		Yes (77%)
Do you think simulators should be used for qualification testing in robotic surgery?	32	1.84 (0.88)		Yes (45%)
Likert scale: 1=poor to 5=excellent
Rate the simulator on the following features:
Foot controls	31	3.54 (1.23)	4.69 (0.54)	<0.001
Haptic feedback	31	2.93 (1.12)	4.03 (1.08)	<0.001
3D view	31	3.61 (1.05)	4.69 (0.47)	<0.001
Movement of masters	31	2.61 (1.17)	4.77 (0.43)	<0.001

Dvt, da Vinci Trainer; dVSSS, da Vinci Surgical Skills Simulator; SD, standard deviation.

Discussion

With the advent of endoscopy, laparoscopy, and robotic surgery in current practice, urologic training is fraught with immense challenges. The surgeon must not only learn the complex surgical techniques but also keep abreast of the continually changing platforms of the surgical equipment or device. Although animal models have greater face validity, they are limited by cost, ethics, and logistics, and have largely been limited to universities to support such endeavors.¹⁴ Virtual reality simulation and computer-based training assist the surgeon in successfully achieving the learning curve of the newer surgical platforms, especially in the case of the da Vinci robotic surgical system (Intuitive Surgical, Inc.).^3,15 In a survey involving 41 urology program directors, 97% indicated that there was a need for training outside the operating room and 100% felt that simulators are useful for residency training by providing a risk-free environment for repetitive practice to a predetermined proficiency level.¹⁶

The dVT and the dVSSS have both demonstrated face and content validity with various researchers and have been noted to have distinct advantages and disadvantages.¹⁷ However, comparison of these two robotic simulators has not been previously reported.¹⁸ Our study has confirmed the face and content validity of the dVT and the dVSSS. By utilizing the high-level skill task that closely mimics the urologically applicable and complex technique of suturing tubes together, we have demonstrated that both simulators have similar construct validity. The score that both the dVT and the dVSSS provide on the performance of this particular skill task can reliably differentiate between surgeons of different robotic surgical experience (Fig. 4).^{9

–13,18

–21} Our data show that novice subjects, with no specific robotic surgery experience, performed with a significantly lower score on the suturing tubes skill task compared with surgeons of intermediate robotic surgery experience, and these intermediate robotic experienced surgeons perform significantly less well than expert robotic surgeons on this same task. Thus, this skill task can be used to assess a surgeon's proficiency with the robotic platform (Fig. 3).

Learner accessibility to the robotic simulator will be paramount in this teaching device providing assistance in successfully completing the learning curve for robotic surgery skill acquisition.^3,6 While the dVSSS did appear to have improved face and content validity compared with the dVT, this concern of availability could be a major issue with this simulator. The dVSSS system is attached and integrated directly to the console of the clinical robot. In one regard, this is ideal for surgeons to incorporate a “warm-up” period performing directed tasks via the simulator prior to starting a robotic surgery. In another regard, the dVSSS can only be used when the actual clinical robot is not in use by a clinician, which is potentially a major limitation of this simulator. In addition, the dVSSS can only be used on the da Vinci Si surgical system so a costly upgrade may be required for those centers currently using the da Vinci S or earlier systems. If a center does have the da Vinci Si robot (dVSi) system, the dVSSS can be purchased for approximately $100,000 (USD), which is similar in price to the dVT.^9,17

Conversely, the dVT is a stand-alone, desktop console attached to a computer running the same Mimic software as that used with the dVSSS. As such, the dVT is unrestricted in its availability—24 hours a day, 7 days a week—thereby allowing easy access to the system without operating room constraints. In addition, the dVT would not accrue the cost of upgrading to the dVSi surgical system.

In comparison, the study subjects tended to score higher on the corresponding skill task with the dVSSS compared with the dVT. It appeared that the dVSSS was an easier platform to use than the dVT. The dVT has a more limited space due to its desktop design—thereby requiring more camera clutching and advanced maneuvers to perform the skill tasks, which tend to reduce the scores especially in the less-experienced robotic surgeons. While this feature may negatively impact the face validity testing of the dVT, it may actually enhance the novice surgeon learning experience and better prepare them for the more limited working space as encountered in the narrow pelvis of the prostatectomy procedure. This is an example of when poor face validity may not be a detrimental characteristic of a training platform.

In addition, we wanted to correlate the findings on both surgical systems to investigate their agreement as a self-evaluating robotic surgical simulation system. We found that the tubes suturing task overall score was highly correlated between both systems (Pearson correlation r=0.519, p=0.002). This affirms that use of either simulator for purposes of self-directed practice is essentially equivalent with high agreement between construct validity. If the dVSSS is not available, use of the dVT could still provide similar practice.

The major limitation of this study is the small sample size limiting a randomization of simulator sequence. However, with >30 subjects, our study is similar to many of the other reported robotic simulation studies in the literature.^{9

–13,17

–20} We felt that maintaining the same sequence of simulator used to be valid, as all participants had the same opportunity to gain “experience” and proficiency on the simulator, given that the same skill tasks were tested in the same order on both simulators. It could be argued that the testing process of this study constituted a learning curve phenomenon only. We specifically designed this study such that all subjects performed the tasks and the two simulators in the same sequence order to reduce the risk of this effect. In addition, specific questions regarding previous exposure to the particular simulators were not performed and could change the results of a particular group. However, the dVSSS was only at our medical campus during the time of this study and therefore it is unlikely that the study subjects had prior experience with this simulator. The dVT may have only been used by chief residents previously, which could have skewed the favorable results on the dVT in this group.

Conclusions

The dVT and dVSSS have similar face and content validity confirming both to be effective platforms on which to practice robotic skills. Both of these simulators reliably distinguish the level of robotic surgery experience the skill task performer has based on their score achieved in the specific skill task performance, thereby establishing construct validity. The dVT and dVSSS are both acceptable platforms for urologic surgeons to learn the skill tasks associated with these challenging surgical techniques. Further evaluation of the robotic simulators is warranted to determine predictive validity of these teaching devices.

Footnotes

Disclosure Statement

No competing financial interests exist.

Abbreviations Used

References

Albani

, Lee

. Virtual reality-assisted robotic surgery simulation. J Endourol, 2007; 21:285–287.

Ahlering

, Skarecky

, Lee

, Clayman

. Successful transfer of open surgical skills to a laparoscopic environment using a robotic interface: Initial experience with laparoscopic radical prostatectomy. J Urol, 2003; 170:1738–1741.

, Yang

, Wang

et al. The learning curve for reducing complications of robotic-assisted laparoscopic radical prostatectomy by a single surgeon. BJU Int, 2011; 108:420–425.

Samadi

, Levinson

, Hakimi

, Shabsigh

, Benson

. From proficiency to expert, when does the learning curve for robotic-assisted prostatectomies plateau? The Columbia University experience. World J Urol, 2007; 25:105–110.

Grantcharov

, Kristiansen

, Bendix

, Bardram

, Rosenberg

, Funch-Jensen

. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg, 2004; 91:146–150.

Seymour

, Gallagher

, Roman

et al. Virtual reality training improves operating room performance: Results of a randomized, double-blinded study. Ann Surg, 2002; 236:458–463; discussion 463–454.

McDougall

. Validation of surgical simulators. J Endourol, 2007; 21:244–247.

Lee

, Mucksavage

, Sundaram

, McDougall

. Best practices for robotic surgery training and credentialing. J Urol, 2011; 185:1191–1197.

Hung

, Patil

, Zehnder

et al. Concurrent and predictive validation of a novel robotic surgery simulator: A prospective, randomized study. J Urol, 2012; 187:630–637.

10.

Hung

, Zehnder

, Patil

et al. Face, content and construct validity of a novel robotic surgery simulator. J Urol, 2011; 186:1019–1024.

11.

Kenney

, Wszolek

, Gould

, Libertino

, Moinzadeh

. Face, content, and construct validity of dV-trainer, a novel virtual reality simulator for robotic surgery. Urology, 2009; 73:1288–1292.

12.

Lee

, Mucksavage

, Kerbl

, Huynh

, Etafy

, McDougall

. Validation study of a virtual reality robotic simulator—role as an assessment tool? J Urol, 2012; 187:998–1002.

13.

Sethi

, Peine

, Mohammadi

, Sundaram

. Validation of a novel virtual reality robotic simulator. J Endourol, 2009; 23:503–508.

14.

Reznick

, MacRae

. Teaching surgical skills—changes in the wind. N Engl J Med, 2006; 355:2664–2669.

15.

Lerner

, Ayalew

, Peine

, Sundaram

. Does training on a virtual reality robotic simulator improve performance on the da Vinci surgical system? J Endourol, 2010; 24:467–472.

16.

, Lightner

, VanderLei

, Segura

, Gettman

. The current role of medical simulation in american urological residency training programs: An assessment by program directors. J Urol, 2007; 177:288–291.

17.

Korets

, Mues

, Graversen

et al. Validating the use of the Mimic dV-trainer for robotic surgery skill acquisition among urology residents. Urology, 2011; 78:1326–1330.

18.

Lallas

, Davis . Members Of The Society Of Urologic Robotic Surgeons JW. Robotic surgery training with commercially available simulation systems in 2011: A current review and practice pattern survey from the society of urologic robotic surgeons. J Endourol, 2012; 26:283–293.

19.

Perrenot

, Perez

, Tran

et al. The virtual reality simulator dV-Trainer((R)) is a valid assessment tool for robotic surgical skills. Surg Endosc, 2012; 26:2587–2593.

20.

Seixas-Mikelus

, Kesavadas

, Srimathveeravalli

, Chandrasekhar

, Wilding

, Guru

. Face validation of a novel robotic surgical simulator. Urology, 2010; 76:357–360.

21.

Lendvay

, Casale

, Sweet

, Peters

. VR robotic surgery: Randomized blinded study of the dV-Trainer robotic simulator. Stud Health Technol Inform, 2008; 132:242–244.