Impact of Virtual Reality-Simulated Training on Urology Residents' Performance of Transurethral Resection of the Prostate

Abstract

Purpose:

There are virtual reality simulators for practicing the transurethral resection of prostate (TURP) procedure, but only few data on its effect on surgical performance are available. The purpose of this study was to test if practicing the TURP procedure in a virtual reality simulator (PelvicVision) increases the skills and dexterity of urology residents when performing the procedure on patients.

Materials and Methods:

Twenty-four urology residents attended a 5-day course on diagnosis and treatment of benign prostatic enlargement. Each of the residents performed three video-recorded TURP procedures. Between two of the procedures (on day 2 and 3, or 3 and 4) the residents underwent criterion-based practice using a simulator. The TURP procedure was evaluated using objective structured assessment of technical skills. Video-recordings of the procedures were analyzed on a minute-to-minute basis.

Results:

Mean practice time in the simulator was 198 minutes before reaching the criterion level. Comparison of the first and last TURP procedures showed an increase in autonomous operation time and time spent on resection, and a tendency to decreased hemostasis time without increased blood loss. The proportion of residents believed to be able to perform a simple TURP procedure increased from 10% to about 75%. Objective structured assessment of technical skills scores and self-evaluations were significantly improved. The scores increased significantly more with than without simulator practice. The resident's self-evaluations showed increased knowledge about the procedure and the technical equipment used. Patient follow-up showed no increased risks.

Conclusions:

Practice in a simulator-based environment improves the skills and dexterity of urology residents when performing the procedure on patients, without increased risks for the patients.

Introduction

A number of virtual reality (VR) simulators for practicing operating skills have been developed and there is proof that simulator training is an efficient way of introducing, for example, the laparoscopic procedure to surgical residents.¹ The value of these new learning tools has been assessed through the use of different validity tests.^2

–5 Most of the VR simulators have undergone the initial procedures involved in these tests and several studies demonstrate the transfer of skills from the VR practice room to the operating room (OR).⁵ A Cochrane analysis in 2009 shows that practice in a VR-simulated environment is at least as good as practice in a more traditional and accepted “box-trainer” environment and concludes that individuals with such training have shorter operation time and make fewer errors.¹ One problem with validation studies is that surgical dexterity and skills may be difficult to assess in an objective way.^2,6

–9 One method used in many studies is the objective structured assessment of technical skills.^8,10
–12 It comprises the combination of a checklist (steps and safety of the procedure) and a global assessment list (skills and dexterity) and has been used to demonstrate the benefits of simulator practice.^13
–15

Several simulators are used today for training in endourologic procedures.¹⁶ Our study group has previously presented validation data for a simulator for transurethral resection of prostate (TURP) training.¹⁷ We concluded that it is safe to use the simulator for training urology residents. The hypothesis of the present study is that practicing the TURP procedure in a VR-simulated environment will increase the skills, dexterity, and efficiency of urology residents when performing the procedures on patients.

Materials and Methods

The regional research ethics committee at the Linköping University approved the study (M163-06).

Simulator

The VR simulator PelvicVision (MeleritMedical AB) used in the present study has been previously described and validated regarding face, content, and construct validity.^17,18 The simulator hardware consists of a standard PC, a modified resectoscope connected to a haptic robot (Phantom, Senseable), a “pelvic floor,” and a pair of “legs.” The software makes it possible to create different patient cases with respect to anatomy and physiology.

Target group

The target group for the evaluation was urology residents in Sweden with some, but minor, experience with transurethral procedures. To participate in the study the residents must have some experience with cystoscopy, transurethral resection of bladder tumors, and TURP procedures. They were required to have performed parts of and, at most, a complete TURP procedure, but without the skills to be able to do this without a supervisor.

The course

The study was implemented as a 5-day course on treatment of benign enlargement of the prostate. The course included theory, diagnostic methods, the instrumentation used in the TURP procedure, and risk factors. The participants were randomly distributed to one of the two groups (Fig. 1) and the supervisors followed their group during the entire course. Three supervised TURP procedures were performed by each participant on selected patients (prostate volume <50 cc and American Society of Anesthesiologists' physical status classification system I–II) in two hospitals (Linköping University Hospital and Vrinnevi Hospital). Supervised VR-simulated TURP procedures were performed (at day 2 and 3, or 3 and 4) until the participant reached at least the criterion level.

FIG. 1.

Course schedule (G1: group 1; G2: group 2; Sim: simulation practice).

Peroperative evaluation

The participants' peroperative performances were rated by two supervisors, one for each group. The supervisors had previously gone through the validation forms and had practiced the rating procedure together. The validation was done according to the objective structured assessment of technical skills,¹⁰ including a checklist (Table 1) and a global assessment form (Table 2) with a pass/fail question, and the participants themselves completed a self-evaluation form.¹⁸ If the participant failed to perform any of the items on the checklist, the supervisor was told to give the correct instructions immediately. The supervisor was also instructed to give instructions when necessary, or to take over the procedure if there was any risk for the patient or if the procedure was estimated to take >60 minutes. The procedures were also video-recorded and analyzed at a later stage.

Table 1.

The Task-Specific Checklist Used Peroperatively

	Yes	No
Check patient ID
Check operation data (prostate volume, risks, antibiotics, etc.)
Check instruments
Check foot pedals
Check diathermia apparatus
Check focus on video screen
Check white balance
Correction of working position
Safe introduction of resectoscope
Observation of anatomy
Active control of rinse solution
Empties rinse solution regularly
Identification of the capsule
Inverted grip on resectoscope when needed
Control of patient status
Realizes when to end or when to ask supervisor to take over
Empties bladder of resection chips
Use of an appropriate catheter
Use of rinse solution when needed
Informs the patient about the results
Informs the staff about the results and the postoperative plan

Table 2.

The Global Five-Point Rating Scale and Pass/Failure Score Used Peroperatively

	1	3	5
Respect for tissue	Resectoscope or sling frequently forced against tissue. Not careful.	Resectoscope or sling sometimes forced against tissue. Mostly careful.	No unnecessary forces against tissue. Careful.
Time and movements	Many unnecessary movements.	Some unnecessary movements, mostly time effective.	No unnecessary movements, optimal use of time.
Eye–hand coordination	Glances back and forth between monitor and hand/resectoscope most of the time.	Glances back and forth between monitor and hand/resectoscope some of the time.	Almost never glances back and forth between monitor and hand/resectoscope.
Foot pedals	“Searches” for the foot pedals a great deal of the time.	“Searches” for the foot pedals some of the time.	Almost never “searches” for the foot pedals.
Videoscope	Frequent problems with aiming the instrument. Poor identification.	Mostly correct aim and identification.	Always good aim and optimal identification.
Resection	Frequently cuts too large or too small chips.		Frequently cuts good-sized chips.
Strategy	Frequently interrupts procedure and needs supervision.	Ability to plan ahead and relatively good progress.	Obvious planning ahead from start to finish.
Tempo	Progresses at very slow or too fast a tempo.	Progresses at a somewhat low or fast tempo but without obvious risks for damage.	Progresses at a good tempo without risks.
Use of assistance	Does not communicate with the staff.	Sometimes communicates with the staff.	Strategic communication with the staff during the entire procedure.
Stress level	Clearly stressed.	Sometimes stressed.	Calm, methodical.
Supervision	During most of the procedure.	During parts of the procedure.	Almost none.
Communication with supervisor	Does not ask for advice despite obvious need.	Sometimes asks for advice but not for enough advice	No need for advice or asks for advice when necessary.
Knowledge about the procedure	Poor knowledge, need for specific instructions during most of the procedure.	Knowledge about the important aspects of the procedure.	Good knowledge about all aspects of the procedure.
Final result	Not near completion.	Requires some completion by the supervisor.	Completed.
□ Supervisor takes over the procedure because of inappropriate performance by the resident
Do you believe the resident is competent enough to independently perform this procedure on a small-to-moderately enlarged prostate?
	Yes	No

Video analysis

The analyses of the video-recordings were done “blindly” (without knowledge of the course, participant, supervisor, patient identity, or the order of the procedure). Each analysis was done using a minute-by-minute rating according to an assessment form (Table 3, modified from Seymour et al¹⁹). The rating was based on what the main action was during that minute and if this action was successful or not. Serious errors such as suspected perforations of the capsule and damage to the bladder wall, uretheral orifices, sphincter area, or urethra were also noted.

Table 3.

The Assessment Form Used When Analyzing the Video-Recordings

	Minute 1 2 3 4 5 6 7 8 9 10 11 12 13 14 …
Lack of progress
Attending takeover
Damage (ca, s, b, u, co)
Resection
Hemostasis
Emptying
Orientation
Technical issues
Progress

ca = capsule; s = sphincter; b = bladder; u = urethra; co = colliculus.

Criterion-based simulation practice

The participants were instructed during their simulated procedures and data from the simulator procedures were saved for later analysis. Each participant was obliged to practice until reaching the “expert” level on a specific “patient”—the test procedure. The expert level was based on the results from nine surgically experienced urologists performing the test procedure (Table 4).¹⁸

Table 4.

The Minimal Criterion Level Form Used When Assessing the Simulation Practice

	Trial no. 1 2 3 4 5 6 …
Parameter	No. of procedures
Total procedure time	<852 seconds (14.2 minutes)
Resectoscope tip movement	<7.76 m
Inactive time	<64%
Resection volume/minute	>0.94 cc/min
Chip size (mean)	>0.098 cc
Absorption orabsorption/minute<0.54 mL	<0.08 dL
High pressure time	<2.2%
Bleeding	<250 mL
Resected volume	>13.3 cc

Patient follow-up

Patient follow-up was done 6–12 months postoperatively using the International Prostate Symptom Score, the bother question, maximum urinary flow rate, and an incontinence score (Linköping incontinence questionnaire).²⁰ Data concerning reoperations and mortality rate were obtained from patient charts from all healthcare facilities in the region.

Statistics

Statistics used were the paired samples t-test and sign test (for nonparametric data), and the statistical analysis was done using SPSS (version 16.0.2).

Results

During 2006 and 2007, urology residents in Sweden were invited to participate in the course, which was held on three occasions with eight different participants in each.

Group statistics

There were 24 participants from all over Sweden who were divided randomly into each of the two groups. There were no significant differences between the groups regarding sex (women:men 1:3), age (mean 33 years), or prior experience with transurethral procedures (mean 14 months of residency, mean 1.2 TURP procedures performed completely, and mean 6.2 incompletely).

Course evaluation

The participants believed that they had learned most from the real TURP procedures (rated 5.0 of 5) and that patient safety was high (4.8 of 5). The simulated exercises were also rated high. During the course there was a significant improvement in knowledge concerning the instrumentation (3.4–4.7 of 5) and procedure (2.8–4.4 of 5).

Simulation learning curve

Figure 2 shows the results from the test procedures. The mean practice time before reaching the expert level was 198 minutes (not including the last validation procedure). The total mean practice time was 254 minutes distributed over 8.3 procedures, of which 3.7 were test procedures.

FIG. 2.

Mean values from the simulated procedures on the test patient. All participants performed a procedure on the test patient as their first procedure, and thereafter they performed the test procedure whenever they wanted to see if they could meet the criterion levels. Data from the other training procedures are not shown here. At most, 17 procedures were performed by one participant before approval.

Interrater analysis

Two surgically experienced urologists analyzed the video-recordings. They analyzed about 10 procedures together and then independently analyzed the same recordings until reaching an interrater agreement of >90% (agreements/total number of observations). When the calculation was based on how many times each action/error and progression score was noted for each recording, agreement reached 98%. This took into account the fact that the timestamps were sometimes out of sync between the raters by 1 to 2 minutes, depending on the video-player used to show the recordings.

Patient cases

The mean prostate volume (as measured by transrectal ultrasound) was 37.7 cc (19–67 cc) and the mean age was 72.5 years. There were no significant differences in prostate volumes or in the order of operations between the groups. In three procedures, it was decided to perform an incision of the bladder neck because of small prostate volumes. One patient was excluded because of an upper respiratory tract infection.

Comparison of the first and last TURP procedures

There was a significant increase in the amount of autonomous procedure time, resection time, and a tendency to decreased hemostasis time and increased successful orientation (Fig. 3). The resection effectiveness measured as total resection weight and resection weight per minute was significantly lower (p = 0.003 and 0.004). The number of aborted procedures due to poor skills/dexterity/judgment decreased from 30% to 0% (p = 0.016, sign test), and the proportion of residents believed to be able to perform the procedure independent of a supervisor increased from <10% to about 75% (p = 0.000, sign test). There were better scores on the checklist (p = 0.000), global assessment (p = 0.000), and self-evaluation (p = 0.000) (Fig. 4).

FIG. 3.

Results from the analysis of the video-recordings. There was a significant increase in resident procedure time between procedures 1 and 3 (p = 0.000), resection time (p = 0.029), and a tendency to decrease in hemostasis time (p = 0.073) and increase in successful orientation (p = 0.078) between procedures 1 and 3.

FIG. 4.

Mean values of pre- and postoperative data and data from the checklist, global assessment, aborted procedures, judgment of competency, and self-evaluation form.

Comparison of procedures preceded by and not preceded by simulation practice

Analysis of the effect of simulation practice was done by comparing the change in skills for each participant. The change in scores for two operations with simulation practice in between was compared with scores for two operations without added simulation practice. Although it was not possible to measure any significant difference in single parameters (except for resident operation time, which increased, p = 0.025), there was a tendency for improved skills with simulation practice. When the number of participants who improved or showed no change in skills after simulation practice was compared with the results for procedures without simulation practice, there was a significant difference (p = 0.021) indicating that simulation practice resulted in increased skills (Fig. 5). Sixteen participants showed greater improvement after simulation practice compared with seven participants who showed greater improvement without practice. One participant could not be evaluated regarding the effect of simulation practice because of exclusion of one patient.

FIG. 5.

The difference in performance between two procedures with simulator practice in between the procedures compared with two procedures without simulator practice.

Patient follow-up

The 71 patients who underwent surgery had a follow-up time of 2.3 to 3.8 years. The International Prostate Symptom Score, bother score, maximum urinary flow rate, and incontinence were evaluated at 6 to 12 months postoperatively. However, the patient response rate regarding functional outcome was low (mean 46%), which makes these figures difficult to interpret (Table 5).

Table 5.

Follow-Up Data for the 23 Patients Whose Transurethral Resection Of Prostate Procedures Were Performed During the Study

	Preoperative	Reference	Postoperative	Reference
IPSS (0–35)	21.1	18–24	7.6	3–9
Bother score (0–5)	3.7	4–5	1.5	1–2
Qmax (mL/minute)	8.2	6–12	20.6	22–25
Incontinence %	41	4–46	1–13	1–16
ReTURP %			5.6	6–10
Mortality rate %			1.4–4.2	2–13

The data on IPSS, bother score, maximal flow rate, and incontinence were collected at 6 to 12 months postoperatively. There was a follow-up time of 2.3 to 3.8 years for the data on postoperative TURP procedures and mortality rate. The reference data are those of large postoperative studies.

IPSS = International Prostate Symptom Score; TURP = transurethral resection of prostate; Qmax = maximum urinary flow rate; ReTURP =Repeated TURP.

The charts of all included patients were analyzed. The mortality rate was 8 (11%); 3 (4.2%) of these deaths may be associated with the surgery, and the other 5 died because of causes not related to the surgery (accidents, cancer, etc.). Two of the patients had a cerebrovascular insult at 26 and 51 days postoperatively, respectively, and they died at 5 and 12 months later, respectively. The last case was more closely associated in time with the surgery. This patient had known cardiovascular disease, 45 minutes of operation time, 13 g of resected tissue, and 140 mL of bleeding. In the postoperative period, the patient suffered from hematuria and angina pectoris. After 9 days, the patient was discharged to his home, and on the following day he had a myocardial infarction and was hospitalized for 8 days. He was found dead in his home at 78 days postoperatively. This gives a surgery-associated mortality rate of 1.4% to 4.2%. Two cases had surgery in the immediate postoperative period (misplaced suprapubic catheter, gross hematuria). Three patients were readmitted because of hematuria. Six patients underwent additional surgery (one meatus-stenosis, one urethral stricture, four TURPs). Nine patients (12.7%) suffered from incontinence at 6 months postoperatively, of whom eight had low-grade incontinence (score 1 of 5). One patient with complete incontinence was preoperatively diagnosed as having an overactive bladder. When analyzing the video-recording it was not possible to identify any resection or damage to the area distal to the colliculus (sphincter area).

Discussion

This study shows that the surgical skills of urology residents increase more with than without preoperative simulator practice. There was an improvement in skills during the course, and when the first operation was compared with the last, the majority of participants went from being unable to independently perform a simple TURP procedure to acquiring this ability. This was also reflected in a more skilled performance as assessed from the video-recordings. The trainees tended to be more effective regarding successful actions and they spent less time attaining hemostasis, without any increase in blood loss or absorption of rinse solution. Resection effectiveness was lower regarding total resected weight and resection speed. This can be explained by the fact that the supervisor performed most of the resection in the first operation, and in the last operation, many of the participants performed the entire procedure independently.

The TURP procedure itself generates few serious errors, and the supervisors were also very aware and observant regarding possible errors and interrupted the participant or took over the procedure before the error was made. There were no significant differences in serious errors, which were very few in all, making the comparison difficult. Nevertheless, there was a tendency for a successive reduction in errors during the course week.

It was not possible to measure more than a few significant differences when analyzing single parameters. This may be explained by the small number of participants, the short practice time in the simulator environment, and/or too low level of difficulty of the criterion levels for the test procedure. There is greater complexity when the trainee performs the entire procedure compared with when the supervisor takes over the procedure because of poor progress or poor dexterity. When the supervisor takes over there is an accumulation of difficulties to take care of, and this is not reflected in the participant's score.

The scores for dexterity (video analysis), judgment (global assessment), safety-focused thinking (checklist), and confidence (self-evaluation) were combined to get an overall judgment concerning how well the entire patient procedure was performed. In Figure 5, the data favor VR practice except for total procedure time, participant operation time, global assessment, and the pass/fail question. This might be explained by the fact that in the OR these data were very dependent on the supervisor. The procedures performed without VR practice in between were done on 2 consecutive days, and both the supervisor and the participant might have felt more comfortable with the situation and with knowledge of the participant's skills fresh in mind.

The study design tried to compensate for bias by means of a strict validation structure and circumstances that were as homogenous as possible. The participants had the same supervisor/evaluator during the entire course, which made the absolute value of the rating less important. The differences in skill were measured between the procedures and compared for each participant. The instrumentation, staff, and OR were the same during the week and the only factor that differed substantially was the patients.

The procedure outcomes and patient safety were good compared with large studies (Table 5).^21,22 Different studies report a large variation regarding functional outcome after TURP.^20,23,24 The present data seem to be well within the reported range with the exception of peak urine flow, which is poorer than what is reported in other studies. This might be explained by the low resection volume (<50% of the preoperative volume), or by the fact that there was little follow-up data (12%) for this figure. The incontinence rate varies greatly between different studies (0%–16%), which is probably due to different definitions of incontinence. In the present study, there was an incontinence rate of 1.2% to 12.7% depending on how incontinence is defined. The follow-up shows no increased risks or poorer results for the patients compared with other studies.

The study shows that criterion-based training of the TURP procedure in a VR-simulated environment significantly improves operative performance and raises the level of skill and dexterity of inexperienced urology residents more than without training. A course for learning the TURP procedure that includes simulator training can significantly improve the participant's performance with the same patient outcome as when done in the usual clinical environment. Simulator training in the early learning phase can, to some extent, replace live operations. To get the most out of a surgical simulator as a training tool, it should comprise part of a larger curriculum that includes theoretical education concerning diagnostic tools, patient selection, and treatment options.

Footnotes

Acknowledgments

The authors thank Kalle Wahlin, lecturer, Department of Computer and Information Science, Linköping University, for statistical support. The study was supported by the City Council in Östergötland.

Disclosure Statement

J. Svanvik has later become a member of the advisory board of Melerit AB. Melerit AB is the manufacturer of the simulation device used in the present study.

Abbreviations Used

References

Gurusamy

, Aggarwal

, Palanivelu

, Davidson

. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev, 2009; 1:CD006575.

Satava

, Gallagher

, Pellegrini

. Surgical competence and surgical proficiency: Definitions, taxonomy, and metrics. J Am Coll Surg, 2003; 196:933–937.

Carter

, Schijven

, Aggarwal

et al. Consensus guidelines for validation of virtual reality surgical simulators. Surg Endosc, 2005; 19:1523–1532.

McDougall

. Validation of surgical simulators. J Endourol, 2007; 21:244–247.

Schout

, Hendrikx

, Scheele

, Bemelmans

, Scherpbier

. Validation and implementation of surgical simulators: A critical review of present, past, and future. Surg Endosc, 2010; 24:536–546.

Reznick

. Teaching and testing technical skills. Am J Surg, 1993; 165:358–361.

Goff

, Mandel

, Lentz

et al.

Assessment of resident surgical skills: Is testing feasible?

Am J Obstet Gynecol, 2005; 192:1331–1338discussion 1338–1340.

Aggarwal

, Grantcharov

, Moorthy

, Milland

, Darzi

. Toward feasible, valid, and reliable video-based assessments of technical surgical skills in the operating room. Ann Surg, 2008; 247:372–379.

Tavakol

, Mohagheghi

, Dennick

. Assessing the skills of surgical residents using simulation. J Surg Educ, 2008; 65:77–83.

10.

Martin

, Regehr

, Reznick

et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg, 1997; 84:273–278.

11.

Vassiliou

, Feldman

, Andrew

et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg, 2005; 190:107–113.

12.

Larsen

, Grantcharov

, Schouenborg

, Ottosen

, Soerensen

, Ottesen

. Objective assessment of surgical competence in gynaecological laparoscopy: Development and validation of a procedure-specific rating scale. BJOG, 2008; 115:908–916.

13.

Lucas

, Tuncel

, Bensalah

et al. Virtual reality training improves simulated laparoscopic surgery performance in laparoscopy naive medical students. J Endourol, 2008; 22:1047–1051.

14.

Larsen

, Soerensen

, Grantcharov

et al. Effect of virtual reality training on laparoscopic surgery: Randomised controlled trial. BMJ, 2009; 338:b1802.

15.

Seymour

. VR to OR: a review of the evidence that virtual reality simulation improves operating room performance. World J Surg, 2008; 32:182–188.

16.

Schout

, Hendrikx

, Scherpbier

, Bemelmans

. Update on training models in endourology: A qualitative systematic review of the literature between January 1980 and April 2008. Eur Urol, 2008; 54:1247–1261.

17.

Kallstrom

, Hjertberg

, Kjolhede

, Svanvik

. Use of a virtual reality, real-time, simulation model for the training of urologists in transurethral resection of the prostate. Scand J Urol Nephrol, 2005; 39:313–320.

18.

Kallstrom

, Hjertberg

, Svanvik

. Construct validity of a full procedure, virtual reality, real-time, simulation model for training in transurethral resection of the prostate. J Endourol, 2010; 24:109–115.

19.

Seymour

, Gallagher

, Roman

, O'Brien

, Andersen

, Satava

. Analysis of errors in laparoscopic surgical procedures. Surg Endosc, 2004; 18:592–595.

20.

Marklund-Bau

, Edell-Gustafsson

, Spangberg

. Bothersome urinary symptoms and disease-specific quality of life in patients with benign prostatic obstruction. Scand J Urol Nephrol, 2007; 41:32–41.

21.

Madersbacher

, Lackner

, Brossner

et al. Reoperation, myocardial infarction and mortality after transurethral and open prostatectomy: A nation-wide, long-term analysis of 23,123 cases. Eur Urol, 2005; 47:499–504.

22.

Doll

, Black

, McPherson

, Flood

, Williams

, Smith

. Mortality, morbidity and complications following transurethral resection of the prostate for benign prostatic hypertrophy. J Urol, 1992; 147:1566–1573.

23.

Lourenco

, Armstrong

, N'Dow

et al. Systematic review and economic modelling of effectiveness and cost utility of surgical treatments for men with benign prostatic enlargement. Health Technol Assess, 2008; 12:iiiix–x, 1–146, 169–515.

24.

Melville

, Donovon

, Sheldon

, Peters

. Benign prostatic hyperplasia. Qual Health Care, 1996; 5:111–119.