Validity and Acceptability of a High-Fidelity Physical Simulation Model for Training of Laparoscopic Pyeloplasty

Abstract

Purpose:

The objective was to determine the acceptability and preliminary construct validity for a high-fidelity synthetic renal pelvis/ureter tissue analogue model for use as a simulation model for training of laparoscopic pyeloplasty.

Materials and Methods:

The pyeloplasty model was designed with incorporated assessment lines for use in post-task Black Light Assessment of Surgical Technique (BLAST)™. Practicing urologists participating in the 2011 and 2012 American Urological Association Mentored Renal Laparoscopy courses performed a simulated laparoscopic pyeloplasty procedure and completed a post-task evaluation of the model.

Results:

Practicing urologists found the model acceptable and rated the model favorably in terms of content and face validity. Urologists who had performed a laparoscopic pyeloplasty procedure in the last 5 years outperformed those who had not by demonstrating increased patency (P<0.05), decreased twisting (P<0.05), and decreased leakage (P<0.10) at the anastomosis.

Conclusions:

The BLAST™ pyeloplasty model demonstrated evidence of acceptability and content, face, and construct validity for training practicing urologists to perform laparoscopic pyeloplasty.

Introduction

Established curricula using physical simulation models for training and assessment have been designed and validated for basic level laparoscopic skills. The Fundamentals of Laparoscopic Surgery (FLS^™) has been validated for assessment of basic laparoscopic skills¹ and is a prerequisite for general surgery residents taking the American Board of Surgery qualifying examination.² The Basic Laparoscopic Urologic Surgery (BLUS^©) skills curriculum was developed and validated as a method to train and assess basic laparoscopic skills that are valuable specifically for urologic surgeons.³

There also is a need for the development of low-cost, high-fidelity physical tissue analogue simulators for training and assessment of advanced laparoscopic skills. The American Urological Association (AUA) Laparoscopic, Robotic, and New Surgical Technology (LRNST) Committee has identified five procedures for which development of simulator models is desired. These include laparoscopic pyeloplasty, Y-V plasty, vesicourethral anastomosis, and control of aortic and inferior vena caval injury. The laparoscopic dismembered pyeloplasty procedure requires advanced laparoscopic skills including excision of the ureteropelvic junction obstruction (UPJO), spatulation of the renal pelvis and proximal ureter, and intracorporeal suturing of the anastomosis.⁴ These skills are technically demanding and difficult for the inexperienced surgeon to perform.⁵ In addition, the LRNST committee required that the tissue analogue models have inherent means of objective assessment of surgical skill. Other simulator models for training of laparoscopic pyeloplasty with various levels of fidelity have been presented previously in the literature.^6

–10

The Simulation PeriOperative Resource for Training and Learning (SimPortal) at the University of Minnesota has designed a disposable, low-cost, high-fidelity, physical renal pelvis/ureter tissue analogue model to address one of the five procedures identified by the AUA LRNST committee. The model was developed and built, used as part of the 2011 and 2012 AUA Mentored Renal Laparoscopy courses, and evaluated for acceptability and certain aspects of validity. Although acceptability does not contribute to validity, it is an important aspect of establishing opinions on the likelihood of simulator use among a cohort of experts.¹¹ In addition, data were gathered to assess preliminary evidence of face, content, construct, and discriminate validity as defined by Gallagher and associates.¹²

Materials and Methods

Pyeloplasty simulator model

The pyeloplasty simulator model (Fig. 1) was created using organosilicate-based materials and cast using a patient-specific, anatomically accurate, three-dimensional printed mold with postprinting modifications at the UPJO to allow for demolding. Material formulations were determined based on referencing mechanical properties data of urologic tissues from the University of Minnesota Property Human Tissue Database and subjective feedback from practicing urologic surgeons on tissue analogue material interaction with suture and laparoscopic instruments. Minor modifications to the material formulation were made following the 2011 course at the request of the course director.

FIG. 1.

(a) Top, (b), bottom and (c) endoscopic view of the pyeloplasty simulator model.

The renal pelvis is approximately 6 cm between superior and inferior surfaces and 3 cm between anterior and posterior surfaces. The UPJO has an outer diameter of 0.5 cm and internal diameter of 0.2 cm. The ureter is approximately 18 cm from ureteropelvic junction (UPJ) to distal end and has an outer diameter of 0.8 cm and an inner diameter of 0.6 cm. The production cost including materials and labor is approximately $55 per model.

Ultraviolet (UV) light sensitive assessment lines were incorporated into the base material for use in Black Light Assessment of Surgical Technique (BLAST^™). The assessment lines are not visible under room (Fig. 1a,b) or endoscopic (Fig. 1c) lighting conditions. After completion of the exercise, an evaluator places the model under UV lighting conditions (wavelength of 340–380 nm) so that the assessment lines can be visualized (Fig. 2). Four colored lines (yellow, red, blue, and green) were incorporated into the synthetic material along the renal pelvis, UPJ, and ureter and spaced 90 degrees apart. A holder was created for the 2012 course to secure the model within a laparoscopic training box or on another surface (Fig. 3a) and to position the model at a 45-degree angle in relation to the endoscope and laparoscopic instruments (Fig. 3b). The holder has a length of 25.5 cm, width of 10 cm at the base, and height of 10 cm.

FIG. 2.

Pyeloplasty model under ultraviolet light (a) before procedure. Postprocedure examples of pyeloplasty models with twisting scores of (b) 4 and (c) 0.

FIG. 3.

Securing pyeloplasty model in (a) holder and (b) within laparoscopy training box.

AUA study

This study was approved by the Institutional Review Board through the University of Minnesota. Participants in the study were practicing clinical urologists participating in the 2011 and 2012 AUA Mentored Renal Laparoscopy courses. Participants gave informed consent before participation in the study, filled out a demographics form, and used the simulator model to complete the laparoscopic pyeloplasty procedure. Participants in the 2012 course were given a post-task questionnaire to evaluate acceptability, content validity, and face validity of the model using a five-point Likert scale.

Performance metrics and analysis

Performance metrics were collected post-task. Course directors determined that patency, leakage, and twisting at the anastomosis were clinically relevant outcomes that should be included as performance metrics. Patency at the anastomosis is an essential component of defining a successful pyeloplasty procedure¹³ and aids in prevention of secondary UPJO. Leakage at the anastomosis may lead to complications including urinoma, ascites, and stricture formation.¹⁴ Presence of twisting at the anastomosis may lead to future obstruction at the UPJ.¹⁵ Time to completion of the pyeloplasty procedure was not collected by request of the course directors to not pressure course participants.

Patency, leakage, flow rate, and volume remaining in renal pelvis were determined by positioning the pyeloplasty model vertically and adding 12 mL of water to the lateral aspect of the renal pelvis at a constant rate over 3 seconds. Patency was defined as the volume of water that exited the renal pelvis, passed through the anastomosis, and successfully exited the distal ureter. Leakage was defined as the amount of water that exited at the anastomosis. The minimum and maximum values were 0 mL and 12 mL, respectively, for both patency and leakage. Flow rate was defined as the total volume of water that exited the renal pelvis (patency plus leakage) divided by the time needed for the renal pelvis to empty. The minimum time was 3 seconds, and the maximum time was defined as when the water stopped moving out of the renal pelvis.

Models that had complete obstruction were excluded from statistical analysis of the flow rate metric because of having a theoretically infinite emptying time. The remaining volume in the renal pelvis was defined as the volume in the renal pelvis after water stopped flowing out of the renal pelvis. A needle and syringe were used to draw out and measure the remaining volume. The minimum remaining volume was 0 mL and the maximum volume was 12 mL. An additional test for patency was performed by attempting to pass a 7F catheter retrograde across the anastomosis.

Twist angle at the anastomosis (Fig. 2) was measured using BLAST. Each colored line was compared across the anastomosis, and each model was given a score from 0 to 4 quantifying the number of lines that were matched across the anastomosis within a 10-degree rotation left or right. Comparisons between experience levels among the practicing urologists were completed using Mann-Whitney U and chi-square tests with significance level of P<0.05 and potential trend indicated with P<0.10.

Results

Demographics

There were a total of 31 participants in the study with 19 participating in the 2011 course and 12 participating in the 2012 course. Subject ages ranged from 36 to 66 with an average of 49 years (standard deviation=8.7). All participants were attending practicing clinical urologists. There were 30 male participants and 1 female participant; 72% of participants had previously heard of FLS.

Acceptability, content validity, and face validity

Study participants were given post-task questions related to the acceptability, content validity, and face validity of the pyeloplasty simulation model. Averages scores and standard deviations are shown in Figure 4. All participants answered “yes” when asked if laparoscopic/endoscopic simulation should be a part of residency training.

FIG. 4.

Average scores of acceptability, content validity, and face validity ratings using five-point Likert scale. (1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree).

Construct validity

The construct tested was whether previous experience performing laparoscopic, open, or robotic pyeloplasty procedures predicted performance on the simulator (Table 1). Using a Mann-Whitney U test, among the practicing clinical urologist participants, those who had experience in performing a laparoscopic pyeloplasty procedure in the last 5 years outperformed those who had not in terms of increased patency (P<0.05), decreased twisting (P<0.05), and decreased leakage (P<0.10) at the anastomosis. There was no significant difference in performance between the two groups for volume remaining in renal pelvis or flow rate. There was no significant difference in performance between those who had performed an open pyeloplasty procedure in the past 5 years compared with those who had not. Likewise, there was no significant difference in performance between those who had performed a robotic pyeloplasty procedure in the past 5 years compared with those who had not.

Table 1.

Construct Validity Comparing Performance Among Practicing Urologists with Varying Experience in Laparoscopic, Robotic, and Open Pyeloplasty Procedures

	Number of laparoscopic pyeloplasty procedures in last 5 years			Number of robotic pyeloplasty procedures in last 5 years			Number of open pyeloplasty procedures in last 5 years
	<1	≥1	P-value	<1	≥1	P-value	<1	≥1	P-value
N	20	11	N/A	28	3	N/A	10	21	N/A
Patency at anastomosis (mL)	0.8 (0–12)	11.2 (0–12)	0.005	4.2 (0–12)	0 (0–5)	0.316	4.9 (0–12)	1.6 (0–12)	0.492
Leakage at anastomosis (mL)	8.9 (0–12)	0.8 (0–12)	0.079	7.8 (0–12)	7 (0–12)	0.777	7.3 (0–12)	7.9 (0–12)	0.950
Twisting at anastomosis	0 (0–4)	2 (0–4)	0.029	1 (0–4)	0 (0–2)	0.422	1 (0–4)	1 (0–4)	0.950
Volume remaining in renal elvis (mL)	0 (0–12)	0 (0–0)	0.502	0 (0–12)	0 (0–12)	0.462	0 (0–12)	0 (0–0)	0.546
Flow rate (mL/s)	3.88 (0–4), N=18	2.86 (0–4)	0.492	3 (0–4), N=27	0 (0–0), N=2	0.207	4 (0–4)	3 (0–4), N=19	0.195

Values are reported as median (range). N values are specified if different from column labels.

Using a chi-square test, there was no significant difference in performance for whether it was possible to pass a 7F catheter through the anastomosis between those who had or had not performed a laparoscopic (P=0.639), open (P=0.139), or robotic (P=0.267) pyeloplasty in the last 5 years.

Discussion

The development of novel technologies to effectively train urologic surgeons is essential to the field of endourology, and this study indicates that the pyeloplasty simulation model was found to have value and be accepted by a cohort of practicing urologic surgeons. The positive ratings of content and face validity demonstrate the success of this model in allowing for reproduction of appropriate skills needed for laparoscopic pyeloplasty and for behaving similarly to human tissue as shown in Figure 4. The aspects of face validity have lower ratings compared with the acceptability and content validity scores. This model, however, allows for training and assessment of appropriate content used in the pyeloplasty procedure with evidence from high content validity scores. Therefore, we conclude that although aspects of face validity were rated less favorably, participants felt the accuracy was adequate to properly execute the procedure.

The results related to preliminary construct validity were also encouraging, because we were able to distinguish between experience levels in laparoscopic pyeloplasty using the BLAST technique. We found that there was no significant difference between experience levels for the flow rate metric, and in light of the analysis being complex, suggest not including this metric in future analysis. This metric included water that left the renal pelvis via leakage or patency and therefore includes factors that demonstrate favorable and unfavorable outcomes.

As evidence for discriminate validity, the significant differences in performance levels were only present for experience differences in laparoscopic cases and not for open or robotic as shown in Table 1. This demonstrates that performance correlates with factors that we expect to correlate in that we would expect a person's experience in open or robotic procedures to not show significant difference between groups because these skills were not demonstrated by completing the laparoscopic pyeloplasty procedure on the model. Limitations for construct validity in this study include low subject numbers, variation in experience level within the more experienced group, and minor differences in material used during the 2011 and 2012 courses.

The integration of objective means of assessment into the simulation models as required by the AUA LRNST committee was addressed in our model design through the use of BLAST. Current methods used in evaluating surgical technique involve subjective ratings that are inadequate because of their inability to objectively quantify technical skill. Our method allows for post-task quantification of relative alignment of approximated edges at the anastomosis, giving objective feedback to the user or evaluator. The assessment lines used in BLAST are present conditionally, eliminating the possibility that the user could reference them to benefit his or her performance, making it feasible to use BLAST in a testing situation. In addition, the applications of BLAST are not limited to the pyeloplasty procedure.

This type of objective assessment can be applied to any reconstructive procedure where analysis of alignment is needed at the tissue level. Although this model was studied for application to laparoscopic pyeloplasty techniques, it can also be used for training in open or robotic pyeloplasty techniques. The holder provides flexibility in that the model can be positioned in a setting conducive for training in open or robotic pyeloplasty.

The objective means of assessment at the tissue level are especially important in the context of advanced laparoscopic skills. In simulation models used for training and assessment of basic laparoscopic skills such as those used in FLS and BLUS, each model is representative of one task with clearly defined metrics and errors. In contrast, models used in advanced laparoscopic skills represent entire anatomic structures and require the user to perform multiple skills on the same area of tissue. Therefore, analysis of individual deconstructed tasks is challenging because of the difficulty in creating markers for identifying and then objectively quantifying each individually.

To address this issue, we suggest it may be most beneficial to use post-task evaluation of outcomes at the tissue level for simulated advanced laparoscopic skills. Outcomes at the tissue level are ultimately what impact the patient, and final outcomes are independent of variation in technical methodologies or style discrepancies between surgeons.

Conclusions

Our novel simulation model for training of laparoscopic pyeloplasty is acceptable and shows preliminary evidence of face, content, and construct validity. This study demonstrates the value of high-fidelity training models with incorporated objective means of assessment. Continued development of novel methods for objective quantification of surgical skill in the simulation environment is essential to properly train and assess advanced laparoscopic skills. This type of innovation is needed to provide resources to effectively train and assess minimally invasive skills as new surgical technologies become available.

Footnotes

Acknowledgments

Thank you to the American Urological Association Office of Education staff for helping to facilitate this study. We would like to acknowledge Christopher Allam, Timothy Averch, Sara Best, Sean Hedican, Stephen Jackman, Richard Link, Wesley Mayer, Elspeth McDougall, and Sri Sivalingam for their contributions. Thank you to Jason Speich and Sneha Somani for assistance in model building and analysis.

Disclosure Statement

No competing financial interests exist.

Abbreviations Used

References

Fried

, Feldman

, Vassiliou

, et al. Proving the value of simulation in laparoscopic surgery. Ann Surg, 2004; 240:518–528.

Soper

, Fried

. The fundamentals of laparoscopic surgery: Its time has come. Bull Am Coll Surg, 2008; 93:30–32.

Sweet

, Beach

, Sainfort

, et al. Introduction and validation of the American Urological Association Basic Laparoscopic Urologic Surgery skills curriculum. J Endourol, 2012; 26:190–196.

Carr

, Casale

. Anomalies and surgery of the ureter in children. In: Wien

, Kavoussi

, Novick

, et al., eds. Campbell-Walsh Urology. 10th ed. Philadelphia, PA: Elsevier Inc., 2012, pp 3217–3221.

Yanke

, Lallas

, Pagnani

, et al. The minimally invasive treatment of ureteropelvic junction obstruction: A review of our experience during the last decade. J Urol, 2008; 180:1397–1402.

Raza

, Soomroo

, Ather

. “Latex glove” laparoscopic pyeloplasty model: A novel method for simulated training. Urol J, 2011; 8:283–286.

Ooi

, Lawrentschuk

, Murphy

. Training model for open or laparoscopic pyeloplasty. J Endourol, 2006; 20:149–152.

Ramachandran

, Kurien

, Patil

, et al. A novel training model for laparoscopic pyeloplasty using chicken crop. J Endourol, 2008; 22:725–728.

, Zhang

, Lang

, et al. New model for training in laparoscopic dismembered ureteropyeloplasty. J Endourol, 2007; 21:1381–1385.

10.

Yang

, Zhang

, Xiao

, et al. A novel training model for retroperitoneal laparoscopic dismembered pyeloplasty. J Endourol, 2010; 24:1345–1349.

11.

Sweet

, Hananel

, Lawrenz

. A unified approach to validation, reliability, and education study design for surgical technical skills training. Arch Surg, 2010; 145:197–201.

12.

Gallagher

, Ritter

, Satava

. Fundamental principles of validation, and reliability: Rigorous science for the assessment of surgical education and training. Surg Endosc, 2003; 17:1525–1529.

13.

Sundaram

, Grubb

3rd , Rehman

, et al. Laparoscopic pyeloplasty for secondary ureteropelvic junction obstruction. J Urol, 2003; 169:2037–2040.

14.

Matsumoto

, Cadeddu

: Laparoscopic reconstructive renal surgery. In: Wetter

, Kavic

, Levinson

, et al., eds. Prevention and Management of Laparoendoscopic Surgical Complications. 2nd ed. Miami: Society of Laparoendoscopic Surgeons, Inc., 2005, p 491.

15.

Neulander

, Romanowski

, Assali

, et al. Renal pelvis flap-guide for ureteral spatulation and handling during dismembered pyeloplasty. Urology, 2006; 68:1336–1338.