Simulation-Based Training Models for Urolithiasis: A Systematic Review

Abstract

Objectives:

Urolithiasis is one of the most common presentations in urological practice and it is becoming increasingly important to provide structured, simulation-based training using validated training models. This systematic review aims to identify current simulation-based training models and to evaluate their validity and effectiveness.

Methods:

Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, a comprehensive search was performed on the Medline and EMBASE databases for English language articles published between 2000 and 2018 describing and/or assessing validity of simulation models for urolithiasis procedures. Studies were qualitatively assessed for validity using the Messick validity framework and models were assigned levels of recommendation using the McGaghie model of translational outcomes.

Results:

A total of 98 studies were included in this study assessing 51 models, with 28 studies concerning models for urethrocystoscopy, 46 studies for ureterorenoscopy, and 39 studies for percutaneous access and/or nephrolithotomy. Only four models demonstrated a level of recommendation of 4. The most validated models were the URO/PERC-Mentor (Simbionix, Lod, Israel) with multiple studies for each across various procedural skills.

Conclusion:

There is a wide spectrum of simulation-based models currently available for urolithiasis procedures, mostly with limited validity evidence from small studies. Further research is required with higher levels of evidence including randomized controlled trials. In addition, long-term transfer of skills to the operating room should be assessed to establish whether there is genuine skill development and retention using simulation models and whether this helps to reduce surgical complications.

Introduction

Urolithiasis is one of the most common presentations in urology, with trends showing numbers of uretero(reno)scopy (URS) performed worldwide increasing by 252% in the last two decades.¹ During this period, we have also witnessed the introduction of modern minimally invasive surgery (MIS) techniques, such as percutaneous nephrolithotomy (PCNL). These relatively new interventions have reduced transfusion rates and hospital costs.² MIS training, however, is more challenging and associated with higher costs, which consequently requires time and money to establish skills required for these procedures.³

Simulation is thought to be effective during the initial phase of training.⁴ Previously, residents underwent traditional methods to acquire surgical skills in the operating room (OR) on patients. Strict ethical imperatives have further reduced training opportunities and fast-tracked the growth of simulator use to form the gateway of the next level of surgical training. This approach allows exposure to crucial surgical skills without presenting any risk to patients. In terms of achieving desired outcomes, simulation-based practice in surgical education illustrates a dose–response relationship, whereby more practice yields better results.⁵

Much of the current literature is still loosely centered around a subjective and outdated concept of validity. Although this taxonomy maintains some relevance to simulation-based training, behavioral scientists no longer accept this method of assessment. The newer concept considers validity as a unitary construct with an added focus on the interpretation of simulator data.⁶

The principal objectives of this review are follows:

To identify current simulation-based modalities pertaining to urolithiasis procedures,

to evaluate the evidence behind its validity status according to updated definitions, and

to make recommendations for use in surgical training based on evidence.

Materials and Methods

This study was conducted using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement guidelines⁷ and registered on PROSPERO (CRD42019118734).⁸

Eligibility criteria

Articles selected for review included various study types including observational, randomized controlled trials (RCTs), nonrandomized studies of interventions, pilot studies, and questionnaires. Review articles were included, subject to them presenting new information not already mentioned in original studies. Articles that mentioned use of a simulation-based model for urolithiasis surgical training and/or had its validity assessed were also considered eligible for selection. Conference abstracts were included if they matched the search criteria and presented novel information. Reference lists in retrieved studies were hand-searched for other relevant studies. The exclusion criteria included research articles that did not discuss a relevant procedure, were not in the English language, or published before 2000. Articles that did not explicitly evaluate a model/simulator for use in urolithiasis training were also excluded, as were articles that did not present sufficient information.

Information sources and search method

The Medline and EMBASE databases were searched for articles discussing simulated training methods for urolithiasis procedures and/or assessing its validity. A wide initial search was conducted to identify existing articles dated between 2000 and November 2018. The terms of this literature search were set out to identify the majority of relevant articles, and to include three principle concepts:

urolithiasis (the condition)

endourology/urethrocystoscopy/ureterorenoscopy/ureteroscopy (the intervention) and

training/models/simulation.

Search terms included “urology,” “endourology,” “ureteroscopy,” “endourology,” “urethrocystoscopy,” “cystoscopy,” “ureterorenoscopy,” “PCNL,” “percutaneous,” and “percutaneous nephrolithotomy” followed by combinations of the terms “model*,” “train*” or “simulat*.” Articles that mentioned the term “ureter*” in the abstract were also screened and considered for inclusion.

Study selection

Titles and abstracts of retrieved studies were examined to identify relevant studies to be included. Duplicates of articles were removed. The remaining studies were then further examined with a full text review. Where disputes on inclusion occurred, a consensus was determined among the authors.

Data items

Information obtained from each study included authors, year of publication, study design, number of participants, training time, interventions and outcomes as well as data reflecting simulator use such as time taken to compete the task, number of attempts, number of adverse outcomes, Objective Structured Assessment of Technical Skills (OSATS) scores in addition to expert and participant feedback. Details of the models such as the name, manufacturer, mechanism of function, and the cost of production were also retrieved. Data analysis of study data were also utilized to establish whether there were any statistically significant differences between control and intervention groups. Statistical evidence presented to evaluate interrater/test-retest reliability, such as Cohen's kappa or Cronbach's alpha, were also considered as a measure for reliability and internal consistency, respectively.

The selected models were classified into distinct groups: Augmented Reality (AR), Virtual Reality (VR), Bench, Organic, Animal and Cadaveric models.

Data analysis

Studies were analyzed according to the modern concept of validity set out by Messick's framework to identify its level of evidence (LoE).⁶ These were given a rating subject to the study addressing parameters such as content, response processes, internal structure, relations to other variables, and consequences of using the simulated model. Studies were rated from N to 2, where N represents no discussion regarding validity evidence and zero indicates mention of validity with no evidence presented. A score of 1 or 2 suggests that the data support the studies validity.

Selected models were given levels of recommendation (LoR) as assessed per McGaghie's model of translational outcomes to establish whether it is an effective tool recommended for surgical training.⁹ This was quantified using an LoR score from level 1 to 5, where LoR 1 indicates reference to a single parameter mentioned in the model. An LoR 5 indicates a cumulative reference of internal acceptability, contained effects, downstream effects, and target effects with the introduction of data signifying a wider systematic level of consequence.⁹ Where an LoR could not be awarded, it was denoted with N/A (not applicable or not available). Studies that presented insufficient information were hyphenated in the relevant sections of the table (Table 1). Because of the heterogeneity of studies, it was not possible to perform a quantitative analysis.

Table 1.

Validation Studies on Urological Training Models (2000–2018) for Urethrocystoscopy

Name of model	Type of model	Study	n	Participants	Training time	Validity	LoR
Glass globe (Catharina Hospital, Eindhoven, the Netherlands)	Bench	¹⁴	32	Medical students	15 minutes	Content: 0 Response processes: 2 Internal structure: 2 Relations to other variables: 2 Consequences: N	2
ETXY Uro Adam (ProDelphus, Olinda, Brazil)	Bench	¹⁶	40	10 urologists, 30 residents	—	Content: 2 Response processes: 2 Internal structure: 2 Relations to other variables: 2 Consequences: N	2
ETXY Uro Adam (ProDelphus, Olinda, Brazil)	Bench	²⁰	60	Urologists	6.5 hours		2
Uro Scopic Trainer (Limbs & Things, Bristol, United Kingdom)	Bench	³⁹	16	Residents	3 days	Content: N Response processes: 2 Internal structure: N Relations to other variables: 1 Consequences: 0	2
Transparent Simulator Model (The Third Military Medical University, ChongQing, China)	Bench	²²	20	Urologists	—	Content: N Response processes: 2 Internal structure: 2 Relations to other variables: N Consequences: 1	1
	Bench	²³	46	10 Urologists, 36 medical students	12 hours		1
URO Mentor (Simbionix, Lod, Israel)	VR	¹⁴	32	Medical students	15 minutes	Content: 2 Response processes: 2 Internal structure: 2 Relations to other variables: 2 Consequences: 0	3
		³⁹	16	Residents	3 days
		²⁴	16	Medical students	—
		²⁵	21	Medical students	—
		²⁹	14	Urology nurse practitioners	6.5 hours
		³⁰	17	10 medical students, 7 urologists	—
		³¹	57	30 medical students, 27 expert cystoscopists	—
		¹¹	10	Novices	6 sessions (45 minutes)
		²⁶	89	56 urologists, 33 residents	—
		¹³	71	Interns	30 minutes (15 × 2)
		¹²	80	30 experts, 50 novices	15 minutes
		³²	38	33 specialist registrars, 5 urological nurses	—
		³³	18	Urologists	—
URO-Trainer (Karl Storz GmbH, Tuttlingen, Germany)	VR	³⁴	10	5 residents, 5 medical students	—	Content: 2 Response processes: N Internal structure: N Relations to other variables: 2 Consequences: N	3
URO-Trainer (Karl Storz GmbH, Tuttlingen, Germany)	VR	³⁵	36	24 medical students, 12 residents	—		3
George Washington University Medical Center, Washington, DC	VR	³⁶	8	Residents	3 sessions	Content: 1 Response processes: 1 Internal structure: N Relations to other variables: 0 Consequences: N	N/A
Porcine Liver in pumpkin	Animal	¹⁷	—			Content: NResponse processes: NInternal structure: NRelations to other variables: NConsequences: N	1
Porcine Bladder	Animal	³⁷	11	Trainees	2 hours	Content: 1Response processes: NInternal structure: 0Relations to other variables: NConsequences: N	1
Boar Urinary Tract	Animal	¹⁵	2	Residents	—	Content: NResponse processes: 1Internal structure: 2Relations to other variables: 1Consequences: N	3
Live Porcine	Animal	¹⁶	40	10 urologists 30 residents	—	Content: 2Response processes: 1Internal structure: 1Relations to other variables: 2Consequences: 0	3
Live Porcine	Animal	²⁰	60	Urologists	2 days		3
Fresh Frozen Cadavers	Cadaver	¹⁸	29	Residents	2 hours	Content: 2Response processes: 2Internal structure: 2Relations to other variables: 2Consequences: N	3
Fresh Frozen Cadavers	Cadaver	¹⁹	81	Residents	3 days		3
Thiel-Embalmed Cadavers	Cadaver	²⁷	12	Urologists	—	Content: 2Response processes: 1Internal structure: 1Relations to other variables: NConsequences: N	3
		²⁸	6	Urologists	2 days
		³⁸	—
Pumpkin in green pepper	Organic	¹⁷	—			Content: 0Response processes: NInternal structure: NRelations to other variables: NConsequences: N	2

LoR = level of recommendation; VR = virtual reality.

Quality assessment

LoE for each study was ascertained according to the guidelines indicated by the validated modified Oxford Centre for Evidence-Based Medicine assessment tool.¹⁰ Scores were assigned to each study that were then ranked according to the LoE presented (levels 1–5). Studies that attained LoE 5 were discarded and ineligible for inclusion owing to the strong risk of bias it presented. Study scores may be upgraded/downgraded based on study quality, precision, and consistency between other similar studies. Three examiners independently reviewed all identified studies to determine if they met the inclusion criteria and whether they matched the quality of evidence required. In the event of disagreement surrounding the inclusion of studies, named authors reached a consensus on inclusion.

Results

A total of 98 studies were included from 2525 initially retrieved studies (Fig. 1) and reference screening (Tables 1 –3).

FIG. 1.

Study selection process, according to the PRISMA statement. PRISMA = Preferred Reporting Items for Systematic reviews and Meta-Analysis.

Table 2.

Validation Studies on Urological Training Models (2000–2018) for Ureterenoscopy

Name of model	Type of model	Type of URS	Study	n	Participants	Training time	Validity	LoR
Adult Ureteroscopy Trainer (Ideal Anatomic Modeling, Holt, MI)	Bench	fURS	⁴⁰	46	Urologists, residents, medical students, industry representatives	—	Content: 2Response processes: 1Internal structure: NRelations to other variables: 2Consequences: N	3
Endo-Urologie Modell (Karl Storz GmbH)	Bench	fURS	⁴¹	21	Urologists	5 hours	Content: 2Response processes: 2Internal structure: 1Relations to other variables: NConsequences: 1	2
Penrose drain (University of Toronto, Canada)	Bench	rURS	⁴²	40	Medical students	75 minutes	Content: 0Response processes: 2Internal structure: 2Relations to other variables: 2Consequences: 0	2
Smart Simulator (Kansai Medical University, Hirakata, Japan)	Bench	fURS	⁴³	16	n = 16 (Urologists)	—	Content: 2Response processes: NInternal structure: NRelations to other variables: NConsequences: 1	4
Scope Trainer (Mediskills, Northampton, United Kingdom)	Bench	rURS	⁴⁴	14	Urology trainees, consultants	—	Content: 0Response processes: 2Internal structure: 2Relations to other variables: 2Consequences: N	3
Scope Trainer (Mediskills, Northampton, United Kingdom)	Bench	rURS	⁴⁵	26	Residents	4 days		3
Advanced Scope Trainer (Mediskills)	Bench	fURS	⁴⁶	53	19 medical students, 34 trainees	—	Content: 2Response processes: 2Internal structure: NRelations to other variables: 2Consequences: 0	3
CREST KU/KUB Model (SimPORTAL, University of Minnesota, MN)	Bench	rURS	⁴⁹	—		—	Content: 0Response processes: NInternal structure: NRelations to other variables: NConsequences: N	N/A
		fURS	⁴⁷	10	Residents	—	Content: 1Response processes: 1Internal structure: 1Relations to other variables: 2Consequences: 0	1
			⁴⁸	30	Residents	—
			⁴⁹	—		—
Cook URS Trainer (Cook Medical, Bloomington, IN)	Bench	fURS	⁵⁰	15	Residents	90 minutes	Content: 2Response processes: 2Internal structure: 2Relations to other variables: 2Consequences: 0	3
Cook URS Trainer (Cook Medical, Bloomington, IN)	Bench	fURS	⁵¹	10	Residents	3 sessions		3
Kidney Box (K-Box; Porges-Coloplast, Rosny-sous-Bois, France)	Bench	fURS	⁵²	—			Content: 2Response processes: 2Internal structure: 1Relations to other variables: 2Consequences: N	2
			⁵³	16	Medical students	10 days
			⁵⁴	10	Medical students	100 minutes
			⁵⁵	231		45 minutes
ETXY Uro Adam (ProDelphus)	Bench	rURS	¹⁶	40	10 Urologists 30 residents	—	Content: 2Response processes: 1Internal structure: 2Relations to other variables: 2Consequences: N	2
ETXY Uro Adam (ProDelphus)	Bench	fURS	²⁰	60	Urologists	16 hours	Content: 2Response processes: 1Internal structure: 2Relations to other variables: 2Consequences: N	2
Uro-Scopic Trainer (Limbs & Things)	Bench	rURS	³⁹	16	Residents	3 days	Content: 2Response processes: 2Internal structure: 2Relations to other variables: 2Consequences: 0	4
			⁴²	40	Medical students	—
			⁵⁶	17	Residents	2 hours
			⁵⁸	32
			⁵⁹	32
		fURS	⁴¹	21	Urologists	5 hours	Content: 2Response processes: 1Internal structure: 1Relations to other variables: NConsequences: N	2
		fURS	⁵⁷	16	Medical students	2 hours		2
EndoUro-Trainer (SAMED, Dresden, Germany)	Bench	rURS	⁵⁵	231	n = 231	45 minutes	Content: 1Response processes: NInternal structure: 0Relations to other variables: NConsequences: 0	2
EndoUro-Trainer (SAMED, Dresden, Germany)	Bench	fURS	⁵⁵	231	n = 231	45 minutes	Content: 2Response processes: NInternal structure: 0Relations to other variables: NConsequences: 0	2
Transparent Simulator Model (The Third Military Medical University)	Bench	rURS	²³	46	10 urologists, 36 medical students	12 hours	Content: 0Response processes: 2Internal structure: 2Relations to other variables: 1Consequences: N	2
URO Mentor (Simbionix)	VR	rURS	³⁹	16	Residents	3 days	Content: 2Response processes: 2Internal structure: 1Relations to other variables: 2Consequences: 0	3
			²³	46	10 urologists, 36 medical students	12 hours
			²⁴	16	Medical students	30 minutes
			²⁶	89	56 urologists, 33 residents	—
			⁵⁸	32	Medical students	—
			⁵⁹	32	—	—
			⁶⁰	20	Trainees	—
			⁶¹	5	Residents	—
			⁶⁸	16	Residents	5 days
		fURS	²⁴	16	Medical students	30 minutes	Content: 2Response processes: 2Internal structure: 1Relations to other variables: 2Consequences: 1	4
			²⁵	21	Medical students	—
			⁴¹	21	Urologists	90 minutes
			⁶¹	20	Urologists	—
			⁶²	30	20 urologists, 10 residents	30 minutes
			⁶³	16	Medical students	5 hours
			⁶⁴	30	17 physicians, 13 associate chief physicians	5 hours
			⁶⁵	32	16 residents, 16 medical students	5 hours
			⁶⁶	13	Residents	3 hours
			⁶⁷	30	Residents	—
			⁶⁸	16	Residents	5 days
			⁶⁹	30	Residents	—
ICSAD+ROVIMAS (Imperial College, London, United Kingdom)	Bench	rURS	⁷⁰	20	Endourologists, residents	—	Content: NResponse processes: 1Internal structure: 2Relations to other variables: 2Consequences: N	N/A
Porcine kidney/ureter model	Animal	rURS	¹⁷	—		—	Content: 2Response processes: 2Internal structure: 2Relations to other variables: 2Consequences: N	3
			¹⁶	40	10 urologists 30 residents	—
			⁷¹	—		—
		fURS	¹⁷	40	—	—	Content: 2Response processes: 2Internal structure: 2Relations to other variables: 2Consequences: N	3
			²²	20	Urologists	—
			²⁰	60	Urologists	9.5 hours
			⁵⁷	16	Medical students	—
Siloah Porcine-in-Box model (Richard Wolf GmbH, Pforzheim, Germany)	Animal+Bench	rURS+fURS	⁷²	350		—	Content: 1Response processes: NInternal structure: NRelations to other variables: NConsequences: N	1
Live porcine	Animal	fURS	²⁰	60	Urologists	9.5 hours	Content: 2Response processes: 2Internal structure: 1Relations to other variables: 2Consequences: N	2
Thiel-Embalmed Cadavers	Cadaver	rURS	²⁷	12	Urologists	—	Content: 2Response processes: 1Internal structure: 1Relations to other variables: NConsequences: N	2
		rURS	²⁸	6	Urologists	—		2
		fURS	²⁸	6	Urologists	—	Content: 2Response processes: 1Internal structure: 1Relations to other variables: NConsequences: N	2
Fresh Frozen Cadavers	Cadaver	rURS	²⁴	32	Urologists, medical students	—	Content: 2Response processes: 0Internal structure: 1Relations to other variables: NConsequences: N	2
		rURS	¹⁹	13	Urologists, residents	3 days		2
		fURS	²⁴	32	Urologists, medical students	—	Content: 2Response processes: 1Internal structure: 1Relations to other variables: 2Consequences: N	2
			¹⁹	13	Urologists, residents	3 days
			⁶⁵	32	16 residents, 16 medical students	5 hours
			⁷³	12	Urologists	—

fURS = flexible ureterorenoscopy; rURS = semi-rigid ureteroscopy; URS = uretero(reno)scopy.

Table 3.

Validation Studies on Urological Training Models (2000–2018) for Percutaneous Nephrolithotomy

Name of model	Type of model	Part of PCNL	Study	n	Participants	Training time	Validity	LoR
Percutaneous Nephrolithotomy Trainer (Limbs & Things, Bristol, United Kingdom)	Bench	Full PCNL	⁸²		—	—	Content: 0Response processes: NInternal structure: NRelations to other variables: NConsequences: N	N/A
iPERC (Instituto Mexicano del Seguro Social, Mexico City, Mexico)	Bench	Access	⁸⁴	30	16 urologists, 14 residents	—	Content: NResponse processes: 1Internal structure: NRelations to other variables: 1Consequences: 0	2
			⁸⁵	16	Residents	20 sessions
			⁸⁶	30	16 urologists, 14 Residents	—
ManiPERC (Instituto Mexicano del Seguro Social, Mexico City, Mexico)	Bench	Access	⁸⁵	16	Residents	20 sessions	Content: 0Response processes: 1Internal structure: NRelations to other variables: 1Consequences: 0	2
CAT (SimPORTAL, University of Minnesota)	Bench	Access	⁸⁷	14	Residents	—	Content: 1Response processes: 1Internal structure: 1Relations to other variables: 0Consequences: N	2
			⁸⁸	23	Residents	1 day
			⁸⁹	38	Physicians	30 minutes
Rapid prototype model (University Hospital of Tours, Tours, France)	Bench	Access	⁹⁰	1	Patient	6 sessions	Content: NResponse processes: NInternal structure: 0Relations to other variables: NConsequences: 1	1
		Full PCNL	⁹¹	15	Patients	7 sessions	Content: 0Response processes: NInternal structure: NRelations to other variables: NConsequences: 1	3
		Full PCNL	⁹²	15	Patients	—		3
Sponge trainer (Tanta University, Tanta, Egypt)	Bench	Access	⁹³	18	Residents	4 sessions	Content: 2Response processes: 1Internal structure: 1Relations to other variables: 1Consequences: N	4
PCNL Simulated Inanimate Model (SIM; University of Rochester, Rochester, NY)	Bench	Access	⁹⁴	15	5 experts, 10 novices	—	Content: 2Response processes: 1Internal structure: NRelations to other variables: 2Consequences: N	3
Model 071A Abdominal Phantom (CIRS, Inc., Norfolk, VA)	Bench	Access	⁹⁵	15	Residents	1 hour	Content: NResponse processes: 1Internal structure: NRelations to other variables: 2Consequences: N	2
PCNL Boz Kidney Trainer (Encoris, Holland, MI)	Bench	Full PCNL	⁹⁶	25	Residents	—	Content: 2Response processes: NInternal structure: NRelations to other variables: NConsequences: N	3
PERC Mentor (Simbionix)	VR	Access	⁷⁵	63	Trainees	1 hour	Content: 2Response processes: 2Internal structure: 2Relations to other variables: 2Consequences: N	3
			⁷⁶	24	9 experts, 15 novices	—
			⁷⁷	24	Experts	—
			⁷⁸	14	Residents	2 hours
			⁷⁹	36	Trainees	—
			⁸⁰	24	9 experts, 15 novices	—
			⁸¹	24	Experts	—
			¹¹⁰	26	Residents	—
		Full PCNL	³²	38	33 registrars, 5 nurses	—	Content: 2Response processes: 1Internal structure: 2Relations to other variables: 2Consequences: N	2
Perk Tutor (Queen's University, Kingston, ON, Canada)	AR	Access	⁹⁷	—			Content: NResponse processes: 1Internal structure: 1Relations to other variables: NConsequences: N	1
Perk Tutor (Queen's University, Kingston, ON, Canada)	AR	Access	⁹⁸	4	Residents	—		1
Porcine Kidney	Animal	Access	⁸³	30	16 urologists, 14 residents	—	Content: 1Response processes: NInternal structure: NRelations to other variables: 1Consequences: N	2
			¹⁰⁰	—		—
			¹⁰¹	—		—
			¹⁰²	8	3 experts, 5 novices	—
			¹⁰³	36	Urologists	3 hours
			¹⁰⁴	11	Urologists	—
		Full PCNL	¹⁷	—		—	Content: 0Response processes: 1Internal structure: 1Relations to other variables: NConsequences: N	2
			⁹⁹	33	Trainees	—
			¹¹¹	—		—
			¹⁰⁰	—		—
			¹⁰¹	—		—
			¹⁰⁴	11	Urologists	—
Porcine Kidney in Chicken model	Animal	Access	¹⁰⁵	—		—	Content: 1Response processes: 0Internal structure: NRelations to other variables: NConsequences: N	2
		Full PCNL	¹⁰⁵	—		—	Content: 1Response processes: 1Internal structure: 0Relations to other variables: 0Consequences: N	2
		Full PCNL	¹⁰⁶	—		—		2
Live Porcine	Animal	Access	⁷⁷	24	Experts	—	Content: 2Response processes: 1Internal structure: 1Relations to other variables: 2Consequences: 0	2
			⁸⁰	24	9 experts, 15 novices	—
			⁸¹	24	Experts	—
Vegetable PCNL model (NU Hospitals, Bangalore, Karnataka, India)	Organic	Access	¹⁰⁷	—		—	Content: 0Response processes: 1Internal structure: NRelations to other variables: NConsequences: 0	1
Training Box (Uludag University Faculty of Medicine, Bursa, Turkey)	Bench	Access	¹⁰⁸	26	5 experts, 21 novices	2 hours	Content: 2Response processes: 2Internal structure: 1Relations to other variables: 2Consequences: N	3
Peking University, Shougang Hospital, Beijing	Bench	Access	¹⁰⁹	48	11 experts, 37 novices	2 hours	Content: 2Response processes: 1Internal structure: 1Relations to other variables: 2Consequences: 0	3

AR = augmented reality; CAT = C-Arm Trainer; PCNL = percutaneous nephrolithotomy.

Urethrocystoscopy

Twenty-eight studies^{11

–38} were identified describing a total of 14 simulation-based training models, consisting of bench (n = 4), VR (n = 3), animal (n = 4), cadaver (n = 2), and organic (n = 1) subtypes. The mean number of participants was 32.9 (range 2–89). An LoR of 3 was achieved by six models, whereas four demonstrated LoR 2. Only one model had maximum validity for the consequences concept and six for content validity, the internal structure concept, and the response processes concept. The URO Mentor (Simbionix, Lod, Israel) was the most commonly reported model with a total of 13 studies retrieved. The primary function of this VR simulator was to provide training for ureterenoscopy, and also includes a number of basic tasks on systematic bladder inspection and bladder biopsies, which enables urethrocystoscopy (UCS) training.

Validated models

Twenty-seven studies were identified that validated 12 simulation models for UCS training, with 13 studies pertaining to the URO Mentor. This model demonstrated a good level of validity (Table 1) achieving an LoR of 3 owing to favorable responses and evidence of skills development.

Gettman and colleagues¹¹ used the URO Mentor to develop a curriculum and showed significant improvement in simulator time among 10 novices (p < 0.001). A complementary study¹² with 80 participants highlighted the simulator's ability to produce scores that correlate with user's experience level. The practical significance of the results were quantified using effect sizes (ES), where scores of 0.10, 0.30, and 0.50 were considered small, medium, and large, respectively. Data on novice performance demonstrated significant improvement with large ES scores in time (p < 0.001, ES = 0.66) and mean Global Rating Scale scores (p < 0.001, ES = 0.84). Furthermore, an RCT by Schout and colleagues¹³ of 100 participants highlighted the transfer of skills from the VR simulator to the OR (p ≤ 0.003). Another RCT¹⁴ utilized this VR simulator to teach UCS skills to 32 medical students. The VR simulator was used in combination with a low-cost, low-fidelity glass model. The intervention arm utilized both models together and outperformed the control arm, which used the URO Mentor alone.

Three studies were identified, each of which described an animal model for UCS. Grimsby and colleagues¹⁵ used a suspended boar urinary tract utilized by two novices, which described its use as a high-fidelity tissue surrogate. This model can be used to enhance core urological skills including cystoscopy in novices. It was also able to differentiate between levels of experience for foreign body retrieval and bladder biopsy procedures. Similarly, Soria and colleagues¹⁶ developed a training model for UCS using a live female porcine bladder. This study used 50 participants and demonstrated realistic domains that accurately represent human anatomy. It also determined that the simulator was a useful tool and realistically simulated the procedure while discriminating between independent theoretical relations of skill (p = 0.000). Hammond and colleagues¹⁷ described a novel model that uses a porcine liver submerged in irrigant within a cored-out pumpkin. This model is yet to be validated; however, it received an LoR of 1 owing to its high fidelity-to-cost ratio.

Two studies^18,19 were identified describing the use of fresh frozen cadavers (FFC) for rigid and flexible UCS skills. Ahmed and colleagues¹⁹ described evaluation of the British Association of Urological Surgeons (BAUS) Cadaveric Training Programme, with 81 residents and demonstrated that FFCs give an accurate representation of a number of surgical procedures including rigid/flexible cystoscopies and bladder biopsies. FFCs displayed feasibility, cost-effectiveness, and a high value for educational influence for cadaveric simulation and therefore achieved an LoR of 3. An RCT study by Bowling and colleagues¹⁸ on 29 obstetric residents highlighted the models ability to correlate with theoretical relationships of skill and therefore achieved a score of 2 for relations with other variables.

Uretero(reno)scopy (URS)

Forty-seven studies^{16,17,19,20,22

–28,39

–74} were identified that described a total of 20 models used for URS, consisting of bench (n = 14), VR (n = 1), animal (n = 3), and cadaver-based (n = 2) models. Seventeen studies described the URO Mentor (Simbionix), a computer-based VR simulator for training in both semi-rigid/flexible URS and rigid/flexible UCS procedures. The Uro-Scopic Trainer (Limbs & Things, Bristol, United Kingdom) was the second most described trainer and mentioned in seven studies. This high-fidelity bench model allows for both semi-rigid and flexible URS training with real instruments and consumables.

Three models demonstrated LoR of 4 and another three demonstrated LoR of 3, with the majority achieving an LoR of 2 (n = 14). Three models had maximum validity for the consequences concept, 17 for content validity, 10 had maximum score for the internal structure concept, 17 for relations to other variables, and 12 for the response processes concept.

Validated models

Forty-five studies were identified, validating 20 models for URS. Seventeen of these studies were aimed at the URO Mentor, which was by far the most evaluated option for URS training. Matsumoto and colleagues³⁹ used the URO Mentor as an assessment tool on a case of distal ureteral stones with 16 residents, whereas Wilhelm and colleagues²⁵ utilized it as a training tool in their RCT of 21 medical students and demonstrated statistically significant improvement in the trained group in parameters such as ability to perform the task (p = 0.035), total evaluator score (p < 0.001), and overall performance (p = 0.004). Mishra and colleagues⁴¹ identified a mean realism score of 7.16 on a 10-point Likert scale (7.16 ± 1.69), among a group of 21 urologists performing flexible URS.

The study conducted by Ogan and colleagues⁶⁵ used cadavers and the URO Mentor for simple diagnostic URS among medical students and residents to determine the ability of these models to accurately predict OR performance. Within the medical student cohort, the performances of the VR model and cadaver showed a strong correlation for several measured parameters including total anatomy score (p = 0.008), time for task completion (p = 0.022), and overall score (p = 0.010). However, there was little correlation between the two models in residents, with statistically significant differences observed in manipulation (p = 0.043) and tasks (p = 0.046) only. The URO Mentor was accredited with an LoR of 4.

Chou and colleagues⁵⁷ used a porcine model to assess 16 medical students trained either by the Uro-Scopic Trainer or the URO Mentor. This study established that the ability of medical students to perform a basic URS procedure was independent of the training modality, thus confirming that incorporation of either simulator may improve clinical performance. Brunckhorst and colleagues⁵⁸ also developed a curriculum, incorporating a selection of tasks from models. The authors demonstrated the accuracy of the Uro-Scopic Trainer in mimicking the procedure and its ability to reflect the user's level of experience.

In addition to the Uro-Scopic Trainer, 13 other bench models were validated by a total of 20 studies. A high-fidelity bench model, the Adult Ureteroscopy trainer (Ideal Anatomic Modeling, Holt, MI), was evaluated by White and colleagues⁴⁰ on a cohort of 46 participants. This model demonstrated great accuracy in illustrating a realistic procedure with all participants rating the trainer as realistic and easy to use. It also displayed its ability to differentiate user levels of skill effectively with experienced surgeons scoring significantly higher on the global rating score (GRS) (p < 0.0001) and checklist (p = 0.004). They also required less time to completion on the task (p = 0.01). The Adult Ureteroscopy trainer achieved an LoR score of 2.

An anatomically accurate, transparent model of the urinary tract was assembled by Hu and colleagues²³ This was then compared with a similar model layered with black paint. Thirty-six medical students participated in this study and concluded that the cohort that trained with the transparent simulator had superior OSATS scores (p = 0.001).

Soria and colleagues¹⁶ described utilizing the ETXY Uro Adam (ProDelphus, Olinda, Brazil) high-fidelity bench model, a porcine “renoureteral tissue” model, and a live porcine model for URS training among 40 trainees and 10 experts. The authors concluded that this model demonstrated satisfactory evidence for its ability to correlate with operating experience in all assessed parameters, and as a useful tool for residents-in-training (p < 0.001). Time for completing the task reduced with successive training, eventually plateauing at the sixth attempt (18 ± 3.4 to 11 ± 1.2 minutes, p < 0.05). This model received an LoR of 3.

Use of Thiel-embalmed cadavers (TECs) for URS was reported in three studies^27,28,74 effectively highlighting its usefulness as a tool for OR preparation. The use of FFCs for semi-rigid and flexible URS was described by Ahmed and colleagues¹⁹ as part of the BAUS Training Programme and again demonstrated to be beneficial as a teaching model and in its detail at mimicking the operating environment. The use of TECs and FFCs both received an LoR of 3.

Percutaneous access and/or nephrolithotomy

Thirty-nine studies^{17,32,75

–111} were identified pertaining to a total of 17 models used for percutaneous renal procedure, consisting of bench (n = 11), VR (n = 1), AR (n = 1), animal (n = 3), and organic (n = 1) modalities. One model demonstrated LoR of 4 and six models demonstrated LoR of 3 with the majority otherwise scoring an LoR of 2 (n = 10). Two models had maximum validity for the consequences concept, eight for content validity, two for the internal structure concept, seven for relations to other variables, and two for the response processes concept.

Nine studies described the PERC Mentor (Simbionix) VR simulator,^{32,75

–81,110} which is a platform often combined with the URO Mentor and allows for percutaneous puncturing of a mannequin under fluoroscopy guidance, using a virtual C-arm through a range of clinical cases. The mannequin has simulated layers of the epidermis and palpable ribs, enabling haptic feedback.

Bruyere and colleagues⁹⁰ created a rapid prototyping model that utilized patient CT images allowing for patient-specific simulation before the procedure. The PCNL Trainer (Limbs and Things) consists of a slab comprising multiple calices where stones may be placed. It also responds realistically to the use of instruments.⁸²

Three main animal models were utilized for training. The porcine kidney was used while wrapped in silicon,¹¹¹ foam,¹⁰⁰ or placed inside a chicken carcass to simulate the posterior tissue layers.^17,105,106 Several other studies have reported use of live porcine models for training.^77,80,81

Validated models

Thirty-nine studies were identified validating 17 models for percutaneous renal procedures. Nine were attributed to the PERC Mentor.^{32,75

–81,110} These studies established the model's usefulness as a surgical learning tool and also its ability to correlate with external, independent factors. This was highlighted by Knudsen and colleagues,⁷⁵ who concluded that training using the simulator significantly improved performance in complete novices (n = 31) and residents (n = 32). Shamim Khan and colleagues³² reinforced this concept in their evaluation of a dedicated urology simulation program utilizing the PERC Mentor among other simulators where senior trainees performed significantly better than juniors in all simulation sessions (p < 0.001). The studies evaluating this model cumulatively displayed good quality control using standardized, validated assessment scales thus minimizing examiner bias. The PERC Mentor's ability to predict OR performance was established by Mishra and colleagues⁸⁰ where novices were trained using the PERC Mentor. Before training, 1/5 novices attained baseline success in contrast to 5/5 novices who demonstrated post-training success. Participants who were unavailable to perform testing on the porcine model were excluded from the final analysis to avoid test result bias.

Another study by Noureldin and colleagues¹¹⁰ employed a cohort of 26 postgraduate trainees who used the PERC Mentor to assess competency in attaining percutaneous renal access. The correlation between previous experience with the PERC Mentor and other endourological procedures were assessed comparing GRS. Results revealed that 100% of the trainees with previous experience with the simulator were competent, compared with 38% in the group without sufficient experience (p < 0.001). All postgraduate trainees who had utilized the simulator for training also performed the task with significantly shorter operating (p < 0.001) and fluoroscopy times (p < 0.001) and achieving a significantly higher number of successful punctures to access renal calices (p = 0.001). This model attained an LoR score of 3.

Six bench models were identified,^84,87,89 which negate the need for fluoroscopy and subsequent radiation exposure. The Fluro-less C-arm Trainer (CAT; SimPORTAL, MN) was evaluated by 14 urologists⁸⁷ demonstrating validity evidence for its content, its ability to accurately reflect a realistic operating environment, and its worth as a surgical learning tool. Chrouser and colleagues⁸⁸ conducted a 2-day training program utilizing the CAT among 23 residents, following which significant improvements in performance were observed in upper pole renal access (p = 0.0015), but with no changes seen in lower pole access. Furthermore, participants perceived the CAT as a “very helpful” training tool for familiarizing with PCNL access. These results highlight the model's ability to correlate with levels of experience, thus achieving an LoR of 2.

Three animal models were validated^{17,77,80,81,83,99

–106,111} and used for training experts and novices. A porcine kidney including its own skin, subcutaneous fat, fascia, and muscle layers was described by Zhang and colleagues⁹⁹ where the authors recruited 42 urologists to perform percutaneous procedures on the model. All participants rated the model “helpful” or “very helpful” and it received an LoR of 2. Mishra and colleagues⁸¹ described the use of live anesthetized pigs for percutaneous renal access and the PERC Mentor with 24 experts. The study concluded that both models were equally useful (mean [SD], 4.6/5 [0.6] vs 4.65 [0.5]). Although the former was more realistic (p < 0.001), the PERC Mentor proved to be more useful in terms of repeating tasks (4.85 [0.4] for both, p < 0.001). The use of live porcine received an LoR of 2.

Discussion

For many years, the fundamentals of surgical skills were acquired using the Halstedian principle of “see one, do one, teach one”; however, this outdated method presents ethical and safety issues for patients. Without underestimating its importance, it is now being swept aside by more contemporary forms of training with the introduction of simulation-based models to further master the practical and theoretical knowledge required for urolithiasis procedures. Simulation-based training not only allows assessment of technical skill, but also allows evaluation of imperative surgical qualities such as decision making, time management, and communication skills that all have a significant bearing on surgical outcomes.

This study found that among the validated VR trainers available, the majority of studies were conducted on the URO/PERC Mentor, which is, to date, the only available and most thoroughly evaluated VR simulator with the highest LoE. They both display a high level of validity with substantial evidence supporting each domain. The URO/PERC Mentor VR simulators have multiple advantages including reusability, quick set-up time, data capture, and objective performance evaluation metrics. As a result, this model has established itself as the forefront VR simulator for urolithiasis procedures; however, more evidence is required for interpretation of the scores achieved from the simulator and its implications in the workplace. There are other limitations to this model such as its cost, maintenance, and lack of real instruments.

Bench models have become the mainstay of surgical training in urolithiasis owing to their portability, reusability, and use of real instruments. Advanced bench models that allow for complex procedures are better suited for senior residents who have more expertise in the field. Basic models may be more useful to novices as an introduction to the field, allowing for effective allocation of resources. A large proportion of bench models have demonstrated its ability at accurately resemble the surgical environment. It also displays qualities in correlating with theoretical relationships of skill, time, and experience and consequently makes it a useful teaching tool for novices.

The Uro-Scopic Trainer and CREST KUB model demonstrated the highest LoE among the bench models. Of the validated models, the Adult Ureteroscopy Trainer (Ideal Anatomic Modeling) and Endo-Urologie Modell (Karl Storz, Tuttlingen, Germany) do not seem to be commercially available any longer. PCNL models displayed the least amount of evidence and future studies ought to focus assessment of validity evidence. The majority of the available models are of high fidelity and therefore expensive, nevertheless significantly cheaper than the URO Mentor.

In contrast, studies evaluating animal models have been limited. The principal benefits of animal models are their similarity to human anatomy, and the ability to get tactile feedback. These qualities befit the use of these models for advanced procedural training. However, the use of animals presents potential ethical issues and consequently affects their availability for training. At present, the only well-validated model is ex vivo porcine kidney-ureter tissue, which has shown to be very useful and effective for URS.^16,20,22 Studies supporting its use also scored well with its internal structure and response processes both achieving a score of 2, highlighting the quality of evidence presented, and its reproducibility, respectively.

Although cadavers have long been utilized for surgical training, there is poor evidence supporting their use. Most conducted studies are of either low participant size or constitute evaluation surveys. Recently, it is now thought that TECs have a greater degree of realism in contrast to FFCs, but is yet to be comprehensively compared in studies. Nevertheless, participants and experts alike both highly recommend cadaveric training in the form of masterclasses.

In light of the current evidence, we recommend that simulation-based urolithiasis training (Fig. 2) for each procedure may be initiated with VR training using the URO/PERC Mentor, as it is the modality that least replicates procedures for the OR setting but may prove to be beneficial in gaining familiarity of concepts, instruments, and steps of the procedures. This can be followed with bench models with good LoR, using real surgical instruments and irrigation. Animal tissue models such as porcine kidney-ureter should be at the next phase of training owing to its resemblance with human anatomy. Moreover, nontechnical skills training using the validated full immersion simulation training should be incorporated alongside selected models, as recommended by Brunckhorst and colleagues⁵⁸ The final stage of procedural training should constitute masterclasses with live porcine and/or human cadavers, because of associated costs and ethics in gaining access to such facilities.¹⁹

FIG. 2.

Figure illustrating simulation models and the recommended stepwise training approach. Color images are available online.

The majority of simulation studies have been conducted on a small number of participants with poor design and thus the field requires further research to critically compare simulators with reasonable LoRs against each other. Furthermore, longer follow-up periods must take place post-training to assess whether there is genuine skill development and retention. This will also reveal its transferability in the OR and will reflect whether they affect patient outcomes. Similarly, it is important that available models should each be analyzed to identify strengths and weaknesses to determine their best area of use for incorporation within formal curricula. Furthermore, future studies should focus on recruiting residents in-training, as they will ultimately be at the receiving end of such training programs.

There is also a growing literature base surrounding the use of objective assessment tools to evaluate manual dexterity using motion-tracking devices.¹¹² These have been extensively validated as an accurate motion-tracking device for evaluation of surgical dexterity and could be used alongside established assessment tools such as GRS and OSATS. In addition, substantiated by the work of Van Herzeele and colleagues,¹¹³ cognitive training for urolithiasis procedures should be further explored for incorporation into training curricula. Its application to urolithiasis procedures must be evaluated and its integration into curricula alongside recommended models should be considered.

There was a clear lack of simulation data utilization for deciding progression in training and for use in selection criteria for surgical applicants. These are two areas with great potential for simulator data incorporation to prioritize higher ranking applicants, making it a great resource for program directors, managers, admission officers, and researchers alike.

This review accounted for a range of urolithiasis models, directing the scope of further research by giving each model an impartial retrospective score for their LoR, emphasizing their potential for clinical use in addition to evaluating studies using the updated definition of validity. To our knowledge, these are unique aspects of this study, which is a clear strength and separates itself from similar reviews performed in the past.

Limitations

The absence of training time data in many studies for participants was a clear limitation identified. Publication bias may have also skewed the validation studies of the simulators. The exclusion of non-English language articles, articles outside 2000 to 2018 and lack of randomization may also suggest evidence of selection bias to our analysis. Similarly, although systematic, the assignment of LoE and LoR was a subjective process and human error may have occurred. Another important limitation is that there are clearly a low number of RCTs assessing the validity evidence of simulators, most with few participants recruited. Furthermore, it is unclear from the current literature how skills obtained from a model-based training system are transferred into real-life surgical procedures; thus, the number of cases to reach competence is yet to be defined.¹¹⁴

Conclusion

This study has highlighted the models that are available in this field and the evidence supporting their integration in training in urolithiasis. It is recommended for surgical trainers to utilize a combination of models mentioned in this study in a modular curriculum with preference given to models with the highest LoE and LoR. Further research is needed surrounding the transferability of skills through high-quality RCTs and the use of well-powered studies. Long-term follow-up studies are essential to establish whether these simulators genuinely retain the skills gained through simulator use and whether this helps to reduce surgical complications in the future.

Footnotes

Acknowledgments

The authors acknowledge support from The Urology Foundation, NIHR Biomedical Research Centre, MRC Centre for Transplantation, King's Health Partners, Guy's and St. Thomas' Charity, School of Surgery, Health Education England, Royal College of Surgeons of England, Olympus, The Pelican Group, Technology Strategy Board, and The Vattikuti Foundation.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was received for this study.

Abbreviations Used

References

Pietropaolo

, Proietti

, Geraghty

, et al. Trends of ‘urolithiasis: Interventions, simulation, and laser technology’ over the last 16 years (2000–2015) as published in the literature (PubMed): A systematic review from European section of Uro-technology (ESUT). World J Urol, 2017; 35:1651–1658.

Auge

, Preminger

. Surgical management of urolithiasis. Endocrinol Metab Clin North Am, 2002; 31:1065–1082.

van der Poel

, Brinkman

, van Cleynenbreugel

, et al. Training in minimally invasive surgery in urology: European Association of Urology/International Consultation of Urological Diseases consultation. BJU Int, 2016; 117:515–530.

Aydin

, Shafi

, Shamim Khan

, Dasgupta

, Ahmed

. Current status of simulation and training models in urological surgery: A systematic review. J Urol, 2016; 196:312–320.

McGaghie

, Issenberg

, Petrusa

, Scalese

. A critical review of simulation-based medical education research: 2003–2009. Med Educ, 2010; 44:50–63.

Noureldin

, Lee

, McDougall

, Sweet

. Competency-based training and simulation: Making a “Valid” argument. J Endourol, 2018; 32:84–93.

Liberati

, Altman

, Tetzlaff

, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. J Clin Epidemiol, 2009; 62:e1–e34.

Chien

, Khan

, Siassakos

. Registration of systematic reviews: PROSPERO. BJOG, 2012:119:903–905.

McGaghie

, Issenberg

, Petrusa

, Scalese

. A critical review of simulation-based medical education research: 2003–2009. Med Educ, 2010; 44:50–63.

10.

Howick

, Chalmers

, Glasziou

, et al. OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence.” 2011 [Cited January 16, 2019]. Available from: www.cebm.net/index.aspx?o=5653 (accessed 16 January 2019).

11.

Gettman

, Le

, Rangel

, Slezak

, Bergstralh

, Krambeck

. Development of a standardized curriculum for teaching cystoscopic skills using a computer-based endourologic simulator. Simul Healthc, 2009; 4:92–97.

12.

Schout

, Muijtjens

, Hendrikx

, et al. Acquisition of flexible cystoscopy skills on a virtual reality simulator by experts and novices. BJU Int, 2010; 105:234–239.

13.

Schout

, Ananias

, Bemelmans

, et al. Transfer of cysto-urethroscopy skills from a virtual-reality simulator to the operating room: A randomized controlled trial. BJU Int, 2010; 106:226–231.

14.

Persoon

, Schout

, Muijtjens

, Hendrikx

, Witjes

, Scherpbier

. The effect of a low-fidelity model on cystoscopic skill training: A single-blinded randomized controlled trial. Simul Healthc, 2010; 5:213–218.

15.

Grimsby

, Andrews

, Castle

, Wolter

, Patel

, Humphreys

. Urologic surgical simulation: An endoscopic bladder model. Simul Healthc, 2011; 6:352–355.

16.

Soria

, Morcillo

, Sanz

, Budia

, Serrano

, Sanchez-Margallo

. Description and validation of realistic and structured endourology training model. Am J Clin Exp Urol, 2014; 2:258–265.

17.

Hammond

, Ketchum

, Schwartz

. Accreditation council on graduate medical education technical skills competency compliance: Urologic surgical skills. J Am Coll Surg, 2005; 201:454–457.

18.

Bowling

, Greer

, Bryant

, et al. Testing and validation of a low-cost cystoscopy teaching model: A randomized controlled trial. Obstet Gynecol, 2010; 116:85–91.

19.

Ahmed

, Aydin

, Dasgupta

, Khan

, McCabe

. A novel cadaveric simulation program in urology. J Surg Educ, 2015; 72:556–565.

20.

Soria

, Morcillo

, Serrano

, et al. Development and validation of a novel skills training model for retrograde intrarenal surgery. J Endourol, 2015; 29:1276–1281.

21.

Matsumoto

, Pace

, D'A Honey

. Virtual reality ureteroscopy simulator as a valid tool for assessing endourological skills. Int J Urol, 2006; 13:896–901.

22.

, Liu

, Wang

. Flexible ureteroscopy training for surgeons using isolated porcine kidneys in vitro. BMC Urol, 2015; 15:71.

23.

, Feng

, Wang

, et al. Ureteroscopy and cystoscopy training: Comparison between transparent and non-transparent simulators. BMC Med Educ, 2015; 15:93.

24.

Matsumoto

, Kondraske

, Ogan

, et al. Assessment of basic human performance resources predicts performance of ureteroscopy. Am J Surg, 2006; 191:817–820.

25.

Wilhelm

, Ogan

, Roehrborn

, Cadeddu

, Pearle

. Assessment of basic endoscopic performance using a virtual reality simulator. J Am Coll Surg, 2002; 195:675–681.

26.

Dolmans

, Schout

, de Beer

, Bemelmans

, Scherpbier

, Hendrikx

. The virtual reality endourologic simulator is realistic and useful for educational purposes. J Endourol, 2009; 23:1175–1181.

27.

Bele

, Kelc

. Upper and lower urinary tract endoscopy training on Thiel-embalmed Cadavers. Urology, 2016; 93:27–32.

28.

Mains

, Tang

, Golabek

, et al. Ureterorenoscopy training on cadavers embalmed by Thiel's method: Simulation or a further step towards reality? Initial report. Cent Eur J Urol, 2017; 70:81–87.

29.

Shah

, Montgomery

, Langley

, Darzi

. Validation of a flexible cystoscopy course. BJU Int, 2002; 90:833–835.

30.

Shah

, Darzi

. Virtual reality flexible cystoscopy: A validation study. BJU Int, 2002; 90:828–832.

31.

Gettman

, Le

, Rangel

, Slezak

, Bergstralh

, Krambeck

. Analysis of a computer based simulator as an educational tool for cystoscopy: Subjective and objective results. J Urol, 2008; 179:267–271.

32.

Shamim Khan

, Ahmed

, Gavazzi

, et al. Development and implementation of centralized simulation training: Evaluation of feasibility, acceptability and construct validity. BJU Int, 2013; 111:518–523.

33.

Zhang

, Liu

, Wang

, Yu

, Zhu

, Na

. Effectiveness of the UroMentor virtual reality simulator in the skill acquisition of flexible cystoscopy. Chin Med J (Engl), 2013; 126:2079–2082.

34.

Kruck

, Horstmann

, Gakis

, et al. 130 Virtual Tur simulator training for novice urologists—First-time transurethral resection (Tur) of the Bladder. Eur Urol Suppl, 2009; 8:153.

35.

Reich

, Noll

, Gratzke

, et al. High-level virtual reality simulator for endourologic procedures of lower urinary tract. Urology, 2006; 67:1144–1148.

36.

Manyak

, Santangelo

, Hahn

, et al. Virtual reality surgical simulation for lower urinary tract endoscopy and procedures. JEndourol Endourol Soc, 2002; 16:185–190.

37.

Soltani

, Hidas

, Kelly

, et al. Endoscopic correction of vesicoureteral reflux simulator curriculum as an effective teaching tool: Pilot study. J Pediatr Urol, 2016; 12:45.e1–e6.

38.

Healy

, Rai

, Biyani

, Eisma

, Soames

, Nabi

. Thiel embalming method for cadaver preservation: A review of new training model for urologic skills training. Urology, 2015; 85:499–504.

39.

Matsumoto

, Pace

, D'A Honey

. Virtual reality ureteroscopy simulator as a valid tool for assessing endourological skills. Int J Urol, 2006; 13:896–901.

40.

White

, Dehaan

, Stephens

, Maes

, Maatman

. Validation of a high fidelity adult ureteroscopy and renoscopy simulator. J Urol, 2010; 183:673–677.

41.

Mishra

, Sharma

, Kumar

, Ganatra

, Sabnis

, Desai

. Comparative performance of high-fidelity training models for flexible ureteroscopy: Are all models effective?. Indian J Urol, 2011; 27:451–456.

42.

Matsumoto

, Hamstra

, Radomski

, Cusimano

. The effect of bench model fidelity on endourological skills: A randomized controlled study. J Urol, 2002; 167:1243–1247.

43.

Inoue

, Okada

, Hamamoto

, Matsuda

. New advanced bench model for flexible ureteroscopic training: The Smart Simulator. J Endourol, 2018; 32:22–27.

44.

Brehmer

, Tolley

. Validation of a bench model for endoscopic surgery in the upper urinary tract. Eur Urol, 2002; 42:175–179.

45.

Brehmer

, Swartz

. Training on bench models improves dexterity in ureteroscopy. Eur Urol, 2005; 48:458–463.

46.

Al-Jabir

, Aydin

, Abe

, et al. Validation of the advanced scope trainer for flexible ureterorenoscopy training. Urology, 2017; 110:45–50.

47.

Kishore

, Pedro

, Monga

, Sweet

. Assessment of validity of an OSATS for cystoscopic and ureteroscopic cognitive and psychomotor skills. J Endourol, 2008; 22:2707–2711.

48.

Argun

, Chrouser

, Chauhan

, et al. Multi-institutional validation of an OSATS for the assessment of cystoscopic and ureteroscopic skills. J Urol, 2015; 194:1098–1105.

49.

Sweet

. The CREST simulation development process: Training the next generation. J Endourol, 2017; 31:S69–S75.

50.

Blankstein

, Lantz AG

DAH, Pace

, Ordon

, Lee

. Simulation-based flexible ureteroscopy training using a novel ureteroscopy part-task trainer. Can Urol Assoc J, 2015; 9:331–335.

51.

Lantz

, Blankstein

, Honey

RJD

, Ordon

, Pace

, Lee

. Pd6-06 validation of a novel inanimate ureteroscopy training model and a simulation-based flexible ureteroscopy training course. J Urol, 2014; 191:E132.

52.

Villa

, Somani

, Sener

, et al. Comprehensive flexible ureteroscopy (FURS) simulator for training in endourology: The K-box model. Cent Eur J Urol, 2016; 69:118–120.

53.

Villa

, Sener

, Somani

, et al. Initial content validation results of a new simulation model for flexible ureteroscopy: The Key-Box. J Endourol, 2017; 31:72–77.

54.

Proietti

, Dragos

, Emiliani

, et al. Ureteroscopic skills with and without Roboflex Avicenna in the K-box((R)) simulator. Cent Eur J Urol, 2017; 70:76–80.

55.

Veneziano

, Ahmed

, Van Cleynenbreugel

, et al. Development methodology of the novel Endoscopic stone treatment step 1 (EST s1) training/assessment curriculum. J Endourol, 2017; 31. DOI: 10.1089/end.2017.0248.

56.

Matsumoto

, Hamstra

, Radomski

, Cusimano

. A novel approach to endourological training: Training at the Surgical Skills Center. J Urol, 2001; 166:1261–1266.

57.

Chou

, Abdelshehid

, Clayman

, McDougall

. Comparison of results of virtual-reality simulator and training model for basic ureteroscopy training. J Endourol, 2006; 20:266–271.

58.

Brunckhorst

, Shahid

, Aydin

, et al. Simulation-based ureteroscopy skills training curriculum with integration of technical and non-technical skills: A randomised controlled trial. Surg Endosc, 2015; 29:2728–2735.

59.

Brunckhorst

, Shahid

, Aydin

, et al. The relationship between technical and nontechnical skills within a simulation-based ureteroscopy training environment. J Surg Educ, 2015; 72:1039–1044.

60.

Watterson

, Beiko

, Kuan

, Denstedt

. Randomized prospective blinded study validating acquistion of ureteroscopy skills using computer based virtual reality endourological simulator. J Urol, 2002; 168:1928–1932.

61.

Knoll

, Trojan

, Hatzinger

, Alken

, Kohrmann K

. M. Ureterorenoscopy: Impact of simulator-training on clinical skills. Eur Urol Suppl, 2003; 2:199.

62.

Knoll

, Trojan

, Haecker

, Alken

, Michel

. Validation of computer-based training in ureterorenoscopy. BJU Int, 2005; 95:1276–1279.

63.

Jacomides

, Ogan

, Cadeddu

, Pearle

. Use of a virtual reality simulator for ureteroscopy training. J Urol, 2004; 171:320–323.

64.

Cai

, Zhang

, Sun

, Li

, Yuan

, Na

. Proficiency of virtual reality simulator training in flexible retrograde ureteroscopy renal stone management. Chin Med J (Engl), 2013; 126:3940–3943.

65.

Ogan

, Jacomides

, Shulman

, Roehrborn

, Cadeddu

, Pearle

. Virtual ureteroscopy predicts ureteroscopic proficiency of medical students on a cadaver. J Urol, 2004; 172:667–671.

66.

Aloosh

, Noureldin

, Andonian

. Transfer of flexible ureteroscopic stone-extraction skill from a virtual reality simulator to the operating theatre: A pilot study. J Endourol, 2016; 30:1120–1125.

67.

Aloosh

, Couture

, Andonian

. Mp51-06 assessment of flexible ureteroscopic stone extraction skills of urology postgraduate trainees during objective structured clinical examinations: Is there a place for virtual reality simulators?. J Urol, 2017; 197:E695.

68.

Biyani

, Hanchanale

, Rajpal

, et al. First urology simulation boot camp in the United Kingdom. Afr J Urol, 2017; 23:258–267.

69.

Aloosh

, Couture

, Fahmy

, Elhilali

, Andonian

. Assessment of urology postgraduate trainees' competencies in flexible ureteroscopic stone extraction. Can Urol Assoc J, 2018; 12:52–58.

70.

Salvado

, Oyanedel

, Sepulveda

, et al. Validation of a high-fidelity model in ureteroscopy incorporating hand motion analysis. Int Urol Nephrol, 2015; 47:1265–1269.

71.

Strohmaier

, Giese

. Porcine urinary tract as a training model for ureteroscopy. Urol Int, 2001; 66:30–32.

72.

Liske

, Zimmermanns

, Lahme

. 1820 New Modified training model for upper urinary tract endoscopy—Using porcine urinary tract. J Urol, 2013; 189:E748.

73.

Huri

, Skolarikos

, Tatar

, et al. Simulation of RIRS in soft cadavers: A novel training model by the Cadaveric Research On Endourology Training (CRET) Study Group. World J Urol, 2016; 34:741–746.

74.

Rai

, Healy

, Raslan

, Tang

, Nabi

. Face validity study of Cadavers using Thiel method of embalming for endoscopic surgery in urology. Br J Surg, 2015; 102:210.

75.

Knudsen

, Matsumoto

, Chew

, et al. A randomized, controlled, prospective study validating the acquisition of percutaneous renal collecting system access skills using a computer based hybrid virtual reality surgical simulator: Phase I. J Urol, 2006; 176:2173–2178.

76.

Ganpule

, Mishra

, Sabnis

, Muthu

, Desai

. Evaluation and validation of virtual reality (Vr) based simulation to develop endourological percutaneous renal access technique for urological trainees. J Urol, 2009; 181:491–492.

77.

Jagtap

. 1333 Surgical skills lab for percutaneous renal access training: Content validation comparison between live porcine and Vr simulation model. J Urol, 2010; 183:E515.

78.

Patel

, El-Husseiny

, Moraitis

, et al. 1332 Assessing and developing percutaneous renal access skills of trainees using the state of the Art Perc Mentor™ simulation trainer. J Urol, 2010; 183:E514.

79.

Papatsoris

, Shaikh

, Patel

, et al. Use of a virtual reality simulator to improve percutaneous renal access skills: A prospective study in urology trainees. Urol Int, 2012; 89:185–190.

80.

Mishra

, Kurien

, Patel

, et al. Validation of virtual reality simulation for percutaneous renal access training. J Endourol Endourol Soc, 2010; 24:635–640.

81.

Mishra

, Kurien

, Ganpule

, Muthu

, Sabnis

, Desai

. Percutaneous renal access training: Content validation comparison between a live porcine and a virtual reality (VR) simulation model. BJU Int, 2010; 106:1753–1756.

82.

Wignall

, Denstedt

, Preminger

, et al. Surgical simulation: A urological perspective. J Urol, 2008; 179:1690–1699.

83.

Hoznek

, Salomon

, de la Taille

, et al. Simulation training in video-assisted urologic surgery. Curr Urol Rep, 2006; 7:107–113.

84.

Maldonado-Alcaraz

, Morena-Palacois

, Serrano-Brambila

, Rodriguez-Silverio

, Torres-Mercade

, Lopez-Samano

. Use of a novel radiation-free fluoroscopy emulator (iPERC) to improve surgical skills in percutaneous nephrolithotomy. Eur Urol Suppl, 2015; 14:eV2.

85.

Maldonado-Alcaraz

, Gonzalez-Meza Garcia

, Serrano-Brambila

. Evaluation of 2 inanimate models to improve percutaneous fluoroscopy-guided renal access time. Cir Cir, 2015; 83:402–408.

86.

Anguiano

JRT

, Alcaraz

, Palacios

, et al. Training in a novel fluoroscopy simulator (Iperc) reduces emulated radiation time associated to puncture and improves kidney access time in percutaneous nephrolithotomy simulation among urologists with scarce surgical experience. J Urol, 2016; 195:E661–E662.

87.

Veneziano

, Smith

, Reihsen

, Speich

, Sweet

. The SimPORTAL fluoro-less C-arm trainer: An innovative device for percutaneous kidney access. J Endourol, 2015; 29:240–245.

88.

Chrouser

, Marsh

, Sweet

. Mp20-03 Percutaneous access skill improvement after the Aua PCNL training course using the simportal fluoro-less C-Arm trainer. J Urol, 2016; 195:E213.

89.

Noureldin

, Hoenig

, Zhao

, et al. Incorporation of the fluoroless C-Arm Trainer at the American Urological Association hands on training percutaneous renal access. World J Urol, 2018; 36:1149–1155.

90.

Bruyere

, Leroux

, Brunereau

, Lermusiaux

. Rapid prototyping model for percutaneous nephrolithotomy training. J Endourol, 2008; 22:91–96.

91.

Ghazi

, Erturk

. Improvement of surgical outcomes after preoperative rehearsal using 3D printed patient specific simulation for percutaneous nephrolithotomy (PCNL). J Endourol, 2017; 31:A419–A420.

92.

, Chen

, Liu

, Li

, Xu

, Bao

. Construction of a three-dimensional model of renal stones: Comprehensive planning for percutaneous nephrolithotomy and assistance in surgery. World J Urol, 2013; 31:1587–1592.

93.

Tawfik

, El-Abd

, El-Enen

, et al. Validity of a sponge trainer as a simple training model for percutaneous renal access. Arab J Urol, 2017; 15:204–210.

94.

Ghazi

, Campbell

, Melnyk

, et al. Validation of a full-immersion simulation platform for percutaneous nephrolithotomy using three-dimensional printing technology. J Endourol, 2017; 31:1314–1320.

95.

Filippou

, Odisho

, Ramaswamy

, et al. Using an abdominal phantom to teach urology residentes ultrasound-guided percutaneous needle placement. Int Braz J Urol, 2016; 42:717–726.

96.

Bozzini

, Provenzano

, Seveso

, et al. Mp20-01 realising a new model to train residents in nephrostomy tube placement and PCNL. J Urol, 2016; 195:E212.

97.

Ungi

, Sargent

, Moult

, et al. Perk Tutor: An open-source training platform for ultrasound-guided needle insertions. IEEE Trans Biomed Eng, 2012; 59:3475–3481.

98.

Fuoco

, Ungi

, Siemens

, Fichtinger

, Beiko

. Pd36-05 percutaneous nephrostomy for dummies: Electromagnetic needle guidance with tracked ultrasound snapshots in a simulation model. J Urol, 2014; 191:E903.

99.

Zhang

, Ou

, Jia

, et al. Novel biologic model for percutaneous renal surgery learning and training in the laboratory. Urology, 2008; 72:513–516.

100.

Earp

. Percutaneous renal surgery—New model for learning and training. Int Braz J Urol, 2003; 29:151–154.

101.

Qiu

, Yang

, Zhang

, Sun

. Modified biological training model for percutaneous renal surgery with ultrasound and fluroscopy guidance. Chin Med J (Engl), 2011; 124:1286–1289.

102.

Klein

, Rassweiler

, Rassweiler-Seyfried

. Validation of a novel cost effective easy to produce and durable in vitro model for kidney-puncture and percutaneous nephrolitholapaxy-simulation. J Endourol, 2018; 32:871–876.

103.

Imkamp

, von Klot

, Nagele

, Herrmann

. New ex-vivo organ model for percutaneous renal surgery. Int Braz J Urol, 2011; 37:388–394.

104.

Jutzi

, Imkamp

, Kuczyk

, Walcher

, Nagele

, Herrmann

. New ex vivo organ model for percutaneous renal surgery using a laparoendoscopic training box: The sandwich model. World J Urol, 2014; 32:783–789.

105.

Hammond

, Ketchum

, Schwartz

. A new approach to urology training: A laboratory model for percutaneous nephrolithotomy. J Urol, 2004; 172:1950–1952.

106.

Hacker

, Wendt-Nordahl

, Honeck

, Michel

, Alken

, Knoll

. A biological model to teach percutaneous nephrolithotomy technique with ultrasound- and fluoroscopy-guided access. J Endourol, 2007; 21:545–550.

107.

Sinha

, Krishnamoorthy

. Use of a vegetable model as a training tool for PCNL puncture. Indian J Urol, 2015; 31:156–159.

108.

Karagozlu Akgul

, Unal

, Demirbas

, et al. A simple, non-biological model for percutaneous renal access training. Urol J, 2018; 15:1–5.

109.

Zhang

, Yu

, Jin

, Li

, Na

. Validation of a novel non-biological bench model for the training of percutaneous renal access. Int Braz J Urol, 2014; 40:87–92.

110.

Noureldin

, Fahmy

, Anidjar

, Andonian

. Is there a place for virtual reality simulators in assessment of competency in percutaneous renal access?. World J Urol, 2016; 34:733–739.

111.

Strohmaier

, Giese

. Ex vivo training model for percutaneous renal surgery. Urol Res, 2005; 33:191–193.

112.

Dosis

, Aggarwal

, Bello

, et al. Synchronized video and motion analysis for the assessment of procedures in the operating theater. Arch Surg, 2005; 140:293–299.

113.

Van Herzeele

, Aggarwal

, Neequaye

, Darzi

, Vermassen

, Cheshire

. Cognitive training improves clinically relevant outcomes during simulated endovascular procedures. J Vasc Surg, 2008; 48:1223–1230, 30.e1.

114.

Aydin

, Ahmed

, Van Hemelrijck

, et al. Simulation in Urological Training and Education (SIMULATE): Protocol and curriculum development of the first multicentre international randomized controlled trial assessing the transferability of simulation-based surgical training. BJU Int, 2020; 126:202–211.