Development Methodology of the Novel Endoscopic Stone Treatment Step 1 Training/Assessment Curriculum: An International Collaborative Work by European Association of Urology Sections

Abstract

Background:

Simulation-based technical skill assessment is a core topic of debate, especially in high-risk environments. After the introduction of the E-BLUS (European Basic Laparoscopic Urological Skills) exam for basic laparoscopy, no more technical training/assessment urological protocols have been developed in Europe.

Objective:

We describe the methodology used in the development of the novel Endoscopic Stone Treatment step 1 (EST s1) assessment curriculum.

Materials and Methods:

The “full life cycle curriculum development” template was followed for curriculum development. A cognitive task analysis was run to define the most important steps and details of retrograde intrarenal surgery, in accordance with European Association of Urology (EAU) Urolithiasis guidelines. Training tasks were created between April 2015 and September 2015. Tasks and metrics were further analyzed by a consensus meeting with the European Section of Urolithiasis (EULIS) board in February 2016. A review, aimed to study available simulators and their accordance with task requirements, was subsequently run in London in March 2016. After initial feedback and further tests, content validity of this protocol was achieved during European Urology Residents Education Programme (EUREP) 2016.

Results:

The EST s1 curriculum development, took 23 months. Seventy-two participants tested the five preliminary tasks during EUREP 2015, with sessions of 45 minutes each. Likert-scale questionnaires were filled out to score the quality of training. The protocol was modified accordingly and 25 participants tested the four tasks during the hands-on training sessions of the European Section of Uro-Technology (ESUT) 2016 congress. One hundred thirty-four participants finally participated in the validation study in EUREP 2016. During the same event, 10 experts confirmed content validity by filling out a Likert-scale questionnaire.

Conclusion:

We described a reliable and replicable methodology that can be followed to develop training/assessment protocols for surgical procedures. The expert consensus meetings, strict adherence to guidelines, and updated literature search toward an Endourology curriculum allowed correct training and assessment protocol development. It is the first step toward standardized simulation training in Endourology with a potential for worldwide adoption.

Introduction

Simulation-based technical skill assessment has been a core topic of debate, especially in high-risk environments. It was introduced in the aviation field in the early 30s with the “Link Trainer.”¹ Simulation training allows for safe and methodological acquisition of skills necessary for trainees, thereby also shortening their learning curve in this process. The first urology-specific curriculum came out in 2011, when the basic laparoscopic skills (E-BLUS [European Basic Laparoscopic Urological Skills])² exam was delivered for the first time by the European Association of Urology (EAU) and subsequently adopted worldwide. After the introduction of the E-BLUS exam, which was heavily inspired by the fundamentals of laparoscopic surgery study carried out by the American Society of Gastroenterological Surgeons (SAGES),³ no more technical training/assessment urological protocols have been developed in Europe. With a recent surge in Endourology and stone treatment, a development of dedicated protocol-based training and assessment relating to endourological techniques was essential. Although an OSATS-based (Objective Structured Assessment of Technical Skills) assessment tool for ureteroscopy (URS) skills has been already described,⁴ no structured training curriculum has yet been developed.

Objective

Aim of this article is to describe the methodology we followed to develop the Endoscopic Stone Treatment step 1 (EST s1), the first step of a novel modular training/assessment curriculum for EST. Our goals were to develop a set of replicable, standardized low-cost exercises, providing objective assessment and applicability to 45–60 minutes hands-on training sessions. Given the structured pathway followed, our process could work as an example for developing new training protocols with highly reliable methodology and evidence.

Materials and Methods

The “full life cycle curriculum development” template (Table 1), described by Richard Satava, was followed for curriculum development. As described by the author,⁵ the process starts by defining the outcomes and metrics and ends with the certification that the outcomes, as planned at the beginning, have been properly achieved. The name “full life cycle” derives straight from this concept. Our whole development process started in November 2014 and is today at the stage of validation.

Table 1.

Full Life Cycle Curriculum Template by Richard Satava, MD

WHAT	Outcomes and metrics	Curriculum development	Simulator development	Validation studies	Implement	Issue certification
					Survey training certification
How	Consensus conference	Standard curriculum template	Engineering physical simulator	Standard validation template	Current procedures	Issue mandates and certificates
EST s1	November 2014 YAUWP CTA	April 2015 cognitive/HoT development	March 2016 simulator review	September 2016 validation EUREP16	March 2017 final tests EAU17	September 2017 first exams EUREP15
	February 2015 ESUT meeting Davos	September 2015 preliminary test EUREP15	June 2016 simulator adaptation	December 2016 EULIS consensus	2017 ART in flexible
		February 2016 EULIS consensus	July 2016 simulator test ESUT16

CTA = cognitive task analysis; EAU = European Association of Urology; EST s1 = Endoscopic Stone Treatment step 1; ESUT = European Section of Uro-Technology; EULIS = European Section of Urolithiasis; EUREP = European Urology Residents Education Programme; YAUWP = Young Academic Urologist Working Party.

Phase 1: outcomes and metrics

Before defining outcomes and metrics as described by the template, a cognitive task analysis (CTA)^6,7 was run by the Endourology and Stone Treatment group of the Young Academic Urologist Working Party (YAUWP).

The CTA, as described by Clark and Estes, aims at defining the most important steps and details of a procedure and is the most appropriate way to objectively analyze it. In this case, its goal was to study the details of retrograde intrarenal surgery (RIRS), so CTA was run in accordance with EAU Urolithiasis guidelines⁸ and in parallel with a focused updated literature review. Preoperative, procedural continuous variables and completion details were analyzed in relationship to each phase of the technique: cystoscopy, URS, and stone fragmentation. Furthermore, indications, contraindications, equipment, patient positioning, procedural steps, and “dos and don'ts” were described in relationship to each of the aforementioned procedural phases.

The CTA led to a complete description of the procedure, as suggested by the EAU guidelines, and defined its main steps, as depicted in Table 2. Given the detailed procedural steps, we defined a modular training system, as described in Figure 1: basic skills with focus on navigation and basic use of the operative channels, intermediate skills with focus on lithotripsy techniques, and advanced skills with focus on complete endoscopic lithotripsy procedures. This protocol underwent a first consensus by Delphi method with European Section of Uro-Technology (ESUT) experts in February 2015.

FIG. 1.

Modular hands-on training template for endoscopic stone treatment.

Table 2.

The Main Steps of the Retrograde Intrarenal Surgery Procedure Defined by the Cognitive Task Analysis Process

RIRS procedural steps
Preoperatory phase	Continuous variables procedural phase
1. Check patient-related details (correct patient, indication, allergies, culture, antibiotics, anticoagulants)	1. Change of instruments during procedure
2. Check material-related details (equipment up to date, presence materials, settings diathermy, irrigation fluid)	2. Orientation in the bladder
3. Patient preparation (positioning, disinfection, sterile drapes)	3. Regulation of irrigation and emptying bladder
	4. Maintenance of visibility
	5. Intrarenal pressure awareness
	6. Identification of urothelial injuries
Procedural phase (RIRS)	Completion phase
1. Assembling instruments and connecting tubes	1. Documentation of stone-free status/residual fragments
2. Adjustment of light settings, focus camera, white balance	2. Removal of instruments
3. Instillation of lubricant into meatus and introduction of the cystoscope	3. Ensuring bladder emptying
4. Inspection of bladder, including orientation, identification of orifices, and eventual bladder tumors
5. Insertion of ureteral catheter	4. Debriefing (check count materials and stone specimen/biopsy, discussion complications, and postoperative policy)
6. Retrograde pyelography	6. Registration (operating report, eventual pathology file, patient file, financial registration)
7. Ureteral mapping (identification of stones/strictures/filling defects)
8. X-ray-guided placement of the guidewire (safety guidewire, X-ray safety precautions)
9. Semirigid URS (inspection of the ureter)
10a. In case of a ureteral stone, proceed with fragmentation (laser/ballistics)
10b. In case of a renal stone, placement of the working guidewire (through the semirigid ureteroscope)
11. Placement of the access sheath (choosing the optimal size and length)
12. Insertion of the flexible ureteroscope
13. Inspection of calices
14a. Insertion of the laser fiber (laser safety precautions, e.g., Glasses)
14b. Introduction of biopsy forceps in case of suspected lesions
15. Stone fragmentation/dusting and basketing as appropriate
16. Double check under fluoroscopy and endoscopy for residual fragments
17. Stent placement if clinically indicated

Roman type indicates skills to be included in the following EST s2 protocol. Bold type indicates skills relevant to the EST s1 protocol development. Italic type indicates theoretical skills.

RIRS = retrograde intrarenal surgery; URS = ureteroscopy.

Phase 2: curriculum development

This phase was carried out in strict accordance with the CTA and was aimed to help the development of step 1 of the curriculum (basic skills). Training tasks were created from April 2015, until September 2015. “Indication and contraindication” information were used to structure the theoretical part of the trials, “equipment” was useful to exactly define the correct tools for each exercise, and “patient positioning” and “procedure steps” were noted to define simulator requirements, whereas “dos and don'ts” were critical to define errors and metrics for each training task. The first set of exercises was composed as follows: task 1, flexible cystoscopy; task 2, safety guidewire placement with rigid cystoscope; task 3, rigid URS and working guidewire placement; task 4, access sheath placement; and task 5, flexible ureterorenoscopy. After a preliminary definition of the tasks, these underwent the first test during “European Urology Residents Education Programme (EUREP)” 2015. The aims of this test were to collect tutor feedback about the possibility of completing the defined protocol during a standard 45 minute hands-on training session. The preliminary hands-on training step-1 protocol was delivered 72 times on four stations, with four expert tutors. No major issue was reported during the preliminary test. Likert scale-based quality feedback questionnaires were collected from the participants, with focus on several aspects of this training session (Fig. 2): duration of the session, new skills acquired, basic skills improvement, expectations, and overall evaluation of the course. Scores went from 1 (poor) to 5 (very good).

FIG. 2.

Results from the quality feedbacks collected during EUREP 2015. EUREP, European Urology Residents Education Programme.

The results were collected along with a detailed description and refinement of these tasks. The tasks and metrics were further analyzed by a consensus meeting with the “EAU Section of Urolithiasis (EULIS)” board in February 2016. Following the feedback given by experts, these were reconfigured into a total of four tasks (Table 3): task 1, flexible cystoscopy; task 2, rigid cystoscopy; task 3, semirigid URS; and task 4, flexible URS.

Table 3.

Preliminary and Final Tasks of the Endoscopic Stone Treatment Step 1 Training Curriculum

Preliminary task list	Final task list
1. Cystoscopy	1. Flexible cystoscopy
2. Safety guidewire placement with rigid scope	2. Rigid cystoscopy and safety guidewire placement
3. Rigid URS and working guidewire placement	3. Semirigid URS, working guidewire, and access sheath placement
4. Access sheath placement	4. Flexible ureteroscope navigation
5. Flexible ureterorenoscopy

Phase 3: simulator development

Based on the simulator requirements of the final task set, a review of the existing simulators was run at Guy's Hospital, London, in March 2016. This was done in full overview of the information collected with the CTA at the beginning of the whole process. The simulators available for the trial were as follows: Uro-Mentor (Simbionix, Fig. 3A), Endo-Uro Trainer (Samed, Fig. 3B), Advanced Scope Trainer (Mediskills, Fig. 3C), Uro-Scopic Trainer (Limbs and Things, Fig. 3D), K-box (Coloplast, Fig. 3E), and kidney-ureter-bladder (KUB) Model (University of Minnesota, Fig. 3F). Once all the simulators had been tested, each experienced panel member independently filled out a simulator evaluation questionnaire. Panelists were asked to score with a 5-point Likert scale quality of the simulators in relationship to different maneuvers and characteristics: anatomic resemblance, bladder visualization, instrument handling, ureteral catheterization, ureteral navigation, semirigid URS, and flexible URS. They were also given space for free-text comments. A final round table discussion was then conducted to summarize their collective thoughts and findings.

FIG. 3.

The simulators tested for training requirements: Uro-Mentor (Simbionix, A), Endo-Uro Trainer (Samed, B), Advanced Scope Trainer (Mediskills, C), Uro-Scopic Trainer (Limbs and Things, D), K-box (Coloplast, E), and KUB Model (University of Minnesota, F). KUB = kidney-ureter-bladder.

Based on the results collected, two simulators (Endo-Uro Trainer by SAMED and K-box by Coloplast) were chosen as best fitting the needs of the protocol. The chosen models underwent some slight adaptation to allow full adoption of the task rules. The silicon bladder of the Endo-Uro Trainer was added with black marker dots to be used as targets (Fig. 4). The K-box was considered in one standardized configuration (Fig. 5), to allow for rotation, in–out and up–down movements with the flexible ureteroscope.

FIG. 4.

Endoscopic view of the modified Endo-Uro Trainer simulator with the black marker balls.

FIG. 5.

Modified K-box with numbered cavities.

Phase 4: validation

The custom-modified simulators were used for the first official test hands-on training sessions in Athens during “EAU Section of Uro-Technology (ESUT)” congress 2016, where they were used for a total of 25 one-hour sessions. Feedback from tutors was collected about the reliability of the simulators. Following the suggestions given during the tests, the black marker dots were replaced with 3 mm black marker balls on the wall of the bladder models, whereas K-boxes were provided with numbered cavities, which allowed for an even more standardized task. After the described development process, preliminary validation of the protocol was run during EUREP 2016. To assess content validity of the developed protocol, 10 experts from high volume stone treatment centers were asked to fill out a Likert-score questionnaire, focused on each single task and its characteristics.

Results

The EST s1 curriculum development, from early data collection to validation, took a total of 23 months and involved EAU sections (ESUT, EULIS) and one YAUWP group, with guidance and coordination from the European School of Urology (ESU)/ESUT training group. The CTA, a detailed word document (6165 words), was structured in five paragraphs dedicated to different parts of the RIRS procedure. Based on the CTA, a list of 3 preoperative steps, 17 procedural steps, and 5 completion steps was developed, defining the critical procedural phases to be simulated. This list was used to divide basic skills from intermediate tasks and cognitive contents (Table 2). Seventy-two participants tested the five preliminary tasks during EUREP 2015, with sessions of 45 minutes each, and the tutors reported no procedural issues. Eighty-six percent of the trainees scored the training session duration as “good” or “very good.” Ninety-five percent of them stated that they acquired new skills during the session (Fig. 2), whereas 87% declared that their basic skills improved during the course. The course, based on the preliminary EST s1 curriculum, met the expectations of 92% of the participants, who scored it as overall “good” in 20% of cases and as “very good” in 77% of cases. The five-task curriculum was further optimized and after feedback, it was readjusted to four tasks. During the simulator review phase, EndoUro Trainer by SAMED and the KUB model by University of Minnesota collected the highest scores based on the set simulator requirements (Fig. 6). Due to the nature of these tasks, K-box was selected just for task 4 (flexible URS).

FIG. 6.

Results from the simulator review test.

Subsequently, the new EST s1 protocol with four tasks was tested by 25 participants during the hands-on training sessions of the ESUT 2016 congress, and 134 participants with different expertise ranging from 0 to >1000 real-life URS cases finally participated in the validation study in EUREP 2016. The 10 experts involved (average of 23.7 URS stone treatment procedures/month) scored individually for each task from the final task list (Table 4). The statement “EST s1 should be used for basic URS training & assessment” was scored with a mode of 5 (±0.3), confirming the quality of the development process. Experts suggested that the curriculum should be applied to the third year of residency (±0.8). All tasks were scored as “valid basic training tools” with a mode of 5 (task 1 ± 0.6; task 3 ± 0.5; task 4 ± 0.4) except for task 2, scored with a mode of 4 (±0.5). The experts considered all tasks as proper parts of the curriculum (mode 5).

Table 4.

Results from the Content Evaluation Performed by Experts During European Urology Residents Education Programme 2016

Question	Mode (± avg SD)
Task 1—flexible cystoscopy
It is a valid basic training tool	5 (±0.6)
It correctly represents a flex cystoscopy	4 (±0.5)
It should be part of the EST s1	5 (±0.5)
Task 2—rigid cystoscopy
It is a valid basic training tool	4 (±0.5)
It correctly represents a rigid cystoscopy	5 (±0.4)
It should be part of the EST s1	5 (±0.5)
Task 3—semirigid URS
It is a valid basic training tool	5 (±0.5)
It correctly represents a semirigid URS	4 (±0.4)
It should be part of the EST s1	5 (±0.5)
Task 4—flexible URS
It is a valid basic training tool	5 (±0.4)
It correctly represents a flex URS	5 (±0.6)
It should be part of the EST s1	5 (±0.5)
Overall rating
It should be used for basic URS training and assessment	5 (±0.3)
When would you suggest the EST s1 training/exam to be performed?
Year of residency	3 (±0.8)

Discussion

Meaning of our study

The new EST s1 protocol is the first such structured and streamlined Endourology curriculum simulation protocol, which has had collaborative input from various EAU subsections. The methodology has been done as a step-by-step replicable process following CTA, curriculum building, simulator review/development, and feedback-based protocol refinement. The result is a successful full cycle Endourology curriculum development, which has now undergone a formal validation process.

Strengths and weakness of our study

The strength of our study is the use of recognized methodology with a phased curriculum development. The CTA gave a solid background to the whole process, with a strong connection to the best achievable standards and guidelines. This allowed a detailed curriculum building, which gave the training tasks an intrinsic validity, even before the popular “validation process,” as depicted by several authors.^9,10 Indeed, even the preliminary task set tested during EUREP 2015 achieved high quality scores from the participants involved. Moreover, the process defined brought to the identification of specific simulator requirements, which lead to testing and scoring various simulators already available on the market and finally modifying them to meet the requirements of the training curriculum.

This whole process successfully allowed the clinicians to wisely choose what best fit their needs, instead of simply using the plethora of available simulators. We consider this as one of the most important goals of our work, which should be considered a priority for any curriculum development process.

In consideration of the expert involvement and the scientific background given by the CTA, we acknowledge that our basic protocol has already achieved content validity, which has been anyway confirmed by the scores given by experts during EUREP 2016. Our statement is strengthened by the concept of “basic” surgical training curriculum, which is a set of basic skills that can be easily found or replicated in any procedure of similar nature. The analogy is equivalent to the E-BLUS that contains simple skills such as cutting, bimanual dexterity, and knot tying, which are common to any laparoscopic procedure in urology, general surgery, and gynecology. The same way, EST s1 tasks focus not only on navigation and basic use of operative channels, skills that can be found in urological endoscopy, but also in gastrointestinal or gynecological endoscopy. Moving to more complex procedural steps, training becomes more specialty specific, just like laparoscopy.

For complete endourology training, technical skills with a validated curriculum need to be supplemented with theoretical background, patient-specific information, and nontechnical skills,⁷ which can help in training and assessment of communication skills in high stress or emergency situations. Cognitive information about the procedure were analyzed and collected in a dedicated theory module by the educational group of the EULIS, following the latest guidelines on this topic. The module will be integrated inside the curriculum as an addendum to the technical part. Nontechnical skills related to the EST s1 protocol have not yet been analyzed, even though their assessment can improve behavior, team work, and communication in these anxiety-driven stressful situations.^11,12 Up to now, no weaknesses have been identified in the process, which appeared to flow smoothly and in full collaboration between different entities. The data collection from EUREP16 will eventually confirm the quality of the whole study, by integrating the already achieved content validity with face and construct validity.

Area of future research

Although EST s1 is the first step in the integration of simulation in the Endourology curriculum, validation and further work with intermediate and advanced steps will be necessary for a comprehensive curriculum. Once this is established, subsequent work should involve fellowship in a recognized training program in an Endourology unit.

From the information collected with the CTA and thanks to the preliminary expert consensus meetings, we already know that the intermediate step will focus on stone treatment, whereas the advanced step will focus on full procedures and complication mastery. This work and the acquired methodology will allow for a faster development of the training protocols. Simulators enhance acquisition of skills, thereby improving surgical training. The optimal duration and level of training need to be targeted on trainee requirements and available resources. A modular training structure using low- and high-fidelity simulators that is realistic, yet cost-effective, seems to be the best model for increased uptake and worldwide adoption.

Conclusion

The process summarized in this article is a reliable and replicable methodology that can be followed to develop training/assessment protocols for any surgical procedure. The expert consensus meetings, strict adherence to guidelines, and updated literature search toward an Endourology curriculum have allowed successful achievement of content validity for the EST s1 training and assessment protocol. It is the first step in standardized simulation training in Endourology with a potential for worldwide adoption.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

Abbreviations Used

References

Link Trainer. Wikipedia: https://en.wikipedia.org/wiki/Link_Trainer (accessed 30 July 2016 ).

Brinkman

, Tjiam

, Schout

BMA

, Muijtjens

AMM

, Van Cleynenbreugel

, Koldewijn

, Witjes

. Results of the European Basic Laparoscopic Urological Skills examination. Eur Urol, 2014; 65:490–496.

Fried

GM.

FLS assessment of competency using simulated laparoscopic tasks. J Gastrointest Surg, 2008; 12:210–212.

Argun

, Chrouser

, Chauhan

, et al. Multi-institutional validation of an OSATS for the Assessment of Cystoscopic and Ureteroscopic Skills. J Urol, 2015; 194:1098–1105.

Satava

, Gallagher

. Next generation of procedural skills curriculum development: Proficiency-based progression. JHS, 2015; 3:198–205.

Clark

, Estes

. Cognitive task analysis. Int J Educ Res, 1996; 25:403–417.

Hananel

, Sweet

. Simulator development—From idea to prototype to product. In: Aggarwal

, Korndorffer

, Cannon-Bowers

, eds. ACS Principles and Practice for Simulation and Surgical Education Research. Chicago, IL: American College of Surgeons, 2015.

Türk

, Neisius

, Petrik

, Seitz

, Skolarikos

, Tepeler

, Thomas

. Guidelines Associates: Dabestani

, Drake

, Grivas

, Ruhayel

2014 publication of the EAU Urolithiasis Guidelines.

McDougall

EM.

Validation of surgical simulators. J Endourol, 2007; 21:244–247.

10.

Fried

, Feldman

, Vassiliou

, Fraser

, Stanbridge

, Ghitulescu

, Andrew

. Proving the value of simulation in laparoscopic surgery. Ann Surg, 2004; 240:518–525; discussion 525–528.

11.

Mishra

, Catchpole

, McCulloch

. The Oxford NOTECHS System: Reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care, 2009; 18:104–108.

12.

Wood

, Raison

, Haldar

, et al. Training tools for nontechnical skills for surgeons—A systematic review. J Surg Educ, 2017; 74:548–578.