Abstract
Objective:
To develop and validate a high-fidelity, nonbiohazardous simulator model for the ultrasound-guided percutaneous nephrolithotomy procedure.
Methods:
We employed a systematic framework based on Delphi consensus and modern education theory to design a simulation model. Twelve expert surgeons provided input through a hierarchal task analysis and identified procedural tasks, anatomical landmarks, and potential errors. These were translated into engineering deliverables by a team of biomedical engineers and surgical educators. A prototype was developed using three-dimensional printing and hydrogel molding, followed by expert validation through recorded simulations and subsequent multicenter trails with 48 participants.
Results:
A hydrogel prototype with realistic anatomical features was created using results from the Delphi process. It received positive feedback in areas such as anatomy, procedural fidelity, and education effectiveness, with overall high satisfaction ratings. Validation studies showed a significant difference in performance between novices and experts. Residents demonstrated significant skill improvement and retention after repeated simulations.
Conclusions:
The developed simulator provides a realistic, effective training tool for urologic education, addressing the need for safer and more accessible surgical training modalities.
Introduction
Percutaneous nephrolithotomy (PCNL) is a well-established and minimally invasive technique for the management of large renal calculi. PCNL procedures are conventionally performed with the assistance of fluoroscopy, a method employed by 74.5% of urologists worldwide. 1 Nevertheless, a significant concern associated with fluoroscopy guidance is the potential exposure to ionizing radiation experienced by patients, surgeons, and other personnel in the operating room. 2,3 Ultrasound-guided PCNL (US-PCNL) is an effective and safer alternative compared with fluoroscopy-guided PCNL 4 –6 with evidence demonstrating a cost advantage when compared with fluoroscopy 7 ; however, adoption of this technique has been limited, with a growing need for a high-fidelity benchtop simulation trainer for US-PCNL. Currently, simulators for US-PCNL are limited to cadaver and modified biological models. 8,9 In this article, we present the development and validation of a custom-built, nonbiohazardous, biodegradable, high-fidelity simulator model for US-PCNL training and education using the Delphi consensus and modern education theory.
Methods
We adopted a previously documented methodological framework to identify pivotal elements, procedural steps, and evaluation criteria through expert consensus (content evidence validity). 10 Subsequently, this culminated into the development of a benchtop simulator, using three-dimensional (3D) printing and hydrogel molding techniques. This process transpired across multiple phases utilizing an approach developed and published by our laboratory. 11
Phase 1: Physician requirements
Twelve experts from nine institutions in the United States with an average 12 years of experience and 352 PCNL average caseload were recruited to develop this model. Physicians were tasked with generating a hierarchal task analysis (HTA) to systematically break down the US-PCNL procedure into a series of tasks and subtasks (see Supplementary Data S1). This comprehensive analysis encompassed procedural steps, delineation of important anatomical landmarks, identification of necessary surgical equipment, and anticipation of potential errors. The identified errors were further categorized into three distinct types: commission errors, denoting instances where actions were wrongly carried out; omission errors, indicating actions that were not undertaken; and execution errors, characterizing situations where actions deviated from their intended completion.
Phase 2: Engineering deliverables
The outcomes of the HTA were aggregated and a collaborative effort between a team of biomedical engineers and a surgical educator ensued to translate identified tasks and subtasks into engineering deliverables. The criteria employed for the generation of these deliverables included considerations of anatomical and physiological fidelity, procedural relevance, and methodologies for error determination.
Phase 3: Expert consensus
Employing the Delphi methodology, expert consensus was established through a series of three rounds of surveys conducted via Google Forms (Google, USA), where the outcomes of preceding rounds informed subsequent questions. The three rounds comprised of 284 questions divided across four distinct sections: overall utility of the model, components of the model, tissue fidelity of the model, and assessment of surgical performance. Following each survey round, experts had the opportunity to review and modify responses based on the anonymized feedback from their peers in the prior round. Additionally, experts could provide comments or suggest alternative answers. Questions achieving at least 80% consensus in each round were considered for the development of the prototype. In addition, a grading system checklist for the assessment of surgical performance was devised to score participants based on if tasks were performed correctly, performed partially or incorrectly, or not performed at all. Scores range from 0 to 110 with each task’s subscore weighted based on difficulty and complexity of the task (see Supplementary Data S2).
Phase 4: Prototype development
We utilized methodologies previously established by our research group to devise and fabricate a prototype model for US-PCNL utilizing a combination approach involving 3D printing and hydrogel molding. 12,13 Materialise Mimics (Materialise, Belgium) facilitated the segmentation of consensus-approved pertinent anatomy from a CT scan, encompassing the kidney, pelvicalyceal system with a 2 cm pelvic stone, 11th and 12th ribs, and iliac crest. A 3D-printed solid base was designed (Autodesk Meshmixer, Autodesk, USA) using the outline of the segmented flank from the patient’s CT into which a US-PCNL hydrogel insert is registered allowing for effortless exchange and ease of portability. The base, hydrogel molds, and spine and ribs were all 3D printed (see Fig. 1). Each model had five calyces present with clear anterior/posterior and superior/inferior distinctions. Fluid was injected into the model’s urinary system before each PCNL attempt. After access, users could flush more fluid into the system to confirm they were in the collecting system. The prototype model was used to complete an US-PCNL simulation in the operating room by an expert and both external and internal video feeds recorded. The recorded video was sent to 10 experts and evaluated using an assessment survey covering aspects such as anatomy, procedure, education effectiveness, external appearance, ultrasound appearance, tissue fidelity, and evaluation of metrics to confirm the developed simulator prototype aligns with the intended construct identified in the consensus statement (response process evidence).
Model development pathway.
Phase 5: Simulator validation
A multicenter validation study was conducted at five institutions with 28 novices and 20 experts. Their performance was recorded and scored using the surgical performance assessment checklist by three blinded independent reviewers (relations with other variables evidence). Expert participants were asked to rate the model on a 5-point Likert scale for eight questions regarding model utility, tissue fidelity, and evaluation metrics. Furthermore, a localized study was completed with 12 residents to access the ability of repeated training on this US-PCNL simulator to improve skills and performance (internal structure evidence). Each resident performed five simulations in a simulation center equipped with all necessary equipment and instruments, scheduled evenly over 2 weeks with an expert proctor to gain lower pole and upper pole access. A 2-month follow-up simulation was performed to measure retention of their skill. Their performance throughout the study was assessed using the surgical performance assessment checklist developed through the Delphi consensus from Phase 3.
Results
Consensus
Per the Delphi methodology, expert consensus was reached in 89 questions out of 284 (31.3%), with 58 questions (65.2%) achieving 100% agreement. Expert participants concluded that despite several advantages of US-PCNL, there has been slow adoption due to lack of adequate training platforms and pathways, and a simulation-based solution with a realistic and safe anatomical model would be beneficial in the training of this approach. Factors considered to be important features to include in the training model included nonbiohazardous tissue realism with accurate anatomy including the kidney, pelvicalyceal system with stone, 11th and 12th ribs, iliac crest, and abdominal wall. In addition, procedural steps including ultrasound-guided imaging and puncture, guidewire positioning, tract dilation, and endoscopic navigation were also deemed important features of the model.
Prototype development
Several iterations were required to fine-tune the complete prototype model. Ten experts provided feedback on if the prerecorded prototype simulation aligned with the consensus. The results revealed high satisfaction across all domains: anatomy (98.3%), procedure (82.5%), education effectiveness (83.3%), external appearance (97.2%), ultrasound appearance (100%), tissue fidelity (68.1%), and evaluation metrics (94.4%) (see Fig. 2).
Evaluation of simulator based on consensus results.
All items in the anatomy (6/6), external appearance (4/4), ultrasound appearance (6/6), and evaluation (22/22) categories had 100% of items agreed upon with high satisfaction and consensus. However, in the education effectiveness (6/8; 75%), procedure (6/8; 75%), and tissue fidelity (3/8; 37.5%) categories, a slightly lower proportion of items achieved consensus with satisfaction levels greater than 70%.
Simulator validation
Forty-eight participants (28 novices and 20 experts) across five institutions completed a full simulation validation study on the final model. Performance based on the assessment of surgical performance checklist revealed a significant difference between average scores by novices and experts (42.3% vs 93.5%, p < 0.01) (see Fig. 3).
Novice vs expert percent score. ***p < 0.01.
Expert participants were also tasked to rate the model based on a 5-point Likert scale with eight questions (see Fig. 4). Results for all questions were 39.7% strongly agreed, 52.5% agreed, 5.0% neutral, 2.8% disagreed, 0% strongly disagreed, and 2.8% not applicable.
Expert survey results for US-PCNL validation. US-PCNL = ultrasound-guided percutaneous nephrolithotomy.
In the localized study where 12 residents (3 Post-Graduate Year (PGY)1s, 3 PGY2s, 3 PGY3s, 1 PGY4, 1PGY5, 1 PGY6) performed five simulations gaining lower pole and upper pole access, both sequential tests and a 2-month follow-up retention test demonstrated significant improvement compared with the first test when evaluated based on the assessment of surgical performance checklist. Lower pole percentage scores increased on average from 45.5% on the first test to 75.3% on the fifth test, with a 2-month average retention score of 66.0% (see Table 1). Upper pole percentage scores increased on average from 39.6% on the first test to 64.1% on the fifth test, with a 2-month average retention score of 55.2%.
Average Percentage Scores for Residents in Localized Study
Average percentage score for residents with lower and upper pole access over five attempts and with 2-month retention. *p < 0.1, **p < 0.05, ***p < 0.01 compared with first test.
SD = standard deviation.
Discussion
Despite the advantages of US-PCNL compared with fluoroscopy-guided PCNL, there is a low adoption rate of this technique, potentially due to limited resources available for training and education. Given the procedure’s inherent complexity and potential for adverse outcomes, urologists require comprehensive training, adept surgical skills, and familiarity with ultrasound utilization for a PCNL. Teaching the skills required for PCNL has been difficult due to the complexity of obtaining safe access to the kidney for lithotripsy with potential injury of anatomical structures from the skin to the renal capsule involving vascular, pleural, and colonic injuries. 14 As a result, the demand for a safe, high-fidelity, and nonbiohazardous model arises from the need to enhance procedural proficiency and patient safety. A training model offers a controlled, realistic, and repetitive environment for trainees to familiarize themselves with anatomical landmarks, procedural steps, and instrument manipulation to practice and obtain procedural competence before starting with human procedures.
In order to create this model, we utilized a previously established backward design, described in Understanding by Design, for development of simulation platforms. 10 This methodology is deconstructed into four distinct phases: outlining physician needs, converting these requirements into engineerable tasks called deliverables, creating and evaluating prototypes, and the development of a validation process. Our team utilized this backward design educational framework to create a training model for US-PCNL with a surgical performance checklist as an objective evaluation metric for assessment. The hydrogel model was validated by 48 participants across five institutions, and a mean difference of 51.2% in performance was statistically different (p < 0.01) between novices (n = 28) and experts (n = 20) based on the surgical performance checklist.
Significant increases in average scores with each additional simulation for both upper and lower pole US-PCNL access were demonstrated in 12 residents performing five repetitive simulations. Subsequent assessment conducted after a 2-month interval revealed sustained performance levels among residents with average scores significantly higher than the first simulation scores in both upper pole (p < 0.05) and lower pole access (p < 0.01). These findings suggest a robust retention of skills among trainees utilizing our US-PCNL hydrogel models, particularly when supplemented with expert-guided training interventions for learning technical skills.
Previous other models have been developed for the training of PCNL. Animal models using an ex vivo perfused porcine kidney preimplanted with artificial stone material can be placed within intact chicken carcasses. 9,15 Advantages of this model include the simplicity of assembly, low cost, presence of tactile feedback, and the ability for the model to be reused for at least three procedures by suturing the access wounds closed. However, a drawback of this model is the obvious lack of human anatomical similarities and landmarks. For a more human and realistic model, thiel-embalmed cadavers may serve as training models for PCNL and ultrasound-guided supine endoscopic combined intrarenal surgery. 8 Human cadaver models have several advantages, the most prominent being life-like anatomy with tactile feedback and the possibility of a full procedural simulation. However, human cadavers are expensive and skin penetration may be more difficult and less realistic. Lastly, several virtual reality (VR) simulation models have been explored as teaching modalities for PCNL. 16,17 Advantages of VR platforms for PCNL training include the lack of radiation exposure, ability to be used repeatedly and regularly, objective assessments, and the opportunity for various difficulty training levels. However, VR models may have a high barrier to entry with expensive technologies, limited to no sensory tactile feedback and depth perception, and the lack of a full procedural simulation environment.
Our model aims to bridge human cadaveric and VR simulation models by incorporating life-like kidney models with anatomical landmarks, decreasing costs, and providing realistic tactile feedback, with the ability to objectively assess performance. The model is continuously evolving to enhance its realism and fidelity, aiming to provide a more authentic training experience for users. Currently, materials for a single model cost under 100 US dollars. However, this cost does not factor in the extensive time and labor required to develop the models by biomedical engineers. Like all models, our model degrades over time if not properly stored. With proper storage in a vacuum-sealed bag, the model can last as much as 6 months. Each model can be punctured approximately 10 times before the previous track marks are too prominent. If the user can overlook previous marks, the model may be used for additional attempts. Through iterative development cycles informed by expert feedback, future iterations will include more features like complex stone morphologies such as staghorn kidney stones. Additional future directions may include modifications to streamline the model fabrication pipeline to be able to generate personalized models that are patient specific for simulation and practice prior to the actual human procedure. This is currently the only hydrogel model that has been custom-built for US-PCNL simulation training using the most modern and advanced approach to simulation validation.
Conclusions
We conceptualized, developed, and validated a nonbiohazardous high-fidelity benchtop simulator for US-PCNL using modern education theory. The utilization of our model in urologic education will provide a much-needed means of training and adopting US-PCNL.
Footnotes
Authors’ Contributions
A.G.: Conceptualization, methodology, formal analysis, investigation, resources, data curation, supervision, writing, review, and editing. L.S.: Conceptualization, methodology, formal analysis, investigation, resources, data curation, supervision, writing, review, and editing. O.L.: Formal analysis, writing, review, and editing. C.I.: Formal analysis, review, and editing. N.S.: Formal analysis, review, and editing. P.S.: Formal analysis, review, and editing. E.B.-L.: Review and editing. S.K.B. Review and editing. R.S.H.: Review and editing. N.K.: Review and editing. S.W.: Review and editing. D.T.T.: Review and editing. R.L.S.: Review and editing. H.C.C.: Review and editing; T.T.: Review and editing; J.A.: Review and editing. D.B.: Review and editing. T.C.: Conceptualization, methodology, resources, review, and editing.
Author Disclosure Statement
No competing financial interests exist.
Funding Information
No funding was received for this article.
Supplementary Material
Supplementary Data S1
Supplementary Data S2
Abbreviations Used
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
