Abstract
Mastering a surgical skill requires experience and repetition, yet opportunities for surgical trainees to gain real experience are variable and limited by case load. Surgical simulators have emerged in an attempt to overcome these limitations. However, the few currently available skills simulators for flexible endoscopy are costly, have limited accessibility and versatility, lack portability, and require dedicated time for practice. The use of a portable skills simulator to teach flexible endoscopy may provide a feasible alternative. This study introduces a novel, low-cost, portable, endoscopic simulation system for training basic endoscopic skills. Using custom software, the simulator presents a virtual environment featuring 3D models of anatomy, endoscopes, and endoscopic tools. The virtual endoscope and its tools are directly controlled in the simulation by motion input from a custom-manufactured portable endoscopic controller that communicates data via a Bluetooth interface. This two-part study presents proof of concept and initial pilot data examining the face/content validity and preliminary construct validity of the portable endoscopic simulator. In part 1, experts (n = 2) and novices (n = 6) provided ratings of fidelity and utility as a training tool. In part 2, experts (n = 4) and novices (n = 4) completed 10 simulated sequential basic endoscopic tasks, and time to completion was assessed. Findings indicate that the simulator has good utility as a training tool, but some features require modification to be more realistic. Furthermore, both novices and experts improved on the task with repeated measurements (p < 0.001), but there were no significant differences between experts and novices in time to completion. Although more robust validation is required, this simulator appears promising as a feasible and cost-effective tool for providing simulation training on basic endoscopic skills.
Introduction
Simulation has been a novel modality used to remedy recent challenges experienced by surgical trainees. 1 This has highlighted an emerging theme in surgical education—the importance of integrating skills simulation into postgraduate urology curriculums. 2 This need for skills simulation is accentuated by the recent introduction of competency-based medical education (CBME) to urology residency programs. The potential benefits of skills simulation in urologic education are improved patient care and safety (minimization of errors), improved learning efficiency by providing residents with a safe standardized environment and reducing equipment spoilage. 3 –6
The movement toward tangible evaluation through milestones and entrustable professional activities dictated by CBME suggests that skills simulation may one day become a curricular requirement. 7 However, the currently available skills simulators for flexible endoscopy are costly, have limited accessibility, and require dedicated time for practice. 8 These limitations currently create a logistical challenge for their integration into current urology curriculums. Flexible endoscopic skills simulators include the Uro-Mentor (Simbionix), Uro-Trainer (Karl Storz GmbH), and bench models by Limbs and Things and Mediskills. 8 However, the barriers to curricular integration of these simulators include cost, maintenance, and accessibility (dedicated time). 9 For example, the PERC-Mentor percutaneous access training system is quoted around 100,000 USD. 10
To bypass these limitations, Rashid and Gianduzzo argue for the development and use of mobile simulators. 3 A class of mobile simulators that combine the processing capabilities, accessibility, and availability of mobile (smartphone or tablet) technology with the cost-effectiveness, versatility, and user-friendliness of an endoscopic joystick (controller) offer an obvious platform for developing a portable endoscopic simulator. The use of a portable skills simulator to teach flexible endoscopy may provide a feasible alternative to current simulation options. As such, we present this study as a proof of concept assessing the face/content and construct validity of a portable endoscopic simulator, developed at our institution, comprising a mobile platform simulation application and a Bluetooth endoscopic controller to teach flexible endoscopic skills.
Materials and Methods
Portable endoscopic simulator
Custom simulation software was developed using a videogame software development platform (Unity v.2021.1.9f1; Unity Software). The simulation consists of a virtual environment featuring 3D models of anatomy (posterior urethra, bladder, ureters, and renal collecting system), interactable models of pathology (e.g., stones, tumors), endoscopes (cystoscope and ureteroscope), and endoscopic tools (e.g., wires, laser fibers, electrodes). The software is compiled to run on an Android platform, and for the purposes of this study was installed on a Samsung Galaxy Tab S7+ tablet (Model SM-T970; Samsung Electronics).
The virtual endoscope and its tools are directly controlled in the simulation by motion input from a custom-manufactured portable endoscopic controller that communicates data via a Bluetooth interface. The physical controller has a similar design and feel as common flexible cystoscopes and ureteroscopes. Orientation of the controller is determined by a digital inertial measurement unit (Bosch BNO080; Bosch), and controller translational motion is measured by a custom stereoscopic optical tracking system. The optical tracking system consists of two parallel cameras with infrared band-pass optical filters, and tracks the x, y, and z positions of an infrared light source fixed to the controller.
Finally, the controller features physical inputs, including a flexion knob, buttons for device reset/start and activation of energy sources, and a push/pull cylinder that emulates the user manipulating a wire or other tool within the scope's working channel. Haptic feedback is built into the controller in the form of a vibration motor, but this feature was not enabled during the study to simplify the number of features that needed to be tuned. The cost to manufacture the controller and tracking system was less than 1000 CAD.
The user interface resembles that of a videogame with a set of levels. Each level consists of a set of specific tasks and objectives within the virtual environment (e.g., survey the bladder, perform lithotripsy of a renal pelvis stone), with each successive level being more difficult than the previous. The user begins on the most basic level (surveillance) and may only unlock the next level by achieving a minimum target level of performance (e.g., greater than 80% of the bladder has been surveyed). See Table 1 for a complete list of levels and the corresponding metrics used to grant access to the next level. This design forces the user to practice each task until they have gained enough skill to move on to a more complex one, a process that emulates training in real life.
Simulator Levels and Performance Metrics
URS = ureteroscopy.
Damage to healthy tissue is detectable by the software, but the display of realistic dynamic changes in surfaces was beyond the scope of our available resources and was not implemented at this stage. The graphics, controller refinements, and content of each level were developed and refined in consultation with the following authors: J.D., H.R., J.B., and P.W. For a detailed video demonstration of the simulator, scan the QR code seen in Figure 1.
Components of the simulation system: a mobile device to display the simulated scope viewport, portable endoscopic controller, and optical tracking system. Scan QR code for video demonstration.
Study design
Participants were enrolled and consented for the study by invitation. The study was approved by the institutional research ethics board (REB# 109688). As part of this proof of concept, we examined the face/content validity of the simulator (part 1) and construct validity (part 2) by examining repeated measurements of a predetermined task for both experts (urology consultants) and novices. Novices for part 1 were PGY1 urology residents to maximize the difference in experience level and perspectives compared with the experts (consultants). Novices for part 2 were urology residents at any level of training, who were invited based on convenience within limitations of the COVID-19 pandemic. The trials for part 1 took place at our annual urology “Boot Camp” for first-year residents in July 2020, and part 2 in an office setting throughout April 2021.
Part 1: face/content validity
Participants were provided with basic instruction in setting up the system and selecting the various levels. They were then asked to make at least one attempt at each of the first five levels. Levels trialed included (1) surveillance of the bladder, (2) inspecting three bladder lesions, (3) inserting a wire into ureteral orifices, (4) inserting a wire into ureteral orifices with prostate median lobe, and (5) performing laser lithotripsy of a bladder calculus.
After using the system, participants were provided a 15-item survey that assessed the perceived realism and usefulness of the system. Nine items assessed how useful this tool would be in the context of teaching and learning (e.g., useful for teaching cystoscopy; 1 = not useful at all to 5 = extremely useful) and six items assessed how realistic the system was (e.g., tissue appears realistic; 1 = not realistic at all to 5 = extremely realistic). Participants were also asked to provide any relevant feedback using an open-ended question format.
Part 2: construct validity
As a result of findings from part 1, the system was upgraded with the ability to track six degrees of freedom of the controller using a custom stereovision optical tracking system (the first iteration could only track the height of the scope). This modification permitted more realistic hand motions to be captured in the simulation. For part 2, a new sample of experts and novices were asked to complete a basic task of simulated insertion of a guidewire sequentially into both ureteral orifices (Figs. 2 and 3). Participants were asked to complete 10 sequential trials of the task and were provided limited instruction on how to use the device. Time to complete the task was used as a competency metric and was compared between the two groups.
Examples of tasks performed in three levels of increasing difficulty:
A more complex level involving fluoroscopic-guided ureteroscopy and laser lithotripsy of a renal pelvis stone.
Data analytic strategy
Descriptive statistics and frequency analyses were conducted on the 15-item survey to assess face/content validity of the simulation task. Construct validity was assessed by determining whether the simulation task could differentiate between experts and novices, and whether it is useful as a training tool by assessing repeated measurements. We conducted a mixed ANOVA with trials (n = 10) as the within-subjects factor, and experience level (expert vs novice) as the between-subjects factor. The dependent variable was time to complete the task in seconds. An alpha correction of 0.006 was applied to control for type 1 error due to multiple comparisons.
Results
Part 1: face/content validity
Participants for part 1 of this study consisted of two experts (one female; M age = 45) and six novices (five males; M age = 26) from the urology division at our institution. Overall, findings indicate that the system would be useful for teaching and training, with 98.6% of responses consisting of a four or five rating on the utility questions (M expert = 4.5, SD: 0.24; M novice = 4.67, SD: 0.41). However, the findings indicate that the realism of the system could be improved, particularly in terms of tissue interaction, bladder entry anatomy, and lithotripsy (M expert = 3.5, SD: 1.65; M novice = 3.64, SD: 0.75) (Fig. 4). From the open-ended comments, the most prominent feedback was that the optical tracking required improvement to allow for more natural movements.
Average ratings of
Part 2: construct validity
Participants for part 2 of this study consisted of four urology consultants (three males; M
age = 48) and four residents (three males; M
age = 30) from the same urology division. Completion times across the 10 trials varied while holding the experience level constant (p < 0.001;
Average Time in Seconds to Complete the Task for Experts and Novices
SD, standard deviation.
While time to complete the sixth trial (M = 23.58) was faster than the first trial, this effect did not reach significance with the alpha correction applied (p = 0.008). Thus, on average, regardless of the experience level, the participants' time improved on the task with repeated measurements (Fig. 5). However, there were no statistically significant differences between experts and novices in time to completion (p = 0.204;
Average time to complete the simulation task across 10 trials for experts and novices.
Discussion
The use of a portable skills simulator to teach flexible endoscopy may provide a feasible alternative to the costly and often inaccessible simulator options currently available. The first step in this endeavor is to demonstrate proof of concept and examine face/content validity of this class of simulator. 11 Our assessment of face/content validity indicated that there is room for improvement in terms of how realistic our simulator is. Specifically, the interaction with tissue and the lithotripsy components was rated as only moderately realistic by both experts and novices. On other categories however, such as structure, size, and tissue appearance, the novices rated the simulator as more realistic than the experts. This is consistent with findings from a face/content validity study of the Uro-Trainer simulator and may be due to the relative inexperience of the novices. 12
Furthermore, open-ended feedback indicated that the application's optical tracking could be improved, which we upgraded before proceeding with part 2 of this study. Despite this, the feedback from expert and novice users was promising in terms of the utility of the simulator. The overwhelming response from users was that the simulator would be a useful teaching tool.
In terms of construct validity, our study demonstrated time improvements with repeated use of this portable endoscopic simulation. While no statistically significant differences were observed between experts and novices, graphically there appears to be differences between the groups in completion time on most trials. Furthermore, the magnitude of the effect size suggests that a large proportion of the variance in completion time may be due to variance in participant experience level. It also appears that while there was an initial learning curve for both groups, it was reconciled more quickly for the experts than the novices. Validation of the Uro-Mentor endoscopic simulator demonstrated a similar learning curve trend in novices, but not in the experts. 13
This may be due to the novelty of the simulator developed in the present study, compared with the Uro-Mentor that was developed nearly two decades ago and may have been more familiar to expert participants. For example, several of the consultants initially attempted to grasp the flexible shaft of the scope during manipulation, which is not present on the controller. We also suspect that, given the simplicity of the task, the novices may have gained substantial skill in completing it after several trials. This may have trended the results toward a nonsignificant difference in performance.
Implications
The potential impact of this type of portable skills simulator may be immense. The portable nature of the simulator could lead to improved availability and accessibility and mitigate the need for dedicated education time. 3,8 This would allow learners the freedom to practice at their own pace and within their schedule availability. Portable skills simulators will also be cost-effective, circumventing the need for bulky complex equipment and the need for a dedicated facility. 3,8 In addition, this type of simulator will provide educators with an effective, feasible, and more cost-effective method of integrating simulation into current urology curriculums to help achieve the goals set forth by CBME. 14
The use of such portable simulators may also be expanded to teaching other endoscopic skills such as percutaneous nephroscopy and transurethral resection of the prostate and bladder tumors by developing different controllers to emulate the equipment needed for each of these different skill sets. Furthermore, the use of a portable skills simulator to teach basic flexible endoscopic skills is not limited to urology and has potential applications in gastroenterology, anesthesia, general surgery, and orthopedic surgery. Lastly, if these simulators are efficacious, it will inevitably have a positive impact on patient care by improving the rate and quality of acquisition of these skills outside of a clinical setting.
Limitations and future directions
While this initial study provided preliminary positive findings including proof of concept, further validation is required with larger samples to substantiate the construct validity of this simulator, as our sample size was insufficient to statistically determine construct validity. A formal power analysis was not performed a priori, in part, due to a lack of reliable data for an estimate of effect size; we had attempted to maximize the sample size within the limits of COVID-19 pandemic restrictions at the time of study. We also recognize potential bias in our results, particularly given that participants were enrolled by invitation. In our future studies, we aim to be more inclusive and extend the scope to be multicentered.
In addition, the outcome for the present study did not assess performance comprehensively, only time to complete the task. Thus, the addition of performance-based metrics and penalties for missteps or poor performance (e.g., damage to lumen, number of failed attempts) need to be implemented. These features are necessary for meaningful integration into a training curriculum, as users require specific feedback on their performance to gain proficiency on subsequent trials. For example, the users may be presented with their overall performance relative to other users at the same level of training, a summary of performance metrics (e.g., economy of motion, total laser time), and unacceptable errors (e.g., perforated bladder with the laser). A summary of these results could be provided to a training program to evaluate each trainee and help them focus on areas in need of improvement.
The immediate next steps of this project include further incorporating user feedback. We intend to implement several improvements in the application to enhance its realism, utility, and customizability (by both learners and educators). In addition, we aim to refine the scope dynamics such that the application models a portion of the scope outside the patient, improving the overall feel of scope control. Together, these changes should improve both how useful this simulator is as a training tool and how realistic it is.
Long-term goals for this project include incorporating the simulation of more complex tasks with a realistic workflow, resulting in a more translatable skills training tool. We have made steps in this direction by creating a level that involves gaining access to a renal pelvis under fluoroscopic guidance and the subsequent laser lithotripsy of a renal pelvis calculus. These more comprehensive procedures will need further validation studies to prove usefulness in a training curriculum. We hypothesize that more complex tasks may differentiate novices more clearly from experts, regardless of an initial learning curve. Lastly, we ultimately intend to enhance the simulator into a fully immersive 3D virtual reality environment. This will facilitate the assessment of predictive validity to determine if simulator training results in improved real-life performance on patients.
Conclusion
This study has demonstrated moderately good face/content validity for a novel portable endoscopic simulator, and that with repeated measurements this simulator appears promising as a training tool for practicing basic endoscopic skills. The next phase will incorporate further enhancements to the realism of the simulation, followed by a more robust construct validity study involving the simulation of more complex procedures.
Footnotes
Authors' Contributions
J.C.: Methodology (lead); data curation (lead); investigation (lead); software (lead); formal analysis (supporting); writing—original draft (supporting); and writing—review and editing (equal). C.W.: Formal analysis (lead); writing—original draft (lead); and writing—review and editing (equal). J.D.: Conceptualization (supporting); and writing—review and editing (equal). N.S.: Investigation (supporting); and writing—review and editing (equal). H.R.: Investigation (supporting); and writing—review and editing (equal). J.B.: Investigation (supporting); and writing—review and editing (equal). S.D.: Conceptualization (supporting); investigation (supporting); and writing—review and editing (equal). P.W.: Conceptualization (lead); methodology (lead); funding acquisition; supervision; and writing—review and editing (equal).
Author Disclosure Statement
The simulator described in the present study is owned by CystoSim Corp and patented by authors Jeremy Cepek and Peter Wang (Application No. 63/193,166).
Funding Information
The authors gratefully acknowledge funding from the CUASF New Investigator Grant (K7510) and the Western University Department of Surgery Internal Research Fund (R5939A04).