Evaluation of Optimal Timing of Expert Feedback in a Simulated Flexible Ureteroscopy Course

Introduction

The exponential growth of novel surgical techniques, mandated changes to resident working hours, heightened transparency and increased patient accountability, and increasing complex surgical patients has drastically changed the surgical training landscape. ^1,2 As a result, the traditional model of apprenticeship-style surgical training is no longer feasible as the sole method of surgical training, and there have been efforts to transition some of the clinical learning into a simulated environment to allow for development of new skills to be applied to the clinical setting. Furthermore, the paradigm shift into the competence-based training model emphasizes the need for adequate surgical exposure for residents to be skilled in both novel and traditional surgical techniques, while balancing academic and increasing ward duties with complex medical patients. The use of simulation-based training (SBT) has increased in recent years with many academic centers building high-tech surgical skills centers or laboratories to help residents develop important surgical techniques outside of the operating room (OR). Benefits of this are many, including the opportunity to engage in deliberate independent practice (IP) without the time constraints, pressures from staff or senior residents, the safety concerns of being in the OR, or the ethical concerns surrounding developing trainee competence at the risk of providing patients with the best possible care.

Despite its touted benefits, one of the biggest limitations of SBT includes the prohibitive costs associated with this method of training, especially in smaller academic centers to acquire, maintain, and organize SBT sessions with up-to-date simulators. These include the financial and human resources needed to build, support, maintain, and improve advanced simulation centers as well the need for expert feedback. SBT is ultimately a supplementary tool to improve the training and enhance the trainee experience under the supervision of skilled expert educators, and timely feedback from experts is critical to any effective educational curriculum, especially so in technical skill development. ^{3,4 –11} In addition, expert feedback has been shown to be critical to allow for correction, proper skill development, and eventually to transition skills from the simulated to the clinical setting. ^{8 –13}

To minimize complications and prevent compromising patient care while training residents in new surgical techniques, especially in endoscopic surgeries with a steep learning curve, SBT has been shown to be a useful educational tool to allow for novice trainees to gain comfort with these new modalities in a low-risk and low-pressure situation. ^3,4 In the field of urology, flexible ureteroscopy (fURS) remains a challenging skill for junior residents to develop; therefore, various SBT models to help learners begin to feel comfortable in fURS have been developed. ^14,15 Previous simulators have shown to improve novices' skill in fURS; however, given the need for expert teaching and the limited amount of time that staff have, optimization of this valuable resource should be undertaken. There have only been two previous studies done, looking at the timing of SBT that has shown that early or immediate feedback is optimal for technical skill acquisition in medical students. ^16,17 As such, this study aims to determine whether there is an optimal time for expert feedback during an SBT curriculum for fURS. ^3,18 Specifically, this study hopes to determine whether expert late feedback (LF) provided after several IP is as effective as early feedback (EF) before engaging in IP in training novice learners in fURS. These findings may be useful to ensure experts' time is best used when developing an SBT curriculum.

Materials and Methods

Junior-level urology residents (postgraduate years [PGYs] 1–3) and medical students who were accepted into urology residency programs at two academic centers were recruited to participate in the SBT curriculum and feedback study. In total, there were three fourth year medical students, and four residents each from PGYs 1 to 3. The previously validated Cook^® Medical URS model designed by Dr. John Honey was used in this study. ¹⁹ This model is composed of polyurethane tubing shaped to model a dual caliceal system, a complete bladder–ureter–kidney system, and a tortuous ureter system (Fig. 1). This model has previously been shown to be effective in helping residents gain short-term technical skills in fURS. ³ The model fills with water and has a motorized suction system to remove debris from the working areas, simulating the high-pressure irrigation during fURS, and provides improved visibility while using the model. We also used flexible Uscope (Storz Flex X-2) and Nitinol baskets during the pre- and postassessments and IP sessions. ³

OPEN IN VIEWER

FIG. 1.

The Cook^® Medical URS model with the artificial dual caliceal system, a mock bladder–ureter–kidney system, and tortuous ureter system from left to right, respectively. Flexible Uscope (Storz^® Flex X-2) and Nitinol baskets were used on this model to complete all assessments and IP sessions. IP = independent practice.

An initial intake questionnaire was given to participants to capture demographic information, such as age, gender, level of training, prior operative exposure, and self-assessment of their endourologic skills (Table 1). Participants were randomized to receiving EF and LF using a free online random number generator using 0 or 1 to assign participants into the EF (number 0) or LF (number 1) groups. All participants first watched an introduction video about fURS. After this, there was a didactic teaching session focusing on the basics of fURS and stone management. Lastly, each participant received one-on-one time with experts for demonstration of fURS without performing fURS themselves. After this one-on-one session, all participants underwent their baseline skill assessment. In the EF group, their expert feedback was provided right after their baseline assessment (Fig. 2). For the LF group, their expert feedback was provided after their third and last IP session before their final assessment (Fig. 2).

OPEN IN VIEWER

FIG. 2.

The study flow diagram outlining the differences between the curriculum between the EF and LF study groups. EF = early feedback; LF = late feedback.

All participants in the study underwent their IP sessions at their respective skills laboratories (the Surgical Skills Lab at the Queen Elizabeth II Health Sciences Center in Halifax, Nova Scotia, and Allan Waters Family Simulation Center at St. Michael's Hospital in Toronto, Ontario). Staff surgeons were present to enforce the time limits on the individual practice sessions (30 minutes) but participants had full access to any equipment and there were no limitations in terms of their own self-training. All participants were only allowed to undergo three IP sessions, 30 minutes each, all within 2 weeks of the original SBT didactic session. Feedback was given by a single expert endourologist at each skills center lasting a minimum of 10 minutes to 15 minutes maximum for standardization within each center. Feedback only pertained to the following domains: respect for tissue, time and motion efficiency, instrument handling abilities, ureteroscope handling techniques, overall flow of procedure and forward planning, and the use of assistants to allow for further standardization.

The initial and post-SBT assessment of fURS skills consisted of the same simulated task; to perform fURS repositioning of a lower pole renal stone into the upper pole using an fURS and stone basket. The baseline and post-training tasks were video recorded and evaluated by two expert endourologists who viewed an anonymized version of the recording. Assessments were performed using the validated assessment device as published by Matsumoto et al. ^20,21 The final assessment was completed 1 week after each participant's last IP session to minimize the risk of recall bias. After their last assessment, all participants completed an exit questionnaire detailing their evaluation of the SBT curriculum, self-assessed performance, and skill level.

Data analysis was performed to determine the impact of the SBT curriculum, as well as the impact of EF compared with LF. Demographic data were correlated with performance scores to determine the significance of variables on skill acquisition. Student's t-test was used for parametric data, whereas Mann–Whitney U and Wilcoxin matched pair analysis testing were performed for nonparametric data. The IBM Statistical Package for the Social Sciences version 21 was used for all analyses.

Results

There was a total of 15 participants (12 residents PGYs 1–3, and 3 fourth year medical students who were accepted into a urology residency program) included in the study (Table 1). Five trainees completed the study at Dalhousie University and 10 trainees at the University of Toronto. Eight participants were randomized to EF and seven participants to LF. None of the participants had used the Cook Medical URS model before. There was a relatively even distribution of junior residents per year; however, most of the participants were male (87%). Most of the residents had significant flexible cystoscopy experience with >80% completing >20 flexible cystoscopies. However, only 27% had completed >10 fURS procedures. There was no significant difference in prior surgical experience among the EF and LF groups.

There was significant improvement in both mean task completion times (15.2 minutes vs 9.1 minutes, p < 0.0001) and performance score (18.1 vs 24.2, total out of 35, p = 0.008) for the overall cohort (Table 2). We did not assess how the participants' performance changed throughout their SBT curriculum based on the number, quality, and time of their IP sessions. The EF and LF groups did not differ significantly on their mean precourse task completion time nor performance scores (Table 3). The time to completion improved in both groups, more so in the EF group (8.8 minutes vs 9.6 minutes), although this was not statistically significant (Table 3). The EF group had a significantly higher mean postcourse performance score (25.7 vs 22.5 out of 35, p = 0.05) (Table 3). Other participant characteristics, such as PGY, gender, handedness, and prior experience did not significantly correlate with the degree of post-SBT score or times. Both EF and LF groups felt that they received adequate feedback throughout the SBT; however, 40% of participants wanted more feedback. There was also a higher incidence of evaluators being comfortable with allowing the participant to perform the procedure in an OR setting after the SBT (p = 0.003), and this evaluation was similar between the EF and LF groups (p = 0.44). On the exit survey, most participants (75%) preferred EF, and this did not differ among the different cohort groups.

Discussion

The rapid development of new surgical techniques, time constraints of training, and increased patient requests have made it difficult for novice residents to gain adequate exposure in clinical settings to become comfortable in tackling the variety of novel techniques needed to become competent surgeons. The development of SBT has helped bridge this gap, provides junior residents with the opportunity to undergo targeted IP to improve their skills before applying them in a clinical setting, and has demonstrated to improve learning and skill retention. ^{21 –24} Within the field of urology, SBT has had the most impact in the field of endourology, with widespread use including validated models, curriculum development, and use for evaluation. ²⁵ Despite the widespread acceptance of SBT and the importance of expert feedback in SBT, the timing of this valuable resource has not been rigorously evaluated. To our knowledge, this study is the first of its kind in evaluating timing of SBT in residents.

This study demonstrates that the SBT was effective for short-term learning of fURS endourologic skills, and that ultimately both cohorts (EF and LF) were equal at the end of the training. There was an overall improvement in time to perform the fURS task in both the EF and LF groups, which was statistically significant for the overall cohort; however, this change was not statistically significant when the EF and LF groups were compared, likely because of the sample size. However, there seems to be a trend toward the EF group being more improved, with the total task completion time decreasing more in the EF group (6.6 minutes vs 5.5 minutes) and increased performance score (7.3 vs 4.8 out of 35). Both improvements in mean score and time between both groups are likely meaningful for the residents' educational activity. The success of the SBT curriculum in improving participant scores with their assessment is likely secondary to their familiarity with the procedure, as well as practice with using the ureteroscope and basket.

Timely feedback is accepted to be extremely important in improving cognitive retention in technical skills training. ³ This study demonstrates that EF might offer some advantages to LF, primarily because residents can target their practice sessions with the feedback they receive from their mentors, which may help to prevent the development of bad habits and may help maximize resident training time with the simulators. EF was also preferred by SBT participants; however, the qualitative reasoning for this was not determined, as the exit survey did not ask participants their reasoning for this.

This study's biggest limitation was the small sample size, which may result in bias and limits to generalizability. Other limitations of our study include our inability to control for any additional training that participants might have had in fURS during the SBT period. Although we standardized the evaluators across the two different institutions, the participation and education of learners would have been different for each resident, despite efforts to control and standardize the feedback session. Furthermore, this study did not evaluate the long-term skill retention or how these skills may translate into better fURS performance in the OR. Instead, we focused on short-term technical skill development and a standardized test instead.

This study did not include senior trainees, such as senior residents or fellows, which limits generalizability to junior learners; however, this was done by design to focus this study on novice learners. In addition, there was a lower number of completed fURS procedures in the study groups, and, therefore, the SBT curriculum in our study may be most beneficial and applied to learners in their early stages of training with minimal pre-SBT exposure to fURS.

Additional studies with bigger sample sizes will need to be done to look into how SBT and the timing of valuable feedback might best be used. Looking into why participants preferred one form of feedback (EF vs LF) on the exit survey should also be done in future studies. Furthermore, having senior residents who have achieved competency in surgical skills to teach junior residents would not only benefit the junior resident but also the senior in helping them teach novice trainees as attending staff, and would be a great topic for future studies and SBT curriculum development.

Conclusions

This study demonstrates the efficacy of our fURS SBT curriculum in improving short-term learning of technical skills among junior level urology residents with little prior fURS experience. EF resulted in improved overall scores, and was preferred by residents compared with LF. This study helps target and support expert feedback timing in SBT curriculum to improve the educational experience and benefit for trainees, staff, and, ultimately, our patients.