The Evaluation of Computer Vision-Based Automated Performance Metrics for Endoscopic Kidney Stone Surgery

Introduction

Ureteroscopy has been utilized since the 1980s and remains a central tool in the armamentarium for management of urinary calculi, as well as other benign and malignant urologic conditions. ^1,2 Evaluating and facilitating surgical competency of flexible ureteroscopy (fURS) is critical during urologic surgical training because of the high prevalence of stone disease. However, efforts to improve skill evaluation and acquisition in the operating room are inherently limited by time constraints and patient safety concerns. In addition, many of the current methods of surgical evaluation are subjective.

Feedback to trainees is usually in the form of verbal communication from the attending surgeon after the case or in real time, but is inherently subjective. This is particularly true for endoscopic kidney stone operation, where there are few ways to evaluate competency and provide feedback objectively. Widely used flexible endoscopes give no kinematic data (unlike robotic procedures), and the manual review of endoscopic video recordings has not been able to distinguish endoscopic surgical expertise. ³ In contrast, several automated performance metrics (APMs) have been identified in robotic procedure and provide an objective method to assess surgical performance. ^4,5 Validated, objective real-time tools to evaluate surgical expertise and provide feedback to trainees during endoscopic stone procedure are necessary for both fostering patient safety and improving surgical outcomes.

We have previously demonstrated that computer vision models (i.e., an application of machine learning, which automatically analyzes image data) could accurately and automatically segment stones in endoscopic surgical videos. ⁶ Not only do these models accurately track kidney stones during the operation in real time but also provide an objective way to evaluate surgical videos of kidney stone treatment for surgical skills. In this study, we sought to define APMs that may help in distinguishing surgical expertise. Specifically, we performed a pixel-based analysis of stone segmentation and assessed performance using novel APMs for both expert and trainee videos during endoscopic kidney stone procedure.

Materials and Methods

Study design and video collection

After internal review board approval, we performed a computer vision-based analysis of prospectively collected surgical videos of fURS for renal stones between January 2022 and October 2022. We collected 50 surgical videos and identified the associated surgeons. Surgeons were categorized as “expert” (n = 3, fellowship-trained endourologist, case volume of >100 fURS per year) or “trainee” (n = 6, postgraduate year 3–4 resident, <100 fURS per year). We compared the expert and trainee videos for two specific tasks as follows: (1) stone localization and (2) stone laser ablation (Fig. 1). For each included case, experts and trainees were randomized to perform either the localization or ablation task (i.e., one task was performed by the expert endourologist and one by the trainee; Fig. 1). For the study, stone localization was standardized to include systematic evaluation of the entire collecting system from upper to lower pole along with observation of any stones in the collecting system. If a stone was seen early in the process, the entire collecting system still was required to be seen before the task was considered complete. For stone ablation, we analyzed the first 20 seconds after laser activation. Stone ablation was performed using dusting settings (starting at 0.3 J and 50 Hz) using a 200-μm holmium laser fiber with pulse modulation (MOSES®). Before beginning stone ablation, the surgeon was permitted to move the stone into an easily accessible calix. Continuous pressure irrigation was used using either a pressure bag or fluid management system. When a trainee was assigned to either task, the experts were permitted to take control of the scope and confirm complete collecting system evaluation (after the stone localization task) or complete stone treatment (after the stone ablation task). All surgeons are aware of the videos being recorded, but were not aware of the purpose of this study. We also reviewed postoperative imaging associated with each video for stone-free status. We defined stone free as having no residual stone fragments seen on postoperative renal ultrasound. Because of cost and limited radiation exposure, renal ultrasound imaging is obtained routinely at our institution to evaluate for residual stone fragments and hydronephrosis around 6 weeks postoperatively.

OPEN IN VIEWER

FIG. 1.

Experimental protocol. Expert and trainee surgeons were assigned to stone localization or ablation tasks. Computer vision models were applied to each video to determine and compare pixel-based, automated performance metrics.

Karl Storz Flex XC digital ureteroscopes (1920 × 1080 pixels) were used for all cases, and all surgeons were right-handed. A 10/12 Fr access sheath was routinely placed into the ipsilateral ureter at the beginning of each case. Patients with anatomical abnormalities of their collecting systems were excluded from analysis. All videos were visually validated for quality and frames extracted at 30 frames per second (fps).

Results

Computer vision analysis of stone localization and ablation tasks

On average, the stone localization task took 72 ± 31 seconds (± standard deviation [SD]) for trainees and 58 ± 17 seconds for experts (p < 0.01). For this task, 2163 frames were analyzed from trainee surgeon videos and compared with 1731 frames from expert surgeon videos (Table 2). Loss of stone visibility was greater in trainee videos compared with expert videos (27% vs 4%, p < 0.01). In addition, trainee videos had a significantly lower screen occupancy by stone compared with expert videos during the localization task (5% vs 14%, p < 0.01). Frame-to-frame change in stone occupancy was the same between expert and trainee videos.

OPEN IN VIEWER

Table 2.

Frame-Based Comparison of Trainee and Expert Videos During the Stone Localization and Stone Ablation Tasks Using Stone Segmentation Model

	Trainee	Expert	p-Value
Localization task (total number of frames)	N = 2163	N = 1731
Loss of stone visibility, a N (%)	592 (27)	79 (4)	<0.01
Screen occupancy by stone, b median percentage (IQR)	5 (0–11)	14 (5–28)	0.03
Frame-to-frame change in stone occupancy, c median percentage (IQR)	1 (0–2)	1 (0–3)	0.7
Stone ablation task (total number of frames)	N = 5202	N = 5202
Loss of stone visibility (%)	428 (8)	359 (7)	0.99
Screen occupancy by stone, median percentage (IQR)	22 (8–34)	13 (6–22)	0.11
Frame-to-frame change in stone occupancy, median percentage (IQR)	3 (1–6)	2 (1–4)	<0.01

a

Loss of stone visibility is defined as proportion of frames with no stone seen out of all frames per task.

b

Screen occupancy by stone is defined as the percentage of total pixels on screen occupied by stone for each frame.

c

Frame-to-frame change in stone occupancy is defined as the change in frame-to-frame occupancy of stone.

For the stone ablation task, 5202 frames were analyzed from both trainee and expert videos, corresponding to the first 20 seconds of stone ablation. There were no differences in loss of stone visibility nor screen occupancy by stone in trainee and expert videos during stone ablation. However, during the stone ablation task, we observed greater frame-to-frame change in stone occupancy in trainee videos compared with expert videos (3% vs 2%, p < 0.01). A representative figure showing screen occupancy by stone over time for two videos (one trainee and one expert) is shown in Figure 2. In addition, individual metrics for trainees and experts can be seen in Supplementary Table S3 with comparisons of loss of stone and stone occupancy for individual surgeons depicted in Supplementary Figures S3 and Figure S4.

OPEN IN VIEWER

FIG. 2.

Example of trainee (red) vs expert (blue) observation of kidney stone as measured by percentage of frame occupied by stone. Corresponding frames are shown on lower panel. (A) The raw ureteroscopy image. (B) The ureteroscopy image and a blue overlay of the stone segmentation as identified by our computer vision model. The plot exemplifies instances of loss of stone observation and greater movement of the stone between frames (i.e., frame-to-frame change in stone occupancy) between experts and trainees.

Discussion

In this study, we demonstrate an application of a computer vision-based analysis to evaluate APMs in both expert and trainee surgical videos depicting fURS for the treatment of renal stones. We found differences between expert and trainee surgical videos when evaluating each APM. The application of an automated computer vision-based analysis for endoscopic stone treatment could be helpful as an objective feedback tool and chart for changes to trainee efficiency over time. Given the time limitations impressed on academic surgeons and trainees alike, these automated objective feedback tools could help supplement traditional methods of feedback.

The ability to adequately navigate the collecting system and observe renal stones depends on hand–eye coordination, memory, and spatial reasoning. Surgical experience impacts these factors and likely reflects the differences seen in the evaluated APMs. For example, during both stone localization and ablation, the trainee videos depicted more frequent loss of stone visibility compared with expert videos. This is likely because of absence of efficiency of motion and scope control while surveying the collecting system and treating the stones completely. Moreover, during the stone localization task, analysis of the videos revealed that experts observed the stones more completely than trainees with the median screen occupancy by stone being 14% and 5%, respectively. Less observation of stones by trainees compared with experts may reflect absence of scope control or a greater uncertainty in understanding stone position within the surgical anatomy.

In addition, we assessed the frame-to-frame change in stone occupancy during both the localization and ablation tasks. The frame-to-frame change in stone occupancy reflects an increase in endoscopic motion from movement of either the ureteroscope or the stone. Although there was no difference in frame-to-frame change in stone occupancy during stone localization, we found a greater frame-to-frame change in stone occupancy in trainee compared with expert videos during stone ablation. This likely represents less hand–eye coordination during laser ablation of kidney stones for trainees. This is expected as trainees may have more difficulty in maintaining scope position when trying to treat a stone or cause more stone motion during active lasering compared with experts. An example of frame-to-frame change in stone occupancy can be seen in Figure 2. Other factors such as respiratory movement, surgical anatomy, laser modality, and laser settings may impact stone motion during treatment. Future work is aimed at evaluating how these factors may impact surgical technique and efficiency of stone ablation for both trainees and experts. We choose to analyze only the first 20 seconds of stone ablation to allow for a direct comparison of dusting technique between groups and avoid potential confounding variables such as alternate treatment strategies (i.e., dusting vs pop-dusting) toward the end of stone ablation. Although isolating to the first 20 seconds of ablation allows for a more direct comparison of dusting technique between trainees and experts, it is not as applicable to other techniques of stone ablation.

Efforts to improve training for endoscopic surgical techniques predominantly focus on ex vivo simulation to supplement the intraoperative training experience. Multiple endoscopic simulators are currently available, and simulator-based training is associated with subsequent improved surgical performance in patients. ^8,9 However, ex vivo simulators are limited. They often lack surgical realism, are associated with additional costs, and are limited in applicability for skill acquisition in more advanced learners. ^8,10 There is a need for training tools that objectively evaluate competency during endoscopic urologic procedures in real time. In this study, we describe an objective real-time tool for evaluating expertise during endoscopic procedure and define specific APMs that distinguish surgical expertise. This tool is software based and could easily be included in training programs as an adjunct to surgical video review. Furthermore, application of this computer vision model could be used as a performance metric to aid in the assessment of completion of stone ablation during dusting.

Using machine learning-based tools to evaluate surgical performance has not been described for endoscopic stone procedure. However, these technologies have been applied for laparoscopic–robotic procedures. For example, Hung and colleagues previously found that APMs based on kinematic surgical data could reliably distinguish between trainee and expert surgeons performing steps of a robotic prostatectomy. ⁴ Several other studies have demonstrated the use of machine learning methods to identify objective measures such as task-evoked pupillary responses and substitch handling during vesicourethral anastomosis to distinguish robotic surgeon expertise during live procedure. ^11,12 Application of our models from a training perspective may clarify the learning curve for endoscopic stone procedure and provide actionable steps for improvement for trainees. In addition, such models could eventually provide real-time intraoperative feedback for endoscopic surgeons. The concept of automated segmentation is broadly applicable to other operations and can be used for things like identification of surgical anatomy and pathology report or image-guided procedure.

Beyond education, there is an association between APMs and surgical outcomes in robotic procedure. ^3,13 For instance, Lee and coworkers previously demonstrated that APMs captured during radical prostatectomy by the Intuitive Surgical System such as camera path length or instrument idle time could delineate risk of positive surgical margins. ¹³ Our findings demonstrate that APMs can be automatically identified during endoscopic kidney stone procedure.

We did find that experts more closely observed stone during the stone localization task, and these APMs correlated with quicker time to completion of the task. Limited postoperative imaging was performed in this study with most patients having a renal ultrasound done within 6 weeks. Although we observed no differences in overall stone-free rate between groups, this study was not specifically designed to assess stone-free rates, and this finding is limited by the low specificity of renal ultrasound. Further work is needed to assess whether the APMs identified in this study correlate with objective postoperative markers of quality such as stone-specific outcomes (e.g., stone-free rate on cross-sectional imaging), efficiency of stone ablation, or complications. Specifically, ongoing work is underway utilizing computer vision models to aid in real-time identification of sufficiently small dust that correlates with subsequent stone-free status. ¹⁴

The findings of this study must be interpreted in the context of some limitations. This was a relatively small, single-institution study at a tertiary referral center and, therefore, may not be applicable to all clinical settings. Only three expert surgeons and six trainees were evaluated, and all were right-handed. This does not provide enough diversity to evaluate the broad variety in surgical skill among both groups on a larger scale or in different clinical settings. In addition, the accuracy of our computer vision models is limited, particularly during active stone treatment as small stone debris generated during treatment may decrease accuracy of segmentation. ⁶ We plan on including more surgical videos demonstrating debris to “robustify” future versions of our model. We also plan on incorporating models that remove visual impairments from images to mask debris during endoscopic procedure. ^15,16

In this study, we were unable to control for type of anesthetic use during each case, introducing a potential confounding variable as differing degrees of respiratory motion could impact degree of difficulty of stone observation. Additional videos from a broader set of patients, clinical settings, surgeon experience, and scope types could improve future computer vision models. Future work will involve obtaining videos from multiple other institutions—including both trainees of differing experience levels and different surgeon-performed tasks. It will be vital to obtain videos across the spectrum of resident experience to help maximize the impact on applications to surgical training. Compiling these videos will help increase robustness of our metrics, as well as help associate them with surgical outcomes, including stone-free rate. Experimental design will need to account for the potential impact of each task on stone-free rate. Despite these limitations, we defined and evaluated APMs for endoscopic kidney stone procedure by evaluating differences between expert and trainee surgical videos. Future efforts will aim to improve and validate these APMs, as well as evaluate how they correlate with clinical outcomes.

Conclusions

Using an automated computer vision-based analysis, we evaluated APMs from surgical videos of endoscopic kidney stone procedure. Differences between expert and trainee videos were seen both during stone localization and stone treatment. Augmentation of this model with a broader set of videos could facilitate creation of enhanced educational and real-time feedback tools for fURS.