Editorial Comment on: Crowdsourcing Evaluation of Ureteroscopic Videos Using the Post-Ureteroscopic Lesion Scale to Assess Ureteral Injury by Kaler et al.

In this study, the authors tested the hypothesis that through a brief virtual training suite, crowd workers, presumably nonmedically trained, could be educated in recognizing whether ureteral injuries after ureteroscopy have occurred. This is the first study in which crowdsourcing has been used to identify surgical errors and injury. There is a growing body of literature validating crowdsourcing for ascribing degrees of technical proficiency; however, surgical outcomes are not driven by technique alone. Judgment and error recognition also keep our patients safe. Methods to rapidly and objectively identify errors could prove transformative and aid surgeons in optimal patient care. The results showed correlations in injury with expert surgeons reviewing deidentified ureteroscopy videos.

Certain limitations included the low number of videos in which injuries actually occurred and the reasonable, but not strong, correlations among the expert surgeon reviewers. Expert reviewers were provided access to videos through a YouTube link, whereas crowd workers used a standardized assessment platform provided by the company that has aggregated crowds for surgical skills assessment. This discrepancy may introduce reviewer platform bias that could influence scoring. Furthermore, crowd workers were reluctant to score videos at the extremes of the scoring tool, which is an observation seen in other studies. ¹ Tailoring the preassessment curricula to encourage a broader range of scoring and, as the authors note, describing for the crowd workers additional markers of injury could aid in increasing both the correlations with experts and the value of crowd review.

The major advantage of the crowdsourcing methodology over expert review is that a large number of surgical performances can be assessed in a short period of time. Labeling of injury events could ultimately be used to train machine learning and artificial intelligence algorithms. This is the initial step toward real-time error recognition and decision-support systems that will ultimately be in place to assist operative teams.