Editorial Comment for Holst et al.

The authors developed a unique surgical skills assessment system using crowdsourcing, named the Crowd Sourced Assessment of Technical Skills (CSATS). The crowdworkers had to have a greater than 95% rating in Human Intelligence Tasks, and should have correctly selected the better of two performance videos showing the Robotic Fundamentals of Laparoscopic Surgery block transfer task. Based on this simple task, 206 crowdworkers were selected from 250, and they scored at least one of five videos of robotic suturing performed by three urology residents (PGY-2, -4, -5) and two urology faculty members. The mean score of the crowdworkers' assessments correlated very well with the mean assessments score by the expert faculty, even though 176 of 206 crowdworkers assessed only one video and only one worker assessed all five videos. The time taken to assess the crowdworkers, however, was only 2 hours and 50 minutes.

Skill assessment is an important element of surgical education, not only for simple surgical tasks but also for complicated real surgery. In Japan, we have conducted a laparoscopic surgical skill assessment system, called the Endoscopic Surgical Skills Qualification system, for 10 years not only in urologic laparoscopy ^{1 –3} but also other surgical subspecialties including gastroenterology, ⁴ pediatrics, ⁵ and gynecology. The total number of applicants every year is now about 800, and there are now 300 referees. Over the 10-year period, the qualification rate for the total of 6096 applicants was 48.2%. Skill assessment on unedited videotapes of entire surgical performances of real surgery is a hard task, taking a lot of time, at least 2 hours for one assessment, which restricts the widespread use of skill assessments of real surgery.

If CSATS could play some role in the skill assessment of real surgery, such as the ESSQ system, surgical skill assessment would be more widely performed worldwide. The rapid response by crowdworkers as shown in this article is also impressive and useful.

A big question arising from this article, however, is whether this CSATS system would work for skills assessment of real human surgery. Could surgery-naïve crowdworkers assess complicated real surgery appropriately? The length of the videos assessed by crowdworkers in this article was only 1 to 5 minutes, while real surgery would be more than 1 hour. The next step for CSATS should be the challenge of effectively assessing the skills in a part of real surgery, such as mobilization of the colon in transperitoneal kidney surgery.