Abstract
The present study examined whether inexperienced individuals can evaluate ultrasonography (US) images using a new rating scale and visual aids. A US image assessment tool and visual aids in the form of alignable comparison standards (one good and one poor example) were developed to rate images of the hepatorenal interface along eight characteristics. Twenty novice participants from Amazon.com’s Mechanical Turk service used the visual aids to evaluate 42 US images in good and poor categories. The results revealed that crowdworkers could distinguish the good from poor US images. Their ratings were highly and positively correlated with those of experts. However, the crowdworkers’ ratings were more severe than those of the experts. These results suggest that inexperienced individuals can use a rating scale and visual aids to reliably distinguish good from poor US images but may require a more extensive set of visual cues or additional training to match ratings provide by experts.