Validation of a Skin-Lesion Image-Matching Algorithm Based on Computer Vision Technology

Introduction

Accurate, resource-conserving methods to classify skin lesions are in great demand given the rapidly increasing incidence of melanoma globally ¹ and the challenges of accessing timely, high-quality care due to the global dermatology workforce shortage. ² Dermoscopy, or epiluminescent microscopy, has improved the accuracy of melanoma classification compared with unaided visual diagnosis by clinicians, but dermoscopes are relatively expensive, limiting their usefulness in low-resource countries, and the accuracy rates vary markedly depending on the experience level of the user. ^{3 –7} Several groups have developed computer algorithms to improve classification of skin lesions from digital images, but many require dermoscope images, and most have only been scored on small datasets. ^{8 –10}

We have developed a unique computerized classification system that combines aspects of computer vision technology with big data methodology. The system uses a patented algorithm combined with a large proprietary database of diagnosed lesion images to match new images with database images. In the current study, we evaluated the accuracy of the algorithm to classify melanoma lesions 10 mm or larger in a test database of histopathologically diagnosed skin lesions.

Materials and Methods

Results

In total, 1,900 participants were recruited by study personnel and agreed to allow study team members to capture digital images of their skin lesions for inclusion in the skin-lesion image database. Study personnel asked participants to report information about specific demographic variables, and for each variable over 90% of all recruited participants provided responses (Table 1). In addition to the recruited participants, we enriched the study database with images of histopathologically diagnosed melanoma lesions, for a total of 2,202 individual image donors. Demographic data were unavailable for the individuals who provided the additional melanoma images that we acquired to enrich the study database.

Overall, the recruited participants were fairly young, with slightly more than half under the age of 35 years, but older participants, who are likely to present with different conditions and skin types and tones, were well represented (Table 1). Participants were predominantly female and ethnically diverse, with just under one-third identifying as white/Caucasian and nearly 40% identifying as Hispanic or Latino.

The study database of 11,780 images (Table 2) included all 11,478 images from the 1,900 participants and an additional 302 images from the DermNet NZ database. Melanoma diagnoses accounted for about 1 in 10 database images and nearly two-thirds of query images. The distribution of images by diagnosis was similar for nonmelanoma query images and the database images, as expected given that the nonmelanoma query images were randomly selected from the database images. The combined nevus diagnoses accounted for more than four-fifths of all nonmelanoma database images and just over three-quarters of all query diagnoses.

Nearly every measure of accuracy of the algorithm to correctly identify melanoma and nonmelanoma lesions 10 mm or larger exceeded 90% (Table 3). The algorithm accurately identified more than 90% of true melanoma and true nonmelanoma lesions. Of those the algorithm identified as having melanoma, more than 94% in fact had melanoma; of those the algorithm identified as not containing melanoma, more than 85% did not have melanoma. Overall, the algorithm accurately identified nearly 91% of true melanomas and true nonmelanomas, with a lower confidence limit greater than 87%.

In addition, the likelihood ratio tests were highly discriminatory. The odds that a true melanoma would be accurately identified were more than 10 times greater than the likelihood that a true nonmelanoma would be incorrectly identified as melanoma. The odds that a true melanoma would be incorrectly identified as nonmelanoma were only 1/10th the odds that a true nonmelanoma would be accurately identified.

Discussion

The image-matching algorithm performed with high accuracy for the classification of larger melanoma lesions, exceeding reported accuracy rates that typically range from 70% to 86% among experienced board-certified dermatologists, the current gold standard for clinical diagnosis of skin lesions. ^{15 –17} It is of importance that, compared with a study by Carli et al. ¹⁸ that specifically examined accuracy rates for classification of large lesions (10 mm or larger) as in the present study, the reported accuracy rates of the image-matching algorithm outperformed those of practicing dermatologists visually diagnosing lesions (algorithm versus naked eye examination: sensitivity, 90.4% versus 82.9%; specificity, 91.5% versus 75.8%; and overall accuracy, 94.5% versus 79.5%). The same study ¹⁸ also reported accuracy rates for visual examination combined with dermoscopy, and the reported sensitivity of the algorithm was slightly lower (algorithm versus visual examination plus dermoscopy, 90.4% versus 93.3%), but the specificity and overall accuracy were substantially higher for the algorithm (algorithm versus dermoscopy: specificity, 91.5% versus 77.3%; overall accuracy, 90.8% versus 85.6%). Several other studies reported dermatologists' diagnostic accuracy rates using dermoscopy, and the image-matching algorithm generally achieved comparable or higher sensitivity and specificity. ^{3 –7,15 –17}

The algorithm also demonstrated high PPV and NPV in this test database enriched for melanoma images. These findings were not surprising given that PPV and NPV are affected by the prevalence of the condition, and the prevalence in this test dataset was high. We would therefore expect the algorithm to achieve lower PPV in a population with a lower prevalence of melanoma, as would be encountered in a typical dermatology or primary care setting. However, the lower PPV and NPV expected in a general or screened population do not negate the value of this or any other screening test because if the sensitivity of the test is very high, the potential benefits of the test due to increased survival and reduced healthcare costs through earlier detection may be greater than the cost of performing the test.

The likelihood ratio test results, mathematically related to sensitivity and specificity, are not affected by the population prevalence of the disease. The algorithm's positive likelihood ratio was above the threshold of 10, which is considered strong conclusive evidence that the disease is likely to be present, and the algorithm's small negative likelihood ratio was at the threshold of 0.1, which is considered conclusive evidence that the disease is not likely to be present. ¹³

We designed the algorithm to model dermatologists' approach to skin-lesion analysis: mental matching of a query image (a patient's lesion) with a personal “database” of images learned during medical school, residency, and routine clinical practice. Accurate classification is limited only by the size of the database and the user's recall ability. Our system replicated this model by creating a proprietary skin-lesion image database that included diagnosed lesions from participants diverse in age, race/ethnicity, skin type, and specific diagnosis. The system will increase in robustness over time through the addition of new participant records to the system database, similar to the continually increasing expertise of dermatologists through the daily examination of skin lesions. However, unlike dermatologists, who will ultimately retire and take their expertise with them, our image database will continually increase in the number of records and the diversity of participants, lesion types, and presentations, including skin conditions in children.

During algorithm development, we discovered that lesion size and pathology were complex drivers of image-signature design. We therefore elected first to maximize algorithm accuracy for the identification of larger lesions that are more likely to be clinically important (presented here). However, we believe that optimization for larger, riskier lesions does not diminish the algorithm's usefulness given the critical global need for simple, inexpensive, and accurate tools to classify skin lesions. Other highly effective medical technologies have been developed that have a lower limit of resolution, such as positron emission tomography, which has been established as an important clinical tool in the evaluation and management of cancer despite the limitation of its use to larger tumors. ¹⁹ However, in a second phase of algorithm development, we will conduct additional assessments of smaller lesions and apply that information to improve algorithm performance for those lesions as well as for nonmelanoma malignancies, including basal and squamous cell carcinomas, and pediatric lesions. Subsequent phases could include development of lesion evolution tracking to compare multiple images of single lesions over time because detection of changes in the growth or visual presentation of lesions is another important dimension to the classification of suspicious lesions.

A potential limitation of the system is the lack of demographic information associated with images acquired from DermNet NZ that were used to enrich the database for melanoma and that were used as query images. We elected to enrich the database with 302 histopathologically confirmed melanoma lesion images due to the low prevalence of melanoma in the population of individuals we recruited to donate skin-lesion images for the study database. The additional images provided the algorithm with a greater representation of melanoma characteristics upon which to optimize the image-matching query. If the query images and the enriched melanoma images were obtained predominantly from white individuals, then the present study results may not directly reflect the robustness of the matching algorithm within diverse populations. However, all nonmelanoma images in the study database were derived from a highly diverse population of skin-lesion image donors, and the algorithm demonstrated very high specificity, NPV, negative likelihood ratio, and overall accuracy, suggesting that the algorithm classification works well even in diverse populations. It is of importance, however, that the algorithm uses only the information in the image, without supplementary demographic information, to classify the image, and the accuracy estimates—the correct classification of melanoma and nonmelanoma images—are internally valid and not affected by the lack of demographic data for this subset of images.

Another potential limitation of our image-matching algorithm is that it currently relies on a database of clinically diagnosed, rather than biopsy-proven, skin lesions. However, the CBIR technology that drives our algorithm exploits visual characteristics of the skin-lesion images that dermatologists have identified as common to a given lesion type. Therefore, even though diagnostic error is likely more frequent in our database of clinically diagnosed lesions than it would be in a database with biopsy-proven diagnoses, the visual characteristics consistent with the lesion types assigned by the dermatologists are sufficient to inform the algorithm and result in exceptionally high sensitivity and specificity of the matches. It is possible that a comparably sized database of biopsied images would give higher classification accuracy results, but the standard of care for lesion types that are clearly benign is clinical diagnosis, and therefore development of this algorithm would not be feasible if the database were restricted to biopsy-proven lesions.

In conclusion, this newly developed algorithm has the potential to improve classification of larger melanoma skin lesions, and ultimately all skin cancers, using digital images captured with low-cost cameras.