Skin cancer image recognition based on similarity clustering and attention transfer

Abstract

BACKGROUND:

Melanoma is a tumor caused by melanocytes with a high degree of malignancy, easy local recurrence, distant metastasis, and poor prognosis. It is also difficult to be detected by inexperienced dermatologist due to their similar appearances, such as color, shape, and contour.

OBJECTIVE:

To develop and test a new computer-aided diagnosis scheme to detect melanoma skin cancer.

METHODS:

In this new scheme, the unsupervised clustering based on deep metric learning is first conducted to make images with high similarity together and the corresponding model weights are utilized as teacher-model for the next stage. Second, benefit from the knowledge distillation, the attention transfer is adopted to make the classification model enable to learn the similarity features and information of categories simultaneously which improve the diagnosis accuracy than the common classification method.

RESULTS:

In validation sets, 8 categories were included, and 2443 samples were calculated. The highest accuracy of the new scheme is 0.7253, which is 5% points higher than the baseline (0.6794). Specifically, the F1-Score of three malignant lesions BCC (Basal cell carcinoma), SCC (Squamous cell carcinomas), and MEL (Melanoma) increase from 0.65 to 0.73, 0.28 to 0.37, and 0.54 to 0.58, respectively. In two test sets of HAN including 3844 samples and BCN including 6375 samples, the highest accuracies are 0.68 and 0.53 for HAM and BCN datasets, respectively, which are higher than the baseline (0.649 and 0.516). Additionally, F1 scores of BCC, SCC, MEL are 0.49, 0.2, 0.45 in HAM dataset and 0.6, 0.14, 0.55 in BCN dataset, respectively, which are also higher than F1 scores the results of baseline.

CONCLUSIONS:

This study demonstrates that the similarity clustering method enables to extract the related feature information to gather similar images together. Moreover, based on the attention transfer, the proposed classification framework can improve total accuracy and F1-score of skin lesion diagnosis.

Keywords

Skin cancer melanoma similarity clustering attention transfer deep metric learning

1 Introduction

Skin is the largest organ in the body, making it possible for malignant neoplasms to grow in most anatomical sites. There are two main types of skin cancer: melanoma and non-melanoma. The most common non-melanoma tumors are basal cell carcinoma and squamous cell carcinoma. The degree of infiltration of them is shown in Fig. 1. Although melanoma accounts for about 1% of all skin cancers diagnosed, it causes most of the deaths. More specifically, according to the reports from the American Cancer Society, the number of melanoma deaths is expected to increase by 6.5 percent in 2022 [1]. Besides, based on the research published by the National Cancer Institute of China in March, there were 3,800 deaths from cutaneous malignant melanoma in 2016, including 2,100 men and 1,700 women [2]. The estimated five-year survival rate for melanoma is over 99 percent if detected in the early stage. However, the survival rate falls to 68 percent when cancer reaches the lymph nodes and 30 percent when cancer metastasizes to distant organs [1]. It is thus clear that early detection is vital for patients with skin cancers, especially for those with melanoma.

Fig. 1

Three types of skin cancer. Obviously, melanoma is an extremely invasive cancer that can penetrate basal layer.

The existing routine clinical examination for skin lesions is primarily diagnosed visually [3] which could be affected by the similarity between skin lesions. Figure 2 exhibits the skin lesion images collected by the dermatoscope and there is remarkable visual similarity among them. More precisely, both the same disease showing different morphological images illustrated in Fig. 2(a) and the different diseases showing similar morphological images illustrated in Fig. 2(b) are phenomena particularly prominent in skin lesions, which exacerbates the difficulty of automatic classification of skin cancers. In the clinical scenario, there are three challenges in using Artificial Intelligence (AI) to diagnose skin cancers. Firstly, the skin cancer images exhibit diversity as a result of the differences in the diagnostic equipment and individuality. Secondly, the annotations of skin cancer images are laborious and must be authorized by experts. Finally, as shown in Fig. 2(b), due to the presence of high similarity between the melanoma and melanocytic nevi in patterns such as colors, shapes, and size, it makes the common classification methods difficult to distinguish them.

Fig. 2

From the top row to the bottom row illustrated in (a), there contains blue nevus, mixed nevus, intradermal nevus, congenital nevus, and dysplastic pigmented nevus sequentially. In (b), the green background images in rows 1 and 3 are benign melanocytic nevi, and the red background images in rows 2 and 4 are malignant melanomas. High degree of similarity between melanocytic nevus and melanoma in terms of color, texture, and contour are presented.

Therefore, the objective of this paper is to explore a novel classification method for skin cancers from the perspective of similarity. We utilized deep metric learning to make similarity clustering and then adopted the attention transfer strategy to transfer similarity knowledge for the diagnosis of skin cancers. Deep metric learning aims to learn a representational metric space where the same category data points are close to each other and the different ones are apart from each other. The attention transfer strategy belongs to the technical approach of knowledge distillation. Consequently, for better utilization of the similarity information of skin cancer images, we integrated these two methods to form an automatic skin cancer recognition framework that includes data pre-processing, data training, and data inference.

The rest of this paper is outlined as follows. Section 2 presents a literature review of skin cancer classification and medical applications with deep metric learning. Technical methodologies of similarity clustering and classification with descriptions of the utilized datasets are provided in Section 3. The experimental results are presented in Section 4. Section 5 presents the discussion of this study and this paper ends with conclusions in Section 6.

2 Related works

2.1 Skin cancer classification

Recently, with the development of Deep Learning (DL), the skin lesion image classification methods sprung up. The major advance in DL-based skin cancer diagnosis is the research work of Esteva et al. [3] published in Nature. They utilized the InceptionV3 to fit the 129,450 clinical and dermatoscopic images consisting of 2032 different categories of skin lesion diseases. In a three-class disease partition, the CNN achieved 72.1±0.9% overall accuracy and two dermatologists attained 65.56% and 66.0% accuracy on a subset of the validation set. In the nine-class disease partition, the CNN achieved 55.4±1.7% overall accuracy whereas the same two dermatologists attained 53.3% and 55.0% accuracy.

However, there are two deficiencies of DL-based skin cancer diagnosis. Firstly, from the data perspective, the robustness of the deep learning model is inadequate because of the semantic gap between skin cancer images generated by the inter-class similarity and intra-class dissimilarity. Moreover, long-tailed data distribution makes representative information of tail classes learned by the DL model insufficient which brings barely satisfactory results. Secondly, although the ensemble DL models could enhance the ability of feature extraction of skin cancer images, the massive parameters could cause over-fitting and inconvenient deployment applications. As a result, a balanced relationship between the model and data should be established with an appropriate method to keep skin cancer classification accuracy.

The International Skin Imaging Collaboration (ISIC) provide high-quality dermatoscopic images to boost the research on automated diagnosis. Maria et.al [22, 23] proposed skin lesion segmentation algorithms with the active contour model. References [24 –26] were DL-based methods for skin lesion segmentation. These works are important process for subsequent diagnosis.

2.2 Medical application of deep metric learning

Distance measurement is a technical method of traditional metric learning, which is widely used in the CBIR system. It is not only an important technical procedure of retrieval but also a mathematical criterion of similarity measurement. In 2006, Rahman et al. achieved the skin disease image retrieval by color space transformation, lesion region segmentation, texture feature extraction, and constructed similarity matching function combined with Bhattacharyya distance and Euclidean distance on 358 skin disease images [4]. With the development of deep learning, it promotes the progress of metric learning technology, which makes it develop into deep metric learning (DML). A variety of notable applications of deep metric learning have occurred such as face verification [5], and person re-identification [6]. In 2019, Yang et al. [7] firstly developed a retrieval system for pathological image analysis by using deep metric learning. Based on the published dataset Kimia path24, the Recall@1 reached 97.89%. Under the giant influence of pandemic Covid-19, Zhong et al. [8] utilized deep metric learning to obtain superior performance both in retrieval and diagnosis tasks. The averaged area under the curve (AUC) of CXR-EHR combined prediction is 91.3%. According to the literature [9] published by Cornel Tech and Facebook in 2020, after 13-year exploration and application, deep metric learning has become one of the most attractive research fields of machine learning. Therefore, the implementation of deep metric learning in skin cancer image research would be a certain degree of improvement for solving the semantic gap, which in turn improve the diagnosis accuracy.

From the related works described above, we can observe that the limitations of previous studies are that the utilization of the similarity information is not considered and the deep metric learning is not adopted in the skin cancer images. In our study, the similarity clustering was conducted by deep metric learning to obtain the similarity information. Moreover, we combined the similarity information and category information by attention transfer strategy for final skin cancer classification.

3 Materials and methods

3.1 Datasets

The International Skin Imaging Collaboration (ISIC) is an international effort to improve melanoma diagnosis and the ISIC Archive contains the largest publicly available collection of quality controlled dermatoscopic images of skin lesions. In order to evaluate and test the proposed automatic skin cancer recognition framework, experiments were performed with the ISIC-2019 Dataset containing BCN2000 (BCN) [10], HAM10000 (HAM) [11], and MSK [12, 13]. There are 8 categories of the ISIC-2019 Dataset including 3 malignant ones: Basal cell carcinoma (BCC), Melanoma (MEL), Squamous cell carcinomas (SCC) and 5 benign ones: Actinic keratosis (AK), Benign keratosis (BKL), Dermatofibroma (DF), Nevus (NV), Vascular (VASC). Table 1 lists the number of each category and the unk means the data is not part of those three main datasets. Table 2 presents the distribution on Five main anatomical predilection sites: Head/Neck (HN), anterior torso (A), posterior torso (P), lower extremity (Lo), and upper extremity (Up). Besides, the number of lesions on palms/soles, oral/genital and lateral torso is 398, 59 and 54 respectively. And 2631 samples are not labeled with anatomical site. Visualization is demonstrated on Fig. 3.

Table 1
Original data distribution of the ISIC-2019

Class\Dataset HAM BCN MSK UNK Total

Cancer MEL 1113 2857 215 337 4522

BCC 514 2809 0 0 3323

SCC 197 431 0 0 628

Non-Cancer AK 130 737 0 0 867

NV 6705 4206 415 1549 12875

BKL 1099 1138 189 198 2624

VASC 142 111 0 0 253

DF 115 124 0 0 239

Total 10015 12413 819 2084 25331

Class\Dataset		HAM	BCN	MSK	UNK	Total
Cancer		MEL	1113	2857	215	337	4522
	BCC	514	2809	0	0	3323
	SCC	197	431	0	0	628
Non-Cancer	AK	130	737	0	0	867
	NV	6705	4206	415	1549	12875
	BKL	1099	1138	189	198	2624
	VASC	142	111	0	0	253
	DF	115	124	0	0	239
Total	10015	12413	819	2084	25331

Table 2

Original data distribution of the ISIC-2019 on five main anatomical sites: head/neck (HN), anterior torso (A), posterior torso (P), lower extremity (Lo), upper extremity (Up)

Class\Site	HN	A	P	Lo	Up	Total
MEL	880	1331	430	796	724	4161
NV	767	3699	1888	2876	1325	10555
BCC	1131	1114	186	462	358	3251
AK	598	92	4	66	79	839
BKL	1001	445	230	412	234	2322
DF	0	43	2	138	52	235
VASC	36	78	22	58	23	217
SCC	174	113	25	182	115	609
Total	4587	6915	2787	4990	2910	22189

Table 3

Original data resolution of ISIC-2019

	HAM	BCN	MSK
Resolution (h×w)	450×600	1024×1024	680×1024
			704×1007
			685×1024
			602×639
			682×1024
			717×1017

Fig. 3

Two histograms are presented in (a). The left histogram shows the number of the eight skin lesions on the five main anatomical sites and the right one shows the number of skin lesions of four datasets. (b) presents the examples of eight skin lesions.

3.2 Pre-processing and similarity clustering

3.2.1 Pre-processing

Fine-grained appearance variations of skin lesions are evident as the different imaging devices and physical conditions. The common labeling process ignores the inter-class similarity and intra-class dissimilarity which only concerns the category information. Moreover, the randomly splitting data would not guarantee the validity of evaluation indexes due to the unbalanced data distribution in categories and appearances. Therefore, based on the CLEAR Derm published on JAMA Dermatology [14], we proposed the four-step approach to make reasonable processing of the skin lesion images. The order is as follows: (1) Resize, (2) Similarity Cluster, (3) Black Edge Removal, (4) Data Partitioning.

Resize was the first. For efficient numerical computing and avoiding information redundancy, we uniformly resized the images to 224×224. Similarity clustering was an unsupervised learning process to extract the features with high degree similarity in skin lesion images. After that, it would gather them in a specific cluster for the following operations.

Black edge removal was a traditional image processing method that includes Otsu threshold segmentation, morphological binary operations, and contour extraction. According to the clustering results of BCN in previous step, the removal operation was concentrated on the clusters containing images with black edge and 3506 of them were processed. Data partitioning was conducted with the clustering results to ensure 8 categories of skin lesion images were in both training and validation sets.

3.2.2 Similarity clustering

The common deep learning classification methods are as follows: an image is sequentially wrapped into probability distribution over clinical classes of skin disease which makes the ability to handle images with high similarity not enough. Therefore, we adopted the unsupervised clustering method with deep metric learning to take advantage of similarity feature information within among of them.

Deep Cluster [15] is representative prototype learning work that introduces a “prototype” as the centroid for a cluster and the feature representations are fed into a clustering algorithm to produce cluster assignments. K-means is utilized as the clustering algorithm to cluster the feature vectors z = f_θ (x) into K distinct groups where the f_θ is the network and x is the input image. The groups are used as pseudo labels and the cross-entropy loss function is used in the training stage. In this work, the cross-entropy loss function was replaced by deep metric learning loss function. This loss function contains different similarity characteristics to improve the accordance of the cluster assignments.

There are two classic loss functions used in deep metric learning. They are contrastive loss [16] and triplet loss [17]. The goal of contrastive loss is to make the distance between positive pairs smaller and the distance between negative pairs larger than another threshold λ as shown in Equation (1): $L = (1 - Y) ⌊ S_{ij} - λ ⌋ + S_{ij}$ (1)

This characteristic is known as self-similarity. It is only computed by the feature vector pairs. A negative pair with large cosine similarity value is named as hard negative pair. Instead of considering pairs, triplet loss shown in Eq. (2) uses the triplet where an anchor is used to construct the positive pairs to consider the relationship from other pairs of the same anchor. $L = ⌊ S_{an} - S_{ap} + λ ⌋$ (2)

Consequently, this loss function pays more attention on positive relative similarity and enforces the similarity of a negative pair to be smaller than that of a positive one over a given margin λ.

However, Multi-Similarity (MS) loss [18] could be more appropriate for skin cancers images. This loss function includes the property of inter-class similarity and intra-class dissimilarity. The formulation of multi-similarity loss is as follows: $L = \frac{1}{m} \sum_{i = 1}^{m} {\frac{1}{a} log [1 + \sum_{k \in P_{i}} e^{- a (S_{ik} - λ)}] + \frac{1}{β} log [1 + \sum_{k \in N_{i}} e^{β (S_{ik} - λ)}]}$ (3) where the m refers to the batch of the samples, the α, β, and λ are the hyper-parameters. We set α = 2, β = 40 and λ = 0.5 in the experiment. The S_ik means the similarity metric calculated on normalized embedding vectors with cosine similarity as shown in Equation (4). $S_{ik} = \frac{dot (E (x_{i}), E (x_{k}))}{{∥ E (x_{i}) ∥}_{2} \cdot {∥ E (x_{k}) ∥}_{2}}$ (4)

The pair of samples x_i and x_j is labeled y_i and y_j respectively, and it is a positive pair if the y_i is equal to y_j, otherwise, it is a negative pair. The hard mining process is based on similarity metric and defined as Equations (5) and (6) where ɛ is a given margin. For an anchor x_i, a positive pair {x_i, x_j} is selected if S_ij satisfies the Equation (5). $S_{ij}^{+} < max_{y_{k} \neq y_{i}} S_{ik} + ɛ$ (5) $S_{ij}^{-} < min_{y_{k} = y_{i}} S_{ik} - ɛ$ (6)

3.3 Classification framework

In 2014, Hinton et al. proposed the concept of knowledge distillation for model compression, which allowed small models with poor generalization to learn the “knowledge” of large teacher models with good generalization. In the distillation process of the classification model, the probability distribution of the SoftMax function becomes softer as the temperature coefficient increases, i.e., it provides more information as to which classes the teacher found more similar to the predicted class, and this information is called the “knowledge” of the model. This process can be expressed as follows: $p_{i} = \frac{exp (\frac{z_{i}}{T})}{\sum_{j} exp (\frac{z_{j}}{T})}$ (7) where p_i is the category probability, z_i and z_j are the category output vectors, and T is the temperature coefficient. During the training process, knowledge is transferred from the teacher model to the student model by minimizing the loss function. In 2017, Zagoruyko et al. [19] proposed the attention transfer method to make the student model mimic the attention maps of the teacher model, the process can be represented as: $L_{AT} = L (W_{s}, x) + \frac{β}{2} \sum_{j \in I} {∥ \frac{Q_{s}^{j}}{{∥ Q_{s}^{j} ∥}_{2}} - \frac{Q_{T}^{j}}{{∥ Q_{T}^{j} ∥}_{2}} ∥}_{2}$ (8) where T, S, W_s denote the teacher model, student model, and student model weights, respectively. I denotes the total number of feature layers, $Q_{S}^{j}$ and $Q_{T}^{j}$ represent the normalized attentional feature maps of the student model and teacher model at the j-th layer, β is the hyperparameter; L (W_s, x) denotes the cross-entropy loss function, the sum of the cross-entropy loss function and feature maps difference function is L_AT. According to the back-propagation principle and the chain derivation rule, the L_AT loss function can update the model parameters using both category information and attention feature map information. In our method, the attention maps of the deep metric learning model were used as “knowledge” to transfer to the classification model, and the number of parameters of both models is kept consistent.

The framework is illustrated in Fig. 4. In the training stage, a batch of images is wrapped into the metric teacher model (MTM) and classification student model (CSM) simultaneously. In MTM (CSM), images are converted to intermediate layer features illustrated in A (B), according to the attention transfer theory, the output features in C are utilized for CSM parameters update and gratitude calculation with the cross-entropy loss function. Besides, the MTM is frozen in the training stage. In the inference stage, an unknown image is wrapped into a probability distribution over clinical classes of skin lesion and the final diagnosis result corresponds to the maximum probability value.

Fig. 4

The skin lesion diagnosis framework consists of (a) and (b). In (a), there exists a four-step approach to address the original images. Red box contains the images with black edge and corresponding results are presented in green box. In data partitioning step, the boxes with three different colors contain the images for training, validation and inference respectively. After the sequential operations, the well processed images and clustering model parameters are ready for fitting the classification model based on the attention transfer method as shown in (b).

3.4 Evaluation

The following table shows the concepts of the TP, TN, FP and FN.

For evaluating the classification performances, the F1-Score of 8 categories were calculated respectively. This metric was considered as a harmonic mean of precision and recall. Accuracy was also used to measure the total classification performance. $Accuracy = \frac{Predicted Right Samples}{Total Samples}$ (9) $precision = \frac{tp}{tp + fp}$ (10) $recall = \frac{tp}{tp + fn}$ (11) $F 1 Score = \frac{2 * precision * recall}{precision + recall}$ (12)

3.5 Implementation

This automatic skin cancer diagnosis framework was built on Pytorch [20] with ResNet [21]. The common classification method of ResNet was considered as baseline which would be compared with our classification framework. All experiments were conducted on a workstation with a GeForce RTX 3090 NVIDIA GPU.

Table 4
Concepts of the TP, TN, FP and FN

Original Label Detected Positive Detected Negative

Positive TP FN

Negative FP TN

Original Label	Detected Positive	Detected Negative
Positive	TP	FN
Negative	FP	TN

4 Experiment results

Our method consists Similarity Clustering and Attention Transfer to make automatic diagnosis for skin cancer images. For better fitting the parameters, in training process, we set 100 epochs for both stages and they could be completed in an hour. Corresponding results are illustrated as follows.

4.1 The results of similarity clustering

As shown in Fig. 5, there exhibits the cluster results on HAM, BCN and unk datasets, MSK is excluded due to the relatively small quantity and consistent in form. It can be observed that there are various similar appearances and dissimilar ones existing the images such as skin color, lesion size, hair and markers. In the similarity clustering, the similar images were formed as clusters. We set the number of clusters as 16 according to the number of the images. Then, we pickled 8 clusters for testing, the rest ones are chosen as training and validation. There are artifacts existing in the skin images such as hair (Cluster 3), rulers (Cluster 7) and markings (Cluster 8). Besides, different lighting conditions (Cluster 1 and 2) are presented in the dataset.

Fig. 5

The results of similarity clustering.

4.2 The results of attention transfer

In our Attention Transfer experiments, we only utilized HAM and BCN dataset for recognition. The table of the dataset split is listed as follows.

Table 5
Dataset split in train, validation and test

Dataset Train Validation Test Total number

HAM 4936 1235 3844 10015

BCN 4830 1208 6375 12413

Dataset	Train	Validation	Test	Total number
HAM	4936	1235	3844	10015
BCN	4830	1208	6375	12413

The validation results are listed in Tables 6 and 7. It is worth mentioning that, we merged the samples of the HAM and BCN both in the training and validation stage for better feature extraction and model selection. The test results of HAM are listed in Tables 8, 9. Tables 10, 11 consist the test results of BCN. The total accuracy was calculated with 8 categories. Layers marked as √ indicate that their parameters were trained in the training process of Attention Transfer. Layer 1 and 2 usually extract the low-level features, on the contrary, layer 3 and 4 extract the high-level features. We listed 9 combinations of layers to make performance comparison with baseline.

Table 6

The total accuracy of eight categories of validation set

Method	Layer 1	Layer 2	Layer 3	Layer 4	accuracy
Attention Transfer	√	√	√	√	0.7171
		√	√	√	0.7073
			√	√	0.7253
				√	0.7077
		√		√	0.7171
		√	√		0.6864
	√	√			0.6831
	√			√	0.7118
	√		√		0.7126
Baseline	√	√	√	√	0.6794

Table 7

The F1 score of eight categories of validation set

Method	Layer1	Layer2	Layer3	Layer4	F1 Score
					NV	BCC	SCC	MEL	AK	BKL	DF	VASC
Attention Transfer	√	√	√	√	0.85	0.7	0.34	0.58	0.44	0.46	0.42	0.52
		√	√	√	0.84	0.71	0.37	0.56	0.37	0.48	0.34	0.53
			√	√	0.85	0.73	0.37	0.58	0.48	0.47	0.41	0.53
				√	0.84	0.69	0.32	0.58	0.43	0.47	0.36	0.52
		√		√	0.85	0.71	0.34	0.55	0.44	0.45	0.40	0.59
		√	√		0.82	0.65	0.33	0.57	0.38	0.44	0.42	0.51
	√	√			0.82	0.67	0.2	0.56	0.4	0.43	0.24	0.53
	√			√	0.84	0.71	0.36	0.58	0.44	0.44	0.33	0.54
	√		√		0.85	0.69	0.33	0.55	0.38	0.46	0.27	0.47
Baseline	√	√	√	√	0.82	0.65	0.28	0.54	0.37	0.42	0.19	0.58

Table 8

The total accuracy of test set of HAM

Method	Layer1	Layer2	Layer3	Layer4	accuracy
Attention Transfer	√	√	√	√	0.655
		√	√	√	0.669
			√	√	0.656
				√	0.658
		√		√	0.680
		√	√		0.636
	√	√			0.649
	√			√	0.652
	√		√		0.671
Baseline	√	√	√	√	0.649

Table 9

The F1 score of test set of HAM

Method	Layer1	Layer2	Layer3	Layer4	F1 Score
					NV	BCC	SCC	MEL	AK	BKL	DF	VASC
Attention Transfer	√	√	√	√	0.82	0.49	0.2	0.45	0.05	0.35	0.23	0.42
		√	√	√	0.83	0.49	0.11	0.46	0.13	0.39	0.15	0.46
			√	√	0.82	0.48	0.16	0.42	0.1	0.38	0.23	0.43
				√	0.82	0.47	0.16	0.44	0.14	0.4	0.19	0.53
		√		√	0.84	0.50	0.16	0.43	0.11	0.39	0.27	0.51
		√	√		0.80	0.44	0.16	0.41	0.13	0.39	0.21	0.62
	√	√			0.81	0.49	0.18	0.46	0.19	0.42	0.18	0.61
	√			√	0.82	0.45	0.20	0.43	0.08	0.35	0.19	0.50
	√		√		0.83	0.51	0.14	0.42	0.05	0.36	0.10	0.67
Baseline	√	√	√	√	0.81	0.45	0.21	0.42	0.16	0.38	0.21	0.59

Table 10

The total accuracy of test set of BCN

Method	Layer1	Layer2	Layer3	Layer4	accuracy
Attention Transfer	√	√	√	√	0.521
		√	√	√	0.512
			√	√	0.526
				√	0.510
		√		√	0.530
		√	√		0.508
	√	√			0.500
	√			√	0.530
	√		√		0.525
Baseline	√	√	√	√	0.516

Table 11

The F1 score of test set of BCN

Method	Layer1	Layer2	Layer3	Layer4	F1 Score
					NV	BCC	SCC	MEL	AK	BKL	DF	VASC
Attention Transfer	√	√	√	√	0.65	0.59	0.11	0.54	0.17	0.22	0.16	0.17
		√	√	√	0.64	0.58	0.11	0.53	0.18	0.21	0.18	0.10
			√	√	0.65	0.6	0.1	0.56	0.2	0.20	0.17	0.18
				√	0.63	0.58	0.12	0.54	0.16	0.22	0.13	0.23
		√		√	0.66	0.61	0.1	0.54	0.19	0.19	0.1	0.17
		√	√		0.63	0.59	0.11	0.53	0.18	0.20	0.11	0.25
	√	√			0.61	0.58	0.04	0.53	0.17	0.19	0.06	0.17
	√			√	0.65	0.6	0.14	0.55	0.19	0.22	0.16	0.20
	√		√		0.65	0.6	0.11	0.52	0.18	0.18	0.12	0.14
Baseline	√	√	√	√	0.63	0.58	0.13	0.54	0.19	0.24	0.10	0.22

In validation sets, the total accuracy results are listed in Table 6. and the √ means this layer would be utilized in the processing of attention transfer. The accuracy of baseline is 0.6794 marked as blue which is 5 percentage points lower than our method obtaining 0.7253 marked as red. From the results, we can observe that the proposed method plays a positive role in improving accuracy. However, it is worth mentioning that the more layers we migrate, the better results may not be obtained. We consider that this situation would be result from the parameter redundancy. Table 7 comprises the f1-score on 8 categories. The best improving score marked as red indicates that our method could be beneficial for diagnosis of three malignant skin cancers.

In test sets, the best total accuracy of HAM dataset is 0.68 which is higher than the baseline (0.649) and the best total accuracy of BCN dataset is 0.53 which is higher than the result of baseline (0.516). Besides, the F1 score of BCC, SCC, MEL in HAM dataset is 0.49, 0.2, 0.45 respectively. Among of them, the results of BCC and MEL in our method are better than the results of baseline (0.45 and 0.42). Not only that, the F1 score of BCC, SCC, MEL in BCN dataset is 0.6, 0.14, 0.55 which are higher than the results of baseline (0.58, 0.13, 0.54).

4.3 Results analysis

References [27, 28] describe why a linear interpolation of points on the precision-recall curve (PR Curve) provides an overly optimistic measure of classifier performance. Precision-Recall is a useful measure of prediction when the classes are very imbalanced.

In our study, the test results of HAM and BCN were chosen and we calculated the values of precision and recall for three malignant lesions (BCC, SCC, MEL) and benign lesions (NV). 9 combinations were made comparisons with the baseline. Corresponding 9 figures of PR curves (a-i) are presented in Figs. 6, 7. In these figures, the solid lines are the PR curves of our method, and the dotted lines are related to the baseline. For NV (black), the differences between these two lines are not obvious. Our view on this is that the feature redundancy of NV makes the performance not obvious. On the contrary, for BCC (red) and MEL (blue), solid lines are higher than dotted lines in HAM and BCN testing sets which indicates that the similarity features are helpful for their recognitions. However, as shown in Fig. 6, the situation of SCC (green) is opposite where the solid lines were lower than the dotted lines especially for Fig. 6(h) and Fig. 6(i). This could be the number of SCC samples is too small to make the good generalization, therefore, the final performance was lower than baseline.

Fig. 6

The precision-recall curve of testing sets of HAM.

Fig. 7

The precision-recall curve of testing sets of BCN.

5 Discussion

In Fig. 2, there is remarkable visual similarity between benign nevus and malignant melanoma. In this work, we designed the framework and conducted the experiments from the perspective of similarity. In the process of similarity clustering, based on the equation (3), we utilized the deep metric learning to make feature extractions and the inter-class similarity and intra-class dissimilarity were concerned. Such features would be helpful for diagnosis. In previous studies, only category information was used. The advantage of our study is that both the category information and the similarity information were utilized in the training process. However, it is worth mentioning that the more layers we migrate, the better results may not be obtained. We consider that this situation would be result from the parameter redundancy.

The limitations of our study are: 1) the features are not established numerical relationship between samples with different categories; 2) the datasets are unbalanced, and we could collect more malignant samples to balance the categories. They are also the challenges in our future research works.

In conclusion, this study adopted a two-step approach for skin cancer image recognition. Firstly, unsupervised clustering was utilized to make the model learn the similarity information and serve as a teacher model subsequently. Secondly, based on the method of attention transfer, the student classification model was trained. During the training, the model learned the similarity information of the feature layers and the category information at the same time. Eventually, the student classification model improved the recognition rate of the lesions.

Footnotes

Acknowledgments

This work was supported by the Application and Basic Research project of Sichuan Province [grant No.2019YJ0055]; the Enterprise Commissioned Technology Development Project of Sichuan University [grant No.18H0832]; and the Achievement Conversion and Guidance Project of Chengdu Science and Technology Bureau [grant No.2017-CY02-00027-GX]. We also thank Highong Intellimage Medical Technology (Tianjin) Co., Ltd for clinical consulting in this research.

References

A.C. Society, Cancer Facts & Figures 2022. Atlanta: American Cancer Society, 2022.

Zheng

, Zhang

, Zeng

, et al., Cancer incidence and mortality in China, 2016, Journal of the National Cancer Center 2(1) (2022), 1–9.

Esteva

, Kuprel

, Novoa

R.A.

, et al., Dermatologist-level classification of skin cancer with deep neural networks, Nature 542(7639) (2017), 115–118.

Rahman

, Desai

B.C.

, Bhattacharya

, et al., Image retrieval-based decision support system for dermatoscopic images, in 19th IEEE Symposium on Computer-based Medical System, 2006, pp. 285–290.

Schroff

, Kalenichenko

and Philbin

, Facenet: A unified embedding for face recognition and clustering, in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.

Hermans

, Beyer

and Leibe

, In defense of the triplet loss for person re-identification, CoRR abs/1703.07737 (2017).

Yang.

, Zhai.

, Li

, et al., A deep metric learning approach for histopathological image retrieval - sciencedirect, Methods 179 (2020), 14–25.

Zhong

, Li

, Wu

, et al., Deep metric learning-based image retrieval system for chest radiograph and its clinical applications in covid-19, Medical Image Analysis (2021), 101993.

Musgrave

, Belongie

and Lim

S.N.

, A metric learning reality check, arXiv preprint (2020), arXiv:200308505.

10.

Combalia

, Codella

N.C.F.

, Rotemberg

, et al., Bcn0: Dermoscopic lesions in the wild, arXiv preprint (2019), -arXiv:-1908.02288.

11.

Tschandl

, Rosendahl

and Kittler

, The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions, Scientific Data 5(1) (2018), 180161.

12.

Codella

N.C.F.

, Gutman

, Celebi

M.E.

, et al., Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic), arXiv preprint (2017), -arXiv:-1710.05006.

13.

Codella

, Rotemberg

, Tschandl

, et al., Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic), arXiv preprint (2019), -arXiv:-1902.03368.

14.

Daneshjou

, Barata

, Betz-Stablein

, et al., Checklist for evaluation of image-based artificial intelligence reports in dermatology: CLEAR Derm Consensus Guidelines From the International Skin Imaging Collaboration Artificial Intelligence Working Group, JAMA Dermatology 158(1) (2022), 90–96.

15.

Caron

, Bojanowski

, Joulin

and Douze

, Deep clustering for unsupervised learning of visual features, in European Conference on Computer Vision, 2018.

16.

Hadsell

, Chopra

and Lecun

, Dimensionality reduction by learning an invariant mapping, in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), 2006.

17.

Hoffer

and Ailon

, Deep metric learning using triplet network, in International Workshop on Similarity-based Pattern Recognition, 2015, pp. 84–92.

18.

Wang

, Han

, Huang

, et al., Multi-similarity loss with general pair weighting for deep metric learning, arXiv preprint (2019), arXiv:1904.06627.

19.

Zagoruyko

and Komodakis

, Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer, in International Conference on Learning Representations, 2017.

20.

Paszke

, Gross

, Massa

, et al., Pytorch: An imperative style, high-performance deep learning library, in NeurIPS, 2019, pp. 8024–8035.

21.

, Zhang

, Ren

and Sun

, Deep residual learning for image recognition, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.

22.

Tamoor

and Younas

, Automatic segmentation of medical images using a novel Harris Hawk optimization method and an active contour model, Journal of X-Ray Science and Technology 29(4) (2021), 721–739.

23.

Malik

Y.S.

, Tamoor

, Naseer

, et al., Applying an adaptive Otsu-based initialization algorithm to optimize active contour models for skin lesion segmentation, Journal of X-Ray Science and Technology 30(6) (2022), 1169–1184.

24.

Kazaj

P.M.

, Koosheshi

, Shahedi

, et al., U-Net-based models for skin lesion segmentation: More attention and augmentation, arXiv preprint (2022), arXiv:2210.16399.

25.

Taghizadeh

and Mohammadi

, The fast and accurate approach to detection and segmentation of melanoma skin cancer using fine-tuned Yolov3 and SegNet based on deep transfer learning, arXiv preprint (2022), arXiv:2210.05167.

26.

Srivastava

, Jha

, Chanda

, et al., Msrf-net: A multi-scale residual fusion network for biomedical image segmentation, IEEE Journal of Biomedical and Health Informatics 26(5) (2021), 2252–2263.

27.

Davis

, The relationship between precision-recall and ROC curves, Proceedings of the 23rd International Conference on Machine Learning, 2006.

28.

Flach

P.A.

and Kull

, Precision-recall-gain curves: PR analysis done right, Neural Information Processing Systems Curran Associates, Inc, 2015.