Long-distance contextual attention network for skin disease segmentation

Abstract

In recent years, the incidence of skin diseases has increased significantly, and some malignant tumors caused by skin diseases have brought great hidden dangers to people’s health. In order to help experts perform lesion measurement and auxiliary diagnosis, automatic segmentation methods are very needed in clinical practice. Deep learning and contextual information extraction methods have been applied to many image segmentation tasks. However, their performance is limited due to insufficient training of a large number of parameters and these parameters sometimes fail to capture long-term dependencies. In addition, due to the many interfering factors of the skin disease image, the complex boundary and the uncertain size and shape of the lesion, the segmentation of the skin disease image is still a challenging problem. To solve these problems, we propose a long-distance contextual attention network(LCA-Net). By connecting the non-local module and the channel attention (CAM) in parallel to form a non-local operation, the long-term dependence is captured from the two dimensions of space and channel to enhance the network’s ability to extract features of skin diseases. Our method has an average Jaccard index of 0.771 on the ISIC2017 dataset, which represents a 0.6%improvement over the ISIC2017 Challenge Champion model. The average Jaccard index of 5-fold cross-validation on the ISIC2018 dataset is 0.8256. At the same time, we also compared with some advanced methods of image segmentation, the experimental results show our proposed method has a competitive performance.

Keywords

Skin lesion segmentation attentional mechanism artificial intelligence deep learning

1 Introduction

The rise of medical images has played a vital role in medical diagnosis. Clinical medicine uses MRI, CT and other methods to image human lesions, which provides the possibility for early detection of malignant tumors, cancerous sites and other lesions, so that patients can find the lesions and get treatment in the early stage of cancer. Among the many types of cancers, skin cancer is one of the most prevalent cancers in the world. According to the American Cancer Society’s 2019 annual report, the number of new skin cancer patients in the United States in 2019 was 104,350, and 11,650 deaths were caused. Among them, melanoma is the deadliest form of skin cancer, accounting for 92.46%and 62.06%of new cases and deaths of skin cancer, respectively [1]. But in fact, because the pigmentation lesions occur on the surface of the skin, melanoma can be detected early through visual inspection by experts. If skin lesions are found in the early stage of illness and treated in time, skin lesions such as melanoma can have a higher cure rate [2].

In recent years, as the main diagnosis and treatment method for skin lesions in clinical medicine, Dermatoscope has provided clinical medicine with clear and visualized images of skin diseases. Dermatologists can more effectively judge the type of lesions through dermatoscope. When subjective factors have a greater impact on the diagnosis results, manual reading of the film will inevitably lead to misdiagnosis [3]. In order to support and help doctors to make more accurate skin disease diagnosis, automatic segmentation computer aided diagnosis system to achieve skin disease image segmentation has become an urgent need for clinical diagnosis.

Deep learning learns the characteristics of multiple abstract levels of data from a specific large number of sample data, and has demonstrated excellent performance in the field of image processing, especially in the field of computer vision that uses natural images as the processing object to make breakthroughs [4, 5]. These characteristics make the application of deep learning methods and medical image segmentation possible. In the field of skin lesion segmentation, scholars are constantly researching new models for the segmentation of medical images, and the image segmentation of skin diseases has also made substantial progress. At present, the more mature skin lesion segmentation methods are mainly based on the encoder-decoder structure, such as a series of widely used network structures such as U-Net[6] and U-net++ [7]. In addition, the atrous spatial pyramid pooling (ASPP) [8] and attention mechanism have also been introduced to accurately predict the image by obtaining multi-scale feature maps. However, due to the complexity of the boundary of skin lesions and the characteristics of different shapes and sizes, In the previous work, the model did not pay enough attention to the boundary of skin lesions and the segmentation accuracy was not high enough. In order to solve these problems, we introduced the LCA module and proposed a long-distance contextual attention network based on the traditional U-Net. We Uses a non-local module to calculate the similarity of two locations to capture long-distance dependence, and uses CAM[9] to integrate channel features. In this paper, we have made the following contributions:

(1) We preprocessed the skin disease datasets to solve the problem of skin disease image color imbalance.

(2) We proposed LCANet to integrate the long-distance dependencies of images and performed a more accurate segmentation of the boundary of the skin lesion.

(3) Experiments were conducted on ISIC2017 and ISIC2018 datasets, and the experimental results were compared with the existing skin disease segmentation methods.

(4) We reproduced the current advanced network model, and conducted experiments on ISIC2017, and compared and analyzed the experimental results.

2 Related work

Attention mechanism: Visual attention is a brain signal processing mechanism unique to human vision. Humans do not pay attention to all the information of a scene when they see a scene, but quickly observe and scan the global image through vision. Image information is transmitted to the cerebral cortex, and the brain quickly captures the target area or target entity that needs to be focused, and then puts more attention on the focus of attention, ignoring useless interference information. The attention mechanism allows humans to quickly screen out high-value information targets in the complex information flow, and improve the efficiency and accuracy of information processing. In recent years, the attention mechanism has been tried to be applied to various computer vision tasks to improve the performance of the model. Jie et al. [10] proposed SENet to model the interdependence between channels of convolution features by introducing an attention module to improve the performance of the network. Li et al. [11] proposed SKNet, they proposed a nonlinear method called selective kernel convolution to aggregate multiple convolution kernel information. It contains three operations of split, fuse and select to achieve different sizes of neuron adaptive receptive fields. Woo et al. [9] proposed the Convolutional Attention Module (CBAM), which performs attention mapping along the channel dimension and spatial dimension, and then performs adaptive feature refinement on the attention mapping, which can be applied to the CNN architecture to improve the performance of the model. Wang et al. [12] proposed Non-local Neural Networks, which introduces a component to capture the long-term dependence of deep neural networks, calculates the response of a certain location as the weighted sum of all location features, and calculates the relationship between two pixels. This component can be combined with any existing architecture, and it shows good performance in the fields of video classification, target detection and segmentation, and pose estimation tasks.

U-shaped networks: U-Net immediately received widespread attention after it was proposed at MICCAI in 2015 [6], and was widely used in medical image segmentation. It uses four up-sampling operations, four down-sampling operations and skip connections. Down-sampling and skip connection make the feature maps of the original image processed by the network merge with more scale features, and the features of different scales are merged to perform multi-scale prediction. The four up-sampling operations make the edge information of the segmentation result more refined. According to the characteristics of the U-shaped network, there have been continuous improved networks based on U-Net used in medical image segmentation research in recent years. Zhou et al. proposed Nested U-Net (U-Net++) [7], which is composed of U-Nets of different depths, and redesigned the jumper between the encoder and the decoder. This structure improves the fixed-depth U-Net and reduces the semantic gap between the feature maps of the encoder and decoder sub-networks. Alom et al. [13] proposed the R2U-Net network structure, which applies the cyclic residual network to the work of medical image segmentation, so that the network can achieve better performance under the same number of parameters. Jha et al. [14] proposed DoubleU-Net, which superimposes two U-Nets to form a double network structure, and uses Atrous Spatial Pyramid Pooling (ASPP) to capture contextual information. The network achieved good segmentation performance on several different medical images. However, in the process of replicating other models for experiments, we found that the traditional deep learning network lacked the ability to distinguish interference factors on skin disease images. They often pay more attention to the extraction of semantic features in images, but ignore the complex boundary of skin disease images.

Skin disease image segmentation: In the current work on skin image segmentation, skin image is mainly an open data set provided by the International Skin Imaging Collaboration (ISIC) for the International Symposiumon Biomedical Imaging (ISBI) Challenge. The Challenge has been held for three years –ISIC2016, 2017 and 2018. Yuan et al. [15] proposed an automatic segmentation model based on ISIC2016 dataset, they used 19-layer deep convolutional neural networks for end-to-end training and designed a new loss function based on the Jaccard distance, which obtained 0.847 Jaccard value on the ISIC2016 dataset. Later, Yuan et al. [4] extended their previous work on ISICs by proposing a deeper network architecture with smaller cores to improve network discrimination. The method achieved a Jaccard value of 0.765 on the ISIC2017 dataset, making it the winner of the 2017 ISBI Challenge. In order to alleviate the need for large-scale labeled datasets in medical image segmentation, Fayjie et al. [16] integrated situational training into the multi-shot semantic segmentation of medical images and proposed a new multi-shot learning framework for semantic segmentation, where unlabeled images are also made available at each episode. This approach is independent of the general model and can be combined with different deep learning frameworks. Ren et al. [17] proposed a serial network by combining different attention modules in different forms, which improved the accuracy of skin disease image segmentation. This work obtained a Jaccard index of 0.7692 on the ISIC2017 dataset. Tomar et al. [18] proposed a feedback neural network that unifies the previous epoch mask with the feature map of the current training epoch, and then used previous epoch mask to provide a hard attention to the learnt feature maps at different convolutional layers, demonstrated the effectiveness of their approach in ISIC2018 and other different medical segmentation datasets. Sabari et al. [19] randomly split the training images of ISIC2018 dataset into 60-20-20 ratio, they proposed an architecture which leverage the benefits of coordinate convolutional layer and deep residual block in an encoding-decoding framework for skin lesion segmentation. Huang et al. [20] proposed a new method for analyzing and visualizing patterns in MRI images, which adopted the fuzzy information gain (FIG) function and K-means clustering algorithm. Compared with other image segmentation methods, various indexes were improved. In addition, some researchers have explored the feasibility of image segmentation and edge prediction in the task of image classification. Simon et al. [21] divided skin tissue into 12 meaningful skin categories (including hair follicles, sweat glands, and explicit skin stratification), providing highly interpretable outputs that provide technical support for routine pathological work and lesion edge recognition. Laith et al. [22] proposed a new deep convolutional neural network model. They trained the model on a large number of unlabeled medical data sets in combination with the method of transfer learning, and then transferred the learned features on a small number of labeled medical images. Their research proves that large amounts of unlabeled image data are available and transfer learning methods can be used to improve the image processing effect of small data sets.

3 Network

In this section, we will introduce our proposed skin disease image segmentation model. Our backbone network uses an encoder-decoder structure, adds X-blocks [23] with depth deconvolution to the encoder and decoder to extract high-dimensional features, and the LCA module is proposed to extract image features from space and channel dimensions. Figure 1 shows the specific process of our segmentation method.

Fig. 1

Overview of the long-distance contextual attention network.

When using convolutional networks for segmentation tasks, we can regard each channel map as a feature detector. Emphasizing the dependency between channels can help us improve the feature representation of specific semantics. On the other hand, the boundary of the lesion area of the skin disease image is very complicated, and the convolution operation will generate a local receptive field, and the key information will be unclear when processing the complex boundary segmentation task, and there will be a lack of judgment on the "where" feature that needs to be emphasized and the "where" need to be suppressed. Therefore, it is necessary to extract a wide range of location context information and encode it into the feature map. However, current context information extraction methods have limited performance due to insufficient training of a large number of parameters for the overly complex boundaries of medical image segmentation, and cannot capture long-distance dependencies. Based on the above problems, we designed an LCA module, which connects a channel attention module and a non-local module in parallel to integrate and deeply mine the channel information and long-distance context information of the image, and fully train the parameters of the context information. In addition, due to the superiority of U-NET in extracting medical image features, we choose U-NET as our backbone network. The structure of the LCA module is as follows:

As shown in Fig. 2, given a local feature map, We first feed it into a non-local operation and a channel attention module to extract its spatial features and channel features respectively, and Then we combine the results from the two modules and input them into the decoder.

Fig. 2

Long-distance contextual attention module.

3.1 Channel attention module

In order to integrate information along the channel dimension to extract contextual features, we use the channel attention module to generate channel attention feature maps. We first perform a maximum pooling operation and an average pooling operation on the feature map F to aggregate channel information and generate two new mappings F_avg ∈ R^1×H×W and F_max ∈ R^1×H×W to represent the average feature and the maximum feature. Then we input these two new mappings into the multilayer perceptron to generate the channel attention map M: $\begin{matrix} M_{s} (F) & = σ (f^{7 \times 7} ([AvgPool (F); MaxPool (F))])) \\ = σ (f^{7 \times 7} ([F_{avg}^{s} \cdot F_{\max}^{s}])) \end{matrix}$ (1)

3.2 Long-distance contextual attention module

According to the characteristics of the skin disease image, we first feed the feature map F ∈ R^H×W×C into a 3 × 3 convolutional layer to form a new feature map F₀ ∈ R^H×W×C₀ to filter out some unimportant features, where $C_{0} = \frac{1}{8} C$ . In order to capture long-distance dependencies, we add non-local operations to the model. The definition of non-local operations is as follows: $y_{i} = \frac{1}{C (X)} \sum_{\forall j} f (X_{i}, X_{j}) g (X_{j})$ (2)

Where i is the index of an output position (in space, time, or spacetime) whose response is to be computed and j is the index that enumerates all possible positions. X represents the input feature map. g (X_j) computes a representation of the input signal at the position j. C (X) represents the normalization process, and the function f (X_i, X_j) represents the similarity relationship mapping between X_i and X_j. We use the embedded Gaussian formula to express the similarity relationship:

$f (X_{i}, X_{j}) = e^{{θ (X_{i})}^{T} φ (X_{j})}$ (3)

At this time f (X_i, X_j) g (X_j) can be regarded as the softmax calculation along the j dimension, we can get: $y = softmax (X^{T} W_{θ}^{T} W_{φ} X) g (X)$ (4)

$f (X_{i}, X_{j}) = \frac{\exp (θ (X_{i})^{T} φ (X_{j}))}{\sum_{j = 1}^{N} \exp (θ (X_{i})^{T}) φ (X_{j})}$ (5)

θ (X) and φ (X) are embedding layers realized by two 1 × 1 convolutions, and N is the number of all x in the feature map. Finally, we get the relationship between all positions in the original feature map and the captured dense context information f (X_i, X_j). At the same time, we input F₀ into a 1 × 1 convolutional layer to generate a new feature map Y ∈ R^H×W×C to represent the input signal and reshape it into R^N×C. Then we multiply f (X_i, X_j) with Y, and perform the element-wise sum operation with F₀: $Z_{i} = \sum_{j = 1}^{N} f (X_{i}, X_{j}) Y_{j} + X_{i}$ (6)

It can be inferred that the feature map Z obtained is the sum of the relationship feature obtained by the non-local module and the original feature, which effectively aggregates long-distance features. Finally, we input Z into the convolutional layer, generate a feature map with the same shape as F, and perform a summation operation with the output of the channel attention module, and integrate channel information for adaptive feature refinement and avoid overfitting.

From previous work we can see that The down-sampling factor of 16 is known to preserve most of the information required to parse the original image correctly at the pixel level [17]. Although the 16×16 thumbnail is small, it is enough to distinguish the target entity in the image and extract the features of the feature map. An increase in resolution will incur additional computing costs, when the resolution of the feature map increases by 2 times in each dimension, the memory consumption of the feature map increases by 4 times. Considering the computational cost of the experiment, we chose to use the LCA module in the last layer of the encoder.

4 Experiments

In order to evaluate the performance of our proposed method in skin lesion segmentation, we trained and validated our method on the open-source datasets ISIC2017 [24] and ISIC2018 [25], and compared the results with existing advanced network model methods. The experimental results show that our method has achieved good results on the datasets. In this part, we will introduce the following work in detail: (1) Introduce our datasets and experimental evaluation metrics; (2) Introduce the details of the experimental implementation; (3) Introduce experimental preprocessing operations; (4) We conducted ablation experiments on ISIC2017 to evaluate the influence of the attention module on the results; (5) We conducted experiments on the existing model based on ISIC2017 and ISIC2018, and compared our experimental results with the obtained data, so as to judge the effect of the model more comprehensively.

4.1 Datasets and evaluation metrics

4.1.1 Datasets

We conducted experiments on ISIC 2017 and ISIC2018. ISIC2017 contains a total of 2750 dermoscopy images composed of moles, seborrheic keratosis and malignant melanoma. Among them, 2000 sheets are the training set, 150 sheets are the validation set, and 600 sheets are the test set. The ISIC2018 images available include 2594 training set images ranging in size from 540x576 pixels to 4499x6748 pixels. In order to more rigorously verify the performance of the model and the robustness of our model, we performed 5-fold cross-validation on ISIC2018. These skin disease images have been marked by a professional dermatologist on the skin lesion area according to pixels. In the datasets, the size of the original image is not uniform. In order to obtain better segmentation performance, we use bi-linear interpolation to adjust the image to a uniform size before training.

4.1.2 Evaluation metrics

In order to measure the segmentation performance of the model, we use the Dice coefficient (Dice) and Jaccard index (JA) as the main evaluation indicators, which are the same as those of the ISBI Challenge. In addition, we also calculated accuracy (Acc), precision (Prec), specificity (SP), sensitivity (SE) and negative predictive value (NPV), which are commonly used metrics in medical segmentation. Their formulas are as follows: $Dice = 2 \cdot TP / (2 \cdot TP + FN + FP)$ (7) $JA = TP / (TP + FN + TN)$ (8) $Acc = (TP + TN) / (TP + TN + FN + FP)$ (9) $Prec = TP / (TP + FP)$ (10) $SP = TN / (TN + FP)$ (11) $SE = TN / (TP + FN)$ (12)

4.2 Implementation

Our experiment is implemented based on the Keras deep learning framework method, and uses the Adam optimizer to optimize our model. All experiments are performed on NVIDIA Tesla V100 GPU, which has 16g of memory. We set the initial learning rate to 1e-4 and adjust the learning rate according to the loss of the validation set during training. When the performance index is stable on the validation set, we adjust the learning rate to 0.1 times the initial learning rate. If the loss does not decrease after 15 epochs, we will stop the learning of the model. Before inputting the training image into the network, we perform data enhancement operations on the image by randomly rotating and flipping the image. In addition, we will use the sum of the dice loss and the binary cross-entropy loss as the training loss function, and set the batch size to 4. Finally, we save the model with the least loss on the validation set.

4.3 Pre-processing of the experiment

Due to the color constancy of the visual system, the human eye obtains a constant object color under different lighting and imaging environments when capturing an image. However, the imaging device does not have this feature, which results in different lighting environments that will cause the color of the collected image to deviate from the true color, as shown in Figure 3.

Fig. 3

Images imaged under different conditions in the ISIC2017 dataset and the results after preprocessing.

In order to eliminate the impact of different lighting and shooting environments on the skin disease images, we use the Grayscale World Algorithm (GWA) [26] to grayscale the image. GWA is based on the gray world hypothesis, which assumes that an image with a large number of color changes, its three RGB color components approach the same gray value Gray. Based on this assumption, we take the average values of the three channels of R, G and B as $\bar{R}$ , $\bar{G}$ and $\bar{B}$ respectively, and calculate their average value as the gray value $\bar{Gray}$ : $\bar{Gray} = \frac{\bar{R} + \bar{G} + \bar{B}}{3}$ (13)

Then calculate the gain coefficient of the three channels: ${\begin{matrix} {gain}_{R} = \frac{\bar{Gray}}{\bar{R}} \\ {gain}_{G} = \frac{\bar{Gray}}{\bar{G}} \\ {gain}_{B} = \frac{\bar{Gray}}{\bar{B}} \end{matrix}$ (14)

Finally, adjust the color components of the original image, and adjust the R, G, and B values of each pixel C to R′, G′, B′ respectively: ${\begin{matrix} R^{'} = {gain}_{R} * R \\ G^{'} = {gain}_{G} * G \\ B^{'} = {gain}_{B} * B \end{matrix}$ (15)

We performed grayscale world operation on the skin disease images on the datasets. and then input the processed images into our model. Meanwhile, we also use pre-processed images when conducting comparative experiments with other advanced models.

4.4 Ablation study for attention modules

In order to verify the performance of the module, we conducted different ablation experiments on the ISIC2017 dataset. It can be concluded from Table 1 that using the attention module can improve the segmentation performance of the model. Adding CAM to the model increases the Jaccard index of the model by 0.28%on the basis of the baseline, and adding the non-local attention module increases Jaccard index by 0.82%. We connected the CAM and non-local attention modules in parallel, and added them to the model as the LCA module as a whole, the Jacquard index obtained increased by 1.04%on the basis of the baseline. The results show that the application of the attention mechanism in the skin disease segmentation network can effectively enhance the feature representation of medical images.

Table 1
Ablation Study for Attention Modules (Implemented on ISIC2017)

name ja di acc se sp

Baseline 0.7608 0.8442 0.9309 0.8352 0.9711

+CAM 0.7636 0.8460 0.9352 0.7112 0.9773

+non-local 0.7690 0.8508 0.9366 0.8204 0.9758

+LCA(ours) 0.7712 0.8533 0.9370 0.8222 0.9791

name	ja	di	acc	se	sp
Baseline	0.7608	0.8442	0.9309	0.8352	0.9711
+CAM	0.7636	0.8460	0.9352	0.7112	0.9773
+non-local	0.7690	0.8508	0.9366	0.8204	0.9758
+LCA(ours)	0.7712	0.8533	0.9370	0.8222	0.9791

Figure 4 is a visual heat map of skin lesions constructed based on the module ablation experiment, showing us that the attention module can be more complete and more effective in capturing contextual features. When the boundary is blurred and there are many interference factors, our model can make a more accurate judgment on the boundary of the lesion through non-local long-distance learning context information.

Fig. 4

Visualization of ablation experiment prediction probability map.

4.5 Comparison with other advanced models

In order to evaluate our model more comprehensively, in this section, we will first compare our method with the results of other segmentation methods based on the ISIC2017. In order to further verify the generalization of our model in skin disease image segmentation, we also conducted experiments on ISIC2018 dataset and compared the experimental results with previous work. Then we reproduced some advanced networks (UNET, DANET, etc.), and fed the preprocessed images of ISIC2017 dataset into these networks for experiment, Finally, we compare compared the experimental results of the contrast experiment with the segmentation results of our model to evaluate the model segmentation performance.

4.5.1 Comparison with previous works

We compared the existing related research results based on the ISIC2017 dataset. As shown in Table 2, it can be seen that compared with the existing models, our method has achieved better results in Jaccard index, accuracy and Dice coefficient.

Table 2
Comparison with other Advanced Models on ISIC2017 (These results come directly from their paper)

Method ja di acc se sp

II-FCN (Wen 2017) [27] 0.6990 0.7940 0.9290 0.8410 0.9840

Thao and Quang (2017) [28] 0.5065 0.6317 0.8772 0.6513 0.9421

FCN-8s (Li and Shen 2018) [29] 0.6960 0.7830 0.9330 0.8060 0.9540

U-Net (Kaul et al. 2019) [30] 0.6510 0.7680 0.9200 0.8530 0.9570

Garcia-Arroyo and Garcia-Zapirain (2019) [31] 0.6650 0.7600 0.8840 0.8690 0.9230

FocusNet (Kaul et al. 2019) [30] 0.7562 0.8315 0.9214 0.7673 0.9896

MtSinai (Yuan et al. 2017; Yuan and Lo 2019) [15] 0.7650 0.8490 0.9340 0.8250 0.9750

LCA(ours) 0.7712 0.8533 0.9370 0.8222 0.9791

Method	ja	di	acc	se	sp
II-FCN (Wen 2017) [27]	0.6990	0.7940	0.9290	0.8410	0.9840
Thao and Quang (2017) [28]	0.5065	0.6317	0.8772	0.6513	0.9421
FCN-8s (Li and Shen 2018) [29]	0.6960	0.7830	0.9330	0.8060	0.9540
U-Net (Kaul et al. 2019) [30]	0.6510	0.7680	0.9200	0.8530	0.9570
Garcia-Arroyo and Garcia-Zapirain (2019) [31]	0.6650	0.7600	0.8840	0.8690	0.9230
FocusNet (Kaul et al. 2019) [30]	0.7562	0.8315	0.9214	0.7673	0.9896
MtSinai (Yuan et al. 2017; Yuan and Lo 2019) [15]	0.7650	0.8490	0.9340	0.8250	0.9750
LCA(ours)	0.7712	0.8533	0.9370	0.8222	0.9791

The available part of the ISIC2018 dataset includes 2594 training sets. We performed a 5-fold cross-validation on the ISIC2018 dataset to measure the performance of the model. We divided the skin disease images into five sections, one of which was used as a test set and the other four groups as a training set. After completing this set of experiments, we take another set of experiments and repeat the process. Table 3 shows the results of the 5-fold cross-validation and the mean value. At the same time, in order to make a horizontal judgment on the performance of our model, we compare our results with the existing segmentation work based on ISIC2018 in Table 4.

Table 3

Results of 5-fold cross-validation based on ISIC2018

Kdold-N	ja	di	acc	se	sp
Kfold-1	0.8228	0.8901	0.9585	0.8883	0.9774
Kfold-2	0.8208	0.8914	0.9610	0.9050	0.9772
Kfold-3	0.8100	0.8803	0.9610	0.8992	0.9755
Kfold-4	0.8383	0.9036	0.9630	0.9188	0.9736
Kfold-5	0.8361	0.9024	0.9613	0.9098	0.9725
Mean value	0.8256	0.89356	0.96096	0.90422	0.97524

Table 4

Comparison with other Advanced Models on ISIC2018(These results come directly from their paper)

Kdold-N	ja	di	acc	se	sp
Shreshth and Divij(2019) [32]	0.809	0.915	0.959	0.968	0.973
Anindo et al.(2020) [33]	0.819	0.891	-	0.943	0.932
DRU-NET(Mina et al. 2020) [34]	0.755	0.861	0.919	-	-
ours(mean value)	0.8256	0.89356	0.96096	0.90422	0.97524

4.5.2 Comparison with other advanced networks

In order to evaluate our approach more comprehensively, we have implemented some existing advanced models based on keras and conducted experiments on the ISIC2017 dataset. These models include CENet [35], DANet [36], Deeplab-v3+ [37], DoubleU-Net [14], HRNet [8], NestedU-Net [7], PSPNet [38], R2UNet [13], ResUNet [39], SCSEUNet [40], SegNet [41], and SEUNet [40]. We use the same training strategies as in our experiments to train these models. We also use the Adam optimizer, set the initial learning rate to le-4, and use the same hyperparameters as our model. In addition, we mainly design the loss function based on the original settings of these models. If the specific loss function type is not mentioned in the original paper, we use the cross-entropy loss function by default. The results of the experiment are shown in Table 3.

Table 5
Comparison with other Advanced Models on ISIC2017(These results come from our reimplementation)

name ja di acc se sp

CENet 0.7585 0.8441 0.9302 0.8444 0.9742

DANet 0.7696 0.8530 0.9359 0.8593 0.9662

Deeplab-v3+ 0.7195 0.8104 0.9247 0.7743 0.9736

DoubleUnet 0.7546 0.8375 0.9310 0.8001 0.9834

HRUnet 0.7502 0.8352 0.9285 0.8269 0.9695

NestedUnet 0.7497 0.8360 0.9274 0.8092 0.9819

PSPNet 0.7320 0.8204 0.9267 0.7990 0.9800

R2Unet 0.7579 0.8445 0.9293 0.8363 0.9717

ResUnet 0.7385 0.8265 0.9239 0.8185 0.9729

SCSEUnet 0.7558 0.8384 0.9320 0.8252 0.9778

SegNet 0.7095 0.7946 0.9187 0.7750 0.9779

SEUnet 0.7350 0.8210 0.9274 0.7890 0.9782

LCANet(ours) 0.7712 0.8533 0.9370 0.8222 0.9791

name	ja	di	acc	se	sp
CENet	0.7585	0.8441	0.9302	0.8444	0.9742
DANet	0.7696	0.8530	0.9359	0.8593	0.9662
Deeplab-v3+	0.7195	0.8104	0.9247	0.7743	0.9736
DoubleUnet	0.7546	0.8375	0.9310	0.8001	0.9834
HRUnet	0.7502	0.8352	0.9285	0.8269	0.9695
NestedUnet	0.7497	0.8360	0.9274	0.8092	0.9819
PSPNet	0.7320	0.8204	0.9267	0.7990	0.9800
R2Unet	0.7579	0.8445	0.9293	0.8363	0.9717
ResUnet	0.7385	0.8265	0.9239	0.8185	0.9729
SCSEUnet	0.7558	0.8384	0.9320	0.8252	0.9778
SegNet	0.7095	0.7946	0.9187	0.7750	0.9779
SEUnet	0.7350	0.8210	0.9274	0.7890	0.9782
LCANet(ours)	0.7712	0.8533	0.9370	0.8222	0.9791

It can be seen from Table 4 that networks without attention mechanisms such as deeplabv3+ and PSPNet, SegNet perform poorly on the ISIC2017 dataset, while the models with attention mechanism such as CENet, DANet and LCANet have achieved good segmentation results. Compared with SegNet, our model improves Jaccard index, Dice coefficient, and accuracy by 6.51%, 5.87%, and 1.83%, respectively. This is because the attention mechanism can better capture contextual feature information. Compared with other types of images, the lesion area of the skin disease image has a relatively fixed structure, does not have too much semantic information and the feature information is more concentrated. Meanwhile, the boundaries of skin disease images are complex, and the network needs to combine contextual information to highlight features in key areas. The attention mechanism can help the network to discard complex interference information and learn the main features. Therefore, the application of attention can enable the network to obtain better results in the segmentation of skin diseases. In addition, we help the network to obtain non-local long-distance context information by introducing a non-local attention module, and add deep separable convolution to the backbone network. Compared with the experimental results of danet, our model has improved by 0.16%, 0.03%, and 0.11%on the three indicators of Jaccard index, Dice coefficient, and accuracy, respectively.

Figure 5 shows the segmentation results of our model and some advanced models on ISIC2017, these results clearly show that models such as SegNet, Deeplab-v3+ and HRNet, as more advanced models in the field of semantic segmentation, are more inclined to obtain a higher global average to solve multi-class segmentation problems. Since these networks pay attention to the global characteristics of the image, it will appear that the interference factors in the skin disease image, such as stains, are recognized as the focus area. As a result, the segmentation results of these networks on the skin disease image are relatively rough.

Fig. 5

LCANet and some advanced model segmentation results

Many models applied to medical images are improved on the basis of Unet, such as DANet, CENet and R2Unet, DoubleUnet, etc. Medical images have fuzzy boundaries and complex gradients, requiring more high-resolution information for accurate segmentation. The U-shaped network combines low-resolution information (providing a basis for object category recognition) and high-resolution information (providing a basis for precise segmentation and positioning), and is more suitable for medical image segmentation than other backbone networks. Our model is also based on a U-shaped network, which can be better used for accurate segmentation of the boundaries of skin disease images.

Although CENet and R2Unet are also improved based on Unet, they do not introduce an attention mechanism, which causes the model to be distracted by secondary feature information. The results in segmentation tasks with strong interference information are not ideal. Our method combines the channel attention module and the non-local attention module to help the network focus on key features, integrate channel and spatial feature information, and combine long-distance context information to make the model achieve better segmentation results. Through the comparison with the above advanced models, it can be found that compared with other advanced models, our model shows better performance in dealing with skin disease images with complex boundary and many interference factors.

5 Conclusion

In this paper, we propose a long-distance contextual attention network (LCANet) for skin disease image segmentation. The network can capture spatial long-distance dependence through the attention module and integrate it with channel feature information. We conducted a series of experiments based on the ISIC2017 and ISIC2018 datasets and tested the performance of our model on these two datasets. We compared the experimental results with the existing skin disease image segmentation methods. we have implemented some existing advanced models based on keras and conducted experiments on datasets. The experimental results show that our model has better performance on skin disease image segmentation tasks compared with other existing models. It can effectively eliminate irrelevant information, grasp the main features of the segmentation target, and more accurately identify the boundary of the lesion in the skin disease image.

There are still deficiencies waiting to be solved in our research. How to reduce the complexity of the model while improving the segmentation accuracy of the model is the main direction we will overcome in future research. At the same time, we will also apply the proposed model to more medical segmentation datasets.

Acknowledgments

This research is partially supported by Xinjiang Autonomous Region key research and development project: 2021B03001-4, Xinjiang Uygur Autonomous Region (CN) Postgraduate Research and Innovation Project (XJ2020G072), Science and Technology Department of Xinjiang Uyghur Autonomous Region Fund Project (2020E0234). We would also like to thank our tutor for the careful guidance and all the participants for their insightful comments.

Disclosures

The authors declare no conflicts of interest.

References

Desantis

C.E.

, et al., Breast cancer statistics, CA: ACancer Journal for Clinicians 69(6) (2019).

Berseth

, ISIC 2017 - Skin Lesion Analysis Towards Melanoma Detection, (2017).

pmhdev and Pub Med Health, Dermoscopy compared with naked eye examination for the diagnosis of primary melanoma: a meta-analysis of studies performed in a clinical setting –National Library of Medicine –Pub Med Health, Centre for Reviews & Dissemination (2008).

Yuan

and Lo

Y.C.

, Improving Dermoscopic Image Segmentation withEnhanced Convolutional-Deconvolutional Networks, IEEE Journalof Biomedical and Health Informatics 99(1) (2017).

, Yu

, Chen

, Fu

C.W.

and Heng

P.A.

, Semisupervised Skin Lesion Segmentation via Transformation Consistent Self-ensembling Model,” (2018).

“U-Net: Convolutional Networks for Biomedical Image Segmentation,” (2015).

Zhou

, Siddiquee

, Tajbakhsh

and Liang

, UNet++: A NestedU-Net Architecture for Medical Image Segmentation, (2018).

Sun

, Xiao

, Liu

and Wang

, Deep High-Resolution Representation Learning for Human Pose Estimation, (2019).

Woo

, Park

, Lee

J.Y.

and Kweon

I.S.

, CBAM: Convolutional Block Attention Module, Springer, Cham, (2018).

10.

Jie

, Li

, Gang

and Albanie

, Squeeze and-Excitation Networks, IEEE Transactions on Pattern Analysis and Machine Intelligence (99) (2017).

11.

, Wang

, Hu

and Yang

, Selective Kernel Networks, IEEE (2019).

12.

Wang

, Girshick

, Gupta

and He

, Non-local Neural Networks, (2017).

13.

Alom

M.Z.

, Hasan

, Yakopcic

, Taha

T.M.

and AsariRecurrent

V.K.

, Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation, (2018).

14.

Jha

, Riegler

M.A.

, Johansen

, Halvorsen

and JohansenDoubleU-Net:

H.D.

, DoubleU-Net:ADeep Convolutional Neural Network for Medical Image Segmentation, (2020).

15.

Yuan

, Chao

and Lo

Y.C.

, Automatic Skin Lesion SegmentationUsing Deep Fully Convolutional Networks With Jaccard Distance, IEEE Trans Med Imaging (9) (2017), 1.

16.

Feyjie

A.R.

, Azad

, Pedersoli

, Kauffman

and Dolz

, Semi-supervised few-shot learning for medical image segmentation, (2020).

17.

Ren

, Yu

, Tian

, Cheng

, Guo

and Zhang

, Serial attention network for skin lesion segmentation, Journal of Ambient Intelligence and Humanized Computing 1–12.

18.

Tomar

N.K.

, et al., FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation, arXiv preprint arXiv:2103.17235 (2021).

19.

Tomar

N.K.

, et al., FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation, arXiv preprint arXiv:2103.17235, (2021).

20.

Huang

Y.P.

, Singh

and Kuo

H.C.

, A Hybrid Fuzzy ClusteringApproach for the Recognition and Visualization of MRI Images ofParkinson’s Disease, IEEE Access 8(1) (2020).

21.

Thomas

S.M.

, Lefevre

J.G.

, Baxter

and HamiltonInterpretable

N.A.

, deep learning systems for multi-class segmentation andclassification of non-melanoma skin cancer, Medical ImageAnalysis 68 (2021), 101915.

22.

Alzubaidi

, Al-Amidie

, Al-Asadi

, Humaidi

A.J.

and YeNovel

, Transfer Learning Approach for Medical Imaging with LimitedLabeled Data, Cancers 13(7) (2021).

23.

, et al., X-Net: Brain Stroke Lesion Segmentation Based on Depthwise Separable Convolution and Longrange Dependencies, (2019).

24.

Codella

, et al., Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC), (2017).

25.

Tschandl

, Rosendahl

and Kittler

, The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions, Scientific Data (2018).

26.

Ebner

, A parallel algorithm for color constancy, Journal ofParallel & Distributed Computing 64(1) (2004), 79–88.

27.

Wen

, II-FCN for skin lesion analysis towards melanoma detection, (2017).

28.

T.T.

and Quang

N.H.

, Automatic skin lesion analysis towards melanoma detection, (2017).

29.

and Shen

, Skin Lesion Analysis towards Melanoma DetectionUsing Deep Learning Network, Sensors 18(2) (2018), 322.

30.

Kaul

, Manandhar

and Pears

, FocusNet: An attentionbased Fully Convolutional Network for Medical Image Segmentation, IEEE (2019).

31.

Jose

, Garcia-Arroyo

and Garcia-Zapirain , Segmentation of skin lesions in dermoscopy images using fuzzy classification of pixels and histogram thresholding, Computer Methods & Programs in Biomedicine, (2018).

32.

Saini

, Gupta

and Tiwari

A.K.

, Detector-SegMentor Network for Skin Lesion Localization and Segmentation, in National Conference on Computer Vision, Pattern Recognition, Image Processing, and Graphics, (2019), 589–599.

33.

Saha

, Prasad

and Thabit

, Leveraging adaptive color augmentation in convolutional neural networks for deep skin lesion segmentation, in 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI) (2020), 2014–2017.

34.

Jafari

, Auer

, Francis

, Garibaldi

and Chen

, DRU-net: An Efficient Deep Convolutional Neural Network for Medical Image Segmentation, IEEE (2020).

35.

, et al., CE-Net: Context Encoder Network for 2D Medical Image Segmentation, IEEE Transactions on Medical Imaging (2019), 1.

36.

, et al., Dual Attention Network for Scene Segmentation, (2020).

37.

Chen

L.C.

, Zhu

, Papandreou

, Schroff

and Adam

, Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, Springer, Cham, (2018).

38.

Zhao

, Shi

, Qi

, Wang

and Jia

, Pyramid Scene Parsing Network, IEEE Computer Society (2016).

39.

Alom

M.Z.

, Yakopcic

, Hasan

, Taha

T.M.

and AsariRecurrent

V.K.

, residual U-Net for medical image segmentation, Journal of Medical Imaging 6(1) (2019).

40.

Roy

A.G.

, NavAb

and Wachinger

, Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks, Springer, Cham, (2018).

41.

Badrinarayanan

, Kendall

and Cipolla

, SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, IEEE Transactions on Pattern Analysis & Machine Intelligence (2017), 1.