ACEANet: Ambiguous Context Enhanced Attention Network for skin lesion segmentation

Abstract

Skin lesion segmentation from dermatoscopic images is essential for the diagnosis of skin cancer. However, it is still a challenging task due to the ambiguity of the skin lesions, the irregular shape of the lesions and the presence of various interfering factors. In this paper, we propose a novel Ambiguous Context Enhanced Attention Network (ACEANet) based on the classical encoder-decoder architecture, which is able to accurately and reliably segment a variety of lesions with efficiency. Specifically, a novel Ambiguous Context Enhanced Attention module is embedded in the skip connection to augment the ambiguous boundary information. A Dilated Gated Fusion block is employed in the end of the encoding phase, which effectively reduces the loss of spatial location information due to continuous downsampling. In addition, we propose a novel Cascading Global Context Attention to fuse feature information generated by the encoder with features generated by the decoder of the corresponding layer. In order to verify the effectiveness and advantages of the proposed network, we have performed comparative experiments on ISIC2018 dataset and PH2 dataset. Experiments results demonstrate that the proposed model has superior segmentation performance for skin lesions.

Keywords

Skin lesion segmentation medical image processing feature extraction encoder-decoder architecture

1. Introduction

Skin cancer is one of the most common cancers worldwide and the rate of growth is increasing. Melanoma has a 92% incidence rate and a 60% mortality rate among skin cancers, according to a report by the American Cancer Society [1]. Although skin cancer is highly lethal, death can be avoided both if the tumour is detected and removed in early diagnosis. Dermoscopy is widely used as a non-invasive diagnostic tool for the analysis of skin diseases [2]. The segmentation of lesions by dermoscopic images can obtain the exact area of the lesion, thus greatly facilitating the diagnosis of skin diseases and is clinically important. However, diagnosis through dermoscopic images is a tedious task which requires a lot of time. Therefore, automated computer-aided diagnostic (CAD) systems have been introduced to improve the accuracy and efficiency of diagnosis, helping doctors immensely. In these CAD systems, one of the most critical factors is the segmentation of the lesion from the dermoscopic image. Therefore, attaining accurate skin lesion segmentation results is crucial for dermoscopic image analysis and dermatological diagnosis. However, there is a lot of noise around the lesion in the dermoscopic image, such as hair, blood vessels and air bubbles, as well as a huge variation in lesions from one dermoscopic image to another [3]. Hence the accurate segmentation of skin lesions is still an adjustment task. Traditional methods of skin lesion segmentation rely excessively on fine manual features and clever image pre-processing, resulting in poor performance under application [4, 5, 6, 7].

In recent years, Deep Convolutional Neural Network (DCNN) models have turned skin lesion segmentation into a pixel-level classification problem with remarkable success [8, 9, 10, 11, 12], such as the well-known Full Convolutional Neural Network FCN [13] and its variant network U-Net [14], where the FCN converts a classification convolutional neural network into a segmentation network using fully convolutional layers instead of fully connected layers, allowing for an end-to-end training process by eliminating the need for sliding window slices and allowing for the input of whole images. The U-Net model is unique compared to other fully convolutional neural networks in that it introduces skip connections, which allow information to be passed directly between the encoder and decoder, greatly enriching the contextual information and achieving better results on a large number of medical image segmentation datasets. However, U-Net network directly fuses the features of the encoder with the features of the decoder of the corresponding layer, so that the information of the feature map on the channel will become coarse, which will eventually lead to less than fine pixel segmentation at the edges of the lesion area. Especially for highly noisy dermoscopic images, U-Net directly fuses the features in the skip connection, which may embed the wrong feature information containing noise in the channel and will eventually affect the segmentation results of the model. Moreover, the continuous pooling of U-Net in the encoder leads to a decreasing spatial resolution of the feature map and the loss of spatial location information. Similar to the encoder-decoder architecture of U-Net, Dai et al. [15] proposed a novel Multi-scale Residual Encoding and Decoding network for skin lesion segmentation, which mines the semantic information of the feature map by using multi-scale residuals in the encoding and decoding paths, and proposes a new pooling method (Soft Pooling) in the process of down-sampling, which can obtain more useful information and can obtain more effective segmentation results. In CA-Net [16], a one joint spatial attention module is designed to make the network more focused on foreground regions, and a novel channel attention module is proposed to adaptively recalibrate the feature responses between channels and highlight the most relevant feature channels. In BA-Net [17], a multi-task learning module was designed for joint learning of segmentation of target masks and detection of lesion boundaries, and an interactive attention (IA) was designed to bridge the two tasks to achieve complementary information between the different tasks, effectively exploiting the boundary information and providing a strong cue for better segmentation prediction. Xie et al. [18] proposed a Mutual Bootstrapping Model, which combines dermoscopic segmentation and classification into one model, using the mask generated by segmentation in the model to assist in classification and the results of classification to assist in segmentation, so that the information between classification and segmentation is complementary and thus the performance of the model is improved. Cao et al. [19] proposed a Global and Local Inter-Pixel Correlations Learning Network for skin lesion segmentation that captures non-local contextual information at different levels and further explores global pixel-level relations to handle large variance in shape and size. Gu et al. [20] proposed a deep edge network with boundary information for automatic skin lesion segmentation and designed an entirety-center-edge loss (ECE) function that can be based on the necessary segmentation results Further optimization of the boundary details is possible. In WNet [21], the network of two encoder-decoders is connected in series to form a “W” shaped model and the encoder and decoder directly complement each other’s information, effectively enhancing the performance of the model. Hassanat [22] make use of some color spaces to segment pixels as either objects of interest or non-objects using artificial neural networks.

The attention mechanism have been introduced into deep learning as an effective means of enhancing features and are widely used in computer vision tasks. The ATTU-Net [23] based on U-Net incorporates an attention mechanism in the skip connection, which reinforces crucial feature information, leading to detailed processing of local target information for segmentation. Hu et al. [24] proposes an attention synergy module that optimally integrates spatial and channel information and introduces a weighted binary cross-entropy loss function to emphasize foreground lesion regions. DANet [25] solves the scene segmentation task by capturing rich contextual relevance based on a self-attention mechanism, as well as applying spatial and channel attention to gather information around the feature map. In SA-Net [26], an efficient Shuffle Attention (SA) is proposed based on channel attention and spatial attention, which effectively integrates contextual information. Wu et al. [27] designed a Feature Adaptive Transformation Network, which effectively captures global contextual information and resolves long-distance dependencies by integrating two-way features from the encoding stage transformer and convolution. He et al. [28] proposes a Fully transformer network for skin lesion analysis and designs a Spatial Pyramid Transformer that introduces spatial pyramidal pooling to multi-headed attention, greatly reducing computational resources. SENet [29] has proposed a novel and efficient module called Squeeze and Excitation, which explores the relationships between channels. Although the above works can accomplish automatically skin lesion segmentation, the ability to process the boundaries of lesions is not precise enough and the performance of these methods may be affected when the skin lesion is disturbed by some noises such as hairs, bubbles, etc.

In this paper, we propose an Ambiguous Context Enhanced Attention Network for skin lesion segmentation based on the encoder-decoder architecture. The Ambiguous Context Enhanced Attention module (ACEA) and the Cascading Global Context Attention module (CGCA) are designed for the skip connection between the encoder and the decoder. The ACEA uses a self-attention mechanism to fuse the feature information generated in the encoding stage with the coarse segmentation map output in the decoding stage, leading the model to enhance the ambiguous boundaries information. The CGCA module extracts features by convolution of different sizes to understand the information comprehensively, and passes the feature information generated by the encoder to the corresponding decoder layer so that the decoder can reply more accurately to the feature map at upsampling. To reduce the loss of spatial location information due to successive down-sampling in encoder, we introduced a Dilated Gated Fusion block (DFG) at the end of the encoding phase, which effectively mitigates the loss of spatial location information without changing the feature map size and number of channels, and preserves local and global semantic information in the corresponding regions. To evaluate the performance of the proposed model, we conducted experiments on the ISIC2018 dataset and the PH2 dataset. Our model obtains competitive results in four metrics: Dice score, Jaccard coefficient, Accuracy and Recall. The visualisation results show that our model makes accurate segmentation of skin lesions. In summary, the contributions of this article are as follows:

(1)
Based on the encoder-decoder architecture, we propose an Ambiguous Context Enhanced Attention Network for skin lesion segmentation, in which we employed a pre-trained resnet34 network in the encoder, and the coarse segmentation results output in each decoder layer are optimised by deep supervision manner.
(2)
We propose a novel Ambiguous Context Enhanced Attention module that leverages feature information from the encoder and the coarse segmentation map generated by the decoder using self-attention. This module enhances the boundary features of the lesion and guides the model in accurately segmenting the skin lesion.
(3)
At the end of the encoding phase, we incorporated a Dilated Gated Fusion block. This block captures multi-scale information from various perceptual fields and effectively mitigates the loss of spatial location information caused by repeated down-sampling.
(4)
In the skip connection, we introduced a Cascading Global Context Attention module. This module utilizes perceptual fields of different sizes to extract and transmit feature information from the encoder to the corresponding decoder layer. By incorporating features with abundant semantic information into the decoder, it enhances the accuracy of feature map restoration.

Figure 1.
The overall of the proposed model.

2. Method

The overall structure of the model is shown in Fig. 1, where the encoder uses a pre-trained Resnet34 [30] network. In each layer of the encoder, the feature map goes through two layers of 3 $\times$ 3 convolution to change the channel to twice its original size, and then the size is scaled to 1/2 of the original size by a pooling operation. After five layers of encoder processing, the feature map size becomes 8 $\times$ 8 $\times$ 512, and the number of channels changes from 3 to 512. The increase of channel number makes it contains rich information in the channel direction. At the same time, the continuous down-sampling leads to the loss of spatial information in the feature map, so at the end of the encoding phase we embed a DFG module, which effectively alleviates the problem of missing spatial location information and preserves the detail information in the channel direction. In the decoder part of each layer, the feature map will be output as a coarse segmentation map after two layers of 3 $\times$ 3 convolution and upsampling. We use deep supervision to calculate the loss and optimize the coarse segmentation map at each decoder layer, and these coarse segmentation map is multiplied by different weights as the final output of the model. We have designed the ACEA module and the CGCA module in the skip connection section between the encoder and the decoder. The ACEA module is based on a self-attention mechanism, which guides the model to enhance ambiguous boundary regions by fusing the feature information generated in the encoding phase with the coarse segmentation map generated in the decoding phase. The CGCA module fully captures the feature information generated by the encoder using convolution of different sizes. which obtains more contextual semantic information while preserving the detailed features of the data.

2.1 Ambiguous Context Enhanced Attention

In general, the edges of lesions are not smooth and the pixels in the transition region between the lesion and normal tissue have similar colour, texture characteristics. As a result, the boundaries of lesions are often ambiguous areas that are difficult to segment in medical image segmentation, and false negatives often occur in the prediction process. Specifically, the predicted probability value of lesion boundary pixels in the final segmentation result is usually close to 0.5, so it is difficult to accurately classify them as background or lesion.

To process the pixels at the boundary of the lesion, we output a portion of the coarse segmentation map in the decoding stage and optimise the pixels at the boundary of the lesion using deep supervision. Meanwhile, we design a novel Ambiguous Context Enhanced Attention module, which is based on a self-attention mechanism to enhance the boundary information of lesions by fusing the feature information generated in the encoding phase with the coarse segmentation map.

The structure of Ambiguous Context Enhanced Attention Module is shown in Fig. 2. where F is from the feature map generated by the encoder and M is the coarse segmentation map generated by the decoder. The foreground region, background region and ambiguous boundary region of M, respectively $M_{f}$ , $M_{b}$ and $M_{a}$ , are calculated as follows:

$\displaystyle M_{f}=\max(M-0.5)$ (1) $\displaystyle M_{b}=\max(0.5-M)$ (2) $\displaystyle M_{a}=0.5-\textit{abs}(M-0.5)$ (3)

where “max” means take the maximum value and “abs” means take the absolute value. we use “max” and “abs” operation to calculate the foreground region, the background region and the ambiguous boundary region, since the probability value of the ambiguous region is close to 0.5 in the predicted probability map, so we calculate the foreground and the background using 0.5 as a bound. The foreground, the background and the ambiguous boundary are then connected in the channel direction.

We calculate the representative vectors of $M_{f}$ , $M_{b}$ and $M_{a}$ by multiplying each pixel with F. The resulting representative vectors are $k_{f}$ , $k_{b}$ and $k_{a}$ . The calculation process is as follows:

$\displaystyle k_{f}=\sum\limits_{i\in p}{M_{f,i}}\times F_{i}$ (4) $\displaystyle k_{b}=\sum\limits_{i\in p}{M_{b,i}}\times F_{i}$ (5) $\displaystyle k_{a}=\sum\limits_{i\in p}{M_{a,i}}\times F_{i}$ (6)

where $k_{f}$ , $k_{b}$ and $k_{a}$ represent the foreground, the background and the ambiguous region, respectively. Then we calculate the similarity score between $k_{f}$ , $k_{b}$ and $k_{a}$ using F. The calculation process is as follows:

$\displaystyle x_{f}=(F_{i})^{T}\times k_{f}\quad x_{u}=(F_{i})^{T}\times k_{u}% \quad x_{a}=(F_{i})^{T}\times k_{a}$ (7) $\displaystyle s_{f}=\frac{e^{x_{f}}}{n}\quad s_{u}=\frac{e^{x_{u}}}{n}\quad s_% {a}=\frac{e^{x_{a}}}{n}\quad n=e^{x_{f}}+e^{x_{u}}+e^{x_{a}}$ (8) $\displaystyle r=\omega[s_{f}\times\alpha(k_{f})+s_{u}\times\alpha(k_{u})+s_{a}% \times\alpha(k_{a})]$ (9)

where $\omega$ and $\alpha$ are 1 $\times$ 1 convolution layer. $r$ is the similarity score of $k_{f}$ , $k_{b}$ and $k_{a}$ . Finally, the softmax [31] function is used to normalize the similarity scores to obtain a weight matrix. The weight matrix is multiplied by the representative vector and passed through a 1 $\times$ 1 convolution, after which it is added with F to obtain F’.

Figure 2.

The illustration of ambiguous context enhanced attention module.

2.2 Dilated gated fusion block

In the continuous encoding process, the number of channels of the feature map keeps increasing, and these channels contain rich semantic information. At the same time, the continuous down-sampling makes the resolution of the feature map keep decreasing, resulting in the loss of spatial location information. To solve this problem, we propose the Dilated Gated Fusion block, which captures the spatial information of the feature map at different scales through the atrous convolution with different dilation rates, passing more comprehensive and rich semantic information to the decoder.

Figure 3 shows the structure of the Dilated Gated Fusion block. Firstly, the feature maps are randomly divided into four groups, each containing 128 channels. These groups of feature maps are then subjected to atrous convolution with different dilation rates: 1, 2, 5, and 7. Next, four groups of feature maps are concatenated on the channels to obtain F’, and F’ is passed through a global average pooling and two layers of full concatenation action, aiming to learn the global feature information. Finally, a weight matrix is obtained after a sigmoid function, and the weight matrix is multiplied with F’ to enhance the spatial location information of the feature map.

Figure 3.

The illustration of dilated gated fusion block.

2.3 Cascading global context attention

The skip connection is a connection between the output of the encoder and the input of its corresponding decoder, which scales the feature information upwards for a specific layer. Adding skip connections between the encoder and decoder can preserve the detailed features of the input data. These connections allow the model to find local detail information and avoid missing important global information, and this connection helps the decoder to restore the original feature map more accurately, leading to improved segmentation accuracy. However, there would be a large semantic gap in the channel direction if the feature information generated by the encoder was directly connected to the decoder with the corresponding layer, so that there would be an imbalance of feature information on the channel. Therefore, to better control the feature information in the skip connection, we designed a Cascading Global Context Attention module. In this module, the feature information generated by the encoder is fused with the features of the decoder after convolutional coding with different dilation rates, mining the information on the channel and effectively conveying more detailed information and spatial contextual features to the decoder.

The structure of the CGCA is shown in Fig. 4. Firstly, the feature map $F_{l}$ is passed through convolution with a convolution kernel size of 3, 5, 7 and atrous convolution with a dilation rate of 3, 5, 7 respectively. By using different sizes of convolution, multiple scales of information can be extracted and the feature map can be fully understood. Then, the feature maps are processed by Adaptive pooling of different sizes and 1 $\times$ 1 convolution to obtain feature maps $p_{1}$ , $p_{2}$ and $p_{3}$ , where Adaptive pooling reduces the size of the feature maps and retains key information, 1 $\times$ 1 convolution adjusts the number of channels and removes redundant information. Finally, $p_{1}$ , $p_{2}$ and $p_{3}$ are concatenated in the channel direction and passed through a 1 $\times$ 1 convolution added with $F_{l-1}$ to obtain the final output result $F_{l+1}$ , where $F_{l-1}$ is the output of the CGCA from the previous layer.

Figure 4.

The illustration of cascading global context attention module.

2.4 Loss function

The loss function plays an important role in the training of the model. To reduce overfitting and to obtain excellent convergence speed, we adopt the Dice loss [32] and the binary cross-entropy loss [33] as the overall loss function. The Dice loss is not affected by the category imbalance problem and has been extensively validated in different medical image segmentation tasks and can be written as:

$\displaystyle{\cal L}_{\textit{Dice}}=1-2\times\frac{\sum_{i}P_{i}G_{i}}{\sum_% {i}P_{i}+\sum_{i}G_{i}}$ (10)

where i represents the ith pixels in the probability distribution map. $P_{i}$ is the probability that the ith pixel belongs to the segmented region and $G_{i}$ is the label of the ith pixel.

At the same time, skin lesion segmentation is a binary problem, so we also use the binary cross-entropy loss:

$\displaystyle{\cal L}_{\textit{Bce}}=\sum_{i}[(1-G_{i})\text{In}(1-P_{i})+G_{i% }\text{In}(P_{i})]$ (11)

where $P_{i}$ is the probability that the ith pixel belongs to the segmented area and $G_{i}$ is the label of the ith pixel.

Eventually, our overall loss function consists of a binary cross-entropy loss and a Dice loss function:

$\displaystyle{\cal L}=\lambda_{1}{\cal L}_{\textit{Bce}}+\lambda_{2}{\cal L}_{% \textit{Dice}}$ (12)

where $\lambda_{1}$ and $\lambda_{2}$ are empirically set to 0.6 and 0.4, respectively.

3. Experiment

Our method is conducted in the PyTorch library with an Intel(R) Xeon(R) Gold 5218 CPU@2.30 GHz and an NVIDIA Quadro RTX 6000 GPU(24G). The training epoch is 200. The batch size is 16. The operating system for the experiments was Ubuntu Server 16.04. We use Adam as the overall optimizer, where betas $\beta$ 1 $=$ 0.9, $\beta$ 2 $=$ 0.999 and weight decay $=$ 1e-5. The initial learning rate is set to 1e-4.

3.1 Datasets

We used public dermoscopic image datasets to train the proposed segmentation model, respectively the ISIC2018 dataset [34] and the PH2 dataset [35]. The two datasets are summarized as shown in Table 1. The ISIC-2018 dataset was provided by the International Skin Imaging Collaboration (ISIC), which contains a total of 2594 RGB images (20.0% melanoma, 72.0% nevus and 8.0% seborrheic keratosis) with image sizes ranging from 540 $\times$ 576 pixels to 4499 $\times$ 6748 pixels. As this dataset was not divided into training set, validation set and test set, we randomly divided the dataset into training set of 1816 images, validation set of 260 images and test set of 518 images. The PH2 dataset, the first public dataset to provide segmentation mask, contains 200 (20% melanoma, 40% atypical nevus, 40% normal nevus) dermoscopic images of size 768 $\times$ 560, which we divided into 140 training sets, 20 validation sets, and 40 test sets.

For the data pre-processing of two datasets, all image is resampled to 256 $\times$ 256 pixels and each pixel normalized in [0, 1] by mean and variance normalization. To enlarge the amount of training set, we will flip the data both horizontally and vertically. The transforms are applied to the original images during the training process. Thus, the images are augmented in real-time until the model training is completed.

We strictly separate the datasets into distinct subsets for training, validation, and testing. This ensures that the models are trained solely on the training set and evaluated independently on the unseen test set. When evaluating the performance of ACEANet on the test set, we adhere to strict guidelines and avoid applying any form of data augmentation. The test images remain in their original form, guaranteeing that the model’s performance is assessed solely based on its ability to denoise real-world, unaltered images.

Table 1
Distribution of the ISIC 2018 dataset and PH2 dataset

Dataset	ISIC2018	PH2
Training data	1816	140
Validation data	260	20
Test data	518	20
Resolution	540 $\times$ 576 $\sim$ 4499 $\times$ 6748	768 $\times$ 560
Input size	256 $\times$ 256	256 $\times$ 256

3.2 Evaluation metrics

To quantitatively assess the segmentation ability of the proposed network, we used four commonly used metrics for skin lesion segmentation evaluation: Dice score (Dice), Jaccard coefficient (Jac), Accuracy (Acc) and Recall. The Dice score is defined as the ratio of twice the intersection of the predicted and true segmentations to the sum of the sizes of the two segmentations.

The Dice score ranges from 0 to 1, with 1 indicating a perfect overlap between the predicted and true segmentations. Jaccard coefficient measures the similarity between the predicted segmentation and the ground truth by calculating the ratio of the intersection to the union of the two segmentations. Similar to the Dice score, the Jaccard coefficient also ranges from 0 to 1, with 1 indicating a perfect overlap between the predicted and true segmentations. Accuracy quantifies the percentage of correctly classified pixels or voxels in the segmentation result. The Accuracy metric ranges from 0 to 1, with 1 indicating a perfect segmentation where all pixels or voxels are correctly classified. Recall measures the ability of a segmentation algorithm to correctly identify all positive instances (pixels or voxels) belonging to the target region in the ground truth. The recall metric ranges from 0 to 1, with 1 indicating a perfect recall where all positive instances are correctly identified. Mathematically, they are defined by the following:

$\displaystyle\textit{Dice}=\frac{2\times TP}{2\times TP+FP+FN}$ (13) $\displaystyle\textit{Jac}=\frac{TP}{TP+FN+FP}$ (14) $\displaystyle\textit{Acc}=\frac{TP+TN}{TP+FP+TN+FN}$ (15) $\displaystyle\textit{Recall}=\frac{TP}{TP+FN}$ (16)

where TP is the number of pixels correctly segmented as lesion regions, TN is the number of pixels correctly segmented as non-lesion regions, FP is the number of pixels incorrectly segmented as lesion regions, and FN is the number of pixels incorrectly segmented as non-lesion regions.

Table 2

Comparison results on the ISIC2018 dataset. Those with “–” indicate that the corresponding metric results are not provided

Network	Year	Dice (%)	Jac (%)	Acc (%)	Recall (%)
U-Net [14]	2015	88.09	80.08	92.40	91.64
ATTU-Net [22]	2018	90.18	85.31	92.72	91.83
CENet [36]	2019	90.53	83.60	95.34	90.59
CANet [16]	2020	92.08	–	–	–
Attn_UNet [37]	2021	91.00	83.0	95.00	94.00
Swin-UNet [38]	2021	90.92	83.90	93.81	92.81
FAT-Net [26]	2022	89.03	82.02	95.78	91.00
Ms Red [15]	2022	89.99	83.45	96.19	90.49
AS-Net [23]	2022	89.55	83.09	95.68	93.06
UNeXt [39]	2022	89.76	81.70	–	–
RMMLP [38]	2023	91.98	85.40	–	–
GFANet [39]	2023	90.13	83.66	96.29	90.75
ACCPG-Net [42]	2023	90.81	83.51	96.13	91.18
ACEANet (ours)	2023	92.16	86.46	96.75	92.55

3.3 Results on the ISIC2018 dataset

A comparison of our method with other optimal models on the ISIC2018 dataset is shown in Table 2. As can be seen from the data in the table, it can be concluded that our model achieves better results for all metrics, especially for the three metrics of Dice, Jac and Acc, which all achieved the highest results with 92.16%, 86.46% and 96.75% respectively. Swin-Unet achieved the best results on the Recall metric, but on the other three metrics, the results were lower than our proposed model. FAT-Net achieves better results on Acc and Recall, but there are large gaps compared to other models in the two metrics Dice and Jac, whereas our model achieves better results on all metrics and there are no results with large gaps compared to other models, proving the stability of our model. In comparison with models of encoder-decoder architectures such as U-Net, ATTU-Net and AS-Net, our model outperforms these models in all metrics. Our model adds multiple skip connections between encoders and decoders that fuse features at different levels. This multi-scale feature fusion method can help the model to capture different features at different scales, so as to better segment. By skip connections, the model can utilize features from different levels, making the segmentation results more accurate.

Table 3
Comparison results on the PH2 dataset. Those with “–” indicate that the corresponding metric results are not provided

Network	Year	Dice (%)	Jac (%)	Acc (%)	Recall (%)
U-Net [14]	2015	90.71	86.82	91.04	90.28
ResU-Net [43]	2018	92.82	87.64	94.92	93.37
CENet [34]	2018	90.81	85.69	90.98	91.21
DSNet [44]	2020	91.97	87.15	94.82	96.01
CA-Net [16]	2020	90.45	–	–	–
SwinUNet [36]	2021	92.88	87.16	95.75	95.33
AS-Net [23]	2022	93.05	87.60	95.20	96.23
AMCC-Net [45]	2023	94.00	89.00	97.00	–
ACEANet (ours)	2023	94.18	89.27	96.24	95.65

Figure 5.

Segmentation results of different models on the ISIC2018 dataset. (a) input, (b) label, (c) ours, (d) ATTUNet, (e) Ms Red, (f) UNet.

A visualisation results of our method on the ISIC2018 dataset compared to other models is shown in Fig. 5, where the green area shows the segmentation results of the model. As can be seen from the Fig. 5, our method can accurately segment the area of skin lesions compared to other models. Our model compensates for the ambiguous lesion boundary pixels and optimizes the coarse segmentation map by deep supervision so that the final output is closer to the label. The proposed model pays more attention to important features by introducing an attention mechanism that assigns different weights to different image areas. This allows the model to better capture details and edges in the image, improving the accuracy of segmentation. Based on the segmentation results of U-Net and ATTU-Net, it can be observed that when there are large irregular lesion areas, the segmentation results of U-Net are better than ATTU-Net. It is worth noting that ATTU-Net based on U-Net only adds the attention mechanism in the skip connections. Therefore, we suspect that only adding the attention mechanism in the skip connections may affect the overall anti-interference ability of the model and reduce its robustness. When low contrast of the lesion is present, the MsRed model does not accurately segment the lesion. Overall, our proposed model achieves competitive results both in terms of metrics and in terms of segmentation results.

Figure 6.

Segmentation results of different models on the PH2 dataset.

3.4 Results on the PH2 dataset

The Table 3 summarizes the comparison results between our model and the current optimal models on the PH2 dataset. The data indicates that our model has achieved competitive results in all performance metrics, especially in Dice and Jac, with the best results of 94.18% and 89.27%, respectively. AS-Net achieved the highest result on the Recall metric, which is 0.61% higher than our model. However, our model’s results are higher than AS-Net’s by 1.13%, 1.67%, and 1.04% on the Dice, Jac, and Acc metrics, respectively. AMCC-Net achieved the best results on the Acc metric, exceeding our model by %0.76. However, our model performs better than AMCC-Net by 0.18% and 0.27% on the Dice and Jac metrics, respectively. Our model performs well on all metrics and there are no instances where a metric is too low. This proves that our model has the potential for practical use.

As shown in the Fig. 6, our model is visually compared with other models, and the red circle highlights the differences between segmentation results of the model and label. From the observation of the Fig. 6, it can be concluded that compared to other models, our method has superior segmentation performance, with segmentation results closest to the label. Our model outputs a coarse segmentation map in each layer decoder. Through continuous optimization of the coarse segmentation map, the model can better understand the semantics and structure of the image, and improve the accuracy and robustness of segmentation. For noisy skin lesion images with hair interference, such as the images in rows (b) and (c) in the Fig. 6, the segmentation performance of ResU-Net will be greatly affected, too much noise signal misguides ResU-Net and leads to over-segmentation, while our model can still segment the complete lesion area, proving that our model has strong anti-interference ability. When the boundary of the lesion is vague and the contrast is low, SwinU-Net cannot accurately identify the area of the lesion. For example, the area circled by the red box in row (a) in the Fig. 6 is a non-lesion area, but SwinU-Net classifies it as a lesion area, yet our model can accurately segment the boundary of the lesion. In summary, these results reveal that the ACEANet is more robust than other methods and can accurately segment various types of skin lesions.

3.5 Ablation studies

To evaluate the various modules in our model, we performed step-by-step ablation experiments on the ISIC 2018 dataset and designed relevant models for testing.

Model 1: Uses U-Net as the baseline and uses Resnet34 pre-trained weights in the encoder.

Model 2: Based on Model1 adding ACEA module at the skip connection.

Model 3: Based on Model1 adding the DGF module to the bottom of the encoder.

Model 4: Based on Model1 adding the CGCA module at the skip connection.

Model 5: Based on Model1 adding ACEA module and DGF module.

Model 6: Based on Model1 adding CGCA module and DGF module.

Model 7: Based on Model1 adding ACEA module, CGCA module.

Model 8: Based on Model1 adding ACEA module, CGCA module and DGF module.

The results of our ablation experiments are shown in the Table 4, where “–” indicates that the corresponding module has not been added to the current model and “ $\surd$ ” indicates that the corresponding module has been added to the current model. As can be seen from the data observed in the Table 4, Model2, Model3 and Model4 have improved in all metrics after adding the corresponding modules to Model1, proving that our modules are effective for dermoscopic image segmentation. After adding two modules to Model1, such as Model5, Model6 and Model7, the results of all three models are higher than the results of Model1. Adding the corresponding three modules to the Model1, such as Model 8, all indicators are at their highest. This proves once again that our proposed modules can help to improve the performance of the models.

Table 4
Ablation studies of different models on the ISIC2018 dataset

Model	ACEA	DGF	CGCA	Dice (%)	Jac (%)	Acc (%)	Recall (%)	Time (h)
1	–	–	–	88.09	80.08	92.40	91.64	7.3
2	$\surd$	–	–	90.23	84.98	94.41	92.15	8.2
3	–	$\surd$	–	91.01	84.75	93.89	92.21	7.9
4	–	–	$\surd$	91.15	84.95	94.37	91.19	8.1
5	$\surd$	$\surd$	–	91.73	85.84	95.62	92.32	8.6
6	–	$\surd$	$\surd$	91.97	84.11	96.02	91.98	8.8
7	$\surd$	–	$\surd$	91.03	84.81	94.45	91.51	8.8
8	$\surd$	$\surd$	$\surd$	92.16	86.46	96.75	92.55	9.2

4. Conclusion

The purpose of dermoscopic image segmentation is to accurately distinguish skin lesions from normal tissues in dermoscopic images to help doctors locate and diagnose various skin diseases more accurately. In this paper, we propose an Ambiguous Context Enhanced Attention Network for skin lesion segmentation named ACEANet. In the skip connection, we have designed a novel ACEA module. The ACEA module can guide the model to enhance the ambiguous lesion boundary information by using the coarse segmentation map generated by the decoder and the feature with rich semantic information produced by the encoder. The five-layer encoder gradually deepens the channels of the feature map, allowing the channels to contain rich semantic information. At the same time, continuous down-sampling results in a loss of spatial information in the feature map. Therefore, at the end of the encoding stage, we embedded a DFG module. This module divides the channel information into different groups, and then extracts features with different sizes of receptive fields. It greatly preserves both detail information and global semantic information. In the skip connections between the encoder and corresponding layers of the decoder, we have designed a CGCA module. This module processes the feature information generated by the encoder by using convolutions with different dilation rates and combines it with the corresponding feature map in the decoder to achieve feature fusion. The CGCA module enables the decoder to more accurately recover the information of the feature map. Experimental results have demonstrated the superior segmentation performance of our model compared to other optimal models on the publicly available ISIC2018 and PH2 datasets. Visual comparisons have also shown that our model can accurately segment lesions in dermatoscopic images. The proposed model can help doctors automatically and accurately segment dermoscopic images, thus providing faster and more accurate diagnosis. It has broad application prospect in CAD system.

References

Siegel

R.L.

Miller

K.D.

and Jemal

, Cancer statistics, 2019, CA: A Cancer Journal for Clinicians 69 (2019), 7–34. doi: 10.3322/caac.21551.

Vestergaard

Macaskill

Holt

and Menzies

, Dermoscopy compared with naked eye examination for the diagnosis of primary melanoma: A meta-analysis of studies performed in a clinical setting, British Journal of Dermatology 159 (2008), 669–676.

Mishra

N.K.

and Celebi

M.E.

, An overview of melanoma detection in dermoscopy images using image processing and machine learning, arXiv preprint arXiv:1601.07843, 2016. https://arxiv.org/abs/1601.07843.

Garcia-Arroyo

J.L.

and Garcia-Zapirain

, Segmentation of skin lesions in dermoscopy images using fuzzy classification of pixels and histogram thresholding, Computer Methods and Programs in Biomedicine 168 (2019), 11–19. doi: 10.1016/j.cmpb.2018.11.001.

Yueksel

M.E.

and Borlu

, Accurate segmentation of dermoscopic images by image thresholding based on type-2 fuzzy logic, IEEE Transactions on Fuzzy Systems 17 (2009), 976–982. doi: 10.1109/TFUZZ.2009.2018300.

Wong

Scharcanski

and Fieguth

, Automatic skin lesion segmentation via iterative stochastic region merging, IEEE Transactions on Information Technology in Biomedicine 15 (2011), 929–936. doi: 10.1109/TITB.2011.2157829.

Munawar

H.S.

Khalid

Jilani

and Maqsood

, Version management by time based approach in modern era, Int. J. Educ. Manag. Eng 7 (2017), 13–20. doi: 10.5815/ijeme.2017.04.02.

Yuan

Chao

and Lo

Y.-C.

, Automatic skin lesion segmentation using deep fully convolutional networks with jaccard distance, IEEE Transactions on Medical Imaging 36 (2017), 1876–1886. doi: 10.1111/j.1365-2133.2008.08713.x.

Korotkov

and Garcia

, Computerized analysis of pigmented skin lesions: A review, Artificial Intelligence in Medicine 56 (2012), 69–90. doi: 10.1016/j.artmed.2012.08.002.

10.

Yuan

and Lo

Y.-C.

, Improving dermoscopic image segmentation with enhanced convolutional-deconvolutional networks, IEEE Journal of Biomedical and Health Informatics 23 (2017), 519–526. doi: 10.1109/JBHI.2017.2787487.

11.

Mirikharaji

and Hamarneh

, Star shape prior in fully convolutional networks for skin lesion segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2018, pp. 737–745. doi: 10.48550/arXiv.1806.08437.

12.

Sarker

Kamal

Rashwan

H.A.

Akram

Banu

S.F.

Saleh

Singh

V.K.

Chowdhury

F.U.

Abdulwahab

Romani

et al., Slsdeep: Skin lesion segmentation based on dilated residual and pyramid pooling networks, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2018, pp. 21–29. doi: 10.48550/arXiv.1805.10241.

13.

Long

Shelhamer

and Darrell

, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.

14.

Ronneberger

Fischer

and Brox

, U-net: Convolutional networks for biomedical image segmentation, in: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, Springer International Publishing, 2015, pp. 234–241.

15.

Dai

Dong

Yan

Zhang

and Luo

, Ms red: A novel multi-scale residual encoding and decoding network for skin lesion segmentation, Medical Image Analysis 75 (2022), 102293. doi: 10.1016/j.inffus.2022.10.022.

16.

Wang

Song

Huang

Aertsen

Deprest

and Zhang

, CA-Net: Comprehensive attention convolutional neural networks for explainable medical image segmentation. IEEE Transactions on Medical Imaging 40(2) (2020), 699–711.

17.

Wang

Chen

Fan

and Li

, Boundary-aware context neural network for medical image segmentation, Medical Image Analysis 78 (2022), 102395.

18.

Xie

Zhang

Xia

and Shen

, A mutual bootstrapping model for automated skin lesion segmentation and classification, IEEE Transactions on Medical Imaging 39 (2020), 2482–2493. doi: 10.1109/TMI.2020.2972964.

19.

Cao

Yuan

Liu

Peng

Xie

Yang

and Zheng

, ICL-Net: Global and local inter-pixel correlations learning network for skin lesion segmentation, IEEE Journal of Biomedical and Health Informatics, 2022.

20.

Wang

and Zhang

, DE-net: A deep edge network with boundary information for automatic skin lesion segmentation, Neurocomputing 468 (2022), 71–84.

21.

Khouloud Ahlem

Fadel

and Amel

, W-net and inception residual network for skin lesion segmentation and classification, Applied Intelligence, 2022, 1–19. doi: 10.1007/s10489-021-02652-4.

22.

Hassanat

A.B.A.

Alkasassbeh

Al-awadi

and Alhasanat

E.A.A.

, Color-based object segmentation method using artificial neural network, Simul. Model. Pract. Theory 64 (2016), 3–17.

23.

Oktay

Schlemper

Folgoc

L.L.

Lee

Heinrich

Misawa

Mori

McDonagh

Hammerla

N.Y.

Kainz

et al., Attention u-net: Learning where to look for the pancreas, arXiv preprint arXiv:1804.03999, 2018. doi: 10.48550/arXiv.1804.03999.

24.

Lee

Xiong

and Chen

, As-net: Attention synergy network for skin lesion segmentation, Expert Systems with Applications 201 (2022), 117112. doi: 10.1016/j.eswa.2022.117112.

25.

Liu

Tian

Bao

Fang

and Lu

, Dual attention network for scene segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3146–3154.

26.

Zhang

Q.L.

and Yang

Y.B.

, Sa-net: Shuffle attention for deep convolutional neural networks, in: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021, June, pp. 2235–2239.

27.

Chen

Wang

Lei

and Wen

, Fat-net: Feature adaptive transformers for automated skin lesion segmentation, Medical Image Analysis 76 (2022), 102327. doi: 10.1016/j.media.2021.102327.

28.

Tan

E.L.

Zhang

Zhao

and Lei

, Fully transformer network for skin lesion analysis, Medical Image Analysis 77 (2022), 102357.

29.

Shen

and Sun

, Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.

30.

Zhang

Ren

and Sun

, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

31.

Liu

Wen

et al., Large-Margin Softmax Loss for Convolutional Neural Networks, in: Proceedings of The 33rd International Conference on Machine Learning, 2016, pp. 507–516.

32.

Milletari

Navab

and Ahmadi

S.A.

, V-net: Fully convolutional neural networks for volumetric medical image segmentation, in: 2016 Fourth International Conference on 3D vision (3DV), IEEE, 2016, October, pp. 565–571.

33.

Jadon

, A survey of loss functions for semantic segmentation, in: 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), IEEE, 2020, October, pp. 1–7.

34.

Codella

Rotemberg

Tschandl

Celebi

M.E.

Dusza

Gutman

Helba

Kalloo

Liopyris

Marchetti

et al., Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic), arXiv preprint arXiv:1902.03368, 2019. doi: 10.48550/arXiv.1902.03368.

35.

Mendonça

Ferreira

P.M.

Marques

J.S.

Marcal

A.R.

and Rozeira

, Ph 2-a dermoscopic image database for research and benchmarking, in: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, 2013, pp. 5437–5440. doi: 10.1109/EMBC.2013.6610779.

36.

Cheng

Zhou

Hao

Zhao

Zhang

Gao

and Liu

, Ce-net: Context encoder network for 2d medical image segmentation, IEEE Transactions on Medical Imaging 38 (2019), 2281–2292. doi: 10.1109/TMI.2019.2903562.

37.

Arora

Raman

Nayyar

and Awasthi

, Automated skin lesion segmentation using attention-based deep convolutional neural network, Biomedical Signal Processing and Control 65 (2021), 102358.

38.

Codella

Rotemberg

Tschandl

Emre Celebi

Dusza

Gutman

and Halpern

, Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC), arXiv e-prints, arXiv-1902, 2019.

39.

Valanarasu

J.M.J.

and Patel

V.M.

, Unext: Mlp-based rapid medical image segmentation network, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part V, Cham: Springer Nature Switzerland, 2022, September, pp. 23–33.

40.

Deng

Ding

Zhou

and Xiao

, RMMLP: Rolling MLP and matrix decomposition for skin lesion segmentation, Biomedical Signal Processing and Control 84 (2023), 104825.

41.

Qiu

Feng

Zuo

Liang

and Xu

, GFANet: Gated Fusion Attention Network for skin lesion segmentation, Computers in Biology and Medicine 155 (2023), 106462.

42.

Zhang

Zhao

and Yuan

, ACCPG-Net: A skin lesion segmentation network with Adaptive Channel-Context-Aware Pyramid Attention and Global Feature Fusion, Computers in Biology and Medicine, 2023, 106580.

43.

Zhang

Liu

and Wang

, Road extraction by deep residual u-net, IEEE Geoscience and Remote Sensing Letters 15 (2018), 749–753. doi: 10.1109/LGRS.2018.2802944.

44.

Hasan

M.K.

Dahal

Samarakoon

P.N.

Tushar

F.I.

and Martí

, DSNet: Automatic dermoscopic skin lesion segmentation, Computers in Biology and Medicine 120 (2020), 103738.

45.

Dayananda

Yamanakkanavar

Nguyen

and Lee

, AMCC-Net: An asymmetric multi-cross convolution for skin lesion segmentation on dermoscopic images, Engineering Applications of Artificial Intelligence 122 (2023), 106154.

ACEANet: Ambiguous Context Enhanced Attention Network for skin lesion segmentation

Abstract

Keywords

1. Introduction

2.1 Ambiguous Context Enhanced Attention

3.1 Datasets

Table 1 Distribution of the ISIC 2018 dataset and PH2 dataset

Table 3 Comparison results on the PH2 dataset. Those with “–” indicate that the corresponding metric results are not provided

3.5 Ablation studies

Table 4 Ablation studies of different models on the ISIC2018 dataset

References

Table 1
Distribution of the ISIC 2018 dataset and PH2 dataset

Table 3
Comparison results on the PH2 dataset. Those with “–” indicate that the corresponding metric results are not provided

Table 4
Ablation studies of different models on the ISIC2018 dataset