MTC-Net: Multi-scale feature fusion network for medical image segmentation

2.1 CNN-based methods

Convolutional neural networks are an important network framework in the field of deep learning, particularly in computer vision. CNN began with LeNet in the 1990s to solve the visual task of handwritten digit recognition, and the basic architecture includes convolutional, pooling, and fully-connected layers. Then came the AlexNet architecture, which has five convolutional layers and three fully-connected layers and uses ReLU instead of Sigmoid to accelerate SGD convergence. And GoogleNet replaces all of the later fully connected layers with simple global average pooling, propelling convolutional neural network research to new heights. ResNet, which was recently proposed in [6], implements a simple constant mapping by jumping connections and has demonstrated advanced performance in classification and segmentation tasks.

CNNs are widely used in medical image classification and segmentation tasks due to their efficient and accurate performance. By fusing all layer he features with an edge-guided module and modifying the jump-join part of the traditional UNet, DEA-UNet solves the problem of blurred lesion edges [7]. To better deal with complex images, [8] proposed CA-Unet++, which solved the loss of eigenvalues in the long jump connection process and upsampling process, respectively, using the channel module and attention module to achieve better image segmentation efficiency and accuracy. The proposed MR-UNet model of [9] uses bilinear interpolation instead of deconvolution in the UNet model, and the encoder includes residual blocks to improve segmentation capability. [10] proposed an attention-based residual dense depth network that can better preserve image details by combining the attention mechanism and residual connectivity. Multiple downsampling is prone to information redundancy in the encoder-decoder framework, and [11] proposed a remote image change detection network based on a multi-feature self-attention fusion mechanism, which can well avoid this drawback by adding a multi-functional self-attention mechanism to obtain richer contextual information. Among the skin lesion segmentation tasks, [12] proposed a new semi-automatic skin lesion segmentation method that overcomes the problem of low contrast between the lesion region and the surrounding healthy skin by combining a full convolutional network with multi-scale integration. Similarly, [13] proposed a new dense pooling layer for accurate skin lesion boundary identification and improved segmentation accuracy. FCN (fully connected network) and U-Net, two neural networks, are prone to parameter redundancy. [14] proposed an improved skin lesion segmentation model based on deformable 3D convolution and ResU-NeXt++, which improves the Jaccard index of skin lesion segmentation by standard reverse propagation for efficient training and improved Jaccard index for skin lesion image segmentation. Ms RED [15] improved the network’s learning capability by replacing the traditional convolutional layer in the encoder and decoder networks with a multi-channel feature fusion module. FAT-Net [16] included a novel functional adaptive transformer network based on the classical encoder-decoder architecture that can effectively capture remote dependencies and global contextual information, as well as improve feature fusion between adjacent level features by activating effective channels and suppressing irrelevant background noise. However, the traditional convolutional neural network-based approach has limitations in capturing rich global contextual information, and more progress is required.

2.2 Transformer

Since its introduction, the Transformer model has been widely used in natural language processing, computer vision, and many other fields. It is essentially an Encoder-Decoder architecture, with the encoding component consisting of multiple layers of encoders (Encoder), each of which is composed of two sub-layers: the self attention layer and a feedforward network layer. The Decoder, on the other hand, is made up of three sub-layers: a multi-headed self-attentive layer, an additional layer capable of multi-headed self-attentive of the encoder output, and a fully connected layer. Many improved models have been developed in recent years as a result of Transformer’s satisfactory results in the field of image processing. The VIT model divides the image into multiple patches, each of which is mapped into a fixed dimension using a NN network and input to a subsequent Transformer Encoder; when the image is classified, an identity token is added at the beginning of the sequence. When pre-trained on a large amount of sufficient data, VIT can achieve good results. The TNT model [17] further subdivides the patches in the original VIT into sub-patches, treating each patch as a sentence and the elements obtained by further subdivision of the patches as words. The model can find more similar sub-patch groups in the data, improving the model’s learning ability and generalization on the data. [18] proposed a Transformer model for pixel-level image tasks, incorporating the Pyramid CNN concept into Transformer and capturing finer-grained information by increasing the initial resolution and decreasing the resolution layer by layer, while reducing the running overhead. Swin Transformer [19] uses local attention to divide patches into windows, and attention between patches is only performed within the windows to improve operational efficiency. Swin Transformer also proposes the Shifted Window method, which uses different window configurations in different layers, and the window positions in the next layer will be shifted horizontally and vertically by 2 patches in order to make the patches within different windows in the previous layer interact with information. Transformer’s application in the field of computer vision has gradually matured, making a significant contribution to the field of computer vision and multimodality.

3.1 Network architectures

Figure 1 depicts the network’s overall design. For precise and dependable segmentation of lesion locations in medical imaging, we present a multi-scale feature fusion network (MTC-Net). To improve MTC-Net’s ability to segment lesions and deal with common issues like blurred lesion boundaries, low contrast, and irregular shapes, we built a network with four key components: a fusion encoder, a global cooperative aggregation module (GCAM), a detail enhancement module (DEM), and a multi-branch multiscale feature fusion module (MSFB). The fusion encoder, in particular, is used to extract multi-scale distant dependent features from the input image while keeping rich global context information. The first two layers of the encoder extract high-resolution low-level features with a considerable quantity of boundary information, which serves as a vital guide for the network’s boundary learning. The latter two levels can extract extensive global contextual information. The decoder’s multi-branch multi-scale feature fusion module builds multi-scale feature representations by separating blocks into layers, and GCAM fuses the features learned in each layer of the decoding stage to increase segmentation accuracy even further. DEM accepts encoder features and improves interclass discrimination and intraclass response via two parallel attention modules. The output is subsequently sent to the decoder for improved segmentation performance. Our network can efficiently fulfill the task of segmenting medical image lesions using the approaches described above.

OPEN IN VIEWER

Fig. 1

The proposed MTC-Net network framework.

As shown in Fig. 1, we use a fusion encoder that combines translational invariance, input adaptive weighting, and a global perceptual field based on MBConv blocks and self-focus. Since shallow features mainly include some broad object feature patterns such as texture, color, and orientation, these features are usually not global, but deep features represent object-specific information, which usually requires global information. Therefore, we build a four-stage encoder structure that uses MBConv to capture spatial interactions in the first two stages and then performs deep convolution with inverted residuals. In addition to this, both depth-separated convolution and self-attention can be described as the sum of the weighted values of each dimension in the pre-defined perceptual field. The self-attention module(Multi-head attention module and feedforward network) is used in the latter two stages to provide additional contextual information through the broader perceptual field.

3.3 Multi-branch and multi-scale feature fusion module

To enhance the ability of the network to extract richer hierarchical features, combining the ideas from [20] and [21], we propose the multi-branch multi-scale feature fusion module (MSFB) shown in Fig. 2, whose internal structure consists of three components: multi-branch convolutional layer, null convolution and hierarchical segmentation block. The multi-branch structure is composed of convolutional layers of convolution kernels of different sizes. The feature graph is divided into s groups by adding segmentation blocks, and each group has w channels. Only the first set of filters can be connected directly to the next layer. The second set of feature maps is sent to a 3×3 void convolution to extract features, and then the output feature maps are divided into two subgroups on the channel dimension. One subgroup of feature maps is connected directly to the next level, while another subgroup is connected in series to the next set of input feature maps in the channel dimension. The series feature map consists of a set of 3×3 void convolution operations. This method is repeated until all of the input feature maps have been processed. Finally, all input group feature maps are cascaded and delivered to another layer of 1×1 convolution to reconstruct the features. The reconstructed feature maps produce a more detailed feature representation.

OPEN IN VIEWER

Fig. 2

Multi-branch and multi-scale feature fusion module structure.

We propose GCAM in order to fully fuse the feature information of different levels of the decoder and obtain more accurate segmentation results (as in Fig. 1). The features at layer l in the decoder are denoted as D _l (l ∈ {1, 2, 3, 4}), and they are unified to D₁ size using a bilinear interpolation transform and 3×3 convolution, before being connected and denoted as feature F. Soft pooling with multilayer perception (MLP) is used to obtain the coefficients of each channel of F, and the channel coefficient attention vector is denoted as a ∈ [0, 1], in addition to using a 3×3 null convolution and a 1×1 convolution layer to generate spatial attention coefficients b ∈ [0, 1]. GCAM’s output is represented as: Y GCAM = F · a · b + F · a .(1)

We introduce a detail enhancement module to improve the network’s ability to learn irregular boundary features in the lesion region (e.g., Fig. 1). The DEM is made up of two modules: spatial attention and channel attention. The spatial attention module is used to capture the broader contextual information of local features, and the generated features are represented as Q _y and K _x by two convolutional layers, 1×3 and 3×1, to extract edge information in the vertical and horizontal directions, respectively. 1×1 convolution is also applied to the input features to obtain a new feature map V. The transpose of Q is matrix multiplied with K, which can promote each other in similar spatial points and suppress each other for different spatial points suppress each other; after that, the softmax layer is applied to obtain the similarity note map of each position with the others. Then matrix multiplication is performed with the features V to obtain the enhanced features, and finally the final output features F _s of the spatial attention module are obtained by summing with the initial input features F. The channel attention module exploits the interdependence of channel mapping to enhance the contrast between class-dependent features. By matrix multiplying the input features F with their transpose, similar channels will promote each other and different channels will suppress each other. The channel dependency matrix is then Softmaxed so that the network can enhance the ability to recognize curve structure and background. The output features F _c are then obtained by multiplying and then summing with the features F. Finally, the attentional feature map information generated in both dimensions is integrated by matrix multiplication and convolution layer operations to produce the output of the DEM module. The final feature map looks like this: T = conv ( sab ( x ) ) · cab ( x ) + conv ( sab ( x ) ) .(2) where x is the input feature map, cab (x) is the channel attention module, and sab (x) is the spatial attention module.

Lesion segmentation is a binary classification task, and in this work we use a combination of the binary cross-entropy (BCE) loss function and dice coefficient (Dice) loss in a way that the BCE loss and dice loss are defined as: L BCE = - ∑ i [ ( 1 - G i ) ln ( 1 - P i ) + G i ln ( P i ) ] .(3) L Dice = 1 - 2 × ∑ i P i G i ∑ i P i + ∑ i G i .(4) where P _i is the probability that the i pixel belongs to the segmented region and G _i is the ground truth value of the i pixel. Our loss function consists of a binary cross-entropy loss and a dice loss function: L loss = L BCE + L Dice .(5)

4.1 Datasets

We conducted extensive experiments on four public medical image segmentation datasets to validate the effectiveness of our method, including the skin lesion image dataset ISIC 2018 [22], the breast ultrasound image dataset BUSI [23], the gastrointestinal polyp image dataset Kvasir-SEG [24], and the thyroid nodule dataset TN3K [25]. Among them, the ISIC 2018 dataset is a challenge dataset for "Skin lesion analysis for melanoma detection" sponsored by the International Symposium on Biomedical Imaging (ISBI) 2018.The ISIC 2018 dataset consists of 2594 color RGB images, which we resampled to 224 × 224 pixels and randomly divided into training set (70%), validation set (10%) and test set (20%). The BUSI dataset consists of 780 images in PNG format (133 normal images, 437 benign images, 210 malignant images), which we randomly divided into training set, validation set and test set by 7:2:1. the Kvasir-seg dataset contains 1000 polyps images from the Kvasir dataset v2 and their corresponding ground The resolution of the images contained in the Kvasir-SEG ranged from 332×487 to 1920×1072 pixels, and we resampled the dataset to 224 × 224 pixels and randomly divided it into training set (70%), validation set (10%), and test set (20%). The TN3K dataset contains 3493 ultrasound images, which are also resampled to 224 × 224 pixels and randomly divided into a training set, a validation set and a test set according to 7:2:1.

4.2 Experimental settings and evaluation criteria

Experimental settings. We built our network in PyTorch on a single NVIDIA GeForce RTX 2080Ti GPU card, using adaptive moment estimation (Adam) as the overall optimizer, with the initial learning rate set to 0.001 and the weights decaying to 0.00005, while employing the CosineAnnealingWarmRestarts Restart (T0 = 10, Tmult = 2) learning rate strategy. The network was trained for 200 periods to use the model with the highest Dice index on the validation set for the test set.

Evaluation Criteria. We used four widely used metrics to evaluate the accuracy performance of our proposed method, including Dice coefficient (Dice), Accuracy (ACC), Sensitivity (SE), Precision, intersection over union (IoU), and Specificity (SP), as follows: Dice = 2 · TP 2 · TP + FP + FN .(6) ACC = TP + FN TP + TN + FP + FN .(7) Sensitivity = TP TP + FN .(8) Specificity = TN TN + FP .(9) Precision = TP TP + FP .(10) IoU = TP TP + FN + FP .(11) where TP, TN, FP, and FN are true positive, true negative, false positive, and false negative, respectively. TP indicates the number of correctly segmented lesion pixels. TN indicates the number of correctly segmented background pixels. FP indicates background pixels that were incorrectly labeled as lesion pixels, and FN indicates lesion pixels that were incorrectly labeled as background pixels.

For the ISIC 2018 dataset, we compared with 10 methods, among which there are three general-purpose segmentation networks, U-Net [2], AttU-Net [26], and DeepLabv3+ [27], and seven networks dedicated to skin lesion segmentation, CE-Net [28], BCDU-Net [29], CPF- Net [30], CA-Net [31], BA-Transformer [32], MFS-Net [33], and FAT-Net [16]. All experiments were done under the same computational environment conditions, and none of them performed data enhancement on the dataset. Table 1 depicts the statistical experimental results of skin damage segmentation on the ISIC 2018 dataset for different methods, it is obvious that our method on the four evaluation metrics of Dice coefficient (Dice), Accuracy (ACC), Sensitivity (SE) and Specificity (SP) all showed better results with 93.17%, 96.60%, 91.78% and 97.40%, respectively. Figure 3 compares the segmentation results of different networks on the ISIC 2018 dataset, and our networks all show better processing capabilities.

OPEN IN VIEWER

Fig. 3

Comparison visualization on the ISIC2018 dataset.

We compared our network to seven other approaches on the BUSI dataset: U-Net [2], AttU-Net [26], DeepLabv3+ [27], CE-Net [28], BCDU-Net [29], DA-Net [34], and MCF-Net [35]. Table 2 displays the experimental outcomes. Our network MTC-Net outperforms the state-of-the-art breast nodule segmentation network, and our network performs better in all four evaluation metrics. MTC-Net outperformed DeepLabv3+ by 7.8%, 1.4%, and 0.35% in Dice coefficient (Dice), Accuracy (ACC), and Specificity (SP), respectively. Our MTC-Net performs better in terms of segmentation. Figure 4 compares the segmentation results of various networks on the BUSI dataset, and all of our networks outperform.

OPEN IN VIEWER

Fig. 4

Comparison visualization on BUSI datasets.

On the Kvasir-SEG dataset, we compared with six methods, namely U-Net [2], AttU-Net [26], CE-Net [28], BCDU-Net [29], DA-Net [34], and MCF-Net [35]. The detailed comparison results are shown in Table 3. Our network achieves superior results in all four evaluation metrics, and it can be seen that the best result among the state-of-the-art segmentation networks is DA-Net [34], while our network compares with it in terms of Dice coefficient (Dice), Accuracy (ACC), Sensitivity (SE) and Specificity (SP) by 13.72%, 1.95%, 7.16% and 0.87%, respectively. The segmentation performance of the above networks with our MTC-Net network on the Kvasir-SEG dataset is shown in Fig. 5.

OPEN IN VIEWER

Fig. 5

Comparison visualization on the Kvasir-SEG dataset.

On the TN3K dataset, we compared with seven methods, namely U-Net [2], AttU-Net [26], BCDU-Net [29], SGUNet [36], TRFE [37], TRFE+ [38], and CPF- Net [30]. The detailed comparison results are shown in Table 4. Our network has improved 9.92% and 4.48% in Dice coefficient (Dice) and Specificity (SP), respectively, compared to U-Net. The segmentation performance of our MTC-Net network on the TN3K dataset is shown in Fig. 6.

OPEN IN VIEWER

Fig. 6

Comparative visualization on TN3K datasets.

To demonstrate the effectiveness of our proposed module, we conducted ablation experiments on the ISIC2018 dataset. We compare the following models and present the results in Table 5:

Baseline: Only a hybrid encoder and decoder network.

Model 1: The MSFB module was added on the basis of baseline.

Model 2: The DEM module is added on the basis of baseline.

Model 3: The GCAM module is added on the basis of baseline.

Model 4: The GCAM module was added to Model 1.

The graphs of experimental results before and after adding the detail enhancement module are shown in Fig. 7, which proves the contribution of the module to the fuzzy boundary of network segmentation.

OPEN IN VIEWER

Fig. 7

Effect comparison of DEM on ISIC2018 dataset.