FDB-Net: Fusion double branch network combining CNN and transformer for medical image segmentation

Abstract

BACKGROUND:

The rapid development of deep learning techniques has greatly improved the performance of medical image segmentation, and medical image segmentation networks based on convolutional neural networks and Transformer have been widely used in this field. However, due to the limitation of the restricted receptive field of convolutional operation and the lack of local fine information extraction ability of the self-attention mechanism in Transformer, the current neural networks with pure convolutional or Transformer structure as the backbone still perform poorly in medical image segmentation.

METHODS:

In this paper, we propose FDB-Net (Fusion Double Branch Network, FDB-Net), a double branch medical image segmentation network combining CNN and Transformer, by using a CNN containing g ⁿConv blocks and a Transformer containing Varied-Size Window Attention (VWA) blocks as the feature extraction backbone network, the dual-path encoder ensures that the network has a global receptive field as well as access to the target local detail features. We also propose a new feature fusion module (Deep Feature Fusion, DFF), which helps the image to simultaneously fuse features from two different structural encoders during the encoding process, ensuring the effective fusion of global and local information of the image.

CONCLUSION:

Our model achieves advanced results in all three typical tasks of medical image segmentation, which fully validates the effectiveness of FDB-Net.

Keywords

Medical image processing dual-branch network convolutional neural networks transformer feature fusion

1 Introduction

Medical imaging is important for the diagnosis, treatment and evaluation of diseases, and its non-invasive imaging modality enables localization, characterization and quantitative analysis of diseases. Medical imaging comprises several types of procedures, such as computed tomography, X-ray, magnetic resonance imaging, and positron emission tomography [1]. Medical image segmentation is crucial for computer-aided disease diagnosis (CAD), treatment planning, and surgical pre-evaluation. It effectively assists doctors in observing and analyzing medical images, such as Magnetic Resonance Imaging (MRI), X-rays, and computed tomography images. The accuracy of medical image segmentation has a direct impact on the clinician’s ability to quantify the focal area, assess the patient’s condition, and select treatment.

Since U-Net [2], a medical image segmentation network based on a full convolutional neural network, was proposed in 2015, various CNN networks with U-Net as a variant have become the mainstay of medical image segmentation tasks [3–6]. Among them, the network structure of U-Net and its variants consists of an encoder and a decoder, which is shaped like a U-shape. Compared with the previous full convolutional neural network, U-Net greatly alleviates the network degradation problem caused by the excessive depth of the network layers, due to the innovative skip-connection module of U-Net, which is used to retrieve the detailed information and features of different granularity that are lost in the process of encoder downsampling. Although CNN-based network models have achieved excellent performance in medical image segmentation, CNN networks often have difficulty in obtaining the global receptive field due to the limitations of the convolution operation, and the performance of CNN networks in image segmentation tends to be severely inhibited due to the inherent bias of convolution [7]. Transformer is one of the most widely used models [8] in natural language processing (NLP) and has achieved great success in paraphrase generation [9], text-to-speech synthesis [10], and speech recognition [11]. Inspired by it, in 2020, Google researcher Alexey Dosovitskiy et al. introduced Transformer to computer vision tasks by proposing Vision Transformer [12], which mimics the input image by slicing it into several 16x16 slices and then mapping the slices into a one-dimensional sequence of two-dimensional image chunks (tokens), to mimic the input of an NLP task. The sequence of image blocks is then fed into the Encoder part of the Transformer for global attention computation operation, so that the image features have a global receptive field with long-range dependency that traditional CNN networks do not have. Such a Transformer computation operation can both avoid computing attention between each pixel, which greatly increases the computational and storage burden, and directly apply the advantages of the Transformer model in NLP tasks to the field of computer vision. However, although the Transformer structure can effectively construct remote dependencies between image blocks (Token), it lacks the ability to extract local features from images, and the performance of Transformer relies on the pre-training of large-scale datasets to reach or exceed the level of CNN networks. But due to the specificity of medical images, it is often difficult for researchers to obtain large-scale medical image datasets with doctors’ hand-annotated medical image datasets, so Transformer is often not as good as expected in medical image segmentation. In recent years, hybrid network architectures combining CNN and Transformer have gradually become hotspots for researchers in the field of medical image segmentation. For example, Jieneng Chen et al. proposed TransUNet [13], which innovatively extracts local feature maps of the image to be segmented through the CNN network first to be used in the original Vision Transformer to replace the image input. The Vision Transformer retrieves the local details of the CNN network through skip-connection at the Decoder layer. Guoping Xu et al. proposed LeViT-UNet [14], which integrates the Transformer and CNN into one Encoder, in which the multiscale feature maps generated by the two networks in the Encoder are passed through skip-connection to the decoder, effectively reusing the spatial information of the feature maps. However, existing hybrid network models still have some issues in medical image segmentation tasks: (1) The models fail to fully leverage the feature extraction capabilities of two different networks during the encoding stage, making it difficult to effectively integrate global and local features of different scales. (2) The models cannot utilize the extracted multi-scale features effectively. Inspired by these studies, we propose FDB-Net, a novel double-branch medical image segmentation model that integrates both CNN and Transformer architectures. FDB-Net effectively leverages the local feature extraction capabilities of CNN along with the global attention mechanism of Transformer to achieve precise medical image segmentation. Our proposed FDB-Net model comprises an encoding module, a decoding module, and a feature fusion module. The encoding module includes a CNN network with the g ⁿConv convolutional kernel and a Transformer variant network featuring Varied-Size Window Attention (VWA).

Additionally, to effectively fuse global and local information and enhance the recovery of coarse- and fine-grained details across all scales, our model incorporates the novel feature fusion module DFF. This module dynamically adjusts feature combinations within the jump connections, introducing both channel and spatial attention mechanisms. Furthermore, it utilizes channel attention to normalize feature details, thereby suppressing irrelevant information and improving segmentation accuracy, particularly in key regions. Experimental results across various segmentation tasks, including abdominal multi-organ segmentation, dermatological lesion region segmentation, and heart segmentation, demonstrate that our model outperforms state-of-the-art methods. The primary contributions of this paper are outlined as follows:

•
Considering the different advantages of CNN and Transformer, a new double branch network, FDB-Net, is proposed to efficiently extract feature information from medical images.
•
A new feature fusion module (DFF) is proposed for fusing global and local features extracted from two branches at different scales during the coding process and suppressing feature expressions that are not related to the segmentation target to avoid interference with the segmentation results.
•
The experimental results show that compared with the existing medical image segmentation methods, the proposed FDB-Net in this paper has high effectiveness and superiority.

2 Related works

2.1 CNN-based segmentation networks

In recent years, CNN networks have become a standard in medical image segmentation due to increasing computational power, particularly with the introduction of U-Net[2]. This method, based on convolutional neural networks, has emerged as a mainstream approach in medical image segmentation tasks. In contrast to previous full convolutional neural networks (FCNs [15]), U-Net proposes a U-shaped encoder-decoder structure combined with skip connections. This architecture retrieves detailed information lost during the downsampling process, providing coarse and fine feature maps for the segmentation model. It accelerates convergence and proves highly effective in handling medical image segmentation tasks, resulting in superior segmentation accuracy compared to almost all previous state-of-the-art models. U-Net has become a reference for subsequent model designs, including Res-UNet [16], Res-UNet++ [17], Dense-UNet [18], U-Net++ [19], and U-Net3+ [6]. These models enhance the ability to extract medical image features by refining the U-Net’s network structure, incorporating additional skip connections, and redesigning the codec structure. However, the pure convolutional network’s limited sensory field restricts its ability to establish global long-range dependencies and fully utilize contextual information.

In order to address the issue of the restricted receptive field in CNNs, Koltun et al. proposed Dilated Convolution [20]. This technique expands the convolution kernel size by inserting spaces (zeros) between kernel elements, effectively enlarging the receptive field and capturing multiscale contextual information. However, Dilated Convolution still suffers from the Gridding Effect, leading to insufficient correlation of long-distance information. To mitigate this, Kaiming He et al. introduced the Spatial Pyramid Pooling (SPP) structure [21], leveraging the classical spatial pyramid feature extraction method to extract image features at various scales from the same image. This improves segmentation accuracy. Additionally, the spatial pyramid can be combined with null convolution to create the Atrous Spatial Pyramid Pooling (ASPP) method [22], further optimizing segmentation tasks. Despite extensive efforts in CNN network optimization, inherent convolutional operation deficiencies persist, often introducing new challenges alongside optimization methods.

2.2 Vision transformers

Transformers have long been a fundamental approach in Natural Language Processing (NLP). Recently, researchers have extended their application to Computer Vision (CV) [12], yielding excellent results and establishing Transformers as another cornerstone in CV, following in the footsteps of CNNs. Unlike CNNs, the unique design of Transformers enables them to capture long-range dependencies and perform sequence-to-sequence (seq2seq) tasks. A Transformer consists of encoders and decoders, both incorporating positional encoding, multi-head attention mechanisms, Layer Normalization (LN) [12], Feed Forward Networks (FFNs), and Skip-Connections. The decoder is similar to the encoder but includes a masked multi-head attention mechanism in the input layer. Despite its advantages, Transformers still face challenges such as high computational complexity and reliance on large-scale pre-training, limiting their application in CV, particularly in medical image processing. Touvron et al. proposed DeiT [24], introducing a distillation strategy to enable advanced results with minimal training resources. X. Dong et al. proposed CSWin Transformer [25], which reduces computational costs by computing self-attention in cross-shaped windows both vertically and horizontally in parallel, effectively mitigating the high computational cost of global and local self-attention. Additionally, Local Enhanced Position Encoding (LEPE) is proposed to enhance the handling of local position information, further improving network performance. Z. Liu et al. introduced the Swin Transformer [26], which incorporates scale variation and employs a hierarchical construction method akin to CNN. Moreover, the proposed localization concept and self-attentive computation, featuring non-overlapping window regions, effectively reduce computational complexity. Hu Cao et al. proposed Swin-Unet [27], a pure Transformer architecture within the U-Net framework for medical image segmentation. In this architecture, labeled image blocks are input into a Swin Transformer-based U-shaped Encoder-Decoder via skip connections, facilitating local-global semantic feature learning. This approach yields outstanding performance in medical image segmentation tasks.

2.3 Hybrid CNN-transformer architecture

Pure CNN structures, when used as the backbone of medical image segmentation networks, often lack the ability to model the image globally over long distances due to the inherently small receptive field of convolutional operations, which leads to difficulties in achieving the expected performance of CNN networks in segmenting medical images. Pure Transformer structured networks are poor in image fine-grained feature extraction and rely too much on pre-training with large-scale datasets, which conflicts with the scarcity of medical images. Recently, networks combining CNN and Transformer have been considered by more and more researchers, and TransUNet [13] is the first network that applies a hybrid CNN and Transformer architecture to the field of medical image segmentation. This hybrid approach differs from pure Transformers, as TransUNet first utilizes the CNN layer to extract local features from the input medical image, and then inputs the extracted feature maps into Transformer for training, taking full advantage of the Transformer’s ability to capture long-distance dependencies in medical images and its symmetric encoder-decoder structure. TransUNet achieves superior segmentation accuracy in segmentation tasks compared to pure CNN or Transformer structured networks. However, the network does not effectively integrate multi-scale features extracted by the two different structural encoders, which also limits the further improvement of network performance. LeViT-UNet [14], on the other hand, is a fast inference network embedded in a U-Net-like structure - LeViT [28], which effectively balances the accuracy and efficiency of the Transformer Block.

Furthermore, LeViT employs skip connections to transmit multi-scale feature maps from its Transformer and convolutional blocks to the decoder, effectively reusing spatial information. However, this network only performs simple concatenation of the two feature maps in skip connections, resulting in some loss of detail information. Transfuse [29] introduces a novel parallel dual-branch structure to enhance feature extraction capability, avoiding local detail loss from redundant deep networks. While Transfuse proposes the BiFusion module to fuse multi-level features extracted by Transformer and CNN branches, it only conducts pooling and concatenation on features without separate treatment for global and local features. Transnorm [30] redesigns skip connections using a special attention mechanism to normalize channel and spatial information on each decoding path. However, Transnorm lacks the use of windowed self-attention mechanism, limiting global feature extraction and segmentation performance on medical images of different resolutions. HiFormer [31] utilizes Swin Transformer [26] and CNN as encoders, designing two multi-scale feature representations to capture both local and global features. It introduces the DLF module to fuse different features using cross-scale attention mechanism, ensuring feature consistency but potentially wasting some features as it only performs cross-scale attention calculation on two feature types.

3 Method

Our objective is to construct a dual-branch network that combines CNN and Transformer architectures to achieve accurate segmentation of high-resolution medical images. As illustrated in Fig. 1, the Encoder section of the model adopts a parallel structure, comprising a CNN branch with a g ⁿConv convolutional kernel and a Transformer branch utilizing a Variable Window Attention (VWA Transformer Block). These two branches focus on extracting local detailed features and global features of the input image, respectively. Additionally, at each encoding level, we incorporate a DFF feature fusion module to recombine and adaptively calibrate the features extracted from each branch of the dual-branch network. Subsequently, the processed feature information is passed to the decoding module. The Decoder stage of the model is responsible for recombining and recovering the feature information extracted during the downsampling process of the network, ultimately producing a clear segmentation result. Further elaboration on each component will be provided in the subsequent section.

Fig. 1

The overview of the proposed FDB-Net. FDB-Net includes CNN and Transformer branches for extracting local and global features respectively, as well as DFF Blocks for feature fusion.

3.1 Encoder

As depicted in Fig. 1, the model’s encoder part comprises two main branches and a series of cascaded feature fusion blocks (DFF), with the CNN and Transformer serving as the two branches. The input images are fed into these branches separately to fully exploit the advantages of both architectures. The CNN excels in capturing local correlations and accurately extracting local features from the images, while the Transformer is utilized to extract global features due to its unique mechanism for establishing global dependencies. Subsequently, the extracted features from both branches are fused at different levels through the feature fusion module DFF to ensure the effectiveness of the extracted features in the encoding process.

3.1.1 CNN module

The model proposed in this paper uses CNN as one of the feature extraction branches, and the CNN branch consists of four levels to extract the local spatial information of the image at different scales, inspired by g ⁿConv [27], this paper adopts a kind of CNN module (HorBlock) which contains recursive gated convolution (g ⁿConv) CNN module (HorBlock) for extracting deeper local spatial information, and the structure of this part is shown in Fig. 2.

Fig. 2

CNN Block structure diagram,the following diagram reveals the internal implementation of g ⁿConv block.

Compared with the traditional CNN convolutional operation, g ⁿConv takes into account the lack of spatial interaction of CNN, and fully exploits the deep spatial semantic information of the feature layer only through simple operations such as convolution and fully-connected layers, and different from the lack of spatial interactions in the ordinary convolutional operation and second-order spatial interactions in the convolutional module that includes the mechanism of self-attention, g ⁿConv is realized by gated convolution and recursion with high efficiency. g ⁿConv, on the other hand, realizes arbitrary-order spatial interaction through the efficient implementation of gated convolution and recursion.

First, the gated convolution module gConv is utilized to perform 1st order spatial interactions on the input features x ∈ R ^HW×C, where p ₀ and q ₀ represent two neighboring blocks of spatial features. f denotes the deep convolutional layer, then the output of the gated convolution y = gConv (x) can be written as: $\begin{matrix} [p_{0}^{HW \times C}, q_{0}^{HW \times C}] = φ_{in} (x) \in ℝ^{HW \times 2 C} \\ p_{1} = f (q_{0}) ⊙ p_{0} \in ℝ^{HW \times C}, y = φ_{out} (p_{1}) \in ℝ^{HW \times C} \end{matrix}$ (1) After the first-order spatial interactions are accomplished, higher-order interactions are achieved by recursively using the gConv module to perform the gated convolution recursively through the following steps in sequence:

$p_{k + 1} = f_{k} (q_{k}) ⊙ g_{k} (p_{k}) / α, k = 0, 1, \dots, n - 1$ (2)

Where f _k is a set of deep convolutional layers and g _k is used to put the sequential matching dimension:

$g_{k} = {\begin{matrix} Identity, & k = 0, \\ Linear (C_{k - 1}, C_{k}), & 1 \leq k \leq n - 1 \end{matrix}$ (3) In order for the model to take full advantage of the spatial information extracted by the g ⁿConv convolution kernel, we use the MLP layer after the g ⁿConv convolution operation to further refine the spatial g ⁿConv kernel extraction features.

3.1.2 Transformer module

In Vision Transformer, the model divides the input feature map into fixed-size patches and performs self-attention computation between the patches, but in real image processing tasks, the model tends to degrade its performance due to the overly high computational complexity as the images are usually of high resolution. Swin Transformer restricts the attentional computation to non-overlapping localized windows through the use of sliding windows while also allowing cross-window connections, which brings higher efficiency and lower computational complexity. overlapping localized windows, while also allowing cross-window connections, leading to higher efficiency with lower computational complexity.

Fig. 3

Transformer Block structure diagram, which consists of CPE position coding and Varied-Size Window Attention(VWA) block. The diagram below shows the internal details of the VWA block.

For an input x ∈ R ^H×W×C, Swin Transformer computes global attention through several window-based multi-head self-attention (W-MSA) and shifted window-based multi-head self-attention SW-MSA. In the W-MSA module, the self-attention will be applied to a local window of size m×m. Compared to the original MSA method of VIT to compute the attention, the W-MSA module reduces the computational complexity, however, given that there is no cross-window connectivity and the lack of information exchange between the non-overlapping windows, the model can only compute the self-attention to the feature maps inside the window, and it lacks the ability to sense the information outside of the window, which leads to its limited modeling capability. Therefore, in addition to the W-MSA module, Swin Transformer also uses the SW-MSA module to solve the problem of information exchange between different windows, and then recalculates the self-attention to expand the receptive field and improve the feature capture ability by window shifting after the calculation performed by the W-MSA module, the computational complexity of which is shown in Equation 4: $\begin{matrix} {\hat{z}}^{l} = W - MSA (LN (z^{l - 1})) + z^{l - 1} \\ z^{l} = MLP (LN ({\hat{z}}^{l})) + {\hat{z}}^{l} \\ {\hat{z}}^{l + 1} = SW - MSA (L N (z^{l})) + z^{l} \\ z^{l + 1} = MLP (L N ({\hat{z}}^{l + 1})) + {\hat{z}}^{l + 1} \end{matrix}$ (4) Swin Transformer makes better use of the principle of locality while reducing computational complexity, and is more powerful in feature extraction compared to VIT, but because of the fixed window size and position in the shift window mechanism, this limits the model’s ability to capture long-distance contextual information and to learn better feature representations of objects at different scales. Inspired by VSA [28], we propose to introduce it into Swin Transformer and use the VSAWindowAttention attention computation method to replace the MSA module and S-WSA module in Swin Transformer. For the input feature x, it is firstly divided into several windows X _w of predetermined size according to the method in Swin Transformer, and the corresponding query key value Q _w is obtained by linear operation on each window, the process is shown in Equation 5: $Q_{w} = Linear (X_{w})$ (5) After completing the sampling of each window, in order to realize variable window size and adjustable position, the VSA takes the default size window and position as a reference, and adopts a variable size window regression prediction module (VSR) to dynamically learn the size and positional offsets relative to the default window, and the VSR module consists of a convolutional layer with a convolution kernel size of the default window size and a step size of 1×1, a LeakyReLU activation function, and average pooling layer, and this process is shown in Equation 6: $S_{w}, O_{w} = Conv \circ LeakyReLU \circ AveragePool (X_{w})$ (6) The window generated in this way is referred to as the target window, where Q _w, Q _w ∈ R ^2×N denote the scaling and offset in the horizontal and vertical directions, respectively, with respect to the default window size, and N denotes the number of attention heads. After obtaining the scaling and offset, the feature maps K and V are then obtained from the input features X using linear operations as shown in Equation 7: $K, V = Reshape \circ Linear (X)$ (7)

Then the VSA module uniformly samples M features from each different-sized window on K, V respectively to obtain K _w,v, , which are fed into the MHSA module for self-attention computation together with Q _w. However, the relative position information of q, k, v is deviated due to the offset of the window position during feature sampling, so the VSA module uses conditional position embedding (CPE) [34] to solve the problem of relative spatial position deviation as shown in Equation 8: $X = Z^{l - 1} + CPE (Z^{l - 1})$ (8)

where Z ^l-1 are features from the previous Transformer Block, and the CPE is implemented by a deep convolutional layer with a convolutional kernel size equal to the default window size and a step size of one.

3.1.3 Deep fusion block (DFF)

The primary challenge in a two-branch network lies in effectively integrating the local and global features extracted from both the CNN and Transformer branches while maintaining feature consistency. A common yet ineffective approach is to simply sum the features extracted by the CNN and Transformer branches at each hierarchical level and feed them directly to the decoder for upsampling. However, this method fails to ensure feature consistency, resulting in subpar performance and underutilization of the advantages of two-branch networks. To address this issue, we have proposed a novel feature fusion (DFF) module. This module incorporates a dual-attention mechanism to adaptively recalibrate and integrate these two different types of features, as illustrated in Figure 4.

Fig. 4

Diagram of the internal structure of the DFF block. It contains the introduction of local and global features with the corresponding attention mechanism processing.

In medical image segmentation tasks, the features extracted by CNN networks usually contain more local details and localization information, and for the input features x ∈ R ^H×W×C at the upper level, the DFF module explicitly models the interdependence between the feature channels by splicing the input feature x in the channel dimensions to the local features provided by the CNN branches in the skip connection (Local Feature), and explicitly modeling the interdependence between feature channels through the Channel Attention mechanism, in which we compress the spatial dimension direction of the input feature map and use average pooling and maximum pooling operations to achieve effective aggregation of information between feature layers, which provides a stronger feature representation than a single pooling method capability [35]. After generating spatial context descriptors containing the most evenly pooled and maximally pooled features using the two pooling operations, the spatial context descriptors are fed into a shared MLP layer to generate the channel attention map and are applied to the generated spatial context descriptors, and the element summation and activation functions are applied to generate the output F _c. This part of the process is shown in Equation 9: $\begin{matrix} F_{c} & = σ (MLP (AvgPool (F)) + MLP (MaxPool (F))) \\ = σ (W_{1} (W_{0} (F_{a v g}^{c})) + W_{1} (W_{0} (F_{max}^{c}))) \end{matrix}$ (9)

where σ denotes the sigmoid activation function and W ₀ and W ₁ denote the two shared MLP weights. After completing the aggregation of local features, the DFF module will use the spatial attention mechanism to adaptively calibrate and fuse the global semantic information extracted from the Transformer branch to improve the model’s ability to represent global semantic information with long-range dependencies. The W_s ∈R ^{H
^′×W
^′×C′} output from each level of the Transformer branch with the output Fc from the previous step is computed using vector multiplication for attention calculation: $F_{out} = W_{s} \otimes F_{c}$ (10) By employing the dual-attention mechanism to adaptively calibrate and fuse the features extracted from the two branch networks, the model combines the strengths of both Transformer and CNN networks. It effectively leverages the long-range modeling capability of the Transformer while recovering lost local details and positional information through the fine-grained features of the CNN branch. This facilitates precise localization of segmentation boundaries and mitigates interference from noise, resulting in the model achieving precise segmentation of medical images.

3.2 Decoder

We follow the U-Net-like encoder-decoder architecture, relative to the encoder part, and design a corresponding decoder to combine the different size-resolution features obtained during the encoding process into a unified masked feature. For the input x ∈ R ^H×W×C, after down-sampling by the CNN and Transformer double branch network and calibrated by the DFF Block for feature fusion, the input features are sampled to $x \in R^{\frac{H}{32} \times \frac{W}{32} \times 8 C}$ size, our decoder receives four different sizes of resolution features output from the DFF module on the coding path, and for the different resolutions of the feature maps, the features are up-sampled by bilinear interpolation according to their corresponding scaling levels, and feature splicing is carried out after a convolutional block of 3x3 size, and through an MLP layer to aggregate the information conveyed in different Stages to obtain a richer feature representation. Finally, the output features are passed through a unified segmentation head to obtain accurate segmentation results.

Fig. 5

The decoder module designs.

4 Experiments

4.1 Datasets

Synapse. The synapse multi-organ segmentation dataset is a dataset containing 30 abdominal CT scans with a total of 3779 abdominal axial clinical CT images, each instance contains 13 organs to be segmented. Each CT scan in this dataset contains 85 ∼ 198 slices of 512 × 512 pixels, with a spatial resolution of ([0.54 ∼ 0.54] × [0.98 ∼ 0.98] × [2.5 ∼ 5.0]) mm ³. In this paper, we will refer to references [13, 36] for the pre-processing method to divide the samples into 18 training sets with 12 test sets. And the segmentation results of 8 abdominal organs (aorta, gallbladder, spleen, left kidney, right kidney, liver, pancreas, spleen, stomach) will be evaluated by using the average Dice-Similarity Coefficient (DSC) and the average Hausdorff Distance (HD) as the evaluation indexes.

ISIC2017. The ISIC2017 dataset [37] published by the International Skin Imaging Collaboration (ISIC) is a large-scale dermatological image dataset used for three main tasks, including lesion segmentation (considered in this work), skin feature detection, and skin disease classification.The ISIC 2017 dataset contains 2,000 images and the corresponding training set of labeled images. The size of each sample image is 576×767 pixels. In this paper, we refer to the method of literature [7, 38] to divide the dataset into a training set, a validation set and resize the images to 224×224.

ACDC. The ACDC dataset is a dataset containing 100 cases of cardiac MRI data for evaluating the performance of automated cardiac segmentation algorithms. Each case is examined with two different examination modalities, and the corresponding labels include left ventricle (LV), right ventricle (RV) and myocardium (MYO). In this paper, we will refer to the setting of literature [13] to partition the dataset into 70 training samples, 10 validation samples and 20 test samples.

4.2 Implementation details

For the experiments in this paper, all the code was done based on the Pytorch framework. The training and testing of the network models proposed in this paper are performed on an NVIDIA RTX 3090 GPU equipped with a 24GB video memory. Also, to ensure a fair comparison with other state-of-the-art methods, we uniformly divide the training images into 224x224 size.

Table 1
Training parameters details

Training config Details

Image size 224x224

Optimizer AdamW

Learning rate schedule CosineAnnealing

Base learning rate 1e-4

Min learning rate 1e-6

Weight decay 0.01

Batch size 12

Training epochs 400

Training config	Details
Image size	224x224
Optimizer	AdamW
Learning rate schedule	CosineAnnealing
Base learning rate	1e-4
Min learning rate	1e-6
Weight decay	0.01
Batch size	12
Training epochs	400

During the training process we used the AdamW optimizer to optimize our model and the CosineAnnealing algorithm to dynamically adjust the learning rate, where the baseline learning rate is 1e-4 and the minimum learning rate adjusted by the CosineAnnealing algorithm is 1e-6. More parameter settings are shown in Table 1.

4.3 Evaluation results

We utilize the Dice Similarity Coefficient (DSC) and the 95% Hausdorff Distance (HD95) to assess the segmentation performance of the model on the Synapse dataset. To validate the model’s performance in skin lesion segmentation, we employ several widely used metrics, including accuracy (ACC), sensitivity (SE), specificity (SP), and DSC. On the ACDC dataset, we primarily use DSC to evaluate the model’s performance. These metrics provide different information: accuracy measures the overall correctness of the prediction results, sensitivity and specificity evaluate the model’s ability to correctly predict positive and negative samples, respectively, and DSC quantifies the overlap between predicted segmentation results and ground truth labels. Additionally, the 95% Hausdorff Distance (HD95) provides information about the degree of matching between segmentation results and the edges of true samples, making the performance evaluation more comprehensive. By considering these metrics comprehensively, we can better assess the segmentation performance of the model on three different datasets.

4.3.1 Results of synapse multi-organ segmentation

On the Synapse Multi-Organ Segmentation dataset, we benchmarked the proposed model on two metrics, Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance (HD95), and the comparison of this method with the best previous (SOTA) method is shown in Table 2.

Table 2
Comparison results of the proposed method on the Synapse dataset. blue indicates the best result, and red displays the second-best

Methods DSC↑ HDâ†“ Aorta Gallbladder Kidney(L) Kidney(R) Liver Pancreas Spleen Stomach

DARR [36] 69.77 - 74.74 53.77 72.31 73.24 94.08 54.18 89.90 45.96

R50 U-Net [13] 74.68 36.87 87.74 63.66 80.60 78.19 93.74 56.90 85.87 74.16

U-Net [2] 76.85 39.70 89.07 69.72 77.77 68.60 93.43 53.98 86.67 75.58

R50 Att-UNet [39] 75.57 36.97 55.92 63.91 79.20 72.71 93.56 49.37 87.19 74.95

Att-UNet [40] 77.77 36.02 89.55 68.88 77.98 71.11 93.57 58.04 87.30 75.75

TransUnet [13] 77.48 31.69 87.23 63.13 81.87 77.02 94.08 55.86 85.08 75.62

Swin-Unet [27] 79.13 21.55 85.47 66.53 83.28 79.61 94.29 56.58 90.66 76.60

LeVit-Unet [14] 78.53 16.84 78.53 62.23 84.61 80.25 93.11 59.07 88.86 72.76

TransNorm [30] 78.40 30.25 86.23 65.10 82.18 78.63 94.22 55.34 89.50 76.01

HiFormer [31] 80.39 14.70 86.21 65.69 85.23 79.77 94.61 59.52 90.99 81.08

TransDeepLab [41] 80.16 21.25 86.04 69.16 84.08 79.88 93.53 61.19 89.00 78.40

DA-TransUNet [42] 79.80 23.48 86.54 65.27 81.70 80.45 94.57 61.62 88.53 79.73

FDB-Net(Ours) 81.58 19.01 89.14 73.09 85.89 80.95 94.72 64.11 91.01 73.70

Methods	DSC↑	HDâ†“	Aorta	Gallbladder	Kidney(L)	Kidney(R)	Liver	Pancreas	Spleen	Stomach
DARR [36]	69.77	-	74.74	53.77	72.31	73.24	94.08	54.18	89.90	45.96
R50 U-Net [13]	74.68	36.87	87.74	63.66	80.60	78.19	93.74	56.90	85.87	74.16
U-Net [2]	76.85	39.70	89.07	69.72	77.77	68.60	93.43	53.98	86.67	75.58
R50 Att-UNet [39]	75.57	36.97	55.92	63.91	79.20	72.71	93.56	49.37	87.19	74.95
Att-UNet [40]	77.77	36.02	89.55	68.88	77.98	71.11	93.57	58.04	87.30	75.75
TransUnet [13]	77.48	31.69	87.23	63.13	81.87	77.02	94.08	55.86	85.08	75.62
Swin-Unet [27]	79.13	21.55	85.47	66.53	83.28	79.61	94.29	56.58	90.66	76.60
LeVit-Unet [14]	78.53	16.84	78.53	62.23	84.61	80.25	93.11	59.07	88.86	72.76
TransNorm [30]	78.40	30.25	86.23	65.10	82.18	78.63	94.22	55.34	89.50	76.01
HiFormer [31]	80.39	14.70	86.21	65.69	85.23	79.77	94.61	59.52	90.99	81.08
TransDeepLab [41]	80.16	21.25	86.04	69.16	84.08	79.88	93.53	61.19	89.00	78.40
DA-TransUNet [42]	79.80	23.48	86.54	65.27	81.70	80.45	94.57	61.62	88.53	79.73
FDB-Net(Ours)	81.58	19.01	89.14	73.09	85.89	80.95	94.72	64.11	91.01	73.70

The results presented in Table 2 demonstrate that our proposed method achieves optimal segmentation accuracy for multiple organs in the abdomen. Compared to the previous state-of-the-art (SOTA) method, our approach exhibits a significant improvement of 1.19% in the key metric, Dice coefficient. Particularly, we observe substantial absolute Dice coefficient increases of +3.93%, +0.66%, +0.50%, +0.11%, +2.49%, and +0.02% in the segmentation of the Gallbladder, left kidney, right kidney, Liver, Pancreas, and Spleen, respectively, when compared to the suboptimal method.

This shows that our model has a certain degree of superiority in capturing segmentation target features and locating target segmentation boundaries, and is significantly ahead of previous CNN-based or Transformer-based methods in several metrics. This is due to the fact that the model effectively utilises global and local features provided by a double branch network, which significantly improves the feature representation of the network compared to a traditional single branch network. capability, which is particularly important for segmentation tasks. Compared to recent hybrid CNN and Transformer-based networks, our approach still has a considerable lead in Dice coefficients and segmentation accuracies of multiple major organs, which demonstrates that compared to the simple splicing of features extracted from the CNN and Transformer blocks in the comparison model, our proposed DFF feature fusion block is quite Advancement. Our model adopts a parallel network structure in the encoding path, and in order to prevent the interference caused by the direct interaction of feature information between different branches, instead of letting the feature information of two different branches interact directly, we set up the DFF block to fuse the two kinds of features at each down-sampling level, which achieves the adaptive fusion of the features through the double-attention mechanism, suppresses the noise interference of the complex background on the segmentation task and improves the robustness of the model and accurately segment fine and complex organ structures.

In addition, in terms of visual evaluation, we provide Figure 6. In Figure 6, it can be seen that our method is more consistent with Ground Truth for the boundary segmentation of multiple abdominal organs, and is clearer in the boundary segmentation of gallbladder and left and right kidney than the previous CNN or transformer-based models. This is because our model provides both fine local position information and global image features, which is particularly important in organ segmentation of medical images.

Fig. 6

Our method is visually compared with the segmentation results of other advanced methods on the Synapse dataset.

4.3.2 Results of skin lesion segmentation

On the ISIC2017 dataset, we tested the proposed model on four indicators: Dice Similarity Coefficient (DSC), Specificity (SP), Sensitivity (SE) and Accuracy (ACC). The comparison between this method and the previous best (SOTA) method is shown in Table 3.

Table 3
Comparison results of the proposed method on the ISIC2017 dataset. blue indicates the best result, and red displays the second-best

Methods DSC↑ SE↑ SP↑ ACC↑

U-Net [2] 0.8159 0.8172 0.9680 0.9164

Att-UNet [40] 0.8082 0.7998 0.9776 0.9145

TransUnet [13] 0.8123 0.8263 0.9577 0.9207

DAGAN [43] 0.8425 0.8363 0.9716 0.9304

MCGU-Net [44] 0.8927 0.8502 0.9855 0.9570

MedT [45] 0.8037 0.8064 0.9546 0.9090

FAT-Net [46] 0.8500 0.8392 0.9725 0.9326

TransNorm [30] 0.8933 0.8532 0.9859 0.9582

FDB-Net(Ours) 0.9198 0.9231 0.9715 0.9601

Methods	DSC↑	SE↑	SP↑	ACC↑
U-Net [2]	0.8159	0.8172	0.9680	0.9164
Att-UNet [40]	0.8082	0.7998	0.9776	0.9145
TransUnet [13]	0.8123	0.8263	0.9577	0.9207
DAGAN [43]	0.8425	0.8363	0.9716	0.9304
MCGU-Net [44]	0.8927	0.8502	0.9855	0.9570
MedT [45]	0.8037	0.8064	0.9546	0.9090
FAT-Net [46]	0.8500	0.8392	0.9725	0.9326
TransNorm [30]	0.8933	0.8532	0.9859	0.9582
FDB-Net(Ours)	0.9198	0.9231	0.9715	0.9601

As can be seen from the results in Table 3, our proposed double-branch network model is almost overall superior to the existing SOTA methods in the skin disease segmentation task. In the key DICE coefficient or F1-Score, our method is 2.55% higher than the previous SOTA method, and our method is also superior to the existing methods in the sensitivity (SE) and accuracy (ACC) indicators.

In Figure 7, a partial segmentation result of skin lesions of dermatosis provided by our model is shown, which shows that the model accurately locates the boundary part of the lesion skin and provides smooth and accurate segmentation results. Compared with poorly performing pure CNN network models such as U-net and hybrid networks such as TransUnet, our method shows better generalization, thanks to the dual branching for grabbing features of different levels and the dual attention mechanism for fusing features in skip connections.

Fig. 7

Our method is shown in the visualized segmentation results on the ISIC2017 dataset.

4.3.3 Results of ACDC

To further explore the generality of our method on different datasets, we also conducted tests on the ACDC dataset. The comparison of this method with the previous best (SOTA) methods is shown in Table 4.

Table 4
Comparison results of the proposed method on the ACDC dataset. blue indicates the best result, and red displays the second-best

Methods DSC↑ RV↑ Myo↑ LV↑

R50 U-Net [13] 87.60 84.62 84.52 93.68

Att-UNet [40] 86.90 83.27 84.33 93.53

TransUnet [13] 89.71 86.67 87.27 95.18

Swin-Unet [27] 88.07 85.77 84.42 94.03

MT-Unet [47] 90.43 86.64 89.04 95.62

FDB-Net(Ours) 90.86 88.09 88.86 95.65

Methods	DSC↑	RV↑	Myo↑	LV↑
R50 U-Net [13]	87.60	84.62	84.52	93.68
Att-UNet [40]	86.90	83.27	84.33	93.53
TransUnet [13]	89.71	86.67	87.27	95.18
Swin-Unet [27]	88.07	85.77	84.42	94.03
MT-Unet [47]	90.43	86.64	89.04	95.62
FDB-Net(Ours)	90.86	88.09	88.86	95.65

It can be seen that our method achieves the best or the second best results in the three main organ segmentation indicators of left ventricle (LV), right ventricle (RV) and myocardium (MYO). In the average DSC index, the segmentation accuracy of our method is 0.43% ahead of the current best model MT-Unet. In the right ventricle (RV) segmentation accuracy, we are significantly ahead of the second best method, which indicates that our method has advantages in complex organ localization and segmentation.

5 Ablation study

In order to further explore the effectiveness of the proposed method, in this section, we design ablation experiments to verify the influence of different design choices on the model segmentation effect.

5.1 Impact of DFF block

In this section, we investigate whether the proposed DFF fusion module has a positive effect on the target medical image segmentation task, and whether it can help the model output more accurate segmentation results by using the double attention mechanism at each level of the decoding path to fuse the high-level and low-level features captured by the dual-branch network. We conducted verification on the Synapse Multi-Organ Segmentation and ISIC2017 datasets. For the model without the DFF Block, we adopted the U-Net skip connection scheme, which involves directly concatenating the features extracted by the dual-branch network at each level of the decoding path.The experimental results presented in Tables 5 and 6 clearly demonstrate the effectiveness of using the DFF block to fuse the dual-branch network at the encoding path. The network with the DFF block outperformed the one without it by a large margin across multiple metrics, indicating that the performance of the dual-branch model is dependent on a specific feature fusion approach. Our proposed DFF block is capable of fusing different-scale information hidden in the target image, which leads to a significant improvement in network performance.

Table 5
Influence of DFF block on segmentation results on Synapse Multi-Organ Segmentation dataset

Methods Encoder with DFF DSC↑ HDâ†“

FDB-Net 78.75 25.33

FDB-Net ✓ 81.58 19.03

Methods	Encoder with DFF	DSC↑	HDâ†“
FDB-Net		78.75	25.33
FDB-Net	✓	81.58	19.03

Table 6

Influence of DFF block on segmentation results on ISIC2017 dataset

Methods	Encoder with DFF	DSC↑	SE↑	SP↑	ACC↑
FDB-Net		0.8831	0.8617	0.9791	0.9432
FDB-Net	✓	0.9198	0.9231	0.9715	0.9601

5.2 Influence of different CNN backbone

To investigate the effectiveness of g ⁿConv and the impact of different CNN backbone on model performance, we conducted experiments on the Synapse Multi-Organ Segmentation and ISIC2017 datasets. The results presented in Tables 7 and 8 demonstrate that the CNN block constructed by g ⁿConv achieved the best performance across different CNN backbones. Specifically, on the Synapse Multi-Organ Segmentation dataset, the DSC and HD metrics of the g ⁿConv-based model outperformed the second-best CNN backbone by 0.80% and 0.45%, respectively, while on the ISIC2017 dataset, the g ⁿConv-based model achieved a significant improvement in the crucial DSC metric, outscoring the second-best CNN backbone by 1.44%. These results indicate the potential of the g ⁿConv block in exploring high-order latent features in images.

Table 7
Segmentation results on the Synapse Multi-Organ Segmentation dataset using different CNN backbone

Methods DSC↑ HDâ†“

FDB-Net + ResNet18 79.49 19.92

FDB-Net + ResNet50 80.78 19.48

FDB-Net + DenseNet121 80.44 20.17

FDB-Net + DenseNet169 80.12 21.31

FDB-Net + g ⁿConvBlock 81.58 19.03

Methods	DSC↑	HDâ†“
FDB-Net + ResNet18	79.49	19.92
FDB-Net + ResNet50	80.78	19.48
FDB-Net + DenseNet121	80.44	20.17
FDB-Net + DenseNet169	80.12	21.31
FDB-Net + g ⁿConvBlock	81.58	19.03

Table 8

Segmentation results on the ISIC2017 dataset using different CNN backbone

Methods	DSC↑	SE↑	SP↑	ACC↑
FDB-Net + ResNet18	0.8895	0.8947	0.9690	0.9409
FDB-Net + ResNet50	0.8977	0.9043	0.9689	0.9536
FDB-Net + DenseNet121	0.9042	0.9101	0.9769	0.9522
FDB-Net + DenseNet169	0.9054	0.9128	0.9721	0.9540
FDB-Net + g ⁿConvBlock	0.9198	0.9231	0.9715	0.9601

5.3 Influence of different transformer backbone

In this section, we will explore the impact of using different Transformer backbone on the results of the model in medical image segmentation tasks. The comparison between the Transformer trunk without the variable sliding window Swin Transformer and the Transformer trunk containing VWA Block in this paper is shown in Table 9 and 10. This indicates that in complex medical image segmentation tasks, the improvement of the global modeling ability of the model greatly promotes the performance of segmentation accuracy, and the Transformer branch with stronger global semantic information feature extraction ability can effectively improve the final segmentation results.

Table 9
Segmentation results on the Synapse Multi-Organ

Segmentation dataset using different Transformers backbone

Methods DSC↑ HDâ†“

FDB-Net + Vision Transformer 78.99 22.45

FDB-Net + Swin Transformer 79.83 20.94

FDB-Net + VWA Transformer 81.58 19.03

Methods	DSC↑	HDâ†“
FDB-Net + Vision Transformer	78.99	22.45
FDB-Net + Swin Transformer	79.83	20.94
FDB-Net + VWA Transformer	81.58	19.03

Table 10

Segmentation results on the ISIC2017 dataset using different transformers backbone

Methods	DSC↑	SE↑	SP↑	ACC↑
FDB-Net + Vision Transformer	0.9027	0.9132	0.9690	0.9426
FDB-Net + Swin Transformer	0.9123	0.9203	0.9754	0.9570
FDB-Net + VWA Transformer	0.9198	0.9231	0.9715	0.9601

6 Conclusions

In this paper, we propose a double branch hybrid network model for the task of segmenting medical images, and its excellent performance on three datasets-Synapse Multi-Organ, Skin Lesion, and ACDC has demonstrated the effectiveness of our proposed method. By combining the different focuses of CNN and Transformer in image feature extraction and using them effectively through a feature fusion module, our approach gives full play to the global and local features extracted by the model at different levels, improves the model’s generalisation performance and robustness, and overcomes the problems of noise interference and low segmentation accuracy in medical image segmentation to a certain extent. We hope that our work can further explore the possibility of combining CNN and Transformer, fully utilizing the potential of features at different scales, and provide some reference for other related work.

Ethical approval

Our paper satisfies and fulfills the compliance with ethical standards. The research does not involve human participants and/or animals.

Data availability

The data that support the findings of this study are openly available in [Synapse] at [https://www.synapse.org/#!Synapse:syn3193805/files], [ISIC2017] at [https://challenge.isic-archive.com/data/#2017], [ACDC] at [https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html].

Conflict of interest

The authors declare that there is no conflict of interest.

Funding

National Natural Science Foundation of China (No. 62266011); the Science and Technology Foundation of Guizhou Province(ZK[2022] 119).

Author contributions

Jiang Z. and Wu Y. were responsible for the writing of the main body and experimental sections of this paper. Huang L. created all the figures in this paper. Gu M. organized the experimental results for this paper.

Footnotes

Acknowledgments

This work was financially supported in part by National Natural Science Foundation of China (No. 62266011), the Science and Technology Foundation of Guizhou Province (ZK[2022] 119).

References

Liao

Chen

Wang

Chen

D.Z.

Gao

, D-former: A ushaped dilated transformer for 3d medical image segmentation, Neural Computing and Applications (2022), 1–14.

Ronneberger

Fischer

Brox

U-net: Convolutional networks for biomedical image segmentation, In:Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015:18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 pp. 234–241 (2015). Springer.

Xiao

Lian

Luo

, Weighted res-unet for high-quality retina vessel segmentation, In: 2018 9th International Conference on Information Technology in Medicine and Education (ITME) pp. 327–331 (2018). IEEE.

Guan

Khan

A.A.

Sikdar

Chitnis

P.V.

, Fully dense unet for 2-d sparse photoacoustic tomography artifact removal, IEEE Journal of Biomedical and Health Informatics 24(2) (2019), 568–576.

Zhou

Huang

Dong

Xia

Wang

, D-unet: a dimension-fusion u shape network for chronic stroke lesion segmentation, IEEE/ACM Transactions on Computational Biology and Bioinformatics 18(3) (2019), 940–950.

Huang

Lin

Tong

Zhang

Iwamoto

Han

Chen

Y.-W.

, Unet 3+: A full-scale connected unet for medical image segmentation, In:ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1055–1059 (2020). IEEE.

Azad

Fayjie

A.R.

Kauffmann

Ben Ayed

Pedersoli

Dolz

, On the texture bias for few-shot cnn segmentation, In:Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2674–2683 (2021).

Liu

Shen

, Medical image analysis based on transformer: A review, arXiv preprint arXiv:2208.06643 (2022).

Egonmwan

Chali

, Transformer and seq2seq model for paraphrase generation, In:Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 249–255 (2019).

10.

Chen

L.-W.

Rudnicky

, Fine-grained style control in transformer-based textto-speech synthesis, In:ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7907–7911 (2022). IEEE.

11.

Shi

Wang

Yeh

C.-F.

Chan

Zhang

Seltzer

, Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition, In:ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6783–6787 (2021). IEEE.

12.

Dosovitskiy

Beyer

Kolesnikov

Weissenborn

Zhai

Unterthiner

Dehghani

Minderer

Heigold

Gelly

et al., An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929 (2020).

13.

Chen

Luo

Adeli

Wang

Yuille

A.L.

Zhou

, Transunet: Transformers make strong encoders for medical image segmentation, arXiv preprint arXiv:2102.04306, (2021).

14.

Zhang

, Levit-unet: Make faster encoders with transformer for medical image segmentation, arXiv preprint arXiv:2107.08623, (2021).

15.

Long

Shelhamer

Darrell

, Fully convolutional networks for semantic segmentation, In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015).

16.

Diakogiannis

F.I.

Waldner

Caccetta

, Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data, ISPRS Journal of Photogrammetry and Remote Sensing 162 (2020), 94–114.

17.

Jha

Smedsrud

P.H.

Riegler

M.A.

Johansen

De Lange

Halvorsen

Johansen

H.D.

, Resunet++: An advanced architecture for medical image segmentation, In:2019 IEEE International Symposium on Multimedia (ISM), pp. 225–2255 (2019). IEEE.

18.

Cao

Liu

Peng

, Denseunet: densely connected unet for electron microscopy image segmentation, IET Image Processing 14(12) (2020), 2682–2689.

19.

Zhou

Rahman Siddiquee

M.M.

Tajbakhsh

Liang

, Unet++: A nested u-net architecture for medical image segmentation, In:Deep Learning in Medical image analysis and Multimodal Learning for Clinical Decision Support: 4th InternationalWorkshop, DLMIA 2018, and 8th InternationalWorkshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4, pp. 3–11 (2018). Springer.

20.

Koltun

, Multi-scale context aggregation by dilated convolutions,, arXiv preprint arXiv:1511.07122, (2015).

21.

Zhang

Ren

Sun

, Spatial pyramid pooling in deep convolutional networks for visual recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (2015), 1904–1916.

22.

Chen

L.-C.

Zhu

Papandreou

Schroff

Adam

, Encoder-decoder with atrous separable convolution for semantic image segmentation, In:Proceedings of the European Conference on Computer Vision (ECCV), pp. 801–818 (2018).

23.

J.L.

Kiros

J.R.

Hinton

G.E.

, Layer normalization,, arXiv preprint arXiv:1607.06450, (2016).

24.

Touvron

Cord

Douze

Massa

Sablayrolles

Jégou

, Training data-efficient image transformers & distillation through attention, In:International Conference on Machine Learning, pp. 10347–10357 (2021). PMLR.

25.

Dong

Bao

Chen

Zhang

Yuan

Chen

Guo

, Cswin transformer: A general vision transformer backbone with cross-shaped windows, In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12124–12134 (2022).

26.

Liu

Lin

Cao

Wei

Zhang

Lin

Guo

, Swin transformer: Hierarchical vision transformer using shifted windows, In:Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021).

27.

Cao

Wang

Chen

Jiang

Zhang

Tian

Wang

, Swin-unet: Unet-like pure transformer for medical image segmentation, In:Computer Vision– ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part III, pp. 205–218 (2023). Springer.

28.

Graham

El-Nouby

Touvron

Stock

Joulin

Jégou

Douze

, Levit: a vision transformer in convnet’s clothing for faster inference, In:Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12259–12269 (2021).

29.

Zhang

Liu

, Transfuse: Fusing transformers and cnns for medical image segmentation, In:Medical Image Computing and Computer Assisted Intervention–MICCAI 2021:24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24, pp. 14–24 (2021). Springer.

30.

Azad

Al-Antary

M.T.

Heidari

Merhof

, Transnorm: Transformer provides a strong spatial normalization mechanism for a deep segmentation model, IEEE Access 10 (2022), 108205–108215.

31.

Heidari

Kazerouni

Soltany

Azad

Aghdam

E.K.

Cohen-Adad

Merhof

, Hiformer: Hierarchical multi-scale representations using transformers for medical image segmentation, In:Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6202–6212 (2023).

32.

Rao

Zhao

Tang

Zhou

Lim

S.-N.

, Hornet: Efficient high-order spatial interactions with recursive gated convolutions, arXiv preprint arXiv:2207.14284, (2022).

33.

Zhang

Tao

, Vsa: learning varied-size window attention in vision transformers, In:Computer Vision–ECCV 2022:17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV, pp. 466–483 (2022). Springer.

34.

Chu

Tian

Zhang

Wang

Wei

Xia

Shen

, Conditional positional encodings for vision transformers, arXiv preprint arXiv:2102.10882 (2021).

35.

Woo

Park

Lee

J.-Y.

Kweon

I.S.

, Cbam: Convolutional block attention module, In:Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018).

36.

Wang

Zhou

Shen

Fishman

Yuille

, Domain adaptive relational reasoning for 3d multi-organ segmentation, In:Medical Image Computing and Computer Assisted Intervention–MICCAI 2020:23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part I 23, pp. 656–666 (2020). Springer.

37.

Codella

N.C.

Gutman

Celebi

M.E.

Helba

Marchetti

M.A.

Dusza

S.W.

Kalloo

Liopyris

Mishra

Kittler

et al., Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic), In:2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 168–172 (2018). IEEE.

38.

Alom

M.Z.

Hasan

Yakopcic

Taha

T.M.

Asari

V.K.

, Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation, arXiv preprint arXiv:1802.06955, (2018).

39.

Schlemper

Oktay

Schaap

Heinrich

Kainz

Glocker

Rueckert

, Attention gated networks: Learning to leverage salient regions in medical images, Medical Image Analysis 53 (2019), 197–207.

40.

Oktay

Schlemper

Folgoc

L.L.

Lee

Heinrich

Misawa

Mori

McDonagh

Hammerla

N.Y.

Kainz

et al., Attention u-net: Learning where to look for the pancreas, arXiv preprint arXiv:1804.03999 (2018).

41.

Azad

Heidari

Shariatnia

Aghdam

E.K.

Karimijafarbigloo

Adeli

Merhof

, Transdeeplab: Convolution-free transformer-based deeplab v3+ for medical image segmentation, In:Predictive Intelligence in Medicine: 5th International Workshop, PRIME 2022, Held in Conjunction with MICCAI 2022, Singapore, September 22, 2022, Proceedings, pp. 91–102 (2022). Springer.

42.

Sun

Pan

Kong

Racharak

Nguyen

L.-M.

, Datransunet: Integrating spatial and channel dual attention with transformer u-net for medical image segmentation, arXiv preprint arXiv:2310.12570, (2023).

43.

Lei

Xia

Jiang

Qin

Chen

Wang

, Skin lesion segmentation via generative adversarial networks with dual discriminators,, Medical Image Analysis 64 (2020), 101716.

44.

Asadi-Aghbolaghi

Azad

Fathy

Escalera

, Multi-level context gating of embedded collective knowledge for medical image segmentation, arXiv preprint arXiv:2003.05056, (2020).

45.

Valanarasu

J.M.J.

Oza

Hacihaliloglu

Patel

V.M.

, Medical transformer: Gated axial-attention for medical image segmentation, InMedical Image Computing and Computer Assisted Intervention–MICCAI 2021:24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24, pp. 36–46 (2021). Springer.

46.

Chen

Wang

Lei

Wen

, Fat-net: Feature adaptive transformers for automated skin lesion segmentation, Medical Image Analysis 76 (2022), 102327.

47.

Wang

Xie

Lin

Iwamoto

Han

X.-H.

Chen

Y.-W.

Tong

, Mixed transformer u-net for medical image segmentation, In:ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2390–2394 (2022). IEEE.

FDB-Net: Fusion double branch network combining CNN and transformer for medical image segmentation

Abstract

BACKGROUND:

METHODS:

CONCLUSION:

Keywords

1 Introduction

2.1 CNN-based segmentation networks

2.2 Vision transformers

2.3 Hybrid CNN-transformer architecture

3 Method

3.1.1 CNN module

4.1 Datasets

4.2 Implementation details

Table 1 Training parameters details Training config Details Image size 224x224 Optimizer AdamW Learning rate schedule CosineAnnealing Base learning rate 1e-4 Min learning rate 1e-6 Weight decay 0.01 Batch size 12 Training epochs 400

4.3.1 Results of synapse multi-organ segmentation

5.1 Impact of DFF block

Table 5 Influence of DFF block on segmentation results on Synapse Multi-Organ Segmentation dataset Methods Encoder with DFF DSC↑ HDâ†“ FDB-Net 78.75 25.33 FDB-Net ✓ 81.58 19.03

Table 7 Segmentation results on the Synapse Multi-Organ Segmentation dataset using different CNN backbone Methods DSC↑ HDâ†“ FDB-Net + ResNet18 79.49 19.92 FDB-Net + ResNet50 80.78 19.48 FDB-Net + DenseNet121 80.44 20.17 FDB-Net + DenseNet169 80.12 21.31 FDB-Net + g n ConvBlock 81.58 19.03

Table 9 Segmentation results on the Synapse Multi-Organ Segmentation dataset using different Transformers backbone Methods DSC↑ HDâ†“ FDB-Net + Vision Transformer 78.99 22.45 FDB-Net + Swin Transformer 79.83 20.94 FDB-Net + VWA Transformer 81.58 19.03

Ethical approval

Data availability

Conflict of interest

Funding

Author contributions

Footnotes

Acknowledgments

References

Table 1
Training parameters details

Training config Details

Image size 224x224

Optimizer AdamW

Learning rate schedule CosineAnnealing

Base learning rate 1e-4

Min learning rate 1e-6

Weight decay 0.01

Batch size 12

Training epochs 400

Table 5
Influence of DFF block on segmentation results on Synapse Multi-Organ Segmentation dataset

Methods Encoder with DFF DSC↑ HDâ†“

FDB-Net 78.75 25.33

FDB-Net ✓ 81.58 19.03

Table 9
Segmentation results on the Synapse Multi-Organ

Segmentation dataset using different Transformers backbone

Methods DSC↑ HDâ†“

FDB-Net + Vision Transformer 78.99 22.45

FDB-Net + Swin Transformer 79.83 20.94

FDB-Net + VWA Transformer 81.58 19.03