A multi-stage training and deep supervision based segmentation approach for 3D abdominal multi-organ segmentation

Abstract

Accurate X-ray Computed tomography (CT) image segmentation of the abdominal organs is fundamental for diagnosing abdominal diseases, planning cancer treatment, and formulating radiotherapy strategies. However, the existing deep learning based models for three-dimensional (3D) CT image abdominal multi-organ segmentation face challenges, including complex organ distribution, scarcity of labeled data, and diversity of organ structures, leading to difficulties in model training and convergence and low segmentation accuracy. To address these issues, a novel multi-stage training and a deep supervision model based segmentation approach is proposed. It primary integrates multi-stage training, pseudo- labeling technique, and a developed deep supervision model with attention mechanism (DLAU-Net), specifically designed for 3D abdominal multi-organ segmentation. The DLAU-Net enhances segmentation performance and model adaptability through an improved network architecture. The multi-stage training strategy accelerates model convergence and enhances generalizability, effectively addressing the diversity of abdominal organ structures. The introduction of pseudo-labeling training alleviates the bottleneck of labeled data scarcity and further improves the model's generalization performance and training efficiency. Experiments were conducted on a large dataset provided by the FLARE 2023 Challenge. Comprehensive ablation studies and comparative experiments were conducted to validate the effectiveness of the proposed method. Our method achieves an average organ accuracy (AVG) of 90.5% and a Dice Similarity Coefficient (DSC) of 89.05% and exhibits exceptional performance in terms of training speed and handling data diversity, particularly in the segmentation tasks of critical abdominal organs such as the liver, spleen, and kidneys, significantly outperforming existing comparative methods.

Keywords

abdominal multi-organ segmentation deep supervision learning multi-stage training pseudo-labeling

Introduction

The distribution of human abdominal organs is complex, with the main organs and structures within the abdominal cavity, including liver, gallbladder, stomach, spleen, inferior vena cava, and aorta. The normal liver contour is smooth, and its shape varies with respect to the cross-sectional position. The porta hepatis region typically contains a significant amount of adipose tissue, presenting an irregular shape or resembling a low- density polygon. This area includes the hepatic artery, portal vein, and bile duct structures. The portal vein, located posteriorly, is relatively large, while the hepatic artery is situated anterior to it, and the bile duct (mainly the common hepatic duct) is located anteriorly and laterally. Post-contrast, the portal vein and aorta are easily identified, appearing as round high-density shadows positioned anterior to the inferior vena cava. Abdominal X-ray CT images often exhibit artifacts and noise due to the limitations of imaging equipment and organ peristalsis. Additionally, some organ tissues may appear partially blurred, and the edges of lesions can be unclear, presenting considerable challenges for organ segmentation. It is fundamental to accurately segment abdominal organs from CT images for diagnosing abdominal diseases, planning cancer treatment, and formulating radiotherapy strategies.

Early multi-organ segmentation tasks relied on the region-growing segmentation algorithms, which are semi-automated methods and have achieved promising performance based on the correct positioning of seed points. With the exceptional performance of convolutional neural networks (CNNs) in image processing and segmentation, they have gradually been applied to medical image segmentation. A classic example is the U-Net architecture proposed by Ronneberger et al. in 2015.¹ It incorporates skip connections between upsampling and downsampling layers, enabling the direct transfer of features extracted by downsampling layers to upsampling layers, thereby enhancing pixel-level localization and segmentation performance. However, the U-Net architecture struggles to capture inter-slice correlations, leading to the risk of under-segmentation. To address these limitations, research into the U-Net structure has deepened, resulting in the development of various encoder-decoder architectures, such as U-Net++ and U-Net 3 + . These new network structures continue to refine the U-Net framework, yet with the advancements in deep learning, alternative high-accuracy network models like Vision Transformers (ViT) have emerged. To achieve comparable accuracy with these models, it is necessary to introduce new modules and innovations. A 3D deeply supervised network (3D- DSN) was developed to address liver segmentation issues, demonstrating commendable accuracy. The development of segmentation tasks is also evolving, transitioning from general segmentation research to specialized model studies focused on specific areas such as the brain, chest, abdomen, and pelvis. In recent years, as the pace of development in multi-organ segmentation for brain and chest regions has gradually slowed, an increasing number of researchers have begun to concentrate on abdominal multi-organ tasks. Despite the emergence of excellent new network structures such as 3D-DSN, studies by Geert et al.² indicate that segmentation tasks typically feature prolonged convergence times and excessive iteration counts. Moreover, to ensure accurate convergence of the models, a substantial amount of labeled data is required to enhance model performance. However, in practical applications, labeled datasets are often severely limited.³ Based on the aforementioned considerations, segmentation tasks commonly face two issues: first, the convergence of the model requires numerous iterations and is time-consuming; second, the labeled datasets are often insufficient to achieve optimal convergence results in practical tasks.

Elnaz et al. noted in 2015⁴ that multi-stage training can produce regularization effects and enhance generalization capabilities, thereby accelerating model convergence. In a subsequent study by Yuan et al. in 2018,⁵ an explanation and justification were provided for the question “why does staged training accelerate the reduction of test error on SGD?” In recent years, increasing research has focused on integrating multi-stage training into multi-organ segmentation tasks. The results of these studies consistently demonstrated that multi-stage training significantly accelerates model convergence. The method of training neural networks using pseudo-labels in a semi-supervised manner (self-training with pseudo-labels) was first proposed by Lee et al.⁶ This approach leverages high-confidence pseudo-labeling samples to enhance the model's fitting ability and improve robustness, particularly in scenarios with limited source domain datasets. Subsequent advancements include entropy minimization,⁷ proxy labeling,⁶ noise contrastive estimation,⁸ meta-pseudo-labeling,⁹ multi-view training,^10,11 collaborative training,¹² and tri-training.¹³ These methods represent successful attempts in the field of pseudo-labeling generation. Recently, the medical image segmentation field has faced challenges due to variations in the same organ's condition across different individuals and ages. This variability poses difficulties regarding the sufficiency and diversity of labeled data, while pseudo-labeling training offers an effective solution to address these challenges.

By reviewing the history and current status of abdominal organ segmentation, this paper traces the evolution from early region-growing algorithms to U-Net and its variants, and onto the latest deep learning models such as 3D-DSN and Vision Transformers. It also highlights the developmental trends in this field and the existing challenges, including long training times, reliance on extensive labeled data, and the diversity of organ structures. As a result, We propose an innovative method that combines multi-stage training with pseudo-labeling techniques and a novel network model (DLAU-Net), specifically designed for 3D multi-organ segmentation in the abdomen. We employ multi-stage training to conduct model training at different granularities, effectively addressing the limitations of the traditional models in feature learning. Additionally, through pseudo- labeling techniques, the model can achieve high-accuracy segmentation even with a limited number of labeled data. The key contributions of this study are summarized as the follows:

The DLAU-Net network model is proposed to significantly enhance the performance of downstream segmentation tasks and advance the accuracy, efficiency, and adaptability of the algorithm.

The multi-stage training techniques are applied to accelerates model convergence, improve generalization capabilities, and aid in addressing segmentation challenges posed by the diversity of organ structures.

The pseudo-labeling techniques are used to effectively resolve the issue of scarce labeled data, enhance the model's generalization ability, and improve training efficiency.

Related work

Abdominal multi-organ segmentation tasks typically focus on few larger organs, such as the esophagus and aorta, as well as the kidneys, liver, spleen, left kidney, right kidney, and pancreas.^14,15 Models that perform exceptionally well in these tasks are often based on U-Net and its variants. Although the U-Net is classic, it is difficult to effectively capture inter-slice correlations and is prone to under-segmentation. To address these limitations, various U-Net variants have been developed, including U-Net++,¹⁶ V-Net,¹⁷ nnU-Net,¹⁸ ResU-Net,¹⁹ and DenseU-Net.²⁰ In the work “Deep Layer Aggregation”,²¹ Yu et al. proposed modifications to U-Net's long skip connections and the insertion of new convolutional layers and short skip connections within the encoder and decoder. These modifications enable the model to learn features at different depths, thus avoiding the manual process of determining the optimal model depth. The U-Net++ combines both short and long skip connections to improve feature acquisition and mitigate information loss. The nnU-Net adapts dynamically to various datasets, automatically tuning hyperparameters, thereby enhancing model generalizability and reducing training time and human efforts. The V-Net, which employs a 3D convolutional network for prostate volume segmentation, also introduced the Dice loss function to address the imbalance between foreground and background voxels.

However, with the increasing clinical demands and the growing complexity of segmentation tasks, it proves insufficient to solely rely on innovations in model architecture to meet the requirements for high-precision segmentation, particularly in the context of multi-organ segmentation in the abdomen with issues of data scarcity and annotation costs. Consequently, a series of efficient training strategies have been introduced. Deep supervision, also known as intermediate supervision, involves adding an auxiliary classifier as a branch at certain intermediate hidden layers of a deep neural network to supervise the main network. This technique,²² introduced by Lee et al. in 2014, helps to ensure thorough training of the model and addresses issues related to vanishing gradients and slow convergence in deep neural networks. However, early applications of deep supervision showed suboptimal performance when the network structure was not sufficiently deep or when the auxiliary classifier was a traditional SVM model, particularly in very deep networks. In 2015, Wang et al. proposed an approach²³ to apply deep supervision in deeper network structures. This approach not only retains the feature representation capability of the main network, enabling more diverse expressions, but also promotes multi-level interactions between different branches through an optimization formula with a probabilistic prediction matching loss. This ensures a more robust optimization process and improved expressive capability. In recent years, studies such as those by Zhu et al.²⁴ have integrated U-Net with Deep Supervision for medical image segmentation tasks, achieving notable results. In abdominal multi-organ segmentation tasks, due to the high cost of data annotation and data scarcity, pseudo-labeling techniques are widely used to augment training datasets. Pseudo-labeling techniques involves using the model's predictions as “pseudo” labels to include unlabeled data in the training process. Research has highlighted a method using pseudo-labeling techniques and noise injection in medical image translation tasks to enhance model generalizability and accuracy. Zhang et al.²⁵ introduced a pseudo-labeling-based semi-supervised learning method for medical image segmentation, demonstrating the effectiveness of pseudo-labeling techniques in improving model performance through experiments. Arazo et al.²⁶ explored the application of pseudo-labeling techniques in deep semi-supervised learning, discussing the potential biases introduced by pseudo-labeling techniques and proposing the corresponding solutions. These studies indicate that pseudo-labeling techniques can aid in expanding training datasets and enhancing model performance in abdominal multi-organ image segmentation tasks. In the context of abdominal multi-organ image segmentation, multi-stage training is an effective strategy for handling complex tasks, large-scale data, or multiple sub-tasks. Tao et al. proposed a multi-scale hierarchical training method²⁷ to address the challenges of high resolution and multi-organ complexity in medical image segmentation tasks. Through staged training, this method effectively improves segmentation accuracy and robustness. Research indicates that multi-stage training can help models better manage complexity and diversity in abdominal multi-organ image segmentation tasks, leading to improved segmentation outcomes.

Inspired by the aforementioned methods, we propose an enhanced multi-stage training approach that integrates variants of U-Net, deep supervision, and pseudo-labeling techniques to address challenges such as data scarcity and difficulties in model training, ultimately enhancing model performance and generalizability in abdominal multi-organ image segmentation tasks.

Methods

To achieve better overall segmentation performance of multiple abdominal organs of different sizes, we developed a lightweight VGG-13 network²⁸ as the encoder. This was implemented on the U-Net architecture by incorporating attention mechanisms²⁹ and deep supervision.^30,31 The multi-stage training and pseudo-labeling techniques were introduced to enhance model performance. The multi-stage training method involves initially conducting coarse training using a large dataset, followed by fine-tuning with fully labeled data from the same dataset. The pseudo-labeling method entails generating the first batch of pseudo-labeling data from model predictions using the optimal weights during fine-tuning. The subsequent pseudo-labels are derived from the pseudo- labels of previous generation. The optimal model obtained during the pseudo-labeling training process will be used as the final model for the entire training process.³²

Proposed methodology

X-ray CT images are first preprocessed. It involves intensity normalization by clipping each image to intensity values within [1, 99] percentile range. Z-score normalization is then applied based on the mean and standard deviation of intensity values across the entire training dataset. Additionally, 3D CT images are sliced into 2D images according to their channel count. The architecture of the DLAU-Net is shown in Figure 1. The core model consists of a U-Net architecture, integrated with deep supervision learning modules and attention mechanisms. Forward skip connections are maintained from the decoder stages to the corresponding encoder stages. In the encoder stage, we employ the VGG-13 encoder along with batch normalization layers. The fundamental layer module is composed of multiple layers, including 3 × 3 convolutional layers (conv) with a stride of 1, padding of 1, followed by batch normalization (BN) and rectified linear unit (ReLU) activation. This pattern is repeated twice, followed by 2 × 2 max pooling (MP). The en-coder includes a sequence of four of the above pattern ([conv, BN, ReLU] × 2 + MP), as illustrated in Figure 1. Compared to the original VGG-13, the top layers, including fully connected layers and softmax, are omitted. The fifth [conv, BN, ReLU] × 2 pattern in the original VGG-13 is utilized as the central part for the separation and expansion paths.

Figure 1.

Architecture of the proposed DLAU-Net. In the diagram, up-conv refers to transposed convolution, attention is a combination of the ReLU and Sigmoid functions, and bilinear interpolation is performed.

To achieve a symmetric U-Net architecture, the decoder branch is expanded in a manner similar to the encoder, by adding batch normalization layers and additional feature channels, and incorporating attention modules after the concatenation operations. The added attention module can be expressed as:

O = F \otimes R e s a m p l e r (σ (C o n v_{1 \times 1} (R e L U ([F, F^{'}])))),

(1)

where O represents the output of the attention module. The feature matrix F represents the current feature matrix of the decoder, while F^′ denotes the corresponding feature matrix on the encoder. σ (·) denotes the Sigmoid function. The symbol ⊗ represents the cross product operation. The Resampler module performs resampling operations on the input sequence. Additionally, feature maps output by each intermediate decoder block are upsampled to the size of the input image using bilinear interpolation. A 3 × 3 kernel convolution operation generates a feature matrix with 16 channels at each level. These feature maps are then processed through deep supervision modules to enhance gradient flow and encourage the learning of more useful representations.³¹ After the deep supervision and concatenate operation outputs, the final two convolutional layers perform pixel-wise multi-label segmentation. The concatenate operation can be expressed:

F = c o n c a t (F_{64}, F_{128}, F_{256}, F_{512}),

(2)

where F layer represents the concatenation result with a shape of (512 × 512 × 64), and F₆₄, F₁₂₈, F₂₅₆, and F₅₁₂ correspond to feature matrices obtained through 3 × 3 convolution operations at different levels, where smaller numerical indices indicate that the corresponding feature matrix is closer to the network's output.

After performing pixel-wise multi-label segmentation, post-processing is conducted. During the post-processing, the greatest connected segmentation regions of the voxels are retained and labeled according to 13 organs, such as liver, spleen, and pancreas. Additionally, the network independently processes axial slices to generate 2D segmentation masks, which are subsequently restored to a 3D volume at the end of the process.

Model training process

The model training process consists of three main steps, as shown in Figure 2. (Step 1) Multi-stage Training: Initially, the training is conducted in phases. The first phase involves coarse training, where the model learns large-scale features from the images. Subsequently, fine training follows, enabling the model to grasp finer and more interconnected features. (Step 2) Pseudo-labeling Training: In the pseudo-labeling training, the first-generation pseudo-labeling training is conducted to further educate the model on fine features that were not fully learned in the first phase. Subsequent iterations of pseudo-labeling training continue to refine these labels, progressively enhancing the model's accuracy in recognizing fine features. (Step 3) Optimal Model Selection: Finally, using the evaluation criteria defined in this study, the optimal model weights are identified from those obtained after pseudo-labeling training. This process yields a model with the best performing.

Figure 2.

Illustration of the model training process.

Multi-stage training

The phase-wise training is divided into two stages as depicted in Figure 3: coarse training and fine training. In the coarse training stage, a dataset comprising 1066 CT scans with fewer than 13 label categories and 250 CT scans with exactly 13 label categories is used as the coarse training dataset. This dataset is employed to train the original DL-attention-U-Net model, aiming to achieve the optimal model for coarse training. In the fine training stage, only the 250 CT scans containing exactly 13 label categories are utilized as the fine training dataset. This fine training dataset is used to further train the model based on the optimal model obtained from the coarse training stage, aiming to achieve the optimal model for fine training.

Figure 3.

Coarse and fine training stages in the multi-stage training.

Pseudo-labeling training

Figure 4 shows the process of pseudo-labeling training. It is conducted in an iterative manner, with the process divided into two stages: the first generation pseudo-labeling training and subsequent iterations. The method employs self-training iteration based on the Dice coefficient criteria. (1) First Generation Pseudo-labeling Training Stage: In this stage, the optimal model from fine training predicts labels for 1066 CT images with fewer than 13 label categories and 884 unlabeled CT images. This prediction results in 1950 pseudo-labeled CT mask data containing exactly 13 label categories. (2) Subsequent pseudo-labeling Training Stage: Using the pseudo-labeled mask data from the first generation and their original images, along with 250 CT images already labeled with exactly 13 categories, the original DL-attention-U-Net model is trained to obtain the first generation pseudo-labeling model. (3) Iterative Process: The first generation pseudo- labeling model then predicts labels for the next batch of 1066 CT images with fewer than 13 label categories and 884 unlabeled CT images, generating 1950 pseudo-labeled CT mask data for the second generation. This process repeats iteratively (N times, where N is a hyperparameter) to train subsequent generations of pseudo-labeling models. (4) Final Optimal Pseudo-labeling Model: After N iterations of pseudo-labeling training, the optimal pseudo-labeling model is achieved based on the iterative improvement guided by the Dice coefficient.

Figure 4.

The first generation and the subsequent pseudo-labeling training during the pseudo-labeling training phase.

Loss function

We use the cross-entropy loss function (L_CE) in our network, and it is defined as follow:

L_{C E} = \frac{1}{N} \sum_{c \in C} \sum_{i = 1}^{N} y_{i}^{c} \log {\hat{y}}_{i}^{c},

(1)

where N represents the number of pixels in axial slices,

{\hat{y}}_{i}^{c}

and

y_{i}^{c}

denote the predicted probability and ground truth for class label c at pixel i, respectively. The set C contains labels for the background and 13 types of abdominal organs. The overall loss function L is the weighted sum of cross-entropy losses estimated at different decoder levels involving supervision

L = \sum_{i = 1}^{M} W_{i} L_{C E}^{i} + W^{'} L_{C E}^{'},

(2)

where W_i and

L_{C E}^{i}

represent the weight and loss of the supervised point at the i-th level of the decoder, and M = 4.

W^{'}

and

L_{C E}^{'}

correspond to the weight and loss computed at the final network output (where ʹ denotes final). We set W₁ = 0.8, W₂ = 0.7, W₃ = 0.6, W₄ = 0.5, and

W^{'} = 1

. Note that level i = 1 is closer to the end of the network compared to levels i > 1.

Experiments and results

Dataset

Experimental data are derived from the FLARE 2023 Challenge that is an extension of FLARE 2021–2022,^33,34 aiming at advancing foundational models for abdominal disease analysis. Segmentation labels includes 13 organs and different abdominal pathologies. In this study, image data from 2200 CT scans are utilized, with mask data restricted to those covering abdominal organ labels, excluding masks containing only tumor labels. Case-specific classifications within the dataset are detailed in Figure 5. The training data are gathered under licenses from over multiple sources, including TCIA,³⁵ LiTS,³⁶ MSD,³⁷ KiTS,^38,39 autoPET,^40,41 TotalSegmentator,⁴² and Abdominal CT-1 K.⁴³ The training set includes 2200 CT scans. The validation set consists of 50 CT scans officially released during the competition's online validation phase, incorporating annotations for 13 distinct abdominal organs. Organ annotations are generated by utilizing ITK-SNAP,⁴⁴ nnU-Net,⁴⁵ and MedSAM.⁴⁶

Figure 5.

Case-specific distributions of the Flare 2023 dataset.

To further evaluate the model's generalizability, extended validation experiments are conducted utilizing the publicly available Automatic Cardiac Diagnosis Challenge (ACDC) dataset [https://www.creatis.insa-lyon.fr/Challenge/acdc/ The ACDC dataset comprises 150 fully annotated clinical cases. In this study, we specifically employ this dataset for downstream multi-organ segmentation tasks to focus on three cardiac structures: left ventricle (LV), right ventricle (RV), and myocardium (Myo).

Evaluation metrics

Two widely used evaluation metrics, average accuracy (AVG) and Dice Similarity Coefficient (DSC), are used to assess the segmentation performance of the model. The formulas of these metrics are as follows.

A V G = \frac{T P}{F N + T P},

(5)

D S C = \frac{2 T P}{F N + 2 T P + F P},

(6)

where TP (true positive) is the number of correctly classified as positive samples, FP (false positive) is the number of actually negative samples misdiagnosed as positive, TN (true negtive) is the number of correctly classified negative samples, and FN (false negative) is the number of actually positive samples misdiagnosed as negative ones.

Experimental setup

The iteration settings for the three main training stages of the experiment are as follows. (1) During the multi-stage training, coarse training involves 100 iterations using 1316 CT images and their mask data. Fine training includes 50 iterations using 250 CT images and their mask data. (2) During the training phase with the attention module, the model is trained on a total of 1316 CT images and mask data for coarse training, and 250 CT images and mask data for fine training. This training process involves a total of 150 iterations. (3) For pseudo-labeling training, the first-generation pseudo- labeling training utilizes 1950 CT image data. The fine training model generates the first-generation pseudo-labeling masks. These 1950 new pseudo-labeling data and 250 CT image data serve as the original data for 200 iterations. Subsequent pseudo-labeling training involves predicting new pseudo-labeling data using the model trained with previous pseudo-labeling data, followed by 50 iterations of training for subsequent iteration. This process continues for a total of n times (where n is a hyperparameter, in this case, 5) for subsequent pseudo-labeling training. Specific hardware and software environment requirements for the experiment are detailed in Tables 1 and 2.

Table 1.

Development environments and requirements.

System	Ubuntu 18.04.6 LTS
CPU	Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
RAM	16 × 4GB
GPU (number and type)	NVIDIA GeForce RTX 2080 Ti
CUDA version	11.3
Programming language	Python 3.7
Deep learning framework	torch 1.7.0 + cu92, torchvision 0. 8.1 + cu92
Specific dependencies	scikit-image, nibabel, torch

Table 2.

Training protocols.

Network initialization size	Normal initialization
Batch	2
Patch size	512 × 512 (full axial slices)
Optimizer	Adam
Learning rate (lr)	10⁻⁵
Lr decay schedule	no decay

Learning rate and loss weights sensitivity analysis

The learning rate is empirically set to Lr = 10⁻⁵, with loss weights W ^′=1, W₁ = 0.8, W₂ = 0.7, W₃ = 0.6, W₄ = 0.5 in our experiment. We conduct two sets of sensitivity analysis. For learning rate sensitivity, loss weights are fixed, the learning rate increases by 10 times (Lr = 10⁻⁴) and decreases by 10 times (Lr = 10⁻⁶), respectively. For loss weight sensitivity, the learning rate is fixed, and loss weights are adjusted by ± 0.1 increments (W ^′=1.1, W₁ = 0.9, W₂ = 0.8, W₃ = 0.7, W₄ = 0.6 and W ^′=0.9, W₁ = 0.7, W₂ = 0.6, W₃ = 0.5, W₄ = 0.4, respectively).

Figure 6 shows the variations of training and validation cross-entropy across epochs under fixed loss weights (1, 0.8, 0.7, 0.6, 0.5). When the learning rate is 10⁻⁵(orange line), the model demonstrates relatively stable convergence with minimal fluctuations, achieving the lowest cross-entropy values in both training and validation. When the learning rate is 10⁻⁶ (blue line), the model exhibits pronounced fluctuations in both training and validation, poor stability, and high cross-entropy values. The performance of the green line, corresponding to the learning rate of 10⁻⁴, lies between the former two cases.

Figure 6.

Learning rate sensitivity analysis. (a) the evolution of training cross-entropy with respect to epochs. (b) the evolution of validation cross-entropy with respect to epochs.

A similar trend is observed in Figure 7 when analyzing the loss weight sensitivity at a fixed learning rate (Lr = 10⁻⁵). The loss weight configuration (1, 0.8, 0.7, 0.6, 0.5) shows relatively small fluctuations, with most cross-entropy values remaining at low levels, indicating better generalizability, compared to the other two alternative weight setups.

Figure 7.

Loss weight sensitivity analysis. (a) the evolution of training cross-entropy with respect to epochs. (b) the evolution of validation cross-entropy with respect to epochs.

In summary, the configuration with Lr = 10⁻⁵ and loss weights (1, 0.8, 0.7, 0.6, 0.5) achieves optimal stability and performance in both training and validation phases.

Ablation experiment

Ablation experiments are conducted on the FLARE 2023 dataset to evaluate the contributions of each component in our method. The baseline model is the classic U-Net architecture. In addition to the baseline, two training modules, multi-stage training and pseudo-labeling training, are incorporated, along with an attention mechanism improvement module. The experiments systematically compare the performance from the baseline model to the final model which includes baseline + multi-stage training + attention + pseudo-labeling training (details shown in Table 3).

Table 3.

Experimental results of the ablation experiments.

Method	Average AVG(%)	Average DSC(%)
Baseline	63.90	62.40
Baseline + pseudo-labeling training + attention	72.55	71.34
Baseline + multi-stage training + attention	82.25	80.11
Baseline + multi-stage training + pseudo-labeling training	88.17	87.78
Baseline + multi-stage training + pseudo-labeling training + attention	90.50	89.05

The baseline model in this study has an organ average AVG value of 63.90% and average DSC value of 62.40%. Our complete baseline + multi-stage training + pseudo- labeling training + attention model achieves the best average AVG value of 90.50% and the best average DSC value of 89.05%, showing significant improvements over the baseline model. Compared with the baseline + pseudo-labeling training + attention model, a remarkable increase of 17.95% for AVG and 17.71% for DSC is obtained by our ultimate model respectively. This highlights that the multi-stage training played a vital role in enabling the model to effectively learn the features of most organs and improving the model performance. Our model moves up 7.95% for AVG and 8.94% for DSC respectively in comparison with the baseline + multi-stage training + attention model, indicating the importance of pseudo-labeling training in enhancing the segmentation accuracy by utilizing the unlabeled data. Our proposed method also makes the model gain 2.33% and 1.27% growth for AVG and DSC values respectively over the baseline + multi-stage training + pseudo-labeling training. This demonstrates that the optimization with attention structures enhances the salient features for organ segmentation to a certain extent. These results show that by incorporating multi-stage training, pseudo-labeling training and attention mechanism techniques, our proposed model could effectively promote the model's learning of large-scale features of 13 organs, edge information of organs, and spatial relationships between organs. Ultimately, our strategy outperforms other configurations, validating its effectiveness.

Comparative experiment

We compare our method with several classical or advanced models using Dice coefficient as the evaluation metric. The comparison includes U-Net, DenseU-Net, ResU-Net, and nnU-Net. Comparative experiments are conducted using the original model methods as described in their respective papers and the same dataset in our study. Models like U-Net and nnU-Net are classical and serve as benchmarks in the field of medical image multi-organ segmentation, whereas DenseU-Net and ResU-Net are considered more advanced with superior performance in their model methodologies.

Tables 4 summarizes the DSC metric for different methods in segmenting the abdominal multi organs. The results show that our method outperforms others on the whole. We achieve a DSC value of 89.05%, which is higher than the traditional U- Net by 6.29%, and exceed DenseU-Net (which emphasizes fine-grained features) and ResU-Net (which focuses on model stability and depth) by 4.9% and 2.44%, respectively. Although our DSC is slightly lower than that of nnU-Net on some smaller and label-scarce organs, our model's overall DSC surpasses that of nnU-Net by 0.09%. This demonstrates that our approach is superior to nnU-Net in the task of abdominal multiorgan segmentation.

Table 4.

The experimental results of different methods in terms of DSC metric.

Method	U-Net	DenseU-Net	ResU-Net	nnU-Net	Ours
Organ	U-Net	DenseU-Net	ResU-Net	nnU-Net	Ours
Liver (li)	89.67	91.10	93.76	96.19	96.67
Right kidney (r-ki)	85.06	86.47	89.10	91.31	92.43
Spleen (sp)	88.01	89.43	92.08	94.36	95.52
Pancreas (pa)	79.03	80.41	81.99	85.09	86.13
Aorta (at)	88.84	90.26	92.93	95.23	96.45
Inferior vena cava (i-v-c)	83.40	84.80	87.42	89.59	88.31
Right adrenal gland (r-a-g)	80.67	82.06	84.61	86.75	84.61
Left adrenal gland (l-a-g)	78.29	79.66	81.24	83.77	82.86
Gallbladder (gb)	80.29	81.67	84.27	86.36	85.92
Esophagus (ep)	76.81	78.18	80.75	83.68	84.35
Stomach (sm)	85.53	86.94	89.57	91.79	91.49
Duodenu (dd)	76.78	78.15	80.72	82.72	82.73
Left kidney (l-ki)	83.54	84.94	87.56	89.73	90.19
Average	82.76	84.15	86.61	88.96	89.05

The qualitative comparison of the abdominal multi-organ segmentation results is in Figure 8, illustrating the performance of different models. In comparison to the U-Net, DenseU-Net, and ResU-Net models, our method achieves the best segmentation performance across 13 organ levels. The classical U-Net model performs least effectively in segmenting small, coherent organs such as the pancreas, adrenal glands, gallbladder, and esophagus, almost failing to learn their distinctive features. While the DenseU-Net and ResU-Net models show some improvement in learning features of these organs, they still struggle with precise edge segmentation of these small organs. Compared to the nnU- Net model, our method outperforms it in the majority of organs' segmentation, although our method slightly lags behind in segmenting organs like the inferior vena cava, left and right adrenal glands, gallbladder, and stomach. The nnU-Net shows marginally better segmentation results for organs like the inferior vena cava and adrenal glands, but it exhibits noticeable redundancy and lacks clear learning of organ edge information. Overall, for large or structurally simple organs, such as the liver, spleen, inferior vena cava, aorta, and left kidney, our model achieves highly overlapping segmentation with ground-truth masks, along with sharper and more precise boundaries. In contrast, other models exhibit minor inaccuracies in regional segmentation and boundary blurring. For small or anatomically complex organs including the pancreas, gallbladder, right adrenal gland, stomach, and duodenum, our model outperforms U-Net, DenseU-Net, and ResU- Net in capturing fine details and organ edges. While nnU-Net also demonstrates competitive performance for some small organs, its segmentation suffers from boundary redundancies and reduced edge clarity. Our approach surpasses these limitations, proving its superior ability to learn spatial relationships and edge features among these small and coherent organs.

Figure 8.

Representative results of our method and several state-of-the-art models for abdominal multi-organ segmentation.

Generalization experiment

Generalization experiments are conducted on the ACDC dataset, with comparative validations against the TransUNet⁴⁷ and Swin-UNet.⁴⁸ The dataset is partitioned into three subsets: 100 cases for training, 30 for validation, and 20 for testing. The experimental results are recorded in Table 5. TransUNet achieves the highest accuracy in LV segmentation. Our method obtains the best accuracy in RV segmentation. Swin-UNet yields the highest accuracy for Myo segmentation. Notably, our method achieves the best overall average DSC score among these three approaches, along with more balanced performance across all three regions (LV, RV and Myo). This aligns with its robust performance on the FLARE2023 challenge dataset. Additionally, Table 5 also statistically analyzes the model complexity (parameter count, Params) between different models. Our model has the fewest parameters, proving its advantage in computational efficiency.

Table 5.

DSC scores of different models on ACDC dataset.

Model	LV	RV	Myo	Average DSC(%)	Params(M)
TransUNet	96.03	88.76	84.71	89.83	96.07
Swin-UNet	95.67	87.80	86.11	89.86	41.34
Our Model	93.44	91.33	85.09	89.95	34.50

Discussion

This study proposes a novel training framework and network model to improve the accuracy and training speed of abdominal multi-organ segmentation under conditions of limited labeled data and training time. To this end, we design a segmentation training framework based on multi-stage training and pseudo-labeling techniques, and introduce an innovative DLAU-Net network model, to address the challenges in abdominal multiorgan segmentation.

The DLAU-Net model consists of a U-Net structure, deep supervision learning, and attention modules. Compared to the traditional U-Net, the DLAU-Net model incorporates an attention mechanism that allows the model to dynamically focus on more important regions, ignoring irrelevant background or noise. This improvement enhances the model's precision in boundary segmentation and fine-grained targets. The addition of deep supervision modules helps to improve gradient propagation and enhance feature learning diversity. By introducing auxiliary losses at different layers, the deep supervision module effectively mitigates the vanishing gradient problem, ensuring that each layer receives sufficient gradient signals to better learn low-level image features. The network learns multi-level features across different layers, aiding in a more comprehensive understanding of the input data and improving the model's overall representational capability.

Due to the abundance of organ features in abdominal multi-organ segmentation tasks, achieving high accuracy with only a robust network model is challenging. Therefore, we combine the DLAU-Net model with a training framework that integrates multistage training and pseudo-labeling strategies to better address the challenges in abdominal multi-organ segmentation. According to the ablation study, after incorporating the multi-stage training strategy, the DSC value is significantly improved by 16.73%. With the addition of pseudo-labeling training, the DSC is increased by an additional 9.92%. These improvements in accuracy clearly demonstrate that the multi-stage training effectively helps the model quickly converge by learning the general features of abdominal organs in the early stages of training, while the pseudo-labeling allows the model to learn finer, more detailed features, further enhancing overall accuracy.

Comparative experiments and the qualitative analysis further confirm the effectiveness of our method. These results indicate that our method is superior to the existing methods in quantitative metrics, and it also has the potential to provide more precise and reliable segmentation results in practical applications.

Despite the promising results achieved in abdominal multi-organ segmentation tasks, it is still suboptimal for the performance of our proposed method on small and highly deformable organs. Future efforts could involve more diverse datasets to validate its generalization, or develop targeted modules for segmentation of those challenging organs.

Conclusion

In this paper, we develop a novel multi-organ segmentation method specifically designed for abdominal CT images. Utilizing the DLAU-Net network, pseudo-labeling training, and a multi-stage training strategy, the network achieves rapid convergence and demonstrates sufficiently accurate multi-organ segmentation performance on the Flare2023 dataset. This method provides a novel and effective approach for clinical diagnostic scenarios such as abdominal disease diagnosis, cancer treatment, and radiotherapy planning. Despite achieving commendable segmentation results, there are limitations in segmenting smaller and strongly adherent organs, such as the inferior vena cava and adrenal glands. Moving forward, our team will focus on targeted research for these challenging organs and explore the advanced techniques, such as domain adaptation, federated learning, and multi-model ensemble methods.

Footnotes

ORCID iDs

Panpan Wu

Peng An

Ziping Zhao

Runpeng Guo

Xiaofeng Ma

Yue Qu

Yurou Xu

Hengyong Yu

Funding

This research was funded in part by the National Natural Science Foundation of China under Grant Nos. 61902282 and 62071330.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Ronneberger

Fischer

Brox

. U-net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5–9, 2015, proceedings, part III 18. Springer, 2015, pp. 234–241.

Litjens

Kooi

Bejnordi

, et al. A survey on deep learning in medical image analysis. Med Image Anal 2017; 42: 60–88.

Hesamian

Jia

, et al. Deep learning techniques for medical image segmentation: achievements and challenges. J Digit Imaging 2019; 32: 582–596.

Barshan

Fieguth

. Stage-wise training: An improved feature learning strategy for deep models. In: Feature extraction: Modern questions and challenges. PMLR, 2015, pp. 49–59.

Yuan

Yan

Jin

, et al. Stagewise training accelerates convergence of testing error over SGD. In: Advances in neural information processing systems. 32. Curran Associates, INC., 2019.

Lee

, et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on challenges in representation learning, ICML. vol. 3. Atlanta, 2013, p. 896.

Grandvalet

Bengio

. Semi-supervised learning by entropy minimization. In: Advances in neural information processing systems. 17. MIT Press, 2004.

Kumar

Rao

. Noisy student training using body language dataset improves facial expression recognition. In: European conference on computer vision. Springer, 2020, pp. 756–773.

Pham

Dai

Xie

, et al. Meta pseudo labels. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 11557–11568.

10.

Kumar

Daumé

. A co-training approach for multi-view spectral clustering. In: Proceedings of the 28th international conference on machine learning (ICML-11), 2011, pp. 393–400.

11.

Zhao

Xie

, et al. Multi-view learning overview: recent progress and new challenges. Inf Fusion 2017; 38: 43–54.

12.

Blum

Mitchell

. Combining labeled and unlabeled data with co-training. In: Proceedings of the eleventh annual conference on Computational learning theory. 1998, pp. 92–100.

13.

Zhang

Wang

Chen

. Detecting false data injection attacks in smart grids: a semi-supervised deep learning approach. IEEE Trans Smart Grid 2020; 12: 623–634.

14.

Valindria

Pawlowski

Rajchl

, et al. Multi-modal learning from unpaired images: Application to multi-organ segmentation in CT and MRI. In: 2018 IEEE winter conference on applications of computer vision (WACV). IEEE, 2018, pp. 547–556.

15.

Fang

Yan

. Multi-organ segmentation over partially labeled datasets with multi-scale feature abstraction. IEEE Trans Med Imaging 2020; 39: 3619–3629.

16.

Zhou

Rahman Siddiquee

Tajbakhsh

, et al. Unet++: A nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support: 4th international workshop, DLMIA 2018, and 8th international workshop, ML-CDS 2018, held in conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4. Springer, 2018, pp. 3–11.

17.

Milletari

Navab

Ahmadi

. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international conference on 3D vision (3DV). IEEE, 2016, pp. 565–571.

18.

Isensee

Petersen

Klein

, et al. nnu-net: Self-adapting framework for u-net-based medical image segmentation. arXiv preprint arXiv:180910486. 2018.

19.

Diakogiannis

Waldner

Caccetta

, et al. ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data. ISPRS J Photogramm Remote Sens 2020; 162: 94–114.

20.

Jin

, et al. Dense-U-net: dense encoder–decoder network for holographic imaging of 3D particle fields. Opt Commun 2021; 493: 126970.

21.

Wang

Shelhamer

, et al. Deep layer aggregation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2403–2412.

22.

Lee

Xie

Gallagher

, et al. Deeply-supervised nets. In: Artificial intelligence and statistics. Pmlr, 2015, pp. 562–570.

23.

Wang

Lee

, et al. Training deeper convolutional networks with deep supervision. arXiv preprint arXiv:150502496. 2015.

24.

Zhu

Turkbey

, et al. Deeply-supervised CNN for prostate segmentation. In: 2017 international joint conference on neural networks (IJCNN). IEEE, 2017, pp. 178–184.

25.

Zhang

Zhu

Hallinan

, et al. Boostmis: Boosting medical image semi-supervised learning with adaptive pseudo labeling and informative active annotation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 20666–20676.

26.

Arazo

Ortego

Albert

, et al. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In: 2020 International joint conference on neural networks (IJCNN). IEEE, 2020, pp. 1–8.

27.

Tao

Sapra

Catanzaro

. Hierarchical multi-scale attention for semantic segmentation. arXiv preprint arXiv:200510821. 2020.

28.

Simonyan

Zisserman

. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556. 2014.

29.

Oktay

Schlemper

Folgoc

, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:180403999. 2018.

30.

Xie

. Holistically-nested edge detection. In: Proceedings of the IEEE international conference on computer vision. 2015, pp. 1395–1403.

31.

Dou

Karimi

Rollins

, et al. A deep attentive convolutional neural network for automatic cortical plate segmentation in fetal MRI. IEEE Trans Med Imaging 2020; 40: 1123–1133.

32.

Lyu

Xiong

Fang

, et al. Advancing multi-organ and pan- cancer segmentation in abdominal CT scans through scale-aware and self-attentive modulation. In: MICCAI Challenge on fast and low-resource semi-supervised abdominal organ segmentation. Cham: Springer, 2023, pp.84–101.

33.

Zhang

, et al. Fast and low-GPU-memory abdomen CT organ segmentation: the flare challenge. Med Image Anal 2022; 82: 102616.

34.

Zhang

, et al. Unleashing the strengths of unlabeled data in pan-cancer abdominal organ quantification: the flare22 challenge. arXiv preprint arXiv:230805862. 2023.

35.

Clark

Vendt

Smith

, et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 2013; 26: 1045–1057.

36.

Bilic

Christ

, et al. The liver tumor segmentation benchmark (lits). Med Image Anal 2023; 84: 102680.

37.

Simpson

Antonelli

Bakas

, et al. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXiv:190209063. 2019.

38.

Heller

Isensee

Maier-Hein

, et al. The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: results of the KiTS19 challenge. Med Image Anal 2021; 67: 101821.

39.

Heller

McSweeney

Peterson

, et al. An international challenge to use artificial intelligence to define the state-of-the-art in kidney and kidney tumor segmentation in CT imaging. Am Soc Clin Oncol 2020; 38: 626–626.

40.

Gatidis

Hepp

Früh

, et al. A whole-body FDG-PET/CT dataset with manually annotated tumor lesions. Sci Data 2022; 9: 601.

41.

Gatidis

Früh

Fabritius

, et al. Results from the autopet challenge on fully automated lesion segmentation in oncologic pet/ct imaging. Nat Mach Intell 2024; 6: 1396–1405.

42.

Wasserthal

Breit

Meyer

, et al. Total Segmentator: robust segmentation of 104 anatomic structures in CT images. Radiol Artif Intell 2023; 5: e230024.

43.

Zhang

, et al.

Abdomenct-1k: is abdominal organ segmentation a solved problem?

IEEE Trans Pattern Anal Mach Intell 2021; 44: 6695–6714.

44.

Yushkevich

Gao

Gerig

. ITK-SNAP: An interactive tool for semi-automatic segmentation of multi-modality biomedical images. In: 2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, 2016, pp. 3342–3345.

45.

Isensee

Jaeger

Kohl

, et al. nnU-Net: a self- configuring method for deep learning-based biomedical image segmentation. Nat Methods 2021; 18: 203–211.

46.

, et al. Segment anything in medical images. Nat Commun 2024; 15: 654.

47.

Chen

Mei

, et al. TransUNet: rethinking the U-Net architecture design for medical image segmentation through the lens of transformers. Med Image Anal 2024; 97: 103280.

48.

Cao

Wang

Chen

, et al. Swin-unet: Unet-like pure transformer for medical image segmentation. In: European conference on computer vision. Springer, 2022, pp. 205–218.