Segmentation of liver tumors with abdominal computed tomography using fully convolutional networks

Abstract

BACKGROUND:

Dividing liver organs or lesions depicting on computed tomography (CT) images could be applied to help tumor staging and treatment. However, most existing image segmentation technologies use manual or semi-automatic analysis, making the analysis process costly and time-consuming.

OBJECTIVE:

This research aims to develop and apply a deep learning network architecture to segment liver tumors automatically after fine tuning parameters.

METHODS AND MATERIALS:

The medical imaging is obtained from the International Symposium on Biomedical Imaging (ISBI), which includes 3D abdominal CT scans of 131 patients diagnosed with liver tumors. From these CT scans, there are 7,190 2D CT images along with the labeled binary images. The labeled binary images are regarded as gold standard for evaluation of the segmented results by FCN (Fully Convolutional Network). The backbones of FCN are extracted from Xception, InceptionresNetv2, MobileNetv2, ResNet18, ResNet50 in this study. Meanwhile, the parameters including optimizers (SGDM and ADAM), size of epoch, and size of batch are investigated. CT images are randomly divided into training and testing sets using a ratio of 9:1. Several evaluation indices including Global Accuracy, Mean Accuracy, Mean IoU (Intersection over Union), Weighted IoU and Mean BF Score are applied to evaluate tumor segmentation results in the testing images.

RESULTS:

The Global Accuracy, Mean Accuracy, Mean IoU, Weighted IoU, and Mean BF Scores are 0.999, 0.969, 0.954, 0.998, 0.962 using ResNet50 in FCN with optimizer SGDM, batch size 12, and epoch 9. It is important to fine tuning the parameters in FCN model. Top 20 FNC models enable to achieve higher tumor segmentation accuracy with Mean IoU over 0.900. The occurred frequency of InceptionresNetv2, MobileNetv2, ResNet18, ResNet50, and Xception are 9, 6, 3, 5, and 2 times. Therefore, the InceptionresNetv2 has higher performance than others.

CONCLUSIONS:

This study develop and test an automated liver tumor segmentation model based on FCN. Study results demonstrate that many deep learning models including InceptionresNetv2, MobileNetv2, ResNet18, ResNet50, and Xception have high potential to segment liver tumors from CT images with accuracy exceeding 90%. However, it is still difficult to accurately segment tiny and small size tumors by FCN models.

Keywords

Abdominal computerized tomography (CT)fully convolutional network liver tumor segmentation

1 Introduction

Clinical diagnostic tools for studying liver tumors include sonography, magnetic resonance imaging, single-photon emission computed tomography, and computer tomography (CT). CT imaging is fast, high-resolution, and commonly used to diagnose various diseases. Hence, it is an essential imaging tool to examine liver tumors.

Image segmentation in the field of computer vision refers to the process of selecting or marking digital images based on features or object types. Image segmentation encircles the edges of different organs and lesions or marks them with specific colors. Image segmentation technology in medical imaging is mainly used in tissue volume measurement, diagnosis, and customized treatment plans. It takes much time to segment organs or tumors semi-automatically through images in the early stage [1 –4]. However, with the advancement of technology, deep learning algorithms can automatically segment or identify many different objects from an image. The traditional main algorithms for image segmentation are cluster analysis, histogram, region growing, and level set method. With the rapid development of artificial intelligence, deep learning algorithms to automatically segment different objects in images have gradually matured. It has been applied in various research fields. In 2014, Jonathan Long et al. published the use of Fully Convolutional Networks (FCN) to perform Semantic Segmentation on color images of 2D natural objects [5]. Before this, semantic segmentation was often identified through feature definitions. The content and location in the image. Olaf Ronneberger et al. used the U-Net network architecture to study biomedical image segmentation and gradually began applying deep learning in image segmentation [6 –8]. This research aims to train a segmentation model that can effectively segment the liver and liver tumors based on the computerized tomographic images of the liver through deep learning.

Nowadays, liver disease is one of the most common diseases among Chinese people, especially liver cancer, the second leading cause of cancer death. Image segmentation is an essential technology in medical imaging. At first, tumors or lesions need to be divided for staging and slice positioning in diagnosis; in radiotherapy, lesions and organs must be divided first. The treatment plan has been completed; after the treatment is completed, the operation Tracking requires image segmentation technology to evaluate the efficacy of the lesion after surgery. Segmentation technology is closely related to medical imaging, and the main segmentation method is manual or semi-automatic, which usually consumes much time. In order to reduce the time of image segmentation and improve the accuracy of segmentation, it is expected to improve the efficiency and accuracy of segmenting the liver and its tumors through deep learning.

Fully Convolution Networks (FCN) were developed to perform segmentation of image by classifying each pixel to a specific group. Each pixel needs to take the surrounding pixels as a patch for operation [9]. In 2015, Olaf Ronnebbrger et al. proposed that U-Net was a neural network architecture based on fully convolutional network architecture, suitable for medical imaging applications. U-Net is an encoder-decoder structure [7, 10]. The encoder extracts feature to reduce the spatial dimension, and the decoder is responsible for restoring the details and spatial dimensions. U-Net’s decoder and encoder are connected to repair the details more accurately. In 2017, Han designed a 2.5 D deep convolutional network [11]. Its 2.5D network architecture means that a set of adjacent slices are used as input (3D), and then a cut image of the intermediate slice image (2D) is output. The network architecture is modified based on the U-Net architecture. ResNet is added the residual connections to modify the network architecture [12]. Hence, the parameters in the external network can be effective training in terms of image pre-processing. It was successful that liver tumor segmentation is for the International Biomedical Imaging Symposium competition, with the highest score of 0.670 for liver tumor Dice Score (DSC) which was similar to the index of IoU.

Furthermore, it is important to use efficient network architecture in order to improve the performance of CNNs [13]. In order to know the performance of segmentation by using different famous backbones of convolutional neural networks, the primary purpose of this research is to use neural network architectures such as inception, mobilenetv2, resnet18, resnet50, and xception to train segmented models for abdominal computer tomography images of liver tumors. It is expected that at least one model could be reasonable and feasible to segment the boundary of liver tumors.

2 Materials and methods

2.1 The data

The data used in this research was obtained from the International Biomedical Imaging Symposium [14, 15]. A total of 131 patients with 3D liver CT images is partly shown in Fig. 1(A) as a two-dimensional image. The images were scanned with different imaging protocols on a CT device. Therefore, the image size and resolutions are not the same. The number of slices of 3D CT for patients ranges from 42 to 1026. The tumors in the liver were selected for performing the segmentation. In addition to 3D computed tomography images of the abdomen, each patient also has images marked by a professional radiologist. In order to obtain feasible visual contrast, the radiologist is usually adjusting the intensity levels for the diagnostic purpose, as shown in Fig. 1(B). The marked images were binary format. The liver tumors were marked as a white block, as shown in Fig. 1(C).

The liver tumor is hepatocellular carcinoma (HCC). The axial resolution and thickness of the image are 0.56 mm to 1.0 mm and 0.45 mm to 6.0 mm, respectively. A total of 7,190 2D liver images were used in this study. The image size was 512×512 with format NIFTI. The minimum and maximum counts of tumors in one image are zero to twenty-five. The 90.7% of tumor counts is about no more than five. The over six tumors in one image are about 9.3% (Table 1).

Fig. 1

The original CT images (A), adjusting intensity level (B), and binary masks for liver tumors (C). In these 2D images, there are shown 22 tumors with boundaries.

Table 1

The numbers of tumor were counted from 7,190 2D images

Tumor counts	0	1	2	3	4	5	6	7	8	9	10	11	12
Counts	20	3,533	1,417	759	496	295	171	117	88	53	52	40	28
Accumulation	20	3,553	4,970	5,729	6,225	6,520	6,691	6,808	6,896	6,949	7,001	7,041	7,069
Relative Accumulation	0.003	0.494	0.691	0.797	0.866	0.907	0.931	0.947	0.959	0.966	0.974	0.979	0.983
Tumor counts	13	14	15	16	17	18	19	20	21	22	23	24
Counts	17	18	22	16	16	12	4	4	2	7	2	1
Accumulation	7,086	7,104	7,126	7,142	7,158	7,170	7,174	7,178	7,180	7,187	7,189	7,190
Relative Accumulation	0.986	0.988	0.991	0.993	0.996	0.997	0.998	0.998	0.999	1.000	1.000	1.000

All the 2D CT liver images are none enhancement (i.e., without injection imaging agent.). A physician diagnosed the types of tumor and confirmed by pathological test. Figure 2 shows raw CT 2D liver axial image (A, D), binary labeling tumors (B, E), and fusion image (C, F). The contrast of tumors in images is not significantly differentiated from normal liver tissues (Fig. 2A, D). Therefore, the diagnostic image is poor by human eyes. However, without injection agent means acceptable by most of the people who might be compliant by imaging agent. The used images were not in DICOM (Digital Image Communication on Medicine) format. Hence, the range of gray levels is limited between 0 and 255 by transformed as PNG gray level with 8 bits per pixels (i.e., gray level range between 0 and 255). The contrast, brightness, sharpening, and details are lost or absent on the PNG image. The boundary of raw images was not clear and sharpening. That is why AI (Artificial Intelligent) with deep learning approaches are adopted in this study to segment the tumors on PNG images. Meanwhile, understanding the segmented performance among AI methods is quite important.

Fig. 2

The 2D liver images (A, C), binary labeling images (B, E), and fusion images (C, F). A physician diagnosed the types of tumor and confirmed by pathology. (A–C) and (D–F) were two different tumors.

2.2 Flowchart of research

In order to prevent the over-fitting problem caused by insufficient data, it is necessary to increase the data set through data enhancement. The method used is to rotate, shift, zoom in and out, and mirror the training set images during each training period (Epoch) to improve the accuracy of models. The original image is then set through the above four parameter values (i.e., size of epoch, size of batch, ratio of training set to all images, and optimizers), and each time a new Epoch is entered, the system automatically performs a random image geometric transformation to increase the number of data sets. The flowchart of this study is shown in Fig. 3.

Fig. 3

The flowchart of this research.

2.3 The fully convolutional networks

This study uses Xception [16], InceptionresNetv2 [17], MobileNetv2 [18], ResNet18 [19], and ResNet50 [19] as the backbone of FCN for image segmentation models (Table 2). The architecture of Xception is 36 convolutional layers to form extraction of features. The 36 convolutional layers are organized into 14 modules with linear residual connections. Meanwhile, the performance of xception is not increasing capacity of parameters but rather to prompt the efficiency of parameters in model.

Table 2
The brief properties of investigated FCN models

FCN Layer Connections input image Size (MB) Developer Year

InceptionresNetv2 824 921 299×299 209 Google 2016

MobileNetv2 154 163 224×224 13 Google 2018

ResNet18 71 78 224×224 44 Microsoft 2015

ResNet50 177 192 224×224 96 Microsoft 2015

Xception 170 181 299×299 85 Google 2017

FCN	Layer	Connections	input image	Size (MB)	Developer	Year
InceptionresNetv2	824	921	299×299	209	Google	2016
MobileNetv2	154	163	224×224	13	Google	2018
ResNet18	71	78	224×224	44	Microsoft	2015
ResNet50	177	192	224×224	96	Microsoft	2015
Xception	170	181	299×299	85	Google	2017

The architecture of inceptionresnetv2 is 824 layers and 206 convolutional layers. The main property is the block designed in model. The inceptionresnetv2 is computational cost and fast training than the other CNNs. Meanwhile, inceptionresnetv2 could be significantly improved the performance of recognition problem.

Resnet18 has 71 layers, including one image input layer (224×224×3), followed by a two-dimensional convolution operation with 64 7×7 filters, horizontal and vertical steps of 2 pixels, complementing the zero mechanism designed for 3 pixels. There are 20 batch normalization layers, with the number of channels ranging from 64 to 512 (Channel); one maximum pooling layer and one average pooling layer; additional layers (Addition). There are eight groups in total. Each group includes two linear rectification layers, a two-dimensional convolution and batch normalization layer (Batch Normalization); a fully connected layer; a softmax layer; a classification layer, and an output layer. The main advantages of resnet18 were large depth of layers that was easily training without increasing the error. Resnets overcome the issues of vanishing gradients by using identity mapping.

Resnet50 has a total of 177 layers, including one image input layer (224×224×3), followed by a two-dimensional convolution operation with a 64-channel 7×7 filter size, horizontal and vertical steps of 2 pixels. The zero-padding mechanism is designed for 3 pixels; there are 52 batch normalization layers in total, and the number of passes is 64, 128, 256, 512, 1024, and 2048 (Channel); one maximum pooling layer and one averaging Pooling layer. There are 16 additional layers (Addition). Each group includes three linear rectification layers, a two-dimensional convolution and batch normalization layer (Batch Normalization); a fully connected layer; a softmax layer; a classification layer, and an output layer. The merit of resnet50 was ensured to improve accuracy and less training time by made of 3-layer bottleneck blocks simultaneously.

Mobilenetv2 has a total of 154 layers, including one image input layer (224×224×3), two-dimensional convolution with 32 3×3 filters with horizontal and vertical steps 2 pixels, 32 batch normalization, a linear rectified layer, and a two-dimensional grouped convolutional layer (2D Grouped Convolutional Layer or also known as Depth-Wise Convolution), batch normalization for every 32 samples. It can speed up the convolutional neural network training and reduce the sensitivity to network initialization; then, use the block design (there are 16 blocks). Each block has two linear rectification layers, three two-dimensional convolution layers, three batch standardization layers, an average pooling layer, a fully connected layer, a softmax layer, a classification layer, and an output layer. The prone of mobilenetv2 could achieve competitive accuracy with a few parameters and less computational complexity.

2.4 Investigated optimizers and parameters for training model

Stochastic gradient decent momentum (SGDM) and adaptive momentum (ADAM) are the two commonly used in FCN. Adam optimizer is equivalent to an optimizer that combines momentum optimization and RMSProp [20 –22]. It can be applied to the situation where the gradient is sparse and the demand for memory is small. T represents the updated weights. m_T and ν_T represent the first-moment vector estimate and second-moment vector estimate of the gradients (Equations 1, 2). β₁ and β₂ are the setting parameters, and the recommended settings are 0.999. g_T is the gradient. $\hat{m_{T}}$ and $\hat{ν_{T}}$ are deviation corrections for m_T and ν_T (Equations 3, 4). The deviation correction can limit the learning rate to a certain range and make the parameter updating more stable. W_T+1 is the weight of the update (Equation 5); η is the learning rate; ∈ is a decimal close to 0 added to prevent the denominator from being 0. $m_{T} \leftarrow β_{1} m_{T - 1} + (1 - β_{1}) g_{T}$ (1) $ν_{T} \leftarrow β_{2} ν_{T - 1} + (1 - β_{2}) g_{T}^{2}$ (2) $\hat{m_{T}} \leftarrow \frac{m_{T}}{1 - β_{1}}$ (3) $\hat{ν_{T}} \leftarrow \frac{ν_{T}}{1 - β_{2}}$ (4) $W_{T + 1} \leftarrow W_{T} - η \frac{\hat{m_{T}}}{\sqrt{\hat{ν_{T}} + ɛ}}$ (5)

In this study, the investigated parameters are listed in Table 3. The investigated parameters include two ratios for the training model (70% and 80%), two optimizers (SGDM and ADAM), five FCN models, five design batch sizes (3, 6, 9, 12, 15), and three design epoch size (3, 6, 9). The input image was re-sizing as 300×300 with the learning rate is 0.0001. Total 300 combinations of investigated parameters (i.e., 2×2×5×5×3) for FCN models are investigated in this study. Setting different parameters for the training model could find suitable FCN more efficiently.

Table 3

The investigated parameters of FCN models

Item	Content	Count
FCN	Xception, InceptionresNetv2, MobileNetv2, ResnNt18, ResNet50	5
Optimizers	SGDM, ADAM	2
Ratio for Training	70%, 80%	2
Batch Size	3, 6, 9, 12, 15	5
Epoch Size	3, 6, 9	3
Image Size	300×300	1
Learning rate	0.0001	1

2.5 Performance index for segmentation

Because the training ratio, batch size, and epoch size were affecting the accuracy of the training model. A total of eighteen combinations (i.e., 2×3×3) of investigated parameters were applied to five FCN models. Each FCN model has executed eighteen combinations of investigated parameters. The global accuracy, mean accuracy, mean IoU (Intersection over Union), weighted IoU, and mean BF (Boundary F1) score were applied to evaluate the segmented performance among FCN with all combinations of parameters. Meanwhile, the optimal FCN model with parameters was evaluated using the mean BF Score and mean IoU value.

In Fig. 4, a diagram for computing matric performance with segmentation of one image by using FCN. Let I be an input image. Let T and G be a target and predicted area with blue and red squares. Let T^c and G^c be the outside of target and outside of predicted area. Hence, true positive (TP) area is defined as T ∩ G. False positive (FP) area is defined as T ∩ G^c. False negative (FN) area is defined as T^C ∩ G. The definitions of accuracy, IoU, and BF score are as Equations (6, 7).

$Accuracy = \frac{TP}{TP + FN} = \frac{| T \cap G |}{| T \cap G | + | T^{c} \cap G |}$ (6) $IoU = \frac{TP}{TP + FP + FN} = \frac{| T \cap G |}{| T \cap G | + | T \cap G^{c} | + | T^{c} \cap G |}$ (7) $BF = \frac{2 TP}{2 TP + FP + FN} = \frac{2 | T \cap G |}{2 | T \cap G | + | T \cap G^{c} | + | T^{c} \cap G |}$ (8)

Fig. 4

A diagram for computing matric performance with segmentation using FCN.

Meanwhile, the global accuracy was computed is the ratio of correctly classified pixels, regardless of class, to the total number of pixels. Using the global accuracy metric could attain a quick and computationally inexpensive estimation of an accurate percentage of classified pixels. The mean IoU was computed for the aggregated data set that is the averaging IoU scores of all classes for all images. The weighted IoU was calculated the averaged IoU of each class which was weighted by the number of pixels in each class. The weighted IoU have considered the numbers of pixel in class in order to reduce the estimated errors caused by less pixels in class. Furthermore, the analysis of variance (ANOVA) was applied to test the statistical significance among invested FCN and parameters.

3 Results

According to the validated results, the CV of the index, Global Accuracy, Mean Accuracy, Mean IoU, Weighted IoU, and Mean BFScore generated by mobilenetv2 are the highest among the other FCN in this study (Table 4). The mobilenetv2 is the middle size of layers (157) and minimum size of parameters (13 MB) among the investigated FCN. Both Mean Accuracy and Mean IoU provided by mobilenetv2 and Xception are over 0.8. The index of performance, both Mean Accuracy and Mean IoU are the best among all FCN. The mobilenetv2 is the potential FCN for segmentation of CT images usefully in this study. The Global Accuracy, Mean Accuracy, Mean IoU, and Weighted IoU were maximum among all FCNs generated by mobilenetv2 (P < 0.05). The coefficient of variation (CV) was defined as Mean/Std. The lower CV, the greater the level of dispersion around the mean. All of the performance index generated by mobilenetv2 were highest than the others in this study.

Table 4
The segmented performance among used FCN models

FCN Index Mean Std Minimum Maximum CV

InceptionresNetv2 Global Accuracy 0.9949 0.0028 0.9912 0.9991 358.1408

Mean Accuracy 0.7551 0.1823 0.5000 0.9770 4.1411

Mean IoU 0.7189 0.1634 0.4959 0.9474 4.4000

Weighted IoU 0.9904 0.0053 0.9835 0.9983 185.7429

Mean BFScore 0.9020 0.0766 0.7467 0.9941 11.7763

MobileNetv2 Global Accuracy^* 0.9973 0.0017 0.9917 0.9988 597.3677

Mean Accuracy^* 0.8800 0.1108 0.5000 0.9611 7.9395

Mean IoU^* 0.8449 0.1025 0.4959 0.9318 8.2406

Weighted IoU^* 0.9947 0.0033 0.9835 0.9977 298.6441

Mean BFScore 0.9063 0.0360 0.8382 0.9941 25.1478

ResNet18 Global Accuracy 0.9949 0.0029 0.9917 0.9986 342.0854

Mean Accuracy 0.7291 0.2033 0.5000 0.9600 3.5865

Mean IoU 0.7013 0.1840 0.4959 0.9216 3.8126

Weighted IoU 0.9900 0.0059 0.9835 0.9973 168.3251

Mean BFScore^* 0.9390 0.0530 0.8574 0.9941 17.7061

ResNet50 Global Accuracy 0.9949 0.0027 0.9917 0.9992 367.8909

Mean Accuracy 0.7297 0.1870 0.5000 0.9800 3.9022

Mean IoU 0.7028 0.1702 0.4959 0.9545 4.1289

Weighted IoU 0.9900 0.0055 0.9835 0.9985 181.3420

Mean BFScore 0.9163 0.0704 0.7990 0.9941 13.0187

Xception Global Accuracy 0.9967 0.0019 0.9917 0.9987 520.8983

Mean Accuracy 0.8486 0.1322 0.5000 0.9558 6.4186

Mean IoU 0.8131 0.1194 0.4959 0.9243 6.8105

Weighted IoU 0.9937 0.0038 0.9835 0.9975 259.4399

Mean BFScore 0.8986 0.0451 0.7853 0.9941 19.9227

FCN	Index	Mean	Std	Minimum	Maximum	CV
InceptionresNetv2	Global Accuracy	0.9949	0.0028	0.9912	0.9991	358.1408
	Mean Accuracy	0.7551	0.1823	0.5000	0.9770	4.1411
	Mean IoU	0.7189	0.1634	0.4959	0.9474	4.4000
	Weighted IoU	0.9904	0.0053	0.9835	0.9983	185.7429
	Mean BFScore	0.9020	0.0766	0.7467	0.9941	11.7763
MobileNetv2	Global Accuracy^*	0.9973	0.0017	0.9917	0.9988	597.3677
	Mean Accuracy^*	0.8800	0.1108	0.5000	0.9611	7.9395
	Mean IoU^*	0.8449	0.1025	0.4959	0.9318	8.2406
	Weighted IoU^*	0.9947	0.0033	0.9835	0.9977	298.6441
	Mean BFScore	0.9063	0.0360	0.8382	0.9941	25.1478
ResNet18	Global Accuracy	0.9949	0.0029	0.9917	0.9986	342.0854
	Mean Accuracy	0.7291	0.2033	0.5000	0.9600	3.5865
	Mean IoU	0.7013	0.1840	0.4959	0.9216	3.8126
	Weighted IoU	0.9900	0.0059	0.9835	0.9973	168.3251
	Mean BFScore^*	0.9390	0.0530	0.8574	0.9941	17.7061
ResNet50	Global Accuracy	0.9949	0.0027	0.9917	0.9992	367.8909
	Mean Accuracy	0.7297	0.1870	0.5000	0.9800	3.9022
	Mean IoU	0.7028	0.1702	0.4959	0.9545	4.1289
	Weighted IoU	0.9900	0.0055	0.9835	0.9985	181.3420
	Mean BFScore	0.9163	0.0704	0.7990	0.9941	13.0187
Xception	Global Accuracy	0.9967	0.0019	0.9917	0.9987	520.8983
	Mean Accuracy	0.8486	0.1322	0.5000	0.9558	6.4186
	Mean IoU	0.8131	0.1194	0.4959	0.9243	6.8105
	Weighted IoU	0.9937	0.0038	0.9835	0.9975	259.4399
	Mean BFScore	0.8986	0.0451	0.7853	0.9941	19.9227

^*means a significant difference among all of FCNs and the mean value is the maximum by ANOVA testing. The coefficient of variation (CV) was defined as Mean/Std. The lower CV, the greater the level of dispersion around the mean.

The investigated parameters include batch and epoch sizes for used FCNs. In order to rank the performance of used FCN for segmentation of CT tumors, the standard deviation (Std) among Global Accuracy, Mean Accuracy, Mean IoU, Weighted IoU, and Mean BFScore is applied to evaluate the FCNs. The high performance of segmentation with FCNs for CT images is defined when Std is smaller than 0.04. Meanwhile, a radar plot is used to visualize the performance of selected FCNs. Also, the frequency of selected FCNs is usually to investigate the strength of segmented potential. Table 5 shows the selected FCNs under Std smaller than 0.04, where Fun is the optimizer, FCN is the investigated models, G.Accuracy is the global accuracy, M.Accuracy is the mean accuracy, M.IoU is the mean IoU, W.IoU is the weighted IoU, and M.BFScore is the mean BF score.

Table 5

The selected top-20 FCNs according to the value of mean IoU. G.Accuracy is global accuracy. M.Accuracy is mean accuracy. M.IoU is mean IoU. W.IoU is weighted IoU. M.BFScore is mean BF score. STD is standard deviation

FCN	Batch	Epochs	G.Accuracy	M.Accuracy	M.IoU	W.IoU	M.BFScore	STD
ResNet50	6	9	0.999	0.969	0.954	0.998	0.962	0.021
ResNet50	3	6	0.999	0.973	0.953	0.998	0.962	0.021
InceptionresNetv2	3	9	0.999	0.961	0.947	0.998	0.961	0.024
ResNet50	15	9	0.999	0.970	0.945	0.998	0.956	0.025
InceptionresNetv2	6	9	0.999	0.951	0.941	0.998	0.963	0.027
InceptionresNetv2	12	6	0.999	0.966	0.936	0.998	0.950	0.028
ResNet50	12	9	0.999	0.980	0.937	0.998	0.946	0.029
InceptionresNetv2	12	9	0.999	0.975	0.937	0.998	0.945	0.029
ResNet50	6	3	0.999	0.966	0.933	0.998	0.947	0.030
MobileNetv2	6	9	0.999	0.960	0.931	0.998	0.945	0.031
MobileNetv2	3	9	0.999	0.951	0.932	0.998	0.943	0.032
InceptionresNetv2	6	6	0.999	0.977	0.920	0.997	0.943	0.035
ResNet18	3	6	0.999	0.960	0.922	0.997	0.935	0.035
MobileNetv2	3	6	0.999	0.943	0.921	0.997	0.937	0.036
InceptionresNetv2	6	3	0.998	0.954	0.916	0.997	0.929	0.038
ResNet18	6	3	0.998	0.942	0.912	0.997	0.938	0.038
ResNet18	15	6	0.999	0.933	0.916	0.997	0.940	0.038
Xception	3	9	0.999	0.936	0.915	0.997	0.935	0.039
MobilenNtv2	9	9	0.999	0.935	0.915	0.997	0.934	0.039
ResNet18	9	3	0.998	0.947	0.908	0.997	0.931	0.040

A total of twenty selected FNCs are optimizer SGDM, batch size 3 to 12, and epoch size 3 to 9 under Std no more than 0.04. The frequency of inceptionresnetv2, resnet50, mobilenetv2, resnet18, and Xception are 6, 5, 4, 4, and 1 time. Hence, the radar plot has displayed the performance of selected FCNs (Fig. 5). In this section, the inceptionresnetv2 and resnet50 are the high potential FCNs for segmented tumors with CT images in this study.

Fig. 5

The relationship between true counts and estimated counts of pixel of tumors was demonstration by scatterplot and linear regression.

4 Discussion

The presented methods are compared with related works in Table 6. The most published works were developed the segmented methods with deep learning on the open dataset, including SYSU-CT, subCT, MICCAI, 3Dircadb, LITS, and ISBI. The samples size was counted by patient bases (i.e., the patient numbers). In previous studies, Gao et al. [13], Kim et al. [23], Luan et al. [24], He et al. [25], Zhang et al. [26], Kushnure et al. [27], Sahli et al. [28], Ayalew et al. [29], Jin et al. [30], and Chen et al. [31] had reported the developed methods for CT liver tumors segmentation. The dice scores (DSC) and IoU have relation as description by Eelbode et al. [32]. The DSC ranged from 0.3 to 0.96, and IoUs ranged from 0.59 to 0.87 on average. The presented results are similar to those of related published works.

Table 6
The presented result was comparisons among published works. The relationship between DSC and IoU was demonstrated in Tom et al. 2020 for estimation of missing DSC or IoU in the Table

Author Database Sample Size FCN DSC/IoU

Gao and Almekkawy, 2021 Open dataset: SYSU-CT 135 ASU-Net++ L4 0.932/0.872

Gao and Almekkawy, 2021 Open dataset: subCT 110 ASU-Net++ L4 0.917/0.846

Kim et al. 2021 Retrospective Study 1350 R-CNN 0.903/0.823

Luan et al. 2021 Open Dataset: MICCAI 200 S-Net 0.740/0.587

He et al. 2021 Retrospective Study 63 RA-U-Net 0.730/0.575

Zhang et al. 2021 Open Dataset: MICCAI 200 3D Network 0.627/0.455

Kushnure and Talbar, 2021 Open Dataset: 3Dircadb 20 U-Net 0.842/0.726

Ayalew et al. 2021 Open Dataset: ISBI 130 U-Net 0.740/0.587

Jin et al. 2021 Open Dataset: 3Dircadb 200 3D U-Net 0.853/0.744

Chen, 2021 Open Dataset: ISBI 130 SED 0.930/0.870

Presented Methods, 2022 Open Dataset: ISBI 131 FCN 0.976/0.954

Author	Database	Sample Size	FCN	DSC/IoU
Gao and Almekkawy, 2021	Open dataset: SYSU-CT	135	ASU-Net++ L4	0.932/0.872
Gao and Almekkawy, 2021	Open dataset: subCT	110	ASU-Net++ L4	0.917/0.846
Kim et al. 2021	Retrospective Study	1350	R-CNN	0.903/0.823
Luan et al. 2021	Open Dataset: MICCAI	200	S-Net	0.740/0.587
He et al. 2021	Retrospective Study	63	RA-U-Net	0.730/0.575
Zhang et al. 2021	Open Dataset: MICCAI	200	3D Network	0.627/0.455
Kushnure and Talbar, 2021	Open Dataset: 3Dircadb	20	U-Net	0.842/0.726
Ayalew et al. 2021	Open Dataset: ISBI	130	U-Net	0.740/0.587
Jin et al. 2021	Open Dataset: 3Dircadb	200	3D U-Net	0.853/0.744
Chen, 2021	Open Dataset: ISBI	130	SED	0.930/0.870
Presented Methods, 2022	Open Dataset: ISBI	131	FCN	0.976/0.954

The segmented performance of the presented FCN methods is related to the numbers of tumors in the CT image (Table 7). The global accuracy with 1–5 counts of tumor is higher than those of over five counts of the tumor. The mean accuracy with over five counts of tumor is higher than those of 1–5 counts of the tumor. The weighted IoU and mean BF score with 1–5 counts of tumor are higher than those of over five counts of the tumor. The Dice score with over five counts of tumor is higher than those of 1–5 counts of the tumor. The dice score is evidence to prove that the presented FCN is reliable to segment multiple tumors in a single CT image.

Table 7

The segmented performance was compared with different counts of tumor

Index	Counts of tumor	Mean	Median	Std	Minimum	Maximum
Global Accuracy	1–5	0.99	1.00	0.06	0.00	1.00
	over 5	0.95	1.00	0.14	0.10	1.00
Mean Accuracy	1–5	0.90	0.98	0.17	0.00	1.00
	over 5	0.93	0.98	0.13	0.34	1.00
Mean IoU	1–5	0.85	0.93	0.17	0.00	1.00
	over 5	0.85	0.92	0.19	0.05	0.99
Weighted IoU	1–5	0.99	1.00	0.08	0.00	1.00
	over 5	0.92	0.99	0.20	0.01	1.00
Mean BF Score	1–5	0.96	0.99	0.08	0.00	1.00
	over 5	0.94	0.97	0.09	0.33	1.00
Dice Score	1–5	0.62	0.76	0.32	0.00	0.97
	over 5	0.70	0.78	0.24	0.00	0.93

Meanwhile, the relationship between true counts and estimated counts of tumors was demonstrated in Fig. 5. Total of 131 patients were included in this study. The true counts of tumors with ranged from 31 to 700000 voxels for the 131 3D liver CT images. In the Fig. 5, the true counts of tumors are smaller than 31 voxels, then it could not be easily segmented by AI methods in this study. The positive relation was demonstration between true and estimated sizes of tumor by a regression model. The linear regression equation was shown the slope term 1.6759 and R² 0.8894 (P < 0.05). The estimated numbers of pixels for tumors were closed to the golden of actual counts in this study.

In the Fig. 6, the true counts of tumors are smaller than 31 pixels, then it could not segment FCN models. Therefore, the tiny size of tumors could be easily segmented by AI methods in this study. However, the tiny tumor with size 31 pixels cross 5 slices (white spots in the bottom row) could not segment by the presented FCM models.

Fig. 6

The case was a tiny tumor in liver in this study (white spots in the bottom row). The volume size was 31 voxels and could not segment by FCN models.

5 Conclusions

The top-20 selected FNC models were the candidates to be segmented approaches under Mean Accuracy no less than 0.944. The occurred frequency of inceptionresnetv2, mobilenetv2, resnet18, resnet50, and Xception were 9, 6, 2, 5, and 3 times. Therefore, the InceptionresNetv2 has better performance than the others. Inceptionresnetv2, mobilenetv2, resnet18, resnet50, and Xception can potentially perform tumor segmentation on CT liver imaging with accuracy exceeding 90%.

This study provides an automated segmented model of liver tumors based on FCN (or U-Net). Through experimentation, several points can be found that can be improved in the future. The first point can be found that image standardization is essential and difficult in medical imaging or in the field of deep learning. Especially in medical imaging, there are individual differences between different patients, instruments, and imaging parameters. Correct standardization images could improve the accuracy of neural networks. Secondly, small tumors are not easy to be cut out, and it is challenging to learn the characteristics of small tumors, and specific structures or parameters are required to improve the probability of segmentation of small tumors. The third point is that it is difficult to obtain medical images and labeled images, which also results in limited training samples. This study only segmented 2D liver tumors. In the future, we could consider segmenting for different organs or use a 3D network architecture to segment 3D images and directly obtain 3D segmentation maps.

In this study, the FCN was trained using a full range of intensity in images without truncation. The FCN has approved a useful approach to segment tumors of the CT liver. The similarity of tumors between each other is almost likely and resemblance. To this end, the tiny spots of tumors almost cannot segment out by FCN in this study. Therefore, the probability of successful segmented spots depends on the tumor’s pixel size with the input image.

Meanwhile, the source of images comes from an open dataset in PNG format. Also, the contrast of images is different. The more heterogeneity images are used in FCN; the more difficult it is to obtain a helpful model. The original format of medical images is DICOM. The performance of segmentation is still to study on DICOM format images in the near future.

Author Contributions: Conceptualization, T.-B.C., C.-I.C., and A.M.; methodology, C.-I.C.; software, T.-B.C., N.-H.L. and S.-Y.H.; validation, K.-Y.L., T.-B.C. and N.-H.L.; formal analysis, C.-I.C. and Y.-H.H.; investigation, Y.-M.W. and Y.-H.H.; data curation, N.-H.L.; writing—original draft preparation, C.-I.C.; writing—review and editing, T.-B.C. and Y.-M.W.; project administration, T.-B.C.; funding acquisition, T.-B.C. All authors have read and agreed to the published version of the manuscript

Funding

This research received no external funding.

Institutional review board statement

Not applicable.

Informed consent statement

Not applicable.

Data availability statement

The liver CT was acquired from “LiTS - Liver Tumor Segmentation Challenge” at URL: https://competitions.codalab.org/competitions/17094.

Footnotes

Acknowledgments

The authors would like to thank the Ministry of Science and Technology in Taiwan, for partial financially supporting this research under Contract MOST110-2118-M-214-001. The authors would like to acknowledge AJE for the English editorial assistance.

Conflicts of interest

The authors declare no conflict of interest.

References

Anter

A.M.

and Hassenian

A.E.

, CT liver tumor segmentation hybrid approach using neutrosophic sets, fast fuzzy c-means and adaptive watershed algorithm, Artif Intell Med 97 (2019), 105–117.

Häme

and Pollari

, Semi-automatic liver tumor segmentation with hidden Markov measure field model and non-parametric distribution estimation, Med Image Anal 16 (2012), 140–149.

Pham

D.L.

, Xu

and Prince

J.L.

, Current methods in medical image segmentation, Annu Rev Biomed Eng 2 (2000), 315–337.

, Wu

, Zhou

, et al., 3D liver tumor segmentation in CT images using improved fuzzy C-means and graph cuts, Biomed Res Int 2017 (2017), 5207685.

Shelhamer

, Long

and Darrell

, Fully convolutional networks for semantic segmentation, IEEE Trans Pattern Anal Mach Intell 39 (2017), 640–651.

Chen

, Wu

, Yang

, et al., An efficient Memristor-based circuit implementation of squeeze-and-excitation fully convolutional neural networks, IEEE Trans Neural Networks and Learning Systems 33 (2021), 1779–1790.

Ronneberger

, Fischer

and Brox

. U-net: Convolutional networks for biomedical image segmentation, In: International Conference on Medical Image Computing and Computer-Assisted Intervention, 9351, 2015, pp. 234–241.

Suzuki

, Overview of deep learning in medical imaging, Radiol Phys Technol 10 (2017), 257–273.

Abdollahzadeh

, Belevich

, Jokitalo

, et al., Deep ACSON automated segmentation of white matter in 3D electron microscopy, Commun Biol 4 (2021), 179.

10.

Badrinarayanan

, Kendall

and Cipolla

, SegNet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Trans Pattern Anal Mach Intell 39 (2017), 2481–2495.

11.

Han

, Wu

, Wang

, et al., Automated segmentation of liver segment on portal venous phase MR images using a 3D convolutional neural network, Insights Imaging 13 (2022), 26.

12.

Chen

, Jiang

, Zhang

, et al., ResNet18DNN: Prediction approach of drug-induced liver injury by deep neural network with ResNet18, Brief Bioinform 23 (2022), bbab503.

13.

Gao

and Almekkawy

, ASU-Net++: A nested U-Net with adaptive feature extractions for liver tumor segmentation, Comput Biol Med 136 (2021), 104688.

14.

Bilic

, Christ

P.F.

, Vorontsov

, et al., The liver tumor segmentation benchmark (LiTS), arXiv:1901.04056, 2019.

15.

Kushnure

D.T.

and Talbar

S.N.

, HFRU-Net: High-level feature fusion and recalibration UNet for automatic liver and tumor segmentation in CT images, Comput Methods Programs Biomed 213 (2022), 106501.

16.

François

, Xception: Deep learning with depthwise separable convolutions, arXiv:1610.02357, 2016.

17.

Christian

, Sergey

, Vincent

and Alex

, Inception-v4, Inception-ResNet and the impact of residual connections on learning, arXiv:1602.07261, 2016.

18.

Andrew

G.H.

, Menglong

, Bo

, et al., MobileNets: Efficient convolutional neural networks form mobile vision applications, arXiv:1704.04861, 2017.

19.

Kaiming

, Xiangyu

, Shaoqing

and Jian

, Deep residual learning for image recognition, arXiv:1512.03385, 2015.

20.

Aytaç

U.C.

, Güneş

and Ajlouni

, A novel adaptive momentum method for medical image classification using convolutional neural network, BMC Med Imaging 22 (2022), 34.

21.

Borysenko

and Byshkin

, CoolMomentum: A method for stochastic optimization by langevin dynamics with simulated annealing, Sci Rep 11 (2021), 10705.

22.

Kingma

D.P.

and Jimmy

, Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.

23.

Kim

D.W.

, Lee

, Kim

S.Y.

, et al., Deep learning-based algorithm to detect primary hepatic malignancy in multiphase CT of patients at high risk for HCC, Eur Radiol 31 (2021), 7047–7057.

24.

Luan

, Xue

, Ding

, et al., Adaptive attention convolutional neural network for liver tumor segmentation, Front Oncol 11 (2021), 680807.

25.

, Liu

, Shahzad

, et al., Advanced deep learning approach to automatically segment malignant tumors and ablation zone in the liver with contrast-enhanced CT, Front Oncol 11 (2021), 669437.

26.

Zhang

, Li

, Du

, et al., 3D Multi-attention guided multi-task learning network for automatic gastric tumor segmentation and lymph node classification, IEEE Trans Med Imaging 40 (2021), 1618–1631.

27.

Kushnure

D.T.

and Talbar

S.N.

, MS-UNet: A multi-scale UNet with feature recalibration approach for automatic liver and tumor segmentation in CT images, Comput Med Imaging Graph 89 (2021), 101885.

28.

Sahli

, Slama

A.B.

and Labidi

, U-Net: A valuable encoder-decoder architecture for liver tumor segmentation in CT images, J Xray Sci Technol 30 (2022), 45–56.

29.

Ayalew

Y.A.

, Fante

K.A.

and Mohammed

M.A.

, Modified U-Net for liver cancer segmentation from computed tomography images with a new class balancing method, BMC Biomed Eng 3 (2021), 4.

30.

Jin

, Meng

, Sun

, et al., RA-UNet: A hybrid deep attention-aware network to extract liver and tumor in CT Scans, Front Bioeng Biotechnol 8 (2020), 605132.

31.

Chen

W.F.

, Ou

H.Y.

, Liu

K.H.

, et al., In-series U-Net network to 3D tumor image reconstruction for liver hepatocellular carcinoma recognition, Diagnostics 11 (2020), 11.

32.

Tom

, Jeroen

, Maxim

, et al., Optimization for medical image segmentation: Theory and practice when evaluating with dice score or Jaccard index, IEEE Trans Med Imaging 39 (2020), 3679–3690.