Spine X-ray image segmentation based on deep learning and marker controlled watershed

Abstract

Background

The development of automatic methods for vertebral segmentation provides the objective analysis of each vertebra in the spine image, which is important for the diagnosis of various spinal diseases. However, vertebrae have inter-class similarity and intra-class variability, and some adjacent vertebrae exhibit adhesion.

Objective

To solve the adhesion problem of adjacent vertebrae and ensure that the boundary between adjacent vertebrae can be accurately demarcated, we propose an image segmentation method based on deep learning and marker controlled watershed.

Methods

This method consists of a dual-path model of localization path and segmentation path to achieve automatic vertebral segmentation. For the vertebral localization path, a high-resolution network (HRNet) is used to locate vertebral center. Moreover, based on spine posture, a new bone direction loss (BD-Loss) is designed to constrain HRNet. For the vertebral segmentation path, we proposed a VU-Net network to achieve vertebral preliminary segmentation. Additionally, a position information perception module (PIPM) is introduced to realize the guidance of HRNet to VU-Net. Finally, we novelly use the outputs of HR-Net and VU-Net deep learning networks to initialize the marker controlled watershed algorithm to suppress the adhesion of adjacent vertebrae and achieve vertebral fine segmentation.

Results

The proposed method was evaluated on two spine X-ray datasets using four metrics. The first dataset contains sagittal images of the cervical spine, while the second dataset contains coronal images of the whole spine, both with different health conditions. Our method achieved Recall of 96.82% and 94.38%, Precision of 97.24% and 98.14%, Dice coefficient of 97.03% and 96.22%, Intersection over Union of 94.24% and 92.72% on the cervical spine and whole spine datasets respectively, outperforming current state-of-the-art techniques.

Keywords

spine X-ray image vertebral localization vertebral segmentation deep learning marker controlled watershed

Introduction

The spine plays a vital role in the human body, providing support and stability for daily activities. Unfortunately, sedentary behavior, poor posture, and other lifestyle changes have slowly led to spinal diseases becoming one of the primary health concerns for individuals of all ages.¹ X-ray imaging is the preferred method for the diagnosis of spinal diseases due to its simple operation and low cost. Currently, doctors often manually segment vertebrae and perform parameter estimation based on medical guidelines, which is not only time-consuming but also susceptible to subjective factors.² Automatic segmentation of vertebrae greatly reduces the effort required for marking and guarantees consistent diagnosis.³ Therefore, it is of great significance to explore an automatic vertebral segmentation method to assist doctors in diagnosis. However, the process of automatic vertebral segmentation in X-ray images faces challenging problems such as the similarity between adjacent vertebrae, the variability of the same vertebra in different images and the adhesion of adjacent vertebrae.

Vertebral segmentation is the task of classifying vertebrae into regions of interest at the pixel level.^4,5 In recent years, deep learning has been developed rapidly and used widely in various fields due to its strong learning ability and excellent mobility.⁶ In particular, in the field of vertebral segmentation task, deep learning shows encouraging potential. At present, there are two main methods for vertebral segmentation based on deep learning: single-task segmentation and multi-task segmentation. For the single-task segmentation, it only focuses on segmenting vertebrae in the image into regions of interest. Lessmann et al.⁷ proposed a single fully convolution network that iteratively regresses vertebral labels, independent of the number of visible vertebrae in spine CT and MR images. To accommodate highly variable vertebral shapes, Rehman et al.⁸ integrated a parametric level set algorithm in a deep learning network. The probability map from the pre-trained network is used to initialize the level set to improve the robustness to noise. Additionally, Rak et al.⁹ proposed a novel graph cut formulation that makes full use of the anatomical information of vertebral shapes in spine MR images. In order to obtain vertebral semantic feature, Han et al.¹⁰ proposed a long short-term memory network, which increases the long-range spatial correlation between pixels and realizes pixel-level segmentation. Payer et al.¹¹ proposed to segment each vertebra from coarse to fine by employing a spatial configuration network to determine vertebral coordinates and performing refined binary segmentation on each identified vertebra in spine X-ray images. Pang et al.¹² proposed a new two-stage framework that connects a 3D graph convolutional network for 3D coarse segmentation with a 2D residual U-Net for 2D fine segmentation, which solves the problems of vertebral intra-class variability and inter-class similarity. Besides, Al Arif et al. proposed to combine the three processes of global localization, center localization and vertebrae segmentation in a single thread to realize the segmentation of spine X-ray images.

For the multi-task segmentation, it includes not only segmenting each vertebra but also simultaneously performing other tasks such as vertebral localization and recognition.¹³ Pang et al.¹⁴ proposed a dual-path network that simultaneously locates and segments vertebrae. Zhang et al.¹⁵ introduced a co-attention module to learn the correlation information between vertebral localization, recognition and segmentation tasks during the network propagation. Then, Li et al.¹⁶ proposed a multi-channel cross attention for fusing features from vertebral localization and segmentation networks to generate a comprehensive description of the spine.

In the single-task segmentation methods, the methods^7–9 using vertebral prior knowledge effectively realized vertebral segmentation, while Pang¹² and Al Arif¹⁷ realized vertebral segmentation from coarse to fine through the multi-stage network. However, these single-task methods lack generalization and depend on the quality of the dataset. In the multi-task segmentation methods,^14–16 the relationship between tasks is used to segment vertebrae, which effectively deals with the diversity and complexity of vertebrae images. However, due to the integration of multiple tasks, the network structure becomes complex and is prone to the overfitting problem. Additionally, it is difficult to accurately identify vertebral edge information, and adhesion is prone to occur when predicting adjacent vertebrae. To solve the above problems, we propose a two-stage method which accomplishes automatic segmentation of vertebrae by fusing the vertebral prior knowledge, the relationship between vertebral localization and segmentation networks, and the marker controlled watershed algorithm. The challenges associated with vertebral under-segmentation and over-segmentation, particularly the vertebral adhesion, are effectively addressed. Our main contributions are summarized as follows:

A position information perception module (PIPM) is proposed to convert the localization features of the vertebral localization network into a guidance matrix, directing the vertebral segmentation network to focus on vertebral related regions. The vertebral localization network uses a high-resolution network (HRNet) to locate vertebral center. To solve the problem of information loss caused by direct upsample of HRNet, an atrous unit is proposed. Additionally, a bone orientation loss (BD-Loss) is proposed to constrain HRNet by taking spine posture as a prior knowledge for vertebral precise localization.

A VU-Net, based on the encoder and decoder structure of U-Net, is used as the vertebral segmentation network for vertebral preliminary segmentation. The encoder of VU-Net adopts VGG16 to extract vertebral edge information and introduces dropout layers to address the limited generalization capability during training on small-scale spine datasets. To achieve cross-layer sharing and reuse of features, dense units are introduced in the decoder of VU-Net. Additionally, guidance units are introduced in the decoder allows for the fusion of HRNet and VU-Net features, thereby enabling HRNet to guide VU-Net.

The marker controlled watershed algorithm, combined with deep learning, is employed for vertebral fine segmentation. Utilizing the outputs of the HRNet localization network and the VU-Net segmentation network, the watershed algorithm is initialized to detect vertebral edge by simulating the separation of water flow and dike, enabling the automatic and accurate segmentation of adjacent vertebrae.

The structure of this paper is organized as follows. Section 2 describes the proposed method in detail. Section 3 describes the experimental conditions and evaluation metrics. Section 4 analyzes the experimental results. The final conclusions section is contained in Section 5.

Methods

The framework of the proposed method is shown in Figure 1. As the localization network, the improved HRNet locates vertebral center. The proposed VU-Net constituting the segmentation network achieves vertebral preliminary segmentation under the guidance of the proposed PIPM, which introduce the output of HRNet into VU-Net. Then, based on the outputs of these two networks the marker controlled watershed algorithm accomplishes vertebral fine segmentation in spine X-ray images which is the ultimate aim in this paper.

Figure 1.

The framework of the proposed method, which includes a localization network (HRNet) for vertebral center localization, a segmentation network (VU-Net) for vertebral preliminary segmentation, and a marker controlled watershed algorithm for vertebral fine segmentation.

Vertebral localization and segmentation

Localization network

To extract a set of highly representative vertebral localization features, the HRNet¹⁸ is improved as the localization network, retaining its advantage of being robust to coordinate perturbations. HRNet always maintains high-resolution branches to keep the features of the target edge and other details. And information is iteratively exchanged between parallel high-resolution and low-resolution branches to capture long-range features. However, HRNet adopts direct upsample to fuse feature maps of different resolutions after the final fusion stage, neglecting the global information of the feature maps. To solve this problem, we propose an atrous unit that contains a series of convolution operations with different convolution kernels and dilation rates. With the assistance of the atrous unit, the improved HRNet can capture features of different scales and receptive fields.

The improved HRNet is shown in Figure 2. The main body consists of four fusion stages and three transition stages. Each fusion stage is responsible for the interaction between different branches. The third and fourth fusion stages are stacked four times and two times, respectively. The transition stage generates a branch downsampled by a factor of 2. Before entering the main body, the input image is first downsampled by a factor of 4. After a series of fusion and transition, four features with different resolutions are obtained, denoted as C1, C2, C3, and C4, which correspond to 1/4, 1/8, 1/16, and 1/32 of the input image, respectively. Then, the P1 and P2 feature maps are obtained by the direct upsample and the atrous unit, respectively. As shown in Figure 2, the atrous unit consists of 1 × 1 convolutions that reduce the parameters of HRNet, 3 × 3 convolutions that capture vertebral internal features, and dilated convolutions with different rates that capture broader contextual information of the feature maps. Based on the resolution of the feature maps, ratios of 5, 3 and 1 are used for the C1, C2, and C3 feature maps, respectively. The result of vertebral center localization is obtained by upscaling the fusion of the P1 and P2 feature maps by a factor of 4.

Figure 2.

The architecture of the improved HRNet. P1 is obtained by the direct upsample. P2 is obtained by the atrous unit.

Position information perception module

In order to enhance the ability of the vertebral segmentation network to identify vertebrae in spine X-ray images, a position information perception module (PIPM) is proposed to convert the localization features of HRNet into a guidance matrix M, enabling HRNet to guide the segmentation network effectively.

As shown in Figure 3, the guidance matrix M consists of two sets of weight coefficients, denoted as M1 and M2. The M1 weight coefficient implies all features in the spine image. Some convolution layers are first performed on the localization features to generate different feature maps fa, fb, and fc. The fa and fb feature maps are learned to obtain features that distinguish vertebral intra-class variability and inter-class similarity, respectively. Then the matrix multiplication of fa and fb feature maps is performed, and the Softmax activation function is used to normalize the feature maps into probability distribution maps. Finally, the M1 weight coefficient is obtained by multiplying the probability distribution maps and fc feature map, which constructs the dependencies between pixels. The M2 weight coefficient focuses attention on the related regions of vertebral localization. Max pooling and average pooling are used to capture the spatial information of high-level and low-level features, respectively. Max pooling extracts vertebral salient features while maintaining invariance, and average pooling aggregates the overall information of the image. Finally, the M2 weight coefficient is obtained through the Sigmoid layer. After obtaining the MI and M2 weight coefficients, we fuse their channels to obtain the guidance matrix M. The higher the pixel values in the M guidance matrix, the greater the probability they represent vertebral localization.

Figure 3.

The architecture of the PIPM. M is the guidance matrix. M1 and M2 are the weight coefficient.

Segmentation network

For the vertebral segmentation network, a VU-Net network is proposed to separate the vertebrae from the background, based on the U-Net.¹⁹ When dealing diverse spine images, the simplicity of the U-Net structure often results in inaccurate segmentation of vertebrae. In this paper, we improved the U-Net structure in several aspects, as shown in Figure 4. In the VU-Net encoder stage, the simple and efficient VGG16 network with smaller 3 × 3 convolution kernels is used to capture richer vertebral edge information. In addition, dropout layers are added to prevent the overfitting problem in processing small-scale spine datasets, thereby achieving regularization of VU-Net. In the VU-Net decoder stage, dense units are added to allow each layer to directly obtain the gradient information of the previous layer. These dense units serve to enhance feature representation capabilities, enable deeper image analysis, and alleviate the problem of gradient disappearance. In addition, guidance units are added to fuse the features of HRNet and VU-Net to minimize the interference from extraneous information during the segmentation process. These guidance units enable VU-Net to achieve vertebral preliminary segmentation under the guidance of HRNet.

Figure 4.

The architecture of the vu-net. M is the guidance matrix generated by the PIPM.

In detail, compared to the standard U-Net, the VU-Net encoder stage consists of 12 conv units, 4 max pooling layers, and 3 dropout layers. Each conv unit uses a 3 × 3 convolution with the ReLU activation function to preserve vertebral edge information. Max pooling layers are introduced in the first, second, third, and fourth downsample layers, which have a window of 2 and a step size of 2. By stacking conv units and max pooling layers, VU-Net progressively extracts features and diminishes feature size. Dropout layers are introduced in the third, fourth, and fifth downsample layers to minimize the overfitting. The VU-Net decoder stage consists of 8 conv units, 3 dense units, 3 guidance units and a 1 × 1 convolution. Dense units and guidance units are introduced in the first, second, and third upsample layers. Stacking multiple dense units at a relatively low computational cost creates a large receptive field with fewer parameters. Guidance units utilize the guidance matrix M generated by the PIPM, multiplied element-wise with the segmentation features, facilitating the fusion of HRNet and VU-Net features. Finally, in the last layer of VU-Net, a 1 × 1 convolution generates the probability map of the input image, achieving vertebral preliminary segmentation.

Loss function

For the two deep learning networks HRNet and VU-Net, we use different loss functions. The overall loss function is designed as follows:

L = W_{a} L_{M S E} + W_{b} L_{B D} + W_{c} L_{C E} + W_{d} L_{D ice}

(1)

where

L_{M S E}

and

L_{B D}

are the regression loss of HRNet.

L_{C E}

and

L_{D ice}

are the loss of VU-Net. The weights

W_{a}

W_{b}

W_{c}

and

W_{d}

are the degree of different loss on the L. Among them, we empirically set

W_{b}

to 2 and the other scalars to 1.

The label for HRNet is the heatmap generated by the vertebral center, which is manually labeled. In this paper, $L_{M S E}$ is the mean squared error (MSE) loss, which is simple and easy to optimize, helping HRNet to converge quickly. Additionally, we propose a new loss function, bone direction loss (BD-Loss), denoted as $L_{B D}$ , which leverages spine posture as prior knowledge. $L_{B D}$ enhances the correlation between predicted and labeled values by integrating inherent vertebral direction information. As presented in (2), $L_{B D}$ first calculates the angular difference between the lines formed by adjacent vertebral centers corresponding to the predicted and labeled values, and then converts the angular difference to a cosine value.

L_{B D} = \sum_{i = 1}^{N - 1} (1 - c o s (| a r c t a n \frac{y_{i} - y_{i + 1}}{x_{i} - x_{i + 1}} - a r c t a n \frac{y_{i}^{*} - y_{i + 1}^{*}}{x_{i}^{*} - x_{i + 1}^{*}} |))

(2)

where

(x_{i}, y_{i})

and

(x_{i + 1}, y_{i + 1})

represent the labeled coordinates of the adjacent vertebral centroids, respectively.

(x_{i}^{*}; y_{i}^{*})

and

(x_{i + 1}^{*}; y_{i + 1}^{*})

represent the predicted coordinates of the adjacent vertebral centroids, respectively.

L_{B D}

magnifies the difference between the predicted and labeled values, making HRNet robustness to surgical implants and sensitive to abnormal samples.

In the VU-Net deep learning network, we denote $L_{C E}$ as the cross entropy loss and $L_{D i c e}$ as the dice loss, considering both class differences and pixel-level similarity. $L_{C E}$ emphasizes differences between vertebrae and background, helping VU-Net in focusing on vertebrae. The vertebral region in the spine image is unbalanced compared to the background, and the vertebral boundaries is blurred. $L_{D i c e}$ resolves this imbalance and encourages VU-Net to effectively capture vertebral shape.

Marker controlled watershed

In order to solve the problem of blurred or adherent vertebral boundaries caused by factors such as spinal diseases or surgical implants, as indicated by the red arrow in Figure 5(b). We use the marker controlled watershed algorithm as a post-processing step, incorporating the deep learning to separate the adherent vertebrae and generate continuous boundaries with a single pixel width. The watershed algorithm is a graph-based technique originally proposed by Vincent et al.,²⁰ which relies on the gray gradient information and regards the pixels as the height on the terrain. However, the gradient information of the ribs and other parts in the spine image is similar to vertebrae, which easily leads to the mis-segmentation of vertebrae. The predefined markers specify the region of interest. However, currently methods rely on manually labeled markers, resulting in limitations such as high requirements for expert knowledge, subjectivity, and constrained generalization ability in the segmentation process. In this paper, we used the heatmap generated by HRNet as the marker, which reflects the topographic surface of the watershed. The heatmap exhibits the max response at the vertebral center, with pixel values decreasing as they approach the edge of the vertebra, as shown in Figure 5(c). It is worth noting that the pixel values between adjacent vertebrae are close to negative one, as indicated by the red arrow in Figure 5(c), which determines the position of the watershed line between them. In addition, the vertebral preliminary segmentation result of VU-Net is employed as the image to be fined segmented by the watershed algorithm, which has eliminated substantial background regions, as shown in Figure 5(b). By integrating the outputs of the above two deep learning networks, HRNet and VU-Net, as input to the watershed algorithm, we implement vertebral fine segmentation by simulating the geographic flooding process, where the neighboring vertebrae are separated and the vertebral edges are accurately determined, as shown in Figure 5(d).

Figure 5.

The post processing. (a) is a section of the whole spine input image. (b) is the result of VU-Net. (c) is the result of HRNet. (d) is the result of the marker controlled watershed. Red arrow highlights the boundaries of adjacent vertebrae.

Experiments

Dataset and pre-processing

The experiments are conducted on two spine X-ray datasets for training and validation. The first dataset includes 295 sagittal images of the cervical spine annotated with six vertebrae from the Second Hospital of Shanxi Medical University. Images range in size from 700 to 3000 pixels, with a variety of spine orientations, patient poses, and cropping styles. This dataset includes images of vertebral fractures, degenerative changes, and surgical bone implants. Some of the images had jewelry, clothing and other extraneous items. Each vertebra in the image was labeled by orthopedic experts. The second dataset is the contains 90 coronal images of the whole spine annotated with 18 vertebrae from the public dataset AASCE 2019.²¹ The two datasets contain different conditions such as spinal diseases, surgical implants, and poor image quality, truly reflecting the diversities and challenges of the spine image. In the following part, two datasets denoted as Dataset-CS and Dataset-WS respectively. To prevent the network overfitting problem, we augment the two dataset using data augmentation techniques such as translation, mirroring, and rotation, resulting in 3984 cervical spine images and 1962 whole spine images. The two datasets are divided into training and validation sets with a ratio of 9:1. To ensure that all experiments use the same augmented dataset, the same random state is applied during data augmentation. Before being input into the model, the images of the two datasets are resized to 256 × 256 and 256 × 512 pixels respectively, without distortion compression by adding gray bars on both sides. They are then normalized by subtracting the mean pixel value and dividing by the standard deviation of the pixel values.

The two datasets consist strongly supervised labels of manually labeled vertebral masks and weakly supervised labels of vertebral center coordinates. For HRNet, weakly supervised labels are converted into heatmaps that can be learned as a regression task. Knowing the coordinates $(i *, j *)$ of the vertebral center, we apply the following formula for conversion:

y (i, j) = \exp (- \frac{{(i - i *)}^{2} + {(j - j *)}^{2}}{2 σ^{2}})

(3)

where

y (i, j)

represents the pixel value at coordinates

(i, j)

on the heatmap. The Gaussian kernel radius

σ

value is determined according to the size of the vertebrae. The exponential term in the formula calculates the distance between the vertebral center and each pixel in the spine image.

Training and evaluation metrics

The experiments were conducted on an NVIDIA GeForce RTX 3080. During the training process, we adopted a batch size of 2 setting and used the Adam optimizer with an initial learning rate of 0.01. At the same time, we monitor the loss of training and validation images, and evaluate the performance of the model every 5 epochs. When the training loss keeps decreasing and the validation loss no longer improves or starts increasing, we stop the training process. Four evaluation metrics including Recall, Precision, Dice coefficient, and Intersection over Union (IoU) are used to evaluate our model. Recall measures the ability of the model to recognize vertebrae. Precision gauges the ability of the model to predict how many of the predicted vertebrae belong to the labeled vertebrae. Both Dice and IoU measure the overlap between the predicted vertebrae and the labeled vertebrae, with Dice assigning more weight to true positives in its calculation.

Results and discussion

Results and discussion of the ablation experiment

We conducted an ablation experiment based on the U-Net, successively adding the proposed VU-Net, atrous, PIPM, BD-Loss and marker-controlled watershed modules, which was used to evaluate the performance gains from each module and the impact of their combination improvements on the model. The experimental results are shown in Table 1 and Figure 6. The best results are shown in bold in the table, and the differences between the experimental results are indicated in box form. The final model achieves values of 96.82%, 97.24%, 97.03%, and 94.24% of the four metrics on Dataset-CS, and values of 94.60%, 98.13%, 96.33%, and 92.92% of the four metrics on Dataset-WS. The highest Precision, Dice and IoU are achieved on the two datasets, and vertebrae can be distinguished with clear edges.

Figure 6.

Results of the ablation study on Dataset-CS and Dataset-WS. Rows 1 and 2 are images from Dataset-CS，and rows 3 and 4 are images from Dataset-WS. (a) Ground Truth. (b) U-Net. (c) VU-Net. (d) VU-Net + HRNet + PIPM. (e) VU-Net + HRNet-atrous + PIPM. (f) VU-Net + HRNet-atrous + PIPM + $L_{B D}$ . (g) Ours.

Table 1.

The results of the ablation study on Dataset-CS and Dataset-WS in terms of Recall (%), Precision (%), Dice (%), and IoU (%) metrics.

Methods	Dataset-CS				Dataset-WS
Methods	Recall	Precision	Dice	IoU	Recall	Precision	Dice	IoU
U-Net	97 . 102	87.534	92.070	85.306	92.955	95.078	94.248	89.122
VU-Net	95.224	95.304	95.264	90.956	93.045	95.310	95.178	90.799
VU-Net + HRNet + PIPM	95.495	95.908	95.701	91.756	93.535	97.596	95.522	91.429
VU-Net + HRNet-atrous + PIPM	96.338	96.133	96.235	92.744	95.090	96.272	95.779	91.899
VU-Net + HRNet-atrous + PIPM + $L_{B D}$	96.732	96.675	96.704	93.618	95.133	96.780	95.949	92.214
VU-Net + HRNet-atrous + PIPM + $L_{B D}$ + Watershed	96.821	97.244	97.032	94.235	94.594	98.129	96.329	92.918

In the baseline experiment based on the U-Net, Dataset-CS exhibits the highest Recall, indicating that the U-Net can effectively identify vertebral regions. But the two datasets show the lowest Precision, meaning that there is mis-segmentation. As seen in Figure 6(b), regions such as the pelvis, surgical implants, jaw, and teeth are mislabeled as vertebrae. The introduction of the VU-Net increases Precision values on the two datasets compared to the U-Net, with a notable 7.77% increase on Dataset-CS. As seen in Figure 6(c), the mis-segmentation regions of the two datasets are significantly reduced, indicating that the VU-Net improves the ability to capture global information and reduces false positives. However, there are still some erroneous segmentation regions with low overlap with the ground truth, reflected in the lower IoU. Then, we replaced the localization network with the HRNet and introduced the PIPM, which converts the features of the HRNet into a guidance matrix integrated into the VU-Net. This allows the HRNet to guide the VU-Net to focus on vertebral related regions. The results show that the four metrics are increased on the two datasets, and the vertebral regions are accurately segmented, as shown in Figure 6(d). Nonetheless, issues of over-segmentation and under-segmentation persist, especially around vertebrae with nearby surgical implants. To address this, we introduce an atrous unit to help the HRNet identify more vertebral pixels. This improves the Recall, Dice, and IoU metrics on the two datasets. However, there is a decrease in Precision on Dataset-WS, indicating that while the atrous unit enhancing the ability of the model to obtain more global information, it also focuses the model on noise. In addition, Figure 6(e) shows adhesion of adjacent vertebrae in the two datasets and inaccurate localization of adjacent vertebrae on Dataset-WS. To achieve precise vertebral localization, we design the BD-Loss to constrain the HRNet based on spine posture. This improves all metrics on the two datasets, with the highest Recall achieved on Dataset-WS. As shown in Figure 6(f), each vertebra can be accurately localized regardless of the spine posture changes. However, distinguishing some adjacent vertebrae remains challenging, particularly on Dataset-WS where adjacent vertebrae are close and boundaries unclear. Therefore, we introduce the marker controlled watershed. Experiments show that the Dice metric of the final model reaches 97.03% on Dataset-CS and 96.33% on Dataset-WS, which means that the under-segmentation and over-segmentation problems are effectively balanced. Compared to the U-Net, the final model improves Precision, Dice and IoU metrics on Dataset-CS by 9.71%, 4.96%, and 8.93%, respectively. On Dataset-WS, the final model improves the four metrics by 1.64%, 3.05%, 2.08%, and 3.80%, respectively. Figure 6(e) shows no adhesion of adjacent vertebrae on the two dataset, and vertebrae edges are clearly visible.

Results and discussion of the comparative experiment

Additionally, we conducted a comparative experiment with the current state-of-the-art models, including U2-Net,²² DGMSNet,¹⁴ and GCN.²³ The experimental results, shown in Table 2 and Figure 7. Our model outperforms these latest models in the Recall, Precision, Dice, and IoU metrics on the two datasets. Although all models are capable of detecting vertebral relevant regions, our model stands out in terms of detail retention and effectively mitigates issues including under-segmentation, mis-segmentation, and vertebral adhesion.

Figure 7.

Results of the comparison study on Dataset-CS and Dataset-WS. Rows 1 and 2 are images from Dataset-CS，and rows 3 and 4 are images from Dataset-WS. (a) Ground Truth. (b) U2-Net. (c) DGMSNet. (d) GCN. (e) Ours.

Table 2.

The results of the comparison study on Dataset-CS and Dataset-WS in terms of Recall (%), Precision (%), Dice (%), and IoU (%) metrics.

Methods	Dataset-CS				Dataset-WS
Methods	Recall	Precision	Dice	IoU	Recall	Precision	Dice	IoU
U2-Net	95.503	96.267	95.883	92.092	94.090	94.630	94.359	89.321
DGMSNet	95.490	95.809	95.708	91.765	93.553	97.600	95.532	91.403
GCN	94.331	95.042	94.685	89.906	89.980	91.726	90.844	83.225
Ours	96 . 821	97.244	97.032	94.235	94.378	98.143	96.224	92.723

U2-Net performs well in all metrics on Dataset-CS, with Recall, Precision and Dice all exceeding 95%, but IoU is slightly lower and there is adhesion phenomenon. On Dataset-WS, although the performance is slightly lower than Dataset-CS, Recall, Precision and Dice remained above 94%. However, under-segmentation is evident, with weak overlap between prediction and true segmentation, which is reflected in the low IoU value. DGMSNet performs slightly worse than U2-Net, but still delivers good results. On Dataset-CS, Recall, Precision and Dice are also above 95%, despite a lower IoU and under-segmentation. On Dataset-WS, the Precision is 97.6%, indicating robust segmentation accuracy. However, Dataset-WS contains more vertebrae and interference areas, and mis-segmentation occurs in segmentation. GCN extends the convolution operation to the image domain and can better locate each vertebra. However, GCN is not as effective as other models in detecting vertebral pixels, and there are serious adhesion and under-segmentation problems on both datasets, with all metrics being the lowest in the comparison methods. Compared to other models, the proposed model achieves the highest in the four metrics on the two datasets and performs well in capturing details.

Conclusions

This paper introduces an automatic vertebral segmentation method in spine X-ray images based on deep learning and marker controlled watershed algorithm, which solves the problems of mis-segmentation and vertebral adhesion by utilizing the prior knowledge of vertebrae, the correlation between tasks, and the marker controlled watershed algorithm. Specifically, we propose a BD-Loss based on spine posture, which enhances spatial constraints on HRNet and improves its robustness. Then, we designed a PIPM to integrate HRNet features into a proposed VU-Net network, allowing VU-Net to focus on vertebral related regions while ignoring distant background regions, thus achieving vertebral preliminary segmentation. Finally, the outputs of HRNet and VU-Net are used to initialize the watershed algorithm to achieve vertebral fine segmentation. Extensive experiments on two spine X-ray datasets demonstrate the excellent performance of the proposed model. There is no adhesion between adjacent vertebrae, and each vertebra is segmented with distinct edge. The vertebral segmentation results obtained by our method can efficiently calculate a variety of clinical indicators, which enables the identification and diagnosis of a variety of spinal diseases, including scoliosis, kyphosis, and vertebral fractures, thereby assisting physicians in achieving rapid and reliable diagnosis. However, there exists an underfitting phenomena, particularly in the regions of the vertebral edge, primarily due to insufficient consideration of the specificity of local vertebral deformations. In the future, we will explore more precise methods to accommodate various vertebral variations.

Footnotes

Acknowledgment

We would like to thank the editors and reviewers for improving the content of this article. This work was supported in part by the Basic Research Program of Shanxi Province (202103021224204).

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

, et al. Current development and prospects of deep learning in spine image analysis: a literature review, (in eng). Quant Imaging Med Surg 2022; 12: 3454–3479.

Masood

Taj

Khan

, et al. Deep learning based vertebral body segmentation with extraction of spinal measurements and disorder disease classification. Biomed Signal Process Control Article; Early Access 2022; 71: 103230. Art no. 103230.

Mushtaq

Akram

Alghamdi

, et al. Localization and edge-based segmentation of lumbar spine vertebrae to identify the deformities using deep learning models. Sensors Feb 2022; 22: 1547. Art no. 1547.

Qadri

, et al. CT-Based Automatic spine segmentation using patch-based deep learning. INT J Intell Syst Mar 2023; 2023: 1–14. Art no. 2345835.

Meng

Boyer

Pujades

. Vertebrae localization, segmentation and identification using a graph optimization and an anatomic consistency cycle. Comput Med Imag Graph Article; Early Access 2023; 107: 102235. Art no. 102235.

Cheng

Yang

, et al. Automatic vertebrae localization and segmentation in CT with a two-stage Dense-U-net. Sci Rep Article Nov 12 2021; 11: 22156. Art no. 22156.

Lessmann

van Ginneken

de Jong

, et al. Iterative fully convolutional neural networks for automatic vertebra segmentation and identification. Med Image Anal Article Apr 2019; 53: 142–155.

Rehman

Ali Shah

Riaz

, et al. A robust scheme of vertebrae segmentation for medical diagnosis. IEEE Access 2019; 7: 120387–120398.

Rak

Steffen

Meyer

, et al. Combining convolutional neural networks and star convex cuts for fast whole spine vertebra segmentation in MRI. Comput Methods Programs Biomed 2019; 177: 47–56.

10.

Han

Wei

Mercado

, et al. Spine-GAN: semantic segmentation of multiple spinal structures. Med Image Anal 2018; 50: 23–35.

11.

Payer

Stern

Bischof

, et al. Coarse to fine vertebrae localization and segmentation with SpatialConfiguration-Net and U-Net. VISI GRAPP(5: VISAPP);124–133.

12.

Pang

, et al. Spineparsenet: spine parsing for volumetric MR image by a two-stage segmentation framework with semantic image representation. IEEE Trans Med Imaging Article Jan 2021; 40: 262–273.

13.

Zhang

, et al. Autonomous localization and segmentation for body composition quantization on abdominal CT. Biomed Signal Process Control Article; Early Access 2022; 71: 103172. Art no. 103172.

14.

Pang

, et al. DGMSNet: spine segmentation for MR image by a detection-guided mixed-supervised segmentation network. Med Image Anal Article; Early Access 2022; 75: 102261. Art no. 102261.

15.

Zhang

Xiao

Liu

, et al. MRLN: multi-task relational learning network for MRI vertebral localization, identification, and segmentation. IEEE J Biomed Health Inform Article Oct 2020; 24: 2902–2911.

16.

Wang

Shen

, et al. SSCK-Net: spine segmentation in MRI based on cross attention and key-points recognition-assisted learner. Biomed Signal Process Control Article; Early Access 2023; 86: 105278. Art no. 105278.

17.

Al Arif

SMMR

Knapp

Slabaugh

. Fully automatic cervical vertebrae segmentation framework for X-ray images. Comput Methods Programs Biomed Article Apr 2018; 157: 95–111.

18.

Sun

Xiao

Liu

, et al. Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp.5693–5703.

19.

Ronneberger

Fischer

Brox

. U-net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, 2015, pp.234–241: Springer.

20.

Vincent

Soille

. Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Trans Pattern Anal Mach Intell 1991; 13: 583–598.

21.

Bailey

Rasoulinejad

, et al. Automatic landmark estimation for adolescent idiopathic scoliosis assessment using BoostNet. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, 2017.

22.

Qin

Zhang

Huang

, et al. U2-Net: going deeper with nested U-structure for salient object detection. Pattern Recognition 2020; 106: 107404.

23.

Zhang

Zhan

, et al. Semi-supervised classification of graph convolutional networks with laplacian rank constraints. Neural Process Lett 2022; 54: 2645–2656.