Knee cartilage MR images segmentation based on multi-dimensional hybrid convolutional neural network

2.1 Knee joint segmentation method

Recently, deep convolutional neural networks have achieved achievements in biological image processing tasks such as classification [14] and segmentation [15–17]. Meanwhile, U-Net-like structures have been widely used in knee joint segmentation tasks. For the segmentation task of key knee joint structures, Woo et al. [18] combined anomaly information detection with the segmentation task for the first time. First, anomaly information is extracted using U-Net for knee MRI and then added to the downstream segmentation task to assist segmentation. Mao et al. [19] also proposed a similar two-stage knee segmentation network. First, significance detection was performed to obtain different bone structure location regions, and then two U-Net networks were used to segment different bone structure regions independently. For the first time, Chen et al. [20] suggested using adversarial loss to enhance the segmentation of bone structure images after resampling. In addition, a recovery network was used to restore the bone structure to its pre-sampling resolution. Meanwhile, Ambellan et al. [21] employed a CNN to locate the bone surface in the first stage, followed by the use of SSM to refine the segmentation by CNN and a smaller 3D-CNN for sub-volume segmentation. The above multi-stage segmentation methods bring segmentation performance improvements while increasing computational complexity.

There is a general problem of category imbalance in medical images, and this problem is more worthy of attention in knee images. To address this problem, Lee et al. [22] proposed a do-differential segmentation network that first segments the bone and cartilage complex (BCC) and then the bone. Cartilage is obtained by subtracting bone from BCC. This avoids segmenting the cartilage directly and alleviates the problem of category imbalance in the segmentation task. Dai et al. [23], on the other hand, proposed a new loss function to solve the problem of category imbalance in images. In addition, the method uses a compact convolutional network to reduce the computational complexity so that the 3D knee images can be fed more completely into the network. Similarly, Raj et al. [24] proposed a multi-category loss function to obtain more accurate segmentation of different types of cartilage.

There are also some researchers who hope to find better segmentation methods than U-Net. Sun et al. [25] explored the effect of replacing the standard convolutional blocks in U-Net [12] and DeepLabv3plu [26] with different blocks such as residual block, residual SE block, dense block, and dense SE block on the task of knee image segmentation and found that the variant using SE block has better feature capturing capability. Sengar et al. [27] utilized a knee joint dataset to assess variants of the U-Net architecture, such as U-Net++ [28] and U-Net3+ [29], from various angles, tasks, and embedding methods. Their objective was to obtain a superior and more widely applicable alternative to U-Net. Kessler et al. [30] applied conditional generative adversarial networks (cGANs) [31] for the first time to the segmentation of knee MRI and compared them with U-Net in various aspects, and they concluded that cGANs networks have better robustness. Kessler et al. [32] used the 3D-CaSM [33] method to extract accurate regional measurements of cartilage morphology on 2D and 3D U-Net segmented knee masks. More et al. [34] proposed the “Discrete-MultiResUNet”, which combines the Discrete Wavelet Transform (DWT) and the MultiResUNet architecture and is more effective in extracting salient features for key tissue segmentation. Overall, the UNet-like method has been shown to be effective in knee segmentation tasks. It is worth noting that U-Net downsampling results in the loss of feature information, which is also present in the other UNet-like methods mentioned above. However, this problem has not been given enough attention.

UVNet employs V-Net as the backbone feature extraction network, which has proven its effectiveness in various medical image segmentation tasks. It uses 3D convolution for feature extraction and achieves end-to-end segmentation through an efficient encoding-decoding structure. In the encoding path, each layer of features undergoes two consecutive identical convolutional layers, followed by batch normalization and the ReLU activation function, and then downsampling using a convolution operation with a stride of two. The decoding path is similar to the encoding path, but the feature map scale is recovered by upsampling, and in the last layer, a 1 × 1 ×1 convolution is performed to map each component feature vector to the desired number of classes. Skip-connections are also used between the encoder and decoder to recover the information loss caused by downsampling.

Traditional medical image segmentation methods usually use a separate 2D or 3D network for feature extraction, which is normally effective. But, in order to extract the deep semantic information, it is necessary to downsample the feature maps, which inevitably results in the loss of information. Even though skip-connections are used in V-Net to supplement the lost information, the skip-connections of the next layer are still obtained by downsampling the previous layer. Knee joint cartilage has an elongated morphology and occupies a relatively small proportion of pixels in the entire medical image. After multiple downsampling steps, this information loss becomes even more critical for cartilage features, leading to issues like under-segmentation or even cartilage breakage in the segmentation mask. Therefore, in addition to using V-Net for feature extraction, we also employ U-Net to segment MIP images as additional supplementary information to alleviate information loss in skip-connections. U-Net utilizes two-dimensional convolution and has the same number of layers as V-Net, making it compatible for feature fusion with V-Net.

3.2 Local MIP images and Global MIP image

Maximum Intensity Projection (MIP) is a projection method used for extracting structures and features from medical images. The conventional projection process involves scanning the entire MRI volume in a specific direction to find the maximum pixel intensity. Then, the maximum intensity values from all pixels along that direction are projected onto a two-dimensional image, which we refer to as the global MIP image. The projection process can be represented as illustrated in Fig. 2. By applying this process separately to the MRI volume and corresponding labels, MIP images and MIP labels are obtained. Although this projection process is relatively straightforward, it finds extensive application in medical image analysis, including tasks such as reconstruction [36, 37], detection [38, 39], and segmentation [40, 41]. During the projection process, cartilage structures are typically retained in the MIP image, providing valuable information about the shape and location of the cartilage.

OPEN IN VIEWER

Fig. 2

Maximum Intensity Projection

It is worth noting that during the process of ray projection, lower pixel values will be overwritten by higher pixel values. Therefore, when acquiring the global MIP image, a substantial amount of valuable shape and positional information is lost, and this should be given due consideration. If the global MIP image is used exclusively as supplementary information, it may introduce interference during the training of V-Net, resulting in a decrease in accuracy. Considering the complexity of the segmentation task, relying solely on global projection as additional information may appear overly simplistic and brute-force. To mitigate the aforementioned issue of information loss and enrich the supplementary information structure, we have employed a method for calculating MIP based on the morphology of cartilage, which we call local MIP. We consider that the cartilage morphology in adjacent slices exhibits similarities. By performing maximum intensity projection across multiple neighboring slices, the loss of information is expected to decrease, and the morphological structure of the cartilage will be more prominently represented in the MIP image. As depicted in Fig. 3, we have partitioned the entire cartilage structure into three parts according to the morphological characteristics, denoted as S₁, S₂, and S₃. The S _j  (j = 1, 2, 3) represents the MRI volume between slices p_j,s and p_j,e, where p_j,s denotes the number of the starting slice, and p_j,e denotes the number of the last slice. We then carried out projections on these three parts separately, resulting in three local MIP images, namely L₁, L₂, and L₃. These local MIP images are combined with the global MIP image, denoted as G, and together they serve as inputs to the U-Net.

OPEN IN VIEWER

Fig. 3

For each MRI volume, three local MIP images and one global MIP image were extracted.

To incorporate the extracted MIP image as supplementary information into the V-Net, we need to perform dimension expansion on different MIP images. We propose two feature fusion modules for this process, referred to as the Local Feature Fusion Module (LMFM) and the Global Feature Fusion Module (GMFM), as shown in Fig. 1. U _i  (i = 1, 2, 3, 4) represent the MIP feature maps obtained from U-Net, with dimensions B i u × C _i × W _i × H _i . The hierarchical feature maps in the V-Net are denoted as V _i  (i = 1, 2, 3, 4), with dimensions B i v × C _i × W _i × H _i × D _i . Here, B represents batch size, C represents the number of channels, and C, W, H, D represent the number of channels, width, height, and depth of the feature maps. We set B i v to 1, while B i u is set to 4, represented as B i u = {l_i,1, l_i,2, l_i,3, g _i }. Among these, l_i,1, l_i,2, and l_i,3 correspond to the local MIP feature maps, and g _i represents the global MIP feature map.

4.1 loss function

In the training process, DiceLoss was selected as the loss function for both 2D and 3D models and defined as, L 2 D ( P mip , G mip ) = 1 - 2 | P mip ∩ G mip | | P mip | ∪ | G mip |(5) Where, P _mip is the segmentation result of 2D networks, and G _mip is the MIP ground truth.

L 3 D ( P , G ) = 1 - 2 | P ∩ G | | P | ∪ | G |(6) Where, P is the segmentation result of 3D networks, and G is the ground truth of femoral cartilage and tibial cartilage.

The overall loss function of network is defined as: L = ω L 2 D + ( 1 - ω ) L 3 D(7) Our final loss consists of losses from two paths, where L_2D and L_3D are loss functions of the 2D network and the 3D network, respectively, and ω represents the weight between the two losses. In our experiment, ω was set at 0.35 for optimal experimental performance.

4.3 Experimental detail

In our study, we divided the two datasets into 80% for network training and 20% for testing. Since the diagnosis of osteoarthritis in clinical practice mainly refers to the morphological manifestations of cartilage, only cartilage tissue labels were retained in the data set. After removing the labels of bone, the dataset contained three categories: 0-background, 1-femoral cartilage,and 2-tibial cartilage. Then, the images are randomly cropped into patches with a size of 256 × 256 × 96, which are sent to the V-Net for training.

During the training period, the learning rate is set to 0.002 in the initial stage, the training batch size of V-Net is set to 1, and the training batch size of U-Net is set to 4, so as to ensure that the training of each MIP is non-interference. The total training epochs are set at 500. The Adam [44] optimizer is used for optimization, and the attenuation coefficient is 0.0001.

4.4 Evaluation metrics

We used the Dice Similarity Coefficient (DSC), Recall, and Precision [45–47] as evaluation metrics, all of which were commonly used to evaluate similarity in the field of medical image segmentation. DSC = 2 TP 2 TP + FP + FN(8) Recall = TP TP + FN(9)

Precision = TP TP + FP(10) where TP, FP, and FN are the number of true positive, false positive, and false negative predictions, respectively.

5.1 Comparison of loss hyper-parameters

Firstly, we study the influence of hyper- parameters on the partial performance of the loss function. We experimented with the effect of different values of ω from 0.15 to 0.75 on the performance of the hybrid network. The experimental results are shown in Fig. 4. When ω = 0.35, the experimental performance is the best.

OPEN IN VIEWER

Fig. 4

The influence of hyper-parameter ω on network performance.

In order to evaluate the performance of our proposed network framework, we conducted a comparative analysis with several popular deep learning-based medical image segmentation methods, such as U-Net [12], U-Net++ [28], V-Net [13], and 3D U-Net [48], on the OAI dataset. Table 1 presents the performance comparison of the different methods on the OAI dataset, with the best results emphasized in bold.

Our proposed hybrid network model, UVNet, which incorporates local and global MIPs, achieved impressive DSC scores of 90.35 for femoral cartilage (FC) and 86.79 for tibial cartilage (TC). These scores outperformed those of other commonly used segmentation methods. The U-Net model obtained DSC scores of 86.91 and 83.40, while the V-Net model achieved DSC scores of 87.45 and 85.24. By utilizing a hybrid network model, we successfully enhanced the segmentation performance for both femoral and tibial cartilage.

Table 2 presents the quantitative results for the SKI-10 MICCAI challenge knee cartilage segmentation dataset. According to the experimental evaluation, the FC and TC DSC scores for the UVNet model are 83.57 and 80.12, respectively. These values are 6.04 and 6.44 points higher than the U-Net and are 4.3 and 2.7 points higher than the baseline V-Net. Furthermore, we compared our work with U-Net++ and 3D U-Net, which have served as robust baselines for various image segmentation tasks. Our DSC scores for FC and TC surpass those of U-net++ by 5.66 and 5.96 points, and outperform 3D U-Net by 5.4 and 3.47 points, respectively. Upon careful visual analysis of the qualitative results, it is evident that the UVNet model produces superior segmentation outcomes. In comparison to other network models, UVNet demonstrates better performance in terms of TC integrity segmentation results, which is often a crucial factor in clinical examinations.

We conducted two experimental directions on the OAI-ZIB dataset to validate the structural rationality of our model. The first direction explores the impact of varying the number of local MIP images on model performance, while the second direction examines the influence of different feature embedding methods on model performance.

(1) Different Number of Local MIP Images: To address the issue of information loss in global MIP images, we introduced local MIP images. A combination of local and global MIP images was used as input to the information augmentation subnet, U-Net. Table 3 illustrates the effect of the extracted number of local MIP images on the model’s segmentation performance. Using too many local MIP images can interfere with the feature extraction of the backbone network, while using too few can result in significant information loss, essentially reverting to global MIP images. It can be observed that the model achieves optimal performance when three slices of local MIP images are used. This approach effectively reduces information loss while minimizing interference during fusion with the backbone network. This method ensures a balanced presentation of cartilage structures and surrounding tissues, resulting in more precise segmentation results.

(2) Different Feature Embedding Methods: Another key challenge is how to embed local and global MIP feature maps into the V-Net. We compared four different embedding methods:

(a) L: V₁ V₂ G: V₃ V₄ represent embedding local MIP features into V₁ V₂ and global MIP features into V₃ V₄;

(b) L: V₃ V₄ G: V₁V₂ represent embedding local MIP features into V₃V₄ and global MIP features into V₁V₂;

(c) L: V₁V₂V₃V₄ represent embedding local MIP features into V₁V₂V₃V₄;

(d) G: V₁V₂V₃V₄ represent embedding global MIP features into V₁V₂V₃V₄;

From Table 4, we can see that the model performs best when adopting the (a). This is attributed to the fact that local MIP feature maps focus more on shape and boundary information, while global MIP feature maps emphasize positional information and the overall morphology of the cartilage. Embedding local MIP feature maps into the shallow features V₁V₂ effectively supplements the boundary information lost in the middle cartilage, ensuring cartilage continuity. Embedding global MIP feature maps into the deep semantic features V₃V₄ enhances cartilage localization and reduces morphological deficiencies in the cartilage structure.

In knee MRI, cartilage is wrapped by other tissues, which are very similar in their morphology performance, and the cartilage occupies a small proportion in the whole image, which can easily cause the loss of boundary information during convolution and downsampling. The above phenomenon may lead to the problem of cartilage under-segmentation or even cartilage breakage in the final mask. To address the above problems, UVNet can compensate for missing boundaries by obtaining additional shape and position information from the MIP feature maps.

Figure 5 shows examples of cartilage segmentation on the OAI-ZIB and SKI-10 datasets by the proposed and comparative methods. It can be observed that our method shows smoother and more continuous cartilage with an overall morphology closer to the ground truth. As shown in the second and fourth rows of Fig. 5, TC tends to be under-segmented in the results of the baseline network and the other compared methods, while the segmentation loss of TC is compensated in our proposed method. Similarly, in the first, and third rows of Fig. 5, the cartilage mask obtained from the baseline network performs poorly, and some of the cartilage is even broken, whereas the UVNet corrects this defect and ensures the integrity of the cartilage morphology. It can be concluded that this is mainly attributed to the fact that the local and global MIP images are able to replenish the boundary information lost by downsampling, enhance the cartilage’s morphological representation in the whole image, and make the cartilage’s boundary more continuous. Our method can well guarantee the integrity of cartilage segmentation and reduce the possibility of cartilage breakage in the final segmentation mask.

In Section 3.2, we explain how to combine the morphological representation of knee cartilage to extract MIP images as an information supplement. In Section 3.3, it is explained how they can be fused with the hierarchical features of V-Net and added to the training. Figure 6 demonstrates the advantages of our proposed method by visualizing the MIP feature map and the feature fusion process. The specific fusion process is done by the feature fusion module, and these representative feature maps are obtained by averaging feature maps of different channels. It can be observed that after one downsampling, the cartilage morphology of the feature map (c) becomes blurred and the boundary distinguishability is relatively poor, which will affect the accuracy of the subsequent segmentation. The fusion of the MIP feature map (b) with (c), which contains additional information, can supplement the lost cartilage boundary information in the feature map (c) and give the cartilage a more complete morphology. We can see that the boundary of cartilage in the fused feature map (d) is more recognizable, and the demarcation line between cartilage and bone as well as cartilage and other tissues is clearer. After color filling, we can observe this more clearly in (e) and (f). The richer expression of the information will be very helpful for subsequent segmentation.

OPEN IN VIEWER

Fig. 5

The illustration of two sampled segmentation results of proposed method and the compared methods 3D U-Net, 3D V-Net, U-Net and U-Net++ on OAI-ZBI and SKI-10 datasets. The cartilage structure of the knee joint is described as: femoral cartilage (dark gray), tibial cartilage (light gray).

OPEN IN VIEWER

Fig. 6

Visualization of the feature map fusion process. (a) the MIP image, (b) the MIP feature map, (c) the feature map of V-Net after one downsampling, (d) the hierarchical features fused with the MIP feature, (e) Color fill for (c), (f) Color fill for (d).