Enhancing teeth segmentation using multifusion deep neural net in panoramic X-ray images

Abstract

BACKGROUND:

Precise teeth segmentation from dental panoramic X-ray images is an important task in dental practice. However, several issues including poor image contrast, blurring borders of teeth, presence of jaw bones and other mouth elements, makes reading and examining such images a challenging and time-consuming task for dentists. Thus, developing a precise and automated segmentation technique is required.

OBJECTIVE:

This study aims to develop and test a novel multi-fusion deep neural net consisting of encoder-decoder architecture for automatic and accurate teeth region segmentation from panoramic X-ray images.

METHODS:

The encoder has two different streams based on CNN which include the conventional CNN stream and the Atrous net stream. Next, the fusion of features from these streams is done at each stage to encode the contextual rich information of teeth. A dual-type skip connection is then added between the encoder and decoder to minimise semantic information gaps. Last, the decoder comprises deconvolutional layers for reconstructing the segmented teeth map.

RESULTS:

The assessment of the proposed model is performed on two different dental datasets consisting of 1,500 and 1,000 panoramic X-ray images, respectively. The new model yields accuracy of 97.0% and 97.7%, intersection over union (IoU) score of 91.1% and 90.2%, and dice coefficient score (DCS) of 92.4% and 90.7% for datasets 1 and 2, respectively.

CONCLUSION:

Applying the proposed model to two datasets outperforms the recent state-of-the-art deep models with a relatively smaller number of parameters and higher accuracy, which demonstrates the potential of the new model to help dentists more accurately and efficiently diagnose dental diseases in future clinical practice.

Keywords

Image segmentation teeth segmentation feature fusion X-ray dental images

1 Introduction

In the oral field, analysing dental images is crucial as it provides information about the patient’s oral conditions and helps in drawing better treatment plans. Teeth segmentation is one of the prominent tasks for analysing tooth status in dental imaging [1]. The segmented dental images convey essential information for monitoring teeth development, detecting dental structures and location of teeth for teeth numbering and implant planning [2, 3]. Another application of teeth segmentation is in the field of forensics to identify personal details like age and gender based on dental features [4, 5].

Among many dental imaging modalities, panoramic x-ray images are mostly utilized for diagnosing dental diseases. Dentists examine these X-ray images by visualizing them to determine the presence of various dental problems like cavities, fractured teeth, gum diseases, bone abnormalities and oral cancer [6 –8]. Analysing panoramic X-ray images manually is difficult and time-consuming, requiring expertise and experience. Even for experts examining X-ray images is more challenging because such images suffer from low contrast at boundaries, the noise produced by machines, overlapped images of teeth etc. thus may lead to incorrect judgements. Also, the presence of jaw and nasal bones makes this task difficult for dentists. Thus, to help young interns and dentists to reduce their fatigue workload and to enhance their analysing skills, accurate and automatic segmentation of dental x-rays is required, which would be very beneficial in the dentistry community.

Many conventional-learning methods have been used for teeth segmentation, these methods are based on thresholds [9], boundaries [10], regions [11, 12], and clusters [13]. Additionally, techniques based on morphological operators [14], template matching [15], active contours [16] and level-set [17] methodologies are also employed for automatically segmenting teeth regions. Binary support vector machines [18] with handcrafted features were also used for the segmentation of teeth. However, most of the classical methods depends on the careful identification and extraction of relevant features which directly affects the performance of these methods. Apart from it, when evaluated on small datasets, these techniques produce good results but perform poorly on large datasets.

With the introduction of deep learning in medical image analysis field, the limitations of traditional approaches for dental image segmentation can be greatly enhanced. The deep learning approaches have an advantage over traditional methods as these approaches has ability to learn features straight from unprocessed data instead of using hand-designed features. Several image segmentation models have been proposed are based on convolutional neural networks of which U-Net [19] and SegNet [20] have demonstrated impressive performance in the field of medical image segmentation, including dentistry.

Yang et al. [21] utilized conventional CNN as the backbone and proposed a pipeline for automatic analysis to diagnose tooth diseases using a small dataset having 196 periapical images. A couple-shaped model with the deep neural net was presented by Wirtz et al. [22] for automatic and robust segmentation of each tooth on low-quality dental panoramic radiographs. Due to insufficient data and shape inconsistencies, their approach results in low segmentation accuracy. To show the efficacy of transfer learning in segmenting dental radiographs Caylak et al. [23] employed the pre-trained Inception-ResNet-v2 and pre-trained U-Net models on small datasets. Only 131 dental panoramic radiographs were used. Nishitani et al. [24] presented a teeth segmentation model using U-Net on 162 panoramic images. This method focuses on the hybrid loss function to train U-Net architecture for tooth edges. However, the drawback of this method is suboptimal hyperparameters tuning and the need for optimization of edge width.

Silva et al. [25] proposed the standard dental dataset of panoramic x-ray images which consists of 1500 images. The authors implemented some traditional methods to segment dental images. Additionally, the Mask-RCNN deep neural network was deployed on this dataset in different categories of this dataset. The results obtained were not up to the mark thus leaving scope for other neural networks to be evaluated on this dataset. Jader et al. [26] used mask regional convolutional networks with Resnet-101 as a backbone for instance segmentation on 1500 panoramic radiographs. The main objective is to detect individual teeth or missing teeth. The limitation of this approach is that it lacks in capability for segmenting mouth and teeth components due to the presence of tooth appliances. Oktay et al. [27] and Pinheiro et al. [28] also used the Mask RCNN for segmenting individual teeth for tooth numbering in panoramic x-ray images. These approaches suffer because of the presence of dental implants and the overlapping of teeth. These models are utilized for instance segmentation in which the focus is on segmenting individual teeth which is a complex and laborious task when compared to semantic segmentation as it requires intensive and careful labelling. In this study, the focus is on semantic segmentation.

Modified U-Net was presented by Koch et al. [29] for accurate tooth segmentation. The authors made patches of original images and combines several techniques to improve segmentation performance. The annotated and patches of images were used which hampers model’s performance. Zhao et al. [30] presented a two-stage attention model based on CNN for accurate tooth segmentation. To locate the tooth area both global and local features were extracted using attention modules in first stage. A fully convolutional network was employed to segment dental region in second stage. The model has approximately 78 million trainable parameters and was evaluated on 1500 panoramic radiographs. The limitation of this method is in handling high number of trainable parameters.

Cui et al. [31] investigated the generative adversarial network with certain conditions on dental dataset comprises 1500 panoramic images for tooth segmentation. This method is highly dependent on annotation which compromises genuine feature extraction. Lin et al. [32] presented a lightweight neural network strategy using knowledge distillation for segmenting dental radiographs to deploy on edge devices. 1321 resized dental X-ray images were utilized to measure the performance. Model designed was lightweight but needs significant improvement in terms of dice and intersection over union (IoU) score.

Panetta et al. [33] developed a new benchmarking multimodal dental database consisting of 1000 panoramic x-ray images. Authors implemented recent state-of-the-art segmentation techniques with image enhancement methodologies to evaluate the dataset. The deep models consisting of Atrous net are yet to be explored on this dataset.

Vision-based automatic dental segmentation techniques employing both traditional and deep learning models have been presented in the literature. Deep learning techniques performed better as compared to traditional methods. These techniques are mainly based on conventional CNN, Mask RCNN and modified U-Net architecture for dental image segmentation. Thus, leaving scope to explore different types of CNN specially Diconvolutional (Atrous Net) layers for accurate and automatic dental panoramic X-ray image segmentation. Models based on Mask RCNN are generally used for instance segmentation while in this study, the focus is on the semantic segmentation of dental panoramic X-ray images which helps provide more clarity of tooth structure for better analysis. The deep models especially based on the attention mechanism possess a huge number of trainable parameters and takes longer training and testing time for model since there is a trade-off between accuracy and trainable model parameters. A lightweight deep model was proposed but it suffers with performance on dice score and IoU score which are important evaluation criteria for segmentation.

Thus, to address above mentioned issues, in this study a multi-fusion deep neural network with less trainable parameters is presented for precise and automatic segmentation of teeth region from dental panoramic X-rays. Following is the summary of contributions:

A novel multistage deep fusion model is presented for accurate and automated segmentation of teeth region. The encoder module encapsulates the information gathered from conventional CNN stream features and Dicnovolutional CNN (Atrous CNN) stream features and fuses them at each stage to contain teeth-rich feature information for accurate and sharp teeth region segmentation.

Dual types of skip connections, short-type skip connection and long-type skip connection are presented to store detailed information on teeth region between the encoder and decoder.

The hybrid loss function comprised of binary cross entropy loss and dice loss is proposed to improve the proposed model’s performance.

The proposed deep model is assessed on two different datasets containing 1500 and 1000 panoramic X-ray images and surpasses the state-of-the-art method in terms of IoU, dice score and accuracy while requiring very less trainable parameters for both datasets.

2 Model and methodology

2.1 Overview

The proposed multistage fusion deep model primarily concentrates on the automatically segmenting the entire teeth region from dental panoramic radiographs. The proposed model is based on encoder-decoder architecture. The encoder module is responsible for encoding texture semantic information of dental region by fusing information gathered from two different CNN-based streams i.e., conventional CNN and Atrous (Diconvolutional) Net at each step. Two types of skip connections Long Skip and Short skips are designed to preserve the low-level features of teeth which help in the reconstruction of the segmented teeth region map at the decoder module. The training of the proposed multistage fusion deep net is done in an end-to-end manner.

2.2 Detailed architecture

The major components of the proposed network are convolutional-dilated convolutional based encoder, deconvolutional decoder and dual type of skip connections. The proposed model is trained in an end-to-end way using hybrid loss function. The detailed architecture of the proposed multi-fusion deep model is shown Fig. 1.

Encoder: The encoder part of the presented model is composed of two different CNN-based streams. Each stream of the encoder receives the same input in the form of original panoramic x-ray image. The first stream uses simple convolutional layers to capture essential dental information from panoramic radiographs. This stream comprises of six convolution layers having different kernel sizes followed by batch normalization and uses an activation function ReLu. In this stream three Max pooling layers of dimensions [2×2] are used to minimise the spatial dimensionality of image. Second stream of the encoder part contains six dilated-convolutional (Diconvolution) layers having different dilation rates. These layers also use activation function ReLu with batch normalization. Here also, three Max pooling layers of dimensions [2×2] are used to downsample feature maps. Dilated convolution has the advantage of capturing contextual semantic information of teeth region in bigger receptive field. Additionally, it suppresses non-relevant background region without increase in the number of network parameters. The skip connections are also designed in the encoder module. Short-type skip connections are utilized in both streams of encoder that is in simple CNN stream as well as dilated convolution (Atrous) net stream to enhance the training of proposed deep network. Long-type skip connection are present from an encoder part to the decoder part of the network. To exploit the features of multi-CNN, feature maps obtained from each convolution layer of the CNN stream and dilated convolutional layer of the Atrous Net stream are fused by using concatenation operation. These fused feature maps are provided as input to another stream called fused stream which includes three simple convolutional layers having filter size [3×3] with ReLu as an activation function followed by batch normalization layer. The feature maps obtained from all streams are fused at each stage while maintaining symmetry in the encoder. This enables a network to learn more context from feature maps of dental panoramic images. Finally, all the features of each stream are fused and are passed as input to the decoder.

Fig. 1

The detailed architecture of the proposed model.

Decoder: The features extracted by the encoder are in compressed form and are the representation of original dental panoramic x-ray images. These compressed features are then fed to the decoder to reconstruct the final segmented teeth image. The decoder accumulates the feature information from various layers and concatenates encoder-extracted features with upsampled feature maps. The symmetry is maintained throughout the encoder and the decoder module. The decoder includes eight deconvolution layers with different kernel sizes. The first seven layers utilizes activation function ReLu and batch normalization. The upsampling layers of dimensions [2×2] are added to increase the spatial dimensions which help in reconstructing the original size of the image. The final layer in the decoder is a deconvolution layer of channel 1 and has a [1×1] kernel size and uses the sigmoid activation function for pixel-wise classification. The training of proposed model is completed in an end-to-end way.

Skip Connections: There are two types of skip connections presented in the proposed deep network. The first are the Short-type skip connections and the second are Long-type skip connections. The objective of these skip connections is to enhance information flow between feature maps. The short-type skip connections are proposed in the encoder module while the Long-type skip connections are between the encoder module and decoder module. The short-type skip connections are utilized in conventional CNN and Atrous net streams to reuse the dental semantic features obtained from earlier layers in the other layers. Additionally, they help in faster convergence of the learning process by stabilizing gradient updates and well distribution of parameter updates. Down-sampling in the encoder part results in a loss of spatial structural information due to different types and shapes of teeth. Thus, to preserve low-level spatial features Long-type skip connections are introduced which originate from an encoder and meet the decoder to enhance the restructuring of the segmented teeth map.

Hybrid Loss Function: Two different loss functions are employed, to optimize the proposed model. Binary cross-entropy (BCE) loss is the first loss function, which is based on distribution loss and is commonly utilized in binary classification and segmentation. BCE is expressed as: ${Loss}_{BCE} = - ({Gt}_{i} (log {Pr}_{i} + (1 - {Gt}_{i}) log (1 - {Pr}_{i}))$ (1) where Gt_i ∈ {0, 1} is considered as input image’s ground truth and Pr _i ∈ {0, 1} represents the predicted segmentation map pixel from input image.

The second loss function is a dice loss inspired by the dice coefficient score is based on region loss and is utilized to refine the overlapping between the two images. This loss is mostly utilized in optimizing medical images. It is described as follows: ${Loss}_{DICE} = 1 - \frac{2 \Pr_{i} {Gt}_{i}}{\Pr_{i} + {Gt}_{i}}$ (2) where Gt_i ∈ {0, 1} is considered as input image’s ground truth and Pr _i ∈ {0, 1} represents the predicted segmentation map pixel from input image.

To obtain clear teeth segmented maps a hybrid loss function is presented and is defined as: ${Loss}_{MODEL} = λ . {Loss}_{BCE} + (1 - λ) . {Loss}_{DICE}$ (3) where Loss_BCE is binary cross entropy loss, mostly employed by deep learning-based techniques to classify pixels or data into two categories. Loss_DICE is dice loss which measures the quality of an image. λ is a positive constant range between 0 and 1. Various experiments with different value λ of ranging from 0.1 to 0.9 was carried out and is empirically set to 0.8.

The BCE loss is considered as a pixel-by-pixel loss that is utilized to minimise the gap between each pixel. The dice loss is considered as a loss of foreground and becomes zero as the prediction of foreground increases assisting the proposed network to emphasize on the foreground. Both loss functions have different benefits the combination of BCE and Dice Loss offers a balanced approach for segmentation tasks, as BCE loss helps in class imbalance and captures global context which ensures overall correct pixel predictions whereas Dice loss is effective in measuring similarity which leads to better localization accuracy also it focusses on capturing finer details. Thus, utilizing them jointly optimizes the efficacy of the proposed network.

3 Datasets and evaluation criteria

3.1 Datasets

The two-benchmark dataset are used to assess the proposed deep model. The first dataset was developed at Ivision Lab by Silva et al. [25] and composed of 1500 dental panoramic x-ray images which are categorized into 10 distinct categories based on missing teeth, dental appliances, dental implants and restoration. For training, validation and testing, dataset is segregated in 3 parts with a split ratio of 8:1:1 that is there are 1200 randomly selected images in the training set and remaining 300 images are equally divided between validation set and test set. The original size of the image in this dataset was 1991×1127 which is resized to 512×512 for better memory utilization. The second benchmark dataset used to assess the suggested model is called Tufts dental database prepared by Panetta et al. [33] is openly available for research purposes and has 1000 new dental panoramic x-ray images. This dataset is also segregated in 3 parts with similar split ratio having 800 randomly selected images in training set while 100 image each in test and validation set. The size of the original x-ray image in this dataset is 1615×840 pixels which is downsized to again 512×512 pixels. Both datasets were developed in different conditions with different machines. The two datasets were used to see the generalizability of the model. The sample dental panoramic radiography images of both datasets are shown in Fig. 2.

Fig. 2

Samples of dental panoramic x-ray images of both datasets (a) Sample of dental panoramic radiograph and its ground truth from dataset 1. (b) Sample of dental panoramic radiograph and its ground truth from dataset 2.

3.2 Evaluation criteria

The proposed deep model is assessed on a variety of quantitative metrics including accuracy, recall, precision, dice coefficient and IoU. The dice coefficient score is broadly utilized to assess the performance of image segmentation models specially in medical imaging, it measures the similarity of predicted segmented map and the ground truth map. The dice coefficient is defined as: $dice_coef = \frac{2 . | P_{SP} \cap G_{SP} |}{| P_{SP} | + | G_{SP} |}$ (4) where P_SP is the predicted segmented pixels of teeth region and G_SP is segmented pixels of ground trurth.

Another evaluation metric commonly used for the segmentation of medical images is Jaccard index or IoU and is defined as: $IoU = \frac{| P_{SP} \cap G_{SP} |}{| P_{SP} \cup G_{SP} |}$ (5) where P_SP is the predicted segmented pixels of teeth region and G_SP is the segmented pixels of ground truth.

The image segmentation is regarded as pixel-level classification task therefore common classification metrics accuracy, recall and precision are also employed to assess the performance of the suggested deep model. These are defined as follows: $accuracy = \frac{(T_{TP} + T_{TN})}{T_{TP} + F_{FP} + T_{TN} + F_{FN}}$ (6) $precision = \frac{T_{TP}}{T_{TP} + F_{TP}}$ (7) $recall = \frac{T_{TP}}{T_{TP} + F_{FN}}$ (8) where T_TP is the number of correctly forecasted pixels of teeth region, F_FP is the number of wrongly forecasted pixels of teeth region, T_TN is the number of correctly forecasted pixels of background region, F_FN represents the number of wrongly forecasted pixels of background region.

4 Experiments and result analysis

4.1 Experimental setup

The presented deep model is implemented in an end-to-end way on four NVIDIA GeForce GTX 1080i GPUs using Pytorch framework in python. The learning rate, batch size and the number of epochs is set to 0.0001, 16 and 200 respectively. To avoid overfitting, the early stopping method is adopted with the patience parameter set to 4 that is training will be halted if there will be no change in training loss for 4 consecutive epochs. It can be inferred that the proposed network converges at approximately 60 epochs. The training and validation loss for every epoch is plotted and is shown in Fig. 3. Further, the hybrid loss function is optimized by using RMSprop optimizer.

Fig. 3

Losses of proposed model for both datasets. (a) The plot demonstrates the training loss and validation loss for dataset 1 it seems the model converges approximately at 55th epoch. (b) This plot demonstrates training loss and validation loss for dataset 2 which seems to be converged at epoch 60.

4.2 Result analysis

The proposed model’s performance is compared with recent existing deep learning models all of which can be utilized in segmenting dental panoramic x-ray images. These deep models are SegNet [20], UNet [19], BiseNet [34], CENet [35], Unet++ [36] and Nanonet [37] which were applied to the same benchmark dental image datasets. To have a fair comparison, the same parameter setting has been used and datasets are split in the same ratio for training, testing and validation set. The assessment of the models is done on five different metrics, number of parameters and two different panoramic radiographs datasets. For dataset 1, accuracy, precision, recall, IoU and dice score of the proposed model are 97%, 96.4%, 90.6, 91.1% and 92.4% respectively with only 0.380 million (M) parameters. On dataset 2 the proposed model attains 97.7% accuracy, 95.4 % precision, 92.8% recall, 90.2% IoU and 90.7% dice coefficient while the number of parameters is same.

The quantitative comparison results with other deep models for dataset 1 is represented in Table 1 and for dataset 2 is represented in Table 2. It is observed that the presented deep model completely outperforms the state-of-the-art methods on all evaluation metrics except the number of parameters for both datasets. The presented model is better than all the deep segment state-of-the-art methods in terms of parameters except Nanonet which has only 0.235 trainable parameters. Though Nanonet has shown good performance while having a smaller number of parameters its dice coefficient and IoU score are quite less compared to the proposed model which are considered important segmentation metrics.

Table 1
Performance summary of the proposed model compared with deep segment state-of-the-art methods for Dataset 1

Accuracy Precision Recall IoU Dice Coefficient Parameters (M)

SegNet [20] 96.1 92.1 88.6 82.4 90.3 29.44

UNet [19] 96.2 94.2 86.8 82.4 90.3 31.04

BiseNet [34] 92.7 92.5 69.6 78.7 79.4 23.06

CENet [35] 96.7 93.3 90.2 84.7 91.7 38.69

Unet++ [36] 95.1 92.5 82.6 77.4 87.3 9.20

NanoNet [37] 96.6 95.0 82.8 89.9 91.3 0.235

Proposed Model 97.0 96.4 90.6 91.1 92.4 0.380

	Accuracy	Precision	Recall	IoU	Dice Coefficient	Parameters (M)
SegNet [20]	96.1	92.1	88.6	82.4	90.3	29.44
UNet [19]	96.2	94.2	86.8	82.4	90.3	31.04
BiseNet [34]	92.7	92.5	69.6	78.7	79.4	23.06
CENet [35]	96.7	93.3	90.2	84.7	91.7	38.69
Unet++ [36]	95.1	92.5	82.6	77.4	87.3	9.20
NanoNet [37]	96.6	95.0	82.8	89.9	91.3	0.235
Proposed Model	97.0	96.4	90.6	91.1	92.4	0.380

Table 2

Performance summary of the proposed model compared with deep segment state-of-the-art methods for Dataset 2

	Accuracy	Precision	Recall	IoU	Dice Coefficient	Parameters (M)
SegNet [20]	96.4	88.5	92.1	88.2	89.3	29.44
UNet [19]	96.5	89.7	91.8	88.3	88.5	31.04
BiseNet [34]	96.7	83.4	90.9	86.6	82.0	23.06
CENet [35]	96.7	93.3	90.2	84.7	91.7	38.69
Unet++ [36]	97.1	86.2	90.3	78.9	88.2	9.20
NanoNet [37]	97.4	89.7	91.2	90.0	90.4	0.235
Proposed Model	97.7	95.4	92.8	90.2	90.7	0.380

The visual results of the proposed model are demonstrated in Fig. 4 for dataset 1 and Fig. 5 for dataset 2. It can be observed that UNet, SegNet, BiseNet, Unet++ and CEnet are prone to misclassify the correct teeth pixels with jaws bone and other facial bone pixels whereas the proposed method correctly identifies segmented teeth region. It can be seen that BiseNet is comparatively weak in dealing with low-contrast images. SegNet, CEnet, and Unet++ suffer in predicting the tooth boundaries while NanoNet seems to perform better in detecting tooth boundaries but shows the teeth structure slightly thicker while the proposed method has predicted the clear and smooth teeth region segmented map by suppressing the other background details of mouth.

Fig. 4

Comparison of visual results of proposed model with different segmentation based deep models on Dataset 1. The first two rows represent original image and its corresponding ground truth while the remaining rows represents the predicted segmented teeth map by different methods including the proposed model.

Fig. 5

Comparison of visual results of proposed model with different segmentation based deep models on Dataset 2. The first two rows represent original image and its corresponding ground truth respectively while rest shows segmented teeth region mask predicted by various deep models.

5 Ablation study

The ablation study is carried out to verify the efficacy of the components of the suggested deep model. The experiments are conducted with similar parameter settings but with different components on the same datasets. The taring and validation loss of each module is shown in Figs. 6 and 7 for dataset1 and dataset 2 respectively from which one can infer that all the modules are converged at epoch 60. The assessment metrics used are precision, accuracy, recall, IoU and dice coefficient. Three different modules are developed from the proposed model based on skip connections and are as follows:

Fig. 6

Training vs Validation loss for Dataset 1. (a) This plot represents the training and validation loss for module without skip connections. (b) The loss for module 2 having long skip connections. (c) This plot shows loss for the third module having only short skip connections. (d) This represents the training and validation loss for proposed model. All modules including proposed model is converged at epoch 60.

Fig. 7

Training vs Validation loss for Dataset 2. (a) This plot represents the loss for first module with no skip connections. (b) The training and validation loss for second module having long skip connections is shown. (c) This plot shows loss for the third module having only short skip connections. (d) This represents the training and validation loss for proposed model. All modules including proposed model is converged at epoch 60.

M1-WS: Module1-Without Skip connections. The first module is composed of the encoder and decoder part of the presented deep network but there are no skip connections present, neither in an encoder nor from an encoder to decoder.

M2-LS: Module2-Long Skip. This is the second module which consists of only long skip connections from an encoder to the decoder.

M3-SS: Module3-Short Skip. The third module contains only short skip connections which are present in both conventional CNN stream and Atrous net stream of the encoder module.

The effectiveness of each model is represented by the quantitative results are shown in Tables 3 and 4 for dataset 1 and dataset 2 as well as by the qualitative results shown in Fig. 8 1 and for dataset Fig. 9 for dataset 2. It can be inferred that without any type of skip connections, the dice score and IoU are less as compared to a network having only long skip connections. Modules without skip and with short skip connections have similar results except for precision which is high for networks with only short-type skip connection. The long-type skip connection module has the highest recall. The proposed model which includes both short-type skip connection and long-type skip connection has performed superior to each individual module in all evaluation metrics. The visual results demonstrate the proposed model’s segmented teeth map has better visibility than other modules.

Table 3

Ablation experiment comparison of different modules on Dataset 1

	Accuracy	Precision	Recall	IoU	Dice Coefficient
M1-WS	96.4	94.4	85.5	89.2	90.1
M2-LS	96.7	88.6	96.0	85.5	92.2
M3-SS	96.4	96.3	85.4	89.1	90.5
Proposed Model	97.0	96.4	90.6	91.1	92.4

Table 4

Ablation experiment comparison of different modules on Dataset 2

	Accuracy	Precision	Recall	IoU	Dice Coefficient
M1-WS	97.3	88.8	81.5	87.7	87.9
M2-LS	97.7	88.9	92.4	82.8	90.6
M3-SS	97.4	94.9	83.1	88.6	88.3
Proposed Model	97.7	95.4	92.8	90.2	90.7

Fig. 8

Comparison of visual results from ablation experiment on Dataset 1. The first two rows represent original image and its ground truth respectively. The next three rows demonstrate the predicted segmented map by different modules while the last row represents the predicted segmented map by proposed model.

Fig. 9

Comparison of visual results from ablation experiment on Dataset 2. The first two rows represent original image and its ground truth respectively. The next three rows show the predicted segmented map by different modules while the last row represents the predicted segmented map by proposed model.

The experiments with various loss settings are executed on the suggested model to assess the efficacy of the presented hybrid loss function. The comparison results for dataset 1 are shown Table 5 and for dataset 2 are shown in Table 6. It can be inferred that BCE loss has a higher performance in segmentation dice score, IoU and recall, and performs similarly to Dice loss in terms of segmented pixel accuracy while dice loss has a higher performance in terms of precision. When both the loss functions are joint to form the hybrid loss function there is a dramatic improvement in IoU and dice score.

Table 5

Comparison of ablation study with different loss functions on Dataset 1

	Accuracy	Precision	Recall	IoU	Dice Coefficient
BCE	96.8	95.4	88.4	90.4	91.8
DICE	96.2	96.6	84.1	81.6	89.9
Proposed Model	97.0	96.4	90.6	91.1	92.4

Table 6

Comparison of ablation study with different loss functions on Dataset 2

	Accuracy	Precision	Recall	IoU	Dice Coefficient
BCE	97.4	93.9	86.0	89.4	89.7
DICE	96.9	96.7	76.5	85.6	85.4
Proposed Model	97.7	95.4	92.8	90.2	90.7

6 Conclusion

This article proposed a novel multi-fusion deep model based on encoder-decoder structure to obtain clear and accurate teeth region segmented from dental panoramic radiographs. The encoder of proposed model comprised of two different CNN based architectures with fusion of features at each step for extracting diverse features and contextual rich information. The two types of skip connections were proposed in encoder and from encoder to decoder to regain dental features using different connections. The decoder is made-up of deconvolutional layers to restructure the image. The experiments were executed to demonstrate efficacy of the presented model on two different datasets. The results obtained shows that the presented model outperforms the state-of-the-art deep segmentation methods with requiring relatively a smaller number of trainable parameters. In future work, weaker boundaries of the tooth root would be determined and enhanced using attention networks.

Conflict of interest

The authors declare that they have no conflict of interest.

References

Kong

, Xiong

, Zhang

, Fu

, Zhang

, Weng

, et al., Automated maxillofacial segmentation in panoramic dental x-ray images using an efficient encoder-decoder network, IEEE Access 8 (2020), 207822–207833.

Zheng

, Yan

, Setzer

F.C.

, et al., Anatomically constrained deep learning for automating dental CBCT segmentation and lesion detection, IEEE Trans Autom Sci Eng 18(2) (2020), 603–614.

T.H.

, Lian

, Lee

, et al., Two-stage mesh deep learning for automated tooth segmentation and landmark localization on 3D intraoral scans, IEEE Trans Med Imaging 41(11) (2022), 3158–3166.

Milošević

, Vodanović

, Galić

, Subašić

, Automated estimation of chronological age frompanoramic dental X-ray images using deep learning, Expert SystAppl 189 (2022), 116038.

Nomir

, Abdel-Mottaleb

, Fusion of matching algorithms for human identification using dental X-ray radiographs, IEEE Trans Inf Forensics Secur 3(2) (2008), 223–233.

Chandrashekar

, AlQarni

, Bumann

E.E.

, Lee

, Collaborative deep learning model for tooth segmentation and identification using panoramic radiographs, Comput Biol Med 148 (2022), 105829.

Kumari

A.R.

, Rao

S.N.

, Reddy

P.R.

, Design of hybrid dental caries segmentation and caries detection with meta-heuristic-based ResneXt-RNN, Biomed Signal Process Control 78 (2022), 103961.

Nader

, Smorodin

, De La Fourniere

, et al., Automatic teeth segmentation on panoramic X-rays using deep neural networks, In: 2022 26th International Conference on Pattern Recognition (ICPR), 2022, pp. 4299–305.

Ajaz

, Kathirvelu

, Dental biometrics: Computer aided human identification system using the dental panoramic radiographs, In: 2013 International Conference on Communication and Signal Processing, 2013, pp. 717–721.

10.

Hasan

M.M.

, Ismail

, Hassan

, Yoshitaka

, Automatic segmentation of jaw from panoramic dental X-ray images using GVF snakes, In: 2016 World Automation Congress (WAC), 2016, pp. 1–6.

11.

Modi

C.K.

, Desai

N.P.

, A simple and novel algorithm for automatic selection of ROI for dental radiograph segmentation, In: 2011 24th Canadian Conference on Electrical and Computer Engineering (CCECE), 2011, pp. 329 504–507.

12.

Singh

V.P.

, Srivastava

, Automated and effective content-based image retrieval for digital mammography, J Xray Sci Technol 26(1) (2018), 29–49.

13.

Tuan

T.M.

, A cooperative semi-supervised fuzzy clustering framework for dental X-ray image segmentation, Expert Syst Appl 46 (2016), 380–393.

14.

Said

E.H.

, Nassar

D.E.M.

, Fahmy

, Ammar

H.H.

, Teeth segmentation in digitized dental X-ray films using mathematical morphology, IEEE Trans Inf forensics Secur 1(2) (2006), 178–189.

15.

Poonsri

, Aimjirakul

, Charoenpong

, Sukjamsri

, Teeth segmentation from dental x-ray image by template matching, pp, In: 2016 9th Biomedical Engineering International Conference (BMEiCON) (2016), 1–4.

16.

Shah

, Abaza

, Ross

, Ammar

, Automatic tooth segmentation using active contour without edges, pp, In: 2006 Biometrics Symposium: Special Session on Research at the Biometric Consortium Conference (2006), 1–6.

17.

Rad

A.E.

, Rahim

M.S.M.

, Norouzi

, Digital dental X-ray image segmentation and feature extraction, TELKOMNIKA Indones J Electr Eng 11(6) (2013), 3109–3114.

18.

Prakash

, Gowsika

, Sathiyapriya

, An identification of abnormalities in dental with support vector machine using image processing, In: Emerging Research in Computing, Information, Communication and Applications: ERCICA 2015, 2015, pp. 29–40.

19.

Ronneberger

, Fischer

, Brox

, U-net: Convolutional networks for biomedical image segmentation, In: International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015, pp. 234–241.

20.

Badrinarayanan

, Kendall

, Cipolla

, Segnet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Trans Pattern Anal Mach Intell 39(12) (2017), 2481–2495.

21.

Yang

, Xie

, Liu

, et al., Automated dental image analysis by deep learning on small dataset, In: 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), 2018, pp. 492–497.

22.

Wirtz

, Mirashi

S.G.

, Wesarg

, Automatic teeth segmentation in panoramic X-ray images using a coupled shape model in combination with a neural network, In: International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018, pp. 712–719.

23.

Caylak

, Yetik

I.S.

, Culhaoglu

, et al., Automated dental panoramic image segmentation using transfer learning based CNNs, In: 2022 29th International Conference on Systems, Signals and Image Processing (IWSSIP), 2022, pp. 1–4.

24.

Nishitani

, Nakayama

, Hayashi

, et al., Segmentation of teeth in panoramic dental X-ray images using U-Net with a loss function weighted on the tooth edge, Radiol Phys Technol 14(1) (2021), 64–69.

25.

Silva

, Oliveira

, Pithon

, Automatic segmenting teeth in X-ray images: Trends, a novel data set, benchmarking and future perspectives, Expert Syst Appl 107 (2018), 15–31.

26.

Jader

, Fontineli

, Ruiz

, et al., Deep instance segmentation of teeth in panoramic X-ray images, In: 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), 2018, pp. 400–407.

27.

Oktay

A.B.

, Gurses

, Detection, segmentation, and numbering of teeth in dental panoramic images with mask regions with convolutional neural network features, In: State of the Art in Neural Networks and their Applications, Elsevier, 2021, pp. 73–90.

28.

Pinheiro

, Silva

, Sobrinho

, et al., Numbering permanent and deciduous teeth via deep instance segmentation in panoramic x-rays, In: 17th International Symposium on Medical Information Processing and Analysis, 2021, pp. 95–104.

29.

Koch

T.L.

, Perslev

, Igel

, Brandt

S.S.

, Accurate segmentation of dental panoramic radiographs with U-NETS, In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), 2019, pp. 15–19.

30.

Zhao

, Li

, Gao

, et al., TSASNet: Tooth segmentation on dental panoramic X-ray images by Two-Stage Attention Segmentation Network, Knowledge-Based Syst 206 (2020), 106338.

31.

Cui

, Zeng

, Chong

, Zhang

, Toothpix: Pixel-level tooth segmentation in panoramic X-Ray images based on generative adversarial networks, In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), 2021, pp. 1346–1350.

32.

Lin

, Hao

, Liu

, et al.,Lightweight deep learning methods for panoramic dental X-ray image segmentation, Neural Comput Appl (2022), 1–12.

33.

Panetta

, Rajendran

, Ramesh

, et al., Tufts dental database: A multimodal panoramic X-ray dataset for benchmarking diagnostic systems, IEEE J Biomed Heal Informatics 26(4) (2021), 1650–1659.

34.

, Wang

, Peng

, et al., Bisenet: Bilateral segmentation network for real-time semantic segmentation, In: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 325–341.

35.

, Cheng

, Fu

, et al., Ce-net: Context encoder network for 2d medical image segmentation, IEEE Trans Med Imaging 38(10) (2019), 2281–2292.

36.

Zhou

, Rahman Siddiquee

M.M.

, Tajbakhsh

, Liang

,Unet++: A nested u-net architecture for medical image segmentation, In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, 2018, pp. 3–11.

37.

Jha

, Tomar

N.K.

, Ali

, et al., Nanonet: Real-time polyp segmentation in video capsule endoscopy and colonoscopy, In: 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS), 2021, pp. 37–43.