Automatic extraction of coronary arteries using deep learning in invasive coronary angiograms

Abstract

BACKGROUND:

Accurate extraction of coronary arteries from invasive coronary angiography (ICA) images is essential for the diagnosis and risk stratification of coronary artery disease (CAD).

OBJECTIVE:

In this study, a novel deep learning (DL) method is proposed for automatically extracting coronary arteries from ICA images.

METHODS:

A convolutional neural network (CNN) was developed with full-scale skip connections and full-scale deep supervisions. The encoder architecture was based on the residual and inception modules to obtain multi-scale features from multiple convolutional layers with different window shapes. Transfer learning was utilized to improve both the initial performance and learning efficiency. A hybrid loss function was employed to further optimize the segmentation model.

RESULTS:

The model was tested on a data set of 616 ICAs obtained from 210 patients, composed of 437 images for training, 49 images for validation, and 130 images for testing. The segmentation model achieved a Dice score of 0.8942, a sensitivity of 0.8735, a specificity of 0.9954, and a Hausdorff distance of 6.0794 mm; it could predict arteries for a single ICA frame in 0.2114 seconds.

CONCLUSIONS:

The results showed that our model outperformed the state-of-the-art deep-learning models. Our new method has great potential for clinical use.

Keywords

Coronary artery disease invasive coronary angiography image segmentation deep learning convolutional neural network

1. Introduction

Coronary artery disease (CAD) is a prevalent cardiovascular condition that affects the coronary arteries, leading to symptoms such as angina and acute myocardial infarction [1, 2, 3]. It is recognized as one of the leading causes of mortality worldwide, particularly in developing countries. According to data from the World Health Organization, cardiovascular diseases claim the lives of around 17.9 million people annually, with CAD being one of the most common types [4].

Invasive coronary angiography (ICA), which provides assessments of artery stenosis and plaque characteristics, is an essential tool for CAD evaluation and treatment [5]. The utilization of ICA in CAD diagnosis necessitates manual segmentation by specialized physicians, which is time-consuming and labor-intensive. Automated extraction of coronary arteries from ICAs plays a key role in the clinical practice of interventional cardiology [6]. However, the accurate extraction of the coronary arteries from ICAs is a challenging task for the following reasons: 1) complex noise caused by the non-uniform illumination; 2) poor signal-to-noise ratio; 3) uneven intensity; 4) semantic information confusion. Automatic extraction techniques of blood vessels can be broadly classified into two main categories: traditional image processing and deep learning-based methods. The former includes Gaussian or Gabor filter-based, model-based, and line tracking-based methods [7, 8, 9], which focus primarily on the vascular extraction of low noise and high contrast. However, these methods cannot achieve satisfactory results when dealing with ICAs with extremely complex noise. The latter mainly contains convolution neural network-based methods (CNNs), which have shown excellent feature-extracting performance. Yang designed a method that applied correspondence matching and CNN for automatic coronary artery extraction, and a Dice score of 0.8007 was reported [10]. Nasr-Esfahani reported that using patches of pixels to evaluate the ICAs and a CNN was designed to determine the arteries and background [11]. To further improve the segmentation accuracy, E. Nasr et al. used two CNNs to a combination of primary and secondary features and achieved a Dice score of 0.8151 [12]. Image segmentation based on CNN methods, such as U-Net and U-Net $++$ , has also made outstanding achievements [13, 14]. Huang et al. proposed a novel U-Net 3 $+$ [15] which is a full-scale connected U-Net for medical image segmentation. Zhao et al. proposed FP-U-Net $++$ by introducing the feature pyramid techniques into U-Net $++$ and reached a Dice score of 0.8899 [16]. Although existing methods have achieved positive results, the accuracy of artery segmentation can be further improved.

In this paper, we present a new method for coronary artery extraction from ICAs. Our method uses a network structure based on U-Net 3 $+$ [15], which explores sufficient information from full scales. We first considered the encoder architecture based on the residual and inception modules, then used transfer learning for promoting learning efficiency. For achieving excellent performance, a hybrid loss function that encourages network convergence and penalizes discrepancy was utilized in the training stage. We developed the deep learning framework using 616 ICAs obtained from 210 patients. Experimental results demonstrated that our DL model achieved an average Dice score of 0.8942 and predict arteries for a single ICA frame in 0.2114 seconds. Finally, we compared the segmentation performance between our model and state-of-the-art deep learning models.

2. Materials and methods

2.1 Patient data

Two hundred and ten patients (100 males and 110 females) who received ICA (616 images) were retrospectively analyzed. Institutional review board approval was obtained with no informed consent required for this HIPAA-compliant retrospective analysis. The acquired ICA images were 512 $\times$ 512 pixels per scan, with pixel spacing ranging from 0.200–0.390 mm.

2.2 Artery extraction

The model developed in this study was used to extract the coronary artery from ICAs (Fig. 1). The ICAs were fed into the deep learning (DL) network, and the output was arterial contours.

Figure 1.

Workflow of extracting coronary arteries from ICA images. (a) The input of ICA image. (b) Our deep learning models. (c) Predicted arterial contours.

Figure 2.

Comparison of arterial extraction models of the (a) U-Net; (b) U-Net $++$ .

2.2.1 Deep neural network architecture

The simplified overview of U-Net is illustrated in Fig. 2a. The low-level extracted feature maps from the encoder path are directly concatenated into the corresponding decoder path. The nodes $X^{i,j}$ represent the result of the convolutional layers, where $i$ is the index of the encoder and $j$ is the index of the decoder. The decoder part of the feature maps represented by $X^{i,j}({i=4-j})$ is computed in Eq. (1)

$\displaystyle X^{i,j}=\left\{\begin{array}[]{ll}F(D(X^{i-1,j}))&j=0\\ F([X^{i,0},U(X^{i+1,j-1})])&j>0\\ \end{array}\right.$ (1)

Where $F(\cdot)$ realizes the convolution operation followed by a batch normalization and a ReLU activation function [17]. The $[\cdot]$ denotes the concatenation operation. $D(\cdot)$ and $U(\cdot)$ represents the down-sampling operation and up-sampling operation, respectively.

U-Net $++$ is a modified solution of U-Net. U-Net $++$ upgrades the U-Net techniques by introducing nested and dense skip connections. As shown in Fig. 2b, what separates U-Net $++$ from the original U-Net is that the former has one encoder and multiple decoders and undergo a dense convolution layer. In U-Net $++$ , the decoder part of the feature maps represented by $X^{i,j}$ are computed in Eq. (2).

$\displaystyle X^{i,j}=\left\{\begin{array}[]{ll}F(D(X^{i-1,j}))&j=0\\ F([[X^{i,k}]^{j-1}_{k=0},U(X^{i+1,j-1})])&j>0\\ \end{array}\right.$ (2)

The decoding is not done directly from the encoding, and the number of layers in this convolution depends on the down-sampling layer corresponding to the coding direction [16].

The proposed method for image segmentation, unlike U-Net and U-Net $++$ , incorporates both encoder and decoder feature maps from the same scale in each decoder layer. This is achieved through the use of full-scale skip connections and deep supervisions, which are similar to those used in U-Net 3 $+$ . The encoder architecture in this approach is based on residual and inception modules, as depicted in Fig. 1. Instead of using a single convolutional layer as in the original U-Net 3 $+$ , the proposed method uses four different types of blocks, namely, Inception A block, Inception-ResNet-A block, Inception-ResNet-B block, and Inception-ResNet-C block, which are designed to capture multi-scale features to improve segmentation accuracy.

Figure 3.

(a) Architecture of the Inception A block. (b) Architecture of the Inception-ResNetA block. (c) Architecture of the Inception-ResNetB block. (d) Architecture of the Inception-ResNetC block.

Figure 3a shows the Inception A block, which consists of four branches. The first branch contains an average pooling layer followed by a convolutional layer. The other three branches consist of convolutional layers with different kernel size, such as 1 $\times$ 1, 3 $\times$ 3 and 5 $\times$ 5. Figure 3b depicts the structure of the Inception-ResNet-A block, which is divided into four branches. The first branch is directly output, the second branch is processed by a 2D convolution with a 1 $\times$ 1 kernel, the third branch is processed by 2D convolution with a 1 $\times$ 1 kernel and a 3 $\times$ 3 kernel, the fourth branch is processed by a 2D convolution with a 1 $\times$ 1 kernel and two 2D convolutions with 3 $\times$ 3 kernel. The outputs of the second, third, and fourth branches are concatenated and then processed by a 2D convolution with a 1 $\times$ 1 kernel before being concatenated with the output of the first branch.

Figure 3c shows the structure of the Inception-ResNet-B block, which is divided into three branches. The first branch is directly output, the second branch is processed by a 2D convolution with a 1 $\times$ 1 kernel, and the third branch is processed by a 2D convolution with a 1 $\times$ 1 kernel, 1 $\times$ 7 kernel, and 7 $\times$ 1 kernel. The outputs of the second and third branches are concatenated and then processed by a 2D convolution with a 1 $\times$ 1 kernel before being concatenated with the output of the first branch. Finally, Fig. 3d shows the structure of the Inception-ResNet-C block, which is also divided into three branches. The first branch is directly output, the second branch is processed by a 2D convolution with a 1 $\times$ 1 kernel, and the third branch is processed by a 2D convolution with a 1 $\times$ 1 kernel, 1 $\times$ 3 kernel, and 3 $\times$ 1 kernel. The outputs of the second and third branches are concatenated and then processed by a 2D convolution with a 1 $\times$ 1 kernel before being concatenated with the output of the first branch.

The full-scale deep supervision in our method is employed to supervise each scale. The side outputs of the decoder are fed into a plain 3 $\times$ 3 convolution layer and then up-sampled with a factor of 16, 8, 4 and 2, respectively. Moreover, there are converted into a probability map by the sigmoid function. The network weights are optimized by comparing the differences between the ground truth of all the side outputs and the arterial annotations. Finally, the OTSU algorithm [18] is employed to convert the probability map into binary segmentation results where a pixel value of 1 is represented as coronary artery, and 0 is described as background.

2.2.2 Loss function

A hybrid loss function is employed to control backpropagation. It contains Binary Cross Entropy loss (BCE_loss) [19], dice loss, and an L2 regularization term. Binary Cross Entropy is one of the most common evaluation methods in binary task, which can effectively handle the problem of convergence speed [20]. Coronary artery extraction is a single-label binary classification task whose purpose is to divide arterial targets from ICAs. For fast convergence and application to binary classification tasks, a BCE_loss is utilized in our study. The BCE_loss is defined in Eq. (3).

$\displaystyle\textit{Bce\_loss}=-y\log(\bar{y})-(1-y)\log(1-\bar{y})$ (3)

Where $y$ denotes the ground truth, $\bar{y}$ denotes the predicted binary arterial tree.

The arterial contours are only a tiny fraction of the entire pixel in the ICAs. An ensemble similarity meter function, Dice, is introduced to alleviate the data sample discrepancy problem. It is usually used to calculate the similarity of two samples, and assign greater training weights to smaller samples in the dataset. The Dice score (DSC) is computed in Eq. (4).

$\displaystyle\textit{DSC}=\frac{2|{\bar{y}\cap y}|}{|{\bar{y}}|+|y|}$ (4)

Where $|\bar{y}\cap y|$ denotes the intersection of sets $\bar{y}$ and $y$ . $|\bar{y}|$ and $|y|$ denotes the number of their elements The larger the DSC, the smaller the loss. A DSC of 1 demonstrates that there is no difference between the ground truth and the predicted value, while a DSC of 0 indicates complete dissimilarity.

In the task of coronary artery extraction, there are many complex samples in the foreground class that are difficult to judge correctly, such as thinner blood vessel branches. To deal with the problem of the imbalanced distribution of difficulty in samples, it is necessary to make the model focus more attention on the learning of complex samples, which can be achieved by reducing the contribution of simple samples to the total loss and increasing the penalty weight of complex samples. Therefore, we utilize the overall loss defined as Eq. (5) as the loss function of the backbone network.

$\displaystyle L(0)_{\textit{loss}}=-y\log(\bar{y})-(1-y)\log(1-\bar{y})+\log% \left(\frac{2|{\bar{y}\cap y}|}{|{\bar{y}}|+|y|}\right)+\left\|W\right\|_{2}$ (5)

Due to the particularity caused by the low-contrast and high-noise of the ICAs of coronary artery extraction task, more attention should be paid to multi-scale information. The loss function should be used on each branch of deep supervision to control the training of the network. The deep supervised branching loss function is defined as Eq. (6)

$\displaystyle L(s)_{\textit{loss}}=-y\log(\bar{y})-(1-y)\log(1-\bar{y})+\log% \left(\frac{2|{\bar{y}\cap y}|}{|{\bar{y}}|+|y|}\right)+\left\|W\right\|_{2}$ (6)

The hybrid loss function is defined in Eq. (7) as the objective loss function.

$\displaystyle L_{\textit{loss}}=\frac{L(0)_{\textit{loss}}+L(s)_{\textit{loss}% }}{s}$ (7)

The $s$ is the number of layers of all lateral outputs of the depth neural network.

2.2.3 Optimization strategy

Our method in this study was implemented in Python with a PyTorch backend. The CNN we utilized here is a supervised learning model and requires both data and ground truth labels. To further improve the accuracy of coronary extraction, transfer learning was applied. The advantages of transfer learning include better performance and reduced training time [21, 22]. By transfer learning, we could enable the CNN to retain the weights learned in the previous task and then apply them in the new task. Therefore, we introduced pre-trained weights from ImageNet into our network during the training process. During the training of our present model, the initial learning rate was set to 0.0001 and a decay was set to 0.0001. The RMSProp optimizer [23] was employed to update the weights within the CNN. To ensure the best performance of the model, we adopted a learning rate decay strategy, which gradually decreased the learning rate during the training process.

The manually labeled 616 ICAs were still inadequate to train a 2D CNN. To address this issue and enhance the stability of the CNN model, data augmentation techniques based on geometric transformation and color transformation were applied. Geometric transformation such as flipping, rotating, shifting, and deforming were applied to geometrically transform the image. Color transformations such as Gaussian noise, Gaussian blur and luminance multiplication were applied to modify the content of the image. These techniques were used to make minor changes to the existing datasets randomly. The image obtained after data augmentation will be recognized as two different images by the deep learning model during the network training process, thus achieving the purpose of expanding the dataset. When images were fed into the model for training, a random sampling strategy was applied, and each type of data enhancement technique was executed randomly. Additionally, if an image performs data enhancement, the label performs the same data enhancement technique accordingly.

2.3 Metrics to evaluate the accuracy of artery extraction

Five metrics were used to evaluate the accuracy of coronary artery extraction. There are DSC, sensitivity (SN), specificity (SP), Hausdorff distance (HD) and average surface distance (ASD).

SN denotes the ratio of true positive (TP) samples to all actual positive sample pixels, and SP denotes the ratio of true negative (TN) samples to all actual negative sample pixels. The definitions of SN and SP are given in Eqs (8) and (9).

$\displaystyle SN=\frac{TP}{TP+FN}$ (8) $\displaystyle SP=\frac{TN}{TN+FP}$ (9)

where FN and FP denote false negative sample pixels and false positive sample pixels, respectively.

Hausdorff distance represents the degree of similarity between two sets of points. Assume there are two sets of sets $C=\{{a1\ldots an}\}$ and $D=\{{b1\ldots bn}\}$ . the Hausdorff distance between these two sets of points is defined as Eq. (10).

$\displaystyle H(C,D)=\max(h(C,D),{h}(D,C))$ (10)

The definitions of $h({C,D})$ and $h({D,C})$ are given in Eqs (11) and (12).

$\displaystyle h(C,D)=\mathop{\max}\limits_{a\in C}\left\{{\mathop{\min\left\|{% a-b}\right\|}\limits_{b\in D}}\right\}$ (11) $\displaystyle h(C,D)=\mathop{\max}\limits_{b\in D}\left\{{\mathop{\min\left\|{% b-a}\right\|}\limits_{a\in A}}\right\}$ (12)

The surface distance is a function describing the average variance in the surface between the prediction results and ground truth. Let $P(A)$ indicates the set of surface pixels of A. The shortest distance of an arbitrary pixel $v$ to $P(A)$ is defined as Eq. (13).

$\displaystyle d(v,P(A))=\mathop{\min}\limits_{p_{A}\in P(A)}\left\|{v-p_{A}}\right\|$ (13)

Where $||.||$ denote the Euclidean distance. The surface distance is defined as Eq. (14).

$\displaystyle\textit{ASD}=\frac{\sum\limits_{a\in P(A)}{\mathop{\min}\limits_{% b\in P(B)}\left\|{a-b}\right\|}+\sum\limits_{b\in P(B)}{\mathop{\min}\limits_{% a\in P(A)}\left\|{b-a}\right\|}}{|{P(A)}|+|{P(B)}|}$ (14)

3. Results and analysis

3.1 Dataset

A total of 616 ICA images were used in our work, 405 of which were left coronary artery (LCA) images, and 211 were right coronary artery (RCA) images. All these images were manually annotated by well-trained operators and confirmed by experienced interventional cardiologists. Table 1 contains specific information on the ICA images used in this paper.

Table 1
Indicators of the number of different angles of LCA and RCA. Note that LCA and RCA represents the left coronary artery and right coronary artery. LAO and RAO represent the left anterior oblique and right anterior oblique

	LAO	RAO	Total
LCA	165	240	405
RCA	163	48	211

We divided the dataset into the training set, validation set, and test set according to what ratios of 70%, 10%, and 20%. 210 patients had 616 ICA images, of which 437 images were used for the training set, 49 images were used for the validation set, and 130 images were used for the test set.

3.2 Accuracy of artery extraction

DSC was used to evaluate the vessel extraction ability of the deep learning models applied in our paper on an external test set and used 10-fold cross-validation for the final evaluation.

There different methods are compared with our proposed method including U-Net, U-Net $++$ and U-Net 3 $+$ , all of which adopted Deep Supervision. Our present method without Deep Supervision (Our Method DS $-$ ) was individually used as a comparison. Furthermore, four deep learning models, namely DeepLabV3 [24], DeepLabV3 $+$ [25], PSPNet [26] and CGNet [27] were employed for comparison. Transfer learning and data augmentation were adopted to all deep learning models.

As shown in Fig. 4, examples of extracting arteries from ICAs by different deep learning models are delineated in red. We can observe that our proposed algorithm yields the best arteries extraction with the closest visual appearance to the reference ground truth among all the methods.

Table 2
Quantitative results associated with different methods for the images in the testing dataset. Note that HD and ASD represents the Hausdorff distance and average surface distance. DSC, SN and SP represents Dice similarity coefficient, sensitivity and specificity, respectively

Mertic	HD (mm)	DSC	SN	SP	ASD (mm)
CGNet	7.6218	0.5395	0.4728	0.9898	6.7381
PSPNet	6.7776	0.8257	0.7908	0.9924	0.5668
DeepLabV3	6.8468	0.8340	0.8178	0.9911	0.4950
DeepLabV3 $+$	6.5714	0.8462	0.8123	0.9937	0.7651
U-Net	6.1040	0.8799	0.8305	0.9967	0.4403
U-Net $++$	6.1048	0.8814	0.8331	0.9966	0.4588
U-Net 3 $+$	6.1159	0.8868	0.8601	0.9958	0.4031
Our method DS $-$	6.1024	0.8862	0.8626	0.9955	0.5080
Our proposed method	6.0794	0.8942	0.8735	0.9954	0.3247

Table 3

Comparison among the state-of-the-art methods. Note that DSC, SN and SP represents Dice similarity coefficient, sensitivity and specificity, respectively

Mertic	Data (images)	DSC	SN	SP
CNN [11]	44	0.7562	0.7935	0.9890
CNN [12]	44	0.8151	0.8676	0.9859
Multiple CNNs [10]	120		0.7774	0.9934
FP-U-Net $++$ [16]	314	0.8899	0.8595	0.9960
Our proposed method	616	0.8942	0.8735	0.9954

Figure 4.

The raw ICAs and ground truth are shown in (a) and (b), respectively. Arterial segmentation results of the left coronary artery and the right coronary artery illustrated by (c) CGNet, (d) PSPNet, (e) DeepLabV3, (f) DeepLabV3+, (g) U-Net, (h) U-Net $++$ , (i) U-Net 3 $+$ , (j) Our Method DS $-$ , (k) Our Proposed Method obtained for the arterial segmentation results.

In Table 2, the HD, DSC, SN, SP, and ASD indexes are given in terms of average scores for the 10-fold cross-validation results of the test dataset. The quantitatively compared results of the model performance of CGNet, PSPNet, DeepLabV3, DeepLabV3 $+$ , U-Net, U-Net $++$ , U-Net 3 $+$ , Our Method DS $-$ and Our Method under different metrics were presented. The average DSC for our proposed method, U-Net 3 $+$ , U-Net $++$ , and U-Net are 0.8942, 0.8868, 0.8814, and 0.8799, respectively. The results show in Table 2 that the method we proposed obtains the lowest HD and ASD and achieves the highest scores of DSC. In addition, four existing studies for coronary artery segmentation are presented in Table 3.

Figure 5.

Results of the 10-fold cross-validation experiment. (a) DSC (b) SN (C) SP.

As shown in Fig. 5, 10-fold cross-validation was employed to ensure the stability of the experimental results. The DSC, SP and SN of each experiment was almost on the same level (DSC: 0.8941; 0.8980; 0.8960; 0.8916; 0.8913; 0.8937; 0.8926; 0.8955; 0.8955; 0.8934), so it could be concluded that our experimental results have good reliability and stability.

4. Discussion

4.1 Performance analysis

Various methods based on machine learning techniques have been introduced in healthcare to assist doctors in diagnosis and improve medical efficiency [28, 29, 30]. They have demonstrated extensive applications in the field of cardiology [31]. Deep learning models have been developed and achieved remarkable results on image segmentation tasks [13, 14, 15].

U-Net has been widely used for the segmentation of medical images since it was proposed. However, the plain skip connections in U-Net require the fusion of equal amounts of feature maps from the encoder and decoder networks, even though they may not be semantically identical. This limitation can affect the performance of feature repair in the decoder path, leading to reduced quality and efficiency in coronary segmentation [32].

U-Net $++$ improves on U-Net by introducing nested skip connections that better utilize low- and deep-level features, reducing the loss of semantic information in the encoding and decoding sub-networks [14]. U-Net $++$ is effective at exploring multi-scale information with its nested dense skip connection [33]. However, short connections primarily reprocess decoding features, and with various connections fused together, the original multi-scale information may not be well utilized. In the research of segmentation tasks, high-level and low-level semantic feature maps can capture different information, respectively. However, some of the required information may be diluted in down and up-sampling.

PSPNet, which takes advantage of the pyramid parsing module and can aggregate contextual information from different regions. CGNet learn both local and global features by obtaining contextual texture features. DeepLabV3, which takes advantage of spatial pyramidal pooling based on atrous convolution to augment image-level features.

To improve the accuracy of image segmentation, the U-Net 3 $+$ , which utilizes full-scale skip connections and full-scale deep supervision, has been proposed [15]. The full-scale skip connection in U-Net 3 $+$ can simultaneously combine the low-level semantic features and high-level abstract features in the down-sampling process, increasing the receptive field.

In this study, we propose a novel deep learning model for accurate coronary artery extraction. The full-scale skip connections allow for exploring more semantic information and capturing more detailed features, leading to improved segmentation accuracy. Deep-supervised methods have demonstrated an effective approach for addressing the vanishing gradient and exploding gradient problems that are common in CNNs [34, 35]. The full-scale deep supervision not only avoids over fitting, enables the CNN to learn a more comprehensive representation of arterial information, which is beneficial for segmenting arterial details. In addition, the inception module downscales matrices with larger sizes and concatenates visual information from different scales for feature extraction. The Residual module accelerates neural network training and also improves neural network performance. The Inception and Residual modules in the encoder structure can improve segmentation accuracy by enlarging the receptive field and reducing the feature loss during convolution. Our proposed model can learn arterial semantic information from aggregated feature maps at different scales and enhance the extraction of arterial boundary information, thus improving the performance of coronary artery extraction.

It can be concluded that the method we applied achieved the highest DSC on the ICA dataset (Table 2). The average Dice score of our proposed model was 1.43%, 1.28% and 0.74% higher than that of U-Net, U-Net $++$ and U-Net 3 $+$ . The average Dice score of our model using full-scale deep supervision with hybrid loss functions increased from 0.8862 to 0.8942. The SN was 0.0430 higher than U-Net, 0.0404 higher than U-Net $++$ and 0.0154 higher than U-Net 3 $+$ , respectively. In the evaluation of HD and ASD, our model obtained results of 6.0794 and 0.3247, respectively, which are better than U-Net 3 $+$ (HD of 6.1159 and ASD of 0.4031), U-Net $++$ (HD of 6.1048 and ASD of 0.4588) and U-Net (HD of 6.1040 and ASD of 0.4403). Furthermore, our model could predict arteries for a single ICA frame in 0.2114 seconds.

Figure 6.

Comparison between the segmentation results and the ground truth. Green and red colors indicate the pixels with segmentation errors. (a) ICA, (b) ground truth, (c) CGNet, (d)PSPNet, (e) DeepLabV3, (f) DeepLabV3+, (g) U-Net, (h) U-Net $++$ , (i) U-Net 3 $+$ , (j) Our Method DS, (k) Our Proposed Method. The green, red and yellow regions represent false negative, false positive and true positive pixels, respectively.

As shown in Fig. 6, the green indicates false negative segmentation, where the artery is incorrectly predicted as background. The red color indicates false positive segmentation where the background is incorrectly predicted as an artery. The yellow color indicates the artery that has been accurately extracted. The segmentation model proposed in this paper predicts fewer erroneous pixel regions than other models.

The advantage of the proposed network includes the following: it can achieve excellent DSC when applied to ICAs and the significance of boosting modules in our method was demonstrated by ablation experiments.

4.2 Limitations

There are the following limitations on automatic coronary artery extraction by our proposed model. 1) The sample size is still limited. More ICA images with manual annotation will further improve the performance of our method. 2) Automatic extraction of coronary arteries in this study is based on the contour of single-view single-frame ICA. New methods that incorporate adjacent frames in the ICA videos will improve the segmentation performance of our method, particularly for missing and overlapped vessels.

4.3 Clinical overview and application

At present, ICA is still the reference gold standard imaging technique for the evaluation of clinically significant CAD. The fundamental task required for the interpretation of ICA is to identify and quantify the severity of coronary stenosis [36]. However, manual estimation of coronary stenosis is time-consuming and determination of the degree of stenosis depends on training and visual estimation. In recent years, automatic detection of coronary stenosis in ICA has become an important factor in assisting physicians with clinical diagnosis, so improving the accuracy of automatic stenosis detection is a necessity for computer-aided diagnostic systems.

The general scheme for automated or semi-automated quantitative stenosis assessment in ICA is artery extraction, geometry calculation, and stenosis analysis [37]. Accurate coronary artery extraction is an important step in the stenosis detection task. We propose an automated coronary artery extraction model, which will enhance the accuracy of coronary stenosis assessment. Especially in those patients with diffuse stenosis or continuous angiography, where accurate automated coronary artery extraction is extremely important.

In the assessment of eccentric lesions, relying on accurate coronary artery auto-extraction models, such as the proposed algorithm, can enhance the accuracy of coronary artery auto-extraction, which will facilitate the assessment of coronary stenosis and thus improve the identification of similar complex lesions.

3D arterial anatomy can also be reconstructed from two-dimensional ICAs to aid in clinical diagnosis of CAD. Automated extraction of coronary from ICA is essential for 3D vessel reconstruction [38, 39]. The automatic extraction of coronary arteries in this study will be used with our 3D vessel revascularization for clinical decision-making of CAD diagnosis and treatment.

5. Conclusion

In this paper, we present a deep learning-based method for automatic extraction of coronary arteries from ICA. Several important techniques were designed to improve the performance of our model, including full-scale skip connections, full-scale deep supervisions, and the incorporation of inception and residual modules in the encoder. Our model achieved a Dice score of 0.8942, which outperformed the state-of-the-art deep-learning models. It has great promise for clinical applications.

Footnotes

Acknowledgments

We sincerely thank Zhihui Xu, MD, at the Department of Cardiology in The First Affiliated Hospital of Nanjing Medical University, for providing the ICA images for this research.

Conflict of interest

The authors declare no conflict of interest.

Funding

This research was supported by the Key Science Research Project of Colleges and Universities in Henan Province of China (No. 22A520046) and the Key Science and Technology Program of Henan Province (232102211003).

References

Sanchis-Gomar

Pascual-Figal

Pérez-Gómez

López-González

López-Sendón

López-López

. Epidemiology of coronary heart disease and acute coronary syndrome. Ann. Transl. Med. 2016; 4: 13.

Benjamin

, et al. Heart disease and stroke statistics-2019 update: A report from the american heart association. Circulation. 2019; 139: 56-528.

Moon

Cha

Chung

Lee

Cho

Choi

. Automatic stenosis recognition from coronary angiography using convolutional neural networks. Computer Methods and Programs in Biomedicine. 2021; 198: 105819.

World Health Organization. Cardiovascular diseases [Internet]. Geneva: World Health Organization; [cited 2023 May 25]. Available from: https://www.who.int/health-topics/cardiovascular-diseases.

TXL

Zhang

Liu

Wang

Chen

. Automatic and multimodal analysis for coronary angiography: Training and validation. Appl. Sci. 2018; 6: 837.

Chu

Yang

Gutiérrez-Chico

. Advances in diagnosis, therapy, and prognosis of coronary artery disease powered by deep learning algorithms. JACC: Asia. 2023; 3(1): 1-14.

Zhou

Peng

. A Novel Method of Vessel Segmentation for X-Ray Coronary Angiography Images. In: 2012 Fourth International Conference on Computational and Information Sciences. 2012. pp. 468-471. doi: 10.1109/ICCIS.2012.6375090.

Felfelian

Fazlali

Karimi

Soroushmehr

SMR

Samavi

Nallamothu

Najarian

. Vessel segmentation in low contrast X-ray angiogram images. MDPI Journal of Imaging. 2017; 3(2): 1-8.

Tsai

Y-C

Lee

H-J

Chen

MY-C

. Automatic segmentation of vessels from angiogram sequences using adaptive feature transformation. Comput. Biol. Med. 2015; 62: 239-253.

10.

Yang

Wang

Yang

Wang

. Automatic Coronary Artery Segmentation in X-Ray Angiograms by Multiple Convolutional Neural Networks. In: Proceedings of the 3rd International Conference on Multimedia and Image Processing. 2018. pp. 31-35. doi: 10.3390/conferences-3-01234.

11.

Nasr-Esfahani

Samavi

Karimi

Soroushmehr

Ward

Jafari

Felfeliyan

Nallamothu

Najarian

. Vessel extraction in x-ray angiograms using deep learning. Int J Mol Sci. 2020; 21(2): 643.

12.

Nasr-Esfahani

Karimi

Jafari

Soroushmehr

SMR

Samavi

Nallamothu

Najarian

. Segmentation of vessels in angiograms using convolutional neural networks. Biomed. Signal Process. Control. 2018; 40: 240-251.

13.

Ronneberger

Fischer

Brox

. U-Net: Convolutional networks for biomedical image segmentation. Int. J. Comput. Vis. 2015; 234: 234-241.

14.

Zhou

Siddiquee

MMR

Tajbakhsh

Liang

. Unet+⁣+: A nested u-net architecture for medical image segmentation. Deep Learn. Med. Image Anal. Multimod. Learn. Clin. Decis. Support. 2018; 3: 3-11.

15.

Huang

, et al. UNet 3

+

: A Full-Scale Connected UNet for Medical Image Segmentation. In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2020. pp. 1055-1059.

16.

Zhao

Vij

Malhotra

Tang

Pienta

Zhou

. Automatic extraction and stenosis evaluation of coronary arteries in invasive coronary angiograms. Computers in Biology and Medicine. 2021; 136: 104667. doi: 10.1016/j.compbiomed.2021.104667.

17.

Ioffe

Szegedy

. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In: Proceedings of the International Conference on Machine Learning (ICML’15); Cambridge, MA, USA, 2015. pp. 448-456.

18.

Otsu

. A threshold selection method from gray-level histograms. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on. 1979; 9: 62-66. doi: 10.1109/TSMC.1979.4310076.

19.

Zadeh

Schmid

. Bias in cross-entropy-based training of deep survival networks. IEEE Trans Pattern Anal Mach Intell. 2021 Sep; 43(9): 3126-3137. doi: 10.1109/TPAMI.2020.3037862.

20.

Ramos

Franco-Pedroso

Lozano-Diez

Gonzalez-Rodriguez

. Deconstructing cross-entropy for probabilistic binary classifiers. Entropy. 2018; 20(3): 208. doi: 10.3390/e20030208.

21.

Neyshabur

Sedghi

Zhang

CJA

. What is being transferred in transfer learning? Mater. 2020; 13(10): 4862. doi: 10.3390/ma13104862.

22.

Liu

Chen

Fieguth

Zhao

Chellappa

Pietikäinen

. From BoW to CNN: Two decades of texture representation for texture classification. Int. J. Comput. Vis. 2019; 127: 74-109. doi: 10.1007/s11263-018-1125-z.

23.

Tieleman

Hinton

. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. Neural Networks for Machine Learning. 2012; 4: 26-31.

24.

Chen

L-C

Papandreou

Schroff

Adam

. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv preprint arXiv1706.05587, 2017.

25.

Chen

Zhu

Papandreou

Schroff

Adam

. Encoder-decoder with atrous separable convolution for semantic image segmentation. Proc Eur Conf Comput Vis. 2018; 14: 801-818.

26.

Zhao

Shi

Wang

Jia

. Pyramid scene parsing network. IEEE Conf Comput Vis Pattern Recognit. 2017; 2017: 2881-2890.

27.

Tang

Zhang

Cao

Zhang

. CGNet: A light-weight context guided network for semantic segmentation. IEEE Trans. Image Process. 2021; 30: 1169-1179.

28.

Badnjevic

Cifrek

Koruga

. Classification of Chronic Obstructive Pulmonary Disease (COPD) using integrated software suite. In: XIII Mediterranean Conference on Medical and Biological Engineering and Computing 2013: MEDICON 2013, September 25–28, 2013; Seville, Spain. Springer International Publishing; 2014. pp. 911-914.

29.

Alić

Gurbeta

Badnjević

Badnjević-ćCengić

Malenica

Dujić

Bego

, et al. Classification of metabolic syndrome patients using implemented expert system. In: CMBEBIH 2017: Proceedings of the International Conference on Medical and Biological Engineering 2017, Springer Singapore. 2017. pp. 601-607.

30.

Mustafić

Gurbeta

Badnjevic-Cengic

Badnjević

Hukeljić

Bego

Perva

. Diagnosis of severe aortic stenosis using implemented expert system. In: Proceedings of the International Conference on Medical and Biological Engineering 2019 (CMBEBIH 2019), May 16–18, 2019, Banja Luka, Bosnia and Herzegovina. Springer International Publishing; 2020. pp. 149-153.

31.

Šećkanović

Šehovac

Spahić

Ramić

Mamatnazarova

Pokvić

, et al. Review of artificial intelligence application in cardiology. In: 2020 9th Mediterranean Conference on Embedded Computing (MECO). IEEE; 2020. pp. 1-5.

32.

Zhou

Siddiquee

NMMR

Tajbakhsh

Liang

. UNet++: Redesigning skip connections to exploit multiscale features in image segmentation. Appl. Sci. 2020; 10: 4862.

33.

Zhao

Bober

Tang

Dong

Zhang

Esposito

Zhou

. Semantic segmentation to extract coronary arteries in invasive coronary angiograms. J Adv Appl Comput Math. 2022; 9(1): 76-85.

34.

Çiçek

Abdulkadir

Lienkamp

Brox

Ronneberger

. 3D U-Net: Learning dense volumetric segmentation from sparse annotation. Int. J. Comput. Assist. Radiol. Surg. 2016; 11: 424-432.

35.

Dou

Chen

Jin

Yang

Qin

Heng

. 3D deeply supervised network for automated segmentation of volumetric medical images. Med. Image Anal. 2017; 41: 40-54.

36.

Zhang

Xie

Zhao

Zhang

. Automatic detection of coronary artery stenosis by convolutional neural network with temporal constraint. Comput. Biol. Med. 2020; 118: 103657.

37.

Danilov

Klyshnikov

Gerget

Kutikhin

Ganyukov

Frangi

. Real-time coronary artery stenosis detection based on modern neural networks. Sci. Rep. 2021; 11: 7582.

38.

Tang

Bober

Zhao

Zhang

Zhu

Zhou

. 3D Fusion between Fluoroscopy Angiograms and SPECT Myocardial Perfusion Images to Guide Percutaneous Coronary Intervention. J. Nucl. Cardiol. 2021; 1-15.

39.

Tang

Malhotra

Dong

Zhou

Wang

Zhou

. Three-dimensional fusion of myocardial perfusion SPECT and invasive coronary angiography guides coronary revascularization. J. Nucl. Cardiol. 2020; 27: 2265-2267.

Automatic extraction of coronary arteries using deep learning in invasive coronary angiograms

Abstract

BACKGROUND:

OBJECTIVE:

METHODS:

RESULTS:

CONCLUSIONS:

Keywords

1. Introduction

2. Materials and methods

2.1 Patient data

2.2 Artery extraction

2.3 Metrics to evaluate the accuracy of artery extraction

3.1 Dataset

Table 1 Indicators of the number of different angles of LCA and RCA. Note that LCA and RCA represents the left coronary artery and right coronary artery. LAO and RAO represent the left anterior oblique and right anterior oblique

Table 2 Quantitative results associated with different methods for the images in the testing dataset. Note that HD and ASD represents the Hausdorff distance and average surface distance. DSC, SN and SP represents Dice similarity coefficient, sensitivity and specificity, respectively

4.1 Performance analysis

4.3 Clinical overview and application

5. Conclusion

Footnotes

Acknowledgments

Conflict of interest

Funding

References

Table 1
Indicators of the number of different angles of LCA and RCA. Note that LCA and RCA represents the left coronary artery and right coronary artery. LAO and RAO represent the left anterior oblique and right anterior oblique

Table 2
Quantitative results associated with different methods for the images in the testing dataset. Note that HD and ASD represents the Hausdorff distance and average surface distance. DSC, SN and SP represents Dice similarity coefficient, sensitivity and specificity, respectively