Abstract
PURPOSE:
To establish a machine-learning (ML) model based on coronary computed tomography angiography (CTA) images for evaluating myocardial ischemia in patients diagnosed with coronary atherosclerosis.
METHODS:
This retrospective analysis includes CTA images acquired from 110 patients. Among them, 58 have myocardial ischemia and 52 have normal myocardial blood supply. The patients are divided into training and test datasets with a ratio 7 : 3. Deep learning model-based CQK software is used to automatically segment myocardium on CTA images and extract texture features. Then, seven ML models are constructed to classify between myocardial ischemia and normal myocardial blood supply cases. Predictive performance and stability of the classifiers are determined by receiver operating characteristic curve with cross validation. The optimal ML model is then validated using an independent test dataset.
RESULTS:
Accuracy and areas under ROC curves (AUC) obtained from the support vector machine with extreme gradient boosting linear method are 0.821 and 0.777, respectively, while accuracy and AUC achieved by the neural network (NN) method are 0.818 and 0.757, respectively. The naive Bayes model yields the highest sensitivity (0.942), and the random forest model yields the highest specificity (0.85). The k-nearest neighbors model yields the lowest accuracy (0.74). Additionally, NN model demonstrates the lowest relative standard deviations (0.16 for accuracy and 0.08 for AUC) indicating the high stability of this model, and its AUC applying to the independent test dataset is 0.72.
CONCLUSION:
The NN model demonstrates the best performance in predicting myocardial ischemia using radiomics features computed from CTA images, which suggests that this ML model has promising potential in guiding clinical decision-making.
Introduction
As the average age of the population has continued to increase, the incidence and mortality of coronary heart disease have likewise risen [1–3]. Currently, the most commonly used method for the clinical assessment of coronary artery stenosis is computed tomography angiography (CTA). This technology is highly sensitive for identifying the degree of coronary artery stenosis and plaque type [4, 5], but the tissue contrast offered by CTA is limited, and so the feasibility of using CTA images to estimate myocardial ischemia is low [6]. The current gold standard modalities for the clinical diagnosis of myocardial ischemia include single-photon emission computed tomography (SPECT), positron emission tomography (PET), and fractional flow reserve (FFR) [1]. However, these tests are expensive and invasive, and so they cannot be widely used in clinical practice. An economical and practical method for assessing myocardial ischemia is therefore needed.
Texture analysis can be used on medical images to noninvasively extract a large number of parameters that cannot be visualized with the naked eye or quantified by conventional analysis, and to then construct predictive or prognostic models for disease treatment [7–9]. Recently, research has shown that texture analysis can be used for the preliminary evaluation of myocardial ischemia [10]. However, this technique does have limitations. For instance, many texture features are extremely sensitive to the acquisition protocol; even if the same acquisition protocol is used for multiple analyses, the reproducibility of the features is uncertain, and such reproducibility is required for both accurate follow-up research and clinical applications [11, 12].
Any model based on machine learning (ML) must therefore demonstrate its ability to produce reproducible results in regard to texture parameters. To this end, we sought to build a high-performance ML model that could be used to evaluate myocardial ischemia in patients with coronary heart disease, and we assessed both the number of texture features and the effect of various ML methods.
Methods
Patient information
We obtained approval for this study from the research ethics committee of Changzhou No.2 People’s Hospital, the Affiliated Hospital of Nanjing Medical University, with a waiver of informed consent. We retrospectively analyzed data from patients with myocardial ischemia and patients with normal myocardial blood supply confirmed by FFR or digital subtraction angiography (DSA) at our hospital between September 2018 and January 2021. Patients were included in the study if they had no history of other heart disease, heart bypass surgery, or coronary stent implantation, and if the interval between CTA image acquisition and DSA or FFR diagnosis of myocardial ischemia was < 2 weeks. Patients were excluded from the analysis if they had liver or kidney dysfunction or if the interval between CTA image acquisition and myocardial ischemia diagnosis was < 1 week (Fig. 1).
Flowchart for inclusion and exclusion of patients.
All patients underwent CTA scans on a multislice spiral CT device (64 slices; Siemens AG, Munich, Germany). The scan range was from 1 to 2 cm below the tracheal crest to the bottom of the heart. The following parameters were used = automatic tube current modulation; tube voltage, 120 kVP; and slice thickness, 7.5 mm. The contrast agent iohexol (350 mg I/mL, Omnipaque, GE Healthcare) was intravenously injected. The duration of contrast medium injection was 12s, and the injection rate is 6 ml/s; heart rate≥75 beats/min; for every increase of 5 beats/min, the injection rate was elevated by 0.2 mL/s, and the contrast medium dose was similarly increased. Figures 2 and 3 show the CCTA images of patient with and without myocardial ischaemia, respectively.
A 57-year-old male patient with myocardial ischaemia. A: Axial image, B: Sagittal image, and C: Coronal image.
A 58-year-old female patient with myocardial ischaemia. A: Axial image, B: Sagittal image, and C: Coronal image.
The CTA images were imported into CQK (CT Coronary Artery Quantitative Analysis Kit, GE Healthcare, China) software for automated segmentation and feature extraction. Myocardial segmentation results were validated by a radiologist in the cardiovascular subgroup with at least 10 years of experience. The image analysis software package was imported on the CQK platform to extract texture features, which complied with Image Biomarker Standardisation Initiative (IBSI). Fig. 4 showed the workflow of the construction of the myocardial ischemia model.
Workflow of the construction of the myocardial ischemia model.
The extracted texture features were preprocessed to remove texture features with zero variance. A Mann-Whitney U test was then used to identify texture features with significant difference (P values < 0.01) between the two groups (myocardial ischemia group and normal myocardial blood supply group), and these features were retained. To avoid unnecessary complexities and redundancies of the models, we then performed a minimum redundancy maximum relevance (MRMR) analysis on the selected texture features. Considering to the number of features affecting the final results, after MRMR, 4 feature subsets were obtained with 5, 10, 15, 20 features, respectively.
Model construction and comparison
The patients were divided into the training and test datasets with the ratio of 7 : 3, the training dataset was used to train the model, and independent test dataset was used to validate the final model. Seven ML methods were used to construct models = naive Bayes (NB), random forest (RF), extreme gradient boosting linear (xgbLinear), neural network (NN), k-nearest neighbors (KNN), support vector machines with the radial basis function kernel (svmRadial), and support vector machines with linear kernel (svmLinear). The optimal feature subset was identified based on the area under the curve (AUC) values of the 7 ML methods trained with each subset of features. The mean AUC value of the 7 models was determined, and the subset of texture features with the largest mean AUC was selected as the final subset.
After determining the optimal subset of texture features, a nested cross-validation scheme was performed for each ML method, and the model was repeatedly trained in the training data set. Each ML approach eventually has 100 models and each indicator (ie, accuracy, sensitivity, specificity, positive predictive value [PPV], and negative predictive value [NPV]) would have 100 values. In addition, the robustness of each model was evaluated by calculating the relative standard deviation (RSD) values [13]. The smaller the RSD value, the higher robust of ML model. The following equation was used for RSD calculation:
Model validation and data analysis
According to the performance and stability of 7 ML models, the optimal model was determined, and the performance of the optimal model was validated in the independent test dataset using ROC analysis.
Then, in the statistical data analysis, continuous variables with a normal distribution were expressed as mean±standard deviation (X±S), and Student’s t-test was used for comparing the difference between the two patient groups. Continuous variables with non-normal distribution were expressed as median (interquartile range), and the Mann-Whitney U test was employed for comparing the difference between the groups. Categorical variables were presented as frequency (percentage) and were analyzed using a chi-square test. All statistical tests were conducted with SPSS v26.0 and R v3.6.1 (https://www.r-project.org) software. The two-tailed P-value of < 0.05 was deemed statistically significant.
Results
In this study, 58 patients with normal myocardial blood supply and 52 patients with myocardial ischemia were enrolled in this study. Among them, 77 patients were assigned in training dataset, and 33 patients were in test dataset. There were no obvious differences between the groups with respect to sex; age; presence of hypertension, hyperlipidemia or diabetes; and history of alcohol use or smoking (Table 1).
Clinical characteristics of study patients
Clinical characteristics of study patients
We used 4 subsets with different numbers of features (5, 10, 15, and 20 features) to train the model. The results showed that the model using 5 features demonstrated the highest average AUC. A heat map was used to identify the top 5 differences between the patients in normal myocardial blood supply group and those in myocardial ischemia group (Fig. 5).
Heatmap of the 5 selected features for each patient. The larger the value, the color was closer to red, and the smaller the value, the color was closer to blue.
As shown in Fig. 6 and Table 2, we found that among the 7 ML models, xgbLinear had the highest accuracy (0.821) and AUC value (0.777), followed by NN (accuracy = 0.818; AUC = 0.757). NB had the highest sensitivity value (0.942), and RF had the highest specificity value (0.85). KNN had the lowest accuracy (0.74), svmLinear had the lowest AUC value (0.732), RF had the lowest sensitivity value (0.775), and NB had the lowest specificity value (0.68).
Performance metrics of 7 predictive models built with different machine-learning methods. RF, random forest; symRadial, support vector machines with the radial basis function kernel; symLinear, support vector machines with linear kernel; NB, naive Bayes; NN, neural network; xgbLinear, extreme gradient boosting linear; KNN, k-nearest neighbors; AUC, area under the curve; NPV, negative predictive value; PPV, positive predictive value. “•” indicates outliers.
Performance metrics for seven ML models
KNN, k-nearest neighbors; NB, naive Bayes; NN, neural network; RF, random forest; symLinear, support vector machines with linear kernel; symRadial, support vector machines with the radial basis function kernel; xgbLinear, extreme gradient boosting linear; AUC, area under the curve; NPV, negative predictive value; PPV, positive predictive value
The most robust classifier was NN (RSD = 0.16 for accuracy, 0.08 for AUC), followed by RF (RSD = 0.14 for accuracy, 0.10 for AUC), svmRadial (RSD = 0.16 for accuracy, 0.11 for AUC), xgbLinear (RSD = 0.22 for accuracy, 0.13 for AUC), NB (RSD = 0.21 for accuracy, 0.15 for AUC), KNN (RSD = 0.20 for accuracy, 0.17 for AUC), and svmLinear (RSD = 0.26 for accuracy, 0.18 for AUC) (Fig. 7).
Model performance metrics (Left: accuracy, Right: area under the curve [AUC]) vs model stability. KNN, k-nearest neighbors; NB, naive Bayes; NN, neural network; RF, random forest; symLinear, support vector machines with linear kernel; symRadial, support vector machines with the radial basis function kernel; xgbLinear, extreme gradient boosting linear; RSD, relative standard deviation.
In general, we observed that NN had the best performance of the ML models in the training dataset. The performance of the NN model in the independent test dataset demonstrated an AUC value of 0.721 (95% confidence interval = 0.529 –0.882), with accuracy, sensitivity, specificity, PPV, NPV of 0.67, 0.69, 0.65, 0.65 and 0.69, respectively (cutoff = 0.396) (Fig. 8).
Receiver operating characteristic curve of the model built using the neural network machine learning method. AUC, area under the curve.
In this study, we extracted and selected the optimal myocardial radiomic features from CTA images and then used these features to construct 7 ML models for predicting and evaluating myocardial ischemia. We found that among these 7 models, NN model demonstrated relatively good predictive performance and the best stability. These results suggest that the NN method should be the model of choice for the prediction of myocardial ischemia.
Previous studies have shown that myocardial ischemia can cause changes in myocardial function and myocardial tissue, and these early changes cannot be observed with the naked eye. However, these subtle changes can be identified by assessing myocardial texture [14]. As in our study, the final selected feature set consisted of high-order texture features calculated from wavelet-transformed image. Similarly, Hu and co-workers [15] evaluated 1,409 radiomics features from CTA images from patients with myocardial ischemia and constructed a logistic regression model that demonstrated AUC values of 0.762 and 0.671 for the training and test cohorts, respectively. Shu et al. [10] similarly used ML to develop a model that could predict chronic myocardial ischemia. From CTA images, 378 textural radiomic features were captured based on 3-dimensional myocardial segmentations, and the resulting model demonstrated accuracy values of 0.839, 0.832 and 0.816 for the training, test and validation cohorts, respectively. In another study, Zhao and colleagues [16] extracted 385 radiomics features from target lesions on CTA images and constructed a logistic regression model, which demonstrated AUCs for predicting myocardial ischemia of 0.835 and 0.717 for the training and test cohorts, respectively.
In order to obtain optimized machine learning model, performance and stability were both to be considered. Performance determines the capability of the model, while stability determines the reliability of the model, and the latter is more important. In our study, we found 7 model had similar performance as shown in Table 2, but their stabilities were different, so we chose the optimized model which had relative high performance but most stable one. Several studies have also assessed and compared various ML-based models for predicting disease. For instance, Li et al. [17] evaluated the ability of 9 ML methods to predict local and distant metastasis in advanced nasopharyngeal cancer. They observed that the Adaboost (AUC = 0.82) and RF (AUC = 0.85) methods had excellent predictive performance and stability. Yin et al. [13] studied the ability of 7 ML models to estimate the pathological grades of clear cell renal cell carcinoma, and svmRadial (AUC = 0.86; RSD = 0.17 for accuracy, 0.13 for AUC) demonstrated an excellent prognostic performance. These findings suggest that there is no “one-size-fits-all” prognostic model for tumor prediction [18], and we believe that this also holds true for predicting myocardial ischemia based on radiomics features.
Many studies of prognostic model use random resampling to split the data set into training and test cohorts, which may lead to bias due to unreasonable partitioning schemes. Additionally, many elements can affect the performance of these models, such as the quantity of texture features and ML models; however, many studies ignore these potential differences and simply use an ML method to train and test the final model. In this study, a nested cross-validation was employed to train the prediction model, the data set was divided 100 times repeatedly, and the average performance indicators were derived to demonstrate the robustness of the model. In addition, we used different numbers of texture feature sets and ML models to train and identify optimal models.
This study did have some limitations. First, the data used to train and test the model were from a single institution, which may lead to selection bias. Our results must be verified with a large-scale, multicenter, prospective study. Second, the model included only radiomic features from CTA images. To improve the predictive performance of the model, laboratory indicators should be included. Our focus in this study was the effectiveness of radiomics features; in future studies, these features plus laboratory indicators will need to be assessed.
In summary, we studied the role of myocardial texture features in predicting myocardial ischemia and compared the effectiveness of 7 ML models. Among these models, the NN method using 5 features had the most outstanding diagnostic performance and highest stability. This radiomics-based model can further improve the clinical performance of CTA by allowing clinicians to more accurately diagnose myocardial ischemia and formulate more effective treatment plans.