Abstract
BACKGROUND:
Ultrasound is one of the non-invasive techniques that are used in clinical diagnostics of carotid artery disease.
OBJECTIVE:
This paper presents software methodology that can be used in combination with this imaging technique to provide additional information about the state of patient-specific artery.
METHODS:
Overall three modules are combined within the proposed methodology. A clinical dataset is used within the deep learning module to extract the contours of the carotid artery. This data is then used within the second module to perform the three-dimensional reconstruction of the geometry of the carotid bifurcation and ultimately this geometry is used within the third module, where the hemodynamic analysis is performed. The obtained distributions of hemodynamic quantities enable a more detailed analysis of the blood flow and state of the arterial wall and could be useful to predict further progress of present abnormalities in the carotid bifurcation.
RESULTS:
The performance of the deep learning module was demonstrated through the high values of relevant common classification metric parameters. Also, the accuracy of the proposed methodology was shown through the validation of results for the reconstructed parameters against the clinically measured values.
CONCLUSION:
The presented methodology could be used in combination with standard clinical ultrasound examination to quickly provide additional quantitative and qualitative information about the state of the patient’s carotid bifurcation and thus ensure a treatment that is more adapted to the specific patient.
Keywords
Introduction
Carotid artery disease is one of the primary causes of ischemic cerebrovascular events. This disease is manifested by the creation of atherosclerotic plaques inside the wall of the arteries. The formation of plaques causes changes in the geometry of the blood vessels and disturbance of blood flow and this further results in the reduction of blood supplies to the brain, and ultimately increases the risk for stroke. During clinical diagnostics, three relevant parameters of this disease are analyzed: the percentage of stenosis, the occurrence of symptoms and the recency of the symptoms [1]. A patient (either with or without symptoms) is considered at high risk and in need for surgical intervention (carotid endarterectomy or stent implantation) if there is a diagnosed stenosis larger than 70%. An asymptomatic patient with a stenosis less than 70% is subjected to a medical treatment alone [2]. A symptomatic patient with recent events and with a diagnosed stenosis greater than 50% is treated as high risk too. However, these criteria are too general and can lead to many unnecessary surgeries and inappropriately treated patients. Thus, the necessity surges to carry out diagnostics that are more patient-specific. There are several non-invasive imaging techniques that have been used in clinical diagnostics to detect plaques and determine the degree of stenosis, including high resolution ultrasound (US), magnetic resonance imaging (MRI) and computed tomography (CT). The main benefits of the US technique are that it is noninvasive and low cost, and it is therefore commonly applied in clinical diagnostics of CAD. Using US, two-dimensional (2D) images that represent a cross-section of the analyzed artery are created and analyzed by medical staff. But, there is space for improvement. One of the ways is to create new tools that can be used together with this imaging technique to provide additional quantitative and qualitative information about the state of the patient’s carotid artery.
The quality of US images of the carotid arteries (CA) is generally lower and the commonly present noise reduces the visibility of the analyzed structures. This makes it difficult to use standard imaging features during image analysis. Also, due to the mentioned drawbacks of US images, it can be highly demanding and sometimes unreliable to detect and analyze the plaque within patient’s CA during clinical examination, even for medical experts. Imaging analysis of the plaque [3] has proven to be useful in helping medical experts determine the plaque size, position and even plaque composition, further enabling more accurate identification of patients that have higher risk of plaque rupture and more serious complications. Several computer techniques were applied for the analysis of US images and they can be divided in two categories: traditional and deep learning approaches. Traditional US computer-aided design (CAD) systems focus on the features of the US images during their classification and their main components are feature selection and extraction [4]. Some of the most common features that are extracted from US images are texture features – Laws Texture Energy (LTE) [5], wavelet features [6] and morphology features [7, 8]. The classification is performed mostly using Bayesian Classifier, Support Vector Machine, Decision Tree, Artificial Neural Network, AdaBoost, etc.
The deep learning techniques have gained popularity in recent years and have shown great potential for the automatic image segmentation and extraction of relevant medical data. This approach has been applied for analysis of medical images in many diverse modalities, including images of lung, brain and breast [9, 10] and it can also be applied for the segmentation of blood vessels [11, 12], and specifically the segmentation of US images of CA [13]. The advantages of deep learning approach over other techniques are that it uses the available information in a more optimal way and has better prediction rate. In traditional US CAD systems the generic imaging features are extracted and analyzed, while the deep learning techniques use complex processing layers that are more adapted to the specific data that is being analyzed. Like it was already mentioned, US images can contain significant noise that reduces the quality of the image and artifacts that can potentially obscure important segments of the vessel. Due to this fact, the deep learning approach can be a very good and reliable alternative to segment this type of images. Convolutional neural networks (CNNs) can be used to extract new discriminative features from the US images, by using a combination of both global and local imaging information. The prediction rate of several machine learning and deep learning approaches was compared in literature [14] and the efficiency of these approaches in identification of plaque components and classification of symptomatic/asymptomatic patients was analyzed. Automated system for characterization and classification of plaque within internal carotid artery was presented in [15], where the plaque was classified into binary classes (symptomatic and asymptomatic types) using the classic CNN. A semiautomatic segmentation method was presented in [16] and uses Dynamic CNN and U-Net models to extract media-adventitia and lumen-intima boundaries from three-dimensional (3D) US images, in order to provide quantified data about carotid plaques. Deep-learning based US imaging was combined with flow measurement techniques in [17] to quantitatively analyze dynamics of blood flow and this approach was applied to analyze flow in murine carotid arteries. Attention-channel-based UNet deep learning (DL) model was used in [18] to identify plaque in US images of carotid arteries. U-Net and DenseNet networks were combined in [13] to perform an automated segmentation of atherosclerotic plaques in US images of carotid arteries.
A more detailed visualization of the state of the arteries is limited by the 2D cross-sections obtained during US examination. It would therefore be useful to perform 3D reconstruction of the both the lumen and wall of the specific vessel, in order to provide more data to the clinical experts. Such 3D patient-specific model can also be used as input for the numerical simulation that can offer even more quantitative information about the state of blood flow. Numerical simulations using methods of computational fluid dynamics (CFD) have already been successfully applied to model blood flow through arteries [19]. Using these techniques it is possible to quantitatively analyze the process of creation of atherosclerotic plaque and predict regions where further progression of the disease would be possible. The effect of blood flow on the plaque progression was analyzed in literature [20, 21] for coronary arteries, but also for CA. It was concluded that the plaque progression occurs in regions with low wall shear stress (WSS). Similarly, it was concluded by Sakellarios et al. [22, 23] that high concentration of low-density lipoprotein (LDL) can be associated with creation of atherosclerotic plaques.
In this study, three mentioned segments are combined into one system that can provide clinical experts with relevant information and upgrade data obtained from US examination. First deep learning module is used to detect the structures of interest from US images. This data is then processed to extract contours of the vessel lumen and wall and these contours are later on used to within the 3D reconstruction module to generate the geometry of the carotid bifurcation. Finite element simulations are performed using the reconstructed arterial geometries to perform hemodynamic analysis. This complex numerical analysis includes blood flow but also the transport and distribution of molecules relevant for the development of atherosclerotic plaque (LDL, macrophages and cytokines) within the arterial wall. By examining the distribution of these quantities of interest, it is possible to observe parts of the carotid bifurcation with low WSS and high LDL concentration and therefore predict possible risks for further progression of the plaque for the specific patient.
Materials and methods
The scheme of the entire methodology employed within this study is shown in Fig. 1 and details of the methodology will be given in this Section. First, the US examination protocol and the dataset containing US images (shown in Fig. 1a) are described in Section 2.1. These images are annotated and preprocessed as explained in Section 2.2 and shown in Fig. 1b. These pairs of original and annotated images are used for the training of the CNNs. Then, a new (and previously unused) set of US images (shown in Fig. 1c) is used to extract the segments of the CA by using the trained models, as shown in Fig. 1d. Image segmentation methodology is explained in more detail in Section 2.3. Then, the information obtained in the deep learning module is further directed to the reconstruction module, where the relevant shapes of the CA are created, like shown in Fig. 1e. These shapes are further used to generate the finite element mesh of the reconstructed geometry, shown in Fig. 1f. The reconstruction process is explained in Section 2.4. Finally, the finite element mesh is used within the CFD module for the blood flow simulations that are discussed in Section 2.5.
The proposed methodology for the analysis of state of carotid arteries using US imaging.
The US examination was performed within the Clinical Centre of Serbia. This center is a major hospital in the cardiology field that operates in the everyday clinical practice. Examination of the CAs is performed in a supine position of the patient, the neck is in a slight extension, and the head is turned opposite to the examined side of the neck. It begins with an examination in transversal projection. The transducer is placed anteromedially or laterally from the sternocleidomastoid muscle, in the direction of the CA from the supraclavicular fossa, to the corner of the mandible. The origin of the CCA on the left side is deeper and more difficult to visualize. The carotid bulb is identified as a slight extension of CCA at the bifurcation level. After bifurcation, the internal (ICA) and external (ECA) carotid arteries are examined. After scanning in transversal projection, the same areas of the CCA, ICA and ECA are also examined by a longitudinal scan. The examination is performed in B and Color Doppler mode. The measurement of systolic and diastolic flow rates of the CA is also performed. The area of the residual lumen and the area of the original lumen were measured and these numbers were used to determine the degree of carotid stenosis. Also, the length of the plaque on longitudinal images was measured manually.
Using this approach, a dataset of US images has been collected. Overall 108 patients were involved in this study. For each patient an average of 8.7 images was collected, including both longitudinal and trasnversal images. The overall number of images that were included in this study is 939.
This study was performed in line with the principles of the Declaration of Helsinki of 1964 and its later amendments. Approval was granted by the Ethics Committee of University of Belgrade, Faculty of Medicine (Date 27.12.2017./No 29/XII-26). All data protection and safety standards were respected during the collection and processing of the medical data and all images were anonymized. Informed consent was obtained from all individual participants included in the study.
Within the presented approach, the original images were first preprocessed and then these images were annotated by the expert clinicians. Each image was labeled manually by medical doctors with more than 10 years of clinical experience and this ensured the reliability of the annotated segmentation masks. The quality of the annotation was guaranteed by applying a multi-human annotation strategy, i.e. each image was labeled by two doctors independently. If the mean Intersection over Union (mIoU) was less than 0.95 for a particular image, this image was placed back into the unlabeled set and annotated again. If the second annotation for this image was still inconsistent, it was considered as a confused image and excluded from the clinical dataset. This ensured that all labeled images were checked by two annotators. Finally, one of two masks was chosen after the agreement between two annotators. The original and annotated images were further used during the development of the US deep learning module. The main task of this module is to detect and extract the regions of interest.
Image preprocessing
In order to use the US images from the described dataset for the deep learning algorithms, it is necessary to pre-process the images first, because the appearance of the images can vary depending on a patient’s age, or the doctor’s method of recording. Within the preprocessing phase, first the region of the image containing the analyzed the arterial tree is automatically isolated. The isolation consists of choosing a static window with fixed dimensions of 512
Image segmentation methodology
Nowadays, there are many different CNN architectures which can be used for image segmentation task. In this study, three most common architectures in healthcare were considered for the automatic segmentation of the CA lumen and wall: FCN-8s with VGG16 as a backbone classifier [24, 25], SegNet [26], and U-Net [27] deep CNNs. Also, within this study, modified versions of U-Net and SegNet networks were used to analyze the influence of depth on the recognition of the regions of interest. Note that different networks use different number of parameters for training. The SegNet model has around 29M of parameters for training, U-Net modified models use around 60M parameters, which is twice more than SegNet, and more than 130M parameters are used by FCN. However, the U-Net was our first choice because model accuracy is more important than model training time in this particular task. The text below gives a brief description of each network used within this study. More details are available in the original papers from cited literature.
Long et al. [24] proposed an approach for the segmentation of images using the deep learning architecture, more precisely using a fully convolutional network (FCN). In this study a modified version of CNN architectures (including VGG16 and GoogLeNet) was applied for the image classification task. The main modifications consisted in replacing the fully connected layers with the fully convolutional layers, in order to overcome the problem of input and output that didn’t have fixed size. Due to these modifications, the output of the model is a spatial segmentation map instead of standard classification score. FCN combines semantic information and appearance information by using skip connections, and this ensures that the feature maps from the final layers of the model are up-sampled and fused with feature maps of earlier layers. This way, the model produces accurate and more detailed segmentations. However, even though the FCN model is very popular and effective, it is not adequate to perform real-time inference since it is not fast enough, and this represents its main disadvantage. Some improvements came with the U-Net architecture.
The segmentation of medical images has been successfully performed using the U-Net CNN. This deep learning approach is based on encoder-decoder model. Encoder phase is in charge of extracting the features of interest and it comprises of the gradual decrease of the spatial resolution of the image and of the increase of the number of channels. This is done using several convolutional and max-pooling layers. The decoder phase on the other hand, symmetrically performs upconvolution and convolution operations. The main goal of this phase is to double the spatial dimensions of the features, and at the same time reduce the number of channels. This is actually the opposite of the operation within the encoder block, where the image resolution is reduced and the depth increased. Again, the skip connections are used within this model, to improve the quality of the decoder features and achieve more precise segmentation masks. These connections concatenate the features from both the encoder and decoder. This way, the shallow fine appearance information from the encoder and combined with the deep coarse semantic features from the decoder to obtain segmentation masks of the higher quality.
In this study we also used a slightly modified version of standard U-Net architecture. This modified architecture was presented in more detail in [28]. The main modifications are that two more blocks were added in both encoder and decoder, and after each convolutional layer within the model, the batch normalization is used. Also, after each batch normalization layer, ReLU activation procedure is applied.
SegNet architecture is also commonly used for segmentation tasks. This model consists of an encoder-decoder combination. After the decoder phase, a pixel-wise classification layer is used for segmentation. Encoder part consists of thirteen convolutional layers, where batch normalization and ReLU activation are applied. Max-pooling with a 2
Three-dimensional reconstruction
During the 3D reconstruction phase, the characteristics of the patient’s CA are incorporated into the generalized model of the carotid bifurcation presented in literature [30, 31]. The following parameters of the generalized model were adopted in the presented methodology:
The ICA and ECA branches are positioned at an angle of 25∘ with respect to the CCA. The lumen and wall of the ECA are assumed to have circular cross section; the diameter of the lumen is equal to 0.59*dCCA (where dCCA denotes the diameter of the CCA branch), and the diameter of the wall is 25% larger than the diameter of the lumen. The ECA branch has the length equal to 0.5*lICA (where lICA denotes the length of the ICA branch). The CCA branch has the length equal to 1.2*dCCA.
In the clinical dataset that was used within this study there are overall 3 US images containing transversal cuts of the carotid bifurcation – one within the CCA, one within the ICA and one close to the bifurcation. These 3 images are used for the reconstruction, together with the longitudinal cut of the ICA. These images represent the inputs for the trained CNN within the deep learning module (see Fig. 1c) and the segmented lumen and wall areas are obtained as the output of the deep learning module. These extracted segments are shown in Fig. 1d and these same segments are augmented and shown separately in Fig. 2 for better clarity. The segmented data is further used to extract the boundary lines (the red lines on transversal cuts and the blue lines on the longitudinal cut in Figs 1d and 2). The centerline of the ICA branch is extracted using these boundary lines from the longitudinal US cut (centerline is shown in red in Figs 1d and 2). During the extraction of centerline, the diameters of cross-sections of ICA branch are also calculated (and illustrated in yellow in Figs 1d and 2). The boundary lines for the lumen and wall of the transversal cuts are converted to non-uniform B-spline curves and used to define the cross-sections of the branches. The transversal cut from CCA is used to extract the contour that is further used to define all cross-sections along the entire length of this branch. The transversal cut taken close to the bifurcation is used to define the cross-sections of the ICA close to the bifurcation and the transversal cut of the ICA branch is used to define the cross-sections of this branch close to the stenosis (since this cut was taken at that position during US examination). The rest of the cross-sections of the ICA are assumed to have a circular cross-section. The reconstructed cross-sections and centerlines of the branches are shown in Fig. 1e. The final phases within 3D reconstruction are the generation of the NURBS surfaces and subsequently the generation of 3D finite element mesh. These steps are performed using the procedure described in [32] and the obtained 3D mesh for one particular patient is shown in Fig. 1f.
Segmented lumen and wall areas for a particular patient and marked lines used during the 3D reconstruction.
During the FP7 project ARTreat, our team developed a continuum based mathematical model for prediction of plaque growth [19] and implemented it in finite element (FE) software which models blood flow, LDL transport, macrophages, cytokines and plaque development [33, 34, 35]. The simulations using this software are incorporated in this study to depict the distribution of relevant quantities and analyze regions of low WSS and high LDL, since there was a defined correlation between these regions and the probability of further plaque growth. All the equations within the numerical model are presented in the Supplement. These equations are solved numerically using the finite element method (FEM). The reconstructed patient-specific geometry of the carotid bifurcation was represented as finite element mesh, consisting of around 200.000 nodes and around 100.000 finite elements. All the equations of the numerical model were converted in their incremental-iterative form and they are solved in iterations and all quantities of interest are calculated in all nodes of the mesh.
The blood was considered as an incompressible Newtonian fluid, with density and viscosity equal to 1.05 g/cm3 and 0.035 cm2/s, respectively. There are studies in literature that suggest different nonlinear models should be used to define the viscosity of the blood in CFD simulations [36, 37]. But since in this study the blood flow is analyzed in a rather large artery (carotid artery) it was considered that the effect potentially introduced with a nonlinear viscosity could be neglected, as suggested in literature [38, 39]. The numerical model considers the arterial wall to be rigid and nondeformable. Again, similar reasoning is also valid for this assumption – it was reported in [39] that this assumption can be acceptable for large arteries.
Results
GIGABYTE NVIDIA GeForce graphics card GTX 1080 Ti 11GB, GDDR5X, 352bit was used for the development and testing of the entire proposed metholodogy. For the deep learning module, Python V3.6.7 was used and Keras framework was used for the implementation of the related code, while Tensorflow framework was used as the backend. The 3D reconstruction module was developed in C
Results of the deep learning module
As it was already mentioned, all implemented methods are based on TensorFlow 2, an end-to-end open-source platform for machine learning. The effectiveness of the tested models was evaluated using 5-fold cross-validation strategy. In order to increase the number of images in the training set, all available samples were processed using standard data augmentation techniques (translation, mirroring, shifting, zooming and flipping).
The loss function that represents a combination of binary cross-entropy and soft dice coefficient [29] was used during the training of the model and it is defined as:
where
Five different models were tested: original and modified (deeper) version of the U-Net architecture, original SegNet and SegNet with pretrained VGG weights, as well as FCN-8s with VGG-16 backbone. The models are trained by the Adam optimizer with 0.9 Nesterov momentum from scratch, with batch size 8. The value of the learning rate is initialized to 1e-4. The optimal number of epochs should be defined for the model, to ensure that the overfitting of the model is minimized and the generalization capacity of the model is maximized. The moment when the model starts overfitting is determined based on the values of loss and accuracy of both training and validation sets. In our case, EarlyStopping strategy was used to monitor model training phase, and approximately 50 epochs are needed to achieve the best performance.
The resulting loss curves during the training and validation phases, for one fold, are shown in Fig. 3a, while Dice coefficient curves are shown in Fig. 3b. The solid and dashed lines represent the training and validation phase, respectively. These graphics are obtained when the models are trained over the dataset that consists of lumen images obtained from B-mode transversal cut. Similar graphics are obtained for other datasets (longitudinal cut, lumen and wall).
Obtained results of the considered models during the training and validation phases; a – resulting loss curves; b – Dice coefficient curves.
It could be seen in Fig. 3b that the highest Dice coefficient value was achieved by models FCN-8s with VGG-16 backbone and Deeper version of U-Net. It should also be noted that for the dataset used in this study better results were obtained using the deeper version of the U-Net model in comparison to the original U-Net model.
The Deeper U-Net version was selected as a trade-off model which achieved the highest Dice coefficient value on validation and test datasets compared to other models, although it uses twice as many parameters compared to SegNet whose Dice coefficient is also relatively high. We justify this decision by the fact that we need more precise segmentation for more reliable 3D models. On the other hand, the Deeper U-Net uses almost two times less parameters during training in comparison to FCN-8s VGG-16 with batch normalization model and this gives an advantage to the first over the second model. SegNet with pretrained VGG weights was not considered due to poor results in both training and validation phases.
As mentioned above, the 5-fold cross validation strategy was used to analyze the performance of mentioned models and to compare them with each other. The average quantitative results of different methods on transversal US images for the binary classification task of the lumen of CA are shown in Table 1. The Deeper U-Net achieved the highest Dice coefficient value among all models tested, justifying our choice of this model as the one that is used further within the methodology.
Performance of different methods on transversal US images for the lumen segmentation task
Binary classification task for image segmentation was again used to evaluate the performance of the chosen CNN and the developed deep learning module. If we consider the segmented lumen (or wall) region as positive and background as negative and define terms true positive (TP), true negative (TN), false positive (FP) and false negative (FN), we can define the following three classification metrics that are commonly used in literature for the quantitative evaluation of the performance of deep learning models – precision (P), recall (R), and Dice coefficient (DC):
As it was previously mentioned, overall 5 different models were developed, for 5 different types of US images – transversal B mode US images and Color Doppler US images were used for segmentation of both lumen and wall and the longitudinal US images. The average of the performance metrics reported by 5-fold cross-validation for all considered models is shown in Table 2. High average values of Dice coefficient for each of the developed models confirm their segmentation ability.
Results of the performance of the deep learning module
Example of an original and segmented image with borders for a case with noise included within US image at one end of the considered vessel.
Illustration of the process of plaque determination; a – original US image with overlayed segmented lines for lumen (blue) and wall (red), extracted centerline (yellow) and clinically marked plaque length (green); b – graphs showing the change of vessel diameters along the centerline.
Validation of the results of the deep learning and reconstruction module; a and b – linear regression and Bland-Altman plots for the validation of percentage of stenosis; c and d – linear regression and Bland-Altman plots for the validation of plaque length.
Segmentation results for a specific patient. The original US images are shown in the first row; predicted lumen regions are shown in the second row; predicted wall regions are shown in the third row; the segmented data on top of original US images is shown in the fourth row.
Results of numerical simulation for a specific patient – fluid flow; a – generated mesh with arterial wall; b – velocity streamlines; c – pressure distribution; d – WSS distribution.
Results of numerical simulation for a specific patient – distribution of quantities in the arterial wall; a – Macrophages concentration; b – Cytokines concentration; c – Oxidized LDL distribution.
Note that the model for longitudinal images performed slightly worse, because the boundaries for lumen and wall are sometimes not clear. Significant noise, reverberation, shadowing and sometimes artifacts are incorporated in these images and sometimes it is difficult even for medical experts to recognize the exact location of these boundaries (as illustrated in Fig. 4). Other models applied for the transversal US images showed high performance in recognizing the regions of interest. This problem with recognition of boundaries in longitudinal images caused smaller accuracy of this model, but has however no influence on the reconstruction module, since the extracted lines are then further processed to extract relevant diameters.
The clinicians measured two parameters during US examination – the length of the region of the artery containing plaque and the percentage of stenosis. The area of the residual lumen on the ICA transversal cut and the area of the original lumen were measured by the clinical device during US examination and the percentual ratio of these two numbers represents the percentage of stenosis. This percentage was provided together with the US images within the clinical dataset. The plaque length is determined from the ICA longitudinal image, as the length of the ICA where the stenosis is present. These two parameters were calculated during the 3D reconstruction as well, in order to validate the segmentation and reconstruction modules of the proposed methodology. As it was already mentioned, the diameters of the ICA are calculated from the segmented longitudinal images within the reconstruction module. The diameters are measured approximately on every 0.2 mm. The ratio between diameters of wall and lumen is analyzed along the centerline of the ICA branch and the plaque start and end locations are determined based on this ratio. Namely, it is considered that the plaque is located on the segment of the ICA where the gradient of mentioned ratio has a significant change of value (more than 20%). This process is shown in Fig. 5. The original US image with overlayed segmented lines is shown Fig. 5a. The segmented lines for lumen are colored in blue, while segmented lines for wall are colored in red. The centerline that was extracted from this data within the reconstruction module is shown as the dashed yellow line. The plaque that was marked by the clinical experts is shown with green line. The graphs of change of lumen and wall diameters, as well as the change of wall/lumen ratio are shown in Fig. 5b. The yellow arrows illustrate the positions of plaque start and end that are detected using the previously described procedure.
Overall 20 patients from the testing dataset are considered for the validation of the methodology and Fig. 6 shows the obtained results. The percentage of stenosis obtained after reconstruction shows good correlation with clinical measurements and the parameter of linear regression is equal to
Example of patient-specific 3D reconstruction and blood flow simulation
The results of the segmentation within the deep learning module for a specific patient are shown in Fig. 7. The original US images are shown in the first row, the predicted images for the lumen are shown in the second row, while the predicted images for the wall regions are shown in the third row. The segmented data is overlayed to the original US images in the fourth row, to enable better visualization of the extracted data. The results of the blood flow simulation for the same patient are shown in Fig. 8. Figure 8a shows the reconstructed geometry, and the remaining images in this figure show the distribution of velocity, pressure and WSS. The results of the plaque progression simulation are shown in Fig. 9. The distribution of relevant quantities within the arterial wall is shown for several slices of the 3D model – macrophages distribution is shown in Fig. 9a, cytokines distribution is shown in Fig. 9b and distribution of oxidized LDL is shown in Fig. 9c. These slices were manually angulated normal to the branches and chosen in all segments of the carotid bifurcation. As it can be observed from Figs 8b and 9c, the regions of the CA with low values of WSS and high values of LDL mostly coincide and are located close to the already stenotic segment of the ICA, indicating that this is the region where plaque will further progress.
Discussion
The segmentation and classification of medical images has been extensively studied in literature. Per example, several algorithms for the classification of plaque types from computed tomography angiography (CTA) images are discussed in [40]. The authors applied diverse classification techniques including decision tree, linear discriminant analysis, quadratic discriminant analysis, support vector machine, k-nearest neighbor, and probabilistic neural network (PNN), but they did not consider deep learning techniques such as CNNs. Deep CNNs were applied in [11] to perform automatic segmentation of intraluminal thrombosis of abdominal aortic aneurysm from CTA images and good prediction results were reported. CNNs were also used in [12] to segment coronary arteries from CTA images and the authors reported that the presented model achieved the Dice coefficient of 0.8942. Densely connected convolutional neural networks (DenseNets) were used in [41] to estimate the thickness of the carotid intima-media region and detect plaque. The authors reported a correlation coefficient of 0.81 and the accuracy for plaque detection of 0.9645 in CCA. Saba et al. [15] applied classical CNN to separate plaque segments from US images and performed the classification of symptomatic or asymptomatic plaque. They reported the accuracy of 0.8617 and 0.897 using two considered techniques. Dynamic CNN and U-Net models were used in [16] to segment regions of interest of the carotid artery from 3D US images. The segmentation results were compared with manual segmentations performed by a clinical expert and the average Dice coefficients were 0.9646 and 0.9284 for the two considered regions. Park et al. [17] presented an integrated approach that performs automatic wall segmentation from longitudinal US images and estimates tissue motion and applies specific flow measurement techniques to analyze blood flow within murine carotid arteries. The authors used U-net based neural networks and reported mean Dice coefficient of over 0.94 for the synthetic test images and over 0.90 for the in vivo test images. Jain et al. [18] analyzed several architectures, including UNet and several modifications, to identify carotid plaques in ICA and CCA longitudinal US images. The obtained values of Dice coefficient were between 0.8494 and 0.9002. Automatic segmentation of atherosclerotic plaques within carotid arteries from US images was performed in [13], using several deep learning networks – the traditional U-Net network, the ResUNet network and the newly proposed combined U-Net and DenseNet network. The authors reported that the combined Dense-UNet network had the best performance. In this study the automatic segmentation of US images was performed. Several deep learning architectures were tested, including SegNet and UNet architectures. Also, a modified deeper UNet model was used, where the number of blocks in both the encoder and decoder phases was increased by two. This Deeper UNet model was the one that resulted to have the best performance metrics and was therefore used within the proposed methodology to extract lumen and wall regions. The values of Dice coefficient for the models presented in this study ranged from 0.84 for longitudinal US images to 0.98 for the transversal US images. These results are comparable to the previously discussed results from literature. However, what differentiates the present study from other similar studies in literature [13, 16, 41] is that the segmentation was not the final goal and instead the results were further employed to perform both 3D model generation and numerical simulations of blood flow and distribution of molecules relevant for the development of atherosclerotic plaque.
The 3D visualization of the CA would be rather straight-forward if a series of cross-sectional B-mode US images were available. The B-mode US images were used by Rosenfield et al. [42] to perform 3D reconstruction of the carotid bifurcation. 2D transversal US images were also used more recently by Yeom et al. [43] to perform 3D reconstruction. In this study a similar approach is used, but the number of US images was limited due to the procedure applied during clinical examination and the lack of transversal cuts has been overcome using the parameters from the generalized CA model.
Blood flow through CAs has been studied in literature, using both in-vivo data and numerical simulations. Markl et al. [44] analyzed in vivo distribution of WSS in the carotid bifurcation using time resolved phase-contrast MRI with 3-directional velocity encoding. Morales et al. [45] performed an analysis of morphological and hemodynamic patterns of stenosed CA after diverse clinical treatments using duplex US. Zhang et al. [46] analyzed blood flow in stenosed CA of one patient using digital subtraction angiography (DSA). Color Doppler ultrasonography was applied in a study [47] that involved patients with head and neck cancers. The main goal was to analyze the vascular changes and subsequent blood flow abnormalities that appear after external radiotherapy. The benefits of Doppler ultrasound technique for the examination of intracranial vascular stenosis were analyzed in [48]. In this study, patients with a history of stroke underwent standard examination techniques together with the transcranial colour Doppler ultrasound and it was concluded that US technique can be succesfully applied for the diagnosis of arterial stenosis. Blood flow through anatomically realistic carotid artery bifurcation was modelled in [49], both with rigid and deformable walls. The MRI scan was used in [40] to reconstruct the geometry of the patient’s CA using commercial image analysis software. This model was used for CFD simulations and the obtained results were compared with values measured from phase-contrast MRI examination. In another study [51] the authors used semi-automatic threshold-based segmentation to reconstruct the geometries of the patient-specific CAs from CT scan images. They also performed US examinations to quantify the blood flow parameters used for the boundary conditions defined within CFD simulations. The main goal was to analyze hemodynamic parameters after two diverse clinical treatments of stenosed arteries. Hemodynamic analysis of an ascending human aorta with and without aneurysm was performed using FEM in [52]. The authors performed blood flow simulations of a cardiac cycle and compared the values of WSS for both considered cases. He et al. [53] also performed FEM simulations of blood flow to examine the influence of the location of stenosis on the distribution of blood velocity, pressure and WSS. In this study the FEM simulations are extended from the standard blood flow simulations performed in [52, 53] and many other papers in literature, into a more complex hemodynamic analysis. The applied complex model provided a way to analyze not only the distribution of blood flow parameters, but also other relevant parameters within arterial wall, such as LDL, macrophages and cytokines.
The main contributions and advantages of the proposed methodology are as follows:
The deep learning techniques are applied for the segmentation of the US images, where a completely automatic detection of relevant regions of the carotid bifurcation (lumen and wall) was performed. The experiments demonstrated that a deep learning approach can provide very accurate image segmentation results in real-time, which is a very important segment in computer-aided diagnosis. The performance of the deep learning technique was demonstrated through the high values of relevant common classification metric parameters (Precision, Recall and Dice coefficient). Also, the accuracy of the proposed methodology was shown through the validation of results for the reconstructed parameters (stenosis and plaque length) against the clinically measured values. The segmented data extracted from US images is used to reconstruct the vessel geometry in 3D. A 3D geometry of the considered vessel that can be obtained using the approach proposed in this paper is more relevant and appropriate in comparison to the 2D images obtained during US examination. One of the reasons is that 3D geometry enables better visualization of the state of the CA, but a more important reason is related to hemodynamics. The clinical examinations that rely on the 2D Doppler US analyze the blood velocity and WSS either in the axial direction or as an average of the entire lumen cross-section. But, the blood flow parameters, including velocity and WSS have complex directions in 3D. Thus it is preferable to perform analysis of these parameters using their actual magnitude and direction. The reconstructed geometry is used to perform hemodynamic simulations. This complex analysis includes blood flow but also the transport and distribution of molecules relevant for the development of atherosclerotic plaque (LDL, macrophages and cytokines) within the arterial wall and provides a way to investigate several quantitative hemodynamic parameters of the stenosed CA.
The main limitation of this methodology is the database that consisted of US images from 108 patients. Also, the dataset consisted of images from only one clinical center. In order to ensure higher representativeness of the US images, the dataset should be enlarged in future studies by including data from other clinical centers. Another problem that may arise from including more clinical centers could be the fact that different US devices produce images of slightly different features. When more versatile data from diverse US devices were included, it would enable to test the scalability of the proposed methodology. Another future improvement will be to train the deep learning module to segment also the plaque components (fibrous, lipid and calcified plaque) from the US images.
There are also some possible ways to additionally improve the performance of the deep learning module. Hongchun et al. [54] proposed an approach to improve the segmentation problems that unclear image boundaries can cause. The boundary-enhanced guided packet rotation dual attention decoder network was proposed to increase segmentation accuracy while reducing the number of model parameters. Yongtao et al. [55] also proposed some modifications of the U-Net architecture that would ensure better feature expression and improve segmentation performance of the model by 0.75%. These proposed improvements and some others should be considered in the future improvements of the methodology presented in this study.Another limitation is the rather small number of US transversal images that are used during reconstruction. This again was mainly due to the data that the used clinical dataset contained. However, if observed differently, this can also be seen as a benefit of the proposed methodology. This model is capable of performing reconstruction using assumptions from the generalized model when there is a lack of US images obtained during examination. If there are more US transversal images available, they can be easily included in the reconstruction to obtain a more personalized model of the patient’s CA. This enables the clinicians to determine the important segments that differ from the generalized model and include them in the modeling phase, thus lowering the time and cost of the US examination.
In the presented methodology the shapes of CCA and ECA are completely generic instead of being specific for the particular patient. With introducing new clinical datasets with more US images for the specific patient, this limitation will be overcome and these branches will also be segmentedin the future improvements, which will ensure an even more realistic 3D model.
Conclusion
The methodology presented in this study enables the segmentation and 3D reconstruction of patient-specific CA without any assistance or manual intervention of the clinical staff. The presented methodology could be used in combination with standard clinical ultrasound examination to quickly provide additional quantitative and qualitative information about the state of the patient’s CA (including information that cannot be easily measured, such as the WSS or distribution of molecules of interest within arterial wall) and thus ensure a treatment that is more adapted to the specific patient.
Funding
The research presented in this study was part of the project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 755320-2 – TAXINOMISIS. This article reflects only the author’s view. The Commission is not responsible for any use that may be made of the information it contains. This research is also funded by the Serbian Ministry of Education, Science, and Technological Development [451-03-47/2023-01/200378 (Institute for Information Technologies, University of Kragujevac) and 451-03-47/2023-01/200107 (Faculty of Engineering, University of Kragujevac)]. All listed funding sources had no involvement in the study design; in the collection, analysis and interpretation of data; in the writing of the manuscript; and in the decision to submit the manuscript for publication.
Supplementary data
The supplementary files are available to download from https://dx-doi-org.web.bisu.edu.cn/10.3233/THC-231306.
Footnotes
Acknowledgments
The authors have no acknowledgments.
Conflict of interest
The authors declare that they have no conflict of interest.
