External Validation of Fully-Automated Infrarenal Maximum Aortic Aneurysm Diameter Measurements in Computed Tomography Angiography Scans Using Artificial Intelligence (PRAEVAorta 2)

Study Design

This is a multicenter, retrospective validation study using prospectively collected data from the Zenith alpha for aneurysm Repair Registry (ZEPHYR). Core laboratory assessed maximum AAA diameter measurements were compared with fully-automated measurements generated by a deep-learning–based software (PRAEVAorta 2).

Data Set

All available images from the ZEPHYR registry at the time of the initiation of the present study were included. A total of 871 CTA images were obtained from patients with a juxtarenal or infrarenal AAA who had multidetector CT scans of the entire abdominal aorta and arterial phase intravenous injection of contrast medium. Patients were recruited for the ZEPHYR registry between 2016 and 2019. Images were provided in Digital Imaging and Communications in Medicine (DICOM) format with a matrix size of 512 × 512 for each slice. Data were de-identified prior to the upload on the core lab’s medidata platform. Scans were performed according to local protocols in all participating sites. The study protocol was approved by the local Institutional Review Board (S-652/2023). Informed consent for the purposes of this study was waived. Informed consent for the ZEPHYR registry was obtained from all patients.

Core Laboratory Diameter Measurements (Syntactx)

Preoperative and postoperative CTA scans were uploaded via a web browser to the medidata platform. Among others, the core laboratory performed measurements of the outer-to-outer maximum diameter of the abdominal aortic aneurysm on a slice orthogonal to the outer wall of the aorta in multiplanar reconstruction images (MPRs). A representative image of the methodology is displayed in Supplemental Figure 1. Measurements were performed in TeraRecon iNtuition (Fremont, California). The reviewers are measurement technologists trained to and utilizing established measurement standards, overseen by vascular surgeons and radiologists. Each image is reviewed by an initial reviewer with a quality check by a second reviewer. The images were collected over a 21 CFR part 11 compliant image transfer system.

PRAEVAorta 2 (Nurea)

All available CTA scans were uploaded to an offline research-only version of the PRAEVAorta 2 software installed at one of the participating study sites (Software version 20230321). The method of aneurysm segmentation consists of 4 sequential steps: (1) image preprocessing, (2) segmentation of the aortic lumen and thrombus, (3) centerline generation, and (4) parameter computation. Models were created using the PyTorch framework. The models designed by PRAEVAorta 2 are based on convolutional neural networks (U-Net). Image preprocessing was fully-automated by the software. Standard image filtering and denoising algorithms for medical imaging were used. The maximum transverse diameter was defined as the maximum diameter computed in the planes orthogonal to the centerline, including lumen and thrombus. The software divides the aorta in distinct segments by planes orthogonally cutting the centerline at the level of predefined anatomical landmarks. The infrarenal segment is defined proximally by the lowest point of the lowest renal artery, orthogonally cutting the centerline. The distal end of the infrarenal segment is defined by the plane orthogonally cutting the centerline at the level of the right and left common iliac artery ostium.

The application offers a dashboard view displaying all previously uploaded scans. Custom software that was provided for the purposes of this study by Nurea enabled the queuing of larger badges of CTA scans, thereby enabling to analyze multiple scans without further intervention of the user. The analysis of each scan requires about 1 to 5 minutes, depending on the number of images per scan. After selection of the DICOM CTA images, no further intervention of the user was required. The software then produces a report in pdf format for every scan that can be accessed through the dashboard view within the application. Larger sets of results can be exported in tabular format as well. An excerpt of the report is presented in Supplemental Figure 2. A full report is presented in Supplementary Material 1. The software recommends for the user to visually review the report to verify plausibility of the segmentation and provides 2-dimensional and 3-dimensional visualization of the segmented volumes, including an overview and imaging slices, from which the measurements are derived.

Statistical Analysis

Statistical analysis was performed using “R” (version 4.3.1, 64 bit, Vienna, Austria). ¹⁶ For the validation of the PRAEVAorta software’s accuracy in measuring the maximum AAA diameter (outer-to-outer), descriptive statistical analysis is used with mean values and standard deviations (SDs). Furthermore, a correlation of the manual and fully-automated diameter measurements is calculated as well as a Bland-Altman plot and intraclass correlations (ICCs). The ICC is presented with 95% confidence intervals (95% CIs). An absolute, maximum difference of <5 mm was defined as clinically acceptable, considering interobservership differences in the literature.^5,7,17–19

Imaging Data

There were 871 contrast CT scans available for the purposes of this study displaying the full abdominal aorto-iliac anatomy in arterial phase. These 871 scans were analyzed fully-automatically by the PRAEVAorta software. After analysis, generated pdf reports were independently assessed by 2 observers (J.H., A.B.). Subsequently, 87 scans (10.0%) were excluded for obvious segmentation errors due to misdetection of the aortic segments or misdetection of the outer circumference of the aortic wall. Representative images of excluded scans are presented in Figure 1. The remaining 784 scans were matched with the corelab data set, which resulted in 762 available matched pairs for analysis. For 22 scans (2.8%), there were no corelab measurements available at the time of data curation for this study.

OPEN IN VIEWER

Figure 1.

Scans with obvious errors in segmentations (N=87/871, 10.0%). All reports were scanned for these errors and independently assessed for plausibility by 2 observers (J.H., A.B.). Obviously flawed segmentations were excluded from further analysis.

There were 263 preoperative scans and 499 postoperative scans. Most scans had a slice thickness of 1 to 3 mm (N=477, 62.6%), followed by a slice thickness of <1 mm (N=213, 30.0%) and a slice thickness of >3 mm (N=72, 9.4%). The mean slice thickness across the included scans was 1.2 mm (SD=0.7). The excluded scans had a slice thickness of 1.4 mm (SD=0.9) with 33 scans <1 mm (37.9%), 41 with 1 to 3 mm (47.1%), and 13 with >3 mm (15.0%). A flowchart presenting included scans is presented in Figure 2.

OPEN IN VIEWER

Figure 2.

Flow chart displaying the data set of the study.

PRAEVAorta 2 vs Core Laboratory

The mean aortic diameter across all scans as measured by core laboratory and PRAEVAorta 2 was 58.8 mm (SD=10.0) and 58.5 mm (SD=10.2), respectively. The mean absolute difference on a per scan basis between core laboratory and PRAEVAorta 2 was 0.28 mm (SD=2.4). 727 of 762 (95.4%) measurements showed a clinically acceptable accuracy with an absolute difference of <5 mm. A total of 628 (82.4%) scans showed a measurement difference of <3 mm. There was a strong correlation between the measurements provided by PRAEVAorta and the core laboratory with a correlation coefficient of 0.97. The correlation is displayed in Supplemental Figure 3. A total of 727 analyzed matched pairs (95.4%) were within a range of 5 mm in difference. The Bland-Altman plot is presented in Figure 3. The limits of agreement (95% confidence interval) were −4.36 mm for the lower and 4.93 mm for the upper limit of agreement with a bias of 0.28 mm.

OPEN IN VIEWER

Figure 3.

Bland-Altman plot presenting the agreement between both methods. The lower limit of agreement was −4.36 mm, the upper limit was 4.93 mm with a bias of 0.28 mm. In total, 95.4% of measurements were within a ±5 mm absolute difference.

Intraclass Correlation

The ICC for all included scans was 0.972 (95% CI=0.968-0.976). For preoperative CT scans, ICC was 0.953 (95% CI=0.941-0.963), and for postoperative scans, ICC was 0.979 (95% CI=0.975-0.983), respectively. Violin plots are displayed in Figure 4.

OPEN IN VIEWER

Figure 4.

Violin plots. The intraclass correlation for preoperative scans was 0.953, and for postoperative scans, it was 0.979. For preoperative and postoperative scans combined, it was 0.972.

Limitations

This study presents with several limitations. Although it was a multicenter study with many included scans from an international registry, patients were treated for infrarenal abdominal aortic aneurysm with standard bifurcated endovascular repair (Zenith Alpha Abdominal Endovascular Graft, Cook Medical, Inc). Therefore, results might not be generalizable to situations with aneurysms following non-standard EVAR, such as fenestrated or branched procedures. In addition, this study only investigated infrarenal maximum AAA diameter, and results are not applicable to other aortic segments, in contrary to other current AI tools. ¹² Slight differences in definition of core laboratory and PRAEVAorta 2 measurements prevented the external validation of additional measurements, such as neck or iliac morphology. Furthermore, while an external core laboratory, blinded for the software’s measurements with internal quality checks, was used as the gold standard in this study, human errors in establishing gold standards for the validation of any AI tools cannot be excluded with absolute certainty.