Higher agreement in endovascular treatment decision-making than in parametric quantifications among automated CT perfusion software packages in acute ischemic stroke

Abstract

BACKGROUND AND OBJECTIVE:

Estimates of parameters used to select patients for endovascular thrombectomy (EVT) for acute ischemic stroke differ among software packages for automated computed tomography (CT) perfusion analysis. To determine impact of these differences in decision making, we analyzed intra-observer and inter-observer agreement in recommendations about whether to perform EVT based on perfusion maps from 4 packages.

METHODS:

Perfusion CT datasets from 63 consecutive patients with suspected acute ischemic stroke were retrospectively postprocessed with 4 packages of Minerva, RAPID, Olea, and IntelliSpace Portal (ISP). We used Pearson correlation coefficients and Bland-Altman analysis to compare volumes of infarct core, penumbra, and mismatch calculated by Minerva and RAPID. We used kappa analysis to assess agreement among decisions of 3 radiologists about whether to recommend EVT based on maps generated by 4 packages.

RESULTS:

We found significant differences between using Minerva and RAPID to estimate penumbra (67.39±41.37mL vs. 78.35±45.38 mL, p < 0.001) and mismatch (48.41±32.03 vs. 61.27±32.73mL, p < 0.001), but not of infarct core (p = 0.230). Pearson correlation coefficients were 0.94 (95%CI:0.90–0.96) for infarct core, 0.87 (95%CI:0.79–0.91) for penumbra, and 0.72 (95%CI:0.57–0.83) for mismatch volumes (p < 0.001). Limits of agreements were (–21.22–25.02) for infarct core volumes, (–54.79–32.88) for penumbra volumes, and (–60.16–34.45) for mismatch volumes. Final agreement for EVT decision-making was substantial between Minerva vs. RAPID (k = 0.722), Minerva vs. Olea (k = 0.761), and RAPID vs. Olea (k = 0.782), but moderate for ISP vs. the other three.

CONCLUSIONS:

Despite quantitative differences in estimates of infarct core, penumbra, and mismatch using 4 software packages, their impact on radiologists’ decisions about EVT is relatively small.

Keywords

Acute ischemic stroke computed tomography perfusion (CTP)endovascular thrombectomy (EVT)estimates of infarct core penumbra and mismatch

1 Introduction

Growing evidence supports the effectiveness of endovascular thrombectomy (EVT) in treating patients with acute ischemic stroke due to large-vessel occlusion [1 –7]. Imaging-based selection of patients likely to benefit from EVT is fundamental for research and clinical practice, especially since the DAWN [8] and DEFUSE 3 [9] trials showed that EVT is highly beneficial beyond the 6-hour window in patients selected with perfusion computed tomography (CT) or perfusion magnetic resonance imaging.

Perfusion CT data are useful for identifying and quantifying the infarct core (irreversibly damaged tissue) and penumbra (potentially salvageable tissue); this information is essential for selecting appropriate treatments for individual patients and determining the time window for different treatment options [10]. However, the applicability of perfusion CT data in acute ischemic stroke has been impaired by a lack of standardized acquisition and postprocessing protocols. Software packages for postprocessing perfusion CT data use diverse algorithms to quantify the relevant parameters, and there is no consensus on the best approach [11 –15]. It is crucial to ensure that volumes of infarct core and penumbra quantified by software packages from different vendors are comparable and accurate, especially when selecting patients for EVT. The DAWN [8] and DEFUSE 3 [9] trials selected patients based on infarct core volumes estimated with RAPID software (version 2017; iSchemaView Inc, Menlo Park, CA, USA), and these landmark interventional trials demonstrated the utility of this software package despite the lack of validation against a firm gold standard. RAPID software is been used in nearly 2000 worldwide (https://www.rapidai.com/). The Olea Sphere 3.0.12 software (Olea Medical Solutions, La Ciotat, France) is a commercially available and FDA-approved automated software adopted by more than 200 clinical and research institutions (https://www.olea-medical.com/en). IntelliSpace Portal (ISP) CT Brain Perfusion is another software package for postprocessing perfusion CT data used worldwide (https://www.philips.co.in/). Minerva (version 2020, Lincbiotech, Santiago de Compostela, Spain) has not been formaly tested in academic studies or trials. Comparative studies have found that software packages’ measurements of infarct core, penumbra, and mismatch volumes can differ considerably; however, the impact of these differences on clinical decisions is unknown.

Thus, we evaluated the volume of infarct core, penumbra, and mismatch estimated by two postprocessing software packages [RAPID and Minerva] in a cohort of patients with acute ischemic stroke. Furthermore, we evaluated the impact of CT perfusion maps elaborated with four software packages [RAPID, MINERVA, Olea Sphere 3.0.12, and ISP (version 7.0; Royal Philips Healthcare, Best, The Netherlands)] on radiologists’ decisions to indicate EVT.

2 Material and methods

2.1 Patients

We retrospectively selected consecutive patients with suspected acute ischemic stroke who underwent CT perfusion imaging between June 2019 and December 2019 and met the following inclusion criteria: clinical diagnosis of acute ischemic stroke, deficit on the National Institutes of Health Stroke Scale (NIHSS)≥2 points, and no hemorrhage on CT. We excluded patients with severe motion artifacts or poor-quality CT scans. Our university hospital’s ethics committee approved this retrospective analysis of anonymized patient data, waiving informed consent because the analysis concerned only routine clinical imaging studies and the retrospective image evaluation could not influence clinical decisions. A total of 63 patients were included. Fifty-two patients (82.53%) were treated with EVT.

2.2 Imaging acquisition

All CT examinations were performed on a 128-slice scanner (Ingenuity; Philips Healthcare, Best, the Netherlands) using the following parameters to acquire perfusion datasets: cine scanning mode, 100 kVp, 100 mAs, and four 10 mm slices centered on the basal ganglia. Acquisition began 5 seconds after starting injection of a 50 mL bolus of iodinated contrast agent (Ultravist 300 mgI/mL; Bayer HealthCare Pharmaceutical Inc., Leverkusen, Germany) into an antecubital vein at 5 mL/second with a power injector. We acquired series of 25 scans every 2 seconds. The total acquisition time was 50 seconds.

2.3 Imaging postprocessing for automated PCT software packages

All datasets were postprocessed with four fully automated software packages: RAPID (version 2017; iSchemaView, Menlo Park, CA, USA), Olea Sphere 3.0.12 (Olea Medical Solutions, La Ciotat, France), IntelliSpace Portal (ISP) CT Brain Perfusion (version 7.0; Royal Philips Healthcare, Best, The Netherlands), and Minerva (version 2020, Lincbiotech, Santiago de Compostela, Spain). All software packages used automated registration, segmentation, and motion correction. Cerebral blood volume (CBV) is defined as the total volume of flowing blood in a given volume in the brain, with units of milliliters of blood per 100 g of brain tissue. cerebral blow flow (CBF) refers to the volume of blood moving through a given volume of brain per unit time, with units of milliliters of blood per 100 g of brain tissue per minute. Mean transit time (MTT) is defined as the average transit time of blood through a given brain region, measured in seconds. On the other hand, maximum value of the residual function after deconvolution (Tmax) reflects the time delay between the contrast bolus arriving in the proximal large vessel arterial circulation (arterial input function) and the brain tissue. Mismatch is defined as the difference between infarct core and penumbra.

RAPID uses a delay-insensitive algorithm in which the infarct core was defined as the area in which cerebral blow flow (CBF) is < 30%of the CBF in normal tissue and the penumbra as the area in which the time to the maximum value of the residual function after deconvolution (Tmax) is > 6 seconds. By contrast, in Olea Sphere 3.0.12 (Olea Medical Solutions, La Ciotat, France) the algorithm to define the infarct core is based on relative CBF (rCBF). Two thresholds are recommended (1) rCBF < 30%, which matches RAPID [16] and (2) rCBF < 40%, which is considered the default and most accurate setting for detecting an acute infarct [13]; both these thresholds are used in conjunction with Tmax > 2 seconds to rule out old infarcts. Olea Sphere 3.0.12 defines the penumbra as the area in which Tmax is > 6 seconds. ISP uses a delay-sensitive algorithm with 3D motion correction and filtering based on relative mean transit time (rMTT) and cerebral blood volume (CBV), where the infarct core is defined as rMTT > 1.45 and CBV < 2.0 mL/100gr and the penumbra is defined as rMTT > 1.45 and CBV > 2 mL/100 gr. ISP requires manual 3D motion correction and filtering. By contrast, Minerva incorporates automatic motion correction and uses a delay-insensitive algorithm that defines the penumbra as the area in which Tmax is > 6 seconds. To define the infarct core, Minerva uses a specialized iterative and statistical method to determine the best threshold from CBV maps. This relative, patient-dependent threshold is calculated by comparing CBV values within the penumbra versus those found in the unaffected brain tissue.

2.4 Study design

We used two approaches to compare the software packages. First, we quantitatively compared the volumes of infarct core, penumbra, and mismatch measured with RAPID and Minerva. Next, to determine the impact of the software package on the decision to perform EVT, three radiologists blinded to the clinical data and to any other imaging information from non-contrast CT (NCCT) or CT angiography independently evaluated the presence of mismatch between infarct core and penumbra on maps generated by the RAPID, Olea, ISP, and Minerva software packages. Therefore, all cohort of 63 cases were scored consecutively using RAPID. After this reading, the cases were evaluated using Olea, ISP, and finally Minerva. After a qualitative evaluation of the perfusion maps to assess vascular anatomic consistency of the ischemic lesion, radiologists classified their recommendations for EVT in each case based on the volume of infarct core and penumbra in the maps into four categories: (1) recommended (infarct core volume≤70 ml and≥20%of penumbra volume); (2) not recommended (infarct core volume > 70 ml and < 20%of penumbra volume); (3) questionable (infarct core volume > 70 ml and≥20%of penumbra volume); or (4) not applicable (no abnormalities or inconsistent findings that do not correspond to the arterial vascular territory identified). Each radiologist scored each map on two separate occasions two weeks apart. Intra-observer and interobserver reliabilities were calculated. Final consensus was obtained among the three observers after reviewing the case again if there was any discordant rating about whether to recommend EVT based on maps generated by any of four packages.

2.5 Statistical analysis

We report continuous variables as means and standard deviations when normally distributed and as medians and interquartile ranges when non-normally distributed; we used the 1-sample Kolmogorov-Smirnov test to determine whether variables were normally distributed. To compare the volumes of infarct core, penumbra, and mismatch measured with RAPID and Minerva, we calculated the means and standard deviations of the absolute and relative differences between measurements and compared them with the Wilcoxon signed-rank test (for skewed data) or paired sample t-test (for normally distributed data). The relative difference between the volumes measured was defined as the ratio of the difference in volume to the average volume and was represented as a percentage. To assess the relation between measurements with the two software packages, we used scatterplots and linear regression with Spearman correlation tests for non-normally distributed data and Pearson correlation tests for normally distributed data. To determine the agreement between volume measurements with the two packages, we carried out a Bland-Altman analysis, calculating the systematic error (bias) and the 95%limits of agreement, defined as bias±1.96 of the individual differences. To check for proportional bias between the two methods of measurement, we used linear regression of differences; if the two methods are equally variable, the slope of the linear regression line is zero. Statistical significance was set at p < 0.05.

To determine the interobserver and intra-observer agreement in the clinical recommendations based on the maps generated from the four automated software packages, we used the kappa coefficient, considering k≤20 slight agreement, 0.20 < k≤0.40 fair agreement, 0.40 < k≤0.60 moderate agreement, 0.60 < k≤0.80 as substantial agreement, and k > 80 nearly perfect agreement [17].

3 Results

Minerva successfully generated CT perfusion maps for all patients. The remaining three software packages were unable to generate maps for some patients [RAPID: 1 (1.58%) patient, Olea: 2 (3.17%) patients, and ISP: 7 (11.11%) patients]. Therefore, the analysis included 63 perfusion CT studies processed with Minerva, 62 processed with RAPID, 61 processed with Olea, and 56 processed with ISP. There were some inconsistencies or absence of any disturbances on PCT (5 cases for Minerva (7.9%), 6 (9.5%) for RAPID, 6 (9.5%) for Olea, 2 (3.2%) for ISP).

3.1 Quantitative comparison

The infarct core volumes estimated with RAPID and Minerva were similar [4.50 mL (0–19.50) and 5.96 mL (2.43–18.27), respectively (p = 0.23)]. However, the penumbra volumes and mismatch volumes were significantly larger with Minerva than with RAPID (Table 1). Figure 1 presents the scatterplots of agreement between infarct core, penumbra, and mismatch volumes calculated by RAPID and Minerva. Figure 2 presents the Bland-Altman plots illustrating the absolute and relative differences from the average for the volumes of infarct core, penumbra, and mismatch. Table 2 presents the results of the statistical analysis comparing the two software packages for the infarct core, penumbra, and mismatch volumes. The limits of agreement were smallest for infarct core volumes (–21.22 –25.02), being (–54.79 –32.88) for penumbra volumes and (–60.16 –34.45) for mismatch volumes. The correlation coefficient was highest for infarct volume core (r = 0.94; 95%CI (0.90 –0.96); p < 0.001).

Table 1
Descriptive statistical analysis of the area of infarct core, penumbra, and mismatch volumes estimated by the two CT perfusion software packages included in the quantitative analysis

RAPID Minerva p-value

Infarct core, mL

Mean (SD) 19.02±31.1 17.1±23.73 0.230

Median (IQR) 4.50 (0–19.50) 5.96 (2.43–18.27)

Penumbra, mL

Mean (SD) 67.39±41.37 78.35±45.38 < 0.001

Median (IQR) 60 (29.75–101) 74.16 (44.75–109.93)

Mismatch, mL

Mean (SD) 48.41±32.03 61.27±32.73 < 0.001

Median (IQR) 45 (23.75–66) 62.34 (34.51–76.05)

	RAPID	Minerva	p-value
Infarct core, mL
Mean (SD)	19.02±31.1	17.1±23.73	0.230
Median (IQR)	4.50 (0–19.50)	5.96 (2.43–18.27)
Penumbra, mL
Mean (SD)	67.39±41.37	78.35±45.38	< 0.001
Median (IQR)	60 (29.75–101)	74.16 (44.75–109.93)
Mismatch, mL
Mean (SD)	48.41±32.03	61.27±32.73	< 0.001
Median (IQR)	45 (23.75–66)	62.34 (34.51–76.05)

Fig. 1

Scatterplots of infarct, penumbra, and mismatch volumes estimated with RAPID versus Minerva. The blue line represents the regression line from Pearson correlation. The red line represents the reference line (x = y).

Fig. 2

Bland-Altman plot including the limits of agreement between the two software packages in the infarct core (A), penumbra (B), and area of perfusion abnormality (C). For Bland-Altman plots, only the significant linear regression lines are shown. For the penumbra area, the slope is 1.62 (P < 0.001); for the area of perfusion abnormality, the slope is 0.43 (p < 0.001).

Table 2

Comparison of infarct core, penumbra, and mismatch volume estimates between RAPID and Minerva

Measurement	Difference between measurements (mean±SD), in mL	Right/left	(Right/left)/(right-left)%	Bland-Altman limits of agreement	Pearson correlation coefficient (95%CI)	Slope of regression line
Infarct core	1.92±11.80	0.74±0.87	44.08±226.05, 0 (0–17.42)	(–21.22, 25.05)	0.94 (0.90–0.96) p < 0.001	0.277 p < 0.001
Penumbra	–10.96±22.36	0.88±0.34	13.69±130.8, –2.12 (–6.54–6.35)	(–54.79,32.88)	0.87 (0.79–0.91) p < 0.001	–0.099 p = 0.172
Mismatch	–12.86±24.14	0.81±0.44	–10.74±62.93, –1.84 (–8.31–1.98)	(–60.16,34.45)	0.72 (0.57–0.83) p < 0.001	–0.025 p = 0.818

3.2 Impact of software packages on the decision to recommend EVT

Table 3 summarizes the analyses of the intra-observer and inter-observer agreement regarding the recommendations for EVT based on the CT perfusion maps generated by the four software packages (Fig. 3). Intra-observer agreement between the first and second evaluations based on CT perfusion maps generated by each software package was nearly perfect, except for one observer’s evaluations of ISP maps, which was substantial (k = 0.767). Inter-observer agreement was nearly perfect or substantial for evaluations of each of the four software packages. Comparisons of the different observers’ evaluations based on the maps generated from the different packages found substantial agreement between Minerva, RAPID, and Olea (k = 0.722 for Minerva vs. RAPID, k = 0.761 for Minerva vs. Olea, and k = 0.782 for RAPID vs. Olea) and moderate agreement between ISP and the other packages (k = 0.566 for ISP vs. Minerva; k = 0.512 for ISP vs RAPID; and k = 0.528 for ISP vs. Olea). Table 4 reports the percentage of cases in which EVT was recommended, non-recommended, and questionable in evaluations based on maps from each software package.

Table 3
Intra-observer and inter-observer agreement about the decision to recommend endovascular thrombectomy from CT perfusion maps generated by different postprocessing software packages

Intra-observer agreement (between first and second evaluations) Inter-observer agreement (for first and second evaluations)

Software for perfusion CT analysis 1 vs. 1 2 vs. 2 3 vs. 3 1 vs. 2 1 vs. 3 2 vs. 3 Consensus among 3 observers

Minerva 0.950 0.880 0.913 0.724 0.724 0.915 NA*

(n = 63) 0.833 0.769 0.795

RAPID 0.918 0.821 0.832 0.827 0.866 0.954 NA*

(n = 62) 0.832 0.922 0.877

IntelliSpace Portal 0.767 0.917 0.926 0.834 0.691 0.706

(n = 56) 0.844 0.853 0.833

Olea 0.920 0.917 0.965 0.831 0.736 0.812 NA*

(n = 61) 0.767 0.783 0.852

Minerva vs. RAPID 0.769 0.568 0.598 0.623 0.623 0.598 0.722

(n = 62) 0.784 0.633 0.630 0.729 0.663 0.641

Minerva vs. Olea 0.666 0.653 0.688 0.566 0.610 0.740 0.761

(n = 61) 0.611 0.680 0.680 0.650 0.553 0.695

Minerva vs. IntelliSpace Portal (n = 56) 0.489 0.319 0.627 0.419 0.548 0.524 0.566

0.565 0.358 0.542 0.458 0.478 0.382

RAPID vs. Olea (n = 60) 0.782 0.687 0.675 0.687 0.722 0.645 0.782

0.840 0.782 0.831 0.789 0.734 0.762

RAPID vs. IntelliSpace Portal (n = 55) 0.506 0.478 0.442 0.412 0.494 0.442 0.512

0.513 0.610 0.515 0.586 0.438 0.499

Olea vs. IntelliSpace Portal (n = 54) 0.483 0.507 0.429 0.507 0.471 0.450 0.528

0.493 0.451 0.409 0.549 0.415 0.472

	Intra-observer agreement (between first and second evaluations)	Inter-observer agreement (for first and second evaluations)
Minerva	0.950	0.880	0.913	0.724	0.724	0.915	NA*
(n = 63)				0.833	0.769	0.795
RAPID	0.918	0.821	0.832	0.827	0.866	0.954	NA*
(n = 62)				0.832	0.922	0.877
IntelliSpace Portal	0.767	0.917	0.926	0.834	0.691	0.706
(n = 56)				0.844	0.853	0.833
Olea	0.920	0.917	0.965	0.831	0.736	0.812	NA*
(n = 61)				0.767	0.783	0.852
Minerva vs. RAPID	0.769	0.568	0.598	0.623	0.623	0.598	0.722
(n = 62)	0.784	0.633	0.630	0.729	0.663	0.641
Minerva vs. Olea	0.666	0.653	0.688	0.566	0.610	0.740	0.761
(n = 61)	0.611	0.680	0.680	0.650	0.553	0.695
Minerva vs. IntelliSpace Portal (n = 56)	0.489	0.319	0.627	0.419	0.548	0.524	0.566
	0.565	0.358	0.542	0.458	0.478	0.382
RAPID vs. Olea (n = 60)	0.782	0.687	0.675	0.687	0.722	0.645	0.782
	0.840	0.782	0.831	0.789	0.734	0.762
RAPID vs. IntelliSpace Portal (n = 55)	0.506	0.478	0.442	0.412	0.494	0.442	0.512
	0.513	0.610	0.515	0.586	0.438	0.499
Olea vs. IntelliSpace Portal (n = 54)	0.483	0.507	0.429	0.507	0.471	0.450	0.528
	0.493	0.451	0.409	0.549	0.415	0.472

(*) NA indicates ‘not available’ because there is not comparison between two software packages.

Fig. 3

Summary maps of a patient with an acute ischemic stroke in the left MCA vascular territory. The area of abnormality of both the ischemic core (red/purple/orange) and penumbra (green/yellow) is visually similar in all mismatch maps provided by the four software packages.

Table 4

Three independent observers’ decisions about whether to recommend endovascular treatment based on CT perfusion generated by different software packages

	Reader	Reading	Recommended	Not recommended	Questionable	Not applicable
Minerva (n = 63)	1	1st	52 (82.54%)	3 (4.76%)	3 (4.76%)	5 (7.94%)
		2nd	51 (80.95%)	3 (4.76%)	4 (6.35%)	5 (7.94%)
	2	1st	49 (77.78%)	2 (3.17%)	7 (11.11%)	5 (7.94%)
		2nd	47 (74.60%)	3 (4.76%)	8 (12.70%)	5 (7.94%)
	3	1st	49 (77.78%)	2 (3.17%)	7 (11.11%)	5 (7.94%)
		2nd	50 (79.37%)	1 (1.29%)	6 (9.52%)	6 (9.52%)
RAPID (n = 62)	1	1st	48 (77.42%)	5 (8.06%)	3 (4.84%)	6 (9.68%)
		2nd	47 (75.81%)	4 (6.45%)	5 (8.06%)	6 (9.68%)
	2	1st	50 (80.65%)	3 (4.84%)	3 (4.84%)	6 (9.68%)
		2nd	49 (79.03%)	3 (4.84%)	4 (6.45%)	6 (9.68%)
	3	1st	50 (80.65%)	3 (4.84%)	3 (4.84%)	6 (9.68%)
		2nd	46 (74.19%)	3 (4.84%)	7 (11.29%)	6 (9.68%)
Olea (n = 61)	1	1st	47 (77.05%)	5 (8.20%)	4 (6.56%)	5 (8.20%)
		2nd	45 (73.77%)	5 (8.20%)	6 (9.84%)	5 (8.20%)
	2	1st	47 (77.05%)	5 (8.20%)	4 (6.56%)	5 (8.20%)
		2nd	46 (75.41%)	4 (6.56%)	6 (9.84%)	5 (8.20%)
	3	1st	43 (70.49%)	4 (6.56%)	9 (14.75%)	5 (8.20%)
		2nd	43 (70.49%)	3 (4.92%)	10 (16.39%)	5 (8.20%)
ISP (n = 56)	1	1st	50 (89.29%)	3 (5.36%)	1 (1.79%)	2 (3.57%)
		2nd	49 (87.50%)	2 (3.57%)	3 (5.36%)	2 (3.57%)
	2	1st	49 (87.50%)	2 (3.57%)	3 (5.36%)	2 (3.57%)
		2nd	50 (89.29%)	2 (3.57%)	2 (3.57%)	2 (3.57%)
	3	1st	48 (85.71%)	1 (1.79%)	5 (8.93%)	2 (3.57%)
		2nd	50 (89.29%)	1 (1.79%)	4 (7.02%)	2 (3.57%)

4 Discussion

In the current study, we compared the volumes of infarct core, penumbra, and mismatch measured by Minerva and RAPID. The two software packages yielded similar quantification of infarct core volumes, but the quantification of penumbra volumes and mismatch volumes differed significantly between the two packages. In previous studies, RAPID has provided the best agreement with final infarct volume [18 –21]. In a recent study, Koopman et al. [22] compared estimates of infarct core volumes obtained with ISP, syngo.via CT Neuro Perfusion (version 2017; Siemens Healthcare, Erlangen, Germany), and RAPID, finding large differences between different software packages. The best agreement between syngo.via and RAPID was found when they applied an additional smoothing filter to the syngo.via standard settings rather than when the settings from the RAPID algorithm was used in syngo.via. Austein et al. [12] directly compared measurements of infarct core volume obtained with three software packages against final infarct volume (FIV), finding significant differences between the packages, including both overestimations and underestimations. FIV was assessed between days 1 and 8, using a dedicated open-source imaging software (OsiriX v.5.8.2 32-bit, Pixmeo SARL, Bernex, Switzerland) to analyze NCCT or diffusion-weighted imaging in MRI. RAPID yielded the best agreement with FIV after EVT. Interestingly, ISP classified the mismatch profile as malignant in more than twice as many patients as the other packages (p < 0.001) defined as ischemic core > 70 mL, or absolute volume

difference < 10 mL, or ratio of hypoperfusion and ischemic core < 1.2. In another study, Xiong et al. [23] compared the diagnostic utility and accuracy of RAPID and Olea for determining infarct core volume, defined as the area with a relative CBF < 30%on RAPID and relative CBF < 40%on Olea. They also evaluated relative CBF < 30%on Olea to match RAPID’s default setting. Although RAPID failed to generate maps for 4.7%of patients, infarct core volume was more closely correlated with diffusion-weighted imaging infarct volume on RAPID (r = 0.64) than on Olea (r = 0.42). For all these reasons, we decided to compare Minerva’s performance against RAPID.

Interestingly, Minerva provided postprocessing maps for all cases; the other software packages failed to detect ischemic brain regions in some patients. Despite the rapidly growing use of perfusion CT in stroke decision making, manufacturers have yet to reach a consensus about the best parameters to define infarct core and penumbra. Previous comparisons between different software packages have found wide variation in quantitative measurements of volumes from perfusion CT [11 , 24]. Famhi et al. [11] assessed the variability of the area of infarct core and penumbra in summary maps generated by ISP and syngo.via in a sample of 26 patients with acute ischemic stroke. They found significant differences between the two software packages in infarct core area (–23.6±25.6 cm²) and penumbra area (15.8±25.3 cm²), and the Bland-Altman analysis revealed significant disagreement, with wide 95%limits of agreement and differences surpassing 100 cm². Our results along the same lines underline the need to standardize and validate the algorithms used to analyze CT perfusion data. Similarly, Kudo et al. [15] found wide variation in brain perfusion maps processed by five software packages using delay-sensitive or delay-insensitive algorithms from five commercial providers (GE Healthcare, Philips Healthcare, Siemens, Toshiba, and Hitachi). Thus, CT perfusion results clearly differ depending on the software package used to process the datasets. The factors that contribute to these differences include CT scan parameters (voltage, current, scan time), contrast delivery, type of postprocessing algorithm, and selected thresholds [25, 26]. Moreover, differences in postprocessing steps such as defining the arterial input function and venous input function, motion correction, and smoothing can also affect the results. Thus, automated software can improve the quality, reliability, and reproducibility of perfusion CT studies compared with manual postprocessing. [27]. Differences in acquisition algorithms can also affect the results. Whereas Minerva, RAPID, and Olea use delay-insensitive algorithms that compensate for the delayed arrival of contrast agents in the brain, ISP uses a delay-sensitive algorithm. Kudo et al. [15] found that delay-sensitive algorithms overestimate CBF and MTT.

Underestimation or overestimation of CT perfusion parameters can have important consequences in clinical practice, affecting clinicians’ decisions about whether to perform EVT. To assess these effects in the current study, three radiologists decided whether to recommend EVT based on the presence of mismatch in perfusion maps generated by Minerva, RAPID, Olea, and ISP after quantifying infarct core, penumbra, and mismatch volumes. The high levels of intra-observer and interobserver agreement about the clinical interpretation of the CT perfusion maps show that, despite differences in the different packages’ estimates of specific parameters, clinical decisions based on their output were highly similar. To our knowledge, this is the first study to examine possible differences in the impact of differences in the output of postprocessing software packages on radiologists’ decisions to recommend EVT. These findings should increase confidence in the reliability of perfusion maps as a clinical tool.

This study has strengths and limitations. Analyzing the data through both quantitative and qualitative approaches enabled us to better determine the clinical impact of differences among the output of the software packages. Perfusion CT studies were done on the same scanner, ensuring the homogeneity of the data processed by the four software packages and strengthening the validity and reproducibility of our results. However, in ordinary clinical practice, various scanners and acquisition protocols are used, so this approach reduces the generalizability of our results. Recent trials about late-window EVT underline the importance of these differences in scanners and acquisition protocols when selecting patients. Moreover, including studies done on other scanners at other centers would have enabled us to include more patients; nevertheless, we were able to achieve a moderate size sample (63 patients). Further studies in larger groups of patients are required to validate these findings across vendors.

In conclusion, this study demonstrates that there are quantitative differences in using CT perfusion software packages to estimate infarct core, penumbra and mismatch underline, which need standardization and validation. However, we also observe that the impact of these differences on radiologists’ decisions about whether to recommend endovascular thrombectomy is relatively small.

Footnotes

Acknowledgments

None.

Disclosure statements

This study has been approved by the CEIM of the University Hospital Dr Josep Trueta de Girona (code: 2020/163).

References

Berkhemer

O.A.

, Fransen

P.S.

, Beumer

, et al., A randomized trial of intraarterial treatment for acute ischemic stroke, N Engl J Med 372 (2015), 11–20.

Campbell

B.C.

, Mitchell

P.J.

, Kleinig

T.J.

, et al., Endovascular therapy for ischemic stroke with perfusion- imaging selection, N Engl J Med 372 (2015), 1009–1018.

Goyal

, Demchuk

A.M.

, Menon

B.K.

, et al., Randomized assessment of rapid endovascular treatment of ischemic stroke, N Engl J Med 372 (2015), 1019–1030.

Saver

J.L.

, Goyal

, Bonafe

, et al., Stent-retriever thrombectomy after intravenous t-PA vs. t-PA alone in stroke, N Engl J Med 372 (2015), 2285–2295.

Williams

M.M.

, Wilson

T.A.

, Leslie-Mazwi

, et al., The burden of neurothrombectomy call: a multicenter prospective study, J Neurointerv Surg 10 (2018), 1143–1148.

Bracard

, Ducrocq

, Mas

J.L.

, et al., Mechanical thrombectomy after intravenous alteplase versus alteplase alone after stroke (Thrace): a randomised controlled trial, Lancet Neurol 15 (2016), 1138–1147.

Jovin

T.G.

, Chamorro

, Cobo

, et al., Thrombectomy within 8 hours after symptom onset in ischemic stroke, N Engl J Med 372 (2015), 2296–2306.

Nogueira

R.G.

, Jadhav

A.P.

, Haussen

D.C.

, et al., Thrombectomy 6 to 24 hours after stroke with a mismatch between deficit and infarct, N Engl J Med 378(1) (2018), 11–21.

Albers

G.W.

, Marks

M.P.

, Kemp

, et al., Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging, N Engl J Med 378 (2018), 708–718.

10.

Warach

S.J.

, Luby

, Albers

G.W.

, et al., Stroke Imaging Research (STIR) and VISTA-Imaging Investigators. Acute Stroke Imaging Research Roadmap III imaging selection and outcomes in acute stroke reperfusion clinical trials: consensus recommendations and further research priorities, Stroke 47 (2016), 1389–1398.

11.

Fahmi

, Marquering

H.A.

, Streekstra

G.J.

, et al., Differences in CT perfusion summary maps for patients with acute ischemic stroke generated by 2 software packages, AJNR Am J Neuroradiol 33 (2012), 2074–2080.

12.

Austein

, Riedel

, Kerby

, et al., Comparison of perfusion CT software to predict the final infarct volume after thrombectomy, Stroke 47 (2016), 2311–2317.

13.

Bivard

, Levi

, Spratt

, et al., Perfusion CT in acute stroke: a comprehensive analysis of infarct and penumbra, Radiology 267 (2013), 543–550.

14.

Mazzei

M.A.

, Squitieri

N.C.

, Sani

, et al., Differences in perfusion CT parameter values with commercial software upgrades: a preliminary report about algorithm consistency and stability, Acta Radiol 54(7) (2013), 805–811.

15.

Kudo

, Sasaki

, Yamada

, et al., Differences in CT perfusion maps generated by different commercial software: quantitative analysis by using identical source data of acute stroke patients, Radiology 254 (2010), 200–209.

16.

Sakai

, Delman

B.N.

, Fifi

J.T.

, et al., Estimation of ischemic core volume using computed tomographic perfusion, Stroke 49 (2018), 2345–2352.

17.

Landis

J.R.

and Koch

G.G.

The measurement of observer agreement for categorical data, Biometrics 33 (1977), 159–174.

18.

Cereda

C.W.

, Christensen

, Campbell

B.C.V.

, et al., A benchmarking tool to evaluate computer tomography perfusion infarct core predictions against a DWI standard, J Cereb Blood Flow Metab 36 (2016), 1780–1789.

19.

Mokin

, Levy

E.I.

, Saver

J.L.

, et al., Predictive value of RAPID assessed perfusion thresholds on final infarct volume in SWIFT PRIME (Solitaire with the intention for thrombectomy as primary endovascular treatment), Stroke 48 (2017), 932–938.

20.

Hoving

J.W.

, Marquering

H.A.

, Majoie

, et al., Volumetric and spatial accuracy of computed tomography perfusion estimated ischemic core volume in patients with acute ischemic stroke, Stroke 49 (2018), 2368–2375.

21.

Albers

G.W.

, Goyal

, Jahan

, et al., Ischemic core and hypoperfusion volumes predict infarct size in SWIFT PRIME, Ann Neurol 79 (2016), 76–89.

22.

Koopman

M.S.

, Berkhemer

O.A.

, Geuskens

R.R.E.G.

, et al., Comparison of three commonly used CT perfusion software packages in patients with acute ischemic stroke, J Neurointerv Surg 11 (2019), 1249–1256.

23.

Xiong

, Huang

C.C.

, Fisher

, et al., Comparison of automated CT perfusion softwares in evaluation of acute ischemic stroke, J Stroke Cerebrovasc Dis 28 (2019), 104392.

24.

Abels

, Villablanca

J.P.

, Tomandl

B.F.

, et al., Acute stroke: a comparison of different CT perfusion algorithms and validation of ischaemic lesions by follow-up imaging, Eur Radiol 22 (2012), 2559–2567.

25.

Wintermark

, Maeder

, Verdun

F.R.

, et al., Using 80 kVp versus 120 kVp in perfusion CT measurement of regional cerebral blood flow, AJNR Am J Neuroradiol 21 (2000), 1881–1884.

26.

Borst

, Marquering

H.A.

, Beenen

L.F.

, et al., Effect of extended CT perfusion acquisition time on ischemic core and penumbra volume estimation in patients with acute ischemic stroke due to a large vessel occlusion, PLoS One 10 (2015), e0119409.

27.

Soares

B.P.

, Dankbaar

J.W.

, Bredno

, et al., Automated versus manual post-processing of perfusion-CT data in patients with acute cerebral ischemia: influence on interobserver variability, Neuroradiology 51 (2009), 445–451.

	Intra-observer agreement (between first and second evaluations)			Inter-observer agreement (for first and second evaluations)
Software for perfusion CT analysis	1 vs. 1	2 vs. 2	3 vs. 3	1 vs. 2	1 vs. 3	2 vs. 3	Consensus among 3 observers
Minerva	0.950	0.880	0.913	0.724	0.724	0.915	NA*
(n = 63)				0.833	0.769	0.795
RAPID	0.918	0.821	0.832	0.827	0.866	0.954	NA*
(n = 62)				0.832	0.922	0.877
IntelliSpace Portal	0.767	0.917	0.926	0.834	0.691	0.706
(n = 56)				0.844	0.853	0.833
Olea	0.920	0.917	0.965	0.831	0.736	0.812	NA*
(n = 61)				0.767	0.783	0.852
Minerva vs. RAPID	0.769	0.568	0.598	0.623	0.623	0.598	0.722
(n = 62)	0.784	0.633	0.630	0.729	0.663	0.641
Minerva vs. Olea	0.666	0.653	0.688	0.566	0.610	0.740	0.761
(n = 61)	0.611	0.680	0.680	0.650	0.553	0.695
Minerva vs. IntelliSpace Portal (n = 56)	0.489	0.319	0.627	0.419	0.548	0.524	0.566
	0.565	0.358	0.542	0.458	0.478	0.382
RAPID vs. Olea (n = 60)	0.782	0.687	0.675	0.687	0.722	0.645	0.782
	0.840	0.782	0.831	0.789	0.734	0.762
RAPID vs. IntelliSpace Portal (n = 55)	0.506	0.478	0.442	0.412	0.494	0.442	0.512
	0.513	0.610	0.515	0.586	0.438	0.499
Olea vs. IntelliSpace Portal (n = 54)	0.483	0.507	0.429	0.507	0.471	0.450	0.528
	0.493	0.451	0.409	0.549	0.415	0.472