Prediction of delayed reperfusion in patients with incomplete reperfusion following thrombectomy

Study design

We performed a retrospective observational analysis of a single-center registry for all consecutive acute ischemic stroke (AIS) patients admitted between 02/2015 and 12/2021 who had undergone MT. Written informed consent was obtained from the patients. This study received ethics committee approval (Kantonale Ethikkommission Bern, reference ID 2019-00547) and was performed in accordance with the Declaration of Helsinki and its later amendments. Reporting has been performed according to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement. Study data are available from the corresponding author upon reasonable request following receipt of a research plan and clearance by the ethics committee.

Perfusion imaging

All patients underwent MRI or CT follow-up imaging 24 ± 12 h after the intervention as part of standard care in our institution. All perfusion imaging was performed as part of clinical routine. Olea sphere software (Olea Sphere v2.3; Olea Medical, La Ciotat, France) was used to generate perfusion images from MRI and syngo.via (Siemens) from CT scans. Perfusion imaging outcome was dichotomized into DR and PPD. DR was defined as resolution of a focal angiographic perfusion deficit from the final thrombectomy angiography with normalization of tissue perfusion at the corresponding neuroanatomical territory on the 24 ±12 h follow-up perfusion imaging maps (Figure 1(a)). While PPD was observed in case of a perfusion imaging abnormality, which corresponded to the antegrade capillary phase deficit from the final thrombectomy angiography imaging (Figure 1(b)). Patients who presented with reocclusions on the follow-up imaging (n = 17) were classified as having a PPD because it cannot be stated with certainty whether these patients will develop a reocclusion after the procedure or not. Details on grading and evaluation of perfusion imaging outcome have been described previously. ⁸ Only large-vessel occlusion anterior circulation AIS patients who had final eTICI grade 2a–2c and available 24 ± 12-h follow-up perfusion imaging were included in the final cohort (Supplemental Figure I).

OPEN IN VIEWER

Figure 1.

Delayed reperfusion and persistent perfusion deficit.

Model development and specifications

We have developed a prediction model “PROCEED” (PeRfusion OutComE prEDiction), which can be used to estimate the likelihood of DR occurring after incomplete thrombectomy has been established on the final angiography run. The aim of PROCEED is to assist in identification and selection of patients with a high likelihood of DR, and who therefore may be less likely to benefit from any adjunctive reperfusion efforts when incomplete angiographic reperfusion is established on the final angiography run (Supplemental Figure II).

The model was corrected for under- and overfitting, avoiding systematic under- or overestimation with steps described below in this paragraph. Missing values were replaced with median and mode values for continuous and categorical variables, respectively. Considering low number of missing values (<5%) no multiple imputations were used. Patients’ baseline and interventional characteristics that could be obtained during routine stroke admission and work-up were included in the initial analyses (Supplemental Table I). In order to reduce the number of variables and find the ones which will aid in model’s ability to discriminate, we have performed variable pre-selection via bootstrapped (n = 1000) backward stepwise logistic regression based on the Akaike information criteria (AIC) score. AIC-based selection was chosen because it measures the quality of the entire model, unlike the p-value, which only provides information on specific variables within the model. Following this statistical pre-selection we have executed final variable selection with interval validation via bootstrapping method and have used the output of interval validation to choose the most optimal classification algorithm. Lastly, random forest algorithm was selected as the one with the highest accuracy for final model development. Bootstrapped resampling of the final model was performed to obtain the 95% confidence interval (CI) for all model performance metrics (Supplemental Figure I). The final version of PROCEED has been uploaded in a digital repository as an interactive open access tool: https://proceed.shinyapps.io/model/. Time points shown in an online tool have been calculated with the Cox regression that analyzed time from the end of the intervention to follow-up perfusion imaging within the model.

Model variables

Variable pre-selection was based on evidence in current literature and AIC score of the logistic regression output. Variables included in the final model are: age, sex, atrial fibrillation, anticoagulants and antiplatelets prestroke, National Institutes of Health Stroke Scale (NIHSS) score on admission, Onset-to-Door time, Intervention-To-Follow-Up time, intravenous thrombolysis, number of maneuver counts, eTICI, and collateral score. Stroke severity was evaluated with the NIHSS score upon admission. Sites of arterial occlusion were evaluated on initial imaging: intracranial carotid artery (ICA), proximal segment of the middle cerebral artery (M1), Sylvian segment of the middle cerebral artery (M2), cortical segment of the middle cerebral artery (M3), pre-communicating and post-communicating segment of the anterior cerebral artery (A1–2). The eTICI scale was used to grade reperfusion success on the final angiography series as follows: 1%–49% reperfusion of the affected territory was graded as eTICI 2a, 50%–66% as eTICI 2b50; 67%–89% as eTICI 2b67; and 90%–99% as eTICI 2c. The American Society of Intervention and Therapeutic Neuroradiology and Society of Interventional Radiology (ASITN/SIR) Collateral Flow Grading System was used for collateral grading on pretreatment angiography. Grades range from 0 (no visible collateral) to 4 (complete and rapid collateral blood flow in the entire ischemic territory), as described previously. ⁹ Onset-to-Door time refers to the time from symptom onset until the admission of the patient to the emergency department of the treating hospital. Intervention-to-Follow-Up time is the captured time between the last angiography series run and the time of the first follow-up imaging.

Statistical analysis

Fischer’s exact and Chi-squared tests were used for categorical variables, and Mann-Whitney-U and Welsch’s T-test were used for continuous variables. Results are reported as “median [interquartile range (IQR)]” or “n (%).” Statistical handling of variables is reported in Supplemental Table I. The model’s discrimination is reported with C-statistics (Harrell’s concordance), which varies from 0.5 to 1, where 0.5–0.7 indicates good, 0.7–0.8 strong, and 1 perfect discrimination. It estimates the likelihood that a randomly selected patient, who has a higher predicted probability of achieving DR, will actually have DR. For binary outcomes, C-statistics is similar to the area under the receiver operating characteristics curve (AUC). Discrimination and calibration were quantified together with a Brier score, which evaluates the goodness-of-fit for a predicted probability. Brier scores ranges from 0 for total accuracy to 1 for complete inaccuracy. This is presented graphically with a calibration plot for achieving DR, together with calibration intercept and slope. The intercept serves as a measurement of predicted probabilities indicating wherever they are too low or too high, while the slope suggests the predictor’s strength in the cohort. ¹⁰ Ideally, the intercept should be equal to 0, and slope to 1. ¹⁰ Importance of variables included in the model is presented with the Mean Decrease Accuracy (MDA) index, and MDA index points were plotted in descending order of importance. MDA expresses how much accuracy the model losses by excluding a certain variable. Variables with higher MDA index are more important for successful performance of the model. Summary of model’s performance is presented with a confusion matrix in a contingency table. Most often used metrics for presenting the results of the confusion matrix are precision and an F1 score. Precision of the matrix accounts for all the positive cases and provides a rate on how many of them were predicted correctly, while the F1 score harmonizes precision and sensitivity, making them comparable. All reported model metrics are presented with a mean and 95% CI from bootstrapped model replication. All statistical analyses were performed in R v4.0.0 with the packages outlined in the Supplemental Table II.

Clinical decision curve

Decision curve analyses are used to add information on the clinical utility of a certain prediction model. ¹¹ Two important metrics in clinical decision curves are net benefit and threshold probability. ¹¹ Net benefit is a weighted difference of true and false positives: it increases with true positive and decreases with false positive cases. Threshold probabilities decide how important doing an additional attempt or maneuver is in a patient that would have DR (false positives) compared to not doing an additional attempt or maneuver in patients that would develop DR in any case (true positives). Higher threshold probabilities give more weight to false rather than true positives. Conversely, higher net benefits give more weights to true rather than false positives. Depending on the weights, context and scenario in which the model is being used, model might favor high or low threshold probabilities. ¹¹ In the present framework, model should have low threshold probabilities and with high net benefit. We predefined a threshold probability range between 20% and 40%, which corresponds to the odds range of 1:4–2:3. This means that pursuing additional attempts in a patient who would be likely to have DR would be 4–1.5 times worse than not pursuing additional attempts. The lower and upper ends of the threshold probability range are based on typical patients who would unquestionably be considered for adjuvant attempts after incomplete reperfusion, considering potential risks and benefits. When the golden reference standard does not exist, threshold probability range is based on the “Treat All” scenario. In our case, lower end of the threshold probability range was set at 20%, as the net benefit between “Treat All” option and the prediction model is almost identical below this point. Upper end was set at 40%, as the “Treat All” option shows zero benefit after this cutoff, meaning no further consideration would be reasonable. Decision curves also provide information on net reduction in additional attempts or maneuvers, where differences between true and false negatives are weighted: net reduction becomes higher the more true negatives there are. ¹¹ Net reduction is usually reported as a rate per 100 patients. Reporting net reduction is especially valuable if the current gold standard for a certain condition is to treat all patients. ¹¹

Potential model implications

Patients with incomplete reperfusion (e.g. eTICI 2b50) are expected to have small remaining perfusion deficits caused by the occlusion of medium and small distal vessels. The decision on proceeding or stopping with an intervention in these patients is surrounded by uncertainties and is dependent on many factors.^3–5 Potential adjunctive reperfusion efforts include secondary distal MT, administration of intra-arterial lytic, or additional antithrombotics.^12–17

Data on distal stent retrievers or aspiration thrombectomy are heterogeneous with limited generalizability due to selection bias.^3–5,18 Some observational studies report better reperfusion rates after performing additional maneuvers.^3,4 However, complications associated with mechanical maneuvers seem to increase the more distal the occlusion site is.^12,13,18 Other adjunctive reperfusion efforts would include the administration of intra-arterial lytics or antithrombotics.^14–17 Although intra-arterial lytics are thought to be less invasive than mechanical therapies, they may increase the risk of bleeding.^19,20

In summary, several propositions have been made for different adjunctive reperfusion strategies, but none of them are devoid of risk.^12–17 Taking these risks may be unwarranted in patients who are likely to develop DR, which is associated with lack of infract growth and a comparable outcome to patients with primary eTICI3 reperfusion. ⁸ Therefore, a better prediction of what patients show a naturally benign disease progression may be of value. While the present model offers fair prediction of perfusion imaging outcomes, it is not clear if there may still be a benefit of immediate complete reperfusion even in patients which show DR.

Predictor values

PROCEED is based on seven baseline and interventional characteristics: age, sex, atrial fibrillation, Intervention-to-Follow-Up time, maneuver count, eTICI, and collateral status.

Association between age, sex, and perfusion outcome has already been established. Younger patients and males tend to have significantly better perfusion outcome with lower rates of incomplete reperfusion. ²¹ Our findings are also supported by previous descriptions of reperfusion status serving as a function of time, showing gradual increase in complete reperfusion rates as time passes.^8,22 Atrial fibrillation (AF) has been previously described as associated with lower recanalization rates in patients receiving intravenous thrombolysis.^23,24 However, these analyses were conducted before the widespread use of MT. A more recent multicenter registry analysis investigated the effects of MT in AF patients and found comparable recanalization rates between AF and non-AF patients. ²⁵ Another recent single-center analysis has reported substantially higher reperfusion rates achieved with MT when comparing AF to non-AF patients. ²⁶ A subtle unexpected finding from the present analysis is that AF seems to favor DR. As there is a lack of literature on the relationship between AF and DR, we presently do not have a tangible pathophysiological explanation for this finding. AF could serve as a proxy for another variable of interest, as we have only focused on baseline and interventional variables which are easily obtainable in the setting of acute patient management (see Methods).

Number of ⩽2 maneuver counts has shown a strong association for achieving successful reperfusion.^9,27 It is known that with an increase of maneuver counts the likelihood of complete reperfusion strongly reduces: for example, if reperfusion is not achieved within five maneuvers, the likelihood of successful reperfusion decreases by half. ²⁷ Potential explanation for this would be that thrombi imperviable to mechanical manipulation are also more likely to be resistant to the effects of lytics or to succumb to the effects of autolysis on follow-up, leaving a PPD in the distal tissue. eTICI score appeared to be the strongest predictor of delayed reperfusion with also the highest MDA index. This has also promoted international societies to recommend achieving near-complete or complete reperfusion whenever possible in acute stroke patient care.^1,2 As eTICI grade increases so does the percentage of reperfused territory where for example eTICI 2c patients will have substantially smaller distal deficits when compared to eTICI 2b67 patients. ²⁸ Far distal perfusion deficit entails possible presence of a smaller thrombi which is more likely to spontaneously dissolve and enable complete DR. ²⁹ Pursuit in achieving the highest possible eTICI score is offset by an increased risk of interventional complications from adjuvant reperfusion efforts (e.g. vessel puncture or perforation). True equipoise of this cost-benefit ratio is presently unknown and should be regarded when deciding to pursue additional reperfusion attempts. Good collateral circulation has been continually associated with successful reperfusion. ³⁰ Developed collaterals may allow continued perfusion to the area distal of the occlusion, so even if complete reperfusion is not achieved at the end of MT, patient might still experience complete DR due to increased collateral flow and vascular remodeling. ³⁰

Despite having similar baseline and interventional characteristics (as shown in our example in the Results: 50%–66% vs 67%–89% reperfusion of target downstream territory), predictions of perfusion imaging outcome can widely differ. When considering individual patient outcome independent value of these characteristics is reduced, because in reality outcome is not influenced by just one single factor, but rather by a combination of them. ³¹ The combination of these interventional metrics together with baseline values increases the accuracy by which heterogeneous perfusion imaging outcomes, that are clinically relevant, can be identified.

Model validation

ESO, ESMINT, and AHA guidelines all recommend an individualized decision-making approach for patients with incomplete reperfusion after thrombectomy.^1,2 For these purposes, validated prediction models tend to be preferable to physicians’ estimates, which are inherently limited compared to information contained in large datasets used for building prediction models. ³¹ However, a recently conducted systematic review highlighted several methodological concerns with many published clinical prediction models. ³² Several studies had not internally validated their models nor corrected for over- and under-fitting, whereas other models excluded patients with missing values, making the use and interpretation of these models worrisome. ³² We did not exclude any patients with missing values and implemented thorough methodological safeguards during validation and overall fitting of the model (see Methods). Great care was taken when initially screening for potential confounding variables, as all variables ought to be easily obtainable in all comprehensive stroke centers, so that all required information are available when making the “proceed-or-stop” decision in the angiography suite. Despite having good discrimination and calibration, the present model tended to overestimate true rates of PPD. Predictive value of the model should generally be as high as possible, ³² to ensure that necessary further treatment options are not deferred for patients who would not have DR and consequently would not achieve a good functional outcome without adjunctive reperfusion efforts. Further updates on the current model are planned. External validation will be performed on patients enrolled in the Endovascular Therapy for Ischemic Stroke with Perfusion-Imaging Selection (EXTEND-IA), Tenecteplase Versus Alteplase Before Endovascular Therapy for Ischemic Stroke (EXTEND-IA TNK) part 1 and part 2 trials, who have available follow-up perfusion imaging data obtained 24 ± 12 h after the intervention. (clinicaltrials.gov, unique identifier: NCT01492725, NCT02388061, and NCT03340493, respectively).

Limitations

The present model has several limitations. Its single-center retrospective study design might limit the generalizability of reported findings. More than half of patients did not receive intravenous alteplase before MT, which is a lower frequency than reported in national registries.^33,34 However, the impact of intravenous thrombolysis on the occurrence of DR remains unclear. ³⁵ Patients without perfusion follow-up imaging were excluded from the present analysis. This selection bias might overestimate absolute rates of DR, as these might differ in the patients not undergoing perfusion imaging on follow-up. Performance of the model was compared to reference scenarios “Treat All” and “Treat None”; however, in the real world the reference scenario is the physician making the individual decision. Future options should explore this as a reference for benefit. Validations using external datasets are required to fully explore the model’s performance.