Inadequacies and pitfalls of network meta-analysis applied to femoropopliteal endovascular interventions

To the Editors,

Recently, Zhou et al. published a network meta-analysis (NMA) on femoropopliteal interventions. ¹ Some of the major hindrances of any meta-analysis are as follows: heterogeneity of data being presented which relates to dispersion and standardization across the included studies, the quality of the included studies, publication bias, and studies effect size. ² An NMA uses indirect inferences among different intervention modalities that have not being compared head-to-head. A close look at the recent NMA published by Zhou et al. reveals some issues that have to be considered while considering its conclusions.

Often, device-driven research is based on limited effect size, non-inferiority trials aiming at device approval. The main outcomes selected by Zhou et al. refer to target lesion revascularization and binary restenosis that lack standardized criteria to gauge restenosis across the included studies. Heterogeneity and sample size were not objectively defined or described. Small-study effect sizes were assessed by visual inspection, which is grossly not reproducible, and heterogeneity, which is considerably high in the available femoropopliteral treatment literature, was not clearly reported. Sample size in several included studies was smaller than 50 patients in each arm studied, which greatly reduces their statistical power and it cannot be completely corrected by using random-effects analysis.^3–6

The readers are then exposed to a plethora of comparisons and Forrest plots that show apple-to-oranges comparisons due to heterogeneous studies included, such as some that describe first and others that describe third-generation atherectomy devices. In addition, control groups from each included study in a NMA are used as “anchors”, in this case plain balloon angioplasty, for the indirect comparisons among different intervention arms. The concept of transitivity, which states no significant difference based on “anchor” characteristics should exist, is not observed in Zhou et al.’s manuscripts.

An emphatic conclusion about drug-eluting stents and covered stent having substantial advantages in terms of restenosis should not be made either, if there were no loops available to compare binary restenosis among the treatment modalities investigated. The authors also report that mortality were not analyzed in an NMA fashion, but a statement in the conclusion clearly states drug-eluting stents and covered stents did not impact mortality. The authors provide a statement acknowledging that the inclusion of several small sample size studies might have affected the authenticity of the reported results. This is indeed the ultimate and most worrisome piece of information contained in this NMA. To control for some of the methodological issues described, a better approach would be to include only larger effect size, higher quality randomized trials that were performed with more recently released technology instead of providing a comparison based on technology that are no longer used or available in the market.

We believe that bringing to light some of the limitations and concerns related to this current NMA can assist the readers to critically review this type of publication. Further, we believe it will help future researchers to overcome some of the methodological biases and design imperfections that can hinder generalization and validity.