Do Small Pilot Randomized Controlled Trials Provide Evidence? The Answer Is No

To the Editor:

We read with great interest the pilot randomized controlled trial by Olavarria et al. [1] comparing the use of porcine acellular dermal matrix and polypropylene meshes in patients undergoing complex ventral hernia repair. We found some aspects of the study perplexing and would like to comment on the validity of the findings.

A small sample size introduces the risk of not achieving balance in potential confounders. The standardized mean difference (SMD) is a metric for assessment of between-group balance that does not depend on sample size, with values <0.10 considered an acceptable degree of balance [2]. In fact, we calculated SMD for the baseline characteristics and found that more patients in the biologic mesh arm had diabetes mellitus (SMD = 0.24), defects >10 cm (SMD = 0.16), and concomitant gastrointestinal procedures (SMD = 0.22).

The overall loss to follow-up was 19.5% and the authors did not use a method for imputing missing data. Cook and Zea [3] noted that even loss rates of 10% to 15% may be unacceptable in randomized trials. The proportion of loss to follow-up was higher in the biologic mesh group (25%). This differential loss to follow-up introduces attrition bias that may occur if there are between-group imbalances at the time of follow-up that are linked to attrition.

Although the authors used blinded outcome assessors to protect against detection bias, measurement error may still have occurred. The authors have not clearly defined the core end points of the study, namely wound dehiscence, hernia recurrence, and mesh infection. By clear definition, we mean standardized criteria and means of detection (screening or diagnosis). This omission could result in considering borderline adverse events as positive events in one group and non-events in the other group [4].

Intention-to-treat analysis was carried out notwithstanding the fact that missing follow-up data were not controlled for using multiple imputation and/or sensitivity analysis with different assumptions for missing outcomes [3,5]. One of the ways to control for such loss to follow-up in intention-to-treat analysis is the worst-case versus best-case scenario approach. For instance, compared with the synthetic mesh group, the patients in the biologic mesh group had a higher rate of major complications at one-year post-operatively (14/33 vs. 8/37; relative risk = 1.96). We could assume that all lost patients in the biologic mesh group had the best outcome and all synthetic mesh patients had the worst outcome. This would result in similar rates of major complications at one year (14/44 vs. 14/43, respectively) with the relative risk value of 0.98, which may represent one bound of sensitivity analyses.

Finally, lack of standardization in surgical technique including mesh to defect ratio, suture technique, and size of mesh utilized, in our opinion, is a major issue. Needless to say, not all surgeons have the same expertise.

In summary, the conclusions drawn in this study are premature and not justified in view of the uncertainty surrounding the single small randomized trial available so far and a number of above-mentioned biases and limitations.