Poorly Recognized and Uncommonly Acknowledged Limitations of Randomized Controlled Trials

Commonly Acknowledged Limitations

For ethical, medicolegal, and other reasons, and to increase sample homogeneity and hence improve signal detection, ² RCTs usually have narrow sample selection criteria. Thus, patients may be selected only if they meet specifications for symptoms and illness severity; they may be excluded if they are suicidal, if they have psychotic symptoms, if they have major medical or neuropsychiatric comorbidity, if they have concurrent alcohol or substance use disorder, if they also have a personality disorder, and so on. Consequently, samples in RCTs tend to be unrepresentative of patients at large. External validity in such RCTs is thereby reduced. RCTs are also commonly performed on convenience and purposive samples and sometimes on enriched samples, further limiting their external validity. ³

The internal validity of RCTs can be compromised by faulty randomization methods, poor blinding, use of assessments with uncertain reliability and validity, selection of inappropriate rating instruments, inadequate rater training, lack of standardization of methods within and across centers, and others. Problems such as inadequate recruitment and underpowering, poor treatment adherence, high dropout, high placebo response, and ceiling and floor effects can scuttle RCTs. Finally, the trustworthiness of RCT literature is compromised by poor or unscrupulous statistical methods, selective publication, selective reporting, investigator and industry bias in these regards, and others. Some of these limitations are avoidable, others are out of the control of the researcher. These limitations are well-recognized, have been discussed elsewhere, and are not repeated here.^4,5

Unobvious Examples of Poor External and Internal Validity

Reasons for lack of validity often hide in plain sight and are seldom discussed because attention is focused on the findings of the study and not on the bias in the sample. Consider the CONSORT diagram, scrutiny of which can often provide an idea of how filtered the sample was. In some RCTs, the filtration is so extensive that external validity is seriously compromised.^6-8 Regrettably, the CONSORT diagram presented by authors often does not provide information about how many patients were obviously ineligible and hence not formally screened. As an example, patients who are out of age range, who are already on treatment, and who have longstanding illnesses may be obviously ineligible and not screened. In such studies, therefore, only obviously eligible subjects are screened, and despite stringent study selection criteria, most of these patients may be found eligible to enter the randomization phase. This gives the reader a false impression that few patients were found ineligible during screening and creates a false impression of external validity. ⁹ Judgment about external validity should then be made after examining the study selection criteria and not based on what the CONSORT diagram states.

Consider RCTs of interventions such as psychotherapy, yoga, meditation, acupuncture, and aerobic exercise. Patients cannot be blinded to these interventions. Patients who volunteer and/or consent for such RCTs would, therefore, experience a placebo response that is contaminated by their expectations and beliefs. As an example, patients who believe that meditation is helpful could select themselves into an RCT on meditation and experience an enhanced placebo response if they are in the meditation group or a diminished placebo response if they are in the control group. Likewise, in psychotherapy RCTs, patients in waitlisted groups would not experience a placebo response. In such RCTs, patient cooperation with study protocols and dropout could also be shaped by preexisting beliefs. Internal validity is thus seriously compromised in such RCTs.^10,11 The subject is important in Eastern countries where interventions such as yoga and tai chi are increasingly being investigated.

In a special example of bias in RCTs of interventions that cannot be blinded, intravenous ketamine was found to be non-inferior to electroconvulsive therapy in patients with depression. ¹² In this RCT, out of 403 patients who had been randomized, 31 patients dropped out in the ECT arm before starting treatment; this number was just 4 in the ketamine arm. The implication is that these dropouts were due to dissatisfaction with the assigned treatment. Thus, randomization was compromised even before the RCT began, and there is no assurance that the patients who remained in the study were free of the biases that the dropouts appeared to display.

As a less common but nonetheless influential example, RCTs have seriously compromised internal validity in maintenance therapy studies when clinically stabilized subjects are randomized to continue on active intervention or to rapidly or even abruptly switch to placebo; this has been done, for example, in studies of quetiapine, lamotrigine, and iloperidone.^6,13,14 The compromise to internal validity is serious because an assumption of RCTs is that, at the start of the study, subjects are similar across groups in all regards except for the allocated intervention. The assumption is violated in patients who abruptly switch to placebo because these patients experience physiological perturbations related to sudden discontinuation of active treatment, whereas patients who continue on active treatment do not experience such perturbations. The efficacy of even lithium was “established” through historical RCTs that effected such sudden switches; as we know today, the sudden discontinuation of lithium is associated with a heightened risk of early relapse.^15,16

Post-randomization Bias

Randomization begins to lose its integrity soon after the commencement of the trial. Most people are aware of the issues, but few realize the extent to which these issues disturb the assumption that, in RCTs, the groups are similar in all regards except for the intervention being studied. In a nutshell, the issues are events and exposures, called post-randomization biases, that disturb the balance between groups; their occurrence is also known as post-randomization confounding. ¹⁷

Examples of post-randomization biases include dropout due to adverse events (more likely in the drug group), dropout due to inefficacy (more likely in the placebo group), use of rescue medication (more likely in the placebo group), and use of unreported out of trial medication (more likely in the placebo group). There is also no reason to expect that drop out due to other reasons, nonadherence to study medication, adherence to study protocols, exposure to external destabilizing events such as stress, and other biases will be neatly balanced between the groups in the study.

Some of the biases, such as those related to missing data, can be partially addressed using methods of imputation. However, to identify and address all post-randomization biases is next to impossible.^17-19 We can recognize and accept that the RCT has been compromised to the extent that the biases were detected and reported, as with the use of rescue medication, study dropout, and reasons for study dropout. We are unable to recognize the compromised internal validity with regard to biases that are not recognized and/or not reported, as with the use of out-of-study medication and exposure to external destabilizing events.

As an example of post-randomization bias, in an RCT of vitamin D supplementation in depressed patients with vitamin D deficiency, some patients in the trial did not take the study medication, and others self-medicated with vitamin D purchased over the counter. This was detected when blood levels were checked at the end of the study. Extensive statistical reworking was required to address the problem. ²⁰ One might imagine similar problems biasing results in other supplementation RCTs, such as RCTs on omega-3 fatty acids. Control group contamination may likewise occur in RCTs of educational interventions, given the ubiquitous availability of information today.

Parting Notes

When an RCT ends, treatment responders in each group may be followed for a further period ²¹ ; such a follow-up study is a nonrandomized observational study and not the continuation phase of an RCT because selectively following responders alters the composition of the groups, thereby breaking the randomization.

The longer the duration of an RCT, the greater the likelihood of contamination by post-randomization biases. The resultant statistical noise may reduce the measured effect size of the intervention and may make a true intervention effect difficult to detect. This was apparent in a 7-year RCT of a cash intervention to reduce memory decline and dementia in an impoverished setting. ²²

When RCTs measure proxies, the proxies (e.g., neuropsychological test findings) should not be misunderstood to represent clinical outcomes (e.g., cognitive symptoms in depression that impair workplace efficiency). Authors may fail to draw the distinction. ²³

As final notes, limitations described in this article notwithstanding, RCTs are still the best among research designs; other research designs have even greater imperfections. Because different RCTs are different, it is important to recognize the imperfections present in each RCT so that the findings of the RCT can be soberly interpreted.