How should one analyse and interpret clinical trials in which patients don't take the treatments assigned to them?

Two main ways to analyse clinical trial data

There are two main ways to analyse clinical trial data, and these have been recognized since the beginning of the modern era of clinical trials. In 1941, in his report of a controlled trial of a whooping cough vaccine, Joseph Bell wrote as follows:

‘Obviously it is not practically possible to preselect two large strictly random groups of children who are representative of the general population and to insure that every child in one group receives the vaccine while every child in the other group receives no vaccine during the observation period. Children in the general population have the prerogative to refuse vaccine offered and the liberty to obtain other vaccine when desired. In these premises there is no known way of changing the two groups so that one would include only children actually vaccinated, and the other include only children not vaccinated, without destroying the randomness of the selection and to that extent possibly invalidating the answer to the question asked. After it has been established that the vaccine confers protection, then questions concerning the amount and duration of such protection might in part demand direct comparison of the experience of the children actually vaccinated with those not vaccinated, providing adequate data are at hand to equalize the two groups with respect to attributes which apparently influence the occurrence of the disease.

For this report, the approach to the primary problem involved the preselection of two large strictly random groups of children and the subsequent injection of a large proportion of only one group with the vaccine. All analyses herein presented are a comparison of the experience of such preselected groups regardless of their actual status with respect to receiving the vaccine. The difficulties encountered in this approach are chronologically described in detail so that the reader may evaluate any possible errors involved.’ 1

The primary analysis used by Joseph Bell thus required that all randomized participants be included in the analysis according to their original study group allocation. This is called the intention-to-treat approach. It retains the principal benefit of randomization, namely that comparison groups remain comparable with respect to known and unknown risk factors.

Joseph Bell also refers to the alternative approach to analysis, however. This involves direct comparison of the experience of the children actually vaccinated with those not vaccinated. This is sometimes referred to as ‘analysis by treatment administered’. Another term is ‘per-protocol’, which is a misnomer and should be avoided. The argument against the intention-to-treat approach is that certain randomized participants should not be included in the analysis, for example, those later found to be ineligible, those not taking their study medication as stated in the protocol, and those with poor or missing data. Reasonable people might well observe that including such people in the analysis will tend to underestimate any favourable (and unfavourable) treatment effects – participants who don't receive the allocated intervention as prescribed cannot benefit or suffer from it. Additionally, patients may want to know what will happen if they are good adherers. The drawback of this approach to the analysis is that it erodes the advantage of randomization. Bell recognized this when he referred to the need for adequate data ‘to equalize the two groups with respect to attributes which apparently influence the occurrence of the disease’. Unfortunately such ‘statistical equalization’ can only be pursued using measured factors of potential prognostic importance. There is no way of taking account – as random allocation does – of unmeasured factors.

The critical question is – to whom should the adherent participants in the intervention group(s) be compared? It is obviously important that the study subgroups are comparable in terms of prognosis or risk – that like will be compared with like. Unfortunately, one cannot assume that adherent participants in one group are always comparable to adherent participants in another group. After all, many participants receiving active treatment withdraw due to adverse effects, while placebo participants may withdraw due to perceived lack of benefit. In addition, the proportion of non-adherers in the study groups may be very different. Will withdrawing potentially different subsets of participants from the study groups invalidate the comparability achieved by random allocation?

A landmark study

The Coronary Drug Project evaluated several lipid-modifying drugs, including clofibrate, in survivors of a myocardial infarction. 2 The overall five-year mortality in the clofibrate group was 18.2% compared to 19.4% in the placebo group, a statistically non-significant difference. Among patients allocated to clofibrate who took 80% of the study medication, the mortality was 15.0%. This was strikingly lower than the mortality among those patients with lower adherence, which was 24.6%. One possible explanation for this difference is that clofibrate did, after all, have an impressive beneficial mortality effect when post-infarction patients actually took the drug: the proponents of the per-protocol approach to analysis appeared to have a case. However, the level of medication adherence was similar among those allocated placebo as it was in the clofibrate group, so an analysis was done to compare mortality among adherers and non-adherers to placebo. This revealed a difference in five-year mortality mirroring that for adherers and non-adherers to clofibrate: placebo adherers had a mortality of 15.1%, compared with 28.2% among poor adherers. Importantly, attempts at ‘equalizing’ these non-randomized subgroup comparisons using 40 baseline characteristics associated with five-year mortality had only a small effect on the observed differences.

What inferences can be drawn from these analyses? First, they demonstrated that participants with high and low adherence in both comparison groups were different in terms of five-year mortality risk. Good adherers are very different from low adherers. Second, the similarity in mortality and adherence rates across the study groups suggests that, in this population, any effects of clofibrate were, on average, similar to placebo.

The next questions are – Have these remarkable findings been confirmed in other studies? Replications are important in research. What are the causes of this adherence effect?

A confirmatory study

A similar pattern to that observed in the Coronary Drug Project was observed in the Aspirin Myocardial Infarction Study. 3 Overall, no statistically significant difference in mortality was observed between patients assigned to aspirin and those assigned to placebo (10.9% vs. 9.7%). The level of medication adherence was high in both groups. However, participants with good adherence had mortality rates of 6.1% and 5.1%, respectively, and those with low adherence rates of 21.9% and 22.0%, respectively. Thus, in a similar population, there were marked mortality differences between participants with high and low adherence rates and this difference was observed in both the aspirin and placebo groups. These analyses of the Aspirin Myocardial Infarction Study thus confirmed the analyses of the Coronary Drug Project. The next question is – Are the analyses of these two studies reflected in other patient populations receiving interventions with documented mortality effects?

Other observations

Published findings from other trials in treatments of myocardial infarction, congestive heart failure, arrhythmias, cancer and schizophrenia have also confirmed the findings from the Coronary Drug Project, 4 and a recent analysis of observational data found that better adherence to statin treatment was associated with lower mortality. 5 The association between good adherence and favourable health outcomes was confirmed in a recent meta-analysis (which unaccountably failed to include the findings from the Coronary Drug Project and the Aspirin Myocardial Infarction Study). 6 The overall odds ratios were 0.56 (95% CI 0.43–0.74) for adherence to placebo and 0.55 (95% CI 0.49–0.62) for adherence to drug therapy. The authors speculate that good medication adherence may be ‘a surrogate marker for overall healthy behaviour’. Interestingly, the meta-analysis also reported that good adherence to harmful drugs was associated with increased mortality (odds ratio 2.90, 95% CI 1.04–8.11).

Lack of any association between adherence and outcomes has also been observed or reported. The findings for the clofibrate group in the Coronary Drug Project was not confirmed in its high-dose oestrogen group (unpublished data). A possible explanation for this difference might be that oestrogen had serious adverse effects and that there were twice as many non-adherers in the oestrogen group as in the placebo group. Two published analyses that failed to find any relationship between medication adherence and health outcomes differed from those mentioned above in two regards. 4 First, the treatment outcome was not all-cause mortality; second, they were analysed using adherence as a continuous variable. One of them was the Cardiac Arrhythmia Suppression Trial, which was stopped early due to harm. 7 The other was the Lipid Research Clinics – Coronary Primary Prevention Trial. The primary outcome included fatal and nonfatal myocardial infarction. 8

The picture is clearly complex and various explanations have been offered for the observations outlined in this commentary. The ‘clofibrate findings’ could reflect a ‘healthy adherer’ effect, with good adherence being a marker of better health or a healthier lifestyle. Sicker patients may not tolerate a study medication and stop taking it. This could explain a difference in the active-treated group but not among placebo participants. Another explanation could be that patients develop medical conditions or serious complications that could lead to low adherence as well as worse prognosis.

What lessons have we learned so far?

Good and poor medication adherers seem to have different prognoses.

Footnotes