‘PACE-Gate’: When clinical trial evidence meets open data access

Introduction

The recent release of data from the largest clinical trial of psychotherapy treatments for chronic fatigue syndrome (CFS), the ‘PACE-Trial’, has triggered a perfect storm of patient anger and professional defensiveness. The data were only released after a protracted freedom of information case brought by a patient with CFS. A tribunal ordered the lead author’s institution to release their data. Upon release, re-analysis showed that the levels of improvement and recovery observed in the released data were much lower than the levels reported in the published report (White et al., 2011a) and other related publications. The released data showed that the effectiveness of cognitive behavioural therapy (CBT) and graded exercise therapy (GET), in comparison to standard medical care (SMC) and adaptive pacing therapy (APT), fell by almost two-thirds.

Patient groups and independent experts have remarked that without data access, the medical establishment would have been left to accept the outcomes from the PACE-Trial, as robust evidence that CBT and GET are effective treatments for CFS. Instead, patients are calling for the wider scientific community to investigate their claim that the PACE-Trial authors overstated the benefits of CBT and GET. This editorial considers the ramifications of this unfolding story for patients with CFS, and its impact on the science of clinical trials of psycho-behavioural therapies.

Background

In 2011, a group of UK researchers published results from the PACE-Trial, a large randomised controlled trial of treatments for CFS, with 641 participants (White et al., 2011a). This is the largest clinical trial ever conducted on therapies for CFS, with a combined cost of almost £5 million; funded by the Medical Research Council, the Department for Work and Pensions, the Department of Health for England and the Scottish Chief Scientist Office. The trial report carried the bold claim that 59 per cent of CFS patients receiving CBT and 61 per cent receiving GET had improved (White et al., 2011a). An accompanying editorial in The Lancet suggested a recovery rate of 30 per cent using CBT and GET in the PACE-Trial (Knoop and Bleijenberg, 2011). This figure was later lowered to a 22 per cent recovery rate in a follow-up paper by the PACE-Trial team (White et al., 2013). If credible, a 61 per cent improvement rate and a 22 per cent recovery rate would represent a significant breakthrough in the treatment of CFS; a serious condition affecting millions of patients worldwide.

Doctors and scientists have long struggled to understand the causes of CFS and have had little in the way of treatments to offer sufferers. Following the PACE-Trial, CBT and GET gained prominence as positive interventions. The National Health Service (NHS) has promoted CBT and GET as effective and safe treatments for CFS on their ‘NHS Choices’ website. The PACE-Trial also solidified the status of CBT and GET as ‘evidence-based treatments’ recommended by the National Institute for Health and Care Excellence (NICE, 2014).

While the health authorities accepted the findings from the PACE-Trial as bona fide and media outlets reported that CBT could cure CFS, patient groups raised concerns about the trial early on, particularly its methodology and findings. Individual patients and advocacy groups pointed to the fact that the trial investigators had altered assessment thresholds (primary outcome measures) after the start of the trial; had applied a broad inclusion criteria (the Oxford Criteria), rather than more strict sampling criteria; and had contaminated the trial by promoting the success of CBT and GET in newsletters to trial participants during the trial (Goldin, 2016; Newsletter available at Queen Mary University London (QMUL, 2016b: FAQ link). The President of the Royal College of Psychiatry, Professor Sir Simon Wessely, who had helped to recruit patients into the trial wrote, ‘I think that we can have confidence in the principal findings of PACE’ but referred to the controversy around the trial being like ‘a ship voyage across a stormy sea’ (Wessely, 2015).

A long and bitter battle ensued in the years following the end of the trial that pitted patients against the trial investigators (Chalder and Goldsmith, 2015). Patients and patient advocacy groups called for access to the trial data. They were not the only critics of the trial. The Lancet received two open letters from a list of distinguished academics, calling for the PACE-Trial to be independently analysed (Racaniello, 2016).

Patient scrutiny and data analysis

While patients and academics called for the release of the PACE-Trial dataset, the authors refused on the grounds that the data included sensitive patient information (QMUL, 2016b). In response, patients proceeded to submit freedom of information requests to the lead author, Professor Peter White at QMUL. However, QMUL turned down many of these requests. One patient, Mr Alem Matthees, took his case to appeal. QMUL lost, but appealed to seek to overturn this decision. The PACE-Trial lead authors submitted evidence to a subsequent information tribunal that they had already shared their trial data with the independent reviewers at the Cochrane Group (QMUL, 2016b), but also divulged that some of the PACE team were in fact the Cochrane reviewers (HMTS et al., 2016). In August 2016, the tribunal dismissed the QMUL appeal and ordered the release of anonymised trial data. Immediately prior to the 1-month deadline, the PACE team published an analysis of the trial’s original protocol and outcome measures, and QMUL released the requested data (QMUL, 2016a, 2016b).

Within days of the release, patients examined the authors’ new data publication and the raw data found its way on to patient forums. It quickly became apparent that the improvements reported by White et al. (2011a) were much reduced when the original protocol thresholds (White et al., 2007) were applied. Using the trial’s original markers for improvement, the effectiveness of CBT and GET fell from the reported 59 and 61 per cent, to just 20 and 21 per cent, respectively (Figure 1). Patients took to social media with statements that the PACE team had overstated claims of efficacy in their data analysis. Indeed, the Standard Medical Care group, which was the de facto control condition, had produced a 10 per cent improvement, meaning that CBT resulted in just a modest 10 per cent added benefit over usual medical care. In light of this revelation, there is little alternative but to conclude that the PACE team utilised methods that showed CBT and GET to be vastly more beneficial than would have been the case, if the authors used their original trial protocol. The impact of this troubling conclusion is made far worse by the fact that the PACE-Trial authors did not disclose these findings until after the information tribunal. The authors have steadfastly maintained that their changes to the thresholds were justified and approved by their oversight committees (Walwyn et al., 2013). However, figure 1 clearly shows that the size of the added benefit of CBT and GET falls to just 10–11 per cent applying the original trial protocol.

OPEN IN VIEWER

Figure 1.

PACE trial improvers: published (blue) versus original protocol (red).

Over the past 5 years, the PACE team have published multiple follow-on papers from the trial, such as a long-term patient follow-up and an economic evaluation of the cost effectiveness of CBT and GET compared with APT and SMC (McCrone et al., 2012; Sharpe et al., 2015). Patients have scrutinised these outputs and highlighted some clear inconsistencies: specifically that the initial gains reported at trial end (52 weeks) mostly disappeared between groups at follow-up (2.5 years), with SMC and APT having improved to a similar degree to those in the CBT and GET groups (Sharpe et al., 2015). The trial authors have since argued that the SMC or APT patients probably went to get CBT or GET privately after the end of the formal trial (QMUL, 2016b). Patients also pointed to the fact that within the CBT and GET groups, the reported levels of improvement and recovery were not matched by clear reductions in secondary outcome measures; with levels of unemployment, health care usage and sickness benefit claims, remaining relatively unchanged in all treatment groups (McCrone et al., 2012; White et al., 2013). The PACE authors have stated this was most likely the result of a harsh economic climate and patients being given welfare advice during the trial (QMUL, 2016b: FAQ).

Methodological concerns

Patients have pointed to several methodological concerns about the PACE-Trial. First, the trial authors had favoured subjective self-report measurement instruments over objective tests of physical function. For example, the PACE team used the Chalder Fatigue Scale and a quality-of-life survey (SF-36) to assess improvement, yet dropped plans to assess patients’ activity levels using electronic activity monitors, reporting that such tests were too complex to undertake (QMUL, 2016b; Walwyn et al., 2013).

Critics have also pointed out a crucial methodological anomaly, that the PACE team had lowered the threshold for improvement and recovery from a score of 85 on SF-36, to a score of 60, at the analysis stage. This change meant that some trial participants had reached the level required to be classified as improved or recovered at trial entry, before they had even taken any treatment course (Walwyn et al., 2013; White et al., 2013). The trial authors have not offered a reasonable explanation for this observation. The other parameters rested on patients reporting feeling better using self-report measures and no longer meeting the Oxford Criteria (White et al., 2007).

Finally, it was noted that the PACE-Trial team derived a rather convoluted ‘operational definition’ of recovery using a mix of no longer meeting the criteria for CFS and a patient reporting feeling ‘better’ or ‘much better’, rather than objective markers of physical improvement. Much of the reported benefits in the PACE-Trial rested on patients’ perceptions of mood and fatigue, rather than measurements of their physical improvement benchmarked against norms within a healthy population group. Despite the size and complexity of the PACE-Trial, the investigators did not explore whether participants were able to return to normal physical activities, such as walking outside, standing upright, doing shopping and socialising. Given CFS is a physically disabling condition, any assessment of recovery should have assessed these factors, particularly as ‘self-reported improvements’ in psychotherapy trials may be influenced by placebo and therapist effects (Geraghty and Blease, 2016).

The impact of the belated and enforced data release

The release of the PACE-Trial dataset was hard won. It took patients 5 years to win the right to obtain the data. Over this period there have been many discussions in academic circles about the need for open data access. As a backdrop, the most-read paper in PLoS Medicine claims that ‘most research evidence in medicine is false’ (Ioannidis, 2005). A major replication project in psychology could only reproduce 39 per cent of published results, suggesting as much as 61 per cent of studies are unreliable (Open Science Collaboration, 2015). These observations support the need for greater transparency in clinical trials. Ben Goldacre at the Evidence-Based Medicine DataLab in Oxford has dedicated much of his time to developing tools to register trials and promote open access (e.g. the Open Trials Initiative, Goldacre, 2010, 2014). However, not all scientists are in favour of sharing data. For example, Lewandowsky and Bishop (2016) wrote in Nature Online about how data access requests might be used as weapons of harassment by militants, specifically referring to CFS. Interestingly, the freedom of information tribunal heard from the PACE authors that they refused to release data partly on the grounds that they viewed requesters as vexatious patients who were using freedom of Information (FOI) for illegitimate reasons (HMTS et al., 2016).

The PACE-Trial stands out as a showcase example of why data transparency is needed in contemporary science. Patients suffering from health conditions like CFS, and independent scientists, should have the right to see the evidence behind the claims of any scientific study, especially if this evidence is used to direct health policy or promote certain treatments – as was the case for the PACE-Trial.

The status of CBT and psychotherapy in CFS

There are other reasons why patients and independent critics challenged the PACE-Trial authors. Many CFS patients and advocacy groups reject the model that asserts that CFS is ‘perpetuated’ or ‘maintained’ by ‘dysfunctional illness beliefs’ (Moss-Morris et al., 2013). Many patients have raised concerns that CBT is being promoted as a cure for CFS, when there is little evidence to support this claim (Geraghty and Blease, 2016). The majority of patients are pragmatic. They are aware that there is currently no agreed cause for the condition, although an increasing amount of research points to immune and cellular alterations as important clues (Green et al., 2015). Many health professionals are in the dark about the condition, most likely the result of education programmes that do not adequately cover CFS. What patients appear to want is better recognition of the condition among all professionals, doctors, nurses and those in psychotherapy; deeper scientific enquiry that does not only focus on the social-psychology of CFS but explores potential biological aetiology and pathophysiology; and they also want better support within current health structures.

Many patients with CFS may need psychological support, particularly help with coping with the distress the condition can cause; but this is a far cry from a CBT-GET intervention regime that emerges from the fear avoidance model that seeks to convince a CFS patient that the pain or fatigue they are suffering, are dysfunctional cognitions that need to be altered (White et al., 2007, 2011b). Patients with CFS might value CBT more if it was offered as an adjunct support therapy, alongside good quality care from a knowledgeable physician. There is a role for psychologists to support patients with chronic health conditions that can include secondary depression, anxiety and vulnerability to suicide (Fuller-Thomson and Nimigon, 2008; Jason et al., 2006). The problem for many CFS patients has been that CBT is not offered as a support, but as a therapy to reframe illness beliefs. However, the re-analysis of the PACE-Trial data shows that CBT is largely ineffective at restoring physical function. Irrespective of the protocol changes, the majority of trial participants did not report benefit and the PACE team concede this, and they also agree that their trial was only applicable to milder cases of CFS, those fit enough to undertake the treatments (QMUL, 2016a).

The fall-out and the future

‘PACE-Gate’ stands as an example of how science is not always a simple process of discovery and reflects the ills of contemporary science in microcosm. As a result of the PACE-Trial saga, it is likely that patients with CFS will be less trusting of doctors, scientists and psychotherapy practitioners. To win this trust back, the medical-scientific community must learn lessons from PACE-Gate. First, was it wise to commission a small group of scientists that held very strong published views in favour of CBT/GET as treatments for CFS, to be the ones to test the efficacy of these treatments? Conflicts of interest have always been the thorn in the side of clinical trials and a major source of investigator bias (Goldacre, 2010, 2014; Marks, 2017). Funders of future trials must consider the independence of those entrusted to carry out clinical trials. Second, what role did the PACE-Trial steering committee and external adjudicators play in this saga? Why did it take patients to spot anomalies and irregularities in PACE-Trial publications? Third, we must question why the PACE authors were never required to publish the original trial protocol results alongside the results from the amended protocol? It appears that none of the funders, steering committees or peer reviewers called for this. The Lancet editor, Richard Horton, made a spirited defence of PACE in the media post publication, claiming critics were ‘a small but highly vocal minority’ (ABC Radio, 2011). PACE-Gate has exposed how it may be too easy to rush to a judgement about critics. In the PACE tribunal hearing, we saw evidence that a team of distinguished professors based in elite institutions characterised critical patients as ‘vexatious’ (HMTS et al., 2016).

There are important lessons here: dissenting voices must be heard, clinical trials must be conducted by independent investigators (as much as is possible) and trial data must be publicly accessible. No scientific study should be immune from criticism. Without criticism, science will suffer and progress will arguably take longer. There is no utopia in research and no trial is free of all bias and error. However, there are accepted scientific procedures and standards that appear to have been neglected, or bypassed, by the PACE-Trial team. Their actions have arguably caused distress to patients, added a million pounds of additional costs to a publically funded trial and have left us with two versions of ‘truth’ concerning the trial’s findings – the published analysis versus the recent re-analysis. It will be up with others and health authorities to decide which version to accept.

Conclusion

The PACE-Trial has been a controversial clinical trial for several reasons. First, CFS is a controversial and contested illness domain. Many CFS patients reject the theoretical rationale for the use of CBT and GET. PACE-Gate exposes the long-running acrimony between doctors and patient groups over the cause of the illness and the most appropriate approaches to treatment. Second, the lowering of improvement criteria after the trial had begun appears to have significantly inflated the benefits of CBT and GET. Third, patients had to engage in a long-drawn-out battle to gain access to the trial data. This exemplifies why data sharing standards are needed. The fact it took a tribunal to order the release of data has done much to damage the reputation of the trial and has added fuel to the fire concerning the conduct of the trial. Fourth, the release of the PACE-Trial data revealed a dramatic reduction in the benefit of CBT and GET as treatments for CFS, applying the original trial protocol. Collectively, PACE-Gate has damaged the trust CFS patients place in health professionals and science. It will now be up to health authorities to make a judgement on the revelations that CBT and GET may be less beneficial than first reported. If CBT and GET bring about improvement in self-reported mood and fatigue for just 10 per cent more patients than SMC, with little impact on restoring physical function, this indicates that these therapies are non-curative and should be downgraded to adjunct support-level status.