Abstract
Plans for new research should use systematic reviews of relevant existing research to provide the scientific and ethical justification for the design and conduct of new studies. Updated systematic reviews should be used to place new findings in context and thus provide evidence relevant to informing decisions in policy and practice. This helps to avoid research waste and ensures that users of reports of randomised controlled trials (RCTs) will have ready access to up-to-date accounts of the available evidence on the topics of interest.1–3 These principles were promoted in the Explanation and Elaboration document for the 2010 CONSORT statement, which recommends ‘… that, at a minimum, the discussion should be systematic and based on a comprehensive search, rather than being limited to studies that support the results of the current trial’. 4
Over a period of 25 years (1997 to 2022), we conducted a series of audits to assess the extent to which these principles were observed in five high-profile general medical journals: Annals of Internal Medicine, BMJ, JAMA, Lancet and New England Journal of Medicine. In the first audit in this series, we concluded by paraphrasing John Donne 5 : ‘No trial is an island, entire of itself; every trial is a piece of the continent, a part of the main’. 6
Our first audit showed that only 2 of the 26 reports of RCTs published in our ‘sentinel’ journals in May 1997 included updated systematic reviews in their Discussion sections, although four others mentioned systematic reviews in that section of the reports. Our audit was expanded in 2005 to include assessments of the use of systematic reviews in the Introduction sections of the RCT reports. 7 This, and our audits in 2009 8 and 2012, 9 showed that most reports of RCTs did not use their Introduction sections to mention the use of systematic reviews for trial design or justification.
We used similar methods to those in our earlier audits to identify reports of RCTs reported in one of the five ‘sentinel’ journals and to determine how systematic reviews were used in their Introduction and Discussion sections.6–10 For this final article, we assessed the Introduction sections of the reports from 1997 and 2001 to provide a complete dataset for both the Introduction and Discussion sections across the 25 years.
Across the 25 years, we identified 175 reports of RCTs in Annals of Internal Medicine, BMJ, JAMA, Lancet or New England Journal of Medicine in the month of May in 1997, 2001, 2005, 2009, 2012 or 2022.
How did these five high-profile general medical journals perform?
The Introduction sections of only 2.9% (5/175) of these reports contained references to up-to-date systematic reviews to inform the design of the new RCT; 31.4% (55/175) mentioned previous systematic reviews in the topic area; 37.7% (66/175) cited other RCTs; and 18.9% (33/175) did not claim to be the first RCT addressing the topic in question and did not contain references to other RCTs or systematic reviews.
In their Discussion sections, the findings of the new RCT were integrated into an updated systematic review in 3.4% (6/175) of reports; 28.6% (50/175) cited previous systematic reviews but did not integrate the findings of the new RCT in them; 12.0% (21/175) claimed to be the first RCT addressing the topic; and the remaining 56.0% (98/175) did not provide information to suggest that any citations for other RCTs arose from a systematic attempt to set the new results in context.
Figures 1 and 2 present the findings for the Introduction sections and for the Discussion sections, respectively, for each of the six audits across the 25 years. Comparing May 1997 and May 2022, the proportion of reports of RCTs mentioning a systematic review in their Introduction sections increased from 11.5% (3/26) to 61.8% (21/34). The proportion that mentioned a systematic review in their Discussion sections increased from 23.1% (6/26) to 52.9% (18/34).
Classification of Introduction sections in reports of randomised published in May in Annals of Internal Medicine, BMJ, JAMA, Lancet or New England Journal of Medicine.
Classification of Discussion sections in reports of randomised published in May in Annals of Internal Medicine, BMJ, JAMA, Lancet or New England Journal of Medicine.
What have others found?
In 2022, two systematic reviews were published of studies that, similar to ours, had investigated the use of systematic reviews in the design of new research or in reports of the findings of research. The findings of those reviews are not inconsistent with our overall findings. Nørgaard et al. sought to identify and synthesise the results of meta-research studies of the use of systematic reviews to design new health research studies. They included 16 studies published between 2005 and 2021. In these reports, the proportion using a systematic review to inform the design of a new study was in the range of 0% to 73% (mean: 17%). 11 Draborg et al. identified and synthesised results from meta-research studies that had assessed whether reports of health research studies had used systematic reviews to present their findings in the context of data from similar studies. They included 15 meta-research studies published between 1998 and 2021 (including our previous five audits). They calculated that a mean percentage of 30.7% (range: 9%–48%) of the original studies had placed their findings in the context of existing studies. 12
More recently, Jia et al. 13 reported a cross-sectional survey that used 737 Cochrane Reviews published up to October 2021 to identify 4003 RCTs that had been published at least two years after the first version of the review and were included in an updated version. These RCTs were published between 2007 and 2021 and Jia et al. screened their references to identify citations of prior Cochrane or other systematic reviews. They found that 56.6% (2265/4003) of the studies cited one or both types of systematic review, and 43.4% (1738/4003) did not cite any systematic reviews. The percentage of RCTs not citing any systematic review decreased from 64.5% (118/183) in reports published in 2007 or 2008, to 28.4% (29/102) of reports published in 2020 or 2021. 13
How might these deficiencies be overcome?
In keeping with those other studies, we have found that over the 25 years of our audits many reports of RCTs failed to make best use of systematic reviews. They have thus failed to provide readers of their research with robust evidence to support decision making in practice or in planning new research. However, over the past decade, there has been an increase in the use of systematic reviews in the Introduction and Discussion sections of reports of RCTs. In the five major general medicine journals we assessed, the proportions citing systematic reviews increased from 27.7% and 27.0%, respectively, in 1997–2012, to 61.8% and 52.9% in the 2022 sample. It is uncertain whether this represents progress with the concept of evidence-based research14,15 or simply that systematic reviews have become more common and therefore easier to identify and cite, 16 with a particular boost during the COVID-19 pandemic. 17 However, there remains substantial room for improvement, with 37.9% (10/27) of the reports from May 2022 that did not claim to be the first RCT making no systematic attempt to place the RCT’s design in context in their Introduction sections, and 33.3% (9/27) failing to do so for the findings of the RCT in their Discussion sections.
Improvements in this area are now much easier than when our ‘Islands audits’ began in 1997, with tens of thousands of systematic reviews being published every year.
16
Furthermore, Glasziou et al. recently illustrated three ways to incorporate new findings in systematic reviews in the Discussion sections of a new study, viz:
conduct an updated systematic review after running an updated search and present an up-to-date meta-analysis; add the new RCT result to an existing systematic review, which requires extracting the summary results for each RCT from a previous meta-analysis, adding those for the new RCT and combining the results, without a new search; or add the results of the new RCT to the previous summary results to generate an updated summary result, without a new search.
18
Where the existing review is a Cochrane review, the second and third methods are particularly easy because data files can be downloaded from the Cochrane Library for all Cochrane Reviews. This facility provides the basis for calculating a revised estimate of the average effects of the intervention and also makes it easier to compare the findings of the new RCT with those of the studies that preceded it.
This type of cumulative meta-analysis would also highlight the value, or not, of the new RCT to the existing evidence base. 19 This is important. If practitioners, policy makers and the public do not have access to the most up-to-date evidence on the effects of interventions, there will be delays in implementing effective treatments and in abandoning interventions that are ineffective or harmful. Failure to address these deficiencies will affect health and sometimes lead to preventable deaths and chronic morbidity.
In an article entitled ‘Modifying the process of science’, Hahn and Teutsch used the findings of the 1992 study reported by Antman and colleagues 20 of what cumulative meta-analyses would have shown about the effects of interventions for patients with acute myocardial infarction. They estimated that failure to conduct routine prospective cumulative systematic reviews had resulted in more than 150,000 deaths annually in the United States alone, from non-use of intravenous vasodilators, aspirin and ß-blockers and from continued use of anti-arrhythmic drugs. 21 They called for ‘modification of the process of science’, 21 like others before them. For example, Collins et al. estimated that tens of thousands of preventable deaths had occurred due to the informed consent procedure required for extension of recruitment to the ISIS-2 trial in the United States. 22
What next after Islands 25?
People making decisions about healthcare need to feel confident when using reports of RCTs. This requires RCTs to be designed in the light of systematic assessments of other similar research, so ensuring that they are really needed, and that they will address relevant, unanswered questions. It also requires RCTs to be reported in the context of other similar research, to ensure that users are able to consider the totality of the relevant evidence. This will need increased recognition that the conduct of RCTs and systematic reviews should not be seen as independent endeavours to be done by separate researchers, but that they should go hand-in-hand. 23
Those who fund and do RCTs and those who oversee their publication and use have an ethical responsibility to ensure that this research is reported in proper context. If this happens, anyone who repeats our ‘Islands audit’ in years to come will be able to conclude with confidence that, in the medical literature: ‘no trial is an island, entire of itself’, and that the findings of every RCT can be seen ‘in the context of the main’. This means the totality of the evidence for improving health and preventing premature deaths. However, unless the deficiency we have shown over the past 25 years is addressed by the research community, people are likely to continue to suffer and die unnecessarily because of a lack of a reliable evidence base for healthcare.