Abstract
T
Subjectivity is not a new problem for Cochrane.
2
Despite the objectivity of Cochrane's processes once papers are selected,
3
there are four areas surrounding reviews that may undermine their credibility:
1. Topic choice is based on researchers' interests rather than patient or health policy priorities. This inability to set priorities for researchers limits the utility of many reviews. One example is reviews of highly specific questions where it is already known there are few or no quality data. Time spent on such reviews would be better invested in generating new evidence. 2. Timing includes (1) doing reviews as soon as there are sufficient data to be analyzed and (2) including all relevant years in the search strategy.
The Cochrane logo's forest plot summarizes the randomized controlled trial (RCT) evidence on administering a short, inexpensive course of corticosteroids to women about to give birth prematurely, which were available in the 1980s. Because no review occurred until two decades after the first RCT, innumerable babies died.
Which years are included in a review is an arbitrary decision of the reviewing team. Some older studies may be still seminal and must be included. For the most recent studies, searching databases for registered trials is an example. Cochrane reviews or their updates should be published as every major trial is completed, not at arbitrary intervals. This decision should not be at the discretion of authors, but a Cochrane policy. Publishing a Cochrane review where it is known a major study is already available but only in abstract form and not including it is equally problematic, especially where a new trial could change the overall conclusions. 2 Minimally, Cochrane authors should conduct a sensitivity analysis to explore how their conclusions would change if the abstract's trial results may change findings.
3. Authors of a Cochrane review may also author one or more major papers in the field. This has strengths and weaknesses. Excluding someone because he or she is an author of a pertinent paper may disqualify people who are best positioned to do the review. Conversely, relying on authors of key papers to define eligibility criteria may allow competing papers to be excluded. Such authors should set a level of accountability about a paper's eligibility that is higher than normal. Eligibility, as with any study, is a major determinant of the study's outcomes and reviewers are able to influence this subjectively. Transparency in this process is crucial for long-term confidence in Cochrane's processes.
4. Cochrane has started to withdraw systematic reviews. Only the lead author on the team is asked if the team wants to update the review. If the lead author declines or is uncontactable, the review is withdrawn. Sadly, the more important and yet missed question is whether there is anything that allows the review to be meaningfully updated. Are new data available that are of value? This is a highly subjective process if one has not done another systematic review. Further, not all reviews are withdrawn at the same age.
A Cochrane review may be out of date two weeks after being published if a major study has been omitted, but may still be current 10 years after the original review if no new studies have been completed.
The term ‘withdrawn’ tarnishes the original review. Published papers and reviews in other journals are withdrawn only when the work is flawed, and readers naturally assume the same for Cochrane reviews. Having a study ‘withdrawn’ on PubMed and other publicly available bibliographies does not convey adequately what has happened—was this withdrawn because it was perceived to be out of date? there are new data? or of an arbitrary decision? Does Cochrane believe that their readers cannot read the date on the review, and account for this? Cochrane should abandon ‘withdrawing’ reviews and instead encourage updates as pertinent new data become available.
The Cochrane Collaboration still makes a major contribution to the ability of clinicians to make a “conscientious, explicit, and judicious use of current best evidence” in the care of individual patients. 4 It is, however, still exposed to subjective or arbitrary decisions, leading to potential inconsistencies. Ensuring that tools are adequately nuanced in order to deliver synthesized data that can inform clinical and policy decision making is absolutely crucial. Strengthening the processes of Cochrane is an investment in the health outcomes for patients in the future.