Unlimited Questions at the Limits of Science

T he House of Representatives of the 112th Congress of the United States has voted to repeal the Health Care Reform Act early in its session, and in response to the campaign promise of the Republican majority. Although it is extremely unlikely that the Bill to repeal would be approved in the Senate, and it is a certainty that it would be vetoed by President Obama, serious efforts are underway to develop and pass more palatable replacement legislation. It is clear that healthcare costs and expenditures to provide care require careful management and diligent assessment given the increasingly large portion of the economy that they occupy. This reality is not under dispute and is understood by Republicans and Democrats, employers and employees, providers, insurance carriers, and consumers alike. Exactly how all of this should be accomplished is much more contentious, however, depending upon the particular stakeholder.

Using data to help drive decisions and optimize the use of resources and the outcomes of treatment is a key component of the healthcare transformation. Readers of this Journal are not strangers to this concept. Scientists and clinicians assess and solve problems by gathering and analyzing data as a matter of routine. Both gather and use knowledge gleaned from attempting to answer one or more questions relevant to the issue at hand. Clinical science is moving toward a greater dependence upon the results of Randomized Controlled Trials (RCT) and Comparative-Effectiveness Research (CER) as mechanisms to define the evidence upon which Evidence-Based Medicine (EBM) is to be practiced. Some recent articles pertinent to these topics warrant our attention. ^{1 –3}

Robinson and Goodman investigated the frequency with which RCTs cited prior trials that studied a similar intervention. ¹ They identified 227 meta-analyses that comprised some 1,523 trials published from 1963 to 2004 in 44 medical disciplines. They found that 23% of 1,101 RCTs that had 5 or more prior trials to cite, failed to cite any prior RCTs, and another 23% cited only a single prior trial. The median number of prior trials cited was 2 regardless of the number of citable trials available. The studies that were cited were not the largest ones, and the strength of qualitative or quantitative evidence did not predict the probability that a prior trial would be cited. Older articles, defined as having been published 10 or more years previously, were cited less frequently than those that were published more recently. These findings were consistent with investigations of citation bias in research. ¹ Fewer than 25% of preceding trials were cited, comprising fewer than 25% of the participants enrolled in all relevant prior trials.

This means that evidence from older trials tends to be neglected by “new” trials. Potential implications of these practices include ethically unjustifiable trials, wasted resources, incorrect conclusions, and unnecessary risks for trial participants. All of these are of course unintended consequences. The problem of turning a blind eye to “old” information is certainly not new. Such thoughts are implanted at a relatively early age in one's professional education or career. I have vivid memories of a professor of pediatrics on the faculty of my medical school who had a particular penchant for requiring students on his service, myself included, to research particular clinical conditions and prepare a written report and deliver the findings at a case conference. His emphatic position was that any material or references that were 5 or more years old were “suspect” and “useless” information.

This brings to mind the often paraphrased quotation of the American philosopher George Santayana (1863–1952), who is credited with saying “Those who do not remember the past are condemned to repeat it”. ⁴ My professor was correct to the extent that medicine and therapy evolve with new knowledge and time. Improved treatments and better drugs replace older, more dangerous, or potentially lethal paradigms. Examples of this abound. Consider for example how the development and use of penicillin and the myriad antibiotics that followed saved countless lives and obviated the morbidity of arsenicals, and drugs based on salts of mercury and other heavy metals that were used to treat venereal and other infectious diseases in the pre-antibiotic era. The new treatment rendered the management paradigm of the past truly useless. However, the older references and strategies still hold nuggets of useful information relative to the natural history or clinical picture of a disease or process and might well offer a more complete picture, particularly for something that is much rarer in our modern society that it was in bygone times.

The American Recovery and Reinvestment Act earmarked $1.1 billion as a stimulus for translational and health services research. ³ The latter is generally intended to promote CER, which is hoped to provide new information and insights about the effectiveness of drugs, medical interventions, and medical technologies. Sullivan and Goldmann ³ note that better evidence is needed to address the translational gap between clinical studies and everyday practice in order for this investment to result in better, more cost-effective healthcare. They note that the new knowledge of strategies to implement the findings of CER will come largely from prospective studies. However, they caution that RCTs tend to be slow, expensive, and insensitive to the heterogeneous real-world contexts in which their findings are ultimately to be applied. ³ They note RCT shortcomings including restrictive entry criteria that limit the rate of recruitment and ability to generalize results, and stringent protocols that are not adaptable to emerging new knowledge about interventions, or changes in the environment of the particular trial's application. ³

A second prong of the CER push is the use of data mining techniques to harvest information from previously published RCTs as a means of answering specific questions and directing efficient and effective healthcare interventions. Djulbegovic and Djulbegovic ² argue that mining efforts cannot provide definitive answers to the questions asked by the CER program. They state that CER should be considered hypothesis-generating research, which should in turn be used to provide the basis for future prospective studies. They further note that these future RCTs and other “observational studies” will invariably require new and better data collection than was used or was available at the time of the earlier studies analyzed by the CER program. ² They further argue that the data mining approach can never result in credible discoveries that will obviate the need for the collection of new data. The basis for this conclusion stems from the thesis that the authors of the original database used for retrospective data mining could not possibly have anticipated or realized what new advances in medical science will be introduced and what kind of new discoveries will be made. They point out that this creates a paradox, which is particularly evident when searching for treatment effects in subgroups. As new research generates new evidence of the importance of tailoring treatments to a given subpopulation of patients, the existing databases will need to be updated, in turn undermining the original purpose, to discover new relationships via existing records. ³ In other words, as one question is answered and explanations are offered, more questions are raised.

Science is indeed an open-ended system, as these authors point out. This is undoubtedly part of the beauty of science and scientific research. I have often quipped that it is the reason for the “RE” in research. Our own work often generates more questions and themes for further study after the present project and data have been analyzed and explained. Therefore, we often find ourselves at the limits of science. As we understand more, we begin to realize that our capacity and adaptability are constantly moving and adapting. Although thorough understanding of prior work is prudent, we must recognize that hindsight and retrospective analyses cannot answer all of the questions that time and new knowledge generate.

As Santayana observed: “[w]e must welcome the future, remembering that soon it will be the past; and we must respect the past, remembering that it was once all that was humanly possible.” There are many unanswered questions even at the limits of modern science.