Abstract
I
As a scientist, I fully support the well-established expectation that one should document and provide, distribute, and make available sufficient details of experimental materials, reagents, equipment, resources, etc. that would enable another scientist to repeat those experiments and determine whether or not your approaches and data hold up to scrutiny and are valid. In the United States at least, if federally funded, you may even be required to share material. Given the maximum word limits for peer-reviewed publications, some journals now give authors the option of providing supplemental material so that full details of an experiment can be publicly available.
Unless data are controversial or challenge someone's hypothesis, it is rare that anyone tries to repeat exactly an experiment that has already been published. Most published articles already include replicates, so if someone repeats an experiment and gets the same results, then the data lack the novelty that most journals want to publish. Nonetheless, the expectation that an experiment is reproducible remains as a standard.
Now to the crux of the October meeting discussion. The meeting was organized to discuss the need—be it real or perceived—that not only should scientists provide details of materials used in experiments, but they should also prove that the details are what they say they are or believe they are.
Philosophically, I understand the rationale for this, but at the same time, I have some concerns. During the October meeting, I described a relatively recent additional requirement that has been added by the U.S. National Institutes of Health (see NIH SF424 application instructions) as part of the grant application process. This is included in an NIH resource chart on Rigor and Reproducibility in NIH applications (infographic attributed to Swan et al.).
An applicant must now complete a form that clearly states the sources of, for example, pathogens, cell lines, strains of experimental mice, and more. At face value, this is merely additional time, effort, and paperwork for the applicant. Of course, based on experience, it is possible that what is now required by a major funding agency will be adopted by other agencies and become mandated for scientists throughout at least the United States. One consequence from Open Access requirements is that the number of well-established journals to which federally funded researchers can submit their articles has to some extent been reduced. Coincidentally, as likely many of you know, based on a daily deluge of email solicitations that I receive, there has been a spawning of new journals offering open access, rapid peer review (perhaps), and even guaranteed acceptance with almost instantaneous publication.
My concern is that the move to formalize what ethical scientists are doing already could have adverse effects on science, especially collaborative science. Moving from having to state your source of a biological agent is one thing, but having to prove the authenticity of a reagent may not only be time consuming and expensive, but in some cases impossible.
While it may be relatively simple technically to sequence specific genes to validate the identity of a virus, demonstrating that, for example, a cell line really comprises cells from Aedes albopictus (C6/36) or the E6 clone of Vero cells, or that the albino mice that were used are actually Swiss TO and not Balb/c may be more difficult. Furthermore, is stating the commercial source really adequate authentication? It is implied in the NIH request that stating the source may be interpreted as fulfilling the requirement of authentication, but as described below, there may be a time component of this. Even though one can sequence a viral genome, this still requires reagents, technology, and analytical capabilities for comparative analysis. You may outsource this and mail a sample to a specialist company, but new regulations may require rigorous proof of inactivation. The time, effort, and costs involved may be bearable in developed countries, but even if possible may be unaffordable in others. Mandatory authentication could effectively prevent science in some circumstances. We all know how well intentioned recommendations become regulations, or are interpreted as such by institutional review bodies, which amount to the same thing.
There are other issues related to a demand for authentication. A virus, cell, or mosquito line obtained from a reputable commercial source may be used in a laboratory for years. Over time that original sample may be propagated to generate stocks and experimental cohorts. Either deliberately or as an unintended consequence of the passaging, the original material may accumulate mutations, or dare one say, even be contaminated with other cells, mycoplasma, viruses, or other entities. Phenotypically, slight changes in phenotype may be undetectable, but in the meeting, we discussed how, when samples of the widely used G3 line of Anopheles gambiae mosquitoes from different laboratories were examined, it was revealed that there was considerable genetic variation. Defining a “standard strain,” although desirable, perhaps might be technically difficult and with undefined true value. One cannot help but recall that the enormous costs and efforts that were dedicated to sequencing the A. gambiae genome ultimately resulted in publication of a genome for a strain that had become extinct.
One question that must be answered is how many times after procurement from an acceptable source can a virus or cell line be passaged, and how many generations of a mosquito can you produce before you must reauthenticate? Viruses and cell lines can be stored for years and can be maintained and stored as a large number of samples produced immediately after procurement, but for mosquitoes this is impossible.
Implicit in the discussion is the assumption that something is what you say it is. That is a good thing, if correct, but one might ask is it good enough? If we take mosquitoes as an example, for over 30 years I have been involved in discussion on the optimal choice of mosquitoes for experiments. An argument can be made that the purpose of performing many experiments in which mosquito species X is infected with virus Y is to provide an understanding of what happens naturally in the field. Obviously, there are other purposes, but if we assume that this is the motivation, then it seems intuitive that one should use mosquitoes and viruses that are as close to what is in nature as possible. Complicating even a well-designed study are the use of mice, guinea pigs, rabbits, and other vertebrate hosts, cell cultures, etc., all artificial hosts, although the best available to us. Ideally, one might prefer to use mosquitoes collected directly in the field and used directly in experiments without breeding in the laboratory (i.e., generation G0), and viruses isolated from a viremic host with the assumption that these reflect the natural situation. Unfortunately, this is an unverifiable assumption and perhaps though ideal, often technically challenging. Mosquitoes collected in the field do not necessarily thrive or feed well under laboratory conditions, and primary isolates of viruses may be of low titers or small volumes, or the viruses may not be well adapted so as to be inadequate for use in experiments. Consequently, before being able to conduct experiments it may be necessary to rear mosquitoes for several generations and to pass viruses in vitro or in vivo to obtain enough high-titer viruses for presentation to mosquitoes. Although necessary to facilitate an experiment, such manipulation may very well make the arthropod vectors or the viruses into an artificial system.
The key question is how many generation cycles or passages are acceptable before the question of “authentic” is raised? Using mosquitoes that are fewer than five generations from the field may sound “reasonable,” but in reality, it is arbitrary. Since we do not know the genetic determinants of vector susceptibility to virus infection, we cannot evaluate when a laboratory mosquito is not a field mosquito for even this very important phenotype. There are many studies that, with respect to susceptibility to viral infection, show great variation between even geographically close mosquito populations. Because natural populations can vary between highly susceptible and highly resistant to viral infection, in some respects any laboratory-colonized mosquito could be representative of a wild population. Some researchers have crossed mosquitoes from different populations with different susceptibility phenotypes to produce what could be regarded as a line that represents the typical species phenotype.
Walter Tabachnick performed detailed and elegant experiments that ultimately demonstrated considerable variation in the susceptibility of different Aedes aegypti populations to be infected with, and the ability to transmit, yellow fever virus. Many years ago he told me something that has stayed with me, perhaps one could say haunted me, ever since. He said, “Data from experiments in which mosquitoes are challenged with a virus are only valid for those particular mosquitoes and the specific virus under the conditions used at the time of the experiment. They should not, and perhaps could not be extrapolated to any other experiment beyond this one.” (WJ Tabachnick,personal communication)
Perhaps there is a lesson here for the well-meaning advocates for authentication. Since there is so much variation in all phenotypes due to genetic and environmental causes just what would be the purpose of certifying the authenticity of any virus or mosquito strain. If Tabachnick is correct, then attempts to generalize many experiments beyond the experimental conditions is problematic since we have such little understanding of the complexities of the controlling factors in the laboratory and in nature. Therefore, before developing a recommendation about authentication of resource materials, one must ensure that it achieves its intent, especially since all too often recommendations can be translated into federal requirements. We must consider if the recommendation can achieve the intent, the purpose, the practicalities, and unforeseen consequences, even if the associated procedures are practical and affordable. Although it is stated “NIH encourages the scientific community to establish guidelines for the authentication of various types of key resources,” a May 25, 2016 response written by R. Kahn to the NIH Extramural Nexus blog (Lauer 2016) suggests that “the focus be on journals and editors to enforce a standard, which should be openly displayed and then adhered to. Currently, journals have abdicated almost all gatekeeper responsibilities, yet they remain the gold currency for productivity.” Certainly, peer-reviewed publications remain as a fundamental metric by which the value of research is measured, and for example, by which appointment, promotion, and tenure decisions are made. The well-established peer review process as used by Vector-Borne and Zoonotic Diseases and other reputable journals is used to obtain unbiased evaluations from subject matter experts of research described in submitted articles. Furthermore, they consider experiment rationale, ethics, appropriateness of procedures and materials, validity of data, statistics, and legitimacy of conclusions. One has to ask how much more should an editor, hard-working volunteer reviewer, and journal do? With relatively few, although some well-publicized breakdowns in this system, typically because of unethical and deliberate fabrication of results by unscrupulous investigators, I think that what might be termed a self-policing system works very well.
As an editor, scientist, and human, I defer to my basic belief that most people are honest, trustworthy, and have high ethical standards. What we work on and what Vector-Borne and Zoonotic Diseases publishes is complex and focuses on human and animal health. In my opinion, this is too important to allow politics, personal egos, or ambitions to get in the way and take precedence over doing the right thing. I believe in collaborative science and resource sharing. Over the last 30 years, only a very few individuals have disappointed me with respect to ethical behavior, treatment of others—especially more junior people—and their willingness to collaborate and give credit to those who deserve it, but, despite them, I maintain my beliefs and never enter a meeting with the assumption that I cannot trust the participants. As an editor, I trust authors, believe the data that they submit, and do not want to be a gatekeeper that questions their materials and methods, especially when there is little to be gained by authentication of resources that have little to no value.
As we have seen with other regulations, they can have a negative impact on our ability to do good science and may prevent resource sharing between scientists. Federally funded research often has the requirement to share resources that have been developed with these funds. This promotes experimental validation and collaborative research. A requirement for rigid authentication could disrupt this sharing, especially in the United States, where scientists might be reluctant to share because of the fear of complaint or even litigation if it is subsequently shown that what they honestly believed to be a certain cell line, for example, was ultimately proved not to be what they thought it was. Certainly if one receives, for example, a virus, and it is permitted to redistribute it then one might be concerned that during propagation it may have accumulated mutations so that the sequence of the virus that you send to others is not exactly the same as the seed stock. Sadly, if we go down the path of demanding authentication, not only may scientists be reluctant to send material to others, but perhaps, those who need material for their research may not accept them without authentication. Either way, progress is stifled.
Caveat emptor? Or perhaps one should say when it comes to evaluating resources used in a scientific article “Buyer beware—especially if it is free!”
My thanks to Dr. Charles H. Calisher, Dr. Peter K. Dorhout, and Dr. Walter Tabachnick for their useful comments and suggestions.
Footnotes
Author Disclosure Statement
No competing financial interests exist.