Abstract
R
The aim of an evaluation of the scientific literature is to determine whether a specific article contains valid evidence that will aid in the care and management of patients. This evaluation is serious business. It is hard work that requires discipline and effort.
On the one hand, we have allies for the effort. These allies are strong and include our intellect, our fund of medical knowledge, our computers, and our common sense. On the other hand, our enemies are also strong and include a lack of time, fatigue, busy schedules, family responsibilities, and, sometimes, bright scientific minds that are more interested in being published than they are in the niceties of pursuing scientific truth and intellectual honesty.
Let me give some examples. The New York Times in The Opinion Page of May 23, 2015, reported that every day a scientific article is retracted because of misconduct. Every day! Two percent of scientists admit to tinkering with their data in some kind of improper way. If that seems small, remember that researchers publish some 2 million articles a year, often with taxpayer funding. That is about 40,000 published articles a year that have been “tinkered” with. 2
Recently, Hindawi Publishing Corporation—which oversees 437 academic journals—identified 32 articles with possible fraud. BioMed Central—publisher of 277 journals—announced the retraction of 43 articles after discovery of a “fabricated” peer-review process. The Journal of Vibration and Control (JVC) retracted 60 articles because of a peer review and citation ring. 3,4
How then to proceed:
When reading a scientific article, first consider the source. Is the journal recognized? Is it abstracted in Index Medicus? Is it included in a large database such as PubMed Central?
Is the journal peer reviewed? Review by peers helps—but does not guarantee—scientific appropriateness and validity. Peer review also helps identify possible plagiarism. Ask how many reviewers are involved with each article. Many journals require three independent reviewers for each submitted article.
Inquire if the journal employs means—other than peer review—to detect plagiarism? A number of journals today employ plagiarism detection software to scrutinize articles submitted for possible publication. Excessive similarities between published articles and a submitted article prompt a letter of inquiry to the author(s).
When all is said and done, however, scientific article review boils down to a matter of trust—trust in the scientific integrity of the journal. The journal editorial board and publication staff need to do everything possible to assure the content they publish is of the highest quality and truthfulness. Of course, to quote former president Ronald Regan—who was quoting an old Russian proverb—“Doveryai no Proveryai.” Trust, but verify. 5
To get back to task.
Most journals use an IMRD structure beginning with an abstract followed by an Introduction, Methods, Results, and Discussion.
Abstracts usually contain the following: • The purpose of the study (why they did it) • Methods (how they did it) • Results (what they found) • Conclusions (what does it mean)
Take notes as you read the article—it improves recall and comprehension.
Now to the main structure of an article.
Introduction
The earliest portion of an article that leads from what is known to what is not known and to what question the author(s) asked and answered.
Methods
This section tells what (experiment) was done to answer the question stated in the introduction.
Results
Herein are contained statements of what was found and reference to the data in figures and tables.
Discussion
This section places the work in context of the broad field. It also should provide a clear answer to the question posed in the introduction and explore how the results support the conclusions.
Now … the hard part. This is how I do it.
Begin by skimming the articles for a macro view. Look for the main points. Generate questions before, during, and after reading. Write them down.
Perhaps the worst way to approach the reading of scientific articles for validity is to read it through from the title through to the references as if you were reading a textbook. That is not the way. Be inquisitive, be questioning; try to see whether the material makes sense. Take your time.
Draw inferences based on your own experience and knowledge. Does it pass the “sniff” test? If you are reviewing for a journal—and not just for yourself—read the entire article line-by-line, word-for-word. Read it multiple times.
Do a PubMed search, read other journal articles. Google lots of things—just about anything or any word you do not understand. Check references—are they from reputable sources, are they up to date?
Then think it out. Use prior knowledge and use new knowledge to decide whether the article has merit.
Look for validity—look for bias.
Recall that validity means the extent to which a concept, conclusion, or measurement is well formed and accurately corresponds to the real world.
A bias is a systematic error, a deviation from the truth in results or inference, or an overestimation or an underestimation of the true intervention effect.
Bear in mind that with a large enough sample, even miniscule differences can become statistically significant. This significance is typically assessed with an index called the p value. A p value is a useful statistical measure, but it has all too commonly been misinterpreted and misused. The misplaced emphasis on p values has caused some academic leaders to dissuade the use of p values and some journals to limit their use. 5
Benjamin Disraeli (1804–1881) and, more famously, Mark Twain (1835–1910) best summed this up with the quote: “There are three kinds of lies: lies, damned lies and statistics.”
To recap, to determine the merit of any topic in the scientific literature, a reviewer must be mindful of historical precedent and established facts. The mentioned suggestions for review are guidelines that will help outline a course of action. They work for me, but they are not all inconclusive or the final word on the matter. Be skeptical, always skeptical, and use outside references for things that are not clear. Make liberal use of a Google search.
Above all, use common sense along with your own experience and knowledge to guide you through the process. It is not easy; there are no shortcuts. But the satisfaction and benefits derived from careful study are well worth the effort involved in teasing out valid scientific evidence from the available scientific literature.