Abstract
Using checklists in manuscripts are perceived to indicate quality, transparency, and rigor. Generally, these checklists consist of a list of all of the strategies that may be used to ensure rigor and transparency. Beside each item, there is usually a box to check (or tick) to indicate whether a component is present, and a space on which to note the page each item is listed in the manuscript. Some of these forms also include space for the author to make brief comments to the reviewer. The intent is that the checklist guides the review process to ensure that all components are present in the article, and therefore, that the article is solid enough to publish.
However, these checklists consist only of technical/mechanical management of the creation and sorting of data. These lists ignore the value of the product of the research: They do not address the originality, the substance, the contribution, and the potential results to the actual topic—which is after all the purpose of the project itself.
Paradoxically, these checklist reviews are undermining the quality of qualitative inquiry. In seeking quality, the criteria for systematic reviews, clinical trials, and evidence have spilled over to represent quality criteria for all qualitative research. They are becoming commonplace for evaluating qualitative research by journal editors, directing the review process, and subsequent evaluation of the research. Of greatest concern is that checklists items are being used by authors themselves to represent their actual text (e.g., “data were saturated”), and the items for completing these forms are read by the reviewers and editors in lieu of reading the article itself (e.g., for signs of “saturation”). Furthermore, the use of these criteria by authors/researchers to guide the conduct of their research, yet meeting all these criteria, whether relevant or pertinent or necessary for their project, and may even invalidate the findings. In this way, these criteria are redefining processes of qualitative inquiry.
Descriptive and Interpretive Research Require Different Criteria for Review
Different agendas exist for descriptive and interpretive research. Accuracy and replicability are important for descriptive research, but less so for interpretive research where the concern is to get in-depth data that provide understanding as the research unfolds. In qualitative descriptive research, because the semi-structured interviews have not changed over the course of the study, interviews are comparable and data analysis permits the comparison of each item in the participant pool. This is not possible with interpretive research where interviews are not standardized, but each interview incrementally adds understanding to the phenomena under study, hence, different criteria must be used for descriptive and interpretive qualitative research, and to use descriptive methods for interpretive inquiry, or interpretive methods for ensuring rigor in descriptive inquiry, invalidates the research. In interpretive research, a single quotation or comment may have important meaning to the interviewer (who can recognize the implications of the text within the context of the entire interview) but may be taken at face value and insignificant when rated by another researcher. Hence, in interpretive research, interrater reliability used in descriptive research will keep the findings shallow and superficial, and even invalidate the study. (This has been explained elsewhere—see, Morse, 1997.) Of course, careful description may move toward interpretation as the project develops, which means that some of the checklist strategies may be used at different stages of the research—obscuring the checklist ratings. To complicate matters even further, all checklist strategies do not fit all qualitative methods, yet checklists do not accommodate for these differences. The original developers of each method have outlined strategies for excellence in their texts, and these should be used in evaluation rather than a generic list of techniques.
Other Unintended Consequences
In addition to the mis-fit of the use of the criteria for all qualitative research, there are other “side effects” to their use that are seriously changing our research.
The Format of Checklists
Developed for standardization, comprehensiveness, and efficiency, checklists may be foregrounded in the process so that checking/ticking the checklist takes priority over thoughtfully reading the document and considering the substantive content of the research. Just because the reviewer notes that something was mentioned (i.e., “member checking”) we cannot assume because it was listed that it was actually done. In the case of member checking, we may not know when (following transcription for accuracy? or following analysis for agreement with interpretation?), or how participant input affected the results (Schwartz-Shea, 2020). Thus, with checklist reviews, the technical aspects of the research, whether appropriate or not, are given priority by the reviewer rather than as a contribution of the research per se.
Checklist as a Method
Increasingly, researchers are using checklists to guide their research, believing that the checklist is the path to perfection, hence publication. Excellent research is not attained by the mechanical application of strategies or techniques but by the careful attention to the content of the interviews, to synthesizing, the linking of this content to what is known in the library, and the logical, in-depth narrative presentation of the findings.
The Checklist Review
The purpose of reviewing an article is not only to determine whether the article is “publishable.” Reviews also provide input for the researcher, not only on the presence or absence of a strategy but also to describe how the strategy has enhanced the research—or conversely, if the strategy is absent, how to correct that deficiency and make the article stronger. Thus, from the review process, the author learns (from strengths and mistakes), and the reviewers learn (by reading great—and less great—articles). As our articles improve, so our discipline advances. Narrative reviews benefit both the author and the reviewer. By replacing a narrative review with a checklist, we are doing ourselves a disservice.
The Narrative Review
The alternative, a narrative review, uses questions that the reviewers may ask of the article; the balance of its pros and cons when writing the review are invaluable. I am proud of the quality of our reviewers, the care, attention, and time that reviewers give our authors, and the impact that these reviews have on the quality of our publications. Reviewers routinely write several single-spaced pages in their review, and I think “This author is so lucky!?”
Following, are the questions in the Qualitative Health Research narrative review form that reviewers are asked to consider when reviewing a manuscript:
Qualitative Health Research Review Form
Please give specific advice to the author on how to improve the manuscript:
Notice that the first question asks about the importance/significance of the topic. Also note that the questions are open enough to permit the reviewer to assess the research within its method and context.
Solution
Checklists are taking over our world. A recent review by Munthe-Kaas et al. (2019) surveyed 102 checklists. Even the APA manual (American Psychological Association, 2020, pp. 77–81) which is charged with dictating format, is a qualitative research checklist that includes criteria for quality of qualitative research. We are losing control of our discipline.
What shall we do? There has been little attempt on the part of checklist developers to explain or to describe how they were developed and tested, or the appropriate use of these checklists—either for the type of research, the methods used, or the phenomena being studied. Given that, and the problems inherent in their application for qualitative inquiry, it seems legitimate that we refuse to use checklists in qualitative reviews, either as authors or as reviewers. As we review articles, we comprehensively assess the phenomena we study, the value, and contribution. Why should we rigidly, shallowly, and even invalidly rate and discard our submissions?
Footnotes
Acknowledgements
This Editorial was adapted from: Morse, J. M. (in press). The fallacy of rigor: Examining checklist criteria as an indicator of quality. In U. Flick (Ed.), The SAGE handbook of qualitative data analysis. Sage. I thank Julianne Cheek and Alex Broom for their comments on an earlier draft.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.