Maintaining the Integrity of Qualitatively Driven Mixed Methods: Avoiding the “This Work Is Part of a Larger Study” Syndrome

In an editorial in this journal over a decade ago, Morse (2010) commented that many submissions to QHR were metaphorically “throwing some crumbs from the banquet table” (p. 147) to the readers of QHR. These submissions described what was being reported as “this work is part of a larger project” (p. 147). However, details of what was on the banquet table, the larger study, were not given in the submission to QHR. In fact, these details may not have been available at all, if the larger study referred to in the submission had not yet been published (or in some cases never was). This made it very difficult to make decisions about the trustworthiness and significance of what was being reported.

For Morse as editor, the statement raised questions of duplication and what has often been referred to in the literature as redundant publication, or “salami slicing” (Smart, 2017). Was QHR getting the crumbs or the most significant part of the study? And what does this say about QHR and the way it, and qualitative inquiry, is viewed by those submitting this partial study?

For reviewers, it raised the issue of how to review part of a study when they did not know what the other parts were, or have a sense of the overall larger study. Often the submission would not describe or explain either the larger study, or the other parts making up that study. They were simply referred to as forthcoming or “in press”. This made it impossible to check on both the larger study itself and the role that the manuscript being submitted played or contributed to the “larger study.” Just reading that these reports, or this information, are “forthcoming” or “submitted for review elsewhere” is not all that convincing in terms of giving a reviewer a sense of the whole picture of which the submission is part.

Fast-forwarding to the present, we can ask: has anything changed? Unfortunately, the answer to this is that not much has. The editorial board of QHR continues to see variations of the “this work is part of a larger study” syndrome in submissions to the journal. However, one change has been the emergence and consolidation of a mixed-methods-related variant of this syndrome. These are submissions that claim to have used a mixed-methods research design of some sort. What QHR receives is the “qualitative bit” of that design.

In the vast majority of these manuscripts, “the larger study” is actually a single mixed-methods study—usually a QUAN-qual design. As such, it comprises a core quantitative component (QUAN) and a supplementary qualitative component (qual). What is submitted for publication to QHR is the supplementary qualitative component (qual). This is despite the fact that, by definition, the supplementary component cannot stand alone from the core component (Morse & Niehaus, 2009).

Thus, by definition, a qual component is not a “complete qualitative project”. Rather it is a more minor, and therefore incomplete, component. It is created to provide supplementary data to answer a part of the research questions that the core component (most often QUAN) cannot access. Hence it is called a supplementary component. This means that this supplementary component does not make much sense when it is reported in isolation, disconnected from the overall study of which it is part. Supplementary data are just that—supplementary to something else and in some way incomplete in their own right. Therefore, unless a qual supplementary component is reported in relation to the core component of the research of which it is supplementary it is inadequate and makes no sense.

Yet we continue to receive submissions to QHR where the supplementary data from the qual part of the study is reported in isolation, disconnected from the overall project, and submitted as a qualitative study in its own right. It is described as “the qualitative part” of a larger mixed-methods study. In such submissions, what we are seeing is a refracted mixed-method-related version of salami slicing involving “the practice of dividing data sets from one study into multiple journals articles with differing levels of depth and breadth of content” (Happell, 2016, p. 29).

It is a refracted version of this practice as it is no longer “simply” dividing data sets to make more articles. Rather, it is actually slicing and dividing an incomplete data set (qual) away from the core component (QUAN). It is this core component that justifies why that incomplete data set in the form of a supplementary component (qual) was a necessary and essential part of the study in the first place. This type of slicing creates serious problems for both mixed-methods research and qualitative inquiry.

With respect to mixed-methods research, it devalues the promise offered by mixed methods to add depth and richness to the research of complex problems. This is because it fractures mixed-methods projects into segments that are torn apart when submitted for publication. In so doing, it violates the rationale and reason for the mixed-methods approach to be employed in the first place (Bryman, 2007; Cheek & Morse, in press; Greene, 2007; Maxwell et al., 2015). The understandings that emerge from the integration of the different components in a mixed-method design is lost.

(R)emoving the supplemental component from the core, the researcher destroys the mixed-method aims, and the publication resulting from the core component is more limited (and less interesting) than it would be if it were published as an integrated mixed-method study. (Morse, 2015, p. 220)

With respect to qualitative inquiry, the most pressing problem is that viewing an incomplete/supplementary data set as the “qualitative part” of a larger study, can reduce understandings, or create misguided impressions, of what makes excellent qualitative research. Reporting on four interviews or two focus groups ¹ and then analyzing them to answer descriptively several basic, and somewhat obvious, questions is not what excellent qualitative inquiry is. This type of reporting results in findings that are shallow, descriptive, uninterpreted, and consequently unconvincing and insignificant. How can they be anything else when they are only part of a study and not even a complete part in their own right at that?

One trend that has exacerbated all this is that many mixed-methods text books and procedural manuals have used published studies of “mixed-methods” research as exemplars for how the research (including the qualitative component) can, and even should be, conducted and reported. Using an incomplete supplementary qualitative component as an exemplar of qualitative research, runs the risk that the danger is that this form of qual, (something I think of as little qual to reflect the belittling of qualitative inquiry), over time becomes synonymous with what qualitative inquiry itself is.

This is even more of a possibility if those reviewing the qualitative research component in a mixed-methods study do not have extensive expertise in qualitative inquiry per se. Rather, they base their judgments of what “good” qualitative inquiry is on what they have read about qual components of mixed-methods studies. After all, a dominant definition in mixed-methods-related research is the use of quantitative and qualitative methods or approaches or inquiry in the study design. Therefore, it is not too much of a stretch to imagine that if what is being reported as the qualitative “bit” or “part” of a study (see Cheek & Morse, in press) is a qual component, then gradually that qual component could be become synonymous with what qualitative inquiry is. The issue arising from this is the question of “how much adaptation of qualitative inquiry is possible for a supplemental component to still be considered qualitative inquiry” (Morse et al., 2018, p. 569). This is an issue that has not yet been adequately addressed, but urgently needs to be.

Thinking about all this more raises even more questions. If a supplementary qual component is not considered to be qualitative research then how can a submission based solely on that supplemental component be called a report of the “qualitative part” of a study? (see Cheek & Morse, in press). Or how can it be described as the “qualitative article” of the often-recommended suite of three articles for a mixed-methods study: “a helpful way of thinking about writing empirical articles for mixed methods research is to think about generating three written products from a single study: a quantitative article, a qualitative article, and an overall mixed methods article” (Creswell, 2015, p. 92). How does this advice tally with the assertion by Morse that “the supplementary component is not a complete method in itself, and it is not interesting enough or complete enough to be publishable by itself” (Morse, 2017, p. 4)? Which in turn raises the really big questions of who can say what about what can be considered as qualitative inquiry? Based on what? Where do these norms (or rules) come from?

So, we have a problem. The “this work is part of a larger project/study” malaise is still alive and well in submissions to the QHR journal. This is particularly so in many submissions related to mixed-methods research that purport to be reporting a qualitative study but are in fact reporting the qualitative supplementary component of a mixed-methods study. Given some of the undesirable and potentially destructive effects that this can have on qualitative inquiry, how can this malaise be treated and the potential contribution that qualitative research can make in a mixed-methods research design be realized?

One way to do this is to require that the emphasis when reporting mixed-methods research is on what was mixed, and how. How were the methods integrated, where and why did the data interface, and how did this design contribute to richer and fuller understandings of the focus of the research? Another way is to require authors to explain how it is possible to report only part of the research that makes up a mixed-methods design given

(t)he core meaning of mixing methods in social inquiry is to invite multiple mental models into the same inquiry space for purposes of respectful conversation, dialogue, and learning one from the other, toward a collective generation of better understanding of the phenomena being studied. (Greene, 2007, p. 13)

How can we achieve this core meaning of mixing methods if we then sever them from each other and attempt to report each of them as self-contained studies?

Simply writing “this work is part of a larger study” is not good enough. Why it is appropriate to submit work that is part of a single larger mixed-methods study for publication in its own right needs to be clarified and explained. This is especially so when it is the incomplete qual supplementary component that is being reported as a stand-alone study. Such clarification will include demonstrating why a qual component can be considered qualitative research

“(i)f there is no longer saturation of data, or no clear information given about how the questions were asked/areas probed in the qual component of the study, be it QUAL-qual or QUAN-qual, or if the participants were not purposively chosen,” (Morse et al., 2018, p. 569),

Therefore, when we come across an article reporting what is described as “the qualitative part” of a larger mixed-methods study, we need to ask ourself the following types of questions to make decisions about the trustworthiness and significance of that qualitative research:

Do I know what the larger study is of which this research is part?

Which part of the study am I reading—the core or a supplementary component?

Is the report complete enough in itself to stand alone as a report of qualitative inquiry?

If the answer to question 3 is yes, then how was this stand-alone study integrated with the other parts of the larger study and published as an integrated mixed-methods article which presumably is the “larger study” that the authors refer to?

We have also found these types of questions useful when reviewing and making decisions about whether to accept, or not to accept, manuscripts that describe what is being reported as “part of a larger study.” Consequently, they can prove useful points for authors to think about when they are thinking about submitting manuscripts related to mixed-methods studies to QHR. As we have indicated previously (Morse & Cheek, 2014, 2015), QHR publishes qualitatively driven mixed-methods articles—QUAL-qual, QUAL-quan studies. We do not publish the qualitative supplemental component from a quantitatively driven study—QUAN-qual.