Living Within Blurry Boundaries: The Value of Distinguishing Between Qualitative and Quantitative Research

The Indistinguishability of Data: Words and Numbers

One of the oldest and most frequently discussed means of differentiating QUAL and QUANT research are the data they produce: words in the case of QUAL and numbers in the case of QUANT. The indistinguishability thesis argument is based on the observation that QUAL results often include statements that imply a numerical analysis through the use of words such as many, more, and most (see Hammersley, 1992, 1996, for a detailed review of this argument and its history). There is no doubt that QUAL researchers talk in terms of the size and frequency of things. Indeed, it is hard to imagine how anyone could talk about patterns and regularities in behavior without concepts such as “more” and “less.”

The strength of this observation in support of the indistinguishability thesis makes it useful for examining the general format of other supporting arguments. These arguments begin with the assumption that any distinction between QUAL and QUANT research must consist of an either/or characteristic with no possibility of misclassification. The arguments then demonstrate that it is impossible to make such hard-and-fast distinctions in practice. In the case of data, the argument first asserts that QUAL research must rely exclusively on words and QUANT research must rely exclusively on numbers. It then asserts that because QUAL researchers use counting words, no study can be classified exclusively QUAL or QUANT on the basis of data.

Overall, it seems safe to conclude that the distinction between words and numbers as data cannot be used to create mutually exclusive categories of QUAL and QUANT research.

The Indistinguishability of Qualitative and Quantitative Methods

Another potential source of criteria for distinguishing QUAL and QUANT is the methods used to produce the data, which would identify open-ended interviewing and participant observation with QUAL, and questionnaire-based interviewing and experimental designs with QUANT. Once again, the strategy of attack used by proponents of the indistinguishability thesis shows that it is not possible to draw sharp lines, in this case between methods. Sandelowski (2014) is typical:

The idea, for example, that the closed-ended and highly structured questionnaire constitutes a [QUANT] element and the open-ended minimally structured interview a [QUAL] element effaces the countless variations in how questionnaires and interviews may be conceived, developed, conducted, or administered, in the purposes they are intended to fulfill, and in the way questionnaire and interview data may be analyzed, interpreted, and represented. (p. 5)

An example that fits this argument would be survey research incorporates both closed-ended and open-ended questions. Thus, proponents of the indistinguishability thesis highlight that interviewing and other research methods include a diverse range of possible practices, with so much overlap that QUAL and QUANT methods cannot be distinguished. From this point of view, there must be definitive differences between methods for distinctions to be of any use. Acceptance of this point of view leads to the conclusion that it is impossible to state essential differences between QUAL and QUANT methods.

The Indistinguishability of Purposes in Qualitative and Quantitative Research

The search for criteria that distinguish QUAL and QUANT research turns to more abstract purposes such as subjectivity versus objectivity and induction versus deduction. But it is easy to question any claim that QUAL research relies on subjective procedures and QUANT research procedures pursue objectivity by minimizing the researcher’s personal influence on the data collection. Every QUAL research project has to pay attention to the accuracy of the researcher’s observations and conclusions. At the same time, QUANT research always depends on subjective human judgments about what to study and how to study it, no matter how much effort goes into meeting criteria such as replicability. From this standpoint, the acceptance of subjectivity and the pursuit of objectivity are always present in both QUAL and QUANT research, and hence cannot serve as defining criteria between them.

Similarly, Sandelowski (2014) notes that, “To characterize [QUAL] research as inductive and [QUANT] research as deductive is to move to the background the iterative cycling between induction and deduction that characterizes all inquiry” (p. 6). According to this line of thinking, there is no way to make a distinction on this basis because every piece of QUAL research involves at least some element of deductive thinking, and vice versa.

The Indistinguishability of Qualitative and Quantitative Research as Different Paradigms?

Another possibility is that QUAL and QUANT could be distinguished through their reliance on different paradigms. Although there are a variety of paradigms available for MMR, none of them contradict the indistinguishability thesis. This is certainly the case with the approach of Guba and Lincoln (1994), who treat paradigms for social research as consisting of ontological, epistemological, and methodological assumptions. From this perspective, the two primary paradigms are constructivism and postpositivism, which are associated with QUAL and QUANT research, respectively, but without any one-to-one correspondence. In particular, Guba and Lincoln (1994) were quite clear that beliefs about the nature of reality (ontology) and what can be known (epistemology) do not force choices about whether to do QUAL or QUANT research, let alone what kind of method to use or data to collect.

Likewise, pragmatism as a paradigm for MMR (Beista, 2010; Morgan, 2007, 2014) does not draw hard-and-fast distinctions between QUAL and QUANT. Instead, pragmatism would treat QUAL and QUANT as tools that could be used in potentially interchangeable or overlapping ways, depending on the need at hand. So, once again, we have a supposedly defining characteristic of QUAL and QUANT research that is only a tendency, rather than a hard-and-fast criterion that would distinguish QUAL from QUANT.

The inability of paradigms to distinguish between QUAL and QUANT research is the last in a string of failures. Up until this point, I have not disputed any of these arguments, and I am now willing to stipulate that it is not possible to find a set of criteria that will draw an absolute distinction between QUAL and QUANT research. Instead, I believe that the insistence on criteria that create an all-or-nothing, either-or boundary is the problem.

Problems With the Essentialist Approach to Categorization

The first problem with essentialist approach to categorization is that it denies the value of classification criteria because there are always exceptions to any of the proposed rules. In the present case, this means that each of the proposed criteria for distinguishing QUAL and QUANT might increase the probability of being in one category or the other category, but none of them can definitively determine category membership. But how serious a criticism is this? People rely on probabilistic reasoning all the time, because demanding certainty can paralyze decision making. Unless one expects a perfect, one-to-one relationship between any of the proposed characteristics and the ability to classify studies as either QUAL or QUANT, the real question is not one of essences but rather of probabilities. So, if a study uses words rather than numbers as data, the use of words does not define it as a piece of QUAL research, but it greatly increases the probability that it is. All four of the previous proposed criteria are strongly associated with predicting whether any given study can be categorized as QUAL or QUANT research, even if no one of them is conclusive.

The second problem with the essentialist approach to distinguishing between QUAL and QUANT is that authors in this tradition argue against each of the proposed classification criteria one at a time. But these criteria are interdependent and mutually reinforcing. On the QUAL side, constructivism is linked to methodological purposes that are inductive and subjective, which are in turn matched to methods such as open-ended interviewing and participant observation, which typically produce textual data. On the QUANT side, postpositivism is linked to purposes that are deductive and objective, which are matched by survey research and experimental designs, which typically produce numerical data. All of the four proposed criteria are associated with each other, and the combination of these criteria will be highly successful in predicting whether the members of a field categorize any given study as QUAL or QUANT research.

Alternatives to the Essentialist Approach to Categorization

Once we recognize the problems with using essentialist logic as the basis for the indistinguishability thesis, it is not hard to find an alternative approach to classification that avoids these issues. In particular, the principal argument against the essentialist approach arises from the concept of “family resemblance,” which comes from the later work of Ludwig Wittgenstein (1953) and his interest in how we perceive a world that is made up of “things.” According to Wittgenstein (1953), concepts and systems for classifying those concepts only rarely depend on essential, specifiable characteristics. His famous example is “games” (e.g., solitaire, Frisbee, Monopoly, football, etc.), which he claims cannot be defined by any set of necessary or sufficient criteria, but instead share a set of features, where any actual game will have many, but not all of those features. Hence, we recognize the similarity among games by a generalized pattern of shared features, and not by a set of essential characteristics.

In Wittgensein’s argument, family resemblances are based on sets of attributes that family members tend to have in common. I use of the word “tend” because not all members will share every characteristic. This is crucial for getting away from the rule-based notion that there are essential, defining features for categories. Wittgenstein (1953) states that when we examine family resemblance “we see a complicated network of similarities overlapping and criss-crossing” (pp. 66-67). It is this network of similarities that resolves the two problems with the essentialist approach, as noted above. First, there are no unique rules that define the features that must be present for a category to be recognizable; instead, it is enough if any given member of a category has a clear likeness to other members of that category. Second, the resemblances do not take form of a specifiable set of required features; instead, there is a “network” of interconnected features, many of which generate a high degree of similarity.

Interestingly, Willis (2007) has used the concept of family resemblances to argue that there is no specific definition of what constitutes QUAL research. One could of course make a similar argument for QUANT research, and then extend that argument to QUAL and QUANT as two basic forms of social science research. Within such a categorization system, there would be a very broad recognition of both QUAL and QUANT as devoted to generating evidence through collecting and analyzing data, at the same time that more specific family resemblances would correspond to the types of data they emphasize, the methods they use, the purposes they pursue, and the they operate within.

Of course, one could still argue that there should be an essential set of characteristics that separate QUAL from QUANT research, in order for that distinction to be meaningful. In particular, the proponents of sharply defined categorization systems believe that this kind of separation is an absolute standard which the division between QUAL and QUANT research must meet. For those authors, the failure to generate this kind of separation between QUAL and QUANT is an important failure that must be due to the subject matter in question. Yet according to Wittgenstein (1953), the whole essentialist approach to categorization routinely fails, then there is nothing surprising about its failure in this specific case.

Following directly from Wittgenstein’s (1953) philosophical work, the study of how humans classify things has been a major topic of empirical research in the field of social cognition. This research began in the 1970s with the work of Eleanor Rosch and others (Rosch, 1978; Rosch & Mervis, 1975; for a more recent review, see Rips, Smith, & Medin, 2012). Fortis (2015) summarizes two of the key ideas that Rosch added to Wittengstein’s original concept of family resemblances as probabilistic membership in a category and prototypical members of a category.

The results of the research by Rosch and others demonstrate that users of a category have no difficulty stating the features that are most commonly shared among the members of that category, and the more of these features that are present, the higher the probability of belonging to a category. Thus, category membership is recognized through a set of overlapping features, where each feature contributes to the family resemblance. Furthermore, the users of a particular category system will see some members of a category as more typical than others, and these prototypes help define the core of the category. Hence, membership in a category will always be probabilistic, with prototypical members representing the highest probabilities.

From this perspective, the inability to classify things according to rule-based characteristics is not just a logical deficiency; rather, it is a defining aspect of human perception. Birds are the classic example. In Europe and North America, robins are considered a prototypical bird because they fly, lay eggs, sing, have feathers, and so on. In contrast, most of us have to be taught to add penguins to the bird category, and many of us would need to consult an ornithologist to understand why kiwis are included. Or we could use the example from U.S. Supreme Court Justice Potter Stewart, who famously stated that he could not definitively define pornography, but he knew it when he saw it. The same approach applies to QUAL and QUANT as categories of research. Even if we cannot create a perfect set of rules to distinguish them, we know them when we see them, based on interconnected patterns across the kinds of data, methods, purposes, and paradigms that they use.

The ability to create category systems by defining the characteristics of typical members erases the claim that there must be a set of unambiguous rules for distinguishing between QUAL and QUANT research. This probabilistic approach to category membership means that some pieces of QUAL and QUANT research can be assigned to a category with almost absolute certainty, while other cases will be less clear-cut. This is what creates the blurry boundaries between QUAL and QUANT research, as shown in the Venn diagram in Figure 1.

OPEN IN VIEWER

Figure 1.

The blurry boundaries between QUAL and QUANT research.

Although the absence of a sharp separation in Figure 1 indicates that there is a “continuum” between QUAL and QUANT research, the shape of that continuum is not a flat distribution with QUAL and QUANT evenly distributed at each point. Instead, there are strong central tendencies at either end of the continuum. In technical terms, the distribution is bimodal rather than uniform.

Overall, the repeated failure of essentialist classification systems is the general point behind Wittgenstein’s (1953) shift to family resemblances and Rosch’s introduction of probabilistic membership and prototypical members. Many of the things that matter to humans are too complex to match narrowly specifiable sets of presences and absences; instead, they arise from deeper and more subtle ways of recognizing meaningful differences. Indeed, it may be impossible for humans to build any large-scale classification system without running into problems (Bowker & Star, 1999; Gieryn, 1999). As a result, there will almost certainly be blurry boundaries between complex categories such as QUAL and QUANT research, but that does not make the underlying distinctions meaningless to those who use them.

Indistinguishability and the Different Strengths of Different Methods

If we were to accept the indistinguishability of QUAL and QUANT methods and the purposes they serve, how could we assign different strengths to those methods? It is not surprising that Sandelowski (2012, 2016) is also opposed to the notion that methods can be assessed in terms of their strengths: “The strong/weak comparison—whether explicitly deployed or implied—is yet another binary that like the qualitative/quantitative binary itself continues to bind. The terms strong and weak designate idiosyncratic judgments about modes of inquiry, not attributes of them” (Sandelowski, 2012, p. 326, italics in the original). The problem with this statement is in the assertion that judgments about the strengths are “idiosyncratic.” Instead, consensual standards about the strengths of methods are the outcome of considerable prior experience in a field, which individual researchers strive to assimilate during their training.

Consider one of the examples that Sandelowski (2012) offers:

To state that individual interviews allow more intensive exploration of individual experience than focus group interviews is to state something about the nature of these data collection techniques and thereby to signal their appropriate use; it is not to state that individual interviews are stronger in any absolute sense than focus group interviews. (p. 325)

But this is a distinction without a difference. How can we say that signaling when one method is more “appropriate” than another is anything other than a statement about the strengths of that method? Every research project requires decisions about the appropriateness of one method versus another, and the relevant strengths of each method are the basis for those decisions.

The rejection of strengths as a basis for selecting methods also appears in Sandelowski’s (2014) article on unmixing mixed methods:

Strength and weakness are not attributes of research approaches but rather judgments researchers make about them. . . . The choice of combination of elements to use must be defended as meeting the specific objectives of a study, and it is the choice researchers make that will be judged as strong or weak. (p. 4)

But once again, there must be criteria for such judgments. Without consensual standards within a field, we cannot say when the use of a method was strong or weak, appropriate or inappropriate, and so on.

As a counterexample to the claim that it is only researchers’ judgments that matter, consider a common situation from MMR practice: A researcher wants to develop the content for a survey, and uses a preliminary set of focus groups to accomplish that goal. Twenty-five years ago, this design would have been unusual enough that it might have required explicit justification. Today, the strengths of focus groups for this purpose are so widely accepted that it would be rare to offer an explicit justification for using this design. This is not to say that focus groups only have strengths for exploratory research, or that focus groups are the only method that has strengths for exploratory work. The availability of further options in no way diminishes the recognition that focus groups offer broadly agreed on strengths for designs that include exploratory research.

The phrases “widely accepted” and “broadly agreed on” that I used above also imply blurry boundaries, but this does not mean that those boundaries are nonexistent. Just as we rely on family resemblances to distinguish between QUAL and QUANT research, we recognize that our standards are both reasonably well-defined and context-dependent. Consequently, researchers are often advised to make their decisions about research design by anticipating the standards peer reviewers will use—which emphasizes the importance relying on shared assumptions whenever that is possible.

Indistinguishability and the Ability to Compare Research Designs

One of the most important outcomes of relying on the different strengths that QUAL and QUANT methods can offer is the ability to make comparisons between research designs used in MMR. Over the past decades, the field has gone to great lengths to generate systematic typologies of research designs in terms of the purposes that each design serves (e.g., Creswell & Plano Clark, 2011; Johnson & Onwuegbuzie, 2004; Leech & Onwuegbuzie, 2009; Morgan, 2013; Morse & Niehaus, 2009; Nastasi, Hitchcock, & Brown, 2010; Teddlie & Tashakkori, 2009). As the number of textbooks in this list indicates, the idea that different methods have different strengths is a central element of how newcomers are introduced to our field.

It is a sign of the maturity of our field that we do not have to make all these connections explicit when we use designs that are “widely accepted” and “well understood.” As an example, when we speak about designs that can be summarized as QUAL → QUANT, most experienced MMR researchers will immediately recognize this as a class of designs that includes using the strengths of QUAL methods for exploratory purposes to contribute to the design of QUANT studies. Of course, this matches the earlier, more specific example of using focus groups to generate content for a survey as one such design.

The example above illustrates how a general understanding of research designs can provide straightforward connections to detailed choices about research methods. More technically, the criteria for the QUAL portion of the project need to serve the purposes of the QUANT study, rather than for QUAL research more generally. For example, assessing the sampling decisions for the QUAL portion depends on their ability to meet the needs of the core QUANT component. This connectivity points to the ability to distinguish between the strengths of QUAL and QUANT methods at the level of research design, while also demonstrating the importance of those strengths for a variety of more specific applications. Thus, the current emphasis on research design as the location for considering the differences between QUAL and QUANT methods has implications throughout the research process.

Of course, there are dissenters who argue that the tendency to generate typologies of research designs may stifle creativity in our thinking about how to combine methods (e.g., Guest, 2013; Maxwell et al., 2015). Although that concern certainly has merit, the value of developing well-understood designs should be to provide researchers with a starting point for their thinking, and not to create a cookie-cutter mentality where one’s work has to fit within some predetermined set of options. From this point of view, the argument that different methods have different strengths serves as a fundamental basis for understanding the consensus that we already have about the prominence of many research designs and the purposes that they serve. The bottom line is that MMR as a field depends on our ability to distinguish QUAL and QUANT methods by matching different strengths to different purposes in the design of mixed methods studies.