The Problem of “Cameo Appearances” in Mixed-methods Research: Implications for Twenty-first-century Ethnography

Introduction

Studies that incorporate ethnographic and quantitative data have risen in prevalence and prominence over the last two decades, so much so that “Mixed-Methods Research” (MMR) has apparently become a “thing” (Hesse-Biber 2015). The reification of MMR as a distinctive form of twenty-first-century social science can be readily observed by the proliferation of special journal issues and symposia dedicated to questions of quantitative-qualitative integration, the emergence of dedicated MMR journals, and a growing literature that grapples with how to define, classify, and conduct MMR (e.g., Axinn and Pearce 2006; Franzosi 2012; Small 2011; Tashakkori and Teddlie 2003a). These formalization processes have transformed MMR from an eclectic set of research approaches premised on the “synergy” of using multiple methodologies to study the social world into a methodology unto itself and a concept sufficiently bounded and well-recognized to serve as a keyword (Hesse-Biber 2015). While the mixed-methods turn may hold promise for advancing research in a variety of fields, recent controversies about the validity of social inequality research that blends qualitative and quantitative data and methods signal the potential perils, not only of research formally practiced as MMR but also of research practices steeped less formally in the mixed-methods zeitgeist.

Of what relevance is the mixed-methods turn for scholars who conduct ethnographic research that does not blend quantitative and qualitative data and/or methods? Relatedly, of what concern are emerging critiques of MMR for ethnographic “purists” and researchers who intend to remain purely qualitative in their approach and analyses?

We argue that cameo appearances made either by quantitative analyses in ethnography or by ethnographic accounts in quantitative studies should concern not only avowedly mixed-methods scholars but should also animate the concerns of qualitative and quantitative scholars alike. While the aim of generalizability is not always explicit in undertaking ethnographic research, quantitative data have been used by ethnographers to legitimate a study as “scientific” (Small 2009, 2011). Simultaneously, ethnographic research may be summoned by quantitative scholars to provide interpretive lenses for statistical analyses (Sykes, Verma, and Hancock 2017), and “a number of people have maintained that the best way to integrate ethnographic and quantitative research is to use ethnography to generate hypotheses that could be tested with quantitative data” (Wilson and Chaddha 2009:550).

When ethnography makes its “cameo appearance” in quantitative studies, ethnographic findings from a single or small number of cases may be used to add narrative or illustrative flair to instantiate statistically derived findings from entirely distinct populations, or to develop hypotheses about broader populations to be quantitatively tested. Moreover, ethnographic findings may be summoned onto stage with or without the participation of the ethnographer and whether or not the qualitative inferences were intended to perform roles according to the ontological and epistemological premises of quantitative knowledge production. Conversely, in the social inequality field, ethnographic studies increasingly draw on nationally (or locally) representative quantitative data to stage the broader context for research, summoning a “cameo” performance by quantitative research to bolster generalizability assertions. “Cameo appearances” raise equally important, if often overlooked, methodological tensions in mixed-methods studies, as well as in what may present as more or less self-contained ethnographies.

Qualitative and quantitative researchers are implicated in the process of cameo research, we argue, because either form of “cameo appearance”—whether by ethnographic research in quantitative studies, or by quantitative research in ethnography—amounts to a form of methodological tokenism under the guise of “pluralism,” the pure aim of which is to transcend rather than introduce new methodological masks for “tribalism” and the perpetuation of rigid research boundaries (Lamont and Swidler 2014:153; see also Lamont and White 2009).

MMR proceeds largely on the philosophical basis of pragmatism, which, as described by Abbas Tashakkori and Charles Teddlie (2003a), provides a framework for deploying a range of methods based on the research question at hand and getting beyond “paradigm wars” (see also Onwuegbuzie, Johnson, and Collins 2009). The classic pragmatist paradigm entailed an experimentally fallibilist methodological orientation that privileged close, detailed observation of everyday experience and focused on the use-value of research for moral and political purposes (e.g., Addams [1902] 2002; Dewey 1938; Peirce [1992] 1999; see also Tavory and Timmermans 2013). As extended to MMR, pragmatism serves as the contemporary rationale for selecting a set of methods deemed most likely to yield useful findings.

The rise of public sociology, which encourages active engagement in policy debates, particularly on systems of inequality (see, for example, Burawoy 2005), has also generated a common assumption that the deployment of multiple methods enhances the validity of research findings (Greene 2007; O’Cathain, Murphy, and Nicholl 2007; O’Cathain and Thomas 2006; Twinn 2003). Increasingly, findings from ethnographic and mixed-methods social inequality research may be understood as explanations sufficient to inform policy interventions—whether or not they are offered as such, and regardless of their inferential bases and particular claims of scientific validity and generalizability. In a remarkable moment in the twenty-first century when all manner of previously unrealistic, even unthinkable, policy reforms hold newfound potential with significant consequences for the production and reproduction of inequality, transparency about the extent of bias that may exist in findings derived from all modes of research, whether quantitative, ethnographic, or mixed-methods, is crucial for social policy informed by and crafted on the basis of research—regardless of whether that scientific basis may be rhetorical or real (see Jackman 2017).

The implications extend even to ethnographers who reject generalizability in the statistical sense and/or broader impacts on policy as research goals, not only because ethnographic findings may perform as unintended tokens by way of cameo appearance in separate quantitative research but because apparently distinct and discrete notions of generalizability within the field of ethnography itself may be shifting in the process and as a result of the so-called pragmatic mixed-methods turn. In the practice of ethnographic research, a theoretical standpoint on generalizability may conflict with the practical implications of that standpoint, both for the formulation of policy and in assessments and review for scholarly publication and other forms of public dissemination. Few, if any, ethnographers enjoy the privilege of avoiding the need to imagine how they would explain their ethnographic findings to a group of respondents or reviewers who are mainly quantitative researchers. More often, ethnographers must attempt such explanations as a fact of the peer-review process in academic publishing and in the pursuit of research funds, wherein they can expect quantitative scholars to be in the position of reviewing their work (see Small 2009). ¹

Approaches to sampling and case selection across methods have consequences for how translation, integration, and scientific commensurability occur—or fail to occur—between ethnographic and quantitative epistemologies. By examining sampling and case selection approaches, in theory and in practice, we reveal fault lines in the larger and recurring problematics of generalizability and validity. Our provocation is that, in encounters (chosen or unchosen) between ethnography and quantitative research, which often provoke contested claims of generalizability, qualitative researchers have more to gain from a mixed-methods dialogue than they may fear losing through methodological assimilation. The pressure for qualitative research to assimilate to quantitative conceptions of knowledge and conventions of knowledge production is a historical fact in the field of social science. New, however, is that qualitative research has become more mainstream (Lamont and Swidler 2014) and thus increasingly shapes social scientific conventions to which quantitative studies must also strain to adapt, often through the use of ethnographic “cameo appearances.” We proceed from the premise that a common research goal unites the field of ethnography, which is to reveal what the world under analysis tells us about the world at large, or at least about larger worlds. Sampling concepts and procedures used in quantitative analyses offer potential tools in service of this core ethnographic objective and stand to enhance rather than assimilate theory, representation, and notions of generalizability underpinning qualitative research.

In this paper, the coauthors, each with distinct methodological orientations—Hancock, an ethnographer; Sykes, a demographer and quantitative sociologist; and Verma, who uses qualitative and quantitative methods in interdisciplinary research—draw on their experiences grappling with the practical consequences of these tensions for research design and implementation. The authors assert that ethnography should not be content with making mere “cameo appearances” in purely quantitative or so-called MMR, nor should ethnography deploy cameo appearances of quantitative data to buttress claims of generalizability. Sampling, as we show, is an underlying explanation and therefore a promising source of solutions in both ethnographic and purely qualitative research, as well as in MMR approaches.

Gary Alan Fine and Black Hawk Hancock (2017) document four major developments in ethnography over the last 15 years, each of which has become salient for understanding ethnography as it is now practiced (see also Sánchez-Jankowski 2017): (1) the role of technology in conceptualizing, capturing, and visualizing data ² ; (2) embodiment and use of the body as an instrument of interpretation; (3) the growth of “urban ethnography”; and (4) new models of public sociology (see also Hancock 2016). To these four developments, we add a fifth—the availability of quantitative data and the possibility of conducting mixed-methods ethnographic research. While the implications of this new development also extend to quantitative research and a variety of mixed-methods approaches, here we focus on the stakes for ethnography in particular.

Thus, the new work of the ethnographer now entails thinking through quantitative demands on research, which helps to crystallize a thematic set of problems in ethnography anew: translation, integration, and commensurability. We first describe the tensions that have arisen in each field with respect to notions of representation and validity, reproducibility and replication, and generalizability and theory in social science research. We then articulate sampling as an explanation for dilemmas and controversies within each of the fields of ethnography and MMR. Sampling explanations lead the authors to contend, first, that MMR is problematic without qualitative-quantitative integration at the research design stage—that is, that MMR studies must push well beyond the “cameo appearance” approach if they are to overcome mere methodological tokenism—and, second, that what we conceive as design-level integration cannot occur without sampling specification and alignment across quantitative and qualitative methods in a given study. As a first-order solution, we reference several potential innovations that may be deployed in ethnographic research to specify sampling and to align sampling, when qualitative or ethnographic methods are blended with quantitative methods, thus helping to resolve (or, at the very least, more properly translate between) otherwise incommensurate notions of generalizability across methods. We conclude with longer-term solutions, by framing the possibilities and provocations of a mixed-methods dialogue for twenty-first-century research, curriculum development, and training, including for purely qualitative scholarship and within the particular field of ethnography.

The Challenges of Translation, Integration, and Commensurability in Ethnographic and MMR

Controversies around translation, integration, and commensurability across methods and disciplines have taken on particular urgency among ethnographers and quantitative researchers with explicit commitments to methodological “pluralism” (Lamont and Swidler 2014:153; see also Lamont and White 2009). Purely qualitative and ethnographic researchers, however, are implicated no less. In these debates, the core issues of translation, integration, and commensurability have often remained implicit. We are not the first to recognize that it is not just the terminology used by qualitative and quantitative researchers to evaluate scholarship that differs, but often their epistemological and ontological premises of the social world are also implicated in these debates. Methodological translation refers to common and reliable scientific understandings that retain their integrity despite local contingencies and across domains of time and space (Star and Griesemer 1989). The surface-level problem of translation reflects the underlying substantive problems of scientific commensurability and integration across methods. Scientific commensurability entails “the expression or measurement of characteristics normally represented by different units according to a common metric” (Espeland and Stevens 1998:315). The craft of integration, then, is in rendering distinct methodological modes of inquiry into scientifically commensurate forms of knowledge production.

Like Andrew Abbott’s (2004:3) framing of science as a “conversation” between methods, Mario Luis Small (2009:10) describes scientific methods as “languages” in “that they constitute systems of thought, with terms and ways of framing problems that are specific to their systems.” The problem ethnographers increasingly face in the field of social inequality research, according to Small (2009:8, emphasis added), is that ethnographic studies may be reviewed and evaluated by “scholars who are experts on the subject matter without necessarily being experts on the method.” Small (2009:10) argues that, rather than “imitating” the language of classical statistics, for ethnographers, “solutions should involve developing alternative languages and clarifying their separate objectives.”

The challenge of translation occurs not only between the ethnographer and the quantitative methodologist who may be reviewing his or her work; increasingly, an even deeper kind of translation must occur within a single research design that blends quantitative and ethnographic methods:

Probably the most important skill for the mixed methods researcher today will be the ability to write and think across not only methodological techniques but also epistemological perspectives . . . Translation involves not merely defining such terms and their significance but also the more difficult task of communicating convincingly core assumptions about quality, reliability, validity and the aims of research inquiry. (Small 2011:79)

The challenge of translation is thus symptomatic of the underlying challenge of integration in MMR designs (Bazeley 2009; Bazeley and Kemp 2012). The multiple ways in which MMR designs have been classified in the literature speak to the various levels of integration that have actually (or conceptually) been achieved (Axinn and Pearce 2006). For example, integration has been defined in MMR literature according to the stage of research at which qualitative and quantitative data and modes of analysis come together (in initial sampling decisions, data collection, and/or in the later stages of data analysis). Alternatively, MMR integration can be described in terms of whether qualitative and quantitative methods are deployed “sequentially” or “concurrently,” as well as whether MMR research designs are “nested” or “non-nested”—that is, whether multiple data types are collected from the same individual or case, allowing for within-subject confirmation (Lieberman 2005). A final form of MMR integration entails whether an individual or case is studied from multiple methodological standpoints (Small 2011).

There remains much disagreement among mixed-methods researchers not only about how best to classify these various forms of MMR but more essentially which kinds of designs should qualify as truly integrated MMR (e.g., Guest 2012; Hall and Howard 2008; Leech and Onwuegbuzie 2009). For example, despite the common understanding of “triangulation” in qualitative research as the measurement or observation of a given individual or case from multiple locations or standpoints as a way of validating findings (May and Pattillo-McCoy 2000), “triangulation” in MMR may be akin to “non-nested” designs, which represent a more limited form of quantitative-qualitative integration than “nested” designs (e.g., Denzin 2012; Mertens and Hesse-Biber 2012; Tashakkori and Teddlie 2003b).

The twin quests for integration and translation—not only across methods but within MMR designs—are, in turn, manifestations of the core question of commensurability (Small 2011). How can quantitative and qualitative methods be truly integrated within a single research design without attending to and somehow satisfying their distinct epistemological and ontological premises? Quantitative research, often using large-n datasets based on randomly selected samples so that causal inferences and associations can be drawn and generalized to a larger population, has been described as proceeding on a “positivist” or “postpositivist” paradigm, wherein “truth” is understood as having an objective, independent, and knowable existence. Qualitative research, on the contrary, can be described as proceeding on a “constructivist,” “interpretivist,” or “hermeneutic” paradigm, wherein the existence of an objective, knowable, fixed “truth” is questioned and viewed as dependent on situated, subjective interpretations (Lincoln and Guba 2005; Hancock and Lorr 2013; Haraway 1988; May and Pattillo-McCoy 2000; Smith and Heshusius 1986).

While probability-based sampling is predominant in quantitative research, nonprobabilistic purposive and convenience sampling approaches are most often used in qualitative research (Lucas 2016), which tends to emphasize the depth (rather than breadth) of information that can be gleaned from each selected unit. Moreover, nonprobability observations enable qualitative researchers to examine “outlier” cases with greater detail because such cases are extreme, deviant, or disconfirming of theory or findings (Burawoy 1979; see also Axinn and Pearce 2006; Teddlie and Yu 2007). Deviant case analysis assumes that the outlier case can be investigated in relation to nondeviant or normative cases. However, when cases are nonrandomly selected, identifying the location of a “deviant” case in a distribution of cases sampled nonprobabilistically makes little sense without having a reference point of the normative case within what Loïc Wacquant (2002:1479) calls “the broader galaxy” of potential cases. A random sample is first required to construct the distribution of cases from which the deviant case is then identified and known to, in fact, deviate from the norm of cases.

Despite the distinct logics undergirding qualitative and quantitative sampling schemes, contemporary ethnographic studies published in mainstream U.S. sociological journals commonly present findings in terms of causation and generalizability (see Abend, Petre, and Sauder 2013). In response to ensuing critiques that ethnography has adopted little more than the patina of causality, and in recognition of what seems to be an epistemic paradigm that demands publishable ethnographic research to yield causal, generalizable claims, a dawning methodological literature has attempted to articulate a kind of causation (and associated causal logics and languages) tailored specifically to ethnography (e.g., Katz 2001, 2002, 2015; Luker 2008; Small 2009, 2013; Sørenson 1998; Tavory and Timmermans 2013). As in the MMR field, pragmatism provides the philosophical basis for this alternative conception of causality in the ethnographic research field. The pragmatic logic of abduction is put forth as an independent basis for scientific discovery, in which knowledge production proceeds by pursuing the best available explanation for the time being and following chains of reasoning to continually seek to improve current explanations (e.g., Tavory and Timmermans 2013). Abductive reasoning leads to a particular conception of theory as explaining processes and mechanisms that relate various sets of conditions (e.g., Sørenson 1998) rather than the positivistic conception of theory as identifying and isolating discrete causes of particular outcomes (e.g., Popper 1963). The implication is that knowledge production and theory building proceed pragmatically, by following chains of reasoning to continually improve the “good enough” explanation, rather than through falsification.

However, pragmatic conceptions of causality may be limited in their ability to arbitrate the divergent premises of quantitative and qualitative methodologies. As a result, qualitative-quantitative integration without attending to divergent epistemologies may introduce bias in MMR research findings, as well as when ethnography or quantitative research makes its “cameo” appearance in less formalistic blendings of methods and data. We examine the manifestation of these biases through three vectors of controversy in ongoing ethnographic and MMR: representation and validity, reproducibility and replication, and generalizability and theory.

New Work for the Twenty-first-century Ethnographer: Engaging Generalizability through Sampling

Ethnographers need not cede ground on generalizability. They can go beyond theoretical abduction and approach generalizability in the statistically representative sense by integrating basic sampling premises and procedures. This ability to make generalizable claims, we contend, is the value-added of engaging mixed-methods approaches in ethnography—in other words, to optimize breadth as well as depth (Sykes et al. 2017).

Ethnography is subject to recurring critiques over case selection, which implicate representativeness, generalizability, and ultimately validity. For example, Wacquant (2002) advances a particular critique of a number of ethnographic accounts, challenging ethnographic validity and conclusions on the grounds that case selection, observations, and interviews within cases did not proceed according to standard research methods and thus failed to adequately connect data with theory. As iterative as this process may be for ethnographers, Wacquant is not the only scholar to argue that a prior conceptualization of the research question(s) at hand, and the possible empirical and theoretical landscapes within which the objects of inquiry could be situated, remains a first-order task in the design of ethnographic investigations (see, for example, Desmond 2014). Here, the problem is that some ethnographies begin the research process, including case selection and data collection, based on interest, curiosity, and/or convenience. Wacquant (2002:1481) characterizes these problematics as “happening upon” field sites, “fishing for questions,” and the dangers of researcher “projection” of their personal interests.

However, such critiques are by no means unique to ethnography. Even purely quantitative studies can often only approximate unknown social distributions in the real world. Critiques such as Wacquant’s (2002:1479) take issue not with the ethnographer’s inability to adequately approximate a context she or he is often only just discovering but rather that ethnography rarely seems to consider “the broader galaxy” from which cases might be selected. Quantitative research uses probability-based sampling to avoid such pitfalls. Ethnography can utilize such sampling concepts and procedures as solutions to these common problematics of case selection (see Lucas 2016; Sykes et al. 2017).

For example, Wacquant (2002) draws attention to how quantitative data could have been leveraged in several ethnographic studies, which could be taken as an implicit argument for MMR. Indeed, engaging multiple methods, we argue, is emerging as a distinct practice of ethnographic work in the twenty-first century. At the same time, Sykes et al. (2017) illustrate the dangers of MMR when sampling is not aligned across ethnographic and quantitative research strands within a single study, which represents an increasingly prevalent form of “cameo appearance.” Even the most rigorous empirical assessments, whether qualitative or quantitative, must contend with the inherent assumptions upon which their particular methods depend. A consequence of failing to recognize the limits of particular methods can lead to incomplete and invalid conclusions as if they were accurate, scientific representations of the social world.

The field of MMR has already begun to engage sampling problematics. Charles Teddlie and Fen Yu (2007:91–92) observe that, “[i]n particular, very few articles that we analyzed included a concurrent MM design with an explicit discussion of both the purposive and probability sampling techniques used to generate it.” In their “Mixed Methods Sampling: A Typology with Examples,” Teddlie and Yu (2007:97) go on to emphasize,

Researchers should be sure to follow the assumptions of the probability and purposive sampling techniques that they are using. In several of the MM studies that we analyzed, the researcher started out with established probability and purposive techniques but violated the assumptions of one or the other during the course of the study . . .

They offer another guideline as well:

The sampling strategy should allow the research team to transfer or generalize the conclusions of their study to other individuals, groups, contexts, and so forth, if that is a purpose of the MM research . . . It should be noted that not all MM studies are intended to be transferred or generalized.

Teddlie and Yu’s (2007:98) final, if obvious, guideline is that “researchers should describe their sampling strategy in enough detail so that other investigators can understand what they actually did . . .” Following Teddlie and Yu’s cue, we draw attention to one source of the problem: the lack of social science doctoral training in blending qualitative and quantitative research, which points to the need for curriculum development and requirements for all PhD students in MMR design. Many social science graduate students are required to take at least one foundational statistics course, which conveys the basic principles of probability-based sampling. However, students who go on to use qualitative and ethnographic methods may never return to the statistical logics of sampling in constructing future research designs. At the same time, summoning cameo appearances by quantitative data and analyses in ethnographic work is increasingly common in research designs that attempt, perhaps unknowingly, to assimilate statistical logics that seemingly afford them generalizable claims. It would seem, then, that courses are necessary for both qualitative and quantitatively oriented students that make explicit the differences in sampling and case selection premises across multiple methods, while training them in how to align sampling in MMR.

Training the Twenty-first-century Ethnographer: Establishing a Mixed-methods Curriculum in the Social Sciences

The promises and perils of twenty-first-century ethnography require structural changes to how graduate students are trained in PhD programs. We constructed a list of the top 20 sociology programs from the 2017 U.S. News and World Report Best Graduate Schools rankings and coded whether each program required a quantitative, qualitative, and mixed-methods course for advancement to candidacy based on information listed on each department’s Web site. ⁷ We also coded whether the department offered a formal course or workshop on MMR as a possible elective for graduate students. Although mixed-methods designs may be covered in subsections of required general research methods courses, we exclude general courses from our analysis because we were interested in identifying classes dedicated specifically to mixed-methods training.

Table 1 presents the percentage of top 20 sociology programs that require at least one graduate course in quantitative (probability, elementary statistics, linear models, etc.) and qualitative (ethnography, historical/archival, interview, etc.) methods. ⁸ The results are striking. All departments, at both public and private universities, require PhD students to be trained in quantitative methods, but only a fraction of the same departments require students to be trained in qualitative methods. Notably, only one in five programs requires graduate student coursework in qualitative methods. These asymmetrical requirements in PhD curricula have the capacity to leave quantitative researchers untrained in methods that they may not only be in the position of evaluating as peer reviewers (see Small 2009:5) but may also employ later in their careers, which may lead them to be less thoughtful about summoning ethnographic “cameo appearances” to explain statistical findings. Similarly, qualitative scholars who elide their quantitative training risk recreating the same research dilemmas and problematics that lead to recurring debates about validity, representation, and generalizability, particularly when they summon quantitative “cameo appearances” to contextualize field sites without systematically engaging the distributional properties and characteristics of their selected cases (Sykes et al. 2017).

OPEN IN VIEWER

Table 1.

The Percentage of Top 20 Sociology Doctoral Programs that Require at Least One Graduate Course in Quantitative and Qualitative Methods, or that Offer a Mixed-methods Course as an Elective, by University Type in 2017.

	Required coursework			Possible elective
University Type	Quantitative	Qualitative	Mixed-methods	Mixed-methods
Private (n = 10)	100%	20.0%	0.0%	10.0%
Public (n = 11)	100%	18.2%	0.0%	0.0%
Total (n = 21)	100%	19.0%	0.0%	4.8%

Source. Authors’ calculations of curriculum requirements.

Note. The top 20 sociology departments included in this analysis are based on the 2017 U.S. News and World Report Best Graduate Schools ranking. Five departments were tied for #17, resulting in a sample size of 21 instead of 20. See Table A1 of the appendix for a list of all departments included in the analysis.

Our solution for resolving a number of ethnographic controversies and preventing future cameo appearances in and by both quantitative and qualitative research is to increase training in MMR for all social science PhD students, regardless of whether they intend to conduct avowedly mixed-methods studies in their future careers and scholarship. Currently, no program in our analysis requires mixed-methods coursework, and only one program offers a mixed-methods course or workshop to its graduate students as an elective. It is worth noting that the doctoral program that offers mixed methods as an elective also requires coursework in both quantitative and qualitative methods. Formal training in these three distinct epistemological approaches, we argue, is critical for articulating and explicating the relative embedded assumptions of each methodological orientation and for resolving discordant or divergent findings when multiple methods are “mixed” together.

The proposed curriculum for MMR must engage how to align sampling in studies that blend multiple methods, particularly when cases have been selected probabilistically and nonprobabilistically. Scholars have effectively utilized many viable options for designing research that fully integrates quantitative and qualitative sampling and analytic methods in MMR (see, for example, Axinn and Pearce 2006; Pearce 2002; Sánchez-Jankowski 1992; Small, Jacobs, and Massengill 2008; see Sykes et al. 2017 for extended discussion of these approaches). Our call to broaden the methodological training of all graduate students is consistent with scholars who have identified curricular limitations in qualitative and mixed-methods training (see Mahoney 2004; Pager and Quillian 2005). As top sociological journal editors increasingly publish MMR (Jacobs 2005), the practical implications for the careers of future sociologists may partly depend on being well-trained in how to mix multiple methods.

Conclusion

Is the discipline “social” or “science”? Or, under what conditions can it be both? Sociology seems to walk a tightrope that attempts to balance social observations of the world with methods that subject the social world itself to the scientific gaze. Pierre Bourdieu (1990:25) argues that, regardless of which position one takes, most sociological work fails to break away from the dichotomy between subjectivity and objectivity as the “theoretical modes of knowledge.” Yet, the blending of subjective observations and objective categorizations matters for how we have come to practice ethnography in its mixed-methods form by the twenty-first century. Similarly, the integration of distinct epistemologies to make sense of the social world has inured quantitative researchers to contextualize their “objective” statistical findings with “subjective” ethnographic accounts in ways that may be invalid.

The emergence of MMR in the social scientific enterprise calls for more than a first-order Bourdieuian “reflexivity”; a deeper mode of introspection about translation, integration, commensurability, and ultimately the validity of methods used in scientific investigation requires Kantian (Kant 1998:367) transcendental reflection, or reflection on reflection,

the action through which [a researcher] makes the comparison of representations in general with the cognitive power in which they are situated, and through which [the researcher] distinguishes whether [those representations] are to be compared to one another, [thusly] belonging to the pure understanding or to pure intuition. ⁹

Transcendental reflection, then, is the second-order activity of thinking about what is conceptually possible methodologically, including the limits of existing epistemologies and the boundaries they place on the genesis of knowledge (see, for example, Goldman 2012). This mode of reflection encompasses inquiries into the theoretical, practical, and empirical limits of discovering knowledge about the world, and, consequently, how research design decisions at the outset limit or enhance scientific knowledge production and eventual claims to scientific expertise. The dialectics of reflection provide a Kantian method for critically interrogating the contingencies of categories of thought as the foundations upon which all knowledge is attained, as well as how existing categories are products of methodological judgment, which may be limited in capacity and, therefore, biased in application (Longuenesse 1998). Ethical social science depends not only on Institutional Review Board (IRB) approval but more foundationally on transcendental reflection, which Kant articulates as “a duty which no one can neglect who wishes to establish an a priori judgment upon things” (Kant 1998:368).

Quantitative and qualitative researchers, given their distinct intellectual trajectories and specialized training, start with very different presuppositions in their commitments to creating knowledge in the social sciences. These presuppositions, both epistemological and ontological—in terms of what is identified; how what is identified is conceptualized; and in their attendant levels of systematicity, translation, validity, and generalizability—need to be reflected upon transcendentally if social science is to establish a more robust curriculum and deeper training in MMR for twenty-first-century ethnography. Researcher “reflexivity” and “cameo appearances”—in both quantitative and qualitative scholarship—are insufficient scientific approaches for achieving the potential of MMR to balance “a conversation between rigor and imagination” (Abbott 2004:3).

To have a science of what people actually do and how they understand and act in the world, we must take account of how those actions occur within the external structures that, although they may not be formally conceptualized, nevertheless shape and mold social life. Only by making explicit this gap between the subjective knowledge that actors have of the world and the objective knowledge gained through structural analysis, can we begin to construct a reflexive theory of society that combines the objective and the subjective in the construction of knowledge without falling into a false dualism. Small (2009:28) uses the metaphor of building “boats that try to fly” rather than building “boats that sail effectively.” MMR offers a way out of this dualism, and even the possibility for building research vehicles that can both fly and sail, but only if the blueprints fully integrate and align sampling and case logics across qualitative and quantitative methods. If we are to build mixed-methods vehicles that effectively sail greater distances while ascending to new heights, the engines that propel this multiple method vehicle require curricula and training for future social scientists that transcend the performance of “cameo appearances” to stage pseudo-replicas of validity and generalizability.