Experimenting With Qualitative Inquiry

Introduction

This article arises out of and seeks to explore the continuing “stand-off” between qualitative inquiry and experimental design. Rather than simply rehearsing the issues, however, the article looks to think about the issues in a different way. The article briefly reviews the debate and the case for designing experiments in social research (specifically randomized control trials [RCTs]). The article argues that RCTs only comprise a small element of much broader scientific inquiries, but it also further argues that qualitative research itself has a substantial history of undertaking experimental interventions. The article goes on to suggest that if we conceptualize the idea of an experiment differently, then thinking about different forms of intervention in social research could generate different forms of knowledge about what “might work” to change social formations. The argument made in the article relates most clearly to applied qualitative research, although it also carries implications for the ways in which qualitative inquiry is conceptualized more generally.

Qualitative Inquiry and Scientific Research

Qualitative inquiry, and qualitative approaches to social research more generally, has been under severe pressure from the “what works” science-based research and policy movement for 15 years and more. Qualitative research is positioned, at best, as an adjunct to experimental research, specifically RCTs, as a “pre-scientific” stage of exploratory investigation and/or additional evidence gathering conducted before or alongside an experimental design to help interpret the experimental results (Cook & Payne, 2002; Shavelson, Phillips, Towne, & Feuer, 2003). At worst, qualitative research is dismissed as of little value and not worth pursuing or funding (No Child Left Behind, 2002). The impact of such criticisms has perhaps been most acute in educational research, and in the United States, where a recent review talked in terms of a “major earthquake” hitting educational research (Walters, Lareau, & Ranis, 2009, p. 1). However similar arguments have been made in the United Kingdom and elsewhere (Goldacre, 2013; Hargreaves, 1996; Yates, 2004). Many responses and rejoinders have been published, both in what we might call a “first wave” of debate (e.g., special issues of Educational Researcher, 2002; Qualitative Inquiry, 2004; Teachers College Record, 2005) and more recently as the arguments have been re-visited and re-stated with renewed vigor in the United States and the United Kingdom (Goldacre, 2013; Haynes, Service, Goldacre, & Torgerson, 2012; Walters et al., 2009). Recent further “second wave” responses have sought to

re-state some of the philosophical issues in identifying cause and effect and establish actionable knowledge in human affairs (Maxwell, 2012; Morgan, 2014);

understand some of the pressures that policymakers are under when trying to collect and evaluate evidence (Donmoyer, 2012);

make visible some of the problems of publishing qualitative research in these new times (Ceglowski, Bacigalupa, & Peck, 2011);

and reassert the complexities of research and the distinctiveness and social justice aspirations of qualitative research, (Denzin, 2010; Lather, 2010)

An additional response has been to engage with and argue the case for more mixed-methods approaches to social and educational research, specifically linking qualitative methods with RCTs in mixed-method research designs (Christ, 2014; Hesse-Biber, 2012). The arguments for mixed-methods approaches are many and varied, and in many respects, they have a long and fairly unremarkable history in social research (Bryman, 1988; Denscombe, 2008). The logic of choosing different research methods to address different research problems is well established, although mixing methods raises many problems of commensurability and integration. The issue with current developments in mixed methods, however, is that they tend to position qualitative methods as subservient to experimental design (Hesse-Biber, 2010) in much the same way as what we might term mainstream experimentalism—an adjunct to RCTs rather than a fully integrated contributory element.

Thus, we seem to have two research communities largely talking past each other for the best part of 20 years with little constructive engagement over legitimate criticisms or potentially common concerns. Proponents of experimental design are clearly in the ascendancy however and, as such, perhaps do not feel the need to engage. A key issue for this article, however, is why definitions of science and scientific research are so closely identified with RCTs, and indeed are taken to be synonymous with RCTs. RCTs only ever form one element of a much longer and more substantial chain of investigative activity. A more general and inclusive understanding of science and experimentation must be reclaimed for qualitative inquiry. Furthermore, the article also argues that qualitative inquiry should move beyond reactive responses and notions of critique, and look forward to creating new forms of experimental research, new forms of “intervention,” which seek to explore what might be produced if research was conducted differently.

The Logic of Experimental Design

Arguments in favor of RCTs are grounded in assumptions about cause and effect and what counts as “proof” of what works in social research and social policy. An intervention is designed and then tested through random assignment to control and experimental samples. However, the intervention itself has to be produced through much prior investigation of the nature of the problem to be addressed and substantial theorization as to how intervening variables might interact to produce the desired outcome. Once designed, the intervention is then tested through random assignment to control and experimental groups. If the intervention works (and it is a big “if”), it is expected that the experimental group will show better results than the control (better early reading, better maths learning, etc.). The overall results should show positive effects on the experimental group as a whole, although this will always vary in individual cases and positive impact cannot be guaranteed for all. The effect—the “proof” that is produced—is at the level of statistical probability, not certainty, although this is not always apparent in the rhetoric of the “what works” movement. Claims are made for what does work, rather than what probably would work in particular circumstances, if properly implemented (Goldacre, 2013).

Nevertheless, RCTs can be very good at answering very specific questions and attributing cause in terms of statistical probability. What they cannot do is produce the questions in the first place: As we have noted, that depends on much prior, often qualitative, investigation, not to mention value judgments about what is significant in the qualitative data and what is the nature of the problem to be addressed by a particular program intervention. Nor can RCTs provide an explanation of why something has happened, or not happened (i.e., the underlying causal mechanisms at work). This, likewise, will depend on much prior investigation and, if possible, parallel qualitative investigation of the phenomenon under study, to inform understandings of implementation and further development of a theory about what the researchers think may be happening. This is where qualitative methods could be fully integrated into RCT designs, and indeed, they have been so in some key studies. Pawson and Tilley (1997), Grissmer, Subotnik, and Orland (2009), and Schorr and Farrow (2011), among others, argue the case for incorporating qualitative work into RCT designs and cite instances of where such work has been effectively deployed. Thus, it is apparent that RCTs, even within their own paradigmatic terms, are actually only one part of a much longer chain of necessary and integral research activities, as indicated below in Figure 1.

OPEN IN VIEWER

Figure 1.

Logic and sequencing of RCT research design.

By “successful,” in column 5, it is meant that the RCT shows positive, statistically significant results, demonstrating the efficacy of the intervention. Often however, although the experimental design has been carried out correctly, no significant differences are shown between the control and experimental groups. Educational outcomes, in particular, seem highly resistant to direct single-cause interventions, partly, at least, because they are the product of so many complex interacting variables. Interestingly, Campbell and Stanley’s (1963) classic article on “Experimental and Quasi-Experimental Designs for Research” notes regular periods of RCT advocacy and RCT disillusionment in educational research, going back to McCall’s “How to Experiment in Education,” published in 1923.

Thus, the conceptualization and practice of a “scientific experiment,” or indeed “science” more generally, involve much more than the design of an RCT. It involves an attitude of curiosity, persistence, and open investigation, pursued by specific acts of observation, categorization, correlation, theorization, and, in due course, experimentation. It also involves replication and returning to observation to better understand what happens when experiments fail—when they do not produce the predicted results. In many respects, it might even be argued that science is produced through a succession of “qualitative” engagements with the world, including pursuing hunches and occasional creative insights. Yet, as we have noted, most of this activity is not recognized in the political arguments over the quality and utility of qualitative social and educational research.

The Orientation of Qualitative Inquiry

Equally, however, qualitative inquiry can often be rather limited in its investigation of social issues and its deployment of critique—looking backward to investigate what has happened, rather than forward to explore what might happen—as if social research, in and of itself, will lead to change. Recently, Ron Pelias (2014) in a short review of how he thought qualitative inquiry “should” develop, and “speaking on behalf of the traditions we have forged” (p. 3), argued that

our intent should never be to offer the final account; instead we should give the best account we can in given circumstances. (p. 2)

This neatly makes my point for me—“the traditions we have forged” seek to render accounts, descriptions, analyses, “in given circumstances,” rather than seek to intervene to produce something new.

Indeed, extensive theoretical and methodological debate in qualitative research has focused on minimizing the impact of the researcher on the phenomenon under study. Debate has revolved around the role of the researcher as “research instrument” and whether or not, and if so how, intervention in the field could and should be minimized (Hammersley, 1999; Hammersley & Atkinson, 1983; Lee, 2000). Institutional Review Boards (IRBs—Ethics committees in the United Kingdom) often still take this view of social research (Lincoln & Tierney, 2004). The argument is that the researcher should not alter or interfere with the phenomenon under study. Recently, the U.K. government has taken this injunction even further and is introducing a new clause into all government-funded research contracts banning what it terms “lobbying.” The clause reads, “The following costs are not Eligible Expenditure: payments that support activity intended to influence or attempt to influence Parliament, government or political parties . . .” (Cabinet Office, 2016, p. 1). This is a strange development indeed in the context of debates about using public money to find out “what works.” The implication is that “science,” including social science, is producing awkward findings about “what doesn’t work,” which the government does not want to be widely disseminated.

Of course, many researchers argue that eliminating or minimizing intervention in the field and in reporting is neither feasible nor desirable (Denzin, 1989; Stake, 1995). The researcher will always generate some perturbation in the data and indeed can be said to produce data through his or her actions. Furthermore, it can be argued that not only can the impact of, for example, participant observation or the research interview on research respondents not be controlled, but could in fact also be turned to advantage as “better accounts” might be produced by respondents reviewing interview transcripts or responding to draft reports (House & Howe, 1999). However, this is a still a long way from arguing that specific experimental qualitative interventions should be designed and implemented. The intention of the mainstream tradition is still to “give the best [retrospective] account we can in given circumstances” (p. 2).

It is this orientation that, at least in part, produces qualitative inquiry’s uneasy relation to current policy. Much current policy debate in the social science community as well as among politicians and policy makers argues that research should have more impact on the topics and issues that it is investigating. Indeed, this is the context in which the evidence-based policy and practice movement have arisen. Given uncertainty over policy options, it is understandable that governments and policy makers will look to research for assistance, even if they do not always use it when it does not support their prior policy interests and prejudices. Equally, evidence-based practice also appeals to ideas of rational planning and the self-interest of some sections of the research community, privileging research knowledge above what is often characterized as traditional, ineffective professional practice. Whatever the drivers, certainly in the United Kingdom, it is the case that research funding now depends on researchers being able to demonstrate that their research has had an impact on social, cultural, and/or economic activity and that new proposed research will have an impact on such matters (Blunkett, 2000; Higher Education Funding Council for England [HEFCE], 2009, 2014). There is not space here to review this trend in more detail (see Torrance, 2014, for a longer account) but suffice to say that social research in general and qualitative social and educational research in particular have attracted considerable attention and criticism for their lack of engagement with an impact on significant policy issues.

Interestingly, however, such criticism does not simply arise from the policy community. Major debates are also apparent in the research community. A key rejoinder to policy makers can be that qualitative research does indeed engage with policy issues, but from a different and often critical perspective—looking to investigate the underlying determinants of social issues and problems rather than produce superficial policy fixes. However, a substantial body of theoretical and methodological work is now emerging within the field of qualitative inquiry itself, which challenges the taken-for-granted assumptions of much qualitative research, especially the assumptions that micro-level investigations of social processes will inevitably throw light on macro-level social products and afford insights into processes of social change. Thus, for example, much criticism has been articulated recently about the overly empiricist, optimistic, and even romantic notions of privileging and celebrating “voice,” as if interview data and disenfranchised voices can unequivocally and unambiguously “speak for themselves” and, in so doing, produce possibilities for change (A. Jackson & Mazzei, 2009, 2012). Similarly, St. Pierre (2011, 2014) laments the often desiccated product of much qualitative research, as textbooks, handbooks, and graduate courses, not to mention many qualitative researchers themselves, reproduce and pursue technical and procedural versions of what she terms “conventional humanist qualitative methodology” (St. Pierre, 2011, p. 611). By this, she means that observations and interviews are conducted and recorded, and “data” are assumed to be produced as a result. These linguistic representations of human activity are then “analysed” via approaches such as “grounded theory” and “findings” produced. St. Pierre despairs of the formulaic procedurality of such work and the trivial outcomes that are so often produced. MacLure (2015) reviews the limited impact of critique and the untenable ontology it implies and on which it depends—of critical social researchers being capable of investigating and unmasking, and therefore somehow being outside, or more critically aware of, the social processes and outcomes that actually embrace us all, and which are produced by us all. Keating (2013) argues that “binary either/or thinking . . . [and] dichotomous oppositionality locks us into the status quo and a reactionary stance.” She makes the case for what she terms the “post-oppositional politics of change” (Keating, 2014).

The essence of these criticisms seems to be that qualitative inquiry, and indeed, social research more generally, too often investigates what is and what has been, rather than what might be. It too often privileges empirical data and critique over theoretical reconceptualization, describing the nature of a problem rather than thinking through or trying to promote alternatives. Thus, successive generations of social and educational researchers, including qualitative researchers, too often discover and rediscover social issues and problems rather than explore or attempt to produce solutions. A significant illustration might be the cumulative work of researchers such as Hargreaves (1967), P. Jackson (1971), Willis (1979), McNeil (1986), McLaren (1989), Delpit (1995), and Lipman (2004), on the social organization of schooling and its impact on disadvantaged groups. It is not that the studies just cited are without merit. Quite the reverse, they are exemplars of the very best of their kind and constitute a formidable body of knowledge about the ways in which schooling reproduces social inequality, often despite the best intentions of teachers, administrators, and, sometimes, the researchers themselves as they have sought to feedback findings to promote change. However, this body of work has documented the nature of the problem in terms of empirical evidence and the production of theoretical and analytic understanding, rather than produced change.

Of course, it is also the case that change can be produced in many different ways. It is unlikely to arise directly from the impact of a single study or even series of studies. Change arises cumulatively and often in serendipitous and unanticipated ways, including perhaps from mid-career professionals reading such studies on graduate degree programs and the like, and attempting to make changes in their own professional practice. But again, my point is that it is puzzling, even striking, that such an orientation to research, often motivated by a sense of injustice and the need for social change, is not directed more specifically toward such change. “Interventions” are something that seem to belong in the RCT camp, rather than the qualitative research camp. Although proponents of RCTs certainly need to acknowledge that there is more to scientific method than designing and evaluating interventions, equally qualitative researchers might think more creatively about what experiments in educational and social research could and should involve.

Experimenting With Qualitative Research

It was not always so. Experimentation is about interrupting the taken-for-granted, doing something different, trying something out to see what happens, creating the new. It actually has a long and distinguished history in qualitative research—back through Garfinkel’s “breaching experiments” in ethnomethodology in the 1960s (Garfinkel, 1967) to the origins of action research in the 1940s (Lewin, 1946, 1947). Garfinkel (1967) was interested in how social action is accomplished and how social life is realized in action. By asking his students simply to refuse to accept the taken-for-granted premises of everyday conversation, his experiments identified those premises and the ways in which routine habits and practices operate to manifest social expectations:

Case 6

[“E” is Garfinkel’s student, experimenting with the norms of social interaction; “S” is his exasperated friend]

(S) How are you?

(E) How am I in regard to what? My health, my finances, my school work, my peace of mind, my . . . ?

(S) (Red in the face and suddenly out of control) Look! I was just trying to be polite. Frankly I don’t give a damn how you are.

(Garfinkel, 1967, p. 44)

Of course, this is a very mundane instance of disrupting (experimenting with) social interaction, but one could imagine more challenging experiments (interventions) with regard to assumptions of race and gender equality, for example. Whether Ethics Committees (IRBs) would tolerate and sanction such interventions is another matter. Challenging them to do so would be part of the experiment.

Similarly, action research might be seen as a rather limited empirical, technical, and realist approach to research and social change that need not even involve much, if any, qualitative inquiry. The action research cycle involves the following: Identify a problem or issue, gather data on the nature of the problem, design an intervention to address the problem, gather data on the effects of the intervention, and start the cycle again. Nevertheless, and however limited and incremental may be the focus of change, there is still an orientation toward change underpinned by a theory of how such change may be accomplished—that is, by continual research and reflection on the nature of the problem, rather than, as with RCTs, by trying to demonstrate that one approach is necessarily better than another. Change is seen as deriving from investigation and deliberation, not irrefutable proof. Moreover, the aspirations of some action researchers are clearly more ambitious than others (Carr & Kemmis, 1986; Elliott, 1989; Noffke & Somekh, 2009), and the development of many different forms of participatory and community-based action research provides interesting possibilities that could be better explored. As Fine, Ayala, and Zaal (2012) argue,

Public science . . . is . . . strategic research for organizing campaigns on the ground . . . [ ] . . . critical, participatory forms of inquiry, organizing and policy development. (p. 687)

Mason and Dale (2011), in an edited collection produced from a U.K. research council–funded methodological seminar series, have similarly noted that in qualitative research, the “logic of introducing a stimulus and observing the result is not particularly common” (p. 9). However, they argue that there might be a good case “to give the social world a poke and see what happens” (p. 9). Mason and Davies (2011) in the same collection, drawing on Douglas (1985), use “creative interviewing” (p. 36) with photographic prompts to encourage people to reflect on matters they might not otherwise consider relevant or important.

Other examples of change-oriented qualitative research could be cited here, including performance ethnography and a range of arts-based qualitative research whose whole raison d’etre is to try to think and do differently—to intervene and produce to challenge and provoke, through making as well as through writing (Smithbell, 2010). Such approaches stand at some distance from social science and social policy development of course. They appeal to the arts and humanities rather than social science for their ontology and epistemology. But the point is that traditions of experimentation and intervention do exist in qualitative research, they are not the sole property of RCT-based experimentalism, and we should look to them for alternative models of engaging with the demands of policy.

A New Empiricism—the New Materialism

Beyond these existing traditions, however, there are also new conceptualizations of the problem of understanding the emergence and realization of social processes, and engaging with critique in social and cultural studies. Arguments deriving from Deleuze and Guattari (1987; Deleuze, 1994) emphasize movement, flow, and the emergence or unfolding of social life. In turn, this produces the immanence of change and the constant possibility of change. Massumi (2002) notes that critique underplays the routine incrementalism of everyday life and overplays ideas of revolutionary rupture. As such critique can be very conservative, ignoring the possibilities for change that are constantly available in routine activity,

The slightness of ongoing qualitative change paled in comparison to the grandness of periodic “rupture” . . . The everyday was the place where nothing ever happens . . . [yet] . . . subtracting movement from the picture . . . catches the body in cultural freeze-frame . . . [thus] . . . passage and indeterminancy . . . constitute the field of emergence, while positionings are what emerge . . . Potential is the immanence of a thing to its still indeterminate variation . . .

(Massumi, 2002, pp. 1, 3, 8, & 9)

The implication of such insights, at least for the argument made in this article, would seem to be twofold. First, there is the analytic argument that “the new” emerges in situ, through engagement and entanglement in and with the world. The emergence of something different is always possible, we are all implicated in the process, so we all have the possibility to produce change. Second, therefore, we should make more of this possibility and our own inventiveness. Massumi (2002) further argues that

the world is self-augmenting . . . perhaps “productivism” would be better than “constructivism” because it connotes emergence . . . Critical thinking disavows its own inventiveness . . . it sees itself as uncovering something . . . [it is] basically descriptive . . . the balance has to shift to affirmative methods: techniques which embrace their own inventiveness and are not afraid to own up to the fact that they add (if so meagerly) to reality . . . (pp. 12-13)

Thus the focus of our activity should be intervention and experimentation—in the fullest sense of the word—rather than, or at least in addition to, description and analysis. Baugh (2010), summarizing Deleuze and Guattari, similarly states,

To experiment is to try new actions, methods, techniques and combinations . . . we experiment when we do not know what the result will be and have no preconceptions concerning what it should be . . . [ ] . . . Experimentation by its nature breaks free of the past and dismantles old assemblages . . . (pp. 93 & 95)

We are a long way from an RCT with these definitions of experimentation, but equally, we are a long way from the practice of much qualitative inquiry, and it may be time to try to reassert the broader definitions and goals of science and qualitative inquiry’s place within that broader definition. Massumi (2002) continues,

It is not that critique is wrong . . . It is a question of dosage . . . foster or debunk . . . It’s a strategic question . . . a question of timing and proportion . . . pragmatic. (p. 13)

There are many questions that one could put to such arguments. Not least how do they differ from or augment earlier insights from Berger and Luckman (1967) about the “social construction of reality” and indeed Deweyan pragmatism about how knowledge is produced through interaction with the world (Biesta & Burbules, 2003). One answer would probably involve criticism of the overly idealist ontology of Berger and Luckman (pace Barad, 2007); another might revolve around the emphasis on movement, action, and intervention rather than the production of knowledge per se. Overall, however, such arguments put renewed emphasis on recognizing our capacity to make something happen, and indeed making that something happen. Thus, rather than qualitative research ignoring RCTs, or criticizing RCTs, or attempting to ingratiate itself with RCTs by providing additional evidence about how things work, the argument of this article is that we should look to expand definitions of science and experimentalism and design our own experiments. Such experiments, such interventions, would further explore how things (“old assemblages”) work by disrupting them and, in so doing, also potentially produce something new, which might work differently. Such activity would raise issues of scope and scale: Would a form of qualitative inquiry directed toward experimentation seek to disturb one instance of a phenomenon, several, or many? How could the depth associated with quality inquiry be maintained across several experimental sites? To remain faithful to the richness and complexity of qualitative inquiry, such an endeavor would probably have to operate at the level of a single experiment, rather than a program or an accumulation (although Garfinkel’s deployment of his students gives us some indication of how a program might be designed). However, as has been expressed in a different context, we only need to find one black swan to demonstrate that not all swans are white (Popper, 1959), that producing something different is possible.

Norman Denzin (2014) has recently welcomed “paradigm proliferation” and argued that “Paths for fruitful dialogue between and across paradigms need to be explored” (p. 1125). I am not sure he had RCTs in mind when he called for a “bigger tent” (Denzin, 2008), but in this context, we should certainly argue for a broader definition of science and experimental design that includes experimenting with qualitative inquiry. Reasserting a wider definition of scientific method grounded in curiosity, observation, interpretation, and judgment, and of experimental work, which involves the exploration of new ideas and activities in situ, will provide us with a better basis for dialogue about the purpose and methods of research. Despite all the recent criticisms of qualitative research, it is qualitative research that is best placed to recover and advance new forms of science (and indeed policy), precisely because it rests on direct engagement with research participants. A broader vision of science and experimentalism proposes research as a system of provocative and reflective inquiry that might help practitioners and policy makers alike think more productively about the nature of the problems they face and how they might be better addressed. In turn, qualitative inquiry might evolve more oriented toward change and intervention, and toward supporting the role of research participants in producing knowledge and action for their own benefit and that of their communities.