Evidential Pluralism and Epistemic Reliability in Political Science: Deciphering Contradictions between Process Tracing Methodologies

1. Introduction

Evidential pluralism argues that since there is no single right causal evidential theory for science, researchers ought to combine evidence of different causal theories (probabilistic, mechanistic, etc.) in an attempt to triangulate and thereby support a causal claim. In social science, evidential pluralism is often applied to mixed methods research which combines quantitative and qualitative data in the same study. The combination of causal methodologies within (qualitative) case study analysis, however, has received little attention from philosophers of science. Yet process tracing, a popular case study method in amongst others political science, would benefit directly from such attention, as this article will demonstrate.

The term “process tracing” hides a variety of methodologies, each with assumptions linked to a different causal evidential theory. In this article, I show that some of these methodologies contradict one another, causing a tension for pluralists. I focus in particular on what I call epistemic incompatibility, where one methodology does not accept some of the more fundamental causal claims of the other as genuine knowledge.

In the paper, I couch this methodological disagreement in terms of the epistemic reliability of evidence for fundamental causal claims. As a working definition, epistemic reliability refers to the likelihood that a social scientist judges a piece of evidence to be true, given that the evidence is actually true (cf. Nozick 1981). Process tracing methodologists often seem to assume that the epistemic reliability of their opponents’ evidence (e.g., the epistemic reliability of counterfactuals) is always low. However, as I will argue, the epistemic reliability of evidence in process tracing research depends on the evidential context of the case under study. For example, as I will show, in some contexts we are more likely to believe a counterfatual is true, given that it actually is true. As such, opposing methodologists should not reject counterfactual evidence without considering the evidential context in which the counterfactual claim was made, i.e., by judging how likely it is in the context that we will recognize the counterfactual as true given that it is. This framework, focused on epistemic reliability, benefits philosophers working on evidential pluralism and methodologists in social science alike, because it shows that methodologies that seem incompatible at the surface are in fact contextually compatible, i.e., compatible in certain epistemic contexts. This leaves room for evidential pluralism in process tracing research.

The structure of this article is as follows. In section 1, I briefly connect the debates on evidential pluralism in the philosophical literature to similar debates in methodological research. In section 2, I explore three potential issues with evidential pluralism, focusing on epistemic incompatibility of methodologies. In section 3, I show that epistemic incompatibility causes a tension between different methodologies for process tracing, by detailing the contradictory assumptions of two such methodologies: the systems view and the interventionist view. In section 4, I argue for a more modest version of evidential pluralism which avoids the “pluralist tension.” I argue that considering epistemic reliability should play a key role in choosing the appropriate methodology for process tracing analysis. Section 5 looks at how one might expand this article’s argument beyond the political science context.

2. Evidential Pluralism

An increasingly popular argument in philosophy of causation is that there is no single right causal evidential theory for science, since all such theories have counterexamples and areas of worse fit. Reiss (2009), for example, points out that evidence of probabilistic causation is susceptible to the old adage that correlation does not imply causation: “correlation is (. . .) neither necessary nor sufficient for causation” (p. 24). On the other hand, he argues, evidence of mechanistic causation also has its difficulties, for example, with cases of omission, where the absence of an event X causes an event Y , since in that case there is no process connecting X and Y (Reiss 2009, 25). So, he concludes, neither theory is universal, in the sense that neither provides a complete set of necessary and sufficient conditions for the truth of any arbitrary causal relation; the same goes for all the other fundamental theories of causation in the literature, none of which are good candidates for a “monistic” theory either.

If we take these limitations seriously, proponents of evidential pluralism argue, this means we ought to combine evidence of different sorts of causal theories (like probabilistic, mechanistic, or regularity causation) in order to “bear on a causal hypothesis and strengthen it” (Reiss 2009, 27). In other words, the pluralists suggest that evidence of “different sorts of things” (Illari 2011), such as correlations, interventions, entities and activities, can corroborate a causal hypothesis ¹ .

Russo and Williamson (2007) put this idea of “triangulation” into practice in their Russo-Williamson thesis, which states that researchers must use both correlational and mechanistic evidence to strengthen a causal hypothesis in the medical sciences. Either kind of evidence by itself is not sufficient to corroborate a causal claim. This “epistemic theory” involves, they argue, “a monistic conception of causality which is neither a purely mechanistic account nor a purely difference-making account—it involves aspects of the two types of approach” (Russo and Williamson 2007, 167); combining approaches is “required to capture the full complexity of causal evidence” (p. 169).

The idea that researchers can use evidence of different types of causation in triangulation also underlies multi-method research in the social sciences. We see this move, for example, when Sharon Crasnow applies Reiss’ views to political science:

“If multimethod researchers in political science are evidential pluralists, then the various different methods in the social sciences are all worth pursuing because they are all various ways of confirming or disconfirming the causal hypothesis. We might generate statistical evidence of causal connection through multiple regression analysis, a core statistical technique in political science research, but a case study that traced a causal process or identified a causal mechanism would make us more confident that we had indeed established a cause.” (Crasnow 2010, 37, emphasis added).

The idea of triangulating with evidence of different types of causation also comes up in the methodological literature. For example, Brewer and Hunter (2006) argue in their guide to multimethod research that “if our various methods have weaknesses that are truly different, then their convergent findings can be accepted with far greater confidence than any single method’s findings would warrant” (p. 4). For multi-method researchers, often the triangulation uses both quantitative and qualitative methods (e.g., Fearon and Laitin 2011). However, multi-method research can also mean “combin[ing] discrete techniques within one of these families of methods” (Ahmed and Sil 2012, 935), such as when “a multi-method approach can feature two or more types of qualitative analysis (say, a structured focused small-N comparison alongside an ethnographic study of a single community)” (p. 936).

Different views of causation coexist even within the “qualitative family,” as I will show below in my analysis of the case study method process tracing. Can we, and should we, combine evidence of different causal theories within process tracing? I will argue that a straightforward combination of methodologies is not always possible. In the next section, I will first discuss in more general terms the different ways that a combination of causal theories can go awry.

3. Epistemic Incompatibility

When we wish to use different causal evidential theories in a triangulating, “mixed-methods” approach, we must first ask to what extent the methods implied by the different theories are compatible. In general, the assumption is we can fruitfully combine knowledge of one type of causation linking a putative cause X and effect of interest Y (e.g., probabilistic causation) with knowledge of another type of causation linking X and Y (e.g., a causal mechanism), and that such a combination increases our confidence that indeed X is (or was) a cause of Y .

Let us now consider what might problematize this assumption. There are three main types of incompatibility between different evidential theories of causation: practical incompatibility, epistemic incompatibility, and ontological incompatibility. Under practical incompatibility, I here include such cases as time or funding constraints which make it difficult to use both methods in one study. Practical incompatibility also includes situations where due to a lack in training or experience by the research team, using both methods is difficult. After all, not everyone is equally well versed in, say, using mixed effect logistic regression and process tracing in individual cases. While practical incompatibility may seem relatively innocent, seeing as we could resolve time, funding, or training issues with a boost in resources, this type of incompatibility lies behind some of the recent pushback against multi-method research (cf. Ahmed and Sil 2012).

In epistemic incompatibility the issues lie deeper. Two methods are epistemically incompatible when one of the methods refuses to recognize as genuine knowledge some of the statements the other method claims to have evidence for. For example, we will see below that if one method assumes that counterfactuals are reliable evidence for a causal claim, while the other method is adamant we can never gain knowledge of the truth or falsity of counterfactual statements, this conflict is an example of epistemic incompatibility. So, while we may overcome practical incompatibility, if two methods have epistemically incompatible assumptions this conflict is not easily solved. We must question the validity of results from a combination of those methods.

The third, most fundamental type of incompatibility is ontological incompatibility. In the literature, we can find this type under the label “conceptual pluralism” (cf. Cartwright 2002; Psillos 2010). Conceptual pluralism is the view that “there is no unique nature of causation that [evidence of different types] track” (Psillos 2010, 132). We should separate this metaphysical claim from the epistemological claim that we cannot find a “single, unified and all-encompassing theory of causation” (Psillos 2010, 134); conceptual pluralists claim that “there is no metaphysical fact of the matter as to what causation is; no deep, simple and unified nature of causation” (p. 134). If we indeed make this assumption, then “triangulation” becomes impossible since each method will speak for a different fundamental type of causal claim (Cartwright 2002).

In short, we should differentiate between evidential pluralism and conceptual pluralism (Reiss 2009). However, conceptual pluralism is beyond the scope of my article. The pluralism I consider in this article is more closely related to that of Jon Williamson, who argues one ought to “accept that our causal beliefs are generated by a wide variety of indicators, but to deny that this variety of indicators yields a variety of concepts of cause” (Williamson 2006, 69). In the next section, I will focus my attention on the epistemic incompatibility of different methodologies developed for the case-study context, within process tracing.

4. Process Tracing and Evidential Pluralism

Process tracing is “the analysis of evidence on processes, sequences, and conjunctures of events within a case for the purposes of either developing or testing hypotheses about causal mechanisms that might causally explain the case” (Bennett and Checkel 2015, 7). Process tracing is used to find out how a particular outcome or event we are interested in in a particular case study came about. Process tracing is increasingly popular in the social sciences and a key technique in political science (cf. Hall 2013; Crasnow 2017). Bennett and Checkel (2015) distinguish “bottom up” and “top down” process tracing, that is, developing hypotheses and theories using process tracing versus testing them. “Top down” process tracing is sometimes labelled “theory testing” process tracing. The test relies on the development of “case-specific observable implications of the theories in question” (p. 18). So, in short, “top down” process tracers aim to establish which causal mechanism(s) led to an effect of interest in a case study; to do so, they identify and attempt to verify the observable implications of these mechanisms.

A representative example of this method is Tannenwald’s (1999) study of the United States’ non-use of nuclear weapons during the Korean War, the Vietnam War, and the Persian Gulf War. Tannenwald hypothesizes that a “nuclear taboo” was a key mechanism behind non-use. Here, the “taboo” refers to the idea that use of nuclear weapons was constrained by people’s revulsion against the use of such weapons. Some of the more specific causal mechanisms behind the non-use of nuclear weapons were adverse domestic public opinion, adverse opinion of world publics and leaders, and adverse personal conviction of political and military leaders.

Tannenwald attempts to identify and verify the observable implications of her hypothesis. She does so by “tracing” evidence of the taboo mechanisms in archives, witness accounts, etc. Among the events which give credence to her hypothesis are, firstly, the personal “moral concerns” that U.S. President Harry S. Truman and his advisors expressed about using such a “disproportionate” weapon (Tannenwald 1999, 446) during the Korean War. Secondly, while the subsequent president, Dwight D. Eisenhower, was not himself against using nuclear weapons, and in fact deplored the “taboo,” adverse public and world opinion constrained him from using such weapons.

Part and parcel of the top-down process tracing method is social scientists’ contrasting of their own proposed causal mechanism with alternative mechanisms found in the literature. Process tracing researchers search for observable implications (traces) of both. For example, Tannenwald contrasts her “nuclear taboo” to the theory of deterrence, and finds that because the theory of deterrence cannot explain the two “traces” mentioned above, her “nuclear taboo” is the more likely causal mechanism behind non-use.

Now that I have sketched process tracing in general terms, let us turn to the issue of evidential pluralism in process tracing. While the general method of process tracing as described above seems straightforward, the underlying methodology is far from settled. A key reason for this variability in methodologies is that we should take any reference to “causal mechanisms” as a metaphor; unlike in biology, the researcher cannot rely on a physical mechanism of “cogs and wheels” (or proteins and RNA sequences). We should not assume that all uses of the term “causal mechanism” in this literature belie a commitment to a “mechanistic theory of causation.” In fact, for some social scientists causal mechanisms are simply “inferences, reason, or argument,” and so we should be wary about thinking of causal mechanisms in social science using the terms of deeper philosophical theories of mechanism (cf. Craver and Tabery 2019).

There is no consensus about what the term “causal mechanism” does refer to (Mahoney 2001), even among process tracers. The methodological debates within process tracing center on one key question: for what kinds of things should we collect evidence, if we wish to process trace properly? As elsewhere in the sciences, the answer to this question depends on a methodologist’s commitments to a fundamental theory of causation. These commitments inform “divergent arguments about how process tracing should proceed” (Jacobs 2016, 14) Thinking back to the Tannenwald study above, we may ask: what should Tannenwald have collected evidence for? Probability raising? Counterfactuals? Some form of regularities?

So, in sum, “mechanism” as a concept hides a large variety of different notions of causation. Depending on methodologists’ underlying theory of causation, the researcher ought to collect evidence of different “things.” Evidential pluralists about process tracing, then, believe we ought to collect evidence of all the different indicators of causation. Evidential pluralists would here argue that although (or even because) methodologists disagree on what kinds of things we should collect evidence of, we can take the advice of each of them and jointly use these pieces of evidence to inform our belief in the purported causal mechanisms.

As I argued above, pluralists assume that the search for evidence of one type of causation is not somehow epistemically incompatible with the search for the other type. I defined two methods as epistemically incompatible if one of the methods refuses to recognize as genuine knowledge some of the statements the other method claims to have evidence for. I will argue in what follows that some of the methodologies for process tracing meet this definition. I will consider two conflicting methodologies in particular: the systems view and the interventionist view of process tracing.

5. Saving Evidential Pluralism

So far, we have seen a case where two different methodologies for process tracing are reliant on directly conflicting epistemic assumptions; each method refuses to accept some of the other’s central claims as genuine knowledge. This causes a clear tension for evidential pluralism. The key question now becomes: can we alleviate this tension? What are our options going forward?

Firstly, we might consider giving up on evidential pluralism altogether. However, I would argue that to accept the idea that only one view of causation is correct overall ignores the argument for pluralism in the first place, that is, that there are counterexamples to all different views of causation, as well as areas of “best fit” for each view. As such, giving up on evidential pluralism is too drastic.

Secondly, we might get out of the tension for evidential pluralism by barring certain methodologies as somehow “unsuitable” for the aims of case study analysis. For example, we might be tempted to restrict evidential pluralism to only those methodologies that are non-probabilistic, letting the researcher combine at will all non-probabilistic views. However, it is unclear to me how this restriction would proceed in practice; the previous example shows that even within non-probabilistic approaches, we may run into a “pluralist tension.”

In the remainder of this article, I will outline a third solution for evidential pluralism in case study research. To set up this solution, let me begin by investigating the epistemic disagreements between the systems view and interventionist view further.

As I have set out the methods above, the systems view seems to exclude counterfactual evidence as genuine knowledge, while the interventionist view excludes singularist observations. Let us unpack what this might mean. In this section of the article, I will argue that both camps believe that the other’s (counterfactual or singularist) propositions are not justified. I will refine this criticism by introducing the evidential context in which process tracers operate to the discussion. Instead of arguing that it is inherently impossible to justify the truth of counterfactuals or make truthful singularist observations, I will argue that evidence of counterfactuals and singularist observations each have a certain degree of reliability, depending on the evidential context. Doing so, I will argue, allows us to save a more modest version of evidential pluralism.

The concept of reliability stems from theoretical epistemology (cf. Nozick 1981), where it is sometimes broken down into positive and negative reliability (List 2006). These terms indicate respectively how likely it is that a subject judges a claim to be true if indeed the proposition is true, and how likely it is that a subject judges a proposition to be false if the proposition is false. Adapting this to our discussion for causal mechanisms in case studies, positive reliability, here, should be seen in terms of how likely it is that a social scientist judges, for example, a counterfactual to be true, given that the counterfactual is actually true ³ . Negative reliability indicates how likely it is that a social scientist judges the counterfactual to be false, given that the counterfactual is actually false.

Phrased in terms of reliability, we see that the systems view believes our judgements of the truth or falsity of counterfactuals are unreliable, while the interventionist view believes our judgements of causal claims based on singularist observations are unreliable. Yet this is too strong a claim. Neither source (counterfactuals or singularist observations) should be judged to be inherently reliable or unreliable without also specifying the evidential context. ⁴ We must consider the study’s evidential context to decide which type of causal evidence is most applicable, based on the reliability of, for example, counterfactual or singularist observation claims in that evidential context ⁵ . Neither evidence is intrinsically reliable or unreliable.

Using the process tracing case study introduced earlier in this article (Tannenwald 1999), we can see how a study’s evidential context will help us choose the appropriate methodology. Arguably, in the Tannenwald example a search for what would have happened if the putative cause had been different (following the interventionist view) fits with what the author thinks her evidential context is like. A counterfactual analysis of causation is reliable in Tannenwald’s context.

The first reason for taking counterfactuals as reliable in Tannenwald’s case is that part of her work is aimed at showing what would have happened if there had been no nuclear taboo: “If there had been no normative opprobrium [due to the taboo]—that is, if the “rules” had been different—it is likely that we would have seen resort to [nuclear] weapons at some point during the Cold War” (Tannenwald 1999, 463) Tannenwald believes that she can reliably judge such counterfactual claims based on the evidence she collected.

Secondly, Tannenwald wishes to use the American case study to test a more general hypothesis, viz. that the nuclear taboo explains any state’s non-use of nuclear weapons better than alternative theories like deterrence. Imagine Tannenwald were able to collect independent evidence that the states she is interested in are sufficiently similar (e.g., using statistical analysis or a combination of a large number of case studies). This will then also indicate that the truth and falsity of a within-case counterfactual, such as the one about the United States, is easier to track; in other words, the counterfactual evidence will be more reliable. After all, the simplest counterfactuals ask us what would have happened if the putative cause had been absent, all other things being equal. By comparing with other cases where this cause was actually absent, we thereby corroborate this counterfactual, but only if these cases are sufficiently similar to the original. ⁶

I have briefly sketched why counterfactual claims are arguably more reliable in the context of Tannenwald’s study. However, there are also contexts in which evidence for counterfactual claims is unreliable. Think, for example, of trying to establish a counterfactual in a highly heterogeneous population: comparing cases would be out of the question, which means the truth value of counterfactuals is much more difficult to ascertain. Thus, all other things being equal we cannot judge the truth value of counterfactuals with any great reliability.

When might singularist observations be more reliable? This would require the causal links to be somehow “visible.” Beach and Pedersen (2019) establish that if one can collect as evidence, for example, witness statements from a reliable source with no ulterior motives, who could have known about the events (p. 207-213), or “‘hard primary evidence’—for example, official internal documents produced by public authorities that describe what took place behind closed doors” (p. 215), this makes singularist observations more reliable. Thereby, the systems view is arguably applicable in such cases.

The systems view may also be more appropriate in relatively unexplored cases, where one has a limited idea of what the causal network might look like. The interventionist view requires that we have information about all the factors in the causal network under study. Evidence for interventions for a putative relationship X → Y requires us to separate effects of the intervention on Y through X from effects of the intervention on Y not through X . However, this does require that we are aware of all other potential causal factors connected with Y . If we are not so well informed, this makes interventionist claims unreliable; it does not detract, however, from the reliability of singular observations, as long as these observations meet Beach and Pedersen’s demands above.

Finally, the counterfactuals in the interventionist view will only be reliable in cases where the causal mechanism under investigation is modular: that is, “the components of a mechanism should be independent in the sense that it should be possible in principle to intervene to change or interfere with the behavior of one component without necessarily interfering with the behavior of others” (Woodward 2002, 374). A broader discussion of modularity is beyond the scope of this article; I will only note here that we can trace the idea that modularity is not a universal characteristic of causal systems back to Cartwright (2007), who argues that while modularity is “epistemically convenient” (p. 81), “we can have causality without modularity” (p. 82). Again, the systems view has no such demands and singularist observations can be reliable where modularity fails.

Let us sum up. By now, I have argued that we ought to consider the epistemic reliability of evidence for the fundamental concepts of a methodology we are interested in using. Evidence of such concepts is never completely reliable or unreliable; rather, their reliability depends on the evidential context of the case study. Consequently, while the systems view may be appropriate in certain case studies, the interventionist view will be more appropriate in others. Before attempting to combine the two, we must take the evidential context into account. Based on my initial analysis above, it may be unlikely that their reliability overlaps. As such, we have not erased the issue of epistemic incompatibility altogether; rather, this conclusion might be labelled contextual compatibility.

Nevertheless, this conclusion leaves room for a type of pluralism. This pluralism must be more modest than the strong theory of evidential pluralism from the start of this article: instead of arguing that each case study analysis benefits from evidence of different kinds of causation, in a triangulation-style approach, we ought to say that every different analysis potentially benefits from a different kind of causation. In other words, two different case studies might be compatible with two different methodologies, depending on how reliable the assumptions of that search are in the context of the case ⁷ .

6. Conclusion

In this article, I have argued that a simple combination of methodologies in the case study method process tracing is ill-advised. I have shown that different methodologies may contradict because their epistemological assumptions are fundamentally in disagreement, as is the case in the systems vs. interventionist views. I have argued that the choice for a methodology depends at least on a case study’s evidential context, since evidence for the fundamental type of causal evidence in these methodologies will be more reliable in some contexts than others.

Further research on this matter can go in at least two directions. Firstly, we may ask how the paradoxes and limitations to theories of causation (like redundancy and overdetermination), which formed the motivation for evidential pluralism in the first place, link to the concrete evidential contexts that are found during process tracing. By working out which theory of causation is called for in particular case study analyses, we will make progress in both philosophy of causation and methodology. The varying epistemic reliability of evidence is a problem for evidential pluralism outside process tracing as well. We might be able to apply the concept of epistemic reliability when we analyze research that mixes quantitative and qualitative approaches, for example.

Secondly, we may explore whether the pluralist tension as I have outlined it in this article is unique to the examples given, that is, to these particular methodologies for process tracing in political science, or occurs more broadly in the social sciences. Are there certain methodologies which are reliable at the same time (in the same evidential contexts)? The analysis in this article may also impact on work outside of the social sciences: the desirability of evidential pluralism is also under discussion in areas like the health sciences, and the notion of epistemic reliability may fit well with Williamson’s argument there for interpreting causality as an epistemic relation (Russo and Williamson 2007; Williamson 2006, 2009).