Assessing Set-Theoretic Comparative Methods

The Appeal of STCM

Much of the interest in STCM is rooted in the causal concepts used in this method. Proponents of STCM are correct in defining causation in terms of necessary and sufficient conditions. In terms of logic, it is appropriate to state that X causes Y means “X if and only if Y.” They are also right in thinking about causation in deterministic terms. There are grounds to think that the social world has a stochastic component and hence it makes sense to state that X causes Y with probability p. But proponents of a probabilistic concept of causation overlook a basic point. Causes are not propensities. Thus, the concept of causation is logically prior to the concept of conditional probability. Valid points made in the STCM literature are frequently dismissed by statistically oriented researchers.

Proponents of STCM also offer a useful diagnosis of some practices in regression analysis. For example, an argument introduced in Ragin’s (1987) work that recurs in the STCM literature is that regression analysis suffers from weaknesses such as the tendency to rely on additive models and hence to ignore the potential impact of configurations of causes, and the inclination to largely assume unifinality rather than place an accent on the possibility of equifinality. These critiques are well taken. Moreover, they reflect long-held concerns of many qualitative researchers about regression analysis.

In other words, advocates of STCM underscore some useful ideas about causation that offer a counterpoint to a statistical understanding of causation and, more specifically, regression analysis, and that have appeal among qualitative researchers. However, these ideas about causation and causal complexity are not enough to establish the comparative advantage of STCM relative to regression analysis and to justify an embrace of STCM by qualitative researchers. Indeed, as the contributions by Thiem et al. and Paine show, the case for STCM is anything but simple or undisputed. Moreover, as these comments argue, STCM has some problems that are quite fundamental.

Formalizing Causal Relations

Thiem et al. move the discussion about STCM forward in some important ways. They make good use of the basic distinction between syntax and semantics, and draw attention to sometimes ignored differences in the semantics of formal languages. More pointedly, they avoid some common mistakes in the characterization of STCM and clarify the distinctiveness of the formal arguments offered in the STCM literature.

However, Thiem et al.’s arguments about STCM raise many questions. A key problem in Thiem et al.’s discussion is that they posit, as do other advocates of STCM, an analysis of causation entirely in formal terms. This is a basic oversight, because a causal relation is not a logical relation but, rather, a relation between events or, more precisely, between changes in the properties of things. But Thiem et al. have nothing to say about the semantics of empirical sciences, as opposed to the semantics of logic and mathematics, and fail to recognize that logical connections do not occur in real processes as they do in relationships of logical equivalence.

This oversight has consequences for the way Thiem et al. propose to formalize causal relations. By ignoring that causation must be understood ontologically and implicitly reducing causation to a logical relation, Thiem et al. err in proposing that causal hypotheses can be adequately formalized as a relation of material implication. Among other things, such a formalization neglects to acknowledge that causal relations are asymmetric, that is, that events X and Y are not interchangeable as they are in the biconditional “X if and only if Y.”

Causal theories should be built with due attention to the rules of logic. But Thiem et al. offer a logical transcription of the cause-effect connection that is inadequate. Thus, inasmuch as the reliance on implication hypotheses that link a condition with an outcome is a distinctive and central feature of STCM, its appeal is tarnished. Both qualitative and quantitative researchers are better off presenting their causal arguments in the general form ΔX → ΔY, meaning that a change in property X causes a change in property Y, and treating the covariation between X and Y as an essential means for checking whether their causal arguments are true.

Testing Causal Relations: STCM and Regression Analysis

Another problem in Thiem et al.’s discussion is their insistence that STCM and regression analysis are “incommensurable”; they use the term 7 times. It is undeniable that there are differences between STCM and regression analysis. However, this does not mean that they cannot be compared. Moreover, as Paine demonstrates, it is valuable to compare the empirical performance of STCM and standard regression analysis. Indeed, though Thiem et al. imply that such a comparison would reinforce some basic misunderstandings about the differences between STCM and regression analysis, and block progress in the field of methodology, Paine’s comparison of STCM and standard regression analysis is particularly instructive.

One issue Paine addresses is the view of advocates of STCM that, in testing hypotheses about necessary conditions, only two cells in a 2 × 2 table are relevant: those in which the outcome of interest is present. However, as Paine argues, even if the purpose is to test hypotheses about necessary conditions—and accepting the STCM premise that empirical counterexamples do not falsify a deterministic hypothesis—dismissing information about some cells may lead to false positives.

The solution Paine proposes is to use information about the cases in the other two cells so as to be able to compare the percentage of X = 1, Y = 1 cases with the percentage of X = 0, Y = 1 cases. And this useful solution leads to different results from those produced by STCM. Thus, the doubts Paine raises, from a regression perspective, about a central procedure in STCM deserve to be taken seriously.

Testing Causal Relations: STCM and Process Tracing

Yet another problem with STCM likewise affects its potential as a tool for testing causal hypotheses. Inasmuch as a causal relation can be represented most generally in the form ΔX → ΔY, what needs to be compared is a set of cases when X = 0 with the same set of cases when X = 1. That is, if the aim of the analysis of data is to interpret results in causal terms, information about change over time is essential. But as Thiem et al. point out, STCM relies on a static comparison of the presence/absence of some condition in different cases as opposed to variation in some condition and, more specifically, change in the value of conditions or variables over time. In effect, STCM’s reliance on a concept of causation understood in purely logical terms leads to the exclusion of information about change over time, severely handicapping STCM for the purpose of causal assessment.

This problem with STCM is not addressed in the articles in this symposium. But it is critical to an evaluation of the method. Moreover, it is very relevant to the claim that STCM is particularly well suited to the kind of concerns expressed by qualitative researchers. Thus, a few comments regarding STCM’s reliance on static as opposed to dynamic comparisons are in order.

One possible response to this critique of STCM, implicit in proposals by advocates of this method, is that the problem of static comparisons can be solved by subsuming process tracing under STCM. This response might seem adequate, in that process tracing is a method that is concerned with change over time and that can be used to study causal mechanisms. Indeed, this response might even seem persuasive, particularly for qualitative researchers. Quantitative researchers study mechanisms through mediation analysis that treats mechanisms as intervening variables, thus relying as usual on covariation to estimate causal effects. In contrast, though qualitative researchers think about mechanisms in various ways, they tend to frame the study of mechanisms differently, not as a matter of ascertaining if there is a causal effect and how strong it is, but rather of learning how a causal effect is transmitted.

This is a genuine difference: Process tracing is not mediation analysis. What is more, the understanding of mechanisms favored by qualitative researchers is arguably superior. It is an error to equate a mechanism with an intervening variable. To describe a mechanism is not merely to show that something happened but rather to shed light on the nature of a process. And it makes no sense to assess mechanisms in terms of the strength of causal effects. Causal mechanisms are not a special case of causal effects, and there are no strong or weak mechanisms. Thus, the advantage of STCM over regression analysis, as well as the reason why qualitative researchers should use STCM, could lie in the role of process tracing in STCM.

Nonetheless, process tracing is not part of STCM. The idea of process tracing was first introduced in the literature on traditional qualitative methods, where it is treated as the core of a method. In contrast, it is not intrinsic to STCM. Instead, process tracing is a recent add-on to STCM that actually clashes with the idea of causal relations as logical relations. In other words, the discussion of process tracing in the STCM literature is an indication of its failure to address dynamics and does not offer a solution to the limitation of a static analysis consistent with the principles and procedures of STCM.

Let’s Set Aside STCM

In conclusion, STCM’s approach to formalizing and testing causal relations is problematic. Advocates of STCM have neither established the advantage of STCM relative to regression analysis nor justified why qualitative researchers should embrace STCM. Indeed, this last point deserves to be made bluntly: Qualitative comparativists who are already inclined to assess causal arguments through within-case over-time studies would weaken their power of causal inference if they traded in their methods for STCM.

The strand of methodological research launched by Ragin, and continued by Goertz, Mahoney, Rihoux, Schneider, Wagemann, and others, has addressed key issues and made important points ignored in other literatures. In addition, the effort of these scholars to develop methods suited to the goals that qualitative researchers pursue is laudable, and their critique of attempts to homogenize research under restrictive and even skewed rules should be taken seriously. The drive to impose a partial view of how to do science has been a recurring trend in the social sciences and is especially strong these days.

Yet, the vision of diversity in unity offered in the introduction of Paine’s article, with its explicit recognition of the contributions made by the qualitative and quantitative traditions, points in directions that are far more compelling. The arguments elaborated in Paine’s article, along with the comments offered here, are not remotely intended to imply that there is one, or even a best, solution to the problem of causation. Likewise, these commentaries are not aimed at challenging the qualitative tradition of research. Therefore, this discussion of STCM should be treated as an opportunity to rethink the suitability of different tools for qualitative comparative research and to better orient future methodological work on pressing issues such as the relationship between covariational analysis and research on mechanisms.

In that spirit, it is time that this discussion of STCM be brought to a close and that energies turn to other options.