Comment

Worrisome Simulation Studies

Simulation studies raise serious concerns about QCA. Hug (2013) reported unstable findings with “drop-one” and “drop-two” tests and with simulated measurement error. In Lucas and Szatrowski’s (this volume, pp. 1–79) 70 simulations, QCA returns the correct causal story only 3 times, yielding many false positives and false negatives. Krogslund, Choi, and Poertner (2013) discovered great sensitivity to error in setting the parameters for fuzzy-set calibration and causal inference ³ ; and Seawright (this volume, pp. 118–21) carefully incorporates QCA’s concern with limited diversity, reporting false positives and failures in analyzing causal complexity. Krogslund and Michel (2014) drew on a key concept in current thinking about causal inference: the “data-generating process,” that is, the often unobserved causal process that generates the data. Using simulations to create a “known” data-generating process, Krogslund and Michel demonstrated QCA’s limitations in making causal inferences.

Schneider and Wagemann’s (2012) discussion of simulations and robustness is more encouraging, but they concluded with great caution: “QCA is not vastly inferior to other comparative methods in the social sciences” (p. 294). Given sharp criticism noted below of conventional quantitative analysis—a key method of comparison—this is faint praise.

How should scholars assess these simulations?

First, they must consider the simulations themselves. Simulations must fit the method being evaluated, and scholars should and will scrutinize this fit. ⁴ In this respect, the new simulations are a major step forward. They use QCA software and carefully seek to match the simulation to the method. Unquestionably, there will be room for improvement and refinement in future simulations, yet the cumulative evidence of unstable findings and error raises major concerns.

Second, scholars must ask why QCA findings might be unstable and prone to error, and whether there are basic problems with QCA’s approach to causal inference. Various problems merit attention. For example, even with the fuzzy-set version that incorporates gradations, the algorithms ultimately reduce the findings to dichotomies, which can be more vulnerable to error than gradations (Elkins 2000:298–99). The number of cases per causal path is often surprisingly small, raising major concerns about the stability of findings. QCA has a procedure that helps avoid false negatives, but appears to lack a corresponding procedure to protect against false positives.

Rethinking Causal Inference

More broadly, QCA’s emergence as a potentially important method coincides with a wide-ranging challenge relevant to diverse methods, including conventional quantitative analysis as well as QCA. These methods are seen as too confidently making causal inferences, based on complex algorithms applied to observational data. Paine’s (2013) argument that the algorithms of these two methods are more similar than has been recognized makes this critique all the more salient here. Given this new challenge, the simulation findings might come as no surprise.

QCA has thus failed to draw on the recent rethinking of causal inference in the social sciences. This rethinking is anchored in skepticism about complex algorithms, commitment to parsimony, and caution about inferences from observational data. Many scholars now seek the most “simple and intuitive” tools adequate to the task at hand (Sekhon 2009:494), for example, comparison of frequencies or percentages, cross-tabulations, and scatterplots. The eminent statistician Freedman (2008) recommended “cross-tabulation before regression” (p. 241). Achen (2002:446) considered it “meaningless” to test more than three explanatory factors. These warnings have focused on conventional quantitative analysis, yet they are a wake-up call for other methods as well, including QCA.

This rethinking gives new authority to case knowledge, process tracing, and the use of qualitative data in constructing good research designs. Freedman (2010) argued that qualitative reasoning is a “type of scientific inquiry” in its own right.

Implications for Qualitative Comparative Analysis

QCA is well aligned with this focus on case knowledge, but in other respects, it has followed a different path. The method gives inadequate attention to (a) obstacles in making compelling causal inferences (e.g., about necessary and sufficient conditions); (b) error and instability of findings that routinely arise with algorithms (i.e., to reiterate, with the method’s basic analytic procedures; see again note 2); and (c) vicissitudes of working with observational data.

QCA scholars would do well to join the skepticism about algorithms and shift to core building blocks: the methodology of case knowledge and possibly the basic truth table, used specifically not as a logical construct but as a revealing data display. The contrast between the parsimonious clarity of the truth table, versus the QCA algorithms (including the algorithms for analyzing these tables), may be a good example of “simple is better.”

To conclude, QCA has valuable objectives: understanding context, interactions, casual complexity, and asymmetry. However, the algorithms may be poorly suited to these objectives. Lucas and Szatrowski’s and Seawright’s conclusions are salient here: Attention should shift from QCA, as currently practiced, to more conventional qualitative methods. Standard tools such as case studies and process tracing are a better point of departure. Case knowledge should not be an adjunct to the algorithms but rather the primary method of analysis. Rohlfing and Schneider (2013) contributed to this much needed reorientation of QCA. The algorithms themselves appear ill suited to QCA’s goals.