Still Searching for the Value-Added

Inferring Necessity and Sufficiency: Relative Frequency Is Crucial

The focus on necessary and sufficient causal conditions is a fundamental feature of STCM. The point of departure in this discussion is the standard STCM argument that scholars do not need to incorporate data from all four cells in a 2 × 2 table. ¹ Schneider’s (2016) proposal to focus on all four cells is a valuable step forward, yet the standard rejection of using data from all four cells is so common in the STCM literature that my critique of this position hardly involves a “straw man,” as he puts it (p. 5). ²

Recognizing that all four cells matter raises crucial issues about how to combine information from the cells to make valid inferences. Because STCM encourages researchers to ignore certain cells, scholars in this tradition have generally overlooked these issues. Furthermore, even contributions such as Schneider and Wagemann (2012) that have proposed STCM procedures for productively incorporating all four cells face important shortcomings. When the distribution of X = 0 and X = 1 cases is skewed, set-theoretic concepts—which do not account for relative frequency—do not adequately capture notions of necessity or sufficiency. They likewise can generate easily avoidable false negatives or false positives. Therefore, core STCM questions such as “is X a superset of Y ?” (Schneider, 2016, p. 5) may provide misleading inferences about necessity or sufficiency. It is crucial to instead use metrics that adjust for relative frequency, as I have sought to do, in contrast to Schneider’s rejection of my proposed consistency measure (Paine, 2016).

How STCM Can Produce False Positives

Throughout the history of major league baseball, is a non-Babe Ruth at-bat necessary for a home run? Table 1 provides data to address this question posed in my article (Paine, 2016).

OPEN IN VIEWER

Table 1.

Home Runs by Batter, 1871 to 2015.

Batter	Home run
	No	Yes
Not Babe Ruth	15,835,542	282,336
Babe Ruth	9,908	714

Source. Data from Sports Reference (2015).

The set of home runs hit by individuals other than Babe Ruth is almost a perfect subset (in percentage terms, 99 . 7 % ) of all home runs hit in baseball history—consistent, by the STCM metric, with the hypothesis that an at-bat by a player other than Babe Ruth is almost a necessary condition for a home run. Although Schneider correctly notes that using Schneider and Wagemann’s (2012) secondary triviality metric would guard against this mistaken inference, Schneider’s conclusion overlooks two important conceptual concerns with his procedure. First, it is troubling that a “consistency” measure can suggest that data are almost completely consistent with a deterministic hypothesis—because of skewed case distribution—when they are obviously not.

The second concern is that Schneider and Wagemann (2012) bracket two very different ideas under the label “trivial.” The first is standard: one value of Y occurs rarely. For example, although oxygen may be a necessary condition for a social revolution, the presence of oxygen carries almost no predictive power for when social revolutions occur because social revolutions happen infrequently. The second idea is non-standard: one value of X occurs rarely. The two Babe Ruth examples above represent the second idea because there was only one Babe Ruth. Crucially, in the Babe Ruth examples, the presence or absence of X provides strong predictive power for the outcome. Therefore, it is not clear why X should be considered a trivial necessary condition. STCM scholars have introduced confusion in their causal analysis by conflating disparate concepts under the same label. Alternatively, if one rejects Schneider and Wagemann’s (2012) unsatisfying second type of triviality, then the problem remains of false positives caused by skewed case distribution.

Inferring Causal Complexity: Regression Interaction Terms Provide Valuable Insight

The focus on causal complexity—in the sense of multiple conjunctions of causal conditions—is another STCM centerpiece. Thiem et al. (2016) and Schneider (2016) expand on existing arguments by claiming irreconcilable methodological differences, arguing that regression interaction terms simply fail to address casual complexity as they conceptualize it. ³ I counter that both methods can be used to study causally complex relationships. I also question whether distinctive STCM approaches to limited diversity and multichotomous variables truly advance political methodology.

My article provides an extended example demonstrating how, in a binary setting, modeling data with regression interaction terms can provide the same information as an STCM truth table (Paine, 2016, Tables 6 and 7). Hence, Schneider’s (2016) contention that I failed to show how regression distinguishes between necessity and sufficiency, or to explain the outcome versus non-outcome, is based on a misunderstanding of my argument. Because a fully saturated regression model can recover all four cells in a complex 2 × 2 table, a scholar can use whatever metric they want to (a) compute necessity or sufficiency scores, (b) analyze the outcome versus non-outcome, or (c) carry out any other procedure that one could perform with STCM tools.

Thiem et al. (2016) are correct that STCM’s conjunctions and statistical interactions are different concepts. However, their argument does not preclude the possibility that regression interaction terms can provide insight into causal complexity. To provide a simple example, suppose there are two posited causal conditions, A i ∈ { 0 , 1 } and B i ∈ { 0 , 1 } , and an outcome Y i ∈ { 0 , 1 } . To assess the working hypothesis that A i = 1 is a necessary condition for Y i = 1 , a scholar should estimate the regression model in Equation 1:

Y i = β 0 + β A ⋅ A i + β B ⋅ β i + β A B ⋅ A i ⋅ B i + ε i

If Y i = 1 indeed never occurs empirically when A i = 0 , then estimating Equation 1 with ordinary least squares will produce β 0 = β B = 0 . These coefficient estimates—whatever the frequency of Y i = 1 when A i = 1 (which determines β A and β A B )—imply E [ Y i | A i = 0 ] = 0 . Furthermore, fitting the model with heteroskedastic-robust standard errors will estimate Var ( ε i | A i = 0 ) = 0 . These estimates correspond perfectly with a necessary condition hypothesis. ⁴

A similar exercise could be performed for any necessary or sufficient condition hypothesis with any number of binary variables. Despite true differences between conjunctions and interactions, they provide the same insights into the social world in a binary setting.

In two other regards, the results yielded by STCM are distinct from that of regression, but STCM’s methodological value-added is questionable. First, Schneider (2016) mentions limited diversity. One useful consideration is that quantitative scholars frequently, albeit implicitly, focus on limited diversity when evaluating multicollinearity in their data—a topic covered in any introductory econometrics textbook. If there are few cases with certain values on a higher order term, then including both the higher and lower order terms in the same regression model will introduce considerable multicollinearity (or, in some cases, perfect multicollinearity, making it impossible to estimate the higher order terms). STCM scholars have commendably devoted considerable attention to limited diversity. However, it may often be preferable to avoid estimating coefficients based on data characterized by limited diversity, rather than imposing very strong and often unverifiable assumptions about “logical remainders,” that is, the combinations of conditions not observed empirically.

Finally, whereas STCM is less distinctive than claimed with binary variables, the value-added of its distinctiveness with multichotomous variables remains to be established. Although regression does not perfectly mimic STCM when using fuzzy, multi-valued, or generalized sets (Schneider, 2016), this observation does not necessarily support using STCM. Brady (2013) expresses concern that necessity and sufficiency become less meaningful concepts when one moves beyond binary variables. In addition, concerns arise about aggregating fuzzy sets to evaluate necessity and sufficiency (Dunning, 2013) and using unreliable calibration procedures (Krogslund, Choi, & Poertner, 2015). Finally, the strong STCM claim to be an alternative to statistical methods is called into question by the crucial importance of using conventional statistics to assess whether a particular STCM finding arose due to chance (Braumoeller, 2015).

Conclusion: Qualitative Methodologists Should Turn From STCM to Other Tools

These arguments support the core contention that STCM is less distinctive than claimed vis-à-vis quantitative methods. Furthermore, many novel STCM techniques produce problematic inferences, which—in addition to the points emphasized above—include the low standards for inferring deterministic causation. Paine (2016) and Schneider (2016) agree on this latter point.

Furthermore, Munck (2016) is correct in arguing that these critiques of STCM do not reject qualitative methods more broadly. The field of qualitative methodology would be better served by focusing on tools such as process tracing that do possess distinctive advantages relative to quantitative methods—specifically, using knowledge about the causal process to draw causal inferences. This focus is far preferable to perpetuating claims about STCM’s distinctiveness that yield an unproductive methodological divide.