Assessing ESA on What It Is Designed for

Common Ground

Before going into differences, it is useful to list where CG and we seem to be standing on common ground. First, since CG do not express any explicit criticism of the standard analysis (SA), we infer that they, like us, are fine with using logical remainders for (simplifying) assumptions. Second, we agree that statements of necessity usually imply assumptions on logical remainders. Third, we share the conviction that untenable assumptions must be avoided. One form of untenability is when assumptions on logical remainders contradict each other. We think the uncontested understanding of a contradiction is that X cannot be sufficient for Y and ∼Y at the same time (X → Y and ∼Y), or that X cannot be necessary for Y and ∼Y at the same time (X ← Y and ∼Y).

ESA Does Not Postulate Necessary Conditions

We call ours the ESA because, compared to the SA, ESA does limit the use of simplifying assumptions to only those that are tenable. Assumptions can be untenable for various reasons. One of them is that they contradict claims of necessity. It is important to note that ESA is absolutely silent as to whether a researcher ought to declare a given condition as necessary.

CG’s critique crucially rests on the meaning they attribute to the term “empirically necessary conditions.” In their reading, ESA entails that researchers must always declare any condition as a necessary condition when it is empirically found to be a superset of the outcome. This not only is a far cry from what we state in our treatment of ESA, it is also not in line with our writings on the strategies for the analysis of necessity (e.g., Schneider and Wagemann 2012:220–232). We quite clearly say that, in addition to empirical support for a statement of necessity, there must be theoretical and conceptual arguments as to why it is plausible to declare a given condition as necessary for an outcome. This is why, in our examples used for illustrating ESA, we chose to qualify conditions as simply being “empirically necessary.” Whether such conditions ought to be considered necessary conditions depends on theoretical and conceptual arguments, none of which is the business of ESA. ESA only kicks in once a researcher, such as Stokke in one of our examples also used by CG, has declared a condition as necessary. CG misread our point into the exact opposite and wrongly attribute to us the claim that ESA forces researchers to declare as necessary any condition that is a superset of the outcome.

The erroneous consequence of charging ESA with giving misleading advice to researchers as to which conditions ought to be considered necessary becomes apparent when CG write:

It is true, in the Stokke case, that if A is necessary for SUCCESS, then ∼A should not appear with SUCCESS. But of the 16 possible rows involving ∼A, only one is present. In using the ENC to rule out the 15 reminders having the outcome SUCCESS, ESA goes beyond the available evidence. It does this algorithmically rather than on the basis of theoretical reflection. Can we really be sure that advice from a regime’s own scientists (A) would be needed for SUCCESS in all conceivable cases.

What CG ignore here is that it is Stokke himself (and later Ragin as well) who, based on empirical evidence and theoretical arguments, declare A as a necessary condition, not ESA. ESA simply takes it from there and states that once a researcher claims A ← S, then any logical remainder including ∼A cannot be implied by the sufficiency solution for S. Since CG agree with the latter, and since, again, ESA is silent as to why a condition ought to be considered as necessary, their objections to ESA are unfounded.

The Use of DeMorgan in Incomplete Truth Tables

CG write that because of the claim A ← S, we would say that ∼A → ∼S. It is true that to identify simplifying assumptions in the sufficiency analysis for outcome S that contradict a statement of necessity for outcome S, we make use of DeMorgan’s law, following which A ← S also means ∼A → ∼S. With this we do not say that the sufficiency solution formula for outcome ∼S will automatically contain ∼A as a single sufficient condition. In our book (pp. 83, 114–15, 279), we repeatedly stress that unless the truth table is fully specified (no logical remainders and no inconsistent rows), it is wrong to infer solutions for outcome ∼S by applying DeMorgan to the solution for outcome S. One reason is precisely that this would lead to contradictory claims about truth table rows. What we do say is simply that any sufficient term for outcome S cannot be a subset of condition ∼A (pp. 201–203). Whether ∼A alone is or is not a subset of ∼S (as required by DeMorgan) is an empirical question and unrelated to the claim that ∼A cannot be a subset of S.

Related to the issue of what the noninclusion of a remainder into a logical minimization means, CG make a, we think, wrong statement when they write “that the complex solution … effectively sets all the remainders to “false” (i.e., assumes that, were the missing configurations to exist, they would have the outcome ∼SUCCESS).” However, not including a remainder row into the analysis of SUCCESS simply means that this row is not sufficient for SUCCESS. This is not the same as saying that it is sufficient for ∼SUCCESS because the same remainder could be set to false in the analysis of ∼SUCCESS as well. In other words, it is possible that a row is neither sufficient for SUCCESS nor for ∼SUCCESS.

Contradictory Simplifying Assumptions in the Analysis of Necessity

CG wonder if ESA can defeat itself. As a demonstration that, according to them, it can, they claim, still using the Stokke data that ESA force researchers to claim that I ← ∼S. And because ESA has already allegedly forced the same researcher to state that A ← S, they cogently show that this researcher has engaged in contradictory assumptions. Given the Stokke data at hand, for both necessity statements to be true, any remainder row containing ∼A × ∼I would need to be sufficient for both S and ∼S. This is a clear logical contradiction, and the two necessity statements cannot be made at the same time. CG see this as a self-defeat of ESA. Quite the contrary: Not only is ESA silent as to whether a researcher declares a condition as necessary or not, it is also precisely the sole purpose of ESA to rule out any logically contradictory assumption on logical remainders—just what CG also argue for.

Everything CG write about the incoherence of the two necessity claims is fully in line with ESA: Do not make assumptions that contradict a statement of necessity. Given the Stokke data, researchers cannot claim both A ← S and I ← ∼S. Instead, they must settle on either one or the other (or none of the) necessity claims. Just as ESA is silent as to which conditions ought to be considered necessary, it is also agnostic as which of the two necessity statements to maintain and which one to drop. ESA simply states that you cannot make both necessity claims.

With this example, CG draw our attention to the insight that the well-known and long-discussed problem of contradictory assumptions (Yamasaki and Rihoux 2009) not only pertains to the analysis of sufficiency of outcomes Y and ∼Y but is also salient for the analysis of necessity of Y and ∼Y. ESA is precisely about not making such contradictory assumptions.

Model Ambiguity

CG’s discussion on contradictory assumptions in the analysis of necessity implicitly points to a general problem that is worth being spelled out explicitly. It can be summarized as follows: There is only a limited amount of inferences that can be drawn based on a single truth table. If researchers, based on one and the same incomplete truth table, make claims of necessity and sufficiency for both Y and ∼Y, then the risk of making untenable assumptions rises and the number of tenable simplifying assumptions shrinks. Our experience shows that with more remainder rows being ruled out as untenable, not only does the solution formula become more complex; it also increases the likelihood of encountering the phenomenon of model ambiguity—that is, the fact that there are multiple, logically equivalent solution formulas (Thiem 2014). Again, this is not a problem of ESA, which simply brings the problem to the light by ruling out untenable assumptions. The real problem is the attempt at drawing too many inferences on too little information.

Conclusion

CG’s argument against ESA rests on the misbelief that ESA is a strategy based on which researchers must always claim a condition X to be necessary for Y if and when there is empirical evidence at hand. This is wrong. ESA is agnostic as to why researchers decide to (not) declare a given X as necessary for Y. ESA simply states that if and when a statement of necessity is made, then no remainder row containing the negation of the necessary condition can be used for the analysis of sufficiency for Y.

CG also take issue with efforts of advancing the methodological debate by formulating good practices, though it is not clear whether this applies only to our efforts or to any such attempts in general. We leave the evaluation of the tone of our suggestions to the readers. Throughout all our best practice writings, however, we always stress that guidelines should not and cannot turn Qualitative Comparative Analysis into a point-and-click method in which users are exempted from thinking for themselves.

CG deserve credit, though, for reminding researchers that contradictory simplifying assumptions can occur not only in the analysis of sufficiency but also necessity when analyzing outcomes Y and ∼Y.

Toward the end of their article, CG seem to endorse coincidence analysis (CNA; see Baumgartner 2008, 2009) not being based on the QuineMcClusky algorithm (QM), as a solution to the problem of making untenable assumptions. “Without QM, the rules offered by ESA may no longer be required.” To this, CG omit to add what else would change if CNA replaced QM. For one, there would be no different solution formulas—conservative, intermediate, most parsimonious—but simply one: the most parsimonious one. Furthermore, this solution would be produced without the researcher engaging into counterfactual reasoning, precisely the practice CG so warm-heartedly (and correctly, in our eyes) advise for.