Qualitative Comparative Analysis in Critical Perspective

2.1. Using QCA

We briefly describe QCA below. (For didactic explications see, variously, Berg-Schlosser et al. 2009; Ragin 2000, 2008a, 2008b, 2009; Rihoux and Grimm 2006; Rihoux and Ragin 2009). QCA researchers seek to determine the causal recipes that produce the occurrence (e.g., Y = 1) and nonoccurrence (e.g., Y = 0) of an outcome; indeed, they investigate occurrence and nonoccurrence separately to assess asymmetric causality, which is understood to mean that the opposite of the cause of Y = 1 (or Y = high) is not necessarily a cause of Y = 0 (or Y = low). Causal recipes are composed of exact values of selected variables; for example, if dichotomous variables A, B, and C might cause dichotomous Y = 1, one possible causal recipe is A = 1, B = 0, C = 1, while A = 1, B = 1, C = (0 or 1) is another. Different causal recipes may cause the same outcome, reflecting the presence of equifinality.

Analysts may use crisp-set QCA (csQCA) on binary variables that indicate membership in a crisp set with sharp boundaries. However, fuzzy-set QCA (fsQCA) can use continuous variables recoded into ordered categories of fuzzy-set membership. Whereas crisp sets do not allow degrees of membership, fuzzy sets do. For example, a fuzzy-set measure might classify nations as full members in the set of nuclear powers, mostly in that set, about equally inside and outside that set, and so on down to fully outside that set. In contrast, in crisp-set terms, nations either have nuclear weapons or do not.

2.2. Analyzing Data Using QCA

Although csQCA and fsQCA are similar, one key difference between them is that fsQCA requires analysts to calibrate their measures. Ragin (2008a:88–97) proposes two calibration strategies, with each mapping a variable onto the logistic distribution. The first strategy anchors key substantive points at theoretically important values and then arrays the remaining values along that continuum, after which an analyst uses the logistic distribution to transform the data into a scale that asymptotically approaches zero and one. A second strategy transforms the variable using the logistic distribution directly, without theoretically based anchors. Because Ragin (2008a) highlights theory, the first method is preferred. Either way, some values short of the asymptotes are selected to mark complete membership and nonmembership in the set. ³

Once calibration is accomplished, csQCA and fsQCA analysts may follow the same procedures. Analysts first construct a truth table, a table containing a row for each permutation of the variables (e.g., one row for X₁ = 0, X₂ = 0, X₃ = 0, one row for X₁ = 1, X₂ = 0, X₃ = 0, and so on; an fsQCA truth table row reflects a corner in multidimensional space [Ragin 2008a:126–33]). Each row also indicates the frequency of cases matching the permutation, and the proportion of cases in the row for which the outcome is observed (e.g., Pr[Y = 1]).

Analysts next remove from the table all configurations that fall below a chosen frequency-of-cases threshold and then code as causal for the outcome all configurations that exceed a chosen consistency threshold, defined as the proportion of cases for which the outcome (e.g., Y = 1) is observed. Analysts are advised to use a consistency threshold of .75 or higher (Ragin 2008a:143–44). On this basis, QCA produces a complex solution.

Some combinations of variable values may lack cases, meaning that some configurations that could provide useful analytic leverage may not be observed. For example, there may be no Marxist, Buddhist, European Union states. A QCA analyst may assign to nonexistent (i.e., counterfactual) configurations the outcome the analyst believes the configuration would produce if the configuration were to exist. Once the analyst makes these assignments, a more parsimonious intermediate solution based on invoking those assignments may be produced. A still more parsimonious solution is produced by QCA’s mechanical imposition of additional simplifying assignments to any remaining nonexistent configurations. The complex solution does not depend on the assignment of such counterfactual configurations. Analysts are encouraged to highlight the type of solution most appropriate for their research (Schneider and Wagemann 2010:408). Because of length, in what follows we usually highlight the complex solution, for it reflects fewer a priori researcher assumptions. Nevertheless, we often present multiple solutions to help readers evaluate our results, and sometimes we discuss more than just the complex solution.

QCA also reports the level of consistency and coverage of the solution and of each causal recipe. In crisp-set analyses, consistency reflects the proportion of cases for which the outcome occurs and coverage reflects the percentage of all outcome occurrences associated with the causal recipe(s). In fuzzy-set analyses, the mathematics differs, but consistency and coverage are conceptually analogous in crisp-set and fuzzy-set analyses. ⁴

Prior to fsQCA, QCA analysts needed to transform all data into dichotomies. Ragin (2008a:141) now recommends fsQCA unless the data possess a strong nominal character.

2.3. Stated Epistemological Bases of QCA

QCA is built on the epistemological base of John Stuart Mill’s methods (Ragin 1987); two key such methods are conveyed in Table 1.

OPEN IN VIEWER

Table 1.

Selected Mill’s Methods

Name	Definition	Example
Method of agreement	“If two or more instances of the phenomenon under investigation have only one circumstance in common, the circumstance in which alone all the instances agree, is the cause (or effect) of the given phenomenon” (Mill [1843] 1906:255).	If two nations have interethnic violence, and the only circumstance they share is the same language, then this circumstance is the cause or the effect of interethnic violence.
Method of difference	“If an instance in which the phenomenon under investigation occurs, and an instance in which it does not occur, have every circumstance in common save one, that one occurring only in the former; the circumstance in which alone the two instances differ, is the effect, or the cause, or an indispensable part of the cause, of the phenomenon” (Mill [1843] 1906:256).	If two nations are under study, and they differ in that revolution occurs in one instance but not in the other, the only candidates for causing the revolution are those factors that differ between the two nations.

Describing QCA as “set theoretic,” conceiving of variables as indicators of set membership and of relations between (groups of) variables as relations between sets, QCA analysts attempt to discern sufficient causes, necessary causes, and combinations thereof (see Table 2), where necessary causes are in all causal recipes but sufficient causes are not. Thus, QCA is seen as revealing explicit causal connections between sets of conditions and outcomes. ⁵

OPEN IN VIEWER

Table 2.

Four Forms of Causation

Name	Definition	Example
Necessary	A cause that is always present in the case of the outcome. However, sometimes the cause will be present but the outcome will not be observed.	If external war is a necessary cause of revolution, then all cases of revolution will occur in states engaged in external war. However, some states engaged in external war may not experience revolutions.
Sufficient	A cause that always produces the outcome, though other causes may also produce the outcome. Sometimes the outcome will be present but the cause will be absent.	If state repression is a sufficient cause of austerity protest, then the outcome (austerity protest) must arise in every instance in which the sufficient cause (state repression) occurs. Still, it would be possible for an austerity protest to occur even without state repression
INUS	A cause X that is an insufficient but necessary part of condition Z, where Z is a sufficient cause of Y.	If failed harvest is a necessary part of sufficient condition Z, composed of failed harvest and civil war, and Z is a sufficient cause of revolution, then all cases with failed harvests and civil war will cause revolution, revolution will also be caused without sufficient condition Z, and civil war without failed harvests but some other factor may also cause revolution, and revolution may also occur in the absence of civil war.
SUIN	A cause Q that is a sufficient but unnecessary part of P, where P is a necessary cause of Y.	If failed harvest is a sufficient but unnecessary part of condition P, composed of failed harvest and civil war, and the condition is a necessary cause of revolution, then all revolutions will occur in states with failed harvests regardless of the existence or nonexistence of civil war, some states will have revolutions despite successful harvests such that failed harvest/civil war is an unnecessary case of revolution.

Another key claim, described as a fundamental difference between QCA and quantitative correlation analyses (e.g., Ragin 2008a:102, 139), is that QCA allows for asymmetric relationships between causal conditions and outcomes. Illustratively, statistical researchers are said to assess the causal role of a dichotomous X on a dichotomous Y by using the odds ratio of the two-by-two cross-tabulation table. The odds ratio compares the distributions of Y|X = 1 and Y|X = 0. If they differ, then X and Y are associated and a causal relationship between X and Y is possible. In studying these distributions, statistical analysts consider all cells of the table.

In contrast, Ragin (2008a) contends that some table cells are irrelevant to whether X causes Y, arguing that analysts interested in causal conditions shared by an outcome will consider cells in which Y = 1 (cells [0,1] and [1,1]) and that analysts interested in outcomes produced for cases sharing the same causal condition would consider cells in which X = 1 (cells [1,0] and [1,1]). The cell signifying absence of both cause and outcome (X = 0, Y = 0) “is not directly relevant to the assessment of either of the two types of explicit connections” (Ragin 2008a:22). In this way, QCA analysts find asymmetric causal relations other analysts cannot.

For proponents, QCA is a set-theoretic method that embraces asymmetric relationships and detects causal links. The contention is that new, perhaps deeper insight can be gained by using such methods (Fujita 2009). Indeed, Ragin (2008a:190–212) contends that QCA can be applied to very large data sets and, addressing Liao (2001), illustrates the utility of QCA by reanalyzing more than 12,000 cases from the 1979 National Longitudinal Survey of Youth (NLSY) to reassess Herrnstein and Murray’s (1994)The Bell Curve.

3.1. Determinism and Stochasticism

Much of the debate about the utility of QCA pivots around one key epistemological disagreement: Are deterministic analyses useful or not? ⁶ This central issue is debated directly and is evident in two additional issues: (1) whether QCA uses Mill’s methods and to what effect and (2) whether QCA is undone by measurement error.

Mahoney (2008) embraces both determinism and QCA as a deterministic technique, arguing that case-oriented researchers use several variations on necessary and sufficient causation (see again Table 2). He then writes that adopting any of these types of causality is

to assume that the world of human beings is deterministic. This assumption does not of course mean that a given researcher will be successful at identifying the causes of any outcome. Nor does this assumption mean that the researcher will achieve valid measurement or carry out good research. The assumption of an ontologically deterministic world in no way implies that researchers will successfully analyze causal processes in this world. But it does mean that randomness and chance appear only because of limitations in theories, models, measurement, and data.

The only alternative to ontological determinism is to assume that, at least in part, “things just happen”; that is, to assume that truly stochastic factors—whatever those may be—randomly produce outcomes. (p. 420, endnotes and citations suppressed)

Mahoney (2000) identifies necessary and sufficient causation as deterministic, and he indicates (Mahoney 2008) that QCA, as a means to study necessary and sufficient causes, is deterministic.

An additional claim is that case-oriented researchers are attempting to understand a specific case or set of cases (Mahoney and Goertz 2006). In that view, the French Revolution—or one roll of the dice—is a deterministic phenomenon in that the probability of the outcome we observe is 1, and the probability of all other outcomes is 0 (Mahoney 2008:415–16). Seen in this way, case-oriented researchers seek a comprehensive explanation, that is, an explanation that completely accounts for what is observed. This aim is contrasted with efforts to estimate the effects of variables in populations (Mahoney 2008:420); the claim is that in such studies, multiple cases are deployed to estimate population average effects.

Mahoney embraced determinism, but some analysts, seeing determinism as undesirable for many reasons, prefer stochasticism. First, determinism implies that analysts could offer a nontrivial explanation that would explain every case perfectly (Lieberson 1994, 2004). ⁷ Second, an effort to establish full explanations easily leads analysts to capitalize on chance data patterns, that is, to overfit the data (Lieberson 2004). Findings thus produced are unlikely to be replicated. Third, if measurement error exists, analysts will misidentify causes (Goldthorpe 2000; Lieberson 1991, 1994). Thus, the only way to use QCA is to assume perfect measurement.

Ragin (2000, 2008a), attempting to strike a middle position, contends that fsQCA provides a stochastic tool and noted that “common data and evidence problems (error, randomness) provide a very strong motivation to employ analytic techniques that make some use of probability theory, especially techniques that address the problem of drawing inferences from imperfect evidence” (Ragin 2000:109). However, Ragin (2000:15) also asserts the “centrality of the analysis of . . . necessary and sufficient causation to social scientific inquiry,” a contention that seems to require a deterministic epistemology. Ragin (2000, 2008a) attempts to resolve the two positions by referencing the concepts of almost necessary cause and almost sufficient cause.

Clearly, different analysts have different postures on determinism and stochasticism. Thus, when Lieberson (2004) states that QCA is deterministic, he regards the critique as conclusive, but Mahoney, unfazed by the criticism because he embraces QCA as a deterministic method that matches his ontological commitments, sees QCA as a breakthrough.

Two other issues follow from the division between determinism and stochasticism. QCA proponents view QCA as a proper implementation or extension of Mill’s methods (e.g., Mahoney 2008; Ragin 2008a; Wagemann and Schneider 2010). Others see QCA as flawed to the extent it implements Mill’s methods in social analysis (e.g., Lieberson 1991).

Critics have claimed that with no allowance for measurement error, QCA may easily produce incorrect results (e.g., Goldthorpe 2000; Lieberson 1991). Ragin (2000), however, introduces probabilistic criteria, possibly muting this threat.

3.2. Other Central Aspects of Debate

The above discussion references central points of contention, but others exist as well. Were it not for the mega-issues noted above, these additional points of contention would themselves be major foci for analytic attention. Hence, these smaller (but still large) issues demand mention as well.

QCA analysts argue that to assess asymmetric causation, one must separately analyze Y = 1 and Y = 0 (or Y = high and Y = low) (De Meur and Rihoux 2002) and ignore some combinations of the putative causal variable and the outcome (Ragin 2008a:20–23). These counsels run counter to common claims of statistical analysts (e.g., Grofman and Schneider 2009:669).

Another argument is that QCA searches for invariant causal configurations, which creates doubt concerning the causal status of interaction effects for, by definition, interaction effects imply that causal configurations are not invariant (Lieberson 1991, 1994). In response, Savolainen (1994) contends that interactions are incredibly flexible, and the only way to sustain Lieberson’s critique is to deny that complexity. More pointedly, Vaisey (2007) claims that there is a major difference between interactions and complex conjunctural arrangements, and QCA allows assessment of the latter.

Another argument is that QCA is no advance over statistical modeling because astute statistical modelers already follow QCA advice to attend to context, base measurement on theory, and more (Achen 2005; Seawright 2005). QCA supporters respond by contending that QCA is meant to supplement, not supplant, statistical modeling (Ragin and Rihoux 2004).

Another argument is that QCA requires restrictive or contradictory assumptions for omitted variables and a functional form assumption that is as necessary as the functional form assumptions statistical analysts must make (Seawright 2005). Yet QCA analysts point to calibration in fsQCA as a response to the functional form issue (Ragin 2008a).

Another argument is that QCA abandons useful features of statistics such as standard errors (Achen 2005:30; Seawright 2005:14), prompting Seawright to suggest that the insights of QCA—most notably interest in complex conjunctures and nonlinear functional forms—be integrated into other methodological frameworks, including both statistical analysis and other methods of case-study research. Yet Ragin (e.g., 2000:107–18) shows that one may use statistical testing in QCA, undercutting this claim

In what follows we address these contentious issues.

6.1. Study 1 Results

We describe Table 4 in detail, for its features are common to many of the later tables. The first column identifies the complexity of the solution. Next, the “R” column indicates how many truth-table rows were read for the analysis (Ragin 2008b:68). In the next column, each line in the cell is a causal recipe. The variable name preceded by ~ indicates that the absence of the condition is part of the causal recipe; the variable name without ~ indicates that the presence of the condition is part of the causal recipe, while an asterisk indicates “and” and thus joins two causal factors. The table’s final column contains the coverage and consistency statistics.

OPEN IN VIEWER

Table 4.

Study 1, QCA Analysis with Equifinal Causes (n = 40)

	Outcome Y = 1			Outcome Y = 0
Solution	R	Causal Recipes	Coverage, Consistency	R	Causal Recipes	Coverage, Consistency
Complex	14	x3*x4 x1*x2*∼x4	.8750, 1.0000	14	∼x2*∼x4 ∼x1*∼x4 ∼x1*∼x2*∼x3	1.0000, 1.0000
Intermediate	6	x3*x4 x1*x2*∼x4	.8750, 1.0000	12	∼x4*∼x1 ∼x4*∼x2 ∼x3*∼x2*∼x1	1.0000, 1.0000
Parsimonious	14	x3*x4 x1*x2*∼x4	.8750, 1.0000	14	∼x2*∼x4 ∼x1*∼x4 ∼x2*∼x3 ∼x1*∼x3	1.0000, 1.0000

Considering the analysis of Y = 1, one causal recipe from the complex solution is “X₃ and X₄”; given the coding of the variables, this can be translated as X₃ = X₄ = 1. Another causal recipe is “X₁ and X₂ and not X₄,” that is, X₁ = X₂ = 1 while X₄ = 0. None of the factors are necessary causes of Y = 1 or Y = 0, for the same value of a given X does not appear in all causal recipes. The complex solution is identical to the intermediate and parsimonious solutions.

Although csQCA identified one of the two causal recipes (X₃ = X₄ = 1) correctly, it incorrectly identified the other, claiming that the value of X₄ was relevant for whether X₁ = X₂ = 1 is causal for Y. We know that this is incorrect, by construction. This failure is especially concerning, because this is exactly the case for which QCA is advertised: equifinality, deterministic data, and analysts’ knowing the set of causal factors (on the basis of theory and/or extensive case-specific study).

Following advised procedure, we then analyze Y = 0. Given how we constructed Y = 1, the fundamental causal recipe of Y = 0 is X₁ × X₂ = 0 and X₃ × X₄ = 0, which occurs if X₁ = 0 and X₃ or X₄ = 0 or if X₂ = 0 and X₃ or X₄ = 0. Results for the Y = 0 analysis are not clearly opposite those obtained for Y = 1. Thus, QCA appears to find asymmetry.

Indeed, the asymmetry is intriguing, for although the complex and intermediate solutions to Y = 0 and Y = 1 are wrong, the parsimonious solution for Y = 1 is wrong but for Y = 0 is correct. The half-right, half-wrong pattern of parsimonious results raises serious questions of QCA.

The analysis in Table 4 uses all causal factors and only causal factors. Analysts sometimes mistakenly omit a causal factor or include a noncausal factor. The costs of omitting a cause are well known for many other methods (e.g., Goldberger 1991:189–91), and QCA should not be held to the untenable standard of producing correct causal conclusions in such a case.

However, if analysts cannot use methods to reveal that factors they believe matter do not matter, empirical analysis of observational data is of little value. Statistical and qualitative analysts routinely appear able to eliminate some factors as causal (e.g., Hauser and Anderson 1991; Vaughan 1996). It is fair to require QCA to succeed at this task as well. To proceed, we added a dichotomous variable, Z₁, to the analysis. Z, the sum of a random component, X₁, X₂, X₃, and X₄, is dichotomized at .49 to produce Z₁; Z₁ is correlated at .198, .156, .383, .088, and .245 with X₁ to X₄ and Y, respectively. Being constructed later than Y and having the lowest correlation with Y, Z₁ has no causal role in Y and is not a viable candidate for causal priority.

The short answer from Table 5 is that QCA does not eliminate the noncausal factor from the causal recipes. In fact, not only does QCA fail to eliminate the noncausal factor, it identifies eight four-way interactions as causal recipes in the complex solution for Y = 1, five of which involve Z₁, while half of the four complex solutions for Y = 0 also involve Z₁. These results are produced even though X₁ through X₄ deterministically cause Y. Far from identifying Z₁ as noncausal, QCA deeply embeds the noncausal Z₁ into the causal story.

OPEN IN VIEWER

Table 5.

Study 1, QCA Analysis with Equifinal Causes and Noncausal Factor (n = 40)

	Outcome Y = 1			Outcome Y = 0
Solution	R	Causal Recipes	Coverage, Consistency	R	Causal Recipes	Coverage, Consistency
Complex	22	∼x1*x3*x4*∼z1 x1*x2*x3*∼x4 x1*x2*∼x3*x4 x1*x3*x4*z1 x1*x2*∼x4*∼z1 x1*x2*∼x3*∼z1 ∼x1*x2*x3*x4 x2*x3*x4*z1	.9375, 1.0000	22	∼x1*∼x4*∼z1 x1*∼x2*∼x4 ∼x1*x3*∼x4 ∼x1*∼x2*∼x3*∼z1	1.0000, 1.0000
Intermediate	20	∼z1*x4*x3*∼x1 x4*∼x3*x2*x1 ∼x4*x3*x2*x1 z1*x4*x3*x1 ∼z1*∼x3*x2*x1 ∼z1*∼x4*x2*x1 x4*x3*x2*∼x1 z1*x4*x3*x2	.9375, 1.0000	16	∼z1*∼x4*∼x1 ∼x4*x3*∼x1 ∼x4*∼x2*x1 ∼z1*∼x3*∼x2*∼x1	1.0000, 1.0000
Parsimonious	22	x4*z1 x1*x2*∼x4 ∼x1*x3*x4 x1*x2*∼x3 x1*∼x3*x4 x2*∼x3*x4	.9375, 1.0000	22	∼x2*∼x4 ∼x1*∼x4 ∼x2*∼x3 ∼x1*∼x3	1.000, 1.000

Note, however, that the parsimonious solution for Y = 0 is correct, whereas its counterpart parsimonious solution for Y = 1 remains incorrect. This further deepens the questions about QCA.

Faced with these results, one might wonder whether the data actually follow the pattern we stated. One well-known impediment to providing that confirmation is that one cannot use the usual logistic regression model estimated via maximum likelihood to analyze data if a pattern of values invariably produces an outcome, because coefficients march toward infinity in that case (Hosmer and Lemeshow 1989). However, there are other models one can use for such data.

We use Firth’s (1993) penalized maximum likelihood (PML) approach to logistic regression estimation as implemented by Heinze and Schemper (2002) in Stata.

Model 1 in Table 6 identified three of the four variables as causal. Technically, this specification is incorrect in that we did not construct Y on the basis of single variables. Thus, we should expect the model to fail. Still, the model does pick out three of the four elements of the causal story. However, the analogous QCA analysis, despite the method’s purported ability to identify complex causal recipes on the basis of specification of the separate variables, missed one of two causal configurations while identifying some noncausal factors as causal.

OPEN IN VIEWER

Table 6.

Penalized Maximum Likelihood Logistic Regression Analysis of Study 1 Data of Y = 1 and Y = 0

	Model 1		Model 2		Model 3		Model 4		Model 5
Outcome →	Y = 1	Y = 0	Y = 1	Y = 0	Y = 1	Y = 0	Y = 1	Y = 0	Y = 1	Y = 0
Constant	−8.735* (3.427)	8.735* (3.427)	−3.891* (1.429)	3.891* (1.429)	−3.545* (1.788)	3.545* (1.788)	−3.888* (1.429)	3.888* (1.429)	−1.490* (.472)	1.490* (.472)
X 1	5.367* (2.332)	−5.367* (2.332)
X 2	4.392* (2.135)	−4.392* (2.135)
X 3	1.841 (1.733)	−1.841 (1.733)
X 4	7.446 * (3.251)	−7.446* (3.251)
X1×X2			6.598* (2.043)	−6.598* (2.043)	6.028* (1.843)	−6.028* (1.843)	6.286* (2.055)	−6.286* (2.055)
X3×X4			6.455* (2.048)	−6.455* (2.048)	5.889* (1.860)	−5.889* (1.860)	6.449* (2.050)	−6.449* (2.050)	4.033* (1.545)	−4.033* (1.545)
Z 1					−.043 (1.933)	.043 (1.933)
X1×X2×X4							−.793 (2.141)	.793 (2.141)	3.078 (1.627)	−3.078 (1.627)

Note: Values in parentheses are standard errors.

*

Coefficient is discernibly different from zero at or below α = .05.

Model 2 specifies the correct causal model and finds that both interactions are discernibly different from zero; model 3, which adds a noncausal variable, correctly identifies the causal factors. Table 6 also contains a model with X₁ × X₂, X₃ × X₄, and X₁ × X₂ × X₄, which tests (1) whether QCA finds a relationship we overlooked and (2) whether this three-way interaction dominates X₁ × X₂, as QCA implies. We find, however, that coefficients for X₁ × X₂ and X₃ × X₄ are statistically significant, while the X₁ × X₂ × X₄ coefficient is not. The result matches the construction of Y.

Model 5 specifies the result QCA identified, including only the three-way interaction and X₃ × X₄. In this specification, X₃ × X₄ is still statistically significant, while the X₁ × X₂ × X₄ coefficient is indiscernible from zero. Thus, Table 6 indicates that QCA does not find the correct causal process and that PML logistic regression offers a statistical means for studying such data. ⁹ These results, coupled with our knowledge of the construction of Y, make it clear that QCA is wrong.

Finally, note that the Y = 1 and Y = 0 PML results are equivalent, as only the signs differ. Study 2 explores how QCA finds asymmetric causes, but first we discuss key implications of study 1.

6.2. Discussion of Study 1 Results

6.2.1. Deterministic epistemology

Study 1 uses nonparametric data; specifies deterministic, sufficient, causes; eschews sampling; and imposes equifinality. QCA proponents have identified these conditions as important ontological commitments (e.g., Mahoney 2010; Ragin 2008a; Ragin and Rihoux 2004); thus, study 1 tests the ability of QCA under analytic “best conditions,” as described by QCA proponents. If real-world analysts meet the conditions above and possess data on the full set of causal factors and have all the relevant cases, then study 1 mimics a real-world analysis that completely matches QCA proponents’ conditions, plus the rare state of having full knowledge. Even so, QCA rarely selects the correct causal recipes, raising serious questions about its effectiveness even when real-world conditions match stated QCA assumptions.

At this juncture, the most important issue to address is the foundational assumption of determinism. To proceed, we first distinguish research method and theoretical perspective. Our question is whether the method and/or applied epistemology is or can be deterministic. We argue that sociologists, like physicists, are free to prefer deterministic theories or stochastic theories. For all we know, the social universe may operate with the sociological analog of relativity, such that all proceeds inexorably from origin to a fated end along a path completely determined by first principles, even if humans are still in the process of discovering those principles. This is the Einsteinian position on matter and energy, hence his doubt that God plays dice with the cosmos (Einstein [1926] 2005). Alternatively, the social universe may operate according to the equivalent of quantum theory, understood as constituted by an irreducibly probabilistic process and thus an impossibility of fully perfect prediction (Gribbin 1984:66). Because the true state of matters is debatable, substantive theories may be either deterministic or probabilistic.

A deterministic ontology is one position; a deterministic epistemology is another. Mahoney (2008) embraces determinism both ontologically and epistemologically. Alas, Table 7 indicates the risky gamble a deterministic epistemology involves. One row reflects a stochastic epistemology, and the other reflects a deterministic epistemology. One column reflects a stochastic ontological reality, and the other reflects a deterministic one. All is well for either epistemology if the world is consistent with its assumption (as shown by the unbiased results on the main diagonal). However, because an epistemological stochasticist estimates an error term, adopting this position in a deterministic world simply produces error estimates of zero. Thus, epistemological stochasticism can produce unbiased results even when its assumption is ontologically incorrect. ¹⁰

OPEN IN VIEWER

Table 7.

Expected Bias Using Stochastic and Deterministic Epistemologies in Stochastic and Deterministic Worlds

Strategic Epistemology	True Ontological Condition
	|ε| > 0, Stochastic	ε = 0, Deterministic
|ε*| estimated as ≥0, stochastic	E ( | ε ^ − ε * | ) = 0 → results unbiased	E ( | ε ^ − ε * | ) = 0 → results unbiased
ε* fixed at 0, deterministic	E ( | ε ^ − ε ∗ | ) > 0 → results biased	E ( | ε ^ − ε * | ) = 0 → results unbiased

Note:ε = true error; ε* = assumed error; ε ^ = error in estimate.

In contrast, a deterministic epistemology activated in a stochastic world will produce biased estimates because, making no space for error, results will necessarily misallocate any nonzero error, biasing analysis results. Because we are and may forever be ignorant of the true state of the universe, the pragmatic response is an epistemological presumption of stochasticism. Determinist epistemologies entail an unnecessary gamble, a gamble determinists can never confirm they have won nor even discover how much debt the analysis incurs owing to the wager.

6.3. Implications of Study 1 Results for Remaining Analyses

Because we have shown that QCA is not deterministic, we no longer construct data using deterministic methods. Relaxing this constraint simplifies data construction and increases the flexibility of tests we can apply.

7.1. Study 2 Methods

In Study 1, 10 of 12 QCA solutions were incorrect. Curiously, each correct solution for Y = 0 was paired with an incorrect solution for Y = 1, a possibility given QCA strategy for studying asymmetric causality. Study 2 interrogates the QCA strategy for studying asymmetric causality.

The advised QCA procedure for assessing asymmetric causality is to analyze Y = 1 and Y = 0 (or Y = high and Y = low) separately (De Meur and Rihoux 2002; Ragin 2008a:137–38). We followed QCA advice. Our aim was to see whether QCA will find asymmetry even with symmetric causes; to ensure symmetry, we built the data using a parametric model. ¹²

7.2. Study 2 Results

We present the complex and parsimonious solutions, highlighting the complex solution (see Table 8). Symmetry might be difficult to discern, but one sign would entail a recipe of, for example, X₁ = 1, X₂ = 0 for Y = 1 and X₁ = 0, X₂ = 1 for Y = 0. However, we find no clear evidence of symmetry, finding four causal recipes of Y = 1 and three causal recipes of Y = 0 across the solutions.

OPEN IN VIEWER

Table 8.

Study 2, Results for Analyses of Y = 1 and Y = 0 for CG2 Data, Consistency Threshold of .8 (n = 20)

	R	Causal Recipes	Coverage, Consistency	R	Causal Recipes	Coverage, Consistency
	Y = 1			Y = 0
Solution	One-case Threshold			One-case Threshold
Complex	13	x2*~x4~x1*x2x1*~x3*~x4	.5385, 1.0000	13	x1*~x2*x3~x1*~x2*~x3*x4	.4286, 1.0000
Parsimonious	13	x1*~x3x2*~x4~x1*x2	.5385, 1.0000	13	x1*~x2*x3~x2*~x3*x4	.4286, 1.0000

One might contend that this particular data set is rife with asymmetries, which would explain our inability to locate any evidence of symmetric causality. The problem, however, is that we know the data were constructed using a symmetric process. ¹³ Thus, any claim of asymmetry is incorrect, raising major questions about the method’s ability to discern asymmetric causality.

Closer inspection reveals that this strategy of analyzing both Y = 1 and Y = 0 is the root of the problem. If one follows this strategy, the two analyses will divide all causal configurations into three categories, with λ and τ as consistency thresholds:

Analysts are advised that the minimum consistency threshold, expressed as a percentage, is 75 (Ragin 2008a). Using percentages, if one sets λ + τ > 100 (as counseled), then category C may contain one or more configurations. Because analyses of Y = 1 and Y = 0 both set aside category C, the existence of this category masks any symmetry embedded in the causal process.

To see the implications of this fact, we reanalyzed the study 2 data using the truth table in Table 9. The last column lists the number of cases that fall into each causal configuration. Rows E, J, and L have no observations and thus are eliminated for failing to meet the frequency threshold of 1. The Y = 1 column indicates the proportion of cases in the causal configuration for which Y = 1; the Y = 0 column reports the analogous proportion for Y = 0.

OPEN IN VIEWER

Table 9.

Study 2 Truth Table for Analyses of Y = 1 and Y = 0, CG2 (n = 20)

Causal Configuration	X 1	X 2	X 3	X 4	Y = 1	Y = 0	Count
A	0	0	0	0	.67	.33	3
B	1	0	0	0	1.00	.00	1
C	0	1	0	0	1.00	.00	1
D	1	1	0	0	1.00	.00	1
E	0	0	1	0	—	—	0
F	1	0	1	0	.00	1.00	1
G	0	1	1	0	1.00	.00	1
H	1	1	1	0	1.00	.00	1
I	0	0	0	1	.00	1.00	1
J	1	0	0	1	—	—	0
K	0	1	0	1	1.00	.00	1
L	1	1	0	1	—	—	0
M	0	0	1	1	.33	.67	3
N	1	0	1	1	.00	1.00	1
O	0	1	1	1	1.00	.00	1
P	1	1	1	1	.75	.25	4

The recommended analytic approach appears in the first two panels of Table 10. Columns 5 and 6 indicate which causal configurations are specified to produce outcome Y = 1 or Y = 0 and the causal configurations implicitly claimed to not produce the outcome. Thus, an analyst investigating Y = 1 specifies seven causal configurations—B, C, D, G, H, K, and O—as causal. In doing so, the analyst implicitly claims that the remaining configurations do not produce Y = 1. In contrast, an analyst studying Y = 0, using the recommended procedure, specifies only three configurations—F, I, and N—as causal. By implication, therefore, this analyst specifies that the remaining configurations do not produce the outcome Y = 0. Note that three causal configurations—A, M, and P—are never included as possible determinants of Y. We contend that it is the neglect of AMP that is feeding back to produce distortion that generates a report of asymmetric causality.

OPEN IN VIEWER

Table 10.

Study 2, Results for Analyses of Asymmetric Causality for CG2 Using Alternative Strategies of Configuration Assignment, One-case Threshold (n = 20)

Outcome/Solution	R	Causal Recipes	Coverage, Consistency	Specified Expectation of Causal Configurations		Consistency Threshold
Recommended analytic approach
Panel 1, Y = 1				Y = 1	Y≠ 1
Complex	13	x2*~x4~x1*x2x1*~x3 *~x4	.5385, 1.0000	B, C, D, G, H, K, O	A, F, I, M, N, P	.75
Parsimonious	13	x1*~x3x2*~x4~x1*x2	.5385, 1.0000	B, C, D, G, H, K, O	A, F, I, M, N, P	.75
Panel 2, Y = 0				Y = 0	Y≠ 0
Complex	13	x1*~x2*x3~x1*~x2*~x3*x4	.4286, 1.0000	F, I, N	A, B, C, D, G, H, K, M, O, P	.75
Parsimonious	13	x1*~x2*x3~x2*~x3*x4	.4286, 1.0000	F, I, N	A, B, C, D, G, H, K, M, O, P	.75
Mutually exclusive specification of configurations
Panel 3, Y = 0				Y≠ 0 →Y = 1	Y = 0
Complex	13	∼x1*~x2*~x3x1*~x2*x3x1*x3*x4~x1*~x2*x4~x2*x3*x4	1.000, .5385	B, C, D, G, H, K, O	A, F, I, M, N, P	.25
Parsimonious	13	∼x1*~x2~x2*~x3x1*x4	1.000, .5385	B, C, D, G, H, K, O	A, F, I, M, N, P	.25

Note: Causal configuration table: A = 0000, B = 1000, C = 0100, D = 1100, E = 0010, F = 1010, G = 0110, H = 1110, I = 0001, J = 1001, K = 0101, L = 1101, M = 0011, N = 1011, O = 0111, P = 1111. E, J, and L are always counterfactual configurations, with no observed cases.

If we presume that asymmetric causation is possible, and thus causes of Y = 1 do not constrain causes of Y = 0, then if we specify that AMP does not cause Y = 1, what justifies the switch to presuming that AMP is noncausal when we move to the analysis of Y = 0? Although this assumption is defensible, given the claim that AMP is not causal for Y = 1 and a belief in a possibility of asymmetric causation as defined in QCA, should one not at least specify a provisional Y = 0 analysis that explores whether Y = 0 is produced by AMP? Panel 3 specifies just such an exploratory analysis. Implementing this specification as the companion of the Y = 1 analysis violates recommended QCA procedure but fulfills a deeper principle of analytic completeness by exploring the possible causal implications of all observed causal configurations. ¹⁴ Thus, no configuration is ruled out of contributing to some outcome. ¹⁵

Summary statistics for panel 3 are the opposite of those obtained in panel 1, suggesting that explanations for Y = 1 and Y = 0 are essentially the same. Yet causal recipes for Y = 1 and Y = 0 are different and not clearly opposite. There are two signs of symmetry, but they are opaque. First, the complex solution contains x1*~x3*~x4 for Y = 1 and x1*x3*x4 for Y = 0. Thus, once we set aside x1, the role of x3*x4 is opposite for the two outcomes. Second, for Y = 1, the parsimonious solution contains ~x1*x2, and for Y = 0, it contains ~x1*~x2. Thus, with the proviso that ~x1 is required, the role of x2 in producing Y = 1 is just opposite its role in producing Y = 0. But these two sets of recipes offer a minimal sign of the symmetry embedded deep in the results.

To excavate that symmetry Table 11 traces the implications of the causal recipes from the complex solutions of panels 1 and 3 of Table 10. The first column contains causal recipes. In column 2, which translates the causal recipe into X₁, X₂, X₃, and X₄, zeros indicate nonmembership in the positionally identified set, ones signify membership, and dashes indicate the variable is not relevant for the causal recipe. When the factor is irrelevant, replacing the dash with either a zero or a one is equally correct. Column 3 provides just such a set of substitutions, expanding directly the implications of the recipes to reveal the actual causal configurations they summarize.

OPEN IN VIEWER

Table 11.

Study 2, Implications of Panel 2 Causal Recipes for Y = 1 and Y = 0, Complex Solution, CG2 Data

	Translation into X1, X2, X3, X4	Implications of Translation	Summary of the Disaggregation
Causal recipes for Y = 1
x2*~x4	— 1 — 0	0100 == C (Y = 1 dup 1)	7 nonredundant causal configurations of Y = 1 implied in the 3 causal recipes
		1100 == D (Y = 1 dup 2)
		0110 == G (Y = 1 dup 3)
		1110 == H
~x1*x2	0 1 ——	0100 == C (Y = 1 dup 1)
		0101 == K
		0110 == G (Y = 1 dup 3)
		0111 == O
x1*~x3*~x4	1 — 0 0	1000 == B
		1100 == D (Y = 1 dup 2)
Causal recipes for Y = 0
~x1*~x2*~x3	0 0 0 —	0000 == a	6 nonredundant causal configurations of Y = 0 implied in the 5 causal recipesImplications in boldface type were configurations implicated in the recommended analysis of Y = 0 (0001 was implied by the 4-way interaction identified in Table 10, panel 2)
		0001 == i (Y = 0 dup 1)
x1*~x2*x3	1 0 1 —	1010 == f
		1011 == n (Y = 0 dup 2)
x1*x3*x4	1 — 1 1	1011 == n (Y = 0 dup 2)
		1111 == p
~x1*~x2*x4	0 0 — 1	0001 == i (Y = 0 dup 1)
		0011 == m (Y = 0 dup 3)
~x2*x3*x4	— 0 1 1	0011 == m (Y = 0 dup 3)
		1011 == n (Y = 0 dup 2)
Conclusion 1			All 13 observed causal configurations are causal
Conclusion 2			There is no asymmetric content: the solution to Y = 1 implies the solution to Y = 0

Note: Causal configuration table: A = 0000, B = 1000, C = 0100, D = 1100, E = 0010, F = 1010, G = 0110, H = 1110, I = 0001, J = 1001, K = 0101, L = 1101, M = 0011, N = 1011, O = 0111, P = 1111. E, J, and L are always counterfactual configurations, with no observed cases.

Column 3 indicates that some implications of a causal recipe duplicate the implications of some other causal recipe. Thus, the three causal recipes for Y = 1 imply 10 causal configurations, but there really are only 7 nonredundant causal configurations.

The key conclusion is that when we compare the causal recipes for Y = 1 with those for Y = 0, it appears that asymmetry holds. However, Table 11 reveals that a deeper symmetry is hidden by the recipes: The causal pathways are such that knowing the causal recipes (or, at least, their implied causal configurations) for one outcome implies the causal recipes for the other. The final column of Table 11 reveals this to be the case: once we know the recipes for Y = 1, we need not analyze Y = 0, for its causal recipes are a perfectly symmetric set (save for the counterfactual configurations about which one, truly, can say nothing).

This conclusion becomes apparent, however, only through the analysis presented in panel 3 of Table 10. If one conducts the advised analyses and sets λ+τ > 100, as in panel 2 of Table 10, then symmetry will likely vanish. This is noted by the recipes, translation, and implications in boldface type in Table 11; these are the results obtained in panel 2 of Table 10. The reason those results are only a subset of those obtained in panel 3 is that in the panel 2 analysis, several causal configurations claimed not to cause Y = 1 are eliminated from consideration of possibly causing Y = 0 a priori. This is the approach Ragin (2008a) recommends, and if one follows that recommendation one will be able to obtain results that reflect burgeoning asymmetry. Yet it appears the asymmetry obtained is a spurious result of QCA.

7.3. Discussion of Study 2 Results

7.3.2. Defining asymmetric causation

In QCA, different causes of Y = 0 and Y = 1 (or Y = high and Y = low) reflect asymmetric causation, and QCA may find asymmetry because QCA can find necessary or sufficient causation (Ragin 2008a:17–20) using the ignore-a-cell approach (Ragin 2000).

We distinguish two kinds of asymmetry. In “functional form asymmetry,” the difference in Y when X = 1 is compared with X = 0 differs from the difference in Y when X = −1 is compared with X = 0; that is, the absolute value of the results of “moving” from some center point on X differ depending on the direction moved (0 moved to 1 vs. 0 moved to −1). Yet the absolute value of the results of “moving” between two points are equal regardless of the direction moved (0 moved to 1 vs. 1 moved to 0). Functional form asymmetry is a basic hypothesis statistical analysts may explore using higher order terms, asymmetric link functions (e.g., complementary log-log), or in other ways, such as piecewise regression and locally weighted scatterplot smoothing regression (Cleveland 1979).

We view necessity and sufficiency as referencing functional form asymmetry, reasoning as follows. First, assume dichotomous Y and trichotomous X, with X = 3 a necessary cause of Y = 1 (i.e., if X ≤ 2, Y = 0, but if X = 3, Y = 1). Thus, all of the change in Y occurs in the move from X = 2 to X = 3. Valid studies will show that

| Pr ( Y = 1 | X = 3 ) − Pr ( Y = 1 | X = 2 ) | ≠ | Pr ( Y = 1 | X = 2 ) − Pr ( Y = 1 | X = 1 ) | ,

but

| Pr ( Y = 1 | X = 3 ) − Pr ( Y = 1 | X = 2 ) | = | Pr ( Y = 1 | X = 2 ) − Pr ( Y = 1 | X = 3 ) | .

Sufficient causation could be similarly expressed. Equations (3.1) and (3.2) reflect functional form asymmetry; hence, necessary causation and sufficient causation reflect functional form asymmetry.

Another type of asymmetry refers to time, à la the English dominance example Lieberson (1985) offered in a chapter titled “Asymmetrical Forms of Causation.” The example considers the rising dominance of the English language over the course of the twentieth century, dominance that might be partly explained by the pre-1939 power of Britain and the postwar rise of the United States. Yet even if those causes were undone (e.g., the United States drowns in rising debt), English may remain dominant. Reversing the causes need not reverse their effects.

We term this kind of asymmetry “dynamic asymmetry” because it is inaccessible without cross-time data. Dynamic asymmetry is not a functional form issue. To clarify with an example, assume that an autumn survey of college sophomores ascertained the number of hours they studied a given week that term and the number of classes in which they enrolled. We find students with two classes studied 5 hours a week and students with four classes studied 20 hours a week. In a spring resurvey, we find that students who formerly had two classes and now have four classes now study 20 hours a week. We also find students who had four classes the previous term and now have two classes also study 20 hours a week. However, students with two (four) classes both terms studied 5 (20) hours a week in both terms. In dynamic asymmetry, the direction of change between two points matters. In the example, moving a student from taking two classes to four classes raises her studying by 15 hours a week, but moving a student from taking four classes to two leaves her studying time unchanged.

In equations, functional form asymmetry is such that

| E [ Y X = 0 ] − E [ Y X = 1 ] | ≠ | E [ Y X = 0 ] − E [ Y X = − 1 ] |

but

| Δ E [ Y X = 0 → Y X = 1 ] | = | Δ E [ Y X = 1 → Y X = 0 ] | ,

while dynamic asymmetry is such that

| Δ E [ Y X = 0 → Y X = 1 ] | ≠ | Δ E [ Y X = 1 → Y X = 0 ] | ,

¹⁶

where the arrow (→) signifies the direction of change in X.

Our example of dynamic asymmetry is a form of lock-in: once exposed to a need to study 20 hours, students allocate 20 hours to study even when they take fewer courses. Lock-in is a familiar form of dynamic asymmetry: English dominance, the QWERTY keyboard, driving on the right side of the road in the United States, substance addiction, and other social phenomena collected under the concept of habit variously defined (Camic 1986) are all examples. Notably, QCA’s technique for finding asymmetric causation does not access dynamic asymmetry and does not effectively access functional form asymmetry either.

7.3.3. The pivotal role of information

Lieberson’s (1985) example introduced information on event sequencing. Narrative analysis (Mahoney 1999) or process-tracing (George and Bennett 2005) also rely on additional often nonstandard information. Some approaches to investigating sequencing (Abbott and Hrycak 1990) or duration (Wu 2003) might also be useful. There may be other possibilities. The QCA strategy, in contrast, creates a serious identification problem (Manski 1995) and attempts to solve it by eliminating rather than supplementing the information available.

Consider a social phenomenon and its reverse, such that

Pr ( Y = 1 ) = f ( β 0 + β 1 X 1 j + β 2 X 2 j + β 3 X 3 j + ε j ) ,

and

Pr ( Y = 0 ) = f ( γ 0 + γ 1 X 1 j + γ 2 X 2 j + γ 3 X 3 j + δ j ) ,

with f a not necessarily parametric function and errors (ε, δ) equal to zero if determinism holds. Moving from expression (6) to (7) reverses the dependent variable, just as QCA advocates. However, as Grofman and Schneider (2009:669) assert, γ₀ = −β₀, γ₁ = −β₁, γ₂ = −β₂, and γ₃ = −β₃. The proof of this transmethodological conclusion can begin by assuming equations (8.1) and (8.2):

Pr ( Y i = 1 ) = e b x / ( 1 + e b x ) ,

and

Pr ( Y i = 0 ) = e gx / ( 1 + e gx ) .

Thus,

1 = Pr ( Y i = 1 ) + Pr ( Y i = 0 ) ,

which implies

1 = ( e bx / ( 1 + e bx ) ) + ( e gx / ( 1 + e gx ) ) .

Simplifying,

1 = ( e b x + 2 e b x + g x + e g x ) / [ ( 1 + e b x ) ( 1 + e g x ) ] .

Solving for b, most terms cancel, reducing to

b = − g .

An asymmetric link function for Pr(Y = 1) requires its opposite link function for Pr(Y = 0) to test whether β = −γ. The test would show the problem remains.

In short, to separately analyze the reverse of a dependent variable’s coding is to attempt to distinguish different absolute values of a variable’s effect on two perfectly collinear outcomes. Absent introduction of new information, this cannot be done.

New information is key. Assumptions are one kind of new information one can invoke to identify unidentified parameters. However, estimates identified via strong assumptions contain no content independent of the assumptions invoked. In specifying λ + τ > 100, analysts impose different, strong, and contradictory assumptions on analyses of Y = 1 and Y = 0, which can easily mask the fundamental symmetry of the causal structure of Y = 0 and Y = 1. Thus, the pairing in study 1 of a correct Y = 0 solution with an incorrect Y = 1 solution is revealed to flow from the invocation of contradictory assumptions that superficially resolve identification problems. The implication is that there is no asymmetric content in QCA results beyond that imposed by assumption.

7.4. Implications of Study 2 Results for Remaining Analyses

One important finding is that there is no value to analyzing both Y = 1 and Y = 0 (or Y = high and Y = low). As we continue, therefore, we will analyze only Y = 1 (or Y = high).

8.1. Does QCA Have a Consequential Unstated Assumption?

Study 1 data were designed to match the stated epistemological assumptions of QCA proponents: nonparametric, deterministic data reflecting sufficient, equifinal causes measured on the full population. Yet QCA failed. Study 3 explores whether QCA’s failure is explained by an unstated QCA assumption both Lieberson (1991) and Tilly (1984) posited for Mill’s methods.

In study 1 (and in reality), an outcome can be overdetermined; all X’s for a case can equal 1 even if subsets of X _k = 1 are sufficient for Y = 1. The social world is rife with such situations. There may be two sufficient reasons for “Joan’s” hiring: (1) her 4.0 grade point average and her physics degree on one hand and (2) her mother owning the company on the other. As another example, there may be three constellations of reasons why prospective students are admitted to elite colleges: (1) academic excellence (measured by grades and scholarly honors), (2) athletic excellence (measured by varsity letters) combined with academic adequacy (measured by the lack of grades below C+), and (3) parental wealth (measured by income, liquid assets, and illiquid assets) coupled with a high school diploma or equivalent. Some applicants (e.g., valedictorian varsity quarterbacks) will satisfy multiple causal constellations (e.g., both 1 and 2 above), making their admission overdetermined. A “no overdetermination” assumption denies such possibilities. Thus, we ask, Does QCA deny the possibility of overdetermination despite its common real-world occurrence?

To assess how QCA handles overdetermined data, we deleted the three overdetermined cases from the data used in study 1. Because QCA produced incorrect causal solutions when the overdetermined cases were present, if we obtain the correct causal solution after removing the overdetermined cases, it would appear that QCA implicitly assumes that overdetermination is not possible. Table 12 contains the results of our analysis.

OPEN IN VIEWER

Table 12.

Study 3, Overdetermined Cases Eliminated, Analysis of Y = 1, Crisp Group Data, Consistency Threshold of .8, One-case Threshold (n = 37)

Solution	R	Causal Recipes	Coverage, Consistency
Complex	13	x1*x2*~x4x1*x2*~x3~x2*x3*x4~x1*x3*x4	1.0000, 1.0000
Intermediate	13	x1*x2*~x4x1*x2*~x3~x2*x3*x4~x1*x3*x4	1.0000, 1.0000
Parsimonious	13	x1*x2x3*x4	1.0000, 1.0000

8.2. QCA Results after Deletion of Overdetermined Cases

Curiously, QCA still produces an incorrect complex solution, repeatedly finding that irrelevant factors matter. For example, QCA finds that X₄ is relevant for whether the X₁, X₂ combination is causal. By construction, we know that this is incorrect. We find, therefore, that all four causal recipes in the complex solution are wrong.

However, QCA finds the correct solution with its parsimonious solution, when formerly the parsimonious solution for Y = 1 was incorrect. This result, however, far from providing a basis for accepting QCA, raises serious doubts about the method.

8.3. Discussion of Study 3 Results

9.1. Study 4 Methods

We constructed a 40-case population using a parametric model reflecting a known set of causal relations and then randomly allocated 20 cases to fuzzy group 1 (FG1) and 20 cases to fuzzy group 2 (FG2). We used a parametric causal structure because it is simpler than the alternative and should be accessible to fsQCA, a method designed for use with continuous variables. Note that because the cases were constructed using coefficients that apply to every case, despite their later placement in different data sets, their causes are the same.

A key part of fsQCA analysis entails calibration. We used multiple calibrations and two calibration logics, empirical calibration and range calibration, explicated in Table 13. We know that few substantive researchers would use multiple calibrations, but we did so in our methodological investigation to decrease the chance that a failure to replicate the known causal structure would be attributed to poor calibration. And, to facilitate a test of stability case-oriented researchers might find germane, we used a strategy that allows us to compare results for one group using different calibration scores and across groups with the same scores.

OPEN IN VIEWER

Table 13.

Calibration Strategies

Strategy	Description
Empirical calibration	A given group’s 5th, 50th, and 95th percentiles anchor the thresholds of full nonmembership, crossover, and full membership in the set. One may obtain an empirical calibration for any continuous variable. For example, for earnings, the empirical calibration may provide a midpoint around $40,000.
Range calibration	The 5th percentile, midpoint, and 95th percentile of the logical range of the variable anchors the thresholds of full nonmembership, crossover, and full membership in the set. For example, for earnings, the range calibration may provide a midpoint around $50,000,000.

Note: Although our variables do not have ranges that are so large, anchor values may differ across the calibration logics.

9.2. Study 4 Results

Table 14 contains the calibrations, while Table 15 contains the analysis results. The first issue to consider is whether fsQCA was able to identify the correct causal recipes. The raw data, before fsQCA calibration, match equation (9):

Y i = . 613 − . 004 × X 1 i + . 030 × X 2 i − . 029 × X 3 i + . 010 × X 4 i + ε i .

OPEN IN VIEWER

Table 14.

Four Sets of Calibration Anchors for Study 4

	Variables and Thresholds
Calibrations	X1 out	X1 middle	X1 in	X2 out	X2 middle	X2 in
1. G1 data, range	.0500000	.5000000	.9500000	1.00000	10.00000	19.00000
2. G1 data, empirical	.0262917	.4986896	.9362063	1.86918	13.15776	19.51347
3. G2 data, empirical	.0600459	.4423967	.9710752	1.72417	9.633014	17.55905
4. All data, empirical	.0125181	.4986896	.9775083	1.009517	11.75234	19.51347
	X3 out	X3 middle	X3 in	X4 out	X4 middle	X4 in
1. G1 data, range	1.00000	10.00000	19.00000	.0500000	.5000000	.9500000
2. G1 data, empirical	2.970067	11.19308	17.32408	.2269875	.5228702	.9442348
3. G2 data, empirical	2.347759	9.72293	19.62373	.0320468	.553797	.9623687
4. All data, empirical	1.645022	10.34501	19.29447	.0373015	.5307223	.978161
	Y out	Y middle	Y in
1. G1 data, range	.0500000	.5000000	.9500000
2. G1 data, empirical	.2753639	.739311	.9708401
3. G2 data, empirical	.2757255	.4770155	.7160342
4. All data, empirical	.2553267	.6818468	.9708401

OPEN IN VIEWER

Table 15.

Study 4, Results for Two Fuzzy Groups of 20 from Fuzzy Simulated Data Set of n = 40, Y = 1 Analyses

	R	Causal Recipes	Coverage, Consistency	R	Causal Recipes	Coverage, Consistency
	Analyses of FG1 Data			Analyses of FG2 Data
Solution	One-case Threshold, Consistency Threshold of .8			One-case Threshold, Consistency Threshold of .8
Range calibration using FG1
Complex	11	∼x1*x2~x3*x4x2*x3x1*x4	.8763, .9028	12	∼x2*~x3x1*~x3x1*~x2x2*x4~x1*x2*x3	.8896, .8256
Intermediate	16	∼x1*x2~x3*x4x2*x3x1*x4	.8763, .9028		All configurations meet frequency threshold
Parsimonious	11	x2x4	.9413, .8624		All configurations meet frequency threshold
Empirical calibration using FG1 data
Complex	11	∼x1*~x3~x3*x4~x1*x2*x4	.8016, .8474	12	x1*x2*~x3x2*~x3*x4x1*x2*x4	.6152, .9268
Intermediate	14	∼x1*~x3~x3*x4~x1*x2*x4	.8016, .8474	8	x2*~x3*x4x1*x2*~x3x1*x2*x4	.6152, .9268
Parsimonious	11	∼x3~x1*x2~x1*x4x2*x4	.9332, .756	12	x2*~x3x1*x2	.6152, .9268
Empirical calibration using FG2 data
Complex	10	x2*x3x1*~x3*x4~x1*x2	.7645, .9434	11	x2*x4~x1*~x2*~x3x1*~x3*~x4~x1*x2*x3	.7599, .9085
Intermediate	10	x2*x3x1*~x3*x4~x1*x2	.7645, .9434	10	x2*x4~x1*~x2*~x3x1*~x3*~x4~x1*x2*x3	.7599, .9085
Parsimonious	10	∼x3x2	.9357, .9304	11	∼x3x2	.8914, .8983

Note: FG1 = fuzzy group 1; FG2 = fuzzy group 2.

The R² value from the OLS regression is .981, signifying that the error variance, σ ε 2 , is very small. A purportedly deterministic method should be more likely to succeed for processes with high R² values. Coefficients for the constant, X₂, and X₃ have p values below .0005; other p values exceed .57. No interaction effects were used in constructing Y. Even so, every causal recipe in the complex solutions identifies two- or three-way interactions. The parsimonious solutions fare better, with the majority of the recipes referencing single causes. Yet even when single causes are referenced, they are more often wrong than right. Thus, even though an OLS regression model can explain 98 percent of the variance, fsQCA consistently fails to identify the correct causal factors.

Looking more closely, we consider the range calibration analysis first. None of the causal recipes match across FG1 and FG2. In fact, one recipe is exactly the opposite in the two data sets: A causal recipe for Y = 1 in FG1 is X₂ = X₃ = 1, while a causal recipe for Y = 1 in FG2 is X₂ = X₃ = 0. These results indicate a failure to replicate.

Looking down the FG1 analyses, however, we also find that calibration matters. For example, in the complex solutions, only 3 of the 7 causal recipes appear in two of the three calibrations, and none appear in all three. Matters barely improve when we move to the FG2 data; of the 10 causal recipes in the complex solutions, only 2 match across two calibrations, and again, none match across all three calibrations.

All told, we produce 7 unique recipes for FG1 data, and 10 unique recipes for FG2 data, simply by perturbing the calibration anchors. Thus, complex solutions identify 17 unique recipes or, put another way, depending on the calibration, the outcomes of the 40 cases appear to have been produced through 17 unique causal recipes. In sum, fsQCA appears unable to replicate findings across data sets drawn from the same population.

One might maintain that the problem is that calibrations are based on subsets of the data; if that is the problem, using all of the data in the calibration process should make group-specific analysis findings match. Table 16 contains results based on calibrations based on all the data.

OPEN IN VIEWER

Table 16.

Study 4, Calibration Assessment, Fuzzy Groups 1, 2, and All Data, All Data Empirical Calibration via Logistic and Complementary Log-log, One-case Threshold, Consistency Threshold of .8

	R	Causal Recipes	Coverage, Consistency	R	Causal Recipes	Coverage, Consistency	R	Causal Recipes	Coverage, Consistency
CalibrationSource/Solution	Proposed (Logit) Calibration			Complementary Log-log Calibration (X)			Complementary Log-log Calibration (X and Y)
FG1 data
Complex	11	∼x1*x2~x1*~x3*~x4x1*~x3*x4x1*~x2*x3*~x4	.7449, .8229	4	x1*x2*x3*x4	.4430, .8419	4	x1*x2*~x3x1*x2*x4	.7263, .9514
Parsimonious	11	∼x3~x1*x2x1*~x2*~x4	.8951, .8128	4	x3*x4	.4690, .8207	4	∼x3x4	.7882, .9164
FG2 data
Complex	11	x2*x4x1*x2*~x3~x1*~x2*~x3*~x4	.7642, .8226	No solution possible, consistency threshold not met			4	x1*x2*~x3x1*x2*~x4	.6361, .8879
Parsimonious	11	x2*x4x2*~x3x1*x2~x1*~x3*~x4~x1*~x2*~x4	.7762, .7788	No solution possible, consistency threshold not met			4	∼x3~x4	.6584, .8786
All data
Complex	15	∼x3x1*x2x2*x4	.9544, .7669	No solution possible, consistency threshold not met			4	x1*x2*~x3x1*x2*x4	.7355, .8792
Parsimonious	15	∼x3x1*x2~x1*x4x2*x4	.9544, .7438	No solution possible, consistency threshold not met			4	∼x3x4	.7923, .8668

Note: Italics indicates the data source for the calibration of the variables.

Furthermore, to assess the sensitivity of the results to calibration assumptions, we used two additional calibration alternatives: a complementary log-log distribution calibration of all X’s and a complementary log-log distribution calibration of both the X’s and Y. We thus compare results from calibrations using logit (symmetric) and complementary log-log (asymmetric) distributions.

Results for the logit calibration indicate that using all of the data in the calibration process does not produce stable results across groups. Only one of the nine complex solutions appears in two different analyses, and none appear in all three.

Table 16 also indicates that complementary log-log calibration results do not match logit calibration results. Taken together, these results suggest that one should provide an explicit theory that justifies the specific fsQCA calibration, because results are highly sensitive to calibration method. If one lacks such a theory, any results produced are arguably arbitrary.

9.3. Discussion of Study 4 Results

10.1. Study 5 Methods

We constructed a 40-case data set using a known set of causal relations and randomly allocate the 40 cases to crisp group 1 (CG1) or crisp group 2 (CG2). Cases in CG1 and CG2 were produced via the same causal process. (The CG2 data were used in study 2.) For study 5, each author separately analyzed one of the two data sets. Afterward, we reanalyze the combined data to give QCA another chance to find the correct causal recipes. Finally, we explore the stability of results using delete-one-at-a-time (D1) simulation, a strategy akin to jack-knifing.

10.2. Study 5 Results

Results for CG1 data are in Table 17, quadrant I. Each solution has three causal recipes. Complex and intermediate solutions are identical, differing slightly from the parsimonious one. Quadrant IV contains results for the CG2 data. Because the cases were randomly assigned to CG1 and CG2, the analysis of CG2 data should obtain the same causal recipes that were produced by analysis of cases randomly assigned to CG1. However, none of the causal recipes in quadrant IV for CG2 match the causal recipes found in quadrant I for CG1 data.

OPEN IN VIEWER

Table 17.

Study 5, Results for Two Random Samples of 20 from One Simulated Data Set of n = 40

Solution	R	Causal Recipes	Coverage, Consistency	R	Causal Recipes	Coverage, Consistency
CG1 data	Quadrant I method 8			Quadrant II method 6
	First analysis of group 1 data: one-case threshold, consistency threshold of .8			Second analysis of group 1 data: one-case threshold, consistency threshold of .6
Complex	11	∼x2*~x3*~x4~x1*x2*~x4x2*~x3*x4	.7692, 1.0000	11	∼x3*~x4x2*~x3~x1*x2*~x4	.9231, .9231
Intermediate	6	∼x2*~x3*~x4~x1*x2*~x4x2*~x3*x4	.7692, 1.0000	10	∼x3*~x4x2*~x3~x1*x2*~x4	.9231, .9231
Parsimonious	11	∼x1*~x4~x2*~x3*~x4x2*~x3*x4	.7692, 1.0000	11	∼x3*~x4~x1*~x4x2*~x3	.9231, .9231
CG2 data	Quadrant III method 8			Quadrant IV method 6
	Second analysis of group 2 data: one-case threshold, consistency threshold of .8			First analysis of group 2 data: one-case threshold, consistency threshold of .6
Complex	13	x2 *~x4~x1 *x2x1 *~x3 *~x4	.5385, 1.0000	13	∼x3*~x4~x1*x2x2*x3	.9231, .8571
Intermediate	10	∼x1*x2x2 *~x4x1 *~x3 *~x4	.5385, 1.0000	12	∼x3 *~x4~x1*x2x2*x3	.9231, .8571
Parsimonious	13	x1 *~x3x2 *~x4~x1 *x2	.5385, 1.0000	13	x2~x3 *~x4	.9231, .8571

Note: CG1 = crisp group 1; CG2 = crisp group 2.

Quadrant IV is the result of different analysts’ decision making while unaware of the findings and procedures for the analysis of CG1. The only analytic decision on which the researchers diverged is that the CG2 analyst chose a consistency threshold of .6 rather than .8. One concern is whether this lone specification difference underlies the differences between quadrants I and IV. The decision is potentially consequential; in the fsQCA manual, Ragin (2008b:46) indicates that consistency scores below .75 reflect nontrivial inconsistency. The analyst of CG2 data accepted a lower consistency threshold because one configuration of the data had a consistency score of .667, just below the .75 cutoff, while the next highest consistency score was .333, substantially lower. Still, because this decision might have so damaged the enterprise that it undermined replication, we conducted additional analyses. ¹⁹

In Table 17, quadrants II and III reveal what each analyst found when using the analytic decisions that had been made in the analysis of the other group. If the different analytic decisions drive the differences between quadrants I and IV, then the solutions for quadrants I and III should match each other, and the solutions for quadrants II and IV should match each other.

Considering quadrants I and III first, for their results are based on the more stringent, more advised consistency threshold, we find that not one of the four causal recipes appears in both quadrants. Quadrants II and IV are also illuminating. One of the four recipes obtained in quadrant IV is found in quadrant II, but the other two recipes in quadrant IV are not in quadrant II, and two recipes in quadrant II are not found in quadrant IV. It seems the widely divergent results are well matched to their samples, just as one would expect were overfitting occurring.

The analyses produced 12 different causal recipes, of which only 1 (X₃ = X₄ = 0) is replicated across the two data sets. Even so, none of the causal recipes match the known causal structure. Collecting the results, QCA fails to replicate causal recipes across data sets.

Combining CG1 and CG2 data allows us to directly determine whether the findings above are a result of problems produced by randomly allocating cases to groups. This is an important hypothesis to assess, for some analysts view QCA as designed for case-oriented research. Many such analysts reject probability sampling, rendering any analysis using random allocation potentially less persuasive. Moreover, the subsamples were necessarily small, lowering their precision. Finally, many QCA studies may use the full population, and it could be that with the full population, the issues above dissipate. Thus, we reassess the matter using all 40 cases.

Table 18 presents the complex and parsimonious solutions for two analyses of the full population. Considering the one-case threshold analysis first, three of the five recipes do not appear in the earlier group-specific analyses. All of the causal recipes involve at least three factors; one of the new recipes that does not appear in the subsample analysis involves all four factors, which suggests that adding cases may cause the algorithm to identify more complex causal recipes.

OPEN IN VIEWER

Table 18.

Study 5 Analysis of Crisp Simulated Data Set of n = 40 (All Sample)

	One-case Threshold, Consistency Threshold of .8			Two-case Threshold, Consistency Threshold of .8
Solution	R	Causal Recipes	Coverage, Consistency	R	Causal Recipes	Coverage, Consistency
Complex	14	x2*x3*~x4x2*~x3*x4x1*~x2*~x3*~x4~x1*x2*~x4~x1*x2*~x3	.5385, 1.000	11	∼x1*x2 *~x4~x1*x2*~x3x1 *~x2 *~x3 *~x4	.4615, 1.000
Parsimonious	14	x1*~x2*~x3~x1*x2*~x3x2*x3*~x4x2*~x3*x4	.5385, 1.000	11	x1 *~x2 *~x3~x1 *x2 *~x4~x1 *x2 *~x3	.4615, 1.000

The two-case threshold analysis produces only three rather than the five recipes identified in the former analysis. Two of the three recipes are not observed in the subsample analyses. Again, a four-way interaction is a chosen causal recipe.

To further consider the stability of the estimates, we conducted a D1 analysis. This approach resembles a Monte Carlo analysis in that results are obtained for multiple subsets of the data, but it is not a Monte Carlo analysis in that we obtain all results possible under a given (and in this analysis a one-case) deletion regimen. Monte Carlo simulations may use only a small proportion of the possible subsets of the data, relying instead on statistical theory to draw inferences. Here, however, relying on the same statistical inference logic QCA analysts may question might only increase the distance between our analysis and QCA proponents’ claims and thus seems inappropriate. Consequently, we opt for D1 simulation instead.

Considering the complex solution (see Table 19), of the 81 causal recipes possible, we consistently obtained the same 5. Fully 95 percent of the replicates produced solutions 1, 2, 3, 5, and 6. The parsimonious solutions were similarly concentrated, identifying the same four causal recipes 95 percent of the time. Thus, QCA D1 results are much more stable, indicating that these recipes accurately reflect QCA analysis of the data.

OPEN IN VIEWER

Table 19.

Study 5, Delete One Analysis, Crisp All Sample, One-case Threshold, Consistency Threshold of .8

	Ingredients and Recipes				Count of Solutions
	X 1	X 2	X 3	X 4	Complex Result	Parsimonious Result
1	—	1	1	0	38	38
2	—	1	0	1	38	38
3	1	0	0	0	38	0
4	1	0	0	—	0	38
5	0	1	0	—	38	38
6	0	1	—	0	38	1
7	1	—	0	0	1	0
8	—	1	0	—	1	1
9	—	1	—	0	1	1
10	0	—	0	1	1	0
11	—	0	0	0	1	1
12	1	—	0	—	0	1
13	0	—	—	0	0	1

Note: In the recipe, if X = 0, it is scored 0; if X = 1, it is scored 1; if X = —, its value is irrelevant to the recipe.

One solution that appears in 95 percent of the complex solutions is a four-way interaction, and all of the other “95 percent” solutions of the D1A analyses involve three-way interactions. Indeed, all solutions identified in study 5, except for one, involved conjunctions of two or more variables. Yet the actual causal structure producing Y = 1 set coefficients for all interactions equal to zero. Thus, results are usually unstable, yet even stable results are consistently incorrect.

10.3. Discussion of Study 5 Results

10.3.1. Frequency of cases thresholds and measurement error

Measurement error is an issue for all research methods. For example, measurement error in linear regression models attenuates coefficients and renders them inconsistent. ²⁰

Critics contend that measurement error is especially debilitating for QCA because imperfect measurement could cause a case to be misclassified. The higher the consistency threshold, the more easily a misclassified case can distort findings. One partial response to this problem is to remove causal configurations that fall beneath a frequency of cases threshold. Yet this strategy is ineffective.

In Figure 1, numbers signify the errorless location of a nation on X*, a latent variable mapped onto the dichotomous variable X. Dots indicate the location an observer believes the nation falls on the latent variable; in fsQCA terms, this could be the set membership score. Values to the right of the vertical line fall inside the crisp set of X, while values to the left of the vertical line fall outside the crisp set of X. Error is always potentially consequential for fsQCA but is consequential for csQCA only if it flips X from its actual value to its opposite. Focusing on csQCA, two conditions increase the chance of a flipped value: (1) the errorless measure is close to the threshold or (2) the error is large. In Figure 1, nation 1 has a small error but because the nation is close to the threshold error moves its observation from X = 0 to X = 1. Nation 2 has the same amount of error, but it is inconsequential because nation 2 is far from the threshold. Nations 3 and 4 illustrate the same possibility for large errors.

OPEN IN VIEWER

Figure 1.

Representation of error for four nations.

For csQCA, errors can be large and have no effect or small and have big effects. This is always true for any threshold method. Because all cases have both informational content and error, Figure 1 also implies that removing rare causal configurations is insufficient, for it leaves intact the error in the cases in the remaining causal configurations.

QCA measures configuration consistency with the conditional mean of Y. By the central limit theorem, the more cases in the configuration, the more precise its estimated conditional mean. Thus, the implicit assumption behind deleting rare causal configurations is that error may play a bigger role in their consistency scores than in the consistency score for common configurations. Hence, removing rare causal configurations removes configurations most likely to harm QCA analyses because of measurement error.

Yet problems ensue. First, there is no necessary connection at the case level between size or consequentiality of error on one hand and placement in a rare configuration on the other. Second, deleting rare configurations removes configurations that might be most crucial for understanding the process at issue, because a case in the rare configuration is arguably far more informative than is yet another case in the common configurations. Thus, the deletion strategy throws away information, is poorly targeted, and may reduce theoretical leverage. Consequently, measurement error harms QCA, and one response to the problem further harms QCA.

11.1. Study 6 Methods

We generated a noncausal data set (NCD), such that Y has no causal relationship to the X’s, by randomly assigning cases to 0 or 1 on each variable. One should expect the association among the variables to be zero, but, randomly it might not be. Yet, correlations and partial correlations confirm that X₁, X₂, and X₃ do not have a statistically significant linear association with Y. Because all variables are dichotomous, there is insufficient measurement precision for the variables to reflect a curvilinear relationship.

11.2. Study 6 Results

Complex and intermediate solutions are identical—“X₁ and not X₂ and not X₃” and “not X₁ and X₂ and X₃,”—and have an overall coverage of .44 and a consistency of 1.00 (see Table 20). ²¹ The solutions are inverses of each other. The parsimonious solution is slightly different: “X₁ and not X₃” and “not X₁ and X₂ and X₃.” Thus, in the analysis of noncausal data, QCA has generated multiple causal recipes that reflect the technique’s ability to detect conjunctural patterns correlational analysis would not. However, in this case, the patterns are spurious; the method detects complex causality even though the data are noncausal, which is the outcome Lieberson anticipated. The results raise serious doubts about QCA’s ability to avoid certifying spurious causal patterns.

OPEN IN VIEWER

Table 20.

Study 6, Causal Recipes and Summary Statistics for Analysis of Noncausal Data, One-case Threshold, Consistency Threshold of .8

Solution	Rows	Causal Recipes	Coverage	Consistency
Complex	7	x1*~x2*~x3~x1*x2*x3	.4444	1.0000
Intermediate	2	∼x3*~x2*x1~x1*x2*x3	.4444	1.0000
Parsimonious	7	x1*~x3~x1*x2*x3	.4444	1.0000

11.3. Discussion of Study 6 Results