Abstract
The field of qualitative comparative analysis (QCA) is witnessing a heated debate on which one of the QCA’s main solution types should be at the center of substantive interpretation. This article argues that the different QCA solutions have complementary strengths. Therefore, researchers should interpret the three solution types in an integrated way, in order to get as much information as possible on the causal structure behind the phenomenon under investigation. The parsimonious solution is capable of identifying causally relevant conditions, the conservative solution of identifying contextually irrelevant conditions. In addition to conditions for which the data provide evidence that they are causally relevant or contextually irrelevant, there will be conditions for which the data neither suggest that they are relevant nor contextually irrelevant. In line with the procedure for crafting the intermediate solution, it is possible to make clear for which of these ambiguous conditions it is not plausible that they are relevant in the context of the research.
Keywords
The last few years, a debate has emerged on which one of the qualitative comparative analysis’s (QCA) main solution types should be at the center of substantive interpretation (Haesebrouck and Thomann 2021). Before the debate started, the highly cited standards of good practice of Schneider and Wagemann (2012:279) suggested that the “intermediate solution should be at the center of the substantive discussion.” In response to this standard, Baumgartner (2015:839) argued that only “maximally parsimonious solutions reflect causal structures” and Baumgartner and Thiem (2020:279) maintained that QCA is “incorrect when generating the conservative and intermediate solution type.” Duşa (2019:1), in turn, reacted to these publications by arguing that the intermediate solution is the most suitable solution for causal analysis.
This article aims to move the debate forward by focusing on the question: “What can be inferred from QCA’s different solution types?” More specifically, this article explores how to draw correct causal inferences from the results of QCA. One of the distinctive features of QCA is that it produces results that can be causally interpreted (Schneider and Wagemann 2012:8). QCA can be used for other purposes, such as summarizing a data set or assessing its coherence. However, the solution types remove conditions from sufficient combinations, which suggests that these conditions are somehow irrelevant. Clear criteria or reasons for removing conditions when a QCA only pursues descriptive objectives have not yet been spelled out. In consequence, shying away from interpreting QCA solutions in causal terms can result in a lack of transparency on the reasons for removing conditions from combinations.
Correctly interpreting the results of a QCA is necessary but definitely not sufficient for avoiding causal fallacies. In fact, many other obstacles can keep researchers from drawing the correct conclusions on causal relations from the analysis of observational data, for example, data deficiencies, errors in study design, or faulty background theories (Baumgartner and Ambühl 2020). Therefore, the use of QCA’s analytical techniques might only be one step in a multimethod study (Schneider 2018:253). However, a correct interpretation of the QCA solutions is also necessary for drawing correct causal inferences from multimethod studies, as will be argued in the conclusions of this article.
This article is structured as follows. The first section introduces the three main solution types, after which it summarizes the debate on how these solutions should be interpreted. The second section builds on regularity theories of causation to define causal relevance, contextual irrelevance, and contextually generalizable sufficiency. The third section assesses which of the solution types offers evidence for causal relevance and contextual irrelevance. The fourth section discusses an integrated way to interpret the solution types, after which the conclusions recapitulate the most important arguments of this article.
QCA Solution Types: The Debate So Far
This section introduces the three main solution types and summarizes the debate on their interpretation.
QCA Solution Types
Table 1 presents a truth table of a fictional study that aims to explain what causes sick people to get better. There are four pills that might cause these patients to get better: A, B, C, and D. The four conditions in this study are thus taking pills A, B, C, and D, and the outcome is being cured (O). The first four rows of the truth table are perfectly sufficient for the outcome (i.e., they have a consistency of 1), rows 5–9 are not sufficient. Seven truth table rows correspond to logical remainders, combinations of conditions that do not correspond to empirical cases.
Fictional Truth Table.
QCA solutions are generally expressed in a language that follows the conventions of Boolean Algebra and set theory. A tilde [∼] indicates that a condition is absent, [*] represents logical AND, which indicates the conjunction of two conditions, [+] represents logical OR, which indicates the disjunction of two (combinations of) conditions, and → indicates that a combination of conditions is sufficient for an outcome.
A first solution type is the conservative solution. This solution applies the following minimization rule: “If two Boolean expressions differ in only one causal condition yet produce the same outcome, then the causal condition that distinguishes the two expressions can be considered irrelevant and can be removed to create a simpler, combined expression” (Ragin 1987/2014:93). Applied to the truth table, this rule results in the following formula:
The second solution type is the parsimonious solution. Publications on this solution generally argue that it is produced by assuming that some logical remainders are sufficient for an outcome and including these logical remainders in the minimization process. More specifically, logical remainders are included in the minimization process if they lead to a less complex solution. Applied to the truth table, this results in the following formula:
The third solution type is the intermediate solution (Ragin and Sonnett 2005). The intermediate solution is based on the distinction between easy and difficult counterfactuals, which, in turn, rests on theoretical and substantive knowledge. In the example, prior knowledge suggests that taking the pills will contribute to the presence rather than the absence of the outcome. Building on these directional expectations, it is more plausible for some logical remainders that they correspond to a sufficient combination than it was for sufficient combinations that correspond to empirical cases. These logical remainders are referred to as easy counterfactuals. The intermediate solution results from Boolean minimization of the rows with a positive outcome and easy counterfactuals. Applied to the truth table, the procedure results in the following formula:
There is an alternative procedure for crafting the intermediate solution, which involves dropping all conditions from the conservative solution that are not in line with theoretical expectations and are not included in the parsimonious solution (Schneider and Wagemann 2012). In the example, the intermediate solution can also be produced by dropping ∼C from the first conjunct of the conservative solution and ∼A from its second conjunct.
The Debate on QCA Solution Types
After the introduction of the intermediate solution, the latter became the preferred output of QCA applications (Baumgartner 2015:851, Schneider and Wagemann 2012). An article by Baumgartner (2015) started a debate on whether this should be considered good practice. In this article, Baumgartner (2015) argues that only the parsimonious solution identifies the conditions that meet the definition of causal relevance provided by regularity theories of causation. Nevertheless, many methodologists and researchers continue to prefer alternative solutions for substantive interpretation. Strikingly however, opponents of the parsimonious solution have explicitly argued that the claim that “from the perspective of a regularity theory of causation” only the most parsimonious solution allows for causal inference is convincing (Schneider 2018:252). This suggests that proponents of other solutions do not (all) agree that the goal of QCA is to identify causally relevant conditions.
The opponents and proponents of the parsimonious solution agree that QCA solutions correspond to sufficient (combinations of) conditions and that a condition X is sufficient for Y if and only if it is true that if X is the case, then Y is the case as well (Baumgartner 2015:842; Schneider and Wagemann 2012:57). Moreover, as argued by Thomann and Maggetti (2020:365) in their overview of different approaches to QCA, both sides of the debate agree that sufficiency alone is “not enough” to draw causal inferences. However, there is a clear disagreement on the criteria that render sufficient (combinations of) conditions causally interpretable.
On the one hand, Thomann and Maggetti (2020) identify an approach to QCA that emphasizes redundancy free (the RF approach) models and prefers parsimonious solutions. According to the RF approach, QCA searches for Boolean difference makers of an outcome, which are conditions that are indispensable parts of sufficient combinations that are indispensable parts of necessary disjunctions (Baumgartner 2015:844). In line with regularity theories of causation, the RF approach argues that these Boolean difference makers are causally relevant.
On the other hand, Thomann and Maggetti (2020) identify an approach to QCA that emphasizes the substantive interpretability (the SI approach) of QCA results and prefers intermediate and conservative solutions. Researchers who follow this approach generally focus on the procedure that is followed for producing the different solution types when they justify their choice of solution type. More specifically, the most invoked argument against the parsimonious solution is that the latter can be based on implausible and/or untenable assumptions on logical remainders. Ragin (2008:175), for example, argues that “parsimonious solutions can be unrealistically simple, due to the incorporation of many (easy and difficult) counterfactual combinations.” Schneider and Wagemann (2012:200), in turn, argue that researchers should only rely on solution terms that are exclusively based on counterfactual claims that are both good and tenable, which is only guaranteed in their “enhanced” intermediate solution.
The argument that the parsimonious solution should not be at the center of interpretation because it builds on untenable or implausible assumptions is not very convincing, given that there are procedures for producing the parsimonious solution that do not rely on logical remainders. For example, the QCA package for R uses an algorithm in which “the remainders are never actually used in the minimization,” but, nevertheless, manages to produce the different solution types (Duşa 2018). Likewise, the minimization procedure applied in the original version of coincidence analysis also allows to produce QCA’s parsimonious solution without assumptions on logical remainders (Baumgartner 2009; Baumgartner and Ambühl 2020).
Opponents of the parsimonious solution do not only argue that parsimonious solutions build on logical remainders but also suggest that QCA solutions imply something about these remainders. According to the SI approach, the parsimonious solution is either identical to the intermediate and/or conservative solution or “claims more” than these solutions (Thomann and Maggetti 2020). More specifically, scholars who follow this approach maintain that the parsimonious solution implies that “the outcome would have also occurred” in the logical remainder rows that are covered by the solution, that is, the logical remainder rows that correspond to combinations that are a subset of the solution (Thomann and Maggetti 2020:370). In the example above, the second path of the parsimonious solution indicates that taking pill C is sufficient for the outcome. When following the SI approach, this would imply that the logical remainder that corresponds to row 16, in which B is absent, would also have a positive outcome. Opponents of the parsimonious solution would argue that this implication is unlikely to be true, because the truth table does not contain evidence that C is sufficient for O without B and B is expected to have a positive impact on the outcome.
The intermediate solution also has implications for logical remainders in the SI approach. In the example above, the second conjunct of the intermediate solution suggests that taking pills B and C is sufficient for the outcome in combination with taking pill A and, thus, that the logical remainders in rows 13 and 14 are sufficient for the outcome. There is not a single case that actually shows that combinations of B, C, and A are sufficient for the outcome. However, proponents of the intermediate solution suggest that it is very likely that the combination of taking pills A, B, and C will always result in the outcome, because taking pill A is expected to have a positive impact on the outcome.
By arguing that the parsimonious solution implies that an outcome would also have occurred in logical remainder rows, opponents of the parsimonious solution suggest that the sufficiency of QCA solutions is (at least to some degree) generalizable beyond the cases included in the study. The idea that QCA should result in generalizable sufficient conditions is also implied by the concept of “robust sufficiency,” which was introduced by Duşa (2019:12) and defined as follows: X is robustly sufficient for Y if and only if Y is “guaranteed to occur” when X is present. According to Duşa (2019), “Boolean minimization guarantees the outcome will always occur” (emphasis added). Unfortunately, identifying such a “robust” sufficient combination, that guarantees that the outcome will always occur, would require knowing the full cause of an effect, which is beyond the grasp of empirical research (cf. infra). Other authors introduce clear limits on the generalizability of QCA solutions and argue that the goal of QCA is arriving at a solution that allows for a “modest” degree of generalization (Thomann 2020:264). “Modest generalization” is defined by Berg-Schlosser et al. (2009:12) as the formulation of propositions that “we can apply, with appropriate caution, to other similar cases—e.g. cases that share a reasonable number of characteristics with those that were the subject of the QCA.”
The opposition against the parsimonious solution, thus, might be based on the idea that the goal of QCA is to identify (limited) generalizable sufficient (combinations of) conditions. This idea is related to a second argument that is sometimes invoked against the parsimonious solution: If all logical remainders that result in a less complex solution are included in the minimization process, the solution might miss causally relevant conditions and, therefore, be incomplete. After providing an example in which the inclusion of logical remainders in the minimization procedure resulted in a solution that did not include an (allegedly) relevant condition, Ragin (2008:175) argues that “parsimonious solutions can be unrealistically simple.” Moreover, the quote from Ragin (1987/2014:93; cf. supra) suggests that Boolean minimization is about eliminating irrelevant conditions, not identifying relevant conditions. Opponents of the parsimonious solution might only want to eliminate conditions from sufficient combinations for which there is evidence that they are irrelevant for explaining the presence (or absence) of the outcome, at least for the cases under investigation and cases that share the same background conditions as these cases.
The arguments against the parsimonious solution, thus, seem to be based on the idea that the goal of QCA is to identify (limited) generalizable sufficient (combinations of) conditions and, therefore, only conditions can be removed from sufficient combinations if there is evidence that they are irrelevant for explaining the presence (or absence) of the outcome (at least in the cases under investigation and cases that share “a reasonable number of characteristics” with these cases). However, proponents of the parsimonious solution never suggest that QCA solutions should be interpreted as sufficient conditions that can be generalized beyond the cases under investigation. In this approach to QCA, the identification of sufficient conditions only constitutes one step in a procedure that aims to identify causally relevant conditions. Moreover, proponents of the parsimonious solution explicitly argue that the parsimonious solution does not make any claims about causal irrelevance (Baumgartner 2015:841; Baumgartner and Thiem 2020).
The different positions in the debate on QCA’s solution types might, thus, be a consequence of a disagreement on the appropriate search target for QCA: identifying causally relevant conditions or identifying (limited) generalizable sufficient combinations of conditions by eliminating conditions that are irrelevant for explaining the presence of the outcome (at least in the cases under investigation and cases that “share a reasonable number of characteristics” with the cases under investigation). In line with this argument, the parsimonious solution might aim to identify causally relevant conditions, while the other solution types might aim to eliminate conditions for which there is evidence that they are irrelevant for explaining the presence of the outcome (at least in cases that meet certain background conditions) and identify (limited) generalizable sufficient (combinations of) conditions. While proponents of the parsimonious solution have explicitly defined causal relevance, a clear definition of contextually generalizable sufficiency or contextual irrelevance has not yet been formulated. The next section aims to fill this gap in the academic debate on solution types.
Causal Relevance, Contextual Irrelevance, and Contextually Generalizable Sufficiency
A method that aims at causal inference must be underpinned by a theory of causation that establishes “criteria that allow separating causal from noncausal relations” (Rohlfing and Zuber 2019:2). There are good reasons to anchor QCA in regularity theories of causation (Rohlfing and Zuber 2019:22). A lot of methodological publications on QCA use concepts derived from regularity theories and “refer to the INUS-theory of causation” (Ragin 2009; Schneider and Wagemann 2012; Thomann 2020:21). Moreover, the work of Baumgartner (2015) shows that QCA can be underpinned by modern regularity theories. In contrast, it is not clear how other theories of causation could undergird QCA (Rohlfing and Zuber 2019; Schneider 2018). This section builds on regularity theories to define causal relevance, contextual irrelevance, and contextually generalizable sufficiency.
Causal Relevance
The definition of causal relevance builds on modern regularity theories of causation, which are inspired by Mackie’s theory of Insufficient but Necessary part of an Unnecessary but Sufficient (INUS) causation. According to Mackie (1965:247), a factor is causally relevant if and only if it is at least an “INUS condition.” The two crucial parts of this definition are “necessary part” and “sufficient condition.” “Sufficient condition” implies a deterministic view of causation. Given that a condition is only causally relevant if it is part of a combination that guarantees the outcome, causes are presumed to determine their effects (Baumgartner 2009:74; Graßhoff and May 2001:88). “Necessary part” implies that a condition must be indispensable in a sufficient combination. More specifically, a condition can only be considered causally relevant if it cannot be removed from a combination without changing the latter’s status as a sufficient combination (Baumgartner 2015:842).
Nonredundancy is essential for distinguishing causally relevant conditions from conditions that are not relevant because of the law of monotony: Sufficient conditions can be conjunctively supplemented with arbitrary factors without altering their status as sufficient conditions (Baumgartner 2015:842). Row 1 of Table 1 corresponds to a sufficient combination: “A*B*∼C*D.” However, not every condition in this combination is guaranteed to be causally relevant. The truth table contains evidence for the causal relevance of A: Row 7 shows that “B*∼C*D” is not sufficient in the absence of A. In contrast, there is no evidence that ∼C is causally relevant: There is no truth table row that shows that “A*B*C*D” is not sufficient for the outcome.
In addition to indicating that a condition must be a necessary part of a sufficient combination, the acronym INUS suggests that it must be an “insufficient part” of an “unnecessary” combination. This indicates that factors are not INUS conditions if they guarantee an effect by themselves or if there is only one minimally sufficient combination (which is hereby also a necessary combination). While Mackie (1965:246) argues that such factors are not INUS conditions, he does consider them causally relevant. More specifically, Mackie (1965:247) argues that a factor is causally relevant when it is “at least an INUS-condition,” a category that includes INUS conditions, conditions that are part of the only sufficient combination and conditions that are necessary and sufficient by themselves. The conditions that Mackie (1965:247) considers causally relevant are all “necessary parts of sufficient conditions,” so the I (i.e., insufficient) and U (i.e., unnecessary) are not really needed in the acronym INUS to describe which conditions are causally relevant.
Nevertheless, there are two good reasons to make causal claims in terms of “at least an INUS condition.” First of all, this acronym makes clear that causally relevant conditions do not bring about their effects in isolation and that there are generally different combinations of conditions that produce an effect. In line with the concept of conjunctural causation, “insufficient part” emphasizes that individual conditions only produce an effect in combination with other conditions. In line with the idea of equifinality or multiple causation, unnecessary combination makes clear that there are generally multiple causes of an effect. Second, even if there would be conditions that are sufficient on their own or if there would be only one combination that produces an effect, it would be very difficult to provide conclusive evidence that this is indeed the case. This is because it is rarely possible to identify all necessary parts of all sufficient combinations. Causal knowledge is typically incomplete, as illustrated by the following example of Mackie (1974:66): Even in a matter of such intimate and absorbing interest as the death of human beings, we cannot confidently assert any complete regularity of this kind. We do not know all the causes of death, that is, all the different closely preceding assemblages of conditions that are minimally sufficient for death. And even with any one cause, we do not know all the possible counteracting causes, all the factors the negations of which would have to be conjoined with our positive factors to make up just one minimal sufficient condition.
As demonstrated by Baumgartner and Thiem (2020), QCA only guarantees fallacy-free inferences on causal relevance if the assumption of causal homogeneity is adopted: All causally relevant conditions that are not included in the analysis must be assumed to be constant across all cases or, at least, not be instantiated in a combination that causes the outcome. If the assumption of causal homogeneity is not satisfied, uncontrolled causes might be covertly responsible for the dependencies that have been uncovered in a data set (Baumgartner and Ambühl 2020). In truth Table 1, A is only guaranteed to be causally relevant if the causally relevant conditions that are not included in the analysis are present or absent in all cases or, at least, not be instantiated in a combination that causes the outcome. If all cases in rows 1 and 2 would also have taken pill E and the cases in rows 7 and 8 would not have taken this pill, A would no longer be an indispensable part of the sufficient combination presented in rows 1 and 2 and the conclusion that A is causally relevant might be incorrect. This does not imply that conclusions on causal relevance will always be incorrect if the assumption of causal homogeneity is not satisfied, only that correctness is not guaranteed if this assumption is not satisfied.
Contextual Irrelevance
Mackie’s INUS theory provides a clear definition of causal relevance, which includes two essential elements: A condition must be a necessary part (1) of a sufficient combination (2). If irrelevance is the negation of relevance, we can deduce from this definition that a condition is causally irrelevant if it is not a necessary part of any sufficient combination. Unfortunately, it is rarely possible to provide evidence that something is causally irrelevant, given that this would require knowing every single combination that is sufficient for the outcome (Baumgartner 2006:83; Mackie 1974:71-72; Nickelsen and Graßhoff 2011). Providing evidence for universal causal irrelevance would require demonstrating that there is not a single sufficient combination in which a condition is indispensable (Baumgartner and Thiem 2020). This would require knowing every single causal path toward an outcome, which is rarely a realistic goal for applied research, especially in the social sciences.
In contrast, it might be possible to show that a condition is at least irrelevant in a specific context or causal path. 2 Just like it is possible to provide evidence that a condition is at least a necessary part of one sufficient combination, it might be possible to show that it is not an essential part of at least one sufficient combination. In other words, it might be possible to demonstrate that a condition is at least an Unnecessary Part of an Unnecessary but Sufficient combination or an UPUS condition. The crucial element of the UPUS definition of contextual irrelevance is that a condition must be an unnecessary part of a sufficient combination. A condition is irrelevant in a combination if this combination guarantees an outcome irrespective of whether this condition is present or absent. An UPUS condition is by definition an unnecessary condition, but unnecessary conditions are not by definition UPUS conditions. Demonstrating that a condition is not necessary only requires showing that the outcome can occur in its absence, which only requires one case that shows the outcome but not the condition. In contrast, an UPUS condition is a part of a sufficient combination that is not necessary for the sufficiency of this combination and, thus, is irrelevant in this specific combination. Demonstrating that a condition is an UPUS condition requires demonstrating that there is a context in which it does not matter whether the condition is present or absent. 3
In addition to being an unnecessary part of a sufficient combination, the acronym UPUS indicates that UPUS conditions are part of sufficient combinations that are unnecessary for the outcome. Hereby, the acronym makes clear that an UPUS condition is only irrelevant in a specific context and not universally causally irrelevant. If a condition would be an unnecessary part of a sufficient and necessary combination, there would not be a single context in which the condition matters and the condition would be universally causally irrelevant. Given that it is usually not possible to know every causal path toward an outcome, claims about the irrelevance of a condition can generally only be made in terms of at least an UPUS combination. Such claims only imply that there is evidence that there is a context or causal path in which it is irrelevant whether or not the condition is present, not that the condition is irrelevant in every context. Conditions that are at least UPUS conditions are “contextually irrelevant” or “path irrelevant,” irrelevant in a specific context or path, but are not universally causally irrelevant. Conditions that are contextually (or path-)irrelevant are at least UPUS conditions but might also be universally causally irrelevant. 4
Rows 1 and 2 of the truth table presented in Table 1, for example, suggest that it does not matter whether or not a patient takes pill D if she or he takes A and B and does not take C, suggesting that D and ∼D are not relevant in the path “A*B*∼C” and, thus, are UPUS conditions. However, we cannot conclude that there is not a single context in which pill D is relevant. All patients included in the study might, for example, have a healthy lifestyle and taking pill D might be indispensable for getting better if a patient does not have healthy lifestyle. As with causally relevant conditions, conclusions on contextual irrelevance are only guaranteed to be fallacy free if the assumption of causal homogeneity is adopted. We can, for example, only conclude from truth Table 1 that D and ∼D are irrelevant if “A*B*∼C” is present if we assume that the causally relevant conditions that are not included in the analysis are constant in all cases or, at least, not be instantiated in a combination that causes the outcome. If the cases in row 1 would all have an unhealthy lifestyle and the cases in row 2 would have a healthy lifestyle, it is possible that the “unhealthy” patients in row 1 only got better because they also took pill D. However, if we assume that “healthy lifestyle” was present (or absent) in all cases, there is at least one context in which it did not matter whether or not a patient took pill D.
Contextually Generalizable Sufficiency
Because it is generally not possible to find all indispensable parts of a sufficient combination, it is difficult to generalize sufficiency beyond the observed cases. While the combination in row 1 of the truth table might be consistently linked to the outcome in the cases under investigation, there is no evidence that cases that are not included in the truth table and take pills A, B, and D and do not take pill C will also be cured. However, if we adopt the assumption of causal homogeneity, sufficiency can be generalized to cases in which the same confounders are present and absent as in the examined cases. Causally relevant conditions that are not included in the analysis are referred to as background conditions in the remainder of this article. Cases in which the same background conditions are present as in the examined cases are referred to as cases with the same causal background. The assumption of causal homogeneity allows to find “contextually generalizable sufficient conditions”: (combinations of) conditions that consistently co-occur with the outcome in a data set and that are guaranteed to consistently co-occur in cases with the same causal background as the cases in the data set.
Not all sufficient (combinations of) conditions are contextually generalizable sufficient. C is, for example, sufficient for the outcome in truth Table 1. However, it is not clear whether all patients that take pill C and have the same causal background as the examined patients will also get better. For example, given that there is no patient that took pill C without taking pill B, there is no empirical evidence that taking pill C will result in getting better without taking pill B. Sufficient (combinations of conditions) are contextually generalizable if only contextually irrelevant conditions are removed from sufficient combinations. Rows 3 and 4, for example, show that D is irrelevant if “∼A*B*C” is present in cases that have the same causal background conditions as the examined cases. In consequence, the combination “∼A*B*C” is contextually generalizable sufficient. Cases that have the same background conditions as the examined cases and show the combination “∼A*B*C” will also show the outcome. Contextually generalizable sufficient conditions can, however, contain conditions that are not relevant. D is, for example, part of the contextually generalizable sufficient combination “∼A*B*C*D” but is contextually irrelevant.
Causal Relevance, Contextual Irrelevance, and QCA Solutions
The next subsections assess which solution types provide evidence for causal relevance, contextual irrelevance, and contextually generalizable sufficiency under the assumption of causal homogeneity. The discussion relies on the interwar project, a textbook example used in Ragin and Rihoux (2009), that is based on the data of Berg-Schlosser and De Meur (1994). In the example, QCA is used to examine the breakdown (BRK) of democracy in Europe during the interwar period. 5 The study includes five conditions that could be relevant for explaining the outcome “BRK”: economically developed (DEV), urbanized (URB), industrialized (IND), politically stable (STA), and high literacy rate (LIT). Table 2 presents the truth table of the interwar project.
Truth Table Interwar Project.
Note: DEV = economically developed; URB= urbanized; IND = industrialized; STA = politically stable; LIT = high literacy rate; BRK = breakdown.
The example was selected for two main reasons. First, there are logical remainders in the truth table and the parsimonious, conservative, and intermediate solution do not coincide. If there are no logical remainders, the solutions will coincide and the same conclusions can be drawn from the different solution types. The arguments raised in this section (and the remainder of this article) only apply if the solutions are not identical. Second, model ambiguities did not arise in the analysis of the outcome “BRK”: There is only one solution formula for each solution type. QCA often results in more than one solution formula, a phenomenon known as model ambiguities. A general discussion on the impact of model ambiguities on the conclusions that can be drawn from QCA is presented in Baumgartner and Thiem (2017). The Online Appendix (which can be found at http://smr.sagepub.com/supplemental) explores the impact of model ambiguities on the conclusions that can be drawn on contextual irrelevance. More specifically, the Online Appendix (which can be found at http://smr.sagepub.com/supplemental/) argues that not every condition that is removed in the conservative solution is guaranteed to be contextually irrelevant if there are multiple conservative or parsimonious solutions. The conservative and intermediate solution might miss causally relevant conditions if there are multiple parsimonious or conservative solutions.
Parsimonious Solution
The parsimonious solution of the interwar project includes two paths toward the outcome:
Are the conditions in this solution indispensable parts of sufficient combinations and, thus, causally relevant? Truth table rows 1–4 show that “∼DEV” always leads to “BRK,” while row 7 shows that the combination of conditions displayed in row 1 is not sufficient without “∼DEV.” Likewise, rows 3–6 show that “∼STA” is sufficient for the outcome, while row 9 shows that the combination of conditions displayed in row 6 is not sufficient without “∼STA.” The conditions in the parsimonious solution are, thus, indispensable parts of sufficient combinations.
Conclusions on contextual irrelevance or contextually generalizable sufficiency cannot be drawn from the parsimonious solution if there are logical remainders. “∼IND” is, for example, not included in the first path of the parsimonious solution (∼DEV). However, there are no cases that show “∼DEV” in combination with “IND.” In consequence, there is no empirical evidence that “∼DEV” will result in the outcome if “IND” (and the background conditions) are present. In other words, there is no evidence that the absence of economic development (∼DEV) will result in the BRK of democracy in IND countries (in which the background conditions are present), so we cannot be certain that the absence of industrialization (∼IND) is not an indispensable part of this combination. The sufficiency of the combinations in the parsimonious solution (“∼DEV” and “∼STA”) can, thus, not be generalized to cases that share the same background conditions as the examined cases. If “∼IND” is causally relevant in the path “∼DEV,” cases that have the same causal background as the examined cases and show “∼DEV” but not “∼IND” are not guaranteed to show the outcome.
To sum up, under the assumption of causal homogeneity, the parsimonious solution identifies causally relevant conditions. However, if there are logical remainders, we cannot deduce from the parsimonious solution which conditions are contextually irrelevant or generalize conclusions on sufficiency to cases that share the same background conditions as the examined cases.
Conservative Solution
The conservative solution of the interwar project includes two paths:
There is no evidence that all conditions included in the conservative solution are indispensable parts of sufficient combinations. In addition to the condition included in the first conjunct of the parsimonious solution (“∼DEV”), the first sufficient combination of the conservative solution includes “∼URB” and “∼IND.” However, there is no truth table row that demonstrates that “∼DEV” is not sufficient if “URB” and/or “DEV” are present.
Conditions are included in the conservative solution if two prerequisites are met. First, there must be sufficient combinations for the outcome that include these conditions. If this would be considered evidence for causal relevance, every single property of every case that shows the outcome would be considered causally relevant. Second, conditions are included in the conservative solution if they are not the only difference between two sufficient combinations. However, the fact that no cases are included in a study that only differ on a condition does not constitute evidence for the causal relevance of this condition, reality might simply not provide all possible combinations of conditions.
The conservative solution does remove the conditions that are unnecessary parts of unnecessary but sufficient combinations (or UPUS conditions). 6 For example, “URB” and “∼URB” are removed from the combinations displayed in truth table rows 5 and 6. Truth table rows 3 and 4 demonstrate that it does not matter whether “URB” is present or absent if “∼STA*DEV*LIT*IND” is present. This indicates that “URB” and “∼URB” are not indispensable parts of the sufficient combinations displayed in truth table rows 5 and 6.
Given that the conservative solution only removes conditions for which there is evidence that they are not relevant in a specific combination, the sufficiency of the combinations in the conservative solution can be generalized to cases that share the same background conditions as the cases under investigation. In other words, the conservative solution identifies contextually generalizable sufficient conditions. This does not imply that the conservative solution represents the complete cause or even a more complete cause than the parsimonious solution. Given that there is no evidence for the causal relevance of the conditions that are only in the conservative solution, all conditions that are in the conservative solution but not also in the parsimonious solution might be causally or contextually irrelevant.
To sum up, if the solutions do not coincide, the conservative solution removes conditions for which there is evidence that they are contextually irrelevant but does not show which conditions are causally relevant. Conclusions on contextual irrelevance can only be drawn if causal homogeneity is assumed and are only generalizable to cases that have the same causal background as the examined cases. The conservative solution represents a contextually generalizable sufficient condition but might include a lot of contextually irrelevant conditions.
Intermediate Solution
The intermediate solution is based on theoretical expectations on the conditions. In the interwar project, the absence of the conditions is expected to have a positive impact on the outcome “BRK.” The first conjunct of the intermediate solution of the interwar project includes two conditions that are not included in the parsimonious solution (“∼URB” and “∼IND”):
There is no empirical evidence that “∼URB” and “∼IND” are indispensable parts of this sufficient combination: There is no truth table row that demonstrates that “∼DEV” is not sufficient if “URB” or “IND” are present. This would require cases that combine “∼DEV” with “URB” or “IND,” which are not included in the analysis. Conditions are represented in the intermediate solution if two prerequisites are met. First, they must be part of the conservative solution. Second, they are either part of the parsimonious solution or prior theoretical expectations suggest that they have a positive impact on the outcome. As argued above, the conservative solution does not provide evidence for causal relevance. It seems self-evident that being in line with theoretical expectations can also not be considered empirical evidence for the relevance of a condition.
One could argue that the above argument neglects the rationale behind the intermediate solution, which is based on a counterfactual analysis of the logical remainders (Schneider and Wagemann 2012:171). Ragin and Sonnett (2005:184) refer to the methodological literature on counterfactual thought experiments, which are used by social scientists and historians to make causal claims in single case studies (Tetlock and Belkin 1996). However, counterfactual thought experiments are far more sophisticated and elaborate than the procedure used for crafting the intermediate solution. To be convincing, thought experiments must follow far more stringent guidelines than being in line with prior theoretical and substantive knowledge (Lebow 2010; Tetlock and Belkin 1996). In fact, literature on such thought experiments warns very explicitly against theory-driven thinking, which according to Tetlock and Belkin (1996:34) is “perhaps the most lethal threat to the validity of counterfactual thought experiments.” However, the procedure for crafting the intermediate solution is strongly based on prior theoretical knowledge and actually requires theory-driven thinking.
Demonstrating causal relevance requires demonstrating that an outcome would not occur in the absence of a condition. In order to draw conclusions on causal relevance from the intermediate solution, researchers must assume that the so-called difficult counterfactuals are not sufficient for the outcome. The conclusion that “∼IND” is an indispensable part of the sufficient combination “∼DEV*∼URB*∼IND” would be based on the assumption that “∼DEV*∼URB*IND” is not sufficient. However, assuming that “∼DEV*∼URB*IND” is not sufficient for the outcome, while the truth table indicates that “∼DEV*∼URB*∼IND” is sufficient, requires the assumption that “∼IND” is indispensable for the sufficiency of “∼DEV*∼URB*∼IND.” If “∼IND” would not be indispensable in this sufficient combination, “∼DEV*∼URB” would also be sufficient and, because of the law of monotony, “∼DEV*∼URB*IND” would also be sufficient. In other words, assigning causal relevance to “∼IND” in the combination “∼DEV*∼URB*∼IND” requires the assumption that it is indispensable and hereby causally relevant, which amounts to circular reasoning.
Attributing causal relevance to conditions because prior theoretical expectations suggest that they might be causally relevant seems a textbook case of theory-driven thinking, of trying to “assimilate ‘what happened’ to some prior knowledge structure or schema that specifies cause-effect relationships for events of that type” (Tetlock and Belkin 1996:34). Given that it is the task of researchers to evaluate the evidence at hand and come to an unbiased conclusion, interpreting evidence in a way that is likely to confirm existing beliefs seems something that should be avoided at all cost.
The intermediate solution also does not provide empirical evidence for contextual irrelevance. In contrast to the second sufficient combination of the conservative solution, the second sufficient combination of the intermediate solution does not include “LIT,” “DEV,” or “IND.” However, there is no empirical evidence that these conditions are not necessary in the sufficient combination: “∼STA*DEV*LIT*IND.” Conditions are dropped from the intermediate solution if they are not in the parsimonious solution and not in line with theoretical expectations. As argued above, the parsimonious solution does not provide evidence for contextual irrelevance. Dropping a condition because it is not in line with theoretical expectations does not constitute empirical evidence for contextual irrelevance. However, it seems less problematic than adding conditions because theory suggests they might be causally relevant. To eliminate contextually irrelevant conditions, it suffices to build on directional expectations and assume that only the presence (or absence) of a condition can be causally relevant (Schneider and Wagemann 2012:325). For many conditions, researchers have good reasons to expect that their absence (or presence) is unlikely to be causally relevant for the presence of the outcome. In the example above, it is not impossible, but unlikely that the presence of “DEV” has a positive impact on the BRK of democracies, given that theory and previous studies suggest that economic development actually has a positive impact on the survival of democracy.
More generally, it seems impossible to avoid the assumption that some conditions are contextually irrelevant when drawing causal conclusions from a QCA. Causal homogeneity could never be assumed without assuming that some conditions are contextually irrelevant, otherwise every single difference between the examined cases should be included as a condition. In the hypothetical example presented in the first section, there are probably some cases with blue eyes and some with brown eyes, some with large feet and some with small feet, some with long hair and some with short hair…. However, few scholars would argue that eye color, shoe size, or hair length should be included or kept constant because prior knowledge suggests they are irrelevant. Mackie (1965:257) mentions that a choice of “a range of possibly relevant factors” must be made before causal discovery is possible and argues that an “already-developed body of causal knowledge” can be used to restrict the range of possibly relevant factors (Mackie 1965:258; 1974:74). The body of causal knowledge, for example, suggests that hair length, eye color, or shoe size do not have an impact on diseases getting cured, so these can be assumed to be irrelevant. Prior causal knowledge on the irrelevance of the absence of a condition, of which the presence is possibly relevant for the outcome, can be equally well-developed. For example, not exposing a match to fire can be assumed to be irrelevant for a match to catch fire. Likewise, we can assume it is unlikely that not taking a pill is causally relevant for getting better or that the presence of inconvenience has a positive impact on compliance with rules. Moreover, we only need to assume that a condition is irrelevant in a particular context, not that it is always irrelevant.
Whether or not the conditions that are removed from the intermediate solution are actually contextually irrelevant fully depends on the correctness of directional expectations. In consequence, whether or not the intermediate solution represents contextually generalizable sufficient (combinations of) conditions also depends on whether directional expectations are correct. The correctness of directional expectations should not be taken for granted. In social sciences, the prior knowledge on which directional expectations are based is not always solid. Moreover, causal uniformity should not be taken for granted: “a given condition may, combined with different others, sometimes act in favor of the outcome and sometimes, differently combined, act against it” (Berg-Schlosser et al. 2009:9). Therefore, researchers should clearly differentiate between the conditions that are removed because of directional expectations and the conditions for which there is empirical evidence that they are contextually irrelevant.
To sum up, the intermediate solution neither provides empirical evidence for causal relevance nor contextual irrelevance. The intermediate solution excludes conditions for which there is empirical evidence that they are contextually irrelevant as well as conditions for which prior knowledge suggests that it is very unlikely that they are relevant. The intermediate solution includes conditions for which there is evidence that they are relevant and conditions for which prior knowledge suggests that they might be relevant. While assumptions on causal relevance are problematic, assumptions on contextual irrelevance can be more firmly based on an “already-developed body of causal knowledge” (Mackie 1965:258). The intermediate solution could be interpreted as a way to make clear that some ambiguous conditions are unlikely to be causally relevant but should not be considered a way to add causally relevant conditions to the parsimonious solution. Whether or not the intermediate solution represents a contextually generalizable sufficient condition depends on the correctness of directional expectations.
Causal Relevance, Contextual Irrelevance, and QCA Solutions
If the different solution types do not coincide, there are important differences in the conclusions that can be drawn from the different solution types. The parsimonious solution effectively identifies causally relevant conditions but does not provide evidence for contextual irrelevance and does not represent a contextually generalizable sufficient condition. The conservative solution shows that some conditions are irrelevant in a specific context and represents a contextually generalizable sufficient condition but does not provide evidence for causal relevance and might include many contextually irrelevant conditions. Beside the conditions for which the QCA provides evidence that they are causally relevant or contextually irrelevant, there will be conditions for which the empirical evidence neither suggests that they are relevant nor that they are irrelevant. In line with the procedure for crafting the intermediate solution, it might be interesting to make clear for which of these ambiguous conditions it is not plausible that they are relevant. However, it is important to emphasize that the data do not provide evidence for the relevance or contextual irrelevance of these conditions.
Toward an Integrated Interpretation of the QCA Solutions
The discussion above makes clear that, if the solutions do not coincide, (1) the parsimonious solution identifies causally relevant conditions, (2) the conservative solution removes contextually irrelevant conditions and represents a contextually generalizable sufficient condition, and (3) the intermediate solution offers a procedure for removing conditions from the conservative solution for which it is implausible that they are causally relevant. However, unless the solutions coincide, none of the solutions offers evidence for causal relevance and represents a contextually generalizable sufficient condition. The answer to the question “how should the different QCA solutions be interpreted?” primarily depends on what is considered the appropriate search target of QCA: identifying causally relevant conditions, identifying contextually generalizable sufficient combinations, or both?
The practical use of a study that exclusively focuses on identifying causally relevant conditions is limited. Whether or not a causally relevant condition produces an effect depends on whether or not other causally relevant conditions are present. In the fictional example presented in the first section, knowing that taking pill A is causally relevant is not very helpful for giving advice to patients if we do not know what other pills they also need to take to get better. While it is not possible to know all other conditions that need to be present for the effect to occur, the conservative solution does identify sufficient combinations that can be generalized to cases that share the same background conditions as the cases under investigation. The conservative solution, thus, provides information on at least one context in which the causally relevant conditions have an impact on the outcome. Unfortunately, the conservative solution can include many contextually irrelevant conditions.
Researchers, thus, face a dilemma between (1) focusing on the parsimonious solution, which identifies the causally relevant conditions but does not represent contextually generalizable sufficient conditions, or (2) focusing on the conservative solution, which identifies contextually generalizable sufficient combinations but does not show for which conditions there is evidence that they are relevant. The intermediate solution seems to provide an easy way out of this dilemma. However, given that there is no evidence for the causal relevance of the conditions in the intermediate solution, the intermediate solution can also include many contextually irrelevant conditions. Moreover, if directional expectations are incorrect, the intermediate solution might result in focusing on conditions that do not produce the desired outcome on their own. If the body of causal knowledge is well-developed, the risk of missing relevant conditions by focusing on the intermediate solution is limited. However, the risk of arriving at unnecessarily complex solutions will always be considerable.
Researchers should not pretend to have evidence for the relevance or contextual irrelevance of conditions if such evidence is missing, especially when giving policy advice. While omitting indispensable parts of a policy mix results in ineffective policy measures, adding irrelevant conditions might result in overly complicated or expensive policy measures. An appropriate standard of good practice for QCA would be to be fully transparent about the conditions on which the data provide evidence that they are causally relevant or contextually irrelevant and the conditions on which the data are inconclusive and should be subjected to further research.
The current standards of good practice in QCA advise scholars to focus on one solution, “usually” the intermediate solution (Schneider, Vis, and Koivu 2019:8; Schneider and Wagemann 2012). However, the intermediate solution includes conditions for which the data provide evidence that they are causally relevant and conditions for which there is no such evidence. Researchers who only focus on this solution, thus, have no way of knowing for what conditions the data provide evidence that they are causally relevant and for what conditions this is not the case. While researchers should not pretend to have evidence for the causal relevance of conditions for which the data do not provide such evidence, they should also not ignore the evidence that the data provide for causal relevance. Exactly the same assumption is needed to guarantee that correct conclusions on causal relevance can be drawn from the parsimonious solution as to guarantee that correct conclusions on contextual irrelevance can be drawn from the conservative solution: causal homogeneity. In consequence, it is inconsistent to eliminate conditions from a solution because the analysis suggests they are contextually irrelevant, but not draw conclusions on the causal relevance of the conditions.
Given that the different solution types clearly have complementary strengths and weaknesses, researchers should be advised to present the three solutions in an integrated way, which shows for every condition in every path whether there is evidence that it is causally relevant or contextually irrelevant or whether there is no such evidence. The conditions for which the data do not make clear whether they are causally relevant or contextually irrelevant can be referred to as ambiguous conditions, for which researchers can make clear whether or not it is plausible that they are causally relevant. Tables 3 and 4 present different approaches to visualize for each path which conditions are causally relevant, which are contextually irrelevant and which are plausible and implausible ambiguous.
Presentation Qualitative Comparative Analysis Results.
Note: DEV = economically developed; URB= urbanized; IND = industrialized; STA = politically stable; LIT = high literacy rate.
Configuration Chart Qualitative Comparative Analysis Results.
Note: 
/
= causally relevant condition present/absent; 
/
= plausible ambiguous condition present/absent; 
/
= implausible ambiguous condition present/absent; 
/ = contextually irrelevant condition; DEV = economically developed; URB= urbanized; IND = industrialized; STA = politically stable; LIT = high literacy rate.
Conclusion
Over the last few years, a debate has emerged on which one of the QCA’s main solution types should be at the center of substantive interpretation. This article suggests that the different QCA solutions have complementary strengths. The parsimonious solution is capable of identifying causally relevant conditions, the conservative solution of identifying contextually irrelevant conditions. In addition to conditions for which the data provide evidence that they are relevant or contextually irrelevant, there will be conditions for which the data neither suggest that they are relevant nor contextually irrelevant. In line with the procedure for crafting the intermediate solution, it might be interesting to make clear for which of these ambiguous conditions, it is not plausible that they are relevant in the context of the research. It is important to reiterate here that condition that is removed from the conservative solution is only guaranteed to be contextually irrelevant if there are multiple conservative or parsimonious solutions. In consequence, researchers should be careful when interpreting their results when they are faced with model ambiguities (cf. Online Appendix, which can be found at http://smr.sagepub.com/supplemental/).
The most important point of this article is that researchers should not focus on only one solution type when they interpret the results of their QCA. All three QCA solutions provide relevant information and combining them allows to gain as much information as possible on the causal structure behind the phenomenon under investigation. Researchers should also be transparent about the limitations of their QCA results. First of all, the generalizability of sufficiency is very limited: Sufficiency claims can only be generalized to cases that share the same background conditions as the cases under investigation. In consequence, researchers aiming at generalizable results should explicitly discuss these conditions in order to specify the external validity of the results of their analysis (Thomann and Maggetti 2020). Second, researchers should be fully transparent about the conditions on which their analysis is inconclusive and, therefore, should be subjected to further research.
In line with the growing body of literature on multimethod set-theoretic research, researchers could combine QCA with other methods to examine these ambiguous conditions. A correct interpretation of the QCA solutions is also necessary for drawing correct conclusions from such multimethod research. A follow-up case study that shows that the causally relevant conditions identified in the parsimonious solution were not relevant or that the contextually irrelevant conditions removed in the conservative solution were causally relevant would disconfirm the conclusions of the QCA. More specifically, this would suggest that an important condition was missing in the QCA and, thus, that the assumption of causal homogeneity was incorrect. In contrast, if a post-QCA case study would indicate that a condition that is only included in the intermediate solution is not relevant, this would complement but definitely not disconfirm the QCA results.
Supplemental Material
Supplemental Material, sj-pdf-1-smr-10.1177_00491241211036153 - Relevant, Irrelevant, or Ambiguous? Toward a New Interpretation of QCA’s Solution Types
Supplemental Material, sj-pdf-1-smr-10.1177_00491241211036153 for Relevant, Irrelevant, or Ambiguous? Toward a New Interpretation of QCA’s Solution Types by Tim Haesebrouck in Sociological Methods & Research
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by fonds wetenschappelijk onderzoek (1219221N).
Supplemental Material
The supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
