Choosing a qualitative comparative analysis solution in multi-method impact evaluation

Abstract

Qualitative comparative analysis is increasingly popular as a methodological option in the evaluator’s toolkit. However, evaluators who are willing to apply it face inconsistent suggestions regarding the choice of the ‘solution term’. These inconsistent suggestions reflect a current broad debate among proponents of two approaches to qualitative comparative analysis. The first approach focuses on substantial interpretability and the second on redundancy-free results. We offer three questions to guide the choice of a solution term in the context of impact multi-method evaluation research. They are related to the intended use of the findings, goals of the analysis and regularity theory of causality. Finally, we showcase guidelines through three potential applications of qualitative comparative analysis. The guiding questions would almost always lead to choosing the substantial interpretability approach. However, the redundancy-free approach should not be disregarded. Whatever the choice, researchers should be aware of the assumptions each approach is based on and the risks involved.

Keywords

innovation subsidies process tracing qualitative comparative analysis quasi-experimental design solution types

Introduction

A solution formula (hereafter ‘solution’) is one of the most important products of qualitative comparative analysis (QCA). A solution is a Boolean expression that describes the relationship between conditions and outcome in the ‘set-theoretic terms’ (Ragin, 2014). If we analyse conditions ‘A’, ‘B’ and ‘C’ along with an effect ‘E’, the solution formula might be:

A * B * C \to E

If all three conditions are met, ‘E’ will happen.

However, the different goals that are pursued by researchers and incompleteness of the analysed data mean that three types of solutions may be obtained. The first is conservative, the second is intermediate and the third is parsimonious. The conservative solution includes the most conditions:

A * B * C \to E

the intermediate excludes some conditions that are considered redundant:

A * B \to E

and the parsimonious solution has the fewest conditions:

A \to E

Note that the conditions and outcome are expressed in set-theoretic terms.¹ In the given example, ‘A’ could denote a set of mid-sized firms and ‘E’ a set of companies which carried out their projects successfully. The final choice of the solution includes decision which part of the logical expression should be kept and which could be ignored; this choice is a researcher’s job and responsibility.

Until recently, the recommendations for choosing a solution were relatively straightforward. Researchers were advised to treat all three solutions as valid, report them for transparency and explicitly explain their final choice of the solution after the analysis (Schneider and Wagemann, 2012). This approach to QCA is focused on the substantial interpretability of the results and is thus called the ‘substantial interpretability approach’ (SI). However, another approach has recently emerged, called the redundancy-free approach (RF). It emphasises the need to find solutions that lack redundancy – that is, which minimise the risk of including conditions that do not influence the outcome or are not causally relevant.² Under the RF approach, only the parsimonious solution is considered to truly describe causal relations between conditions and outcomes (Baumgartner, 2015; Baumgartner and Thiem, 2020). The SI approach is older and remains more popular, and many more empirical examples exist (Thomann, 2020). Nonetheless, researchers are developing procedures for the RF approach, and it is gaining popularity (Thiem, 2017). Differing suggestions about the solution choice arise from disagreements about the goal of the analysis and the assumptions needed for a causal claim. The debate over approaches has been reviewed in other articles (Haesebrouck and Thomann, 2021; Schneider, 2018; Thiem, 2018; Thomann, 2020; Thomann and Maggetti, 2020).

The approach should be chosen before the actual analysis is planned and conducted. The choice of RF implies choosing a parsimonious solution type, whereas SI does not unequivocally determine a specific solution type. The mainstream SI approach suggests a conservative or intermediate solution (Haesebrouck and Thomann, 2021; Schneider and Wagemann, 2019). However, it is still possible to choose the parsimonious solution (Rihoux and de Meur, 2009) or all solution types (Fiss, 2011; Haesebrouck, 2022) under SI. Once the approach is chosen, adequate procedures to match the choice are provided (for both approaches).

This article focuses mostly on the first stage of choosing the approach, which in turn suggests a solution type. The subsequent step of choosing the specific solution formula is not discussed in our article.

Different approaches lead to different suggestions for evaluators. Most guidelines for implementing QCA in evaluation research either do not discuss the topic (Gerrits and Verweij, 2016) or recommend an intermediate solution (Befani, 2016; Pattyn et al., 2019). However, Thiem (2017) advocated for the RF approach and thus the parsimonious solution as the only correct one. Kahwati and Kane (2020) noted the existence of different approaches, and Thomann (2020) briefly compared the two approaches in the broad context of comparative policy analysis.

The differing suggestions can hinder the employment of QCA in impact evaluation or may lead to unsatisfactory practices. Evaluators might limit themselves to a single familiar approach, whether SI or RF, and would thus ignore the assumptions involved – or they might incorrectly interpret the chosen solution. That is, the lack of a conscious choice of the approach and solution can lead to incorrect interpretation of QCA results.

To provide guidelines for evaluators employing QCA, we introduce three questions to facilitate the choice of an approach and solution. The questions relate to the intended use of the findings, goals of the analysis and regularity theory of causality. The questions are predicated on the discussion between advocates of the SI and RF approaches (Haesebrouck and Thomann, 2021) and consider the context of impact evaluation research.

In the first section of the article, we place an impact evaluation research within the context of the debate about SI and RF. In the second section, the three guiding questions are proposed to facilitate the choice of a solution; for each question, the consequences of the answers are discussed. The third section provides three examples of study designs which follow our proposed guidelines. The examples present various research designs: stand-alone QCA, QCA with process tracing and QCA with a quasi-experimental design.

Background to QCA solution choice in impact evaluation research

Impact evaluation is understood in this article as ‘any evaluation which refers to impact (or often outcome) indicators’ (White, 2010: 154). Impact evaluation is thus interested in claims related to causal relations between the programme or the policy and the impact. A thorough review of methodological approaches in impact evaluation, including QCA, was provided by Stern et al. (2012). As the review indicates, methodological approaches are based on diverse theories of causality. Therefore, evaluators wanting to make a causal claim should be transparent about their assumptions and the evidence they use.

It is common knowledge that a chosen research method should enable achieving the goal of a study. Hence, in this section, we discuss the specificity of impact evaluation research goals. Such goals reflect in the decision to use QCA as a method of investigation and in the decision about which approach to employ.

Intended use of results guides evaluation questions

A distinctive feature of evaluation studies is their focus on the ‘primary intended use’ of the results (Alkin and Christie, 2012; Patton, 2008). This means that the application of QCA must be not only methodologically correct but – more importantly – of practical relevance (Haesebrouck and Thomann, 2021; Tanner, 2014; Thiem, 2017; Thomann, 2020). The primary intended use varies, but two main categories emerge from the literature: instrumental use and conceptual use (Alkin and Taut, 2003; Johnson et al., 2009). The first is related to specific decisions by commissioners, such as programme continuation or adjustment. Conceptual use focuses on understanding how the programme works; the aim here is to build knowledge to plan further interventions or to adjust an existing programme.

The primary intended use should reflect in the evaluation questions of a study. The two categories described above are, respectively, albeit loosely related to a dominant dichotomy in impact evaluation questions (European Commission, 2013). One category investigates the effects of a programme (i.e. whether it works) and the other the causes and circumstances related to its effects (how and why it works). Hence, the evaluation questions indicate which research methods would be appropriate. The two types of evaluation questions have equivalents in actual evaluation methods. To examine whether a programme works, an experimental or quasi-experimental method is often recommended (White, 2010). To determine how and why a programme works, a possible method is QCA (Befani, 2016; Pattyn et al., 2019).³

The ‘how and why?’ question is a broad one (Stern et al., 2012). The QCA method addresses only the second aspect, the ‘why’. This requires identifying configurations of conditions which are ‘necessary’ and ‘sufficient’ and those which ‘make a difference’ for the effect to occur. Therefore, combining QCA with explanatory approaches can provide insight into the actual mechanisms⁴ which lead to the outcome. Examples are process tracing, contribution analysis and realistic evaluation (Befani, 2020). Using both configurational and mechanistic approaches provides more accurate picture of the investigated reality than stand-alone QCA and better answers to the research questions (Pattyn et al., 2020).

While ‘necessary’ and ‘sufficient’ are relatively unequivocal terms, ‘making a difference’ (i.e. being a cause) can have many interpretations (Johnson et al., 2019). At the most basic level, making a difference relates to situations in which ‘presence or absence [of a condition] makes a difference to the presence or absence of the effect, assuming binary causes and effects’ (Rohlfing and Zuber, 2019: 6). The possible operationalisation of the term in QCA differs in many regards. Let us discuss two most important differences related to foundations causal claims are based on, and possible errors while making the claims.

Causal claims may be based on diverse theories of causality. Each theory has specific conditions for its validity (Rohlfing and Zuber, 2019). Stern et al. (2012) offer review of theories in the context of impact evaluation. Haesebrouck and Thomann (2021) discuss theories QCA may be based on. The RF approach bases causal claims solely on regularity theory of causation, within which the most important criterion is that causes are regularly followed by the effects. The theory was proposed by Hume (2003) and has been refined by many authors, including Mackie (1974) and Baumgartner (2008, 2013). The theory offers coherent and elegant foundations for causal claims. However, it is argued that in real-world research, especially applied research, its conditions are rarely met because of less than perfect data (Schneider, 2018).

The second important difference is related to the trade-off between two possible errors (Braumoeller, 2015; Rohlfing, 2018). In the first error, conditions are unnecessarily included in the solution. In the second, conditions are omitted which should be included in the solution (Radaelli and Wagemann, 2019). These errors have important consequences for recommendations based on evaluation studies. Haesebrouck (2022) stated that, ‘While omitting indispensable parts of a policy mix results in ineffective policy measures, adding irrelevant conditions might result in overly complicated or expensive policy measures’. The errors are presented in Table 1.

Table 1.

Qualitative comparative analysis model versus actual causal relations.

		Reality
		Condition makes a difference for an outcome	Condition does not make a difference for an outcome
Model	Condition included	OK	Error: redundant condition (redundancy-free approach focuses on avoiding it)
Model	Condition excluded	Error: omitted condition (mainstream substantial interpretability approach focuses on avoiding it)	OK

The RF approach focuses on excluding redundant conditions.⁵ Therefore, the parsimonious solution is recommended. In our example from the introduction, the solution A → E would be chosen as an RF approach because conditions B and C may be redundant.

The SI approach is less specific in this regard. Mainstream SI literature suggests a preference for avoiding the omission of conditions (Haesebrouck and Thomann, 2021: 11). Therefore, depending on the goal of the analysis, either a conservative (A*B*C → E) or an intermediate (A*B → E) solution is suggested. In this example, the researcher should be able to explain why C is included or excluded, based on theoretical and substantive knowledge of the investigated phenomenon. This difference between the two approaches is further illustrated in our first two examples of research design.

This section has discussed the importance of three factors when deciding on the solution choice for QCA: (1) the intended use of findings, (2) the goal of the analysis and (3) the regularity theory of causality. The factors are elaborated in the section titled ‘Guidelines for solution choice’ and are illustrated in ‘Examples: innovation subsidies offered to enterprises’. First, multi-method designs for QCA are briefly introduced.

Combining QCA with other methods

Because of the various evaluation questions to be answered in one study, there is a strong preference for using a multi-method approach in impact evaluation (European Commission, 2013). This predilection is reflected in the suggestion that QCA should not be used as a stand-alone approach (Befani, 2016; Kahwati and Kane, 2020; Pattyn et al., 2019). For a thorough and recent review of applying QCA in mixed-methods and multi-method research strategies, readers may consult Rihoux et al. (2021).

Often, QCA is used jointly with case studies (Schneider and Rohlfing, 2016; Thomann and Maggetti, 2020). Specifically, the sequencing of QCA with process tracing to identify mechanisms behind the configuration leading to the effect is gaining popularity (Álamos-Concha et al., 2021; Beach, 2018; Beach and Rohlfing, 2018; Schneider and Rohlfing, 2013). For example, Álamos-Concha et al. (2020) investigated configurations of conditions leading to the effective transfer of training. In their follow-up, they applied process tracing to identify mechanisms that enabled such training transfer.

The QCA approach can also be combined with quantitative methods (Meuer and Rupietta, 2017), especially with a quasi-experimental design, which is often used in impact evaluation research (European Commission, 2013).⁶ Our focus is on designs in which QCA and a quasi-experimental method are used in a sequence (Green et al., 2015; Kahwati and Kane, 2020). Using both QCA and a quasi-experimental design in one study allows the researcher to answer both types of evaluation questions, that is, first, whether a programme works, second, how and why it works). For example, Pelucha et al. (2019) used QCA after quasi-experiment to deepen the understanding of causal relationships between public support for training in a firm and positive change in the profit per employee (i.e. an effect). Configurations of several conditions were found to mediate the impact of public support on the effect.

This short review of combining QCA with other methods creates the context for the rest of this article. Applying QCA in mixed-methods and multi-method research strategies can influence the choice of a solution. This point is discussed and illustrated in the following sections.

Guidelines for choosing a solution

We argue that evaluators who apply QCA should choose either an RF or an SI approach and justify their decision. We propose a set of guiding questions in this regard. These questions are related to the intended use of the findings, the goals of the analysis, and the regularity theory of causality. The questions are based on the debate about the SI and RF approaches (Haesebrouck and Thomann, 2021; Thomann and Ege, 2020; Thomann and Maggetti, 2020) and they reflect the context of impact evaluation research. The questions are introduced in this section.

Are you expected to choose one solution type?

Although it is good practice within the SI approach to report all solutions, it is standard practice to highlight one solution type at the end of the analysis (Befani, 2016; Kahwati and Kane, 2020; Pattyn et al., 2019; Thiem, 2014). Some exceptions apply. Here, we consider deploying QCA in which the results are instrumental to the next stage of the study. In this case, the primary intended use involves further analysis. For example, the goal of the QCA may be to identify the configurations of conditions that make a difference for the effect. Those configurations would then be used further in a quasi-experimental module of the study.

The described situation is not located in the mainstream SI approach but may be associated with it (Fiss, 2011; Haesebrouck, 2022). Therefore, if the researchers wish to provide multiple solutions to the primary intended users, they should choose the SI approach. The situation of applying an SI approach when there is no need for only one solution is elaborated in the third example of research design.

Which feature of the model do you mainly want to avoid: Including redundant conditions or omitting conditions which make a difference?

The choice is related to the goals of the two approaches (Baumgartner and Ambühl, 2020; Dușa, 2019; Haesebrouck and Thomann, 2021; Thiem, 2017; Thomann and Ege, 2020). The RF approach claims that identifying redundancy-free models is the only goal of the method. Within the substantial interpretability approach, making a difference is generally associated with necessary conditions and a robustly sufficient configuration of conditions. The robustly sufficient configuration of conditions refers to the complete cause of a given effect. Therefore, the configuration ‘guarantees the outcome will always occur’ (Dușa, 2019: 12); hence, it cannot omit any relevant conditions. At the same time, choosing a parsimonious solution remains feasible within the SI approach (Rihoux and De Meur, 2009).

Do you want to base your causal claims solely on the regularity theory of causation?

Choosing the regularity theory as the only foundation for causal claims, while elegant and coherent, has two important negative consequences for real-world applications. First, the data requirements are difficult to meet. Second, QCA cannot be combined with investigation of specific cases. Let us discuss them in turn. The crucial assumption of the redundancy-free approach is configurational homogeneity in the analysed data (Baumgartner and Ambühl, 2020: 12). The tenability of the assumption depends on including all causally relevant conditions that influence the outcome and the completeness of the data – that is, the degree to which each configuration of conditions is represented by the cases at hand. The RF approach can be chosen only if the researcher is able to claim configurational homogeneity without collecting further evidence.

A configurational homogeneity statement is a bold one to make in evaluation and generally in social science research. Almost always in real-world research, certain configurations of conditions are not represented by the data; the configurations are called ‘logical remainders’ and this challenge ‘limited diversity’. Therefore, the assumption of configuration homogeneity is treated as a notably weak point of the RF approach (Schneider, 2018) and is debated by methodologists (Haesebrouck and Thomann, 2021). Evaluators wanting to apply the approach should be aware of this issue and directly discuss the assumption when presenting the research results.

Within the SI approach, configurational homogeneity is also a goal. However, researchers acknowledge that it can be difficult to attain. Hence, they are expected to provide additional evidence for a causal claim.⁷ Such evidence is most often collected through case studies or process tracing (Beach and Rohlfing, 2018). Often, in an evaluation study, the researcher uses diverse sources of data and collects assorted evidence to support a causal claim; in such instances, the SI approach is preferred. It offers guidelines for the design and examples of studies. By contrast, proponents of RF had not provided similar instructions at the time of writing this article.

Researchers who apply the SI approach and face limited diversity may introduce simplifying assumptions to the conservative solution based on theoretical or substantial knowledge of the field (Schneider and Wagemann, 2012: 167–169). They may identify situations in which – if other conditions are held constant – the presence (or absence) of a given condition is expected to contribute to the occurrence of the effect. To put it simply, they treat some logical remainders as if there were observed cases. The solution with includes assumptions is called intermediate. While sometimes there may not be enough basis for the assumptions or stakeholders involved in the evaluation may not feel comfortable in making them, the assumptions are optional.⁸

The second important negative consequence of choosing the regularity theory as the only foundation for causal claims is related to the level of analysis. The scope of the regularity theory is limited to cross-case patterns. Therefore, within-case results cannot be used as additional evidence for the causal claim. While in impact evaluation using both cross-case and within-case evidence is a standard, this limitation may discourage many evaluators. If the researchers want to use within case evidence, they should base their claims on other theories of causation. While substantial interpretability approach does not offer one consistent theory for making causal claims, it is much more flexible in terms of dealing with both imperfections of real-world data and diversity of evidence (Haesebrouck and Thomann, 2021).

Table 2 summarises the guidelines we provide. The table presents the questions, possible answers and consequences for the decisions regarding each approach and the solution type. The RF approach should be used only if (1) only one solution type should be chosen, (2) the goal of QCA is to identify an RF model and (3) causal claims are to be based only on the regularity theory of causality. If any of these conditions are not met, researchers should choose the SI approach.

Table 2.

Criteria guiding the choice of a solution in multi-method impact evaluation.

Approach	Redundancy-free	Substantial interpretability
Intended use of findings: Are you expected to choose one solution type?	Yes	Yes		No
Goal of QCA: Which feature of the model do you mainly want to avoid: including redundant conditions or omitting conditions which make a difference?	Including redundant conditions	Including redundant conditions	Omitting conditions which make a difference	N/A
Theory of causality: Do you want to base your causal claims solely on the regularity theory of causation?	Yes	No
Recommended solution type	Parsimonious	Parsimonious	Conservative or intermediate	Utilise all available

QCA: qualitative comparative analysis.

The order of the questions reflects the process of thinking about research design but not their importance. In most studies, researchers will be interested in choosing one solution type (positive answer to the first question). Answers to the second question are not conclusive for the choice of the approach. Therefore, the third question is often decisive.

Examples: Innovation subsidies offered to enterprises

There are many possible configurations of answers to the questions proposed in the previous section. However, only the three most instructive are discussed here. The examples correspond to the RF (first example) and SI approaches (second example), including a situation where there is no need to choose a single solution (third example). They also correspond to three different research designs: stand-alone QCA, QCA with process tracing and QCA with quasi-experimental design. The number of examples given here is a compromise between the goal of showcasing all different applications of QCA and the limits on the length of the article. The examples are loosely inspired by research ideas implemented in evaluation studies conducted by the authors (Krupnik, 2012; Krupnik et al., 2015). We keep the descriptions simplistic for illustrative purposes.

We first consider a study to evaluate innovation subsidies offered to enterprises. The intervention scheme is described in Supplementary Material. In the examples discussed below, the outcome ‘E’ of a subsidy was defined as a non-zero income from the sold results of R&D activities. Such an outcome must have been reported by the company within 2 years after completing the project. A lack of self-reported income was treated as if it was a lack of outcome ‘~E’. Based on our literature review, several conditions that influenced the outcome ‘E’ were identified:

Microenterprise ‘M’;

Received an income from innovations introduced no more than 3 years before submitting the application for subsidy ‘I’;

High market demand for the product or service resulting from R&D activities proposed by an enterprise ‘D’; and

Manufacturing sector ‘P’.

All conditions were operationalised in the binary form (0, absent; 1, present) and were saturated by the secondary data.

In all three examples below, analysis would lead to at least two types of solutions for the effect:

Conservative

~ M * I * ~ D * P + M = > E

This pertains to enterprises which (1) are larger than microenterprises, (2) have been receiving income from introducing their innovations to the market, (3) lack high demand for the outputs from of their R&D activities and (4) operate in the manufacturing sector. Alternatively, they may be microenterprises that achieve the effect, regardless of other characteristics:

Parsimonious

~ M * I + M = > E

This pertains to enterprises which (1) are larger than microenterprises and (2) have been receiving income from introducing innovations to the market. Alternatively, they may be microenterprises which achieve the effect.

A separate analysis for lack of the effect (~E) would indicate the same solution as conservative and parsimonious:

~ M * ~ I = > ~ E

Here, enterprises (1) larger than microenterprises and (2) which have not been receiving income from introducing innovations to the market do not achieve the effect.

Applying an redundancy-free approach (first example)

Commissioners were interested in disaggregating the existing programme into smaller programmes to better match them to the target audiences. Therefore, the evaluation question was, ‘Which features of enterprises are certain to make a difference to the effect?’ The researchers understood that they should focus on avoiding redundant conditions in the final solution; not doing so would yield too many small and overly specific groups that would be difficult to address.

The researchers based their analysis on the regularity theory of causation and claimed that all assumptions were met. Hence, the researchers decided that a RF approach would be most appropriate for the research questions. They chose a parsimonious solution indicating that configuration of conditions:

~ M * I + M = > E

leads to the effect and configuration:

~ M * ~ I = > ~ E

to a lack of the effect.

Based on their results, the researchers answered the evaluation question by stating that for two groups of enterprises, the programme should be provided without any major changes. The first group was enterprises larger than micro which had been receiving an income from introducing innovations to the market; the second group was microenterprises. For other enterprises, especially those larger than microenterprises and those which had not been receiving such income, a new programme should be designed and implemented.

Applying a substantial interpretability approach (second example)

The commissioners were interested in increasing the effectiveness of subsidies, as measured by the effect ‘E’. They wished to identify features of enterprises which were robustly sufficient for achieving the outcome. The evaluation question was therefore, ‘Which configurations of conditions would ensure the occurrence of the effect?’ The researchers understood that they should focus on including all relevant conditions and that the search target in QCA involved robust sufficiency. A SI approach was the obvious choice.

Analysis led to conservative solutions for the effect:

~ M * I * ~ D * P + M = > E

It was reasonable to assume that if an outcome occurred even in the absence of D, it would also happen in the presence of the condition. Hence, the adequate directional expectation was introduced. While there were no cases for ~M*I*D*P, the simplifying assumption was that the outcome would occur for this configuration of conditions. Therefore, the researchers obtained an intermediate solution:

~ M * I * P + M = > E

Moreover, they obtained a conservative solution for the lack of the effect:

~ M * ~ I = > ~ E

In the SI approach, the results of QCA were insufficient to make a causal statement. However, the researchers applied process tracing to investigate specific causes, which enabled them to identify the mechanisms behind the configurations that led to the effect (Supplementary Table S1).

Finally, the researchers provided an answer to the evaluation question. They concluded that to secure the presence of the effect, subsidies should be given mainly to two groups of enterprises. The first was manufacturing enterprises that were larger than microenterprises and had been receiving income from introducing innovations to the market. The second group was microenterprises. Moreover, the provision of subsidies to enterprises larger than microenterprises and those which had not been receiving income from introducing innovations to the market should be reconsidered.

A substantial interpretability approach without the need for one solution (third example)

The overall evaluation question here was to estimate the net effect of the intervention. It was expected, based on earlier studies, that the net effect would be relatively small and that the effect might vary significantly among different groups. Hence, the evaluation question was formulated as two questions: ‘What is the net effect for diverse groups of beneficiaries?’ and ‘What are the factors causing the diversity in the effect?’

To answer the evaluation questions thoroughly, QCA was applied before quantitative analysis. In this case, regression discontinuity analysis was employed. The question that QCA addressed was ‘What configurations of conditions form groups of beneficiaries that display the highest and lowest net effects?’

In the first stage of the study, QCA provided the configuration of conditions for the effect to occur. This step identified the groups with the highest and lowest net effects. In the second stage, a quasi-experimental analysis was conducted. It was possible to introduce all solutions into the analysis (Supplementary Table S2).

This study offered several benefits for the commissioners. They were informed about expected effect sizes for companies that corresponded to conservative, intermediate and parsimonious solutions, along with group sizes. Based on this information and information about available funding, policy makers could decide on the selection criteria (set conditions from a given solution) for the next round of funding. They knew how many potential applicants would meet the criteria and what effect size to expect by the end of the programme.

Criteria guiding the choice of a solution for each example are presented in Table 3. The table illustrates how different intended uses of findings from seemingly similar analyses translate into different questions and answers. While each scenario shows the proper use of the QCA approach, it is crucial for evaluators who apply QCA to know which type of situation they face.

Table 3.

Summary of three different approaches for configurational comparative methods.

	First example (redundancy-free approach)	Second example (substantial interpretability approach with one solution chosen)	Third example (substantial interpretability approach without choosing one solution)
Intended use of findings	Disaggregating the existing programme into smaller ones to better match segments of target audiences: expectation to choose one solution	Identifying features of enterprises which were sufficient for achieving the effect: expectation to choose one solution	Improving the net effect measurement within counterfactual analysis; no expectation to choose one solution
Goal of QCA	Which features of enterprises are certain to make a difference for the effect? Focus on excluding redundant conditions	Which configurations of conditions are robustly sufficient for the effect to occur? Focus on including conditions making a difference	What configurations of conditions form groups of beneficiaries with the highest and the lowest net effect?
Causal claim based only on the regularity theory of causation	Yes	No	No
Chosen solutions	~MI + M = > E ~M~I = > ~E	~M*IP + M = > E* ~M~I = > ~E*	~MI~DP + M = > E* ~MIP + M = > E ~MI + M => E* ~M~I = > ~E*

Source: Own elaboration based on Haesebrouck and Thomann (2021).

QCA: qualitative comparative analysis.

Discussion

The notion that evaluation research should ensure both a high scientific standard and practical relevance has been promoted by many authors (Alkin and Christie, 2012; Patton, 2008). Some have aptly pointed out that multi-method designs should be used to overcome the methodological and practical limitations of stand-alone methods (Coryn et al., 2011; White, 2009). The goal of this article was to provide guidelines for a QCA solution choice in multi-method impact evaluation studies. These guidelines are embedded in a broader discussion comparing the substantial interpretability and redundancy-free approaches to QCA (Schneider, 2018; Thiem, 2018). Some publications have aimed at bridging the two approaches, leading to more favourable and useful research schemes (Haesebrouck and Thomann, 2021; Thomann, 2020; Thomann and Maggetti, 2020). The current article is intended to contribute to this stream of literature.

Our article makes two contributions to the emerging literature on the topic. First, it demonstrates how the primary intended use of the results of a QCA study influences the solution choice. This focus on the main use of results is an indispensable element of applied research; it lends an argument to debates about all methods, not just QCA (Haesebrouck and Thomann, 2021; Tanner, 2014; Thomann, 2020). Second, the article exemplifies under which scenario the RF approach and not choosing one specific solution could be reasonable options. However, we acknowledge that such scenarios are rare.

It has been argued that the SI approach is of most interest for public policy analysis (Haesebrouck and Thomann, 2021; Schneider, 2018; Thomann, 2020). We believe that the guiding questions we have provided would almost always lead to choosing the SI approach. This is because researchers most often prefer to avoid omitting relevant conditions in the explanation. Moreover, configurational homogeneity is generally a bold statement to make in evaluation research. The SI approach promotes a diversity of evidence and the inclusion of theoretical, substantial and in-depth case knowledge to enrich the analysis.

In addition, the SI approach is relatively flexible. It allows for choosing between solution types at the end of the analysis. Hence, different competitive solutions may be discussed with the commissioner. In our view, it is important to allow for the commissioner’s criteria to influence the final choice of the solution. Such a criterion could be, for example, real-life consequences resulting from a given solution. These consequences can be identified only if competitive solutions are presented. In the RF approach, this scenario is impossible because a commissioner is given only one solution, chosen by the researcher. Regardless of the chosen approach, researchers cannot foresee all uses of their results. Therefore, we argue that it is crucial to report all solutions and to ensure that the limitations of the conclusions based on the choice are clearly communicated.

This article might be criticised for treating the RF approach as legitimate and comparable to the SI approach in the context of impact evaluation. A version of this criticism was advanced by an anonymous reviewer of the article, who argued that conditions for the RF approach do not exist in commissioned evaluation research conducted in cooperation with stakeholders. The reviewer pointed out that stakeholders will not be willing to agree for all the assumptions of the RF approach. Although RF is a new approach and its practical usefulness is still being explored, peer reviews in public policy analysis have already applied this approach. For example, Andreas et al. (2017) analysed the attitude of European Union (EU) member states to renewable energy transition during an economic recession (2008–2013). Haesebrouck and Thiem (2018) investigated the defence policy of European countries. Lankoski and Thiem (2020) analysed the impact of agricultural policies on sustainable development. We are not aware of the employment of RF-approach in the commissioned impact evaluation. However, we believe such employment to be only a matter of time. While we agree that the SI approach is typically chosen, we believe it would be premature to declare the RF approach as unviable and illegitimate for impact evaluation research.

To illustrate the application of our guiding questions, we implemented them in three examples of stylised evaluation studies. In our examples, we do not share detailed methodological parameters and data, as they were meant to serve solely an illustrative purpose. We believe the examples may provide inspiration for evaluators and other readers.

Given the relative novelty of the debate in the evaluation field, there is a need for empirical studies which would further explore the proposed guidelines. In particular, a comparative use of approaches regarding simulated and real-life data could deepen the understanding of solution choices and their consequences. Unresolved issues include the following:

In what situations (other than those presented here) could the RF approach be applied?

How can QCA be further implemented together with other methods used in the evaluation field? What consequences does such joint use have for the choice of approach?

How often can the RF approach be implemented in actual impact evaluation research?

How can the study of processes be combined with the RF approach?

In what other circumstances – and how often – is there no need to choose only one solution?

What are the consequences of the choice of approach for the generalisability of results?

The debate regarding the SI and RF approaches contributes to expanding the range of prospective uses of QCA in impact evaluation research. However, an abundance of choices may breed confusion. Whatever the choice, researchers should be aware of the assumptions their model is based on and the risks involved. We hope that our guidelines will make it easier for others to navigate among the available options.

Supplemental Material

sj-doc-1-evi-10.1177_13563890221088015 – Supplemental material for Choosing a qualitative comparative analysis solution in multi-method impact evaluation

Supplemental material, sj-doc-1-evi-10.1177_13563890221088015 for Choosing a qualitative comparative analysis solution in multi-method impact evaluation by Seweryn Krupnik and Maciej Koniewski in Evaluation

Supplemental Material

sj-doc-2-evi-10.1177_13563890221088015 – Supplemental material for Choosing a qualitative comparative analysis solution in multi-method impact evaluation

Supplemental material, sj-doc-2-evi-10.1177_13563890221088015 for Choosing a qualitative comparative analysis solution in multi-method impact evaluation by Seweryn Krupnik and Maciej Koniewski in Evaluation

Footnotes

Acknowledgements

The authors are grateful for the valuable suggestions and comments from Ingo Rohlfing, Anna Szczucka and Monika Woźniak as well as the three anonymous reviewers.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This research is financed by the Polish National Science Center (Project 2019/35/B/HS5/04238).

ORCID iDs

Seweryn Krupnik

Maciej Koniewski

Supplemental material

Supplemental material for this article is available online.

Notes

Seweryn Krupnik holds an Assistant Professor position at the Jagiellonian University (JU). He is also a Project Manager and Researcher at the Center for Evaluation and Analysis of Public Policies of the JU. His main research interests are innovation policy and qualitative comparative analysis.

Maciej Koniewski holds an Assistant Professor position at the JU. In addition, as a Consultant of the Center for Evaluation and Analysis of Public Policies of the JU, he has completed many evaluation projects regarding education policy and subsidy programmes for companies. Clients have included nongovernmental organisations, government agencies, public administration and private sector entities.

References

Álamos-Concha

Cambré

Pattyn

, et al. (2020) What drives training transfer effectiveness and how does this transfer work? Antwerp Management School. Available at: https://publicaties.vlaanderen.be/view-file/40312b

Álamos-Concha

Pattyn

Rihoux

, et al. (2021) Conservative solutions for progress: On solution types when combining QCA with in-depth process-tracing. Quality and Quantity. Epub ahead of print 3 August. DOI: 10.1007/s11135-021-01191-x.

Alkin

Christie

(2012) An evaluation theory tree. In: Alkin

(ed.) Evaluation Roots. Thousand Oaks, CA: SAGE, pp. 13–65.

Alkin

Taut

(2003) Unbundling evaluation use. Studies in Educational Evaluation 29: 1–12.

Andreas

Burns

Touza

(2017) Renewable energy as a luxury? A qualitative comparative analysis of the role of the economy in the EU’s renewable energy transitions during the ‘double crisis’. Ecological Economics 142: 81–90.

Baumgartner

(2008) Regularity theories reassessed. Philosophia 36(3): 327–54.

Baumgartner

(2013) A regularity theoretic approach to actual causation. Erkenntnis 78(Suppl. 1): 85–109.

Baumgartner

(2015) Parsimony and causality. Quality and Quantity 49(2): 839–56.

Baumgartner

Ambühl

(2020) Causal modeling with multi-value and fuzzy-set coincidence analysis. Political Science Research and Methods 8: 526–42.

10.

Baumgartner

Thiem

(2020) Often trusted but never (properly) tested: Evaluating qualitative comparative analysis. Sociological Methods and Research 49: 279–311.

11.

Beach

(2018) Achieving methodological alignment when combining QCA and process tracing in practice. Sociological Methods and Research 47(1): 64–99.

12.

Beach

Pedersen

(2018) Process-Tracing Methods – Foundations and Guidelines. Ann Arbor, MI: The University of Michigan Press.

13.

Beach

Rohlfing

(2018) Integrating cross-case analyses and process tracing in set-theoretic research: Strategies and parameters of debate. Sociological Methods and Research 47(1): 3–36.

14.

Befani

(2016) Pathways to Change: Evaluating Development Interventions with Qualitative Comparative Analysis (QCA). Pathways to Change: Evaluating Development Interventions with QCA, Rapport Till Expertgruppen För Biståndsanalys (EBA). Available at: http://eba.se/wp-content/uploads/2016/07/QCA_BarbaraBefani-201605.pdf

15.

Befani

(2020) Diagnostic evaluation and Bayesian updating: Practical solutions to common problems. Evaluation 26(4): 499–515.

16.

Braumoeller

(2015) Guarding against false positives in qualitative comparative analysis. Political Analysis 23(4): 844–58.

17.

Coryn

CLS

Noakes

Westine

, et al. (2011) A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation 32(2): 199–226.

18.

Dușa

(2019) Critical tension: Sufficiency and parsimony in QCA. Sociological Methods and Research. Epub ahead of print 18 November. DOI: 10.1177/0049124119882456.

19.

European Commission (2013) EVALSED – The Resource for the Evaluation of Socio-economic Development: Sourcebook – Method and Techniques. Available at: https://ec.europa.eu/regional_policy/en/information/publications/evaluations-guidance-documents/2013/evalsed-the-resource-for-the-evaluation-of-socio-economic-development-sourcebook-method-and-techniques

20.

Fiss

(2011) Building better causal theories: A fuzzy set approach to typologies in organization research. Academy of Management Journal 54(2): 393–420.

21.

Gerrits

Verweij

(2016) Qualitative comparative analysis as a method of evaluating complex cases. Zeitschrift für Evaluation 15: 7–22.

22.

Green

Roberts

Petticrew

, et al. (2015) Integrating quasi-experimental and inductive designs in evaluation: A case study of the impact of free bus travel on public health. Evaluation 21(4): 391–406.

23.

Haesebrouck

(2022) Relevant, irrelevant, or ambiguous? Towards a correct interpretation of QCA’s solution types. Sociological Methods and Research. Epub ahead of print 25 January. DOI: 10.1177/00491241211036153.

24.

Haesebrouck

Thiem

(2018) Burden sharing in CSDP military operations. Defence and Peace Economics 29(7): 748–65.

25.

Haesebrouck

Thomann

(2021) Introduction: Causation, inferences, and solution types in configurational comparative methods. Quality and Quantity. DOI: 10.1007/s11135-021-01209-4.

26.

Hume

(2003) A Treatise of Human Nature. Mineola, NY: Dover Publications.

27.

Johnson

Greenseid

Toal

, et al. (2009) Research on evaluation use. American Journal of Evaluation 30(3): 377–410.

28.

Johnson

Russo

Schoonenboom

(2019) Causation in mixed methods research: The meeting of philosophy, science, and practice. Journal of Mixed Methods Research 13(2): 143–62.

29.

Kahwati

Kane

(2020) Qualitative Comparative Analysis in Mixed Methods Research and Evaluation. Thousand Oaks, CA: SAGE.

30.

Krupnik

(2012) Evaluating the social mechanism of investment subsidies using an abductive approach. Evaluation 18(4): 466–476. DOI: 10.1177/1356389012460958.

31.

Krupnik

Koniewski

Skórska

, et al. (2015) Analiza efektów netto wybranych działań Programu Operacyjnego Innowacyjna Gospodarka 2007-2013, z wykorzystaniem podejścia counterfactual impact evaluation w ramach projektu ewaluacyjnego PARP „BAROMETR INNOWACYJNOŚCI”, Polish Agency for Enterprise Development. Available at: http://badania.parp.gov.pl/images/badania/Raport_Barometr_netto_POIG.pdf

32.

Lankoski

Thiem

(2020) Linkages between agricultural policies, productivity and environmental sustainability. Ecological Economics 178. DOI: 10.1016/j.ecolecon.2020.106809.

33.

Machamer

Lindley

Craver

(2000) Thinking about mechanisms. Philosophy of Science 67(1): 1–25.

34.

Mackie

(1974) The Cement of the Universe: A Study of Causation. Oxford: Oxford University Press.

35.

Meuer

Rupietta

(2017) A review of integrated QCA and statistical analyses. Quality and Quantity 51(5): 2063–83.

36.

Patton

(2008) Utilization-focused Evaluation, 4th edn. Thousand Oaks, CA: SAGE.

37.

Pattyn

Álamos-Concha

Cambré

, et al. (2020) Policy effectiveness through configurational and mechanistic lenses: Lessons for concept development. Journal of Comparative Policy Analysis: Research and Practice 24: 33–50.

38.

Pattyn

Molenveld

Befani

(2019) Qualitative comparative analysis as an evaluation tool: Lessons from an application in development cooperation. American Journal of Evaluation 40(1): 55–74.

39.

Peck

Gorzalski

(2009) An evaluation use framework and empirical assessment. Journal of Multidisciplinary Evaluation 6(12): 139–56.

40.

Pelucha

Kveton

Potluka

(2019) Using mixed method approach in measuring effects of training in firms: Case study of the European Social Fund support. Evaluation and Program Planning 73: 146–55.

41.

Petrosino

(2000) Whether and why? The potential benefits of including program theory in meta-analysis. New Directions for Evaluation. Program Theory in Evaluation: Challenges and Opportunities 87: 59–70.

42.

Radaelli

Wagemann

(2019) What did I leave out? Omitted variables in regression and qualitative comparative analysis. European Political Science 18(2): 275–90.

43.

Ragin

(2014) The Comparative Method : Moving beyond Qualitative and Quantitative Strategies. Berkeley, CA: University of California Press.

44.

Rihoux

Alamos-Concha

Lobe

(2021) Qualitative Comparative Analysis (QCA): An integrative approach suited for diverse Mixed Methods and Multimethod research strategies. In: Onwuegbuzie

Johnson

(eds) The Routledge reviewer’s guide to mixed methods analysis. New York: Routledge, pp. 185–98.

45.

Rihoux

de Meur

(2009) Crisp-set qualitative comparative analysis (csQCA). In: Rihoux

Ragin

(eds) Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques. Thousand Oaks, CA: SAGE, pp. 33–68.

46.

Rihoux

Rezsöhazy

Bol

(2011) Qualitative comparative analysis (QCA) in public policy analysis: An extensive review. German Policy Studies 7(3): 9–82.

47.

Rohlfing

(2018) Power and false negatives in qualitative comparative analysis: Foundations, simulation and estimation for empirical studies. Political Analysis 26(1): 72–89.

48.

Rohlfing

Schneider

(2018) A unifying framework for causal analysis in set-theoretic multimethod research. Sociological Methods and Research 47(1): 37–63.

49.

Rohlfing

Zuber

(2019) Check your truth conditions! Clarifying the relationship between theories of causation and social science methods for causal inference. Sociological Methods and Research 50: 1623–59.

50.

Schneider

Wagemann

(2019) Set-theoretic Methods for Social Sciences. Cambridge: Cambridge University Press.

51.

Schneider

(2018) Realists and idealists in QCA. Political Analysis 26(2): 246–54.

52.

Schneider

Rohlfing

(2013) Combining QCA and process tracing in set-theoretic multi-method research. Sociological Methods and Research 42(4): 559–97.

53.

Schneider

Rohlfing

(2016) Case studies nested in fuzzy-set QCA on sufficiency: Formalizing case selection and causal inference. Sociological Methods and Research 45(3): 526–68.

54.

Schneider

Wagemann

(2012) Set-theoretic Methods for the Social Sciences. A Guide to Qualitative Comparative Analysis. Cambridge: Cambridge University Press.

55.

Stern

Stame

Mayne

, et al. (2012) Broadening the range of designs and methods for impact evaluations. DFID Working Paper 38, April. London. Available at: https://www.oecd.org/derec/50399683.pdf

56.

Tanner (2014) QCA is of questionable value for policy research. Policy and Society 33(3): 287–98.

57.

Thiem

(2014) Navigating the complexities of qualitative comparative analysis: Case numbers, necessity relations, and model ambiguities. Evaluation Review 38(6): 487–513.

58.

Thiem

(2017) Conducting configurational comparative research with qualitative comparative analysis: A hands-on tutorial for applied evaluation scholars and practitioners. American Journal of Evaluation 38(3): 420–33.

59.

Thiem

(2018) ‘Realists’ and ‘Idealists’ in QCA ? A Rejoinder to Schneider (2018). Available at: http://alrik-thiem.net/data/documents/Realists-and-Idealists-in-QCA.pdf

60.

Thiem

Baumgartner

Bol

(2016) Still lost in translation! A correction of three misunderstandings between configurational comparativists and regressional analysts. Comparative Political Studies 49(6): 742–74.

61.

Thomann

(2020) Qualitative comparative analysis for comparative policy analysis. In: Peters

Fontaine

(eds) Handbook of Research Methods and Applications in Comparative Policy Analysis. Cheltenham: Edward Elgar Publishing, pp. 254–76.

62.

Thomann

Ege

(2020) Qualitative comparative analysis (QCA) in public administration. Oxford Research Encyclopedia of Politics. DOI: 10.1093/acrefore/9780190228637.013.1444.

63.

Thomann

Maggetti

(2020) Designing research with qualitative comparative analysis (QCA): Approaches, challenges, and tools. Sociological Methods and Research 49: 356–86.

64.

White

(2009) Theory-based impact evaluation: Principles and practice. Journal of Development Effectiveness 1(3): 271–84.

65.

White

(2010) A contribution to current debates in impact evaluation. Evaluation 16(2): 153–64.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.08 MB