Addressing the Challenges Related to Transforming Qualitative Into Quantitative Data in Qualitative Comparative Analysis

How to Calibrate Qualitative Data in QCA?

An important issue in QCA is the calibration of the raw data. When using crisp-set QCA (csQCA), all cases are either “in” (1) or “out” (0) of the sets. In fuzzy-set QCA (fsQCA), the raw data are calibrated from “fully in” (1) and “fully out” (0) of the sets, with additional gradations of set membership (e.g., “almost fully in” [.83] or “more out than in” [.40]). The 1 and the 0 are two of the so-called qualitative thresholds; the crossover point at 0.5 is the third.

The literature on calibration is mainly concentrated on quantitative data. For example, Ragin (2008, chap. 5) focuses exclusively on this issue, while providing no practical advice for researchers on how to calibrate qualitative data. The same holds for Schneider and Wagemann (2012, pp. 32-41).

The only two studies offering explicit methodological advice on how to calibrate qualitative data in QCA are Basurto and Speer (2012) and Tóth, Henneberg, and Naudé (2017) (see De Block & Vis, 2017 for a more extensive discussion). Basurto and Speer (2012) propose a stepwise procedure to calibrate qualitative (interview) data into qualitative classifications with associated fuzzy-set values (see Appendix B). Tóth et al. (2017) introduce the so-called Generic Membership Evaluation Template (GMET) to assign membership scores to conditions based on qualitative data ² (see Appendix C).

Yet although Basurto and Speer (2012) and Tóth et al. (2017) provide valuable guidelines on how to calibrate qualitative data, some important questions remain. It remains unclear, for example, how qualitative data can properly inform and justify the determination of the qualitative thresholds—especially regarding the crossover point. What is more, while filling in the GMET is rather straightforward, decisions about how to attribute the final fuzzy-set score remain somewhat subjective.

Both Basurto and Speer (2012, p. 169) and Tóth et al. (2017, p. 195) note that existing studies using qualitative data in QCA are typically unclear about how exactly they calibrated their data. The studies usually are not transparent about (1) where they placed the thresholds for inclusion and exclusion of a set (respectively, the 1 and the 0) and (2) how they established the degree to which a case is “in” (0.5 < x≤ 1) or “out” of the set (0 ≤x < 0.5), that is, the degree of set membership. Since results of a QCA analysis can differ substantively depending on researchers’ specific choices on these issues, such transparency is important.

Determining the Thresholds for Inclusion and Exclusion of a Set

How did the studies we reviewed determine where to place the thresholds for inclusion and exclusion of a set? Table 1 summarizes the five main strategies employed, while Appendix D provides additional details for all reviewed studies.

OPEN IN VIEWER

Table 1.

Different Strategies to Determine the Thresholds for Inclusion and Exclusion of a Set.

Strategy	Examples
Develop a rubric/coding scheme to assign codes to outcome and conditions.	Chatterley et al. (2014), Chatterley, Linden, and Javernick-Will (2013), Fischer (2015), Henik (2015), Iannacci and Cornford (2018), and Kirchherr, Charles, and Walton (2016)
Construct an imaginary case for full membership based on the case context and a case for nonmembership based on theoretical knowledge. The thresholds for inclusion and exclusion are then placed somewhere in-between these values.	Basurto and Speer (2012) and Iannacci and Cornford (2018)
Apply the GMET where qualitative anchor points are based on a combination of the positive or negative direction on a case’s membership and the relative importance of the attribute.	Tóth et al. (2017)
Conduct a cluster analysis by using, for example, Tosmana (Cronqvist, 2016).	Kim and Verweij (2016), Li, Kopennjan, and Verweij (2016), and Vergne and Depeyre (2016)
Base the thresholds on a large gap in the numerical data between the various cases (and preferably complement this with other approaches).	Li et al. (2016) and Vergne and Depeyre (2016)

Note. GMET = Generic Membership Evaluation Template.

A first strategy is developing a rubric or coding scheme to assign codes for the outcome and the conditions. Chatterley et al. (2014), for example, develop a rubric to assign codes based on their data from interviews, focus groups, and observations (see Appendix E for an overview of the types of qualitative data used in all the reviewed studies). Whereas these codes are useful to rate the conditions and outcome for each case, Chatterley et al. (2014) do not provide a justification for assigning the thresholds for inclusion and exclusion of a set. Kirchherr et al. (2016) base the calibration of some fuzzy-set values on existing quantitative indices and of other values on an iterative process of multiple semistructured expert interviews and an online survey. While the thresholds for inclusion and exclusion of a set are rather straightforward for data based on indices (e.g., when a ranking is used), it is unclear how Kirchherr et al. (2016) determined thresholds based on the qualitative data.

A second strategy is suggested by Basurto and Speer (2012, see Iannacci & Cornford, 2018, for an application). Basurto and Speer (2012) construct two imaginary ideal cases, one representing full membership in a set (1) and one representing full nonmembership (0). The thresholds for inclusion and exclusion of the set, then, are put in-between the two “extreme” values.

A third—inductive—strategy that several studies adopt is to set the thresholds using QCA software, particularly the threshold setter in Tosmana (Cronqvist, 2016). ³ Exploring the possibly large gaps in the data is another inductive strategy. Note that these inductive strategies are applicable only when the raw data are already numerical. Yet for a study based exclusively on qualitative data, these strategies are not an option; as a result, researchers are restricted in these cases to applying Tóth et al.’s (2017) GMET or using one of the first two strategies listed above.

Establishing the Degree of Set Membership

While the overview in Appendix D shows that almost all fsQCA studies are careful about establishing the degree of set membership, ⁴ it also reveals that many studies are not fully transparent regarding how the qualitative data were used to this end.

For example, Verweij (2015) used both qualitative and quantitative data to calibrate the outcome and the conditions. As with studies using a similar approach (e.g., Vis, 2010), the quantitative material “dominated” the calibration (i.e., it was the benchmark that could be adjusted based on the qualitative material). One of Verweij’s (2015) conditions was calibrated based on various qualitative documents, with codes assigned using qualitative data analysis software. The few small coding contradictions were then recalibrated in a final step (Verweij, 2015, p. 1883). While the latter is common practice in QCA—as well as in many qualitative studies—it is not clear exactly how and why this recalibration was done. As a consequence, it is difficult to replicate studies that employ this approach. The same holds for Verweij, Klijn, Edelenbos, and Van Buuren’s (2013) study, which used various qualitative sources to calibrate their outcome and conditions. In line with good QCA practice, Verweij et al. (2013) published their coding scheme and the resulting scores in an appendix, allowing other researchers to assess whether “the observations meaningfully capture the ideas contained in the concepts” (Adcock & Collier, 2001, p. 529) and thereby are “valid” (i.e., that a given value makes sense given existing empirical and theoretical knowledge). Yet these tables do not include the reasoning behind the coding decisions and, therefore, cannot be reproduced fully.

Similarly, Van der Heijden (2015) used a systematic coding scheme and qualitative data analysis software to explore data on voluntary environmental programs systematically and gain insights “into the ‘repetitiveness’ and ‘rarity’ of experiences shared by the interviewees, and those reported in the existing information studied” (p. 581). However, Van der Heijden (2015) did not discuss how this information was subsequently used to code cases as “in” or “out” of the set. Other studies go over the coding decisions only briefly (e.g., Chatterley et al., 2014) or provide no information on how the interview data were translated into the fuzzy-set values (e.g., Basurto, 2013). To varying degrees, this lack of transparency inhibits the studies’ replicability.

Some studies use multiple coders to establish the degree of set membership. In Henik (2015), for example, two coders applied a coding rubric on transcribed interviews, with the average of these coders’ scores constituting the final set attribute. Henik (2015) notes that the coders “agreed within 0.25 set membership points on more than 90% of the 960 items” (p. 445). However, it is unclear how qualitatively important differences across coders were addressed, existing when one coder codes an item as being “in” the set and the other as “out.” In this regard, a discrepancy of 0.15 (e.g., 0.45 vs. 0.6) can be more relevant than one of 0.3 (e.g., 0.6 vs. 0.9).

Regarding which values to assign to qualitative data (i.e., the quantitizing), the studies we reviewed offer some suggestions. Table 2 lists the strategies, while Appendix D provides a more comprehensive overview.

OPEN IN VIEWER

Table 2.

Different Strategies to Determine the Degree of Set Membership.

Strategy	Examples
Use predetermined options in an interview (e.g., Likert scale)	Fischer (2014)
Use a coding scheme (e.g., 4-value and 2-value fuzzy sets) to assign membership scores to attributes and subsequently: a. Average the calibrated values. b. Take the minimum value (when all attributes of a concept are necessary). c. Take the maximum value (when all attributes are sufficient).	Kirchherr et al. (2016) Chatterley (2014) Basurto and Speer (2012)

One strategy is to directly ask interviewees to provide answers on a Likert-type scale (or one based on other predetermined options). This strategy is applied by Fischer (2014), who calibrated his outcome (policy change) by asking approximately 250 interviewees to rate their perception of policy change from 1 to 5. Next, Fischer (2014, pp. 350-351) averaged the perceptions of actors and subsequently calibrated these data into fuzzy sets by rescaling the average value to a 0 to 1 scale. ⁵ Another strategy is adopted by Kirchherr et al. (2016), who used a 4-value and 2-value coding scheme to assign set-membership scores to the attributes. Subsequently, Kirchherr et al. (2016) averaged the calibrated values for the different attributes of the conditions. They addressed this strategy’s potential weakness, as it “could introduce misfits between the verbal meaning of a concept and its operationalization” (Kirchherr et al., 2016, p. 39), by reviewing all averaged calibrations of the conditions and changing or recalibrating the attributes when they found that the conditions’ values did not correspond to their averaged operationalization. Alternatives for taking the average value are substitutability (i.e., taking their maximum value) or taking the weakest link (i.e., the minimum value of the attributes of the concept; Ragin, 2000, see Basurto & Speer, 2012; Chatterley et al., 2014, for examples).

How to Present the Calibration Process and the Data?

To make studies replicable, the data sources and calibration process need to be presented transparently and comprehensively (Gerring, 2012). Ideally, this should also be done concisely to make the material easily accessible. These goals—transparency and comprehensiveness versus conciseness—often conflict. What is more, even transparency and comprehensiveness may conflict, as researchers aiming to be comprehensive risk burying their readers in details, thereby hindering transparency. How QCA scholars present the calibration process, and hence the actual possibility for replication, varies strongly across the reviewed studies. Table 3 summarizes the material from Appendix D on this.

OPEN IN VIEWER

Table 3.

Different Strategies to Present the Calibration Procedure.

Approach	Examples
Table in main text, full information	Kirchherr et al. (2016) and Tóth et al. (2017) for 1 GMET
Table in main text, partial information	Basurto (2013), Chai and Schoon (2016), Chatterley et al. (2013), Chatterley et al. (2014), Crilly (2011), Hodson and Roscigno (2004), Hodson, Roscigno, and Lopez (2006), Iannacci and Cornford (2018), Kim and Verweij (2016), Li et al. (2016), Metelits (2009), Summers Holtrop et al. (2016), Vergne and Depeyre (2016), Verweij (2015), Verweij et al. (2013), and Verweij and Gerrits (2015)
Text boxes	Basurto and Speer (2012) and Mishra et al. (2017)
Discussed in words in main text, typically partial	Chai and Schoon (2016), Chatterley et al. (2013), Crilly (2011), Henik (2015), Iannacci and Cornford (2018), Kim and Verweij (2016), Kirchherr et al. (2016), Li et al. (2016), and Verweij (2015)
Discussed in words in appendix, typically partial	Smilde (2005) and Vergne and Depeyre (2016)
Table(s) in appendix, full information	Fischer (2014, 2015), Iannacci and Cornford (2018), Kirchherr et al. (2016), Li et al. (2016), Thomann (2015), Van der Heijden (2015), Verweij et al. (2013), and Wang (2016)
Table(s) in appendix, partial information	Basurto (2013) and Hodson and Roscigno (2004)

Note. GMET = Generic Membership Evaluation Template.

Table 3 demonstrates that most reviewed studies (n = 27) provide some information on the calibration procedure (Aversa, Furnari, & Haefliger, 2015; Crowley, 2012, provide too little information). Numerous studies provide substantial information, but not all that would be required for full transparency.

Some studies’ data calibration procedures make them easier to replicate than others. Kim and Verweij (2016), for example, included a table with the motivation of the assignment of U.S. states to a specific category based on a combination of descriptions and secondary survey data. Fischer (2014) presented the calibration of outcome and conditions in tables in appendices. Both studies use a rather straightforward approach to calibration by, respectively, referring to survey results and directly asking interviewees to “score” their outcome and conditions, subsequently taking the average. Hence, replicating these findings is also rather straightforward.

Arriving at similar results becomes more complicated when the data needed for a specific attribute cannot be directly derived from interviewees’ answers. While journal space limitations often make the disclosure of all details of the calibration process challenging, using (online) appendices, an option available at a growing number of journals, is one way to give more insight in the argumentation of researchers (Basurto & Speer, 2012). This suggestion is taken up by a variety of the reviewed studies (Basurto, 2013; Fischer, 2014; Kirchherr et al., 2016; Thomann, 2015).

Which Sensitivity Tests to Conduct?

Testing findings’ robustness by means of sensitivity analyses should be part of a good QCA study (Schneider & Wagemann, 2012). The methodological literature on QCA pays increasing attention to sensitivity tests (Baumgartner & Thiem, 2017a; Marx, 2010; Skaaning, 2011; Thiem, 2014; Thiem, Baumgartner, et al., 2016), including how to deal with different types of errors (Maggetti & Levi-Faur, 2013). In addition, the literature criticizing QCA (e.g., Hug, 2013; Lucas & Szatrowski, 2014; Paine, 2016) regularly indicates that the alleged lack of findings’ robustness is a key problem (but see Baumgartner & Thiem, 2017b).

The QCA literature provides several suggestions on how to assess the robustness of QCA findings using sensitivity tests. A nonexhaustive list includes (1) dropping or adding cases and conditions, (2) changing fuzzy-set membership functions, (3) altering consistency thresholds (Schneider & Wagemann, 2012; Thiem, 2014; Thiem, Spöhel, & Duşa, 2016), (4) changing the definitions of the set values, (5) using alternative measures for a concept (Basurto & Speer, 2012), (6) changing the calibration thresholds of raw data into set membership, and (7) altering the frequency of cases linked to configurations (Skaaning, 2011). These suggestions are not specific to qualitative data. Changing the consistency thresholds, for example, can be done irrespective of whether the data used are qualitative, quantitative, or both (see for examples with qualitative data Kim & Verweij, 2016; Tóth et al., 2017). Similarly, changing the frequency of cases linked to the configuration can be done irrespective of the kind of data used. Still, the higher the number of cases, the more appropriate this sensitivity test becomes. Since studies using qualitative data often—though not always—have a relatively low number of cases, this will in many cases not be the most important sensitivity test to conduct. Some researchers conduct additional statistical analyses to assess the robustness of their findings, despite criticism about the comparability of the two methods (e.g., Thiem, Spöhel, et al., 2016). For example, Hodson et al. (2006) investigated whether their QCA-generated configurations were associated with the outcome and whether the association was statistically significant. Hodson et al. (2006) also introduced multivariate controls by creating dummy variables specifying key configurations and including them in a linear model. Note that while combining QCA and statistical analyses might be of interest to the readership of JMMR, we do not discuss this further since it is not specific to QCA studies using qualitative data.

Based on the reviewed literature, we selected those sensitivity tests that are relevant for QCA studies using qualitative data. We list these in Table 4. Appendix D provides a more detailed overview for all reviewed studies.

OPEN IN VIEWER

Table 4.

Relevant Sensitivity Tests for Assessing the Robustness of Qualitative Comparative Analysis Findings Based on Qualitative Data.

Approach	Examples
Drop or add cases motivated by extensive case knowledge.	Kirchherr et al. (2016)
Alter the attributes of a condition based on knowledge about the case context.	Kirchherr et al. (2016)
Replace conditions by one of their attributes based on the importance that the data from the interviews, documents, or literature assigned to a specific attribute.	Kirchherr et al. (2016)
Rerun the analysis with a new, more extreme, outcome that has—consequently—a different qualitative breakpoint (anchor point) for being “in” the set. Go back to the qualitative data to calibrate this new outcome (which can be done starting from the original outcome’s calibration).	Fiss (2011) and Tóth et al. (2017)

First, the available qualitative data can be a strong motivator to decide which cases to drop or add in the sensitivity analysis. Dropping cases can be a useful way to assess findings’ robustness. Kirchherr et al. (2016), for example, included an extensive section on robustness in which they tailor their choices to exclude certain cases based on case descriptions presented in an appendix. However, when dropping cases, it is important to make sure that the cases-to-conditions ratio is still acceptable—typically one condition to three cases (Marx, 2010). If this ratio becomes too low, the results become unreliable.

A second type of sensitivity test is conducted by altering the different attributes of the condition (Kirchherr et al., 2016)—for example, to base the membership score on only one attribute rather than on multiple ones. Here as well, the motivation for such choices must be based on knowledge about case context (e.g., that the now omitted attributes introduced noise to the condition’s operationalization). Another related option is to replace the condition by one of its attributes, a decision that can, for example, be based on the importance assigned to the specific attribute in the interviews, relevant documents, or literature.

Another type of test, which we subsume here under the heading of sensitivity tests but which is technically a test to better determine which factors or mechanisms “drive” the outcome, is conducted by Tóth et al. (2017, p. 202), who follow Fiss (2011). A new outcome is introduced that is more extreme than the original (in Tóth et al., 2017: very high relational attractiveness of the customer [RAC]). The qualitative threshold (the “anchor point,” in Tóth et al.’s (2017) terminology) for being “in the set” is higher for “very high RAC” than it was for “RAC,” meaning that some cases will no longer be “in” the set of this new outcome. The calibration of the outcome requires returning to the qualitative data and assigning appropriate (fuzzy) set values, where the calibration of the original outcome can be used as a starting point.

Considerations When Using Qualitative Data in QCA

Based on the studies we reviewed, we highlight five considerations for using qualitative data for QCA (summarized in Appendix F). First, QCA researchers should be more explicit about how they arrive at certain thresholds for inclusion and exclusion of a set. Depending on the type of data (to be) collected, these thresholds might, for example, be determined by constructing an imaginary ideal case or be based on a classification of interview responses.

Second, researchers should be more explicit about how they determined the degree of set membership. More specifically, the reasoning behind the coding of qualitative data and the subsequent translation of qualitative codes into fuzzy-set scores should be more clearly communicated in articles or (online) appendices (see also the fourth point below). Qualitative data or codes can be linked to values on a Likert-type or other predetermined numerical scale (potentially based on quantitative material) and subsequently translated into fuzzy-set values. Moreover, rubrics or coding schemes (e.g., with two or four values) or predetermined qualitative classifications can be used as an intermediate step for assigning fuzzy-set values to qualitative data.

Third, QCA researchers should pay more attention to the zeros in their calibrated data. Crucially, they must be careful about distinguishing between cases whose condition(s) or outcomes are coded zero because they are “not mentioned” (or not identified in, e.g., documents) versus cases whose condition(s) or outcomes are coded zero because they are “truly absent.” To avoid this ambiguity when using interview data, researchers should attempt to construct their interview scheme such that all concepts are addressed during the interview (although Sandelowski et al.’s (2009) option—that the analyst did not see “it,” even though it was there—would then still be a possibility). Creating a separate section for each condition and the outcome in the interview guideline, as proposed by Basurto and Speer (2012), is one possibility to doing so. The same holds for Tóth et al.’s (2017) suggestion to draw up an initial template based on previous literature. When all concepts are addressed in an interview, a value of “0” would then be assigned only to attributes or conditions that are truly absent. However, due to the iterative nature of QCA, which allows for the inclusion and exclusion of conditions during the process, a lack of data about one or more attributes or conditions cannot always be avoided.

A similar data deficiency can also occur when analyzing preexisting data for QCA. We provide two options to deal with such data gaps. First, in cases where such an approach is possible, interviewees can be recontacted about the attributes or conditions for which information is missing. This is the ideal solution, since it allows researchers to establish whether it was indeed absent, or whether it was just not mentioned in the initial interview. When it is not possible to go back to the interviewees however—for example, because of practical constraints—a second-best option is to conduct sensitivity analyses. Three sensitivity analyses are particularly apt for addressing the zero issue: (1) removing the conditions where this problem occurs and assessing the effect, (2) assigning the value “0.51” (i.e., just “in” the set) to cases of which the researcher is not sure whether the condition is “truly” absent to differentiate between the two findings, and (3) excluding the cases where the concept is “not mentioned” from the analysis.

Fourth, to increase a study’s transparency and comprehensiveness, and hence its replicability, QCA researchers should explicitly delineate the choices they made (to the extent that this is possible given issues of, e.g., confidentiality). We agree with Schneider and Wagemann’s (2010) advice to publish the raw data matrix in addition to a detailed discussion of the calibration of the set membership scores. When a data set is too large to be published, the original data should be made available on the Internet or, if that is impossible, on demand. Large data sets, including transcribed interviews and reports, often exist when using qualitative data for QCA. To present the data in a transparent yet concise way, a balance should be sought in giving brief explanations and/or illustrations in the main text and using tables in the main text and/or in (online) appendices.

Finally, our review showed that although conducting sensitivity tests in (qualitative) QCA should be common practice, this is still not the case. Various tests are particularly suited to dealing with qualitative data, such as changing the number of cases, altering the conditions, or rerunning the analysis with a more extreme outcome.

Transforming Qualitative Into Quantitative Data in QCA: What Lessons for Mixed Methods Research?

The considerations in the previous section are first and foremost meant for QCA researchers using qualitative data. However, as Cragun et al. (2016) show, QCA’s hybrid nature offers several advantages over other methods and is therefore interesting for mixed methods researchers more generally.

Our considerations regarding calibration specifically relate to the discussions in JMMR on quantitizing. Discussions have been held about “the foundational assumptions, judgments, and compromises involved in converting qualitative into quantitative data” (Sandelowski et al. 2009, p. 208)—for example, on what and how to count. Debates about how to quantitize qualitative data are not new to MMR (e.g., Boyatzis, 1998), and the topic is usually included in MMR text books (e.g., Miles, Huberman, & Saldaña, 2014). Typically, as in Teddlie and Tashakkori (2009, pp. 270-271), examples are presented as to how qualitative data have been quantitized or on how researchers have generally proceeded—for instance, by Sandelowski et al. (2009):

A common approach to quantitizing is to use the results of a prior quantitative analysis of quantitative data as the framework for the conversion of qualitative into quantitative data. This framework provides the decision rules for a directed form of content analysis whereby a priori codes are derived from a quantitative data set and applied to a qualitative data set. (p. 218)

However, as with the studies reviewed above, the more detailed choices made by researchers frequently go undiscussed, alongside their underlying reasoning. Consequently, the methodological MMR literature provides little guidance for researchers seeking to quantitize their qualitative data. Since such choices may also influence the substantive results of an MMR study, they must be clearly communicated. What is more, the transparency and hence replicability of MMR studies would increase if they were more explicit about the choices made and the reasoning underlying these choices regarding quantitizing.

Conversions from qualitative into quantitative data “are by no means transparent and uncontentious” (Love, Pritchard, Maguire, McCarthy, & Paddock, 2005, p. 287, in Sandelowski et al., 2009). Our considerations regarding the presentation of the calibration process increase the transparency and replicability of studies where quantitization is used.

Given that quantitizing in MMR is to some extent subjective, it is relevant for MMR to conduct sensitivity tests to assess the robustness of the findings. Some of the sensitivity tests that we identified as relevant for QCA using qualitative data are also relevant for MMR that includes quantitizing; this is especially the case for studies in which the (in)dependent variables (conditions) include several subdimensions (attributes). Specifically, three of the sensitivity tests mentioned above are particularly appropriate to MMR: (1) dropping or adding cases based on extensive case knowledge, (2) altering the attributes of a condition based on knowledge of the case context, (3) and replacing conditions by one of their attributes.

Considerations on Quantitizing Beyond the QCA Literature

Although this article focused on QCA studies, research using methodologies other than QCA also provide valuable insights about quantitization. This can be illustrated using examples from various scientific fields. In education research, the study of Gilmore, Maher, Feldon, and Timmerman (2014) quantitized data from 65 interviews to assess the relationship of participants’ teaching experiences and teaching support systems with changes in their teaching orientation over time. They covered this longitudinal aspect by calculating the changes in coding scores between pre- and postinterviews. Moyer-Packenham et al. (2016) conducted pre- and postassessments of quantitized video data when studying the role of affordances in children’s learning performance. As their study makes clear, using quantitized codes derived from sources based on different points in time is a useful consideration when investigating developments over time.

When considering on how to deal with zeros in the data, Gilmore et al. (2014) suggest using multiple imputation procedures to fill the missing data. In the area of health research, Chang, Voils, Sandelowski, Hasselblad, and Crandell (2009) describe how qualitative labels for the number of respondents per specific finding on antiretroviral adherence—such as “few” or “many”—can be transformed in exact numbers—such as 2 or 50. They conducted an online survey at nursing school faculty to obtain lower and upper limits for specific verbal labels and, subsequently, used the responses in regression analyses to estimate a plausible range of respondents in a given study. Sandelowski (2000), in turn, uses the study of Borkan, Quirk, and Sullivan (1991) as an example of quantitizing. In this study, the researchers use narrative analysis to determine the main categories of how elderly people viewed the hip fractures from which they suffered. A series of reliability tests were then conducted to ensure the consistency of the categories. Both studies provide additional insights on the issue of how to establish the degree of set membership.

An example from economics comes from Vaitkevicius (2013), who suggests a systematic coding procedure based on hermeneutics to code qualitative data and subsequently analyze these data quantitatively. This procedure is, for instance, applicable to code and analyze closed-ended and open-ended questions. A final example also proposes a procedure for open-ended—qualitative—survey questions. Rohrer, Brummer, Schmukle, Goebel, and Wagner (2017) suggest the employment of tools from natural language processing to process and analyze potentially large numbers of answers to open-ended questions. They demonstrate their procedure by analyzing the more than 35,000 answers to the question “What else are you worried about?” from the participants of a German socioeconomic panel study. These examples can be used as a starting point for expanding the list of considerations to be reflected on in MMR.

Conclusion

This article addressed the challenges that researchers face when using qualitative data in QCA, especially when it comes to transforming it into quantitative data. Although QCA training courses are offered worldwide and several textbooks and journal articles that include hands-on instructions have been published, specific guidance for the use of qualitative data in QCA has been largely absent. We addressed this lacuna by exploring the various ways in which researchers currently use qualitative data in QCA and by laying considerations on three key issues: (1) the calibration of qualitative data (known as quantitization in MMR), (2) the presentation of the calibration process and the data, and (3) sensitivity testing. Overall, our study demonstrates that many QCA studies using qualitative data are not as transparent in their procedures as would be required to enable proper replicability.

We thus presented five main considerations for QCA researchers aiming to enhance their studies’ transparency: first, researchers should be more explicit as to how they arrive at the thresholds for inclusion and exclusion of a set; second, they should be clear about how they determined the degree of set membership; third, more attention should be paid to the “zeros” in the calibrated data; fourth, researchers should make more explicit and present clearly the choices they made during the calibration process; and finally, conducting sensitivity tests should become common practice. These considerations contribute to the methodological discussions on data calibration and quantitization. Moreover, our study provides QCA users, and readers of JMMR more generally, with ideas about how to transform qualitative data into quantitative form in their empirical studies. Which consideration(s) a given researcher ultimately takes into account will depend, among other things, on the specific research question, the type of data, and available time and resources.