Clarifying a basis for qualitative generalization using approaches that identify shared culture

Introduction

Researchers typically obtain data from samples of various sorts and seldom study an entire population. The sample data are analysed and interpreted, and researchers are then faced with considering the extent to which the findings can be generalized. In quantitative research designs, there is a broad consensus that generalization is achieved using random samples and statistical methods. In qualitative research designs, typically with non-random samples, the views about generalization are more varied. Some argue that generalization is either not a priority (Denzin, 1983) or is not possible (Hammersley, 1990). For others (e.g. Sarantakos, 1993: 53), higher-level analytic or conceptual work provide a basis for generalization.

While the topic of generalization is well documented for quantitative research, it is less well documented for qualitative research. Generic sources of advice such as encyclopaedias are relatively mute on the topic of qualitative generalization. For example, The Sage Encyclopedia of Qualitative Research Methods (Given, 2008) has a modest entry (Donmeyer, 2008) under ‘generalizability’, which splits its account evenly between quantitative and qualitative issues. Its treatment of qualitative generalization briefly gives two explanations of how knowledge from a single or a limited number of cases can be applied to other settings. The Sage Dictionary of Social Science Methods (Jupp, 2006) has no entries explicitly on qualitative generalization but does include an entry on quantitative generalization under the topic of inferential statistics. A fuller treatment of qualitative generalization is found in The Sage Encylopedia of Social Science Research Methods (Lewis-Beck et al., 2004) in which Williams (2004: 420) states that this issues lies at the heart of the qualitative approach but ‘is discussed less than one would expect’. He suggests that this occurs because many qualitative advocates see the issue as a preoccupation of those taking a nomothetic approach. Countering those who deny the possibility of generalization, Williams goes on to introduce the idea of ‘moderatum generalization’ or moderate claims about the social world that depend on shared culture or cultural consistency in the social environment.

An exception among the reference books is The Sage Handbook of Social Research Methods (Alasuutari et al., 2009) in which Gobo (2009) thoroughly re-conceptualizes generalization. He rebuts the methodological denigration of generalization in qualitative research as overly severe and unjustified in three ways: by noting problems associated with statistical inference; by showing that there are numerous disciplines whose theories are based on few cases; and by arguing for ‘idiographic sampling theory’. He reviews five positions that qualitative researchers have taken on generalization and argues that there is a lack of reflection on how to emancipate qualitative generalization from subordination to statistical inference. Consequently, there remains a need for a more rigorous sampling theory. Gobo (2009) develops four suggestions for idiographic sampling theory, one of which is to search for social regularities via three modes of inference – the deductive inference, the comparative inference, and the emblematic case. We will return to Gobo (2009) in the conclusion to this article where we observe that the mixed method approaches proposed here are consistent with the emblematic case mode of inference he advocates.

The modest coverage of qualitative generalization in the encyclopaedias is matched in the journal literature. A Web of Science search yielded only one article (Payne and Williams, 2005) with the title key words of ‘qualitative generalization’. (Using ‘qualitative inference’ yielded more articles but most of these relate to Bayesian applications in computer science.) Payne and Williams (2005) note that in one year of publications in the journal Sociology, some of the qualitative studies did make, to greater or lesser degree, generalizations, in some cases while denying that this was their purpose. They advocate that qualitative social science researchers should more explicitly consider the extent of their generalizations and the grounds for their claims. While many researchers working with qualitative data have given thought to this issue, it remains true that for many others the only basis for generalization is statistical inference. We contend that it remains important to consider methods and approaches that are germane to the topic of qualitative generalization.

This article contributes to knowledge by showing how some mixed methods research approaches provide a basis for qualitative generalization. These approaches include a first step that is largely quantitative and establishes shared culture, and then a second step that is largely qualitative and provides an in-depth understanding of the topic in question. In this article we will focus on two such mixed-method approaches. One is cultural consensus analysis (CCA) (Romney et al., 1986; 1987) combined with cultural modelling (D’Andrade, 1984; Holland and Quinn, 1987). Recent advocates include Garro (2000) and Rinne and Fairweather (in press). This approach is referred to here as CCA&CM. The other approach is Q methodology (Brown, 1980; McKeown and Thomas, 1988; Stephenson, 1953), which involves a quantitative Q sort and data analysis phase then a qualitative interpretation phase. These two approaches are similar and offer a basis for qualitative generalization.

There are some important qualifiers to our argument. Other approaches may well provide a basis to generalize and we are not suggesting that the two approaches covered here are the only available bases. Further, we are not implying that qualitative researchers satisfied with the status quo and their own basis for belief in qualitative generalization need to revize their thinking. Our purpose is to contribute to knowledge about qualitative generalization and in doing this we highlight two approaches that appear particularly well suited to achieving this goal.

This article begins with an account of a basis of qualitative generalization in shared culture following Williams (2000) and then gives a description of CCA&CM and Q methodology. It then describes how the two approaches are similar. Finally, we argue that these two approaches provide a basis for moderatum generalization by virtue of their explicit focus on shared culture and are consistent with idiographic sampling theory (Gobo, 2009). We also suggest that some other mixed method approaches provide a similar basis for qualitative generalization.

Qualitative generalization via ‘moderatum generalization’

For Williams (2000), moderatum generalizations are moderate claims about the social world that are not meant to hold true over long time periods or across cultures. They are moderately held and therefore open to change. This implies that they need to be regularly tested and modified if necessary. Williams rejects the pure interpretivist position that generalization is not possible. In establishing this position, he cites Geertz’s (1979) study of Balinese cock fighting as telling us about Balinese society not just cock fighting, and Fisher’s (1993) study of teenage fruit machine players and their motivations for gambling as indicative of gambling behavior in general. In his study of fruit machine players, Fisher contends that policy developed from the findings of the study can be applied effectively to other instances of gambling behaviour.

Williams (2000) specifies three types of generalizations – total (where the phenomenon is an instance of a general law), statistical, and moderatum – and explains how the latter considers the sample as bearing those characteristics necessary to infer to a wider population. He considers two possible foundations for moderatum generalization, analytic induction and theoretical inference. Analytic induction is the iterative process of starting with a definition of the phenomenon to be explained, examining a few cases, formulating hypotheses about these cases, examining further cases to test these hypotheses, and, where there is negative evidence, either redefining the phenomenon to exclude the case or reformulating the hypothesis. Alternatively, theoretical inference entails developing knowledge of the necessary relationships that exist among categories of phenomena and this, in turn, provides the basis of generalization.

Williams (2000) rejects analytic induction as an adequate basis for generalization since it finds only cases that fit the causal pattern and, therefore, only finds necessary conditions, but not the sufficient conditions, for the outcome to occur. Regarding theoretical inference, Williams argues that a promising basis for this is the Weberian ideal type (Weber, 1949) where the study of a limited number of cases can provide a basis on which to generalize. As part of using ideal types as the basis of theoretical inference, Williams (2000) observes that some generalizing statements are intuitively more correct than others and the basis for this is the cultural consistency that makes social life possible. Social life requires patterns in behaviour that are distinctive for the particular culture in which it occurs. These patterns allow social interaction to occur and provide for regular patterns of social behaviour, rather than reinventing them prior to each interaction. In any culture there is a given format of appropriate behaviours and associated meanings in all social settings. For example, in some countries it is typical that people do not bring photographs of the deceased to the funeral, while in other countries (e.g. Tonga) people do bring photographs of the deceased to funerals. Cultural consistency in each country thus occurs and observation of these and the associated meanings makes possible the development of ideal types and moderatum generalization. Cultural consistency for us means the same thing as shared culture.

For Weber (Coser, 1977: 222, quoting Shils and Finch, 1949) ideal types are ‘formed by the one sided accentuation of one or more points of view and by the synthesis of a great many diffuse, discrete, more or less present, and occasionally absent concrete individual phenomena, which are arranged according to those one-sidedly emphasized viewpoints into a unified analytic construct’. The ideal type specifies how the necessary relationships between categories of phenomena work and focuses on patterns of behaviour made possible because of shared culture. Ideal types may include patterns of macro-sociological development such as the Protestant Ethic, or patterns that may be found in a variety of historical and cultural contexts such as bureaucracy (Coser, 1977). An ideal type never corresponds to concrete reality but abstracts from it in a logically precise and coherent whole (Coser, 1977). Even everyday activities such as drinking in a public bar can be studied in these terms since people know what behaviours work in bar settings and usually follow these conventions. Observation and interpretation of these behaviours can lead to the development of an ideal type.

Having explained how ideal types and shared culture provide a basis for moderatum generalization it is appropriate now to describe two research approaches that specifically focus on cultural consistency and therefore provide a basis for qualitative generalization.

Cultural consensus analysis and cultural modelling (CCA&CM)

Cultural consensus analysis is a quantitative method while cultural modelling is a qualitative method based on discourse analysis. The use of two different yet converging methods can aid in the development of ‘a more complete and representative model than is possible through the use of only one approach’ (Garro, 1988: 99). According to Garro (2000: 285) ‘both cultural consensus and cultural models approaches could productively be applied in a converging manner to learn about underlying cultural knowledge and intracultural variation for a given topic’. Garro (2000) utilized cultural consensus analysis and cultural models to study how the Ojibway understood diabetes and its causes.

Cultural consensus analysis (CCA) (Romney et al., 1986, 1987) uses quantitative techniques to assess the nature of the culture that is shared. It asks three primary questions. First, it asks if shared knowledge of a specific cultural domain exists within a group of subjects. Second, consensus analysis assesses the relationship of each subject’s knowledge of the domain in question with the aggregate knowledge of that domain. Third, consensus analysis determines the ‘culturally correct’ answers to the survey questions without knowing or assuming the correct answers ahead of time. In other words, consensus analysis does not compare subject’s responses to survey items to an established answer key. Rather, the answer key, the content of ‘culture’, is estimated mathematically from the patterns of data.

In effect, the analysis determines what is consistent or reliable across the subject responses. The focus is on each subject, specifically on how they have responded to each question and how their responses across questions compare to the general pattern of other participant responses. CCA asks ‘What is the relationship of each individual informant’s command of the knowledge of the domain to the knowledge possessed by the aggregate?’ (Swora, 2003: 343). Cultural competence refers to ‘How much of a given domain of culture each individual informant “knows”’ (Romney et al., 1986). Cultural competence scores for each person are estimated by factoring a matrix of person-by-person similarity coefficients. Thus, if a subject is estimated to have a cultural competence in a particular domain of 0.8, he or she is estimated to command 80 per cent of the knowledge of that domain. More ‘expert’ subjects, those with higher cultural competence scores, agree with each other more frequently, and demonstrate greater cultural competence.

Overall, CCA allows the researcher to estimate the social distribution of knowledge, the culturally correct answers to the questions, and the average knowledge possessed by each member of the sampled population. In particular, it enables a researcher to determine if there is sufficient sharing in response to structured questions to make it reasonable to infer that participants are drawing on a single cultural model (Romney et al., 1986). If this condition is met, then it is appropriate to prepare a single cultural model. The resulting model would be basic and include a list of the key attributes of shared culture. Such models can be developed further, however, by using qualitative data derived from interviews or surveys using open-ended questions. While CCA may focus on a single model, it can also show the presence of multiple models.

Cultural models are those presupposed, taken-for-granted models of knowledge and thought that are used in the course of everyday life to guide a person’s understanding of the world and their behaviour (D’Andrade, 1984). They are also the constructed representations made by researchers in order to describe shared knowledge and perceptions used by groups of people in their daily lives (Blount, 2002; Cooley, 2003). Cultural models systematically draw on personal discourse – the representations, practices and performances through which meanings are produced, connected into networks and legitimized (Gregory, 2000). Discourse analysis allows researchers to get the insider’s perspective on participant knowledge, thought and word meaning. In some cases, the cultural models include causal explanations of the topic being considered. Advocates of cultural modelling may consider that the models themselves provide a basis for generalization and we do not take issue with this view. However, in contexts in which such a claim is likely to be contested it may be necessary to provide additional analysis to support such claims, and this is provided by combining qualitative cultural modelling with quantitative cultural consensus analysis.

Q methodology

Q methodology was invented and developed by William Stephenson (1953), applied and codified by Brown (1980), and in recent years has been argued to be relevant to geography (Robbins and Kreuger, 2000), rural research (Previte et al., 2007), environmental policy (Addams and Proops, 2000), and psychology (Watts and Stenner, 2005). Like CCA&CM, Q methodology identifies patterns of shared belief, but unlike CCA&CM it emphasizes distinctive patterns of belief, usually from two to four patterns. Fundamental to Q methodology is the goal of understanding human subjectivity.

The research process entails uncovering patterns of beliefs about the phenomenon being studied and the researcher seeks to understand how the participants experience their world. Essentially, summarizing Previte et al. (2007), Q methodology requires the researcher to give attention to the discourse under investigation; that is, the views held by the participants, and then identify a concourse or range of issues within that discourse. From this concourse a Q sample of 30 to 60 statements (or other items such as photographs) is prepared in either a structured or unstructured way. Participants are then asked to sort the items into a Q sort distribution (usually a normal distribution) following a ‘condition of instruction’ (e.g. ‘please identify the statements with which you strongly agree and strongly disagree’). Statements placed at the ends of the distribution are allocated high scores while statements placed in the middle of the distribution are allocated low or neutral scores. While the statement selection focuses on including a representative range of items, the participant selection, typically of about 30–40 people, emphasizes diversity. Finally, data from the assigned scores in the Q sort are used for correlation, factor analysis and factor rotation using specific computer programmes. Participants load onto Q factors where each Q factor is formed as an archetype of those who sort in a similar way. Each Q factor thus represents the views of the group of participants but is distinct from any one person’s Q sort. Each Q factor has a distinctive array of statements, which match the Q sort distribution, and each statement has a score. The Q factor is interpreted using the process of abduction to sequentially develop an explanation that fits the data associated with that array of statements, supplemented by comments or discussion recorded during the Q sort.

In our view, Weber’s ideal type best applies to the interpreted Q factor since the factor is an idealized amalgam derived from those subjects who load on to it. While some subjects have a high loading or correlation to the Q factor, none are actually identical to it and it remains an abstraction derived from the properties of its constituent members. However, Stephenson’s view was different, and since our argument builds on ideal types it is relevant to consider what Stephenson thought on this topic. Stephenson (1962) recognized compatibility between Q methodology and ideal types. He argued that Q methodology provides the basis of a pure social science so that what leading social scientists were doing was ‘readily reducible to Q-methodological terms’ (p. 12). However, he considered a single individual’s Q sort to correspond to an ideal type (p. 12), although, in addition, he considered that when social scientists express types in abstract formal ways, that ‘Q-sorts, Q-factors and factor space pursue these formal ways’ (p. 13). Further, for the task of ideal types being used as conceptual instruments for comparison and measurement with reality, he said that Q method was comparative but that ‘[i]t is a mistake to think of ideal types as factors attainable by many persons or as average of many Q-sorts or as stereotypes. Operationally, one expert’s Q-sort may represent the most significant data in factor space’ (p. 14). Perhaps because he was emphasizing how single cases represented by single Q sorts can still be important he overlooked how Q factors can be seen as ideal types.

Q methodologists have considered the issue of qualitative generalization. Bass and Brown (1973) address this issue and argue that research based on a single case can be generalized. Their psychiatric study of displacement behaviour was based on one person performing 30 Q sorts, analysis of which yielded four Q factors. In effect, they used one person to generate a number of cases. Bass and Brown (1973: 182) presume that there are other people who have similar ‘structured mental worlds that they can represent according to the rules prescribed’. Further, Brown and Ungs (1970) argue that findings from a Q study of responses to the Kent State University shootings in 1970 identified general patterns of thinking because Q methodology emphasizes the comprehensiveness and representativeness of the sample of items presented to subjects, rather than on a comprehensive and large random sample of people. What is needed, however, is a fuller explanation of the basis for making qualitative generalizations, even if care is used in the selection of items used for Q sorting. A fuller explanation is what this article provides.

Methodological similarities between CCA&CM and Q methodology

There are remarkable similarities between CCA&CM and Q methodology. Both methods use small, non-random samples, subjects not variables are analysed, the data analysis is weighted toward those subjects who are found to be more strongly reflective of the general pattern, and one or more patterns in the data are interpreted. Each of these similarities is considered in turn. Of these four similarities, three are quantitative in nature. Our comparisons, therefore, focus more attention on quantitative similarities. These quantitative similarities are the means for establishing the extent to which culture is shared and for showing the key elements of that shared culture. Although we focus more strongly on the quantitative components of these two methods in this note, we do not want to draw attention away from the importance of the qualitative part. It is the qualitative component that provides the full account of the nature of shared culture.

CCA requires very small sample sizes to reach statistical significance, as can be seen in work by Dressler et al. (1998), Garro (1988), Romney et al. (1987) and Swora (2003), with samples ranging from 10 to 20 participants. According to Weller and Romney (1988), sample size should be determined by focusing on three factors: the cultural competence of the participants, the confidence levels required, and the proportion of questions required to be classified correctly. Assuming an average cultural competence level among informants of 0.5 or higher, which seems reasonable in studies of general culture patterns, and classifying 90 per cent of the questions correctly, a confidence level of 0.95 can be achieved using only 13 participants.

In Q methodology, the standard approach is to select a diverse, non-random sample of people. It is suggested that samples sizes be no more than 40 (Brown [1980: 92] cited in Addams [2000]). McKeown and Thomas (1988) state that a sample sizes can range from one to 100 but they emphasize that sample size depends on the nature and purpose of the study. Recent applications of Q methodology typically focus on limited diversity (2–4 factors) with samples of less than 60 people (e.g. Cairns, in press).

In both approaches, there is, as in any research that records quantitative responses to questions, a subject by variable data matrix. Quantitative analyses of such data focus on patterns in the variables and responses to each question, and the scores across all respondents are examined. In CCA&CM and Q methodology the focus is on finding consistency or stability in the pattern of responses across the subjects.

During this analysis, those subjects who more strongly typify the patterns found are given more weighting: in CCA&CM, the more ‘expert’ subjects have a proportionately greater influence on the patterns found, and in Q methodology, subjects with a higher factor loading have a proportionately greater influence on the characteristics of the Q factor.

Both approaches can identify one or more patterns of culture. Q methodology is perhaps more dedicated to characterizing a small number of patterns. It does this by adding the additional step of factor rotation, in which the variance associated with the first main factor is distributed across other factors. It can also identify single patterns by using just the un-rotated first factor. In addition, the analysis identifies consensus items, those items that receive a similar score from all subjects.

A qualification is needed about the two methods. The use of statistical methods should not be taken to imply that the quantitative phase in each method provides a single or invariant indication of the cultural patterns found. In fact, each method admits some indeterminacy, in large part because the statistical analysis entails many options. This is quite pronounced in Q methodology since there are different factor algorithms available and ample options for researcher judgements. In her use of CCA, Garro (2000) specifically documented departure from consensus analysis to provide a more detailed account of cultural variation. This indeterminacy is entirely consistent with the qualitative focus of these methods. For an ideal type, there is no one way in which the ‘one-sided accentuation’ necessarily occurs.

CCA&CM and Q methodology as a basis for qualitative generalization

Both approaches have a quantitative phase that establishes shared culture, and a second qualitative phase that enables in-depth interpretation. Since both approaches identify shared culture, their results provide a basis for generalization to a wider population. In the qualitative phase, interpretations are built on patterns derived from earlier quantitative results. Because culture has been established as shared, it is then possible to establish the particular character of the common patterns and express these as ideal types. In essence, both approaches highlight shared culture, that is, what is culturally consistent. This consistency allows the ideal type patterns to have relevance in other settings. Researchers can be confident that others in the population will share the patterns of belief and meaning identified in the sample. In studies where only one general pattern is identified there can be greater confidence that the results from the sample give a good indication of population characteristics. What is not addressed by qualitative research is the issue of the proportion of the types that may exist in the population. The frequency of the different types in the population can be determined using statistical methods.

Examples of Q methodology in New Zealand research on public perceptions of nature and the environment (Fairweather and Swaffield, 2001, 2002; Newton et al., 2002) provide evidence supporting the validity of generalization. Their research identified two perceptions of the environment: ‘pure nature’ and ‘cultured nature’. These two perceptions were found for different stakeholders across multiple settings in more than one study. These consistent empirical findings support the claim that the two patterns in the way landscapes are seen can be expected to occur across the population.

Some other mixed-methods approaches have characteristics similar to CCA&CM and Q methodology in that they identify patterns or ideal-typical structures of meaning in social life and therefore offer a basis for qualitative generalization. These include ethnographic decision tree modelling (Gladwin, 1989), qualitative comparative analysis (QCA) (Ragin, 1987, 2000, 2008), and causal mapping (Bryson at al., 2004; Fairweather, 2009; Fairweather and Hunt, 2011). In ethnographic decision tree modelling, the tree specifies how someone comes to a decision to do something. There are multiple pathways through the logical structure of the decision tree that specifies the particular combinations of motivations and other factors that bear on the decision. Each combination is, in effect, an ideal type. Further, the tree enables prediction of decision outcomes from a given set of decision criteria – people sharing the same set of beliefs as those in one branch of the decision tree will likely reach the same decision outcome. This structure of the decision tree allows generalization beyond the sample. In QCA, the analysis specifies the conditions under which the study outcome occurs. The inference is that where these conditions exist in other settings the same outcome will occur because of the combination of causally necessary and sufficient conditions. For causal mapping, each individual prepares a causal map that shows the factors considered to be important in the settings under study and the causal relationships between factors. The maps are then aggregated to show the common patterns of understanding of how the system works. This understanding will apply in other similar settings.

All of the methods mentioned in this article rely on the application of abductive reasoning to generate the synthetic findings. Abduction requires the researcher to come up with explanations that are consistent with the raw data. In CCA&CM, the preliminary CCA process is relatively straightforward in that the key cultural elements are readily identified but cultural modelling, in which the relationships between the cultural elements are elaborated, requires considerable effort and the developed model must remain true to the data. In Q methodology, factor interpretation demands that the researcher develop a viable explanation for the particular ordering of items in each factor. In ethnographic decision tree modelling, the decision tree emerges from repetitive and cumulative interrogation of each interview transcript until the tree is developed. The process can become unwieldy and appear to expand uncontrollably with multi-stage models being developed as the researcher develops a deeper understanding of the subtleties of decision making. In QCA, there is constant interaction between theory and data as the best set of causal conditions are winnowed from the available evidence. In causal mapping, the multiple linkages between factors demand an account that identifies the core elements of the system under study and how these elements interact. It needs to be acknowledged that this abductive feature of these mixed methods means that they are not so amenable to easy teaching, or of providing a set of key steps to follow in a simple logical order. At times during the research process there are no clear guidelines and the researcher works slowly to a conclusion that is defensible. The process can be exhaustive and, at times, frustratingly indeterminate. Arguably, because of this character they are demanding methods to use but no more demanding than other qualitative approaches.

Conclusion

This article contributes to knowledge by showing how mixed method approaches such as CCA&CM and Q methodology offer a basis for qualitative generalization by showing that there is shared culture and providing a means to characterize it. Some qualitative researchers may find the considerations in this article to be largely self evident, if not redundant, but as Payne and Williams (2005) argue, more care is needed when it comes to generalization in qualitative research.

Extending this position, Gobo (2009) has argued that we need to reconceptualize generalization to provide a stronger basis for qualitative inference and he has advanced the idea of idiographic generalization. One of his suggestions is to search for social regularities via three modes of inference (pp. 204–7). The first is deductive inference (using a critical or deviant case to test a theory), and the second is comparative inference (choosing diverse cases). The third is the typical or emblematic case in which the cases represent a significant feature of a phenomenon with the focus on identifying general structures detached from individual social practices. This mode of inference is not generalizing an individual case or event but the ‘key structural features of which it is made up, and that are to be found in other case or events of the same species or class’ (p. 206). Clearly, our emphasis on ideal types is consistent with Gobo’s third mode of inference, the typical or emblematic case. Further, the use of diverse, non-random samples in the approaches described is consistent with Gobo’s second mode of inference, comparative inference derived from choosing diverse cases.

These considerations on qualitative generalization or idiographic sampling are important for advancing knowledge about qualitative research. These advances also provide a platform for making convincing arguments to those who seek to better understand the generalizability of qualitative research.