Abstract
Quantitative cross-cultural databases can help uncover structure and diversity across human populations. These databases have been constructed using a variety of methodologies and have been instrumental for building and testing theories in the social sciences. The processes and assumptions behind the construction of cross-cultural databases are not always openly discussed by creators or fully appreciated by their users. Here, we scrutinize the processes used to generate quantitative cross-cultural databases, from the point of ethnographic fieldwork to the processing of quantitative cross-cultural data. We outline challenges that arise at each stage of this process and discuss the strengths and limitations of how existing databases have handled these challenges. We suggest a host of best practices for cross-cultural database construction, and stress the importance of coding source meta-data and using this meta-data to identify and adjust for source biases. This paper explicitly discusses the processes, problems, and principles behind cross-cultural database construction, and ultimately seeks to promote rigorous cross-cultural comparative research.
Introduction
Humans speak thousands of different languages, revere hundreds of thousands of gods, and engage in a wide range of social and moral customs. Understanding cross-cultural diversity and patterns in human cognition and behavior is a central goal of the social sciences. Comparative databases built from existing ethnographic source materials provide a powerful way to systematically aggregate and analyze cross-cultural information. Researchers have used ethnography based cross-cultural databases—and independent cross-cultural codes—to derive remarkable insights into cultural diversity (Kirby et al., 2016). Because of cross-cultural comparative research, we now have a better understanding of how different gender roles and marriage systems evolve (Colleran et al., 2015; Cowlishaw & Mace, 1996; Wood & Eagly, 2002), how ecological and social pressures shape religious belief systems (Botero et al., 2014; Watts et al., 2015a), how subsistence styles influence cooperation norms (Peoples & Marlowe, 2012; Sheehan et al., 2018; Strassmann & Kurapati, 2016), why cultural groups develop such diverse forms of pedagogical practices (Barry & Roberts, 1972; Ellis et al., 1978; Ember & Ember, 1994), and how warfare influences cultural norms (Ross, 1983; Sosis et al., 2007) amongst many other findings 1 . Much of this research focuses on cultural groups prior to processes of economic and technological modernization, colonization, and conversion to major world religions. With care, this information can help build a picture of the diversity of human cultural systems prior to the homogenizing processes of globalization.
Cross-cultural comparative databases have traditionally been the domain of anthropologists, but have now become an interdisciplinary endeavor. Researchers from fields such as psychology (Jackson et al., 2020), sociology (Stark, 2001), religious studies (Slingerland & Sullivan, 2015), and history (Turchin et al., 2015), as well as interdisciplinary teams combining data and theory from multiple fields, have all used comparative cross-cultural methods to supplement and expand traditional disciplinary methodologies. Unlike field site comparisons, cross-cultural coding studies can sample large numbers of cultural groups (anywhere from several dozen to over a thousand), offering high power for culture-level comparisons. Additionally, cross-cultural coding can develop samples of both contemporary and historical cultural groups. This enables comparative research to examine trajectories of cultural change over time, retrodict patterns from the past, and minimize issues that stem from globalization and cultural convergence among contemporary nations.
Scholars have been building large-scale cross-cultural databases since the 19th century (Tylor, 1871), but it was not until the 1960s that systematic cross-cultural studies began to be used regularly. Examples of published cross-cultural databases include the Ethnographic Atlas (Murdock, 1967), the Standard Cross-Cultural Sample (Murdock & White, 1969), Binford’s (2001) comparative Database of Hunter-Gatherer societies, and the Pulotu database of Pacific societies (Watts et al., 2015b). These databases follow a “standard” model of database construction that has a number of shared features. First, these databases have been built using existing ethnographic materials as their source of research data, rather than collecting data directly from the cultural groups of interest. Second, they focus on transforming qualitative descriptions into quantitatively coded variables (M. Ember, 1997). Third, they focus on comparing populations or groups of individuals, not the individuals themselves. Here, we focus specifically on discussing this standard model of database construction.
Recent cross-cultural databases still rely on the standard model of database construction, but have expanded and adapted this approach in a number of ways. For example, the Seshat database of polities over time (Turchin et al., 2015) and the Database of Religious History (Slingerland & Sullivan, 2015), include historical summaries and archaeological records in addition to ethnographic materials. The Natural History of Song also differs from this standard model in that it focuses on coding the features of songs within a population, rather than the characteristics of the population itself (Mehr et al., 2019). We focus specifically on the standard approach to cross-cultural database construction, though many of the points we raise here also apply to other cross-cultural databases as well.
Since their popularization in the 1960s, there has been critical discussion of methods and procedures for cross-cultural coding (Bradley, 1989; Ember, 1986; Ember & Ember, 2009; Ember & Otterbein, 1991; Naroll, 1968, 1970; Otterbein, 1976; Whiting, 1968). The issues raised in these discussions tend to focus on the practices and concerns of early cross-cultural databases, such as the Standard Cross-Cultural Sample and Ethnographic Atlas. Recent cross-cultural databases raise additional potential issues, as well as illustrate that many of the concerns raised in these debates remain relevant and unresolved. Slingerland et al. (2020) highlight some issues in contemporary cross-cultural databases, focusing on the relative merits of RA and expert based coding decisions, documentation and accountability in coded data, and issues around maintaining database infrastructure. This existing literature contains a wealth of resources for those interested in building cross-cultural databases, but has a number of important gaps. For example, with the exception of Slingerland et al. (2020), there has been relatively little explicit discussion of open science principles, such as transparency and documentation of research methodologies. Although discussion of bias and data quality were in the past widely discussed by cross-cultural researchers (see Ethnographic Research Process and Ethnographic Record), cross-cultural datasets published recently rarely contain such discussions. These methodological concerns are foundational to the reliability and validity of cross-cultural database construction (Ember et al., 1991; LeCompte, 1987), but there remains little consensus on how best to resolve them.
Here, we critically examine the data creation process behind ethnographic records, raise potential issues that this process poses for the construction of cross-cultural comparative datasets, and provide an overview of methods for dealing with these biases. We start by discussing three difficulties to defining cultural populations (Population). This section includes a discussion of nested populations structures (Challenge: Nested Population Structures), variation within populations (Challenge: Variation within Populations), and change in populations over time (Challenge: Change in Populations Over Time). In the second section, we discuss biases in the ethnographic research process (Ethnographic Research Process). This section includes discussion of an ethnographer’s personal biases (Challenge: Ethnographers Personal Biases), biases in the interactions of an ethnographer with others (Challenge: Interaction Biases), and biases in the features of a culture that are published (Challenge: Recording Biases). In the third section, we discuss challenges in interpreting the ethnographic record (Ethnographic Record). This section focuses on the ambiguity in source materials (Challenge: Ambiguity Within Source Materials), and inconsistency between sources (Challenge: Inconsistencies Between Sources). Across all of these sections, we both present challenges to researchers, as well as evaluate potential solutions. We conclude by discussing general principles for building comparative cross-cultural databases from ethnographic records. We aim to highlight the complexities of building cross-cultural databases, best practices in database construction, and hope to encourage dialog between people actively building these databases.
Population
The first stage of the data creation process involves determining a focal population of people (or specific cultural units). People live in communities that change in structure and features over time, contain internal variation, and have fuzzy and layered boundaries. Ethnographers often define their population of interest by wherever they happen to pitch their tent (Whiting, 1954). This means that ethnographers often describe focal communities within a broader culture. A challenge of building cross-cultural databases from ethnographies is ensuring that the units of analysis are meaningfully comparable.
The difficulties of defining units of analysis are not unique to cross-cultural research. Linguists face similar difficulties when defining languages and biologists face similar difficulties when defining species (Calude et al., 2017; Mallet, 2005). Despite these difficulties, comparative research methods in these fields have vibrant and productive research programs (Bromham et al., 2015; Dunn et al., 2011; Harvey & Pagel, 1991). These fields have achieved productive research programs by using pragmatic definitions for the purposes of research, while continuing to debate and refine theoretical issues in more philosophical literature (Currie, 2016). There is no reason that cross-cultural research needs to be any different. Below we outline challenges for selecting populations when building cross-cultural comparative datasets, discuss how existing databases have defined cultural units, and then evaluate the approaches that these databases have taken.
Challenge: Nested Population Structures
Humans have long lived in complex and nested social groupings (Bird et al., 2019). The Island of Sumatra provides an illustrative example. Over the course of the second millennium, this island was part of the Melayu Kingdom, the Singhasari Kingdom, the Kingdom of Aceh Darussalam, the Dutch East Indies, the Japanese occupation of the Dutch East Indies, and Indonesia (Loeb, 1935). Throughout this time, the island retained numerous distinct cultural families with distinct languages and cultural systems. One of these families, called the Batak, inhabit an inland region of Northern Sumatra. This family of peoples speak related but mutually unintelligible languages, have distinct customs and practices, and are politically and geographically separate. The largest of these groups, the Toba Batak, had over half a million people during the Dutch rule of Sumatra, and recognized their own “priest-king” called Si Singamangaraja (Sibeth, 1991). This priest-king ruler was the figurehead of a hierarchical political system comprising districts, villages and clans. Researchers could potentially be interested in writing about the kingdoms that Sumatra was a part, the Island of Sumatra as a unit, the Batak cultural family, the Toba Batak, and/or specific communities within the Toba Batak (Sibeth, 1991). While different groupings will be better suited to particular research questions, a challenge here is deciding which of these groupings to include in a comparative database that can be used to address a range of research questions.
Early databases such as the World Ethnographic Sample (Murdock, 1957), the Ethnographic Atlas (Murdock, 1967), and the Standard Cross-Cultural Sample (Murdock & White, 1969) interchangeably use the terms “Societies” and “Cultures” to refer to their units of analysis. The authors of these databases went to great lengths to spell out how populations in these samples were selected from different world regions. However, the papers documenting the construction of these databases provide little explanation on how the boundaries of each population were actually defined. In the case of Sumatra discussed above, the terms “society” and “culture” could conceivably refer to the group of villages that make up a district, the Toba Batak, the Batak peoples, or the broader political state of the Island of Sumatra. In practice, the Ethnographic Atlas identifies the Toba Batak as their unit of analysis, but it remains unclear how and why the authors chose this particular grouping.
Approach: Selecting Units of Analysis
Recent cross-cultural databases have more clearly defined their units of analysis, but have taken substantially different approaches to defining them. The Pulotu database defines its units of analysis as “a group of people living in a similar physical, social and economic environment that speak mutually intelligible languages and have relatively homogenous supernatural beliefs and practices” (Watts et al., 2015b). This narrows the definition down considerably more than “society,” but of course still leaves open what other high-inference factors (e.g., “relatively homogenous,” “supernatural”) mean. As a way of clarifying what is meant by homogeneity, the Pulotu definition is followed by examples of what would be counted as relatively homogenous for the purposes of the database, and what would not. While ostensive definitions might not provide an exact definition, they can help provide a benchmark for comparing units of analysis. In the case of Sumatra, the Pulotu database includes the Toba Batak as a cultural unit. This is also the same unit used in the Ethnographic Atlas, which suggests that despite lacking a clear definition of “society,” similar principles were applied, at least implicitly (Murdock, 1967).
In the Database of Religious History, coders are able to specify a wide range of grouping units and define their own time spans (Slingerland & Sullivan, 2015). The predefined categories of units in this database include regions, religious groups, religious places, places, and polities. The variables coded are different across unit categories, limiting the ability to draw comparisons between them. Furthermore, there is substantial room for variation even within each predefined category. For example, the category called religious groups includes The Oneida Community, a group of 300 people living within the city of Oneida, New York during a period of 33 years. The category religious groups also includes the Central African Iron Age, a grouping that includes everyone within a region of Africa that now spans eight nation states during a period of 1850 years. Comparing vastly different units means that differences in traits, such as population size, ritual frequency, and social complexity, could potentially be products of differences in the scale of units chosen by coders. For example, the region of Africa included in the Central Africa Iron Age may itself have included new religious movements similar in scale and complexity to The Oneida Community. This presents a challenge to systematic cross-cultural comparisons as researchers must account for differences in scale when comparing the traits of religious groups.
The Seshat database, on the other hand, focuses on “natural geographic areas” which code human populations in these areas over time, irrespective of the people inhabiting that region at that time (Turchin et al., 2015). This means that there is not necessarily any cultural continuity in populations over time. For example, the people inhabiting the Deccan area were conquered by people from the Kachi Plain area, marking a cultural discontinuity in the Deccan area profile (Beheim et al., 2021). This provides an opportunity to study the impact of warfare on social change (Turchin, 2007) but means that patterns of change in traits (e.g., marriage customs, gender norms, and political systems) over time do not necessarily reflect processes of cultural evolution within a cultural lineage. As such, the Seshat database might be more appropriately considered a comparative database of geographic populations rather than a comparative database of cultural populations. The focus on natural geographic areas in Seshat may reflect the focus of the kinds of historiographical literature it incorporates in addition to ethnographic sources.
Ultimately, the most appropriate unit of analysis will depend on the kinds of research questions that a study seeks to answer, as well as the kinds of sources being used to code cultural groups (Naroll et al., 1964, p. 305). If the researcher is conducting a study involving political variables, polities may be the appropriate unit, whereas smaller communities may be chosen as the unit of study by a researcher investigating practices of gift exchange (Whiting, 1954). In any case, it is important to ensure that researchers consistently define populations to make the units as comparable as possible, and that there is a standard system of defining one’s population of interest (see Slingerland et al. (2020) for further discussion). Explicitly defining and documenting units of analysis also helps other researchers identify which datasets can be meaningfully combined and compared 2 .
Challenge: Variation Within Populations
People within communities clearly vary in many traits due to processes of cultural transmission, genetic differences, and individual experiences (D’Andrade, 1987). For example, in Kédang culture, people are widely reported to believe that humans emerged from some other physical form, but differ about whether this form was a goat, bamboo or a rock (Barnes, 1974). How should one characterize their explanations of human origin? There can also be variation in higher-level features of a population, such as the organizational structure of populations. For example, the Bauans of Central Fiji had ruling chiefs who held authority over confederacies of vanua 3 (Scarr, 1984). Some vanua included a single koro (village), while other vanua were composed of several koro linked by a koro turanga (capital city/political center). This means those koro on their own vanua only had one level of hierarchy beyond the local community (the ruling chief), while other koro had two levels of hierarchy beyond the local community (the koro turanga and then the ruling chief). While this description ignores additional complexities in supralocal hierarchies (e.g., tikina) and sublocal hierarchies, it suggests that Bauans could be coded as having either one or two levels of political hierarchy beyond the level of a local community.
Existing cross-cultural databases generally code an entire culture (or community within a culture, if that is the unit of analysis) with a single value per trait (Murdock & White, 1969; Watts et al., 2015b), though some exceptions do exist (for instance, some of the codes in the Ethnographic Atlas are followed with additional codes indicating alternative values (Murdock, 1967)). Coding an entire culture with a single value can be straightforward when the trait of interest is at the population level, such as the total number of people in the culture or levels of political integration. However, things become more complicated when coding traits that can vary in the population of interest, such as in the case of Bauan political complexity.
One of the difficulties of interpreting data from the Ethnographic Atlas is that there is no explicit explanation of how these kinds of variation are dealt with (Murdock, 1967). For example, Tahitians are coded in the Ethnographic Atlas as having two levels of jurisdictional hierarchy beyond the local community. Like the example of Bauan political complexity highlighted above, Tahitians also had multiple politically autonomous groups with different levels of political complexity (Oliver, 1974). The reason that the Ethnographic Atlas codes Tahitians as having two levels of jurisdictional hierarchy could be because this is the most representative state within the cultural group, that it is the highest level of the variable observed, or simply based on limited information about the culture. Without knowing how this decision was reached, it can be difficult to understand what the data actually represents, how reliably it is coded, and what kinds of comparisons can be made in cross-cultural analyses. While it has been recommended that researchers re-code a sample of data to gain a better understanding of the extent of variation within populations and how it was coded (Whiting, 1968), we argue that databases should make their coding procedures publicly available and as explicit as possible.
Approach 1: Collapsing Variation Within Populations
One approach to addressing variation within populations is to process and simplify data according to a predefined method. For example, when coding something like post-marital residence locations, the most common pattern of occurrence within the community might be coded. Pulotu provides an example of a database that systematically collapses variation within populations. The approach taken by Pulotu was to establish general principles for dealing with variation, and only deviate from these principles when specified in the variable definition (Watts et al., 2015b). For example, when there was variation in political complexity within a cultural group, the highest level was coded. An advantage of collapsing variation is that it can be more efficient and reduce the amount of time taken to code variables for a culture. Collapsing variables can also result in a simpler data structure, which can minimize the complexity of subsequent data analyses.
A drawback of collapsing variation within a cultural group is that information about a variable is lost. As a result, a narrower range of research questions can be asked of the data than the kinds of approaches outlined below. For example, if researchers wanted information on the most representative state of political complexity for the Pulotu sample, rather than the highest level, the variable would have to be re-coded. Coding separate variables on the highest and most representative states levels of political complexity in a society would partly address this issue. However, this approach would still not capture the full range of variation in political complexity within a society. Variation within populations is interesting to study in its own right (e.g., Shaver, 2015) and collapsing to the cultural group level generally precludes the possibility of studying within-population variation. Because of these limitations, collapsing variation within cultural groups may only be advisable when a dataset is built to test specific hypotheses rather than to provide a general resource.
Approach 2: Quantifying Variation Within Populations
Another approach to dealing with variation within and between communities is to design a coding system that quantifies the extent to which a trait varies within a population. For example, the Database of Religious History (Slingerland & Sullivan, 2015) allows coders to specify whether the coding decisions for a variable apply to (a) religious specialists, (b) elites, and/or (c) non-elites. This means that the codes of religious practices, such as sacrificial funeral rites, can tell researchers whether this practice occurred generally in the cultural group, or only for those of high or low status. This provides a principled way of quantifying variation specifically designed to understand how different segments of a population may vary, though care must be taken to ensure any variation coded occurs within the same time period, and is not simply the result of cultural change (see Challenge: Change in Populations Over Time below).
Examples of Approaches to Quantifying Variation Within Populations.
Quantifying variation within populations addresses issues of collapsing variation within populations and enables variation to be studied in its own right. Understanding cultural variation within populations remains largely under-explored by cross-cultural researchers, but has important theoretical implications for social psychology, anthropology, and cultural group selection (Soltis et al., 1995). For example, understanding what aspects of a cultural system are most standardized within a population might help identify group identity markers, social pressures, and provide a marker of trait functionality.
Challenge: Change in Populations Over Time
The features and boundaries of human populations change over time. These changes can occur through processes such as disease epidemics, warfare, colonization, technological innovations, trade, drift, and migration. Change in populations presents both a challenge and an opportunity for cross-cultural datasets. Challenges include how best to define time foci, how to ensure that time foci are meaningfully comparable, and how to use ethnographic records based on observations by different ethnographers—whose observations each may be colored with particular biases and theoretical perspectives (see Challenge: Ethnographers Personal Biases. below)—at different points in time. Opportunities provided by cultural change include being able to study the processes that shape cultural systems and testing causal processes about the relationships between traits. All major existing databases specify the point(s) in time a cultural profile documents, but do so in different ways.
Approach 1: Specifying a Single Time Foci
Many comparative databases, such as the Standard Cross-Cultural Sample, Ethnographic Atlas, and Binford’s Database of Hunter-Gatherer Societies, include a variable that specifies a single year that a cultural profile aims to capture (Binford, 2001; Murdock, 1967; Murdock & White, 1969). In practice, these databases draw on source information from a range of different periods, rather than just from a single year. For example, Binford’s cultural profile on the Ona has a time focus of 1880 but cites texts without dates of fieldwork (Cooper, 1946), fieldwork from 1919–1923 (Gusinde, 1931), and fieldwork from 1965–1976 (Champman, 1982). Differences between the time foci specified in databases and the year of fieldwork research means that cross-cultural databases sometimes make coding decisions entirely based on retrospective accounts of informants, or inferences about the earlier features of cultural groups based on later observations. These kinds of retrospective sources and inferences can be particularly challenging when cultural groups have undergone substantial social change between the time focus and the date(s) of fieldwork. In the case of the Ona, the population suffered from genocide, resettlement and missionization between the time focus and the dates of fieldwork (Beierle, 1996). While many of the sources used by Binford (2001) do provide a rich account of the Ona at the time point of interest, presenting a single year time focus glosses over the complexity and uncertainty in the coding process. If this approach is taken, one could consider coding meta-data for variables based on whether the coding decision has been made from retrospective accounts or from data that falls outside the coded time focus for a cultural profile (see Approach 2: Coding Source Meta-Data. below). This would allow researchers using the database to consider these variables separately and avoid having to individually verify that the codes for each variable apply to the time point of interest (Ember & Ember, 1992).
More recent databases specify the time range that a cultural profile documents. For example, the Database of Religious History allows coders to specify the time range for which a cultural profile applies (Slingerland & Sullivan, 2015) and the Pulotu database specifies a time span of 25 years for its early time foci (Watts et al., 2015b). These ranges more accurately reflect the nature of the source material used to code cultural profiles but can still include materials from within and outside the time foci. Providing a time range also has the potential to more accurately reflect the nature of variables that are based on the frequency of an event over longer periods of history, such as the frequency of warfare (Ember & Ember, 1992).
A general issue with defining a single time focus for each culture—irrespective of whether it is a single year or time range—is the challenge of ensuring the time foci used in cultural profiles are consistently suited to the research questions. This does not necessarily mean that data for each cultural profile should be coded for the same time period. For example, if researchers are interested in understanding non-industrial subsistence practices, or religion prior to influence by major world religions, the most appropriate point in time to code will depend on the specific histories of each cultural population.
Approach 2: Multiple Time Foci Per Cultural Profile
Recent databases include multiple time foci per cultural profile. The Seshat database is based on Natural Geographic Regions coded at 100-year intervals from the late Neolithic to the early modern period (Turchin et al., 2015). The Database of Religious History allows people to generate profiles for the same or overlapping populations at different points in time (Slingerland & Sullivan, 2015). The Pulotu database codes cultural groups before modernization, their contemporary state, and the time span in between (Watts et al., 2015b). In principle, coding multiple time foci per population has the advantage of allowing researchers to test hypotheses about how populations change over time and the relative sequence with which different traits have arisen in human history.
Coding populations over time creates a challenge of ensuring that the populations coded at different points are meaningfully related. As previously mentioned, the natural geographic unit approach taken by Seshat means that there is not necessarily any cultural continuity between units over time (Turchin et al., 2015). In the case of the Database of Religious History, it is up to particular researchers to define the units and boundaries of a religious group (Slingerland & Sullivan, 2015). This means that similar cultural groups coded at different points in time may differ in the scale of population and time spans covered. When using data to study change over time, researchers need to take care to distinguish between differences in traits due to changes in the cultural system itself, differences in the way cultural groups are defined, and differences in the time scale being coded.
The Pulotu database codes the same linguistic and cultural group, living in the same geographic region, at each time focus (Watts et al., 2015b). In principle, this approach would help ensure comparability of groups over time. However, the vast majority of the variables in this database concern the earliest time point of a culture, and these variables do not overlap with the variables at subsequent time foci. This reflects the relatively short time depth of written records of Austronesian speaking peoples and the researchers’ interest in the religious systems prior to the spread of major world religions (Watts et al., 2015b). It means, however, that researchers must rely upon modeling approaches to infer patterns of cultural change over time (Gray & Watts, 2017), rather than being able to observe these directly within the data.
As is evident from this discussion, there are limitations to how all existing databases have coded change in cultural profiles over time. In part, these limitations reflect the fact that ethnographic research is a relatively recent activity, and that detailed written records of cultural systems were relatively scarce prior to the rise of ethnographic fieldwork in the 19th century. We now turn to discussing how some of the content of these sources raises concerns regarding the reliability of source material and its subsequent coding.
Ethnographic Research Process
Challenge: Ethnographers Personal Biases
Early missionary, explorers, and naturalist records provide some of the first written descriptions of many world cultures. These authors were typically white, middle-aged, educated, European men from Christian-majority states. This means that the interests and biases of this demographic are over-represented in source material. These qualities also constrain the kind of experiences, events and information that the writer is likely to have, find noteworthy, and record. For example, in cultural groups where there are gender divisions in roles and events, male ethnographers are unable to participate or potentially learn the details of a substantial portion of a cultural system such as gender segregated ritual practices (e.g., Slocum, 1975; Weiner, 1976).
Additionally, many of these early sources are written by authors untrained in ethnographic fieldwork and who varied in the extent to which they recognize their own theoretical, religious and cultural biases. In some cases, authors funneled observations of other cultural groups into their own cultural framework. For example, one common form of bias among missionaries and Christian scholars in the early 20th century was the doctrine of Urmonotheismus, also known as “primitive monotheism.” According to this doctrine, all peoples started with Abrahamic-like monotheistic religions, but these religions were corrupted to different extents by polytheistic beliefs. Evidence of this bias can be seen when authors explicitly support original monotheism, talk about the elements of Abrahamic-like religious systems that have been lost, or regularly seek to interpret the beliefs and practices of a foreign culture as components of Christianity (e.g., Schmidt, 1931).
In other cases, authors seek to demean, exoticize or over-emphasize cultural differences. References to groups of people as “primitive,” “savage,” “noble-savage,” the use of ethnocentric language, or sweeping derogatory statements about a culture are usually indicative of this over-emphasis on differences, and can make it difficult to determine whether recorded observations reflect the characteristics of a culture or are a result of biased ethnography. Sometimes cultural differences may be inadvertently over-emphasized even by trained anthropologists, a phenomenon Naroll and Naroll (1963) refer to as the “bias of exotic data,” where researchers who have been taught to look for behavior different to that of their own culture end up overlooking or exoticizing behavior with which they are already familiar.
While trained ethnographers and anthropologists may be more likely to explicitly acknowledge their own biases and perspectives, they are also subject to theoretical commitments that bias their interpretation of cultural systems. For example, in the late 19th and early 20th century, European scholars widely endorsed unilineal and hierarchical conceptions of socio-cultural evolution (Tylor, 1889). These accounts assume that all cultural groups evolved along a common trajectory which progresses from “savage” peoples, up to the ultimate (Euro-centric) form of civilized society. Aside from the racism inherent in unilineal models of evolution perspectives, they are also problematic because they encourage authors to fit the features of the culture into specific cultural prototypes, in which things like political leadership, kinship systems, and inheritance systems were assumed to be tightly clustered. Scholars now recognize that there are no rigid pathways of cultural evolution, that features of cultural systems are rarely tightly coupled, and cultural systems cannot be ordered in terms of the extent of their evolution as they are simply shaped by different processes and histories (Mesoudi, 2011). This being said, both historical and contemporary research is guided by theory (Bernard et al., 1986), so it should also not be assumed that more recent ethnographic work is entirely free from bias.
In some instances, anthropologists and ethnographers may unconsciously project ideological biases onto their observations, distorting the accuracy of the information they record. Of the 20 cultural groups from the Human Relations Area Files surveyed by Precourt (1979), using five indexes designed to measure ideological bias, 12 groups showed evidence of this bias appearing in the sources describing them. The indexes he used include "reductionistic stereotyping" (an overly simplified representation of a culture) and "racial connotation" (where it is suggested there is an innate biological basis for certain behavior found only in a particular group of people). Conversely, Rohner et al. (1973) have also found evidence of a “bias of romanticism,” in which anthropologists are reluctant to see or document the “negative” features of another cultural group.
The potential for gender bias in both ethnographic observations and interpretations by anthropologists, and the subsequent coding of this data by others, has previously been raised (Slocum, 1975). In a study of female status and societal complexity, Divale (1976) found a significant correlation between the gender of an ethnographer and the reported status of women, with male ethnographers more likely to report lower female status. On the other hand, Whyte (1978) compared the gender of both ethnographers and coders with a variety of variables on women’s status. While Whyte found some significant correlations, these were no more common than would likely occur by chance, and he suggested gender bias by coders could be largely avoided if codes were kept as simple as possible and required little inference from the coder. It should be noted that even when bias may exist within source data, this does not necessarily mean it will affect every variable that is measured. For example, a variable such as settlement size is less likely to be affected by bias as it relies less on interpretation by an ethnographer, and indeed, the aforementioned study by Divale (1976) found no significant relationship between settlement size and the gender of the ethnographer.
Biases in ethnographic source materials, and others, are widely acknowledged in anthropological literature (LeCompte, 1987), and have been discussed in the past in by cross-cultural researchers (Divale, 1976; Slocum, 1975; Naroll & Naroll, 1963; Precourt, 1979; Rohner et al., 1973). However, few cross-cultural databases have explicitly documented how they address author biases. Databases need a way to systematically deal with author biases, and potentially enable for author biases to be identified, filtered and adjusted for by researchers. In the following sections, we detail approaches to dealing with author biases.
Approach 1: Coding at the Source Level
Even with databases that list specific time and place foci for a cultural group (e.g., the Ethnographic Atlas and the Standard Cross-Cultural Sample), it is not always clear whether the published codes of specific researchers strictly adhered to that focus. For example, White (1989) gives an analysis of seven different studies to show how much other sources (other than the principal authorities) were used to make coding decisions. Nonetheless, such databases usually end up with one coded value (Binford, 2001; Murdock & White, 1969; Turchin et al., 2015; Watts et al., 2015b). The resulting data are easily visualized and readily amenable to statistical analysis. However, this approach also makes it difficult to understand the weight given to different sources, and to identify systematic differences and biases across sources.
An alternative approach is to code variables at the level of ethnographic sources, rather than cultural groups. Instead of aggregating information across different sources when making coding decisions, this approach codes what each source item has to say about each variable at a specific time and place. This means that cultural groups can have multiple values for the same variable, one for each source used to code the variable. Ethnographic sources can also contain data on multiple time and place foci. Descriptions of multiple time foci can occur when a source includes both retrospective data from elders and a description of the contemporary culture. In addition, descriptions of multiple place foci can occur when a source includes a descriptions or comparisons with neighboring communities. . In such instances, a single source can be used to code multiple time and/or place foci for the same variable. When different sources converge on the same coding for a variable at the same time and place, this can strengthen the confidence researchers have in the state of the culture. When sources diverge, researchers can investigate why these differences occur (see Challenge: Inconsistencies Between Sources. for sources of inconsistency). When multiple sources are from different time periods, coding values for a variable from each source allows the researcher to assess change over time. Coding at the source level has the potential to allow researchers to identify potential author biases, for either a single culture, or those that arise systematically across cultural groups. This approach also opens up new analytical approaches, such as treating cultural groups as higher-order clustering variables in statistical models and including multiple observations of variables per cultural group.
We are not aware of any published databases that have coded cultural groups at the source level by design. The closest we are aware of is the Database of Religious History which allows multiple different people to code the same time-specific cultural unit and to publicly raise disagreements over coding decisions (Slingerland & Sullivan, 2015). One of the advantages of the approach taken by the Database of Religious History is that it sets up a framework for experts to contrast and debate interpretations of historical records. Allowing multiple experts to code the same population is similar to coding at the source level in that it allows for the comparison of multiple coding decisions for each variable, but it differs in that these coders are each likely to draw on multiple historic sources. This system also lends itself to analytically pooling multiple coders across time points and geographies to examine inter-coder variability and consistency.
Approach 2: Coding Source Meta-Data
Meta-data variables represent the features of source materials and provide a way to quantify potential biases in source material (Naroll, 1962). These potential biases may include the length of stay in the field by the ethnographer, the ethnographer’s familiarity with the language of the culture studied, the role of the author (for example, anthropologist, or missionary), the author’s field experience, the author’s country of origin, the primary purpose of the study, and the year in which the work was written and published (Naroll, 1962, 1970). While very few databases have public meta-data, the sources used in the Standard Cross-Cultural Sample have been independently coded on whether the author used multiple informants, the age of the informants, and the author’s involvement in the community studied (Rohner et al., 1973; Rohner et al., 1982).
Recording source meta-data enables researchers to filter out particular kinds of source materials and/or model potential author biases at the stage of data analysis. For example, researchers could exclude missionary accounts from analyses of religion if there was reason to believe these accounts might be biased. Additionally, one could treat such meta-data as a varying intercept in a hierarchical model, thus pooling estimates around author and source.
Coding source meta-data also means that the source materials can become the subject of study themselves. For example, researchers could identify how author characteristics correlate with the subject matter that they write about, identify potential biases in how different sources describe cultural groups, and adjust for these biases within statistical models. When there are multiple sources available to code a single cultural group, it also becomes possible to make coding decisions that are independently supported by multiple sources, and to identify what kinds of variables and authors are most likely to result in conflicting accounts.
While data quality control measures have been provided by Rohner et al. (1982) for the 186 cultural groups in the Standard Cross-Cultural Sample (Murdock & White, 1969), we are not aware of any recently published quantitative databases that include variables on meta-data. The Pulotu database includes a general variable on the overall quality of ethnographic record for a population (Watts et al., 2015b), but this provides little additional information about the biases of authors and how they impacted specific coding decisions. Coding meta-data at the source level adds some complexity to database construction and analysis but is clearly feasible. The eHRAF World Cultures qualitative online database provides an example of how this can be achieved (M. Ember, 1997). This rich qualitative database codes the document type (e.g., monograph or essay), the year of ethnographic fieldwork, the social role of the author in the cultural group (e.g., missionary or ethnographer), the subjects covered in the text, and a rating of the overall source quality on a five-point scale. These codes provide a useful starting point for meta-data variables, and could be extended to include a multi-dimensional rating system of source quality and measures of a broader range of author and source biases.
Challenge: Interaction Biases
The nature of an author’s interactions with a cultural group determines their opportunities to learn about it. One important component of this interaction is the amount of time authors spend interacting with people. These differences in time create differences in opportunities to observe social and cultural traits. For example, if an author has spent less than a year with a cultural group, they will be less likely to witness seasonal variation in subsistence activities and ceremonial rituals, or infrequent events such as warfare and the funerals of highly ranked personages. A significant correlation between length of stay in the field and witchcraft attribution reports was detected by Naroll (1962) in a study on stress. Naroll argues this could present an issue if another variable, such as warfare, was similarly underreported by those staying only a short time in the field, thus giving the spurious impression of a relationship between witchcraft and warfare (Naroll, 1962, p. 25). This data quality variable was later tested by Witkowski (1978), who found that length of field stay influenced the accuracy and reliability of ethnographic data. It has been pointed out by Ember and Ember (2009), however, that a correlation such as the one Naroll discovered between length of stay and witchcraft attribution could be spurious if, for instance, ethnographers were less likely to stay for long periods in more complex cultural groups, which in turn may be less likely to possess witchcraft beliefs. A significant correlation between a data quality variable and a specific trait can therefore warn of a potential bias, but does not necessarily entail a bias.
Another important component of an author’s interaction with a culture is the effect that being observed has on a population. This includes the disruptions to normal community events resulting from a foreigner, potential suspicion about the observer’s intentions, shyness, restricted behaviors and social desirability biases. For example, in situations where a cultural group has colonial laws imposed upon them, the presence of someone associated with a colonial country may make them less likely to engage in illegal behavior. The issue here is that, even if an ethnographer impartially documented their observations of a culture, their presence is likely to bias what they observe.
Much ethnographic literature is written by authors originating outside the cultural group of interest. There is, however, a growing body of ethnographic literature, known as endogenous ethnography, written by authors from the cultural groups of interest. It has been argued that endogenous ethnographies avoid many of the potential biases outlined above (Hale, 1972; Rohner, 1996), though we note that endogenous ethnographies can entail their own challenges. For example, informants may not disclose information they assume a local researcher already knows, regardless of whether this is really the case (Jones, 1970). While endogenous ethnographies can offer rich cultural insights and avoid some biases, they could also introduce other biases.
As discussed above in the section Challenge: Ethnographers Personal Biases Author Biases, coding variable meta-data can help control for, and identify, potential source biases in some instances. Meta-data could include variables on the amount of time that an ethnographer has spent with a cultural group and the nature of their interactions. While we strongly advocate coding variable meta-data, we note that researchers should be able to provide theoretical justifications for the meta-variables they code (Ember et al., 1991), and significant correlation between meta-data codes variable of interest does not necessarily mean that there is bias in the ethnographic sources (Bernard et al., 1986; Ember, 1986).
Challenge: Recording Biases
Source materials vary widely in the scope of cultural domains that they describe. Some ethnographies aim to provide general overviews of cultural systems while others focus on specific topics of interest such as tattooing (Firth, 1936), politics (Schulte Nordholt, 1971), or kinship systems (Scheffler, 1962). Authors also make decisions about which cultural groups to study, and what aspects of these groups to write about based on what they perceive as interesting and novel, which are in themselves the product of the cultural context of the author (Berreman, 1968; Gough, 1968). These decisions have the potential to introduce biases into cross-cultural databases if researchers are not careful in specifying how coders are to treat traits that authors do not discuss, but that might not necessarily be absent.
Novelty biases mean that cultural traits that are deemed unusual or noteworthy are more likely to be documented than commonplace traits. For example, ethnographers are likely to write about a practice such as the use of traditional plant-based medicines, but less likely to write about the procurement of commercial medicines from a pharmacy (Naroll & Naroll, 1963). This bias means that it will be easier to find evidence for the presence of novel cultural traits than the presence of common or familiar cultural traits. Novelty biases can make it challenging to tell whether an ethnographic source has failed to mention a trait because it is absent, because they did not learn about it, or because it was not sufficiently interesting to them (Purzycki & Watts, 2018).
Another form of bias is that authors often choose to write about topics assumed to be of interest to their intended audience. Theoretical perspectives, as well as both academic and public interests change over time. This means that subjects such as kinship systems, political order, and gods, which has long been of academic and general interest tend to be well described. However, subjects like the ratio of maternal and paternal investment in children, which is of contemporary theoretical interest, were often not documented by early ethnographers (Shaver et al., 2020). One challenge that results from this bias is that it is easier to build databases documenting the kinds of cultural features that have long been of interest to academics and general audiences, but more difficult to build databases to represent niche or contemporary topics.
Approach: Variable Meta-Data
The standard practice in cross-cultural database construction has been to withhold details of how particular coding decisions have been reached. As a result, it is generally not possible to tell from a database itself whether a trait was coded as absent because the coder inferred its absence from a lack of mention in source materials, or whether the trait was coded as absent due to the source material explicitly describing it as absent. This also means that it is difficult to investigate the extent that things such as recording bias might influence coding decisions.
Cross-cultural databases could address the lack of information about coding decisions by explicitly recording the evidence used to make a coding decision. For example, instead of binary coding the (0) absence or (1) presence of a trait, researchers could code (0a) absence directly stated in source materials, (0b) absence inferred from source material, (1a) presence directly stated in source materials, (1b) presence inferred from source materials. While the Database of Religious History can include qualitative justification of coding decisions, we are not aware of any existing cross-cultural databases that have integrated this information into variable coding schemes. We agree with both Whiting (1954) and Ember and Ember’s (2009) warnings against inferring absence of a trait not reported in the source materials unless the source materials include a thorough discussion of the context in which the trait would be expected to occur. For example, if an ethnographer spent considerable time describing species of domestic animals kept, and failed to mention chickens, one might infer their absence. However, if types of domestic animals kept were mentioned only in passing or not at all, absence of chickens should not be inferred. Coding meta-data for a trait as “inferred absence” or “not attainable” as suggested by Whiting (1954) can allow researchers using a database to understand whether the absence of a trait can be reasonably inferred from the ethnographic source, or whether it is unknown.
Explicitly coding the difference between inferred absences of traits and described absences of traits is important because, when not done appropriately, this process can have major implications for the results of statistical models (Beheim et al., 2021). Explicitly coding the way in which absences of traits have been inferred allows researchers to test the sensitivity of researchers’ results to coding assumptions. Providing meta-data on the processes of variable coding also helps primary researchers to identify topics that are of theoretical interest but under-documented in ethnographic research.
Ethnographic Record
Ethnographic sources vary widely in their quality and specificity. As a result, researchers regularly have to critically evaluate the sources they are using and develop strategies for dealing with textual data that is challenging to interpret. These challenges can range from minor uncertainty in wording to clear contradictions within source materials.
Challenge: Ambiguity Within Source Materials
As an example of an apparent contradiction, Everett’s (2009) ethnography described the Pirahã of Brazil as having a “relative lack of ritual” (81). Everett also defines ritual as “a set of prescribed actions with symbolic significance for the culture.” Yet, the Pirahã have a variety of behaviors that would count as ritual, construed in Everett’s own terms: traditions associated with burying the dead (p. 74), village dances with venomous snakes, and spirit possession (p. 139). Contradictions within sources can make it challenging to interpret and code sources, but databases have rarely outlined how they deal with such challenges.
A more common challenge when coding variables is that a source can be informative but insufficient to define a single response on a variable. Take the following hypothetical variable codes on the presence of different kind of shaman practitioners; (0) absent, (1) unpaid, (2) paid and semi-professional, or (3) paid and fully professional. The kind of under-specification of ethnographic source material that coders might encounter is provided by Howard’s (1965) statement that Bungi shamans “earn a handsome income.” This description on its own would narrow the most appropriate coding decision down to the responses 2 or 3, but might not be sufficient to code one over the other. How should coders deal with such a case and how can researchers ensure that coders process ambiguous source material consistently? To address these issues, there needs to be clearly defined procedures for dealing with ambiguity in source materials. Below we discuss three approaches.
Approach 1: Discarding Ambiguity
The simplest approach to dealing with ambiguity in source materials is to code such a variable as having missing data. For example, in the case of the Bungi, the variable on shaman would simply be treated as missing data. This approach has the advantage of being straightforward to code, principled, and relatively simple to analyze. However, it comes at the cost of discarding information. For these reasons, a multi-state coding approach may be preferable.
Approach 2: Multi-State Coding of Ambiguity
Another approach to accounting for ambiguity in source materials is to allow for the multiple states of a variable to be coded. This could be done using either a mid-range value or by using multiple values. For example, in the case of the Bungi shaman, they could be coded as "2.5," or as both "2" and "3," respectively. This is useful when there is sufficient information to narrow down the relevant range of coding decisions, but there is insufficient information to code an exact value. Along these lines, The Database of Religious History allows coders to enter multiple responses to the same variable (Slingerland & Sullivan, 2015).
A multiple coding approach has the advantage of being able to accurately represent the level of detail available in the ethnographic record. This means that partial information can be retained, which opens up additional post-coding data processing opportunities. For example, if a researcher decided they were simply interested in the presence or absence of shamans in general, a coding decision of “2 or 3” in the case of the Bungi could then be coded as “present”. Multi-state coding increases the amount of information retained in datasets with minimal additional costs of time and effort for coders, but introduces additional complexity to data analysis.
Approach 3: Meta-Variable Coding
Another way of representing uncertainty in datasets is to include a meta-variable that represents the strength of evidence for each data point. This was suggested by Ember (1986) as well as Ember and Ember (2009) as a more profitable alternative to Naroll’s data quality measures, with the advantage of providing a direct assessment of the quality of each variable without having to code for the many additional data quality variables that Naroll proposed. For example, along with coding a response value for “2” on a scale, coders might rate the strength of evidence supporting a coding decision on a 5-point scale ranging from (1) barely supported to, say, (5) multiple lines of clear and decisive support. This approach has the advantage of quantifying the level of information available in the ethnographic record, which can also be incorporated into statistical models as weights or used for filtering data. As far as we are aware, this approach has not yet been used in cross-cultural databases.
We note that the approaches outlined in this section can be used together and address different kinds of problems arising from the uncertainty in source materials. The approach of coding a meta-data variable provides a general measure of the quality of the source material for the variable of interest, while the multi-state coding approach provides a way of representing partially informative information.
We caution, however, that meta-variable coding should not be used as a way to incorporate substandard data or make speculative inferences. When information is ambiguous and does not narrow the range of possible values, this should be coded as either missing data or given a code to represent ambiguous source materials that researchers can choose how to process. The approaches outlined here are not intended to be used to lower the standard of data required to make coding decisions, only to quantify the extent of support for a given coding decision and to accurately represent the information available in the ethnographic record.
Challenge: Inconsistencies Between Sources
In addition to uncertainty and contradictions within sources, researchers are also likely to encounter inconsistencies between sources. Returning to the case of the Pirahã, there has been some debate regarding whether or not they have creation stories or myths. On the one hand, Everett (2005) states that the Pirahã “have no creation stories or myths,” yet others (Gonçalves, 1993; Nevins et al., 2009) have pieced together fragments of myths detailing creation and a mythical cosmology. One of the challenges for addressing these inconsistencies between sources is that there are many potential sources of these inconsistencies. Such instances can arise for many reasons, including: (a). Cultural change over time. Source materials are often based on ethnographic research at different points in time and changes are not always easily documented or observed. (b). Different sub-populations described. Cultural groups can include multiple communities, these communities can exhibit variation, and some ethnographers spend more time with one community than another. (c). Differences in interpretation of an event. Ethnographers can witness an individual perform the same sequence of actions, but interpret them in different ways based on their contextual knowledge and the ideas they have about the world. (d). Difference in definitions of terms. The same terms can be used in different ways due to shifts in the meaning of terms over time, or various meanings that a single term can have. (e). Low quality sources. Some inconsistencies between sources can arise due to sensationalist, inaccurate or otherwise erroneous writing.
Some of the issues concerning inconsistencies between sources may be avoided if the researcher is able to choose a single time and place focus, while others can be addressed, at least in part, through the coding strategies already highlighted. For example, researchers can use source-level data and meta-variable coding to select the highest quality sources coded for a given variable. These approaches, combined with using only high-quality sources, can drastically reduce inconsistencies between sources.
General Principles for Building Comparative Databases
In this article, we have argued that there is no single best approach to building cross-cultural comparative datasets. For example, the most appropriate way to deal with variation within a population depends on the units of analysis, the hypotheses the database is built to test, and the resources available to coders. However, we do believe that three general principles should be used across comparative datasets.
First, cross-cultural databases should be transparent. This means that databases should provide clear and detailed descriptions of how they defined variables, how they reached coding decisions, how they have addressed limitations in ethnographic source materials, as well as clear records and motivations of decisions (see below). For example, databases need to make it clear when the ethnographic and/or historical records provide insufficient information about a trait, and how these limitations are addressed (Beheim et al., 2021).
Second, cross-cultural databases should be as internally consistent as possible. This includes consistency in the units of analysis and the coding procedures used across culture profiles. For example, database managers need to ensure that different coders are treating variation within populations in the same way across the cultural groups.
Third, cross-cultural databases should retain as much information as is practical. This includes information about uncertainty in coding decisions, variation in traits within populations, and the qualitative descriptions used to make coding decisions. Retaining information about sources and coding decisions enables databases to be more easily validated, extended, and thus to be used to answer a broad range of research questions.
In addition to these three principles, we are in broad agreement with the issues highlighted in previous literature (Ember & Ember, 2009; Slingerland et al., 2020). These include issues such as building links between different datasets, and the practical issues regarding maintaining database infrastructure. We also believe that a major next step in cross-cultural database construction is to critically analyze the source materials from which cross-cultural databases are built. Coding variables at the level of text sources, and coding the features of the authors provide a means of achieving this in future databases. Another issue we encourage those compiling cross-cultural databases to engage with is that of Indigenous data sovereignty (Walter et al., 2021). While these issues are beyond the scope of this paper, we encourage those constructing databases to give consideration to the growing literature on this topic, much of which has been published in recent years (e.g., Kukutai & Taylor 2016). Finally, we believe the issues highlighted in this paper will be valuable for contemporary ethnographers collecting new data to consider when planning and undertaking their research. By giving careful consideration to how they select units of analysis to study, and how they collect data on variation within a population so that it can be quantified, ethnographers can ensure the data they collect and publish will be amenable to cross-cultural analyses. Today, many ethnographies focus on specific domains of culture from particular theoretical perspectives. In contrast, early anthropological records were often structured around broad, standardized checklists of topics, such as the list of topics provided in Notes and Queries on Anthropology (Garson & Read, 1899) or the Outline of Cultural Materials (Murdock et al., 2008). While some of the content and theories behind these questions is now outdated (and in some cases ethically problematic) the general approach to providing broad overviews of cultural systems is immensely useful for researchers interested in building cross-cultural comparative databases.
Conclusion
Cross-cultural databases provide a powerful way to summarize vast amounts of historical textual data. This power is achieved by abstracting away details about sources and populations, which can result in several pitfalls. Here, we have outlined issues with the identification of populations, biases in ethnographic fieldwork and literature, as well as ambiguity and uncertainty in ethnographic records. Existing databases have addressed these issues with varying degrees of success but all existing solutions have their limitations. Here, we have proposed new ways forward for the construction of cross-cultural databases that we hope will generate constructive dialog about best practice in cross-cultural database construction. This continuing dialog is necessary to ensure that comparative databases continue to expand our understanding of human culture.
Footnotes
Acknowledgments
We would like to thank three anonymous reviewers and Carol Ember for their detailed and constructive feedback.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: JW acknowledges funding from the Marsden Fund, Royal Society Te Apārangi (19-UOO-1932). JS acknowledges funding from the Marsden Fund, Royal Society Te Apārangi (19-UOO-090). BGP acknowledges support from the Max Planck Institute for Evolutionary Anthropology, an Understanding Unbelief Project grant that was managed by the University of Kent (JTF 60624), and the Aarhus University Research Foundations. JW, JCJ, and BGP, acknowledge support from a Templeton funded grant (JTF 61111). None of these funders were involved in the design of writing of this article.