Cultural Change Over Time: Why Replicability Should Not Be the Gold Standard in Psychological Science

Moving Toward the Challenge

Klein et al. (2012) suggest that social context may be a factor in failures to replicate. Nonetheless, in their view, contextual variables are interfering factors to be controlled or at least assessed so that they can be ruled out. Investigating this idea empirically, Van Bavel, Mende-Siedlecki, Brady, and Reinero (2016a) found that the contextual sensitivity of research topics in psychology was associated with replication success. Their analysis focused on broad, macro-level contextual influences—time, culture, location, and population. This type of analysis often goes under the rubric of “hidden moderators.”

In the Van Bavel et al. study, coders rated the strength of contextual influences likely to have affected 100 studies that had been the subject of an influential study on replicability in psychological science. That prior study had demonstrated that a minority of psychological studies in cognitive and social psychology could be replicated (Open Science Collaboration, 2015). When rating a study for the Van Bavel et al. study, the coder assessed how likely the effect reported in the abstract of the original study was to vary by context—defined broadly as differing in time (e.g., pre- vs. post-Recession), culture (e.g., individualistic vs. collectivistic culture), location (e.g., rural vs. urban setting), or population (e.g., a racially diverse population vs. a predominantly White population). This coding scheme concerned broad classes of macro-level contextual influences that could reasonably be expected to influence the reproducibility of psychological research. As predicted, studies whose effects were rated as more likely to vary by context were significantly less likely to replicate. These findings implied that contextual change rather than lack of validity might lead to nonreplication. They advocate that “future work should . . . develop a more nuanced model of the influence of context on reproducibility” (Van Bavel et al., 2016a, p. 6458). That is the goal of the present article. While the aforementioned aspects of context are all relevant to the present argument, I try to identify a theoretically derived mechanism by which they operate and to use this to predict exactly why certain studies should not be expected to replicate.

Going One Step Further: A Theory of Social Change and Human Development

Van Bavel et al. have made an important start in challenging the notion of replicability as the gold standard in psychological science. But those of us who study cohort effects or use cross-temporal research designs—that is, methods for studying changes in behavior and values over time—are in a position to take this critique to the next level. Based on the idea that social change drives cultural change, which, in turn, drives behavioral change, my collaborators and I have studied the behavioral effects of environmental shifts over time.

But we have gone a step beyond this: My theory of social change and human development has enabled us to specify what kinds of social changes will produce what kind of cultural and behavioral changes. Our most recent publication on the effects of decades of social change ended with a challenge to the notion of replication as a standard for psychological science (Garcia et al., 2015).

Unlike the replicability studies, we have not studied environmental change as nuisance variables or new behavioral patterns as failed replication. I have developed a theory that integrates sociological, cultural, and psychological variables to predict when and how behavior will change over time; to put it in an alternative language—when and how behavior will not replicate (Greenfield, 2009, 2016). Incorporating sociodemographic change and culture change into a theoretical framework—as independent variables if you will—constitutes a paradigm shift from nonreplication of results to theoretically based predictions of behavior change. In other words, I am advocating that psychological science move from the issue of whether or not a psychological phenomenon replicates to studying the effects of sociodemographic change on culture and behavior. Note an important methodological implication: One must call into question the automatic judgment that a failed replication means that the original study lacked validity.

According to my theory of social change and human development, as environments become more urban, commercial, technological, and wealthier, providing more educational opportunity (classified as facets of social change; top level of Fig. 1; e.g., Kağ ıtçıbaş ı, 2007), social behavior becomes more independent and cognitive performances become more abstract (Greenfield, 2009, 2016; bottom level of Fig. 1). These behavioral changes embody shifts in cultural values toward individualism (e.g., Zeng, 2015), gender equality (e.g., Manago, 2012), and abstract, decontextualized cognition (e.g., Luria, 1976; classified as cultural change and shown on the second, value, level of Fig. 1).

OPEN IN VIEWER

Fig. 1.

Graphic representation of Greenfield’s theory of social change and human development. Causal influences go from higher levels to lower. Sociodemographic changes are seen as the governing causal influence that can influence the next level, which is conceived as the cultural level, or can have a direct influence on the level of learning environment/socialization. Source: Greenfield (2016).

The basic direction of causality is from the top down. However, one can also envision the opposite direction, especially between generations. That is, say members of the current generation are socialized to become more individualistic (bottom level). Then they become parents and socialize their children in an individualistic direction (second level up). This phenomenon occurred with the first generation of professional Maya women when they became parents (Manago & Greenfield, 2011).

Note that the sociodemographic level on the left side is anchored and summarized by Tönnies’s (1887/1955) concept of Gemeinschaft, usually translated as community, while the right side is anchored by Tönnies’s concept of Gesellschaft, usually translated as society. Gemeinschafts are the traditional subject of study in anthropology; Gesellschafts are the traditional subject of study in sociology. The term social change encompasses all of the sociodemographic shifts shown on the top level of the diagram. Hence, it is clear from the outset that the theory is interdisciplinary. While owing a debt to German sociologist Tönnies (1887/1955) and to early 20th-century U.S. anthropologist Robert Redfield (1941), the present theory goes beyond them in several ways, the most important being the consideration of social change and the linkage of ecological features to psychological and behavioral characteristics. The term social change refers to changes on the sociodemographic level of the figure; the term cultural change refers to change on the level of cultural values.

In terms of time scale, the rate of cultural, socialization, and behavioral change depends on the rate of social change in a given society. For example, technology has been advancing so quickly that cultural change caused by the spread and advancement of communications technologies is detected in periods of time much smaller than a full generation (e.g., Uhls & Greenfield, 2011). The same is true of the impact of social change on child behavior and adjustment in China, arguably the most rapidly changing society in the world (Zeng & Greenfield, 2015). Chen, Cen, Li, and He (2005) reflected accelerating social change by showing differences in the evaluation of child behavior first in two cohorts 8 years apart (1990 to 1998) and later in two cohorts only 4 years apart (1998–2002).

Studies in Zinacantan, a Maya community in Chiapas, Mexico, tend to reveal various manifestations of social change as sequential (e.g., shift from subsistence to commercial economy preceded the expansion of education; Maynard et al., 2015). Changes on the value level also appear to have the feature of sequentiality rather than simultaneity. For example, Manago’s (2014) study of changes in gender roles and independence/over the generations in Zinacantan showed movement away from family obligation values to independence was not occurring as rapidly as the value shift toward egalitarian gender roles. This uneven rate of change suggests a sequential process. These examples and the theory itself suggest that sequentiality characterizes all levels of the model and sequential ordering as effects move from one level to the next.

Gemeinschaft and Gesellschaft summarize a group of characteristics that co-occur at the extremes in the ideal types. Learning environments and behavior that are adapted to one Gemeinschaft characteristic tend to be adapted to the others as well. At the sociodemographic level, the ideal types of Gemeinschaft and Gesellschaft are ecological complexes of co-occurring and synergistic features. These synergistic relations—for example, between education and wealth—do not fit well with the idea in psychology that we can and should separate out variables, one at a time. One important feature of the theory is that all of the sociodemographic shifts are equipotential—they move values, socialization, and behavior in the same direction. Whichever sociodemographic variable (the top level of the figure) is shifting in a particular period in a particular place, that will be the one to drive shifts on the lower levels—cultural values, learning environments/socialization, and behavior.

Studies of Social, Cultural, and Behavioral Change in the United States

Because most of the controversy concerning replication has focused on studies done in the United States, I too will focus on U.S. social and cultural change and how behavior has changed in concert with these changes.

Cultural change and sociodemographic change (Greenfield, 2016)

On the cultural level, the content of millions of books, analyzed by means of the Google Ngram Viewer, revealed that individualistic words (e.g., choose, personal, individual, self, unique, special), first-person singular pronouns (e.g., I, me, mine), unique children’s names, and the word child itself (indexing a rise in child-centeredness) became more frequent as urbanization progressed between 1800 and 2000. In this same period of time, collectivistic words (e.g., duty, give, harmony, belong, compassion), first-person plural pronouns (we, us, ours), words signifying hierarchical social relations (obedience, authority), and words related to the practice of religion in everyday life (pray, worship) also became less frequent (DeWall, Pond, Campbell, & Twenge, 2011; Greenfield, 2013; Grossmann & Varnum, 2015; Kesebir & Kesebir, 2012; Twenge, 2015; Twenge, Campbell, & Gentile, 2012b). Figure 2 shows examples of these trends from Greenfield (2013). Another individualistic cultural trend, the promotion of equal rights in the realm of gender, was manifest in a rising proportion of female pronouns relative to male pronouns in books between the 1960s and the 2000s (Twenge et al., 2012b).

OPEN IN VIEWER

Fig. 2.

Top panel: Increases in frequency of words indexing hypothesized adaptations to more urban, educated, wealthy, and technological environments in the United States from 1800 through 2000. Bottom panel: Decreases in frequency of words indexing hypothesized adaptations to more rural, less educated, poorer, and less technological environments from 1800 through 2000. Source: Greenfield (2013).

As an index of long-term sociodemographic change, the United States went from being predominately rural in 1800 to being overwhelmingly urban in 2010 (Greenfield, 2013; Fig. 3). Between the late 1800s and the 2000s, family size declined and multigenerational households became less frequent. In contrast, single-child families and living alone became more frequent family and household structures (Grossmann & Varnum, 2015; top level, Fig. 1).

OPEN IN VIEWER

Fig. 3.

Percentage of U.S. population living in rural and urban areas from the years 1800 to 2000. Data were drawn from the following sources—1800–1980: U.S. Census Bureau (2004); 1990: U.S. Census Bureau (1992); and 2000: U.S. Census Bureau (2004). The definition of urban population changed over the years, and two different definitions were both used in 1950 and 1960, so there are double data points for those years. Source: Greenfield (2013).

Of all the indices of a Gesellschaft environment, urban population was selected for the 2013 study of cultural change (Fig. 2) because it was the only sociodemographic variable that was continuously assessed in the U.S. census from 1800 to 2000, the period of that study. Greenfield (2013) considers urbanization to be a stand-in for the whole set of Gesellschaft characteristics: education, wealth, and technological environments, to name the three most important ones. While the connection between the changes on the cultural level and the rise of urbanization are not, in this study, definitively proven by statistical analysis, two studies, to be described next, provide relevant statistical analysis.

Comparing youth in Taipai (urban environment) with youth in Taitung (rural environment) on the island of Taiwan, Lee, Beckert, and Goodrich (2010) found that rural youth, both males and females, had a more collectivistic value system, whereas urban youths had a more individualistic one. In the United States, shifts toward greater urbanization significantly predicted shifts toward greater individualism 30 years later (Grossmann & Varnum, 2015). In contrast, there was little evidence for concurrent shifts in urbanization and individualism; nor did shifts toward greater urbanization follow shifts toward greater individualism.

Reversing These Trends

However, the theory also predicts that reversing sociodemographic trends, for example, wealth reduction, will reverse cultural and psychological trends. In line with this prediction, yearly national surveys of high school students from 1976 to 2010 showed that concern for others and for the environment were higher during times of relative economic deprivation, while materialism and positive self-views were higher in better economic times. With respect to positive self-views, the Great Recession was an historical exception, probably because of conflicting sociodemographic trends: The increasing role of technology seemed to overwhelm reduced economic level; and self-views continued to become more positive, even as concern for others and for the environment rose (Park, Twenge, & Greenfield, 2014).

All of this evidence indicates that cultural variability exists over time as well as space, hence its relevance to replications, which also occur over time. Next, I will show that two studies that did not replicate in the Open Science Collaboration (2015) instantiate these general trends of social, cultural, and behavioral change.

Applying This Framework and Findings to Understand Why Certain Studies Did Not (and Should Not Have Been Expected to) Replicate

The two studies were selected because the earlier described trends of cultural change would have led to the prediction that they would not replicate. And they did not.

Implications for Psychological Science

A major, often unstated assumption is that the behavior and cultural values of adults in a particular country or ethnic group are stable over time. Theorizing about the effects of social change challenges this assumption and invites a whole new field of inquiry. For example, what have been labeled WEIRD cultures (Western, educated, industrial, rich, and democratic) were, for most of history, in the minority of the world’s population, but furnished participants for the majority of psychology studies (Henrich, Heine, & Norenzayan, 2010). Therefore, the authors point out that many common psychological findings, thought to be universal, are found only in WEIRD environments. However, with globalized social change in the Gesellschaft direction (e.g., more formal education, growing wealth, and industrialization), the whole world is becoming increasingly WEIRD, with concomitant changes in behavior. Psychological science needs to pay attention to these changes.

Many cross-cultural studies ignore the influence of these same sociodemographic factors at their own peril—college populations are compared across different countries in different parts of the world. However, sociodemo-graphically, these populations are rather similar to each other: highly educated and relatively wealthy, and they tend to be WEIRDer in their psychology—for example, more individualistic—than their non-college-educated compatriots. Similarly, many cross-ethnic comparisons in the United States ignore known sociodemographic differences between the groups. I hypothesize that this issue has created inconsistent results in studies of individualism and collectivism, where the hypothesis relates to cross-national or ethnic differences in these value systems (Oyserman, Coon, & Kemmelmeier, 2002).

My hypothesis is that this inconsistency occurs because sociodemographic variability is rarely taken into account. In other words, one implication for psychological science more generally is that the same sociodemographic factors invoked in explaining the psychological impact of social change also must be taken into account in cross-cultural and cross-ethnic comparison as well. Sociodemographic factors can explain, for example, why members of different ethnic groups can have the same values (e.g., samples from two different ethnic groups are college educated and urban), while members of the same ethnic group can have different values (e.g., urban vs. rural Chinese; Greenfield, 2009).

Implications for the Role of Replication

Inbar (2016) notes that the social psychology studies assessed by Van Bavel and colleagues (2016a) were less likely to replicate than cognitive psychology studies. This is because social psychology studies are more likely than cognitive psychology studies to examine contextually sensitive topics (Van Bavel et al., 2016b). It is therefore no coincidence that the two studies selected for in-depth analysis here were both social psychology studies.

However, there is another side to this coin: One conclusion might be that methodological artifacts are more likely to affect cognitive psychology studies than social psychology studies. One possible methodological artifact that I noticed in the Open Science Collaboration (2015) was that studies originally done in the laboratory were being replicated online with participants recruited through MTurk, an online task force that receives minimal monetary compensation for their work. While there is evidence that participants carefully recruited through MTurk are more attentive and responsive to experimental manipulations than college students taking a survey online (Hauser & Schwarz, 2016), such successful replication does not necessarily apply to the effect of moving from an in-person lab experiment to an online experiment.

In terms of boundary conditions for the domain in which failure to replicate a study is actually a failure, I would say that failing to replicate a strictly cognitive experiment with no social stimuli replicated in the same presentational modality as the original study is the most likely to justify impugning the original results. A future direction might be to compare the failure rate in the Open Science Collaboration for cognitive experiments that were replicated in the original modality versus those that went from laboratory administration to online administration.

The preceding analysis provides specificity and theoretical underpinning for understanding the important factors involved in the passage of time. I do not like the term hidden moderators to describe these factors because they should not be “hidden.” Instead they should be overt parts of any research design that can be affected by social and cultural change.

If it were not for the concept of culture and, more specifically, the concept of culture change, I would have believed that failures to replicate studies were a threat to our science. However, armed with (a) a theory of social change and human development, (b) empirical studies supporting this theory, and (c) the foregoing analysis of two sample studies illustrating the predictability of nonreplication, the field of psychological science should recognize that replication cannot be the sine qua non of validity. Instead, what is needed is expanding scientific exploration of how and why an array of behaviors shifts with the passage of time.