Abstract
Replication is the scientific gold standard that enables the confirmation of research findings. Concerns related to publication bias, flexibility in data analysis, and high-profile cases of academic misconduct have led to recent calls for more replication and systematic accumulation of scientific knowledge in psychological science. This renewed emphasis on replication may pose specific challenges to cross-cultural research due to inherent practical difficulties in emulating an original study in other cultural groups. The purpose of the present article is to discuss how the core concepts of this replication debate apply to cross-cultural psychology. Distinct to replications in cross-cultural research are examinations of bias and equivalence in manipulations and procedures, and that targeted research populations may differ in meaningful ways. We identify issues in current psychological research (analytic flexibility, low power) and possible solutions (preregistration, power analysis), and discuss ways to implement best practices in cross-cultural replication attempts.
Keywords
Introduction
Understanding cultural influences on psychological phenomena is the backbone of cross-cultural research, and the production of cultural knowledge when two or more cultural groups are compared requires the findings to be replicated and reproduced to ascertain the trustworthiness of the findings. Indeed, replication is an uncontroversial expectation of standard scientific practice that enables the accumulation of reliable results and is essential for scientific progress; it is regarded as one of the key ingredients of science (Schmidt, 2009) and the scientific gold standard (Jasny, Chin, Chong, & Vignieri, 2011). As Lykken (1968) noted several decades ago, “[m]ost theories should be tested by multiple corroboration and most empirical generalizations by constructive replication” (p. 151).
Novel, exciting findings have historically been easier to publish and the incentive to conduct replications has been low (Bakker, van Dijk, & Wicherts, 2012). It is true that multistudy packages are the norm for prominent journals, with later studies expanding upon earlier studies. However, these later studies are rarely identified as replications (Makel, Plucker, & Hegarty, 2012), the final set of studies may be subject to selective reporting excluding null results (Ferguson & Heene, 2012), and ultimately these internal replications do not seem to increase the odds of successful external replication (Kunert, 2016). The frequency of replication in psychological science and the extent to which findings are reproducible is thus uncertain. Moreover, there is serious concern that the evidence in the published literature may not be as robust as previously assumed. A recent large-scale test of reproducibility in psychology successfully replicated fewer than 40% of the 100 included studies (Open Science Collaboration, 2015).
Recent concerns about reproducibility have highlighted the effect of publication bias, high-profile cases of academic misconduct, flexibility in data analysis generating false-positive findings (Kerr, 1998; Simmons, Nelson, & Simonsohn, 2011), widespread statistically underpowered research (Cohen, 1992; Richard, Bond, & Stokes-Zoota, 2003), and difficulty replicating some findings (see, for example, Doyen, Klein, Pichon, & Cleeremans, 2012; Stroebe, Postmes, & Spears, 2012). These concerns drive recent calls for more replication and systematic accumulation of scientific knowledge (e.g., Braver, Thoemmes, & Rosenthal, 2014; Munafò et al., 2017; Roediger, 2012). This new emphasis on replication may pose specific challenges to cross-cultural psychology due to inherent methodological difficulties in emulating an original study in other cultural groups. As noted by van de Vijver and Matsumoto (2010), “The risk of producing cultural knowledge that is incorrect or not replicable is too great if methodological pitfalls [involving a host of issues and problems that are unique to cross-cultural studies] are not understood and addressed” (p. 2).
Despite intrinsic difficulties, though, replication attempts should be fostered in our discipline to ensure a high degree of confidence in our findings. The present article provides a broad discussion on replication with a focus on cross-cultural research, with discussion around ways to conduct cross-cultural research that would be reproducible, as well as how to conduct replications of findings in different cultures. Our goal is to address the intricacies of replication in combination with cultural variation, as well as to discuss practices to increase the probability of producing original research that will be reproducible.
Replication in Psychology
Replication is a method used to confirm the results and conclusions of one study independently in another study. If the methods of the original study can be repeated and yield similar results, we gain increased confidence in those findings and conclusions. More specifically, the function of replications includes (a) to control for sampling error in the original finding (chance result), (b) to control for artifacts (lack of internal validity), (c) to encourage transparent reporting of results and potentially deter fraud, (d) to generalize results to a larger or to a different population, and (e) to test the underlying hypothesis of the earlier experiment (Schmidt, 2009; see also Lykken, 1968; Roediger, 2012).
Many cross-cultural replications have been published in this journal (e.g., Gabriel et al., 2001; Irwin, Engle, Klein, & Yarbrough, 1976; Milfont, Sibley, & Duckitt, 2010; Vauclair, Hanke, Fischer, & Fontaine, 2011) and elsewhere (e.g., Ekman & Heider, 1988; Milfont, Vilar, Araujo, & Stanley, 2017; Sidanius, Levin, Liu, & Pratto, 2000). However, there has been an increased focus on replication in recent years, and several large projects have started to investigate the reproducibility of psychological findings. The previously mentioned “Reproducibility Project: Psychology” (Open Science Collaboration, 2015) involved 270 authors from several countries examining the reproducibility of a sample of 100 cognitive and social-psychological findings published in the 2008 issues of three leading psychology journals (Journal of Personality and Social Psychology, Psychological Science, and Journal of Experimental Psychology: Learning, Memory, and Cognition). Multiple methods of evaluating replication “success” indicated that approximately 40% of the 100 included replications provided support for the original findings.
A separate project line, the “Many Labs Replication Projects,” sought to assess the variability in replication associated with administering the same protocol across independent samples and settings. So far the Many Labs projects have attempted to replicate 51 findings in more than 150 samples around the world across three distinct initiatives. In Many Labs 1 (R. A. Klein et al., 2014), 10 of 13 findings replicated robustly, and variation in effect size across sites and contexts generally did not exceed variation expected by chance. If a finding replicated in one sample, it tended to replicate similarly across all others. Many Labs 2 (R. A. Klein et al., under review) replicated 28 findings across more than 120 samples with a greater emphasis on international labs. Results from that project are not yet publicly available. In Many Labs 3 (Ebersole et al., 2016), 10 studies were replicated across 20 participant pools. Researchers kept the studies running throughout the academic semester to test whether potential differences in participants (e.g., highly conscientious participants might complete participation obligations earlier in the semester) would affect study outcomes. Overall, three of 10 findings replicated and there was little to no evidence that replication outcomes varied based on time of academic semester or the site of data collection.
Key Conceptual Issues in (Cross-Cultural) Replications
The recent intense discussion about replication in psychology, the large-scale replication initiatives, and journal incentives for replication projects (e.g., Simons, Holcombe, & Spellman, 2014) indicate a widespread recognition of the importance of replication for cumulative scientific progress. At the same time, a number of conceptual issues bear discussion about replication in general and replication in cross-cultural research in particular.
One main issue refers to the level of specificity present in psychological theory (e.g., how precisely theories describe the conditions necessary to produce an effect). A lack of specificity may explain replication failures of psychological findings, as key moderators are either unreported or unknown (S. B. Klein, 2014). In this context, expectation of replication may serve as an incentive to increase specificity, and potentially discover these moderators. This specificity may include the populations being tested as well as dimensions of cultural variability, making cross-cultural replications important for theory and generalizability.
The degree of precision expected of different theories, and indeed how much different findings warrant replication at all, is another key issue. Highly impactful findings with broad potential implications warrant more resources (e.g., participant time, money, researcher effort, etc.) than less impactful studies. Effect size is one factor that may affect the determination of impact, and much published psychological research has relatively small effect size that may limit their predictive value and suggest replication resources could be better spent elsewhere (Barrett, 2015; Ferguson, 2009). However, researchers should not consider effect size in a vacuum. Large effect sizes are not automatically of practical or theoretical importance, and small effect sizes can have broad impact depending on the finding (see Greenwald, Banaji, & Nosek, 2015, for an example from Implicit Bias research). To complicate the matter further, published effect sizes are likely highly inflated due to publication biases favoring significant results (Simonsohn, Nelson, & Simmons, 2014). The Reproducibility Project: Psychology (Open Science Collaboration, 2015) found empirical evidence for this, as across the 100 studies replication effect sizes were on average half as large as the originally published effect size. Nonetheless, in combination with other factors, effect size should be one factor to consider when determining which cross-cultural replication efforts should take priority. Topics with the largest differences across cultural groups may be the most impactful. For instance, a recent study has indicated that contents not often studied cross-culturally, such as religiousness, regularity-norm behaviors, family roles, and ethno-nationalism, show the largest cross-cultural differences in samples from 33 countries (see Saucier et al., 2015). These and related topics may make for highly impactful replications.
Another issue refers to the fact that “true” experiments are essentially impossible in cross-cultural research (van de Vijver & Leung, 1997). The large replication projects noted above focused on original effects from experimental studies that are based on random assignment of participants. Interpreting findings about similarities and differences between groups in experimental studies is more straightforward because random allocation makes it less likely that group characteristics are relevant in influencing the dependent variable. In cross-cultural research, however, the issue is more difficult because research participants already belong to a cultural group and cannot be assigned meaningfully to another cultural group, making culture a variable that is beyond experimental control.
Perhaps the most critical issue in conducting replication in cross-cultural research is bias and equivalence of manipulations and procedures, or whether measurement obtained in different cultural groups can be interpreted in the same way. As nicely put by Poortinga (1989), A comparison between two persons, or groups or persons, A and B, can be misleading for two reasons. First, the attribute of A in terms of which the comparison is made may not be the same as the attribute of B. This would happen if one were to compare the weight of A with the length of B. Second, the scale units for a common attribute may not be the same for A and B. This would be the case if the length of A measured in centimeters were to be compared with the length of B measured in inches. Thus, an essential ingredient in any comparison is a variable that forms a scale with identical or invariance scale properties for the persons or groups to be compared. (p. 737)
The psychological phenomena cross-cultural psychologists seek to study might not exhibit the invariance that supports direct replication in other research areas. Ensuring that the same underlying psychological processes are activated in the compared cultures is more important in testing a theory than following the same exact procedure, particularly because cross-cultural research often requires adaptation of the method used (from translation to field implementation). Indeed, observable cross-cultural differences in psychological functioning could arise from distinctions on how instruments and instructions are understood (Campbell, 1964), and the meaning of stimuli may vary by language, time, and cohort. A recent examination of some failures to replicate in the priming literature argues how both language and cohort effects may contribute to failures to replicate across groups (Ramscar, Shaoul, & Baayen, 2015). In particular, Ramscar and colleagues (2015) argued that many priming results cannot and should not be expected to replicate because the learned behaviors researchers try to prime (e.g., words related to old age inducing slow walking pace) do not exhibit the invariance observed in other scientific disciplines due to cultural and linguistic change.
Insights into addressing bias and equivalence in cross-cultural replications may be derived from the same principles and standards applied to original cross-cultural investigations, as well as from broad methodological principles (e.g., utilizing manipulation checks and pretesting manipulations). A detailed discussion of bias and equivalence is beyond the scope of this article, but we strongly encourage readers to read in-depth discussions of these issues and specific ways to deal with them in specialized publications (e.g., Chen, 2008; Milfont & Fischer, 2010; Poortinga, 1989; Poortinga, van de Vijver, & van Hemert, 2002; van de Vijver & Leung, 2010).
In brief, bias in cross-cultural research indicates a presence of nuisance factors in the measurement instrument that influences its meaning within and across cultures when scores of specific groups are compared. Common types of bias that can negatively influence cross-cultural research are construct, method (relative to sample, administration, or instrument), and item biases (van de Vijver & Leung, 2010). Besides these types of bias, four levels of equivalence are typically distinguished in the cross-cultural literature (see, for example, Fontaine, 2005; van de Vijver & Leung, 2010): construct equivalence (whether the theoretical variable or construct has the same psychological meaning in all groups studied), structural equivalence (whether the observed indicators relate to the same construct in each group studied), metric equivalence (whether the relation between the observed indicators and the construct has the same value in all groups), and scalar equivalence (whether the measurement unit is the same in all groups and can be interpreted in the same way). Importantly, scalar equivalence must be met to meaningfully compare groups in terms of a given psychological construct. For example, comparing mean levels of depressive symptomatology across groups (e.g., Szabo et al., 2014) is only meaningful after statistically establishing that depressive symptomatology scores have the same meaning regardless of the cultural background of the respondents. For further discussion on bias and equivalence in cross-cultural research, see the article in this special issue by Boer, Hanke, and He (2018).
In the absence of a robust method to replicate an effect, it is difficult to attribute a high degree of confidence to the reported research finding; however, in the absence of equivalence of the measure/stimuli, it is impossible to interpret precisely the results of a replication attempt. Direct replication failures could be attributed to the original method functioning differently in other societies, and if an adapted procedure is used, replication failure could be attributed to problems with the transfer of the method. Likewise, differences across populations could be due to this methodological variance as opposed to underlying psychological differences between cultures. Nonetheless, a string of failures to replicate in good-faith replication efforts warrants serious reconsideration of the utility of the original finding. Replications, like original studies, cannot determine absolute truth; they can only increase or decrease our confidence in a psychological phenomenon. In the absence of empirically demonstrating moderators to an effect, and conditions under which the effect is reliably obtained, the finding has decreased relevance moving forward without the need to conclusively determine whether the original was a false-positive or not.
Clearly, these conceptual issues are broad and do not have simple answers. However, we hold the view that any replication should be evaluated in the broader context of the research evidence (e.g., in a meta-analytic sense), with a keen eye toward statistical power and generalizability. There is a strong argument that meta-analysis could be a path to address many of the current validation concerns in psychology if issues of questionable research practices, publication bias (e.g., Braver et al., 2014; Schmidt & Oh, 2016), and overestimation of effect sizes due to such bias (van Aert, Wicherts, & van Assen, 2016) are also addressed. Similarly, replication is not a panacea in and of itself, and will not be successful in improving reproducibility unless issues such as publication bias and selective reporting are also addressed (Nuijten, van Assen, Veldkamp, & Wicherts, 2015).
We argue that replication and meta-analysis go hand-in-hand—particularly in the current publishing environment where statistically significant findings are reported selectively—and preregistered replications are the best way to create an unbiased pool of studies for meta-analysis. Indeed, there is evidence of selective reporting in multistudy packages (Kunert, 2016), suggesting that meta-analyzing internal replications without preregistration can lead to biased results (see van Elk et al., 2015, for a discussion of how meta-analysis can lead to ambiguous results even using current bias-correction procedures).
Recommendations for Reproducible Research and Cross-Cultural Replications
In this section, we present recommendations for conducting robust, informative, and replicable cross-cultural research (see also Asendorpf et al., 2013; Brandt et al., 2014; Munafò et al., 2017; Open Science Collaboration, 2017). Although provided with replications in mind, most of the following guidelines are equally applicable to original research designs. The gold-standard, ideal research design would incorporate all recommendations. However, failing to meet certain criteria does not in-and-of-itself result in an uninformative or unpublishable outcome. We encourage researchers and reviewers to consider these guidelines and critically examine their application to any given project. Ultimately, replications should be evaluated individually on their own merits, and practical constraints or the specific goals of a given project will make some recommendations less appropriate. Recommendations are organized by project timeline, starting with the design phase and going to the research implementation, reporting, and post-publication phases.
Design Phase
Select an appropriate effect for replication
There is no universal set of rules for selecting an effect for replication, but generally one should try to maximize the added value of the replication regardless of outcome. What does one learn about the procedure or theory if the replication is successful? What does one learn if it fails? Often the answers may depend on a combination of the centrality of a finding to theory, the breadth of impact of the finding, and the degree of existing certainty about the effect. Recent developments have caused some to reconsider what we define as an established, rock-solid finding. At the same time, it may be less useful to replicate a study that has already been convincingly replicated on a large scale, compared with a high-impact but less replicated effect. As noted above, effect size may also be considered both for the possible real-world impact of the effect and for statistical power considerations.
Plan for high statistical power
High statistical power is a prerequisite for providing an informative investigation of a phenomenon. A failed replication is more ambiguous to the extent it could simply be a false-negative due to chance (e.g., a study with 60% statistical power will fail to find an effect 40% of the time, even if the effect is real). If the probability that the replication failed due to chance is low, it allows more serious consideration to be given to the failure being due to a methodological artifact, a true cultural difference, or the original finding being a false-positive. Therefore, the common guideline of 80% power with alpha at .05 should be viewed as a bare-minimum, and targeting 90% or greater power will make for a more convincing study.
In some cases, practical limitations will mean a cross-cultural project is not feasible given the resources of the replicating lab in the first place. For instance, researchers without access to a large sample of the population of interest may find it difficult to study effects with a small effect size on their own. However, several solutions do exist to increase the sample size in these cases. When appropriate, one might consider using an online participant pool (e.g., Amazon Mechanical Turk, Prolific Academic, or other research panel services), or supplementing in-lab data collections with additional data from these online sources. Alternatively, researchers are increasingly collaborating with other labs and pooling resources to ensure large sample sizes (see the Many Lab, https://osf.io/89vqh/, for one initiative facilitating these collaborations), and cross-cultural researchers already work together with collaborators from multiple countries/cultural groups.
Aside from increasing sample size, researchers might consider alternatives to increase power such as using within-subjects designs, or increasing the strength of the manipulation (see McClelland, 2000, for a review of alternatives). In light of evidence suggesting published effect sizes are inflated (e.g., Open Science Collaboration, 2015; Simonsohn et al., 2014), when calculating statistical power, it is prudent to use conservative estimates of effect size. When no reliable estimate is available in the published literature, it may be useful to estimate assuming a small-to-medium effect size. One might also consider using Safeguard Power, which considers the imprecision in the effect size estimate when planning for power (Perugini, Gallucci, & Costantini, 2014). Many programs exist to estimate power, and we recommend G*Power as one free and flexible option (Faul, Erdfelder, Lang, & Buchner, 2007).
Determine a sampling plan
Careful consideration should be given to the number and types of cultures to be sampled. If the project is a replication of a cross-cultural effect, utilizing the original cultures provides the closest replication. However, extending the finding to additional cultures that vary on the dimension of interest will increase the information gained from the results (see, for example, Amir & Sharon, 1987). If the key finding to be replicated is proposed to be a specific moderating variable between cultures, at least one additional culture provides a confirmatory opportunity to isolate and test the influence of the proposed cultural moderator. The proposed moderator should be measured across all cultures, and to the extent one can predict the presence or absence of the relationship across cultures, depending on the level of the moderator, one can afford a high degree of confidence to that explanation. The same thinking also applies when exploring cultural variation within countries (e.g., Talhelm et al., 2014). Ideally, enough populations might be sampled to allow for multilevel analyses (see below), and the sampling plan is critical in the process of unpackaging culture (see Bond & van de Vijver, 2010).
Consider meta-analysis and multilevel modeling
Both meta-analysis and multilevel modeling are effective tools for summarizing research conducted across multiple samples and allow for quantification of potential moderating variables (e.g., Nezlek, 2010; van Hemert, 2010). Meta-analytical studies can be conducted to replicate cross-cultural findings (e.g., Vauclair et al., 2011), and both methodologies are useful in estimating the strength of the effect of interest and examining boundary conditions (e.g., Bain et al., 2016; Pratto et al., 2013). For example, Milfont and colleagues (2017) examined whether individuals’ support for the domination of “inferior” groups by “superior” groups influence proenvironmental actions. They meta-analyzed the effect sizes across samples and then employed multilevel methodology for unpackaging cultural variability observed.
Preregistration
Often, the same dataset can be analyzed in any number of justifiable ways, and in the absence of preregistration, it is impossible to know if the reported results came from the only test that was conducted, or if 20 comparisons were made before a given effect emerged and was reported. These multiple comparisons inflate the chances of finding a spurious result. Even experts working with each other to answer the same question from the same dataset will often disagree on the optimal analysis (e.g., Silberzahn et al., under review). Preregistration, and the related concept of Registered Reports journal formats (Nosek & Lakens, 2014), allow for more convincing, informative, and transparent research.
Preregistration entails specifying in advance the sampling plan, methods, hypotheses, and analysis plan. These details are then timestamped and frozen as an unchangeable record. The results stemming from this preregistered plan are thus confirmatory—It is verifiable whether they were predicted and it is known that researcher degrees of freedom in the analysis were limited. These results can be treated with high confidence. Results outside the analysis plan are exploratory. Importantly, these results may still be informative, and exploratory analyses should be encouraged in the context of discovery. However, exploratory analyses must be reported and interpreted with the caveat that they were not predicted in advance, and may be more likely to be false-positives due to multiple comparisons or other researcher degrees of freedom (see Simmons et al., 2011). To afford a high degree of confidence in an exploratory result, a preregistered follow-up study is usually required.
In sum, preregistration helps control for false-positives in results reporting and allows the reader to evaluate the research from a more informed position (see Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012, for a review of the benefits of preregistration, and Van‘t Veer & Giner-Sorolla, 2016, for a review of the benefits/drawbacks as well as key elements of a preregistration). Registered Reports journal formats take this concept one-step further. The entire plan for the study is detailed and preregistered in advance, but it is also submitted as a proposal to a journal. This submission is reviewed as a normal journal submission would be, but in advance of the project beginning. Successful submissions are conditionally accepted for publication regardless of the results, which eliminates publication bias stemming from significant/non-significant findings. For example, Registered Replication Reports at Perspectives on Psychological Science is one initiative that provides a publishing format explicitly for high-powered replications (Simons et al., 2014). While the previously mentioned large-scale replication initiatives focus on numerous results, these Registered Replication Reports investigate a single psychological finding in-depth across multiple labs. Currently, two reports are available from this new publishing format (see Alogna et al., 2014; Eerland et al., 2016), and a growing number of journals are now accepting this format (see https://cos.io/rr/ for a continually updated list).
Implementation Phase
Communicate with original authors
Ideally, contact with the original author(s) who reported the effect to be replicated would be unnecessary, and all required materials and facts relating to a project would be available in a format such that a similar expert in the field would be able to independently replicate the effect. It is unknown to what extent that is currently the case in psychology. In our own research, there have been instances in which original authors caught mistakes or noted otherwise unknown moderators to effects that increased the quality of our replication efforts (R. A. Klein et al., under review). In other projects, endorsement of original authors was associated with increased probability of successful replication (e.g., Open Science Collaboration, 2015), although multiple factors aside from researcher expertise may factor into the decision to endorse or not endorse a replication. If nothing else, communicating with the original authors should help to facilitate a collegial and constructive atmosphere for replication.
Translating and adapting materials
Replications strive to activate the same psychological mechanisms as an original demonstration, and thus researchers should take steps to ensure the replication is appropriately adapted to all sampling contexts. Cross-cultural researchers are likely well versed in these issues already, but we highlight five strategies for ensuring materials are understood similarly across samples. First, a local researcher with deep familiarity with the to-be-sampled population and culture should be involved. Second, in cases where translation is necessary, we recommend the bilingual committee approach for translating and adapting the materials (see Harkness, 2003; van de Vijver & Leung, 1997) and consideration of established guidelines (International Test Commission, 2016). Third, for experimental cross-cultural research, it may be even more critical to include a manipulation check, a measure that independently examines whether the experimental manipulation had the intended effect on the participants. For instance, a study inducing positive or negative moods might include a self-report mood measure to test whether those in the positive mood condition report more positive mood compared with those in the control condition. This can provide confirmation that the materials are functioning similarly across both samples, or indicate a problem if the manipulation does not affect the manipulation check in certain cultures. Importantly, if the procedure fails to produce a difference on the manipulation check, it is almost certainly not a valid test of the underlying theory. Fourth, materials should be pilot tested across all studied cultures to ensure instructions are understood, manipulation checks are functioning as expected, and measures are similarly reliable across settings. Finally, during this adaptation/translation process, it may be prudent to again consult with the original authors. They may have insights from unpublished experiments or other experience that can be used to ensure the critical elements are adapted appropriately.
Reporting Phase
Invariance testing
As argued above, addressing bias and equivalence are important in replication especially because of the quasi-experimental nature of cross-cultural studies. Cross-cultural replications should thus report results of measurement invariance testing whenever possible (Boer, Hanke, & He, 2018) and whether the measurement instrument is indeed equivalent for all compared groups (see Fabrigar & Wegener, 2015).
Report effect size and emphasize confidence intervals
A strict focus on p values can understate variability in the estimates and reveals little about the practical importance of the findings. Focusing on confidence intervals and effect size (e.g., correlation coefficients, Cohen’s d) provides the reader with a fuller interpretation of the precision of the estimates (see Cumming, 2012).
Data analytic decisions
As noted above, researcher degrees of freedom in data analytic decisions is problematic (Simmons et al., 2011), and experts often disagree on the optimal analysis to be conducted (Silberzahn et al., under review). Ideally data analytical decisions would be considered and determined in the design phase of a research project, but it is likely that some decisions would be made after data gathering. When this happens, it is important to consider ways to reduce the impact of arbitrary analytical decisions. A new approach, named specification-curve analysis, has been proposed in helping with this task (Simonsohn, Simmons, & Nelson, 2015). It consists of reporting results for all “reasonable specifications” that are (a) consistent with the underlying theory, (b) expected to be statistically valid, and (c) not redundant with other specifications in the set. Future work is needed to confirm whether this is a viable approach to implement within cross-cultural research, but researchers today would benefit from paying greater attention to the influence of analytic flexibility.
Evaluating replication results
Researchers must report whether the cross-cultural replication succeeded or failed. Currently there is no single standard for evaluating replication results. For example, the Open Science Collaboration (2015) considered five distinct indicators to conclude whether a finding replicated (i.e., significance and p values, effect sizes, subjective assessments of replication teams, and meta-analysis of effect sizes), while Verhagen and Wagenmakers (2014) proposed a test using Bayesian statistics. These indicators provide different information that collectively allows a conclusion about a replication attempt. It is important to note, however, that these indicators focus only on two standard approaches for interpreting replication results (see Simonsohn, 2015): (a) whether the magnitude of the effect observed in the replication study is significantly different from zero, and (b) whether the effect observed in the replication study is significantly different from the effect found in the original study.
Simonsohn (2015) proposed a new approach that focuses on the more consequential question of whether the effect of interest is undetectably different from zero. According to this new approach, A replication that obtains an effect size that is statistically significantly smaller than d33% [effect size giving 33% power to the original study] is inconsistent with the notion that the studied effect is large enough to have been detectable with the original sample size. (p. 562)
We recommend researchers to consider all these approaches when evaluating replication results, and ideally these considerations are discussed in the preregistration step.
Post-Publication Phase
Open data
One of the cornerstones of science is being able to check and confirm results. Perhaps the first step, then, is ensuring data from published work is available to other researchers for further analyses or for testing findings. This is particularly important because willingness to share data has been shown to be associated with fewer errors and greater strength of evidence for the reported finding (see Wicherts, Bakker, & Molenaar, 2011). Data availability policies are commonplace in academic journals and societies, including the American Psychological Association (APA), which requires data be made available to other researchers upon request after publication. However, these policies may not guarantee data availability in practice. Vanpaemel, Vermorgen, Deriemaecker, and Storms (2015) requested data from all articles published in 2012 in four APA journals, and of 394 requests obtained data for only 38% of articles.
Our recommendation is to plan for data sharing up front. A number of options exist where data can be documented, uploaded, and shared, such as the Open Science Framework (https://osf.io) or Harvard’s Dataverse (https://dataverse.harvard.edu/). We recommend preparing and documenting the data to the extent that it does not require any further explanation or preparation on the author’s part when they receive a request. In addition to ensuring these data remain useful years later, authors benefit by being able to easily reference and reanalyze their own past datasets. As further incentive, one might consider publishing the data in a data journal, such as the Journal of Open Psychology Data. These journals allow researchers to receive credit for their data through citations whenever it is used, and allow for a review that the data are sufficiently documented and prepared for others’ use.
Sharing analysis scripts
In addition to reporting the outcome of analyses and data sharing, providing the analysis scripts themselves allows for greater transparency about the analyses conducted. This transparency allows for confirmation of analysis results and greater chances that future researchers will be able to reproduce those results. While this may seem a burdensome extra step, consider that upon obtaining data from 67 papers in economics, researchers were able to reproduce the key finding from only half of those papers, even when original authors were consulted (Chang & Li, 2015). The New Zealand Attitudes and Values Survey (NZAVS) provides a very good example of this practice. The project team provides Mplus syntax for all statistical models of published research using the NZAVS dataset (http://nzvalues.org).
Sharing materials
Publicly available materials make it easier for future researchers to utilize the original materials in future experiments, and likely increase the probability of being able to replicate or extend the effect. In the case of translated materials, this in-and-of-itself may be of great value to future researchers. Perhaps especially critical for cross-cultural research, making stimuli, wordings, and translations available also allows researchers to evaluate the protocol and procedure in fine detail. In addition, requiring open materials at publication may help future-proof our research endeavors and free us from relying on authors to keep and retrieve meticulous private records or recall procedural details years, or decades, after the fact.
Avoid the file drawer
Publication bias threatens the validity and representativeness of the published literature (see Ferguson & Heene, 2012; Franco, Malhotra, & Simonovits, 2016). Public preregistrations help with this, but replications can only possibly be informative to the extent others know about them. A common goal with most work is publication, and one mechanism to ensure replications are surfaced is the Registered Reports format, which is a strong form of a preregistration (see details above). A traditional publication is another option, and journals are beginning to accept replications, or packages of replications, for publication at a greater rate.
Other options that require little or no peer review but still allow for public consumption of research are Psych File Drawer (psychfiledrawer.org), and the Open Science Framework (osf.io). These tools allow researchers to document their replication (or, in the latter case, original research as well) and make it discoverable to other researchers searching for it. Also worth noting is the Curate Science initiative (curatescience.org) where replication attempts are bundled together and real-time meta-analysis of the results is performed. Especially in the case of failed replications or null effects, researchers may find they have little personal incentive to complete this last step at the end of the research process. It is our recommendation to commit to surfacing the research in some way in advance of beginning the project, and plan for contingencies should publication become an untenable plan.
Suggestions for Fostering a Culture of Replication in Cross-Cultural Research
In this article, we have reviewed the increasing focus on replication in psychological science and have discussed and provided recommendations to implement cross-cultural replications. While asking whether and why a replication failed is of interest to any replication project (see Simonsohn, 2015), the why question is particularly central to replication efforts in cross-cultural psychology. The context sensitivity of psychological phenomena means an effect found in a well-powered study might not replicate in another context for a number of reasons, including differences in the sociocultural characteristics between the participants in the original and replication study. Explaining replication failures in terms of cultural differences might be reasonable if theoretically grounded, but it can be problematic because researchers might use culture as a deus ex machina (Poortinga, 2011) and overemphasize cultural differences (Brouwers, van Hemert, Breugelmans, & van de Vijver, 2004).
Given this context and based on the discussion and recommendations above, we suggest good-faith replication efforts in cross-cultural psychology should achieve bare-minimum conditions. Cross-cultural replications should (a) preregister the study design with detailed discussion of sampling plan and contextual moderators of the effect of interest, (b) have 80% power, (c) test and confirm the measurement invariance of the measures, and (d) include control variables to test for alternative explanations. 1 As individualism-collectivism and independent-interdependent views of the self currently are the main dimensions of cultural variability examined in our field, cross-cultural studies explicitly examining the replicability, size and boundary conditions of effects in this domain are encouraged.
We conclude by suggesting practical actions that we could implement to nurture a culture of replication in our field. Our recommendations for improving research practices and conducting cross-cultural replications put most of the responsibility on researchers, but journal editors and reviewers also have an important role in improving psychological science (see Maner, 2014). Indeed, we believe the Journal of Cross-Cultural Psychology could facilitate replication initiatives, accept Registered Replication Reports and enforce “best practices” via editorial policies that will positively benefit cross-cultural research (see, e.g., Eich, 2014; Funder, Levine, Mackie, Morf, Vazire, & West, 2013). Editors and reviewers could consider the recommendations discussed in this article when examining replication attempts more specifically and recommendations made to facilitate open data reporting more generally. As noted by Maner (2014), A culture shift in research practices requires a corresponding shift in the manuscript evaluation system—one that is marked by greater acceptance of imperfections in patterns of data and, at the same time, greater attention to overall scientific rigor and theoretical importance. (p. 344).
One important initiative is to make replication of recent findings a key part in courses of cross-cultural research methods, which will help students to appreciate replication better and will increase the number of independent replications (see Frank & Saxe, 2012). There are examples of such initiatives in social psychology that implement a hands-on approach to teaching replication (see https://osf.io/nxytf/), which has led to publications (e.g., Connors, Khamitov, Moroz, Campbell, & Henderson, 2016). Courses on cross-cultural research methods can foster replication research and open-science practices in young scholars, which will lead to better reporting and interpretation of research findings (see Tullett, 2015) and consideration of methodological issues in replication attempts (see Braver et al., 2014; Cumming, 2014; Earp & Trafimow, 2015; Simmons et al., 2011). Considerable progress has been made in cross-cultural research in the last several decades, and we hope planned replication will become an important aspect of studies examining cultural influences on mind and behavior in years to come.
Footnotes
Acknowledgements
Ype Poortinga and Ronald Fischer provided invaluable suggestions and encouragement throughout the editorial process. The authors also thank one anonymous reviewer and Michèle Nuijten (signed review) for important critics and suggestions to improve the manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.