Preregistered Direct Replications in Psychological Science

I am pleased to announce a new category of articles in Psychological Science: Preregistered Direct Replications (PDRs). PDR articles report high-quality, preregistered, direct replications of studies published in Psychological Science. Our inaugural PDR (Lumian & McRae, 2017) reports a replication of a functional MRI study originally conducted by William A. Cunningham, Jay J. Van Bavel, and Ingrid R. Johnsen and published in Psychological Science in 2008. The original article reported evidence that participants’ processing goals can modulate their amygdala activity. Daniel Lumian and Kateri McRae conducted a high-powered replication of that experiment in consultation with Cunningham and Van Bavel. Lumian and McRae replicated Cunningham et al.’s central finding, and the new study disambiguates some of the earlier findings by virtue of its greater statistical power and multiple converging analyses.

The fact that a study was published in Psychological Science is not a sufficient basis for submitting a PDR of that study to this journal. Authors of a PDR must make a convincing case that the replication itself will make a valuable contribution to understanding a phenomenon or theory of broad current interest to psychologists. As with all Psychological Science submissions, the primary criterion is general theoretical significance. I plan to set a high bar for acceptance of PDRs, and expect that we will publish only a few per year.

Direct replications should reproduce the original methods and procedures as closely as possible, with the goal of measuring the same effect as in the original study. It is impossible to conduct a study twice in exactly the same way, and doing so would not always be desirable: For example, often it would be better to test a larger number of subjects than in the original study, and if the original study included some problematic items or unclear instructions, it is probably best to correct those shortcomings. Also, if the replication study will be conducted in a different cultural context than the original study was, then it may be appropriate to change surface-level aspects of the procedure. The aim of a direct replication is to create conditions that experts agree test the same hypotheses in essentially the same way as the original study did. To that end, researchers undertaking a PDR project are encouraged to consult with the author or authors of the original article.

PDRs will be subject to external review. Typically, an author of the target piece will be invited to provide a review, along with at least two independent experts. In addition to having a high degree of general theoretical significance, PDRs are expected to include strong justification for sample sizes and outcome-independent quality-control checks that maximize the credibility of reported findings. It is expected that data and materials associated with PDRs will be available to reviewers on a third-party Web site and, if the work is accepted for publication, to other psychologists (with appropriate safeguards for confidentiality, etc.; see Lindsay, 2017).

Researchers are strongly encouraged to submit proposals for PDRs for review before data collection. Proposed PDRs will be externally reviewed, and those that meet criteria will be accepted in principle prior to data collection (acceptance will often be contingent on passing manipulation checks, avoiding floor and ceiling effects, etc.; see Center for Open Science, 2017). After data are collected, the manuscript reporting the replication will be reviewed with the expectation that it will be accepted if the agreed-upon criteria are met. One virtue of pre-data-collection submissions is that decisions about scientific merit are independent of the study’s results. Another is that reviewers have an opportunity to improve a study before it is conducted. We may in the future require that PDRs be submitted as proposals prior to data collection, but for now, we will also consider PDRs based on data that have already been analyzed if the studies were preregistered (Lindsay, Simons, & Lilienfeld, 2016).

PDRs are limited to 1,500 words, excluding Method and Results sections. Manuscripts should be no longer than they need to be. Authors are encouraged to use online Supplemental Materials for minutiae that would be of interest only to experts.

PDRs are distinct from Many Labs projects or Registered Replication Reports (RRRs), in that a PDR is a single study, typically conducted by an individual lab. RRRs and other multilab empirical reports will henceforth be published in the new APS journal, Advances in Methods and Practices in Psychological Science, under the continued oversight of Daniel Simons as Editor.

One of the motivations for adding PDRs to Psychological Science is the belief that a journal is responsible for the works it publishes (as per Sanjay Srivastava’s, 2012, “Pottery Barn rule” blog post). That said, the motivation for adding PDRs is not to sow contention. Rather, inviting PDRs is one of many steps we are taking to increase the extent to which this journal contributes to efforts to make psychology a more cumulative science (e.g., Center for Open Science; Chambers, 2017; Eich, 2014; Lindsay, 2015, 2017). Some PDRs will report “successes” in which the original findings are closely replicated, and some will report unambiguous failures to replicate (made compelling by fidelity to the original studies, high statistical power, and appropriate analyses). Both of those outcomes are valuable and informative. Some direct replications will produce ambiguous results, indicating that better methods are needed to resolve the issues in question. And, as Walter Mischel (2009) noted in an APS Observer Presidential column, replications sometimes yield more nuanced results that spark new hypotheses and contribute to the elaboration of psychological theories.