Introduction to the Registered Replication Report

Registered Replication Reports (RRRs) combine many direct replications of a study, all following an identical protocol, to arrive at a precise and relatively unbiased estimate of the size of an effect (Simons, Holcombe, & Spellman, 2014). The first RRR, a set of replications of an important finding in the area of eyewitness memory, appeared in Perspectives on Psychological Science last year. The second completed RRR is in this issue (Eerland et al., 2016).

RRRs allow us to assess the size and reproducibility of important findings, bypassing the problems of publication bias (all results are published, regardless of outcome) and variability or flexibility in design and analysis (a common protocol is followed and pre-registered). This RRR examined a finding reported by William Hart and Dolores Albarracín in Psychological Science in 2011. Their study was motivated by a broader literature on how grammatical aspect affects one’s interpretation of the description of an event. Reading that a person “did” something suggests a completed event, whereas reading that a person “was doing” something implies a dynamic, ongoing sequence, one that might yield a more vivid and rich memory. The latter form of the verb (“doing”) is called imperfective aspect and the former is known as perfective aspect.

Hart and Albarracín (2011) hypothesized that imperfective aspect would lead people to see a person’s behavior as more intentional or deliberate. In their third experiment, participants in one group who read a vignette in which a shooter “was pointing” his gun and “was firing” shots (imperfective) rated the shooter’s actions as more intentional and deliberate than did participants in a second group who read a version in which the shooter “pointed” his gun and “fired” shots (perfective). The size of the effect was more than 70% of the standard deviation of the ratings within each group (Cohen’s d = 0.76). An effect this large could have practical importance for decision making in the justice system. For instance, describing a suspect’s actions using the imperfective conceivably could result in the judge or jury seeing the suspect as more culpable.

Anita Eerland, Andrew Sherrill, Joseph Magliano, and Rolf Zwaan were intrigued by the size and potential importance of this effect and proposed an RRR. They drafted a detailed protocol for the labs to follow. As editor, I consulted with Hart and Albarracín to refine it. Hart provided the original materials from their study for use in the RRR, and the studies used the same design as the original, with one exception: The two uses of imperfective in a sentence about the critical event sequence clashed, and collectively we agreed to eliminate one, although doing so reduced the number of imperfective instances in the vignette from three to two.

Prior to data collection, each of the twelve participating labs predicted the effect they would observe. Eight expected an effect size of at least d = 0.55 in the same direction as the original. You can view these predictions and the details of how each lab implemented the study on the Open Science Framework pages listed at https://osf.io/79bhs/wiki/home/. The individual results and the result of the planned meta-analysis surprised all twelve of the teams.

While the RRR manuscript was in press, a separate investigation into the effect of grammatical aspect on intentionality judgments was published by some of the RRR authors (Sherrill et al., 2015). Two of those studies found an effect, and two others did not produce significant effects. These experiments were conducted before the RRR was initiated and helped motivate the RRR project.