You Could Have Just Asked

Many research projects include a “file drawer” of studies that did not make it into the final paper and that tend to be less supportive of the hypothesis than those included in the published work (Rosenthal, 1979). Even though this publication bias has already been widely acknowledged within the field of psychology, Francis (2012) has set out to estimate the probability of this bias for individual papers. He does so by identifying papers with many replications of the same effect and estimating the likelihood that at least one nonsignificant study was left in the file drawer. When this likelihood is sufficiently high (as he has claimed is the case for our paper, Galak & Meyvis, 2011), he then writes a paper in which he meticulously assesses this probability according to different assumptions.

Although we respect Francis’s diligence, we propose an alternative method that is not only more accurate but also much easier: Simply contact the authors and ask them for their file drawer. In fact, it is this same innovative approach that Simonsohn (2012) used to find out whether Francis himself had conducted any undisclosed tests on other papers (it turned out that he had). Had Francis contacted us, we could have saved him a lot of time and revealed up front what he was trying to discern: We do have a file drawer and, not surprisingly, the studies that made it into our paper tend to provide stronger evidence of the phenomenon than those that did not.

Moreover, had he come to us, he would also have learned that our result is quite robust, even allowing for our file drawer. We know this because we is conducted a meta-analysis of all our data. Our basic finding is that people remember an unpleasant experience as more aversive when they think they will experience it again. We reported eight successful demonstrations of this phenomenon in our paper, but we also conducted five additional studies whose results either did not reach conventional levels of significance or did reach significance but ended up being rhetorically redundant. Following the same procedure used by Francis (2012), we computed effect sizes for all of our 12 experiments (one study is correctly excluded from both our and Francis’s analyses for reasons beyond the scope of this reply) and conducted a simple meta-analysis (Hedges & Olkin, 1985). We found an average effect size (g*) of 0.38 with a 95% confidence interval of [0.25, 0.51]. A z test of g* against zero results in a p value less than 10⁻⁸. In other words, the support for our hypothesis is overwhelming. It is worth noting that the effect size estimated using all our data is smaller than the effect size estimated using only the published data. However, as is the case for many papers in experimental psychology, the goal was never to assess the exact size of the effect, but rather to test between competing theoretical predictions. Our paper does just that.

Francis has argued that if there is a high probability that a paper has a file drawer with a null finding, then that paper should be ignored because the set of reported experiments is incomplete. We think this recommendation is neither logical nor productive. If a researcher is interested in estimating the size of an effect reported in a published paper, we recommend asking the authors for their file drawer and conducting a meta-analysis. For projects that do not aim to estimate the exact size of the effect, we maintain that replications are the best way to mitigate file-drawer effects because producing many replications in the absence of a real phenomenon is not practically feasible. Francis acknowledges this, but he also indicates that such replications become more feasible when one engages in data peeking or other forms of utilizing “researcher degrees of freedom” (Simmons, Nelson, & Simonsohn, 2011). However, we contend that replications can actually reduce researcher degrees of freedom, provided that authors are consistent across replications (e.g., use similar measures and analyses), as was the case in our paper.

In closing, we thank Francis for bringing attention to our paper, but we also want to emphasize that when an answer is out there, it makes more sense to ask for it than to estimate it.