Abstract
Under the strong leadership of Vincent Yzerbyt (2009-2012) and Allen McConnell (2012-2015), and their editorial teams, Social Psychological and Personality Science (SPPS) has had an impressive first 6 years. The journal now has a circulation of about 7,700, publishes 8 issues a year, and has an impact factor of 2.56. These accomplishments reflect the strong reputation SPPS has achieved as a premier outlet for social and personality psychology research and provide a foundation for an auspicious future.
I am thrilled and humbled to be the journal's third editor in chief. My overarching goals as editor in chief of SPPS will be to keep it on this extraordinary trajectory and to adapt the submission guidelines and review process to further improve transparency and rigor of published research in our field. The lively discussion our field has been having in the last few years about replicability and best research practices has led to many new ideas about and approaches to fundamental issues such as statistical power, replication, and transparency. Before discussing the changes happening at SPPS to keep pace with some of these new developments, I want to highlight several features of SPPS that have set it apart and will continue to serve as the foundation of the journal.
First, SPPS complements other excellent journals in our field by publishing exclusively short report format articles. One benefit of this format is that it allows us to have an exceptionally fast turnaround time. The average time from submission to first decisions is 39 days, and the average time from acceptance to online publication is 20 days. Although the prototypical article in SPPS presents one or two novel empirical studies, the short report format can also accommodate multistudy papers, replications, methodological papers, and short theoretical pieces or commentaries.
Second, SPPS benefits from the support of its four founding organizations (the Association for Research in Personality, the European Association of Social Psychology, the Society of Experimental Social Psychology, and the Society for Personality and Social Psychology), and the two organizations that co-sponsor the journal (the Asian Association of Social Psychology, and the Society of Australasian Social Psychologists). These organizations provide guidance to the editorial team, and they help increase the visibility of the journal within the field of social/personality psychology and beyond.
Third, SPPS publishes a broad range of research. A glance at recent tables of contents shows that our articles cover a range of topics, methods, research designs, and subject populations. Indeed, many of the articles in SPPS showcase a range of methods and designs within individual articles. Moreover, SPPS's international scope means that we attract submissions from around the world, and the research published in SPPS reflects this cultural and geographic diversity.
These features of the journal give SPPS its unique identity and strong reputation. They will remain the foundation of SPPS. Building on this foundation, I will also make some changes to ensure that SPPS keeps up with new developments in the field and new standards for best practices in psychological science. Below, I describe my vision for the next few years of SPPS and new policies that are going into effect.
Looking Forward
These are tumultuous times for the world of science in general, but the basic values of our field have not changed. We want to incentivize research that pushes the frontiers of our science, both in terms of studying new topics and developing new methods, as well as research that deepens our understanding of core topics in our field. We want to publish research that is cutting edge as well as research that builds incrementally on our existing knowledge base. This is nothing new. However, the norms for how we conduct, evaluate, and present our research are changing.
There is a lively debate going on around us about whether the field of psychology has a reproducibility problem. Some people take the results of coordinated efforts such as the Reproducibility Project (Open Science Collaboration, 2015), special issues on replication (e.g., Social Psychology; Cohoon, Nosek, & Lakens, 2013), and the Many Labs projects (e.g., Ebersole et al., 2015; Klein et al., 2014) as evidence that our published findings are not as replicable as we had thought (see also John, Loewenstein, & Prelec, 2012). Others think the jury is still out on whether these sets of findings are representative. And still others are confident that the large majority of research published in our top journals is reproducible. Regardless of where one's beliefs lie on this spectrum, one unequivocally positive outcome of this debate is the increased attention to studying how we can evaluate and improve the reproducibility of our research.
Even if the current state is not as dire as some think, improving the reproducibility of our research should always be a goal. Some caution is warranted because some proposed methods for increasing reproducibility are likely to have undesirable side effects or unacceptably high costs. My goal as editor is to balance these perspectives by implementing those changes that have been shown to improve reproducibility at a relatively low cost. It is also important that the pace of change is not so fast that researchers will be left in the lurch, unable to meet the new standards. Thus, the current changes are those that the SPPS consortium and I view as the most reasonable, empirically justified changes that the field is currently ready to adopt.
Aims and Their Implementation.
In this editorial, I aim to describe the changes happening at SPPS and explain the justification behind these changes. I hope this will help guide researchers in making their decisions about which journal to submit their work to, and what to expect if they submit to SPPS. It may also help guide reviewers in evaluating manuscripts for SPPS and readers of SPPS in evaluating the papers they read.
Statistical Power
Among all proposed reforms to make our research more reproducible, I am most interested in increasing the statistical power of our published studies. The statistical power of a study is a good predictor of whether a finding will be replicable (Gervais, Jewell, Najle, & Ng, 2015). However, there is ample evidence that, despite many calls for greater attention to power, the field of social/personality psychology has not done very well in recent decades when it comes to statistical power (Fraley & Vazire, 2014; Maxwell, 2004). Moreover, increasing statistical power is one of the few practices that reduce both Type I error (indirectly) and Type II error. Thus, one of my goals as editor in chief is to place more emphasis on the statistical power of the studies we evaluate.
What does this mean? First, it means asking authors to justify their sample sizes. If the effect being examined is likely to be in the range of typical published effects in social/personality psychology, which is an r of .21 or d of .43 (Richard, Bond, & Stokes-Zoota, 2003), then researchers should aim for a sample size that will provide at least 80% power to detect such an effect or more (given that this is the mean effect size, a large proportion of published effects are actually smaller than this). In addition, because this meta-analysis is based on published studies, and because publication bias leads to inflated effect size estimates, it likely overestimates the size of true effects. Thus, I encourage researchers to consider recruiting a large enough sample to have 80% power to detect an effect smaller than this if possible.
Moreover, if a researcher is interested in an interaction effect, in comparing two effects, or in mediated effects, those effects are likely to be smaller, and thus require a larger sample to reliably detect. In addition, if researchers are interested in interpreting null effects, they should aim to recruit a sample large enough such that the 95% confidence interval around a null effect will exclude effect sizes that would be considered practically meaningful. For a very clear explanation of planning for precision (similar to power planning), see Cumming’s invited article in Psychological Science (2014). Another useful document to consult is the SPSP Taskforce on Best Practice's 2014 publication in Personality and Social Psychology Bulletin (Funder et al., 2013).
However, emphasizing statistical power does not mean requiring large samples in all cases. There is not enough room in this editorial to list all of the good reasons why a study with a small sample might be worthy of publication in SPPS, but here are a few: a design that does not require large samples to achieve statistical power (e.g., some within-subjects designs); a study on a special population whose members are very difficult to recruit; a study using a useful and unique measure that is so expensive or time intensive that it is very difficult to recruit large samples; a study that capitalizes on an event that is not reproducible (e.g., a pre-post study on the effects of a natural disaster); and a set of studies that have small samples but are preregistered. SPPS is not committed to exclusively publishing articles that contain definitive evidence but rather to ensuring that an article's conclusions are justified by the evidence it contains.
In short, because the statistical power of a study is strongly related to its likelihood of being replicable, we will expect authors to plan studies with adequate statistical power to detect small-to-medium effects or to explain why this standard should not apply to their study. If a study has a simple design (e.g., a cross-sectional study of college students or MTurkers using self-reports, reaction time measures, and/or laboratory tasks) and is underpowered, it will have a high chance of being rejected without review. We recognize that there are many reasons to make exceptions to this general rule. By asking authors to provide a justification for their sample size, reviewers and editors will be able to evaluate each study on a case-by-case basis. Of course the justifications themselves, like other aspects of the research design, are subject to critique by reviewers and editors.
What about effects that are genuinely large? Because it is difficult for readers and reviewers to tell the difference between genuinely large effects and effect size estimates that are inflated by flexibility in data analysis, researchers who are confident that they are studying large effects should consider preregistering their hypotheses, research design, and analysis plan to help bolster readers’ and reviewers’ confidence in the results.
What about multistudy papers? It is also useful to think meta-analytically about the total evidence presented in a multistudy paper. A multistudy paper that includes multiple demonstrations of an effect and a meta-analytic summary of the evidence can be very compelling. However, a string of underpowered studies that are all statistically significant is not much comfort–indeed, it is an unlikely pattern even when the effect is real (Schimmack, 2012). Thus, individual studies in multistudy papers should also be well powered or preregistered. When an exception is warranted, and a series of underpowered studies is worthy of publication, we should expect (and tolerate) a mix of significant and not-quite-significant findings.
Less Emphasis on p < .05
Another step we can take to increase the reproducibility of our research is to focus less on whether a p value crosses the .05 threshold or not and to evaluate the strength of the evidence in a more continuous manner. In this spirit, we will now require authors to report exact p values (not just whether the p value is less than .05; for p values above .001), effect size estimates, and 95% confidence intervals around the effect size estimates for key findings (see Cumming, 2014 for a tutorial). The goal of this change is to help authors and reviewers accurately assess the strength of the evidence for their conclusion, rather than thinking in a binary fashion about significant versus nonsignificant results.
In a similar vein, we welcome studies whose primary finding is a non-significant result so long as they have enough power/precision to warrant confidence in the conclusion that the effect is zero or very close to zero (see the section on statistical power). Allowing informative, nonsignificant results to make their way into our journals will help reduce the publication bias that leads to inflated effect size estimates in original research and meta-analyses and should ultimately reduce the incentives to engage in research practices that help researchers achieve statistically significant results at the cost of increasing the rate of false positives (i.e., “p-hacking”; Simmons, Nelson, & Simonsohn, 2011).
Increasing Transparency
One of the key features of the scientific method is its transparency and reproducibility. As Lupia and Elman (2014) state, “The credibility of scientific claims comes, in part, from the fact that their meaning is, at a minimum, available for other scholars to rigorously evaluate. [ …] Such open access to the origins of others’ claims is the hallmark of scientific ways of knowing. Accordingly, when social scientists fail to document their assumptions, decisions, and actions [ … ] it limits others’ abilities to understand the meaning of the scientists’ claims.” (p. 20).
Author disclosures
Of course total transparency is an impossible ideal. We must pick and choose what information to report. The new author disclosure requirements are an attempt to increase transparency about those parts of the research process that are most important to give readers and other researchers the information they need to evaluate the evidence, conduct meta-science, and attempt replications. Specifically, authors will now be asked to report the following in the text of the manuscript: How sample size was determined for each study. All data exclusions (e.g., dropped outliers) and how decisions about data exclusions were made. All measures or conditions for variables of interest to the research question(s), whether they were included in the analyses or not.
The rationale behind these new requirements is to provide information to readers about flexibility in data collection and analysis. Knowing whether the decision about when to stop collecting data was contingent on the results, whether the results are affected by data exclusions, and whether any potentially relevant results have been omitted will encourage authors and readers to evaluate the strength of the evidence in the context of the power/precision of the study and the risk of increased Type I error due to flexibility in data analysis.
Tables and figures do not count against the word limit
Many authors likely wish to disclose more details of their materials, procedure, data, and results. We encourage authors to make use of resources such as the Open Science Framework for making materials, data, and code publicly available. However, to facilitate authors providing valuable basic information about their results, we have now removed tables, table notes, figures, and figure captions from counting toward the 5,000 word limit. This change will allow authors to thoroughly report descriptive statistics and all the important details of their results.
Editor disclosure
We also want to increase the transparency of our side of the peer-review process. One step we are taking is to publish the name of the handling editor (or “action editor”) for each published article. This change will facilitate meta-science and provide reassurances (and incentives) against conflicts of interest between editors and authors.
Valuing Both Cutting Edge and Incremental Research
What kinds of articles should a journal publish? In my view, one way to think about this is like an investment portfolio. It is important to diversify, not just in terms of topics, populations, and methods but also in terms of riskiness or uncertainty of the conclusions. As Srivastava (2011) wrote on the SPSP blog, an individual paper is unlikely to be both “groundbreaking” and “definitive,” and journals should not expect papers to do (or to claim to do) both. However, a journal can value and publish both types of research. Traditionally, short report journals like SPPS focus mostly on the “groundbreaking” side of that trade-off. But SPPS can be a home for both kinds of papers and everything in between.
Some examples of provocative but preliminary research include research that is difficult to conduct because of the population, method, or design that is used but tackles an important question. In these cases, it might be necessary to publish smaller, less conclusive studies that provide preliminary evidence and are likely to stimulate further research. Examples of less novel but more conclusive research include rigorous replication studies, incremental work that improves on the method or design of previous studies in significant ways, and studies that help resolve apparent inconsistencies in a literature.
I hope that explicitly recognizing the value of both types of research, and welcoming submissions along the entire spectrum, will allow authors to freely present preliminary studies as such, with an explanation of why the results should be published now rather than after collecting more evidence, and without pressure to make unwarranted claims about the strength of the evidence. Likewise, I hope that authors will feel free to present incremental studies as such, with an explanation of how the study improves our understanding or certainty about an important phenomenon, and without pressure to make strong claims about novelty.
Replications
In the spirit of valuing incremental research that moves us closer to a definitive conclusion, SPPS is now accepting replication papers. We have not developed specific guidelines for replication papers, but they will be evaluated, like all other papers, based on the rigor of the study, the importance of the finding, and the strength of the evidence. We encourage authors to think carefully about how to interpret the results of replication studies (see, for example, Simonsohn, 2015). Preregistered replications are especially likely to fare well, and although we will consider replications of any social/personality study, replications of studies published in SPPS will be especially valued. If the need arises, we will develop more specific guidelines.
In addition, we welcome replications within a paper. All else being equal, a submission that includes a new study and a direct replication will have a better chance of acceptance than one that includes only a new study without replication. If the replication is preregistered, this will further increase the strength of the evidence for the result.
Exploratory research
We also encourage authors to submit exploratory research. Exploratory research should be presented as exploratory, rather than dressed up in confirmatory language. Whenever possible, exploratory findings should be followed up with a confirmatory study, but when this is prohibitively costly, exploratory findings may be worthy of publication on their own.
Concluding Remarks
I have been accused of being idealistic about the future of our field, and I will admit, I am wildly optimistic. This is mostly the result of my experiences as an associate editor (for SPPS, Journal of Personality and Social Psychology, Journal of Research in Personality, Perspectives on Psychological Science, Frontiers in Social and Personality Psychology, and Collabra). Editing has given me a new window into the scientific process, and my experiences have been almost uniformly positive. I am constantly awed by the extremely rigorous and generous service that reviewers provide—spending hours reading and critiquing strangers’ work for no direct reward. I am humbled by the excellent work done by authors and by their openness to reviewers’ and editors’ suggestions. I am impressed with the constructive, respectful tone of communications among authors, editors, and reviewers. I have learned a great deal from editing, both about the excellent research being conducted in social/personality psychology and about the strong commitment many of us have to our field and to good science. The talent, dedication, and good will on display during the peer-review process are truly remarkable. These experiences make me optimistic about the future of our field and are the reason I enthusiastically agreed to take on this role.
I am equally optimistic about the future of SPPS. I am delighted to have recruited a team of superb associate editors and I am very grateful to have them on board. They are: Wiebke Bleidorn, Lorne Campbell, Jesse Graham, Dominique Muller, Nickola Overall, Kate Ratliff, Joe Simmons, Gerben van Kleef, Greg Webster, and Tessa West. I also wish to thank Amanda Richardson, Dan Sawney, Calyx Schentrup, Tzveta Mihaylov, Kelly O’Rear, and the rest of the team at SAGE, who help keep the publication process running smoothly, and the consortium of SPPS, chaired by Linda Skitka, for their guidance and support. Finally, I want to express my gratitude to the editorial board members and ad hoc reviewers for contributing your time and effort, and to authors for choosing SPPS as an outlet for your excellent research.