From Outcome to Process Focus: Fostering a More Robust Psychological Science Through Registered Reports and Results-Blind Reviewing

Abstract

A variety of alternative mechanisms, strategies, and “ways of doing” have been proposed for improving the rigor and robustness of published research in the psychological sciences in recent years. In this article, we describe two existing but underused publication models—registered reporting (RR) and results-blind reviewing (RBR)—that we believe would contribute in important ways to improving both the conduct and evaluation of psychological research. We first outline the procedures and distinguishing features of both publication pathways and note their value for promoting positive changes to current scientific practices. We posit that a significant value of RR and RBR is their potential to promote a greater focus on the research process (i.e., how and why research is conducted) relative to research outcomes (i.e., what was observed or concluded from research). We conclude by discussing what we perceive to be five common beliefs about RR and RBR practices and attempt to provide a balanced perspective of the realities likely to be experienced with these systems.

Keywords

robust science registered reporting results-blind reviewing publication practices

Growing concern over the state of research in the psychological sciences (C. J. Anderson et al., 2016; Gilbert, King, Pettigrew, & Wilson, 2016; Klein et al., 2014; Open Science Collaboration, 2015) has spurred interest in identifying new means for strengthening how we conduct, evaluate, and disseminate our scientific research—necessity, as the saying goes, is the mother of invention. Indeed, many compelling visions have been advanced for what scientific practices in psychology could look like under alternative systems (e.g., Nosek & Bar-Anan, 2012; Nosek, Spies, & Motyl, 2012; Simmons, Nelson, & Simonsohn, 2011; Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012). Changing our large and decentralized research ecosystem will certainly be no small feat (Grand et al., 2018; National Academies of Sciences, Engineering, & Medicine, 2017). However, organizational change researchers have long noted that providing transparent accounts of current institutional dilemmas coupled with knowledge of possible ways forward is among the most effective mechanisms for empowering desired change (Lewin, 1947; Thundiyil, Chiaburu, Oh, Banks, & Peng, 2015).

To this end, we hope to contribute to this conversation by highlighting two alternative publication mechanisms—registered reporting (RR) and results-blind reviewing (RBR)¹—that we believe could have far ranging implications for improving how our research is conducted, evaluated, and interpreted and that ultimately promote a more robust psychological science. We make no claim to the originality or novelty of either mechanism; both have existed in the scientific publishing domain for some time. However, journals that use RR or RBR are relatively rare in psychology, and thus opportunities to experience and evaluate these techniques have been limited. As of this writing, only 91 journals offer RR and only 11 offer RBR as options for reviewing and publishing manuscripts (for a regularly updated list, see https://cos.io/rr).² As a result, we suspect that many in our community may be less familiar with how these practices operate, with how they interface with the conduct and dissemination of research, and with their potential for addressing a host of detrimental research practices (DRPs) observed within the current research environment—hypothesizing after results are known (HARKing), p-hacking, etc. (Murphy & Aguinis, 2017).³ The goal of this article is to first describe RR and RBR publishing models and their potential value for fostering robust science and then address what we see as common beliefs and realities associated with these practices.

Overview of Registered Reporting and Results-Blind Reviewing

The manuscript submission and review process for research submitted under an RR or RBR system follows a nearly identical two-stage process.⁴ During Stage 1, authors submit a manuscript for review that contains only the following sections: introduction (e.g., theoretical or conceptual rationale, relevant literature review, specific hypotheses), methods (e.g., detailed study or data collection procedures, description and justification of all measures including psychometric and any other relevant measurement information, equipment to be used, recruitment plan for sample), and analysis plan (e.g., data wrangling/munging/cleaning procedures, statistical tests and criteria for evaluating hypotheses, results of power analyses).⁵ Notably absent from an initial submission under either publication model is any description of findings, conclusions, or related interpretations. This allows—and in fact requires—all editorial actions and decisions during this stage to be driven by the plausibility, importance, and soundness of the research questions and the approach to answering them, and not on what was found.

On the basis of these criteria, a Stage 1 manuscript may be rejected, be invited for further revisions, or receive an in-principle acceptance. The latter outcome is a formal indication from an editorial team that the research is meritorious, methodologically sound, and will be published so long as the procedures described in the initial submission are followed, all analyses are competently performed, any implications or recommendations advanced follow from the results, and limitations are transparently discussed. If an in-principle acceptance is received, authors finalize data-collection procedures, conduct all analyses in the agreed-on manner, and add the remaining results and discussion to the previously submitted manuscript. The full manuscript is then submitted for a Stage 2 review. During this review, the editorial team assesses only whether (a) the previously stated study protocols were followed; (b) results were reported accurately and clearly and the interpretations provided by the authors warranted; (c) the discussion section effectively contextualizes the theoretical, conceptual, and/or practical implications of the results; and (d) any deviations or post hoc analyses included in the manuscript are justified, informative, and fully transparent. If these criteria are met, the manuscript is published. Review teams are not asked to evaluate the potential or perceived contribution of the manuscript in light of the study results, and any revisions requested during Stage 2 are to clarify procedural elements. An intended consequence of both RR and RBR evaluation systems is that a much larger proportion of the review time for manuscript submissions—and by extension the strongest weight when making acceptance decisions—occurs during Stage 1 (manuscript without results) rather than during Stage 2 (manuscript with results).

Although the submission process is similar for both publication methods, RR and RBR differ in two important respects that hold implications for how research is evaluated: (a) the timing of data-collection activities relative to a Stage 1 submission and (b) preregistration of methods and analyses. In the case of RR, no data collection occurs before the Stage 1 review process; in contrast, data collection and analysis may be ongoing or already completed before submission of manuscripts using RBR. The fundamental consequence of this difference is that RR submissions allow editorial teams and authors to (a) evaluate, suggest, and implement revisions to a study’s rationale and methodology to address perceived limitations before authors or reviewers know anything about results, and (b) collectively agree on a “prospective contract” for how data will be collected and analyzed. In this sense, research articles generated through RR methods have greater potential to be “cocreated” by authors, reviewers, and editors and for substantive changes to be made to a project’s hypotheses, methods, and analyses before knowledge of results, therefore minimizing the likelihood of DRPs throughout the entirety of the research process.⁶

In contrast, authors may have already collected data, conducted analyses, and may be aware of findings in studies using RBR. Submissions under this system are thus more akin to “retrospective contracts” that cannot incorporate revisions to a study’s design or data collection procedures beyond requesting alternative analytic tests or gathering of new or additional data. RBR processes are instead meant to alter the lens through which researchers and reviewers approach empirical work (results are secondary to well-articulated and transparent rationale, methods, and execution), but are not able or intended to directly influence how that research is designed and conducted. The goal of the RBR approach is simply to reduce any perceived rewards or advantages that engaging in DRPs during data analysis or manuscript preparation (e.g., HARKing, selectively including only significant findings, excluding null or contradictory results, removing “uncooperative” data) holds for publication decisions and therefore reduce the instrumentality of these practices for researchers.

The Value of RR and RBR Practices: Shifting From Outcome Focus to Process Focus

Table 1 summarizes the procedures, advantages, and limitations of RR, RBR, and status quo publication models for fostering robust science. Although each offers different strengths and weakness for disseminating research, we believe the most significant value of RR and RBR publication models is their potential to strike a healthier balance between a focus on process versus outcome when conducting and reviewing research (Table 1, last row). We define process focus as the degree to which the accumulation and dissemination of scientific research in a field is driven by evaluations of how and why research was conducted relative to what finding or conclusions are presented. The mantra of a process focused research community is that research studies that are relevant to science or society and that are conducted well should have a home in the scientific literature base, irrespective of findings (Grand et al., 2018).

Table 1.

Comparison of Publication Practices

Category of comparison	Registered reporting (RR)	Results-blind reviewing (RBR)	Traditional/status quo
Description	Authors initially submit a Stage 1 manuscript that includes only theoretical rationale, hypotheses, detailed study protocol, and analysis plan for review before any data collection. Reviewers can request revisions relevant to conceptual rationale, methodology, data collection procedures, and analysis plan. If Stage 1 submission is approved, authors receive in-principle acceptance and perform study. After data collection and analysis, authors submit full manuscript with results for final review as Stage 2 manuscript to ensure there are no deviations from original submission. If no unjustifiable deviations are noted, manuscript is accepted and published.	Authors initially submit a Stage 1 manuscript that includes only theoretical rationale, hypotheses, detailed study protocol, and analysis plan for review. Data collection can be in progress or can have already occurred. Reviewers can request revisions relevant to conceptual rationale and analysis plan. If Stage 1 submission is approved, authors receive in-principle acceptance and incorporate results into manuscript. Authors submit full manuscript with results for final review as Stage 2 manuscript to ensure no deviations from original submission. If no unjustifiable deviations are noted, manuscript is accepted and published.	Authors submit complete papers that include theoretical rationale, hypotheses, methods, analyses/results, and discussion/interpretation for review. Manuscript acceptance is based on the extent to which theoretical rationale is well explicated, methods and analyses are accurate, conclusions are appropriate to the methodology, and potential contribution/impact of results are significant, novel, or counterintuitive.
Advantages	• Manuscript decisions are not influenced by outcomes of research • All methodology, data-collection procedures, and analyses are prospectively vetted and agreed on by authors and reviewers before knowledge of results/data • Reduces motivations to engage in DRPs during all stages of research • Opportunity to receive feedback and revise methods before data collection • Encourages precision in theoretical rationale, hypothesis development, methods, and analysis plan • Process for authors similar to grant/dissertation proposals (i.e., only rationale, methods, analysis plan submitted for review)	• Manuscript decisions not influenced by outcomes of research • Analysis plan is vetted and agreed on by reviewers before knowledge of results • Reduces motivations to engage in DRPs during analysis stage of research • Flexibility in data collection timing requirements (vs. RR) more amenable to “opportunistic” research • Encourages precision in theoretical rationale, hypothesis development, methods, and analysis plan • Process for authors similar to grant/dissertation proposals (i.e., only rationale, methods, analysis plan submitted for review)	• Can easily accommodate multiple manuscript formats (e.g., theory papers, literature reviews, multistudy papers) • Infrastructure and norms are established and common knowledge
Limitations	• Not easily adaptable for all types of manuscripts • No flexibility in data collection timing requirements (vs. RBR), less amenable to “opportunistic” research • Norms for evaluating potential contribution of study are ambiguous (e.g., “Is this study still valuable if it returns only null results?”)	• Not easily adaptable for all types of manuscripts • Less opportunity to identify, improve, and regulate methodological concerns during review (vs. RR) • Norms for evaluating potential contribution of study are ambiguous (e.g., “Is this study still valuable if it only returns null results?”)	• No mechanisms for identifying or ensuring that authors do not engage in DRPs during research process • Explicit reliance on subjective decision criteria related to contribution and impact of research results • Implicit focus on research outcomes in manuscript decisions (novelty, counterintuitiveness, significance) contributes to DRPs during research process
Process focus	Highest	Moderate to high	Lowest

Note: DRP = detrimental research practices (e.g., HARKing, p-hacking, manipulating measures, selecting result-favorable analytical strategies; National Academies of Sciences, Engineering, & Medicine, 2017).

We believe publication practices such as RR and RBR, though only one piece of the puzzle, provide an important infrastructure to support these goals. RR procedures are likely the most rigorous and strict in engendering a process focus given that the evaluations, actions, and proposed revisions of researchers and editorial teams revolve wholly around ensuring the rationale, methods, and analytical strategies of research are robust before data collection. RBR publication models are likely moderate to high in process focus in that dialogue during the review stages still concerns only the rationale, methods, and analytical strategies of research (i.e., the process), but opportunities to revise or correct methods are more limited given that data collection may be complete or in progress. An advantage of RBR, however, is that it can more readily accommodate a broader suite of research methodologies (e.g., “opportunistic” data collections, applied or field-based research contexts). In any event, RR and RBR both facilitate a process focus to a much greater extent than do status quo publishing practices in which the interpretation of results (i.e., outcomes) of research are not divorced from the review process.

Common Beliefs and Realities of Registered Reports and Results-Blind Reviews

We recognize that there are concerns about the utility, pragmatism, and limitations of both RR and RBR practices. Some of these point to genuine boundary conditions or matters still to be resolved if such systems were adopted. However, others reveal areas where resistance to changing the status quo may stem from uncertainty or lack of reliable information. Consequently, we describe five common perceptions about RR and RBR systems and attempt to provide a balanced picture of the reality that could be expected if such models were implemented. We do not suggest that RR or RBR models are infallible or capable of correcting any and all issues faced in the conduct and evaluation of science (Chambers, 2015; Chambers, Feredoes, Muthukumaraswamy, & Etchells, 2014). However, we believe they represent viable methods for improving the robustness of our science that cannot be adequately addressed by current publication practices and thus merit consideration.

Belief 1: RR and RBR procedures limit opportunity for discovery or revised thinking

RR and RBR allow for inductive exploratory analyses to be presented (whether planned or unanticipated). During either the initial submission or following an in-principle acceptance, authors can include exploratory research questions and/or additional analyses alongside those previously disclosed. The key is that such publication models require authors and editorial teams to clearly identify whether any substantive analysis or research questions were proposed before or following data collection. Under RR and RBR publication models, all analyses can still be recognized as valuable and appropriate as long as there is transparency.

We do believe it likely that authors, editors, and reviewers in an RR or RBR system may have to (re)train themselves on how to evaluate the utility of exploratory analyses given the reigning emphasis of hypothetico-deductive and confirmatory theory testing in our science (Hambrick, 2007; Wagenmakers et al., 2012) to recognize that transparency is more beneficial to science than the packaging of findings into a nice and neat artificial box. A robust science is one capable of embracing both inductive and deductive philosophies while simultaneously attempting to bolster their inherent limitations; science is the pursuit of truth, and all practices that help our research better identify those truths should be supported (M. S. Anderson, Martinson, & De Vries, 2007; Grand et al., 2018; Locke, 2007). In this respect, RR and RBR methods offer a cleaner and more natural means for transparently distinguishing between confirmatory and exploratory analyses so that the scientific community can apply the appropriate lens for interpreting the results of scientific research.

Belief 2: RR and RBR approaches result in more “null” results that do not contribute to understanding

Such comments imply two separate beliefs: (a) RR and RBR will lead to more null results in the literature, and (b) null results do not contribute to our understanding of focal phenomena. If there is merit to the assertion that current publication practices tend to favor studies that report statistically significant results, we agree that the first of these hypotheses is plausible (e.g., Campbell, 2014; de Winter & Happee, 2013; Ioannidis, 2005; Rosenthal, 1979). This is, however, ultimately an empirical question (e.g., for a discussion of the prevalence of null results in a special RBR issue for government and politics research, see Findley, Jensen, Malesky, & Pepinsky, 2016). We should not simply conjecture that the literature will become rife with null results without implementing these approaches and evaluating the fruits of such efforts. Our working hypothesis is that we would not expect to see published papers overrun with null results per se; rather, we would see papers with more mixed or equivocal results (as we frequently see in graduate-student theses or dissertations). After all, science is a messy business (McGrath, 1982). However, the upside of such work is that it identifies where future research need be invested to precisely tease apart the critical process mechanisms and boundary conditions of phenomena (Edwards & Berry, 2010; Meehl, 1967). Unfortunately, if only a handful of journals offer such opportunities, we do not give RR and RBR approaches an adequate test of their effectiveness.

With respect to the second point and rather than focusing on whether more null results will appear in the literature under RR and RBR practices, we should reconsider how or whether the quality and rigor of accumulated knowledge will be adversely affected if null findings are published. Given that RR and RBR models emphasize publication decisions based on sound conceptual or theoretical rationale and proposed methodology and analyses, there is no reason to believe that accepted manuscripts should be less robust than those accepted under more traditional reviewing procedures. Results (be they statistically significant or not) that are associated with well-designed studies and address important phenomena can still advance our understanding in meaningful ways (e.g., Landis, James, Lance, Pierce, & Rogelberg, 2014; Landis & Rogelberg, 2013; Rosenthal, 1979). For example, publishing evidence of null findings from well-conducted research with appropriate statistical power is important to ensuring that more complete and accurate distributions of effect sizes are available for meta-analyses (Banks, Rogelberg, Woznyj, Landis, & Rupp, 2016; Ioannidis, 2016). In short, no intrinsic characteristics of RR or RBR practices suggest that they will add uninteresting or poorly designed research into the published literature.

Belief 3: RR and RBR procedures are more work for authors and reviewers

From the perspective of authors, we acknowledge that the RBR and RR process is different but do not agree it is more laborious. An initial submission is typically shorter than a full manuscript because it contains only an introduction, methods, and analysis plan. Anecdotally, the analysis plan section appears to be the most angst-provoking out of concern that it connotes too much rigidity and unrealistic forethought (e.g., analyses cannot be completely specified as they are dependent on what is observed). However, analysis plans in a Stage 1 RR or RBR submission can specify contingent or exploratory analyses, and revisions to these plans are likely to be a significant focus of the review process during this phase. Furthermore, nothing prevents authors from pursuing previously unplanned analyses after data collection so long as they are identified as such in the Stage 2 manuscript. The purpose of the analysis plan is simply to provide a mechanism through which authors and review teams can vet how substantive research questions will be examined in ways that are not influenced by simultaneous presentation of results. Furthermore, in the case of an RR submission, constructive, process-focused feedback can be received during a stage where potential revisions can improve the rigor of a study (i.e., before data are collected). Such a system stands to actually lessen the time between idea generation and publication because it can reduce the current norm of “shopping” completed but flawed papers across multiple journals until they find a home (Nosek & Bar-Anan, 2012).

From the perspective of reviewers, the cadence of the review process will also be different, but again not necessarily longer or more difficult. Initial RR and RBR manuscript submissions are shorter and the review mission more focused on whether the research questions—irrespective of results—are compelling and proposed to be answered in a rigorous manner. When reviewing Stage 2 manuscripts with an in-principle acceptance, the review task is further simplified and focuses on whether the authors delivered on what they proposed, interpreted results correctly, and presented an appropriate discussion of findings. It is also worth noting that reviewers (and authors) already have plenty of experience managing manuscripts under an RR or RBR system because it is effectively the same procedure used to prepare, review, and evaluate graduate-student theses and dissertations and grant applications. In this regard, RR and RBR processes reflect how many of us learned to do science and how we teach others to do science.

Belief 4: RR and RBR procedures are not suitable for all types of research

RR was originally intended for quantitative primary studies to help distinguish between confirmatory and exploratory analyses, with an emphasis on studies conducting confirmatory analyses (Chambers et al., 2014; Wagenmakers et al., 2012). RBR was proposed as a less restrictive means for accomplishing the same (Kepes & McDaniel, 2013). Yet RR and RBR can also be adapted for quantitative inductive studies, traditional qualitative studies, and meta-analytic reviews with little to no loss of their benefits. Again, the point of the RR and RBR process is to place emphasis on the evaluation of the methodological rigor of a study as well as its potential theoretical and/or practical contribution; this is a goal that can be accomplished irrespective of study design.

That said, we do believe that new or unanticipated complications may emerge for different types of studies that might prove more challenging in an RR or RBR system. For example, editors may be hesitant to offer an in-principle acceptance to meta-analytic reviews via an RR or RBR process without knowing whether there is a sufficient number of primary samples for a meta-analysis. Editorial teams may also have difficulty evaluating qualitative inductive studies in which the foci of research or data collection evolves organically as understanding of the phenomenon is accumulated. A similar challenge exists with multiple study designs intended to build on one another, particularly if the first study is more exploratory in nature. We do not believe these issues are insurmountable, but we predict that they will require additional items as part of a Stage 1 RR or RBR submission than a typical RR or RBR submission (e.g., evidence of number of studies or maturity of research topic for meta-analyses, appendix describing interview methodology and criteria for establishing saturation, contingencies for multipart studies if initial experiments support or fail to support main hypotheses). Alternatively, they may require authors to seek counsel from editors more regularly than is currently the norm before submitting. In sum, we believe RR and RBR models are suitable for many types of research studies, but we acknowledge that unique issues may emerge for different methodologies (see also Chambers et al., 2014).

Belief 5: RR and RBR procedures cannot eliminate gaming the system

Neither RR nor RBR is a panacea; they will not solve every problem identified within the current research enterprise. For example, neither model explicitly incentivizes replications (though both can be used for this purpose and the ability to provide authors with an in-principle acceptance may mitigate the perceived “risk” of investing time and resources into replication research). In addition, researchers submitting manuscripts in an RR or RBR system still possess opportunistic degrees of freedom that can influence the publication decision. For example, manuscripts submitted via RR and RBR will require power analyses to justify sample size, and authors might be tempted to use larger estimates of probable effect sizes to mitigate the need for large samples (S. F. Anderson, Kelley, & Maxwell, 2017). Furthermore, new problems during the manuscript review process may emerge or be introduced in an RR or RBR system. With reviews no longer focused on the statistical significance of findings, editorial teams may turn their attention to other unanticipated criteria or engage in review practices that are equally unrelated to research quality (e.g., reviewers and editorial teams become too influential in how research is conducted).

Despite such potential shortcomings, RR and RBR can help alleviate DRPs that are motivated by a desire to obtain a significant result (e.g., selective use of covariates, dropping nonsignificant hypotheses) because results are not part of the manuscript acceptance decision. More important though, we should not fall prey to the perfect solution fallacy and be so quick to reject RR and RBR because manipulation of the publication process could still occur. The goal of improving scientific practice through RR and RBR is not to eliminate ways for people to game the system; acts of commission and omission will always exist. The goal is to redirect the focus of attention away from a system in which interpretations of research quality are primarily driven by outcomes (what was observed) toward one that is more driven by process (how and why observations were gathered and the relevance of doing it that way). In other words, we do not propose that RR or RBR will eliminate “game playing”; rather, we believe they can change the research ecosystem in a way that decreases the implicit “need” for game playing, at least with respect to attaining publication success.

Conclusion

Many have noted that a driver of the “research crisis” in the psychological sciences and beyond stem from norms within the academic environment that associate research quality (and subsequent rewards such as publications, status, recognition) with the novelty, significance, or unexpectedness of research findings (Antonakis, 2017; Banks, O’Boyle, et al., 2016; O’Boyle, Banks, & Gonzalez-Mule, 2017). These norms can be addressed only by changes to the culture of our science—its practices, policies, and procedures—that reduce potential advantages of or opportunities to conduct and disseminate research in ways that eschew principles of robust science. Changes in our publication methods, such as the use of RR and RBR approaches, represent a unique and currently underused point of leverage for reaching these goals.

One of the provocative questions raised in this symposium is whether altering norms and practices in ways that promote robust science might inadvertently discourage scientific creativity. Regardless of changes made to the scientific enterprise, the generation of unique research ideas, methods, and analytical frameworks will (and should) continue to be valued; innovation, discovery, and exploration of the unknown are central to the fabric of science (Astley, 1985; Davis, 1971; Pfeffer & Fong, 2002; Weick, 1989). In this regard, creativity in idea generation and methodologies are undoubtedly important to our science and should be fostered. However, one can (and should) also be concerned with the “creative license” afforded when testing theories and managing, analyzing, and reporting data. It is our sense that a great deal of creative effort is currently geared toward “winning” the publication game and ensuring one’s results present a pristine narrative rather than focusing those creative efforts toward developing rigorous scientific explanations that embraces, reconciles, and integrates inevitably messy data. Furthermore, a scientific ecosystem that too strongly favors creative ideas at the expense of transparency, rigor, reproducibility, and accumulative evidence threatens its credibility and can lead to situations that erode stakeholder trust in its products (Institute of Medicine, National Academy of Sciences, & National Academy of Engineering, 1995). Efforts to improve the quality and rigor of our science (such as through embracing RR or RBR publications models) need not come at the expense of curiosity, originality, and ingenuity. Creativity and rigor is not an either/or proposition in science; both are necessary and essential to advancing understanding of the natural and social worlds in which we live.

To this end, we believe a healthier psychological science would be facilitated if the evaluations and foci of research were less concerned with the outcomes of research (i.e., Does a finding replicate or not? Is a finding significant or not? Is a finding statistically powered or not?) and more concerned with the manner by which research is conducted (i.e., to what extent are rationale and hypotheses precise about mechanisms, magnitude, conditions, etc.? To what extent are data reliable and unambiguous indicators of the phenomenon? To what extent do methods and analyses permit uncertainty in interpretation?). We believe RR and RBR publication models offer a means for incrementally advancing these changes. Both methods ensure that the determination of scientific quality during the peer review process has strong theoretical conceptual grounding, methodological rigor, and transparency of research in ways that cannot be directly influenced by knowledge of results. Consequently, both the producers and evaluators of science are encouraged to focus more directly on how and why research is conducted rather than what was observed. No single change will ever be capable of correcting all current or possible flaws in the scientific enterprise. However, we believe both RR and RBR publication models have the potential to change certain norms and rules within our ecosystem in ways that foster a more robust psychological science and hope more members of our research community will explore and entertain these and other alternative methods.

Footnotes

Action Editor

Robert J. Sternberg served as action editor and editor-in-chief for this article.

Declaration of Conflicting Interests

The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article.

Notes

References

Anderson

C. J.

Bahnik

Barnett-Cowan

Bosco

F. A.

Chandler

Chartier

C. R.

. . . Zuni

(2016). Response to comment on “Estimating the reproducibility of psychological science.” Science, 351, 1037.

Anderson

M. S.

Martinson

B. C.

De Vries

(2007). Normative dissonance in science: Results from a national survey of U.S. scientists. Journal of Empirical Research on Human Research Ethics, 2, 3–14.

Anderson

S. F.

Kelley

Maxwell

S. E.

(2017). Sample-size planning for more accurate statistical power: A method adjusting sample effect sizes for publication bias and uncertainty. Psychological Science, 28, 1547–1562.

Antonakis

(2017). On doing better science: From thrill of discovery to policy implications. Leadership Quarterly, 28, 5–21.

Astley

W. G.

(1985). Administrative science as socially constructed truth. Administrative Science Quarterly, 30, 497–513.

Banks

G. C.

O’Boyle

Jr. Pollack

J. M.

White

C. D.

Batchelor

J. H.

Whelpley

C. E.

. . . Adkins

C. L.

(2016). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42, 5–20.

Banks

G. C.

Rogelberg

S. G.

Woznyj

H. M.

Landis

R. S.

Rupp

D. E.

(2016). Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology, 31, 323–338.

Campbell

(2014). Registered reports: A step change in scientific publishing. Retrieved from the Elsevier Connect website: https://www-elsevier-com-s.web.bisu.edu.cn/reviewers-update/story/innovation-in-publishing/registered-reports-a-step-change-in-scientific-publishing

Chambers

C. D.

(2015). Ten reasons why journals must review manuscripts before results are known. Addiction, 110, 10–11.

10.

Chambers

C. D.

Feredoes

Muthukumaraswamy

S. D.

Etchells

P. J.

(2014). Instead of “playing the game” it is time to change the rules: Registered reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1, 4–17.

11.

Davis

M. S.

(1971). That’s interesting! Towards a phenomenology of sociology and a sociology of phenomenology. Philosophy of Social Science, 1, 309–344.

12.

de Winter

Happee

. (2013). Why selective publication of statistically significant results can be effective. PLOS ONE, 8(6), Article e66463. doi:10.1371/journal.pone.0066463

13.

Edwards

J. R.

Berry

J. W.

(2010). The presence of something or the absence of nothing: Increasing theoretical precision in management research. Organizational Research Methods, 13, 668–689.

14.

Findley

M. G.

Jensen

N. M.

Malesky

E. J.

Pepinsky

T. B.

(2016). Can results-free review reduce publication bias? The results and implications of a pilot study. Comparative Political Studies, 49, 1667–1703.

15.

Gilbert

D. T.

King

Pettigrew

Wilson

T. D.

(2016). Comment on “Estimating the reproducibility of psychological science.” Science, 351, 1037.

16.

Grand

J. A.

Rogelberg

S. G.

Allen

T. D.

Landis

R. S.

Reynolds

Scott

J. C.

. . . Truxillo

D. M.

(2018). A systems-based approach to fostering robust science in industrial-organizational psychology. Industrial and Organizational Psychology, 11, 4–42.

17.

Greve

Broder

Erdfelder

(2013). Results-blind peer reviews and editorial decisions: A missing pillar of scientific culture. European Psychologist, 18, 286–294.

18.

Hambrick

D. C.

(2007). The field of management’s devotion to theory: Too much of a good thing? Academy of Management Journal, 50, 1346–1352.

19.

Institute of Medicine, National Academy of Sciences, & National Academy of Engineering. (1995). On being a scientist: Responsible conduct in research (2nd ed.). Washington, DC: National Academies Press.

20.

Ioannidis

J. P. A.

(2005). Why most published research findings are false. PLOS Medicine, 2(8), Article e124. doi:10.1371/journal.pmed.0020124

21.

Ioannidis

J. P.

(2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Quarterly, 94, 485–514.

22.

Kepes

McDaniel

M. A.

(2013). How trustworthy is the scientific literature in I-O psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 252–268.

23.

Klein

R. A.

Ratliff

K. A.

Vianello

Adams

R. B.

Jr. Bahnik

Bernstein

M. J.

. . . Nosek

B. A.

(2014). Investigating variation in replicability. Social Psychology, 45, 142–152.

24.

Landis

R. S.

James

L. R.

Lance

C. E.

Pierce

C. A.

Rogelberg

S. G.

(2014). When is nothing something? Editorial for the null results special issue of Journal of Business and Psychology. Journal of Business and Psychology, 29, 163–167.

25.

Landis

R. S.

Rogelberg

S. G.

(2013). Our scholarly practices are derailing our progress: The importance of “nothing” in the organizational sciences. Industrial and Organizational Psychology, 6, 299–302.

26.

Lewin

(1947) Frontiers in group dynamics: Concept, method and reality in social science; equilibrium and social change. Human Relations, 1, 5–41.

27.

Locke

E. A.

(2007). The case for inductive theory building. Journal of Management, 33, 867–890.

28.

McGrath

J. E.

(1982). Dilemmatics: The study of research choices and dilemmas. In McGrath

J. E.

Martin

Kulka

R. A.

(Eds.), Judgment calls in research (pp. 69–102). Beverly Hills, CA: Sage.

29.

Meehl

P. E.

(1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103–115.

30.

Murphy

K. R.

Aguinis

(2017). HARKing: How badly can cherry picking and question trolling produce bias in published results? Journal of Business and Psychology. Advance online publication. doi:10.1007/s10869-017-9524-7

31.

National Academies of Sciences, Engineering, & Medicine. (2017). Fostering integrity in research. Washington, DC: National Academies Press.

32.

Nosek

B. A.

Bar-Anan

(2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217–243.

33.

Nosek

B. A.

Lakens

(2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137–141.

34.

Nosek

B. A.

Spies

J. R.

Motyl

(2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 76, 615–631.

35.

O’Boyle

E. H.

Jr. Banks

G. C.

Gonzalez-Mule

(2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43, 400–425.

36.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. doi:10.1126/science.aac4716

37.

Pfeffer

Fong

C. T.

(2002). The end of business schools? Less success than meets the eye. Academy of Management Learning & Education, 1, 78–95.

38.

Rosenthal

(1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.

39.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

40.

Thundiyil

T. G.

Chiaburu

D. S.

I.-S.

Banks

G. C.

Peng

A. C.

(2015). Cynical about change? A preliminary meta-analysis and future research agenda. Journal of Applied Behavioral Science, 51, 429–450.

41.

Wagenmakers

E.-J.

Wetzels

Borsboom

van der Maas

H. L.

Kievit

R. A.

(2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.

42.

Weick

(1989). Theory construction as disciplined imagination. Academy of Management Review, 14, 516–531.