Abstract
As most of you know, a number of concerns have been raised regarding the integrity of research in the social and behavioral sciences. These concerns include, but are not limited to, data fabrication, misrepresenting or misreporting of results, generating hypotheses after results are known (or HARKing), as well as other questionable practices that may result in various forms of publication bias and/or contribute to failures to replicate previously published findings.
Reactions to these concerns have been diverse. At one extreme, there are those who would argue that such problems have always existed, but in small isolated pockets. As such, they would argue that the peer-review system (while imperfect) has worked reasonably well in the past; it is currently working reasonably well; and it will likely continue to work well in the future. As such, they are reticent to embrace changes to a system that, from their vantage, appears to be working reasonably well.
At another extreme, there are those who have concluded that these problems are widespread and that the current peer-review system is fundamentally broken. They would argue that the reward structure in the social sciences incentivizes questionable research practices while not supporting or incentivizing transparency (and thus not allowing for a truly meaningful peer-review). Consequently, they are more comfortable embracing changes to the peer-review system, as they believe such changes are required for reestablishing the integrity of our science.
Personally, I find myself falling somewhere in between these two schools of thought. I certainly believe there are problems, but I also believe that most scholars are playing by the rules and trying to get it right (at least most of the time). Of course, we as a discipline strive for excellence and integrity all of the time. Thus, I do believe some (measured) changes to the current peer-review system (and the broader reward structure in the social sciences) may go a long way toward improving the overall quality of inferences we are able to draw from our research findings.
It was with this mind-set that I decided to augment the review protocols used at ORM.
Beginning in January 2016, authors submitting their work to ORM will have the option of requesting that their manuscript be reviewed using a new review protocol—Hybrid Registered Reports. To be clear, this new protocol is NOT replacing the existing protocols, but rather expands the options available to authors and simply serves as an alternative “pathway to publication.”
ORM joins with the following journals, each of which has agreed to expand review protocols to include a Hybrid Registered Reports option (if only on a trial basis): European Journal of Work and Organizational Psychology Group and Organization Management International Journal of Selection and Assessment Journal of Business and Psychology Journal of Managerial Psychology Journal of Personnel Psychology Journal of Vocational Behavior Leadership Quarterly
Caveat
Given the nature of the manuscripts we review at ORM, the Hybrid Registered Report review protocol will not be applicable to many (if not most) of the submissions. To be clear, my goal as editor continues to be publishing articles that make a clear and compelling methodological contribution to our field. Manuscripts that simply apply existing methods to test substantive hypotheses are NOT appropriate for ORM. However, manuscripts that document/demonstrate their methodological contributions (e.g., introducing a new method, providing an extension of existing methods, comparing/contrasting/critiquing existing methods) using illustrative examples have always been welcome at ORM (and will continue to be welcome at ORM). For a manuscript to be reviewed under this new protocol it must (at a minimum): Make a clear and important methodological contribution to our field Articulate a set of substantive, a priori hypotheses Use the test of those hypotheses to demonstrate/document the methodological contribution
For example, in this issue of ORM, Zyphur, Zammuto, and Zhang (2016) illustrated how multilevel latent polynomial regression may be used to improve the inferences made in the organizational sciences. They document their methodological contribution, not by relying solely on proofs or simulations, but rather by applying their techniques to test a specific set of hypotheses derived from their work involving organizational culture. Although the article by Zyphur et al. was reviewed using the traditional protocols, it represents an excellent example of the type of manuscript that would be appropriate for consideration under the Hybrid Registered Reports review protocol.
The following description of Hybrid Registered Reports was initially assembled by Steven Rogelberg and shared with the editors of the aforementioned journals for additional feedback/comments. This description summarizes the basic Hybrid Registered Reports review protocol I plan to implement at ORM.
Hybrid Registered Reports
Guidelines for Authors and Reviewers
Registered Reports are a form of empirical article in which the introduction, methods, and proposed analyses are preregistered and reviewed prior to research being conducted. This format of article seeks to neutralize a variety of questionable research practices, including submitting research based on inadequate statistical power, selective reporting of results, P-Hacking, HARKING, and publication bias (Chambers, 2013). Our hybrid Registered Reports model tweaks this format by allowing researchers to submit work that has been completed, but only the Introduction, Method, and Proposed Analyses are submitted for review. Thus the work will be judged on the merits of the research question and methodology, not the findings.
Initial submissions will be reviewed by the editor using standard screening criteria (e.g., comprehensiveness, fit, and clarity). Those that pass this review will then be sent for in-depth blind peer review (Stage 1). Following review, the article will then either be rejected, receive an R&R, or be accepted in principle for publication. Following in-principle acceptance (IPA), the authors will then produce a more traditionally formatted manuscript that includes Results and Discussion sections (Stage 2). Assuming that the Stage 2 submission is consistent with what was committed to in Stage 1, and a sensible interpretation and discussion of the findings, the manuscript will be published regardless of the results. It is critical to note that this process is designed to incent the authors to have compelling research questions, excellent methodology, and then search for “truth” in their data. It is recognized that the search for “truth” can be messy, may produce nonsignificant results, and may lead to more questions than answers.
The hybrid registered reports approach is appropriate for inductive, deductive, mixed methods, and papers involving multiple studies. In the case of the latter type, the author is encouraged to contact the editor to decide on the best approach for submission. There are multiple options as the hybrid registered reports approach is highly flexible. At times, it may make sense to include the first study with results, and then the follow-up studies with out results (this is sometimes ideal for research that starts inductive and then moves to deductive approaches). Other times it is useful to include all studies with out results. Overall, an initial conversation with the editor can clarify an approach that makes sense for the project in question—ultimately it is not useful to assume a one size fits all approach given the eclectic nature of research.
Stage 1: Initial Manuscript Submission and Review
Before sending for in-depth T include the manuscript (details below) and a brief cover letter.
The cover letter should include A statement identifying any conflicts of interests A statement identifying whether data used have been used in any other research study, and if so, how
Manuscript Preparation Guidelines—Stage 1
Initial Stage 1 submissions should include the following sections:
Introduction
A review of the relevant literature that motivates the research question and a full description of the research aims and hypotheses if applicable. The work proposed can be inductive or deductive in nature. A strong theoretical (especially for deductive work) and/or conceptual rationale (especially for inductive work) is provided.
Methods
Full description of proposed sample characteristics, including criteria for participant inclusion and exclusion, and description of procedures/practices for defining outliers or excluding data.
A detailed description of research procedures. These procedures must be consistently reported in the Stage 2 manuscript or the paper will be summarily rejected (unless approved in advance).
Proposed analysis pipeline, including a precise description of all planned analyses. Any covariates or regressors must be stated including a rationale for their inclusion. Often, proposed analyses involving covariates should be reported with and without the covariate(s) included. Where analysis decisions are contingent on the outcome of prior analyses, these contingencies should be specified. Only preplanned analyses can be reported in the main Results section of Stage 2 submissions. However, unplanned post hoc analyses will be admissible in a separate section of the Results.
Studies involving null hypothesis significance testing must include a priori statistical power analysis. Estimated effect sizes for power analysis should be justified with reference to the existing literature. For studies involving Bayesian hypothesis testing, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified.
Discussion of the sample and sampling methods to assure that the research questions can be answered in a fair and appropriate manner (e.g., no floor or ceiling effects).
Key Note
ALTHOUGH STAGE 1 SUBMISSIONS RESEMBLE THESIS AND DISSERTATION PROPOSALS IN FORM, PAPERS SUBMITTED TO ORM MUST HAVE THE TIGHTNESS AND COHERENCE OF WELL-CONSTRUCTED JOURNAL MANUSCRIPTS.
In considering Stage 1 papers, reviewers will be asked to assess The significance/importance of the research and research question(s) The logic, rationale, and plausibility of the proposed hypotheses and/or research questions The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where applicable) Measurement/psychometric information where applicable Whether the authors have used measures and procedures that are of sufficient methodological rigor for ensuring that the results obtained are able to test the stated hypotheses Whether the sample used provides a good test of the stated hypotheses (e.g., no major floor or ceiling effects, no systematic bias or distortion)
Following Stage 1 peer review, manuscripts will be rejected outright, offered the opportunity to revise, or receive in-principle acceptances (IPA) indicating that the article will be published pending accurate and complete analyses, as well as a defensible and evidence-bound interpretation of the results. Note, IPA decisions are time-bound. Time limits will be noted if an IPA is granted (e.g., typically 2-3 months). Time extensions will be considered on a case-by-case basis.
Please note that any deviation from the stated research procedures or inconsistencies between methods from Stage 1 to Stage 2, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript. Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in a final manuscript.
Stage 2: Full Manuscript Review
Authors prepare and resubmit their manuscript for full review. The cover letter should acknowledge any deviations or new analyses from Stage 1. This is a full and traditional manuscript, with the following additions: Background and Rationale
Please note that the Introduction should not be altered from the approved Stage 1 submission, and the stated hypotheses should not be amended or appended. Depending on the timeframe, new relevant literature may have appeared between Stage 1 and Stage 2. Any such literature should be covered in the Discussion.
Results and Discussion
The outcome of all registered analyses must be reported in the manuscript, except in rare instances where a registered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and editor must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases the analysis would still be mentioned in the Methods but omitted with justification from the Results.
It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between IPA and full review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be reported in a separate section of the Results titled “exploratory analyses” for deductive research or “additional analyses” for inductive work. Where the interpretation depends on inferential statistical analysis, authors should be careful not to base their conclusions entirely on the outcome of statistically significant exploratory tests.
Exact p values and effect sizes must be reported for all null hypothesis significance tests
Misc
Any other pertinent information; it is better to provide too much information than too little information
The resubmission will ideally be considered by the same reviewers as in the registration stage. In considering papers at Stage 2, reviewers will be asked to decide Whether the Introduction and methods are the same as the approved Stage 1 submission (analysis plan is omitted) Whether results are reported clearly, appropriately and consistently with what was registered Whether any unregistered exploratory or additional statistical analyses are justified, methodologically sound, and informative Whether the authors’ conclusions are justified given the data (e.g., absent meaningful floor and/or ceiling effects) Whether the discussion section effectively contextualizes the results theoretically, conceptually and practically
Basically, Stage 2 reviews are consistent with current practices for reviewing results and discussion sections.
Crucially, reviewers will be informed that editorial decisions will not be based on the support or lack of support of the hypotheses.
There is very little risk for the author in Stage 2 reviews. The author may be asked to revise and resubmit, however, based on reviewer/editor feedback.