Conducting Registered Report Research: A Conversation With Matthew McBee and Scott Peters

Kettler

Registered report research is a new approach for conducting and publishing research. How would you describe the origins of the registered report process?

McBee

The registered reports format is a new model of academic publishing that is designed to alter the incentives presented to researchers, peer reviewers, and editors to improve the quality, rigor, and trustworthiness of research findings.

Peters

I think the idea of improvement is a really big one. When the analyses and entire research plan are more or less fixed in stone a priori, then the reader can have more confidence in the results. It might not sound like that big of a thing, but when you consider the replication crisis that’s happening in most social science fields right now, it’s huge.

McBee

This model is one of a set of responses to the crisis of confidence that has swept through the social and biomedical sciences as a result of widespread findings of low replicability and high rates of false positives in the literature. It is thought that a major contributor to this problem is that the incentives confronting researchers and editors facilitate the proliferation of bad science. The registered reports format alters those incentives, creating a closer alignment between what is good for individual researchers and what is good for the quality of research. Registered reports also have the advantage of being a more humane and efficient model of scientific dissemination from the researcher’s point of view. Researchers win, and the quality of the literature wins. If anyone loses, it would only be readers who are accustomed to turning to academic journal articles for entertainment. Widespread adoption of the registered reports format likely means that a much larger proportion of published papers will consist of null findings that, perhaps, are not extremely exciting to read. I’m not sure that such people exist, and in any case, I do not consider this interest to be legitimate.

I will describe some of the relevant facts as I see them. First, academic journals exist to advance the profit motives of academic publishing companies.

Peters

I agree that this is a huge problem, but I don’t see it as really that related to registered reports. Sure, registered reports mean less crap research being published, which might mean fewer journals or articles, but only if the promotion and tenure process is revised as well. The existence of registered reports isn’t likely to change the for-profit motives of Wiley or Sage. Combating the negative side effects of for-profit publishing platforms will take different actions. Registered reports could prevent the journals from focusing so much on sexy findings, because with registered reports, they will not know which articles will produce sexy findings, but without registered reports being mandatory, the journal incentive is likely to remain.

McBee

Most faculty members are required to publish research as a condition of employment, promotion, and professional standing. However, there is no systemic process for adequately funding such work, nor are all faculty intrinsically inclined to engage in it. As a result, there exists a glut of academic production, much of which is low quality. The supply of research exceeds the demand.

Peters

Again, I see the cause of bad research (mainly publish or perish tenure and promotion requirements at universities coupled with for-profit motives from journals) as a different issue. I do not see how registered reports really fix this. Instead, universities need to change how points or credit is awarded with regard to research work or productivity. Simply having registered reports as an option for publishing in some journals does not change what my university requires for tenure.

McBee

In the past, publication was a costly process requiring specialized and expensive equipment. Nowadays, dissemination online is free. Therefore, journals have become almost exclusively a branding and marketing mechanism for papers. Additionally, there exists a hierarchy of journals within each field. Publishing in high-status journals yields disproportionate rewards of circulation, prestige, and press interest for researchers. Therefore, competition for space in high-ranked journals is intense. The impact factor is the primary means of establishing the hierarchy of journals. It is a measure of the citation rate of papers published by the journal. Journals compete with one another for status, for this increases the profits they generate. Each journal would like to be perceived as the premium brand within its space. Increasing citations, and therefore the impact factor, is the most effective means of moving up the ranking.

Peters

I think the issue this creates is an incentive for authors to publish novel or sexy findings so that the journals can get more readers and more media attention. Registered reports help combat this by removing the results or findings from the equation. No longer can authors or journals publish only what’s sexy or can journals give preference to sexy findings.

McBee

Because of the positive feedback generated by high status, the stakes for inter-journal competition are high. Prestige journals attract many more highly appealing submissions, making it easier for editors to select the subset of papers whose publication will maintain or increase the journal’s status. But here is where it gets messy. Studies describing affirmative findings—discoveries, counterintuitive phenomena, and strong effects—are perceived as more interesting and exciting than papers presenting null findings. The same could be said of novel studies as opposed to replications of previous work. Exciting, interesting, and captivating findings generate more citations and attention. Publishing novel discoveries serves the journal’s goal of moving up the hierarchy.

Peters

This is very relevant to registered reports. Readers, editors, the media, promotion committees, all like new and sexy findings. They do not tend to like either replications or null findings. They see null findings as essentially having found nothing and having been a waste of time. Registered reports fix that problem by having articles reviewed before there are any findings. That is a huge shift. It is also what would require the most change to the current review system. So many reviewers look at the findings as a major factor on whether or not to publish a study. That’s 100% nonscientific. Every one of my graduate students, when first learning to read and approach articles, looks at the introduction and then the results. They do not think there is anything to be learned from the methods.

McBee

Since the supply of submissions so strongly outstrips the journal’s artificial space constraints, especially for high-ranked outlets, journals can impose severe selection criteria on submissions. They can essentially demand perfection in the form of a coherent narrative and completely internally consistent, novel, and exciting results. The higher the standard demanded, the stronger the journal’s branding effect, and the more valuable publication in that outlet becomes for authors.

Peters

Again, very true, but not related to registered reports unless they were universally required. If they remain just an option for some researchers, then the journal incentive to still only publish sexy findings remains, and registered reports will be less attractive. It will be interesting to see why journals choose to accept registered reports as an option. In my experience, it’s the big shot journals that are not as interested because they are happy with the status quo. They receive lots of submissions, reject most, have a high impact factor. Why change?

McBee

Every study has a chance of producing a false positive outcome or an overestimated effect size. Making publication conditional on the criterion of exciting discoveries guarantees that such false positives and inflated effect size estimates are overrepresented in the literature, even if the error rates within individual papers were strongly controlled (which they are not). Similarly, researchers respond to journal selection criteria by dutifully generating perfect studies. As socially consequential discoveries are rare and perfect results are nigh-impossible to honestly obtain, techniques that have been dubbed questionable research practices (QRP) are often employed to make study results appear better than they would otherwise be. These include techniques such as Hypothesizing After Results are Known (HARKing), p-hacking to statistical significance, dropping aims or outcomes that are unsuccessful, and other associated techniques.

Peters

But as long as registered reports are optional, traditional articles will still be published that have been fortunate enough to produce sexy, even if HARKed findings. These will then have the potential to crowd out registered reports. That’s really the next step for this field. How does the field of academic publishing prevent this short of solely accepting registered reports?

Kettler

Scott, that may be the big question looming in the corner, but the problem seems to be deeply systemic.

McBee

Because overall publication acceptance rates are low, researchers may be better served by running several small and relatively weak studies rather than a single strong study. If the strong study is rejected (perhaps due to a null finding), they will have no evidence of productivity and risk losing their careers. Further, p-hacking techniques work more effectively on small samples, and the resulting unstable estimated effects more often veer into the truly exciting “moderate to large” category. Further, some reviewers operate under the misperception that a statistically significant and large effect size generated from a small sample somehow constitutes stronger evidence than the same generated from a large study. This is completely, devastatingly wrong, by the way.

Kettler

Matt, explain why you see this as devastatingly wrong.

McBee

A large proportion of the scientific literature produced under this publication model consists largely of false positives and dramatically overstated claims. Even research synthesis techniques such as meta-analysis cannot recover real effects, as they suffer from the well-known garbage-in/garbage-out problem. It is worth reflecting on why this publication model continues to exist. The reason is that it serves the ends of publishers by creating a product for them to sell—a product consisting of the donated contributions of researchers and peer reviewers, no less! And it serves the needs of researchers by providing a means of dissemination and a system for assigning social and professional status that appears to be based on merit. The costs of this system—the untrustworthy literature, the phantom findings, the discoveries that turn out to be carefully curated noise—were not visible until researchers began investing in replication studies, as has taken place in psychology, biomedical science, and a few other fields, but sadly not ours. Then it was discovered, to utter shock and dismay, that a large proportion of the claims presented in the literature in those fields was not replicable. This has been the case in every field where it has been attempted, and though gifted education has not yet engaged in replication efforts of the necessary magnitude, there is little reason to think that our field stands apart. And the lowest bar that any scientific or educational finding must be able to clear before it can lay claim to truth-value is the possibility of being repeatedly observed.

Peters

I think this is a big argument for those faculty members who have tenure to discontinue publishing in for-profit, pay-walled journals. Editors and associations should also start investigating how they can move current journals to open access with as low of publishing costs as possible.

McBee

In short, the traditional publication model has created a tragedy of the commons. The journals and researchers have collaboratively created a system in which all parties privately benefit from the degradation of a public good—the body of the scientific literature. The set of events circa 2011-2018 collectively known as the replication crisis are analogous to the Cuyahoga River fire of 1969. The noxious water was easy to ignore until the river burst into flame. That event culminated in the Clean Water Act of 1972. And the replication crisis is culminating in a set of reforms known as Open Science, into which the registered reports format is situated.

Peters

All of this is true and important background regarding the larger cultural shift and its causes going on right now, but a lot of it is not directly related to registered reports. Research that has been conducted under the framework of a registered report is more likely to be true, but that only makes that individual study better. It doesn’t necessarily change researcher or journal incentives.

Kettler

How does registered report research differ from what has typically been done in educational research?

Peters

The thing that will stand out the most to people familiar with the current system of scholarly research and publishing is that with registered reports, peer review happens before the study is conducted—in most cases before any data are even collected. The authors draft their introduction, review of the literature, and proposed methods of analysis and then submit this for peer review to a journal. In this way, it’s a little similar to what researchers submit for IRB [institutional review board] or a student for a dissertation proposal. The major difference is that a registered report is meant to be a full recipe such that another researcher could replicate the study completely based on what is provided. That recipe is what is reviewed on the basis of whether or not, if conducted, it would actually test the hypothesis or answer the question.

McBee

Right. And I think it’s important to note that the study plan can, and in many cases must, contain branching “if-then” logic outlining a decision tree of contingency plans. For example, imagine that a researcher is planning to fit a complex latent variable model to a dataset. What if the planned model does not converge or the data indicates that certain assumptions were violated? The researcher will have to turn to Plan B—perhaps fitting a simpler model to the data, using a different estimator, or transforming certain variables. These may be reasonable responses, but staying in the confirmatory context requires that they be pre-planned. The point is to discourage or prevent researchers from being able to make data-informed choices through an unconstrained multiplicity of possible decisions. So, the data analysis plan is not really the analysis plan, but rather a set of plans.

Kettler

How is the registered report process used in fields outside of education?

McBee

The registered report format has been adopted by 125 journals, with the number increasing weekly. Gifted Child Quarterly and the Journal of Advanced Academics seem to be the first two education journals to experiment with them. It is great to work in a field that has positioned itself at the cutting edge of the methodological reform movement.

Peters

Although this is true that many journals have now, at least in principle, adopted registered reports as a form of submission that is eligible for publication, many have done so very recently. This means (a) we do not know to what degree they actually accept and publish registered reports, (b) we do not know how good their review system for registered reports actually is, and (c) we do not actually know how this world is changing incentives or structures like promotion and tenure procedures. It’s just too early yet. Still, there is a major change happening in fields like medicine, genetics, and psychology.

McBee

Arguably, the highest-profile use of the format to date has been in the Registered Replication Reports series published by Perspectives on Psychological Science (PoPS) and now also offered by the new journal Advances in Methods and Practices in Psychological Science (AMPPS). Registered replication reports present the results of preregistered multi-lab collaborative replication studies of topics or findings deemed to be of widespread interest. In these studies, the project leads work with the original authors to create a study protocol and analysis plan. The project is submitted and peer reviewed. Upon receipt of the Stage 1 in-principle acceptance, the project is pre-registered, and participating labs are recruited. Each lab implements the study with the same materials, stimuli, scripts, and so on. After data collection is complete, each lab submits its data to the project lead. The project lead analyzes the data according to the approved analysis plan and combines all the findings via meta-analysis. The results and discussion sections are written, and the completed paper is submitted for Stage 2 peer review. The final paper, incorporating reviewer feedback, is published, along with a commentary from the original authors.

There have been eight such studies published thus far, investigating the verbal overshadowing effect (Registered Replication Reports 1 and 2) ¹ , the effect of grammar on perceived intentionality (Registered Replication Report 3), ego depletion (Registered Replication Report 4), the effect of commitment on forgiveness (Registered Replication Report 5), the facial feedback hypothesis (Registered Replication Report 6), the intuitive-cooperation effect (Registered Replication Report 7), and the professor priming effect (Registered Replication Report 8). Of these, only the verbal overshadowing effect emerged with any support. The effect sizes for the other effects were zero to a high approximation.

Peters

I want to emphasize that registered reports and registered replication reports are different things. A registered replication report is a multigroup collaboration on a registered report, but it is also a model that is less likely to take off in education than in psychology because there aren’t really labs in the same way there are in psychology. However, there’s nothing stopping multiple education graduate programs from grouping together to do something similar. Still, I want readers to know that a registered report is something they can do for a dissertation or even a single study. They don’t need 10 labs to collaborate. Most people that do quantitative confirmatory/inferential research can and should move all of their future studies to registered reports. There aren’t many downsides for the individual researcher except that he or she will no longer be able to p-hack your way to a significant finding.

McBee

It is difficult to overstate the impact of these studies in psychology. For example, ego depletion was a dominant paradigm for understanding self-control, having amassed hundreds of publications seeming to demonstrate it in various ways. This was a consequence of publishing only the hits, and either suppressing the misses into the file drawer or using questionable research practices to transform them into seeming hits. But a single, very rigorous study has obliterated the consensus and made a space for new theoretical work on self-control, work that will one day be able to withstand the rigors of a multi-lab preregistered study. I have no doubt that future generations will look to Registered Replication Report 4 as a pivotal moment in the history of the field.

I want that for our field of gifted education and advanced academic research. I suspect that our replication success rate will be better than social psychology’s for various reasons, but I also suspect that some of the things we think we know in our field are not as true as we might hope for them to be.

Kettler

In what ways do registered reports address specific weaknesses in existing educational research?

Peters

Here it is important to distinguish the benefits from registered reports over pre-registration. In pre-registration, researchers prepare and register their study before they conduct any analyses. Doing so (and sticking to that pre-registered set of analyses for confirmatory research) helps assure the reader that the study results were not gained via any questionable research practices such as p-hacking or Hypothesizing After Results were Known (HARKing). In both of those cases, the authors are not testing a specific question, but rather are just mining the data for a significant result. Then they change their research questions or hypotheses to make it seem like that was the a priori question they were asking all along. By pre-registering analyses a priori, readers can be more confident in the validity of a study.

Registered reports go a step further. Not only do they tend to be more detailed than pre-registrations (the latter often including methods with much less introduction or literature review), but also, they are actually submitted for peer review before being conducted. This extra step has two major benefits—to the author and to the science. First, the author gets feedback and has any incorrect or imperfect methods caught by reviewers before any analyses have been conducted or data collected. I want to note that it is possible to submit a registered report on pre-existing data, but here I am referring to the most common situation where data have yet to be collected.

All authors have had the experience of having to go back and re-do analyses based on reviewer comments. In a registered report, those types of re-analysis do not happen because the methods are agreed upon before the actual analyses are conducted. This early peer review also prevents any bad data collection methods that might not be correctable after the fact. Imagine an instrument is used incorrectly and this is not caught until traditional peer review. There’s no going back and fixing it! Registered reports help prevent this.

The second major benefit is that the study is reviewed based on the scientific merits alone—not based on the desirability of the findings. An article showing the ineffectiveness of a gifted program or the lack of racial/ethnic proportionality might meet with hostile reviewers who do not want to see such ideas or examples in print. There could also be general hostility toward the topic (vis-à-vis gifted education in nongifted education journals, for example). But the social desirability of the outcome is absolutely irrelevant to study quality. Registered reports move the judgment of a study to the quality of the science and away from the sexiness, novelty, or desirability of the findings.

McBee

The legacy model gives readers no verifiable means of distinguishing between exploratory and confirmatory findings. Knowing this is critical because they offer very different levels of evidence. The virtue of preregistration in general and registered reports specifically is that readers can finally know this and calibrate their interpretation of the results.

Kettler

What about specific weaknesses in advanced academic and gifted education research?

Peters

The desirability of the results is a serious issue in gifted education research. I know of many papers that were rejected from publication or received critical reviews because the findings went against a reviewer’s personal beliefs, challenged the conceptual definition under which the reviewer was trained, or simply portrayed the field in a negative light. But again, the actual results of a study should have zero bearing on the quality of the science. It’s all about the methods. By moving peer review before the results are known, this can be addressed.

McBee

Absolutely. I think desirability bias exists in every field, but it can be a particular problem in an advocacy field like ours. Increasing the credibility of our research efforts—especially outside our field—could end up helping us more effectively facilitate our advocacy goals. It will be more difficult for outsiders to discount our findings as the product of confirmation bias when publication decisions are made on the basis of rigor alone, and not influenced by the findings.

Kettler

Let’s talk more specifically, how would you describe the steps in the process of registered report research?

McBee

It is quite like that used for thesis or dissertation work (see Figure 1). The researcher develops an idea, reviews the relevant literature, and designs a study, including the proposed hypotheses, instruments, sample size, exclusion criteria, and statistical analysis plan. The researcher begins drafting a paper explaining these details, consisting of the introduction, method, and proposed analysis. The incomplete manuscript is submitted to the journal at this point.

OPEN IN VIEWER

Figure 1.

Registered reports process.

Peters

It is also very similar to the human subjects review board (IRB) process, but I think Matt’s comparison to a dissertation proposal is quite apt. The reason students do this with their dissertation, in theory, is so the faculty can provide feedback, make sure there are not any mistakes in the method, and generally give the student confidence that if he or she conducts the study as designed, the committee members are not going to change their mind at the end. Advocates for registered reports want to inject that same confidence into all of educational research.

McBee

In the registered reports process, the editors review and triage the manuscript. If it meets the appropriate standards, the proposal is sent out for peer review. The reviewers carefully consider the strengths and weaknesses of the proposal and provide feedback and a recommended decision to the editor, who can decide to reject the proposal out of hand, reject but recommend resubmission with revisions, or issue an in-principle acceptance (with or without revisions). An in-principle acceptance means that the journal has committed to publishing the study, regardless of how the results may come out, so long as the researchers faithfully execute the agreed-upon plan. After receiving the in-principle acceptance at Stage 1, the researcher preregisters the study and begins data collection.

Peters

This is where the greatest barrier for scalability comes in. How closely are the reviewers expected to judge the degree to which the authors didn’t stray 1% from their proposal in Phase 1? What if they do? How much post hoc exploratory research is okay? There are just some further details that will need to be worked out.

It is also important to note that registered reports can be done on pre-existing or extant data. This might include something like National Assessment of Educational Progress (NAEP), Early Childhood Longitudinal Study (ECLS), or even the Office of Civil Rights data collection. The main requirement with such approaches is that the author needs to not peek before-hand. The author needs to write up the entire registered report and get it accepted in principle before even opening the data file or requesting it. Otherwise, he or she could be intentionally or unintentionally HARKing the study.

McBee

After completing the data analysis according to the protocol, the results and discussion sections of the paper are written. The complete manuscript is resubmitted to the journal for Stage 2 peer review. At this point, the focus is on ensuring that the protocol was faithfully executed, with any deviations documented and justified, as well as on the reviewing the discussion and implications section. The Stage 2 review is perfunctory compared to the Stage 1 review process. Once the reviewers and editor are satisfied, the paper receives final acceptance and enters the production queue.

Peters

Again, note that even in Stage 2, there is no real evaluation or judgment of the findings. All that is happening in this review is checking to make sure the Stage 1 plan was followed. Even if a journal does not adopt registered reports, all journal editors should push back when a reviewer focuses too much on a paper’s results. Results do not matter with regard to study quality. I suggest that even if a journal does not have formally accepted registered reports, editors should modify their review process to cut back on how much weight reviewers give to results or findings.

Kettler

How would you describe the potential of registered report research to improve the research in advanced academics or gifted education?

McBee

Registered reports offer the same benefits to our field as they do to others—which are realigning the incentive structure and imposing results-blind review at Stage 1, when the principle acceptance or rejection decision is made.

Peters

Matt, I disagree with the first part of this. I do not think registered reports fix the existing terrible incentive structure, especially if registered reports are optional (which they will be). Instead, registered reports do something much more powerful—they can increase the probability of study findings being true and replicable. They can also increase reader trust in scientific research. That’s huge! They also have the benefit of saving time. No longer do authors need to conduct and write up an entire study before knowing if it will be published. No more do authors need to shop a paper around for an outlet. No more do three or four sets of reviewers from multiple journals need to review the same paper. Sure, there is more work with registered reports because there are two phases of review, but on the whole, it should result in less time and work for the author and the reviewers.

Registered reports do offer a cleaner and more reliable pathway by nature of the flow in Figure 1. However, people who do not want to learn or do this new pathway won’t change. Plus, registered reports may not change any incentive structures for most qualitative researchers or those who do case study, or small sample studies. Registered reports are mostly focused on the crowd that wants to do externally valid research that generalizes to broader groups. I tend to believe that should be the majority of educational research, but right now it is not. It is also possible for registered reports to be done for qualitative research, but that’s another conversation.

McBee

Good point, Scott. At Stage 1, the reviewers are presented the key pieces of information that enable them to understand how the proposed research is positioned to contribute to the literature, the hypotheses to be investigated, and all aspects related to the study design and analysis plan. Crucially, the rigor of the study depends on these factors and only these factors. Reviewers and editors cannot select or accept on affirmative findings or the sexiness, excitingness, or counterintuitive quality of the results. Registered report proposals are, therefore, evaluated on quality and topical importance. Studies must be designed to produce informational value regardless of the result. This implies, for example, that statistical power and precision must be substantial, for a null result from a poorly powered study is uninformative. In other words, for a null result to be interesting, it must indicate with high probability that the hypothesized relationship is either zero or so small as to be effectively zero.

Since the publication decision is independent of the result, the incentive to engage in questionable research practices like HARKing or p-hacking is tremendously reduced. And since the protocol is preregistered, and adherence to the protocol will be scrutinized at Stage 2, it becomes very difficult for researchers to get away with any shenanigans, anyway, because someone is watching. Several someones, in fact. The Stage 2 review further incentivizes other transparency and credibility-increasing actions, such as open data, open materials, and open analysis code. The easiest way to show faithful adherence to the protocol is to share the data, code, and materials such that external parties can reproduce the analysis. Thus, there is no ambiguity about the process leading to the conclusion.

The other tremendous advantage of registered reports is simply that it is a much more efficient and humane model of dissemination than the legacy model. How much time is wasted by researchers shopping their manuscripts down the prestige hierarchy of journals? How much time is wasted by different sets of reviewers examining the manuscript as it wends its way down to the outlet that will publish it? And worst of all is the tragedy of learning that your research contains some fatal flaw that invalidates its findings only after the study is complete and it is too late to repair. I am sure that many of us have been on both sides of this situation; I certainly have, and it is awful for everyone involved. The proper time to receive feedback on your idea is before you have invested in implementing it.

Peters

These are such major benefits; it still blows my mind talk about them again. I think so many of us are simply accustomed to the existing model that we don’t even think about how silly it is. We waste so much time. I have a manuscript right now that is on its fourth journal. We’ve been shopping it around for a year! What a waste of reviewer and editor time.

McBee

Absolutely. Investing in a research idea is a risky proposition when there’s no guarantee that the final result will be published. Graduate students, postdoctoral researchers, and junior faculty only have so much time to generate evidence of productivity. One of the little-discussed aspects of the replication crisis in psychology is the tremendous but largely invisible career wreckage, in which people spent futile years trying to build on ideas such as ego depletion or social priming. Worse is that such conditions create a selection mechanism that favors the least ethical researchers, the ones most willing to engage in questionable research practices in order to turn those frowns upside down. We want the most principled people to build scientific careers, not the ones most willing to cut corners. How many students failed to find jobs because they had nothing to show for years of work that came to nothing? How much more quickly could progress in these fields have taken place if the literature contained an accurate record of the rigorous studies that had been performed and not just the hits?

Kettler

Scott mentioned this earlier, and I want to return to it. Can qualitative or mixed-methods researchers also use the registered reports protocol?

Peters

Absolutely! So far, we have talked almost exclusively about quantitative, confirmatory research where there is an a priori, theory-based hypothesis being tested. Registered reports can also include qualitative research and exploratory research, though they are less important here than in confirmatory and qualitative research. The overall ideal of a registered report is not new—that a reader should be able to reproduce the study based on what is in the study’s methods section. In qualitative research, this is often called trustworthiness. The reader should be able to trust the results based on a thick description of the methods, participants, and protocols.

The problem is we have gotten away from this level of detail in the typical methods section—often due to length restrictions in print journals and because so many people do not like to read methods sections. The issue is really about honesty and transparency. The clearer I am to my readers regarding how I arrived at my results (whether qualitative or quantitative), the greater the trust the reader can have in the findings. The reader should never have to take my word for it.

McBee

I have a slightly different perspective. Registered reports are a method for enforcing a distinction between exploratory and confirmatory research. I’m not sure that this is needed by or applicable to qualitative work, which I have always thought of as exploratory in nature; I view qualitative research as generally being more suited to hypothesis generation rather than hypothesis testing. My preferred qualitative methodology, grounded theory, is about as exploratory as it is possible to be and has generated many valuable and interesting insights into the lives and experiences of bright students. I cannot imagine how one would even approach a registered report for a grounded theory qualitative study. Only the barest outline of the intended study is knowable prior to encountering the data in such a circumstance.

Some journals (e.g., Cortex) are now offering a submission format known as an Exploratory Report, ² which extends some of the features of registered reports to the exploratory studies, including findings-free peer review. Rather than trying to shoehorn qualitative research into the confirmatory registered report format, my preference would be to consider adopting exploratory reports for qualitative studies.

To be frank, the open science practice that would most improve the transparency and trustworthiness of qualitative studies would be open data—Posting the transcripts, field notes, and so on, in an online repository would allow readers to determine the extent to which the quotes, themes, and narrative of the paper are truly representative of the data.

Kettler

Let’s take this a little farther. Give me some examples of what conducting registered report research looks like.

Peters

Jennifer Jolly and I conducted a study looking at the relationship between various levels of teacher training in gifted education and self-reported instructional practices. It is not a registered report, but we did pre-register it. In other words, we time-stamped the methods before collecting data. We work in gifted education and teach in the very programs we were evaluating, so we had a huge incentive to show positive results. By pre-registering the study, readers do not have to worry about that because we said what we were going to do before-hand. We couldn’t just p-hack our way to a significant result. A registered report would have been even better, but at the time, the journal we were shooting for did not offer them. See here for the entire pre-registration form as well as the article: https://osf.io/9b7av/

Karen Rambo-Hernandez, Matt Makel, Michael Matthews, Jonathan Plucker, and I are in the middle of a registered report on the effect of various alternative norming criteria on racial/ethnic underrepresentation in gifted education. It is part of a special issue on registered reports in AERA Open. The Phase 1 proposal has received an in-principle acceptance, so we are in the middle of conducting the research now. People can read the actual pre-registration and accepted proposal here: https://osf.io/kazy9/. Note that part of the reason I can share this without fear of getting scooped by someone else is that the study has already been accepted. Thus, I can start to share things right away, even if the full paper is not in print.

McBee

I’d like to refer readers to three examples of study formats that could fit within the registered reports format used in our field. First is the multilab registered replication report model. I want to cite the Professor Priming Registered Replication Report (O’Donnell et al., 2018) as an example. This format could be used in our field if a sufficient number of stakeholders agreed on one or more foundational studies in our field that we would like to rigorously re-evaluate. In this case, the registered reports format helps to overwhelm the usual bias against publishing replications.

Second is a fascinating study titled “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results” (Silberzahn et al., 2018). In this study, many research teams attempted to answer the question of whether dark-skinned soccer players receive more penalties using a complex observational dataset. A huge range of analytic models were employed, and while most of the approaches yielded similar estimates, some produced quite divergent findings. As our field’s research becomes increasingly reliant on complex statistical models for inference, the potential impact of analytic decisions is unknown. This study indicates that researchers don’t necessarily have to preregister just one analysis plan; they could actually plan to do several versions of the analysis to understand how sensitive the conclusions are to arbitrary analytic decisions.

Third, and most exciting to me, is the preregistered adversarial collaboration. A great example is Matzke et al.’s (2015) study of the effect of horizontal eye movements on free recall. In this format, researchers on opposite sides of a theoretical dispute jointly design a study to (hopefully) settle the question. The editor and reviewers of the registered report can serve as neutral third parties to resolve any disputes that may arise in the course of the study. It becomes very difficult for the losing side of such a debate to mount a convincing argument for why the study was flawed or unfair—in other words, to engage in the usual sort of post hoc rationalization that stymies scientific progress—after the results are revealed. Given that our field is characterized by many such disputes, I am optimistic that the preregistered adversarial collaboration format implemented as a registered report could finally allow for our field to make progress on some of its most basic questions.

Kettler

What is the role of reviewers in registered report research? How is it different than traditional peer-review processes?

McBee

I have reviewed one Stage 1 registered report proposal to date, and I found the experience to be very positive, but quite different from the typical review. It feels more like a collaboration than evaluation, though it is evaluative, of course. The reviewers are forced to develop an exceptionally clear notion of exactly what the researchers plan to do and how they plan to do it, and to think deeply about possible failure points. Rather than simply pointing out flaws and leaving their resolution up to the authors, it becomes incumbent on the reviewers to propose workable solutions to address those flaws. I found the review process to be somewhat more fraught than I am used to experiencing, because in some months you will discover together with the research team whether this collaboratively developed research plan was, in fact, workable. Reviewers must think about maximizing evidentiary value under many possible result scenarios. Having gone through the review experience has left no doubt in my mind regarding the superiority of this model. The research team expressed the same in their revision letter. It is difficult to return to the legacy model after experiencing the benefits of registered reports.

Peters

I think being a registered reports reviewer takes more expertise. As Matt said, you are almost like a collaborator. Plus, if you are not a real expert on the methods, how can you accept or reject an article solely based on the methods? I think this is why so many reviews focus on the paper’s results—because the reviewer really was not qualified to review the methods.

Reviewers become a little more like co-authors in registered reports. Some have even suggested that reviewers get some kind of credit for their work. The idea is that reviewers help to catch mistakes, oversights, or provide suggestions for better analyses before the author has done any analysis or even data collections. This way, mistakes are avoided saving time and making for better and more efficient science in the end. This also prevents the uncomfortable conflict that can occur when reviewers want one kind of analysis and the author thinks he or she is done. All of that is worked out before-hand. I can say from experience that I have gotten substantial, helpful feedback in all of my pre-registrations or registered reports that have made the study better. This has speeded time to final publication and involved fewer instances of rejection and resubmission at other journals.

McBee

That’s true, Scott. Reviewers take on a more collaborative, involved role in the research project. Of course, there is an evaluative element as well. For example, is this study worth doing? However, the focus is on how to make it stronger as opposed to deciding whether it should be disseminated. I have only reviewed on a registered report proposal thus far, but I found the experience to be quite a bit more enjoyable and involving than the usual review experience.

Kettler

What do you think are the hardest skills to learn to begin successfully conducing registered report research?

McBee

Registered reports enforce a strict separation between confirmatory and exploratory research. The reality is that very few of us have ever run a completely confirmatory study before. I was trained and mentored in graduate school and beyond to run the type of hybrid exploratory/confirmatory studies that have been the problematic de facto standard in psychology and education for decades. Successfully running a confirmatory study requires great skill in anticipating how a study could potentially go awry and building in mechanisms to prevent it. This is a skillset that few of us have. We weren’t trained to do it for the most part, and since it has not been necessary, we have not yet absorbed the necessary lessons from the school of hard knocks.

For a truly confirmatory study, it should be possible to write the data analysis code before the data collection begins. Even the auxiliary decisions that we commonly leverage on the data need to be decided in advance. A mad-libs style results section template can be written—with only the specific values filled in when the analysis is complete. I know that these things are possible because I participated in the Professor Priming Registered Replication Report. The project leads sent out the data analysis script and the results template for review by the project participants before data collection was complete. Once the data were collected, all that was necessary was for someone to click run.

My point is simply that this is a model of research that is new to us and involves developing a set of skills and wisdom that will only be gained with effort and some inevitable failures along the way. Over time, training programs will be updated so students won’t have to learn to do confirmatory research the hard way as we must do.

Peters

I do not know that there are a lot of new skills per say, aside from the skills necessary to write the proposed methods for a study in the first place. I do think the exploratory/confirmatory distinction is something most people need to learn. The main skill that is new with a registered report is just writing 100% of the actions and methods before-hand. This is hard to do! How many of us know every code or analysis step we will take? In a recent registered report, we forgot to say what we would do for students who took a certain test twice (because they were held back or moved schools mid-year). This wasn’t a big deal, but any choice we made wasn’t in the registered report. So, what were we to do? Submitting registered reports takes a lot of planning that most researchers simply do not do until they’re in the thick of data analysis. But that post hoc process has a lot of negative side effects.

McBee

I’ve already spoken about how inexperienced and unskilled I suspect that we all are at coming up with comprehensive and workable plans that will withstand contact with the vagaries of real life and real data. The nature of working with schools makes this much more difficult than trying to do a confirmatory lab study with undergrads. But maybe the deeper challenge will be rethinking and reframing the definition of success. In the legacy model, success involves getting a paper accepted for publication in a quality journal. In the registered report model, success involves designing the most informative study that your resources allow you to execute—and then executing it according to the plan.

Kettler

Describe some specific ways that registered report research could positively impact gifted education and advanced academics.

McBee

One disadvantage of working in a field like ours, which is one-part dispassionate research and one-part advocacy, is that outsiders discount the credibility of our work, assuming that it always suffers from an inherent conflict of interest. I don’t agree with this, but it does not have to be true in order to affect our standing in the larger educational community. We can see this so clearly in the way that the press and others have embraced and celebrated gifted education research done by economists over what we do within our field. Registered reports offer us a way to increase the credibility of our research both within, but especially outside our field.

Peters

I think that concern is often valid—that too many gifted education researchers are really engaging in a process of seeking proof for their own existence—that some gifted education intervention or training program actually works. That is why we need registered reports so badly—to remove that potential or perceived conflict of interest and increase credibility. Science can be used for advocacy purposes, but they are not at all the same thing. If people enter into research seeking proof for something they already believe, they are hurting the field.

Kettler

Taking that one step farther, in what ways could the use of registered reports change the way research methods are taught in educational programs?

Peters

Everyone who reads research or is a researcher herself needs to stop looking for novelty in research studies. A good rule of thumb, that I should attribute to Matt McBee, is that the more sexy the findings, the more likely they are to be nonreplicable/spurious/false. I have had many graduate students tell me they just read the abstract and the results, even though the quality of a study’s findings is completely conditional on methodological quality. Registered reports also point to the importance of actual study design—not just analysis methods. Hopefully this will get more PhD programs to focus on methodological design and not just complex statistical methods. Everyone takes classes on analytic methods like regression or multivariate statistics. How many of us took courses on causal inference?

McBee

The first thing I suggest is that everyone needs to learn about the replication crisis/credibility revolution. This will entail substantial reading outside of our field, though Scott and I (along with Matt Makel and Michael Matthews) have a piece on open science in gifted education in press at Gifted Child Quarterly that reviews some of this literature and might (he suggests, humbly) provide a place to begin. Tim van der Zee and Justin Reich published a paper titled “Open Educational Science” in AERA Open that is excellent and highly recommended. To understand why registered reports are needed, it is helpful to understand the context of how we got here. I also recommend spending some time browsing the website curatescience.org, which curates replication studies in psychology, to see just how many ideas have failed to replicate. The problem is very real and worthy of deep consideration. It is nothing short of an emergency.

Peters

The other thing registered reports might change is how much time in training programs is spent on proposal writing and critique. Rather than thinking so much about how to apply results from a study, we should spend time parsing the nuance of the study design and how it influences the utility, validity, and reproducibility of the findings. I could imagine a great research methods course assignment where students must design a replication study proposal for their favorite gifted education study from the literature.

McBee

I still go back to the replication crisis. The question is what to do about it. Our training in methods and statistics will need to change in order to teach students how to run actual confirmatory studies. But the knowledge just isn’t there yet. In order words, the faculty who would presumably be teaching such content themselves don’t know how to run completely confirmatory studies because they’ve never done it. I certainly count myself in that group. So, there will be a period of building the ship as we are flying it. But this is a healthy and necessary transition period. As lessons are learned, they will be incorporated into our training programs. The next generation of scholars will be much better researchers than we are—if we do our jobs. We owe that to the future.

Kettler

How do you see the immediate and long-term future of registered report research in advanced academics and gifted education scholarship?

Peters

The vast majority of research should take the form of registered reports. Period. There really are few downsides that are unique to registered reports. I think in the intermediate or long term, as they become more common, the mechanism for actually checking that the authors followed their registered analyses to a sufficient degree will become more important. It’s not always clear when a significant deviation has occurred. There will also be a big need to re-train peer reviewers in what to look for. I would guess 75% of readers base a lot of their rating on the results of a study even though that is completely inappropriate—except when evaluating the degree to which the results actually stem from the methods. In a world of volunteer reviewers, this will be a challenge. There are also data privacy and legal concerns. Not everything can be publicly shared, and in a world of registered reports, a lot more transparency is the ethos.

McBee

The legacy publication model has failed to produce a generally trustworthy and reliable literature. We should stop using it and explore other alternatives. The extent to which registered reports will improve the situation is currently an open empirical question, though I believe there are convincing reasons to suspect that it will. I do hope that our field will also adopt the exploratory reports format as well. I do not want anyone to be left behind. In the future, and increasingly, the present, journals will no longer be the sole means of dissemination for scholarly work. Editors and peer reviewers will no longer possess the power of censorship that they currently do. Peer review will become open, public, and transparent. However, human nature will not change, and it will always be important to judge the rigor and scientific value of a project separately from its findings.

Kettler

Are there other thoughts or comments you want to make about the registered report protocol for educational research?

Peters

I would love to see some kind of incentive put into place to get current faculty to try registered reports or even to try pre-registering their studies. Universities should offer some additional funding for those who try this new system, or maybe conferences could even reserve slots for registered report presentations. Registered reports will happen going forward, but I’d also like to see some incentives for existing faculty to change as well.

McBee

I commend you and Anne Rinn for your leadership in bringing registered reports to the Journal of Advanced Academics. It’s an incredibly exciting time to be in this field. Gifted education is a field that has traditionally lagged behind in the research innovation space, but that has changed. We are now at the forefront of the methodological revolution in education.

Peters

It is worth noting that pre-registration and registered reports are part of a larger emphasis on open science in research. A few colleagues and I wrote a piece about open science practices in giftedness research for Gifted Child Quarterly (McBee, Makel, Peters, & Matthews, 2018). A free version is available here: https://osf.io/tu9mq/

McBee

Between Scott and me, I think we’ve promoted that paper quite extensively! More important than reading our paper, I think, is reading the papers that we reference. Understanding the replication crisis is one of the most important things that researchers in our field can do. It is difficult to consider these issues deeply without concluding that some fundamental changes are needed in the way we conduct and disseminate research. Interested readers can learn more about registered reports at https://cos.io/registeredreports/.