Registered Reports in Special Education: Introduction to the Special Series

Abstract

This article introduces a special series of registered reports in Remedial and Special Education. Registered reports are an innovative approach to publishing that aim to increase the credibility of research. Registered reports are provisionally accepted for publication before a study is conducted, based on the importance of the research questions and the rigor of the proposed methods. If provisionally accepted, the journal agrees to publish the study if researchers adhere to accepted plans and report the study appropriately, regardless of study findings. In this article, we describe how registered reports work, review their benefits (e.g., combatting questionable research practices and publication bias, allowing expert reviewers to provide constructive feedback before a study is conducted) and limitations (e.g., requires additional time and effort, cannot be applied to all studies), review the application of registered reports in education and special education, and make recommendations for implementing registered reports in special education.

Keywords

research methodology quantitative evidence-based practice

Chambers and Tzavella (2020) described a paradox potentially underlying one source of bias in the scientific literature. On one hand, researchers are trained to act like detectives and dispassionately follow where the data lead, analyzing and reporting research objectively to build a valid evidence base. On the other hand, researchers are incentivized to behave like lawyers, who fashion the most compelling narrative from the evidence, to maximize the likelihood of publication in prestigious journals with low acceptance rates. Because opportunities to publish in high-impact journals are scarce and extremely competitive, researchers may feel pressure to submit manuscripts that will stand out (Giner-Sorolla, 2012). For example, studies with clear positive results that support innovative hypotheses may be perceived as more likely to be published. Whereas the detective’s approach benefits science as a whole by generating objective evidence, the lawyerly approach benefits researchers personally by enhancing career advancement. To the degree contemporary scientists work within “an incentive structure that preferentially values aesthetics over authenticity” (Hardwicke & Ioannidis, 2018, p. 3), it is logical they may follow the method of the lawyer rather than the detective.

Emerging evidence suggests some researchers sometimes have to manipulate the research process to generate studies well suited for publication. For example, a surprising proportion of researchers, including those in education (Makel et al., in press), report they and their colleagues engage in questionable research practices such as p-hacking (i.e., trying different approaches for analyzing data until statistical significance is attained), hypothesizing after results are known (i.e., HARKing, or generating a hypothesis after analyzing the data), and cherry-picking (i.e., selectively reporting positive outcomes; see Fraser et al., 2018; John et al., 2012). Moreover, evidence of publication bias and scarcity of published null findings has been documented in many fields (Dwan et al., 2013; Fanelli, 2012; Polanin et al., 2016), including special education (Chow & Ekholm, 2018; Therrien & Cook, 2018; Gage et al., 2017). Indeed, Franco et al. (2014) found many researchers in the social sciences simply choose to not write up studies with null findings, perhaps due to the belief they will not be published. Finally, it appears researchers often “spin” (i.e., present and interpret research results in a favorable manner) negative findings in articles (Chiu et al., 2017).

Although eschewing null findings, engaging in questionable research practices, and spinning results may enhance the likelihood of high-profile publications, it may also lead to invalid, misleading, and irreproducible research. Indeed, a series of large-scale projects in the social sciences have found many research findings are not supported upon replication (e.g., Klein et al., 2018; Open Science Collaboration, 2015), spawning concerns about a reproducibility crisis (Baker, 2016). Scholars have compared the effects of questionable research practices and publication bias on the validity of research to photoshopping pictures (Chambers, 2019) and to steroid use (Gehlbach & Robinson, 2018). That is, researchers engaging in questionable research practices and not submitting null results produce a research base that looks better than it really is, which provides a misleading basis for policy and practice.

Scholars have proposed a number of open-science practices to help reduce the occurrence of questionable research practices, publication bias, and—ultimately—irreproducible science (e.g., preregistration, open data, preprints; see Cook et al., 2018; van Dijk et al., 2021). One of the most intriguing open practices is registered reports. In short, registered reports shift the primary basis for publication to methodological rigor without consideration of the direction or magnitude of results, thereby incentivizing rigorous science rather than attainment of a certain type of result. The purpose of this article is to provide an overview of registered reports—including what they are, their benefits, their limitations, their use in education research, and recommendations for applying them in special education scholarship—and introduce a special series of registered reports in Remedial and Special Education, which we believe represents the first application of registered reports to research focused on students with and risk for disabilities.

How Registered Reports Work

Registered reports were developed to increase transparency and reduce bias in research by making changes in the conduct and review of studies. First, as in preregistration (i.e., publicly posting study plans prior to collecting data; see Gehlbach & Robinson, 2018), registered reports involve researchers delineating research questions and methods before beginning to conduct the study. Registered reports go beyond preregistration by making those research plans the focus of peer review and the primary basis for acceptance for publication. The ideas underlying registered reports are not new. As Chambers and Tzavella (2020) noted, Rosenthal (1966) proposed over 50 years ago that,

What we may need is a system for evaluating research based only on the procedures employed. If the procedures are judged appropriate, sensible, and sufficiently rigorous to permit conclusions from the results, the research cannot then be judged inconclusive on the basis of the results and rejected by the referees or editors. (p. 36)

The distinguishing feature of registered reports is that research and peer review take place in two distinct but related stages (see Figure 1).

Figure 1.

The process of registered reports.

Stage 1

In Stage 1, researchers first conceptualize a study and then develop and write up detailed study plans, which are submitted to a journal and undergo peer review. As in preregistration, researchers decide on and write up detailed study plans before recruiting participants or collecting data. Specifically, Stage 1 manuscripts consist of complete, but prospective, Introduction and Method sections that describe a study rationale; research questions and hypotheses; and detailed, planned methods—including a power analyses to justify sample size, as relevant. See Kiyonaga and Scimeca (2019) for a description of important study elements to include when writing a Stage 1 manuscript. Although registered reports were designed originally to improve the validity of hypothesis-testing studies using group-quantitative designs, Grand et al. (2018) proposed the approach can be used with single-case, qualitative, secondary data analyses, systematic reviews, and purely descriptive studies—all of which are commonly used in special education research. So long as researchers can concretely describe their methods before conducting a study, the registered report approach can be used. However, we are not aware of any published registered reports using qualitative or single-case designs. In theory, the same basic procedures for registered reports described in this article will apply, and many of the same benefits will be conferred by registered reports regardless of research design.

Unlike preregistration, in registered reports the completed Stage 1 manuscript is then submitted to a journal for peer review. Stage 1 manuscripts are submitted to journals and assigned to reviewers in the same way as a typical manuscript. However, reviewers only evaluate the introduction and proposed methods, as the study has not yet been conducted. As such, the focus of the review is solely on the justification and importance of the research questions and hypotheses, and the rigor of the proposed methods for examining those research questions, with no consideration given to direction or magnitude of findings. Because the study has yet to be conducted, reviewers and the editor can provide constructive feedback and recommendations for improving study design. Stage 1 can involve one or more rounds of review and revisions, culminating in the editors deciding to either reject or grant in-principle acceptance to the proposed study. In-principle acceptance means the journal agrees to publish the final research report, regardless of study findings, so long as the researchers adhere closely to the accepted study protocol, and report and discuss results appropriately. Because the study has yet to be conducted, researchers can incorporate feasible recommendations made by the reviewers and editor, which has resulted in high in-principle acceptance rates of Stage 1 manuscripts (Chambers, 2019). Typically, authors formally preregister the Stage 1 manuscript after in-principle acceptance on a registry such as the Open Science Framework to make it accessible by Stage 2 reviewers and other research consumers.

Stage 2

After a Stage 1 manuscript receives in-principle acceptance and is preregistered, the researchers recruit a sample and conduct the study. When conducting the study and analyzing study data, it is important for researchers to adhere to the accepted Stage 1 study plans. Researchers should notify the editor if and when substantive deviations from the approved protocol occur. Trivial changes that have little or no effect on study findings, such as wording changes to enhance clarity to a fidelity checklist, do not require correspondence with the editor. However, changes that reasonably effect study findings (e.g., researchers are unable to recruit the anticipated number of participants, resulting in an underpowered study) should be communicated with the editor as soon as possible. At this point, the editor decides whether the change is trivial or if it needs to be considered by reviewers. If the latter, the editor and reviewers determine whether the change (a) threatens the validity of study (and will result in the study not being published) or (b) may affect study results but does not pose a significant threat to study validity (and should be reported clearly in the Stage 2 manuscript).

When the study is completed and data analyzed, researchers write up the Stage 2 manuscript (i.e., a complete research report that provides a link to the accepted Stage 1 manuscript). The Introduction and Method sections in a Stage 2 manuscript should be essentially the same as in the approved Stage 1 manuscript. Results and Discussion sections reporting and interpreting the findings of the approved analyses are then added. Researchers can conduct and report exploratory analyses not approved in the Stage 1 manuscript, but exploratory analyses and findings must be clearly designated as such and reported separately from approved, confirmatory analyses in the Stage 2 manuscript.

Editors typically send out a Stage 2 manuscript to the same reviewers who reviewed the Stage 1 manuscript. Reviewers’ evaluations of Stage 2 manuscripts consist of examining substantive adherence to the approved protocol, and appropriate reporting and discussion of results. Importantly, reviewers cannot consider the direction, magnitude, or perceived interestingness of findings in their reviews. Given this specific charge for reviewers and editors during Stage 2 review, it is perhaps not surprising that Chambers (2019) noted a 100% acceptance rate of Stage 2 manuscripts at the journal Cortex.

The process of registered reports is similar to traditional dissertations (Mehlenbacher, 2019; Peters et al., 2020), but with reviewers instead of committee members and an editor instead of committee chair. The dissertation proposal is analogous to a Stage 1 manuscript, with committee members (or, in the case of registered reports, reviewers and editor) providing feedback to be incorporated into study plans before the proposal is accepted. The student (researcher) then conducts and writes up the study, and submits the completed dissertation (or Stage 2 manuscript) to committee members (or the editor and reviewer) for final approval (or acceptance). If the student (researcher) conducted the study as proposed, and wrote it up and defended it appropriately, committee members (or reviewers and editor) must approve (or accept) the manuscript regardless of the results.

Potential Benefits of Registered Reports

Registered reports can impart multiple benefits to the research base by discouraging questionable research practices and publication bias, and by promoting high-quality research. In addition, registered reports provide benefits to researchers and for increasing the transparency and credibility of research.

Promoting Adherence to Research Plans

Preregistration discourages and makes discoverable many questionable research practices by making available preregistered research plans posted before a study was conducted, which research consumers can compare with the final research report. In this way, using a different analytic approach to reach statistical significance (p-hacking), developing new hypotheses after seeing study results (HARKing), and selectively reporting findings only for analyses with desired findings (cherry-picking) can be discovered by identifying discrepancies between preregistered plans and the research report (Gehlbach & Robinson, 2018). However, researchers do not always adhere to preregistrations or report deviations from preregistered plans in their published papers (Claesen et al., 2019). Moreover, given that preregistration is still relatively novel in special education, reviewers and other researcher consumers in the field seem unlikely to investigate whether a study is preregistered, let alone compare preregistered plans to reported methods to identify potential questionable research practices. Therefore, preregistration, at least thus far, is unlikely to have reduced questionable research practices in a meaningful way.

Registered reports promote adherence to the proposed research plan by making study publication contingent on adhering to the preregistration, with compliance monitored and enforced during Stage 2 peer review. Under the traditional publication paradigm, researchers may feel pressure to engage in questionable research practices to generate desired outcomes to enhance the likelihood of publication. Registered reports flip this incentive structure by making publication conditional on adherence to accepted plans. Indeed, one of the only ways for a Stage 2 manuscript to be rejected is if authors deviate unjustifiably from approved plans by engaging in these questionable research practices. Registered reports are primarily associated with hypothesis-testing research because deviating from planned research methods designed to test a priori hypotheses invalidates tests of the hypotheses. For example, it is not meaningful to report a p value of .01 if the planned analysis was not significant, but the researcher conducted multiple other analyses until obtaining statistical significance and only reported the statistically significant finding.

Reducing Likelihood of Publication Bias

At least three mechanisms underlie publication bias (i.e., studies with null findings being published less often than similar studies with positive findings): (a) reviewers and editors being less likely to accept studies with null findings for publication, (b) researchers not writing up and submitting studies with null findings, and (c) researchers engaging in questionable research practices to change null findings into positive results. Registered reports address each of these issues (Grand et al., 2018). Because the decision to grant in-principle acceptance is made before the study is conducted or results known, reviewers and editors are undeniably blind to results, and in-principle acceptance is based solely on the importance and rigor of the proposed study. Moreover, direction of results cannot be used as a rationale for reviewers and editors to reject a study once in-principle acceptance is granted. As such, authors are incentivized to write up Stage 2 manuscripts with null findings. Finally, registered reports obviate the primary incentive for engaging in questionable research practices (i.e., publication) because publication decisions are unrelated to study results. Indeed, one of the only ways for a study with in-principle acceptance not to be published is engaging in questionable research practices that deviate from accepted plans.

Findings from initial meta-research on registered reports suggest they may result in different findings than traditional publications. Whereas 96% of the first-reported hypotheses were supported in 152 randomly selected traditional hypothesis-testing studies, only 44% of corresponding hypotheses were supported among 71 registered reports published in psychology and psychiatry (Scheel et al., 2020; see also Wiseman et al., 2019). Similarly, Allen and Mehler (2019) found only 39% of hypotheses in 113 registered reports in psychology and biomedicine were supported, a markedly lower proportion than in traditional publications. Although alternative explanations exist (e.g., researchers may examine more risky hypotheses in registered reports), these findings lend support to the notion that registered reports combat publication bias and the use of questionable research practices to generate significant findings (Chambers & Tzavella, 2020).

Strengthening Research Procedures

Registered reports also support the conduct of high-quality research by (a) allowing peer reviewers to provide constructive feedback before a study is conducted and (b) focusing on statistical power. In traditional peer review, peer reviewers can only critique a study after the fact. With registered reports, expert reviewers provide input for improving a study before it is conducted, when the researcher can incorporate (or argue against) such recommendations. Indeed, when 353 researchers reviewed 29 published registered reports and 57 nonregistered report comparison studies, they rated issues such as rigor of methodology, rigor of analysis, and overall study quality substantially higher in the registered reports (Soderberg et al., 2020). The authors suggested that effects may be due to registered reports authors receiving substantive feedback on critical study elements from reviewers and the editor before conducting the study. The benefit of expert feedback on study methods can strengthen a study using any design. In addition, insufficient power threatens validity in group-quantitative research by increasing the likelihood of both Type I and Type II error (Button et al., 2013). Because registered reports typically require a defensible power analysis in Stage 1 manuscripts (when relevant) to justify the planned sample size, registered reports should be adequately powered.

Benefits to Researchers

In addition to potential benefits for science and the validity of research bases, registered reports can also benefit researchers and reviewers. For example, the majority of registered report reviewers surveyed by DeHaven et al. (2019) reported enjoying reviewing registered reports more than traditional manuscripts. The most frequently mentioned positive aspect of reviewing registered reports was the ability to provide constructive feedback for improving the study before it was conducted. Although all researchers can benefit from constructive feedback from expert peer reviewers before conducting a study, early career researchers may find this aspect of registered reports particularly advantageous. In addition, although registered reports may require additional time for authors to incorporate recommendations from reviewers, they tend to have high acceptance rates (Chambers, 2019)—which may also be especially appealing to early career researchers.

Potential Limitations of Registered Reports

Although registered reports have many potential benefits, important limitations and challenges also exist. We describe three potential limitations to registered reports: they cannot be applied to all studies, they require increased time and effort, and they are perceived to decrease researcher flexibility (Chambers, 2019).

It is not appropriate to submit a study as a registered report for which research questions and methods cannot be concretely described before the study is conducted. This may be more likely to occur in single-case and qualitative research. For example, it would be impossible to meaningfully review a Stage 1 manuscript for a single-case study that did not describe the outcome measures or an independent variable because these determinations were to be made based on the results of functional behavior assessments. That being said, researchers conducting studies in which some study decisions will be made during the course of a study can be published as registered reports if the researchers can describe concretely how such decisions will be made in a Stage 1 manuscript. This might involve specifying decision-making trees to indicate under which conditions different approaches will be used (see Kiyonaga & Scimeca, 2019). For example, although a single-case researcher might not be able to specify in advance how many baseline data points they will collect in a study, they can specify a minimum number and the specific conditions under which they will collect more baseline data (Johnson & Cook, 2019). Using this approach, registered reports can be used for many different types of research designs, including many single-case and qualitative studies, even if all procedures cannot be fully specified in advance.

In addition, registered reports are not a good choice for researchers who must begin data collection right away, perhaps because they are investigating a timely issue (e.g., interviewing teachers about how a natural disaster that just happened affected their teaching) or are under a tight time frame (e.g., to meet a deadline imposed by a funder, to finish collecting data in schools before the end of the school year). Stage 1 review must occur before research for a registered report can begin, and the length of Stage 1 review for any specific study will depend on many factors (e.g., the timeliness of reviewers, the number of rounds of review before in-principle acceptance). Thus, researchers must build in sufficient time for Stage 1 review (e.g., 6 months) between submission of a registered report and the planned start date of their study.

An additional limitation is that researchers who do not have flexibility in their research plans should not consider registered reports. For example, if a doctoral student’s dissertation committee or a researcher’s funding agency absolutely require previously approved methods be used, the researcher cannot respond to reviewers’ recommendations in Stage 1 review, thereby jeopardizing in-principle acceptance. In these cases, we recommend simply preregistering study plans. We hope many funding agencies and dissertation committees would grant some flexibility to incorporate reasonable, constructive feedback from expert reviewers. Indeed, intriguing proposals exist for combining the peer review processes for funding and registered report publication (e.g., Chambers & Tzavella, 2020; Munafò, 2017) that we believe could also be applied in dissertation work.

Another challenge for registered reports is the additional time and work involved. Most special education researchers do not customarily write out formal and detailed study justifications, research questions, hypotheses, and complete study methods before beginning a study (i.e., a Stage 1 manuscript), and doing so requires significant time and effort. However, this is balanced to some degree by time and effort saved analyzing data (i.e., only running the approved analyses) and writing the final (Stage 2) manuscript (as much of the Introduction and Method sections are written already; see Peters et al., 2020; Robinson et al., 2020). Importantly, editors and reviewers must go through two separate stages of review, each potentially involving multiple rounds of review. However, Stage 2 review is typically not taxing for editors and reviewers, as in-principle acceptance has already been granted and review is focused just on making sure approved plans were adhered to and results reported and discussed appropriately.

A final concern often raised is that registered reports may stifle researcher flexibility and freedom, which Chambers (2019) suggested is a misperception. Registered reports do require researchers to adhere to accepted study plans and not engage in questionable research practice such as p-hacking, HARKing, and cherry-picking. So, in some sense, freedom to engage in questionable research practice undetected is reduced. However, researchers can and should engage in exploratory analyses within the structure of registered reports. For example, as a researcher conducts a registered study and corresponding analysis, they may discover additional analyses they wish to explore with the data collected even though those analyses were not specified in the approved Stage 1 manuscript. Researchers have the freedom to do so in registered reports, but they must clearly identify which analyses are exploratory and report them in a separate subsection of the Results. In this way, the distinction between confirmatory (i.e., with hypotheses posed before data is collected) and exploratory analyses is clear, and researchers cannot present exploratory analysis as if they are confirmatory (i.e., HARKing).

Applications of Registered Reports

Although registered reports can improve the validity of research bases and are growing in popularity, there are few published registered reports in education and special education. Chambers and Tzavella (2020) reported that although 225 journals now accept registered report submissions, only 284 registered reports had been published in 54 journals. The Center for Open Science (n.d.) curates lists of (a) journals that have adopted registered reports and (b) published registered reports. Among the listed journals that accept registered reports as a regular submission option are a few in the fields of education (i.e., British Journal of Educational Psychology; Frontiers in Education; Journal of Research in Reading; Learning and Instruction; Mind, Brain, and Education; and Scientific Studies of Reading) and special education/exceptionality (i.e., Exceptional Children, Gifted Child Quarterly). Although not included on the curated list, we note AERA Open has recently begun to accept registered reports as regular submissions (Reich et al., 2020).

To identify published registered reports in education, we conducted advanced searches on Google Scholar using “registered report” as an exact phrase and each of the above journals as sources on October 11, 2020. We identified the following completed registered reports:

“Does what you get depend on who you are with? Effects of student composition on teaching quality” (Fauth et al., 2021; in Learning and Instruction)

“Should I stay or should I go? Predictors and effects of studying abroad during high school” (Hübner et al., 2021; in Learning and Instruction)

“Disentangling the process of epistemic change: The role of epistemic volition” (Kerwer et al., 2020; in British Journal of Educational Psychology)

The Center for Open Science (n.d.) also curates a list of 36 journals (as of October 11, 2020) that have published or have in progress special issues on registered reports. Journals related to education include AERA Open, Child Development, and Learning Disability Quarterly. A special topics series on registered reports in AERA Open (Reich et al., 2020) included seven studies:

“Large-scale estimates of LGBQ-heterosexual disparities in the presence of potentially mischievous responders: A preregistered replication and comparison of methods” (Cimpian & Timmer, 2019)

“Academic harms of missing high school and the accuracy of current policy thresholds: Analysis of preregistered administrative data from a California school district” (Kirksey, 2019)

“Save your strokes: Chinese handwriting practice makes for ineffective set of instructional time in second language classrooms” (Lu et al., 2019)

“Experimental evidence on the effects (or lack thereof) of informational framing during the college transition” (Kramer, 2020)

“Smart but evil? Student-teachers’ perception of educational researchers’ epistemic trustworthiness” (Merk & Rosman, 2019)

“Effect of local norms on racial and ethnic representation in gifted education” (Peters et al., 2019)

“Taking it to the next level: A field experiment to improve instructor-student relationships in college” (Robinson et al., 2019)

Four of the studies in the special series used a group experimental design, whereas three studies were secondary data analyses. One study involved students with exceptionalities, a secondary data analysis involving gifted students (Peters et al., 2019). The special issue of Child Development has not yet been published.

We are aware of three special issues or series on registered reports in special education journals. Two of the special issues/series are in progress and, as of October 11, 2020, do not have any completed registered reports accepted. One special series will appear in Learning Disability Quarterly (Cook et al., in progress) and will feature registered reports using a variety of research designs to examine research questions related to students with and at risk for learning disabilities. Another special issue will appear in Remedial and Special Education and will feature meta-research examining the research base in special education (Therrien & Cook, in progress).

This article serves as the introduction for the third special series on registered reports in a special education journal. In addition to this introductory article, the special series will include three registered reports. The first registered report, which appears in this issue, is a randomized control trial examining the effects of a science program on the outcomes of second-grade students, including students with and at risk for disabilities (Doabler et al., this issue). To the best of our knowledge, this is the first registered report published targeting students with and at risk for disabilities. The other two studies in the special issues have received in-principle acceptance, but data collection for both was delayed by the COVID-19 pandemic. Both of the other two studies in the special series are single-case designs, which are—to the best of our knowledge—the first single-case design studies to receive in-principle acceptance as registered reports. They will be published in Remedial and Special Education when the studies are completed and Stage 2 manuscripts written up and accepted.

Recommendations

In this section, we briefly provide some recommendations related to registered reports for editors, authors, and reviewers. See also Kiyonaga and Scimeca (2019) for recommendations for authors; Chambers and Tzavella (2020) for tips and advice for editors, authors, reviewers; and Center for Open Science (n.d.) for a variety of resources on registered reports.

Given the scarcity of journals in education generally and special education specifically that accept registered reports, our primary recommendation is for journal editors to consider adopting registered reports. One approach for editors is to do a trial run by conducting a special issue or series, which allows them to experience and develop procedures for registered reports without committing their journal to the practice. For instance, AERA Open adopted registered reports after the positive experience of doing a special topics series on the practice (Reich et al., 2020). One potential concern for editors is the level of methodological detail required in Stage 1 manuscripts may result in Stage 2 manuscripts exceeding journal page limits. Reich et al. (2020) recommended editors work with authors to determine whether some of the detailed methods that may be needed for Stage 1 review (e.g., definitions and coding options for variables) can be made available as supplementary material rather than published in the Stage 2 manuscript. We also recommend editors provide clear and explicit guidance to authors and reviewers. The registered reports process will be new for many special education scholars, and providing clear and complete guidelines will help scholars decide whether and how to submit Stage 1 manuscripts and serve as reviewers. Center for Open Science (n.d.) makes available detailed templates for editors to communicate with authors and reviewers, which we adapted for this special series. Extant guidance focuses on group-quantitative research and will need to be adapted for other designs.

Like editors, we encourage authors to consider trying this new format for publishing research. One obstacle is the limited number of journals and special issues in special education that accept registered reports. Researchers might reach out to editors who do not currently accept registered reports and ask whether they would consider accepting a submission as a trial, which is how Gifted Child Quarterly first considered registered reports (J. Adelson, personal communication, October 14, 2020). Because reviewers do not have study results to help tie a study together in a Stage 1 manuscript, authors should provide a detailed and well organized Stage 1 manuscript with each research question, hypothesis, sampling plan, and analysis plan clearly aligned (Chambers & Tzavella, 2020). In addition, researchers should take care to build in sufficient time for Stage 1 review to occur if they must begin or end the study within a particular time frame, as is common in school-based research.

Reviewers tend to enjoy reviewing registered reports more than typical manuscripts (DeHaven et al., 2019), likely because they are able to provide constructive feedback to improve study design rather than just critique it. We recommend reviewers carefully read guidelines provided for conducting Stage 1 and Stage 2 reviews, as they will (a) be different from typical reviews that scholars are used to conducting, and (b) may vary across journals. Reviewers should know they can point out methodological flaws during Stage 2 review, but authors cannot change approved methods. Rather, reviewers can request authors note identified issues as limitations (Chambers & Tzavella, 2020).

Conclusion

Although scholarly research articles have remained remarkably stable over time, Mehlenbacher (2019) argued, “the Registered Report marks a new genre of empirical research reporting” (p. 61) that provide researchers “an important moment . . . [to] reflect on the alignment of their communication practices with the norms and values that circulate within the enterprise of science” (p. 62). In this article, we have suggested registered reports can help improve the credibility of published research studies and the research bases they comprise by basing the decision to publish a study on the importance of the research question and the rigor of the methods to investigate those questions before a study is conducted. Despite the potential benefits of registered reports, this approach is relatively new and has not yet been applied in research focused on students with and risk for disabilities. The first registered report in our special series (Doabler et al., this issue) appears in this issue. We thank the authors and reviewers involved in the special series, and the editors of Remedial and Special Education for their support. Given the importance of a credible research base in applied fields such as special education, we hope to see an increase in the use of registered reports in the field.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Bryan G. Cook

References

Allen

Mehler

D. M.

(2019). Open science challenges, benefits and tips in early career and beyond. PLOS Biology, 17(5), Article e3000246. https://doi.org/10.1371/journal.pbio.3000587

Baker

(2016). Reproducibility crisis? Nature, 533(26), 353–366. https://doi.org/10.1038/533452a

Button

K. S.

Ioannidis

J. P.

Mokrysz

Nosek

B. A.

Flint

Robinson

E. S.

Munafò

M. R.

(2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. https://doi.org/10.1038/nrn3475

Center for Open Science. (n.d.). Registered reports: Peer review before results are known to align scientific values and practices. https://www.cos.io/initiatives/registered-reports

Chambers

(2019). What’s next for registered reports? Nature, 573, 187–189. https://doi.org/10.1038/d41586-019-02674-6

Chambers

Tzavella

(2020, February 10). Registered reports: Past, present and future. https://doi.org/10.31222/osf.io/43298

Chiu

Grundy

Bero

(2017). “Spin” in published biomedical literature: A methodological systematic review. PLOS Biology, 15(9), Article e2002173. https://doi.org/10.1371/journal.pbio.2002173

Chow

J. C.

Ekholm

(2018). Do published studies yield larger effect sizes than unpublished studies in education and special education? A meta-review. Educational Psychology Review, 30, 727–744. https://doi.org/10.1007/s10648-018-9437-7

Cimpian

J. R.

Timmer

J. D.

(2019). Large-scale estimates of LGBQ-heterosexual disparities in the presence of potentially mischievous responders: A preregistered replication and comparison of methods. AERA Open, 5(4). https://doi.org/10.1177/2332858419888892

10.

Claesen

Gomes

Tuerlinckx

Vanpaemel

(2019, May 9). Preregistration: Comparing dream to reality. https://doi.org/10.31234/osf.io/d8wex

11.

Cook

B. G.

Lloyd

J. W.

Mellor

Nosek

B. A.

Therrien

W. J.

(2018). Promoting open science to increase the trustworthiness of evidence in special education. Exceptional Children, 85(1), 104–118. https://doi.org/10.1177/0014402918793138

12.

Cook

B. G.

Therrien

W. J.

Hart

(in progress). Registered reports in the field of learning disabilities [special series]. Learning Disability Quarterly.

13.

DeHaven

A. C.

Graf

Mellor

D. T.

Morris

Moylan

Pedder

Tan

(2019, September 17). Registered reports: Views from editors, reviewers and authors. https://doi.org/10.31222/osf.io/ndvek

14.

Doabler

Longhi

Therrien

Roberts

Hess

Maddox

. . . Toprac

. (this issue). Efficacy of a second-grade science program: Increasing science outcomes for all students. Remedial and Special Education. https://doi.org/10.1177/0741932521989091

15.

Dwan

Gamble

Williamson

P. R.

Kirkham

J. J.

(2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias—An updated review. PLOS ONE, 8(7), Article e66844. https://doi.org/10.1371/journal.pone.0066844

16.

Fanelli

(2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904. https://doi.org/10.1007/s11192-011-0494-7

17.

Fauth

Atlay

Dumont

Decristan

(2021). Does what you get depend on who you are with? Effects of student composition on teaching quality. Learning and Instruction, 71, Article 101355. https://doi.org/10.1016/j.learninstruc.2020.101355

18.

Franco

Malhotra

Simonovits

(2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484

19.

Fraser

Parker

Nakagawa

Barnett

Fidler

(2018). Questionable research practices in ecology and evolution. PLOS ONE, 13(7), Article e0200303. https://doi.org/10.1371/journal.pone.0200303

20.

Gage

N. A.

Cook

B. G.

Reichow

(2017). Publication bias in special education meta-analyses. Exceptional Children, 83(4), 428–445. https://doi.org/10.1177/0014402917691016

21.

Gehlbach

Robinson

C. D.

(2018). Mitigating illusory results through preregistration in education. Journal of Research on Educational Effectiveness, 11(2), 296–315. https://doi.org/10.1080/19345747.2017.1387950

22.

Giner-Sorolla

(2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7(6), 562–571. https://doi.org/10.1177/1745691612457576

23.

Grand

J. A.

Rogelberg

S. G.

Banks

G. C.

Landis

R. S.

Tonidandel

(2018). From outcome to process focus: Fostering a more robust psychological science through registered reports and results-blind reviewing. Perspectives on Psychological Science, 13(4), 448–456. https://doi.org/10.1177/1745691618767883

24.

Hardwicke

T. E.

Ioannidis

J. P.

(2018). Mapping the universe of registered reports. Nature Human Behaviour, 2(11), 793–796. https://doi.org/10.1038/s41562-018-0444-y

25.

Hübner

Trautwein

Nagengast

(2021). Should I stay or should I go? Predictors and effects of studying abroad during high school. Learning and Instruction, 71, Article 101398. https://doi.org/10.1016/j.learninstruc.2020.101398

26.

John

L. K.

Loewenstein

Prelec

(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

27.

Johnson

A. H.

Cook

B. G.

(2019). Preregistration in single-case design research. Exceptional Children, 86(1), 95–112. https://doi.org/10.1177/0014402919868529

28.

Kerwer

Rosman

Wedderhoff

Chasiotis

(2020). Disentangling the process of epistemic change: The role of epistemic volition. British Journal of Educational Psychology. Advance online publication. https://doi.org/10.1111/bjep.12372

29.

Kirksey

J. J.

(2019). Academic harms of missing high school and the accuracy of current policy thresholds: Analysis of preregistered administrative data from a California school district. AERA Open, 5(3). https://doi.org/10.1177/2332858419867692

30.

Kiyonaga

Scimeca

J. M.

(2019). Practical considerations for navigating registered reports. Trends in Neurosciences, 42(9), 568–572. https://doi.org/10.1016/j.tins.2019.07.003

31.

Klein

R. A.

Vianello

Hasselman

Adams

B. G.

Adams

R. B.

Jr. Alper

. . . Batra

(2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225

32.

Kramer

J. W.

(2020). Experimental evidence on the effects (or lack thereof) of informational framing during the college transition. AERA Open, 6(1). https://doi.org/10.1177/2332858420908536

33.

Ostrow

K. S.

Heffernan

N. T.

(2019). Save your strokes: Chinese handwriting practice makes for ineffective use of instructional time in second language classrooms. AERA Open, 5(4). https://doi.org/10.1177/2332858419890326

34.

Makel

M. C.

Hodges

Cook

B. G.

Plucker

, (in press). Both questionable and open research practices are prevalent in education research. Educational Researcher.

35.

Mehlenbacher

A. R.

(2019). Registered reports: Genre evolution and the research article. Written Communication, 36(1), 38–67. https://doi.org/10.1177/0741088318804534

36.

Merk

Rosman

(2019). Smart but evil? student-teachers’ perception of educational researchers’ epistemic trustworthiness. AERA Open, 5(3). https://doi.org/10.1177/2332858419868158

37.

Munafò

M. R.

(2017). Improving the efficiency of grant and journal peer review: Registered reports funding. Nicotine & Tobacco Research, 19(7), Article 773. https://doi.org/10.1093/ntr/ntx081

38.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

39.

Peters

S. J.

Rambo-Hernandez

K. E.

Makel

M. C.

Matthews

M. S.

Plucker

J. A.

(2019). Effect of local norms on racial and ethnic representation in gifted education. AERA Open, 5(2). https://doi.org/10.1177/2332858419848446

40.

Peters

S. J.

Rambo-Hernandez

K. E.

Makel

M. C.

Matthews

M. S.

Plucker

J. A.

(2020). Reflections on the registered report process for “Effect of local norms on racial and ethnic representation in gifted education.” AERA Open, 6(2). https://doi.org/10.1177/2332858420919054

41.

Polanin

J. R.

Tanner-Smith

E. E.

Hennessy

E. A.

(2016). Estimating the difference between published and unpublished effect sizes: A meta-review. Review of Educational Research, 86(1), 207–236. https://doi.org/10.3102/0034654315582067

42.

Reich

Gehlbach

Albers

C. J.

(2020). “Like upgrading from a typewriter to a computer”: Registered reports in education research. AERA Open, 6(2). https://doi.org/10.1177/2332858420917640

43.

Robinson

C. D.

Scott

Gottfried

M. A.

(2019). Taking it to the next level: A field experiment to improve instructor-student relationships in college. AERA Open, 5(1). https://doi.org/10.1177/2332858419839707

44.

Robinson

C. D.

Scott

Gottfried

M. A.

(2020). Taking it to the next level: A field experiment to improve instructor-student relationships in college. AERA Open, 6(2). https://doi.org/10.1177/2332858420919055

45.

Rosenthal

(1966). Experimenter effects in behavioral research. Appleton-Century-Crofts.

46.

Scheel

A. M.

Schijen

Lakens

(2020, February 5). An excess of positive results: Comparing the standard psychology literature with registered reports. https://doi.org/10.31234/osf.io/p6e9c

47.

Soderberg

C. K.

Errington

T. M.

Schiavone

S. R.

Bottesini

J. G.

Singleton Thorn

Vazire

. . . Nosek

B. A.

(2020, November 16). Research quality of registered reports compared to the traditional publishing model. https://doi.org/10.31222/osf.io/7x9vy

48.

Therrien

W. J.

Cook

B. G.

(2018). Null effects and publication bias in learning disabilities research. Learning Disabilities Research & Practice, 33(1), 5–10. https://doi.org/10.1111/ldrp.12163

49.

Therrien

W. J.

Cook

B. G.

(in progress). Examining the special education research base [special issue]. Remedial and Special Education.

50.

van Dijk

Schatschneider

Hart

S. A

. (2021). Open science in education science. Journal of Learning Disabilities, 54(2), 139–152. https://doi.org/10.1177/0022219420945267

51.

Wiseman

Watt

Kornbrot

(2019). Registered reports: An early example and analysis. PeerJ, 7, Article e6232. https://doi.org/10.7717/peerj.6232