Abstract
The Canadian Journal of School Psychology (CJSP) is offering scholars the opportunity to register research reports and make research protocols publicly available to promote replication, transparency, credibility, and utility for clinical practice. The purpose of this article is to outline the challenges regarding replication, reproducibility, and evidence-based practices, as well as describe the submission protocol and criteria for acceptance of registered reports. Advances and criticisms of the registered reports model are discussed. Although CJSP will accept submissions through the traditional peer-review model, registered reports and support of replication studies have the objective of promoting high-quality research to improve the research foundation for evidence-based practices in the profession of school psychology.
Strong science is based upon robust findings that lead to clearer understanding of a given concept (Nosek, Ebersole, DeHaven, & Mellor, 2018). Replication of results is an essential component of the scientific process that ensures the robustness of findings (Baker, 2016). However, there is currently a replication crisis in many scientific domains, including medicine, life sciences, and natural sciences; largely due to the prevalence of questionable research practices (Baker, 2016; Ioannidis, 2005). Efforts to replicate 100 major studies, which established the evidential foundation of psychology and several other fields, have found that 61% of the studies cannot be replicated or are extremely difficult to replicate (Ioannidis, 2017; Open Science Collaboration, 2015). Even most studies that were replicated found smaller effect sizes than the originally published studies. Certainly, if studies are difficult to replicate in a laboratory context, then implementing studies in clinical practice (i.e., schools) are likely to be even more problematic (Tajika, Ogawa, Takeshima, Hayasaka, & Furukawa, 2015). Questionable research practices refer to the lack of replication, data sharing, and transparency of methodological procedures as well as low statistical power, P-hacking, harking (i.e., Hypotheses After Results are Known), and publication bias (John, Loewenstein, & Prelec, 2012; Lilienfeld, 2017; McGill, Ward, & Canivaz, 2018; Nosek et al., 2015). Concerns about these questionable research practices have existed in the sciences for decades, but the replication crisis and advances in methodology have brought these issues to the forefront (Ioannidis, 2016a; Mellor, 2017). Consequently, the identification of questionable research practices has become important to the development of further knowledge and scientific endeavors, as well as the evaluation, selection, and implementation of evidence-based practices (EBPs) in school psychology (Greenhalgh, Howick, & Maskrey, 2014; Rycroft-Malone et al., 2004).
Studies published in peer-reviewed scholarly journals are among the most influential sources of credible information upon which to base EBPs. Even the most influential and high-impact journals may include research with questionable research practices that may not result in quality EBPs. The impact factor is a metric of influence based on the number of times papers that were published in the specific journal are cited by other research papers published in scholarly journals (Garfield, 2006). Although impact factors were initially created to track the influence of journals for the purposes of highlighting important ideas, they are not without flaws (Garfield, 2006). For example, scholarly journals can increase their impact factor by publishing exciting and novel scientific findings, rather than high-quality methods. This often results in emphasis being placed upon the publication of statistically significant results and not research design quality, and rejection of replication studies because they are insufficiently novel (Dwan, Gamble, Williamson, & Kirkham, 2013; Fraley & Vazire, 2014). At the extreme, incentives regarding prestige, such as impact factors, may encourage the publication of statistically significant findings at the risk of disseminating lesser quality research, resulting in a higher prevalence of questionable research practices (Kühberger, Fritz, & Scherndl, 2014). This publication bias for novel information also provides less opportunity to validate or question previous scientific results through publication (Fraley & Vazire, 2014; Makel & Plucker, 2014). Thus, relying on peer reviewed research may not result in a dramatic improvement of EBPs.
The field of school psychology is especially susceptible to the effects of questionable research practices. Research that is based upon questionable research practices has direct influence on implementation of research that contributes to the education, mental health and well-being of children, schools, families, and communities, and therefore, must be actively avoided (La Greca, Silverman, & Lochman, 2009). To address the educational and mental health concerns of children and their families, school psychology requires EBPs to serve as the basis for assessment and intervention (Levant & Hasan, 2008). Oftentimes, these interventions must be tailored to the client and community needs, as well as to the availability of resources, resulting in a conceptual rather than identical application. Findings that are difficult to replicate in controlled laboratory settings, will almost certainly be harder to replicate when implemented in the specific context of the classroom, therapeutic counseling session, and systemic intervention. Questionable research practices have direct influence on whether specific interventions and professional activities can be called “evidence-based” with confidence.
EBP is the explicit use of the best available research evidence to inform research practices and service delivery (Levant & Hasan, 2008). Originating from the medical sciences, the goal of EBP was to base clinical decision-making on empirical evidence and avoid biases, such as adherence to tradition, unsystematic observation, and intuition (Guyatt et al., 1992). EBP has become increasingly important to the identity of school psychology. In psychology, EBP has the potential to influence and significantly improve the quality of services provided by psychologists. However, EBP can only be a useful concept if the evidential support of our profession is reproducible, replicable, robust, transparent, and credible (Tajika et al., 2015). Scholarly journals are the main resource for disseminating research findings and are therefore able to incentivize quality research and promote school psychology as an evidence-based profession.
In response to the identification of questionable research practices, an open science movement has been launched (Allen & Mehler, 2018). The movement emphasizes improving research methods and collaboration, increasing transparency, expanding access, and redirecting incentives for scholars and scholarly journals. Although the movement has influenced research practices in the medical sciences, neurosciences, social psychology, and other fields of study; it has yet to make a significant impact on education, mental-health service delivery, and professional school psychology (Simmons, Nelson, & Simonsohn, 2011; van der Zee & Reich, 2018).
The Canadian Journal of School Psychology (CJSP) is launching two initiatives to begin the process of addressing questionable research practices and improving the quality of the evidence behind EBPs that are the foundation of professional school psychology. Although CJSP will continue to accept brief reports, original articles, literature reviews and meta-analyses, book and test reviews, and special issues; CJSP will now accept and value replications and registered reports.
Reproduction and Replication
Reproduction and replication of scientific findings are foundational components of all science (Ioannidis, 2016a, 2016b). Reproduction refers to the same procedures being repeated by the same scholar or scientific team to determine if the phenomenon can be observed again or if the observed situation is due to error, extraneous variables, or sampling issues (Tajika et al., 2015). In most fields of study, when a phenomenon is observed, the next step is to attempt to reproduce the phenomenon. However, in clinical psychological and educational research, the reproduction stage is rarely conducted (van der Zee & Reich, 2018). There are pragmatic challenges in conducting reproduction studies in these fields; however, the scientific limitations of forgoing reproduction are important. The validity of conclusions drawn from such findings is questionable as it becomes impossible to distinguish between important findings and sampling, measurement, or data errors (Maxwell, Lau, & Howard, 2015).
In addition to reproduction, replication is a vital component of the scientific enterprise and research. Replication refers to a different scholar or scientific team applying a previously reported methodology to determine if the phenomena can be observed in different settings (Baker, 2016; Open Science Collaboration, 2015). There are two types of replications: exact replications and conceptual replications. Exact replications refer to strictly following the methodology and research protocol used in the original study (Open Science Collaboration, 2015). The purpose is to determine whether the procedure and protocol of the study is sufficiently robust to result in the same phenomenon with different personnel involved and a different sample of participants. Conceptual replications refer to attempting to reproduce the phenomenon, yet involving variations from the methodology and research protocol used in the original study. In many fields of science, conceptual replications are of limited utility. However, in school psychology, conceptual replications are essential because some aspect of the original methodology and research protocol will almost certainly need to be modified to match the individual needs of the student or client, availability of resources, or other systemic and contextual issues within the environment.
Because replications are not novel or striking endeavors, they are less likely to be undertaken by researchers or published in scholarly journals. Considering the emphasis on high-impact factor metrics, replications are also less pursued because they do not generate unique citations (Ferguson & Heene, 2012). The result is less than 2% of scholarly research papers in school psychology journals have been replicated or are themselves replications (Shaw, Lysenko, Wang, & Nieva Bitencourt, 2018).
The CJSP values and welcomes exact and conceptual replications of previously published and reported research. Conducting replications is an excellent training opportunity for graduate students who are first learning how to conduct research. However, even advanced researchers benefit from testing well-known research to determine the viability of the evidence for implementation in an evidence-based intervention and for determining how robust a phenomenon is to minor changes in procedures and protocol. CJSP welcomes such submissions as a step toward strengthening the evidential base in the profession of school psychology.
Registered Reports
The current model of publication provides incentives to produce novel findings, which places greatest importance on the results of an experiment (Dwan et al., 2013). Consequently, effect sizes and p-values are two ways in which the value of a paper is evaluated (Jefferson, Alderson, Wager, & Davidoff, 2002). The rationale, theoretical importance, quality of scientific procedures and protocol, measurement quality, and data analysis are frequently secondary considerations. The result is that published studies tend to be of low power, have sampling concerns, poor control of variables, and containing otherwise questionable research practices (Rosenthal, 1979). There is a positive relationship between large effect sizes and poor methodological quality (Ioannidis, 2005). The current incentives for scholars to publish in high impact journals compromise the objective of promoting high-quality research and a strong evidential foundation of EBPs (Chambers, Dienes, McIntosh, Rothstein, & Willmes, 2015).
Although the CJSP will continue to accept traditional research submissions, CJSP will also consider a radical reshaping of the publication review process, known as registered reports. This approach is new to school psychology journals and reflects a different way of thinking about research. Registered reports have the effect of minimizing publication bias, P hacking, harking, low statistical power, and increasing transparency (Anderson, Kelley, & Maxwell, 2017; Chambers, 2013; Nosek et al., 2018). In this fashion, the evidence published as a registered report is more likely to be robust and relevant to EBP (Nosek & Lakens, 2014).
The logic of a registered report is that the introduction, research methodology, and data analysis plan are submitted for peer review before data are collected. Proposals that address the important issues and use the strongest research methods will receive an in-principle acceptance letter before data are collected. Once the proposal, known as Stage I, has been approved by the editor, the researcher then conducts the experiment, collects data, analyzes data, and writes the results and conclusions sections. This final version of the manuscript, known as Stage II, is submitted. Providing that the researcher followed the approved methodology, the manuscript will be published regardless of whether the results were statistically significant or not. The paper will be accepted if the data are collected in accordance with the pre-approved plan and if the conclusions are reasonable and are related to the theory or rationale established in Stage I. Pre-registering reports reduces the opportunity for publication bias and several other questionable research practices, such as harking, p-hacking, or selectively choosing statistically significant results (Nosek et al., 2018). Registered reports place emphasis on high-quality research design.
For the CJSP, the following steps will be in place for registered reports:
1. The author submits the introduction, methods, proposed data analysis (including power analysis, if applicable), and pilot data (if applicable) for review through the Scholar One manuscript submission portal.
2. The proposal is then reviewed by the editor to determine appropriateness and fit with CJSP. At this stage, the manuscript could be rejected or sent for peer review.
3. The criteria for the review of the Stage I protocol are as follows: A. The research questions contribute to the clinical, theoretical, or general knowledge base in the field of school psychology. B. The authors provide a strong, logical, and organized rationale with accompanying literature support for the hypothesis. C. The proposed methodology, sampling, measurement instruments, and research protocols are feasible, detailed, clear, and well-planned. D. The author proposes appropriate data analysis to test the hypotheses, which will most often include an analysis of statistical power, as well as proposed statistical corrections (e.g., post hoc tests, addressing assumptions, managing outliers, etc.) as necessary. E. The methodological procedures and protocols are sufficiently detailed and clear to facilitate future replication of the proposed methodologies, procedures, and data analyses. F. The authors have made sufficient consideration of additional variables that may influence the procedures, analyses, results, or interpretation of data, as well as how those factors may influence the evaluation of the proposed hypotheses.
4. The Stage I protocol receives an editorial decision to either reject, reject and recommend revisions, accept with minor revisions, or is offered an in-principle acceptance. After receiving an in-principle acceptance, data collection begins in accordance with the methods and research protocol.
The accepted proposal is then registered in a repository so that there is a clear timeline established and the procedures are entirely transparent. Depositing the Stage I proposal makes later comparisons between the proposed (Stage I) and final reported methods and protocols (Stage II) possible. Moreover, the repository helps to limit the potential for switched outcomes, data analysis errors, and other questionable research practices. The primary logic is that the deposited Stage I proposal results in a timestamp that makes it impossible to develop hypotheses after the results are known and limits opportunity to change data analysis plans to obtain statistically significant results. There is a dedicated portal available for Stage I registered reports at the Center for Open Science portal (https://cos.io/rr/).
Some authors may have concerns that their publicly available Stage I protocol could be adopted or scooped by another scholar. Although this is unlikely, authors concerned about making their protocols public before collecting data have the option of registering a private embargo that expires after a fixed time (e.g., 4 years on the Centre for Open Science portal) or at the point of Stage II final acceptance.
Authors may withdraw from the process if they find that conducting the study in a manner consistent with the Stage I proposal is not possible. This does not preclude authors from submitting the study as a regular original research manuscript. However, the manuscript will no longer be recognized as a registered report.
When faced with pragmatic issues of data collection, the authors may find that they need to deviate from the proposal. Any proposed deviations require consultation with the editor. Minor deviations from the Stage I proposal may require an updated registered proposal in the repository, but are acceptable amendments to the proposed procedure. Major deviations may require a second peer review or removal from consideration as a registered report. Typically, studies are submitted for the institutional research ethics board for approval after Stage I approval. Sometimes, a deviation from the Stage I approved methodology and research protocol is required for local institutional research ethics board approval. Such deviation will require consultation with the editor to determine whether this deviation is substantially consistent with the approved Stage I proposal. A large percentage of registered reports eventually have some deviation from the Stage I proposal due to the reality of collecting data in schools and mental health settings.
5. The study is conducted, data are collected and analyzed, and the results and conclusions section of the paper are completed by the author.
6. The paper is submitted as a Stage II registered report and reviewed with the following criteria. A. The introduction, rationale, and stated hypotheses are identical (or have approved amendments) to those that were recorded during the Stage I approved protocol. B. The authors adhered to the registered experimental methods and research protocol as described in the approved Stage I protocol. C. Any unregistered post-hoc analyses are justified, based in strong rationale and statistical logic, and are necessary to answer the research questions and hypotheses. D. The author’s conclusions are justified given the data and analyses.
7. An editorial decision is made concerning request for revisions or acceptance for final publication.
8. Once there is a final acceptance, authors have the option to place raw data and statistical analyses into a repository that is publicly available. This extra layer of transparency allows outside scholars to re-analyze and verify the published findings, but is not required.
9. Authors are encouraged to post their manuscripts as preprints in repositories at any time during the process. CJSP encourages authors to place the Digital Object Identifier (DOI) of the preprint in the last line of the abstract of the published paper so that there is increased access to the manuscript for clinicians. This allows improved access to high-quality information, which further contributes to the evidential base for high-quality EBPs in school psychology.
10. Published registered reports will have an editor’s note regarding the level of transparency of the study and the quality of research methodology. The DOI of the deposited Stage I proposal, location of the data set (if applicable), and the DOI for the preprint (if applicable) will all be available with the published article.
Advantages of registered reports for scholars
Registered reports have advantages for the general scientific community, for improving EBPs, and for individual scholars. First, given that data collection is expensive and time-consuming, registered reports can help scholars to allocate their time, energy, and resources more efficiently (Chambers, 2013). For researchers, knowing that a paper will be accepted before the time-consuming data collection process begins is constructive and reassuring. Similarly, rejection at the Stage I may save time and energy in data collection for a project that may not be publishable, thereby helping researchers to focus their energy and resources.
Second, suggestions and revisions during the Stage I review process may result in an improved quality of research methodology and protocol (Chambers et al., 2015). In addition, editors are more invested in the process of developing a strong methodology and are available for consultation throughout the process. The goal of relationship between scholar and editor is to promote and encourage strong research methods in published papers rather than limiting the role of an editor to evaluating the final product of research. As such, editors can encourage a higher quality of research methodologies through Stage I reviews, and the public availability of approved Stage I protocols and data can improve the likelihood of replication. In addition, there is always a concern that replications may not be appropriate for publishing in some scholarly journals. A successful Stage I proposal with an in-principle acceptance indicates that replication is valued before going through the time and energy of collecting data on an exact or conceptual replication study.
In traditional publishing models, data collection occurs before the research methodologies have been peer reviewed (Jefferson et al., 2002). In comparison, registered reports change the scholars’ relationship to data collection. For example, in a traditional publishing model, there are concerns regarding whether the results will be statistically significant or whether the results are valued as publishable findings prior to submitting the manuscript for publication (Chambers et al., 2015). As such, scholars are tempted to analyze existing data using every possible permutation of analysis to identify statistically significant results. However, in a registered report, the emphasis on data collection is simply on accuracy and following the protocol with integrity. Integrating registered reports into the peer review process allows for greater opportunity to yield accurate, replicable, and robust results than the traditional publication model.
Registered reports involve detailed reviews of the methodologies and research protocols, which provides scholars with protection from post-publication critique. Inevitably, in traditional publication models, research is often scrutinized post publication for limitations regarding study design, sampling, measurement, or methodology, among other variables. In a registered reports publication model this discussion and review occurs prior to publication, which contributes to more constructive research, editorial involvement, and publication process. Although there is always room for differing interpretations, major flaws in study design are less likely to appear in registered reports.
An important component of scholarship is developing a reputation as an expert in a field of research (Allen & Mehler, 2018). Registered reports allow scholars to predict phenomena effectively. This allows for insight, prediction based on theory, or prediction based on clinical experience to be demonstrated transparently. Moreover, registered reports are sufficiently new in the field of school psychology that participants could quickly gain a reputation as an innovative scholar who minimizes questionable research practices, and supports the professional school psychology in their effort to promote an evidence-based profession.
Criticisms of Registered Reports
There have been some criticisms of registered reports that are worthy of consideration. For example, registered reports have been accused of limiting opportunity for scientific creativity (Chambers, 2017). Rarely do scholars in the proposal or Stage I consider all pragmatic issues and procedural variations that can occur in data collection. The ability to make substantial changes after starting a study leads to innovative procedures and exciting new findings. Registered reports require that researchers strictly adhere to a specific set of procedures, and as such, much of the creativity during the data collection phase is limited. However, registered reports place more emphasis on being creative, innovative, and thorough when initially designing the study than after data are collected.
A related concept is that serendipitous findings, which have been extremely important in all of science, will be minimized and that registered reports would make these findings impossible to identify. There is a fine line between changing analyses with the goal of obtaining statistically significant results and allowing for surprising or accidental findings to be discovered. Should important, surprising, or accidental findings be discovered in the context of a registered report, there are two major options. These findings can be included in the Stage II registered report manuscript with identification that these findings were outside of the original protocol. Such findings often add to the impact and quality of the research. The second option is to create a second manuscript that highlights the serendipitous findings, and this study would be published as a traditional original research manuscript.
Determining the fidelity with which scholars follow the Stage I registered protocol is difficult to operationalize. How will an editor determine whether a minor deviation from the agreed upon protocol is too much? It is unclear as to whether a minor change in a measure, sampling from a different population, or a new component of the procedure will pass a threshold that moves the study from following a pre-registered protocol to a failure to follow the protocol and to no longer be considered a registered report. The definition of this threshold will be especially important and difficult to determine in classroom or mental health settings that tend to have more variables to control and are more dynamic than bench science. As registered reports become more commonplace, more specific evaluation procedures regarding protocol changes for approved reports will help to address this criticism.
Exploratory Versus Explanatory Research
Registered reports are ideal for evaluating the effectiveness of programs or interventions in an experimental or quasi-experimental approach. These forms of studies are essential for developing EBPs. However, a great deal of research in school psychology is exploratory in nature. Registered reports can be used for exploratory research. Rather than hypotheses being related to null hypothesis statistical testing, a series of possible outcomes based on theory or clinical observations can be posited at the Stage I level. The goal of the registered report for exploratory research is that rather than a wide-open fishing expedition of a large and ill-defined number of variables, theoretically consistent and clinically important variables can be named and tested ahead of time. Therefore, the benefits of registering exploratory studies are to reduce publication bias by ensuring that all aspects of the studies are discoverable and to minimize alpha inflation for that part of analysis. Although exploratory research is important, explanatory research is required to develop evidence-based interventions with a high probability of causal and clinical impact. The use of registered reports will assist in differentiating exploratory from explanatory research and help to ensure that conclusions based on these different forms of research do not overlap.
Conclusion
CJSP maintains the objective of promoting high-quality research and best research practices. Two ways in which CJSP plans to achieve this objective are to provide scholars with opportunity to register reports and to make protocols and data publicly available, and to promote the replication of research findings. However, CJSP also recognizes that not all research is appropriate for registered reports. Therefore, CJSP will continue to publish traditional original research through the conventional peer-review process. Registered reports do not address every possible concern with the quality of research and school psychology. Yet, registered reports represent an important mechanism to increase reproducibility and replicability in scientific findings. Building the evidential foundation of research to be strong, credible, transparent, replicable, and useful will contribute to the development of higher-quality evidence-based interventions and practices for the benefit of children and their families, and the profession of school psychology. CJSP emphasizes an accurate and transparent science instead of focusing on what is most striking, statistically significant, or will garner most attention. Although the use of registered reports has the potential to improve all scientific areas of inquiry, it is especially important for school psychology as the evidence supporting interventions that affect the academic and mental health of children needs to be as reproducible, robust, and accurate as possible. CJSP contends the use of registered reports will promote stronger evidence within school psychology.
If there are additional questions about how to prepare a Stage I protocol for registered report in CJSP or if additional support is required, please contact the editor at
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is supported, in part, by Fonds de recherche société et culture (D’Intino).
