A How-to Guide for Open-Science Practices in Special Education Research

Benefits and limitations

Nosek et al. (2019) noted three primary benefits of preregistration. First, by making study plans transparent, preregistration provides a record of which analyses are planned a priori (i.e., hypothesis testing or confirmatory analyses) and which are post hoc (i.e., exploratory analyses). Second, by enabling comparisons between preregistered study plans and reported methods and findings, preregistration makes questionable research practices such as p-hacking, selective outcome reporting, data-peeking, and HARKing more readily discoverable, thereby discouraging their use. Finally, preregistration can help combat publication bias by making all planned studies discoverable regardless of whether the study was published.

A key concern with preregistration is that it might diminish or demean exploratory analysis. However, as DeHaven (2017) noted, researchers are encouraged to conduct and report exploratory analyses in preregistered studies, but they should clearly report them as exploratory and separate from preregistered analyses. Another challenge is simply that preregistration is time-consuming and can be challenging (Nosek et al., 2019). Finally, preregistration only works to (a) combat publication bias if researchers search for and include preregistered but unpublished research in their research syntheses, and (b) reduce questionable research practices if editors, reviewers, and other research consumers examine and hold researchers accountable for discrepancies between study plans and research reports. Yet, Claesen et al. (2019) reported most published preregistered studies they examined contained unexplained deviations from preregistered plans.

Applications to single-case and qualitative research

Although preregistration has been most commonly applied with group quantitative studies, researchers can also preregister other research designs, including single-case designs, qualitative research, meta-analyses, and systematic reviews (see Haven & Van Grootel, 2019; Johnson & Cook, 2019). Although single-case and qualitative studies often involve making decisions after data collection has begun (e.g., deciding to introduce an intervention according to participant baseline responding, determining questions to ask in a focus group depending on participants’ previous responses), researchers can preregister the process and criteria for making decisions and update the preregistration when decisions are made.

Guidelines

Before starting the preregistration process, it is important to plan out in detail the critical elements of one’s study (e.g., research questions and hypotheses, sampling plan, independent variables, dependent variables, study procedures, data analyses). After developing a research plan, researchers should select a registry and respond to the prompts and questions as completely as possible. If researchers do not know one or more aspects of a study in advance, they can preregister the criteria by which they will make such determinations. For example, a decision tree for determining which statistical analysis will be used depending on whether assumptions are met can be preregistered (Kiyonaga & Scimeca, 2019). Once the decision is made, it can be described in an update to the preregistration.

Two registries commonly used by special education researchers are the Open Science Framework (OSF; https://osf.io/prereg/) and the Registry of Efficacy and Effectiveness Studies (REES; www.sreereg.org). OSF is a multi-disciplinary registry with multiple templates for preregistering different types of studies. For example, OSF’s standard template, which can be used for group experiments, observational studies, and meta-analyses, asks researchers to report the study design, sampling plan, study variables, and analysis plan. The standard template provides a variety of multiple-choice and short answer questions for researchers to complete, provides examples, and prompts researchers to explain and justify their plans. OSF also provides preregistration templates for qualitative research and secondary data analysis, and allows researchers to save blinded copies of preregistrations, which can be cited in manuscripts submitted for blind peer review. A preregistration guide (OSF, n.d.) and a how-to video (Mellor et al., 2017) are also provided.

REES is designed for preregistering causal inference studies (i.e., randomized trials, quasi-experimental designs, regression discontinuity designs, and single-case designs) in education and related fields. When preregistering a study on REES, researchers respond to a series of questions and prompts in each of eight areas (i.e., general study information, description of study, research questions, study design, sample characteristics, outcomes, analysis plan, and additional materials). For example, when preregistering a multiple-baseline single-case design, researchers are prompted to specify type of multiple-baseline design (across participants, places, materials, or behaviors), start of baseline (concurrent or nonconcurrent, and for how many cases), type of participants (students, teachers, principals, schools, other), and total number of participants. Prompts for preregistering a study on REES are straightforward, with definitions and examples provided. A user guide (REES, n.d.-b), a how-to video (REES, 2020), and a checklist of required information for each design (REES, n.d.-a) are provided.

Once completed, preregistrations are time-stamped, assigned a unique digital object identifier (DOI), and made public. The published preregistration is permanent, but researchers can and should post updates describing and providing a rationale for any changes to preregistered plans. For example, researchers may need to deviate from preregistered plans if collected data do not meet assumptions for a planned analysis or if a school site requests a change to a planned intervention. Transparency is paramount and changes should be completed before data analysis is conducted. Researchers can update their preregistration on REES by navigating to “My Registry” and clicking “Update.” This will create a time-stamped copy of the updated registration that will also be made public. Researchers are provided additional space to justify or explain changes. When updating a preregistration on OSF, researchers must add a Transparent Changes Document (https://osf.io/yrvcg/) as a supplemental file on the overview page of the public registration. This document should note the changes to the study, rationale for the changes, and how the changes potentially affect the outcome of the study.

Benefits and limitations

Registered reports are designed to improve the transparency, quality, and credibility of research in at least three ways. First, reviewers of registered reports are able to provide constructive feedback before the study is conducted, when authors can make adjustments to their study. Second, registered reports reduce questionable research practices such as p-hacking by not only detailing research plans prior to conducting a study, as in preregistration, but also by devoting a stage of review to ensure that plans are adhered to. Finally, registered reports help combat publication bias. Because the decision to accept a study in principle is made before results are known, reviewer and editor decisions are not influenced by the direction of study findings. Indeed, engaging in unplanned, questionable research practices to obtain positive findings is one of the only ways for a study with in-principle acceptance not to be published. Null findings are, in fact, significantly more likely to be reported in registered reports compared with traditional studies (Scheel et al., 2020).

The same potential limitations noted for preregistrations apply to registered reports. An additional concern for registered reports is that (a) planning for and writing up Stage 1 manuscripts and (b) conducting two stages of review entail increased time and effort for researchers, reviewers, and editors. Second, not all research is appropriate for registered reports. For example, authors are unlikely to be able to write a prospective method section suitable for review for purely exploratory research.

Applications to single-case and qualitative research

Most published registered reports have used group quantitative designs. For example, all seven studies in Reich et al.’s (2020) special topic collection in AERA Open on registered reports in education used group quantitative designs. Yet, any study that can be preregistered, including single-case design, qualitative research, meta-analyses, and systematic reviews, can be a registered report.

Guidelines

First, researchers must select a study that is appropriate for registered reports (e.g., a study with primarily predetermined methods). It is also important to allot sufficient time for the Stage 1 review process when conducting school-based studies with fixed timelines to ensure that study plans are accepted in principle before the study must begin. Researchers must also identify a journal that accepts registered reports. Only a few journals in special education currently accept registered reports as a regular submission option (e.g., Exceptional Children, Gifted Education Quarterly; see https://www.cos.io/our-services/registered-reports for a list of journals accepting registered reports). We encourage interested authors to ask editors of other journals if they would accept a registered report submission to pilot the process.

Stage 1 submissions should include complete, but prospective, introduction and method sections. Journals accepting registered reports have their own guidelines for Stage 1 submissions that should be adhered to. Researchers need to clearly report (a) well-justified research questions and accompanying hypotheses, (b) a power analysis to justify the proposed sample (as relevant), and (c) reproducible methods in their Stage 1 manuscript. A cover letter in which authors describe the feasibility of conducting the study and the importance of study findings regardless of direction and magnitude often accompany Stage 1 submissions. For elements of the study that cannot be fully determined in advance, Kiyonaga and Scimeca (2019) recommended using decision trees in Stage 1 manuscripts to clarify criteria for determining how the study will be conducted.

The Stage 1 submission and review process is similar to typical peer review. When and if in-principle acceptance is granted, authors should preregister the accepted study plans, and then proceed to conduct and write up the study. Researchers should notify the editor if substantive deviations from approved plans will occur. Any deviations from accepted study plans need to be clearly noted and justified in the final manuscript. Any exploratory analyses conducted should be clearly reported as such. Stage 2 review typically is conducted by the same reviewers as in Stage 1, and focuses on whether accepted study plans were followed and deviations are noted and justified. See Kiyonaga and Scimeca (2019) for practical considerations and Registered Reports FAQ (n.d.) for responses to frequently asked questions regarding registered reports.

Benefits and limitations

Shared data expand the possibilities of what can be done with a data set, including allowing independent reanalysis and reproduction of published findings to assess their trustworthiness. In addition, allowing more researchers to analyze a data set promotes (a) examination of novel research questions and (b) application of diverse perspectives and techniques to robust research questions (Vision, 2010). Sharing data in a repository also allows data sets to be assigned a DOI and become a citable and reportable product for the data sharer. Furthermore, publications with shared data are cited more often than publications without (Piwowar & Vision, 2013). Given the typically resource-intensive nature of data collection in special education, open data also benefit other researchers by democratizing access to data and allowing them to analyze otherwise inaccessible data (Fleming et al., 2021; Mangul et al., 2019).

However, sharing high-quality data takes time and expertise: data must be checked, cleaned, and fully de-identified. For many investigators, these processes may fall outside of their expertise and necessitate a change in workflow. Resources are available to aid with sharing data (see van Dijk et al., 2021) and, for those who write grants, we recommend budgeting time for staff with expertise in data sharing. In addition, not all data can be shared (e.g., highly sensitive data that cannot be fully de-identified).

Applications to single-case and qualitative research

Data often can be shared in single-case and qualitative research. Researchers conducting qualitative research can use specialized data repositories for sharing their data (i.e., the Qualitative Data Repository [https://qdr.syr.edu/] for text-based data, and Databrary [https://nyu.databrary.org] for video-based data). Both repositories offer services to help curate and document data. Although values for dependent variables typically are shown in single-case studies in single-case graphs, greater transparency and accuracy can be achieved by openly sharing the actual values for each data point (Cook, Johnson, et al., 2021). In addition, single-case researchers can share data related to procedural fidelity, intervention intensity, reliability of the dependent variable(s), social validity, and visual analysis (e.g., trend and stability calculation results); as well as their code for calculating effect sizes. Because single-case and qualitative studies often have few participants, researchers should take extra care de-identifying their data to protect participants’ identities.

Guidelines

The first step to sharing data is to check participants’ informed consent. Given the new Common Rule for Protection of Human Subjects (U.S. Department of Health & Human Services, n.d.), de-identified data are considered not human subjects if researchers did not explicitly state what would happen to participant’s data. Thus, if data sharing is not mentioned in informed consent, it is possible that data may be shared. We recommend checking with one’s institutional review board (IRB) for guidance. If informed consent indicates data will not be shared, researchers can request a waiver of consent from the IRB for sharing de-identified data. Researchers planning to share their data should design consent agreements and research protocols to facilitate data sharing. As appropriate, we recommend using language permitting sharing of de-identified data in informed consents (see Shero & Hart, 2020, for templates for informed consent that facilitate data sharing).

The second step is cleaning and de-identifying the data. This involves checking for data entry mistakes (e.g., out-of-range values), creating any necessary summary scores (e.g., sum scores, developmental scale scores), and checking for identifiable variables (e.g., names, addresses) and cells (e.g., labeling of low-incidence disabilities). Although most available guidance on data de-identification involves Health Insurance Portability and Accountability Act (HIPAA)-related data in medical settings (see U.S. Department of Health & Human Services, 2012), a general rule of thumb is to ensure obvious identifiers are removed (such as name, exact date of birth, address), and then consider the frequency of the knowable characteristics of that participant to the region the data were collected. For example, if a participant is a child with a rare genetic disorder, that child is likely identifiable when coupled with the setting of the study, and caution is warranted in sharing that data. However, a participant who is a male with a learning disability in a study conducted in a large city is likely not identifiable from these variables, which can therefore be shared. See Edwards and Schatschneider (2020a, 2020b) for guides to de-identifying education data and Meyer (2018, pp. 133–135) for other considerations when de-identifying data.

The third step is creating metadata. Researchers need to create documentation providing information about the study, the sample, protocols, and measures, as well as create a data dictionary that details missing data values, variable names, variable labels, recoding strategies, questionnaire items, scoring logic, and the like. The fourth step is selecting an appropriate data repository. There are many data repositories, including domain general (e.g., Inter-university Consortium for Political and Social Research [ICPSR], OSF) and domain-specific (e.g., LDbase, Databrary) repositories, and researchers should consider where the data will be best discoverable by their research community. It is important that the repository provides a DOI for the data and the option to assign a copyright license to the data. Most repositories give researchers an option of an Open Data Commons (https://opendatacommons.org/) or Creative Commons (https://creativecommons.org/) license, which will tell data users how they can use and cite the data and data products.

The final step is to upload the data and metadata into the data repository and set an access level, if applicable. Some data repositories allow researchers to choose if the data will be posted openly or be available only by managed access, which requires data reusers to apply for permission to access the data. This may be a good choice for special education researchers who have sensitive data or other concerns, as they can require a data reuser to have an active IRB protocol before accessing the data, as an example. For more information on open data, the open data handbook (https://opendatahandbook.org/) hosts a wealth of information. Other resources to help researchers prepare their data to be shared include Gilmore et al. (2018) and ICPSR (n.d.).

Benefits and limitations

Making study materials open is beneficial for multiple stakeholders. By sharing study materials, researchers increase their impact by allowing others to use and build upon their work. These products can be listed on authors’ CVs and can garner citations. Open materials also provide benefits to the field and other special education stakeholders by (a) facilitating replication of previous studies, which is essential to establish evidence-based practices in special education, and (b) providing researchers, practitioners, parents, and other stakeholders access to free, research-validated materials to use with students.

Copyright concerns are the main obstacle to materials sharing. Authors cannot share material, such as a published norm-referenced assessment, for which they do not own the copyright, even if they used the materials in a study. Yet, they can share study material, such as a treatment-integrity protocol, that they developed for the project and own (original work is under the creator’s copyright once it is shared in a fixed and tangible form; U.S. Copyright Office, 2019). Unfortunately, determining ownership is not always straightforward. For example, it may be unclear whether and how one can share adapted versions of someone else’s copyrighted materials, materials developed as part of one’s professional duties at a university, and materials developed from work funded by external sources (e.g., grants). In addition, like other open practices, materials sharing requires time and planning (van Dijk et al., 2021).

Applications to single-case and qualitative research

For researchers conducting single-case studies, authors can share materials such as observation coding forms, intervention protocols, intervention materials, social validity measures, and personnel training protocols. Qualitative researchers can share materials such as interview and probing question protocols, field notes/observation data collection forms, positionality and reflexivity statements, data analytic strategies and detailed descriptions of the data analytic process, and deductive or inductive codebooks with supportive interview quotes or field notes/observations. The process for sharing materials is the same regardless of study design.

Guidelines

First, decide what materials to share. We encourage authors to share all relevant materials possible and provide detailed descriptions in the manuscript of material they are unable to share. Second, check for any potential copyright or intellectual-property restrictions that might preclude legally sharing the material. We suggest researchers check with their funding agencies, employers, and university copyright lawyers (often employed through a university’s library system) regarding sharing researcher-created materials. When seeking to share others’ materials they have adapted, authors must ensure the materials have a license permitting sharing and adaptations. Any materials created from others’ open work should be properly attributed, cited, and licensed according to the permissions granted by the original creator. Some materials are licensed by their original authors in a manner that allows for adaptations but does not permit sharing of the adapted materials (i.e., the original authors retain exclusive rights to redistribute their materials). In these instances, we recommend sharing a detailed account of the adaptations made to the original materials with proper attribution to the original authors.

Third, format materials to facilitate easy access and reuse. Open materials should be saved and shared in an easily accessible format such as .txt or .html that allows reusers to edit the materials without the need for atypical software or expertise in editing (Hilton et al., 2010). Authors should also provide relevant instructions for using shared materials. For example, authors might include a coding guide alongside a shared observational protocol or the implementation steps and procedures for a training manual.

Fourth, determine and apply a copyright license for sharing in consultation with everyone who played a role in creating the material. Creative Commons (https://creativecommons.org) is a widely used provider of public copyright licenses for sharing content with varying degrees of openness. Two popular licenses are CC BY and CC BY-NC. The CC BY license allows others to reuse, modify, edit, build upon, and distribute shared materials as long as attribution is given. The CC BY-NC license restricts such reuse to noncommercial purposes. Once selected and shared, licensing options on materials cannot be changed, so care should be taken when selecting the license. The license should be indicated clearly on the shared materials in a machine-readable format. Creative Commons provides copyright license images that can be copied and pasted into the shared document or included as a watermark. Alternatively, authors can indicate the license within a header or footer by using the copyright symbol (©) followed by their names, the year, and the license (e.g., CC BY-NC). See Figure 1 and online supplemental Table 1, which are shared on OSF, for example.

Fifth, decide whether to share materials on a data repository or as an online supplement to an article. We recommend selecting the option that makes the material best available to one’s target audience. For example, if a researcher is sharing an intervention protocol that is intended primarily to be shared with practitioners, and they are publishing in a practitioner journal read by many teachers, sharing through the journal may be the best choice. Alternatively, if the article is being published in a research journal read by few practitioners, a data repository may be advisable. Note that some repositories allow sharers to select the level of availability of their shared materials (e.g., only shared after the owner approves an application), which may be an attractive option. Finally, upload one’s materials with the chosen licensing level, and add the shared materials and associated DOI, if available, to one’s CV.

Benefits and limitations

A fundamental benefit of OA publishing is democratization of access to research and scholarship (Fleming et al., 2021). Providing special education practitioners, for example, access to research can help bridge the gap between research and practice. Preprints can also speed dissemination of research by allowing authors to immediately share their research on preprint servers, personal websites, social media, or institutional repositories without having to wait for what can sometimes be a lengthy process of peer review. OA publishing is also associated with increased impact of research; articles posted as preprints before publication receive more citations, downloads, and social media attention (Fu & Hughey, 2019; Piwowar et al., 2018). Finally, preprints can help combat publication bias by providing a forum for dissemination of studies that might not otherwise be published (e.g., studies with null results).

In terms of limitations, gold and hybrid OA can be costly for researchers, as the average article-processing charge for special education journals is just below US$3,000 (Fleming & Cook, 2021). It is also important to recognize preprints are not typically peer reviewed. Thus, seriously flawed and misleading research can be posted as preprints, and research consumers should critically evaluate preprinted research for potential bias. In addition, preprints can unmask the identities of study authors, thereby potentially reducing the pool of available blind peer reviewers if and when the paper is submitted for publication (Fleming et al., 2021).

Applications to single-case and qualitative research

Single-case and qualitative research, as well as any other type of scholarship, can be made openly accessible through gold, hybrid, bronze, and green OA. OA operates in the same way regardless of the type of research.

Guidelines

Journal OA policies vary. For example, some journals do not accept submissions that have been preprinted, and policies differ on whether, where, and when papers can be postprinted (Fleming & Cook, 2021). Researchers should review OA policies of journals to which they may submit a manuscript as early as possible in the research process. Most journals with academic publishers have posted OA information on their websites. For some small or independent journals, researchers may need to contact editors to obtain OA policies.

As with shared data and materials, authors must choose their copyright license when publishing OA. For all OA models except bronze, authors should select a copyright license to clarify how others can reuse and adapt the work. For preprints and postprints, print repositories often require selection of a copyright license. Similarly, for gold and hybrid OA, authors will typically choose a copyright license as part of the publication process.

For green OA, authors must select a repository. Repositories can be general (e.g., OSF Preprints, https://osf.io/preprints/) or discipline specific (e.g., EdArXiv, https://edarxiv.org). Researchers typically must create an account at the selected repository, select a copyright license, and upload the manuscript. When submitting to EdArXiv, for example, authors upload their unformatted manuscript (typically a pdf file) and answer a series of questions. EdArXiv requests the following information about submissions: title, abstract, license for copyright, key words, discipline and subdiscipline(s), author(s), statement on conflict of interest, supplemental materials, a link to the study’s publicly available data (if applicable), a link to the study’s preregistration (if applicable), and the DOI to the corresponding article if the manuscript is already published.

Print repositories typically do not peer review manuscripts but often screen them for appropriateness (e.g., content aligns with focus of discipline-specific repositories) before posting. If and when a preprinted manuscript is submitted and accepted for publication in a journal, authors should add an updated version of the paper with an author note indicating acceptance and specifying the journal. After publication of a paper, authors should update the preprint with the corresponding citation and link DOIs of the preprint and published paper. If the journal in which an article is published allows postprints, authors may update a preprint with a postprint to reflect changes made during peer review. Many journals impose an embargo period of 1 or 2 years before authors can postprint a manuscript accepted for publication.

As an example of how this process can work, two of the authors concurrently submitted a recent manuscript to EdArXiv (Fleming & Cook, 2020) and to a journal for publication. Because the journal allowed postprints without an embargo period, we updated our preprint following each round of peer review. Following acceptance, we uploaded the accepted version of the manuscript with an author’s note that indicates the journal in which the paper is accepted. Researchers can also share preprints and postprints on social media, personal websites, and institutional repositories (Laakso, 2014). See Fleming and Cook (2021) for a review of OA policies of special education journals, and Sherpa Romeo (https://v2.sherpa.ac.uk/romeo/) to search journal OA policies.