Abstract
Replication research is essential to scientific knowledge. Reviews of replication studies often electronically search for replicat* as a textword, which does not identify studies that replicate previous research but do not self-identify as such. We examined whether the 83 intervention studies published in six non-categorical research journals in special education in 2013 and 2014 might be considered replications regardless of using the term replicat* by applying criteria related to (a) the stated purpose of the study and (b) comparing the findings of the study with the results of previous studies. We coded 26 intervention studies as replications. Authors of 17 of these studies reported that their findings solely agreed with the results of the original study(ies). Author overlap occurred for half of the replicative studies. The likelihood of findings being reproduced did not vary as a function of author overlap. We discuss implications and recommendations based on these findings.
Although a variety of factors shape special education practice and policy (e.g., advocacy, funding), science has and continues to play an important role in the field (Chard, 2004; Kauffman, 1999). Indeed, recent legislation (e.g., Individuals With Disabilities Education Act, 2004; No Child Left Behind, 2002) and evidence-based reforms (B. G. Cook & Odom, 2013; Odom et al., 2005) highlight the use of scientific research to guide training and instruction in special education. The influence of science is generally embraced in the field because the scientific method is distinguished from other approaches to knowledge generation (e.g., authority, personal experience, intuition) by “experiments that confirm or deny ideas . . . and in its willingness to abandon ideas that have been found wanting” (Sagan, 1997, p. 252). As such, science is generally regarded as the most valid approach for, among other things, determining the effectiveness of instructional practices.
Experimental research is designed to control for common threats to validity in determining whether a practice is effective for study participants (L. Cook, Cook, Landrum, & Tankersley, 2008). Nonetheless, many factors (e.g., low power, measurement error, chance, researcher bias), some of which researchers may not be able to control or even be consciously aware, exist throughout the research process that can markedly decrease the validity of study results (see B. G. Cook, 2014). Consequently, Ioannidis (2005b) theorized that the results of most research studies might in fact be false. It is perhaps not surprising, then, that the history of special education, like other fields, is strewn with examples of study findings that subsequent research showed to be invalid. In one infamous example, Wakefield et al.’s (1998) study suggesting that vaccines caused autism was exposed as flawed by subsequent “studies consistently [finding] no evidence of a link between the MMR vaccine and autism” (Godlee, Smith, & Marcovitch, 2011, p. 64). In this and many other examples, the scientific method worked as it is supposed to: Researchers independently re-examined research findings to shed light on their validity.
As such examples demonstrate, replication is critical to the scientific enterprise (Francis, 2012; Open Science Collaboration, 2015; Popper, 1959). As Schmidt (2009) noted, To confirm results or hypotheses by a repetition procedure is at the basis of any scientific conception. A replication experiment to demonstrate that the same findings can be obtained in any other place by any other researcher is conceived as an operationalization of objectivity. It is the proof that the experiment reflects knowledge that can be separated from the specific circumstances (such as time, place, or persons) under which it was gained. (p. 90)
Despite a common understanding among researchers as to the general meaning and importance of replication, definitions and classifications of replication are varied (Schmidt, 2009). Schmidt suggested that replications can be grouped into two basic types: (a) direct replications, in which researchers try to duplicate as exactly as possible the methods used in a previous study, and (b) conceptual replications, in which researchers change one or more aspects of a previous study. Thus, if researchers duplicate all aspects (e.g., sample population, setting, independent variable, outcome measure[s]) of a previous study, it is considered a direct replication; if one or more of these elements are varied (e.g., examining whether an intervention is similarly effective on a given outcome for a different population of learners in a different setting), it is a conceptual replication. Direct replications directly assess the reproducibility of the findings of previous research; if valid, results of a previous study should be reproduced when researchers replicate their methods. Alternatively, conceptual replications serve to explore the generalizability of the findings of the original study (e.g., does the intervention work with middle school students too?). However, conceptual replications do not directly assess the validity of a previous study. Failure to reproduce findings in a conceptual replication does not necessarily cast doubt on the results of the original study because the differences between the studies may explain any discrepancies in findings (e.g., different populations respond differently to the same intervention; Travers, Cook, Therrien, & Coyne, 2016, In this issue).
Given the critical role of replications in research, it is surprising how little attention has been paid to them in the scientific literature until recently (Nosek, Spies, & Motyl, 2012). Researchers have begun to examine the rate at which replication studies are conducted in fields related to special education. For example, Makel, Plucker, and Hegarty (2012) reported that only 1.1% of articles published in the top 100 psychology journals were replications (direct or conceptual). Makel and Plucker (2014) reported even lower rates of replications (direct or conceptual) among articles in the top 100 educational journals (0.1%). Moreover, when replications are conducted, findings frequently differ from those reported in the original research (Ioannidis, 2005b). In the largest examination of reproducibility to date, the Open Science Collaboration (2015) conducted 100 independent, direct replications in the field of psychology. Whereas 97 of the original studies reported significant results, the results of only 36 of the replications were statistically significant.
Author overlap (i.e., when one or more individuals authored both the original and replication studies) may further impede the role of replication research in advancing scientific knowledge. If the same researchers conduct both the replicated and replicating studies, they may infuse both studies with the same biases and be reluctant to publish findings that contradict their previous research. Indeed, replications with author overlap have been found to be significantly more likely to successfully reproduce the findings of the studies being replicated than replications without author overlap (Makel & Plucker, 2014; Makel et al., 2012). Issues such as low rates of replication studies, findings not consistently being reproduced in replication studies (indicating a high level of false-positives in the scientific literature), and high frequency of author overlap in replications are so consequential that some scholars have suggested they represent a crisis that threatens the self-correcting nature of science (Ioannidis, 2012; Pashler & Wagenmakers, 2012).
Although B. G. Cook’s (2014) cursory review indicated that replication studies are rare in special education, prior to this special issue, we are unaware of any systematic reviews examining issues related to replication in the special education research base. Thus, it is unknown whether problems related to replication demonstrated in other fields exist in special education. In this special issue, we undertook four independent reviews of replication in special education to address this gap in the knowledge base. Like many of the reviews in related fields (e.g., Makel & Plucker, 2014; Makel et al., 2012), Lemons et al. (2016, In this issue) and Makel et al. (2016, In this issue) identified replication studies using electronic searches for the keyword replicat* (i.e., variations of replicate). This search procedure enables researchers to efficiently search through scores of journals. However, this approach may under-estimate the prevalence of replications in that it only detects studies that self-identify as replications. Although researchers may value and conduct replication research, they may not explicitly refer to their work as such because replications are perceived as (a) less likely to garner professional notoriety and institutional rewards than novel discoveries (Nosek et al., 2012), (b) less likely to be published by journal editors and reviewers (Neuliep & Crandall, 1990, 1993), and (c) hostile attempts to refute the work of one’s colleagues (Koole & Lakens, 2012). Thus, the primary purpose of our review was to examine the prevalence of replication research in special education regardless of whether the authors used certain terms, such as “replicate,” to describe their studies.
Given the lack of commonly accepted, operationalized criteria for replications (Schmidt, 2009), we identified replications from the perspective of a consumer of special education research. We reasoned that most research consumers do not search for, locate, and examine other studies to determine whether and how a study duplicated the methods of a previously conducted study. Rather, when reading an intervention study, it seems reasonable that a research consumer could consider a study as replicative if it meets two criteria: (a) When discussing the purpose of their study, the authors state that they used one or more previous study(ies) as a basis for their study, and (b) when discussing the results of their study, the authors compared their findings with the results of the study(ies) being replicated. Using this functional approach for identifying replications, we also sought to determine the agreement rate between findings of original and replicating studies. Furthermore, we examined the prevalence of author overlap in replication studies in special education research and the association between author overlap and the agreement of findings between original and replication studies. Finally, we also examined the association between prevalence of replications and research design. Because of the relatively small number of participants, more replications of single-case studies are required to establish evidence-based practices (e.g., B. G. Cook et al., 2015; Horner et al., 2005). For example, the Council for Exceptional Children (2014) requires that, whereas evidence-based practices in special education must be supported by only two high-quality group experiments, they must be supported by five single-case studies. It is possible, then, that more replications might be conducted using single-case research designs.
The specific research questions that guided our review are as follows:
Method
To address our research questions, we identified intervention studies from non-categorical research journals in special education and then coded those studies for elements related to replication. In this section, we describe our search and coding methods as well as data analysis.
Identifying Non-Categorical Research Journals
Because our search entailed reading and coding each identified study to determine whether it was a replication, it was not feasible for us to examine the large volume of articles that Makel et al. (2016) and Lemons et al. (2016) did in their electronic searches. To avoid biasing our review by focusing on an area of special education scholarship that might publish more or fewer replications than others, we targeted research journals in special education without a categorical focus. We defined non-categorical research journals using two criteria: The journal’s title, description, mission, aim, and/or scope (as stated on the journal’s website) (a) indicated the publication of research and (b) did not emphasize a specific sub-population of learners with or at risk of disabilities (e.g., learning disabilities, severe disabilities, young children) or a particular outcome area (e.g., reading, writing). We applied these criteria to the 36 special education journals included in the Journal Citations Report, 2013 Social Science Edition and identified six journals for inclusion: Exceptional Children; Exceptionality; International Journal of Disability, Development and Education; The Journal of Special Education; Journal of Positive Behavior Interventions; and Remedial and Special Education.
Identifying Intervention Studies
With the exception of book reviews and editors’ commentaries, we examined all articles (N = 289) published in these six journals in 2013 and 2014 to identify intervention studies. Although non-intervention studies can be replicated, we targeted intervention studies because they are the primary source of information regarding the effectiveness of instructional techniques for students with and at risk of disabilities. The first three authors coded articles to identify intervention studies. We randomly assigned each of the 289 articles to two authors to code as an intervention study or other publication. We defined intervention studies as original, empirical research that investigated the effect or interaction of an independent variable that was implemented under the control of researchers on a specified outcome. We excluded purely descriptive, correlational, and “natural experiment” studies from the review. All articles that met the criterion, regardless of the disability status or age of participants, were included (e.g., we included studies that focused on adult participants such as teachers or parents). We identified 83 articles as intervention studies with an inter-rater agreement rate of 97.6%. As with all inter-rater agreement levels less than 100%, we reconciled disagreements through discussion between all authors until agreement was reached (see Table 1 for the number of intervention studies identified from each journal).
Number of Articles, Intervention Studies, and Replications Published 2013–2014, by Journal.
Identifying Replication Studies
We used two criteria to classify studies as replications (either direct or conceptual): Did the authors (a) indicate that they are replicating a previous study(ies) when discussing the rationale/purpose of the study (i.e., in the Introduction, Method, or both) and (b) compare the findings of the current study with the results of the study(ies) being replicated when discussing the findings of the current study i.e., in the Results, Discussion, or both sections? See Figure 1 for a graphical overview of these coding procedures.
Coding procedures for replication studies.
We were purposefully liberal when coding “replication statements” (i.e., a statement indicating that a purpose of the study was to replicate, extend, further investigate, or use as its basis one or more previously conducted studies), as the intent of our review was to identify all intervention studies that research consumers might reasonably conclude were replications, rather than identify a particular type of replication study (e.g., direct replication). We used the following definition to guide our coding of replication statements: A replication statement indicates that the study purposefully replicates, extends, further investigates, or uses as its basis one or more previously conducted studies. Indicating that one or more gaps exist in one or more previous studies (e.g., not addressing a particular outcome area or learner population) and then noting in the same paragraph or in consecutive sentences in separate paragraphs that the present study addresses the gap(s) constitutes a replication statement. The source for a replicated study must be a previously conducted study (i.e., a review or meta-analysis cannot be replicated).
Merely indicating that an intervention was based on previous research was not considered sufficient for a replication statement because we felt that a replication statement should allude to the larger purpose of the study. For example, although O’Keeffe, Slocum, and Magnusson (2013) cited multiple studies in providing a context and basis for their study, their purpose statement did not reference previously conducted research: The purpose of this study was to examine the effects of a fluency training package on the presentation rate, praise rate, and error correction accuracy of paraprofessionals presenting a research-based reading intervention to students who were at risk for reading disabilities within an RTI system. (p. 16)
Thus, we coded this study as not having a replication statement. Study authors sometimes indicated that a purpose of their study was to replicate a research base or previously conducted research without citing specific studies. We coded such statements as replication statements if the authors had previously cited specific studies that constituted or were part of that research base. When we coded a study as containing a replication statement, we recorded the previously conducted studies referred to in the replication statement. Inter-rater agreement for 30 double-coded intervention studies for including a replication statement was 90.0%.
We then read the Results and Discussion sections of each article and coded whether findings from previous research were noted and compared with the results of the current study. Typically, when authors noted the findings of one or more previously conducted studies, they did so explicitly (e.g., citing a previously conducted study and describing its findings). However, in some instances, it was difficult to discern whether authors were noting the findings of previous research. For example, it was not unusual to find these types of statements in the Discussion sections of research reports: Reading outcomes have been improved by phonics-based training (Author, year). In these cases, we considered that findings of previous research had been noted when we could infer (a) an intervention (e.g., phonics-based training), (b) a dependent variable (e.g., reading outcomes), and (c) findings (improved outcomes) from previous research. When authors noted the findings of a previously conducted study in the Results or Discussion section, we coded whether they compared those findings with the results of the present study. We used the following criterion to guide our coding of comparison of findings: Findings from previous research are considered to be compared to findings from the current study when the authors use language that indicates comparing or contrasting findings (e.g., as in, similar to, in contrast to) or when findings from previously conducted research and current study are noted in the same or consecutive sentences (e.g., Author A (2000) reported positive effects of the intervention. In this study, we found no effects.). Noting findings from the current study and a previous study in different places (e.g., in different sections of the manuscript, in different paragraphs) does not constitute a comparison of findings.
Inter-rater agreement for 30 double-coded intervention studies for results of previous studies being noted and compared with findings of current study was 90.0%.
For a study to be coded as a replication, the study must meet both criteria (i.e., replication statement, findings of previous research noted and compared with results of current study) with regard to one or more common “parent” studies (i.e., studies being replicated). For example, a study would be coded as a replication if the authors referred to Study A in a replication statement (e.g., stated that a purpose of the current study was to extend Study A) and compared the findings of the current study with the findings of Study A in the Discussion section (e.g., indicated that similar to Study A, the current study found that the intervention was effective). If only one condition was met (e.g., the authors compared their findings with the results of Study A in the Discussion section, but did not indicate that a purpose of the current study was to replicate Study A) or if both criteria were met, but only with regard to different parent studies (e.g., the authors stated that a purpose of the current study was to extend Study A, but compared the findings of the current study only with Study B), then we did not code the study as a replication. Inter-rater agreement for 30 double-coded intervention studies on replication status was 90.0%.
We then coded whether identified replication studies were direct or conceptual replications. To be coded as a direct replication, the author(s) of the study must have clearly stated that the participant sample, setting, intervention, and outcome measure(s) were the same as a parent study. If authors indicated that one or more study elements differed from replicated studies or it was unstated/unclear whether all study elements were the same as a replicated study), we coded the replication as conceptual. Inter-rater agreement for 30 double-coded intervention studies for type of replication was 100%.
Coding for Research Design, Author Overlap, and Agreement in Findings
We coded all 83 intervention studies for research design. We defined group studies as group experimental and quasi-experimental studies that examined the effects of an intervention using inferential statistics to evaluate the performance of one or more groups of participants. We defined single-case studies as any study using a traditional single-case design (e.g., reversal/ABAB, multiple baseline, multiple-probe, changing criterion, alternating treatment/multiple element) in which the individual case serves as its own comparison. We coded studies that fit into neither the group nor the single-case research categories (e.g., qualitative studies, case studies, studies that used both group and single-case designs) as “other.” We coded mixed-methods studies that used a group or single-case design in addition to qualitative methods according to the quantitative design used. Inter-rater agreement for 30 double-coded intervention studies for research design was 93.3%.
For replication studies, we coded whether any of the authors of the study also authored or co-authored one or more of the studies being replicated (i.e., parent studies). Coding was dichotomous, indicating whether any author overlap occurred between the current study and any of the parent studies (many replication studies had more than one parent study). Inter-rater agreement for 30 double-coded intervention studies on author overlap was 93.3%. Disagreements related to author overlap occurred when coders disagreed on which studies (and, therefore, study authors) were parent studies for the article being coded.
Finally, we coded whether and how authors of the current (replication) study reported that their findings agreed with the results of parent (replicated) studies. We used four options to code for agreement in findings:
It is unclear whether comparison(s) indicate(s) agreement or disagreement between replicated and current studies
Findings of replicated and current studies compared and reported only to agree
Findings of replicated and current studies compared and reported only to disagree
Findings of replicated and current studies compared and reported to have mixed agreement/disagreement
The second and third coding options were only used when all comparisons between the findings of current and all parent studies indicated agreement or disagreement, respectively. We used the fourth coding option to indicate when findings of the current study were compared with one or more parent studies and reported to agree in some ways but disagree in other ways (e.g., findings between studies agreed on one outcome variable but not another). Inter-rater agreement for 30 double-coded intervention studies on reported level of agreement in findings was 83.3%. The majority of disagreements were due to coders differing on whether statements indicated solely agreement or a mix of agreement/disagreement. For example, coders disagreed on this statement from Bottge et al. (2014): “The previous version of this intervention resulted in similar improvements on the problem solving measures (Bottge, Rueda, Grant, Stephens, & LaRoque, 2010), but this is the first time that students made such large gains in fraction computation” (p. 434). Ultimately, we discussed this item and decided it should be coded as mixed agreement/disagreement because the authors noted a clear difference (i.e., larger gains) in their results in comparison with the findings of a study being replicated.
Coding Procedures and Statistical Analyses
Prior to coding the 83 intervention articles, we (the four co-authors) calibrated our coding by collaboratively evaluating three intervention studies from special education journals not included in this review. From this practice coding, we refined our coding criteria and generated examples and non-examples for various coding options. We coded intervention studies one journal at a time. We randomly assigned each intervention article to a co-author for coding. We also randomly assigned at least one third of the intervention articles from each journal to a second co-author for double coding to assess inter-rater agreement. After we finished coding articles for a journal, all four co-authors examined and resolved any coding discrepancies as a group before proceeding to the next journal. A total of 36% of the intervention studies (n = 30) were double coded by two randomly assigned co-authors. Inter-rater agreement was calculated using the formula (Agreements / [Agreements + Disagreements]) × 100. To examine whether (a) agreement in findings differed as a function of author overlap in replication studies and (b) prevalence of replications differed as a function of research design (i.e., group studies and single-case designs), we conducted chi-square tests of independence using alpha levels of .05.
Results
Rate of Replication Studies
Of the 83 intervention studies published in 2013–2014 in six, non-categorical research journals in special education, (a) 53% (n = 44) indicated that a purpose of the study was to replicate, expand, extend, add to, or use as their basis one or more previous studies in the Introduction or Method section and (b) 73% (n = 61) of the studies included a comparison of the findings of previously conducted research with the results of the current study in the Results or Discussion section. Overall, we categorized the 31% (n = 26) intervention studies that addressed both of these criteria in relation to one or more common parent studies as replications. In none of the replications did authors specify that they attempted to duplicate all aspects of a previous study; therefore, we considered all identified replications as conceptual replications and none as direct replications. In the 26 replication studies, the number of parent studies (i.e., previously conducted studies that authors both referred to in a replication statement and to which they compared the findings of the current study) ranged from one to four. Specifically, 10 of the replication studies had one parent study, four had two parent studies, nine had three parent studies, and three had four parent studies.
Forty-four of the intervention studies included a replication statement, compared current findings with the results of previous studies, or both, but were nonetheless not classified as replications. Nine of these studies included a replication statement but did not compare findings of their study with previous research; 27 studies compared their findings with the results of previous research but did not include a replication statement; and eight studies both included a replication statement and compared their findings with the results of previous studies, but compared their findings with different previously conducted studies than the studies the authors referred to in the replication statement.
Agreement in findings
Of the 26 replication studies, authors of 17 studies (65%) solely indicated that their findings agreed with the findings of all parent studies. Authors of five studies (19%) noted findings both agreed and disagreed with findings of parent study(ies), authors of two studies (8%) solely noted their findings disagreed with parent study(ies), and the comparison was unclear in two studies (8%).
Author Overlap
Author overlap occurred for half (13/26) of the replication studies (see Table 2 for a breakdown of reported agreement between replicated studies and their parent studies by author overlap status). Agreement of findings did not differ as a function of author overlap, χ2(1) = 0.17, p > .05 (we combined all comparisons that did not solely indicate agreement for this analysis because of low n).
Reported Agreement Between Findings of Replicated Studies and Their Parent Studies, by Author Overlap Status.
Research Design
Of the 83 intervention studies, 46 were single-case designs, 31 group studies, and six “other” (e.g., qualitative studies, case studies, studies using both group and single-case methods). Of the 26 replication studies, 42% used a group design and 58% used single-case designs. Of the non-replication intervention studies that used group or single-case designs, 39% used group designs and 61% used single-case designs (see Table 3). The distribution of replication and non-replication intervention studies did not differ significantly as a function of research design, χ2(1) = 0.68, p > .05 (we did not include studies using “other” designs in this analysis because of low n).
Number of Intervention Studies Using Group, Single-Case, and Other Designs, by Replication Status.
Discussion
Using a two-part definition of replication, we found that 31% of intervention articles published in six non-categorical research journals in special education in 2013 and 2014 were replications. This rate is considerably higher than replication rates in special education reported by Lemons et al. (2016; 0.4%) and Makel et al. (2016; 0.5%), but considerably lower than the six of eight (75%) intervention studies in special education found to be replicated by Therrien, Mathews, Hirsch, and Solis (2016, In this issue). We propose two primary reasons for these differences: types of publications considered and procedures used to identify replications.
One variable that helps to explain the variability in replication rates between reviews is the type of publications examined. Lemons et al. (2016) and Makel et al. (2016) calculated replication rate using all articles, including non-empirical publications, as the denominator. In contrast, we (and Therrien et al., 2016, who also reported a high replication rate) considered only intervention studies when determining replication rate. Non-empirical publications, which comprise a sizable portion of the special education literature (see Mastropieri et al., 2009), cannot be replicated. Indeed, the 26 replication studies we identified represent only 9.0% of all publications in the journal issues we examined.
The definitions and procedures used to identify replications represent a fundamental difference in the reviews in this special issue that likely accounts for considerable variability in estimated replication rates. Lemons et al. (2016) and Makel et al. (2016), who identified the lowest proportion of replication studies, searched electronically for replicat*. Any replication studies that did not use the word replication were, therefore, not identified. In contrast, we read all intervention studies to identify critical elements of replications and identified studies as replications regardless of whether authors referred to them as replications. In fact, only six of the 26 studies we identified as replications used a version of “replicate” in their replication statements. Thus, if we only considered studies that used replicat*, we would have identified only 2.1% of the 289 articles as replications, a rate not dissimilar to that reported by Lemons et al. and Makel et al. In contrast to Lemons et al. and Makel et al., Therrien et al. (2016) found that the majority of intervention studies they examined were replicated multiple times when they identified replications by examining similarity between the interventions used in parent and child studies, regardless of whether the child studies indicated that they replicated the parent study. Thus, special education researchers appear to regularly conduct what we identified as replications without explicitly referring to them as such. This is consistent with the observation that “researchers who conduct . . . replications are looked down on as bricklayers and not advancing knowledge” (Makel et al., 2012, p. 537). When reporting research that builds on other studies, researchers may elect to not highlight the replicative aspects of their studies due to perceptions that replications are not prestigious, difficult to publish, and hostile to previous research (e.g., Koole & Lakens, 2012; Nosek et al., 2012).
The scarcity of direct replications was a consistent finding across the reviews in this special issue. We identified no direct replications among 289 articles published in six non-categorical research journals in special education. Therrien et al. (2016) reported that none of the 39 replicative “child” studies in their review were direct replications. And although Makel et al. (2016) and Lemons et al. (2016) identified some direct replications, the overall rate of direct replications was very low—0.24% and 0.02% of all articles, respectively. Similar to the findings of Therrien et al. and Lemons et al., we found that special education researchers conducted conceptual replications at a much higher rate than direct replications. In contrast, Makel et al. reported a similar number of direct and conceptual replications.
When comparing their findings with the results of parent studies, we found that the authors of most replicative studies (17 of 26) solely reported that findings agreed. This is consistent with the findings of the other reviews in this special issue (Lemons et al., 2016; Makel et al., 2016; Therrien et al., 2016), but contrasts with reports from medicine (e.g., Ioannidis, 2005a; Prinz, Schlange, & Asadullah, 2011) and psychology (Open Science Collaboration, 2015) that results are often not reproduced by replication studies. It is important to note that low rates of reproducibility reported in other fields were derived from direct replications, whereas the high agreement rate in our review was derived primarily from conceptual replications. As such, it appears that replications in special education typically indicate that previously reported effects of interventions tend to generalize (e.g., to other populations of learners, to other outcome areas, when changes are made to aspects of the intervention) rather than demonstrate that previously reported studies have been directly replicated. It is possible that the high level of agreement between child (conceptually replicating) and parent (replicated) studies is due to the potency and generalizability of interventions being researched in special education, publication bias, selective reporting of findings, or all of these factors. For example, special education researchers may value agreement in findings between parent and child studies (e.g., to establish a practice as research-based across multiple studies)—and therefore may choose not to publish replication studies with discrepant findings or to selectively focus on reporting agreement of findings when discussing their results.
Implications for Policy and Practice
Our findings suggest that special education researchers conduct conceptual replications at a higher rate than reported by larger reviews that identified only studies that used the term replicat* (Lemons et al., 2016; Makel et al., 2016). Given that standards in the field require evidence-based practices be supported as effective by multiple studies (e.g., Gersten et al., 2005; Horner et al., 2005), this finding bodes well for the field’s continuing efforts to identify evidence-based practices for learners with and at risk of disabilities. Indeed, the growing number of meta-analyses and evidence-based reviews being conducted in special education would not be possible without the existence of multiple conceptual replications. However, because authors of replication studies seldom explicitly identified replication as a purpose of their study, the replicative nature of many of these studies may go unrecognized by research consumers outside the context of research syntheses.
As much as the identification of conceptual replications in our review is encouraging, the lack of direct replications is alarming. Direct replications are uniquely well-suited for verifying the validity of research findings (Simons, 2014) on which theory, policy, and practice are based. The scarcity of direct replications in the special education research base (see also Lemons et al., 2016; Therrien et al., 2016) suggests that the observed effects of relatively few interventions have been directly corroborated, which implies that any number of observed positive effects in the special education literature may be unchallenged fallacies (see Ioannidis, 2012). However, van IJzendoorn (1994) theorized that direct replications may not always be necessary. That is, if a number of conceptual replications (e.g., with different populations of learners, different outcome measures) consistently reproduce a previous study’s findings (e.g., that an intervention is effective), the findings of the original study are supported indirectly and shown to be robust across varied conditions. Thus, systematic syntheses (e.g., meta-analyses, evidence-based reviews) that show consistently positive effects of an intervention across multiple conceptual replication studies may provide meaningful support for the effectiveness of the practice even in the absence of direct replication.
In contrast to Lemons et al. (2016) and Makel et al. (2016), we did not find an association between author overlap and agreement in findings between parent (replicated) and child (replicating) studies. Nonetheless, the relatively high rate of author overlap across the reviews in this special issue seems problematic. Logically, if the same authors are involved in both the replicated and replication studies, they are likely to imbue both studies with similar biases. Moreover, replications with author overlap may be particularly vulnerable to publication bias (i.e., the file-drawer problem) because researchers may be reluctant to publish findings that contradict their own previous work (Makel et al., 2012). Thus, replications with author overlap, which we found to be relatively common in the special education research base, should be interpreted with caution.
Recommendations
Perhaps not surprisingly, we recommend that our findings be replicated. To directly test the validity of our findings, we hope that an independent team of researchers will conduct a direct replication of our review. In addition, to explore the degree to which our findings generalize to other areas of special education, we recommend that other researchers conduct conceptual replications to examine the rate and type of replications conducted in different areas of special education (e.g., journals focused on early childhood special education and learners with severe or low-incidence disabilities).
As we reviewed intervention studies, we were struck by the fine line between contextualizing one’s study with a discussion of previous research and conducting a conceptual replication. Even in the studies we considered to be replications, authors were often vague about whether and the degree to which they replicated previous research. Whereas 14 of the 26 studies we identified as replications used language explicitly indicating the replicative purpose of their study (most commonly using variations of “extend” and “replicate”), we needed to infer the replicative nature of 12 studies—most commonly by authors identifying a gap or limitation in previous studies and then noting that their research addressed the gap or limitation. Moreover, authors seldom explicitly indicated which elements of a parent study were and were not replicated beyond one or two variables (e.g., participants, outcome). Therefore, we recommend that authors of research reports clearly and fully report whether and how they replicate previously conducted studies.
Finally, we recommend that steps be taken to support conducting and publishing replications, especially direct replications, in special education. Perhaps foremost, special education researchers should work toward adopting a replication culture that clearly values and rewards replication research (Ioannidis, 2014). Nosek et al. (2012) argued that to change researchers’ behavior (e.g., conduct more replication studies), the antecedents (e.g., publishing opportunities) and consequences (e.g., institutional rewards) that currently promote innovation over replication must be changed. For example, the value of replication research might be explicitly recognized in tenure and promotion guidelines. In addition, the Transparency and Openness Promotion (TOP) Guidelines (Nosek et al., 2015) recommend that journals adopt policies that (a) make data, analytic methods, and study materials transparent and available to other researchers to facilitate direct replication and (b) explicitly encourage the submission of direct replications.
Limitations
Our review and its finding should be interpreted within the context of a number of important limitations. First, our procedure for identifying replications, though reliable, may not coincide with others’ perspectives on what should be considered a replication. Indeed, we suspect that many of the authors of studies we identified as replications would not consider their studies to be replications. Second, we examined only 2 years of articles published in six non-categorical research journals in special education. Replication rates may differ in other journals, in other years, or both. Similarly, we examined only intervention studies. Replication rates may differ for other types of empirical research. In addition, our chi-square analyses involved a relatively small number of studies. It is possible that results would have differed if these analyses had greater power. Moreover, our analysis of author overlap only considered direct author overlap. A broader treatment of author overlap (e.g., reflecting when authors were trained by or had collaborated extensively with one or more authors of replicated studies) may have resulted in a higher number of replications with author overlap. Finally, we relied solely on information provided in the articles to code study elements. For example, when authors indicated that their study was based on a previously conducted study, we did not verify that the two studies shared common elements. Similarly, when authors noted that their findings agreed with results of previous studies, we did not examine the findings of parent (replicated) studies to determine whether findings did in fact agree or may have also differed in some ways. It is possible that independently comparing findings of replicated and replicating studies may have yielded a different level of agreement.
Conclusion
Reproducibility is the cornerstone of science. The idea that direct replication undergirds science has a simple premise: If an effect is real and robust, any competent researcher should be able to obtain it when using the same procedures with adequate statistical power. (Simons, 2014, p. 76)
Despite the importance of replication, prior to this special issue, we are not aware of any systematic reviews of the rate, type, or findings of replication research in special education. We examined 83 intervention studies published in 2013 and 2014 in six non-categorical research journals in special education and identified 26 replication studies, all of which were conceptual replications. We also found authors reported that the findings of the majority of replication studies solely agreed with the results of the studies being replicated and that, although author overlap occurred for half of the replication studies, agreement in findings did not differ as a function of author overlap. Although the presence of conceptual replications in the special education research base is encouraging, we recommend that special education researchers conduct more direct replications to directly examine the validity of findings from previous research.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.