Abstract
Special education researchers conduct studies that can be considered replications. However, they do not often refer to them as replication studies. The purpose of this article is to consider the potential benefits of conceptualizing special education intervention research within a framework of systematic, conceptual replication. Specifically, we advocate for the value and importance of replication research that includes both closely aligned and distal conceptual replications. We acknowledge the challenges associated with conducting replications in applied school-based research and also provide recommendations for how to design, conduct, and report replication studies in special education research with the goal of supporting the identification of effective practices for individuals with disabilities.
Replication research involves examining the validity of previous research by conducting a similar, subsequent study. Because no single study should be assumed to indicate “the truth,” replications are essential to scientific credibility (e.g., Francis, 2012; Ioannidis, 2012; Jasny, Chin, Chong, & Vignieri, 2011; Simons, 2014). Despite the generally acknowledged importance of replication to the validity of knowledge claims, the topic has received little attention in the special education literature (Cook, 2014). Accordingly, this special issue includes four reviews examining the frequency and characteristics of special education replication research. The differences among the findings from our reviews are striking. Two reviews reported that replication studies in special education are extremely rare, making up only 0.4% or 0.5% of papers published in special education journals (Lemons et al., 2016, In this issue; Makel et al., 2016, In this issue). The two other reviews suggested something very different—that a significant percentage of the intervention studies in leading special education journals include a replication component (30%; Cook, Collins, Cook, & Cook, 2016, In this issue) and that a large majority of intervention studies (75%; Therrien, Mathews, Hirsch, & Solis, 2016, In this issue) have been replicated at least once.
What explains these divergent findings? Although a number of differences in procedures and methodology across the reviews may help to explain these discrepant results, one explanation stands out. The two reviews that found low rates of replications searched for articles that included the term replicate or replication whereas the two reviews that found relatively large proportions of replications searched for articles that included the features of replication studies, but that did not necessarily use the term replication. Therefore, it appears that in special education, we conduct replications, but we often do not call them replication studies. Special education researchers clearly value programs of research that produce cumulative and converging evidence about practices and interventions; however, we do not necessarily think about these programs of research within a replication framework.
The purpose of this article is to consider the potential benefits of conceptualizing special education intervention research within a framework of systematic replication while acknowledging the challenges associated with conducting replications in applied school-based research. We also provide some initial recommendations for how to design, conduct, and report replication studies with the goal of supporting the identification of effective practices for individuals with disabilities.
Direct Replications
When we think about replication, we tend to think about direct replication. Direct replications duplicate an original study in all aspects, including participants, setting, independent and dependent variables, research design, and analysis plan (see Schmidt, 2009). Although the concept of a direct replication is straightforward, the reality of conducting one in special education research is much more challenging and complex, especially in applied settings. For example, consider the challenges associated with duplicating the following dimensions of a study in a school setting: (a) identifying a second sample drawn from the same population as the original study and (b) locating a setting that reproduces all the contextual variables from the original study. These dimensions represent only two of many that could influence study outcomes and, therefore, would be necessary to reproduce when conducting a direct replication.
The difficulties associated with conducting direct replications lead to the question of whether we can truly duplicate all aspects of an original study, especially when conducting special education research in applied school settings that are constantly shifting and changing. Because of this, scholars have suggested that true, direct replications may be impossible to conduct in the social sciences (Asendorpf et al., 2013; IJzerman, Brandt, & Van Wolfer, 2013; Simonsohn, 2015). The reviews in this special issue suggest that if direct replications are not impossible to conduct in special education research, then they are extremely rare. All four reviews identified few to no direct replication studies. Therefore, it is apparent to us that direct replications are rarely, if ever, feasible in special education research and that even if researchers’ goal is to directly replicate a study, it is likely that they will be unable to duplicate at least a few of the variables from the original study.
Conceptual Replications
Direct replications are usually not feasible in special education research. What is feasible are conceptual replications. In a conceptual replication, one or more features vary from the original study (Schmidt, 2009). For example, participants could be middle school students instead of second graders, or second graders from another school or district that serves students with different demographic or achievement profiles. Because they do not exactly duplicate a previous study, conceptual replications do not constitute an unequivocal test of the validity of prior findings (Simons, 2014). For example, if a previous study showed an intervention was effective with second graders with learning disabilities, but a conceptual replication indicated that the intervention was ineffective for middle schoolers with intellectual disabilities, the findings of the previous study are not necessarily called into question. It may be that the intervention is less effective for older learners, students with intellectual disabilities, or both. Although they do not directly examine the validity of previous studies, conceptual replications are useful for identifying the generalizability of prior findings (e.g., identifying for which populations, in which settings, for which outcomes an intervention is and is not effective; see Earp & Trafimow, 2015; Schmidt, 2009). In addition, conceptual replications can be used to identify the critical elements of interventions. For example, a conceptual replication might involve implementing an intervention for more or less time or with fewer or additional instructional components.
Special education researchers do conduct conceptual replications. In fact, we have a well-established tradition of conducting programs of research that include a series of studies evaluating the same strategy, approach, or intervention across different settings and contexts. Examples include lines of research on Self-Regulated Strategy Development (SRSD; Harris, Graham, & Mason, 2006), Peer-Assisted Learning Strategies (PALS; Fuchs & Fuchs, 2005), and many others. Special education researchers value programs of research evaluating common interventions for good and important reasons. The field of special education is committed to identifying evidence-based practices because we know that students with disabilities require the most effective instruction and intervention to accelerate learning outcomes (Odom et al., 2005). Established standards for identifying evidence-based practices require evidence from multiple studies (e.g., from conceptual replications). For example, Gersten et al. (2005) proposed that practices be considered evidence based when an intervention is supported as effective by two high-quality group studies or four acceptable group studies, which necessitates conducting replication studies. Special education researchers are also comfortable with contextual variability across research studies. We understand the importance of explicitly acknowledging and accounting for the role of context in our research designs and analysis plans. We regularly evaluate the effects of interventions with different participants and in different settings because we do not assume that an intervention that works with some students under certain conditions will work with other students under different conditions.
What special education researchers do not appear to do very often, however, is think about these programs of research within a replication framework. The finding that very few studies published in special education journals use the term replicate or replication (Lemons et al., 2016; Makel et al., 2016) supports this assertion. It may be that because direct replications are impractical, if not impossible, in special education research, we do not consider replication as an option when conceptualizing or designing studies. The idea of conceptual replication and its role in validating previous findings may be less understood than direct replication. It is also possible that special education researchers avoid identifying their research as replications because of negative perceptions of this type of research. Although dated, empirical evidence suggests that journal editors and reviewers in the social sciences hold negative views toward publishing replication research (e.g., Madden, Easley, & Dunn, 1995; Neuliep & Crandall, 1990, 1993). Even if the views of editors have changed in recent years, it appears that negative perceptions of replication research persist among many scholars. For example, replications continue to be discussed as difficult to publish (Ritchie, Wiseman, & French, 2012; Schmidt, 2009; Spellman, 2012), less valued and rewarded than novel research (e.g., Ioannidis, 2014; Makel & Plucker, 2014; Nosek, Spies, & Motyl, 2012), and as potentially hostile toward the original researcher(s) (Koole & Lakens, 2012). Therefore, even if a study includes some features of a conceptual replication, we may not explicitly consider or report what we are doing as “replication.” Rather, special education researchers may think more broadly and generally about conducting programs of research that add converging evidence in support of an intervention, strategy, or approach.
If special education researchers are engaged in conducting conceptual replication studies, but just not calling them replications, is this merely a problem of semantics rather than substance? We believe that when we do not explicitly conceptualize and report special education research within a replication framework, we may be missing an opportunity to more systematically accumulate evidence about our interventions. Valentine et al. (2011) suggested that rather than being conceptualized within a replication framework, most replications are ad hoc in nature. Authors are unlikely to identify ad hoc replications as replications or specifically indicate in which ways the study is the same as and different from previous research on the topic, and—although they may investigate the same general practice—ad hoc replications and original studies “vary from one another in multiple known and unknown ways” (Valentine et al., 2011, p. 106). Thus, it is difficult for research consumers to make clear sense of whether and how the results of ad hoc replications verify, contradict, or extend the findings of previous research. The benefits of thinking explicitly about replication become clearer when we consider a continuum of replication research within a larger framework of generalizability.
Replication in Support of Generalizability
Replication research plays a central role in systemically investigating whether the effects of an intervention are valid and generalize across different participants, settings, and other contextual and instructional dimensions. In a sense, results from replication studies, especially conceptual replications, act as a bridge between the tentative and narrow findings from an initial study and established evidence of generalization. For example, the following paragraph illustrates a path from an original study through a series of replication studies to eventually meta-analysis.
An individual study provides initial evidence about the efficacy of an intervention. A direct replication (or “as direct as possible”) then ensures that findings from the initial study can be duplicated and are not the result of error, bias, or chance. Once effectiveness of the intervention is supported by a direct (or close to direct) replication, a series of conceptual replications that systematically manipulate different theoretically important variables provide accumulating evidence about whether the impact of the intervention, or variations of the intervention, generalize across different settings, populations, and other contextual variables. Finally, after a series of planned conceptual replications has been conducted, a meta-analysis can be performed to quantitatively aggregate data from all studies to determine average treatment effects and also investigate variables that may moderate those effects. The ultimate results of a program of research informed by a framework of replication and generalization is robust, converging, and trustworthy evidence about the effects of an intervention, whether the effects are generalizable, and whether effects are robust to contextual variability.
When considering programs of research within a larger framework of replication and generalization, it becomes clear that a continuum (Rosenthal, 1991) of possible conceptual replication studies ranges from more closely aligned to an original study (i.e., that include what Brandt et al., 2014 termed close replications, in which the replication—although not a truly direct replication—is as similar to the original study as is possible) to more distal replications in which a number of variables differ.
Special Education Research Informed by a Continuum of Conceptual Replications
There are potential benefits for thinking about special education research within a framework of systematic replication in support of generalization, particularly a framework that classifies conceptual replications on a continuum from more closely aligned to an original study to more distal. For example, a framework that includes a continuum of conceptual replications may help address challenges that special education researchers face when conducting studies designed to accumulate evidence about an intervention. For clarity, it may be useful to quantify the amount of overlap between a replication and an initial study based on (a) how many dimensions vary, (b) how much those dimensions vary, and (c) how theoretically important those dimension are.
Closely Aligned Conceptual Replications
A closely aligned conceptual replication can serve the same practical purpose as a direct replication (Brandt et al., 2014). Although not as theoretically pure as a direct replication, the results of a closely aligned conceptual replication can provide valuable evidence about whether treatment effects from an original study replicate under very similar conditions. Evidence from multiple, closely aligned replications strengthens initial findings even more. Most important, unlike direct replications, closely aligned conceptual replications are feasible to design and carry out in special education research.
For example, a closely aligned conceptual replication could consist of implementing a reading intervention with similar participants in a second school district with comparable demographics and approach to reading instruction. In this case, only two dimensions of the original study are varied (i.e., participants and setting), the dimensions do not vary substantially (i.e., the participants and setting are very similar to the original study), and these dimensions are not central to the theoretical mechanisms hypothesized to produce the treatment effect (i.e., the intervention should produce similar effects with comparable participants in a similar setting). Although still technically a conceptual replication, this “close replication” example is probably as near to a direct replication that is possible in school-based special education research and can serve the same purpose of examining the validity of initial findings regarding the effectiveness of an intervention.
If the findings from a closely aligned conceptual replication mirror those from an original study, it provides compelling evidence supporting the efficacy of the intervention and reduces the possibility that the original findings were the result of error or chance. If findings do not replicate, it is unlikely that the discrepant findings are attributable to any differences between the initial study and the replication because few variables differed across studies and those that did were not theoretically central to the intervention.
Distal Conceptual Replications
We know that context matters in special education. Therefore, determining the effectiveness of an intervention is not as simple as conducting an initial study and confirming those findings through close conceptual replications. We expect that the effects of an intervention will differ depending on a number of contextual factors that reflect the natural variability found among students, teachers, and schools. In fact, we may never be able to determine a “true” estimate of the effects of an intervention but, instead, only accumulate evidence about the relative effectiveness of an intervention in particular contexts under certain conditions. The role of more distal replications, as part of a carefully designed series of conceptual replications, is to systematically investigate the effects of an intervention under different conditions by purposely manipulating theoretically important instructional and contextual variables (Schmidt, 2009).
For example, by conducting a distal conceptual replication, a reading intervention that was initially found to be effective for students with learning disabilities and confirmed via close conceptual replication studies could then be evaluated for its effect on students’ with Down syndrome reading achievement. Researchers who design distal conceptual replications, which Bonett (2012) referred to as replication-extension studies, have the freedom to manipulate theoretically interesting aspects of an intervention, such as instructional delivery, content focus, and dosage.
Findings from more distal conceptual replications that are consistent with findings from the original study and closely aligned replications contribute evidence about the generalizability of the intervention across important student, instructional, and contextual variables. Distal conceptual replications that do not find comparable treatment effects provide important information that helps define the conditions under which an intervention is more or less efficacious—in effect, determining how robust an intervention is to variability. For example, it may be that some highly impactful interventions are only effective for very specified populations implemented under very specific conditions, while the effects of other equally impactful interventions are more generalizable across populations and instructional conditions.
Variations Across Replication Studies
The preceding discussion and examples illustrate how conceptual replications exist on a continuum from more closely duplicating an original study to purposely varying key dimensions. The degree of overlap or similarity between a conceptual replication and an original study can be determined by examining the number of variables that vary between the studies and the degree to which these variables vary. When designing or interpreting studies within a replication framework, it may be helpful to delineate and categorize dimensions of variables that vary and remain constant across studies. In this way, researchers could more purposely consider and report the degree of overlap between studies. Broad categories of variables could include participants, setting, intervention (IV), outcome measures (DV), and research design/analyses. Examples of study dimensions within these categories that could be held constant or systematically and intentionally varied across studies are outlined in Table 1.
Study Dimensions That Could Be Held Constant or Intentionally Varied Between a Conceptual Replication and an Initial Study.
Note. SES = socioeconomic status.
Replication in Support of Meta-Analyses
Meta-analyses synthesize findings across studies to provide reliable information on the average magnitude of effect. Meta-analysis plays a critical role in identifying interventions that tend to have large and small effects for learners with disabilities (e.g., Forness, Kavale, Blum, & Lloyd, 1997). Simonsohn (2015) suggested that although meta-analyses and replications are related, they are motivated by different questions: Whereas meta-analyses answer, “what is the aggregate magnitude of effect across studies?” replications seek to determine if a previously reported effect is valid (see also van Elk et al., 2015). To illustrate these different purposes, Simonsohn uses the hypothetical example of an initial study showing that people can levitate an average of 9 inches in the air, which is followed by a study showing that people could not levitate at all. A meta-analysis might report that the average magnitude of levitation is 4.5 inches across studies. In contrast, if analyzed as a replication, the results of the second study (assuming it was adequately powered and methodologically sound) would cast considerable doubt on the validity of the previous findings.
A series of carefully designed conceptual replications also supports better and more useful meta-analyses. For example, it is difficult to interpret the results of a meta-analysis that investigates a group of individual studies evaluating some version of a common intervention in which the relationships among the studies have not been explicitly defined. If, as Valentine et al. (2011) suggested, studies investigating the effects of an intervention are typically ad hoc replications in which the practice and other variables vary from study to study in multiple, meaningful, and often unreported ways, an average effect size across studies may not address the issue of whether the original version of intervention is effective (especially if effects are heterogeneous across studies). Moreover, it is particularly difficult to tease out moderators in meta-analyses of studies that do not build on each other systematically. As noted, ad hoc replications may involve settings, participants, outcomes, and procedures that vary in many important ways. Thus, moderators can only be reliably analyzed after a large number of studies in which these elements vary more or less randomly across studies have been conducted. Such an approach is highly inefficient, especially given the considerable resources needed to conduct high-quality intervention studies in applied settings.
In contrast, a coherent series of conceptual replications that make explicit how each study either duplicates or purposefully varies the dimensions of earlier studies provides a strong conceptual and empirical foundation for conducting and interpreting the results of a meta-analysis. For example, after support for the validity of an intervention has been established through a close replication, researchers might conduct a series of studies that systematically build upon one another in which only one or two theoretically important variables related to the intervention, participants, setting, or outcomes are varied per study. A meta-analysis on such a series of carefully designed conceptual replications will require markedly fewer studies to generate meaningful outcomes (e.g., average magnitude of effect of the intervention across studies, significant moderators of that effect) than an ad hoc approach to intervention research.
Suggestions for Reporting, Designing, Conducting, and Interpreting Replication Studies in Special Education
Embracing a framework of systematic replication could support special education researchers in accumulating converging evidence about interventions and identifying evidence-based practices. What follows are initial recommendations for how to better align our intervention research in special education with a framework of systematic replication. Recommendations are summarized in Table 2.
Recommendations for Designing, Conducting, Reporting, and Interpreting Replication Research in Special Education.
Describe Replication Components of Research Studies Directly
A relatively easy recommendation is to call what we already do replication (i.e., call replications replications). Even though special education researchers appear to infrequently conduct direct replications, the types of studies we do conduct can provide important evidence about whether the effects of interventions replicate under certain conditions. As Cook et al. (2016) and Therrien et al. (2016) report, many of the intervention studies that special education researchers publish can be considered conceptual replications. Essentially, these are studies that investigate an intervention that has previously been studied. If researchers called these studies replications, it would help research consumers better situate the research in the larger body of literature and interpret findings within that broader context—not to mention making the job easier for authors of review articles, such as those in this special issue.
Even if the primary goal of the study was not to purposefully replicate an earlier study, acknowledging any replication component directly would both provide important information to support interpretation and also promote a growing understanding of the role of replication in special education research. Following American Psychological Association (APA) guidelines, authors should clearly identify whether and how their study replicates one or more previous studies in the Introduction (APA, 2010). We also echo Brandt et al.’s (2014) call for authors of replication studies to specify which aspects of their study directly, closely, or conceptually (i.e., distally) replicate previous research. Moreover, to enable designers of future replication studies to accurately identify whether and how they are replicating different aspects of previous studies, researchers should fully report all aspects of their studies (Brandt et al., 2014; IJzerman et al., 2013), which might involve posting details on publisher or university websites if sufficient space for full reporting is not provided in a journal (Asendorpf et al., 2013).
Design and Conduct Closely Aligned Conceptual Replications
Direct replications are almost impossible to conduct in applied special education research. Therefore, special education researchers need a way to ensure that intervention effects found in initial experiments are not the result of error, bias, or chance. Closely aligned conceptual replication studies can provide valuable and rigorous evidence about whether the findings from an intervention study can be reproduced under very similar conditions. In a framework of systematic replication, closely aligned conceptual replications play a critical role in building a foundational evidence base for an intervention.
Acknowledging the value of closely aligned replications may require a shift in thinking for both researchers designing intervention studies and journal editors and reviewers evaluating the contribution of research manuscripts. Special education researchers clearly value research examining whether the effects of interventions generalize across populations, settings, outcomes, and contexts. Therefore, a research study that duplicates many features of existing research may be seen as less significant because it fails to extend or expand knowledge of an intervention beyond what is already known from an earlier study (see Koole & Lakens, 2012; Makel & Plucker, 2014; Nosek et al., 2012; Schmidt, 2009). To counter this perception, a number of reforms may help to promote a replication culture (Ioannidis, 2014) in special education, such as (a) journals adopting policies to encourage submission and publication of replication studies regardless of whether they reproduce the findings of the original study (Nosek et al., 2015); (b) policies that recognize and reward methodologically sound research (e.g., high-quality, close replications) in tenure, promotion, and merit decisions rather than rely on journal-based metrics (e.g., impact factors; Asendorpf et al., 2013; San Francisco Declaration on Research Assessment, 2013); and (c) using a “crowd sourcing” approach to access the resources needed to conduct independent replications (e.g., Nosek et al., 2012; Open Science Collaboration, 2015).
Special education researchers should design and conduct closely aligned replications of initial intervention studies that duplicate as faithfully as possible the features of previous studies. Decisions about which features vary should be based on both what is feasible and an understanding of the mechanisms central to the hypothesized treatment effect. The goal is to design a study that duplicates the theoretically important features of an earlier study to provide strong evidence of reproducibility. Moreover, multiple close replication studies are warranted to fully investigate the validity of important findings. For example, just as the outcomes of an initial study should not be uncritically accepted, neither should the results of a single close replication. This is especially the case if the replication study has insufficient power, does not address contemporary standards for high-quality intervention research (e.g., Council for Exceptional Children, 2014; What Works Clearinghouse, 2014), or both. Thus, it is critical that replication studies be sufficiently powered and of high methodological quality (Asendorpf et al., 2013; Brandt et al., 2014; Simonsohn, 2015)
One relatively feasible way to increase the number of closely aligned replications in special education is for research teams to plan for a replication study in conjunction with an initial efficacy study. This enables researchers to more easily duplicate many of the features of the original study. For example, a research team involving the first author recently conducted a program of research that included both a well-powered randomized control trial and a closely aligned replication (Coyne, McCoach, Ware, & Rattan, 2015). The replication study evaluated our early vocabulary intervention in the same schools with the same teachers and interventionists but with a new cohort of kindergarten students. This study was not a direct replication because the students were different, teachers had a year’s experience delivering the intervention, and our team refined the way we supported implementation. However, we were able to duplicate many more features of the original study efficiently and economically by planning for a replication a priori rather than considering a replication later. Special education researchers should consider planning for, or proposing, closely aligned replications when designing an intervention study or developing a grant application.
Although a research team that conducts an initial efficacy study is well positioned to conduct a closely aligned replication, this approach leaves open the possibility that unintentional bias or flaws in the logic or procedures of the study could have influenced results. These biases or flaws would likely be reproduced in a replication conducted by the same research team. In addition, researchers might be motivated to reproduce and validate their previous findings and, therefore, make decisions to bring about those results (Nix & Barnette, 1998). Indeed, replications sharing one or more common authors with the original studies are more likely to reproduce findings (Lemons et al., 2016; Makel & Plucker, 2014; Makel et al., 2016; Makel, Plucker, & Hegarty, 2012). As such, Koole and Lakens (2012) recommended that, “the most compelling direct replications are therefore conducted independently by different researchers than the original study” (p. 609). Therefore, researchers not associated with the authors of or interests related to the original research should ideally conduct closely aligned replications. This is particularly important in special education research in which a high percentage of replication studies are authored by the same team as initial studies.
Design and Conduct Distal Conceptual Replications
According to Cook et al. (2016) and Therrien et al. (2016), special education researchers already conduct conceptual replications with some frequency. If we begin to think about intervention research as existing on a continuum of increasingly more distal conceptual replications, we may be better able to purposefully design our studies so that they contribute more systematic evidence in support of generalization.
Key to establishing a knowledge base built on systematic replication is considering the replication component of intervention studies a priori during the conceptualization and design stage. If special education researchers designed intervention studies with the goal of serving as conceptual replications, it would encourage them to be more specific about similarities and differences between their study and other earlier studies investigating the same intervention. Because conceptual replications are defined by the degree of overlap between the replication and earlier studies, researchers could specify which features of the current study vary from other studies, how much they vary, why these variations are theoretically important, and how these variations support reproducibility and/or generalization (see Brandt et al., 2014).
In practical terms, situating studies within a framework of systematic replication would dissuade researchers from generic statements such as “Few studies have examined this intervention with students at this grade level” and encourage more precise statements such as “The design of the current study varies from Smith et al., in these important ways: . . . .” Similarly, when delineating directions for future study, researchers would be less likely to provide general recommendations such as “results should be replicated in different settings.” Instead, they would be better able to provide informed suggestions such as “a logical next step would be to evaluate whether this intervention is equally effective under the following conditions . . .”
When designing more distal conceptual replications, special education researchers should consider whether to duplicate or intentionally vary each of the essential components of earlier studies, including participants, setting, intervention features, outcome measures, and analyses. Variations should be informed by theoretically important questions about whether interventions produce effects under different conditions from earlier studies. Although the design of research studies in applied settings is certainly driven to a large extent by issues of feasibility and access, understanding the key variables central to the goal of a well-defined conceptual replication can help researchers select and target certain study conditions. For example, if the goal of a more distal conceptual replication is to evaluate the differential impact of an intervention implemented for 30 min compared with 60 min, the research team could prioritize that dimension of the study when recruiting schools and negotiating study procedures.
Interpret Findings of Replication Studies Using Multiple Approaches
Determining whether the results of a study successfully replicate the findings of previous research is not as straightforward as it might seem. The common approach of comparing significance levels (e.g., statistically significant findings from an original study are considered to be replicated if the findings of a replication study are also significant) is likely insufficient (Brandt et al., 2014; Simonsohn, 2015). For example, two studies might generate highly similar effects, yet statistical significance might “disagree” if (perhaps due to differences in sample size) findings from the original study were marginally statistically significant whereas findings from the replication study were just above the threshold for statistical significance. Similarly, simply comparing the direction of effects may also be problematic; although the directions of effects agree, a study with very small positive effects should not be interpreted as replicating an initial study with very large effects (Valentine et al., 2011). Scholars recommend using multiple approaches to collectively examine whether and how replication studies support previous research findings, much in the same way as qualitative researchers triangulate evidence from multiple sources to draw conclusions. In addition to considering statistical significance and directions of effects, researchers should consider (a) examining whether the effect size of the replication study falls within the confidence interval of the original study, (b) conducting homogeneity analyses to examine whether study effects are comparable, and (c) meta-analytic aggregation of effects (Asendorpf et al., 2013; Brandt et al., 2014; Valentine et al., 2011).
Conclusion
The goal of intervention research in special education is to identify effective practices for students with disabilities and accumulate rigorous and trustworthy evidence about the conditions under which these practices are more or less effective. Conceptualizing intervention research within a framework of systematic replication, particularly a continuum of increasingly distal conceptual replications, could help special education researchers better design, conduct, and report research that promotes that goal.
Closely aligned conceptual replications provide evidence about the efficacy of an intervention and help determine whether findings from an initial study are reproducible under very similar conditions. Closely aligned conceptual replications play a particularly important role in special education research because of the difficulty of conducting direct replications in applied settings and increasing concerns about the lack of reproducibility of research findings across scientific disciplines (e.g., Earp & Trafimow, 2015; Ioannidis, 2005; Lehrer, 2010; Open Science Collaboration, 2015). More distal conceptual replications provide evidence of generalizability and help determine whether the effects of interventions are robust across student, instructional, and contextual variables. Distal conceptual replications are critical because context matters in special education research and practice, and the effects of interventions are likely to differ based on variability in contextual factors. A series of conceptual replications that each explicitly specifies the degree of overlap between the current study and earlier studies enables more coherent interpretation of the body of work and supports comprehensive meta-analyses.
There are concerns that despite its importance, journal editors, funding agencies, and promotion and tenure committees do not value replication research and, as a result, many contingencies exist to dissuade researchers from conducting replication studies (Ioannidis, 2014; Makel & Plucker, 2014; Nosek et al., 2012). We support a more optimistic outlook. We believe that our field’s commitment to fostering an evidence-based profession puts special education in a better position than other disciplines to embrace the value of replication research. We believe that editors of special education journals and agencies that fund special education research are increasingly recognizing the important contribution of high-quality, well-powered replications and are becoming more likely to look favorably on manuscripts and proposals that are explicitly presented within a replication framework. The landscape is promising for special education researchers who intentionally and purposefully design and conduct replication research.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.