Building Evidence in Early Childhood Special Education: A Systematic Review of Replication Intervention Studies

Abstract

Valid and evidence-informed practices are critical to help young children with disabilities and their families with highly effective interventions and instruction to reach their potentials. Replication research is critical for appraising research and identifying evidence-based practices. The purpose of this study was to replicate the methods used by Cook and colleagues and conduct a systematic review of the quantity, type, and findings of replication research in early childhood special education (ECSE). An intervention study was considered a replication if (a) the stated purpose of the study was to replicate, expand, further investigate, or use a previously conducted study as its basis and (b) the findings of the study were compared with the results of previous studies. Two hundred ninety-seven published articles in three journals relevant to ECSE from 2012 to 2016 were reviewed to identify intervention studies. From the 78 identified intervention studies, we found 44 studies (56%) that met the criteria for replication. Author overlap occurred for 20 studies out of the 44 replication studies (46%). Our investigation revealed that ECSE journals published a significantly larger number of replication studies (56%) than those reported by Cook and colleagues (31%); however, the scarce number of direct replication was concerning. We recommend intentional and proactive steps be taken to promote and value replication research in ECSE.

Keywords

evidence-based practice replication early childhood special education research methodology

The primary goal of early childhood special education (ECSE) is to use the most effective practices to improve the positive outcomes for infants and young children with or at risk for disabilities and their families (Odom, 2009). In the past two decades, evidence-based practices (EBP) have become a central focus and part of the lexicon in early childhood research (Buysse, Wesley, Snyder, & Winton, 2006). EBP have been operationally defined as, “practices that are informed by research, in which the characteristics and consequences of environmental variables are empirically established and the relationship directly informs what a practitioner can do to produce a desired outcome” (Dunst, Trivette, & Cutspec, 2002, p. 3). EBP are particularly important for children with or at risk for disabilities as they require the most effective, efficient, and intensive instruction to accelerate learning (Fuchs & Fuchs, 2014).

EBP in ECSE involve empirical evidence, clinical expertise, family values, and ongoing data collection which are adapted as necessary with individual children and families (Barton & Smith, 2015). The Individuals With Disabilities Education Act (2004) and the Every Student Succeeds Act (2015–2016) have emphasized the application of scientific research for the purpose of training and instruction in special education (Cook & Cook, 2013). However, the most significant challenge regarding EBP is the lack of ecological validity of the research findings leading to limited implementation in everyday practices (Cook & Odom, 2013). The origins of EBP in early childhood can be traced to the long-standing gap between research and practice (Buysse et al., 2006).

Replication of EBP has been considered critical to the scientific process and credibility because no single study can be considered to present “the truth” (e.g., Francis, 2012; Ioannidis, 2012; Jasny, Chin, Chong, & Vignieri, 2011; Simons, 2014). Replication is “the process of repeating a research study with a different group of research participants using the same or similar conditions, for the purpose of increasing confidence in the original study’s findings” (Gall, Gall, & Borg, 2003, p. 635). Replication studies that result in similar findings can increase scientific confidence, advance science, and provide a more comprehensive understanding of research findings (Cook, Collins, Cook, & Cook, 2016). Claims that rely on a replicated body of evidence (i.e., multiple studies conducted using similar methods by different researchers in different geographic locations) and report similar findings across similar populations should generally be treated as valid and scientific (Travers, Cook, Therrien, & Coyne, 2016). In contrast, research findings based on a small nonreplicated body of evidence are generally considered unconvincing. In addition, insufficient replication of novel research can result in failure to uncover nuances, false, or invalid research findings; these findings may be presumed to be true and remain uncorrected or misunderstood (Burns & Ysseldyke, 2009; Makel, Plucker, & Hegarty, 2012). A lack of replications can be associated with the underuse of EBP known to positively affect student performance. For example, special education practitioners reported regularly using some practices demonstrated to be ineffective (e.g., modality instruction), but indicated implementing other practices validated by research (e.g., mnemonic strategies) relatively infrequently (Burns & Ysseldyke, 2009).

Despite the importance of replication research, recent empirical and theoretical works indicate researchers (a) seldom try to replicate previous findings, and (b) often are unsuccessful in regenerating previous findings when they do (Nosek, Bar-Anan, & Motyl, 2012; Pashler & Harris, 2012). When comparing the field of special education with other fields that have broader scope and evidence, such as medicine and psychology, special education researchers have not adequately examined these issues (Makel & Plucker, 2014; Makel et al., 2012). Furthermore, in special education, although researchers conduct replication studies, they often do not clearly stipulate that the studies are replications. Special education researchers support attempts that generate cumulative and converging evidence in addressing practical interventions and practices, but often fail to consider these efforts within a replication framework (Coyne, Cook, & Therrien, 2016). The very small number of published studies in special education journals that include the terms replicat* in their abstract, introduction, or methodology support this postulation (Lemons et al., 2016; Makel et al., 2016).

Replications may be direct or conceptual. Direct replications are defined as duplicating an original study including all aspects such as participants, setting, independent and dependent variables, research design, and analysis plan (Schmidt, 2009). According to Coyne and colleagues (2016), the reality of conducting direct replications in special education is challenging and complex despite appearing to be straightforward. This is particularly true for direct replications conducted in applied settings. In contrast, conceptual replications are more feasible and doable. In a conceptual replication, one or some features differ from the original or parent study (Schmidt, 2009). Although validity of previous studies is not directly examined through conceptual replications, they can produce results related to identifying the generalizability of prior findings regarding population, settings, and outcomes of an effective and ineffective intervention (Earp & Trafimow, 2015; Schmidt, 2009). Special education researchers tend to conduct conceptual replication studies (Fuchs & Fuchs, 2005). However, as Coyne and colleagues (2016) noted, the idea of conceptual replication and its role in validating previous findings may be less understood than direct replication.

In a systematic examination of replication studies in special education, Makel and colleagues (2016) conducted a comprehensive search for studies from special education research that included different variations of the word replicat* and a two-pronged definition of replication. These search terms produced a replication rate of 0.5% in special education journals, which was considered significantly lower than the reported rate for psychology, but markedly higher than the rate reported for education as a whole. Authors also reported that the majority of replications found similar findings to the parent study. Replication studies that included at least one author from the parent study were more likely to report similar results compared with replication studies with no author overlap with the original study (86.3% vs. 68.4%). In another investigation of direct replication rates in special education journals, Lemons and colleagues (2016) identified 70 replication studies published between 1997 and 2013, indicating that 0.41% of published articles in special education journals are replication studies. Similar to findings reported by Makel and colleagues, the review by Lemon and colleagues indicated that most replications are successful and that successful replications are more likely when there is author overlap.

In contrast, Cook and colleagues (2016) conducted a systematic review of intervention studies published in six noncategorical special education journals in a 2-year period to examine the extent and nature of replication studies using a broader and functional definition of replication. Within the reviewed journals, they concluded that 31% of published intervention studies were replications. This rate of conceptual replication was significantly higher than that found by Makel and colleagues (2016) and Lemon and colleagues (2016) who used a more restrictive definition of replication. Similar to Makel and colleagues (2016), Cook and colleagues (2016) found that the majority of replication studies reported similar findings to the parent study. However, unlike Makel and colleagues, they did not report a significant relation in the similarity in findings between the parent and child studies with author overlap compared with replication studies with no author overlap.

Research in ECSE is focused on the identification of effective practices to improve positive outcomes for young children with exceptionalities, including those with or at risk for disabilities and their families. As ECSE works to minimize the research to practice gap, the factors that promote practitioners’ use of effective intervention practices must be considered and defined (Artman-Meeker, Fettig, Barton, Penney, & Zeng, 2015). Encouraging more replication research is one approach for narrowing this gap and establishing scientific credibility of practices. However, to date, there has not been any systematic investigation of the number and type of replication studies conducted in ECSE.

The purpose of this study was to examine the prevalence of replicated intervention studies in ECSE, the types of replications being conducted (i.e., direct, conceptual), rate of agreement between the original study and the replicating study, the extent to which studies were published by the same authors and to determine if the author overlap influenced the rate of agreement and whether the rate of replications varied by research methodology. To achieve these goals, we replicated the methods used by Cook and colleagues (2016). Cook and colleagues investigated similar questions for the overall field of special education by targeting six research journals in special education without a categorical focus. However, our study was specific to the research published in ECSE. The research questions addressed by Cook and colleagues (2016) guided the development of the research questions used for this study. They were as follows:

Research Question 1: Using a functional definition of replication, what is the rate of replication studies in ECSE intervention research (overall, and for direct and conceptual replications)?

Research Question 2: As reported by study authors, what is the rate of agreements between findings of original and replicating studies?

Research Question 3: What is the rate of author overlap in replication studies? Are replication studies with author overlap more likely to have findings that agree with original studies?

Research Question 4: Does the rate of replication vary as a function of the research design?

Method

As a first step, noncategorical research journals in ECSE were identified. The second step included identification of intervention studies and coding them as a replication based on criteria outlined by Cook and colleagues (2016). Below, we describe our search, coding procedures, and process for data analysis.

Step 1: Identification Process of Noncategorical Research Journals in ECSE

We followed Cook and colleagues’ process in selecting noncategorical journals for the study to prevent bias in the review procedure. Journals were first shortlisted based on a categorical search in the Journal Citation Report for 2015. From a search of more than 11,000 journal titles, the following categories were used to generate the initial list of journals (n = 402): Family Studies; Education, Special; Psychology, Education; Psychology, Developmental; Education & Education Research. Each of the 402 journal titles were randomly assigned to one of the four researchers and coded for inclusion in the study. In this step, we reviewed the aims and scope of each journal title as published on the journal website. The purpose of this coding was to identify journals that included (a) original research, (b) noncategorical studies, and (c) published articles specific to young children aged birth through 8. The decision tree illustrated in Figure 1 describes this process. A list of 10 journal titles in early childhood was generated; seven journals were eliminated because our examination of each indicated that there were very few, if any, intervention studies that focused on ECSE. The final list of three selected journal titles that specifically publish articles relevant to ECSE is as follows: Topics in Early Childhood Special Education, Journal of Early Intervention, and Infants and Young Children.

Figure 1.

Decision process used to identify noncategorical research journals in ECSE.

Identification Process for Intervention Studies

To keep the task manageable and to closely align our methods with those of Cook and colleagues, all published articles in the three journals for the past 5 years (2012–2016) were reviewed (n = 297) to identify intervention studies. Future topics and editorials were excluded from this review. Only studies that focused directly on children birth through 8, and their families or professionals who worked with them, were coded. Interventions were defined as “original, empirical research that investigated the effect or interaction of an independent variable that was implemented under the control of researchers on a specified outcome” (Cook et al., 2016, p. 225). Correlational studies and articles that only described interventions or programs without investigating the impact were excluded. Two researchers coded each study simultaneously. Disagreements, if any, were reconciled between the two researchers who coded those studies through discussion. If an agreement could not be reached, the two remaining researchers reviewed the study to reach a mutual agreement for inclusion or exclusion of the intervention study in the review. Interrater agreement for the 297 articles reviewed was 93%. Using percentage of agreement estimates method, the calculation used for interrater agreement was (Agreements / [Agreements + Disagreements] × 100). Articles that met the criterion represented a wide range of exceptionalities, specific ages of children (birth to 8 years), settings, and participants, including preservice teachers, teachers, or parents who received training.

Identification Process for Replication Studies

Slightly modified criteria and coding sheets developed by Cook and colleagues (2016) were used to identify replications within the identified intervention studies (see Figure 2). Specific to ECSE, we added information about targeted age group (0–3, 3–5, 5–8), participants, and the setting for each study in the coding sheet. The coding criteria included two initial questions: (a) Is there a replication statement in the Introduction or Method that cites one or more studies or refers to study/studies/literature base/research base/research? and (b) Are findings from previous research compared with findings from current study in either the Results or Discussion?

Figure 2.

Coding procedures for replication studies.

As discussed earlier, studies may be replicated in one of two ways: direct replications (i.e., studies duplicating an original study including all aspects such as participants, setting, independent and dependent variables, research design, and analysis plan) or conceptual replications (i.e., studies where one or some features differ from the original or parent study). For the purpose of this study, we decided to intentionally use a broad, functional definition of replication that described conceptual replications, similar to the definition used by Cook and colleagues:

A replication statement indicates that the study purposefully replicates, extends, further investigates, or uses one or more previously conducted studies as its basis. Indicating that one or more gaps exist in one or more previous studies (e.g., not addressing a particular outcome area or learner population) and then noting in the same paragraph or in consecutive sentences in separate paragraphs that the present study addresses the gap(s) constitute a replication statement. The source for a replicated study must be a previously conducted study (i.e., a review or meta-analysis cannot be replicated). (p. 226)

To be considered a replication, a study should reference previous research in the Introduction or Method and the authors needed to include a statement that the purpose of their study was to extend previously conducted studies. For example, Meadan and colleagues (2016) wrote, “The training and coaching procedures replicated those used by Meadan, Angell, and colleagues (2014) but were delivered to parents from a distance using online videoconferencing instead of in person” (p. 6). Following the procedures suggested by Cook and colleagues (2016), studies that noted the purpose was to extend previous research but did not cite those studies were not coded as replications. Although this decision may have limited studies indicated as replications, we believed this process was necessary because without explicit connection and citation to the parent study, the scientific review would be compromised and may have led to confusion among researchers resulting in lower interrater agreement.

For studies that were coded “yes” for a replication statement in the Introduction or Method, the Results and Discussion sections were analyzed for documentation of further evidence. The determination was made if the findings of the parent study(ies) cited in the Introduction or Method were again referenced and compared with the Results of the current study. The following criterion was used to guide our coding:

Findings from previous research are considered to be compared to findings from the current study when the authors use language that indicates comparing or contrasting findings (e.g., as in, similar to, in contrast to) or when findings from previously conducted research and current study are noted in the same or consecutive sentences (e.g., Author A (2000) reported positive effects of the intervention. In this study, we found no effects.). Noting findings from the current study and a previous study in different places (e.g., in different sections of the manuscript, in different paragraphs) does not constitute a comparison of findings. (Cook et al., 2016, p. 226)

When coding intervention studies for replication statements, researchers reviewed all studies referenced by the “child” study in potential replication statements. A study was included as a parent study only if there was a clearly identifiable intervention, such as parent–infant communication training, with a dependent variable and findings. Studies that met both criteria and referenced one or more “parent” study(ies) in the Introduction or Method and Results or Discussion were coded as replication studies. If the parent study(ies) referenced did not include at least one intervention study under each of the two criteria, the study was not coded as a replication. Also, if only one criterion was met, the study was not coded as a replication study.

To train for establishing interrater agreement for this step, the four researchers selected one article that was included in this study, coded the selected article for replication statements, and discussed the coding procedures. The coding process of this first article was done collaboratively until all four researchers agreed upon coding procedures. The interpretation of whether the study “purposefully replicates, extends, further investigates, or uses one or more previously conducted studies as its basis” differed among researchers. The group finally reached a consensus to code broadly to include all possible replications. Examples of replication statements provided by Cook and colleagues (2016) in the coding sheets afforded guidance and clarity. Two additional intervention studies, not included in the sample for this study, were then reviewed by all four researchers for further calibration. Any disagreements were discussed until consensus was reached.

After the training, each of the identified intervention studies was randomly assigned to one of the four researchers for coding. In addition, each researcher randomly selected six studies from each other’s list (total 24 studies or one third of the studies) to double-code for interrater agreement. Thus, each researcher coded 25 or 26 intervention studies. For the 24 double-coded studies, interrater agreement for studies that met both criteria and were thus coded as including a replication statement was 77%. The interrater agreement was 71% for the first criterion and 83% for the second criterion. The somewhat lower interrater agreement was caused by disagreements among researchers when identifying the specific and explicit connection between the “child” study and “parent” study in the Introduction or Method sections. These disagreements were further discussed until consensus was achieved.

The next step included coding the replication studies as either direct or conceptual replications. Studies were coded as direct replications if the participant sample, setting, intervention, and outcome measures were the same as the parent study. Outcome measures were categorized by the five commonly used early childhood domains. Studies that did not meet this description were coded as conceptual replication studies. Interrater agreement for coding of the direct and conceptual studies was 96%.

Coding Process for Research Design, Author Overlap, and Agreement in Findings

All intervention studies were coded for research design using the following categories: single case, group, and other. Studies that examined the effects of an intervention on a group or groups of participants were coded as group studies. Single-case studies were defined as studies in which results of individuals were compared with their baselines. One case study was reported among the coded studies. Interrater agreement for research design for the 24 double-coded studies was 96%. Studies that were coded as replications were also coded for author overlap and agreement in findings. Author overlap was considered when one or more of the authors of the child study also authored one or more of the parent studies. Interrater agreement for author overlap was 94%. We also determined if the author overlap in replication studies and prevalence of replications differed as a function of research design using chi-square tests.

The studies were then categorized for agreement in findings. The four categories of agreement in findings as defined by Cook and colleagues (2016) included the following:

Findings of parent studies not explicitly compared with findings from child study OR not clear whether comparison(s) indicate(s) agreement or disagreement between parent and child studies,

Findings of parent studies explicitly compared and noted only to agree,

Findings of parent studies explicitly compared and noted only to disagree, and

Findings of parent studies explicitly compared and noted to have mixed agreement/disagreement.

Interrater agreement for agreement in findings within the 16 identified replication studies out of the 24 double-coded studies was 81%.

Results

Rate of Replication in ECSE Intervention Research

To answer Research Question 1, 78 intervention studies published during 2012–2016 in three noncategorical ECSE journals were coded to ascertain if they were replication studies. Fifty-eight of the 78 intervention studies (74%) indicated that the purpose of the study was to replicate, expand, extend, add to, or use as their basis one or more previous intervention studies in the Introduction or Method section. Forty-seven (60%) of the studies included a comparison of the findings of previously conducted research with the results of the current study in the Results or Discussion sections. Of these studies, we identified 44 studies (56%) that included statements both in the Introduction or Method section and in the Results or Discussion section with at least one common parent study. In seven of these studies, (16%) authors specified that they attempted to replicate the previous study and were thus coded as direct replication. The authors of the remaining studies (n = 37; 84%) did not make this claim and were, therefore, coded as conceptual replications (see Table 1).

Table 1.

Journal Articles Reviewed (2012–2016).

Journal	Articles	Intervention studies	Replications
Journal	Articles	Intervention studies	Conceptual	Direct
Topics in Early Childhood Special Education	104	34 (33%)	17	3
Journal of Early Intervention	78	29 (37%)	11	2
Infants and Young Children	115	15 (13%)	9	2
Total	297	78 (31%)	37	7

Rate of Agreement in Findings

Of the 44 replication intervention studies, authors of 28 studies (64%) agreed with the findings of the parent studies. Authors of two studies (5%) reported that their findings did not agree with that of the parent study and authors of nine studies (21%) reported that their findings both agreed and disagreed with the parent studies. Authors of the remaining studies (n = 5; 10%) did not explicitly compare the findings to the parent study or their findings were not clearly presented to gauge whether they agreed or disagreed with the parent study.

Rate of Author Overlap

For the replication studies, we also calculated author overlap between the child study and the parent study and the percentage of overlap. Author overlap occurred for 20 studies out of 44 replications (46%). Chi-square analysis for whether the intervention study was a replication or nonreplication did not differ as a function of author overlap, χ²(1) = 0.2, p > .05. Furthermore, whether the agreements for studies differed as a function of author overlap also suggested a nonsignificant relationship, χ²(1) = 1.76, p > .05. Makel and colleagues (2012) cautioned that the high rate of author overlap between the replication studies is problematic and should be interpreted with caution due to the possible inherent author bias.

Research Design

The most often reported research design in the 78 coded intervention studies was single-case design (n = 43; 55%), followed by group design (n = 34; 44%); one case study design was reported. Twenty-one replication studies (61%) used single-case designs and 39% (n = 17) of the studies used group designs. Fifty percent (n = 17) of nonreplication studies used single-case designs and 47% (n = 16) of the studies used group designs. One nonreplication study used Case Study as the research design. Also, chi-square analysis for whether the agreements for studies differed as a function of study design was nonsignificant, χ²(1) = 0.7468, p > .05. We did not include studies using “other” designs in this analysis because of the small n. We further investigated trend of research designs in published intervention studies. As evident in Table 2, no clear pattern emerged.

Table 2.

Studies by Year and Design.

Year	Single case	Group^a	Total intervention studies
2012	4	4	8
2013	9	6	15
2014	6	10	16
2015	16	6	22
2016	8	9	17
Total studies	43	35	78

Includes one case study reported.

Discussion

EBP are intended to help practitioners provide effective interventions to those they serve. Dunst (2016) suggested four types of research syntheses of early childhood intervention research studies that can be used to analyze intervention research and can inform the identification of EBPs—narrative or traditional literature reviews, summative reviews that include quantification, systematic reviews that use exclusionary and inclusionary criteria to definite search strategies and analysis methods, and meta-analysis. In addition, Reichow (2016) operationalized EBP as a five-step process involving

(1) question formulation describing the characteristics of the patient or population, intervention, comparator, and outcome (e.g., PICO); (2) searching for and retrieving the best available research evidence; (3) critical appraisal of the research evidence; (4) selection of the intervention or intervention method based on available evidence, clinical expertise, and patient values and choice; and (5) continual progress monitoring. (p. 109)

The appraisal of research evidence is integral to this process and is critical to establishing an evidence base. Research appraisal can be addressed by replication of intervention studies by different researchers across settings and participants. This study investigated the quantity, type, and findings of replication research in ECSE.

Findings Compared

To ascertain the rate of published replication research in ECSE journals, we coded 297 articles published in three primary ECSE research journals between 2012 and 2016. This study replicates the methods described by Cook and colleagues (2016) whereby they investigated 289 articles published in six noncategorical research journals in special education in 2013 and 2014. Although Cook and colleagues reviewed research in special education in general and primarily research published in K–12 settings, the scope of this study was specific to research published in ECSE. Out of the 297 articles, 78 were coded as intervention studies (26%). This rate is slightly lower than that reported by Cook and colleagues (2016; 29%). However, using the same definition of replication as used by Cook and colleagues (2016; a replication statement in both Introduction and in the Results sections with one common parent study), we found that ECSE journals published a significantly larger number of replication studies (56%) than those reported by Cook and colleagues (31%; see Table 3).

Table 3.

Comparison of Findings of the Current Study With Cook, Collins, Cook, and Cook (2016).

Criteria/item	Current study	Cook et al. (2016)
Number of journals	3	6
Number of years	5	2
Scope of journals	Special education: Early childhood	Special education: Noncategorical
Total articles	297	289
Intervention studies	78 (26%)	83 (29%)
Replication statements reported in the Introduction	58	44
Replication statements reported in the Results	47	61
Replication studies	44 (56%)	26 (31%)
Direct Replications	9	0
Author Overlap	20	13

Another notable finding was that while Cook and colleagues did not find any direct replications, our investigation revealed 16% of studies reported direct replication statements. One of the possible reasons for this difference could be the considerably higher rate of single-case research designs we found in ECSE journals. Single-case designs are perhaps easier to replicate with fidelity due to the clinical and manualized nature of the interventions as well as the clear reporting tradition of the methods and results in research publications. Another possible reason for the higher rate of direct replications in ECSE could be a result of how the ECSE field views replication. Cook and colleagues (2016) observed that researchers often “look down upon” replication research and find them “difficult to publish” or “not prestigious” because journal editors and reviewers may think they do not provide a unique contribution; thus, authors may not explicitly report an investigation as a replication. The overall higher rate of both conceptual and direct replication in this study suggests that phenomemon might not hold true in ECSE research.

Current Trends in Intervention Research Published in Early Childhood Research Journals

Although the primary purpose of this investigation was to report on trends in replication research in ECSE, we also extrapolated additional findings on the trend in overall intervention research in ECSE. Our investigation clearly suggests a trend in single-case design studies, and a strong professional development element, conducted with adults (families or practitioners) with primary targeted participants being preschool-aged children in center-based settings. Six of the 78 intervention studies were conducted in home-based settings and 64 in center-based settings, including Head Start classrooms, university, community, and church-based preschools, district preschools, or elementary schools. In addition, 11 intervention studies were conducted in an online environment or in a clinical setting such as a hospital, facilitator’s office, or in therapy rooms. Some studies were conducted in more than one environment; for example, assessment may have been conducted in a therapist’s office but the intervention took place in the child’s preschool or an online professional development training was supplemented with onsite classroom training.

Forty-seven of the 78 intervention studies were conducted with preschool-aged children as the primary beneficiaries, 16 studies with infants or toddlers, and four studies reported they worked with children in primary grades. The remaining studies reported mixed child age groups. The outcome domain most reported was social-emotional, with language and literacy being the second most reported outcome domain. Other outcome areas reported included play, communication, and professional development focusing on collaboration, teacher burnout, goal development, and performance feedback.

Of note in this investigation was the increasing focus on interventions preparing professionals and family members to implement practices with fidelity. Thirty-seven studies in this investigation were conducted with children only; the remaining studies (n = 41) were conducted with adults such as teachers or family members only or as adult–child dyads. Different forms of coaching, mentorship were reported, as were varied modes of professional development (e.g., synchronous and asynchronous coaching and mentoring using email, in-person and online coaching methods; different combinations of these methods). There is growing evidence that prolonged coaching supplemented with some forms of technology-based training can play a role in helping improve practices in inclusive environments. Simultaneously, there has been an identified need in the field of ECSE for more “rigorous experimental examination of EC PD [early childhood professional development] in recent years” (Snyder, Hemmeter, & McLaughlin, 2011, p. 364). Our findings align with the recent focus from researchers, policy makers, and practitioners on professional development for early childhood professionals as accountability for outcomes is high and demands for qualified early childhood practitioners have increased.

Limitations

This exploration of replication research in ECSE was based on methods used by Cook and colleagues (2016). Thus, our limitations mirror some of the limitations they mention in their publication. For example, the definition of replication itself used in this study may be of debate. Two-part definition of replication used by Cook and colleagues (2016), discussed above, was used in this study. However, other researchers have suggested more rigid definitions of replication (e.g., Lemons et al., 2016; Makel et al., 2016) or more generous definitions of replication (e.g., Therrien, Mathews, Hirsch, & Solis, 2016), which could result in different and even contradictory results. Lemons and colleagues (2016) and Makel and colleagues (2016) searched electronically for replicat* in all articles, including nonempirical publications. Thus, studies where the authors did not clearly describe their study as a replication were not identified. Using this strict criterion, their investigation produced the least proportion of studies. In contrast, Therrien and colleagues (2016) identified replications by examining the similarity between the interventions used in the parent and child studies, regardless of whether the child studies indicated that they replicated the parent study. Using this broad definition, they found that the majority of intervention studies they examined were replicated multiple times. In our study, we used a somewhat middle ground whereby we examined functional replications that met our two criteria mentioned earlier regardless of whether the authors themselves referred to them as replications. In fact, in our study, we found only eight out of the 44 replication studies used the word “replicat.” Thus, if we had used the formal and restrictive definition of replication, we would have identified only 2.7% of the 297 published articles (vs. the 56% we found) as replications; a rate similar to Lemon and colleagues, Makel and colleagues, and Cook and colleagues.

Second, in an effort to answer Research Question 2, we completed a title and reference analysis of the parent studies to determine if they were indeed intervention studies. We did not do a more thorough independent review to verify the findings of the parent studies to determine whether or not the findings agreed or differed from what was reported in the child study. Finally, we used a small sample of publications in three ECSE journals over 5 years only. Although these three journals are the primary publication avenue for ECSE researchers, they by no means represent the entire trend of published research in ECSE. Replication rates may differ by journals or by the years of publication or both. Furthermore, Cook and colleagues mention other limitations that apply to our investigation as well (e.g., small number of studies for chi-square analysis resulting nonsignificant results of author overlap).

Recommendations for Research, Policy, and Practice

Although an increased percentage of conceptual and direct replication than those reported by Cook and colleagues (2016) were observed, the overall lack of more direct replication studies published in ECSE journals is concerning. Authors seldom explicitly state their intent to replicate due to some of the biases toward replication noted above. One example of direct replication was reported by Appelman, Vail, and Lieberman-Betz (2014). The authors explicitly stated in the introduction, “The current study attempted to replicate and extend findings of Ledford and colleagues (2008)” (p. 132). The authors very clearly articulated whether and the degree to which they replicated previous research and which elements of a parent study were and were not replicated in their research. They further provided a clear discussion about the agreement and disagreement between their findings and those of the parent study. For example, in their discussion, they state that “the mean number of sessions and ranges reported for acquisition of target information in the current study is comparable with the mean number of sessions to criterion for targeted information reported by Ledford and colleagues (2008) . . . ” followed by “In contrast to the study by Ledford and colleagues (2008) . . . ” (p. 145). Another example of replication was described by McCollum, Hemmeter, and Hsieh (2013) where the authors stated the explicit intent to replicate their study with another previous study and provided comparison of their findings with that of the parent study.

We recommend that steps be taken to support conducting and publishing replications, especially direct replications, and building a culture in early childhood that promotes and values replication research—both single-case and group designs. For example, future and early researchers in ECSE should be encouraged and provided ample opportunities to conduct research when in the doctoral leadership program with an explicit and clear intent to replicate and extend a previously conducted parent study. The direct replication of peer-reviewed, published research not only prepares the future leaders in strong research methodology, it also builds on the scientific evidence for practices.

Journal editors and reviewers also will need to be more open to publishing replication research when a clear goal of building evidence is presented. Furthermore, reviewers and journal editors may require more transparency from authors about the origins of their research and the data presented in their research so that future researchers may replicate the methods with fidelity and compare findings with the parent study. Similarly, replication of large group designs with fidelity will require significant funding and thus, support from agencies to finance replication research is critical. Higher education programs too will need to demonstrate a culture of replication research by valuing direct and conceptual replication studies and publications in promotion and tenure decisions (Cook et al., 2016).

With regard to the overall trends in ECSE research, it is encouraging to see the increased focus on professional development with families and early childhood professionals to increase fidelity of implementation and scaling of practices in early childhood settings. However, additional findings observed in our study suggest that more research is needed in early intervention environments, with families, and use of group designs. Only 21% of the research was conducted with infants and toddlers as the beneficiaries of the intervention, compared with 60% of the research with preschool-aged children. In addition, early childhood researchers seem to rely heavily on single-case designs as their primary investigation methodology. This is understandable given the diversity of the interventions, populations, and settings of the research being undertaken. It also reinforces the importance of conducting replication research that adheres to the quality indicators for single-case design recommended by Council for Exceptional Children (2014) and the What Works Clearinghouse (WWC; Institute of Education Sciences, 2013) for determining EBP in special education. Specifically, to be considered a body of evidence equaling an EBP by the WWC, a group of single-case design studies needs to (a) be evaluated through a minimum of five single-case design studies that meet the quality indicators, (b) be conducted by research teams at three different institutions, and (c) have a total of 20 or more participants across the studies being evaluated (Kratochwill et al., 2010). However, group experimental designs using randomized controlled trials (RCT) would require a smaller number of investigations to consider a practice as evidence based. Two RCT conducted by two independent research groups are considered the “gold-standard” criteria regarding the amount of evidence required for an intervention to be considered evidence based (Reichow, 2016). Thus, in conclusion, the results of this study indicate that there is a need for increased replication of research studies that use both single-case designs that attend to the quality indicators as well as high-quality RCT research that involves large groups of participants conducted by independent researchers across settings and age groups.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Appelman

Vail

C. O.

Lieberman-Betz

R. G.

(2014). The effects of constant time delay and instructive feedback on the acquisition of English and Spanish sight words. Journal of Early Intervention, 36, 131–148.

Artman-Meeker

Fettig

Barton

E. E.

Penney

Zeng

(2015). Applying an evidence-based framework to the early childhood coaching literature. Topics in Early Childhood Special Education, 35, 183–196. doi:10.1177/0271121415595550

Barton

E. E.

Smith

B. J.

(2015). The preschool inclusion toolbox: How to build and lead a high-quality program. Baltimore, MD: Brookes.

Burns

M. K.

Ysseldyke

J. E.

(2009). Reported prevalence of evidence-based instructional practices in special education. The Journal of Special Education, 43, 3–11. doi:10.1177/0022466908315563

Buysse

Wesley

P. W.

Snyder

Winton

(2006). Evidence-based practice: What does it really mean for the early childhood field? Young Exceptional Children, 9, 2–11. doi:10.1177/109625060600900401

Cook

B. G.

Collins

L. W.

Cook

S. C.

Cook

(2016). A replication by any other name: A systematic review of replicative intervention studies. Remedial and Special Education, 37, 223–234. doi:10.1177/0741932516637198

Cook

B. G.

Cook

S. C.

(2013). Unraveling evidence-based practices in special education. The Journal of Special Education, 47, 71–82. doi:10.1177/0022466911420877

Cook

B. G.

Odom

S. L.

(2013). Evidence-based practices and implementation science in special education. Exceptional Children, 79, 135–144. doi:10.1177/001440291307900201

Council for Exceptional Children. (2014). Council for Exceptional Children standards for evidence-based practices in special education. Exceptional Children, 80, 504–511. doi:10.1177/0014402914531388

10.

Coyne

M. D.

Cook

B. G.

Therrien

W. J.

(2016). Recommendations for replication research in special education: A framework of systematic, conceptual replications. Remedial and Special Education, 37, 244–253. doi:10.1177/0741932516648463

11.

Dunst

C. J.

(2016). Role of research synthesis for identifying evidence-based early childhood intervention practices. In Reichow

Boyd

B. A.

Barton

E. E.

Odom

S. L.

(Eds.), Handbook of early childhood special education (pp. 541–563). Basel, Switzerland: Springer.

12.

Dunst

C. J.

Trivette

C. M.

Cutspec

P. A.

(2002). Toward an operational definition of evidence-based practice. CenterScope, 1, 1–10.

13.

Earp

B. D.

Trafimow

(2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6, 1–11. doi:10.3389/fpsyg.2015.00621

14.

Every Student Succeeds Act of 2015, Pub. L. No. 114-95, 114 Stat. 1177 (2015–2016).

15.

Francis

G. L.

(2012). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585–594. doi:10.1177/1745691612459520

16.

Fuchs

L. S.

(2005). Peer-assisted learning strategies promoting word recognition, fluency, and reading comprehension in young children. The Journal of Special Education, 39, 34–44.

17.

Fuchs

L. S.

(2014). Rethinking service delivery for students with significant learning problems: Developing and implementing intensive instruction. Remedial and Special Education, 36, 105–111. doi:10.1177/0741932514558337

18.

Gall

M. D.

Gall

J. P.

Borg

W. R.

(2003). Educational research: An introduction (7th ed.). Boston, MA: Pearson Education.

19.

Individuals With Disabilities Education Act, 20 U.S.C. § 1400 (2004).

20.

Institute of Education Sciences. (2013). WWC procedures and standards handbook version 3.0. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_v3_0_standards_handbook.pdf

21.

Ioannidis

J. P. A.

(2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7, 645–654. doi:10.1177/1745691612464056

22.

Jasny

Chin

Chong

Vignieri

(2011). Again, and again, and again . . . Science, 334, Article 1225. doi:10.1126/science.334.6060.1225

23.

Kratochwill

T. R.

Hitchcock

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2010). Single-case designs technical documentation. Retrieved from https://ies.ed.gov/ncee/wwc/Document/229

24.

Lemons

C. J.

King

S. A.

Davidson

K. A.

Berryessa

T. L.

Gajjar

S. A.

Sacks

L. H.

(2016). An inadvertent concurrent replication: Same roadmap, different journey. Remedial and Special Education, 37, 213–222. doi:10.1177/0741932516631116

25.

Makel

M. C.

Plucker

J. A.

(2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43, 304–316. doi:10.1037/a0035811

26.

Makel

M. C.

Plucker

J. A.

Freeman

Lombardi

Simonsen

Coyne

(2016). Replication of special education research: Necessary but far too rare. Remedial and Special Education, 37, 205–212. doi:10.1177/0741932516646083

27.

Makel

M. C.

Plucker

J. A.

Hegarty

(2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7, 537–542. doi:10.1177/1745691612460688

28.

McCollum

Hemmeter

M. L.

Hsieh

(2013). Coaching teachers on emergent literacy instruction using performance feedback. Topics in Early Childhood Special Education, 33, 28–37.

29.

Meadan

Angell

M. E.

Stoner

J. B.

Daczewitz

M. E.

(2014). Parent-implemented social-pragmatic communication intervention: A pilot study. Focus on Autism and Other Developmental Disabilities, 29, 95–110.

30.

Meadan

Snodgrass

M. R.

Meyer

L. E.

Fisher

K. W.

Chung

M. Y.

Halle

J. W.

(2016). Internet-based parent implemented intervention for young children with autism: A pilot study. Journal of Early Intervention, 38, 3–23. doi:10.1177/1053815116630327

31.

Nosek

Bar-Anan

Motyl

(2012). Scientific utopia II: Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. doi:10.1177/1745691612459058

32.

Odom

S. L.

(2009). The tie that binds: Evidence-based practice, implementation science, and outcomes for children. Topics in Early Childhood Special Education, 29, 53–61. doi:10.1177/0271121408329171

33.

Pashler

Harris

(2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531–536. doi:10.1177/1745691612463401

34.

Reichow

(2016). Evidence-based practice in the context of early childhood special education. In Reichow

Boyd

B. A.

Barton

E. E.

Odom

S. L.

(Eds.), Handbook of early childhood special education (pp. 107–121). Switzerland: Springer. doi:10.1007/978-3-319-28492-7

35.

Schmidt

(2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90–100. doi:10.1037/a0015108

36.

Simons

D. J.

(2014). The value of direct replication. Perspectives on Psychological Science, 9, 76–80. doi:10.1177/1745691613514755

37.

Snyder

Hemmeter

M. L.

McLaughlin

(2011). Professional development in early childhood intervention: Where we stand on the silver anniversary of PL 99-457. Journal of Early Intervention, 33, 357–370.

38.

Therrien

W. J.

Mathews

H. M.

Hirsch

S. E.

Solis

(2016). Progeny review: An alternative approach for examining the replication of intervention studies in special education. Remedial and Special Education, 37, 235–243. doi:10.1177/0741932516646081

39.

Travers

J. C.

Cook

B. G.

Therrien

W. J.

Coyne

M. D.

(2016). Replication research and special education. Remedial and Special Education, 37, 195–204. doi:10.1177/0741932516648462