Becoming Critical Consumers of Research: Understanding Replication

Abstract

Special educators are tasked with understanding and implementing evidence-based practices. The purpose of this article is to highlight the salient features of replication in special education research and explain how practitioners can use this knowledge to become critical consumers of research. To do this we describe replication, emphasize the importance of understanding replication in special education research, provide a list of questions for consumers of research, explain the process of replication using an example, and offer resources for critical consumers to access.

Keywords

evidence-based practices replication intervention special education curriculum

The following vignette describes an experience shared by many teachers as they engage with new research. (see Note 1). Mrs. Harris and Mr. Williams just returned from a professional development seminar focused on research-based practices for teaching students with disabilities. Though these special education teachers attended the same presentation, their reactions to the new practice are different. Mrs. Harris, excited to have a new tool at her fingertips, is ready to put these new practices in place. On the other hand, Mr. Williams, tired of hearing about the revolving door of research-based district initiatives that seem to lack quality and practicality, is ready to put the binder describing the new practices in the recycling bin. Is one of these approaches to research correct or do these teachers’ reactions bring to light potentially problematic ways teachers consume research? Could a deeper understanding of how an evidence base develops provide these teachers with the tools to make informed decisions regarding the practices, strategies, and interventions they choose to implement in their classrooms?

Questioning Research

Mrs. Harris’s and Mr. Williams’s reactions are indicative of teachers’ experiences with research in their professional environment (Carnine, 1997; Jones, 2009). Through professional development (PD), new policy initiatives, and their own reading and research, special educators like Mrs. Harris and Mr. Williams frequently encounter new practices, exciting ideas, and the latest trends in educational practice. Because special educators are in the business of finding solutions and want the best for students with disabilities (SWD), they often seek out strategies that are supported by research. Understandably, all this information can lead to questions about the value of research and the ways in which the quality and quantity of research supporting a practice add to or detract from the evidence base. Questioning research, becoming a critical consumer, is one of the roles special educators can take on as they work to affect classroom-, school-, and district-level decisions.

It is important that practitioners understand that not all research is equal. The quality, depth, and accumulation of research vary, and, for this reason, it is important to consider each of these three factors when deciding whether to implement a strategy or practice. With quality, depth, and accumulation of research as their focus, special educators can become critical consumers of research to select practices that are strongly rooted in empirical evidence and therefore advocate for SWD. But what does this really mean for teachers like Mr. Williams and Mrs. Harris and other individuals in their schools and districts tasked with selecting practices, strategies, and interventions for SWD? What tools can they use to evaluate instructional strategies or practices? More important, how might they decide which practices are most likely to return positive results for their students?

Replication Research Is Relevant to Classroom Work

One way for individuals like Mrs. Harris and Mr. Williams to better support SWD is to adopt a replication framework when evaluating the strategies and practices promoted in their professional setting. Replication is the practice of confirming the results of previous research by conducting a similar study (Coyne, Cook, & Therrien, 2016). Though practitioners are not necessarily in a position to replicate research themselves, they are responsible for selecting and implementing empirically sound practice in their classroom. Because of this, it is vital that teachers and other practitioners understand the difference between a practice that has been tested through high-quality replication and one that has been researched. This is the foundation of a replication framework. Understanding the meaning of replication better equips teachers and other consumers of research to critically examine decisions in the planning and delivery of instruction and intervention for SWD.

To help teachers, school-based teams, and district-level personnel select evidence-based practices (EBPs) that are supported by rigorous, replicated scientific research, this article includes a discussion of the meaning and importance of replication and the following tools to support critical consumption of research: (a) a list of terms to know when evaluating and critiquing research, (b) guidance regarding resources practitioners can use to access replicated research, (c) a questionnaire for evaluating the research base for practices, strategies, and interventions, and (d) an example of a practice that has been tested through replication.

The Importance of Replication

Although the terms evidence-based practice and research-based practice are often used synonymously, they hold different meanings. An EBP is an instructional strategy or practice that has returned positive results across multiple rigorous studies (Cook & Cook, 2013). Simply stated, EBPs must show a clear cause and effect relationship with improved student outcomes using group comparison (e.g., randomized control studies) or replicated single-subject design research. EBPs are evaluated by high-quality experimental tests and published in high-quality peer-reviewed journals (Travers, 2016). In contrast, a research-based practice is an instructional strategy or program that has been studied in some way but has not been replicated by other researchers (Kretlow & Blatz, 2011). Thus, EBP and research-based practice are not the same. What often differentiates the two is the extent to which research has been replicated. When considering a new practice, understanding the extent and quality of the research serves as the linchpin for ascertaining whether the practice is evidence-based or research-based. Table 1 offers some critical definitions to know when evaluating and critiquing research.

Table 1.

Terms to Know When Evaluating Research.

Term	Definition
Conceptual replication	Tests the same strategy or practice as the original study but purposefully adjusts some aspect of the original study, such as grade level or exceptionality
Direct replication	An exact copy of an original study using the same procedures with no purposeful changes from those procedures
Evidence-based practice	An instructional practice or strategy that has been found to be effective by multiple rigorous and reliable studies
Meta-analysis	Systematic summary of statistical findings derived from multiple studies of a practice or strategy with results combined and presented across studies
Replication	Repeated, purposeful research regarding a strategy or practice
Synthesis	Summary of the evidence related to a particular strategy or practice; includes multiple studies but does not calculate statistical effects

In scientific research, each replication is a “brick in the wall” of evidence surrounding a practice across multiple, varied contexts. For example, a researcher may study the same social skills curriculum multiple times with different groups of students, in different grade levels, and across different settings. With each replication, researchers work to enhance their understanding of the strengths and limitations of the tested educational practice. In this, the replications serve as a way to determine whether previously positive reports were the result of chance, or lack of rigorous study, and as a way to demonstrate the applicability of research findings in practice (Schmidt, 2009). Replications also test the bounds of an intervention. For example, a practice may work for elementary school students but not for middle school students.

Types of Replication

Each type of replication provides critical consumers with a different window into the strength and applicability of an educational practice. Researchers often divide replications into categories: (a) direct and (b) conceptual (Makel et al., 2016). Direct replications “attempt to copy the same research procedures as the original study as closely as possible with no purposeful deviation” (Makel et al., 2016, p. 207). They are the strictest type of replication and, therefore, are the least common in educational research. For critical consumers, direct replications serve as a check on the strength of an intervention as it was originally conducted and tested with the same population under the same circumstances.

Conceptual replications, on the other hand, “purposefully alter some aspect [of the original study]” (Makel et al., 2016, p. 207). In conceptual replications, researchers might study an intervention with different students or test the effectiveness of an intervention using a different type of measure. Conceptual replications are more common in educational research. Conceptual replications provide teachers, or critical consumers, with a better understanding of the ability of certain practices to reach different groups of students with varied needs across numerous classrooms.

Replication Is Important

Coyne and colleagues (2016) called for researchers to conduct series of conceptual replications that involve systematically changing one or more features of the original study (e.g., setting, participant, outcome measure, research design, and analysis). In repeatedly testing a practice, researchers are able to build an argument for its validity and meaningfulness across contexts (Coyne et al., 2016; Therrien, Mathews, Hirsch, & Solis, 2016). For critical consumers, applying their understanding of conceptual replications can help guide the selection of practices that will be most likely to produce positive effects for the students in their classrooms. Replication is important because it provides a way for researchers, teachers, and policy makers alike to understand the effectiveness of a practice across a wide body of students, settings, and circumstances.

Systematic replications are a way of building an argument in support of an intervention or practice across diverse settings. For example, a researcher might first study the effectiveness of a social skills intervention with third graders with autism spectrum disorders (ASDs) in self-contained settings and then replicate the study with third graders with ASD in inclusive settings. If the results of the original study are confirmed in a subsequent conceptual replication, this adds to the body of evidence regarding the efficacy of this practice. Replications are important because they provide a way to answer the following questions:

For whom has the intervention or practice been proven effective?

In which settings has this intervention or practice been proven effective?

Which components are essential for this intervention or practice to be effective?

Which assessments have proven the effectiveness of this intervention or practice?

Student population

To start, conceptual replications are instructive in assessing the population of students that benefit from an intervention or practice (Coyne et al., 2016). While the aforementioned example noted the setting changed from a third-grade self-contained setting to a third-grade inclusive setting, the population was held constant. In examining the effects of the intervention across developmental ages, researchers might assess whether it is effective for students with autism who are in middle or high school. A researcher looking to test the effectiveness of an intervention across different populations might also take the same social skills curriculum and, because it has been found to be effective with third graders with ASD, decide to test it with third graders with learning disabilities or traumatic brain injuries. In this way, the researchers purposefully alter a characteristic of the sample but hold the intervention constant. As critical consumers review replication research, they can consider the characteristics of the students for whom the practice has been found to be effective and whether their students’ needs are consistent with regard to key variables such as developmental age or disability.

Setting

Travers, Cook, Therrien, and Coyne (2016) noted that one of the strengths of conceptual replication is that research can attest to the effects of an intervention across similar, but not identical settings. In the previous example, the researcher changed the setting in which the intervention was delivered from a self-contained to an inclusive classroom with a similar population of students (i.e., third graders with ASD). When considering a conceptual replication of this work, researchers could alter the instructional setting (e.g., self-contained, cotaught, general education, community) or the classroom makeup (e.g., homogeneous grouping or heterogeneous grouping). Each time a researcher alters the setting, this adds to or delimits the practice’s evidence base.

Essential components

Another important aspect to consider is which components of the intervention have been tested across replications (Makel et al., 2016). This includes knowing any number of variables that are a part of implementing a practice in school settings: (a) frequency and duration of treatment, (b) the materials and supports used, (c) the individual delivering the intervention, and (d) the focus of the intervention. Examining the social skills intervention described above, researchers might alter the frequency of instructional sessions to assess whether it can have the same efficacy in fewer sessions. Alternatively, a researcher might adjust the intervention to see if there is a way to scale back the materials a student or interventionist would need to manage during instruction. Attending to the essential instructional components across replications also highlights the extent to which a replication is actually replicating the practice as intended. If something essential was altered from the original study, it might have different effects in the classroom.

Measurement

Replication is also important because studies can produce different results on different measures. There is a wide variety of ways researchers can measure a skill; these fall along a continuum of researcher-created measures (i.e., those that are closely aligned to the intervention) to norm-referenced measures (i.e., more global measures of knowledge and skills). While both tools measure the effectiveness of an intervention, they may vary with regard to the validity and reliability of the assessment. An original study might use a series of researcher-created measures, while a replication might attempt to verify the veracity of an intervention through more stringent, norm-referenced measures. Because special education often uses norm-referenced measures to assess students’ needs and gains, considering the ways in which researchers have assessed the efficacy of a practice is extremely important.

Finding Replicated Research

Practitioners and researchers alike are often concerned that much of the research conducted on educational practice is inaccessible (Carnine, 1997). However, there are a number of ways for critical consumers to access empirically sound practice beyond the formal PD experience. These tools are not meant to overwhelm practitioners. Instead, access to these tools can empower practitioners to examine their own practice and question the practices, strategies, and interventions presented to them as research-based through other avenues.

First, there are several online resources that provide critical consumers with free access to replicated research. These include the IRIS Center’s Evidence-Based Practice summaries (http://iris.peabody.vanderbilt.edu), the What Works Clearinghouse Practice Guides (http://ies.ed.gov/ncee/Wwc/), and the National Professional Development Center on Autism Spectrum Disorder’s Autism Focused Intervention Resources and Modules (http://afirm.fpg.unc.edu/afirm-modules). Each of these resources includes organized and easily searchable information regarding many of the programs and practices used in schools today. Because of this, they are helpful when considering how implemented practice have been replicated. These dissemination platforms are part of the larger movement to support teachers, collaborative teams, and building and district leaders in implementing EBPs in the classroom.

Two other important tools are research syntheses and meta-analyses. Both are systematic summaries that combine the results of many studies to determine the effectiveness of an intervention, strategy, or practice for a specific population of learners. Syntheses are narrative reviews that provide summaries of several studies without statistically combining results. Meta-analyses are systematic summaries of research that combine the results of multiple studies into a statistical value called an effect size. This value, the effect size, can help critical consumers understand how beneficial an intervention, practice, or strategy is likely to be for a specific population of students. Both syntheses and meta-analyses can describe the results, population, and settings across which practices and strategies are studied, but meta-analyses’ attention to effect size is important for critical consumers to consider. In examining effect sizes across multiple studies, or replications, presented in meta-analyses, consumers can better understand the empirical landscape and select practices that researchers have found to be statistically effective across multiple studies. Banda and Therrien (2008) provided a practitioner-friendly description of how to interpret effect sizes in intervention research.

Syntheses and meta-analyses are the resources professional organizations such as those listed above, educational advocates, and educators with experience in research use to access the body of evidence surrounding a practice (Carnine, 1997). Using simple search terms (i.e., mathematics for students with disabilities + meta-analysis), individuals and collaborative teams interested in digging deeper into the research can often access these tools through Google Scholar (https://scholar.google.com). Though this approach is more complicated than using the WWC, IRIS, or the AFFIRM modules, it is not out of the realm of possibility for educators.

Evaluating Research in a Replication Framework

Critical research consumers are often in the position of trying to decide whether the strategies being presented in PD, in district initiatives, and more broadly in popular culture will be effective for their students. As special educators collaborate with parents, general educators, administrators, and students, they know one of their responsibilities is to plan and monitor the effectiveness of instruction. This is certainly not something to be taken lightly and is precisely why it is important to understand replication.

It is imperative that special educators become informed consumers of research. One way to assume this professional empowerment is to evaluate the research base available for interventions, strategies, and practices presented by researchers and policy makers through PD experiences and school, district, and other policy initiatives. Figure 1 outlines six questions that, when answered, give critical consumers of research a better understanding of the extent to which practices have been replicated and might result in positive student outcomes. These questions are connected to the reasons why understanding replication is important and useful in the classroom. They serve as the basis for a practitioner-friendly replication framework—a way to move past the frustrations of competing or confusing messages about research-based strategies and toward understanding effective EBPs. Asking these questions of research and researchers is one way that practitioners can increase the likelihood that the practices they implement in the classroom will return positive results for students. Based on the Therrien et al. (2016) review of replication in special education research, six critical questions emerged. Each of the six questions falls into one of three categories: (a) before, (b) during, or (c) after implementation.

Figure 1.

Questions for Critical Consumers to Ask.

Before Implementation

Prior to identifying an EBP to implement, four questions are important to answer. The first question is: Has the practice or strategy been replicated? If the answer is no then, without replication, the results might be due to a number of factors (e.g., chance, poorly designed research). Research that has not been replicated cannot be considered an EBP. The second question is: Who was included in the replication sample? An instructional approach that has been validated only by research conducted on groups of students who differ from your student(s) on key variables (e.g., disability, age) may be ineffective with your student(s). The third question is: What was the setting of the study? A strategy, practice, or intervention that has only been proven effective in a single grade level (e.g., preschool, elementary, secondary) or in a specific setting (e.g., self-contained, general education) may not be effective in your particular classroom setting. The fourth question is: Were the results consistent? Consistently positive results indicate that a practice is likely to be effective in the classroom. When the results of research are inconsistent, a practice or strategy is less likely to be effective.

During Implementation

Critical consumers should ask question five during implementation: Did I implement the practice or strategy with fidelity? Fidelity refers to implementing a practice as originally designed (Keller-Margulis, 2012). Direct and indirect fidelity methods are available for teachers to record data (Keller-Margulis, 2012). Direct methods involve an observation by an outside observer who watches and collects data using an implementation checklist. Indirect methods, on the other hand, involve self-report. For example, a fidelity checklist or rating scale can be completed to note the implemented components of the practice. In addition to being a critical component of determining whether the strategy or practice was implemented as intended, treatment fidelity also helps consumers of research make accurate decisions regarding a student’s response to an intervention (Bruhn, Hirsch, Gorsh, & Hannan, 2014).

After Implementation

Critical consumers should ask question six after they have implemented a practice or strategy: To what extent did student data demonstrate desired effects? Even interventions that have been found effective by many replication studies may not work with your students. For this reason, it is important to collect data to evaluate strategies and practices with students in a classroom. These data should include student-level outcomes as well as fidelity data. For example, if the intervention was not effective and fidelity was less than optimal, then it might be beneficial to pursue additional training or support to improve implementation in your school setting (Bruhn et al., 2014). On the contrary, if the intervention was implemented with fidelity and student-level data do not reveal desired changes, consider (a) a different or more intense intervention or (b) additional assessment (Bruhn et al., 2014).

Applying a Replication Framework: Anchored Instruction

Researchers have tested many practices in special education and found them to be effective through multiple replications. The following example illustrates how understanding replication could improve the likelihood of classroom practice returning positive results. Anchored instruction (AI) is an instructional strategy designed to help students build background knowledge and develop mental models (Bottge, 1999). AI provides students with information often in the form of an image (moving, still, or live) about a topic and then anchoring the image in successive learning experiences. The following section presents some of the replication research on AI. This demonstrates how critical consumers like Mrs. Harris and Mr. Williams might apply the questions from the replication framework to determine if AI is an appropriate practice to use with their students. In Figure 2, you will be able to trace the development of the effectiveness of AI for various groups of students and draw conclusions regarding AI.

Figure 2.

Applying the Critical Questions Before Selecting Interventions and Practices: An Example.

Bottge’s (1999) original study examined the effects of AI on eighth graders’ ability to solve contextualized problems and included students with learning disabilities, other health impairment, speech or language impairment, emotional and behavioral disorders, and attention-deficit/hyperactivity disorder. The intervention was conducted over 10 days across two settings: (a) prealgebra and (b) remedial mathematics classrooms. Using multiple measures, Bottge (1999) found that students who received the AI intervention across both settings had significantly better scores on a test of contextualized problem solving than their peers who had not been exposed to the AI intervention. However, in this study, the researcher did not find differences on word problem and fraction computation measures. In the next steps, the information on the original research is used to answer the Before Implementation questions to assess the practice.

Has the Practice or Strategy Been Replicated?

The answer to the first question is yes. Research teams have replicated Bottge’s (1999) work regarding AI more than 12 times since the original research (Bottge, Grant, Stephens, & Rueda, 2010; Bottge, Heinrichs, Chan, Mehta, & Watson, 2003; Bottge, Rueda, LaRoque, Serlin, & Kwon, 2007; Bottge, Rueda, & Skivington, 2006; Bottge & Watson, 2002; Cho, Cohen, & Bottge, 2013; Mulcahy & Krezmien, 2009; Zydney, Stegeman, Bristol, & Hasselbring, 2010). Given that AI has been the subject of prolific replication, one can consider for whom and in which settings AI is most likely to be effective.

Who Was Included in the Replication Sample?

A variety of different types of students were included in the replication studies. Six studies used samples that were similar to the original study to thoroughly evaluate the effectiveness of AI for middle school students with high-incidence disabilities (Bottge et al., 2003; Bottge et al., 2007; Bottge et al., 2010). In addition, researchers have studied AI with middle school students with mathematics learning disabilities (Bottge et al., 2007), samples of middle school students including those with more severe cognitive disabilities and ASD (i.e., Cho et al., 2013), middle school students with emotional and behavioral disorders (i.e., Mulcahy & Krezmien, 2009), at-risk high school students with challenging behaviors (Bottge et al., 2006), and young adults with comorbid severe mental illness and learning disabilities in reading and math (Bottge & Watson, 2002). One study examined the effects of AI for fifth-grade students with and without disabilities (Zydney et al., 2010).

From this information, it is evident that researchers have studied AI with many groups of adolescents, but that they have not comprehensively examined the effectiveness of the practice with upper elementary school students. Furthermore, one can ascertain that researchers have validated AI as a powerful instructional approach for addressing the needs of mixed groups of middle schools students. Using this information, special educators could decide whether the research around this practice points to its utility in their classroom and whether considering the next set of questions would be beneficial.

What Was the Setting of the Study?

In examining the setting, a critical consumer might look at the subject area and the classroom setting. Whereas the original study was conducted in prealgebra and remedial mathematics classes (Bottge, 1999), replications were conducted in a number of different settings. However, while the classroom setting was varied, the use of AI for teaching mathematics content was consistent across all reviewed studies. The settings studied by researchers include (a) general education (Zydney et al., 2010), (b) technology education (Bottge et al., 2010), (c) remedial mathematics (Bottge et al., 2003), (d) self-contained classrooms (Bottge et al., 2007; Cho et al., 2013; Mulcahy & Krezmien, 2009), (e) alternative charter schools (Bottge et al., 2006), and (f) forensic mental health facilities (Bottge & Watson, 2002). From this, the critical consumer can see that AI has been tested across a continuum of classroom settings.

Thus far it is clear that researchers have replicated AI studies, and the majority of the studies help to understand the effects of AI on adolescent students’ performance in mathematics across settings. However, at this point, it is not yet clear whether the results of these replications tell a consistent story about AI. For this, move on to the final Before Implementation question.

Were the Results Consistent?

Finally, the critical consumer needs to know how effective AI was for these varying groups of students. Across multiple studies, researchers have found that AI improves students’ performance on contextualized problems (Bottge et al., 2003; Bottge et al., 2007; Bottge et al., 2010) and that AI helps students in remedial mathematics classes make progress similar to their peers in general education (Bottge et al., 2001).

For SWD, the results of the replications are more complex and highlight some important instructional implications. First, multiple studies found that the results of AI are not significant for SWD across all measures. SWD show improvements on contextualized problem measures but not on computation problems or word problems Bottge et al., 2006). Second, SWD need additional support to address deficits in background knowledge related to the context of the problems and computational skills (Bottge et al., 2007; Mulcahy & Krezmien, 2009). One study addressed this discrepancy by examining teacher practice and student need (Cho et al., 2013). These researchers found that, without organized, carefully planned instruction and fidelity of implementation, the majority of students with low ability would not make the same progress as students with high ability. Based on this, they concluded that, even though they made marked progress, students with cognitive disabilities would benefit from more than an 18-week intervention.

These findings underscore some important considerations for critical consumers as they decide whether to implement AI (i.e., our example practice) with SWD. First, it is clear that AI has the potential to support the development of conceptual understanding across settings for middle school SWDs. However, it is also recommended that if teachers were to implement this practice, they would need to provide explicit instruction in computation. In addition, they would need to consider students’ potential deficits in background knowledge and the extent to which they could implement the practice with fidelity. After examining these studies in a replication framework, a critical consumer might conclude that AI cannot be the only instructional program for SWDs, but that it could be an integral component of a larger program.

Where Do Critical Research Consumers Go from Here?

The example of AI highlights the importance of considering replication in selecting instructional practices. By applying a replication framework when selecting educational practices, special education teachers and other school and district personnel can take on the role of the critical consumer and evaluate the practices, strategies, and interventions presented to them through PD, district initiatives, changes in policy, or even in popular culture. By asking the “Before Implementation” questions of proponents of different strategies, critical consumers like Mrs. Harris can carefully evaluate new practices instead of blindly accepting research and those like Mr. Williams can advocate for EBPs in their schools instead of rejecting new learning. They can assert that they have choices in the practices they implement. By basing their instructional decisions in a replication framework, critical consumers are empowered to make evidence-based decisions and to advocate for best practice for their students. These questions become a tool that consumers can use to guide their instruction and as a form of advocacy for the most effective practice for SWDs.

Returning to the Vignette

Mrs. Harris and Mr. Williams opted to get together to discuss the instructional approach they learned during the PD: Anchored Instruction (Bottge, 1999). They reviewed the questions for critical consumers and came to some important decisions. Because Mrs. Harris works in an elementary school self-contained setting, she decided that AI might not be right for her students. While she found the work interesting, AI did not seem to meet the needs of her students. However, Mr. Williams, a high-school special educator who frequently collaborated with the engineering teacher to help promote student engagement, realized that AI might be an important addition to his students’ learning experiences when paired with explicit instruction in computation. However, he also knew that choosing AI as an instructional component for his students was not the end of the road. He and Mrs. Harris agreed that going forward they would meet periodically as colleagues and friends and continue to evaluate whether and how effective AI is for his students.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Banda

D. R.

Therrien

W. J.

(2008). A teacher’s guide to meta-analysis. Teaching Exceptional Children, 41(2), 66–71.

Bottge

B. A.

(1999). Effects of contextualized math instruction on problem solving of average and below-average achieving students. Journal of Special Education, 33, 81–92. doi:10.1177/001440290907500301

Bottge

B. A.

Heinrichs

Chan

S. Y.

Serlin

R. C.

(2001). Anchoring adolescents’ understanding of math concepts in rich problem-solving environments. Remedial and Special Education, 22, 299–314.

Bottge

B. A.

Grant

T. S.

Stephens

A. C.

Rueda

(2010). Advancing the math skills of middle school students in technology education classrooms. NASSP Bulletin, 94, 81–106.

Bottge

B. A.

Heinrichs

Chan

S. Y.

Mehta

Z. D.

Watson

(2003). Effects of video-based and applied problems on the procedural math skills of average-and low-achieving adolescents. Journal of Special Education Technology, 18, 5–22.

Bottge

B. A.

Rueda

LaRoque

P. T.

Serlin

R. C.

Kwon

(2007). Integrating reform-oriented math instruction in special education settings. Learning Disabilities Research & Practice, 22, 96–109.

Bottge

B. A.

Rueda

Skivington

(2006). Situating math instruction in rich problem-solving contexts: Effects on adolescents with challenging behaviors. Behavioral Disorders, 31, 394–407.

Bottge

B. A.

Watson

E. A.

(2002). Using video-based math problems to connect the skills and understandings of incarcerated adults with disabilities. Journal of Special Education Technology, 17, 25–38.

Bruhn

A. L.

Hirsch

S. E.

Gorsh

Hannan

(2014). Simple strategies for reflecting on and responding to common criticisms of PBIS. Journal of Special Education Leadership, 27, 13–25.

10.

Carnine

(1997). Bridging the research-to-practice gap. Exceptional Children, 63, 513–521.

11.

Cho

S. J.

Cohen

A. S.

Bottge

(2013). Detecting intervention effects using a multilevel latent transition analysis with a mixture IRT model. Psychometrika, 78, 576–600.

12.

Cook

B. G.

Cook

S. C.

(2013). Unraveling evidence-based practices in special education. Journal of Special Education, 47, 71–82.

13.

Coyne

M. D.

Cook

B. G.

Therrien

W. J.

(2016). Recommendations for replication research in special education: A framework for systematic, conceptual replications. Remedial and Special Education, 37, 244–253. doi:10.1177/0741932516648463

14.

Jones

M. L.

(2009). A study of novice special educators’ views of evidence-based practices. Teacher Education and Special Education, 32, 101–120.

15.

Keller-Margulis

M. A.

(2012). Fidelity of implementation framework: A critical need for response to intervention models. Psychology in the Schools, 49, 342–352.

16.

Kretlow

A. G.

Blatz

S. L.

(2011). The ABCs of evidence-based practice for teachers. Teaching Exceptional Children, 43(5), 8–19.

17.

Makel

M. C.

Plucker

J. A.

Freeman

Lombardi

Simonsen

Coyne

(2016). Replication of special education research necessary but far too rare. Remedial and Special Education, 37, 205–212, doi:10.1177/0741932516646083

18.

Mulcahy

C. A.

Krezmien

M. P.

(2009). Effects of a contextualized instructional package on the mathematics performance of secondary students with EBD. Behavioral Disorders, 34, 136–150.

19.

Schmidt

(2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90–100.

20.

Therrien

W. J.

Mathews

H. M.

Hirsch

S. E.

Solis

(2016). Progeny review: An alternative approach for examining the replication of intervention studies in special education. Remedial and Special Education, 37, 235–243. doi:10.1177/0741932516646081

21.

Travers

J. C.

(2016). Evaluating claims to avoid pseudoscientific and unproven practices in special education. Intervention in School and Clinic, 52, 195–203. doi:10.1177/1053451216659466

22.

Travers

J. C.

Cook

B. G.

Therrien

W. J.

Coyne

M. D.

(2016). Replication and special education. Remedial and Special Education, 37, 195–204. doi:10.1177/0741932516648462

23.

Zydney

J. M.

Stegeman

Bristol

Hasselbring

T. S.

(2010). Improving a multimedia learning environment to enhance students’ learning, transfer, attitudes, and engagement. International Journal of Learning Technology, 5, 147–165.