Abstract
Increasingly, mixed methods research (MMR) is used in special education, although the extent to which MMR is used in intervention research in the field of emotional/behavioral disorders (EBD) remains unknown. The purpose of this article is to examine the use of MMR in EBD intervention research to describe its application within the field. We conducted a targeted review of two prominent journals to assess the use of MMR over the last 20 years in the field. Following the identification of MMR intervention studies, we applied several frameworks to describe the application of MMR across identified studies. Results of this targeted review indicate that very few EBD intervention studies used MMR. Most studies used singular quantitative approaches (e.g., group experimental or single case research design) and only a small portion used qualitative methods. We did find a number of studies using multimethods by combining two or more quantitative approaches. Our assertion is that researchers consider increasing their use of MMR to advance the science in the field of EBD and to gain a better understanding of the “why” and “how” interventions are or are not effective.
Over the past several decades, a great deal of emphasis has been placed on advancing research quality and ensuring that researchers in the field of special education utilize rigorous methods, but those efforts have largely focused on individual research methods (Corr et al., 2020). Notably, special issues of Exceptional Children have expounded quality indicators and standards for group experimental and quasi-experimental (Gersten et al., 2005), single-case experimental (Horner et al., 2005), correlational (Thompson et al., 2005), and qualitative (Brantlinger et al., 2005) research within special education (Odom et al., 2005). Although special education researchers have provided these standards and guidance concerning individual research methods, it has become apparent that singular methodological approaches alone may limit the special education knowledge base and that there is an increased need for mixed methods research (MMR; Corr et al., 2020; Houchins et al., 2022; Leko et al., 2022; Newman & Houchins, 2018). The benefit of MMR is that it addresses complex issues to meet the needs of those with disabilities (Klingner & Boardman, 2011), including students with emotional/behavioral disorders (EBD; Anderson, 2018; Marsh et al., 2022).
Nearly 20 years ago, social science researchers began to consider the relevance of MMR (Chatterji, 2004; Johnson & Christensen, 2004; Johnson & Onwuegbuzie, 2004; Onwuegbuzie & Corrigan, 2018; Raudenbush, 2005; Tashakkori & Teddlie, 2003). Although numerous definitions exist, an MMR design can be defined as a study in which a researcher(s) collect[s] and analyze[s] both qualitative and quantitative data rigorously in response to research questions and hypotheses, integrates (or mixes or combines) the two forms of data and their results, organizes these procedures into specific research designs that provide the logic and procedures for conducting the study, and frames these procedures within theory and philosophy. (Creswell & Clark, 2017, p. 5)
In 2003, Tashakkori and Teddlie published a comprehensive handbook of MMR in social and behavioral research. More recently, Hitchcock and Onwuegbuzie (2022) published an updated handbook emphasizing the role of integration when using MMR. Since 2003, the use of MMR in the social sciences has increased. Yet, the field of special education is just beginning to acknowledge the benefits of rigorous MMR research. In a recent systematic review of articles published between 2007 and 2019 in 15 top-ranked special education journals, Corr et al. (2020) found that less than 1% of original research studies clearly reported using MMR. While Corr et al.’s literature review represents one of the most complete and thorough MMR literature reviews in special education, the authors focused on a broad array of special education journals rather than journals in a particular area of study (e.g., EBD; learning disabilities).
The limited use of MMR in special education, including EBD, may be attributed to several factors. For instance, rigorous MMR designs are often more labor-intensive and require a great deal of logistical planning (Collins et al., 2006), requiring more resources and effort to plan and implement (Tashakkori & Teddlie, 2003). Furthermore, planning an MMR study requires that researchers have specific expertise in both qualitative and quantitative design (Tashakkori & Teddlie, 2003). Although training researchers in special education across methodologies is becoming more common, special educators, particularly in the area of EBD, tend to specialize in quantitative research methods and may lack the training in MMR or qualitative methods. In addition, MMR may create a large amount of data that poses challenges. Researchers may have difficulty with combining or mixing quantitative and qualitative data in a meaningful way. In addition, funding agencies may have not prioritized MMR, which might discourage researchers from proposing MMR studies. Beyond logistical challenges, differences in theoretical perspectives among researchers may result in disagreement among researchers in the most appropriate methodology to use in order to best answer research questions (Collins et al., 2006). For example, historically researchers in the field of EBD have argued for approaching research from a positivist or post-positivist paradigm and have often rejected using a post-modernism frame that often uses qualitative approaches toward research (e.g., see Kauffman & Sasso, 2006; Sasso, 2001). Yet, answering many of the perplexing and seemingly intractable issues in the field of EBD may benefit from a more pragmatic paradigm that builds upon the strengths of one paradigm while also utilizing the benefit of other theoretical approaches.
Benefits of MMR in the Field of EBD
Despite the apparent challenges in conducting MMR, researchers have discussed several benefits of its rigorous design in advancing not only the field of special education, but its potential to improve outcomes for students with EBD. Chiefly, the use of MMR minimizes a “one size fits all” approach to special education research (Klingner & Boardman, 2011) and allows researchers to explore the intersection of multiple constructs or phenomena within a single study (Snodgrass et al., 2018). In addition, the use of a singular method and an overreliance on experimental research have faced criticism from educational researchers (Collins et al., 2006; Klingner & Boardman, 2011), who noted that the strength of experimental research is in its ability to identify cause and effect relationships, but often fails to explain “how” or “why” an intervention may be effective or not (Collins et al., 2006). Similarly, overdependence on using qualitative research as a singular methodology has limitations as it is a nonexperimental design and does not address questions of whether something does or does not work (i.e., causal inferences), but can help researchers understand nuances in the findings and may further inform further implementation of the intervention. Thus, Collins et al. (2006) suggested that special education researchers consider using MMR to answer complex questions where a more nuanced understanding of why and how something does or does not work. Similarly, Houchins et al. (this special issue) presented an aspirational framework that describes 17 areas where MMR can enhance EBD intervention research quality. The framework includes (a) research question development; (b) intervention development; (c) instrumentation development; (d) selection; (e) sampling; (f) multi-lens causality; (g) social validity; (h) acceptability; (i) stakeholder perspectives; (j) treatment integrity; (k) contamination; (l) carry-over effects; (m) outcome variability; (n) attrition; (o) replication; (p) data analysis and interpretation; and (q) dissemination.
Anderson (2018) and Maggin et al. (2016) also described the relevance of MMR for researchers in EBD, noting that students with EBD are not a homogeneous group, but, rather, have complex and diverse social, emotional, academic, and mental health needs. To meet those needs, interventions must account for the risk and protective factors among children and their families. Although randomized controlled trials (RCTs) are widely recognized as the gold standard in intervention research (Odom et al., 2005), there are several factors that may create significant challenges and impact researchers’ ability to make causal inferences (e.g., the heterogeneous nature of students identified with an EBD, issues around small sample sizes and attrition, ethical concerns with random assignment). Anderson (2018) suggested that broader analytic approaches are necessary to further develop our understanding of interventions that are effective for students with EBD. Thus, there may be tremendous benefit in increased use of MMR studies that allow for multiple ways of gathering and analyzing evidence of cause and effect.
Differentiation Between Mixed and Multimethods
Whereas the definition of MMR specifies that researcher(s) integrate, mix, or combine quantitative and qualitative data and results (Creswell & Clark, 2017), many studies may utilize more than one singular research method to answer distinct research questions (e.g., Lane et al., 2001; Mason et al., 2013; McDaniel et al., 2013). A multiple method or multimethod design refers to the practice of employing two or more different research methods within the same study rather than confining the research to the use of a single method (Brewer & Hunter, 1989, 2006; Onwuegbuzie & Hitchcock, 2019). Multimethod studies can fall into one of four categories: (a) multiple quantitative and qualitative design, (b) multiple quantitative designs, (c) multiple qualitative designs, or (d) a combination of multiple quantitative and qualitative designs (Nastasi et al., 2007). For example, a researcher may utilize multimethods to answer research questions related to both the efficacy of an intervention and its social validity. As an illustration, Lane et al. (2001) conducted a multiple baseline single-case design across intervention groups to investigate the effectiveness of the phonological awareness training for reading. Following the implementation of the intervention program, the researchers reported social validity findings based on the composite scores derived from Likert-type scale items. In this example, researchers used two distinct research methods to best answer the research questions, but data and results were not integrated or mixed in this case. Conversely, MMR would allow researchers to design a study a priori employing and examining quantitative and qualitative data simultaneously and integrate the relationships between the findings, leading to a more in-depth understanding of the intervention.
Rationale for Current Targeted Review in EBD Intervention Research
Although the landscape of MMR in special education has been recently examined (e.g., Corr et al., 2020) and is expanding (see Leko et al., 2022), the prevalence of MMR studies in the field of EBD has yet to be explored. For example, studies from two of the top-tier journals focused on EBD intervention research (e.g., Behavioral Disorders, Journal of Emotional and Behavioral Disorders) were not included in the Corr et al. review. Given the benefits of using a rigorous research approach to improve outcomes for children with EBD (Anderson, 2018), and the historical emphasis on using quantitative methods, it is important to examine the extent to which EBD researchers have utilized more than a single method within intervention research and further consider how often studies using multiple methods are integrated or combined in a meaningful way. Thus, the purpose of this article is to identify and examine the use of MMR within intervention research in the field of EBD. An additional purpose is to identify intervention studies that employed multimethods to answer research questions, but did not purposefully integrate those findings. Finally, the framework proposed by Houchins et al. (this special issue) is applied to the identified studies to further describe the application of MMR.
Targeted EBD Intervention Literature Review
To explore the prevalence of mixed and multimethod intervention studies in the field of EBD, we conducted a targeted review (i.e., not a systematic review of the literature) of the two leading journals in the field of EBD (i.e., Behavioral Disorders and Journal of Emotional and Behavioral Disorders). All articles from the year 2000 through 2020 in each journal were screened for inclusion. A beginning date of 2000 was selected because MMR began to increase after the dissemination of Tashakkori and Teddlie’s seminal work in MMR in social research in 1998, followed by a handbook of MMR in social and behavioral research in 2003.
Study Screening and Identification Steps
Two doctoral students in special education with a concentration in early childhood studies completed the primary coding of all articles. Empirical intervention studies that utilized mixed and multimethod designs were identified though the following five-step screening process.
Step 1—Identification of Articles Published 2000–2020
All articles from 2000 through 2020 in Behavioral Disorders (Volumes 25 to 46) and Journal of Emotional and Behavioral Disorders (Volumes 8 to 28) were downloaded from journal websites or electronic databases.
Step 2—Identification of Empirical Research Studies
All articles were screened to determine whether they were empirical or nonempirical. Types of nonempirical articles excluded were editorial notes (e.g., E. M. Farmer & Rauktis, 2017, media reviews (e.g., Barr & Parrett, 2002), commentaries (e.g., Wagner et al., 2009), theoretical articles (e.g., T. W. Farmer et al., 2018), systematic reviews without a meta-analysis component (e.g., Bruhn et al., 2022), legal analyses (e.g., Etscheidt, 2002), and memoriam (e.g., Kutash, 2009).
Step 3—Identification of Empirical Intervention Studies
All empirical studies identified at the second step were screened to determine whether the study examined an intervention or not. Intervention studies were defined as those that examined the effects of exposure to an intervention or treatment on some or all of the study participants. Examples of intervention studies identified at this step include experimental group design (e.g., Iovannone et al., 2009), quasi-experimental group design (e.g., Hogue et al., 2016), one-group pretest–posttest design (e.g., Borgmeier et al., 2015), single-case design (e.g., Escarpio & Barbetta, 2016), and qualitative methods (e.g., Lewis et al., 2020).
Empirical studies that did not examine an intervention were excluded from further review at this step. These excluded empirical studies fell into one of the following categories: measurement study (e.g., Heyne et al., 2017), correlational study (e.g., Hurley et al., 2017), descriptive study (e.g., Flannery et al., 2013), meta-analysis (e.g., Theule et al., 2013), or methodological study (e.g., Rakap et al., 2014).
Step 4—Identification of Multimethod Studies
To identify multimethod research reports, all intervention studies retained in Step 3 were screened according to two criteria.
Step 4.1
Studies were screened to identify the primary research design (i.e., group design, single-case design, or qualitative design). Group design was defined as a research study that aggregated data collected from multiple participants and used inferential statistics to draw conclusions about group differences (e.g., Borgmeier et al., 2015; Hogue et al., 2016; Iovannone et al., 2009).
Single-case design was defined as a research study in which the dependent variable was repeatedly measured during a baseline condition, followed by an intervention condition with changes in level, trend, and variability that are used to assess the presence of experimental control (Gast & Ledford, 2014). Examples of single-case designs included withdrawal and reversal design, multiple baseline design, multiple probe design, multitreatment design, alternating treatment design, multi-element design, adapted alternating treatments design, parallel treatments design, changing criterion design, concurrent operant design, repeated acquisition design, brief experimental design, and combination designs (e.g., Hustus et al., 2020).
Qualitative design was defined as a nonexperimental study that explored attitudes, opinions, and beliefs of research participants on an educational phenomenon (Brantlinger et al., 2005). For the purposes of this review, studies using a qualitative design examined the outcomes of an intervention and primarily included case studies and semi-structured interviews to gather informants’ perspectives on the potential outcomes of the intervention (e.g., Hyter, 2002; Lewis et al., 2020).
Step 4.2
All intervention studies were screened a second time to determine whether authors utilized a multimethod research design. A multimethod design was defined as the practice of employing two or more different methods within the same study rather than confining the research to the use of a single method (Brewer & Hunter, 1989, 2006; Onwuegbuzie & Hitchcock, 2019). In addition, if a multimethod design was employed, we identified the type (i.e., multiple quantitative design [e.g., QUANT + quant], multiple qualitative design [e.g., QUAL + qual], and multiple quantitative and qualitative design [e.g., QUANT + QUAL]).
We defined multiple quantitative designs (e.g., Lane et al., 2001; McDaniel et al., 2013) as those studies in which the research team collected quantitative data (and only quantitative data), presented more than one type of quantitative design to answer different research questions, and analyzed according to design (adapted from Hunter & Brewer, 2015; Onwuegbuzie & Hitchcock, 2019). For example, Lane et al. (2001) conducted a multiple baseline single-case design across intervention groups to investigate the effectiveness of the phonological awareness training for reading (QUANT). Following the implementation of the intervention, the researchers reported social validity findings based on the composite scores derived from a Likert-type scale (quant).
Multiple qualitative design was defined as a study in which the research team collected qualitative data (e.g., focus group, interview with open-ended questions, written response to open-ended survey questions), presented more than one type of qualitative design to answer different research questions (e.g., content analysis, narrative analysis, grounded theory), and analyzed according to design (adapted from Hunter & Brewer, 2015; Onwuegbuzie & Hitchcock, 2019).
Multiple quantitative and qualitative design was defined as a study that collected quantitative data (with one or more quantitative data collection methods) and qualitative data (with one or more qualitative data collection methods) and presented data analysis results for data collected through the quantitative and qualitative methods (adapted from Hunter & Brewer, 2015; Onwuegbuzie & Hitchcock, 2019). To illustrate, Mason et al. (2013) evaluated the effectiveness of the Self-Regulated Strategy Development (SRSD) instruction for quick writes with three high school students with EBD using a multiple baseline across participants design. In addition to examining the effects of the SRSD, the number of persuasive parts written, and the number of words written using the quantitative data, the researchers also investigated intervention acceptability by having participating students answered six open-ended questions regarding their impressions of the SRSD instruction. Although Mason and colleagues employed both quantitative and qualitative methods, they did not integrate (or mix) their data, and thus, we did not code their study as an MMR study.
Step 5—Identification of Mixed Methods Studies
Finally, studies previously identified as using a multiple quantitative and qualitative design were screened to determine whether the study used an MMR design. As indicated earlier, MMR designs were defined as a study in which a researcher(s) collect[s] and analyzes both qualitative and quantitative data rigorously in response to research questions and hypotheses, integrates (or mixes or combines) the two forms of data and their results, organizes these procedures into specific research designs that provide the logic and procedures for conducting the study, and frames these procedures within theory and philosophy. (Creswell & Clark, 2017, p. 5)
Therefore, only study that met this definition was coded as an MMR study.
Intercoder agreement was calculated for 20% of the articles identified at Steps 2 through 5 of the coding process. For each coding step, the percentage of agreement was calculated by dividing the number of agreements between the coders by the sum of the number of agreements and disagreements, multiplied by 100%. The intercoder agreement for Step 2 (n = 195), identification of empirical research studies, was 97.44%. At Step 3 (n = 136), identification of empirical intervention studies, agreement was 95.59%. At Step 4.1 (n = 49), identification of primary research design, agreement was 100%. At Step 4.2 (n = 44), agreement was 89.90% for the use of multiple methods and 89.90% for type of multiple methods employed. At Step 5 (n = 6), use of MMR design, agreement was 100%. The research team, comprised of the two doctoral students and three doctoral-level researchers with expertise in EBD, discussed disagreements and final coding decisions were determined by consensus.
Extent and Applicability of Mixed Methods Intervention Research in EBD
Of the 975 articles reviewed in Behavioral Disorders and Journal of Emotional and Behavioral Disorders, a total of 678 empirical research studies were identified. The remaining nonempirical studies (n = 297) included commentaries, theoretical articles, descriptive studies, correlational studies, measurement studies, meta-analyses, or other related articles. Of the empirical research studies identified, 239 represented studies that examined the effects of an intervention. These studies were further categorized into types of primary research designs, including group experimental design, single-case design, or qualitative design. Of the 239 intervention studies, 109 (45.61%) were classified as group experimental design, 126 (52.72%) were single-case design, and four (1.67%) employed a qualitative design (e.g., case studies). Following the determination of primary research design, each intervention study was further coded to determine whether multimethods were used within the study. Of the 239 intervention studies, 152 (63.60%) used more than a single research method to address the research questions. Quantitative and quantitative methods were used in 123 (80.92%) of the 152 multimethods studies, no studies used qualitative plus qualitative, and quantitative and qualitative methods were used in 29 (19.08%). Only two studies (0.84%) out of the total intervention studies were identified as using an MMR design. See Table 1.
Summary of Results at Each Step.
Note. BD = Behavioral Disorders; JEBD = Journal of Emotional and Behavioral Disorders.
Description of Mixed Methods in EBD Intervention Research
To further describe intervention research using MMR in the field of EBD, a three-step process was used. First, publication guidelines from multiple MMR resources as described by Houchins et al. (2022) were applied to determine whether the studies were aligned with recommended journal dissemination guidelines. Next, an examination of the legitimation types (see Houchins et al., this special issue) occurred. Finally, the MMR EBD Intervention Inquiry Framework outlined in Houchins et al. (this special issue) was applied to the two studies that were located.
Bottage et al
Bottage et al. (2006) employed an MMR design to examine the effects of enhanced anchored instruction (EAI) on the math achievement of teenage students at risk of EBD. Following the implementation of the intervention, the results were mixed. There was a positive effect for curriculum-aligned problem-solving, but not for fraction computation or on the standardized measures used in the study. Following post-testing, the researchers conducted classroom observations and interviews to further explore their findings.
Application of journal dissemination guidelines
The authors refer to the use of MMR in the abstract of the article and describe their design in the method section of the paper. Their research process was transparent, and a description of both the quantitative and qualitative approaches was described in detail. In this study, the researchers used a concurrent MMR approach to address two different research questions. First, using a quantitative approach (i.e., one-group nonequivalent dependent variables design, they examined the effects of EAI on math achievement of students at risk of EBD (i.e., Did it work?). Next, they conducted direct classroom observations and video-recorded to document what occurred during the intervention. Teachers also recorded their impressions into a logbook during the intervention phase of the study. At the end of the study, interviews with the principal, classroom observers, and the students were conducted to identify and describe factors that may have influenced the effectiveness of the intervention on different outcomes. These interviews were transcribed, and the researchers identified potential factors to further understand the results. They also examined the information in the teacher logbooks and video recordings to further explore information shared during the interviews. The results of the quantitative and qualitative analyses were reported separately, and findings did not appear to be fully integrated.
Legitimation types
In this study, the researchers used MMR to further explain the findings (i.e., commensurability approximation). They also included the views of the stakeholders to further the researchers’ understanding of the quantitative data findings (i.e., inside-outside). Finally, they used weakness minimization by including qualitative methods to further understand the limitations of the quantitative methods.
Houchins’ et al. framework
The framework outlined by Houchins et al.’s (this special issue) was applied to this study. These researchers used an MMR design to examine the social validity of the intervention through their interviews with stakeholders, including the students themselves. In addition, they used MMR to understand intervention outcome variability. The quantitative data indicated positive effects on problem-solving tests, but not on standardized scores or fractions. The qualitative data results indicated that students were motivated to engage in problem-solving activities even though they found them difficult. They also reported that they disliked basic math skills such as fractions. In addition, the context in which instruction occurred and the hands-on projects contributed to the positive findings. Specifically, they were examining contextual factors within the classroom that may have impacted the effectiveness. They also examined critical intervention elements (i.e., the key components likely to produce the effects) through their interviews. Given the varied outcomes of the intervention, one of the primary reasons for the MMR was research finding interpretation. The authors stated that fidelity of implementation (e.g., treatment integrity) was assessed using qualitative data sources (teacher logbook, camera captures of learning activities, field notes, video of instructional procedures, classroom observations) structured around lesson objectives and activities. While not explicitly stated in the fidelity of implementation section of the article, duration of instruction was documented through lesson plans corresponding to lesson objectives and activities. Similarly, although not explicitly stated in this article, interviewing the students might have led to using participant perspectives to enhance intervention development and accessing stakeholder buy-in or acceptability.
Woodbridge et al
A second example of a study that used an MMR design to further our understanding of an intervention for students with EBD was identified. Woodbridge et al. (2014) conducted a national effectiveness study of the First Step to Success from 2006 to 2010. The researchers examined the effectiveness of the program on students’ behavior and academic outcomes and followed up on teachers’ implementation of the program after the study was conducted. The researchers discovered that only one of the research sites continued to implement the program in most of the schools 2 years after the effectiveness study was conducted. Thus, qualitative data were collected to investigate how program characteristics and participants’ perceptions of social validity and feasibility influenced the sustainability of the program in the research site. This study helped to share insights into why the intervention program (i.e., First Step to Success) sustained and the factors that might influence sustainability in other sites.
Application of journal dissemination guidelines
Neither the title, abstract, or introduction indicated that an MMR study was conducted or discussed the use of MMR. Both the quantitative approach and qualitative approach were described in detail in the method section of the article and were quite rigorous. First, using quantitative methods, the researchers examined the effectiveness of the intervention on outcomes for children as well as the effectiveness of study implementation measures. Following the quantitative approach, the researchers used qualitative methods (i.e., case study focus group and interviews) to identify factors influencing the sustainability of the intervention. Sampling schemes were discussed and appear appropriate for each approach. The methods were appropriate for the purposes of the study and the added value of various data sources was discussed, but a description of intentional integration is missing from the article. Findings reported quantitative and qualitative data separately; however, the discussion expanded on the importance of the qualitative approach in furthering their understanding of the sustainability of the intervention.
Legitimation types
This article used several legitimation types including commensurability approximation to further their understanding of the sustainability of the intervention. The methods included strategies to ensure rigor of both the quantitative and qualitative approaches for multiple validities was described. Weakness minimization occurred using a sequential design to identify the factors that influence sustainment.
Houchin’s et al. framework
While the Woodbridge and colleagues (2014) did not provide separate quantitative, qualitative, and MMR research questions, they did state that a purpose of the study was to investigate the acceptability and social validity of the First Steps to Success (Walker et al., 1997) intervention. Selection procedures and sampling schemes were described. They assessed the social validity of the intervention by interviewing stakeholders about the reasons the intervention was accepted and sustained. Adherence, adaptation, duration, and quality as components of fidelity of implementation (e.g., treatment integrity) were documented quantitatively while qualitative data were collected from case studies and interviews. In addition, through interviews, they identified critical intervention elements, which also supported their interpretation of research findings. In particular, the authors analyzed various program characteristics in relation to stakeholders’ perspective of the social validity and feasibility that influences sustainability of the intervention. One of their goals was to identify factors that may have influenced sustainability and adoption of the intervention of one site in a larger RCT. This site was of particular interest, because when they examined sustainment and adoption across sites, they found that many of the original sites did not sustain the intervention after the research study was completed. This school district sustained and adopted the intervention. Therefore, part of the purpose of this study was to understand these replication differences.
Discussion
The purpose of this article was to examine the use and application of MMR designs in intervention research conducted in the field of EBD. Two prominent EBD journals were reviewed: Behavioral Disorders and Journal of Emotional and Behavioral Disorders. Like Corr et al. (2020), we found few studies that used an MMR approach. Not surprisingly, several intervention studies were published in both journals, but the majority of these employed a sole quantitative approach (e.g., single-case design or group experimental design). Only a few studies utilized a qualitative approach. One promising finding is that some intervention studies incorporated a multimethod approach. Of the studies employing a multimethod approach, the majority of these combined two quantitative methods. For example, many studies using a multimethod approach employed a single-case design to examine effects of an intervention and a Likert-type scale to examine social validity (e.g., see Zimmerman et al., 2020). Other multimethod studies used two different group designs to address two different research questions (e.g., see McDaniel et al., 2013). Using multimethods to further explore and understand the effects of an intervention can provide additional information in comparison with a singular method and illustrates advances in intervention research conducted in the field of EBD. However, using an MMR approach and integrating findings from both methods could even further advance our understanding of “how” or “why” interventions are effective (Collins et al., 2006; Snodgrass et al., 2018) and advance the field.
The targeted review conducted for this article has several limitations that should be noted. First, a systematic literature review was not conducted and only two journals in the field of EBD were targeted. Conducting a more comprehensive and systematic literature, which includes additional journals publishing intervention research for individuals with EBD, may have led to the identification of more MMR studies and changed the findings. Nevertheless, the two journals selected are prominent in the field of EBD and were not represented in previous MMR literature reviews. Furthermore, many special education researchers who examine interventions for students with EBD publish in these journals. Another limitation is that the review was focused solely on intervention studies using an MMR design. Several additional studies were located in the journals that used MMR to answer research questions but did not implement an intervention. For example, Poduska and Kurki (2014) used an MMR approach to examine the implementation of the Good Behavior Game (GBG; Barrish et al., 1969). Both quantitative (i.e., teacher implementation and student outcomes) and qualitative data (i.e., teacher interviews) were collected to provide information on how the professional development component of the intervention performs when implemented by coaches and teachers within school settings. Although this study was conducted within a larger RCT investigating the effects of the GBG, the focus of this study was on how the training and coaching model works rather than the efficacy of the intervention. An MMR approach was used, but how the findings were integrated was less clear. This was not an intervention study, but the results from this MMR study also help to inform the researchers on factors that influenced teacher implementation and can guide adaptations they might consider to the GBG to increase usability and sustainability.
Future Research Directions
As suggested by Anderson (2018), given the complexity and diverse needs of students with EBD, understanding whether an intervention is effective for individuals as well as the factors that might influence the effectiveness can further our understanding of the intervention. As illustrated in Woodbridge et al. (2014), interventions may or may not be sustained after the research study is over; yet identifying the factors that influence sustainability may help to make interventions more acceptable or socially valid to the end-users. In addition, MMR approaches may help to inform whether the intervention is implemented with fidelity and why (or why not). Again, identifying factors that influence stakeholders from implementing an intervention with fidelity or not may help researchers understand how to modify or adapt intervention to ensure feasibility of implementation. Much of the EBD intervention research is conducted in classroom settings and the intervention is delivered by authentic providers such as teachers. Therefore, it is important that the interventions are acceptable and feasible to implement in these settings.
Clearly, MMR approaches have several advantages and more MMR is needed in the field of EBD (Anderson, 2018; Hitchcock & Onwuegbuzie, 2022). Using MMR while implementing an intervention can help inform researchers if teachers or other change agents can deliver the intervention with fidelity to students with EBD and identify factors that might influence their initial delivery, adoption, and sustainability. As illustrated by Bottage and colleagues (2006), MMR can also help identify other factors that might influence the effectiveness of the intervention (e.g., classroom context, individual student characteristics). Currently, the field of EBD has identified several evidence-based programs and practices that we know “work” for students. More research is needed to address research questions such as what works, with whom, and under what conditions. Conducting intervention studies conceptualized using a MMR approach and integrating the methods can further our understanding of these effective interventions.
Another area of intervention research that could be enhanced using MMR is examining the acceptability or social validity of the intervention. Often researchers use Likert-type scales and asked teachers or other stakeholders to rate the acceptability of the intervention. Although this method produces quantitative data indicating the stakeholder’s perspective, using a Likert-type scale fails to provide more detailed information on why or why not the intervention is acceptable. Mixing methods such as single-case design plus stakeholder interviews or focus groups may provide a more in-depth understanding of why the intervention is acceptable or not and what might increase the acceptability.
Summary
In summary, a targeted review of the intervention research conducted over the past 20 years for students with EBD in two journals was examined. Most of the studies in these two journals used a singular quantitative method, either single case or group designs. Only several MMR intervention studies were identified. As the research base in the field of EBD continues to grow, identification of factors that influence the effectiveness of interventions is important to identify. The MMR can further the knowledge-base and inform practice in the field.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.