Abstract
Special education policy and scholarship have increasingly emphasized the use of research that encompasses interdependent methodological approaches designed to address specific questions. Consequently, the knowledge available to practitioners, policymakers, and other consumers is a function of the research designs featured in the literature. This study describes the prevalence of methodological approaches appearing in special education journals (n = 33). An assessment of a sample of articles (n = 12,669) published from 1999 to 2019 found that much of the published research (n = 9,543) emphasizes nonexperimental, quantitative inquiry (57.86%). Experimental research is comparatively less common (25.51%) and encompasses both single-case (14.22%) and group experimental designs (11.29%). Qualitative studies (9.38%) appear far less frequently. Patterns of publication vary based on journal emphasis. A description of results is followed by a discussion of implications for the consumption and generation of evidence in special education.
The historical emphasis on research within special education has contributed to the development of effective instruction for students with disabilities and the dissemination of practices to other fields (Odom et al., 2005; Vaughn & Swanson, 2015). Increasing recognition of the importance of research has coincided with the “evidence-based practice movement,” which is predicated on the use of experimentally verified instructional methods (Cook & Odom, 2013). While the emphasis on studies that identify evidence-based practices potentially conflates science—which refers to a wide range of practices predicated on objective observation, making predictions, and analyzing data—with intervention research, scholars in special education have repeatedly emphasized the need for a full spectrum of methods to address questions relevant to people with disabilities (Cook & Cook, 2016; Odom et al., 2005). Such work may involve studies with only an indirect relation to intervention (e.g., attitudes of service providers or recipients).
Research is valued based on the extent to which it allows the researcher to address questions of interest; no methodology is inherently superior to another (Vaughn & Dammann, 2001). Broad research designs (e.g., experiments) are linked to research questions that serve a specific purpose (e.g., survey; Cook & Cook, 2016). For example, studies concerning teachers’ attitudes toward inclusion (Saloviita, 2020) describe observable phenomena in quantitative terms (Odom et al., 2005; Plomp, 2013). Experimental studies, which are most closely associated with evidence-based practices, involve the management of conditions to determine whether the use of an intervention has an impact on quantitative measures of student performance (e.g., reading fluency; King, Rodgers, & Lemons, 2022). However, research methods are amenable to a variety of important questions in special education and need not relate to the identification of evidence-based practices to contribute to the field.
Methods used by researchers provide insight into the questions under consideration by the field, publication trends, and areas with the potential for additional inquiry (Mastropieri et al., 2009). Whether special education research addresses the range of questions that may be posed by consumers is partly a function of the prevalence of specific methodologies in professional journals (Demchak et al., 2019). Consequently, information regarding the patterns of publication in special education journals is of interest to editors, policymakers, and practitioners—all of whom are encouraged to seek answers from research. This article briefly describes the primary types of designs featured in special education. We then summarize previous research assessing peer-reviewed special education publications. Finally, we examine the prevalence of different approaches to research in a large sample of special education journals.
A Brief Overview of Research Designs in Special Education
The following section provides an overview of research designs commonly featured in special education. Research categorization invariably oversimplifies the wide range of designs encompassing individual studies, and no universally accepted method for design classification exists (Cook & Cook, 2016). We focus on broad categories of research addressed in previous scholarship: descriptive-correlational, experimental, and qualitative (e.g., Gersten et al., 2005; Horner et al., 2005; Thompson et al., 2005; Trainor & Graue, 2014). For experimental research, we provide additional detail regarding specific approaches employed in special education: group and single-case designs.
Descriptive-correlational research
Descriptive-correlational research refers to a variety of studies (e.g., surveys, assessment administration) that differ in terms of specific analyses but share an emphasis on observation. The purpose of descriptive research is to quantitatively describe features of variables as they exist, without any manipulation on the part of the researcher (Cook & Cook, 2016). In a purely descriptive study, the researchers do not attempt to formally establish relationships between variables. For example, Villani et al. (2012) collected the frequency and duration of restraint and seclusion observed in a middle and high school over 6 years. The purpose of the study was to determine how restraints were used in practice and whether there were any lessons to be learned—and potentially generalized—from situations in which restraints were not used. Such information has the potential to guide practice. However, additional analysis is needed to precisely identify variables that predict changes in the variables of interest.
Correlational research represents an extension of descriptive research in which researchers use statistical analysis (e.g., analysis of variance [ANOVA], regression) to determine the relationship between variables collected through observation (Thompson et al., 2005). This analysis can provide evidence of relationships among variables, which can provide useful information to practitioners. For example, Sheaffer et al. (2021) observed a correlation between student gender and teachers’ rating of problem behavior; specifically, females were rated as exhibiting more problematic behaviors than males. Statistical analyses did not reveal a similar relationship between direct observations of student behavior and gender. As the authors suggest, such findings could influence future research and inspire trainings aimed at addressing potential sources of teacher bias.
Researchers may choose to conduct a nonexperimental design because the researcher cannot manipulate a variable (e.g., gender) or because it would be unethical to manipulate a variable (e.g., access to breakfast; Cook & Cook, 2016). Thus, correlational research addresses questions that may not be amenable to experimental research. Researchers may also conduct either descriptive or correlational research as a first step in determining which variables to study later in an experiment. In addition, correlational research may be an appealing option for researchers because it is relatively economical and feasible. Experimentally examining questions regarding the implementation of interventions at a national scale (e.g., positive behavior interventions and supports; Simonsen et al., 2021) would be cost-prohibitive.
Because variables are not actively controlled by the researcher (e.g., participants exposed to a condition as part of a study), however, results of correlational studies cannot support a causal relationship between variables. Returning to Scheaffer et al. (2021), the study revealed an association between gender and teacher ratings, but due to the lack of control over the full range of variables (e.g., other potential sources of bias), results do not suggest differences in gender cause differences in teacher ratings. Consumers must therefore be cautious in interpreting the results of correlational studies (Thompson et al., 2005).
Experimental research
Experimental research involves arranging conditions in such a way as to permit the determination of causality between variables (Cook & Cook, 2016). As with descriptive-correlational studies, many experiments involve some form of statistical analysis. Thompson et al. (2005) suggest the arrangement of conditions by the researcher distinguishes experiments from other forms of evidence, rather than statistical analysis, and permits the assessment of causality. Many putative experiments actually lack sufficient rigor to permit causal inference, however. Thus, design quality—though beyond the scope of this review (see Gersten et al., 2005; Horner et al., 2005; What Works Clearinghouse [WWC], 2020)—represents an important consideration in recognizing and interpreting experiments in special education. Notwithstanding these considerations, determining whether instruction and other interventions associated with special education cause positive change for people with disabilities is of obvious and critical importance to the field.
As policymakers, practitioners, and consumers are increasingly interested in effective practices, increasing emphasis has been placed on experimental designs (Cook & Cook, 2013). Nonetheless, experimental designs have notable limitations, such as expense and the inability to provide insight into how an intervention works or whether teachers are likely to continue using the practice outside of the experimental context (Cook & Odom, 2013). Special education researchers use two broad methodologies to evaluate the effectiveness of interventions and instructional practices: group designs and single-case designs (Cook & Cook, 2016).
Group design research
Prevalent in many research fields, group designs typically demonstrate the effectiveness of treatment by examining the differences between participants randomly assigned to groups (Gersten et al., 2005). The researcher then (a) differentially exposes groups to the independent variable (e.g., often some form of instruction), with one group receiving access (i.e., the treatment group) and another receiving some alternative (i.e., the control group); and (b) measures a variable of interest (i.e., the dependent variable) both before and after the introduction of the intervention (Cook & Cook, 2016). In evaluating the efficacy of three separate interventions on fourth graders’ ability to solve word problems with fractions, Fuchs et al. (2016) randomly assigned participants (n = 213) to groups corresponding with each form of instruction and measured their performance before and after 12 weeks of intervention. Subsequent analyses statistically determined both the magnitude (i.e., effect size) and significance of the differences between groups—a process that is partially contingent on the size of the samples featured in the experiment.
Random assignment of participants and the researcher’s control of treatment ensure changes observed in the treatment group, relative to the control group, are caused by treatment rather than another factor (Gersten et al., 2005). When researchers must work with intact groups, such as classrooms, randomization may be maintained by randomly assigning groups to treatment or control conditions. The resulting quasi-experimental design introduces another potential explanation for results—any commonality between members of each of the intact groups—which must be controlled for statistically (Odom & Lane, 2014).
Although historically considered the “gold standard” of experimental design, group designs have several limitations (Spooner & Browder, 2003). The emphasis on groups is incompatible with the prevalence and instructional needs of students with disabilities—relatively small populations who may have unique characteristics that necessitate intensive, personalized instruction. The resources needed to facilitate group design (e.g., personnel) can also be especially prohibitive (Cook & Cook, 2016). In addition, the aggregation of effects across participants in treatment and control conditions potentially obscures the performance of unique individuals whose results may differ from the average participant (Cooper et al., 2020). Subgroup analyses have the potential to address concerns regarding the aggregation of effects; however, such procedures are not always adequate for small samples and are associated with the risk of misleading findings in certain circumstances (Brookes et al., 2004). Consequently, special education researchers often use alternative forms of experimental design.
Single-case design research
A range of within-subject experimental methodologies and single-case designs emphasize repeated assessment over time, systematic implementation of an independent variable, and the replication of intervention effects across conditions, individuals, or groups (e.g., Hurtado-Parrado & López-López, 2015). Single-case designs often appear in disciplines, such as special education, where the difficulty in obtaining large groups would otherwise inhibit the experimental validation of interventions. In single-case designs, the individual generally serves as their own control, and performance across varying conditions provides evidence of an independent variable’s effect (Ledford & Gast, 2018). The focus on the individual allows flexibility and adaptation to address idiosyncratic student needs. When student performance is stable and demonstrates a consistent pattern across similar conditions, single-case designs also allow researchers to determine the effect of an intervention (Cooper et al., 2020).
Single-case designs encompass a wide array of experimental arrangements, including reversal and multiple-baseline designs (Ledford & Gast, 2018). Reversal designs evaluate changes in behavior by repeatedly introducing and withdrawing an intervention (e.g., access to tokens in a token economy; Sleiman et al., 2020). Multiple-baseline designs stagger intervention implementation across behaviors, individuals, or settings (e.g., start date of reading instruction across participants; King, Rodgers, & Lemons, 2022). Unlike group designs, which evaluate effects using statistical analysis and null-hypothesis testing, researchers typically analyze single-case designs through the visual analysis of data displayed on a linear graph (Ledford & Gast, 2018). Patterns in steady states of responding across conditions demonstrate a functional relation between variables. The strongest demonstration of a functional relation occurs when behavior change is replicated repeatedly across adjacent experimental conditions.
Although purportedly a large portion of experimental studies were conducted in special education (Mastropieri et al., 2009), the wider education community has only recently acknowledged the utility of single-case designs (Hurtado-Parrado & López-López, 2015). A major concer in single-case designs is the potential threat to externally validity posed by the use of small samples that may not be representative of the broader population (Ledford & Gast, 2018). Researchers provide evidence of generalizability in single-case designs through repeated replication of effects within and across studies, suggesting a large quantity of single-case research is necessary for the generalization of findings (Sidman, 1960). The routine use of idiosyncratic, proximal measures rather than the standardized measures more typically featured in group designs (e.g., Dessemontet et al., 2019) represents an additional challenge to generalizing the results of single-case designs. Demonstrating an intervention effect across large numbers of individuals in a group design potentially provides a greater indication of what might be probable in practice; in contrast, a single-case design provides an indication of what might be possible for an individual and generally results in a much larger effect size relative to group designs (King, Rodgers, & Lemons, 2022). Researchers have also encountered difficulty comparing or meta-analyzing single-case designs, which prevents the identification of moderators (i.e., variables with the potential to influence intervention effectiveness; King, Wang, et al., 2022).
Qualitative research
Qualitative research provides deep descriptions and explanations of a phenomenon within a certain context from a participant’s perspective (Merriam & Tisdell, 2015). Unlike quantitative studies, qualitative research relies on text derived from questionnaires, interviews, and observations to address research questions (Cook & Cook, 2016; McDuffie & Scruggs, 2008). Researchers using a qualitative approach emphasize collecting and analyzing verbal or visual data. Qualitative research encompasses multiple approaches, including grounded theory, ethnography, and the case study (Merriam & Tisdell, 2015). Grounded theory is used to develop a theory to explain the phenomena under observation. Ethnography explores a social group or community by immersing researchers into the group as a participant. The case study conducts deep investigations by focusing on specific individuals, classrooms, or schools.
Researchers often use qualitative research to explore the attitudes, motivations, experiences, and behaviors of participants (e.g., Scruggs et al., 2007). Due to the emphasis on perception, critics of qualitative research have alleged the design is unscientific or misleading (Vaughn & Dammann, 2001). However, the capacity of research to mislead is independent of whether data are quantitative or qualitative, and qualitative work in special education provides important insight into factors that may be difficult to interpret quantitatively. Although qualitative studies typically do not address questions of causality, they do provide insight into how and why interventions work (Cook & Cook, 2016; McDuffie & Scruggs, 2008; Merriam & Tisdell, 2015).
Previous research suggests qualitative studies rarely appear in special education literature (Trainor & Graue, 2014); nonetheless, qualitative methods play an important role, especially in relation to social validity (Kozleski, 2017). Studies focusing on whether an intervention works provides limited insight into the perceptions of stakeholders. Qualitative research analyzes data from participants’ perspectives (through interview or observation) and potentially deepens the researchers’ understanding of both participants and intervention. As such, qualitative research may be an essential research tool for uncovering the needs of targeted individuals (e.g., teachers, students, or parents; McDuffie & Scruggs, 2008).
Previous Journal Analyses
Evidence revealed through one form of research facilitates work with related, yet different aims. Just as evidence obtained through descriptive-correlational or qualitative designs often informs experiments (e.g., selection of independent or dependent variables), successful experiments often inspire qualitative and descriptive-correlational studies designed to examine outcomes beyond those documented in the original study (e.g., perceived benefits of an intervention or large-scale outcomes; Odom et al., 2005). An optimal special education “ecosystem” would therefore feature a variety of research designs capable of addressing the full range of questions relevant to people with disabilities (Demchak et al., 2019; Plomp, 2013).
Although researchers have examined the types of research methods featured in special education journals, previous analyses have frequently focused on a single journal or journals related to a specific disability category. For example, McFarland et al. (2013) reviewed 841 articles published in three journals concerning students with learning disabilities (LD). The majority of articles were empirical (66.7%), with 20% pertaining to experimental research. Demchak et al. (2019) evaluated the methodological characteristics of articles (n = 527) in five journals concerning people with developmental disabilities published between 2012 and 2014. Results suggest nonexperimental studies (e.g., qualitative, descriptive-correlational) represented 55% of empirical studies (n = 90); qualitative research, however, represented 2% of the total sample. Of the experimental studies identified (n = 40), single-case designs appeared in 80%.
In a broader review, Mastropieri et al. (2009) assessed the prevalence of research designs that appeared in 11 peer-reviewed special education journals from 1988 to 2006. Results suggest that, of the articles evaluated (n = 6,724), 58% were research articles. Of these, the majority (56.2 8%) featured descriptive-correlational designs (e.g., surveys). Experimental research represented 32.70% of studies, with single-case designs constituting just over half of all intervention studies (50.50%). A smaller proportion of studies from the total sample were qualitative studies (11.02%). The rate of published empirical studies and intervention work increased over time; however, the authors did not report design-specific information for journal categories. Zanuttini’s (2020) review of seven journals concerning various disability categories focused on experimental research published between 2014 and 2019 and found that single-case designs represented 45% of experimental studies, with clear discrepancies based on the categorical emphasis of each journal. Specifically, journals for students with emotional/behavioral disorders (EBD) published higher proportions of single-case designs than journals for students with LD or cross-categorical journals.
Purpose
Publication trends in special education provide an indication of what the field has accomplished as well as potential gaps in the knowledge base. The absence of qualitative research, for example, may be indicative of an infrequent consideration of the collateral effects of intervention. Similarly, reliance on single-case designs may have implications for the external validity of experimental research. Earlier reviews have either not been updated within the last decade (Mastropieri et al., 2009) or included a limited number of publications. Thus, this study examines the prevalence of research designs in special education journals. Given that previous studies have linked types of research to specific categories (e.g., single-case designs and EBD; Zanuttini, 2020), we target cross-categorical special education journals—those with the highest impact factors that address a range of issues and disabilities—as well as publications related to developmental disabilities, EBD, and LD. The following were the research questions:
Method
We addressed our research questions using a multi-part strategy. Databases were used to identify the size of the population of interest (i.e., articles in special education journals), which we then divided based on publication year. From this population, we selected a random sample of articles for each year of publication. Articles were then coded based on design characteristics.
Article Identification
Journal selection
English-language journals comprising the special education category of the 2018 Web of Science (WOS) Journal Citation Reports (JCR) Social Science Edition (Clarivate Analytics, 2019) were selected for review. The most accepted subject classification system in the world, the WOS predicates categories on citation patterns (Wang & Waltman, 2016). The WOS categories served as the basis for journal inclusion due to their acceptance and use in similar reviews (e.g., King, Davidson, et al., 2020; King, Kostewicz, et al., 2020). Moreover, reviews in journals outside of special education may not reflect publication practices observed within the field.
From the list of 40 journals identified under the JCR special education category, we selected the journals that corresponded with the field of special education (i.e., cross-categorical journals), developmental disabilities, EBD, or LD. Journals related to other categories were excluded (e.g., gifted and talented, n = 6). In addition, we excluded practitioner-oriented journals. We determined the orientation of each journal by examining both the titles and the aim and scope featured on each journal’s website. Cross-categorical journals included publications that did not identify a specific disability as a focal point or that concerned the general field of special education. Journals focusing on people with developmental disabilities explicitly addressed either developmental disabilities or disabilities categorized as developmental disabilities (e.g., autism). We placed journals dedicated to behavioral issues of people with intellectual disabilities in the developmental disability category because (a) intellectual disabilities are correlated with mental health issues (Grondhuis, 2020) and (b) intellectual disabilities often preclude a diagnosis of EBD (Forness & Knitzer, 1992). Journals focusing on people with EBD identified students with EBD or chronic behavior problems as the primary focus of the journal. For example, the description of the aims and scope of Education and Treatment of Children (ETC) defines the focus of the journal as “behavioral assessments or interventions for children and youth who are at-risk for or experiencing emotional or behavioral problems” (Springer Nature, 2022). We therefore identified ETC as a journal focusing on people with EBD. Journals focusing on people with LD identified LD or related conditions (e.g., learning difficulties) as the area of focus. The list of journals appears in Table 1.
Titles and Identified Records of Included Journals.
Note. Order of journals based on 2018 impact factor. Margin of error for the total sample is ±0.6%. Margin of error (MOE) for category estimates does not exceed ±2.10%; journal MOE does not exceed ±6.41%. Database column refers to records retrieved in the initial search. IN% refers to percentage of articles in a category. Journals published within the full search range unless otherwise noted. DB = database search; IN = included; N = number of studies; S = single-case design; G = group design; D = descriptive study; Q = qualitative study; R = systematic review/meta-analysis; C = commentary; MOE = margin of error.
Superscript denotes alternate titles included in search: aAmerican Journal on Mental Retardation. bMental Retardation. cEducation and Training in Mental Retardation and Developmental and Disabilities and Education and Training in Developmental Disabilities. dThe British Journal of Developmental Disabilities.
Following the selection of journals, we consulted with a graduate-level library scientist at a research university to identify the number of articles comprising the population of special education articles published between 1999 and 2019. We observed this publication range based on the earlier time frame reviewed by Mastropieri et al. (2009) and the review process of the WWC (2020), which restricts searches to within 20 years of the date of the review. We selected databases with coverage of journals for every year of the search. A single database was used to separately search each year of individual publications to avoid duplicates. We searched the majority of journals (n = 24) exclusively through Scopus. The remaining journals were acquired through Education Source (n = 8) or PsycInfo (n = 1).
Sampling and Article Selection
The first author, a special education faculty member with experience conducting systematic literature reviews, and two doctoral-level graduate students with research experience obtained journal records by entering current and alternate titles journals within the source or publication name database fields. Searches were conducted in February, 2020. For each journal and year of publication, the research team randomly selected 50% of articles for review. Specifically, the search team (a) recorded the number of articles published per year for each journal in an Excel spreadsheet, (b) placed the numbers in a random order, and (c) selected the first 50% of the list. Articles therefore constituted a random sample comprising approximately 50% of the content released within each year of publication for all journals. From a total of 25,025 possible articles, we selected 12,669, or 50.62%.
Following article selection, we calculated the error for the total sample, journals, and journal categories given established parameters for confidence, population, and sample size. The error indicates the extent to which estimates of methodological prevalence obtained from samples of the population may deviate from the true value obtained by assessing the entire population (i.e., examining every article). In survey research involving the estimation of proportions from known populations, an acceptable sample size is generally derived using Cochran’s (1977) formula, where z is the standardized confidence interval (CI; 95% CI = 1.96), p is the proportion of a population expected to exhibit a specific response (set to .5 to provide the most conservative sample size requirement; Blair & Blair, 2015), e is the margin of error (i.e., extent to which estimate varies from the true value), and N is the population size:
Due to variations in population (i.e., total articles published as indicated through database searches) and sample size (i.e., number of articles selected from each database), the error associated with estimates of prevalence varied across specific samples. Assuming a CI of 95%, the sample (n = 12,669) from the population of all journals (N = 25,025) was representative, with a negligible margin of error (±0.6%; i.e., .006; Daniel & Cross, 2018). Margins of error for samples of cross-categorical (±1.25%), developmental disability (±86%), EBD (±2.10%), and LD (±1.484%) journals were also acceptable. Although detailed claims regarding individual publications are beyond the scope of this publication, the error associated with individual journals (M = ±3.88%, range = 1.72%–6.41%, SD = 0.93) was also acceptable, with larger sample and population sizes (see Table 1) associated with smaller degrees of error. The average error for individual publications comprising the journal categories for cross-categorical (M = ±4.10%, range = 3.15%–4.90%, SD = 0.59), developmental disability (M = ±3.50%, range = 1.72%–6.41%, SD = 1.24), EBD (M = ±4.23%, range = 3.87%–4.59%, SD = 0.51), and LD (M = ±4.15%, range = 2.89%–5.54%, SD = 0.87) was also acceptable.
Coding
A team of coders consisting of the first author and graduate students with experience coding research articles (n = 5) coded all articles. Codes were mutually exclusive and assigned according to a hierarchy, meaning that articles qualifying for a higher level category could not be considered for a lower level category regardless of whether the study contained features relevant to multiple designs. This method was used to reduce ambiguity. Mixed-methods studies encompassing multiple designs were coded as the highest design category featured in the article. Articles received the following classifications: (a) single-case designs, (b) group designs, (c) descriptive-correlational research, (d) qualitative articles, (e) systematic reviews, and (f) commentary.
We defined a single-case design as any article in which the authors graphically presented original data in accordance with recognized single-case arrangements (e.g., Ledford & Gast, 2018). A group design study included any article that, as stated in the abstract or method, used original data to determine the difference between groups or within a single group following a treatment managed by the researchers. Articles determining the effect of changes beyond the control of the experimenters or that exclusively analyzed a secondary data set were not classified as group design. Descriptive-correlational studies included nonexperimental research with quantitative outcomes and included articles featuring examinations of simulated data, surveys, or secondary data. We defined a qualitative study as an article in which text taken from participants represented the primary data source. Articles were coded as qualitative if they presented original data and were described as a qualitative study, case study, interview, or questionnaire. Articles in which authors coded text and statistically analyzed quantified responses were coded as descriptive-correlational studies. Literature reviews included any article which examined previously conducted studies in accordance with an explicit search and selection procedure. As such articles feature procedures associated with primary research, we classified literature reviews as empirical articles. We coded articles without an explicit methodology (i.e., nonempirical; e.g., editorials, book reviews) as commentary.
Analysis
Given the nature of the research questions, analysis consisted of calculating descriptive statistics related to the prevalence of article types. Procedures entailed dividing the number of article types by the total of articles published within the sample, article category, or journal throughout the targeted publication range or within a specific year. We performed a similar procedure within each category to determine the relative prevalence of single-case and group experimental designs. We performed all calculations in Excel.
Interrater Agreement
Interrater agreement (IRA) was established through routine training procedures for staff and assessed at multiple points throughout the study. We collected IRA during article search, selection, and coding procedures. Across all procedures, we calculated IRA by dividing the number of agreements by the total number of coding opportunities and multiplying by 100%.
Search and article selection
Prior to identifying the number of articles in a journal, graduate assistants (n = 2) completed a training in which they (a) reviewed a written description of procedures and (b) collected yearly publication data for a journal previously completed by the first author. Agreement, determined via the point-by-point method (Ledford & Gast, 2018), was defined as an exact match between the number of articles identified for a specific year. Training concluded when agreement between graduate students and the first author reached 100% across every year for a single journal. Following training, the authors and graduate students independently determined the number of articles published in 100% of journals, with an IRA of 100%.
Graduate students also received training on random sampling and the selection of specific articles. Training procedures and criterion matched those employed in identifying articles. Agreement was defined as selecting an identical article for each sampled number. Two graduate students independently coded 27.27% of journals (n = 9). The average IRA was 100%.
Coding
Project staff completed a training prior to coding the research methodology featured in each article. Specifically, coders watched a series of video-recorded lectures in which the first author described each code and guided viewers through coding practice articles. Training articles were selected from journals outside of the scope of review. Following the lecture, each coder was required to obtain 100% IRA with the first author on three consecutive articles. Coders who did not successfully code practice articles were assigned additional articles until the criterion was met. Agreement was defined as selecting an identical method for a specific article. Two coders randomly selected and scored 20% (n = 2,542) of the total sample. The average IRA was 97% (range = 66.67%–100%; SD = 9.45). Coders reconciled disagreements through discussion.
Results
Total Sample
We identified 12,669 articles. On average, each journal contributed 384 articles (range = 106–1,615; SD = 277.62). Yearly, an average of 603 articles (range = 492–824; SD = 107.58) was published across journals. Journals published an average of 18.28 articles per year (range = 5.05–76.90, SD = 13.22). Commentaries represented 24.67% of articles (n =3,126); for the purposes of comparing empirical designs, we exclude commentaries from the remaining results. Study proportions relative to commentaries appear in Table 1.
Distribution of article types
The majority of empirical articles (n = 9,543) were descriptive-correlational studies (57.86%; n = 5,522). Other article types, including single-case designs (14.22%, n = 1,357), group designs (11.29%, n = 1,077), qualitative studies (9.38%, n = 895), and systematic reviews (7.25%, n = 692), appeared less frequently. When accounting for the number of journals, descriptive-correlational studies (M = 167.33; range = 12–1,037; SD = 207.08) were the most common, followed by single-case (M = 41.12; range = 0–162; SD = 49.67) and group designs (M = 32.63; range = 5–222; SD = 36.50). Qualitative studies (M = 27.12; range = 0–167; SD = 31.78) and systematic reviews (M = 20.96; range = 0–124; SD = 22.25) were less common.
Experimental articles
The total number of experimental (n = 2,434) and nonexperimental (n = 7,109) articles published each year appear in Figure 1. The appearance of experimental articles has gradually increased since 1999. In contrast, nonexperimental articles increased sharply in 2008. Excluding all nonexperimental article types, single-case designs represented over half of the published articles (55.75%; n = 1,357), with group designs (44.25%; n = 1,077) encompassing fewer studies. On average, journals published 45.05% single-case designs (range = 0%–88.89%, SD = 27.16) and 54.95% group designs (range = 11.11%–100%, SD = 27.16). The yearly publication rate of single-case designs (M = 55.61%; range = 44.76%–63.16%; SD = 5.14) exceeded that of group designs (44.39%; range = 34.72%–55.24%; SD = 5.14). Single-case designs initially represented more than 50% of experiments published, but exhibited a declining trend between 1999 and 2008. Nonetheless, the percentage of single-case designs published each year surpassed group designs except in 2005–2006 and 2008.
Studies in the total sample and cross-categorical journals published from 1999 to 2019.
Cross-Categorical Journals
Cross-categorical journals included 10 journals and 3,001 articles (22.82% of all articles). On average, 300 articles (range = 196–452; SD = 93.50) appeared in each journal. Yearly, an average of 142.86 articles (range = 127–158; SD = 7.94) were published across journals.
Distribution of article types
Of the empirical articles (n = 1,969), descriptive-correlational studies (46.80%; n = 922) represented the majority of published articles. Single-case designs (16.9%; n = 332), group designs (12.4%; n = 245), qualitative studies (13.8%; n = 272), and reviews (10.1%; n =198) appeared less frequently. Journals published an average of 92.2 descriptive-correlational (range = 12–165; SD = 44.34), 33.2 single-case designs (range = 5–143; SD = 40.30), 27.2 qualitative studies (range = 0–70; SD = 212.81), 24.5 group designs (range = 11–34; SD = 8.32), and 19.8 review articles (range = 8–47; SD = 12.42).
Experimental articles
Nonexperimental articles (n = 1,392) have exceeded experimental articles (n = 577) since 1999; however, both methodologies have increased (see Figure 1). Single-case designs represented the greater proportion of experiments (57.54%; n = 332) compared with group designs (42.46%; n = 245). Journals published an average of 47.63% single-case (range = 16.67%–87.20%; SD = 22.14) and 52.37% group designs (range = 12.80%–83.33%; SD = 22.14). The yearly publication rate of single-case designs (56.34%; range = 32.34%–74.20%; SD = 11.47) surpassed that of group designs (43.67%; range = 25.81%–68.96%; SD = 11.47). From 1999 to 2008, single-case and group design studies were approximately equivalent. After 2008, single-case designs represented over half of experimental articles published per year (see Figure 1).
Developmental Disability Journals
Twelve journals focused on students with developmental disabilities, encompassing 6,384 articles (48.55% of all articles). On average, 531.83 articles (range = 106–1,615; SD = 409.38) appeared in each journal, with an average of 303.9 articles (range = 185–535; SD = 111.25) appearing each year.
Distribution of article types
Descriptive-correlational articles represented 64.5% (n = 3,452) of empirical studies (n = 5,351). The remaining articles included single-case designs (12.9%; n = 689), group designs (9.5%; n = 507), systematic reviews (6.8%; n = 363), and qualitative studies (6.4%; n = 340). Journals focused on students with developmental disabilities published an average of 287.67 descriptive-correlational (range = 59–1,047; SD = 302.9), 57.42 single-case designs (range = 5–162; SD = 58.82), 42.25 group designs (range = 6–222; SD = 58.98), 30.25 systematic reviews (range = 5–124; SD = 32.84), and 28.33 qualitative articles (range = 2–60; SD = 21.3).
Experimental articles
Until 2008, the rate of increase in nonexperimental articles (n = 5,351) was consistent with the change in experimental articles (n = 1,196). Beginning in 2008—the approximate time in which two journals were added (see Table 1), the trend of nonexperimental articles sharply increased, before declining in 2013 (see Figure 2). Group designs (42.40%; n = 507) appeared in fewer articles than single-case designs (57.6%; n = 689). Journals published an average of 52.82% single-case (range = 18.52%–87.02%; SD = 23.82) and 47.18% group designs (range = 12%–81.49%; SD = 23.82). Yearly rates of single-case designs (60.72%; range = 53.19%–72.31%; SD = 5.32) exceeded group designs (39.2%; range = 27.69%–46.81%; SD = 5.32). Single-case designs represented over half of the studies between 1999 and 2005. From 2005 to 2009, single-case and group designs were roughly equivalent. After 2009, single-case designs increased above 50% and remained the most common experiment. The proportion of single-case designs declined following 2014 (see Figure 2).
Article types published from 1999 to 2019 within the journals for individuals with developmental disabilities, emotional/behavioral disorders, and learning disabilities.
Emotional and Behavioral Disorder Journals
The EBD category included four journals with a total of 1,117 articles (11.7% of all articles). On average, 279.25 articles (range = 234–333; SD = 39.58) appeared in each journal. Yearly, an average of 38.18 articles (range = 32–43; SD = 3.04) was published across journals.
Distribution of article types
Out of the empirical articles (n = 802), 315 descriptive-correlational (39.1%), 297 single-case designs (37.03%), 98 group designs (12.21%), and 67 systematic reviews (8.35%) appeared. Qualitative studies (3.11%; n = 25) were rarely identified. Journals published an average of 78.75 descriptive-correlational articles (range = 47–131; SD = 38.69), 74.25 single-case designs (range = 11–138; SD = 54.87), 24.5 group designs (range = 12–39; SD = 12.66), 16.75 systematic reviews (range = 9–27; SD = 7.59), and 6.25 qualitative articles (range = 2–12; SD = 4.19).
Experimental articles
A roughly equivalent number of experimental (n = 395) and nonexperimental (n = 407) designs have appeared each year since 1999 (see Figure 2). Single-case design (75.19%; n = 297) represented the majority of articles. Fewer group designs (24.81%; n = 98) appeared in EBD journals. Journals published an average of 67.25% single-case (range = 22.0%–88.89%; SD = 30.6) and 32.75% group designs (range = 11.11%–78%; SD = 30.59). Though variable, single-case designs consistently represented over half of the articles published each year (see Figure 2).
Learning Disability Journals
Seven journals focused on students with LD, encompassing 2,167 articles (16.48% of all articles). On average, 309.57 articles (range = 151–570; SD = 144.07) appeared in each journal, with an average of 67.67 articles (range = 49–98; SD = 13.79) appearing each year.
Distribution of article types
Descriptive-correlational articles represented 58.6% (n = 833) of empirical studies (n = 1,421). The remaining articles included qualitative studies (18.2%; n = 258), group designs (16.0%; n = 227), systematic reviews (4.5%; n = 64), and single-case designs (2.7%; n = 39). Journals focused on people with LD published an average of 119 descriptive-correlational (range = 57–344; SD = 102.35), 38.86 qualitative articles (range = 1–167; SD = 58.57), 32.43 group designs (range = 5–49; SD = 14.33), 9.14 systematic reviews (range = 0–20; SD = 7.47), and 5.57 single-case designs (range = 0–13; SD = 5.59).
Experimental articles
Although both types of articles exhibited positive trends since 1999, the rate of nonexperimental studies (n = 1,155) increased more sharply than experimental designs (n = 266; see Figure 2). Single-case designs (14.66%; n = 39) appeared in fewer articles than group designs (85.33%; n = 227). Journals published an average of 84.64% group designs (range = 62.5%–100%; SD = 14.25) and 15.36% single-case designs (range = 0%–37.5%; SD = 14.25). Yearly rates of group designs (84.49%; range = 57.14%–100%; SD = 12.56) exceeded single-case designs (15.52%; range = 0%–42.86%; SD = 12.56). Despite an increasing trend following 2008, single-case designs consistently represented less than half of the articles published each year (see Figure 2).
Discussion
This study examined the prevalence of research designs in peer-reviewed special education journals. The majority of identified articles were empirical. Descriptive-correlational studies represented the most frequently identified study regardless of journal category. Qualitative work, though more common than previously reported (Trainor & Graue, 2014), remains scarce, particularly in journals regarding students with EBD. Although the number of experimental designs has steadily increased, nonexperimental articles represented the clear majority of studies within the total sample and across categories, with the exception of EBD. Single-case designs represented the majority of experimental studies and appeared more commonly following 2008 and also represented the most common experiment featured in journals for people with developmental disabilities. Nonetheless, the publication rate of single-case designs in developmental disability journals has routinely declined since 2014. In contrast, single-case designs represented the dominant experimental design featured in publications focused on students with EBD, yet consistently represented very few of the experimental studies conducted among students with LD.
Differences in the methods used in previous studies to assess the prevalence of research design prevent clear comparisons of findings. Nonetheless, our results in regard to the high prevalence of descriptive-correlational studies, the continued decline of commentary (i.e., 42% of articles published between 1988 and 2006, compared with 24.67% more recently), and the scarcity of qualitative work are generally consistent with earlier work (Mastropieri et al., 2009). Our findings also suggest that single-case designs represent a higher proportion of experimental studies than previously indicated (e.g., Zanuttini, 2020), with a similar pattern observed in cross-categorical journals. However, cross-categorical journals included an outlier with a high number of single-case designs (i.e., Journal of Behavioral Education; see Table 1), suggesting single-case design is concentrated in journals dedicated to people with developmental disabilities and EBD. That the average proportion of single-case designs increased within these journals over the course of the following decade possibly reflects the greater acceptance of single-case designs accompanying detailed WWC guidelines (e.g., WWC, 2011).
Although the initial evidence-based practice movement implicitly rejected qualitative research and other nonexperimental approaches (e.g., Song & Herman, 2010), our findings suggest there is no shortage of alternatives to experimental research. Descriptive-correlational studies comprising the majority of published research provide insight into areas typically resistant to experimentation (Thompson et al., 2005). However, Reinhart et al. (2013) suggest such work has the potential to displace experimental research or overstate relationships between variables through sophisticated statistical modeling. An evaluation of this considerable research base in future work may therefore be warranted. Our results further suggest that qualitative work continues to be underrepresented in general and among specific populations (e.g., EBD).
Patterns of research prevalence may stem from the costs and incentives associated with individual designs. Substantial funding is often unnecessary for single-case designs, and agencies such as the Institute of Education Sciences (IES) fund single-case design research for the purposes of pilot and primary studies (Okagaki, 2014). Similarly, descriptive-correlational studies may often be conducted using publicly accessible data or through online surveys that quickly facilitate data collection. Although IES funds qualitative research for exploratory projects, some researchers have suggested this designation fails to recognize the full potential of the design (Trainor & Graue, 2014). The suppressive effect of limited funding is likely compounded by the resource-intensiveness of many qualitative designs.
The disparities in design rates across journal types are consistent with the evidence provided in previous reviews (e.g., Zanuttini, 2020). The prevalence of single-case designs across journals for students with EBD, relative to qualitative or group designs, is potentially problematic. First, experimental studies provide limited insight into the perceptions of consumers, which does little to allay concerns regarding the disproportionate or inappropriate use of behavior interventions among historically marginalized groups (e.g., Simmons-Reed & Cartledge, 2014). Notwithstanding the limitations of group designs (e.g., Ioannidis, 2005) and the need for replications to provide evidence of generality (Sidman, 1960), the reliance on single-case designs may raise issues regarding the external validity of interventions used among students with EBD (see WWC, 2020). On the contrary, the predominance of group design in LD journals could result in interventions targeted toward the “average” at-risk student rather than students who require more intense support (King, Wang, et al., 2022).
The use of single-case designs in EBD journals relative to publications involving developmental disabilities is surprising given that (a) many of the methods used among both populations stem from behavior analysis and (b) researchers frequently examine manualized interventions for children with EBD using group designs (Brookman-Frazee et al., 2006). The unanticipated downward trend of single-case designs in journals involving developmental disabilities contradicts findings identifying single-case designs as the most common form of experiment (Demchak et al., 2019) and potentially reflects recent attempts by scholars to address questions amenable to group design research (e.g., moderator analysis; Kasari & Smith, 2016). The increasing emphasis on nonexperimental studies in journals for students with developmental disabilities may have also played a role in the decline of single-case studies.
Limitations
This study has several notable limitations. Aggregating journals into categories potentially obscures trends related to specific subcategories of disabilities. This is particularly true given the eclectic nature of specific journals. Publications such as the Journal of Positive Behavioral interventions— though previously identified as having an emphasis on EBD (e.g., Spear et al., 2013)—present articles pertinent to various disabilities. Concerns regarding categorization accuracy are partially mitigated by the disclosure of results for each journal in Table 1.
Reviews must be kept up to date to avoid potentially misleading consumers (Bashir et al., 2018). The search featured in this study failed to account for articles published beyond 2019. A number of specific extenuating considerations warrant further discussion. First, the study is a survey of thousands of papers, rather than a more focused systematic review limited to the evaluation of dozens of articles. Second, previous surveys of relatively smaller portions of the literature (e.g., Mastropieri et al., 2009) have reported an extended latency between the date of the search and the final date of publication (e.g., 2006–2009) due to the effort involved in coding and the delays associated with peer review. Third, the study relates to trends in research methodology rather than any specific intervention. The age of the search should certainly be considered by consumers, but is unlikely to pose threats to practitioners or service recipients.
The current method had the advantage of identifying the relative prevalence of research regardless of whether authors described design characteristics in the title, abstract, or keywords. Though comprehensive, the search did not feature all journals within or outside of special education relevant to people with disabilities. Included journals appeared in a widely accepted subject classification system and exceeded the number included in previous examinations of research prevalence (e.g., Zanuttini, 2020). Regardless, the final sample may not meet the conception of special education journals held by all readers.
Future Directions
Variability in the use of research may not be limited to prevalence. Future examinations of methodological issues should consider assessing characteristics across journal categories. For example, LD-focused publications featuring more eclectic research teams with access to larger populations may be more likely to eschew methods associated with smaller populations (e.g., single-case designs). Similarly, research featuring children who demonstrate issues with short-term memory may be more likely to feature methods less suitable among general populations. (e.g., multiple-baseline across settings; Ledford & Gast, 2018).
The volume of studies retrieved in the current review has additional implications for the evaluation of special education research. Previous scholarship has attempted to identify publication or methodological trends (e.g., study quality) using traditional systematic reviews (e.g., system of least prompts, n = 86; Shepley et al., 2021). The breadth of published scholarship suggests consumers should exercise caution in extrapolating such findings to the larger research base. We recommend authors interested in assessing the broader special education literature consider methods to procure representative samples.
As noted by Mastropieri et al. (2009), the quantity of research provides limited insight into its quality or impact. Nonetheless, our findings suggest that the research base of special education has continued to grow, with an increasing emphasis on experimental designs. While this emphasis certainly contributes to the evidence-based practice movement, the full range of research designs is needed to identify effective instruction and potential challenges to implementation (Cook & Odom, 2013). Diversification of research is likely to manifest in different ways across publications given the distinct pattern of publication observed across disability categories. Mixed-methods research, which accounts for less than 1% of research in special education (Corr et al., 2021), may be used to mitigate the weaknesses of research designs used in isolation. Given the increase in research over the past several decades, we are confident the field will continue to produce work commensurate to the challenge of serving children with disabilities.
Footnotes
Author’s Note
Olivia Enders is now affiliated with Coastal Carolina University, South Carolina, USA.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
