We must improve the low standards underlying “evidence-based practice”

What is evidence-based practice?

Evidence-based practice is the process of identifying the best available evidence to make decisions about practices that should be deployed to support individuals in a given population (McKibbon, 1998, see Vivanti, 2022, for a review in relation to autism). Practices that meet a predefined set of evidentiary criteria are labeled “evidence-based practices” (EBPs ¹ ), to promote their adoption by service providers. A tenet of EBP is that the research used to designate EBPs should be rigorous, with the fewest risks of bias possible (Slavin, 2008). Several groups have developed EBP frameworks for conveying the extent to which intervention practices designed for autistic people are supported by evidence (e.g. National Autism Center, 2015; Reichow et al., 2008; Smith & Iadarola, 2015; Steinbrenner et al., 2020). These frameworks include algorithms for determining the number and types of studies showing an effect that must be available, and set standards for determining which studies qualify as evidence. The amount and type of evidence that should be required to designate EBPs is controversial, and autism researchers have debated about how standards should be operationalized in EBP frameworks since their initial development (Odom et al., 2005). Critics of autism EBP frameworks have argued that they: do not consider the scope of change indexed by outcome measures so that broad, developmental change and narrow, context-bound change are conflated (Sandbank et al., 2021) ² ; lead to an overestimation of effectiveness by tallying studies that show effects while ignoring gray literature, studies showing null effects, and studies showing iatrogenic effects (Sandbank et al., 2020; Slavin, 2008); and use taxonomies for categorizing practices that confuse practices and specific components of those practices (Ledford et al., 2021). The aim of this editorial is to point out another limitation of autism EBP frameworks, which is that research quality thresholds are much too low for making determinations about which interventions are likely to be efficacious. Low standards result in practices with questionable efficacy being labeled EBPs and promoted for use, and perpetuate the continued production of low-quality autism intervention research.

How have EBP frameworks been used in autism research?

The Autism Working Group for the National Institutes of Health produced one of the first formal reports on the “state of the science” in autism research, which included a review of social and behavioral interventions (Bristol et al., 1996). This report concluded that robust experimental designs that, among other features, used randomization procedures and included assessors who were free from conflicts of interest (COIs) were needed to identify which practices showed evidence of gains on the outcomes that were measured. Reichow and colleagues (2008) cite this report, and several subsequent reports that also failed to find sufficient evidence for any specific practice, as an impetus for developing an autism-specific EBP framework that would be more “sensitive” to the available evidence (p. 1312). In their framework, Reichow and colleagues developed evaluation rubrics for single case and group design studies that distinguished between “primary” and “secondary” quality indicators. The former were considered crucial to establishing rigor, while the latter were considered useful but not necessary. Primary quality indicators included adequate descriptions of participant characteristics, independent variables, and comparison groups or conditions; appropriately conceptualized and described dependent variables; strong links between research questions and data analysis; and appropriate statistical tests. Important design features such as random assignment, masked assessors, interobserver agreement on coded variables, measures of attrition, and effect size reporting were considered secondary, and researchers being free of COIs was not listed among as either primary or secondary quality indicators. According to this EBP framework, a practice could be considered “established” even if there were no available studies that used randomized designs and masked assessors, and all of the available evidence was produced by researchers who were also intervention providers. Disregarding quality standards in this way is a significant departure from cross-disciplinary evaluation frameworks such as those developed by the Cochrane Collaboration, which emphasize that all risks of bias undermine our ability to interpret the evidence provided by intervention studies. Despite these shortcomings, Reichow and colleagues’ EBP framework has been used numerous times to evaluate autism intervention practices (at the time of this writing, it has been cited more than 500 times).

Later, the National Professional Development Center (NPDC), which was later named the National Clearinghouse on Autism Evidence and Practice (NCAEP), developed their own framework for designating EBPs, and released scoping reviews of autism intervention research from early childhood through early adulthood, in 2014 (Wong et al., 2015) and again in 2020 (Steinbrenner et al., 2020). Their EBP framework is similar to Reichow and colleagues,’ in that research designs with risks of bias such as non-randomization and unmasked assessors were considered sufficient for establishing EBPs, so long as a set number of reports were published by two or more research groups. In the most recent report, 28 focused intervention practices were designated as EBPs (Steinbrenner et al., 2020). The NPDC/NCAEP reports have been enormously influential; the 2014 report was cited over 1500 times at the time of this writing, and the 2020 report was cited more than 300 times in just over 2 years since publication. Other influential, although less ambitious reviews and meta-analyses have also been conducted with the goal of describing the extent to which practices are evidence-based, each with quality review frameworks that consist of lowered standards relative to other fields of research (e.g. Odom et al., 2003; Smith & Iadarola, 2015; Virués-Ortega, 2010; Virués-Ortega et al., 2013).

Crucially, none of these EBP frameworks considers whether intervention researchers measure or report on adverse events, which are unintended negative consequences of interventions that can cause short- or long-term harms. This is problematic because selecting interventions should involve appropriate weighting of the potential for benefit against the potential for harm. The pairing of low standards with insufficient consideration of adverse events that is common to each of these frameworks could mean that researchers routinely recommend interventions that confer little or no benefit, while also inadvertently putting autistic people at risk of harm.

How do EBP reports compare to rigorous risk of bias assessments?

In collaboration with my colleagues, I have worked on two comprehensive risk of bias evaluations of autism intervention research that provide contrasts to the EBP frameworks commonly used in autism research, and to the categorization of EBPs that result from applying these frameworks. The first is Project AIM (Sandbank et al., 2020), a meta-analysis that synthesized intervention effects for 150 group design intervention studies conducted with autistic children up to age 8. In addition to synthesizing effect sizes, this study applied a tool created for cross-disciplinary use to identify risks of bias in group design studies (Higgins et al., 2021). This review found that there were very few studies that were free from risks of bias, and effect sizes decreased and were no longer significantly different from zero when studies with significant risks of bias were excluded. A second systematic review evaluated autism intervention research on transition-age youth (Bottema-Beutel et al., 2022). This study used two risk of bias tools; a recently created risk of bias tool for single-case design studies meant to be analogous to the Cochrane Collaboration’s group design tool (Reichow et al., 2018), ³ in addition to the risk of bias tool for group design studies. This study found that very few studies described sufficient randomization procedures or used masked assessors, and none of the 1258 outcomes were measured in designs that used both sufficient randomization procedures and masked assessors. Across these two reviews, we found that adverse events were rarely mentioned (they were mentioned in 7% of studies in our review on young children, and in only 2% of studies in our review on transition-age youth), but there is nevertheless evidence that they do occur (Bottema-Beutel et al., 2021a, 2022).

The conclusions from these two quality reviews starkly contrast with findings from EBP reports. For example, nearly half of the 28 practices designated as “evidence-based” in the most recent NCAEP report were behavioral (i.e. practices that rely on manipulating behavioral antecedents and consequences to shape new behavior). ⁴ Similarly, Smith and Iadarola’s (2015) report concluded that behavioral practices either alone or in combination with developmental practices were “well established,” and the National Autism Center (2015) considered a variety of behaviorally-based interventions to be “established.” However, in Sandbank et al. (2020) , we showed that there were too few randomized controlled trials of behavioral interventions to make any conclusions about their efficacy for autistic children. In our review of interventions for transition-age autistic youth (Bottema-Beutel et al., 2022), we found that although 70% of the interventions tested were behaviorally-based, quality concerns prevented us from considering any intervention practice to have sufficient evidence. Because autism EBP frameworks do not distinguish between research that adheres to some quality standards but is still designed with significant risks of bias, and research with minimized risks of bias, the reports may mislead researchers, practitioners, and commissioners of services to conclude that behavioral interventions are better supported by research evidence than other kinds of interventions, given the high number of behavioral strategies labeled as EBPs. In reality, behavioral intervention research has more risks of bias relative to research examining other types of interventions (Sandbank et al., 2020).

Some researchers have argued that reviews that only examine group design studies miss contributions of single-case designs to the overall body of evidence supporting interventions, and that including single-case evidence would lead to conclusions that are aligned with autism EBP frameworks (e.g. Steinbrenner et al., 2020; Vivanti, 2022). There are two reasons to argue that this is not likely the case. First, single-case design research is not well suited to providing evidence of effects that extend beyond the specific behaviors that are the focus of the intervention, and therefore have limited utility for understanding the kinds of outcomes that are most likely to be of developmental or social importance to autistic people (Sandbank et al., 2021). Second, our review of intervention research on transition-age autistic youth (which examined both group- and single-case designs) showed that significant quality concerns extend to single-case design research. Therefore, the inclusion of single-case designs in EBP frameworks does not increase confidence that they draw appropriate conclusions regarding intervention efficacy.

What is the potential fallout of using low standards in EBP frameworks?

There are at least two consequences of autism EBP frameworks that count low-quality research as sufficient evidence for establishing efficacy; they perpetuate the production of low-quality research, and they lead to the proliferation of autism support services that may be ineffective and/or are potentially harmful. Some EBP guidelines suggest that, when research is scarce, lowered standards of rigor may help to reflect the small amount of available evidence (Slavin, 2008). At present, however, autism intervention research is decidedly not scarce. Comprehensive reviews across age groups have identified nearly 1000 reports over the last 30 years, and the rate of production of autism intervention research continues to increase (Steinbrenner et al., 2020). Although these efforts represent millions of dollars of yearly research expenditures (Cervantes et al., 2021), the number of high-quality studies remains scarce. It could be that EBP designations currently used in autism research, which gloss over important risks of bias, disincentivize researchers from conducting studies with appropriate rigor. Researchers may misinterpret EBP quality criteria as the “gold standard” of research design, which is especially problematic in autism research because the standards are so low. Current EBP frameworks may also perpetuate the inaccurate notion that if a sufficient number of studies with threats to internal validity are available, those threats are somehow offset (Slavin, 2008).

It is important to consider that autism intervention research, as a field, has historically mixed the provision of clinical services with the evaluation of those same services. That is, COIs abound (Bottema-Beutel et al., 2021b). COIs may hinder calls for increased rigor in EBP frameworks, because researchers may benefit from seemingly neutral third-party organizations declaring the practices they study “evidence-based” after applying lax quality criteria (note however that at least one EBP framework is produced by clinical providers, and therefore is not a “neutral” third party; the May Institute, a conglomerate of behaviorally-based autism service providers, produced the Autism Standards Project Reports).

The conclusions drawn from EBP reports, or other reviews that aim to categorize practices based on levels of evidence, are taken up by implementation researchers and policy-making bodies that make these practices widely available. Dillenburger and colleagues (2014) trace a long history of researcher and policymaker recommendations (across the United States, Canada, and United Kingdom, Ireland, and Australia) for providing young autistic children with early intensive behavioral interventions, citing evidence from these reports without mentioning research quality concerns. More recently, influential stakeholder groups, including for example, the American Academy of Pediatrics (Hyman et al., 2020) and the Lancet Commission on the Future of Care and Clinical Research in Autism (Lord et al., 2022) reference EBP reports as a rationale for their recommendations to practitioners and policymakers about the types of support that should be provided to autistic people. Other initiatives, such as the Autism Focused Intervention Resources and Modules project, have built infrastructure for wide dissemination of training modules to practitioners to educate them on the use of practices that are designated EBPs (Sam et al., 2020), and researchers continue to call for increased implementation of EBPs (e.g. Boyd et al., 2021). Each of these initiatives recommends intervention approaches that do not hold up to scrutiny when research supporting them is evaluated using rigorous risk of bias tools.

It is concerning that, at least in the United States, the infrastructure for providing intervention is massively tipped toward intervention approaches that do not have the most rigorous support, likely because these approaches are labeled EBPs (Sridhar et al., 2022; Xu et al., 2019). Thus, even if families wanted interventions that incorporate practices with a firmer evidence base, it could be difficult to find a service provider or community setting that offers them (Hampton & Sandbank, 2022; Summers, 2021). When widely disseminated practices receive criticism (from researchers, autistic people, and people belonging to both groups), practitioners use EBP reports to deflect against such critique—without any consideration of the significant shortcomings of these reports (e.g. Leaf et al., 2022). An additional problem is that when the service infrastructure for providing an intervention outpaces the production of quality evidence, standard efficacy trials become impractical, because business-as-usual control groups receive much of the same types of supports as those provided to the intervention group (Lord et al., 2022). This creates a double bind; the evidence is not robust enough to support the adoption of a particular practice, yet the practice is already so widely adopted that robust evidence can longer be obtained.

Conclusion and looking forward

Designating EBPs is high stakes and shapes the kinds of support that are available to autistic people. Lowered quality standards mean that interventions with uncertain efficacy and unknown, unacknowledged potential for harm are now widely recommended for autistic children, youth, and adults. In addition, the widespread acceptance and adoption of EBPs backed by poor-quality research appears to have stalled innovation for creating new interventions that could be more rigorously tested.

Given the high volume of autism intervention studies produced each year, a reimagining of how EBPs are designated and understood is long overdue. A path forward could include abandoning autism-specific, arbitrarily lowered standards for evaluating research evidence. Instead, risk of bias tools that have been developed for cross-disciplinary purposes and that include consideration of adverse events could be used to determine which types of interventions are supported by high-quality research that minimizes all risks of bias (Higgins et al., 2021; Reichow et al., 2018). At the primary-study level, researchers could improve research quality in relatively low-cost ways by attending to research design elements such as appropriately randomizing participants (so that it is not possible for researchers to inadvertently manipulate group or phase assignment), selecting measures that can be administered in such a way that assessors are masked to group assignment, incorporating robust procedures for monitoring adverse events (which can be adaptations of measures developed in related fields; e.g. Hall et al., 2019), and building protections against COIs into their research designs. This latter suggestion could be implemented by incentivizing independent replications where none of the researchers hold COIs related to the intervention or, in preliminary research, by partitioning researchers with COIs from aspects of the study that involve participant assignment, data collection, data analysis, or interpretation of effects. As a final note, researchers (including those who conduct primary intervention studies and those who conduct evaluations or develop/use EBP frameworks) should exercise humility in their interpretations of research findings. This includes increased transparency about what is known and what remains uncertain regarding the efficacy of autism interventions, given the limitations of the research designs used to test them. Hopefully, an overhaul of EBP frameworks will culminate in an increase in autism intervention research that adheres to rigorous quality standards, and the practices available to autistic people will better reflect the most rigorously tested interventions.