The good,the bad and systematic reviews

Countless interventions have claimed to treat all aspects of autism spectrum disorder (ASD). Some of these claims are doubtful, some more reasonable; some treatments and associated claims we may deeply distrust and others might be our favorites; no matter our personal opinion, only a few demonstrate consistently positive effects under rigorous conditions (Bölte, 2014). Currently, exhaustive systematic reviews of the scientific literature are deemed the gold standard for evaluating evidence in clinical science, especially if the reviews conform to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement or the standards of the Cochrane Collaboration (http://www.cochrane.org). Many researchers (including me) and national health care authorities around the world use systematic reviews increasingly to summarize the empirical evidence for the efficacy of therapeutic methods. For example, the Cochrane Collaboration has published more than 20 autism-related reports since 2005. The most recent ones (social skills training groups, parent-mediated early intensive treatment, theory of mind-based intervention) (Fletcher-Watson et al., 2014; Oono et al., 2013; Reichow et al., 2012) all concluded that there was some evidence to support the class of interventions they reviewed, but all also emphasized that the quality of most studies is too low to draw firm conclusions. Some other reviews, including those for auditory integration, acupuncture, and gluten- and casein-free diets, found no support for the interventions they reviewed (Cheuk et al., 2011; Millward et al., 2008; Sinha et al., 2011), but also judged study quality to be poor. More Cochrane systematic reviews on intranasal oxytocin, chelation, and hyperbaric oxygen treatment are underway. The rise of systematic reviews is an excellent and vital development for the field of ASD, helping to strengthen and prioritize methods in clinical practice endorsed by rigorous studies, instead of methods based only on hopes and beliefs. Systematic reviews provide clarity in the muddy waters of studies that vary in kind, size, and quality; make research needs more visible; and motivate funding. Thus, systematic reviews (and also meta-analysis) are invaluable in contemporary applied clinical research.

However, systematic reviews do not always tell the whole truth either. For many of you, this may be self-evident and not worth an editorial at all. Still, I now have encountered many situations in which members of the general public, stakeholders, and colleagues are unfamiliar with the significance and limitations of systematic reviews. Systematic reviews often receive wide public attention. They have the potential not only to improve understanding but also to confuse and unnecessarily upset people, if the results are not carefully conveyed.

Let me explain by providing an example from Scandinavia. In Sweden, the so-called Swedish Council of Health and Technology Assessment (Statens Beredning för Medicinsk Utvärdering, SBU) has the mandate from the Swedish Government to comprehensively assess health care technology. Recently, it published a systematic review on treatments (and diagnostic instruments) for ASD and attention deficit hyperactivity disorder (ADHD) (SBU, 2012). The findings were quite in line with other international studies: the review found that the evidence base for the diagnostic accuracy of tools and the efficacy of blinded non-pharmacological treatments was limited when strict criteria for scientific quality were applied. The SBU, although a national authority, obviously had neither prepared a proper communication strategy nor involved or informed interested organizations or professional societies prior to publishing the report. Moreover, SBU’s brief press release could easily be misread as “No evidence for anything” and was uncritically summarized this way by the popular media. This was warmly welcomed by the anti-psychiatry movement, which took the opportunity to use the news as official scientific support for their far-reaching negative conclusions about psychiatry, psychiatric diagnoses, and treatments. What followed was a cascade of accusations and hard feelings among stakeholders, service providers, and researchers and some desperate attempts by SBU to communicate that the true message from the systematic review was only that more research is needed. However, the damage was done, and the aftermath seems like that existing serious ASD and ADHD research and clinical services were jeopardized and stakeholders troubled. Health care policy makers felt the need to revisit whether current treatment, such as early intervention for children with ASD, was really worth financing. This story may appear extreme or strange and not really specific to systematic reviews. Still, the high value placed on systematic reviews and associated attention suggests the particular need to reexamine how we communicate research findings in general and those from systematic reviews in particular.

Let us start with science communication. SBU obviously did not act sensitively in this case. A national authority should be aware that a thoughtful plan for communication and prior exchange with interest organizations and opinion leaders is required when publishing a report that is highly relevant for a broad audience. Different audiences need different information. What information is relevant to clients and their families, clinicians, the public, politicians, and how must it be framed to be comprehensible? These questions were largely ignored. Actually, SBU’s own manual for conducting systematic reviews stresses the significance of social responsibility when carrying out this kind of research, but, unfortunately, it seems it was not taken that seriously. We all might want to learn from this. On one hand, we cannot expect that the public and even our own peers understand research findings right away without background information. On the other hand, we need to be aware that what we do is potentially a lot more significant and of more interest to a broader audience than we think. SBU had apparently no sense of how important their report was and that it could be misunderstood. Just recently, my research group conducted a systematic review (PROSPERO, reg. no. CRD42013003780) on interventions to improve social communication in ASD for the “Swedish Board of Health and Welfare” (Socialstyrelsen). In order to comply with the social requirements of mental health research, we introduced a bottom-up driven approach. To incorporate viewpoints from stakeholders on the focus, objectives, and results of the project, we conducted two surveys with three major Swedish interest organizations. This strategy identified the significance of interventions involving significant others, a group of methods that have not been systematically evaluated previously. In addition, also in cooperation with interest organizations as well as a range of clinicians and the Swedish Board of Health and Welfare, we based the choice of methods on the current applicability in Swedish clinical practice. That is, we prioritized to examine those methods for which there is a wide consensus about their potential usefulness and which are actually used in clinical practice in Sweden. Thus, we selected those treatments that are from a societal point of view at the moment, probably the most relevant ones to be critically reviewed. Hopefully, such strategies and actions help to increase science communication and understanding.

How should systematic reviews be understood, and what background information should be provided to the public? Most importantly, it must be clear that systematic reviews on a certain treatment must not be confused with the quality of local services. While systematic reviews examine and condense primary intervention research studies on one method on a group level and a narrow set of outcomes (e.g. ASD specific symptoms), services can be adaptive and individualized, with combined and integrative approaches. Although there are no data published to support the hypothesis that eclectic approaches lead to better outcomes than single-method treatment, it is worthwhile to remember that it might be the case. This is even truer for diagnostic instruments, which usually are combined with expert clinician judgment to achieve a best estimate clinical diagnosis. Of equal importance, in community settings, the goals of treatment (e.g. functional adaptation, school performance, quality of life, perceived stress, comorbidities) often are very different than what is measured in university-based clinical trials. The trials may therefore not report on outcomes of greatest importance to people with ASD and their families.

Systematic reviews also have some limitations that hamper their use as a strategy for selecting scientifically supported methods for evidence-based practice. For instance, they traditionally focus on internal validity, while external validity including implementability has been overlooked (Jonsson et al., in preparation). The lack of consideration of external validity is the most frequent criticism by clinicians (Rothwell, 2006), and the need for increased attention on the selection of participants and their characteristics, differences between the protocol and routine practice, choice of outcome measures, follow-up timing, providers’ characteristics and training, treatment fidelity, and the number of participants have been highlighted (Gardner et al., 2013; Grant et al., 2013). A second, perhaps more critical limitation of systematic reviews is that the quality of treatment delivery is rarely measured or addressed in the review, despite consistent findings that treatment fidelity often is low, especially in community-based studies. There are also limitations concerning the quality (psychometric properties) and elasticity (sensitivity to change) of outcome measures commonly used in clinical trials. Several studies have either used scales without sufficiently established psychometric properties or the outcome measures were not constructed to assess change. This limitation, in combination with studies being of small sample size and limited power to detect significant differences, makes it difficult sometimes to identify treatment effects in primary research studies. Although this is also true for the synopsis of a systematic review, the concept might connote that it could overcome such limitations. A third limitation relates to the fact that autism intervention studies are notoriously difficult to conduct. There are ethical challenges (e.g. compliance to participate, randomization to control treatments), societal issue (e.g. funding opportunities for large scale thorough randomized controlled trials (RCTs)), practical issues (e.g. conducting a study in a community setting), and scientific issues (e.g. blindness, drop-out, missing data). Surely, there are no perfect studies, but all obstacles in intervention research make even almost “perfect” studies unlikely. Because systematic reviews are strict regarding their inclusion criteria and evaluation of bias risk, many studies that could contribute important information often are not included. Finally, as research is progressing rapidly in autism intervention, many systematic reviews are already obsolete when they are published. Because of varying lag times in publication, different reviews appearing in the same period may come to different conclusions. For instance, the Cochrane Review on social skills training in ASD (Reichow et al., 2012) that was published about the same time as the one by SBU included five RCT studies and found some evidence to support these interventions, while the SBU report only included three RCT studies and concluded no evidence for this group of methods.

In recent Cochrane reports, the concluding remarks often are balanced and understandable for many readers. Cochrane reports with null findings based on a few studies tend to read “still too little available high-quality research,” rather than “no evidence for anything.” I would urge government bodies such as SBU to adopt a similar cautious tone, especially when reviewing treatments that may represent the best hope for the people to whom they are delivered.