Abstract
The extensive literature on ADHD describes students with ADHD as having difficulty with attention, excess motor activity, behavioral impulsivity, and deficits in neurocognitive abilities across a wide range of cognitive skills (e.g., working memory; Goepel, Kissler, Rockstroh, & Paul-Jordanov, 2011; Oades, Dauvermann, Schimmelmann, Schwartz, & Myint, 2010). ADHD often co-occurs with at least one type of learning disability (LD) in reading, writing, or mathematics (Al-Yagon, 2009; Capano, Minden, Chen, Scharchar, & Ickowicz, 2008; Eden & Vaidia, 2008; Mayers, Calhoun, & Crowell, 2000). Many research studies have found significant differences in behavior and cognitive skills between students with ADHD, ADHD-LD, and their peer groups (Doyle, Biedeman, Seidman, Weber, & Faraone, 2000; Goepel et al., 2011; Jakobson & Kikas, 2007; Reddy, Hale, & Brodzinsky, 2011). Students with this disorder face many difficulties in life and are at risk of academic failure and social isolation (Goepel et al., 2011; Salmeron, 2009; Westby & Watson, 2010). Because ADHD can negatively affect a student’s quality of life, the need for effective treatment and intervention is evident.
Pharmacologic intervention with drugs such as methylphenidate and dextro-amphetamine are typically the first line of treatment used to address the behavioral symptoms of ADHD in children. Although medication can reduce some of the negative behavior symptoms and improve cognitive functions (e.g., working memory; Chelonis et al., 2002), it is not the only way to address the problems associated with ADHD. Compliance and long-term use of medications can be problematic. In addition, some parents are reluctant to use medication because of the various side effects (e.g., poor appetite, tics, heart problems; Elia & Vetter, 2010). Furthermore, many students with ADHD frequently exhibit comorbid disorders (e.g., LDs, oppositional defiant disorder, and bipolar disorder; Mayers et al., 2000; Woziniack, 2001). Often researchers caution that pills do not substitute for the skills students with ADHD need to acquire and use (O’Leary, 1984). Accordingly, medication alone is not sufficient and other types of interventions must be considered.
Several interventions have been described in the realm of what may be considered behavioral or psychosocial treatments for ADHD (e.g., psychotherapy, cognitive behavior therapy, family counseling). Included are studies that examine outcomes of psychosocial and cognitive interventions between psychologists and children with ADHD or their parents, and educational interventions used by teachers in the classrooms. For example, behavioral treatment with antecedent- and consequence-based interventions is one school-based intervention that targets social and academic behaviors (DuPaul & Weyandt, 2006). Although psychosocial treatments are varied in scope, all tend to focus on modifying disruptive or distracting behaviors through an interaction between clinician/teacher and student/parent. For the purposes of this article, the use of the general term psychosocial treatments will be used to represent the variety of interventions subsumed in clinical practice. Current education practices and legislation (e.g., No Child Left Behind) require the use of research-based strategies to promote student learning and achievement. For this reason and because ADHD is such a disabling disorder, many individuals have conducted reviews of studies that have examined psychosocial treatments for ADHD (e.g., Murphy, 2005).
One of the highest levels of scientific evidence to support research-based interventions is high-quality systematic reviews and meta-analyses (Odom et al., 2005; Rubin, 2008). In fact, dozens of systematic reviews and meta-analyses have been conducted to coalesce the scientific evidence for the effects of different behavioral or psychosocial treatment approaches for ADHD in children. One important characteristic of systematic reviews is that they provide nonbiased, prefiltered evidence of evidence-based practice. Systematic reviews, however, need to be conducted with rigorous methodologies to avoid bias in their conclusions (Schlosser, Wendt, & Sigafoos, 2007). Jadad et al. (1999) published an appraisal of all systematic reviews to date of treatments for ADHD, both pharmacologic and psychosocial. At that time, they reported that most reviews were lacking in methodological rigor, therefore, the clinical implications of their appraisal were limited.
Several rating scales and checklists have been developed to guide the analysis of the quality of systematic reviews (Barker, 2010; Rubin, 2008; Schlosser et al., 2007). An Agency for Healthcare Research and Quality (AHRQ) committee report (West et al., 2002) noted that one of the strongest tools available at the time was a scale developed by Sacks, Reitman, Pagano, and Kupelnick (1996), later revised by Auperin, Pignon, and Pynard (1997). The internationally accepted Auperin et al. criteria scale addresses 27 key appraisal considerations for determining if a systematic review or meta-analysis has been conducted in a rigorous, unbiased fashion. The purpose of this article is to report an evaluation of the quality of systematic reviews and meta-analyses conducted for psychosocial treatments of ADHD in children using the Auperin et al. scale. We evaluated systematic reviews and meta-analyses, that were published since the Jadad et al. (1999) appraisal.
Method
The research team conducted a thorough search of the literature to identify systematic reviews and meta-analyses that examined the effects of some form of psychosocial intervention for ADHD. Electronic databases were searched (PubMed, CINAHL, PsycINFO, Cochrane Database of Systematic Reviews, American Speech-Language-Hearing Association Compendium) as were reference lists from retrieved articles. Search terms included “ADHD and systematic review” and “ADHD and meta-analysis.” English language studies were selected for the review, if they were published in peer-reviewed journals or book chapters from 1998, the time of the Jadad et al. (1999) appraisal article, to August 2010. From identified titles, the reviewers examined abstracts to determine whether the article met several inclusion criteria, including a systematic review or meta-analysis of some form of psychosocial treatment for ADHD, included studies conducted in children up to 18 years of age, published in English up to August 2010. Articles were excluded if they did not include intervention studies, only reviewed the pharmacologic literature and did not address psychosocial interventions for ADHD, focused on biofeedback studies that represented interventions between the student and a computerized system, were general review articles that did not use a systematic review methodology, or discussed interventions in adults. Reviews that discussed pharmacologic and psychosocial interventions for children were retained for this study to allow the psychosocial interventions to be considered. After a review of abstracts, 26 articles were identified that appeared to meet inclusion criteria to be evaluated by our research team. During the appraisal process, five additional articles were removed from consideration when it was determined that they did not represent systematic reviews (Carr, 2009), did not include a review of psychosocial treatments (Danckaerts et al., 2010), were practice guidelines rather than systematic reviews (New Zealand, 2001), and described reviews in progress without conclusions (Storebo et al., 2010; Zwi et al., 2009). Ultimately, 13 systematic reviews and 8 meta-analyses conducted from 1998 to 2010 that met criteria for inclusion in this study were identified (i.e., systematic reviews or meta-analyses on psychosocial interventions for children with ADHD).
Several scales and checklists are available to evaluate the quality of systematic reviews (West et al., 2002). We elected to use the Auperin et al. (1997) scale in our analysis. This checklist was selected because it was one of two checklists that were identified in an AHRQ report (West et al., 2002) to meet their standards of evaluating the strength of methodologic quality, the magnitude of effects across studies, and the consistency of findings across studies. The Auperin et al. scale was one of only two scales identified by West et al. (2002) that addressed seven specific criteria viewed as critical to the methodological quality of systematic reviews, that is, criteria that are pertinent to the internal validity of the studies included. The 27 quality-rating criteria proposed by Auperin et al. are listed in Table 1. The criteria examined the identification of protocol and selection of trials for each review (6 items), description of clinical trials (4 items), evaluation of study quality (3 items), description of data collection procedures (3 items), statistical analyses (8 items), and application of results (3 items). Each criterion was rated on a scale of 0 (not included), 1 (addressed partially), and 2 (addressed adequately). The maximum score on the 27-item scale was 54. Because the majority of the reviewed articles were systematic reviews, we also calculated a modified score that removed the 8 items that examined statistical analyses that are relevant to meta-analysis, allowing for a maximum systematic review score of 38.
Quality Review Rating Criteria From Auperin, Pignon, and Pynard (1997).
The four coders trained on the use of the Auperin rating system in two articles, meeting to discuss each item and reach agreement when there were discrepancies in scoring. Two reviewers then independently rated 6 of the 21 articles on the set of 27 quality criteria. Interexaminer reliability of scoring ranged from 81.5% to 97.3%, with an average of 89.4% agreement, which represents an acceptable level of reliability for the extensive list of rating criteria. A third reviewer was consulted for discrepancies in scoring. All other articles were coded by one examiner. During the review process, the researchers also tabulated other characteristics of the review articles, including number of intervention trials, the type of study designs included (e.g., only randomized controlled trials [RCTs], all group designs, single-participant research designs), type of psychosocial intervention (e.g., parent, school, cognitive behavioral, family), age group of the study participants, types of outcome measures employed (e.g., academic, behavioral, parent, and teacher rating scales), and study conclusions.
Results
Table 2 provides the characteristics of the reviewed studies and the quality scores for each of the 21 reviews included in this study. The number of intervention trials included in each article reviewed ranged from 2 (Bjornstad & Montgomery, 2010) to 116 (Fabiano et al., 2009). This large range is due, in part, to the inclusion criteria incorporated within each of the systematic reviews. Smaller numbers of studies were included in reviews that targeted only RCTs, which are viewed as the gold standard of intervention studies (Rubin, 2008). The RCT-only reviews varied from 2 (Bjornstad & Montegomery, 2010) to 26 studies (Klassen, Miller, Raina, Lee, & Olsen, 1999). Six of the reviews included single-participant treatment designs (SSDs) in the trials; Reid, Trout, and Schartz (2005) reviewed only SSDs for cognitive-behavioral ADHD interventions. When all treatment designs were included in a review, larger numbers of trials were identified. Two studies did not state inclusion criteria for research design.
Characteristics of Review Articles and Quality Scores, Ranked Chronologically.
Note: RCT = randomized controlled trial; SSD = single-participant treatment design. Grp = group; cog Beh= cognitive/behavioral; NS = not stated; Behav = behavioral Instead of institute for Clinical Systems Improvement, 2010, see Dobie et al., 2010.
Indicates meta-analysis.
Studies shown in boldface received top scores for the total and modified quality-rating scores.
Five of the reviews included trials of pharmacologic and psychosocial interventions, whereas all others focused only on some form of psychosocial intervention. A variety of interventions were subsumed under the psychosocial heading, including cognitive-behavioral interventions directly with children, parent training, teacher conducted behavior modification approaches, and family counseling.
Evaluations of the systematic reviews and meta-analyses showed that quality scores for the 21 studies ranged from 8 to 49 points, with a mean of 19.8 out of 54 possible points. Because a majority of the reviews were only systematic reviews, we calculated a modified score, excluding the criteria relevant to statistical analysis. The modified scores ranged from 8 to 35, with a mean of 16.1 of 38 possible points. Five studies, indicated in bold in Table 2, received the top scores for the total and modified quality-rating scores (Bjornstad & Montgomery, 2010; Fabiano et al., 2009; Purdie, Hattie, & Carroll, 2002; Schachar et al., 2002; Van der Oord, Prins, Oosterlaan, & Emmelkamp, 2008). The top two reviews included only RCTs (Bjornstad & Montgomery, 2010; Schachar et al., 2002). The highest quality meta-analysis was from the Cochrane Collaboration (Bjornstad & Montgomery, 2010), which examined RCT studies of the effectiveness of family therapy for children with ADHD without medication. Although high in methodological quality, Bjornstad and Montgomery (2010) reviewed only two studies.
Table 3 shows the number of reviews that achieved an adequately addressed score for the 27 quality criteria. Only 8 out of the 27 criteria were observed in most of the systematic reviews and meta-analyses. Those 8 criteria were largely focused on methods for identifying and describing the trials included in the review. On the other end of the spectrum, only four of the reviews described data extraction methods and five described interobserver agreement, 2 criteria important to verifying that bias was avoided in the review methods. Also of note is that only nine of the reviews reported a quality rating for the trials included. Several criteria important for avoiding biased conclusions were missing, as were those pertaining to statistical analyses, because the majority of the studies were systematic reviews and not meta-analyses.
Number of Studies Achieving an “Adequately Addressed” Score for Each Criterion.
Items pertaining to meta-analysis.
Discussion
Children with LDs often experience co-occurring ADHD. Intervention for those children will be complicated by the comorbidities and may require targeted intervention for the ADHD in order for instruction to be maximized for learning and social success. Evidence-based methods need to be implemented with these children.
Since 1998, many systematic reviews and meta-analyses have been conducted summarizing results of treatment studies for a variety of psychosocial treatments for children with ADHD. The quality of methodology used in those reviews has varied greatly over the years. There is a trend, however, toward increased rigor in the more recent reviews (e.g., Bjornstad & Montgomery, 2010; Fabiano et al., 2009). Perhaps this increased rigor is an outgrowth of the evidence-based practices movement influenced by legislation such as No Child Left Behind, which has led to the call for thoroughness and control for bias in systematic reviews (Schlosser, Wendt, & Sigafoos, 2007). Although the 21 review articles included in this study were generally good at describing and listing the studies included within their reviews, there were key features of the systematic review and meta-analysis process that were often omitted in the methodology, thereby reducing the overall quality of the reviews. For example, almost all reviews failed to report data extraction methods and interobserver agreement on this extraction process. These are key features that are important to assure the lack of bias in the review conclusions. Another important criterion included in only 9 of the 21 reviews was a trial quality assessment. Without trial quality assessment, it is difficult to balance the findings of the individual trials with the adequacy of the methods used to assure internal validity of the studies. Studies with positive findings need to be balanced with a consideration of the methodologic features incorporated to assure unbiased conclusions. Finally, more meta-analyses are needed that address the statistical analysis criteria (e.g., statistical methods, confidence intervals, statistical error), to provide quantitative measurements of the impact of psychosocial interventions for various behavioral and academic outcomes in children with ADHD.
Although the primary purpose of this study was to evaluate the quality of systematic reviews and meta-analyses conducted for psychosocial treatments of ADHD in children, we also wanted to identify effective research-based interventions for students with ADHD. However, findings from this study demonstrated patterns of methodological weaknesses that need to be addressed in future meta-analyses and systematic reviews to increase confidence in the conclusions of most of these reviews. Moreover, the complications posed by comorbid disabilities in children with ADHD are rarely systematically evaluated.
The types of interventions used in the 21 reviews varied from family intervention to behavior modification in schools and self-regulation methods with the children. Three of the five highest rated reviews were completed in recent years, when the methodology for conducting high-quality, rigorous, and unbiased systematic reviews had been developed more thoroughly. The highest rated review by Bjornstad and Montgomery (2010) received a score of 45 and found two RCTs that examined the effects of family intervention for children with ADHD. However, the fact that only a small number of trials were included in the review precluded any strong conclusions about the positive benefits reported for family intervention.
The second highest rated quality article (score = 33) was the systematic review of Schachar et al. (2002). They reviewed studies on pharmacological, cognitive-behavioral therapy, parent training, electroencephalogram (EEG) biofeedback combined pharmacological and behavioral interventions, and combined psychosocial treatments without nonpharmacological interventions. Although the reviewed studies provided intervention for a minimum of 12 weeks, they measured different outcomes and used 26 tests to measure those outcomes. They noted that pharmacologic plus cognitive-behavioral intervention had the best benefit for ADHD behaviors and academic and social outcomes. Among their conclusions were observations of the lack of high-quality methodological research across studies, inadequate description of interventions, small numbers of participants, and incomplete description of participants (e.g., missing gender, ethnicity), all of which make generalization of results difficult.
The third highest scored (32 points) review was the meta-analysis of Purdie et al. (2002). They examined 74 studies that used various types of interventions (e.g., pharmacological and behavioral interventions), provided effect sizes, and reported on five different outcomes. They concluded that pharmacological and multimodal behavioral treatments had the greatest effects on behavioral outcomes, and multimodal treatments have the most effect on social outcomes.
Implications for Practice
A large literature exists exploring a variety of behavioral interventions for children with ADHD. Multiple factors reflected in the literature suggested that psychostimulant medication along with behavioral strategies implemented in multiple educational and counseling settings are the most effective treatments for students with ADHD (Conners et al., 2001; DuPaul & Weyandt, 2006; Stroh, Frankenberger, Cornell-Swanson, Wood, & Pahl, 2008). Among psychosocial interventions, teacher’s use of classroom behavioral management methods has good effects for academic outcomes. Although positive effect sizes have been reported in the literature (e.g., Reid et al., 2005), little can be concluded about self-regulation interventions proposed for children with ADHD, possibly because self-regulatory behaviors take time to develop.
Our appraisal of systematic reviews and meta-analyses of behavioral interventions for children with ADHD clearly indicates that future reviews must improve the methods used to increase confidence in the conclusions of the reviews. At a minimum, review studies, whether systematic reviews or meta-analyses, need to describe their data extraction methods and interobserver agreement to increase confidence in the lack of bias for any conclusions drawn. Furthermore, reviews need to assess the quality of the intervention studies included, as any conclusions need to be balanced by the integrity of the studies that led to those conclusions. In addition, more well-conducted meta-analyses are needed. Furthermore, the interplay between comorbid LDs and ADHD in intervention outcomes needs to be explored.
The general conclusion that resonates across this series of reviews is that psychosocial treatments can have some positive impact for children with ADHD beyond the impact of pharmacologic treatment alone. Future systematic reviews must be conducted more rigorously to better guide educators who work with students with ADHD. Considering what is known about the nature of ADHD, that is, a neurobehavioral disorder concomitant with executive function deficits (Barkley, 2006; Douglas, 2005; Reddy et al., 2011), some researchers (e.g., Miranda, Jarque, & Tarraga, 2006; MTA Cooperative Group, 2004; Reeves & Anthony, 2009) suggest that a multimodal approach to ADHD that includes pharmacologic treatment along with psychosocial interventions seems to better address the multifaceted needs of children with ADHD.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.