Abstract
Attention-deficit/hyperactivity disorder (ADHD) is one of the most common neurodevelopmental conditions among school-age children. Early intervention and ongoing evaluation of treatment effectiveness are essential to minimize the life-long negative impact of ADHD. Neurocognitive functions have been reported to improve with pharmacological and cognitive training interventions for children with ADHD. We evaluated the value of measuring change in neurocognitive functions following ADHD interventions as a treatment outcome. We systematically reviewed randomized control trials of two distinctive types of ADHD interventions—pharmacological treatments and cognitive training—and summarized the changes in neurocognitive and clinical outcomes using a series of meta-analyses. Both pharmacological and cognitive training interventions showed positive effects on some aspects of neurocognitive functions. However, there were no significant correlations between changes in neurocognitive function (e.g., inhibition) and changes in ADHD behavioral symptoms (e.g., impulsive behavior). Although the associations between changes in neurocognitive function and clinical outcomes are not well studied, based on current findings, it is not suitable to use change in neurocognitive outcomes as a proxy for change in ADHD clinical symptom-based outcomes. There is, however, notable value in monitoring changes in neurocognitive function associated with ADHD interventions to achieve the following aims: (1) understanding full treatment effect on children with ADHD, (2) identifying ancillary indicators of subclinical changes, and (3) provision of objective and less biased measures of treatment effects. These findings are important evidence that changes in neurocognitive function could be a co-occurring objective indication that parallels the clinical effects of ADHD treatments.
Introduction
Attention-deficit/hyperactivity disorder (ADHD) is a highly prevalent neurodevelopmental condition among school-age children, affecting ∼7.2% of the childhood population (Thomas et al, 2015). Children with ADHD present with inattentive and/or hyperactive-impulsive behaviors across different settings (e.g., school and home) that impair their personal, social, and/or academic functioning (American Psychiatric Association, 2013).
Subsequently, children with ADHD are more vulnerable to lifelong difficulties, directly and indirectly as a result of their ADHD symptoms, including neurocognitive deficits (Coghill et al, 2014b), academic and occupational challenges, social isolation, risk-taking behaviors, and psychiatric comorbidities (Faraone et al, 2021; Faraone et al, 2015; Wehmeier et al, 2010). Accordingly, when treating ADHD, intervention outcomes should look beyond symptomatic improvement and instead focus on cognitive improvements, day-to-day functioning, and subjective quality of life (QoL) (Adamo et al, 2015; Mulraney and Coghill, 2018; National Guideline, 2018).
Although there is no known single major cause or cure for ADHD, early diagnosis, early intervention, and measurement-based care (i.e., clinical decision making based on treatment response data) are critical in providing optimal management and improving QoL (National Guideline, 2018). Pharmacological interventions are currently the most effective intervention based on ADHD symptom reduction (Cortese et al, 2018; Sonuga-Barke et al, 2013). Pharmacological interventions for ADHD include psychostimulants (e.g., methylphenidate [MPH] and amphetamines) and/or non-psychostimulants (e.g., atomoxetine [ATX]).
Both types of pharmacological agents intend to alter the effectiveness of neurotransmitters (e.g., dopamine or norepinephrine) that are hypothesized to be imbalanced in the brain in individuals with ADHD (Brown et al, 2018).
More recently, cognitive training has been proposed as an alternative or supplementary intervention for ADHD. Cognitive training involves repetitive performance on a set of game-like tasks that are intended to strengthen specific cognitive processing networks in the brain, such as attention, working memory, or inhibition (Cortese et al, 2015; Simons et al, 2016a; Simons et al, 2016b). The intervention is typically delivered via a computerized program with adaptive procedures (i.e., the difficulty of the training task(s) increases with session progression based on the individual's performance; Cortese et al, 2015; Farah et al, 2004).
The investigation of treatment effects of cognitive training to treat ADHD is still in its infancy, and findings are somewhat controversial (Spencer-Smith and Klingberg, 2016; Spencer-Smith and Klingberg, 2015; Spencer-Smith et al, 2020). Consequently, the clinical implications of cognitive training for ADHD are not yet established (Cortese et al, 2015).
Among the rigorously controlled trials to evaluate the efficacy of ADHD interventions, the treatment outcomes are commonly measured by changes in subjectively rated clinical outcomes such as severity of ADHD symptoms or impairment. This was evidenced in several systematic reviews and meta-analyses of the treatments of ADHD [e.g., cognitive training (Cortese et al, 2015) and pharmacological intervention (Cortese et al, 2018)], with the treatment effects predominantly measured by clinical outcomes, including ADHD symptoms and impairment rated by a parent, a teacher, or a clinician.
Some trials used more objective measures of impairment such as academic skills. QoL was not commonly used as a primary outcome in the trials included in the two meta-analyses mentioned earlier. However, QoL, either rated as a proxy by parents or directly by children with ADHD, has been increasingly recognised as an important and effective treatment outcome measure in recent years (Coghill et al, 2019; Danckaerts et al, 2010; Escobar et al, 2010; Mulraney and Coghill, 2018).
Unlike rigorously controlled research settings, in practice, clinicians have reported that measurement-based care is limited by a reliance on subjectively rated treatment outcomes (Hall et al, 2016b). Common issues include difficulties in obtaining teacher reports, low test–retest reliability on subjective symptom questionnaires, and variability in the interpretation of impairment and QoL among parents, patients, and teachers (Danckaerts et al, 2010; Hall et al, 2016b). To assist these challenges, valid, reliable, and practical ancillary measures that can objectively monitor treatment response are necessary to inform and enhance clinical judgment.
One potential area to address these issues is neurocognition. Neurocognitive function or neurocognition is a set of abilities to process information in the mind that reflects functions in an individual's specific brain pathways or networks (Sharafkhaneh and Grogan, 2015). Over the past two decades, ADHD researchers have identified differences in various neurocognitive domains that may distinguish groups of children and adolescents with ADHD from typically developing controls (Banaschewski et al, 2018; Coghill et al, 2014b; Willcutt et al, 2005).
In addition, some neurocognitive functions have been reported to improve with pharmacological interventions such as response inhibition (i.e., ability to suppress automatic response to stimuli; Coghill et al, 2014; Griffiths et al, 2018), working memory (i.e., ability to manipulate incoming information; Coghill et al, 2014), non-executive aspects of short-term memory (i.e., ability to mentally hold incoming information for a short-period; Coghill et al, 2014a; Coghill et al, 2014b), attention (i.e., ability to control focus on goal-directed tasks; Epstein et al, 2011), response reaction time (RT; i.e., how quickly react to stimuli; Coghill et al, 2014b), and reaction time variability (RTV; i.e., differences of response time across stimuli presented in the same task; Coghill et al, 2014b).
Cognitive training interventions for ADHD also reported positive treatment effect on working memory and short-term memory (Cortese et al, 2015; Rapport et al, 2013). These neurocognitive markers may have potential as an objective indicator of treatment response (Aman and Turbott, 1991). However, whether these changes in neurocognitive functions can be used as an objective indication of changes in clinical outcomes for ADHD has not yet been investigated.
Most of the previous investigation focused on associations between neurocognitive function and ADHD symptoms in the intention to assist the diagnostic process instead of relationships of a wider breadth of clinical outcomes during the treatment phases (Hall et al, 2016a; Tallberg et al, 2019; Willcutt et al, 2005). These studies reported weak or absent correlation between neurocognitive function and ADHD symptoms (Coghill et al, 2014a; Coghill et al, 2007; Hall et al, 2016a). However, the evidence for the relationships between changes in neurocognitive functions and changes in a wider breadth of clinical outcomes are inconclusive and under investigated (Coghill et al, 2014a; Cortese et al, 2015).
Pharmacological interventions have been shown to improve both clinical and neurocognitive outcomes across several domains (Cortese et al, 2018). Although differences in treatment effects have been observed depending on the outcome rater (e.g., parents or teachers), medication type (e.g., stimulants or non-stimulants), or medication dosage, overall trends of pharmacological interventions of ADHD demonstrate improvements in clinical outcomes such as reducing severity of ADHD symptoms, subjective impairment, and improving QoL for children (Cortese et al, 2015).
In addition, a meta-analysis of pharmacological studies (Coghill et al, 2014b) showed positive effects of MPH on working memory, short-term memory, RT, RTV, and inhibition in those treated with medication compared with placebo. These studies did not, however, assess whether there were associations between changes in neurocognitive functions and clinical symptomology outcomes of ADHD. Cognitive training interventions also have been reported improving both clinical and neurocognitive outcomes across several domains (Cortese et al, 2015).
A recent systematic review and meta-analyses of 15 individual cognitive training trials including 759 children with ADHD (Cortese et al, 2015) found statistically significant improvements in visual (standardized mean differences [SMD] = 0.47, 95% CI 0.23–0.70) and verbal (SMD = 0.52, 95% CI 0.24–0.80) working memory performances after cognitive training that targeted working memory abilities when rated by either blinded or unblinded individuals. They also found positive effects of cognitive training on ADHD inattentive symptoms and ADHD total symptoms scores when these were rated by unblinded (i.e., parents or teachers) but not from blinded raters.
Overall, some aspects of neurocognitive function improve in both pharmacological and cognitive training interventions for children with ADHD. Neurocognitive functions may not directly reflect ADHD symptom changes. It is, however, not yet clear whether there are associations among changes in neurocognitive functions and a wider breadth of ADHD clinical outcomes (i.e., impairment or QoL). Lack of, or small correlation between, objectively measured neurocognitive function and subjectively measured ADHD symptoms at a time point (usually after treatment effect) are a consistently reported phenomenon in ADHD diagnostic setting (Emser et al, 2018) and other health conditions (Song et al, 2015; van der Werf-Eldering et al, 2011).
This could be reflecting complex associations between performance-based neurocognition and behavior (Toplak et al, 2013), inter-individual differences (Kanai and Rees 2011), and/or psychometric differences (i.e., validity or reliability issues; van der Werf-Eldering et al, 2011). Despite this background understanding, there is a lack of comprehensive investigation into whether changes in two measures (i.e., the difference between two time points) are correlated. These relationships among the changes of ADHD treatment outcomes have never been systematically reviewed. Consequently, the clinical value of measuring neurocognitive functions has not been explored.
These issues merit a comprehensive systematic review to understand the potential relationships or differences of changes in neurocognitive functions and clinical outcomes post-ADHD interventions, and to evaluate the clinical potential to utilize neurocognitive outcomes as one of the indicators of ADHD treatment effects. To address this concern, we simultaneously investigated the pattern of changes of treatment outcomes in two types of ADHD interventions known to improve neurocognitive functions: pharmacological and cognitive training interventions. We hypothesized that if neurocognitive outcomes are associated with ADHD clinical outcomes, the pattern of changes would be the same or similar for both types of interventions.
To our knowledge, this is the first review that systematically summarizes and evaluates the relationships between changes in neurocognitive functions and clinical outcomes in the two distinctive ADHD interventions that are most likely to have an impact on cognition. The outcomes of this review will inform the value of measuring neurocognitive functions to track individualized responses to the ADHD interventions. We also aimed at assessing which neurocognitive functions and specific tasks measuring them are more likely to change with each type of intervention. Direct comparisons of treatment effects between pharmacological and cognitive training interventions, especially within the same trial, were not the scope of this review.
The objectives of the current systematic review were to: Systematically review the impact of two distinctive types of ADHD interventions (i.e., pharmacological, and cognitive training interventions) on neurocognitive and clinical outcomes. Meta-analyse each domain of treatment outcomes using the findings from eligible trials. Examine the associations between changes in neurocognitive and clinical outcomes. Consider the potential clinical implications of these findings with respect to the potential for neurocognitive functions to track treatment response in ADHD.
Methods
Inclusion and exclusion criteria
Types of studies
We included randomized controlled trials (RCTs) of pharmacological and cognitive training interventions for ADHD. Included trials had to measure at least one outcome from each of the neurocognitive (i.e., inhibition, attention, working memory, short-term memory, RT, or RTV) and clinical (i.e., ADHD symptoms, impairments, or QoL) domains. We included studies with either a between-subject parallel-controlled or within-subject crossover control design. We included placebo, active, waitlist, or training as usual controls. Due to the nature of cognitive training interventions, we included trials with single- or non-blinded designs as well as double-blinded designs.
Types of participants
We included trials of children and adolescents aged 6 to 17 years with a diagnosis of ADHD using Diagnostic and Statistical Manual of Mental Disorders or International Classification of Diseases criteria (any versions). We included any RCTs with a portion of their sample having common comorbid conditions with ADHD such as oppositional defiant disorder (ODD), conduct disorder (CD), anxiety disorder, sleep difficulties, mood disorder, specific learning disorder (SLD), or a tic disorder. If any trials had multiple subgroups with different comorbid conditions, we included data from the ADHD subgroup without a comorbid condition.
Exclusion criteria
We excluded any RCTs that specifically selected all participants with a comorbid condition (e.g., all participants had ADHD and autism spectrum disorder, or all participants had ADHD and SLD). We excluded RCTs that included participants with any other developmental, neurological disorder, or psychiatric disorder such as pervasive developmental disorder, intellectual disability, epilepsy, bipolar disorders, acquired brain injury, or Fragile X syndrome. We excluded any interventions, including the elements other than pharmacological treatment or cognitive training such as cognitive behavioral treatment, meditation, physical exercise, or Omega-3.
Meta-analyses
In our meta-analyses, we included trials reported means and standard deviations (SDs) for their pre- and post-treatment outcome measures.
Search methods of identification of studies
Electronic searches
We searched the following electronic databases in January 2021 and in March 2021. We did not limit the searches by date or language (Appendix A).
Cochrane Central Register for Controlled Trials (CENTRAL; Issue 3 of March 12, 2021) (searched the first week of January 2021 and March 19, 2021)
MEDLINE Ovid (1946 to March 18, 2021)
PubMed (1965 to March 18, 2021)
PsychINFO Ovid (1806 to second week of March 2021)
CINAHL EBSCO (1986 to March 18, 2021)
Scopus (1972 to March 18, 2021)
Searching other resources
We checked references in previous systematic reviews and the bibliographies of included and excluded studies when developing a protocol for the current review. The review protocol was published through PROSPERO on December 18, 2020 (Lee et al, 2020). Registration number is: CRD42020221432. Refer to Figure 1 for the hand search results.
PRISMA flow diagram.
Data collection and analysis
Identification of studies
Search results were managed via a web-based software Covidence. Two authors (S.L., T.R.H.) independently reviewed the retrieved titles and abstracts and determined inclusion of the trials in the current meta-analysis based on the inclusion and exclusion criteria. They then reviewed the full text of potentially relevant trials or trials for which more information was needed. Any disagreements were resolved by discussion or consultation with a third review author (D.R.C.) who acted as an arbiter. Reasons for excluding studies after a full-text review are presented in the PRISMA flow diagram (Fig. 1) (Moher et al, 2009).
Data extraction and management
At least two authors (S.L., V.P., and/or T.R.H.) extracted data from each included RCT. Extracted data included details regarding study design, participants (age, sample size, study location, comorbidity, diagnostic criteria, inclusion and exclusion criteria), intervention (treatment type, type of control group), and outcome measures (name of measurements, numeric data, data collection timepoints). The two authors resolved any disagreements by consensus, and any disagreements were resolved by discussion or consultation with a third review author (D.R.C.) who acted as an arbiter.
Assessment of risk of bias in included studies
At least two authors (S.L., T.R.H., and/or V.P.) used the Cochrane Risk of Bias Assessment Tool (Higgins et al, 2019) to independently assess the quality of included RCTs. The level of risk of bias was categorized as either “low,” “high,” or “unclear,” and it was considered based on six general domains and one treatment specific domain, including (1) random sequence generation, (2) allocation concealment, (3) blinding of participants and personnel, (4) blinding of the outcome, (5) incomplete outcome data, (6) selective reporting, and (7) treatment-specific risk of bias (pharmacological treatment: compliance with medication, order effect, and carry-over effect; cognitive training: integrity of intervention). Any disagreements were resolved by discussion or consultation with a third review author (D.R.C.) who acted as an arbiter.
Measures of treatment response
We considered four aspects of change to evaluate the potential of change in ADHD treatment outcome measures: (1) differences in post-treatment outcomes between treatment and control conditions, (2) differences in within-group outcomes between pre-treatment and post-treatment, and (3) relationships between neurocognitive and clinical outcomes.
Continuous data
For the systematic summaries, we assessed differences in post-treatment outcomes between treatment and control conditions using effect sizes and p-values. For the parallel control design trials, we also analyzed within-group changes between pre-treatment and post-treatment to evaluate placebo effect. We interpreted results as either an “improvement” (i.e., positive effect on a child, such as a decline in ADHD symptom severity or improvement in working memory), “no change” (i.e., no statistically significant difference between treatment and control conditions), or “worsening” (i.e., negative effect on a child, such as increase in ADHD impairment or worsening of neurocognitive function).
For the meta-analyses, we analyzed SMD for all eligible variables using random-effects models. For the parallel (between-group) comparisons, we calculated post-treatment differences between treatment and control groups using means (M), SDs and sample size. For the crossover (within-group) comparisons, we used differences between treatment and control conditions using standard SMD and standard error (SE).
Associations between changes in neurocognitive and clinical outcomes
We narratively summarized all reported independent correlations of changes in neurocognitive and clinical outcomes.
Characteristics of outcomes
We summarized methods of outcome measures from each trial to qualitatively evaluate differences between the neurocognitive and clinical outcomes.
Unit of analysis issues for meta-analysis: heterogeneity of study design
The two types of study design (parallel and crossover) included in this review differ intrinsically in research design, duration of the treatment, form of delivery, control method, underline change mechanism, sample size, outcome measures, and interval between pre- and post-treatment. To address this heterogeneity, we analyzed the trials separately both by treatment type (pharmacological treatment and cognitive training) and then, within each treatment condition we grouped the trials by the two types of study designs to perform meta-analyses.
Parallel control design
All included cognitive training trials employed a parallel control design in which the outcome data were measured independently in the two arms. There were four parallel design pharmacological treatment trials.
Cross-over design
Most of the pharmacological treatment trials were crossover placebo-controlled trials, which may include paired data for each subject. We evaluated whether each crossover trial had a risk of carry-over or order effects and excluded trials with a high risk of bias from the meta-analysis. We followed the Cochrane Handbook for Systematic Reviews of Interventions—approximate analyses of crossover trials for a meta-analysis, to control unit-of-analysis error (Higgins et al, 2021). We used SMD and SE of the SMD and input the data as the generic inverse-variance outcome model in RevMan.
Dealing with missing data
We documented details of missing data (e.g., type, number) in the “Risk of Bias” table under “Incomplete outcome data” (see section 3. Characteristics of included studies in the Supplementary Data). We used data with low risk of attrition bias. We did not attempt to impute missing data.
Assessment of heterogeneity
We assumed each trial would introduce its own heterogeneity into the meta-analysis. We incorporated differences in sample demographics across studies using a random effect model in analyses and reported heterogeneity measures Tau 2, X 2, and I 2.
Assessment of reporting biases
Funnel plots were examined to examine publication bias.
Data synthesis
Trends of changes
We qualitatively summarized results from all included trials. We indicated the direction or statistical significance of the changes using three different symbols (i.e., ⇑: improvement post-treatment, ⇔: no change post-treatment, ∩: inverse U shape changes). For the within-subject comparison trials, we used post-treatment differences between treatment and control groups.
For the between-subject comparison trials, we independently indicated differences between pre- and post-treatment from treatment and control groups. We reported direct pre- and post-treatment comparisons between treatment and control groups when this was available. Effect sizes reported by each study indicated the magnitude of the changes, and the specific measures used were reported. We computed effect sizes using SMD for trials that did not report effect size but reported adequate raw data (means and SDs).
We used Cohen's approaches (Schäfer and Schwarz 2019) as a guideline to determine small (i.e., equal, or above d = 0.2, r = 0.1 or η 2 = 0.01), medium (i.e., equal or above d = 0.5, r = 0.3 or η 2 = 0.06), and large (i.e., equal or above d = 0.8, r = 0.5 or η 2 = 0.14) effect sizes. When a trial contained multiple control or treatment conditions, we reported comparisons only from the included treatment and control groups and indicated which subgroups were included in the review.
Meta-analyses
Multiple meta-analyses were conducted for each outcome, including inhibition, attention, working memory, short-term memory, RT, RTV, ADHD symptoms, behavioral impairment, functional impairment, and QoL.
Subgroup and sensitivity analyses and investigation of heterogeneity
For neurocognitive outcomes, we performed two subgroup analyses based on method measurement (i.e., traditional pencil and paper and computerized tasks). For clinical outcomes, we performed subgroup analyses based on the rater, that is, parent, teacher, or clinician. Differences among the subgroups were investigated using p-value and heterogeneity measures (i.e., Tau2, X 2, and I 2). We did not include trials or variables with a high risk of selection bias or attrition bias. For pharmacological treatment trials, we did not perform sensitivity analyses as we did not include trials with a high risk of bias.
For cognitive training trials, we performed sensitivity analyses to determine the effect of removing from the analysis: trials with a high risk of performance and outcome detection bias associated with blinding issues. For example, trials systematically differ from the control group, which cannot guarantee that the outcome raters or participants were free from the knowledge of which intervention they were exposed to.
Results
Description of included studies
Results of the searches
We retrieved 8882 records (4253 unique records) from the six databases (not limit the searches by date or language, latest update in March 2021) and a review of previous meta-analysis in the field. After removing duplicates, we screened 4121 records that did not meet inclusion criteria and identified 132 records that were eligible for full-text review (Fig. 1). We excluded 94 records at full-text review and identified 38 eligible RCTs (19 cognitive training, 18 pharmacological treatments, one trial with both conditions). Of these 38 trials, 23 were included in meta-analyses. Six trials were included in analyses examining correlations between cognitive and clinical outcomes. Reasons for exclusion are summarized in the Reasons for Excluded Studies from the Full-Text Review table in the Supplementary Data.
Characteristics of included studies
The characteristics of included RCTs as well as the interventions evaluated are described in Table 1. Full details of each included trial are reported in section 3. Characteristics of included studies in the Supplementary Data.
Characteristic of Trials Included in the Review
ATX, atomoxetine; DEX, dextroamphetamine; EF, executive function, MPH, methylphenidate; OROS MPH, osmotic-release oral system methylphenidate; WM, working memory.
Participants
All participants in the included trials were clinically diagnosed with ADHD. A total of 2054 children (1075 in pharmacological treatment; 979 in cognitive training) aged between 6 and 17 years old were included. Sample sizes of each trial ranged from 26 to 125.
Interventions
Four different types of ADHD pharmacological treatments were evaluated, including immediate release MPH, osmotic-release oral system methylphenidate (OROS MPH), dextroamphetamine (DEX), and ATX. Cognitive training trials evaluated interventions that intended to improve working memory, attention, executive functions, and multiple cognitive domains.
Neurocognitive outcome measures
We summarized the number of trials reporting on each neurocognitive domain and the measures in Table 2. Inhibition and attention were the most common cognitive domains used as an outcome measure to evaluate both types of ADHD treatments. Short-term memory and working memory were more commonly measured in cognitive training trials, as the interventions primarily targeted to enhance the same cognitive domains. Some trials reported short-term memory results in the working memory domain (Green et al, 2012; Hovik et al, 2013; Johnstone et al, 2012; Klingberg et al, 2005; Tamm et al, 2013).
Characteristics of Neuropsychological Test Used
ASS, attentional set-shifting task; AWMA, Automatic Working Memory Assessment; CANTAB, Cambridge Neuropsychological Test Automated Battery; CBT-B, Corsi block tapping backward span; CBTT, Corsi block tapping task; CCT, children's checking task; CDT, choice discrimination task; CPT, continuous performance test; DKEFS, Delis-Kaplan Executive Function System; DSB, digit span backward; DM, dot matrix; DMtS, delayed matching to sample; DR, digit recall; DS, digit span, DSF, digit span forward; FK, flanker test; FWB, finger window backward; FWF, finger window forward; LNS, letter number sequencing; LoB, Locations of Blocks; LST, List Sorting Working Memory Test; MFFT, Matching Familiar Figures Test; OST, Operational Span Test; RD, reading disorder; RT, reaction time; RTV, reaction time variability; RVP, rapid visual information processing; STM, short term memory; SWM, spatial working memory; TEAC, Test of Everyday Attention for Children; TM, traditional measures; TMT, Trail Making Test; TOVA, Test of variables of attention; VWM, visuospatial working memory; WISC, The Wechsler Intelligence Scale for Children.
We reported short-term memory results separately from working memory when it was possible. RT or RTV predominantly were reported from the computerized tests, which were often incorporated into an attentional domain. We counted trials that reported RT or RTV separately.
Clinical outcomes
We summarized the number of trials reporting on each clinical outcome and the measures in Table 3. Subjective ADHD symptoms and impairment questionnaires were the most used clinical outcome measures across both types of ADHD treatment trials. The questionnaires mainly were rated by the participant's parents. Teacher's rating was used as a secondary measure on some trials. Clinician-rated ADHD symptom or impairment severity was less popular but used in some included trials (Chacko et al, 2014; Coghill et al, 2007; Green et al, 2012; Solanto et al, 2009; Tamm et al, 2013). Only two trials (Dovis et al, 2015; van der Donk et al, 2015) measured QoL. In cognitive training trials, researchers used objective tests or subjective questionnaires to measure academic skills as one of their outcome measures.
Characteristics of Clinical Outcome Measures Used
ACTeRS, Attention-Deficit Disorder with Hyperactivity Comprehensive teacher's Rating Scale; ADHD, attention-deficit/hyperactivity disorder; ADHD-Q, ADHD questionnaire; ADHD-RS, ADHD rating scale; CPRS, Conners' Parent Rating Scale; CTRS, Conners Teacher Rating Scale; PACS, parental account of childhood symptoms; PEDsQL, Pediatric Quality of Life Inventory; PERMP, Permanent Product Measure of Performance; QoL, quality of life; WRAT, Wide Range Achievement Test.
Excluded studies
The primary reasons for exclusion were ineligible: (1) outcome measures, (2) study design (e.g., not RCT, ineligible control condition), (3) participants (e.g., preschool age or 18 years of age and older), (4) intervention (e.g., supplementary foods or vitamins trials, including aspects of behavioral intervention), and (5) quality of the trial (e.g., high risk of bias in multiple domains, protocol papers, not peer-reviewed papers). The reasons for exclusion in the meta-analyses were not providing means and SDs and parallel control design in pharmacological trials (see Excluded Studies Table in Supplementary Data).
Risk of bias in included studies
Out of the total 38 RCTs in qualitative summary, we included 12 RCTs with a high risk of performance or/and detection bias(es) due to their nature of blinding their control condition (i.e., waitlist). Of those 12 RCTs with a high risk of blinding, 2 RCTs were pharmacological interventions. Among the 23 RCTs included in the meta-analyses, we included data from 9 cognitive training RCTs with a high risk of either performance or detection bias(es) and one pharmacological intervention with a high risk of performance bias. Due to the considerable number of cognitive training RCTs with a high risk of bias, we also reported adjusted results for all cognitive training variables after accounting for a high risk of bias (Figs. 2 and 3). Please see the full detail of the risk of bias assessment results in the Supplementary Data.
Summary of meta-analyses of neurocognitive outcomes by ADHD treatments. Effect sizes = standard mean difference between intervention and control groups. *p < 0.05. ADHD, attention-deficit/hyperactivity disorder; NA, not available due to lack of raw data.
Summary of meta-analyses of clinical outcomes by ADHD treatments. Effect sizes = standard mean difference between intervention and control groups. *p < 0.05.
Qualitative synthesis of treatment outcomes
Trends of outcome changes in pharmacological interventions
As indicated in Table 4, the pattern of changes in neurocognitive and clinical treatment outcomes showed the same positive direction. Almost all trials (18 out of 19) reported statistically significant improvement in neurocognitive and clinical outcomes. However, it is important to acknowledge that only a handful of trials measured a comprehensive list of neurocognitive outcomes (e.g., Coghill et al., 2007; Hawk Jr et al., 2018; Wigal et al., 2011; Zeiner et al., 1999). The rest of trials mainly measured inhibition and attention that are traditionally assumed underlying cognitive weakness in children with ADHD. Clinical outcomes were mostly measured by subjective ADHD symptoms and impairment severity. No trials measured treatment outcomes using objective impairment (e.g., academic skills) or QoL.
Summary of Trend and Pattern for Pharmacological Treatment Outcomes
∩, a quadratic relationship with medication dosages/inverse U shape improvement; ⇑, statistically significant improvement post-treatment, a linear relationship with medication dosages; ⇓, statistically significant reduction after treatment, a reverse linear relationship with treatments; ⇔, No changes post-treatment; —, not assessed.
ADD, attention deficit disorder; ADD+H, attention deficit disorder with hyperactivity; ADD-H, attention deficit disorder without hyperactivity; ADD-RS, attention deficit disorder rating scale; C, clinician or researcher; CANT, Child Attentional Network Task; CAP, child attention problems; CBT-F, Corsi block tapping forward span; CCI, child conflict index; CGI-I, clinical global impressions-improvement scale; CGI-P, Conners' global index-parent; CGI-S, clinical global impressions severity scale; CGI-T, Conners' global index-teachers; CH, child; CPRS-AX, Conners' Parent Rating Scale Anxious-Shy; CPT, continuous performance test; CTRS-10, Conners 10-item hyperkinesis index; D, imputed standard mean difference d = Cohen's d; CV, coefficient of variation; FBB-HKS, the German ADHD Rating scale; g, Hedges effect size; GDS, Gordon Diagnostic Symptom Continuous Performance Test; GNG, GoNoGo; GS, good sleep group; HC, healthy control; HSQ, Home Situations Questionnaire; HY, hyperactivity; INH, inhibition; INT, intervention group; IOWA, Inattention/Overactivity with Aggression; KMFFT, Kagan matching familiar figures test; MP, math productivity; P, parent;; PASAT-A, paced auditory serial addition task A version; PASAT-R, paced auditory serial addition task R version; PPI, prepulse inhibition; PS, processing speed; QbTest, Qb tech infrared motion tracking device; r, correlation coefficient; RCT, randomized controlled trial; SDs, standard deviations; SKAMP, Swanson, Kotkin, Agler, M-F1ynn and Pelham Scale; SS, spatial span; SSQ, School Situations Questionnaire; SSRT, Stop-Signal Reaction Time; SST, stop signal task; T, teacher; TOVA, Test of variables of attention; VARS, Vanderbilt ADHD Teacher Rating Scale; VCT, Virtual Classroom Task Assessment; WREMB-R, Weekly Ratings of Morning and Evening Behavior-Revised-Investigator Rated.
Response to medication dosage
Findings from the qualitative summary in pharmacological treatments indicated differences in response to medication dose between neurocognitive and ADHD symptoms. There was a linear relationship between ADHD medication dosage and ADHD symptoms, where a higher dosage of medication was associated with better improvement in clinical outcomes. Interestingly, changes in neurocognitive outcomes may not have a linear relationship with medication dosage. Neurocognitive function was less optimal in the high dosage of ADHD medication compared with the medium dosage (Barkley et al, 1991; Coghill et al, 2007; Konrad et al, 2005; Solanto et al, 2009).
Trends of outcome changes in cognitive training
Improvement in both neurocognitive and clinical outcomes was reported in 10 out of 20 trials (see Table 5). Six trials reported improvement in at least one aspect of neurocognitive function, whereas no change occurred in clinical outcomes. Three trials reported no change in both neurocognitive and clinical outcomes. In trials that reported no difference between treatment and control groups, one trial (van der Donk et al, 2015) reported reduced performance in automated math tasks whereas short-term memory was improved post-treatment.
Summary of Trend and Pattern for Cognitive Training outcomes
a: A comparison between intervention and control groups. b: A pre- and post-treatment comparison within INT. c: A pre- and post-treatment comparison within control group.
Reported in the QoL column as children's own reflection of their ability to focus, however this is not a conventional measure of QoL.
∩ = a quadratic relationship with medication dosages/inverse U shape improvement, ⇑ = improvement post-treatment, a linear relationship with medication dosages, ⇓ = reduction after treatment, a reverse linear relationship with medication dosages, ⇔ = No changes post-treatment.
AC, active control; ARS, Attention-Deficit/Hyperactivity Disorder-Rating Scale; AST, Attention Switching Task; ATTC, Attentional Control Scale, BASC, Behavioral Assessment System for Children; BRIEF, Behavior Rating Inventory of Executive Function; BRS, Behavioral Rating Scale; CBCL, Child Behavior Checklist; CBT, cognitive behavioral treatment; CGAS, Children's Global Assessment Scale; CGI, Clinical Global Impressions Rating Scale; COC, cognitive measure by computer; COT, cognition measured by traditional methods; CPRS-R, Conners' Parent Rating Scale; CPT-II, Conners' Continuous Performance Test-version 2; CWI, color-word interference; DBD, Disruptive Behavior Disorder; DBDRS, Disruptive Behavior Disorder Rating Scale; DSM, Diagnostic and Statistical Manual of Mental Disorders; GNG, GoNoGo; IN, inattention; LOGOS, computerized reading ability test; NEPSY, A Developmental Neuropsychological Assessment; NLD, Netherlands; PS, processing speed; RAST, Restricted Academic Situations Task; RTI, reaction time in a inhibition test; SBB, span board backwards; SBF, span board forward; SC, standardized change; SCWT, Stroop Color and Word Test; SNAP, Swanson, Nolan, and Pelham; TAU, training as usual; TW, DKEFS Tower; WDQ: word decoding quality; WDS: word decoding speed; WJ-III, Woodcock Johnson III; WMI, Working Memory Index.
Controlling for concomitant medication effect
There were no qualitative differences in the pattern of outcome changes between trials that fully controlled or partially controlled for ADHD medications. Fully controlled for medication trials included all participants who used pharmacological interventions (Ackermann et al, 2018; de Oliveira Rosa et al, 2021; Nejati, 2020), and three other trials (Azami et al, 2016; Bigorra et al, 2015; Klingberg et al, 2005) reported that all participants were medication naive or were not on medication during the cognitive training trial period.
The rest of the 14 trials included a portion of participants prescribed ADHD medications during the intervention period, and we defined these trials as partially controlled for medication.
Parent-rated clinical outcomes improved in both treatment and control conditions
Parent-rated clinical outcomes were less likely an effective measure for treatment effect. Most cognitive training trials (16 out of 20) reported significant improvement in parent-rated clinical outcomes compared with within-subject pre- and post-treatment outcomes. However, 8 out of these 16 trials reported no significant difference between treatment and control groups.
Factors might impact the treatment effects
Although not a primary objective of this review, we identified possible variables to affect treatment effects, such as sleep quality (Gruber et al, 2007), anxiety (Griffiths et al, 2018), a subtype of ADHD (Barkley et al, 1991), or quality of cognitive training coach/program (Bioulac et al, 2020). In the interest of consistency across the RCTs among the cognitive training treatments, we have excluded cognitive training programs, including social/behavioral aspects (de Oliveira Rosa et al, 2021; Kermani et al, 2016; Qian et al, 2017), and top-up cognitive training (Qian et al, 2021; Qian et al, 2017).
Meta-analyses: changes in the outcomes post-ADHD treatments
Changes in neurocognitive outcomes
The results of meta-analyses examining neurocognitive outcomes are summarized in Figure 2. The results indicate small to medium positive effects (p < 0.05) of pharmacological treatments and cognitive training in inhibition, working memory, and short-term memory. Attention and RTV improved with pharmacological treatments only. The RT showed a trend toward improvement with pharmacological treatments (p = 0.05) but not cognitive training. Cognitive training did not improve inhibition when we excluded trials with a high risk of blinding issues (performance and detection bias).
However, the effects of cognitive training on working memory (small effect) and short-term memory (medium effect) remained positive after removing the high risk of bias trials. In the interest of cognitive training outcomes, working memory and short-term memory results are from the tests that participants were not directly trained during the intervention.
Changes in clinical outcomes
The results of meta-analyses examining clinical outcomes are summarized in Figure 3. Pharmacological treatments showed medium positive effects on all subtypes of ADHD symptoms and subjectively rated impairment. There were too few pharmacological treatments trials that measured impairment objectively to conduct a meta-analysis. Cognitive training showed, regardless of the risk of blinding (i.e., performance and detection biases), no difference in any clinical outcome compared with controls. We could not conduct a meta-analysis on QoL for either treatment type due to lack of data.
Subgroup analyses: types of neurocognitive measures
There was no statistical difference between traditional and computerized neurocognitive measures in all four neurocognitive domains (i.e., inhibition, attention, working memory, and short-term memory). Table 6 summarized effect sizes of these subgroup analyses; only for inhibition and working memory effect size was slightly greater in computerized continuous performance tests (CPTs) than a traditional pencil and paper task. The RT and RTV were mostly measure by computerized CPT, and there was an inadequate number of trials measuring these domains with traditional measures (refer to section 2. Meta-analysis in the Supplementary Data).
Summary of Effect Size Subgroup Analysis for Neurocognitive Outcomes
p < 0.05.
CT, cognitive training; NA, not available due to lack of raw data; PI, pharmacological interventions; PPT, pencil and paper tasks; SMD, standardized mean differences.
Subgroup analyses: subjectively measured clinical outcomes by rater
We conducted three subgroup analyses based on the outcome raters, which were parent, teacher, and clinician/researcher (Table 7). For pharmacological trials, no significant level of subgroup heterogeneity was detected in ADHD symptom total (I 2 = 0%, X 2 = 0.65, p = 0.72), ADHD symptom inattention (I 2 = 0%, X 2 = 0, p = 0.97), ADHD symptom hyperactivity (I 2 = 17.6%, X 2 = 1.21, p = 0.27), and subjective impairment (I 2 = 57.8%, X 2 = 4.73, p = 0.09). For cognitive training, ADHD symptom total was mostly rated by parents, and we could not conduct the subgroup analysis due to a lack of teachers' rated data.
Summary of Effect Size Subgroup Analysis for Clinical Outcomes
p < 0.05.
Meanwhile, ADHD symptom inattention (I 2 = 91.3%, X 2 = 23.01, p < 0.0001) showed substantial differences between parent-, teacher-, and clinician-rated subgroups. The ADHD symptom hyperactivity (I 2 = 47%, X 2 = 3.78, p = 0.15) showed no significant difference among the three subgroups. In both outcomes, parent rating showed a trend toward favoring treatment over control conditions, whereas teacher ratings showed no difference between treatment and control conditions. Similar subgroup heterogeneity was observed in the subjective impairment outcomes.
There were substantial differences by outcomes rated by parent, teacher, or clinician/researcher (I 2 = 85.6%, X 2 = 13.94, p = 0.0009). Clinician- and parent-rated impairment level favored treatment over control conditions, whereas teacher-rated impairment showed no differences.
Correlations of changes between neurocognitive and clinical outcomes
We could not conduct meta-analyses for corrections between changes of neurocognitive functions and clinical outcomes post-ADHD treatments due to the lack of adequate raw data reported by researchers. Therefore, we qualitatively synthesized the results from the correlation studies. There were six pharmacological trials investigating correlations between changes in neurocognitive function and clinical outcome (Table 8). For cognitive training trials, no eligible trials directly investigated correlations between neurocognitive and clinical outcomes.
Correlation Coefficient (r) Summary
p < 0.05.
Correlations between changes in neurocognitive functions and ADHD symptoms
Konrad et al (2004) reported nonsignificant correlation between changes in inattention and inhibition and teacher-rated ADHD symptom severity. Coghill et al (2007) reported significant regression (r 2 = 0.17) between Go-NoGo Block 2 commission errors and small dosage MPH (i.e., 3 mg/kg) responders. They also reported no significant correlation between neurocognitive outcomes and high dosage MPH (i.e., 6 mg/kg) responders. Blum et al (2011) reported nonsignificant correlation (r = −0.24 to 0.32) between changes in neurocognitive function and changes in severity of parent- and teacher-rated ADHD symptoms. Wehmeier et al (2012) also reported no significant correlation between changes in QbTest scores and ADHD-Rating Scale scores.
Correlations between changes in neurocognitive function and impairment
One pharmacological intervention trial (Froehlich et al, 2014) reported statistically significant positive correlations (r = 0.22–0.47, p < 0.05) between neurocognitive improvement and math productivity (i.e., number of math problems attempted regardless of whether the answer was correct). None of the included cognitive training trials reported on correlations between change in neurocognitive function and clinical outcomes. Some trials investigated improvement in treatment targeted neurocognition (i.e., near transfer effect) and improved functionally related impairment (i.e., far transfer effect, e.g., academic skills, behavioral working memory difficulties).
Positive near transfer effects of cognitive training were reported by most of the RCTs (Bigorra et al, 2015; Bioulac et al, 2020; Chacko et al, 2014; Egeland et al, 2013; Green et al, 2012; Johnstone et al, 2012; Johnstone et al, 2010; Kermani et al, 2016; Steeger et al, 2016; Tamm et al, 2013; van der Donk et al, 2015). However, none of these RCTs found positive far transfer effects (i.e., improved neurocognitive associated with changes in participants' impairment). In addition, Tamm et al (2013) reported improvement in a child's subjective rating of ability to focus and shift their attention post-attention targeted cognitive training, which was measured by Attentional Control Scale.
Correlations between changes in neurocognitive function and QoL
QoL was measured in two trials (Dovis et al, 2015; van der Donk et al, 2015) using the Kidscreen-27 (i.d., a questionnaire to measure QoL, parent- and child-rated) and Pediatric Quality of Life Inventory (PedsQL; parent- and child-rated). There were no changes in parent- and child-rated QoL measured by PedsQL and Kidscreen-27.
Discussion
Summary of main results
Findings from the current systematic review and meta-analyses demonstrate positive neurocognitive effects of pharmacological interventions (improvements in inhibition, attention, working memory, short-term memory, and RTV) and cognitive training (improvements in inhibition, working memory, and short-term memory). Pharmacological interventions improve performance across a broad range of neurocognitive functions, whereas cognitive training interventions selectively improve specific neurocognitive domains.
For example, treatment effect sizes (i.e., SMD) were greater for cognitive training in the domains of working memory and short-term memory than pharmacological interventions. However, these results need to be interpreted carefully as not all cognitive training RCTs fully controlled for ADHD medication effects, the possible practice effect of cognitive training, or other psychological support factors from the personnel delivering the cognitive training (Kable et al, 2017). Regarding the clinical outcomes, pharmacological interventions reduced the severity of ADHD symptoms (I 2 = 0%, X 2 = 0.65, p > 0.05) and subjective impairment (I 2 = 57.8%, X 2 = 4.73, p > 0.05), with no obvious differences between parent-, teacher-, and clinician-rated outcomes.
The direction of change indicated positive treatment effects in both neurocognitive and clinical outcomes in pharmacological trials. However, no significant correlations were identified in studies that investigated associations between change in symptoms and change in neurocognitive functions. Interestingly, clinical and neurocognitive outcomes differed not only by individual differences but also by the pattern of changes in the different dosages of the medications. Higher doses of the medications were associated with better improvement in clinical outcomes. However, neurocognitive outcomes were less optimal at a higher dosage than a medium dosage. For cognitive training, parent-rated ADHD symptoms (SMD = 0.65, 95% CI [0.890.40], p < 0.05) and subjective impairment (SMD = 0.70, 95% CI [0.810.59], p < 0.05) improved when compared with within-group pre- and post-treatment outcomes.
When we compared the differences in post-treatment outcomes between treatment and control groups, no single domain of clinical outcomes was significant (refer to Fig. 3). These results did not differ when we controlled for the risk of blinding issues (i.e., trials with a higher risk of performance and detection biases due to use of waitlist as their controls) in cognitive training trials. The changes in clinical outcomes in the treatment group did not statistically differ from those reported from the controls. No trials for either treatment approach directly investigated the associations between changes in neurocognitive functions and subjective impairment.
Measures of objective impairment (e.g., academic achievement) or QoL were not commonly included as outcome measures in both pharmacological and cognitive training interventions. Consequently, we were not able to investigate correlations quantitatively. Only one trial (Froehlich et al, 2014) reported positive correlation between improved neurocognitive function and objective impairment (i.e., improvement in math productivity). Another trial (Tamm et al, 2013) reported positive correlations between improved neurocognitive function and self-efficacy (i.e., subjective confidence level of attention control). This could be directly or indirectly (e.g., test anxiety or practice effect) related to cognitive training treatment effect (Bloemsma et al, 2013; Heilbronner et al, 2010).
Overall completeness and applicability of evidence
Overall, for both interventions, an adequate amount of data (i.e., more than three raw data for each specified variable; Higgins et al, 2021) reported for inhibition, attention, working memory, short-term memory, ADHD symptoms, and subjective impairment to conduct meta-analyses.
There was a possible carry-over effect among a small number of within-subject crossover pharmacological trials, which could reduce the effect size between placebo and medication conditions. Although most of the included cognitive training RCTs were reportedly delivered in a standardized manner, extraneous factors were not well considered or controlled, including test anxiety, mood, locus of control, or variability in personnel delivering the training program (Bloemsma et al, 2013; Heilbronner et al, 2010).
Agreements and disagreements with other studies and reviews
No previous systematic review or meta-analysis has covered the full scope of the current review that explored the relationships of ADHD treatment outcomes. Several previous meta-analyses investigated treatment effects for either pharmacological or cognitive training interventions for individuals with ADHD (Cortese et al, 2018; Sonuga-Barke et al, 2013). A previous systematic review and meta-analysis of the effect of MPH on neurocognitive functions (Coghill et al, 2014c) similarly reported significant improvement in RT, RTV, response inhibition, and short-term memory for individuals with ADHD.
Similar to current findings, Cortese et al (2018) investigated the treatment effect of pharmacological interventions on clinical outcomes and reported improvement in ADHD symptoms rated by clinicians and teachers. Specifically, clinician-rated symptoms improved with all included medications, whereas teacher-rated symptoms only improved after MPH or modafinil. Two earlier meta-analyses (Cortese et al, 2015; Rapport et al, 2013) investigated treatment effects for cognitive training in individuals with ADHD. Cortese et al (2015) reported significant improvement in only working memory and parent-rated executive function (i.e., subjective impairments).
However, they found no effect on ADHD symptoms rated by either blinded or most proximal (i.e., probably unblinded) raters. When measured by non-blinded parents (not bias controlled), ADHD symptoms improved with cognitive training. Plausible explanations for this discrepancy between blind and non-blinded parents' ratings would be the reflections of parents' anticipation of the treatments or appreciation of their child spending time training. Cortese et al (2015) reported no improvement in academic performance after cognitive training interventions. QoL was not included as one of the treatment outcomes in the aforementioned systematic reviews and meta-analyses.
We found significant changes (SMD = 0.160.55, p < 0.05) after ADHD pharmacological interventions in inhibition, attention, working memory, RTV, and short-term memory. Changes in RT were marginal (SMD = 0.15, p = 0.05). Our findings differ slightly from Coghill et al (2014a), who found significant changes for all cognitive measures. However, these differences may reflect our smaller sample size as we included only trials that measured both clinical and cognitive outcomes, and fewer studies that measured RT. Similar to Cortese et al (2018), we included trials evaluating MPH, ATX, and DEX, and all of these trials showed improvement in ADHD symptoms and behavioral impairment.
There have been several trials published subsequent to (de Oliveira Rosa et al, 2021; Jones et al, 2020; Nejati, 2020) the review of cognitive training by Cortese et al (2015). In addition to working memory improvements after cognitive training reported via Cortese et al (2015), we found significant improvements in inhibition and short-term memory. This finding is interesting given that none of the included trials used cognitive training interventions that directly targeted these two neurocognitive domains, suggesting the far transfer effect (i.e., learned one domain of neurocognitive function influences to improve another neurocognitive domain that is not directly taught) of the cognitive training interventions.
However, consistent with Cortese et al (2015), we found no evidence for improvement in academic achievement or functional impairment, in line with findings of Rapport et al (2013). We identified only three trials measuring QoL, in highlighting the importance of measuring QoL and other functional outcomes and the current lack of data in this respect. The potential associations between improvement in neurocognitive functions and QoL remain unexplored.
Authors' Conclusions
Findings from this systematic review and meta-analyses indicate that both pharmacological and cognitive training interventions can improve neurocognitive functions in children with ADHD. However, changes in neurocognitive function cannot be assumed to be an indicator of changes in clinical outcomes. The review findings support previous observations of a dissociation between changes in neurocognitive function and subjectively measured ADHD symptoms. Unfortunately, there is not enough evidence to conceptualize clinical relevance of improved neurocognitive functioning in terms of improvements in either subjective or objective impairment or QoL.
Despite this lack of evidence, we believe that changes in neurocognitive function should be measured and that they should be interpreted independently from symptomatic outcomes. The impact that these treatment-related improvements in neurocognitive functions have on the daily life of children with ADHD is not yet understood. This is particularly relevant for cognitive training interventions, where we see improvements in selected neurocognitive functions but no clear impact on core ADHD symptoms in well-blinded studies.
Thus, cognitive training interventions may benefit those with ADHD irrespective of the effect on core symptoms. It would, however, be much more meaningful if we could show evidence that the improvement in neurocognitive function was associated with improvements in impairment and/or QoL. Further research investigating whether cognitive training does, in fact, impact functioning and QoL vis clearly needed.
There are several other unanswered issues to consider. Neurocognitive improvement may influence subclinical changes, such as increased ability to stay longer on a task (Froehlich et al, 2014) or improving a child's subjective feeling of their ability to stay focused (Tamm et al, 2013). These may provide a neurocognitive state that children can learn and practice their life skills to live with ADHD that are independent from a measurable attenuation of ADHD symptoms or related impairment using rating scales.
In this review, we only compared outcomes immediately post-treatment. Neurocognitive change may require more time to influence behavioral or psychological changes among children with ADHD (Roberts et al, 2016; Spencer-Smith et al, 2020). There was a 4-year post-MPH treatment follow-up study (Coghill et al, 2014a) that showed a significant association between improvement in short-term memory and ADHD symptoms, but no association for other neurocognitive functions. The results highlighted the importance of longitudinal investigation of correlations between changes in neurocognitive and clinical outcomes, cautioning on the reliance of cross-sectional associations.
In contrast, the long-term effects of cognitive training trials are less promising to predict behavioral changes. According to Roberts et al (2016), Cogmed working memory training improved select aspects of working memory performance among children with low working memory at 6 and 12 months, but not at a 24-month follow-up. However, the training did not change literacy or subjectively rated impairment (i.e., behavioral difficulties). The intervention group performed worse in math at a 2-year follow-up only.
Similarly, Spencer-Smith et al (2020) reported no difference in change in working memory between the intervention and control group at the 6-month follow-up among children with elevated ADHD symptoms. Despite the less promising predictable value of working memory reported from the two previous studies, the long-term effects of improvement in multiple neurocognitive domains and its relation to the broader concept of clinical outcomes still need to be explored (Coghill, 2021).
Lastly, the link between neurocognitive changes and behavioral changes is not clear among children with ADHD. Researchers and clinicians of ADHD have previously largely assumed this link, which commonly appears in other health conditions (e.g., neurocognitive disorders; American Psychiatric Association, 2013; World Health Organization, 2021). However, the discrepancy between objectively measured neurocognitive function (e.g., inattention) and subjectively measured behavioral representation (e.g., inattentive behavior) was a well-known phenomenon in the broader context of research filed (Toplak et al, 2013).
We need to investigate the association between neurocognitive functions and ADHD symptoms and impairment with much more complex models such as consideration of moderators or mediators (Fig. 4). In addition, further investigation is required to evaluate the independent value of measuring neurocognitive changes when treating ADHD. Nevertheless, potential values of tracking neurocognitive changes indicated from this review include: (1) understanding full treatment effect on children with ADHD, (2) ancillary indication for subclinical changes, and (3) providing less biased objective aspects of treatment effect.
Models to understand relationships among ADHD treatments outcomes. Adapted from Coghill et al (2014a).
Limitations
The scope of this study was limited to investigating neurocognitive functions and its relations to ADHD clinical outcomes, which innately included following limitations. First, this review included a lack of consideration of possible covariate or moderator variables between neurocognitive and clinical changes, such as psychological difficulties, social skills, family support, or parenting style (Faraone, 2015; Meer et al, 2014; Sonuga-Barke et al, 2013). Second, due to the nature of the research design, we did not have full control for subtypes of ADHD (i.e., predominantly Inattentive, predominantly Hyperactive-Impulsive or Combined; American Psychiatric Association., 2013) and some aspects of comorbidity (e.g., sleep quality, special learning difficulties, anxiety, ODD, and mood).
Although the effect of pharmacological interventions does not significantly differ by sub-types of ADHD (Cortese et al, 2018), it is inconclusive whether the treatment effects of cognitive training would be differed by sub-types of ADHD (Spencer-Smith et al, 2020). However, it is plausible to hypothesize that the effect of cognitive training could differ by subtype of ADHD (van der Donk et al, 2020). Third, we conceptualized pharmacological treatments as one construct, which included four different forms of stimulants (i.e., MPH, OROS MPH, DEX, ATX). Meta-analysis results were mainly driven by MPH trials.
Similarly, for cognitive training, we conceptualized cognitive training as one construct; however, data were driven by mainly working memory training programs. Sub-group analyses (i.e., attention, worming memory, vs. executive function) were not possible due to a lack of raw data. Fourth, most cognitive training trials did not fully control concomitant medication effects. Although randomization should have cancelled out the effects of medication, especially for trials with a large sample size, an influence of medications cannot be fully excluded.
Last, we have excluded many cognitive training programs, including psychological, physical (i.e., exercise), and/or behavioral components, due to the validity of the treatments on neurocognitive functions. However, this may have been limited, encapsulating the full treatment effects of cognitive training for other clinical outcomes (i.e., ADHD symptoms, subjective impairments). The results from this review should not be used to compare the treatment effects of specific pharmacological and cognitive training interventions.
Despite its limitations, the review and meta-analyses certainly offers valuable insights into how we better conceptualize neurocognitive functions when treating ADHD. Based on the current review and meta-analysis limitations, the current findings should not be generalized to directly compare the treatment effects of two types of ADHD interventions. Likewise, comparing which type of treatment is better than the other was neither the aim nor the current review's scope.
Implications
Clinical practice
The potential clinical value of measuring neurocognitive functions has been highlighted by this systematic review of RCTs for ADHD interventions. It is important to monitor changes with ADHD treatments in both neurocognitive and clinical outcome domains. These are two independent domains that cannot be indicators for one or another. For example, working memory behaviors measured by a subjective questionnaire cannot be an indicator for neurocognitive working memory function (Biggs et al, 2011). An individual's neurocognitive strengths and weaknesses need to be recognized and measured with appropriate methods. During the titration or ongoing pharmacological interventions, clinicians can monitor changes in neurocognitive functions.
These objective data can provide additional information for clinical decision making, regardless of its apparent connection to the clinical outcomes. However, the clinicians should be encouraged to monitor the individual experience of the changes in neurocognitive function regarding how it reflects into individual's functional difficulties or QoL (Coghill, 2021; World Health Organization, 2021).
Future research
Future studies investigating the effects of ADHD treatments are encouraged to explore the impact on neurocognitive functions in a broader context of improving children's life with ADHD, which is above and beyond the reductions in ADHD symptoms and symptom-related functional/behavioral impairment (Fig. 4). It is recommended to develop comprehensive and multifactorial outcome measures, and to better assess what and how we measure these treatment outcomes. These measures should require good psychometric properties, in addition to being practical within the clinical setting (Hall et al, 2016b).
Communication with vulnerable populations
For researchers and clinicians, we need to interpret the potential treatment effects of ADHD interventions carefully to participants, families, teachers, and patients. We need to be more transparent about what we are assuming (i.e., associations between neurocognitive function and QoL) and what we know about it (i.e., currently, there is no association between neurocognitive improvement and improvement in ADHD symptoms).
Although there was no clear indication of associations between changes in neurocognitive functions and ADHD clinical outcomes, the authors identified independent values to measure changes in neurocognitive functions. The findings highlighted a lack of understating of how these improved neurocognitive functions affect an individual's day-to-day function or their QoL when living with ADHD. Further research directions are suggested with comprehensive outcomes measures and a complex theoretical model to understand the full effect of ADHD interventions.
Differences Between Protocol and Review
Search syntax was amended during the first week of January 2021. The main difference was in the research design set, in which we indicated an error in using a Boolean search command “OR” in “crossover design.” We changed this Boolean search command to “AND.” We also included attention as one of the neurocognitive outcome measures. The definition for cognitive training was refined, which included cognitive intervention only and was most typically delivered via a computerized program. We excluded any training, including behavioral, social, or psychological components.
Footnotes
Disclosures
S.L., T.R.H., B.J., R.T., V.P., and M.S.-S. have no disclosures. D.C. has received royalties from Oxford University Press but was not related to the work described here.
Supplementary Material
Supplementary Data
Appendix
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
