Working Memory Training in ADHD: Controlling for Engagement,Motivation,and Expectancy of Improvement (Pilot Study)

Abstract

Objective: The aim of this study was to evaluate whether a shortened-length session of CogMed Working Memory Training (CWMT) would be a suitable active control group and evaluate study protocol to aid in design refinements for a larger randomized controlled trial (RCT). Method: Thirty-eight post-secondary students diagnosed with ADHD were randomized into 25 sessions of standard (45 min/session) or shortened (15 min/session) CWMT, or into a waitlist control group. Results: There was no significant difference in completion rate or training index score between the standard- and shortened-length groups indicating that both groups showed improvement and put forth good effort during training. Conclusion: Preliminary findings suggest that shorter training sessions may induce similar levels of engagement, motivation, and expectancy of improvement in participants. We conclude that a larger scale RCT that utilizes shortened-length training as an active control group is warranted, but that a few modifications to the study protocol will be required.

Keywords

ADHD adults working memory training randomized controlled trial

Introduction

In recent years, computer-based training programs have emerged as a treatment option for working memory (WM) impairments. Initial studies showed that these programs improved WM, generalized to other cognitive domains, and had lasted beneficial effects for at least 3 to 6 months after training was completed (Green et al., 2012; Holmes, Gathercole, & Dunning, 2009; Johnstone, Roodenrys, Phillips, Watt, & Mantz, 2010; Klingberg et al., 2005). However, recent meta-analyses have found that WM training produces short-term, training-specific effects that do not generalize to other domains (Melby-Lervåg & Hulme, 2013; Rapport, Orban, Kofler, & Friedman, 2013). Moreover, WM training literature in general has come under scrutiny due to the inclusion of inadequate control groups, which are critical to controlling for the non-specific effects of not only training but also being a participant in a research study (Shipstead, Redick, & Engle, 2012).

Thus, the purpose of this pilot study was to ascertain whether a shortened-length WM training program would be a suitable active control group that controlled for non-specific effects such as engagement, motivation, and expectancy of improvement. The CogMed Working Memory Training (CWMT) program was chosen as the intervention for this study and participants were randomized into one of three arms: (a) a treatment arm, in which participants underwent standard CWMT (25 daily, adaptive training sessions of 45 min over a 5-week period and weekly calls from a certified CWMT coach); (b) an active control arm, in which participants received a shortened version of CWMT that was still adaptive and involved weekly coach calls over a 5-week period, but the daily training sessions were shortened to 15 min; and (c) a waitlist control arm, in which participants received weekly coach calls, but did not undergo any WM training (until after completing the study), to control for possible effects of coaching.

CWMT was built on a “physical-energetic” model that equates WM to a muscle that is strengthened by repeated use (Melby-Lervåg & Hulme, 2013) and according to Klingberg (2010) training must not only be adaptive but also intensive for there to be improvement. Thus, the inclusion of the shorter length version not only served as an active control to account for non-specific effects but also allowed us to examine “dose effects” of training. We hypothesized that for outcome measures we would find either (1) a dose effect, in which the standard-length training confers the most benefit while the shortened-length training confers lesser benefits, or (2) that the standard-length training confers beneficial effects while the shortened-length training will be no different than waitlist control. We also hypothesize that the standard- and shortened-length training groups will have comparable completion rates as well as improvement index scores indicating that two groups are equivalent in engagement, motivation, and expectancy of improvement.

Post-secondary students with ADHD were used as the patient population for this study as there is little literature available on non-medical interventions for this ADHD subgroup. While seemingly high functioning, these students do struggle compared with their non-disordered college peers; they are more likely to have lower grade point averages, higher drop-out rates, and more difficulties with their concentration, memory, and time management skills (e.g., DuPaul, Weyandt, O’Dell, & Varejao, 2009; Dvorsky & Langberg, 2014; Woltering, Liu, Rokeach, & Tannock, 2013). There is a need for evidence-based interventions that target not just their ADHD symptomatology but also the cognitive deficits they experience in everyday life (Gropper & Tannock, 2009).

As this was a pilot study, our overall objective was to evaluate study components, such as the utility of the proposed treatment and control groups, intake criteria, coach calls, and other aspects of the general protocol, so to aid in design refinements for a larger randomized controlled trial (RCT). We aimed to (a) explore reasons for any attrition such as inherent differences between compliant and non-compliant participants, possible cohort effects, or inadequate intake procedure and (b) interpret effect sizes, and (c) conduct a power analysis to determine the sample sizes needed for the larger RCT. Also, given that post-secondary students with ADHD have received negative attention in the literature for symptom feigning and abuse of stimulant medication (Sollman, Ranseen, & Berry, 2010), we explored the use of a simple assessment protocol for validating current ADHD symptomatology to ascertain whether it might assist in detecting possible malingering.

Method

Participants

A total of 38 students with a diagnosis of ADHD who were enrolled in post-secondary education (52.6% male), aged 18 to 35 (M = 23.39, SD = 4.02), participated in this pilot study (see Figure 1). Prior to commencing the study, counselors from Disability Service Centers at several Colleges and Universities were invited to attend a start-up meeting to request their support with recruitment, discuss the feasibility of the study procedures, and seek advice about working with a post-secondary ADHD population. The first two cohorts of participants completed WM training in the summer based on the advice of disability service counselors, as they believed students would have fewer academic commitments during this time and thus optimize compliance with training. We did not specify a sample size a priori, but aimed for 8 to 10 participants per cohort—a sample size that could be feasibly recruited and handled by the community agency providing the WM training within the first couple of months of the planned large-scale study. Students were recruited through information sessions held on campus at four universities in a major metropolitan city in Canada. Information sessions were advertised through emails sent from Student Disability Services to students registered with ADHD. A study brochure was also developed and distributed at Student Disability Services Centers.

Figure 1.

Flow of participants through the trial.

In Canada, registration with Student Disability Services requires students to provide comprehensive documentation or undergo a new assessment to confirm their diagnosis. About one third of the students (28.9%) were also registered with a comorbid learning disability. Of the 23 (60.5%) participants who were being treated with medication for their ADHD symptoms, 2 received non-stimulant medication (atomoxetine), while the majority (n = 21) received psychostimulants: specifically dextroamphetamine (8), lisdexamfetamine (4), methylphenidate short release (4), and methylphenidate extended release (5). Medication status was not controlled in this study, but participants were advised to maintain their current pharmacological treatment throughout the study. Medication status and dose were recorded at each visit: 3 of the 23 (13%) participants, one in each of the three groups, reported taking a higher dose of medication at the post-test assessment than at the pre-test assessment.

Semi-structured telephone interviews were conducted to assess students’ eligibility to participate in the study. Inclusion criteria were as follows: (a) previous diagnosis of ADHD, (b) current enrollment in a post-secondary educational institution, (c) registered with Student Accessibility/Disability Services with a diagnosis of ADHD, (d) current symptoms consistent with diagnostic criteria for ADHD, and (e) between 18 to 35 years of age. Exclusion criteria included (a) major neurological dysfunction or psychosis, (b) current use of sedating or mood-altering medication other than medication provided for ADHD, (c) uncorrected sensory impairment, (d) motor or perceptual handicap that would prevent use of a computer program, or (e) a history of concussion or traumatic brain injury prior to ADHD diagnosis. Exclusion criteria were ascertained from self-report during the intake interview.

We used a number of approaches to validate current ADHD symptomatology and explore possible malingering. Specifically, participants (a) had to be registered with a diagnosis of ADHD at their post-secondary Disability Service Center, (b) needed to meet the criterion score (detailed below) on the 6-item Adult ADHD Self-Report Scale–Part A (ASRS-A) during the telephone intake interview, during which they were required to provide real-life examples for each of the 6 items, to ensure they understood the question and that the reported behavior was a reasonable example of that ADHD symptom, (c) obtained an elevated score on the self-report paper-and-pencil version of the 18-item Adult ADHD Self-Report Scale (ASRS) completed at the pre-test assessment, and (d) obtained an elevated score on an adapted 18-item version of Adult ASRS completed by a significant other (ASRS-other). Significant others, such as a partner, parent, sibling, or friend, completed the ASRS-other using a secure, online software program (http://www.surveymonkey.com): the majority of respondents were parents of the participant. Conventional scoring of the ASRS requires a score of ≥2 (sometimes, often, very often) on Items 1 to 3 and ≥3 (often, very often) on Items 4 to 6, with at least 4 of the 6 items meeting these criteria to indicate a current symptom profile consistent with a diagnosis of ADHD.

Eligible participants were randomized into one of three treatment arms: standard-length training (45 min), shortened-length training (15 min), or a delayed-training waitlist control group (described below). There were no significant group differences at baseline for age, IQ, comorbidity, or severity of ADHD symptoms. Demographic and descriptive characteristics of each treatment arm are shown in Table 1. Participants were assessed individually at two different time points; prior to training (T1) and 3 weeks after completing training (T2). On each occasion, participants completed a behavioral assessment, which included a battery of neuropsychological tests and behavioral rating scales, and a neural assessment, in which they underwent electroencephalography while completing a battery of executive function tasks. The behavioral and neural assessments were counterbalanced, took a total of 5 hr at each time point, and included the same battery of measures planned for the large-scale study. Data from the pre-test neural assessment are reported elsewhere (Woltering et al., 2013) and post-test neural data are not included in this report.

Table 1.

Participant Characteristics.

	Standard-length group		Shortened-length group		Waitlist control group		Total
	M	SD	M	SD	M	SD	M	SD
As randomized
Males (n)	10		5		5		20
Females (n)	8		3		7		18
Age (years)	23.67	5.12	21.40	2.70	23.11	4.11	23.39	4.02
WASI IQ^a	122.11	12.62	103.00	7.71	112.11	11.129	112.26	14.68
TOWRE^a	107.67	14.19	112.20	13.44	99.33	11.158	104.89	11.907
WCJ Math Fluency^a	98.33	12.61	92.80	14.04	92.44	12.82	93.95	13.63
ASRS-A Interview^b	18.11	2.84	17.4	2.51	17.33	1.35	17.63	2.76
ASRS-A Interview Symptom Count	5.46	0.66	5.5	0.17	5.21	0.21	5.38	0.68
ASRS T1 Total^b	51.33	9.41	50.20	10.26	48.44	11.83	49.43	8.74
ASRS-A T1^b	17.65	3.50	17.89	2.37	18.09	3.21	17.84	3.10
ASRS T1 Symptom count	4.71	2.06	4.78	0.36	4.91	0.29	4.78	0.92
ASRS Other^b	47.11	7.32	47.80	8.58	44.89	8.70	46.39	7.87
SA-45^a	61.07	9.82	64.86	9.23	61.20	8.63	61.94	9.17
GRIT^b	3.25	0.38	2.99	0.43	2.89	0.84	3.08	0.58
WAIS Digit Span^a	9.50	2.23	9.00	2.69	9.82	3.19	9.47	2.59
As analyzed
Males (n)	5		3		5		13
Females (n)	3		3		4		11
Age (years)	24.25	4.34	21.40	2.70	22.86	3.18	23.13	3.75
WASI IQ^a	116.00	14.72	103.00	7.71	114.14	9.77	111.88	13.02
TOWRE^a	106.75	12.84	112.20	13.44	97.43	9.03	104.08	11.43
WCJ Math Fluency^a	96.25	5.19	92.80	14.04	90.29	13.90	94.08	13.68
ASRS-A Interview^b	18.00	1.83	17.40	2.51	18.00	3.51	17.21	2.69
ASRS-A Interview Symptom Count	5.50	0.19	5.50	0.22	5.10	0.233	5.33	0.64
ASRS T1 Total^b	49.00	10.23	50.20	10.26	51.86	11.02	49.61	8.89
ASRS-A T1^b	17.14	1.26	17.85	0.94	18.78	1.04	18.00	2.95
ASRS T1 Symptom count	4.57	0.37	4.71	0.47	5.11	0.26	4.83	0.98
ASRS Other^b	46.25	5.25	47.80	8.58	43.57	8.22	45.56	7.47
SA-45^a	55.29	5.77	66.50	8.92	60.89	9.09	60.64	8.90
GRIT^b	3.26	0.42	2.89	0.36	2.62	0.62	2.90	0.55
WAIS Digit Span	9.25	1.83	9.43	2.93	9.00	2.47	9.21	2.4

Note. WASI = Wechsler Abbreviated Scale of Intelligence; TOWRE = Test of Word Reading Efficiency; WCJ = Woodcock Johnson; ASRS-A = ADHD Self-Report Scale–Part A; T1 = prior to training; SA-45 = Symptom Assessment–45; WAIS = Wechsler Adult Intelligence Scale.

Standardized scores.

Raw scores.

Intervention Program

The CogMed WM Training program, developed by CogMed Cognitive Medical Systems AB (Stockholm, Sweden), was used as the intervention program in this study, as this program has the most empirical evidence to support its efficacy in improving WM (Klingberg, 2010). Based on an initial survey of participants’ preference, we used the RM version of the Cogmed program. The RM version has an interface similar to a video game and was originally designed for school-aged children but with the same exercises as the adult QM version. The program was implemented online through the Cogmed-Pearson’s website (http://www.cogmed.com) and completed in participants’ homes or place of residence: 64% of participants in this study were living at home with their parents.

The standard RM version of CogMed consists of 12 auditory-verbal and visual-spatial WM tasks that involve the storage and manipulation of particular sequences of stimuli. For each task, an adaptive algorithm automatically adjusts the difficulty level based on trial-by-trial performance to ensure individuals are always working at the upper limit of their WM capacity. Positive reinforcement is provided at the end of each trial through computerized verbal feedback. The program requires 25 training sessions, each taking about 45 min, to be completed over 5 to 6 weeks. Weekly telephone calls from a certified CogMed coach are conducted to provide feedback on training performance, address any training challenges, make recommendations for the next week of training, and encourage compliance with the training schedule. Certified CogMed coaches from a community-based psychological services agency, independent from the research team, provided participants with 30 min of telephone-based support each week.

In this study, participants in the standard-length training group engaged in 45 min of training, completing 2 “core” tasks per session that were used throughout training plus another 6 of the remaining 10 possible tasks per session, which were chosen based on random computer selection of tasks, for a total of 90 WM trials. Those in the shortened-length training group engaged in 15 min of training, completing 45 trials of 4 WM tasks per session, which consisted of the two “core” tasks used throughout the training plus two additional tasks that changed during each training session based on random computer selection (see task descriptions in the appendix). Participants were requested to engage in five training sessions (one session per day of the specified length) per week for 5 weeks. Both the standard- and shortened-length training programs used the adaptive algorithm to adjust task-level difficulty and received the same number (5) of coach calls. Thus, the training groups only differed in duration of daily training sessions.

Waitlist control participants did not undergo any training during the 5-week period, but did receive weekly calls from a certified CogMed coach, who was independent from the research team, to control for possible effects of attention and motivation, with each call lasting approximately 30 min. Coach calls to waitlist control participants were a unique component of this pilot study. During these calls, coaches used a semi-structured script to discuss participant’s academic functioning (e.g., Did you make it to class, did you do your specified hours of studying for each course, what can you do differently to make these commitments work next week?). The feasibility of coach calls in this post-secondary ADHD sample formed one objective of this pilot study. After their T2 assessment, waitlist participants were able to choose between the standard- and shortened-length WM training: none opted to undergo training.

Procedure

This study was approved by the Institutional Research Ethics Boards of the participating universities, as well as by the community agency providing the WM training program. Informed written consent was obtained from all participants prior to entering the study. Information given to students during recruitment indicated that participants would be randomized into a standard-length (45-min) training group, shortened-length (15-min) training group, or waitlist control group. Prior to enrolling in the study, participants were asked whether they were going on vacation, moving or undergoing any major life changes during the training period. If so, it was suggested that they enroll in the next cohort. Moreover, as the training was administered through the Internet, participants were able to complete training regardless of their location, so they were not obligated to stay at home. However, they were encouraged to complete training in a quiet, private room. Due to the nature of the treatment arms, it was not possible to keep participants or CogMed Coaches blind to condition. This is a limitation of the study as participants have investment in the intervention being a success, which may inflate subjective ratings on outcome measures (Sonuga-Barke et al., 2013). However, results of the randomization were not revealed to participants until after the T1 assessment. Prior to beginning training, participants were invited to attend an in-person start-up session, held in a centrally located university conference room, to be familiarized with their CogMed coaches and the WM training program. Only about 40% (n = 15) attended a start-up session prior to beginning training; the remaining 60.5% received this information over the phone. Participants were run in cohorts of 10 to 14 to ensure that the CogMed coach was able to provide each participant with adequate individual attention. Weekly coach calls were able to keep most participants on track and the few participants who did skip trainings withdrew from the study prior to completing the program. Randomization, with stratification for sex, was carried out separately for each cohort by an investigator independent of the research team. During this study, three cohorts were trained within 5 months, with two cohorts completing training during the summer and one during the fall semester.

Measures

Baseline measures

(a) Vocabulary and Matrix Reasoning subtests from the Wechsler Abbreviated Scale of Intelligence–Second Edition (WASI-II) were administered at T1 as an estimate of general intellectual ability (Wechsler, 1999); (b) two subtests, the Math Fluency subtest from The Woodcock Johnson–Third Edition (WCJ-III; Woodcock, Mcgrew & Mather, 2001) and the Test of Word Reading Efficiency–Second Edition (TOWRE-II; Wagner, Torgesen, & Rashotte, 1999), were used to screen basic math and reading skills; (c) Symptom Assessment–45 (SA-45) was used to assess general psychiatric symptomology (Maruish, Bershadsky, & Goldstein, 1998); and (d) a scale called the “GRIT” was used to assess ambition (e.g., I aim to be the best at what I do), perseverance of effort (e.g., I have overcome setbacks to conquer important challenges), and consistency of interest (e.g., I often set goals but later choose to pursue a different one) in terms of long-term goals (Duckworth & Quinn, 2009). The internal consistency and test–retest reliability of the Grit has been good (Duckworth, Peterson, Matthews, & Kelly, 2007). Formally tested norms are not yet present, though data published from a normative sample (aged 25-35 years old) suggested average scores were around 3.2 (SD = 0.7; Duckworth & Quinn, 2009).

Symptom validation measures

The following measures were used to assess current ADHD symptoms: (a) ASRS-Part A was administered during the intake interview; (b) ASRS, Parts A and B were administered at T1; (c) Parts A and B of ASRS-other were completed by a significant other prior to T1.

Compliance measures

(a) Duration of training: the number of daily sessions participants completed and number of weeks participants took to complete the 25 required training sessions. (b) Coach calls: the number of the five scheduled coach calls that participants completed. (c) CogMed Training Index Score: this score, which measures the users improvement on selected training tasks, is calculated by subtracting the results of Days 2 and 3 of training from the best 2 days during the training period; the mean index score for individuals 18 to 65 years is 30 (normal range = 18-42), with a higher score suggesting good effort during training (CogMed, 2011). (d) Attrition: examining whether there was differential attrition (e.g., the drop-out rate) between standard- versus shortened-length programs and between cohorts (summer vs. regular academic year).

Outcome measures

Outcome measures were categorized into (a) Criterion Measures: assessments that closely resemble tasks from the WM program, (b) Near-Transfer Measures: measures that tap WM but do not resemble trained tasks, and (c) Far-Transfer Measures: measures of transfer to everyday functioning, academic performance, and ADHD symptomology. The majority of these measures were selected based on the results of previous studies (Gray et al., 2012; Klingberg et al., 2005). Outcome measures were analyzed using raw scores.

Criterion measures

(a) Digit Span Forwards (DSF), Digit Span Backwards (DSB), and Digit Span Sequencing (DSS) from the Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV) were used to assess auditory-verbal working and short-term memory (Wechsler, 2008); (b) The Spatial Span Forwards (SSF) and Spatial Span Backwards (SSB) from The Cambridge Neuropsychological Testing Automated Battery (CANTAB) was used to assess visual-spatial short-term and WM (Fray, Robbins, & Sahakian, 1996); (c) The Finger Windows Forwards (FWF) and Finger Windows Backwards (FWB) subtest from The Wide Range Assessment of Memory and Learning–Second Edition (WRAML-II) was used as another measure of visual-spatial short-term and WM (Sheslow & Adams, 2003).

Near-transfer measures

(a) The CANTAB Spatial WM task was used to assess strategy skills and visual-spatial WM (Fray et al., 1996), (b) The CANTAB Pattern Recognition Memory task assessed visual short-term memory (Fray et al., 1996), (c) an adapted version of Kahneman’s WM task was used to assess visual WM. In the latter task, participants mentally add 1 or 3 to each number in a visually presented sequence of three or four digits ranging from 1 to 9, and say the resulting answer out loud. There are five trials at each of the four levels of difficulty in the following sequence: add 1 to a three digit sequence, add 1 to a four digit sequence, add 3 to a three digit sequence, and add 3 to a four digit sequence. The stimulus duration is 4 s and the inter-trial interval is 3 s.

Far-transfer measures

(a) The 18-item Adult ASRS (ASRS v1.1) was used to evaluate current manifestation of ADHD symptoms; this scale is based on the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; American Psychiatric Association, 1994) and has high internal consistency and concurrent validity (Adler et al., 2006, p. 2). The Cognitive Failures Questionnaire (CFQ) was used to assess self-reported errors in memory, perception, and motor function when completing everyday tasks; the questionnaire has been found to have good external validity and stability over time (Broadbent, Cooper, FitzGerald, & Parkes, 1982, p. 3). The Barkley Deficits in Executive Functioning Scale–Short Form (BDEFS-SF) was used to evaluate executive functioning deficits in everyday life activities (Barkley, 2011).

Statistical analysis

Data were first checked to ascertain the shape of the score distributions using SPSS v.21. A Winsorizing technique described in Tabachnick and Fidell (2007) was used to minimize the effect of 1 to 2 outliers in each of the following variables: CANTAB Pattern Recognition Memory, CANTAB Spatial WM Between Errors Score, WAIS backwards, WCJ Math Fluency, Kahneman’s addition task, and ASRS (Tabachnick & Fidell, 2007). An “As-Treated” approach was used to test for training effects, as missing data at post-test (40% attrition) precluded an Intent-to-Treat analysis of these data. An ANCOVA was conducted with baseline as a covariate, Group (standard-length, shortened-length, and waitlist control) as a between-subjects factor and post-test scores on target indices as the dependent variables. Partial eta-squared values were used to obtain a rough estimate of effect sizes that could be expected in a larger scale RCT. According to Vacha-Haase and Thompson (2004), $η_{p}^{2}$ = .01 corresponds to a small effect, $η_{p}^{2}$ = .10 corresponds to a medium effect, and $η_{p}^{2}$ = .25 represents a large effect. Power analyses were conducted using GPower (Faul, Erdfelder, Buchner, & Lang, 2009) to determine the optimal sample size needed for a larger randomized control trial. Primary measures of change used for the analysis were the CANTAB Spatial WM (between errors score) and the ASRS.

Results

Baseline Measures

One-way ANOVAs showed there were no differences between the three treatment groups at baseline on WASI-II, WCJ-III, TOWRE-II, SA-45, or GRIT (all p > .1); this showed that our randomization was effective and that the three groups of participants had comparable levels of IQ, math fluency, reading fluency, psychiatric symptomology, and perseverance (Table 1).

As evident from the mean scaled score for the estimated IQ score (M = 112.3, SD = 14.7) and WASI Digit Span (M = 9.47, SD = 2.6), this was a high functioning sample, with only 26% having poor auditory-verbal WM, as indicated by scaled scores below the 16th percentile. Six (15.8%) participants scored at or below the 7th percentile (standardized score ≤ 78) on the Math Fluency Subtest from the WCJ-III, indicating ongoing academic difficulties in basic mathematical skills. However, no participant manifested low scores on the TOWRE, and none of the 11 students who self-reported a learning disability during the intake interview had a score below 78 on either of these subtests. However, the mean T score (61.94) for the SA-45 indicated that this sample had elevated psychiatric symptomatology with 54% of participants having T scores above 60.

Symptom Validation Measures

Paired t tests showed that there was no significant differences between ASRS-A (6 items) completed during the intake interview and the first 6 items of the ASRS (18 item) administered at T1. There were also no significant differences between the collateral report using ASRS-other and the self-report ASRS administered at T1. Summary data for the ASRS-A, ASRS, and ASRS-other are shown in Table 1. As shown, the mean raw scores for the ASRS were greater than 30, corresponding to the 95th percentile, across informants and modality of ASRS administration, indicating a symptom profile consistent with an ADHD diagnosis.

Compliance Measures

The attrition rate for this study was high, with 14 of the 38 participants (36.8%) not completing their assigned treatment condition. In the standard-length group, 44.4% (n = 8) of participants completed the training in 5.63 weeks (SD = 0.74, range = 5-7 weeks), with an average Training Index Score of 30.29 (SD = 8.11, range = 22-42). Participants who did not complete the standard-length training (55.6%) completed an average of 11.6 training sessions (SD = 5.25, range = 0-16 sessions). In the shortened-length group, 75% (n = 6) completed the training in 6.29 weeks (SD = 1.38, range = 5-9 weeks), with an average Training Index Score of 25.71 (SD = 13.43, range = all scores were between 15 and 50, except for one whose index was 7). Participants who dropped out from the shortened-length training group (25%) did not complete any training sessions prior to leaving the study. The majority of participants (94%) who completed the training obtained a Training Index Improvement Score within the expected range (i.e., a score of 18-41). The two training groups did not significantly differ in the completion rate, χ²(1) = 2.7, p = .1, or Training Index Improvement Score, t(12) = .77, p = .46. In the waitlist group, the majority of participants (75%, n = 9) completed all five coach calls: the remaining 25% did not receive any calls as they dropped out after the initial T1 assessment. There were no significant differences in attrition rates between cohorts, χ²(2) = .14, p = .93. Attrition in the cohorts that completed training in the summer was 64%, whereas attrition in the cohort that completed training in the winter was 50%. A total of 24 participants completed the study requirements and their data were used in the analysis (see Table 1).

Post hoc analysis (t tests) of potential baseline differences between completers and non-completers revealed significant differences only on subscales of the SA-45 and the GRIT; non-completers had higher scores on one (Paranoia) of the nine subscales of the SA-45, t(34, 1) = 2.24, p = .03, and lower scores on the GRIT subscales, specifically Ambition, t(x) = 2.06, p = .05, and Perseverance of Effort, t(x) = 2.23, p = .03.

Outcome Measures

Descriptive statistics are reported in Table 2. A table showing 95% confidence intervals is available from the authors on request.

Table 2.

Descriptive Statistics for Criterion, Near-Transfer, and Far-Transfer Measures at Pre- and Post-Test.

	Standard-length training group (n = 8)				Shortened-length training group (n = 7)				Waitlist control group (n = 9)
	Pre-test		Post-test		Pre-test		Post-test		Pre-test		Post-test
Measure	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
Criterion
WAIS Digit Span: Forwards	9.38	1.30	12.50	2.67	10.57	1.62	11.71	2.81	9.67	2.60	10.67	2.24
WAIS Digit Span: Backwards	8.75	2.77	12.75	2.49	7.86	2.27	10.43	2.51	8.56	1.88	8.78	2.49
WAIS Digit Span: Sequencing	8.75	1.49	10.12	2.10	8.86	2.27	9.29	1.38	8.33	2.18	9.33	2.65
CANTAB Spatial Span: Forwards	6.75	2.50	7.88	1.13	6.57	1.40	6.86	1.57	7.00	1.41	7.22	1.79
CANTAB Spatial Span: Backwards	5.88	1.64	7.75	1.28	6.29	1.98	5.86	2.04	6.56	1.81	6.56	1.67
WRAML Finger Windows: Forwards	18.38	3.25	22.13	2.23	15.29	4.31	17.29	5.47	17.89	3.59	18.78	4.66
WRAML Finger Windows: Backwards	16.63	5.85	20.12	3.044	14.00	5.39	14.43	7.53	17.44	5.41	26.56	4.39
Near-transfer
CANTAB Spatial Working Memory: Strategy Score	15.25	15.76	5.38	5.55	23.14	16.27	20.43	13.43	19.44	20.49	17.89	23.38
CANTAB Spatial Working Memory: Between Errors Score	26.88	6.45	22.50	4.21	33.86	4.49	30.29	4.61	31.44	7.11	29.00	7.78
CANTAB Pattern Recognition Memory	78.13	19.76	83.85	15.18	83.93	14.52	85.71	7.93	89.35	4.71	80.09	31.79
Kahneman’s Addition Task	16.75	4.03	18.25	2.19	16.14	4.88	17.71	2.14	15.22	5.33	18.33	2.55
Far-transfer
ASRS	47.71	7.87	46.57	8.44	49.29	9.69	44.71	9.76	51.33	9.68	51.44	11.23
CFQ	51.66	5.92	44.67	6.38	58.75	12.95	52.50	14.01	66.86	6.34	69.00	8.98
BDEFS	41.14	11.64	41.71	12.15	54.71	8.60	55.29	9.62	52.63	12.49	52.88	12.33

Note. WAIS = Wechsler Adult Intelligence Scale; CANTAB = Cambridge Neuropsychological Testing Automated Battery; WRAML = Wide Range Assessment of Memory and Learning; ASRS = ADHD Self-Report Scale; CFQ = Cognitive Failures Questionnaire; BDEFS = Barkley Deficits in Executive Functioning Scale.

Criterion WM measures

ANCOVA were conducted on the post-test scores with the relevant baseline score as a covariate, and Group (standard-length, shortened-length, and waitlist control) as a between-subjects factor. The ANCOVA for the WAIS-IV DSB was significant, F(1, 24) = 7.45, p = .004 (see Figure 2). The strength of the relationship between training group and dependent variable was large, as assessed by a $η_{p}^{2}$ of .43. Post hoc tests showed that participants in the standard-length group did significantly better than waitlist control groups (p = .001), and that there was no difference between the standard- and shortened-length group, or the waitlist control group and the shortened-length group (all p > .059). No differences between the three groups were found at post-test for the WAIS-IV DSF, F(1, 24) = 1.29, p = .3, and Sequencing, F(1, 24) = .35, p = .71, subtests; however, the observed differences at T2 were of medium ( $η_{p}^{2}$ = .11) and small ( $η_{p}^{2}$ = .034) effect size, respectively.

Figure 2.

Post-test effects of working memory training.

The ANCOVA for the CANTAB SSB subtest was significant, F(1, 24) = 4.60, p = .023 (see Figure 2). The strength of the relationship between the group and dependent variable was large, as assessed by a $η_{p}^{2}$ of .32. Post hoc tests showed that participants in the standard-length group did significantly better than both the shortened-length and waitlist control groups (all p < .036), which did not differ (p = .44). Results for the CANTAB SSF subtest showed no significant effects, F(1, 24) = 1.11, p = .35; however, the observed differences at T2 were of medium effect size ( $η_{p}^{2}$ = .10). With respect to the FWF, F(1, 24) = 2.04, p = .16, and FWB, F(1, 24) = 2.06, p = .15, subtests, there were no significant differences found between the three groups at post-test. However, the observed group differences were of medium effect size (both ps = .17).

Near-transfer WM measures

No differences were found between the three groups at post-test for any of the near-transfer measures: CANTAB Spatial WM strategy score, F(1, 24) = 1.36, p = .28; CANTAB Spatial WM between errors score, F(1, 24) = 1.99, p = .16; CANTAB Pattern Recognition Memory, F(1, 24) = .27, p = .76; or Kahneman’s Addition Task, F(1, 24) = .414, p = .67. However, medium effect sizes were found for the CANTAB Spatial WM Between Errors Score ( $η_{p}^{2}$ = .12) and Strategy Score ( $η_{p}^{2}$ = .17), and small effect sizes were found for CANTAB Pattern Recognition Memory ( $η_{p}^{2}$ = .03) and Kahneman’s Addition Task ( $η_{p}^{2}$ = .04).

Far-transfer measures

There were no significant differences found between the three groups at post-test for the ASRS, F(1, 24) = 1.64, p = .22, but observed group differences at T2 were of medium effect size for ASRS ( $η_{p}^{2}$ = .15). The ANCOVA conducted for CFQ was not significant but the trend was toward greater improvement in both CWMT groups, F(1, 24) = 3.46, p = .06 (see Figure 2). The strength of the relationship between the group and dependent variable was large, as assessed by a $η_{p}^{2}$ of .32. Post hoc analysis showed that participants in the standard- and shortened-length groups did not differ (p = .83), and both groups did significantly better at T2 than the waitlist control group (all p < .038). With respect to BDEFS, no significant differences found between the three groups at post-test, F(1, 24) = .04, p = .961.

Power Analysis

A power analysis was conducted on the CANTAB Between Errors Score and the ASRS to estimate the required sample size for a future study utilizing an ANCOVA analysis with three groups and one covariate. For a medium effect size to be detected (80% chance) in the CANTAB Between errors ( $η_{p}^{2}$ = .12—as computed in present pilot) with an alpha set at .05, a sample of 25 participants in each group would be required. Similarly, for the ASRS ( $η_{p}^{2}$ = .15), a sample of 19 participants would be required.

Discussion

The purpose of this pilot study was to ascertain whether shortened-length CogMed WM training (15 min) would be a suitable active control group for a standard WM training program (45 min), meaning that it would control for non-specific effects such as engagement, motivation, and expectancy of improvement. To determine whether the shortened-length program would be a suitable active control group for the larger RCT, we looked at the completion rate and index of improvement between the standard- and shortened-length treatment groups. As we hypothesized, there was no significant difference in the completion rate between the groups; however, more participants did complete the shortened-length training (77.8%) compared with the standard-length training group (44%). Moreover, there was no significant difference in the Training Index Scores and both groups’ scores fell in the expected range. This latter finding indicates that both groups showed improvement and put forth good effort during training, a concern that has beleaguered previous studies that used an active but non-adaptive control group. Based on these findings, we believe there is utility of the shortened-length training program as an active control group.

In regards to outcome measures, given Klingberg’s (2010) premise that cognitive training must not only be adaptive but also intensive, we predicted that we would see (a) a “dose effect” in which the shortened-length training group shows more improvement than the waitlist group, but less than that of the standard-length training group, or (b) that the standard-length group shows improvement but not the shortened-length group, which would not differ from the waitlist control. However, the pattern of findings was mixed. On the criterion visual-spatial WM task (CANTAB SSB subtest), the standard-length training group performed better at post-test compared with the shortened-length training and waitlist control groups, which showed little to no change over time. The standard-length group also did significantly better than waitlist control group on the criterion auditory-verbal WM task (WAIS-IV DSB subtest); however, there was no difference between the standard- and shortened-length group, or between the waitlist control group and the shortened-length group. There were also no near- or far-transfer effects found. Although these patterns of findings are surprising, it is important to keep in mind that this pilot study was not intended to or powered for inferential statistics.

In addition to the shortened-length control group, we also evaluated a novel study procedure, coach calls to participants randomized to the waitlist control group. This was designed to also control for the effects of motivation and expectation. The majority of these participants (81.8%) attended all of their coach calls, indicating that “distance coaching” is feasible in studies involving a post-secondary ADHD population. However, based on participants’ feedback, we better standardize the coach calls for the waitlist control in the larger RCT and provide the students with more practicable information about time management, organization, and mnemonic strategies.

A secondary objective of this pilot study sought to evaluate the protocol for validating current ADHD symptomatology and detecting possible malingering by administering the ASRS during the telephone intake interview and asking for real-life example of symptoms at T1, and having a second informant (a significant other) complete an adapted version of the ASRS. The analysis revealed no significant differences between the ASRS-A (6 items) completed during the telephone interview and the first 6 items of the 18-item self-report ASRS administered at pre-test assessment, nor between the total scores (based on 18-items) on the second-informant’s ASRS and the participant’s self-report ASRS administered at pre-test assessment. These results show evidence of ongoing elevated symptom count, indicating that participants were reporting real symptoms and that malingering in these participants was unlikely.

The overall objective of this pilot was to evaluate study components, such as the utility and potential benefits of the proposed treatment and control groups, intake criteria, coach calls, and other aspects of the general protocol, so to aid in design refinements for a larger RCT. The attrition rate for this study was high, with over one third (about 37%) of participants not completing their assigned treatment condition. However, analysis revealed inherent differences between completers and non-completers. Non-completers were more likely to have elevated symptoms of paranoia (SA-45) and lower ambition and perseverance of effort (GRIT), suggesting that dropout may not necessarily be related to the WM training requirement, but instead may be related to participant characteristics that increase likelihood of attrition. Screening for general psychiatric symptomatology and goal persistence during the intake process may alert interventionists to provide additional support to participants who are anxious about this type of intervention or who tend to be poorly motivated to put forth effort, which in turn might reduce attrition. Moreover, there were no significant cohort effects in relation to attrition indicating that participants could be recruited year round.

To determine the sample size needed for a larger confirmatory study, we obtained rough estimates of effect sizes that could be expected. Treatment effect sizes were large for some criterion measures (WAIS DSB, CANTAB SSB) and one far-transfer measure (CFQ), whereas others were of medium effect size (WAIS DSF subtest, CANTAB SSF, FWF and FWB subtest, CANTAB Spatial WM Strategy Score and Between Errors score, and ASRS). Small effect sizes were also found for WAIS DSS subtest, CANTAB Pattern Recognition Memory Task, and Kahneman’s Addition task. These findings suggest that significant effects could be found using larger sample sizes for many of the criterion and near-transfer measures, and that measures such as Kahneman’s Addition task may not be sensitive to treatment effects. Power analysis conducted for one near-transfer measure (CANTAB Spatial WM Between Errors Score) and one far-transfer measure (ASRS) found that to detect a small effect size (80% chance), significant at the 5% level, a sample of at least 19 to 25 participants per group would be required in a larger, randomized control study.

This pilot study has indicated the need for several important design refinements prior to conducting a larger trial. Our findings indicate that (a) the intervention could be administered during the summer or regular academic year; (b) additional procedures are required to minimize attrition, such as the use of a timetable or planner to ensure students’ understanding of the time requirements and the schedule of study assessments; (c) the provision of coach calls to participants in the waitlist control group was deemed useful but students requested more practicable information (e.g., organization, time management, etc.); and (d) the shortened-length CWMT showed promise as an active control group that control for motivational and expectancy effects, but post-training interviews are needed to further explore this issue and ascertain whether those in the shortened-length training felt as engaged and motivated as those in the standard group.

Limitations

Although this study had several strengths such as the use of an active control group and rigorous intake procedures to validate ADHD symptoms, there were also several limitations such as high attrition and that participants were not blinded, which may inflate ratings on subjective outcome measures as participants are invested in the intervention being a success (Sonuga-Barke et al., 2013). Also, post-secondary students from colleges were not included in this study only those from universities. Compliance with the five coach calls made to the standard- and shortened-length training group were not recorded, which will be an important measure to include in the larger RCT to determine whether telephone-based coach calls to a post-secondary population are feasible. Moreover, by design, this pilot study was not powered for inferential statistical analysis of the training outcomes. Thus, our preliminary reports of group differences in training outcomes need to be interpreted with great caution, and at most, they may indicate the magnitude of training outcomes that could be reasonably expected in a larger RCT.

Conclusion

This pilot study is the first to examine whether shortened-training sessions for an adaptive WM training program would serve as a suitable active control group that would induce comparable levels of engagement, motivation, and expectancy of improvement to those associated with the standard-length training. Both training programs used the same adaptive training algorithm over a 5-week period and received the same coaching support. Preliminary findings suggest that shorter training sessions may yield similar, if not higher, compliance rates and that participants showed comparable levels of motivation during training. However, results of preliminary outcome measures were mixed. We conclude that a larger scale RCT that utilizes shortened-length training as an active control group is warranted, but that improvement in intake and start-up procedures is required.

Footnotes

Appendix

Description of WM Tasks in the Cogmed WM Training Program.

Task	Description
1. Reproducing a light sequence in a visuo-spatial grid	Lamps arranged in a 4 × 4 grid are displayed. Participants watch several lamps light up and then reproduce the same sequence.
2. Reproducing a light sequence in a rotated grid	A rotating version of the grid task described above. After the sequence of lamps lights up, the panel rotates 90 degrees clockwise and participants reproduce the sequence in the panel’s new position.
3. Repeating numbers in reverse order	A keyboard with numbers is displayed and then numbers are read aloud. Participants respond by repeating the digits in reverse order.
4. Repeating numbers in reverse order with an invisible keyboard	Similar to the task described above, but numbers are not visible on the screen until participants are required to respond.
5. Identifying letter positions in a sequence	Letters are read aloud, one at a time. Participants have to remember the letters in the order in which they are read. A row of lights becomes visible and a flashing light cues the participant to respond. For example, if light number 3 lights up, then participants report the third letter they heard.
6. Identifying letter sequences	A sequence of letters is read aloud. Then, the participant is presented with three letters and must select the one that was presented.
7. Reproducing a light sequence in a rotating circle	A set of lamps is arranged in a rotating circle. Participants watch several lights light up and then reproduce the sequence, even though the lamps are constantly shifting position.
8. Reproducing a light sequence	A number of moving circles appear on screen and participants must reproduce the order in which they appeared.
9. Reproducing a sequence of moving shapes	A number of moving shapes light up and participants must reproduce the sequence in which they lit up.
10. Reproducing a light sequence in a 3D visuo-spatial grid	Lights are arranged in a “3D room” with 20 segments. Participants watch several lights go on and then reproduce the sequence.
11. Reproducing a light sequence in a 3D visuo-spatial cube	Lights are positioned in a 3D cube with 12 segments. Participants watch several lights go on and then reproduce the sequence.
12. Reproducing a sequence of numbers on a visual grid	A 4 × 4 grid with 16 latches is shown. A sequence of latches is opened displaying a set of numbers. Participants sort the numbers by clicking on the latch that contained the numbers in numerical order.

Acknowledgements

We would like to thank JVS-Toronto for their collaboration on this project. Thank you to Farhad Pantwala and Rachel Goldberg, whose support and enthusiasm were integral to the completion of this study. Many thanks to the Post-Secondary Disability Service Centers across Ontario for their help with recruitment, to the students who participated in this study, and to Daniel Glizer, Jessica Jung, and Soyeon Kim for their help with data collection.

Declaration of Conflicting Interests

The author(s) declared following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr. Tannock has served as a consultant for and received honoraria from Shire, Eli Lilly, and Purdue Pharmaceuticals.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Cogmed provided the software without cost for this study. The study was funded by Canadian Institute of Health Research (CIHR) (Grant #482246; R.T.) and by Canada Research Chairs Program (R.T.).

Author Biographies

Karizma Mawjee is currently a Certified Clinical Research Coordinator at the University of Toronto and Holland Bloorview Kids Rehabilitation Hospital for the Ontario Brain Institute’s Cerebral Palsy and Epilepsy Integrated Discovery Programs. She graduated from the University of Toronto with a Masters in Developmental Science. Her research interests focus on the efficacy of clinical interventions for atypically developing populations.

Steven Woltering is currently an assistant professor at Texas A&M University in the department of Educational Psychology. He has graduated from the University of Toronto in the area of developmental neuroscience. His research interests focus on the development and training of self-regulation in typical and atypical populations.

Nathan Lai is currently an undergraduate student at the University of Toronto pursuing an Honors Bachelor of Science in Psychology and Neuroscience. His research interests lie in clinical disorders in pediatric populations.

Howell Gotlieb, MA, C. Psych. Associate, is currently Senior Psychological Associate of JVS Toronto Psychological Services. He received his MA from the Department of Applied School-School Psychology program at the Ontario Institute for Studies in Education in the University of Toronto.

Reena Kronitz, PhD, C. Psych., is Chief Psychologist and Senior Director of Programs and Services (Psychological and Career Services) at JVS Toronto. She leads a department of about 20 staff who provide school- and clinic-based psychological assessment and intervention services. She received her BA in Psychology from McGill University and an MA and PhD in Developmental Psychology from York University.

Rosemary Tannock is currently a Senior Scientist with the Neuroscience & Mental Health Research Program in the Research Institute at The Hospital for Sick Children, Toronto, and Professor Emerita at the University of Toronto. Formerly she held the Canada Research Chair in Special Education & Adaptive Technology (2006-2013).

References

Adler

L. A.

Spencer

Faraone

S. V.

Kessler

R. C.

Howes

M. J.

Biederman

Secnik

(2006). Validity of pilot adult ADHD Self-Report Scale (ASRS) to rate adult ADHD symptoms. Annals of Clinical Psychiatry, 18, 145-148.

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

Barkley

R. A.

(2011). Barkley Deficits in Executive Functioning Scale (BDEFS). New York, NY: Guilford Press.

Broadbent

D. E.

Cooper

P. F.

FitzGerald

Parkes

K. R.

(1982). The Cognitive Failures Questionnaire (CFQ) and its correlates. British Journal of Clinical Psychology, 21, 1-16.

CogMed. (2011). CogMed coaching manual. Retrieved from http://cogmed.com/wp-content/uploads/2010/07/Coaching-manual-US-1.0.9.pdf

Duckworth

A. L.

Peterson

Matthews

M. D.

Kelly

D. R.

(2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 9, 1087-1101.

Duckworth

A. L.

Quinn

P. D.

(2009). Development and Validation of the Short Grit Scale (GRIT–S). Journal of Personality Assessment, 91, 166-174.

DuPaul

G. J.

Weyandt

L. L.

O’Dell

S. M.

Varejao

(2009). College students with ADHD current status and future directions. Journal of Attention Disorders, 13, 234-250.

Dvorsky

M. R.

Langberg

J. M.

(2014). Predicting impairment in college students with ADHD the role of executive functions. Journal of Attention Disorders. Advance online publication. doi:10.1177/1087054714548037

10.

Faul

Erdfelder

Buchner

Lang

A. G.

(2009). Statistical power analyses using G* Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160.

11.

Fray

P. J.

Robbins

T. W.

Sahakian

B. J.

(1996). Neuropsychiatric applications of CANTAB. International Journal of Geriatric Psychiatry, 11, 329-336.

12.

Gray

S. A.

Chaban

Martinussen

Goldberg

Gotlieb

Kronitz

. . . Tannock

(2012). Effects of a computerized working memory training program on working memory, attention, and academics in adolescents with severe LD and comorbid ADHD: A randomized controlled trial. Journal of Child Psychology and Psychiatry, 53, 1277-1284.

13.

Green

C. T.

Long

D. L.

Green

Iosif

A. M.

Dixon

J. F.

Miller

M. R.

. . . Schweitzer

J. B.

(2012). Will working memory training generalize to improve off-task behavior in children with attention-deficit/hyperactivity disorder? Neurotherapeutics, 9, 639-648.

14.

Gropper

R. J.

Tannock

(2009). A pilot study of working memory and academic achievement in college students with ADHD. Journal of Attention Disorders, 12, 574-581.

15.

Holmes

Gathercole

S. E.

Dunning

D. L.

(2009). Adaptive training leads to sustained enhancement of poor working memory in children. Developmental Science, 12, F9-F15.

16.

Johnstone

S. J.

Roodenrys

Phillips

Watt

A. J.

Mantz

(2010). A pilot study of combined working memory and inhibition training for children with AD/HD. Attention Deficit and Hyperactivity Disorders, 2, 31-42.

17.

Klingberg

(2010). Training and plasticity of working memory. Trends in Cognitive Sciences, 14, 317-324.

18.

Klingberg

Fernell

Olesen

P. J.

Johnson

Gustafsson

Dahlström

. . . Westerberg

(2005). Computerized training of working memory in children with ADHD—A randomized, controlled trial. Journal of the American Academy of Child & Adolescent Psychiatry, 44, 177-186.

19.

Maruish

M. E.

Bershadsky

Goldstein

(1998). Reliability and validity of the SA–45: Further evidence from a primary care setting. Assessment, 5, 407-420.

20.

Melby-Lervåg

Hulme

(2013). Can working memory training ameliorate ADHD and other learning disorders? A systematic meta-analytic review. The ADHD Report, 21(2), 1-5.

21.

Rapport

M. D.

Orban

S. A.

Kofler

M. J.

Friedman

L. M.

(2013). Do programs designed to train working memory, other executive functions, and attention benefit children with ADHD? A meta-analytic review of cognitive, academic, and behavioral outcomes. Clinical Psychology Review, 33, 1237-1252.

22.

Sheslow

Adams

(2003). Wide range assessment of memory and learning second edition administration and technical manual. Lutz, FL: Psychological Assessment Resources.

23.

Shipstead

Redick

T. S.

Engle

R. W.

(2012). Is working memory training effective? Psychological Bulletin, 138, 628-654.

24.

Sollman

M. J.

Ranseen

J. D.

Berry

D. T.

(2010). Detection of feigned ADHD in college students. Psychological Assessment, 22, 325-335.

25.

Sonuga-Barke

E. J.

Brandeis

Cortese

Daley

Ferrin

Holtmann

. . . Sergeant

(2013). Nonpharmacological interventions for ADHD: Systematic review and meta-analyses of randomized controlled trials of dietary and psychological treatments. American Journal of Psychiatry, 170, 275-289.

26.

Tabachnick

B. G.

Fidell

L. S.

(2007). Experimental designs using ANOVA. Belmont, CA: Thomson/Brooks/Cole.

27.

Vacha-Haase

Thompson

(2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473-481.

28.

Wagner

R. K.

Torgesen

J. K.

Rashotte

C. A.

(1999). Comprehensive test of phonological processing: CTOPP. Austin, TX: Pro-Ed.

29.

Wechsler

(1999). Wechsler Abbreviated Scale of Intelligence. San Antonio, TX: Psychological Corporation.

30.

Wechsler

(2008). Wechsler Adult Intelligence Scale–Fourth Edition (WAIS–IV). San Antonio, TX: NCS Pearson.

31.

Woltering

Liu

Rokeach

Tannock

(2013). Neurophysiological differences in inhibitory control between adults with ADHD and their peers. Neuropsychologia, 51, 1888-1895.

32.

Woodcock

R. W.

McGrew

K. S.

Mather

(2001). Woodcock-Johnson III Tests of Cognitive Abilities. Rolling Meadows, IL: Riverside.