Screening Amazon’s Mechanical Turk for Adults With ADHD

Abstract

Objective: Researchers are increasingly using Amazon’s Mechanical Turk (MTurk; www.mturk.com) to recruit study participants. However, the utility of MTurk for investigations of ADHD in adulthood is unknown. Method: A total of 6,526 MTurk workers (median age range = 26-35 years) completed an online screening survey assessing their diagnostic histories and symptoms of ADHD, as well as relevant demographic correlates and diagnostic comorbidity. Results: The prevalence of MTurk workers who reported being diagnosed as children and as adults, and the percentage of workers whose ADHD appeared to persist from childhood into adulthood, are consistent with those observed in “offline” samples. Relative to MTurk workers diagnosed with ADHD as adults, workers diagnosed with ADHD as children were more likely to be male and without college degrees, as well as less likely to have comorbid depression or anxiety disorders. Conclusion: Amazon’s MTurk holds promise as a recruitment tool for studying adults with ADHD.

Keywords

adult ADHD online sample assessment

Once considered a disorder exclusive to childhood (American Psychiatric Association [APA], 1994), it is now generally accepted that ADHD symptoms and impairment persist into adulthood for many individuals (2.5%-4.4% of adults in the United States; APA, 2013; Kessler et al., 2006). However, researchers have only recently begun to highlight the considerable impairment corollary to ADHD in adulthood, with the bulk of the evidence informed by samples of emerging adults (aged 18-25), particularly those in college. These studies indicate that emerging adults with ADHD report more antisocial behavior, substance abuse, educational and occupational difficulties, hazardous driving, financial mismanagement, and discordant interpersonal relationships than those without ADHD (for review, see Barkley, Murphy, & Fischer, 2008). Still, we know much less about adults with ADHD than children with the disorder. For example, we know very little about how ADHD presents in non-referred, non-college samples or among adults beyond their early- to mid-20s. Non-referred adults with ADHD are important to study because they may differ in important ways from those seeking treatment. It is also necessary to study adults with a wider range of education histories than just students attending 4-year universities given that adults with ADHD often do not go to, let alone finish college (Biederman et al., 2006; Kuriyan et al., 2013). In addition, it is reasonable to suspect that older adults have different occupational, financial, and interpersonal difficulties than 18- to 25-year-olds. To reach these less convenient samples, appropriate strategies for doing so must be identified. One method to engage a demographically heterogeneous sample of adults with ADHD may be to conduct research online using “crowdsourcing” portals, which are web-based interfaces designed to facilitate recruitment of a diverse workforce to complete a specific task (Behrend, Sharek, Meade, & Wiebe, 2011). A popular crowdsourcing website is Amazon’s Mechanical Turk (MTurk; www.mturk.com), which employs hundreds of thousands of adults internationally as “workers” who receive compensation for completing tasks or surveys online (Mason & Suri, 2012). Owing largely to the opportunity to gather large amounts of high-quality data quickly and cheaply from diverse samples (Buhrmester, Kwang, & Gosling, 2011), clinical researchers have relied increasingly on sampling MTurk workers for their research (e.g., Shapiro, Chandler, & Mueller, 2013). The present study sought to investigate the utility of MTurk for studying adults with ADHD by screening a large sample of MTurk workers for symptoms and diagnostic histories of ADHD as well as relevant demographic correlates and diagnostic comorbidities.

MTurk and Psychological Research

There are numerous advantages of conducting research via crowdsourcing websites such as MTurk (Mason & Suri, 2012; Paolacci, Chandler, & Ipeirotis, 2010). First and foremost, data can be collected quickly and at a low cost from large samples (Buhrmester et al., 2011). Researchers have been successful in recruiting hundreds of MTurk workers to complete 30-min surveys for as little as $0.20 per participant (e.g., Gardner, Brown, & Boice, 2012) and others have recruited dozens of participants within hours who are willing to complete similar length surveys for even less money ($0.01-$0.02/subject; Buhrmester et al., 2011). However, workers are still more likely to take surveys offering better compensation (Mason & Suri, 2012). Second, data quality is reported to be high regardless of payment amount. For example, Litman, Robinson, and Rosenzweig (2014) found that MTurk workers with U.S. residence provided data with high internal consistency, and routinely passed attention check questions, whether they made 20 cents per hour or 20 times minimum wage per hour. Third, the MTurk workforce is similar to college/university subject pool samples in terms of gender and racial/ethnicity demographics (i.e., predominantly female and Caucasian; Behrend et al., 2011), but is more diverse in other traits. Behrend and colleagues (2011) found that MTurk samples are older, more likely to be employed, and less well-educated than typical college samples. Regarding education level, many MTurk workers (about 50%) also report that their highest level of education is a 2-year degree or less (Behrend et al., 2011), which is consistent with the median education level in the United States (i.e., some years of college, but no degree; U.S. Census Bureau, 2014). The fact that MTurk workers have diverse educational backgrounds is particularly relevant for studies of adults with ADHD, a group at risk for not matriculating to or beyond college (Biederman et al., 2006; Kuriyan et al., 2013). Fourth, efforts are taken to ensure the relative anonymity of MTurk workers, which may increase the likelihood of honest reporting of sensitive topics, such as the presence or history of mental health symptoms or diagnoses. Fifth, MTurk workers are registered with unique ID codes associated with basic information, which allows researchers to screen for some inclusion criteria (e.g., U.S. residents or workers with particularly reliable work), and exclude participants who have participated in an investigator’s prior MTurk research projects, which is often the case (Chandler, Mueller, & Paolacci, 2014).

Despite the many advantages of MTurk for clinical research, very few studies have assessed rates of psychopathology among MTurk workers. Shapiro et al. (2013) were the first to assess the utility of MTurk for clinical research among U.S. residents. They recruited a sample of 530 adults (M age = 32.6 years) in 2 days for their study, and found that these workers provided quality data (as evidenced by the high internal consistencies, test–retest reliabilities, and criterion-related validity) for the measures they used to assess psychopathology (i.e., Beck Depression Inventory and Beck Anxiety Inventory; Beck, Epstein, Brown, & Steer, 1988; Beck, Ward, Mendelson, Mock, & Erbaugh, 1961). With regard to rates of clinical diagnoses, Shapiro and colleagues (2013) reported that 21% of their sample indicated being previously diagnosed with a mental health disorder, which is consistent with rates of mental health problems among adults in the United States (Substance Abuse and Mental Health Services Administration, 2013). However, these researchers did not report diagnostic rates for specific disorders, which may be useful information for researchers interested in the utility of MTurk for studies of individuals with unique diagnostic histories. Regarding symptom presentations, Shapiro and colleagues (2013) reported that rates of clinically elevated depressive and anxious symptoms were consistent with population estimates (Kessler, Chiu, Demler, & Walters, 2005). Unfortunately, these researchers did not assess for symptoms of externalizing disorders, such as ADHD, in their MTurk sample nor did they explore demographic correlates or rates of comorbidity among workers endorsing diagnostic histories or clinically elevated symptoms.

Since the study by Shapiro and colleagues (2013), a number of researchers have examined clinically relevant concerns among MTurk workers (e.g., Arch, 2014; Bardeen, Fergus, & Orcutt, 2013; Brochu, Pearl, Puhl, & Brownell, 2014; Henshaw, 2014; Lebowitz, Ahn, & Nolen-Hoeksema, 2013; Papa, Lancaster, & Kahler, 2014). However, to our knowledge, only one study has compared clinical profiles among those meeting or not meeting criteria for a specific mental health concern: Victor and Klonsky (2014) found that workers (M age = 30.3 years) endorsing histories of non-suicidal self-injury were at greater risk of other forms of psychopathology, including depression and anxiety, than workers not endorsing such histories. Clearly, there is need for additional studies to examine the utility of MTurk for clinical research on psychopathology, including studies comparing clinical profiles (e.g., demographic correlates, comorbidity) of those with and without diagnostic histories or clinically elevated symptoms of specific mental health disorders. There is also a dearth of evidence on externalizing disorders, namely ADHD, among MTurk workers.

Utility of MTurk for Studying Adults With ADHD

Although the literature is replete with studies on the demographic correlates and diagnostic comorbidities of children and adolescents with ADHD (Barkley, 2006), far less is known regarding the same characteristics of adults with ADHD. As mentioned above, the dearth of studies evaluating ADHD in adults who are non-referred, not in college, and/or over the age of 25 is particularly notable. Screening for relevant correlates and comorbidities among demographically heterogeneous MTurk workers reporting or denying ADHD diagnoses and/or elevated current symptoms could address these limitations.

An emerging pattern of results from early studies of ADHD in adulthood that needs further investigation is that adults diagnosed with ADHD as children or adolescents have different demographic and comorbidity presentations as those diagnosed with ADHD as adults. First, it appears that individuals diagnosed with ADHD as children are more likely to be male than individuals diagnosed with ADHD as adults. For example, in their follow-up studies of adults diagnosed with ADHD as children and as adults, Barkley and colleagues (2008) demonstrated that the prevalence of men far outnumbered the prevalence of women in their sample of adults diagnosed with ADHD as children, regardless of whether they continued to meet diagnostic criteria for ADHD as adults (84% men) or not (88% men). In comparison, the gender ratio was weighted less in favor of a male majority in their sample of individuals diagnosed with ADHD as adults (68% men). Additional studies of individuals diagnosed with ADHD as adults conducted by Kessler and colleagues (2006; 62% men) and Wilens and colleagues (2009; 49% men) also had more gender-balanced samples than those including adults diagnosed with ADHD as children. Although these findings appear to be convincing, further research is needed to compare rates of childhood and adulthood ADHD diagnoses among men and women because none of the prior studies actually statistically compared gender ratios across diagnostic histories.

Second, early evidence indicates that adults diagnosed with ADHD as children attain lower levels of education than those diagnosed with ADHD as adults. As reported by Barkley and colleagues (2008), the highest level of education (in years) appeared to be lower in their sample of adults diagnosed with ADHD as children, regardless of diagnostic persistence (M = 12.2, SD = 2.2) or not (M = 12.8, SD = 2.1), than the highest level of education in their sample of individuals diagnosed with ADHD as adults (M = 14.2, SD = 2.2). Again, the highest level of education was not compared statistically between adults diagnosed with ADHD as children/adolescents versus as adults. Thus, an investigation is warranted to clarify whether education level is indeed associated with the timing of diagnosis.

Third, another characteristic that appears to differ between those diagnosed with ADHD as children versus those diagnosed as adults is comorbid diagnoses. For example, Barkley and colleagues (2008) reported that between 2.6% and 13.2% of adults diagnosed with ADHD as children had been diagnosed with a mood disorder (depending on whether they did or did not meet ADHD diagnostic criteria as adults). Klein and colleagues (2012) reported that adults diagnosed with ADHD as children had a similar rate of mood disorder diagnoses (8.9%). Conversely, mood disorders tend to be more common in samples of individuals diagnosed with ADHD as adults. For example, Kessler and colleagues (2006) reported that 38.3% of their sample of adults with ADHD reported having a mood disorder diagnosis in the past 12 months. The same pattern appears to hold with anxiety disorders as well. Specifically, adults diagnosed with ADHD as children/adolescents reported lower rates of anxiety disorders as adults (7.7%-12.6% met criteria for any anxiety disorder; Barkley et al., 2008; Klein et al., 2012) than individuals diagnosed with ADHD as adults (47.1% had been diagnosed with any anxiety disorder in past year; Kessler et al., 2006). Investigations with demographically heterogeneous samples are needed to replicate these findings, to extend them by statistically comparing rates of diagnostic comorbidities between adults diagnosed with ADHD as children/adolescents and those diagnosed as adults, and to examine whether correlate and comorbidity profiles vary depending on the level of current ADHD symptoms (i.e., reporting clinically significant current ADHD symptoms vs. non-significant current ADHD symptoms).

Study Goals

The primary aim of this study was to screen MTurk workers, a demographically heterogeneous online population, for ADHD diagnostic histories, symptoms, relevant demographic correlates, and diagnostic comorbidities. We recruited over 6,500 MTurk workers for this investigation, and screened them for ADHD diagnoses and current ADHD symptoms using an evidence-based screening tool (Kessler, Adler, et al., 2005). We also screened the same MTurk workers for relevant demographic correlates (i.e., gender and level of education) and for comorbid diagnoses (i.e., depression and anxiety disorders).

We hypothesized that rates of ADHD diagnoses in our MTurk sample would be consistent with current estimates for the national prevalence of ADHD in childhood (5%-11%; APA, 2013; Visser et al., 2014), in adulthood (2.5%-4.4%; APA, 2013; Kessler et al., 2006), and across the life span (8.1%; Kessler, Berglund, et al., 2005). We also expected that rates of ADHD persistence from childhood (as determined by positive self-reported diagnostic histories) to adulthood (as determined by self-reported current symptoms exceeding clinical cutoff) in our MTurk sample would also be comparable with prior research (44%; Barkley et al., 2008). Based on evidence suggesting that adults diagnosed with ADHD as children view themselves as having few symptoms as adults (Loney, Ledolter, Kramer, & Volpe, 2007), we anticipated that adults diagnosed with ADHD as children would report fewer current ADHD symptoms than individuals who were diagnosed with ADHD as adults. Consistent with prior research (e.g., Barkley et al., 2008; Kessler et al., 2006; Klein et al., 2012; Wilens et al., 2009), we hypothesized that MTurk workers reporting childhood ADHD diagnoses (regardless of current symptoms) would be more likely than workers diagnosed with ADHD as adults to be men, to have lower levels of education, and fewer histories of depressive or anxiety disorder diagnoses.

Method

Participants

The sample for the current study was recruited through Amazon’s MTurk (https://www.mturk.com). Male and female MTurk workers who were at least 18 years of age and U.S. residents were eligible for this study. Data were collected from two sample cohorts. The first cohort of 2,753 eligible workers was recruited in 16 days from January to February, 2014. The second cohort of 3,902 eligible workers was recruited in 36 days from October to November, 2014. A qualifier was required so that individuals participating in the first cohort were ineligible to participate in the second cohort. Of the 6,655 participants who were successfully recruited, 129 (1.9% of sample) were excluded for having missing data on at least one primary measure (ADHD diagnosis or current ADHD symptoms). Among the eligible participants with complete data (n = 6,526), the sample was nearly half (50.1%) male and nearly half (49.9%) female. Workers were asked to indicate their age by selecting one of the following year ranges: 18 to 25 years (33.8% endorsed), 26 to 35 years (41.1% endorsed), 36 to 45 years (13.9% endorsed), 46 to 55 years (7.2% endorsed), and 55 years or older (4.0% endorsed). Our sample included workers with a variety of different education backgrounds: not completing high school (0.7%), high school diploma or its equivalent (12.7%), some college but no degree (30.2%), 2-year degree (e.g., AA, AS; 10.7%), bachelor’s degree (35.0%), and a master’s degree or higher (10.8%). Nearly all participants (96.7%) reported that English was their native language.

Procedures

An advertisement was posted on the MTurk worker website highlighting the opportunity to participate in an academic survey. The population of interest was not identified so as to not bias workers responses. Mturk workers who were interested in taking this survey were directed to click on a weblink that would direct them to our secure online data collection portal (i.e., Research Electronic Data Capture [REDCap]; Harris et al., 2009). Once participants completed the brief survey (after approximately 2 min), participants were provided with a code to enter into the MTurk system to indicate that they completed the survey. The researchers were then able to examine the code provided by individual workers and then “approve” or “reject” the work based on the accuracy of the code. Participants were compensated $0.25 for providing the correct code, which is typical pay for MTurk workers who complete similar brief tasks (Buhrmester et al., 2011). Prior to completing this brief survey, participants were required to acknowledge informed consent digitally. This study was approved by the University Social-Behavioral Institutional Review Board.

Measures

Participants completed the demographic questionnaire and the following measures as part of a single online survey.

Diagnostic histories

Participants were asked whether they had ever been diagnosed with ADHD. If they endorsed having a history of ADHD, they were asked to indicate what age range they were diagnosed (4 years or younger, 5-12 years, 13-17 years, 18-25 years, 26-35 years, 36-45 years, 46 years or older). Diagnostic histories have been assessed using similar procedures in prior epidemiological studies of ADHD (Biederman et al., 2006; Visser et al., 2014). For ease of presentation, we will only present whether they reported being diagnosed as a child (i.e., by age 17 years) or as an adult (i.e., 18 years of age or older). Parallel questions also assessed whether participants were ever diagnosed with depression or anxiety disorders and, if so, during what age range they were diagnosed.

ADHD symptoms

Current symptoms of ADHD were screened using the six-item World Health Organization Adult ADHD Self-Report Scale–Screener (ASRS-S; Kessler, Adler, et al., 2005). The ASRS-S assesses for four symptoms of inattention and two symptoms of hyperactivity/impulsivity listed as criteria for ADHD in the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; APA, 1994). Respondents are asked to indicate how often they exhibit symptoms (0 = never, 1 = rarely, 2 = sometimes, 3 = often, and 4 = very often). Kessler, Adler, and colleagues (2005) reported that the ASRS-S had adequate sensitivity (68.7%), excellent specificity (99.5%), and excellent classification concordance with a clinical interview (97.9%). In a replication study, the same research group found that the ASRS-S had marginal (α = .63) to acceptable (α range .70-.72) internal consistency as well as marginal (r = .58) to acceptable (r range = .74-.77) test–retest reliability (Kessler et al., 2007). Kessler et al. (2007) demonstrated that the ASRS-S should be scored on a 0 to 24 scale, as it most strongly aligns with rigorous clinical diagnoses (area under the curve [AUC] = .87). The same authors found an optimal clinical cutoff of 14 or greater (0-13 vs. 14-24) to predict having a diagnosis (AUC = .79), but recommended classifying general population samples using a four-stratum approach (0-9; 10-13; 14-17; 18-24; AUC = .90; Kessler et al., 2007). Internal consistency of the ASRS-S within this sample was adequate (α = .74). ASRS-S total scores were normally distributed in this sample (M = 9.93, SD = 4.10) and did not differ statistically between males and females.

Results

ADHD Diagnostic Histories and Current Symptoms

Of the 6,526 MTurk workers in this sample, 10.5% (n = 684) endorsed having been diagnosed with ADHD in their lifetime, with 6.0% (n = 392) reporting a childhood diagnostic history and 4.5% (n = 292) reporting that they were diagnosed as adults. About one fifth of our sample (18.5%; n = 1,208) had ASRS-S total scores exceeding the recommended clinical cutoff for this measure (i.e., ASRS-S raw score ≥ 14), including 164 (2.5%) who were diagnosed with ADHD as a child and 166 (2.6%) who were diagnosed with ADHD as an adult. Table 1 presents rates of current ADHD symptoms reported by MTurk workers with and without ADHD diagnostic histories.

Table 1.

ADHD Symptoms Reported by MTurk Workers With and Without ADHD Diagnoses.

ASRS-STotal scores	Never diagnosed with ADHD (n = 5,842)	Diagnosed with ADHD as child (n = 392)	Diagnosed with ADHD as adult (n = 292)
0-9	50.84% (n = 2,970)	21.68% (n = 85)	11.99% (n = −35)
10-13	34.13% (n = 1,994)	36.48% (n = 143)	31.16% (n = 91)
14-17	12.29% (n = 718)	27.81% (n = 109)	33.90% (n = 99)
18-24	2.74% (n = 160)	14.03% (n = 55)	22.95% (n = 67)
≤13	84.97% (n = 4,964)	58.16% (n = 228)	43.15% (n = 126)
≥14	15.03% (n = 878)	41.84% (n = 164)	56.85% (n = 166)

Note: ASRS-S total score ranges are presented in four strata as recommended by Kessler and colleagues (2007). Higher scores indicate greater ADHD symptoms. An ASRS-S total score ≥ 14 is considered the optimal clinical cutoff for adults at risk for having ADHD. Diagnosed with ADHD as child = reporting diagnosis by age 17; Diagnosed with ADHD as adult = reporting diagnosis at/after age 18; MTurk = Mechanical Turk; ASRS-S = ADHD Self-Report Scale–Screener.

MTurk workers reporting that they were diagnosed with ADHD as children/adolescents were more likely to report current ADHD symptoms exceeding clinical cutoff (41.84%) than workers who were never diagnosed with ADHD (15.03%), χ²(6234) = 189.64, p < .001; odds ratio (OR) = 4.07, 95% confidence interval (CI) [3.29, 5.03]. Similarly, workers who reported being diagnosed with ADHD as adults were also more likely to report current ADHD symptoms exceeding clinical cutoff (56.85%) than those who were never diagnosed with ADHD (15.03%), χ²(6134) = 344.38, p < .001; OR = 7.45, 95% CI [5.85, 9.49]. Among those reporting a prior ADHD diagnosis, MTurk workers diagnosed as adults were more likely to report current ADHD symptoms exceeding clinical cutoff (56.85%) than workers diagnosed as children/adolescents (41.84%), χ²(684) = 15.10, p < .001; OR = 1.83, 95% CI [1.35, 2.49].

Demographic Correlates of ADHD Diagnoses and Current Symptoms

Table 2 presents demographic characteristics of MTurk workers split by whether they reported prior ADHD diagnoses. Regarding gender comparisons, MTurk workers diagnosed with ADHD as children or adolescents were more likely to be men (63.01%) than workers never diagnosed with ADHD (49.37%), χ²(6234) = 27.35, p < .001; OR = 1.75, 95% CI [1.41, 2.16], and workers diagnosed with ADHD as adults (47.60%), χ²(684) = 16.16, p < .001; OR = 1.88, 95% CI [1.38, 2.55]. Regarding highest education level, MTurk workers who reported being diagnosed with ADHD as children or adolescents were less likely to have received a college degree (2-year degrees or higher; 45.41%) than those who never were diagnosed with ADHD (56.97%), χ²(6235) = 19.97, p < .001; OR = .63, 95% CI [0.51, 0.77], and those who reported being diagnosed with ADHD as adults (60.27%), χ²(684) = 14.81, p < .001; OR = .55, 95% CI [0.40, 0.75].

Table 2.

Demographic Characteristics of MTurk Workers by ADHD Diagnostic History and Current Symptoms.

	Never diagnosed with ADHD (n = 5,842)	Diagnosed with ADHD as child (n = 392)	Diagnosed with ADHD as adult (n = 292)
Gender
Male	49.37% (n = 2,884)	63.01% (n = 247)	47.60% (n = 139)
Female	50.63% (n = 2,958)	36.99% (n = 145)	52.40% (n = 153)
Education
Did not complete high school	0.74% (n = 43)	0.77% (n = 3)	0.34% (n = 1)
High school diploma or equivalent	12.63% (n = 738)	17.35% (n = 68)	7.53% (n = 22)
Some college but no degree	29.66% (n = 1,733)	36.48% (n = 143)	31.85% (n = 93)
Associate’s degree	10.48% (n = 612)	11.48% (n = 45)	13.70% (n = 40)
Bachelor’s degree	35.72% (n = 2,078)	26.28% (n = 103)	34.59% (n = 101)
Master’s degree or higher	10.94% (n = 639)	7.65% (n = 30)	11.99% (n = 35)
<Associate’s degree	43.03% (n = 2,514)	54.59% (n = 214)	39.73% (n = 116)
≥Associate’s degree	56.97% (n = 3,329)	45.41% (n = 178)	60.27% (n = 176)

Note. Diagnosed with ADHD as child = reporting diagnosis by age 17; Diagnosed with ADHD as adult = reporting diagnosis at/after age 18; MTurk = Mechanical Turk.

Comorbidity

Of participants in the full sample, 31.61% reported having been diagnosed with a depressive disorder and 24.59% reported they were previously diagnosed with an anxiety disorder. Table 3 presents prevalence of depressive and anxiety disorder diagnoses among MTurk workers split by whether they reported ever being diagnosed with ADHD.

Table 3.

Depressive and Anxiety Disorder Diagnoses by ADHD Diagnostic History and Current Symptoms.

	Never diagnosed with ADHD (n = 5,838)	Diagnosed with ADHD as child (n = 392)	Diagnosed with ADHD as adult (n = 292)
Diagnosed with depressive disorder	28.79% (n = 1,681)	48.21% (n = 189)	66.55% (n = 193)
Diagnosed as child	10.34% (n = 604)	33.16% (n = 130)	18.62% (n = 54)
Diagnosed as adult	18.45% (n = 1,077)	15.05% (n = 59)	47.93% (n = 139)
Diagnosed with anxiety disorder	22.27% (n = 1,300)	39.54% (n = 155)	51.37% (n = 150)
Diagnosed as child	5.40% (n = 315)	21.43% (n = 84)	7.19% (n = 21)
Diagnosed adult	16.87% (n = 985)	18.11% (n = 71)	44.18% (n = 129)

Note. Diagnosed as child = reporting diagnosis by age 17; Diagnosed as adult = reporting diagnosis at/after age 18.

MTurk workers who reported being diagnosed with ADHD as children/adolescents were more likely than those who were never diagnosed with ADHD to endorse histories of depressive (48.21% vs. 28.79%), χ²(6230) = 65.95, p < .001; OR = 2.30, 95% CI [1.87, 2.83], and anxiety disorder diagnoses (39.54% vs. 22.27%), χ²(6230) = 61.23, p < .001; OR = 2.28, 95% CI [1.85, 2.82]. MTurk workers who reported that they were diagnosed with ADHD in childhood/adolescence were also more likely than those who were never diagnosed with ADHD to report having been diagnosed with a depressive disorder in childhood (33.16% vs. 10.34%), χ²(6230) = 184.00, p < .001; OR = 4.30, 95% CI [3.43, 5.39]. There was no difference between workers diagnosed with ADHD as children and those without ADHD histories in their rates of being diagnosed with a depressive disorder as an adult (15.05% vs. 18.45%), χ²(6230) = 2.84, p = .092; OR = 0.78, 95% CI [0.59, 1.04]. Following the same pattern, MTurk workers who reported being diagnosed with ADHD in childhood were more likely than those who were never diagnosed with ADHD to report histories of anxiety disorder diagnoses in childhood (21.43% vs. 5.4%), χ²(6230) = 157.52, p < .001; OR = 4.78, 95% CI [3.66, 6.24], but not in adulthood (18.11% vs. 16.87%), χ²(6230) = 0.40, p = .527; OR = 1.09, 95% CI [0.84, 1.42].

Regardless of current ADHD symptoms, MTurk workers reporting that they were diagnosed with ADHD as adults were more likely than those diagnosed with ADHD in childhood to report having histories of depressive (66.55% vs. 48.21%), χ²(684) = 21.70, p < .001; OR = 2.09, 95% CI [1.53-2.86], and anxiety disorder diagnoses (51.37% vs. 39.54%), χ²(684) = 9.48, p = .002; OR = 1.62, 95% CI [1.19, 2.19]. More specifically, MTurk workers who reported that they were diagnosed with ADHD as adults were more likely than those diagnosed with ADHD as children to report having been diagnosed with a depressive disorder as adults (47.93% vs. 15.05%), χ²(684) = 86.21, p < .001; OR = 5.13, 95% CI [3.58, 7.35]. Conversely, MTurk workers who reported that they were diagnosed with ADHD as adults were less likely than those diagnosed with ADHD as children to report having been diagnosed with a depressive disorder as a child/adolescent (18.62% vs. 33.16%), χ²(684) = 18.31, p < .001; OR = .46, 95% CI [0.32, 0.66]. Following the same pattern, MTurk workers diagnosed with ADHD as adults were more likely than those diagnosed with ADHD as children/adolescents to report being diagnosed with an anxiety disorder as adults (44.18% vs. 18.11%), χ²(684) = 54.95, p < .001; OR = 3.58, 95% CI [2.53, 5.06], but were less likely than those diagnosed with ADHD as children/adolescents to be diagnosed with an anxiety disorder in childhood/adolescence (7.19% vs. 21.43%), χ²(684) = 26.10, p < .001; OR = 0.28, 95% CI [0.17, 0.47].

Discussion

In this study, we screened over 6,500 MTurk workers for ADHD diagnostic histories and current ADHD symptoms as well as for relevant demographic correlates (i.e., gender, education level) and diagnostic comorbidities (i.e., anxiety, depression). Based on data we collected from the workers regarding their ADHD diagnostic histories and current symptoms, relevant demographic correlates, and rates of comorbid conditions, it appears as though MTurk may be a valuable source of data for ADHD researchers. First, 6% of our MTurk sample reported being diagnosed with ADHD by age 17. As hypothesized, this rate of ADHD diagnoses in children and adolescents is consistent with prevalence rates endorsed by Diagnostic and Statistical Manual of Mental Disorders ([5th ed.; DSM-V], 5%; APA, 2013) and the Centers for Disease Control and Prevention (11%; Visser et al., 2014). Similarly, 2.5% of the workers endorsed meeting study criteria for ADHD in adulthood (i.e., endorsing childhood history and exceeding clinical cutoff for self-reported current symptoms). Again, the prevalence of adults with ADHD in this sample is comparable with those indicated by prior research (2.5%-4.4%; APA, 2013; Kessler et al., 2006). Finally, 10.5% of our sample reported being diagnosed with ADHD as a child or as an adult, which is fairly consistent with life span prevalence rates reported previously (8.1%; Kessler, Chiu, et al., 2005). Thus, rates of ADHD diagnoses reported by our “online” national sample appear to be equivalent to national samples collected “offline.”

Second, about two fifths (42%) of the MTurk workers in this study who reported being diagnosed with ADHD as children or adolescents endorsed current ADHD symptoms above clinical cutoffs as adults. The prevalence of ADHD persisting from childhood to adulthood among workers in this sample was strikingly similar to the rate of clinically significant symptoms of ADHD self-reported by the 44% of Barkley and colleagues’ (2008) sample diagnosed with ADHD as children. Again, the rate of self-reported ADHD persistence in our online, MTurk sample was nearly identical to the rate of self-reported persistence in Barkley and colleagues’ (2008) offline sample.

Third, results indicate that the specificity and sensitivity of the ASRS-S clinical cutoff within our sample of MTurk workers was comparable with findings within the offline sample used by Kessler and colleagues (2007). With regard to specificity, 15% of the workers who reported never being diagnosed with ADHD endorsed symptoms meeting or exceeding the ASRS-S clinical cutoff. This is slightly more than the percentage observed by Kessler et al. (2007), who found that 6% of those in their sample who did not have ADHD histories reported symptoms greater than the same clinical cutoff used within this study. With regard to sensitivity, 57% of the MTurk workers who were diagnosed with ADHD as adults self-reported symptoms at or above the clinical cutoff. This figure was slightly less than what Kessler and colleagues (2007) found in their offline sample of adults with ADHD (65%). Sensitivity and specificity results may not have been identical between studies owing, in part, to our sample relying on self-reported diagnoses instead of diagnoses resulting from clinical interviews, as Kessler and colleagues used in their study. Nonetheless, despite relying entirely on self-reports, the fact that the ASRS-S was found to be only slightly less specific and sensitive with the MTurk workers than it was within Kessler et al.’s offline sample highlights the potential utility of using the ASRS-S to screen in workers with ADHD for future studies.

Fourth, MTurk workers diagnosed with ADHD as children or adolescents presented with different symptom, demographic, and comorbidity profiles than those diagnosed with ADHD as adults. MTurk workers who reported being diagnosed with ADHD as children were significantly less likely than workers who were diagnosed with ADHD as adults to report current symptoms in excess of clinical cutoffs. It may be that many of the adults diagnosed with ADHD as children were treated successfully and, as a result, their symptoms resolved. Then again, there is reason to suspect that adults diagnosed with ADHD in childhood have a tendency to under-report symptoms relative to the perceptions of others (e.g., Loney et al., 2007). This tendency, “positive illusory bias,” is a well-known phenomenon in children with ADHD (Owens, Goldfine, Evangelista, Hoza, & Kaiser, 2007). Preliminary evidence indicates that this pattern of inaccurate self-appraisals may extend into at least young adulthood (e.g., Knouse, Bagwell, Barkley, & Murphy, 2005; Loney et al., 2007) and wane thereafter as older adults with ADHD may eventually become more able or willing to recognize their symptoms (Barkley et al., 2008). The potential for adults with childhood histories of ADHD to under-report their own symptoms highlights the importance of gathering collateral reports with this group, and also underscores the need for researchers to examine the persistence of positive illusory bias into adulthood among individuals with ADHD.

As hypothesized, ORs indicated that MTurk workers with childhood histories of ADHD were nearly twice as likely to be male as those diagnosed with ADHD as adults. These data are consistent with the well-known pattern where ADHD diagnoses are more common among boys than girls in childhood but are less weighted toward males in adulthood (APA, 2013; Barkley et al., 2008; Kessler et al., 2006). However, it is notable that the gender ratio detected herein is slightly weighted toward females among those diagnosed with ADHD as adults, which is consistent with the ratio reported by Wilens and colleagues (2009), though less weighted toward females than some treatment-seeking samples (e.g., Solanto et al., 2010). It may be that women are more willing than men to come forward and endorse having ADHD as adults. For researchers interested in studying adult women with ADHD, MTurk may be an attractive option for participant recruitment and data collection.

MTurk workers reporting childhood ADHD diagnoses were about half as likely to have received a 2-year college degree (or better) as workers diagnosed with ADHD as adults. In fact, there was no difference in the likelihood of having received a 2-year degree or better among workers without ADHD histories compared with those diagnosed with ADHD as adults. This pattern of results is again consistent with those reported by Barkley and colleagues (2008), who highlighted that years of education were lower for adults diagnosed with ADHD in childhood than those diagnosed as adults, but inconsistent in that we did not observe a difference in degree attainment between workers without ADHD histories and those diagnosed with ADHD as adults. This inconsistency may have been the result of our MTurk sample being somewhat more educated than typical ADHD clinical samples. Nonetheless, these data re-affirm the need for researchers studying individuals diagnosed with ADHD as children in college/university samples to focus on students who are in their first or second years, as this might be the best window to collect data from this population. Realistically, our findings indicate that a substantial percentage of adults diagnosed with ADHD as children are likely missed by researchers focusing exclusively on studying students attending 4-year colleges/universities. It is recommended that researchers studying emerging adults with ADHD, especially those who were diagnosed in childhood, consider sampling beyond the confines of their campus, and attend to collecting data from students attending GED courses, vocational classes, or community colleges as well as national online populations, like MTurk workers.

Although MTurk workers diagnosed with ADHD as children or adolescents were significantly more likely to report being diagnosed with a depressive or anxiety disorder than workers never diagnosed with ADHD, they were significantly less likely to report being diagnosed with depressive or anxiety disorders as workers diagnosed with ADHD as adults. Similar comorbidity patterns were reported by Barkley and colleagues (2008). However, the present study was the first to statistically compare rates of depressive and anxiety disorder diagnoses between adults diagnosed with ADHD as children versus as adults. Notably, rates of ever being diagnosed with comorbid depressive (66.1%) or anxiety (51.4%) disorders reported by MTurk workers diagnosed with ADHD as adults were greater than those reported by Kessler and colleagues (2006), who found that approximately 38% and 47% of those diagnosed with ADHD as adults had histories of any depressive disorder or any anxiety disorder, respectively. There is precedent for above average rates of clinically elevated anxiety (especially social anxiety), but not depression, to be a concern in MTurk workers (Shapiro et al., 2013). Additional studies are needed to evaluate whether rates of depressive and anxiety disorders are indeed higher among MTurk workers, including those with ADHD diagnostic histories, than among offline samples. Future research of ADHD in adults via MTurk should also broaden the assessment of comorbid conditions to others that are commonly found in this population (e.g., substance use disorders, antisocial personality disorder). Finally, we also found that MTurk workers diagnosed with ADHD as children/adolescents primarily reported being diagnosed with depressive or anxiety disorders as children or adolescents whereas those diagnosed with ADHD as adults tended to report being diagnosed with depressive or anxiety disorders as adults. As neither Barkley and colleagues (2008) nor Kessler and colleagues (2006) reported the age of onset for comorbid diagnoses in their studies, investigations are needed to replicate our findings regarding age-of-onset differences between those diagnosed with ADHD as children and as adults.

The present study has several strengths, including a large sample size, evaluating for diagnostic histories in childhood and adulthood, as well as assessing for clinically elevated ADHD symptoms using a psychometrically sound screening tool. However, this study also has several weaknesses. First, a limitation of this study is our reliance on self-reported diagnoses and symptom ratings. We are unable to confirm the veracity of diagnoses or current symptoms, the quality of the assessment leading to diagnoses, or the accuracy of the age of diagnosis reported by MTurk workers in this study. One solution for future studies may be to collect ratings from collateral informants (e.g., romantic partner/spouse, co-worker). As asserted by Barkley and colleagues (2008), many additional adults with childhood ADHD (up to 30%) may meet criteria as adults when including collateral perspectives than when relying solely on self-report data. Research is needed to explore whether a similar percentage of MTurk workers reporting childhood ADHD diagnostic histories, but non-significant current symptoms, would meet criteria for ADHD based on ratings from significant others. In addition, the validity of self-report ratings in this population could also be examined by following up MTurk workers who consent to complete self-report ratings and then to undergo a diagnostic interview via phone/skype. Second, treatment histories and additional demographic correlates of ADHD (e.g., race/ethnicity, income/employment status, marital status) were not assessed in this study. Current symptom ratings may have been affected by not controlling for whether participants were taking stimulant medication or receiving other forms of therapy. Moreover, symptom ratings may have differed across relevant demographic traits, and this information could have been useful toward clarifying whether MTurk is an equally useful recruitment tool across different demographic groups of adults with ADHD. Third, although there is a deep pool of potential research participants in MTurk, the potential for researchers to enroll the same individuals across multiple MTurk studies is cause for concern (Chandler et al., 2014). This can be remedied to some extent within an individual investigator’s program of research by restricting enrollment in their future studies to those who have not participated in their prior studies (via qualifiers within MTurk capabilities; we did this when recruiting the second cohort in this study). However, this procedure cannot be used to restrict MTurk workers from participating in multiple concurrent studies across several investigators. Thus, in the event that a number of investigators begin using MTurk as an avenue to advance their research on adults with ADHD, our knowledge of ADHD in adulthood may become limited to a relatively small number of workers with the disorder in a single population (i.e., MTurk) that may not generalize to the larger population.

Taken together, MTurk appears to be a promising avenue to pursue research examining adults with ADHD. Diagnostic prevalence, symptom profiles, demographic correlates, and internalizing comorbidities appear to be consistent with studies of “offline” populations of adults with ADHD, including those with symptoms persisting from childhood. Moreover, the opportunity to gather large quantities of clinically relevant data efficiently and inexpensively without sacrificing quality is attractive. One could envision, for example, using MTurk and other crowdsourcing portals to conduct large-scale longitudinal or online intervention studies (e.g., computerized interventions or tele-health interventions) for a fraction of the cost and time typical in parallel research with offline samples. In the present funding climate, MTurk and related online sampling interfaces could provide opportunities for many to conduct large-scale studies without significant grant support.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Brian T. Wymbs, PhD, is an assistant professor of Psychology at Ohio University. His primary research interests focus on examining links between ADHD and romantic relationship difficulties. He has examined rates of interparental discord and divorce in families including children with ADHD, as well as rates of intimate partner violence and potential explanatory factors among young adults with ADHD.

Anne E. Dawson, MS, is a graduate student in clinical psychology at Ohio University. Her research focuses on the negative sequelae of ADHD across development, specifically the impact of ADHD in couples, particularly parents, and in emerging adulthood.

References

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Arch

J. J.

(2014). Cognitive behavior therapy and pharmacotherapy for anxiety: Treatment preferences and credibility among pregnant and non-pregnant women. Behavior Research and Therapy, 52, 53-60.

Bardeen

J. R.

Fergus

T. A.

Orcutt

H. K.

(2013). Experiential avoidance as a moderator of the relationship between anxiety sensitivity and perceived stress. Behavior Therapy, 44, 459-469.

Barkley

R. A.

(2006). Attention-deficit hyperactivity disorder (3rd ed.). New York, NY: Guilford Press.

Barkley

R. A.

Murphy

K. R.

Fischer

(2008). ADHD in adults: What the science says. New York, NY: Guilford Press.

Beck

A. T.

Epstein

Brown

Steer

R. A.

(1988). An inventory for measuring clinical anxiety: Psychometric properties. Journal of Consulting and Clinical Psychology, 56, 893-897.

Beck

A. T.

Ward

C. H.

Mendelson

Mock

Erbaugh

(1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561-571.

Behrend

T. S.

Sharek

D. J.

Meade

A. W.

Wiebe

E. N.

(2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800-813.

10.

Biederman

Faraone

S. V.

Spencer

T. J.

Mick

Monuteaux

M. C.

Aleardi

(2006). Functional impairments in adults with self-reports of diagnosed ADHD: A controlled study of 1001 adults in the community. Journal of Clinical Psychiatry, 67, 524-540.

11.

Brochu

P. M.

Pearl

R. L.

Puhl

R. M.

Brownell

K. D.

(2014). Do media portrayals of obesity influence support for weight-related medical policy? Health Psychology, 33, 197-200.

12.

Buhrmester

Kwang

Gosling

S. D.

(2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.

13.

Chandler

Mueller

Paolacci

(2014). Nonnaivete among Amazon Mechanical Turk workers: Consequences and solutions for behavior researchers. Behavior Research Methods, 46, 112-130.

14.

Gardner

R. M.

Brown

D. L.

Boice

(2012). Using Amazon’s Mechanical Turk website to measure accuracy of body size estimation and body dissatisfaction. Body Image, 9, 532-534.

15.

Harris

P. A.

Taylor

Thielke

Payne

Gonzalez

Conde

J. G.

(2009). Research electronic data capture (REDCap): A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42, 377-381.

16.

Henshaw

E. J.

(2014). Too sick, not sick enough? Effects of treatment type and timing on depression stigma. Journal of Nervous and Mental Disease, 202, 292-299.

17.

Kessler

R. C.

Adler

Ames

Demler

Faraone

Hiripi

. . . Walters

E. E.

(2005). The World Health Organization adult ADHD Self-Report Scale (ASRS): A short screening scale for use in the general population. Psychological Medicine, 35, 245-256.

18.

Kessler

R. C.

Adler

Barkley

Biederman

Conners

C. K.

Demler

. . . Zaslavsky

A. M.

(2006). The prevalence and correlates of adult ADHD in the United States: Results from the National Comorbidity Survey replication. The American Journal of Psychiatry, 163, 716-723.

19.

Kessler

R. C.

Adler

Gruber

M. J.

Sarawate

C. A.

Spencer

Van Brunt

D. L.

(2007). Validity of the World Health Organization adult ADHD Self-Report Scale (ASRS) screener in a representative sample of health plan members. International Journal of Methods in Psychiatric Research, 16, 52-65.

20.

Kessler

R. C.

Berglund

Demler

Jin

Merikangas

K. R.

Walters

E. E.

(2005). Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey replication. Archives of General Psychiatry, 62, 593-602.

21.

Kessler

R. C.

Chiu

W. T.

Demler

Walters

E. E.

(2005). Prevalence, severity, and comorbidity of twelve-month DSM-IV disorders in the National Comorbidity Survey Replication (NCSR). Archives of General Psychiatry, 62, 617-627.

22.

Klein

R. G.

Mannuzza

Olazagasti

M. A. R.

Roizen

Hutchison

J. A.

Lashua

E. C.

Castellanos

(2012). Clinical and functional outcome of childhood attention-deficit/hyperactive disorder 33 years later. Archives of General Psychiatry, 69, 1295-1303.

23.

Knouse

L. E.

Bagwell

C. L.

Barkley

R. A.

Murphy

K. R.

(2005). Accuracy of self-evaluation in adults with attention-deficit hyperactivity disorder. Journal of Attention Disorders, 8, 221-234.

24.

Kuriyan

A. B.

Pelham

W. E.

Molina

B. S. G.

Waschbusch

D. A.

Gnagy

E. M.

Sibley

M. H.

. . . Kent

K. M.

(2013). Young adult educational and vocational outcomes of children diagnosed with ADHD. Journal of Abnormal Child Psychology, 41, 27-41.

25.

Lebowitz

M. S.

Ahn

Nolen-Hoeksema

(2013). Fixable or fate? Perceptions of the biology of depression. Journal of Consulting and Clinical Psychology, 81, 518-527.

26.

Litman

Robinson

Rosenzweig

(2014). The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behavior Research Methods, 47, 519-528.

27.

Loney

Ledolter

Kramer

J. R.

Volpe

R. J.

(2007). Retrospective ratings of ADHD symptoms made at young adulthood by clinic-referred boys with ADHD-related problems, their brothers without ADHD, and control participants. Psychological Assessment, 19, 269-280.

28.

Mason

Suri

(2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1-23.

29.

Owens

J. S.

Goldfine

M. E.

Evangelista

N. M.

Hoza

Kaiser

N. M.

(2007). A critical review of self-perceptions and the positive illusory bias in children with ADHD. Clinical Child and Family Psychology Review, 10, 335-351.

30.

Paolacci

Chandler

Ipeirotis

P. G.

(2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.

31.

Papa

Lancaster

N. G.

Kahler

(2014). Commonalities in grief responding across bereavement and non-bereavement losses. Journal of Affective Disorders, 161, 136-143.

32.

Shapiro

D. N.

Chandler

Mueller

P. A.

(2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213-220.

33.

Solanto

M. V.

Marks

D. J.

Wasserstain

Mitchell

Abikoff

Alvir

J. M. J.

Kofman

M. D.

(2010). Efficacy of meta-cognitive therapy for adult ADHD. American Journal of Psychiatry, 167, 958-968.

34.

Substance Abuse and Mental Health Services Administration. (2013). Results from the 2012 National Survey on Drug Use and Health: Mental health findings. Rockville, MD: Author.

35.

U.S. Census Bureau. (2014). Current Population Survey, 2014 annual social and economic supplement. Retrieved from https://www.census.gov/hhes/socdemo/education/data/cps/2014/tables.html

36.

Victor

S. E.

Klonsky

E. D.

(2014). Daily emotion in non-suicidal self-injury. Journal of Clinical Psychology, 70, 364-375.

37.

Visser

S. N.

Danielson

M. L.

Bitsko

R. H.

Holbrook

J. R.

Kogan

M. D.

Ghandour

R. M.

. . . Blummberg

S. J.

(2014). Trends in the parent-report of health care provider-diagnosis and medication treatment for ADHD disorder: United States, 2003–2011. Journal of the American Academy of Child & Adolescent Psychiatry, 53, 34-46.

38.

Wilens

T. E.

Biederman

Faraone

S. V.

Martelon

Westerberg

Spencer

T. J.

(2009). Presenting ADHD symptoms, subtypes, and comorbid disorders in clinically referred adults with ADHD. Journal of Clinical Psychiatry, 70, 1557-1562.