Accuracy of Electronic Health Record–Derived Data for the Identification of Incident ADHD

Abstract

Objective: To assess the accuracy of electronic health record (EHR)–derived diagnoses in identifying children with incident (i.e., newly diagnosed) ADHD. Method: In 10 large health care organizations, electronic diagnoses data were used to identify all potential cases of incident ADHD among 3- through 9-year-old children. A random sample of records was manually reviewed to determine whether a diagnosis of ADHD was documented in clinician notes. Results: From electronic diagnoses data, a total of 7,362 children with incident ADHD were identified. Upon manual review of 500 records, the diagnosis of incident ADHD was confirmed in clinician notes for 71.5% (95% confidence interval [CI] = [56.5, 86.4]) of records for 3- through 5-year-old children and 73.6% (95% CI = [65.6, 81.6]) of records for 6- through 9-year-old children. Conclusion: Studies predicated on the identification of incident ADHD cases will need to carefully consider study designs that minimize the likelihood of case misclassification.

Keywords

ADHD electronic health records child accuracy

Introduction

ADHD is the most common mental health disorder of childhood, with a prevalence rate of 8.7% to 9.5% (Centers for Disease Control and Prevention, 2010; Froehlich et al., 2007; Visser, Lesesne, & Perou, 2007). The burden of illness from ADHD is high, with affected individuals often experiencing significant functional limitations, such as poor academic achievement and impaired interactions with peers and family members (Larson, Russ, Kahn, & Halfon, 2011). Although a number of pharmacologic and non-pharmacologic approaches are available for the treatment of ADHD (American Academy of Pediatrics, 2011), the majority of clinical trials of ADHD treatments have been conducted in university settings using strict study protocols (MTA Cooperative Group, 1999). Most children with ADHD are seen and treated in community-based primary care and mental health clinics; however, data are limited regarding the comparative effectiveness of various ADHD treatments in children seen in these settings (Epstein et al., 2007; Epstein et al., 2010).

Having a well-defined cohort of children treated for ADHD in routine care settings will be necessary to conduct comparative effectiveness research in this population. While a number of studies have utilized electronic data (such as from claims data or electronic health record [EHRs]) to examine health care utilization (DeBar, Lynch, & Boles, 2004; Guevara, Lozano, Wickizer, Mell, & Gephart, 2001; Ray et al., 2006), medication use (Chen, Gerhard, & Winterstein, 2009; Christensen, Sasane, Hodgkins, Harley, & Tetali, 2010; Guevara, Lozano, Wickizer, Mell, & Gephart, 2002), and quality of care (Zima et al., 2010) in children with ADHD, few studies have examined the validity of using electronic data to identify children with ADHD (Guevara et al., 2002). In addition, it is not known whether incident (i.e., newly diagnosed) cases of ADHD can be identified from electronic data; use of incident cases can be important to comparative effectiveness research as a means of controlling for potential sources of bias (Johnson et al., 2013). The objective of the current study was to assess, in a large network of health care organizations, the accuracy of EHR-derived diagnoses in identifying children with incident ADHD.

Method

Overview

We conducted a multi-site retrospective cohort study to evaluate the validity of using EHR-derived diagnosis codes to identify children 3 through 9 years of age with incident ADHD. The study period was January 1, 2004, through December 31, 2010. Potential ADHD cases were identified using International Classification of Diseases–Ninth Revision–Clinical Modification (ICD-9-CM) diagnosis codes, and a random sample of potential cases was selected for manual review. The validity of EHR-based diagnosis codes was assessed compared with clinician-diagnosed ADHD documented within progress notes and other fields in the EHR. The study was approved by the human subjects review board at the lead study site. Eight participating non-lead sites ceded human subjects oversight to the lead site; one additional site retained research oversight and also approved the study. Written informed consent was not required.

Study Setting

This study was conducted in 10 large health care organizations participating in the Scalable Partnering Network (SPAN) for Comparative Effectiveness Research (www.span-network.org). The SPAN network was funded by the Agency for Healthcare Research and Quality (AHRQ) to develop the informatics platform, governance processes, and scientific expertise to conduct research in a distributed health data network (Brown et al., 2010). All sites participating in the SPAN network use EHRs in routine care delivery.

Eight participating sites were managed care organizations (MCOs), which provided comprehensive care within a closed or open network managed care model. These MCOs maintained health plan enrollment information on their members; enrollment in this context meant that an individual was insured by, and could seek care from, their respective MCO. These sites provided behavioral and mental health services for their members.

The two additional study sites were community health systems: One site was a rural health care network in the upper Midwest and the second was an urban integrated community health care network in Denver, Colorado. Unlike the MCOs, these sites were not health insurers, and therefore did not have health plan enrollment information on their patients. While the community health sites offered behavioral and mental health services for their patients, their patients could also seek care elsewhere, and such care would not result in an insurance claim to the community health system.

Identification of Potential ADHD Cases

An algorithm was developed to identify potential cases of ADHD. Because the algorithm was specifically designed to identify incident cases of ADHD, a more stringent set of criteria was developed than has been utilized in other epidemiologic studies of ADHD (Christensen et al., 2010; Getahun et al., 2013). Once the algorithm was developed for MCO sites, it was modified for community sites to accommodate the absence of health plan enrollment information at those sites.

At MCO sites, we required that children were enrolled in their MCO on or before their 4th birthday, and were continuously enrolled until their 10th birthday, the end of the study period (December 31, 2010), or at least 60 days after their second diagnosed encounter for ADHD. This enrollment requirement was used to reduce the likelihood that individuals had ADHD diagnosed outside of the MCO, such as by a prior provider in another system. We then excluded individuals with a diagnosis of mental retardation or pervasive developmental disorder (ICD-9-CM codes 299.x, 317.x, 318.x, 319.x); per Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; American Psychiatric Association [APA], 1994) criteria, ADHD is not generally diagnosed in the presence of these other conditions (APA, 1994). Within this cohort, we then identified potential cases of incident ADHD, utilizing the following case definition: (a) age 3 through 9 years at the time of initial ADHD diagnosis (ICD-9-CM code 314.0x); (b) two separate encounters with a diagnosis of ADHD; (c) the two diagnoses of ADHD were at least 7 days and at most 365 days apart from each other; and (d) the ADHD diagnoses were made in a primary care, pediatric, or mental health setting. Email or telephone encounters for ADHD were not included in the case definition.

At community sites, the algorithm to identify potential cases of incident ADHD was modified as follows. As these sites did not have health plan enrollment information, we required children to have at least one outpatient medical encounter at the community site before their fifth birthday. We also required that individuals did not have any gaps between outpatient medical encounters of greater than 18 months. Individuals with mental retardation or pervasive developmental disorders were excluded as described above. The case definition for potential incident ADHD was the same as that used at MCO sites.

Manual Review of Potential ADHD Cases

After identifying potential incident ADHD cases at MCO and community sites, we conducted a manual review of a stratified random sample of records to assess the validity of the algorithm used to identify incident ADHD. Sampling of records was stratified by study site and by age: Within each site, we randomly selected 40 records for children 6 through 9 years of age at the time of their first ADHD diagnosis, and 10 records for children 3 through 5 years of age at their first ADHD diagnosis. Records were reviewed for the younger age group because of the increasing trend toward diagnosis and treatment of ADHD in pre-school age children, and their inclusion in the most recent pediatric practice guidelines (American Academy of Pediatrics, 2011). However, this was exploratory, and resource limitations prevented the review of additional records in the younger age group.

A standardized, paper-based abstraction tool was developed to guide the medical record review process. The tool was developed and pilot-tested at two sites, and subsequently revised. According to DSM-IV criteria, the diagnosis of ADHD is primarily based on the presence of a number of inattentive and/or hyperactive symptoms/behaviors, and the determination that these symptoms/behaviors impair functioning in at least two different settings such as school and home (APA, 1994). Given the complexity of making an ADHD diagnosis, national organizations encourage using standardized ADHD screening checklists to help establish a diagnosis (American Academy of Pediatrics, 2000). The abstraction tool included the following elements: whether a clinician-diagnosed ADHD; whether ADHD had been diagnosed previously in another setting; whether inattentive and/or hyperactive symptoms were present; whether these symptoms were associated with impairment in school or at home; and whether a standardized ADHD screening checklist was utilized.

The medical record review was conducted at each site by trained abstractors. For each record randomly selected for review, abstractors were instructed to review all notes and records for all encounters within a 60-day window before and after the dates of the first (referred to as the “index”) and second (referred to as the “follow-up”) encounters with ADHD diagnoses. Abstractors may have also reviewed other time periods if necessary to complete the abstraction tool, such as to search for evidence of a prior ADHD diagnosis. Before starting their reviews, abstractors participated in 3 hr of Internet-based training. After completion of abstractions, the paper-based forms were reviewed at the lead site by a research specialist (L.P.) and a child psychiatrist (D.N.), and entered into an electronic database.

Analytic Methods

Frequencies and descriptive statistics were calculated for all variables of interest. ADHD cases were considered prevalent cases if the abstractor noted that ADHD had been diagnosed greater than 30 days before the index diagnosis date. We then calculated five confirmation rates, each representing a different definition of a confirmed case of ADHD. For each confirmation rate, the denominator was the number of children randomly sampled for manual record review. For the five different confirmation rates, the numerator was calculated based on the five definitions listed below. The primary outcome of interest was chart confirmation of incident ADHD (Definition 2):

Clinician note confirming the diagnosis of ADHD in either the index or follow-up window.

Clinician note confirming the diagnosis of ADHD in either the index or follow-up window, with all prevalent ADHD cases excluded.

Clinician note confirming the diagnosis of ADHD in either the index or follow-up window, prevalent cases excluded, including only those cases with at least one positive ADHD screening checklist documented.

Clinician note confirming the diagnosis of ADHD in either the index or follow-up window, prevalent cases excluded, including only those cases with at least six of nine inattentive and/or six of nine hyperactive/impulsive symptoms documented.

We calculated weighted confirmation rates and 95% confidence intervals. Each sampled case was weighted by the inverse selection probability, defined as 1 divided by the sampled population over the entire eligible ADHD population for that site. For example, if there were 100 potential incident ADHD cases among 6- through 9-year-old children at one site, and 40 were randomly sampled, then the inverse selection probability for each sampled case would be 1/(40/100) = 2.5. Confirmation rates were examined by site, by type of site (MCO vs. community site), and by age group (3 through 5 vs. 6 through 9 years of age).

Results

ADHD Cases Identified From EHRs

The process of identifying potential incident ADHD cases at the eight MCO sites is shown in Figure 1. As can be seen, from a study cohort of 184,712, we identified 6,839 children (3.7%) who met the case definition of incident ADHD. At the two community sites, we identified 523 children who met the case definition of incident ADHD. However, because of the lack of health plan enrollment information at community sites, we were unable to determine a denominator from which children at community sites were selected.

Figure 1.

Identification of potential incident ADHD cases from EHR diagnosis data at eight MCOs, 2004-2010.

The demographic characteristics of the children with potential incident ADHD are presented in Table 1. Seventy-four percent of the identified children were male. The age at diagnosis is also shown in Table 1; however, because of the requirement that participants be enrolled in their health plan on their fourth birthday (or for community sites, have an outpatient visit prior to their fifth birthday), the data shown do not represent the age-stratified prevalence of ADHD in the entire population. The demographic characteristics of those participants selected for manual record review are also shown in Table 1.

Table 1.

Potential Incident ADHD Cases Identified From Electronic Health Record–Derived Diagnosis Information and ADHD Cases Randomly Sampled for Manual Record Review,^a 2004-2010.

	Potential ADHD cases	Sampled ADHD cases	Sampled ADHD cases
Characteristic	n = 7,362^b	3- through 5-year-olds, n = 100	6- through 9-year-olds, n = 400
Sex, n (%)
Male	5,426 (73.7)	77 (77.0)	269 (67.3)
Female	1,936 (26.4)	23 (23.0)	131 (32.8)
Age at first ADHD diagnosis in years, n (%)
3	87 (1.2)	6 (6.0)	NA
4	497 (6.8)	28 (28.0)	NA
5	1,220 (16.6)	66 (66.0)	NA
6	2,062 (28.0)	NA	159 (39.8)
7	1,882 (25.6)	NA	133 (33.3)
8	1,142 (15.5)	NA	75 (18.8)
9	472 (6.4)	NA	33 (8.3)
Race, n (%)
White	4,481 (60.9)	57 (57.0)	260 (65.0)
Black	696 (9.5)	10 (10.0)	39 (9.8)
Other	663 (9.0)	11 (11.0)	46 (11.5)
Unknown or missing	1,522 (20.7)	22 (22.0)	55 (13.8)
Hispanic ethnicity, n (%)
Yes	997 (13.5)	14 (14.0)	61 (15.3)
No	1,281 (17.4)	41 (41.0)	171 (42.8)
Unknown or missing	5,084 (69.1)	45 (45.0)	168 (42.0)

Note. NA = not applicable.

Within each site, 10 records for children 3 through 5 years of age at the time of their first ADHD diagnosis and 40 records for children 6 through 9 years of age at the time of their first ADHD diagnosis were randomly sampled.

Among these cases, n = 6,839 were identified from the eight managed care sites, and n = 523 were identified from the two community sites.

ADHD was diagnosed in a variety of outpatient clinical settings. For 41.4% of incident ADHD cases, the index and follow-up diagnoses were made in pediatric clinics; for 37.9%, the index and follow-up diagnoses were made in mental health clinics; and for 4.6%, the diagnoses were made in a primary care setting (“primary care” in the participating sites is typically used to designate a clinic that may have some combination of internal medicine, family medicine, and/or pediatrics within one clinic). The remaining 16.1% were diagnosed in a combination of these settings.

Manual Record Review

Five hundred participants were randomly selected for a manual review of their EHR: 40 records from 6- through 9-year-old children and 10 records from 3- through 5-year-old children at each of the 10 study sites. Seven out of 400 records (1.8%) for 6- through 9-year-old children and 3 out of 100 records (3.0%) for 3- through 5-year-old children were not available for manual review.

Descriptive information about the results of the manual record review is presented in Table 2. As shown, in most cases (85.6% of records for 3- through 5-year-olds and 90.1% of records for 6- through 9-year-olds), a clinician documented a diagnosis of ADHD in the EHR on the index date. An ADHD diagnosis was also documented in most cases on the follow-up visit date. More detailed information about how the diagnosis of ADHD was established was often lacking from records. For example, clinicians often documented some inattentive or hyperactive symptoms. However, few records explicitly documented six or more inattentive symptoms (9.3% of index date records for 3- through 5-year-olds and 14.2% of index date records for 6- through 9-year-olds), or six or more hyperactive/impulsive symptoms (13.4% and 11.5% of index date records for younger and older children, respectively).

Table 2.

Descriptive Information From Manual Review of Electronic Health Records of Potential Incident ADHD Cases,^a 2004-2010.

Characteristic	n = 97 records for 3- through 5-year-olds, n (%)	n = 393 records for 6- through 9-year-olds, n (%)
Index date/index window^b
Medical encounter on ADHD diagnosis index date	87 (89.7)	371 (94.4)
Clinician note confirming diagnosis of ADHD on index date	83 (85.6)	354 (90.1)
Number of DSM-IV inattentive symptoms documented in clinician notes during the index window
1-5 symptoms	61 (62.9)	245 (62.3)
6-9 symptoms	9 (9.3)	56 (14.2)
Number of DSM-IV hyperactive symptoms documented in clinician notes during the index window
1-5 symptoms	55 (56.7)	193 (49.1)
6-9 symptoms	13 (13.4)	45 (11.5)
Clinician note documenting that symptoms impair functioning at school	46 (47.4)	230 (58.5)
Clinician note documenting that symptoms impair functioning at home	29 (29.9)	128 (32.6)
Number of ADHD screening checklists per child documented during the index window
None	46 (47.4)	149 (37.9)
1	39 (40.2)	183 (46.6)
2	9 (9.3)	44 (11.2)
3 or more	3 (3.1)	17 (4.3)
Follow-up date/follow-up window^c
Medical encounter on ADHD diagnosis follow-up date	89 (91.8)	363 (92.4)
Clinician note confirming the diagnosis of ADHD on follow-up date	86 (88.7)	354 (90.1)
Number of DSM-IV inattentive symptoms documented in clinician notes during the follow-up window
1-5 symptoms	30 (30.9)	161 (41.0)
6-9 symptoms	4 (4.1)	21 (5.3)
Number of DSM-IV hyperactive symptoms documented in clinician notes during the follow-up window
1-5 symptoms	30 (30.9)	98 (24.9)
6-9 symptoms	9 (9.3)	16 (4.1)
Clinician note documenting that symptoms impair functioning at school	29 (29.9)	134 (34.1)
Clinician note documenting that symptoms impair functioning at home	15 (15.5)	90 (22.9)
Number of ADHD screening checklists per child documented during the follow-up window
None	76 (78.4)	313 (79.6)
1	14 (14.4)	67 (17.0)
2	4 (4.1)	13 (3.3)
3 or more	3 (3.1)	0 (0.0)

Note. DSM-IV = Diagnostic and Statistical Manual of Mental Disorders–4th ed.

n = 3 records for 3- through 5-year-old children and n = 7 records for 6- through 9-year-old children were not available for manual record review.

The index date refers to the date of the first diagnosis code for ADHD; the index window includes the 60 days before and after the index date.

The follow-up date refers to the date of the second diagnosis code for ADHD; the follow-up window includes the 60 days before and after the follow-up date.

Although ADHD symptoms were often not documented in clinician notes, standardized ADHD screening checklists may have been used. As seen in Table 2, one or more standardized checklists were documented on the index date for n = 51 (52.6%) of 3- through 5-year-old and n = 244 (62.1%) of 6- through 9-year-old children whose records were reviewed. ADHD screening checklists were more often documented on the index date than the follow-up date. Among all ADHD screening checklists used, the Vanderbilt checklist was used most frequently (43.7% of all checklists), followed by the Connors’ checklist (20.6%) and the SNAP [Swanson, Nolan, and Pelham questionnaire] checklist (5.5%). For 10.3% of younger children and 15.5% of older children, an ADHD diagnosis had been made prior to the index date, indicating that the ADHD was a prevalent rather than an incident case.

Confirmation Rates

The extent to which electronic diagnoses of ADHD were confirmed upon medical record review varied depending on the definition of “confirmation” used. As shown in Table 3, when incident and prevalent cases were included (Definition 1), the confirmation rate was high. The confirmation rate for the primary outcome of clinician-documented incident ADHD was 71.5% for younger and 73.6% for older children (Definition 2). The confirmation rate was lower when more stringent diagnostic criteria were applied: For example, requiring documentation of at least one positive ADHD screening checklist, the confirmation rate was 48.9% for younger and 59.1% for older children. The confirmation rate did not differ substantively between MCO sites and community sites: For example, for 6- through 9-year-old children, the confirmation rate for clinician-documented incident ADHD (Definition 2) was 73.7% for MCOs and 72.0% for community sites (p = .82).

Table 3.

Weighted Percentage of Potential ADHD Cases Confirmed Upon Manual Review of Electronic Health Records, Stratified by the Age at Incident Diagnosis, for Five Different Definitions of Confirmed ADHD, 2004-2010.

		3- through 5-year-olds, n = 100	6- through 9-year-olds, n = 400
Definition number	Definition	Confirmation percentage^a [95% CI]	Confirmation percentage^a [95% CI]
1	Clinician note confirming the diagnosis of ADHD in either the index or follow-up window.	89.8 [80.6, 99.0]	94.2 [89.8, 98.5]
2	Clinician note confirming the diagnosis of ADHD in either the index or follow-up window, with all prevalent ADHD cases excluded.	71.5 [56.5, 86.4]	73.6 [65.6, 81.6]
3	Clinician note confirming the diagnosis of ADHD in either the index or follow-up window, prevalent cases excluded, including only those cases with at least one positive ADHD screening checklist documented.	48.9 [33.4, 64.3]	59.1 [50.8, 67.5]
4	Clinician note confirming the diagnosis of ADHD in either the index or follow-up window, prevalent cases excluded, including only those cases with at least six of nine inattentive and/or six of nine hyperactive symptoms documented.	32.8 [17.1, 48.5]	30.9 [22.2, 39.6]
5	Clinician note confirming the diagnosis of ADHD in either the index or follow-up window, prevalent cases excluded, including only those cases with at least one positive ADHD screening checklist, or at least six of nine inattentive and/or six of nine hyperactive symptoms documented.	65.8 [52.2, 79.4]	68.5 [60.8, 76.1]

Note. CI = confidence interval.

Each sampled case was weighted by the inverse selection probability, defined as 1 divided by the sampled population over the entire eligible ADHD population for that site.

Discussion

In this study from eight MCOs and two community health sites, we sought to assess the validity of using EHR-derived diagnosis codes to identify children with incident ADHD. To identify potential ADHD cases, we used an algorithm requiring health plan enrollment (for MCO sites) or health care utilization (for community sites), as well as two separate ICD-9-CM coded visits for ADHD within a 1-year period, and then conducted manual review of a random sample of identified ADHD cases. While more than 89% of potential ADHD cases were confirmed to have clinician documentation of ADHD within the EHR, these represented a mix of incident and prevalent cases. With exclusion of prevalent cases, the confirmation rate for clinician-diagnosed ADHD ranged from 71.5% for 3- through 5-year-old children to 73.6% for 6- through 9-year-old children. The confirmation rate for incident ADHD was lower (48.9%-59.1%) if documentation of one or more ADHD screening checklists was required. The community sites have different care delivery models and serve different patient populations than the MCOs; despite these differences, confirmation rates did not differ substantively between these types of organizations.

ADHD is a relatively high prevalence condition (Centers for Disease Control and Prevention, 2010; Froehlich et al., 2007; Visser et al., 2007) and investigating the comparative effectiveness of different treatments for ADHD is considered a research priority (Institute of Medicine, 2009). The use of EHR-derived information, collected during the routine process of care, should facilitate large-scale comparative effectiveness studies, as well as epidemiologic and health services studies regarding ADHD (Institute of Medicine, 2009). While the accurate identification of cases is critical to such investigations, and many studies have used ICD-9-CM diagnosis codes in EHR-derived data to identify children with ADHD (Chen et al., 2009; Christensen et al., 2010; DeBar et al., 2004; Guevara et al., 2001; Guevara et al., 2002; Getahun et al., 2013; Ray et al., 2006; Zima et al., 2010), few have reported on the accuracy of these codes (Guevara et al., 2002). This should not be surprising: A recent review of studies using electronic diagnostic or procedural codes found that only 14% measured or reported the accuracy of the codes being used to identify cases (van Walraven, Bennett, & Forster, 2011).

The current study also raises a number of challenging questions with respect to diagnostic certainty and the use of standardized case definitions. The diagnostic criteria for ADHD are well-established and widely disseminated to clinicians (American Academy of Pediatrics, 2011; Wolraich et al., 2011). However, our manual review of EHR data revealed that clinicians rarely documented that patients met all of the DSM-IV criteria for ADHD. Several patterns were evident: Clinicians may have made a diagnosis of ADHD without reference to any diagnostic criteria; they may have documented one or more positive ADHD symptom checklists, but those checklists were not available for review; or they may have listed various symptoms of ADHD, but without mentioning impairment in two or more settings. From the data available, it was not possible to determine whether this represented an issue of diagnosis versus of documentation: Clinicians could have made the diagnosis of ADHD without reference to DSM-IV criteria, or they may have used these criteria but not provided the documentation in their textual notes. In either circumstance, relatively few cases of ADHD identified from diagnosis codes would meet a standardized ADHD case definition.

Although the algorithm used in the present study was designed to identify incident cases of ADHD, prevalent cases were also identified. This could have occurred, for example, if ADHD was initially diagnosed by a mental health professional outside of the health care organization. The misclassification of prevalent as incident cases could have a number of implications for epidemiologic and comparative effectiveness studies. In particular, for observational studies comparing outcomes of children on different ADHD medications, this misclassification could contribute substantial bias to the findings (Berger et al., 2012; Johnson et al., 2013). Children with prevalent (i.e., previously diagnosed) ADHD may have been tried on, and failed, multiple medications in the past, and therefore their outcome on the medication of interest would not be independent of the decision to start that medication versus an alternative ADHD treatment.

This study is subject to a number of limitations. We did not manually review EHRs for children who did not have an ICD-9-CM diagnosis of ADHD; we therefore could not calculate the sensitivity of ICD-9-CM codes for the identification of children with ADHD. As discussed above, it was often not possible to distinguish between issues of diagnosis (clinicians not using formal DSM-IV criteria for diagnosing ADHD) and issues of documentation (clinicians using DSM-IV criteria, but not documenting it within their notes). Our algorithm to identify potential cases required two separate in-person encounters with an ICD-9-CM code for ADHD, and we did not assess the accuracy of using only one encounter to identify cases. In addition, our algorithm did not include email or telephone encounters for ADHD. While the study settings involved both managed care and community health organizations, the results may not generalize to other settings such as private practices. Finally, our ability to identify prevalent cases of ADHD was dependent on clinician documentation of such within the EHR; we did not have access to medical records from outside the participating study sites.

In conclusion, while a number of studies have utilized ICD-9-CM diagnosis codes to conduct epidemiologic and health services studies among children with ADHD, few studies have assessed and reported on the accuracy of these codes to identify children with ADHD. In the current investigation, we used a multi-step algorithm to identify potential cases of incident ADHD, and found that while more than 89% of cases represented clinician-diagnosed ADHD, the resultant cases were a mix of incident and prevalent cases. Therefore, studies predicated on the identification of incident ADHD cases, such as comparative effectiveness studies of different ADHD medication regimens, will need to carefully consider study designs that minimize the likelihood of misclassifying prevalent ADHD cases as incident cases.

Footnotes

Acknowledgements

The authors would like to thank the programmers, analysts, and research specialists from the following Scalable Partnering Network (SPAN) sites for providing data for this project: Denver Health and Hospital Authority (DHHA), Essentia Institute of Rural Health (EIRH), Geisinger Health System (GHS), Group Health Research Institute (GHRI), HealthPartners Institute for Education and Research (HPIER), Kaiser Permanente Hawaii (KPHI), Kaiser Permanente Northern California (KPNC), Kaiser Permanente Northwest (KPNW), Kaiser Permanente Georgia (KPGA), and Kaiser Permanente Colorado (KPCO). We are indebted to the following SPAN investigators for the collaborations that made this study possible: GHRI (Denise Boudreau, PhD, RPh); DHHA (Art Davidson, MD); KPGA (Melissa Butler, PharmD, PhD); EIRH (Thomas Elliott, MD). We would also like to thank David C. Tabano, MS; Andrew T. Sterrett, PhD; Jennifer Dickman-Portz, MSW, PhD; Barbara McCray; and Michael Shainline, MS, for their helpful technical and administrative assistance.

Authors’ Note

The findings and conclusions in this report are solely the responsibility of the authors and do not necessarily represent the official views of AHRQ or the Department of Health and Human Services. The first author had full access to all of the data in this study and takes responsibility for the integrity of the data and the accuracy of the data analyses.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr. Herrinton has received funding from Proctor and Gamble, Centocor, Genentech, Roch, Medimmune, and Pfizer, all unrelated to the current study.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This investigation was supported by grant number R01HS019912 from the Agency for Healthcare Research and Quality (AHRQ).

Author Biographies

Matthew F. Daley, MD, is a senior investigator at Kaiser Permanente Colorado’s Institute for Health Research and an associate professor at the University of Colorado School of Medicine. He is also a practicing pediatrician and provides clinical care to patients with ADHD.

Douglas A. Newton, MD, MPH, is the chief of child and adolescent psychiatry at Kaiser Permanente Colorado. His responsibilities include clinical practice and policy and administrative support for the Denver-Boulder region.

Lynn DeBar, PhD, MPS, is a senior investigator at the Kaiser Permanente Center for Health Research whose previous health services research includes a focus on psychotropic treatment for ADHD in children.

Sophia R. Newcomer, MPH, is a biostatistician with Kaiser Permanente Colorado’s Institute for Health Research and provided statistical and data management support for this study.

Lisa Pieper, MBA, MSHA, works in the Kaiser Permanente Colorado’s Institute for Health Research and provided project manager and research assistant support for this study.

Joseph A. Boscarino, PhD, MPH, is a senior scientist with the Center for Health Research at Geisinger Clinic in Danville, PA, and a professor of psychiatry (adjunct) in the Department of Psychiatry, Temple University School of Medicine in Philadelphia, PA. He has conducted research and has published extensively in the fields of trauma psychology and PTSD.

Sengwee Toh, ScD, is an assistant professor at Harvard Medical School and Harvard Pilgrim Health Care Institute. He has published articles on the comparative effectiveness and safety of medical products.

Pamala Pawloski, PharmD, is a research investigator with HealthPartners Institute for Education and Research and an adjunct associate professor in the Department of Experimental and Clinical Pharmacology at the University of Minnesota College of Pharmacy.

James D. Nordin, MD, MPH, is a practicing pediatrician at HealthPartners Institute for Education and Research who specializes in ADHD, and a clinical investigator interested in many pediatric topics.

Cynthia Nakasato, MD, is a pediatric investigator at the Center for Health Research, Kaiser Permanente Hawaii.

Lisa J. Herrinton, PhD, is an epidemiologist and health services researcher. She is currently engaged in implementation of measurement systems needed for population management and quality improvement of rare diseases.

Regina Bussing, MD, MSHS, is a tenured professor at the University of Florida with appointments in psychiatry, clinical psychology, and pediatrics. She has published extensively on the topic of ADHD and is a leading authority on ADHD-focused mental health services research.

References

American Academy of Pediatrics. (2000). Clinical practice guideline: Diagnosis and evaluation of the child with attention-deficit/hyperactivity disorder. Pediatrics, 105, 1158-1170.

American Academy of Pediatrics. (2011). ADHD: Clinical practice guideline for the diagnosis, evaluation and treatment of attention-deficit/hyperactivity disorder in children and adolescents. Pediatrics, 128, 1007-1022.

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

Berger

M. L.

Dreyer

Anderson

Towse

Sedrakyan

Normand

S. L.

(2012). Prospective observational studies to assess comparative effectiveness: The ISPOR good research practices task force report. Value in Health, 15, 217-230.

Brown

J. S.

Holmes

J. H.

Shah

Hall

Lazarus

Platt

(2010). Distributed health data networks: A practical and preferred approach to multi-institutional evaluations of comparative effectiveness, safety, and quality of care. Medical Care, 48(Suppl.), S45-S51.

Centers for Disease Control and Prevention. (2010). Increasing prevalence of parent-reported attention-deficit/hyperactivity disorder among children—United States, 2003 and 2007. MMWR Morbidity and Mortality Weekly Report, 59, 1439-1443.

Chen

C. Y.

Gerhard

Winterstein

A. G.

(2009). Determinants of initial pharmacological treatment for youths with attention-deficit/hyperactivity disorder. Journal Child and Adolescent Psychopharmacology, 19, 187-195.

Christensen

Sasane

Hodgkins

Harley

Tetali

(2010). Pharmacological treatment patterns among patients with attention-deficit/hyperactivity disorder: Retrospective claims-based analysis of a managed care population. Current Medical Research & Opinion, 26, 977-989.

DeBar

L. L.

Lynch

F. L.

Boles

(2004). Healthcare use by children with attention deficit/hyperactivity disorder with and without psychiatric comorbidities. Journal Behavioral Health Services & Research, 31, 312-323.

10.

Epstein

J. N.

Langberg

J. M.

Lichtenstein

P. K.

Altaye

Brinkman

W. B.

House

Stark

L. J.

(2010). Attention-deficit/hyperactivity disorder outcomes for children treated in community-based pediatric settings. Archives Pediatrics & Adolescent Medicine, 164, 160-165.

11.

Epstein

J. N.

Rabiner

Johnson

D. E.

Fitzgerald

D. P.

Chrisman

Erkanli

. . .Conners

C. K.

(2007). Improving attention-deficit/hyperactivity disorder treatment outcomes through use of a collaborative consultation treatment service by community-based pediatricians: A cluster randomized trial. Archives Pediatrics & Adolescent Medicine, 161, 835-840.

12.

Froehlich

T. E.

Lanphear

B. P.

Epstein

J. N.

Barbaresi

W. J.

Katusic

S. K.

Kahn

R. S.

(2007). Prevalence, recognition, and treatment of attention-deficit/hyperactivity disorder in a national sample of US children. Archives Pediatrics & Adolescent Medicine, 161, 857-864.

13.

Getahun

Jacobsen

S. J.

Fassett

M. J.

Chen

Demissie

Rhoads

G. G.

(2013). Recent trends in childhood attention-deficit/hyperactivity disorder. JAMA Pediatrics, 167, 282-288.

14.

Guevara

Lozano

Wickizer

Mell

Gephart

(2001). Utilization and cost of health care services for children with attention-deficit/hyperactivity disorder. Pediatrics, 108, 71-78.

15.

Guevara

Lozano

Wickizer

Mell

Gephart

(2002). Psychotropic medication use in a population of children who have attention-deficit/hyperactivity disorder. Pediatrics, 109, 733-739.

16.

Institute of Medicine. (2009). Initial national priorities for comparative effectiveness research. Washington, DC: National Academies Press.

17.

Johnson

E. S.

Bartman

B. A.

Briesacher

B. A.

Fleming

N. S.

Gerhard

Kornegay

C. J.

. . .Schneeweiss

(2013). The incident user design in comparative effectiveness research. Pharmacoepidemiology & Drug Safety, 22, 1-6.

18.

Larson

Russ

S. A.

Kahn

R. S.

Halfon

(2011). Patterns of comorbidity, functioning, and service use for US children with ADHD, 2007. Pediatrics, 127, 462-470.

19.

MTA Cooperative Group. (1999). A 14-month randomized clinical trial of treatment strategies for attention-deficit/hyperactivity disorder. The MTA Cooperative Group. Multimodal Treatment Study of Children with ADHD. Archives of General Psychiatry, 56, 1073-1086.

20.

Ray

G. T.

Levine

Croen

L. A.

Bokhari

F. A.

T. W.

Habel

L. A.

(2006). Attention-deficit/hyperactivity disorder in children: Excess costs before and after initial diagnosis and treatment cost differences by ethnicity. Archives of Pediatrics & Adolescent Medicine, 160, 1063-1069.

21.

van Walraven

Bennett

Forster

A. J.

(2011). Administrative database research infrequently used validated diagnostic or procedural codes. Journal of Clinical Epidemiology, 64, 1054-1059.

22.

Visser

S. N.

Lesesne

C. A.

Perou

(2007). National estimates and factors associated with medication treatment for childhood attention-deficit/hyperactivity disorder. Pediatrics, 119(Suppl. 1), S99-S106.

23.

Wolraich

Brown

R. T.

DuPaul

Earls

Feldman

H. M.

. . .Visser

(2011). ADHD: Clinical practice guideline for the diagnosis, evaluation, and treatment of attention-deficit/hyperactivity disorder in children and adolescents. Pediatrics, 128, 1007-1022.

24.

Zima

B. T.

Bussing

Tang

Zhang

Ettner

Belin

T. R.

Wells

K. B.

(2010). Quality of care for childhood attention-deficit/hyperactivity disorder in a managed care medicaid program. Journal of the American Academy of Child & Adolescent Psychiatry, 49, 1225-1237.