Validation of the Use of Electronic Health Records for Classification of ADHD Status

Abstract

Objective: To validate an electronic health record (EHR)–based algorithm to classify ADHD status of pediatric patients. Method: As part of an applied study, we identified all primary care patients of The Children’s Hospital of Philadelphia [CHOP] health care network who were born 1987-1995 and residents of New Jersey. Patients were classified with ADHD if their EHR indicated an International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis code of “314.x” at a clinical visit or on a list of known conditions. We manually reviewed EHRs for ADHD patients (n = 2,030) and a random weighted sample of non-ADHD patients (n = 807 of 13,579) to confirm the presence or absence of ADHD. Results: Depending on assumptions for inconclusive cases, sensitivity ranged from 0.96 to 0.97 (95% confidence interval [CI] = [0.95, 0.97]), specificity from 0.98 to 0.99 [0.97, 0.99], and positive predictive value from 0.83 to 0.98 [0.81, 0.99]. Conclusion: EHR-based diagnostic codes can accurately classify ADHD status among pediatric patients and can be used by large-scale epidemiologic and clinical studies with high sensitivity and specificity.

Keywords

accuracy adolescents attention deficit disorder medical records sensitivity

Introduction

ADHD is the most common childhood neurodevelopmental disorder, with a prevalence of 5% to 10% among 4- to 17-year-olds in the United States and 5% worldwide (Polanczyk, Willcutt, Salum, Kieling, & Rohde, 2014; Thomas, Sanders, Doust, Beller, & Glasziou, 2015). ADHD is currently defined by persistent patterns of inattention and/or hyperactivity/impulsivity that are inconsistent with one’s developmental level, usually continue across the life span, and lead to impairments in social, educational, and work activities (American Psychiatric Association, 2013; Barkley, 1997; Barkley, Fischer, Smallish, & Fletcher, 2002; Levy, 2014).

The current best practice method used to diagnose ADHD in childhood is a comprehensive approach that combines information from multiple methods and informants, including parent and teacher ratings, and clinical information derived from parent interviews (Dupaul et al., 2016; Goldman, Genel, Bezman, & Slanetz, 1998; Willcutt, 2012). These methods, however, generally are not feasible to implement on a consistent basis in high-volume primary care settings (Epstein et al., 2014), which may preclude their use in large-scale clinical or population-based studies of ADHD. Such studies—which play an important role in addressing key ADHD-related research questions—frequently utilize existing administrative databases such as medical or insurance claims to identify patients with ADHD or classify ADHD status among study patients (Chen et al., 2014; Christensen, Sasané, Hodgkins, Harley, & Tetali, 2010; Guevara, Lozano, Wickizer, Mell, & Gephart, 2001, 2002; Leibson, Katusic, Barbaresi, Ransom, & Brien, 2001; Mandell, Guevara, Rostain, & Hadley, 2003; Zima et al., 2010). These databases can be efficiently utilized in research as they contain diagnostic and health service information about large numbers of individuals, are time efficient, relatively inexpensive to access, and can be linked with other data sets to examine relationships between health services and outcomes (Burke et al., 2014; Leibson et al., 2001; Losina, Barrett, Baron, & Katz, 2003; Mandell et al., 2003; Roos, Walld, Wajda, Bond, & Hartford, 1996; Virnig & McBean, 2001). In particular, with the advent of the “meaningful use” criteria released by the Department of Health and Human Services and recent government investment to incentivize the use of medical electronic health records (EHRs; Jha, 2010), the amount of EHR data readily available for research purposes is rapidly growing and likely to increase substantially in the future.

One potential limitation of utilizing EHR data for research is the uncertain validity of these data given that they are not collected for research purposes, may not be collected systematically, and may be incomplete or contain coding errors (Van Walraven & Austin, 2012; Virnig & McBean, 2001; Walkup & Yanos, 2005). To address this concern, several prior studies have examined different administrative databases—Medicaid and Medicare claims, pharmacy claims, or institution-level EHRs—to assess validity of diagnoses of psychiatric conditions (Walkup & Yanos, 2005), including depression (Rawson, Malcolm, & D’Arcy, 1997; Spettell et al., 2003), schizophrenia (Lurie, Popkin, Dysken, Moscovice, & Finch, 1992; Rawson et al., 1997), and autism (Burke et al., 2014). Results have been mixed; some case definitions were found to have high specificity but low sensitivity and positive predictive value (PPV; Lurie et al., 1992; Spettell et al., 2003), while others have found high PPVs or a high proportion of agreement between hospital and medical chart data (Burke et al., 2014; Rawson et al., 1997). Similar to the manner in which ADHD is assessed, these mental health conditions are diagnosed in community settings using labor-intensive best practice methods that are difficult to implement in high-volume primary care settings. Validation studies of administrative data are vital to understanding the potential misclassification of ADHD status and allowing researchers to estimate the magnitude and direction of misclassification bias in applied studies.

Despite the potential utility of EHR data for use in epidemiologic studies related to ADHD, only two studies have assessed the utility of EHR in identifying children diagnosed with ADHD (Daley et al., 2014; Guevara et al., 2001). Daley et al. (2014) calculated confirmation rates, that is, the percentage of diagnoses of ADHD indicated in EHRs that were confirmed via medical record review. Within the context of a larger study determining the cost of health care for those with ADHD, Guevara et al. (2001) reviewed the medical charts of 70 randomly selected patients with an ADHD diagnosis to assess the diagnosis’ accuracy and to conduct sensitivity analyses. To our knowledge, no study has examined the sensitivity and specificity of an algorithm to identify ADHD cases using existing administrative EHR data. Thus, our objective was to validate an EHR-based algorithm to classify ADHD status among a cohort of primary care patients of CHOP pediatric health care network. This internal validation study was part of a larger retrospective cohort study that aimed to examine the association between ADHD and driving outcomes among adolescents.

Method

CHOP EHR

The CHOP health care network encompasses more than 50 locations throughout southeastern Pennsylvania and southern New Jersey, including 31 primary care centers, 14 specialty care centers, a 535-bed inpatient hospital, two emergency departments, and two urgent care centers supporting more than one million visits annually. The network serves a socioeconomically and racially diverse population and accepts most insurance plans, including Medicaid. CHOP first implemented its EHR system (EpicCare®, Epic Systems, Inc, Madison, WI) in 2001, and it is currently used for all aspects of clinical care in all outpatient locations and within the hospital. For every hospital encounter or office visit, the CHOP EHR captures (a) identifiable data (e.g., name, date of birth, address); (b) demographic data (e.g., sex, race/ethnicity); (c) International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis codes that indicate the reason(s) for the visit (required for a provider to bill for services under that code); (d) ICD-9-CM diagnosis codes that document ongoing or historical problems (known as the “problem list”); (e) medications prescribed by CHOP or non-CHOP providers; and (f) provider notes, letters, communication with non-CHOP health care providers, and other qualitative descriptions of each visit that may provide important supplemental information on ADHD diagnosis (e.g., results of diagnostic testing, documentation of parent and teacher reports).

ADHD Classification: Algorithm Development

We first conducted an initial analysis of EHR data to develop an algorithm to identify whether each patient had been diagnosed with ADHD. To this end, we identified all patients of the CHOP health care network who were born 1987-1995 and New Jersey residents at their last CHOP visit (N = 113,966). We then identified those who fit the following hierarchical criteria and, at each level, extracted a random sample of records for manual chart abstraction: (a) had an ICD-9-CM diagnosis code beginning with “314” (i.e., indicative of ADHD) at a CHOP office or hospital visit (n = 7,567; 172 sampled), (b) had an ICD-9-CM diagnosis code beginning with “314” on their problem list (n = 302; 30 sampled), (c) were prescribed a medication with primary indication for ADHD (n = 880; 88 sampled) in the absence of an ADHD diagnostic code, and (d) had select ADHD-related keywords (e.g., “ADHD,” “hyperactivity”) in provider progress notes or qualitative descriptions of visits (n = 3,055; 161 sampled) in the absence of an ADHD diagnostic code. This final step was a search for key terms within the provider notes and not a formal natural language processing evaluation. With oversight by two study authors (BY, TP), two trained abstractors performed a manual review of each sampled patient’s entire EHR to search for independent sources of information that confirmed ADHD case or non-case status. These sources included letters, patient history, evaluations, appointment notes, telephone notes, problem list addendums, uploaded media (e.g., scanned prescription, external evaluations), laboratory results, emergency department notes, and office or hospital visits if they indicated a separate calendar date or different physician (i.e., separate visit) than the originally entered ICD-9-CM code. Very few patients who were selected based on visit-or problem list-level ICD-9-CM codes were found to be non-cases (<2% and 0%, respectively). In addition, neither our keyword search in provider notes nor a medication prescription with primary indication for ADHD in the absence of ICD-9-CM diagnostic codes yielded additional confirmed cases of ADHD. Based on these results, we developed the following final algorithm to identify patients who had an ADHD diagnosis in our cohort study: Patients were classified as having ADHD if their EHR indicated an ICD-9-CM diagnostic code beginning with “314” either at a CHOP office or hospital visit or on their problem list.

Formal Internal Validation Study

The aim of this validation study was to develop an applied method to classify ADHD status for a retrospective cohort study examining the association between ADHD and motor vehicle crash outcomes. Eligibility for this formal validation study mirrored criteria for the larger applied study. Eligible patients included individuals who were born 1987-1995, had two or more office visits at any of the six CHOP primary care practices located in New Jersey, were New Jersey residents at the time of their last CHOP network visit, were not identified as having an intellectual disability, and had their last primary care visit at the age of 12 years or older to ensure the individual was seen by a primary care provider at an old enough age to confirm ADHD status (Diagnostic and Statistical Manual of Mental Disorders [5th ed.; DSM-5; American Psychiatric Association, 2013] criteria calls for ADHD symptom onset by age 12). The final study cohort included 15,609 patients (Figure 1). Using the above-described ADHD algorithm, 2,030 (13.0%) of these patients were classified as having ADHD.

Figure 1.

Flowchart depicting selection of study cohort.

We conducted an EHR review for all patients who were classified with ADHD and a weighted random sample of patients classified as non-ADHD with the goal of confirming the presence or absence of ADHD in this cohort. Similar to the algorithm development phase, two trained abstractors reviewed patient EHRs with oversight by two study authors (BY, TP). The review process for the 2,030 patients classified as having ADHD is shown in Figure 2. Using EHR data, the following were automatically confirmed as having ADHD: 940 patients who had three or more ADHD-related CHOP visits and an additional 284 who were prescribed an ADHD medication by a CHOP provider (as medication was an independent source of confirmation). The remaining 806 patients included those who had either one or two ADHD-related CHOP visits or a problem list diagnosis and had no history of being prescribed an ADHD medication. For each of these patients, abstractors were instructed to manually review the entire EHR to locate any of the aforementioned independent sources confirming ADHD case status. When contradicting information was present in the EHR, abstractors weighed letters, evaluations, and appointment notes more in making a final diagnostic decision. A patient was determined to be a “confirmed ADHD case” when at least one independent source confirming ADHD diagnoses was recorded. Similarly, if an independent source was found that indicated the patient did not have ADHD, the patient was determined to be a “confirmed non-ADHD case,” weighing the most recent independent source more. Only one patient had information both confirming ADHD status and other information confirming non-ADHD status; this patient was confirmed as a non-ADHD case because one letter and two patient forms documented no history of ADHD. Finally, patients for whom neither a source confirming the presence or absence of ADHD was located were considered to have an unknown case status. Twenty-four percent of records were reviewed by both abstractors to assess interrater reliability. All disagreements were reviewed at consensus meetings, and final decision was made by the two co-authors (BY, TP).

Figure 2.

Flowchart depicting manual review process of ADHD cohort to confirm case status.

We also reviewed a sample of 807 patients in the non-ADHD cohort to determine whether there were sources in their medical records indicating the presence of an ADHD diagnosis. To optimize the rigor of our assessment, we oversampled patients with visit- or problem list-level ICD-9-CM codes for disruptive behavior disorder (“313.81” or “312.X”) and learning disorder (“315.X”)—conditions frequently comorbid among children with ADHD; approximately 70% of patients in each group were sampled. In addition, a 3.5% sample of the 13,005 patients who did not have a comorbid conditions were reviewed. Sizes of samples selected for review were based on power calculations; for each sampled group, we had at least 80% power to detect a 5% misclassification rate with a confidence interval (CI) width of 2%.

Statistical Analysis

We estimated the proportion of patients with ADHD that our classification algorithm correctly identified as having ADHD (i.e., sensitivity), the proportion of patients without ADHD that our classification correctly identified as not having ADHD (i.e., specificity), the proportion of patients classified as ADHD by our algorithm who had an confirmed ADHD diagnoses (i.e., PPV), and the proportion of patients classified has non-ADHD by our algorithm who were confirmed non-ADHD patients (i.e., negative predictive value [NPV]), along with exact 95% CIs using standard measures for proportions. With respect to patients classified as “unknown ADHD case status” during the review of our ADHD cohort, we identified two distinct groups. The first was patients who had their last CHOP primary care visit on or before 2005, the year the EHR was fully implemented in New Jersey primary care practices; these patients were considered to have “incomplete records” as their EHR had very little information available beyond visit dates and diagnosis codes. The second group was patients who had visits after 2005 but did not have information in their EHR confirming the absence or presence of ADHD; we classified these patients as “inconclusive.” In analyses, patients with incomplete records were assumed to have the same distribution of ADHD, non-ADHD, and inconclusive status as patients for whom a case status classification was determined. Validation analyses were conducted under two different assumptions for those with inconclusive status, representing extreme conditions: (a) all were assumed to be true ADHD cases and (b) all were assumed to be true non-ADHD cases. We then weighted the manual review results of the sample of non-ADHD patients to the entire non-ADHD cohort. In addition to estimating validity for the overall sample, we estimated the PPV separately for each origin of diagnostic codes (visit-level vs. problem list). Analyses were conducted in SAS Version 9.3 (SAS Institute, Inc., Cary, NC). This study was approved by CHOP institutional review board.

Results

Overall, patients were last seen at CHOP primary care at a median age of 17.9 years (interquartile range [IQR] = 15.9, 19.1 years; Table 1). Patients in the ADHD cohort had a median of 27 (16, 44) office visits to the CHOP network compared with 18 (9, 31) for patients in the non-ADHD cohort. The majority of patients in both cohorts were non-Hispanic White (ADHD: 69%; non-ADHD: 59%), and patients with ADHD were more likely to be male (72% vs. 47%, p < .001).

Table 1.

Demographic and Relevant Characteristics for Study Participants in ADHD and Non-ADHD Cohorts (N = 15,609).

Characteristic	All participants (N = 15,609)	Cohort
Characteristic	All participants (N = 15,609)	Non-ADHD (n = 13,579)	ADHD (n = 2,030)	p value
Age at last CHOP care visit, median (IQR)	17.9 (15.9-19.1)	17.9 (15.9-20.3)	18.1 (16.5-19.5)	<.0001
Number of CHOP primary care visits, median (IQR)	17 (9-29)	16 (8-28)	23 (13-37)	<.0001
Number of total CHOP visits, median (IQR)	19 (10-33)	18 (9-31)	27 (16-44)	<.0001
Sex, n (%)				<.0001
Female	7,004 (49.4)	7,138 (52.6)	566 (27.9)
Male	7,905 (50.6)	6,441 (47.4)	1,464 (72.1)
Race/ethnicity, n (%)				<.0001
Hispanic	421 (2.7)	373 (2.7)	48 (2.4)
Non-Hispanic White	9,395 (60.2)	7,990 (58.8)	1,405 (69.2)
Non-Hispanic Black	2,762 (17.7)	2,458 (18.1)	304 (15.0)
Non-Hispanic Asian/Other/multiple	3,029 (19.4)	2,756 (20.3)	273 (13.4)
Unknown/refused	2	2	0

Note. Unless otherwise indicated, data are expressed as number (percentage) of participants. Denominators for percentages exclude missing data. Percentages are rounded and may not total to 100. IQR = interquartile range.

A total of 806 EHRs of patients classified as having ADHD were manually reviewed (Figure 2). Of these, we were able to confirm the presence of ADHD in 376; when combined with those automatically confirmed based on three plus visits or medication use, a total of 1,600 of the 2,030 (79%) were determined to be confirmed cases of ADHD. Only 1% of patients were confirmed as non-ADHD cases (n = 25), while the remaining 405 (20%) patients were determined to have an unknown case status. Of these 405 patients, 26% had incomplete records while 74% were labeled inconclusive.

Of the 807 patients in the non-ADHD group who were manually reviewed, 9 (1%) were misclassified as non-ADHD patients when in fact they had been diagnosed with ADHD. After fully weighting these results up to the larger non-ADHD comparison group, we estimated that 67 (0.5%) patients in the non-ADHD cohort were in fact diagnosed with ADHD.

The overall interrater agreement for ADHD case classification was 96.4% with a kappa of 0.87 (95% CI = [0.75, 0.99]). We also estimated the sensitivity, specificity, PPV, and NPV of the ADHD algorithm under the two assumptions (Table 2). When all inconclusive cases were assumed to be true ADHD cases, the overall sensitivity was 0.97 (95% CI = [0.96, 0.97]), the specificity was 0.99 (95% CI = [0.99, 0.99]), and the PPV was 0.98 (95% CI = [0.98, 0.99]). Estimates among those who were classified via visit-level codes and those classified via problem list codes were similar. When all inconclusive cases were assumed to be true non-ADHD cases, the sensitivity was 0.96 (95% CI = [0.95, 0.97]), specificity was 0.98 (95% CI = [0.97, 0.98]), and PPV was 0.83 (95% CI = [0.81, 0.85]). The estimated PPV among patients diagnosed using problem list codes was lower (0.67; 95% CI = [0.58, 0.75]) than those diagnosed using visit-level codes (0.84; 95% CI = [0.83, 0.86]). Notably, we found no evidence that validity varied appreciably by site; estimated sensitivity and specificity was greater than 92% for all sites.

Table 2.

ADHD Status Classified via Algorithm and After Manual Review of Electronic Health Records (i.e., True Case Status), Along With SE, SP, PPV, and NPV (95% CI) of Algorithm, Overall and Stratified by Origin of Diagnostic Code (N = 15,609).

All inconclusive cases assumed to be . . .	ADHD (n = 2,030)		Non-ADHD (n = 13,579)		SE (95% CI)	SP (95% CI)	PPV (95% CI)	NPV (95% CI)
All inconclusive cases assumed to be . . .	True ADHD cases	True non-ADHD cases	True ADHD cases^a	True non-ADHD cases^a	SE (95% CI)	SP (95% CI)	PPV (95% CI)	NPV (95% CI)
Overall sample
True ADHD cases	2,004	26	67	13,512	0.97 [0.96, 0.97]	0.99 [0.99, 0.99]	0.98 [0.98, 0.99]	0.99 [0.99, 0.99]
True non-ADHD cases	1,688	342	67	13,512	0.96 [0.95, 0.97]	0.98 [0.97, 0.98]	0.83 [0.81, 0.85]	0.99 [0.99, 0.99]
Classified based on visit-level codes^b
True ADHD cases	1,878	26					0.98 [0.98, 0.99]
True non-ADHD cases	1,604	300					0.84 [0.83, 0.86]
Classified based on problem list codes^c
True ADHD cases	126	0					1.00 [0.97, 1.00]
True non-ADHD cases	84	42					0.67 [0.58, 0.75]

Note. SE = sensitivity; SP = specificity; PPV = positive predictive value; NPV = negative predictive value; CI = confidence intervals; ICD-9-CM = International Classification of Diseases, Ninth Revision, Clinical Modification.

True case status estimated based on manual review of sample of records.

Case status determined by ICD-9-CM diagnosis codes documented on an office or hospital visit that indicate the reason(s) for the visit.

Case status determined by ICD-9-CM diagnosis codes documented on an list of ongoing or historical problems.

Discussion

This study developed and validated an EHR-based algorithm that classified ADHD status among a large cohort of primary care patients in a regional health care network. Overall, findings demonstrate that an algorithm that seeks to capture ADHD case status among primary care patients can do so with a high degree of sensitivity using only ICD-9-CM codes contained in visit-level diagnosis fields or in the problem list. In addition, a very low proportion of patients without ADHD diagnosis codes were in fact diagnosed with ADHD, resulting in a very robust estimate of specificity for the algorithm. Notably, we were unable to use EHRs to independently confirm the presence or absence of an ADHD diagnosis for one in five patients with ADHD diagnosis codes; this likely reflects the lack of documentation of how ADHD diagnoses were made. Despite this limitation, we demonstrated that the selected ICD-based algorithm was able to identify the presence of ADHD with a high degree of accuracy.

Our findings establish the utility of EHR’s in classifying ADHD status in large population-based studies and quality improvement projects and uniquely extend results of two prior studies. Daley et al. (2014) used EHR data to identify potential cases of incident (i.e., newly diagnosed) ADHD among 3- to 9-year-old patients in 10 large health care organizations. To assess the accuracy of the EHR diagnosis data, the authors randomly selected 40 patients within each site for whom medical charts were reviewed to determine confirmation of incident ADHD. The investigators used five definitions of a confirmed case of ADHD and, for each, calculated the proportion of potential ADHD cases that were confirmed (Daley et al., 2014). When incident and prevalent cases were included, the proportion of confirmed ADHD cases was high (89.8%-94.2%); however, the proportions were lower when a more conservative definition was used (48.9%-59.1%). To assess the validity of ADHD diagnoses in claims data and evaluate the health care costs of ADHD, Guevara et al. (2001) conducted a sensitivity analyses within the context of the larger study. The investigators randomly selected 70 patients for whom medical charts were reviewed and categorized ADHD case status based on different ADHD definitions: probable, possible, and doubtful diagnoses. Seventy-two percent of the 70 patients were classified as having a probable diagnosis, 21% had a possible diagnosis, and 7% had a doubtful diagnosis (Guevara et al., 2001). While both studies estimated the PPV of an existing ADHD diagnosis in health records, neither included patients without ADHD; thus, they were not able to fully assess the quality of their classification by calculating sensitivity or specificity. Although PPV is an important measure of classification quality and can be used to quantitatively estimate the extent of and correct for misclassification bias in applied studies, its dependence on the prevalence of the condition in the population makes portability across studies with different populations problematic (Lash, Fox, & Fink, 2009). To our knowledge, this study is the first to estimate sensitivity and specificity of ADHD diagnosis codes, which can quantify a classification’s performance across populations. More generally, validation studies make it possible to conduct quantitative bias analyses, allowing researchers to account for potential bias within an applied study and ultimately enhance the robustness of study results. While epidemiology methodolgists strongly encourage the conduct of bias analysis (Lash et al., 2009), the process is not often used in applied clinical research.

There are several limitations to this study. First, this method is limited by the quality of diagnostic information in medical records, which was highly variable; some records had parent and teacher rating scales and the results of evaluations conducted by external mental health experts, while others had no information beyond the visit date and diagnosis code. While ADHD diagnoses in the EHR accurately reflect real-world diagnostic decisions rendered by providers in primary care, these decisions may differ to some extent from those rendered by expert mental health providers (e.g., psychologists, psychiatrists, developmental pediatricians) applying best practice methods for assessing ADHD in a systematic manner. Furthermore, inherent in the process of diagnostic decision making for mental health conditions is that, even when best practice methods for diagnosis are systematically applied, there is always an element of uncertainty in a small percentage of cases given that this diagnosis relies on clinical judgment. Accordingly, although this study is able to make conclusions about the extent of an EHR-based algorithm to distinguish patients who have and have not been diagnosed with ADHD in clinical settings, it was not designed to identify “true” ADHD status. Another potential limitation is that we only examined EHR data from a single health care network, raising questions about generalizability. However, the network includes a variety of providers across multiple settings and serves a broad range of patients with regard to race/ethnicity and socioeconomic status. Furthermore, the network utilizes a single EHR system, which promotes standardized practice, similar to other large health care systems. In addition, although there may have been some variability in how providers diagnosed ADHD across sites, we found no appreciable difference in accuracy across the six practices. As such, the practices, providers, and patients represented in this study likely reflect those in other large health care systems, although replications in other systems are needed to confirm generalizability. Study findings are likely more generalizable to studies that are conducted within large health care systems—in particular, those that standardize practices—than to studies that combine data from multiple institutions. In addition, abstractors were not blinded to the ADHD status of the patients. It is possible that this knowledge might have biased the reviewers; however, we expect this was minimized given that we developed strict criteria on information that was considered “confirmatory,” included a classification category of “inconclusive,” and ensured final results were reviewed by two licensed psychologists (BY, TP). Finally, this method does not capture patients who do have ADHD but either do not seek care or were unrecognized by CHOP providers.

Conclusion

Health service and applied clinical studies have utilized ICD-9-CM codes derived from EHRs to classify ADHD status; however, internal validity of study results is dependent on the extent to which diagnosis codes can accurately classify status. The current study describes the development an EHR-based approach to classify ADHD status among primary care patients and, for the first time, demonstrates that such an algorithm can be used by large epidemiologic and clinical studies with high sensitivity and specificity. Estimates of sensitivity and specificity from the current study can be utilized by future applied studies to conduct analyses that correct estimates of measures of association (e.g., odds ratios, risk ratios) for misclassification of ADHD status.

Footnotes

Acknowledgements

The authors thank Melissa Pfeiffer for her expertise in data management and for her review of the manuscript and Michael Elliott for his guidance on statistical analysis. In addition, the authors thank Meghan Kirk and Sayaka Ogawa for their assistance reviewing electronic health records.

Authors’ Note

The views presented are those of the authors and not necessarily the views of The Children’s Hospital of Philadelphia (CHOP) or National Institutes of Health (NIH).

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development at the National Institutes of Health (Grant R01HD079398, Principal Investigator: Curry).

Author Biographies

Siobhan M. Gruschow, MPH, MEd, is a clinical research coordinator at for PolicyLab and the Division of Adolescent Medicine at The Children’s Hospital of Philadelphia (CHOP).

Benjamin E. Yerys, PhD, is a psychologist in the department of Child and Adolescent Psychiatry and Behavioral Sciences and in the Center for Autism Research (CAR) at The Children’s Hospital of Philadelphia. He leads CAR’s research on the brain basis and treatment of attention and executive function. He has published extensively on the presence of executive function impairments in children with autism spectrum disorder (ASD) and how they relate to core repetitive behavior symptoms of ASD or common co-occurring disorders like ADHD.

Thomas J. Power, PhD, is chief psychologist and associate chief of academic affairs in the department of Child and Adolescent Psychiatry and Behavioral Sciences and director of the Center for Management of ADHD at The Children’s Hospital of Philadelphia. As director of the Center for Management of ADHD, he oversees the clinical practice, training, and research activities of a large interdisciplinary team, including developmental and behavioral pediatricians, child and adolescent psychiatrists, clinical and school psychologists, and advanced practice nurses. He has extensive experience developing interventions for children and families coping with ADHD, and he is an expert in promoting the school success of these children and youth.

Dennis R. Durbin, MD, MSCE, is an attending physician in the department of Emergency Medicine, Clinical Epidemiologist, and Chief Clinical Research Officer at The Children’s Hospital of Philadelphia Research Institute. His research interests are in the prevention of motor vehicle occupant injuries to children and the prevention of teen driver crashes, and he has been a major contributor to policy planning and guidance efforts in reducing vehicle-related injuries and deaths among children.

Allison E. Curry, PhD, MPH, is a senior scientist and director of epidemiology and biostatistics at the CIRP at The Children’s Hospital of Philadelphia. Her program of research focuses on the epidemiology and prevention of motor vehicle crashes in children and adolescents. She is the principal investigator of a program of research that aims to establish the scientific foundation for driving safety among teens with ADHD and autism.

References

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Barkley

R. A.

(1997). Behavioral inhibition, sustained attention, and executive functions: Constructing a unifying theory of ADHD. Psychological Bulletin, 121, 65-94.

Barkley

R. A.

Fischer

Smallish

Fletcher

(2002). The persistence of attention-deficit/hyperactivity disorder into young adulthood as a function of reporting source and definition of disorder. Journal of Abnormal Psychology, 111, 279-289.

Burke

J. P.

Jain

Yang

Kelly

J. P.

Kaiser

Becker

. . . Newschaffer

C. J.

(2014). Does a claims diagnosis of autism mean a true case? Autism: The International Journal of Research and Practice, 18, 321-30.

Chen

Sjölander

Runeson

D’Onofrio

B. M.

Lichtenstein

Larsson

(2014). Drug treatment for attention-deficit/hyperactivity disorder and suicidal behaviour: Register based study. British Medical Journal, 348, g3769.

Christensen

Sasané

Hodgkins

Harley

Tetali

(2010). Pharmacological treatment patterns among patients with attention-deficit/hyperactivity disorder: Retrospective claims-based analysis of a managed care population. Current Medical Research and Opinion, 26, 977-989.

Daley

M. F.

Newton

D. A

Debar

Newcomer

S. R.

Pieper

Boscarino

J. A

. . . Bussing

(2014). Accuracy of electronic health record-derived data for the identification of incident ADHD. Journal of Attention Disorders. Advance online publication. doi: 10.1177/1087054713520616.

Dupaul

G. J.

Reid

Anastopoulos

A. D.

Lambert

M. C.

Watkins

M. W.

Power

T. J.

(2016). Parent and teacher ratings of attention-deficit/hyperactivity disorder symptoms: Factor structure and normative data. Psychological Assessment, 28, 214-225.

Epstein

J. N.

Kelleher

K. J.

Baum

Brinkman

W. B.

Peugh

Gardner

. . . Langberg

(2014). Variability in ADHD care in community-based pediatrics. Pediatrics, 134, 1136-1143.

10.

Goldman

L. S.

Genel

Bezman

R. J.

Slanetz

P. J.

(1998). Diagnosis and treatment of attention-deficit/hyperactivity disorder in children and adolescents. Journal of the American Medical Association, 279, 1100-1107.

11.

Guevara

Lozano

Wickizer

Mell

Gephart

(2001). Utilization and cost of health care services for children with attention-deficit/hyperactivity disorder. Pediatrics, 108, 71-78.

12.

Guevara

Lozano

Wickizer

Mell

Gephart

(2002). Psychotropic medication use in a population of children who have attention-deficit/hyperactivity disorder. Pediatrics, 109, 733-739.

13.

Jha

A. K.

(2010). Meaningful use of electronic health records. Journal of the American Medical Association, 304, 1709-1710.

14.

Lash

Fox

Fink

(2009). Applying quantitative bias analysis to epidemiologic data. New York, NY: Springer.

15.

Leibson

Katusic

S. K.

Barbaresi

W. J.

Ransom

Brien

P. C.

(2001). Use and costs of medical care for children attention-deficit/hyperactivity disorder. Journal of the American Medical Association, 285, 60-66.

16.

Levy

(2014). DSM-5, ICD-11, RDOC and ADHD diagnosis. Retrieved from http://anp.sagepub.com/content/early/2014/11/01/0004867414557527.full

17.

Losina

Barrett

Baron

J. A.

Katz

J. N.

(2003). Accuracy of Medicare claims data for rheumatologic diagnoses in total hip replacement recipients. Journal of Clinical Epidemiology, 56, 515-519.

18.

Lurie

Popkin

Dysken

Moscovice

Finch

(1992). Accuracy of diagnoses of schizophrenia in Medicaid claims. Hospital & Community Psychiatry, 43, 69-71.

19.

Mandell

D. S.

Guevara

J. P.

Rostain

A. L.

Hadley

T. R.

(2003). Medical expenditures among children with psychiatric disorders in a Medicaid population. Psychiatric Services, 54, 465-467.

20.

Polanczyk

G. V.

Willcutt

E. G.

Salum

G. A.

Kieling

Rohde

L. A.

(2014). ADHD prevalence estimates across three decades: An updated systematic review and meta-regression analysis. International Journal of Epidemiology, 43, 434-442.

21.

Rawson

N. S. B.

Malcolm

D’Arcy

(1997). Reliability of the recording of schizophrenia and depressive disorder in the Saskatchewan health care datafiles. Social Psychiatry and Psychiatric Epidemiology, 32, 191-199.

22.

Roos

L. L.

Walld

Wajda

Bond

Hartford

(1996). Record linkage strategies, outpatient procedures, and administrative data. Medical Care, 34, 570-582.

23.

Spettell

C. M.

Wall

T. C.

Allison

Calhoun

Kobylinski

Fargason

Kiefe

C. I.

(2003). Identifying physician-recognized depression from administrative data: Consequences for quality measurement. Health Services Research, 38, 1081-1102.

24.

Thomas

Sanders

Doust

Beller

Glasziou

(2015). Prevalence of attention-deficit/hyperactivity disorder: A systematic review and meta-analysis. Pediatrics, 135, e994-e1001.

25.

Van Walraven

Austin

. (2012). Administrative database research has unique characteristics that can risk biased results. Journal of Clinical Epidemiology, 65, 126-131.

26.

Virnig

B. A.

McBean

(2001). Administrative data for public health surveillance and planning. Annual Review of Public Health, 22, 213-230.

27.

Walkup

Yanos

(2005). Psychological research with administrative data sets: An underutilized strategy for mental health services research. Professional Psychology: Research and Practice, 36, 551-557.

28.

Willcutt

E. G.

(2012). The prevalence of DSM-IV attention-deficit/hyperactivity disorder: A meta-analytic review. Neurotherapeutics, 9, 490-499.

29.

Zima

B. T.

Bussing

Tang

Zhang

Ettner

Belin

T. R.

Wells

K. B.

(2010). Quality of care for childhood attention deficit/hyperactivity disorder in a managed care Medicaid program. Journal of the American Academy of Child and Adolescent Psychiatry, 49(12), 1-17.