Comparison of Performance on ADHD Quality of Care Indicators: Practitioner Self-Report Versus Chart Review

Abstract

Objective: This study compared practitioner self-report of ADHD quality of care measures with actual performance, as documented by chart review. Method: In total, 188 practitioners from 50 pediatric practices completed questionnaires in which they self-reported estimates of ADHD quality of care indicators. A total of 1,599 charts were reviewed. Results: The percentage of patients for whom practitioners self-reported that they used evidence-based care was higher in every performance category when compared with chart review, including higher use of parent and teacher rating scales during assessment and treatment compared with chart review. Self-reported use of Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV) criteria during assessment was also higher than by chart review. The actual number of days until the first contact after starting medication was nearly three times longer than self-report estimates. Conclusion: Practitioners overreport performance on quality of care indicators. These differences were large and consistent across ADHD diagnostic and treatment monitoring practices. Practitioner self-report of ADHD guideline adherence should not be considered a valid measure of performance.

Keywords

ADHD self-report chart review quality improvement

The American Academy of Pediatrics (AAP) released consensus guidelines defining evidence-based ADHD care in 2000 and 2001 and updated them in 2011 (AAP, 2000, 2001, 2011). These guidelines recommended that pediatricians use both Diagnostic and Statistical Manual of Mental Disorders (DSM) criteria and standardized rating scales during assessments for ADHD and that they systematically monitor response to treatment.

Previous studies conducted using physician self-report suggest modest adherence to these guidelines (Chan, Hopkins, Perrin, Herrerias, & Homer, 2005; Gardner, Kelleher, Pajer, & Camp, 2004; Rushton, Fant, & Clark, 2004). For example, physicians reported using parent and teacher rating scales during ADHD assessment for approximately 70% of patients, and 58% of physicians reported using formal DSM criteria to diagnose ADHD (Chan et al., 2005; Rushton et al., 2004). More recently, a physician self-report survey indicated that the majority of physicians were adhering to the AAP guideline for ADHD care, including frequent use of a validated screening tool and appropriate follow-up after initiation of medication (McElligott et al., 2014). However, another study using external chart review suggests that physician adherence to the AAP guidelines is lower than what has been reported in studies using physician self-report. In this study, chart reviews revealed that parent and teacher rating scales were used during assessment with 55% and 52% of patients, respectively, and rating scales were rarely ever used to assess treatment response (Epstein et al., 2008).

Physicians’ tendencies to overestimate performance are a documented phenomenon that has led some investigators to raise concerns about the validity of data obtained by self-report (Adams, Soumerai, Lomas, & Ross-Degnan, 1999; Davis et al., 2006). For example, Adams et al. (1999) reported that physicians overestimate performance by nearly 30% on a range of adult-focused quality indicators. In a systematic review of 20 self-to-external comparisons published in 17 articles, Davis et al. (2006) found self-report to be poorly correlated with actual performance and questioned the usefulness and accuracy of self-report as an assessment tool for physicians.

As ADHD is one of the most common pediatric behavioral health conditions (Perou et al., 2013), accurate assessment of practitioner performance is important. Recent studies have raised concerns about the possibility of ADHD overdiagnosis with limited reliance on diagnostic criteria (Bruchmüller, Margraf, & Schneider, 2012). Health care quality has been identified as a key target area by the Institute of Medicine and patients alike (Institute of Medicine, 2001), and quality improvement efforts are a major focus area of the American Board of Pediatrics. An accurate assessment of care quality is necessary to determine if improvement is warranted and if improvement efforts are successful. Similarly, the shift from volume-based to value-based reimbursement (U.S. Department of Health and Human Services, n.d.) requires an assessment of actual performance on quality indicators to fully realize this approach.

Although the available literature would suggest that ADHD guideline adherence is likely higher as assessed by practitioner self-report as compared with chart review, no study has directly compared practitioner self-report with ADHD care documented by comprehensive chart review. Accordingly, the purpose of this study was to utilize a large sample that included both practitioner self-report and external chart review to systematically evaluate the validity of practitioner self-report of ADHD quality of care indicators.

Method

This study was approved by the Institutional Review Boards at Cincinnati Children’s Hospital Medical Center and Nationwide Children’s Hospital. Data were collected from 50 practices and 188 practitioners (184 physicians and four nurse practitioners) in central and northern Ohio from August 2010 to December 2012. Practitioner characteristics are presented in Table 1. All physicians were pediatricians.

Table 1.

Practitioner Characteristics.

Primary practitioner type	Physicians (98%)
Mean age	43.5 years (SE = 0.7)
Average time since training	12.9 years (SE = 0.7)
Race	White (86%)
Sex	Female (64%)
Practice location	Suburban (55%)
	Urban (28%)
	Rural (9%)
	Unspecified (8%)
Practice type	Non-academically affiliated (75%)
	Academically affiliated (25%)
Previous participation in ADHD quality improvement activities	8%

Quality indicators used in both the self-report questionnaire and chart reviews were selected based upon key components of the AAP’s most recent ADHD guideline, such that deviation from this guideline would indicate lower-quality care. All practitioners completed the self-report questionnaire in which they estimated the frequency of their ADHD care behaviors consistent with the AAP ADHD guidelines for the assessment and treatment of ADHD. The questionnaire was based upon the survey tool created by Kwasman, Tinsley, and Leper (1995) and modified to include self-report of ADHD care behaviors.

A random sample of at least 10 charts per practitioner was selected by billing audit of ADHD cases seen in the last year. To review both assessment and treatment behavior, at least five charts per practitioner with an ADHD assessment were selected. A total of 1,599 charts of children in Grades 1 to 5 were reviewed for the presence or absence of ADHD care behaviors across all 188 practitioners. There was overlap between assessment and treatment charts. As these chart reviews required a retrospective review of patient charts, a waiver of consent was granted on the condition that we abstracted only information related to quality of ADHD care and no identifying or demographic information from the patient charts be recorded. Research assistants were trained to use a standardized chart review procedure using a structured form. In total, 10% of charts were double rated to ensure consistency (intra-class correlation = 0.86 for days to first contact; kappa = 0.71 for all other variables). Chart review data included the percentage of patients for whom parent or teacher ADHD rating scales were collected during assessment, the percentage of patients for whom documentation of Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV; American Psychiatric Association, 1994) ADHD criteria was present during assessment, the time to first contact after the initiation of medication, and the percentage of patients for whom parent or teacher ADHD rating scale were used to monitor treatment response. Documentation of DSM-IV criteria was defined as the presence of completed parent and teacher rating scales or the presence of an assessment report from a specialty provider. Time to first contact after the initiation of medication was determined by comparing the date of first prescription (according to chart review) with date of first contact via either an in-person encounter, e-mail, or telephone call. For the purposes of the study, the use of ADHD rating scales during treatment was fulfilled if any ADHD rating scale completed after being prescribed medication was in the chart during the first year of treatment.

Chart review data about each practitioner were compared against the practitioner’s self-reports on the same quality domains. To compare practitioner self-reports against chart review results, we calculated the difference between them. We then tested the null hypothesis that the mean of the difference scores equaled zero. These tests were conducted using a linear mixed-effects model that tested the null hypothesis that the mean of the practitioners’ difference scores—that is, the intercept of a linear mixed model—was zero versus a two-sided alternative hypothesis. The physician self-report data were modeled to reflect nesting at the practice level using a random intercept associated with the practice to account for variance attributable to practices. The patient-level chart review data were modeled to reflect nesting at the physician and practice level. Analyses were conducted using the linear mixed-effects model function in the non-linear mixed-effects package of R version 3.0 (www.r-project.org).

Results

Practitioners overreported the quality of care in every category of performance as compared with chart review (Table 2). Practitioners overestimated use of parent and teacher rating scales during assessment by 51% and 47%, respectively, as compared with chart review. Also, practitioners overestimated the use of parent and teacher rating scales during treatment by 241% and 332%, respectively, as compared with chart review. There was a 12% overestimation in the use of DSM-IV criteria to diagnose ADHD comparing practitioner self-report to chart review. Chart review estimates of days to contact with families were nearly three times longer than estimated by self-report. All of the differences between data obtained by practitioner self-report and chart review reached statistical significance at the p < .05 level.

Table 2.

Performance as Reported by Practitioner Self-Report and Chart Review.

Performance measure	Self-report			Chart review			Difference between self-report and chart review means	Significance level, p	ES (95th CI)
Performance measure	Number of practitioners	M	SE	Number of charts	M	SE	Difference between self-report and chart review means	Significance level, p	ES (95th CI)
Parent scales, assessment (%)	176	85.4	1.8	1,098	56.7	3.7	28.7	.000	0.25 [0.09, 0.41]
Teacher scales, assessment (%)	178	81.8	2.0	1,098	55.5	3.6	26.3	.000	0.24 [0.08, 0.40]
DSM criteria (%)	159	78.7	2.2	708	70.4	3.4	8.3	.014	0.10 [−0.07, 0.27]
Days to contact (days)	179	26.1	2.2	1,405	72.4	5.4	46.3	.000	0.24 [−0.40, −0.09]
Parent scales, treatment monitoring (%)	173	36.8	2.7	784	10.8	2.7	26.0	.000	0.37 [0.21, 0.54]
Teacher scales, treatment monitoring (%)	172	32.4	2.3	784	7.5	1.5	25.5	.000	0.62 [0.45, 0.79]

Note. ES = effect size; CI = confidence interval; DSM = Diagnostic and Statistical Manual of Mental Disorders.

Conclusion

Our study design allowed us to directly compare individual practitioner’s self-report of ADHD care with actual care as determined by patient chart review, thus highlighting discrepancies between perceived and actual rates of ADHD care. We found that practitioners appeared to estimate some practices more accurately (e.g., use of DSM-IV criteria; 8.3% discrepancy between self-report and chart review) than others (e.g., days to first contact; 46.3% discrepancy). It is possible that certain contextual factors at the physician, practice, or patient level could moderate these differences. However, higher rates of ADHD care were evident based on self-report compared with chart review across all ADHD assessment and treatment monitoring practices. Given the results of this study, as well as similar results found by other investigators in other disease states, we suggest that practitioner self-report of ADHD guideline adherence should not be considered a valid measure of performance.

Accurate documentation of care is essential if we are to improve ADHD care, as we cannot change care unless we can measure it. Practitioners are thus faced with multiple questions: (a) how to efficiently document and assess the care that they provide, (b) how to receive feedback about their performance in a timely manner, and (c) how to improve suboptimal care. Although chart reviews are more accurate than self-report, they are cumbersome and time-consuming for busy practitioners. Manual patient registries and tracking systems can be similarly challenging. Electronic Health Records (EHR) hold promise but pose particular difficulties with regard to obtaining teacher information, as most EHRs are unable to effectively process information from external sources. Moreover, extracting data from EHRs has proven to be difficult in practice, as it can still be time-intensive and is “dependent on the resources of the organization to pull and analyze the data” (Baum, Epstein, & Kelleher, 2013, p. 371). Finally, quality measures calculated from data extracted from EHRs by computers often correlate poorly with measures calculated from data extracted by human chart reviewers (Gardner et al., 2014).

It is likely that novel technologies will be necessary to meet this need. Web portals allow parents and teachers to document rating scale responses directly, can provide rapid aggregation of quality indicators to allow for practitioner feedback, and can automate certain aspects of care (e.g., e-mailing parent and teacher reminders to complete rating scales; Epstein et al., 2011). Automated reminder systems in web portals or EHRs may be especially helpful to improve care behaviors that occur infrequently, such as the use of parent and teacher rating scales during treatment. Internet-based applications or “apps” are also becoming a popular tool for both documentation and patient self-management (Mosa, Yoo, & Sheets, 2012). Although a clear solution to accurately and efficiently capture care provided has not yet emerged, it is necessary for efforts to improve ADHD care to succeed.

Practitioners must also address opportunities for improvement once they are identified. Clinical practice guidelines provide a benchmark against which quality care can be measured, and strategies from the field of quality improvement are increasingly being applied to help practitioners achieve guideline-based care (American Board of Pediatrics, n.d.; Epstein, Langberg, Lichtenstein, Kolb, & Simon, 2013). Pay for performance strategies that reward practitioners for achieving quality indicators may provide additional incentives. Ultimately, however, system change will be necessary to help practitioners analyze their practice patterns and institute process changes to support improved care.

Our study contains certain limitations. Participating practitioners were recruited to participate in an ADHD quality improvement effort in Ohio, such that our sample may be biased toward a group of practitioners interested in improving care, and generalization to care provided in other areas may be limited. Data collected by chart reviews may underrepresent practitioner performance, as only care documented in the medical record can be captured. Hence, the relatively lower rates of care obtained through chart review may not reflect overreporting but may reflect underdocumentation of care. Analysis of patient, practice, and physician characteristics that may affect the magnitude of the discrepancy between physician self-report and chart review were outside the scope of our manuscript but may be explored as a future direction. Last, our chart review definition of the use of DSM-IV criteria (i.e., documentation of parent and teacher rating scale or presence of a specialty assessment) may not be fully reflective of actual practice (i.e., reviewing ADHD symptoms during the parent history). This definition may have artificially increased chart review estimates of the use of DSM-IV criteria, leading to a smaller discrepancy between self-report and chart review results.

Regardless of these limitations, the findings in this study suggest that improvements in community-based ADHD care are warranted and that practitioner self-report should not be considered a valid measure of performance. Further study will be needed to determine the most effective and efficient ways for practitioners and other stakeholders to monitor and improve performance.

Footnotes

Authors’ Note

The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health (NIH).

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Jeffery N. Epstein, PhD, is the developer of the ADHD web portal () and, along with his medical institution (Cincinnati Children’s Hospital Medical Center), owns this intellectual property.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by Grant R01 MH083665 from the National Institute of Mental Health and Grant UL1 TR000077 from the National Center for Advancing Translational Sciences of the National Institutes of Health (NIH). Drs. Epstein and Brinkman are supported by Grants K24MH064478 and K23MH083027 from the National Institute of Mental Health, respectively.

Author Biographies

Megan K. Gordon is a graduate student at the University of Illinois at Chicago College of Nursing and previously worked as a Study Coordinator for the Center for Innovation in Pediatric Practice at Nationwide Children’s Hospital. Her current research interests include postpartum care, family planning, and stigma related to both HIV and Tuberculosis.

Rebecca A. Baum, MD, is a Developmental Behavioral Pediatrician at Nationwide Children’s Hospital and a Clinical Assistant Professor of Pediatrics at The Ohio State University College of Medicine. Dr. Baum’s research interests include mental health service delivery, the integration of mental health and primary care, and dissemination of best practice care.

William Gardner is a Psychologist and Statistician. He is the University of Ottawa and Children’s Hospital of Eastern Ontario Chair in Child and Adolescent Mental Health Research.

Kelly J. Kelleher is ADS Professor of Pediatrics and Psychiatry at Nationwide Children’s Hospital and The Ohio State University. He studies the quality of health and mental health services to children, adolescents, and their families affected by poverty, violence and behavioral health issues.

Joshua M. Langberg, PhD, is an Associate Professor in the Department of Psychology at Virginia Commonwealth University (VCU) where he serves as co-director for the Center for ADHD Research, Education, and Service. Dr. Langberg’s research and professional interests are focused on improving the academic performance of children and adolescents with ADHD and on dissemination and implementation of interventions for youth with ADHD in community and school settings.

William B. Brinkman, MD, MEd, MSc, is an Associate Professor of Pediatrics at Cincinnati Children’s Hospital Medical Center. His research focuses on family engagement in the treatment of ADHD.

Jeffery N. Epstein, PhD, is Professor of Pediatrics in the Division of Behavioral Medicine and Clinical Psychology at Cincinnati Children’s Hospital Medical Center. He is also the Director of the Cincinnati Children’s Center for ADHD. Much of his research has focused on the promotion of evidence-based ADHD care in community settings.

References

Adams

A. S.

Soumerai

S. B.

Lomas

Ross-Degnan

(1999). Evidence of self-report bias in assessing adherence to guidelines. International Journal for Quality in Health Care, 11, 187-192.

American Academy of Pediatrics. (2000). Clinical practice guideline: Diagnosis and evaluation of the child with attention deficit/hyperactivity disorder. Pediatrics, 105, 1158-1170.

American Academy of Pediatrics. (2001). Clinical practice guideline: Treatment of the school-aged child with attention deficit/hyperactivity disorder. Pediatrics, 108, 1033-1044.

American Academy of Pediatrics. (2011). ADHD: Clinical practice guideline for the diagnosis, evaluation, and treatment of attention-deficit/hyperactivity disorder in children and adolescents. Pediatrics, 128, 1007-1022.

American Board of Pediatrics. (n.d.). ABP Performance Improvement Modules (PIMS). Retrieved from https://www.abp.org/content/abp-performance-improvement-modules-pims

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.

Baum

R. A.

Epstein

J. N.

Kelleher

(2013). Healthcare reform, quality, and technology: ADHD as a case study. Current Psychiatry Reports, 15, 369-375.

Bruchmüller

Margraf

Schneider

(2012). Is ADHD diagnosed in accord with diagnostic criteria? Overdiagnosis and influence of client gender on diagnosis. Journal of Consulting and Clinical Psychology, 80, 128-138.

Chan

Hopkins

M. R.

Perrin

J. M.

Herrerias

Homer

C. J.

(2005). Diagnostic practices for attention deficit hyperactivity disorder: A national survey of primary care physicians. Ambulatory Pediatrics, 5, 201-208.

10.

Davis

D. A.

Mazmanian

P. E.

Fordis

Van Harrison

Thorpe

K. E.

Perrier

(2006). Accuracy of physician self-assessment compared with observed measures of competence: A systematic review. Journal of the American Medical Association, 296, 1094-1102.

11.

Epstein

J. N.

Langberg

J. M.

Lichtenstein

P. K.

Kolb

Altaye

Simon

J. O.

(2011). Use of an Internet portal to improve community-based pediatric ADHD care: A clustered randomized trial. Pediatrics, 128, e1210-e1217.

12.

Epstein

J. N.

Langberg

J. M.

Lichtenstein

P. K.

Kolb

Simon

J. O.

(2013). The myADHDportal.com improvement project: An innovative quality improvement intervention for improving the quality of ADHD care among community-based pediatricians. Clinical Practice in Pediatric Psychology, 1, 55-67.

13.

Epstein

J. N.

Langberg

J. M.

Lichtenstein

P. K.

Mainwaring

B. A.

Luzader

C. P.

Stark

L. J.

(2008). Community-wide intervention to improve the attention-deficit/hyperactivity disorder assessment and treatment of community physicians. Pediatrics, 122, 19-27.

14.

Gardner

Kelleher

K. J.

Pajer

Camp

J. V.

(2004). Follow-up care of children identified with ADHD by primary care clinicians: A prospective cohort study. Journal of Pediatrics, 145, 767-771.

15.

Gardner

Morton

Byron

Tinoco

Canan

Leonhart

. . .Scholle

S. H.

(2014). Using computer-extracted data from electronic health records to measure the quality of adolescent well-care. Health Services Research, 49, 1226-1243.

16.

Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the 21st century. Washington, DC: The National Academies Press.

17.

Kwasman

Tinsley

B. J.

Leper

H. S.

(1995). Pediatricians’ knowledge and attitudes concerning diagnosis and treatment of attention deficit and hyperactivity disorders: A national survey approach. Archives of Pediatric & Adolescent Medicine, 149, 1211-1216.

18.

McElligott

J. T.

Lemay

J. R.

O’Brien

E. S.

Roland

V. A.

Basco

W. T.

Jr. Roberts

J. R.

(2014). Practice patterns and guideline adherence in the management of attention deficit/hyperactivity disorder. Clinical Pediatrics, 53, 960-966.

19.

Mosa

A. S. M.

Yoo

Sheets

(2012). A systematic review of healthcare applications for smartphones. BMC Medical Informatics and Decision Making, 12, Article 67. doi:10.1186/1472-6947-12-67

20.

Perou

Bitsko

R. H.

Blumberg

S. J.

Pastor

Ghandour

R. M.

Gfroerer

J. C.

. . .Huang

L. N.

(2013). Mental health surveillance among children—United States, 2005-2011. Morbidity and Mortality Weekly Report, 62(02), 1-35.

21.

Rushton

J. L.

Fant

K. E.

Clark

S. J.

(2004). Use of practice guidelines in the primary care of children with attention-deficit/hyperactivity disorder. Pediatrics, 114, e23-e28.

22.

U.S. Department of Health and Human Services. (n.d.). The Affordable Care Act, section by section. Retrieved from http://www.healthcare.gov/law/full/index.html