An Audit and Feedback Intervention to Improve Internal Medicine Residents' Performance on Ambulatory Quality Measures: A Randomized Controlled Trial

Abstract

Audit and feedback is an effective method to improve attending physician performance. However, there are limited data on how audit and feedback impacts care provided by resident physicians. The authors conducted a 3-arm randomized clinical trial among internal medicine resident physicians to examine the impact of an audit and feedback intervention on ambulatory quality measures (AQMs). Residents in all 3 groups received an email containing the contact information of a population health coordinator and a list of AQMs (control). In addition, the Practice Target group received individual AQM data compared to the target AQM goals for all primary care practices. The Peer Comparison group received information on individual AQM data compared to the average performance of residents in the same postgraduate year. Residents in each intervention group received updated information 6 months later. Ten AQMs related to diabetes care, hypertension management, lipid control, and cancer screening, as well as a composite quality score, were examined at baseline, 6 months, and 13 months. At 13 months follow-up, the Practice Target group had statistically significant improvement in cervical cancer screening rate (77% vs. 65.3%), colorectal cancer screening rate (72.5% vs. 64.6%), and composite quality score (71.7% vs 65.4%) compared to baseline. Providing internal medicine residents with individual AQMs data compared to target goal for the practice led to statistically significant improvement in cancer screening rates and the composite quality score. Audit and feedback may be a relatively simple yet effective tool to improve population health in the resident clinic setting.

Introduction

Previous studies have demonstrated that patients of resident physicians have worse outcomes on ambulatory quality measures (AQMs), such as cancer screening rates and blood pressure control, when compared to patients of staff physicians.^1

–4 As all internal medicine residency programs in the United States are mandated by the Accreditation Council for Graduate Medical Education to offer longitudinal continuity clinic experiences, there is an imperative to address this disparity in quality of care. As the health care system transitions from fee for service to value-based payment systems, addressing these quality gaps also will become increasingly important for reimbursement.

Audit and feedback has been shown to be an effective method to improve physician performance.⁵ However, there are limited data on how audit and feedback impacts care provided by resident physicians, as previous studies were limited to patients with specific comorbidities⁶ and small sets of preventive measures,⁷ or had a small sample size.⁸ The study team hypothesized that providing internal medicine residents with data on their panels' AQMs performance compared to practice-wide targets or to their peers' panel data would lead to improved outcomes. The team anticipated greater improvement among residents who were provided with peer data as behavioral economics research has demonstrated that physician behaviors are influenced by their perception of relative social ranking.⁹

Methods

Study design

This study was a 3-arm randomized clinical trial conducted among resident physicians from the Massachusetts General Hospital (MGH) internal medicine residency program. The trial was conducted between May 2017 and June 2018. The institutional review board of Partners HealthCare approved the study.

Participants and interventions

The Microsoft Excel (Microsoft Corporation, Redmond, WA, USA) RAND function was used to randomize 96 first- and second-year internal medicine residents at MGH (Boston, MA) into 3 groups: (1) Control, (2) Practice Target, and (3) Peer Comparison (Fig. 1). All 3 groups received an email containing information on how to contact a population health coordinator and a list of AQMs and their definitions. Population health coordinators assigned to each primary care practice were available to assist residents in identifying patients who were overdue or not at goal for a certain AQM, schedule follow-up appointments and tests, and obtain home blood pressure values and outside tests or laboratory results.

FIG. 1.

CONSORT diagram

Additionally, the Practice Target group received information on individual AQMs performance compared to the target AQMs goals set for all MGH primary care practices. The Peer Comparison group received information on individual AQM performance compared to the average panel performance for residents in the same postgraduate training year. Residents in each group received email follow-up with updated information 6 months later. Residents graduating during the academic year ending within 1 month of baseline (May 2017) were excluded from the study given the lack of an adequate follow-up period.

Outcome measures

The primary outcome measure was change in performance on composite quality score at 6 months (November 2017) and 13 months follow-up (June 2018) in each group. The composite quality score was calculated as the number of quality metrics for which patients were at goal divided by the number of quality metrics for which patients were eligible. The secondary outcome measure was change in performance on 10 individual AQMs related to diabetes care, hypertension management, lipid management in patients with cardiovascular disease, and colorectal, breast, and cervical cancer screening (Table 1).

Table 1.

Ambulatory Quality Measures Definitions

Quality measure		Goals
Diabetes	Hemoglobin A1c (HbA1c)	HbA1c ≤9 and checked within 6 months.
	Lipid control	Patient is on moderate or high-dose statin. If not on a moderate or high-dose statin, LDL must be done in the past 12 months and LDL is <100.
	Blood pressure (BP)	Last BP or average of last 3 blood BPs within the past 18 months ≤140/90 or on 3 antihypertensive medications. If patient is ≥60 years old, also pass if last diastolic blood pressure or average of last 3 diastolic blood pressures ≤70.
	Urine microalbumin	Checked every 12 months or on angiotensin-converting enzyme inhibitor/angiotensin receptor blocker.
	Eye exam	Retinal or dilated eye exam by an eye care professional in the last 24 months.
Hypertension	BP	Patients <60 years and all patients with diabetes: At goal if last BP ≤140/90 or average of last 3 BPs ≤140/90 within the past 18 months or on 3 antihypertensive medications.
Hypertension	BP	Patients ≥60 years old (unless diabetic): At goal if last BP ≤150/90 or diastolic BP ≤70 or average of last 3 BPs ≤150/90, or diastolic BP <70 within the past 18 months or on 3 antihypertensive medications.
Cardiovascular disease (coronary artery disease, cerebral vascular accident and peripheral vascular disease)	Lipid control	If patient is on a high-dose statin, automatically passes measure regardless of LDL value. If patient is not on a high-dose statin, the patient passes if LDL was done in the past 12 months and LDL is <100.
Cancer screening	Breast cancer	Every 2 years for patients ages 50–74 years unless health maintenance modifier for annual mammography is added to health maintenance by primary care team.
	Cervical cancer	PAP smear every 3 years for patients ages 21–65 years unless health maintenance modifier for annual, 2-, or 5-year interval is added to health maintenance by primary care team.
	Colorectal cancer	Colonoscopy every 10 years or virtual colonography, sigmoidoscopy with FIT co-testing every 5 years, or FIT or FOBT annually for patients ages 50–75 years unless health maintenance modifier for more frequent screening is added by primary care team.

BP, blood pressure; FIT, fecal immunochemical test; FOBT, fecal occult blood test; LDL, low-density lipoprotein cholesterol; PAP, Papanicolaou.

Statistical analysis

Data on residents and their patients were obtained from the electronic health record (EHR). Resident variables assessed include sex, postgraduate year, continuity clinic site, and panel size. Patient characteristics including demographics (age, sex, race), comorbidities (diabetes, hypertension, cardiovascular disease, depression), Charlson score, and acute care utilization were examined. The study team compared patient and resident characteristics using chi-square tests, analysis of variance, or Kruskal-Wallis tests as appropriate. Mixed effects multivariable regression models were used for both intragroup (comparing postintervention AQM performance over time within each group) and intergroup contrasts (comparing among the 3 groups at each follow-up period), adjusting for patient characteristics and acute care utilization, while taking physician clustering into account. The analysis was restricted to patients who remained part of the residents' panels throughout the study period to ensure that residents had adequate time to work with patients on the quality metrics. Statistical significance was defined as 2-sided P ≤ 0.05. All statistical analyses were conducted using SAS version 9.4 software (SAS Institute Inc., Cary, NC, USA).

Results

Baseline comparison

A total of 96 first and second year internal medicine residents and 6363 patients were randomized (Fig. 1). The Peer Comparison group had higher baseline colorectal cancer screening rates than the Control group (67.7% vs. 59.6%, P = 0.02) while the Control group had the highest mean number of hospitalizations (P < 0.01) and emergency department visits (P < 0.01) among the 3 groups (Tables 2 and 3). There were no other significant differences in patient and resident physician characteristics (Table 2), or baseline AQM performance among the 3 groups (Table 3).

Table 2.

Baseline Patient and Resident Characteristics

Patient characteristics	Control (N = 1858)	Practice target (N = 2056)	Peer comparison (N = 2449)	P
Age, mean (SD)	49.4 (17.6)	48.3 (17.1)	48.5 (17.4)	0.11
Female, n (%)	888 (47.8)	1014 (49.3)	1237 (50.5)	0.21
White, n (%)	1330 (71.6)	1482 (72.1)	1679 (68.6)	0.054
Diabetes, n (%)	269 (14.5)	294 (14.3)	315 (12.9)	0.23
HTN, n (%)	687 (37.0)	708 (34.4)	842 (34.4)	0.15
Cardiovascular disease, n (%)	233 (12.5)	247 (12.0)	290 (11.8)	0.78
Depression, n (%)	272 (14.6)	314 (15.3)	319 (13.0)	0.082
Charlson score, mean (SD)	1.21 (1.90)	1.04 (1.65)	1.01 (1.71)	0.082
Number of hospitalizations, median (SD)	0.18 (0.73)	0.14 (0.55)	0.11 (0.47)	0.0004
Number of ED visits, mean (SD)	0.47 (1.40)	0.42 (1.52)	0.34 (1.09)	0.001

Resident characteristics	Control (N = 30)	Practice target (N = 32)	Peer comparison (N = 34)	P
Female, n (%)	14 (46.7)	18 (56.3)	15 (44.1)	0.59
PGY-1, n (%)	19 (63.3)	18 (56.3)	17 (50.0)	0.56
Community health center, n (%)	8 (26.7)	6 (18.8)	5 (14.7)	0.48
Panel size, mean (SD)	73.2 (17.7)	72.6 (18.0)	82.8 (26.7)	0.10

ED, emergency department; HTN, hypertension; PGY, postgraduate year; SD, standard deviation.

Table 3.

Baseline Ambulatory Quality Measures Performance

AQM measure		Control	Practice target	Peer comparison
Diabetes, n (%)	Hemoglobin A1c	161 (59.9)	156 (53.1)	185 (58.7)
	Lipid control	199 (74.0)	220 (74.8)	236 (74.9)
	Blood pressure	199 (74.0)	226 (76.9)	244 (77.5)
	Nephropathy	215 (79.9)	243 (82.7)	268 (85.1)
	Retinopathy	180 (66.9)	201 (68.4)	232 (73.7)
Hypertension, n (%)	Blood pressure	471 (68.7)	503 (71.1)	597 (70.9)
Cardiovascular disease, n (%)	Lipid control	166 (71.2)	186 (75.3)	216 (74.5)
Cancer screening, n (%)	Breast cancer	196 (63.8)	227 (71.6)	298 (71.8)
	Cervical cancer	457 (67.1)	515 (65.3)	656 (68.0)
	Colorectal cancer	439 (59.6)	499 (64.6)	628 (67.7)^*
Quality composite, mean (SD)		0.648 (0.405)	0.654 (0.411)	0.673 (0.399)

P = 0.02 when compared to control group after adjusting for baseline patient characteristics.

AQM, ambulatory quality measure; SD, standard deviation.

Intragroup comparisons

For the Control group, there was significant improvement in blood pressure control at 6 months compared to baseline (73.6% vs. 68.7%, P = 0.03) followed by a significant decline at 13 months (73.6% vs. 70.6%, P = 0.01) among patients with hypertension (Fig. 2). There also was significant improvement in colorectal cancer screening rates at 6 months (64.7%, P = 0.03) that was sustained at 13 months (65.6%, P = 0.049) when compared to baseline (59.6%). In addition, the composite quality score was significantly higher at 6 months compared to baseline (67.6% vs 64.8%, P = 0.047), but there was no significant difference at 13 months (67.1%). Lastly, there was a significant decline in retinopathy screening rates at 13 months compared to baseline (56.3% vs 66.9%, P < 0.01) among diabetes patients (Fig. 2).

FIG. 2.

Intragroup ambulatory quality measure performance comparison at 6 months and 13 months follow-up. *P < 0.05. CVD, cardiovascular disease; DM, diabetes mellitus; HTN, hypertension.

For the Practice Target group, there was significant improvement in cervical cancer screening rates compared to baseline (65.3%), both at 6 months (68.1%, P < 0.01) and 13 months (77.0%, P < 0.01) (Fig. 2). Colorectal cancer screening rates and composite quality score were significantly higher at 13 months compared to baseline (72.5% vs 64.6%, P < 0.01 for colorectal cancer screening and 71.7% vs 65.4%, P < 0.01 for composite quality score). Hemoglobin A1c control and retinopathy screening both improved at 6 months compared to baseline (65.0% vs. 53.1%, P < 0.01 for A1c and 80.0% vs. 68.4%, P = 0.03 for retinopathy), but the improvement was not sustained for A1c control and rates declined for retinopathy screening at 13 months (Fig. 2).

For the Peer Comparison group, there was significant decline in hemoglobin A1c control at 13 months compared to 6 months (55.9% vs. 62.7%, P = 0.03) (Fig. 2). Retinopathy screening rate was significantly lower at 13 months (60.7%) when compared to both baseline (73.7%, P < 0.01) and 6 months follow-up (75.5%, P < 0.01) (Fig. 2).

Intergroup comparisons

The Practice Target group had a significantly higher diabetic retinopathy screening rate (80.0%) than Control (71.6%, P = 0.04) at 6 months but the difference was not sustained at 13 months. There was a trend toward statistically significant difference in cervical cancer screening rates (69.3% vs 77%, P = 0.057) and composite quality score (67.1% vs 71.7%, P = 0.086) between the Control and Practice Target groups at 13 months, but no statistically significant differences were found in other AQMs either between the Control and intervention groups or between the Practice Target and Peer Comparison groups.

Discussion

This study examined the impact of an audit and feedback intervention on the quality of primary care provided by resident physicians. Although the control group demonstrated short-term improvement in panel AQM performance that was not sustained in the long term, providing audit and feedback in the form of individual panel performance data compared to practice targets led to statistically significant improvement in cancer screening rates and composite quality score after 13 months compared to baseline. The differences in AQMs between the Control and intervention groups did not reach statistical significance, but an increasing gap favoring the Practice Target group was noted by the end of the study, specifically for cervical cancer screening rates and composite quality score. Given that significant time, effort, and coordination are required to implement and demonstrate major clinical improvements in diabetes, hypertension, and lipid management as well as cancer screening rates, future studies should examine whether a longer follow-up period leads to greater differences in performance when audit and feedback interventions are implemented.

Contrary to the study hypothesis, peer comparison did not lead to greater improvement in performance. Although there were statistically significant improvements in cervical and colorectal cancer screening rates and composite quality score at 13 months for the Practice Target group, there were no statistically significant improvements in AQMs at 13 months compared to baseline in the Peer Comparison group. A possible explanation for this finding is that the average panel AQM performance for residents in a given postgraduate year often was markedly lower than the set practice-wide target. This finding highlights the importance of pairing individual performance data with an appropriate comparator that will serve as a motivator rather than underscore poor average performance as a group. A prior randomized controlled trial demonstrated the effectiveness of using the achievable benchmark, which is the average performance of the top 10% of physicians, in improving quality.¹⁰ Therefore, future studies should examine whether providing similar data on high performers leads to a greater improvement in performance among resident physicians.

Given the US health system's growing emphasis on population health management and value-based reimbursement, and the ubiquity of resident primary care clinics in the graduate medical education community, these findings have important care delivery implications. Resident patient panels that often underperform on quality metrics are logical targets for population management interventions. Because this intervention consisted of a simple email containing information that can be easily extracted from the EHR, it can be readily replicated.

Data provided to residents must be accurate, timely, and personalized, as physicians do not respond positively to feedback that is perceived to be based on old, inaccurate information or unrealistic measures.¹¹ In 2014, Partners Healthcare (Boston, MA) transitioned away from traditional claims-based Healthcare Effectiveness Data and Information Set measures to a set of internally-defined quality measures that are derived from the EHR and deemed to be most clinically relevant. This change made it possible to track quality of care for the entire patient population and allow real-time feedback to practices and providers. Feedback also must be provided in a nonpunitive manner to avoid generating negative emotions or defensive behavior. Despite carefully worded messages to assure residents that AQM data were not being provided for evaluative purposes, several residents in this study replied to explain why their panel performance may be lower than target or their peers.

Another important feature of effective feedback is that it is actionable. For example, a prior study showed that use of an online dashboard that allowed physicians to not only identify patients who meet criteria for statin therapy but also to conveniently prescribe atorvastatin led to statistically significant improvement in statin prescribing rates.¹² Therefore, future study should consider ways to facilitate provider actions to address gaps in AQMs through the use of an integrated tool in the EHR that allows resident physicians to both identify and address AQM deficiencies (eg, to batch order a screening colonoscopy, schedule a blood pressure check).

In addition, audit and feedback interventions are more effective when feedback is provided more frequently, presented in both verbal and written format, and delivered by a supervisor or senior colleague.⁵ Given these findings, residency programs planning to implement an audit and feedback intervention should aim to provide feedback more than once a year and have supervisors (eg, program directors, preceptors) deliver verbal feedback in addition to providing written reports on residents' performance.

This study has several limitations. First, the study was restricted to residents in 1 internal medicine training program and therefore may not be generalizable to other settings. However, residents at the study institution provide primary care in diverse settings, including community health centers, large academic practices, and suburban clinics, which increases the generalizability of the findings. Second, there was a limited follow-up period given the turnover in resident providers who graduate from the program after 2 or 3 years. Third, because the data were obtained from the EHR of one health system, labs, blood pressure measurements, and other tests that were performed outside of the network may not have been captured. However, this is mitigated by the fact that the study was randomized and missing information likely was balanced across arms.

In summary, this study found that providing internal medicine residents with individual AQM performance data in comparison to set practice targets led to statistically significant improvement in composite quality scores. Given that health systems are increasingly being held accountable for quality measures, audit and feedback may be a relatively simple yet effective tool to improve population health.

Footnotes

Acknowledgment

The authors would like to thank Steve Condon for his help with data acquisition.

Author Disclosure Statement

The authors declare that there are no conflicts of interest. The authors received no financial support for this study.

References

Lynn

, Hess

, Weng

, Lipner

, Holmboe

. Gaps in quality of diabetes care in internal medicine residency clinics suggest the need for better ambulatory care training. Health Aff (Millwood), 2012; 31:150–158.

Essien

, Abraham

, Ray

, He

, Atlas

. Disparities in quality of primary care by residents and staff physicians: is there a conflict between training needs and equity in care?. Paper presented at 2017 AcademyHealth Annual Research Meeting, June 26, 2017, New Orleans, LA.

Cardozo

, Steinberg

, Lepczyk

, Binns-Emerick

, Cardozo

, Aranha

. Improving preventive health care in a medical resident practice. Arch Intern Med, 1998; 158:261–264.

Zack

, DiBaise

, Quigley

, Roy

. Colorectal cancer screening compliance by medicine residents: perceived and actual. Am J Gastroenterol, 2001; 96:3004–3008.

Ivers

, Jamtvedt

, Flottorp

, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev, 2012; CD000259.

Thomas

, Thomas

, Stroebel

, et al. Use of a registry-generated audit, feedback, and patient reminder intervention in an internal medicine resident clinic—a randomized trial. J Gen Intern Med, 2007; 22:1740–1744.

Holmboe

, Scranton

, Sumption

, Hawkins

. Effect of medical record audit and feedback on residents' compliance with preventive health care guidelines. Acad Med, 1998; 73:901–903.

Kogan

, Reynolds

, Shea

. Effectiveness of report cards based on chart audits of residents' adherence to practice guidelines on practice performance: a randomized controlled trial. Teach Learn Med, 2003; 15:25–30.

Emanuel

, Ubel

, Kessler

, et al. Using behavioral economics to design physician incentives that deliver high-value care. Ann Intern Med, 2016; 164:114–119.

10.

Kiefe

, Allison

, Williams

, Person

, Weaver

, Weissman

. Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA, 2001; 285:2871–2879.

11.

Payne

, Hysong

. Model depicting aspects of audit and feedback that impact physicians' acceptance of clinical performance feedback. BMC Health Serv Res, 2016; 16:260.

12.

Patel

, Kurtzman

, Kannan

, et al. Effect of an automated patient dashboard using active choice and peer comparison performance feedback to physicians on statin prescribing: the PRESCRIBE cluster randomized clinical trial. JAMA Netw Open, 2018; 1:e180818.