An Adult Immunization Best Practices Learning Collaborative: Impact,Scale Up,and Spread

Abstract

The research objective was to rapidly scale up and spread a proven learning collaborative approach (intervention) for adult vaccination rates for influenza and pneumococcal disease from 7 to 39 US health care organizations and to examine improvement in adult immunization rates after scale-up. Comparative analyses were conducted between intervention and nonintervention propensity score-matched providers on vaccination rates using a difference-in-differences approach. Qualitative data, collected during site visits and in-person and virtual meetings, were used to enhance understanding of quantitative results. In 2017–2018, an analysis of a subset of sites (n = 9) from 2 intervention cohorts (∼20 sites each) demonstrated greater improvement than their matched providers in pneumococcal vaccinations (PV) for patients ages ≥65 years (treatment effect range: 1.4%-3.7%, P < 0.01) and PV for high-risk patients (eg, with immunocompromising conditions) aged 19–64 years (0.8%-1.6%, P < 0.01). Significant effects were observed in one of the study cohorts for PV for at-risk patients (eg, with diabetes) aged 19–64 years (1.7%, P < 0.01), and influenza vaccination rates (2.4%, P < 0.001). Individual health systems demonstrated even greater improvements across all 4 vaccinations: 9.5% influenza; 8.7% PV ages ≥65 years; 11.8% PV high-risk; 16.3% PV at-risk (all P < 0.01). Results demonstrated that a 7-site pilot could be successfully scaled to 39 additional sites, with similar improvements in vaccination rates. Between 2014 and 2018, vaccination improvements among all 46 groups (7 pilot, 39 in subsequent cohorts) resulted in an estimated 5.5 million adult vaccinations administered or documented in 27 states.

Introduction

According to the Institute for Healthcare Improvement (IHI), the key to a best practice becoming a common practice is the ability of providers and health systems to rapidly spread new ideas and successful innovations.¹ IHI's Framework for Spread¹ is a useful model for identifying key components for the spread of best practices and innovations and aligns with AMGA's efforts to spread its Adult Immunization Best Practices Learning Collaborative to 39 health care organizations. This was accomplished by bringing health systems and their providers and staff together, both in-person and virtually, in an iterative learning collaborative format. This article demonstrates the scale-up and spread that followed a pilot learning collaborative in efforts to increase adult immunization rates nationwide.

According to the Centers for Disease Control and Prevention (CDC), introduction of pneumococcal conjugate vaccines (PCV) in the United States reduced invasive pneumococcal disease by 56% among adults aged 19–64 years and by 61% among adults aged ≥65 years between 1998–2015.² For the last flu season on record (2017–2018), adult influenza vaccines prevented an estimated 7 million flu illnesses, 109,000 flu hospitalizations, and 8000 flu deaths.³ Pneumococcal vaccination (PV) also reduces the likelihood of prolonged hospitalization among adults, especially those at high risk of disease.⁴ A study among patients at high risk for cardiovascular disease found that individuals who received vaccinations for influenza had significantly lower risks of having adverse cardiac events, a leading cause of death in the United States.⁵

Despite the strong evidence base for adult immunization, millions of illnesses and tens of thousands of deaths occur every year from vaccine-preventable diseases. In 2017, CDC estimated 450,000 hospitalizations related to pneumococcal pneumonia and 959,000 hospitalizations resulting in 79,400 influenza-related deaths (2017–2018 flu season).⁶ As many as 70% of influenza hospitalizations and 70%-85% of all influenza-related deaths were in older adults age ≥65 years.⁷ Estimates of the health care costs associated with influenza in adults are $5.8 billion per year, while pneumococcal disease is estimated to cost another $1.9 billion annually.⁸

Notwithstanding the pressure to provide more efficient, affordable, value-based care, gaps in immunization among adult populations persist. In the 2017–2018 flu season, CDC estimated the percentage of adults aged 18–64 vaccinated for influenza in the United States was 31% (Healthy People 2020 goal = 70%) and the percentage of adults aged ≥65 years vaccinated for influenza was ∼60% (Healthy People 2020 goal = 90%). Although there has been significant progress in reducing the gap in influenza vaccination for high-risk adults with compromised health conditions, still only 39% aged 18–64 received an influenza vaccine.⁹ The greatest gap in PV was among at-risk adults ≥19 years, with 24.5% ever having received a pneumococcal vaccine.¹⁰ The economic and disease burden associated with vaccine-preventable illnesses suggests adult immunization is an area of prevention with many opportunities to improve population health. This is particularly true for at-risk adults with health conditions that place them at higher risk of developing complexities that can lead to costly and serious health complications.

In response to this serious public health problem, AMGA conducted 3, one-year learning collaboratives, the Adult Immunization Best Practices Learning Collaboratives (“the Collaboratives”), to address barriers, share best practices, and improve pneumococcal and influenza vaccination rates. The results of the first pilot collaborative (Cohort 1) are reported elsewhere.¹¹ In an effort to scale up and spread improvements realized in the pilot program, 2 subsequent collaboratives (Cohorts 2 and 3) leveraged learnings from pilot Cohort 1, which included a formal qualitative component that identified organizational, individual, cultural, and contextual factors that influenced the success of the programs. The current study examined the ability to scale up and spread the findings from the 7-site pilot program (Cohort 1) to 39 additional health care organizations that participated in 2 subsequent collaboratives (Cohorts 2 and 3) by applying IHI's Framework for Spread.¹ Vaccination rates were assessed through a rigorous quantitative analysis using an approach that enabled true effects to be estimated by comparing providers at participating organizations with providers at other organizations with similar baseline characteristics and therefore a similar propensity to join a learning collaborative. This matching technique prevented overestimation of the impact of the intervention. An informal qualitative analysis helped support the quantitative data and validated the findings from the pilot study.

Methods

Ethics approval and consent to participate

The study was determined exempt by the New England Independent Review Board (NEIRB# WO 1-776-1) for pilot study. Further information and documentation of Institutional Review Board exemption is available upon request.

Study sample

The study population included adult patients who received health care services between 2017 and 2018 at 9 large US health care organizations in 7 states, including independent medical groups and integrated delivery systems, that participated in the 2 Collaboratives, described in the Intervention section), and patients from 29 matched-control organizations. These 9 health care organizations are a subset of the 39 organizations that participated in the Collaboratives and were selected based on patient- and provider-level data availability in the Optum^® database. Study sites used a shared Optum population health analytics platform and contributed data to the common data repository (CDR) (described in the Data Source section). Table 1 provides descriptive information about each cohort. The 9 organizations included in the analysis reflected the characteristics of the 30 remaining organizations in terms of size, structure (eg, integration), and baseline immunization rates. In other words, the 9 study organizations were representative of the remaining 30. Three organizations participated in both Cohort 1 and either Cohort 2 or 3.

Table 1.

Collaborative Cohort Descriptions¹

	Cohort 2	Cohort 3
Collaborative Intervention Periods	1/2017–3/2018²	7/2017–6/2018
Baseline Measurement Periods	1/2016–12/2016²	7/2016–6/2017
Sites, n	5	4
Providers, n	414	152
Control sites used, n	21	27
Control sites considered	23	27
Integrated Systems, %	94	53
Eligible patients, pneumococcal, n	∼541,000
Eligible patients, influenza, n	∼858,000
Provider-level data within each cohort
Average Patient Volume, n ± SD	397 ± 237	463 ± 266
Ethnic Minority Rate, % ± SD	11 ± 14	5 ± 4
Medicare Rate, % ± SD	26 ± 14	40 ± 16
Medicaid Rate, % ± SD	9 ± 8	9 ± 11
Medicare Wellness Visit Rate, % ± SD	40 ± 30	29 ± 25
Baseline ≥65 Pneumococcal Vaccine Rate, Any Vaccine, % ± SD	85 ± 12	79 ± 16
Baseline ≥65 Pneumococcal Vaccine Rate, PCV & PPSV Vaccines, % ± SD	51 ± 18	48 ± 21

For illustrative purposes, these numbers reflect the pneumococcal vaccination measure for adults ≥65 years. The rates differ slightly for each of the 4 quality measures.

For cohort 2, influenza baseline and intervention time periods were adjusted to contain measurements within a single flu season.

PCV, pneumococcal conjugate vaccine; PPSV, pneumococcal polysaccharide vaccine; SD, standard deviation.

Data source

AMGA is a nonprofit trade association representing 420 multispecialty medical groups and integrated health care delivery systems with a total of 175,000 full-time equivalent physicians. As AMGA's distinguished data and analytics collaborator, Optum provides access to data from AMGA members, many of whom use Optum Analytics' population health tools. In addition to claims data, Optum Analytics tools include clinical data from members' electronic health records (EHRs), mapped and normalized to allow valid, reliable comparisons. Detailed EHR data enable discovery of differences in care process as well as clinical outcomes. The CDR that pools longitudinal EHR data from 54 health care organizations, including records for approximately 79 million patients, was accessed for this study. Vaccination data were pulled from several areas in the EHR including medication tables, medication patient reports, vaccination tables, health maintenance tables, procedure codes, and diagnosis codes.

Study design

In order to assess the success of the scale-up and spread of a pilot learning collaborative (Cohort 1), vaccination data were examined to identify changes in adult vaccination rates over time. Provider-level propensity scores were used to match intervention to nonintervention providers to control for inherent selection bias of participating organizations. Based on a review of existing literature and subject matter expertise, 6 variables chosen for the propensity score models included baseline patient count, baseline vaccination rate(s), baseline minority patient rate, baseline Medicare patient rate, baseline Medicaid patient rate, and baseline Medicare wellness visit rate. Comparative analyses were conducted between intervention sites and nonintervention sites on vaccination rates using a difference-in-differences approach. Table 1 includes provider baseline rates by cohort. Qualitative data collected during site visits and in-person and virtual meetings helped identify system- and clinic-level interventions, and organizational factors contributing to successful programs.

Lists of participating providers were obtained from the 9 collaborative health systems. In order to be included in each provider's patient population for a given measure, the patient needed to satisfy the age restriction for the measure (age at start of time period), have a qualifying ambulatory visit during the time period, be alive for the full time period, and be documented in the EHR data as having a designated primary care provider (PCP) for the most days during the 24 months leading up to the end of the time period. For the pneumococcal high-risk and pneumococcal at-risk measures, the patient also needed to have a diagnosis for one of the high- or at-risk conditions between January 2013 and the end date of each intervention period. Table 2 displays specifications for each measure.

Table 2.

Measure Specifications

Measure	Eligible ages	Eligible conditions
1: Pneumococcal ≥65	≥65	All
2: Pneumococcal High Risk	19–64	cerebrospinal fluid leak, cochlear implants, hemoglobinopathies, asplenia, chronic renal failure, nephrotic syndrome, organ transplant, kidney disease, immunodeficiencies, leukemia, lymphoma, Hodgkin's disease, multiple myeloma, general malignancy
2a: Pneumococcal At Risk	19–64	diabetes, chronic heart conditions, lung disease, chronic liver disease, alcoholism, smoker
3: Influenza	≥18	All

Matching variable statistics were obtained for the aforementioned 6 variables used for propensity score matching. Provider-level vaccination rates were calculated for each measure. For the pneumococcal 65+ and pneumococcal high-risk measures, vaccination rates of interest included “ANY” (patients with any vaccination, including records of pneumococcal polysaccharide vaccine [PPSV] administration, PCV administration, or administration of unknown pneumococcal vaccine) and “BOTH” (patients with documentation of receiving both PPSV and PCV vaccinations). For the pneumococcal at-risk measure, vaccination rates of interest included “ANY” (patients with documented PPSV or administration of unknown pneumococcal vaccine) and “PPSV” (patients with documented PPSV). In the case of the pneumococcal 65+ measure, vaccinations were only considered if they occurred on or after the patient's 65^th birthday. For the pneumococcal high- and at-risk measures, vaccinations were only considered if they occurred during ages 19 through 64. For the influenza measure, vaccinations had to occur within the current flu season (ie, July 1 through June 30), and on or after a patient's 18^th birthday to be considered.

Minority rate was defined as the proportion of nonwhite patients; patients with unknown race were classified as white and patients reporting multiple races were classified as nonwhite. Medicare and Medicaid rates were defined as the proportion of patients who had at least 1 insurance record of the given product type or at least 1 month of eligibility (where claims data were available) during the baseline period. Patients with dual membership for both Medicare and Medicaid products were included in the numerator for both rates. Medicare wellness visit rate was defined as the proportion of patients aged ≥65 years who had at least 1 Medicare wellness visit (Healthcare Common Procedure Coding System codes G0402, G0439, and G0438) during the baseline period.

The comparison provider population was created using 29 health systems and organizations available in the database, but not participating in the adult vaccination learning collaborative. Data on the 6 matching variables were collected for all providers who saw at least 50 patients during both the baseline and intervention periods.

Propensity score matching

The propensity score was estimated using a logistic regression model, in which treatment status (collaborative participant or not) was regressed on the aforementioned 6 observed baseline characteristics. The estimated propensity score is the predicted probability of participating in the collaborative, derived from the fitted regression model. Participating collaborative providers (intervention) were one-to-one matched to non-collaborative providers (control), with replacement, on the logit of the propensity score. All providers were matched. Matching without replacement and/or with a caliper also were explored. To enable a subgroup analysis comparing integrated delivery system (IDS) providers with ambulatory providers, a block-randomized design was mimicked by propensity score matching within these 2 subgroups (IDS providers were matched to IDS providers and ambulatory providers to ambulatory providers). To allow for effect estimates at the health system level, individual propensity score models were created for each health system so that the model would predict the best matches for providers within each individual health system. Because the ultimate goal of propensity score models is balance—not satisfaction of model diagnostics—transformations and interaction terms were explored. Using these transformations and higher order terms, several propensity score models were created per health system.

For each propensity score model, 500 iterations were tested, producing different matches with each iteration. The different matches were prompted by using a “distance tolerance,” in which all control providers within a certain distance are treated as ties, and one provider is randomly selected from the tying set. This allows for greater fluctuation when matching with replacement. In the end, more than 5000 distinct matched sets were produced per health system. To evaluate the quality of the matched sets, standardized differences were calculated. The sum of the standardized differences across all covariates was calculated to select the best matched set for each health system. The best matched sets for each health system were aggregated to evaluate the sum standardized difference overall, within the IDS and ambulatory subgroups, and at the individual health system level.

Other matching diagnostics included constructing quantile-quantile plots, overlaying histograms, and overlaying density plots to compare the distribution of continuous baseline covariates between intervention and control providers. Means and standardized differences of each covariate were computed for the intervention and control provider groups. These balance diagnostics were evaluated overall and within the IDS and ambulatory subgroups. The number of times each control was used overall and for each individual health system also were monitored.

Using the best matched set, generalized estimating equations (GEEs) were applied to compute a difference-in-differences estimate for each vaccination rate of interest, controlling for the inherent correlation between a provider's baseline and intervention rates. As a part of the GEE, variances were adjusted to account for some controls being reused (because of matching with replacement). The reported estimates represent the improvement of intervention providers above that of their matched controls.

Finally, sensitivity of the reported estimates to a particular matched set was analyzed using the best 500 matched sets (of the total ≥5000 matched sets). These matched sets have nearly as good, though not the best, matching diagnostics. Presumably, unmeasured confounders are shifting around in these distinct matched sets, so significant differences demonstrated in all 500 matched sets is indicative of robustness to unmeasured confounders.

Qualitative analysis

Lessons learned from the formal qualitative analysis collected during the pilot program¹¹ were leveraged to inform collaborative Cohorts 2 and 3 (current study). Interventions and self-identified best practices were recorded during Cohort 1 and organized into a framework that was provided to Cohorts 2 and 3. This allowed for learnings from Cohort 1 to be transferred to Cohorts 2 and 3. The framework grouped interventions into domains and provided an estimated level of difficulty of each intervention to accommodate the varying levels of resources and implementation readiness of each organization. As in other learning collaboratives, a non-prescriptive approach was taken; interventions other than those in the framework could be implemented. In addition, in this study informal qualitative data were collected during site visits and during both in-person and virtual meetings. These data were collected to confirm findings from the pilot study and to discover potential new strategies used during Cohorts 2 and 3. Barriers, facilitators, and most successful strategies (as perceived by participants) were identified.

Framework for spread

The IHI created a “Framework for Spread”¹ more than a decade ago that serves as a useful guide to identifying key components that contribute to effective scale-up and spread of new ideas, innovations, and operational systems both within and across organizations. Although collaboratives themselves have proved successful interventions^12,13 in scale-up and spread, the Framework for Spread helps hone in on the specific, necessary components, including leadership support, identification of better ideas, communication, strengthening the social system, measurement and feedback, and knowledge management. The authors recognize that organizational context plays an important role and factors such as culture, infrastructure, size, strength of social system, and the intervention or system being spread influence how components of the framework are applied.

Intervention

The study intervention was the learning collaborative approach taken to improve adult vaccination rates over 3 one-year collaborative periods and is reported in detail elsewhere.¹¹ Briefly, the learning collaboratives were led by an expert advisory committee and included in-person meetings, webinars, sharing of best practices, education, site visits, goal setting, outreach and coaching, peer-to-peer learning, case studies, and clinical outcome measurement. Participating organizations were encouraged to choose from a selection of strategies to improve their vaccination rates including, but not limited to, nurse standing orders, use of a 1-way or 2-way state vaccination information system registry, patient and provider education, patient outreach, EHR registries, health maintenance and best practice alerts, working with specialists, working with pharmacies, designating provider champions, and identifying and learning from high performers. Participating organizations chose a variety of strategies. Learnings from the pilot study¹¹ were heavily leveraged in Cohorts 2 and 3 in the spirit of progressive and iterative learning. Although a total of 39 organizations participated in either Cohort 2 or 3, only 9 were included in this study because of their data availability in the aforementioned Optum data set.

Results

Quantitative

Among the 9 study organizations, there were 250k, 99k, 192k, and 858k eligible patients for pneumococcal ages ≥65 years, pneumococcal high risk, pneumococcal at risk, and influenza vaccines, respectively. Table 3 shows baseline unadjusted (ie, unmatched) and adjusted (ie, matched) characteristics of the intervention and comparison cohorts for PV for patients aged ≥65 group for Cohort 2. (Cohort 3 characteristics are similar; data available upon request.) For the pneumococcal ≥65 measure, a total of 414 collaborative PCPs were matched to 268 non-collaborative providers out of a pool of 4290 providers. The sum standardized difference (SSD) for this measure and cohort was reduced from 5.46 to 0.309; similar reductions were observed across other measures and cohorts, indicating well-matched study samples. Across 4 measures in Cohorts 2 and 3, the overall adjusted SSDs ranged from 0.113 to 0.353.

Table 3.

Descriptive Statistics of Matching Variables: Unadjusted versus Adjusted Data Set (Cohort 2, Pneumococcal ≥65 Example)^1,2

	Original sample (unadjusted)			1:1 Matched set (adjusted)
Baseline variable	Collaborative PCPs	Non-collaborative PCPs	Standardizeddifference	Collaborative PCPs	Non-collaborative PCPs	Standardizeddifference
Overall	N = 414	N = 4290		N = 414	N = 268
Count Patients	397 ± 237	203 ± 201	0.883	397 ± 237	408 ± 740	0.020
Minority Rate	0.11 ± 0.14	0.10 ± 0.15	0.058	0.11 ± 0.14	0.11 ± 0.14	0.019
Medicare Well Visit Rate	0.40 ± 0.30	0.24 ± 0.27	0.559	0.40 ± 0.30	0.39 ± 0.33	0.029
Medicare Rate	0.26 ± 0.14	0.42 ± 0.22	0.841	0.26 ± 0.14	0.27 ± 0.15	0.050
Medicaid Rate	0.09 ± 0.08	0.11 ± 0.11	0.187	0.09 ± 0.08	0.09 ± 0.1	0.024
Ambulatory Group Type	25 (6%)	354 (8.3%)	0.086	25 (6%)	25 (6%)	0
Baseline Immuniz. Rate (ANY)	85 (20.5%)	354 (8.3%)	0.355	0.85 ± 0.12	0.84 ± 0.17	0.060
Baseline Immuniz. Rate (BOTH)	0.85 ± 0.12	0.55 ± 0.29	1.334	0.51 ± 0.18	0.49 ± 0.22	0.107
IDS	N = 389	N = 3936		N = 389	N = 259
Count Patients	393 ± 238	194 ± 189	0.927	393 ± 238	408 ± 763	0.025
Minority Rate	0.11 ± 0.14	0.10 ± 0.15	0.079	0.11 ± 0.14	0.11 ± 0.15	0.018
Medicare Well Visit Rate	0.37 ± 0.29	0.25 ± 0.27	0.465	0.37 ± 0.29	0.37 ± 0.33	0.028
Medicare Rate	0.26 ± 0.14	0.42 ± 0.22	0.827	0.26 ± 0.14	0.27 ± 0.16	0.048
Medicaid Rate	0.09 ± 0.08	0.10 ± 0.11	0.167	0.09 ± 0.08	0.09 ± 0.1	0.025
Baseline Immuniz. Rate (ANY)	0.84 ± 0.12	0.56 ± 0.28	1.292	0.84 ± 0.12	0.83 ± 0.17	0.061
Baseline Immuniz. Rate (BOTH)	0.50 ± 0.18	0.21 ± 0.22	1.465	0.50 ± 0.18	0.48 ± 0.22	0.117
Ambulatory	N = 25	N = 354		N = 25	N = 9
Count Patients	444 ± 218	297 ± 285	0.579	444 ± 218	404 ± 166	0.206
Minority Rate	0.10 ± 0.09	0.13 ± 0.09	0.334	0.10 ± 0.09	0.09 ± 0.05	0.064
Medicare Well Visit Rate	0.80 ± 0.09	0.20 ± 0.28	2.88	0.80 ± 0.09	0.78 ± 0.05	0.272
Medicare Rate	0.19 ± 0.06	0.40 ± 0.24	1.178	0.19 ± 0.06	0.20 ± 0.07	0.144
Medicaid Rate	0.06 ± 0.03	0.11 ± 0.12	0.546	0.06 ± 0.03	0.06 ± 0.03	0.021
Baseline Immuniz. Rate (ANY)	0.90 ± 0.09	0.48 ± 0.30	1.931	0.90 ± 0.09	0.90 ± 0.05	0.044
Baseline Immuniz. Rate (BOTH)	0.63 ± 0.17	0.18 ± 0.21	2.348	0.63 ± 0.17	0.64 ± 0.13	0.057
Sum Standardized Difference Overall = 0.309
Sum Standardized Difference IDS = 0.322
Sum Standardized Difference Ambulatory = 0.807

Best matched set using both ANY and BOTH immunization rates; using separate propensity score models for each group; matching with replacement, without caliper, distance.tolerance = 0.02.

For the pneumococcal ≥65 measure, “Any” vaccination includes pneumococcal conjugate vaccine (PCV), pneumococcal polysaccharide vaccine (PPSV), or unknown vaccination type; “Both” vaccinations refers to patients who received both PCV13 and PPSV23.

IDS, integrated delivery system; PCP, primary care provider.

Table 4 shows the average treatment effect of the intervention for each vaccination and type of vaccination overall for each cohort. In 2017–2018, both intervention cohorts demonstrated greater improvement than their matched providers in PVs (both vaccines PPSV and PCV) for patients age ≥65 (treatment effect range: 1.4%-3.7%, P < 0.01) and PV for high-risk patients (eg, with immunocompromising conditions) aged 19–64 (0.8%-1.6%, P < 0.01). Significant effects were observed in Cohort 3 for PV for at-risk patients (eg, with diabetes) aged 19–64 (1.7%, P < 0.01), and influenza vaccination rates (2.4%, P < 0.001). Individual health systems demonstrated even greater improvements across all 4 vaccinations: 9.5% influenza; 8.7% PV age ≥65; 11.8% PV high risk; 16.3% PV at risk (all P < 0.01). Table 5 presents pre–post vaccination rates for intervention and matched control groups across immunization type and cohort, for adult vaccinations recommended by the CDC.¹⁴

Table 4.

Adjusted Intervention Treatment Effect by Immunization Category and Type

Immunization	Cohort 2			Cohort 3
Immunization	Estimate¹	95% CI	P	Estimate¹	95% CI	P
Pneumococcal ≥65	(n = 414)			(n = 152)
Any Immunization²	0.004	-0.002, 0.011	0.205	0.010	-0.004, 0.024	0.145
PPSV and PCV	0.014	0.004, 0.024	0.007	0.037	0.020, 0.054	<0.001
Pneumococcal High Risk	(n = 385)			(n = 135)
Any Immunization²	0.014	0.001, 0.027	0.030	0.015	-0.002, 0.033	0.089
PPSV and PCV	0.008	0.004, 0.013	0.001	0.016	0.005, 0.028	0.004
Pneumococcal At Risk	(n = 423)			(n = 153)
Any Immunization³	0.005	-0.008, 0.017	0.468	0.019	0.006, 0.031	0.004
PPSV	0.002	-0.011, 0.014	0.792	0.017	0.005, 0.029	0.006
Influenza	(n = 513)			(n = 186)
Influenza	-0.005	-0.018, 0.008	0.461	0.024	0.010, 0.039	<0.001

Treatment effect estimate compared to matched control providers.

Any immunization for Pneumococcal ≥65 and High Risk refers to PPSV, PCV, or type unknown. Patient only needs to receive one of these to be included.

Any immunization for Pneumococcal At Risk PPSV or type unknown. Patient only needs to receive one of these to be included.

CI, confidence interval; PCV, pneumococcal conjugate vaccine; PPSV, pneumococcal polysaccharide vaccine.

Table 5.

Adjusted Pre–Post Intervention Immunization Rates Compared to Matched Controls

Immunization	Intervention		Matched controls		P ¹
Immunization	Pre	Post	Pre	Post	P ¹
Pneumococcal 65+: Cohort 2
PPSV and PCV	51.0%	59.4%	48.9%	55.9%	<0.01
Pneumococcal 65+: Cohort 3
PPSV and PCV	47.8%	56.0%	46.9%	51.4%	<0.0001
Pneumococcal High Risk: Cohort 2
PPSV and PCV	5.9%	7.8%	5.8%	6.9%	<0.001
Pneumococcal High Risk: Cohort 3
PPSV and PCV	9.1%	10.7%	9.0%	8.9%	<0.01
Pneumococcal At Risk: Cohort 2
PPSV	39.3%	39.6%	38.7%	38.9%	0.79
Pneumococcal At Risk: Cohort 3
PPSV	28.1%	29.7%	28.2%	28.1%	<0.01
Influenza: Cohort 2
Influenza	37.5%	41.1%	36.7%	40.8%	0.46
Influenza: Cohort 3
Influenza	38.1%	42.6%	37.8%	39.8%	<0.001

Difference-in-difference analysis comparing intervention to matched controls.

PCV, pneumococcal conjugate vaccine; PPSV, pneumococcal polysaccharide vaccine.

Qualitative

Although formal qualitative interviews were not conducted for Cohorts 2 and 3, site visits were conducted at 20 of the 39 health care organizations. During the site visits, AMGA staff members informally discussed progress on sites' individual adult immunization programs. Site visit qualitative findings were supplemented with information gathered during 2 in-person meetings, 12 webinars, and regular conference calls.

Commonly reported facilitators of adult immunization were similar to the pilot cohort and included provider education, implementing best practice alerts in the EHR, and collaboration with specialists, particularly for patients who were at risk or high risk for pneumococcal disease, and the collaborative itself. When asked what they liked best about the collaboratives, participant comments included “coming together with multiple organizations around the country tackling the same issue, networking very enjoyable, hearing best practices, collaboration, great discussions, ideas shared, great way to connect with peers.” Engagement of pharmacists to administer immunizations was noted more often as a facilitator in Cohorts 2 and 3. This was attributed to additional education about the role pharmacists can play in increasing immunization rates provided during Cohorts 2 and 3. Additional facilitators were the presence of immunization “champions” within the organization, automated patient outreach, and improving the exchange of data with state immunization registries.

A commonly reported barrier was the complexity of the immunization guidelines for patients with high-risk and at-risk conditions. Documentation issues were a lesser barrier for Cohorts 2 and 3 than they were for the pilot cohort. This was attributed to the addition over time of structured fields in EHRs to reflect changes in adult pneumococcal vaccine guidelines.

The most influential factors in successful spread align with IHI's Framework for Spread, including the importance of leadership buy-in and support, the identification of better ideas or practices than what was currently being done, measurement and feedback to providers and other key program staff, and knowledge management in the form of education for all staff members, especially around PVs for high- and at-risk patients.

Discussion

The adult immunization best practices learning collaborative was successfully scaled from 7 pilot sites (Cohort 1) to 39 expansion sites (Cohorts 2 and 3). Results demonstrated that expansion cohorts performed as well or better than the pilot sites. For example, expansion cohorts showed greater treatment effects for the recommended vaccinations for PV (eg, PPSV and PCV for PV age ≥65 and PV high risk) versus any vaccine (ie, PPSV or PCV). This suggests there is progress toward patients receiving the recommended vaccines as there are time restrictions preventing meeting requirements immediately (eg, recommended administration of PPSV is 1 year following receipt of PCV). Only Cohort 3 showed a positive treatment effect for influenza vaccination. This might reflect better documentation of vaccinations received outside the clinic environment. This was a noted primary barrier in the pilot cohort and therefore received significant focus during the expansion collaboratives. Improvements to state immunization registries may have contributed to better documentation.

An impressive finding also demonstrated during the pilot collaborative (Cohort 1), among high- and at-risk populations was that PV improvement rates increased following the difference-in-difference analysis because of decreasing immunization rates among control group providers. For example, at one site rates for at-risk patients increased from 52% to 65% while their matched control providers demonstrated a rate decrease from 55% to 52%. Again, the high- and at-risk populations were a greater challenge and less attention has been paid to these groups nationally.

The Collaborative itself was an important intervention leading to successful scale-up and spread, as were the key components identified during the pilot (Cohort 1) and emphasized during the expansion collaboratives that aligned with IHI's Framework for Spread. While participating in the Collaborative, and even upon conclusion, health care organization representatives learned and borrowed ideas from each other, troubleshot problems, and received guidance from national experts. The key components emphasized across collaboratives included leadership support and buy-in, measurement and feedback, communication, testing of innovations, and education or knowledge management. These align with the successful components identified and supported by the Framework for Spread.

As with the pilot study, a rigorous analytic approach was taken to compare the intervention organizations with other motivated and committed organizations. Typically, quality improvement programs are evaluated using a pretest–posttest design, often overestimating treatment effects and, ultimately, the impact of interventions. The approach taken in this study enabled true effects to be estimated by comparing providers at participating organizations with providers at other organizations with similar baseline characteristics and therefore a similar propensity to join a learning collaborative. This matching technique prevented the overestimation of the impact of a learning collaborative approach. Indeed, had participating organizations been compared to all health systems in the United States, a much greater difference would have been observed. Therefore, the authors can conclude that participating systems, even when compared to the most high-performing health systems available, still improved at least twice as much.

Limitations

There are limitations to the quantitative analyses. Three health care organizations that participated in Cohort 1 also participated in either Cohort 2 or 3. Although having had a longer overall time to make improvements may have favored their improvement, it also could have been more difficult for them to continue to improve on some measures. The limited pool of high-performing providers available as control providers meant the authors needed to match with replacements, which caused variance estimates to increase, and some control providers potentially had a greater influence on the reported effect estimates. In addition, there only were data on 9 of 30 organizations for the comparison analysis. However, the authors were able to confirm that the 9 study groups were representative of the larger group of 30 in terms of organizational size, structure, and baseline immunization rates. Although qualitative data collected during site visits and meetings supported findings from the qualitative data collection in the pilot study, formalized interviews conducted in Cohorts 2 and 3 may have enriched the analysis. A cost analysis of this approach was not conducted. The focus was on clinical population health outcomes and methods to improve these outcomes at a population level. It also was beyond the scope of the study to follow vaccinated patients to determine whether they experienced improved outcomes in terms of fewer hospitalizations or lower overall costs. Finally, the authors did not formally examine race/ethnic disparities in adult immunization rates. However, one participating organization did focus on disparities and was able to improve influenza vaccination rates for Hispanic and African American patients, decreasing the overall gap in rates.

Conclusion

In summary, these findings suggest a successful scale-up and spread of an intervention tested as a pilot study. These results further support a learning collaborative approach to improving population health outcomes in the area of adult vaccinations, especially for vaccinations with lower baseline national rates. In an era when health care dollars are scarce, leaders need to know with a degree of certainty that the resources they deploy to improve population health will result in desired outcomes. This study demonstrates how a learning collaborative approach can be successfully piloted and subsequently scaled up and spread. But perhaps even more importantly, it demonstrates how to identify those strategies and interventions that work best in real world, clinic, and organizational settings.

Footnotes

Acknowledgments

The authors wish to acknowledge the 39 AMGA member health care organizations that participated in the 2 learning collaboratives described in this manuscript, as well as the 7 organizations that “paved the way” by participating in the pilot study. Their commitment to improving adult immunization rates made this study possible. We also acknowledge the Population Health Initiatives Team at AMGA Foundation, without whom the best practices learning collaboratives would not have been possible. Their dedication, skill, and experience in conducting best practices learning collaboratives make work like this possible.

Author Disclosure Statement

The authors declared that there are no conflicts of interest.

Funding Information

The AMGA Adult Immunization Best Practices Learning Collaboratives were made possible by funding from Pfizer, Inc. However, the funding source did not play any role in the intervention design; collection, analysis, and interpretation of data; in the writing of the manuscript; or in the decision to submit the manuscript for publication. The content is solely the responsibility of the authors.

References

Massoud

, Nielsen

, Nolan

, Schall

, Sevin

A framework for spread: from local improvements to system-wide change. Cambridge, MA: Institute for Healthcare Improvement, 2006.

Centers for Disease Control and Prevention, National Center for Immunization and Respiratory Diseases, Division of Bacterial Diseases (NCIRD). Pneumococcal Disease Surveillance and Reporting Trends. 2017. https://www.cdc.gov/pneumococcal/surveillance.html. Accessed July 12, 2019 .

Rolfes

, Flannery

, Chung

, et al. Effects of influenza vaccination in the United States during the 2017–2018 influenza season. Clin Infect Dis, 2019; 69:1845–1853.

Centers for Disease Control and Prevention, National Center for Immunization and Respiratory Diseases (NCIRD). What are the benefits of flu vaccination?. 2019. https://www.cdc.gov/flu/prevent/vaccine-benefits.htm. Accessed November 4, 2019 .

Udell

, Zawi

, Bhatt

, et al. Association between influenza vaccination and cardiovascular outcomes in high-risk patients: a meta-analysis. JAMA, 2013; 310:1711–1720.

Centers for Disease Control and Prevention, National Center for Immunization and Respiratory Diseases (NCIRD). Estimated Influenza Illnesses, Medical visits, Hospitalizations, and Deaths in the United States—2017–2018 influenza season. 2018. https://www.cdc.gov/flu/about/burden/2017–2018.htm#Table 1. Accessed July 12, 2019 .

Centers for Disease Control and Prevention, National Center for Immunization and Respiratory Diseases (NCIRD). People 65 Years and Older & Influenza. 2019. https://www.cdc.gov/flu/highrisk/65over.htm. Accessed November 4, 2019 .

Ozawa

, Portnoy

, Getaneh

, et al. Modeling the economic burden of adult vaccine-preventable diseases in the United States. Health Aff (Millwood), 2016; 35:2124–2132.

Centers for Disease Control and Prevention, National Center for Immunization and Respiratory Diseases (NCIRD). Flu Vaccination Coverage by Age Group, United States, 2017–18 Season. Flu Vaccination Coverage, by Age Group, Adults, United States, 2017–18 Season. 2018. https://www.cdc.gov/flu/fluvaxview/coverage-1718estimates.htm#figure 1. Accessed November 4, 2019 .

10.

Centers for Disease Control and Prevention, National Center for Immunization and Respiratory Diseases. TABLE 2. Estimated proportion of adults aged ≥19 years who ever received pneumococcal vaccination* by increased-risk status† and race/ethnicity§—National Health Interview Survey, United States, 2017. 2018. https://www.cdc.gov/vaccines/imz-managers/coverage/adultvaxview/pubs-resources/NHIS-2017.html#pneumo. Accessed November 4, 2019 .

11.

Ciemins

, Jerry

, Powelson

, et al. Impact of a learning collaborative approach on influenza and pneumococcal immunization rates in US adults: a mixed methods approach. Popul Health Manag, 2019 [Epub ahead of print].

12.

Lynn

, Schall

, Milne

, Nolan

, Kabcenell

. Quality improvements in end of life care: insights from two collaboratives. Jt Comm J Qual Improv, 2000; 26:254–267.

13.

Wilson

, Berwick

, Cleary

. What do collaborative improvement projects do? Experience from seven countries. Jt Comm J Qual Saf, 2003; 29:85–93.

14.

Centers for Disease Control and Prevention. Recommended Adult Immunization Schedule, United States, 2019. 2019. https://www.cdc.gov/vaccines/schedules/hcp/imz/adult.html. Accessed November 4, 2019 .