Using Quality Measures to Drive Improvements in Immunization Rates: Findings from a Real-World Evaluation from 3 US Health Care Organizations

Abstract

Adult immunization rates are low and continue to fail to meet national targets. The coronavirus disease 2019 pandemic halted routine health care services for many, including vaccinations. This study explored whether the National Committee for Quality Assurance's Adult Immunization Status (AIS) measure, which had previously only been evaluated for reporting by health plans, could be leveraged by health care organizations (HCOs) as a tactic to improve immunization rates. Methods included a quantitative analysis of deidentified patient electronic health record data from 3 HCOs, supplemented by qualitative interviews to further understand opportunities and barriers. The analysis indicated that the data necessary for calculation of the AIS measure are available within HCOs and that measure performance could be enhanced with supplemental data from external sources, such as state immunization registries. Although HCOs rates were consistent with national estimates, this research further validated that adult immunization rates in the United States are low and highlighted the profound disparities that exist. For instance, the likelihood of completing all age-appropriate vaccinations was lower if patients were Black or African American, enrolled in Medicaid, or without health insurance. As a result of this study, the authors concluded that the AIS measure is feasible for use in medical groups and could potentially help drive quality improvements in immunization rates; however, there are considerations for implementation particularly if providers are being held accountable for measure performance.

Introduction

Despite strong evidence for vaccine effectiveness, the number of adults immunized against critical vaccine-preventable diseases, such as pneumonia, tetanus, diphtheria, pertussis, and herpes zoster (shingles), in the United States remains consistently below national targets.¹ Adult vaccination rates are lower among ethnic and racial minority groups²—a disparity that has been highlighted and exacerbated by the coronavirus disease 2019 (COVID-19) pandemic.³ COVID-19 has also reinforced the importance of strengthening the United States immunization data infrastructure and eliminating barriers to vaccine access. However, although COVID-19 vaccination is emerging as one of the greatest public health priorities in recent history, the pandemic has dramatically decreased overall adult vaccination rates due to profound declines in elective and routine care.⁴

To support quality-based initiatives aimed at increasing adult immunization rates, the National Adult Immunization and Influenza Summit (NAIIS) called for the development of a quality measure for adult immunizations in 2012. Since this recommendation, organizations such as the Indian Health Services and the National Committee for Quality Assurance (NCQA) have been involved in developing, testing, and specifying an adult immunization composite measure. In 2019, NCQA introduced the Adult Immunization Status (AIS) composite measure, an electronic clinical data set measure that was specified to assess the proportion of individuals, aged 19 years and older, who are up to date on age-recommended routine vaccines for 4 different vaccines—influenza; tetanus and diphtheria (Td) or tetanus, diphtheria, and acellular pertussis (Tdap); zoster; and pneumococcal.⁵ The measure was added to the Healthcare Effectiveness Data Information Set (HEDIS) as a voluntary measure for health plans to report in 2019. In 2020, NCQA updated the AIS measure specification so that only the individual vaccine rates are reported without an overall composite rate.⁶

Although HEDIS measures are designed for use by health plans, measures may be used across the health care system to improve quality in multiple settings, such as clinician group practices, integrated delivery systems, and hospitals. Therefore, it is important to evaluate the feasibility, usability, and implications of adapting HEDIS measures for use in assessing performance at a level beyond health plans.

Methods

The authors used a mixed methods approach that included quantitative retrospective analysis of deidentified patient electronic health record (EHR) data from 3 health care organizations (HCOs) (“test sites”) and qualitative analysis of information collected through interviews and a focus group with test site representatives. The study was reviewed and determined to fall under the exempt research category from the New England Independent Review Board (NEIRB).

The HCOs recruited for this study differed in geographic location within the United States, relative size, and capability to exchange data with their state immunization information system (IIS). Test sites identified and extracted information for all patients 19 years or older as of July 1, 2016, who were seen by a primary care provider between July 1, 2016, and June 30, 2018. At the patient level, test sites extracted limited demographic data (eg, age, gender, race, ethnicity, primary payer) as well as documentation of immunizations received that met the following criteria:

Influenza immunizations received between July 1, 2016, and June 30, 2018

Shingles (zoster) immunizations that occurred on or after an individual's 50th birthday

Pneumococcal immunizations that occurred after an individual's 60th birthday

Any Td or Tdap immunizations that occurred between July 1, 2007, and June 30, 2018

Patient-level data were removed and deidentified by the test sites then sent to the researchers through a secure file transport before the analysis. It should be noted that, consistent with the measure specifications, the authors requested documentation for any pneumococcal vaccinations received by patients after their 60th birthday to calculate the numerator; however, the denominator criterion for the pneumococcal rate is patients aged 66 years or older.

Because patients can receive immunizations at a variety of places (eg, at a pharmacy), test sites were encouraged to submit data from immunizations occurring outside of their organization (“external sources”) such as those reported in an IIS. Due to its relatively short look-back period, influenza was the only vaccine for which patient self-report was accepted as a valid external source of data. Note, test site 2 did not submit alternative documentation from external sources.

Quantitative analyses

Measure performance rates were calculated for each test site for 2 measurement periods: July 1, 2016, to June 30, 2017, and from July 1, 2017, to June 30, 2018. Data were submitted for a total of 856,744 unique patients across all test sites. There were 849,561 total individuals remaining in the analysis after 7,183 patients were excluded due to contraindications or other patient factors (eg, active chemotherapy, bone marrow transplant, history of immunocompromising conditions, hospice use).

Several rates were calculated (see Table 1 for detailed descriptions):

Table 1.

Measure Calculation Logic

	Rate	Denominator	Numerator
1	Influenza	Individuals aged 19+ who had an encounter during the measurement period	Individuals who received an influenza immunization during measurement period
2	Tdap/Td	Patients ages 19+ who had an encounter during the measurement period	Individuals who received a Tdap or Td vaccine during the measurement period or in the 9 years preceding the measurement period
3	Zoster	Individuals aged 50+ who had an encounter during the measurement period	Individuals who received a single Zostavax immunization or the complete 2-dose Shingrix series separated by at least 28 days
4	Pneumococcal	Individuals aged 66+ who had an encounter during the measurement period	Individuals who received the 2-dose pneumococcal series with both occurrences after the age of 60 years and at least 12 months apart
5	AIS composite	The sum of the denominators in rows 1–4	The sum of the numerators in rows 1–4
6	All age-appropriate immunizations	Individuals aged 19+ who had an encounter during the measurement period	The sum of individuals: Ages 19–49 years that are included in numerators 1 and 2 Ages 50–65 years that are included in numerators 1, 2, and 3 Ages 66+ years that are included in numerators 1, 2, 3, and 4

AIS, Adult Immunization Status; Td, tetanus and diphtheria; Tdap, tetanus, diphtheria, and acellular pertussis.

Individual immunization rates for influenza, Td/Tdap, zoster, and pneumococcal.

The AIS composite rate, as detailed in the 2019 HEDIS measure specifications, calculated by combining the 4 individual immunization rate numerators and dividing that total by the sum of the 4 denominators.

Receipt of all age-appropriate immunizations, calculated by stratifying the age groups and assessing whether each individual was up-to-date on all immunizations that are recommended for their age.

Qualitative analysis

In addition to the quantitative data analysis, the researchers performed 5 semistructured interviews with a total of 14 representatives from the 3 HCOs between October 15 and November 15, 2019. Interviews were used to gather information about test sites' immunization practices and factors that affect adult immunization rates. Individuals who participated in the interviews included physicians, chief quality officers, directors of quality improvement, a clinical informaticist, a pharmacist, and a manager of clinical services. The researchers also conducted 1 focus group with 7 participants from 1 of the participating health systems.

Results

The majority of patients were female, White, non-Hispanic/Latino, and had commercial insurance. Mean age across the 3 test sites ranged from 49.8 to 56.5 years old. On average across the 3 sites, patients had between 2.6 and 4.9 physician encounters per measurement year. See Table 2 for detailed descriptive statistics by test site.

Table 2.

Pilot Site Characteristics and Demographics

	Site 1	Site 2	Site 3
Location (United States)	East	Midwest	Midwest
Size of organization	200–500 employees	10,000+ employees	1000–5000 employees
Bidirectional exchange with IIS	No	Yes (limited)	Yes
Total unique patients included in study	38,828	759,192	58,724
Average no. of encounters
2016 (SD)	3.6 (2.8)	4.9 (6.3)	2.7 (3.4)
2017 (SD)	3.6 (2.8)	4.8 (6.1)	2.6 (3.3)
Gender (%)
Female	56.8	56.7	54.9
Male	43.2	43.3	45.1
Average age in 2016 (SD)	56.5 (17.6)	50.1 (18.4)	49.8 (18.0)
Race (%)
Asian	0.9	1.7	2.6
Black or African American	8.4	4.4	4.5
Native American or other	0.1	3.5	1.7
Declined, not reported, or unknown	1.4	3.5	0.4
White	89.2	86.9	90.8
Ethnicity (%)
Declined, not reported, or unknown	19.6	3.7	16.8
Hispanic or Latino	0.8	3.4	2.5
Non-Hispanic or Latino	79.6	92.9	79.3
Payer (%)
Commercial	61.6	55.9	63.7
Medicare	26.9	23.4	12.1
Medicaid	3.7	6.0	10.3
Self-pay	0	4.7	2.8
Other public	0.3	0.8	0.9
Worker's compensation	0	0.5	0.4
Uncategorized	7.6	8.7	9.8

IIS, immunization information system.

AIS composite measure performance ranged from 32.6% to 61.9% across the 3 sites (Table 3). Performance was highest for Td/Tdap immunizations and lowest for zoster. In general, rates on the individual vaccine measure components and the composite measure increased from the first measurement year to the second.

Table 3.

Measure Performance

Rate (%)	Site 1		Site 2		Site 3
Rate (%)	2016–2017	2017–2018	2016–2017	2017–2018	2016–2017	2017–2018
Influenza	51.9	57.4	32.6	36.2	35.1	35.1
Td/Tdap	68.6	77.3	54.3	61.2	88.9	89.8
Zoster	41.5	43.6	3.6	4.5	29.9	29.3
Pneumococcal	63.7	66.9	7.0	15.0	67.9	68.5
AIS composite	56.4	61.9	32.6	37.3	56.3	56.5
All age-appropriate immunizations	22.6	26.7	7.0	9.0	19.0	18.8

AIS, adult immunization status; Td, tetanus and diphtheria; Tdap, tetanus, diphtheria, and acellular pertussis.

The proportion of patients who had received all age-appropriate immunizations ranged from 7.0% to 26.7% (Table 3). The demographic effects on the odds of a patient receiving all age-appropriate immunizations were calculated (Table 4). Across all test sites, the odds of receiving all age-appropriate immunizations included in the AIS measure significantly increased with each additional encounter a patient had with the test site during the measurement year. The odds of completing all age-appropriate vaccinations were lower if patients were Black or African American, did not report ethnicity, received Medicaid, or did not have health insurance.

Table 4.

Demographic Effects on Receipt of All Age-Appropriate Immunizations in Adult Immunization Status Composite Measure

Covariates	Odds ratio (95% confidence interval)
Covariates	Site 1	Site 2	Site 3
No. of encounters in 2017–2018	1.13 (1.12–1.14)	1.08 (1.079–1.082)	1.16 (1.15–1.17)
Gender
Male	1.00 (reference)	1.00 (reference)	1.00 (reference)
Female	1.02 (0.97–1.07)	1.36 (1.33–1.39)	1.49 (1.42–1.57)
Race
White	1.00 (reference)	1.00 (reference)	1.00 (reference)
Asian	1.25 (0.97–1.61)	1.20 (1.13–1.27)	0.91 (0.77–1.07)
Black or African American	0.57 (0.52–0.64)	0.60 (0.57–0.63)	0.49 (0.42–0.57)
Another race	0.65 (0.28–1.51)	0.74 (0.68–0.80)	0.69 (0.56–0.86)
Unreported	0.87 (0.70–1.08)	0.93 (0.89–0.98)	0.46 (0.40–0.54)
Ethnicity
Non-Hispanic or Latino	1.00 (reference)	1.00 (reference)	1.00 (reference)
Hispanic or Latino	0.89 (0.67–1.19)	0.96 (0.89–1.03)	0.80 (0.68–0.95)
Unreported	0.89 (0.84–0.95)	0.71 (0.67–0.76)	0.69 (0.58–0.81)
Primary payer at last encounter
Commercial	1.00 (reference)	1.00 (reference)	1.00 (Reference)
Medicare	1.07 (0.99–1.14)	0.97 (0.92–1.02)	0.84 (0.78–0.91)
Medicaid	0.61 (0.53–0.71)	0.55 (0.52–0.57)	0.59 (0.54–0.65)
Self-pay	n/a	0.48 (0.46–0.51)	0.08 (0.05–0.13)
Another governmental/public	1.60 (1.07–2.40)	1.28 (1.17–1.40)	0.90 (0.70–1.16)
Worker's compensation	n/a	0.76 (0.66–0.87)	0.65 (0.40–1.02)
Uncategorized	0.72 (0.65–0.80)	1.07 (1.03–1.10)	1.08 (0.99–1.11)

Bold text indicates statically significant results.

n/a, not reported by site.

The authors calculated the AIS composite rate with and without external sources of data to determine how external sources of data contributed to measure performance. As shown in Figure 1, including external sources of data increased the measure rate by 18.2% to 29.4%. Test site 2 did not submit external sources of data and, therefore, only the rate without external data was calculated.

Fig. 1.

Using measures to improve immunization rates.

Site perspectives on immunization measurement

Test site representatives indicated that the 3 HCOs had a strong interest in improving adult immunization rates among their patient populations. The representatives were familiar with the AIS measure and its 4 vaccine components, but no HCO had implemented or reported it because it is not tied to accountability (eg, pay-for-performance) or external reporting requirements. All test sites had the capability to electronically capture data on the 4 vaccines included in the AIS measure through their own EHR systems or external sources and shared that they routinely used these data for internal quality tracking purposes, regardless of whether they reported the information externally.

Test site representatives indicated that although their HCOs can capture data on all 4 of the vaccines in the AIS measure, they experience challenges that can inhibit measure performance and quality improvement. For example, their EHR and/or state IIS may not be robust enough to effectively support bidirectional exchange of immunization data. In states with a robust IIS and in HCOs with advanced systems, data can be automatically downloaded into the EHR, and reconciled in both directions and from several sources. Furthermore, some states do not require pharmacies or other immunization providers to report immunizations to their IIS. Therefore, even with a bidirectional EHR interface and a robust state IIS, gaps in patient immunization data may occur if pharmacies and other immunization providers do not report immunizations to the state's IIS.

Although there may be challenges in reporting, test site representatives believed reporting the AIS measure would shift providers' focus toward all 4 vaccines, and, therefore, could improve immunization rates in their patient populations.

Discussion

The purpose of this study was to assess whether the AIS measure could be leveraged by HCOs as a tactic to improve immunization rates, as well as describe any challenges and considerations for implementation of the measure. In summary, the authors concluded that HCOs have the data to calculate the AIS measure and implementation of the measure could be used as a tactic to increase adult immunization rates.

This research further validates that adult immunization rates in the United States are low. As shown in Figure 2, performance rates calculated from the 3 test sites were consistent with national estimates from the Centers for Disease Control and Prevention.⁷ In addition, although rates for certain vaccines (eg, pneumococcal and Tdap/Td) are high relative to other vaccines, a significant proportion of adults are still without at least 1 Advisory Committee on Immunization Practices (ACIP)-recommended immunization. For example, the current research found the proportion of patients who received all the recommended vaccinations for their age group ranged from 7% to 26.7% across test sites, and profound disparities in vaccine coverage existed across all test sites. Black or African American patients were less likely to have received all recommended vaccines than White patients. Medicaid beneficiaries and uninsured patients were also less likely than people with private insurance to have received all recommended vaccines.

Fig. 2.

Immunization rates for four vaccines (year2) compared to national estimates.

Providers need access to comprehensive patient data to accurately assess and counsel patients on required immunizations without undue administrative burden. Unlike payers, HCOs do not necessarily have access to all patient immunization data, particularly if immunizations are administered outside of their organization. Interoperability and the use of external data sources, such as an IIS, can improve vaccine measure performance, better reflect real-world patient immunization rates, and give providers the information they need so they can have meaningful conversations with their patients about required vaccinations. These findings showed that AIS measure rates were higher for the 2 test sites that included submission data from external sources compared with the test site that did not. However, patients from these 2 sites only represented 11% of the total patient population in the study, therefore, these findings should be validated in subsequent studies.

Although these results show the impact and importance of using external sources, there remain challenges in the real-world application that should be noted. First, site 2, the largest of the 3 sites as well as the only site that had locations across several states, had challenges related to extracting data from external sources, despite having some bidirectional interoperability capabilities. It may be more difficult for larger organizations, such as site 2, to navigate the bidirectional interface across their system, especially when the HCO spans multiple states. For this reason, efforts to increase immunization exchange of data and interoperability should include both state-level initiatives at improving IIS and support for HCOs in setting up their EHRs to be fully interoperable with the IIS. Second, even when HCOs can utilize data from external sources, gaps in data likely persist due to the number of vaccinations that are received in pharmacies or other sites of care that are not all fully integrated into the IIS. This study did not evaluate which external sources (eg, downloaded from IIS, directly from a pharmacy) contributed to the improved rates; however, this would be an important area for future research and could help inform policy-related discussions. For instance, although some states have mandates that entities must report administered immunizations to the IIS, not all states have this requirement.

Ultimately, this research suggests that implementing the AIS measure in HCOs could be a useful tool to track and improve adult immunization rates. Although there are challenges to using the AIS measure, there needs to be a concerted effort to improve adult immunization rates. Incentivizing performance on an immunization quality performance measure would likely increase providers' focus and counseling of patients on recommended immunizations, ultimately leading to increased vaccine administration and improvements in population health.

Limitations

There are limitations to this study. First is the nature of retrospective data. Based on prior published research, as well as information gathered from the qualitative interviews, the authors concluded that immunization measures could be used as a tool to increase performance; however, they cannot determine the extent of improvement without prospective data. It should be noted that the authors did have multiple years of data that showed measure performance on the individual immunization components, as well as performance on all-recommended immunizations, generally improved from measurement year 1 to measurement year 2. Second, the AIS measure is currently specified for use by health plans, not medical groups. Therefore, minor modifications (eg, data collection methods) and assumptions were made in the measure specifications to fit the needs of medical groups and this project. Lastly, although the analysis included a relatively large number of patients, data were only collected from 3 test sites. In addition, only 2 sites, representing ∼11% of unique patients, were able to submit external data sources. Adding additional test sites, and ensuring sites have the capability to submit external sources of data, would increase the validity and generalizability of the research, especially given the homogeneous population represented across the test sites. Future research in this area should consider a larger, more diverse, and more comprehensive sample to increase generalizability of the findings.

Conclusion

This research provides an initial evaluation of the feasibility of using the AIS measure in HCOs. The current findings suggest that the AIS composite measure would be feasible to implement in HCOs and has the potential to increase adult immunization rates. HCO representatives reported that they believed the measure would increase adult immunizations and improve population health, which is particularly important given that most patients did not receive all age-recommended immunizations. Barriers to implementation of the measure include challenges accessing data, which is important to consider since external sources of immunization data heavily contributed to measure performance.

Footnotes

Acknowledgments

The authors thank the 3 HCOs that participated in this study for their focus on increasing immunization rates and dedication to improving quality of care for their patients.

Authors' Contributions

Ms. Esselman led the quantitative portions of the research. She was involved in study design, data collection from test sites, data analysis and interpretation, and all aspects of the article drafting and finalization. Dr. Ciemins led the qualitative components of the research. She provided input into the overall study design, conducted the qualitative interviews, interpreted the findings, and reviewed the quantitative outputs. In addition, she contributed to the writing of the article, and critically reviewed and provided input on the article. Ms. Donckels was involved in study design, data collection from test sites, data analysis and interpretation, and all aspects of the article drafting and finalization. Ms. Barbera was involved in study design, data collection from test sites, data interpretation, and all aspects of the article drafting and finalization. Mr. D'Andrea was involved in project conceptualization, study design, and review of findings. In addition, he critically reviewed and provided input on the article. Ms. McBride was involved in project conceptualization and study design and provided thought leadership throughout the research. In addition, she critically reviewed and provided input on the article.

Author Disclosure Statement

Ms. Esselman, Dr. Ciemins, Ms. Barbera, Ms. Donckels, and Mr. D'Andrea disclose no personal conflicts of interest; however, Discern Health received funding to conduct the research. Ms. McBride was an employee at GSK at the time of the research.

Funding Information

This research was sponsored by GlaxoSmithKline (GSK). The sponsor was involved in project conceptualization and study design and provided thought leadership throughout the research. However, the sponsor was not involved in the collection, analysis, or interpretation of data. In addition, the study sponsor was not involved in the selection of participating test sites and does not have knowledge of which HCOs were included in the study.

References

Terrie

. Adult vaccination rates are rising but fall short. 2019. https://www.pharmacytimes.com/view/adult-vaccination-rates-are-rising-but-fall-short Accessed October 27, 2021 .

, O'Halloran

, Williams

, Lindley

, Farrall

, Bridges

. Racial and ethnic disparities in vaccination coverage among adult populations in the U.S. Am J Prev Med, 2015; 49(6 Suppl 4):S412–S425.

National Foundation for Infectious Diseases. Issue brief: the impact of COVID-19 on US vaccination rates. 2020. https://www.nfid.org/keep-up-the-rates/issue-brief-the-impact-of-covid-19-on-us-vaccination-rates/ Accessed October 27, 2021 .

Shen

, Sobczyk

. Unraveling a composite measure to drive immunization uptake in adults: a critical compromise. Health

Affairs

. https://www.healthaffairs.org/do/10.1377/hblog20200817.901186/full/ Accessed October 27, 2021 .

Nguygen

, Joshi

, Drew

, et al. Racial and ethnic differences in COVID-19 vaccine hesitancy and uptake. medRxiv, 2021;2021.02.25.21252402.

National Committee for Quality Assurance. Adult Immunization Status. https://www.ncqa.org/hedis/measures/adult-immunization-status/

, Hung

, Srivastav

, et al. Surveillance of vaccination coverage among adult populations—United States, 2018. MMWR Surveill Summ, 2021; 70(No. SS-3):1–26. DOI: https://dx-doi-org.web.bisu.edu.cn/10.15585/mmwr.ss7003a1externalicon Accessed October 27, 2021 .