Impact of Alternative Encounter Types on HIV Viral Suppression Rates in an Integrated Health System

Abstract

Kaiser Permanente Mid-Atlantic States (KPMAS) members are increasingly utilizing electronic encounter types, such as telephone appointments and secure messaging for healthcare purposes, although their impact on health outcomes is unknown. We evaluated whether use of alternative encounters by adult human immunodeficiency virus (HIV)-infected patients affected the likelihood of achieving viral suppression (VS). Our study population of 3114 patients contributed 6520 patient-years between 2014 and 2016. We compared VS (HIV RNA <200 copies/mL) by number of in-person visits (1 or ≥2), with further stratification for additional phone and/or e-mail encounters (none, phone only, e-mail only, and both phone and e-mail). Rate ratios (RRs) for VS by number of in-person visits and encounter types were obtained from Poisson modeling, adjusting for age, sex, race/ethnicity, and HIV risk. Compared to those with ≥2 visits, patients with one in-person visit alone were significantly less likely to achieve VS (RR = 0.93; 95% confidence interval, CI: [0.87–1.00]), as were those with one in-person visit plus a telephone encounter (0.93; [0.90–0.97]). We did not find significant differences in VS comparing patients with one in-person visit plus e-mail only (RR = 1.00; 95% CI: [0.97–1.02]) or plus e-mail and telephone (0.99; [0.97–1.01]) to those with ≥2 in-person visits. If supplemented by e-mail communications (with or without telephone contact), patients with one in-person visit per year had similar estimated rates of VS compared with ≥2 in-person visits. More research is needed to know if these findings apply to other care systems.

Introduction

It is well established that linkage to and retention in clinical care is critical for patients diagnosed with human immunodeficiency virus (HIV) for achieving viral suppression (VS), improving health outcomes and reducing risk of transmission.^{1

–14} The US National HIV/AIDS Strategy (NHAS) recognizes the importance of these steps in the HIV Care Cascade¹⁵ and has set ambitious goals through 2020,¹ although challenges to establishing and sustaining engagement in care remain barriers in attaining these goals. While there is no clear gold standard retention measure, and proposed definitions vary in their metrics,^{3,5,9,10,16,17} retention in care is often defined as two in-person visits annually with either a primary care provider or HIV specialist.^{4,8,15,18

–25} However, there is increasing use of alternatives to in-person patient–provider visits, including telephone visits and e-mail communications, and their effect on health outcomes is unknown.

Previous research into use of telephone and electronic communication in HIV care has focused on HIV prevention and intervention through text message visit reminders and follow-up calls to supplement in-person visits.^{26

–33} Reviews of the literature on use of electronic and mobile communication devices as vehicles for HIV care, known as mobile health (mHealth),^26,30 found these alternative methods to be embraced among patients and efficacious in care delivery, contributing to retention in care.^26,33 In addition, Gardner et al. examined the effect of “enhanced contact,” consisting of follow-up and reminder phone calls, on visit constancy and visit adherence.²⁸ These retention in care improvements seen by Gardner et al. led to the intervention's inclusion in the CDC's Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention.³⁴ While these studies provide strong evidence that alternative methods of communication are useful supplements to in-person clinical care, there is a noticeable lack of research into whether these encounter types are comparable to face-to-face visits with physicians with respect to health outcomes.

At Kaiser Permanente Mid-Atlantic States (KPMAS), there is increasing use of telephone and/or e-mail encounters in place of additional in-person visits for managing patient care.³⁵ For HIV patients in care, it is essential to determine the impact of these alternate encounters on achieving VS, which according to the CDC Care Continuum, is the ultimate goal for HIV treatment.³⁶ The current study aims to expand upon previous research and observe the impact that alternate encounter types, including telephone and e-mail communications, may have on the likelihood of VS. We hypothesized that use of alternative encounter types in combination with one in-person visit annually would allow patients to achieve similar HIV VS outcomes when compared with the two-visit standard in our integrated care population.

Methods

Study design and subjects

To evaluate the impact of type and frequency of healthcare visits on VS among HIV-infected patients, we conducted a retrospective study using electronic health record (EHR) data of eligible KPMAS HIV-infected members that were at least 18 years old. Between January 1, 2014 and December 31, 2016, observations were recorded in each calendar year during which eligible members had at least 6 months Kaiser Permanente (KP) membership, at least one in-person office visit to an HIV/Infectious Disease (ID) specialist, and at least one HIV RNA viral load (VL) measurement. A single patient could contribute a patient-year of data for each calendar year, in which these three criteria were met. Of the 7222 total HIV-infected patient-years identified, 219 (3.0%) patient-years were omitted due to having no in-person visits, and 483 additional (6.7%) patient-years were omitted due to having no VL measurement. This study is under the approval of the KPMAS Institutional Review Board (IRB), which waived the requirement for informed patient consent as this is a data-only study, involving no more than minimal risk to members, with numerous data safeguards in place. In addition, there were no ethical concerns expressed.

Setting

KP is one of the largest not-for-profit integrated health systems in the United States, currently serving over 11.8 million members in eight geographic regions.³⁷ KP provides comprehensive care for its members; HIV-infected patients benefit from multidisciplinary HIV care teams, a dedicated HIV quality measurement and improvement program, and continual provider education.¹⁵ Currently, KPMAS provides care to over 3200 HIV-infected members in the District of Columbia, Maryland, and Virginia.³⁸ The HIV-infected population in KP (including KPMAS) is representative of its respective states; data indicate that members overall are very similar to the general population with regard to age, gender, and race/ethnicity. Moreover, HIV-infected patients in KP reflect the general HIV-infected population, in part, because patients with low incomes are eligible for state Medicaid coverage in KP.^15,39

Data source

All KP patient data are collected in EHRs through KP HealthConnect^®, a comprehensive health information system that is Meaningful Use compliant, described elsewhere.^40

–43 One of the many integrative features of this EHR system is the ability for KP providers to log all patient encounter types, including phone visits and e-mail communications. Secure e-mail messages between patients and providers are sent through the KP patient portal, My Health Manager, on kp.org. In addition to the secure messaging function, the portal provides patients with clinical information, such as laboratory test results and medication information, as well as other resources, such as self-care instructions, educational materials, and the ability to refill prescriptions through KP pharmacies (including by mail order).⁴⁰

Measurements

The KP EHR was queried to collect all in-person visits to an HIV/ID specialist or primary care provider as well as utilization of alternative encounter types with an HIV/ID specialist. The number of in-person visits was dichotomized into groups of one or ≥2 visits per year, regardless of whether there were additional e-mail or telephone encounters. Patients with one or ≥2 visits were further stratified by additional use of phone and/or e-mail encounters to an HIV/ID specialist. Use of these alternative phone and/or e-mail encounters was categorized into four subgroups: (1) neither phone nor e-mail; (2) phone only; (3) e-mail only; and (4) both phone and e-mail. We did not specify the number of phone or e-mail encounters per patient.

We compared VS (last HIV RNA in year <200 copies/mL) by number of in-person visits (1 or ≥2) to an HIV/ID specialist or primary care physician. Our goal was to determine whether patients having only one in-person visit that was supplemented with phone and/or e-mail encounters would achieve similar VS outcomes as those with two or more in-person visits.

Demographic characteristics were obtained from EHR data. Age, measured on July 1st of the given patient-year, was categorized as 18–29, 30–39, 40–49, 50–59, 60–69, and ≥70 years. We explored an alternate categorization for age of 18–29, 30–50, and >50 years, but found that results were similar. Self-reported race/ethnicity was classified as Asian/Pacific Islander, black, Hispanic, white, or other/unknown. HIV risk categories were classified as intravenous drug use (IDU), men who have sex with men (MSM) or bisexual behavior, heterosexual behavior, and other/unknown. Patients who were identified both sexual and IDU risk factors were categorized as IDU. In-person visits by patients to HIV/ID specialists during the calendar year were dichotomized into one or more visits per year and no visits per year.

Statistical analyses

Demographic characteristics were reported for unique subjects using frequency distributions. HIV VS rates were reported annually using frequency distributions, and comparing between one and ≥2 in-person visits. We compared rates of VS by the number of in-person visits to an HIV/ID specialist or primary care provider and, among those with one in-person visit, by use of additional telephone or e-mail communications with an HIV/ID specialist. Our reference population for all comparisons was ≥2 in-person visits, with or without additional telephone or e-mail communications. Adjusted rate ratios (RRs) for VS were computed by multivariable Poisson regression⁴⁴ using generalized estimating equations (GEE) with an exchangeable working correlation structure⁴⁵ to account for members who contributed data in more than 1 year, adjusting for age, sex, race/ethnicity, HIV risk, and whether the patient visited an HIV/ID specialist during the calendar year.

A sensitivity analysis was conducted to determine the impact of excluding patients with missing VL measurements in our study population. In the sensitivity analysis, patients with no measurements in a given year were included in the models described above, but were counted as not virally suppressed. Patients with no in-person visits were excluded from all analyses, as they would not be able to meet any definition of retention in care. Data set preparation and quality assurance were performed using SAS software version 9.4 (SAS Institute, Inc., Cary, NC), and statistical analyses were conducted using R version 3.1.1 (R Foundation for Statistical Computing, Vienna, Austria).

Results

Study population and demographics

A total of 3114 patients contributed 6520 patient-years to our analysis. Selected population characteristics are presented in Table 1. Our study population was predominantly male (70.2%), black (75.0%), and 30 years of age or older (87.2%). HIV risk was predominantly MSM or bisexual (36.0%) or heterosexual (36.4%). In addition, most patients (97.5%) had seen an HIV/ID specialist during the calendar year. The distributions of all specified demographic characteristics remained consistent across the years of the study (data not shown).

Table 1.

Description of Characteristics of HIV Patients, 2014–2016

Total population^a	3114
Distribution by sex
Female	927 (29.8)
Male	2187 (70.2)
Distribution by age range
18–29	399 (12.8)
30–39	500 (16.1)
40–49	764 (24.5)
50–59	926 (29.7)
60–69	429 (13.8)
70+	96 (3.1)
Distribution by race/ethnicity
Asian/Pacific Islander	36 (1.2)
Black	2335 (75.0)
Hispanic	147 (4.7)
White	479 (15.4)
Other/unknown	117 (3.8)
Distribution by HIV risk
Injection drug use	379 (12.2)
Heterosexual	1133 (36.4)
Men who have sex with men or bisexual	1120 (36.0)
Other/unknown	482 (15.5)
One or more ID visit during calendar year	3035 (97.5)
No ID visit during calendar year	79 (2.5)

Of the 3114 patients who were eligible for inclusion (≥1 in-person visit, ≥1 HIV viral load laboratory result, and ≥6 months of membership between 2014 and 2016), 987 met the inclusion criteria in 1 study year, 848 met the criteria in 2 study years, and 1279 met the criteria in all 3 study years.

HIV, human immunodeficiency virus; ID, infectious disease.

Encounter patterns and viral suppression

Overall, patients were virally suppressed in 88.6% of years, in which they met the eligibility criteria (Table 2). Of those patients with one in-person visit, 87.3% achieved or maintained VS, while 89.4% of patients with ≥2 in-person visits were virally suppressed. VS rates increased over time, from 86.8% in 2014 to 90.5% in 2016. Similar increases in VS rates over time were seen among patients with one in-person visit (84.2–89.6%) and patients with ≥2 in-person visits (88.2–91.1%).

Table 2.

Summary of HIV Patients' Visits and Viral Suppression, 2014–2016

Year	Patients ^a	No. of patients virally suppressed (HIV RNA <200 copies/mL)	Percent of patients virally suppressed (HIV RNA <200 copies/mL)
All in-person visits
2014	1959	1700	86.8
2015	2182	1923	88.1
2016	2379	2153	90.5
Total	6520	5776	88.6
One in-person visit
2014	683	575	84.2
2015	827	720	87.1
2016	995	892	89.6
Total	2505	2187	87.3
≥2 In-person visits
2014	1276	1125	88.2
2015	1355	1203	88.8
2016	1384	1261	91.1
Total	4015	3589	89.4

Patients with missing laboratories are excluded.

HIV, human immunodeficiency virus.

We stratified VS rates by year and encounter pattern, defined by visit frequency (1 vs. ≥2) and type of visit (in-person, with or without telephone and/or e-mail). Overall, inclusion of alternate patient–provider communication was associated with a higher percentage of virally suppressed patients, regardless of number of in-person visits (Fig. 1). The lowest rate of VS (77.9%) was observed in 2014 among patients who had one in-person visit and telephone communication, while the highest rate of VS (94.9%) was observed in 2016 among patients who had ≥2 in-person visits and e-mail communication (data not shown).

FIG. 1.

Differences in viral suppression rates by encounter types for HIV patients in KPMAS. This figure compares HIV patients with ≥2 in-person visits in a calendar year versus patients with one in-person visit, stratified by alternative encounter types, and excludes patients with missing laboratory values. HIV, human immunodeficiency virus; KPMAS, Kaiser Permanente Mid-Atlantic States.

Impact of alternative encounter patterns on viral suppression

Poisson regression models were estimated to compare rates of VS among patients with one in-person visit, with and without alternative encounter types, to those with ≥2 in-person visits (Table 3). In the adjusted model, when compared to those with ≥2 visits, patients with one in-person visit alone were significantly less likely to achieve VS (RR = 0.93; 95% confidence interval, CI: [0.87–1.00]), as were those with one in-person visit plus a telephone encounter (0.93; [0.90–0.97]).

Table 3.

Association Between Encounter Types and Viral Suppression

	Unadjusted analysis		Adjusted analysis ^a
	Rate ratio ^b	95% CI	Rate ratio ^b	95% CI
One in-person only	0.92	0.86–0.98	0.93	0.87–1.00
One in-person + telephone only	0.94	0.90–0.97	0.93	0.90–0.97
One in-person + e-mail only	1.00	0.98–1.03	1.00	0.97–1.02
One in-person + telephone + e-mail	0.99	0.97–1.01	0.99	0.97–1.01

Of the 3114 patients included in this analysis, with exclusions due to missing laboratory values; represents 6520 patient-years.

Poisson regression models adjusted for sex, age, race/ethnicity, HIV risk behavior, and ID visit. Additional significant differences seen for: younger age, black, or unknown race/ethnicity (all less likely).

Reference group: ≥2 in-person visits annually.

CI, confidence interval; HIV, human immunodeficiency virus; ID, infectious disease.

We found no significant difference in VS comparing patients with one in-person visit plus e-mail only (RR = 1.00; 95% CI: [0.97–1.02]) or plus e-mail and telephone (0.99; [0.97–1.01]) to those with ≥2 in-person visits. When one annual visit was accompanied by an e-mail encounter, with or without a telephone encounter, the estimated rates of VS were similar to the traditional two or more in-person visits.

Our sensitivity analysis (Table 4) included patients without HIV VL measurements in the specified year, and assumed that they did not have VS. Results were largely consistent with our analysis that excluded patients without HIV VL measurements. Patients having one in-person visit with both e-mail and telephone encounters achieved similar rates of VS to those having two in-person visits. Patients having one in-person visit with telephone encounters only or with no telephone and no e-mail encounters had lower rates of VS than those with two in-person visits. However, patients having one in-person visit supplemented with only e-mail encounters had lower rates of VS than those having two in-person visits, differing from the results of the other analysis, in which these two groups of patients had more similar rates of VS.

Table 4.

Sensitivity Analysis: Association Between Encounter Types and Viral Suppression, Including Patients with Missing Laboratory Values as Not Virally Suppressed

	Unadjusted analysis		Adjusted analysis ^a
	Rate ratio ^b	95% CI	Rate ratio ^b	95% CI
One in-person only	0.51	0.45–0.57	0.83	0.75–0.91
One in-person + telephone only	0.90	0.86–0.95	0.89	0.85–0.93
One in-person + e-mail only	0.95	0.92–0.98	0.92	0.89–0.95
One in-person + telephone + e-mail	1.01	0.98–1.03	0.98	0.96–1.01

Of the 3249 patients included in this analysis, including patients with missing laboratory values as not virally suppressed; represents 7003 patient-years.

Poisson regression models adjusted for sex, age, race/ethnicity, HIV risk behavior, and ID visit.

Reference group: ≥2 in-person visits annually.

CI, confidence interval; HIV, human immunodeficiency virus; ID, infectious disease.

Discussion

Our results indicate that one in-person visit without alternative patient/provider communications (telephone or e-mail) was inferior to two or more in-person visits, with respect to VS among HIV-infected patients. If supplemented by e-mail communications (with or without telephone contact), results suggest that one in-person visit per year may result in similar rates of VS, when compared to two or more in-person visits. The landscape of HIV care continues to change as new technologies are used and embraced, including alternative patient–provider encounter types.

Our results are consistent with those found in other studies of care quality and efficiency among KP patients for other chronic conditions. In KP Southern California, Zhou et al. observed improvements in several HEDIS measures among patients with diabetes and/or hypertension who engaged in secure e-mail messages with their provider.⁴⁰ An internal KP study revealed that nine in ten patients felt that the KP patient portal enabled them to manage their conditions more effectively and made it more convenient for them to interact with their care teams.^46

–49 Previous work also suggests that patients who use the patient portal for prescription refills experienced greater medication adherence and improved outcomes.^49

–52 While the specific mechanisms underlying the association between e-mail encounters and improved outcomes are unclear, contributing factors may include increased continuity of care, greater patient–physician connectedness, and better support for patient self-management.^40,51 Although telephone visits provide additional communication lines between patients and providers, e-mail encounters are more directly tied to a patient's EHR and KP's online resources, providing a centralized location for health information and allowing patients to feel more in control of their health. These factors may explain our finding that one in-person visit supplemented by e-mail has comparable rates of VS to the two in-person visit standard, while supplemental telephone encounters did not have the same effect.

The objective of this study was to examine whether the standard definition of retention in care can or should be altered to incorporate additional measures and broaden our definition of what constitutes high-quality HIV care. Quality metrics are vital measures of HIV care performance within a health system as well as at the national and international levels. In KP, each step of the HIV Care Cascade is evaluated and the results are used to identify potential gaps in care as well as highlight performance best practices.¹⁵ Inaccurate measurement of the HIV Care Cascade can result in misdirected interventions and policy recommendations. An important implication of our findings is the need to update and harmonize HIV quality metrics, including retention in care, which supports earlier research.^9,15,39

We believe that the definition of retention in care merits revision to reflect alternative communications. Our results suggest that the most often cited definition of retention in care (≥2 in-person visits in a calendar year) may not be inclusive enough, given the evolving nature of HIV care and healthcare in general; even public healthcare sites are developing alternatives to in-person visits, including online patient portals for electronic communications.^53,54 The current study shows that excluding alternative encounter types when calculating retention in care would underestimate the number of HIV-infected patients who are virally suppressed and are engaged in care. It is possible that HIV-infected patients who are well established in care, adherent to their ART regimen, and have maximally suppressed virus may not require the same level of traditional, in-person engagement in care compared with those who are newly diagnosed, nonadherent, and not consistently virally suppressed.^55,56 HIV care quality measures must adapt to changes in care delivery and frequency, as more and more research indicates that alternative models of care are not detrimental to patients⁵⁷ and, in some cases, are positively associated with clinical outcomes and health behaviors, such as retention in care.^58,59 However, the extent to which alternative encounter types can fully replace in-person visits is still unclear and warrants more research.

It is important to note that while VS is a key element of successful engagement in care, it is not the only outcome. The HIV care visit also may include management of comorbidities, screening for sexually transmitted infections, screening for behavioral health issues, as well as substance use screening and counseling. However, even here, these health concerns may not require in-person visits either.

While not surprising, our sensitivity analysis did indicate that a single in-person visit plus e-mail cannot guarantee that patients will go and have their laboratory work performed, a caution always worth remembering when initiating alternative encounters. However, among those patients who do go and have their laboratories done, the rates of VS among patients with one in-person visit plus e-mail are not significantly different from the rates of VS among patients with ≥2 in-person visits. These subtle differences suggest that monitoring of getting laboratory work done is key to effective HIV care.

Our study does have some limitations. First, we chose to exclude HIV-infected patients who were missing HIV RNA laboratory results, as we would be unable to determine their VS status, our primary outcome. These exclusions could have skewed our estimates of VS as well as our measurements of retention in care; however, the number of patients with missing laboratories was deemed negligible (6.7%). We also assessed VS based on a single measurement (last HIV RNA in each calendar year), which may not accurately represent each patient's VS status, as VLs can fluctuate over time due to various factors.⁵⁶ Finally, it is important to note that patients who utilize e-mail encounters may be inherently more compliant with their care or potentially different in other unmeasured ways that result in better outcomes.

Despite these limitations, our study was strengthened by the use of data from a large, integrated health system. The detailed medical records of KP contain all encounter types, including the alternatives investigated here, as well as HIV RNA laboratory results in a centralized electronic location. We also accounted for HIV risk categories in our analysis, which has been a limitation in previous research.¹⁵

There is demonstrated evidence to suggest that we should reexamine the standard definition of retention in care to include the expanding use of alternative encounter types, especially as technological advances increase and our patients utilize them more extensively. It is still unclear as to whether alternative encounters can improve other aspects of quality HIV care such as management of comorbidities, screening for sexually transmitted diseases, and screening for behavioral health issues, including substance use. Further study of use of e-mail and telephone visits in improving clinical outcomes beyond HIV RNA VL measurement is warranted. More research is needed to know if these findings apply to other care systems.

Footnotes

Acknowledgments

Funding: Support for this study was provided by the KPMAS Community Benefit Program.

Authors' Contributions

M.A.H., J.G.B., and K.B.R. performed the research. M.A.H., J.G.B., K.B.R., L.B.H., P.M.K., D.B.K., and M.J.S. designed the study. M.A.H., J.G.B., K.B.R., J.M.C., L.B.H., P.M.K., D.B.K., and M.J.S. analyzed the data. M.A.H., K.B.R., J.M.C., and M.J.S. wrote the article. M.A.H., J.G.B., K.B.R., J.M.C., L.B.H., P.M.K., D.B.K., and M.J.S. edited the article.

Author Disclosure Statement

The authors have no conflicts of interest to declare.

References

Office of National AIDS Policy. National HIV/AIDS Strategy for the United States. Updated to 2020. July 2015. Available at: https://www.hiv.gov/federal-response/national-hiv-aids-strategy/nhas-update (Last accessed July 8, 2017 ).

Cyrus

, Dawson

, Fennie

, et al. Disparity in retention in care and viral suppression for black Caribbean-born immigrants living with HIV in Florida. Int J Environ Res Public Health, 2017; 14:285.

Sabin

, Howarth

, Jose

, et al. Association between engagement in-care and mortality in HIV-positive persons. AIDS, 2017; 31:653–660.

Rebeiro

, Gange

, Horberg

, et al. Geographic variations in retention in care among HIV-infected adults in the United States. PloS One, 2016; 11:e0146119.

Yehia

, Rebeiro

, Althoff

, et al. Impact of age on retention in care and viral suppression. J Acquir Immune Defic Syndr, 2015; 68:413–419.

Rebeiro

, Althoff

, Buchacz

, et al. Retention among North American HIV-infected persons in clinical care, 2000–2008. J Acquir Immune Defic Syndr, 2013; 62:356–362.

Horberg

, Hurley

, Silverberg

, Klein

, Quesenberry

, Mugavero

. Missed office visits and risk of mortality among HIV-infected subjects in a large healthcare system in the United States. AIDS Patient Care STDS, 2013; 27:442–449.

Committee on Review Data Systems for Monitoring HIV Care; Institute of Medicine; Ford

, Spicer

, eds. Monitoring HIV Care in the United States: Indicators and Data Systems. Washington, DC: National Academies Press, 2012.

Mugavero

, Westfall

, Zinski

, et al. Measuring retention in HIV care. J Acquir Immune Defic Syndr, 2012; 61:574–580.

10.

Yehia

, Fleishman

, Metlay

, et al. Comparing different measures of retention in outpatient HIV care. AIDS, 2012; 26:1131–1139.

11.

Gardner

, Marks

, Craw

, et al. A low-effort, clinic-wide intervention improves attendance for HIV primary care. Clin Infect Dis, 2012; 55:1124–1134.

12.

Hall

, Gray

, Tang

, Li

, Shouse

, Mermin

. Retention in care of adults and adolescents living with HIV in 13 US areas. J Acquir Immune Defic Syndr, 2012; 60:77–82.

13.

Mugavero

, Davila

, Nevin

, Giordano

. From access to engagement: Measuring retention in outpatient HIV clinical care. AIDS Patient Care STDS, 2010; 24:607–613.

14.

Skarbinski

, Rosenberg

, Paz-Bailey

, et al. Human immunodeficiency virus transmission at each step of the care continuum in the United States. JAMA Intern Med, 2015; 175:588.

15.

Horberg

, Hurley

, Klein

, et al. The HIV care cascade measured over time and by age, sex, and race in a large national integrated care system. AIDS Patient Care STDS, 2015; 29:582–590.

16.

Rebeiro

, Horberg

, Gange

, et al. Strong agreement of nationally recommended retention measures from the Institute of Medicine and Department of Health and Human Services. PLoS One, 2014; 9:e111772.

17.

Thompson

, Mugavero

, Amico

, et al. Guidelines for improving entry into and retention in care and antiretroviral adherence for persons with HIV: Evidence-based recommendations from an International Association of Physicians in AIDS Care Panel. Ann Intern Med, 2012; 156:817–833.

18.

Rhodes

, Chang

, Regan

, Signer

, Triant

. HIV quality indicators are similar across HIV care delivery models. Open Forum Infect Dis, 2017; 4:ofw240.

19.

Office of National AIDS Policy. National HIV/AIDS Strategy for the United States: Indicator Supplement Updated to 2020. December 2016. Available at: https://files.hiv.gov/s3fs-public/nhas-indicators-supplement-dec-2016.pdf (Last accessed July 8, 2017 ).

20.

Halperin J

, Richey

. Laboratory markers slightly overestimate retention in HIV care among newly diagnosed individuals. AIDS Care, 2016; 28:1188–1191.

21.

Colasanti

, Kelly

, Pennisi

, et al. Continuous retention and viral suppression provide further insights into the HIV care continuum compared to the cross-sectional HIV care cascade. Clin Infect Dis, 2016; 62:648–654.

22.

Rebeiro

, Althoff

, Lau

, et al. Laboratory measures as proxies for primary care encounters: Implications for quantifying clinical retention among HIV-infected adults in North America. Am J Epidemiol, 2015; 182:952–960.

23.

Yehia

, Stephens-Shields

, Fleishman

, et al. The HIV care continuum: Changes over time in retention in care and viral suppression. PloS One, 2015; 10:e0129376.

24.

Valdiserri

, Forsyth

, Yakovchenko

, et al. Measuring what matters: Development of standard HIV core indicators across the U.S. Department of Health and Human Services. Public Health Rep, 2013; 1285:354–359.

25.

Health Resources and Service Administration. HIV/AIDS Bureau Performance Measures. January, 2015. Available at: https://hab.hrsa.gov/sites/default/files/hab/clinical-quality-management/coremeasures.pdf (Last accessed July 7, 2017 ).

26.

Catalani

, Philbrick

, Fraser

, Mechael

, Israelski

. mHealth for HIV treatment & prevention: A systematic review of the literature. Open AIDS J, 2013; 7:17–41.

27.

Collier

, Ribaudo

, Mukherjee

, et al. A randomized study of serial telephone call support to increase adherence and thereby improve virologic outcome in persons initiating antiretroviral therapy. J Infect Dis, 2005; 192:1398–1406.

28.

Gardner

, Giordano

, Marks

, et al. Enhanced personal contact with HIV patients improves retention in primary care: A randomized trial in 6 US HIV clinics. Clin Infect Dis, 2014; 59:725–734.

29.

Henry

, Goetz

, Asch

. The effect of automated telephone appointment reminders on HIV primary care no-shows by veterans. J Assoc Nurses AIDS Care, 2012; 23:409–418.

30.

Krishnan

, Ferro

, Weikum

, et al. Communication technology use and mHealth acceptance among HIV-infected men who have sex with men in Peru: Implications for HIV prevention and treatment. AIDS Care, 2014; 27:273–282.

31.

Mannheimer

, Morse

, Matts

, et al. Sustained benefit from a long-term antiretroviral adherence intervention: Results of a large randomized clinical trial. J Acquir Immune Defic Syndr, 2006; 43:S41–S47.

32.

Reynolds

, Testa

, Su

, et al. Telephone support to improve antiretroviral medication adherence: A multisite, randomized controlled trial. J Acquir Immune Defic Syndr, 2008; 47:62–68.

33.

Swendeman

, Rotheram-Borus

. Innovation in sexually transmitted disease and HIV prevention: Internet and mobile phone delivery vehicles for global diffusion. Curr Opin Psychiatry, 2010; 23:139–144.

34.

Centers for Disease Control and Prevention. Retention Through Enhanced Personal Contacts. Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention. March 2017. Available at: https://www.cdc.gov/hiv/pdf/research/interventionresearch/compendium/lrc/cdc-hiv-lrc-retention-through-enhanced-personal-contacts.pdf (Last accessed July 12, 2017 ).

35.

Pearl

Kaiser Permanente Northern California: Current experiences with internet, mobile, and video technologies. Health Aff (Millwood), 2014; 33:251–257.

36.

National Center for HIV/AIDS, Viral Hepatitis STD, and TB Prevention. Understanding the HIV Care Continuum. July 2017. Available at: https://www.cdc.gov/hiv/pdf/library/factsheets/cdc-hiv-care-continuum.pdf (Last accessed December 5, 2017 ).

37.

Kaiser Permanente. Fast Facts About Kaiser Permanente. About Us: Fast Facts. 2017. Available at: https://share.kaiserpermanente.org/article/fast-facts-about-kaiser-permanente (Last accessed August 1, 2017 ).

38.

Kaiser Permanente. Mid-Atlantic States Fast Facts. About Us: Fast Facts. 2017. Available at: https://share.kaiserpermanente.org/article/mid-atlantic-states-fast-facts (Last accessed August 1, 2017 ).

39.

Horberg

, Hurley

, Towner

, et al. HIV Quality performance measures in a large integrated health care system. AIDS Patient Care STDS, 2011; 25:21–28.

40.

Zhou

, Kanter

, Wang

, Garrido

. Improved quality at Kaiser Permanente through e-mail between physicians and patients. Health Aff (Millwood), 2010; 29:1370–1375.

41.

Chen

, Garrido

, Chock

, Okawa

, Liang

. The Kaiser Permanente electronic health record: Transforming and streamlining modalities of care. Health Aff (Millwood), 2009; 28:323–333.

42.

Silvestre

, Sue

, Allen

. If you build it, will they come? The Kaiser Permanente model of online health care. Health Aff (Millwood), 2009; 28:334–344.

43.

Zhou

, Garrido

, Chin

, Wiesenthal

, Liang

. Patient access to an electronic health record with secure messaging: Impact on primary care utilization. Am J Manag Care, 2007; 13:418–424.

44.

Zou

, Donner

. Extension of the modified Poisson regression model to prospective studies with correlated binary data. Stat Methods Med Res, 2013; 22:661–670.

45.

Liang

K-Y

, Zeger

. Longitudinal data analysis using generalized linear models. Biometrika, 1986; 73:13–22.

46.

Derman

, Garrido

, Radler

, et al. Impact of kp.org Personal Health Record with Secure Messaging on Office Visits and Patients' Calls. Paper presented at: 20th Annual. National Forum on Quality Improvement in Healthcare, December 10, 2008. Nashville, Tennessee.

47.

McCarthy

, Mueller

, Wrenn

. Kaiser Permanente: Bridging the Quality Divide with Integrated Practice, Group Accountability, and Health Information Technology. Commonwealth Fund: Organized Health Care Delivery System, 2009; 17:1–27.

48.

IOM (Institute of Medicine). Appendix B: Case Studies for the Digital Health Infrastructure. Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care: Workshop Series Summary. Washington, DC: The National Academies Press, 2011.

49.

Garrido

, Raymond

, Wheatley

Lessons from more than a decade in patient portals. April 2016. Available at: https://www.healthaffairs.org/do/10.1377/hblog20160407.054362/full/ (Last accessed September 27, 2017 ).

50.

Sarkar

, Lyles

, Parker

, et al. Use of the refill function through an online patient portal is associated with improved adherence to statins in an integrated health system. Med Care, 2014; 52:194–201.

51.

Garrido

, Meng

, Wang

, Palen

, Kanter

. Secure e-mailing between physicians and patients. J Ambulatory Care Manage, 2014; 37:211–218.

52.

Saberi

, Catz

, Leyden

, et al. Antiretroviral therapy adherence and use of an electronic shared medical record among people living with HIV. AIDS Behav, 2015; 19(Suppl 2):177–185.

53.

US Department of Veterans Affairs My HealtheVet. Secure messaging spotlight. 2016. Available at: www.myhealth.va.gov/mhv-portal-web/web/myhealthevet/secure-messaging-spotlight (Last accessed September 27, 2017 ).

54.

Haun

, Lind

, Shimada

, et al. Evaluating user experiences of the secure messaging tool on the veterans affairs' patient portal system. J Med Internet Res, 2014; 16:e75.

55.

Horberg

MA.

Editorial commentary: HIV quality measures and outcomes: The next phase. Clin Infect Dis, 2016; 62:240–241.

56.

Crepaz

, Tang

, Marks

, Hall

. Changes in viral suppression status among HIV patients receiving care: United States, 2014. AIDS, 2017; 31:2421–2425.

57.

Gardner

, Marks

, Patel

, et al. Gaps up to 9 months between HIV primary care visits do not worsen viral load. AIDS Patient Care STDS, 2018; 32:157–164.

58.

Bean

, Scott

, Kilby

, Richey

. Use of an outreach coordinator to reengage and retain patients with HIV in care. AIDS Patient Care STDS, 2017; 31:222–226.

59.

Hawk

, Coulter

RWS

, Egan

, et al. Exploring the healthcare environment and associations with clinical outcomes of people living with HIV/AIDS. AIDS Patient Care STDS, 2017; 31:495–503.