Internalized HIV Stigma Predicts Suboptimal Retention in Care Among People Living with HIV in the United States

Abstract

HIV-related stigma is a known barrier to retention in care. However, no large-scale, multi-site studies have prospectively evaluated the effect of internalized stigma on retention in care. The Centers for AIDS Research Network of Integrated Clinical Systems (CNICS) cohort study integrates medical record and survey data from people living with HIV (PLWH) seen in HIV primary care clinics across the United States, and assesses internalized stigma yearly using a validated 4-item Likert scale. We used multivariable logistic regression models to evaluate associations between mean internalized stigma and two prospective retention in care outcomes: keeping the next primary care appointment and keeping all scheduled primary care appointments in the 12 months following stigma assessment. From February 2016 to November 2017, 5968 PLWH completed the stigma assessment and had adequate follow-up time. Mean stigma was 1.9 (standard deviation 1.08). Increased mean stigma scores were associated with decreased odds of attending the next primary care appointment [adjusted odds ratio (aOR) = 0.93, 95% confidence interval (CI) 0.88–0.99, p = 0.02], and all primary care appointments in the subsequent 12 months (aOR = 0.94, 95% CI 0.89–0.99, p = 0.02). In both models, younger age and Black race were also independently associated with suboptimal appointment attendance. There was no support for interactions between internalized stigma and covariates. Internalized HIV stigma had an independent negative effect on the odds of subsequent appointment attendance. This study highlights the importance of identifying even low levels of internalized stigma. Interventions to address internalized HIV stigma are critical to supporting retention in care and improving clinical outcomes.

Introduction

Throughout the HIV epidemic, HIV-related stigma has negatively impacted both the societal response to the disease and the individual health of people living with HIV (PLWH).^1,2 However, its effects on retention in HIV care are not well understood.

HIV stigma research is framed around three conceptual definitions: anticipated, enacted, and internalized stigma.³ Internalized stigma, the degree to which PLWH endorse the negative beliefs associated with HIV about themselves, is unique in that it does not imply an individual has disclosed his or her status, so may be most inclusive of all individuals living with HIV. Internalized stigma in particular has also been associated with adverse interpersonal consequences such as not disclosing status to recent partners or family members, potentially leading to less community support or engagement in one's health.⁴

The bulk of HIV-related stigma literature shows associations with antiretroviral therapy (ART) adherence, both directly and through mediators of self-efficacy, depression, or other psychological factors.^5
–7 In the last 10 years, recognizing the association of missed HIV primary care visits with higher mortality,⁸ research has increasingly explored the effect of stigma on retention in care. However, studies have often been theoretical and qualitative^9,10 and the empirical data are conflicting. Two studies of HIV clinic patients, one in Birmingham, Alabama, and the other in Miami, Florida, found that higher internalized stigma was associated with lower visit adherence.^11,12 A study of HIV clinic patients in rural Georgia demonstrated an association between cumulative stigma and disengagement from care over an 18-month period.¹³ However, a study of men who have sex with men (MSM) in Boston, Massachusetts, found that internalized HIV stigma was not associated with HIV care appointment adherence.¹⁴ A retrospective study of Latino and African American MSM in HIV care in Los Angeles demonstrated no association of HIV stigma with retention in care, but did show an association between stigma related to sexual minority identity and retention.¹⁵ Finally, among mostly MSM PLWH newly presenting to a safety-net HIV clinic in San Francisco, higher internalized stigma at clinic intake was paradoxically associated with more successful linkage to primary care and better retention over the first year of care, raising the possibility that an unmeasured mediator, such as social support from the clinic, helped promote retention.¹⁶ In summary, these single-site studies, many of which focus on specific populations or are retrospective in design, have yielded inconsistent results.

Given that a prospective association between internalized HIV stigma and retention in care has not been definitively demonstrated, we sought to leverage the multi-site, geographically diverse Centers for AIDS Research Network of Integrated Clinical Systems (CNICS) cohort to investigate this question, especially as prior work by our group found evidence of a cross-sectional association.¹⁷

Methods

Study setting and participants

CNICS is a network of eight academic HIV primary care clinics across the United States that includes the University of Alabama Birmingham (UAB), the University of California San Diego (UCSD), Johns Hopkins University (JHU), University of Washington (UW), Fenway Health, University of California San Francisco (UCSF), University of North Carolina (UNC), and Case Western Reserve University (CWRU).¹⁸ The cohort integrates electronic medical record (EMR) data with patient-reported and measured outcome (PRO) data that are collected via tablet-based surveys every 4–6 months during primary care visits.¹⁹ Each CNICS site has institutional review board approval to send deidentified EMR and PRO data to a coordinating center at UW. Each site followed approved local IRB protocols for data collection and data security, and the authors adhered to UCSF Committee on Human Research policies for working with deidentified data.

For this study, we included all PLWH who completed a PRO survey with stigma instrument from February 2016 to November 2017. Of note, CWRU began to administer PRO surveys after the time period of this analysis and UNC did not have data on no-show primary care visits, and thus, these sites were excluded.

The PRO surveys collect data on sexual orientation and gender identity and include validated measures of ART adherence, depressive symptoms, substance use, and sexual risk behavior. To these, we added a validated 4-item assessment of internalized HIV stigma with Likert response scale: 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree (Cronbach's alpha 0.91) on a yearly basis. Survey items included, “Having HIV makes me feel like I am a bad person,” “Having HIV is disgusting to me,” “I feel ashamed of having HIV,” and “I think less of myself because I have HIV.” The assessment was adapted from the validated 6-item assessment developed by Earnshaw et al., shortened to be incorporated into primary care from a research setting.²⁰

Predictor

Mean stigma score was the primary predictor, with higher scores representing greater internalized stigma. Responses were on a continuous scale from 1 to 5 and we calculated the mean only for participants who completed at least 3 of the 4 items.

Outcomes

Outcomes for this study were as follows: (1) keeping the next primary care appointment following the stigma assessment, and (2) keeping all scheduled primary care appointments in the 12 months following the stigma assessment.

Covariates

We controlled for categorized age at the time of stigma assessment, gender, race/ethnicity, sexual orientation, number of scheduled primary care appointments, time since CNICS enrollment in years, and CNICS site. For the second outcome, we also adjusted for the number of scheduled appointments in the follow-up period.

Statistical analysis

We used descriptive statistics to characterize the sample. Of the 5968 participants who had a stigma assessment, 96.6% had at least one appointment scheduled in the 12 months of follow-up and were included in analyses. We assumed that the 3.4% who did not have scheduled follow-up had transferred care.²¹

We used direct maximum likelihood (ML) estimation in Mplus 8.3 to fit unadjusted and adjusted multivariable logistic regression models of mean stigma, covariates, and the two outcomes. ML permits the inclusion of all cases with missing data on the predictor or the covariates under the assumption of being conditionally missing at random. We screened for two-way interactions between stigma and each covariate individually, choosing between full and nested models; in each case, the model with the lowest Bayesian information criterion (BIC) would determine if the interaction should be retained. We investigated the assumption of a linear relationship between mean stigma and the log odds of the two outcomes by substituting restricted cubic splines for mean stigma and by comparing the BIC between the models, including all splines to the nested models where estimates for splines were constrained to zero.

For statistical significance, we used an a priori alpha of 0.05, and all testing was two-sided.

Results

Between February 2016 and November 2017, 5968 participants completed the stigma assessment. Of these, 45 participants (0.75%) completed fewer than 3 of the 4 stigma items and therefore were coded as missing on stigma assessment. Complete covariate data were available for 5825 participants.

At the time of stigma assessment, the median age of the study population was 49 years, with an interquartile range (IQR) of 39–56 years. Forty-nine percent (49%) of participants were 50 years or older. Gender identity distribution was 80% cis-male, 17% cis-female, and 3% gender minority. Thirty-two percent (32%) of participants identified as heterosexual and 68% not heterosexual. Forty-one percent (41%) of participants were White, 39% Black, 15% Latinx, and 4.5% other (Table 1). Distribution of participants among sites was 28.7% UCSD, 31.3% UAB, 10.8% UW, 9.1% Fenway Health, 14.7% JHU, and 5.4% UCSF.

Table 1.

Sample Characteristics (N = 5968)

Characteristic	N (%)
Age, years
19–29	469 (7.9)
30–39	1090 (18.3)
40–49	1487 (24.9)
50+	2922 (49.0)
Current gender
Cis-male	4788 (80.2)
Cis-female	1018 (17.1)
Gender minority	162 (2.7)
Heterosexual orientation	1889 (32.2)
Race
Black	2317 (39.1)
White	2435 (41.1)
Latinx	907 (15.3)
Other	267 (4.5)
CD4,^a median (IQR)	563 (365–802)
Current ART	5384 (92.8)
Unsuppressed viral load (>200 copies/mL)^b	707 (12.8)
Years in CNICS, median (IQR)	6.3 (2.8–11.4)

Variables with missing data (number missing) include: race (42), heterosexual orientation (102), and current ART (164).

CD4 closest to the stigma assessment from −180 days before to 90 days after stigma assessment (N = 3440).

VL closest to the stigma assessment from +/−90-day window (N = 5506).

ART, antiretroviral therapy; CNICS, Centers for AIDS Research Network of Integrated Clinical Systems; IQR, interquartile range.

Mean stigma value was 1.9 (standard deviation 1.08) of a possible range of 1–5. Median stigma was 1.75 (IQR 1–2.75). A minority of participants agreed or strongly agreed with the stigma items: 9.6% for “Having HIV makes me feel like I am a bad person,” 24.7% for “I feel ashamed of having HIV,” 17.5% for “I think less of myself because I have HIV,” and 11.6% for “Having HIV is disgusting to me.” However, 29% of participants agreed or strongly agreed with at least one stigma item.

A median of 3 HIV primary care appointments (IQR 2–5) were scheduled in the 12 months after stigma assessment. Eighty percent of participants kept the first appointment and 56% kept all appointments. Each unit increase in mean stigma was statistically significantly associated with decreased odds of keeping the next primary care appointment [adjusted odds ratio (aOR) = 0.93, 95% confidence interval (CI) 0.88–0.99, p = 0.02], and keeping all primary care appointments in the subsequent 12 months (aOR = 0.94, 95% CI 0.89–0.99, p = 0.02) (Table 2). In both models, younger age and Black race were independently associated with suboptimal appointment attendance. Non-cis gender identification was associated with decreased odds of keeping the next appointment. Each increase in scheduled appointments was associated with 30% decreased odds of keeping all appointments in the subsequent 12 months. Interaction screening yielded models that retained no stigma-by-covariate interactions. The linearity assumption held for both outcomes.

Table 2.

Unadjusted and Adjusted Odds Ratios for the Association Between Stigma and Keeping Next Primary Care Appointment and Keeping All Primary Care Appointments in the Following Year

Characteristic	First primary care appointment				All primary care appointments
Characteristic	OR (95% CI)	p	aOR (95% CI)	p	OR (95% CI)	p	aOR (95% CI)	p
Mean stigma	0.89 (0.84–0.95)	<0.0001	0.93 (0.88–0.99)	0.02	0.86 (0.82–0.91)	<0.0001	0.94 (0.89–0.99)	0.02
Age
18–29	0.43 (0.35–0.54)	<0.0001	0.39 (0.31–0.51)	<0.0001	0.46 (0.38–0.57)	<0.0001	0.34 (0.27–0.43)	<0.0001
30–39	0.55 (0.46–0.65)	<0.0001	0.52 (0.43–0.63)	<0.0001	0.55 (0.47–0.63)	<0.0001	0.43 (0.37–0.52)	<0.0001
40–49	0.65 (0.55–0.76)	<0.0001	0.63 (0.54–0.74)	<0.0001	0.63 (0.55–0.71)	<0.0001	0.51 (0.44–0.60)	<0.0001
50+	Reference		Reference		Reference
Race/ethnicity
Black	0.70 (0.61–0.81)	<0.0001	0.60 (0.50–0.72)	<0.0001	0.66 (0.59–0.75)	<0.0001	0.61 (0.52–0.72)	<0.0001
Latinx	0.65 (0.54–0.79)	<0.0001	0.85 (0.70–1.04)	0.09	0.58 (0.50–0.68)	<0.0001	0.92 (0.76–1.12)	0.41
Other	0.82 (0.59–1.13)	0.22	1.15 (0.82–1.61)	0.44	0.66 (0.50–0.85)	0.002	0.99 (0.67–1.20)	0.46
White	Reference		Reference		Reference
Gender identity
Cis-female	0.95 (0.80–1.12)	0.52	0.97 (0.79–1.20)	0.80	0.86 (0.75–0.98)	0.03	1.11 (0.92–1.34)	0.89
Gender minority	0.60 (0.42–0.86)	0.005	0.71 (0.49–1.03)	0.03	0.55 (0.40–0.76)	<0.0001	0.76 (0.53–1.09)	0.14
Cis-male	Reference		Reference		Reference
Sexual identity
Heterosexual	0.99 (0.87–1.14)	0.93	1.01 (0.83–1.22)	0.95	0.83 (0.75–0.93)	0.001	0.89 (0.74–1.03)	0.11
Sexual minority	Reference		Reference		Reference		Reference
Number of appointments	—	—	—	—	0.69 (0.67–0.72)	<0.0001	0.70 (0.67–0.72)	<0.0001
Years in CNICS	1.02 (1.01–1.03)	0.001	1.00 (0.98–1.01)	0.69	1.03 (1.02–1.04)	<0.0001	1.01 (0.99–1.02)	0.37

The models were also adjusted for site, but results are not shown here for clarity.

aOR, adjusted odds ratio; CI, confidence interval; CNICS, Centers for AIDS Research Network of Integrated Clinical Systems.

Bold values represent statistical significance.

Discussion

Greater internalized stigma was associated with worse short-term retention in care in a large, multi-site study of PLWH at the US academic HIV primary care clinics, defining retention as attendance at the next primary care appointment and attendance at all primary care appointments in the subsequent 12 months.

These findings differ from those of prior studies that had focused specifically on MSM populations,^14
–16 but reinforce study findings of clinic populations with different sexual orientations.^11,12 Heterosexual-identified individuals have been shown previously to perceive more negative reactions to HIV from their social environment,^22,23 although sexual orientation was not a significant independent predictor of retention in our models. While sites had different levels of mean stigma (data not shown), we found no evidence of interaction between stigma and site, suggesting that the site variable was not driving the demonstrated effect. Black race was independently associated with suboptimal retention in care in both models. This association has been reported previously and is likely due to multiple structural and interpersonal barriers, including racial inequities in health insurance status, a concentration of health facilities in non-Black communities, among others.^24
–26 Latinx ethnicity was associated with poor retention in the unadjusted analysis only, but this association was not statistically significant in the adjusted models. Prior literature is robust in showing that Latinx populations have demonstrated good HIV care cascade outcomes, aside from late diagnosis rates, and despite significant barriers to care.²⁷ However, much of this research is not unpacked by place of birth, language, and years in the United States. The Latinx population seen in CNICS is predominantly English speaking. As English is a proxy for immigration status, our findings may more closely approximate findings in nonimmigrant Latin American populations.

This study has many strengths: prospective study design, large sample size, and use of a multi-site cohort and validated stigma scale, but there are limitations. The study sites are academic medical centers, and so, findings may not be generalizable to community settings. We assessed internalized stigma only. There may be selection bias as not all PLWH presenting to clinic complete the PROs. However, our prior cross-sectional work on stigma in CNICS used inverse probability weighting to account for differences between those who do and do not complete the PRO assessment and found very similar results to unweighted models.¹⁷ There may be covariates we did not include that modify or explain the results, such as depressive symptoms, self-efficacy, or perceived social support.^15,28
–30

As we move closer to ending the epidemic, it is increasingly important to identify and address barriers to sustained HIV care even among small groups of PLWH. Missed clinic visits have been associated with greater cumulative viral load, negatively impacting both individual mortality and public health outcomes.⁸ Therefore, while the mean stigma score in this study was low and had modest effects on retention in care, the implications are important.

This study provides strong evidence that internalized HIV stigma negatively impacts retention in care. Future research should now explore the mechanistic pathways underlying or modifying this association, strategies for decreasing internalized stigma, and interventions to mitigate the effects of stigma on retention, such as provider/patient engagement.¹² Future research may also examine the role of other types of stigma in retention, most importantly intersectional stigma, which nests HIV stigma within stigma related to interdependent social identities such as sexual orientation, race, and gender, as well as health care-specific enacted stigma, which has been recently shown to be associated with nonadherence and nonsuppression.^31

–34

Addressing HIV stigma is critical to improving both the individual health of those living with HIV and the health of our communities.

Footnotes

Authors' Contributions

C.A.P. was the primary writer of the article. M.O.J. and K.A.C. aided C.A.P. in the conception of the work, analysis, and provided critical edits to the article. T.B.N. and S.E.D. contributed to the statistical analysis as well as to the edits to the section of the work. All the other authors, J.A.S., M.J.M., H.M.C., R.J.F., W.C.M., R.D.M., S.N., and K.H.M., contributed to the critical edits of the final article. All authors approved the final article.

Acknowledgments

The authors thank Valerie Earnshaw and Laramie Smith for the 4-item version of their 6-item scale.

Meetings at Which These Data Have Been Presented

Presented at the 27th Conference on Retroviruses and Opportunistic Infections, Boston, MA (Virtual), March 8–11, 2020.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This work was supported by the National Institutes of Health R01 MH102198-S1, R24 AI067039, and T32 AI 7641-17.

References

Herek

, Capitanio

. AIDS stigma and sexual prejudice. Am Behav Sci, 1999; 42:1130–1147.

Parker

, Aggleton

. HIV and AIDS-related stigma and discrimination: A conceptual framework and implications for action. Soc Sci Med, 2003; 57:13–24.

Earnshaw

, Chaudoir

. From conceptualizing to measuring HIV stigma: A review of HIV stigma mechanism measures. AIDS Behav, 2009; 13:1160–1177.

Overstreet

, Earnshaw

, Kalichman

, Quinn

. Internalized stigma and HIV status disclosure among HIV-positive black men who have sex with men. AIDS Care, 2013; 25:466–471.

Sumari-de Boer

, Sprangers

, Prins

, Nieuwkerk

. HIV stigma and depressive symptoms are related to adherence and virological response to antiretroviral treatment among immigrant and indigenous HIV infected patients. AIDS Behav, 2012; 16:1681–1689.

Sweeney

, Vanable

. The association of HIV-related stigma to HIV medication adherence: A systematic review and synthesis of the literature. AIDS Behav, 2016; 20:29–50.

Blake Helms

, Turan

, Atkins

, et al. Interpersonal mechanisms contributing to the association between HIV-related internalized stigma and medication adherence. AIDS Behav, 2017; 21:238–247.

Mugavero

, Lin

, Willig

, et al. Missed visits and mortality among patients establishing initial outpatient HIV treatment. Clin Infect Dis, 2009; 48:248–256.

Mahajan

, Sayles

, Patel

, et al. Stigma in the HIV/AIDS epidemic: A review of the literature and recommendations for the way forward. AIDS, 2008; 22(Suppl 2):S67–S79.

10.

Yehia

, Stewart

, Momplaisir

, et al. Barriers and facilitators to patient retention in HIV care. BMC Infect Dis, 2015; 15:246.

11.

Rice

, Crockett

, Mugavero

, Raper

, Atkins

, Turan

. Association between internalized HIV-related stigma and HIV care visit adherence. J Acquir Immune Defic Syndr, 2017; 76:482–487.

12.

Valverde

, Rodriguez

, White

, Guo

, Waldrop-Valverde

. Understanding the association of internalized HIV stigma with retention in HIV care. J HIV AIDS, 2018; 4.

13.

Kalichman

, Katner

, Banas

, Hill

, Kalichman

. Cumulative effects of stigma experiences on retention in HIV care among men and women in the rural Southeastern United States. AIDS Patient Care STDS, 2020; 34:484–490.

14.

Traeger

, O'Cleirigh

, Skeer

, Mayer

, Safren

. Risk factors for missed HIV primary care visits among men who have sex with men. J Behav Med, 2012; 35:548–556.

15.

Wohl

, Galvan

, Myers

, et al. Do social support, stress, disclosure and stigma influence retention in HIV care for Latino and African American men who have sex with men and women?. AIDS Behav, 2011; 15:1098–1110.

16.

Christopoulos KA

, Dilworth

, Neilands

, Gandhi

, Geng

Higher levels of internalized HIV stigma at clinic intake paradoxically predict more successful linkage to primary care. In: 11th International Conference on HIV Treatment and Prevention Adherence. Fort Lauderdale, FL, May 9–11, 2016.

17.

Christopoulos

, Neilands

, Hartogensis

, et al. Internalized HIV stigma is associated with concurrent viremia and poor retention in a cohort of US patients in HIV care. J Acquir Immune Defic Syndr, 2019; 82:116–123.

18.

Kitahata

, Rodriguez

, Haubrich

, et al. Cohort profile: The Centers for AIDS Research Network of Integrated Clinical Systems. Int J Epidemiol, 2008; 37:948–955.

19.

Kozak

, Mugavero

, Ye

, et al. Patient reported outcomes in routine care: Advancing data capture for HIV cohort research. Clin Infect Dis, 2012; 54:141–147.

20.

Earnshaw

, Smith

, Chaudoir

, Amico

, Copenhaver

. HIV stigma mechanisms and well-being among PLWH: A test of the HIV stigma framework. AIDS Behav, 2013; 17:1785–1795.

21.

Christopoulos

, Scheer

, Steward

, et al. Examining clinic-based and public health approaches to ascertainment of HIV care status. J Acquir Immune Defic Syndr, 2015; 69(Suppl 1):S56–S62.

22.

Giordano

, Hartman

, Gifford

, Backus

, Morgan

. Predictors of retention in HIV care among a national cohort of US veterans. HIV Clin Trials, 2009; 10:299–305.

23.

Christopoulos

, Das

, Colfax

. Linkage and retention in HIV care among men who have sex with men in the United States. Clin Infect Dis, 2011; 52(Suppl 2):S214–S222.

24.

Gaskin

, Dinwiddie

, Chan

, McCleary

. Residential segregation and disparities in health care services utilization. Med Care Res Rev, 2012; 69:158–175.

25.

Adeyemi

, Livak

, McLoyd

, Smith

, French

. Racial/ethnic disparities in engagement in care and viral suppression in a large urban HIV clinic. Clin Infect Dis, 2013; 56:1512–1514.

26.

Dailey

, Johnson

, Wu

. HIV care outcomes among Blacks with diagnosed HIV—United States, 2014. MMWR Morb Mortal Wkly Rep, 2017; 66:97–103.

27.

Ruiz

, Steffen

, Smith

. Hispanic mortality paradox: A systematic review and meta-analysis of the longitudinal literature. Am J Public Health, 2013; 103:e52–e60.

28.

Yigit

, Bayramoglu

, Weiser

, et al. Changes in internalized stigma and HIV health outcomes in individuals new to HIV care: The mediating roles of depression and treatment self-efficacy. AIDS Patient Care STDS, 2020; 34:491–497.

29.

Johnson

, Neilands

, Dilworth

, Morin

, Remien

, Chesney

. The role of self-efficacy in HIV treatment adherence: Validation of the HIV Treatment Adherence Self-Efficacy Scale (HIV-ASES). J Behav Med, 2007; 30:359–370.

30.

Fredericksen

, Fitzsimmons

, Gibbons

, et al. Development and content validation of the Multifactoral assessment of perceived social support (MAPSS), a brief, patient-reported measure of social support for use in HIV care. AIDS Care, 2019; 31:1–9.

31.

Turan

, Elafros

, Logie

, et al. Challenges and opportunities in examining and addressing intersectional stigma and health. BMC Med, 2019; 17:7.

32.

Sangaramoorthy

, Jamison

, Dyer

. Intersectional stigma among midlife and older Black women living with HIV. Cult Health Sex, 2017; 19:1329–1343.

33.

Rice

, Logie

, Napoles

, et al. Perceptions of intersectional stigma among diverse women living with HIV in the United States. Soc Sci Med, 2018; 208:9–17.

34.

Algarin

, Sheehan

, Varas-Diaz

, et al. Health care-specific enacted HIV-related stigma's association with antiretroviral therapy adherence and viral suppression among people living with HIV in Florida. AIDS Patient Care STDS, 2020; 34:316–326.