Providing Online-Based Sexual Health Outreach to Gay,Bisexual,and Queer Men in Ontario,Canada: Qualitative Interviews with Multisectoral Frontline Service Providers and Managers

Abstract

The Internet is a common tool for gay, bisexual, and other men who have sex with men (MSM) to find sexual partners and sexual health information. Given persistently high human immunodeficiency virus (HIV) infection rates among MSM, it is important to examine the role of online outreach for MSM as part of HIV prevention and care. We provide an overview of the unique perspectives of online sexual health outreach, delivered through AIDS Service Organizations (ASOs) through sociosexual Internet sites and mobile applications. Data were drawn from the qualitative arm of the community-based Cruising Counts study conducted across Ontario from December 2013 to January 2014. ASO online outreach providers and managers (n = 22) were recruited to complete a 1-h in-person/telephone interview to explore in-depth their experiences with, and perspectives on, delivering online outreach services for MSM in Ontario. Thematic analyses were conducted inductively using NVivo 10. Service providers suggested a high demand for online outreach services for MSM. Strengths and advantages of online outreach over face-to-face outreach included anonymity, instant access to services, peer model, and accessing hard-to-reach populations of MSM. Barriers included consistent quality of service, collaborations between companies that own online technologies and outreach service agencies, budgetary and staff capacity issues, and uncertainty of best practices and evaluation parameters for online outreach. Findings from these interviews can inform service providers, policy makers, and researchers on how online sexual health outreach can play a greater role in HIV prevention by better acknowledging and addressing the opportunities and barriers experienced by service providers working with MSM communities online.

Introduction

The use of mobile smartphone applications (apps) and Internet websites (from now on referred to as online technologies) have gained increased popularity among gay, bisexual men, and other men who have sex with men^1
–3 (from now on, referred to as MSM). In Ontario (Canada's most populous province), communication through online/mobile technologies has become the most common means for MSM to find sexual partners.⁴

Online technologies have significantly changed the ways MSM access health information.^5,6 These technologies have also become important tools for sexual identity development and social support for MSM,^7,8 as well as platforms for online-based sexual health outreach.⁴

The rise of the Internet as a facilitator of sexual networking has been proposed to have contributed to the reemergence of human immunodeficiency virus (HIV) and sexually transmitted infections (STIs) among MSM.^9
–11 This is an important public health concern, given that HIV and a range of other STIs disproportionately affect MSM.¹² As of 2011, 56% of people living with HIV in Ontario were MSM.¹³ In Ontario, over half of all new HIV infections (52%) are in MSM.¹⁴

Consistently, scholars have documented evidence linking Internet use with new cases of STIs and HIV among MSM.^7,15

–19 Increases in the incidence of syphilis among MSM have also been associated with increases in the use of the Internet for meeting sex partners.^20
–22

Some studies suggest that those MSM who seek sexual partners online are more likely to report having condomless anal intercourse, more sex partners, substance use during sex, a previous STI, and lower HIV testing rates.^11,23 For example, a study of 2058 MSM in France demonstrated a significant increase in unprotected anal intercourse (UAI) among MSM who met online.²⁴ Similarly, a meta-analysis examined 22 articles and showed that UAI was more likely between MSM who sought partners online, as opposed to physical venues.²⁵ Therefore, identifying online intervention to decrease sexual risk practices, encourage HIV testing, or complement existing HIV care and treatment efforts, could enable more effective targeting of HIV and STIs.

If harnessed properly, online technologies may yield vast benefits across the whole HIV care continuum, including identification of new HIV infections, linkage to care, initiation of antiretroviral therapy, retention in care, and the maintenance of undetectable viral loads.²³ Online information and communication technologies, such as Internet websites, smartphone apps, short message services (SMS), synchronous chat platforms, and social media platforms are currently utilized globally across the HIV continuum of prevention, treatment, and care.^23,26 These technologies bring multiple efficiencies to health service delivery, including cost-effectiveness and innovation,²⁷ the ability to keep track of clients for HIV testing, treatment, and care,²⁸ as well as the ability to quickly disseminate information on sexual health and HIV.²⁹

There is now growing evidence that online interventions have the potential for promoting HIV prevention practices and behaviors in MSM.^30,31 Online outreach, which is defined as the delivery of information and services regarding HIV, STIs, and more generally sexual health through online technologies,⁴ offers an opportunity to influence how HIV prevention and care initiatives are delivered to, and accessed by MSM, particularly men who are not traditionally reached, such as rural and non-gay-identifying MSM,^32
–34 as well as young MSM.³⁵ Furthermore, a systematic review of online HIV prevention interventions for MSM in the United States found that online HIV outreach is effective in increasing testing and decreasing sexual risk practices that may lead to HIV transmission.²⁷ Less is known about the Canadian context.

While emerging research from Canada suggests that online outreach services in Ontario are a relatively new and underdeveloped area of intervention,³⁶ it also highlights that online sexual health outreach is a promising health promotion strategy to provide services, education, and referrals to diverse groups of MSM.⁴

To date, there remains a lack of evidence supporting the use of online sexual health outreach with MSM in Canada. The purpose of this study was to investigate the experiences of providers at a variety of AIDS Service Organizations (ASOs), public health agencies, and other community-based organizations (CBOs) across Ontario in the implementation and use of online sexual health outreach services for MSM communities. The main goals of this study were: (1) to examine the experiences and perspectives of outreach workers in delivering HIV prevention and sexual health outreach to MSM communities online; (2) identify facilitators of and barriers to effective online outreach work across agencies serving MSM in Ontario; (3) to examine evaluation practices and parameters for online sexual health outreach for MSM in Ontario.

Methods

The Cruising Counts research project had a mixed-methods study design, which consisted of qualitative (interviews) and quantitative (online survey) arms that ran simultaneously. This community-based study was derived from, and included, an active community advisory board with representatives from diverse MSM communities across the province (MSM ethnoracialized as non-white, MSM living with HIV, and MSM who live or work outside of urban locations). This article focuses on data from the qualitative arm (interviews) of the study.

Recruitment

From November 2013 to April 2014, ASO/CBO/public health online outreach providers and managers (n = 22) were recruited to complete a 1-h in-person/telephone interview to explore in-depth their experiences with, and perspectives on, delivering online outreach services for MSM in Ontario, Canada. Interview participants were frontline service providers, public health nurses, and managers who work (previously or currently, paid or volunteer) in a variety of organizations and public health units that are involved in online outreach programs and services for gay, bisexual, two-spirit, straight, queer, cis-, and transgender men of all ages, and immigration statuses who have sex with other men.

Participants were recruited using listservs of the ASOs/CBOs, the AIDS Bureau at Ontario's Ministry of Health, and the Gay Men's Sexual Health Alliance of Ontario. Individuals that were recruited worked with organizations that fell into one of three categories: (1) currently providing online outreach services to MSM, (2) interested in or planning to provide (but are not currently providing) online outreach services to MSM, or (3) formerly provided or not planning to provide online outreach services to MSM.

All participants engaged in an informed consent process before beginning the interview. Everyone who participated was entered into a random draw to receive $100.00 (CDN) as compensation for their time.

Data collection

Data collection for the overall qualitative study consisted of exploring the issues, core values, and principles associated with doing online outreach with diverse communities of MSM. Participants were asked to identify the organizational structures, policies, or programs supportive of this work and were also asked to comment on evaluation practices, opportunities for, as well as challenges in providing online outreach. Each interview utilized a semistructured guide with scripted probes to provide an in-depth exploration of the providers' experiences and perspectives of online outreach programs and services for MSM. Some of the questions were: Can you describe how you currently do online or app-based outreach? Describe what you see as the strengths or best parts of your current online outreach programming? What areas need strengthening? Can you describe how you currently evaluate online outreach? Interviews were digitally recorded and transcribed verbatim.

Data analyses

Thematic analysis³⁷ was conducted using NVivo10 data analysis software, and was primarily framed by a SWOT framework (which focused on identifying strengths, weaknesses, opportunities, and threats/challenges of online outreach). The main task involved in drawing inferences from the data was following inductive analysis and was closely grounded in the study's objectives.

All data were compared, and similar narratives were grouped together into themes. Themes were then differentiated from one another. Based on this approach, we identified themes, which describe the findings we summarized under each category. Member checking procedures with the community advisory board and the research team were used to support validity and data trustworthiness.

Results

Twenty-two key informants took part in our interviews, including 18 frontline outreach workers from ASOs/CBOs, 2 managers who worked at ASOs/CBOs, and 2 informants who worked in public health. Seven individuals lived in Toronto (Ontario's largest city), 11 lived in Ontario outside Toronto, and 4 informants lived outside Ontario. In total four themes emerged across interviews regarding online outreach for MSM in Ontario. Overall, the findings indicate that although there is still some uncertainty about how to do online sexual health outreach well, service providers and managers consider online outreach a vital tool for HIV and STI prevention.

The goals and methods of online sexual health outreach

Service providers commented that the advent of online technologies have reorganized gay/queer community and modified traditional means of social and sexual interactions among these men: “This is where people meet now. It's the new community. There's no gay community, physically, out here anymore. So it makes sense for organizations to step up and be where everybody else is” (Outreach worker; Toronto). Respondents often commented how this service may be especially beneficial to those MSM who are not connected to the mainstream community: “Online outreach is very crucial to those people who are not getting information, are not connecting to the right community that they need … or where to get tested or just navigating coming out” (Outreach worker; Ontario).

Online outreach staff also described the methods they used to conduct online outreach and the services they provided: “We give opportunities for men who use that service to ask us questions on safe sex. Certainly, we have a lot on STIs, HIV itself, obviously. We get a lot of questions about HIV disclosure. We get a lot of questions about testing” (Outreach worker; Ontario).

Furthermore, we uncovered that most outreach on websites and online apps represents passive, nonintrusive attempts at promoting health online, which requires service users to initiate conversations with service providers: “I never initiate conversations. It's one of the policies, it is passive outreach because with a lot of the apps and the sites, it's all based on sort of the consent of the community for you to be there … So I'll post an ad that says, ‘hey, do you have questions about hooking up, oral sex, STI transmission, barebacking’” (Outreach worker; Ontario).

The strengths of online sexual health outreach

In this section, we present data on how participants described what they saw as the strengths or best parts of the way they currently provide online outreach programming. Service providers highlighted the strengths and advantages of online outreach over face-to-face outreach, such as anonymity, instant access to services, peer-to-peer model of online outreach, and accessing hard-to-reach populations of MSM. One outreach worker made comments regarding the benefits of online sexual health promotion: “I would definitely say the best part is the anonymity part. Another good thing about the online outreach that we do is the fact that it can be very instant, real time accessing a person” (Outreach worker; Ontario).

Importantly, outreach staff made the point that an issue of trust in healthcare professionals was key in MSM's use of their services: “We capitalize on the strength of our name. It's the name associated with, I think, good care for MSM.” (Public Health Nurse; Toronto). One outreach worker also expressed an appreciation of the peer-to-peer model and commented on its usefulness and relevance to the type of work that they do: “That's the type of model that we work on. We basically recruit people who are online who understand the technology world, understand the smartphone world, understand the now world. The way we cruise now is very different than the way we cruised 10 years ago” (Outreach worker; Ontario).

Another service provider commented on the ability of online sexual health outreach services in accessing hard-to-reach populations of MSM, and recalled the appreciation of service users for information that is engaging and relevant (specifically to marginalized and/or underserved communities of MSM): “We have really become able to disseminate so much information. So guys who are maybe more isolated and don't know what's going on in the community can use it as perhaps an access point to that information. Which I think is really important, especially so many of the men are older like 40 and up on [app]. So there are definitely some strength there and because the services never existed we get a lot of positive feedback and thank you's” (Outreach worker; Toronto).

Barriers to provision of effective online sexual health outreach services

Specific barriers to effective online outreach service included quality of service, collaborations between outreach service agencies and companies that own apps and websites, budgetary and staff capacity constraints, as well as work with volunteers. Within these themes, respondents also commented on the aspects of their programming that they struggle to maintain, as well as vulnerabilities that their online outreach services have. One aspect of programming that outreach workers struggled to maintain was the provision of a consistent quality of service online: “I think one of the challenges is going to be consistent messaging, and from a quality control standpoint, that everyone is providing the same message, the same way.” (Outreach worker; outside Ontario). Another common barrier identified by service providers is related to data security and safety.

As evidenced from the quote below, one of the limitations of online sexual health outreach is that agencies cannot guarantee service users that information collected by agencies, as part of online outreach, will remain confidential: “There's a lot of risk with online communication and interactions … You don't know where it's going to go, how it's going to be shared. We know that email can bounce around to many servers and can be accessed in other ways” (ASO Manager; Ontario).

Another major barrier that was identified by outreach workers is the relationships between agencies that conduct online outreach and companies that own websites and other online technologies: “One of the biggest fears that I might have, is that we really rely on the service providers and the site administrator, to allow us to be there. There are certain rules on each site and I know that over time, volunteers have mistakenly broken those rules. We've heard from the site administrators ‘watch out or we can take away your membership’ … so that's a vulnerability” (Outreach Worker; Ontario).

In a very similar fashion, the importance of dialogs and partnerships between agencies that do online sexual health outreach work and site administrators was pointed out by another participant: “I would really like to sit down with the owners of these apps and websites and have an honest conversation about the need for the presence of community organizations and being more receptive to having them online, and having some thought about what kinds of policies to put in place around the presence of an outreach worker on the site” (Outreach Worker; Toronto).

The discussion surrounding staff capacity and budgetary issues was a common narrative, endorsed by many outreach workers. Outreach workers identified important structural barriers of incorporating online outreach in their workload, discussed their struggles in meeting the quotas and deliverables for their program: “It's difficult for one guy in my agency to figure out how to logistically be present on these sites and work that into a seven-hour workday” (Outreach Worker; Toronto).

Staff capacity issues also emerged in other participants' narratives: “I think one of the things, unless you've got a number of people that are involved in working as a team, doing it on a common site, or email address, it's the length of time that it may take to respond. You need sufficient staffing, and, of course, anything online is 24 hours a day, so, that's inherently a challenge, and I think any agency that does stuff online will probably say that” (Outreach Worker; Ontario).

Outreach staff also identified important challenges related to working with volunteers, in particular the reliance on volunteer workforce and the high turnover among volunteer staff: “For volunteers, I noticed that every agency has a high turnover” (Outreach Worker, Toronto).

Online outreach service evaluation

Overall, outreach staff reported that no official (streamlined, regulated) evaluations regarding online sexual health outreach services have been conducted by their agencies: “We don't presently evaluate it, in part because it's never been a funded program” (Outreach Worker; outside Ontario). Similarly others commented: “I have never put out a community survey to see if people use it, if they like it, if they've benefited from it. I have never evaluated it that way at all” (Outreach Worker; Ontario).

Although outreach staff reported no official evaluations, most outreach workers, nevertheless, had a particular evaluation plan and indications of success, which included counting the number of contacts made during various times of days, descriptions of encounters, or the number of referrals. For example: “Indicators for success for our online outreach is when we are successfully able to answer a question properly. For instance, if they ask us information about herpes, HIV, anything like that, and if we're able to answer it, and we entice them to keep on talking, that's considered successful” (Outreach Worker; Ontario).

Many participants' narratives highlighted uncertainty about how to evaluate this service: “I think right now, it's more of the statistics. Are we getting the amount of hits and questions? Does that warrant the amount of time that we're spending? And, I think right now, we're figuring out that actually, it does, it is worth that time and effort to continue to do that” (Outreach Worker; Ontario).

Discussion

This study adds to the growing evidence^30

–33 that online technologies have the potential for HIV prevention and care with MSM. This is the first study of its kind to examine MSM's online outreach contexts from the service providers' perspective. Service providers commented on the importance, goals, and context of online sexual health outreach and considered online sexual health outreach a vital tool for HIV and STI prevention and sexual health outreach for MSM. They highlighted that there is still some uncertainty about how to do online sexual health outreach well.³⁸ Adapting current in-person outreach to online milieus and bridging public health programs into the Internet age may be a significant task. Nevertheless, service providers also described various strengths and advantages of online outreach over face-to-face outreach, such as anonymity, instant access to services, peer model, and accessing hard-to-reach populations of MSM.

Across interviews, the following elements were identified by service providers as barriers to providing effective online outreach: consistent quality of service, collaborations between companies that own online technologies and outreach service agencies, as well as budgetary and staff capacity issues. Our findings are similar to other studies that describe training and staffing needs, and highlight the difficulty of evaluation of online outreach programs.^15,39 These findings can inform HIV policy makers and funders on how to support ASOs and increase the effectiveness of online outreach services by removing the barriers identified by service providers in this study.

Collaborations with online venue owners and budgetary issues in particular were identified as a major obstacle to effective online outreach. Managers at AIDS, healthcare, and social service organizations that serve MSM in Ontario should facilitate the development of capacities and opportunities for employees involved in online outreach. This, in particular, should include capacity building, training, and continuing education opportunities for agency employees.

Furthermore, these findings highlight that it is essential to continue funding organizations that serve MSM. Partnerships between service agencies and online venue owners to promote change in these online venues may be beneficial. For example, given the significant budgetary constraints of health and social care organizations that serve MSM, online app/website owners could provide free accounts to health officials and service providers at AIDS service agencies or other CBOs for the purposes of online outreach.

Importantly, our findings offered a nuanced understanding of the challenges associated with research and program evaluations of online sexual health outreach services for MSM. These results are important given that without a clear understanding of how to evaluate online outreach work many agencies are missing a growing opportunity to improve their services to effectively engage MSM, increase condom use, and prevent new STIs.

Research on evaluation of online sexual health outreach services is scarce, due to the fact that such programs and services are a relatively new invention. However, one study documented the varied success of Internet-based interventions in reducing syphilis transmission among gay and bisexual men in San Francisco, and called for greater emphasis on impact measures in the evaluation of Internet-based outreach services.⁴⁰ Taking this into account, the evaluation of online programs for MSM is an urgent need. These findings can assist social service, healthcare, and public health agencies in meeting the needs of outreach workers in implementing and improving service evaluations.

Limitations, future directions, and implications

While our approach from the beginning was to gather and document the work of service providers, the perspectives of services users, online platform providers, or policy makers were not captured in this study. Future qualitative research should evaluate the perspectives of MSM on these services. In addition, the findings of this study may not be representative of the experiences and perspectives of all online outreach service providers, as we were primarily able to interview those service providers who worked in Ontario.

Nevertheless, even with these limitations, the findings from this study advance our knowledge by identifying the strengths and gaps in services and logistical concerns in providing this outreach.

The findings from this study can be used in several important ways. These findings have potential importance for those who provide a broad range of services to MSM. Healthcare professionals and social service providers should consider the role of facilitators, as well as the barriers identified in this study as factors involved in the provision of effective online outreach service. In particular, we recommend the production and dissemination of a range of resources that provide information and best practices for service providers involved in online outreach programs.

The findings from this study highlight the importance of a coordinated engagement (local, national, and international) between companies that own online networking apps or websites, and agencies that serve MSM to explore opportunities for health promotion for MSM. It is also essential that public health agencies partner with HIV prevention organizations and providers of these online technologies to develop, implement, and evaluate a broad range of innovative online-based prevention interventions for MSM. As a result, this study provides a platform to foster dialogs with researchers, community providers, and funders to recognize online outreach as an essential HIV care and prevention service for MSM communities in Canada, and other countries.

Footnotes

Acknowledgments

The study was supported by the Ontario HIV Treatment Network. Dr. David J. Brennan (Principal Investigator) is an Ontario HIV Treatment Network Applied HIV Research Chair (funded by the Ontario HIV Treatment Network).

Author Disclosure Statement

No competing financial interests exist.

References

Beymer

, Weiss

, Bolan

, et al. Sex on demand: Geosocial networking phone apps and risk of sexually transmitted infections among a cross-sectional sample of men who have sex with men in Los Angeles county. Sex Transm Infect, 2014; 90:567–572.

Bien

, Best

, Muessig

, Wei

, Han

, Tucker

. Gay apps for seeking sex partners in China: Implications for MSM sexual health. AIDS Behav, 2015; 19:941–946.

Blackwell

, Birnholtz

, Abbott

. Seeing and being seen: Co-situation and impression formation using Grindr, a location-aware gay dating app. New Media Soc, 2015; 17:1117–1136.

Brennan

, Lachowsky

, Georgievski

, et al. Online outreach services among men who use the Internet to seek sex with other men (MISM) in Ontario, Canada: An online survey. J Med Internet Res, 2015; 17:e277.

Grov

, Rendina

, Jimenez

, Parsons

. Using online settings to identify gay and bisexual men willing to take or with experience taking PrEP: Implications for researchers and providers. AIDS Educ Prev, 2016; 28:378–392.

Krakower

, Mimiaga

, Rosenberger

, et al. Limited awareness and low immediate uptake of pre-exposure prophylaxis among men who have sex with men using an Internet social networking site. PloS One, 2012; 7:e33119.

Grov

, Breslow

, Newcomb

, Rosenberger

, Bauermeister

. Gay and bisexual men's use of the Internet: Research from the 1990s through 2013. J Sex Res, 2014; 51:390–409.

Mustanski

, Greene

, Ryan

, Whitton

. Feasibility, acceptability, and initial efficacy of an online sexual health promotion program for LGBT youth: The Queer Sex Ed intervention. J Sex Res, 2015; 52:220–230.

Ciesielski

. Sexually transmitted diseases in men who have sex with men: An epidemiologic review. Curr Infect Dis Rep, 2003; 5:145–152.

10.

Wong

, Chaw

, Kent

, Klausner

. Risk factors for early syphilis among gay and bisexual men seen in an STD clinic: San Francisco, 2002–2003. Sex Transm Dis, 2005; 32:458–463.

11.

Rhodes

, Vissman

, Stowers

, et al. A CBPR partnership increases HIV testing among men who have sex with men (MSM): Outcome findings from a pilot test of the CyBER/testing internet intervention. Health Educ Behav, 2011; 38:311–320.

12.

Beyrer

, Baral

, Collins

, et al. The global response to HIV in men who have sex with men. Lancet, 2016; 388:198–206.

13.

Public Health Agency of Canada. Summary: Estimates of HIV prevalence and incidence in Canada, 2014. Surveillance and Epidemiology Division, Professional Guidelines and Public Health Practice Division, Centre for Communicable Diseases and Infection Control, Public Health Agency of Canada, Ottawa, Canada, 2015.

14.

Public Health Agency of Canada. HIV/AIDS Epi updates: National HIV prevalence and incidence estimates for 2011. Centre for Communicable Diseases and Infection Control, Public Health Agency of Canada, Ottawa, Canada, 2014.

15.

McFarlane

, Kachur

, Klausner

, Roland

, Cohen

. Internet-based health promotion and disease control in the 8 cities: Successes, barriers, and future plans. Sex Transm Dis, 2005; 32:S60–S64.

16.

Ostergren

, Rosser

, Horvath

. Reasons for non-use of condoms among men who have sex with men: A comparison of receptive and insertive role in sex and online and offline meeting venue. Cult Health Sex, 2011; 13:123–140.

17.

Tsui

, Lau

. Comparison of risk behaviors and socio-cultural profile of men who have sex with men survey respondents recruited via venues and the Internet. BMC Public Health, 2010; 10:232.

18.

Young

, Szekeres

, Coates

. The relationship between online social networking and sexual risk behaviors among men who have sex with men (MSM). PloS One, 2013; 8:e62271.

19.

Yang

, Zhang

, Dong

, Jin

, Han

. Prevalence of unprotected anal intercourse in men who have sex with men recruited online versus offline: A meta-analysis. BMC Public Health, 2014; 14:508.

20.

Ashton

, Sopwith

, Clark

, McKelvey

, Lighton

, Mandal

. An outbreak no longer: Factors contributing to the return of syphilis in Greater Manchester. Sex Transm Infect, 2003; 79:291–293.

21.

Centers for Disease Control and Prevention. Internet use and early syphilis infection among men who have sex with men—San Francisco, California, 1999–2003. MMWR Morb Mortal Wkly Rep, 2003; 52:1229.

22.

Ehlman

, Jackson

, Saenz

, et al. Evaluation of an innovative Internet-based partner notification program for early syphilis case management, Washington, DC, January 2007–June 2008. Sex Transm Dis, 2010; 37:478–485.

23.

Rhodes

, McCoy

, Tanner

, et al. Using social media to increase HIV testing among gay and bisexual men, other men who have sex with men, and transgender persons: Outcomes from a randomized community trial. Clin Infect Dis, 2016; 62:1450–1453.

24.

Adam

, Murphy

, de Wit

. When do online sexual fantasies become reality? The contribution of erotic chatting via the Internet to sexual risk-taking in gay and other men who have sex with men. Health Educ Res, 2011; 26:506–515.

25.

Liau

, Millett

, Marks

. Meta-analytic examination of online sex-seeking and sexual risk behavior among men who have sex with men. Sex Transm Dis, 2006; 33:576–584.

26.

Muessig

, Nekkanti

, Bauermeister

, Bull

, Hightow-Weidman

. A systematic review of recent smartphone, Internet and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep, 2015; 12:173–190.

27.

Schnall

, Travers

, Rojas

, Carballo-Diéguez

. eHealth interventions for HIV prevention in high-risk men who have sex with men: A systematic review. J Med Internet Res, 2014; 16:e134.

28.

Global Forum on MSM & HIV. Rights in action: Access to HIV services among men who have sex with men. 2015. Available at: http://msmgf.org/wp-content/uploads/2015/12/KP-Brief-MSM-FINAL.pdf (Last accessed May 12, 2018 ). Global Forum on MSM & HIV, Oakland, California.

29.

Adams

, Klindera

, Walsh

, Wolf

. Innovative programmatic approaches to HIV prevention and care services for gay men, other men who have sex with men (MSM) and transgender persons using information and communication technology (ICT). Dig Cult Educ, 2014; 6:152–163.

30.

Bourne

, Knight

, Guy

, Wand

, Lu

, McNulty

. Short message service reminder intervention doubles sexually transmitted infection/HIV re-testing rates among men who have sex with men. Sex Transm Infect, 2011; 87:229–231.

31.

Reback

, Grant

, Fletcher

, et al. Text messaging reduces HIV risk behaviors among methamphetamine-using men who have sex with men. AIDS Behav, 2012; 16:1993–2002.

32.

Saberi

, Siedle-Khan

, Sheon

, Lightfoot

. The use of mobile health applications among youth and young adults living with HIV: Focus group findings. AIDS Patient Care STDS, 2016; 30:254–260.

33.

Rana

, van den Berg

, Lamy

, Beckwith

. Using a mobile health intervention to support HIV treatment adherence and retention among patients at risk for disengaging with care. AIDS Patient Care STDS, 2016; 30:178–184.

34.

Fields

, Wharton

, Marrero

, Little

, Pannell

, Morgan

. Internet chat rooms: Connecting with a new generation of young men of color at risk for HIV infection who have sex with other men. J Assoc Nurses AIDS Care, 2006; 17:53–60.

35.

Sullivan

, Carballo-Diéguez

, Coates

, et al. Successes and challenges of HIV prevention in men who have sex with men. Lancet, 2012; 380:388–399.

36.

Ybarra

, Bull

. Current trends in Internet-and cell phone-based HIV prevention and intervention programs. Curr HIV/AIDS Rep, 2007; 4:201–207.

37.

Holloway

, Rice

, Gibbs

, Winetrobe

, Dunlap

, Rhoades

. Acceptability of smartphone application-based HIV prevention among young men who have sex with men. AIDS Behav, 2014; 18:285–296.

38.

Fantus

, Souleymanov

, Lachowsky

, Brennan

. The emergence of ethical issues in the provision of online sexual health outreach for gay, bisexual, two-spirit and other men who have sex with men: Perspectives of online outreach workers. BMC Med Ethics, 2017; 18:1–12.

39.

Hallett

, Brown

, Maycock

, Langdon

. Changing communities, changing spaces: The challenges of health promotion outreach in cyberspace. Promot Educ, 2007; 14:150–154.

40.

Klausner

, Levine

, Kent

. Internet-based site-specific interventions for syphilis prevention among gay and bisexual men. AIDS Care, 2004; 16:964–970.