Provider Perspectives on Evidence-Based HIV Prevention Interventions: Barriers and Facilitators to Implementation

Abstract

Since the beginning of the HIV/AIDS epidemic, community-based organizations (CBOs) have been key players in combating this disease through grassroots prevention programs and close ties to at-risk populations. Increasingly, both funding agencies and public health institutions require that CBOs implement evidence-based HIV prevention interventions, most of which are researcher developed. However, after completing training for these evidence-based interventions (EBIs), agencies may either abandon plans to implement them or significantly modify the intervention. Based on 22 semistructured interviews with HIV prevention service providers, this article explores the barriers and facilitators to dissemination and implementation of EBIs included in the Centers for Disease Control and Prevention's (CDC) Diffusion of Effective Behavioral Interventions (DEBI) program. Results suggest that there is a tension between the need to implement interventions with fidelity and the lack of guidance on how to adapt the interventions for their constituencies and organizational contexts. Findings suggest the need for HIV prevention intervention development and dissemination that integrate community partners in all phases of research and dissemination.

Introduction

S ince 1999, the Center for Disease Control and Prevention (CDC) has heavily invested financial, personnel, and other resources in the identification, dissemination, and training of service providers in evidence-based HIV prevention interventions. The CDC's Diffusion of Effective Behavioral Interventions (DEBI) program is one of the primary mechanisms in this dissemination effort.^1
–3 Between 1999 and 2008, the CDC trained over 10,000 service provider participants from more than 5,000 agencies as part of this program. Community-based service providers are key players in the fight against HIV/AIDS. They often have the ability to reach populations at greatest need of services, can quickly develop new programs in response to client needs, and have the potential for high levels of community participation.⁴ Despite increased demand for evidence-based HIV prevention interventions, service providers often lack access to these programs,⁵ selectively implement program components,^6,7 or never implement the programs at all.⁸ In recognition that training in the intervention alone is insufficient to ensure adoption and implementation,⁹ the CDC has spearheaded efforts to build the organizational capacity of community-based organizations (CBOs) to implement these interventions.¹⁰ These capacity-building efforts have concentrated on fiscal, administrative, and program management; community mobilization; and leadership development.^{10, 11} Despite these efforts, CBOs participating in the CDC's DEBI and technical assistance programs continue to experience high rates of staff turnover, lack of access to and use of technical assistance resource, and inconsistent program evaluation and implementation.¹² This lack of wholesale incorporation of DEBI interventions into local programming raises questions regarding the additional factors—beyond organizational capacity—that contribute to service providers' ability and desire to adopt and implement EBIs. Providers encounter additional barriers and facilitators to intervention adoption and implementation that stem from factors beyond the organization.

Factors that affect organizations' ability to adopt and implement externally developed can be divided into three (albeit overlapping and interconnected) categories: the specific intervention, characteristics of the implementing agency, and the training and technical assistance support system.¹³ At the intervention level, the programs themselves need to be well-specified, easy to use, and draw on providers' existing skills.¹⁴ At the organization level, factors associated with implementation include leadership, organizational commitment to and support of the innovation, participatory decision-making, stable and adequate resources, shared vision and goals within the organization, willingness to initiate change, and dedicated staff to implement the intervention.^14
–16 Other organizational factors affecting implementation include the fit between the new program and the agency's mission and goals.¹⁷ At the level of the training and support system, training needs to ensure that providers are adequately trained to conduct the intervention and that the resources to support providers once implementation begins, such as retraining and mechanisms for local problem solving, are in place.¹³

For the CDC's DEBI program, little research has been conducted that explores these critical factors in dissemination and implementation from the level of the interventions, implementing agencies, or the program itself. This paper explores barriers and facilitators to the adoption and implementation process from the perspective of service providers. Through in-depth, semi-structured interviews with 22 service providers in Wisconsin who have participated in DEBI training, this article explores intervention, agency, and programmatic factors in the adoption of evidence-based HIV prevention interventions among community-based service providers. Understanding the role of these factors in agencies' implementation experiences with evidence-based interventions can be used to increase provider buy-in to the program, improve the dissemination process, and facilitate the identification and development of new interventions for inclusion in the DEBI program that reflect the needs of CBOs.

Methods

Data collection

Since 2004, approximately 100 people have been trained in six different DEBI interventions through the Wisconsin AIDS/HIV Program, the lead governmental agency in Wisconsin responsible for coordinating the state's public health response to the AIDS/HIV epidemic. From 2008–2009 in-depth, semi-structured interviews were conducted with 22 frontline service providers in Wisconsin who had completed DEBI training (Table 1). Service providers were purposively selected to represent different types of organizations (e.g., AIDS-related service provision only, population-focused mission) and different regions of the state (e.g., large cities, rural areas). In total, 8 agencies were included in the study. One constituted a comprehensive AIDS service organization that offered HIV treatment, care, support, and prevention services to a broad population. The remainder of the organizations incorporated HIV prevention into a broader spectrum of programs and services targeting specific populations, such as the Hispanic community, LGBT youth and adults, and African American women.

Table 1.

Demographic Characteristics of Study Participants

	n
Gender
Male	9
Female	13
Age
Under 30	4
31–39	9
40–49	3
50–59	6
Ethnic/racial background
White	14
African American	3
Hispanic	5
Position
Director	4
Supervise staff/provide services	5
Direct service staff	13
Time at agency
1–5 years	9
6–9 years	9
10 or more years	4

We assured directors and staff that participation in the interview was voluntary and confidential, and we obtained written informed consent before the interview began. We obtained IRB approval for the study. Interview participants held a variety of positions in their respective organizations, including HIV prevention directors, organization directors, case managers, outreach workers, and prevention specialists. Interview participants included intervention staff who did not implement the target intervention, despite DEBI training (n = 5), and those who did (n = 17). Of the 22 interview participants, 8 (36%) completed more than one DEBI training. DEBI trainings included SISTA (n = 5), VOICES/VOCES (n = 8), Street Smart (n = 1), MPowerment (n = 8), Many Men, Many Voices (n = 5), and Safety Counts (n = 3). Interviews were digitally recorded, and lasted between 1.5 and 3 h. Interviews addressed motivations to attend DEBI training, opinions about the activities learned at the training, perceptions of implementation feasibility, barriers to implementation, and assessment of client reaction and receptivity to the new program. Interviews addressed overall perceptions of the quality, relevance, and effectiveness of the training and the intervention. To explore issues of “fit” between DEBI program interventions and agency goals and mission, and their influence on implementation, interviews also addressed needs of target populations, perceptions of the scope and severity of HIV risk and EBI program relevance; service providers' history with the organization; information about the organization itself and its current or past HIV prevention projects; and public policy that affects the work of the organization and/or interviewee.

Analysis

All interview recordings were transcribed verbatim and entered into a computer-based text file. Transcripts were then transferred to the software program MAXQDA, to be coded and sorted. Transcripts were first coded by gender, ethnicity, participant's title or role in the agency (e.g., director, paid staff, volunteer), population served by the agency (e.g., African American men, youth), DEBI training attended, agency type, and whether the agency implemented the intervention. Then, the documents were coded with text codes that reflected key analytical concepts, including type of HIV prevention services provided in the organization as a whole, perceived barriers to EBI implementation, posttraining follow-up and support, motivations for participating in the DEBI training, and satisfaction with the training. Analysis grouped together “implementers,” or CBO staff that implemented at least one of the DEBIs in which they were trained, and “nonimplementers,” or CBO staff that did not implement any of the DEBIs in which they were trained. Perceptions of barriers and facilitators overlapped between these two groups, and neither group identified a barrier or facilitator not mentioned by the other group.

Results

Barriers and facilitators to intervention implementation providers discussed can be grouped into three different categories, following Durlak and DuPre¹³: those that occurred at the level of the intervention itself; those that stemmed from specific aspects of organizations, (i.e., implementing and funding organizations); and those that arose from the training and support program itself (i.e., from the structure of the CDC's DEBI program).¹³ Some interview participants were trained in more than one DEBI, and too few participants were trained on a single DEBI intervention to draw conclusions about one particular intervention. Study participants drew on experiences with training in multiple evidence-based HIV prevention interventions, including some outside the structure of the DEBI program.

Intervention level factors

Intervention level barriers to implementation

Discrepancies between the intended target of the DEBI intervention and the agency's target population were a primary factor in agencies' decisions to either completely abandon plans to implement an intervention or significantly modify it. All providers mentioned “lack of fit” as a barrier to implementation, although it varied in form and degree among the agencies. At the most basic level, some organizations worked with populations for which a DEBI had yet to be developed. In these cases, an agency staff member initially attended DEBI training in order to explore the possibilities for modifying the intervention to fit their particular constituencies. For example, one agency that worked primarily with Latino MSM sent staff members to MPowerment training, an intervention developed for young MSM in general. After attending the training, the staff and prevention directors concluded that MPowerment would be more appropriate for an African American population, and therefore decided not to implement it. In other circumstances, staff judged the intervention inappropriate and never implemented it for less obvious reasons. The Safety Counts program, for example, involves a seven session intervention with hard-to-reach populations of injection drug users. One agency that considered implementing it concluded that it would be unfeasible due to the transient nature of its client base: “After having conversations with [the prevention director] about it, the people we work with are so random. There are some people we see all the time, and some we don't see for months, and then some people move” (Female, Associate Director, 9 years with agency). Given the instability of this population, the agency decided not to implement Safety Counts, even though they ostensibly worked with the intended target population. Likewise, differences between the age of the original research intervention population and the agency's target population, and resulting lifestyle differences (e.g., different time demands on teen and young adult populations) made a program appear unfeasible.

In instances in which an agency did implement the intervention, unique characteristics of the target population made it difficult to implement the DEBI as originally conceptualized by researchers. One agency implementing SISTA, an intervention developed for heterosexually active African American women, with women in a correctional setting observed:

Well, when you go into the jail system you can't leave any condoms. You can't leave literature. Initially the staff people [at the jail] wanted to sit in on a group, and that kept everybody not being open and honest. But then, they [staff people—JO] stopped coming. You can tell those women don't want to say what's going on with their life. They may be doing something right up in there [in jail], and they don't want to say. And the lack of their ability to go outside. Some women that are in those institutions whereby they're court mandated to be there, and they can't have passes and stuff. So it's really hard to gauge a behavioral change if they really aren't doing anything. [Female, direct service staff,10 years with agency]

The intended audience for the SISTA intervention and the target population of the agency matched, but the institutional context in which the program was implemented constrained what facilitators could do with the program. It also did not allow women the freedom to either talk honestly or to practice their new skills. Seventeen of the 22 interview participants mentioned challenges they faced in participant recruitment and retention. Many of the agencies encouraged participation in interventions through incentives such as gift cards for grocery stores and other retailers, cash, small gifts such as toiletries, raffles for larger prizes, and food and beverages. Despite these strategies, participant recruitment and retention remained a challenge for all agencies, particularly for agencies implementing multiple-session interventions. The research projects through which DEBI interventions are developed and tested have dedicated staff for participant recruitment and retention, and often have significant resources to use for incentive payments to encourage attendance. Therefore, researchers are often able to provide larger cohorts than many CBOs. As one service provider explained, his agency could not get the same participants to come to seven consecutive sessions as part of the Many Men, Many Voices (MMMV) program. Their clients had unpredictable work schedules and often worked irregular hours. To accommodate these realities of clients' lives, the agency decided to modify MMMV by turning it into a “retreat” in which participants attended 6- to 7-h long sessions over 2 days during the same weekend (Male, prevention specialist, 8 years with agency). Another agency implemented the five-session SISTA intervention fully aware that they would not be able to keep the same group of women through all sessions. To account for attrition, the facilitator developed a system of “certificates” in which some participants were given certificates of participation and others certificates of completion.

Many DEBIs are time-intensive, including the process of training, obtaining funding, adapting the program, and implementing the intervention. As one provider explained:

We talked about it [the possibility of implementing a DEBI—JO] and talked about it, and tried to do it and just couldn't. It was—basically we discovered that it was full time. It was a full time job just to do that program. [Another staff member] was really, really interested in trying that and doing that but 2 years later we just were like it's, it's, because those, those staff out there were one man. They do everything. They do needle exchange groups, bar outreach, internet outreach … [S]o if we had had a specific person that could be dedicated just to doing that program that would've been just great but [not] with the limitations we have. [Female, Associate Director, 9 years with agency]

Agency programs already occupy staff time and resources. These programs have demonstrated effectiveness and popularity within the agency and for their clients. Agencies often have little motivation to shift resources away from these longstanding programs to implement a program with unknown (to the providers) success or questionable appeal for target populations. The expensive and time-intensive nature of DEBIs, therefore, forced agencies to make decisions regarding their priorities:

I think one of the challenges, and this was a decision I had to make this week, actually, on the budget. DEBIs, because of their sustained pieces, it's really important to have the pieces like the food and the pieces like a $40 gift card. The reality is, it's very expensive to run them. And if you have a limited budget and you need to pay your people, you need to have pieces of your overhead in there, your indirect costs in there, all of a sudden then you're making choices. I had to make a choice this week: do we want to run the program or do I want to have tape, or do I want to have a travel budget so that I can have staff development for my staff? I chose we need to run the program for next year. [Female, Director, 1 year with agency]

The convergence of DEBIs' high cost, restrictions placed on use of funds, and what the agency needs to provide for its employees created conflicts when budget decisions were made.

Finally, a few agencies perceived the intervention to conflict with its broader organizational identity, and either abandoned the intervention or made significant modifications to it. For example, one agency worked to provide the most comprehensive and inclusive programming as practically possible. A prevention provider there criticized the narrow focus of the MPowerment program:

It was too focused. It's too rigid of a focus when you are segregating down to that [young MSM—JO]. I thought back to our group and I [realized that] I've got more women, young women that come to group than the young men. And if we are facilitating this, if we are going to be doing this and this is going to be beneficial, it needs to be broader or it needs to be more encompassing. It needs to be a space—regardless of how you identify or what you do or how you do it—it's going to be a place for you to go. I expressed that it was just too rigid, too rigid to sit down and have a group of 10 young women that identifies as lesbians and focus this thing directly at them because you get diversity within that group. [Male, Youth Specialist, 8 years with agency]

In the end, this agency implemented a modified version of MPowerment but opened participation to all interested LGBTQ youth, as well as their friends and allies. Another agency was concerned that a program's narrow focus would further isolate already marginalized populations, such as transgender individuals, and undermine the agency's central mission to combat discrimination against sexual minorities (Female, direct service staff, 1 year with agency). More broadly, some agencies observed that the interventions included as part of the DEBI program were unable to meet their clients' diverse needs, in addition to HIV prevention. One prevention director, for example, described that the agency's clients often faced problems such as alcohol and drug dependency. As a result, prevention staff dedicated significant time and effort to “ad hoc prevention case management,” which decreased staff availability for implementing a DEBI (Female, Director, 1 year with agency).

Intervention level implementation facilitators

Despite these numerous implementation barriers, interview participants also identified factors at the intervention level that facilitated their agencies' implementation of the particular DEBI. At the level of the individual DEBI interventions, the fit between the DEBI's target population and the agencies' client base was key. One agency in particular described that because these two populations were very similar, adaptation and implementation were relatively straightforward processes:

[The SISTA intervention] fit very well. But most of the trainings [prevention interventions—JO] that I was providing at first were just single trainings, you know, the DEBI became a multi-session [intervention] for one. It was more interactive in that, normally, when I'm doing an HIV 101 type of training, I am standing there just pushing information off on people. I may poll people to ask them their opinions about different things, but with DEBIs, [there are] all types of interactive exercises that people can—they don't get lost. [Female, direct service staff, 10 years with agency]

This agency already served African American women, the target population of SISTA. It built on its existing HIV programs, mobilized existing contacts, and recruited participants from its current programs for the new one. There was close alignment between the DEBI's target population and the agency's constituency (e.g., in terms of age, ethnic background, gender, and sexual identity), as well as an identified pool of potential participants. Therefore, the service providers did not need to invest resources—either staff time or money—into significant modifications to the SISTA protocol. The director of this agency described that the process of DEBI adoption and implementation “evolved really smoothly.” She emphasized that this new intervention built on the agency's existing programmatic strengths and infrastructure, and helped them expand the number of women they were able to serve (Female, Director, 2 years with agency).

Organization level factors

Organizational/institutional level barriers to implementation

Other barriers and facilitators to implementation emerge from particular aspects of the organizations and supporting institutions (e.g., funding agencies). As Somlai et al. (1999) documented, the median time for frontline service providers to hold their positions is one year.¹⁸ Even though the individuals interviewed for this study had been at the agencies for a median of 8 years, staff turnover remained a significant barrier to intervention implementation. Interview participants suggested that high rates of staff turnover affected their ability to implement new interventions, particularly those that required staff members to undergo intensive training provided by external facilitators. In this study, four interview participants held the position of agency director, and an additional five people oversaw other staff. Of these nine participants, eight mentioned staff turnover as a significant problem. One prevention director of an agency that eventually implemented a DEBI after a long delay, described the problem:

I know a challenge was staffing. Early on, we'd get somebody hired, trained and the momentum going and then [he] leaves. And then hire and trained and momentum going and now left again. This is like the third staff person in 2 years. So that's been a huge challenge. And I think a lot of that has to do with probably the credentials of the person that we're looking for in the position. Ideally, we like somebody that has at least started the process with AODA [alcohol and other drug abuse—JO] counseling, just the group facilitation piece. And then, the salary maybe is the issue—that we're not paying an AODA counselor's salary. Then they're getting a job in an AODA field and then moving on. And then also too, our grant target has one of the primary population's Hispanics. So getting somebody bilingual also has been a huge challenge. Bilingual individuals, especially AODA counselors, are pretty hard to keep because they're sought after. [Male, Prevention Director, 9 years with agency]

The inability to retain highly skilled staff made it difficult for this agency to implement and sustain complex HIV prevention programs. In other cases, changes in leadership resulted in similar disruptions in plans to implement interventions. Vacant leadership positions created breakdowns in communication during periods of restructuring and shifts in agency-level priorities that came with new leadership.

Prior to adopting a DEBI, most agencies had existing programs that they had been conducting, with anecdotal success, for many years. Therefore, the adoption of new programs often required agencies to make decisions regarding resource allocation, use of staff time, and agency-level priorities. The time intensive nature of many DEBI interventions meant that agencies would have to cut back on existing programs to dedicate sufficient staff time to DEBI implementation. One agency, for example, had to make the decision to allocate funds for either its HIV testing program or the MPowerment program. The agency prioritized HIV testing, and therefore decided against implementing MPowerment (Male, Prevention Specialist, 9 years with agency). Likewise, for agencies with a more comprehensive mandate beyond HIV prevention, for example the provision of HIV care-related services, money for prevention programming often decreases as these services expand and costs increase. Without additional funds, many service providers found that they had fewer resources available to dedicate to HIV prevention, including DEBI interventions:

I do know that prevention dollars for education were cut and therefore programs … When I was a prevention education specialist, that program was cut. It was the HRH, High Risk Heterosexual funding we were getting. That was cut, so there was not that much money for any more education. And that was probably close to the time that maybe they just decided not to implement [VOICES/VOCES], I don't know. Because initially, the agency has to take in the cost of whatever these programs are going to cost before the CDC or someone else … gives money for it. [Female, Prevention Director, 5 years with agency]

In this case, scarce resources for prevention forced the agency to consider whether the DEBI would offer something new, and concluded that it duplicated services they already provided through existing programs. Therefore, they abandoned plans to implement VOICES/VOCES.

The competition between programs emerged from lack of available resources, particularly financial resources. From service providers' perspectives, funders' restrictions on the ways in which money can be used, unpredictable and low funding levels, and funders' lack of awareness of community prevention needs impeded DEBI implementation. One prevention director, for example, criticized both state and federal agencies for being too rigid in the populations an agency could target with prevention dollars and the types of programs they were willing to fund. She observed that funders were slow to recognize that, “The demographics around [the agency] have been shifting. …We were hired to do prevention in the Latino and Latina community, and there was no mention of the fact that there still needed to be gay outreach” (Female, Director, 1 year with agency). Even the frontline providers—those individuals actually implementing intervention and not directly involved in funding applications and decisions—recognized the limitations that inadequate funding levels placed on the agencies' ability to conduct its programs. Several providers commented that while funding levels decreased, they were still expected to meet the same participant numbers. In some cases, other program costs were so high that no money remained to provide participant incentives, which providers saw as essential for recruitment and retention of participants through multisession interventions (JO13).

Organizational/institutional level implementation facilitators

Providers described favorable conditions within their agencies that enabled them to implement new interventions. Particularly, broad support within the organization at all levels (e.g., directors, other frontline staff) was key:

My supervisors have always been like, “How are things going? Are we implementing it? Have you had to make any adaptations? What is working and what is not working?” [They] do a pretty good job of it but it's because … they both have been doing HIV work in excess of 15 years, so they have been trained in the models as well. So they know they get [understand—JO] them and they follow up: “Is this working? Oh, OK.” [Male, direct service staff, 4 years with agency]

This provider highlighted the support of upper level management within the agency as key to successful intervention implementation. As another provider at a second agency observed, “You gotta send people [to training]—not only your outreach staff—but you got to send managers, or people that are going to supervise” (Female, direct service staff, 14 years with agency). From her perspective, supervisors and managers needed to be familiar with the intervention in order to appropriately allocate resources, guide adaptation, and support implementation.

At the same time, frontline outreach staff expressed that they also needed flexibility at all levels—from funders, from other prevention staff, and supervisors—to respond to needs created by new programs. Funder flexibility with respect to how agencies spent prevention money, for example allowing them to use funds to pay for incentives, enabled agencies to increase participation and retention, and therefore continue to offer the intervention. Likewise, flexibility within the organization to expand programs and rethink the agency's mission, based on outreach staff's determination of need, also facilitated implementation. As one frontline provider observed, the balance between management involvement and flexibility was key:

I have to say that … the President and the Vice-President of this here organization … allow me to the full capacity to be able to serve them without … stepping in. They do guide the program. They will oversee the parts of what it takes in that part of management, but as far as delivering with fidelity and different things, they never hinder, and they really allow me to be able to go and do my thing … They're really good. [Female, direct service staff, 8 years with agency]

Management provided support where it was needed, but the agency valued this provider's experience, and gave her latitude to implement the program as she saw best.

Organization support for programs also involved guidance regarding implementation and adaptation, including communication between agencies regarding their implementation experiences, and capacity-building organizations to assist in the process. A central agency, in this case the State of Wisconsin's Division of Public Health, played a pivotal role in coordinating these various forms of support. As one program director summarized:

I actually think that the State does a really good job with that. What has happened in the past, when we've had a training from the State, they will bring the trainers in … We go through the training. Then, a couple things can happen. [Sometimes] there is a specific group that is trained, and they're followed on the State level and the State will support us—when we got trained for VOICES/VOCES they made extra money available to buy a variety of condoms because a lot of people can't go out and buy a whole bunch … They've supported us in either having follow up meetings. They've had follow up conference calls, which can be helpful. [Our grant monitor has] CBO meetings, and every other month all of the folks who are funded by the State meet either in Madison or in the Milwaukee area. We get together and we talk about what we're implementing, how we're implementing it, what's really going well, what's not going well. Something that's not working for us may or may not be working for somebody else. And if [our grant monitor] keeps hearing there's something that really isn't working for everybody, she'll [say], “Ok, well we've got to do something about that,” versus if somebody is doing it well and then there's one person just struggling, then that's another opportunity to fit people together so that they have support from their other peers out doing HIV. [Female, Director, 1 year with agency]

Coordination through state agencies enabled organizations to communicate with one another about their implementation experiences. It also allowed them to use the success and failures of these other agencies to guide their own program development.

Program level factors

Programmatic level barriers to implementation

Interview participants identified significant barriers to DEBI adoption and implementation that they saw as products of the DEBI program itself. Both implementing and nonimplementing agencies noted similar barriers at the level of the DEBI program. All interview participants expressed concerns about DEBI adaptation, modification, and fidelity as barriers to intervention implementation. Some interview participants expressed confusion regarding what elements of and the degree to which a DEBI could be modified, the impact of modification on effectiveness, and who should be responsible for intervention modification to adapt a DEBI for a new target population. One agency serving a predominately Hispanic population, for example, understood the need to implement evidence-based HIV prevention interventions. They turned to the menu of DEBI interventions for a possible program. The organization also worked with women, including transgender individuals, and the SISTA program (originally developed for African American women) seemed to be the best fit. Adapting this intervention for a new population, however, presented a significant barrier to implementation, as the agency director explained:

I don't think anybody thinks that adapting is a bad reason, is a bad idea. I think that—this is just my impression—that the state thinks that this is something that the CDC developed [so] it's their responsibility to come up with a Latino one … the state isn't going to take it on … For an individual agency to take it on it would be a huge amount of work. [Female, Prevention Director, 6 years with agency]

In this example, adapting SISTA for a Latina population involved more than creating Spanish-language materials. It would involve translating an entire culture into the intervention. For a community-based organization, this process would constitute “a huge undertaking” for which they do not necessarily have the staff and training.

Respondents also indicated that different agencies interpreted issues of intervention fidelity and modification differently. Some agencies reviewed interventions and concluded that if they could not implement it with high fidelity, they would not implement it at all:

We looked at MPowerment really closely. [My colleague] and I wanted to do it, but knew we couldn't do it like it was written—to the extent that they wanted it done … [We] thought maybe we could adapt it, but we were told we shouldn't adapt it. [Female, Associate Director, 9 years with agency]

Likewise, a service provider at a second agency reported that training participants were told that they were not permitted to “deviate” from the intervention, but they were permitted to “modify” it. From his perspective, this meant that they could do things such as cover material in more or fewer sessions than written in the manual, or change language, such as using colloquial terms for “penis” (JO02). In his understanding of the distinction between “deviation” and “modification,” skipping sections constituted unallowable “deviations.” He lamented the lack of any further guidance about what could be changed and to what degree. Others acknowledged that existence of post-training support, in the form of consultation with trainers, was offered. However, some service providers deliberately did not raise their questions about acceptable intervention modification. After being “really kind of encouraged to follow [the curriculum] as closely as we could,” service providers at one agency realized they would need to modify MPowerment to fit its constituency and mission:

P: Some of these other things that they suggested don't seem 100% vital to it and they don't fit for us anyway. So we are just going to change those but we felt like we really kept true to the core of the initiative, like the sessions and have it be group led, and building community. That was most important that we kept that.

I: So you decided on your own what you saw as the core elements of this program?

P: Yeah we did it and we did not go back to the people and say this is how we are going to do it. Partially because we knew they might say, “Well, that is not going to work.” So … we knew in our heads probably what they were going to say but we didn't really want to hear them say because we knew we had to do this way anyway. We are going to have to do [it] this way and we just did it. [Male, Associate Director, 1 year with agency]

In this case, the agency deliberately avoided raising issues of fidelity and modification with trainers, preferring to determine on their own how they thought the intervention needed to be modified.

That this particular organization purposefully avoided accessing post-training supports hints at another concern raised by training participants: the lack of follow-up and accountability built into the DEBI program. One participant observed that most of the communication between participants and trainers was “one way:”

P: There wasn't a ton of follow up. I think that a lot of the assistance was one sided, us reaching out to them. There wasn't necessarily a lot—or any, that I can remember—follow up on their end. Like, “Alright you came to this thing. What have you guys done?” The door was open but just us to talk to them.

I: Do you think that that would've been helpful if they would've made more effort to contact you?

P: Yeah, I think it would have been helpful. I think that it definitely would've helped with accountability if we didn't do anything and they call us and we may feel guilty like, “Oh my gosh! We totally went to this training nine months ago and we haven't done anything with it.” Also, it could just help us reinforce … we may have then asked other questions that didn't feel important enough for us to call and ask this question. [Male, Associate Director of Prevention, 1 year with agency]

Likewise, training participants lamented that they did not have the opportunity to discuss the implementation process either with the facilitators or with other implementing agencies:

I think it will be good if the same people who were involved in facilitating the training came [back for] meetings and discussed certain things [such as] what else can be done differently. I know that's a lot of money—those strangers came from Colorado, but I think giving us continual education about the curriculum and how people implemented it … the experiences of other people doing it. That would be helpful. [Male, prevention specialist, 8 years with agency]

Similarly, other training participants suggested that it would have been helpful to have a “tune up” session that involved discussions between facilitators and regional agencies about their experiences implementing the intervention, raise questions about modifications, and verify that they are doing the intervention correctly (JO10, JO16).

Several participants also raised concerns about the training facilitators, particularly related to their hands-on experience with the intervention and their familiarity with the participating agencies' target populations. As one participant summarized:

I think there was a lot of book knowledge … It sounded beautiful, but the reality was, “OK, this is going to be hard.” For MPowerment, it was like someone coming from California, where there is a population that we are trying to reach, and that's a big minority number. Being MSM themselves, it was more effective [than] having someone having the book knowledge and just talking about it, saying, “This is what you do next” and “This is what you will do at the end.” And I'm thinking, “OK, it sounded nice.” And you get out there thinking, “I can do this,” but then you sit down and see, “OK, where are we going to get these people from.” [Male, direct service staff, 10 years with agency]

In this instance, the trainers' lack of direct experience with the target population and understanding of the on-the-ground challenges the agencies would face with basic components of the program, such as participant recruitment, limited their usefulness to trainees. Another provider at a different agency commented that, “[The trainers] were very focused on, well, ‘This is how you spend your 40 hours. You have plenty of time to get this and this done.’ But they didn't take into consideration that we don't have that luxury. We are one person and have to do the testing, the driving back and forth, and the office stuff” (Male, prevention specialist, 5 years with agency). In other words, the trainers had unrealistic expectations of what types of program would be possible for this agency to implement, given its staff limitations, resource constraints, and competing demands on staff time. This same service provider, however, also suggested that perhaps if both upper level management and frontline providers had attended the training, the agency as a whole would have been better prepared to make the necessary resource allocation decisions. This raises a final point regarding implementation barriers: many times, service providers attend training without any clear agency-wide plan to implement the intervention. Rather, agencies sent participants to trainings as part of generalized skill-building, or in order to make initial determinations if this particular intervention would be desirable or feasible for the agency to implement.

Programmatic level implementation facilitators

At the level of the DEBI program itself, accessible facilitators who could answer questions about the curriculum after the training, guidance regarding intervention fidelity, and an emphasis on “core elements” helped agencies in the implementation process. Most interviewees noted that DEBI training facilitators continually emphasized the importance of core elements and fidelity to them during implementation. Moreover, interviewees indicated that trainers made themselves available to the participants through phone and email after the training ended. However, as described above, many trainees reported that they did not initiate contact with their trainers. Again, the balance between fidelity to the core elements and the ability to adapt the program to the agency's specific needs was critical:

We kind of checked in with the trainers and the trainers really kind of encouraged [us] to follow as closely as we could … We took that into mind as best we could adapt it to our population. We knew that we didn't want to exclude women because we thought that would not send the right message from our program. So we knew we wanted to include women, so that was something that we had to stray from. We looked at the initiative and said, “This is what we see as the core—these are the most important things—and we are going to do those things.” Some of these other things that they suggested don't seem 100% vital to it and they don't fit for us anyway. So, we are just going to change those but we felt like we really kept true to the core of the initiative. [Male, Associate Director of Prevention, 1 year with agency]

A clear fit between the intervention and agency target populations resulted in easier intervention implementation, and required less staff time and resources for adaptation and modification.

Discussion

As previous research has demonstrated, organizational capacity significantly influences providers' adoption and implementation of evidence-based interventions. However, this study's findings illustrate the role of institutional, intervention, and programmatic factors in the diffusion process. The barriers to intervention implementation outlined by participants suggest that HIV prevention interventions are often developed without a thorough understanding of the actual capabilities and needs of the implementing organizations. For example, scientists often design multi-session interventions as the “gold standard” of HIV prevention programs.¹⁸

However, as the comments above illustrate, the small client base of many CBOs or limited organizational resources for prevention are not conducive to this type of program. In addition, there may be value incongruence between the ideological principles that guide the agency's programs and those that underlie the evidence-based prevention program; the existence of programs within an agency that they see as working; and a wide range of stakeholders within the agency that need to be in favor of any new programs before they can be implemented.

Providers pointed to the tension between fidelity and the lack of guidance on how to adapt the intervention for their specific constituencies and organizational contexts. Service providers indicated that many of the DEBIs needed to be adapted to fit their particular organizational context, address the needs of their target populations, and respond to diversity within their constituency. Providers also recognized that, in order for an intervention to be considered a DEBI and retain the label “effective,” it must be implemented with fidelity to certain “core elements.” However, study participants also highlighted that attention to core elements was not sufficient for their adoption and adaptation efforts. In many cases, providers recognized the importance of core elements but still modified programs or abandoned plans to implement them. In particular, many providers expressed greater concern for meeting the diverse needs of their constituents and remaining consistent with agency core mission and values, than implementing a program with fidelity to core elements. More broadly, agencies highlighted the need to develop interventions that reflect their resource context, for example programs that work well with small client bases or highly mobile target populations. That is, while particular DEBIs may closely reflect the ethnic and demographic characteristics of the implementing agencies, these criteria alone do not determine whether a particular intervention will work within the context of the particular agency or its constituent communities.

In addition, many of the providers in this study observed that the interventions included in the DEBI program did not reflect their particular client base. In response, agencies either modified an intervention to make it more suitable for their clients, or abandoned plans to implement an intervention after they determined it was not a good fit. These providers experiences reflect that the prevention needs of certain groups are underrepresented in HIV prevention research in general, and in the DEBI program in particular. For example, there are currently no DEBI interventions for older individuals^20,21 or transgender populations.²² Other groups, such as adolescents, remain underrepresented as well.²³

The results of this study offer several places where the DEBI program can be changed to improve its acceptance by community organizations, especially in the areas of adaptation and follow-up. Clearer guidelines as to aspects of an intervention that can be changed to respond to agency and client needs, and still retain its effectiveness, should be developed. Resources should be elaborated for CBOs and technical staff to appropriately modify/adapt interventions. Attention should be paid to developing the kind of expertise (e.g., knowledge of the theoretical foundations of behavioral science based interventions), particularly within CBOs, that is needed to make these changes and do them well. Guidance on adaptation versus fidelity could be improved through increased follow-up after training. Follow-up could provide an opportunity for providers to ask questions regarding the implementation process and possibilities for modification of the intervention, and contribute to an increased sense of accountability to implement the intervention. Service providers also suggested the need for clearer delineation of responsibilities related to the process of DEBI adaptation, especially given the time, staffing, and resource constraints of agencies.

This study presents several limitations. First, some interview questions asked participants to recall training and subsequent implementation experiences several years after they were completed. Interviewees often had difficulty recalling specific details about the implementation process, including how decisions were made regarding DEBI “fit,” what aspects of the prevention intervention were retained and which were changed, and particular aspects of the training. Second, due to the small sample size, it is difficult to make conclusions regarding differences in attitudes toward the DEBI program and whether an agency implemented the intervention in which they were trained. Finally, the interviewees participated in trainings for very different interventions, conducted by very different facilitators and in different settings. The small sample size and research design do not allow for the effects of particular facilitators or training experiences to be explored.

Because adoption, implementation, and modification processes are influenced by organizational factors, it is worth exploring how aspects of organizational capacity, systems of support and technical assistance, and elements particular to one intervention interact and affect implementation of a single EBI. Further study of the implementation process needs to compare the preparations different types of agencies make in order to implement a new intervention, understand what agencies do with these interventions (e.g., how they modify them) when they do implement them, and explore how adopting these interventions changes the services agencies provide and the populations they are able to serve. Finally, more effectiveness studies that examine the impact of interventions in community settings will allow both researchers and practitioners to identify organizational and programmatic resources needed to improve these interventions' success in applied settings.

Author Disclosure Statement

No competing financial interests exist.

References

Lyles

, Crepaz

, Herbst

, Kay

. Evidence-based HIV behavioral prevention from the perspective of the CDC's HIV/AIDS prevention research synthesis team. AIDS Educ Prev, 2006; 18,Suppl A:21–31.

Nutley

, Walters

, Davies

. From knowing to doing: A framework for understanding the evidence-into-practice agenda. Evaluation, 2003; 9:125–148.

Rugg

, Buehler

, Renaud

et al. Evaluating HIV prevention: A framework for national, state, and local levels. Am J Eval, 1999; 20:35–56.

Kelly

, Somlai

, Benotsch

et al. Programmes, resources, and needs of HIV-prevention nongovernmental organizations (NGOs) in Africa, Central/Eastern Europe and Central Asia, Latin America and the Caribbean. AIDS Care, 2006; 18:12–21.

Collins

, Harshbarger

, Sawyer

, Hamdallah

. The diffusion of effective behavioral interventions project: Development, implementation, and lessons learned. AIDS Educ Prev, 2006; 18,Suppl A:5–20.

Galbraith

, Stanton

, Boekeloo

et al. Exploring implementation and fidelity of evidence-based behavioral interventions for HIV preventions: Lessons learned from the Focus on Kids Diffusion Case Study. Health Educ Behav, 2009; 36:532–549.

Palinkas

, Schoenwald

, Hoagwood

et al. An ethnographic study of implementation of evidence-based treatments in child mental health: First steps. Psychiatr Serv Jul, 2008; 59:738–746.

Veniegas

, Kao

, Fosales

, Arellanes

. HIV prevention technology transfer: Challenges and strategies in the real world. Am J Public Health, 2009; 99,S1:S124–S130.

Dusenbury

, Brannigan

, Falco

, Hansen

. A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Educ Res, 2003; 18:237–256.

10.

Collins

, Phields

, Duncan

. An agency capacity model to facilitate implementation of evidence-based behavioral interventions by community-based organizations. J Public Health Manag Pract, 2007; 1,suppl:S16–S23.

11.

Thomas

, Smith

, Wright-Deaguero

. The program evaluation and monitoring system: A key source of data for monitoring evidence-based HIV prevention program processes and outcomes. AIDS Educ Prev, 2006; 18,Suppl A:74–80.

12.

Taveras

, Duncan

, Gentry

, Gilliam

, Kimbrough

, Minaya

. The evolution of the CDC HIV prevention capacity-building assistance initiative. J Public Health Manag Pract, 2007; January,Suppl:S8–S15.

13.

Durlak

, DuPre

. Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol, 2008; 41:327.

14.

Rohrbach

, Grana

, Sussman

, Valente

. Type II translation: transporting prevention interventions from research to real-world settings. Eval Health Prof, 2006; 29:302–333.

15.

DiFranceisco

, Kelly

, Otto-Salaj

et al. Factors influencing attitudes within AIDS service organizations toward the use of research-based HIV prevention interventions. AIDS Educ Prev, 1999; 11:72–86.

16.

Livet

, Courser

, Wandersman

. The prevention delivery system: Organizational context and use of comprehensive programming frameworks. Am J Community Psychol, 2008; 41:361.

17.

Miller

. Innovation in HIV prevention: Organizational and intervention characteristics affecting program adoption. Am J Community Psychol, 2001; 29:621–647.

18.

Somlai

, Kelly

, Otto-Salaj

et al. Current HIV prevention activities for women and gay men among 77 ASOs. J Public Health Manag Pract, 1999; 5:23–33.

19.

Miller

, Shinn

. Learning from communities: Overcoming difficulties in dissemination of prevention and promotion efforts. Am J Community Psychol, 2005; 35:169–183.

20.

Theall

, Elifson

, Sterk

, Klein

. Perceived susceptibility to HIV among women: Differences according to age. Res Aging, 2003; 25:405–432.

21.

Orel

, Stelle

, Watson

, Bunner

. No one is immune: A community education partnership addressing HIV/AIDS and older adults. J Appl Gerontol, 2010; 29:352–370.

22.

Reisner

, Perkovich

, Mimiaga

. A mixed methods study of the sexual health needs of New England transmen who have sex with nontransgender men. AIDS Patient Care STDs, 2010; 24:501–513.

23.

Bouris

, Guilama-Ramos

, Jaccard

, McCoy

, Aranda

, Pickard

, Boyer

. The feasibility of a clinic-based parent intervention to prevention HIV, sexually transmitted infections, and unintended pregnancies among Latino and African American adolescents. AIDS Patient Care STDs, 2010; 24:381–387.