Recruiting and retaining service agencies and public health providers in longitudinal studies: Implications for community-engaged implementation research

Abstract

This article addresses a lack of attention in the implementation science literature regarding how to overcome recruitment and retention challenges in longitudinal studies involving large samples of service agencies and health service providers (“providers”). Herein, we provide a case-illustration of procedures that improved recruitment and retention in a longitudinal, mixed-method study—Project Interprofessional Collaboration Implementation—funded by the US National Institute of Mental Health. Project Interprofessional Collaboration Implementation included counselors, program workers, educators, and supervisors. We present a research-engagement model to overcome barriers that included developing a low-burden study, social gatherings to engage stakeholders, protocols to recruit agencies and providers, comprehensive record-keeping, research procedures as incentives to participation, a plan to retain hard-to-reach participants, and strategies for modifying incentives over time. Using our model, we retained 36 agencies over the life of the project. Between baseline (N = 379) and 12-month follow-up (N = 285), we retained 75% of the sample and between the 12- (N = 285) and 24-month follow-ups (N = 256), we retained 90%. For qualitative interviews (between baseline and 12-month follow-up and between 12- and 24-month follow-ups), we retained 100% of the sample (N = 20). We provide a summary of frequency of contacts required to initiate data collection and time required for data collection. The model responded to environmental changes in policy and priorities that would not have been achievable without the expertise of community partners. To recruit and retain large samples longitudinally, researchers must strategically engage community partners. The strategies imbedded in our model can be performed with moderate levels of effort and human resources. Creating opportunities for research partners to participate in all phases of the research cycle is recommended, which can help build research capacity for future research.

Keywords

Community-engaged research longitudinal design cohort retention implementation health research HIV/AIDS mixed methods

To advance the implementation of evidence-based practices, we must engage and retain large cohorts of community-based service agencies (“agencies”) and public health services providers (“providers”) in research studies. Nonetheless, providers and the agencies for which they work are often overburdened with large caseloads and lack of time and resources. Therefore, “implementation research is beset by a ‘small n’ problem”; creative methods are required (Proctor et al., 2009). To be relevant and to succeed, researchers must strategically and fundamentally engage community partners. Principles and the best practices of community-engaged research have been employed worldwide to guide the work of researchers, providers, and community members as equal partners in each phase of the research cycle (Belone et al., 2016; Cashman et al., 2008). Researchers have typically been involved in the procedural aspects of research such as advising on recruitment strategies, gaining entry into hard-to-reach populations, and communicating community risks and benefits to their partners. They have also been involved in substantive aspects of research such as the specification of aims, data collection, analysis and interpretation, and dissemination (Spector and Pinto, 2017). A thorough assessment of community-engaged research outcomes has shown improved overall research quality (Viswanathan et al., 2004); however, publications seldom detail methodological practices for recruiting and retaining large cohorts of providers over long periods of time.

To improve practitioners’ use of evidence-based practices, it is essential to document and understand the long-term course of partnerships between researchers and practitioners. The use of checklists of key concern in designing and conducting longitudinal studies, and the employment of well-defined procedures for data collection, have both been suggested to help overcome barriers to conducting longitudinal studies in practice settings such as recruitment and retention of large cohorts (Ployhart and Vandenberg, 2010; Van Weel et al., 2006). When followed, these recommendations can improve researchers’ capacity to collect organizational- and provider-level data over time. For decades, strategies with the potential to improve recruitment and retention in health and public health research have been described in the literature: developing promotional materials that speak of the cultural beliefs of potential participants, offering incentives to participate, and providing childcare and transportation whenever needed (Swanson and Ward, 1995).

More specifically, flexible scheduling of interviews and multiple reminder phone calls have been recommended to sustain communication among research teams and both agencies and individual research participants (Witte et al., 2004). But missing in the literature are strategies for overcoming challenges in recruiting and retaining health and public health providers, particularly for longitudinal studies that require repeated measures and thus demand more time for sustained participation. Potential strategies for overcoming challenges regarding recruitment and retention of study sites (e.g. service agencies) and service providers include involving agency staff as research collaborators to design and develop research methods and procedures (Witte et al., 2004). Isolated strategies to recruit and retain participants over time in longitudinal studies are not in themselves great innovations; however, the engagement of community partners in performing a systematized combination of research tasks and procedures known to have a positive effect on recruitment and retention is innovative (Faridi et al., 2007; Viswanathan et al., 2004). This will be demonstrated by a case illustration below.

Missing data are common in any research effort, but this is particularly true in more complex studies, such as longitudinal and mixed-methods designs, where attrition (loss of follow-up) of study participants leads to decreased sample sizes over time and may cause bias in data analysis and interpretation (Ibrahim and Molenberghs, 2009; Little and Rubin, 1987). The combination of qualitative and quantitative approaches within longitudinal research is feasible and encouraged, but it also introduces further biases that may arise from the absence of quantitative or qualitative data or both (Clark et al., 2015). Studies that call for the collection of longitudinal qualitative and survey data measures may end up with distorted results when there is loss of both types of data. Imputation techniques may be used to correct quantitative analyses (Asendorpf et al., 2014; Biering et al., 2015), but these methods rely on untestable or random assumptions about the lost participants and may thus lead to biased results and some distortion of the findings. Qualitative data can help assuage distortions by describing, explaining, and contextualizing human behavior and social phenomena.

Loss of data may jeopardize data analysis and interpretation, but studies that employ rigorous methods for recruitment and retention have the potential to generate the comprehensive datasets needed for innovative data analysis. For example, multilevel data (from providers nested within agencies) and longitudinal survey data (from repeated surveys across time within the same providers) allow for multilevel structural equation modeling (SEM) to study temporal and interactive relationships at the provider and agency levels simultaneously (Rabe-Hesketh et al., 2004). Moreover, comprehensive datasets, including qualitative data (e.g. archival data or in-depth interviews), allow for rich descriptions of organizational changes, analysis of different levels of influence, and predictive models, including myriad influences on organizational and provider changes over time. These innovations require researchers to secure participation and subsequent recruitment and retention of large samples of agencies, of the individuals who manage these agencies, and of providers who offer services to clients/patients.

Although the literature provides recommendations for best research practices, specific examples or explorations of the steps researchers may take to improve retention and avoid attrition are hard to find. Research on service practices requires a comprehensive examination of both organizational data (e.g. capacity, size, number, and types of services provided) and provider-related data (e.g. demographics, including years of experience, attitudes about evidence-based practices, and frequency of provision of services). Models of organizational evolution have been described in the literature, characterizing organizations facing constraints (the inertial model), those that are highly influenced by environmental conditions (the external-control model), and those in which senior managers determine patterns of change (the strategic-management model; Romanelli and Tushman, 1986). In choosing a research design involving large samples of service agencies, it is thus important to consider and collect data on the influence of managers, service providers, and the environment on organizational changes.

This article addresses key gaps in the literature—a lack of details about how previous studies overcame recruitment and retention challenges in longitudinal studies involving large samples of service agencies and providers, and a lack of specific examples to improve retention and avoid attrition. We have used a case-illustration approach to narrow these gaps. In using a specific research project to illustrate the methodological issues we seek to uncover and address, this article describes tested procedures that improved actual recruitment and retention in a longitudinal study involving 36 agencies and 379 providers. Our case illustration focuses on Project Interprofessional Collaboration Implementation (ICI), a 5-year study (2012–17) funded by the US National Institute of Mental Health (NIMH; R01MH095676). Project ICI is the first longitudinal, mixed-method study designed to examine the impact of collaboration among providers in service agencies (“interprofessional collaboration”) as they implement evidence-based HIV-prevention services (“HIV-prevention services”) in New York City. Providers include counselors, program workers, educators, and supervisors.

With an eye toward narrowing the gaps identified above, herein we demonstrate how we reframed the original study aims to respond to environmental changes (federal and local policy changes) that hindered service agencies’ and providers’ capacity to participate in a longitudinal study. We also provide a model for engaging, recruiting, and retaining large samples of service agencies and providers. The case illustration offers detailed descriptions of barriers to recruitment and retention and how to address them, and examples of two innovations (criteria sampling and SEM) made possible because of the completeness of the datasets produced by Project ICI.

Case illustration

Project ICI is a 5-year, mixed-methods, and longitudinal study whose design includes the following:

Recruitment and retention of 250 providers in 36 nonprofit agencies in New York City, an epicenter of the HIV pandemic, over a baseline and 12- and 24-month follow-ups. Baseline and follow-up surveys were conducted via face-to-face interviews.

In-depth qualitative interviews with a randomly selected sub-sample of 20 providers. Qualitative interviews took place between the baseline and the 12-month follow-up, and between the 12- and 24-month follow-ups. Participants were selected using criteria sampling and provided face-to-face interviews.

Training providers in collaborative practices to improve implementation of evidence-based HIV-prevention services. Project ICI training took place between the baseline and the 12-month follow-up and it was meant to spark interprofessional collaboration among the providers in the study.

Table 1 provides a summary of Project ICI’s design and methods. Our sequential mixed-method approach is grounded in diffusion of innovations, institutional, and behavioral theories. Diffusion of innovations and institutional theories suggests that implementation involves providers who routinely facilitate evidence-based interventions in agencies of local diffusion systems (Ajzen, 1991; Ajzen and Fishbein, 1980; Gotham, 2004; Lehman et al., 2002; Rogers, 1995). Reasoned action and planned behavior define the key factors influencing implementation: providers’ attitudes, self-efficacy, and subjective norms. Project ICI examines the role of interprofessional collaboration in resolving day-to-day challenges providers face as they implement HIV-prevention services. The project is grounded in the evidence that interprofessional collaboration—mutual referral making, information sharing, service evaluation, and outcome dissemination—may help improve agencies’ and providers’ ability to deliver HIV-prevention services.

Table 1.

Summary of Project ICI design and methods.

Recruitment, incentives, and baseline interviews	Before submitting this proposal to the NIMH, the research team approached agency representatives by letter, inviting them to join the study. Upon confirmation, agency representatives were reminded about recruitment procedures and inclusion criteria. Agency executive directors were asked to encourage their staff to participate. Because some agencies operate with severe financial constraints, flexibility was needed to recruit providers. Thus, different schedule arrangements for data collection were put in place. To be eligible, providers had to work 75% of the time for their agencies and be at least 18 years old. Each participant received US$20–US$50 as an incentive, following completion of each interview. Baseline interviews were conducted face-to-face. Agency representatives completed organizational surveys about their agencies, including the number of consumers they served, the types of services they offered, and barriers they experienced with regard to service delivery
ICI training	The ICI training was proposed as a strategy to spark and stimulate interprofessional collaboration, a practice already in place in the service environment in New York City, so that the relationships between interprofessional collaboration and service delivery could be studied. After the baseline was established, providers were offered ICI training. Before we submitted this study proposal, an ICI training component was piloted with 20 providers from diverse agencies and backgrounds. We constructed training that was aligned with CDC’s vision of improving collaboration in order to enhance HIV/AIDS prevention. We collaborated with pilot participants to ensure ICI training acceptability, structure, content, and language. We used a satisfaction survey to assess participants’ satisfaction with the training and the extent to which the training met its objectives. The 12-month follow-up captured immediate post-training changes in providers’ interprofessional collaboration and their implementation of HIV-prevention services. The 24-month follow-up detected further changes
12- and 24-month follow-ups	Follow-up surveys were conducted via face-to-face interviews by research staff
In-depth interviews	We used sampling criteria to select providers at different levels, and longitudinal patterns of interprofessional collaboration and service delivery (e.g. effective behavioral intervention implementation and referral-making). We compared levels (on a 5-point Likert-type scale) of interprofessional collaboration and service delivery at the baseline and the 12-month follow-up. If an individual provider’s level of interprofessional collaboration and service delivery changed, we considered that provider’s service delivery as having improved or declined. Random sampling was then used sequentially to select five providers in each category (improvement, decline, and consistently high or low). Each of these providers then gave two in-depth interviews: one after the 12-month follow-up and another after the 24-month follow-up. Being able to interview the same providers a second time added further detail to the longitudinal data. Providers were asked to describe strategies that optimized domains of interprofessional collaboration, and to give examples of barriers and resources that could help them overcome barriers. Interviews were digitally recorded, transcribed, and downloaded into NVivo software for data management and analysis

ICI: Interprofessional Collaboration Implementation; NIMH: National Institute of Mental Health; CDC: Centers for Disease Control and Prevention.

It has been well documented that single-cohort, long-term longitudinal survey design has key advantages over cross-sectional survey design, including the opportunity to collect baseline information about demographics and cognitive aspects of participants, and to track these variables over time. Longitudinal data provide insight into how natural changes occur and thus allow for behavioral prediction. Nonetheless, long-term longitudinal survey research can delay results and suffer from cumulative attrition. Therefore, use of multiple single cohorts, each one starting at a different age, has been recommended for achieving the benefits of longitudinal methods while reducing attrition (Galbraith et al., 2017). Since our research participants were service providers, their ages were important in that the literature shows that age is related to provider use of research findings in practice (Chagnon et al., 2010; Owczarzak and Dickson-Gomez, 2011; Pinto et al., 2013; Spector and Pinto, 2017). Moreover, regardless of the age at which a provider starts his or her career, providers learn more about all issues related to practice as time goes on. Therefore, Project ICI involved service providers of different ages at different points in their careers.

Study context

The literature suggests that in choosing a research design involving large samples of service agencies, researchers ought to consider the influence of the socioeconomic environment on organizational changes (Romanelli and Tushman, 1986). Project ICI’s procedures have been influenced by environmental changes that will likely not be unique to HIV-prevention studies, given global economic and public health policy shifts. We present these changes and then discuss their implications for future studies. Initially, Project ICI aimed to study exclusively the Centers for Disease Control and Prevention (CDC) Diffusion of Effective Behavioral Interventions (DEBI) program, which took off in 1999 (CDC, 2010). Dissemination of evidence-based HIV prevention in the United States has taken place largely through the DEBI program, in which agencies received funding and technical assistance (TA) to deliver “packaged” behaviorally based programs that had been selected as “best evidence” programs (CDC, 2013; Collins et al., 2006). This national approach is based on best evidence from Kelly et al. (2000a, 2000b); it offered paper-based manuals, training workshops, and follow-up TA.

Between 2012 and 2017, despite the need for its benefits, DEBI implementation remained limited (Durlak and DuPre, 2008; National Institutes of Health, 2006; Norton et al., 2009; Tinkle et al., 2013; Witte et al., 2015). The HIV-prevention field was experiencing shifts in scientific priorities based on changing global health policies and economics. These shifts included movement from behavioral to biomedical prevention approaches; promotion of behavioral couple-based models to overcome barriers in traditional individually based models; and the recognition that efficacious multisession programs need to be briefer for sustainability in service settings and that programming should target individuals living with HIV and their intimate partners as opposed to HIV-negative individuals from the general public or from specific high-transmission risk groups (Mermin and Fenton, 2012; Rotheram-Borus et al., 2012). The United States was also slowly rebounding from an economic recession. This context, in light of the traditional barriers to implementation efforts in real-world HIV-services agencies, made it even more challenging for agencies to participate in research efforts.

As a part of these priority shifts, in 2011, the CDC began to deemphasize the DEBI program and launched the High-Impact HIV Prevention approach, emphasizing only the interventions that were most cost-effective and whose key behavioral changes related to HIV testing and primary care access (CDC, 2011, 2015). The high-impact approach requires providers to offer individuals of unknown HIV status access to HIV testing, primary care, biobehavioral interventions, support services, and structural interventions such as syringe-exchange programs.

This consolidation of programming was taking place just after the economic recession that began in 2008, which had a devastating effect on HIV-services agencies in New York State; some lost as much as 30% of their funding within 3–6 months. It was already a time of low or dwindling resources and of considerable organizational disruption in the provision of services due to the recession and related cuts. Together, these events resulted in many agencies losing funds and resources, thus hindering their ability to focus on anything but recovering from their losses. Many agencies had to dismantle entire HIV-prevention departments and terminate scores of providers, then hire and train less-experienced staff to fill the gaps. Service providers, a workforce characterized by high burnout and turnover, became more difficult to recruit and retain. Although these challenges appeared unique to this project, changes in the fiscal and geopolitical environment are relevant to any research project. Therefore, how we addressed and adapted to a new context in response to external changes may help researchers facing different challenges in other geographic contexts. Below, we describe how best to engage and retain large numbers of service agencies and providers over long periods of time in longitudinal studies. Specifically, we provide a model of community engagement and set forth the specific adaptations we made to our research procedures in order to respond to policies affecting the agencies and providers involved in Project ICI.

Research-engagement model

Globally, community-engaged research is the leading paradigm with a theoretical and empirical basis for understanding and advancing implementation readiness, for evaluating collaborations between researchers and service providers, and for developing strategies to boost the acceptability of health interventions (Blevins et al., 2010; James et al., 2013; Maar et al., 2015). Project ICI adheres to principles and practices of community-engaged research, encouraging researchers to involve community partners (e.g. providers) in each phase of the research cycle—from defining study aims to protocol development to dissemination of findings (Israel et al., 1998).

The basis for a research-engagement model

Using qualitative and quantitative data, our research team has conducted several studies to examine provider willingness to become involved in research. Our work shows that providers’ perceptions of researchers’ availability, the benefits of research, and agency preparedness are associated with their willingness to engage with researchers to pursue scientific research (Pinto, 2013). Our evaluation of mediated and nonlinear relationships between personal and agency factors and provider willingness to become involved in research has generated a predictive model of practitioner involvement: the Provider-Researcher Partnership Model (Pinto et al., 2014), showing that providers favor “balanced partnerships” in which researchers and providers conduct research tasks and procedures in ways that reflect their unique knowledge and skill sets. Our work also reveals that research conducted in partnership with community collaborative boards, involving agency representatives, providers, and individuals who receive services in these agencies, improves recruitment and retention in large studies. Based on these findings, and on the most rigorous review of the community-engaged research literature presented by Viswanathan et al. (2004), we developed and conducted Project ICI, including its design and methods (e.g. recruitment and retention procedures). Clear evidence indicates that all the strategies we describe below are effective in improving recruitment and retention. It also suggests that combining different tasks and procedures may increase the impact of each individual strategy.

Implementation Community Collaboration Board processes and dynamics

Project ICI’s methods were developed and conducted by the Implementation Community Collaborative Board (ICCB), composed of 8–10 university researchers, providers, managers, and consumers whose unique contributions included research, practice experience, delivery of behavioral interventions, and intervention facilitation. The ICCB was developed using six published steps meant to engage members in procedural and substantive research roles: engaging members, developing relationships, exchanging information, negotiating and making decisions, retaining members, and studying dynamic processes (Pinto et al., 2011, 2015). ICCB members are trained in community-engaged principles and practices, protection of human participants, cultural competence, survey development, recruitment scripts, and data collection and analysis. The ICCB generally meets monthly (sometimes every 2 weeks or every 2 months, based on the methodological needs of the project). Meetings are usually 2 hours long. Participants receive appropriate compensation for attending meetings and for work outside meetings (e.g. drafting surveys and qualitative protocols, and collecting and interpreting data).

During meetings, we use dialectic processes, mutual support, problem-solving, and procedural triangulation to facilitate the implementation of research methods and to respond to environmental challenges that may hinder the recruitment and retention of research participants. For example, in order to develop the Project ICI survey instrument, ICCB members worked individually to create questions in specific areas of inquiry that matched their HIV skills and knowledge. These questions were then examined by the group during scheduled meetings. Discussions allowed for the exchange of knowledge and experiences (dialectic processes) that resulted in survey questions that could be answered by providers with diverse levels of HIV knowledge and skills. ICCB members expressed their individual ideas while supporting other members’ ideas despite differences of opinion (social support), and combined all the ideas into one project. In working out differences of opinion (problem-solving) about the best language to use in the survey questions, the sequence of questions, and the length of the survey, ICCB members collectively created a survey that was not only valid but readily accepted by the providers who participated in the study.

The ICCB reframed Project ICI’s aims to respond to new policies and changing contexts. The original study aims and corresponding methods and procedures had focused exclusively on issues regarding dissemination of the effective behavioral interventions from the CDC. The launching of the high-impact approach, in 2011, gave us the opportunity also to study myriad HIV-prevention services encompassed by the new program, such as expanded HIV testing and links to primary care (National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention, 2015). Our modified aims reflected the mandate requiring providers to engage in the HIV Continuum of Care and in the Test and Treat Approach. This approach integrates myriad services (e.g. HIV-, STI-, and hepatitis-C testing; primary care; and mental health services) to maximize the public health impact and calls for complex longitudinal designs, large samples of providers retained over time, and mixed methods to capture providers’ implementation behaviors over time (Palinkas and Soydan, 2011).

The ICCB has been instrumental in developing strategies for conducting research in response to the consolidation of programming at the federal level and its local consequences, such as the loss of funding and resources, dismantling of HIV-prevention departments, and termination of scores of providers, to strengthen methods for future research efforts. Specifically, the ICCB broadened the scope of the study and developed new recruitment and retention strategies, as described below.

Recruiting and retaining service agencies and providers longitudinally

Strategies for obtaining responses to surveys have been described in the literature. One strategy commonly used is Dillman et al.’s (2014) Total Design Survey Method (TDSM), which offers a sequence of specific steps researchers need to take in order to secure participation in survey research. Grounded in social-exchange theory, the TDSM helps researchers identify ways to increase the number of and enhance the quality of survey responses. The TDSM has shown to yield 75%–80% return rates for mail and telephone surveys. It encourages researchers to systematize data-collection tasks and procedures, to convey to survey respondents the importance of the research, and to help respondents weigh the costs and benefits of participation. The method was developed for using both mail and telephone surveys, but its basic tenets have been updated over the years and have been helpful in implementing face-to-face, Internet, and mixed-mode surveys. Dillman et al. recommend that researchers follow steps such as sending cover letters describing the purpose of the research and why respondents’ opinions are needed, and follow-up letters to nonresponders formatted in a specified manner.

Designs calling for in-person survey and qualitative-data collection require techniques beyond those provided by Dillman et al. As other researchers have done in the past, we followed the fundamental ideas contained in the TDSM to guide the specific tasks and procedures for Project ICI, but added several other strategies that improved recruitment and retention in Project ICI, which may do the same for studies with more complex designs (Hoddinott and Bass, 1986). The tasks and procedures developed for Project ICI are innovations that went far beyond those recommended for cross-sectional, in-person, mail, and Internet surveys. These included developing a low-burden study; social gatherings to help engage stakeholders; protocols to identify a point person at each service agency who would help recruit service providers; comprehensive record-keeping; research procedures that served as incentives to participation; a plan to retain hard-to-reach participants; and strategies for modifying incentives to participation over time. These are described below.

Social gatherings

In order to get buy-in from service agencies, we went through a pre-study phase in which we held recruitment brunches—community-focused events to engage agencies. Our goals for these events were to make the research team available for one-on-one conversations with agency representatives and providers, to communicate to them the opportunities offered by the study, to understand their concerns about the time commitments for their participation, and to create a safe space for discussing research aims and methods. ICCB members were instrumental in this phase of the project; they came to the social gatherings and spoke about Project ICI from their own experience. Managers and decision-makers heard well-informed presentations about Project ICI from providers who had been directly involved in the development of the grant that funded the project. Stakeholders from many agencies revealed that they made the decision to participate in the study because of the providers’ endorsement and direct knowledge of the project. Momentum from the brunches carried us into our first phase of data collection. We began at agencies where we had the closest partnerships because of past collaborations or the presence of ICCB members on the agency staff. We held an additional event 2 months later with more agencies. Eight of these agencies joined the study several months after we had held the first recruitment brunch, demonstrating what may be the long-term influence of community-based gatherings.

Barriers to recruitment and how to address them

In our attempts to recruit service agencies, we experienced pushback. Some agencies did not wish their providers to make even the minimal time commitments; others felt their internal systems were too overwhelmed for them to add promotion and explanation of the study to their internal conversations.

Developing low-burden studies

In order to facilitate recruitment of providers, we emphasized to agency representatives that participation in Project ICI would involve a low burden. Members of ICCB provided incentives and traveled to each agency for data collection so that providers did not need to travel. We scheduled our data collection at times that were convenient to both agencies and providers; we surveyed up to four participants at a time, with a time commitment of only about 1 hour per person.

Identifying a point person

Each participating agency designated a “point person” to coordinate interview schedules and obtain the approvals necessary to confirm participation. Point people were often administrators with multiple duties, and we struggled because these individuals sometimes neglected to return emails and telephone calls. In response to this challenge, we used e-blasts and personalized phone calls from the principal investigator or the Project ICI director. We also faced challenges in recruiting and scheduling individual providers. Providers were often called away to see clients just when they were scheduled to take our survey. Client needs and program crises extended beyond the providers’ available time. Often, if providers were called away to address a program issue, they expressed a commitment to the interview and made attempts to resume or reschedule the interview for the same day, but this process frequently ended in rescheduling. Sometimes, providers were ill or occupied with family or other personal issues.

Developing comprehensive record-keeping

We developed and established a system of record-keeping, including a Google calendar for appointments and a detailed and standardized checklist for data collectors. All research staff were trained in how to implement and document each item on the checklist, including the informed-consent script; signed consent forms from each participant; agency and personal contact information for all participants; notes on data collection for each agency and for providers; and signed receipts for incentives. We found at every stage that good record-keeping improved our ability to have a fluid workflow and retention of participants. This also allowed for easy replacement of support staff—an important consideration because most longitudinal projects face research-staff turnover.

Giving back to the community as a facilitator of retention

Community-engaged research calls for opportunities to give back to those who make time in their lives to be involved in research and who willingly give myriad types of data that are used by researchers to advance science and inform both practice and policy (e.g. in the areas of social work, medicine, and public health). Here, we discuss three strategies from our community-engaged model, grounded in the literature discussed above, which have helped us with recruitment and retention in several studies, including Project ICI.

Appealing to altruism and stressing Project ICI benefits

In community-engaged research, the study goal is rooted in community members’ interests and intentions. Our providers, including agency administrators, work in HIV-prevention services; this is a service community with a long history of loyalty generated from personal connection with the communities served and activism to improve access to care. During all phases of the study, staff were trained and supervised to remind participants of the potential benefits study participation and success might offer to the local community. Providers shared feelings of strong commitment to their clients and the larger community. A great many providers expressed interest in the dissemination process—the use of findings as they related to the betterment of HIV-prevention services. Personal benefits experienced by participants were amplified and these supported the motivation to be “part of the solution” by increasing interprofessional collaboration across agencies.

Offering training as incentives for research participation

After the baseline was established, providers were offered ICI training (Table 1), a full day of programming delivered in line with CDC’s vision of improved collaboration to enhance HIV/AIDS prevention (CDC, 2009). ICI training was developed by the ICCB; it involved conceptualizing goals, specific modules used to teach participants the benefits of evidence-based services, subjective norms and barriers to interprofessional collaboration, self-efficacy techniques, and specific steps participants could use to engage in collaboration. All members of the ICCB participated in the pilot training, and several taught different modules of the training when it was offered in the full-day workshop. ICI training was provided between the baseline and the 12-month follow-up, and it served as an opportunity for us to solidify our partnerships with participating agencies and providers. Interactions prior to and during the training helped participants develop a sense of belonging to a cohort and thus being committed to one another’s retention for the 24-month follow-up. Some 142 study participants from 34 agencies attended the trainings. Providers’ schedules, workloads, and personal and family issues were identified as obstacles to attendance. Agencies found it challenging to afford staff a day for training due to work responsibilities and program needs (e.g. limited staff, client crises, and mandatory program events). There were instances in which providers scheduled to attend the training had to miss it because of personal obligations. Following the advice of the ICCB, the research team responded by scripting and shooting a 37-minute video version of the training that could be shown to our participants at their agencies and at their convenience. Several ICCB members participated in writing and performing in the video. The video followed the same arc as the live training, and it discussed all the same elements of collaboration. Although interactive group activities could not be included in the video, stopping places were built into the video (whose total training time was 60 minutes), so that those watching could stop to discuss critical points and engage in question-and-answer sessions. An additional 84 providers were trained using the video approach. The ICI training, in both forms, served as an engagement tool, providing a needed service for the community of agencies and providers in the study.

Sharing research findings with the community: Project ICI dissemination symposium

Community-engaged research calls for the dissemination of findings through different media; specifically, it calls for researchers to make certain research findings available to service providers, managers, stakeholders, and policy makers so that the findings can be used to guide practice and policy. Research findings are commonly shared through publications in journals, but such results often do not extend past the academic community. Community providers are often engaged in different stages of the research cycle, but are seldom reengaged after the research has been completed. Reengagement after data collection, training, and other research participation is needed to help researchers contextualize findings and empower members of the community to use those findings in their lives—in this case, to empower providers, managers, and policy makers to use findings to guide practice and policy.

In order to decrease the gap between research and practice, Project ICI, at the end of its final year, hosted a dissemination symposium to bring together community members, providers, administrators, and researchers (Ahead Magazine: Social work research at University of Michigan, 2017).

All Project ICI participants were invited to attend this event. An important incentive for providers who participated was continuing education units (CEUs) for training. Social workers who attended were eligible for 6 hours of CEUs, so the incentive was equivalent to an opportunity for professional development. Symposium participants represented a cross-section of HIV-prevention and treatment professionals in New York City. The event offered an opportunity for researchers to tackle difficult questions with providers, explore their experiences, and discern areas of impact for future research; participants had their opinions and experiences validated by peers.

The 6-hour day included social activities such as breakfast, lunch, and afternoon coffee. These activities echoed how Project ICI started with social gatherings to offer opportunities for providers to network and compare and contrast their experiences with the research project. Keynote addresses—by Dr Chris Gordon, chief of the HIV Treatment and Translational Science Branch at the Division of AIDS Research at the NIMH, and Dr Julie Myers, the director for HIV prevention at the New York City Department of Health and Mental Hygiene—focused on the importance of community engagement in research on combating HIV. Most important, nearly 60 participants heard from and engaged with a panel of administrators, providers, and researchers who discussed and analyzed the Project ICI findings. After the panel, ICCB members led discussions with small groups of participants about the issues revealed by the findings. This afforded us an opportunity to ask providers to help researchers interpret the findings; this represents a large-scale example of “member-checking,” a technique used in qualitative research that involves including research participants in the analysis and interpretation of findings (Padgett et al., 2012). An issue of Ahead, the research magazine published by the University of Michigan School of Social Work, highlighted the dissemination symposium and was distributed to all Project ICI participants and to administration at participating agencies.

Barriers to cohort retention and how to address them

When we began to collect the 12-month follow-up surveys, we realized that provider attrition rates were high due to the environmental challenges discussed above. In some cases, the point person himself or herself had left the agency. The project staff was typically notified of such situations during contact attempts. This information was relayed in various ways: by a new employee who had filled the position, by another participating provider, or even by a receptionist who would then refer us to an alternate representative within the agency. Not until we started reaching out to this “hard-to-reach” group did we come to understand the myriad challenges we would have in getting in contact with each provider.

Developing a plan to retain hard-to-reach participants

We found that the information we had for many providers—home addresses, personal email addresses, and personal phone numbers (usually for mobile phones)—was no longer valid. We took note of those who did not respond to our outreach efforts—those whose postal mailings were returned to us, or whose email accounts, personal or professional, bounced our messages back. Returned postal mail, bounced emails, and a lack of responses to phone calls were the only indications that certain individuals’ contact information had changed. In retrospect, we would like to have had some indication about how many participants might become hard to reach. We did not completely understand why there was such an attrition rate. We knew the roles policy and priority changes had played in recruitment and retention, but could attrition also be due to job dissatisfaction, or to economic or demographic trends at participants’ workplaces? Were people taking positions at new agencies that would offer better benefits or working conditions? As we began to interview hard-to-reach participants, we learned that all these issues were present. Agencies experiencing downsizing tended not to have the capacity to absorb providers into other roles within the agencies. Providers who had left their agencies of their own accord typically expressed a desire for better supervision or felt undervalued and deserving of more compensation for the quality of work expected in the role.

We pursued hard-to-reach individuals by mailing a letter with an ICI Project fact sheet to all for whom we had mailing addresses. We then followed up by calling all those from whom we had not heard. We called each individual twice within the space of 2–3 days, most often leaving messages that reacquainted participants with Project ICI, informed them of our need to follow-up, and offered to come to their new workplaces to survey them. If we received no response from a participant, we made a third attempt, usually via text message. Provider recall of and reengagement in the study were reinforced by the ability of project staffers to provide details from the previous interview process. The project director supported this process through supervision and information-sharing with project staff. Details could include the location and time of year of the interview, idiosyncrasies involving the scheduling of interviews or shared life events, and reminders of names of the project staff members who had been involved in the scheduling process.

Working with Institutional Review Boards to modify incentives

We based our decisions about how much to use incentives to recruit and retain participants based on the well-known concept of undue influence, which occurs when compensation or incentives are used to induce potential research participants who otherwise would not participate in a study. Grounding its thoughts in community-engaged research best practices, the ICCB discussed this issue several times and consulted with the Institutional Review Board at our institution. Our main concern was to avoid inducing providers, particularly those with lower salaries, to make decisions that were not in their best interests (Williams and Walter, 2015). We resolved the retention issues by requesting approval from our Institutional Review Board to double the incentive we were offering each point person for facilitating recruitment for the 24-month follow-up. Furthermore, we also nearly doubled the incentive for providers to participate in the 12- and 24-month follow-up interviews. Although the cost of conducting the study grew higher, we know that the datasets we have developed, because of their completeness, have allowed us to use innovative sampling strategies for a major component of the mixed-method design (in-depth interviews) and for quantitative data analyses, both described below.

Lessons learned

With an eye toward narrowing identified gaps in the literature, we sought to demonstrate how researchers can respond to environmental changes (such as federal and local policy changes) that hinder participation in research, particularly in longitudinal studies characterized by large samples of service agencies and providers. We provided a model of research engagement shown to yield rates of recruitment and retention higher than those for previously described strategies for cross-sectional studies. Project ICI is a complex study whose design required engagement of a large number of service agencies in order to yield a dataset containing survey data for 250 individual participants at the baseline and 12- and 24-month follow-ups, and another dataset containing in-depth interview data for 20 providers. Moreover, the study provided a full-day (N = 142) or video (N = 84) training session on interprofessional collaboration to a total of 226 providers.

We learned that our research-engagement model yielded high rates of retention in its collection of survey data at the baseline and the 12- and 24-month follow-ups; of qualitative data between the baseline and the 12-month follow-up and between the 12- and 24-month follow-ups; and of interprofessional collaboration training between the baseline and the 12-month follow-up. By developing a comprehensive series of tasks and procedures, we were able to recruit 36 service agencies in New York City and a total of 379 providers at the baseline. We retained all 36 agencies over the 5-year study, even though two pairs of agencies merged between the baseline and the 12-month follow-up, resulting in a final sample of 34 agencies. Using the community-engaged model, between the baseline (N = 379) and the 12-month follow-up (N = 285) we retained 75% of the sample, and between the 12-month follow-up (N = 285) and the 24-month follow-up (N = 256) we retained 90% of the sample. For the qualitative interviews, we retained 100% of the sample. This degree of retention appears to surpass return rates for studies with simpler designs described in the literature (Dillman et al., 2014; Hoddinott and Bass, 1986).

By using an innovative approach—a systematic combination of specific tasks and procedures over time—we achieved excellent recruitment and retention rates. Because we retained a large number of agencies and providers over time, we have datasets with crucial data that have allowed us to engage in innovative sampling and data analyses that would not be possible without complete datasets. For example, in selecting providers for in-depth interviews, we used criterion sampling to focus on a predefined group of participants (Patton, 2002; Ragin and Becker, 1992). We selected 20 providers out of 250 participants who provided survey data at the baseline and the 12- and 24-month follow-ups. We based our selection on two variables from the survey: the frequency with which the providers made links to HIV services and interprofessional collaboration experience. This allowed us to select providers with different patterns of link-making (“low” and “high” link-makers) and collaboration (“low” and “high” collaborators). We randomly assigned five providers to each of the four groups (N = 20), and interviewed them between the baseline and the 12-month follow-up, and then between the 12- and 24-month follow-ups. Another innovation achieved only because of the completeness of the dataset was the combination of multilevel survey data (from providers nested within agencies) and longitudinal survey data, which allowed us to conduct hierarchical or “multilevel” SEM, which allows temporal regression-type relationships to be examined simultaneously. Multilevel SEM allows provider- and agency-level predictors and outcomes to be treated (when appropriate) as latent variables, and allows for joint modeling of longitudinal trajectories.

The speed and innovation with which we responded to environmental and historical changes in policy and priorities (without compromising rigor) would not have been achievable without the expertise and commitment of our community partners. US national directives and funding opportunities continue to urge researchers to engage community partners, yet few studies fully describe community-engaged principles and practices (Institute of Medicine, 2000; National Institutes of Health, 2014; Proctor et al., 2009; United States Department of Health and Human Services, 2006). This article helps fill this gap by providing a detailed account of how our model for community engagement helped us improve Project ICI’s aims and procedures. Our model offered us the immediate capacity to develop sustainable solutions to changes in policy and priorities that might have hindered our study design and methods. Therefore, it has potential application for other studies.

Implementation-system policies are fluid, and science must keep pace, yet we seldom publish our efforts or relate how we address research challenges. Project ICI was developed with a collaborative board whose members have firsthand experience with the practices—and the policy shifts that alter practices—at community-based agencies. The involvement of community experts has helped us and will continue to help us respond to environmental shifts and improve recruitment and retention. Box 1 shows a summary of the time that was required for data collection, the frequency of contacts required to initiate data collection, and the time required for training 226 providers. These data are not sufficient to conduct a cost–benefit analysis; however, our team will keep collecting this type of data in future studies so as to be able to describe and weigh the costs and benefits of using a research-engagement model. Since we work under the paradigm of community-engaged research, we do not possess any data to show differences between our model and other models of engagement, recruitment, and retention. Nonetheless, our efforts to recruit and retain agencies and providers adds evidence to a large body of research showing that community engagement can improve overall recruitment and retention (Viswanathan et al., 2004). Our work further validates how community-engaged research puts researchers at an advantage when responding to environmental changes that otherwise might hinder research design, methods, and procedures. The strategies provided here have implications for research using different designs and methods in myriad other contexts.

Box 1.

Recruitment and retention of 36 agencies and 250 participants.

Time required for data collection: It took approximately 23 months to collect baseline data from 379 participants, and 22 and 16 months for the 12- and 24-month follow-ups, respectively. We have longitudinal data (baseline and 12- and 24-month follow-ups) from a sample of 256 participants.

Contacts required for data collection: The number of direct contacts (US mail, emails, and telephone calls) with agency representatives, who sometimes became our point people, ranged from one to five. The completion of each survey interview required one to five contacts between staff and participants. Each interview, from initial contact to survey completion, required 1–5 hours of research staff time.

Time required for training: We provided full-day training (6 hours) in interprofessional collaboration to 142 participants and video training (60 minutes) to 84. Completion of the training for each participant required an average of 1.5–2 hours of direct contact for recruiting and scheduling.

Project ICI has public health implications because it promotes the implementation of evidence-based HIV-prevention services. Knowledge of how interprofessional collaboration influences service implementation informs the diffusion of various services across multiple HIV-prevention delivery systems. This project’s design will enhance our capacity to interpret longitudinal patterns, especially those patterns influenced by such factors as scarcity of resources and provider turnover faced by participating agencies during the study. An advantage of conducting the longitudinal study in New York City is the diversity of providers and agencies that will facilitate the ability to generalize our findings.

These lessons learned may also be applicable in low- and middle-income countries. Our model of community-engaged research has been used to guide health research in Brazil, Mongolia, and Kazakhstan and has been published elsewhere (Pinto et al., 2014). We believe that the strategies we used here can be performed with moderate levels of effort and human resources (see Box 1). This is particularly true in the initial phases of partnership-building and formative low-budget research, when partners may have only seed funding. Creating opportunities for research partners to participate in all phases of the research cycle is recommended; this can help build research capacity usually absent in low- and middle-income countries. Research partners can be involved in data collection—particularly through qualitative interviews, which can generate findings more accessible to lay individuals than are quantitative ones.

Conclusion

We wish to encourage researchers and their community partners to watch for policy and priority shifts that might present an opportunity to expand the scope of their research aims to respond to new trends. Researchers will need to maintain partnerships with policy makers, who are the primary sources of information about new practices and the policies guiding them. We hope that researchers, guided by community-engaged values and practices, will disseminate any and all methodological innovations that might enhance recruitment and retention of research participants. Our work suggests that engaging and retaining community partners over time may help build structural capacity—for example, through ongoing collaborative research boards. Community experts on these boards who are involved in the conceptualization of research aims will, from one research project to the next, improve their capacity to formulate solutions to the problem of attrition in research. Capacity can also be built by establishing a thorough and detailed system of record-keeping, by training staff and community partners in how to use that system, and by evaluating the system and making changes to address environmental challenges.

Footnotes

Acknowledgements

We wish to thank all members of the Implementation Community Collaborative Board who generously provided inspiration and ideas for this paper and for the grant proposal that funded this study: Angela Ghesquiere; Charles Sanky; Jeannette Ruffins; Rosa Bramble; Karen Baird; and Wendy Whitman.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research presented here, the preparation for this manuscript, and publication of this article were supported by NIH grant R01MH095676 (Principal Investigator: R. M. Pinto).

ORCID iD

Rogério M Pinto

Author biographies

Rogério M Pinto is associate dean for research at the University of Michigan, School of Social Work. He is a Brazilian-born psychiatric social work researcher with more than a decade of clinical and community practice. He is an expert in Community Based Participatory Research and mixed-method research. In the United States, Brazil and Spain. Dr. Pinto examines how interprofessional collaboration and practitioners’ involvement in research improves delivery of evidence-based services. He is a scientific reviewer for NIH and Brazil’s Ministry of Health and Associate Editor for the Journal of Mixed Method Research.

Susan S Witte is associate professor at Columbia University School of Social Work. Susan’s research interests include design, testing and dissemination of HIV/STI prevention interventions for women and their male partners; multimedia technologies in research, design, testing and dissemination of services targeting sex workers; female-initiated STD/HIV barrier methods; and intervention research.

Melanie M Wall is a professor at Columbia University, School of Public Health. She is a biostatistician with a record of innovative data analysis methods research in latent variable modeling and spatial and longitudinal data analysis. She has made contributions in latent variable modeling, including methods allowing for nonlinear relationships to be examined among latent variables, and an innovative line of research incorporating latent variables into spatial data analyses.

Prema L Filippone is a doctoral student at Columbia School of Social Work. She was the project director for Project ICI. She has supervised several federal National Institutes of Mental Health grants. As a licensed master social worker, she served as program director for Girls Educational and Mentoring Services, a leading service provider for victims of sexual exploitation of children and domestic sex trafficking. She holds Dual Degree Bachelor of Arts degrees in Psychology and Sociology from Rutgers University and a Master of Science in Social Work from Columbia University.

References

Ajzen

(1991) The theory of planned behavior. Organizational Behavior and Human Decision Processes 50(2): 179–211.

Ajzen

Fishbein

(1980) Understanding Attitudes and Predicting Social Behaviour. Englewood Cliffs, NJ: Prentice-Hall.

Asendorpf

Van De Schoot

Denissen

JJA

et al . (2014) Reducing bias due to systematic attrition in longitudinal studies: The benefits of multiple imputation. International Journal of Behavioral Development 38(5): 453–460.

Belone

Lucero

Duran

et al . (2016) Community-based participatory research conceptual model: Community partner consultation and face validity. Qualitative Health Research 26(1): 117–135.

Biering

Hjollund

Frydenberg

(2015) Using multiple imputation to deal with missing data and attrition in longitudinal studies with repeated measures of patient-reported outcomes. Clinical Epidemiology 7: 91–106.

Blevins

Farmer

Edlund

et al . (2010) Collaborative research between clinicians and researchers: A multiple case study of implementation. Implementation Science 5: 76.

Cashman

Adeky

Allen

et al . (2008) The power and the promise: Working with communities to analyze data, interpret findings, and get to outcomes. American Journal of Public Health 98(8): 1407–1417.

Centers for Disease Control and Prevention (CDC) (2009) Program Collaboration and Service Integration: Enhancing the Prevention and Control of HIV/AIDS, Viral Hepatitis, Sexually Transmitted Diseases, and Tuberculosis in the United States. Atlanta, GA: Centers for Disease Control and Prevention.

Centers for Disease Control and Prevention (CDC) (2010) Welcome to effective interventions. Available at: http://www.effectiveinterventions.org/

10.

Centers for Disease Control and Prevention (CDC) (2011) High-impact HIV Prevention: CDC’s Approach to Reducing HIV Infections in the United States. Atlanta, GA: National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention and Division of HIV/AIDS Prevention. Available at: http://www.cdc.gov/hiv/pdf/policies_NHPC_Booklet.pdf

11.

Centers for Disease Control and Prevention (CDC) (2013) Compendium of evidence-based behavioral interventions. Available at: http://www.cdc.gov/hiv/prevention/research/compendium/rr/index.html

12.

Centers for Disease Control and Prevention (CDC) (2015) High-impact HIV Prevention (HIP): Overview of Select Interventions & Strategies. Washington, DC: Danya International. Available at: https://effectiveinterventions.cdc.gov/docs/default-source/general-docs/15-1106-hip-overview-factsheet.pdf?sfvrsn=2

13.

Chagnon

Pouliot

Malo

et al . (2010) Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implementation Science 5: 41.

14.

Clark

VLP

Anderson

Wertz

et al . (2015) Conceptualizing longitudinal mixed methods designs: A methodological review of health sciences research. Journal of Mixed Methods Research 9(4): 297–319.

15.

Collins

Harshbarger

Sawyer

et al . (2006) The diffusion of effective behavioral interventions project: Development, implementation, and lessons learned. AIDS Education and Prevention: Official Publication of the International Society for AIDS Education 18(4 Suppl. A): 5–20.

16.

Dillman

Smyth

Christian

(2014) Internet, Phone, Mail, and Mixed-mode Surveys: The Tailored Design Method. Hoboken, NJ: John Wiley & Sons.

17.

Durlak

DuPre

(2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology 41(3–4): 327–350.

18.

Faridi

Grunbaum

Gray

et al . (2007) Community-based participatory research: Necessary next steps. Preventing Chronic Disease 4(3): A70.

19.

Galbraith

Bowden

Mander

(2017) Accelerated longitudinal designs: An overview of modeling, power, costs, and handling missing data. Statistical Methods in Medical Research 26(1): 374–398.

20.

Gotham

(2004) Diffusion of mental health and substance abuse treatments: Development, dissemination, and implementation. Clinical Psychology: Science and Practice 11(2): 160–176.

21.

Hoddinott

Bass

(1986) The Dillman total design survey method. Canadian Family Physician 32: 2366–2368.

22.

Ibrahim

Molenberghs

(2009) Missing data methods in longitudinal studies: A review. Test 18(1): 1–43.

23.

Institute of Medicine (2000) Crossing the Quality Chasm: A New Health System for the Twenty-first Century. Washington, DC: National Academy of Science.

24.

Israel

Schulz

Parker

et al . (1998) Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health 19(1): 173–202.

25.

James

Richardson

Wang

et al . (2013) Systems intervention to promote colon cancer screening in safety-net settings: Protocol for a community-based participatory randomized controlled trial. Implementation Science 8: 58.

26.

Kelly

Heckman

Stevenson

et al . (2000a) Transfer of research-based HIV prevention interventions to community service providers: Fidelity and adaptation. AIDS Education and Prevention: Official Publication of the International Society for AIDS Education 12(Suppl. 5): 87–98.

27.

Kelly

Somlai

DiFranceisco

et al . (2000b) Bridging the gap between the science and service of HIV prevention: Transferring effective research-based HIV-prevention interventions to community AIDS service providers. American Journal of Public Health 90(7): 1082–1088.

28.

Lehman

WEK

Greener

Simpson

(2002) Assessing organizational readiness for change. Journal of Substance Abuse Treatment 22(4): 197–209.

29.

Little

Rubin

(1987) Statistical Analysis of Missing Data. New York: Wiley.

30.

Maar

Yeates

Barron

et al . (2015) I-RREACH: An engagement and assessment tool for improving implementation readiness of researchers, organizations, and communities in complex interventions. Implementation Science 10: 64.

31.

Mermin

Fenton

(2012) The future of HIV prevention in the United States. JAMA 308(4): 347–348.

32.

National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention (2015) CDC Awards $216 Million to Community-Based Organizations to Deliver the Most-Effective HIV-Prevention Strategies to those in Greatest Need (Community-based Organization Awards for HIV Prevention). Washington, DC: Department of Health and Human Services, Centers for Disease Control and Prevention.

33.

National Institutes of Health (2006) Dissemination and Implementation Research in Health (PAR-06-039). Bethesda, MD: National Institutes of Health.

34.

National Institutes of Health (2014) Community Partnerships to Advance Research (CPAR R01). Bethesda, MD: National Institutes of Health.

35.

Norton

Amico

Cornman

et al . (2009) An agenda for advancing the science of implementation of evidence-based HIV-prevention interventions. AIDS and Behavior 13(3): 424–429.

36.

Owczarzak

Dickson-Gomez

(2011) Providers’ perceptions of and receptivity toward evidence-based HIV prevention interventions. AIDS Education and Prevention: Official Publication of the International Society for AIDS Education 23(2): 105–117.

37.

Padgett

Smith

Henwood

et al . (2012) Life-course adversity in the lives of formerly homeless persons with serious mental illness: Context and meaning. American Journal of Orthopsychiatry 82(3): 421–430.

38.

Palinkas

Soydan

(2011) Translation and Implementation of Evidence-Based Practice. New York: Oxford University Press.

39.

Patton

(2002) Qualitative Research and Evaluation Methods (3rd edn). London: SAGE.

40.

Pinto

(2013) What makes or breaks provider–researcher collaborations in HIV research? A mixed method analysis of providers’ willingness to partner. Health Education & Behavior 40(2): 223–230.

41.

Pinto

Spector

Valera

(2011) Exploring group dynamics for integrating scientific and experiential knowledge in community advisory boards for HIV research. AIDS Care 23(8): 1006–1013.

42.

Pinto

Spector

Rahman

et al . (2015) Research advisory board members’ contributions and expectations in the USA. Health Promotion International 30(2): 328–338.

43.

Pinto

Spector

Witte

et al . (2014) Systematizing planning and formative phases of HIV-prevention research: Case studies from Brazil, Mongolia, and Kazakhstan. Global Social Welfare 1(3): 137–144.

44.

Pinto

Spector

et al . (2013) Transdisciplinary collaboration and endorsement of pharmacological and psychosocial evidence-based practices by medical and psychosocial substance-abuse treatment providers in the United States. Drugs (Abingdon, England) 20(5): 408–416.

45.

Pinto

Wall

Spector

(2014) Modeling the structure of partnership between researchers and front-line service providers: Strengthening collaborative public health research. Journal of Mixed Methods Research 8: 83–106.

46.

Ployhart

Vandenberg

(2010) Longitudinal research: The theory, design, and analysis of change. Journal of Management 36(1): 94–120.

47.

Proctor

Landsverk

Aarons

et al . (2009) Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research 36(1): 24–34.

48.

Rabe-Hesketh

Skrondal

Pickles

(2004) Generalized multilevel structural equation modeling. Psychometrika 69(2): 167–190.

49.

Ragin

Becker

(1992) What Is a Case? Exploring the Foundations of Social Inquiry. Cambridge: Cambridge University Press.

50.

Rogers

(1995) Diffusion of Innovations (4th edn). New York: Simon & Schuster.

51.

Romanelli

Tushman

(1986) Inertia, environments, and strategic choice: A quasi-experimental design for comparative-longitudinal research. Management Science 32(5): 608–621.

52.

Rotheram-Borus

Swendeman

Chorpita

(2012) Disruptive innovations for designing and diffusing evidence-based interventions. American Psychologist 67(6): 463–476.

53.

Spector

Pinto

(2017) Partnership matters in health services research: A mixed-methods study of practitioners’ involvement in research and subsequent use of evidence-based interventions. Journal of Mixed Methods Research 11(3): 374–393.

54.

Swanson

Ward

(1995) Recruiting minorities into clinical trials toward a participant-friendly system. JNCI: Journal of the National Cancer Institute 87(23): 1747–1759.

55.

Tinkle

Kimball

Haozous

et al . (2013) Dissemination and implementation research funded by the U.S. Nursing Research and Practice 2013: 909606.

56.

United States Department of Health and Human Services (2006) The Road Ahead: Research Partnerships to Transform Services. Bethesda, MD: United States Department of Health and Human Services.

57.

University of Michigan (2017) Ahead Magazine: Social work research at the University of Michigan (Project ICI, Issue 1). Available at: https://issuu.com/uofmsocialwork/docs/ahead-magazine-issue-one/1 (18 January 2018).

58.

Van Weel

Van Weel-Baumgarten

Mold

(2006) The importance of longitudinal studies in family medicine: Experiences of two practice-based research networks. Journal of the American Board of Family Medicine 19(1): 69–74.

59.

Viswanathan

Ammerman

Eng

et al . (2004) Community-based participatory research: Assessing the evidence, summary. Agency for Healthcare Research and Quality (US). Available at: https://www.ncbi.nlm.nih.gov/books/NBK11852/

60.

Williams

Walter

(2015) When does the amount we pay research participants become “undue influence”? AMA Journal of Ethics 17(12): 1116–1121.

61.

Witte

Aira

Tsai

et al . (2015) Efficacy of a savings-led microfinance intervention to reduce sexual risk for HIV among women engaged in sex work: A randomized clinical trial. American Journal of Public Health 105(3): e95–e102.

62.

Witte

El-Bassel

Gilbert

et al . (2004) Recruitment of minority women and their main sexual partners in an HIV/STI-prevention trial. Journal of Womens Health 13(10): 1137–1147.