Abstract
Randomized controlled trials (RCTs) are a long-standing and important design for conducting rigorous tests of the effectiveness of health interventions. However, many questions have been raised about the external validity of RCTs, their utility in explicating mechanisms of intervention and participants’ intervention experiences, and their feasibility and acceptability. In the current mixed-methods study, academic and community partners developed and implemented an RCT to test the effectiveness of a collaboratively developed community-based advocacy, learning, and social support intervention. The goals of the intervention were to address social determinants of health and build trust and connections with other mental health services in order to reduce mental health disparities among Afghan, Great Lakes Region African, and Iraqi refugee adults and to engage and retain refugees in trauma-focused treatment, if needed. Two cohorts completed the intervention between 2013 and 2015. Ninety-three adult refugees were randomly assigned to intervention or control group and completed four research interviews (pre-, mid-, and postintervention, and follow-up). Several challenges to conducting a community-based RCT emerged, including issues related to interviewer intervention to assist participants in the control group, diffusion of intervention resources throughout the small refugee communities, and staff and community concerns about the RCT design and what evidence is meaningful to demonstrate intervention effectiveness. These findings highlight important epistemological, methodological, and ethical challenges that should be considered when conducting community-based RCTs and interpreting results from them. In addition, several innovations were developed to address these challenges, which may be useful for other community–academic partnerships engaged in RCTs.
Keywords
Randomized Controlled Trials
In continued attempts to develop effective interventions to improve and promote health and well-being and eliminate health inequities, randomized controlled trials (RCTs) are an important design for conducting rigorous tests of interventions with high internal validity. Despite significant questions and concerns, RCTs remain a critical tool and the “gold standard” for measuring intervention effectiveness (Meldrum, 2000). There is growing recognition, however, that certain limitations should be addressed in any study that employs an RCT design, including examining external validity (Rothwell, 2005), incorporating methods for explicating mechanisms of change and understanding participants’ intervention experiences (Hawe, Shiell, & Riley, 2004), and carefully exploring the feasibility and acceptability of an experimental design (Rychetnik, Frommer, Hawe, & Shiell, 2002). For example, in terms of external validity, it is essential to understand contextual factors that may affect the success of an intervention for any particular population (Braveman, Egerter, & Williams, 2011; Hawe et al., 2004). Thus, contextual factors relevant to any test of intervention effectiveness should be measured and reported (Lifsey, Cash, Anthony, Mathis, & Silva, 2015).
As health interventions have moved from clinics to communities, issues of feasibility and acceptability of RCTs have been increasingly highlighted. For example, Lam, Hartwell, and Jekel (1994) examined the impact of randomization on research participants, relationships with community organizations, and research staff in an RCT of an intensive residential treatment for homeless men with substance abuse problems. They found that participants had different and mixed reactions to the reasons they did not receive the intervention. Unexpectedly, they found that service providers and community organizations had stronger negative feelings about the randomization process, which affected their willingness to refer participants to the study. Furthermore, research staff experienced difficulties in seeing participants not receive the intervention. One approach to addressing these issues has been to engage in collaborative partnerships with communities to plan and conduct intervention studies.
Community-Based Participatory Research and RCTs
Community-based participatory research (CBPR) involves genuine collaboration among researchers and community members to identify the goals of research, research questions, methods, interventions, data analyses and interpretation, and dissemination of results. CBPR approaches rely on mutual learning among community members and researchers and aim to recognize and build on the strengths of everyone involved. CBPR also has an explicit focus on using the research process to improve the lives of individuals and communities and to promote social justice (Minkler & Wallerstein, 2008). Thus, CBPR is particularly appropriate for research that addresses health inequities.
In some ways, CBPR and RCTs seem to have divergent values and goals. Trickett (2011) has highlighted some of the tensions inherent in combining RCTs and CBPR. In particular, he emphasizes that CBPR is often inappropriately viewed as an instrumental strategy employed “to accomplish predetermined aims or goals not collaboratively developed or locally defined” (p. 1353). He suggests that RCT and CBPR paradigms are typically incompatible because of the assumption within an RCT design that context should be controlled or “ruled out” to allow for a rigorous test of intervention effectiveness. As Trickett (2011) explains, a CBPR worldview requires attention to context, systems change, sustainability, capacity building, and empowerment.
However, there are examples in which a CBPR approach has been successfully combined with an RCT design (Horn, McCracken, Dino, & Brayboy, 2008; Jones, Koegel, & Wells, 2008; Krieger, Takaro, Song, & Weaver, 2005, Krieger, Takaro, Song, Beaudet, & Edwards, 2009; Parker et al., 2008; Salvatore et al., 2009). Jones et al. (2008) highlight the importance of having sufficient time and resources for a long planning phase. They also note that there are several reasons communities might consider employing an RCT design, including a community’s interest in strong evaluation data that could support causal inference and expansion of the scope of designs that communities are comfortable employing. Innovative approaches to combining CBPR and an RCT design have also been proposed, such as the multisite translational community trial (Katz, Murimi, Gonzalez, Njike, & Green, 2011), which provides a detailed method for maintaining the key attributes of a multisite RCT while allowing for customization of community actions/interventions.
In sum, the appropriateness of using an RCT design within a CBPR framework remains contested and merits further exploration and testing to understand more about the possibilities of combining these while remaining true to the values and goals of CBPR. It is clear, however, that attempts to integrate an RCT design and CBPR approach should recognize and address inherent tensions and challenges.
Method
Refugee Well-Being Project
Among the populations that bear the burden of social inequities and health disparities are the increasing numbers of refugees worldwide, who typically have higher rates of psychological distress, limited material resources, lingering physical ailments, and loss of meaningful social roles and support, which can be compounded by poverty, racism, discrimination, and devaluation of cultural practices (Edberg, Cleary, & Vyas, 2011). In the current study, academic and community partners developed and implemented an RCT to test the effectiveness of a community-based advocacy, learning, and social support intervention that addresses social determinants of mental health and builds trust and connections with other mental health services to reduce mental health disparities among low-income Afghan, African, and Iraqi refugee adults in the United States, and engages and retains refugees in trauma-focused treatment, if needed.
The Refugee Well-being Project (RWP) intervention emphasizes a sustainable and replicable partnership model between refugees, community organizations that work with refugees, and universities that involves refugee adults and undergraduate advocates working together to (1) increase refugees’ abilities to navigate their new communities; (2) improve refugees’ access to community resources; (3) enhance meaningful social roles by valuing refugees’ cultures, experiences, and knowledge; (4) reduce refugees’ social isolation; and (4) increase communities’ responsiveness to refugees. The RWP intervention is delivered by university undergraduate students enrolled in a 2-semester service learning course, 1 and has two elements: (1) Learning Circles, which involve cultural exchange and one-on-one learning opportunities, and (2) Advocacy, which involves collaborative efforts to mobilize community resources related to health, housing, employment, education, and legal issues.
Of note, the intervention was initially developed by academic and community partners over 4 years in Michigan (Goodkind, Hang, & Yang, 2004) and adapted and implemented 6 additional years by academic and community partners in New Mexico (Goodkind, Githinji, & Isakson, 2011) before collaboratively developing an RCT design and acquiring funding from the National Institutes of Health in 2013. The initial pilot testing of the RWP demonstrated feasibility, appropriateness, acceptability, and preliminary evidence that the intervention decreased Hmong, African, and Iraqi participants’ psychological distress and increased protective factors (Goodkind, 2005, 2006; Goodkind et al., 2014). Findings also indicated that undergraduate students engaged in mutual learning with their refugee partners (Goodkind, 2006) and that the RWP fostered transformative learning experiences through which refugees and students came to new understandings of the relationship between social inequities and well-being. For many, these new understandings also provided an impetus to work toward social change at multiple levels (Hess et al., 2014).
In the current study, a mixed-methods strategy with data collected from each participant at four time points over a period of 14 months is being used to test the effectiveness of the 6-month intervention to reduce psychological distress, increase protective factors, and engage and retain refugee adults with posttraumatic stress disorder (PTSD) in an evidence-based trauma treatment (Narrative Exposure Therapy [NET]). Mechanisms of intervention effectiveness will be explored by testing mediating relationships between protective factors and psychological distress. Qualitative interviews are being used to explore participants’ experiences in the intervention, inform interpretation of quantitative data, and investigate unexpected impacts. Additional qualitative data collected through participant observation at Learning Circles, community advisory council (CAC) meetings, research team meetings, and other community meetings will be analyzed in order to try to explicate the context in which the intervention is being implemented and examine multilevel changes in families and the community.
Community Collaboration
From its inception, the RWP and the accompanying research on its impacts have involved collaboration between refugee community members and academic partners. The intervention was initially designed by the first author and a CAC of Hmong refugee women in Michigan and was implemented and evaluated by the first author and the CAC as the first author’s dissertation. Except for the first author, all study staff in Michigan were Hmong refugees. In New Mexico, the study has been guided by a CAC of refugees, former students, and community service providers that has been in existence for almost 10 years. The CAC has been involved in all aspects of the study, including designing the interview protocols, participant recruitment, intervention implementation, data analysis, and dissemination. Furthermore, many key research team members and almost all the interpreters and interviewers are refugees. The second and eighth authors of this article are members of the refugee community, and most papers from the study have been coauthored by refugee partners.
Participants
In the current RCT of the RWP, two of four cohorts have been enrolled (current n = 93, planned n = 200). Half of each cohort was randomly assigned to the intervention group and half was randomly assigned to stress management control group (randomization was stratified by nationality and PTSD status). Among the 93 participants, 55 are Iraqi, 26 Afghan, and 12 Great Lakes Region African. They range in age from 19 to 71 years (M = 36.1 years, SD = 11.1). Forty-five (48%) are women, and 61 (66%) are married, 25 (27%) single, 6 widowed, and 1 divorced. Most participants have children (77%), with a range of 0 to 7 (M = 2.4, SD = 1.8). At the time of enrollment, participants had been in the United States an average of 7.6 months (SD = 7.5, range = 1-27). In terms of meeting initial screening criteria for trauma treatment, 33 (35%) scored above the PTSD symptom threshold.
Interviews
Bilingual/bicultural interviewers conducted four mixed-method interviews with each participant: pre (before random assignment), mid (14 weeks), post (28 weeks), and follow-up (56 weeks). The quantitative component of each interview included measures of depression, anxiety, trauma exposure, PTSD, culturally specific distress, quality of life, social support, acculturation, access to resources, English proficiency, use of mental health services, and spirituality. All participants responded to an initial qualitative interview, which included questions about the impact of the resettlement experience in the United States on multiple aspects of their life (e.g., health, family, work, culture, and access to resources and social support). A purposive sample of participants (n = 16 for each cohort) also had qualitative components in their subsequent three interviews, which included additional questions about their experiences in the intervention or stress management session (whichever was applicable). We conducted purposive sampling with the goal of having equal representation across four groups: people in the intervention who did not meet the PTSD symptom threshold for NET eligibility, people in the intervention who were eligible for NET, people in the control group not eligible for NET, and people in the control group who were eligible for NET. Then, we selected for variation in each of those four groups for gender and national origin. If there was more than one person eligible for each group (e.g., two Iraqi women), we examined preinterview transcripts and selected those who had more to say, either negative or positive, about their experiences. We excluded spouses and other family members who were already selected in another category. After completion of all four cohorts, longitudinal multilevel modeling will be employed to analyze quantitative data; qualitative data analyses are ongoing.
Data Analysis
Data analyzed for this article included transcripts of qualitative interviews of participants from the first two cohorts, including interviewer notes on participant requests for help and questions about study design. We also conducted a textual analysis of research meeting notes and CAC notes, which included sessions where study design was explicitly addressed. Qualitative data analysis began immediately after study initiation and is an ongoing iterative process. Data sources were imported into NVivo 10, a qualitative data analysis software package (QSR International, Melbourne, Australia). Coding was done in multiple phases, beginning with autocoding, which allows each question from a structured interview guide to be analyzed across the data. Initial coding was primarily descriptive, sorting text into broad themes associated with the intervention, but also allowed for the creation of new themes that emerge from the data. The second phase, focused coding, involved analysis of specific themes, looking for patterns and anomalies according to demographic and other patterns (Charmaz, 2014). Themes explored included “Study Implementation Issues” and subthemes, including “Control Group Influenced by RWP,” “Explaining Purpose, Design of RCT,” and “Interviewers or Interpreters Helping Participants.”
Results
We have completed intervention and most data collection with two of the four cohorts in the study. Although study implementation has proceeded as planned, our experiences and data highlight important epistemological, methodological, and ethical challenges that should be considered when conducting and interpreting results from community-based RCTs.
Epistemological Challenges
Epistemological questions have arisen throughout the development of the intervention and various iterations of studying its impact, including the current RCT. The research team and CAC have continually considered the question, “What evidence is meaningful for demonstrating intervention effectiveness?” This is certainly not a new question and many, particularly indigenous researchers, have highlighted the concern that current emphases on “evidence-based treatments” privilege Western forms of knowledge building and ignore other ways of knowing that have successfully informed healing approaches for thousands of years (Gone, 2012).
Part of the challenge in our study is that we have been implementing the intervention for many years in the community without an RCT design (but with a longitudinal mixed-methods design), and our CAC, interpreters/interviewers, and other members of the refugee communities have experienced and observed the positive effects of the intervention for themselves. Thus, they “already know it works,” and high demand among refugee community members for the intervention is further evidence of its impact. Although we decided as a team to undertake the RCT, continual concern and confusion about the RCT occurred. These questions and concerns have not impeded study implementation, but many research team and CAC members view the RCT design as (1) unnecessary because they have already seen evidence of its effectiveness that is consistent with their epistemological perspectives and (2) potentially damaging in terms of eroding community trust and withholding a helpful intervention from some people.
Methodological Challenges
Implementation of the RCT has also raised several methodological challenges that pose potential problems for the validity of our results. First, although we considered the possibility of diffusion of effects of the intervention from the intervention to control group, we did not anticipate that we would have participants in the intervention who were closely related to participants in the control group, which we would not find out until midintervention. We were careful to randomize by household, and we thought we considered all close family relationships before randomization, but some relationships were unknown to study staff. In addition, sometimes after participants in the intervention group realized that their advocates were an important resource for refugees, they referred other refugee families to their advocates for support. Advocates, conducting their role ethically, have provided various supports to participants in the control group or nonparticipants, such as helping people find employment or access health care.
A related issue is the methodological complication that diffusion of the positive effects of the intervention to achieve community-level change is an explicit goal of the intervention. 2 Thus, “contamination” of control group participants could in some sense be a demonstration of intervention success. To address these methodological concerns, we are carefully documenting relationships among intervention and control group participants that we become aware of as well as any intervention-related activities that we know control group participants have received, and diffusion of positive effects throughout the community.
Finally, our interviewers have observed that the interviews themselves may have positive effects on participants. For example, many participants have thanked their interviewers for the opportunity to talk about their lives and have frequently remarked that they feel better after having shared their experiences. For some participants, particularly many in the control group, the interviews interrupt prolonged periods of social isolation. In addition, interviews have frequently resulted in referrals for mental health services for participants who report thoughts of suicide or who request a mental health referral.
Ethical Challenges
Most important to consider are the ethical challenges we have encountered. Foremost is the fact that for many families who have recently moved to the United States, the interviewer who comes to their home may be the only person they know who speaks their language and is in the position to translate documents or facilitate communication. Because the study is designed to test the effectiveness of the advocacy and learning intervention, having research staff help participants access resources is problematic from a research design perspective. However, after extensive discussion, our study team agreed that these types of requests are reasonable and necessary. As the eighth author pointed out during one of our research team discussions, “These are our neighbors. It is our responsibility to help them.” Thus, to some extent, we have prioritized what we view as an ethical response above study design considerations.
Examples of help that interviewers have provided include helping participants read and complete forms, calling service providers or community resources, providing rides to mental health appointments for participants who were suicidal, giving advice about enrolling in ESL (English as a second language) classes at community college, pointing out particular resources on a community resource list, explaining a smoke alarm and replacing its batteries, providing interpretation for appointments, and referring participants to the study’s clinical psychologist. However, we are carefully documenting all requests for and provisions of assistance so that we can examine their potential impact on our results. This is particularly important because participants in the control group frequently make more requests of their interviewers for assistance because they do not have a student advocate working with them. The research team has also discussed and noted that from a practical standpoint, not helping participants would likely affect the study reputation, which is critical to maintain for its continued success and could adversely affect the relationship between research staff and participants, which is important for ensuring openness in future interviews.
Innovations to Address Challenges
As expected, implementing a community-based RCT that strives to genuinely adhere to a CBPR approach has been challenging (see Table 1 for a summary of challenges and the resolutions or innovations we implemented to address them). Some of our initial efforts to address these challenges included our mixed-methods design, which allows us to include multiple forms of data that measure intervention processes and outcomes at multiple levels. In terms of ethical and community/research team concerns about not everyone getting the RWP intervention, one innovation in our study design involves offering evidence-based trauma treatment (NET) to all participants who meet eligibility criteria in both intervention and control groups. This not only ensures that we respond to participants’ distress but also allows us to test the ability of the advocacy and learning intervention (RWP) to increase refugees’ engagement in individual trauma-focused treatment, when warranted.
Challenges, Resolutions, and Innovations in the Refugee Well-Being Project Community-Based RCT.
Note. RCT = randomized controlled trial; CAC = community advisory council; PTSD = posttraumatic stress disorder.
To address concerns of refugee community mistrust, our CAC decided that random assignment should occur at a public meeting, to which all participants were invited. Although staff were somewhat apprehensive about potential reactions from participants who were randomized into the control group, we used this strategy with great success. After completion of all preinterviews, participants’ ID numbers were placed into a box. ID numbers were color-coded by our stratification variables (national origin and NET eligibility, determined by ascertaining whether any adult in the household had a PTSD symptom score above the threshold for NET). Randomization meetings were held at a community center where the Learning Circles would also occur. All participants were informed of the time and location of the randomization meeting, and transportation was provided, if requested. At the randomization meetings, interpreters were present, and the process of selecting ID numbers was explained to all who attended. In addition to the benefit of participants being able to observe that the process was unbiased (further strengthened by using ID numbers instead of names on the slips of paper), an advantage of this approach was that research staff could meet with participants after the randomization process to make plans for the first Learning Circle for those in the intervention group or the stress management session for those in the control group.
Our attempts to address participants’ requests for help from interviewers have involved establishing a clear protocol for interviewer assistance. Through discussions with the CAC and among the research team, we negotiated a defined scope of allowable assistance and trained interpreters and interviewers to recognize what types of support were appropriate to provide. In addition, we asked interpreters/interviewers to, when possible, refer participants to resources on the community resource list provided to all participants or to their student advocate. Although interviewers/interpreters were not always able to confine their help within the agreed-on limits, the primacy we have given to ethical considerations and mutual learning and trust has created a team environment in which staff are comfortable to share these “breaches” of the protocol. For example, several months after an interview, one of our interpreters received a call from a participant in the control group who needed interpretation for an appointment with the attorney who was completing their families’ permanent legal residency applications. The resettlement agency refused to provide interpretation, and the attorney would meet with the family only if they brought an in-person interpreter. The interpreter felt she could not refuse to help.
Discussion
Our experience implementing a community-based participatory RCT suggests numerous conclusions and implications. First is the importance of using mixed methods in community-based RCTs. This ensures that the complexity of challenges and their potential impact will be more fully explored and understood. Incorporation of qualitative interviews and participant observation also allows us to examine the processes and context of the intervention, which are typically not visible in RCTs, as well as social relationships and power structures that may affect intervention outcomes (Smith-Morris, Lopez, Ottomanelli, Goetz, & Dixon-Lawson, 2014).
Second, research team training must involve bidirectional learning and support for interviewers who are faced with ethical challenges on a daily basis. We have incorporated opportunities for reflection and support in multiple ways, including in-depth interviewer trainings, weekly debriefing meetings with interviewers, and ongoing discussion at weekly research team meetings and monthly CAC meetings. Importantly, we have continued to be flexible and make changes to our approach and procedures, when warranted, with explicit emphasis on listening to the interviewers’ experiences and learning from them.
We have also found that discussions with CAC members, other community partners, and the research team about the purpose and requirements of RCTs must be ongoing. This can be easily overlooked, but it is essential to have continual dialogue, both because research team members’ different positionalities in the community and the academy frequently result in divergent perspectives on what is most salient and important to address and because dialogue leads to mutual learning across all team members. This further highlights that community-based RCTs must be conducted in genuine collaboration with community partners and must include them as key members of the research team (Jones et al., 2008).
Finally, our experiences demonstrate that it is essential to develop novel research designs and methods to rigorously assess interventions that have intended community-level outcomes. Although our research design has been innovative in allowing us to test whether an empowering intervention that addresses daily stressors and social determinants of mental health can improve refugees’ mental health and increase their engagement in more specialized trauma treatment when necessary, randomization at the family-level has compromised our ability to observe community-level outcomes. This is because our study is situated within the paradoxical situation of wanting to avoid diffusion of intervention effects to families in the control group, while simultaneously having an explicit intervention goal of sustainable, community-level change. Our CAC chose an RCT design to maximize our chances of obtaining funding and of having our findings be seen as credible (building an evidence base for the intervention). However, the inherent tension in applying this research design to our intervention study and the challenges in implementing an RCT design demonstrate the need to shift to a community intervention paradigm (Trickett et al., 2011) that recognizes the importance of the context of intervention processes and of community capacity building to reduce health disparities and create sustainable change. This shift, as Trickett et al. (2011) note, requires us to critically examine our culture of science, including the current context of funding for intervention research, our epistemological assumptions about what constitutes evidence of intervention effectiveness, tenure and promotion guidelines, requirements for student training, and ideas about what is publishable in academic journals. The reflections and innovations presented in this article aim to address some of these issues, particularly the need to recognize and share some of the complexities and challenges of community-based intervention research.
Footnotes
Acknowledgements
We wish to acknowledge the contributions of our community advisory council members (Jerome Ndabirorere, Freedance Nibakiza, Mohammed Alkwaz, Sulaf Alkawaz, Seyyed Qasim Sadat, Sadiqa Sadat, Hala Al-kurdi, Muslim Al-kurdi, Jean Paul Ninziza, Nina Nahimana, Marshall Jenson, Kiri Mathsen, Kresta Opperman, Ngerina Nyankundwakazi, Danielle Parker, Mandy Ortaa, and Azhar N. Al-Jarry), interviewers/interpreters (Ahmed Mashhadi, Hadeer Faris Albazzaz, Yousif Dabbach, Eric Ndaheba, Sharief Hadi, Arif Khan, Babak Shahsiah, Rana Alkhafaji, Noha Ghaly, Safaa Abid, Mohammed Alkwaz, Danielle Parker, Sherry MacKay, Samia Ayoubi, Larissa Messier, Alex Miller, Sanaa Yaqoob, Ebtisam Ali, Abdul Fahim, Arianna Martinez, Lorraine Pacheco, Courtney Howe, Alexandra Cervantes, Angelica Romero, Sahar Shirzada, Asma Yusufi, and Chonour Varyani), other research assistants (Mika Armenta, Venes Barlas, Chanda Begin, Kaveri Bisht, Mallory Fallin, Julia George-Jones, Kerstin Kalke, Casey Smith, and Sage Vogel), and all of the research participants.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by a grant to the first author from the National Institute on Minority Health and Health Disparities (R01MD007712).