Abstract
Congregation-based health program evaluations often rely on surveys, but little documentation is available regarding specific methods and challenges. Here we describe methods used to achieve acceptable response rates (73–79%) in a survey of HIV-related attitudes and behaviors in two African American and three Latino churches in high HIV-prevalence communities in Los Angeles County. Survey participation was enhanced by conducting survey sessions at church-based meetings (e.g., women’s Bible study) and after worship services; employing diverse survey staff; providing participation incentives for pastors, church coordinators, and survey participants; and working collaboratively and respectfully with congregational leaders. Achieving broad participation in church-based surveys on sensitive health topics is feasible when done collaboratively with congregational leaders and with a flexible protocol, which permits tailoring survey approaches to cultural and organizational contexts and leverages available resources appropriately.
A sizable research literature describes how church-based health programming is important for reaching underserved, minority populations to reduce health disparities. Since the 1990s, over 400 articles have been published about congregation-based health programs and the actual or potential health effects of such activities (Campbell et al. 2007; DeHaven et al. 2004). Further, there is increasing interest in how church-level interventions may be particularly effective because they operate at multiple levels of influence (Campbell et al. 2007; Winett et al. 1999). However, little has been published about the methodological challenges involved in evaluating church-level programs, including lessons that could guide future studies.
Most outcome evaluations of church-based health interventions have relied on data collected from congregants, usually through church-based surveys that were self-administered (Berkley-Patton et al. 2010; Berkley-Patton et al. 2013; Davis et al. 1994; DeHaven et al. 2011; Frank and Grubbs 2008; Lindley et al. 2010; Resnicow et al. 2004; Resnicow et al. 2001). A smaller number of studies have used telephone interviews (Campbell et al. 1999; Campbell et al. 2004; Duan et al. 2000; Resnicow et al. 2004; Young and Stewart 2006) or in-person interviews (Erwin et al. 1999).
Yet, despite this fairly extensive literature on church-based surveys, there are surprisingly few implementation details such as sampling, recruitment and survey procedures, and response rates. For example, while articles routinely include the number of churches and the total number of survey participants, few clearly indicate congregation type and the total number of congregants in each church (Berkley-Patton et al. 2010; Campbell et al. 1999; Derose et al. 2000; Duan et al. 2000), making it difficult to ascertain representativeness. Few report having selected participants randomly (Campbell et al. 1999); more commonly, studies rely on church liaisons to recruit and survey a convenience sample at each church (DeHaven et al. 2011; Resnicow et al. 2004). Moreover, the literature typically provides little information about survey design and implementation (e.g., exactly how and where self-administered surveys were done) or the number and spacing of survey waves. Thus, there is little information on how to achieve high response rates in church-based surveys of congregants.
In addition, few church-based studies have been conducted across congregations of diverse denominations, racial and ethnic makeups, and sizes. The church-based literature has been focused almost exclusively on African American churches (Campbell et al. 2007). The Los Angeles Mammography Promotion Program recruited 1,967 women ages 50–80 from 18 predominantly African American, 15 predominantly white, and 12 predominantly Latino churches through self-administered screener surveys. However, the proportion of age-eligible women recruited per church ranged from 10% of age-eligible women in very large Latino churches (400+ age-eligible women) to 75% of age-eligible women in small white churches (<100 age eligible; Derose et al. 2000). Another multirace/ethnicity study in Los Angeles (Davis et al. 1994) surveyed 943 women ages 21–89 from 23 churches participating in a cervical cancer education program, but no response rates were provided nor were any details on how the surveys were conducted.
Finally, there is little information about surveying congregants on sensitive health topics (e.g., HIV, drug use, and sexual behavior). There is a growing literature about church-based HIV interventions with African American churches (Berkley-Patton et al. 2010; Berkley-Patton et al. 2012a; Berkley-Patton et al. 2012b; Coleman et al. 2012; Francis et al. 2009; Griffith et al. 2010; Hawes and Berkley-Patton 2014; Lindley et al. 2010; Moore et al. 2012), all of which relied on interviews, focus groups, or self-administered questionnaires with limited numbers of clergy and/or individual congregants to inform and assess intervention design, acceptability, and feasibility. Data collection efforts focused primarily on a small subset of congregants and have tended to measure HIV-related knowledge and stigmatizing attitudes, not personal HIV risk factors (e.g., drug use and sexual behavior) and HIV status. Berkley-Patton and colleagues (2012a) included an extensive HIV behavioral risk assessment in their congregant surveys (Hawes and Berkley-Patton 2014); however, participants numbered 211 across eight churches with reported average sizes of 200 members each and thus represented a small minority of congregational members at each church.
In sum, a better understanding is needed of the most effective methods for facilitating broad survey participation within and across diverse congregations. In an era of growing public resistance to participation in surveys because of concerns about privacy (Morton et al. 2006), researchers conducting church-based projects on sensitive health topics are likely to encounter challenges in surveying diverse congregants. Congregations tap into important social networks and thus offer great potential for health programming among populations less connected to health-care services. However, since most previous church-based studies have used self-administered surveys, it is important to understand how feasible this approach is among populations of diverse literacy levels. Further, it is critical to identify the most promising solutions for maximizing response rates in church-based surveys to evaluate effectiveness of health interventions designed to reach the underserved.
This article describes the procedures used to achieve high response rates in congregational surveys at African American and Latino churches regarding HIV-related attitudes and behaviors. Because of the gaps in the literature on congregational health surveys noted above, an in-depth discussion of our procedures and lessons learned can inform future evaluations of church-based programming.
Methods
Study Overview
The purpose of the Facilitating Awareness to Increase Testing for HIV (FAITH) project is to raise awareness and reduce stigma about HIV, encourage congregants to get regular HIV testing, and develop strategies congregations and their leaders can adopt to assist in the community response to HIV. We pilot tested a multifaceted, congregation-based intervention to reduce HIV stigma and promote HIV testing among African Americans and Latinos that was developed using community-based participatory research (CBPR) and extensive qualitative and case study research (Bluthenthal et al. 2012; Derose et al. 2014; Derose et al. 2010; Derose et al. 2011). The multicomponent intervention was implemented over approximately three to four months in each church and involved congregant HIV education workshops, peer leader workshops, an HIV sermon and imagined contact scenario, and church-based HIV testing events (see Derose et al. [2014] for a full description). All congregants 18 years and older were eligible to participate in intervention activities, but some activities reached a broader proportion of the congregation than did others. As a church-level intervention, baseline and follow-up surveys approximately six months apart of all adult congregants were used to assess preliminary effectiveness in reducing HIV-related stigma and increasing HIV testing.
Community Partners
A community advisory board (CAB) composed of local leaders from religious and public health communities and chaired by two clergy consultants (Rev. Dr. Clyde W. Oden and Rev. Michael A. Mata) guided all phases of the study. Pastors of each participating church designated a church coordinator to serve as a liaison between the project team and the congregation to plan, schedule, and implement all intervention and evaluation activities. The project survey team, which included several project outreach workers and a number of part-time survey assistants, was racially ethnically diverse and possessed the language and cultural skills to work well with study congregations.
Survey Development and Piloting
We conducted church-wide, self-administered baseline and follow-up health surveys that focused on participants’ attitudes and behaviors related to health and HIV, including HIV risk factors (e.g., drug use and sexual behavior) and attitudes toward homosexuality, drug addiction, and people with HIV. Prior to implementing the pilot intervention trial, we pilot tested the survey questionnaire and procedures in English and Spanish with 68 congregants from two congregations similar to those in the pilot intervention, and feedback was incorporated into the survey development and implementation (Table 1). (Intervention activities were also piloted at these two churches prior to implementation.) Survey content was vetted and approved by the CAB and the pastors and church coordinators of the participating congregations.
Exemplary Feedback from Baseline Survey Pilot Participants.
Note: n = 68.
Survey Participants and Recruitment Strategies
Over 1,300 adult participants were surveyed from five churches: one large (1,200+ adult attendees) Latino Catholic, two small (<100 adult attendees) Latino Pentecostal, and two medium-sized (125–250 adult attendees) African American Baptist in high HIV-prevalence communities. All adults (18+ years) who attended participating churches were invited to participate.
Based on feedback from the survey pilot and consultation with the CAB, the survey approach and methodology utilized the following strategies to encourage survey participation: (1) the project outreach team worked closely with the designated church coordinator to establish rapport and engage key clergy and lay leaders; (2) surveys were integrated into each church’s regular schedule of services, meetings, and events; (3) survey participants were not asked to provide their names or contact information (instead they were asked four questions that generated a unique survey code to link baseline and follow-up surveys); and (4) participation incentives were offered to the church, pastor, coordinator, and congregants. The church, pastor, and coordinator each received honoraria (US$500 each) for their assistance; congregants received a US$20 Target gift card and a light meal for completing a survey.
Survey Promotion Period
During a minimum two- to four-week survey promotion period, advance information about the purpose and content of the surveys was disseminated broadly via church channels to encourage congregants to participate. Church staff and volunteers distributed recruitment flyers after worship services and during ministry meetings, placed flyers in the church foyer, put notices in church bulletins, and made announcements at church meetings.
Church-based Survey Administration
Survey sessions were conducted at times when congregants normally came to church, such as weekday evenings and weekends, commonly after worship services and/or during regularly scheduled ministry meetings or church events. Surveys were administered in group settings in classrooms, meeting rooms, or, in the smaller churches, the main sanctuary. Participants received an introductory letter and a project brochure that covered all the required elements of informed consent, which were then reviewed orally at the beginning of the survey session by the project survey administrators. Survey sessions were generally scheduled for 60 minutes each, although the survey took, on average, 30 minutes to complete. All survey materials were available in Spanish and English and trained survey staff provided one-on-one assistance to those who needed help completing the survey.
Attendance at the survey sessions varied greatly and some adaptations were made to accommodate specific congregation concerns. One African American church was reluctant to host multiple survey sessions after Sunday services, and church leaders asked us to modify our procedures so that surveys and consent materials were distributed after Sunday worship services; individuals wishing to participate filled them out at home and returned them to survey administrators at the church the following two weeks. To accommodate a large Catholic church (1,200 adult congregants) with limited availability of meeting rooms, concurrent survey sessions were conducted in adjoining classrooms and a queue was set up to organize congregants.
Survey sessions were repeated multiple times over a minimum of three weeks, but the number of sessions and the length of the field period varied based on the size of the congregation. At the large Catholic church, we conducted four baseline survey sessions during established ministry or group meeting times during weekday evenings and multiple survey sessions open to all adults congregants on three different weekend days; these sessions were spread over two months. At the small and medium-sized congregations, we had fewer sessions (usually three per church), and the time frame was three to four weeks.
Coordinated Management of the Survey Sessions
The survey sessions were managed by trained project staff; because of the sensitivity of some questions (e.g., drug use, sexual behavior, and HIV status), church staff were not involved in the distribution and collection of the surveys, and project staff placed completed surveys in lock boxes immediately after each survey session and delivered the surveys to the research office at the end of each day. For each survey session, a designated project staff member served as the lead survey administrator who ran the session, gave oral instructions, reviewed the informed consent materials, and answered participants’ questions before they began the survey. Survey assistants helped with logistics, such as greeting participants as they arrived, handing out consent materials and surveys, suggesting seating arrangements to minimize the likelihood that survey responses would be seen by others, answering questions about the survey, distributing incentives, and collecting completed survey materials.
The survey team brought child activity kits to each session with age-appropriate books and games and asked each church to provide some space within or near the survey room, where children could use these items while their parents filled out the survey. We also provided a light meal for accompanying children (in addition to parent survey participants). During the baseline survey, project staff purchased most of the food from an outside vendor, but two churches catered their own survey sessions and were paid for each meal provided. Participating churches viewed the catering experience as a welcome fund-raising opportunity, and church personnel involved in catering often became unofficial “recruiters” for the survey. Given this mutually beneficial experience, each church catered remaining survey sessions.
Ongoing Survey Feedback to the Church
We provided ongoing (e.g., weekly) feedback about our progress during the survey period to church coordinators and sought their advice when we encountered any challenges such as low turnout at a survey session. After the survey was completed, we provided each church with a customized report, which had easy-to-read bar charts showing the church’s aggregate results on each survey item, and offered to meet to answer questions.
Survey Response Rates
The number of completed surveys collected at baseline versus follow-up was almost identical: 1,319 compared to 1,320. However, as shown in Table 2, the overall estimated response rates increased from 73% at baseline to 79% at follow-up due to an observed drop in church attendance between baseline and follow-up at three churches (estimates provided by the church leaders who track attendance). Church-specific survey response rates varied from 68–99% at baseline to 75–99% at follow-up. At a small Latino church and an African American church, we surveyed nearly all adult attendees. At the large Latino Catholic church, it appeared that a large number of noncongregant community members (who were demographically different from the predominately Latino parish and indicated on their surveys that they had not attended the church in the past year) heard about the baseline survey and attended one afternoon survey session. Thus, a more conservative estimate of the baseline response rate in this parish would be 59% instead of 68%. We did not experience an influx of noncongregant community members at the follow-up survey sessions in this parish.
FAITH Project Congregant Health Survey Response Rates at Baseline and Follow-up by Church.
Note: Response rates = number of completed surveys divided by the estimated number of adult congregants (age 18 years or older) who were attending services on average (per church leaders) at the time the baseline and follow-up health surveys were conducted.
Participant Reactions to the Survey
The survey team received feedback about the survey, which was documented in field notes. Participants commented that they thought the survey was good, because the questions made them think about HIV and how it affects their community. Only one survey respondent provided negative feedback—an older congregant indicated that she was offended by some of the questions, though she opted to complete the survey. While other adults may have had concerns about surveys, none approached our staff or the church coordinators to express them.
Survey Data Quality
The overall item-level nonresponse in the pre- and post-survey questionnaires was fairly low (less than 2% per item) and appeared to be well within the typical range for self-administered surveys. There were a relatively small number of returned questionnaires (two at baseline and four at follow-up) that were entirely blank or had only a couple of questions filled out, signaling that the respondent was unwilling or unable to participate. About 30 (2%) participants at baseline and 50 (4%) at follow-up required substantial assistance from the survey administrators to complete the questionnaire due to literacy issues.
Discussion
Our results suggest that acceptable survey response rates (over 70%) in church-wide surveys can be achieved across diverse churches. Together with our CAB, we considered but ultimately ruled out alternative methods such as telephone surveys and in-person interviews both because of the cost and because we anticipated resistance to providing personal information. These privacy concerns were validated by congregational participants in our baseline survey pretests and leaders at participating churches.
Instead, we opted for survey sessions with established ministry group meetings and after religious services. The costs involved participation incentives and survey staff labor; however, the costs would likely have been much higher had we attempted telephone or personal interview surveys. A mail survey would have been very difficult since many congregations do not have updated membership lists with mailing addresses; even if such lists exist, intensive follow-up with nonresponders would have been required.
Many survey design and implementation challenges must be addressed to maximize congregants’ willingness and ability to participate. Below we discuss what worked well in our survey and describe remaining challenges and limitations.
A Collaborative Approach
Close collaboration between study staff and congregation leaders, similar to Berkeley-Patton’s Taking It to the Pews study (Berkley-Patton et al. 2010), helped establish a constructive staff–congregation working relationship. Having the same, primarily bilingual, survey administrators and project team members involved over the course of the study helped establish rapport with clergy, lay leaders, and individual congregants and allowed our team to become familiar with the culture and traditions of each congregation. This relationship-building helped congregants feel comfortable in asking for help and improved the survey team’s ability to handle unexpected problems that required on-the-spot problem-solving, such as a reserved meeting room being mistakenly assigned for another concurrent use, or a suspected minor attempting to take the survey. Further, the survey procedures were acceptable and successful because of the CBPR approach, which, as others have advocated, used culturally matched research personnel, linguistically appropriate materials, and a personalized recruitment process with endorsement from trusted congregational leaders (George et al. 2014).
Although the support of congregational leaders is vital to success, appropriate roles will vary. Previous studies that relied on church liaisons to administer congregant surveys (DeHaven et al. 2011; Resnicow et al. 2004) explored dietary behavior and physical activity. Similar procedures for our study, which collected data on drug use, sexual behavior, and HIV status, would likely not been approved by our internal review board and would have placed too large a burden on our church coordinators. Thus, each study will need to consider appropriate roles for all partners.
Church-based Survey Development and Administration
We worked hard to make the survey instrument and administration as easy as possible for the congregations and survey participants, and assistance was provided to those who had trouble completing the survey on their own. Similar to other church-based studies (Berkley-Patton et al. 2010; Davis et al. 1994; Wiist and Flack 1990), scheduling survey sessions around regularly scheduled meetings proved to be easier for the congregations and was an effective means to recruit large groups of survey participants.
Understanding unique church characteristics and working closely with the pastor and church coordinators were crucial to identifying appropriate survey content and develop a customized survey plan for each church to achieve acceptable response rates. For example, the group survey sessions conducted after services or during ministry meetings worked extremely well at most churches but not at one of the African American churches. After brainstorming with church leaders, we implemented an alternative internal review board-approved procedure (take-home surveys) that enabled us to boost our initial baseline response rate at this church from 20% to 71%.
Other logistical challenges included scheduling constraints with other church events, a shortage of large meeting spaces in which to conduct the survey sessions, and managing congregants during the survey sessions. Ongoing communication with the church coordinators and other key congregational leaders like church secretaries was important to secure space and make announcements about the survey sessions well in advance. The child activity kits and meals for children facilitated participation among parents.
Attempting to include all adult participants in the survey sessions regardless of literacy level was also important to achieving high participation rates and representing the congregation as a whole. However, variable literacy levels in the Latino congregations challenged the staffing and management of survey sessions. Our efforts to accommodate low-literate participants were largely successful, but some congregants may have opted not to come to a survey session because they could not read.
Conclusions
As church-based health promotion literature has proliferated, calls have been made to increase evaluation of these efforts and diversify them beyond African American churches and in particular to include Latino churches (Campbell et al. 2007; DeHaven et al. 2004). However, despite the importance of collaborative approaches, such as CBPR to program design and evaluation (Campbell et al. 2007), the literature provides little guidance as to how to conduct church-wide surveys of congregants in minority communities in ways that maximize response rates without overburdening congregational partners or compromising confidentiality of participants.
This study demonstrates that acceptable response rates can be achieved surveying congregants in churches of various denominations and sizes using self-administered surveys conducted in English and Spanish. An important key to successful implementation in this study was a lengthy collaborative planning process with our CAB and the leaders of participating congregations who helped design culturally and organizationally appropriate implementation plans and actively promoted the survey. Fielding the survey in established ministry group meetings and after religious services was important in achieving a broad reach. Finally, employing experienced survey field staff—who were racially and ethnically diverse, bilingual, trained in protecting confidentiality, and comfortable with the content of the surveys and the church setting—was important in establishing trust with congregants. Applying these design elements may help future efforts to survey congregants on sensitive health topics, thereby improving the science and execution of FAITH-based health programs.
Footnotes
Authors’ Note
The contents of this article are solely the responsibility of the authors and do not represent the official views of National Institute of Child Health and Human Development (NICHD).
Acknowledgments
The authors thank other members of the study’s CAB, especially Delis Alejandro, Deborah Owens Collins, the Rev. Chris Ponnet, and Richard Zaldivar. We also recognize Kristin Leuschner for her very helpful comments and excellent editorial advice. We are also grateful to the other members of the research team, especially David Kanouse for his critical comment on this article, as well as Laura Bogart, Malcolm Williams, Karen Flórez, and Alexandria Smith.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by grant no. 1 R01 HD050150 (Derose) from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD).