Abstract
Achieving proportionate minority representation in community surveys is often difficult. Such was the case in previous attempts to obtain a demographically representative sample of adults living in one southern city whose adult population of slightly more than 26,000 adults was majority white but with large numbers of black and Hispanic residents. Bringing together in a survey about community perceptions a combination of methods often used in health and marketing research and depending heavily on the cooperation of minority ministerial alliances and their member churches, the intended numbers of respondents in every one of 18 strata of a quota sample were exceeded—sometimes by a little, sometimes by a lot. The original strata quotas were based on the known age, sex, and race/Hispanic status of the adult population. A simple weighting procedure easily adaptable to other community surveys was then used to restore population proportions to what had become a disproportionate sample. This research brief describes that weighting procedure along with the techniques employed to obtain respondents, the design of the quota sample, and a synopsis of the differences found, particularly by race/Hispanic status and by age, in both importance of and satisfaction with 20 different city characteristics.
Introduction
Minority representation in community surveys is often difficult to obtain due to underlying residential, educational, economic, political, and social divisions within the community that limit interaction across racial and ethnic boundaries. This limited interaction can result in an absence of shared interests and distrust between groups (Hatchett et al. 2000; Milburn et al. 1991; Reed et al. 2003; Yancey, Ortega, and Kumanyika 2006). The limited participation of minorities in community surveys may also arise from methodological missteps such as failure to actively seek minority respondents, failure to include in the survey issues of interest to minority respondents, dependence on a survey distribution method that limits the likelihood of reaching minority respondents, and use of English-only surveys (Hatchett et al. 2000; UyBico, Pavel, and Gross 2007; Vesey 2002).
This research brief describes a successful effort to obtain a representative sample of adults residing in a small southern city of approximately 32,000 residents. The 2010 Census reported the slightly more than 26,000 adult residents of the city (i.e., those 18 years of age and older) to be 13 percent Hispanic, 27 percent non-Hispanic black, and 56 percent non-Hispanic white. The remaining 4 percent were non-Hispanic members of other races and non-Hispanics reporting multiple racial membership.
Prior sample surveys of the target city’s residents rarely reported the demographic characteristics of respondents and, when reported, consistently underrepresented minorities within the city. These prior surveys typically used some form of convenience or accidental sampling such as surveyors soliciting respondents at retail or recreational sites, surveys being made available to pick up and complete at stores, surveys inserted into the local newspaper, and online surveys available to online subscribers of the local newspaper.
The research design to be described here obtained a large and more demographically representative sample of the population in a way that could be easily replicated in other small cities. It relied heavily on four techniques, all previously cited in the literature: engagement of minority community leaders, strong reliance on minority churches, quota sampling, and postdata collection weighting of responses to bring the sample back in line with the demographic characteristics of the population. It is the bringing together of these techniques in a successful community perceptions survey that makes this effort noteworthy.
Engagement of Minority Community Leaders
The survey methodology literature has long cited the advantages of involving minority community leaders in the effort to elicit minority group participation in community surveys. Already in 1977, Weiss published a meta-analysis of 194 surveys in minority communities and reported that most researchers began by seeking the cooperation of minority group leaders, and many created minority community advisory groups. While much of the literature on seeking the cooperation of minority community leaders has come out of health assessment and illness prevention efforts (Hatchett et al. 2000; UyBico et al. 2007; Yancey et al. 2006), examples are also present in the social survey literature (Weiss 1977). Researchers have sought the assistance of minority community members in numerous ways, including participation in focus groups (Vesey 2002), advisory panels (McLean and Campbell 2003; Milburn et al. 1991; Payne 1999; Reed et al. 2003; Vesey 2002), suggestions for and critiques of survey instruments (Reed et al. 2003; Vesey 2002; Weiss 1977), testimonials in support of research efforts (Payne 1999; Reed et al. 2003; Weiss 1977), recruiting of minority community interviewers (Milburn et al. 1991; Vesey 2002), and interpretation of survey results (Weiss 1977).
Use of Minority Group Churches
Several authors have cited churches and church leaders as particularly valuable allies in survey success due to the centrality and credibility of churches within many minority communities (McLean and Campbell 2003; Reed et al. 2003; Yancey et al. 2006). UyBico et al. (2007) reported contact with local churches to be the most common form of community outreach for recruiting minority survey participants but found community outreach, in general, to be less effective than social marketing in recruiting minority participants. McLean and Campbell (2003) noted that the effectiveness of church-based recruitment of minority survey respondents varies by minority group, and Odulana et al. (2014) reported that the willingness of church members to participate in survey research is related to the education and age of the pastor. Yancey et al. (2006) found recruitment of research participants to be less effective in more religiously conservative churches, and both Yancey et al. (2006) and Reed et al. (2003) noted recruitment through churches runs a slight risk that participation is viewed by church members as compulsory rather than voluntary.
Quota Sampling
Probability sampling techniques have proven difficult to use in local community surveys due to limited sampling frames, low response rates, and high ratios of cost per completed survey (Vesey 2002; Yancey et al. 2006). This has been true for minority participants but also for nonminority participants (Brick 2011). Quota sampling offers an alternative approach.
In quota sampling (Vogt and Johnson 2011), the population is first divided into categories or strata based on one or more known, typically demographic, dimensions, and within each category, survey respondents are selected. There usually exists for each stratum a target or minimum number of respondents, thus, the name “quota.” Within each stratum, quota sampling resorts to convenience or purposive sampling, both nonprobability sampling techniques. This contrasts with stratified probability sampling in which samples within each stratum are selected using some form of random sampling.
Quota sampling techniques have long been used in opinion polling, being the primary method of sampling used by nongovernmental opinion research centers in the 1930s and 1940s (Berinsky 2006). Quota sampling cannot produce margins of error because nonprobability sampling is done within the individual strata, but it does assure representativeness on those dimensions used to stratify the population (Payne 1999). If those stratifying dimensions are relevant to the attitudes and/or behaviors being researched, quota sampling can be very useful (Brick 2011; Cumming 1990).
Postdata Collection Weighting
In proportionate quota sampling, the number of respondents within each stratum corresponds exactly to that stratum’s proportion of the larger population. In disproportionate quota sampling, the number of respondents within each stratum does not correspond exactly to the stratum’s share of the population but, when summarizing results from the entire sample, weights are assigned to respondents based on their quota so that each stratum’s contribution to the overall results corresponds to the stratum’s share of the total population (Edwards et al. 1997; Henry 1990; Kish 2004; Orr 1999). Even if not intended, disproportionate quota sampling is likely to result if the researchers cannot quickly stop the collection of data from a particular stratum once the quota is reached or if the stratum to which a respondent belongs is not known until the questionnaire is processed.
Method
The survey reported here was part of a larger project to identify ways to improve the quality of life within the city. The project included a review of past studies of the city, the convening of focus groups, a survey of the city’s resident adult population, and the making of recommendations for practical improvements that could be made in years and not decades. The present paper is only concerned with the survey of the city’s resident adult population and those other parts of the study that impacted the survey.
Self-administered questionnaires were chosen as the method of survey administration for a variety of reasons. Compared with personal interviewing, self-administered questionnaires were less expensive, required less of the respondent’s time, and offered respondents anonymity. And online surveys would have reached relatively few of the minority households in the community.
A total of 18 focus groups were held prior to the survey. Different focus groups addressed different aspects of the community’s quality of life, including racial and ethnic diversity and cooperation. Individuals from the community known to the researchers to be interested in particular issues were asked to attend the focus groups with an open invitation to bring others who shared their interest. These focus groups were valuable not only for generating content for the coming survey of city residents but also because the focus group attendees were then able to attest to other members of their community about the legitimacy of the survey they would be receiving. Several of the later focus groups were shown a draft of the survey and made suggestions for changes that would increase minority response rates.
This city includes more than 40 churches, and because church attendance was perceived to be high across all race/Hispanic status groups within the city, a major thrust in collecting data was to approach selected local pastors and ask them to invite their members to complete the survey following the Sunday service. Several predominantly white, predominantly black, and predominantly Hispanic churches were asked to serve as distribution points for the survey. Churches were given $0.50 for each completed survey. The presence of a Black Ministerial Alliance greatly helped in obtaining black survey respondents. That many of the black ministers meet together on a regular basis not only meant saving explaining the survey to each minister individually but it also meant that ministers enthusiastic about the project could persuade other ministers less enthusiastic.
The absence of something like a Hispanic Ministerial Alliance meant that pastors of primarily Hispanic churches were individually invited to have their church participate in the survey. While survey response was good in Hispanic churches that chose to participate, not all pastors of Hispanic churches agreed to participate.
Church pastors were asked to announce the survey to their members and encourage participation. Parishioners could take the survey home and return it to the church the next time they attended but they were urged, instead, to complete the survey immediately after the service. For this reason, surveys were designed to be short and self-administered. Surveys were printed, and pencils were made available.
To keep the survey to two sides of an 8.5″ × 11″ sheet of paper, the list of city characteristics that respondents were asked to evaluate in terms of personal importance and satisfaction was pared down to 20. To identify the stratum into which a completed survey belonged, respondents were asked their sex, their age (three categories), if they were of Hispanic status, and their race. Respondents were also asked if they resided within the city’s limits or not. This last item was essential as most of our collection methods were likely to also include some noncity residents. The survey was printed in both English and Spanish versions.
Citing limitations in previous city surveys, the project researchers had promised city officials a large sample of the adults residing in the city that would be representative of the city’s racial/Hispanic status makeup. To achieve that, a quota sampling design was adopted. The 26,342 adults residing in the city as reported in the 2010 Census were divided on the basis of sex, age, and race/Hispanic status. Stratifying on the basis of income was considered but focus groups confirmed researcher suspicions that respondents would be reluctant to reveal even approximate income levels. More detailed age and race/Hispanic divisions were considered but rejected because they would make reaching quotas for small population groups too difficult. Table 1 shows the size of each population category (indicated by the letter P). The large number of 18- to 29-year-olds in the city reflects the presence of a university within the city. University students whose place of usual residence while attending school is within the city (either on or off campus) are city residents and, therefore, included in the target population.
Strata by Population Size (P), 3 Percent Quota (Q), and Actual Sample Size (S).
A 3 percent sample of the population was judged to yield not only one of the largest non-Census samples of the city but also a reasonably good number of respondents in all but a few of the 18 Sex by Age by Race/Hispanic categories. Table 1 shows the quota for each of the categories (indicated by the letter Q).
Not surprisingly, surveys from the churches enabled us to meet and exceed our quotas for persons 55 and older. In addition, they provided many respondents age 30 to 54; however, they resulted in relatively few 18- to 29-year-old respondents. Also, not surprisingly, surveys from churches provided more female respondents than male respondents. Approximately three fifths of the eventual total number of usable surveys were collected through the churches.
At this point, other methods for obtaining survey respondents were employed. Surveys were distributed at retail stores, barber shops, and beauty salons. They were also distributed at some large local businesses. Participants in those early focus groups were asked to invite their friends in the community to participate in the survey. University faculty and students were invited to participate, and student organizations with ties to groups within the city were asked to bring surveys to those city residents the organization meets with or serves. By processing surveys as they arrived and tallying responses by stratum, it was possible to direct surveyors specifically to locations potentially rich in population groups inadequately represented in the sample. These nonchurch-related efforts produced additional respondents primarily in strata that had not reached their quota but also added some respondents in all other strata, thereby lessening any bias introduced by too heavy a reliance on church attenders.
It took eight weeks to complete the data collection but eventually the quota in each category was met and exceeded. The hardest respondents to obtain were middle-aged black males and black and white young adult males. Table 1 shows the final sample size by category (indicated by the letter S). In addition to the 1,095 surveys from city residents that could be placed within the quota sampling categories, 433 surveys were received from persons residing outside the city or from city residents who could not be placed in the quota matrix because they failed to answer the sex, age, or race/Hispanic questions. A small number of surveys were also not used because of obvious carelessness in completing the questionnaire (e.g., a continuous line down a single column of answer choices in a matrix table).
That each of the strata quotas was met was in large part because the data collection phase could be kept open as long as it was. That the quotas were exceeded was not surprising and, in fact, to be expected in community surveys such as this. Surveys were being completed simultaneously at different churches as well as at a variety of local businesses. How many completed surveys would be returned from any particular site was unknown. And while toward the end of the data collection phase, attempts were made to target locations where respondents with needed demographic characteristics were more likely to be, other respondents could and did complete surveys. Furthermore, all of the sites where surveys were being distributed would have both city residents and persons living outside the city. Only when the surveys were processed could those from noncity residents and those that failed to answer the demographic questions needed to place them in a stratum be set aside.
So, why not simply stop processing completed questionnaires once a stratum’s quota was reached? To do so would be to leave legitimate data unused. Particularly when a simple weighting procedure was available, there was no reason to not use all the surveys collected.
Because the quota was exceeded in some categories by as little as 3 percent (middle-aged white males) but in other categories by as much as 139 percent (young Hispanic females), postdata collection weighting was employed so that categories in which quotas were dramatically exceeded do not unfairly distort sample results. The weight assigned to each case in a stratum was simply the stratum’s proportion of the city’s adult population multiplied by the study’s total sample size divided by the stratum’s sample size.
So, for example, the 950 Hispanic males ages 18 to 29 in the population represent 3.61 percent of the city’s adult population. If our sample of 1,095 usable surveys had been proportionate, we would have had approximately 39 surveys from Hispanic males ages 18 to 29. But we obtained 43. Thus, each survey in this stratum receives a weight of 0.9184.
By contrast, black males ages 18 to 24 make up 5.47 percent of the adult population of the city. In a proportionate sample of 1,095 surveys, we would expect about 60 surveys from this group. We received just 47. Thus, each survey in this stratum receives a weight of 1.2736.
Of course, weighting a disproportionate quota sample does not make it a probability or random sample. While the sample becomes representative of the city on the dimensions on which the strata were created (age, sex, and race/Hispanic status in our case), sampling within the stratum remains nonrandom—purposive, at best, in an attempt to sample different types of persons within the stratum. However, as all previous community samples were similarly nonrandom, achieving representativeness on these demographic characteristics in itself represents an accomplishment.
Results
Table 2 shows how the weighted sample ranked the 20 city characteristics first in terms of how the city rated and then in terms of how important the characteristic was. The city rated best in terms of its religious life (possibly an artifact of our data-gathering method), its historic downtown area, and its low crime rate. It rated least well in terms of shopping opportunities, entertainment venues, and opportunities for business start-ups.
City Rating on 20 Characteristics.
Of the 20 characteristics, the weighted sample saw as most important the level of crime, the availability of jobs, and health care. Judged least important to the respondents were bilingual services, bike paths, and public transit.
Table 3 reports differences by race/Hispanic status while Table 4 reports differences by age. Differences by sex are not reported here because they were small for both how the city rates and for how important the individual characteristics were.
Hispanic, Black, and White City Residents.
Scale: very bad (−10) to very good (+10).
Scale: not important (0) to very important (+10).
Young, Middle-aged, and Older City Residents.
Scale: very bad (−10) to very good (+10).
Scale: not important (0) to very important (+10).
In terms of race/Hispanic status differences (Table 3), whites generally gave the city the highest marks while blacks generally gave it its lowest marks. The race/Hispanic status groups most differed (as indicated by the distance between the highest and lowest group average) in how they rated the city in terms of recreation opportunities, available housing, the historic downtown area, and race relations. The race/Hispanic status groups most differed in the importance they gave to bilingual services, public transit, shopping opportunities, and race relations.
In terms of age group differences (Table 4), middle-aged adults were generally more critical of the city’s features than were young adults and older adults. The age groups most differed in how they rated the city in terms of public schools, availability of jobs, public transit, and bike paths. The age groups most differed in the importance they gave to public schools, local government, opportunities for business start-ups, and religious life.
Conclusion
This paper demonstrates how a number of well-established survey and sampling techniques can be easily combined in a community perceptions survey to yield a sample in which minority groups are included proportionate to their numbers in the population. Making use of a short, self-administered paper-and-pencil questionnaire, quota sampling with postdata collection weighting, and heavy reliance on minority church cooperation, a local community survey known to be representative on sex, age, and race/Hispanic status was achieved. Quota sampling as used here remains very much a nonprobability sampling technique with no guarantee of representativeness beyond the dimensions used to divide the population—in this case, those dimensions were age, sex, and race/Hispanic status. Within each stratum, efforts should be made to achieve as representative a sample as possible, in other words, a purposive sample. In communities where regular church attendance continues to be the norm and where minority participation has historically been difficult to obtain, dependence on churches as survey distribution points can be invaluable. The methodology used here can be replicated with relative ease in other community surveys.
Survey results documented age differences and race/Hispanic differences in judgments in both the importance of and satisfaction with city characteristics. These results, in turn, can be brought back to the focus groups convened at the start of the study for interpretation and reaction, thereby combining qualitative insights with quantitative data.
Footnotes
Author’s Note
Interested readers may request a copy of the original community study proposal of which the survey described in this research brief was one part. Email requests to
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received financial support for the research, authorship, and/or publication of this article: Financial support for data collection came from a grant from the city being surveyed.