Abstract
One of the main obstacles for survey researchers—especially those conducting surveys in difficult contexts such as postconflict areas—is accessing respondents. In order to address this problem, this article draws on an ongoing research project to reflect on the utility of mobile phones to connect with hard-to-access populations in conflict affected, low-income countries. It considers the strengths and weaknesses of a number of different mobile phone survey modes. The article goes a step further and discusses how (potential) survey respondents can be included in the survey design process thereby increasing the relevance of the research to them and hopefully encouraging them to participate. We conclude by considering the issue of “good enough” methodologies, or the need to balance methodological rigor with an understanding of the exigencies of suboptimal research contexts.
Introduction
Survey methods allow researchers to examine broad patterns of experience across communities and allow for relative generalizability through random selection of respondents. However, one of the main obstacles for survey researchers—especially those conducting surveys in hard-to-access contexts—is accessing potential respondents. This article draws on an experimental research project in sub-Saharan Africa to consider how mobile phones can be used to reach hard-to-access populations as a tool for survey research. Our principal aim is to show how the experiences from our own research project (anonymized here for review purposes) may be instructive for other researchers, especially in the use of multiple survey methodologies. As well as drawing on innovations from the research project we seek to build on the emerging corpus on the utility of the mobile phones as a research tool (Raento, Oulasvirte and Eagle 2009; Sugie 2016; Stern, Bilgen, and Dillman 2014).
The article has secondary aims such as discussing the possible tensions between the desire for “authentic,” inclusive and bottom-up methodological rigor and the demand for scientific rigor by parts of the social sciences. We consider if research can be true to the perspectives of the researched and the methodological demands of the social scientific community. Another secondary aim is to investigate how survey target groups can be included in the research process, therefore extending the potential to access hard-to-reach populations by using their own vernacular and indicators for survey questions. That is, how can hard-to-reach groups, once found, be included in the survey design, thereby increasing the relevance and inclusive nature of the research? As this was experimental research, we aimed to create a virtuous circle whereby community members could see the relevance of the research to their own lives (especially given that the indicators were sourced from the bottom-up) and therefore be motivated to participate in the research.
Our project uses focus groups to find out peoples’ perceptions of peace, safety, and social change in specific localities in societies that have been affected by violent conflict and/or authoritarianism. Rather than a research team based in the global north developing a set of indicators and then taking these to “the field,” the research asks local people, through focus groups, to develop their own set of indicators and then applies these via a rolling survey. The research is designed and administered by local researchers and communities as a way of encouraging the identification of issues that are relevant to communities at the neighborhood or village level. The aim has been to pilot a civic epistemology that moves away from imposed research methodologies and addresses the barriers between the researched and the researchers, something that has been on the agenda for mixed methods scholars for several years now (Mertens 2007). Our research project is interested in local knowledge and perspectives and is anxious not to impose terminology and narratives on communities, thus further stripping away their power and agency. A key problem, however, was how to access hard-to-reach populations for the survey segment of the research.
While the focus of our research project was on highly localized views of peace, safety, and security in societies emerging from violent conflict, we posit that our discussion and experiences have relevance across the social sciences in relation to accessing hard-to-reach populations. Our approach was not without its problems, both ethical and practical, and it is hoped that these experiences can be instructive for future researchers deploying participatory action research in research involving hard-to-access populations.
In terms of structure, the article begins with an outline of our research project, with a particular focus on the challenges it faced in accessing target populations in conflict-affected areas. The article then discusses the use of mobile phone surveys and our attempts to use them to combine bottom-up research with social scientific strictures. Possible survey modes are discussed: interactive voice response (IVR) surveys, mobile phone handset surveys, simple voice calls, text message surveys, and mixed mode surveys. The third section goes on to discuss the possibility of including hard-to-access populations in the survey design, thus addressing the aim to lessen the barriers between the researcher and the researched. The conclusion deals with the broader issue of the balance between faithfulness to methodological strictures, and the need to make compromises when operating in suboptimal contexts for research. When faced with the judgment call of “when are methodologies good enough?” we hope that our research methodology was able to meet both minimum scientific criteria and respect the sensitivities of the communities under research.
Accessing Hard-to-Reach Populations
The Everyday Peace Indicators project sought to pilot a methodology that would ascertain localized attitudes to peace, security, and safety in communities affected by violent conflict. It aimed to capture local knowledge in changing contexts by community-sourcing (via focus groups) indicators of change. Once these peace, security, and safety indicators had been identified and verified by the community (i.e., not by the research team) the indicators were then turned into survey questions and taken to a wider sample of the same community using multiple survey modes (i.e., those in the locality who had not participated in the focus groups). Using community generated indicators allowed us to develop more rigorous survey questionnaires, since these had immediate meaning and relevancy to survey respondents using vernacular familiar to their everyday lives. The community generation of indicators, and the wider epistemology that this reflects, also steered us toward adopting multiple survey modes. The project operated in three locations each in conflict-affected countries and was as interested in the research methodology as it was in research findings.
A key problem facing our research was how to access survey respondents in the postconflict states of South Africa, Uganda, and Zimbabwe. Given that the research was seeking bottom-up views of peace, safety, and security the research was necessarily focused on sensitive matters. While South Africa, Uganda, and Zimbabwe may be formally termed as “postconflict,” they each have characteristics that may make potential respondents reticent about participating in perception research. South Africa suffers from high levels of gang violence and crime, has seen major xenophobic and labor-related violence in recent years, and is dominated by a single political party. Uganda experienced a long-running insurgency by the Lord’s Resistance Army with significant numbers of young people being abducted and thereafter experiencing reintegration problems. In recent years, the fall out from the civil war in neighboring South Sudan has brought large numbers of refugees into northern Uganda. The polity is well described by the term “big man politics”—President Museveni has been in power for over 30 years. Zimbabwe also has centrifugal tendencies, with President Mugabe in power for 29 years, and his regime jealous to clamp down on alternative sources of power.
For understandable reasons, potential survey respondents may wish to mask their true feeling from state surveillance, neighbors, criminal gangs, and groups in society who see a particular narrative as inviolable. In addition to political sensitivity and personal safety, there were geographical and logistical reasons that made some potential survey respondents difficult to access. The research project worked in three locations in each country, with some of those locations being rural and remote. Robert Chambers’ (1997) observation of a main road bias in much field research still holds true due to the sheer inaccessibility of many locations to outside researchers.
There is now a substantial body of work, from a variety of disciplines, that has confronted the hard-to-access population issue. Heckathorn (2011), Kogan (2011), and others have recommended respondent-driven sampling or chain referral methods, whereby respondents recommend a limited number of other potential respondents using a coupon system. For Gile and Handcock (2010:286) respondent-driven sampling “reduces the confidentiality concerns generally associated with sampling from stigmatized populations.” Ellard-Gray et al. (2015:8) recommend “a combination of sampling strategies” as well as “flexibility and creativity,” while Huan et al. (2015) stress the importance of anonymity. Monahan and Fisher (2015:721) note how the language that research projects use is often “encoded with politics” and thus recommend nonstigmatizing language and, if possible, finding project advocates within the populations or organizations targeted by the research. Benoit et al. (2005:266) also recommend close community–academic cooperation so as to develop reassurance among the researched. For Marpsat and Razafindratsina (2010:9) “capture and recapture” is a suitable method of minimizing harm to both researchers and respondents.
An important difference between our research and much of the research cited above was that we were not necessarily targeting a subsection of society such as those possibly stigmatized because of addiction, illness, or sexual behavior. Our research sought to find a cross section of society in a particular geographical location of a conflict-affected community, and so did not have to take additional measures to identify and secure the participation of a subgroup, nor provide a clinical setting for the research. The primary obstacle facing our research was geographical and logistical. Our research sample may have included people who were minorities or stigmatized for a variety of reasons, but these identities/conditions were not the primary aim for including them in the research. The reason these populations can be deemed “hard-to-access,” however, is primarily related to three factors: (1) their locations—some locations were difficult and expensive for the researchers to access because of uneven terrain, poor roads, and long distances from the centers where the researchers were based, (2) the political sensitivity of the survey questions and high levels of societal suspicion and distrust due to conflict and displacement, and (3) the security risks involved in reaching them. There is general agreement that these three impediments are the primary difficulties for collecting micro-level data in conflict-affected societies (Van der Windt and Humphreys 2016).
Before progressing, we would stress that research techniques for reaching hard-to-access populations should be placed within a wider set of methodological and epistemological concerns. In relation to our research, which can be most comfortably categorized as peace and conflict studies, we are aware of trends in anthropology and sociology that promise to deliver more human-centered research (Brewer 2010; Brigg and Bleiker 2010; Lie 2013; Mac Ginty and Richmond 2013; Sylvester 2010). Our research project reflects a trend toward bottom-up forms of research that, it is hoped, may be able to access the apparently “authentic” voice of communities under research (Herring 2008; Millar 2014). It is aware, however, of the pitfalls of “romanticizing” all things local (Briggs 2005; Richmond 2009) and of the dangers of attempting to speak for the subaltern (Spivak 1988). Qualitative research on peace and conflict increasingly discusses the positionality of the researchers and shows an awareness of the responsibilities that researchers hold to the researched (Chabal 2012; Fischer 2007; Sabaratnum 2013).
Quantitative methodologies are also prominent in peace and conflict studies. The promise of scientific, objective, robust, and replicable studies chimes with donor demands for greater specificity in reporting from recipients (de Coning and Romita 2009:6-8). Quantitative approaches have been criticized for fetishizing methodological rigor (Cramer 2002; Pugh, Cooper, and Turner 2011:4-5). In this view, they become data harvesting and data set interrogating projects that seem to have little connection with the topic under study.
Our project draws from both the qualitative and quantitative toolkit and so is in the tradition of interdisciplinary and mixed method research (King, Keohane, and Sidney Verba 1994; Mahoney 2010:139; Mertens 2007:222). The project is aware of the possible advantages and pitfalls associated with participatory action research–related methodologies in conflict-affected societies. Many of these are connected with the partiality of respondents, their freedom and ability to express their opinion, and gatekeeping by those anxious to maintain a particular community or group narrative (Lundy and McGovern 2008). We are also aware of the opportunities and challenges entailed by allowing others to participate in the research design process. As Robins (2010:193) observes “a participatory research design demands that the researcher yields some control of the research agenda, and indeed the research question, to the researched.”
Having given a brief overview of our research project, its interest in accessing hard-to-reach populations, and its wider epistemological worldview, our task is now to make a case for the utility of mobile phone surveys in reaching these populations.
Mobile Phone Surveys and Hard-to-Reach Populations
Just how viable is mobile phone survey research, especially in developing world contexts? As the number of mobile phone subscriptions approaches global population figures, the rate of mobile phone availability in Africa continues to grow. In 2013, out of every 100 inhabitants on the African continent, 63 owned a mobile phone with an active subscription (ITU 2013). In South Africa in 2014, mobile phone ownership stood at 89 percent of the population—the same figure for the United States (although smartphone ownership in the United States was much greater; Pew Research Center 2015). Nigeria, Ghana, and Kenya all had over 80 percent mobile phone ownership. Interestingly, age is not a predictor of phone ownership, with about the same percentage of under and over 35s owning a phone (Pew Research Center 2015). Mobile phone penetration reflects poor infrastructure and maintenance of, often state-owned, landline communication networks. The median of households with landlines across 23 sub-Saharan countries was a mere 2 percent (Tortora 2014). People in these countries are using their mobile phones for traditional purposes such as text messages and phone calls but also for making or receiving payments with mobile money applications (Pew Research 2014). Indeed, a common sight during the fieldwork for this project was roadside and storefront signs advertising “mobile money” or the ability to send and receive funds via mobile phone transfers. A significant source of revenue in small villages where there is little electricity is offering to charge mobile phones with solar panels for a small fee.
The spread of mobile phones has allowed researchers interested in community-based knowledge to see the potential of this technology for data collection in seemingly inaccessible areas. Given logistical and security concerns, conducting research using mobile phones provides an attractive alternative or complement to existing field-based surveys, interviews, and ethnographic research methods, which may require extensive fieldwork. Previous mobile phone studies have been conducted across the continent in South Sudan, Tanzania, South Africa, Uganda, Central African Republic, Liberia as well as in Latin America and Asia (Demombynes & Gubbins 2013; Dillon 2012; Tomlinson and Solomon 2009, peacebuildingdata.org). Private sector firms, such as Mobenzi, developed mobile-based research platforms that have been extensively utilized in health and agricultural research in multiple African contexts. Importantly, they also developed software to efficiently interrogate research results.
Having established the widespread availability of mobile phones, and the possibility of using them to access hard-to-reach populations, our next task was to consider how particular survey modes would work in particular circumstances. Our project is a longitudinal study, which gathers data to track changes in perceptions of security and peace over time. Therefore, the same survey is repeated several times to be able to track whether or not people’s perceptions of peace and safety in their communities has changed. As noted earlier, the indicators were created by the community members themselves through a series of focus groups. Crucial to our project was the notion that indicators would be bottom-up and community produced. Our aim was that this would make the research relevant to community members (rather than imposed or remote) and that this might have an impact on participation levels—in terms of both focus group and survey participation.
Through partnerships with community-based organizations, we were able to access focus group participants who were reimbursed for their travel costs to the focus group locations. Their journeys were sometimes treacherous and long, involving significant time spent on boda-bodas (motorcycle taxis) or, in some contexts, danger. We sought to gather focus groups that were as representative as possible of the communities we were working in and therefore required participants from even the most remote or dangerous areas. These indicators were turned into a survey that was run in the community at large through mobile phone surveys. In order to do this, we integrated different survey modes depending on the particularity of each context. The contexts were variously urban and rural, subject to extensive or light peacebuilding interventions, and stable or prone to continuing instability, and each context had a bearing on how accessible potential respondents would be. The surveys were more difficult to administer in hard-to-access communities because of two reasons: first, we strived to be as representative as possible of the communities and therefore needed to cover as much ground as possible, which was often difficult due to rough terrain, long distances between households, and mobile phone coverage. Second, surveys that work with representative samples must send enumerators into the field at least once to collect telephone numbers. Enumerators can be at risk, especially those who are not from the locality because community members do not know them. This makes surveys additionally difficult to conduct in hard-to-access localities. Telephonic IVR and simple voice call surveys were piloted in highly fragile environments where it was too dangerous to send enumerators or fieldworkers out into the community, mobile phone handset surveys were conducted in areas where there was little mobile phone reception or penetration such as in the rural areas of Uganda. Some South African project locations experienced serious gang warfare when the survey was about to go into the field, and therefore we opted to have local volunteers randomly collect mobile phone numbers in their own neighborhoods in order to conduct simple voice call surveys. We determined that text message surveys were too dangerous to conduct in politically sensitive areas (e.g., Zimbabwe) because we did not want respondents to be caught with potentially subversive information on their phones.
In each of the survey modes, the questions were generated by the communities through a series of focus groups and participatory action research methods. Survey participants in the communities were asked to answer questions on a 1 to 5 Likert-type scale: 1 = never; 2 = rarely; 3 = sometimes; 4 = often; or 5 = always. Wherever possible, all participants, regardless of survey mode, were provided with a printed Likert-type scale card, which provided visual representation and local language translation of the possible answers. In the event that this was not possible, enumerators were trained to repeat the Likert-type scale answers after each question. A Likert-type scale response was chosen because it provided the most detailed account of people’s perceptions while still providing the researchers with a quantifiable response.
What follows is a brief summary of each of the survey modes and their strengths and weaknesses for accessing hard-to-reach communities. Based on our experience, the pros and cons of different mobile phone–mediated survey modes are summarized in Table 1.
Different Types of Mobile Phone Data Collection.
IVR Surveys and Simple Voice Calls
Our project made use of IVR surveys as a means to reach hard-to-access populations. IVR research calls for a straightforward recording that is easy to understand, which generally is best achieved by a human voice because it is recorded in a way to achieve a higher quality of conversation with the respondent than is true for a synthesized voice (Steiger and Conroy 2008:290). Respondents subsequently answer questions by entering a number into their phone’s dial pad. IVR is one of the survey methods of choice for participants who have difficulty reading and writing and can be very useful in areas where there are high rates of analphabetism. It provides consistent delivery of survey questions without any interviewer variability or unintended influence and may provide more honest reporting on sensitive items than surveys administered by a field-worker or telephone operator. Survey researchers have recognized IVR’s future potential as a survey tool in the developing world and in areas where smart phones are ubiquitous (Stern et al. 2014:293). IVR offers the power and complexity of computerization combined with the privacy of self-administration and can be administered to populations who do not have easy access to expensive technology such as computers. Although IVR interviewing does not use live interviewers, it can still be subject to bias if the choice of voice somehow influences the response. IVR has been found to be an extremely efficient methodology for cross-cultural studies being conducted in multiple languages, since it is relatively inexpensive to translate and program a survey into additional languages. In addition, IVR is ideally used when the data desired are numeric or can easily be linked to a numeric code, such as “press 1 if yes, press 2 if no” (Steiger and Conroy 2008:286-289).
Our project also used simple voice calls, which can be used instead of IVR surveys and require enumerators to call respondents and enter data remotely. One such study found that phone surveys were particularly useful and cost-effective for projects with high-frequency enumeration of a single set of respondents and those that require data at levels of aggregation above the household or individual, such as our study (Dillon 2012). One technique is to have an operator call respondents and remind them of the survey and its elements and then forward participants to the recorded IVR survey. This allows operators to troubleshoot during the call and also clarify questions and remind participants of the purpose of the study. Another technique, and the one mainly used in this study, is to have operators call respondents directly and enter answers into a mobile phone handset survey or directly into a computer.
IVR and simple voice surveys are subject to the so-called nonobservational errors or survey errors that arise because part of the population failed to be measured due to inaccessibility to mobile phones to participate in the survey. Also, inconsistent network coverage creates sampling problems at the village selection stage and is most likely correlated with important characteristics, such as distance from major towns, road quality, water supply, and average income (Dillon 2012). In addition, it has been found that aural mode respondents tend to give more extreme responses on the positive ends of the scale compared to visual mode respondents, even when controlling for demographics (Dillman et al. 2009:16). Unlike IVR surveys, simple voice calls are also plagued by many of the same issues as interviews. Respondents are more likely to give answers that are guided by the interviewer and they are also more likely to give socially desirable answers (Stern et al. 2014). However, they offer the advantages of having an enumerator able to guide and encourage respondents to complete their surveys as well as to call respondents back if the call is made at an inconvenient time. In addition, simple voice calls allow for more flexibility and a higher response rate than IVR surveys. They also ensure the safety of enumerators in highly violent or sensitive contexts since phone calls can be made from the safety of one’s home or office.
Finally, in highly sensitive contexts where sending field-workers into the community in order to randomly collect phone numbers can be life-threatening, IVR survey respondents can be contacted and trained through snowball sampling or a community meeting in order to collect cell phone numbers if randomization is not necessary. The lack of randomization can be remedied in some instances by using community members to collect mobile phone numbers randomly in their own neighborhoods and following up with IVR or simple voice calls. In this study, participants in one IVR survey were recruited through a community meeting in their locality. During this meeting, their mobile phone numbers were collected along with demographic details. At this time, participants were provided with instructions on how to use the IVR system as well as a basic training on the Likert-type scale and an explanation of our project. Following this training, participants were called in the morning on a predetermined date and first-time nonrespondents were called again that afternoon. Respondents were introduced to the survey and the instructions were explained: Welcome to the Everyday Peace Indicators project. Press 1 for English. Druk 2 vir Afrikaans. Press 0 to repeat this message. (If pressed 1) This survey contains 13 questions and will require 5–10 minutes to complete. After each question, please select a response that matches your everyday experience in Atlantis. Press 1 to continue. Press 0 to repeat this message.
Participants were given incentives to complete the survey in the form of airtime credit: (If pressed 1) If you complete this survey, your cell phone account will be credited with 20 rand of airtime by the end of the week. If for whatever reason you are unable to complete the survey during this call, you will be called one more time later today. If you have any further questions, please SMS K at xxx or call xxx. Press 1 to continue. Press 0 to repeat this message.
Mobile Phone Handset Surveys
The use of mobile phone handsets as a data collection tool has been found to be an effective tool for data collection which is preferable to a paper-based approach and has been used primarily in large-scale public health and agricultural studies in low-income countries (Demombynes, Gubbins, and Romeo, 2013; Tomlinson et al. 2009). This relatively new data collection process, also known as mobile phone–assisted personal interviewing (MPAPI; van Heerden et al. 2014), offers advantages over paper-based surveys by ensuring less margin of error in data entry (due to paper survey loss, destruction, or enumerator error), automatic uploading and real-time data analysis, random selection of respondents through a random operator, GPS, photo integration into surveys, encrypted access to a web-based interface provides data security and respondent confidentiality, and multilingual survey options (vs. carrying multiple paper surveys to remote locations). Disadvantages can be associated with the risk of the enumerator carrying an item of relative value in areas with high rates of crime, battery life issues, and the need to connect to a network in order to update surveys.
Various companies offer applications that can be downloaded to mobile phones or tablets and open-source applications are now available. 1 These applications offer researchers the possibility to upload a survey, which is distributed to multiple handsets and can be amended in real-time, depending on need, by pressing a button. Field-workers use mobile phones to administer the surveys in the field and once submitted, the data are sent directly to the computer interface and can be accessed by the researcher. Mobile phone handset surveys are also appropriate for contexts with little mobile phone reception or access to mobile phones, since they allow for data storage and automatically upload surveys when linked to a network. They allow for respondents to be randomly selected and for fieldworkers to collect data via face-to-face interviews. In our surveys, the mobile application surveys had the highest response rates of all the mobile phone survey modes we piloted. However, simple voice call and IVR surveys may be more amenable to panel surveys where the same people must be contacted at different intervals over time. The challenge of mobile application surveys can be daunting to return to the same people in villages and homes that are often not marked and which may be populated by multiple family members. This can be reconciled by using the GPS feature on the phone as well as recording names and location descriptions or by contacting respondents before returning to the village and agreeing to meet again at a predetermined location and time. Van Heerden et al. (2014) found that the advantages of a particular survey mode depend on the context in which the survey is being undertaken. Handset surveys may be more amenable to large, complex studies lasting several years across multiple sites with poor infrastructure (such as this one), whereas paper surveys may be more useful when start-up time is limited.
Text Message Surveys
Text message surveys can be administered either in a free-form format or with a set text-input guide that requires a certain sequence of information to be entered into the survey response. The latter is restrictive in that it requires substantial training and also makes changes to questions on an ongoing basis difficult. In addition, text message surveys have been found to have a lower response rate than face-to-face surveys, although they are more cost-effective particularly in longitudinal studies (Li and Wang 2013). However, they have been found to be effective in increasing the response rate of other mobile phone survey modes when a text message was sent in advance in order to inform respondents that a survey will be administered shortly (Steeh et al. 2007).
Text messages are also widely used in crowdsourcing and crowdseeding surveys (Van der Windt and Humphreys 2016). The crowdsourcing approach was pioneered by Ushahidi, which gathered and stored data collected by SMS (Short Message Service) on a central platform and then visualized it on a map for monitoring purposes. Another Uganda-based crowdsourcing approach is U Report. This United Nations Children’s Fund (UNICEF)-funded project is being conducted in Uganda by sending out weekly texts to U reporters across Uganda. A group of nine partner organizations meet regularly to determine the issues to be discussed with Uganda’s youth who make up the majority of U reporters. Topics have included female genital mutilation, outbreaks of disease, safe water, early marriage, education, health, and inflation. Once a topic is decided, UNICEF sends a question via SMS text to U reporters who can respond either with a simple menu-based reply or with personal messages. Text messages are free to receive and respond to, and U Report has been instrumental in addressing the recent outbreak of a mysterious epilepsy-related illness known as “nodding disease,” which mainly affects children under age 15. 2 UNICEF has found that this system has been an effective way of tapping into local communities: “U-report offers a cost-effective, easy-to-implement means of assuring accountability by tapping community knowledge to learn the local and personal impact of policy and development schemes, health interventions and outbreaks.” 3 Crowdseeding approaches, where respondents are preidentified and provided with mobile phones and training, also tend to use text messages as a data collection tool. One such recent study by Van der Windt and Humphreys (2016) uses a crowdseeding system to collect information from villagers texting information to a research platform in South Kivu, Eastern Democratic Republic of the Congo. Through their project Voix de Kivus, they were able to study the micro-level dynamics of conflict and the effects of development aid on conflict.
Mixed Mode Surveys
A combination of modes in one single survey has been found to help overcome some of the individual mode limitations (Stern et al. 2014). This study has found that a mixed mode approach to surveys is most beneficial wherever possible in longitudinal studies with high-frequency enumeration of a single set of respondents in low-income settings. The option of mixed mode surveys also allows for a more accurate sample of the greater community and mixed mode surveys, in which people are surveyed using a mixture of different kinds of methods, has been found to reduce nonresponse error and increase response rates (Dillman et al. 2009:1).
Where appropriate, our project uses a combination of simple voice calls and mobile application to collect survey data over time. These allow for the random selection of respondents but also ensure that the same people will be contacted in future surveys. In this case, the same respondents are surveyed at different time points using different modes. This is practically very efficient in a longitudinal study because it allows for random selection of respondents but assures that the same respondents can be contacted on the second and third rounds of the survey administration.
One of the main benefits of using a mixed mode approach was that we were able to assure random sampling of respondents while still maintaining a cost-effective longitudinal survey. Random sampling was done according to context and resources available and was used in all the pilot projects except for Atlantis (a town 40 km to the north of Cape Town) where we piloted the IVR surveys, which were snowball sampled. In Uganda, we randomly selected the areas we would visit by writing down the names of the areas of each village and then randomly selecting them out of a hat. In South Africa, we used Google Earth snapshots of the community and then randomly selected areas by separating out public housing apartment blocks. For both the IVR and simple voice call surveys, respondents were first asked whether they exclusively used the phones that we called. We collected phone numbers only of respondents who owned their own phones or used the phone of an immediate family member. On the next iteration of the survey, they were asked whether they had responded to the first survey. In the cases where survey respondents were different, we provided them with the opportunity to find the original respondent or to do the survey themselves, in which case all demographic details were collected.
Drawbacks to mixed mode surveys are related to the possibility that survey mode may have an effect on the data that are collected. In other words, different modes may have different effects on the question and answer process and therefore lead to different answers to the survey questions. However, although these are recognized as important methodological issues, mode effects have been found to be small in well-conducted surveys (de Leeuw and Hox 2008). Yet, mixed mode data collection does present questions of data integrity, which can be seen as problematic for those who perceive mixed-mode approaches as having negative consequences for data collection. In cases where research is being conducted in conflict-sensitive environments, it is sometimes necessary and preferable to employ a mixed mode design and important to remain flexible. Therefore, the benefits and drawbacks must be weighed in order to determine whether or not “good enough” methods are preferable to not being able to conduct studies at all.
Bringing Populations in: Using Focus Groups in Survey Design
It is widely recognized in the social sciences that using focus groups is appropriate as a pretest for assessing measurement validity in surveys and other types of instruments (Cyr 2015; Fuller et al. 1993; O’Brien 1993; Wolff, Knodel, and Sittitrai 1993). Focus groups can help guide researchers and give us more context and detail in order to test whether the questions we use in surveys mean what we want them to mean. Researchers in the social sciences concerned with measurement validity often use focus groups as a strategy to inform questionnaire design, from the formulation of whole question categories to the fine-tuning of wording on particular questions (Morgan 1996:134; Wolff et al. 1993:120). In order to avoid systematic biases and the data distortions surveys can produce, researchers use focus groups to confirm that the questions indeed measure what they seek to measure. They also can use focus groups to help to interpret results or to assess reactions to the surveys among respondents or simply to triangulate results (Cyr 2015:11; Wolff et al. 1993:120). Cyr (2015) gives examples of some of the more recent research conducted in this way in top political science and sociology journals. She finds that out of the twelve articles that used this approach, five used focus groups as a pretest to test the validity of their survey instruments. These studies used focus groups to help code dependent variables, to pretest a video used in an experiment, and to pretest interview questions, among others (Cyr 2015:9). Focus groups have also been used before surveys to anticipate survey nonresponse or refusal problems in hard-to-reach populations and to explore ways to minimize these potential sources of sampling bias (Desvousges and Frey 1989).
In order to include populations in the survey design, this study takes the use of focus groups in survey questionnaire development a step further. It heeds concerns of measurement bias, in particular in light of the many concerns expressed by those apprehensive about inclusion and participation about survey question design and the imposition of concepts and categories on communities by outsiders (Chambers 2013:51; Merry and Wood 2015). Instead of merely using focus groups to inform the survey generation process, community members themselves actually create and select the indicators that provide the basis of the survey questionnaires. This article is based on an experimental research project called (anonymized here for peer review purposes) in which we sought to elicit indicators of peace, safety, and social change from community members and then test those indicators among the community. The project is run in communities in three sub-Saharan African countries and takes focus groups as its starting point. These, complemented by discussions with local civil society partners, form the basis of our survey questions. We did not work from a set of hypotheses in order to create survey questions for the communities. Instead, we looked to local communities to identify their own indicators of peace, conflict, or change. The focus groups played a key role in this; they developed the project’s unprompted survey questionnaire according to what local community members felt were the most appropriate indicators of peace in their locality.
In each community we conducted separate focus groups with women, men, and youth, and asked general questions about their perceptions of peace and safety. The focus groups were held in local languages and included questions like, “What factors do you look to in your daily life to determine whether you are more or less peaceful?” and “How do you recognize peace in your own community and everyday life?” Once the focus group transcripts were transcribed and translated, we intervened to create long lists of indicators from the discussions. We generated these by reading through the transcripts and picking out factors people mentioned they used to determine peace and safety in their surroundings. It was important for us not to give any examples of indicators or to lead the groups into certain directions. Therefore, we decided it was best to ask as general questions as possible and then generate the lists of indicators (sometimes up to forty indicators) from these discussions.
Once the lists were developed, we were ready for our verification focus groups (VFGs). The purpose of the VFGs was to come up with a final list of approximately 10 indicators that would be included in the community-wide survey. The VFGs also allow for the verification of the indicators gleaned from the focus group transcripts. Participants often had a much clearer idea of what we were trying to collect once they saw these lists and were able to add more relevant indicators. In addition, some of the original indicators had little or no value and were merely anecdotal. Therefore, the VFGs offered an opportunity to discuss the lists with the groups again and ensure that the final selected indicators were the most representative of a particular community. In the VFGs, five representatives of the original groups met with the representatives of the other groups in order to discuss and determine the final lists. This was because it was unfeasible to include all participants in a second focus group due to time constraints and conflicting schedules. During this discussion, the participants were explained the goals and intentions of the project and asked to help create the indicators. We felt comfortable at this juncture, giving participants lists since they had been created by their own previous conversations and were not examples or indicators written by us and imposed on them.
Through two group exercises, the participants narrowed down their lists in order to decide which indicators or questions were included in the final survey. First the participants discussed their indicator lists in the smaller groups and decided what their final ten indicators would be. These groups were facilitated and supported by local moderators, especially in areas where there was illiteracy. In these groups, they were asked to rank their indicators on a scale from 1 to 10; 1 was the indicator they most used to perceive whether there was peace and safety in their community and 10 was the indicator they least used to perceive their peace and safety. When deciding which indicators to include in their top 10 list, participants were asked to consider the following questions as a group: Which indicators do you use most often in determining whether there is peace in your community? Which indicators will also be recognizable to the other groups?
Once these groups determined their lists, the participants came together and were each given ten stickers or pieces of paper. The community at large was invited to join these groups, which were often held on the weekends or afternoons and evenings so as to maximize potential representation from all sectors of society. All of the areas where the research was conducted suffered from high unemployment so the pool of potential participants was considerable. Participation was voluntary and so it was probable that focus group participation was skewed in favor of those who were particularly interested in local issues—certainly focus group discussions were often passionate. The lists were taped to a wall or a tree in a communal space and participants voted with their stickers or pieces of paper for their favorite indicators. The final list was discussed to finalize the process and was then included in the mobile phone surveys conducted in the communities. The active facilitation by a skilled moderator was key in this second stage of the VFG because individuals were sometimes pressured by their groups to choose certain indicators. Therefore, ensuring that people were free to choose whichever indicators they felt most reflected their perceptions was important.
We took a number of steps to maximize the confidence that potential participants would have in the research process. All participation was voluntary and based on informed consent. There was little incentive for participation, other than light refreshment or travel reimbursement. Our local research partner organizations had links in each of the areas where the focus groups were run and so were able to judge issues of safety. They guided us by identifying whether local conditions were unsuitable to hold focus groups. The separation of focus groups into sessions for men, women, and youth was meant to facilitate free conversation but also to minimize the possibility of one group intimidating the other. No personal identification data were taken from focus group participants other than their sex and age. The focus group transcripts did not record the names of participants (instead, using “participant 1” and “participant 2”). The focus group facilitators, who were experienced, used a very light touch style of discussion moderation and did not press issues if participants did not want to discuss them. The focus group transcripts reveal that participants were more open and engaged in some contexts than in others but overall gave quality answers since the sensitivity of our questions was not very high. The focus groups were held in public or semi-public spaces such as under a tree in a village or in a community hall. There was no way that these spaces could be safeguarded against eavesdropping, for example, by nonparticipating neighbors or state agents.
A primary aim of using focus groups to inform the questionnaire design was to ensure that the indicators of peace, safety, and security were locally meaningful and conceptually sound. This, we hoped, would aid us in reaching hard-to-access populations. If potential respondents could see that a survey was tailored to local perceptions, and tapped into vernacular descriptions of peace and security, then they may be encouraged to see the survey as relevant to their lives, and worth participating in. Certainly our experience in the focus groups was that community members were exercised by conditions in their communities and anxious that something be done about it. It was more difficult to gauge community emotional investment in issues via the mobile phone surveys (where follow-up questions, face-to-face discussions, etc., were not possible).
Concluding Discussion: Is Good Enough Enough?
We argue that it is possible to construct and use methodological tools that satisfy the desire for “authentic,” inclusive and bottom-up approaches and satisfy some of the methodological strictures favored by the more scientifically rigorous parts of the social sciences. When working in complex and highly sensitive environments, it is difficult if not impossible to maintain the methodological purity some scholars insist upon (Gullette and Rosenberg 2015). No society is a “laboratory,” and conflict-affected societies or those undergoing significant transition often present additional challenges to researchers and the researched (Smyth and Robinson 2001). This complexity is recognized by survey researchers who advocate for survey design to be adjusted according to the needs of the population of interest (Stern et al. 2014:295).
The debate about whether scientific approaches can be incorporated with indigenous ones has been ongoing for several decades mainly by development practitioners and scholars interested in agricultural indigenous knowledge. Initially, western science was seen to be universal and global, rigorous, objective, and dependable, whereas indigenous knowledge was seen to be provincial and limited, subjective, primitive, and untrustworthy (Agrawal 1995; Briggs 2005; Mercer et al. 2009). These artificial distinctions were then later criticized by development scholars and calls were made for more integrated frameworks using participatory approaches in which relevant indigenous and scientific knowledge could be used (Agrawal 1995; Briggs 2005; Mercer et al. 2009). Although indigenous and western approaches may use different epistemological and ontological frames, they often overlap and feed off one another. Dichotomizing began to be seen as unhelpful because indigenous and scientific knowledge are not always distinguishable and knowledge is dynamic, fluid, and adaptable (Agrawal 1995; Mercer et al. 2009:218). Indeed, Agrawal (1995) goes as far as calling ridiculous the distinction of forms of knowledge as “western” and “indigenous” (Agrawal 1995:30). He advocates a more integrated approach to distinguish only by discussing multiple domains and types of knowledge rather than creating fixed categories. Briggs (2005) points out that western science is as much a social construction as indigenous knowledge.
However, our challenge was perhaps more straightforward than more abstract and theoretical discussions on indigenous knowledge and scientific strictures. We wanted to know how survey research could be conducted in a methodologically satisfactory way in a conflict-affected environment and still remain true to the bottom-up ambitions of the project? In many ways, our project has been an exercise in attempting to square this circle. Survey purism and scientific rigor pulls the project in one way. The anthropological and sociological ambitions of the project pull it in another. And this is made more difficult by demands from anthropologists and sociologists for reflexivity in research and that we recognize the importance of thinking about what is gained and lost in research decision-making rather than what might be ideal (Luttrell 2000; Mazurana, Jacobsen, and Andrews Gale 2013; Miller 2014). Matters are complicated by a series of logistical problems affecting research in societies affected by conflict. The result has been compromise and the use of mixed methods. Good enough methods allow the researcher to be flexible and use the most rigorous methods possible while still being able to conduct research in these kinds of contexts. Mobile phone data collection is especially suited to this kind of research, and multimode approaches allow for more cost-effective and consistent results. Multimethod approaches also allow researchers to work in conflict-affected contexts they might otherwise be unable to access.
In many cases, conflict-sensitive research depends on face-to-face encounters that allow the researcher to use emotional intelligence and make professional judgments about risk to the research subject and their ability to exercise informed consent. Mobile phones and other technological interfaces do not always offer a face-to-face dimension that might bring assurance to the research subject. But phones do offer the advantages of privacy and anonymity—very significant factors in conflict-affected settings in which it may be necessary to mask one’s true opinions from family members, neighbors, community leaders, or state agents.
It is clear that all research involves compromise, especially where human subjects and conflict-affected societies are involved. We argue that the criteria for this decision needs to be carefully calibrated in a way that the views and sensitivities of the researched are not overshadowed by a task-oriented wish to collect data. If the balance is between methodological purism and the ethos of including the researched, then we opt for the latter. Of course, surveys necessarily involve the compression and standardization of responses. This usually means compromises in including local and bottom-up voices. Including the researched in the survey design process is one way to ensure that local voices are heard and that the important questions are asked. In this way, the survey methodology can be inclusive of bottom-up voices. Finally, there are limits to the extent to which any epistemology can be civic, plural, and inclusive. All research needs originators and designers. Most of it requires funding and organization. This raises issues of power and the location and identity of the principal instigators of that research.
We are not so naive to believe that our research process, however sensitive we seek to make it, has left no footprints. Despite trying to be as nondirective as possible, the very fact that we have asked questions about “peace,” “conflict,” and “safety” will have directed respondents in particular directions. Research on such areas is necessarily political. Yet there are some indications that our research did pass a major goal set by ourselves: can bottom-up voices be heard and then be used in survey research? At the outset of the research, we were curious to see whether the indicators formulated by communities would be expressed in the language of Inyernational Non-Governmental Organizations (INGOs) and Non-Governmental Organizations (NGOs), of political leaders, or in a community vernacular. In other words, would the community-identified indicators be a form of mimicry whereby participants “speak back” using formulaic, often western-originated, terminology (Bhabha 1994)? In fact, by giving research participants as much free rein as possible, many did use their own vernacular to describe their situation and formulate indicators. For example, in communities in two countries (many thousands of miles apart) respondents referred to the barking of dogs at night as an indicator of prowlers and thus of insecurity. What was refreshing about this is that the academic and policy literature on insecurity in societies coming out of conflict does not—to the best of our knowledge—mention this phenomenon, nor use it as an indicator. This, and similar examples, found their way into community-chosen final lists of indicators that were then used in the mobile phone surveys. It was reassuring to us, as a research team, that we had not imposed our own indicators of peace and, consequently, survey questions on communities.
The debate on the balance between scientific rigor and the conflict sensitivity of research connects directly with the issue of hard-to-reach populations. Strict scientific rigor can, in some cases, create barriers to accessing potential survey respondents. An understandable push to ensure the validity of a research sample may make it difficult for individuals to participate in the research. This might particularly be the case in conflict-affected contexts or in relation to sensitive topics. Our experience recommends the calibration of specific types of survey modes to varying circumstances.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The author wish to acknowledge financial support from the Carnegie Corporation of New York.