Field Lessons From the Delivery of Questionnaires to Young Adults Using Mobile Phones

Retaining Contact With an Increasingly Mobile Population

No activity is more challenging when working with an adolescent cohort than minimizing selective attrition (Das Gupta, Aaby, Garenne, & Gilles, 1997). Common strategies used to increase retention include collecting a variety of contact information, assessing a participants’ long-term willingness to participate, bonding with participants through the creation of a study identity (e.g., a logo), ensuring frequent contact, hiring well-trained and enthusiastic study staff, and providing small incentives or tokens of appreciation for participation (Hunt & White, 1998). For BT20, the past 20 years have shown that the key driver of attrition is mobility and migration out of the study area (Richter et al., 2009), often as the result of job seeking. By the end of year 15, migration accounted for 27% of loss to follow-up cases. During any given data collection wave between 4% and 49% of families report a change of address (Norris et al., 2007).

Collecting Reliable Information Pertaining to Sensitive Behaviors

Adolescent smoking, alcohol, violence, conflict with the law and risky sex practices are important and active areas of research (Galambos & Leadbeater, 2000; Reyna & Farley, 2006). However, collecting accurate information about these behaviors is not easy. Parental reports are not always informed or accurate (Christian, Pearce, Roberson, & Rothwell, 2010; Hadley, Smith, Gallo, Angst, & Knafl, 2008; Schilling et al., 2007), and face-to-face interviews produce more socially desirable responses from young adults than do self-report surveys (O’Reilly, Hubbard, Lessler, & Turner, 1994; Schwarz, Strack, Hippler, & Bishop, 1991; Waruru, Nduati, & Tylleskar, 2005). To increase the reliability of sensitive data collected, the BT20 cohort has evolved its data collection methods from face-to-face questionnaires to self-completion with secret ballot submission and, most recently, to audio computer-assisted self-interviewing.

Accurately Measuring Infrequently or Irregularly Occurring Behaviors

Many risky behaviors are erratic, occurring only infrequently and sporadically (Oindo, 2002; World Health Organization [WHO], 1993). Relying on questionnaires that require young adults to recall the number and detail of these risky events over long periods between follow-ups can result in both underreporting and overreporting (Cleland, Boerma, Carael, & Weir, 2004; Slaymaker, 2004). As a result, some studies have evaluated the use of diaries as a way to record ecological momentary assessment data (Hedeker, Mermelstein, Berbaum, & Campbell, 2009).

Rapid development in mobile computing has led to modern devices being well placed to provide a supportive solution to the problems of mobility, self-report, and the measurement of irregularly occurring behaviors. In low- and middle-income countries mobile phones are widely available, providing a personal, asynchronous, and inexpensive means of communication. This article reports on our first effort to collect data from these young adults using a mobile phone–assisted self-interviewing (MPASI) platform. The objectives of the study were to assess feasibility, calculate response rates and item response characteristics, and monitor response stability across two waves of data collection.

Study Population and Sample

The study was designed as a prospective intervention evaluation. One thousand handsets were purchased and loaded with the Mobenzi Researcher (MR) survey software (Clyral, 2011). These handsets were then distributed between January 2010 and November 2010 during regularly scheduled BT20 participant visits. At the visit, participants consented to their involvement in the study and were then guided through a practice survey by trained research assistants.

A telephonic follow-up survey, using one of the alternate numbers available in the participants file, was undertaken in December 2010 to validate that participants who had received a donated handset were still able to receive and complete surveys. As this was the first attempt by BT20 to conduct an MPASI survey and software systems were untested, a targeted sampling frame was used to randomly select 435 participants confirmed in the telephonic follow-up to have survey ready handsets (see Figure 1). All 435 participants were born within the Soweto municipality between April and June 1990. This gives the sample a median age of 20 at the time of the mobile survey (Richter et al., 2007). The sample was composed of 227 females (52.2%) with 386 (88.7%) being Black. Ethical approval for the study was obtained from the Committee for Research on Human Subjects (Medical) Witwatersrand Human Ethics Committee.

OPEN IN VIEWER

Figure 1.

Phones not available for use from the 1,000 donated handsets that were distributed.

Procedures

During training and handset distribution, users were informed that they would be notified by a short messaging service (SMS) text message when a survey was ready for them to complete. Two notifications were dispatched, 2 weeks apart, requesting the completion of a 60-item survey. The first test message was sent on April 20, 2011, and the second on May 5, 2011. Both notifications included an airtime voucher (R5; US$0.69) to ensure that participants were able to respond as this was indicated to us in pilot work to be a potential constrain. After MR received an uploaded survey response, a text message was sent back to the participant to thank them for their time.

Data Analysis and Outcome Indicators

Data were analyzed using proportions, frequency tables, and other descriptive techniques. Key indicators of “success” were defined for each step in process. This provided the means by which to evaluate the outcomes of the study. These indicators were the following: (1) the number of participants possessing donated handsets capable of submitting surveys; (2) the proportion of these donated handsets, included in the study, contactable by text message; and (3) the proportion of survey responses received from donated handsets that were contactable by SMS.

Outcome 1: Number of Participants Possessing Handsets Capable of Submitting Surveys

During 2010, 1,000 handsets were purchased, of which 996 were distributed to participants because in the bulk delivery two handsets were found to have faulty batteries and two subscriber identity module (SIM) cards were not adequately registered. Among the 996 participants who received handsets, a telephonic follow-up survey conducted soon after the last handset was distributed revealed that 25 (2.5%) had inserted a new SIM card, 32 (3.2%) had deleted the MR software, and 205 (20.6%) reported having lost, stolen, broken, or given away their phone. Figure 1 presents this attrition visually.

Outcome 2: Texting the 435 Handsets Included in the Study

Of the 435 text message notifications sent out in Wave 1, 288 (66.2%) were reported by the mobile network as delivered. The remaining 147 messages were undeliverable due to the phone no longer being active on the network. It was not possible to determine whether this was due to the phone been switched off for an extended period or that the SIM card had been removed from the handset. Similarly, 271 (62.3%) of the 435 second wave SMSs were delivered. Twenty-one (4.8%) participants received SMS 1 but were not reachable for SMS 2, 4 (0.9%) received SMS 2 but were not reachable for SMS 1, and 267 (61.4%) were reached by both SMS 1 and SMS 2.

Outcome 3: Survey Responses Rates

Of the 288 handsets that received Notification 1, 105 (36.5%) individuals returned a response. Of the 271 handsets that received Notification 2, 84 (30.9%) individuals returned a response. Although only one response was expected for each survey, some individuals responded multiple times. During Wave 1, the 105 responding individuals sent in a total of 237 responses. During Wave 2, 159 responses were received from 84 individuals. The median number of responses per participant was 1, with a maximum of 24. Among the 105 respondents who uploaded a response in Wave 1, 59 (56.0%) also submitted a response in Wave 2. There were 25 (of 84 Wave 2 respondents) who only submitted a Wave 2 response (29.8%).

Challenges and Difficulties

Within 11 months of the first donated mobile phone handsets being issued to 996 BT20 participants, 734 (73.7%) reported that their phones were still capable of sending and receiving MR surveys. Of the 435 randomly selected participants who were enrolled in this study, response rates of 36.5% (105 respondents to the 288 delivered text messages) and 30.9% (84 respondents to the 271 delivered text messages) were achieved during Waves 1 and 2, respectively. Item nonresponse was found to be higher for numeric than free-text, date, or multiple-choice items.

Two clear challenges emerged from these results. First, the effort required to provision, support, and manage large numbers of cell phones for the purpose of data collection among young adults should not be underestimated. There was a high initial rate of attrition, with 205 handsets reported lost, broken, or stolen within an 11-month period. Technical support for participants having problems with their handset was an everyday activity. In addition, it was challenging to keep track of phones and SIM cards as participant reports indicated that SIM card swapping and ownership of multiple SIM cards is common. Prior to the start of the study, all 435 phones were validated as being available for participation, but 4 months later only one third of these handsets were not contactable by SMS.

The second challenge that emerged was the struggle that participants faced when trying to use the survey software. The authors had previously used MR very successfully for mobile-assisted personal interviewing. However, this study highlighted that the process and user interface design of the software needs to be modified and simplified significantly for MPASI, even with young people who are proficient in the use of mobile phones and associated applications. To successfully complete a survey required participants to receive and read the SMS, know how to find the software application on the phone, load the application, complete the appropriate permissions (e.g., allowing the software to access the network), download the correct survey given that prior surveys including tutorials remained on the phone if not deleted, successfully work through the survey, and, finally, upload the completed response. Despite thorough training, text messages sent to the research team by participants suggested that the process was too complicated for a significant number of participants’ to complete. Participants report deleting the application, not being able to find it or the surveys and not having the application available in the phone they have inserted the donated SIM card into.

Although the use of “apps” is widespread among high-income countries where smartphones are more common, the “app” approach to MPASI in the youth of low- and middle-income countries (at this time) should be considered tentatively. SIM swapping, phone “loss,” and app deletion all contributed to the low response rate achieved. Similar challenges were found in a recent review of SMS applications in the prevention of disease in low-resource countries (Déglise, Suggs, & Odermatt, 2012). Barriers included high mobile phone turnover and system maintenance difficulties.

Learning and Best Practices

Results suggest that participants were chiefly enthusiastic of the data collection format. At their own cost, and without further prompting, many participants continued to submit responses for over a year since the original data collection wave. Furthermore, the response rate of 36.5% compares favorably with rates reported for web surveys. In a recent meta-analysis of 39 mode comparison studies from the past 10 years, Shih and Fan (2008) report average response rates of 45.0% for mail surveys and 34.0% for web surveys. In a second meta-analysis (Shih & Fan, 2009), a slightly higher response rate of 53.0% was found for mail surveys, with 33.0% for e-mail surveys.

Given the challenges with handset management, theft, and loss, a best practice recommendation–when working with similar populations–is to explore the use of individuals' personal handsets. In 2010, 66.2% of the cohort reported owning their own personal handset (van Heerden, Norris, & Richter, 2010). Using individuals’ own handsets would alleviate several challenges faced with managing study-provided handsets, although not knowing the type of phones owned by the cohort would introduce complexities for software development. A further recommendation for mobile data collection from young adults in low- and middle-income countries would be to conduct mobile-assisted personal interviewing through an SMS-based survey system or through social network sites (Wicks, Vaughan, Massagli, & Heywood, 2011). This would be more familiar to participants than having to install and manage sophisticated survey software. Improving participant engagement in the mobile survey process is the final learning to emerge from this study. Given the many stimuli competing for attention in a mobile survey setting, questions need to be carefully designed to ensure that they are short, focused, and interesting. Possible approaches that could be tried to increase interest are the use of feedback to participants on their responses, simple social rewards, and gamification strategies.

In conclusion, mobile phones are an interesting and challenging method for collecting data and staying in contact with adolescents from low- and middle-income countries. Approaches need to be kept simple, streamlined, and engaging while remaining cognizant of the challenges associated with giving up control of the survey environment. By carefully considering the results, challenges, and suggestions presented above, future designs employing mobile phone–based self-interviewing can be strengthened.