Feasibility and Quality of a National RDD Smartphone Web Survey: Comparison With a Cell Phone CATI Survey

Abstract

Internet-enabled smartphones and wireless communication technologies are opening new ways to conduct web-based self-administered data collection for academic or nonacademic research. Considering the relative advantages of self-administration such as the low cost, overall convenience, and collection of better data about sensitive topics, survey researchers are eager to explore conducting national web surveys of the general population via smartphones, especially if they can use probability-based random-digit-dialing (RDD) sampling methods. But questions about the feasibility of such surveys remain. We conducted an experiment using national samples drawn from an RDD wireless sampling frame to compare two administration methods: a smartphone web survey using SMS (text messages) invitations and a cell phone (smartphone or feature phone) survey through computer-assisted telephone interviewing (CATI). This study was conducted using the National Survey of Smoking and Health in South Korea, a country with a very high rate of smartphone ownership. The geographic and demographic representativeness of respondents in the smartphone web mode (self-administered mode) was similar to that of the cell phone CATI survey (interviewer-administered mode), although the completion rate in the former was nearly half that of the latter. A majority of demographic variables and measures of experiences, attitudes, and perceptions of cigarettes or smoking showed significant mode differences for both unweighted and weighted estimates. The total cost of the web survey was about one fifth that of the telephone survey. The results demonstrate the potential of a smartphone web survey as a stand-alone or primary mode of data collection, if carefully designed and implemented.

Keywords

smartphone web survey cell phone CATI survey single-frame random-digit-dialing design SMS text messages sample representativeness response rate survey costs

In the last few decades, many interviewer-administered surveys, including random-digit-dialing (RDD) telephone surveys, have been dealing with rising costs and declining response rates, with potential implications for data quality (e.g., see American Association for Public Opinion Research [AAPOR], 2019; Groves & Harris-Kojetin, 2017, pp. 23–27). At the same time, advances in wireless or cellular technologies and telecommunication have presented a number of opportunities and challenges for survey researchers. Internet-enabled smartphones offer new opportunities for web-based self-administered data collection. Reflecting this, a number of recent studies have explored the capabilities of smartphones for both active and passive data collection (e.g., see Conrad et al., 2017; Elevelt et al., 2019; Kreuter et al., 2020; Revilla & Couper, 2019).

Given the advantages of self-administration such as relatively low cost, overall convenience, and improved data quality (especially for sensitive topics), survey researchers are exploring ways to make use of smartphones for national probability-based surveys. But conducting web surveys for survey research at the national level still presents challenges related to coverage and sampling (see, e.g., Couper, 2017). With regard to sampling, no email list of addresses exists to serve as a sampling frame for a national survey, so other methods of sampling (such as RDD or address-based sampling) must be employed (see, e.g., AAPOR, 2014; Couper, 2017; Schonlau & Couper, 2017). With regard to coverage, while the rate of smartphone penetration has increased dramatically in recent years, coverage is still far from universal (see, e.g., Couper et al., 2018; Heffner & Mull, 2017; Gummer et al., 2018; Peterson et al., 2017). Researchers conducting web-based surveys must either rely on nonprobability samples and recruitment methods (e.g., volunteer online panels, convenience sample of potential respondents, etc.) or randomly select a sample using a different frame, contact sample persons using another mode (telephone call, mail, or face-to-face), and invite them to complete a web survey on a smartphone, tablet, or personal computer.

Internet-enabled smartphones offer the opportunity to combine a sampling method (RDD) with a mode of data collection (web surveys) that may mitigate the sampling challenges of web surveys while still taking advantage of the self-administered mode. As smartphone penetration rises, this approach may offer increasing promise. However, while RDD remains a useful method for sampling, the method of invitation remains a challenge. Short message service (SMS) or text messaging offers one possible solution. As Smith (2015) reported, SMS is the most widely used smartphone feature in the United States (100% among ages 18–29, 98% among ages 30–49, 92% among those 50 or older). In South Korea, just as in the United States, SMS has become a part of people’s daily life. A survey of South Korean smartphone owners reported that about half (51.2%) of their time on smartphones was spent on average in making voice/video calls (34.7%) or sending SMS (16.5%), with the other half spent using other functions such as Internet, social networking service, and email (see Lee et al., 2019). Thus, if an invitation is sent via SMS with a link to a web survey, smartphone users can access the web survey directly on their smartphones. But several countries have laws regulating unsolicited text messages. In the United States, the Telephone Consumer Protection Act does not exempt research from the restriction requiring prior consent to receive text messages (see Marlar & Hoover, 2019). Similar restrictions are in place in Europe under the General Data Protection Regulations (see https://gdpr.eu/). In South Korea, unsolicited text messages without prior consent are similarly proscribed; however, the regulations are limited to advertising information for commercial use. Text messages for noncommercial use including survey research by nonprofit organizations are exempt from such regulations (see Korea Communications Commission, 2017). South Korea also has the highest rate of smartphone ownership in the world (95% of Korean adults in 2018; Pew Research Center, 2019) and the world’s fastest mobile Internet connection speed (Opensignal, 2017).

Given these opportunities and challenges, we designed an experiment using RDD probability samples selected from a national wireless (cell phone) sampling frame in South Korea in order to compare two survey administration methods: a smartphone web survey using invitations by SMS text message and a cell phone survey conducted using computer-assisted telephone interviewing (CATI). We explore response rate differences between the two modes as well as the geographic and demographic representativeness of respondents. We also investigate differences in responses between the two modes and compare them to official statistics. In addition, we compare the relative costs of the two modes.

Review of Previous Research

There are two broad uses of SMS in survey research: one as a mode of data collection (see, e.g., Hoe & Grunwald, 2015; Lau et al., 2019; Marlar et al., 2014; Schober et al., 2015; Van der Heijden, 2017; West et al., 2015) and the other as a method of invitation to a web survey, either directly (see, e.g., De Bruijne & Wijnant, 2014; Mavletova & Couper, 2014) or as a supplement to another invitation mode, for example, as a prenotification method (e.g., Bosnjak et al., 2008; Dal Grande et al., 2016; Holland et al., 2014) or as a reminder method (e.g., Respi & Sala, 2017). Our primary focus here is on using SMS as the primary means to invite sample persons to complete a survey online. There are relatively few papers that have examined SMS text messages as the only or primary method of inviting people to complete a web survey, especially for an initial contact or “cold calling.”

In one of the first studies to use SMS to invite sample persons to a survey, Steeh et al. (2007) conducted an experiment among subscribers of one mobile service in the United States. They compared SMS-only invitations to an inbound CATI survey to a combination of SMS and calls, and a call-only protocol. They obtained a response rate of 2.6% for the text-only condition compared with 23.2% for the cell-only condition and 24.2% for the text/cell combination.

Marlar et al. (2014) reported on experiments conducted by Gallup in the United States, among people who were previously recruited by telephone and agreed to receive text messages. They tested an SMS survey, an SMS invitation to a web survey, and a telephone survey, using a short (six questions) and long (two questions) questionnaire. They obtained response rates of 13%, 13%, and 38%, respectively, for the short version (with similar results for the long version).

De Bruijne and Wijnant (2014) tested the use of SMS versus email for invitations to a web survey in an online probability panel in the Netherlands. They found a higher overall response rate for the SMS group (74%) than for the email group (70%), but this was particularly true of responses via smartphone (71% vs. 57%). Mavletova and Couper (2014) similarly tested SMS versus email invitations among members of a nonprobability panel in Russia. They randomly assigned panelists to SMS versus email invitations and reminders in a 2 × 2 design. The SMS-only condition (SMS invitation and SMS reminder) resulted in a higher response rate (53.2%) than the email-only condition (44.1%), but the highest response rate was obtained in the group sent an email invitation and SMS reminder (60.9%).

While the evidence presented above that SMS as a supplemental method—whether for prenotification or additional reminders—may increase response rates over email invitations only, we are aware of no other studies testing the exclusive use of SMS invitations to web surveys, particularly among sample members that had not been previously recruited (i.e., part of a panel). The only evidence of likely response rates to an SMS “cold-call” invitation comes from Lau and colleagues (2019). They conducted an SMS survey in four African countries based on lists of mobile numbers obtained from providers. In three of the countries where regulations permitted cold-call SMS invitations, they obtained response rates (using AAPOR RR1) of 4.83%, 0.36%, and 0.19%, respectively, for Kenya, Ghana, and Nigeria.

Study Design and Implementation

This experiment was embedded in the 2018 National Survey of Smoking and Health (NSSH) conducted by the Survey and Health Policy Research Center (SHPRC) at Dongguk University. The main purpose of the NSSH is to produce national data on tobacco use of adults (19 years of age or older). The NSSH used a national dual-frame (landline and cell phone) RDD survey in 2016 and 2017, and analysis of data on telephone use (a subset of survey questions in the NSSH) showed that nearly all responding adults (97%) owned a cell phone (smartphone or feature phone), only 3% of adults were landline only, and cell phone RDD samples were more demographically representative than landline RDD samples (S. W. Kim et al., 2017). Based on these earlier results, we decided to conduct the 2018 NSSH using a single-frame cell phone RDD sample design instead of a dual-frame RDD sample design (see Peytchev & Neely, 2013, and Kennedy et al., 2018, for similar developments in the United States).

Single-Frame Cell Phone RDD Sample Design

Cell phone RDD surveys have been used in South Korea since the mid-2000s, starting with dual-frame designs (see Lepkowski et al., 2005). RDD samples selected using these methods have been used at SHPRC for a variety of studies (see, e.g., Kim et al., 2012, 2014, 2017; Park et al., 2012).

The SHPRC generates a cell phone RDD sample using active seven-digit cell phone 10,000 banks. All 10,000 possible suffixes, from 0000 to 9999, are appended to the seven-digit codes to generate 11-digit cell phone numbers. The cell phone RDD frame size is about 77 million numbers, with about 63.6 million cell phone subscribers in 2017 (Statistics Korea, 2018), suggesting that about 83% of numbers are potentially working numbers.

For this study, we selected an RDD sample of 30,900 cell phone numbers using a single-stage equal probability of selection method (EPSEM). These sampled cell phone numbers were randomly assigned to each of the two survey modes with slightly different sizes: 15,900 to the smartphone web survey and 15,000 to the cell phone CATI survey. This unequal allocation is based on findings from the 2017 NSSH, where about 6% of adults were found to own feature phones rather than smartphones. While feature phones do not preclude users from completing web surveys, they make it very difficult to do so (see Mavletova & Couper, 2014). We thus slightly inflated the number of cell phone numbers for the smartphone web survey (by 15,000 × 0.06 = 900) to reflect this.

Based on prior experience, we expected about 1,000 completed interviews from 15,000 sample numbers in a cell phone CATI survey. We had no comparable experience for a national smartphone web survey, but based on prior research (reviewed earlier), a small pretest for the current study, and smartphone web surveys among university students conducted by SHPRC (see Lee et al., 2019; Woo et al., 2015), we expected the yield to be very low (i.e., less than 200). Given this uncertainty, and to efficiently manage SMS text messaging and callbacks, we used four sample replicates, each made up of a random subset of about 3,000 numbers, obtained after screening to remove business or nonworking numbers among the 15,900 numbers. The details on survey completion in each sample replicate are shown in Table 1.

Table 1.

Distribution of SMS Delivery Results and Survey Completions by Sample Replicates in Smartphone Web Survey.

	Sample Replicates (SR)				Total	Percent A^a	Percent B^b
	SR1	SR2	SR3	SR4	Total	Percent A^a	Percent B^b
Initial RDD sample size	15,900				15,900	100.00
Size after screening	11,991
Random assignment	2,998	2,998	2,997	2,998
SMS sent	2,998	2,998	2,997	2,998	11,991	75.42
Successful delivery of SMS	2,510	2,560	2,538	2,534	10,142	63.79	100.00
Completed interviews	60	70	61	50	241		2.38
Follow-ups (F1, F2, F3)
F1
SMS sent ^c	2,452	2,490	2,471	2,486	9,899	62.26
Successful delivery of SMS	2,407	2,458	2,429	2,451	9,745	61.29
Completed interviews	39	32	28	24	123		1.21
F2
SMS sent ^c	2,363	2,420	2,395	2,413	9,591	60.32
Successful delivery of SMS	2,337	2,393	2,363	2,383	9,476	59.60
Completed interviews	32	27	24	33	116		1.14
F3
SMS sent ^c	2,307	2,368	2,336	2,347	9,358	58.86
Successful delivery of SMS	2,275	2,332	2,303	2,325	9,235	58.08
Completed interviews	11	17	19	10	57		0.56
Subtotal of completed interviews	142	146	132	117
Total of completed interviews	537						5.29

Note. RDD = random-digit-dialing.

^a Percentage calculated from the number in the “Total” column divided by 15,900.

^b Percentage calculated from the number in the “Total” column divided by 10,142.

^c Some follow-up SMS were sent to completed interviews. Respondents were asked to report the last five digits of their cell phone numbers for verification purposes. A small number shared the same last five digits.

National Data Collection Protocols

Survey period

We gathered data in March and April 2018, with 4 weeks for each sample replicate in the smartphone web survey and 4 weeks in total for the cell phone CATI survey. The two modes started on the same day. In the smartphone web mode, the subsequent replicates started a week later, with data collection lasting a total of 7 weeks.

SMS text message invitations

The invitation text message to encourage people to participate in the smartphone web survey consisted of 68 words. It included the name of university-based survey organization (SHPRC), contact number, purpose and importance of the survey, and the statement “We are not advertising or selling anything” as well as a unique URL to the survey website. It also emphasized that only smartphone owners could participate in the survey. The follow-up reminders were sent once a week for 3 weeks. A commercial SMS text messaging service (see https://smartsms.aligo.in/) was used to send the invitations to sample numbers in each sample replicate at the same time in a batch process. Some SMS messages were not delivered successfully for a variety of reasons including an incorrect or disconnected phone number or the device unable to receive SMS. The service provided detailed information on the delivery status of each SMS invitation. Cases that were resolved (e.g., completed interviews, break-offs, business numbers, nonadults, do-not-send-SMS requests, etc.) were not sent reminders. Screening questions were asked at the beginning of the survey to determine whether it was a business phone number and whether the sample person was an adult. Some recipients contacted the SHPRC by phone to opt out of receiving SMS or to ask about the survey. The comparison of delivery results for SMS by sample replicate is shown in Table 1.

Call attempts in the cell phone CATI mode

A total of 10 interviewers worked on the survey, with two supervisors. Call attempts were mainly made on weekday evenings and weekends, using specified calling schemes (e.g., days and times of the calls, number of callbacks, days and times of callbacks, and time lag between calls) implemented and controlled by the CATI system. At least 10 call attempts were made to sample numbers that had not yet been contacted. Additional callbacks were also made to reluctant sample persons.

Incentives

Incentives were offered in both modes. In the smartphone web mode, incentives were offered to both nonsmokers and smokers. Offering cell phone coupons as incentives was mentioned in the introductory statement of the questionnaire, but the amount was not specified. When respondents were identified as current smokers in the initial filter questions, they were asked to complete additional detailed questions about smoking, and a cell phone coupon worth 4,000 KRW (about US$3.48) was offered as an incentive. Respondents identified as nonsmokers were offered a 2,000 KRW coupon (worth about US$1.74) for completing the survey. In the cell phone CATI mode, incentives were not mentioned in the introductory statement, and a 4,000 KRW coupon was provided to current smokers only, with no incentive given to nonsmokers.

Questionnaire

For both modes, we used the same NSSH questionnaire, which consisted of 119 questions that cover demographics, smoking and tobacco use, attitudes, and perceptions of smoking or cigarettes as well as an introductory statement to the survey. For the smartphone web mode, it was emphasized in red that only smartphone owners could participate in the survey, and participation was restricted to smartphone devices only (i.e., not PCs or tablets). A subset of 30 questions, including nine filter questions, was asked of all respondents regardless of smoking status. The remaining 89 questions were more detailed follow-up questions to those who answered “yes” to filter questions or identified as smokers. These included 10 open-ended frequency questions (nine of which were only asked of smokers) and one narrative question (asked of everyone). Two key questions in the survey were used to estimate the prevalence of cigarette smoking among adults: (1) whether the respondent had smoked at least 100 cigarettes in their life and (2) if so, whether they now smoked either daily or less frequently.

The web survey was optimized for smartphones. The survey included no grid or matrix questions. Response options were displayed in a single vertical list. Most (76%) of the web pages contained a single question, while the balance had two or three questions per page.

Survey Software

For the smartphone web survey, we used SurveyMonkey (see https://ko.surveymonkey.com), which is available in the Korean language, has been used for previous web surveys at SHPRC (e.g., Lee et al., 2019), and allows for web surveys to be optimized for various types of smartphones in service. Since the survey may still appear in different forms on different mobile operating systems (Android, iOS, Windows, RIM, or others) or screen sizes, we fully tested different models of smartphones (iPhone, Samsung Galaxy, Google Nexus, LG X4+, etc.) to identify any problems.

For the cell phone CATI survey, we used the Blaise software (see http://blaise.com/) developed by Statistics Netherlands. The SHPRC has used Blaise in a centralized telephone facility for many national CATI surveys since 2005.

Results

SMS Delivery Results and Survey Completion in Smartphone Web Surveys

The initial sample of 15,900 cell phone numbers was screened by trained operators at the SHPRC to remove nonworking or business numbers (similar approaches are used in the United States; see Marketing Systems Group, 2020). The set of 11,991 active numbers (75.42% of 15,900) was then randomly assigned to the four equal-size samples. Table 1 shows the delivery results for the SMS messages as well as survey completions by sample replicate and overall, including three follow-up messages.

First, it can be seen that SMS messages were successfully delivered to 63.8% of all sampled numbers (or 84.6% of those screened). This rate of successful delivery stayed relatively constant across the three follow-ups, ranging from 58.1% to 61.3% (see Percent A in Table 1). Second, as shown in the “Percent B” column, each additional reminder brought in more completed interviews, ranging from 2.4% of successfully delivery messages in the initial invitation to 0.6% for the third follow-up. As seen in the last row of the table, a total of 537 completed web surveys were obtained, representing 4.5% of screened cell phone numbers and 5.3% of successful delivery of SMS invitations. The table also shows stability across the sample replicates over 7 weeks of data collection.

Response Rates

Table 2 shows the final dispositions and response rates for the two modes. Like the smartphone web mode mentioned above, the initial sample of 15,000 cell phone numbers was screened to remove nonworking or business numbers, yielding 11,420 active numbers. CATI call attempts were directed at these numbers over a 4-week period. This effort yielded a total of 968 completed cases or 8.5% of 11,420 numbers. This completion rate was almost double that of the smartphone web mode (537 cases, 4.5% completion rate). There were no partial interviews (answered at least 50% of questions) in either mode. Of those who started the web survey, 122 broke off before completing half of the questions, representing a break-off rate of 18.5% (122/659). A similar break-off rate was obtained for the CATI mode (208/1,176 = 17.7%).

Table 2.

Final Survey Dispositions and Response Rates by Mode.

	Smartphone Web Survey	Cell Phone CATI
Initial RDD sample size	15,900	15,000
Size (of active numbers) after screening	11,991 (100.0%)	11,420 (100.0%)
Completed interview ^a	537 (4.5%)	968 (8.5%)
Eligible, noninterview
Refusal		2,214 (19.4%)
Break-off or partial ^b	122 (1.0%)	208 (1.8%)
Noncontact		55 (0.5%)
Other, nonrefusals		949 (8.3%)
Unknown eligibility, noninterview
Always busy		46 (0.4%)
No response/no answer	9,402 (78.4%)	4,075 (35.7%)
Other		592 (5.2%)
Not eligible
Failed to deliver SMS ^c	1,849 (15.5%)
Business	4 (0.0%)	50 (0.4%)
Nonadult	73 (0.6%)	202 (1.8%)
Explicit opt out of SMS	4 (0.0%)
Other ^d		2,061 (18.0%)
Response rate (AAPOR RR1)	5.3%	10.6%

Note. Final dispositions based on AAPOR’s Standard Definitions, Version 9 (2016). RDD = random-digit-dialing; CATI = computer-assisted telephone interviewing; AAPOR = American Association for Public Opinion Research.

^a Answered more than 80% of the questions.

^b Answered fewer than 50% of the questions.

^c SMS delivery can fail if a phone number is incorrect, disconnected, or otherwise invalid or if the device is unable to receive SMS. This number is based on failures to deliver the initial invitation (not the follow-ups).

^d Includes disconnect, temporarily out of service, non-working, etc.

Despite the prescreening of cell phone numbers, a relatively large percentage (15.5%) resulted in failure to deliver the SMS invitation because the number was disconnected or invalid or the device was incapable of receiving text messages. This was larger than the percentage we anticipated. A similar percentage of cases in the CATI group (18.0%) turned out to be nonworking (or unassigned), temporarily out of service, or disconnected numbers. A larger percentage of cases was screened out as age ineligible in CATI (1.8%) than web (0.6%), largely because contact was made with a greater proportion of CATI numbers.

The response rate was 10.6% (using RR1; AAPOR, 2016) in the cell phone CATI mode compared with 5.3% in the smartphone web mode. These low rates are comparable to the RDD telephone survey rates in the United States (Keeter et al., 2017; Kennedy & Hartig, 2019).

Sample Representativeness

The first step to assess data quality was to identify whether respondents are representative of the adult general population. Many studies examine demographic variables such as geographical location of residence, gender, and age. But identifying geographical location using cell phone frames presents challenges (see Kim et al., 2017). In the United States, for example, telephone number portability has meant that state of residence can no longer reliably be determined from area codes. In South Korea, cell phone numbers share a single mobile prefix “010” instead of area codes. Thus, the geographical representativeness of the respondents needs to be determined based on self-reports. The survey included a question on geographic area of residence classified into 17 administrative divisions (seven metropolitan cities, one special autonomous city, and nine provinces) that have often been used for statistical purposes in South Korea.

Table 3 shows the respondent and population percentages for the three demographic variables (administrative divisions, gender, and age) in the two modes, the signed differences between the respondent and population percentages, and the p values for χ² tests of the differences in the respondent percentages between the two modes. Cities and provinces making up less than 5% of the population are combined in Table 3 for ease of presentation. For administrative divisions, we can see in both modes that most signed differences between the respondent and population percentages were comparatively small, and there was no statistically significant difference between the modes (p = .433). One exception is Seoul, which was overrepresented in both survey modes relative to the population. For gender, we can see that the smartphone web group was closer to the population value (difference of ±3.6% for web and ±9.8% for CATI), and the differences between modes were statistically significant (p < .05). With regard to age, both modes overrepresented the younger age-group (19–29) and underrepresented the older group (60+). But for the latter, the difference was larger for the smartphone web group (−19.2 percentage points) than for the CATI group (−10.4 percentage points), with the overall age distributions differing significantly (p < .001) between modes. One potential reason for this is because older adults may experience more difficulties using their smartphones and be less familiar with SMS (see Olmsted-Hawala et al., 2018). Another possible reason is undercoverage in smartphone ownership among this age-group. In 2018, only 59% of those aged 60 and older were reported to have a smartphone, relative to 97% for those under 60 (Korea Information Society Development Institute, 2019). In summary, despite the relatively low response rates (5.3% for smartphone web and 10.6% for cell phone CATI), respondents from both modes were generally geographically representative of the adult population, and the smartphone web mode was not markedly inferior to the cell phone CATI mode with respect to gender or age representativeness.

Table 3.

Demographic Comparison Between Respondent and Population Distributions.

Characteristics	Response Distributions % (Signed Difference)		p Value^a	Adult Population (%)
Characteristics	Smartphone Web Survey	Cell Phone CATI	p Value^a	Adult Population (%)
Administrative divisions ^b			.433
Eight cities
Seoul	24.6 (5.3)	27.4 (8.1)		19.3
Pusan	7.1 (0.4)	6.2 (−0.5)		6.7
Incheon	6.5 (0.8)	6.2 (0.5)		5.7
Five other cities	12.8 (−0.4)	12.2 (−1.0)		13.2
Nine provinces
Gyeonggi	25.5 (0.5)	25.6 (0.6)		25.0
Gyeongnam	5.8 (−0.6)	3.3 (−3.1)		6.4
Gyeongbuk	3.7 (−1.5)	3.6 (−1.6)		5.2
Six other provinces	14.0 (−4.5)	15.4 (−3.1)		18.5
Gender			.021**
Male	53.4 (3.6)	59.6 (9.8)		49.8
Female	46.6 (−3.6)	40.4 (−9.8)		50.2
Age groups			.000***
19–29	31.5 (13.7)	32.7 (14.9)		17.8
30–39	28.3 (11.0)	19.9 (2.6)		17.3
40–49	20.9 (1.3)	16.5 (−3.1)		19.6
50–59	13.0 (−6.8)	15.8 (−4.0)		19.8
60 or older	6.3 (−19.2)	15.1 (−10.4)		25.5

Note. The signed difference indicates the difference between the percentage in each mode and the population percentage. The percentage of the adult population (19 years of age or older) in each administrative division comes from the 2018 Population and Housing Census and is provided by the Korean Statistical Information Service (http://kosis.kr/index/index.do). CATI = computer-assisted telephone interviewing.

^a p Values in χ² tests for differences between modes.

^b South Korea is made up of 17 administrative divisions: seven metropolitan cities, one special autonomous city, and nine provinces. The adult population size is 42,932,956 in 2018. Seoul is the capital city, and its adult population is 8,277,128. The largest province is Gyeonggi, and its adult population is 10,708,858. The five smallest cities and six smallest provinces are combined for ease of presentation.

**p value < .05. ***p value < .01.

Responses to Questions Asked of All Respondents

The next step was to look at differences in demographic and substantive questions by mode. We examined both unweighted and weighted estimates. Poststratification weights were calculated using the three domains (administrative divisions, gender, and age groups) in Table 3. There were 10 poststrata within each administrative division. Small poststrata were combined with neighboring ones to meet minimum size requirements (see Kim et al., 2007, p. 145). The design effects due to unequal weighting (Kish, 1992) were 2.29 and 1.62 for the smartphone web mode and the cell phone CATI mode, respectively.

Tables 4 –6 present the unweighted and weighted estimates for 23 of the remaining 27 questions asked of all respondents (excluded are two screening questions and two open-ended questions). Table 4 presents differences between the two modes in responses to five demographic questions whose population percentages were not available from the 2018 Population and Housing Census, Table 5 focuses on twelve behavioral questions, and Table 6 presents differences for six attitudinal questions. The significance tests for the unweighted estimates are based on Pearson’s χ² tests, whereas for the weighted estimates, they are based on Rao and Scott’s (1987) χ² tests.

Table 4.

Demographic Questions Asked of All Respondents: Differences in Unweighted and Weighted Estimates Between Modes.

Questions	Unweighted Estimates Within Mode (%)		p Value^a	Weighted Estimates Within Mode (%)		p Value^a
Questions	SP Web	CP CATI	p Value^a	SP Web	CP CATI	p Value^a
Highest level of educational attainment			.000***			.000***
a. High school graduate or less	30.9	40.0		31.1	41.8
b. Two-year or 3-year bachelor’s degree	17.9	14.9		17.8	15.6
c. Four-year undergraduate degree	39.1	38.6		36.1	36.1
d. Postgraduate degree	12.1	6.5		15.0	6.5
Number of people who live together, including themselves			.011**			.004***
a. 1	10.4	16.0		6.8	14.3
b. 2	21.2	23.6		27.3	26.8
c. 3	24.4	23.0		23.9	22.1
d. 4	33.5	27.5		33.7	25.8
e. 5 or more	10.4	9.8		8.2	11.0
Marital status			.201			.021**
a. Single	38.8	43.3		24.1	31.3
b. Currently married (married, common law marriage, domestic partnership)	56.6	53.0		67.2	64.2
c. Formerly married (divorced, separated, widowed)	4.7	3.8		8.7	4.5
Did you work for a living at least 1 hr last week?			.181			.095*
a. Yes	78.0	74.9		78.8	73.0
b. No	22.0	25.1		21.2	27.0
Average personal income per month before taxes over the last year (Korean Won)			.137			.200
a. No income	14.5	18.5		12.7	18.6
b. Less than 1,000,000	10.2	13.3		10.5	14.4
c. 1,000,000–less than 2,000,000	17.9	17.0		18.2	17.2
d. 2,000,000–less than 3,000,000	20.3	19.0		21.1	17.2
e. 3,000,000–less than 4,000,000	14.0	13.0		13.1	12.1
f. 4,000,000–less than 5,000,000	8.8	8.3		10.2	9.3
g. 5,000,000 or more	14.3	10.9		14.2	11.2

Note. 1,000,000 Korean Won (KRW) = $870 (exchange rate, approximately US$1 = 1,150 KRW). CATI = computer-assisted telephone interviewing; SP web = smartphone web; CP CATI = cell phone CATI.

^a p Values in χ² tests for differences between modes. For the weighted estimates, the p values from Rao–Scott χ² tests are given.

*p value < .10. **p value < .05. ***p value < .01.

Table 5.

Behavioral Questions Asked of All Respondents: Differences in Unweighted and Weighted Estimates Between Modes.

Questions	Unweighted Estimates Within Mode (%)		p Value^a	Weighted Estimates Within Mode (%)		p Value^a
Questions	SP Web	CP CATI	p Value^a	SP Web	CP CATI	p Value^a
Have you ever smoked cigarettes in your lifetime? (% yes)	59.2	51.7	.005***	55.4	45.2	.009***
Have you ever seen the information on chemicals (e.g., nicotine, tar) on cigarette packages? (% yes)	84.7	68.3	.000***	78.7	70.2	.042**
Have you ever seen a list of substances other than nicotine or tar on cigarette packages? (% yes)	46.2	22.7	.000***	42.3	19.7	.000***
Have you ever heard about roll-your-own cigarettes? (% yes)	70.6	63.2	.004***	69.8	61.5	.031**
Have you ever heard about cigars? (% yes)	88.6	70.5	.000***	89.0	64.1	.000***
Have you ever heard about pipe tobacco? (% yes)	86.0	61.8	.000***	88.9	61.1	.000***
Have you ever heard about water-pipe tobacco?(% yes)	57.4	47.6	.000***	56.0	39.6	.000***
Have you ever heard about snuff tobacco? (% yes)	5.0	8.9	.007***	5.4	9.4	.050*
Have you ever heard about dipping tobacco? (% yes)	5.0	7.5	.061*	4.1	7.6	.027**
Have you ever heard about liquid electronic cigarettes? (% yes)	73.4	62.9	.000***	62.6	56.3	.121
Have you ever heard about electronic cigarettes such as IQOS, lil, and glo? (% yes)	67.2	70.1	.249	64.2	67.1	.445
Have you ever heard about chewing tobacco? (% yes)	37.2	38.9	.515	41.2	37.2	.291

Note. CATI = computer-assisted telephone interviewing; SP web = smartphone web; CP CATI = cell phone CATI.

^a p Values in χ² tests for differences between modes. For the weighted estimates, the p values from Rao–Scott χ² tests are given.

*p value < .10. ** p value < .05. ***p value < .01.

Table 6.

Attitudinal Questions Asked of All Respondents: Differences in Unweighted and Weighted Estimates Between Modes.

Questions	Unweighted Estimates Within Mode (%)		p Value^a	Weighted Estimates Within Mode (%)		p Value^a
Questions	SP Web	CP CATI	p Value^a	SP Web	CP CATI	p Value^a
Cigarette packages are currently required to display the content of nicotine and tar. Do you think that the content of harmful substances other than those is needed to display on cigarette packages?			.000***			.000***
a. Yes	86.0	68.3		86.8	70.2
b. No	14.0	31.7		13.2	29.8
On a 1–7 scale anchored by not at all harmful at 1 and very harmful at 7, how harmful do you think cigarette smoking is to a person’s health?			.009***			.007***
a. 4 or less	11.9	8.4		9.4	8.1
b. 5	10.2	14.9		8.5	14.1
c. 6	18.8	16.0		20.4	13.5
d. 7	59.0	60.6		61.8	64.2
On a 1–7 scale anchored by not at all harmful at 1 and very harmful at 7, how harmful do you think any flavored cigarettes are to a person’s health?			.000***			.006***
a. 4 or less	17.9	10.7		17.0	10.6
b. 5	11.7	14.9		11.3	14.9
c. 6	17.7	15.4		19.0	13.4
d. 7	52.7	59.0		52.8	61.2
On a 1–7 scale anchored by not at all addictive at 1 and very addictive at 7, how addictive do you think cigarette smoking is to a person’s health?			.002***			.101
a. 4 or less	19.0	18.5		16.4	16.9
b. 5	16.9	24.0		15.2	21.4
c. 6	22.3	16.2		20.1	15.8
d. 7	41.7	41.3		48.3	45.9
On a 1–7 scale anchored by not at all addictive at 1 and very addictive at 7, how addictive do you think any flavored cigarettes are to a person’s health?			.000***			.001***
a. 4 or less	38.0	27.7		37.8	26.0
b. 5	14.5	21.7		12.7	20.4
c. 6	17.9	15.6		18.1	15.3
d. 7	29.6	35.0		31.4	38.3
Self-assessed health status			.110			.403
a. Excellent	18.6	20.0		14.4	17.0
b. Very good	50.5	44.0		49.9	43.3
c. Good	25.7	29.8		28.9	31.6
d. Fair or poor	5.2	6.2		6.7	8.0

Note. CATI = computer-assisted telephone interviewing; SP web = smartphone web; CP CATI = cell phone CATI.

^a p Values from χ² tests for differences between modes. For the weighted estimates, the p values from Rao–Scott χ² tests are given.

***p value < .01.

As shown in Table 4, while four demographic questions (education, number of people living together, marital status, employment status) showed significant differences (p < .10) between the two modes for both unweighted or weighted estimates, income did not differ significantly by mode.

Table 5 shows that most behavioral questions (experience with cigarettes or tobacco and cigarette packages) had significant differences (p < .10) between the two modes for both unweighted and weighted estimates. For only two of the questions were the mode differences not significant. In general, respondents in the web mode reported significantly higher rates of cigarette and tobacco use than CATI respondents. For example, the unweighted estimate for lifetime smoking was 7.5 percentage points higher for web than CATI, while the weighted estimate was 10.2 percentage points higher. This is consistent with research finding differences in reporting of socially undesirable behaviors by mode (see Kreuter et al., 2008; Tourangeau et al., 2000; Tourangeau & Yan, 2007). The fact that the weighted estimates (which account for key demographic differences in response to the surveys) still show differences supports the above interpretation.

Table 6 also shows that most attitudinal questions (harm and addiction beliefs about cigarette use) showed significant differences (p < .01) between the two modes for both unweighted and weighted estimates, whereas only self-assessed health status showed no significant difference between modes. For example, the first question on displaying other harmful substances on packaging showed a much higher agreement rate for web than CATI (an unweighted difference of 17.7 percentage points and a weighted difference of 16.6 percentage points).

Item Nonresponse

Excluding one voluntary item (personal opinion on tobacco control policy), the questionnaire included a total of 118 questions. Item nonresponse was very rare in the smartphone web group because the survey was designed so that respondents could not proceed without answering, although they could go back and change responses to earlier questions, and could suspend and resume the survey at their convenience.

In contrast, “don’t know” responses or refusals were allowed in the cell phone CATI group, although they were not offered as explicit response choices. About a third of the questions (35 of 118) had any item nonresponse. Of these, 60% (21 of 35 questions) had fewer than 10 missing responses. Four questions, one on income (with 73 missing) and three on harm and beliefs about cigarette use addiction (with 97, 108, and 114 missing, respectively) had higher rates.

Response to Open-Ended Questions

There were 10 open-ended numeric (frequency) questions. With the exception of age, the rest were follow-up questions asked of those who answered “yes” to filter questions. Table 7 shows the comparison of the unweighted and weighted estimates (both means and medians) between the two modes for the five follow-up questions where the minimum number of respondents exceeded 20. The differences between smartphone web and cell phone CATI smartphone in each question were small and did not reach statistical significance for either unweighted or weighted average estimates.

Table 7.

Comparison of Responses to Frequency Questions Between Modes.

Follow-Up Questions	Average Estimates		p Value^a
	Unweighted/Weighted		p Value^a
	Smartphone Web(Median)	Cell Phone CATI(Median)	Unweighted/Weighted
Average number of cigarettes smoked per day (asked of respondents who now smoke cigarettes every day)	14.4/15.5(15)n = 126	15.8/15.9(15)n = 200	0.102/0.701
Number of the past 30 days smoked cigarettes (asked of respondents who now smoke cigarettes some days)	14.4/14.4(15)n = 33	15.9/15.6(15)n = 35	0.517/0.655
Average number of cigarettes smoked on the day smoked (asked of respondents who now smoke cigarettes some days)	4.6/4.2(4)n = 33	4.9/5.1(5)n = 35	0.686/0.240
Months since completely stopping smoking cigarettes (asked of respondents who no longer smoke cigarettes)	104.7/130.2(73)n = 79	107.9/130.0(85)n = 198	0.774/0.971
Average number of electronic cigarettes smoked per day (asked of respondents who now smoke electronic cigarettes every day)	11.8/12.2(10)n = 28	11.1/10.4(10)n = 24	0.479/0.274

Note. “n” denotes the number of respondents to each question. CATI = computer-assisted telephone interviewing.

^a p Value in t test for difference of average estimates between modes.

Completion Times

In order to appropriately compare the completion times between the modes, we divided respondents into two groups, smokers (those who have smoked at least 100 cigarettes in their life and now smoke at least on some days) and nonsmokers. There were no outliers in the CATI mode, but 29 respondents (5%) in the web mode took longer than an hour to complete the survey. After removing these outliers, smokers took an average of 12.6 min (median of 10.5) for the smartphone web mode (n = 145) and 14.6 min (median of 14.0) for the cell phone CATI mode (n = 235), a significant difference (p < .05). Nonsmokers took an average of 9.9 min (median of 7.4) for the smartphone web mode (n = 363) and 10.8 min (median of 11.0) for the cell phone CATI mode (n = 733), with the difference also being significant (p < .05). In both groups, CATI took slightly longer than web.

Comparison With Official Statistics

The 2018 Korea National Health and Nutrition Examination Survey (KNHANES), conducted by the Korea Centers for Disease Control and Prevention, was a nationwide survey examining the general health and nutrition status of the Korean people. Cigarette smoking prevalence was among the topics covered by the KNHANES, and these topics are administered using computer-assisted self-interviewing as part of the computer-assisted personal interviews. This national survey collected data from approximately 10,000 individuals who were selected using multistage cluster sampling, which tends to have a lower efficiency (e.g., a larger standard error or variance or design effect) compared to a single-stage EPSEM (e.g., simple random sampling) for the same sample size (see Kish, 1995, Section 5.4). The sample weights were constructed for the KNHANES respondents to represent the Korean population. Although both the sample sizes and the modes were different between the KNHANES and the surveys in our study, it is still meaningful to compare national smoking prevalence estimates, based on the two key questions that asked whether the respondents have smoked at least 100 cigarettes in their lifetime and whether they now smoked either daily or on some days. In the 2018 KNHANES, the published prevalence of current cigarette smoking among adults was 22.4% with the standard error (SE) of 0.8% (Korea Statistical Information Service, 2020). The corresponding estimates were calculated for the two modes in our study. The weighted estimate in the smartphone web mode was 25.0% (SE = 2.40%), while in the cell phone CATI mode, it was 20.6% (SE = 1.50%). The larger standard errors reflect the smaller sample sizes in our study. The estimate in the smartphone web mode (self-administered) was slightly higher than in the KNHANES (self-administered in the presence of an interviewer), while that in the cell phone mode (interviewer-administered) was slightly lower than in the KNHANES.

Survey Costs

Cost is an important component in comparing the two modes. The data collection cost per completed case for each mode was calculated as follows. The cell phone CATI survey was conducted at the SHPRC, which has a centralized telephone facility. To screen business or nonworking numbers from 15,000 sample cell phone numbers, four trained operators spent a total of 109 hr at a rate of 7,530 KRW ($6.55 at the exchange rate of 1,150 KRW to USD at the time of the study) per hour, resulting in a total cost of 820,770 KRW (about $714). During the 4 weeks of data collection, 10 telephone interviewers worked a total of 551 hr, after receiving a total of 236 hr of training for CATI. The cost per interviewer was 7,650 KRW per hour for training and 8,254 KRW per hour for interviewing, yielding a total interviewing cost of 6,353,354 KRW ($5,525). Two supervisors worked a total of 697 hr at a rate of 17,000 KRW per hour, and the total cost was 11,849,000 KRW ($10,303). A total of 5,760,000 KRW ($5,009) was paid for CATI programming from a total of 320 hr at a rate of 18,000 KRW per hour. The total labor cost for screening, training, interviewing, supervision, and programming was 24,783,124 KRW ($21,551). The charge for cell phone calls was 2,006,250 KSW ($1,745). The cost of incentives provided to respondents was 940,000 KRW ($817). The cost for operation and maintenance of the survey was 2,220,000 KRW ($1,930). Thus, the total cost for the cell phone CATI was 29,949,374 KRW ($26,043), coming to about 30,939 KRW ($26.9) per completed case for 968 completed interviews. It is noted that this total does not include other costs for sampling, questionnaire design, and data analysis, assumed to be equivalent across the two modes.

In the smartphone web survey, business or nonworking numbers were screened from 15,900 sample cell phone numbers by four trained operators who worked a total of 115 hr at the same rate, for a total cost of 865,950 KRW ($753). A total of 38,598 SMS text message invitations were delivered using a commercial service, at a total cost of 1,003,548 KRW ($873). The cost per SMS message was 26 KRW ($0.0226). Two web survey specialists worked a total of 126 hr at a rate of 18,000 KRW ($15.65) per hour, yielding a total cost of 2,268,000 KRW ($1,972). The cost of incentives was 1,388,000 KRW ($1,207). The cost for operation and maintenance including software license during the survey period of 7 weeks was 300,000 KRW ($261). The total cost for the smartphone web survey was about 5,825,498 KRW ($5,066). The cost per completed case was 10,848 KRW ($9.43) for 537 completed interviews. As a result, the smartphone web survey was conducted at about one fifth of the total cost for the cell phone CATI or about one third the cost per completed case. Increasing the sample size of the smartphone web survey to yield an equivalent number of completed cases to the cell phone CATI survey (from 537 to 968) would yield an estimated cost of 10,500,864 KRW (about $9,131), still significantly less than the total cost of the cell phone CATI survey (29,949,374 KRW or $26,043). Put another way, spending the same amount on the smartphone web survey as was spent on the cell phone CATI survey, the number of completed cases would increase from 1,505 to about 2,760 (assuming the same per complete web survey costs).

Discussion

Web surveys offer a great deal potential for survey researchers, and this is only likely to increase over time. There is no doubt that sampling or coverage problems have presented big challenges for web surveys of the general population in the past. Meanwhile, survey costs for RDD telephone surveys in many countries have increased due to the difficulty in contacting sample numbers on the phone and to dramatically lower participation rates. South Korea is no exception. The costs for national surveys (whether telephone or face-to-face) conducted at the SHPRC have been steadily increasing while response rate has been declining. This raises the question whether or not we are using the best ways to collect survey data from the general public, in terms of both to cost-effectiveness and accuracy.

The advantages of self-administered surveys conducted over the Internet are well-known: no interviewer effects, faster data collection, respondent convenience, and lower social desirability bias, yielding more accurate data on sensitive topics. Based on those advantages, researchers are likely to reduce costs and improve data quality if the sampling and coverage problems facing web surveys of the general population can be overcome. While initially viewed with concern, smartphones now provide opportunities to address these challenges without compromising data quality (see Couper et al., 2017).

We have demonstrated the feasibility of conducting a national RDD smartphone web survey using SMS text message invitations to address sampling or coverage problems in collecting data from the general population in South Korea, which has a wireless sampling frame with very high coverage of adult population and permits unsolicited SMS for noncommercial research purposes. This suggests promise for countries with similar conditions. In countries where unsolicited text messages are prohibited, other methods need to be found to sample and recruit persons for surveys completed online. But for South Korea, at least, a smartphone web survey based on an RDD sample with SMS invitations provides a viable alternative as a stand-alone or primary mode to traditional RDD telephone surveys, if carefully designed and implemented. Also, if a further study could be conducted to achieve a larger number of respondents in a smartphone web survey, more detailed subgroup analyses could be undertaken to explore the reasons for the observed mode effects more fully.

Footnotes

Authors’ Note

The authors thank Junbeom Ryu for data analysis, Kihyun Um and Youngje Woo for data collection, and many others including Seha Park and Gisung Cho for their assistance.

Data Availability

The anonymized data are available from the first author at the given email address.

Declaration of Conflicting Interests

The authors declared no conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The SHPRC’s own funds were used for this study.

Software Information

The analyses were conducted using SPSS Version 23 and SAS Version 9.4.

References

American Association for Public Opinion Research. (2014). Mobile technologies for conducting, augmenting and potentially replacing surveys: Report of the AAPOR task force on emerging technologies in public opinion research. https://targeting.alnap.org/system/files/content/resource/files/main/REVISED_Mobile_Technology_Report_Final_revised10June14.pdf

American Association for Public Opinion Research. (2016). Standard definitions: Final dispositions of case codes and outcome rates for surveys (9th ed.). https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf

American Association for Public Opinion Research. (2019). Report of the AAPOR task force on transitions from telephone surveys to self-administered and mixed-mode surveys. https://www.aapor.org/getattachment/Education-Resources/Reports/AAPOR-Mixed-Mode-Task-Force-Executive-Summary-FINAL.pdf.aspx

Bosnjak

Neubarth

Couper

M. P.

Bandilla

Kaczmirek

(2008). Prenotification in web-based access panel surveys: The influence of mobile text messaging versus e-mail on response rates and sample composition. Social Science Computer Review, 26, 213–223.

Conrad

F. G.

Schober

M. F.

Antoun

Yan

H. Y.

Hupp

A. L.

Johnston

Ehlen

Vickers

Zhang

(2017). Respondents mode choice in a smartphone survey. Public Opinion Quarterly, 81, 307–337.

Couper

M. P.

(2017). New developments in survey data collection. In Cook

K. S.

Massey

D. S.

(Eds.), Annual review of sociology (vol. 43, pp. 121–145). Annual Reviews.

Couper

M. P.

Antoun

Mavletova

(2017). Mobile web surveys: A total survey error perspective. In Biemer

Eckman

Edwards

de Leeuw

Kreuter

Lyberg

Tucker

West

(Eds.), Total survey error in practice (pp. 133–154). Wiley.

Couper

M. P.

Gremel

Axinn

Guyer

Wagner

West

B. T.

(2018). New options for national population surveys: The implications of internet and smartphone coverage. Social Science Research, 73, 221–235.

Dal Grande

Chittleborough

C. R.

Campostrini

Dollard

Taylor

A. W.

(2016). Pre-survey text messages (SMS) improve participation rate in an Australian mobile telephone survey: An experimental study. PLoS ONE, 11, e0150231.

10.

De Bruijne

Wijnant

(2014). Improving response rates and questionnaire design for mobile web surveys. Public Opinion Quarterly, 78, 951–962.

11.

Elevelt

Bernasco

Lugtig

Ruiter

B. M. S.

Toepoel

(2019). Where you at? Using GPS locations in an electronic time use diary study to derive functional locations. Social Science Computer Review. Advance online publication. https://doi.org/10.1177/0894439319877872

12.

Groves

R. M.

Harris-Kojetin

B. A.

(2017). Innovations in federal statistics: Combining data sources while protecting privacy. The National Academies Press.

13.

Gummer

Quoß

Roßmann

(2018). Does increasing mobile device coverage reduce heterogeneity in completing web surveys on smartphones? Social Science Computer Review, 37, 371–384.

14.

Heffner

J. L.

Mull

K. E.

(2017). Smartphone ownership among US adult cigarette smokers: 2014 Health Information National Trends Survey (HINTS) data. Journal of Medical Internet Research, 19(8), e305. https://doi.org/10.2196/jmir.7953

15.

Hoe

N. D.

Grunwald

H. E.

(2015). The role of automated SMS text messaging in survey research. Survey Practice, 8. https://www.surveypractice.org/article/2829.pdf

16.

Holland

Couper

M. P.

Schroeder

H. M.

(2014, May). Pre-notification strategies for mixed-mode data collection [Paper presentation]. Annual Meeting of the American Association for Public Opinion Research, Anaheim, CA, USA.

17.

Keeter

Hatley

Kennedy

Lau

(2017). What low response rates mean for telephone surveys. Pew Research Center. https://www.pewresearch.org/methods/2017/05/15/what-low-response-rates-mean-for-telephone-surveys/

18.

Kennedy

Hartig

(2019). Response rates in telephone surveys have resumed their decline. Pew Research Center. https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/

19.

Kennedy

Mcgeeney

Keeter

Patten

Perrin

Lee

Best

(2018). Implications of moving public opinion surveys to a single-frame cell-phone random-digit-dial design. Public Opinion Quarterly, 82, 279–299.

20.

Kim

J. J.

Valliant

(2007). Cell collapsing in poststratification. Survey Methodology, 33, 139–150.

21.

Kim

S. W.

Lee

S. K.

Hong

S. J.

Park

S. H.

(2012). List-assisted RDD sampling in Korea: Testing the feasibility of national survey under within-household selection. International Journal of Public Opinion Research, 24, 79–92.

22.

Kim

S. W.

Traugott

M. W.

Kwak

N. J.

Choi

E. H.

Lee

H. N.

(2014, September 26). Using two-wave dual frame RDD telephone pre-election poll in the 2012 Korean presidential election [Paper presentation]. Sixty-Seventh Annual Conference of the World Association for Public Opinion Research, Nice, France.

23.

Kim

S. W.

Woo

Y. J.

Kim

N. H.

(2017). Assuring quality in dual frame RDD national or sub-national surveys using cell phone numbers without area codes in South Korea [Paper presentation]. Joint Statistical Meetings, Baltimore, MD, USA.

24.

Kish

(1992). Weighting for unequal Pi. Journal of Official Statistics, 8, 183–200.

25.

Kish

(1995). Survey sampling. Wiley.

26.

Korea Communications Commission. (2017). Guidelines for preventing spam. https://spam.kisa.or.kr/customer/sub2_R.do?boardNo=1004

27.

Korea Information Society Development Institute. (2019). Research on digital divide. KISDI STAT report Vol. 19–22. http://m.kisdi.re.kr/mobile/repo/stat_view.m?key1=14651&selectPage=1

28.

Korea Statistical Information Service. (2020). Korea National Health and Nutrition Examination Survey. http://kosis.kr/statisticsList/statisticsListIndex.do?menuId=M_01_01&vwcd=MT_ZTITLE&parmTabId=M_01_01&statId=1973001&themaId=D#117_11702.3

29.

Kreuter

Haas

G.-C.

Keusch

Bähr

Trappmann

(2020). Collecting survey and smartphone sensor data with an app: Opportunities and challenges around privacy and informed consent. Social Science Computer Review, 38, 533–549. https://doi.org/10.1177/0894439318816389

30.

Kreuter

Presser

Tourangeau

(2008). Social desirability bias in CATI, IVR, and web surveys. Public Opinion Quarterly, 72, 847–865.

31.

Lau

C. Q.

Lombaard

Baker

Eyerman

Thalji

(2019). How representative are SMS surveys in Africa? Experimental evidence from four countries. International Journal of Public Opinion Research, 31, 309–330.

32.

Lee

H. N.

Kim

S. W.

Couper

M. P.

Woo

Y. J.

(2019). Experimental comparison of PC web, smartphone web and telephone surveys in the new technology era. Social Science Computer Review, 37, 234–247.

33.

Lepkowski

J. M.

Kim

S. W.

Steeh

(2005). Dual-frame landline/cellular telephone survey design [Paper presentation]. Invited Session of the Survey Research Methods Section, American Statistical Association, Minneapolis, MN, USA.

34.

Marketing Systems Group (2020). Cellular working identification number service (CELL-WINS). http://www.m-s-g.com/CMS/ServerGallery/MSGWebNew/Documents/GENESYS/whitepapers/Cell-WINS.pdf.

35.

Marlar

Hoover

(2019, June 20). Leveraging SMS for survey research [Webinar]. AAPOR.

36.

Marlar

McGeeney

Chattopadhyay

(2014, May 15–18). SMS surveys: Testing multiple modes to reach respondents from a wireless frame. Annual Meeting of the American Association for Public Opinion Research, Anaheim, CA, USA.

37.

Mavletova

Couper

M. P.

(2014). Mobile web survey design: Scrolling versus paging, SMS versus e-mail invitations. Journal of Survey Statistics and Methodology, 2, 498–518.

38.

Olmsted-Hawala

Nichols

Falcone

Figueroa

Antoun

Wang

(2018). Optimal data entry designs in mobile web surveys for older adults. Proceedings of the 4th International Conference, ITAP 2018, held as part of HCI International 2018, Las Vegas, NV, USA.

39.

Opensignal. (2017). Global state of mobile networks (February 2017). https://www.opensignal.com/reports/2017/02/global-state-of-the-mobile-network

40.

Park

S. H.

Lee

Kim

S. W.

Lee

S. K.

(2012, June 14–16). A comparison of response patterns between landline and cell phone RDD surveys. Sixty-Fifth Annual Conference of the World Association for Public Opinion Research, Hong Kong.

41.

Peterson

Griffin

LaFrance

(2017). Smartphone participation in web surveys: Choosing between the potential for coverage, nonresponse, and measurement error. In Biemer

Eckman

Edwards

de Leeuw

Kreuter

Lyberg

Tucker

West

(Eds.), Total survey error in practice (pp. 203–233). Wiley.

42.

Pew Research Center. (2019). Smartphone ownership is growing rapidly around the world, but not always equally. https://www.pewresearch.org/global/2019/02/05/smartphone-ownership-is-growing-rapidly-around-the-world-but-not-always-equally/

43.

Peytchev

Neely

(2013). RDD telephone surveys: Toward a single-frame cell-phone design. Public Opinion Quarterly, 77, 283–304.

44.

Rao

J. N. K.

Scott

A. J.

(1987). On simple adjustments to chi-square tests with survey data. Annals of Statistics, 15, 385–397.

45.

Respi

Sala

(2017). Personalized SMS, survey participation and data quality—The Italian case. Bulletin de Méthodologie Sociologique, 136, 21–39.

46.

Revilla

Couper

M. P.

(2019). Improving the use of voice recording in a smartphone survey. Social Science Computer Review. Advance online publication. https://doi.org/10.1177/0894439319888708

47.

Schober

M. F.

Conrad

F. G.

Antoun

Ehlen

Fail

Hupp

A. L.

Johnston

Vickers

Yan

H. Y.

Zhang

(2015). Precision and disclosure in text and voice interviews on smartphones. PLoS ONE, 10, https://doi.org/10.1371/journal.pone.0128337

48.

Schonlau

Couper

M. P.

(2017). Options for conducting web surveys. Statistical Science, 32, 279–292.

49.

Smith

(2015). U.S. smartphone use in 2015. Pew Research Center. http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/

50.

Statistics Korea. (2018). South Korea mobile phone subscribers. http://kosis.kr/statHtml/statHtml.do?orgId=101&tblId=DT_2KAAA10

51.

Steeh

Buskirk

T. D.

Callegaro

(2007). Using text messages in U.S. mobile phone surveys. Field Methods, 19, 59–75.

52.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge University Press.

53.

Tourangeau

Yan

(2007). Sensitive questions in surveys. Psychological Bulletin, 133, 859–883.

54.

Van der Heijden

(2017). The practicalities of SMS research. International Journal of Market Research, 59, 157–172.

55.

West

B. T.

Ghimire

Axinn

W. G.

(2015). Evaluating a modular design approach to collecting survey data using text messages. Survey Research Methods, 9, 111–123.

56.

Woo

Y. J.

Kim

S. W.

Couper

M. P.

(2015). Comparing a cell phone survey and a web survey of university students. Social Science Computer Review, 33, 399–410.