A National RDD Smartphone Web Survey: Comparison With a Large-Scale CAPI Survey

Abstract

The most important national surveys of the general population for creating official statistics or public policymaking in many countries, including South Korea, are still conducted using face-to-face interviews with household members. Recently face-to-face surveys have faced threats to data quality from decreasing response rates and rising costs of in-person visits. The COVID-19 pandemic and associated lockdown exacerbated the situation for face-to-face surveys. Survey organizations suspended fieldwork or began to explore alternate means of collecting data. One alternative was a shift to telephone surveys; however, telephone interviews have encountered similar difficulties with declining response rates and increasing costs. Could a self-administered web survey be a viable alternative to interviewer-administered modes such as telephone interviews (CATI) or face-to-face interviews (CAPI)? Smartphones may offer opportunities not offered by other modes. We conducted a smartphone web survey using SMS invitations where a sample of cell phone numbers was selected by random digit dialing (RDD) and compared it with a large-scale national face-to-face survey (CAPI) where a sample of households was selected by stratified cluster sampling. The two surveys were conducted during the COVID pandemic in the second half of 2020. The coverage and sample representation of the smartphone web survey were comparable to that of the face-to-face survey. Despite the relatively small number of respondents, the quality of the smartphone web survey was sufficient to provide accurate data and compared favorably with the CAPI survey. The smartphone web survey yielded more reports of socially undesirable attitudes and behavior than the CAPI survey. The findings will guide researchers to explore new opportunities in establishing a web survey methodology that obtains data more conveniently, efficiently, accurately, and with less cost.

Keywords

smartphone web survey short message service text messages single frame random-digit-dialing design computer-assisted personal interviewing survey COVID-19 pandemic response rates,sample representativeness accuracy social desirability survey costs

Introduction

The last half-century has seen the rapid rise and subsequent decline of telephone surveys. A key attraction of telephone surveys was being able to use random digit dialing (RDD) methods, which provide a sample of households or persons through a random selection of telephone numbers, and to conduct inexpensive interviews via telephone. Given the recent precipitous decline in telephone survey response rates, and the rising costs of face-to-face data collection, researchers have been looking for alternative ways to conduct surveys among randomly selected samples representative of national populations (AAPOR, 2019). Web surveys offer a cost-effective alternative to interviewer-administered surveys, but no list or procedures exist for sampling directly into web surveys; alternatives such as address-based sampling (ABS) are needed. The recent COVID-19 pandemic and lockdown accelerated the need to find alternative methods for sampling and surveying general populations.

The rapid rise in the use of smartphones worldwide offers a potential solution. RDD methods could be used to sample mobile telephone numbers and invite users via SMS (short message service) or text message to complete a survey online. To be effective, the selection properties of the proposed solution need to be similar to those of the face-to-face “gold standard” (i.e., coverage bias should be minimized) and the measurement properties of the two approaches should be similar (i.e., mode effects should be minimized). In this paper we explore a smartphone web survey using RDD sampling and an invitation sent as a text message via SMS (i.e., an SMS invitation) to compare data quality and estimates to a traditional face-to-face survey in South Korea, focusing on health.

Background and Relevant Literature

We review the literature and related evidence regarding two features necessary for a smartphone web survey to be effective. First, we look at smartphone coverage (a necessary requirement for RDD sampling to be effective) and the use of SMS as a mode of invitation to a web survey (a necessary requirement to reach sample persons for a web survey). Second, we look at relevant mode comparisons of web and face-to-face surveys, focusing on probability samples of the general population.

RDD Sampling of Mobile Phone Numbers and SMS Invitations

Several features are necessary for this approach to be effective. First, there needs to be sufficiently high coverage of the population using mobile phones and low coverage bias. The Pew Research Center (2019) estimated that 95% of adults in South Korea owned a smartphone in 2018, compared to a median of 76% across 18 advanced economies (e.g., Netherlands 87%, U.S. 81%, and UK 76%). Thus, not only is mobile phone coverage relatively high (paralleling the heyday of RDD telephone surveys in the U.S.; see Groves et al., 1988; Lepkowski et al., 2007), but the vast majority of these are smartphones, enabling users to access the Internet and complete web surveys. The coverage of subpopulations using smartphones in South Korea is also very high. For example, smartphone ownership by 10-year age groups was estimated to be 99.6% for 20s, 99.5% for 30s, 99.5% for 40s, 99.9% for 50s, 93.1% for 60s, and 53.8% for 70 or above (National Information Society Agency, 2021).

Second, a mechanism is needed for sampling mobile phone numbers. Since Kim and Lepkowski (2002) reported on the rise of cell phone-only households and the decline of landlines, the Survey and Health Policy Research Center (SHPRC) at Dongguk University established its own dual frame (landline and cell phone RDD) and has used it for national telephone surveys on a variety of topics (see e.g., Kim et al., 2012; Kim et al., 2014; Lepkowski et al., 2005; Park et al., 2012). A study by Kim et al. (2017) found that only 3% of adults in South Korea were landline-only, almost all adults (97%) owned a cell phone (smartphone 91% and feature phone 6%), and cell phone RDD samples were much more demographically representative than landline RDD samples. Subsequently, SHPRC dropped the landline RDD frame and adopted a cell phone RDD frame only. Similar trends toward the adoption of a single cell phone frame have occurred elsewhere (see Kennedy et al., 2018; Peytchev & Neely, 2013). As the AAPOR task force (AAPOR, 2017, p. 2) noted “For most surveys in the future that use a telephone frame to reach the general population of the United States, the cell phone RDD frame will be sufficient because it will not lead to a meaningful amount of unit-level coverage error.”

Third, text messaging or SMS is sufficiently widespread to be an accepted method of survey invitation. Text messaging in South Korea is a popular feature (see National Information Society Agency, 2021): 97.0% of South Koreans in 2020 used text messaging via smartphone or tablet. Similar high rates of use occur in the U.S. (see Smith, 2015) and elsewhere. Further, findings regarding the use of text messages to invite individuals could differ depending on whether the recipient or sender pays for the message (Andreadis, 2020). In South Korea, the cost of SMS is borne by the sender.

A final potential barrier to the use of SMS is legal restrictions on unsolicited text messages. In the U.S., the Telephone Consumer Protection Act (TCPA) prohibits text messages without the recipient’s prior expressed consent (see Dayton et al., 2020; Marlar & Hoover, 2019). In Europe, the General Data Protection Regulation (GDPR) prohibits SMS without prior consent for commercial purposes (see European Commission, 2022). While interpretations vary across countries, non-commercial actors such as researchers are often exempt from such restrictions (see Andreadis, 2020; Kongsgard et al., 2014). In South Korea, unsolicited text messages without prior consent are proscribed by the Information and Communication Network Act; however, this only applies to advertising information for commercial use. Text messages for non-commercial use by public offices or non-profit organizations are exempt (see Korea Communications Commission, 2020).

Given these requirements, there is relatively little research involving the use of RDD sampling and SMS for inviting the general population to complete a web survey. There are a few examples of using SMS in combination with other modes or to conduct SMS surveys. In an early study, Steeh et al. (2007) sent text messages to 1884 randomly sampled numbers from one mobile phone provider in the U.S., asking them to call a toll-free number to complete a short survey by telephone. They did not offer incentives. Response rates were 2.6% for those who received only a text message, compared with 24.2% for those who received a text message followed by a telephone call and 23.2% for those who received the call only. In another early example, Balabanis et al. (2007) conducted a small experiment (n = 200) among randomly dialed UK mobile phone numbers. They sent SMS messages inviting recipients to participate in a survey online or by telephone, varying topic salience and survey sponsorship, but without incentives. They reported an overall response rate of 36.5% after three contact attempts.

Kim and Couper (2021) demonstrated the feasibility of this approach in South Korea, drawing a national RDD cell phone sample to compare two modes: a smartphone web survey using SMS invitations and a cell phone survey using computer-assisted telephone interviewing (CATI). Incentives were offered in both modes. Response rates (AAPOR RR1) were 5.3% for the web survey and 10.6% for CATI. For some demographic variables (e.g., gender), the web survey estimate was closer to national benchmarks, while for others (e.g., age) the CATI estimate was closer. Weighted estimates on substantive measures (smoking behaviors and attitudes) suggested more socially undesirable reports in the web mode (e.g., higher rates of smoking).

Two more recent studies have used RDD for sampling and SMS for invitations to a web survey. Bucher and Sand (2022) drew a simple random sample of 30,102 cellphone numbers from a frame obtained by four providers in Germany. Home location register lookups, to reduce the proportion of unassigned numbers, were only possible for two of the four providers, leaving 13,338 numbers in the sample. Text messages were sent, providing a link to complete a web survey. No incentives were offered. Only 6016 of the SMS messages sent were successfully delivered. A total of 161 people answered at least one question, and 102 completed the 36-item survey, yielding a response rate (AAPOR RR1) of .8%. Younger persons and those with higher education were overrepresented among the respondents. They concluded that “although recruiting respondents for web surveys via text messaging to smartphones is feasible, this approach is hampered by considerable issues relating to its implementation and the resulting data” (Bucher & Sand, 2022, p. 893-894), and urged further research on this topic.

Kocar (2022) randomly generated a set of Australian mobile numbers, which were then matched to a database of known users to remove non-working numbers and append selected demographics (age and gender). About 7% of the numbers were matched. The resulting sample of 27,000 numbers was split into 48 experimental groups to test the effects of survey reminders, day of the week, and time of day of the initial SMS, different types of incentives, and invitation text. The overall response rate (AAPOR RR2; see AAPOR, 2016) was 1.6%. Respondent demographics were compared to census data. The survey overrepresented females, those aged 55–74, those with a college education, and those born in Australia.

A few studies have used SMS both for inviting participants and for conducting the survey. Hoe and Grunwald (2015) conducted a short SMS survey among a simple random sample of mobile numbers associated with six zip codes surrounding a park in Philadelphia. Of the 1000 SMS invitations where a raffle of iPads was offered as an incentive, a total of 70 (7%) began the survey and 36 (3.6%) completed it. Lau et al. (2019) report on SMS surveys in four African countries (Ghana, Kenya, Nigeria, and Uganda). Uganda requires prior consent to send SMS messages, but the other three countries permit SMS invitations without prior consent. Lists of active mobile phone numbers were obtained from mobile phone network operators. Response rates to the 16-item SMS survey ranged from .3% in Nigeria and .6% in Ghana, to 12.1% in Kenya. Lau and colleagues compared the estimates from these SMS surveys with benchmark data from face-to-face surveys conducted in these countries. After weighting on age and sex, they found that the SMS surveys significantly underrepresented those with low education and the self-employed, and overrepresented those with Internet access. Some of these differences were very large. Sending reminders improved representativeness, but offering shorter surveys or higher incentives did not. They conclude that “SMS cannot replace high-quality FTF surveys” (Lau et al., 2019, p. 325).

Web versus Face-To-Face Comparisons

We note that our design does not permit us to disentangle selection effects from mode effects. Rather, we are comparing two systems of survey design, where both the sampling mechanism and the mode of data collection are varied. Relatively few studies have explored differences in estimates between national face-to-face and web surveys.

Some examples of cross-sectional surveys comparing web to face-to-face data collection include the American National Election Studies (ANES; DeBell et al., 2018), the U.S. Residential Energy Consumption Survey (RECS; Biemer et al., 2018) and the European Values Survey (EVS; Wolf et al., 2021). Most of these used ABS and mailed invitations to complete a web survey.

Most relevant to our study are two recent health surveys. Braekman et al. (2022) conducted a web survey in parallel to the Belgian Health Interview Survey (conducted face-to-face) in 2018. The national population register was used as a sampling frame for both surveys, and the sampled individuals aged 16–85 for the web survey were invited by mail. The substitution was used to replace nonrespondents in both surveys. The web survey questionnaire was shorter than the face-to-face questionnaire. Among 6183 individuals invited to the web survey, 16.3% (1010) completed the survey, and among 7698 those invited to the face-to-face survey, 43.1% (3316) did. A lower response rate was observed for the web survey among all compared sociodemographic groups (especially, those aged 65 years and older, lower educated people, a non-Belgian nationality, and people living alone). The cost per completed questionnaire was €41 (US$48) for the web mode, compared with €111 (US$131) for the face-to-face mode. Braekman et al. (2022) did not report on differences in substantive estimates between the two surveys.

In 2021, the UK National Health Service conducted a feasibility study (FS) of converting the Health Survey for England (HSE) from face-to-face interviewing to self-completion modes (see NHS Digital, 2021). This was in response to the cessation of face-to-face interviewing in 2020 due to the COVID-19 pandemic. ABS was used to select addresses from the Postcode Address File (PAF). A stratified random sample of 19,500 unclustered addresses was selected, with up to two adults aged 16 or older per household invited to respond to the survey online. A paper questionnaire was sent to nonresponding households. Including paper questionnaire response, the adjusted (after removing ineligible addresses) address-level response rate was 25%, substantially lower than the equivalent rate of 60% obtained in the 2019 HSE. The adjusted individual-level response rate was 17%. Seventy percent of respondents completed the survey online.

In terms of demographic differences, both the web/paper and face-to-face surveys overrepresented women relative to the population. Similarly, in both surveys, younger age groups were underrepresented compared with older age groups. Responding adults in the web/paper survey were also more likely to be older, from white backgrounds, living in less deprived areas of the country, and less likely to be renting their homes compared with those in the face-to-face survey.

In terms of substantive differences, the report concluded: “…results across a number of questions suggest that the HSE FS may have missed less healthy individuals and individuals who had less healthy lifestyles.” The study concluded that “The HSE FS achieved a less representative sample of participants than the face-to-face surveys which weighting may not completely compensate for” (NHS Digital, 2021).

While there are many other experimental comparisons of mode effects between web and face-to-face surveys, most of these involve pre-recruited samples or panel studies, or mixed-mode designs (e.g., Bianchi et al., 2017; Biemer et al., 2022; Jäckle et al., 2015; McGonagle & Freedman, 2017; Ofstedal et al., 2022). We do not review these further as they do not directly pertain to our comparison.

With the critical requirements for conducting an RDD-based survey using SMS invitations to a web survey in place, we conducted a national RDD smartphone web survey to compare estimates with those from a face-to-face health survey in South Korea, the Korea Community Health Survey (KCHS) sponsored by the Korea Centers for Disease Control and Prevention.

Study Design, Implementation, and Data Analysis

There were several reasons for choosing the KCHS as a comparison target among many face-to-face household surveys conducted in 2020. First, to minimize temporal changes, we wanted a survey that was conducted in the same period. Second, we wanted a survey administered using computer-assisted personal interviewing (CAPI) rather than paper-and-pencil, making it more similar to the web-based survey. Third, we wanted a survey focusing on health issues of broad interest to the general population. Fourth, we wanted a large-scale household survey conducted or sponsored by a government agency that provides accurate data with a high response rate. The KCHS satisfied all these conditions (see Korea Centers for Disease Control and Prevention, 2020; 2021).

The national RDD smartphone web survey for this study, named the 2020 National Survey of Life and Health (NSLH), was conducted by the Survey and Health Policy Research Center (SHPRC) at Dongguk University. The sample design, implementation, and data analysis of the two surveys are described below.

NSLH

Sample Design

Since 2018, SHPRC has been using a cell phone-only RDD frame for two major reasons. First, cell phone RDD samples in South Korea were much more demographically representative than landline RDD samples. Second, interviewing and supervision costs in landline RDD surveys were three times more expensive than those in cell phone RDD surveys due to a large difference between the numbers of completed interviews per hour (.65 for landlines vs. 1.83 for cell phones in 2017). This differs from the U.S., where the costs of cell phone RDD surveys were reported to be substantially greater than those of landline RDD surveys (see AAPOR, 2010).

In general, survey organizations in South Korea use a cell phone RDD frame because public access to a comprehensive record of individual subscribers of cell phone numbers, such as the national directory databases owned by telecommunication companies, is strictly forbidden by telecommunications law. The cell phone RDD frame at the SHPRC has been constructed using active seven-digit cell phone number prefixes. Some seven-digit number prefixes used by specific businesses or public organizations have been identified and removed using publicly available information. All 10,000 possible suffixes from 0000 to 9999 are then appended to the remaining seven-digit prefixes to create an RDD frame of eleven-digit cell phone numbers. The cell phone RDD frame size consisted of 69,720,000 numbers. Considering that there are 55,857,980 individual cell phone subscribers (Ministry of Science and ICT, 2020), it is estimated that 80% of phone numbers in the cell phone RDD frame are used by the general population.

Sample size

We aimed to achieve about 1000 completed interviews as a final sample size, typically required for nationwide polls or social research. A previous study by Kim and Couper (2021) obtained a total of 537 completed interviews from a smartphone web survey using an initial sample of 15,900 RDD cell phone numbers. Assuming similar response rates, we decided to select an initial sample of 30,000 numbers, to achieve a sample size of 1000. A reserve sample was not considered.

Sampling Method

We used an unstratified and unclustered single-stage equal probability of selection method (EPSEM) to select an initial sample of 30,000 eleven-digit numbers from the cell phone RDD frame at the SHPRC. In South Korea, geographical stratification for eleven-digit cell phone numbers by area codes that match administrative divisions is not possible because they share a single mobile prefix “010” instead of area codes. The single-stage EPSEM provides equal probabilities of selection for all cell phone numbers in the frame, and hence the sample is self-weighted, that is, the reciprocal of the probability of selection of each cell phone number in the sample is the same. Self-weighting samples are often preferred for many surveys because they possess considerable advantages in reduced variances, simplicity, and robustness (see Kish, 1992, pp. 194-195).

Data Collection

Four sample replicates of 7500 numbers each were used to manage and monitor data collection and evaluate the consistency of outcomes. The web survey for each replicate was open for four weeks, with replicates released a week apart. Data collection lasted for a total of 7 weeks, from October 12 to November 28, 2020.

Survey Invitations

The design and content of the SMS text message invitation, which is the sample person’s first exposure to the study, are critical for maximizing the response rate. The invitation was written to engender trust and was limited to 135 words to be concise and to distinguish it from unwanted or unsolicited commercial text messages. It included a brief description of the university-based survey organization (SHPRC), the primary purpose and importance of the survey, a link to the university website with detailed information on the NSLH, requirements for participation (use of a smartphone), voluntary participation, explanation of the incentive, advance notice of screening questions at the beginning of the survey, expected survey duration, the method of selection (“We randomly selected your phone number by a computer, and do not have any of your personal information”), the statement “We are not advertising or selling anything,” and a unique short link to the survey. The invitation was sent on Mondays (weekdays) using an SMS text messaging service.

Reminders

Three follow-up SMS reminders were sent once a week for three weeks. The first reminders were sent on Mondays; the second and third reminders were sent on Sundays. In each reminder, the dates of the previous invitations were stated and the importance of voluntary participation was re-emphasized with the original invitation message.

SMS Text Messaging Service

In South Korea, many commercial SMS text messaging services can assist in sending out invitations and reminders to sample numbers in a batch process. We used Aligo (see https://smartsms.aligo.in/). It costs only a few cents per SMS message and provides real-time information on the delivery status and reasons for delivery failures when a message is queued, sent, delivered successfully, or not delivered by the carrier. We allowed recipients to refuse to receive SMS messages at any time by simply clicking the Refusal button at the bottom of the SMS message.

KCHS

Sample Design

Begun in 2008, the KCHS is a community (municipality)-based large annual health survey covering the adult household population. This survey was conducted by 255 community health centers located in cities and counties across the country, in cooperation with universities within the communities. A sample of households was selected by stratified two-stage cluster sampling in each community. All adults were selected (interviewed) within each sample household. A similar number of adults (about 900) were interviewed in each community, and about 230,000 adults nationally. Since all communities have different numbers of households, the selection probabilities of households vary, accordingly survey weights should be applied during the analysis to compensate for unequal selection probabilities.

Data Collection

The KCHS was conducted from August 16 to October 31, partially overlapping with the NSLH. The 2020 Population and Housing Census conducted by Statistics Korea, which provides detailed information on the demographic characteristics of the general population, was conducted from October 15 to November 18 in the same year. The demographic profiles from the Census data are useful for evaluating the quality of the NSLH and HCHS, as described later.

Survey Invitations

A one-page prenotification letter from the director of the community health center was sent via mail with a survey brochure to each sampled household (Korea Centers for Disease Control and Prevention, 2020, p. 29).

Reminders and Callbacks

The guidelines specified a minimum of three callbacks (attempts) to reach members of the sample household. In practice, interviewers often made more than three callbacks to reduce non-contacts or refusals (Korea Centers for Disease Control and Prevention, 2020, pp. 40-41).

Questionnaires

National Survey of Life and Health

The NSLH questionnaire consisted of 50 questions, divided into an introductory statement (similar to the invitation text message) and five sections. The survey specified that only smartphone users could participate in the survey, but did not specify which device (smartphones, PCs, or tablets) should be used to respond, and asked what device was used both at the beginning and end of the survey. The first section asked four screener questions (last 6 digits of the cell phone number; geographic location of residence; cell phone use; and age) to determine eligibility for the survey. The second section asked 4 subjective questions and 19 factual questions about key health topics, selected from 142 common questions at the national level from the KCHS questionnaire. The third section asked about the devices used for responding to the web surveys and the incentive (choice between a mobile coupon or cash). The fourth section asked 11 questions about various daily activities including smartphone use. The fifth section asked standard demographic questions (gender, education, employment, marital status, household size, and housing occupancy status), which replicated questions used in the Census and on the KCHS. The questionnaire was optimized for smartphone respondents. There were no grid or matrix questions. Each web page contained a single question, that is, one question per smartphone screen. Response options were displayed in a single vertical list to minimize the need for horizontal scrolling.

Excluding extremely high outliers (respondents could suspend and resume the survey), the NSLH took an average of 10.7 minutes (a median of 8.2 minutes) to complete. There were no extremely low outliers.

Korea Community Health Survey

The number of questions in the KCHS questionnaire varied depending on the community. The questionnaire covered a variety of topics on personal health practices and behaviors. The average completion time at the national level for the KCHS was 28 minutes (Korea Centers for Disease Control and Prevention, 2020).

Survey Software

For the NSLH, we used SurveyMonkey (see https://ko.surveymonkey.com). It is available in the Korean language and web surveys are automatically adapted for smartphones. We tested the instrument on various smartphones and operating systems (e.g., Samsung Galaxy and Apple iPhone). A user agent string (UAS) helps identify which browser is being used, what version, and on which operating system, according to the type of device used (smartphone, tablet, laptop, or desktop). However, SurveyMonkey does not make the UAS available to researchers, so we relied on self-reports of devices used at the beginning and end of the survey.

The KCHS was administered using CAPI software developed by a company in South Korea. The interview data was transferred to a central database and monitored in real-time (Korea Centers for Disease Control and Prevention, 2020, pp. 21-22).

Incentives

In the NSLH, offering a mobile coupon or cash as an incentive was mentioned in both the SMS text message invitation and the introductory statement of the questionnaire. The amount was not specified and was revealed only if there was a phone inquiry. Respondents were offered a 2000 KRW mobile coupon (worth about $1.74USD) or 2000 KRW in cash for completing the survey.

In the KCHS, the incentive amount was also not mentioned in the prenotification letter or the introductory statement of the questionnaire. A 10,000 KRW gift certificate (worth about $8.70USD) was offered for completing the survey (Korea Centers for Disease Control and Prevention, 2020, p. 44).

Weighting and Data Analysis

The KCHS is designed to produce separate estimates for each of the 255 community health centers. Reports and microdata (in the form of SAS files) can be downloaded by request from the KCHS website (https://chs.kdca.go.kr/chs/main.do) in Korean. Given the community focus of the KCHS, survey weights for calculating national estimates of population parameters are not available. The results of national-level analyses are officially reported as a median of 255 community-based weighted estimates. Thus, national-level survey weights are needed to compare the KCHS with the NSLH. In addition, the KCHS attempts to interview all adults from each sample household, whereas the NSLH was a sample of persons. Given this, we first randomly subsampled one person from each household in the KCHS, leaving us with 125,585 of the 229,269 respondents (55% of KCHS respondents). The base weight (the household weight in the sample) in the microdata was then multiplied by the number of adults in the household.

Poststratification was used to reduce sampling variance and nonresponse and noncoverage biases (see Kish, 1992, p. 187). We evaluated two forms of poststratification. For the first, the base weights were adjusted so that the weighted totals within each of 170 post-strata divided by the domains (17 administrative divisions of 8 cities and 9 provinces, 2 gender groups, and 5 age groups) equal the population totals in the 2020 Census. For the second, the base weights were adjusted using 40 post-strata divided by the domains (2 gender groups, 5 age groups, and 4 levels of education) in the same way as the first. The two forms of postratification yielded very similar results, so we present only the results from the second version here.

For the NSLH, the EPSEM RDD sample was self-weighted, so the base weight was the same for all cell phone numbers in the sample. We evaluated two forms of poststratification, similar to the KCHS. Again we only present the results from the second version.

Often called the unequal weighting effect (UWE), the design effects due to unequal weighting (Kish, 1992), indicating an increase in the variance of the survey estimates resulting from final weights, were 1.97 for the NSLH and 2.06 for the KCHS. These final weights in the two surveys were used to analyze survey results, described later.

Item Missing Data

The web survey for the NSLH was designed so that the respondents could change their responses to earlier questions but could not advance without answering a question, so item nonresponse was minimal among completed surveys. There was no explicit “Decline to Answer” response option such as “Refuse to Answer” or “Don’t Know.”

For the KCHS, which allowed an explicit “Decline to Answer” response option, most of the common questions used in the analysis had very little item missing data. Four questions (To what extent do you think the government’s ability to respond to COVID-19 is appropriate?; Do you know your blood pressure level?; Do you know your blood sugar level?; BMI) had higher item missing rates (1239 cases, 382 cases, 715 cases, and 3596 cases respectively), but still less than 3% of the sample (125,585). If there is missing data for a question, we analyzed the rest of the data excluding the missing data for that question.

Results

Distributions of Cell Phone Numbers in the Smartphone Web Survey

A total of 1532 adults completed the NSLH survey, exceeding the target of about 1000 adults. Table 1 presents the frequencies and percentages of cell phone numbers in each stage of the data collection process, and identifies the potential source of error at each stage.

Table 1.

Distribution of Cell Phone Numbers and Potential Errors in Sampling and Data Collection of the NSLH

	Cell Phone RDD Frame	Initial sample	SMS Text Messages				Completed interviews
			Invitation Delivered^a	Follow-ups
			Invitation Delivered^a	First Reminder Delivered^a	Second reminder Delivered^a	Third Reminder Delivered^a
Size	69,720,000	30,000	20,529	19,832	18,869	17,837	1532
Percent		100%	68.4%	66.1%	62.9%	59.5%	5.1%
Sources of Errors	Coverage Error	Sampling Error	Errors in SMS Messaging/Reporting system				Nonresponse Error

^a‘Delivered’ means that an SMS text message successfully reached the recipient’s phone.

More than two-thirds (68.4%) of the SMS invitations were successfully delivered, with a declining percentage of reminders being delivered. Completed surveys were obtained from 5.1% of sampled numbers, and from 7.5% of numbers with a successful SMS invitation delivery. Appendix Table A1 shows the added yield with each successive reminder and additional details by sample replicate.

Efficiency of Screening the RDD Sample by SMS

Using the real-time delivery report of the SMS messaging system eliminated the need to screen the initial sample of 30,000 RDD cell phone numbers by trained operators or automatic systems to remove non-working numbers. The 68.4% delivery rate can be viewed as the working number rate in an RDD smartphone web survey. This was slightly lower than the working number rate of 75.4% obtained by Kim and Couper (2021, p. 1223) using trained operators to screen and remove non-working numbers. Using SMS, for which there is no charge for non-delivered SMS messages in South Korea, can help in reducing time, effort, and costs for smartphone web data collection.

Self-Report of Device Used in the NSLH

We mentioned in the SMS message and the introductory statement of the questionnaire that the survey was restricted to smartphone owners. However, as noted earlier, we asked respondents what kind of device was used, both at the beginning and end of the survey. Appendix Table A2 shows the distribution of reported devices used. Most of the NSLH respondents (93.9%) reported using a smartphone both at the beginning and end of the survey. Given this, we based our analyses on all respondents, regardless of the device used.

Response Rates

Table 2 shows the final disposition distributions of sample cases and response rates for the smartphone web survey (NSLH), following AAPOR’s (2016) standard definitions for web surveys of the specifically named person (codes for an RDD smartphone web survey are not available).

Table 2.

Final Disposition Distributions and Response Rates in the NSLH

Initial RDD sample size	30,000 (100.0%)
1. Returned questionnaire		4. Not eligible
Completed interview	1532 (5.1%)	Failed to deliver SMS message	9471 (31.6%)
Partial or Break-off	414 (1.4%)	Foreign resident	7 (.0%)
2. Eligible, Non-interview		Business	22 (.0%)
Refusal, Non-contact or Other	No Cases	Non-adult	294 (1.0%)
3. Unknown eligibility, Non-interview
Nothing returned	16,971 (56.6%)
Refusal by phone	32 (.1%)
Implicit opt-out of SMS	1257 (4.2%)
AAPOR Response Rates	RR1 7.6%, RR2 9.6%, RR3 30.8%, RR4 39.1%

Note. Final dispositions are based on AAPOR (2016) and response rates were calculated by AAPOR Response Rate Calculator 4.1 (2020).

Response rate 1 (RR1) was 7.6%. This was slightly higher than the 5.3% RR1 obtained in a recent RDD smartphone web survey (Kim & Couper, 2021). RR2, which counts partial interviews as complete, was 9.6%. RR3 was 30.8%. This rate was calculated by using the estimated proportion of unknown eligibility cases that are eligible, often referred to as “e,” which was .166 (16.6%). This proportion is comparatively low due to a high number of cases being ineligible, such as non-adults or businesses. For example, the smartphone ownership rate among non-adults (ages 6–18) is about 95% (National Information Society Agency, 2021). About 15 million numbers out of a total of 70 million cell phone numbers are used for business and industrial purposes based on IoT technology (Ministry of Science and ICT, 2021). The RR3 (30.8%) was substantially higher than the corresponding rates (e.g., 6% in 2018) for RDD telephone surveys in the U.S. (Keeter et al., 2017; Kennedy & Hartig, 2019). RR4, which is similar to RR3 but includes partial interviews in the numerator, was 39.1%.

Directly comparable rates cannot be calculated for the KCHS. The household response rate (AAPOR RR6) at the national level is officially reported as an average and median of the 255 community-based rates. In the 2020 KCHS, these were 92.5% (average) and 94.4% (median) (see Korea Centers for Disease Control and Prevention, 2021).¹

Representation of Respondents Using Unweighted Estimates

We selected five key demographic variables for the unweighted comparison of the two surveys to each other and to official statistics on the adult population from the 2020 Census provided by the Korean Statistical Information Service (https://kosis.kr/index/index.do) (see Table 3). Administrative division in the smartphone web survey was obtained by asking a screener question, while this variable was derived from the addresses on the sample frame for the CAPI survey.

Table 3.

Key Demographic Comparison Between Unweighted Sample Estimates and Adult Population Distributions.

Variables	Unweighted Sample Estimates % (Signed Difference)		p Value^a	Adult Population %
Variables	Smartphone Web Survey (NSLH)	CAPI (KCHS)	p Value^a	Adult Population %
Administrative divisions			.000^***
8 cities
Seoul	22.9 (3.7)	9.6 (−9.6)		19.2
Pusan	5.2 (−1.5)	6.5 (−.2)		6.7
Incheon	5.4 (−.3)	4.0 (−1.7)		5.7
5 other cities	13.9 (.9)	9.2 (−3.8)		13.0
9 provinces
Gyeonggi	27.3 (2.0)	16.8 (−8.5)		25.3
Gyeongnam	5.2 (−1.2)	8.0 (1.6)		6.4
Gyeongbuk	4.4 (−.8)	10.2 (5.0)		5.2
6 other provinces	15.7 (−2.8)	35.7 (17.2)		18.5
(Average absolute difference)	(1.7)	(6.0)
Gender			.000^***
Male	50.9 (1.3)	43.3 (−6.3)		49.6
Female	49.1 (−1.3)	56.7 (6.3)		50.4
(Average absolute difference)	(1.3)	(6.3)
Age groups			.000^***
19–29	33.7 (16.7)	9.0 (−8.0)		17.0
30–39	23.9 (7.9)	10.8 (−5.2)		16.0
40–49	19.2 (.1)	15.3 (−3.8)		19.1
50–59	13.6 (−6.3)	18.0 (−1.9)		19.9
60 or older	9.6 (−18.4)	46.9 (18.9)		28.0
(Average absolute difference)	(9.9)	(7.6)
Level of education			.000^***
High school graduate or less	23.2 (−26.0)	65.3 (16.1)		49.2
Two/three-year degree	18.2 (3.5)	10.3 (−4.4)		14.7
Four-year degree	46.2 (15.5)	21.0 (−9.7)		30.7
Postgraduate degree	12.4 (7.0)	3.4 (−2.0)		5.4
(Average absolute difference)	(13.0)	(8.1)
Current marital status			.000^***
Married	49.9 (−8.5)	59.2 (.8)		58.4
Widowed	1.2 (−6.3)	18.0 (10.5)		7.5
Divorced	4.0 (−2.0)	6.6 (.6)		6.0
Single	44.9 (16.8)	16.2 (−11.9)		28.1
(Average absolute difference)	(8.4)	(6.0)

Note. The signed difference is between the survey estimates and the adult population percentage. The adult population sizes are provided by the Korean Statistical Information Service (https://kosis.kr/index/index.do).

^ap values using Pearson’s chi-square tests for differences in the unweighted estimates between surveys.

^***p < .01.

Table 3 shows the unweighted percentages of the respondent and adult population percentages for the five demographic variables in the two surveys, the signed differences between those percentages, and the p-values for the differences in the respondent percentages between the two surveys. Despite concerns about geographic representation by administrative division (area of residence) in the NSLH, the signed differences between the respondent and population percentages were relatively small (less than ±4 percentage points). In contrast, the signed differences in the KCHS were larger for administrative divisions, reflecting the fact that an equal number of adults (about 900) were surveyed in each of the 255 communities. Regarding gender, the NSLH distributions were closer to the population percentages. For age, the NSLH overrepresented (+16.7%) the youngest age group (19–29) and underrepresented (−18.4%) the oldest group (60+), whereas the KCHS underrepresented (−8.0%) the youngest age group (19-29) and overrepresented (+18.9%) the older group (60+). Two possible reasons account for this age discrepancy. One is that some older people may be less familiar with SMS and be not accustomed to using their smartphones for survey response (e.g., see Olmsted-Hawala et al., 2018). Another is undercoverage in smartphone ownership in old age groups. In 2020, only about half of those aged 70 and older (54%) had a smartphone, compared with at least 93% for younger age groups (National Information Society Agency, 2021).

The NSLH substantially underrepresented (−26.0%) those with the lowest level of education, whereas the KCHS overrepresented (16.1%) this group, resulting in significant differences in education (p < .01) between the two surveys. For marital status, the NSLH overrepresented the “single” category (16.8%), whereas the KCHS underrepresented this group (−11.9%), again yielding significant differences (p < .01) between surveys.

A comparison of the average absolute differences between the two surveys shows that for administrative division and gender, the NSLH respondents were more representative of the population than the KCHS. But for the remaining variables, the NSLH was slightly less representative of the population than the KCHS. This suggests that neither survey was superior in terms of these demographic variables.

In the NSLH, every individual has an equal chance of being selected, while in the KCHS the selection probabilities differ. To better understand the representation of respondents between the two surveys, it would be more beneficial to compare the unweighted percentages in the NSLH with the weighted percentages based on unequal selection probabilities in the KCHS. However, the KCHS website does not provide microdata on the selection probabilities.

Accuracy of Weighted Estimates

Table 4 compares weighted demographic estimates to the population distribution. As expected, the signed differences are very small (mostly within ±1.0) and not statistically significant for the three poststratification variables (gender, age, and education) in both surveys.

Table 4.

Key Demographic Comparison Between Respondents’ Weighted Estimates and Adult Population Distributions.

Variables	Weighted estimates % (signed Differences)		p Value^a	Adult Population %
Variables	Smartphone Web survey (NSLH)	CAPI (KCHS)	p Value^a	Adult Population %
Administrative divisions			.0.559
8 cities
Seoul	21.6 (2.4)	19.2 (.0)		19.2
Pusan	5.5 (−1.2)	6.5 (−.2)		6.7
Incheon	5.4 (−.3)	5.8 (.1)		5.7
5 other cities	11.6 (−1.4)	13.0 (.0)		13.0
9 provinces
Gyeonggi	26.7 (1.4)	25.5 (.2)		25.3
Gyeongnam	6.5 (.1)	6.6 (.2)		6.4
Gyeongbuk	5.3 (.1)	4.9 (−.3)		5.2
6 other provinces	17.4 (−1.1)	18.5 (.0)		18.5
(Average absolute difference)	(1.0)	(.1)
Gender			1.000
Male	49.5 (−.1)	49.5 (−.1)		49.6
Female	50.5 (.1)	50.5 (.1)		50.4
(Average absolute difference)	(.1)	(.1)
Age groups			1.000
19–29	17.1 (.1)	17.1 (.1)		17.0
30–39	15.7 (−.3)	15.7 (−.3)		16.0
40–49	18.9 (−.2)	18.9 (−.2)		19.1
50–59	19.8 (−.1)	19.8 (−.1)		19.9
60 or older	28.5 (.5)	28.5 (.5)		28.0
(Average absolute difference)	(.2)	(.2)
Level of education			1.000
High school graduate or less	49.2 (.0)	49.2 (.0)		49.2
Two/three-year degree	14.7 (.0)	14.7 (.0)		14.7
Four-year degree	30.7 (.0)	30.7 (.0)		30.7
Postgraduate degree	5.4 (.0)	5.4 (.0)		5.4
(Average absolute difference)	(.0)	(.0)
Current marital status			.012^**
Married	60.5 (2.1)	63.2 (4.8)		58.4
Widowed	5.3 (−2.2)	7.4 (−.1)		7.5
Divorced	7.4 (1.4)	4.7 (−1.3)		6.0
Single	26.8 (−1.3)	24.7 (−3.4)		28.1
(Average absolute difference)	(1.8)	(2.4)

^ap values using Rao-Scott (1987)’s chi-square tests for differences in the weighted estimates between surveys.

^**p < .05.

For the remaining two variables not used in poststratification (administrative divisions and marital status), the differences are still small (mostly within ±1.0 and some within ±2.5 for administrative divisions; all within ±2.5 for marital status) in the NSLH and small likewise (all within ±1.0; all within ±5.0) in the KCHS. In summary, poststratification markedly improved the accuracy of the estimates relative to population parameters.

Responses to Substantive Questions

Tables 5 and 6 present the unweighted and weighted estimates for a series of substantive questions in common between the two surveys. All four subjective questions in Table 5 show significant differences between the two surveys for both unweighted and weighted estimates. For example, those interviewed in CAPI (KCHS) were almost twice as likely to report being extremely concerned about getting COVID-19 than those completing the smartphone web survey (NSLH). The smartphone web survey consistently yielded fewer reports of socially desirable responses (“very good” in the first question, “extremely” in the second, “very appropriate” or “appropriate” in the third, and “almost never” in the fourth) than the CAPI survey (KCHS).

Table 5.

Subjective Questions: Differences in Unweighted and Weighted Estimates Between Surveys.

Questions	Unweighted estimates (%)		p Value^a	Weighted estimates (%)		p Value^a
Questions	SP Web (NSLH)	CAPI (KCHS)	p Value^a	SP Web (NSLH)	CAPI (KCHS)	p Value^a
How would you describe your general state of health?			.000^***			.000^***
a. Very good	6.3	9.3		5.5	11.8
b. Good	34.3	36.5		30.8	40.8
c. Fair	48.1	38.8		52.4	37.8
d. Bad	10.6	12.3		10.6	8.0
e. Very bad	0.7	3.1		0.7	1.6
n	1,532	125,581		1,532	125,422
How concerned are you about getting COVID-19?			.000^***			.000^***
a. Extremely	16.8	34.4		17.0	30.9
b. Very	39.2	36.2		38.0	37.8
c. Somewhat	31.4	19.0		32.0	21.7
d. Not very	9.3	8.4		9.6	7.9
e. Not at all	3.3	2.0		3.4	1.7
n	1,532	125,541		1,532	125,383
To what extent do you think the government (including the Ministry of Health and Welfare and the Korea Centers for Disease Control and Prevention)’s ability to respond to COVID-19 is appropriate?			.000^***			.011^**
a. Very appropriate	21.9	27.2		26.4	23.9
b. Appropriate	43.3	45.8		42.0	47.2
c. Moderate	22.7	19.6		21.7	21.2
d. Not appropriate	9.5	6.2		8.0	6.4
e. Not appropriate at all	2.6	1.2		1.9	1.3
n	1,532	124,346		1,532	124,202
How often do you experience stress in your daily life?			.000^***			.000^***
a. Very often	9.1	3.1		6.5	3.5
b. Fairly often	32.8	19.0		30.4	22.0
c. Sometimes	51.4	50.9		54.2	54.2
d. Almost never	6.7	27.0		8.9	20.3
n	1,532	125,538		1,532	125,381

Note. The ‘n’ represents the number responding to each question. For the unweighted estimates, the NSLH had a total of 1532 respondents, and the KCHS had a total of 125,585 respondents. For the weighted estimates, the NSLH had a total of 1532 respondents, and the KCHS had a total of 125,426 respondents due to 159 nonrespondents in the level of education.

^ap Values in chi-square tests for differences between surveys. For the weighted estimates, the p values from Rao-Scott (1987)’s chi-square tests are given.

^***p < .01.

^**p < .05.

Table 6.

Factual and Behavioral Questions: Differences in Unweighted and Weighted Estimates Between Surveys.

Questions	Unweighted estimates (%)		p Value^a	Weighted estimates (%)		p Value^a
Questions	SP web (NSLH)	CAPI (KCHS)	p Value^a	SP web (NSLH)	CAPI (KCHS)	p Value^a
Have you ever been quarantined or hospitalized for COVID-19? (% Yes)	1.8	0.4	.000^***	1.1	0.6	.006^***
n	1,532	125,585		1,532	125,426
Have you experienced any respiratory symptoms such as fever or cough in the past 3 months? (% Yes)	8.0	1.2	.000^***	5.8	1.4	.000^***
n	1,532	125,584		1,532	125,425
How many regular cigarettes have you smoked in your lifetime?			.000^***			.000^***
a. Less than 100 cigarettes	6.6	1.7		5.5	2.2
b. More than 100 cigarettes	35.8	32.6		37.4	34.5
c. Never smoked	57.6	65.7		57.1	63.3
n	1,532	125,570		1,532	125,411
(Follow-up question)			.000^***			.001^***
How often do you currently smoke regular cigarettes?
a. Smoke everyday	39.7	43.1		38.0	43.5
b. Smoke occasionally	11.4	5.2		9.7	5.9
c. Smoked in the past, but do not smoke now	48.9	51.7		52.3	50.6
n	650	43,048		650	42,996
Have you ever smoked a cigarette-type e-cigarette (heated tobacco; e.g., Glo and Lil, etc.)? (% Yes)	16.9	3.7	.000^***	12.9	5.7	.000^***
n	1,532	125,574		1,532	125,416
(Follow-up question)			.076^*			.221
How often do you currently smoke a cigarette-type e-cigarette?
a. Smoke everyday	28.6	35.4		29.9	35.7
b. Smoke occasionally	12.7	10.8		12.0	11.3
c. Smoked in the past, but do not smoke now	58.7	53.8		58.1	53.0
n	259	4606		259	4605
Have you ever smoked a liquid e-cigarette containing nicotine? (% Yes)	14.4	2.7	.000^***	10.5	4.2	.000^***
n	1,532	125,570		1,532	125,412
Have you ever had alcohol more than one drink in your life? (% Yes)	94.3	74.1	.000^***	91.2	81.5	.000^***
n	1,532	125,576		1,532	125,419
(Follow-up question)			.000^***			.111
How often have you been drinking in the last year?
a. Have not drunk at all	13.4	24.8		17.4	19.1
b. Less than once a month	21.9	16.9		21.3	17.8
c. About once a month	13.5	11.7		13.3	13.3
d. About 2-4 times a month	29.1	22.8		24.2	26.4
e. About 2-3 times a week	16.9	16.2		16.4	17.1
f. 4 or more times a week	5.2	7.6		7.4	6.3
n	1,445	93,026		1,445	92,919
When driving a car, how often do you wear your seat belt?			.000^***			.000^***
a. Do not drive	21.7	45.6		23.9	38.7
b. Never	0.2	0.1		0.1	0.1
c. Rarely	0.9	0.4		1.0	0.3
d. Some of the time	3.3	1.5		3.4	1.1
e. Most of the time	11.0	3.0		11.8	2.5
f. All of the time	62.9	49.4		59.8	57.3
n	1,532	125,577		1,532	125,419
(Follow-up question)	6.4	3.8	.000^***	8.3	2.3	.000^***
Have you ever driven a car after drinking alcohol even a little in the past year? (% Yes)	6.4	3.8	.000^***	8.3	2.3	.000^***
n	985	47,274		985	47,239
How many times a week did you have breakfast in the past year?			.000^***			.000^***
a. 5-7 times a week	32.1	68.6		44.3	57.7
b. 3-4 times a week	16.7	7.6		15.3	9.5
c. 1-2 times a week	16.7	5.7		14.8	7.9
d. Hardly ever have breakfast	34.5	18.1		25.6	24.9
n	1,532	125,584		1,532	125,425
(Follow-up question)	16.0	11.0	.000^***	18.2	10.6	.000^***
Have you ever consulted a doctor because of stress? (% Yes)	16.0	11.0	.000^***	18.2	10.6	.000^***
n	642	27,741		642	27,694
Do you know your blood pressure level? (% Yes)	63.0	64.4	.267	73.5	64.8	.000^***
n	1,532	125,203		1,532	125,046
Have you ever been diagnosed with high blood pressure by a doctor? (% Yes)	15.2	30.4	.000^***	26.4	21.2	.002^***
n	1,532	125,566		1,532	125,408
Do you know your blood sugar level? (% Yes)	28.2	30.2	.087	38.1	31.0	.000^***
n	1,532	124,870		1,532	124,713
Have you ever been diagnosed with diabetes by a doctor? (% Yes)	6.8	12.6	.000^***	11.7	9.1	.039^**
n	1,532	125,564		1,532	125,406
Last week, did you work for 1 hour or more for pay or do any unpaid work for 18 hours or more in the family business? (% Yes)	61.3	58.9	.061	61.2	61.5	.874
n	1,532	125,539		1,532	125,385

Note. The “n” represents the number responding to each question. For the unweighted estimates, the NSLH had a total of 1532 respondents, and the KCHS had a total of 125,585 respondents. For the weighted estimates, the NSLH had a total of 1532 respondents, and the KCHS had a total of 125,426 respondents due to 159 nonrespondents in the level of education.

^ap Values in chi-square tests for differences between modes. For the weighted estimates, the p values from Rao-Scott (1987)’s chi-square tests are given.

^***p < .01.

^**p < .05.

As seen in Table 6, twelve of the 18 factual or behavioral questions show significant differences (p < .01) between the two surveys for both unweighted and weighted estimates and two questions show significant differences (p < .01) between the two surveys for weighted estimates only. The differences between the smartphone web survey and the CAPI survey are consistent with research on reporting socially undesirable behaviors by mode (see Kreuter et al., 2008; Tourangeau et al., 2000; Tourangeau & Yan, 2007), suggesting greater honesty of reporting in the web mode.

Finally, height and weight were asked as open questions of all respondents. The microdata in the CAPI survey (KCHS) is released as body mass index (BMI), defined as the weight (in kilograms) divided by the square of the height (in centimeters). The unweighted mean BMI in the smartphone web survey and CAPI survey was 23.60 and 23.56, respectively. The weighted mean BMI in the two surveys was 23.85 and 23.63, respectively. The unweighted and weighted means did not differ significantly (p > .05) between the two surveys.

Design Effects

The estimated design effect is the ratio of the variance of a weighted estimate relative to an unweighted estimate assuming the sample was selected using simple random sampling. The larger the design effect, the more sample required to obtain the same variance of an estimate as would have been obtained in simple random sampling (see Kish, 1995, pp. 257-259). For 27 variables (in Tables 4 -6) in the smartphone web survey using a sample selected by single-stage RDD sampling, the mean (median) of the design effects for the variables was 1.84 (1.82). The mean (median) of the design effects for the same variables in the CAPI survey using a sample selected by stratified multi-stage cluster sampling was 2.18 (2.18). This suggests that if the smartphone web survey was conducted with the same sample size as the CAPI survey, the variance of the weighted estimates (or margin of error) on average for those variables would very likely be smaller than in the CAPI survey.

Survey Costs

NSLH. The cost of a smartphone web survey of the general population using an RDD cell phone sample is an important element in evaluating this approach. A total of 77,067 SMS text messages (invitations and reminders) were delivered at a total cost of 2,200,000 KRW ($1913 at an exchange rate of 1150 KRW to 1USD) or 28.6 KRW ($0.0249) per SMS message. Respondents were offered a 2000 KRW mobile coupon ($1.74) or 2000 KRW in cash. A small number (127 or 8.3%) of NSLH respondents declined the incentives. The total cost of incentives (including bank fees) was 2,455,830 KRW ($2136). Web survey specialists worked a total of 280 hours at a rate of 18,000 KRW ($15.65) per hour to deploy and manage the survey, for a total cost of 5,040,000 KRW ($4383). The cost for operation and maintenance, including the license fee for web survey software, was 480,000 KRW ($417). As a result, the total cost for the smartphone web survey (1532 respondents) was 10,175,830 KRW ($8849), resulting in a cost of 6642 KRW ($5.78) per respondent. The total cost did not include the costs for RDD sampling and questionnaire design for a balanced comparison with the CAPI survey.

KCHS

The costs for the CAPI survey, involving field supervisors and interviewers, programmers, laptops, and travel, are more varied, making it more difficult to calculate the data collection cost per respondent. However, the KCHS budget for each community is public, resulting in an approximate total cost (for 125,585 respondents) of about 8,790,950,000 KRW ($7,644,304) or an average of 70,000 KRW ($60.87) per respondent. This is about 10 times the per-respondent cost of the smartphone web survey, suggesting a much larger sample could be surveyed using the smartphone web survey to increase the precision of estimates.

Discussion

Face-to-face surveys have long been regarded as the “gold standard” for obtaining high-quality data, but have suffered recently due to rising costs of in-person visits and decreasing response rates. This threat to data quality has been exacerbated by the COVID-19 pandemic. These problems are likely to grow over time, necessitating the exploration of alternative approaches.

Web surveys have become a popular alternative because they do not require interviewers, and have low costs, speed of deployment, respondent convenience, and lower social desirability bias for sensitive topics. The COVID-19 pandemic and subsequent lockdown made self-administered web surveys even more attractive. However, web surveys of the general population still face significant methodological issues including undercoverage, lack of sample representativeness, and low response rates.

We conducted a national RDD smartphone web survey using SMS text message invitations, which combines telephone sampling and mobile technology, for comparison with a large-scale national CAPI survey during the COVID-19 pandemic. The results demonstrate the potential of this approach for yielding a probability sample of the general population in countries with high rates of smartphone use and literacy, and few restrictions on SMS messages.

While the response rate to the smartphone web survey (RR1 = 7.6%) was substantially lower than that for the CAPI survey (median RR6 = 94.4%), the costs per complete were also substantially lower (by a factor of 10) for the smartphone web survey. In terms of demographic comparisons in unweighted estimates, the smartphone web survey performed well relative to population benchmarks, being closer to those benchmarks for two variables (administrative division and gender) and only slightly higher for the other three (age, education, and marital status). Poststratification largely eliminated these demographic differences between the two surveys.

There were significant differences for several substantive variables, whether examining unweighted or weighted estimates. Without benchmark data, it is hard to determine which estimates are more “accurate”; however, the (self-administered) smartphone web survey mode tended to have fewer reports of socially desirable attitudes or behaviors and more reports of socially undesirable ones than the (interviewer-administered) CAPI survey, especially for sensitive questions. This suggests that data quality (accuracy) may be improved for such topics using a smartphone web survey. Further, given the relative costs, a substantially larger smartphone web survey could be conducted for the same or less money than the CAPI survey, yielding greater (statistical) precision of the estimates.

We hope that these findings will encourage researchers to explore alternative approaches to conducting web surveys using RDD sampling methods and smartphone technology. In countries such as the U.S., where there are legal constraints against unsolicited text messages, this approach may not be practical. In many other countries, there are no such restrictions, and this approach is a promising alternative. We encourage others to replicate and extend these findings to other countries with high rates of smartphone penetration.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Authors’ Note

The authors would like to sincerely thank Soohyun Oh for the meticulous data analysis, Jaehoon Kim for careful assistance in preparing and revising the manuscript, and Junbeom Ryu and Jeongbin Kim for collecting data during the problematic situation of the COVID-19 pandemic.

Funding

The SHPRC’s own funds were used for this study. The author(s) received no financial support for the research, authorship, and/or publication of this article.

Software Information

The analyses were conducted using SAS 9.4.

Note

Data Availability Statement

The anonymized data for the smartphone web survey (NSLH) are available from an author at a given email address. The reports and microdata for the CAPI survey can be downloaded by request from the KCHS website ().

Appendix

Table A2.

Distribution of Devices Used to Respond to the NSLH

		Device used at end
		Smartphone	Tablet	Laptop	Desktop	Total
Device used At beginning	Smartphone	1438(93.9%)	2(.2%)	9(.6%)	2(.2%)	1451(94.9%)
	Tablet	1 (.0%)	7 (.5%)			8 (.5%)
	Laptop	11 (.8%)	1 (.0%)	17 (1.1%)	1 (.0%)	30 (1.9%)
	Desktop	15 (1.0%)	1 (.0%)	1 (.0%)	26 (1.7%)	43 (2.7%)
	Total	1465 (95.7%)	11 (.7%)	27 (1.7%)	29 (1.9%)	1532 (100.0%)

Author Biographies

Sunwoong Kim is a professor at the Department of Statistics and the director of the Survey & Health Policy Research Center, Dongguk University.

Mick P. Couper is a research professor at the Survey Research Center, Institute for Social Research, University of Michigan.

References

American Association for Public Opinion Research (AAPOR) (2010). Cell phone task force report: Costs in RDD cell phone surveys. AAPOR.

American Association for Public Opinion Research (AAPOR) (2016). Standard definitions: Final dispositions of case codes and outcome rates for surveys (9th ed.). AAPOR.

American Association for Public Opinion Research (AAPOR) (2017). Report of the AAPOR task force on the future of U.S. general population telephone survey research. AAPOR.

American Association for Public Opinion Research (AAPOR) (2019). Report of the AAPOR task force on transitions from telephone surveys to self-administered and mixed-mode surveys. AAPOR.

Andreadis

(2020). Text message (SMS) pre-notifications, invitations and reminders for web surveys in Survey Methods: Insights from the Field, Special issue: ‘Advancements in Online and Mobile Survey Methods’. https://surveyinsights.org/?p=13551

Balabanis

Mitchell

V.-W.

Heinonen-Mavrovouniotis

(2007). SMS-based surveys: Strategies to improve participation. International Journal of Advertising, 26(3), 369–385. https://doi.org/10.1080/02650487.2007.11073019

Bianchi

Biffignandi

Lynn

(2017). Web-face-to-face mixed-mode design in a longitudinal survey: Effects on participation rates, sample composition, and costs. Journal of Official Statistics, 33(2), 385–408. https://doi.org/10.1515/jos-2017-0019

Biemer

P. P.

Harris

K. M.

Burke

B. J.

Liao

Halpern

C. T.

(2022). Transitioning a panel survey from in-person to predominantly web data collection: Results and lessons learned. Journal of the Royal Statistical Society - Series A: Statistics in Society, 185(3), 798–821. https://doi.org/10.1111/rssa.12750

Biemer

P. P.

Murphy

Zimmer

Berry

Deng

Lewis

(2018). Using bonus monetary incentives to encourage web response in mixed-mode household surveys. Journal of Survey Statistics and Methodology, 6(2), 240–261. https://doi.org/10.1093/jssam/smx015

10.

Braekman

Demarest

Charafeddine

Drieskens

Berete

Gisle

Van der Heyden

Van Hal

(2022). Unit response and costs in web versus face-to-face data collection: Comparison of two cross-sectional health surveys. Journal of Medical Internet Research , 24(1), Article e26299. https://doi.org/10.2196/26299

11.

Bucher

Sand

(2022). Exploring the feasibility of recruiting respondents and collecting web data via smartphone: A case study of text-to-web recruitment for a general population survey in Germany. Journal of Survey Statistics and Methodology, 10(4), 886–897. https://doi.org/10.1093/jssam/smab006

12.

Dayton

Locke

Kinder

DuellZuWallack

J. R.

Troutman

(2020). Using SMS as a survey recruitment tool - challenges convincing respondents and carriers of legitimacy. Paper presented at the virtual annual of the American Association for Public Opinion Research.

13.

DeBell

Amsbary

Meldener

Brock

Maisel

(2018). Methodology report for te ANES 2016 time series study. Stanford University and the University of Michigan.

14.

European Commission (2022). Can we use data for another purpose? Retrieved from: https://ec.europa.eu/info/law/law-topic/data-protection/reform/rules-business-and-organisations/principles-gdpr/purpose-data-processing/can-we-use-data-another-purpose_en

15.

Groves

R. M.

Biemer

P. P.

Lyberg

L. E.

Massey

J. T.

Nicholls

W. L.

Waksberg

(1988). Telephone survey methodology. John Wiley.

16.

Hoe

Grunwald

(2015). The role of automated SMS text messaging in survey research. Survey Practice, 8(6), 1–15. https://doi.org/10.29115/sp-2015-0027

17.

Jäckle

Lynn

Burton

(2015). Going online with a face-to-face household panel: Effects of a mixed mode design on item and unit non-response. Survey Research Methods, 9(1), 55–70.

18.

Keeter

Hatley

Kennedy

Lau

(2017). What low response rates mean for telephone surveys. Pew Research Center. Retrieved from: https://www.pewresearch.org/methods/2017/05/15/what-low-response-rates-mean-for-telephone-surveys/

19.

Kennedy

Hartig

(2019). Response rates in telephone surveys have resumed their decline. Pew Research Center. Retrieved from: https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/

20.

Kennedy

Mcgeeney

Keeter

Patten

Perrin

Lee

Best

(2018). Implications of moving public opinion surveys to a single-frame cell-phone random-digit-dial design. Public Opinion Quarterly, 82(2), 279–299. https://doi.org/10.1093/poq/nfy016

21.

Kim

S. W.

Couper

M. P.

(2021). Feasibility and quality of a national RDD smartphone web survey: Comparison with a cell phone CATI survey. Social Science Computer Review, 39(6), 1218–1236. https://doi.org/10.1177/0894439320964135

22.

Kim

S. W.

Lee

S. K.

Hong

S. J.

Park

S. H.

(2012). List-assisted RDD sampling in Korea: Testing the feasibility of national survey under within-household selection. International Journal of Public Opinion Research, 24(1), 79–92. https://doi.org/10.1093/ijpor/edr022

23.

Kim

S. W.

Lepkowski

J. M.

(2002). Telephone household non-coverage and mobile telephones. In Proceedings of the Survey Research Methods Section (pp. 1845–1850). American Statistical Association.

24.

Kim

S. W.

Traugott

M. W.

Kwak

N. J.

Choi

E. H.

Lee

H. N.

(2014). Using two-wave dual frame RDD telephone pre-election poll in the 2012 Korean presidential election. In Paper presented at the 67th annual conference of the World Association for Public Opinion Research. Nice, France.

25.

Kim

S. W.

Woo

Y. J.

Kim

N. H.

(2017). Assuring quality in dual frame RDD national or sub-national surveys using cell phone numbers without area codes in South Korea. In Paper presented at the Joint Statistical Meetings. Baltimore, Maryland.

26.

Kish

(1992). Weighting for unequal Pi. Journal of Official Statistics, 8(2), 183–200.

27.

Kish

(1995). Survey sampling. Wiley.

28.

Kocar

(2022). Survey response in RDD-sampling SMS-invitation web-push study. Survey Research Methods, 16(3), 283–299. https://doi.org/10.18148/srm/2022.v16i3.7846

29.

Kongsgard

H. W.

Syversen

Krokstad

(2014). SMS phone surveys and mass-messaging: Promises and pitfalls. Epidemiology: Open Access, 4(4), 177. https://doi.org/10.4172/2161-1165.1000177

30.

Korea Centers for Disease Control and

Prevention (2021). Korea Community Health Survey. Retrieved from: https://chs.kdca.go.kr/chs/index.do

31.

Korea Centers for Disease Control and Prevention (2020). Statistical information report on Korea Community Health Survey. Retrieved from: https://kostat.go.kr/portal/korea/kor_pi/8/7/index.board?bmode=read&aSeq=387560&pageNo=&rowNum=10&amSeq=&sTarget=&sTxt=

32.

Korea Communications Commission (2020). Information and Communications Network Act Guide for Prevention of Illegal Spam (the fifth edition). Retrieved from: https://spam.kisa.or.kr/spam/na/ntt/selectNttList.do?mi=1020&bbsId=1002

33.

Kreuter

Presser

Tourangeau

(2008). Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly, 72(5), 847–865. https://doi.org/10.1093/poq/nfn063

34.

Lau

C. Q.

Lombaard

Baker

Eyerman

Thalji

(2019). How representative are SMS surveys in Africa? Experimental evidence from four countries. International Journal of Public Opinion Research, 31(2), 309–330. https://doi.org/10.1093/ijpor/edy008

35.

Lepkowski

J. M.

Kim

S. W.

Steeh

(2005). Dual-frame landline/cellular telephone survey design. In Presented at an invited session of the Survey Research Methods Section. American Statistical Association.

36.

Lepkowski

J. M.

Tucker

N. C.

Brick

J. M.

de Leeuw

E. D.

Japec

Lavrakas

P. J.

Link

M. W.

Sangster

R. L.

(eds.), (2007). Advances in telephone survey methodology. Wiley.

37.

Marlar

Hoover

(2019). Leveraging SMS for survey research. AAPOR webinar, June 20th.

38.

McGonagle

K. A.

Freedman

V. A.

(2017). The effects of a delayed incentive on response rates, response mode, data quality, and sample bias in a nationally representative mixed mode study. Field Methods, 29(3), 221–237. https://doi.org/10.1177/1525822X16671701

39.

Ministry of Science and ICT (2020). Wireless communication service subscriber statistics. Retrieved from: https://www.msit.go.kr/bbs/view.do?sCode=user&mId=99&mPid=74&pageIndex=8&bbsSeqNo=79&nttSeqNo=3173370&searchOpt=ALL&searchTxt=

40.

Ministry of Science and ICT (2021). Wireless communication service statistics status. Retrieved from: https://www.msit.go.kr/bbs/view.do?sCode=user&mId=99&mPid=74&bbsSeqNo=79&nttSeqNo=3173370&formMode=&pageIndex=&searchCtgry=&searchCtgry2=&searchCtgry3=&RLS_YN=&searchOpt=ALL&searchTxt=

41.

National Information Society Agency (2021). 2020 Survey on the Internet usage. Retrieved from: https://www.nia.or.kr/site/nia_kor/ex/bbs/List.do?cbIdx=99870

42.

NHS Digital . (2021). The health survey for England 2020/2021 feasibility study. NHS Digital. https://digital.nhs.uk/data-and-information/areas-of-interest/public-health/health-survey-for-england---health-social-care-and-lifestyles/feasibility-study-2020---2021

43.

Ofstedal

M. B.

Kézdi

Couper

M. P.

(2022). Data quality and response distributions in a mixed-mode survey. Longitudinal and Life Course Studies, 13(4), 621–646. https://doi.org/10.1332/175795921X16494126913909

44.

Olmsted-Hawala

Nichols

Falcone

Figueroa

Antoun

Wang

(2018). Optimal data entry designs in mobile web surveys for older adults. In Proceedings of the 4th international conference, ITAP 2018, held as part of HCI International 2018. Las Vegas.

45.

Park

S. H.

Lee

Kim

S. W.

Lee

S. K.

(2012). A comparison of response patterns between landline and cell phone RDD surveys. In Paper presented at the 65th annual conference of the World Association for Public Opinion Research. Hong Kong.

46.

Pew Research Center (2019). Smartphone ownership is growing rapidly around the world, but not always equally. Report. Retrieved from: https://www.pewresearch.org/global/2019/02/05/smartphone-ownership-is-growing-rapidly-around-the-world-but-not-always-equally/

47.

Peytchev

Neely

(2013). RDD telephone surveys: Toward a single-frame cell-phone design. Public Opinion Quarterly, 77(1), 283–304. https://doi.org/10.1093/poq/nft003

48.

Rao

J. N. K.

Scott

A. J.

(1987). On simple Adjustments to chi-square tests with sample survey data. Annals of Statistics, 15(1), 385–397. https://doi.org/10.1214/aos/1176350273

49.

Smith

(2015). U.S. smartphone use in 2015. Pew Research Center. Retrieved from: https://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/

50.

Steeh

Buskirk

T. D.

Callegaro

(2007). Using text messages in U.S. mobile phone surveys. Field Methods, 19(1), 59–75. https://doi.org/10.1177/1525822x06292852

51.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge University Press.

52.

Tourangeau

Yan

(2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859–883. https://doi.org/10.1037/0033-2909.133.5.859

53.

Wolf

Christmann

Gummer

Schnaudt

Verhoeven

(2021). Conducting general social surveys as self-administered mixed-mode surveys. Public Opinion Quarterly, 85(2), 623–648. https://doi.org/10.1093/poq/nfab039