Mobile Response in Web Panels

Abstract

This article investigates unintended mobile access to surveys in online, probability-based panels. We find that spontaneous tablet usage is drastically increasing in web surveys, while smartphone usage remains low. Further, we analyze the bias of respondent profiles using smartphones and tablets compared to those using computers, on the basis of several sociodemographic characteristics. Our results indicate not only that mobile web respondents differ from PC users but also that tablet users differ from smartphone users. While tablets are used for survey completion by working (young) adults, smartphones are used merely by the young. In addition, our results indicate that mobile web respondents are more progressive and describe themselves more often as pioneers or forerunners in adopting new technology, compared to PC respondents. We further discover that respondents’ preferences for devices to complete surveys are clearly in line with unintended mobile response. Finally, we present a similar analysis on intended mobile response in an experiment where smartphone users were requested to complete a mobile survey. Based on these findings, testing on tablets is strongly recommended in online surveys. If the goal is to reach young respondents, enabling surveys via smartphones should be considered.

Keywords

mobile web survey unintended mobile response survey error web panel respondent preference

Introduction

As mobile access rates to the Internet continue to grow (Statistics Netherlands, 2012), survey researchers question whether they should take this new online channel into account when conducting a web survey. Our article aims to answer this question in three ways.

First, we present the rate of spontaneous mobile web response in two online probability panels in the Netherlands. We investigate developments in the distribution of web and mobile respondents over the past years. This trend is examined in relation to that of general mobile web usage. Based on the results, we discuss the past, current, and expected rate of unintended use of mobile devices in online panels.

Second, we identify the unintended mobile respondents and investigate in what ways they differ from the PC web respondents. We compare the distributions of several demographic and social behavior-related variables between groups and investigate which characteristics predict the likelihood of unintended mobile response. We also consider mobile web responding from the perspective of the technology adoption life cycle theory and show how the respondents’ general progressiveness and attitudes toward new technologies relate to mobile responding. In addition, we compare these results with a similar analysis on intended mobile web response. The latter findings are based on an experiment in which the smartphone users of an online panel were requested to complete a mobile-adapted survey using their smartphones. Here, we include general mobile web usage and device-related factors into a further logistic regression analysis on the propensity of mobile responding.

Third, we discuss the level of demand for mobile surveys among online panel members. We present respondent preferences for different devices when taking web surveys and when using the mobile web in general. We investigate whether mobile demand is higher for certain respondent groups by demographic characteristics.

Based on all findings, we offer suggestions for further research and formulate practical recommendations for survey researchers on immediate actions necessary to restrict possible nonresponse and measurement error in web surveys.

Background and Research Questions

One type of mobile use in survey research can be defined as “respondents completing web surveys using mobile devices” (Couper, 2013). Approaches to the mobile use in web surveys can roughly be divided into two, either taking respondents’ mobile access to a survey into account or not. For the first approach, Callegaro (2010) has suggested a few main strategies, varying from blocking mobile respondents to adapting questionnaires for each type of device used to access an online survey. The latter, inactive approach has been termed as the “passive-mobile browser survey approach” by Buskirk and Andrus (2012). Describing the same phenomenon, Peterson (2012) and Wells, Bailey, and Link (2013) speak of “unintended mobile respondents.” Also “unintentional mobile response” is used (Peterson, 2012). In this article, we define unintended mobile response as attempts to complete web surveys using mobile devices, namely tablets and smartphones, while the survey was designed to be taken on computers and was not adapted for mobile browsers or smaller screen sizes.

Whether it is necessary to take action due to unintended mobile response in a web survey depends on several factors. Does not adapting the questionnaire design lead to an increase in survey error or respondent burden? What is the extent of unintended mobile response in web surveys and is this biased, compared to the total sample?

Concerning the first question, earlier research has found small screens to have various limitations or disadvantages (Chae & Kim, 2004; Jones, Buchanan, & Thimbleby, 2003; Jones, Marsden, Mohd-Nasir, Boone, & Buchanan, 1999; Watters, Duffy, & Duffy, 2003) and visibility has been found to affect results in web surveys (Couper, Tourangeau, & Conrad, 2004). Buskirk and Andrus (2012) discuss the many disadvantages for respondents of the not adapting, passive mobile approach, such as high page loading times with images and likely need for pinching, scrolling, or zooming on small smartphone screens. One signal of an increased nonresponse error is the relatively high survey breakoff by mobile respondents, reported by several researchers (Bosnjak et al., 2013; Callegaro, 2010; Guidry, 2012; Wells, Bailey, & Link, 2013). As earlier studies have shown that survey design may improve response rates (Porter, 2004) and reduce breakoffs in web surveys (Crawford, Couper, & Lamias, 2001), it is not surprising that many experiments on offering adapted surveys to mobile users have started to appear (Boreham & Wijnant, 2013; de Bruijne & Wijnant, 2013; Mavletova, 2013; Peytchev & Hill, 2010; Stapleton, 2013, Wells, Bailey, & Link, 2014; Wells et al., 2013).

However, on the question of the extent of unintended mobile response in general web surveys, only a few recent studies have presented estimates. Poggio, Bosnjak, and Weyandt (2013) reported the mobile response rate to range from 2.8% to 4.2% in GESIS Pilot Panel, which is an online, probability-based panel in Germany. Their results were based on eight survey waves conducted in 2011 and 2012. They found a slight but not significant increase in mobile response over the waves. Callegaro (2010) presented the results of three customer satisfaction surveys fielded in 2010 in Asia, North America, and Europe, in which the spontaneous mobile response ranged from 1.2% to 2.6%.

Since mobile web usage is increasing very rapidly, we think it is worthwhile to continue the existing research with an extended background analysis on the rate and bias of unintended mobile response, using recent data based on online, probability-based scientific panels. We investigate the mobile access attempts in two online probability-based panels in the Netherlands, the CentERpanel and the Longitudinal Internet Studies for the Social sciences (LISS) panel, from 2012 onward. Since the two most typical devices to access the mobile Internet, tablets and smartphones, differ significantly in terms of screen size and other characteristics, we specify the response rates per type of device.

The development of mobile access to surveys is furthermore compared to that of the general mobile web access as a share of the total web traffic. The general mobile Internet rate compared to the total web traffic has been estimated to have increased from 8% at the start of 2012 to 18% in September 2013 (StatCounter, 2013), which leads us to expect that respondent access to online surveys using mobile browsers has likewise increased in the past 2 years.

As unintended mobile responding may increase measurement and nonresponse error, it is important to investigate whether the distribution of mobile respondents is biased by respondent background characteristics. To estimate the bias, we examine the extent to which personal characteristics influence the likelihood of mobile attempts to access a survey. There are some earlier findings on this topic. Wells et al. (2013) found that when conducting an online survey among smartphone owners, unintended smartphone respondents were significantly more likely to be young, female, to live in larger households, and to access the Internet primarily using their smartphones. In the same study, tablet respondents were more likely to have at least a bachelor’s degree, to be married, homeowners, and a household income of at least US$75,000. Peterson (2012) found that females and younger respondents were more likely to use mobile phones in an online survey. We will study the effect of not only demographic variables such as age, sex, education, urbanization, and housing condition but also social behavior-related characteristics such working status, progressiveness, and general adoption of new technology. Adding to previous research, we present the results separately for tablets and smartphones and within a general sample of a probability-based panel.

In addition to the differences between PC and mobile web respondents, we signal another phenomenon among mobile respondents that calls for closer attention. Although some respondents spontaneously switch to mobile devices, most experiments so far have faced difficulties in persuading people to use such devices even when specifically requested to do so. In our earlier study (de Bruijne & Wijnant, 2013) and in that of Mavletova (2013), both of which targeted mobile device users, the response rates were lower in the mobile web condition than in the computer web condition. We also found that groups of respondents, while in possession of a mobile device, are unwilling to engage in mobile surveys and complete the survey using a computer instead, even when the survey is adapted to mobile browsers. Fuchs (2012) reported that 5% of nonrespondents and noncontacts in a mobile telephone survey accessed a mobile web survey alternative. Millar and Dillman (2012) experimented with a mobile-friendly survey among undergraduate students and reported a mere 5.5% mobile share of the total web response. In a condition where the researchers specifically prompted mobile responding, 45.5% of the respondents reported owning a smartphone but only 7.0% used one for survey completion. Further, only 6.6% of all respondents reported to prefer completing surveys on smartphones. Also, higher mobile rates have been presented. In a recent study of Toepoel and Lugtig (2014) among smartphone owners, with a clear prompt for mobile usage and a mobile-optimized version, 57% of the respondents who completed the survey did so by using a mobile phone.

These findings imply that simply possessing a mobile device does not necessarily indicate a willingness to use it for mobile responding. Also, Fuchs and Busse (2009) suggest that the mere coverage of the technology does not imply that all respondents who have access to the mobile Internet are able or willing to use it to complete a survey.

We therefore wish to take a deeper look at the relationship between general mobile Internet usage and mobile survey completion. Based on earlier findings, we expect that the mobile device user group is scattered and contains different types of users. Among users, we expect those who very frequently use mobile Internet to be more likely to participate in a mobile web survey. We will examine this in a background analysis based on an intended mobile web survey among smartphone users, of whom various mobile Internet usage and device-related background characteristics are known.

Finally, we estimate the level of demand among online panel members to be able to complete surveys using tablets or smartphones. We address this question based on self-report data on respondent preferences for different devices. These results are compared with the preferences for general web usage. As for the unintended mobile response analysis, we examine respondent preferences by respondent characteristics and investigate which respondent subgroups are more likely to favor mobile devices.

Method

The data for this study were gathered in two Dutch panels: the LISS panel and CentERpanel. Both panels are online probability-based panels in the Netherlands. Also, people without a prior computer or Internet connection are able to participate in these panels. To avoid sampling bias, these participants are loaned equipment to afford them access to the Internet via a broadband connection. Persons not included in the drawn panel samples cannot participate, so there can be no self-selection. The LISS panel consists of 5,000 households, comprising 8,000 individuals. Panel members complete online questionnaires every month. For this study, data of the Tilburg Consumer Outlook Monitor (a quarterly survey on consumer behavior administered to a random 2/3 of the panel) were used from the waves from March 2012 to September 2013. The CentERpanel is a household panel that completes web questionnaires every week. The panel consists of over 2,000 members. For the CentERpanel, we used the data of the Health Monitor, fielded every 8 weeks, from February 2012 to October 2013. Due to a programming lapse, user agent strings were not registered for the first half of 2013 and therefore we had no continuous data on this panel.

We investigated the mobile web response in two ways. First, we registered the user agent strings in studies from 2012 through 2013 in the aforementioned panels, to estimate the level and development of spontaneous, unintended mobile web response compared to computer-assisted web response. This method of capturing a text variable when connecting to a website is recommended by Callegaro (2013) and has been used in several other studies (Millar & Dillman, 2012; de Bruijne&Wijnant, 2013, Mavletova, 2013, Wells et al., 2013). Since many demographic and social behavior variables of the panel members are known, we were able to conduct an extended analysis as to which respondent groups access the surveys on mobile devices.

Second, we examined intended mobile response by administering a survey to the CentERpanel in which panel members were requested to complete a survey using their smartphone. We investigated the effect of several predicting factors on the response. The survey mainly consisted of adjusted questions from the World Values Survey (2011) and was part of a larger experiment on questionnaire design. For this study, we focused on analyzing the background characteristics of the mobile respondents. In Week 11 (March 15–19) of 2013, a premeasurement was fielded in the panel to inventory which respondents would be eligible for the main measurement, that is, used a smartphone to access the Internet. The premeasurement also served to collect background information on the respondents, such as preferences for different devices and frequency of mobile Internet use. The main measurement was conducted 1 week later. Finally, in Week 18 (May 3–7) of 2013, an additional survey on attitudes toward new technology was conducted in order to analyze the relationship between the technology adoption life cycle and mobile responding.

Results

Trend of Unintended Mobile Access to Surveys

The results show that the spontaneous, unintended mobile attempts to complete online questionnaires have significantly increased between 2012 and 2013. In the LISS panel, the share of mobile response increased from 3% in March 2012 to 11% in September 2013. In the CentERpanel, the spontaneous mobile response increased from 3% in February 2012 to 16% in October 2013. The higher rise in the CentERpanel is possibly due to a recruitment of new panel members in February 2013, which especially targeted younger respondents.

The presented rates are based on all respondents who had logged in for the analyzed questionnaires. The user agent string of the first attempt to log onto the questionnaire was used for each respondent. We also examined the last attempt to access the surveys and found no difference in results. An analysis of only those who fully completed the surveys resulted in the same outcome. In total, there were only a handful of respondents in each wave who switched from computer to tablets or smartphones or vice versa. As the panel respondents receive their incentive only at the end of the questionnaire, breakoff in the panels was minimal and did not yield enough cases for an analysis. The development of the mobile access rates in the LISS panel is presented in Figure 1.

Figure 1.

Respondents’ access to surveys using mobile devices in the Longitudinal Internet Studies for the Social sciences (LISS) panel as a share of total response.

The trend of unintended mobile response in the LISS panel appears similar to that of general mobile web use. In Figure 2, the mobile response rate in the LISS panel is given next to general, worldwide mobile Internet access rates. For this, we used the StatCounter (2013) statistics that are directly derived from hits, not unique visitors, from 3 million sites using StatCounter, totaling more than 15 billion hits per month. According to StatCounter, the share of Internet access using mobile web browsers has increased from 8% in January 2012 to 18% in September 2013. The shown trend lines are based on linear growth, both appearing consistent with r ² values of .98 for the LISS panel data and .97 for the StatCounter data. The difference between general mobile traffic and mobile web response in the panel has remained relatively stable in the past 2 years, averaging 7 percentage points. When a linear growth expectation is used, the available data result in an average growth rate of 5.9% for unintended mobile access to web surveys and in a growth rate of 6.3% for general mobile web access. However, since the period of data collection is relatively short due to a lack of data from earlier than 2012, it is important to notice that forecasts for the future remain highly speculative.

Figure 2.

Comparison of the shares of general mobile web traffic and mobile web access in the Longitudinal Internet Studies for the Social sciences (LISS) panel.

Bias of Unintended Mobile Access to Surveys

In the following, we discuss the role of the demographic and social behavior-related respondent characteristics in predicting unintended mobile responding. For the analysis, we used the LISS panel data of the Tilburg Consumer Outlook Monitor from June 2013. We compared the first attempts to access the survey by browser type, which allowed us to identify whether the respondent had used a computer, tablet, or smartphone to do so. As we were interested in the demand for mobile web surveys in different subgroups, these data were collected for all who accessed the questionnaire, regardless of whether they finished the survey. The computer group included desktop computers, laptops, and SimPC laptops (which are provided to panel members who do not possess a computer when recruited to the panel). Of the total of 2,508 respondents, 186 used a tablet and only 41 a smartphone to access the survey. The usage of each device type by the subgroups based on respondent characteristics is presented in Table 1. A logistic regression analysis was conducted to predict mobile responding using tablets and smartphones. The analysis gives the likelihood of responding by tablet against any other input device and, respectively, by smartphone compared to other devices. These results are shown in Table 2.

Table 1.

Access Rates to a Survey by Device Within Demographic and Social Behavior–Related Subgroups, LISS Panel, June 2013.

	Gender**		Age***						Education*^a						Working status***		Urbanization					Household composition***^b
	Male	Female	16–24	25–34	35–44	45–54	55–64	65+	A	B	C	D	E	F	Paid Work	No Paid Work	Not Urban	Little Urban	Somewhat Urban	Strongly Urban	Highly Urban	G	H	I	J	K	Total
Computer^c	93%	89%	89%	80%	88%	88%	93%	97%	95%	93%	90%	88%	90%	92%	88%	94%	93%	89%	91%	90%	92%	96%	92%	88%	86%	88%	91%
Tablet	6%	9%	6%	14%	10%	11%	6%	3%	4%	6%	7%	10%	9%	7%	11%	5%	6%	8%	8%	8%	6%	4%	7%	9%	9%	9%	7%
Smartphone	1%	2%	5%	6%	2%	1%	1%	0%	1%	1%	3%	2%	1%	1%	2%	1%	1%	3%	1%	2%	2%	1%	1%	3%	4%	3%	2%
Total	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
N	1,201	1,307	235	235	361	419	556	702	231	653	273	576	570	192	1,186	1,322	386	552	613	656	293	474	1,047	835	118	34	2,508

Note. LISS = Longitudinal Internet Studies for the Social sciences. ^aEducation: A = primary school, B = secondary education; C = higher secondary education, D = intermediate vocational education, E = higher vocational education, F = university. The education level was missing for 13 respondents, which are excluded here. ^bHousehold composition = G = living alone, H = living together, without children, I = living together, with children, J = single, with children, K = other. ^c Computer: including desktop computers, laptops, and SimPCs.

*p < 0.05. **p < 0.01. ***p < 0.001 (chi-square: gender, working status/Monte Carlo two sided, 99% confidence interval: age, education, urbanization, and household composition).

Table 2.

Logistic Regression Model Predicting the Likelihood of Unintended Mobile Responding, LISS Panel, June 2013.

	Tablet		Smartphone
	Coefficients	Odds	Coefficients	Odds
Age	−.01 (.01)**	0.99	−.07 (.01)***	0.94
Sex: male	−.50 (.16)**	0.61	−.69 (.35)*	0.50
Working status: paid work	.76 (.17)***	2.14	.34 (.35)	1.41
Urbanization (low to high)	.02 (.06)	1.02	.10 (.13)	1.11
Higher education (vocational or university)	.04 (.17)	1.04	−.32 (.39)	0.73
Living alone	−.81 (.26)**	0.44	−.77 (.61)	0.46
Constant	−2.04 (.34)***	0.13	−1.41 (.58)*	0.24
Nagelkerke R ²	.06		.14
Number of observations	2,487		2,487

Note. LISS = Longitudinal Internet Studies for the Social sciences. References: female, no paid work, secondary/lower/intermediate education, living with other people.

*p < .05. **p < .01. ***p < .001.

The results show that age and sex predict unintended smartphone access to online surveys. In general, smartphone access is very low and remains at 1–3% throughout the subgroups, except for the younger respondents. Among respondents younger than 35, the smartphone access rate to surveys is 5–6%. Contrary to our initial expectations, women appear more likely to access surveys on smartphones than men. However, both the effects of age and sex are in line with the findings of Peterson (2012) who reported that respondents younger than 35 were clearly more likely to access surveys on a smartphone, with females slightly more likely to respond using a mobile device than men.

For survey access using tablets, age, sex, working status, and housing composition act as significant predictors. Having paid work appears to have the strongest relation to tablet response, as those who work are twice more likely to access the survey using tablets than those who do not work. Also, people sharing their household with others are more likely to access the survey using tablets than those living alone.

Education level does not seem to have a clear relation to mobile web access to surveys, although there seems to be a slight tendency among the respondents with an intermediate and higher vocational education to access surveys using tablets. We could find no effect of the level of urbanization on the likelihood of unintended mobile response.

Unintended Mobile Response and the Technology Adoption Life Cycle

Not much research has been conducted on identifying the mobile web respondent as an early adopter of new technology although one might expect these two groups to be strongly correlated. Regarding general mobile usage, early adopters of mobile phones have been found to be typically young, well-educated men with high incomes (Fuchs & Busse, 2009), and early adopters of mobile web technology are assumed to differ similarly from the general population. To explore the role of the technology adoption life cycle and unintended mobile web access to surveys, we used data from two surveys in CentERpanel. We combined the Health Monitor data on response mode, collected in May 2013, with a survey fielded in the same panel in May 2013 about the progressiveness of the respondent. In the latter survey, the respondents were asked to report how conservative or progressive they considered themselves to be, on a scale from 1 = very conservative to 5 = very progressive. Of the 1,479 respondents, 73 (5%) had used a mobile device. Since the mobile group was relatively small, we did not differentiate between tablet and smartphone users in the analyses that follow.

The respondents who had accessed the Health Monitor using a mobile device considered themselves more progressive than those who had not. Among the mobile web respondents, 49% saw themselves as progressive or very progressive, while only 30% of the PC respondents termed themselves as progressive or very progressive, χ²(2, N = 1,479) = 12.69, p < .01; Cramer’s V = .09, p <.01. When conducting a logistic regression analysis, full model, χ²(7, N = 1,469) = 48.72, p < .001; Hosmer–Lemeshow: p = .90, the results confirm that mobile usage can not only be explained by the included socioeconomic factors (age, sex, education level, working status, urbanization and living conditions) but progressiveness remains a significant predictor (odds ratio 1.46, p = .02).

The respondents were also presented definitions of typical groups along the technology adoption curve and asked to indicate with which group they associated themselves most. The distributions for the groups differed significantly by response mode, χ²(4, N = 1,477) = 35.53, p < .001; Cramer’s V = .16, p < .001. The distributions are presented in Figure 3. As we had expected, the respondents who had accessed the survey using a mobile device identified themselves more often as pioneers or forerunners when it comes to adopting new technology.

Figure 3.

Respondent distributions along the technology adoption life cycle by survey access mode, May 2013.

Bias of Intended Mobile Response

In addition to the analyses of unintended mobile web access to surveys, we investigated the role of respondent characteristics when people are requested to complete a survey using a mobile device. For this analysis on intended mobile web responding, we used the data from an experiment¹ in the CentERpanel where smartphone users were asked to complete a survey using their smartphone. The experiment was fielded in Week 12 (March 22–26) of 2013. In this study, we knew not only the demographic variables of the respondents but also smartphone usage-related background variables.

Two logistic regression models were used to estimate the predictive, multivariate effects of the different characteristics on the response rate (see Table 3). The dependent variable indicated whether the respondent had used a smartphone to complete the questionnaire, as was requested, against using other devices and nonresponse. The input device for this analysis was based on the user agent string of the respondent’s last attempt to access the survey. Model 1 indicates the demographic and social behavior characteristics of the respondents. Here, age and education level have a significant effect on responding using smartphones. The younger and highly educated are more likely to respond using a smartphone.

Table 3.

Two Logistic Regression Models Predicting the Likelihood of Intended Mobile Responding, CentERpanel.

	Model 1		Model 2
	Coefficients	Odds	Coefficient	Odds
Age	−.02 (.01)**	0.98	−.00 (.01)	1.00
Sex: male	−.14 (.17)	0.87	−.20 (.18)	0.82
Working status: paid work	.21 (.19)	1.24	.22 (.20)	1.24
Urbanization (low to high)	.09 (.07)	1.09	.05 (.07)	1.06
Higher education (vocational or university)	.34 (.17)*	1.41	.24 (.18)	1.27
Living alone	.15 (.23)	1.16	.23 (.25)	1.26
Frequency visiting websites on smartphone (low to high)		—	.07 (.06)	1.08
Frequency reading e-mails on smartphone (low to high)		—	.22 (.05)***	1.25
Frequency participating surveys on smartphone (low to high)		—	.34 (.13)*	1.41
Touchscreen fingertip navigation		—	.62 (.28)*	1.86
Not sharing of smartphone		—	−.26 (.24)	0.77
Constant	−.12 (.35)	0.89	−2.64 (.62)***	0.07
Nagelkerke R ²	.04		.16
Number of observations	616		616

Note. References: female, no paid work, secondary/lower/intermediate education, living with other people, no true touchscreen navigation, smartphone sharing.

*p < .05. **p < .01. ***p < .001.

In Model 2, smartphone usage characteristics were added to the equation, including the frequency of visiting websites, and reading e-mails, using a smartphone. These frequencies were reported on a 7-point scale from “every day or almost every day” to “never done this.” We also included a variable indicating whether the smartphone was used solely by the respondent or shared with others and whether the respondent used a smartphone with a true touchscreen interface, navigated only by fingertips. Here, the characteristics related to mobile usage appear to function as intermediate variables. Although the demographic variables predict general smartphone usage, the variations in the latter in turn predict participation in a smartphone survey.

In the second analysis, the frequency of reading e-mails on a smartphone had a significant effect on responding, while visiting websites using a smartphone did not. As these variables are correlated with each other (r = .51, n = 622, p < .001), this seems to indicate that reading e-mails on a smartphone is a stronger predictor of responding than general mobile website usage. It is also important to note that the invitations to the respondents were sent by e-mail.

Another clear predictor was the type of user interface. Respondents with a solely fingertip-navigated touchscreen were more likely to respond than those without (odds ratio 1.86), implying that the respondents’ willingness to complete a mobile web survey is related to the type of device he or she is using. The modern type of smartphone interface is likely to increase the comfort in web task handling, and respondents with relatively sophisticated smartphones might generally be more inclined to use their device. Further, as could be expected, respondents completing other surveys using a smartphone were also more likely to participate in this survey.

Respondent Preferences

In Week 11 (March 15–19) of 2013, we asked the CentERpanel members with which device they prefer to access the Internet and to complete the panel surveys. On the whole, desktop computers and laptops are the most preferred devices among the panel members. In line with the proportion of unintended mobile survey response, 11% reported to prefer tablets for visiting website or completing surveys, while only 1–2% prefer smartphones.

These two preferences result in very similar outcomes, as shown in Table 4. The correlation between the two preferences is relatively strong, χ²(36, N = 2,090) = 4,580.09, p < .001; Cramer’s V = .60, p < .001. The device that respondents prefer to use for general web surfing is likely to be preferred for completing surveys as well. This is particularly true for people who prefer computers (92%) and laptops (88%) but slightly less so for those who prefer tablets (78%) and even less likely (38%) for those who prefer smartphones for general use.

Table 4.

Preferred Device for Visiting Websites and Completing Surveys, CentERpanel, March 2013.

Stch	To Visit Websites	To Complete Surveys
Desktop computer	48%	47%
Laptop	39%	36%
Tablet	11%	11%
Smartphone	1%	2%
SimPC	1%	1%
Cannot choose	1%	3%
Other	0.1%	0.1%
Total	100%	100%
N	2,090*	2,119

Note. Excluding 29 respondents who reported not to visit any websites, except for completing the questionnaires in the panel.

When examining the preferred device to complete surveys by demographic characteristics, we find clear differences by subgroup (Table 5). Desktops are the most preferred device for men and laptops for women. Regarding tablets, women appear to prefer them more often than men. Tablets are also mainly favored by the higher educated and those between 25 and 44 years old. For smartphones, a different pattern arises. Any interest for smartphone surveys can be found only among the youth and the younger adults. The preference for smartphone surveys among people more than 45 years is strikingly close to nonexistent. Under the 24-year-old respondents, on the contrary, smartphones are more often preferred (8%) than tablets (2%), while tablets are more preferred than smartphones in all other age groups. Logistic regression analysis confirms that age, sex, working status, education level, and housing composition all significantly predict respondent preference for tablets as input device (Table 6). For smartphones, only age and education significantly contribute to the model.

Table 5.

Preferred Device for Completing Surveys by Demographic Characteristics, CentERpanel, March 2013.

	Sex***		Age***						Education**^a						Working Status***		Urbanization					House hold Composition***^b	House hold Composition***^b			House hold Composition***^b
	Male	Female	16– 24	25– 34	35– 44	45– 54	55– 64	65+	A	B	C	D	E	F	Paid work	No paid work	Not urban	Little urban	Some what urban	Strongly urban	Highly urban	G	H	I	J	K	Total
Desktop computer	54%	39%	34%	25%	34%	47%	53%	59%	50%	53%	49%	46%	43%	42%	40%	54%	51%	48%	47%	46%	41%	46%	53%	40%	31%	51%	47%
Laptop	32%	41%	52%	41%	38%	38%	35%	32%	36%	35%	39%	41%	36%	33%	38%	35%	34%	37%	37%	36%	39%	40%	32%	38%	49%	44%	36%
Tablet	10%	14%	2%	24%	19%	11%	10%	6%	7%	9%	8%	10%	14%	18%	16%	7%	10%	10%	11%	13%	13%	8%	11%	15%	15%	0%	11%
Smartphone	2%	3%	8%	8%	5%	0%	1%	0%	2%	0%	2%	2%	3%	3%	1%	2%	2%	2%	2%	2%	2%	1%	4%	0%	2%	2%
SimPC	1%	1%	0%	0%	0%	0%	0%	2%	1%	1%	1%	0%	1%	0%	0%	1%	0%	1%	0%	1%	1%	2%	1%	0%	0%	0%	1%
Cannot choose	2%	3%	4%	3%	4%	3%	1%	2%	5%	3%	1%	1%	3%	3%	3%	2%	3%	2%	2%	2%	4%	2%	2%	3%	5%	2%	3%
Other	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	0%	1%	0%	0%	0%	0%	0%	0%	0%	0%	0%	1%	0%	0%	0%
Total	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
N	1,126	993	95	212	331	380	472	629	105	532	246	352	590	291	1,049	1,070	365	459	428	562	289	400	935	661	80	43	2,119

Note. ^aEducation: A = primary school, B = lower secondary education; C = higher secondary education, D = intermediate vocational education, E = higher vocational education, F = university. The education level was missing for three respondents, which are excluded here. ^bHousehold composition: G = living alone, H = living together, without children, I = living together, with children, J = single, with children, K = other.

*p < .05. **p < .01. ***p < .001 (Monte Carlo two sided, 99% confidence interval).

Table 6.

Multinomial Logistic Regression Model Predicting the Likelihood of Preference for Mobile Devices When Completing Surveys, CentERpanel, March 2013.

	Tablet		Smartphone
	Coefficients	Odds	Coefficients	Odds
Age	−.01 (.01)**	0.99	−.10 (.01)***	0.91
Sex: male	−.44 (.14)**	.65	−.34 (.33)	.72
Working status: paid work	.60 (.17)***	1.81	.72 (.41)	2.05
Urbanization (low to high)	.07 (.06)	1.07	−.01 (.13)	0.99
Higher education (vocational or university)	.57 (.14)***	1.77	.78 (.36)*	2.17
Living alone	−.55 (.21)*	0.58	.39 (.41)	1.48
Number of observations	2,100		2,100

Note. Reference group: other preferences, including desktop, laptop, SimPC, cannot choose, and other. Missing cases: 19. References: female, no paid work, secondary/lower/intermediate education, living with other people.

*p < .05. **p < .01. ***p < .001.

Conclusions, Suggestions, and Discussion

Our findings indicate an increasing scattering of the web sample, as a significant group of respondents refuses to take web surveys using computers but uses mobile devices instead, whether or not the surveys are adapted for mobile browsers. The recent, clear increase in the unintended mobile access rate to web surveys is mainly attributable to tablets, while the overall smartphone access remains at a couple of percentage. Accordingly, the question arises whether web panels as we know them—respondents completing web surveys on a computer—still exist.

In web surveys, mobile respondents differ from PC users by respondent background characteristics. The analysis also reveals that tablet users differ from those using smartphones to complete surveys. Smartphones are used to a slight extent among the young, while tablets are used mostly by working adults between 25 and 54 years old. Both devices are used more among females than males to complete surveys. Respondent preferences for an ideal device to complete online surveys show similar differences: age, sex, working status, education level, and housing composition predict tablet preference, but only age and education predict the preference for smartphones.

We also find that there are differences within the mobile user group related to device usage and that this affects the likelihood of responding. When smartphone web response is intended, those having the most advanced input interface and those who use the smartphone frequently for purposes such as reading e-mails are more likely to respond.

Although smartphones are still in minority as access method to web surveys, the tablet user group is already too large to be ignored in online panels. As tablets are fairly comparable to computers in terms of screen size, their use does not necessarily lead to a significant increase in respondent burden when completing unadapted web surveys. However, they run different operating systems than desktops or laptops and the touchscreen input interface differs from the mouse–keyboard interface. Therefore, we recommend testing web surveys on tablets, at the very least. A more advanced step could be to investigate whether the survey design should be adapted such that it can take account of the differences in input interfaces.

It would be interesting to examine the reasons for the low usage rates of smartphones compared to tablets in web surveys, as this cannot be explained by the general Internet access rates using smartphones. In unadapted web surveys, this would seem likely to be attributed to technical reasons such as cumbersome task handling on the small screens. It could also be that surveys are associated with a longer task, which is preferred to be completed under similar conditions as tasks on a computer, namely in a sitting position and a more stationary than on-the-go mode, especially in a panel setting.

Since the mobile web respondent group is still dominated by progressive forerunners in adopting new technology, unintended mobile web response could be expected to continue to increase as the technology matures and mobile Internet penetration in the general population rises. The era of a PC-based web environment has gone, but it is possible to detect the type of device the respondents are using and, if necessary, to act on it. It seems more necessary than ever to monitor how people access online surveys.

Footnotes

Authors’ Notes

The authors gratefully acknowledge Corrie Vis, Marcel Das, and Eric Balster for their invaluable encouragement and support.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Note

References

Boreham

Wijnant

(2013). Developing a web-smartphone-telephone questionnaire. Conference paper. IBUC 2013 15th International Blaise Users Conference. Retrieved from http://www.blaiseusers.org/2013/papers/4c.pdf

Bosnjak

Poggio

Becker

K. R.

Funke

Wachenfeld

Fischer

(2013). Online survey participation via mobile devices. Conference Paper, The American Association for Public Opinion Research (AAPOR) 68th Annual Conference, 2013, Boston.

Buskirk

T. D.

Andrus

(2012). Smart surveys for smart phones: Exploring various approaches for conducting online mobile surveys via smartphones. Survey Practice, 5 (1).

Callegaro

(2010). Do you know which device your respondent has used to take your online survey? Survey Practice, 3.

Chae

Kim

(2004). Do size and structure matter to mobile users? An empirical study of the effects of screen size, information structure, and task complexity on user activities with standard web phones. Behaviour & Information Technology, 23, 165–181.

Couper

M. P.

(2013). Is the sky falling? New technology, changing media, and the future of surveys. Keynote speech presented at 5th European Survey Research Conference, Ljubljana, 18 July 2013. Retrieved from http://www.europeansurveyresearch.org/conference/couper

Couper

M. P.

Tourangeau

Conrad

F. G.

Crawford

S. D.

(2004). What they see is what we get: Response options for web surveys. Social Science Computer Review, 22, 111–127.

Crawford

S. D.

Couper

M. P.

Lamias

M. J.

(2001). Web surveys: Perceptions of burden. Social Science Computer Review, 19, 146–162.

de Bruijne

Wijnant

(2013). Comparing survey results obtained via mobile devices and computers: An experiment with a mobile web survey on a heterogeneous group of mobile devices versus a computer-assisted web survey. Social Science Computer Review, 31, 483–505.

10.

Fuchs

(2012). Nonresponse and panel attrition in a mobile phone panel survey. Conference paper, presented at the Federal Committee on Statistical Methodology Research Conference in Washington, January 10–12, 2012.

11.

Fuchs

Busse

(2009). The coverage bias of mobile web surveys across european countries. International Journal of Internet Science, 4, 21–33.

12.

Guidry

K. R.

(2012). Response quality and demographic characteristics of respondents using a mobile device on a web-based survey. Conference paper, presented at the American Association for Public Opinion Research (AAPOR), 67th Annual Conference, in Orlando, Florida, May 18, 2012.

13.

Jones

Buchanan

Thimbleby

(2003). Improving web search on small screen devices. Interacting with Computers, 15, 479–495.

14.

Jones

Marsden

Mohd-Nasir

Boone

Buchanan

(1999). Improving web interaction on small displays. Computer Networks, 31, 1129–1137.

15.

Mavletova

(2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31.

16.

Millar

M. M.

Dillman

D. A.

(2012). Encouraging survey response via smartphones: Effects on respondents’ use of mobile devices and survey response rates. Survey Practice, 5.

17.

Peterson

(2012). Unintended mobile respondents. Conference paper, CASRO Technology Conference, May 31 2012, New York City.

18.

Peytchev

Hill

C. A.

(2010). Experiments in mobile web survey design: Similarities to other modes and unique considerations. Social Science Computer Review, 28, 319–335.

19.

Poggio

Bosnjak

Weyandt

(2013). Online survey participation via mobile devices: Implications for nonresponse. Conference paper. ESRA Conference, Ljubljana, July 15-19, 2013.

20.

Porter

S. R.

(2004). Raising response rates: What works? New Directions for Institutional Research, 121, 5–21.

21.

Stapleton

C. E.

(2013). The smart(Phone) way to collect survey data. Survey Practice, 6. Retrieved from http://www.surveypractice.org/index.php/SurveyPractice/article/view/75

22.

StatCounter Global Stats. (2013). Mobile vs. desktop. Retrieved October 25, 2013, from http://gs.statcounter.com/#mobile_vs_desktop-ww-monthly-201201-201309

23.

Statistics Netherlands. (2012). Mobile internet use continues to grow. [Press release], PB12-060, 23 October 2012. Retrieved June 2, 2013, from http://www.cbs.nl/en-GB/menu/themas/vrije-tijd-cultuur/publicaties/artikelen/archief/2012/2012-060-pb.htm

24.

Toepoel

Lugtig

(2014). What happens if you offer a mobile option to your web panel? Evidence from a probability-based panel of Internet users. Social Science Computer Review 32(4). doi: 10.1177/0894439313510482

25.

Watters

Duffy

(2003). Using large tables on small display devices. International Journal of Human-Computer Studies, 58, 21–37.

26.

Wells

Bailey

J. T.

Link

M. W.

(2013). Filling the void: Gaining a better understanding of tablet-based surveys. Survey Practice, 6.

27.

Wells

Bailey

J. T.

Link

M. W.

(2014). Comparison of smartphone and online computer survey administration. Social Science Computer Review 32(2), 238–255.

28.

World Values Survey. (2011). Wave 2010-2012 Revised Oct 2011. Retrieved June 14, 2013, from http://www.worldvaluessurvey.org/wvs/articles/folder_published/article_base_136/files/WVS_2010-2012_REVISED_OCT_2011.pdf