Does Increasing Mobile Device Coverage Reduce Heterogeneity in Completing Web Surveys on Smartphones?

Abstract

Mobile coverage recently has reached an all-time high, and in most countries, high-speed Internet connections are widely available. Due to technological development, smartphones and tablets have become increasingly popular. Accordingly, we have observed an increasing use of mobile devices to complete web surveys and, hence, survey methodologists have shifted their attention to the challenges that stem from this development. The present study investigated whether the growing use of smartphones has decreased how systematically this choice of device varies between groups of respondents (i.e., how selective smartphone usage for completing web surveys is). We collected a data set of 18,520 respondents from 18 web surveys that were fielded in Germany between 2012 and 2016. Based on these data, we show that while the use of smartphones to complete web surveys has considerably increased over time, selectivity with respect to using this device has remained stable.

Keywords

mobile devices web surveys smartphone survey error selectivity

In recent years, the use of smartphones to complete web surveys has attracted the widespread attention of survey methodologists. This popularity is motivated partially by a population-wide trend toward increasing smartphones coverage, and the observation that the use of smartphones may affect response behavior, and thus data quality. For example, in 2011, only 19% of Europeans used a smartphone to access the Internet, but by 2016, this number had risen to 56% (Eurostat, 2017). One consequence of the availability and usage of mobile Internet options is an increase in surveys that are filled out on smartphones (Gummer & Roßmann, 2015; Lugtig, Toepoel, & Amin, 2016; Revilla, Toninelli, Ochoa, & Loewe, 2016).

Web survey research has focused on the impacts of mobile device usage on survey errors and data quality and has found some significant effects. For example, some research investigated the effects on completion rates, finding that mobile respondents had significantly lower completion rates (Mavletova, 2013; Sommer, Diedenhofen, & Musch, 2017), including break-off rates for smartphones that were 2–3 times higher than for personal computers (PCs; Bosnjak et al., 2013; Poggio, Bosnjak, & Weyandt, 2015). A number of studies also studied the effects of mobile devices on response times, finding that participants on mobile devices needed longer to complete a survey (Antoun, Couper, & Conrad, 2017; Gummer & Roßmann, 2015; Revilla, Toninelli, & Ochoa, 2016; Sommer et al., 2017). Schlosser and Mays (2017) also found longer completion times for mobiles, although controlling for the type of Internet connection and hardware minimized those differences. In addition, Couper and Peterson (2016) found that only one fifth of the time difference could be accounted for by transmission time, whereas other characteristics were responsible for the remaining four fifths. Keusch and Yan (2017) found significantly longer response times both for iPhone users who voluntarily used their device to do a survey and the respondents who were told to switch to their mobile device after starting a survey on their PC. Regarding the length and quality of open answers, Bosnjak et al. (2013) found that mobile respondents answered more open questions than PC users when dropping out of a survey, whereas Mavletova (2013) and Revilla, Toninelli, and Ochoa (2016) found the open answers of mobile respondents to be significantly shorter. In addition, Antoun, Couper, and Conrad (2017) found that respondents gave longer answers to open questions on their smartphone, hypothesizing that these respondents may have become more comfortable with writing on their smartphones or using technological support such as voice-to-text typing. Last, concerning missingness on the item level, Tourangeau et al. (2017), Schlosser and Mays (2017), and Toepoel and Lugtig (2014) reported no differences in item nonresponse rates between mobile and PC respondents. Based on these findings, some researchers have stressed the importance of survey design adaptions to minimize the effects of different devices (de Bruijne & Wijnant, 2014; Mavletova & Couper, 2016).

A second line of research has examined the coverage and usage patterns of smartphones to complete web surveys. For example, de Bruijne and Wijnant (2013) found that smartphones were used by more highly educated, younger members of the LISS panel and that men were more likely to use a smartphone than women. At the same time, they also found that out of all smartphone and tablet owners, only a tiny fraction of 0.9% completed surveys on their smartphone, 13.5% on their tablet, and 0.8% on both of those devices (de Bruijne & Wijnant, 2013, p. 486). Along these same lines, Toepoel and Lugtig (2014) found that “age, income, household composition, and household size are predictors of mobile completion” but that “gender, education, and urbanization did not affect mobile completion” (pp. 548–549) and also pointed to younger respondents with higher income and living in smaller households as the first wave of smartphone adopters for completing surveys. In addition, Revilla, Toninelli, Ochoa, and Loewe (2016) reported ambiguous findings with respect to the use of mobile devices in web surveys in seven countries; they did not find any explanatory power for education, and moreover, they found inconsistent effects for age and gender across the seven countries.

From these two lines of research, we have learned that survey errors and data quality may be related to the use of smartphones to complete web surveys. The choice of using this particular device seems to be selective with respect to which groups of respondents are more prone to complete a survey using such a mobile device. In other words, the choice of a mobile device for completing surveys varies systematically between groups of respondents. On the one hand, this could lead to survey errors that are heterogeneously distributed among different groups of respondents; yet on the other hand, it seems entirely reasonable to assume that after an initial period of selective use by innovators and early adopters (Geroski, 2000; Rogers, 2003), the new technology (i.e., innovation) spreads to the broader public, and thus the selectivity in mobile device usage decreases over time. Following this line of reasoning, Lee and Lee (2014) and Lee, Lee, and Chan-Olmsted (2017) have built on previous studies that propose an S-shaped curve for mobile technology diffusion and have used Gompertz models to depict the early diffusion of smartphones and tablets in different countries. These authors have reported that after the initial phase during which early adopters use smartphones, network penetration increases and smartphone prices decrease, which leads to a higher diffusion of smartphones in the general public.

As argued previously, some research has reported rising rates of mobile device usage for completing web surveys. However, research is surprisingly lacking with respect to how the selectivity of using smartphones has changed over time. Participation via smartphones (i.e., selectivity) has at least two elements: ownership of a smartphone (i.e., coverage) and the actual device choice. Previous research in this area has focused primarily on different coverage rates. For example, Fuchs and Busse (2009) analyzed smartphone penetration rates, and thus coverage bias in web surveys within Europe between 2005 and 2007. Although this research provides important insights regarding coverage bias, the availability of smartphones to respondents does not necessarily imply that they actually will use a smartphone to complete web surveys. Therefore, Fuchs and Busse (2009) rightly caution “that the mere coverage of the population with a new technology does not imply that all respondents who have access to the mobile Internet are actually capable and willing to use it in order to take a survey” (p. 30). Similarly, Mohorko, de Leeuw, and Hox (2013) investigated Internet coverage and coverage bias in Europe, finding that “in general, coverage bias decreases when education, employment, life expectancy, and urbanicity increase” and that the “differences in age, sex, education, and life satisfaction between those with and without Internet access are diminishing” (p. 618).¹ Again, a higher Internet coverage does not lead necessarily to lower selectivity; rather, it provides a necessary basic prerequisite.

In the present study, we aim to contribute to this research lacuna by addressing two research questions. First, we investigate whether growing Internet coverage and the population-wide availability of smartphones have resulted in higher smartphone usage for completing surveys, and second, whether the increasing use of smartphones to complete surveys has diminished the selectivity of smartphone usage over time. In other words, we ask the following important question—Is the self-selectivity of using smartphones to complete web surveys reduced as these mobile devices become more available and commonly used in daily life? To address this research gap, we relied on web surveys that were fielded in Germany. Germany is an illustrative case for our study, since it is an advanced economy that has reached high levels of Internet and smartphone coverage in recent years (Poushter, 2016), which are the basic prerequisites for a potentially diminishing selectivity. Since we were interested in the overall selectivity of using a smartphone to complete surveys, we have not attempted to disentangle the elements of ownership and device choice, and thus we leave this challenge to future studies.

Data and Method

To investigate our research questions, we pooled 18 web surveys that were conducted between May 2012 and December 2016 as part of the German Longitudinal Election Study (Rattinger, Roßteutscher, Schmitt-Beck, Weßels, & Wolf, 2012–2016). The questionnaires featured a common core of similar questions that covered topics related to political attitudes and behaviors. Each of these surveys was sampled based on the same quotas from a large German offline-recruited probability-based access panel. On average, 1,029 respondents participated per survey, which provided us a total of 18,520 cases. Detailed figures on respondents who accepted the survey invitation, the numbers of screen-outs and break-offs, and the break-off rate of each survey as proposed by Callegaro and DiSogra (2008) are provided in Table 1.

Table 1.

Field Period and Outcome Rates of 18 Web Surveys.

Survey	Field Period	N			Break-off Rate in %
Survey	Field Period	Accepted Invitation	Screened Out	Break-off	Break-off Rate in %
1	May 2 to May 15, 2012	1,709	427	266	20.7
2	September 17 to October 1, 2012	1,517	254	188	14.9
3	January 4 to January 19, 2013	1,532	326	172	14.3
4	May 24 to June 8, 2013	1,626	350	228	17.9
5	September 9 to September 21, 2013	1,373	184	177	14.9
6	November 29 to December 13, 2013	1,648	384	215	17.0
7	February 21 to March 7, 2014	1,493	265	205	16.7
8	May 9 to May 23, 2014	1,446	199	203	16.3
9	August 29 to September 13, 2014	1,404	231	162	13.8
10	November 21 to December 5, 2014	1,446	174	253	19.9
11	February 27 to March 13, 2015	1,375	165	181	15.0
12	June 5 to June 19, 2015	1,569	388	162	13.7
13	September 11 to September 25, 2015	1,460	282	151	12.8
14	December 4 to December 18, 2015	1,563	346	186	15.3
15	February 26 to March 11, 2016	1,498	320	163	13.8
16	June 3 to June 17, 2016	1,509	278	208	16.9
17	August 19 to September 2, 2016	1,544	337	184	15.2
18	December 2 to December 16, 2016	1,349	123	204	16.6

To assess the levels of selectivity with respect to the use of smartphones in each of our 18 surveys, we fitted separate logistic regression models with “using a smartphone to complete the survey” as the dependent variable. To assess what device respondents used to complete the survey, we drew on the user agent strings that the survey software automatically collected (for a discussion of these strings, see Roßmann, Gummer, and Kaczmirek, in press). We used the Stata module parseuas (Roßmann & Gummer, 2014) to parse the strings into useable information and to identify whether respondents used a PC, tablet, or smartphone to complete the survey. Based on this information, we created a binary variable on using a smartphone (0 = no, 1 = yes).

With respect to the independent variables, we added a set of sociodemographic and key substantive variables from the surveys. We operationalized sex (0 = male, 1 = female) and region of residence (0 = East, 1 = West) as dummy variables. We categorized age into four groups ranging from “20–29 years,” “30–39 years,” “40–49 years,” to “50 years and older.” We also categorized education, interest in politics, and strength of party identification into “low,” “intermediate,” and “high.” We metrically coded participation in political discussions from not at all to 7 days a week. Table 2 provides descriptive statistics of all the variables in the pooled data set.

Table 2.

Descriptive Statistics on the Pooled Data of 18 Web Surveys.

Variable	M	SD	Min.	Max.
Independent variables
Region of residence: West	0.81	0.39	0	1
Sex: Female	0.49	0.50	0	1
Age
20–29 Years	0.22	0.42	0	1
30–39 Years	0.19	0.39	0	1
40–49 Years	0.24	0.43	0	1
50 Years and older	0.34	0.47	0	1
Education
Low	0.26	0.44	0	1
Intermediate	0.40	0.49	0	1
High	0.34	0.47	0	1
Interest in politics
Low	0.19	0.40	0	1
Intermediate	0.43	0.50	0	1
High	0.37	0.48	0	1
Political discussions	2.34	1.98	0	7
Strength of party identification
Low	0.24	0.43	0	1
Intermediate	0.31	0.46	0	1
High	0.45	0.50	0	1
Dependent variable
Smartphone use in survey: Yes	0.12	0.33	0	1

Note. Statistics are calculated at the respondent level with n = number of respondents. M = mean; SD = standard deviation.

If selectivity in smartphone usage decreases over time (i.e., across the surveys), significant differences between groups should vanish or become less pronounced. In other words, the effects of the covariates in our 18 regression models should diminish, and the predicted propensities for smartphone usage should become similar for different subgroups.² To assess these indicators of selectivity, we calculated the average marginal effects for all regressions. Due to the difficulties associated with comparing the coefficients of the logistic models (Best & Wolf, 2015), we further compared the differences in the predicted probabilities of the selected groups and analyzed how (and if) these differences changed across time (i.e., surveys). To indicate significant differences between predicted probabilities, we calculated 95% confidence intervals (CIs) for each probability. Overlapping CIs of the predicted probabilities indicated nonsignificant differences, whereas nonoverlapping CIs indicated significant differences (with α = .05).

Results

Based on the data we used, Figure 1 shows an increasing use of smartphones and tablets to do web surveys, which also suggests that the use of PCs to complete web surveys has declined. This finding is in line with what we know from previous studies on the use of mobile devices for completing surveys (e.g., Lugtig & Toepoel, 2016; Revilla, Toninelli, Ochoa, & Loewe, 2016). Moreover, the increasing use of smartphones coincides with ongoing technological development—the increasing availability of mobile devices to the broad public and increasing Internet coverage facilitating usage. Figure 1 also relies on supplemental data on Internet access in households and via smartphones provided by Eurostat (2017). Supporting our reasoning, we found that Internet access in households remains nearly stable at an already high level, whereas Internet access via smartphones dramatically increased between 2012 and 2016.

Figure 1.

Device usage in 18 web surveys between 2012 and 2016.

To shed light on whether the increasing use of mobile devices has reduced selectivity in the usage patterns of these devices, we report the results of our regression models in Table 3. The model fits showed that selectivity occurred in all of our 18 surveys because a moderate share of variation was explained by the covariates (pseudo R ² between .074 and .175). If selectivity was not present, the explanatory power of the models should have been lower. This finding is supported partially by the specific independent variables. In line with previous research (de Bruijne & Wijnant, 2013; Toepoel & Lugtig, 2014), in our surveys, we found a consistent effect of age on the likelihood of using a smartphone. Younger respondents were more likely to complete a survey on a smartphone—we found this effect in each of the 18 models. Figure 2 details the predicted probabilities for age groups across the surveys. Surprisingly, the difference in probabilities between age groups did not diminish and remained at a stable level. Moreover, Figure 2 also shows a generally increased likelihood across all age groups to use a smartphone to complete web surveys. In other words, as mobile devices became increasingly available and were used by the broader public, a gap remained between age groups with respect to using these devices to complete web surveys.

Table 3.

Use of Smartphones for Completing Web Questionnaires: Results From Logistic Regressions in 18 Surveys.

Survey	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18
Survey	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)	AME (SE)
Sex
Male	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref
Female	−.015 (.012)	.023 (.014)	−.013 (.013)	−.009 (.014)	.017 (.015)	.029 (.017)	.042* (.020)	.015 (.020)	.006 (.020)	.038 (.022)	.005 (.023)	.052* (.021)	.040 (.022)	.049* (.023)	.021 (.023)	.035 (.025)	.061* (.026)	.054* (.025)
Age
29 Years or younger	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref
30–39 Years	−.008 (.023)	−.037 (.027)	−.047 (.024)	−.073* (.032)	−.042 (.029)	−.037 (.038)	−.084* (.038)	−.084* (.039)	−.095** (.036)	−.096* (.044)	−.100* (.041)	−.051 (.046)	−.075 (.046)	−.107* (.046)	−.086 (.047)	−.016 (.050)	−.089 (.049)	−.034 (.047)
40 –49 Years	−.061*** (.018)	−.071** (.024)	−.067** (.023)	−.125*** (.030)	−.083** (.026)	−.173*** (.031)	−.171*** (.034)	−.178*** (.034)	−.101** (.036)	−.225*** (.040)	−.206*** (.036)	−.190*** (.038)	−.162*** (.041)	−.249*** (.039)	−.199*** (.042)	−.248*** (.040)	−.266*** (.044)	−.171*** (.041)
50 Years and older	−.058** (.019)	−.088*** (.023)	−.091*** (.021)	−.138*** (.029)	−.112*** (.024)	−.183*** (.031)	−.225*** (.031)	−.235*** (.031)	−.186*** (.032)	−.277*** (.037)	−.277*** (.033)	−.256*** (.035)	−.287*** (.035)	−.328*** (.035)	−.291*** (.038)	−.271*** (.038)	−.361*** (.040)	−.276*** (.036)
Education
Low	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref
Intermediate	−.038 (.036)	−.029 (.020)	−.005 (.018)	.011 (.020)	.027 (.019)	.006 (.026)	.067** (.022)	−.021 (.029)	.019 (.024)	−.038 (.033)	−.057 (.029)	−.037 (.030)	.012 (.030)	−.029 (.031)	−.023 (.034)	.035 (.032)	−.028 (.037)	.043 (.034)
High	−.040 (.036)	−.023 (.022)	−.004 (.019)	−.004 (.020)	.006 (.018)	−.022 (.025)	.043 (.025)	−.041 (.030)	.036 (.028)	−.037 (.035)	−.077* (.031)	−.039 (.031)	.001 (.030)	−.060 (.031)	−.045 (.033)	.012 (.032)	−.068 (.037)	−.044 (.034)
Region
East	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref
West	.007 (.014)	−.014 (.018)	.015 (.015)	.015 (.014)	−.012 (.017)	.038* (.017)	.042* (.021)	.015 (.024)	.037 (.026)	.006 (.025)	.047 (.025)	.010 (.025)	−.006 (.027)	.010 (.026)	−.026 (.028)	.015 (.029)	−.026 (.031)	−.045 (.030)
Political discussions	−.004 (.004)	−.002 (.005)	−.001 (.005)	−.002 (.005)	.003 (.004)	.011* (.005)	.009 (.006)	−.000 (.006)	−.002 (.006)	.003 (.007)	.002 (.007)	−.010 (.007)	−.003 (.006)	.002 (.006)	−.002 (.006)	.005 (.007)	.009 (.007)	.001 (.007)
Interest in politics
Low	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref
Intermediate	−.017 (.017)	−.003 (.015)	−.050* (.020)	.006 (.018)	.003 (.018)	.019 (.023)	−.041 (.028)	−.024 (.026)	−.043 (.029)	−.019 (.029)	.010 (.029)	.053* (.023)	−.053 (.030)	−.016 (.030)	.051 (.030)	−.027 (.035)	−.055 (.040)	−.010 (.035)
High	−.015 (.020)	.027 (.026)	−.046* (.023)	−.018 (.021)	.003 (.021)	−.025 (.025)	−.067* (.033)	−.029 (.031)	−.063 (.035)	−.047 (.035)	−.062 (.034)	.067* (.033)	−.022 (.038)	−.030 (.036)	.011 (.035)	−.056 (.041)	−.017 (.047)	−.013 (.040)
Strength of party identification
Low	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref	Ref
Intermediate	.028 (.015)	.004 (.019)	.016 (.014)	−.001 (.015)	−.008 (.020)	−.018 (.022)	.018 (.025)	.014 (.024)	−.047 (.027)	.033 (.028)	−.005 (.028)	−.035 (.027)	.012 (.027)	−.013 (.029)	−.013 (.029)	−.008 (.033)	.002 (.035)	.009 (.031)
High	.000 (.013)	−.019 (.016)	.021 (.016)	.022 (.018)	−.019 (.018)	−.016 (.021)	.010 (.023)	−.002 (.023)	−.040 (.028)	−.000 (.025)	−.001 (.028)	−.022 (.027)	.041 (.028)	−.033 (.027)	.026 (.029)	−.043 (.032)	−.015 (.033)	.014 (.030)
Pseudo R ²	.127	.074	.132	.152	.118	.175	.167	.133	.122	.139	.129	.139	.147	.160	.112	.120	.123	.112
N	1,003	1,059	1,008	1,023	995	1,030	1,004	1,025	997	1,002	1,017	996	1,007	1,016	994	1,003	1,015	1,007

Note. AME = averaged marginal effect; SE = standard error; Ref = reference category.

*p < .05. **p < .01. ***p < .001.

Figure 2.

Predicted probabilities to use a smartphone to complete a survey by age.

Previous studies have reported nonconclusive findings regarding other factors for smartphone use such as sex, education, and region of residence. In our models, the female respondents appeared to be more likely to use smartphones compared to men, but this effect was only significant in 5 of the 18 models. A reasonable pattern would have been finding this effect in the first five surveys and then seeing it diminish afterward, but this was not the case, since the five significant effects appeared to vary nonsystematically over time. To investigate the matter further, we again plotted the predicted probabilities in Figure 3. The visual presentation illustrates our previous findings. Overall, the likelihood of using a smartphone to complete web surveys increased across surveys. However, the differences in propensities between male and female respondents did not change. Given the inconsistent pattern of these effects and that previous research had presented similar findings (e.g., Revilla, Toninelli, Ochoa, & Loewe, 2016), further research is warranted to shed light on the causal mechanisms that lie behind these effects.

Figure 3.

Predicted probabilities to use a smartphone to complete a survey by gender.

Our analysis adds to the ambiguous findings from prior studies. We did not find selection effects for different education groups, and the region of residence in Germany did not have a consistent effect. In addition, our analysis did not show any trends across time with respect to how intergroup differences changed (see predicted probability plots in Appendix Figures A1 and A2).

Overall, we found a general pattern that using smartphones to complete web surveys has become more likely. This finding is in line with previous studies that investigated either how the coverage and availability of mobile devices developed or who were the users of smartphones to complete web surveys. We also found that younger respondents were more likely to adopt this new technology for completing surveys compared to older respondents. This pattern did not change across time despite the fact that the overall likelihood increased. We did not find a distinct pattern for other covariates that previous studies had reported to facilitate smartphone use, despite our findings with respect to the mixed effects for respondents’ gender. In summary, our study illustrates that a macrolevel process (the increasing spread of a technological innovation) does not manifest necessarily in a microlevel phenomenon (decreasing selectivity).

Conclusion

The present study set out to gain knowledge about how smartphone usage and patterns of use developed with respect to web surveys over the recent years. This line of research was motivated by the observation that previous studies had reported increasing coverage rates of mobile devices in the general public and web surveys in particular. However, none of these studies focused on whether the increasing use of smartphones had manifested in a decreasing selectivity of use. Methodological studies frequently have cautioned that completing surveys via a smartphone may result in lower data quality if the survey design was not adapted properly (de Bruijne & Wijnant, 2013). If smartphone use to complete web surveys varies systematically (i.e., is selective), survey errors also will vary across different respondent groups, what may be misinterpreted as substantive differences between groups.

First, in line with previous research, we found that the use of smartphones to complete web surveys has risen over recent years. Second, while the general likelihood of using a smartphone to complete a survey has increased over time, the selectivity for doing so has remained stable over a period of 5 years. Despite the growing and persistent popularity of mobile devices in everyday life, the use of smartphones has not become an accepted practice for all our respondents. In other words, the problem remains that specific groups are more prone to using mobile devices compared to others and, in turn, are more likely subject to device-specific influences with respect to their response behavior. The factor that was outstanding and showed a consistent pattern across all 18 surveys was age. While this finding may appear straightforward at first and even seem like selectivity may only exist with respect to the age dimension, it is a crucial indicator of a more serious problem. Age (as well as cohort and birth year) is commonly and frequently used in the social sciences as a control variable because it is assumed to be correlated with a variety of constructs in which a researcher may have an interest. If specific age groups are more likely to answer a survey via a smartphone, and thus are more prone to measurement error, specific groups of respondents with characteristics correlated with age also are more prone to this error. For instance, assume that younger respondents are more likely to use smartphones and, hence, their measures are more likely to be of lower quality. If age is now positively correlated with a variable such as computational skills, the respondents who have these higher skills are also more likely to provide data of lower quality. In other words, due to the correlation with age, measurement error spills over into other variables.

Given the pertinacity of selectivity, our results have implications for the ongoing discussion about pushing respondents to the web mode (McMaster, LeardMann, Speigle, & Dillman, 2017; Millar & Dillman, 2011). Compared to face-to-face interviews, interviewing respondents online has an appeal due to lower costs and other beneficial side effects (e.g., reducing interview effects, more flexibility, and implementation of visual cues and media applications). However, despite the fact that mobile devices are available and frequently are used to complete surveys, pushing respondents to the web may result in specific subgroups using specific devices, which, again, could result in heterogeneous survey errors in the data. To better evaluate the potential outcome of push-to-web approaches, we believe it is necessary to further shed light on which error mobile use might induce and what drives the choice of a device to complete a survey. In this context, it might be interesting to push specific respondent groups to PCs, tablets, or even smartphones to balance any potential survey errors in a sample.

The findings of the present study yield opportunities for future research. First, while we were in the fortunate position to have 18 comparable surveys that enabled us to assess smartphone use over 5 years, it might be interesting to extend our approach to investigate larger time intervals. Future research that tackles this question will be confronted, however, with the challenge to ensure the comparability of different surveys. Second, in the present study, we focused on web surveys in Germany. One reason for our choice was that Germany is an advanced economy that has reached an all-time high in mobile coverage rates and thus fulfills the requirements for testing the assumption of decreasing selectivity. In addition, Internet access in Germany has reached a level comparable to other advanced economies such as the United Kingdom, the United States, and Canada (Poushter, 2016, p. 9), and thus web surveys can be considered an important survey mode for covering larger parts of the population. However, relying on data from Germany alone may impair the generalizability of our findings. If comparable data sets can be obtained for other countries, we would consider a cross-national comparison to be a formidable challenge for future studies. In our view, these studies should focus on country-level differences that influence adaption rates and investigate what influences subgroups of respondents to use mobile devices (i.e., what drives selectivity). Third, the present study covered 5 years, and as our initial analysis showed, in our surveys, this period coincided with the rise in smartphone usage rates necessary to address our research questions. A long-term analysis of trends in mobile device use seems mandatory to monitor whether selectivity remains stable, diminishes, or even starts to rise. Fourth and finally, with the present study, we showed that selectivity remains an important challenge for survey methodologists and practitioners. In our analyses, we mostly were concerned with measuring selectivity and whether it changed between surveys. Further advancing this line of research warrants an identification of the causal mechanisms related to answering a survey on a smartphone (or not). This research will include disentangling the elements that constitute the selectivity of smartphone usage: ownership of a smartphone (i.e., who could participate via a smartphone) and the device choice (i.e., who favors participation via a smartphone above other devices). However, investigating this question will require research designs that facilitate a causal interpretation (i.e., experimental designs or panel data). We believe that addressing this research gap is an important next step and a necessary requirement to counteract the selectivity of survey respondents’ device choices.

Footnotes

Appendix

Data Availability

All 18 web surveys used in our study are archived in the German Data Archive for the Social Sciences at the GESIS Leibniz Institute for the Social Sciences (). Study numbers are ZA5350, ZA5351, and ZA5719–ZA5734.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Antoun

Couper

M. P.

Conrad

F. G.

(2017). Effects of mobile versus PC web on survey response quality: A crossover experiment in a probability web panel. Public Opinion Quarterly, 81, 280–306.

Best

Wolf

(2015). The SAGE handbook of regression analysis and causal inference. London, England: Sage.

Bosnjak

Poggio

Becker

K. R.

Funke

Wachenfeld

Fischer

(2013). Online survey participation via mobile devices. Conference Paper, The American Association for Public Opinion Research (AAPOR) 68th Annual Conference, 2013, Boston.

Callegaro

DiSogra

(2008). Computing response metrics for online panels. Public Opinion Quarterly, 72, 1008–1032. doi:10.1093/poq/nfn065

Couper

M. P.

Peterson

G. J.

(2016). Why do web surveys take longer on smartphones? Social Science Computer Review, 35, 357–377. doi:10.1177/0894439316629932

de Bruijne

Wijnant

(2013). Comparing survey results obtained via mobile devices and computers. Social Science Computer Review, 31, 482–504. doi:10.1177/0894439313483976

de Bruijne

Wijnant

(2014). Improving response rates and questionnaire design for mobile web surveys. Public Opinion Quarterly, 78, 951–962. doi:10.1093/poq/nfu046

Eurostat. (2017). Digital Economy and Society: ICT usage in households and by individuals. Retrieved from http://ec.europa.eu/eurostat/web/digital-economy-and-society/data/database

Fuchs

Busse

(2009). The coverage bias of mobile web surveys across European countries. International Journal of Internet Science, 4, 21–33.

10.

Geroski

P. A.

(2000). Models of technology diffusion. Research Policy, 29, 603–625.

11.

Gummer

Roßmann

(2015). Explaining interview duration in web surveys: A multilevel approach. Social Science Computer Review, 33, 217–234. doi:10.1177/0894439314533479

12.

Keusch

Yan

(2017). Web versus mobile web: An experimental study of device effects and self-selection effects. Social Science Computer Review, 35, 751–769. doi:10.1177/0894439316675566

13.

Lee

(2014). Early diffusion of smartphones in OECD and BRICS countries: An examination of the effects of platform competition and indirect network effects. Telematics and Informatics, 31, 345–355.

14.

Lee

Chan-Olmsted

(2017). An empirical analysis of tablet PC diffusion. Telematics and Informatics, 34, 518–527.

15.

Lugtig

Toepoel

(2016). The use of PCs, smartphones, and tablets in a probability-based panel survey. Social Science Computer Review, 34, 78–94. doi:10.1177/0894439315574248

16.

Lugtig

Toepoel

Amin

(2016). Mobile-only web survey respondents. Survey Practice, 9.

17.

Mavletova

(2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31, 725–743. doi:10.1177/0894439313485201

18.

Mavletova

Couper

M. P.

(2016). Grouping of items in mobile web questionnaires. Field Methods, 28, 170–193. doi:10.1177/1525822x15595151

19.

McMaster

H. S.

LeardMann

C. A.

Speigle

Dillman

D. A

. (2017). An experimental comparison of web-push vs. paper-only survey procedures for conducting an in-depth health survey of military spouses. BMC Medical Research Methodology, 17. doi:10.1186/s12874-017-0337-1

20.

Millar

M. M.

Dillman

D. A.

(2011). Improving response to web and mixed-mode surveys. Public Opinion Quarterly, 75, 249–269. doi:10.1093/poq/nfr003

21.

Mohorko

de Leeuw

Hox

(2013). Internet coverage and coverage bias in Europe: Developments across countries and over time. Journal of Official Statistics, 29, 609–622. doi:10.2478/jos-2013-0042

22.

Poggio

Bosnjak

Weyandt

(2015). Survey participation via mobile devices in a probability-based online-panel: Prevalence, determinants, and implications for nonresponse. Survey Practice, 8.

23.

Poushter

(2016). Smartphone ownership and Internet usage continues to climb in emerging economies. Washington, DC: Pew Research Center.

24.

Rattinger

Roßteutscher

Schmitt-Beck

Weßels

Wolf

. (2012–2016). Long-term online tracking T17-T34 (ZA5350, ZA5351, ZA5719–ZA5734). Cologne, Germany: GESIS Data Archive.

25.

Revilla

Toninelli

Ochoa

(2016). PCs versus smartphones in answering web surveys: Does the device make a difference? Survey Practice, 9(4).

26.

Revilla

Toninelli

Ochoa

Loewe

(2016). Do online access panels need to adapt surveys for mobile devices? Internet Research, 26, 1209–1227. doi:10.1108/IntR-02-2015-0032

27.

Rogers

E. M.

(2003). Diffusion of innovation. New York, NY: Free Press.

28.

Roßmann

Gummer

(2014). PARSEUAS: Stata module to extract detailed information from user agent strings (Version: 1.0) [Computer Software]. Chestnut Hill, MA: Boston College.

29.

Roßmann

Gummer

Kaczmirek

(in press). Working with user agent strings in Stata: The PARSEUAS command. Journal of Statistical Software.

30.

Schlosser

Mays

(2017). Mobile and dirty: Does using mobile devices affect the data quality and the response process of online surveys? Social Science Computer Review. Advanced online Publication. doi:10.1177/0894439317698437

31.

Sommer

Diedenhofen

Musch

(2017). Not to be considered harmful: Mobile-device users do not spoil data quality in web surveys. Social Science Computer Review, 35, 378–387. doi:10.1177/0894439316633452

32.

Sterrett

Malato

Benz

Tompson

English

(2017). Assessing changes in coverage bias of web surveys in the United States. Public Opinion Quarterly, 81, 338–356. doi:10.1093/poq/nfx002

33.

Toepoel

Lugtig

(2014). What happens if you offer a mobile option to your web panel? Evidence from a probability-based panel of Internet users. Social Science Computer Review, 32, 544–560. doi:10.1177/0894439313510482

34.

Tourangeau

Sun

Yan

Maitland

Rivero

Williams

(2017). Web surveys by smartphones and tablets: Effects on data quality. Social Science Computer Review. Advanced online Publication. doi:10.1177/0894439317719438