Abstract
Mobile coverage recently has reached an all-time high, and in most countries, high-speed Internet connections are widely available. Due to technological development, smartphones and tablets have become increasingly popular. Accordingly, we have observed an increasing use of mobile devices to complete web surveys and, hence, survey methodologists have shifted their attention to the challenges that stem from this development. The present study investigated whether the growing use of smartphones has decreased how systematically this choice of device varies between groups of respondents (i.e., how selective smartphone usage for completing web surveys is). We collected a data set of 18,520 respondents from 18 web surveys that were fielded in Germany between 2012 and 2016. Based on these data, we show that while the use of smartphones to complete web surveys has considerably increased over time, selectivity with respect to using this device has remained stable.
In recent years, the use of smartphones to complete web surveys has attracted the widespread attention of survey methodologists. This popularity is motivated partially by a population-wide trend toward increasing smartphones coverage, and the observation that the use of smartphones may affect response behavior, and thus data quality. For example, in 2011, only 19% of Europeans used a smartphone to access the Internet, but by 2016, this number had risen to 56% (Eurostat, 2017). One consequence of the availability and usage of mobile Internet options is an increase in surveys that are filled out on smartphones (Gummer & Roßmann, 2015; Lugtig, Toepoel, & Amin, 2016; Revilla, Toninelli, Ochoa, & Loewe, 2016).
Web survey research has focused on the impacts of mobile device usage on survey errors and data quality and has found some significant effects. For example, some research investigated the effects on completion rates, finding that mobile respondents had significantly lower completion rates (Mavletova, 2013; Sommer, Diedenhofen, & Musch, 2017), including break-off rates for smartphones that were 2–3 times higher than for personal computers (PCs; Bosnjak et al., 2013; Poggio, Bosnjak, & Weyandt, 2015). A number of studies also studied the effects of mobile devices on response times, finding that participants on mobile devices needed longer to complete a survey (Antoun, Couper, & Conrad, 2017; Gummer & Roßmann, 2015; Revilla, Toninelli, & Ochoa, 2016; Sommer et al., 2017). Schlosser and Mays (2017) also found longer completion times for mobiles, although controlling for the type of Internet connection and hardware minimized those differences. In addition, Couper and Peterson (2016) found that only one fifth of the time difference could be accounted for by transmission time, whereas other characteristics were responsible for the remaining four fifths. Keusch and Yan (2017) found significantly longer response times both for iPhone users who voluntarily used their device to do a survey and the respondents who were told to switch to their mobile device after starting a survey on their PC. Regarding the length and quality of open answers, Bosnjak et al. (2013) found that mobile respondents answered more open questions than PC users when dropping out of a survey, whereas Mavletova (2013) and Revilla, Toninelli, and Ochoa (2016) found the open answers of mobile respondents to be significantly shorter. In addition, Antoun, Couper, and Conrad (2017) found that respondents gave longer answers to open questions on their smartphone, hypothesizing that these respondents may have become more comfortable with writing on their smartphones or using technological support such as voice-to-text typing. Last, concerning missingness on the item level, Tourangeau et al. (2017), Schlosser and Mays (2017), and Toepoel and Lugtig (2014) reported no differences in item nonresponse rates between mobile and PC respondents. Based on these findings, some researchers have stressed the importance of survey design adaptions to minimize the effects of different devices (de Bruijne & Wijnant, 2014; Mavletova & Couper, 2016).
A second line of research has examined the coverage and usage patterns of smartphones to complete web surveys. For example, de Bruijne and Wijnant (2013) found that smartphones were used by more highly educated, younger members of the LISS panel and that men were more likely to use a smartphone than women. At the same time, they also found that out of all smartphone and tablet owners, only a tiny fraction of 0.9% completed surveys on their smartphone, 13.5% on their tablet, and 0.8% on both of those devices (de Bruijne & Wijnant, 2013, p. 486). Along these same lines, Toepoel and Lugtig (2014) found that “age, income, household composition, and household size are predictors of mobile completion” but that “gender, education, and urbanization did not affect mobile completion” (pp. 548–549) and also pointed to younger respondents with higher income and living in smaller households as the first wave of smartphone adopters for completing surveys. In addition, Revilla, Toninelli, Ochoa, and Loewe (2016) reported ambiguous findings with respect to the use of mobile devices in web surveys in seven countries; they did not find any explanatory power for education, and moreover, they found inconsistent effects for age and gender across the seven countries.
From these two lines of research, we have learned that survey errors and data quality may be related to the use of smartphones to complete web surveys. The choice of using this particular device seems to be selective with respect to which groups of respondents are more prone to complete a survey using such a mobile device. In other words, the choice of a mobile device for completing surveys varies systematically between groups of respondents. On the one hand, this could lead to survey errors that are heterogeneously distributed among different groups of respondents; yet on the other hand, it seems entirely reasonable to assume that after an initial period of selective use by innovators and early adopters (Geroski, 2000; Rogers, 2003), the new technology (i.e., innovation) spreads to the broader public, and thus the selectivity in mobile device usage decreases over time. Following this line of reasoning, Lee and Lee (2014) and Lee, Lee, and Chan-Olmsted (2017) have built on previous studies that propose an S-shaped curve for mobile technology diffusion and have used Gompertz models to depict the early diffusion of smartphones and tablets in different countries. These authors have reported that after the initial phase during which early adopters use smartphones, network penetration increases and smartphone prices decrease, which leads to a higher diffusion of smartphones in the general public.
As argued previously, some research has reported rising rates of mobile device usage for completing web surveys. However, research is surprisingly lacking with respect to how the selectivity of using smartphones has changed over time. Participation via smartphones (i.e., selectivity) has at least two elements: ownership of a smartphone (i.e., coverage) and the actual device choice. Previous research in this area has focused primarily on different coverage rates. For example, Fuchs and Busse (2009) analyzed smartphone penetration rates, and thus coverage bias in web surveys within Europe between 2005 and 2007. Although this research provides important insights regarding coverage bias, the availability of smartphones to respondents does not necessarily imply that they actually will use a smartphone to complete web surveys. Therefore, Fuchs and Busse (2009) rightly caution “that the mere coverage of the population with a new technology does not imply that all respondents who have access to the mobile Internet are actually capable and willing to use it in order to take a survey” (p. 30). Similarly, Mohorko, de Leeuw, and Hox (2013) investigated Internet coverage and coverage bias in Europe, finding that “in general, coverage bias decreases when education, employment, life expectancy, and urbanicity increase” and that the “differences in age, sex, education, and life satisfaction between those with and without Internet access are diminishing” (p. 618). 1 Again, a higher Internet coverage does not lead necessarily to lower selectivity; rather, it provides a necessary basic prerequisite.
In the present study, we aim to contribute to this research lacuna by addressing two research questions. First, we investigate whether growing Internet coverage and the population-wide availability of smartphones have resulted in higher smartphone usage for completing surveys, and second, whether the increasing use of smartphones to complete surveys has diminished the selectivity of smartphone usage over time. In other words, we ask the following important question—Is the self-selectivity of using smartphones to complete web surveys reduced as these mobile devices become more available and commonly used in daily life? To address this research gap, we relied on web surveys that were fielded in Germany. Germany is an illustrative case for our study, since it is an advanced economy that has reached high levels of Internet and smartphone coverage in recent years (Poushter, 2016), which are the basic prerequisites for a potentially diminishing selectivity. Since we were interested in the overall selectivity of using a smartphone to complete surveys, we have not attempted to disentangle the elements of ownership and device choice, and thus we leave this challenge to future studies.
Data and Method
To investigate our research questions, we pooled 18 web surveys that were conducted between May 2012 and December 2016 as part of the German Longitudinal Election Study (Rattinger, Roßteutscher, Schmitt-Beck, Weßels, & Wolf, 2012–2016). The questionnaires featured a common core of similar questions that covered topics related to political attitudes and behaviors. Each of these surveys was sampled based on the same quotas from a large German offline-recruited probability-based access panel. On average, 1,029 respondents participated per survey, which provided us a total of 18,520 cases. Detailed figures on respondents who accepted the survey invitation, the numbers of screen-outs and break-offs, and the break-off rate of each survey as proposed by Callegaro and DiSogra (2008) are provided in Table 1.
Field Period and Outcome Rates of 18 Web Surveys.
To assess the levels of selectivity with respect to the use of smartphones in each of our 18 surveys, we fitted separate logistic regression models with “using a smartphone to complete the survey” as the dependent variable. To assess what device respondents used to complete the survey, we drew on the user agent strings that the survey software automatically collected (for a discussion of these strings, see Roßmann, Gummer, and Kaczmirek, in press). We used the Stata module parseuas (Roßmann & Gummer, 2014) to parse the strings into useable information and to identify whether respondents used a PC, tablet, or smartphone to complete the survey. Based on this information, we created a binary variable on using a smartphone (0 = no, 1 = yes).
With respect to the independent variables, we added a set of sociodemographic and key substantive variables from the surveys. We operationalized sex (0 = male, 1 = female) and region of residence (0 = East, 1 = West) as dummy variables. We categorized age into four groups ranging from “20–29 years,” “30–39 years,” “40–49 years,” to “50 years and older.” We also categorized education, interest in politics, and strength of party identification into “low,” “intermediate,” and “high.” We metrically coded participation in political discussions from not at all to 7 days a week. Table 2 provides descriptive statistics of all the variables in the pooled data set.
Descriptive Statistics on the Pooled Data of 18 Web Surveys.
Note. Statistics are calculated at the respondent level with n = number of respondents. M = mean; SD = standard deviation.
If selectivity in smartphone usage decreases over time (i.e., across the surveys), significant differences between groups should vanish or become less pronounced. In other words, the effects of the covariates in our 18 regression models should diminish, and the predicted propensities for smartphone usage should become similar for different subgroups. 2 To assess these indicators of selectivity, we calculated the average marginal effects for all regressions. Due to the difficulties associated with comparing the coefficients of the logistic models (Best & Wolf, 2015), we further compared the differences in the predicted probabilities of the selected groups and analyzed how (and if) these differences changed across time (i.e., surveys). To indicate significant differences between predicted probabilities, we calculated 95% confidence intervals (CIs) for each probability. Overlapping CIs of the predicted probabilities indicated nonsignificant differences, whereas nonoverlapping CIs indicated significant differences (with α = .05).
Results
Based on the data we used, Figure 1 shows an increasing use of smartphones and tablets to do web surveys, which also suggests that the use of PCs to complete web surveys has declined. This finding is in line with what we know from previous studies on the use of mobile devices for completing surveys (e.g., Lugtig & Toepoel, 2016; Revilla, Toninelli, Ochoa, & Loewe, 2016). Moreover, the increasing use of smartphones coincides with ongoing technological development—the increasing availability of mobile devices to the broad public and increasing Internet coverage facilitating usage. Figure 1 also relies on supplemental data on Internet access in households and via smartphones provided by Eurostat (2017). Supporting our reasoning, we found that Internet access in households remains nearly stable at an already high level, whereas Internet access via smartphones dramatically increased between 2012 and 2016.
Device usage in 18 web surveys between 2012 and 2016.
To shed light on whether the increasing use of mobile devices has reduced selectivity in the usage patterns of these devices, we report the results of our regression models in Table 3. The model fits showed that selectivity occurred in all of our 18 surveys because a moderate share of variation was explained by the covariates (pseudo R 2 between .074 and .175). If selectivity was not present, the explanatory power of the models should have been lower. This finding is supported partially by the specific independent variables. In line with previous research (de Bruijne & Wijnant, 2013; Toepoel & Lugtig, 2014), in our surveys, we found a consistent effect of age on the likelihood of using a smartphone. Younger respondents were more likely to complete a survey on a smartphone—we found this effect in each of the 18 models. Figure 2 details the predicted probabilities for age groups across the surveys. Surprisingly, the difference in probabilities between age groups did not diminish and remained at a stable level. Moreover, Figure 2 also shows a generally increased likelihood across all age groups to use a smartphone to complete web surveys. In other words, as mobile devices became increasingly available and were used by the broader public, a gap remained between age groups with respect to using these devices to complete web surveys.
Use of Smartphones for Completing Web Questionnaires: Results From Logistic Regressions in 18 Surveys.
Note. AME = averaged marginal effect; SE = standard error; Ref = reference category.
*p < .05. **p < .01. ***p < .001.
Predicted probabilities to use a smartphone to complete a survey by age.
Previous studies have reported nonconclusive findings regarding other factors for smartphone use such as sex, education, and region of residence. In our models, the female respondents appeared to be more likely to use smartphones compared to men, but this effect was only significant in 5 of the 18 models. A reasonable pattern would have been finding this effect in the first five surveys and then seeing it diminish afterward, but this was not the case, since the five significant effects appeared to vary nonsystematically over time. To investigate the matter further, we again plotted the predicted probabilities in Figure 3. The visual presentation illustrates our previous findings. Overall, the likelihood of using a smartphone to complete web surveys increased across surveys. However, the differences in propensities between male and female respondents did not change. Given the inconsistent pattern of these effects and that previous research had presented similar findings (e.g., Revilla, Toninelli, Ochoa, & Loewe, 2016), further research is warranted to shed light on the causal mechanisms that lie behind these effects.
Predicted probabilities to use a smartphone to complete a survey by gender.
Our analysis adds to the ambiguous findings from prior studies. We did not find selection effects for different education groups, and the region of residence in Germany did not have a consistent effect. In addition, our analysis did not show any trends across time with respect to how intergroup differences changed (see predicted probability plots in Appendix Figures A1 and A2).
Overall, we found a general pattern that using smartphones to complete web surveys has become more likely. This finding is in line with previous studies that investigated either how the coverage and availability of mobile devices developed or who were the users of smartphones to complete web surveys. We also found that younger respondents were more likely to adopt this new technology for completing surveys compared to older respondents. This pattern did not change across time despite the fact that the overall likelihood increased. We did not find a distinct pattern for other covariates that previous studies had reported to facilitate smartphone use, despite our findings with respect to the mixed effects for respondents’ gender. In summary, our study illustrates that a macrolevel process (the increasing spread of a technological innovation) does not manifest necessarily in a microlevel phenomenon (decreasing selectivity).
Conclusion
The present study set out to gain knowledge about how smartphone usage and patterns of use developed with respect to web surveys over the recent years. This line of research was motivated by the observation that previous studies had reported increasing coverage rates of mobile devices in the general public and web surveys in particular. However, none of these studies focused on whether the increasing use of smartphones had manifested in a decreasing selectivity of use. Methodological studies frequently have cautioned that completing surveys via a smartphone may result in lower data quality if the survey design was not adapted properly (de Bruijne & Wijnant, 2013). If smartphone use to complete web surveys varies systematically (i.e., is selective), survey errors also will vary across different respondent groups, what may be misinterpreted as substantive differences between groups.
First, in line with previous research, we found that the use of smartphones to complete web surveys has risen over recent years. Second, while the general likelihood of using a smartphone to complete a survey has increased over time, the selectivity for doing so has remained stable over a period of 5 years. Despite the growing and persistent popularity of mobile devices in everyday life, the use of smartphones has not become an accepted practice for all our respondents. In other words, the problem remains that specific groups are more prone to using mobile devices compared to others and, in turn, are more likely subject to device-specific influences with respect to their response behavior. The factor that was outstanding and showed a consistent pattern across all 18 surveys was age. While this finding may appear straightforward at first and even seem like selectivity may only exist with respect to the age dimension, it is a crucial indicator of a more serious problem. Age (as well as cohort and birth year) is commonly and frequently used in the social sciences as a control variable because it is assumed to be correlated with a variety of constructs in which a researcher may have an interest. If specific age groups are more likely to answer a survey via a smartphone, and thus are more prone to measurement error, specific groups of respondents with characteristics correlated with age also are more prone to this error. For instance, assume that younger respondents are more likely to use smartphones and, hence, their measures are more likely to be of lower quality. If age is now positively correlated with a variable such as computational skills, the respondents who have these higher skills are also more likely to provide data of lower quality. In other words, due to the correlation with age, measurement error spills over into other variables.
Given the pertinacity of selectivity, our results have implications for the ongoing discussion about pushing respondents to the web mode (McMaster, LeardMann, Speigle, & Dillman, 2017; Millar & Dillman, 2011). Compared to face-to-face interviews, interviewing respondents online has an appeal due to lower costs and other beneficial side effects (e.g., reducing interview effects, more flexibility, and implementation of visual cues and media applications). However, despite the fact that mobile devices are available and frequently are used to complete surveys, pushing respondents to the web may result in specific subgroups using specific devices, which, again, could result in heterogeneous survey errors in the data. To better evaluate the potential outcome of push-to-web approaches, we believe it is necessary to further shed light on which error mobile use might induce and what drives the choice of a device to complete a survey. In this context, it might be interesting to push specific respondent groups to PCs, tablets, or even smartphones to balance any potential survey errors in a sample.
The findings of the present study yield opportunities for future research. First, while we were in the fortunate position to have 18 comparable surveys that enabled us to assess smartphone use over 5 years, it might be interesting to extend our approach to investigate larger time intervals. Future research that tackles this question will be confronted, however, with the challenge to ensure the comparability of different surveys. Second, in the present study, we focused on web surveys in Germany. One reason for our choice was that Germany is an advanced economy that has reached an all-time high in mobile coverage rates and thus fulfills the requirements for testing the assumption of decreasing selectivity. In addition, Internet access in Germany has reached a level comparable to other advanced economies such as the United Kingdom, the United States, and Canada (Poushter, 2016, p. 9), and thus web surveys can be considered an important survey mode for covering larger parts of the population. However, relying on data from Germany alone may impair the generalizability of our findings. If comparable data sets can be obtained for other countries, we would consider a cross-national comparison to be a formidable challenge for future studies. In our view, these studies should focus on country-level differences that influence adaption rates and investigate what influences subgroups of respondents to use mobile devices (i.e., what drives selectivity). Third, the present study covered 5 years, and as our initial analysis showed, in our surveys, this period coincided with the rise in smartphone usage rates necessary to address our research questions. A long-term analysis of trends in mobile device use seems mandatory to monitor whether selectivity remains stable, diminishes, or even starts to rise. Fourth and finally, with the present study, we showed that selectivity remains an important challenge for survey methodologists and practitioners. In our analyses, we mostly were concerned with measuring selectivity and whether it changed between surveys. Further advancing this line of research warrants an identification of the causal mechanisms related to answering a survey on a smartphone (or not). This research will include disentangling the elements that constitute the selectivity of smartphone usage: ownership of a smartphone (i.e., who could participate via a smartphone) and the device choice (i.e., who favors participation via a smartphone above other devices). However, investigating this question will require research designs that facilitate a causal interpretation (i.e., experimental designs or panel data). We believe that addressing this research gap is an important next step and a necessary requirement to counteract the selectivity of survey respondents’ device choices.
Footnotes
Appendix
Data Availability
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.