Abstract
This article compares the factors affecting completion times (CTs) to web survey questions when they are answered using two different devices: personal computers (PCs) and smartphones. Several studies have reported longer CTs when respondents use smartphones than PCs. This is a concern to survey researchers because longer CTs may increase respondent burden and the risk of breakoff. However, few studies have analyzed the specific reasons for the time difference. In this analysis, we analyzed timing data from 836 respondents who completed the same web survey twice, once using a smartphone and once using PC, as part of a randomized crossover experiment in the Longitudinal Internet Studies for the Social Sciences panel. The survey contained a mix of questions (single choice, numeric entry, and text entry) that were displayed on separate pages. We included both page-level and respondent-level factors that may have contributed to the time difference between devices in cross-classified multilevel models. We found that respondents took about 1.4 times longer when using smartphones than PCs. This difference was larger when a page had more than one question or required text entry. The difference was also larger among respondents who had relatively low levels of familiarity and experience using smartphones. Respondent multitasking was associated with slower CTs, regardless of the device used. Practical implications and avenues for future research are discussed.
The substantial increase in smartphone use over the past decade has corresponded with an increase in smartphone use among online survey respondents (see, e.g., Gummer, Quoß, & Roßmann, 2018). Survey researchers studying the implications of this shift away from personal computers (PCs) have mostly focused on response quality (i.e., whether respondents can provide high-quality answers when using mobile devices) and to a lesser extent on response rates by device (for reviews, see Couper, Antoun, & Mavletova, 2017; Link et al., 2014). Other issues are only now starting to receive attention, including why respondents tend to take longer to complete surveys when using smartphones than PCs. Longer completion times (CTs) are a concern because they are associated with increased respondent burden and increased breakoffs (Mavletova & Couper, 2015). In fact, many web surveys are already considered quite long when respondents use PCs. As an example, consider the online version of the American Community Survey that takes 40 min to complete (on average). It is a concern if this survey takes even longer for the large numbers of respondents (approximately 8% of online respondents) who now use smartphones to access it (Horwitz, 2016). Research is needed to inform efforts to reduce the smartphone-PC time difference. This article intends to address this issue by investigating the factors that affect how long it takes respondents to answer web survey questions when using smartphones and when using PCs.
The article begins with a review of the literature. It follows with a description of our data and analysis plan and concludes with our results and their implications.
Previous Literature
A consistent finding is that web surveys take longer to complete on mobile devices, particularly smartphones, than on PCs. Most evidence comes from secondary analyses of timing data from web surveys where respondents choose to use different devices including PCs, tablets, and smartphones. For example, in a secondary data analysis of 21 web surveys taken on different devices, Gummer and Roßmann (2015) report that respondents using smartphones took significantly longer than those using PCs, even after controlling for demographic differences between the groups. Couper and Peterson (2017) present ratios of web survey CTs on mobile devices and PCs across 26 studies that included a mix of experiments with random assignment to device and nonexperiments. For 24 of them, the ratios were positive (with a median ratio of 1.4), indicating longer CTs among respondents using mobile devices. The differences seem to be larger in surveys that are not optimized for mobile devices but are still apparent in mobile-optimized surveys. Thus, optimization seems to help reduce the time difference but does not eliminate it. In one of the few studies that looked at question-level CTs by device in a web survey, Andreadis (2015) report that respondents using smartphones spent about 1.5 s longer per question (on average) than those using PCs. By contrast, recent investigations suggest that respondents using tablets do not necessarily take more time to complete web surveys than those using PCs (see, e.g., Gummer & Roßmann, 2015). Given these findings, we focus our attention on the smartphone-PC time difference.
Why do web surveys take longer on smartphones even when the surveys are optimized for such devices? The current state of knowledge about the topic is still evolving. Gummer and Roßmann (2015) outline three potential explanations. One is that mobile respondents take more time because of the extra scrolling required to view and answer questions on a small device. Another explanation is that survey pages load more slowly on smartphones because they rely on a cellular Internet connection (e.g., 3G, 4G) that is slower and sometimes less reliable than a typical Wi-Fi connection. A third explanation is that respondents respond to questions more slowly because of increased multitasking or distractions when using smartphones.
Unfortunately, Gummer and Roßmann were not able to test their proposed mechanisms. Couper and Peterson (2017) expand upon these explanations and describe two others. The first is that respondent may take longer to record their answers using touch input. In particular, typing answers to open-ended questions may take longer on a smaller virtual keypad than a larger physical one. The other explanation is that some of the respondents using smartphones have low levels of comfort and familiarity with their devices. These novice users might take more time than experienced users to navigate through the questionnaire.
Couper and Peterson (2017) tested some of these mechanisms in a secondary data analysis of page-level time data from three web surveys of college students. They found that survey pages do indeed load more slowly on mobile devices. They estimated the time to load a new survey page after clicking the “Next” button by taking the difference between the time spent on a page according to server-side time stamp (for total time) and client-side time stamp (for time when a page is loaded in a respondent’s browser). They report that average transmission times across the three surveys were between 0.31 and 0.57 s longer per page on smartphones than PCs. This is consistent with Mavletova and Couper (2016) who also investigated transmission times in a survey in Russia, finding that it took about 2 s longer to load each survey page (on average) on mobile devices than PCs. Across the two studies, this extra loading time accounted for about between 13% and 28% of the difference in timings between devices. It seems that other things being equal, the faster the cellular Internet connection, the smaller the time difference will be.
Couper and Peterson (2017) also found support for the idea that screen size affects CTs because additional scrolling is often required on smaller devices to complete pages with long questions (e.g., grids) or more than one question. They report that smartphone users scrolled on a higher percentage of screens (49%) than PC users (4%) and were especially likely to scroll on grid questions. Mobile users took about 1.5 times longer than PC users for screens that required scrolling, which in their analysis explained the majority of the difference in timings.
Another possible explanation for the time difference by device is that mobile respondents take longer to read words because they are displayed on the screen in small fonts. However, no studies have found evidence for this. In a secondary data analysis of page-level timings from a web survey conducted in Greece, Andreadis (2015) tested for interactions between device and question length and found no significant results (reported at the .01 significance level). Likewise, Couper and Peterson (2017) report that the difference in timings was comparable for questions with few words and questions with many words.
Another possible explanation for the difference in timings by device, that as far as we know has not yet been tested, is that some mobile respondents take more time because of physical limitations that make it difficult to use smartphones. For example, those with poor eyesight may have greater difficultly reading questions on a small screen. Those who have problems using their hands (i.e., low dexterity) may find it harder to select a response and type on smartphones.
It should be noted that all of these prior investigations made an implicit assumption that group differences between mobile users and PC users in CTs reflected device differences rather than unaccounted for differences in the composition of the groups. By contrast, the analysis presented here makes within-respondent comparisons using data from an experiment where participants were invited to complete the same survey on both a smartphone and a PC. Further, the study was conducted in a probability-based web panel, and the panel members who did not own smartphones were provided with devices. Thus, the resulting study sample is quite diverse in terms of respondents’ demographics and their levels of smartphone experience.
Hypotheses
Here, we test six of the hypotheses described above: the smartphone-PC time difference is explained by (Hypothesis 1) slower completion of longer pages (i.e., more than one question) on smartphones, (Hypothesis 2) slower reading speeds on smartphones, (Hypothesis 3) slower typing speeds on smartphones, (Hypothesis 4) increased completion of other tasks (multitasking) on smartphones, (Hypothesis 5) slower completion on smartphones by those with low familiarity with their devices, and (Hypothesis 6) slower completion on smartphones by those with physical limitations.
In sum, initial work shows that web surveys take longer on smartphones than PCs. Several explanations for this have been offered. It is important to continue to test these explanations as well as to test new ones using a carefully designed experiment and diverse study sample.
Method
Data
The data are from an experiment conducted in the Longitudinal Internet Studies for the Social Sciences (LISS) panel, a probability-based online panel consisting of about 7,000 individuals from the Netherlands who were originally recruited from the national population register. The panel members are invited to complete surveys for 15–30 min each month in exchange for payment (see Scherpenzeel, 2011).
The experiment used a crossover design: Panelists were invited to complete the same survey twice, once on a smartphone and once on a PC, with the order randomized and with a monthlong break in between. It was conducted in 2013—from October 7 to 29 for Wave 1 and from December 2 to 31 for Wave 2.
Panelists who did not have their own smartphone were provided one for the wave in which they were invited to participate in the mobile web survey. While providing devices is not common in practice, it is sometimes used as a way to include mobile users in smartphone surveys conducted in online panels (e.g., Fernee & Sonck, 2013).
The mobile version of the questionnaire was adapted for small screens by eliminating the sponsor’s logo from each page and by using larger fonts and wide, rectangular buttons for response options.
The questionnaire contained 46 attitude and behavioral questions on topics ranging from health to politics. They were displayed on 32 survey pages (consisting of one or more questions). Two pages near the end of the questionnaire contained special formats (a slider question and spinner question) and were excluded because of evidence that respondents had difficulty recording their answers using these formats (reported in Antoun, Couper, & Conrad, 2017). Eight pages at the end of the questionnaire contained nontraditional questions about respondents’ experience taking the survey (e.g., whether they multitasked during the survey) and were also excluded. Thus, our analysis focuses on 22 pages. Twenty pages contained only one question, one page contained five questions, and one page contained six questions. The two pages that had more than one question displayed only one type of question (numeric entry attitudinal questions).
The pages were generally short (median: 24 words per page). Respondents entered their responses by either selecting one option from a closed radio-button response scale (9 pages), typing a number into a numeric input field (11 pages), typing letters into a text box for a narrative-style open question (1 page), or selecting one response from a closed scale with the option to instead type an “Other” response (i.e., half-open other; 1 page).
See Figures 1 and 2 for examples of the first two types of questions in mobile and PC web. We classified the questions as attitude (10 pages), behavioral recall (4 pages), behavioral concurrent, that is, about the current time period (5 pages), and cognitive ability (3 pages). The latter questions were based on the Cognitive Reflection Test (Frederick, 2005). They had correct and incorrect answers and were designed to assess a respondent engagement with the survey.
Single-choice question in mobile (left) and PC (personal computer) web (right).
Numeric-entry question in mobile (left) and PC web (right).
Panelists were first asked in a screener questionnaire whether they were willing to participate in the experiment. Of the 5,486 panelists who responded to the screener, 2,263 indicated that they were willing to participate in the experiment. A sample of willing panelists was drawn. Of the 1,390 panelists who were selected and invited to participate, 895 completed both surveys using the assigned device for a study completion rate of 64% (Callagaro & DiSogra, 2008). User agent strings were used to classify device type (Callegaro, 2010). When more than one device was used for a single survey, we used information from the last one under the assumption that respondents switched devices early in the questionnaire. Those respondents who completed the survey using an unassigned device (approximately 58 in mobile web and 36 in PC web) were excluded from the analysis. Cases with missing values for at least one of the variables used in our multivariate analysis were also removed, resulting in final sample size of 836. The final sample was quite diverse in terms of age: 22% (16–30 years), 27% (31–45 years), 30% (46–60 years), and 21% (61–75 years); education: 60% (no college) and 40% (college); and gender: 51% (male) and 49% (female). The dependent variable for our analysis is page-level timings captured by the survey software (i.e., server-side paradata). This reflects the time from transmission of a survey page by the server to the receipt of data by the server. We did not have timings for when a page loaded on a respondent’s browser and when it was submitted (i.e., client-side paradata).
The CTs included some extreme values on the lower and higher end of the distribution. In order to ensure an approximately normal distribution, we excluded negative CTs (50 in total) as well as the longest 1% CTs and we log-transformed the dependent variable. The negative values may have been due to instances when respondents backed up to earlier questions, then when navigating forward, the load-page time stamp updated but the submit-page time stamp did not. A sensitivity analysis that excluded the shortest 1% of CTs did not lead to changes in our conclusions. The excluded response times at the higher end of the distribution correspond to values of 295 s or longer (i.e., 4 min and 55 s); we assume that such extreme timings do not reflect time spent actively completing the survey. The complete case data set used in the multilevel models has 36,403 rows.
Several characteristics were used as predictors (Table 1). Four page-level characteristics were included: (1) number of words, (2) whether there was more than one item on the page, (3) type of question, and (4) type of input field. Five respondent-level characteristics were included, all of which were self-reported. One was an indicator of multitasking, specifically whether respondents reported completing other activities during the survey. This was measured at the end of the survey. Two indicators about respondents’ familiarity with smartphones were included. The first is a three-level indicator of experience using smartphones. Those who received a smartphone for the experiment were classified as “low”; those who owned a smartphone but reported not using it to complete surveys were classified as “medium”; and those who owned a smartphone and reported sometimes using it to complete surveys were classified as “high.” The second indicator is a five-level indicator of how often participants reported using smartphones to access the Internet (1 = never; 2 = rarely; 3 = some days; 4 = most days; 5 = every day). Both indicators were measured 1 month before the experiment in a baseline survey. Finally, two indicators of physical limitations were included: (1) visual acuity and (2) whether someone had problems working with their hands. These indicators were measured in an unrelated LISS panel survey (“Health Wave 7”) conducted in between the two waves of the experiment.
Description and Distribution of Page-Level and Respondent-Level Predictors.
a Nonresponse and measurement error in mobile web surveys—baseline (September 2013).
bLongitudinal Internet Studies for the Social Sciences Core Study—Health Wave 7 (November to December 2013).
Analytic Approach
We estimated cross-classified multilevel models of pages nested within respondents (see, e.g., Snijders & Bosker, 2011; Yan & Tourangeau, 2008). We estimated random effects at the page and respondent level. The advantage of this approach, in addition to estimating regression coefficients and standard errors that account for the hierarchal structure of our data, is that we could investigate the proportion of variation that is due to page characteristics and respondent characteristics for CTs. Models were estimated using R 3.5 with the package lme4.
In our crossover experiment with two experimental waves, the CTs come from when respondents used PCs to complete the survey and when the same respondents used smartphones to complete the survey. We included CTs from both waves in the same model. The device that respondents used was included as a fixed effect to measure within-respondent change in CTs by device. We also included a control variable for the experimental wave in which the survey was completed (first vs. second). In order to answer our research questions, we estimated interactions between device and the predictors in the model.
Results
Descriptive Results
Respondents took an average of 34.8 s per page when using smartphones and an average of 24.8 s per page when using PCs. Thus, respondents took an average of 10 s (or 1.4 times) longer to complete each survey page when using smartphones than PCs.
Multivariate Models
Table 2 shows the results of three multilevel models predicting page-level CTs. Our modeling approach was to start with an empty model (Model 1) that allowed us to investigate the page- and respondent-level variance components. We then added a variable for the device that respondents used as well as our other predictor variables (Model 2). Finally, we tested our hypotheses by adding interactions between device and the predictor variables (Model 3). To assist interpretation of the regression coefficients, we calculated exponentiated coefficients (ECs) that can be interpreted as the expected change in the average CT per page (in seconds) for a unit change in a predictor variable after controlling for other variables. The models are based on 36,403 timings, 22 pages, and 836 respondents.
Results From Multilevel Models Explaining (Log-Transformed) Page-Level Completion Times in a Web Survey.
Starting with the variance decomposition, the null model showed that 45% of the variance in CTs was at the page level and that 15% was at the respondent level. Thus, we concluded that page-level characteristics contributed more to variation in CTs than respondent-level characteristics.
The variance components in the final model suggested that we were able to explain 46% of the total variation in CTs with our variables. At the page level and respondent level, we explained 80.3% and 45.5%, respectively. From this we concluded that our variables were more successful in accounting for variation at the page level than respondent level.
Next, we added our predictor variables (Model 2) in order to test main effects. Consistent with our descriptive results, the model showed that smartphone use is associated with an increase in response times (EC: +8 s). Not surprisingly, pages with more than one question took longer to answer (EC: +30 s) than other pages. Cognitive ability questions, which were intended to be challenging, were more time-consuming (EC: +27 s) than other types of questions. Questions requiring a numeric entry took longer (EC: +19 s) than single-choice questions.
We found a significant wave effect, indicating that respondents were faster (EC: −2 s) when answering questions for the second time. The effects of age seem to be approximately linear, with older respondents taking more time. Those with a college degree were slightly faster (EC: −2 s) than those without a college degree. Frequent smartphone users tended to be faster at answering the questions (EC: −9 s) than others. Finally, we found that respondents who reported multitasked during the survey took longer to complete each page (EC: +2 s) than those who did not multitask.
Next, we added interactions between device and the predictor variables for which we had device-specific hypotheses (Model 3). This model indicates whether our predictors had a differential effect on CTs based on the device that respondents used. To facilitate interpretation of the interactions, we computed the ratio of CTs for smartphones versus PCs based on the ECs for each level of a predictor variable.
We found an interaction between the device that respondents used and the number of items on the page that supported our first hypothesis (Hypothesis 1). The pattern is that the ratio of mobile to PC time is larger for pages with more than one question than single-item pages (1.54 vs. 1.47). This is perhaps due to increased scrolling on smartphones for multi-item pages.
The model also revealed an interaction between device and the number of words on the page, and the pattern was a reversal of our second hypothesis (Hypothesis 2). The time ratio decreased slightly as the number of words on the page increased. This result indicates that respondents took less time to read words when displayed on smartphones than PCs.
We expected slower typing speed on smartphones (Hypothesis 3) and this was partially supported by the model results. In line with our expectation, the time ratio increased when respondents answered a text-entry question rather than single-choice questions (1.8 vs. 1.6). However, the time ratio decreased when respondents answered numeric entry items compared to single-choice questions (1.4 vs. 1.6). Thus, respondents using smartphones tended to slow down when entering several characters of text but not when entering a single number.
The model also revealed interactions involving the device that respondents used and their familiarity with smartphones. All of the interactions were in the expected direction, that is, the more experience respondents had using smartphones, the smaller the time difference was (Hypothesis 5). Specifically, the time ratio was 1.8 for those who used devices provided to them for the study (“low familiarity”), 1.3 for those who reported using smartphones for some online tasks but never for surveys (“medium familiarity”), and 1.2 for those who reported previously using their smartphones to complete surveys (“high familiarity”). Frequency of smartphone use affected the time ratios in a similar way that was nearly linear as shown in Table 3. At the extremes, those who reported never using smartphones took 1.8 s longer when using such devices, whereas those who reported using smartphones every day took 1.3 s longer.
Mean Page-Level Completion Times (in seconds) by Frequency of Smartphone Use.
We did not find an interaction between device and multitasking (Hypothesis 4), suggesting that multitasking had approximately the same slowing effect on either device. Nor did we find an interaction between device and either of our indicators of physical limitations (Hypothesis 6).
Discussion
Web surveys tend to take longer to complete on smartphones than PCs, and long CTs are a concern because they are associated with increased respondent burden and increased breakoffs. In our study, we found that respondents took 1.4 times longer when using smartphones. The size of this difference is comparable to several other studies (e.g., Mavletova & Couper, 2015; McGeeney & Marlar, 2013) and happens to be the same as the median difference from Couper and Peterson’s (2017) summary of 26 studies.
Our analysis highlighted some of the factors that differentially affect how long it takes respondents to answer web survey questions when using smartphones and PCs. We find that the ratio of mobile to PC time is larger for multi-item pages than single-item pages. One explanation supported by Couper and Peterson’s (2017) findings is that the extra step of scrolling on smartphones adds time, though we could not verify this because we did not have passively measured indicators of whether respondents actually scrolled.
We also found that the smartphone-PC time gap was strongly related to respondents’ levels of familiarity and experience using smartphones. The gap was smallest for frequent smartphone users and those who had previously used their phones to complete surveys, and the gap was largest for infrequent smartphone users and those who were provided phones for the experiment. Those in the latter group were using unfamiliar devices and probably had difficulty navigating through the questionnaire.
Our analysis represents the first attempt (to our knowledge) to examine variation in CTs due to multitasking. It was clear that this behavior slowed respondents down. Even though the effect of multitasking did not vary by device, it may have contributed to the time difference because more respondents engaged in multitasking when using smartphones than PCs. However, the fact that only 5% more respondents multitasked when using smartphone means that this was probably a small contributor to the overall time difference. Given that multitasking added about 2 s per page, in our calculations, the diffused effect on the overall time difference between devices was only about 0.10 s per page (2 s × .05).
We found no evidence that respondents took longer to read words on the screen when displayed on smartphones, which is consistent with other research (Andreadis, 2015; Couper & Peterson, 2017). In fact, we found the opposite pattern which we attribute to the relatively large font sizes used in the mobile-optimized survey. Nor did not find evidence that respondents’ level of vision or level of dexterity account for the time difference between devices. Because the panel respondents in this study had already participated in prior surveys and indicated their willingness to participate in this research, it is possible that their physical limitations were not so severe as to have noticeable effects. It is also possible that the text and buttons in the optimized mobile survey were sufficiently large for those with low levels of visual acuity or dexterity to read and select without slowing down.
There are limitations with this analysis. Our analysis considered a limited number of question formats (i.e., single-choice questions and text and numerical entry questions) and was based on a limited number of pages. Several respondent-level measures used in our analysis were self-reported and thus subject to measurement error. Further, multitasking was reported at the end of the survey even though this activity is likely intermittent with respondent attention fluctuating throughout the survey. Nonetheless, we think our study raises important practical questions.
Will the time difference between devices eventually disappear? Our findings suggest that it will diminish over time as respondents become more comfortable using smartphones, especially in online surveys where respondents can choose which device they use. However, there are at least two reasons why this prediction may not hold. There are signs that smartphone ownership and use is starting to level off (Gummer et al., 2018; Pew Research Center, 2018). Secondly, some respondents may choose to use a smartphone for reasons unrelated to their level of comfort and familiarity, for example, it happens to be nearby, they are on the go, or it is their only device. Given this, the time difference between devices may not disappear any time soon.
Can improvements to mobile questionnaires eliminate the time difference? Our analysis suggests that relatively straightforward questionnaire design changes can help. One change is to use closed response scales and numeric entry questions rather than text-entry questions and presumably other more complicated formats (slider, drop boxes). Another potential change is to use single-item pages, though we need to be cautious about recommending this, given that we did not have a measure of transmission time required to load each new page and given that other studies have found advantages to using scrolling designs (de Bruijne & Wijnant, 2014; Mavletova & Couper, 2014). However, even if such changes are effective, they are likely to have modest effects that do not erase the time difference; whether the difference can be eliminated through the development of innovative mobile-friendly design features is a topic for future research.
Our findings also open other topics for future work. For example, there would be value if research uses passively measured indicators of respondents’ activities (see Sendelbah, Vehovar, Slavec, & Petrovčič, 2016). This would allow for a more precise investigation of the nature of multitasking on each type of device and its impact on CTs. There would also be value investigating the time difference between devices for other question types that require different touch gestures or more scrolling (e.g., grids).
In conclusion, we encourage further research on the implications of the shift away from PCs to smartphones for online data collection. This can inform new strategies to improve the design and implementation of mobile surveys. Simply ignoring the issues that arise—such as respondents taking longer to complete surveys when using smartphones—is not tenable as it could lead to increased breakoffs and increased error in survey estimates.
Footnotes
Authors’ Note
The data for this analysis come from an experiment carried out by the Longitudinal Internet Studies for the Social Sciences (LISS), which is administered by CentERdata (Tilburg University, the Netherlands). We are grateful to them for conducting this experiment and for providing the timing data. We also thank two reviewers for their helpful suggestions.
Data Availability
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Software Information
Analyses were conducted in R Version 3.5.1 using the following packages: ggthemes, haven, knitr, lme4, texre, and tidyverse. The code is available from the corresponding author at