Mobile and Dirty

Abstract

In this article, we present a study on the data quality and the response process of mobile online surveys using an experimental design as compared to a standard computer. We used the following indicators to measure data quality and response properties: reaction time to survey invitation, break-off rate, item nonresponse, length of responses to open-ended questions and survey transmission, processing, and completion time. With regard to completion time, we also explored the significance of the place as well as the situation in which the survey was completed, the kind of Internet connection the respondents had as well as the hardware properties of the devices used to answer the online survey. Our results suggest comparable data quality and response properties in most aspects: There were no noticeable differences between computer and mobile users as regards break-off rate, item nonresponse, and length of responses to open-ended questions, nor the place where the survey was completed. However, it took respondents in the mobile group longer to complete the survey as compared to respondents answering the online survey on their computer. In terms of the completion time, there was a significant decrease in the differences between mobile devices and PCs when respondents used technically advanced mobile devices and had access to a fast Internet connection.

Keywords

online survey mobile device response time data quality mode effect paradata reaction time to survey invitation break-off rate item nonresponse length of responses to open-ended questions

Despite existing methodological problems, online surveys have become more and more popular among researchers and practitioners over the past two decades. In addition to the standard online surveys designed for computers, those meant for completion on mobile devices such as smartphones and tablet computers have additionally been used to collect data for several years. In light of the growing availability of suitable mobile devices and their inherent potential for data collection, the use of these types of surveys will increase significantly in the future (Mavletova, 2013). Given that PC and mobile surveys differ in several aspects, this may have a strong impact on the quality of data as well as on response properties obtained using different survey modes. For instance, mobile devices have smaller displays, influencing perception and comprehension of a survey (Mavletova, 2013). Navigation takes place via a touch screen and keypad instead of a mouse and keyboard. Furthermore, the processing power of PCs is usually superior to that of mobile devices. These aspects can affect response time and accuracy of data input (Peytchev & Hill, 2010; Stapleton, 2013). Moreover, since mobile devices are made for mobile use and are usually used in many different places and situations, concentration and motivation may differ among respondents (Wells, Bailey, & Link, 2014).

To date, there have only been a few empirical studies carried out on the data quality and the response properties of mobile online surveys as compared to traditional PC online surveys. An analysis of this issue as it pertains to German-speaking countries is especially lacking. Furthermore, it is often unclear whether the observed differences in data quality and the response process are caused by the different survey modes themselves or merely by the different data transmission times of PCs as opposed to mobile devices as a function of their hardware (Gummer & Roßmann, 2015). In this study, we first analyze the data quality and response process of mobile surveys compared to those completed on PCs, while controlling for technical aspects.

It is conceivable that the differences between the two survey modes would disappear to a significant extent if an adaptive web design was used in the mobile survey and if the study controlled for technical issues such as survey page transmission time as well as browser and processor quality. As for data quality in online surveys, there is, as yet, no coherent concept (Dingelstedt, 2015); to determine data quality and response properties for the purposes of this study, we used the completion rate, the reaction time to survey invitation, item nonresponse, the length of responses to open-ended questions, and time measurements of interview duration.

Against this background, this article has two main goals: First, we use a German sample to compare the data quality and response process of self-administered online surveys completed using a PC to that of surveys completed on mobile devices. Second, we use a novel empirical approach to paradata for time measurements and hardware analyses, which allows us to distinguish between survey page sending and receiving times and the time a respondent spent answering the survey as well as differences which might be due to different hardware features.

Literature Review

Recent studies have investigated the differences in the data quality and response properties of mobile online surveys versus PC online surveys (Buskirk & Andrus, 2014; Couper & Peterson, 2016; Cunningham, Neighbors, Bertholet, & Hendershot, 2013; de Bruijne & Wijbant, 2013; Mavletova, 2013; Mavletova & Couper, 2016; Millar & Dillman, 2012; Peytchev & Hill, 2010; Stapleton, 2013; Toepoel & Lugtig, 2014; Wells et al., 2014; Zahariev, Ferneyhough, & Ryan, 2009). The results of these studies demonstrated significant differences between the two types of surveys; one example of this is the completion rate, which was investigated as an important feature of data quality. There is some evidence that mobile online surveys have a higher dropout rate (break-off rate) than PC surveys. In a German access panel study, for instance, Bosnjak et al. (2013) observed a dropout rate for smartphones that was 3 times higher than that of the PC group (15.5% vs. 4%). Mavletova (2013) in Russia and de Bruijne and Wijbant (2013) in the Netherlands also found evidence for lower completion rates in mobile online surveys than in PC surveys (similar: Millar & Dillman, 2012; Stapleton, 2013). Mavletova (2013) offered the explanation that completing an online survey via a mobile device implies greater effort and is more time-consuming compared to a PC (see also Mavletova & Couper, 2015).

A further indicator for data quality is the rate of item nonresponse (nonsubstantial responses, e.g., “don’t know” as well as absence of answers to specific questions). In this regard, researchers assume that mobile online surveys lead to a higher degree of item nonresponse than PC surveys. However, contrary to the expectations, initial studies showed no effect of survey mode (Mavletova, 2013; Toepoel & Lugtig, 2014).

Another point researchers often argue is that the length of responses to open-ended questions is shorter in mobile surveys than in PC surveys. However, the evidence supporting this is mixed. Bosnjak et al. (2013), for example, observed no differences between mobile users and PC users with respect to open-ended questions (similar: Buskirk & Andrus, 2012; Zahariev et al., 2009). In contrast, Mavletova (2013) found that responses to open-ended questions were significantly shorter on mobile phones than in computer-based surveys. Toepoel and Lugtig (2014) also showed a shorter average length of responses to open-ended questions on mobile devices; however, the results did not reach conventional levels of statistical significance.

Another aspect is the difference in average interview length with regard to individual survey pages or the full length of interviews (Couper & Kreuter, 2013; Yan & Tourangeau, 2008). Most studies indicate that mobile users needed more time completing the survey than PC users (Cunningham et al., 2013; de Bruijne & Wijbant, 2013; Gummer & Roßmann, 2015; Mavletova, 2013; Zahariev et al., 2009). However, a subset of studies found shorter completion times as well as higher break-off rates for mobile devices compared to PCs, which might be caused by special mobile optimization methods (e.g., at the use of drop-down questions, images, or slider bars) for mobile devices used in the survey (Johnson, Kelly, & Stevens, 2012; Mavletova & Couper, 2015, see also Couper & Peterson, 2016; Table 1).

Table 1.
Participation Rate for PC Group Versus Mobile Group.

Invited to PC Group Invited to Mobile Group Total

Responded with PC 510 137 647

Responded with mobile devices 16 157 173

Total 526 294 820

▾ Exclusion of drop-outs ▾

Invited to PC Group Invited to Mobile Group Total

Responded with PC 403 80 483

Responded with mobile device 9 116 125

Total 412 196 608

Note. Valid cases in bold.

The place and/or situation in which the survey was completed is a further important difference between both survey modes. In one of the few empirical studies conducted on this topic, Mavletova (2013) included the place of completion of the survey as a control variable. However, no evidence was found suggesting an impact on the place of completion of the survey with regard to the various aspects of data quality observed. Nevertheless, Mavletova also presumed different response effects due to the different technical characteristics of mobile devices; however, her study was not designed to investigate these aspects and therefore did not produce any results in this regard.

In terms of the current state of research, the question as to whether the described differences between mobile and PC surveys might be due to technical problems—such as processor power or transfer time—rather than to real mode differences is, as yet, still open. However, it seems quite reasonable to assume that the differences between both survey modes will disappear to a great extent if adaptive web designs are used in mobile surveys and if studies control for technical issues such as survey page transmission time, browser and processor quality.

Research Questions and Hypotheses

To investigate these issues, we conducted an experimental study with two survey modes (PCs vs. mobile devices). In total, 820 German university students participated in the study. We used the following nine indicators to measure data quality and response properties: (1) reaction time to survey invitation; (2) completion rate/break-off rate; (3) item nonresponse (rate of nonsubstantive responses as well as the rate of unanswered questions); (4) length of responses to open-ended questions (measured by the number of characters the respondents entered); (5) survey completion time; (6) survey transmission time, that is, regarding the type of Internet connection: local area network (Wi-Fi or fixed-line Internet connection) versus cellular network architecture; (7) survey processing time; (8) where or in which situation the respondent completed the survey; and—for mobile users—(9) the technical characteristics of the device they used.

Based on the described theoretical considerations and empirical findings, we propose the following: First, since most respondents are used to carrying their mobile phone with them to be persistently accessible, we hypothesize that respondents who receive the study invitation on their mobile devices react more quickly than respondents who receive the invitation via their PC (Hypothesis 1). Second, since completing the survey using mobile devices often is less convenient and could be more time-consuming, the break-off rate in the mobile phone group should be higher than in the PC group (Hypothesis 2). For the same reasons, we expect that completing the survey on a smartphone¹ should lead to more item nonresponses (Hypothesis 3a), more nonsubstantive responses (Hypothesis 3b), and shorter replies in the open-ended questions (Hypothesis 4) compared to answering on the PC.

Completing a survey via smartphone can be more burdensome and more prone to failures; in addition, the technical specifications of smartphones (processor, power, etc.) are normally less impressive than those of PCs. For this reason, we assume a longer survey completion time in the mobile group than in the PC group (Hypothesis 5). The completion time consists of two components: the transmission time (survey page loading/sending time) and the processing time (the time elapsing between question presentation on the screen and the time the page was submitted by clicking “Next”). Due to a possibly slower Internet connection, we assume that the transmission time of respondents using a mobile device should be longer than using a PC (Hypothesis 6a). Furthermore, we expect that respondents using a Wi-Fi Internet connection (either mobile or PC group) or fixed-line Internet connection (PC group only) have less time to wait for the transmission of the survey pages than respondents using mobile devices that utilize a cellular network to access the Internet (Hypothesis 6b). Due to the described differences between the two technical devices, we hypothesize that the processing time in the mobile group will be longer than in the PC group (Hypothesis 7).

Mobile devices such as smartphones and tablets are made for portable use, during which respondents are often exposed to interruptions. Therefore, we additionally expect surveys on these devices to have been completed in different places (Hypothesis 8). In this regard, we also expect respondents completing a survey outside of their home to require more time than respondents who complete the survey at home. However, those completing the survey at home can also be interrupted by activities related to the household. Because our study involved a sample of students who, we assume, are mostly free of domestic commitments, we presume that survey participation is less likely to be interrupted at home than outside the home. Finally, we hypothesize differences within the mobile group regarding the technology used; specifically, it can be presumed that respondents who use modern mobile phones require less time to process the survey than respondents with outdated mobile hardware (Hypothesis 9). The screen size and resolution, as well as the computing power, are assumed to have a shortening effect on the processing time.

Sample and Design

The main interest of this study is to examine potential differences in data quality and response properties when different devices are used to respond to an online survey. In June 2014, we conducted two parallel unpaid and short online surveys (about 7 min in median) employing an adaptive web design optimized for mobile editing. We used 10,000 e-mail addresses selected using random onomastic sampling to contact students (Dingelstedt, 2015).² Each respondent was provided a unique URL in the invitation e-mail that directed them to the survey hosted by Unipark. Nonrespondents were recontacted once after 3 days.

To obtain a group of mobile device users (mobile group), and another one in which the respondents should use a desktop computer or laptop (PC group), invitations which differed in text and subject line were sent out via e-mail.³ To prevent self-selection bias by using the preferred device, the respondents were randomly assigned to completion mode (see Couper & Peterson, 2016; Keusch & Yan, 2016). We excluded all respondents for the following analysis who had not participated with the device prescribed in the invitation e-mail or who had dropped out of the survey.⁴ With the exception of the analyses for the dropout rate, however, we also used incomplete survey responses. Based on paradata⁵ from the “user agent strings” (Callegaro, 2010; Kreuter, 2013), we were able to identify the technical equipment used by the respondents and check for compliance with the given instructions.

A total of 526 of the respondents invited to the PC group and 294 of the respondents invited to the mobile group visited the first page; after a 12-day period, 78.3% of the PC group respondents and 67.7% of the mobile group respondents had completed the survey.

With respect to the dropout rate, we found no significant difference between the PC group (20.98%, n = 107) and the mobile group (26.11%, n = 41) if the designated device was used, χ²(1) = 1.55, p = .214. According to these criteria, using a different completion mode than requested meant that especially the mobile group had to be reduced to 116 valid cases. The PC group includes 403 respondents who qualified for further investigation (Table 1).

Similar to the findings of de Bruijne and Wijbant (2013), individuals who responded to the invitation to participate in the survey were more likely to use a computer than to follow the instruction to use a mobile device. This leads us to assume that many of the respondents opened their invitational e-mail on a computer; switching to a different device afterward may have represented an unacceptable additional burden.

The survey itself contained 32 questions. The survey started with key demographic variables (age and gender) and some questions about political issues. These were followed by items about the mayoral and the European election in 2014 and, finally, additional demographic questions (see Table 2 for an overview of descriptive statistics). We also asked respondents to tell us where/in which situation they answered the survey. For most questions, we used four to seven response categories on an ordinal vertical scale with radio buttons to select an answer. Twelve judgment items—the respondents had to decide between two statements—included a construct-specific answer format; three open-ended and one grid question were presented to the students (horizontal scrolling was not needed to see the full scales). Each question was placed on a separate survey page with the additional option to select a nonsubstantial answer (don’t know/prefer not to say). With regard to the survey completion time analysis, we tried to keep the design of the survey as simple as possible. The use of a complex layout and additional media increases files sizes and, as a consequence, increases download times as well (Couper, 2008).

Table 2.
Descriptive Sample Statistics for PC Group Versus Mobile Group.

Variable Experimental Group Mean SD Min Max n

Age ^ns PC group 24.55 3.30 19 38 364

Mobile group 24.36 2.82 19 33 105

Gender (male) ^ns PC group 0.50 0.50 0 1 388

Mobile group 0.47 0.50 0 1 113

Number of semesters (categorized) ^ns PC group 2.81 1.59 1 6 379

Mobile group 3.11 1.61 1 6 115

Political interest ^ns PC group 2.52 1.01 1 5 400

Mobile group 2.58 0.93 1 5 116

Socioeconomic classification ^ns PC group 3.16 0.72 1 5 377

Mobile group 3.21 0.69 1 4 110

Left–right self-identification ^ns PC group 3.23 1.23 1 7 383

Mobile group 3.46 1.10 1 7 114

^ns No significant differences were found (χ² test, p > .05).

For measuring transmission time as well as processing time, we used two sources of time data. On the one hand, server-side time data captured by Unipark, the online survey software we used. These survey data are page level based and measured in seconds. These measurements include the period between when the survey host (server) sends a survey page to the client (browser of the respondent) and when the next survey page is ready for transmission. Therefore, the captured period includes the transmission time (sending and receiving the survey page), the respondents processing time of the survey page, and the time period that is needed by the server to generate the next survey page. On the other hand, we used “Embedded Client Side Paradata” (Schlosser, 2016), a program code that allows researchers to measure time stamps at the page level directly in the respondent’s browser (client side) with precision to the millisecond. Moreover, this JavaScript- and Cascading Style Sheets-based paradata code provides two different kinds of time measurements: first, the period between the moment when a survey page is displayed completely in the respondents’ browser and the moment when the following survey page is also displayed completely and second, a measurement of respondents’ processing time, that is, the time between the moment when the question was presented on the screen and the moment when the page was submitted by clicking the “Next” button. The difference between these two measurements can be used to calculate the transmission time. This difference contains the time needed to send the survey page to the server, the period to generate the next survey page, and the elapsed time until the client receives and displays the new survey page. In our study, we assigned this transmission time to each subsequent survey page. A similar result can be obtained by using server-side-based and client-side-based time data (see Couper & Peterson, 2016), with the difference that using only client-side data (which is captured in milliseconds) to compute transmission times makes the measurement more accurate.

In our last hypothesis, we presumed that respondents who use modern mobile phones need less time to process the survey questions than respondents with outdated mobile hardware (Hypothesis 9). To clarify this, two basic characteristics of smartphones were examined: their physical properties and the computing power they possess. While increasing miniaturization was an objective in the early technical development of feature phones, the recent trend in the development of smartphones has been reversed (Sweeney & Crestani, 2006). Since smartphones are navigated exclusively by touching the display, it seems obvious that a larger screen size simplifies the survey completion process. At the same time, at higher display resolutions, more content—or the same content at a higher detail level—can be displayed. Second, greater computing power allows users to access content more quickly. This applies especially in cases where the user is scrolling, zooming, or flipping the display during the survey. To give an indication of the computing power of a device, we used the size of the available main memory, the clock of the CPU, and the number of cores it contains.

Results

Response Time to the Survey

With respect to our first hypothesis, we assumed that respondents who receive the invitation on their mobile devices react more quickly to the online survey than PC users. When we consider our statistical results, we observe that, in the first 3 days and prior to sending out the reminder, 48.1% of the final respondents in the PC group and 52.6% of the users of mobile devices completed the interview. Although there is a recognizable trend, namely, that mobile users reply faster than PC users, the average reaction time did not significantly differ between the groups in this period (PC group n = 194, median of 289 min; mobile group n = 61, median of 228 min; Mann–Whitney U test, p = .114).⁶ After reminding respondents to participate, the average reaction time differed significantly between the two groups (PC group n = 209, median of 617 min; mobile group n = 55, median of 225 min; p < .05). Over the entire period of the survey, the smartphone and tablet users responded significantly quicker to the survey (PC group n = 403, median of 4293 min; mobile group n = 116, median of 2979 min; p < .05; Figure 1 and Appendix A).

Figure 1.
Differences in the period between sending out the invitation e-mail and starting the survey for PC group versus mobile group. Indicates variables with significant differences (Mann–Whitney U test, p < .05, ^nsp* > .05).

Completion Rate/Break-Off Rate

Contrary to Hypothesis 2, the two experimental groups differ only slightly in terms of their break-off rates. Again, only respondents who completed the survey using the requested device were taken into account. The tendency to complete the survey was lower among users of mobile devices (73.9%, n = 157) compared to PC users (79.0%, n = 510). However, the difference in break-off rates is not statistically significant, χ²(1) = 1.55, p = .214; Table 3.

Table 3.
Differences in Completion Rate, Number of Unanswered Questions, and Nonsubstantive Answers (“Don’t Know,” “Prefer Not to Say”) for PC Group Versus Mobile Group.

Variable Experimental Group Mean SD Min Max n

Completion rate ^ns PC group 0.790 0.408 0 1 510

Mobile group 0.739 0.441 0 1 157

Number of unanswered questions ^ns PC group 0.010 0.199 0 4 403

Mobile group 0.009 0.093 0 1 116

Number of “don’t know” answers ^ns PC group 2.293 2.604 0 13 403

Mobile group 2.241 2.918 0 21 116

Number of “prefer not to say” answers ^ns PC group 1.702 2.988 0 31 403

Mobile group 1.405 1.943 0 9 116

^ns No significant differences were found (χ² test, p > .05).

Item Nonresponse

There were hardly any differences between the amount of nonsubstantive answers in the PC group n = 403 versus the mobile group n = 116, “don’t know,” χ²(10) = 13.91, p = .177; “prefer not to say” χ²(10) = 12.90, p = .229; ≥10 have been pooled; this applies equally to the frequency of blank or incomplete pages, χ²(4) = 3.77, p = .397; see Table 3. Given that, in some cases, the expected frequencies were slightly smaller than five, we additionally conducted Fisher’s exact test (see Choi, Blume, & Dupont, 2015) to validate the results of the χ² tests. All statistical results remained unchanged. With regard to the item nonresponse behavior between the experimental groups, Hypothesis 3 was not supported.

Open-Ended Questions

According to the fourth hypothesis, we would expect mode differences for answering open-ended questions; PC users would be expected to produce longer replies to open-ended questions and therefore use more keystrokes (Table 4). In the survey, three consecutive open-ended questions were asked about the most important political problems in Europe. Although there is a recognizable tendency of PC users to submit detailed statements, for none of the items did the quantity of keystrokes differ significantly between the groups (Mann–Whitney U test Question 1: p = .348, Question 2: p = .148, Question 3: p = .466). The overall willingness to provide an answer differed only slightly, Question 1: χ²(1) = 0.013, p = .908; Question 2: χ²(1) = 0.248, p = .619; Question 3: χ²(1) = 0.829, p = .363; see Appendix B.

Table 4.
Answer Length for Open-Ended Questions Posed to the PC Group Versus the Mobile Group (Number of Keystrokes).

Variable Experimental Group Mean Median SD Min Max n

Open-ended question I ^ns PC group 76.1 34.0 159.3 3 1,899 294

Mobile group 49.7 29.5 52.3 3 232 84

Open-ended question II ^ns PC group 54.0 28.5 99.1 5 1,132 250

Mobile group 35.8 24.0 40.2 5 296 69

Open-ended question III ^ns PC group 44.4 30.0 46.3 4 398 193

Mobile group 40.9 26.0 42.1 4 214 50

^ns No significant differences were found (Mann–Whitney U test, p > .05).

Completion Time

Based on time stamps logged for every single survey page, we were able to measure the actual time needed to complete the survey. In line with Hypothesis 5, our examination of the survey completion times revealed that mobile device users took, in median, 29.7% longer (490 s) than PC users (378 s) to complete the survey (Mann–Whitney U test p < .01; Table 5).

Table 5.
Survey Transmission, Processing, and Completion Times for PC Group Versus Mobile Group.

Variable Experimental Group Mean Median SD n

Survey transmission time (ms)** PC group 3,614,930 23,326 37,921,739 403

Mobile group 3,810,932 50,113 38,668,975 116

Survey processing time (ms)* PC group 483,575 348,395 575,629 403

Mobile group 509,367 414,625 713,485 116

Survey completion time (ms)** PC group 4,098,505 377,538 38,105,487 403

Mobile group 4,320,300 489,696 38,655,654 116

* and ** Indicate variables with significant differences (Mann–Whitney U test, p < .05, p < .01).

We obtained a similar result at the survey page level for separated items. Mode effects were able to be registered for nearly all items. Similar to the used keystrokes count, this effect seems to carry less weight for open-ended questions, for which, in two of the three cases, no differences were recorded (Figure 2).

Figure 2.
Cumulative medians of completion times on survey page level for PC group versus mobile group. “Item no.” refers to our calculation of Mann–Whitney U tests for each survey page separately, where *p < .05 and **p < .01. “Section” refers to the section or topic specified for each questionnaire page, that is, I (intro), D (demographic question), J (judgment question—the respondents had to decide between two statements), L (Likert-type scale question), G (grid question), O (open-ended question).

At least two possible reasons can be cited for the difference in completion time between the survey modes. First, a slower smartphone Internet connection during participation in the survey could cause long periods of downloading and sending (see Buskirk & Andrus, 2012, in regard to iPhones). Second, participation via mobile devices is more time-consuming when answering the survey questions. To investigate these questions and whether this effect is limited only to certain types of questions, the transmission and processing times were analyzed separately for every single item (Hypotheses 6a, 6b, and 7).

When comparing the measured transmission time between mobile devices and the survey host (Hypothesis 6a), we found that these took, on average, about twice as much time in median as the transmission time to a PC (cumulative medians over 32 items: 21.67 s for the PC group vs. 41.56 s for the mobile group; Mann–Whitney U test p < .05). Regardless of the type of question, highly significant differences were observed (Figure 3a).

Figure 3.
(a) Cumulative medians of survey transmission times for PC group versus mobile group. “Item no.” refers to our calculation of Mann–Whitney U tests for each survey page separately, where *p < .05 and **p < .01. (b) Cumulative medians of survey transmission times for PC group, mobile group with Wi-Fi connection, and mobile group connected via cellular network. “Item no.” refers to our calculation of Mann–Whitney U tests between mobile cellular group and mobile Wi-Fi group for each survey page separately, where *p < .05 and **p < .01. See description of Figure 2 for a clarification of “Section.”

Furthermore, we assumed that the type of Internet connection (Wi-Fi vs. cellular network) and its different transmission rates could affect the transmission time (Hypothesis 6b). Using the recorded paradata, the opportunity arose to draw inferences on the connection type used.⁷ Even more clearly than in the previous comparison, it turns out that—just for the use of the cellular network architecture—very long transfer times arise (cumulative medians over 32 items: 62.3 s; Figure 3b). Almost all items differ significantly in terms of their transmission times between the mobile group connected via cellular network, mobile device users connected via Wi-Fi and the PC group (on survey level see Appendix C).

Similar results were found in the comparison of the processing times (Hypothesis 7), in which, at 17.6%, the total differences between the groups turn out slightly lower (cumulative medians over 32 items: 297.02 s for the PC group vs. 349.44 s for the mobile group; Mann–Whitney U test, p < .05), although this effect includes a higher proportion of the total difference (Figure 4; Table 5).

Figure 4.
Cumulative medians of survey processing times for PC group versus mobile group. “Item no.” refers to our calculation of Mann–Whitney U tests for each survey page separately, where *p < .05 and **p < .01. See description of Figure 2 for a clarification of “Section.”

Altogether, it may be concluded that both the transmission time and the processing time contribute to higher completion times for mobile device users.

Home Versus Away

In order to investigate whether the place and/or situation in which respondents completed the survey influences the data quality and the response process, we make a comparison between respondents who indicated that they were at home and others who indicated that they were away from home (Hypothesis 8). Theoretically, being at home or away from home should only affect the processing time during the online survey. Previous investigations have shown that the processing time is influenced by the type of hardware that respondents used (Cunningham et al., 2013; Gummer & Roßmann, 2015; Mavletova, 2013; Zahariev et al., 2009). Therefore, the analysis of a possible location effect while completing the survey will also be made separately for the mobile group versus the PC group. Unexpectedly, the proportion of respondents who indicated that they were not at home was relatively high in the PC group (34.4%) as compared to mobile users (46.5%; see Appendix D). After analyzing the processing times, we observed that only 3 of the 32 items differed because of the location where a mobile device was used (PC group: 9 of the 32 items; Figure 5). Apparently, therefore, location does not affect processing time.

Figure 5.
Cumulative medians of survey processing times for PC group and mobile group—home versus away. “Item no.” refers to our calculation of Mann–Whitney U tests between PC away group and mobile away group for each survey page separately, p < .05, p < .01. See description of Figure 2 for a clarification of “Section.”

This result could be also due to the fact that the item inquiring as to where respondents were completing the online survey was not optimized for interviewing students. Most respondents who indicated that they were away from home chose the answer option “University (library, cafeteria, campus)” (see Appendixxs D). Given the assumption that respondents need more time to complete the survey when they are disturbed in public places, the degree of interference between being at the cafeteria and the library could vary greatly.

Technical Aspects of Mobile Devices

In our last hypothesis, we presumed that respondents using modern mobile phones need less time to process the survey than respondents with outdated mobile hardware (Hypothesis 9). Two basic characteristics of smartphones were examined: firstly, the computing power. We used the size of the available main memory, the clock of the CPU, and the number of cores it contains. Secondly, we examined the physical properties of the smartphones, including their screen size and display resolution. It is not surprising that, in the development of new devices, none of these parameters should be neglected. This strong correlation between the technical characteristics can also be observed in the present data (all r > .65, p < .01). To determine these technical details, the user agent strings can also be used. Based on this information, it is possible to determine the brand and model designation of the device being used to subsequently identify its technical characteristics on the basis of the product description (e.g., the mobile device “Samsung GT-I9506” can be identified from the following user agent string “Mozilla/5.0 [Linux; Android 4.3; GT-I9506 Build/JSS15J] AppleWebKit/537.36 [KHTML, like Gecko] Chrome/35.0.1916.138 Mobile Safari/537.36”). A certain degree of uncertainty in the interpretation of user agent strings could arise if mobile devices with slightly modified hardware were distributed with the same model designation. Another disadvantage of this approach is that, in our study, it is not possible to identify the exact product series of Apple devices.⁸ Accordingly, only Android devices were available for further steps, which reduced the number of cases to n = 45.

Based on the low number of cases, we divide modern and outdated smartphones into two groups using the median of their technical specifications. We classified devices as “hi-end” (n = 25) if one of the investigated aspects (Table 6) was technically superior to the median of all devices. Mobile devices were assigned to the “low-end” group (n = 20) if their computing power or their physical properties were lower than or equal to the respective medians in all respects (main memory ≤ 1 GB; CPU cores ≤ 2; CPU clock speed ≤ 1400 MHz; display size ≤ 4.3 in.; display resolution ≤ 960 pixel).

Table 6.
Technical Characteristics of Android-Based Smartphones.

Variable Mean Median SD Min Max n

Main memory (GB) 1.12 1.00 0.55 0.15 2.00 45

CPU cores 2.64 2.00 1.17 1.00 4.00 45

CPU clock speed (MHz) 1,349 1,400 388.5 600 2,260 45

Display size (in.) 4.39 4.30 0.54 3.00 5.50 45

Resolution (pixel X) 1,061 960 469.9 320 1,920 45

Table 7 shows that, consistent with our hypothesis, respondents in the low-end group required, in median, about 37.6% longer to process the survey (468 s) than those using modern devices (340 s; Mann–Whitney U test, p < .05). We further found no significant differences for processing the survey between the PC group and the mobile group given that a modern smartphone was used (p = .668). Likewise, there no significant results found for transmission times when comparing the mobile groups, although these were slightly higher for the low-end devices (p = .132). As might be expected from preceding investigations, however, both mobile groups needed significantly longer to transfer the survey data (p < .01).

Table 7.
Differences in Survey Transmission, Procession, and Completion Times Due to Technical Characteristics of Mobile Devices Versus PC Group.

Variable Experimental Group Mean Median SD n

Survey transmission time (ms)^{a), b), c)ns} PC group 3,614,930 23,326 37,921,739 403

Mobile hi-end 770,919 47,215 3,407,693 25

Mobile low-end 104,766 67,497 90,530 20

Survey processing time (ms)^{a)ns, b), c)} PC group 483,575 348,395 575,629 403

Mobile hi-end 645,452 339,898 1,477,432 25

Mobile low-end 467,871 468,433 165,868 20

Survey completion time (ms)^{a)ns, b)**, c)} PC group 4,098,505 377,538 38,105,487 403

Mobile hi-end 1,416,371 387,113 3,694,332 25

Mobile low-end 572,637 529,685 217,539 20

Note. Eight respondents in the PC group and three in the mobile group completed the survey after a long interruption. ^a) ^Indicates significant differences between PC group and mobile hi-end group (Mann–Whitney U Test, p < .01, ^a)ns p > .05). PC group vs. mobile low-end group (^b) p < .01). Mobile hi-end group vs. mobile low-end group (^c) p < .05, ^c)ns p > .05).

Nevertheless, this result supports the assumption that the use of the latest smartphones (as compared to older devices) can reduce the processing time in mobile online surveys. In fact, our assumptions were even exceeded, as there were no time differences for up-to-date smartphones versus PCs when processing the survey. Particularly with regard to the low number of cases, and taking into account that students are probably very well versed in dealing with mobile devices, further studies on this issue are needed. Nevertheless, these results can be used to get a basic idea of the possible reasons for breaking off the survey. In the mobile group n = 41, respondents did not complete the survey. Seven of the 14 identified mobile device models could be classified as low-end.⁹

Conclusion and Discussion

The aim of our study was to investigate the data quality and response process of mobile online surveys compared to traditional PC online interviews. Furthermore, using comprehensive paradata, we were able—for the first time—to use a German sample to examine the impact of data transfer and hardware properties on survey completion time.

Our study revealed that the data quality and the response properties was comparable in most aspects. We found that the differences between the types of devices vanish to a great extent when controlling for the type of Internet connection and hardware properties.

The results of comparing self-administered web surveys using a PC to those completed on mobile devices can be summarized as follows: Contrary to the expectations and findings from other studies (Bosnjak et al., 2013; de Bruijne & Wijbant, 2013; Mavletova, 2013), no difference in terms of break-off rate was observed between the PC and the mobile group. We were even able to confirm our hypothesis about item nonresponse and the number of keystrokes used to answer open-ended questions (as well as where respondents answered the questionnaire). In addition, the mobile group performed better than the PC group with respect to one point: Users of mobile devices responded more quickly to our survey invitation. Only with regard to completion time was the mobile survey at a disadvantage in comparison to the PC survey: Mobile device users needed approximately 30% more time than PC users to finish the survey. This effect can be attributed to longer transmission and processing times. Respondents completing the online survey on a mobile device had transmission times 3 times longer than respondents completing it on their computer; the total processing time when using a mobile device was approximately 18% higher than when using a PC. However, upon closer inspection, we observed that the advantage of using a PC in online surveys with regard to transmission time disappears if we take the type of Internet connection into account. If the mobile device was connected via Wi-Fi, the difference decreased significantly. Furthermore, the use of new, up-to-date mobile hardware leads to shorter processing times for mobile devices. With these results in mind, it can be assumed that the differences between mobile devices and PCs found in previous studies (Bosnjak et al., 2013; Mavletova, 2013) are, to a large extent, most likely attributable to the type of Internet connection and hardware characteristics and not to the interview mode used.

Our study has several limitations and opens avenues for future research. Firstly, the results of our study are based on a sample of students who are generally considered to be “digital natives” in terms of their ability to interface with mobile devices. Including a sampling method from the general population and increasing the sample sizes would have significantly strengthened the ability to generalize our results.¹⁰ Additionally, it must be taken into account that our findings are based on a survey conducted in Germany. Especially for the analyses of transmission times, the results may only be transferred to regions with similar mobile infrastructures. Similarly, it should be considered that we were not able to differentiate between the type of cellular Internet connection (e.g., Enhanced Data rates for Global Mobile Communications Evolution (EDGE), third generation Universal Mobile Telecommunications System (3G), and Long Term Evolution (LTE)) and did not have any means of measuring the actual speed of the connection. Secondly, our questionnaire was relatively short and easy to complete. Varying the length and difficulty could be a benefit to future research. Thirdly, the questionnaire was optimized for mobile devices with an adaptive Web design, with the result that the findings should not be transferred to surveys with a “classical design.” Fourthly, based on the restricted technical data available to us, we were only able to examine smartphones with the Android operating system. In further research, the collection of additional paradata could be helpful toward identifying and classifying all types of mobile equipment used. Likewise, dealing directly with these mobile devices would allow more detailed conclusions to be drawn about the usage behavior of respondents.¹¹ Fifthly, we were not able to determine whether respondents had switched devices during the survey. This was only able to be ascertained for 1.99% of the respondents who started the survey on a PC, while 2.59% of those who started the survey on a mobile device left the survey for a certain period of time prior to completing it.

There are several open questions that lie beyond the scope of this study. Due to the low number of respondents using a tablet in this study, we were not able to further pursue the question as to whether the use of a tablet is more similar to using a PC or to using a smartphone (Gummer & Roßmann, 2015). Furthermore, given the small number of cases in the mobile group, we are not able to make a clear statement about whether slow transmission rates caused an increase in break-off rates. In order to gain a more detailed understanding of web-based surveys with mobile devices, it is of great relevance that these issues be clarified and researched further.

In summary, as long as a (reasonably short) web survey is suitable for a certain population, no major quality losses need be feared given the use of mobile devices. With the expansion of the transmission capacity of mobile networks and ever-increasing performance of mobile devices, the differences identified to date between smartphones and standard PCs will continue to diminish in the future.

	Invited to PC Group	Invited to Mobile Group	Total
Responded with PC	510	137	647
Responded with mobile devices	16	157	173
Total	526	294	820
▾ Exclusion of drop-outs ▾
	Invited to PC Group	Invited to Mobile Group	Total
Responded with PC	403	80	483
Responded with mobile device	9	116	125
Total	412	196	608

Variable	Experimental Group	Mean	SD	Min	Max	n
Age ^ns	PC group	24.55	3.30	19	38	364
Mobile group	24.36	2.82	19	33	105
Gender (male) ^ns	PC group	0.50	0.50	0	1	388
Mobile group	0.47	0.50	0	1	113
Number of semesters (categorized) ^ns	PC group	2.81	1.59	1	6	379
Mobile group	3.11	1.61	1	6	115
Political interest ^ns	PC group	2.52	1.01	1	5	400
Mobile group	2.58	0.93	1	5	116
Socioeconomic classification ^ns	PC group	3.16	0.72	1	5	377
Mobile group	3.21	0.69	1	4	110
Left–right self-identification ^ns	PC group	3.23	1.23	1	7	383
Mobile group	3.46	1.10	1	7	114

Variable	Experimental Group	Mean	SD	Min	Max	n
Completion rate ^ns	PC group	0.790	0.408	0	1	510
Mobile group	0.739	0.441	0	1	157
Number of unanswered questions ^ns	PC group	0.010	0.199	0	4	403
Mobile group	0.009	0.093	0	1	116
Number of “don’t know” answers ^ns	PC group	2.293	2.604	0	13	403
Mobile group	2.241	2.918	0	21	116
Number of “prefer not to say” answers ^ns	PC group	1.702	2.988	0	31	403
Mobile group	1.405	1.943	0	9	116

Variable	Experimental Group	Mean	Median	SD	Min	Max	n
Open-ended question I ^ns	PC group	76.1	34.0	159.3	3	1,899	294
Mobile group	49.7	29.5	52.3	3	232	84
Open-ended question II ^ns	PC group	54.0	28.5	99.1	5	1,132	250
Mobile group	35.8	24.0	40.2	5	296	69
Open-ended question III ^ns	PC group	44.4	30.0	46.3	4	398	193
Mobile group	40.9	26.0	42.1	4	214	50

Variable	Experimental Group	Mean	Median	SD	n
Survey transmission time (ms)**	PC group	3,614,930	23,326	37,921,739	403
Mobile group	3,810,932	50,113	38,668,975	116
Survey processing time (ms)*	PC group	483,575	348,395	575,629	403
Mobile group	509,367	414,625	713,485	116
Survey completion time (ms)**	PC group	4,098,505	377,538	38,105,487	403
Mobile group	4,320,300	489,696	38,655,654	116

Variable	Mean	Median	SD	Min	Max	n
Main memory (GB)	1.12	1.00	0.55	0.15	2.00	45
CPU cores	2.64	2.00	1.17	1.00	4.00	45
CPU clock speed (MHz)	1,349	1,400	388.5	600	2,260	45
Display size (in.)	4.39	4.30	0.54	3.00	5.50	45
Resolution (pixel X)	1,061	960	469.9	320	1,920	45

Variable	Experimental Group	Mean	Median	SD	n
Survey transmission time (ms)^{a), b), c)ns}	PC group	3,614,930	23,326	37,921,739	403
Mobile hi-end	770,919	47,215	3,407,693	25
Mobile low-end	104,766	67,497	90,530	20
Survey processing time (ms)^{a)ns, b)*, c)}	PC group	483,575	348,395	575,629	403
Mobile hi-end	645,452	339,898	1,477,432	25
Mobile low-end	467,871	468,433	165,868	20
Survey completion time (ms)^{a)ns, b)*, c)}	PC group	4,098,505	377,538	38,105,487	403
Mobile hi-end	1,416,371	387,113	3,694,332	25
Mobile low-end	572,637	529,685	217,539	20

Footnotes

Appendix A

Table A1.

Differences in the Period Between Sending Out the Invitation E-Mail and Starting the Survey for the PC Group Versus the Mobile Group.

Variable	Experimental Group	Mean	Median	SD	Min	Max	n
Period until starting the survey before the reminder ^ns	PC group	834.5	288.5	1,014.5	1	4,239	194
Period until starting the survey before the reminder ^ns	Mobile group	750.3	228.0	1,080.5	1	4,378	61
Period until starting the survey after the reminder*	PC group	2,586.3	617.0	3,339.5	1	11,844	209
Period until starting the survey after the reminder*	Mobile group	1,921.9	225.0	3,152.9	1	11,365	55
Entire period until starting the survey*	PC group	3,974.1	4,293.0	3,927.8	1	16,215	403
Entire period until starting the survey*	Mobile group	3,344.3	2,978.5	3,579.3	1	15,706	116

*Indicates variables with significant differences (Mann–Whitney U test, p < .05, ^nsp > .05).

Appendix B

Table B1.

Differences in Completion Rate of Open-Ended Questions for PC Group Versus Mobile Group.

Variable	Experimental Group	Mean	SD	Max	n
Completion rate open-ended question I ^ns	PC group	0.730	0.445	1	403
Completion rate open-ended question I ^ns	Mobile group	0.724	0.449	1	116
Completion rate open-ended question II ^ns	PC group	0.620	0.486	1	403
Completion rate open-ended question II ^ns	Mobile group	0.595	0.493	1	116
Completion rate open-ended question III ^ns	PC group	0.479	0.500	1	403
Completion rate open-ended question III ^ns	Mobile group	0.431	0.497	1	116

^ns No significant differences were found (χ² test, p > .05).

Appendix C

Table C1.

Survey Transmission, Processing, and Completion Times for PC Group, Mobile Group With Cellular Internet Connection, and Mobile Group With Wi-Fi Internet Connection.

Variable	Experimental Group	Mean	Median	SD	n
Survey transmission time (ms)^{a), b), c)**}	PC group	3,614,930	23,326	37,921,739	403
	Mobile cellular	9,516,665	72,080	61,368,860	46
	Mobile Wi-Fi	61,452	39,733	84,294	70
Survey processing time (ms)^{a)*, b)ns, c)ns}	PC group	483,575	348,395	575,629	403
	Mobile cellular	596,569	432,157	1,083,550	46
	Mobile Wi-Fi	452,063	411,568	272,635	70
Survey completion time (ms)^{a), b), c)ns}	PC group	4,098,505	377,538	38,105,487	403
	Mobile cellular	10,113,233	530,152	61,332,578	46
	Mobile Wi-Fi	513,515	467,277	293,407	70

^a)*Indicates significant differences between PC group and mobile group with cellular Internet connection (Mann-Whitney U test, p < .05, ^a)** p < .01, ^a)ns p > .05). PC group vs. mobile group with Wi-Fi Internet connection (^b)** p < .01, ^b)ns p > .05). Mobile group with cellular Internet connection vs. mobile group with Wi-Fi Internet connection (^c)** p < .01, ^c)ns p > .05).

Appendix D

Table D1.

Place and/or Situation in Which Respondents Completed the Survey for PC Group and Mobile Group.

Variable	Home Away	Experimental Group	Frequency	%	Valid Percent
At home	Home	PC group	254	63.0	65.6
At home	Home	Mobile group	61	52.6	53.5
University (library, cafeteria, campus)	Away	PC group	88	21.8	22.7
University (library, cafeteria, campus)	Away	Mobile group	35	30.2	30.7
At work	Away	PC group	38	9.4	9.8
At work	Away	Mobile group	7	6.0	6.1
At cafe, bar, restaurant, etc.	Away	PC group	0	0.0	0.0
At cafe, bar, restaurant, etc.	Away	Mobile group	2	1.7	1.8
In bus, train, car, etc.	Away	PC group	0	0.0	0.0
In bus, train, car, etc.	Away	Mobile group	2	1.7	1.8
Elsewhere away from home	Away	PC group	7	1.7	1.8
Elsewhere away from home	Away	Mobile group	7	6.0	6.1
Don’t know		PC group	0	0.0
Don’t know		Mobile group	1	0.9
Prefer not to say		PC group	16	4.0
Prefer not to say		Mobile group	1	0.9
Total		PC group	403	100	100
Total		Mobile group	116	100	100

Authors’ Note

The authors would like to thank Steffen M. Kühnel (University of Göttingen, Germany) as well as Jan K. Höhne (University of Göttingen, Germany) for their excellent comments on this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Bosnjak

Poggio

Becker

K. R.

Funke

Wachenfeld

Fischer

(2013). Online survey participation via mobile devices. Conference Paper, The American Association for Public Opinion Research (AAPOR) 68th Annual Conference 2013, Boston, MA.

Bosnjak

Tuten

T. L.

(2001). Classifying response behaviors in web-based surveys. Computer-Mediated Communication, 6. Retrieved October 20, 2016, from http://onlinelibrary.wiley.com/doi/10.1111/j.1083-6101.2001.tb00124.x/full

Buskirk

T. D.

Andrus

C. H.

(2012). Online surveys aren’t just for computers anymore! Exploring potential mode effects between smartphone and computer-based online surveys. Conference Paper, Annual Conference of the American Association for Public Opinion Research 2012, Orlando, FL.

Buskirk

T. D.

Andrus

C. H.

(2014). Making mobile browser surveys smarter. Results from a randomized experiment comparing online surveys completed via computer or smartphone. Field Methods, 26, 322–342.

Callegaro

(2010). Do you know which device your respondent has used to take your online survey? Survey Practice, 3. Retrieved October 20, 2016, from http://www.surveypractice.org/index.php/SurveyPractice/article/view/250/pdf

Choi

Blume

J. D.

Dupont

W. D.

(2015). Elucidating the foundations of statistical inference with 2 × 2 table. PloS One 10. doi:10.1371/journal.pone.0121263

Couper

M. P.

(2008). Designing effective web surveys. Cambridge, NY: Cambridge University Press.

Couper

M. P.

Kreuter

(2013). Using paradata to explore item level response times in surveys. Journal of the Royal Statistical Society (Series A), 176, 271–286.

Couper

M. P.

Peterson

(2016). Why do web surveys take longer on smartphones? Social Science Computer Review. First published on February 11, 2016. doi:10.1177/0894439316629932

10.

Cunningham

J. A.

Neighbors

Bertholet

Hendershot

C. S.

(2013). Use of mobile devices to answer online surveys: Implications for research. BMC Research Notes, 6, 258.

11.

de Bruijne

Wijbant

(2013). Comparing survey results obtained via mobile devices and computers: An experiment with a mobile web survey on a heterogeneous group of mobile devices versus a computer-assisted web survey. Social Science Computer Review, 31, 482–504.

12.

Dingelstedt

(2015). Die Wirkung von Incentives auf die Antwortqualität in Umfragen. Göttingen, Germany: ePublikation Staats- und Universitätsbibliothek Göttingen.

13.

Gummer

Roßmann

(2015). Explaining interview duration in web surveys. A multilevel approach. Social Science Computer Review, 33, 217–234.

14.

Johnson

Kelly

Stevens

(2012). Modular survey design for mobile devices. A research study on the potential of modular survey design for mobile and online surveys. Conference Paper, CARSO Online Research Conference, March 2, 2012, Las Vegas, NV.

15.

Keusch

Yan

(2016). Web versus mobile web: An experimental study of device effects and self-selection effects. Social Science Computer Review. First published on November 2, 2016. doi:10.1177/0894439316675566

16.

Kreuter

(Ed.). (2013). Improving surveys with paradata. Analytic uses of process information. Hoboken, NJ: Wiley (Wiley series in survey methodology).

17.

Mavletova

(2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31, 725–743.

18.

Mavletova

Couper

M. P.

(2015). A meta-analysis of breakoff rates in mobile web surveys. In Toninelli

Pinter

de Pedraza

(Eds.), Mobile research methods: Opportunities and challenges of mobile research methodologies (pp. 81–98). London, England: Ubiquity Press.

19.

Mavletova

Couper

M. P.

(2016). Grouping of items in mobile web questionnaires. Field Methods, 28, 170–193.

20.

Millar

M. M.

Dillman

D. A.

(2012). Encouraging survey response via smartphones: Effects on respondents use of mobile devices and survey response rates. Survey Practice, 5. Retrieved October 20, 2016, from http://www.surveypractice.org/index.php/SurveyPractice/article/view/19/pdf

21.

Peytchev

Hill

C. A.

(2010). Experiments in mobile web survey design. Similarities to other modes and unique considerations. Social Science Computer Review, 28, 319–335.

22.

Schlosser

(2016). Embedded client side paradata (ECSP). doi:10.5281/zenodo.55169. Retrieved October 20, 2016, from https://zenodo.org/record/55169#.V1gBT-RSE4E

23.

Stapleton

C. E.

(2013). The smartphone way to collect survey data. Survey Practice, 6. Retrieved October 20, 2016, from http://www.surveypractice.org/index.php/SurveyPractice/article/view/75/pdf

24.

Sweeney

Crestani

(2006). Effective search results summary size and device screen size. Is there a relationship? Information Processing & Management, 42, 1056–1074.

25.

Toepoel

Lugtig

(2014). What happens if you offer a mobile option to your web panel? Evidence from a probability-based panel of internet users. Social Science Computer Review, 32, 544–560.

26.

Wells

Bailey

J. T.

Link

M. W.

(2014). Comparison of smartphone and online computer survey administration. Social Science Computer Review, 32, 238–255.

27.

Yan

Tourangeau

(2008). Fast times and easy questions. The effects of age, experience, and question complexity on web survey response times. Applied Cognitive Psychology, 22, 51–68.

28.

Zahariev

Ferneyhough

Ryan

(2009). Best practices in mobile research. Chicago, IL: ESOMAR Online Research.