The Effects of Grids on Web Surveys Completed with Mobile Devices

Abstract

Question grids are common on Web surveys, and studies show that grids can affect how respondents complete surveys. However, there is little research that investigates the effects of grids on Web surveys completed on mobile devices. In this article, we evaluate the effects of question grids on response quality and measurement error for surveys taken on phones or tablets. Our study draws on a probabilistic Web survey. The survey included an experiment in which respondents were assigned to one of three question format conditions: one large grid, two small grids, or single item per page. We analyze how question grids affect response times and nondifferentiation as well as explore the interaction effects between grids and devices. Reductions in time associated with question grids were greater for surveys completed on mobile devices as opposed to those completed on computers. Likewise, the increases in nondifferentiation associated with question grids were greater for surveys completed on mobile devices. We find that effects of question grids on responses in Web surveys can differ across devices, and so researchers should be cautious of using grids on Web surveys as more people opt to do surveys on phones or tablets.

Keywords

web surveys survey research mobile Internet surveys smartphone communication and information technologies methodology sociological practice public sociology

Introduction

Grids, also known as matrices, are commonly used in Web surveys because they allow multiple question items with a similar question stem to appear on a single screen rather than presenting one item per page. Past studies investigating the effects of grids on Web surveys find that they affected substantive survey responses, thus increasing measurement error (Dillman, Smyth, and Christian 2009; Gräf 2002; Poynter 2001; Wojtowicz 2001). However, there is little research on how grids affect surveys completed via mobile devices such as smartphones or tablets.

An increasing portion of survey respondents are opting to complete surveys via mobile phones or tablets (Buskirk 2013; McGeeney and Marlar 2013). Studies indicate that there are differences in how people complete surveys on computers and mobile devices (Peytchev and Hill 2010; Wells, Baily, and Link 2013). This raises an important question that is largely unaddressed in the literature: how do grids affect survey responses and measurement error on surveys completed with mobile devices?

Both the visual design of a Web survey and the device people use to complete it can affect responses and increase measurement error (McCutcheon and Wang 2015). In terms of grids, studies have found that people tend to respond more quickly when questions are presented in a one-page grid rather than on separate pages (Couper, Trougott, and Lamias 2001; Tourangeau, Conrad, and Couper 2013; Tourangeau, Couper, and Conrad 2004). Research also shows that question grids can lead to more item nonresponse and more nondifferentiation (i.e., the selection of the same answer choice for every item in a set of questions; (Manfreda, Batagelj, and Vehovar 2002; Tourangeau et al. 2004). In addition to visual design, the technology of specific devices can also affect responses and data quality on Web surveys. As mobile devices have smaller screens than computers, scrolling across items and typing responses can be more difficult. Studies have found that Web survey design that is optimized for computers may not work well for Web surveys completed on mobile devices such as phones or tablets (Callegaro 2013; Macer 2012), yet the impact of grids is less understood.

In this article, we measure the effects of grids in Web surveys completed via mobile devices. We use data from a survey conducted between November 2013 and April 2014, featuring a representative sample of 494 (466 complete) Illinois residents, including 70 (62 complete) respondents who took the Web survey on mobile phones or tablets. Respondents were randomly assigned to one of three question format conditions: (1) one question item per page, (2) two small question grids, or (3) one large question grid. Within this experimental design, we analyze whether question grids affect response times and nondifferentiation when compared with a single-item-per-page design for mobile and nonmobile respondents.

Theoretical Background

The Importance of Visual Design

A significant amount of extant research on survey design has explored how visual design elements affect respondent behavior (e.g., Christian and Dillman 2004; Jenkins and Dillman 1997; Redline and Dillman 2002); however, fewer studies have applied this work to unimodal Web surveys. Because Web surveys are self-administered, it is especially important for them to be designed intuitively and clearly (Schwarz 1996). Respondents must be able to effectively navigate a survey interface to minimize the potential for measurement error (Casey and Poropat 2014).

The Use of Grids

Grids have advantages and disadvantages over alternative approaches. On one hand, grids reduce the number of pages in a survey and can decrease completion times (Couper et al. 2001; Tourangeau et al. 2013; Tourangeau et al. 2004). One such study even found that completion time reduction comes without sacrificing data quality in exchange (Couper et al. 2001). On the other hand, research shows that grids can also affect response quality and increase measurement error (e.g., Couper 2008; Gräf 2002; Poynter 2001; Wojtowicz 2001). Dillman et al. (2009) found that the instructions and design for grid questions are frequently difficult for respondents to quickly grasp, leading to a greater possibility for measurement error and item nonresponse. Furthermore, studies show higher instances of nondifferentiation and item nonresponse with grid use (Manfreda et al. 2002; Tourangeau et al. 2004), and some research has found that grid usage results in more break offs (Jeavons 1998; Tourangeau et al. 2013). Considering these potential drawbacks, a number of studies have explored ways to improve the design of grids such as dynamic shading and feedback (Couper et al. 2013; Galesic et al. 2007; Kaczmirek 2009). Research also indicates that the size of grids (i.e., the number of items included in a grid) can affect survey quality. In particular, there is evidence of greater potential measurement error or satisficing when more items are included in the same grid (Dillman et al. 2009).

The Use of Mobile Devices to Complete Web Surveys

In recent years, smartphone and tablet ownership has proliferated in the United States, forging a new path for Web survey research. According to a 2015 Pew report, 68 percent of the American adult population owns a smartphone compared with just 35 percent in 2011 (Anderson 2015; Smith 2013). A similar trend has been found regarding tablet and e-reader ownership (Yelton 2012). In 2015, Anderson reported that 45 percent of Americans over the age of 16 own a tablet, up from only 3 percent in 2010 (Anderson 2015; Rainie and Smith 2013). What is more, Pew Internet and American Life Research Center (Miller, Purcel, and Rosenstiel 2012:7) recently reported that the U.S. adults who own a smartphone do the majority of their Web searching on this device instead of a computer.

As more Americans are accessing the Internet on their mobile devices, there is a growing trend in both academic and market research of respondents using mobile devices to complete Web surveys (Return Path 2013). Recent research indicates that respondents are now choosing to complete Web surveys on their own mobile devices (Buskirk 2013; McGeeney and Marlar 2013), with mobile response rates ranging from 5 to 25 percent (Bosnjak, Poggio, and Funke 2013; Comer and Saunders 2012; Wells et al. 2013). Much of the existing research on Web surveys completed on mobile devices has focused on respondent-side factors, such as respondent satisfaction (Bosnjak, Metzger, and Gräf 2010; Conrad et al. 2013; de Bruijne and Wijnant 2013) and respondents’ perceptions of survey difficulty (Peytchev and Hill 2010). More recent research has investigated mode effects on Web surveys, studying a variety of factors such as respondent behavior (Dominguez and de Rada 2015; Peytchev and Hill 2010), responses on open-ended items (Buskirk and Andrus 2012; Mavletova 2013; Zachariev, Ferneyhough, and Ryan 2009), and frequency scales (Wells et al. 2013). However, there is a recognition for the need to understand how to adjust Web surveys for the mobile world (Bilgen, Stern, and Sterrett 2014; Callegaro 2013; Macer 2012).

Method

Data

For this study, we analyzed the data from the Healthy Illinois Survey, an address-based sampling (ABS), probabilistic Web survey conducted by National Opinion Research Center (NORC) from November 2013 to April 2014. All respondents were given a $2 preincentive, and received both e-mail and letter reminders to complete the survey. The response rate was 12 percent (494 total), with 466 complete and 28 incomplete. Of these respondents, 13 percent (62 complete and eight incomplete) reported taking the survey using a mobile device.

The study featured three multiitem experimental grid questions. The first experimental grid question had four items, the second had seven items, and the third had five items. Half of the respondents received the first experimental grid question, and all respondents received the second and third experimental grid questions. Respondents were randomly assigned to one of three groups. Group A was given each of the experimental question items in a single large grid; that is, all question items were presented on one page (see Figure 1). Group B received question items in two smaller grids over the course of a few pages (see Figure 2). Last, Group C was given each question item on a separate page (see Figure 3). The true experimental design alternative to Groups A and B would be the sequence of questions on the same page in a Web survey. However, including all items in one page in Web survey would force respondents to scroll down, which will increase respondent burden as well as may cause respondent to miss several items (specifically in devices with smaller screens). Hence, this control group is not included in our study.

Figure 1.

Question 2 presented in the large grid design.

Figure 2.

Question 2 presented in the small grids design.

Figure 3.

Question 2 presented in the single-item-per-page design.

Key Independent Variables

Mobile

The device respondents used was coded as a dichotomous variable. At the start of the survey, all respondents were asked the following: “First, please tell us how you’re completing this survey. Is it on a . . . desktop computer; laptop computer; tablet such as an iPad, Kindle, or similar device; mobile phone; or other?” The variable was coded as 1 for mobile device (phone or tablet) and 0 for nonmobile device (desktop or laptop computer). There were 62 people who completed with mobile and 404 who completed with computer (Table 1).

Table 1.

Number of Completes in Each Condition.

Taken on Nonmobile or Mobile Device	Single item per page	Small grids	Large grid	Total
Nonmobile	130	140	140	404
Mobile	18	21	27	62
Total	148	155	176	466
		Total for any grid = 331

Small grids

A dichotomous variable was coded as 1 if respondents received the small grids treatment and 0 if they received either the large grid or the single-item-per-page treatments.

Large grid

A dichotomous variable was coded as 1 if respondents received the large grid treatment and 0 if they received either the small grids or single-item-per-page treatments.

Any grid

A dichotomous variable was coded as 1 if respondents received either the large grid or small grid treatment and 0 if they received the single-item-per-page treatment.

Time

We captured how long it took the respondent to answer each of the three experimental question sets. These continuous time variables are coded in seconds. For Question 1 (four items), response times ranged from 13 to 321 seconds, and two outliers were recoded to 240 seconds (four minutes). For Question 2 (seven items), response times ranged from 2 to 693 seconds, and four outliers were recoded to 330 seconds (five-and-a-half minutes). For Question 3 (five items), response times ranged from 2 to 449 seconds, and one outlier was recoded to 180 seconds (three minutes; see Table 2).

Table 2.

Mean Response Time in Seconds.

	Question 1		Question 2		Question 3
	Nonmobile	Mobile	Nonmobile	Mobile	Nonmobile	Mobile
Single item per page	49.6s (58)	80.7s (9)	80.4s (133)	104.1s (21)	37.3s (133)	53.0s (20)
Small grids	47.9s (53)	42.8s (9)	59.7s (139)	66.6s (22)	30.1s (139)	29.4s (22)
Large grid	39.8s (65)	51.5s (11)	57.7s (147)	67.8s (26)	29.3s (147)	31.7s (24)
Any grid	43.5s (118)	47.6s (20)	58.7s (286)	67.3s (48)	28.7s (286)	30.6s (46)

Nondifferentiation

For each of the three experimental questions, we created a variable that identified respondents who chose the same response option (either yes or no) for every question item. The variables were coded as 1 if all response options were the same and 0 if all responses were not the same. Although nondifferentiation can be calculated as the absolute differences across all responses, the binary variable construction is often used when focusing on the clearest examples of straightlining (see Table 3).

Table 3.

Percentage of Nondifferentiation.

	Question 1		Question 2		Question 3
	Nonmobile	Mobile	Nonmobile	Mobile	Nonmobile	Mobile
Single item per page	4% (135)	10% (21)	33% (135)	19% (21)	10% (135)	0% (21)
Small grids	6% (140)	18% (22)	33% (140)	36% (22)	11% (140)	9% (22)
Large grid	5% (149)	11% (27)	44% (149)	44% (27)	10% (149)	26% (27)
Any grid	6% (289)	14% (49)	39% (289)	41% (49)	11% (289)	18% (49)

Control variables

Controls included age, sex, income, education, race, and contact mode where respondents were randomly assigned to one of two contact groups that differed on when they received an e-mail reminding them to complete the survey.

Analysis

First, for each of the three experimental questions, we used two sets of negative binomial regression models to examine differences in item response times between respondents who received the single-item-per-page treatment and those who received any type of grid. The first set of models features a dummy variable for mobile device, a dummy variable for grid (either large or small), and an interaction term for mobile device and grid. The second set of models includes a dummy variable for mobile device, a dummy variable for small grid, a dummy variable for large grid, an interaction term for mobile device and small grids, and an interaction term for mobile device and large grid. Both models control for age, education, income, gender, race, and contact mode.

Next, for each of the three experimental questions, we used two sets of logistic regression models to compare nondifferentiation between respondents who received one question per page versus those who received a grid. The first set of models features a dummy variable for mobile device, a dummy variable for any grid (either large or small), and an interaction term for mobile device and grid. The second set of models includes a dummy variable for mobile device, a dummy variable for small grids, a dummy variable for large grid, an interaction term for mobile device and small grids, and an interaction term for mobile device and large grid. Both models control for age, education, income, gender, race, and contact mode.

Results

Response Times

Our analysis revealed that both using a mobile device and receiving a grid are associated with changes in mean item response times when controlling for demographic factors (see rows 1 and 2 in Table 4). Completing the survey with a mobile device was associated with an increase in mean response time for all three questions: Question 1 (coefficient = 0.28, p < .05), Question 2 (coefficient = 0.15, p < .05), and Question 3 (coefficient = 0.14, p < .10). In contrast, receiving grid questions (whether large or small) is associated with a significant decrease in mean response times for Question 2 (coefficient = −0.33, p < .01) and Question 3 (coefficient = −0.31, p < .01). For Question 1, the grid variable had a negative coefficient as well (indicating a decrease in response times) but was not significant (coefficient = −0.14, n.s.). These results show that mobile device use can increase response times, whereas receiving grid questions can reduce response times.

Table 4.

Response Time Results (Grouping Together Both Grid Types).

	Question 1			Question 2			Question 3
	Base	Control	Interaction	Base	Control	Interaction	Base	Control	Interaction
Mobile	0.23* (0.11)	0.28* (0.13)	0.57** (0.22)	0.18** (0.07)	0.15* (0.07)	0.34* (0.13)	0.16* (0.06)	0.14^† (0.07)	0.38** (0.13)
Any grid	−0.19* (0.08)	−0.14 (0.09)	−0.08 (0.09)	−0.33** (0.05)	−0.33** (0.06)	−0.29** (0.06)	−0.30** (0.05)	−0.31** (0.05)	−0.26** (0.06)
Age		0.02** (0.003)	0.02** (0.003)		0.008** (0.002)	0.008** (0.002)		0.006** (0.002)	0.006** (0.002)
Education		−0.03 (0.02)	−0.03 (0.02)		−0.02 (0.01)	−0.02 (0.01)		−0.01 (0.01)	−0.01 (0.01)
Income		−0.03* (0.01)	−0.03* (0.01)		−0.004 (0.008)	−0.005 (0.008)		−0.006 (0.008)	−0.006 (0.008)
Female		0.08 (0.09)	0.07 (0.08)		−0.02 (0.05)	−0.02 (0.05)		−0.01 (0.05)	−0.02 (0.05)
White		−0.04 (0.10)	−0.03 (0.10)		−0.32** (0.06)	−0.31** (0.06)		−0.15* (0.06)	−0.15* (0.06)
Contact mode		0.19* (0.09)	0.19* (0.08)		0.13** (0.05)	0.14** (0.05)		−0.05 (0.05)	−0.04 (0.05)
Mobile × Any grid			−0.45^† (0.27)			−0.27^† (0.16)			−0.37* (0.15)
_cons	3.94** (0.07)	3.22** (0.27)	3.19** (0.27)	4.40** (0.04)	4.18** (0.16)	4.15** (0.16)	3.65** (0.04)	3.66** (0.16)	3.62** (0.16)
Pseudo R²	.006	.03	.03	.01	.02	.03	.01	.02	.02
n	205	163	163	488	378	378	485	378	374

†

p ≤ .1. *p ≤ .05. **p ≤ .01.

The analysis also illustrates significant interaction effects between grids and mobile devices (see row 9 of Table 4). The reduction in response times associated with grids is greater when a survey is completed on a mobile device for all three questions: Question 1 (coefficient = −0.45, p < .10), Question 2 (coefficient = −0.27, p < .10), and Question 3 (coefficient −0.37, p < .05). In addition, both grids and mobile devices have direct effects on mean response times when the interaction term is included in the models (see rows 1 and 2 in Table 4). The use of a mobile device was associated with increases in response times for Question 1 (coefficient = 0.57, p < .01), Question 2 (coefficient = 0.34, p < .05), and Question 3 (coefficient = 0.38, p < .01). The use of grids was associated with decreases in mean item response times for Question 2 (coefficient = −0.29, p < .01) and Question 3 (coefficient = −0.26, p < .01). For Question 1, the grid variable was negative (indicating a decrease in response times) but not significant (coefficient = −0.08, n.s.).

When subdividing the grid variable according to whether respondents received several small grids or one large grid, the results show that both small and large grids were associated with decreases in mean item response times (see rows 2 and 3 in Table 5). The small grid format was associated with a significant decrease in mean item response times for Question 2 (coefficient = −0.33, p < .01) and Question 3 (coefficient = −0.26, p < .01). For Question 1, the small grid variable was negative (indicating a decrease in mean response time) but not significant (coefficient = −0.07, n.s.). The large grid format was associated with significant decreases in mean response times for all three questions: Question 1 (coefficient = −0.19, p < .10), Question 2 (coefficient = −0.34, p < .01), and Question 3 (coefficient = −0.35, p < .01).

Table 5.

Response Time Results (When Differentiating between Small Grids and Large Grid).

	Question 1			Question 2			Question 3
	Base	Control	Interaction	Base	Control	Interaction	Base	Control	Interaction
Mobile	0.24* (0.11)	0.28* (0.13)	0.57** (0.22)	0.18** (0.07)	0.15* (0.07)	0.34* (0.13)	0.16* (0.06)	0.13^† (0.07)	0.37** (0.13)
Small grid	−0.12 (0.10)	−0.07 (0.11)	0.001 (0.11)	−0.32** (0.06)	−0.33** (0.06)	−0.29** (0.07)	−0.27** (0.05)	−0.26** (0.06)	−0.21** (0.07)
Large grid	−0.25** (0.09)	−0.19^† (0.10)	−0.15 (0.10)	−0.35** (0.06)	−0.34** (0.06)	−0.30** (0.07)	−0.34** (0.05)	−0.35** (0.06)	−0.30** (0.07)
Age		0.02** (0.003)	0.02** (0.003)		0.01** (0.002)	0.01** (0.002)		0.01** (0.002)	0.01** (0.002)
Education		−0.03 (0.02)	−0.03 (0.02)		−0.02 (0.01)	−0.02 (0.01)		−0.01 (0.01)	−0.01 (0.01)
Income		−0.03* (0.01)	−0.03* (0.01)		−0.005 (0.01)	−0.005 (0.01)		−0.01 (0.01)	−0.01 (0.01)
Female		0.09 (0.09)	0.09 (0.08)		−0.02 (0.05)	−0.03 (0.05)		−0.01 (0.05)	−0.03 (0.05)
White		−0.06 (0.11)	−0.04 (0.11)		−0.32** (0.06)	−0.32** (0.06)		−0.16* (0.06)	−0.16* (0.06)
Contact mode		0.19* (0.08)	0.20* (0.08)		0.13** (0.05)	0.14** (0.05)		−0.04 (0.05)	−0.03 (0.05)
Mobile × Small grid			−0.60^† (0.31)			−0.25 (0.18)			−0.30^† (0.17)
Mobile × Large grid			−0.31 (0.31)			−0.30 (0.18)			−0.46** (0.18)
_cons	3.94** (0.07)	3.21** (0.27)	3.17** (0.27)	4.40** (0.04)	4.19** (0.16)	4.16** (0.17)	3.65** (0.04)	3.67** (0.16)	3.64** (0.16)
Pseudo R²	.01	.03	.03	.01	.02	.03	.01	.02	.02
n	205	163	163	488	378	378	485	378	378

†

p ≤ .1. *p ≤ .05. **p ≤ .01.

Our findings also reveal interaction effects between both small and large grids and mobile devices such that decreases in response times associated with grids were greater on mobile devices (see rows 10 and 11 in Table 5). The interaction term for small grids and mobile device use was negative and significant for Question 1 (coefficient = −0.60, p < .10) and Question 3 (coefficient = −0.30, p < .10), whereas the interaction term was negative but not significant for Question 2 (coefficient = −0.25, n.s.). The interaction term for the large grid and mobile device use was negative and significant for Question 3 (coefficient = −0.46, p < .01), and was negative but not significant for Question 1 (coefficient = −0.31, n.s.) and Question 2 (coefficient = −0.30, n.s.). These results show that the reductions of response times associated with both large and small grids was greater when the survey was completed on a mobile device than when completed on a computer.

Nondifferentiation

Our analysis provides evidence that grids are associated with increases in nondifferentiation (see row 2 of Table 6). Grid questions were associated with significantly more nondifferentiation among respondents for Question 2 (coefficient = 0.40, p < .05). The grid coefficient was also positive (indicating more nondifferentiation) but not significant for Question 1 (coefficient = 0.94, n.s.) and Question 3 (coefficient = 0.67, n.s.).

Table 6.

Nondifferentiation Results (Grouping Together Both Grid Types).

	Question 1			Question 2			Question 3
	Base	Control	Interaction	Base	Control	Interaction	Base	Control	Interaction
Mobile	1.04* (0.42)	0.61 (0.55)	0.49 (1.21)	−0.12 (0.27)	−0.12 (0.33)	−1.31 (0.82)	0.21 (0.39)	0.70 (0.49)	−14.32 (934.1)
Grid	0.43 (0.45)	0.94 (0.58)	0.91 (0.66)	0.37^† (0.21)	0.40* (0.25)	0.33 (0.27)	0.31 (0.33)	0.67 (0.43)	0.30 (0.45)
Age		−0.02 (0.02)	−0.02 (0.02)		−0.04** (0.009)	−0.04** (0.009)		0.02 (0.01)	0.02 (0.01)
Education		−0.08 (0.13)	−0.08 (0.13)		−0.18** (0.06)	−0.17** (0.06)		−0.22* (0.10)	−0.22* (0.10)
Income		0.06 (0.07)	0.06 (0.07)		−0.02 (0.04)	−0.02 (0.04)		−0.20** (0.06)	−0.22** (0.06)
Female		0.54 (0.47)	0.54 (0.47)		−0.50* (0.24)	−0.48* (0.24)		−1.17** (0.41)	−1.17** (0.41)
White		0.11 (0.55)	0.11 (0.55)		−0.53* (0.27)	−0.54* (0.27)		−0.54 (0.40)	−0.69^† (0.41)
Contact mode		−0.98^† (0.50)	−0.99* (0.50)		−0.06 (0.23)	−0.10 (0.23)		−0.48 (0.38)	−0.57 (0.38)
Mobile × Grid			0.14 (1.35)			1.55^† (0.91)			15.57 (934.1)
_cons	−3.27** (0.41)	−1.72 (1.44)	−1.59 (1.47)	−0.80** (0.18)	2.95** (0.75)	3.14** (0.76)	−2.35** (0.29)	0.74 (1.06)	1.23 (1.09)
Pseudo R²	.03	.08	.07	.005	.09	.09	.004	.18	.21
n	494	378	378	494	378	378	494	378	378

†

p ≤ .1. *p ≤ .05. **p ≤ .01.

Our results do not point to any relationship between the use of mobile devices and nondifferentiation (see row 1 of Table 5). Mobile device use did not have a significant effect on nondifferentiation for any of the three questions, nor was the direction of the relationship consistent across the three questions.

Our analysis provides evidence of an interaction effect between mobile devices and grids such that the increase in nondifferentiation associated with grids was greater on a mobile device. The interaction effect between grids and mobile devices for nondifferentiation was positive and significant for Question 2 (coefficient = 1.55, p < .10), and positive but not significant for Question 1 (coefficient = 0.14, n.s.) and Question 3 (coefficient = 15.57, n.s.). For Question 3, this insignificance was due to its lack of variance in nondifferentiation (there were no cases of nondifferentiation among mobile users who received the single-item-per-page format). However, the frequencies do reveal that for Question 3, there was a substantially larger increase in the percent of nondifferentiation cases with a grid format among mobile users (see Table 3). Among mobile users, there was an 18-point percentage increase in nondifferentiation when receiving grid questions compared with just a 1-point percentage increase among nonmobile users.

The results provide some evidence of significant nondifferentiation effects for both small grids and large grids (see rows 2 and 3 in Table 7). For the small grids, the nondifferentiation coefficient was positive and significant for Question 1 (coefficient = 1.10, p < .10), and positive but not significant for Question 2 (coefficient = 0.14, n.s.) and Question 3 (coefficient = 0.60, n.s.). Large grids were associated with a significant increase in nondifferentiation for Question 2 (coefficient = 0.80, p < .01), whereas the coefficient was positive but insignificant for Question 1 (coefficient = 0.77, n.s.) and Question 3 (coefficient = 0.73, n.s.).

Table 7.

Nondifferentiation Results (When Differentiating between Small Grids and Large Grid).

	Question 1			Question 2			Question 3
	Base	Control	Interaction	Base	Control	Interaction	Base	Control	Interaction
Mobile	1.05* (0.42)	0.58 (0.55)	0.50 (1.21)	−0.13 (0.27)	−0.09 (0.34)	−1.31 (0.82)	0.21 (0.39)	0.70 (0.49)	−14.80 (1,212.9)
Small grid	0.63 (0.49)	1.10^† (0.62)	0.88 (0.72)	0.12 (0.24)	0.14 (0.30)	−0.05 (0.32)	0.24 (0.38)	0.60 (0.48)	0.32 (0.50)
Large grid	0.22 (0.51)	0.77 (0.65)	0.81 (0.72)	0.59* (0.23)	0.80** (0.28)	0.64* (0.30)	0.37 (0.36)	0.73 (0.48)	0.28 (0.51)
Age		−0.02 (0.02)	−0.02 (0.02)		−0.04** (0.01)	−0.04** (0.01)		0.02 (0.01)	0.02 (0.01)
Education		−0.08 (0.13)	−0.08 (0.13)		−0.18** (0.07)	−0.18** (0.07)		−0.22* (0.10)	−0.22* (0.10)
Income		0.06 (0.07)	0.06 (0.07)		−0.02 (0.04)	−0.02 (0.04)		−0.20** (0.06)	−0.21** (0.06)
Female		0.53 (0.47)	0.50 (0.48)		−0.50* (0.24)	−0.48* (0.24)		−1.17** (0.41)	−1.14** (0.41)
White		0.07 (0.56)	0.02 (0.56)		−0.49^† (0.27)	−0.50^† (0.28)		−0.53 (0.40)	−0.62 (0.42)
Contact mode		−0.97^† (0.50)	−0.99^† (0.51)		−0.09 (0.23)	−0.12 (0.23)		−0.49 (0.38)	−0.57 (0.38)
Mobile × Small grid			−0.41 (1.65)			1.54 (1.00)			16.34 (1,212.9)
Mobile × Large grid			0.40 (1.43)			1.65^† (1.00)			15.62 (1,212.9)
_cons	−3.26** (0.41)	−1.59 (1.43)	−1.48 (1.47)	−0.79** (0.18)	2.83** (0.76)	3.02** (0.77)	−2.35** (0.29)	0.79 (1.07)	1.18 (1.10)
Pseudo R²	.03	.07	.07	.01	.10	.10	.004	.18	.21
n	494	378	378	494	378	378	494	378	378

†

p ≤ .1. *p ≤ .05. **p ≤ .01.

The interaction effects between mobile device use and grids were more consistently significant for large grids than for small grids (see rows 10 and 11 in Table 7). The interaction effect for nondifferentiation in large grid use was positive and significant for Question 2 (coefficient = 1.65, p < .10), and the interaction term was positive but insignificant for Question 1 (coefficient = 0.40, n.s.) and Question 3 (coefficient = 15.62, n.s.). As with the previous model, the interaction coefficients for both large and small grids for Question 3 were insignificant due to the large standard error resulting from the lack of variance in nondifferentiation among those who received the single-item-per-page treatment. The interaction term between mobile device use and small grids was not significant for any of the questions, nor was the direction of the coefficient consistent: Question 1 indicated a negative relationship (coefficient = −0.41, n.s.), whereas Question 2 (coefficient = 1.54, n.s.) and Question 3 (coefficient = 16.34, n.s.) indicated a positive relationship.

Conclusion and Discussion

Our study provides strong evidence of an important new insight into the effects of question grids on responses to Web surveys. Consistent with past studies, we find question grids to be associated with reductions in mean item response times and increases in nondifferentiation when compared with a single-question-per-page design. We expand upon these findings by additionally demonstrating that the effects of question grids on responses are greater when a survey is completed on a mobile device than when completed on a computer. First, the reduction in mean item response time associated with question grids is larger for surveys completed on cell phones or tablets than for surveys completed on computers. Second, the increase in nondifferentiation associated with question grids is greater on mobile devices than on computers. Together, these results demonstrate that the effects of question grids on responses can vary depending on the device a respondent uses to complete the Web survey.

Our findings offer several practical applications for Web survey design. First, question grids may lead to more measurement error (e.g., nondifferentiation) for surveys completed on mobile devices. These differences could create problems for analyzing and comparing the results of a Web survey that has been completed on an array of devices. As more respondents opt to complete Web surveys on mobile devices, researchers should be proportionally more cautious of using question grids.

Second, our study indicates that regardless of grid size, grids can affect survey responses and these effects are greater when presented on mobile devices than when presented on computers. Consequently, researchers and survey designers should be aware that presenting relatively few questions in a grid (e.g., two three-question mini grids instead of one larger six-question grid) may neither eliminate the effects of grids on surveys nor mitigate the interaction effects between grids and mobile devices. We found interaction effects even though our survey featured grids with relatively few questions (seven at most). Therefore, the interaction effects may be even greater for grids featuring more questions as the smaller screens and limited scrolling capabilities of mobile devices present challenges for respondents.

Last, our research provides some evidence that the interaction effects of grids and mobile devices may be greater if the survey is longer and/or grids are used near the end of the survey. We found more significant reductions in item nonresponse and more significant increases in nondifferentiation for the third experimental question than for the first and second experimental questions that were presented earlier in the survey. These results indicate that interaction effects may be partially due to mobile respondents satisficing as they approach the end of the survey. Our survey was relatively short (the mean completion time was below 25 minutes), so grid effects would likely be more significant on longer surveys for which respondents would be more prone to satisfice (Krosnick 1991).

Combined, these findings suggest that researchers should try to avoid using grids on surveys that could be completed on mobile devices. Although grids can reduce the burden on survey respondents, these benefits are likely outweighed by the increased measurement error associated with grids on surveys completed via mobile devices. The optimal presentation of questions depends on a survey’s specific goals, design, and sample population, but researchers should recognize the likely negative effects of grids on response quality for surveys completed with phones or tablets.

The interaction effects between question grids and mobile device use may also vary based on individual characteristics of respondents. We lacked the statistical power to explore how demographic factors could affect the interaction effects of grids and mobile devices, and further research should test whether the differences in effects of question grids on mobile respondents varies depending on demographic characteristics. Future research with a larger sample is necessary to disentangle the grid effects among different devices from the device user characteristics. Our findings will shed light for researchers exploring the impact of visual complexity of matrices and use of question grouping and grids in Web surveys, as well as the comparability of different visual groupings among multiple devices with different screen sizes while controlling for the characteristics of the users with a larger sample size. A larger sample size will provide the statistical power to explore how demographic factors could affect the interaction effects of grids and devices. Future analyses with unfolded grids using mobile apps are needed to explore whether or not further optimization will diminish the impact of grids on data quality measures in Web surveys completed via mobile devices.

Our research highlights the importance of designing Web surveys that are compatible for both computers and mobile devices. Methodologists have long been cautious of mode effects on surveys and have been cognizant of the need to consider how different forms of survey administration can affect responses. However, our research on question grids draws attention to an emerging issue in survey design and administration: device effects. Web surveys are becoming increasingly popular, and an ever-growing number of respondents are choosing to complete these surveys on cell phones or tablets. Consequently, it is crucial that researchers be aware of possible device effects and work to develop designs that mitigate such effects.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Anderson

Monica

. 2015. “Technology Device Ownership: 2015.” Pew Research Center. Retrieved May 30, 2014 (http://www.pewinternet.org/2015/10/29/technology-device-ownership-2015).

Bilgen

Ipek

Stern

Michael

Sterrett

David

. 2014. “Question Grouping and Matrices in Web Surveys: Using Response and Auxiliary Data to Examine Question Grouping Decisions.” Paper presented at the 69th Annual Conference of the American Association for Public Opinion Research, May 15–18, Hollywood, FL.

Bosnjak

Michael

Metzger

Gottfried

Gräf

Lorenz

. 2010. “Understanding the Willingness to Participate in Mobile Surveys: Exploring the Role of Utilitarian, Affective, Hedonic, Social, Self-expressive, and Trust-related Factors.” Social Science Computer Review 28(3): 350–70.

Bosnjak

Michael

Poggio

Teresio

Funke

Frederik

. 2013. “Online Survey Participation via Mobile Devices: Findings from Seven Access Panel Studies.” Paper presented at the 68th Annual Conference of the American Association for Public Opinion Research, May 16–19, Boston, MA.

Buskirk

Trent

. 2013. “Smarter Smartphone Surveys 201: Data Collection Methods and Survey Design Considerations.” Webinar presented for the American Association of Public Opinion Research.

Buskirk

Trent

Andrus

Charles

. 2012. “Online Surveys Aren’t Just for Computers Anymore! Exploring Potential Mode Effects between Smartphone vs. Computer-based Online Surveys.” Paper presented at the 67th Annual Conference of the American Association for Public Opinion Research, May 17–20, Orlando, FL.

Callegaro

Mario

. 2013. “Do You Know Which Device Your Respondent Has Used to Take Your Online Survey?” Survey Practice 3. Retrieved May 30, 2014 (http://surveypractice.org).

Casey

Tristan

Poropat

Arthur

. 2014. “Beauty Is More than Screen Deep: Improving the Web Survey Respondent Experience through Socially-present and Aesthetically-pleasing User Interfaces.” Computers in Human Behavior 30(2):153–63.

Christian

Leah

Dillman

Don

. 2004. “The Influence of Symbolic and Graphical Language Manipulations on Answers to Paper Self-administered Questionnaires.” Public Opinion Quarterly 68(1):57–80.

10.

Comer

Patrick.

Saunders

Ted

. 2012 “Technical Impact of Mobile Devices.” CASRO Technology Conference, May 30–31, New York.

11.

Conrad

Frederick

Schober

Michael

Zhang

Chan

Yan

Huiying

Vickers

Lucas

Johnston

Michael

Hupp

Andrew

Hemingway

Lloyd

Fail

Stefanie

. 2013 “Mode Choice on an iPhone Increases Survey Data Quality.” Paper presented at the 67th Annual Conference of the American Association for Public Opinion Research, May 16–19, Boston, MA.

12.

Couper

Mick

. 2008. Designing Effective Web Surveys. Cambridge, England: Cambridge University Press.

13.

Couper

Mick

Tourangeau

Roger

Conrad

Frederick

Zhang

Chan

. 2013. “The Design of Grids in Web Surveys.” Social Science Computer Review 31(4):322–45.

14.

Couper

Mick

Trougott

Michael

Lamias

Mark

. 2001. “Web Survey Design and Administration.” Public Opinion Quarterly 65(2):230–53.

15.

de Bruijne

Marika

Wijnant

Arnaud

. 2013. “Comparing Survey Results Obtained via Mobile Devices and Computers: An Experiment with a Mobile Web Survey on a Heterogeneous Group of Mobile Devices versus a Computer-assisted Web Survey.” Social Science Computer Review 31(3):244–51.

16.

Dillman

Don

Smyth

Jolene

Christian

Leah

. 2009. Internet, Mail, and Mixed-mode Surveys: The Tailored Design Method. Hoboken, NJ: Wiley.

17.

Dominguez

Juan

de Rada

Vidal Diaz

. 2015. “The Quality of Responses to Grid Questions as Used in Web Questionnaires (Compared with Paper Questionnaires).” International Journal of Social Research Methodology 18(4):337–48.

18.

Galesic

Mirta

Tourangeau

Roger

Couper

Mick

Conrad

Frederick

. 2007. “Using Change to Improve Navigation in Grid Questions.” Paper presented at the General Online Research Conference (GOR’07), March 26–28, Leipzig, Germany.

19.

Gräf

Lorenz

. 2002. “Assessing Internet Questionnaires: The Online Pretest Lab.” Pp. 69–79 in Online Social Sciences, edited by Battinic

Reips

U. D.

Bosnjak

Werner

Seattle, WA: Hogrefe & Huber.

20.

Jeavons

Andrew

. 1998. “Ethology and the Web: Observing Respondent Behavior in Web Surveys.” Proceedings of the Worldwide Internet Conference, ESOMAR, January 28–30, London, England.

21.

Jenkins

Cleo

Dillman

Don

. 1997. “Towards a Theory of Self-administered Questionnaire Design.” Pp. 165-196 in Survey Measurement and Process Quality, edited by Lyberg

Biemer

Collins

Decker

DeLeeuw

Dippo

Schwarz

Trewin

New York: Wiley-Interscience.

22.

Kaczmirek

Lars

. 2009. Human-survey Interaction: Usability and Nonresponse in Online Surveys. Cologne, Germany: Herbert von Halem Verlag.

23.

Krosnick

Jon

. 1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology 5(2):213–36.

24.

Macer

Tim

. 2012. “Developments and the Impact of Smart Technology.” International Journal of Market Research 54(4):567–70.

25.

Manfreda

Katja

Batagelj

Zenel

Vehovar

Vasja

. 2002. “Design of Web Survey Questionnaires: Three Basic Experiments.” Journal of Computer-Mediated Communication 7(3). Retrieved February 23, 2014 (http://onlinelibrary.wiley.com/doi/10.1111/jcmc.2002.7.issue-3/issuetoc).

26.

Mavletova

Aigul

. 2013. “Data Quality in PC and Mobile Web Surveys.” Social Science Computer Review 31(50):725–43.

27.

McCutcheon

Allan

Wang

Mengyang

. 2015. “Grids and Online Panels: A Comparison of Device Type from a Survey Quality Perspective.” Paper presented at the 70th Annual American Association for Public Opinion Research Conference, May 12–15, Austin, TX.

28.

McGeeney

Kyley

Marlar

Jenny

. 2013. “Mobile Browser Web Surveys: Testing Response Rates, Data Quality and Best Practices.” Paper presented at the 67th Annual Conference of the American Association for Public Opinion Research, May 16–19, Boston, MA.

29.

Miller

Carolyn

Purcell

Kristen.

Rosenstiel

Tom

. 2012. 72% of Americans follow local news closely. Pew Research Center. Retrieved April 20, 2015 (http://pewinternet.org/Reports/2012/Local-news-enthusiasts.aspx).

30.

Peytchev

Andy

Hill

Craig

. 2010. “Experiments in Mobile Web Survey Design Similarities to Other Modes and Unique Considerations.” Social Science Computer Review 28(4):319–35.

31.

Poynter

Ray

. 2001. “A Guide to Best Practice in Online Quantitative Research.” Pp. 3–19 in The Challenge of the Internet: Proceedings of the ASC International Conference on Survey Research Methods, edited by Westlake

Sykes

Manners

Rigg

London, England: Association for Survey Computing.

32.

Redline

Cleo

Dillman

Don

. 2002. “The Influence of Alternative Visual Designs on Respondents’ Performances with Branching Instructions in self-administered Questionnaires.” Pp. 179–93 in Survey Nonresponse, edited by Groves

Dillman

Eltinge

Little

New York: John Wiley.

33.

Return Path. 2013. “Email Mostly Mobile.” Retrieved July 20, 2014 (http://www.returnpath.com/wp-content/uploads/resource/email-mostly-mobile/Return-Path-Email-Mostly-Mobile1.jpg).

34.

Schwarz

Norbert

. 1996. Cognition and Communication: Judgmental Biases, Research Methods, and the Logic of Conversation. Mahwah, NJ: Lawrence Erlbaum.

35.

Smith

Aaron

. 2013. “Smartphone Ownership.” Pew Research Center’s Internet and American Life Project. Retrieved July 20, 2014 (http://www.pewinternet.org/2013/06/05/smartphone-ownership-2013/)

36.

Tourangeau

Roger

Conrad

Frederick

Couper

Mick

. 2013. The Science of Web Surveys. Oxford, England: Oxford University Press.

37.

Tourangeau

Roger

Couper

Mick

Conrad

Frederick

. 2004. “Spacing, Position, and Order: Interpretive Heuristics for Visual Features of Survey Questions.” Public Opinion Quarterly 68(3):368–93.

38.

Wells

Tom

Bailey

Justin

Link

Michael

. 2013. “Comparison of Smartphone and Online Computer Survey Administration.” Social Science Computer Review 32 (2): 238–55.

39.

Wojtowicz

Tamara

. 2001. “Designing Lengthy Internet Questionnaires: Suggestions and Solutions.” Pp. 25–32 in The Challenge of the Internet: Proceedings of the ASC International Conference on Survey Research Methods, edited by Westlake

Sykes

Manners

Rigg

London, England: Association for Survey Computing.

40.

Yelton

Andromeda

. 2012. “Chapter 1: Who Are Smartphone Users?” Library Technology Reports 48:5–8.

41.

Zachariev

Manuel

Ferneyhough

Chris

Ryan

Chris

. 2009. “Best Practices in Mobile Research.” Paper presented at the ESOMAR Online Research, October 26, Chicago, IL.