Grid and Item-by-Item Formats in PC and Mobile Web Surveys

Abstract

While grids or matrix questions are a widely used format in PC web surveys, there is no agreement on the format in mobile web surveys. We conducted a two-wave experiment in an opt in panel in Russia, varying the question format (grid format and item-by-item format) and device respondents used for survey completion (smartphone and PC). The 1,678 respondents completed the survey in the assigned conditions in the first wave and 1,079 in the second wave. Overall, we found somewhat higher measurement error in the grid format in both mobile and PC web conditions. We found almost no significant effect of the question format on test–retest correlations between the latent scores in two waves and no differences in breakoff rates between the question formats. The multigroup comparison showed some measurement equivalence between the question formats. However, the difference varied depending on the length of a scale with a longer scale producing some differences in the measurement equivalence between the conditions. The levels of straightlining were higher in the grid than in the item-by-item format. In addition, concurrent validity was lower in the grid format in both PC and mobile web conditions. Finally, subjective indicators of respondent burden showed that the grid format increased reported technical difficulties and decreased subjective evaluation of the survey.

Keywords

grid questions matrix questions item-by-item format web surveys mobile web surveys measurement equivalence measurement error concurrent validity reliability

While grids or matrix questions are a widely used format in PC web surveys (completed on desktop or laptop computers), there is little agreement on the optimal format for mobile web surveys (completed on smartphones). Several practitioners argue against the use of grids on smartphones, and a number of different alternatives have been proposed including item-by-item approaches. However, there is still relatively little empirical research on the performance of grid questions or their alternatives on smartphones, and the findings are generally inconsistent. Given this, we compare data equivalence between two formats (grid and item-by-item questions) and between devices (smartphones and PCs), using a two-wave crossover design. We measure the effect of the question format and device on data quality and whether the effect of the question format varied among PC web and mobile web or smartphone respondents (we use the terms “smartphone” and “mobile web” interchangeably to refer to surveys administered via a browser on a smartphone).

Background and Literature Review

Grid, table, or matrix questions (in which a set of items sharing the same response options are presented together) have been the subject of much attention since the early days of web surveys. Despite researchers and practitioners arguing for avoiding grids altogether or minimizing their use, both in the early days of web surveys (e.g., Gräf, 2002; Poynter, 2001; Wojtowicz, 2001), and more recently (e.g., Callegaro, Lozar Manfreda, & Vehovar, 2015, p. 200; de Leeuw, 2017; Dillman, Smyth, & Christian, 2014, pp. 214–218; Kuper & Glowa, 2014), grids remain a popular web survey design tool (see Revilla, 2017). More recently, the increased use of mobile devices (especially smartphones) to complete web surveys has again raised questions about the use of grid questions (see Couper, Antoun, & Mavletova, 2017). Several new alternatives to grids have been proposed, including concertina or accordion formats (Barlas, Thomas, & Buttermore, 2017; Phillips & Luks, 2016), dynamic grids (Hanson, 2017), menu matrixes (Liu, 2017), and horizontal scrolling matrixes (de Leeuw, Hox, Klausch, Roberts, & de Jongh, 2012; Klausch, de Leeuw, Hox, Roberts, & de Jongh, 2012), again suggesting that grid questions may not be suitable for smartphones. Several studies have also found higher breakoff rates among smartphone users compared to PC users, and grids are a design feature implicated in this effect (see the meta-analysis by Mavletova & Couper, 2015).

Mobile optimization is one of the design features that complicates comparisons of different question formats on smartphones. Mobile optimization (or “mobile friendly” or “mobile first” designs) is implemented in different ways in different software products, ranging from using a liquid design that adjusts to the size of the browser window and replacing standard HTML radio buttons with larger buttons that are easier to select using touch, to systems that automatically convert grids items into some alternative (item-by-item scrolling or accordion formats being two examples). Further, many surveys do not (yet) optimize for mobile devices. In a study cited by York (2017), 29% of surveys submitted to the Research Now survey platform in 2016 were considered “mobile incompatible,” while a further 23% were deemed “mobile possible,” leaving 33% classified as mobile friendly and only 15% as “mobile optimized.” The issue of whether and how to optimize grid or matrix questions for smartphones remains an open one, needing further research.

While arguments both for and against grids have been made, the research evidence on the effect of grids remains mixed. Part of this is related to the many different alternatives to grids, including item-by-item scrolling (with separate items on the same web page) and item-by-item paging (with each item on a separate page), along with the newer alternatives mentioned above. Another reason is that grid questions themselves vary greatly, both in their design and implementation and in their size, including the number of items (rows), the length of the items, the number of response options (columns), whether they are labeled or not, and (if so) the length of the labels. Further, much of the research has focused more on issues of process quality (i.e., breakoffs, completion time, missing data, and straightlining) and less on outcome quality (i.e., the measurement properties of the items).

We briefly summarize some of the findings from the early research on grids before describing our study in greater detail. Extensive reviews of the literature on grids, especially on smartphones, are found elsewhere (e.g., Couper et al., 2017; Revilla & Couper, 2017). As noted above, the early research on grids on PCs yielded mixed results (see Couper et al., 2013). For example, some studies found somewhat higher Cronbach’s α coefficients in the grid format (Couper, Traugott, & Lamias, 2001; Tourangeau, Couper, & Conrad, 2004), while others found no significant differences in inter-item correlations between the formats (Callegaro, Shand-Lubbers, & Dennis, 2009; Toepoel, Das, & Van Soest, 2009; Yan, 2005). Similarly, some studies found higher levels of straightlining and item nonresponse in grids (Callegaro et al., 2009; Toepoel et al., 2009; Tourangeau et al., 2004), but Couper, Traugott, and Lamias (2001) found lower rates of nonsubstantive responses such as “don’t know” or “would rather not answer” in the grid format than a single item per page format.

Several studies have explored grids versus alternative question formats on smartphones. Some of the early studies randomized participants to format (e.g., grid vs. item-by-item) but not to device. This limits the inference if self-selected smartphone users are different from PC users (e.g., they have more experience with the device and greater tolerance of suboptimal survey design). For example, Lattery, Park Bartolone, and Saunders (2013) found no difference in breakoff rates or response distributions between a long 9-item grid format and a shorter 3-item grid format among smartphone users. McClain and Crawford (2013) found shorter median completion times, fewer breakoffs, and lower levels of straightlining in an item-by-item format relative to a grid, but no differences in item nonresponse rates. Peterson, Griffin, LaFrance, and Li (2017) found no significant differences in response distributions or straightlining between a traditional grid and an item-by-item version among self-selected smartphone users, but they did find higher breakoff rates for the grid version.

Liu & Cernat (2016) varied question format (grid vs. item-by-item scrolling) and number of response options (2–11 options) in a 2 × 7 factorial design, with about a third of the sample choosing to complete the survey on mobile devices (smartphones or tablets). They found higher missing data rates for the grid format, reaching significance (p < .05) for three of the seven question sets. But they found no differences in completion times or straightlining between the two formats on mobile devices. Using confirmatory factor analysis (CFA), they found measurement equivalence between the two formats in questions with seven responses or fewer. Questions with nine or more response options showed significant differences in measurement equivalence between formats, with lower reliability for the grid format.

Few experiments have varied both device (PC vs. smartphone) and format (grid vs. item-by-item). Thomas, Barlas, Graham, and Subias (2015) compared grids to several alternatives for three sets of questions, with respondents randomly assigned to device and question format. The item-by-item scrolling version (with horizontal scales) took slightly longer on smartphones than the grid version, but they found no differences in average item means. They concluded that “traditional grids (with 3–4 response categories) performed well across devices.”

Revilla, Toninelli, and Ochoa (2017) randomly assigned respondent to a PC, a nonoptimized smartphone version (both of which used grids) or an optimized smartphone version (with the questions in item-by-item scrolling format) in a two-wave crossover design. They examined two grids, each with 14 items, but one with 5 response options and the other with 11. They found similar inter-item correlations between the two smartphone versions (optimized and nonoptimized). However, they found evidence of higher variances in the optimized (item-by-item) version than the nonoptimized (grid) version, suggesting lower nondifferentiation with the item-by-item version.

In a follow-up study, Revilla and Couper (2017) varied the assigned device (PC vs. smartphone) and format (grid, item-by-item horizontal, and item-by-item vertical) in a single-wave experiment. They found higher rates of missing and nonsubstantive answers, and more nondifferentiation in the grid format, but no differences in completion time. They found few differences in the horizontal versus vertical versions of the item-by-item format. They concluded “…we find no compelling evidence that grids are necessarily bad on smartphones, nor do we find that the alternatives to grids that we tested are consistently superior to grids” (Revilla & Couper, 2017, p. 17).

In summary, the results of studies on alternative formats of grid questions on different devices are mixed (see Couper, Antoun, & Mavletova, 2017, for a review). In part, we attribute this to the wide variety of different design decisions that can be made, one of which relates to the size of the grids (number of items and number of response options). Against this backdrop, we conducted an experiment to explore differences in process and outcome quality between grid and item-by-item scrolling formats on both smartphones and PCs. Respondents were randomized to both format and device using a two-wave crossover design. Further, we examine a range of questions sets, with the number of items ranging from 4 to 11 and the number of response options ranging from 4 to 7. We do this on a survey that is partially optimized for mobile devices.

Our main research hypothesis is that the item-by-item format is associated with higher data quality and stronger measurement equivalence than the grid format across devices (both smartphones and PCs); however, we expect this effect to be stronger among smartphone respondents. We further expect that larger grids (more items and more response options) perform more poorly on smartphones relative to the item-by-item approach. We examine several different outcomes, including indicators of process quality (breakoffs, completion times, and straightlining) that are used in many other studies. However, we focus our attention on measurement equivalence, using methods similar to those used by Liu & Cernat (2016). This addresses the important question of whether the different formats would yield equivalent substantive conclusions across devices. Finally, we also look at subjective indicators of the surveys experience (again, as others have done).

Experimental Design

We conducted a two-wave experiment, in which we varied the question format (grid vs. item-by-item) and device used for survey completion. The item-by-item format with a scrolling design presented the items on the same screen. In the first wave, respondents were randomly assigned to one of the following four conditions: grids on a smartphone, grids on a PC, item-by-item on a smartphone, or item-by-item on a PC. In the second wave, respondents were assigned to a different device to complete the survey (see Table 1). In four conditions, respondents were assigned to the same question format (groups 1, 2, 3, and 4), while in two other conditions, respondents were assigned to the item-by-item format on a smartphone and to the grid format on a PC. We did not include a group with item-by-item questions on a PC and grids on a smartphone, as this is typically not considered in practice.

Table 1.
Experimental Design (N, Excluding Tablets).

Group First Wave N Second Wave N

1 Grids on smartphone 258 Grids on PC 165

2 Grids on PC 285 Grids on smartphone 175

3 Item-by-item on smartphone 278 Item-by-item on PC 193

4 Item-by-item on PC 292 Item-by-item on smartphone 176

5 Item-by-item on smartphone 276 Grids on PC 182

6 Grids on PC 289 Item-by-item on smartphone 188

The item-by-item format was fully optimized for smartphones, but the grid format was not fully optimized for smartphones. While a liquid or fluid design was employed to display the entire grid on the screen, pinching and zooming was required to answer the questions in the grid format on smartphones. Font sizes were not optimized for smartphones in the grid format. The item-by-item format was fully optimized for smartphones: larger buttons and larger font sizes were used, and no horizontal scrolling was required. The screenshots for all conditions are in Online Appendix A. While some have tested horizontal item-by-item formats and others have tested vertical formats (see, e.g., Revilla & Couper, 2017), the format we used is a hybrid approach.

This two-wave crossover design (see Antoun, Couper, & Conrad, 2017; Mavletova & Couper, 2013; Revilla, Toninelli, & Ochoa, 2017) allows us to explore the effect of devices and formats within respondents, allowing us to measure between-wave reliability within and across formats and devices. This design maximizes the internal validity of the results, at some loss to external validity.

Data Collection

The experiment was conducted in Russia using a volunteer online access panel run by Online Market Intelligence (http://www.omirussia.ru/en). All respondents reported using the Internet on their mobile phones in the last 30 days. The first wave was conducted from December 7 to 17, 2016. The second wave was conducted from January 9 to February 3, 2017. All respondents were invited by e-mail. Panelists received the standard panel incentive for participating.

In the PC web condition, we encouraged respondents to use a PC, while in the smartphone condition, we encouraged them to use a smartphone to complete the survey. By capturing user agent string information, we identified the browser and device used by respondents (PC or smartphone). If respondents used another device, they were asked to switch to the requested device and were not allowed to continue otherwise. In the first wave, 57.8% of those who started the survey were redirected because they were using the wrong device; this rate was higher in the smartphone than in the PC web survey condition: 63.7% in the smartphone condition and 51.2% in the PC web condition, χ²(1) = 88.32, p < .001. In the second wave, 15.3% were redirected due to incorrect device use: 23.2% in the smartphone condition and 12.5% in the PC condition, χ²(1) = 31.65, p < .001. Some of the respondents completed the survey via tablets or some other device (n = 82 in the first wave and n = 56 in the second wave) and were also excluded from the analysis.

In the first wave, 1,678 respondents completed the survey in the assigned conditions: 866 in the PC web condition and 812 in the mobile web (smartphone) condition (see Table 2). The completion rate was higher in the PC web condition than in the mobile web condition: 7.1% and 2.3%, respectively, χ²(1) = 603.58, p < .001. The overall completion rate was 3.6%.

Table 2.
Number of Completed Surveys.

Condition Format Device Format × Device

Grids Item-by-Item PC Mobile PC Grids PC Item- by-Item Mobile Grids Mobile Item-by-Item

Wave 1 832 846 866 812 574 292 258 554

Wave 2 522 557 540 539 347 193 175 364

In the second wave, 1,079 respondents completed the survey in the assigned conditions: 540 in the PC web condition and 539 in the mobile web condition (see Table 2). The completion rate was 64.3% with no significant differences between the conditions, χ²(1) = 3.32, p = .07, and between the formats, χ²(1) = 0.51, p = .48. Logistic regression showed that those who completed the survey in the PC web condition in the first wave were expected to complete the second wave in the mobile web condition (odds ratio [OR] = 0.80, p < .05), and females (OR = 0.76, p < .01) had a lower nonresponse rate. No effects of age or question format were found.

Questionnaire

There were a total of 111 items in the questionnaire. The main focus of the survey was trust. The focus of our analysis is on seven sets of questions with 49 items overall (see Table 3 for a short summary and Online Appendix B for question wording). All sets used fully labeled scales, except for Set 4 which used an end point–labeled scale. Most of the questions are from the World Values Survey (see http://www.worldvaluessurvey.org). We conducted 20 cognitive interviews and ran a pretest among 50 respondents on both types of devices before launching the survey. Consistent with other surveys administered to the panel, respondents were required to answer all questions.

Table 3.
Description of Question Sets.

Question sets Number of Items Latent Factors Response Scale

Set 1: Risk willingness 4 items One factor: Risk willingness 4-point scale

Set 2: Trust (Uslaner, 2002; Yamagishi & Yamagishi, 1994) 6 items Two factors: In-group trust and out-group trust 4-point scale

Set 3: Caution (Yamagishi & Yamagishi, 1994) 6 items Two factors: In-group caution and out-group caution 4-point scale

Set 4: Moral and rational trust (Uslaner, 2002; Yamagishi & Yamagishi, 1994) 6 items Two factors: Moral trust and rational trust 7-point scale

Set 5: Institutional trust 8 items Two factors: Institutes and voluntary organizations 5-point scale

Set 6: Tolerance 8 items Three factors: Out-group tolerance, deviant groups’ tolerance, and stigmatized groups’ tolerance 4-point scale

Set 7: Schwartz values (Schwartz, 1992) 11 items Two factors: Self-transcendence (universalism and benevolence) and conservation (security, conformity, and conservation) 6-point scale

Indicators

Process quality indicators

The first set of indicators we examined relate to process, quality, or respondent behavior completing the survey. We examined three indicators commonly used in the literature:
Breakoff rates: Among those who started each survey, this is the proportion failing to complete the survey. We examine overall breakoff rates and breakoffs on the target question sets.

Completion times: We compare the mean time to complete the question sets. The values were truncated at the 95th percentile (following Yan & Tourangeau, 2008).

Straightlining: We computed the number of question sets in which a respondent selected the same response option as the indicator of straightlining.

Internal consistency: We used Cronbach’s α to measure item–total correlations. Since there were some significant differences in age and gender between the conditions in the first wave, we controlled for age and gender to compare reliability between the conditions.

Measurement quality indicators

The next group of indicators we examined relate to measurement quality. These include the following.

Measurement invariance

Measurement invariance indicates that the same latent constructs are measured among different groups (Cernat, Couper, & Ofstedal, 2016; Horn & McArdle, 1992; Hox, de Leeuw, & Zijlmans, 2015). Multigroup CFA was applied to measure data equivalence across formats and devices, as well as across formats in the two devices (Format × Device). The usual practice is to apply ordered categorical CFA if there are four or fewer response options in a scale and continuous CFA if there are five or more response options (Finney & DiStefano, 2006; Hox et al., 2015) and we followed this convention here. We proceeded with the following steps in assessing measurement equivalence (Meredith, 1993; Steenkamp & Baumgartner, 1998; Vandenberg & Lance, 2000):
Configural equivalence—the weakest form of equivalence. Configural equivalence tests whether the same patterns of latent constructs are found across conditions.

Metric equivalence constrains the factor loadings to be equal across conditions. If metric equivalence is achieved, the factor loadings are equal across formats and devices.

Scalar equivalence constrains factor loadings and intercepts (or thresholds in ordered categorical CFA) to be equal across conditions. If scalar equivalence is achieved, the factor loadings and intercepts are equal across formats and devices.

Latent means constrain the means to be equal across conditions.

Residuals constrain error variances to be equal across conditions (not applied in ordered categorical CFA).

Variances of latent variables are then constrained across conditions.

Covariances of latent variables.

If the variances of the latent variables are similar across conditions, we constrained the covariances of the latent variables (if there are at least two factors) to be equal across conditions to measure whether construct validity is similar across conditions. Following Chen’s (2007) recommendation, comparative fit index (CFI) was chosen as the main criterion for measurement equivalence. A change in CFI of 0.01 indicated measurement noninvariance. We used the lavaan package in R (version 0.5-23.1097) (see http://lavaan.org).

Concurrent validity

We measured concurrent validity and compared partial correlation coefficients between the question formats and devices. We included the following three questions in the analysis:

Generalized trust (Question 9)

Generally speaking, would you say that most people can be trusted or that you need to be very careful in dealing with people? (1 = most people can be trusted, 2 = need to be very careful)

Trust (Question 11)

How much would you say one can trust people? (1 = you can trust most people, 7 = you cannot trust most people)

Caution (Question 12)

Some people think that one should be careful in dealing with people, while others think there is no need to be careful in dealing with people. And what do you think? (1 = one should be careful in dealing with people, 7 = there is no need to be careful in dealing with people)

In our study, we compared the partial correlation coefficients between the conditions. We conducted nine models predicting the latent variables while controlling for age and gender based on the following questions:
in-group trust (Set 2) based on generalized trust (Question 9);

out-group trust (Set 2) based on generalized trust (Question 9);

rational trust (Set 4) based on generalized trust (Question 9);

in-group trust (Set 2) based on trust (Question 11);

out-group trust (Set 2) based on trust (Question 11);

rational trust (Set 4) based on trust (Question 11); and

out-group caution (Set 3) based on caution (Question 12).

We conducted a multigroup analysis and compared the partial correlations between the formats and devices. We examined the magnitude of the correlation without regard to the direction of the relationship.

Reliability

We predicted the score of the latent constructs in the second wave based on the following variables:

score of the latent construct in the first wave;

device used in the second wave (PC = 1, smartphone = 0);

gender;

age; and

question format in the two waves.

We divided the question format variable into three conditions: grid format in both waves, item-by-item format in both waves, and different question formats between the waves (the latter was selected as the reference group). An example of the structural equation model for Question Set 1 (risk willingness) is shown in Online Appendix C, Figure 1.

We measured the effect of device and question format on the latent score in the second wave. In addition, we measured whether the correlation between the latent score in the first wave and the latent score in the second wave can be mediated by the question format. This would help to examine whether using different formats (grid vs. item-by-item) has an effect on the test–retest reliability correlation between the scores of the latent construct in two waves.

Subjective evaluation

The final set of indicators we examined relate to the respondents’ subjective experience of the survey and serve as indirect indicators of respondent burden. We used the following four variables (asked at the end of each survey) as subjective indicators of respondent burden:
self-reported difficulty of completing the survey (4-point scale);

self-reported enjoyment of completing the survey (4-point scale);

subjective report of technical problems while completing the survey; and

subjective survey completion time (how many minutes did it take to complete the survey).

We recoded the self-reported difficulty of completing the survey (1 = very easy to complete, 0 = otherwise) and enjoyment of completing the survey (1 = enjoyed a lot, 0 = otherwise) into binary variables. We conducted logistic and ordinary least squares (OLS) regressions to explore whether there were any differences in the subjective indicators of respondent burden between the question formats and devices.

Sample Composition

Overall, 45% of respondents were male in the first wave and 47% in the second wave. The mean age of the participants was 34.4 years. Significant gender and age differences were found between the two devices in the first wave, but no significant differences were found between formats (see Table 4). No significant gender and age differences were found between device and question format in the second wave. We conclude that the randomization worked as intended, and the two-wave design protects us from selection biases.

Table 4.
Sample Composition.

Condition Format Statistics Device Statistics Format × Device Statistics

Grids Item-by-Item PC Mobile PC Grids PC Item- by-Item Mobile Grids Mobile Item-by-Item

Wave 1

N 832 846 866 812 574 292 258 554

Males 44% 45% χ²(1) = 0.64 48% 41% χ²(1) = 9.8** 48% 48% 39% 41% χ²(3) = 10.42*

Age 34.6 (10.3) 34.1 (10.3) t(1671) = 1.01 35.6 (10.2) 33.1 (10.2) t(1671) = 4.98* 35.4 (10.2) 35.9 (10.4) 32.8 (10.3) 33.2 (10.0) F(3, 1669) = 8.45*

Wave 2

N 522 557 540 539 347 193 175 364

Males 49% 45% χ²(1) = 1.13 47% 47% χ²(1) = 0.00 46% 48% 53% 44% χ²(3) = 3.61

Age 34.3 (10.1) 34.7 (10.5) t(1072) = −0.68 34.2 (10.4) 34.9 (10.1) t(1072) = 1.15 34.2 (10.4) 34.1 (10.6) 34.5 (9.5) 35.1 (10.4) F(3, 1070) = 0.58

p < .05. p < .01. p < .001.

Results

Breakoff Rates

If respondents used the device to which they were assigned, they were randomly assigned either to the grid or to the item-by-item format. We found a higher breakoff rate overall in the smartphone than in the PC web condition in the first wave: 15.2% and 10.0%, respectively, χ²(1) = 16.40, p < .001, and no difference between the survey formats: 12.4% in the grid format and 12.5% in the item-by-item format. No statistically significant breakoff rates were found on any of the seven target questions sets between the question formats or devices.

The breakoff rate in the second wave was 4.0%. There was no difference between devices or question formats: 3.9% on PC and 4.1% on smartphones, χ²(1) = 0.02, p = .88; 4.2% in the grid format and 3.8% in the item-by-item format, χ²(1) = 0.13, p = .72.

Completion Time

For all indicators of data quality, we included in the analysis those who completed the first wave in the assigned condition while analyzing the results of the first wave and those who completed both waves while analyzing the results of the second wave.

The mean completion time was 19.9 min in the first wave and 17.7 min in the second wave. The median completion time was 17.1 min and 15.1 min, respectively. As expected, the mean completion time was shorter in the PC web group than in the mobile web group in both waves and shorter in the grid format than in the item-by-item format (see Table 5, also Figure 1 in Online Appendix D). While analyzing only the target question sets, the same significant differences between question format and device were found.

Table 5.
Survey Completion Times (Minutes).

Format Device Format × Device

Condition Overall Grids Item-by-Item PC Mobile PC Grids PC Item-by-Item Mobile Grids Mobile Item-by-Item

Wave 1 t (df = 1,676) t (df = 1,676) F(3, 1674)

Full survey 19.9 (9.8) 18.8 (9.2) 20.9 (10.3) 4.48* 19.1 (9.8) 20.8 (9.8) 3.58* 18.9 (10.0) 19.4 (9.4) 18.7 (6.9) 21.7 (10.7) 10.26*

Target question sets 7.6 (4.1) 6.7 (3.6) 8.6 (4.2) 9.88* 7.1 (4.0) 8.2 (4.0) 5.96 6.7 (4.0) 7.7 (4.0) 6.6 (2.8) 9.0 (4.3) 39.87*

Wave 2 t(df = 1,077) t(df = 1,077) F(3, 1075)

Full survey 17.7 (9.8) 16.8 (9.0) 18.7 (10.4) 3.20 16.8 (9.8) 18.7 (9.7) 3.30* 16.5 (9.9) 17.3 (9.6) 17.4 (6.8) 19.4 (10.7) 5.66*

Target question sets 7.3 (4.1) 6.3 (3.8) 8.1 (4.2) 7.25* 6.7 (4.2) 7.8 (3.9) 4.62* 6.2 (4.2) 7.5 (4.3) 6.6 (2.9) 8.4 (4.1) 20.20*

Note. Standard deviation is given in parentheses.

p < .05. p < .01. p < .001.

Figure 1.
Cronbach’s α by question format and device. p < .05, p < .01, p < .001. PC/gr. = PC grids; PC/it. = PC items; Mob/gr. = mobile grids; Mob/it. = mobile items.

The longest completion time was found in the mobile item-by-item condition (21.7 min in the first wave and 19.4 min in the second wave) and the shortest completion times in the PC and mobile grid conditions.

The OLS regression that controlled for gender and age differences between the conditions showed the same patterns with significant effects of device and format on overall survey completion time and completion time for the target question sets.

Straightlining

Overall, about 25% of respondents straightlined in at least one of the grids (see Table 1 in Online Appendix D for the percentage straightlining in each question set). We ran a negative binomial model to predict the number of grids in which respondents straightlined, controlling for age and gender. The question format was statistically significant: The grid format increased the odds of straightlining by 1.34 in the first wave and by 1.38 in the second wave (p < .05, see Table 6). Device and the interaction between format and device (the interaction is not shown) had no effect on the level of straightlining. The mixed effects negative binomial model predicting the level of straightlining in both waves showed the same effect of the grid format increasing the level of straightlining (β = 0.30, ORs = 1.35, p < .001), no effect of device and no effect of the interaction between format and device.

Table 6.
Negative Binomial Models of Straightlining.

Model Wave 1 Wave 2

β Odds Ratios β Odds Ratios

Intercept −1.51* (0.18) 0.22* [0.16, 0.31] −1.39* (0.23) 0.25* [0.16, 0.38]

Grids (vs. item-by-item format) 0.30 (0.14) 1.34 [1.11, 1.63] 0.32* (0.16) 1.38* [1.09, 1.73]

PC (vs. mobile) 0.13 (0.10) 1.14 [0.94, 1.38] −0.14 (0.12) 0.87 [0.69, 1.09]

Males 0.28 (0.09) 1.32 [1.10, 1.59] −0.15 (0.11) 1.17 [0.93, 1.46]

Age 0.01 (0.00) 1.01 [1.00, 1.02] 0.01 (0.01) 1.01 [1.00, 1.02]

Model Likelihood ratio χ²(4) = 30.83, p < .001 Likelihood ratio χ²(4) = 14.19, p < .01

Note. Standard error is given in parentheses and 95% CI in brackets.

p < .05. p < .01. p < .001.

Internal Consistency

We measured internal consistency within question sets using Cronbach’s α, while controlling for age and gender. We found significantly higher Cronbach’s α coefficients in two of the seven questions in the grid than in the item-by-item format in both waves. The difference was statistically significant in Question Set 4 on moral and rational trust and Set 6 on tolerance (see Figure 1 and Table 4 in Online Appendix D). The Cronbach’s α coefficients were .74 in the grid format and .62 in the item-by-item format in Question Set 4 in the first wave, χ²(1) = 21.44, p < .001, and .75 in the grid format and .64 in the item-by-item format in the second wave, χ²(1) = 12.58, p < .001. The difference was lower in Question Set 6 on tolerance but also statistically significant: .79 in the grid format and .74 in the item-by-item format in the first wave, χ²(1) = 7.39, p < .01, and .80 in the grid format and .75 in the item-by-item format in the second wave, χ²(1) = 5.16, p < .05. We suggest that higher Cronbach’s α coefficients in the grid format are associated with a higher tendency to straightline, as found earlier (see also Couper et al., 2013).

No statistically significant differences were found between devices except for Set 1 on risk willingness in the first wave (Figure 1 and Table 4 in Online Appendix D). The interaction between question format and device was statistically significant for Question Set 4 (moral and rational trust) in both waves and Set 2 (trust) in the second wave with a lower Cronbach’s α in the mobile item-by-item condition.

Measurement Equivalence

Since there were some differences in gender and age between the conditions in the first wave, we controlled for these variables in the models while running a multigroup CFA to measure data equivalence across formats and devices. While configural equivalence is the weakest form of equivalence, equivalence of variances of latent variables or covariances of latent variables (in the case of at least two latent factors) are the strongest forms of measurement equivalence.

The measurement equivalence at the level of covariances of latent variables or variances of latent variables in the case of a single latent factor was reached between devices in all question sets in both waves (see Table 7; for more details, see Tables 2 and 3 in Online Appendix D). We highlight the cells in gray where the stronger form of measurement equivalence was not reached. In all except one 7-point scale (Set 4), construct validity was reached between the question formats. In this 7-point scale grid, the measurement equivalence was reached at the level of residuals. While exploring the measurement equivalence between the four conditions (the interaction between the question format and device), sufficient construct validity was reached in all except two grids: a 4-point scale on caution (Set 3) and a 7-point scale on moral and rational trust (Set 4).

Table 7.
Summary of Measurement Equivalence (Measurement Equivalence Achieved) by Question Format and Device.

Question sets Question setsNumber of Latent Factors Response Scale Format (Grid vs. Item-by-Item) Device (PC vs. Mobile) Format × Device

Set 1: Risk willingness One factor 4-point scale Variances of latent variables Variances of latent variables Variances of latent variables

Set 2: Trust Two factors 4-point scale Covariances of latent variables Covariances of latent variables Covariances of latent variables

Set 3: Caution Two factors 4-point scale Covariances of latent variables Covariances of latent variables Configural (Wave 1)Covariances of latent variables (Wave 2)

Set 4: Moral and rational trust Two factors 7-point scale Residuals Covariances of latent variables Residuals (wave 1)Latent means (wave 2)

Set 5: Institutional trust Two factors 5-point scale Covariances of latent variables Covariances of latent variables Covariances of latent variables

Set 6: Tolerance Three factors 4-point scale Covariances of latent variables Covariances of latent variables Covariances of latent variables

Set 7: Schwartz values Two factors 6-point scale Covariances of latent variables Covariances of latent variables Covariances of latent variables

Note. The measurement equivalence from the weakest to the strongest forms of measurement equivalence (see the Indicators section for more details): (1) configural equivalence, (2) metric equivalence, (3) scalar equivalence, (4) latent means, (5) residuals, (6) variances of latent variables, and (7) covariances of latent variables (when there are at least two latent factors). Cells highlighted in grey: The stronger form of measurement equivalence was not reached.

Concurrent Validity

We measured concurrent validity and compared it between the question formats and devices. We predicted some latent variables while controlling for age and gender based on variables which measured similar constructs (using single-item questions).

We found higher partial correlation coefficients in the item-by-item format than in the grid format. Five of the seven correlations were statistically higher in the item-by-item format in the first wave (see Figure 2 and Table 5 in Online Appendix D). The partial correlation between in-group trust (Question Set 2) and generalized trust was .47 in the grid format and .60 in the item-by-item format (z = 3.74, p < .001); between out-group trust (Set 2) and generalized trust .43 and .57, respectively (z = 3.83, p < .001); between rational trust (Set 4) and generalized trust .43 and .52 (z = 2.38, p < .05); between out-group trust (Set 2) and trust .52 and .62 (z = 3.03, p < .01); and between rational trust (Set 4) and trust .50 and .64, respectively (z = 4.26, p < .001). Only one statistically significant difference was found in the second wave.

Figure 2.
Partial correlation coefficients by question format and device. p < .05, p < .01, p < .001. PC/gr. = PC grids; PC/it. = PC items; Mob/gr. = mobile grids; Mob/it. = mobile items.

There was no clear pattern in the difference in concurrent validity between devices: Three statistically significant differences were found with a higher partial correlation coefficient in the smartphone condition in both waves and one statistically significant higher partial correlation coefficient in the PC condition (see Figure 2). Comparing all four conditions (PC grid, PC item-by-item, smartphone grid, and smartphone item-by-item), we can conclude that both the smartphone and PC item-by-item conditions produced higher partial correlation coefficients in the first wave; however, there was no clear pattern in the second wave (see Figure 2 and Table 5 in Online Appendix D).

Reliability

To evaluate test–retest reliability, we predicted the score of the latent variable in the second wave based on the score of the latent variable in the first wave, as well as based on the question format (condition with the grid format in both waves, condition with the item-by-item format in both waves, and condition with different question formats between the waves) and device, while controlling for age and gender. We found almost no significant differences between the question formats and no clear pattern of the differences between devices (see Table 6 in Online Appendix D). We also measured whether the correlation between the latent score in the first wave and the latent score in the second wave was mediated by the question format. We found no significant interaction effects. This means that using different formats (grid or item-by-item) had no effect on the test–retest reliability correlations between the scores of the latent construct in the two waves. Overall, we conclude that there is no clear answer which question format or device produced lower measurement error in terms of reliability.

Subjective Evaluation

Overall, 65% of the participants reported that it was very easy to complete the survey in the first wave; however, this proportion was higher for PC than smartphone respondents, 74% versus 55%, χ²(1) = 63.56, p < .001, and in the item-by-item than in the grid format, 69% versus 60%, χ²(1) = 12.75, p < .001. Similar patterns were found in the second wave. Logistic regression showed that the grid format was associated with significantly lower odds of reported ease in both the first wave (OR = 0.22, p < .001, see Table 8) and second wave (OR = 0.24, p < .001), while using a PC rather than a smartphone increased the odds of a very easy rating by 1.40 (p < .05) in the first wave and 1.67 (p < .01) in the second wave. A significant interaction between device and question format shows that using grids on smartphones has a negative effect on self-reported ease of completion, but no such effect was found among PC web respondents.

Table 8.
Logistic Regression: Predicting Subjective Indicators of Respondent Burden.

Very Easy to Complete the Survey Enjoyed Completing the Survey a Lot Some Technical Difficulties While Completing the Survey

Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2

Model β Odds Ratios β Odds Ratios β Odds Ratios β Odds Ratios β Odds Ratios β Odds Ratios

Intercept 0.70(0.20) 2.02[1.37, 2.97] 0.17(0.25) 1.19[0.73, 1.93] −0.33(0.19) 0.72[0.50, 1.04] −0.23(0.25) 0.79[0.49, 1.29] −1.70(0.30) 0.18[0.10, .033] −1.88(0.41) 0.15[0.07, 0.34]

Grids (vs. item-by-item format) −1.50(0.16) 0.22[0.16, 0.31] −1.41(0.20) 0.24[0.16, 0.36] −1.53(0.20) 0.22[0.15, 0.32] −1.03(0.22) 0.36[0.23, 0.55] 2.25(0.19) 9.52[6.60, 13.73] 2.08(0.24) 8.00[5.01, 12.78]

PC (vs. mobile) 0.33(0.16) 1.40[1.02, 1.91] 0.51(0.19) 1.67[1.14, 2.43] 0.00(0.15) 1.00[0.75, 1.34] 0.14(0.18) 1.15[0.80, 1.64] −1.26(0.37) 0.28[0.14, 0.58] −1.15(0.45) 0.32[0.13, 0.77]

Grids × PC 1.52(0.23) 4.58[2.92, 7.19] 1.16(0.28) 3.19[1.83, 5.54] 1.69(0.24) 5.41[3.35, 8.74] 0.68(0.29) 1.98[1.12, 3.49] −2.13(0.45) 0.12[0.05, 0.29] −2.05(0.57) 0.13[0.04, 0.39]

Males −0.14(0.11) 0.87[0.70, 1.08] −0.02(0.13) 0.98[0.75, 1.27] 0.08(0.10) 1.09[0.89, 1.34] 0.24(0.13) 1.27[0.97, 1.65] −0.10(0.18) 0.91[0.64, 1.28] −0.03(0.22) 0.97[0.64, 1.48]

Age 0.00(0.01) 1.00[0.99, 1.01] 0.01(0.01) 1.01[1.00, 1.02] 0.00(0.01) 1.00[0.99, 1.01] −0.01(0.01) 0.99[0.98, 1.00] −0.01(0.01) 0.99[0.97, 1.00] −0.01(0.01) 0.99[0.97, 1.01]

Model Likelihood ratio χ²(5) = 156.26, p < .001, AIC = 829.61, BIC = 862.15, Deviance (311) = 388.01 Likelihood ratio χ²(5) = 96.31, p < .001, AIC = 680.33, BIC = 710.21, Deviance (311) = 368.34 Likelihood ratio χ²(5) = 101.54, p < .001, AIC = 875.12, BIC = 907.65, Deviance (293) = 426.75 Likelihood ratio χ²(5) = 33.33, p < .001, AIC = 689.26, BIC = 719.13, Deviance (293) = 379.86 Likelihood ratio χ²(5) = 339.98, p < .001, AIC = 477.12, BIC = 509.65, Deviance (311) = 262.01 Likelihood ratio χ²(5) = 173.65, p < .001, AIC = 366.97, BIC = 396.85, Deviance (293) = 221.92

Note. Standard error of β is given in parentheses, and 95% CI of odds ratios is given in brackets. AIC = Akaike information criterion; BIC = Bayesian information criterion.

p < .05. p < .01. p < .001.

About 41% of the participants reported that they enjoyed completing the survey a lot. More people reported enjoyment in the PC than in the smartphone condition, 47% versus 34%, χ²(1) = 27.26, p < .001, and in the item-by-item than in the grid format, 44% and 38%, χ²(1) = 6.50, p < .05. Similar patterns were found in the second wave. Logistic regressions showed no effect of device and a negative effect of grids on enjoyment. Using grids significantly lowered the odds of reported enjoyment (in both the first OR = 0.22, p < .001, Table 8) and second wave (OR = 0.36, p < .001). A significant interaction between device and format showed that while using grids on smartphones had a negative effect on subjective evaluation, no such effect was found in the PC web condition.

About 13% of the respondents reported having some technical difficulties while completing the questionnaire. This proportion was higher among smartphone than PC respondents, 24% versus 3%, χ²(1) = 153.47, p < .001, and among those in the grid than in the item-by-item format, 19% versus 8%, χ²(1) = 42.70, p < .001. Similar patterns were found in the second wave. Logistic regression showed that grids significantly increased the odds of reporting technical difficulties in both the first (OR = 9.52, p < .001, Table 8) and second wave (OR = 8.00, p < .001). Using a PC rather than a smartphone decreased the odds of reporting technical difficulties in the first (OR = 0.28, p < .001) and second wave (OR = 0.32, p < .05). Similar to other subjective indicators, using a grid format on a smartphone increased the odds of reporting technical difficulties, while using the grid format on a PC had no such effect.

Subjective estimation of completion time showed the same patterns. The OLS regression showed that the grid format increased the subjective length of the survey (β = 2.02, p < .001 in the first wave and β = 1.94, p < .001 in the second wave, data not shown); however, using grids on a PC decreased subjective survey length compared to grids on a smartphone (β = −2.54, p < .001 in the first wave and β = −2.92, p < .001 in the second wave).

Overall, all subjective evaluation indicators showed that the grid format was associated with lower (more negative) subjective evaluation and increased reported technical difficulties and subjective evaluation of survey length. This negative effect was particularly strong for smartphones. However, we should note that these negative effects could be mostly due to the nonoptimized version of the survey in the smartphone condition rather than the grid format per se.

Discussion

Grid or matrix questions are commonly used in web surveys. With a proliferation of mobile devices, some software presents grid questions in an item-by-item format on mobile devices and grid format on PC. Some researchers suggest using the item-by-item format on both PCs and mobile devices (Kuper & Glowa, 2014; Revilla et al., 2017), while others suggest using a grid format on PCs and an item-by-item format on mobile devices (de Bruijne, Marcel, van Soest, & Wijnant, 2015; Stern, Sterrett, & Bilgen, 2016). Still others advocate for new question formats to replace grids on smartphones. We conducted a two-wave experiment and varied the device used (PC vs. smartphone) and the format of the questions (grids vs. item-by-item scrolling) to compare data quality between the formats across devices. The grid format in the mobile web condition was not fully optimized for smartphones.

Our main research hypothesis was that the item-by-item format would produce higher data quality than the grid format for both types of device, but the effect would be stronger among smartphone respondents. Overall, we can conclude with five main research findings.

First, the multigroup comparison showed some measurement equivalence between the question formats. In all except one grid with a 7-point scale, the measurement equivalence was reached at the level of covariances of latent variables (or variances of latent variables in case of one latent factor) between the question formats. The measurement equivalence was reached between devices in all question sets. However, while comparing four conditions (PC grids, PC item-by-item, mobile grids, and mobile item-by-item), the construct validity was not reached in two of the seven sets. The measurement equivalence was not reached in a 7-point scale on moral and rational trust and in a 4-point scale on caution. Liu & Cernat (2016) found measurement equivalence between the grid and item-by-item formats in questions with seven responses or fewer. We found that the questions with a 7-point scale had some differences between the two formats, as well as between the four conditions (Question Format × Device). Overall, we can conclude that in question sets with a 5-point scale or with fewer response categories, the measurement equivalence between the question formats would be reached on both device types. However, there might be some differences in measurement equivalence in the sets with seven or more response categories.

Second, we found almost no significant effects of the question format on test–retest correlations between the latent scores in two waves. We measured whether using a grid format in both waves, using an item-by-item format in both waves, or using different question formats across waves had any effect on predictions of the latent score in the second wave. We found only a few differences and no clear pattern in these differences.

Third, we found no differences in breakoff rates between the question formats. Similar to other studies, we found that mobile web produced a higher breakoff rate than PC web; however, we found no differences between the question formats and between the question formats across devices.

Fourth, similar to other researchers, we found that the grid format is associated with shorter completion times in both PC and mobile web (Callegaro et al., 2009; Couper et al., 2001; Revilla et al., 2017; Toepoel et al., 2009; Tourangeau et al., 2004). However, the price for that shorter completion times is somewhat higher measurement error. The levels of straightlining were also higher in the grid than in the item-by-item format. No stronger effect of the grid format on the level of straightlining was found in the mobile web condition. We also found somewhat higher Cronbach’s α coefficients in the grid than in the item-by-item format, which could be associated with a higher tendency of straightlining. In addition, concurrent validity was lower in the grid than in the item-by-item format in both PC and mobile web conditions. Overall, based on these indicators, the measurement error in the grid format was higher than in the item-by-item format. It seems that the grid format might produce a higher tendency for quicker response, stronger satisficing, and higher probability of less attentive and less thorough responses.

Finally, subjective indicators of respondent burden showed that the grid format increased reported technical difficulties and decreased subjective evaluation of the survey. The effect was substantial in the mobile web condition, while almost no negative effect of the grid format was found in the PC condition. Although the objective completion time was shorter in the grid than in the item-by-item format, subjective evaluation of interview length was longer in the grid format. Again, this effect was strong among mobile web respondents, while no such negative effect of the grid format was found among PC web respondents.

Limitations and Outlook

Some limitations of the experimental study should be noted. First, the study was restricted to members of a single volunteer online opt in panel in a single country. This is not a probability-based sample, and the members of the panel participate in web surveys on a regular basis, which might dampen the effect of different question formats. The sample was further restricted to those who completed two survey waves using the assigned device. While this design limits external validity, it maximizes the internal validity of the comparisons. Second, the grid format in the mobile web condition was implemented using a partially optimized style sheet. This means that implementing grids on smartphones using other forms of mobile optimization may produce different results. It would be useful to replicate these experiments using fully optimized grids (e.g., adjusting font sizes in addition to making the input field touch friendly). Third, we were not able to capture client-side paradata so were not able to measure behaviors such as pinching and zooming or scrolling (see Couper & Peterson, 2017) across formats and devices. Along with Liu & Cernat (2017), we suggest the capture of additional paradata to better understand the reasons for different measurement errors between the question formats on different devices.

A final limitation is that we did not systematically vary the number of response options and number of items in the question sets to explore how these variables may affect nonresponse and measurement error. Liu & Cernat (2017) showed that the number of response categories has an effect on measurement equivalence. We found some support for this in our experimental study but did not exceed seven (end point–labeled) response options. Some researchers showed that measurement errors can be different depending on the number of items included in the grid questions (Peytchev, 2005; Stern et al., 2016; but see Revilla & Couper, 2017). We suggest that further research should vary the number of response options and number of items to measure how these factors might have an effect on different indicators of data quality in PC and mobile web surveys.

Conclusions and Practical Implications

Our findings offer some practical suggestions for implementing grid questions in web surveys. We suggest there are three main implications.

First, based on our results, if one is not able to use mobile optimization for smartphones, then using an item-by-item format rather than a grid format for mobile web surveys may be prudent. This finding replicates several earlier studies that find grids to be less than optimal for nonoptimized surveys on smartphones. Our results suggest that using grids on mobile devices without optimization increases measurement error and decreases the subjective evaluation of the survey.

Second, in questions with a seven or more response options, we recommend using an item-by-item format in both mobile and PC web surveys; otherwise, there may be differences in measurement equivalence between devices. This is important as some survey software optimized automatically for mobile devices by turning grids into item-by-item formats for smartphones, but leaving them as grids on PCs, potentially resulting in compromises in measurement equivalence.

Third, we suggest that using the item-by-item format in PC web surveys may increase concurrent validity and decrease nondifferentiation or straightlining. In addition, we found that although the grid format decreased survey completion time, it had no effect on subjective survey length among PC web respondents.

As we noted at the outset, there are a large number of factors that may affect the choice of a grid format for surveys, including the length and complexity of the survey, the number of items, the number of response options, the proportion of the respondents using a mobile device, the type of mobile optimization (if any) for smartphones, and so on. The choice of whether or not to use grid question should be made on a case-by-case basis and is not an all-or-nothing decision. If the survey software does not optimize for mobile devices (particularly smartphones), we suggest that using an item-by-item format for both mobile and PC web may result in lower measurement error and higher measurement equivalence in a survey.

Supplemental Material

Online_Appendices - Grid and Item-by-Item Formats in PC and Mobile Web Surveys

Online_Appendices for Grid and Item-by-Item Formats in PC and Mobile Web Surveys by Aigul Mavletova, Mick P. Couper, and Daniil Lebedev in Social Science Computer Review

Group	First Wave	N	Second Wave	N
1	Grids on smartphone	258	Grids on PC	165
2	Grids on PC	285	Grids on smartphone	175
3	Item-by-item on smartphone	278	Item-by-item on PC	193
4	Item-by-item on PC	292	Item-by-item on smartphone	176
5	Item-by-item on smartphone	276	Grids on PC	182
6	Grids on PC	289	Item-by-item on smartphone	188

Condition	Format	Device	Format × Device
Wave 1	832	846	866	812	574	292	258	554
Wave 2	522	557	540	539	347	193	175	364

Question sets	Number of Items	Latent Factors	Response Scale
Set 1: Risk willingness	4 items	One factor: Risk willingness	4-point scale
Set 2: Trust (Uslaner, 2002; Yamagishi & Yamagishi, 1994)	6 items	Two factors: In-group trust and out-group trust	4-point scale
Set 3: Caution (Yamagishi & Yamagishi, 1994)	6 items	Two factors: In-group caution and out-group caution	4-point scale
Set 4: Moral and rational trust (Uslaner, 2002; Yamagishi & Yamagishi, 1994)	6 items	Two factors: Moral trust and rational trust	7-point scale
Set 5: Institutional trust	8 items	Two factors: Institutes and voluntary organizations	5-point scale
Set 6: Tolerance	8 items	Three factors: Out-group tolerance, deviant groups’ tolerance, and stigmatized groups’ tolerance	4-point scale
Set 7: Schwartz values (Schwartz, 1992)	11 items	Two factors: Self-transcendence (universalism and benevolence) and conservation (security, conformity, and conservation)	6-point scale

Condition	Format	Statistics	Device	Statistics	Format × Device	Statistics
Wave 1
N	832	846		866	812		574	292	258	554
Males	44%	45%	χ²(1) = 0.64	48%	41%	χ²(1) = 9.8**	48%	48%	39%	41%	χ²(3) = 10.42*
Age	34.6 (10.3)	34.1 (10.3)	t(1671) = 1.01	35.6 (10.2)	33.1 (10.2)	t(1671) = 4.98***	35.4 (10.2)	35.9 (10.4)	32.8 (10.3)	33.2 (10.0)	F(3, 1669) = 8.45***
Wave 2
N	522	557		540	539		347	193	175	364
Males	49%	45%	χ²(1) = 1.13	47%	47%	χ²(1) = 0.00	46%	48%	53%	44%	χ²(3) = 3.61
Age	34.3 (10.1)	34.7 (10.5)	t(1072) = −0.68	34.2 (10.4)	34.9 (10.1)	t(1072) = 1.15	34.2 (10.4)	34.1 (10.6)	34.5 (9.5)	35.1 (10.4)	F(3, 1070) = 0.58

	Format	Device		Format × Device
Wave 1				t (df = 1,676)			t (df = 1,676)					F(3, 1674)
Full survey	19.9 (9.8)	18.8 (9.2)	20.9 (10.3)	4.48***	19.1 (9.8)	20.8 (9.8)	3.58***	18.9 (10.0)	19.4 (9.4)	18.7 (6.9)	21.7 (10.7)	10.26***
Target question sets	7.6 (4.1)	6.7 (3.6)	8.6 (4.2)	9.88***	7.1 (4.0)	8.2 (4.0)	5.96**	6.7 (4.0)	7.7 (4.0)	6.6 (2.8)	9.0 (4.3)	39.87***
Wave 2				t(df = 1,077)			t(df = 1,077)					F(3, 1075)
Full survey	17.7 (9.8)	16.8 (9.0)	18.7 (10.4)	3.20**	16.8 (9.8)	18.7 (9.7)	3.30***	16.5 (9.9)	17.3 (9.6)	17.4 (6.8)	19.4 (10.7)	5.66***
Target question sets	7.3 (4.1)	6.3 (3.8)	8.1 (4.2)	7.25***	6.7 (4.2)	7.8 (3.9)	4.62***	6.2 (4.2)	7.5 (4.3)	6.6 (2.9)	8.4 (4.1)	20.20***

Model	Wave 1	Wave 2
Intercept	−1.51*** (0.18)	0.22*** [0.16, 0.31]	−1.39*** (0.23)	0.25*** [0.16, 0.38]
Grids (vs. item-by-item format)	0.30** (0.14)	1.34** [1.11, 1.63]	0.32* (0.16)	1.38* [1.09, 1.73]
PC (vs. mobile)	0.13 (0.10)	1.14 [0.94, 1.38]	−0.14 (0.12)	0.87 [0.69, 1.09]
Males	0.28** (0.09)	1.32** [1.10, 1.59]	−0.15 (0.11)	1.17 [0.93, 1.46]
Age	0.01 (0.00)	1.01 [1.00, 1.02]	0.01 (0.01)	1.01 [1.00, 1.02]
Model	Likelihood ratio χ²(4) = 30.83, p < .001	Likelihood ratio χ²(4) = 14.19, p < .01

Question sets	Question setsNumber of Latent Factors	Response Scale	Format (Grid vs. Item-by-Item)	Device (PC vs. Mobile)	Format × Device
Set 1: Risk willingness	One factor	4-point scale	Variances of latent variables	Variances of latent variables	Variances of latent variables
Set 2: Trust	Two factors	4-point scale	Covariances of latent variables	Covariances of latent variables	Covariances of latent variables
Set 3: Caution	Two factors	4-point scale	Covariances of latent variables	Covariances of latent variables	Configural (Wave 1)Covariances of latent variables (Wave 2)
Set 4: Moral and rational trust	Two factors	7-point scale	Residuals	Covariances of latent variables	Residuals (wave 1)Latent means (wave 2)
Set 5: Institutional trust	Two factors	5-point scale	Covariances of latent variables	Covariances of latent variables	Covariances of latent variables
Set 6: Tolerance	Three factors	4-point scale	Covariances of latent variables	Covariances of latent variables	Covariances of latent variables
Set 7: Schwartz values	Two factors	6-point scale	Covariances of latent variables	Covariances of latent variables	Covariances of latent variables

	Very Easy to Complete the Survey	Enjoyed Completing the Survey a Lot	Some Technical Difficulties While Completing the Survey
Intercept	0.70***(0.20)	2.02***[1.37, 2.97]	0.17(0.25)	1.19[0.73, 1.93]	−0.33(0.19)	0.72[0.50, 1.04]	−0.23(0.25)	0.79[0.49, 1.29]	−1.70***(0.30)	0.18***[0.10, .033]	−1.88***(0.41)	0.15***[0.07, 0.34]
Grids (vs. item-by-item format)	−1.50***(0.16)	0.22***[0.16, 0.31]	−1.41***(0.20)	0.24***[0.16, 0.36]	−1.53***(0.20)	0.22***[0.15, 0.32]	−1.03***(0.22)	0.36***[0.23, 0.55]	2.25***(0.19)	9.52***[6.60, 13.73]	2.08***(0.24)	8.00***[5.01, 12.78]
PC (vs. mobile)	0.33*(0.16)	1.40*[1.02, 1.91]	0.51**(0.19)	1.67**[1.14, 2.43]	0.00(0.15)	1.00[0.75, 1.34]	0.14(0.18)	1.15[0.80, 1.64]	−1.26***(0.37)	0.28***[0.14, 0.58]	−1.15*(0.45)	0.32*[0.13, 0.77]
Grids × PC	1.52***(0.23)	4.58***[2.92, 7.19]	1.16***(0.28)	3.19***[1.83, 5.54]	1.69***(0.24)	5.41***[3.35, 8.74]	0.68*(0.29)	1.98*[1.12, 3.49]	−2.13***(0.45)	0.12***[0.05, 0.29]	−2.05***(0.57)	0.13***[0.04, 0.39]
Males	−0.14(0.11)	0.87[0.70, 1.08]	−0.02(0.13)	0.98[0.75, 1.27]	0.08(0.10)	1.09[0.89, 1.34]	0.24(0.13)	1.27[0.97, 1.65]	−0.10(0.18)	0.91[0.64, 1.28]	−0.03(0.22)	0.97[0.64, 1.48]
Age	0.00(0.01)	1.00[0.99, 1.01]	0.01(0.01)	1.01[1.00, 1.02]	0.00(0.01)	1.00[0.99, 1.01]	−0.01(0.01)	0.99[0.98, 1.00]	−0.01(0.01)	0.99[0.97, 1.00]	−0.01(0.01)	0.99[0.97, 1.01]
Model	Likelihood ratio χ²(5) = 156.26, p < .001, AIC = 829.61, BIC = 862.15, Deviance (311) = 388.01	Likelihood ratio χ²(5) = 96.31, p < .001, AIC = 680.33, BIC = 710.21, Deviance (311) = 368.34	Likelihood ratio χ²(5) = 101.54, p < .001, AIC = 875.12, BIC = 907.65, Deviance (293) = 426.75	Likelihood ratio χ²(5) = 33.33, p < .001, AIC = 689.26, BIC = 719.13, Deviance (293) = 379.86	Likelihood ratio χ²(5) = 339.98, p < .001, AIC = 477.12, BIC = 509.65, Deviance (311) = 262.01	Likelihood ratio χ²(5) = 173.65, p < .001, AIC = 366.97, BIC = 396.85, Deviance (293) = 221.92

Footnotes

Authors’ Note

Aigul Mavletova would like to thank the Russian Academic Excellence Project “5-100”.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The article is supported by The Russian Science Foundation (Grant No. 17-78-20172).

Supplemental Material

The supplemental material is available in the online version of the article.

References

Antoun

Couper

M. P.

Conrad

F. G.

(2017). Effects of mobile versus PC web on survey response quality: A crossover experiment in a probability web panel. Public Opinion Quarterly, 81, 280–306.

Barlas

F. M.

Thomas

R. K.

Buttermore

(2017, 3). Mobile-friendly grid questions: The accordion grid as an alternative to the traditional grid. Paper presented at the General Online Research conference, Berlin, Germany.

Callegaro

Lozar Manfreda

Vehohar

(2015). Web survey methodology. Los Angeles, CA: Sage.

Callegaro

Shand-Lubbers

Dennis

J. M.

(2009). Presentation of a single item versus a grid: Effects on the vitality and mental health subscales of the SF-36v2 health survey. Retrieved from http://www.knowledgenetworks.com/ganp/docs/jsm2009/Presentation_of_single_item_JSM_2009_submitted.pdf

Cernat

Couper

M. P.

Ofstedal

M. B.

(2016). Estimation of mode effects in the health and retirement study using measurement models. Journal of Survey Statistics and Methodology, 4, 501–524.

Chen

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14, 464–504.

Couper

M. P.

Antoun

Mavletova

(2017). Mobile web surveys: A total survey error perspective. In Biemer

Eckman

Edwards

de Leeuw

Kreuter

Lyberg

Tucker

West

(Eds.), Total survey error in practice (pp. 133–154). New York, NY: Wiley.

Couper

M. P.

Peterson

G. J.

(2017). Why do web surveys take longer on smartphones? Social Science Computer Review, 35, 357–377.

Couper

M. P.

Tourangeau

Conrad

F. G.

Zhang

(2013). The design of grids in web surveys. Social Science Computer Review, 31, 322–345.

10.

Couper

M. P.

Traugott

Lamias

(2001). Web survey design and administration. Public Opinion Quarterly, 65, 230–253.

11.

De Bruijne

Marcel

van Soest

Wijnant

(2015). Adapting grid questions for mobile devices (Working paper). Tilburg, the Netherlands: CentERdata (Tilburg University).

12.

De Leeuw

E. D.

(2017, 7 17). Mixed mode: Past, present, and the future. Keynote presentation at the European Survey Research Association conference, Lisbon, Portugal.

13.

De Leeuw

E. D.

Hox

J. J.

Klausch

Roberts

de Jongh

(2012, 5). Design of web questionnaires: Matrix questions or single question formats. Paper presented at the annual conference of the American Association for Public Opinion Research, Orlando, FL.

14.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). New York, NY: Wiley.

15.

Finney

S. J.

DiStefano

(2006). Non normal and categorical data in structural equation modeling. In Hancock

G. R.

Mueller

R. O.

(Eds.), Structural equation modeling. A second course (pp. 269–314). Greenwich, CT: Information Age Publishing.

16.

Gräf

(2002). Assessing Internet questionnaires: The online pretest lab. In Battinic

Reips

U.-D.

Bosnjak

Werner

(Eds.), Online social sciences (pp. 69–79). Seattle, WA: Hogrefe & Huber.

17.

Hanson

(2017, 3). Adapting questionnaires for smartphones: An experiment on grid format questions. Paper presented at the General Online Research conference, Berlin, Germany.

18.

Horn

J. L.

McArdle

J. J.

(1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18, 117–144.

19.

Hox

J. J.

de Leeuw

E. D.

Zijlmans

E. A. O.

(2015). Measurement equivalence in mixed mode surveys. Frontiers in Psychology, 6. doi: 10.3389/fpsyg.2015.00087.

20.

Klausch

de Leeuw

E. D.

Hox

J. J.

Roberts

de Jongh

(2012, 3). Matrix vs. single question formats in web surveys: Results from a large scale experiment. Paper presented at the General Online Research conference, Mannheim, Germany.

21.

Kuper

Glowa

(2014). Going mobile with your survey? Get off the grid (questions). Quirks Marketing Research Review. Retrieved from http://www.quirks.com/articles/2014/20141225-2.aspx?searchID=1168560474&pg=1

22.

Lattery

Park Bartolone

Saunders

(2013). Optimizing surveys for smartphones: Maximizing response rates while minimizing bias. Presented at CASRO Online Research Conference, March 6–8, San Francisco, CA.

23.

Liu

(2017). Web survey experiments on matrix questions. Computers in Human Behavior, 67, 61–72.

24.

Liu

Cernat

(2018). Item-by-item versus matrix questions: A web survey experiment. Social Science Computer Review, 36, 690–706.

25.

Mavletova

Couper

M. P.

(2013). Sensitive topics in PC web and mobile web surveys. Survey Research Methods, 7, 191–205.

26.

Mavletova

Couper

M. P.

(2015). A meta-analysis of breakoff rates in mobile web surveys. In Toninelli

Pinter

de Pedraza

(Eds.), Mobile research methods: Opportunities and challenges of mobile research methodologies (pp. 81–98). London, England: Ubiquity.

27.

McClain

Crawford

S. D.

(2013). Grid formats, data quality, and mobile device use: A questionnaire design approach. Presented at the AAPOR Annual Conference, May 16–19, Boston, MA.

28.

Meredith

(1993). Measurement invariance, factor-analysis and factorial invariance. Psychometrika, 58, 525–543.

29.

Peterson

Griffin

LaFrance

(2017). Smartphone participation in web surveys: Choosing between the potential for coverage, nonresponse, and measurement error. In Biemer

Eckman

Edwards

de Leeuw

Kreuter

Lyberg

Tucker

West

(Eds.), Total survey error in practice (pp. 203–233). New York, NY: Wiley.

30.

Peytchev

(2005). Web survey design: Effect of layout on measurement error. In van Dijkum

Blasius

Durand

(Eds.), Recent developments and applications in social research methodology. Proceedings of the RC33 Sixth International Conference on Social Science Methodology, Amsterdam. Retrieved from http://konference.fdvinfo.net/rc33/2004/Data/PDF/stream_11-02.pdf

31.

Phillips

Luks

(2016, 5). Measuring survey behavior of smartphone users. Paper presented at the annual conference of the American Association for Public Opinion Research, Austin, TX.

32.

Poynter

(2001). A guide to best practice in online quantitative research. In Westlake

Sykes

Manners

Rigg

(Eds.), The challenge of the Internet; Proceedings of the ASC International Conference on Survey Research Methods (pp. 3–19). London, England: Association for Survey Computing.

33.

Revilla

(2017). Analyzing survey characteristics, participation, and evaluation across 186 surveys in an online opt-in panel in Spain. Methods, Data, and Analysis, 11, 135–162.

34.

Revilla

Couper

M. P.

(2017). Comparing grids with vertical and horizontal item-by-item formats for PCs and smartphones. Social Science Computer Review. doi:10.1177/0894439317715626

35.

Revilla

Toninelli

Ochoa

(2017). An experiment comparing grids and item-by-item formats in web surveys completed through PCs and smartphones. Telematics and Informatics, 34, 30–42.

36.

Schwartz

S. H.

(1992). Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. Advances in Experimental Social Psychology, 25, 1–65.

37.

Steenkamp

J. E. M.

Baumgartner

(1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78–107.

38.

Stern

Sterrett

Bilgen

(2016). The effects of grids on web surveys completed with mobile devices. Social Currents, 3, 217–233.

39.

Thomas

R. K.

Barlas

F. M.

Graham

Subias

(2015). What they can’t see can hurt you: Improving grids for mobile devices. Paper presented at the Annual Conference of the American Association for Public Opinion Research, Hollywood, FL.

40.

Toepoel

Das

Van Soest

(2009). Design of web questionnaires: The effects of the number of items per screen. Field Methods, 21, 200–213.

41.

Tourangeau

Couper

M. P.

Conrad

F. G.

(2004). Spacing, position, and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68, 368–393.

42.

Uslaner

E. M.

(2002). The moral foundations of trust. Cambridge, MA: Cambridge University Press.

43.

Vandenberg

Lance

C. E.

(2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–70.

44.

Wojtowicz

(2001). Designing lengthy Internet questionnaires: Suggestions and solutions. In Westlake

Sykes

Manners

Rigg

(Eds.), The Challenge of the Internet; Proceedings of the ASC International Conference on Survey Research Methods (pp. 25–32). London, England: Association for Survey Computing.

45.

Yamagishi

(1994). Trust and commitment in the United States and Japan. Motivation and Emotion, 18, 129–166.

46.

Yan

(2005). Gricean effects in self-administered surveys (Doctoral dissertation). University of Maryland, College Park, MD. Retrieved from http://drum.lib.umd.edu/bitstream/1903/3080/1/umi-umd-2890.pdf

47.

Yan

Tourangeau

(2008). Fast times and easy questions: The effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology, 22, 51–68.

48.

York

(2017, 7). The journey from device agnostic to mobile first. Paper presented at the European Survey Research Association conference, Lisbon, Portugal. Retrieved from https://www.slideshare.net/SueYork2/sue-york-esra-presentation-lisbon-july-2017

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.34 MB

Condition	Format		Device		Format × Device
Condition	Grids	Item-by-Item	PC	Mobile	PC Grids	PC Item- by-Item	Mobile Grids	Mobile Item-by-Item
Wave 1	832	846	866	812	574	292	258	554
Wave 2	522	557	540	539	347	193	175	364

	Format				Device			Format × Device
Condition	Overall	Grids	Item-by-Item		PC	Mobile		PC Grids	PC Item-by-Item	Mobile Grids	Mobile Item-by-Item
Wave 1				t (df = 1,676)			t (df = 1,676)					F(3, 1674)
Full survey	19.9 (9.8)	18.8 (9.2)	20.9 (10.3)	4.48***	19.1 (9.8)	20.8 (9.8)	3.58***	18.9 (10.0)	19.4 (9.4)	18.7 (6.9)	21.7 (10.7)	10.26***
Target question sets	7.6 (4.1)	6.7 (3.6)	8.6 (4.2)	9.88***	7.1 (4.0)	8.2 (4.0)	5.96**	6.7 (4.0)	7.7 (4.0)	6.6 (2.8)	9.0 (4.3)	39.87***
Wave 2				t(df = 1,077)			t(df = 1,077)					F(3, 1075)
Full survey	17.7 (9.8)	16.8 (9.0)	18.7 (10.4)	3.20**	16.8 (9.8)	18.7 (9.7)	3.30***	16.5 (9.9)	17.3 (9.6)	17.4 (6.8)	19.4 (10.7)	5.66***
Target question sets	7.3 (4.1)	6.3 (3.8)	8.1 (4.2)	7.25***	6.7 (4.2)	7.8 (3.9)	4.62***	6.2 (4.2)	7.5 (4.3)	6.6 (2.9)	8.4 (4.1)	20.20***

Model	Wave 1		Wave 2
Model	β	Odds Ratios	β	Odds Ratios
Intercept	−1.51*** (0.18)	0.22*** [0.16, 0.31]	−1.39*** (0.23)	0.25*** [0.16, 0.38]
Grids (vs. item-by-item format)	0.30** (0.14)	1.34** [1.11, 1.63]	0.32* (0.16)	1.38* [1.09, 1.73]
PC (vs. mobile)	0.13 (0.10)	1.14 [0.94, 1.38]	−0.14 (0.12)	0.87 [0.69, 1.09]
Males	0.28** (0.09)	1.32** [1.10, 1.59]	−0.15 (0.11)	1.17 [0.93, 1.46]
Age	0.01 (0.00)	1.01 [1.00, 1.02]	0.01 (0.01)	1.01 [1.00, 1.02]
Model	Likelihood ratio χ²(4) = 30.83, p < .001		Likelihood ratio χ²(4) = 14.19, p < .01