Left Feels Right

Introduction

When designing web questionnaires, survey practitioners need to make numerous decisions about the visual presentation of the questions. Earlier research has shown that these decisions can have profound effects on responses (e.g., Couper, 2008; Dillman, Smith, & Christian, 2009). One very basic design choice concerns the format and layout of the response options. Typical formats of response options in web surveys include radio buttons, check boxes, drop-down boxes, slider bars, and text boxes. Visual layout of these formats can affect how respondents perceive and answer the questions (e.g., Couper, Tourangeau, Conrad, & Crawford, 2004; Tourangeau, Couper, & Conrad, 2004, 2007).

A question that is yet unanswered is whether the answer boxes (i.e., radio buttons and check boxes ¹ ) should be placed to the left or to the right of the answer options (i.e., the answer text ² ) in closed questions with vertically arranged response categories. Placement to the right has been argued to be more natural and logical, because the answer boxes appear after the words or phrases to which they correspond (Jenkins & Dillman, 1995). Placing the answer boxes to the left instead would require respondents to first perceive the input field, move to the right to read the answer text, and then move back to the left to mark an answer. However, it has also been argued that placement to the right may increase the distance between the answer text and the input field, making it more difficult for respondents to find the correct answer box (Couper, 2008). To alleviate this potential problem, one would have to right align the answer options. However, this layout may also make it more difficult for respondents to navigate through the different answer options and thereby introduce another problem. Finally, cognitive interviews have demonstrated that most respondents are not aware of the position of the answer boxes and do not have a clear preference for either layout (Bowker & Dillman, 2000; Dillman, Carley-Baxter, & Jackson, 1999). All in all, to date “there is no strong empirical evidence supporting placing the input fields to the left or right of the response options in a vertically aligned response set” (Couper, 2008, p. 177).

In this article, we aim to provide empirical evidence about this issue. The criterion we use to evaluate the two placements against is the amount of cognitive effort required to select an answer. The literature on human-computer interaction consistently stresses the importance of reducing the effort required for interacting with a computer in order to improve the experience and enhance usability and comprehension (e.g., Shneiderman, 1992). Hence, the layout that makes it easier for respondents to select an answer is considered to be superior in terms of usability.

In principle, both placements of the answer boxes may facilitate answering, depending on the ways in which web survey respondents process survey questions with vertically arranged response categories. Two aspects of answer behavior seem particularly relevant in this respect: (1) whether respondents decide about each answer option immediately after reading it or only after reading all or at least several of the other options and (2) whether respondents use the mouse pointer as a reading aid. From a usability perspective, the crucial point is whether respondents’ eyes and/or mouse pointers are closer to the left or to the right of the answer text before they select an answer box. Placement of the answer boxes to the right of the answer text would reduce the effort of selecting an answer if respondents show the following behavior:

A. Respondents decide about each answer option immediately after reading it (so that their eyes are closer to the right and thus closer to the answer box when they decide whether to select it or not).

OR

On the other hand, placing the input fields to the left of the answer text would reduce the effort of answering if respondents show the following behavior:

D. Respondents decide about selecting an answer option only after they have read some or all of the other options; then they scan the options they have read previously by reading part of the options (probably only the beginning) and select one (so that their eyes are closer to the middle or left side of the answer text when they decide whether to select the option or not).

OR

If the position of answer boxes is not in accord with the way respondents process the questions, they should experience more cognitive effort.

To examine whether responding to web survey questions with vertically arranged response categories is better described by one or another of these principles, and consequently whether placing the answer boxes to the left or to the right of the answer text reduces the cognitive effort of answering these types of questions, we conducted an eye-tracking experiment. Besides providing a direct window into the ways in which respondents process web survey questions, collecting eye-tracking data also enabled us to analyze relatively direct measures of cognitive effort, such as fixation times and fixation counts (cf., Galesic & Yan, 2011). In eye-tracking studies, longer fixation times and higher numbers of fixations are usually associated with increased cognitive effort (Rayner, 1998). Hence, the layout that produces shorter and fewer fixations is considered to be superior in terms of processing ease.

In this study, we adopt two common assumptions about eye movements: the immediacy assumption and the eye-mind assumption (Just & Carpenter, 1980; Rayner, 1998). The immediacy assumption posits that readers try to interpret every word or visual object as soon as they encounter it. The eye-mind assumption states that the eyes remain fixated on a word or an object as long as it is being processed. Taken together, these assumptions suggest that there is a close connection between fixation times and processing duration: the time spent fixating a word or an object is (more or less) equal to the time it is being processed.

Methods

Design

The eye-tracking experiment reported in this article was conducted in October and November 2012 at the pretest laboratory of GESIS—Leibniz Institute for the Social Sciences in Mannheim, Germany, and was part of a larger study with several unrelated experiments (cf., Neuert & Lenzner, 2013). All experiments were independently randomized to reduce the possibility of any systematic carryover effects. The whole study took about 1.5 hours of which 30 min were devoted to eye tracking and 60 min were devoted to cognitive interviewing. The present experiment was embedded in a web questionnaire that participants completed after participating in a cognitive interview during the second half of the study. Respondents were randomly assigned to one of three question layouts with answer boxes appearing to the left of left-aligned answer options (n = 25), answer boxes appearing to the right of left-aligned answer options (n = 25), or answer boxes appearing to the right of right-aligned answer options (n = 25; Figure 1). The layout with right-aligned answer options was included in the design to allow for an interpretation of the position of the answer boxes independent of the space between the answer text and the answer boxes.

OPEN IN VIEWER

Figure 1.

Screenshots of Q4 for the three layout designs.

To investigate the first aspect of answer behavior (i.e., whether respondents decide about each answer option immediately after reading it or only after reading and rereading several of the other options), we coded the eye-tracking videos for (a) the number of answer options respondents read before they selected an answer box, (b) the number of answer options they reread before they selected an answer box (i.e., to which they returned to after reading at least one other option), and (c) the number of answer options they read after the one they would later select.

To investigate the second aspect of answer behavior (i.e., whether respondents follow the text with the mouse pointer while reading, keep it stationary near the input fields, or do not use it as a reading aid), we coded the position of the mouse pointer (a) when the question appeared on the screen and (b) during reading (see Table 1 for the list of all codes). As indicators of cognitive effort, we collected response latencies, recorded respondents’ fixation times and counts on the answer text and the answer boxes, and coded the number of gaze switches between answer text and answer boxes. The latter indicates the respondent effort to match an answer box to the corresponding answer text.

OPEN IN VIEWER

Table 1.

Reading Patterns, Codes, and Interrater Agreement.

Reading Patterns	Code	Agreement Percent	ICCa/κb
No. of answer options read before selecting an answer box	No. of options	93.0	.99
No. of answer options that were reread	No. of options	94.0	.96
No. of answer options read after the one respondents would later select	No. of options	93.0	.99
Position of mouse pointer when question appeared on screen	1 = top left, 2 = top center, 3 = top right, 4 = middle left, 5 = middle center, 6 = middle right, 7 = bottom left, 8 = bottom center, 9 = bottom right	94.0	.90
Position of mouse pointer while reading an answer option	0 = option not read, 1 = on answer box for that option, 2 = on answer box for some other option, 3 = on answer text for that option, 4 = on answer text for some other option, 5 = elsewhere on page, 6 = tracking answer text of that option with mouse pointer	95.9	.94
No. of gaze switches between answer text and boxes	No. of switches	84.0	.93

Note. Interrater agreement is based on independent ratings of N = 100 questions. ^aIntraclass correlation coefficients (ICCs) were calculated for interval level data. ^bκ values were calculated for nominal level data.

All questions were coded by one coder, and a randomly selected subset of 30% (N = 100) were independently coded by a second coder for purposes of estimating reliability. Interrater agreement was excellent (cf., Fleiss, Levin, & Paik, 2003) with intraclass correlation coefficients (ICCs) ranging from .93 to .99 and κ values ranging from .90 to .94 (Table 1). Discrepancies between the two ratings were examined and discussed between the two coders until consensus was reached.

Results

Processing of Survey Questions

To examine how respondents processed the questions, we first looked at the number of answer options they read and reread before selecting one of the answer boxes, as well as at the number of options they read after the one they would later select. As a second aspect of respondent behavior, we examined the position of the mouse pointer while respondents were reading and answering the questions. We also examined whether there were differences in response distributions between the three conditions but found no such differences for any of the four questions.

Number of answer options read besides the one that was selected

In all four questions, respondents read most of the answer options before checking an answer box and also reread some of the answer options (Table 2). This behavior was particularly prominent in responses to the check-all-that-apply question (Q1), which asked respondents to select the three most relevant answers among a set of 13 answer options and hence required them to perform a comparative judgment. In responding to this question, many respondents first read most of the answer options from top to bottom. Then, they shortly scanned some of the options they had previously read and selected their answers (cf., behavior D discussed above). Answering the rating scale questions (Q2–Q4) required respondents to carry out a different task, namely, to select the one “correct” answer among a set of five options. Hence, in processing these questions, it would suffice to read only as many options as are listed before the one that is later selected. However, the eye-tracking data show that, for all four questions, respondents read additional answer options after having read the one they would later select (see last column of Table 2). With regard to this variable, it is possible that there might be different “types” of respondents: some reading additional answer options after the one they would later select and some deciding immediately whether to select it or not (cf., behavior A discussed above). However, we identified only seven respondents in our experiment who did not read any additional answer options in all four questions and merely nine respondents who did not read any additional answer options when answering questions Q2 to Q4. Hence, most respondents did not decide about each answer option immediately after reading it but after reading and rereading several of the other answer options as well.

OPEN IN VIEWER

Table 2.

Means and Standard Deviations of Question Processing Indicators.

Question	No. of Options Read Before Selecting an Answer Box		No. of Options Reread Before Selecting an Answer Box		No. of Options Read After the One Respondents Would Later Select
	M	(SD)	M	(SD)	M	(SD)
Q1a (13 options)	8.07	(5.72)	1.71	(2.62)	3.61	(4.26)
Q2 (5 options)	3.41	(1.16)	0.68	(0.95)	0.85	(0.93)
Q3 (5 options)	2.92	(1.14)	0.65	(0.86)	0.73	(0.78)
Q4 (5 options)	3.00	(1.17)	0.80	(1.09)	0.68	(0.83)

Note. M = mean; SD = standard deviation. ^aQ1 asked respondents to select three answer options. Respondents’ behavior was only analyzed for their first answer.

Position of the mouse pointer

When respondents first started to read the question, the mouse pointer was positioned most often in the middle region of the screen (for 80, 88, 96, and 92% of the respondents for questions Q1 to Q4, respectively), close to the spot where the “next button” of the previous page was located. Hence, respondents usually did not move the mouse pointer to any part of a question but kept it stationary while starting to read the new question. Tracking the text with the mouse pointer while reading (cf., behavior B discussed above) was rare in our recordings and occurred exclusively in Q1: only four respondents tracked the text in a total of 41 answer options while reading (Respondent 1: 12 options, Respondent 2: 11 options, Respondent 3: 13 options, Respondent 4: 5 options). Sometimes respondents kept the mouse stationary on the answer box of the particular option they were reading (cf., behavior C and E discussed above). For all answer options that were actually read, the average proportion of respondents who held the mouse pointer over the corresponding answer box was 10, 7, 8, and 8% for questions Q1 to Q4, respectively. In most cases, however, they kept the mouse stationary on some other region of the screen while reading the answer options (Q1: 85%, Q2: 93%, Q3: 93%, Q4: 92%). Overall, respondents did not use the mouse pointer as a reading aid.

Cognitive Effort

We examined the response latencies for the four questions, respondents’ fixation times and counts on the answer boxes and on the answer options, and the number of gaze switches between these two regions as indicators of cognitive effort (see Appendix B for means of cognitive effort indicators for each individual question).

Response latencies

We used JavaScript to collect client-side response latencies that were measured from the time a question appeared on the screen to the time respondents clicked on the submit button to receive the next question. Given that the distribution of response times was skewed, which is typical for this kind of data (cf. Yan & Tourangeau, 2008), we applied logarithmic transformations on the response latencies (cf. Fazio, 1990).

Across the four questions, respondents required more time to answer the questions when the boxes were placed to the right (of both left-aligned and right-aligned text) than when they were placed to the left of the answer options (Table 3). To examine the effects of the three different question layouts on response latencies, we conducted an analysis of covariance (ANCOVA) with the mean log-transformed response latency as dependent variable and respondents’ baseline speed as a covariate. The baseline speed was computed for each respondent separately, by averaging the speed of answering to two attitudinal questions asked directly after the experiment reported in this article (see Appendix C for question wording). This covariate was included in the analysis to control for interindividual differences in respondents’ speed of answering questions. The ANCOVA showed a marginally significant effect of the question layout on response latencies, F(2, 71) = 3.01, p = .056. Sidak post hoc tests revealed a marginally significant difference (p = .066) between the “left” condition and the “right” condition (i.e., when the boxes appeared to the right of left-aligned answer options), with respondents requiring less time to answer the questions in the former layout.

OPEN IN VIEWER

Table 3.

Means and Standard Errors (in Parentheses) of Cognitive Effort Indicators in the Three Conditions.

Cognitive Effort Indicators	Condition
	Left	Right	Right aligned
Raw response latencies (in sec.)	17.56+	20.04	21.76+
	(1.23)	(1.23)	(1.23)
Log-transformed response latencies	4.24+	4.31+	4.29
	(0.21)	(0.22)	(0.22)
Fixation times on answer boxes (in sec.)	1.46*	2.55*	2.04
	(0.27)	(0.27)	(0.27)
Fixation counts on answer boxes	3.14*	5.89*	4.41
	(0.62)	(0.62)	(0.63)
Fixation times on answer options (in sec.)	7.81	9.01	9.16
	(0.57)	(0.57)	(0.58)
Fixation counts on answer options	37.08	42.69	42.56
	(2.53)	(2.55)	(2.57)
No. of switches between answer options and answer boxes	5.48*	7.51*/+	6.12+
	(0.44)	(0.44)	(0.44)

Note. For response latencies, fixation times, and fixation counts the table reports estimated marginal means after controlling for the covariates respondent baseline speed, reading rate, and fixation rate, respectively.

*/⁺ = Sidak post hoc test shows that the difference between these conditions is significant at p < .05(*) and p < .10(⁺).

Fixation times and counts

Across the four questions, respondents fixated for a longer time and more often on the answer boxes if these were placed to the right of the answer text (for both the “right” and the “right-aligned” condition) than if they were placed to the left of the answer text (Table 3). To examine the effect of the placement of the answer boxes on fixation times and fixation counts, we conducted ANCOVAs of the means of the four questions with reading rate or fixation rate as covariates, respectively. These covariates were computed from the same two questions as the baseline speed. Reading rate refers to the average fixation time on these questions and fixation rate refers to the average number of fixations on these questions. Again, these covariates were chosen to control for interindividual differences in respondents’ reading rate and fixation rate.

Statistically significant effects were found for both the fixation times on the answer boxes, F(2, 71) = 4.19, p = .019, and the fixation counts on the answer boxes, F(2, 71) = 4.91, p = .010. Sidak post hoc tests showed that respondents required significantly shorter (p = .015) and fewer (p = .008) fixations on the answer boxes when the boxes appeared to the left of left-aligned answer options (“left” condition) than when they appeared to the right of left-aligned answer options (“right” condition). No significant between-group effects were found for fixation times and fixation counts on the answer options, F(2, 71) = 1.65, p = .199 and F(2, 71) = 1.60, p = .210, respectively. This finding is important because it reveals that the placement of the answer boxes to the right increases fixation times and counts on the answer boxes while it does not affect the depth of processing the answer options.

Number of gaze switches between answer text and answer boxes

Across the four questions, respondents made more gaze switches between the answer text and the answer boxes when the boxes were placed to the right (of both left-aligned and right-aligned text) than when they were placed to the left of the answer options (Table 3). An analysis of variance of the means of the four questions revealed a significant between-group effect, F(2, 72) = 5.69, p = .005. Sidak post hoc tests showed that respondents made significantly fewer gaze switches (p = .005) if the answer boxes appeared to the left of left-aligned answer options (“left” condition) than if they appeared to the right of left-aligned options (“right” condition). All in all, these findings indicate that respondents had more difficulties to identify the appropriate input field if the answer boxes were placed to the right (of both left-aligned and right-aligned answer options) than if they were placed to the left of the answer options.

Discussion

This eye-tracking study examined how respondents process and answer closed questions with vertically arranged response categories in web surveys and whether placing the answer boxes to the left of left-aligned answer text, to the right of left-aligned answer text, or to the right of right-aligned answer text makes it easier for respondents to select an answer. Our results show that most respondents do not use the mouse pointer as a response aid but keep it stationary in the middle region of the screen. In this regard, it does not make any difference whether the answer boxes are placed to the left or to the right of the response options: the mouse pointers are about equally close to the answer boxes before selecting an answer in all three conditions. However, with regard to respondents’ reading behavior, we found that most respondents read several answer options before scanning the options briefly again and selecting an answer (cf. behavior D discussed above). Hence, just before selecting an answer box, their eyes are usually closer to the left than to the right of the answer text. Placement of the answer boxes to the left of left-aligned answer text (“left” condition) therefore facilitates the response task by making it easier for respondents to select an answer. This reduced effort is indicated by shorter fixation times and fewer fixation counts on the answer boxes as well as fewer gaze switches between answer text and answer boxes in comparison to the layout, in which the boxes appear to the right of left-aligned answer text (“right” condition).

With respect to response latencies, we identified a marginally significant effect between the “left” and “right” conditions, with respondents requiring more time to answer the questions in the latter condition. Our eye-tracking data revealed that this effect was mostly driven by longer and more numerous fixations on the answer boxes and by a larger number of switches between the answer text and the answer boxes in the “right” condition. In contrast, no significant differences were found for the time respondents spent looking at the answer text, suggesting that the depth of processing the answer options did not differ between conditions.

With regard to our measures of cognitive effort, it is important to note that longer response latencies, longer fixation times, higher numbers of fixations, and higher numbers of gaze switches between answer text and answer boxes do not necessarily indicate processing difficulties. In principle, these measures could also indicate a deeper processing and a more conscientious response style. However, as was mentioned previously, we did not find statistically significant differences in the fixation times or counts on the answer options between the conditions, and hence the longer fixation times did not result from a more careful evaluation of the answer text. In contrast, our findings indicate that the additional time is spent on finding the appropriate answer box.

Our results indicate that the superiority of the placement of answer boxes to the left cannot be explained by the shorter distance between the boxes and the answer text alone. On one hand, the cognitive effort required by the “right-aligned” condition (in which the answer boxes were placed to the right of right-aligned text) was lower than the effort required by the “right” condition (in which the boxes were to the right of left-aligned text). This indicates that the distance between the boxes and the answer options indeed should be kept as short as possible as it influences respondent burden. On the other hand, cognitive effort required by the “left” condition was even lower than the effort required by the “right-aligned” condition. This finding suggests that in addition to keeping the distance between boxes and answer options short, it would be advisable to left align the answer text, because this layout is more in accord with the ways in which respondents process survey questions with vertically arranged response categories.

Respondent effort could also be reduced by shading every other row and thereby visually connecting the answer options and their corresponding answer boxes. This would make it easier for respondents to find the appropriate answer box. However, we would still expect to find the same pattern across the three layouts, except that overall response times and fixation times would be shorter. The eyes of the respondents would still be closer to the left than to the right of the answer text before they decide to select and answer. Hence, shading might attenuate but not eliminate the effects of placement and proximity. However, this notion clearly calls for future studies that systematically explore the interactions between placement, proximity, and shading.

There are some limitations to this study which suggest additional directions for future research. First, our findings are restricted to closed web survey questions (both single-choice and check-all-that-apply) with vertically arranged answer options. Although these types of questions probably form a large part of the questions asked in social science research, further research is needed to examine whether our findings also generalize to other kinds of questions, such as forced-choice questions, for example, in which respondents are asked to provide an answer (e.g., yes/no) for each item in a list. Second, our participants answered the questions in a laboratory while their eyes were being recorded so there is the possibility that they were more conscientious than they would have been in a more private and natural environment. Earlier studies have shown that even when answering questions in front of an eye tracker in the laboratory, respondents often skip some parts of the question text or do not read all of the answer options (Galesic, Tourangeau, Couper, & Conrad, 2008; Graesser, Cai, Louwerse, & Daniel, 2006). We found the same sort of “satisficing” behavior (Krosnick & Alwin, 1987) in our data, so we can at least assume that our participants did not completely change their usual answer behavior. Nevertheless, we cannot rule out the possibility of a laboratory effect, and we encourage future studies to examine whether our findings can be replicated outside of the laboratory. This could be done, for example, by analyzing the response times of these different question layouts when they have been implemented in a regular web survey. Third, our current data do not enable us to examine the quality of the answers obtained by the three different question layouts. Although our findings reveal that placing the answer boxes to the left of left-aligned answer text reduces the cognitive effort for respondents, it remains unclear whether this reduced effort also results in more reliable and valid responses. Given that these measures are the ultimate criteria for judging the quality of a survey question, future research is needed to examine which of the three question layouts produces the most reliable and valid data.

This research can also be viewed in the light of common design choices in different modes of self-administered surveys and mixed-mode surveys. In our experience, most web surveys are designed with left-aligned text and answer boxes on the left side, whereas paper-based surveys commonly implement both right-aligned and left-aligned answer boxes (with left-aligned text, however). Examples of different positions of answer boxes in self-administered surveys can be found in the ISSP 2011 source questionnaire (primarily right aligned) and its German implementation (primarily left aligned) as well as in the U.S. American Community Survey 2013 (left aligned) and the German census 2011 household questionnaire (right aligned). Considering our data and the goal of unimode design in mixed-mode studies (Dillman et al., 2009), it seems advisable to left align the answer boxes both on paper and on the Web.