The Influence of the Answer Box Size on Item Nonresponse to Open-Ended Questions in a Web Survey

Abstract

This article investigates item nonresponse in open-ended survey questions because such item nonresponse is much higher than in closed questions. The difference is a result of the higher cognitive burden placed on the respondent. To study item nonresponse, we manipulate different questionnaire design characteristics, such as the size of the answer box and the inclusion of motivation texts, as well as respondent-specific characteristics, in a randomized web experiment using a student sample. The results show that a motivation text increases the frequency of responses to open-ended questions for both small and large answer boxes. However, large answer boxes earn higher item nonresponse than small answer boxes regardless of the usage of a motivation text. In addition, gender and the respondent’s field of study affected the answering of open-ended questions; being a woman or studying social sciences increased the frequency of a response. As the major finding and in contrast to previous findings, our results indicate that particularly large answer boxes should be avoided, because they reduce respondents’ willingness to respond.

Keywords

open-ended questions item nonresponse answer box size motivation demographic variables

The use of open-ended questions in surveys is helpful to gain additional, more sophisticated information from respondents. Although many surveys use open-ended questions in combination with closed questions, item nonresponse to open-ended question in surveys is much higher than that to closed questions (Andrews, 2005; Reja, Manfreda, Hlebec, & Vehovar, 2003; Scholz & Zuell, 2012). This is due to the answering process being more cognitively demanding for open-ended questions. The cognitive process associated with answering survey questions has been addressed by many researchers (see, e.g., Krosnick, 1999; Schwarz & Strack, 1985; Tourangeau & Rasinski, 1988; Tourangeau, Rips, & Rasinski, 2000). Open-ended questions are associated with a higher burden by respondents and call for more active involvement in the interview than are typical closed questions, because no predefined categories are presented and the respondent has to verbalize the answer.

Although few studies have investigated response behavior of open-ended questions in face-to-face or paper interviews, there is an increase in research with respect to open-ended questions in online surveys (see, e.g., Denscombe, 2008). Many studies investigating response behavior to open-ended questions have focused on measuring answer quality in terms of length of answers and number of topics (see, e.g., Barrios, Villarroya, Borrego, & Ollé, 2010; Denscombe, 2008; Reja et al., 2003; Smyth, Dillman, Christian, & McBride, 2009), while few studies have addressed item nonresponse (Emde & Fuchs, 2012; Smyth et al., 2009). Smyth, Dillman, Christian, and McBride (2009) tested the effects of several questionnaire design features of open-ended questions in online surveys on both quality of answers and item nonresponse in three different studies. The first study addressed the effect of the size of the answer box in combination with or without a remark that “you are not limited in the length of your answer by the size of the box.” The second study investigated different answer box sizes and the use of motivation texts. The third study considered motivation texts that stress the importance of the study and texts encouraging respondents to “take their time.” The authors found that both motivation texts and instructions to “take your time” increased the length of the answers and the number of reported topics. With respect to item nonresponse, the authors reported different results for different studies. In the first study, the size of answer boxes did not affect item nonresponse. In the second study, presenting a motivation text led to longer answers and to an increase in item nonresponse. In their third study, the authors found that stating the importance with and without a prompt to “take your time” significantly decreased item nonresponse among early respondents. None of the introductions affected item nonresponse among late respondents in the third study. In sum, Smyth et al. (2009) found that the size of response boxes does not impact the item nonresponse, whereas motivation texts may.

Similar to Smyth et al. (2009), Emde and Fuchs (2012) analyzed the effects of different types of answer boxes on the length of answers and on item nonresponse. Emde and Fuchs (2012) used small, large, and different dynamically sized answer boxes. Emde and Fuchs (2012) verified the findings of Smyth et al. (2009) who found no reduction in item nonresponse when the answer box size was increased.

The studies by Smyth et al. (2009) and Emde and Fuchs (2012) provide evidence regarding the main possibilities to increase responses to open-ended questions when developing web surveys. However, these studies did not consider other relevant factors, such as the respondents’ demographic characteristics and motivation. Taking such variables into account may help to explain the results or to identify relevant subgroups of respondents that would experience different effects in response to changes in the questionnaire design.

With respect to demographic variables, there is evidence that the gender of the respondent may affect response behavior. Denscombe (2008) found in his web survey that women provide longer answers to open-ended questions than men do. In addition, education of respondents may be a relevant variable. Although all respondents are able to answer open-ended questions (Geer, 1988), it is well known that better educated respondents answer more often and provide longer answers to open-ended questions than less educated respondents, because of different burdens to formulating an answer (see Holland & Christian, 2009; Scholz & Zuell, 2012).

In addition to demographic variables, motivation might play an important role. Many researchers reported that the interest in a topic has a strong positive effect on responses to open-ended questions (see, e.g., Geer, 1991; Holland & Christian, 2009): respondents interested in a topic are much more likely to answer the question. In addition, some researchers reported that they found a negative bias in the answering behavior to open questions about job satisfaction in employee surveys: unsatisfied respondents answer more often and with longer answers to open-ended questions about situations eliciting negative feelings than satisfied respondents do (see, e.g., Borg & Zuell, 2012; Poncheri, Lindberg, Foster Thompson, & Surface, 2008).

In sum, previous research found no increase in item nonresponse by the size of answer boxes, while mixed results were obtained with regard to motivational texts. In addition, respondents’ characteristics such as demographic variables and motivation were not considered in this research. In our article, we examine how response box size, motivational texts, and respondents’ characteristics might influence the willingness to provide a response to open-ended questions in web surveys. This is a departure from the literature because most studies examining answering behavior to open-ended questions focus on measuring answer quality in terms of length of answers and number of topics (e.g., Denscombe, 2008; Reja et al., 2003) and do not address item nonresponse. However, item nonresponse should also be considered, since elaborate (high quality) responses would be less useful for researchers when item nonresponse was high. Nonresponse in the case of open-ended questions of more than 20% is reported in different studies (see, e.g., Andrews, 2005; Scholz & Zuell, 2012), which is also a relevant source of measurement error. Consequently, an important task of researchers is not only to foster a high quality of responses but also to motivate respondents to answer open-ended questions. Next, as it was shown earlier, it is important to simultaneously address the effects of motivation texts, answer box size, and respondents’ demographic and motivation variables on item nonresponse to open-ended questions. The aim of the present study is to fill this gap. With respect to the size of the answer box, we seek to obtain the result found by Smyth et al. (2009) and Emde and Fuchs (2012), namely that large answer boxes do not differ from small answer boxes with respect to item nonresponse. We expect this result to be robust and independent of the impact of motivation texts. The corresponding hypothesis is:

Hypothesis 1: Providing large answer boxes does not increase item nonresponse to open-ended questions.

Next, we expect to confirm the results that demonstrate a positive effect of motivation texts on response rates as it was found by Smyth et al. (2009). We also expect that this result will not be affected by the size of the answer box. Our corresponding hypothesis is:

Hypothesis 2: Providing motivation texts decreases item nonresponse to open-ended questions.

Our next extension of the studies described (Emde & Fuchs, 2012; Smyth et al., 2009) is to consider the effect on item nonresponse of different respondent-specific variables, namely the demographic characteristics of the respondents and their positive or negative attitudes toward the subject addressed by the open-ended question (as it was also investigated by Borg and Zuell, 2012). The respondents’ attitudes reflect the respondents’ motivation to answer. Our hypotheses concerning these variables are:

Hypothesis 3: Demographic characteristics correlate with the item nonresponse in the case of open-ended questions: being a woman increases the frequency of responses to the open-ended question. Indicators of higher education are also expected to positively correlate with the item nonresponse.

Hypothesis 4: Motivation factors are expected to correlate with item nonresponse, for example, being unsatisfied with the subject addressed by the open-ended question is expected to be associated with a lower item nonresponse. Variables related to the interest on the topic of the survey are also expected to be associated with the responses to the open-ended question, that is, in the case of higher interest less item nonresponse is expected to be observed.

Method and Data

We used data from a web survey conducted at the Justus-Liebig University of Giessen in January 2012. A platform (Survey Monkey) provided at the university was used, which offers students and lecturers the opportunity to conduct web surveys in the context of their learning and research activities. The sample was a nonprobability sample of 743 university students. The goal of the study was not to generate a description of the population (where a random sample is crucial) but to determine the effect of the experimental manipulation (where a random assignment of respondents to the experimental groups is crucial for internal validity; Shadish, Cook, & Campbell, 2001). Of the respondents, 76.6% were female and 24.4% were male. The average age of the students was 24.5 years (standard deviation [SD] = 3.9); the average number of semesters enrolled was 5.6 semesters (SD = 4.17). In the sample, most of the university faculties were present: law (4%), medicine (4%), economy (5%), social and cultural sciences (16%), history (6%), language (16%), psychology and sport (8%), physical science (9%), biology (8%), agricultural sciences (16%), and medicine (10%).

The topic of the survey was satisfaction of the students with their studies and the university. The students of the university were contacted by e-mail and asked to participate in the survey. In this e-mail, a link to the survey was provided. The survey was conducted completely anonymously; access to the survey was not restricted.

The questionnaire began with questions about how satisfied the students were with the university and their studies, followed by some special questions about the reasons for being satisfied or not, the levels of satisfaction with some aspects of their university study, the reason why they decided to study, and how easy or difficult the courses are. This portion of the questionnaire was the same for all respondents and used only closed questions. The next question in the questionnaire was an open-ended question concerning the conditions experienced by the students at the university. This question addressed students’ ideas concerning better support for their studies. After this open-ended question, closed questions about demographic characteristics were posed. We applied a 2 × 2 randomized experimental design and varied the size of the answer box and the inclusion of a motivation text for the open-ended question. In the first experimental group (G1, n = 176), we did not present a motivation text and used a small answer box of 5 lines in height and about 12 cm in width. The same small answer box was assigned to Group 2 (G2, n = 170). Prior to answering the open-ended question, this group was primed with a reason for asking the open-ended question and with the importance of their responses for the study (“The following question is very important for my work and helps you to express your opinion and wishes. Please take your time to answer the question”). This motivation text was posed directly below the open-ended question in the form of an instruction on the same page. Large answer boxes (30 lines and width 12 cm) were assigned for Groups 3 and 4 (G3, n = 159 and G4, n = 142); the motivation text was also assigned to Group 4. For the randomized assignment the standard procedure implemented in Survey Monkey was used. The experimental groups did not differ in terms of gender, age, or number of semesters enrolled, multivariate analysis of variance: Wilks-Lambda F(9, 1458) = 0.56, p > .10, nor in terms of faculty, χ² _{(33, N = 623)} = 36.69, p > .10.

To test our hypotheses, we used binary logistic regression to analyze the simultaneous effects of all predictors. The dependent variable was the dichotomously measured response (or lack thereof) to the open-ended question. For predictors, we used two factors of the questionnaire design: answer box size (0 = small; 1 = large) and motivation text (0 = without; 1 = with) as well as the interaction term between these two factors (answer box size × motivation text) to test Hypotheses 1 and 2. To test Hypothesis 3, we used the demographic variables of gender and number of terms enrolled. We did not use variables such as age and formal school education because they do not vary considerably within the typical student population (which was also the case in our data). However, the number of semesters enrolled was considered as a variable with respect to students’ education level. To test Hypothesis 4, we considered the field of the study and general satisfaction with the university, which was measured with one question using a 5-point rating scale ranging from very satisfied to very unsatisfied. The field of the study was relevant, since it could be related to students’ interest in the survey topic and to the students’ motivation to respond to the open-ended question. Our first analysis with the data of the present study showed only a difference in responding to the open-ended question between social science students and students in other fields. Therefore, we used a dummy variable (social sciences yes/no) in the analysis reported subsequently to address the possible motivational differences, which may result from the field of the study. Finally, we considered the item nonresponse of the preceding closed-ended questions. It was important to consider a general tendency of respondents to answer survey questions, which could affect our experimental manipulation, because respondents who are less likely to answer the closed-ended questions would also tend not to answer the open-ended question. The predictor variables were entered into the analysis block wise in two steps because we considered two different types of influences on the response behavior: We started with the variables associated with the questionnaire design modifications followed by the variables on respondent characteristics.

Results

The descriptive statistics (in percentage) with respect to the item nonresponse to the open-ended question in the four experimental groups are reported in Table 1. It can be seen that the respondents answered the open-ended question more often in the case of small answer boxes (G1 and G2) than in the case of large answer boxes (G3 and G4). In addition, the frequencies are higher for both small and large answer boxes when the motivation text was used.

Table 1.

Distribution of Nonresponse in the Four Groups.

	Open-Ended Question
	Not Answered, %	Answered, %
G1 small box, no motivation text	38.6	61.4
G2 small box, motivation text	31.2	68.8
G3 large box, no motivation text	56.0	44.0
G4 large box, motivation text	45.1	54.9

Note. χ² = 22.21; df = 3; N = 743; p < .001

Next, logistic regressions were run (see earlier) to test a simultaneous effect of answer box size and motivation text with a control of respondents’ characteristics. The results of the logistic regression are reported as odds ratios to assess the effect of the variables and Nagelkerke’s pseudo-R ² for the model fit. In our interpretation of the results of the logistic regression reported in Table 2, we concentrate on the significant coefficients displayed in boldface.

Table 2.

Determinants of Responses to the Open-Ended Questions.

	Model 1 (Odds Ratios)	Model 2 (Odds Ratios)
Questionnaire design
Answer box size (0 = small)	.50**	.50**
Motivation text (0 = no)	1.39	1.72*
Answer box size by motivation text	1.12	1.03
Respondent-specific characteristics
Gender		1.94***
Closed questions		.83
General satisfaction		1.17
Field of study		1.50*
N of university terms		1.02
Nagelkerke’s R ²	.046***	.098***

Note. Significance: *p ≤ .05. **p ≤ .005. ***p ≤ .001.

In Model 1, we entered the variables based on questionnaire design: the size of the answer box (small vs. large) and motivation texts (with and without) as well as the interaction term as represented by the four groups. The results in Table 2 show that there was a significant main effect of answer box size, while the propensity to respond to the open-ended question was halved for the large answer box group as compared with the small answer box group. The effect of motivation text and the interaction between answer box size and motivation text did not gain significance in Model 1. Although Nagelkerke’s pseudo-R ² of the first model was not very high (.046), the regression showed that the size of answer boxes has an effect on responses to the open-ended question.

In Model 2, we entered the respondent-specific variables. Nagelkerke’s pseudo-R ² remarkably increased in Model 2 to a value of .098. The effect of the response box size did not change but the effect of motivation text gained on significance. It was observed that providing motivation text increases the propensity to respond to the open-ended question. The interaction between the answer box size and motivation text remained nonsignificant. Regarding the demographic variables, it was found that gender played a significant role when answering the open-ended question: women answered the open-ended question more often than men did. In contrast, the variable “number of terms enrolled” did not significantly influence the providing of responses to the open-ended question.

Next, field of study had a minor significant influence: Respondents studying social sciences answered open-ended question more often than respondents studying in other fields. However, general satisfaction with the university and studies was not a significant predictor of the dependent variable, neither was the effect of item nonresponse for closed questions significant.

Discussion and Conclusion

In summary, the effects of questionnaire design such as answer box size and presence of motivating texts have a significant impact on the likelihood of response to open-ended questions. Additionally, the gender of the respondents and their field of study play a role in nonresponse to the open-ended question.

Our first hypothesis, which we derived from findings by Smyth et al. (2009) and Emde and Fuchs (2012), could not be confirmed; in terms of rates of item nonresponse, the large answer boxes significantly decrease the number of responses to the open-ended question. Thus, large answer boxes discouraged the respondents from providing responses to the open-ended question and this effect was visible independent of the presence of a motivation text. Obviously, large answer box size discourages the respondent from filling in the answer because of the attempt to try to avoid the burden of a longer response that is associated with the larger answer box. However, in our experiment, we tested the differences between two different box sizes - a rather small box and a relative large box. It would be interesting to know which box size would be particularly crucial for the item nonresponse if different box sizes are used to represent a graduated continuum of possible box sizes. Therefore, the results of our study would be generalized as follows: particularly large box sizes (such as those used in our study) seem to have a discouraging effect when an open-ended question has to be answered.

With respect to the expectations of the second hypothesis, the motivation text increased the frequency of responses to the open-ended question in the case of both small and large answer boxes. Thus, our second hypothesis could be confirmed. However, the effect of the motivation text was visible only if respondents’ characteristics were considered as explanatory variables. Next, in the case of a large answer box, the frequency of responses to the open-ended question is lower than in the case of small answer box, even if a motivation text is provided for the large answer box but not for the small answer box.

In addition to these changes in questionnaire design, respondents’ characteristics played a role in the rate of response to the open-ended question; being a woman increased the frequency of responses. This aligned with our Hypothesis 3. With respect to the number of semesters enrolled (a variable we assumed to represent a varying level of education in the student sample), no effect on item nonresponse to the open-ended question was found. However, since it was only one indicator for education and the students are rather homogeneous with respect to the education level, it could not be concluded from our results that education variables do not affect item nonresponse to open-ended questions. Therefore, more studies with populations which are heterogeneous with respect to education are needed.

As a variable that would be related to the motivation to respond to the open-ended questions, we found that studying social science increased response propensity. However, we found that general satisfaction with the university did not significantly predict answering the open-ended question, unlike other studies in the context of job motivation (Borg & Zuell, 2012; Harman, 2011; Poncheri et al., 2008) that reported unsatisfied employees answer the open-ended question more often when the question pertains to problems to be solved or proposals for progress. Thus, our fourth hypothesis could be only partially supported.

In sum, our experimental research design clarified the effects of the relevant characteristics of questionnaire design, such as answer box size and motivation text, on item nonresponse in the case of open-ended questions. In addition, the consideration of several of respondents’ characteristics demonstrated a significant effect of motivation text in the analysis. These results show that using both small answer boxes and motivation texts can encourage respondents to provide answers to open-ended questions. In contrast to previous findings (Emde & Fuchs, 2012; Smyth et al., 2009), our results indicate that particularly large answer boxes should be avoided because they reduce respondents’ willingness to respond.

One limitation of our experiment is that a student sample was used (like Emde & Fuchs, 2012; Smyth et al., 2009). Therefore, the results should be confirmed with samples that are more representative of the population. Compared with respondents in population surveys with probability samples, our respondents would be biased with respect to their higher motivation to respond to the questionnaire and high cognitive skills, both of which are linked to their advanced education. Thus, the discouraging effect of large answer boxes could be even greater in other respondents’ groups with a lower motivation to participate in a survey and with lower cognitive skills.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Andrews

(2005). Who is being heard? Response bias in open-ended responses in a large government employee survey. Paper presented at the 60th Annual Conference of the American Association for Public Opinion Research, Miami Beach, FL.

Barrios

Villarroya

Borrego

Á.

Ollé

(2010). Response rates and data quality in web and mail surveys administered to phd holders. Social Science Computer Review, 29, 208–220.

Borg

Zuell

(2012). Write-in comments in employee surveys. International Journal of Manpower, 33, 206–220. doi:10.1108/01437721211225453

Denscombe

(2008). The length of responses to open-ended questions. A comparison of online and paper questionnaires in terms of a mode effect. Social Science Computer Review, 26, 359–368. doi:10.1177/0894439307309671

Emde

Fuchs

(2012). Using adaptive questionnaire design in open-ended questions: A field experiment. Paper presented at the American Association for Public Opinion Research (AAPOR) 67th Annual Conference, San Diego, USA.

Geer

J. G.

(1988). What do open-ended questions measure? Public Opinion Quarterly, 52, 365–371. doi:10.1086/269113

Geer

J. G.

(1991). Do open-ended questions measure “salient” issues? Public Opinion Quarterly, 55, 360–370. doi:10.1086/269268

Harman

R. P.

(2011). Understanding survey comment and the characteristics of nonresponders. Cambridge, MA: Proquest, Uni Dissertation.

Holland

J. L.

Christian

L. M.

(2009). The influence of topic interest and interactive probing on responses to open-ended questions in web surveys. Social Science Computer Review, 27, 197–212. doi:10.1177/0894439308327481

10.

Krosnick

J. A.

(1999). Survey research. Annual Review of Psychology, 50, 537–567. doi:10.1146/annurev.psych.50.1.537

11.

Poncheri

R. M.

Lindberg

J. T.

Foster Thompson

Surface

E. A.

(2008). A Comment on employee surveys: Negativity bias in open-ended responses. Organizational Research Methods, 11, 614–630. doi:10.1177/1094428106295504

12.

Reja

Manfreda

K. L.

Hlebec

Vehovar

(2003). Open-ended vs. close-ended questions in web questionnaires. Developments in Applied Statistics, 19, 159–177.

13.

Scholz

Zuell

(2012). Item non-response in open-ended questions: Who does not answer on the meaning of left and right? Social Science Research, 41, 1415–1428. doi:10.1016/j.ssresearch.2012.07.006

14.

Schwarz

Strack

(1985). Cognitive and affective processes in judgments of subjective well-being: a preliminary model. In Brandstatter

Kirchler

(Eds.), Economic psychology (pp. 439–447). Linz, Austria: R. Tauner.

15.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2001). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton.

16.

Smyth

J. D.

Dillman

D. A.

Christian

L. M.

McBride

(2009). Open-ended questions in web surveys. Can increasing the size of answer boxes and providing extra verbal instructions improve response quality? Public Opinion Quarterly, 73, 325–337. doi:10.1093/poq/nfp029

17.

Tourangeau

Rasinski

K. A.

(1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, 299–314. doi:10.1037/0033-2909.103.3.299

18.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge, New York: Cambridge University Press.