Asking Good Survey Questions

The Use of the Churchill Procedure as a Mindless Drill

As Churchill himself (1998, p. 30) noted: “The bad news is that measurement seems to almost have become a rote process, with the Paradigm article serving as backdrop for the drill, thereby supposedly lending legitimacy to what seems to be at times thoughtless, rather than thoughtful, efforts . . . the rote approach falls far short of what we should be striving for.”

Survey researchers in tourism largely follow Churchill’s scale development paradigm, although they rarely explicitly say so: of the 279 constructs measured empirically in the reviewed articles, only 20% explicitly state the measurement paradigm under which they operate. However, given the scale quality criteria provided by the authors, it can be assumed that at least 60% of studies using multiple items to measure a construct use parts of the Churchill procedure to develop scales. Whether or not tourism researchers have a tendency to use it as a “rote process” or whether they use it as originally intended is hard to assess because the actual items are rarely provided. Survey items are provided for only 30% of constructs; a justification of the choice of scale development paradigm has never been provided.

The Uncritical Use of the Quality Criteria Proposed by Churchill

To quote Churchill again (1998, p. 35), the “concern is not with these calculations, but with their blind use as decision criteria. I have seen any number of instances where people calculate alpha and the item-to-total correlations and automatically use low item-to-total correlations to throw out items, thereby improving alpha. The danger in doing so is that while that procedure invariably improves alpha, it just as invariably lowers the quality of the measure of the construct.” The criteria Churchill proposed to evaluate the quality of scales have become so widely accepted that journal reviewers and editors appear to judge the measure merely by the size of coefficient alpha (although we all know that size does not really matter). What matters is whether the items are capturing the meaning of the construct. This can never be assessed by looking at coefficient alpha, it can only be assessed by looking at the survey questions people were asked.

The review of recent survey studies in tourism reveals that coefficient alpha values were provided for 56% of the constructs measured using multiple items. Factor analysis was used in 17% of constructs to reduce the number of items, and in 45% of cases to determine underlying dimensions of the constructs. For example, one study investigated, among others, job satisfaction of Scottish chefs. Factor analysis led to a factor that was labeled “promotion,” but the item “people get ahead as fast as they do in other places,” which is clearly conceptually related to promotion, was assigned to a different factor. Similarly, the item “I like the people I work with” was not assigned to the factor labeled “co-workers.” In this instance, factor analysis clearly did not group the items into meaningful components defining the job satisfaction construct for chefs. Similarly, a study aimed at understanding how the needs of disabled air passengers can be met used factor analysis to conceptualize the dimensions, leading to a number of illogical assignments of items to factors, such as the items “barrier-free environment” and “reservation phone number” being assigned to the factor “compensation and improvement schemes.”

Cases such as those illustrated above are not uncommon. Components of constructs are rarely developed conceptually in tourism research, often leaving it up to a statistical method (typically factor analysis) to perform this task which by nature is conceptual, not statistical. Churchill (1979) points out that factor analysis tends to produce more factors than conceptually postulated and thus should not be used to circumvent conceptual construct definition (but does have its place in confirming conceptually postulated dimensions). Rossiter (2011) argues that there is no role for factor analysis at all in item and scale development.

It is also surprising how often survey researchers who have been in control of data collection argue that they used factor analysis to reduce the number of variables they have initially chosen to use in the survey. A group of authors, for example, developed survey questions with the aim to learn more about prospective volunteer tourists, including pretrip expectations. They developed and asked people 18 questions and factor analyzed them to “make the number of variables more manageable.” In so doing, the researchers lost, in this instance, 37% of the information they collected (the variance explained was 63%), but more critically, they can no longer draw conclusions about the very pieces of information they originally decided were critical.

An implicit C-OAR-SE perspective (Rossiter 2002, 2011) can only be found in one single survey study in tourism: Choi et al. (2012) state that the “questionnaire was then reviewed by three tourism researchers to check content validity” (p. 29).

Overall, it can be concluded that the understanding of alternative measurement paradigms is relatively low. The incidence of using items or scales taken or derived from previously published studies is high (63% of constructs in the reviewed articles), risking reduced validity of the measure.

The Assumption That Measuring Every Construct Requires Multiple Survey Questions

The issue of single- versus multi-item measures is hotly debated. Opposing views are due to differences in core beliefs about measurement in the social sciences held by two schools of thought, Domain Sampling Theory which underlies the Churchill procedure and C-OAR-SE theory. Both agree that if the object being measured (e.g., a hotel) has multiple components, or if a single object is being assessed using attributes that have multiple components (e.g., knowledge about online booking of plane tickets), multiple survey questions are required. For example, to investigate

HOTEL GUESTS’ (rater)

SATISFACTION (attributes)

WITH THE HOTELS PERFORMANCE IN DIFFERENT ASPECTS (components of the object)

multiple questions are required because each aspect of the hotel (e.g., room size, room cleanliness, comfort of the bed, size of bathroom) has to be assessed separately. The two schools of thought disagree about how many questions are needed to measure one component. For example, when measuring

HOTEL GUESTS’ (rater) ASSESSMENT

ASSESSMENT OF THE COMFORT (attribute)

OF THE HOTEL BED (object)

C-OAR-SE theory suggests that it is sufficient to ask only one good question (e.g., “Was the bed comfortable?”). Domain Sampling Theory suggests that multiple questions should be asked (e.g., “Was the bed comfortable?” “Did you sleep well on the bed?” “Did you wake up refreshed in the morning?”).

What is the underlying reason for this difference? Followers of Domain Sampling Theory argue that the observed score (the answer given by the respondent on the survey) is a result of the true score (the real answer that is in the respondent’s head) being distorted by systematic and random error (observed score = true score + systematic error + random error). Optimally, the observed score is equal to the true score. But according to Domain Sampling Theory, it is not possible to obtain an observed score that is equal to the true score because systematic error and random error distort the observed score away from the true score. Systematic error is caused by respondents’ tendencies to use extreme or middle answer options. Random error is caused by factors such as respondents’ mood, but also imprecise measures or variation in how the questionnaire is administered. Under Domain Sampling Theory, error is minimized by asking respondents more than one question about each component of interest assuming that any error will average out over multiple questions.

Churchill (1979), a proponent of Domain Sampling Theory, provides three arguments for asking respondents more than one question per component: (1) single items are too specific, and such specificity can be averaged out by using multiple items, (2) individual values for a construct have a finer discrimination if they are derived from multiple questions. For example, a single item with 10 answer options allows for differentiation between 10 groups of people based on their response. If, however, five items with 10 answer options are used, the minimum score is 0 and the maximum score over all items is 50, allowing discrimination between 50 groups of people. Finally, Churchill argues, (3) multi-item measures have higher reliabilities because random error averages out.

The C-OAR-SE position on Churchill’s arguments would be as follows: (1) if an item is too specific, it should not be used; (2) finer discrimination among people is only useful if it is valid and validity does not increase, but rather decreases if additional bad items are added; and (3) there is no random error, errors must be prevented when the measures are developed, and they cannot be corrected retrospectively. Under C-OAR-SE, the following alternative model is proposed: observed score = true score + measure induced distortion + rater error.

Measure-induced distortion is caused by bad measures (a badly formulated query or unsuitable return options). Measure-induced distortion is not random error and can be prevented if survey questions are designed carefully. Rater error occurs when respondents make mistakes, due to fatigue or lack of motivation, etc. This can never be prevented; it is truly out of the control of the researcher and therefore represents a random form of error that leads to the observed score not being identical to the true score. Importantly, however, asking more than one question with slightly different meanings and averaging across observed scores cannot remove rater error.

According to C-OAR-SE, asking multiple questions about the same component means that bad questions will be included and will thus decrease the validity of the overall observed score. C-OAR-SE theory can therefore be seen as the stricter measurement theory in terms of researcher responsibility: it forces the survey researcher to think about every possible error and prevent it before it occurs.

Converse and Presser (1986) raise two practical concerns about multi-item scales: (1) if questions are not well worded or suboptimal answer options are provided, using multiple such items in a scale will increase, rather than cancel out, measurement error; and (2) the validity of scales changes over time, so it is unwise to assume it is still valid a few decades later. Converse and Presser acknowledge that single-item measures play a key role in surveys, but believe that this is primarily the case because they keep surveys short.

Recently, Rossiter and Bergkvist (2009) have investigated under which circumstances it is sufficient to ask one single question. Rossiter (2011) identifies three different kinds of objects (concrete, abstract collective, abstract formed) and four kinds of attributes (concrete perceptual, concrete psychological, abstract achieved, and abstract dispositional). Of particular interest in terms of single-item measures are concrete objects, meaning that the respondents understand clearly and unambiguously what they are being asked about (e.g., their hotel bed) AND a concrete perceptual attribute, meaning that the respondents understand clearly and unambiguously what attribute is being assessed (e.g., comfortable). It is in such cases—which Bergkvist and Rossiter (2007) refer to as doubly concrete—that single-item measures are sufficient and, as they argue, superior to multiitem measures because adding any additional items would blur what exactly is being asked. Using the hotel bed example: Asking three questions (“Was the bed comfortable?” “Did you sleep well on the bed?” “Did you wake up refreshed in the morning?”) reduces the quality of the measure because the additional questions measure slightly different things, but are weighted equally as the main meaning. That this effect indeed occurs has been shown empirically by Bergkvist and Rossiter (2009).

A summary of when single and multiple items are required (based on Rossiter 2011) is provided in Table 2.

OPEN IN VIEWER

Table 2.

When to use single or multiple items?

		Object Type
		Single concrete object	Multiple concrete objects	Complex objects with multiple components
		• Examples:	Examples:	Examples:
		• country, city, town	• ecotourism destinations	• important product amenities
		• airline	• backpacker hostels	• important service elements
		• accommodation option
Attribute type	Single concrete attribute	Single-item measure	Multi-item measure	Multi-item measure
	Examples:	Examples:	(category-representative objects)	(main component objects)
	• demographic characteristics	• Will you visit this city again as a tourist?	Examples:	Examples:
	• recall, recognition	• Is Qantas Australian?	• Are ecotourism destinations clean?	• Overall, how satisfied were you with the
	• perceptual attribute	• Will you recommend this hotel to your friends?	• Are backpacker hostels inexpensive?	○ Hotel staff
	• evaluative attribute			○ Your room
	• specific emotion			○ The breakfast
	• overall evaluation or satisfaction
	• intention
	• behavior
	Complex achieved attribute	Multi-item measure	NOT APPLICABLE because abstract achieved attributes about abstract collective objects are not of interest	Multi-item measure
	Examples:	(main component attributes)		(main component objects)
	• multiattribute image	Examples:		Examples:
	• accumulated knowledge	• Is Australia		• During your stay at the hotel, were you satisfied with
		○ Clean		○ Staff
		○ Safe		○ Friendliness
		○ Expensive		○ Effectiveness
		○ Friendly		○ Room
				○ Size
				○ Cleanliness
	Complex dispositional attribute	Multi-item measure	NOT APPLICABLE because complex dispositional attributes about abstract collective objects are not of interest	NOT APPLICABLE because complex dispositional attributes about abstract formed objects do not need to be measured
	Examples:	(main component attributes)
	• personality trait	Examples:
	• motive or goal	• Is Qantas
		○ Innovative
		○ Honest
		○ Reliable

As can be seen, in most cases (all light-gray cells in Table 2) multiple items are required for a range of different reasons. Only in one situation (dark-gray cell in Table 2) is a single item sufficient to capture the construct, that is, when both the object and the attribute under study are concrete.

Of the 78 recent articles reviewed, only 7% use single-item measures. For example, Bojanic and Warnick (2012) measured both satisfaction with and likelihood to return to an event with a single item each. Many instances can be found where doubly concrete constructs are measured using multiple items. For example, in one study, travel intention was measured with four items, only one of which actually reflected travel intention, the other three read as follows: “I’ll say positive things about cruising to other people”; I’ll recommend cruising to others”; and “I’ll encourage friends and relatives to go on a cruise.” Given that three of four measures do not actually measure intention to travel at all, it may well be that the proposed model of explaining intention to travel actually explains intention to recommend instead.

It can be concluded that the scale development paradigm proposed by Churchill dominates survey research in tourism, although the actual measures are rarely made publicly available, thus making it impossible to assess whether following the Churchill procedure led to good measures. Five key precautionary steps can be taken to avoid pitfalls relating to scale development:

Developmental Qualitative Research

Development of questions may or may not require input from people other than the researcher. For example, if a researcher wishes to measure

TOURISTS’ (rater)

INTENTION TO RETURN (attribute)

TO WOLLONGONG (object),

they can easily formulate a valid survey question that will capture the construct because neither the attribute (intention to return) nor the object (Wollongong) requires further specification by experts or tourists. If, however, the construct under study is

RESIDENTS’ (rater)

BELIEFS (attribute)

ABOUT THE EFFECTS OF TOURISM ON THE LOCAL AREA (object),

it is unlikely that the researcher, through introspection or reading of the literature alone, will be able to capture every important belief local residents may hold. Residents must be asked about their beliefs by means of developmental qualitative research. The term developmental qualitative research is used to describe qualitative research conducted for the sole purpose of informing the development of questionnaires. This term is derived from Converse and Presser’s (1986) term developmental pretest in the context of questionnaire pretesting. Developmental qualitative research can use a range of valid sources of insight. Typical sources include interviews and focus groups, but valuable information could also come, for example, from reading the Letters to the Editor section of the local newspaper or going to the annual general meeting of the local tourism organization.

In cases where the attribute under study is not perceptual, consumers cannot assist in the development of survey questions. For example, if the construct is

RESIDENTS’ (rater)

KNOWLEDGE (attribute)

ABOUT THE ECONOMIC IMPACT OF TOURISM TO THE LOCAL ECONOMY (object),

experts on economic impacts of tourism to communities need to be involved in item generation.

The Wording of the Query

Payne (1980) reports on a large number of split-ballot experiments investigating effects of small wording differences. He found, for example, that using the word should versus the word might in the following question: “Do you think anything should/might be done to make it easier for people to pay doctor or hospital bills?” leads to a 19% difference in agreement: 82% say it “should,” but only 63% say it “might.” In a similar experiment, testing the effects of different wording of answer options, the difference was 28%. Payne compares these differences with sampling errors: the biggest documented historical disasters in opinion polling due to sampling errors occurred in the 1936 and 1948 election polls in the United States. The errors made were only 19% and 4%, respectively. It is worth keeping this in mind when developing a survey study: question wording matters a lot and can affect results more than other survey design factors, including sampling and statistical analysis.

The key challenge in formulating survey questions is to ensure that respondents interpret them the same way. Cantril (1940), Payne (1980), and Converse and Presser (1986) derive from their experiments and experience a number of practical recommendations on how to reduce variability of interpretation:

Order Effects

The order of presentation of questions within the survey as well as answer options for each individual query affects responses. Such effects are referred to as order effects. In an extensive review of experimental work on response order effects, Krosnick (1999) concludes that respondents—when asked categorical questions—tend to choose the first of the response options (primacy effect) when they are presented with options visually (written questionnaire or card with options as part of an interview or online), but they tend to choose the last option (recency effect) when response options are presented to them orally, for example, in phone interviews. Furthermore, Krosnick identifies the following factors to increase response order effects: lower levels of cognitive skills of respondents, fatigued respondents, and difficult and longer questions.

Open or Closed Returns

Open questions play an important role in survey research (Payne 1980; Krosnick 1999), especially for the assessment of salience and the collection of a wide range of responses, such as suggestions for improvement, because respondents are not influenced. The main advantage of closed questions is that they are more specific because all participants see the same responses. If the set of responses has been carefully developed (which often requires developmental qualitative research), closed questions should not lead to the omission of important responses.

Unipolar and Bipolar Response Options

Unipolar answer options go in one direction. Intention to visit a tourism destination, for example, demands offering unipolar answer options to respondents because the chances of visiting a destination lie between zero and 100%, they can never be negative (see Figure 2a). Bipolar answer options allow respondents two directions, with one positive and one negative endpoint. For example, tourists can be highly satisfied (extremely positive evaluation) or highly dissatisfied (extremely negative evaluation) with their stay at the hotel (see Figure 2b).

OPEN IN VIEWER

Figure 2.

Unipolar and bipolar response options.

Compared to other aspects of questionnaire design, the issue of unipolar versus bipolar response options has received little attention. The overwhelming popularity of the Likert-type scale in the social sciences may be partially to blame. Using Likert-type scales (Likert 1932) automatically implies bipolar answer options, with the endpoints representing “strongly agree” and “strongly disagree.”

Rossiter (2011) emphasizes the importance of formulating response options correctly in terms of their direction. It is important to note that whether or not to use unipolar or bipolar response options is not a question of researcher preference, it is determined by the nature of the constructs. The researcher has to determine whether the construct under study conceptually allows positive and negative responses (as is the case for beliefs, attitudes, satisfaction) or whether responses are logically limited by the zero point and a positive extreme (as is the case for number of nights spent at a destination, intention to return to a destination, intention to share the satisfaction about a trip with friends). Importantly, numeric coding of the answer options must reflect the directional nature of response options. For unipolar answer options, numeric coding starts with “0” and increases in numbers; for bipolar constructs, the middle point (if a middle point is offered) is coded as 0, with negative options coded as –1, –2, etc. and positive options coded as 1, 2, etc. Correct coding ensures that results from data analysis are not misinterpreted.

Of the constructs studied in recent tourism studies using nonbinary rating scales, 93% offered respondents bipolar answer formats (although only 7% state explicitly that they have done so). Of all answer scales that were identifiable as bipolar to the reader, 89% incorrectly numerically label the answer options as unipolar.

One of the survey studies reviewed for this article used “bipolar adjective scales, where 1 = negative and 7 = positive” (correctly this should be scored as –3 to +3 to account for the bipolar nature of response options), concluding that “the country’s affective country scores are relatively weak, as low as 3.4 for ‘safe.’” In this study, the value 3 was the middle point, so an average value of 3.4 indicates that the average respondent perceives the country neither as safe nor as unsafe. In another example, a seven-point unipolar answer format ranging from “not at all” to “a very great extent” was used. Respondents were asked six questions measuring the extent to which they felt cheated as tourists. The first five asked whether they felt staff were cheating, duping, acting dishonestly, being deceitful, and swindling. The last item asked if respondents felt they had been undercharged. This item is subsequently reverse coded. So if a respondent stated that they have “not at all” been undercharged this is, incorrectly, interpreted as the highest level of overcharging.

To ensure that the correct direction of the answer scale is chosen, the conceptual nature of the construct needs to be understood. Common unipolar constructs measured in tourism include destination image, perceptions, intention to visit/return, intention to recommend, loyalty, and travel motivations; these should be numerically coded from 0 to the maximum number on the scale. Common bipolar constructs measured in tourism include satisfaction, overall evaluations, and overall attitude; these should be coded in the data set from the maximum negative to the maximum positive number on the scale, with or without a zero in the middle, depending whether a neutral middle option is included in the answer options or not.

The Number of Response Options

Response options can be anticipated answer options, not applicable answer options, and no response options (see Figure 1). Anticipated answer options represent the kinds of responses the researchers expect, not applicable answer options can logically not be answered by the respondent because the question is not applicable to him or her (e.g., if you are asked if you liked your stay in Wollongong and you have never been to Wollongong before), and no response or “Don’t know” options allow respondents to not answer the question although there is no logical reason that they could not answer it (E.g., you are asked what your image of Sydney is. Even if you never visited Sydney you may have a perception of it. You may, however, choose not to answer all questions because you do not feel competent to assess each attribute of Sydney).

In this section, the focus is on anticipated answer options. With respect to anticipated options, three choices are available to survey designers: offering only one answer option, two answer options, or more than two answer options. Examples of a destination image measure with one and two anticipated answer options are provided in Figure 3. They are all referred to as binary because the numerical coding of responses in all cases is 1 if the respondent agrees and 0 otherwise.

OPEN IN VIEWER

Figure 3.

Binary answer options.

In the case of the pick any or pick any out of n answer format (Figure 3a), respondents tick the box next to the attribute if they believe that it applies (in this case, to Wollongong). Alternatively, they do not tick a box, in which case it is not possible for the researcher to determine whether they believe that the attribute does not apply to Wollongong, or whether the respondents wanted to get through the questionnaire quickly and simply skipped the question. Offering only one answer option is common in commercial brand image studies.

The same question can be asked offering two answer options, effectively forcing respondents to answer. This answer format is referred to as forced choice binary (Figure 3b). This answer format does not restrict the way the query is formulated or the way the attributes, in this case characteristics of a tourist destination, are formulated.

A more specific kind of the forced choice binary format is the DLF IIST format. DLF IIST stands for Doubly Level-Free with Individually Inferred Satisfaction Threshold (it is also referred to as the BIT measure, which stands for Binary Internal Threshold) and is illustrated in Figure 3c. This format is characterized by additional restrictions being placed on the basic forced-choice binary format: neither the description of the answer options (“Yes” and “No”) nor the attribute part of the question (“expensive,” “fun,” “safe”) may contain an indication of magnitude (e.g., “very,” “extremely,” “rather,” . . . ). The theory is that, as a consequence, respondents answer the question with respect to their personal threshold for any given attribute. For a detailed discussion of the DLF IIST measure, see Rossiter, Dolnicar, and Grün (2010) and Rossiter (2011), for empirical evidence of its performance see Dolnicar, Rossiter, and Grün (2012) and Dolnicar and Grün (2013).

Alternatively, respondents can be offered multiple anticipated answer options, ranging from three to endlessly many. As soon as three anticipated options are provided (and assuming those do not include a neutral midpoint), respondents are simultaneously providing a direction and magnitude response. This can be positive when it is essential to understand magnitude, for example, to assess the effectiveness of an advertising campaign. It does come, however, at a price: the magnitude response can be biased by people’s tendencies to respond in certain ways (known as response styles; Paulhus 1991), making it impossible to separate the response from the response style.

The discussion about which number of answer options is best has led to a large number of investigations. One of the earliest studies was conducted by Peabody (1962). Peabody finds no relationship between the number of answer options and the reliability of a scale and recommends the use of two answer options that lead to equal reliability but are simpler and more convenient to use and administer. Similarly, Komorita (1963) studied whether responses to Likert scaled items could be reproduced by weighting direction information (binary agree, disagree) with magnitude information, concluding that the magnitude information “had practically no effect on total scores” (p. 332). Jacoby and Matell (1971) extended this line of research to include four assessment criteria: test–retest reliability (stability over two measurements at different points in time), internal consistency reliability (measured using Cronbach’s alpha), concurrent validity (predictive validity to another measure asked in the same survey, using a graphical scale), and predictive validity (predictive validity to another measure asked in the second survey, using a graphical scale). They compare 18 different versions of the Likert-type scale ranging from 2 to 19 answer options. Results confirm Peabody’s and Komorita’s findings: no statistically significant difference was determined in any of the criteria used to compare the performance of the answer formats.

Similar conclusions have been drawn in later studies which demonstrate that reliability of responses increased over time if separate questions were asked to capture direction and extremity (Krosnick and Berent 1993), and that the confounding of direction and magnitude in Likert-type scales leads to a tendency to use middle answer options (Albaum 1997; Albaum et al. 2007; Yu, Albaum, and Swenson 2003). This tendency can be avoided by asking respondents to answer two questions, one about direction only (including a “yes” and “no” option or similar and, if appropriate a “Don’t know” option) and the second one aimed at capturing magnitude only (including, e.g., a “very” and a “somewhat” option).

A large number of empirical studies did not use the underlying concept of direction versus magnitude to drive the investigation. Rather they used a range of different criteria for the assessment of the number of answer options. Key studies are reported in the following sections.

Reliability and Validity

Empirical evidence on whether questions with fewer or more response options are more reliable and valid are mixed; a number of studies conclude that the number of response options does not affect reliability (Bendig 1954; Peabody 1962; Komorita 1963; Komorita and Graham 1965; Matell and Jacoby 1971; Jacoby and Matell 1971; Remington et al. 1979; Preston and Colman 2000), others come to the opposite conclusion (Symonds 1924; Nunnally 1967; Jones 1968; Oaster 1989; Finn 1972). The same is the case for validity, with Matell and Jacoby (1971), Jacoby and Matell (1971), and Preston and Colman (2000) finding no difference and Loken et al. (1987) and Hancock and Klockars (1991) concluding that validity is increased if more answer options are provided.

The above studies are problematic because they vary in their definitions of reliability and validity and, as noted by Chang (1994), they do not decompose systematic method variance and trait variance, giving scales with more answer options an advantage caused purely by computational reasons, specifically the restriction of range effect (see Nunnally 1970; Martin 1973, 1978) which affects correlation-based measures such as Cronbach’s alpha and test–retest measures. Using structural equation models, Chang concludes that criterion-related validity is independent of the number of answer options and reliability can be improved by reducing the number of answer options.

Stability of Responses over Repeated Measurements

When a good survey question is asked, the observed score is close to the true score and it can be expected that respondents will be able to reproduce the observed score response if asked the same question again (in the absence of any environmental changes). If responses are found not to be stable, this can be due to a bad query or bad response options. Therefore, stability is a useful criterion for the comparative assessment of answer options.

Only a small number of studies have investigated stability; Dall’Olmo Riley et al. (1997) and Rungie et al. (2005) examine the problem of instability in brand image measures. Dolnicar and Rossiter (2008) show that this lack of stability can be explained partly by measurement factors. Dolnicar, Grün, and Leisch (2011) find no difference in stability in dependence of the number of answer options offered. These findings are supported by the study conducted by Dolnicar and Grün (2007d) using only repeated agreement as a criterion. If, however, a stricter criterion of stability is used (the requirement to actually choose the exact same response option), the forced-choice binary answer format outperforms all other response options (five-point and six-point formats, both fully verbalized and end-point anchored only) significantly. Additional empirical evidence for the stability of forced binary answer options has been recently provided by Dolnicar, Rossiter, and Grün (2012) and Dolnicar and Grün (2013). The key conclusion from these studies is that—in the context of brand image measurement—the forced binary answer format produces the most stable results without loss of concurrent validity, and that the answer format most frequently used in commercial market research, the pick any answer format, is highly prone to evasion, thus producing substantially fewer “yes” responses than any other answer format.

Interpretation of Findings

The purpose of survey data is to be interpreted so as to advance knowledge (a typical academic aim) or gain market insight (a typical aim of commercial survey studies). Consequently, it is important to understand whether the choice of the number of anticipated answer options affects the interpretation of findings. A number of studies have investigated this question; Green and Rao (1970) constructed artificial data and compared the recovery of data structure in dependence of the number of answer options, concluding that at least six points and at least eight attributes should be used. A different design involved collapsing empirical multicategory data to dichotomous or trichotomous data, computing factor analyses on both data sets and comparing the results (Martin, Fruchter, and Mathis 1974; Percy 1976). These studies lead to the conclusion that the results do not differ significantly using an objective measure of compliance between factor solutions as the criterion.

In those early studies, data transformation was controlled by researchers, not respondents. Later studies avoided this problem by collecting data using different answer formats directly from respondents: Loken et al. (1987) compare responses given on an 11- and a 4-point scale and conclude that there was no difference in discrimination power between sociodemographic groups and capturing of relationships between variables. Similarly, Preston and Colman (2000) empirically compare responses given using 10 different scales, including dichotomous and nearly metric (101 scale points) formats. Using correlation matrices, underlying factor structure, and coefficient alpha values as criteria, they find no differences between formats. Dolnicar, Grün, and Leisch (2011) collected empirical data using different answer formats, and find that results from a positioning analysis for both answer formats were practically the same.

It can be concluded from these studies that there is no reason to believe that any particular answer format is superior with respect to findings derived from the entire study.

User-Friendliness

In a study on respondents’ answer option preferences, Jones (1968) finds that respondents prefer multiple categories. Similarly, Preston and Colman (2000) find that respondents are better able to express themselves when more categories are offered, although they perceive surveys with fewer response options as quicker. Respondents in a study by Dolnicar and Grün (2007a) also perceive the binary format as being the quickest (in the context of attitude and intention measurement) but show no preference for either a binary, 7-point, or metric format; neither do respondents perceive any of these formats as simpler or as better in terms of their ability to express their feelings.

Dolnicar and Grün (2007b) compared the user-friendliness of five different answer formats (forced-choice binary, 3-point scale, 7-point scale, visual analogue scale, and percentage allocation) in the context of responding to questions about brand image and behavioral intentions. They conclude that the 3-point, 7-point, and forced binary formats are preferred over the two metric formats. Respondents indicate that the forced binary answer format was the simplest and the quickest, but there was no significant difference in how pleasant they perceived the answer formats to be, and how well they allowed them to express their feelings.

Dolnicar, Grün, and Leisch (2011) report that 70% of respondents stated that the binary survey was easy, but only 45% indicated this for the multicategory version, supporting earlier findings by Dolnicar (2003) in which respondents perceived binary answer options as easier to answer than ordinal scales.

From these studies, it is concluded that results on user-friendliness are mixed, and no clear “winner” emerges. It is important to note, however, that people certainly do perceive some formats as easier, more pleasant, and quicker. User-friendliness should, therefore, be taken into consideration when developing a survey.

Proneness to Bias

A key concern—one shared among most measurement researchers in the social sciences—is data contamination by response styles. Two positions can be taken on response styles. The first position is that response styles can be removed from the data before data analysis. A wide range of approaches of how to remove bias of such kind has been suggested in the past (Cunningham, Cunningham, and Green 1977; Greenleaf 1992a, 1992b; Heide and Gronhaug 1992; Watson 1992; Van de Vijver and Poortinga 2002; Welkenhuysen-Gybels, Billiet, and Cambré 2003; Dolnicar and Grün 2007c).

The second position is that it is impossible to remove bias from data retrospectively and that it is therefore critical to select answer options which will reduce the likelihood of capturing response styles in the first place. Possibly the first proponent of this view was Cronbach (1950), who stated: “Since response sets are a nuisance, test designers should avoid forms of items which response sets infest” (p. 21) and recommended forced-choice and paired comparison formats to address this problem. The same argument is made by Lee et al. (2006) in favor of best worst scaling, and Rossiter, Dolnicar, and Grün (2010) in favor of the DLF IIST (or BIT) forced binary measure.

Kampen and Swyngedouw (2000) classify ordinal scales into different types and raise serious concerns about Type 5, defined as unstandardized discrete variables with ordered categories such as the agreement with statements or levels of satisfaction: “in many instances the experimenter can only hope that in general respondents or experimentators attach the same meaning to the [response] categories of an ordinal variable” (p. 99). In addition, the fact that scale points are not equidistant causes problems in both data analysis and interpretation.

In multicategory answer formats, a number of response styles can occur, including—most prominently—extreme response bias, where respondents tend to use the extreme positive and negative ends of the scale, tendency to the middle, where respondents tend to choose the middle option or an option close to the middle and acquiescence bias where people tend to respond in the affirmative (yes rather than no, agree rather than disagree). While extreme response bias and tendency to the middle can logically only occur in response formats with more than two options, acquiescence bias has been shown to also affect response options including only two options, such as “agree” versus “disagree”. Specifically, a review of experimental evidence on acquiescence by Krosnick (1999) reveals that “yes” versus “no” questions are less prone to this bias than “true” versus “false” and “agree” versus “disagree” questions.

A different kind of bias occurs with pick-any measures, where respondents are only given one anticipated response option which they tick if they agree but do not tick if they disagree (for an example, see Figure 3a). Such response formats cause substantial levels of evasion (Rossiter 2011; Dolnicar, Rossiter, and Grün 2012; Dolnicar and Grün 2013), probably because respondents can complete the survey without ticking much, and without investing much cognitive effort. Krosnick (1999) would refer to this as strong satisficing.

The forced-choice binary format is able to avoid capturing most response styles, and thus represents an attractive option, if suitable for the measurement of the construct under study.

Speed

There are at least three good reasons to keep a survey as short as possible. First of all, short questionnaires reduce fatigue effects among respondents, which lead to a reduction of data quality (Johnson, Lehmann, and Horne 1990; Drolet and Morrison 2001; Vriens, Wedel, and Sandor 2001). Second, people have become increasingly reluctant to participate in survey studies (Hardie and Kosomitis 2005; Bednell and Shaw 2003), leading to difficulties recruiting representative samples. The length of the survey is a key reason for people to decline participating in market research (Hardie and Kosomitis 2005). Shorter surveys can increase participation rates and improve sample representativity. Finally, and most pragmatically, short surveys reduce fieldwork cost. In online surveys, for example, doubling the length of a questionnaire leads to a 30% cost increase (Dolnicar, Grün, and Leisch 2011). Therefore, if data of equal quality can be collected in a quicker way, it should be.

A number of studies have shown that offering fewer response options speeds up the process of completing a survey significantly without negatively affecting data quality (Dolnicar 2003; Driesener and Romaniuk 2006; Grassi et al. 2007; Dolnicar, Grün, and Leisch 2011), thus leading to the recommendation to offer only one or two response options. Given the proneness of the pick-any format (one answer option only) to evasion, the forced-choice full binary format (two response options) is preferable when the aim is to reduce completion times without capturing systematic bias.

Across all criteria of comparison, however, it has to be concluded that there is no single best number of answer options; different answer formats are suitable for different situations (Lehmann and Hulbert 1972; Preston and Colman 2000; Dolnicar and Grün 2009). Yet, currently, commercial market research heavily relies on the pick-any answer format, which leads to substantial evasion by respondents while academics prefer ordinal answer formats (Van der Eijk 2001) which are prone to capturing response styles.

Among the 279 empirical measures reviewed, 240 offered respondents closed answer options rather than asking them to respond in an open-ended manner or rank. Of these 240, 95% offered respondents more than two answer options. Of those, 1% offered respondents three answer alternatives, 4% provided four answer alternatives, 51% offered five, 5% offered six, 38% allowed respondents to choose between seven answer options, and 1% used 11 answer options. These results show that tourism survey researchers have adopted a standard approach of offering five or seven answer options—a dangerous choice given the known susceptibility of multicategory answer formats to response styles. The problem is exacerbated in instances where respondents from different countries participate in the survey because cultural background is a known cause of response styles. In tourism survey research, respondents from different cultural background are common; 48% of recent survey studies in tourism use such samples.

An example is provided by a study that asked respondents from a wide range of cultural backgrounds about their perceptions of being cheated by tourism service workers using a seven-point scale. Responses are compared across countries of origin. It is unclear how much of the difference is due to differences in feeling cheated versus differences in the way respondents from different cultures use seven-point scales.

It should also be noted that a justification or explanation why a certain number of answer options is chosen has only been provided for 3 of the 240 measures. Sometimes, authors choose a certain response scale, but then report results at a lower level. In one study data on perceived severity of visitor impacts is collected on a four-point scale. The bottom two and the top two answer options are then added up for reporting, leading to the question: why were more response options used in the first place?

So, no single best number of answer options exists. Yet, randomly choosing a number of answer options is not an acceptable solution, neither is the uncritical adoption of answer options used by others in the past. Rather, researchers should be aware of all alternatives and assess whether answer options are logically meaningful given the query, whether respondents feel that they can express themselves well using the answer options provided, whether the chosen answer format is prone to biases (and if so, what will be done to minimize contamination of data), whether respondents display a high level of stability when asked the same question twice using an answer format, and whether the answer format chosen represents the most efficient option in terms of speed of completion. Sometimes, these questions cannot be answered by the researcher, making additional developmental qualitative research necessary before finalizing the decision on the number of answer options to be provided.

Midpoints

In cases where three or more anticipated bipolar answer options are offered to respondents, the researcher has to decide to include or not to include a “neutral” midpoint. Examples of a survey question with and without midpoint are provided in Figure 4.

OPEN IN VIEWER

Figure 4.

Response options with and without midpoint.

A small number of experimental studies have been conducted over the past decades that arrive at similar conclusions: midpoints do not add much value, but have the potential to induce evasion. For example, Schuman and Presser’s (1981) experimental work suggests that people who do not have strong feelings, but still lean toward one of the answer options, will evade by using midpoints and as a consequence the researchers will lose information. Schuman and Presser recommend, should it be critical to the study to capture the level of magnitude of the response, to first offer an answer option capturing direction only (yes, no) and then follow up with a pure intensity question to determine how strongly respondents feel about the issue (e.g.: How strongly do you feel about that?).

Additional support for Schuman and Presser’s results have been provided in recent experiments by Dolnicar and Grün (forthcoming) in which respondents were asked to respond to different survey questionnaire versions, some including “Don’t know” response options, some including midpoints or neither of the two. Results of individual-level comparisons indicate that respondents use the midpoint as a substitute for a “Don’t know” option. This makes them impossible to interpret: they could mean that a respondent cannot provide a response because they have no opinion, cannot provide a response because they do not feel comfortable answering, or even chooses not to provide a response. As a consequence, Dolnicar and Grün recommend avoiding midpoints because of lack of interpretability of their actual meaning.

Garland (1991), from an experimental study comparing responses when a midpoint was offered with responses when a midpoint was not offered, concludes, first, that this manipulation significantly influences responses, and, second, that the elimination of the midpoint removed social desirability bias. Rossiter (2011), who is generally sceptical about the use of midpoint options because of their susceptibility to evasion, recommends them only in cases when respondents are asked to provide overall evaluations of objects. In this case, it may well be that people are genuinely neutral, he argues.

Among recent tourism studies measuring constructs using non-binary rating scales, the vast majority (91%) offer midpoints, although only 8% state explicitly that they do. There is not a single case among the reviewed studies where a reason was stated for providing a midpoint. Given experimental findings relating to midpoints and the frequent use of midpoints in tourism surveys, it is possible that tourism survey data contains substantial counts of evasion bias. Evasion bias cannot be eliminated after data has been collected and, therefore, can negatively affect the validity of findings.

It can be concluded that midpoints should be used only if there is a specific and valid reason to do so (e.g., an overall attitude is measured which may genuinely be neutral for certain respondents). Midpoints are used in different ways by respondents. As such, they cannot be interpreted in a meaningful way; instead they have the potential of distracting respondents from using the anticipated response options provided which can be meaningfully interpreted by the researchers, even if it is “I do not know”.

“Don’t Know” Options

“Don’t know” options in surveys, also referred to as nonresponse options allow respondents not to answer a question. The intention of offering such options is that people who are genuinely unable to answer are not forced to. Imagine being asked the question in Figure 5. Unless you have been to Wollongong, it is impossible for you to know and meaningless for you to guess whether or not there are any Mexican restaurants in Wollongong, making the “Don’t know” option crucial in ensuring high-quality data.

OPEN IN VIEWER

Figure 5.

Response options with and without “don’t know” option.

Generally, two schools of thought exist with respect to offering “Don’t know” options in surveys: one postulates that responding with “Don’t know” is a characteristic of the respondent (a “relatively stable individual trait”; Rapoport 1982) and not related to the content of the question, much like a response style. If this assumption is correct, “Don’t know” responses do not actually reflect respondents’ true answers and reduce data quality.

A number of reasons are put forward by proponents of this view. For example, respondents may not feel confident in answering the question and therefore may tick the “Don’t know” option unnecessarily. Some evidence for this explanation has been provided by Rapoport (1982): women gave more “Don’t know” responses than men, especially when both the women and men had low education levels, so their response is not likely to be driven by knowledge of the answer but to a high extent by their confidence to give a response.

Another reason may be that respondents want to comply with the researcher’s request to complete the survey. Krosnick (1991) refers to this as satisficing. From a satisficing perspective, the “Don’t know” option is a way to comply with the researcher’s request while avoiding the cognitive effort associated with thinking about each question. Krosnick (1999) lists a number of other possible mechanisms that may lead to false “Don’t know” responses such as people not understanding the meaning of the questions, avoiding to answer honestly, or feeling ambivalent about the answer. Empirical evidence for the respondents don’t want to answer, make them! school of thought is scarce, mainly because empirical research is based on observation and subjective interpretation of survey responses by researchers rather than explanations by respondents why they did or did not use a “Don’t know” option. For example, Poe et al. (1988) find no differences in reliability of responses in dependence of offering a “Don’t know” response and base their argument against “Don’t know” options on the fact that fewer anticipated responses are given. They provide no proof, however, that the additional anticipated responses given are valid, which is the most important criterion when aiming to collect high-quality empirical data. Gilljam and Granberg (1993) show that survey answers provided by respondents who used “Don’t know” to avoid other survey questions about nuclear power were predictive of their referendum vote. They conclude that they would have been competent to answer the questions and therefore should be made to. But even if the answers they have provided were predictive, they still may have validly stated not to know the answer to the other questions.

The other school of thought argues that “Don’t know” responses are caused by respondents being unable to answer a question. If this assumption is correct, not offering a “Don’t know” answer option would force them to guess, which, in turn, would reduce data quality. Proponents thus argue for offering a “Don’t know” response option to prevent people who genuinely cannot answer the question (Are there any Mexican restaurants in Wollongong?) to guess. Empirical evidence for this argument has been provided by Hawkins and Coney (1981), Schuman and Presser (1981), and most recently, Dolnicar and Grün (forthcoming). These studies show that offering a “Don’t know” option reduces the number of uninformed or guessing responses using quantitative criteria to determine whether or not respondents were familiar enough with the content of the question to be able to answer it. Payne (1980) summarizes this school of thought’s perspective in saying: “If most respondents have no basis for opinions but in effect would have to flip a coin mentally for their answers, it would be better if their answers were not recoded” (p. 23).

In the context of tourism research, Ryan and Garland (1999) recommend the inclusion of “Don’t know” options, pointing out that the nonrandom patterns of “Don’t know” responses represent valuable information in their own right.

Note that studies on “Don’t know” options come from different disciplines: marketing (Hawkins and Coney), tourism (Ryan and Garland), but mostly public opinion research (Krosnick). In opinion research, it can be argued that everyone can have an opinion on pretty much anything. In other contexts, such as destination image measurement, this is hard to argue; if you have never ever heard of Wollongong before reading this article, how could you possibly be expected to competently answer questions about it? In the brand image context, Dolnicar and Rossiter (2008) specifically recommend using “Don’t know” options to ensure that responses relating to unfamiliar brands can be identified as such.

The key insight that emerges from the large body of work on “Don’t know” response options is, whether a “Don’t know” option is offered or not, it is impossible to avoid some contamination—either by incompetent respondents guessing or by competent respondents choosing not to answer. The key question therefore is not which school of thought is right, but how can data contamination be kept to a minimum. Dolnicar and Grün, for their brand image data set, compare the error rates for both cases empirically. They assume that people who are familiar with a fast-food brand and eat there are competent to answer questions about it. When a “Don’t know” option was not offered to respondents, 9% of respondents gave answers about fast-food restaurants they knew nothing about. This means that 9% of the data was likely to be junk, to use Jacoby’s term. When a “Don’t know” option was provided, 1% of respondents who were familiar with a brand chose the “Don’t know” option. This means that only 1% of data was junk. For this particular data set, including a “Don’t know” option was clearly better.

In recent tourism survey research studies, “Don’t know” options are extremely rare (5%). A good example of using a “Don’t know” option is provided by Choi et al. (2012). They offered respondents a “Don’t know” response option and provided an explanation why they chose to do so: “N/A stands for a ‘no response’ or ‘don’t know’ answer. Some respondents were not able to recollect the exact decision time for each decision item” (p. 31). If respondents in tourism studies can reasonably be expected to answer all questions, the low figure of 5% would be no reason for concern, but this appears to be rather unlikely. Even among the 279 constructs measured in the 78 reviewed articles, brand or destination image alone was measured ten times. It is likely that some respondents would not have felt competent to assess all the attributes for all the brands/destinations listed. As a consequence, contamination of tourism survey data by respondent guessing is likely.

It can be concluded that the researcher has to assess whether some respondents will genuinely be unable to provide an answer before finalizing the measure. This can be done logically or by conducting developmental qualitative research, asking a heterogeneous sample if they can answer the questions. If some cannot, a “Don’t know” option should be provided. The “Don’t know” options should be visually well separated from other response options and should not be overly strongly worded to avoid actively attracting “Don’t know” responses (an effect demonstrated by Hippler and Schwarz in 1989).

Verbal Labeling of Anticipated Answer Options

Whether or not answer options should be verbally labeled appears to be one of the few uncontentious areas of questionnaire design with broad consensus that options should be verbally labeled because verbal labeling reduces the variability in the interpretation of answer options by respondents and thus leads to higher reliability of results (Peters and McCormick 1966; Krosnick and Berent 1993; Krosnick 1999). Note, however, that minor differences in wording can affect respondents’ interpretations substantially and that the assumption that verbally labeled multicategory options have the same distance between them is very unlikely, as demonstrated by Worcester and Burns (1975), who provide empirically derived numerical values for different verbalizations of bipolar answer formats that can be used in the data set, rather than simply coding the values as +2, +1, 0, –1, and –2, implicitly assuming equidistance.

One of the few studies concluding that purely numerically labeled scale was as good as the verbally labeled scale was provided by Finn (1972). The study was conducted in the context of rating job complexity. Context dependence is the key argument made by Rossiter (2011), who argues theoretically for the use of verbal labels when beliefs are being measured because “beliefs are mentally represented as verbal statements” (p. 20), and for the use of (bipolar) numerical scales when respondents are asked for overall evaluations about something. Again, the reason is that it is likely such overall evaluations are mentally represented in a quantitative way.

For recently published empirical tourism studies, it is extremely difficult to assess how frequently respondents are offered a purely verbally labeled answer format, a purely numerically labeled answer format, or both. The reason is that measures are rarely provided and the description of measures tends to be rudimentary. In cases where some detail on the measure is provided, it is often unclear whether the description refers to the presentation of answer options to respondents or to the way data were coded in the data set. A typical example is illustrated by the following: “Measurements of all the constructs were carried out by the statements adopted from previous studies and a 7-point Likert type scale ranging from (1) Strongly Disagree to (7) Strongly Agree as shown in Appendix 1” with Appendix 1 stating “7-point Likert scale: ‘1’ Disagree Strongly and ‘7’ Agree Strongly.” From both the description in the text and the items in the Appendix, it is actually not clear what exactly the respondents saw: the words, the numbers, or both. On the positive side, this is one of the few articles where the actual measures are provided to the reader.

Based on the evidence on the effects of labeling of answer option, it is recommended that answer options be verbally labeled; unless there is a specific, stated reason for not doing so (e.g., overall evaluations are requested from respondents).

Interactive Returns

Online surveys account for approximately 20% of the global research market (Poynter 2010). Of the articles reviewed for this study, 23% report that they collected data online. These figures illustrate that, while uptake in academic survey research is slower than in the commercial market research, online data collection is no longer a marginal occurrence. It is quickly developing to become the main source of survey data.

With the increase in popularity of online surveys and less concern about representativity of data collected online (Dolnicar, Laesser, and Matus 2009), entirely new ways of asking questions in surveys have become feasible. Instead of ticking a box on a piece of paper or even clicking on an icon online, respondents can now drag and drop pictorial symbols of the objects they are asked to assess in the precise order they see fit, or walk through a simulated shop and take products off shelves and place them in a shopping cart.

While market research clients are impressed by the new possibilities and online survey companies emphasize the benefits—primarily the increase in respondent engagement—of these new “funky” question formats, little research has been undertaken to investigate whether they do more good by improving respondents’ engagement or more bad by potentially distracting the respondent with “special effects” from what is being asked and thus negatively affecting the validity of questions.

Research to date has largely been limited to testing whether or not interactive survey questions increase respondent engagement (Reid, Morden, and Reid 2007; Puleston and Sleep 2008; Strube et al. 2008; Sleep and Puleston 2009). Almost unanimously, they conclude that this is indeed the case. These studies do not, however, prove that the collected data are better. There is a possibility that respondents had more fun completing the survey but, maybe as a consequence of perceiving it as entertainment, provided less valid responses.

Two studies have attempted to go beyond respondent engagement as the dependent variables and assess whether data quality is in fact better if respondents are offered an interactive way of responding; Delavande and Rohwedder (2008) developed a survey question where respondents threw virtual balls into bins on the computer screen to express their opinions and compared this with a more traditional answer option. They find the results from the dynamic task to be more usable, with usable being defined as not violating monotonicity. Dolnicar, Grün, and Yanamandram (forthcoming) compared three response formats; a percentage allocation task, a ranking task, and a dynamic drag-and-drop task in the context of determining how much people felt vacations contributed to their quality of life. They conclude that the responses from the dynamic question format had the highest concurrent validity. In addition, respondents perceived the question format as the easiest and preferred option if they had to redo the survey.

Given the lack of experimental data on interactive questions, the key recommendation for survey researchers at this point is rather general; inspect interactive survey questions carefully to ensure that potential advantages such as increased respondent engagement are harvested without sacrifice in measurement validity.