Slider scales and radio buttons scales were experimentally compared in horizontal and vertical orientation. Slider scales lead to statistically significantly higher break-off rates (odds ratio = 6.9) and substantially higher response times. Problems with slider scales were especially prevalent in participants with less than average education, suggesting the slider scale format is more challenging in terms of previous knowledge needed or cognitive load. An alternative explanation, technology-dependent sampling (Buchanan & Reips, 2001), cannot fully account for the current results. The authors clearly advise against the use of Java-based slider scales and advocate low-tech solutions for the design of Web-based data collection. Orientation on screen had no observable effect on data quality or usability of rating scales. Implications of item format for Web-based surveys are discussed.
Arnau, R.C., Thompson, R.L., & Cook, C. ( 2001). Do different response formats change the latent structure of responses? An empirical investigation using taxometric analysis. Educational and Psychological Measurement, 61, 23-44.
2.
Buchanan, T., & Reips, U.-D. ( 2001). Platform-dependent biases in Online research: Do Mac users really think different? In K. J. Jonas, P. Breuer, B. Schauenburg, & M. Boos (Eds.), Perspectives on Internet research: Concepts and methods [WWW document]. Retrieved from http://www.psych.uni-goettingen.de/congress/gor-2001/contrib/buchanan-tom
3.
Christian, L.M., Dillman, D.A., & Smyth, J.D. ( 2005). Instructing Web and telephone respondent to report date answers in format desired by surveyor (Technical Report 05-067). Retrieved July 16, 2008, from http://www.sesrc.wsu.edu/dillman/papers.htm
Couper, M.P. ( 2008). Designing effective Web surveys. Cambridge, UK: Cambridge University Press.
6.
Couper, M.P., Conrad, F.G., & Tourangeau, R. (2007). Visual context effects in Web surveys. Public Opinion Quarterly, 71, 623-634.
7.
Couper, M.P., Tourangeau, R., Conrad, F.G., & Crawford, S.D. (2004). What they see is what we get: Response options for Web surveys. Social Science Computer Review, 22, 111-127.
8.
Couper, M.P., Tourangeau, R., Conrad, F.G., & Singer, E. ( 2006). Evaluating the effectiveness of visual analog scales: A Web experiment. Social Science Computer Review, 24, 227-245.
9.
Couper, M.P., Traugott, M.W., & Lamias, M.J. ( 2001). Web survey design and administration. Public Opinion Quarterly, 65, 230-253.
Downie, W.W., Leatham, P.A., Rhind, V.M., Wright, V., Branco, J.A., & Anderson, J.A. (1978). Studies with pain rating scales. Annals of the Rheumatic Diseases, 37, 378-381.
12.
Fuchs, M. ( 2008). Mobile Web survey: A preliminary discussion of methodological implications. In F. G. Conrad & M. F. Schober (Eds.), Envisioning the survey interview of the future (pp. 77-94). New York, NY: Wiley.
13.
Graham, N.E. ( 1956). The speed and accuracy of reading horizontal, vertical, and circular scales. Journal of Applied Psychology, 40, 228-232.
14.
Healey, B. ( 2007). Drop downs and scroll mice: The effect of response option format and input mechanism employed on data quality in Web surveys. Social Science Computer Review, 25, 111-128.
15.
Heerwegh, D. ( 2003). Explaining response latencies and changing answers using client side paradata from a Web survey. Social Science Computer Review, 21, 360-373.
16.
Heerwegh, D., & Loosveldt, G. ( 2002). An evaluation of the effect of response formats on data quality in Web surveys. Social Science Computer Review, 20, 471-484.
17.
Krosnick, J.A. ( 1999). Survey research. Annual Review of Psychology , 50, 537-567.
18.
Krosnick, J.A., & Fabrigar, L.R. (1997). Designing rating scales for effective measurement in surveys. In L. Lyberg, P. Biemer, M. Collins, E. deLeeuw, C. Dippo, N. Schwarz , & D. Trewin (Eds.), Survey measurement and process quality (pp. 141-164). New York, NY : Wiley.
19.
Lyberg, L., Biemer, P., Collins, M., deLeeuw, E. , Dippo, C., Schwarz, N., & Trewin, D. (Eds.). (1997). Survey measurement and process quality. New York, NY: Wiley.
20.
Reips, U.-D. ( 2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 89-118). San Diego, CA : Academic Press.
21.
Reips, U.-D. ( 2002a). Context effects in Web surveys. In B. Batinic, U.-D. Reips , & M. Bosnjak (Eds.), Online social sciences (pp. 69-80). Seattle, WA: Hogrefe & Huber.
22.
Reips, U.-D. ( 2002b). Internet-based psychological experimenting: Five dos and five don’ts. Social Science Computer Review, 20, 241-249.
23.
Reips, U.-D. ( 2006). Web-based methods. In M. Eid & E. Diener (Eds.), Handbook of multimethod measurement in psychology (pp. 73-85). Washington, DC: American Psychological Association.
24.
Reips, U.-D. ( 2007). The methodology of Internet-based experiments. In A. N. Joinson, K. Y. A. McKenna, T. Postmes, & U.-D. Reips (Eds.), Oxford handbook of Internet psychology (pp. 373-390). Oxford, UK: Oxford University Press.
25.
Reips, U.-D. ( 2010). Design and formatting in Internet-based research. In S. Gosling & J. Johnson (Ed.), Advanced Internet methods in the behavioral sciences (pp. 29-43). Washington, DC: American Psychological Association.
26.
Reips, U.-D., & Funke, F. ( 2008). Interval level measurement with visual analogue scales in Internet-based research: VAS Generator. Behavior Research Methods, 40, 699-704.
27.
Schmidt, W.C. ( 2007). Technical considerations when implementing online research . In A. N. Joinson, K. Y. A. McKenna, T. Postmes, & U.-D. Reips (Eds.), Oxford handbook of Internet psychology (pp. 459-470). Oxford, UK: Oxford University Press.
28.
Scott, J., & Huskisson, E.C. (1979). Vertical or horizontal visual analogue scales. Annals of the Rheumatic Diseases, 38, 560.
29.
Smyth, J.D., Dillman, D.A., Christian, L.M., & Stern, M.J. ( 2006). Effects of using visual design principles to group response options in Web surveys. International Journal of Internet Science, 1, 5-15.
30.
Sudman, S., Bradburn, N.M., & Schwarz, N. ( 1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass.
31.
Tourangeau, R., Couper, M.P., & Conrad, F.G. ( 2004). Spacing, position, and order: Interpretative heuristics for visual features of survey questions. Public Opinion Quarterly , 68, 368-393.
32.
Tourangeau, R., Couper, M.P., & Conrad, F.G. ( 2007). Color, labels, and interpretative heuristics for response scales. Public Opinion Quarterly, 71, 91-112.
33.
Tukey, J.W. ( 1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
34.
United Nations Educational, Scientific and Cultural Organization (1997). International standard classification of education - 1997 version. Retrieved July 16, 2008, from http://www.uis.unesco.org/TEMPLATE/pdf/isced/ISCED_A.pdf
