Abstract
This study assesses the effect of response-mode choices on response rates, and responsemode preferences of hard-to-survey populations: young adults, full-time workers, big city inhabitants, and non-Western immigrants. Using address-based sampling, a stratified sample of 3,496 households was selected. The first group of sample members was contacted face to face and could choose between a CAPI and web response mode. The second group, contacted by telephone, could choose between CATI and web. The third group, contacted by telephone, was randomly allocated to a response mode. Our address-based sampling technique was successful in reaching most of the hard-to-survey groups. Insufficient numbers of non- Western immigrants were reached; therefore this group was excluded from our analyses. In our mixed-effect models, no significant effects on the willingness to participate were found for mode choice. We found that full-time workers and young adults were significantly more likely to choose web over CAPI when contacted face to face.
References
1.
Ament, P. (2008). Most People on Long-Term Low Incomes Live in Major Cities. Statistics Netherlands. Available at: http://www.cbs.nl (accessed August 2013).
2.
American Association for Public Opinion Research (2011). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. Ann Arbor, MI: AAPOR.
3.
Bates, D. (2005). Fitting Linear Mixed Models in R. R News, 5, 27-30.
4.
Bethlehem, J., Cobben, F., and Schouten, B. (2011). Handbook of Nonresponse in Household Surveys. Hoboken, NJ: Wiley.
5.
Blohm, M. and Diehl, C. (2001). Wenn Migranten Migranten befragen Zum Teilnahmeverhalten von Einwanderern bei Bevo¨lkerungsbefragungen. Zeitschrift fu¨ r Soziologie, 30, 223-242.
6.
Blumberg, S.J. and Luke, J.V. (2007). Coverage Bias in Traditional Telephone Surveys of Low-Income and Young Adults. Public Opinion Quarterly, 71, 734-749. DOI: http://www.dx.doi.org/10.1093/poq/nfm047
7.
Brady, S.E., Stapleton, C.N., Bouffard, J.A., and Imel, J.D. (2003). Effect of Alternative Data Collection Modes on Cooperation Rates and Data Quality. Proceedings of the American Statistical Association, Joint Statistical Meetings, Section on Survey Research Methods, 693-700, San Francisco, August 3-7, 2003, http://www.amstat.org/ sections/srms/Proceedings/ (accessed November 2013).
8.
Brøgger, J., Nystad, W., Cappelen, I., and Bakke, P. (2007). No Increase in Response Rate by Adding a Web Response Option to a Postal Population Survey: A Randomized Trial. Journal of Medical Internet Research, 9(5), e40. DOI: http://www.dx.doi.org/10.2196/ jmir.9.5.e40
9.
Campanelli, P., Sturgis, P., and Purdon, S. (1997). Can you Hear Me Knocking: an Investigation into the Impact of Interviewers on Survey Response Rates. London: The Survey Methods Centre SCPR.
10.
Conrad, F.G., Schober, M.F., Zhang, C., Yan, H.G., Vickers, L., Johnston, M., Hupp, A., Hemingway, L., Fail, S., Ehlen, P., and Antoun, C. (2013). Mode Choice on an iPhone Increases Survey Data Quality. Paper presented at the Annual Conference of the American Association for Public Opinion Research, Boston., May 16-19, 2013.
11.
Couper, M.P. and Groves, R.M. (1996). Social Environmental Impacts on Survey Cooperation. Quality & Quantity, 30, 173-188. DOI: http://www.dx.doi.org/10.1007/ BF00153986
12.
Couper, M.P., Kapteyn, A., Schonlau, M., and Winter, J. (2007). Noncoverage and Nonresponse in an Internet Survey. Social Science Research, 36, 131-148. DOI: http://www.dx.doi.org/10.1016/j.ssresearch.2005.10.002
13.
De Gregorio, J. and Lee, J.W. (2002). Education and Income Inequality: New Evidence from Cross-Country Data. Review of Income and Wealth, 48, 395-416. DOI: http://www.dx.doi.org/10.1111/1475-4991.00060
14.
De Leeuw, E.D. (2005). To Mix or Not to Mix Data Collection Modes in Surveys.Journal of Official Statistics, 21, 233-255.
15.
De Leeuw, E.D. and Hox, J.J. (1998). Nonrespons in Surveys: Een Overzicht. Kwantitatieve Methoden, 19, 31-53.
16.
De Leeuw, E.D. and van der Zouwen, J. (1992). Data Quality and Mode of Data Collection: Methodology and Explanatory Model. La qualite´ de l’information dans les enquetes, L. Lebart (ed.). Paris: Dunod, 11-31.
17.
Deutskens, E., Ruyter, K., Wetzels, M., and Oosterveld, P. (2004). Response Rate and Response Quality of Internet-Based Surveys: An Experimental Study. Marketing Letters, 15, 21-36. DOI: http://www.dx.doi.org/10.1023/B:MARK.000002196886465.00
18.
Dhar, R. (1997). Consumer Preference for a No-Choice Option. Journal of Consumer Research, 24, 215-231.
19.
Dillman, D.A. (2007). Mail and Internet Surveys: The Tailored Design Method. Hoboken, NJ: Wiley.
20.
Dillman, D.A., Phelps, G., Tortora, R., Swift, K., Kohrell, K., Berck, J., and Messer, B.L. (2009). Response Rate and Measurement Differences in Mixed-Mode Surveys Using Mail, Telephone, Interactive Voice Response (IVR) and the Internet. Social Science Research, 38, 1-18.
21.
Dillman, D.A., West, K.K., and Clark, J.R. (1994). Influence of an Invitation to Answer by Telephone on Response to Census Questionnaires. Public Opinion Quarterly, 58, 557-568. DOI: http://www.dx.doi.org/10.1086/269447
22.
Diment, K. and Garrett-Jones, S. (2007). How Demographic Characteristics Affect Mode Preference in a Postal/Web Mixed Mode Survey of Australian Researchers. Social Science Computer Review, 25, 410-417. DOI: http://www.dx.doi.org/10.1177/0894439306295393
23.
Durrant, G.B. and Steele, F. (2009). Multilevel Modeling of Refusal and Non-Contact in Household Surveys: Evidence from Six UK Government Surveys. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172, 361-381. DOI: http:// www.dx.doi.org/10.1111/j.1467-985X.2008.00565.x
24.
Feskens, R.C.W. (2009). Difficult Groups in Survey Research and the Development of Tailor-Made Approach Strategies. Utrecht: University of Utrecht.
25.
Feskens, R.C.W., Hox, J.J., Lensvelt-Mulders, G.J.L.M., and Schmeets, J.J.G. (2007). Non-Response Among Ethnic Minorities: a Multivariate Analysis. Journal of Official Statistics, 23, 387-408.
26.
Feskens, R.C.W., Hox, J.J., Lensvelt-Mulders, G.J.L.M., and Schmeets, J.J.G. (2006). Collecting Data Among Ethnic Minorities in an International Perspective. Field Methods, 18, 284-304. DOI: http://www.dx.doi.org/10.1177/1525822X06288756
27.
Friese, C.R., Lee, C.S., O’Brien, S., and Crawford, S.D. (2010). Multi-Mode and Method Experiment in a Study of Nurses. Survey Practice, 3. Available at: http://surveypractice.org/index.php/SurveyPractice/issue/view/32 (accessed August 2013).
28.
Gentry, R. and Good, C. (2008). Offering Respondents a Choice of Survey Mode: Use Patterns of an Internet Response Option in a Mail Survey. Paper presented at the Annual Conference of the American Association for Public Opinion Research, New Orleans., May 15-18, 2008.
29.
Gillian, E., Loosveldt, G., Lynn, P., Martin, P., Revilla, M., Saris, W., and Vannieuwenhuyze, J. (2010). ESS Prep6 - Mixed-Mode Experiment. Deliverable 21
30.
Final Mode Report. Available at: www.europensocialsurvey.org Griffin, D.H., Fischer, D.P., and Morgan, M.T. (2001). Testing an Internet Response Option for the American Community Survey. Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Montreal., May 17-20, 2001.
31.
Groves, R.M. (1977). An Experimental Comparison of National Telephone and Personal.Interview Surveys. Proceedings of the Section on Social Statistics: American Statistical Association, 232-241.
32.
Groves, R.M., Cialdini, R.B., and Couper, M.P. (1992). Understanding the Decision to Participate in a Survey. Public Opinion Quarterly, 56, 475-495. DOI: http://www.dx.doi.org/10.1086/269338
33.
Groves, R.M. and Couper, M.P. (1998). Nonresponse in Household Interview Surveys. New York: Wiley.
34.
Groves, R.M., Dillman, D.A., Eltinge, J.L., and Little, R.J.A. (2002). Survey Nonresponse. New York: Wiley.
35.
Groves, R.M. and Kahn, R.L. (1979). Surveys by Telephone: A National Comparison with Personal Interviews. New York: Academic Press.
36.
Goyder, J. (1987). The Silent Minority. Nonsample Members on Sample Surveys. Cambridge: Polity Press.
37.
Goyder, J., Lock, J., and McNair, T. (1992). Urbanization Effects on Survey Non- Response: A Test Within and Across Cities. Quality and Quantity, 26, 39-48. Haan, M. and Ongena, Y.P. (2014). Tailored and Targeted Designs for Hard-to-Survey Populations. In Hard to Survey Populations, R. Tourangeau et al. (eds). Cambridge: Cambridge University Press. (in press).
38.
Hardigan, P.C., Succar, C.T., and Fleischer, J.M. (2012). An Analysis of Response Rate and Economic Costs Between Mail and Web-Based Surveys Among Practicing Dentists: A Randomized Trial. Journal of Community Health, 37, 383-394. DOI: http:// www.dx.doi.org/10.1007/s10900-011-9455-6
39.
Hoffer, T., Grigorian, K., and Fesco, R. (2007). Effectiveness of Using Respondent Mode Preference Data. Paper presented at the Joint Statistical Meetings of the American Statistical Association, Salt Lake City., July 29-August 2, 2007.
40.
Holbrook, A.L., Green, M.C., and Krosnick, J.A. (2003). Telephone vs. Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias. Public Opinion Quarterly, 67, 79-125. DOI: http://www.dx.doi.org/10.1086/346010
41.
Hox, J.J. and de Leeuw, E.D. (1994). A Comparison of Nonresponse in Mail, Telephone, and Face-to-Face Surveys. Quality and Quantity, 28, 329-344. DOI: http://www. dx.doi.org/10.1007/BF01097014 Israel, G.D. (2010). Using Web-Hosted Surveys to Obtain Responses from Extension Clients: A Cautionary Tale. Journal of Extension, 48, http://www.joe.org/joe/2010august/a8.php (accessed November 2013).
42.
Iyengar, S.S. and Lepper, M.R. (2000). When Choice Is Demotivating: Can One Desire Too Much of a Good Thing? Journal of Personality and Social Psychology, 79, 995-1006. DOI: http://www.dx.doi.org/10.1037/0022-3514.79.6.995
43.
Kaplowitz, M.D., Hadlock, T.D., and Levine, R. (2004). A Comparison of Web and Mail Survey Response Rates. Public Opinion Quarterly, 68, 94-101. DOI: http:// www.dx.doi.org/10.1093/poq/nfh006
44.
Lesser, V.M., Newton, L., and Yang, D. (2010). Does Providing a Choice of Survey Modes Influence Response? Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Chicago., May 13-16, 2010.
45.
Li, B., Lingsma, H.F., Steverberg, E.W., and Lesaffre, E. (2011). Logistic Random Effects Regression Models: A Comparison of Statistical Packages for Binary and Ordinal Outcomes. BMC Medical Research, 11, Article 77. DOI: http://www.dx.doi.org/10.1186/1471-2288
46.
Loges, W.E. and Jung, J. (2001). Exploring the Digital Divide: Internet Connectedness and Age. Communication Research, 28, 536-562. DOI: http://www.dx.doi.org/10.1177/009365001028004007
47.
Lynn, P. (2013). Alternative Sequential Mixed-Mode Designs: Effects on Attrition Rates, Attrition Bias, and Costs. Journal of Survey Statistics and Methodology, 1, 183-205. DOI: http://www.dx.doi.org/10.1093/jssam/smt015
48.
Martin, P. (2011). What Makes a Good Mix? Chances and Challenges of Mixed Mode Data Collection in the ESS. Working Paper No. 02. Centre for Comparative Social Surveys, City University, London.
49.
Medway, R.L. and Fulton, J. (2012). When More Gets You Less: a Meta-Analysis of the Effect of Concurrent Web Options on Mail Survey Response Rates. Public Opinion Quarterly, 76, 733-746. DOI: http://www.dx.doi.org/10.1093/poq/nfs047
50.
Millar, M.M. and Dillman, D.A. (2011). Improving Response to Web and Mixed-Mode Surveys. Public Opinion Quarterly, 75, 249-269. DOI: http://www.dx.doi.org10.1093/poq/nfr003
51.
Millar, M.M., O’Neill, A.C., and Dillman, D.A. (2009). Are Mode Preferences Real? Technical Report 09-003, Social and Economic Sciences Research Center. Pullman: Washington State University.
52.
Miller, T.I., Kobayashi, M.M., Caldwell, E., Thurston, S., and Collett, B. (2002). Citizen Surveys on the Web: General Population Surveys of Community Opinion. Social Science Computer Review, 20, 124-136. DOI: http://www.dx.doi.org/10.1177/089443930202000203
53.
Nicolaas, H., Wobma, E., and Ooijevaar, J. (2010). Demografie van (Niet-Westerse) Allochtonen in Nederland. Statistics Netherlands. Available at: http://www.cbs.nl (accessed August 2013).
54.
Olson, K., Smyth, J.D., and Wood, H. (2012). Does Providing Sample Members with Their Preferred Survey Mode Really Increase Participation Rates? Public Opinion Quarterly, 76, 611-635.
55.
Porter, S.R. and Whitcomb, M.E. (2007). Mixed-Mode Contacts in Web Surveys: Paper Is Not Necessarily Better. Public Opinion Quarterly, 71, 635-648. DOI: http://www.dx.doi.org/10.1093/poq/nfm038
56.
Quene´, H. and van den Bergh, H. (2008). Examples of Mixed-Effects Modeling With Crossed Random Effects and With Binomial Data. Journal of Memory and Language, 59, 413-425. DOI: http://www.dx.doi.org/10.1016/j.jml.2008.02.002
57.
Radon, K., Goldberg, M., Becklake, M., Pindur, U., Hege, I., and Nowak, D. (2002).Low Acceptance of an Internet-Based Online Questionnaire by Young Adults. Epidemiology, 13, 748-749.
58.
Raets, B. (2008). Vinex-Bewoners zijn Geen Doorsnee Stedelingen. Statistics Netherlands. Available at: http://www.cbs.nl (accessed August 2013).
59.
Ryan, J.M., Corry, J.R., Attewell, R., and Smithson, M.J. (2002). A Comparison of an Electronic Version of the SF-36 General Health Questionnaire to the Standard Paper Version. Quality of Life Research, 11, 19-26. DOI: http://www.dx.doi.org10.1023/A:1014415709997
60.
Scherpenzeel, A. and Toepoel, V. (2012). Recruiting a Probability Sample for an Online Panel. Effects of Contact Mode, Incentives and Information. Public Opinion Quarterly, 76, 470-490. DOI: http://www.dx.doi.org/10.1093/poq/nfs037
61.
Schmuhl, P., van Duker, H., Gurley, K.L., Webster, A., and Olson, L. (2010). Reaching Emergency Medical Services Providers: Is One Survey Mode Better Than Another? Prehospital Emergency Care, 14, 361-369.
62.
Schneider, S.J., Cantor, D., Malakhoff, L., Arieira, C., Segel, P., Nguyen, K., and Tancreto, J.G. (2005). Telephone, Internet and Paper Data Collection Modes for the Census 2000 Short Form. Journal of Official Statistics, 21, 89-101.
63.
Schwartz, B. (2004). The Paradox of Choice: Why More Is Less. New York: Harper Perennial.
64.
Shih, T. and Fan, X. (2007). Response Rates and Mode Preferences in Web-Mail Mixed- Mode Surveys: A Meta-Analysis. International Journal of Internet Science, 2, 59-82.
65.
Sinclair, M., O’Toole, J., and Malawaraarachchi, M. (2012). Comparison of Response Rates and Cost-Effectiveness for a Community-Based Survey: Postal, Internet and Telephone Modes with Generic or Personalized Recruitment Approaches. BMC Medical Research Methodology, 12, Article 132. DOI: http://www.dx.doi.org/10.1186/1471-2288-12-132
66.
Smyth, J.D., Dillman, D.A., Christian, L.M., and O’Neill, A.C. (2010). Using the Internet to Survey Small Towns and Communities: Limitations and Possibilities in the Early 21st Century. American Behavioral Scientist, 53, 1423-1448. DOI: http://www.dx.doi.org/10.1177/0002764210361695
67.
Smyth, J.D., Olson, K., and Richards, A. (2009). Are Mode Preferences Real? Paper presented at the Annual Meeting of the American Association for Public Opinion Research. Hollywood, Florida., May 14-17, 2009.
68.
Statistics Netherlands. (2013a). Definitions. Available at: http://www.cbs.nl (accessed November 2013).
69.
Statistics Netherlands. (2013b). ICT Gebruik van Huishoudens naar Huishoudkenmerken Available at: http://statline.cbs.nl (accessed November 2013).
70.
Statistics Netherlands. (2010). Laag en Langdurig Laag Inkomen; Particuliere Huishoudens naar Kenmerken. Available at: http://statline.cbs.nl (accessed August 2013).
71.
Stoop, I. (2007). No time, Too Busy: Time Strain and Survey Cooperation. In Measuring Meaningful Data in Social Research, G. Loosveldt, M. Swyngedouw, and B. Cambre´ (eds). Leuven: Acco, 301-314.
72.
Stoop, I. (2005). The Hunt for the Last Respondent. Non-Response in Sample Surveys. The Hague: Social and Cultural Planning Agency.
73.
Sylvester, D.E. and McGlynn, A.J. (2010). The Digital Divide, Political Participation, and Place. Social Science Computer Review, 28, 64-74. DOI: http://www.dx.doi. org/10.1177/0894439309335148
74.
Tancreto, J.G., Zelenak, M.F., Davis, M., Ruiter, M., and Matthews, B. (2012). 2011 American Community Survey Internet Tests: Results from First Test in April 2011. Final Report. Washington, DC: US Census Bureau.
75.
Tarnai, J. and Paxson, M.C. (2004). Survey Mode Preferences of Business Respondents. Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Phoenix., May 13-16, 2004.
76.
Toffler, A. (1971). Future Shock. United States: Bantam Books.
77.
Tse, A. (1998). Comparing the Response Rate, Response Speed, and Response Quality of Two Methods of Sending Questionnaires: Email vs. Mail. Journal of the Market Research Society, 40, 353-361.
78.
Turner, S., Viera, L., and Marsh, S. (2010). Offering a Web Option in a Mail Survey of Young Adults: Impact on Survey Quality. Poster presented at the Annual Meeting of the American Association for Public Opinion Research, Chicago., May 13-16, 2010.
79.
Vannieuwenhuyze, J. (forthcoming 2014). On the Relative Advantage of Mixed-Mode Surveys. Survey Research Methods.
80.
Vehovar, V., Batagelj, Z., Lozar Manfreda, K., and Zalatel, M. (2002). Nonresponse in Web Surveys. In Survey Nonresponse, R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little (eds). New York: John Wiley and Sons, 229-242.
81.
Vercruyssen, A., Roose, H., Carton, A., and van de Putte, B. (2013). The Effect of Busyness on Survey Participation: Being Too Busy or Feeling Too Busy to Cooperate? International Journal of Social Research Methodology. DOI: http://www.dx.doi.org/10.1080/13645579.2013.799255
82.
Weeks, M.F., Kulka, R.A., Lessler, J.T., and Whitmore, R.W. (1983). Personal versus Telephone Surveys for Collecting Household Health Data at the Local Level. American Journal of Public Health, 73, 1389-1394. DOI: http://www.dx.doi.org/10.2105/AJPH73.12.1389
83.
Werner, P. and Forsman, G. (2005). Mixed Mode Data Collection Using Paper and Web Questionnaires. Proceedings of the American Statistical Association, Section on Survey Research Methods, 4015-4017.
84.
West, B.T. (2012). An Examination of the Quality and Utility of Interviewer Observations in the National Survey of Family Growth. Journal of the Royal Statistical Society: Series A (Statistics in Society), 176, 211-225. DOI: http://www.dx.doi.org/10.1111/ j.1467-985X.2012.01038.x
85.
Wilkins, J.R., Hueston, W.D., MacCrawford, J., Steele, L.L., and Gerken, D.F. (1997). Mixed-Mode Survey of Female Veterinarians Yields High Response Rate.Occupational Medicine, 47, 458-462. DOI: http://www.dx.doi.org/10.1093/occmed/47.8.458
86.
Ziegenfuss, J.Y., Beebe, T.J., Rey, E., Schleck, C., Locke, III, G.R., and Talley, N.J. (2010). Internet Option in a Mail Survey: More Harm Than Good? Epidemiology, 21, 585-586. DOI: http://www.dx.doi.org/10.1097/EDE.0b013e3181e09657
87.
Zickuhr, K. and Smith, A. (2012). Digital Differences. Pew Internet Project Report.Washington, DC: Pew Research Center. Available at: http://www.pewInternet.org/Reports/2012/Digital-differences.aspx (accessed August 2013).
88.