Remember,You Can Complete This Survey Online! Web Survey Links and QR Codes in a Mixed-Mode Web and Mail General Population Survey

Abstract

Recruitment materials for concurrent mixed-mode self-administered web and mail surveys must communicate information about multiple modes simultaneously. Providing the link to the web survey on the cover of the paper questionnaire or including a QR code to access the web survey may increase the visibility of the web mode and thus increase the proportion of people who participate via the web, but whether including either piece of information does so has received surprisingly little empirical attention. In this paper, we examine the results of experiments embedded in two general population probability-based concurrent mixed-mode surveys of Nebraska adults. First, in the Labor Availability Survey of the Greater Omaha Area, respondents were randomly assigned to receive the web link and login information on the cover or the paper questionnaire without this information (all had web information in the cover letter). We then replicated and extended this experiment in the Labor Availability Survey of Northeast Nebraska. The questionnaire cover experiment was fully crossed with the presence or absence of a QR code to access the web survey. Neither of these design features affected response rates or speed of response, but the link on the questionnaire significantly increased the proportion of respondents who participated by web and the QR code significantly increased the proportion of respondents who participated by smartphone. Sample composition was largely unaffected on most characteristics, although the respondent pool was less similar to the population on education when the link was on the questionnaire. About 20% of respondents used a smartphone when typing in a survey link, but virtually all respondents used a smartphone when scanning the QR code. Survey researchers can include a link on the cover of the questionnaire to increase web participation rates in mixed-mode surveys. QR codes can be used when smartphone participation is desired.

Keywords

QR codes web surveys mixed-mode surveys research methodologies survey methodology

Get full access to this article

View all access options for this article.

References

Antoun

Couper

M. P.

Conrad

F. G.

(2017). Effects of mobile versus PC web on survey response quality: A crossover experiment in a probability web panel. Public Opinion Quarterly, 81(S1), 280–306. https://doi.org/10.1093/poq/nfw088

Biemer

P. P.

Murphy

Zimmer

Berry

Deng

Lewis

(2018). Using bonus monetary incentives to encourage web response in mixed-mode household surveys. Journal of Survey Statistics and Methodology, 6(2), 240–261. https://doi.org/10.1093/jssam/smx015

Cohen

(1988). Statistical power analysis (2nd ed.). Erlbaum.

Couper

M. P.

(2017). New developments in survey data collection. Annual Review of Sociology, 43(1), 121–145. https://doi.org/10.1146/annurev-soc-060116-053613

Couper

M. P.

Antoun

Mavletova

(2017). Mobile web surveys: A total survey error perspective. In Biemer

P. P.

De Leeuw

Eckman

Edwards

Kreuter

Lyberg

L. E.

Tucker

West

B. T.

(Eds.), Total survey error in practice (pp. 133–154). John Wiley & Sons.

Davies

Hjorth

Andrejevic

Richardson

DeSouza

(2023). QR codes during the pandemic: Seamful quotidian placemaking. Convergence, 29(5), 1121–1135. https://doi.org/10.1177/13548565231160623

de Bruijne

Wijnant

(2013). Comparing survey results obtained via mobile devices and computers: An experiment with a mobile web survey on a heterogeneous group of mobile devices versus a computer-assisted web survey. Social Science Computer Review, 31(4), 482–504. https://doi.org/10.1177/0894439313483976

de Bruijne

Wijnant

(2014). Mobile response in web panels. Social Science Computer Review, 32(6), 728–742. https://doi.org/10.1177/0894439314525918

Delnevo

C. D.

Singh

(2021). The effect of a web-push survey on physician survey responses rates: A randomized experiment. Survey Practice, 14(1), 1–9. https://doi.org/10.29115/sp-2021-0001

10.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2014). Internet, phone, mail, and mixed mode surveys: The tailored design method. John Wiley & Sons.

11.

Endres

Heiden

E. O.

Park

Losch

M. E.

Harland

K. K.

Abbott

A. L.

(2023). Experimenting with QR codes and envelope size in push-to-web surveys. Journal of Survey Statistics and Methodology, smad008. https://doi.org/10.1093/jssam/smad008

12.

Faggiano

Carugo

(2020). Can the implementation of electronic surveys with quick response (QR) codes be useful in the COVID-19 era? International Journal of Epidemiology, 49(5), 1732–1733. https://doi.org/10.1093/ije/dyaa170

13.

Gelman

Stern

(2006). The difference between “Significant” and “Not Significant” is not itself statistically significant. The American Statistician, 60(4), 328–331. https://doi.org/10.1198/000313006X152649

14.

Gostin

(2021). How the pandemic saved the QR code from extinction. Forbes. https://www.forbes.com/sites/forbescommunicationscouncil/2021/03/25/how-the-pandemic-saved-the-qr-code-from-extinction/?sh=337c615d6905

15.

Greenland

Senn

S. J.

Rothman

K. J.

Carlin

J. B.

Poole

Goodman

S. N.

Altman

D. G.

(2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. https://doi.org/10.1007/s10654-016-0149-3

16.

Griffin

D. H.

Fischer

D. P.

Morgan

M. T.

(2001). Testing an internet response option for the American community survey. In Annual conference of the American association for public opinion research. Montreal, Quebec, Canada.

17.

Groves

R. M.

Singer

Corning

(2000). Leverage-saliency theory of survey participation: Description and an illustration. Public Opinion Quarterly, 64(3), 299–308. https://doi.org/10.1086/317990

18.

Haan

Lugtig

Toepoel

(2019). Can we predict device use? An investigation into mobile device use in surveys. International Journal of Social Research Methodology, 22(5), 517–531. https://doi.org/10.1080/13645579.2019.1593340

19.

Harrison

Henderson

Alderdice

Quigley

M. A.

(2019). Methods to increase response rates to a population-based maternity survey: A comparison of two pilot studies. BMC Medical Research Methodology, 19(1), 65. https://doi.org/10.1186/s12874-019-0702-3

20.

Hill

(2021). How to scan a QR code. Wired. https://www.wired.com/story/how-to-scan-a-qr-code/

21.

Jackson

M. T.

Medway

R. L.

Megra

M. W.

(2023). Can appended auxiliary data be used to tailor the offered response mode in cross-sectional studies? Evidence from an address-based sample. Journal of Survey Statistics and Methodology, 11(1), 47–74. https://doi.org/10.1093/jssam/smab023

22.

Keusch

Conrad

F. G.

(2021). Using smartphones to capture and combine self-reports and passively measured behavior in social research. Journal of Survey Statistics and Methodology, 10(4), 863–885. https://doi.org/10.1093/jssam/smab035

23.

Lesser

V. M.

Newton

L. D.

Yang

D. K.

Sifneos

J. C.

(2016). Mixed-mode surveys compared with single mode surveys: Trends in responses and methods to improve completion. Journal of Rural Social Sciences, 31(3), 7–34. https://egrove.olemiss.edu/jrss/vol31/iss3/2.

24.

Link

M. W.

Murphy

Schober

M. F.

Buskirk

T. D.

Childs

J. H.

Tesfaye

C. L.

(2014). Mobile technologies for conducting, augmenting and potentially replacing surveys.

25.

Long

J. S.

Freese

(2014). Regression models for categorical dependent variables using Stata (3rd ed.). Stata Press.

26.

Lugtig

Luiten

(2021). Do shorter stated survey length and inclusion of a QR code in an invitation letter lead to better response rates? Survey Methods: Insights From the Field. https://surveyinsights.org/?p=14216

27.

Lugtig

Toepoel

Amin

(2016). Mobile-only web survey respondents. Survey Practice, 9(4), 1–8. https://doi.org/10.29115/sp-2016-0020

28.

Lugtig

Toepoel

Haan

Zandvliet

Kranenburg

L. K.

(2019). Recruiting young and urban groups into a probability-based online panel by promoting smartphone use. Methods, Data, Analyses, 13(2), 291–306. https://doi.org/10.12758/mda.2019.04

29.

Marlar

(2018). Do quick response codes enhance or hinder surveys? Gallup. https://news.gallup.com/opinion/methodology/241808/quick-response-codes-enhance-hinder-surveys.aspx?version=print

30.

Marlar

Schreiner

Sullivan

(2023). QR codes for survey access: Revisiting previous findings. In 78th annual meeting of the American association for public opinion research. AAPOR.

31.

Medway

Scardaville

Paek

Dias

Kaiser

Megra

Pulizzi

(2022). National household education surveys program of 2019: Qualitative study of nonresponding addresses. Institute of Education Sciences, National Center for Education Statistics. Retrieved from. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2022043

32.

Medway

R. L.

Fulton

(2012). When more gets you less: A meta-analysis of the effect of concurrent web options on mail survey response rates. Public Opinion Quarterly, 76(4), 733–746. https://doi.org/10.1093/poq/nfs047

33.

Messer

B. L.

Dillman

D. A.

(2011). Surveying the general public over the internet using address-based sampling and mail contact procedures. Public Opinion Quarterly, 75(3), 429–457. https://doi.org/10.1093/poq/nfr021

34.

Millar

M. M.

Dillman

D. A.

(2011). Improving response to web and mixed-mode surveys. Public Opinion Quarterly, 75(2), 249–269. https://doi.org/10.1093/poq/nfr003

35.

Olson

Smyth

J. D.

Horwitz

Keeter

Lesser

Marken

Mathiowetz

N. A.

McCarthy

O’Brien

E. M.

Opsomer

Steiger

D. M.

Sterrett

Suzer-Gurtekin

Z. T.

Turakhia

Wagner

(2019). Transitions from telephone surveys to self-administered and mixed-mode surveys. https://www.aapor.org/Education-Resources/Reports/Transitions-from-Telephone-Surveys-to-Self-Adminis.aspx

36.

Olson

Smyth

J. D.

Horwitz

Keeter

Lesser

Marken

Mathiowetz

N. A.

McCarthy

J. S.

O’Brien

Opsomer

J. D.

Steiger

Sterrett

Suzer-Gurtekin

Z. T.

Turakhia

Wagner

(2021). Transitions from telephone surveys to self-administered and mixed-mode surveys: Aapor task force report. Journal of Survey Statistics and Methodology, 9(3), 381–411. https://doi.org/10.1093/jssam/smz062

37.

Olson

Smyth

J. D.

Wood

(2012). Does giving people their preferred survey mode actually increase survey participation rates? Public Opinion Quarterly, 76(4), 611–635. https://doi.org/10.1093/poq/nfs024

38.

Patrick

M. E.

Couper

M. P.

Jang

B. J.

Laetz

Schulenberg

J. E.

O’Malley

P. M.

Bachman

Johnston

L. D.

(2020). Building on a sequential mixed-mode research design in the monitoring the future study. Journal of Survey Statistics and Methodology, 10(1), 149–160. https://doi.org/10.1093/jssam/smaa030

39.

Pew Research Center . (2024). Mobile fact sheet. Pew Research Center. Retrieved February 2, 2024 from. https://www.pewresearch.org/internet/fact-sheet/mobile/

40.

Peytchev

Hill

C. A.

(2010). Experiments in mobile web survey design: Similarities to other modes and unique considerations. Social Science Computer Review, 28(3), 319–335. https://doi.org/10.1177/0894439309353037

41.

Rivas

A. G.

Schulzetenberg

(2023). QR codes as a method for older adults to access a mobile survey. In Human aspects of IT for the aged population.

42.

Ruiter

Zelenak

M. F.

Davis

M. C.

Horwitz

Tancreto

J. G.

(2012). Testing mail notification strategies for an internet response option in the American community survey. In Annual meeting of the American association for public opinion research.

43.

Schwede

(2013). Cognitive testing of modified American community survey envelopes and letters for use during the 2010 census: ACS messaging project final report on phases 1 and 2. US Department of Commerce, US Census Bureau. Retrieved from. https://www.census.gov/content/dam/Census/library/working-papers/2013/acs/2013_Schwede_01.pdf

44.

Smyth

J. D.

Dillman

D. A.

Christian

L. M.

O'Neill

A. C.

(2010). Using the internet to survey small towns and communities: Limitations and possibilities in the early 21st century. American Behavioral Scientist, 53(9), 1423–1448. https://doi.org/10.1177/0002764210361695

45.

Smyth

J. D.

Olson

Millar

M. M.

(2014). Identifying predictors of survey mode preference. Social Science Research, 48(0), 135–144. https://doi.org/10.1016/j.ssresearch.2014.06.002

46.

Smyth

J. D.

Olson

Stange

(2019). Within-household selection methods: A critical review and experimental examination. In Experimental methods in survey research (pp. 23–45). John Wiley & Sons. https://doi.org/10.1002/9781119083771.ch2

47.

Tancreto

J. G.

Zelenak

M. F.

Davis

Ruiter

Matthews

(2012). 2011 American community survey internet tests: Results from first test in April 2011. US Department of Commerce; US Census Bureau.

48.

U.S. Census Bureau . (2022a). American community survey, ACS 1-year estimates subject tables, Table S1501: Educational attainment. Retrieved from. https://data.census.gov/table/ACSST1Y2022.S1501

49.

U.S. Census Bureau . (2022b). American community survey, ACS 5-year estimates subject tables, Table S0101 age and sex. Retrieved from. https://data.census.gov/table/ACSST1Y2022.S0101?q=Age_and_Sex

50.

U.S. Census Bureau . (2022c). Types of computers and internet subscriptions. https://data.census.gov/table/ACSST1Y2022.S2801?q=Telephone,+Computer,+and+Internet+Access

51.

Zelenak

M. F.

Ruiter

Tancreto

J. G.

Davis

(2012). Results from the April 2011 Puerto Rico community survey internet test. US Department of Commerce, US Census Bureau. Retrieved from. https://www.census.gov/content/dam/Census/library/working-papers/2012/acs/2012_Zelenak_01.pdf

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB