Displaying Videos in Web Surveys

Abstract

Videos are often used in web surveys to assess attitudes. While including videos may allow researchers to test immediate reactions, there may be issues associated with displaying videos that are overlooked. In this article, we examine the effects of using video stimuli on responses in a probability-based web survey. Specifically, we evaluate the association between demographics, mobile device usage, and the ability to view videos; differences in ad recall based on whether respondents saw a video or still images of the video; whether respondents’ complete viewing of videos is related to presentation order; and the data quality of follow-up questions to the videos as a function of presentation order and complete viewing. Overall, we found that respondents using mobile browsers were less likely to be able to view videos in the survey. Those who could view videos were more likely to indicate recall compared to those who viewed images, and videos that were shown later in the survey were viewed in their entirety less frequently than those shown earlier. These results directly pertain to the legitimacy of using videos in web surveys to gather data about attitudes.

Keywords

web surveys measurement videos data quality advertising

Web surveys offer options, such as embedded videos, that previously were not possible with paper and telephone surveys. This has enabled researchers and marketers to test immediate reactions to videos, including the effectiveness of advertisements (e.g., Calfee, 2002; Lemert, Wanta, & Lee, 1999; Mehta, 2000) and to provide videos to respondents who may not remember them when prompted with verbal descriptions or still images. However, videos may pose several problems for web surveys (Shapiro-Luft & Cappella, 2013), and there is little existing literature on the implications for survey measurement.

One issue that may arise when using the web to conduct a survey is that respondents may have technical problems. Compatibility issues, lack of broadband access, or lack of Internet experience may affect the ability for people to view and respond to surveys accurately (Couper, 2008; Fielding, Lee, & Blank, 2008). Mobile device usage can impact responses to surveys (Buskirk & Andrus, 2014; Mavletova, 2013; Peytchev & Hill, 2010), which is of increasing importance as smartphone usage continues to grow (Anderson, 2015). Technical limitations and mobile device usage can impact how people respond to surveys, posing problems for videos in web surveys.

Respondents’ demographics can relate to how they access and respond to surveys. For example, broadband Internet access rates vary widely by sociodemographic characteristics, including income, age, and race/ethnicity (Horrigan & Duggan, 2015); this can affect a participant’s survey experience. Age, education, gender, household Internet access, and employment status also influence survey completion time, video viewing time, and video failure (Couper, Traugott, & Lamias, 2001; Groves & Couper, 1998; Peytchev, 2011; Shapiro-Luft & Cappella, 2013). The decision to display videos in web surveys may involve a choice between imperfect options, including trade-offs between internal and external validity. For example, screening out those who cannot view videos could reduce generalizability, whereas tailoring the survey instrument to a respondent’s technology platform could reduce measurement consistency if the modes elicit different responses.

Within the context of advertising research, little is known about the comparability of video-based and image-based advertising recall methodologies. In this exploratory study, we examined technical issues, device, and demographics of respondents who completed a web-based survey that included either video advertisements or images of the videos and questions about advertisement effectiveness for each video. We aimed to answer the following questions:

Question 1: What is the relationship between demographic characteristics, mobile device usage, and video viewing ability?

Question 2: How does mode affect ad recall?

Question 3: How do video viewing behavior and presentation order affect the data quality of answers to follow-up questions?

Even if respondents can be shown videos, if they do not watch the entire video, this could negatively impact responses to follow-up questions, leading to straightlining, in which a respondent selects the same option for every item in a battery, or selecting indefinite responses, such as “don’t know” or “not sure” (Krosnick, 1991). Questionnaire length can also impact data quality—satisficing behaviors tend to occur more frequently in later items of a long survey (Herzog & Bachman, 1981). Thus, we aimed to answer:

Question 4: How does the order of video presentation affect whether the videos are fully watched?

Method

Data Collection

Data were collected throughout January and March 2013 via the GfK KnowledgePanel^®. Panel members have a known probability of selection, and households that lack Internet access are provided a netbook and Internet service (Dennis, 2010).

The survey is a quarterly tracking study that measures awareness of and reactions to military advertisements among U.S. adults who regularly interact with young people.¹ Of the 11,536 panel members sampled for screening, 8,016 completed the screener (completion rate of 69.5%); of those, 2,914 qualified for the survey (incidence rate of 36.4%). Of eligible panel members, 1,803 were sampled for the survey; 1,237 completed the survey (cumulative response rate CUMRR1 = 7.0%; Callegaro & DiSogra, 2008);² after which an additional 12 observations were removed due to incomplete data, leading to a final data set of 1,225 participants.³ The survey was weighted,⁴ although an unweighted, model-based approach was used for inference.⁵

Advertising Recall Methodology

Participants were shown advertisements from four of the Active Duty Services of the U.S. Military: the Army, Navy, Air Force, and Marine Corps. Two sets of advertising recall measures were asked. The first set measured unaided advertising recall—respondents were asked questions without prompts about the ads. The second set measured aided advertising recall—respondents were shown either images with text or videos of specific advertisements. The mode in which respondents were provided ads (i.e., images or videos) depended on whether the respondents could view embedded videos.

Unaided advertising recall

First, respondents were asked, “In the past couple of months, have you seen, read or heard any advertisements for the U.S. Military or any of its branches? This does not include news coverage of military activities.” Respondents who indicated “yes” or “not sure” were then shown a list of the Active Duty Services (in randomized order) and were asked, “For which of the following have you seen, read or heard an advertisement in the past couple of months?”

Aided advertising recall

The format of the aided recall question was assigned based on whether respondents could view videos embedded in an HTML survey. The videos were in Windows Media Video format, with a resolution of 320 by 285 pixels, and automatically started playing for respondents after the page was loaded. Respondents were shown a test video and asked, “Were you able to view this video?” Respondents who could view the test video were assigned a video-based question format; otherwise, they were assigned an image-based question format.

Respondents in the video format were shown four pairs of 30-second military advertisements, one pair each for the Army, Navy, Marine Corps, and Air Force. The order of the pairs of ads was randomized, and the order within each pair was randomized. For each ad, the respondent saw the ad and was asked, “Do you recall seeing or hearing this ad in the past couple of months?” Respondents were then asked two follow-up question batteries about the ad: first, regarding their reactions to the advertisement (e.g., if the ad was visually appealing, told them something new or different, accurately portrayed people in the particular Service); second, they responded to five items regarding their attitudes after watching the ad (e.g., if the ad made them interested in finding out more). The response options for each question were “agree,” “disagree,” and “not sure.” After all eight sets of questions (i.e., one set per advertisement), respondents were asked whether they encountered technical problems while viewing the videos.

Respondents in the image viewing condition were similarly provided with advertising recall questions for the eight ads, except that instead of being shown videos, respondents were shown screen captures from key scenes and a text summary of the advertisement. Respondents who recalled seeing an advertisement were asked follow-up questions regarding their reactions and attitudes after watching the ad.

Results

Question 1: What Is the Relationship Between Demographic Characteristics, Mobile Device Usage, and Video Viewing Ability?

We used logistic regression to predict whether respondents could view videos, based on demographics and the use of a mobile device (i.e., smartphone or tablet) to access the survey. Respondents were defined as being able to view videos if they responded “yes” to “were you able to view this video?” and did not report any technical problems after being shown all advertisements. Of the 532 respondents who reported that they could view the test video (of 1,225), 46 subsequently reported technical difficulties; these respondents were coded as unable to view videos for the first analysis and were omitted from subsequent analyses.⁶

The demographic variables used in the model were gender, age (continuous), educational attainment, annual household income in U.S. dollars, race/ethnicity, and whether the respondent’s household had Internet access prior to joining the panel;⁷ the model also included likelihood to recommend military service (recoded as binary), as a possible correlate of survey interest (Table 1). Mobile device usage to access the survey was determined using the browser’s user-agent header.⁸ Nine respondents who had missing data on at least one variable were omitted from the analyses.

Table 1.

Sample Characteristics (by Respondent).

	All Respondents (n = 1225)	Video Respondents (n = 486)
Male	48.7%	54.1%
Age (mean)	50.2	49.8
Education
High school or less	33.1%	32.7%
Some college	28.1%	29.6%
College graduate	38.5%	37.7%
(Missing)	0.3%	0.0%
Household income
Less than US$25,000	17.8%	16.0%
US$25,000–US$49,999	24.0%	22.6%
US$50,000–US$74,999	19.1%	18.9%
US$75,000–US$99,999	15.1%	16.5%
US$100,000–US$124,999	12.2%	13.0%
US$125,000 or more	11.8%	13.0%
Race/ethnicity
White, non-Hispanic	72.3%	76.3%
Black, non-Hispanic	11.8%	9.7%
Hispanic	10.4%	4.1%
Other or 2+ races, non-Hispanic	5.4%	9.9%
Native Internet household	86.4%	90.7%
Mobile device used	13.7%	2.1%
Likely recommend Military	40.3%	42.4%

A logistic regression predicting viewing ability from mobile browser and demographic characteristics indicated that mobile browser users had 93% lower odds of being able to view videos than others, Exp( $\hat{β}$ ) = 0.073, z = −7.87, p < .001. Native web households had over twice the odds of being able to view videos than those who were supplied Internet by GfK, Exp( $\hat{β}$ ) = 2.185, z = 4.13, p < .001 (Table 2). Gender was also a significant predictor, with males having 36% higher odds than females of being able to view videos, Exp( $\hat{β}$ ) = 1.357, z = 2.48, p < .05. Education, income, race/ethnicity, and likelihood to recommend the military were not significant predictors of viewing ability.⁹

Table 2.

Logistic Regression Analysis Predicting Ad Viewing Ability.

	Initial Model		Final Model
Predictor	$e^{\hat{β}}$	$\hat{SE} (e^{\hat{β}})$	$e^{\hat{β}}$	$\hat{SE} (e^{\hat{β}})$
Male	1.334*	(0.167)	1.357*	(0.167)
Age (continuous)	0.990	(0.005)
Education
High school or less (ref.)
Some college	1.099	(0.179)
College graduate	0.872	(0.142)
Household income
Less than US$25,000 (ref.)
US$25,000–US$49,999	0.922	(0.185)
US$50,000–US$74,999	0.984	(0.215)
US$75,000–US$99,999	1.187	(0.280)
US$100,000–US$124,999	1.263	(0.323)
US$125,000 or more	1.326	(0.346)
Race/ethnicity
White, non-Hispanic (ref.)
Black, non-Hispanic	0.717	(0.147)
Hispanic	0.571	(0.166)
Other or 2+ races, non-Hispanic	0.887	(0.187)
Native Internet household	1.916***	(0.397)	2.185***	(0.413)
Mobile device used	0.067***	(0.022)	0.073***	(0.024)
Likely recommend military	1.192	(0.153)
Constant	0.669	(0.264)	0.359	(0.066)
Area under the ROC curve (AUROC)	.684		.660
McFadden’s Pseudo R ²	.096		.086
LR χ²	156.5		141.3

*p < .05. **p < .01. ***p < .001.

These results indicate that mobile compatibility and/or bandwidth issues could have sizable impacts on measurement for web surveys. Only 6% of respondents who completed the survey on mobile were able to view videos, compared with 45% of respondents who completed the survey on computers. Given that panel households without Internet access were generally supplied with dial-up rather than broadband, their lower rates of viewing ability may relate to Internet speed.

Question 2: How Does Mode Affect Ad Recall?

Next, we examined differences in aided ad recall (i.e., recognition) based on whether respondents were shown images or videos, using demographics and the unaided recall questions to control for possible confounds. A random effects logistic model was estimated to account for the nested nature of observations within respondents and predict dichotomous ad recall for the eight advertisements (1 = yes, 0 = no, not sure, or refused). The predictors used were demographic characteristics (limited to those that were marginally significant in the previous model), mobile device usage, Service-specific likelihood to recommend military service (as a possible attitudinal correlate of ad recall), and unaided advertising recall for the given Military Service (1 = yes, 0 = no, not sure, or refused). Although mode and recall are confounded, the use of unaided recall as a covariate may help to lessen this effect. Key sample characteristics are displayed in Table 3 (central column for observations included in the Question 2 analyses; rightmost column for the observations included in the Question 3–Question 4 analyses).

Table 3.

Sample Characteristics (by Respondent and Ad, Complete Records Only).

	All Observations (n = 9320; %)	Video Observations (n = 3836; %)
Unaided ad recall	26.5	27.0
Aided ad recall: Yes	16.4	18.4
Aided ad recall: Not sure	12.6	9.1
Likely recommend Service	31.8	33.3
Straightlined both batteries	n/a	18.5
Watched full video	n/a	79.1

Note. The table above excludes observations from individuals who encountered technical problems; similarly, it excludes observations for which there were item-missing data for any covariates used in the Question 2–Question 4 analyses.

Results indicated that being shown videos was associated with greater aided ad recall rates, $\hat{β}$ = .28, p < .01 (Table 4). In addition, respondents shown videos were more likely to give a yes or no response versus a not sure response. A random effects logistic model was estimated to predict whether respondents indicated that they were not sure that they had seen an ad, using the same predictors as the previous model. Being shown videos was associated with fewer not sure responses, $\hat{β}$ = −.60, p < .01 (Table 5).

Table 4.

Random Effects Logistic Model Predicting Aided Ad Recall of Individual Ads.

Aided Ad Recall	$\hat{β}$	$\hat{S E} (\hat{β})$	p
Video condition	0.28**	(0.10)	.00
Male	0.08	(0.09)	.40
Age (continuous)	0.01**	(0.00)	.01
Race: White, non-Hispanic (ref.)
Race: Black, non-Hispanic	0.20	(0.15)	.17
Race: Hispanic	0.04	(0.21)	.84
Race: Other or 2+ races, non-Hispanic	0.21	(0.15)	.15
Native Internet household	0.25	(0.15)	.08
Mobile device used	−0.04	(0.14)	.81
Likely recommend Service	0.25**	(0.08)	.00
Unaided ad recall	1.56***	(0.08)	.00
Constant	−3.57***	(0.29)	.00
$\hat{ρ}$	.25	(0.02)
AUROC	.698
Wald χ²(10)	390.8

*p < .05. **p < .01. ***p < .001.

Table 5.

Random Effects Logistic Model Predicting Not Sure Response to Individual Aided Recall.

Aided Ad Recall	$\hat{β}$	$\hat{S E} (\hat{β})$	p
Video condition	−0.60**	(0.20)	.00
Male	−0.26	(0.18)	.16
Age (continuous)	0.01	(0.01)	.06
Race: White, non-Hispanic (ref.)
Race: Black, non-Hispanic	1.56***	(0.27)	.00
Race: Hispanic	1.02**	(0.38)	.01
Race: Other or 2+ races, non-Hispanic	0.83**	(0.29)	.00
Native Internet household	−0.17	(0.27)	.54
Mobile device used	0.26	(0.27)	.32
Likely recommend Service	0.04	(0.14)	.77
Unaided ad recall	0.50***	(0.13)	.00
Constant	−4.18***	(0.55)	.00
$\hat{ρ}$	.62	(0.02)
AUROC	.651
Wald χ²(10)	76.3

Note. The analysis was restricted to observations among individuals who were shown videos and did not experience technical problems.

*p < .05. **p < .01. ***p < .001.

Question 3: How Do Video Viewing Behavior and Presentation Affect the Data Quality of Follow-Up Questions?

Next, we examined response differentiation and the selection of not sure options in the follow-up ad reaction questions among respondents who were shown videos, based on demographics, whether or not respondents viewed the entire videos, and presentation order. The model was restricted to respondents who successfully viewed videos (n = 486). A random effects logistic model was estimated to predict whether respondents straightlined the follow-up questions. The dependent variable was coded as 1 for respondents who straightlined both follow-up question batteries and 0 for all others. The predictors included the presentation order of the ad, demographics, Service-specific likelihood to recommend military service, and a variable that indicated whether the respondent watched the entire ad (1 = yes, 0 = no). Mobile browser usage was excluded from the model, given that very few respondents successfully viewed videos using a mobile browser.

Respondents who watched the entire ad were significantly less likely to straightline both follow-up batteries, $\hat{β}$ = −1.40, p < .001 (Table 6). Presentation order significantly predicted straightlining, $\hat{β}$ = .17, p < .001, with follow-up batteries shown later in the survey more likely to be straightlined.

Table 6.

Random Effects Logistic Model Predicting Straightlining of Both Follow-Up Batteries Based on Presentation Order.

Straightlining	$\hat{β}$	$\hat{S E} (\hat{β})$	p
Watched full video	−1.40***	(0.23)	.00
Presentation order	0.17***	(0.03)	.00
Male	0.15	(0.33)	.64
Age (continuous)	−0.03*	(0.01)	.04
Race: White, non-Hispanic (ref.)
Race: Black, non-Hispanic	0.61	(0.54)	.26
Race: Hispanic	0.33	(0.80)	.68
Race: Other or 2+ races, non-Hispanic	0.96	(0.53)	.07
Native Internet household	−1.24	(0.56)	.03
Likely recommend Service	−0.56**	(0.22)	.01
Constant	−0.41	(1.00)	.68
$\hat{ρ}$	.74	(0.03)
AUROC	.541
Wald χ²(9)	103.0

Note. The analysis was restricted to observations among individuals who were shown videos and did not experience technical problems.

*p < .05. **p < .01. ***p < .001.

Next, a random effects model was estimated predicting the number of not sure responses in the follow-up items (Table 7). Across all respondents and all ads, the mean number of not sure responses was 2.32 (range 0–12). The predictors were the same as in the previous model. Respondents who watched the full ad provided significantly fewer not sure responses to the follow-ups, $\hat{β}$ = −.24, p < .05. Presentation order was a significant predictor, with respondents providing slightly more not sure responses to ads shown earlier in the survey.

Table 7.

Random Effects Model Predicting Number of Not Sure Responses to Ad Follow-Up Items.

Not Sure Responses	$\hat{β}$	$\hat{S E} (\hat{β})$	p
Watched full video	−0.24*	(0.11)	.02
Presentation order	−0.05***	(0.01)	.00
Male	−0.11	(0.23)	.64
Age (continuous)	−0.01	(0.01)	.20
Race: White, non-Hispanic (ref.)
Race: Black, non-Hispanic	1.02**	(0.39)	.01
Race: Hispanic	0.51	(0.58)	.38
Race: Other or 2+ races, non-Hispanic	−0.05	(0.39)	.90
Native Internet household	−0.19	(0.41)	.64
Likely recommend Service	0.11	(0.09)	.23
Constant	3.45***	(0.71)	.00
$\hat{ρ}$	.73	(0.01)
Wald χ²(9)		38.8

Note. The analysis was restricted to observations among individuals who were shown videos and did not experience technical problems.

*p < .05. **p < .01. ***p < .001.

Thus, data quality indicators for the follow-up questions varied across stimuli and across respondents as a function of presentation order and complete viewing. Ads viewed later in the survey were less likely to be viewed completely, and incomplete viewing of ads was related to straightlining and increased not sure responses.

Question 4: How Does the Order of Video Presentation Affect Whether the Videos Are Fully Watched?

We estimated a random effects model to predict whether respondents who were shown videos viewed the entire advertisements, based on demographics and presentation order (n = 486 respondents who were shown videos and did not experience technical problems). Time spent viewing the ad was coded as 1 for respondents who spent 30 seconds or more watching the ad, based on server-side timestamps, and 0 otherwise. Independent variables included the randomized presentation order of the ad, demographics, and Service-specific likelihood to recommend military service. The demographic variables were the same as in the previous set of regressions.

When ads were shown later in the survey, respondents were less likely to view those ads in their entirety, p < .001 (Table 8). This suggests that ad viewing behavior may be affected by factors such as survey fatigue.

Table 8.

Random Effects Logistic Model Predicting Video Ad Viewing Based on Presentation Order.

Watched Full Video	$\hat{β}$	$\hat{S E} (\hat{β})$	p
Presentation order	−0.29***	(0.03)	.00
Male	−0.58	(0.44)	.19
Age (continuous)	0.09***	(0.02)	.00
Race: White, non-Hispanic (ref.)
Race: Black, non-Hispanic	−1.52*	(0.76)	.05
Race: Hispanic	−0.44	(1.09)	.69
Race: Other or 2+ races, non-Hispanic	−0.99	(0.75)	.19
Native Internet household	0.85	(0.79)	.28
Likely recommend Service	−0.53*	(0.25)	.03
Constant	1.46	(1.41)	.30
$\hat{ρ}$	.89	(0.01)
AUROC	.618
Wald χ²(8)	98.4

Note. The analysis was restricted to individuals who were shown videos and did not experience technical problems.

*p < .05. **p < .01. ***p < .001.

Summary and Discussion

The key findings from this exploratory study are:

Video viewing ability was related to mobile device usage, household Internet access, and gender. Those using a mobile device were less likely to be able to view videos compared with those using computers; those with preexisting Internet access were more likely to view videos than those without; and males were more likely to view videos. These results suggest that mobile compatibility and bandwidth issues impact measurement for web surveys, particularly when rich media are involved.

Mode was associated with ad recall: those who viewed videos were more likely to indicate ad recall and were more likely to give a definite response (yes or no) than those who viewed images. These results suggest that video stimuli may be more effective than image stimuli in prompting memory.

The data quality of follow-up items varied across respondents as a function of presentation order and complete viewing. Respondents who did not view videos in their entirety were more likely to straightline the follow-up items and provide more not sure responses.

Videos that were shown later in the survey were less likely to be viewed in their entirety, suggesting video viewing behavior may be affected by survey fatigue.

Given the fidelity of video stimuli for prompting memories of television advertisements, researchers may favor a video-based question format. However, this format introduces concerns regarding response accuracy and data completeness. Our results indicate that respondents who access a survey through a mobile device may have difficulty viewing videos. Survey designers including videos need to consider whether and to what extent their sample will access the survey through a mobile device and whether the levels of missing data are acceptable. This is particularly important given rising usage of mobile devices for surveys. Although changes to the video delivery format used in our study have led to large increases in ability to view videos, differences between mobile and nonmobile respondents have persisted.¹⁰

Further, respondents may not watch the entirety of videos. Most participants in this study (85%) viewed the whole first ad they were shown, and viewing decreased with order of presentation—only 74% watched the entire eighth ad they were shown. Incomplete ad viewing was significantly associated with not sure responses to the recall question and straightlining responses to subsequent questions. Although these data may truly reflect respondents’ recall, beliefs, or genuine lack of knowledge, these behaviors may indicate satisficing.

Using data from a large field survey in which respondents likely used devices with which they were familiar, we have demonstrated potential advantages and challenges associated with using videos in web-based surveys. While the field survey allowed us to collect data naturalistically, we did not have the advantages of a controlled lab study, such as random assignment to experimental groups or within-person comparisons of video and still image conditions. As such, the ability to view ads and being in the video condition were confounded given that respondents who were unable to view videos were not supplied with a way to view videos; further, nearly all respondents completing the survey using mobile devices were unable to view videos. While we attempted to account for possible demographic confounds by including relevant control variables in the regression models, we were unable to separate mobile device usage and video viewing ability. However, this real-world condition reflects a generalizable experience and informs about what to expect from a survey that includes videos.

To ensure that all respondents could answer questions about the ads of interest, respondents who were unable to view videos were shown still images. Results suggest that video stimuli may be more effective at prompting memory. Having seen videos was generally associated with higher levels of ad recall and increased certainty (fewer participants responding not sure to the recall question) compared to having seen images. It may be that (1) the videos prompted more false memories of the ads or (2) participants who viewed the videos were more likely to have seen the ads before the study. However, the videos were likely to facilitate memory recall since they are more similar to the advertisements being evaluated than the still images in terms of sensory input mode. Remembering involves accessing information that has been encoded in memory. During recall, the brain “replays” the connections that were encoded. Because memory depends on both the nature of the memory and the information available in the retrieval environment (Eysenck & Keane, 2000), it is likely that video stimuli are more effective memory cues than still images for recall of television ads.

Researchers must account for potential incomplete or low-quality data when designing web surveys that require videos. Video length may be reduced in some contexts in an attempt to minimize incomplete viewing, as may the number of videos. Number of videos or number of follow-up questions may be reduced by employing between-person or incomplete counterbalanced designs. If number of videos cannot be reduced, survey designers might consider modifying question order, so that videos are interspersed, shown earlier in a survey, grouped separately from follow-ups, or must be viewed completely before progressing to the next survey item; though modifying the question order rather than reducing the questionnaire length might change the pattern of missing data instead of reducing the overall amount of missing data. Finally, researchers should randomly assign video order when possible since order effects are evident, and incomplete viewing is related to response behavior.

In conclusion, we have demonstrated that survey respondents who used mobile devices were less likely to be able to view videos in a web-based survey, those who could view videos were more likely to indicate recall compared to those who viewed images, and videos that were shown later in the surveys were viewed in their entirety less frequently than those shown earlier in the survey. Our results directly speak to the legitimacy of using videos in web-based surveys to gather data about attitudes.

Footnotes

Authors’ Note

The views, opinions, and findings contained in this paper are solely those of the authors and should not be construed as an official U.S. Department of Defense position, policy, or decision, unless so designated by other documentation.

When this work was conducted, all authors were employed by Fors Marsh Group; Jennifer Romano-Bergstrom is now at Facebook.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

References

The American Association for Public Opinion Research. (2011). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys (7th ed.). Chicago, IL: AAPOR.

Anderson

(2015). Technology Device Ownership: 2015. Washington, DC: Pew Research Center.

Buskirk

T. D.

Andrus

C. H.

(2014). Making mobile browser surveys smarter: Results from a randomized experiment comparing online surveys completed via computer or smartphone. Field Methods, 26, 322–342.

Calfee

J. E.

(2002). Public policy issues in direct-to-consumer advertising of prescription drugs. Journal of Public Policy & Marketing, 21, 174–193.

Callegaro

DiSogra

(2008). Computing response metrics for online panels. Public Opinion Quarterly, 72, 1008–1032.

Couper

M. P.

(2008). Designing effective web surveys. New York, NY: Cambridge University Press.

Couper

M. P.

Traugott

M. W.

Lamias

M. J.

(2001). Web survey design and administration. Public Opinion Quarterly, 65, 230–253.

Dennis

M. J.

(2010). Summary of KnowledgePanel^® Design (2010), Retrieved from http://www.knowledgenetworks.com/knpanel/docs/KnowledgePanel(R)-Design-Summary-Description.pdf

Eysenck

M. W.

Keane

M. T.

(2000). Cognitive psychology. Philadelphia, PA: Taylor & Francis.

10.

Fielding

N. G.

Lee

R. M.

Blank

(2008). The Sage handbook of online research methods. Los Angeles, CA: Sage.

11.

Gelman

(2007). Struggles with survey weighting and regression modeling. Statistical Science 22, 153–164.

12.

Groves

R. M.

Couper

M. P.

(1998). Nonresponse in household interview surveys. New York, NY: Wiley.

13.

Herzog

A. R.

Bachman

J. G.

(1981). Effects of questionnaire length on response quality. Public Opinion Quarterly, 45, 549–559.

14.

Horrigan

J. B.

Duggan

(2015). Home broadband 2015. Washington, DC: Pew Research Center.

15.

Krosnick

J. A.

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.

16.

Lemert

J. B.

Wanta

Lee

T. T.

(1999). Party identification and negative advertising in a US Senate election. Journal of Communication, 49, 123–134.

17.

Mavletova

(2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31, 725–743.

18.

Mehta

(2000). Advertising attitudes and advertising effectiveness. Journal of Advertising Research, 40, 67–71.

19.

Peytchev

(2011). Breakoff and unit nonresponse across web surveys. Journal of Official Statistics, 27, 33–47.

20.

Peytchev

Hill

C. A.

(2010). Experiments in mobile web survey design: Similarities to other modes and unique considerations. Social Science Computer Review, 28, 319–335.

21.

Shapiro-Luft

Cappella

J. N.

(2013). Video content in web surveys: Effects on selection bias and validity. Public Opinion Quarterly, 77, 936–961.

22.

Winship

Radbill

(1994). Sampling weights and regression analysis. Sociological Methods & Research, 23, 230–257.