Assessing digital advertising exposure using a virtual experimental protocol

Introduction

Anti-tobacco mass media campaigns have been successful in changing tobacco-related attitudes, intentions, and behaviors among youth and young adults. Evaluation research has consistently shown that these campaigns can mitigate the influence of pro-tobacco marketing and promotional efforts, prevent tobacco use initiation, and reduce tobacco use prevalence. ¹

The evolution of the media landscape over the past two decades has altered the delivery of many public education campaigns. An increasing proportion of campaign advertising is now delivered partially or fully online to adolescent and young adult audiences, among whom the internet and smartphone use is nearly ubiquitous.^2–4 Use of social media is increasing among all age groups but is still most prevalent among youth and young adults, especially for platforms that stream digital video.^2,5,6 Online videos currently comprise more than 82% of all consumer internet traffic, ⁷ an increasing proportion of which is coming from social video platforms that are most popular among younger users. ⁸ Among young adults, 95% use YouTube, three-quarters use Instagram and Snapchat, 55% use TikTok, ² and use is similar among teens.^9,10 Younger users are also more likely than older adults to use these sites daily and often multiple times per day.^2,6

Similar to other industries, the tobacco industry has leveraged the growth in social media and video platforms to reach and communicate with younger audiences.^11–20 The growing prevalence of exposure to and engagement with tobacco advertising online among youth and young adults^11,21–28 underscores the need for effective anti-tobacco digital campaigns. Public education campaigns at the local, state, and national levels have dramatically shifted media delivery online, including national campaigns such as those from Truth Initiative focused on e-cigarette use prevention among older teens and young adults.^29,30

Despite the promise of digital advertising, research attributing changes in targeted individual-level campaign outcomes to digital advertising exposure is sparse. A recent systematic review of mass media messaging for public health purposes found little evidence of effectiveness for campaigns utilizing digital media channels. ³¹ One factor potentially limiting the success of digital campaigns is the lack of evidence on appropriate levels of exposure for digital advertising. Sufficient exposure to digital campaign ads is critical for achieving campaign targeted outcomes. The Centers for Disease Control recommends a benchmark of 800 to 1200 gross rating points per quarter for a traditional anti-tobacco television advertising campaign. ³² This benchmark can be roughly translated to one ad exposure per week or two exposures every two weeks. These benchmarks are based on evaluation studies of media effects on behavior that either compare those exposed to the unexposed or examine a dose-response curve among varying levels of exposure. ³² Unlike television advertising, there is limited research on effective digital campaigns for tobacco prevention and cessation and no rigorous evidence-based benchmarks associated with adequate levels of digital media exposure for the purpose of ad recognition or impacts on other targeted campaign outcomes. ³³

A small number of campaign evaluation studies have been designed to isolate the effect of a digital campaign component on individual-level campaign outcomes or to assess exposure levels sufficient to impact these outcomes. ³⁴ The majority of these studies have utilized self-reported awareness measures to capture digital exposure, which raises concerns regarding endogeneity (i.e. lack of independence between the exposure and the outcome). Endogeneity can occur as a result of reverse causality (i.e. an individual who uses e-cigarettes is more likely to attend to e-cigarette ads), confounding (i.e. an unobserved variable such as an individual propensity for substance use can be associated with both selective attention to any e-cigarette advertising and a higher likelihood of using e-cigarettes),^35,36 or measurement error such as recall bias (i.e. an individual who uses e-cigarettes is more likely to recall e-cigarette advertising). ³⁵

More recent campaign evaluation studies have utilized alternative approaches to measure digital ad exposure exogenously. Exogenous exposure variables reduce concerns regarding the independence of the exposure and the outcome as they are generally measured separately from the outcome assessment and objectively such that individuals are unaware of the measurement.^35,36 Digital metrics of exogenous ad exposure aggregated by geography ³⁷ or time ³⁸ have been used in campaign evaluation studies but these measures cannot capture the effects of digital ad exposure at the individual level. Understanding the effects of exogenous individual-level digital ad exposure is especially important given that digital media is delivered via precise targeting of individuals’ sociodemographic characteristics and prior behavior rather than solely based on geographic areas, such as with television advertising.

Four studies have examined exogenous individual-level digital ad exposure and campaign outcomes. Of these, three studies utilized quasi-experimental designs and digital ad tracking technology in online panels of consumers to link exogenous individual-level ad exposure during panelists’ natural day-to-day browsing with individual-level campaign outcomes. Kim et al.^39,40 found positive associations between exposure to digital display ads for smoking cessation campaigns and related information-seeking behavior. However, these studies did not examine digital video and only captured a single instance of exposure so were not able to evaluate dose-response effects. Romberg et al's. ⁴¹ quasi-experimental study demonstrated that exposure to digital video e-cigarette prevention ads was positively associated with self-reported frequency of ad recognition in a dose-response manner but some evidence indicated selection bias in the construction of the exposed and unexposed groups. There is only one study utilizing an experimental design to assess individual-level digital campaign ad exposure and individual outcomes. In a lab-based setting, Romberg et al. ⁴² randomized participants to increasing levels of exposure to a mix of forced and skippable digital video ads for e-cigarette use prevention in a naturalistic online browsing environment. The authors found that ad recognition did not vary in a dose-response manner by randomized exposure levels when participants were able to skip ads but ad recognition was sensitive to non-randomized completed ad views. ⁴² Although Romberg utilized a digital environment that mimicked real-world online browsing, the research occurred in a lab setting and required participants to physically attend multiple lab visits over time. ⁴² This type of lab-based protocol can be costly and slow with respect to recruitment and retention efforts, is constrained to local convenience samples and may have limited generalizability to a browsing experience in a person's natural environment.

The limited research on digital anti-tobacco campaign advertising and targeted individual-level outcomes undermines campaign development and evaluation^31,34 as well as our understanding of digital advertising effects on attitude and behavior change. Further, the lack of evidence to inform benchmarks for adequate levels of exposure in digital anti-tobacco campaigns reduces campaign efficiency and effectiveness. With the rapid shift from broadcast to digital media, researchers have called for additional studies focused on developing and evaluating digital campaign advertising exposure, exposure measures, and dose-response effects in a controlled digital environment. ⁴¹ The overall goal of the current study, which is part of a larger project to examine the dose-response effects of digital media on tobacco use behavior (R01CA253013), is to develop a protocol to examine exogenous digital campaign advertising on targeted campaign outcomes in a naturalistic online browsing context among individuals in their real-world environment; and to evaluate dose-response effects of varying levels of digital exposure on outcomes. In this paper, we will describe the protocol and utilize advertising from Truth Initiative's e-cigarette use prevention campaign for older teens and young adults to examine varying levels of randomized digital advertising exposure to non-skippable ads on self-report measures of ad recognition, frequency of ad exposure and main message knowledge, and evaluate dose-response effects on these outcomes. We hypothesize the following:

Hypothesis 1: Digital exposure to non-skippable digital video ads will be positively associated with increased odds of self-reported ad recognition.

Methods

This research was conducted on a study platform designed and implemented by Ipsos, a global market research firm. The sample was drawn from the KnowledgePanel^®, a probability-based nationally representative online panel of adults in the U.S. who are recruited via address-based sampling and provided with computer/internet access if needed. The age of panel respondents is validated through a series of checks during recruitment with corroboration from multiple respondents and is updated each week for each panelist.^43,44 From this panel, we identified eligible respondents as English speakers residing in the U.S. aged 18 to 24 years. The experimental ads presented in this study were designed for an audience aged 15 to 24 years but including individuals younger than 18 years was cost-prohibitive for this study. The study was conducted from 18 February through 16 April 2021 and was approved by the George Washington University Institutional Review Board (IRB).

Randomization

Participants were randomized to one of seven groups of digital advertising exposure to the e-cigarette use prevention ads or a dummy ad (see Table 1). Across three separate viewing sessions, participants were randomized to view between 0 and 6 exposures of ads from the e-cigarette prevention campaign. Participants viewed two advertisements per session, with 0 to 2 of those ads being non-skippable digital video ads from the campaign and/or a dummy ad. The video ads played prior to the short standard YouTube videos on topics unrelated to the ads. Each viewing session was 5 min, each short video was 2.5 to 3 min (including the 15 s ad at the beginning of the video), and respondents were required to watch two videos in each session, ensuring exposure to two ads. The group that was exposed to 0 campaign ads was exposed to the dummy ad twice before each video within each session for all three viewing sessions. Groups exposed to 1 to 5 campaign ads were also exposed to the sneaker ads in some sessions depending on their group assignment. Exposures to the e-cigarette use prevention ads were randomized across the three viewing sessions within groups, with the exception of the treatment groups that received only one or only two campaign ad exposures. For these groups, we ensured that all participants received an e-cigarette use prevention ad on the third and final session to maintain consistent time to the follow-up survey with the other treatment groups, which viewed at least one e-cigarette use prevention ad in their final session.

OPEN IN VIEWER

Table 1.

Randomization of participants into a no exposure group and 6 experimental groups (N = 221).

	No exposuregroup	Experimental group 1	Experimental group 2	Experimental group 3	Experimental group 4	Experimental group 5	Experimental group 6
				Randomize order of sessions	Randomize order of sessions	Randomize order of sessions
Session 1	2 dummy ads	2 dummy ads	Doug the Pugand1 dummy ad(randomized)	Doug the Pug and1 dummy ad (randomized)	Doug the Pug AND Ditch Juul (randomized)	Doug the Pug AND Ditch Juul (randomized)	Doug the Pug AND Ditch Juul (randomized)
Session 2	2 dummy ads	2 dummy ads	2 dummy ads	Ditch Juuland1 dummy ad(randomized)	Doug the Pug and1 dummy ad(randomized)	Doug the Pug AND Ditch Juul (randomized)	Doug the Pug AND Ditch Juul (randomized)
Session 3	2 dummy ads	Doug the Pug OR Ditch Juulmust be in the third sessionand 1 dummy ad(randomized)	Ditch Juul(must be in the third session)and1 dummy ad(randomized)	Doug the Pug OR Ditch Juuland1 dummy ad(randomized)	Ditch Juuland1 dummy ad(randomized)	Doug the Pug OR Ditch Juuland1 dummy ad(randomized)	Doug the Pug AND Ditch Juul (randomized)
Total truth campaign ad exposure	0	1	2	3	4	5	6

The random assignment of eligible participants was independent of any demographic characteristics or study-related behaviors of any sample member to mitigate bias. For the three groups that included a randomized order of treatments (groups 3, 4, and 5), the order of treatment was further randomized for each member of the sample and also was independent of any demographic characteristics or behaviors of any sample members. The allocation sequence was not known to participants until they completed their participation (meaning after they received the treatments randomly assigned), and respondents were not aware of the other treatment arms included in this study.

Prior to recruitment, we conducted a power analysis by utilizing simulated datasets to fit a dose-response curve for digital ad exposure and ad recognition across seven equally-spaced exposure groups and determined that we could recover a dose-response curve with n = 20 to 25 participants per group. Our final sample was N = 221, with a range of 27 to 35 participants per group.

Results

 Table 2 shows the sociodemographics of the sample. Over three-quarters were aged 21 years or older and 65.6% were female. The large majority were white (65.2%), approximately one-quarter were married or living with a partner and over one-third lived in the South. Over three-quarters of the sample had some college education or a bachelor‘s degree or higher. Approximately 10% had used e-cigarettes in the past 30 days and 8.1% had used cigarettes in the past 30 days. We compared these demographics across randomized conditions and found no significant differences between conditions.

OPEN IN VIEWER

Table 2.

Demographic characteristics of the sample (N = 221).

Variable	N	%
Age
18	13	5.9
19	22	10.0
20	19	8.6
21	26	11.8
22	37	16.7
23	50	22.6
24	54	24.4
Education
Less than high school	11	5.0
High school	37	16.7
Some college	104	47.1
Bachelor‘s degree or higher	69	31.2
Race/ethnicity
White non-Hispanic	144	65.2
Black non-Hispanic	9	4.1
Other non-Hispanic	7	3.2
Hispanic	50	22.6
2 + races, non-Hispanic	11	5.0
Gender
Male	76	34.4
Female	145	65.6
Marital status
Married	20	9.1
Divorced/separated	2	0.9
Never married	163	73.8
Living with partner	36	16.3
Region
Northeast	42	19.0
Midwest	54	24.4
South	77	34.8
West	48	21.7
E-cigarette use (past 30 days)
No	199	90.1
Yes	22	9.9
Cigarette use (past 30 days)
No	203	91.9
Yes	18	8.1

Table 3 shows separate logistic regression models with the outcome of the self-reported ad recognition as a function of digital exposure to each separate ad and total digital exposure to either ad. Participants were randomly exposed to each individual ad from 0 to 3 times, and to either ad from 0 to 6 times. For each separate ad, the odds of self-reported recognition approximately doubled with each one-unit increase in digital exposure to the ad, with odds ratios significant at the p < .01 level. For total exposure, the odds of self-reported recognition to either ad were 1.36 times higher, 95% CI (1.16, 1.58) with each one-unit increase in digital exposure and were significant.

OPEN IN VIEWER

Figure 1.

(a) Ad-specific digital exposure and self-reported ad recognition; (b) total digital exposure and self-reported ad recognition.

OPEN IN VIEWER

Figure 2.

(a) Ad-specific digital exposure and self-reported frequency of exposure; (b) total digital exposure and self-reported frequency of exposure.

OPEN IN VIEWER

Figure 3.

(a) Ad-specific digital exposure and message knowledge; (b) total digital exposure and message knowledge.

OPEN IN VIEWER

Table 3.

Logistic regression models of self-reported ad recognition as a function of exogenous digital exposure to Ditch Juul, Doug the Pug, and total exposure to either Ditch Juul or Doug the Pug (N = 221).

Digital exposure	Odds ratio	p-value	95% CI
Ditch Juul	2.06	<0.001	1.55, 2.72
Doug the Pug	1.98	<0.001	1.50, 2.63
Total exposure to either Juul or Doug	1.36	<0.001	1.16, 1.58

Figure 1(A) shows percentages for self-reported ad recognition across digital exposure for each ad. For the Ditch Juul ad, self-reported ad recognition among respondents ranged from 28.3%, 95% CI (17.8, 41.9) for those with 0 digital exposures to 72.7% (57.8, 83.9) for those with 3 digital exposures. For Doug the Pug, recognition was 30.6% (19.3, 44.8) for 0 digital exposures to 75% (60.2, 85.6) for three digital exposures. Pairwise comparisons for each individual ad showed that ad recognition was significantly higher for two digital exposures compared with 1 or 0 digital exposure but there was no incremental benefit for 3 digital exposures compared with 2 digital exposures as well as no significant difference between 1 and 0 digital exposures.

Figure 1(B) shows percentages for ad recognition of either ad across total digital exposure conditions. Self-reported ad recognition of either ad ranged from 54.3% (37.8, 69.9) for those with 0 digital exposures to 85.2% (66.4, 94.4) for those with six digital exposures. Analyses of pairwise comparisons showed a significant improvement in ad recognition for 5 or more digital exposures compared with fewer. There was no benefit for 6 versus 5 digital exposures.

Table 4 shows separate ordinal regression models for self-reported frequency of ad exposure as a function of digital exposure to each individual ad and total digital exposure to either ad. Results demonstrate that each unit increase in digital exposure significantly increased the odds of reporting a higher frequency of exposure by 1.77 times for Ditch Juul, 1.99 times for Doug the Pug, and 1.40 times for Either ad. All odds ratios were highly significant.

OPEN IN VIEWER

Table 4.

Ordinal logistic regression models of self-reported frequency of ad exposure as a function of exogenous digital exposure to Ditch Juul, Doug the Pug, and total exposure to either Ditch Juul or Doug the Pug (N = 221).

Digital exposure	Odds ratio	p-value	95% CI
Ditch Juul	1.77	<0.001	1.38, 2.27
Doug the Pug	1.99	<0.001	1.54, 2.57
Total exposure to either Juul or Doug	1.40	<0.001	1.23, 1.58

Figure 2(A) shows the mean self-reported frequency of ad exposure across digital exposure conditions for each individual ad. For Ditch Juul, the mean self-reported frequency of exposure ranged from 0.57, 95% CI (0.30, 0.83) to 1.32 (1.02, 1.62) for those with 0 to 3 digital exposures, respectively; similarly, for Doug the Pug, mean self-reported frequency ranged from 0.43 (0.21, 0.64) to 1.34 (1.04, 1.64) for those with 0 to 3 digital exposures, respectively. Pairwise comparisons of mean self-reported frequency of ad exposure were significantly higher for the higher digital exposure conditions versus the lower conditions for each pair except for 1 versus 0 and 3 versus 2 digital exposures, which were not significantly different.

Figure 2(B) shows the mean self-reported frequency of exposure to either ad across total digital exposures. The mean self-reported frequency of exposure index ranged from 0.97 (0.61, 1.33) for 0 digital exposures to 2.59 (1.96, 3.23) for 6 digital exposures. Pairwise comparisons between each digital exposure pair showed a significantly higher self-reported frequency of exposure for 5 or 6 digital exposures compared with 3 or fewer, but the self-reported frequency of exposure was not improved with 6 digital exposures versus 4 or with 6 compared with 5.

Table 5 shows ordinal logistic regression models of main message knowledge as a function of digital exposure to each individual ad and total digital exposure to either ad. Results showed that each one-unit increase in digital exposure was associated with a non-significant increase in the odds of improved main message knowledge by 1.23 times for the Ditch Juul ad, 1.10 times for Doug the Pug, and 1.08 times for Either Ad.

OPEN IN VIEWER

Table 5.

Ordinal logistic regression models of main message knowledge as a function of exogenous digital exposure to Ditch Juul, Doug the Pug, and total exposure to either Ditch Juul or Doug the Pug (N = 221).

Digital exposure	Odds ratio	p-value	95% CI
Ditch Juul exposures	1.23	0.09	0.97, 1.58
Doug the Pug exposures	1.10	0.43	0.86, 1.41
Total exposure to either Juul or Doug	1.08	0.23	0.96, 1.21

Figure 3(A) and (B) shows main message knowledge across digital exposure levels for each individual ad and total digital exposures to either ad, respectively. The mean index response on main message knowledge for individual ads ranged from 1.23, 95% CI (1.02, 1.44) for 0 digital exposures to 1.43, 95% CI (1.23, 1.64) for 3 digital exposures for Ditch Juul, and from 1.10, 95% CI (0.93, 1.28) for 0 digital exposures to 1.25, 95% CI (1.07, 1.43) for 3 digital exposures for Doug the Pug. Tests of pairwise comparisons for each ad found no significant differences in mean main message knowledge across digital exposure levels. In Figure 3(B), the mean message index ranged from 2.37 (2.04, 2.71) for 0 digital exposures to 2.67 (2.25, 3.09) for 6 digital exposures and pairwise comparisons showed no significant differences in mean message knowledge across digital exposures.

Discussion

This study presents a virtual experimental protocol for evaluating digital advertising effects on individual-level exposure and targeted campaign outcomes using a naturalistic social media platform among individuals exposed in their own real-world environment. The protocol is valuable for researchers seeking to explore the effects of individual-level exogenous digital advertising on individual outcomes and is also useful for practitioners seeking to rigorously assess advertising content and varying dose effects prior to campaign implementation. This virtual protocol can be extended by examining more distal outcomes beyond ad recognition, such as vaping and smoking intentions, as well as mechanisms of action for ad effects. Increasing the number of digital ad exposures over an extended period of time and evaluating varying dose-response curves for different campaign outcomes would allow assessment of possible threshold and drop-off effects of exposure for targeted outcomes and could also include conditions in which ads are skippable or of different lengths. Such research can inform our understanding of the impact of varying levels of digital ad exposure on longer-term behavior change and inform benchmarks regarding optimal digital exposure levels for digital public education campaigns.

Additional benefits of the approach presented in this paper include potentially greater generalizability of study findings compared with studies examining digital advertising in lab-based or similar settings. Individuals may be less attentive to digital ads in their day-to-day environment where they may have a different cognitive focus, more potential distractions, or the ability to view other online sites while a non-skippable ad is playing. Understanding digital ad effects in the context in which most advertising is consumed is important for improved external validity. An additional benefit is the use of an online panel and a virtual experiment, which allows recruitment from a broader population and will generally be conducted more quickly than studies requiring participants to attend in-person sessions. Lastly, methods for rapid virtual experiments and remote data collection are increasingly important in the post-COVID research context. ⁴⁵

In this study, digital exposure occurred over a period of 15 days, or approximately 2 weeks. For a single ad, there was no effect of one digital exposure on ad recognition over that time period, two exposures achieved peak recognition and three (the maximum) did not provide any incremental benefit above two. For exposure to a mix of campaign video ads, which here consisted of two rotating ads for a total of six exposures, our findings suggest that approximately five digital exposures over a two-week period achieved the highest level of ad recognition. This compares to the television advertising benchmark of roughly two exposures every two weeks. ³² We did not find that increasing digital exposure was associated with increased knowledge of the ads’ main messages. Given that our analyses examined knowledge of two distinct and somewhat nuanced messages from each ad, we also conducted post-hoc analyses to evaluate the comprehension of a simpler message in each ad of “do not vape.” Results were similar in finding no relationship between exposure and message knowledge. It is important to note that the ads in this study were only 15 s each so total exposure time for those exposed ranged from a minimum of 15 s in the lowest exposure level to a maximum of 90 s at the highest level, which occurred over several browsing sessions covering approximately two weeks. Our findings suggest that this level of exposure may be acceptable for basic ad recognition but are not sufficient for improving message knowledge.

Supplemental Material

Supplemental Material