Using Online Message Testing to Evaluate TV Ads,Select Effective Messaging,and Improve Public Health Campaigns

Abstract

Background:

This article describes two examples of online ad testing used to assess effectiveness of tobacco control ads in Minnesota (United States). The U.S. Centers for Disease Control and Prevention (CDC) recommends mass-media campaigns to change tobacco-related attitudes and encourage quitting. ClearWay Minnesota^SM uses online surveys to test ads with target audiences, select the most effective ads, and complement traditional evaluations of its campaigns. Two examples in this article demonstrate the method and how results have been used.

Methods:

Example 1 used an online survey to test two ad campaigns with 600 tobacco users. Participants were randomly assigned to evaluate two CDC Tips From Former Smokers ads and two ClearWay Minnesota QUITPLAN^® Services ads for motivational effectiveness. In Example 2, 805 adult Minnesotans evaluated three of ClearWay Minnesota’s Still a Problem educational ads to determine whether the campaign was having the intended effect on its target audience.

Results:

Results for Example 1 revealed CDC’s ads were more effective than QUITPLAN Services ads in motivating tobacco users to seek help. Results for Example 2 demonstrated that the Still a Problem campaign was having the intended effect on the target audience.

Discussion:

Example 1 results compelled ClearWay Minnesota to tag CDC’s ads with the QUITPLAN Services brand and buy more advertising time. Observed service volumes increased. Example 2 results suggested no need for adjustments to change the Still a Problem campaign because it was working as intended. Traditional evaluation requires longer time frames than online testing. Online message testing can inform social marketing and educational campaign development in real time, typically for less cost than traditional methods.

Conclusions:

Online message testing is an efficient way to test ads, improve program effectiveness, and protect investments in public health marketing campaigns.

Keywords

monitoring and evaluation best practices Internet branding tobacco cessation health research

Introduction

Tobacco use remains the leading cause of preventable death worldwide, causing nearly 5 million deaths annually (World Health Organization, 2012). Every year in the United States, cigarettes kill more than 440,000 people (Koh & Sebelius, 2012; U.S. Department of Health and Human Services, 2010) and cost US$96 billion in direct medical costs and US$97 billion in lost productivity (Centers for Disease Control and Prevention [CDC], 2008). In Minnesota, each year more than 5,000 people die from smoking-related diseases, and US$2.87 billion is spent in tobacco-related health-care costs (Blue Cross and Blue Shield of Minnesota, 2010). The U.S. CDC recommends using mass-media campaigns as part of a comprehensive tobacco control program to change attitudes about tobacco and encourage people to quit (CDC, 2014). These recommendations are based on substantial evidence demonstrating that mass-media campaigns are effective in motivating smokers to quit (Davis, Farrelly, Duke, Kelly, & Willett, 2012; Durkin, Brennan, & Wakefield, 2012; National Cancer Institute, 2008; Wakefield, Loken, & Hornik, 2010; Wilson, Avila Tang, Chander, & al., 2012).

For more than a decade, ClearWay Minnesota^SM has created mass-media campaigns to educate adult Minnesotans about the dangers of tobacco use and secondhand smoke, and to motivate adult tobacco users to quit and to use ClearWay Minnesota’s cessation programs, called QUITPLAN^® Services. Mass media, in the form of television (TV) ads, is one part of a larger social marketing model that ClearWay Minnesota uses to motivate people to change their behavior and quit tobacco use.

While public health educators have traditionally provided information to the general public in hopes that the intended audience will recognize themselves as being at risk and change their behavior as a result, social marketing “targets a particular segment of the population, using persuasive messages and audience research and participation” (ORC Macro/APCO, 2004). In other words, social marketing selects specific audiences to hear a particular message delivered in persuasive and compelling ways in order to change behavior. Marketing, and particularly social marketing, is based on theories of consumer behavior, which in turn stem from the social and behavioral sciences (Novelli, 1990). However, many social marketers do not report on their work in professional journals or at conferences, and those who do rarely focus on the theoretical models that impacted their decisions on target audience selection, formative research strategies, program element development and selection, and outcome measurement (Lefebvre, 2001). This article focuses on the online testing method used to determine whether ads in two examples were effective in changing attitudes and social norms, thereby predicting both intentional and actual behavior change, consistent with the theory of reasoned action.

Online message testing has been one of several important tools for pretesting and improving ClearWay Minnesota’s mass-media campaigns in order to effectively reach and promote behavior change with our intended target audience. This article highlights two examples of how online surveys were used to assess and select ads that most effectively influenced target audiences’ quitting behaviors and attitudes about tobacco use. Example 1 compared two very different ad campaigns to test which would be more effective at getting Minnesota adult tobacco users to contact QUITPLAN Services for help with quitting. Example 2 examined whether a new campaign, called Still a Problem, was achieving its intended educational and attitudinal impact among adult Minnesotans.

Background

There is substantial evidence demonstrating that mass-media campaigns are effective in motivating smokers to quit (Davis et al., 2012; Durkin et al., 2012; National Cancer Institute, 2008; Wakefield et al., 2010; Wilson et al., 2012). In addition to motivating behavior change, mass-media campaigns have been effective in shifting social norms regarding smoking (Abroms & Maibach, 2008; Noar, 2006; Siegel, 1998), which may contribute to increased quit attempts and successful quitting.

Although media campaigns can substantially contribute to declines in tobacco use (Levy, Boyle, & Abrams, 2012), they are expensive, and public health dollars for them are limited. Research to determine potential effectiveness of ads before dollars are invested to produce, air, or continue them is important for the public health community. Marketing research offers tools to do this and has developed into its own industry. An entire peer-reviewed publication, The Journal of Advertising Research, is dedicated to publishing case studies and methodological analyses on this topic. Its parent, the Advertising Research Foundation, represents both practitioners and academics (The Advertising Research Foundation, 2014). Many marketing research firms engage in advertising research and some, such as Ameritest, specialize in it. These resources can be used by the public health community to produce more successful campaigns (Noar, 2006; Rogers, 1996).

The theory of reasoned action (Ajzen & Fishbein, 1980), a classic behavioral model, states that people change their behavior based on their intentions and the outcomes they expect to achieve as a result. Intention, according to the model, is predicted by both attitude (a person’s positive or negative feelings toward the behavior) and subjective norm (a person’s perceived social pressures to think or behave in a certain way; ORC Macro/APCO, 2004). A social marketing campaign that changes either attitudes or subjective norms toward a behavior will, according to the theory of reasoned action, be more effective at changing behavior than campaigns that do not change attitudes or subjective norms.

Specifically, past research has demonstrated that a target audience’s ratings of ad quality or perceived effectiveness (e.g., “attention grabbing,” “credible,” and “believable”) can predict actual effectiveness of ads focusing on behaviors and attitudes related to drug use, seat belt use, and drunk driving (Dillard, Shen, & Vail, 2007; Dillard, Weber, & Vail, 2007). This is true for smoking cessation campaigns as well (Davis, Nonnemaker, Duke, & Farrelly, 2013; Davis, Nonnemaker, Farrelly, & Niederdeppe, 2011; Niederdeppe, Farrelly, Nonnemaker, Davis, & Wagner, 2011). The research supports the theory of reasoned action, since ads perceived as being believable or credible might influence both a person’s attitudes about a behavior and increase perceived social pressure to change behavior. A study by Davis et al. (2013) found that baseline perceived effectiveness for cessation TV ads was associated with increased negative feelings about smoking, increased outcome expectations about the benefits of quitting, and an increase in desire and intention to quit at follow-up. The authors recommended pretesting for perceived effectiveness as part of formative research for cessation TV campaigns.

Formative research may be especially important because antitobacco campaigns have a wide variety of themes, and selecting the most effective ones can ensure good use of scarce public health resources. The most commonly used, researched, and supported themes are cessation-focused messages using graphic images or emotional testimonials about the consequences of smoking, primarily designed to change attitudes and behavioral intentions (Beaudoin, 2002; Davis et al., 2013; Farrelly et al., 2012; Nonnemaker, 2013). These are often referred to as “why-to-quit” messages. Another type of messaging offers quitting resources and encouragement designed to influence social norms based on research suggesting self-efficacy can encourage behavior change (Stephenson & Witte, 1998; Witte, 1992). These messages are referred to as “how-to-quit.” ClearWay Minnesota has used both types of messages to motivate people to quit and to promote QUITPLAN Services.

In addition to formative research, practitioners often use media evaluations to help determine the efficacy of campaigns. This approach is valuable because it can potentially identify and quantify a campaign’s ability to change attitudes and behaviors among target audiences and establish a causal relationship between campaign exposure and attitudinal or behavior changes. However, comprehensive evaluations can be limiting. Based on ClearWay Minnesota’s experience, a rigorous, longitudinal media evaluation with four data collection points in time may cost approximately US$500,000 or more. Additionally, the data collection and analysis can take many months, which may limit real-time application of findings to the campaign it was intended to evaluate.

Media evaluations remain important, and their findings can and should be applied to future campaigns; however, online message surveys are designed to help predict the potential effectiveness of campaign messages by collecting the target audience’s reactions and perceptions of the ads, specifically in our examples, around potential changes in attitudes or social norms. Data can be collected and analyzed in a matter of weeks and can cost approximately US$25,000 per survey. Given the demonstrated link between perceived and actual effectiveness, and the predictive nature of changing attitudes and social norms on behavior change, these surveys can be a time- and cost-effective tool in shaping campaigns and predicting their effectiveness. Their timely nature allows practitioners to make strategic decisions before ads air and to make changes during the course of campaigns: increasing airtime for a particular ad, removing a spot from rotation, selecting the most effective messages for an upcoming broadcast cycle, and so on.

Several methods have been used for acquiring participants for such surveys, including established Internet panels (cf. Pearson, Richardson, Niaura, Vallone, & Abrams, 2012). Internet panels are created by companies as existing groups of people who can be invited to participate in research based on predetermined selection criteria, such as demographics, behavioral characteristics, or areas of expertise or interest. Established panel methods can be both quick and inexpensive and can provide a consistent cross section of a particular population, although there can be great variability with respect to the types of demographic data available. During the pre-Internet era, message and copy testing were often done via intercept studies or central location tests. These approaches did not and could not incorporate random sampling. In this respect, online panels are a huge improvement for the practice.

Literature on online message testing is limited, in part because most such research is proprietary, since commercial firms and advertising agencies rely on it to gain competitive advantages. Campbell, Parent, and Plangger analyzed more than 100 refereed articles that appeared in the Journal of Advertising Research as of 2010 and treated various aspects of online research as applied to advertising. They highlighted the recent explosion of advertising research using online methods and noted both benefits and challenges associated with conducting online research (Campbell, Parent, & Plangger, 2011). One long-running survey of corporate research users found that by 2011, half of all studies used an online research mode (Macer & Wilson, 2014).

For the purpose of assessing TV ads, ClearWay Minnesota used online message surveys to select messages and images that effectively changed attitudes and beliefs as well as motivated our audience to take action and to conduct an assessment of how well a new educational media campaign influenced perceived social norms. Two examples are presented here to illustrate ways ClearWay Minnesota uses online message testing to make strategic decisions about the direction of two different types of campaigns. This article contributes to the literature by describing a practitioner’s use of online message testing to improve practice.

Example 1: Using Online Testing to Assess Ad Effectiveness Among Tobacco-Using Adults

The purpose of Example 1 was to test potential effectiveness of two different TV campaigns designed to change attitudes and beliefs and motivate tobacco users to use cessation services. Based on results, one campaign would be selected for airing. The QUITPLAN Services brand, which directs Minnesota tobacco users to an evidence-based cessation treatment resource, was created in 2003 after extensive formative research and testing with tobacco users to identify key attributes that would shape the brand. Since then, QUITPLAN Services has served more than 130,000 tobacco users. In 2011, almost 80% of Minnesota adult smokers were aware of QUITPLAN Services, and more than 64% of Minnesota adult smokers rated QUITPLAN Services favorably (ClearWay Minnesota, 2011). Despite this, enrollment in services declined dramatically in 2010 and continued to decline in 2011. ClearWay Minnesota staff explored new methods and messages to recruit people to QUITPLAN Services, including the creation of new how-to-quit ads that were lighter in tone and tested well in formative research. These ads were “Inner Conflict” and “Everyone’s Monkey is Different.” (ClearWay Minnesota’s ads can be viewed at http://www.youtube.com/user/ClearWayMinnesota/videos) Inner Conflict (Figure 1) featured a humorous approach, acknowledging smokers likely feel conflicted about quitting and illustrating that conflict. Everyone’s Monkey is Different (Figure 2) depicted different primates clinging to smokers’ backs (illustrating the addiction metaphor of “getting the monkey off your back”) and encouraging them to keep smoking. The campaign was created by Minneapolis ad agency Clarity Coverdale Fury in consultation with ClearWay Minnesota staff.

Figure 1.

“Inner Conflict” (QUITPLAN^® Services) still image.

Figure 2.

“Everyone’s Monkey is Different” (QUITPLAN^® Services) still image.

At the same time, CDC announced that it would launch a hard-hitting, emotional TV advertising campaign in the spring of 2012 called Tips From Former Smokers. (All Tips ads can be viewed at http://www.cdc.gov/tobacco/campaign/tips/resources/videos/) CDC ads tested for this survey were “Buerger’s” (Figure 3) and “Jessica’s Asthma” (Figure 4). Both ads featured real people talking about the impact of smoking on their lives (including Buerger’s disease, amputation, and asthma), and both ads included a call to action to quit smoking. The QUITPLAN and CDC ads included final screen shots listing contact information for cessation services available to tobacco users (Figures 5 and 6). Although CDC already planned to air the ads in Minnesota, ClearWay Minnesota had the option to purchase additional media time to broaden the campaign’s impact. An online survey was developed to determine which approach would be more effective in motivating Minnesota adult tobacco users to use QUITPLAN Services.

Figure 3.

“Buerger’s” (tips campaign) still image.

Figure 4.

“Jessica’s Asthma” (tips campaign) still image.

Figure 5.

Final screen for tips campaign ads “Buerger’s” and “Jessica’s Asthma” with national quitline portal toll-free number.

Figure 6.

QUITPLAN^® Services phone number final image for “Inner Conflict” and “Everyone’s Monkey.”

Example 2: Using Online Testing to Assess an Educational Campaign’s Impact on Its Target Audience

The purpose of Example 2 was to assess whether a new educational advertising campaign was having the intended impact of increasing concern about tobacco use among adult Minnesotans aged 18–54 who are socially active and connected. Results were used to determine whether to make changes to the newly launched campaign. ClearWay Minnesota launched a new TV campaign in 2012 to counter apathy around the issue of tobacco use. Unpublished qualitative research conducted earlier in 2012 showed adult Minnesotans were becoming less concerned about the problem of tobacco—believing it had essentially “been taken care of.” This attitude appeared to result from progress made in reducing smoking in Minnesota and passing an indoor smoking ban. The same research identified strategies to counter this complacency among adults, including leveraging concerns about smoking among youth, and the economic costs that all Minnesotans pay. It was believed that pointing out negative consequences would create an environment where the public supported doing more to reduce tobacco use. The campaign’s key message and tagline was “Tobacco is still a problem in Minnesota.” The campaign was created by Minneapolis agency Clarity Coverdale Fury in consultation with ClearWay Minnesota staff.

Three ads were created for the campaign and tested using online message surveys. “Roller Coaster” (Figure 7) and “School Bus” (Figure 8) focused on teen use of tobacco and were intentionally dark in tone. Both contained disturbing images of teens using or being impacted by tobacco. Roller Coaster depicted youth being taken on the “ride of their lives” on a roller coaster representing tobacco addiction. School Bus showed startling footage of a school bus full of junior high school kids smoking, illustrating that 77,000 Minnesota kids are current tobacco users. The third ad, “Your Share” (Figure 9), took a lighter tone, using absurdity to make its point that we all pay the economic cost for tobacco use. In the spot, an official showed up at a home to collect US$554 from each family member to pay for the nearly US$3 billion in tobacco-related health-care costs in Minnesota. All three ads contained a final screen shot showing the web address for the campaign where viewers could get additional information (www.stillaproblem.org; Figure 10). (These ads can be viewed at http://www.youtube.com/user/ClearWayMinnesota/videos.)

Figure 7.

“Roller Coaster” (still a problem) still image.

Figure 8.

“School Bus” (still a problem) still image.

Figure 9.

“Your Share” (still a problem) still image.

Figure 10.

Still a problem final image for “Roller Coaster,” “School Bus,” and “Your Share.”

Method

Study Design

ClearWay Minnesota’s online message testing used established online panels to obtain participants. Panels vary in their methods, but the firms used for the ClearWay Minnesota studies seek to maintain appropriate demographic balances among their members. For each survey, panel members who meet selection criteria are invited to participate. Participant identities were known only to the panel company, and the surveys’ sponsors were not revealed to participants. Participants accumulated reward “points” for their involvement but were not paid directly. Participants could redeem their points with participating companies such as electronic retailers, hotel chains, and airlines. Response rates average between 5% and 10%. The panels used for both studies adhere to the guidelines set by the Council of Survey Research Organizations.

All surveys were hosted by Action Marketing Research on its own domain using Sawtooth Software’s Sawtooth Software Inc. (SSI)-Web survey platform. Action Marketing Research developed the survey questionnaires in collaboration with ClearWay Minnesota staff. The time frame to design, field, analyze, and use the results was approximately 6 weeks.

The technology used for showing the ads to participants changed over time. In past surveys, some participants experienced technological difficulties viewing the ads. As a result, ClearWay Minnesota began using unlisted links on YouTube. The YouTube player functions on nearly all mobile devices and computers, and the number of technical issues was dramatically reduced once this method was adopted.

Example 1 Details

Sampling

The survey for Example 1 was fielded in February and March 2012. Six hundred tobacco users aged 18–54 completed the survey used for this analysis. A single research aggregator (ReRez) was the point of contact for the study to coordinate logistics and ensure consistency across panels. To obtain a large enough sample of tobacco users, ReRez divided the sample among these other panel organizations: Toluna, MyPoints, Market Cube, and Peanut Labs. Assignment to the spots seen was randomized across panels to avoid any potential bias due to the differences in population makeup for each panel. ReRez sent invitations to panel members (exact numbers are unavailable due to the panels’ proprietary nature). Of those invited, 3,634 started the survey. Screening questions were included within the survey. Based on responses provided, 25% of those who started the survey qualified as being tobacco users of the right age and state(s) of residence. Nine hundred participants completed the survey. Three hundred respondents viewed ads not used for this analysis, leaving 600 tobacco users, aged 18–54, for this example. As the population of tobacco users in the panels was relatively small, it was necessary to include consumers in states adjoining Minnesota. Although demographic differences across the groups selected to view the CDC and ClearWay Minnesota ads were small, the samples were further weighted to assure that the weighted proportions of participants viewing each ad grouping were 60% female, 40% male, and 39% Minnesota residents. Demographics are provided in Table 1.

Table 1.

Demographics and Tobacco Use Characteristics for Example 1 Respondents.

Participant Characteristics	N = 912 (Unweighted)	% (Unweighted)	Weighted Proportion of the Sample (%)
State of residence
Minnesota	346	39	39
Wisconsin	340	38	37
Iowa	171	19	18
North Dakota	28	3	3
South Dakota	27	3	3
Age
18–24	88	10	10
25–34	233	26	25
35–44	241	26	27
45–54	350	38	38
Education
High school or less	259	28	28
Trade/vocational	119	13	13
Some college	191	32	32
College graduate	243	26	26
Income
Less than US$25K	207	23	23
US$25–US$34K	145	16	16
US$35–US$49K	169	19	18
US$50–US$74K	193	21	21
More than US$75K	134	15	15
Refused	64	7	7
Cigarette use
Every day	725	79	79
Some days	150	16	16
Not at all	37	4	4
Other tobacco use
Every day	109	12	12
Some days	222	24	25
Not at all	581	64	63
Quit for 24 hr or more in the past 12 months
Yes	378	41	41
No	504	55	55
Don’t recall	30	3	3
Intention to quit
Next 30 days	108	12	12
Next 6 months	251	28	28
Sometime in the future	279	31	30
No plans	196	21	22
Currently trying	78	9	9

Respondents were randomly assigned to view a 60-s video comprising two 30-s ads from either the CDC’s Tips From Former Smokers campaign or ClearWay Minnesota’s QUITPLAN Services campaign (i.e., 300 per cell). The average survey was 10 min long and consisted of 24 questions. All respondents answered the same questions.

Measures

After viewing the videos, respondents were asked questions about how they perceived the ads and what actions they might take after viewing them.

Attribute measures

Respondents were asked questions focusing on specific ad attributes. Respondents were asked on a 5-point scale whether they strongly disagree (1) to strongly agree (5) with statements such as “This ad grabbed my attention,” “This ad gave me good reasons to quit tobacco,” “This ad was more memorable than other ads,” “This ad was difficult to watch,” “This ad made me think more about quitting smoking,” and so on. Next, respondents were asked whether certain words describe(s) not at all (1) to describe(s) very well (5) ad characteristics including depressing, helpful, encouraging, annoying, believable, motivating, informative, frightening, and emotional. Several of these survey items correspond to perceived effectiveness measures as identified by Dillard, Weber, and Vail (2007), including “This ad grabbed my attention” (compelling), “This ad gave me good reasons to quit tobacco” (persuasive), “this ad was believable” (plausible), and “this ad was informative” (logical). Others measured negative attributes that ClearWay Minnesota wanted to avoid or that are not associated with perceived effectiveness, including “depressing” and “annoying.” A third group were measures of ad attributes that could be indicators of perceived effectiveness or ineffectiveness, depending on interpretation by the individual respondent, including “frightening,” “emotional,” or “difficult to watch.” The proportion of respondents endorsing the highest rating for each type of question (strongly agree, describes very well, etc.) are reported in Table 2. Respondents were also asked whether they would “talk to [their] friends and family about these ads.”

Table 2.

Proportion of Respondents Strongly Endorsing Ad Attributes and Evaluative Statements.

Ad Descriptors	Proportion of Respondents Viewing the CDC Ads Reporting Highest Level of Agreement (Strongly Agree or Describes Very Well) With Descriptive Attribute or Statement (%)	Proportion of Respondents Viewing the QUITPLAN^® Services Ads Reporting Highest Level of Agreement (Strongly Agree or Describes Very Well) With Descriptive Attribute or Statement (%)	Significance of Difference of Proportions by Two-Tailed z-Test
Perceived effectiveness attributes
Powerful	44	16	p < .0001
Helpful	24	14	p < .001
Believable	35	19	p < .0001
Motivating	29	19	p < .002
Informative	36	18	p < .0001
These ads grabbed my attention	34	23	p < .001
These ads gave me good reasons to quit smoking or using tobacco	35	10	p < .0001
These ads gave me a new perspective	25	8	p < .0001
These ads were more memorable than other ads	27	22	p < .08
I would talk to my friends and family about these ads	21	13	p < .005
These ads made me think more about quitting smoking or using tobacco	31	17	p < .0001
Negative attributes
Depressing	28	3	p < .0001
Annoying	6	4	NS
Neutral attributes
I liked these ads	48	70	p < .0001
These ads were difficult to watch	20	3	p < .0001
Frightening	39	2	p < .0001
Emotional	37	11	p < .0001

Note. NS = not significant; CDC = Centers for Disease Control and Prevention.

Intended action measures

Measures of actual effectiveness are restricted here to postmessage attitude and behavioral intention. Respondents were asked how motivated (not at all [1] to very [5]) they were to take action by quitting, reducing smoking, or using the services advertised after seeing these ads.

Service volume measures

Service volumes consist of call volumes, which are the number of tobacco users who call the QUITPLAN Helpline, and web volumes, which are the number of people who register for quitplan.com. Calls and web registrants are tracked weekly by ClearWay Minnesota for QUITPLAN Services.

Analysis

Differences between means on numeric scales were evaluated with two-tailed t-tests. Differences in the distribution of categorical ratings were evaluated with χ² tests. Differences in pairs of proportions were evaluated with z-tests.

Example 2 Details

Sampling

The survey for Example 2 was fielded in January 2013. Eight hundred five adults aged 18–54 completed the survey. All participants belonged to a single large panel (ResearchNow). ResearchNow sent invitation e-mails to panel members, of whom 1,228 began the survey, and 805 (66%) met the screening criteria for being 18–54 years old, socially connected, or having children under 18 in their household and completed the survey. The social connection criteria were selected to help ensure the survey sample included ClearWay Minnesota’s target audience for its Still a Problem campaign: adult Minnesotans who are socially active and connected. (The more socially connected and influential adults see and talk about a campaign, the more credible and widespread its messaging can become.) Parents were oversampled due to the campaign’s inclusion of youth smoking as an issue that should concern adult Minnesotans with children. To be considered “socially connected,” potential participants had to agree with two of the following three statements: (1) “I enjoy talking to people about current events or issues,” (2) “People I am closest to ask for my opinion and follow my advice,” or (3) “I regularly watch the TV news.” Seventy percent of survey participants met the socially connected criteria, 55% had children under 18 in the home, and many participants met both criteria. Demographics and tobacco use characteristics of the sample are provided in Table 3. Respondents were randomly assigned to view either School Bus or Roller Coaster, and all participants viewed Your Share. The average survey length was 10 min long and consisted of 26 questions. All respondents received the same questions.

Table 3.

Participant Characteristics for Example 2 Respondents.

Participant Characteristics	N = 805	Proportion of the Sample (%)
Gender
Men	298	37
Women	507	63
Residence
Seven-county metro area	510	63
Outside of metro area	295	37
Age
25–34	189	23
35–44	228	28
45–54	388	48
Tobacco use
Use any	97	12
Do not use	708	88

Measures

Cognitive responses

Survey respondents were asked diagnostic questions after viewing each ad, such as “How clear was the idea that tobacco use is still a problem in Minnesota?” Answer options were very clear, somewhat clear, not very clear, and not clear at all. They also were asked what they thought each ad was trying to say (e.g., “We need to do more to keep kids from smoking,” “Even nonsmokers are paying the price for smoking,” etc.). Response options were selected to allow ClearWay Minnesota to confirm the intended messages for each ad were clear to viewers. The same list was shown following each ad viewing in random order. Participants could select as many messages as they wanted from the list.

Emotions

Respondents were asked to select words or phrases they felt described each ad based on the emotional responses elicited; for example, “Which of the following word or phrases describes the ad that you saw?” Emotional responses included insulting, frightening, and inspiring.

Perceived effectiveness

Several of the descriptive words and phrases included measures related to perceived effectiveness including “truthful,” “exaggerated” and “believable,” “personally relevant,” “attention getting,” and “memorable.”

Attitudes

Respondents were asked a series of attitude questions about tobacco issues before and after viewing two ads. Respondents viewed one of the two ads focused on the issue of youth smoking and the Your Share ad (the order of ads was rotated). Measures of actual effectiveness are restricted here to postmessage attitude. The questions included whether the participants strongly agreed, agreed, disagreed, or strongly disagreed with the following statements (corresponding to the goals of the campaign):

The tobacco industry is responsible for smokers’ addiction,

Reducing the number of people who smoke should be a priority for Minnesotans,

Too many Minnesota kids use tobacco, and

The harms of tobacco increase health-care costs for everyone.

This campaign was designed to counter the complacency of adults toward the problem of tobacco use in Minnesota. All three ads were assessed for message clarity, perceived effectiveness, and potential to further shift attitudes about the problem of tobacco use, with the hope that adult Minnesotans would more strongly support additional steps to decrease tobacco use. If the testing found that a particular ad was performing exceptionally well or poorly, campaign managers would use this information to adjust airtime or find or create a more effective ad for the campaign.

Analysis

Differences between means on numeric scales were evaluated using two-tailed t-tests. Differences in the distribution of categorical ratings were evaluated using χ² tests. Differences in pairs of proportions were evaluated with z-tests.

Results

Example 1

Ad attributes and perceived effectiveness

While more respondents viewing the CDC ads reported that depressing, frightening, and emotional described them “very well” than respondents viewing the QUITPLAN Services ads, they also perceived the CDC ads to be more effective at potentially changing behavior, with higher proportions reporting the CDC ads to be “powerful,” “helpful,” believable, “motivating,” and “informative” than the QUITPLAN Services ads. More viewers of CDC ads than of QUITPLAN Services ads reported strong agreement that the ads grabbed their attention, gave them good reasons to quit smoking or using tobacco, made them think more about quitting, and gave them a new perspective. At the same time, more viewers of QUITPLAN Services ads reported that they liked the ads than viewers of CDC ads. Also, more viewers of the CDC ads found them “hard to watch” (see Table 2).

Intended action

The CDC ads were more effective than the QUITPLAN Services ads in motivating action. Thirty-two percent of respondents indicated they were very motivated to quit or reduce their smoking after viewing the CDC ads versus 15% of those who viewed the QUITPLAN Services ads. Twenty-three percent of respondents said they were very motivated to use the advertised services after seeing the CDC ads versus 14% who saw the QUITPLAN Services ads. Twenty-one percent of respondents viewing the CDC ads strongly agree that they would “talk to [their] friends and family about these ads” versus 13% of respondents viewing the QUITPLAN Services ads.

Service volumes

Because the goal of this campaign was to motivate people to use QUITPLAN Services, as a result of this testing QUITPLAN Services campaign managers decided that the CDC ads should get more airtime and could be tagged with the QUITPLAN Services brand to drive consumers to these services with little risk of damage to the brand. During the first 3 weeks of the CDC ads running with the QUITPLAN Services tag, there was approximately a doubling of calls to the helpline and of web registrants over the same period the previous year.

Example 2

Cognitive responses

More than 6 in 10 respondents (64% for School Bus, 71% for Your Share, and 78% for Roller Coaster) reported that the ads they viewed made it very clear that tobacco use is still a problem in Minnesota (see Table 4). The responses to the questions about what the ads were trying to say confirmed that the ads clearly conveyed their intended messages. More than 8 in 10 viewers of School Bus (80%) and Roller Coaster (86%) reported that the ads were trying to say that we need to do more to keep kids from smoking, while about half (52% and 43%, respectively) believed that the ads were saying that keeping people from smoking is an ongoing challenge. Viewers of Your Share believed the ad was trying to say that smoking isn’t just a problem for smokers but for everyone (74%) and that even nonsmokers are paying the price for smoking (81%).

Table 4.

Proportion of Respondents Agreeing to Attitude Statements Before and After Viewing Ads.

Attitude Statements	Level of Agreement	Pretest Before Seeing Any Spots, n (%)	Posttest After Seeing Two Spots, n (%)	Significance of Difference of Proportions by Two-Tailed z-Test
The tobacco industry is responsible for smokers’ addiction	Agree OR strongly agree	427 (53%)	459 (57%)	NS
The tobacco industry is responsible for smokers’ addiction	Strongly agree	121 (15%)	113 (14%)	NS
Reducing the number of people who smoke should be a priority for Minnesotans	Agree OR strongly agree	612 (76%)	676 (84%)	p < .01
	Strongly agree	201 (25%)	250 (31%)	p < .01
Too many Minnesota kids use tobacco	Agree OR strongly agree	725 (90%)	757 (94%)	NS
Too many Minnesota kids use tobacco	Strongly agree	242 (30%)	290 (36%)	p < .01
The harms of tobacco increase health-care costs for everyone	Agree OR strongly agree	757 (94%)	773 (96%)	NS
	Strongly agree	411 (51%)	419 (52%)	NS

Note. NS = not significant.

Emotions

About a third of respondents reported feeling anger at the tobacco companies after viewing each of the ads (28% Your Share, 32% School Bus, and 34% Roller Coaster). Two in 10 reported feeling anger at smokers after watching the ads focused on teen smoking (20% Roller Coaster and 21% School Bus). Nearly half (49%) of those viewing Your Share reported feeling anger at smokers while watching the ad, but very few (5%) reported feeling anger at the ad sponsor. And while between one third and one half of participants felt discomfort with the images shown in both Roller Coaster (47%) and School Bus (33%), many fewer reported discomfort with the ad messages (14% and 15%, respectively).

Perceived effectiveness

Between 7 and 9 in 10 (70% School Bus, 79% Roller Coaster, and 87% Your Share) thought the ads were personally relevant, and 9 in 10 (87% School Bus, 87% Roller Coaster, and 91% Your Share) thought the ideas conveyed in the ads were believable. About 8 in 10 (77% School Bus, 80% Roller Coaster, and 77% Your Share) thought these three ads would be much or somewhat more memorable than other ads on TV.

Attitudes

While there was some movement in attitudes, especially for those endorsing “strongly agree” for the statements about reducing tobacco use being a priority (25–31%, p < .01), and about too many Minnesota kids smoking (30–36%, p < .01), the general level of endorsement of all four statements at pretest was very high to start with, leaving little room for movement (see Table 4).

Discussion

Example 1

In response to declining service volumes, ClearWay Minnesota used online message testing in an attempt to identify the most compelling ads to drive tobacco users to QUITPLAN Services and increase the number of tobacco users served. Findings indicated that the CDC ads would likely be more effective at changing behavior than the QUITPLAN Services ads. Respondents who viewed the CDC ads were more likely to be motivated to take action—to quit or reduce their smoking and to contact services—than those viewing the QUITPLAN Services ads. Additionally, the CDC ads scored higher on attribute measures (more memorable and gave me good reasons to quit) and measures of perceived effectiveness (believable, helpful, motivating, informative, and frightening). While more people reported liking the QUITPLAN Services ads, this was not a measure of perceived effectiveness and may not be associated with viewers taking action. Similarly, more people reported the CDC ads were hard to watch, but such an ad may be more effective at spurring action, not less, especially if it conveys hope and helpfulness.

Based on these findings, ClearWay Minnesota campaign managers made the decision to air CDC Tips ads. The only change was the final screen shot which was changed from the Tips tagline and national portal number (Figure 5) to the QUITPLAN Services toll-free number and logo (Figure 6). During the first 3 weeks of CDC ads airing with the QUITPLAN Services tag screen, volumes to QUITPLAN Services approximately doubled from what they had been during those 3 weeks the prior year. Although we cannot be certain that the CDC ads were the cause of this increase in service volumes because other promotions were running at the same time, it is reasonable to conclude that they contributed to this dramatic increase. These results appear consistent with past studies demonstrating the effectiveness of “why-to-quit” ads (Farrelly et al., 2012) and with CDC’s evaluation of the Tips campaign. Results of the Tips evaluation found the campaign increased population-level quit attempts by 12%, resulting in an estimated additional 1.64 million smokers making quit attempts as a result (McAfee, Davis, Alexander, Pechacek, & Bunnell, 2013). Of these, an estimated 220,000 smokers remained quit at follow-up. As a result of increases in quit attempts and successful quits, between 333,000 and 500,000 quality-adjusted life years may have been added to the U.S. population (McAfee et al., 2013).

In this case study, formative message testing with the intended audience of adult tobacco users helped to identify the most effective advertisements to air and compelled the decision to air them. This appears to have been the correct decision based on the observed service volume increases.

Example 2

The findings of the online message testing showed that all three spots successfully communicated their intended key educational messages and scored high on other factors related to their perceived effectiveness, such as being personally relevant, believable, and memorable. The results also demonstrate that although these ads created discomfort and anger among viewers, those feelings did not detract from the intended messages, nor did the emotional response create anger toward the sponsor. ClearWay Minnesota concluded that the campaign was on-course and continued airing all three ads with relatively equal airtime.

General Discussion

The two examples presented here highlight ways that online message testing can be used to gauge public responses to social marketing and educational mass-media content quickly and efficiently. Total time for these online surveys, from creation of the instrument to applying results, was approximately 6 weeks, and the total cost was approximately US$25,000. These examples also demonstrate several ways ClearWay Minnesota used these findings, including strategically selecting media content for airing and evaluating whether an existing campaign was having its intended motivational or educational impact. ClearWay Minnesota was able to use these results to make important decisions using measures of perceived effectiveness and other attributes quickly. For organizations that operate in the public eye, such rapid feedback can assist with pragmatic and strategic decisions that can save an organization time and money and build credibility.

The methodology for online message testing presented here does have several limitations. First, not all segments of the population have equal access to the Internet. Those people without access would be unable to participate in online surveys. Second, like almost all cross-sectional studies, issues of coverage, weighting, and representation exist. Finally, the method by which participants are selected introduces potential biases, and these must be taken into account for both the analysis and the interpretation of results. However, online panels offer a far larger and more diverse source of research participants than was possible when testing of visual messages had to be done in shopping malls or with prerecruited central location tests. In addition, the American Association for Public Opinion Research (AAPOR) Opt-In Online Panel Task Force (2010) acknowledged that in cases where research is not intended to produce precise estimates of population values, “the generally lower cost and unique properties of Web data collection is an acceptable alternative to traditional probability-based methods (p. 5).”

Despite some limitations, businesses large and small continue to use online-based surveys for tactical as well as strategic decision making. Precise point estimates are rarely expected as long as the relative magnitudes of attitudes and interest can be determined. Fully acknowledging the statistical imperfections of the methods, we believe online panel research can assist ClearWay Minnesota and others in making well-informed programmatic decisions.

Conclusions

Online message surveys are an economical and effective way to support testing TV ads for several important purposes, including determining which ads are most effective in promoting behavior change with a target audience and assessing intended and unintended impacts of ads on knowledge, perceptions, and behavior. As such, online message surveys can improve program effectiveness and protect investments in marketing campaigns. Public health mass-media campaigns aimed at changing attitudes or behavior can be very expensive to run, particularly TV ads. For organizations that require close to real-time feedback in order to most efficiently use public health dollars and maximize the impact of public health mass-media campaigns, online message testing is a useful tool to apply.

Footnotes

Acknowledgments

The authors wish to thank Clarity Coverdale Fury, an advertising agency in Minneapolis, Minnesota, for creating ClearWay Minnesota’s advertising campaigns.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: All funding for the studies described here was provided by ClearWay Minnesota.

References

Abroms

L. C.

Maibach

E. W.

(2008). The effectiveness of mass communication to change public behavior [Review]. Annual Review of Public Health, 29, 219–234. doi:10.1146/annurev.publhealth.29.020907.090824

Ajzen

Fishbein

(1980). Understanding attitudes and predicting social behavior. Englewood Cliffs, NJ: Prentice-Hall.

American Association for Public Opinion Research Opt-In Online Panel Task Force. (2010). AAPOR Report on Online Panels. Deerfield, IL: Author.

Beaudoin

C. E.

(2002). Exploring antismoking ads: Appeals, themes, and consequences. Journal of Health Communication, 7, 123–137. doi:10.1080/10810730290088003

Blue Cross and Blue Shield of Minnesota. (2010). Health care costs and smoking in Minnesota: The bottom line. Eagan, MN: Author.

Campbell

Parent

Plangger

(2011). Instant innovation: From zero to full speed in fifteen years; How online offerings have reshaped marketing research. March 2011 Supplement; 72-84. Journal of Advertising Research, 5, 72–87. doi:10.2501/JAR-51-1-072-086

Centers for Disease Control and Prevention. (2008). Smoking-attributable mortality, years of potential life lost, and productivity losses—United States, 2000–2004. MMWR Morbidity and Mortality Weekly Report, 57, 1226–1228.

Centers for Disease Control and Prevention. (2014). Best practices for comprehensive Tobacco control programs—2014. Atlanta: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health.

ClearWay Minnesota. (2011). Media evaluation study: Wave 2. Communications. Minneapolis, MN: Author.

10.

Davis

K. C.

Farrelly

M. C.

Duke

Kelly

Willett

(2012). Antismoking media campaign and smoking cessation outcomes, New York State, 2003–2009. Preventing Chronic Disease, 9, E40.

11.

Davis

K. C.

Nonnemaker

Duke

Farrelly

M. C.

(2013). Perceived effectiveness of cessation advertisements: The importance of audience reactions and practical implications for media campaign planning [Randomized Controlled Trial]. Health Communication, 28, 461–472. doi:10.1080/10410236.2012.696535

12.

Davis

K. C.

Nonnemaker

J. M.

Farrelly

M. C.

Niederdeppe

(2011). Exploring differences in smokers’ perceptions of the effectiveness of cessation media messages [Research Support, Non-U.S. Gov’t]. Tobacco Control, 20, 26–33. doi:10.1136/tc.2009.035568

13.

Dillard

Shen

Vail

(2007). Does perceived message effectiveness cause persuasion or vice versa? 17 consistent answers. Human Communication Research, 33, 467–488. doi:10.1111/j.1468-2958.2007.00308.x

14.

Dillard

Weber

Vail

(2007). The relationship between the perceived and actual effectiveness of persuasive messages: A meta-analysis with implications for formative campaign research. Journal of Communication, 57, 613–631. doi:10.1111/j.1460-2466.2007.00360.x

15.

Durkin

Brennan

Wakefield

(2012). Mass media campaigns to promote smoking cessation among adults: An integrative review. Tobacco Control, 21, 127–138.

16.

Farrelly

M. C.

Duke

J. C.

Davis

K. C.

Nonnemaker

J. M.

Kamyab

Willett

J. G.

Juster

H. R.

(2012). Promotion of smoking cessation with emotional and/or graphic antismoking advertising [Research Support, Non-U.S. Gov’t]. American Journal of Preventive Medicine, 43, 475–482. doi:10.1016/j.amepre.2012.07.023

17.

Koh

H. K.

Sebelius

K. G.

(2012). Ending the Tobacco epidemic. Journal of the American Medical Association, 308, 767–768.

18.

Lefebvre

(2001). Theories and models in social marketing. In Bloom

Gundlach

(Eds.), Handbook of marketing and society (pp. 506–518). Newbury Park, CA: Sage.

19.

Levy

D. T.

Boyle

R. G.

Abrams

D. B.

(2012). The role of public policies in reducing smoking: The Minnesota SimSmoke tobacco policy model [Research Support, Non-U.S. Gov’t Validation Studies]. American Journal of Preventive Medicine, 43, S179–S186. doi:10.1016/j.amepre.2012.07.037

20.

Macer

Wilson

(2014). Cross crosstabs off the list? A Report on the 2013 Market Research Technology Survey. Quirks Marketing Research Review, 38–43. fromhttp://www.quirks.com/pdf/201406_quirks.pdf

21.

McAfee

Davis

K. C.

Alexander

R. L. J.

Pechacek

T. F.

Bunnell

(2013). Effect of the first federally funded US antismoking national media campaign. Lancet, 382, 2003–2011.

22.

National Cancer Institute. (2008). The role of the media in promoting and reducing Tobacco use: Tobacco Control Monograph No. 19. Bethesda, MD: U.S. Department of Health and Human Services, National Institutes of Health, National Cancer Institute.

23.

Niederdeppe

Farrelly

M. C.

Nonnemaker

Davis

K. C.

Wagner

(2011). Socioeconomic variation in recall and perceived effectiveness of campaign advertisements to promote smoking cessation [Research Support, Non-U.S. Gov’t]. Social Science & Medicine, 72, 773–780. doi:10.1016/j.socscimed.2010.12.025

24.

Noar

S. M.

(2006). A 10-year retrospective of research in health mass media campaigns: Where do we go from here? [Review]. Journal of Health Communication, 11, 21–42. doi:10.1080/10810730500461059

25.

Nonnemaker

(2013). Do different styles of antismoking ads influence the types of smokers who call quitlines?Health Education Research, 28, 23–30. doi:10.1093/her/cys080

26.

Novelli

W. D.

(1990). Applying social marketing to health promotion and disease prevention. In Glanz

Lewis

Rimer

(Eds.), Health behavior and health education (pp. 342–369). San Francisco, CA: Jossey-Bass Publishers.

27.

ORC Macro/APCO. (2004). Theoretical framework multi-year mass media campaign plan. Prepared for United States Department of Agriculture, Food Safety and Inspection Service, Food Safety Education Staff. Retrieved November 13, 2015, fromhttp://www.fsis.usda.gov/wps/wcm/connect/8f313f7f-57bc-465b-b78d-c8204db01e48/BFS_Theoretical_Framework.pdf?MOD=AJPERES

28.

Pearson

J. L.

Richardson

Niaura

R. S.

Vallone

D. M.

Abrams

D. B.

(2012). e-Cigarette awareness, use, and harm perceptions in US adults [Research Support, N.I.H., Extramural]. American Journal of Public Health, 102, 1758–1766. doi:10.2105/AJPH.2011.300526

29.

Rogers

(1996). Up-to-date report. Journal of Health Communication, 1, 15–24.

30.

Siegel

(1998). Mass media antismoking campaigns: A powerful tool for health promotion [Historical Article]. Annals of Internal Medicine, 129, 128–132.

31.

Stephenson

M. T.

Witte

(1998). Fear, threat, and perceptions of efficacy from frightening skin cancer messages. Public Health Reviews, 26, 147–174.

32.

The Advertising Research Foundation. (2014, 625). Retrieved June 18, 2014, fromwww.thearf.org

33.

U.S. Department of Health and Human Services. (2010). How tobacco smoke causes disease: The biology and behavioral basis for smoking-attributable disease: A report of the Surgeon General. Atlanta, GA: U.S. Department of Health and Human Services, CDC.

34.

Wakefield

M. A.

Loken

Hornik

R. C.

(2010). Use of mass media campaigns to change health behaviour. Lancet, 376, 1261–1271.

35.

Wilson

L. M.

Avila Tang

Chander

Hutton

H. E.

Odelola

O. A.

Elf

J. L

… Apelberg

B. J.

(2012). Impact of Tobacco control interventions on smoking initiation, cessation, and prevalence: A systematic review. Journal of Environmental and Public Health, 2012, 961724.

36.

Witte

(1992). Putting the fear back in fear appeals: The extended parallel process model. Communication Monographs, 59, 329–349. doi:10.1080/03637759209376276

37.

World Health Organization. (2012). WHO global report: Mortality attributable to Tobacco. Geneva, Switzerland: Author.