Testing Overall and Synergistic Campaign Effects in a Partisan Statewide Election

Abstract

Although studies based on field experiments and large-N surveys have enhanced our understanding of how campaigns affect U.S. elections, few of these projects have (1) considered the synergistic effects of distinct aspects of the campaign, (2) focused on statewide, partisan elections, or (3) considered the durability of any estimated campaign effects. We rely on a massive field experiment from the 2014 Texas gubernatorial campaign to assess the individual, synergistic, and collective impact of a variety of outreach modes on the electorate. The data demonstrate some durable synergistic and overall campaign effects on voters’ attitudes toward the sponsoring candidate, with lesser effects on turnout. In addition, while the results indicate that television is rightly considered the most effective mover of voters, radio and Internet advertising also have notable effects and may, in fact, deliver a better return on investment.

Keywords

campaigns elections field experiments mobilization persuasion

Introduction

Since the late 1990s, political science research has shed considerable light on the influence of common modes of campaign communication—mail, telephones, and in-person contacting—on U.S. voters and elections. But the effects of the dominant modes—broadcast and online advertising—as well as the synergistic and overall effects of electioneering, remain a matter of dispute. The main reason for this is straightforward: American campaigns are noisy, and it is difficult to isolate the independent impact of singular forms of outreach. More generally, calculating the sum of these impacts and offering estimates of the interactive and total effects of the campaign is close to impossible.

Still, the allure of learning more about this essential element of democratic functioning is compelling. As many have observed, campaigns are the connective tissue between voters and elected officials (Campbell 2008). Campaigns are where candidates convey promises and commitments that bind them to specific policy acts as public officials. They establish the basis for accountability and, in no minor way, democracy.

Beyond their importance for democratic theory, we are also fascinated with campaigns because of the time, energy, and money they consume. Are they worth the effort we invest in them? Indeed, the disbelieving reaction of practitioners and pundits to findings of “minimal effects” on the part of some campaign scholars is no doubt caused by the assumption that smart, rational people could not possibly spend so much for so negligible an effect.

Against this backdrop, we want to know if campaigns successfully persuade and mobilize voters. To engage this question, the primary goals of this study are (1) to gauge the distinct and synergistic effects of different forms of campaigning, as designed and executed by real-life practitioners and (2) to estimate the cumulative effects of a campaign. To attain these goals, we rely on a field experiment testing the simultaneous effects of several distinct forms of campaign communication in a statewide, partisan campaign for governor of Texas.¹

Theorizing about Campaign Effects

When considering the impact of electioneering, we distinguish between “modes” of outreach. Different campaign modes vary on two salient dimensions: frequency and targeting precision. Frequency refers to the number of times that an individual voter is exposed to a campaign message. Targeting precision refers to the match between the campaign message and the attitudes of individual voters. Campaign effects are most likely when receptive voters are repeatedly exposed to persuasive messages.

To the chagrin of practitioners, campaign modes that offer the greatest potential with respect to frequency and repetition (television and radio) are also broadcast to many unreceptive voters. Conversely, campaign modes that offer the greatest potential for precise and personal targeting (digital, mail, phones, and face-to-face) are the most easily discarded, blocked, or refused by voters. This conundrum is borne out by empirical analyses, which show substantial but temporary effects for television advertising (Gerber et al. 2010) and small effects for digital, mail, and phones (Gerber and Green 2004; Huber and Arceneaux 2007).

Additional leverage can be gained on the question of aggregate campaign effects by considering voter (or “receiver”) characteristics. Previous studies demonstrate that the ability of campaigns to persuade is substantially conditioned by the attitudinal predispositions of the audience. Most notably, political awareness increases the likelihood of “reception” (Zaller 1992), but it also leads to greater stores of information and thus lessens the likelihood that voters will “yield” to additional information. High intensity election campaigns experienced by low awareness voters are therefore often regarded as the most likely to produce an impact (e.g., Ridout and Franz 2011).²

Besides political awareness, partisanship should moderate how voters react to campaign information. Independents, who lack a partisan filter that causes them to resist messages coming from certain sources, should be most receptive to this information. Partisans, on the contrary, should be moved by messages coming from their side, but will resist information coming from the other side.³

But while it is important to understand that campaigning has distinct forms and that voters come with various levels of political interest and attachment to the parties, it is equally important to acknowledge that both campaigns and voters are evolving. Campaigns have gotten better at identifying voters who are relatively more receptive to specific issue appeals (through large-N surveys and data analytics performed on registered voter lists) while campaign modes have been refined and expanded to improve their reach (through the proliferation of Internet access, as well as through the prevalence of cable and Internet television and satellite radio). Quite simply, campaigns have improved tactically. This improvement has been most apparent since the 2000 election, just as voters have polarized in their views of the two major parties (e.g., Iyengar and Westwood 2015).

These twin developments are consequential for our understanding of campaign effects. For example, in a race with a polarized electorate and precise targeting, we would expect significant effects through the activation or mobilization of partisans. We suspect that most statewide partisan elections of today—presidential, gubernatorial, and U.S. senate races—fall into this category. But many other election contests fall outside this category. For instance, some lower profile statewide races and many local elections feature a polarized environment but campaigning with less reach or with less precise targeting; these should result in less substantial mobilization. Conversely, in a less polarized environment with wide-spread yet targeted electioneering—for example, some mayoral or city council elections—campaigns should have significant persuasive effects on what is likely to be a relatively independent, open-minded electorate.

Research Expectations for the Texas Study

Assuming the posited relationship between reach, targeting precision, and voter receptivity, we have several attendant expectations. First, campaigning ought to affect vote choice. The theoretical literature discussed earlier and most empirical studies (e.g., Hillygus and Jackman 2003; Shaw 2006) suggest that campaigns can influence attitudes toward the candidates, so our focus is on confirming this conventional wisdom.

Second, campaigning should also affect turnout. Unlike studies of campaign persuasion, many turnout studies suggest that effects are small or nonexistent (e.g., Krasno and Green 2008). But recent research on aggregate campaigning and turnout is more positive (Issenberg 2012) and one study of battleground states in the 2012 presidential election estimated a mobilization effect of between 7 and 8 points (Enos and Fowler 2016). Furthermore, there is also the specter of increased polarization, which suggests mobilization is perhaps the most likely campaign effect. Consequently, we focus here on adding something new to a rather fulsome debate.

Our third (but perhaps most important) expectation is that campaigns have synergistic effects on voters. The notion that campaign effects are synergistic has only occasionally been tested, with little supportive evidence being proffered (Cardy 2005; Fieldhouse et al. 2013; Gerber and Green 2004). But these tests have largely been incidental to the studies, and they have focused on the interactive effects between direct mail and paid phone calls. We think the “reach versus targeting precision” conundrum makes it probable that campaigns use an “all of the above” approach. More to the point, we think the combination of different forms of outreach is likely to affect voters.

It is less obvious how synergistic effects occur. We suspect that reach modes are primarily used to increase awareness of a candidate and to establish some positive affect (e.g., Iyengar and Simon 2000). We also suspect they are sometimes used to establish the candidate’s issue priorities and credentials (e.g., Fowler and Ridout 2014). On the other side of the ledger, more precisely targeted modes convey specific messages to receptive voters. This outreach emphasizes conveying substantive and discrete information that can then be accessed when a voter must offer a candidate preference or vote (Hersh and Schaffner 2013). This “substantive and discrete information” is akin to what some opinion scholars call “considerations” (Zaller 1992) and is necessary to give firmness and consistency to the expressed vote choice. “Synergistic effects” therefore occur when someone becomes more likely to cast a vote for a candidate either because specific issue positions reinforce a generally positive feeling toward a candidate or because the personal characteristics of the candidate reinforce agreement on issue positions.

There is unmistakable evidence that campaigns think different forms of campaigning have a synergistic effect (e.g., Decker and Koster 2014). When planning their calendars, campaign practitioners employ television and radio to “set the table” for digital, mail, and phones, which offer more substantive and issue-based appeals. They also use targeted outreach to drive receptive voters to reach modes: mail, emails, or Internet ads invite or link voters back to online videos. This process is called “amplification” and can be viewed as a form of synergistic campaigning (Decker and Koster 2014). However varied its specific execution, synergistic campaigning is a staple of consultant talk in contemporary American elections and we expect that it will affect voters in this study.

Context

As mentioned above, this project flows from a unique opportunity to design and execute a field experiment in conjunction with a major, statewide partisan election campaign. The field experiments were conducted on behalf of Republican Greg Abbott’s gubernatorial campaign in advance of the March 4, 2014, Texas primary election. Abbott was the Attorney General of Texas, a position to which he was reelected twice after first winning the office in 2002. Abbott faced opposition in the Republican primary from Lisa Fritsch, whose writings and media personality had endeared her to Tea Party groups. Early polls, however, showed Abbott well above the 50 percent necessary to avoid a runoff election. In the end, Abbott handily defeated his opponents, amassing almost 1.22 million votes for 91.5 percent of the total tally.

The general election contest, however, was always considered more of a test by the Abbott campaign, with Wendy Davis, a state senator from Fort Worth, looming as the Democratic opponent. Davis burst onto the state and national scenes on June 25, 2013, when she held an eleven-hour filibuster on the floor of the state senate to block legislation that included restrictions on abortions. Following a flood of attention for her role in the abortion showdown, Davis announced her candidacy for the Democratic nomination for Texas governor in October 2013. She immediately became a prohibitive favorite to win the primary and ended up raising approximately $37 million.⁴ Polls from late 2013 showed Abbott with a single-digit lead over Davis, although Davis was better known.

During its December 2013 strategy meetings, the Abbott team decided to use the primary phase of the campaign to test the stand-alone and synergistic effects of different modes of campaign outreach.⁵ The purpose was to identify the most effective ways to influence voters’ perceptions of the candidates, as well as to mobilize likely Abbott supporters. Although the campaign outreach occurred in advance of the primary election, it was mostly designed to enhance Abbott’s standing and chances for the November general election.

Design

The goal of the study was to randomize all of Abbott’s campaign communications for three weeks in February, to test the independent effects of each outreach mode, as well as the overall effects of the campaign. Several practical realities made for some design dilemmas, however. For starters, the Abbott campaign wished to examine broadcast television, cable television, radio, Internet, and mail. The difficulty, of course, is that unlike mail or door knocks, one cannot easily randomize television, radio, or online advertisements at the individual level. The campaign also wanted to analyze both turnout in the primary and attitudes toward the likely general election candidates (Abbott and Davis). This required the acquisition of voting records from the state (for turnout analyses), as well as large-N surveys (for candidate attitudes). Finally, the campaign was concerned about the expense of the state’s three largest media markets: Dallas/Fort Worth, Houston, and San Antonio. These markets were excluded, as was Austin due to concerns about both its expense and “uniqueness” (as one can see from the analysis of clusters in Appendix A, no other markets in the state are easily matched with Austin). This meant that the tests required a design focused on the state’s remaining 12 media markets.⁶

Broadcast Television, Cable Television, and Radio

We began by identifying and gathering data on the media markets in play for broadcast television advertisements: Abilene-Sweetwater, Amarillo, Beaumont-Port Arthur, Corpus Christi, Harlingen-Weslaco-Brownsville, Lubbock, Odessa-Midland, El Paso, San Angelo, Tyler-Longview-Lufkin-Nacogdoches, Waco-Temple-Bryan, and Wichita Falls. Following the design of Gerber et al. (2010), we then sorted the media markets into matched groups based on past partisan voting averages (average Republican share of the two-party vote in statewide races in 2010 and 2012), percent Black and percent Hispanic, percent below 30 years of age, and median household income. The four matched groups (or “clusters”) we created were (1) Odessa, Lubbock, and Amarillo; (2) Harlingen, Corpus Christi, and El Paso; (3) Abilene, San Angelo, and Wichita Falls; and (4) Beaumont, Waco, and Tyler. The first market of each cluster was randomly designated as a treatment market for broadcast TV ads, as well as cable TV, Internet, mail, and some radio outreach; the second (also based on a random draw) was not subjected to broadcast TV ads but was subject to cable TV, Internet, mail, and some radio outreach; the third was randomly designated as a control, with no campaigning at all. To facilitate our analysis of targeted television (see below), Midland and Harlingen were randomly assigned “targeted” broadcast television, while Abilene and Beaumont were randomly assigned “traditional” broadcast television. A table showing treatment-control assignments, by outreach mode, for every Texas media market is presented in the Appendix B. A map of designated market areas in Texas is available in our section 1 of our supplemental material.

At this point, we owe the reader a description of “targeted television” advertising. For years, viewership numbers for broadcast and cable television have been purchased by political campaigns from Nielsen, whose data are also used by television stations to determine advertising rates. These data are analyzed by candidates, who concentrate their ads during dayparts and on shows watched by targeted voters. But Nielsen’s demographic profiles and viewership estimates are not as detailed or as accurate as many campaigns prefer. Recently, consulting companies have begun to offer “targeted television advertising” buy plans, which purchase ads in specific day-parts and on shows based on Nielsen ratings plus analyses of large surveys and commercial vendor lists. The practical consequence of targeted TV ad buys appears to be more advertisements placed on less-prominent, niche shows, such as “Community” or “Rookie Blue” (or, on cable, “Judge Joe Brown” for Democrats or “Friday Night Wrestling” for Republicans). Preliminary, unpublished research suggests that targeted TV may be relatively better at influencing vote choice than traditional TV ad buys. We suspect that this research is correct. In the past, targeted TV has been an oxymoron, but the proliferation of viewing options coupled with better data on viewership could increase the ability of campaigns to focus their messages on the most receptive audiences and influence their behavior.

After consulting with the campaign’s television team, the treatment was set at 600 targeted ratings points per week, for three weeks (February 12–March 4, 2012).⁷ Two spots were alternated in each of the treatment markets. The first, titled “Never Stop Fighting,” was a 30-second ad focusing on Abbott’s personal and professional history. The ad notes that he was paralyzed by a tragic accident and plays up Abbott’s clashes (as Texas’s Attorney General) with President Obama on second amendment issues.⁸ The second, “No More Gimmicks,” lambasts Texas politicians for collecting taxes for roads, education, and public safety, and then spending these funds on other things.

The cable television experiment necessarily differs from its broadcast TV counterpart. Initially, we sorted zip codes from the eight “non-control” markets—Odessa, Lubbock, Harlingen, Corpus Christi, Abilene, San Angelo, Beaumont, and Waco—into matched pairs based on the same demographic and political characteristics we used to identify media market clusters.⁹ For each of these 31 pairs, one was randomly designated to receive cable TV ads, and the other was designated as a control. Cable TV ads were then aired in the treatment zones, and voters in those zones were compared with voters in the matched control zones.

Some aspects of the cable TV ad test, however, are like the broadcast test. As with broadcast TV, the Abbott campaign wanted to test the relative effectiveness of targeted ad buys, so whether cable TV ads were targeted or traditional was randomly determined for each of the treated cable zones. In addition, the same two 30-second spots that ran on broadcast TV also alternated on cable.

The radio test was constructed to parallel the broadcast TV tests, only without a “targeted” versus “traditional” aspect. One market from each of the four clusters—specifically, Abilene, Corpus Christi, Lubbock, and Waco—was randomly designated for radio advertisements.¹⁰ These were treated with 600 points per week for three weeks (February 12–March 4). The radio tests consisted of a single 60-second radio ad, titled “Preserve, Protect, and Defend,” which combines the scripts from the two TV ads, detailing the accident that paralyzed Abbott, the offices that he has held, and his advocacy for Texas in the face of the Obama Administration, the Environmental Protection Agency, and “federal overreach.”

Internet, Pre-roll, and Facebook

After consulting with Abbott’s digital team, we also tested three forms of online and social media outreach: Internet advertising, pre-roll advertising, and Facebook advertising. Internet advertising includes banner and sponsored ads which promote “click-throughs” to an online video. Pre-roll advertising is the video ad shown before a YouTube or other selected video. Facebook advertising invites those who “like” Greg Abbott to click-through to a video. For each of the online tests, the video links directed browsers to one of the two Abbott TV ads.

Although it is possible to target online advertising based on individual-level purchasing and browsing patterns, in 2014 almost all online political advertising was done by buying banner or side-bar ads on websites known to be frequented by targeted voters. In this way, online advertising is like TV or radio advertising. Because website ads vary by zip code—users with IP addresses in a specific zip code see ads associated with that zip code—Internet and pre-roll advertising treatments were each randomized at this level. Following the cable TV experiment design, we randomly assigned one of each matched zip code pair to the treatment group and the other to the control group. We then compared the attitudes and behaviors of those in the treatment zip codes to those in the control zip codes.

Unlike other online ads, Facebook advertising treatments were each randomized at the individual level, as those who “liked” Greg Abbott were randomly assigned into either the treatment or control groups. Note that the Facebook experiment is thus limited to those who have expressed an interest in Abbott.

Direct Mail

Although the Abbott campaign was especially interested in the broadcast and online tests, they also agreed to include experiments for direct mail. To test these effects, 500,000 households in the target universe (described below) were randomly selected to receive mail while another 500,000 randomly selected households received no mail. Within the treatment universe, individuals received four mail pieces: the first was a generic piece stating that exercising your right to vote helps keep Texas great, the second questioned Davis’s claim that she’d be a “pro-life” governor, and the third and fourth presented the individual with their voter turnout record and urged them to participate in the upcoming primary.

A Few Notes on the Design

Several features of the design merit additional comment. First, the professionals designed the outreach. As noted earlier, Abbott’s television team produced and aired two advertisements, while the radio team produced and aired a single 60-second spot. Similarly, the digital team produced unique online ads encouraging people to “click through” to Abbott videos, and the mail vendors designed all four direct mail pieces.

Second, the leaders of the Abbott campaign specifically instructed the professionals to submit budgets that would “allow them to do what they needed to produce an impact.” In other words, there was a concerted effort to allow those being examined to put forth their best work with sufficient force and repetition. The campaign eventually approved all submitted budgets, including approximately $600,000 for broadcast television, $337,000 for cable television, $205,000 for radio, $227,000 for online, and $182,000 for direct mail.

Third, while messages were strikingly consistent across treatments, there were differences owing to both mode and intent. For example, the final mail piece emphasized specifics about when and where to vote. Also, the Facebook and Internet advertisements (and one mailer) included a negative mention of Wendy Davis, whereas other modes were devoid of any reference to the likely Democratic nominee.¹¹ Thus, we allowed the “real world” directive to trump complete message consistency (which we judged impossible due to mode differences anyway). It is therefore possible that effect differences noted here are partly due to message rather than exclusively due to mode.

Finally, the universe under analysis did not include all registered voters. Notably, registrants from the top four media markets were set aside. For 2014, the Texas voter file contained just over 14 million records, of which roughly six million were from the nonmajor media markets. Of these six million records, over two million were in areas designated to receive campaign treatments. Furthermore, across these two million records the experiments focused on the half of the voter file most likely to turn out and support Abbott (the “top fifty percent”).¹² This was an accommodation to the realities of the campaign; the Abbott people did not want to “waste” outreach and measurement on those least likely to cast a ballot for him. That said, there is some advantage to this design, especially with respect to turnout. Campaigns do not attempt to increase turnout across the board; rather, they attempt to increase turnout among people with a decent chance of voting for their candidate. Thus, we focus on that segment of the electorate where both persuasive and (especially) mobilization effects are most likely. In sum, over one million voters were identified for treatment, with seven modes being tested (broadcast television, cable television, radio, Internet advertising, Facebook advertising, pre-roll advertising, and mail).

Measuring Effects

“To assess campaign effects, we rely on large-N surveys (to gauge the impact of campaign treatments on candidate affect), as well as an updated, postelection voter file (to gauge the impact of campaign treatments on turnout). We lean most heavily on the postelection survey, which was conducted on March 5–6, 2014, immediately after the March 4 election.¹³ The postelection survey was conducted on March 5–6, 2014, immediately after the March 4 election. We randomly contacted voters in Abbott’s “top fifty percent” (but outside the state’s four largest media markets) via automated telephone calls. Respondents were asked to rate Abbott and Davis, to register their candidate preference in the Republican Primary election, to indicate whether they had “read, seen, or heard anything from the Abbott campaign for governor in the last week,” and to record their gender, age, and party identification. The text of the survey—which is presented in section 2 of the online supplemental material along with topline results, additional details with respect to methodology, and distributions of basic demographic and political variables by media market cluster—yields no indication whether the sponsor is pro- or anti-Abbott. The total number of respondents in the postelection survey is 2,968. Of the phone numbers attempted, 10 percent yielded interviews; excluding nonworking or unanswered numbers, the response rate was 23 percent. The resultant data were weighted to a demographic profile of Abbott supporters derived from analyses of the voter file.¹⁴

Note that by relying on a postelection survey—conducted well after the last week in February, when most of Abbott’s campaign treatment materials reached voters—we gain some purchase on assessing the durability of effects. As discussed earlier, this is consequential because recent studies strongly indicate that campaign effects are often transitory and decay within a few days (Gerber et al. 2010).

To gauge the effects of distinct types of campaign outreach, we use (1) least squares regression models of Abbott’s net favorability rating,¹⁵ and (2) logistic regression models of Republican primary turnout. The Abbott net favorability models include a dummy variable for voters below 30 years of age and strong Republican partisanship, while the turnout models include a continuous variable rating turnout likelihood. Controlling for potential covariates is necessary because the assignment of broadcast TV and radio treatments was done at an aggregate level and preliminary diagnostic tests indicate minor individual-level imbalances by age, partisanship, and turnout propensity.¹⁶ To further control for the fact that randomization of broadcast TV and radio is within clusters at the media market level, we include separate dummy variables for these clusters in our models and calculate both traditional as well as clustered robust standard errors.

Results

A quick and dirty way to estimate the overall effects of the campaign treatments is to compare opinions from the pre- and postelection surveys. This comparison is, of course, rough because differences might be a function of exposure to campaign treatments but they might also be due to sample differences or random error. Still, survey error is quantifiable and it is minor here due to the large sample sizes.¹⁷ Table 1 presents the pre- and postelection survey results across key attitudinal and behavioral measures. Overall, the campaign appears to have had a positive and statistically significant impact.¹⁸ In the treatment markets, we see that Abbott’s net favorable rating (% favorable – % unfavorable) increased by 7 points, while in the control group his rating decreased by 1 point. Similarly, in the treatment markets, Abbott’s net advantage in the trial ballot increased by 12 points from the pre- to postelection survey (from +34 to +46 points). He also improved in the control markets, but only by about 6 points. In addition, when asked whether they had recently “read, seen, or heard” something from the Abbott campaign, the “yes – no” margin increased by 41 points in the treatment markets over the course of the three-week project. Among voters in the control group, the margin also increased, but by only 16 points.¹⁹ More broadly, the overall effects of the campaign experiment can be gleaned from the treatment-to-control differences: +8 points on Abbott’s net favorability and +6 points on the Republican primary ballot.

Table 1.

Estimates of Campaign Effects in Abbott Experiments.

	Respondents receiving treatment			Respondents in control group			Treatment effect compared with control?
	Pre	Post	Change	Pre	Post	Change	Difference significant at p < .01?
Republican primary ballot Abbott Margin	+34	+46	+12***	+40	+46	+6*	Yes
Greg Abbott favorability Favorable – unfavorable	+22	+29	+7***	+28	+27	−1	Yes
Wendy Davis Favorability Favorable – unfavorable	−27	−24	+3*	−23	−25	−2	No
“Have you read, seen, or heard anything from the Abbott Campaign for Governor in the past week?” Yes – No	−24	+17	+41***	−21	−5	+16**	Yes

Here, we compare preelection survey results (N = 4,198) with those of the postelection (N = 2,968). In data columns 3 and 6, bold values indicate pre to postelection shifts in favorability. Significance estimates are derived from t tests for means from independent samples. For the estimates of the significance of pre- to postelection survey change, they are based on net differences in the preelection poll compared with net differences in the postelection poll. For example, we calculate the significance of Abbott’s preelection ballot margin (+34) compared with his postelection margin (+46). Similarly, we compare net favorability scores and read/seen/heard versus not scores from the pre- and postelection surveys. For the estimates of the significance of the differences between treatment and control effects, we also rely on t tests for means from independent samples. Whether or not the difference between the treatment and control effect is statistically significant at the 0.01 level is indicated by bold entries in the final data column.

p < .05. **p < .01. ***p < .001.

Effects on Candidate Favorability

Given the nature of the Abbott study, simple comparisons are instructive but not dispositive. Multivariate models, though, provide unbiased estimates of experimental campaign effects by correcting for any attitudinal and demographic imbalances between voters in treatment and control markets. Focusing on the postelection survey and the measure of net favorability toward Abbott, Table 2 offers simple least squares regression coefficients and shows that targeted broadcast television, radio, direct mail, and Internet treatments improved Abbott’s net favorability, while traditional broadcast television, cable television, Facebook, and pre-roll advertising tended to decrease it. Figure 1 takes the coefficient estimates from the models and graphs them, including lines representing the (95%) confidence intervals, so that we get a clearer sense of the substantive and statistical magnitude of the effects. This shows that the positive effects associated with targeted broadcast TV, radio, and Internet are very close to conventional levels of statistical significance despite our “conservative” estimation strategy.²⁰

Table 2.

Effects of Treatments on Abbott Favorability.

	Regression coefficient	Standard error	Significance	Lower bound (95% CI)	Upper bound (95% CI)
Dependent variable = net Abbott favorability (100 = favorable, –100 = unfavorable)
Constant	20.88	5.83	0.00	9.45	32.32
Strong Republican	49.61	3.05	0.00	43.64	55.59
Below 30 years of age	−15.80	3.80	0.00	−23.25	−8.34
Broadcast TV (targeted)	11.23	7.09	0.11	−2.68	25.13
Broadcast TV (traditional)	−2.29	5.02	0.65	−12.13	7.54
Cable TV (targeted)	−2.71	4.93	0.58	−12.37	6.96
Cable TV (traditional)	−0.86	6.04	0.89	−12.70	10.99
Radio	8.02	5.19	0.12	−2.16	18.20
Mail	2.31	3.18	0.47	−3.93	8.55
Internet	5.54	3.41	0.10	−1.14	12.23
Facebook	−3.45	3.26	0.29	−9.84	2.94
Pre-Roll	−3.23	3.43	0.35	−9.95	3.48
Beaumont-Waco-Tyler (cluster 2)	−13.53	5.26	0.01	−23.85	−3.21
Brownsville-Corpus Christi-Tyler (cluster 3)	−17.93	6.25	0.00	−30.19	−5.68
Midland-Lubbock-Amarillo (cluster 4)	−6.40	6.09	0.29	−18.35	5.55
N	2,967
Adjusted R²	.100
Standard error	76.67
F statistic	22.93
Durbin-Watson	2.00

Entries are based on a least squares model of Greg Abbott favorability in which the treatment variables, along with age (1 = below 30 years old, 0 = else) and strong Republican voting prediction (1 = strong Republican voter, 0 = other), are the predictor variables. The F statistic is significant at the 0.00 level. Analysis includes markets with media treatments and other campaign treatments, markets with no media treatments but with other campaign treatments, and markets with no campaign treatments at all. The baseline is the Abilene-San Angelo-Wichita Falls cluster. CI = confidence interval.

Figure 1.

Effects of campaign communication on net favorability toward the Republican candidate.

An estimate of the overall effect of campaign treatments can be obtained by substituting the mean values of the treatment variables into the model, calculating Abbott’s net favorability, and then comparing this to Abbott’s net favorability when the treatment variables are set to zero. This exercise pegs the overall effect of all campaign treatments on Abbott’s net favorability at +7.1 points.

Keep in mind that the estimated effect is not across the entire electorate; rather, it is for voters in Abbott’s “top fifty percent.” Still, even if the effect were zero across all other voters the total impact would be about 3.5 points.²¹ As with the earlier comparison of pre- to postsurvey results, this constitutes a significant and durable effect—one that is consistent with our first expectation about the overall persuasive impact of the campaign.

Collectively, the experiments yield unambiguous evidence for our expectation that campaign effects vary considerably by mode. We see treatment effects (p ~ .1) for targeted broadcast television, radio, and Internet advertising. On the other side of the ledger, treatment effects are close to zero for mail, cable television, pre-roll, and Facebook advertising.

Turnout

Moving from persuasion to mobilization, Table 3 presents our logistic regression model of turnout for the 2014 GOP primary. The models analyze data from the Texas voter file and present the log-odds ratios for the different treatments. Note again the conservative modeling strategy—the presence of covariates to control for imbalances, as well as dummy variables for the media market clusters. Two important findings are immediately evident. First, the magnitude of effects is smaller than we saw for the models of net favorability toward the candidate. This is understandable: it is easier to move favorability a few points than to increase the probability that someone will vote in an uncompetitive primary election. Second, campaign outreach had mixed effects overall on GOP primary turnout. On the plus side, broadcast TV (both targeted and traditional) and Internet advertising had positive and statistically significant impacts on turnout. On the minus side, cable TV (both targeted and traditional), and Facebook and pre-roll advertising had negative and statistically significant impacts. The influence of radio and mail on turnout was statistically indistinguishable from zero.

Table 3.

Effects of Treatments on Republican Primary Turnout.

	Regression coefficient	Standard error	Significance	Lower bound (95% CI)	Upper bound (95% CI)
Republican primary turnout (1 = voted, 0 = did not vote) Campaign treatment
Constant	−3.52	0.01	0.00
Turnout propensity	5.98	0.01	0.00	385.65	401.98
Broadcast TV (targeted)	0.14	0.01	0.00	1.12	1.18
Broadcast TV (traditional)	0.19	0.01	0.00	1.18	1.23
Cable TV (targeted)	−0.14	0.01	0.00	0.85	0.88
Cable TV (traditional)	−0.29	0.01	0.00	0.73	0.77
Radio	0.01	0.01	0.23	0.99	1.03
Mail	−0.01	0.01	0.28	0.98	1.01
Internet	0.06	0.01	0.00	1.05	1.08
Facebook	−0.05	0.01	0.00	0.94	0.96
Pre-roll	−0.02	0.01	0.03	0.97	1.00
Beaumont-Waco-Tyler (cluster 2)	−0.33	0.01	0.00	0.71	0.73
Brownsville-Corpus Christi-Tyler (cluster 3)	−0.77	0.01	0.00	0.45	0.48
Midland-Lubbock-Amarillo (cluster 4)	−0.02	0.01	0.09	0.96	1.00
N	1,468,935
Nagelkerke R²	.50
Standard error	0.002
Percent correctly predicted	87.4

Entries are based on a single logistic regression model of Republican primary turnout in which the treatment variables, along with age (1 = below 30 years old, 0 = else) and turnout propensity scores (0 to 1, with 0 representing almost certain not to vote and 1 representing almost certain to vote), are the predictor variables. Analysis includes markets with media treatments and other campaign treatments, markets with no media treatments but with other campaign treatments, and markets with no campaign treatments at all. The baseline is the Abilene-San Angelo-Wichita Falls cluster. CI = confidence interval.

Most of the estimated effects reach conventional levels of statistical significance because the turnout models draw on almost 1.5 million cases. As suggested earlier, their substantive magnitudes are more difficult to gauge, especially given that we are looking at log-odds ratios rather than regression coefficients. Figure 2 converts these ratios into predicted probabilities of voting in the GOP primary and offers a visual representation of how the different campaign mode treatments affected voters. From this, we see that broadcast TV improved the chance that a voter would turnout for the Republican primary by approximately one-half percentage point, with traditional broadcast having a slightly greater impact. The influence of both targeted and traditional cable television advertising, on the contrary, was counterproductive. Exposure to cable TV ads reduced the likelihood of voting in the Republican primary by an average of about one-half percentage point.

Figure 2.

Effects of campaign communication on predicted probability of voting in the Republican primary.

Internet advertising also increased turnout likelihood, by about one-quarter percentage point. Meanwhile, other forms of digital outreach did either nothing (pre-roll advertising) or had a marginally negative effect (Facebook advertising). Mail and radio were similarly ineffective, although radio had a very marginal positive impact.

To ascertain the overall impact of the campaign treatments, we can perform the same sort of calculation for turnout that we did for candidate favorability. From the logistic regression model, we estimate that campaign advertising improved Republican primary turnout by about 0.24 percentage points. This is consistent with our expectations, although hardly the sort of eye-catching result that many consultants would hope for.

Once again, the disparate findings by mode—some of which are statistically and substantively significant—conform to our expectations. We note the ability of Internet advertising and broadcast TV to draw voters to the polls in a low salience primary election. Conversely, the smallish aggregate effects might be attributable to both the relatively uncompetitive nature of the election and the lack of outreach specifically designed to help get voters to the polls.

Distinct and Synergistic Campaign Effects

Figure 3 plots how different modes of electioneering affect both Abbott favorability and Republican primary turnout in Texas, allowing us to better comprehend their overall influences. It is worth reiterating that the wide confidence intervals surrounding our estimates of persuasive effects on Abbott’s favorability—which suggest even the largest effects are, by conventional standards, only marginally significant statistically—reflect our conservative estimation strategy. Still, we see that targeted broadcast television advertising stands out as effective on both dimensions. Radio and Internet advertising have smaller, yet notable positive influences as well. Traditional broadcast TV, on the contrary, boosts turnout but does little for Abbott’s favorability. Other modes of digital advertising and cable television cluster closer to the origin point of the graph, but none has a positive or significant effect on either Abbott’s favorability or turnout.

Figure 3.

Effects of campaign communication on candidate favorability and Republican primary turnout.

From the practitioner’s perspective, Figure 3 suggests a different mix of outreach over the course of the campaign: activities that move favorability numbers should be front-loaded, while those that increase turnout should be held in reserve until the last days of the race. More specifically, a campaign might profitably focus on radio and Internet advertising early in the contest and then switch to broadcast TV (even traditional broadcast TV) late in the campaign.

Now we consider the claim of campaign professionals that distinct forms of outreach have a synergistic within a broadcast environment. Figure 4 shows the singular and interactive effects of different modes of broadcast campaigning. These estimates rely on distinct models: (1) Abbott’s favorability and (2) Republican primary turnout as a function of Republican partisanship/turnout propensity, dummy variables for our media market clusters, and five key dummy variables representing distinct treatments of broadcast campaign outreach. Those include voters who received (1) only broadcast TV ads, (2) only radio ads, (3) only broadcast TV and cable TV ads, (4) only broadcast TV and radio ads, and (5) only broadcast TV, cable TV, and radio ads.²²

Figure 4.

Interactive effects of campaign communication on candidate favorability and Republican primary turnout.

Figure 4 reveals that broadcast advertising’s most powerful positive effects are, in fact, synergistic: the combined effect of broadcast TV + cable TV + radio boosts Abbott’s favorability by over 10 points and increases Republican turnout likelihood by almost a full percentage point. Broadcast TV + radio has a smaller positive effect on Abbott favorability (a little under 3 points) but an even larger effect on turnout (+1.3 points). Other broadcast advertising effects increase favorability toward Abbott but have a slightly depressing effect on turnout. Despite this last fact, the preponderance of evidence is that the combined effect of broadcast advertising was modest but positive.

A similar approach was taken to generate the results behind Figure 5. Here, we modeled Abbott’s favorability and Republican primary turnout as a function of partisanship/turnout propensity, dummy variables for our media market clusters, and seven dummy variables representing those whose exposure to digital advertising was confined to (1) Internet only, (2) Facebook only, (3) pre-roll only, (4) Internet and Facebook only, (5) Internet and pre-roll only, (6) Facebook and pre-roll, and (7) Internet, Facebook, and pre-roll only. As with broadcast advertising, the data support the notion that understanding digital campaigning requires understanding how different elements of digital work (or don’t work) together.

Figure 5.

Interactive effects of campaign communication on candidate favorability and Republican primary turnout.

More specifically, Figure 5 shows that only Internet and pre-roll advertising had a positive effect on both Abbott favorability and turnout, improving Abbott’s rating by approximately 9 points and turnout likelihood by approximately 0.1 points. Internet alone, Facebook alone, and Facebook + pre-roll advertising all improved Abbott’s favorability but had a slightly demobilizing effect for the GOP primary. By contrast, the combined effect of Internet + Facebook + pre-roll slightly decreased Abbott’s favorability, but improved turnout by about 0.3 points. The poorest showings in the model come from pre-roll and Facebook + Internet advertising, which dropped Abbott’s favorability and demobilized Republican primary voters. Pre-roll was especially disappointing, as it managed to decrease Abbott’s rating and GOP turnout by 12 and 0.6 points, respectively.

We would emphasize three take-away points. First, Internet advertising appears to have improved favorability ratings for the candidate except in combination with Facebook. Second, digital advertising does not seem to be especially effective at mobilizing voters for the primary. Third, pre-roll advertising—at least as applied in this campaign—was simply counterproductive. More generally, it does appear that different combinations of digital advertising may produce effects that are quite distinct from one another.

Is it possible that digital advertising is more effective when voters are also exposed to a robust broadcast advertising campaign? We reran the models from Figure 5, focusing solely on voters who were in areas where there was a broadcast TV and radio campaign. There were some changes in estimated effects, but most were quite modest. At best, the estimated impact of synergistic digital modes receives a slight boost within the broadcast television environment. Overall, however, we did not see significant increases in the size of digital campaign effects within the broadcast markets. From our perspective, this constitutes weak evidence for the consultant’s claim that television advertising “primes the pump” and creates a more receptive and persuadable audience for other forms of outreach.

What about Cost?

A reasonable objection to this analysis is that these estimated differences in mode effects do not consider cost differentials. For example, might the relatively modest effects associated with digital outreach in our tests constitute a cost-effective investment? A crude way of equalizing for cost is to take the estimated effects by campaign mode along with the actual amount of money spent by mode, and use these to calculate the estimated return if each mode had been allotted $1,000,000. In the absence of data on the impact of elevated levels of spending for each mode, we assume that campaign investments deliver slightly diminishing returns. More specifically, we assume the logarithmic form is y = log₂x, such that effects beyond those estimated in our models are assumed to adhere to a base 2 log form.²³ This follows precedent from other studies of political advertising and predicts a plausible rate of decline for the persuasive and mobilizing effects associated with heavy campaign spending.

On a cost-per-voter basis, our estimates confirm what online enthusiasts have long maintained: the Internet does seem to provide an excellent relative return on investment. A $1,000,000 investment in Internet campaigning from the Abbott team would have theoretically raised his net favorability rating by +12.9 points and would have increased Republican primary turnout likelihood by +0.14 points. This supports our positive expectation about the relative, cost-controlled effectiveness of Internet advertising.

Even in this cost-sensitive analysis, however, broadcast TV still looks like a solid investment in relative terms. A $1,000,000 targeted TV buy would have produced a +16.9 point increase in favorability and a +0.2 point increase in turnout. Proponents of radio and (to a lesser extent) direct mail advertising might also be cheered by our results. For radio, a $1,000,000 investment in radio would have boosted favorability (+15.2 points) and turnout (+0.02 points). For mail, the effect of $1,000,000 would have been +6.2 on favorability but −0.02 on turnout. As noted earlier, cable television and other forms of digital did not fare well in this exercise, and additional spending on these modes would not have altered this result.

Conclusion

The endgame for any analysis of campaign effects is a statement about whether the campaign mattered for the election. Our study indicates that Abbott’s campaign had a significant overall impact. According to our models, his net favorability improved by 7.1 points over the campaign, and Republican primary turnout propensity improved by 0.24 points. We also have instructive data about the relative effectiveness of several different modes of outreach. Targeted broadcast TV advertising significantly increased Abbott’s favorability, as well as Republican primary turnout. In addition, Internet and radio advertising had positive effects, while the impact of cable TV was negative as were the impacts of other, singular forms of online advertising.

Yet an equally distinctive finding from this project is that the synergistic effects of different campaign modes are occasionally substantial. Specifically, we find that the combination of radio, cable, and broadcast TV moved both candidate favorability and turnout, and that pairing Internet and pre-roll advertising may be the best combination of digital outreach. To our knowledge, no previous study has explored the synergy between and among campaign modes as this one has.

Finally, we put our findings into context by considering the effectiveness of different campaign modes by their respective costs. From this perspective, online advertising effects may be comparable to those we find for broadcast advertising because digital ads remain decidedly less expensive than TV ads.

There are reasons to be cautious about our results. Most obviously, the tests were conducted during a minimally competitive primary election campaign. Furthermore, we excluded half of the electorate: those who were judged—based on vote history and micro-targeting—to be less likely to cast a vote for Abbott. This means that estimated effects are possibly overstated because we exclude from the analysis those judged to be most resistant to the campaign. However, even conceding this point, fifty percent of Texas voters in our test markets were in play based on our criteria. It is also worth observing that Davis was campaigning during this period, and much of her outreach sought to raise doubts about Abbott. Because of this, the tests were conducted during what we would describe as a slightly less intense approximation of the general election campaign.

Some might be concerned that we are studying a single state (Texas) and do not strictly control for message across the mode tests. There is little reason, however, to think Texas is exceptional in this instance. Furthermore, the rejection of individual location studies would put quite a dent in our collective knowledge of campaign effects. As for message effects, the consistency of message is notable across the modes, although differences are impossible to totally neutralize. One might also be worried that we did not systematically analyze Democratic primary turnout in this study. This means that we cannot definitively speak to the possibility that the treatments countermobilized Democrats. After conducting the main analyses, we did briefly examine Democratic primary turnout: it was so low (4.1% of registered voters, 3.0% of the voting age population) and varied so little that it provided almost no additional leverage on the question of turnout.

We should also acknowledge that one of Abbott’s consultants was concerned about the selection of Harlingen-Brownsville-McAllen as a treatment market. Although Harlingen matches with Corpus Christi, the match is not overwhelmingly strong (Harlingen is heavily Democratic and Hispanic) and there are reasons to think that using Harlingen as a treatment market might depress the estimated effects of Republican outreach. We did, of course, include a battery of covariates to ensure that treatment imbalances are controlled for in the individual-level analyses, as well as dummy variables for the different media market clusters. Still, it is possible that these do not completely control for political context or culture in a particularly unique environment. In the end, we were limited in the media markets at our disposal for the study, and did not think this issue was sufficient to warrant excluding it from either the randomization process or the subsequent analyses.

Most of these issues, in our view, are unavoidable and acceptable for a study of multiple modes of campaign outreach in the context of a major partisan statewide election. The exact magnitude of effects estimated here can be debated, but we think we are on firmer ground than most when extrapolating to the real world of campaigns given the scope of the experiments and involvement of expert, interested parties. Hopefully, this sort of cooperation will become much more common as campaigns institutionalize processes for testing, review, and auditing modes of outreach.

Footnotes

Appendix

Appendix B

Experimental Assignments.

	Broadcast television			Cable television			Radio		Internet		Pre-roll		Facebook		Mail		Overall
	Targeted	Traditional	Control	Targeted.	Traditional	Control	Treatment	Control	Treatment	Control	Treatment	Control	Treatment	Control	Treatment	Control	Control
Abilene		x		x		x	x		x	x	x	x	x	x	x	x
San Angelo			x			x		x	x	x	x	x	x	x	x	x
Wichita Falls																	x
Beaumont		x		x	x	x	x	x	x	x	x	x	x	x	x	x
Waco			x	x	x	x	x		x	x	x	x	x	x	x	x
Tyler																	x
Harlingen-Brownsville	x			x		x		x	x	x	x	x	x	x	x	x
Corpus Christi			x	x		x	x		x	x	x	x	x	x	x	x
El Paso																	x
Midland-Odessa	x			x	x	x		x	x	x	x	x	x	x	x	x
Lubbock			x	x	x	x	x		x	x	x	x	x	x	x	x
Amarillo																	x

Acknowledgements

The authors gratefully wish to acknowledge the invaluable contributions of the following individuals: Dave Carney, Wayne Hamilton, David Wilson, Ross Hunt, Jim Gimpel, Chris Wlezien, Lynn Vavreck, Joshua Kalla, David Brookman, and several anonymous referees. All errors and omissions are our own.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental materials for this article are available with the manuscript on the Political Research Quarterly (PRQ) website.

Notes

References

Ansolabehere

Stephen

Iyengar

Shanto

. 1996. Going Negative. New York: Free Press.

Campbell

James

. 2008. The American Campaign. 2nd ed. College Station: Texas A&M University Press.

Cardy

Emily A.

2005. “The Persuasion Effects of Partisan Direct Mail and Phone Calls.” Annals of the American Academy of Political and Social Science 601 (1): 28–40.

Chang

Chingching

. 2003. “Party Bias in Political-Advertising Processing.” Journal of Advertising 32 (summer): 55–67.

Decker

Steffi

Koster

Josh

. 2014. “Five Digital Mistakes to Avoid.” Campaigns and Elections, September 15, 16–18.

Druckman

James

. 2004. “Political Preference Formation: Competition, Deliberation, and the (Ir)relevance of Framing Effects.” American Political Science Review 98 (4): 671–86.

Enos

Ryan

Fowler

Anthony

. 2016. “Aggregate Effects of Large-Scale Campaigns on Voter Turnout.” Political Science Research and Methods May: 1–19.

Fieldhouse

Edward

Cutts

David

Widdop

Paul

John

Peter

. 2013. “Do Impersonal Mobilisation Methods Work? Evidence from a Nationwide Get-Out-the-Vote Experiment in England.” Electoral Studies 32 (1): 113–23.

Fowler

Erika

Ridout

Travis

. 2014. “Political Advertising in 2014: The Year of the Outside Group.” The Forum 12 (4): 663–84.

10.

Gerber

Alan

Green

Donald

Gimpel

James

Shaw

Daron

. 2010. “How Large and Long-Lasting Are the Persuasive Effects of Television Campaign Ads? Results from a Randomized Field Experiment.” American Political Science Review 105 (1): 135–50.

11.

Gerber

Alan S.

Green

Donald P.

2004. Get Out the Vote! Washington, D.C.: Brookings Press.

12.

Hersh

Eitan

Schaffner

Brian F.

2013. “Targeted Campaign Appeals and the Value of Ambiguity.” Journal of Politics 75 (2): 520–34.

13.

Hillygus

D. Sunshine

Jackman

Simon

. 2003. “Voter Decision Making in Election 2000: Campaign Effects, Partisan Activation, and the Clinton Legacy.” American Journal of Political Science 47 (4): 583–96.

14.

Huber

Gregory A.

Arceneaux

Kevin

. 2007. “Identifying the Persuasive Effects of Presidential Advertising.” American Journal of Political Science 51 (4): 957–77.

15.

Issenberg

Sasha

. 2012. The Victory Lab. New York: Random House.

16.

Iyengar

Shanto

Simon

Adam

. 2000. “New Perspectives and Evidence on Political Communication and Campaign Effects.” Annual Review of Psychology 51:149–69.

17.

Iyengar

Shanto

Westwood

Sean

. 2015. “Fear and Loathing across Party Lines: New Evidence on Group Polarization.” American Journal of Political Science 59 (3): 690–707.

18.

Krasno

Jonathan

Green

Donald

. 2008. “Do Televised Presidential Ads Increase Voter Turnout? Evidence from a Natural Experiment.” Journal of Politics 70 (1): 245–61.

19.

Pfau

Michael

Holbert

R. Lance

Szabo

Erin

Kaminski

Kelly

. 2002. “Issue-Advocacy versus Candidate Advertising: Effects on Candidate Preferences and Democratic Process.” Journal of Communication 52 (2): 301–15.

20.

Ridout

Travis

Franz

Michael

. 2011. The Persuasive Power of Campaign Advertising. Philadelphia: Temple University Press.

21.

Shaw

Daron R.

2006. The Race to 270. Chicago: University of Chicago Press.

22.

Zaller

John.

1992. The Nature and Origins of Mass Opinion. New York, NY: Cambridge University Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

1.48 MB

0.00 MB