The Persuasive Effects of Partisan Campaign Mailers

Abstract

A substantial literature has used field experiments to assess the mobilization effects of non-partisan mailers. However, little work has examined whether partisan mailers affect voters as intended. We report findings from two field experiments conducted in cooperation with partisan campaign strategists that allow us to assess the effects of negative and positive mailers. We find that mailers can affect voters—particularly their recognition of candidate names and their intent to turn out to vote. Notably, we find evidence that both negative and positive mailers stimulate intent to turn out.

Keywords

field experiment negative advertising persuasion mobilization

Campaigns spend huge amounts of money on political advertising in each election cycle. The primary intent of these communication efforts is typically to increase the likelihood that a voter will recognize and, ultimately, support the sponsoring candidate on Election Day. Much of this spending on communications—by some estimates as much as 15 percent in the 2012 election cycle—is devoted to direct mail advertising. Although an expansive and growing literature has used field experiments to examine the effects of non-partisan get-out-the-vote appeals, little work has examined the effects of partisan campaign mailers. One factor that has slowed the pace of published research in this area is that because grants and institutional research funds cannot be used to fund partisan efforts, examining the effects of these mailers in the field requires establishing a relationship with campaign strategists who are willing to fund the research and permit publication of findings.

In this article, we report findings from studies that leverage the unique advantages of field experiments. Each was done in collaboration with campaign strategists working on state legislative races during the 2012 general election campaign season. In each case, we are able to assess the effects of mailers that focus on touting the positive characteristics of the sponsoring candidate and the effects of negative messages that attack the opposing candidate. We are also able to compare the effects of these two types of communications. The findings make three contributions to our understanding of the effectiveness of direct mail campaign messaging.

First, our findings demonstrate that partisan campaign mailers affect voters. Most notably, in the first experiment, we find evidence that mailers substantially affect name recognition of the candidates—a factor that appears to play an important role in shaping vote choice (Goldenberg and Traugott 1980; Stokes and Miller 1962). Specifically, positive mailers that focus on the sponsoring candidate increase name recognition of that candidate while negative mailers that attack the opposing candidate increase rates of recognition of that candidate. Although we find suggestive evidence that the mailers affected voter evaluations of the candidates as intended, for the most part, these effects fall short of conventional levels of statistical significance.

Second, an ongoing scholarly debate pertains to whether negative campaign communications stimulate or depress participation. Our evidence indicates that both negative and positive mailers increase reported intent to turn out. Notably, the effects associated with each type of mailer are statistically indistinguishable from one another.

Finally, our evidence suggests that the timing of campaign communications is likely to play a crucial role in determining their effectiveness. The mailers substantially affected name recognition in the first field experiment (conducted several months before the election) but did not affect name recognition in the second study (conducted only weeks before Election Day). In addition, consistent with some existing work, we find that the effects of campaign mailers are short-lived. By the time the surveys associated with the second field experiment were conducted, the treatment effects identified in the first field experiment had evaporated. Similarly, the fact that the mailers stimulated intent to turn out in the second field experiment did not translate into an increase in likelihood of actually turning out to vote several weeks later.

The Effects of Campaign Communications

As we discuss below, little work has assessed the effects of partisan direct mail. However, a voluminous literature has examined the effects of other forms of campaign advertising—especially television advertising. Before continuing, it is important to note that existing work finds evidence that the effectiveness of campaign messages can depend substantially on the medium through which it is conveyed (Green and Gerber 2008; Hillygus and Shields 2009). We emphasize that the findings we present below cannot directly address ongoing debates regarding the effects of other forms of campaign communications. That said, given the dearth of existing work on the effects of partisan campaign mailers, we draw on evidence from these related literatures to clarify our contribution and provide theoretical grounding.

Much of the existing research on the effects of campaign advertising relies on observational analyses of survey data, often in concert with administrative records of turnout behavior or information about respondents’ political advertising environment (e.g., Krupnikov 2011). Other research uses lab or survey experiments (e.g., Ansolabehere et al. 1994; Brader 2005; Schultz and Pancer 1997). As authors of these studies acknowledge, each of these methodological approaches is open to critiques. Researchers who use observational data to assess campaign effects must contend with a variety of issues related to measuring which individuals have been exposed to which messages—a task that is complicated by systematic biases in how respondents describe their media consumption (Stevens 2008). Others are tied to the fact that observed campaign activities are endogenous to election outcomes: decisions about which races to devote resources to and what type of messaging to use are likely to be driven by assessments of which races are winnable, whether the candidate is an incumbent or challenger, and a variety of other strategic judgments.¹

Lab and survey experiments overcome many of the problems that complicate observational studies by randomly assigning exposure to the communication of interest and obviating concerns about the communications being strategically targeted. However, in spite of the efforts researchers make to mask the intents of their research designs, these studies are often criticized as lacking external validity because participants are captive recipients of treatments who are aware that they are being studied or because the treatments used in these experiments differ from the real-world communications they are intended to parallel (Arceneaux 2010; Kinder and Palfrey 1993; McDermott 2002).

Field experiments use randomly assigned treatments to achieve the internal validity benefits of lab experiments but achieve greater external validity by treating participants in a natural setting where they are not aware that they are being studied and that their response to the information they encounter is of interest to a researcher. Although some studies find evidence that survey and lab experiments yield substantively similar conclusions to findings from field experiments and other research designs (Ansolabehere, Iyengar, and Simon 1999; Falk and Heckman 2009; Gerber et al. 2013; Valentino, Traugott, and Hutchings 2002), others find reason to be cautious about claims regarding the external validity of these experiments (Barabas and Jerit 2010; Gneezy and List 2006; Jerit, Barabas, and Clifford 2013). Specifically, there is reason to be concerned that lab and survey experiments may overstate or otherwise distort the real-world effects of a given treatment.

A large literature has examined the effects of non-partisan get-out-the-vote messages on political participation using field experiments (Green and Gerber 2008). More recently, some scholars have conducted field experiments to assess the effects of other types of political communications—typically in cooperation with partisan political organizations or interest groups (Arceneaux and Kolodny 2009a, 2009b; Arceneaux and Nickerson 2010; Arceneaux 2007; Gerber 2004; Gerber et al. 2011; Loewen and Rubenson 2011; Panagopoulos and Green 2008). However, little work has leveraged the advantages of field experiments to assess the effects of partisan mailers. Indeed, we are only aware of one published field experiment that examines the effects of campaign mailers sent as part of a candidate’s campaign effort. That study finds that, in the context of a municipal mayoral election, negative mailers increase turnout by approximately 6 percent over the control group (Niven 2006).

Negative versus Positive Campaign Messaging

Much of the research on campaign advertising has focused on negative messaging. In contrast to positive advertising, which highlights the favorable characteristics and positions of the sponsoring candidate, negative advertising is designed to draw attention to an opponent’s unfavorable policy positions or personal characteristics. Theories regarding the persuasive advantages (or disadvantages) of negative advertising pit the expectation that negative advertising can successfully degrade voters’ evaluations of an opposing candidate against the possibility that voters dislike candidates who attack opponents—particularly if those attacks are perceived to be unnecessarily rude (Roese and Sande 2006).

Similarly, some posit that negative advertising demobilizes voters—perhaps by leaving individuals with the sense that there is no “good” candidate to vote for or degrading their assessments of the integrity or civility of the political process (e.g., Ansolabehere et al. 1994; Finkel and Geer 1998)—while others argue that negative advertising can increase participation by leading voters to see the election as more important or because voters find negative information to be particularly useful (Goldstein and Freedman 2002; Kahneman and Tversky 1979; Skowronski and Carlston 1989). However, to date, findings regarding the effects of negative advertising have been mixed. Ultimately, the authors of an extensive meta-analysis conclude, “There is no consistent evidence . . . that negative political campaigning ‘works’ in achieving the electoral results that attackers desire . . . Nor have we uncovered evidence that negative campaigning tends to demobilize the electorate . . . the overall mean effect is approximately zero” (Lau, Sigelman, and Rovner 2007, 1185–86).

Message Timing

Beyond assessing the relative effectiveness of negative and positive campaign mailers, the studies we report here allow us to examine whether the effects of these messages depend on their timing. Specifically, we fielded similar treatment regiments at two points in the general election cycle—one early in the campaign (mid-August) and another during the peak of the campaign season (mid-October). There are two reasons that this variation in timing may affect whether voters are affected by the mailers.

First, early in a campaign cycle a given political communication may face little competition for voter attention. In contrast, the marginal effect of an additional communication in the late stages of a highly salient election cycle may be dampened by increased competition from other contemporaneous messages from political opponents or candidates involved in other races. Only 40 percent of respondents in the control group in our first experiment reported having received political mail in the previous week. In contrast, the second experiment was conducted later in the campaign cycle when voters were being inundated with messages regarding high-profile ballot initiatives, presidential and congressional candidates, and an array of candidates for state-level office. In this experiment, 83 percent of respondents in the control group reported having received political mail in the previous week.

Second, the effectiveness of mailers may face the problem of diminishing returns from repeated attempts at persuading a fixed pool of voters. The state legislative campaigns that our messages were tied to were competitive, and by the time the second field experiment was fielded, 55 percent of the potential voters who had not been treated with a mailer recognized the Republican candidate and 63 percent recognized the Democratic candidate. Thus, a substantial segment of potential voters who viewed their state Senate race as worthy of consideration may have already come to recognize the candidates and, perhaps, made up their minds about which candidate they preferred by the time they received a treatment mailer. Taken together, these dynamics suggest that the effects of campaign communication efforts conducted late in a campaign will be weaker than those sent earlier in the campaign cycle. Thus, overall, we expect that—assuming we identify any treatment effects—the effects of the treatments in the second field experiment will tend to be weaker than those identified in the first.

It is important to note that scholars posit that the mobilizing (or demobilizing) effects of negative advertising are driven, in large part, by the way voters respond to the tone of political communications in general. Thus, it is possible that exposure to political communications may affect assessments of whether engaging in the political process is likely to be enjoyable, even if it does not affect attitudes about the candidates. Indeed, Krupnikov (2011) finds that negative advertising demobilizes voters, but only when voters encounter that negativity after they have already made up their mind regarding which candidate to support. Thus, even late in the election cycle, exposure to political advertisements may affect whether people are inclined to take the time to go to the polls on Election Day.

Assessing the Effects of Campaign Mailers

We conducted two essentially identical field experiments to compare the effects of negative and positive campaign mailers conceived of and designed by professional political strategists. We examine the effects of these mailers on candidate name recognition, candidate evaluations, and intent to turn out to vote. Given that previous findings regarding the effects of campaign communications have been mixed, we are agnostic in our expectations regarding the nature of these effects. Instead, we rely on random assignment to rule out potential confounds and use two-tailed tests of statistical significance.

As discussed above, we fielded one study relatively early in the 2012 general election cycle and one late in the campaign. The initial field experiment was conducted in two state Senate districts (SD 19 and SD 26) in a battleground state. The follow-up experiment included SDs 19 and 26, as well as SD 35. All three districts were thought likely to be very competitive; the Democratic incumbents in SDs 19 and 26 won by 2 percentage points or less in the previous (2008) election, and there was no incumbent running in SD 35. Prior to the election, political observers were referring to these districts as “swing districts,” “toss up seats,” or “battleground seats” (Hoover 2012a, 2012b), with the newly drawn SD 35 attracting an extraordinary amount of expenditures by outside political action committees (Crummy 2012). The margins of victory for the winning candidates (Democratic incumbents in SDs 19 and 26, and the Republican open-seat candidate in SD 35) ranged from 0.3 to 7.0 percent.

The campaign professionals we worked with were interested in examining the effects of mailers on a particular population—independent likely voters (unaffiliated voters—those who were not formally affiliated with a political party—and who had turned out to vote in either the 2008 or 2010 general election).

In each study, treatment assignment was conducted at the household level. In cases where more than one eligible registered voter (i.e., more than one independent likely voter) lived in a given household, one individual was randomly selected from the voter file for inclusion in the study, and any other eligible voter within that household was dropped from the dataset.² Our final sample for each study consists of individuals who fall into one of three strata: (1) individuals who our records indicate both do not share a phone number with any other registered voter (of any type) and do not live with any other registered voters, (2) individuals who do not share a phone number with any other voters but do share a physical address with other voters, and (3) likely independent voters who share both a phone number and physical address with one other voter. For the first experiment, within each stratum, we randomly assigned individuals in SDs 19 and 26—with equal probability—to one of three conditions: a control condition, a negative mailer condition, or a positive mailer condition.³

Initial Field Experiment

Two identical mailers were sent (two days apart) to targeted individuals in mid-August of 2012. Although these races would ultimately be hotly contested, the organization we worked with reported that none of the four campaigns in question had begun sending out direct mail when we conducted the first experiment. The negative mailers attacked the Democratic candidates’ policy positions and the purported implications of those positions. Specifically, the mailer in each district accused the Democratic candidate of eagerly supporting raising taxes: “Raising taxes. Killing jobs.” was presented in large, bold font at the top of the front of the mailer. The back of the mailer described the candidate with the phrase, “Likes high taxes. How much? $4 billion!” In contrast, the positive mailer focused on the Republican candidate’s background and policy goals. As with the negative mailers, the positive mailers associated with each of the two candidates were almost identical. Each highlighted the candidate’s background (e.g., “Husband, father, veteran”) and promised “Jobs for [STATE], Opportunity for All, and Limited Government.”

Three days after sending out the second mailer, we fielded interactive voice response (IVR) surveys, attempting to contact all individuals in the target population. The IVR surveys were conducted over several days and yielded a final response rate of 9.2 percent.⁴ The survey consisted of five questions. The first two asked respondents to rate each of the candidates (generally favorable opinion, generally unfavorable opinion, never heard of candidate, heard of but unsure; see the appendix for full question wording). These items provide a way to measure candidate name recognition as well as respondents’ ratings of each candidate and—when compared—which candidate (if any) the respondent preferred.

The third question asked whether the respondent recalled receiving any campaign mail in the previous week. The fourth question asked respondents whether they were registered to vote in Colorado. The final question asked respondents whether they intended to vote in the 2012 general election. Although 1,939 individuals provided responses to the first item in the survey, 289 respondents did not complete the entire survey. For simplicity and clarity, we restrict our sample to the cases where the individual provided responses to all five questions in the analysis that follows. We also exclude the 110 of the remaining respondents who indicated that they were not registered to vote in Colorado as this response suggests that the person who completed the survey was not the targeted voter.⁵ These restrictions do not materially affect the findings we report. Summary statistics for this field experiment and the field experiment described in the next section are presented in Table S2 of the Supplementary Analysis Document (see supplementary material at http://prq.sagepub.com/supplemental/).

In Table 1, we estimate the effects of the mailer treatments on a several outcomes of interest. We include only indicators for treatment assignments in these models. Including pre-treatment control variables does not affect the substance of the findings we report here (see Table S3 in the Supplementary Analysis Document). In column 1, we begin by examining responses to the question that asked respondents whether they had received any campaign mail in the previous week. The relative campaign calm during this period is reflected in the fact that only 39.4 percent of individuals in the control groups (see coefficient on the constant) reported having received any campaign mail at all in the previous week.⁶ In contrast, a significantly larger proportion of respondents in the positive and negative mailer conditions, 57.2 and 60.2 percent, respectively, reported having received mail in the previous week (p < .01 for comparisons with control condition; the difference in the proportion of respondents reporting having received mail across the two treatment conditions was not statistically significant, p = .266).

Table 1.

Estimated Treatment Effects (Initial Field Experiment).

	(1)	(2)	(3)	(4)	(5)	(6)	(7)
	Yes, received mail	Recognize Republican	Recognize Democrat	Evaluation of Republican	Evaluation of Democrat	Difference in evaluations (Republican–Democrat)	Intent to vote
	(1 = yes)	(1 = yes, 0 = no, unsure = yes)		(1 = favorable, −1 = unfavorable, 0 = Don’t Know or never heard of)			(1 = definitely not, 4 = definitely will)
Positive Mailer Treatment	0.178*** [0.031]	0.088*** [0.026]	0.003 [0.031]	0.049** [0.020]	−0.007 [0.034]	0.056 [0.042]	0.044 [0.035]
Negative Mailer Treatment	0.212*** [0.030]	0.003 [0.024]	0.059* [0.031]	0.014 [0.018]	−0.033 [0.035]	0.047 [0.041]	0.080** [0.034]
Constant	0.394*** [0.022]	0.175*** [0.017]	0.462*** [0.022]	−0.012 [0.013]	0.074*** [0.024]	−0.085*** [0.028]	3.786*** [0.026]
Observations	1,540	1,540	1,540	1,540	1,540	1,540	1,540
R ²	.035	.010	.003	.004	.001	.001	.004
Positive Mailer p value	.000	.001	.915	.015	.842	.186	.211
Negative Mailer p value	.000	.904	.058	.457	.347	.249	.017
p value of difference between treatments	.266	.001	.074	.085	.452	.835	.255
p value of joint significance of treatments	.000	.001	.103	.050	.617	.343	.058

Cell entries are unstandardized OLS coefficients. Robust standard errors in brackets. OLS = ordinary least squares.

p < .10. **p < .05. ***p < .01.

In this type of state-level race, campaign strategists are often interested in two questions: whether campaign efforts increase candidate name recognition and whether they improve the relative standing of their candidate in the eyes of targeted voters. Accordingly, we analyze the effects of the mailer treatments on recognition of the candidates’ names. As noted above, respondents could rate each candidate favorably, unfavorably, say they had never heard of the candidate, or say that they had heard of the candidate but were unsure about how they feel about the candidate. In columns 2 and 3, we predict candidate name recognition. Respondents who rated the candidate favorably or unfavorably or said they had heard of the candidate but were unsure about their feelings about the candidate are scored 1; those who had not heard of the candidate are scored 0.⁷

The model in column 2 assesses the effects of the treatments on recognition of the Republican candidate. The constant indicates that only 17.5 percent of respondents in the control condition recognized the Republican candidate’s name. The coefficient on the Positive Mailer Treatment indicates that this mailer—which focused exclusively on the positive attributes of the Republican candidate—increased the candidate’s name recognition by 8.8 percentage points (p < .01). This amounts to a substantial proportional increase of approximately 50 percent. In contrast, the Negative Mailer Treatment—which focused exclusively on the negative characteristics of the Democratic incumbent and did not mention the Republican candidate by name—did not significantly affect the Republican candidates’ name recognition.

In column 3, we see that among those in the control group, 46.2 percent recognized the incumbent Democrat’s name. Here the treatment effects are essentially flipped. The Positive Mailer Treatment (which, again, did not mention the Democratic candidate’s name) did not significantly affect the proportion of voters who recognized the Democratic candidate. In contrast, the Negative Mailer Treatment increased the Democrat’s name recognition by an estimated 5.9 percentage points (p < .10).

In columns 4 to 6, we assess how the mailers affected evaluations of the two candidates. The outcome measures in columns 4 and 5 are scored so that those rating the candidate favorably are scored 1, those rating the candidate unfavorably are scored −1, and those who either indicated that they did not recognize the candidate or that they were unsure how they felt about the candidate are scored 0. The results in column 4 indicate that the Positive Mailer Treatment had a small but statistically significant effect on the favorability rating of the Republican candidate. Specifically, it increased this favorability rating by .049 units (approximately one-fifth of a standard deviation; p < .05). In contrast, the Negative Mailer Treatment did not significantly affect ratings of the Republican candidate. The results in column 5 suggest that the Negative Mailer Treatment did not significantly hurt the favorability standing of the incumbent Democratic candidate. The point estimates for both treatment effects are negative, but they fall well short of conventional levels of statistical significance both independently and jointly.

The model in column 6 estimates the effects of the treatments on the standing of the Republican candidate relative to the standing of the Democratic candidate measured by subtracting the Democrat’s favorability rating from the rating of the Republican. This measure can be interpreted as a proxy for vote preference. The evidence suggests that the positive mailer improved the Republican candidate’s relative standing slightly (by approximately .056 units—about 1/10 of a standard deviation). However, the coefficient on this treatment indicator falls short of conventional levels of statistical significance (p = .186). The effect of the Negative Mailer Treatment is also positive but falls short of conventional levels of statistical significance (p = .249). The estimated effects associated with the two treatments are statistically indistinguishable from one another (p = .835) and the two treatment indicators are not jointly significant (p = .343).

Finally, in column 7, we estimate the effects of each treatment on responses to the intent to turn out question. The point estimates on each treatment indicator are positive, and the coefficient on the Negative Mailer Treatment reaches conventional levels of statistical significance (p < .05). The coefficient on the Positive Mailer Treatment falls short of conventional levels of statistical significance (p = .211) but is not statistically distinguishable from the coefficient on the Negative Mailer Treatment (p = .255).

Follow-Up Field Experiment

The follow-up field experiment was designed to assess whether the findings from the first field experiment replicated later in the campaign cycle. The structure of the experiment—including how the sample was identified and which voter was sampled in households with more than one targetable voter—mirrored the first experiment. The layouts of the positive and negative mailers used in this study were slightly different from those used in the first field experiment, but the messaging was virtually identical. As with the first experiment, the negative mailers highlighted the Democratic candidate’s support for raising taxes and said the Democrat’s “bad voting record has hurt [STATE]’s ability to build a strong economy.” The positive mailers, again, emphasized positive aspects of the Republican candidate’s background and commitment to creating jobs through fiscally responsible policies.

This study also extended the design used in the first field experiment in two ways. First, we included likely independent voters from a third state Senate district (SD 35) in the sample. Second, in addition to the positive and negative mailer treatment conditions, we included a third treatment condition that we label the “contrast mailer” condition. This mailer presented information from the negative treatment mailer regarding the Democratic candidate on one side and information from the positive treatment mailer about the Republican candidate on the opposite side. We focus our attention on the two treatments—the positive and negative mailer—that were comparable to those used in the first experiment.

As with the initial experiment, treatment assignment was done within each of the three strata described above with targeted individuals having an equal probability of being assigned to each of the four conditions (three treatment conditions or control). For voters in SDs 19 and 26, this treatment assignment was done independently of the assignment in the initial experiment. Mailers were sent out in the second week of October, 2012. Due to resource constraints, only one mailer was sent to each targeted individual. We attempted IVR surveys identical to those used in the first design beginning three days after the mailers were sent out. The surveys were conducted over several days and yielded a final response rate of 7.0 percent.⁸ As with the analysis presented in Table 1, we restrict the sample to individuals who provided usable responses to each of the five survey questions and exclude the 5.5 percent of respondents who indicated that they were not registered to vote.⁹

In Table 2, we regress each of the outcomes used in Table 1 on indicators for each treatment condition from the follow-up experiment, indicators for treatment assignment from the first experiment, and—because individuals in SD 35 were not included in the first experiment—an indicator for respondents from this district.¹⁰ The substantially higher intensity of campaign activity during this period is reflected in the fact that 82.9 percent of respondents (compared with 39.1% in the first study) who were assigned to the control condition reported having received political mail in the previous week.¹¹ Communications during this period appear to have been so intense that being treated with an additional mailer did not significantly affect reported receipt of political mail (p value of test of joint significance of treatment indicators = .901). In addition, we find little evidence of the treatments in this experiment affecting candidate name recognition or evaluations of the candidate—p values associated with tests of the joint significance of the three treatment indicators in columns 2 to 6 range from .411 to .963.¹²

Table 2.

Estimated Treatment Effects (Second Field Experiment).

	(1)	(2)	(3)	(4)	(5)	(6)	(7)
	Yes, received mail	Recognize Republican	Recognize Democrat	Evaluation of Republican	Evaluation of Democrat	Difference in evaluations (Republican–Democrat)	Intent to vote
	(1 = yes)	(1 = yes, 0 = no, unsure = yes)		(1 = favorable, −1 = unfavorable, 0 = Don’t Know or never heard of)			(1 = definitely not, 4 = definitely will)
Positive Mailer Treatment	0.008 [0.025]	−0.013 [0.035]	−0.008 [0.033]	0.018 [0.039]	0.002 [0.048]	0.015 [0.075]	0.096** [0.038]
Negative Mailer Treatment	0.013 [0.026]	−0.002 [0.036]	0.028 [0.034]	0.002 [0.040]	0.064 [0.052]	−0.061 [0.077]	0.091** [0.040]
Contrast Mailer Treatment	−0.005 [0.026]	−0.016 [0.036]	−0.029 [0.034]	−0.036 [0.039]	0.002 [0.049]	−0.038 [0.075]	0.039 [0.041]
Positive Mailer Treatment (1st round)	0.008 [0.026]	−0.009 [0.035]	−0.006 [0.034]	−0.013 [0.038]	−0.024 [0.046]	0.011 [0.070]	0.021 [0.037]
Negative Mailer Treatment (1st round)	0.029 [0.025]	−0.009 [0.035]	0.007 [0.034]	0.032 [0.038]	−0.039 [0.048]	0.070 [0.072]	0.025 [0.037]
District 35 (1 = yes)	0.033 [0.026]	0.049 [0.036]	0.155*** [0.033]	0.049 [0.042]	−0.012 [0.052]	0.061 [0.080]	0.003 [0.040]
Constant	0.829*** [0.024]	0.550*** [0.032]	0.626*** [0.031]	0.056 [0.036]	−0.004 [0.045]	0.060 [0.069]	3.783*** [0.038]
Observations	1,552	1,552	1,552	1,552	1,552	1,552	1,552
R ²	.002	.002	.020	.003	.002	.002	.006
p value of joint significance of second-round treatments	.901	.963	.411	.582	.545	.731	.040
p value of joint significance of first-round treatments	.487	.958	.928	.489	.712	.585	.772

Cell entries are unstandardized OLS coefficients. Robust standard errors in brackets. OLS = ordinary least squares.

p < .10. **p < .05. ***p < .01.

We do find evidence that the treatments increased intent to turn out. Specifically, in column 7, the coefficients on the negative and positive mailer treatments each reach conventional levels of statistical significance. The Positive Mailer Treatment is associated with a .096 unit increase in Intent to Vote, and the Negative Mailer Treatment is associated with a .091 unit increase. The coefficient on the Contrast Mailer Treatment is positive but falls short of conventional levels of statistical significance (p = .339).

Persistent Effects?

Finally, we assess the durability of the treatment effects we identified in these studies. First, we examine whether the treatment effects identified in the first study were still observable when the second study was conducted. Consistent with findings from recent studies that suggests that campaign effects dissipate rapidly (e.g., Gerber et al. 2011; Hill et al. 2013), we find no evidence that the effects identified in the first experiment were sustained until the time of the second experiment.¹³ The coefficients on the treatments from the first experiment that significantly affected outcomes in that experiment are, on average, one-seventh of the size. The p values associated with tests of the joint significance of the first-round treatment indicators in each of the seven models fall well short of conventional levels of statistical significance, ranging from .487 to .958.

In addition, post-election (February 2013), we acquired updated voter files to assess whether the effects of the treatment mailers on reported intent to turn out in the second study were reflected in actual turnout behavior. Consistent with the null effects of the first-round treatments in the second-round survey, analysis of the effects of the first- and second-round treatments on validated turnout suggests that the mobilization effects associated with receiving campaign mail dissipated rapidly and did not affect actual turnout (see Table S8 in the Supplementary Analysis Document). We note that this null effect could also indicate that variation in respondents’ reported intent to turn out does not meaningfully correspond to variation in actual participation. Although we cannot definitively rule out this explanation, over 90 percent of respondents who indicated that they would definitely vote did, in fact, turn out. In contrast, only 63 percent of those who said that they would either definitely or probably not vote actually turned out.

Discussion

The findings we present here suggest that both positive and negative campaign mailers can affect how voters view the political world. Importantly, apart from their effects on candidate name recognition, our evidence suggests that the effects of negative and positive mailers are statistically indistinguishable (for similar findings, see Arceneaux and Nickerson 2010). Our findings also suggest that the timing of these communications can have at least two important consequences for their effectiveness. First, the results from the first field experiment suggest that, in the early days of the 2012 general election cycle, the mailers increased the probability that likely independent voters would recognize the candidate the mailer focused on. In that experiment, we also found suggestive evidence that the mailers improved the candidates’ electoral prospects by improving their standing with voters. In contrast, in the second field experiment, we find little evidence that the mailers affected recipients’ assessments or recognition of the candidates. Second, our evidence suggests that the effect of these mailers dissipates rapidly. We found no evidence that the effects identified in the first treatment persisted until we fielded the second experiment or that the effects of the treatments on intent to turn out in the second field experiment persisted until Election Day.¹⁴

Our evidence also supports the claim that negative advertising—at least negative direct mail advertising—mobilizes voters rather than demobilizing them. This is consistent with the one previous study we are aware of that has examined the effects of negative direct mail on turnout (Niven 2006). Positive mailers also appear to stimulate intent to turn out. Notably, these effects were identified both early and late in the campaign cycle. Thus, our findings are consistent with the claim that although communications sent late in a campaign may be unlikely to alter potential voters’ views about candidates, they can affect broader assessments of the political environment and, thereby, their eagerness to participate.¹⁵

It is important to note that, as with all research, our evidence has limitations. First, although the mailers used in the second field experiment contained messages that were quite similar to those used in the first experiment, they were not precisely identical. Second, due to resource constraints, treated individuals in the first field experiment received two mailers, while those in the second field experiment received one. Given the similarities in the effects of the treatments on intent to turn out across the studies, we believe that the timing of the study, rather than quantity of the treatments, is the most likely explanation for the differences in findings across the two field experiments. However, some previous studies find that treatment effects associated with negative mailers are amplified by multiple mailings (Niven 2006). In the future, researchers should pursue opportunities to repeat more perfectly identical field experiments within a campaign cycle.

It is also important to note that our analysis relies on responses from IVR surveys that yielded response rates that, although typical for this type of survey, were nonetheless low. We did not find any statistically significant differences between the characteristics of survey respondents and non-respondents. However, we are unable to rule out the possibility that respondents were distinctive on unmeasured characteristics. Similarly, we cannot confidently rule out the existence of complex interactions between treatment assignment and non-response.

Other caveats to our findings stem from our successes in achieving consistency across these studies. We focused exclusively on estimating the effects of campaign mailers sent on behalf of candidates from one political party. In addition, our studies were fielded in the context of specific state legislative races during a presidential election year. However, the effects of campaign messaging may well vary across campaign contexts and depend on factors such as the characteristics of the candidates (e.g., gender, party affiliation, race), whether the campaign is associated with a midterm, presidential, or “off-year” election, and a range of other factors. Similarly, we focused strictly on a target population of unaffiliated registered voters. Many unaffiliated voters—including those who claim to be politically independent when asked—appear to behave much like partisans (Keith et al. 1992). However, just as we cannot definitively generalize the treatment effects we observed among those who responded to our surveys to those who refused, we cannot be confident that our findings would be similar among self-identified partisans.

These limitations aside, our findings constitute an important contribution to our understanding of the effects of campaign mailers. The field experiments we report here are the first that we know of to examine the persuasive effects of both negative and positive campaign mailers by leveraging the advantages of random assignment in a natural setting. This allows us to make clear inference regarding the effects of the treatment mailers. The results from two randomized field experiments demonstrate that partisan campaign mailers can affect candidate name recognition, evaluations of candidates, and intent to turn out. Although the effects we identified appear to be short-lived, the findings suggest that partisan mailers may be a valuable component of a political campaign.

Footnotes

Appendix

Acknowledgements

We are grateful to Kevin Arceneaux, Gregory Huber, and several anonymous reviewers for their feedback on previous versions of this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Ansolabehere

Stephen

Iyengar

Shanto

Simon

Adam

. 1999. “Replicating Experiments Using Aggregate and Survey Data: The Case of Negative Advertising and Turnout.” American Political Science Review 93:901–909.

Ansolabehere

Stephen

Iyengar

Shanto

Simon

Adam

Valentino

Nicholas

. 1994. “Does Attack Advertising Demobilize the Electorate?” American Political Science Review 88:829–38.

Arceneaux

Kevin

. 2007. “I’m Asking for Your Support: The Effects of Personally Delivered Campaign Messages on Voting Decisions and Opinion Formation.” Quarterly Journal of Political Science 2:43–65.

Arceneaux

Kevin

. 2010. “The Benefits of Experimental Methods for the Study of Campaign Effects.” Political Communication 27:199–215.

Arceneaux

Kevin

Kolodny

Robin

. 2009a. “Educating the Least Informed: Group Endorsements in a Grassroots Campaign.” American Journal of Political Science 53:755–70.

Arceneaux

Kevin

Kolodny

Robin

. 2009b. “The Effect of Grassroots Campaigning on Issue Preferences and Issue Salience.” Journal of Elections, Public Opinion and Parties 19:235–49.

Arceneaux

Kevin

Nickerson

David

. 2010. “Comparing Negative and Positive Campaign Messages: Evidence from Two Field Experiments.” American Politics Research 38:54–83.

Barabas

Jason

Jerit

Jennifer

. 2010. “Are Survey Experiments Externally Valid?” American Political Science Review 104:226–42.

Brader

Ted

. 2005. “Striking a Responsive Chord: How Political Ads Motivate and Persuade Voters by Appealing to Emotions.” American Journal of Political Science 49:388–405.

10.

Crummy

Karen

. 2012. “Dems Ramp Up PAC Attack.” Denver Post, October 21, 1A.

11.

Falk

Armin

Heckman

James J.

2009. “Lab Experiments Are a Major Source of Knowledge in the Social Sciences.” Science 326:535–38.

12.

Finkel

Steven E.

Geer

John G.

1998. “A Spot Check: Casting Doubt on the Demobilizing Effect of Attack Advertising.” American Journal of Political Science 42:573–95.

13.

Gerber

Alan S.

2004. “Does Campaign Spending Work? Field Experiments Provide Evidence and Suggest New Theory.” American Behavioral Scientist 47:541–74.

14.

Gerber

Alan S.

Gimpel

James G.

Green

Donald P.

Shaw

Daron R.

2011. “How Large and Long-Lasting Are the Persuasive Effects of Televised Campaign Ads? Results from a Randomized Field Experiment.” American Political Science Review 105:135–50.

15.

Gerber

Alan S.

Huber

Gregory A.

Doherty

David

Dowling

Conor M.

Panagopoulos

Costas

. 2013. “Big Five Personality Traits and Responses to Persuasive Appeals: Results from Voter Turnout Experiments.” Political Behavior 35:687–728.

16.

Gneezy

Uri

List

John A.

2006. “Putting Behavioral Economics to Work: Testing for Gift Exchange in Labor Markets Using Field Experiments.” Econometrica 74:1365–84.

17.

Goldenberg

Edie N.

Traugott

Michael W.

1980. “Congressional Campaign Effects on Candidate Recognition and Evaluation.” Political Behavior 2:61–90.

18.

Goldstein

Ken

Freedman

Paul

. 2002. “Campaign Advertising and Voter Turnout: New Evidence for a Stimulation Effect.” The Journal of Politics 64:721–40.

19.

Green

Donald P.

Gerber

Alan S.

2008. Get Out the Vote: How to Increase Voter Turnout. Washington, DC: The Brookings Institution Press.

20.

Hill

Seth J.

James

Lynn

Vavreck

John

Zaller

. 2013. “How Quickly We Forget: The Duration Of Persuasion Effects From Mass Communication.” Political Communication 30 : 521-47.

21.

Hillygus

D. Sunshine

Shields

Todd G.

2009. The Persuadable Voter: Wedge Issues in Presidential Campaigns. Princeton: Princeton University Press.

22.

Hoover

Tim

. 2012a. “Hudak Fights to Keep Swing-District Seat.” Denver Post, August 2, 5A.

23.

Hoover

Tim

. 2012b. “In Redistricted Battleground, GOP’s Kerber Targets Newell.” Denver Post, July 7, 6A.

24.

Jerit

Jennifer

Barabas

Jason

Clifford

Scott

. 2013. “Comparing Contemporaneous Laboratory and Field Experiments on Media Effects.” Public Opinion Quarterly 77:256–82.

25.

Kahneman

DanieI

Tversky

Amos

. 1979. “Prospect Theory: An Analysis of Decision Under Risk.” Econometrica 47:263–91.

26.

Keith

Bruce E.

Magleby

David B.

Nelson

Candice J.

Orr

Elizabeth

Westlye

Mark C.

Wolfinger

Raymond E.

1992. The Myth of the Independent Voter. Berkeley: University of California Press.

27.

Kinder

Donald R.

Palfrey

Thomas R.

1993. Experimental Foundations of Political Science. Ann Arbor: University of Michigan Press.

28.

Krasno

Jonathan S.

Green

Donald P.

2008. " Do Televised Presidential Ads Increase Voter Turnout? Evidence from a Natural Experiment." Journal of Politics 70: 245-261.

29.

Krupnikov

Yanna

. 2011. “When Does Negativity Demobilize? Tracing the Conditional Effect of Negative Campaigning on Voter Turnout.” American Journal of Political Science 55:797–813.

30.

Lau

Richard R.

Pomper

Gerald M.

2002. "Effectiveness of Negative Campaigning in U.S. Senate Elections." American Journal of Political Science 46: 47-66.

31.

Lau

Richard R.

Sigelman

Lee

Rovner

Ivy B.

2007. “The Effects of Negative Political Campaigns: A Meta-analytic Reassessment.” The Journal of Politics 69:1176–209.

32.

Loewen

P. John

Rubenson

Daniel

. 2011. “For Want of a Nail: Negative Persuasion in a Party Leadership Race.” Party Politics 17:45–65.

33.

McDermott

Rose

. 2002. “Experimental Methodology in Political Science.” Political Analysis 10:325–42.

34.

Niven

David

. 2006. “A Field Experiment on the Effects of Negative Campaign Mail on Voter Turnout in a Municipal Election.” Political Research Quarterly 59:203–10.

35.

Panagopoulos

Costas

Green

Donald P.

2008. “Field Experiments Testing the Impact of Radio Advertisements on Electoral Competition.” American Journal of Political Science 52:156–68.

36.

Roese

Neal J.

Sande

Gerald N.

2006. “Backlash Effects in Attack Politics.” Journal of Applied Social Psychology 23:632–53.

37.

Schultz

Cindy

Pancer

S. Mark

. 1997. “Character Attacks and Their Effects on Perceptions of Male and Female Political Candidates.” Political Psychology 18:93–102.

38.

Skowronski

John J.

Carlston

Donal E.

1989. “Negativity and Extremity Biases in Impression Formation: A Review of Explanations.” Psychological Bulletin 105:131–42.

39.

Stevens

Daniel

. 2008. “Measuring Exposure to Political Advertising in Surveys.” Political Behavior 30:47–72.

40.

Stokes

Donald E.

Miller

Warren E.

1962. “Party Government and the Saliency of Congress.” Public Opinion Quarterly 26:531–46.

41.

Valentino

Nicholas A.

Traugott

Michael W.

Hutchings

Vincent L.

2002. “Group Cues and Ideological Constraint: A Replication of Political Advertising Effects Studies in the Lab and in the Field.” Political Communication 19:29–48.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.28 MB