Abstract
The impact of presidential debates on candidate evaluations remains an open topic. Research has long sought to identify the factors that matter most in citizens’ responses to debate content, including what candidates say, how they say it, and the manner in which they appear. This study uses detailed codings of the first and third 2012 presidential debates to evaluate the impact of candidates’ verbal and nonverbal behaviors on viewers’ “second screen” response—their use of computers, tablets, and mobile devices to express their reactions to the viewing experience. To examine the relationship between candidates’ on-screen behaviors and the social media response, we conduct generalized least squares regression (Prais–Winstein estimation) relating two data sources: (a) a shot-by-shot content analysis coded for rhetorical/functional, tonal, and visual elements of both candidates’ behavior during the debates, and (b) corresponding real-time measures of the volume and valence of online expression about the candidates on Twitter. We find that the nonverbal communication behaviors of candidates—their facial expressions, physical gestures, and blink rate—are consistent, robust, and significant predictors of the volume and valence of public expression during debates, rivaling the power of memes generated by candidates and contributing more than rhetorical strategies and speech tone.
Keywords
Since American presidential debates were first televised more than half a century ago, scholars, pundits, and citizens alike have been fascinated with their ability to crystallize political conflict and public discourse. Though there are growing questions about the impact debates have on elections themselves (Benoit, 2013; Druckman, 2003), interest in American televised debates has remained constant. There are few events in the American electoral cycle that focus national attention and public expression on the political process to the extent that televised debates do. In 2012, for example, the first presidential debate between Barack Obama and Mitt Romney drew an audience of more than 70 million Americans—the most in 32 years—a total viewership matched only by the Super Bowl (Carr, 2012). 1 Likewise, the third presidential debate in 2012 drew a total audience of nearly 60 million viewers. And during these two debates’ 180-minute total duration, at least 16.5 million debate-related comments were posted to Twitter, making them among the most tweeted-about events in U.S. politics up to that time (Sharp, 2012a, 2012b). This “dual screening” of political events—moving between live broadcast viewing and social media interaction—is gaining academic attention (Vaccari, Chadwick, & O’Loughlin, 2015).
As all of this indicates, debates constitute moments of significant national attention and public expression (through social media) among the electorate. Although debates focus national attention in a powerful way, there is little understanding of how the public responds to these crystallizing moments. That is, there is disagreement among scholars about whether citizens’ responses to debates are driven by what the candidates say, how they say it, or how they appear (Benoit, 2013; Cho, Shah, Nah, & Brossard, 2009; Nagel, Maurer, & Reinemann, 2012; Zhu, Milavsky, & Biswas, 1994). Despite scholarly interest in both televised debates and social media, only a handful of studies have examined how the volume and valence of public expression via social media may be shaped by the content of such focal moments (Shah, Hanna, Bucy, Wells, & Quevedo, 2015; Wells, Van Thomme, Maurer, Hanna, Pevehouse, Shah & Bucy, 2016).
Since at least the first televised debates between John F. Kennedy and Richard Nixon, students of presidential campaigns have discussed the role of visual content in political competition, arguing that viewers are heavily “influenced by appearances, gestures, or other nonverbal behaviors” (Kraus, 1996, p. 78). Druckman (2003), for example, found evidence that among subjects with no knowledge of the 1960 debate, the mode in which they first encountered the event—watching the television feed as opposed to only listening to it—affected debate evaluations, increased individual reliance on personality perceptions, and enhanced learning among those with the least preexisting knowledge. Thus, for many Americans, the influence of candidates’ self-presentation goes beyond the rhetorical features of persuasive discourse available through their words alone (e.g., attacking, contrasting, or responding to an opponent; Benoit & Harthcock, 1999). Our research builds on this intuition and explores the wide range of verbal, tonal, and visual elements of presidential debates experienced by the audience.
We explore the power of the visuals in presidential debates by combining insights from biobehavioral research with computational methods tracking the public’s real-time reaction to the 2012 U.S. presidential debates. Existing studies suggest that a debate audience may respond at least as much to a candidate’s physical actions, facial expressions, and autonomic behaviors as they do to his or her words (Grabe & Bucy, 2009; Patterson, Churchill, Burger, & Powell, 1992). In this article, we explore this possibility directly by examining the influence of rhetorical/functional elements alongside voice tone, facial expressions, physical gestures, and blink rates of both candidates during the first and third U.S. presidential debates in 2012. Unlike many other examinations of candidate behavior, we study these elements in real time across debate settings. That is, we code for their existence at the level of the individual segment (ranging from 10 to 30 seconds) during two debates that addressed distinct topics hosted by different moderators.
We then relate differences in segment-level rhetorical/functional, tonal, and visual features to the volume—that is, the actual number of mentions of each candidate on social media or a proxy of the attention a candidate is receiving—and valence—that is, the sentiment posts mentioning each of the candidates or a proxy of the opinion embedded in that expression—of Twitter posts about Obama and Romney on a synchronous and lagged basis. Our social media data comes from an original harvest of 10% of all Twitter content posted during each debate (the Twitter “garden hose”). We then use machine-learning classifiers to score candidate-specific sentiment in all tweets that mention either Barack Obama or Mitt Romney. By comparing the words, tone, and visuals of the candidates to the online expression of debate viewers, we identify the candidate-specific features that most drive responsiveness via the “second screen”—that is, computers, tablets, and mobile devices that enable real-time reactions to the viewing experience (Tsekleves, Cruickshank, Hill, Kondo, & Whitham, 2007).
Literature Review
Our research builds on an extensive body of work examining visual portrayals and candidate nonverbal behavior (e.g., Hellweg & Phillips, 1981; Tiemens, Hellweg, Kipper, & Phillips, 1985) that runs parallel to studies on rhetorical strategies employed by candidates during modern presidential debates (see Benoit, 2013; Jamieson & Birdsell, 1990). As such, this research draws on studies of visual dominance in sensory processing (Colavita, 1974; McGurk & MacDonald, 1976) and considers historical changes in debate presentation and performance that increasingly highlight contentiousness and incivility in such settings.
Debates and the “Second Screen”
In many ways, the ebb and flow of debate formats has followed the evolution of television as a visual medium. The earliest televised debates were shown in black and white from a limited number of candidate perspectives and fixed-shot angles (Kraus, 1996). As the number of television sets and available channels grew rapidly, so too did the technical skill involved in the presentation of the debates. By the early 2000s, a dozen channels were broadcasting presidential debates, including CNN and Fox News, and used a broader variety of camera angles, moving shots, and split-screen displays, providing more visual information, and allowing viewers to continuously monitor the reactions of one candidate while the other was speaking (Cho et al., 2009). These increasingly sophisticated formats may have strengthened debates’ well noted—though not always realized—potential to provide a range of information to the electorate (e.g., Jamieson & Adasiewicz, 2000; Zhu et al., 1994). Benoit, McKinney, and Holbert (2001) argue that while debates enhance the transfer of information, they may also change the nature of the criteria citizens use when deciding whether to vote and whom to support.
The evolution of televised media has also changed the nature of political discourse surrounding the debates themselves. Millions of debate viewers in 2012 discussed the event in real time using networked social media (Sharp, 2012a, 2012b). Emerging research suggests that the distracted nature of viewing induced by “second screen” activity may alter viewers’ perceptions of candidate performance (Stroud, Stephens, & Pye, 2011). Furthermore, given the tendency for online networks to foster homophily and consensus (Conover et al., 2011), online exchanges during debates may support the finding that debate viewing “largely reinforces existing predispositions rather than substantially changing previously held images of candidates, issue orientations, or voting intentions” (Sigelman & Sigelman, 1984, p. 624; also see Abramowitz, 1978).
Similarly, social media use around the debate may reinforce Wald and Lupfer’s (1978) observation that debates increase cynicism and reduce trust. Social media posts are often brief (a single tweet, e.g., can include no more than 140 characters) and may encourage users to focus on broad, generalized assessments of candidate performance, rather than more detailed evaluation of specific policies. This may lend itself to singling out one candidate as the winner of a particular exchange, similar to professional commentators’ response to debates (e.g., Fridkin, Kenney, Gershon, & Woodall, 2008; Jamieson, 1992; Tsfati, 2003). Doing so, however, focuses on the strategic nature of candidate behavior and may thereby increase viewer cynicism (Cappella & Jamieson, 1997; Stroud et al., 2011).
Yet as Shah et al. (2015, p. 228) note, “these and most other analyses fail to consider the tonal and visual elements of candidate behavior that also contribute, if not outperform, the influence of candidate statements.” Mutz and Reeves (2005; also see Mutz, 2007) examine this question in the context of heated exchanges among pundits. They find that when contention is highlighted through more aggressive exchanges and production decisions, despite holding the content constant, the conflict affects viewer evaluations, further indicating that tonal and visual elements matter.
Presentation and Performance
Production practices may thereby significantly affect the way candidate performances are received and often heighten the sense of conflict perceived by television audiences (Alexander, 2010; Davis, 1999; Zettl, 1990). To be sure, candidates have relatively little control over the format in which they are presented during a televised debate and must instead work within the constraints broadcast networks impose (Peters, 2012). Mutz (2007) demonstrates this point and argues that the effect of televised incivility—that is, being disrespectful, interruptive, and inattentive—is amplified by close-up camera shots that highlight interpersonal conflict. The split-screen format also emphasizes debate contentiousness and allows “viewers to constantly monitor the words, gestures, and reactions of each candidate, [intensifying the] perception of conflict in much the same manner as close-up camera shots” (Cho et al., 2009, p. 245).
While research on debate format has shown that presentation does matter, the effect of these changes is still not entirely understood. Split-screen shots during debates generally provide more information, allowing the public to better evaluate subtle aspects of the candidates’ expressive behavior. Consistently and prominently filling the screen with candidates’ faces allows viewers to monitor expressions, demeanor, and gaze, details individuals naturally account for when evaluating politicians’ self-presentation. As Jeffrey Alexander (2012, para. 9) notes, “Political performances are also about eyes and energy, about looking and being looked at, about seeming eager and interested and caring.” The intimate nature of debate coverage in 2012 facilitated this type of evaluation among Americans.
The first and third debates in 2012 thus offer an ideal opportunity to evaluate the influence of both spoken and nonverbal candidate communication. Camera shots in both debates were consistently tight and simultaneously displayed both candidates on screen. As suggested by prior work concerning format and presentation, viewers during these debates likely used candidates’ body language, facial expressions, voice tone, and physical gestures in making political evaluations. Fortunately for our purposes, candidate performance also varied significantly across the two debates, with many commentators suggesting that Romney outperformed Obama in the first debate but that Obama was more successful in the third. 2 These debates thus offer a valuable opportunity to consider the nature and impact of candidate verbal and nonverbal behavior in different moments of the same general election environment, holding the candidates and presentational format constant. In addition to several measures of candidates’ rhetorical style, our analysis therefore examines the influence of three general categories of candidates’ nonverbal communication: voice tone, expressions and gestures, and blink rate.
Voice Tone
Voice tone is a paralinguistic cue present in all spoken communication. Tone modifies the content of speech by imparting emotion and potentially moderating the meaning of the message, informing audience reactions (Hall, 1980). Tone also signals social intent, expressing a variety of purposes such as disapproval or threat, in the case of an angry tone, or reassurance, as in the case of a friendly one. As Laplante and Ambady (2003) find, tone of voice shapes interlocutors’ ratings of politeness or impoliteness. Longitudinal studies of presidential election coverage have found that challengers and debate losers tend to be more aggressive in tone than incumbents and front-runners (Bucy & Grabe, 2008; Grabe & Bucy, 2009), consistent with their tendency to attack (Benoit, 2013).
Expressions and Gestures
Often even more consequential than candidates’ tone of voice is the quality of their facial expressions. The face, more than any other expressive feature, serves as the primary conveyance of emotional communication, signaling affective states and behavioral intentions to observers. Researchers identify a few key categories of nonverbal communication displays, two of which are relevant to political communication: happiness/reassurance and anger/threat (Sullivan & Masters, 1988). Happiness/reassurance displays are intended to facilitate a hedonic or friendly mode of social interaction, lowering the probability of an aggressive or antagonistic encounter. Anger/threat displays, on the other hand, are associated with hostile encounters and attempts to unseat the leader of a social group (Bucy & Grabe, 2008). Although reassurance is more commonly associated with effective leadership, in competitive contexts, partisans also respond positively to anger/threat (Masters, Sullivan, Lanzetta, McHugo, & Englis, 1986).
Similar to voice tone, trailing candidates are shown more often in political news coverage as exhibiting anger/threat and making defiant gestures, including finger pointing or wagging, making and brandishing a fist, or shaking their head in disagreement (Grabe & Bucy, 2009). Leading candidates, by contrast, are more likely to be shown engaging in positive affinity behaviors that imply bonding, compassion, or friendship. Less nuanced than facial expressions, hand gestures usually work in tandem with facial expressions, potentially amplifying their effect. The capacity of nonverbal displays to influence decision making depends, at least in part, on candidates’ political status and expressive ability, as well as the context of the displays.
In close elections where much is at stake, leader displays are likely to take on added significance. High-status leaders are said to have an “attention-binding” quality that draws continued observance by other members of the social group (whether journalists, party activists, or interested voters). Indeed, Sullivan and Masters (1993) have characterized facial displays as leadership behavior. Thus, when the incumbent performs poorly or commits a violation of nonverbal expectations (see Burgoon & Hale, 1988), such as acting in an inappropriate or unexpected manner, we would expect a higher volume of attention to and communication in response to these actions, as well as an outpouring of negatively valenced reactions.
Eye Blink Rate
Another indirect indicator of a candidate’s mental and emotional state during a debate is blink rate. Eye blinks typically involve the rapid closing and opening of the eyelids over the eyes (typically within 300-400 milliseconds) in an autonomic fashion, though blinking can also be deliberate. Physiologically, blinking helps provide moisture to the eye and protects the eye from irritants. But blinking has cognitive and emotional dimensions as well. Indeed, three types of eye blinks have been identified—reflex blinks that protect the eye from environmental intrusions, voluntary blinks, and spontaneous or “endogenous” blinks that occur in the absence of identifiable stimuli and are thought to be influenced by perceptions, affective orientation, or information processing (Stewart & Mosely, 2009).
A variety of physical and psychological factors influence this basic semi-autonomic response (Doughty, 2001). Those experiencing task-related fatigue or anxiety, for example, often blink at a faster rate than otherwise similar individuals (Barbato et al., 2007; Harris, Thackray, & Shoenberger, 1966). A number of studies have found that humans generally blink more when concentrating on a task or are under stress to manage dissonance 3 (Oh, Han, Peterson, & Jeong, 2012), especially when they deem the outcome to be important (Karson et al., 1981; Shultz, Klin, & Jones, 2011). As one study put it, the “eye blink [may be] triggered by aspects of information processing, and . . . blink latencies can be used as one tool for evaluating the level of complexity of such processing under a wide variety of task demands” (Fogarty & Stern, 1989, p. 35).
While laboratory research on blink rate has identified the underlying processes these behaviors signal, social scientists have also used blink rate to explore the social information they provide. Several studies have found that individuals monitor the blinking behavior of interaction partners and use it as a meaningful indicator of social information alongside facial expressions (Cunningham & Wallraven, 2009). Our brains instinctively respond to the social relevance of others’ blink patterns in a manner similar to other facial features and gaze behavior (Brefczynski-Lewis, Berrebi, McNeely, Prostko, & Puce, 2011). Whatever the ultimate reason, people with unusually rapid blink rates are often perceived as careless, less intelligent, or nervous, while those who blink especially rapidly or slowly are considered “unfriendly” (Omori & Miyata, 2001). Thus, both average blink rate and deviations in blink rate may be noted.
Accordingly, in a study of the 1976 presidential debates, Exline (1985) found that increased blinking by both Gerald Ford and Jimmy Carter negatively affected judgments of candidate competency. Similarly, Patterson et al. (1992) showed that elevated blinking by Walter Mondale in the 1984 presidential debates with Ronald Reagan was associated with lower favorability scores. Thus, excessive blinking and significant variation in blink rate carries evaluative consequences in social as well as political settings.
Hypotheses and Research Questions
In this study, we measure communication behavior in response to actual presidential debates as they occurred, by tracking the volume and sentiment of real-time public expression on Twitter. By measuring actual responses from users of a popular social network, this approach benefits from a high degree of ecological validity. Although Twitter users are not representative of the population, they are nonetheless quite diverse, and their voluminous real-time comments allow us to trace, in a highly granular fashion, the connections between the first and second screens that characterize the television viewing experience in a social media age.
We expect that the tonal and visual elements of the candidates’ debate performance, including blink rate, will shape reactions on social media more so than the verbal strategies candidates use to score political points, win favor with the public, and (at times inadvertently) create quotable moments in the form of memes. In light of these considerations, we predict that:
Method
To examine these relationships, we merged two data sets for each debate: (a) a detailed content analysis of the first and third presidential debate between Barack Obama and Mitt Romney that coded for rhetorical/functional, tonal, visual elements and (b) corresponding measures of the volume and sentiment of Twitter posts mentioning the candidates.
Debate Coding
For the verbal, tonal, and visual content analysis, we used C-SPAN’s televised coverage of the first and third debates. Throughout the debate, this feed showed both candidates in split-screen format, using a medium shot from the waist up, enabling coding of all nonverbal responses and blink rate. Given that other broadcasters drew on the same nine camera feeds, but were free to use whichever images they chose, it is notable that all the major news outlets—ABC, CBS, NBC, CNN, and Fox News—favored split-screen shots similar to those used by C-SPAN. PBS, by contrast, used a single camera perspective. To standardize analysis across comparable units, the verbal, tonal, and visual aspects of the debate (including blink rates) were coded at the level of the individual camera shot. In cases where segments exceeded 30 seconds (which was typical), they were divided into 30-second increments. For the first 90-minute debate, this resulted in 169 codable segments ranging from 12 seconds to 30 seconds. For the third debate, this resulted in 181 codable segments ranging from 10 seconds to 30 seconds.
Coding of the debates proceeded in two stages. In the first stage of the analysis, the candidates’ statements were coded for their primary rhetorical function and “memes”—a reproducible unit of culture that is widely shared (Dawkins, 1976). In the second stage, the candidates’ nonverbal behavior was coded for tonal and visual elements, as well as blink rate.
Rhetorical Functions
Major rhetorical functions identified in individual debate segments included attacks on the opponent, contrast statements that highlighted differences between the speaker and his opponent, direct responses to statements (typically, answers to attacks) made by the opponent, and personal narratives from the candidate’s own experience (see Benoit & Harthcock, 1999; Green & Brock, 2000, for details on these categories). 4
Memes
As verified by our reading of the news coverage and social media content surrounding each debate, we identified the statements and phrases used to create one or more memes—that is, intended and inadvertent utterances that produced memorable and highly referenced moments. We identified five memes in the first debate, including Romney’s promise to cut funding for public television, even though he declared, “I love Big Bird”; and Obama’s statement to moderator Jim Lehrer, “I had five seconds before you interrupted me.” Similarly, we coded five memes in the third debate, including Obama’s “fewer horses and bayonets” comment, and Romney’s statement, “I love teachers.” Because memes are memorable debate encounters that flow from the utterances of the candidates, they were included with the verbal elements we coded and were simply identified as occurring during the period in which they were uttered. Via digital media, memes can be reproduced almost immediately, and are discussed by pundits in postdebate analyses and rebroadcast in news reports; as such, these simple utterances become highly quotable.
Tonal Elements
In any persuasive encounter, a large part of nonverbal influence stems not just from semantic content but also from voice tone and variability (Bucy & Grabe, 2008). We therefore coded for the presence or absence of two emotion/intention pairs that play a central role in political competition: anger/threat and happiness/reassurance. These categories reflect the felt emotion and presumed behavioral intention of the communicator (Way & Masters, 1996).
Anger/threat was operationalized as statements in which the candidate’s tone had a menacing or hostile feel; where the candidate used confrontational verbal tactics to challenge his rival; where the candidate revealed a desire to do political battle, or took exception to and forcefully rebutted a claim by his opponent; or, where the overall tone of a segment could be characterized as enraged, feisty, bold, or aggressive.
Happiness/reassurance was operationalized as statements in which the candidate’s tone had an optimistic or cheerful feeling; where the candidate’s voice was upbeat, positive, and conveyed an affiliative intent; where the candidate offered hopeful predictions about what will happen to the country if elected; or, where the tone suggested an attempt at bonding or reinforcing a sense of goodwill with potential supporters.
Visual Elements
Next, we coded aspects of the candidates’ nonverbal behavior, particularly facial expressions and body language. Consistent with voice tone, we coded for the presence of anger/threat and happiness/reassurance in facial expressions. We also documented the occurrence of gestures that signaled affinity (bonding) or defiance (aggression).
Consistent with the biobehavioral approach to nonverbal communication (see Masters et al., 1986), facial expressions that contained one or more of the following key elements were classified as anger/threat displays: lowered eyebrows, a staring gaze, the visibility of lower teeth, lowered mouth corners (frowning), facial rigidity that showed little to no movement, lips pressed firmly together, or an overall expression that was negative or hostile.
Similarly, happiness/reassurance displays were operationalized using very specific behavioral criteria as expressions containing one or more of the following elements: a smile with relaxed mouth position, the visibility of upper or both rows of teeth, nodding up and down, brief eye contact to avoid staring, open or just slightly closed eyes, “Crow’s feet” wrinkles around the eyes, or an overall expression that was welcoming.
Gestures were coded as body language that signaled affinity or defiance (see Grabe & Bucy, 2009). Affinity gestures consisted of hand, body, or facial movements that suggest a friendly relationship or attempt at bonding between the candidate and the audience, opponent, or moderator. Examples include waving or giving a “thumbs-up”; winking or nodding knowingly to the camera, moderator, or other candidate; or, using an open palm when referencing the audience or opponent (rather than a closed fist or pointed finger).
Defiance gestures consisted of hand, body, or facial movements that suggest a threatening or antagonistic relationship between the candidate and their opponent. Examples include finger pointing, wagging, or shaking; raising a fist; shaking one’s head in disagreement; negative expressions accompanied by prolonged stares; or, other behaviors signaling aggression. Frequencies of these manifest tonal and visual elements are provided in Table 1.
Nonverbal Display Frequencies, Key Variables.
Note. Percentages may not add up to 100 because more than one display type may occur in any coded segment. Frequency counts in parentheses.
Blink Rate
In addition, eye blinks of both candidates were manually coded at 5-second intervals, consistent with previous research (see Stewart & Mosely, 2009). Altogether, this resulted in 1,060 codable segments for Debate 1 and 1,083 segments in Debate 3. Obama (M = 4.55, SD = 2.98) exhibited a slightly higher average rate of blinking than Romney in Debate 1 (M = 4.09, SD = 2.31) as well as Debate 3 (Ms = 5.22, 4.64, SDs = 1.73, 1.31, for both candidates, respectively). Obama showed more variability than Romney, with variance scores of 8.87 and 3.01 in Debates 1 and 3, compared with Romney’s 5.34 and 1.72. The maximum number of blinks in any given segment was also highest for Obama, with 13 in Debate 1 (compared with 10 for Romney). Both candidates had a maximum number of 10 blinks in any segment in Debate 3. To facilitate analysis with our other measures, blink rate data were collapsed into corresponding segments (which ranged from 10 to 30 seconds) to parallel the verbal, tonal, and visual variables. For each segment, average blink rates and standard deviations were calculated for candidates.
Intercoder Reliability
For the first debate, the presence or absence of each verbal, tonal, and visual element was coded for in each shot or 30-second segment. For all 169 codable segments, each element was coded 1 if present and 0 if absent. A similar procedure was followed in the third debate, with each verbal, tonal, and visual element rated for the 181 codable segments, with the exception that visual displays were also coded for degree of intensity (2 = vivid display, 1 = visible display, 0 = no display) given the generally negative and aggressive affect of both candidates throughout much of the third debate.
For the verbal elements, a second trained coder assessed each of the 350 retained segments, agreeing on all but 88 of the 3,500 individual codings and producing initial Krippendorff’s alpha values ranging from .89 for Obama’s use of contrast to .99 for Romney’s use of response. Instances of disagreement were discussed between coders until a consensus was achieved. For the tonal and visual variables, 11% of the debate footage, or 40 segments, were assessed by a second coder. Tonal variables had a similarly high level of agreement, with α values of .91 and 1 for Obama and Romney’s anger/threat and happiness/reassurance voice tone, respectively. Gestures’ α values ranged from .90 for Obama’s defiance gestures to 1 for Romney’s defiance gestures. For facial expressions, again of anger/threat and happiness/reassurance, there was .95 agreement, which dropped to α values of .83 and .78. Reliability testing of the blink rate data was conducted on about 15% of the segments from each debate (152 from Debate 1 and 219 from Debate 3) by two trained coders. Intercoder reliability was high, with α values of .91 for Debate 1 and .94 for Debate 3.
Twitter Harvesting
For this study, we archived the Twitter garden hose as a sample of social media activity through its Streaming API. 5 Twitter describes the garden hose as a continuous 10% sample of the 300 to 500 million global tweets per day. Tweet information includes a variety of different fields, including user information, tweet time, geolocation (if available), and platform used to post the tweet. 6 From this archive, we drew posts within a 50-day window before the election, from September 19, 2012, to November 8, 2012, that included “Obama,” “Romney,” “Biden,” or “Paul Ryan” (not case-sensitive). We then identified all tweets from our sample that referenced Obama or Romney and were posted during either the first (October 3) or the third (October 22) debate. We generated volume measures from mentions of only “Obama” or only “Romney” and not both within the text of the tweet. While this undercounts the total candidate mentions, our intent was to label whether a tweet was about a particular candidate and to understand over-time variability in those mentions.
Sentiment analysis was performed using a supervised machine-learning method, in which a function for inferring the positivity or negativity of a post was generated by the labeled training data. To generate a training set, we randomly sampled 1,000 tweets from each debate period, 500 mentioning Romney and 500 mentioning Obama. Two coders then coded these tweets for three possible sentiment values: positive, negative, or neutral. After coding separately, coders met to resolve any disagreements. In the first debate, we found that most tweets (87%) were either positive or negative. The final training set for sentiment analysis of the first debate therefore used only this subset of tweets that were coded as positive or negative, numerically represented as 1 and −1, respectively. In the third debate, we used all three values to assign a sentiment score. Last, we trained a regularized logistic regression classifier using the Python scikit-learn library. The classifier obtained an F1-score of .77 for the first debate and a F1-score of .71 for the third debate, both within acceptable levels of accuracy (Yang & Liu, 1999).
To align Twitter activity to archived video material of the debate, we identified points during both debates where we knew there was high Twitter volume: Romney’s quip, “I love Big Bird”; Obama’s “I had five seconds” remark; Romney’s “apology tour” comment, and Obama’s “horses and bayonets” retort. Assuming a small gap between broadcast and Twitter reaction, we then used the first mention of these sound bites as the measure of the lag. We found a consistent gap between the debate clock on the C-SPAN feed and the UTC (Coordinated Universal Time) timestamp on the Twitter posts. Accordingly, we synchronized Twitter data to the debate feed. Each coded shot of the debate was the unit of observation and analysis, with the volume and sentiment of expression on Twitter normed by the length of the unit.
For each shot, we next generated a synchronous volume metric and average sentiment score for both candidates, along with lagged versions of these variables, at 15, 30, and 45 seconds. These lagged values account for any delayed reaction by Twitter users and observe the robustness of observed effects. This resulted in the generation of 16 variables for each debate, matched or lagged to the start-stop times on the debate clock (volume and sentiment for Obama and Romney synchronous to the debate clock, and at 15-, 30-, and 45-second lags). Because the results at 30 seconds are very consistent with the 15-second lag, we omit presentation of those analyses, opting to make them available as an online appendix (available at http://abs.sagepub.com/content/by/supplemental-data).
Figure 1 shows the volume per minute by candidate for each debate, while Figure 2 shows the average sentiment per minute by candidate. The lines on each graph represent LOESS regression smoothed averages for the candidates over their data points, which provide the minute-by-minute averages. The plots in Figure 1 are comparable with Twitter’s own graphs of total debate volume, not distinguished by candidate. 7 The sentiment plots reveal that Twitter favored Obama over Romney during both debates, a result that held throughout each event, albeit with considerable variation over time. Notably, both candidates averaged sentiment scores below 0, indicating generally negative expression on this social media platform.
Volume per minute by candidate, with LOESS regression smoothed average.
Sentiment per minute by candidate, with LOESS regression smoothed average.
Analysis
Before testing our hypotheses, we thought it important to verify that the debates were indeed moments of national attention and expression. To do so, we charted the volume of keyword references to Obama, Romney, Biden, and Ryan during the 50-day window of our archive draw. The results are presented in Figure 3 and clearly indicate that the debates were high-intensity moments during the campaign, with the first debate between Barack Obama and Mitt Romney being the highest volume moment other than Election Day itself.
Volume of name mentions of presidential and vice-presidential candidates on Twitter.
Next, to estimate the effects of verbal, tonal, and visual factors on the volume and valence of online expression about the debates, we estimate time series regression models that take into account the autoregressive nature of the data-generating process. Durbin–Watson tests on each of our dependent variables reveal significant autocorrelation. To take account of this temporal dependence, we use generalized least squares (Prais–Winstein) regression to estimate our models. These models adjust the variance–covariance matrix to account for a first-order autoregressive process in the error term of the model, allowing for a more conservative test of our hypotheses. We calculate the adjustment (rho) using the single-lag ordinary least squares estimate of the residuals from the original estimating equation. We begin with saturated models with all predictors included. We then removed variables that were consistently not significant predictors across the models, leading us to drop candidates’ use of contrast, response, and narrative verbal strategies and happiness/reassurance facial expressions. Retaining these variables yields no changes on the model estimates discussed below.
We conducted these regression analyses on the normalized volume measures (ratio of number of posts to seconds in the segment) and average sentiment scores (the mean value of scores across seconds in the segment) of the tweets mentioning Obama or Romney in Debate 1 followed by Debate 3. As noted above, these analyses were run with two different versions of the dependent variables: synchronous and 15 seconds after the segment.
For each debate, independent variables were grouped in three blocks that were sequentially added to each regression: verbal, tonal, and visual. Models separately tested the volume and valence of mentions for each candidate. The first block consisted of candidates’ use of attacks and indicators of whether either candidate sparked a meme during the shot. The next block added variables assessing each candidate’s voice tone (anger/threat, happiness/reassurance). The third block added Obama’s and Romney’ facial expressions (anger/threat), affinity and defiance gestures, average blink rate, and standard deviation of that blink rate per segment. Both blink rate variables were included because it is not only the rate but also large deviations that are notable.
Results
The results of the generalized least squares regression analyses present a clear picture of the factors that shaped the volume and valence of responses on Twitter. Beginning with the volume of Obama mentions in the first debate, the results are consistent across the synched and 15-second lag models (see Table 2; columns 1 and 2). The verbal factors account for a sizable amount of variance in the volume of Obama mentions, with F-change scores significant in the synchronized tests, accounted for mainly by Obama’s generation of memes and to a lesser degree by Romney’s generation of memes. Comparatively, when the tonal block is added, the models’ performance does not improve substantially, with F-change scores insignificant for both tests. Only an angry or threatening tone by Romney was negatively related to volume of Obama mentions. Likewise, the addition of the visual block does not contribute significantly to the models’ performance across both F-change tests. Nonetheless, Obama’s angry or threatening facial expressions and defiance gestures contribute to the synchronized and lagged models, respectively. The analysis also indicates that the variability of the blink rate within segment reduced the reaction in terms of the volume of social media posts.
Synchronous and Lagged Models Predicting Normalized Volume of Obama and Romney Mentions for Debate 1—Generalized Least Squares (Prais–Winstein) Regression.
Note. Standard errors in parentheses.
p < .01. **p < .05. *p < .1.
Moving next to the volume of Romney mentions during the first debate (see Table 2; columns 3 and 4), the verbal factors account for significant variance in the immediate volume of Romney mentions, with F-change tests achieving statistical significance, driven mainly by Romney’s memes. The tonal block adds to overall performance of the models, with F-change scores significant in both tests, largely a function of the use of a happy or reassuring tone by Obama. The addition of the visual block does not contribute to the performance of either model. Romney’s angry/threatening facial expressions and defiance gestures account for most of the explained variance within the block, although the latter barely miss conventional levels of statistical significance. These results, in combination with those observed for Obama volume, provide some support for Hypothesis 1 in the context of the first debate.
For the sentiment analysis of posts mentioning Obama during the first debate, we again ran the synched and 15-second lag models (see Table 3; columns 1 and 2). The verbal factors account for a significant amount of variance in Obama sentiment, with the F-change statistic significant for the 15-second lag model, driven mainly by Obama’s and Romney’s memes. The tonal block contributes minimally, failing to reach significance in any of the tests. Conversely, the visual block contributes significantly, again in the 15-second lag model based on the F-change test, mainly a function of Obama’s gestures, both affinity and defiance.
Synchronous and Lagged Models Predicting Sentiment of Obama and Romney Mentions for Debate 1—Generalized Least Squares (Prais–Winstein) Regression.
Note. Standard errors in parentheses.
p < .01. **p < .05. *p < .1.
These findings contrast somewhat with the models predicting sentiment of Romney mentions (see Table 3; columns 3 and 4). The verbal and tonal factors account for little variance in the sentiment scores for Romney, failing to achieve significance in any F-change tests. In sharp contrast, the addition of the visual block contributes substantially and significantly to the synchronous models’ performance. Two visual factors stand out: Obama’s gestures and Romney’s blink rate. These results, in combination with those observed for Obama sentiment during this first debate, provide some support for Hypothesis 2, especially concerning the visual elements.
Turning to the volume of Obama mentions in the third debate, the results are highly consistent across the synched and 15-second lag models (see Table 4; columns 1 and 2). Yet, unlike the first debate, it is not Obama’s memes but his attacks that account for significant variance explained by verbal factors when predicting the volume of Obama mentions. Notably, the F-change is significant in both models. When the tonal block is added, the models’ performance does not improve. Yet, unlike the first debate, the addition of the visual block contributes considerably to the models’ performance, with F-change scores significant for both models. This result is largely due to Obama’s angry or threatening facial expressions and his use of affinity gestures. Romney’s blink rate is also a significant predictor, contributing to the synched and lagged models. We want to be careful not to read too deeply into the differences between the two debates, but the increasing potency of the visual features is notable.
Synchronous and Lagged Models Predicting Normalized Volume of Obama and Romney Mentions for Debate 3—Generalized Least Squares (Prais–Winstein) Regression.
Note. Standard errors in parentheses.
p < .01. **p < .05. *p < .1.
Moving next to the volume of Romney mentions during the third debate, the results are fairly consistent across the synched and 15-second lagged models (see Table 4; columns 3 and 4). The verbal factors account for little variance in the immediate or lagged measures of Romney mentions, with F-change tests failing to achieve significance in either model. The tonal block adds to overall performance in the lagged model, driven by Romney’s expressions of anger and frustration. The addition of the visual block contributes substantially to the models’ performance, but again, only in the lagged model. Romney’s defiance gestures account for most of the explained variance, linked to greater volume of Romney mentions, whereas Obama’s angry and threatening facial expressions and Romney’s heightened blink rate yields fewer mentions, the latter in 15-second lag model only. These results, in combination with those observed for Obama volume, provide strong support for Hypothesis 1, particularly for the visual elements in the third debate, building on results from the first debate. Indeed, visual elements appear to play a somewhat larger role in third debate than in first debate.
For the sentiment of Obama posts during the third debate, we again ran the synched and 15-second lag models (see Table 5; columns 1 and 2). The verbal factors account for immediate variance in Obama sentiment, with F change significant for the synched model, driven mainly by Obama’s and Romney’s use of attacks. The tonal block does not consistently contribute variance across the models, though Romney’s angry or threatening tone is negatively related to positive sentiment. The addition of the visual block contributes significantly to the synchronous model, with Obama’s angry/threatening facial expressions and his blink rate as individual predictors.
Synchronous and Lagged Models Predicting Sentiment of Obama and Romney Mentions for Debate 3—Generalized Least Squares (Prais–Winstein) Regression.
Note. Standard errors in parentheses.
p < .01. **p < .05. *p < .1.
The models predicting sentiment of Romney mentions (see Table 5; columns 3 and 4) in the third debate do not perform well. The verbal factors account for minimal variance in the sentiment scores for Romney, failing to achieve significance in any F-change tests. The addition of the tonal block is significant in just one case, for the synchronized model, with Romney’s angry and threatening tone diminishing sentiment in the synched model but enhancing it in the lagged model. In contrast, the addition of the visual block does not contribute to the models’ performance, with few individual predictors contributing significantly, namely Obama’s angry/threatening facial expressions and defiance gestures enhancing Romney’s sentiment. These results, in combination with those observed for Obama sentiment, provide limited support for Hypothesis 2 for the third debate, with nonverbal elements occasionally predicting sentiment. The lower potency of nonverbal elements in explaining the sentiment of responses in the third debate compared with the first debate runs counter to the pattern observed for the volume of mentions.
Taken as a whole, however, the results between these two candidates across two different debates indicate that the candidates’ nonverbal communication shaped the volume and valence of expression on Twitter, providing somewhat stronger support for Hypothesis 1 than for Hypothesis 2. Without overemphasizing the differences between the two debates, attention across the tests of incremental variance explained finds that the candidates’ rhetorical attacks and meme generation contributed to the models in 4 of 8 tests for the volume of second screen expression about the candidates and in 2 of 8 tests for the sentiment of this expression. With these verbal elements assessed, the tonal elements contributed incremental variance to the predictive models in 3 of 8 tests for volume (Hypothesis 1a) and in 1 of 8 tests for the sentiment (Hypothesis 2a). After accounting for these verbal and tonal elements, the visual elements contributed significant incremental variance to the predictive models in 3 of 8 tests for volume (Hypothesis 1b) and in 3 of 8 tests for sentiment (Hypothesis 2b). This pattern of findings suggests a meaningful and consistent influence of nonverbal communication elements on volume and sentiment. Taken together, the tonal factors contribute incremental variance in only 4 of 16 tests beyond verbal factors. In contrast, candidate expressions, gestures, and blink rate contributed incremental variance to the models in 6 of 16 tests even after accounting for verbal and tonal factors. Overall, this pattern lends support to Hypothesis 1, with tonal (Hypothesis 1a) and visual elements (Hypothesis 1b) jointly contributing to the predictive models. In contrast, results revealed more modest support for Hypothesis 2, with visual markers (Hypothesis 2b) performing better than the tonal variables (Hypothesis 2a) in terms of consistency of contribution to the predictive models.
Discussion
Consistent with theoretical expectations and previous experimental findings, the nonverbal behavior of candidates is consequential in driving social media responses, rivaling what candidates actually say during debates (Benoit, 2013; Cho et al., 2009; Zhu et al., 1994). These findings expand on previous work connecting debate performance to reactions on social media (Shah et al., 2015) by looking across two debates, one focused on domestic policy and the other on foreign policy, and integrating additional nonverbal markers, specifically, average blink rate and its standard deviation, into predictive models. Equally important, the analysis reported here employs generalized least squares (Prais–Winstein) regression, a time series modeling technique that addresses the significant autocorrelation in our data. Notably, these results yield substantially more conservative estimates than ordinary least squares regression.
Given this more conservative approach, it is notable that across both of these debate contexts, and two different dependent variables for two different candidates, the tonal and visual aspects of debate presentations explained variance in the volume and valence of second screen expression beyond the verbal aspects of debates. By linking the content of first and second screens during major political events and examining the immediate and lagged connection between candidates’ behavior and users’ social media expression, this work provides a novel method for analyzing real-time effects of broadcast media on social media. In doing so, it shares some similarity with prior debate analysis by Nagel et al. (2012) and Shah et al. (2015), while improving on the methodological approach employed in these earlier efforts to understand audience responses to debate verbal, tonal, and visual features.
And what does this effort to connect debate presentations and social expression reveal? Certainly, this work confirms that debates are moments of “national conversation,” with peaks of expressive activity that dwarf all other campaign events in the 50 days before the election. But if public expression corresponds to nonverbal cues as much as verbal statements, we may have to rethink what constitutes a debate effect. This analysis suggests that the Twitter-using public, or at least that subset of users who choose to be active during presidential debates, responds as strongly to the visual elements of candidate behavior, including facial displays, expressive gestures, and their blink rates, as they do to verbal elements, particularly candidate memes or disputes. Future research should look beyond the volume and valence of online expression to examine other discursive outcomes.
Given the capacity of candidate nonverbal behavior to spark conservation, the quality of democratic discourse resulting from reactions to debate performances merits closer attention. When witnessed by audiences, do facial expressions, physical gestures, and autonomic responses (blink rates) trigger more attentive processing and encourage members of the viewing public to generate meaningful responses that productively contribute to a democratic dialogue? Or do these visual cues spur superficial utterances, focusing on the appearance or distress of the candidate? These remain open questions and warrant further attention from research following this line of research. Previous work has shown that candidates’ tonal and visual behaviors provide relevant social information, with nonverbal communication treated as a more reliable predictor of leader traits than verbal utterances (see Bucy & Grabe 2008; Masters et al., 1986). Nonverbal cues, for instance, facilitate accurate inferences about candidates’ competence and integrity (Olivola & Todorov, 2010).
Our analysis also reveals that while candidate attack strategies and memes explain some variance in the amount and favorability of expression, they were matched by the influence of nonverbal factors, especially candidate expressions, gestures, and blinking. It is notable that across these two debates, the most powerful verbal elements were memes—the quotable and malleable utterances that are easily repeated or referenced on a short messaging platform such as Twitter. Moreover, it appears that visual factors often absorb the effects attributed to tonal factors, suggesting a complex interplay between some of these interrelated nonverbal elements. Subsequent analyses should consider accumulated or conditional effects, such as how an angry or threatening tone works alongside corresponding facial expressions, or how defiance gestures are enhanced by a rapid blink rate or sudden variance in this response.
Future research should also differentiate among social media users to examine how subgroups respond to discrete moments during debates or other televised events. Classifying social media users based on their ideology or geography would allow their expressions to be differentiated and their responses to be modeled separately. By using profile information, geographic tags, and previous tweets to classify users, these relationships can be examined within subgroups rather than in the aggregate, revealing how different user segments respond to the verbal, tonal, visual, and autonomic aspects of candidates’ debate performance. This would be an appropriate extension of the novel methods advanced in this article, connecting the power of television images to the responses of social media audiences via the second screen.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the National Research Foundation of Korea Grant funded by the Korean Government (NRF-2013S1A3A2055285).