#Ebola: Emergency Risk Messages on Social Media

Abstract

Public health threats require effective communication. Evaluating effectiveness during a situation that requires emergency risk communication is difficult, however, because these events require an immediate response and collecting data may be secondary to more immediate needs. In this article, we draw on research analyzing the effectiveness of social media messages during times of imminent threat and research analyzing the emergency risk communication conceptual model in order to propose a method for evaluating emergency risk communication on social media. We demonstrate this method by evaluating 2,915 messages sent by local, state, and federal public health officials during the 2014 Ebola outbreak in the United States. The results provide empirical support for emergency risk communication and identify message strategies that have the potential to increase exposure to official communication on social media during future public health threats.

Introduction

When a public health emergency threatens, effective messaging by public health officials on social media has the potential to provide reassurance, accurate information, and lifesaving instruction. People turn to social media during public health crises,¹ and public health officials increasingly use social media to correct misinformation, dispel rumors, and provide resources.^2,3

Despite these practices and the breadth of research identifying emergency risk communication as a key element of public health emergency planning and response,⁴ little research has been conducted that evaluates the effectiveness of emergency risk communication on social media. Most research examining official communication on social media during public health threats has been descriptive (eg, see Tang and colleagues).⁵ Few studies have evaluated the social media messages sent by public health officials in terms of message effectiveness.⁶

In general, measuring the effectiveness of emergency risk communication presents challenges for researchers, most notably due to the difficulty of collecting data during events that require immediate response and cannot be recreated in a laboratory. However, the conceptual model of emergency risk communication, developed by the US Centers for Disease Control and Prevention (CDC) in conjunction with scholars,⁴ provides a framework for evaluating emergency risk communication. The model draws on existing literature to articulate the practices used to develop and disseminate emergency risk communication. Many constructs within the model have not been operationalized, and, therefore, the model needs further refinement to allow for measurement and assessment.⁷ In this article we draw on research analyzing the effectiveness of social media messages during times of imminent threat and longer crises such as emerging infectious diseases,^6,8-10 in addition to research analyzing the emergency risk communication conceptual model.⁴ Based on this research we propose a method for evaluating emergency risk communication disseminated on social media. We then demonstrate how to use this method by analyzing social media messages sent by US public health agencies during the 2014 Ebola outbreak in the United States. Findings from this research identify communication strategies that increase the reach of public health messages.

Emergency Risk Communication Conceptual Model

The emergency risk communication conceptual model describes the process that CDC uses to develop and evaluate messages across the life cycle of a public health emergency.⁴ The process described in the model includes both the features of the messages and of the dissemination process. Features of the messages include scientific accuracy, actionable language, openness and transparency, clarity, and audience tailoring; features of the dissemination process include message sufficiency, timely dissemination, and use of multiple channels. The model identifies a series of short-, mid-, and long-term outcomes. Short-term outcomes include factors that relate to the distribution of messages (reach of messages, higher levels of information sharing) and factors that describe psychometric changes in the audience (increases in awareness and information seeking, reduction in uncertainty). The mid- and long-term outcomes identified by the model also describe changes in the audience (increased knowledge, source credibility, self-efficacy, alignment of risk perceptions, engagement in protective behaviors).

Evaluating Emergency Risk Communication on Social Media

Social media messages are distributed through network users. To reach large numbers of users, emergency risk messages have to be passed on (eg, shared on Facebook or retweeted on Twitter). Message passing is a behavioral reaction that indicates a message has been noticed, similar to commenting and liking.¹¹ Although social media platforms make message passing simple, requiring only the tap of a button, most messages are not passed on.¹² For messages that are passed on, however, the effects can be dramatic in terms of reach. For example, a message about cancer testing sent by actor and director Ben Stiller on Twitter reached 5.1 million accounts on its first iteration, and retweeting by others increasing its reach to an additional 20.2 million accounts.¹³ For emergency risk communication, message passing provides a way to measure both audience engagement and increased exposure. Although message passing has not traditionally been used as a measure of message effectiveness in communication studies,¹⁴ the increase in exposure or reach created by message passing is a precursor to more traditional measures of message success, including changing attitudes, intentions, and behaviors in the context of health messages.^15,16

Message passing occurs in complex environments with multiple factors that influence how often a message is passed on.⁶ These factors include the network surrounding the sending account,^17,18 the characteristics of the sender,^9,19 the intrinsic features of the messages itself,^6,20 and the timing of the message in relation to the events.²¹ Some of these factors, like the features of the message itself or the timing of the message in relation to the event, can be controlled by the officials sending the message. Other factors, like the network surrounding the account (eg, the number of followers an individual account has) and the innate characteristics of the sender (eg, the governmental level of the account, the size of the community the account serves) may evolve over time and remain outside of immediate official control. In this article, we evaluate 2 aspects of message passing that officials can easily control during a public health threat: the features of messages and the timing of messages.

Methods and Materials

To examine the success of key message features identified by the emergency risk communication conceptual model⁴ on social media, we analyzed message passing on a social media platform (Twitter) during the 2014 Ebola outbreak in the United States using traditional and computational social science methods. We chose to examine messages on Twitter because (1) at the time of the study, it was one of the more popular platforms in the United States, with 19% of US adults reporting platform use in 2014²²; (2) it is similar to other popular platforms, like Facebook, that allow users to send messages containing text and images; (3) it is widely used by public health organizations to disseminate time-sensitive information; and (4) it encourages users to “broadcast” themselves and as a result, the messages are publicly available.

We chose the 2014 Ebola outbreak in the United States as a context for this study because the event represented a challenging emergency risk communication problem for US public health officials. Days before the current study began, a West African man in Dallas, Texas, became the first person to be diagnosed with Ebola in the United States. This diagnosis occurred in the middle of the largest Ebola outbreak that had occurred to date, infecting more than 28,600 people, mostly in West Africa, and killing 11,300. Shortly after the first US patient was diagnosed, 2 nurses who treated him became sick and had to be hospitalized. One of the nurses had traveled to Ohio, after exposure and before becoming sick. In the following days, schools closed in Texas and in Ohio, community members and health workers were placed in isolation, and rumors spread on social media about the possibility of the virus becoming airborne. Although only 3 cases occurred in the United States, the exotic and deadly disease evoked fear and concern across populations.²

Between October 1 and 29, 2014, we collected messages from for 544 Twitter accounts associated with local, state, and federal health agencies in the United States, using Twitter's REST API. The study time period coincided with increased US attention to the disease: Twitter mentions of Ebola and Google searches spiked at the beginning of the study period and dropped off precipitously shortly before the study period ended.²³

The accounts included in this study were identified by drawing on publicly available lists of public health accounts maintained by CDC, The Nation's Health (a publication of the American Public Health Association), and a public health researcher, Jenine Harris (@jenineharris).²⁴ In addition to collecting the text of the tweet, our collection protocol captured various metadata including the date and time the message was sent and how often a message was retweeted. Our system collects this information every 24 hours, updating and tracking changes to follower numbers and retweeting in the process.

Accounts that produced at least 1 original message that included the word “Ebola” were then selected for analysis. Of the 544 accounts, 236 produced at least 1 original message (a tweet) about Ebola, for a total of 2,915 messages. Local health departments produced the largest number of tweets (n = 1,222, 41.9%), federal health agencies produced approximately one-third (n = 1,009, 34.6%), and state health departments produced approximately one-fifth (n = 684, 23.5%). On average, each account tweeting about Ebola produced 12 tweets during the study period (mode = 1, median = 4).

Evaluating Messages

The emergency risk communication conceptual model identifies 5 types of message features: scientifically accurate messages, actionable messages, open and transparent messages, clear messages, and tailored messages.⁴ In order to evaluate if these message features contribute to retweeting, we identified specific intrinsic message features related to 4 of these categories. Development of a codebook began with deductive coding of a random subset (157 messages or 5%) in order to identify features relevant to the emergency risk communication conceptual model⁴ and previous research on message passing on social media.^9,10 Table 1 presents 12 intrinsic message features identified in this process, in addition to their definitions, descriptive information, coding method and intercoder reliability, and examples. Figure 1 is an example of a tweet from then CDC Director Tom Frieden.

Figure 1.

This image provides an example of how public health organizations used images in Twitter messages to convey information about Ebola.

Table 1.

Message Features: Definitions, Descriptive Information, Coding Method and Reliability, and Examples

Message Feature	Definition	Descriptive Information n (% of total)	Coding Method (Reliability)	Example Message
Scientifically accurate messages
Threat information	Tweet provides information about the disease itself that can facilitate sensemaking	1,200 (41.2)	Human (α = .99)	You can only get Ebola from touching the blood or bodily fluids of a person who is sick with or has died from #Ebola. http://t.co/beEbABuNtP
Actionable messages
Efficacy information	Tweet provides a specific instruction for response	66 (2.3)	Human (α = 1)	Symptoms for the 1st few days of #flu and #Ebola are similar. Help the medical community & get your #flushot now! http://t.co/DzdCBU7XVt
Open and transparent messages
Official action	Tweet provides information about the actions of officials	809 (27.8)	Human (α = .74)	#LACounty Interim Health Officer: we continue to monitor. NO cases of #ebola in #LACounty http://t.co/GLhSpfbllp http://t.co/eE7Z5BKNUS
First person	Message uses the first person (I, we)	612 (21)	Computer	Fight against #Ebola in W. Africa is going to be long, hard fight. The enemy is a virus, but working together, we can contain it.
Clear messages
Image	Messages coded for the presence of an image	305 (10.5)	Human (α = .9)	See Figure 1
Figurative language	Message uses a comparison, an implied comparison, or other trope	97 (3.3)	Human, (α = .54, 95%)	Bottom line on #Ebola in US: we will stop it in its tracks. CDC has 9-person team doing thorough contact tracing & followup.
Second person	Message uses the second person (you, you're)	361 (12.4)	Computer	#Colorado: You can always gather #Ebola info from our website: http://t.co/nPgnO6amW8 @adamscountygov @douglascountyco @ArapahoeCounty
Additional message strategies
Hyperlink	Message contains an hyperlink to a web resource	1,528 (52.4)	Human (α = .8)	What you need to know about #Ebola Q&A w/ @Gstephanopoulos & our @DrJayVarma: http://t.co/hfXgnAv70J http://t.co/nHEDQwbkDw
#Keyword hashtag	Message contains the hashtag #Ebola	1,618 (55.5)	Computer	A wealth of credible info on #ebola from @CDCgov @WHO & more all on one webpage thnx to @NLM_DIMRC http://t.co/L4SwRQu82s
@Mention	Message mentions another user (eg, @cdc)	713 (24.5)	Human (α = .81)	Find out key facts about the #Ebola virus & outbreak in @KaiserFamFound”s infographic with @JAMA current http://t.co/Ma2Uw2MboP
@Reply	Message begins with a mention, directing it to a particular user (eg, @cdc)	529 (18.1)	Human (α = .93)	@StateHealthIN has Ebola information available at http://t.co/ajJ4pZMXJg
Retweet request	Message contains a specific request for the user to retweet or RT	83 (2.8)	Human (α = .85)	October 20 Procedures for Personal Protective Equipment \| #Ebola Hemorrhagic Fever http://t.co/oey9J3WC3M Please retweet #PPE #CDC

Note: Reliability is measured by Krippendorff's alpha (α) and/or percent agreement (%).

Scientifically accurate messages. The emergency risk communication model calls for the development of scientifically accurate messages using a process of scientific review. Although the model does not identify specific content, which changes depending on the nature of the threat, previous research suggests that scientifically accurate messages should include threat information, or information about the hazard or threat that increases knowledge and contributes to sensemaking.²⁵ We, therefore, did not assess scientific accuracy per se; instead we assessed whether the message contained threat information. This type of information fills people's information needs resulting from the uncertainty created by these events and has the potential to increase adoption of official recommendations during a public health threat—a key long-term outcome in the emergency risk communication conceptual model.⁴

Actionable messages. Actionable messages describe specific protective actions individuals can take during a public health threat.^4,26 Within the public health literature, this type of message content is also called efficacy information.²⁷ This type of information—when it increases beliefs that individuals can perform the recommended action (self-efficacy) and affirms that the recommended action will provide protection (response efficacy)—increases the likelihood that individuals will respond to the threat by taking appropriate actions.²⁸

Open and transparent messages. Open and transparent messages provide candid information and have the potential to enhance trust in officials.⁴ Intrinsic message features that contribute to open and transparent messages include providing information about official actions. These “official action” messages describe actions that officials have taken in response to an event. These messages increase trust,²⁹ essentially building the credibility of officials by showing constituents how officials are responding. Credibility enhances the believability and effectiveness of messages from officials³⁰ and has been shown to increase message compliance in both experimental³¹ and observational studies.³² It is also a mid-term outcome in the emergency risk communication model.⁴

Another message strategy that can increase the openness and transparency of messages is the use of first person (ie, “I,” “we”). Writing in the first person to refer to an organization can increase the sense of a human presence in organizational communication³³ and increase trust and identification.³⁴

Clear messages. Public health threats often require officials to communicate complicated public health processes, such as contact tracing or the mode of transmission for an emerging infectious disease. Using clear messages increases the likelihood they will be understood.⁴ Intrinsic message features that have the potential to enhance message clarity include images, figurative language, and the second person.

Images are visual representations of messages. They are theorized to increase message clarity because they provide an additional route for information processing.³⁵ Using images with text messages has been found to increase message recall,³⁶ increase attention to messages,^20,37 and promote message comprehension.³⁸

Figurative language is a group of linguistic strategies, called tropes, that compare 2 different objects in order to make 1 object more understandable. These message strategies include metaphors, similes, and personification.³⁹ During the 2014 Ebola crisis, Ebola was compared to a deadly animal that could be stopped “in its tracks” and personified as wimpy. Although tropes may be extended metaphors with multiple points of comparison,³⁹ they may also be 1-word metaphors. In these cases, a single word creates the dissonance central to metaphoric language.⁴⁰

Writing in the second person (ie, “you”) contributes to message clarity by increasing self-referencing, the process through which individuals relate information to themselves.^41,42 Personalizing a message is a necessary step in the process that leads individuals to act on warning messages.⁴³

Additional Message Strategies Relevant to Reach

Social media has unique features that enable official communicators to provide quick access to information (through hyperlinks), label messages as part of a larger conversation (through keyword hashtags), reply to particular messages, and mention specific user accounts. In some cases, these strategies may relate to goals stated in the emergency risk communication conceptual model. For example, hyperlinks provide quick access to additional information on a website and have the potential to increase message sufficiency, a psychological outcome in the model that relates to the amount of information a person needs about a particular risk.⁴

In other cases, these features are part of the conversational nature of social media. For example, keyword hashtags distribute messages to a sender's followers and to a hashtag, where the messages can be viewed by accounts that do not follow the sender. Hashtags, which emerged on Twitter and have been adopted by other platforms, allow individuals to label their messages (using #keyword) and thereby contribute to larger online conversations about a topic and engage a bigger audience. A mention occurs when a message contains @username anywhere in the message—except at the beginning. At the time of this study, a tweet beginning with @username functions as a reply and is shared only with the mutual followers of the sender and receiver. As a result, sending a reply reduced the size of the potential audience.

In addition to using channel-specific message strategies to indicate the potential audience for a message, officials can also ask audiences members to share a message and, as a result, disseminate the message through additional networks. This strategy, called a retweet request, has been successful in other contexts.^19,44

Establishing Intercoder Reliability

After finalizing the coding scheme, the first and second authors participated in a brief training session and then separately coded 200 randomly selected tweets. Krippendorff's alpha was used to measure the reliability of the manual coding, using the SPSS macro created by Hayes and Krippendorff.^45,46 For the majority of the message feature codes, Krippendorff's alpha was high, .7 or greater.⁴⁶ In 1 case (figurative language), the data were not distributed equally among categories, so percentage agreement was used instead. A percentage agreement of 95% or higher was deemed acceptable. Krippendorff's alpha assumes that distribution among categories is relatively even. In cases with uneven distribution, the measure may underestimate reliability.

Coding Procedures

Following the establishment of intercoder reliability, messages were coded in Microsoft Excel. Computer-driven coding was employed for codes that consisted of a symbol (eg, @) or specific words (eg, I, we, we're). Codes that required interpretation were manually coded.

Additional Factors Influencing Message Passing

To account for other factors that influence message passing on social media,⁶ we included several additional variables in the analysis. Sender characteristics were operationalized as the account itself and the governmental level of the account (local, state, federal). Network characteristics were operationalized as the number of network followers. Messages and account information, including follower numbers were gathered daily at 24-hour intervals. As a result, the exact follower counts between intervals is not known. We estimated the follower count growth between intervals using a negative binomial model, with the time the tweet was produced and the number of followers the account had as fixed effects. The resulting variable was skewed with high kurtosis (mean [M] = 173,925, standard deviation [SD] = 436,874, skewness = 2.76, kurtosis = 6.13). For analysis, we transformed the follower count variable using the natural log function in IBM SPSS Statistics for Windows version 25.0 (IBM Corp, Armonk, NY), (M = 8.85, SD = 2.62, skewness = 0.69, kurtosis = -0.33).

The message timing was operationalized as the day the message was sent. We added this to the model using daily fixed effects. To identify whether daily fixed effects corresponded to key events in the public health threat, we created a timeline of Ebola-related events in the United States (Figure 2).

Figure 2.

This figure shows daily fixed effects for message passing. The study began on October 1 (the point of comparison for the effects displayed above), 1 day after the first patient was diagnosed with Ebola in the United States. Spikes in message passing appear to be associated with key events in the Ebola crisis, suggesting that the relevance of the issue influenced the passing of public health messages. For example, a drastic increase in message passing is associated with the first case of local transmission. Abbreviations: NYC, New York City; US, United States.

Analysis

We modeled the relationship between message features and message passing using negative binomial regression without zero inflation. The outcome variable, message passing, was operationalized as the number of times a message was retweeted (M = 18.74, SD = 71.89, kurtosis = 180.94, skewness = 10.68). Because this variable consisted of count data and was overdispersed and highly skewed, we evaluated negative binomial models with and without zero inflation and Poisson models with and without zero inflation.⁴⁷ We used sample-size-corrected Akaike's information criterion⁴⁸ and out-of-sample prediction or cross-validation⁴⁹ to identify both the model choice and variable selection. The reported model, a negative binomial model without zero inflation, minimized squared error in 10% of held-out data over 300 trials, in addition to minimizing Akaike's information criterion. Analyses were conducted in the R statistical software⁵⁰ using MASS and glmmADMB packages.^51,52 We included a random intercept for each account and fixed effects for all other variables. (Models with accounts as fixed effects did not yield better model fit than the accounts as random effects.)

Results

During the modeling process, 2 variables—the keyword hashtag and use of second person—did not significantly predict message passing. The keyword hashtag variable was discarded, as model fit statistics indicated the model fit better without it. However, the second person variable improved model fit and was retained. In the following analysis, we interpret the results using a frequency interpretation of the results based on the incident rate ratios followed by the beta and significance.²¹ Full model results are reported in Table 2.

Table 2.

Multilevel Negative Binomial Regression Model Predicting Message Passing

Predictor	β	SE	P Value^a	IRR	95% CI
(Intercept)	-5.26	0.47	<.001	-	-
Scientifically accurate information
Threat information	.40	0.06	<.001	1.49	1.33-1.68
Actionable messages
Efficacy information	.40	0.15	.009	1.48	1.10-1.98
Open and transparent messages
Official action	.23	0.06	<.001	1.26	1.13-1.42
First person	.13	0.07	.039	1.14	1.00-1.30
Clear messages
Image	.84	0.09	<.001	2.32	1.96-2.74
Figurative language	.23	0.12	.049	1.26	1.00-1.59
Second person	-.03	0.08	.671	0.97	0.83-1.13
Additional strategies
Hyperlink	.13	0.06	.020	1.14	1.02-1.28
Mention	-.26	0.06	<.001	0.77	0.68-0.87
Reply	-2.39	0.08	<.001	0.09	0.08-0.11
Retweet request	1.33	0.19	<.001	3.78	2.63-5.44
Account and network characteristics
State government	.91	0.22	<.001	2.50	1.62-3.85
Federal government	.78	0.32	.014	2.19	1.18-4.08
Followers (log)	.62	0.06	<.001	1.86	1.64-2.11
Time-related effects^b
October 5	1.05	0.25	<.001	2.85	1.75-4.65
October 6	.40	0.19	.029	1.50	1.04-2.15
October 12	1.11	0.24	<.001	3.03	1.88-4.89
October 15	.58	0.17	<.001	1.79	1.30-2.48
October 16	.48	0.16	.002	1.62	1.18-2.21
October 17	.57	0.15	<.001	1.77	1.31-2.38

P values below .001 reported as <.001.

Only days with significant effects are reported due to space constraints. Model statistics for the other days are available from the first author.

Note: The results predicted the number of times a message from a public health agency was retweeted. The data consisted of 2,915 Twitter messages sent between October 1 and 29, 2014, from 236 public health agencies that included the keyword “Ebola.” Model includes the account sending the message as a random effect. All other effects are fixed.

Abbreviations: CI, confidence interval; IRR, incident rate ratio; SE, standard error.

In general, the message features identified by the emergency risk communication model increased message passing. Messages that contained threat information, the feature related to scientifically accurate messages, were retweeted 49% more often than tweets that did not include threat information (beta [β] = .40, P < .001 ), holding all other predictors constant. Messages that contained efficacy information, the feature related to actionable messages, were retweeted 48% more often than tweets that did not include efficacy information (β = .40, P = .009). Message features related to open and transparent messages also increased message passing. Messages that contained information about official action were retweeted 26% more often (β = .23, P < .001), and messages that used the first person were retweeted 14% more often (β = .13, P = .039).

Message features that increase clarity also increased message passing, with the exception of using the second person. Messages that contained images were passed on 132% more often than messages that did not contain images, holding all other variables constant (β = .84, P < .001), and messages that used figurative language were retweeted 26% more often (β = .23, P = .049. Using the second person did not influence message passing (β = -.03, P = .671).

Conversational message features specific to the social media channel both increased and decreased message passing. Using hyperlinks and requesting a retweet increased message passing. Messages that contained hyperlinks were passed on 14% more often (β = .13, P = .02), and messages that contained a request to retweet were passed on 278% more often (β = 1.33, P < .001). However, messages that contained a mention or a reply (@username) decreased message passing (mentions: 23%, β = -.26, P < .001; replies: 91%, β = -2.39, P < .001).

User account features also influenced message passing. Having a larger number of followers predicted higher rates of retweeting (β = .62, P < .001). Message passing was also predicted by the account itself and the governmental level of the account. Compared to local accounts, both state (β = .91, P < .001) and federal (β = .78, P = .014) accounts exhibited higher rates of message passing. Message passing also varied widely in relation to the timing of the messages, as operationalized by daily fixed effects (Figure 2). Some days did not have a significant effect on message passing, compared to the first day of the study (October 1). On other days, message passing increased by more than 200% (eg, on October 12; β = 1.11, P < .001), holding all other variables constant. The comparison of daily fixed effects to events in the Ebola crisis showed that, in general, days with higher rates of message passing occurred near the time of key events during the health crisis.

Discussion

These results provide empirical evidence that the intrinsic message features described in the emergency risk communication conceptual model influence key short- and mid-term outcomes on social media. These findings demonstrate that scientifically accurate messages (operationalized as containing threat information), open and transparent messages (operationalized as containing official action and use of the first person), and clear messages (operationalized as containing images and use of figurative language) influence audience behaviors, as these message strategies were associated with higher levels of message passing. Message passing signals higher levels of engagement by audience members¹¹ and increases exposure to messages on social media. These short-term outcomes have been shown in previous research to lead to the long-term psychological and behavioral effects identified in the emergency risk communication model.¹⁵

In addition to providing empirical support for the emergency risk communication model, this study also describes a method for evaluating the success of emergency risk communication on social media in the future. Given the difficulty of evaluating emergency risk communication in general,⁴ this method provides a starting point for officials to assess message strategies and make improvements during an event that involves emergency risk communication. It also provides a strategy for evaluating messages after the event and identifying intrinsic message features that contribute to message success. Although the presence of these features does not ensure that a message is more persuasive, it does encourage messages to reach larger audiences, increasing the likelihood that more people would be persuaded.¹⁶

Findings from this study also support the importance of timing in the emergency risk communication conceptual model,⁴ and clarify what timeliness means for social media messages. In this study, messages sent near the time of key events in the Ebola crisis were passed on more often than messages sent at other times. This finding, that the day the message was sent influenced message passing, suggests that topic relevance plays an influential role in message passing. These results are significant not because they are surprising (they are not) but because they provide support for our intuitions about the life cycle of a message on the internet. Future research should continue to investigate this phenomenon as it adds to the research suggesting that risk messages are more relevant at key moments during a public health crisis.⁵³ Although the emergency risk communication model does not consider the role of misinformation in the communication environment, its presence in public health debates has been documented.⁵⁴ Sending messages in a timely fashion and implementing the message strategies described in this article has the potential to increase the reach of information from officials during a crisis and, in doing so, may increase the ability of public health officials to combat misinformation.

Implications for Practice and Theory

For those who write social media messages during a public health crisis, this study identified a set of specific message strategies that should be implemented in future crises. In particular, the results suggest that public health practitioners should create messages that illustrate information visually, as this practice results in the largest increase in message sharing. In addition, they should try to include threat and efficacy information in messages. They should also emphasize how officials are responding. These elements can be used in combination or separately; however, combining elements has an additive effect in terms of message passing, and, therefore, officials should design messages to include more than 1 of these strategies whenever possible.

These message strategies are not new; previous research has established the importance of threat and efficacy information and the need to tell people what officials are doing during a crisis. Yet many messages in this study did not contain these strategies: less than half (41.2%) of the messages contained threat information, less than one-third (27.8%) contained information about official action, and only a handful (2.3%) contained efficacy information.

Practitioners should also use features such as replies sparingly, as this feature decreases message passing. This study is not the first one to find that this channel-specific message feature has this effect.⁹ This may be because a reply signifies that the sender expects the message to be relevant to a narrow audience, or perhaps usernames reduce the clarity of a message as they are often cryptic abbreviations. For example, the username for the San Francisco Department of Public Health is @SF_DPH.

Finally, the findings from this study highlight the need for public health communicators to grow and engage social media audiences before public health crises emerge. Like previous studies,^8,18 this study demonstrated a clear link between follower numbers and message passing: Messages sent by accounts with larger follower numbers exhibited increased retweeting. A growing literature on social media engagement offers some guidance on message strategies that public health organizations could use, both before a public health threat emerges and, in the case of a long-lasting public health threat like novel coronavirus diseases 2019, to continue engaging followers throughout a crisis.⁵⁵ This growing body of research suggests that engagement is both a function of message structure and content.^56,57 In other words, public health officials can use the same strategies identified in this study (using images and providing threat and efficacy information) to engage and build audiences between crises.

In terms of theory, the results from this partial test of the emergency risk communication model suggests that, in order to be empirically tested, the emergency risk communication conceptual model may need to be adjusted to account for channel- and platform-specific characteristics that influence message outcomes. Currently, the model does not consider how messages may need to differ as they are disseminated across platforms and channels.

Limitations and Areas for Future Research

This study has high external validity as it evaluates behavioral reactions to messages sent by public health agencies during an event involving emergency risk communication. However, we cannot rule out the presence of unmeasured mechanisms influencing those reactions. In addition, because of the structure of the data, we do not know who is passing these messages on. Previous research suggests that 41% of followers of local health department Twitter accounts are associated with individuals rather than organizations.⁵⁸ A more recent study found that 46% of retweets related to a CDC smoking cessation campaign came from accounts associated with individuals.⁵⁹ Together, these studies suggest that public health organizations are reaching individuals on Twitter; however, we do not know whether the messages we analyzed created any changes in the mid- and long-term outcomes identified in the emergency risk communication model. In other words, we cannot tell whether these messages influenced individuals to change their behavior. Existing research suggests that individuals have many different motivations for retweeting, including showing approval, arguing, gaining attention, and entertaining.⁶⁰ Future research using different methods is needed to assess motivations for retweeting in the context of public health crises.

Since the data for this study was collected, social media has changed, both at the platform and the channel level. Twitter in particular added more ways to interact with messages, such as quote tweeting, and implemented an algorithm that changed how users saw messages. At the time of the study, messages appeared in a user's Twitter “feed” in chronological order, with the most recent messages appearing first. In February 2016, Twitter began using an algorithm, similar to other social media platforms, like Facebook, that prioritizes certain messages, based on undisclosed factors related to the sender, the message, and the relationship between the sending account and the receiving account. Although the effect of the algorithm changes on public health messages are unknown, research conducted after these changes showed similar results to this study,⁶ suggesting that the algorithm may amplify factors already at work in the social media environment.

In addition to platform-specific changes on Twitter, new social media platforms have emerged (eg, TikTok), and social media audiences have continued to grow since the data for this study was collected.²² Growing numbers of public health leaders have also recognized social media as a tool for community engagement, and local health departments have continued to expand their use of social media.⁶¹ Although not every public health audience can be reached through this platform, researchers and public health organizations should continue to evaluate social media messages and identify evidence-based best practices to inform public health communication on social media, especially during crises. Future research should also take into account the value of social media messaging, given the cost of staff time and other resources to produce effective messages.

Conclusion

This study demonstrated how to assess emergency risk messages on social media by short-term outcomes on social media, providing a starting point for evaluating emergency risk communication in the future. Because of the difficulty of evaluating emergency risk communication, findings from social media may inform the development of messages on other channels.

Future research should continue examining message passing on social media during events that require emergency risk communication and seek to identify other message features related to key constructs in the emergency risk communication model. In doing so, these studies could contribute to a lexicon of social media message strategies that public health communicators could turn to during a public health emergency.

Footnotes

Acknowledgments

We thank Editor-in-Chief Thomas Inglesby and the anonymous reviewers for their helpful review of the manuscript. In addition, we thank the University of Kentucky College of Public Health's Office of Scientific Writing for its assistance in revising this manuscript. This research was support by grants from the National Science Foundation (CMMI-1536347; CMMI-1536319). Some of the initial research for this project was completed as part of the first author's dissertation. The analyses presented in this paper are new.

References

Fraustino

, Liu

, Jin

. Social media use during disasters: a research synthesis and road map. In: Austin

, Jin

, eds. Social Media and Crisis Communication. New York: Routledge;. 2018:283-295.

Dalrymple

, Young

, Tully

. “Facts, not fear”: negotiating uncertainty on social media during the 2014 Ebola Crisis. Sci Commun. 2016; 38(4):442-467.

Wong

, Harris

, Staub

, Bernhardt

. Local health departments tweeting about Ebola: characteristics and messaging. J Public Health Manag Pract. 2017; 23(2):e16-e24.

Seeger

, Pechta

, Price

, et al. A conceptual model for evaluating emergency risk communication in public health. Health Secur. 2018; 16(3):193-203.

Tang

, Bie

, Park

, Zhi

. Social media and outbreaks of emerging infectious diseases: a systematic review of literature. Am J Infect Control. 2018; 46(9):962-972.

Vos

, Sutton

, Yu

, et al. Retweeting risk communication: the role of threat and efficacy. Risk Anal. 2018; 38(12):2580-2598.

O'Keefe

DJ.

Message properties, mediating states, and manipulation checks: claims, evidence, and data analysis in experimental persuasive message effects research. Commun Theory. 2003; 13(3):251-274.

Sutton

, Gibson

, Spiro

, League

, Fitzhugh

, Butts

. What it takes to get passed on: message content, style, and structure as predictors of retransmission in the Boston Marathon Bombing response. PLoS One. 2015; 10(8):e0134452.

Sutton

, Gibson

, Phillips

, et al. Terse message retransmission: a cross-hazard analysis of Twitter warning messages. PNAS. 2015; 112(48):14793-14798.

10.

Sutton

, Spiro

, Johnson

, Fitzhugh

, Gibson

, Butts

. Warning tweets: serial transmission of messages during the warning phase of a disaster event. Inf Commun Soc. 2013; 17:765-787.

11.

Neiger

, Thackeray

, Burton

, Giraud-Carrier

, Fagen

. Evaluating social media's capacity to develop engaged audiences in health promotion settings: use of Twitter metrics as a case study. Health Promot Pract. 2013; 14(2):157-162.

12.

Park

, Oh

, Park

, et al. The source and credibility of colorectal cancer information on Twitter. Medicine (Baltimore). 2016; 95(7):e2775.

13.

Vos

, Sutton

, Gibson

, Butts

. Celebrity cancer on Twitter: mapping a novel opportunity for cancer prevention. Cancer Control. 2019; 26(1):1073274819825826.

14.

O'Keefe

. The relative persuasiveness of different message types does not vary as a function of the persuasive outcome assessed: evidence from 29 meta-analyses of 2,062 effect sizes for 13 message variations. In: Cohen

, ed. Communication Yearbook 37. New York: Routledge;. 2013:221-249.

15.

Hornik

. Public health communication: making sense of contradictory evidence. In: Hornik

, ed. Public Health Communication: Evidence for Behavior Change. Mahwah, NJ: Lawrence Erlbaum Associates;. 2002:1-19.

16.

McGuire

. McGuire's classic input-output framework for constructing persuasvie messages. In: Rice

, Atkin

, eds. Public Communication Campaigns. 4th ed. Los Angeles: Sage;. 2013:133-146.

17.

Kim

, Hou

, Han

, Himelboim

. Predicting retweeting behavior on breast cancer social networks: network and content characteristics. J Health Commun. 2016; 21(4):479-486.

18.

Suh

, Hong

, Pirolli

, Chi

. Want to be retweeted? Large scale analytics on factors impacting retweet in Twitter network. Proceedings of the, 2010 IEEE Second International Conference on Social Computing. Los Alamitos, CA: The Institute of Electrical and Electronics Engineers;2010:177-184. Accessed September 21, 2020. https://ieeexplore.ieee.org/document/5590452

19.

Tan

, Lee

, Pang

The effect of working on message propagation: topic- and author-controlled natural experiments on Twitter. Preprint. arXiv:1405.1438. Submitted May 6, 2014. Accessed October 27,2020. https://arxiv.org/abs/1405.1438

20.

Bail

CA.

Emotional feedback and the viral spread of social media messages about autism spectrum disorders. Am J Public Health. 2016; 106(7):1173-1180.

21.

, Lee

, Goh

DH-L

. Sharing in social news websites: examining the influence of news attributes and news sharers. Proceedings of the Ninth International Conference on Information Technology: New Generations (ITNG). Los Alamitos, CA: The Institute of Electrical and Electronics Engineers; 2012:726-731. Accessed September 21, 2020. https://ieeexplore.ieee.org/document/6209115

22.

Pew Research Center. Social media fact sheet. Published June 12, 2019. Accessed September 21, 2020. https://www.pewresearch.org/internet/fact-sheet/social-media/

23.

Towers

, Afzal

, Bernal

, et al. Mass media and the contagion of fear: the case of Ebola in America. PLoS One. 2015; 10(6):e0129179.

24.

Harris

, Mueller

, Snider

. Social media adoption in local health departments nationwide. Am J Public Health. 2013; 103(9):1700-1707.

25.

Veil

, Reynolds

, Sellnow

, Seeger

. CERC as a theoretical framework for research and practice. Health Promot Pract. 2008; 9(suppl 4):26S-34S.

26.

Coombs

. Conceptualizing crisis communication. In: Heath

, O' Hair

, eds. Handbook of Risk and Crisis Communication. New York: Routledge;. 2009:99-118.

27.

Bandura

Self-efficacy determinants of anticipated fears and calamities. J Pers Soc Psychol. 1983; 45(2):464-469.

28.

Witte

Putting the fear back into fear appeals: the extended parallel process model. Commun Monogr. 1992; 59(4):329-349.

29.

Reynolds

, Seeger

. Crisis and Emergency Risk Communication: 2014 Edition. Atlanta, GA; US Centers for Disease Control and Prevention; 2014. Accessed September 21, 2020. https://emergency.cdc.gov/cerc/resources/pdf/cerc_2014edition.pdf

30.

Reynolds

, Seeger

. Crisis and emergency risk communication as an integrative model. J Health Commun. 2005; 10(1):43-55.

31.

De Meulenaer

, De Pelsmacker

, Dens

. Power distance, uncertainty avoidance, and the effects of source credibility on health risk message compliance. Health Commun. 2018; 33(3):291-298.

32.

Freimuth

, Musa

, Hilyard

, Quinn

, Kim

. Trust during the early stages of the 2009 H1N1 pandemic. J Health Commun. 2014; 19(3):321-339.

33.

Park

, Cameron

. Keeping it real: exploring the roles of conversational human voice and source credibility in crisis communication via blogs. J Mass Commun Q. 2014; 91(3):487-507.

34.

Yang

, Kang

, Johnson

. Effects of narratives, openness to dialogic communication, and credibility on engagement in crisis communication through organizational blogs. Commun Res. 2010; 37(4):473-497.

35.

Paivio

Dual coding theory: retrospect and current status. Can J Psychol. 1991; 45(3):255-287.

36.

Clark

, Paivio

. Dual coding theory and education. Educ Psychol Rev. 1991; 3(3):149-210.

37.

Noar

, Hall

, Francis

, Ribisl

, Pepper

, Brewer

. Pictorial cigarette pack warnings: a meta-analysis of experimental studies. Tob Control. 2016; 25(3):341-354.

38.

Chang

Seeing is believing: the direct and contingent influence of pictures in health promotion advertising. Health Commun. 2013; 28(8):822-834.

39.

Sopory

, Dillard

. The persuasive effects of metaphor. Hum Commun Res. 2002; 28(3):382-419.

40.

McQuarrie

, Mick

. Figures of rhetoric in advertising language. J Consum Res. 1996; 22(4):424-438.

41.

Burnkrant

, Unnava

. Effects of self-referencing on persuasion. J Consum Res. 1995; 22(1):17-26.

42.

Meyers-Levy

, Peracchio

. Moderators of the impact of self-reference on persuasion. J Consum Res. 1996; 22(4):408-423.

43.

Mileti

, Sorensen

. Communication of Emergency Public Warnings: A Social Science Perspective and State-of-the-Art Assessment. Report No. ORNL-6609. Oak Ridge, TN: Oak Ridge National Lab; 1990. Accessed September 21, 2020. https://www.osti.gov/biblio/6137387

44.

Malhotra

, Malhotra

, See

. How to get your messages retweeted. MIT Sloan Manage Rev. 2012; 53(2):61-66.

45.

Hayes

, Krippendorff

. Answering the call for a standard reliability measure for coding data. Commun Methods Meas. 2007; 1(1):77-89.

46.

Krippendorff

Reliability in content analysis: some common misconceptions and recommendations. Hum Commun Res. 2004; 30(3):411-430.

47.

Hilbe

JM.

Negative Binomial Regression. 2nd ed. Cambridge, UK: Cambridge University Press; 2011.

48.

Hurvich

, Tsai

. Regression and time series model selection in small samples. Biometrika. 1989; 76(2):297-307.

49.

Geisser

Predictive Inference: An Introduction. New York: Chapman and Hall; 1993.

50.

R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2018.

51.

Bolker

, Skaug

, Magnusson

, Nielsen

. glmmADMB: generalized linear mixed models with AD Model Builder. R package version 0.8.8.3. GitHub; 2017. Accessed September 21, 2020. https://github.com/bbolker/glmmadmb

52.

Venables

, Ripley

. Modern Applied Statistics with S. 4th ed. New York: Springer; 2002.

53.

Southwell

, Dolina

, Jimenez-Magdaleno

, Squiers

, Kelly

. Zika virus-related news coverage and online behavior, United States, Guatemala, and Brazil. Emerg Infect Dis. 2016; 22(7):1320-1321.

54.

Broniatowski

, Jamison

, Qi

, et al. Weaponized health communication: twitter bots and Russian trolls amplify the vaccine debate. Am J Public Health. 2018; 108(10):1378-1384.

55.

Olson

, Sutton

, Vos

, Prestley

, Renshaw

, Butts

. Build community before the storm: the National Weather Service's social media engagement. J Contingencies Crisis Manage. 2019; 27(4):359-373.

56.

Lovejoy

, Waters

, Saxton

. Engaging stakeholders through Twitter: how nonprofit organizations are getting more out of 140 characters or less. Public Relat Rev. 2012; 38(2):313-318.

57.

Dhanesh

GS.

Putting engagement in its PRoper place: state of the field, definition and model of engagement in public relations. Public Relat Rev. 2017; 43(5):925-933.

58.

Harris

, Choucair

, Maier

, Jolani

, Bernhardt

. Are public health organizations tweeting to the choir? understanding local health department Twitter followership. J Med Internet Res. 2014; 16(2):e31.

59.

Chung

JE.

A smoking cessation campaign on Twitter: understanding the use of Twitter and identifying major players in a health campaign. J Health Commun. 2016; 21(5):517-526.

60.

Majmundar

, Allem

, Boley Cruz

, Unger

. The Why We Retweet scale. PLOS One. 2018; 13(10):e0206076.

61.

Miller

, Snook

, Yoder

. Social media in public health departments: a vital component of community engagement. J Public Health Manag Pract. 2020; 26(1):94-96.