Creating News Literacy Messages to Enhance Expert Corrections of Misinformation on Twitter

Misinformation and Its Correction

A set of best practices for addressing misinformation—both on social media and in other spaces—has emerged encouraging succinct and repeated corrections of misinformation to reduce associated misperceptions (Lewandowsky et al., 2012, 2017). Nonpartisan experts, such as fact-checking organizations and journalistic outlets, serve as effective agents to correct misinformation, although success can depend on features of the correction, including use of humor or visual stimuli (Amazeen et al., 2018; Garrett et al., 2013; Nyhan & Reifler, 2010; Thorson, 2016; Wood & Porter, 2019; Young et al., 2018).

But the news is not entirely good. Corrections can also lead people to accept correct information without changing their attitudes toward the underlying target, creating “belief echoes” (Thorson, 2016; see also Nyhan, Porter, Reifler, & Wood, 2019). Likewise, fact checks may be limited in their reach. Significant selective exposure occurs for viewing and sharing of fact checks (Hameleers & van der Meer, 2019; Shin & Thorson, 2017), meaning they may not reach individuals who are most exposed to misinformation (Guess et al., 2018).

Another potential response to misinformation online puts the burden on platforms, individual users, and experts to correct misinformation. In the realm of social media, conditions are ripe for observational correction—which occurs “when social media users update their own attitudes after witnessing another user being corrected” (Vraga & Bode, 2017, p. 14). Essentially, users decrease their misperceptions when they see (and presumably accept) a correction to misinformation on social media. Importantly, this is true even for those who are misinformed on the issue initially. They may engage in confirmation bias (Taber & Lodge, 2006) and evaluate the correction poorly as a result (Bode & Vraga, 2015), but viewing the correction still decreases misperceptions. Observational correction has been documented when the correction comes from experts like the Centers for Disease Control (Vraga & Bode, 2017), from social media platforms using recommendation algorithms or flagging false stories (Bode & Vraga, 2015; Clayton et al., 2019; Pennycook & Rand, 2017), and from other users, provided that the correction comes from multiple users providing a credible source (Margolin et al., 2018; Vraga & Bode, 2018).

In this study, we expand upon existing work regarding expert correction of misinformation. Past research has emphasized the effectiveness of expert correction (Guillory & Geraci, 2013; Lewandowsky et al., 2012; Vraga & Bode, 2017), often using nonpartisan fact-checking organizations (Amazeen et al., 2018; Garrett et al., 2013; Nyhan & Reifler, 2010). This study further tests who can serve as a corrective agent on social media, considering both a nonpartisan think tank (Study 1) and a nongovernmental medical organization (Study 2). Given past research which finds positive effects of correction from expert organizations, we expect expert correction to continue to reduce misperceptions.

News Literacy

Despite calls to improve media literacy as a method to limit susceptibility to misinformation, little research has tested whether media literacy messages designed to prompt critical processing of news and information interact with misinformation and its correction on social media (Clayton et al., 2019; Craft et al., 2017; Kahne & Bowyer, 2017). We focus on NL because it promotes becoming more mindful news consumers who understand complex media environments (Ashley et al., 2017; Mihailidis & Viotty, 2017; Vraga & Tully, 2015). NL encompasses the knowledge and skills that audiences need to analyze media messages, to distinguish news and high-quality content from other kinds of content—including misinformation, ads, and opinion—and to apply this literacy when consuming news and other media (Ashley et al., 2017; Craft et al., 2017; Potter, 2004).

Typically, NL efforts explore the relationship between journalists, news production, citizens, and democracy in classroom settings or through online courses (Potter, 2019). These efforts are generally limited to those in educational settings, divorcing NL from spaces, like social media, where misinformation spreads (Bulger & Davison, 2018; Marwick & Lewis, 2017). In order for NL to influence news consumption, audiences must develop knowledge and skills and put them to use when actually engaging with media (Potter, 2004).

However, research suggests that people may be familiar with NL concepts but fail to apply them when consuming news (Craft et al., 2016; Tully, Vraga, & Smithson, 2018), highlighting the need to incorporate these reminder messages into online spaces where many get their news and information (Clayton et al., 2019; Tully et al., 2019). These reminders should encourage people to act “in a media literate manner” when engaging with news and (mis)information on social media (Potter, 2004, p. 61).

Inoculation research has found that warning people about misleading tactics can neutralize the effects of misinformation messages (J. Cook et al., 2017; van der Linden et al., 2017). Likewise, NL messages can serve as reinforcement to encourage mindful news consumption (Tully et al., 2019). In contrast, Clayton and colleagues (2019) found that general warnings with a NL style message produced lower accuracy ratings for all content in the feed, rather than helping individuals distinguish between accurate and inaccurate stories and did not make tagging of specific stories as “false” or “disputed” more effective. However, the focus on political content (rather than health and science), as well as the dominance of false news in their study (six out of nine headlines) may explain the inability of NL messages to function effectively in this case.

Research has found that exposure to NL education and the knowledge it produces may generate skepticism toward misinformation and political conspiracy theories, even when these theories match subjects’ political orientations (Craft et al., 2017; Kahne & Bowyer, 2017). Kahne and Bowyer (2017) found that people with greater media literacy were more likely to rate high-quality posts as more accurate than misinformation posts. Therefore, exposure to NL messages should have two outcomes. First, such messages should encourage people to be more skeptical of low-quality information, such as misinformation that lacks a credible source. Second, they should improve reception of high-quality information, such as corrections from expert sources, by prompting people to consider the quality of the source and information. Given the role of source credibility in promoting and rebutting misinformation (Guillory & Geraci, 2013; Walter & Tukachinsky, 2019), recognizing the differing credibility of the misinformation source (low credibility) and correction source (high credibility) should amplify corrective effects.

This study examines the potential for NL messages to enable a more critical response to low-quality misinformation and expert correction on Twitter. But given the space and time constraints of social media (e.g., character limits), NL messages designed to be consumed as part of regular media consumption cannot cover all aspects of NL. These messages need to be tailored for online audiences and contexts to encourage receptiveness to correction, which may depend on which aspects of NL are promoted in the message (Clayton et al., 2019; Tully et al., 2019). Research has shown that short NL videos are effective at conveying NL concepts to diverse audiences and at promoting political engagement, but distilling these concepts into compelling messages on social media may be challenging.

Recognizing the personal nature of social media and its role as a news source, the NL tweets in this study emphasize the responsibility of audiences to be critical news consumers. In Study 1, we adapted validated NL concepts from existing literature to create NL tweets. Given the focus on audience behavior, we chose a message that highlights how personal views or biases influence news choices and interpretation (Klurfeld & Schneider, 2014; Vraga & Tully, 2016).

In one tweet, we used second-person address (“your job”), and in the other, we used third person (“citizens’ job”) to test if the more personal framing (you) or the more citizen-oriented framing would be more effective in this persuasive context (G. Cook, 2001). The “you/your” address is ubiquitous in advertising and is used to present a more familiar or personal style, as compared to a more formal style (G. Cook, 2001; Machin & van Leeuwen, 2005). However, the third-person address of “citizen” is relevant in this case, as it connects news and information choices to democracy and may remind audiences that news plays a critical role in informing self-governing citizens (Mihailidis, 2014; Mindich, 2005). Therefore, the rationale for comparing these forms of address is twofold: (1) given the focus on the role of personal biases in news consumption, we might expect the “your job” tweet to more directly speak to news consumers, as it addresses them directly; (2) however, the allusion to democracy in the “citizens” tweet may make people more receptive to a message that connects NL and news consumption to the greater social good associated with democracy. Both assumptions are supported by work that shows NL messages can effectively convey core concepts that contribute to different outcomes and that persuasive messaging uses both forms of address (G. Cook, 2001; Vraga & Tully, 2016).

In Study 2, we adapted a message from an existing tweet from a well-known media literacy organization (the News Literacy Project). This tweet uses the popularized phrase “fake news” to draw attention to the problem and provides a set of concrete actions that people can take to identify content that does not meet these standards. This tweet has a clearer “call to action” than the tweets in Study 1 but has not been validated by previous research.

Across both studies, we maintain several aspects of the NL message. First, both highlight audiences and individual responsibility, including the hashtag #DoYourPart as a call to action. Second, both are designed to encourage individuals to distinguish high- from low-quality content by recognizing how their own biases influence news choice and interpretation (Study 1) or by looking for concrete indicators in the content itself (Study 2).

Together, research suggests that NL efforts can promote recognition of both low- and high-quality news and information thus leading to differential assessments of that content. With this in mind, we propose that NL messages will enhance expert correction—a post from an expert source providing credible evidence—making it more effective at combating misinformation—a post from an unknown Twitter user providing no evidence (a meme) or a link to a low-quality source (a source that Wikipedia (2019) classifies as a “fake news” website).

The Moderating Role of Misperceptions

In considering possible responses to misinformation, it is important to recognize the role that personal beliefs—specifically how misinformed people are to begin with—may play in receptiveness to corrective efforts. Research on how initial misperceptions affect success of corrective efforts is mixed. Some research shows that motivated reasoning leads people to resist updating their attitudes in light of new conflicting information, resulting in less correction among those with higher misperceptions, for whom the correction is counter-attitudinal (Garrett et al., 2013; Garrett & Weeks, 2013; Nyhan & Reifler, 2015; Taber & Lodge, 2006; Walter & Tukachinsky, 2019). Other research suggests such resistance can be overcome and finds greater effects of correction among those who hold higher misperceptions on the issue (Bode & Vraga, 2015; Hameleers & van der Meer, 2019; Nyhan et al., 2013; Vraga & Bode, 2017). Several explanations exist for these stronger effects. Functionally, there may be more room to move for those who are initially misinformed—that is, those who hold largely correct attitudes are limited by how much they can update their attitudes in the correct direction. Theoretically, motivated reasoning only operates to a “tipping point”; if sufficient counter-attitudinal evidence is presented, people will adjust their attitudes (Festinger, 1957; Redlawsk et al., 2010). Such a process may be especially likely when counter-attitudinal corrections occur immediately after misinformation (Walter & Tukachinsky, 2019), before such misinformation becomes ingrained. As the literature is divided on the issue, we ask,

Given the relative dearth of research on the intersection between NL message and expert corrections on social media (Clayton et al., 2019; Tully et al., 2019), it is unclear whether initial misperceptions would also condition response to NL messages. If motivated reasoning is particularly strong—as would be expected when initial misperceptions are high (Garrett & Weeks, 2013)—people may extend these processes to the NL message, reducing its effectiveness. Indeed, previous research has found that an NL video paired with a political talk show host advancing an incongruent political argument increases the odds that people will rate the NL message as biased (Tully & Vraga, 2017). In contrast, however, Craft and colleagues (2017) found that knowledge of news structures—a key component of NL—reduced endorsement of politically congruent conspiracy beliefs, suggesting effectiveness of NL messages regardless of individual predispositions. Given the competing expectations, we ask,

Study 1 Methods

To test these expectations, we performed an online experiment in September 2017 (N = 1,207) and February 2018 (N = 603) with participants from Amazon’s Mechanical Turk, providing a diverse but not representative sample of the U.S. population (Clifford & Jerit, 2014; Levay et al., 2016; Necka et al., 2016). Our average participant was 36 years old (M = 35.80, SD = 10.86, min = 18, max = 80) and had a Bachelor’s degree; 50% were female; participants were paid $1.00 after completing the 10-minute survey.

Data were collected at two time points to ensure adequate power. We performed a series of t-tests and chi-square tests to examine differences between the samples. The February sample was significantly more educated (3.40 vs. 3.21), wealthier (2.77 vs. 2.64), and more likely to have a Twitter account (60% vs. 53%). No differences in age, party affiliation, ideology, time spent on survey, or misperceptions about genetically modified organisms (GMOs) in the pretest were observed. We controlled for fielding date in all analyses.

For this study, we analyze a 2 (Correction: Misinformation-only vs. Pew correction) × 3 (Promoted tweet: Texting [control], Your job, Citizen job) experimental design. Three additional correction stimuli were not analyzed for this study. First, we had a control condition in which no misinformation on the topic of GMOs was included. Second, we had two additional correction formats: a condition where the response from Pew has 35 favorites rather than 0, and a condition where another Twitter user corrects the misinformation providing a link to the Pew Research Center and tagging the organization in their response, followed by the same response from Pew as in the other conditions. We eliminate these conditions for parity with Study 2.

In this study, all participants read a simulated Twitter feed with 6 tweets (Supplemental Appendix A); four previously validated control tweets on social and news topics (Vraga Bode, Smithson, & Troller-Renfree, 2016) and two manipulated tweets. Participants were required to spend 15 seconds on the stimuli page before the continue button appeared.

We manipulated two tweets for this experiment. Our first manipulation altered the first tweet on the feed. Participants saw a “promoted tweet,” the label Twitter uses for paid posts that appear in feeds whether or not the user follows the account, from either the Ad Council about the dangers of texting and driving, or from the Media Literacy Coalition (a fictitious group) reminding participants that it is either citizens’ job or your job to recognize how citizens’ (your) viewpoints influence news choices and evaluations, and encouraging critical news consumption. These NL tweets highlighted that personal biases shape news beliefs and empowered citizens to make informed news choices, which seems particularly relevant to the correction of misinformation, where preexisting attitudes often intersect with correction efforts (Bode & Vraga, 2015; Garrett & Weeks, 2013; Nyhan & Reifler, 2015; Vraga & Bode, 2017).

The second manipulation varied whether misinformation regarding the safety of GMOs was immediately corrected. In both conditions, a Twitter user posted that scientists “know” GMOs are not safe to eat and shared a visual claiming that most scientists say GMOs are UNSAFE overlaying a visual of a syringe and large words stating “NO GMO” (Supplemental Appendix A). In the Misinformation-only condition, there are no responses to this tweet. In the Correction condition, the Pew Research Center (a nonpartisan think tank) replied that 88% of American Association for the Advancement of Science (AAAS) scientists say GMOs are safe, including a link to the Pew website and a pie chart with 88% scientific agreement on GMO safety, applying best practices for communicating consensus (van der Linden et al., 2015). We used the 88% statistic from the Pew report. Estimates of scientific consensus on GMO safety differ slightly by study, but consistently show high levels of consensus. We selected a gender-neutral name and picture for the user posting the misinformation to mitigate the potential for gender biases to skew responses.

We limit analyses to participants in these four experimental conditions who passed an attention check in the posttest, which asked participants to select “somewhat disagree” if they were paying attention, N = 724 (n = 18 participants failed the attention check), and who spent at least 3 minutes on the survey (n = 5 removed). ¹

Notably, recall for the misinformation tweet was substantively higher than for the NL tweet (see Supplemental Appendix). We focus on the effects of the messages regardless of recall in the main document but include the effects among the “treated” in our supplemental appendices. We believe this is the more appropriate approach in this context for several reasons. First, we cannot distinguish why participants had low recall for the NL tweet. They may not have paid sufficient attention to the messages to encode their experience for recall, or they may not have identified the content of the tweet as being about NL—which would have quite different implications for information processing. Second, our approach increases the generalizability of the results for those considering an NL campaign on social media (that is, it examines effects on everyone who would see the tweet, not just those that recall it). Finally, research suggests that manipulation checks are not necessary when message variations are based on “intrinsic features” (O’Keefe, 2003, p. 251). The NL message is either in the feed or not, whether or not participants recall it.

Study 1 Measures

Study 1 Results

To test our hypotheses and research questions, we use Model 3 in PROCESS (see Supplemental Appendix for a visual representation of PROCESS models), version 3.3 (Hayes, 2017). We contrast correction versus misinformation as our independent variable, with initial estimates of scientific consensus and exposure to the NL tweets as moderators, to examine their impact on posttest perceptions of scientific consensus on GMOs, controlling for field date (Garrett et al., 2013), applying mean centering of the variables and a heteroscedasticity consistent standard error estimator (Field, 2017; Hayes, 2017).

We find partial support for our expectations. First, we find support for H1, as there is a main effect of correction—participants who get corrected have higher posttest perceptions of scientific consensus related to GM safety (b = 6.97, SE = 2.75, p = .01, lower limit confidence interval (LLCI) = 1.57, upper limit confidence interval (ULCI) = 12.37). Likewise, existing GMO estimates of consensus exert a main effect on posttest misperceptions (b = .59, SE = .07, p < .001, CI = [.45, .72]); as expected, those with higher initial estimates of consensus also have higher posttest estimates of scientific consensus on GMO safety.

However, per RQ1, the main effect of the correction on perceptions of consensus in the posttest is qualified by an interaction between exposure to the correction and initial GMO misperceptions (b = −.26, SE = .13, p = .05, CI = [−.52, .00]) (see Figure 1). We probe the interaction using PROCESS model 1 with posttest ratings of scientific consensus as the dependent variable and entering the promoted tweet conditions and field date as controls and using the pick-a-point approach for those with 30%, 50%, and 80% initial estimates of scientific consensus. Here, the interaction is significant, F(1, 712) = 18.28, p < .001, explaining an additional 2.3% of variance posttest estimates. The correction improves ratings of scientific consensus among those with low (b = 20.39, SE = 3.51, p < .001, CI = [13.50, 27.28]), moderate (b = 14.42, SE = 2.26, p < .001, CI = [9.98, 18.86]), and high initial estimates of scientific consensus regarding GMO safety (b = 5.46, SE = 1.27, p < .001, CI = [2.97, 7.95]) compared to the misinformation-only condition. Moreover, using the Johnson–Neyman technique suggests that the effects of correction are significant (p < .05) for all initial values below 88%—the number indicated in the correction as the scientific consensus on the issue. Therefore, it appears our correction was effective for all participants who underestimated scientific consensus after seeing the misinformation, but these effects are stronger among those with low initial estimates of scientific consensus on GMOs (that is, high initial misperceptions), in line with existing research on observational correction (Bode & Vraga, 2015; Vraga & Bode, 2017). However, while corrections significantly and substantially reduced misperceptions, they did not eradicate them, as all three groups (low, medium, and high) still reported significantly lower estimates of scientific consensus than presented in the correction.

OPEN IN VIEWER

Figure 1.

Effects of correction and initial estimates of scientific consensus on posttest perceptions of scientific consensus, Study 1 (control for NL message exposure, field date).

Next, we consider the role the NL tweets may play in this relationship, returning to Model 3 in PROCESS. First, we do not see a main effect of exposure to either the Your Job (b = −1.04, SE = 1.86, p = .58, CI = [−4.70, 2.62]) or Citizens’ Job (b = −1.20, SE = 1.83, p = .51, CI = [−4.80, 2.41]) tweet on posttest misperceptions. In addition, we find no evidence that exposure to the NL tweets moderate the relationship between correction and initial GMO misperceptions, per H2 and RQ2. Neither the omnibus test of the three-way interaction between NL tweets, initial misperceptions, and exposure to the correction was significant, F(2, 706) = .26, p = .77, nor were any of the three-way interactions or two-way interactions involving exposure to the NL tweets (see Supplemental Appendix). Thus, NL tweets do not appear to improve receptiveness to expert correction on the GMO issue.

Given low level of recall for the NL message in particular, we also test all analyses among only people with perfect recall for all manipulated messages. We report these results in the supplemental appendices. In no case do the NL messages produce a significant effect on misperceptions, even using a more conservative test focusing on the “treated” who recalled exposure to all manipulated messages. We consider these findings in the discussion.

Comparing Study 1 and Study 2

We used two studies to better test the boundaries of expert correction and to more rigorously test the possible moderating role of NL messages in this relationship. In Study 1, we found that while expert correction led to more accurate estimates of scientific consensus about GMO safety, the two NL tweets we examined did not make these corrections more effective, as we hypothesized. This first study offered a conservative test of the potential of NL message: we used an established issue (GMO safety) and the NL tweets—while drawn from previously validated work (Vraga & Tully, 2016)—were not tailored to misinformation nor to social media.

In Study 2, we attempted to strengthen the NL message and its relationship to the misinformation and correction. The NL tweet directly encourages audiences to “watch out” for misinformation by checking the source and considering emotional appeal. This message aligned with the misinformation presented in the study—an article commonly shared on social media (Felton, 2018) immediately before fielding the study that claimed the flu shot caused the deadly flu outbreak in the winter of 2018 (from a website rated as “fake” by Wikipedia). In contrast, a “meme” was shared to promote GMO misinformation in Study 1. Moreover, by testing an emerging misinformation story, as compared to the more established issue of GMO safety, we may facilitate correction (Bode & Vraga, 2015). As part of the new issue domain, the correction comes from a credible expert source—the American Medical Association (AMA)—highlighting the quality of the correction as compared to the misinformation. Together, these changes were designed to increase the relevance of the NL message to the misinformation and correction, providing a less conservative test of its impact.

Study 2 Methods

For Study 2, we recruited 1,214 participants from Amazon’s Mechanical Turk in February 2018, who were paid $1.00. Our average participant was 37 years old (M = 36.83, SD = 11.89, min = 18, max = 82) and had a Bachelor’s degree; 54% were female.

This study examines a 2 (Correction: Misinformation-only, AMA correction) × 2 (Promoted tweet: Texting [control], NL tweet) experimental design. The first tweet in the feed varied a control tweet on Texting and Driving (identical to Study 1) versus an NL tweet from the Media Literacy Coalition about how to spot fake news, adapted from a tweet by the News Literacy Project (see Supplemental Appendix A) and pretested for recall and credibility. ² The other tweets remained consistent, and participants were required to spend 15 seconds on the page. We do not analyze participants in a separate control condition, who were not exposed to misinformation about the flu vaccine (N = 406).

Second, we manipulated exposure to misinformation about the flu vaccine, via a false tweeted story from YourNewsWire.com claiming the flu vaccine causes the flu and is responsible for flu-related deaths. ³ In the correction condition, the AMA replied to this tweet saying the flu vaccine prevents the flu, offered a link to their website, and posted a picture reminding people to get the flu shot. We limit all analyses to those who passed the attention check (N = 774, n = 34 removed) and spent at least 3 minutes on the survey (n = 3 removed).

Study 2 Measures

Study 2 Results

We utilize the same analytical approach from Study 1 for Study 2, using Model 3 in PROCESS, version 3.3 (Hayes, 2017). First, we again observe a main effect of initial flu misperceptions (b = .89, SE = .02, p < .001, CI = [.85, .93]), such that those with higher initial flu misperceptions also have higher flu misperceptions in the posttest.

In contrast to Study 1, we find no main effect of correction impacting posttest misperceptions, (b = −.08, SE = .07, p = .22, CI = [−.22, .05]), but the interaction between correction and initial misperceptions is again significant, b = −.08, SE = .04, p = .05, CI = [−.15, .00] (Figure 2). We probe the interaction using PROCESS Model 1, with posttest flu misperceptions as the dependent variable and entering the promoted tweet condition as a control, which again shows a significant interaction, F(1,766) = 3.75, p = .05, explaining an additional 1.4% of variance. We examine those who “disagree” (m = 2), are “neutral” (m = 4), and “agree” (m = 6) that the flu vaccine causes the flu to probe the interaction. These analyses demonstrate that the effects of correction in reducing misperceptions that the flu vaccine causes the flu are observed among those who initially hold high misperceptions (b = −.30, SE = .15, p = .05, CI = [−.60, −.00]) on the flu issue, with marginal effects on those who were initially neutral (b = −.15, SE = .09, p = .09, CI = [−.31, .02]), but not among those low in initial misperceptions (b = .01, SE = .07, p = .90, CI = [−.12, .14]). The Johnson–Neyman technique suggests the effects of correction become significant at M = 5.42 (e.g., at least somewhat agree the flu vaccine causes the flu), and become marginal for people who are neutral on this statement (M = 4). These results reinforce that corrections only work among those who hold at least some level of misperceptions initially, with stronger effects for those with higher initial misperceptions.

In Study 2, we again find that exposure to the NL tweet has no impact on misperceptions. Not only is the main effect of exposure to the NL tweet not significant (b = .05, SE = .07, p = .43, CI = [−.08, .19]), it also does not condition the effects of the correction on posttest flu misperceptions, as it produces no significant omnibus interaction, F(1, 763) = .13, p = .72, nor are the two-way interactions significant.

OPEN IN VIEWER

Figure 2.

Effects of correction and initial flu misperceptions on posttest flu misperceptions, Study 2 (control for NL message exposure).

Discussion

Despite calls to correct misinformation on social media and to improve media literacy education to facilitate its identification and correction, this study suggests such efforts are complicated. While this study found additional evidence for observational correction from two different expert sources, it highlights the difficulty of crafting NL tweets that break through the clutter on social media, and the unlikelihood that they enhance the effectiveness of expert correction. Three different NL tweets across two controversial health issues showed little ability to enhance the effectiveness of correction by expert organizations.

There are several possible reasons that the NL tweets did not produce the expected results. First, these messages were lost in the noise of a busy social media feed—we found that roughly 60% of participants recalled seeing the NL tweet in Study 1 and Study 2—a much lower percentage than who reported seeing the texting and driving tweet (roughly 85% in both studies), the GMO tweet in Study 1 (94%), or the flu tweet in Study 2 (95%). However, the NL messages still do not intersect with the corrective messages when considering only those participants who recalled seeing all the manipulated posts in the feed (see supplemental appendices). Although these supplemental results are limited in power, they do suggest that attention is not the whole story; something is preventing NL messages from persuading the audience to be more receptive to corrective messages from high-quality sources.

Second, our results echo previous findings that general warning messages and forewarnings of misinformation—which are similar to the NL message used here—may be less effective than specific corrective responses and sometimes generate cynicism toward all information rather than helping people distinguish misinformation from other types of content (Clayton et al., 2019; Pennycook & Rand, 2017; Walter & Murphy, 2018).

These results suggest that existing NL messages—both content and tone—cannot simply be transferred to social media environments. While the first study employed previously validated NL concepts shown to effectively convey the message that citizens must move beyond their own biases in evaluating news (Vraga & Tully, 2016), such a message may not have been seen as applicable in this context, where the misinformation did not come from a news source but from a social media user posting a meme. The message in Study 2 was adapted from a tweet from a prominent NL organization providing tips on content cues that might signal that a particular post has misinformation (or “fake news”), but still failed to garner attention or improve the effectiveness of expert correction to misinformation shared from a website known for its dubious content (Wikipedia, 2019). Together, these results suggest that more effort is needed to create NL tweets and other social media interventions that work online. Drawing on “best practices” to gain attention on Twitter, including animated GIFs, videos, or visually interesting graphics could help engage audiences and convey key messages (Adornato, 2017). Furthermore, additional work needs to test how to turn attention into action prompting more critical news and information consumption.

Our results for expert correction were more promising. This study expands what we know about “expert” correction, demonstrating that nonpartisan think tanks and national health agencies can also serve as effective agents of correction. It also reinforces that expert correction appears to be most effective among groups who hold stronger misperceptions on the original issue (Vraga & Bode, 2017).

In Study 1, a corrective response from the Pew Research Center about the scientific consensus on GMO safety led individuals to update their attitudes regarding the scientific consensus on this topic. This effect was particularly pronounced among those who initially had the lowest estimates of scientific consensus—producing a 20-point increase in estimates of scientific agreement compared to the Misinformation-only condition. While their estimates (56%) remain lower than the actual consensus (88%), it does move them to recognizing that a majority of scientists believe GMOs are safe to consume. These results are echoed in our findings regarding the safety of the flu vaccine. A correction from the AMA reduced misperceptions that the flu vaccine could cause the flu, particularly among those who initially believed this to be true.

That the effects in both studies were stronger among those with higher initial misperceptions on the issue merits special attention. This speaks to an ongoing debate in the field about the moderating role of existing predisposition in response to correction. The results from this study align with existing research on observational correction on social media, which has found effects are stronger among those with higher misperceptions (Bode & Vraga, 2015; Vraga & Bode, 2017), in contrast to earlier research on direct correction which found stronger effects among those with low misperceptions (Garrett et al., 2013; Garrett & Weeks, 2013; Nyhan & Reifler, 2015). There may be something unique about social media—and specifically observing an interaction between an individual sharing misinformation and an expert correcting them—that lowers barriers to correction that cannot be easily replicated in other spaces or contexts. Future research should test this possibility, examining whether it is the observation of an exchange or the nature of the social media platform that may explain these contrasting effects.

Moreover, the AMA’s response did not only reduce misperceptions among those who initially believed the flu shot caused the flu but encouraged the entire sample to report greater agreement with their behavioral recommendation that everyone eligible should get the flu shot. The NL message produces a similar main effect, boosting agreement with this statement. We offer several possible explanations for these effects. First, the behavioral recommendation was reinforced both in text and in the visual, which may strengthen the message. Second, people reported agreement with the idea that everyone eligible should get the flu shot, without indicating their own willingness to get the shot. The correction may therefore have led people to say others should get the flu shot, without necessarily affecting their own intentions, echoing research into third-person effects (Davison, 1983). Third, the correction may highlight social norms regarding vaccination, which would also explain the main effect of the NL message boosting agreement.

To better understand the mechanisms underscoring these effects, future research should employ diverse methods to study observational correction. Eye-tracking studies could precisely capture what aspects of the message garner attention from audiences (King et al., 2019) and whether this attention leads to improved recognition and updated attitudes. In addition, eye tracking would allow us to observe the order in which people engage with content in the feed to test if the NL message differentially affects processing of information depending on if it is seen before or after the misinformation and correction (Walter & Murphy, 2018). Likewise, in-depth interviews or focus groups would allow people to discuss nuanced reactions to different components of these messages, offering insight into individual response.

This study is limited in several ways. Our sample, though diverse and likely as good or better than other convenience samples (Clifford & Jerit, 2014; Levay et al., 2016; Necka et al., 2016) and which has been shown to approximate national samples for misinformation (Wood & Porter, 2019), is not representative and is younger and better educated than the U.S. population (see Supplemental Appendix for a comparison). This sample may have different levels of misperceptions about GMOs or the flu vaccine than the population, but if higher education means fewer misperceptions on these issues, it may actually underestimate the effects of expert correction, which appear stronger among those with greater misperceptions. Moreover, we cannot test whether people were familiar with the precise misinformation or story we test. If participants had been exposed to the story previously—especially in Study 2, where we picked an existing misinformation story—they may experience a familiarity effect of the misinformation (Lewandowsky et al., 2012) and be less receptive to the NL message, which is unfamiliar, as compared to the misinformation. However, at least one meta-analysis suggests no differences between real-world and artificial contexts for correction (Walter & Tukachinsky, 2019). As the study is online, our sample may also have more experience with platforms such as Twitter, as well as greater NL. Given that the NL messages are designed to reinforce existing knowledge and encourage their application, the NL messages may be even less successful when tested among a population with fewer NL resources. Alternatively, such reminders may be more powerful among those for whom NL is less salient, a question that future research should test. The data are also generated from an artificial social media feed, which may limit external validity. Likewise, our manipulations lack social cues (e.g., favorites or re-tweets), which may impact the effects of the misinformation or corrections (e.g., Messing & Westwood, 2014), a question that we do not explore here.

Finally, there are a number of differences between Study 1 and Study 2, which limit our ability to parse which differences may cause differing effects. For example, we selected a prominent false news story regarding the flu vaccine on social media, which may have potentially affected people’s familiarity with the issue, and therefore their response to the misinformation; a proposition that we cannot test. Likewise, we cannot be certain whether familiarity or respect for Pew versus the AMA may influence our effects. However, the broad similarity of our effects—notably, the ability of expert correction to reduce misperceptions, especially among those who initially have higher misperceptions on the issue, and the lack of impact that exposure to an NL message has to boost this corrective potential—suggests that our findings hold across at least some contexts.

This study speaks to the complicated environment created by social media that can both promote and hinder the spread of misinformation. It validates the importance of experts immediately responding to misinformation on social media with correct information. Yet despite calls for improved media literacy education as one mechanism to address concerns about misinformation spreading on social media, this study identifies several obstacles involved in such efforts, including designing messages that resonate with audiences. Although creating memorable and actionable NL tweets may be difficult, we believe that reminding people to be critical information consumers, particularly “in the moment” when they are consuming content on social media and empowering experts to correct misinformation will help address an environment rife with opportunities for misinformation to thrive. This study represents an important step in moving toward that reality.

Supplemental Material

Supplemental Material