Abstract
Audiences, who cannot investigate the credibility of most news stories for themselves, rely on noncontent heuristic cues to form credibility judgments. For most media, these heuristics were stable over time. Emerging formats of journalism, however, require audiences to learn to interpret what new heuristics credibility cues mean about the story’s credibility. In an experiment, participants (N = 254) were given instructions about how to interpret the credibility cues in three formats as they read a politicized news story, which were compared with a control condition with no instructions. Results show the timing and source increase the effectiveness of the instructions.
The newly launched Colorado Sun website embedded in news stories a box of credibility indicators. In this box, readers can click to learn more about whether the story is original, on-the-ground reporting; whether the reporters are subject specialists; and what sources are cited in the article (“About Us,” 2019). The funding organization for the Sun, Civil, cited the difficulties audiences have with media literacy as a major reason for the credibility indicator box (Himmel, 2018). As public discourse in the wake of the 2016 presidential election turned to the role of hoax news stories that spread across social media sites such as Facebook, audiences said they were having difficulty telling the difference between credible news and hoax news. A late-2016 Pew survey found 64% of American adults said “completely made up” news was causing them a great deal of confusion about current events and facts (Barthel, Mitchell, & Holcomb, 2016). Furthermore, about 23% of those same adults said they had shared a political news story that was fabricated, whether they knew at the time it was made-up or learned later.
To bring audiences up to speed, several news organizations responded with “How to” articles on spotting “fake” news: The Globe and Mail offered a list of tips such as paying attention to news by trusted news brands and URLs (Annett, 2017). The Associated Press gave audiences a list of five steps that included clicking the link and then reading the story, and paying attention to grammar and emoticons (Ortutary, 2016). Facebook (2017), for its part, listed 10 heuristics to which to look when determining credibility. Similarly, the News Literacy Project distributed tips for as part as a news literacy program for adoption in classrooms across the country, teaching students what to be wary of when reading news online (Foley, 2017).
The appeal of heuristics as a checklist to aid in media literacy worked when the news media environment was stable, but the continued evolution of online news design makes the usefulness of any one list ephemeral. Marsh and Yang (2017) said teaching audiences to focus on source evaluation as a form of media literacy will not work because “typical cues for credibility have been hijacked” to blur the lines between news that is malicious and news that aims to tell the truth. At the same time, the sheer magnitude of information available from innumerable news brands means newsreaders must rely on heuristics and social cues to assess information from unfamiliar news organizations (Lazer et al., 2017). Of particular concern is media literacy skills surrounding news about politics. Political news can trigger a defensive response in audiences, which makes them resistant to updating their beliefs (Festinger, 1957; Lowin, 1967). That reaction has recently been applied to the tendency of audiences to consider news that challenges their worldview as “fake.”
Together, these concerns expose three gaps in the scholarly literature of vital importance for today’s news literacy and participation in democracy. First, how can news organizations teach audiences to understand and interpret the credibility heuristics available to them? Second, how does teaching these heuristics to audiences affect their perception of news credibility of political news? Finally, how does the partisanship of news audiences aid or impede their ability to learn credibility heuristic cues on political news? There is little research on how adults learn best to tell the credibility of news in the face of changing technology and the expanding availability of news. One exception is a report from the Center for Media Engagement, which found that adding a box next to news stories that explains the reporting process can increase reader trust in that story (Chen, Curry, & Whipple, 2019). Additional understanding about learning credibility heuristics comes from child development literature that observes children as they develop a sense of interpersonal credibility. But, adults must continue to develop a system of determining information credibility as the ways they access news change.
The purpose of this study is to test several easy-to-implement cognitive interventions to teach audiences how to understand and interpret credibility heuristic cues, and how ideology plays a part in learning these cues about partisan news. In a survey-embedded experiment, it looks at three formats of interventions in particular: a pop-up before the news story provided by the news site itself, a sidebar next to the story with the same information as the pop-up, and a sidebar next to the story with tips provided by other users. These formats are compared with a control condition, in which participants were provided no additional information about how to interpret the credibility heuristics.
Literature Review
News Credibility
How audiences perceive the credibility of the news they read is the subject of more than six decades of research, which began with Hovland and Weiss (1951), because it is vital to understanding how citizens make decisions in a self-governing society. Although the practices, routines, and conventions of journalists are designed to cultivate credibility (Coddington, 2014; Tuchman, 1972), the decision about what to believe is ultimately in the eyes of the audience (Gunther, 1992). Although logical, analytical, and careful consideration of the argument are the most prized types of message processing when making credibility judgments, people tend to be cognitive misers to make more efficient decisions (Fiske & Taylor, 1991). This means that audiences will process no more cues than they determine is needed to make a relatively risk-free decision about the credibility of news (Sundar, Knoblach-Westerwick, & Hastall, 2007). Pjesivac, Geidner, and Cameron (2018) found modern Internet news readers primarily use the peripheral processing route, paying attention to heuristics such as source expertise rather than the content of news messages. Yet, it seems, when audiences encounter obviously not-credible information, they may more systematically process the news by spending more time with it and demonstrating better recall of it than credible information (Geidner & Cameron, 2017).
Heuristic credibility systems change each time a different news delivery system emerges. Because when audiences first find news on an unfamiliar medium, the modality, design of the interaction, and the story structure are different from past experiences with news media (Kiousis, 2001; Lee, Park, Lee, & Cameron, 2010; Sundar, 2008), they must adjust to the heuristic cues provided by these mediums if they are to reap the benefits of the medium such as convenience and quick delivery to audiences. Transitioning to a new medium causes confusion and disorientation (Bucy, 2003; McDonald & Stevenson, 1996), and audiences are less able to identify cues that were once important to determining credibility such as message source (Vraga, Edgerly, & Wang, 2011). This has been a particular concern as newspapers move online and make changes in the visual design of news (Li, 2002; Lowrey, 1999).
Although the connections between systems of heuristic credibility cues and the perception of credibility seem obvious, much of the scholarly literature focuses on a checklist approach to researching how cues affect the perception of credibility. Studies isolate heuristics such as news brand names (Metzger, 2007; Sundar & Nass, 2001), source credibility (Pjesivac et al., 2018), platform publisher (Bhandari, 2018), number of social recommendations (Xu, 2013), or website features (Flanagin & Metzger, 2007) to find the effects of cues on the perception of credibility. But, little research looks at how knowledge about the interpretation of these heuristics is acquired, and the role that partisanship—which is known to create knowledge gaps in coordination with news media use (Hindman, 2009)—might play in the ability of the audience to learn and interpret these cues. The suggestion that audiences can learn news credibility heuristics comes from two areas of the scholarly literature.
Learning to Judge Credibility
First, theory can draw on early childhood literature to offer suggestions that credibility cues can be learned. Flanagin, Metzger, and Hartsell (2010) found in a survey of school-aged children that the more the students were exposed to Internet information, the more they were able to identify the importance of using analytical cues to weed the false information from the true information. The students, though, reported they more often relied on heuristics such as “gut reaction” or “feelings.” Birch, Akmal, and Frampton (2010) demonstrated that 2- and 3-year-olds could learn credibility cues. When only one adult tried to teach them, with or without nonverbal credibility cues, the children learned equally. But, when they saw more than one adult teaching, some with credibility cues and some without, the children showed preference and learned more from the adults who demonstrated credibility cues. More sophisticated cue use in interpersonal communication, such as a motivation to be biased toward a belief, may not develop until middle school (Mills, 2013).
Second, survey evidence suggests that audiences adjust to these systems over time. The more people use a news medium, the more they consider it credible (Kiousis, 2001). Cross-sectional data suggest that as news technology becomes more prevalent and more familiar, the perception of credibility of news on that technology increases (Jacobson, 1969; Johnson & Kaye, 2004; Mackay & Lowrey, 2011). Audiences, however, are able to develop platform-specific heuristic cue sets (Chung, Nam, & Stefanone, 2012). For example, modern audiences are judging political news on social media by different criteria than they judge political news found on blogs, news websites, or broadcast television (Johnson & Kaye, 2014) and continue to find blogs less credible than traditional news (Bhandari, 2018). The evidence that audiences can learn cues to credibility and that, as they grow familiar with the cues, the perception of credibility increases suggest a positive correlation between the interpretation of credibility cues and the perception of credibility.
Partisanship and Credibility Judgments
Although the scholarly literature suggests that audiences should be able to learn credibility cues, partisanship can also influence news media literacy outcomes. Vraga and Tully (2015) showed partisan audiences a clip of a television talk show about immigration reform in which the host took a liberal, conservative, or neutral position. With half of the audience, they intervened with a public service announcement (PSA) promoting the importance of media literacy before the clip ran. They found that conservatives who saw the PSA before the conservative-host clip rated the host and the program as more biased. That perception of bias, though, did not alter their perception of credibility. Liberals who saw the PSA questioned their own motives and so saw the conservative host as neutral. Where news media literacy is taught, Democrats are more likely to increase their belief in the value of news media literacy (Tully & Vraga, 2017). Similarly, Chen et al. (2019) were able to improve the audience’s perception of credibility of a news story by placing an “Explain the Process” box on both mock and real news stories. Yet, the box did not reduce the audience’s perception that the news organization “had an agenda.” Furthermore, the authors found a similar addition to news stories designed to reduce the perception of partisan bias in the news article was not effective enough for the authors to recommend adoption by newspapers (Chen et al., 2019).
Partisanship may present a barrier to learning credibility heuristics and updating beliefs because of cognitive processes that create a tendency to interpret the world in a way that favors maintaining the status quo. Partisan audiences have the tendency to believe information when it confirms their previously existing beliefs (Festinger, 1957; Lowin, 1967). But beyond swaying opinions, the partisan heuristic about a politicized issue can change the belief about factual information. This pattern of “selective learning” (Jerit & Barabas, 2012) shows what partisans tell pollsters about objective facts such as crime rates are filtered through their political worldview. Barabas and Jerit (2009) and Jerit and Barabas (2012) found that news media coverage of a politicized issue exacerbated the knowledge gap along partisan lines. The more news media report on an issue, the more likely people report that they know more facts that help their party or hurt the opposing party in comparison with facts that would be detrimental to their party (Jerit & Barabas, 2012). Partisans are able to dismiss attitude-discrepant information sources that have a lower credibility more easily than attitude-discrepant information from sources with high credibility.
When it comes to politicized news issues, credibility also plays a part in the information choices of partisans. Westerwick, Kleinman, and Knobloch-Westerwick (2013) in selective exposure experiments found when personal relevance was low, users spent more time with high-credibility sources. But, when personal relevance was high, they favored attitude-discrepant messages from low-credibility sources compared with high-credibility sources. This suggests the audience will take extra steps to avoid attitude-inconsistent information when it is an important issue, even taking the risk of being misinformed by a low-credibility source.
Timing and Credibility
The tendency of audiences to choose news and to believe ideas that agree with their political view is well established. Explanations about what can increase the audience’s willingness to read, consider, or believe an opposing viewpoint is less clear. Two interventions that seem to increase the perception of credibility while simultaneously suppressing the effect of partisan cues emerge from the scholarly literature: the timing of the information given to the audience and positive social recommendations. Although the role of these factors is not directly addressed in news credibility literature, the theories and circumstances of these effects are similar enough that they may provide guidance to how audiences process information meant to teach them how to interpret credibility heuristics.
First, evidence suggests that the timing of credibility cues will matter to a skeptical audience. Sternthal, Dholakia, and Leavitt (1978) first found that when evaluating the persuasive ability of consumer messages, the message source’s degree of credibility had an effect only when it was provided before the message. Where an audience member was negatively predisposed to a message and the source of the message was revealed after the consumer saw the message, whether the source was high or low credibility made no difference on the persuasiveness of the message. Similarly, Vraga and Tully (2015) found an effect of a media literacy PSA on partisan audiences when it was placed before a clip of a political talk television show.
Tormala, Briñol, and Petty (2007) expanded these findings by thoroughly examining the role of timing of the identification of a message source in persuasive messages. In an experiment, participants were told the source of a message about a chemical in soap detergent was either the government (high credibility) or a manufacturer of detergent (low credibility). The source of the message was revealed to the participants either before the message or after the message. When the source was revealed before the message, the credibility manipulation influenced the valence of thoughts so that high-credibility sources invoked more favorable thoughts. When the source of the message was revealed after the message, the credibility of the source had no effects on the audience’s thoughts about the message.
These studies suggest that when a high-credibility source of a message is revealed to the audience, it can change how the audience processes the message so that members are more likely to view the message favorably. Tormala et al. (2007) suggest this effect is observed because the audience is predisposed to believe the message and hence spends less cognitive effort on examining the content of the message. This evidence suggests the timing of source credibility information matters to the audience’s perception of credibility. This study posits, similarly, that the timing of information about how to interpret the credibility cues about a news story will have an effect on the audience’s perception as well.
Social Recommendations and Credibility
The second factor, social recommendations, is an emerging line of literature that reflects the gaining popularity of “crowdsourced” heuristics made possible by interactive Internet technology. Social recommendations—the suggestions of nonelites through individual expressions or aggregated opinions—were shown to be powerful heuristic cues in several studies. Metzger, Flanagin, and Medders (2010) found in focus groups that audiences relied on the recommendations of others to make credibility decisions about monetary transaction websites, such as Amazon or Paypal: Applied to the context of online credibility judgments, when choosing between sources, people are likely to believe that a source whose name they recognize is more credible compared to unfamiliar sources. People appear to reason that name recognition is earned by positive interactions overtime that are spread through social networks. (p. 426)
Furthermore, they found that when others recommended or endorsed a website, participants put more faith in that heuristic than their own firsthand experience with a website: “Regardless of their previous opinion on a product, some participants noted that they often used comments and testimonials to direct their purchase decisions” (Metzger et al., 2010, p. 428). In an experiment, Xu (2013) found that increasing the number of social recommendations on a social networking platform increases both the likelihood of clicking on a news story and the perceived credibility of the news. This study found that as the number of “upvotes” a Reddit story received increased, the audience’s perception of credibility of the story increased.
Because these studies find that social recommendation is a powerful heuristic in determining online credibility, this study hypothesizes that instructions on how to interpret news credibility heuristics provided through social recommendations will positively affect the audience evaluations of the message.
Method
These hypotheses were tested using a mock mobile news application. This context was used because it is an emerging medium for news that continuously changes. These circumstances mean that the absence of familiar credibility heuristics would not communicate a negative credibility and that the audience would not have any preexisting knowledge about how to interpret interpretation credibility heuristics. The stimulus, therefore, manipulated the source and timing of the instructions about how to interpret the credibility heuristics in an environment similar to what happens when news technology creates major shifts in the availability of news credibility cues.
Adults residing in United States (N = 254) were recruited by Survey Sampling International and paid incentives by the service for participating. The participants were mostly White (81%; 8% Asian or Pacific Islander; 7% African American or Black; 7% Hispanic). A slight majority of participants (55%) were men. The average age of participants was 47 years. The participants identified their party affiliation as 31% Democratic, 45% percent Independent, and 24% Republican. This study received approval through the researcher’s institutional review board.
Participants contacted by Survey Sampling International (SSI) were told that they were helping to evaluate a mobile news application before it was available to the public. This application, participants were told, employed an algorithm to assess the credibility of news aggregated from multiple news sources and assign it credibility “grades.” After answering a few pretest questions about media use, opinions about current news topics, and attitudes about technology use in news delivery, the survey software assigned participants to one of the four conditions. All participants saw the same news story with the same credibility heuristics. Depending on condition, the instructions about how to interpret the credibility heuristics were provided in a column to the right of the story or on a screen before the news story. Those in the control condition were not given any instructions on interpreting the cues. Participants were able to view the experimental stimulus on Internet-connected devices, including smartphones, tablets, and laptop or desktop computers, simulating many mobile applications that are available for different sizes of screens. Participants were able to zoom in to see any detail if they wanted (see the appendix).
After reading the news story, participants answered questions about how they interpreted the credibility heuristics and perceived the credibility of the news story. After several demographic and sociographic questions, participants were debriefed about the use of deception in this experiment.
Stimulus
The news story used in the experiment was written by a former journalist to be an unbiased news story about a politicized issue. It followed journalistic conventions and included information from both sides of the issue. The decision to use a mock news article was made to increase internal validity on two levels: First, this ensures that the audience had no prior experience with the issue that could bias their perceptions. Second, the news app was designed to have a slightly futuristic look as to provide the audience with an unfamiliar heuristic environment and replicate conditions where participants would need to learn to judge credibility.
The topic of crime was chosen because crime/policing was politicized during the 2016 presidential campaigns. A Pew Research Center survey found that perceptions of crime rates differed by support of candidate (Gramlich, 2016). While 37% of those who supported Hillary Clinton’s bid for the presidency said that crime in the United States worsened since 2008, 78% of those who supported Donald Trump felt the same. The same report noted that Federal Bureau of Investigation (FBI) statistics showed crime decreased about 20% during the same time period. Therefore, it was an appropriately partisan issue when this experiment was fielded in March 2017.
The news story made visible three specific design elements that were indicators of credibility. This means to interpret the credibility heuristics of the news story, the audience had to learn their meanings. However, these cues are ones not currently used in other news environments. For example, the story gave a number of “points.” The points had no context on the layout of the page, but the instructions explained that the points indicated how reliable the news story was on a scale of 1 to 200.
The experiment provided instructions on their interpretations in three of the four conditions. The fourth condition, the control condition (n = 65), provided the audience no way to assess or interpret these three design elements. Although this condition did not provide any information on how to interpret the cues, the cues were visible. This decision was made to ensure it was the participants’ ability to learn the meaning of the cues that made a difference in their perception of credibility rather than the presence of this design element. To operationalize the factors that the scholarly literature suggests would improve the ability to learn credibility heuristics in an externally valid way, audiences viewed three conditions in a controlled survey-embedded experiment. To examine the role of timing, audiences were shown the exact same wording for the instructions in two conditions. In the “site tips” condition (n = 64), instructions for interpreting the credibility heuristics were provided in a sidebar that was shown to the audience at the same time as the news story.
In the “pop-up” condition (n = 66), instructions were provided to the audience on a screen where they could only read instructions. Next, they saw the same story as those in the control condition. An introduction sentence in these conditions suggested that the source of the instructions was the site itself through pronoun usage.
To examine the role of social recommendations, the “user tips” condition (n = 59) featured tips about how to interpret the credibility heuristics, which were displayed in a sidebar titled “What Readers Are Saying.” Short, tweet-length statements appeared in this column. Each tip was labeled with a username. In addition, the pronoun usage in these tips was altered to indicate that the source of the tips was individual readers.
Variable Construction
The perception of credibility was measured through six questions with an answer scale of 1 to 5, in which larger numbers indicate higher perceived credibility. These items were adapted from Meyer (1988), which has been used to measure audience perceptions of news message credibility in emerging online news contexts (Kiousis, 2001). Participants rated the news story on the degree they thought the story was interesting, fair, accurate, believable, and credible. They also rated the degree to which they thought the story showed bias in favor of lawmakers and in favor of law enforcement. The bias questions were reverse coded and averaged to create one bias item. The Cronbach’s alpha for the six items was .734, indicating a reliable index. Then, the six items were averaged to create the credibility index (M = 3.55, SD = 0.59).
After the participants read the news story and instructions, all participants, including those in the control condition, were asked to recall the interpretation of three credibility heuristics. Participants were given one point for a correctly identified heuristic and zero points for wrong answers and do-not-know answers (M = 1.09, SD = 1.06).
Participants were asked to rate their political party partisanship on a scale from 1 to 7, where 1 was strong Democrat and 7 was strong Republican (M = 3.82, SD = 1.85). Participants who identified as strong, moderate, or leaning Democratic were grouped as Democrats (n = 78). Those who identified as strong, moderate, or leaning Republican were grouped as Republicans (n = 59). Those who identified as independent were so categorized (n = 113).
Results
To check how well the manipulation of the source of the instructions for the credibility heuristics worked, participants were asked to recall the source. In the three treatment conditions, the correct answer was the most common answer (site tips, n = 16; user-tips, n = 10; pop-up, n = 25). A Pearson’s chi-square shows an association between the two categories, χ2(9, 105) = 23.37, p = .005.
To check whether the manipulation of the format of credibility instructions changed the degree of scrutiny participants gave the content of the news story, the participants were asked two recall questions about the content. Participants were given one point for a correct answer and zero points for an incorrect answer or a “don’t know” answer. A one-way analysis of variance (ANOVA) of those content recall questions by condition found no significant difference, F(3, 253) = 1.317, p = .269.
To check that participants interpreted the credibility cues correctly, a Pearson correlation between the number of cues learned and the perceived credibility was computed. The results indicate a significant positive correlation of r = .182, p = .004. This suggests that as the audience is able to identify more credibility heuristics correctly, it perceives the credibility of the news story as higher.
To examine the effect of the intervention condition, an ANOVA was executed. Results suggest that condition had a main effect on the number of cues learned, F(3, 249) = 10.98, p < .001. Pairwise comparisons with Bonferroni correction show that those in the pop-up condition learned more credibility cues than those in the control condition (mean difference [MD] = 0.97, SE = 0.19), p < .001, and more than those in the site tips condition (MD = 0.57, SE = 0.18), p = .012. Furthermore, those in the user-tips condition learned more than those in the control condition (MD = 0.83, SE = 0.2), p < .001. There was no main effect nor an interaction of participant partisanship on the number of cues learned (see Figure 1). These results support
Effect of Condition and Participant Partisanship on Number of Cues Learned
Effect of Condition and Participant Partisanship on Perceived Credibility
Conclusion
This study used simple interventions in the form of instructions to teach participants the meaning of heuristic credibility cues, mimicking the conditions the audiences would face when they encounter news in an emerging journalism format. In a controlled experiment, these manipulations tested an unusual set of circumstances—one where the audience had no familiarity with the credibility heuristics but that unfamiliarity did not communicate a lack of credibility. In this context, the results presented here were able to complement the scholarly literature that finds that, over time, audiences learn to adjust to new credibility heuristics to learn to distinguish news with high credibility from low credibility. In addition, because the scholarly literature suggests that both learning and the perception of credibility are influenced by partisanship, audiences were given a story written with a balanced viewpoint about a politicized news topic to test the role of partisanship in the perception of credibility in the absence of the usual markers of credibility.
The results of this study have important implications for theory and for newsrooms that are looking for ways to increase the audience’s ability to determine the veracity of political news. These results separated the influence of preexisting knowledge about credibility heuristics (or actual increases in the credibility of news on a new format) from the effect on the perception of credibility. The increase in identified credibility cues cannot be attributed to more familiarity with the news or more familiarity with politics.
These results also suggest that two formats for instructions work best to teach audiences how to interpret credibility heuristic cues. First, audiences responded best to instructions that were given to the audience before the politicized news. Second, audiences learn the meaning of more credibility heuristics when the source of the tips is other users. These findings comport with prior research that audiences respond well to social recommendations and that timing matters when processing source credibility information. Furthermore, although partisanship is known to interfere with learning facts, these results showed no evidence that partisanship would prevent differing levels of knowledge about the meaning of credibility cues. With these results, news literacy advocates can take care to provide explicit instructions on how to interpret news credibility heuristics either through a pop-up message before the political news or through crowdsourced tips. Furthermore, research could investigate whether user-generated tips provided through a message presented before the audience reads the news can even further increase the ability of the audience to learn credibility heuristics. All of these formats for instructions were tested here in a controlled setting contrived by the research design. Developing and testing these formats on everyday audiences will be key to knowing how effective these types of interventions will be on mass audiences.
Although these findings offer key insights, like all experiments, this study has its limitations. First, for ecological validity, the design and wording of the user-generated tips were not the same as the site tips nor the pop-up tips. For this reason, the user-tips cannot be directly compared to either, although effort was made to explain the same cues in the same way. Second, while it makes sense that no participant in the control condition correctly answered all three questions about the meaning of the credibility cues—they were given no information to be able to learn what the heuristics on the page meant, after all—this makes statistical comparisons between those who did answer all three correctly in the treatment conditions and the control condition impossible. Finally, the manipulation check to the source of the tips shows about half of the participants could not correctly recall who was providing the instructions about how to interpret the credibility heuristics. Although a chi-square shows the participants did pay some attention to the source, and although this attention likely replicates the attention audiences pay during the normal course of reading, the full effect of the credibility heuristic instructions may not be observed in this study. However, the results of this manipulation check also may be similar to real-world news attention. Mahone (2014) found similar numbers of participants were unable to remember the source of a news story in an experiment.
Nevertheless, this study has two key findings for mass communication scholars and media literacy advocates: (a) Crowdsourced information about the meaning of credibility heuristics increases the number of credibility heuristics audiences learn and (b) the timing of the information makes a difference, as well. Instructions about how to interpret credibility cues before a politicized news story increase the number of cues learned compared with when the information is displayed simultaneously to the news story.
