The effect of news labels on perceived credibility

Abstract

Two online experiments examined whether news labels attract reader attention and affect news story credibility. Both studies showed that labels garner little attention from newsreaders and do not influence perceptions of news story credibility. However, Study 2 demonstrated that a label that explained the type of news story produced better recall of labels and more accurate recommendations of what the label should be. Findings suggest that if labels act as cognitive heuristics, they are weak ones.

Keywords

Credibility experiment labels online news

Public trust in the news media has been steadily declining over the past four decades, reaching an all-time low in 2016 (Jones, 2018). Credibility is among the features that influence Americans’ trust in news organizations (Gallup, 2018). This study examines the potential influence of labeling digital news content so that readers can more easily differentiate between various types, such as opinion and news. Some news organizations have adopted labels like these on their websites (see Micek, 2017; Stead, 2017), although the use of online news labels is neither ubiquitous nor standardized (Iannucci and Adair, 2017). Labels can potentially clarify what type of story people are reading, and, as a result, readers will see the news as more credible. Little empirical work has tested whether labels work as intended. We sought to fill this void by answering two main questions: Do online newsreaders notice news labels? Do digital news labels affect credibility judgments?

Study 1 used an experiment that manipulated the labels on an online news story, not the story itself, to test whether a label alone affects perceptions of news story credibility. Study 2 built on this work by testing the effects of labels on credibility perceptions using a new type of label – which we called an in-story explainer label because it appeared within the story and explained the label (e.g. ‘An opinion piece advocates for ideas . . .’) rather than merely labeling the story (e.g. opinion, sponsored content, etc.). These studies show that online news labels get little notice from newsreaders and have limited influence on perceptions of story credibility. However, the in-story explainer label produced better recall of labels. Overall, our findings suggest that online news labels have limited potential for improving media credibility.

Cues and labeling news

Heuristic processing is one route in the heuristic-systematic model of information processing (Chaiken, 1980) that occurs when people evaluate messages but lack the ability or motivation to engage in effortful systematic processing. When processing heuristically, people rely on past knowledge or simple guidelines – cues or cognitive heuristics – to make quick assessments of information (Chaiken, 1980). Cues ‘operate by activating applicable cognitive schemas that, when deemed usable, are used to make subsequent judgments’ (Shah et al., 2009: 94). Sundar et al. (2007), for example, describe the name of the news outlet and the number of related articles about a topic as cues that influence what users think. They argue that users rely on cues or mental shortcuts to assess content. These quick assessments are particularly important in the context of digital news because people tend to skim content in order to deal with the plethora of available information (Holton and Chyi, 2011).

News labels function as cues that influence users’ perceptions of the content (Metzger and Flanagin, 2013; Sundar, 2008; Sundar et al., 2007). Labeling articles may cue digital news users by setting expectations for the type of content and laying a foundation for how the content should be interpreted. If, for instance, news users notice that an article is labeled ‘Sponsored’, they may anticipate that the sponsored content will be more akin to the persuasive nature of an advertisement than the informative nature of actual news. As a result, people may ascribe less credibility to the advertising-related article.

Driven by the potential of news labels to help restore trust in the news media, Iannucci and Adair (2017) assessed the prominence and practice of news label use across 25 local newspapers and 24 national news and opinion websites. They found that fewer than half of the organizations employed news labels and of those who did, 80% only used labels in the opinion section. The research identified inconsistencies in labeling both within and across news organizations. The research presented here takes the next step in assessing labels by examining whether readers notice news labels and if the labels affect credibility perceptions.

Previous work suggests that labels are not consistently noticed nor universally successful in informing users. Tewksbury et al. (2011) investigated the influence of labeling video news releases (VNRs), or news packages created to influence the public, on users’ recall and perceptions. Just under half (47.1%) of their respondents reported seeing the label, although when labels were present, participants were significantly more likely to name the sponsor of the content correctly than when the labels were absent. In a study of televised government censorship disclaimers, however, Newhagen (1994) did not find a significant relationship between the presence of a disclaimer label and participant recall; participants were correct in their assessments less than 30% of the time. A study examining native advertising labels found only 7% of respondents indicated that they understood that there was advertising on the story page, although the rate varied depending on the words used to label the content (Wojdynski and Evans, 2016). Relatedly, 40% of those given a tutorial about native advertising and then shown a piece of sponsored content believed that they had seen advertising on the news page. Only 25% of those shown a sponsored content label without a tutorial believed that they had seen advertising (Amazeen and Muddiman, 2018; see also Amazeen and Wojdynski, 2019).

Research suggests that only a minority of users notice or benefit from seeing labels. Yet these studies were primarily about labels signaling persuasive intent, which is only a small subset of the labels currently being adopted by news organizations. In our first study, we considered four types of labels, based on the four common types of stories: news, a story based on verified facts; analysis, an interpretation of news based on factual reporting; opinion, an article that advocates for ideas based on the author’s interpretation of facts and data; and sponsored content, a paid-for story that does not meet standards of impartial journalism (Curry and Stroud, 2019). Building on past work finding that few people notice new labels, we first investigate the following:

RQ1: How accurately will people recall digital news labels (news, opinion, analysis, sponsored content, or no labels)?

News credibility

News organizations’ decision to label content is connected to their drive to increase trust and credibility in their process and product (Iannucci and Adair, 2017). Many researchers have taken on the task of defining and measuring media credibility, which is commonly used in tandem – and at times interchangeably – with trustworthiness (see Kohring and Matthes, 2007). Included in the myriad definitions and measures of trust and credibility are notions of fairness, accuracy, and the separation of facts from fiction (Kohring and Matthes, 2007; Meyer, 1988; Yale et al., 2015). We rely on the definition of credibility from Johnson and Kaye (2014): ‘Credibility is a judgment made by the users based on a host of factors, including believability, accuracy, fairness, depth, trustworthiness, bias, completeness’ (p. 958). We focused specifically on the credibility people ascribe to particular news stories.

News labels should provide readers with information that helps them interpret the information presented and, thereby, affect the perceived credibility of the content. News labels also may cue readers’ notions of transparency or openness about the product. Native advertising, for example, has been associated with lower levels of credibility (Amazeen and Muddiman, 2018). Although the main effects of recognizing advertising on attitudes toward the content and the publisher are negative, the relationships become positive when transparency is included in the model. That is, recognizing content as advertising is positively related to sponsorship transparency, which has a positive relationship with attitudes toward content and the publisher (Amazeen and Wojdynski, 2019). Moreover, experimental research has shown that providing additional details about the sponsored content can increase credibility relative to more standard disclosures via perceived transparency (Krouwer et al., 2019). Although there is some debate about the effectiveness of transparency to translate into trust and credibility (Karlsson et al., 2014), transparency still enjoys a place for many among respected news norms (Lasorsa, 2012). For example, Fogg et al. (2009) suggest that the presence of design elements that convey the nature of the information will lead to increased credibility. In addition, research has found that adding explanatory elements to stories can boost credibility perceptions. One study found that adding a box to news stories that explained the reporting process improved perceptions of credibility of the news organization (Chen et al., 2019). Another found that adding elements such as a link to a reporter’s bio or explanations of how and why an article was written, increased credibility perceptions (Curry and Stroud, 2019). Similarly, news labels are meant to provide users a clear marker describing the type of content.

News cues can affect credibility perceptions. In one study, researchers examined three cues: the source, the number of related articles, and recency of the article (Sundar et al., 2007). Although the authors did not find clear direct effects, they did find that news cues interact in ways that affect users’ perceptions of credibility. Peripheral messages, such as mistakes in articles (Karlsson et al., 2017), comments posted on articles (Thorson et al., 2010), and the news sources (Johnson and Kaye, 2016) affect credibility perceptions. Authority cues (Lin and Spence, 2018), platform name and sharing applications (Bhandari, 2018), and social endorsements (Messing and Westwood, 2014) also influence the selection and perception of online news. In short, research shows that even superficial messages can influence how credible users deem content.

Studies focusing specifically on labels provide mixed evidence about the effects of labels on perceived credibility. In the study detailed earlier that labeled video news releases, labels did not influence users’ perceptions of credibility (Tewksbury et al., 2011). A study on native advertising, however, found that those recognizing that the content was an ad rated the news story as less credible (Wojdynski and Evans, 2016). It is possible that labels indicating that a story is something other than news content would lead people to rate the story as less credible because of the perception that other categories of content (e.g. analysis, opinion, and sponsored content) are more subject to biases. Given different theoretical rationales support different patterns of results, we propose the following non-directional hypothesis:

H1a: An article’s label, whether news, analysis, opinion, or sponsored, will affect perceptions of the credibility of the news story compared to no label.

Sponsored content, however, has received more scholarly attention. Previous work indicates that recognizing sponsored content leads to lower perceptions of publisher credibility, particularly for online versus legacy outlets (Amazeen and Muddiman, 2018), more negative evaluations of the content (Amazeen and Wojdynski, 2019) and the publisher (Amazeen and Wojdynski, 2018) as well as diminished perceptions of news credibility (Wojdynski and Evans, 2016). Based on this evidence, we propose:

H1b: An article with a sponsored label will result in lower news story credibility ratings than the other conditions.

Study 1 method

Design and procedures

To test whether news labels affect readers’ perceptions of credibility, we employed a 2 (article topic: science, politics) × 5 (label type: news, analysis, opinion, sponsored, and no label) experimental design with 1000 participants.¹ Institutional Review Board (IRB) approval was obtained. After consenting to participate in the study, users were randomly assigned to one of the 10 conditions.

Two articles were selected for the study to understand whether the conclusions were robust across topics: a science article, ‘Battery That Charges in One Minute Could Beat Tesla in the Race for Tomorrow’s Clean Car’, and a political article, ‘Trump and Pentagon Search for Way to End America's Longest War’. These articles were selected because they contained elements that were arguably consistent with each of the possible labels in the study. The science article came from the science and technology news website Futurism; the political article came from usatoday.com. Both articles were edited for length to 218 and 219 words, respectively.

The headline and content of each article remained constant; the only element that changed was the news label. The label consisted of a small black box with white type that was displayed above the story (Figure 1). The intent was to mimic labels frequently used on actual news sites but to make the labels even more prominent to maximize the probability that they would be noticed. All the stories were posted on a mock news site. After being shown one of the articles, participants answered questions about whether or not they saw a label, what they thought the label was, and a series of items related to credibility. This design allowed us to test whether the news label alone (holding the content of the articles constant) affected readers. Participants spent an average of 8.64 minutes and a median of 6.19 minutes completing the study.

Figure 1.

Examples of the above-story labels, Study 1.

Data were gathered by Survey Sampling International (SSI) between July 26–30, 2018.² SSI invites members of the public to join their online survey panel and uses incentives to encourage participation. SSI matched our sample to national demographics in terms of age, race/ethnicity, gender, education, and census region. Although the sample was not random, it was demographically representative of internet users in the United States, according to data from Pew Research Center (Table 1). There were no significant demographic differences by condition.

Table 1.

Participant demographics compared with U.S. adult Internet population.

	U.S. Adult internet population	Study 1	Study 2
		n = 1000	n = 697
Gender
Male	49%	43%	54%
Female	51	56	45
Race/Ethnicity
Hispanic/Latino	15	12	7
White	70	79	81
Black or African-American	13	11	9
Asian	5	4	7
Mixed race/Other	12	5	4
Age
18–29	24	26	25
30–49	36	40	55
50–64	25	28	16
65+	15	5	4
Household income
Less than $30K	31	28	21
$30K–$49,999	18	17	25
$50K–$74,999	14	23	27
$75K or more	37	32	27
Education
High school graduate or less	34	27	13
Some college/Assoc. degree	33	40	33
College graduate or more	33	34	55

U.S. Internet population is based on data from Pew Research Center when data were collected in January 2018.

Dependent measures

Label recall was captured by asking participants: ‘Which label (if any) appeared on The News Beat article that you just saw?’ Answer options were: news, analysis, opinion, sponsored, no label, or did not notice whether the article was labeled or not. Overall, 13.5% correctly recalled the label.

Perceptions of news story credibility were measured using an index drawn from Yale et al. (2015) and Meyer (1988). Eighteen items asked participants to indicate from 1 (strongly disagree) to 5 (strongly agree) whether the article they saw was: fair, unbiased, concerned mainly about the public interest, balanced, objective, accurate, honest, believable, trustworthy, up-to-date, current, timely, told the whole story, respected people’s privacy, could be trusted, separated facts from opinion, watches out after my interests, and reports the whole story. A principal components factor analysis with promax rotation showed a single factor based on the scree plot (Eigenvalue = 9.87, 54.82% variance explained). The averaged measure (M = 3.51; SD = 0.73) had a reliable Cronbach’s α of .95.

Study 1 results

RQ1 asked whether news users would recall the label they encountered. As both the label displayed and label recalled were nominal variables, we analyzed the data using a chi-square statistic. Similar patterns appeared whether participants saw the politics or science article, so we present the data without separating out which topic participants saw in Table 2. There were no significant differences in which label participants recalled depending on the label condition to which they were assigned (Overall χ²[20] = 30.22, Cramer’s V = .09, p = .06; Science χ²[20] = 28.36, Cramer’s V = .11, p = .17; Politics χ²[20] = 22.23, Cramer’s V = .11, p = .33). Overall, 44.8% of participants reported not noticing whether the article was labeled, and there were no differences by condition. Those who chose a label were most likely to report seeing a news label (regardless of condition); overall, 31.4% reported that the article they saw was labeled news. Examining the Bonferroni-corrected pairwise comparisons, those seeing the opinion label were significantly more likely to say that the article label was opinion than those seeing the analysis label or no label at all. No other significant differences appear. In sum, people were not particularly accurate in their recall of the label conditions.

Table 2.

Cross-tabulations showing label recalled based on label displayed, Study 1.

	Label displayed
	News	Analysis	Opinion	Sponsored	No label
Label recalled
News	34.0%	35.7%	28.9%	29.2%	29.5%
Analysis	8.2	8.0	7.8	6.9	7.5
Opinion	7.2_a,b,c	4.5_c	12.7_b	8.9_a,b,c	3.5_a,c
Sponsored	1.5	2.5	3.4	3.0	1.5
No Label	4.6	7.0	3.9	5.0	10.5
Did Not Notice	44.3	42.2	43.1	47.0	47.5

Different subscripts within a row indicate significant differences between proportions (p < .05) using Bonferroni-corrected pairwise comparisons.

H1a predicted that labeling an article as news, analysis, opinion, or sponsored would affect readers’ perceptions of the credibility of the news content, compared with the no-label condition. H1b proposed that those seeing a sponsored label would report the lowest levels of news story credibility. Using an analysis of variance (ANOVA), labels did not affect the article credibility (F[4, 994] = 2.26, p = .06). Credibility ratings were statistically indistinguishable regardless of whether people saw a news label (M = 3.43, SE = 0.05), an analysis label (M = 3.56, SE = 0.05), an opinion label (M = 3.62, SE = 0.05), a sponsored label (M = 3.46, SE = 0.05), or no label at all (M = 3.49, SE = 0.05). Although there were differences by topic (F[1, 994] = 42.22, p < .01), whereby the science article was rated as more credible than the politics article (science M = 3.66, SE = 0.03; politics M = 3.37, SE = 0.03), the effect of labels did not vary by condition (interaction between label and topic F[4, 990] = 0.44, p = .78). In sum, there was no support for H1a or H1b.

Study 1 discussion

The goal of Study 1 was to assess whether labels on news stories were noticed and/or boosted perceptions of article credibility. We note that our manipulation of the article labels was more obvious than traditional labels because it included a black background and no other visuals were present on the page. Results showed that nearly half the participants could not recall if the story they viewed had a label or not, and the labels produced no significant effects on perceptions of news story credibility. These findings suggest that labels on stories may not be doing the job that news organization hope they would. Labels may not operate as cognitive cues if people do not notice them.

We considered several explanations for our findings. First, it may be that labels have become so routinized on news sites that newsreaders in our study just skipped over them, particularly because our label placement was typical of real news sites. A study examining how native advertising is indicated on news websites found that people were more likely to recognize that the content was an advertisement when a label, such as Sponsored Content, was placed in the middle of the article, as opposed to at the top of the article (Wojdynski and Evans, 2016). Using eye-tracking software, the authors reported that 40% paid attention to the label when it appeared at the top of the page compared to 90% when it appeared in the middle and 60% when it appeared at the end. Second, it is possible that the study participants did not attend to the stimuli as intended. Several aspects of the study ward against this possibility: participants had to correctly answer questions indicating that they were 18 years of age or older, that they were U.S. residents, and that they had successfully seen the stimuli in order to proceed with the study. Further, we examined whether the results were affected by how long people spent with the survey – in no instance did this change the results. In Study 2, however, we add an additional attention check to the questionnaire. Third, it is possible that people do not understand what these labels mean. For example, Loker (2018) reported that at least half of the public is unfamiliar with terms like native advertising and op-ed. To test these explanations, Study 2 involved a test of two different types of labels – the above-story label used in Study 1 and an in-story explainer label that provided a description of the story content categories (e.g. ‘An opinion piece advocates for ideas . . .’). This label was positioned within the story, rather than above the story, because its size might make it cumbersome above the story. We propose the following:

H2: In-story labels that explain the story content categories will increase recall of the accurate label compared to above-story labels that do not explain the story content categories or no labels.

Further, if explainer labels are more influential as we predict, then they may affect credibility more than above-story labels that do not explain the story content categories or no labels. Therefore:

H3: In-story labels that explain the story content categories will have a greater effect on credibility ratings of the news story than above-story labels that do not explain the story content categories or no labels.

A second plausible explanation for our null results in Study 1 is that people make sense of an article based on the content (story and headline), not just the label. This differs from those proposing that news cues function as heuristics and spare people from the cognitive work required to make an assessment about the story content on their own (e.g. Sundar, 2008). For example, when people read a story labeled sponsored that appears more like a news story, they may dismiss the label because it is inconsistent with the content they read. Alternatively, they may ignore the labels and instead rely on their own assessments about the content. It is important to note that in Study 1 we purposefully held the content constant across conditions to parse whether the label functioned independently. It may be that labels do not have enough power to operate independently without content that more clearly matches the label. To test this idea in Study 2, we used four different stories on the same topic (news, analysis, opinion, or sponsored content) and varied whether each was labeled or not. In other words, a news story was either labeled as news or not labeled at all; an opinion piece was either labeled as opinion or not labeled at all. We asked participants to recommend what label the story they read should have, as a means to decipher whether they actually noticed the different types of stories. Because the literature does not provide us guidance to propose a directional hypothesis, we pose a research question:

RQ2: Will label type (above story that does not explain the story content categories, in-story explainer, or no label) affect perceptions of what the story should be labeled?

Study 2 method

Design and procedures

To test the influence of labeling on perceptions of credibility of the news story, a 4 (story type: news, analysis, opinion, and sponsored content) × 3 (label type: above-story label that does not explain story content categories, in-story label that explains story content categories, no label) design was used in an online survey experiment (Figure 2). IRB approval for the project was obtained. The in-story explainer label used story content category definitions based on those proposed by the international consortium of news organizations known as the Trust Project. After consenting, participants (n = 697) were randomly assigned to view one of the four story types and also randomly assigned to see the story with one of the two label types or no label, which served as a control condition.³ Then they answered questions about the stories, explained below, and demographic questions. Data were collected March 17–18, 2019.

Figure 2.

In-story explainer labels, Study 2.

Participants

The stories were posted on a mock website, as in Study 1. However, in this study, the content of the stories and headlines differed by condition to reflect the four types of a story (news, analysis, opinion, and sponsored), and the label (if used) reflected that story type. For example, people randomly assigned to view a news story, saw a news label with a news story (or no label in the control condition). The stories were based on real stories that reflected each story type, and they were edited to be similar in length. All the stories related to gender makeup of the U.S. Congress. The news story (158 words) came from CNN.com, the opinion story (160 words) came from LaTimes.com, the analysis story (158 words) came from WashingtonPost.com, and the sponsored content story (143) was adapted from several nonprofits groups’ webpages. Participants spent an average of 8 minutes and a median of 6.78 minutes completing the study.

Participants

Participants were recruited using TurkPrime. TurkPrime culls participants from Amazon.com’s Mechanical Turk, an opt-in online tool that allows participants to complete tasks for payment, but blocks suspicious geolocations to guard against bots.⁴ Table 1 shows the participants’ demographics. No significant demographic differences were found across the conditions.

Dependent measures

The same measures from Study 1 were used in Study 2. Overall, 38.3% correctly recalled the article label. Perceptions of news story credibility again formed a reliable, single-factor measure. A principal components factor analysis with promax rotation showed one factor, based on the scree plot (Eigenvalue = 9.06, 50.32% variance explained). The 18 items were averaged into an index with high reliability (M = 3.70, SD = 0.70, Cronbach’s α = .94). In Study 2, we added a measure of what participants thought the article should be labeled; 62.4% offered the correct label.

Study 2 results

We investigated whether above-story article labels that did not explain story content categories (RQ1) and in-story labels that explained the story content categories (H2) would affect people’s recall of how the article was labeled. To begin, we present the percentage of people correctly recalling the label by article condition in Table 3 under the header ‘Correctly recalling displayed label’. There were significant differences based on the label used (χ²[2] = 112.10, Cramer’s V = .40, p < .01) and the actual content (χ²[3] = 10.28, Cramer’s V = .12, p < .05). Based on the pairwise comparisons, a higher percentage correctly recalled the label after seeing the in-story label that explained the story content categories than the other labels, and those seeing a sponsored article were more likely to correctly recall the label than those seeing an analysis piece. We then analyzed the two factors together using a logistic regression model in Table 4, as the dependent variable of whether or not the article label was correctly recalled was dichotomous. Consistent with Study 1, the above-story label that did not explain the story content categories had no effect on whether people correctly recalled the displayed label. Seeing the in-story label that explained the story content categories, however, significantly increased the probability that people would correctly name the label displayed (B = 1.84, SE = 0.21, p < .001).⁵ H2 is supported.

Table 3.

Percent correctly labeling the article by actual news content and labels, Study 2.

	Correctly recalling displayed label	Correctly stating how article should be labeled
Labels
Above-story	24.5%_a	55.4%_a
In-story explainer	66.1_b	73.5_b
None	24.8_a	58.5_a
Actual Content
News	36.2%_a,b	59.8%_a
Analysis	32.6_b	60.5_a
Opinion	36.0_a,b	75.6_b
Sponsored	48.0_a	54.2_a

Different subscripts within a row indicate significant differences between proportions (p < .05) using Bonferroni-corrected pairwise comparisons.

Table 4.

Logistic regression analyses of correctly labeling the article by actual news content and labels, Study 2 B (SE).

	Correctly recalling displayed label	Correctly stating how article should be labeled
Constant	–0.66 (0.20)	–0.14 (0.19)
Labels
Above-story	0.02 (0.22)	–0.14 (0.19)
In-story explainer	1.84** (0.21)	0.70** (0.20)
None (reference category)
Actual Content
News	–0.63* (0.24)	0.22 (0.22)
Analysis	–0.81* (0.24)	0.25 (0.22)
Opinion	–0.57* (0.24)	1.00** (0.24)
Sponsored (reference category)
Nagelkerke R-square	0.22	0.07

*p < .05. **p < .01. Interaction between actual content and labels was not significant in either model, recall: χ²(6) = 3.13, p = .79, should: χ²(6) = 9.01, p = .17.

We hypothesized that above-story article labels (H1a, b) and in-story labels that explained the story content categories (H3) would affect news story credibility perceptions. To test these ideas, we ran two analyses of covariance (ANCOVA) with label type as the key independent variable and story type as a covariate.

The main effect of label type was not significant (F[2, 691] = 0.85, p = .43). There was a significant main effect for story type (F[3, 691] = 17.06, p < .001, η² = 0.06), where news (M = 3.96, SE = 0.05) was seen as significantly more credible than analysis (M = 3.77, SE = 0.05), opinion (M = 3.60, SE = 0.05) or sponsored content (M = 3.48, SE = 0.05) based on post-hoc comparisons with a Sidak correction. There was not a significant interaction between labels and story type (F[6, 685] = 1.29, p = .26). As news story credibility was unaffected by the label, H1a, H1b, and H3 were not supported.

RQ2 asked whether the label would affect perceptions of what the story should be labeled (as opposed to recall of what the article was labeled). As before, we present the percentage of people correctly stating the label by article condition in Table 3 under the header ‘Correctly stating how the article should be labeled’. There were significant differences based on the label used (χ²[2] = 18.43, Cramer’s V = .16, p < .01) and the actual content (χ²[3] = 18.67, Cramer’s V = .16, p < .01). Based on the pairwise comparisons, a higher percentage correctly identified what the label should be after seeing the in-story label that explained the story content categories than the other labels, and those seeing an opinion article were more likely to correctly identify what the label should be compared to those seeing other content. As the dependent variable was dichotomous – whether the label people thought a story should have was correct or not – we then used a logistic regression model, with label type as the independent variable and story type as a control. As shown in Table 4, the in-story label that explained the story content categories increased the probability that people would suggest the correct label. Table 5 provides a summary of our findings from both studies.

Table 5.

Summary of hypothesis and research question outcomes.

Hypotheses and research questions	Outcome
Study 1
RQ1: How accurately will people recall digital news labels (news, opinion, analysis, sponsored content, or no labels)?	No significant differences in recall based on label condition.
H1a: An article’s label, whether news, analysis, opinion, or sponsored, will affect perceptions of the credibility of the news story compared to no label.	Not supported.
H1b: An article with a sponsored label will result in lower news story credibility ratings than the other conditions.	Not supported.
Study 2
RQ1: How accurately will people recall digital news labels (news, opinion, analysis, sponsored content, or no labels)?	No significant differences in recall based on label condition to which participants were assigned.
H1a: An article’s label, whether news, analysis, opinion, or sponsored, will affect perceptions of the credibility of the news story compared to no label.	Not supported.
H2: In-story explainer labels will increase recall of the accurate label compared to above-story labels, or no labels.	Supported.
H3: In-story explainer labels will have a greater effect on credibility ratings of the news story than above-story labels or no labels.	Not supported.
RQ2: Will label type (above story, in-story explainer, or no label) affect perceptions of what the story should be labeled?	In-story explainer label increased probability that people would correctly identify what the story should be labeled.

Study 2 discussion

Study 2 sought to answer questions left unclear by Study 1, most notably whether changing the style and content of the label would affect label recall and credibility perceptions. As in Study 1, participants had low recall of story labels, regardless of whether they were exposed to a news, analysis, opinion, or sponsored story. However, for all four types of stories, people were better at recalling the correct label if the in-story label that explained the story content categories was used, compared with the above-story label that did not explain the story content categories or no label. This is notable because it suggests that labels may need to include a description of the type of content in the story to yield recall. This finding is similar to that of Krouwer et al. (2019), who found that more detailed disclosures about sponsored content were more beneficial, although that study did not consider other content types as we did. Despite the better label recall when the in-story label that explained the story content categories was used, labels again did not affect perceptions of news article credibility. Our findings are in line with earlier research that has shown that few people recognize news labels (Amazeen and Muddiman, 2018; Amazeen and Wojdynski, 2018; Tewksbury et al., 2011) or can recall them correctly after exposure (Newhagen, 1994).

Our Study 2 findings also provide important insights into how newsreaders assess the credibility of stories. People could ascertain differences among some story types, and these differences – not the labels – influenced how credibly they viewed the story. When asked to report what label was displayed, participants were more likely to answer the question correctly when the story was sponsored content. When asked to report what label should be displayed, people were most likely to provide the correct label when exposed to an opinion story. When it comes to credibility assessments, news stories were judged as more credible than analysis, opinion, and sponsored content. This finding suggests that newsreaders may intuitively understand that some content is less objective than news, so news organizations should emphasize this subjectivity so readers can easily perceive it, even if news labels are not used. Our result regarding sponsored content is in accordance with research that has found that sponsored content is evaluated as less credible than content not understood as sponsored (Amazeen and Muddiman, 2018; Amazeen and Wojdynski, 2018).

Notably, participants had trouble differentiating a news story from an analysis piece, which makes sense because both rely on more objective reporting techniques compared to opinion or sponsored stories. It may be helpful to readers if the content of analysis stories makes it clear that the articles are interpreting news, rather than merely reporting it, so that newsreaders do not confuse news and analysis, even if labels are not employed. For example, analysis pieces could use terms such as ‘interpret’ or ‘analyze’ to help readers recognize that the stories are not just reporting facts. These findings bolster the evidence that newsreaders can, at least in some cases, tell the difference between story types and use these differences to assess how they view the story.

General discussion

Overall, these two studies sought to understand whether digital news labels operate as cues that affect the perceived credibility of news articles. Our findings do not negate the possibility that labels act as heuristic cues, but they suggest that labels are not very powerful heuristic cues, if they do in fact operate that way. Across two studies, above-story labels that did not explain the story content categories were infrequently recalled – even when they were visually set apart from the other content. In Study 2, in-story labels that explained the story content categories and was embedded at the beginning of the article resulted in greater recall of the article label and of more correct perceptions of what the article should be labeled. Yet the labels did not affect credibility assessments compared to unlabeled news content, suggesting that they do not provide a strong cue for these judgments. People may be engaging in more systematic processing because there were differences in their credibility assessments based on the actual content, irrespective of the label. This finding suggests that newsreaders may understand that different types of content should be viewed with a more skeptical eye (e.g. sponsored content), so news organizations should be clear in the content of the story what type it is – even if they do not use a label.

Americans are not particularly skilled at distinguishing factual news statements from opinion news statement (Barthel, 2018). News labels are one tool that news organizations have implemented to help users make more accurate assessments. Our findings are in line with research that shows a minority of users notice news labels (Amazeen and Muddiman, 2018; Tewksbury et al., 2011) or are able to recall them correctly after exposure (Newhagen, 1994). In a departure from work that indicates sponsored content garners lower credibility ratings and more negative evaluations of content (Amazeen and Muddiman, 2018; Amazeen and Wojdynski, 2019; Wojdynski and Evans, 2016), we found that a sponsored label alone (leaving the content constant) did not affect perceived credibility. However, in accordance with previous findings, sponsored content (not the label) was perceived as less credible than news. As a contribution, our study also indicates that opinion and analysis content also is perceived as less credible than news content. People were more likely to notice the in-story labels that explained the story content categories. This may be due to the fact that these labels actually explained the different story types (e.g. ‘An opinion piece advocates for ideas . . .’), rather than merely labeled the story, like the above-story labels. The description in the in-story labels may have made them more prominent compared to the above-story labels. Amazeen and Wojdynski (2018) found that people are more likely to recognize sponsored content as such when the label is more visually prominent. Whereas that study increased prominence with the size, weight, and contrast of the label’s font, our study shows that an explanation of story content categories also may work to increase label reception.

This study is not without limitations. Although we tested two different types of labels across three topics of text stories, future research should consider other types of labels across different types of stories (e.g. video). Labels could operate differently depending on context and prominence. Krouwer et al. (2019) found that, at least for sponsored content, more detailed disclosures led to higher perceptions of transparency, which ultimately, boosted credibility assessments compared to less detailed disclosures. Moreover, future research should parse how the position of labels and their content distinctly influence credibility perceptions. Our study compared two types of labels with different content and positions, but the design did not allow us to separate the effects. Finally, labels may work more effectively for people who are choosing to read the news from a site, rather than being paid to do so (Curry and Stroud, 2019). Further, Amazeen and Wojdynski (2019) found that younger and better-educated participants and those seeking information were more likely to identify sponsored content, compared with others, and future research should consider whether similar variables may influence how labels operate.

Future research could test labels on real news sites among their newsreaders. This study used a news site with which respondents would be unfamiliar because we created it for the study. Encountering news from an unknown site is not unrealistic; according to the Texas Media and Society Survey (2016), only 11% of Americans never encounter unfamiliar news sources. Yet analyzing how the results play out with an actual news site warrants future attention. Amazeen and Muddiman’s (2018) finding that fictitious digital publishers suffered more if people recognized native advertising on their sites than legacy publishers did underscores the need to test news labels on real news sites. On the one hand, labels may be even less likely to affect credibility assessments for known news brands, where audiences already have a formed opinion. On the other hand, audiences may be more likely to engage in heuristic processing with a known brand, amplifying the effect of news labels. The latter possibility seems less plausible. If the artificial site we used for this study enhanced systematic processing relative to other sites, we suspect that we would have seen greater correct recall of the displayed news label. We tested a single exposure to news labels; exposure to multiple articles may yield different results.

Despite these limitations, our findings offer both practical and theoretical insight. If story labels operate as cognitive heuristics, they are weak cues at best. In-story labels that explained the story content categories were better recalled, suggesting that if news organizations choose to label stories to benefit their audiences, they should use a style that explains the type of story, rather than merely including a label above the story. Even so, labels are not a cure-all for waning trust. We have no evidence that labels boost the perceived credibility of news stories as news organizations might hope. News organizations trying to bolster their credibility via news labels may want to try other methods, such as adding a box to a story that explains how and why a story was done or adding identifying information about the journalists (Chen et al., 2019).

Conclusion

News organizations are searching for ways to boost credibility. Providing audiences with heuristics to help them navigate the news ecosystem seems like a promising strategy, particularly given that audiences sometimes skim and engage in heuristic processing when making news decisions (Holton and Chyi, 2011). Labels represent one such option. The two studies reported here, however, urge caution in relying upon news labels alone to elevate credibility. Neither obtrusive above-story labels that did not explain story content categories nor in-story labels that explained story content categories resulted in higher credibility ratings for the news story compared to articles without a label. In-story labels that explained story content categories did lead people to have more accurate perceptions of what the story was labeled and what the story should be labeled, offering some rationale for their use. Standard labels that are buried in the header of a news article, however, seem unlikely to garner notice. News outlets should try other ways – besides only labeling stories – to improve their credibility.

Footnotes

Declaration of conflicting interests

There is no conflict of interest related to this research.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for this research came from the Moody College of Communication at The University of Texas at Austin and from Democracy Fund.

ORCID iDs

Cynthia Peacock

Gina M Masullo

Notes

Author biographies

Cynthia Peacock (PhD, University of Texas at Austin) is an Assistant Professor in the Communication Studies Department at the University of Alabama and a Faculty Research Associate with the Center for Media Engagement at the University of Texas at Austin. Her research focuses on the ways people form, change, and express their political opinions.

Gina M Masullo (PhD, Syracuse University) is an Associate Director of the Center for Media Engagement and an Associate Professor in the School of Journalism, both at The University of Texas at Austin. Her research focuses on how the digital space both connects and divides people and how that influences society, individuals, and journalism.

Natalie Stroud (PhD, University of Pennsylvania) is the Director of the Center for Media Engagement and a Professor in the Communication Studies Department and School of Journalism at The University of Texas at Austin. Her research focuses on understanding how the media environment affects democracy.

References

Amazeen

Muddiman

(2018) Saving media or trading trust? Digital Journalism 6(2): 176–195. DOI: 10.1080/21670811.2017.1293488.

Amazeen

Wojdynski

(2018) The effects of disclosure format on native advertising recognition and audience perceptions of legacy and online news publishers. Journalism 1–20. Epub ahead of print 7 February 2018. DOI: 10.1177/1464884918754829.

Amazeen

Wojdynski

(2019) Reducing native advertising deception: Revisiting the antecedents and consequences of persuasion knowledge in digital news contexts. Mass Communication and Society 22(2): 222–247. DOI: 10.1080/15205436.2018.1530792.

Barthel

(2018) Distinguishing between factual and opinion statements in the news. Available at: https://www.journalism.org/2018/06/18/distinguishing-between-factual-and-opinion-statements-in-the-news/ (accessed 28 October 2020).

Bhandari

(2018) Social media cues and news site name: What do they mean for online news perception? Newspaper Research Journal 39(2): 169–179. DOI: 10.1177/0739532918775699.

Chaiken

(1980) Heuristic versus systematic information processing and the use of source versus message cues in persuasion. Journal of Personality and Social Psychology 39(5): 752–766.

Chen

Curry

Whipple

(2019, February) Building trust: What works for news organizations. Center for Media Engagement. https://mediaengagement.org/research/building-trust (accessed 28 October 2020).

Curry

Stroud

(2019) The effects of journalistic transparency on credibility assessments and engagement intentions. Journalism: Theory, Practice and Critism. Online First Advance Publication. DOI: 10.1177/1464884919850387.

Fogg

Cuellar

Danielson

(2009) Motivating, influencing, and persuading users: An introduction to captology. In: Sears

Jacko

(eds) Human Computer Interaction Fundamentals. Boca Raton: CRC Press, Inc, pp.109–122.

10.

Gallup (2018) Indicators of news media trust. Available at: https://kf-site-production.s3.amazonaws.com/media_elements/files/000/000/216/original/KnightFoundation_Panel4_Trust_Indicators_FINAL.pdf (accessed 28 October 2020).

11.

Holton

Chyi

(2011) News overload and the overloaded consumer. Cyberpsychology, Behavior, and Social Networking 15(11): 619–624. DOI: 10.1089/cyber.2011.0610.

12.

Iannucci

Adair

(2017) News or opinion? Online, it’s hard to tell. Available at: https://www.poynter.org/ethics-trust/2017/news-or-opinion-online-its-hard-to-tell/ (accessed 28 October 2020).

13.

Johnson

Kaye

B K

(2014) Credibility of social network sites for political information among politically interested Internet users. Journal of Computer-Mediated Communication 19(4): 957–974. DOI: 10.1111/jcc4.12084.

14.

Johnson

Kaye

(2016) Some like it lots: The influence of interactivity and reliance on credibility. Computers in Human Behavior 61: 136–145. DOI: 10.1016/j.chb.2016.03.012.

15.

Jones

(2018) U.S. media trust continues to recover from 2016 low. Available at: https://news.gallup.com/poll/243665/media-trust-continues-recover-2016-low.aspx (accessed 28 October 2020).

16.

Karlsson

Clerwall

Nord

(2014) You ain’t seen nothing yet: Transparency’s (lack of) effect on source and message credibility. Journalism Studies 15(5): 668–678. DOI: 10.1080/1461670X.2014.886837.

17.

Karlsson

Clerwall

Nord

(2017) Do not stand corrected: Transparency and users’ attitudes to inaccurate news and corrections in online journlaism. Journalism & Mass Communication Quarterly 94(1): 148–167. DOI: 10.1177/1077699016654680.

18.

Kohring

Matthes

(2007) Trust in news media: Development and validation of a multidimensional scale. Communication Research 34(2): 231–252. DOI: 10.1177/0093650206298071

19.

Krouwer

Poels

Paulussen

(2019) Moving towards transparency for native advertisements on news websites: A test of more detailed disclosures. International Journal of Advertising, 1–20. Epub ahead of print 20 February 2019. DOI: 10.1080/02650487.2019.1575107.

20.

Lasorsa

(2012) Transparency and other journalistic norms on Twitter: The role of gender. Journalism Studies 13(3): 402–417. DOI: 10.1080/1461670X.2012.657909.

21.

Lin

Spence

(2018) Identity on social networks as a cue: Identity, retweets, and credibility. Communication Studies 69(5): 461–482. DOI: 10.1080/10510974.2018.1489295.

22.

Loker

(2018) Confusion about what’s news and what’s opinion is a big problem, but journalists can help solve it. American Press Institute. Available at: https://www.americanpressinstitute.org/publications/confusion-about-whats-news-and-whats-opinion-is-a-big-problem-but-journalists-can-help-solve-it/ (accessed 28 October 2020).

23.

Messing

Westwood

(2014) Selective exposure in the age of social media: Endorsements trump partisan source affiliation when selecting news online. Communication Research 41(8): 1042–1063. DOI: 10.1177/0093650212466406.

24.

Metzger

Flanagin

(2013) Credibility and trust of information in online environments: The use of cognitive heuristics. Journal of Pragmatics 59: 210–220.

25.

Meyer

(1988) Defining and measuring credibility of newspapers: Developing an index. Journalism Quarterly 65(3): 567–574. DOI: 10.1177/107769908806500301.

26.

Micek

(2017) We’re making it easier to distinguish news from opinion. Available at: https://www.pennlive.com/opinion/2017/07/were_making_it_easier_to_disti.html (accessed 28 October 2020).

27.

Newhagen

(1994) Effects of televised government censorship disclaimers on memory and through elaboration during the gulf war. Journal of Broadcasting & Electronic Media 38(3): 339–351. DOI: 10.1080/08838159409364269.

28.

Shah

McLeod

Gotlieb

, et al. (2009) Framing and agenda setting. In: Nabi

Oliver

(eds) The Sage Handbook of Media Processes and Effects. Thousand Oaks, CA: Sage, (pp.83–98).

29.

Stead

(2017) Public editor: Lines between opinion, analysis and news need to be clearer. Available at: https://www.theglobeandmail.com/community/inside-the-globe/public-editor-lines-between-opinion-analysis-and-news-need-to-be-clearer/article34310619/ (accessed 28 October 2020).

30.

Sundar

(2008) The MAIN model: A heuristic approach to understanding technology effects on credibility. In: Metzger

Flanagin

(eds) Digital Media, Youth and Credibility. Cambridge, MA: MIT Press, (pp.73–100).

31.

Sundar

Knobloch-Westerwick

Hastall

(2007) News cues: Information scent and cognitive heuristics. Journal of the American Society for Information Science and Technology 58(3): 366–378. DOI: 10.1002/asi.20511.

32.

Tewksbury

Jensen

Coe

(2011) Video news releases and the public: The impact of source labeling on the perceived credibility of television news. Journal of Communication 61(2): 328–348. DOI: 10.1111/j.1460-2466.2011.01542.x

33.

Texas Media & Society Survey (2016) Annette Strauss Institute. Available at: https://moody.utexas.edu/sites/default/files/Report_ASI_Media_FOR%20WEB_v01.pdf (accessed 28 October 2020).

34.

Thorson

Vraga

Ekdale

(2010) Credibility in context: How uncivil online commentary affects news credibility. Mass Communication and Society 13(3): 289–313.

35.

Wojdynski

Evans

(2016) Going native: Effects of disclosure position and language on the recognition and evaluation of online native advertising. Journal of Advertising 45(2): 157–168. DOI: 10.1080/00913367.2015.1115380.

36.

Yale

Jensen

Carcioppolo

, et al. (2015) Examining first-and second-order factor structures for news credibility. Communication Methods and Measures 9(3): 152–169. DOI: 10.1080/19312458.2015.1061652.