News story aggregation and perceived credibility

Abstract

The practice of aggregating news content—repurposing content created by other news organizations—raises questions about credibility. This experimental study suggests that news organizations can boost credibility of aggregated content by more clearly identifying originating sources than by increasing or decreasing the use of aggregation. Relationships between levels of aggregation and credibility showed little or no significance, while relationships between credibility and receivers’ confidence in identifying originating sources were significant.

Keywords

credibility aggregation experiment journalism quantitative ethics

News consumption online is rising (Pew Research Center, 2019a, 2019b), but the number of journalists creating original news is declining. Legacy print organizations originate 60% of local news stories nationwide (Mahone et al., 2019), despite a 47% decline in jobs between 2004 and 2018 (Pew Research Center, 2019d); meanwhile, broadcast news employment remains flat despite adding hours of content (Pew Research Center, 2019c). News consumers are continuing to drop print and local broadcasting (Masta, 2018) as their conduits. Some go directly to individual news organization sites or apps, and two-thirds use social media to find content generated by legacy organizations (Gottfried & Shearer, 2018). The result is that news organizations are exchanging advertising dollars from their legacy channels for digital dimes (Barthel, 2018), even as they must produce more content to drive traffic with fewer journalists.

Many online publishers cover the gap through aggregation, which Coddington (2019) defined as “taking news from published sources, reshaping it, and republishing it in an abbreviated form within a single place” (p. 7). In other words, it is the repurposing and posting portions of, or links to, stories produced by other media—and it can be done with or without additional reporting by the writer. This form of aggregation requires humans to find information from other sources and rewrite it for their own organizations’ use. The term’s use for this study differs from two other definitions of “aggregator.” One is databases, such as Newsbank or ProQuest, that let users search for content from multiple sources (see Gilbert & Watkins, 2020) and pull the results to their screen. The other is the technological definition of “aggregation” as the use of software tools such as Really Simple Syndication (RSS) feeds or similar applications programmed to search for topics or sites preselected by users, then push the results to a single site for users (see Doree, 2007). The technology rarely if ever creates new content or revises previous content, making it different from the definition of aggregation used for this study in which writers typically “weigh evidence, evaluate sources, and verify information” originally published by other news organizations (Coddington, 2020, p. 365).

The reshaping of news often occurs by linking to the original source(s), providing context and/or updated information by gleaning updated content from multiple sources and/or adding visual content to text-heavy original content. Aggregation can occur at three levels: (a) the level of the news index website, where all or a substantial portion of a publisher’s content originates elsewhere online and is repurposed as a news index linking to original or intermediary messengers (e.g., Yahoo! News and Google News), (b) the level of the traditional news organization website, where original and aggregated content co-exist, and (c) at the message level, where individual news stories are compiled, in part or in whole, using material from originating or intermediary messengers elsewhere on the internet (Hurley & Tewksbury, 2012). Duffy et al. (2018) call the practice of rewriting others’ news for your own site “Frankenstein journalism,” which raises questions of epistemology (Coddington, 2019, 2020) and the identities of contributing news organizations (Duffy et al., 2018). It also relates to the notion of ethical aggregation (Carr, 2012; Martin, 2015), including the Society of Professional Journalists’ (SPJ, 2014) code’s edicts to “[a]lways attribute” and “provide access to source material when it is appropriate.” Aggregation also has implications for transparency, which for media practitioners includes the notion of revealing the source of information used in stories. Black and Roberts (2011) wrote, “Transparency is required in order to have the accountability needed to be an ethical communicator, but it is not sufficient to boost credibility” (p. 401).

Research on aggregation has focused principally at the website level, with less study at the message level. This is of concern to journalists who routinely see their work co-opted by competitors who use it to leverage advertising revenue from which the original journalists’ news organizations do not benefit. Similarly, as research has focused largely on index aggregation, the influence that the practice of message or story aggregation has on the perceived credibility of news stories has remained unexamined, and it is this issue that this study explores. As more people consume news aggregated from other news sources, it is important to understand the relationships between the messengers, the messages and perceptions of credibility. This research describes relationships between perceived credibility of online news and levels of aggregation. In a world with rising levels of gatewatching (see Bruns, 2005), complaints of “fake news,” and the use of social media mediating channels that distribute news where the original source may be blurry to consumers, this research can provide insight into relationships between credibility and traditional news organizations.

Literature Review

Communication literature offers widespread agreement toward a broad definition of the concept of credibility, yet precise definitions of and measurements for this multidimensional construct vary widely even as social media adds new dimensions. Hovland and Weiss (1951) and Hovland et al. (1953) equated credibility with trustworthiness and expertise. This classic definition has been applied to studying message credibility, with trustworthiness of a message defined as an appearance of reliability, presence of fairness and lack of bias; and expertise in a message defined as a product of firsthand experience with its messenger, secondhand knowledge of its messenger, or by a messenger’s reputation and/or credentials (Hilligoss & Rieh, 2008). Researchers have defined credibility as a perceived quality with multiple dimensions including believability, accuracy, fairness, authority and bias (Johnson & Kaye, 2010a, 2010b; Melican & Dixon, 2008); completeness (Flanagin & Metzger, 2000); factuality (Kiousis, 2001); dynamism, likeability and goodwill (Wathen & Burkell, 2002); scope and currency (Metzger, 2007); and consistency, nature of presentation and professionalism (Appelman & Sundar, 2016). The definition has been reimagined for new media as a perception of the message as a plausible reflection of events (Newhagen & Nass, 1989).

While news credibility research largely splits along three interlocking paths (messenger, message and receiver attributes), messenger credibility has been directly tied to message credibility (Newhagen & Nass, 1989). Internet-focused analyses apply the concepts of conferred and reputed credibility, suggesting that information seen as being the product of or receiving an endorsement from a well-regarded entity will be perceived as more credible than information lacking a similar association (Chung et al., 2012; Flanagin & Metzger, 2008). Most conferred and reputed credibility research focuses on online versions of traditional news media products—so-called “shovelware”—and not message-level aggregation.

Internet-focused credibility research found that the presence of links, which let users click on a word, phrase or image and move to a related website, boosts perceptions of credibility (Borah, 2014). Karlsson et al. (2014) found that links—a tool commonly used for attribution to the original news source of aggregated stories—were the only device associated with transparency that significantly boost message credibility.

Furthermore, recent research regarding the nature of evidence online and the changing role of the gatekeeper suggests that electronic delivery has changed how we think of originating sources. Kovach and Rosenstiel (2011) argued that the shift online has taken much of the responsibility for assessing legitimacy of news away from professional journalists and placed the onus on readers. In aggregated stories, links to originating material can provide a heuristic cue to credibility.

Contemporary research, however, supports the possibility that receivers of news messages online may have misperceptions about originating sources. Pew found that more than half of those surveyed accurately identified the originating source of news if they retrieved the news by visiting a news organization website. Fewer than half identified the originating source if sent there by social media; 10% erroneously cited the social media site as the original news outlet (Mitchell et al., 2017). More recent Pew research showed that at least a third of Americans incorrectly thought that aggregators such as Google News, Apple News and Facebook do their own reporting (Barthel et al., 2020). In a much-earlier study of internet credibility and sourcing at the message level, Sundar and Nass (2001) found a conflation of source and intermediary that suggests receivers might be confused about the sourcing of an aggregated message, resulting in a lower perception of credibility.

Singer’s (2003) analysis of online news noted that the change in gatekeeping roles online supports the argument that journalists are not professionals and provides an opening for those who challenge the legitimacy of objective news. Former President Trump’s dismissal of unfavorable stories from mainstream messengers as “fake news” suggests Singer was prescient (Allcott & Gentzkow, 2017), and that some consumers make up their minds about credibility based on their perceptions of the messenger even before seeing the message and considering levels of attribution. This ties to “gatewatching,” in which non-journalists make their own news decisions by “harnessing the collective intelligence and knowledge of dedicated communities to filter the newsflow and to highlight and debate salient topics” (Bruns, 2008, p. 174). The source of information is among the many considerations that gatewatchers and others use in assessing credibility, but it is not the only one.

Much existing research regarding message credibility leans either on the perceived authority of the message’s author or source, or on the role of heuristics in message processing by receivers. Sundar (2008) suggested that the ease of digital publishing, which enables aggregation, complicates the gatekeeping process and eliminates or hides cues that receivers traditionally use to make credibility judgments. Absent conventional cues, consumers of online news use heuristics to make credibility judgments along four dimensions via the MAIN (modality, agency, interactivity, navigability) model. Modality represents the communication channel, agency represents the messenger, interactivity represents the interaction levels between the receiver and the medium, and navigability represents the availability of tools facilitating movement about a message or related information (Sundar, 2008). The MAIN model is related to the Elaboration Likelihood Model (ELM) (Cacioppo et al., 1986; Petty & Cacioppo, 1986), which suggests that receivers engage along a continuum between two information processing routes—a central route that requires conscious thought, or a peripheral route that relies on message cues and heuristics. The heuristic systematic model (Chaiken, 1980), also assumes two similar routes of processing and suggests that low-motivated receivers use shortcuts and trust experts or majority opinion (Chaiken & Maheswaran, 1994). A recent study of aggregation and credibility (Molyneux & Coddington, 2019) showed that aggregated content was rated lower in credibility if those stories were poorly written and/or had clickbait headlines—both examples of heuristic clues.

This family of theory and heuristics concepts suggests that readers of aggregated news should respond to a lack of traditional message cues by relying more heavily on the presence of attribution when making credibility judgments. In aggregated news stories, attribution can be a hyperlink that identifies and connects to an originating source. As noted by Kovach and Rosenstiel (2011), these links may play the role of supporting evidence, leading readers to conclude that the news story is more credible by virtue of their presence. While there is limited support for the possibility that links in aggregated stories could confuse the reader about the identity of the originating source of the message and undermine its perceived credibility, the balance of research suggests these links are more likely to be viewed as evidence of transparency, resulting in a more pronounced perception of credibility for stories with more aggregation. This also suggests that a story without aggregation—entirely the work of the bylined author—may be perceived as less credible than an aggregated counterpart, which may include no original reporting. Separately, the theory of conferred credibility suggests that messages associated with a familiar or trusted source will be viewed as being more credible than identical messages associated with a generic source. Because attribution links in aggregated messages are expected to boost credibility as readers see them as evidence and/or transparency (Curry & Stroud, 2019), it is anticipated that perceived credibility may rise as aggregation rises.

With this in mind, three research questions and four hypotheses exploring the main effects on perceptions of credibility are proposed. The main effects studied include the separate relationships between perceptions of message credibility and (a) degree of message aggregation, and (b) the familiarity of the publishing organization. Conditions of independent variables analyzed for interaction effects were similarly assessed.

Because the lack of traditional cues in online news was expected to prompt increased reliance on heuristics in message evaluation, it is possible that the multitude of attributed sources in highly aggregated messages will affect receivers’ ability to accurately determine messages’ originating sources. This informs the first research question:

RQ1:What potential sources, and under what conditions, do receivers perceive as being the originating sources of aggregated messages?

Heuristics concepts (Petty & Cacioppo, 1986; Sundar, 2008) suggest that, in the absence of traditional news message cues, receivers of aggregated online news will take shortcuts when making credibility decisions. This informs the second research question:

RQ2:What is the relationship between receivers’ level of confidence in their ability to accurately identify the originating source(s), and receivers’ perception of message credibility?

Conversely, however, because receivers likely employ heuristics along the modality and agency dimensions of the MAIN model, literature also suggests that receivers will see the attributive element of aggregation as evidence similar to attribution of original reporting in print news (Sundar, 2008), perceiving stories that are more highly aggregated as also being more credible by virtue of the presence of more supporting evidence. This informs the first hypothesis:

H1:There will be a positive relationship between degree of message aggregation and receivers’ perception of message credibility.

Because the theory of conferred credibility suggests that messages from a familiar source will be perceived as being more credible than messages from an unfamiliar source, it could be that more highly aggregated stories from a familiar messenger will be perceived by receivers as more credible than identical messages from a messenger with whom the receiver has not established trust. This informs the final hypotheses:

H2:Highly aggregated messages from a familiar source will be perceived as more credible than highly aggregated messages from a generic source.

H3:Moderately aggregated messages from a familiar source will be viewed as more credible than moderately aggregated messages from a generic source.

H4:Non-aggregated messages from a familiar source will be viewed as more credible than non-aggregated messages from a generic source.

Finally, research regarding perceptions of media credibility and news media consumption habits has reached conflicting conclusions (Kiousis, 2001), and relationships between perceived message credibility, consumption habits and degree of attribution remain largely unexplored. Research suggests that people who consume more news may be more familiar with news conventions (Mitchell et al., 2014), suggesting that they may see aggregated content not as more well-sourced, but as less original. This informs the final research question:

RQ3:What are the relationships between perceived message credibility, degree of message aggregation and media consumption habits?

Method

This study analyzed data from 303 participants in an experiment that used a 2 × 2 × 3, between-subjects factorial design presenting subjects with mock news web pages carrying one of two news stories, presented as published by one of two news organizations, and in conditions representing three degrees of aggregation.

Before the experiment, 17 fictional news organization names were generated to select the names of three credited sources of aggregated material and an unfamiliar, fictional news organization to represent one of the two identified web page publishers. Thirty subjects on Amazon’s Mechanical Turk rated those names using a 5-point Likert-type scale, ranging from strongly negative to strongly positive. Results showed that Advocate Gazette (M = 2.9, SD = .96), Real-Time Beacon News (M = 3.1, SD = .87) and CTNews (M = 3.3, SD = .99) generated similarly neutral attitudes among the participants; post hoc analyses showed no significant difference in audiences’ perception, F(2, 87) = .612, p > .05. The names of real-world, familiar news organizations that consistently rank among the most-trusted U.S. news organizations were similarly tested. Post hoc analysis found no significant difference, F(6, 196] = 1.15, p > .05, among means of perceptions of the seven familiar names tested. USA Today, which scored second-highest among print newspapers for trust in an American Trends Panel survey (Mitchell et al., 2014), was chosen for the familiar publisher condition. Mock-ups were created of news web pages representing the fictional, generic news organization identified as “NewsNB.com” and USA Today. The NewsNB.com mock-up was loosely modeled on websites under the umbrella of Advance Local, a subsidiary of the Newhouse family’s Advance Publications. It includes middle-market news websites affiliated with legacy newspapers and that routinely include both aggregated and original content, and content that is a combination of original and aggregated material. The company’s 12 websites representing 30-plus newspapers are among the most-visited news websites in the United States, and their affiliated news organizations’ awards include Pulitzer prizes, Emmys and Peabodys. The USA Today mock-up was based on the real-world web page of that organization. The key researcher is a longtime journalist, familiar with aggregation practices and design conventions common to the industry.

The message stimuli include two, three-paragraph news stories, the first about a fire that destroyed businesses in a fictional neighborhood called “Northside,” and the second about political intrigue in North Korea. Both stories were derived from actual Associated Press stories and were presented identically across conditions, except for the addition of attribution in a manner consistent with common news story aggregation practices. For the no-aggregation condition on both the generic and familiar publication platforms, each of the two, three-paragraph stories appeared below a headline and a byline including a fictitious, gender-neutral journalist’s name, Pat Smith, and the name of the publication on whose website the stories appeared to have been published, NewsNB or USA Today (see Figures 1 –3 for examples). Consistent with industry practice, headlines were written in Associated Press style, and stimuli pages included social media and email links as well as ads. Stimuli stories included the addition of attribution and a link to CTNews in the moderate-aggregation condition and to CTNews, the Advocate Gazette and The Real-Time News Beacon in the high-aggregation condition.

Figure 1

Generic Aggregator, Story No. 1, No Credited Sources

Figure 2

Generic Aggregator, Story No. 1, Moderate Credited Sources

Figure 3

Familiar Aggregator, Story No. 1, High Number of Credited Sources

A power analysis (Institute for Digital Research and Education, n.d.) showed that a sample size of 211 was needed to achieve a moderate effect size of .25 at a probability level of .05 and β of 0.8. Subjects living in the United States were recruited using Amazon’s Mechanical Turk service and used Qualtrics software to complete surveys in Spring 2017. Subjects were paid $1 to complete the survey, with that sum credited to their Amazon account upon completion and approval of the researcher. To be compensated, subjects entered a random number generated by Qualtrics upon completion of the survey, making it unlikely that subjects could receive payment without completing the task.

This study recruited 394 participants; the final sample size was 303 after removing data from subjects who did not complete the survey or completed it in less than 2 min while making selections at only the top or bottom measures on scales. Time spent taking the survey ranged from under 2 min to 41 min, with subjects taking an average of 5.3 min.

The sample was demographically diverse but not reflective of national U.S. Census Bureau (2018) data: The sample was 5% Black, versus 13% of the U.S. population, and 61% male, versus 49% of the U.S. population. The sample also self-identified as being more politically independent and less conservative than the overall population. Forty-four percent of the sample identified as Democrats, versus 48% nationally; 20% identified as Republican, compared with 44% nationally (Pew Research Center, 2016), and 36% identified as being politically independent.

Consideration of Covariates

Because aggregation at the news story level has received little attention from researchers, the possibility of the presence of confounds, or unmeasured variables that influence outcomes, was addressed by conducting a series of two-way analyses of covariance (ANCOVAs) exploring relationships between publication type and aggregation condition on perceived credibility controlling for possible covariates related to subjects’ demographic characteristics and news consumption habits. Explored covariates include subjects’ age, gender, race, level of education, political affiliation, household income, sources of news and frequency of news consumption, which has been found to serve as a proxy for media or news literacy (Eveland & Scheufele, 2000).

Independent Variables

Degree of message aggregation

Stimuli stories were presented with one of three degrees of aggregation: (a) No aggregation, in which the story appears below the byline identifying the author and the author’s news organization; (b) moderate aggregation, in which the story is identical to that of the first condition but with the addition of a single credited, aggregated source in the second of three paragraphs; (c) high aggregation, in which the story is identical to the first condition but has one credited, aggregated source in each of the three paragraphs.

Publisher

The mock-up web pages were credited to the generic news organization NewsNB.com or USA Today. They were identified in the URLs as well as atop each page.

Perceived message credibility

In both the generic and familiar news organization conditions, subjects were asked to rate the credibility of the message using a series of five measures with each employing a 5-point, Likert-type scale, with means of the credibility measures calculated and summed to determine a message credibility score. The credibility component measures of accuracy, completeness, objectivity, spin and professionalism were drawn from components of scales commonly used in credibility research (Austin & Dong, 1994; Flanagin & Metzger, 2007; Gaziano & McGrath, 1986; McCroskey & Teven, 2009; Meyer, 1988; Roberts, 2010).

Perceived identity of originating source

Respondents were asked to identify the originating messenger(s) as (a) the bylined author, (b) the fictional or known publication identified at the top of the page and in the URL, (c) one or more of the fictional news organizations identified as the sources of aggregated material, (d) all of the above, or (e) unknown/other media.

Confidence in one’s ability to accurately identify originating source

Respondents were asked to rate their level of confidence in their ability to identify the originating messenger or messengers on a 5-point, Likert-type scale: 1 (not at all confident), 2 (a little confident), 3 (somewhat confident), 4 (reasonably confident), or 5 (completely confident).

Frequency of exposure to news

Respondents were asked to rate the frequency with which they watch, read or listen to news on a 3-point, Likert-type scale: 1 (weekly or less often), 2 (daily), or 3 (more than one time per day).

Primary sources of news

Respondents were asked to identify their primary source(s) of news from a list of seven options: (a) Print newspaper, (b) newspaper website, (c) television, (d) television news organization website, (e) social media, (f) websites not affiliated with traditional news media (e.g., Huffington Post, Buzzfeed), or (g) radio.

Political affiliation

Respondents were asked to identify their political affiliation, choosing from a list of three options: (a) Republican, (b) independent, or (c) Democrat.

Results

RQ1 required exploration of readers’ perceptions regarding the origins of messages. Table 1 shows that the page publisher (the organization identified at the top of the page and in the URL) was most often identified as an individual originating source in all three conditions, declining in number as the aggregation levels rose. With the addition of aggregation, the percentage of instances in which the bylined author or the identified page publisher were credited as a likely originating source fell by about half. In the high-aggregation condition, subjects selected the page publisher and the last aggregation source identified in the story as their likely originating sources at approximately equal rates. In the high-aggregation condition, subjects were most likely to see aggregated sources presented near the beginning or near the end of the story as originating sources, while the aggregation source identified in the middle of the story was largely overlooked. The bylined author was least likely to be perceived as the originating source under the no- and moderate-aggregation conditions and was picked less often than the publisher and less often than the last aggregator used in the high-aggregation condition.

Table 1

Perceived Originating, Individual Sources as Identified by Subjects, Within Aggregation Condition and in the Order of Presentation

Credited source	Aggregation condition
	None		Moderate		High
	n	%	n	%	n	%

Page publisher	297	62.4	139	40.7	111	26.0
Bylined author	172	36.1	86	25.2	74	17.4
First aggregator	—	—	—	—	72	16.9
Second aggregator	—	—	—	—	65	15.3
Third aggregator	—	—	115	33.7	101	23.7
Unknown	7	0.0	1	0.0	3	0.0

Note. The no-aggregation condition included no apparent aggregation. The moderate aggregation condition included one apparent aggregator. The high-aggregation condition included three apparent aggregators.

RQ2 required exploration of the relationship between receivers’ level of confidence in their ability to accurately identify the originating source(s) of a news story and their perception of its credibility. A one-way, between-groups analysis of variance found a positive relationship between confidence in source identification and perceived credibility: F(8, 294) = 6.3, p < .001. Subjects who reported higher levels of confidence in their ability to identify originating sources were more likely to find stories to be credible. Separately, one-way, between-groups analyses of variance explored the relationship between credibility and confidence in source identification for the domestic and international stories separately. Relationships were found for both, respectively: F(4, 298) = 13.5, p < .001; F(4, 298) = 11.6, p < .001.

Again, H1 predicted a positive relationship between degree of message aggregation and receivers’ perception of message credibility. It was not supported. Results showed a significant negative relationship between degree of aggregation and overall perceived credibility when controlling for subjects’ frequency of exposure to news, which can serve as a proxy for media literacy, F(2, 297) = 4.326, p < .05. With a partial eta-squared value of .014, the effect size is small by Cohen’s (1992) standards. When controlling for the covariate, stories with higher degrees of aggregation were viewed by subjects as being less credible than stories with a lower degree of aggregation. A significant relationship also was found between frequency of news consumption and perceived credibility after controlling for the type of publication(s) identified by subjects as their primary source of news, F(2, 297) = 6.529, p < .05. With a partial eta-squared value of .022, the effect size is small by Cohen’s (1992) standard. Notably, the condition that generated the single highest mean credibility score (M = 45.945) was that in which a story containing no aggregated content was presented by the generic publisher. The mean sum credibility scores for stories with no aggregation were consistently higher (and at significant levels after controlling for the covariate) than stories with moderate or high aggregation.

This negative relationship suggests that including links to more sources seemingly lowered perceived credibility—a finding that can be supported in the literature. Research suggests that both perceived credibility (Kiousis, 2006; Metzger et al., 2010; Sundar, 2008) and perceived authority (Fritch & Cromwell, 2001) may suffer when a message is ambiguous. Several models of message processing also offer support for the idea that ambiguity may have a negative effect on perceptions of credibility. Sundar (2008), and Kovach and Rosenstiel (2011), noted that digital publishing complicates gatekeeping, and the agency component of the MAIN model—consistent with the findings of this study—suggests that receivers are likely to have more trouble identifying the originating messenger in online messages because of the multitude of possibilities along the message’s chain of existence. The ELM suggests that receivers make an unconscious choice between a peripheral or central route of message processing based in part on their level of involvement with the message. If involvement is low, the receiver is more likely to employ heuristics, such as a quick evaluation of sourcing on its face. It follows, logically, that ambiguity in sourcing would mean lower involvement levels, prompting the use of peripheral processing, which itself would again catalyze the use of heuristics and lower involvement in a sort of closed circle or spiral of decline. Previous research also suggests that this relationship may be amplified by the nature of online news, relative to traditional methods of news messaging, because of the absence of salience cues common to traditional delivery mechanisms (Chaiken, 1980).

H2, H3 and H4, respectively, predicted that highly aggregated, moderately aggregated and non-aggregated messages from a familiar source would be perceived as more credible than the same messages from a generic source. Under the non-aggregated condition, a significant relationship was found for the international story. An independent samples t test comparing the international story credibility scores under the no-aggregation condition found a significant difference in scores under the generic (M = 20.28, SD = 4.94) and the familiar conditions, M = 22.84, SD = 3.79; t(107) = −3.04, p < .05, two-tailed. Under the no-aggregation condition, the nature of the publication affected perceived credibility of the international story, with the story from the familiar publisher seen as more credible. The lone supported hypothesis was H4, which predicted that non-aggregated messages from a familiar source would be viewed more credible than non-aggregated messages from a generic source.

RQ3 required exploring the relationships between perceived message credibility, degree of message aggregation and media consumption habits. No significant relationships were found between aggregation levels and frequency of exposure to news or type of news source. A series of two-way analyses of variance explored the relationship between (1) credibility score, degree of aggregation and frequency of news consumption, and (2) credibility score, degree of aggregation and source of news (e.g., newspaper, television). Each analysis was conducted for combined story credibility scores, and for credibility scores for each of the two stories independently. To incorporate preferred news source, subjects’ original survey responses were collapsed from seven categories (print newspaper, newspaper on the web, television, television news on the web, social media, other web and radio) to four categories (print, web, broadcast and some combination of the three mediums).

A series of analyses showed no statistically significant relationships:

Related to the impact of degree of aggregation and subjects’ preferred news source on perceptions of (combined) message credibility;

Between preferred news source and sum of credibility or interaction effect involving aggregation condition and preferred news source;

Between the degree of aggregation and total perceived credibility;

Between preferred news source and domestic story perceived credibility, or between aggregation condition and preferred news source. This also was true for the international story.

Separately, in light of results of rotated matrix analyses, the credibility scale was split into two scales, one comprising variables that are factual in nature and a second comprising variables that are interpretive in nature. Using the interpretable components scale, an analysis found relationships between perceived international story credibility and aggregation condition under the no- and moderate-aggregation conditions. Perceived credibility increased along with frequency of exposure to news as aggregation degree advanced. Under the high-aggregation condition, however, perceived international story credibility decreased among subjects who reported more than daily news exposure. In other words, high-volume news consumers saw the high-aggregation condition international story as less credible than did moderate- and low-volume news consumers when it was assessed using interpretable criteria.

The analysis exploring the impact of frequency of exposure to news on combined stories credibility, as measured by the interpretable components credibility scale identified the significant main effect for frequency, F(2, 293) = 4.26, p < .05. The effect size was small (partial eta-squared value = .028). Post hoc Tukey HSD comparisons indicated that the mean score for the low-frequency condition (M = 14.30, SD = 2.57) was significantly different from the high-frequency condition (M = 15.59, SD = 2.78). Subjects more often exposed to news found the domestic and international stories (combined) to be more credible. The analysis exploring the impact of frequency of exposure to news on domestic story credibility, as measured by the interpretable components credibility scale found a significant main effect for frequency, F (2, 293) = 3.36, p < .05. The effect size was small (partial eta-squared value = .022). Post hoc Tukey HSD comparisons indicated that the mean for the low-frequency condition (M = 7.54, SD = 1.70) was significantly different from the high-frequency condition (M = 8.31, SD = 1.53). Subjects more often exposed to news found the domestic story to be more credible when evaluated using interpretable criteria. The analysis to explore the impact of frequency of exposure to news on international story credibility, as measured by the interpretable components credibility scale, found a significant interaction effect between frequency of news exposure and degree of aggregation, F(4, 293) = 2.63, p < .05. The effect size was small (partial eta-squared value = .035). As frequency of exposure to news increased, message credibility rose under the no- and moderate-aggregation conditions, but fell under the high-aggregation condition. The decline in perceived credibility under the high-aggregation condition is consistent with previous results. The main effect between frequency and perceived credibility also was significant, F(2, 293) = 3.1, p < .05, (.074 for factual) with a partial eta-squared value of .035, a small effect.

Finally, post hoc analysis was conducted to discover potential relationships between perceived credibility and the demographic variables age, gender, race, level of education, political affiliation and household income. The lone relationship was a small effect for age. Ages were collapsed into five categories (20–25, 26–31, 32–37, 38–43, 44–49), and the test showed a relationship between age and overall story credibility at the p < .05 level, F(4, 298) = 2.334, p = .05. To determine effect size, an eta-squared value was calculated, returning a value of 0.03, a small effect size value based on Cohen’s (1992) standards. Although the result is on the borderline standard for significance, it suggests that older consumers may be more likely to view news stories as being credible than younger ones, a possibility consistent with previous findings (Johnson & Kaye, 1998).

Discussion

The results of this research suggest some conflict between common news story aggregation practice and traditional news ethical standards. Journalists are trained to seek original sources, and to identify and credit them in news copy; failure to attribute is often considered an ethical failing (SPJ, 2014). Even with added transparency given through links, however, the mere practice of aggregation remains fraught with ethical peril as aggregators save money by linking to content they did not produce or verify but can profit from, sometimes at the expense of the original producer.

A second conflict reaches into assumptions about the relationship between transparency and credibility. This research suggests that at the message level online, following traditional news ethics standards by citing sources as an aggregation practice—an exercise in transparency and ethics—may undermine credibility rather than support it. This paper’s title uses the term “perceived credibility” as a reminder that credibility is ultimately a receiver attribute. History is filled with examples of people and organizations who did the ethically appropriate thing but were not trusted by the public, and unethical people and organizations perceived as credible.

For working journalists, the most significant finding of this study may be that the more confident message recipients are in their ability to identify the originating source, the more likely they are to believe the message. While little relevant research exists regarding ambiguity of sourcing and perceived message credibility, research regarding ambiguous messages and credibility suggests that ambiguity undermines systematic message processing and encourages the use of heuristics. Chaiken and Maheswaran (1994) found that when heuristic and systematic processing occur simultaneously, heuristic processing may bias the simultaneous systematic processing and “is most likely when message content or other individuating information is ambiguous” (p. 469). This, along with results regarding confidence in source identification and perceived credibility, suggests that in addition to clarity in messaging, sourcing clarity at the message level is an important component for credibility of aggregated news stories.

This study indicates that receivers of highly aggregated messages who were asked to identify originators consistently chose the third of three aggregators most often, followed by the first of the three and the second, respectively. Receivers also identified the bylined author and the aggregating publisher as originators even under conditions where all the information in the story was credited to aggregated sources.

The results of this study suggest that news organizations can do more to build credibility by more clearly identifying aggregated sources, and therefore giving readers ways to judge the significance of their contribution, than by changing the number or proportion of aggregated sources used. Analyses found relationships between confidence in ability to identify an originating source and perceived message credibility under multiple conditions. News organizations have sought but failed to develop industry standards or an ethical framework for aggregation. Proposals to use icons representing degree of aggregation, to manage the use of links and to create an industry council to provide guidance all have failed to gain support (Carr, 2012; Martin, 2015). Perhaps news organizations and others should make another attempt to explain how more sourcing should lead to more trust.

These findings ultimately suggest that some news consumers are confused about the original sources of news-mediated information and also standards of newsgathering, which may mean a fundamental inability to tie perceived credibility to important attributes related to “rewarding” the efforts of original news gatherers and to the quality of information. Aggregation can aid consumers by putting relevant news in a single place, and by combining relevant information from multiple sources, but the practice comes at the peril of hurting the economics of the original news source. Also, as Coddington (2019) noted, aggregation introduces more “uncertainty” as it absorbs information “without getting deep enough into individual pieces of that information to be sure of their validity” (p. 200).

The public’s inability to fully understand aggregation and sourcing ultimately could harm the commonweal. Message recipients who see news messages as not credible are less likely to pay attention to them, increasing the likelihood of uninformed or misinformed voters. Furthermore, less-credible online messengers, which presumably may transmit less-credible messages, can contribute to rising levels of inaccuracy: They may have lower editorial standards than many in legacy media. This means that, while they may be more likely to report information first, they also may be more likely to report it before determining whether it is accurate and more likely to make errors (Johnson & Kaye, 2010a). Unlike traditional mediums such as print newspapers, the murky nature of aggregated news online—with unclear provenance—necessitates “the constant need to critically assess information while consuming it” (Sundar, 2008, p. 73). Furthermore, aggregation shifts the burden of evaluating news to receivers who may not be prepared to make accurate decisions. That receivers may distrust the news is not a new idea, of course.

This research faced several limitations. While the stimuli stories were adapted from Associated Press stories with the intent of limiting emotional response and were tested in manipulation checks, the subject of one story was in world news between the time the stimulus story was selected and adapted and the time the survey was conducted. It is possible that some subjects may have been aware of more recent developments than those portrayed in the stimulus story, influencing their responses to survey questions.

While the data were collected in 2017, the results likely remain valid even as questions of media credibility persist. The gap in news media trust by political affiliation remains wide but did not grow significantly wider between 2017 and 2020 (Brenan, 2020). Moreover, the results showing no relationship between credibility and political affiliation may be related to the story choices—a local story, and a story about a nation (North Korea) disliked by a wide majority of Americans regardless of political affiliation (Jones, 2018). Using more politically charged stories, or better-known news organizations besides USA Today, might have led to different results. Such is the nature of experimental research into perceived credibility.

It is hoped this study contributes to the understanding of how news consumers become informed and how journalists may operationalize transparency in a manner that helps receivers assess information in a new digital world. Its findings lead to two broad conclusions: First, this analysis found no statistically significant relationship between degree of aggregation and perceived news story credibility, suggesting that the employment of aggregation at the message level is not likely to increase or decrease the perceived credibility of news stories. Second, this analysis found a statistically significant relationship between receivers’ confidence in their ability to identify originating sources and their perceptions of credibility. This suggests that news organizations, increasingly reliant on the internet as a delivery mechanism, can build credibility with their audience by doing a better job of identifying sources and disclosing the degree to which they inform content. Furthermore, the same technology that has fractured the industry and forced reconsideration of long-held journalistic values may be employed to reach this end.

Footnotes

ORCID iD

Chris Roberts

References

Allcott

Gentzkow

(2017). Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31(2), 211–236. https://doi.org/10.1257/jep.31.2.211

Appelman

Sundar

S. S.

(2016). Measuring message credibility: Construction and validation of an exclusive scale. Journalism & Mass Communication Quarterly, 93(1), 59–79. https://doi.org/10.1177/1077699015606057

Austin

E. W.

Dong

(1994). Source v. content effects on judgments of news believability. Journalism & Mass Communication Quarterly, 71, 973–983. https://doi.org/10.1177/107769909407100420

Barthel

(2018, August 21). 5 facts about the state of the news media in 2017. Pew Research Center. https://www.pewresearch.org/fact-tank/2018/08/21/5-facts-about-the-state-of-the-news-media-in-2017/

Barthel

Mitchell

Asare-Marfo

Kennedy

Worden

(2020, December 8). Measuring news consumption in a digital era. Pew Research Center. https://www.journalism.org/2020/12/08/measuring-news-consumption-in-a-digital-era/

Black

Roberts

(2011). Doing ethics in media: Theories and practical applications. Routledge.

Borah

(2014). The hyperlinked world: A look at how the interactions of news frames and hyperlinks influence news credibility and willingness to seek information. Journal of Computer-Mediated Communication, 19, 576–590. https://doi.org/10.1111/jcc4.12060

Brenan

(2020, September 30). Americans remain distrustful of mass media. Gallup. https://news.gallup.com/poll/321116/americans-remain-distrustful-mass-media.aspx

Bruns

(2005). Gatewatching: Collaborative online news production. Peter Lang.

10.

Bruns

(2008). The active audience: Transforming journalism from gatekeeping to gatewatching. In Paterson

Domingo

(Eds.), Making news online: The ethnography of new media production (pp. 171–184). Peter Lang.

11.

Cacioppo

J. T.

Petty

R. E.

Kao

C. F.

Rodriguez

(1986). Central and peripheral routes to persuasion: An individual difference perspective. Journal of Personality and Social Psychology, 51, 1032–1043. https://doi.org/10.1037/0022-3514.51.5.1032

12.

Carr

(2012, March 11). A code of conduct for content aggregators. The New York Times. https://www.nytimes.com/2012/03/12/business/media/guidelines-proposed-for-content-aggregation-online.html?searchResultPosition=1

13.

Chaiken

(1980). Heuristic versus systematic information processing and the use of source versus message cues in persuasion. Journal of Personality and Social Psychology, 39(5), 752–766.

14.

Chaiken

Maheswaran

(1994). Heuristic processing can bias systematic processing: Effects of source credibility, argument ambiguity, and task importance on attitude judgment. Journal of Personality and Social Psychology, 66(3), 460–473.

15.

Chung

C. J.

Nam

Stefanone

M. A.

(2012). Exploring online news credibility: The relative influence of traditional and technological factors. Journal of Computer-Mediated Communication, 17(2), 171–186. https://doi.org/10.1111/j.1083-6101.2011.01565.x

16.

Coddington

(2019). Aggregating the news: Secondhand knowledge and the erosion of journalistic authority. Columbia University Press.

17.

Coddington

(2020). Gathering evidence of evidence: News aggregation as an epistemological practice. Journalism, 21(3), 365–380. https://doi.org/10.1177/1464884918817608

18.

Cohen

(1992). A power primer. Psychological Bulletin, 112, 155–159.

19.

Curry

A. L.

Stroud

N. J.

(2019). The effects of journalistic transparency on credibility assessments and engagement intentions. Journalism. Advance online publication. https://doi.org/10.1177/1464884919850387

20.

Doree

(2007). RSS: A brief introduction. Journal of Manipulative Therapy, 15(1), 57–58. https://www-tandfonline-com-s.web.bisu.edu.cn/doi/abs/10.1179/106698107791090169

21.

Duffy

Tandoc

E. C.

Ling

(2018). Frankenstein journalism. Information, Communication & Society, 21(10), 1354–1368. https://doi.org/10.1080/1369118X.2017.1313884

22.

Eveland

W. P.

Jr. Scheufele

D. A.

(2000). Connecting news media use with gaps in knowledge and participation. Political Communication, 17(3), 215–237. https://doi.org/10.1080/105846000414250

23.

Flanagin

A. J.

Metzger

M. J.

(2000). Perceptions of internet information credibility. Journalism & Mass Communication Quarterly, 77, 515–540. https://doi.org/10.1177/107769900007700304

24.

Flanagin

A. J.

Metzger

M. J.

(2007). The role of site features, user attributes, and information verification behaviors on the perceived credibility of web-based information. New Media & Society, 9, 319–342. https://doi.org/10.1177/1461444807075015

25.

Flanagin

A. J.

Metzger

M. J.

(2008). Digital media and youth: Unparalleled opportunity and unprecedented responsibility [Monograph]. In Metzger

M. J.

Flanagin

A. J.

(Eds.), Digital media, youth, and credibility (pp. 5–28). MIT Press. https://doi.org/10.1162/dmal.9780262562324.005

26.

Fritch

J. W.

Cromwell

R. L.

(2001). Evaluating internet resources: Identity, affiliation, and cognitive authority in a networked world. Journal of the American Society for Information Science and Technology, 52, 499–507. https://doi.org/10.1002/asi.1081

27.

Gaziano

McGrath

(1986). Measuring the concept of credibility. Journalism & Mass Communication Quarterly, 63(3), 451–462. https://doi.org/10.1177/107769908606300301

28.

Gilbert

Watkins

(2020). A comparison of news databases’ coverage of digital-native news. Newspaper Research Journal, 41(3), 317–332. https://doi.org/10.1177/0739532920950039

29.

Gottfried

Shearer

(2018, September 10). News use across social media platforms 2018. Pew Research Center. https://www.journalism.org/2018/09/10/news-use-across-social-media-platforms-2018/

30.

Hilligoss

Rieh

S. Y.

(2008). Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context. Information Processing & Management, 44, 1467–1484. https://doi.org/10.1016/j.ipm.2007.10.001

31.

Hovland

C. I.

Janis

I. L.

Kelley

H. H.

(1953). Communication and persuasion: Psychological studies of opinion change. Yale University Press.

32.

Hovland

C. I.

Weiss

(1951). The influence of source credibility on communication effectiveness. Public Opinion Quarterly, 15, 635–650. https://doi.org/10.1086/266350

33.

Hurley

R. J.

Tewksbury

(2012). News aggregation and content differences in online cancer news. Journal of Broadcasting & Electronic Media, 56, 132–149. https://doi.org/10.1080/08838151.2011.648681

34.

Institute for Digital Research and Education. (n.d.). G*POWER [Computer software]. University of California, Los Angeles. https://stats.idre.ucla.edu/other/gpower/

35.

Johnson

T. J.

Kaye

B. K.

(1998). Cruising is believing? Comparing internet and traditional sources on media credibility measures. Journalism & Mass Communication Quarterly, 75(2), 325–340. https://doi.org/10.1177/107769909807500208

36.

Johnson

T. J.

Kaye

B. K.

(2010a). Choosing is believing? How web gratifications and reliance affect internet credibility among politically interested users. Atlantic Journal of Communication, 18, 1–21. https://doi.org/10.1080/15456870903340431

37.

Johnson

T. J.

Kaye

B. K.

(2010b). Still cruising and believing? An analysis of online credibility across three presidential campaigns. American Behavioral Scientist, 54, 57–77. https://doi.org/10.1177/0002764210376311

38.

Jones

J. M.

(2018, July 12). Americans’ opinion of U.S.-North Korea relations less negative. Gallup. https://news.gallup.com/poll/237125/americans-opinions-north-korea-relations-less-negative.aspx

39.

Karlsson

Clerwall

Nord

(2014). You ain’t seen nothing yet: Transparency’s (lack of) effect on source and message credibility. Journalism Studies, 15, 668–678. https://doi.org/10.1080/1461670X.2014.886837

40.

Kiousis

(2001). Public trust or mistrust? Perceptions of media credibility in the information age. Mass Communication & Society, 4, 381–403. https://doi.org/10.1207/S15327825MCS0404_4

41.

Kiousis

(2006). Exploring the impact of modality on perceptions of credibility for online news stories. Journalism Studies, 7, 348–359. https://doi.org/10.1080/14616700500533668

42.

Kovach

Rosenstiel

(2011). Blur: How to know what’s true in the age of information overload. Bloomsbury Publishing.

43.

Mahone

Wang

Napoli

Weber

McCollough

(2019, August). Who’s producing local journalism? Assessing journalist output across different outlet types. https://dewitt.sanford.duke.edu/whos-producing-local-journalism-nmrp-report/

44.

Martin

(2015). The case for curatorial journalism . . . or can you really be an ethical aggregator. In Zion

Craig

(Eds.), Ethics for digital journalists: Emerging best practices (pp. 87–102). Routledge.

45.

Masta

K. E.

(2018, January 5). Fewer Americans rely on TV news; what type they watch varies by who they are. Pew Research Center. https://www.pewresearch.org/fact-tank/2018/01/05/fewer-americans-rely-on-tv-news-what-type-they-watch-varies-by-who-they-are/

46.

McCroskey

J. C.

Teven

J. J.

(2009). Goodwill: A reexamination of the construct and its measurement. Communications Monographs, 66, 90–103. https://doi.org/10.1080/03637759909376464

47.

Melican

D. B.

Dixon

T. L.

(2008). News on the net: Credibility, selective exposure, and racial prejudice. Communication Research, 35, 151–168. https://doi.org/10.1177/0093650207313157

48.

Metzger

M. J.

(2007). Making sense of credibility on the web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology, 58, 2078–2091. https://doi.org/10.1002/asi.20672

49.

Metzger

M. J.

Flanagin

A. J.

Medders

R. B.

(2010). Social and heuristic approaches to credibility evaluation online. Journal of Communication, 60, 413–439. https://doi.org/10.1111/j.1460-2466.2010.01488.x

50.

Meyer

(1988). Defining and measuring credibility of newspapers: Developing an index. Journalism & Mass Communication Quarterly, 65(3), 567–574. https://doi.org/10.1177/107769908806500301

51.

Mitchell

Gottfried

Kiley

Matsa

K. E.

(2014, October 21). Political polarization & media habits: From Fox News to Facebook, how liberals and conservatives keep up with politics. Pew Research Center. http://www.journalism.org/2014/10/21/political-polarization-media-habits/

52.

Mitchell

Gottfried

Shearer

(2017, February 9). How Americans encounter, recall and act upon digital news. Pew Research Center. http://www.journalism.org/2017/02/09/how-americans-encounter-recall-and-act-upon-digital-news/

53.

Molyneux

Coddington

(2019). Aggregation, clickbait and their effect on perceptions of journalistic credibility and quality. Journalism Practice, 14(4), 429–446. https://doi.org/10.1080/17512786.2019.1628658

54.

Newhagen

Nass

(1989). Differential criteria for evaluating credibility of newspapers and TV news. Journalism & Mass Communication Quarterly, 66(2), 277–284. https://doi.org/10.1177/107769908906600202

55.

Petty

R. E.

Cacioppo

J. T.

(1986). Communication and persuasion: Central and peripheral routes to attitude change. Springer.

56.

Pew Research Center. (2016, September 13). The parties on the eve of the 2016 election: Two coalitions, moving further apart. https://www.pewresearch.org/politics/2016/09/13/the-parties-on-the-eve-of-the-2016-election-two-coalitions-moving-further-apart/

57.

Pew Research Center. (2019a, July 23). Digital news fact sheet. http://www.journalism.org/fact-sheet/digital-news/

58.

Pew Research Center. (2019b, September 11). Key findings about the online news landscape in America. https://www.pewresearch.org/fact-tank/2019/09/11/key-findings-about-the-online-news-landscape-in-america/

59.

Pew Research Center. (2019c, June 25). Local TV news fact sheet. https://www.journalism.org/fact-sheet/local-tv-news/

60.

Pew Research Center. (2019d, July 9). Newspapers fact sheet. https://www.journalism.org/fact-sheet/newspapers/

61.

Roberts

(2010). Correlations among variables in message and messenger credibility scales. American Behavioral Scientist, 54(1), 43–56. https://doi.org/10.1177/0002764210376310

62.

Singer

J. B.

(2003). Who are these guys? The online challenge to the notion of journalistic professionalism. Journalism, 4(2), 139–163. https://doi.org/10.1177/146488490342001

63.

Society of Professional Journalists. (2014). SPJ code of ethics. https://www.spj.org/ethicscode.asp

64.

Sundar

S. S.

(2008). The MAIN model: A heuristic approach to understanding technology effects on credibility. In Metzger

Flanagin

(Eds.), Digital media, youth, and credibility (pp. 73–100). MIT Press. https://doi.org/10.1162/dmal.9780262562324.073

65.

Sundar

S. S.

Nass

(2001). Conceptualizing sources in online news. Journal of Communication, 51, 52–72. https://doi.org/10.1111/j.1460-2466.2001.tb02872.x

66.

U.S. Census Bureau. (2018). QuickFacts [Online population data]. https://www.census.gov/quickfacts/fact/table/US/PST045217

67.

Wathen

Burkell

(2002). Believe it or not: Factors influencing credibility on the web. Journal of the American Society for Information Science and Technology, 53, 134–144. https://doi.org/10.1002/asi.10016