Social credibility online: The role of online comments in assessing news article credibility

Social Credibility Online

People are prone to relying on beliefs and opinions of others when fulfilling a variety of actions online, when shopping on online retailers, when looking for a restaurant to dine at or a movie to watch and when choosing personal and home electronics, a specific retailer or the best stocks to invest in. ⁶ Research has found that social information and social influence online are especially significant for credibility assessments. People consider the use of user-generated feedback systems, testimonials and reputation systems as ways to help them make credibility evaluations. They have the habit of checking the reviews online when making evaluations about products or when verifying information needed for offline purposes such as health care, hobbies or other special interests. They consider tips provided by real people to be “highly useful,” “credible” and “trustworthy” and think that knowledge derived from individuals’ uses or practices is often more reliable than information provided by a company or manufacturer. ⁷

Thus, the concept Social Credibility Assessment (SCA) reflects the “reliance on the advice or input of other people, or knowledge of their actions when determining the credibility of information online.” ⁸ Feedback can come from people known or unknown to the person evaluating the information, and is used in a manner that influences the evaluator’s credibility judgments. The reliance on social cues for credibility assessments online stems from a belief that standard conventions of determining credibility break down in cyberspace as abundance of sources makes traditional models of information gatekeeping untenable ⁹ and defer people to external sources of knowledge for trust assessments, ¹⁰ among which is a large scale of user-generated contents. Although the SCA concept is recent (the term mentioned in a 2014 study ¹¹ ), other earlier studies also suggested that individuals can be influenced by others when evaluating online information. Focus group data from 109 participants across the United States showed that most participants relied on others to make credibility assessments online and that they frequently used user-generated content to navigate and evaluate information online. ¹² A 2011 survey and experimental study showed that people rely heavily on user-generated ratings to evaluate the credibility of commercial information they find online. The study found, relative to other credibility cues, that ratings were ranked highly and were less important only than other attributes, such as website security and the currency and completeness of the information. ¹³ In Metzger and Flanagin’s studies, ¹⁴ which used survey data collected over three years, official “stamp of approval” or a recommendation from someone you know was one of the most frequent criteria used for gauging credibility of online information. Also, explicit recommendations were found in Knobloch-Westerwick et al.’s ¹⁵ study to influence people’s choice of online articles, and the strength of the recommendation positively predicted duration of exposure to the article. Another study found that individuals are susceptible to social influence from online groups. Walther et al. ¹⁶ discovered that message postings on YouTube videos about antimarijuana public service announcement (PSA) influenced the evaluations of the PSA. Number of followers was also discovered as an important social cue online, but in a curvilinear manner, such that having too many or too few connections results in lower judgments of expertise and trustworthiness. ¹⁷

This study uses the notion that online audiences rely on the opinions of others when making evaluations of online information and argues that readers’ comments on online news stories can be considered one of the social cues important for credibility evaluations. The social component of online comments is reflected through enabling readers to express their opinions about the news article and give their suggestions in a direct and open manner. This opens the communication between the users and the creators of the news content and allows for readers’ input to be heard, a role once reserved to the letters to the editor. ¹⁸

Information Processing Online

Studies have found that Internet users do not engage in a deep evaluation of the message content but rather use short cuts when assessing the content of online messages. ¹⁹ This heuristic path of information processing is consistent with numerous theories of limited information processing that propose that humans do not have the unlimited cognitive capacity to process messages, nor the time and abundant resources to concentrate on all aspects of the message, but instead choose to focus only on their salient aspects. ²⁰ Recent studies suggest that the social elements embedded in different interactive and user-generated features of online news, such as readers’ comments on online news stories, might act as a new, powerful heuristic in judging the credibility of online information. ²¹ Researchers coined the terms endorsement-based heuristics, ²² conferred credibility, ²³ consensus heuristic ²⁴ and bandwagon heuristic, ²⁵ to suggest that “people are inclined to perceive information and sources as credible if others do so.” ²⁶

Consistent with findings about the importance of social cues in credibility evaluations online ²⁷ and theories of peripheral and limited information processing, ²⁸ this study contends that readers’ comments affect the evaluations of credibility of online news stories. This study further posits that within online comments, Internet users use an additional heuristic, source expertise, when making credibility assessment about news. Source expertise has been an accepted credibility heuristic in the literature, ²⁹ but its effects vary depending on the relevance of the topic. ³⁰ In this study, the topic is kept constant across the experimental conditions, while source expertise is manipulated to measure its impact on article credibility. In addition, previous studies have found that while technological factors play a role in evaluations of index-type of online news sources (e.g., news.google.com, news.yahoo.com), traditional sources of credibility, such as expertise and trustworthiness, become more important if online news is presented in a form of traditional newspaper on the Internet (such as online versions of USA Today or The New York Times). ³¹ As in this study, in which the news story to which the participants were exposed was written in a traditional form but for the Web, it is hypothesized that source expertise in online comments positively influences perceptions of news credibility.

However, the relationship between source expertise and article credibility can be moderated by certain variables. For example, it can be assumed that people who follow news more frequently will be able to better discern source expertise in media messages than people who are not avid news followers, which could consequentially affect credibility perceptions. In addition, previous studies have found that media skepticism can be associated with news exposure. ³² Based on this, it is hypothesized that an interaction effect exists between source expertise and news media use.

Beyond the expertise of the commenter, previous research has found that the valence of the comment can also affect credibility assessments. ³³ These findings suggest that when engaged in evaluating valence of online content, Internet users might process information more systematically, which involves a thoughtful process in which users actively scrutinize the message and elaborate on its content. Based on these findings, it is possible to assume that opinion valence in readers’ comments can also affect the perceptions of article credibility. To further examine this possibility, this study examines the impact of comments’ valence on credibility perceptions by asking the following:

Although previous research is not unanimous on the impact of message valence on people’s assessments of the online content, it can be assumed that the opinion valence expressed in online comments could interact with previous attitudes toward genetically modified organisms (GMOs) in affecting individuals’ assessments of an online news article about GMOs. Research has found that receiver’s initial attitude and the message’s advocated position, considered jointly, will influence the evaluation. ³⁴ Based on this, it is hypothesized that:

Method

A randomized experiment was used to test the above hypotheses and answer the research question. The experiment utilized a 2 (positive vs. negative) × 2 (expert sources vs. nonexpert sources) factorial structure. Students enrolled in communication and journalism courses at a large university in the southern United States were invited to participate in the study in return for extra credit. Data were collected between September 16 and September 28, 2014. A total of 256 participants completed the experiment (M age = 19.5, SD = 2.3; 79 percent female, 21 percent male; 84.6 percent white, 4.7 percent black, 2.4 percent Hispanic, 3.2 percent Asian, 5.2 percent other/not answer). The experiment was completed using Qualtrics online survey software.

Procedures

Before starting the experiment, participants were provided with the appropriate informed consent documentation. After agreeing to participate, participants were asked to complete a short prestimulus questionnaire. The questionnaire included numerous items, including measures of media use and attitudes toward GMOs. This study is part of a larger research project, so additional items, beyond the scope of this study, were collected in the prestimulus and poststimulus questionnaires, which will be the focus of future studies and are not addressed in this article. On completing the prestimulus questionnaire, the participants were asked to read a short news story (see Figure 1). The stimulus was designed to look like a news story from a legitimate news source, such as CNN.com. It included a headline, main photo and short news story (i.e., 287 words) about the use of GMOs in orange crops. The average participant spent about two and a half minutes reading the article (M = 160.42 seconds, SD = 1,283.33 seconds). It was assumed that participants who spent less than 5 seconds or more than 600 seconds on the article were not engaged with the research instrument and were removed from further analyses. This reduced the total sample size to 196 participants.

OPEN IN VIEWER

Figure 1

Stimulus Material: One of the Experimental Conditions, in Which Source Expertise Was High and Opinion Valence Positive

Embedded in the news story, to the left of the main content, was a box labeled “Reader’s Perspective.” It included three comments about the story. This style of embedding comments next to the story, instead of where they are traditionally found (i.e., the bottom of the story), was inspired by an award-winning piece of science journalism recently published by The New York Times. The article, “A Race to Save the Orange by Altering Its DNA” by Amy Harmon, discussed the use of GMOs in orange farming and included “Reader Perspective” boxes in the story sidebar. This study’s researchers, after reading Harmon’s story, believed this style of displaying comments could have profound effects on individuals’ perceptions of credibility, especially when reading about a controversial topic such as GMOs.

It was within the “Reader’s Perspective” box that the two experimental manipulations, which serve as independent variables, were embedded. The first manipulation was of comments’ sources. In the high source expertise condition, comments were attributed to professional sources, such as “John Williams, Director of California Institute of Agriculture.” In the low source expertise condition, comments were attributed to generic Twitter or Facebook users, such as “from Twitter user @JohnNYC.” The second manipulation was of the valence of opinions expressed in comments. In one condition, comments were generally positive toward the story and GMOs. In the second condition, comments were generally negative toward the story and GMOs. To ensure accuracy, information in all the comments was crafted with help from a scientist knowledgeable about GMOs. Participants were randomly assigned across conditions. Note, importantly, that between conditions, the article, the art and headline were held constant. The only component changed was content in the “Reader’s Perspective” box.

After reading the story, participants were asked to complete a poststimulus questionnaire. It included items about the participant’s perceptions of the credibility of the article they read. After they completed the poststimulus questionnaire, participants had completed their involvement in the experiment and were thanked for their time.

Measures

Again, this study is concerned with the relationship between two independent variables (i.e., source expertise and opinion valence), two moderating variables (i.e., news use and attitudes toward GMOs) and one dependent variable (i.e., article credibility). The independent variables were operationalized through the manipulations described above. The moderating and dependent variables were measured via self-reported items in a questionnaire.

News use was measured with a five-item scale asking the participant how often they engaged with various media (e.g., “Please indicate how many days per week you read online news”). The participants answered on an 8-point scale ranging from 0 days to 7 days, and the values for the five items were averaged (Cronbach’s α = 0.52, M = 3.21, SD = 1.26).

Attitudes toward GMOs were measured using a four-item scale that has been shown to be valid and reliable in previous scientific research about GMOs (Gaskell et al., 2006). Participants were asked to rate their levels of agreement on a 7-point Likert scale (1 = strongly disagree) with statements about GMOs, such as “Genetically modified foods are useful for society.” Responses for the four items were averaged (Cronbach’s α = 0.84, M = 3.93, SD = 1.39).

Finally, the dependent variable, article credibility, was measured using an established 8-item scale. ³⁵ The scale asked individuals to rate their levels of agreement with statements about the article, such as “The article was fair.” Participants answered on the same 7-point Likert scale as above, and the responses were averaged (Cronbach’s α = 0.80, M = 3.63, SD = 0.81).

Results

The above hypotheses and research question were addressed using a hierarchical linear regression model predicting article credibility. Linear regression was used instead of the more traditional method, ANOVA, in order to simplify the interpretation of the interactions between the manipulated variables and continuous individual difference variables (i.e., news use and attitudes). The full results can be seen in Table 1.

OPEN IN VIEWER

Table 1

Models Predicting Article Credibility

	Without interactions	With interactions
Source expertise	0.26 (.13)**	−0.77 (.44)*
Opinion valance	0.03 (.13)	0.19 (.40)
News use	0.09 (.05)*	−0.05 (.07)
Attitude toward genetically modified organisms	0.05 (.05)	−0.07 (.06)
News use × Expertise	—	0.24 (.10)**
Attitude × Valence	—	−0.03 (.09)
R 2	.05	.09

Note. Hierarchical regression models predicting perceptions of article credibility. Interaction terms were entered in the second block of the model. The change in R² of approximately .04 was significant at the .05 level. Entries are unstandardized beta coefficients from the given block. Parentheses are standard errors.

*

p < .10. **p < .05.

H1 predicted a positive relationship between source expertise and credibility. Specifically, it was argued that individuals who saw comments from professional sources would judge the article as more credible than individuals who saw comments from Twitter-based sources. This hypothesis was supported (b = 0.26, p < .05). Individuals judged the article more credible when the public comments embedded in the sidebar of the story were from professional sources (e.g., scientists) than when the comments included were from generic Twitter users. This result is especially surprising considering the conservative nature of the manipulation. Across the source expertise conditions, the content of the story and comments were held constant. The only part that changed was the name and title of the commenter.

The second hypothesis further probed the relationship between source expertise and article credibility, positing an interaction between source expertise and news media use, resulting in high-frequency news users being more discerning in their assessment of the article’s credibility. This hypothesis was also supported in data (b = 0.24, p < .05). As can be seen in Figure 2, high-frequency news (i.e., one standard deviation above the sample mean) users found the article more credible than low-frequency news users (i.e., one standard deviation below the sample mean) in the high source expertise condition, and they found the article less credible than low-frequency news users in the low source expertise condition.

OPEN IN VIEWER

Figure 2

Interaction between News Use and Source Expertise on Article Credibility

RQ1 asked whether the valence of the opinions included in the public comments shaped individuals’ assessment of the credibility of the article. No significant difference (b = 0.03, p = .81) was found in perceptions of article credibility between the positive (M = 3.61, SD = 0.86) or negative comment (M = 3.62, SD = 0.79) conditions. Therefore, it can be said that the valence of the comments included in the sidebar did not shape individuals’ perceptions of the credibility of the article.

To further probe this relationship, H3 was advanced. This hypothesis argued that individuals’ level of agreement with the public comments would affect their perceptions of article credibility. For example, individuals with positive attitudes toward GMOs would likely rate an article with positive comments as more credible than an article with negative comments. The hypothesis was tested by looking at the interaction of attitudes toward GMOs and the opinion valence manipulation (i.e., Attitude × Valence in Table 1). This hypothesis was not supported either (b = −0.03, p = .71). While individuals did use the source of the comments to access the credibility of the article, they did not use the valence of the comments, even when those opinions disagreed with the reader’s attitudes toward GMOs.

Post Hoc Test

After reviewing results, the coauthors ran post hoc tests to try to explain why H3 was not supported by data. In gathering data for H3, the interaction between the continuous variable, attitudes toward GMOs and the manipulated variable, valence of opinions was tested. While this tested whether there was a linear moderating effect, it might not provide the proper nuance to fully understand the relationship between these variables. Specifically, individuals with strong attitudes could have very different opinions than individuals with weak attitudes. Furthermore, individuals’ assessment of the credibility of the article could be different within these two groups. In order to test this possibility, an ANOVA model was created.

The ANOVA model examined only individuals who had a generally positive view of GMOs (i.e., above the mean value for the sample). Within that group (M = 4.95, SD = 0.90), a mean-split was used to create a strong (i.e., above 4.95) and weak attitude group (i.e., below 4.95). This new variable, along with the manipulated opinion valence variable, was entered into an ANOVA model predicting article credibility. While neither attitude strength nor opinion valence had a significant main effect on article credibility, the interaction term was marginally significant, F(1, 69) = 2.77, p = .10. Individuals who had strong attitudes toward GMOs rated the credibility of the article higher than individuals with weak attitudes in the positive valence condition and lower than individuals with weak attitudes in the negative valence condition.

The same analysis was run in the group of individuals who had a generally negative view of GMOs. Again, the main effect for opinion valence and attitude strength were nonsignificant, while the interaction term was significant, F(1, 69) = 6.42, p < .05. In this case, the interpretation is not clear. Individuals with weak attitudes judged the article with negative comments as most credible, while individuals with strong attitudes judged the article with positive comments as most credible. While this is the same pattern of effects as in the first analysis, it is the opposite of what was expected. One would expect individuals with the strongest negative-leaning attitudes to judge the article with negative comments as more credible than an article with positive comments. The divergent and confusing results of these analyses provided more questions than answers regarding the role of attitude strength in perceptions of article credibility. They also explain why H3, which predicted a linear interaction between opinion strength and opinion valence, was not supported.

Discussion

Again, the purpose of this study was to test the effects of source expertise and opinion valence in readers’ comments on an online news story about GMOs on article credibility. Based on its results, it seems reasonable that simply identifying the source as an expert or not (average Twitter user) can serve as a heuristic cue when assessing credibility.

These results support findings of previous studies, which indicated that Internet users mainly use peripheral or heuristic route of information processing when assessing information online. ³⁶ Not engaging in laborious thinking about the content of online news, participants in this study used source expertise as a mental shortcut to evaluate the credibility of online news. This is further supported by the finding that the valence of the comments did not influence the credibility assessment, suggesting that the central route of information processing has not been activated. This contradicts the findings that suggested that when evaluating the valence of online comments and reviews, Internet users engage in more systematic information processing by scrutinizing the number and the possible hidden meaning of each comment. ³⁷

This study’s results may be explained by the fact that the participants were neither motivated (motivation was not manipulated in the experimental design) nor had the ability to further engage in a deeper analysis of comments content (as the number of comments was relatively small, and no condition mixed positive and negative comments). (This study was also limited by a convenience sample overwhelmingly comprising female, young, white university students and, therefore, is not representative of even the U.S. population. Its results cannot be generalized beyond its participants or this experiment’s particulars.)

Future studies should expand the experimental design to test the mixed effects of heuristic and central route of information processing on credibility perceptions online. Furthermore, post hoc tests of this study found that individuals with weak negative GMO attitudes judged the article with negative comments as most credible, while individuals with strong negative attitudes judged the article with positive comments as most credible. This goes in the opposite direction of what would be expected, ³⁸ suggesting that theoretical models that treat individuals’ previous attitudes in a linear fashion might be evaluated further. Researchers have shown that there is considerable variability in the degree to which attitudes predict behavior ³⁹ and have specified conditions under which attitudes predict behaviors. ⁴⁰ Since literature on message credibility has dealt more with how credibility perceptions influence attitudes than vice versa, ⁴¹ future studies are needed to explain the role of previous attitudes in how users decide to believe the message.

In addition, this study found that Internet users are indeed led by the opinion of others when assessing the credibility of online content. This indicates that social heuristics have become increasingly important in the era of social computing. This study shows that when making evaluations of Internet content, users reach for the opinion of their peers, when they need advice not only on where to dine, which movie to watch or what products to buy but also on how to judge news content. This may suggest that social aspects online may have an impact not only on immediate purchasing behaviors but also on the formation of credibility judgments about important social, economic and political issues.

In addition to contribution to social aspects of credibility, online comments may have important implications for the industry. Integrating comments increases the utility of the comments to other readers, “highlight[s] the most insightful perspectives” and shows that commenters are valued for their input. ⁴² This research supports industry decisions to integrate identifiable comments, especially when the perspectives come from professional experts. Unlike comments at the end of an online news story, embedded comments within the text changes multiple aspects of the online news environment. First, it brings other readers’ opinions directly into the story context. Second, the likelihood that a reader will attend to the public commentary instead of ignoring all comments as many readers do increases, as comments are made easily accessible, which could have an effect on credibility. Third, placement gives prominence to specific information and commenters; thus, future research should consider how readers view the credibility of the comments specifically. However, both the comments and commenters individually can hold varying levels of credibility. ⁴³ Comments provided by real people personalize information and are useful as “credible,” “trustworthy” and “reliable” in the face of information from entities with stakes in the topic, such as manufacturers or advertisers. ⁴⁴ This study showed that the story with embedded comments from experts was more credible. Additional investigations should consider alternative manipulations (e.g., political orientation, gender or race, in-group/out-group) to elaborate how commenter identity or expertise contributes to more global perceptions of source credibility.

Finally, it has to be acknowledged that industry has adopted numerous filtering processes and identification policies to protect public comment spaces (e.g., algorithmic and human filtering systems for detecting inappropriate content, ranking software for placement of more popular contributors or comments above those deemed less valuable;, requirements of commenters’ sign with real names, “guiding” questions and prioritization by “showing examples of ‘good’ comments”). ⁴⁵ The choice of how and if comments are embedded may reflect an agenda-setting bias, such that adding readers’ perspectives may cue readers that others also believe that topic is important. ⁴⁶ The selection criteria may also prefer comments that help frame a story or issue in a particular way (i.e., organizations favoring comments supportive of its ideological bias). Largely, as comments are singled out and highlighted, they are attributed some level of importance. This analysis was limited to comparing two different production choices. Future research is encouraged to explore how other nuances within this industry practice differ from user-controlled navigation and the effects differential practices have on reader perceptions.